SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse–Linear Attention
Authors: Jintao Zhang
Haoxu Wang
Kai Jiang
Shuo Yang
Kaiwen Zheng
Haocheng Xi
Ziteng Wang
Hongzhou Zhu
Min Zhao
Ion Stoica
Joseph E. Gonzalez
Jianfei Chen
Jun Zhu
Venue: ICLR 2026 Poster (2026)
Keywords: sparse attention
efficient attention
video diffusion model
video generation
diffusion transformer
Primary area: generative models
TL;DR: SLA: a trainable attention method that fuses sparse and linear attention to accelerate diffusion models.

Abstract: In Diffusion Transformer (DiT) models, particularly for video generation, attention latency is a major bottleneck due to the long sequence length and the quadratic complexity. Interestingly, we find that attention weights can be decoupled into two matrices: a small fraction of large weights with high rank and the remaining weights with very low rank. This naturally suggests applying sparse acceleration to the first part and low-rank acceleration to the second. Based on this finding, we propose SLA (**S**parse-**L**inear **A**ttention), a trainable attention method that fuses sparse and linear attention to accelerate diffusion models. SLA classifies attention weights into critical, marginal, and negligible, applying $\mathcal{O}(N^2)$ attention to critical weights, $\mathcal{O}(N)$ attention to marginal weights, and skipping negligible ones. SLA combines these computations into a single GPU kernel and supports both forward and backward passes. With only a few fine-tuning steps using SLA, DiT models achieve a $\textbf{20x}$ reduction in attention computation, resulting in significant acceleration without loss of generation quality. Experiments show that SLA reduces attention computation by $\textbf{95}$\% without degrading end-to-end generation quality, outperforming baseline methods. In addition, we implement an efficient GPU kernel for SLA, which yields a $\textbf{13.7x}$ speedup in attention computation and a $\textbf{2.2x}$ end-to-end speedup in video generation on Wan2.1-1.3B. The code is available at https://github.com/thu-ml/SLA.

Published as a conference paper at ICLR 2026
SLA: BEYOND SPARSITY IN DIFFUSION TRANSFORM- ERS VIA FINE-TUNABLE SPARSE-LINEAR ATTENTION
Jintao Zhang ? Haoxu Wang', Kai Jiang' Shuo Yang", Kaiwen Zheng', Haocheng Xi', Ziteng
Wang' Hongzhou Zhu', Min Zhao', lon "ooica",Jssepp" E. Gonzalez", Jianfei Chen', Jun Zhu 'Dept. of Comp. Sci. & Tech., BNRist Center, Tsinghua-Bon -Bsch MI. Center, Tsinghua University PUC Berkeley
jianfeic, dcszj]@tsiigg sinnhuua. edu. cn
ABSTRACT
In Diffusion Transformer (DiT) models, particularly for video generation, atten- tion latency a major bottleneck due to the long sequence length and the quadratic complexity. Interestingly, we find that attention weights can be decoupled into two matrices: small fraction of large weights with high rank and the remaining weights with very low rank. This naturally suggests applying sparse acceleration to the first part and low-rank acceleration to the second. Based on this finding, we propose SLA (Sparse-Linea Attention), a trainable attention method that fuses sparse and linear attention to accelerate diffusion models. SLA classifies attention weights into critical, marginal, and negligible, applying ((N2) attention to crit- ical weights, (N) attention to marginal weights, and skipping negligible ones. SLA combines these computations into single GPU kernel and supports both forward and backward passes. With only a few fine-tuning steps using SLA, DiT models achieve about 20x reduction in attention computation, resulting in signif- icant acceleration without loss of generation quality. Experiments show that SLA reduces attention computation by about 95% without degrading end-to-end gen- eration quality, outperforming baseline methods. In addition, we implement an efficient GPU kernel for SLA, which yields a 13.7 7 speedup in attention compu- tation and x end-to-end speedup in video generation on Wan2.1-1.35 The code is available at aftttps:: thu... com/ thm-L
INTRODUCTION
Among the operations in Transformers, attention (Vaswani et al. 2017) is the only one with quadratic computation complexity, while others mostly scale linearly with the sequence length N. In Diffu- sion Transformer (DiT) models (Peebles & Xiel (022). especially for video generation, attention becomes the primary computational bottleneck, as the sequence length typically ranges from 10K to 100K. Reducing the cost of attention is therefore critical for improving the efficiency of DiT
models. Existing efficient attention methods for DiTs fall into two main categories: (1) numerous
sparse attention methods (Li et al. 2025 Zhang et al.) 2025b Xi ct al., 2025; Yang al. 2025a
Zhang et al. 2025e Wu et al.] 2025 Shen et al. 2025 Hassani et al. 2023] Liu et al. 2025
which
compute only subset of attention scores, and (2) few linear attention methods (Xie et 2024
Zhu et al. 2025). which reformulate the operation to achieve O(N) complexity.
Limitation. Despite recent progress, both approaches face challenges in substantially reducing attention computation: (L1) Linear attention methods often fail in practice especially on video diffusion models. Existing work on linear attention in diffusion is rare and primarily limited to image generation. Our experiments show that when applied to diffusion models, particularly video generation, linear attention severely degrades video quality. (L2) Sparse attention methods rarely achieve very high sparsity and require a considerable fraction of the full complexity of attention. In practice, they typically reach only 40 60-600% sparsity for sequence length below 50K. Although some recent works (Yang et al., Li et al. 2025) report sparsity of 80-85%, such results are obtained on very long sequences (e.g.. (00K-300K) where achieving high sparsity easier.
"oorreponningg author.

---

Published as a conference paper at ICLR 2026
Key Observation. We find that attention weights in diffusion transformers can be decomposed into two matrices: a small fraction of large weights with high rank and a large fraction of the remaining weights with extremely low rank. This explains why sparse attention or linear attention alone cannot achieve satisfactory results and naturally suggests applying sparse acceleration to the first part and low-rank acceleration the second.
Our Method. Based on the observation above, we propose SLA, a trainable hybrid sparse and linear attention for DiT models. Specifically, attention weights are partitioned into blocks and dynamically classified into three categories: critical, marginal, and negligible. Critical blocks are computed exactly using FlashAttention, negligible blocks are skipped, and, unlike existing methods, marginal blocks are processed with linear attention. This design allows sparsity to increase dramatically (e.g., 70%-+95% while maintaining accuracy. Since linear attention is computationally negligible, costing less than 0.5% of full attention in video generation models, SLA is several times faster than sparse attention alone. Furthermore, we implement efficient forward and backward passes for SLA. With few steps of fine-tuning, SLA significantly reduces the computation complexity and latency of attention while preserving the quality of the generation results.
SLA reduces attention computation by about 95% without degrading video generation quality, even at a moderate sequence length of 30K, which the sequence length in Wam2.1-1.33 In addition, implementation achieves a 13.7x speedup in the attention kernel and a 2.2x end- to-end acceleration for video generation, where the attention time becomes almost negligible. SLA consistently surpasses baselines in both generation quality and efficiency.
Result.
Contribution. We summarize our main contributions as follows. (1) We find that the attention weights in diffusion models can be perfectly decomposed into two parts: highly sparse matrix with high rank and a dense matrix with very low rank.
(2) We propose the first attention method that effectively fuses sparse attention and linear attention. (3) Our method achieves about 95% attention sparsity, corresponding to approximately a 20x reduction in attention computation, while maintaining video generation quality. (4) We implement efficient GPU kernels for SLA.
2 PRELIMINARY
2.1 BLOCK SPARSE ATTENTION
Given queries, keys, and values K, V R.Wxd, the standard attention computes the score matrix = QKT / Vd and the attention weights p Softmax(S) obtain the output O PV. This inefficient for large N as it requires ((N2d) operations. The idea of sparse attention is to reduce computation by applying a mask M E {0, 1| to the attention weights: P <- P @ M, where @ is the element-wise product. A common strategy is to choose a threshold and set Mij if Ps > T. For entries with M,j- 0, the multiplications Q,KJ and P.yV, can be skipped, where Q: Qii.:), K3 = K[. V, = V[j.=l
However, element- wise sparse attention is inefficient on modern GPUs. Practical implementations such as FlashAttentiin (Dao) (023) first partitions Q, K, V,S.P. M into blocks {Q:), (K J. [V1). [S. (P.j), (Maj). where Q₂ e Rd Kj, V, E RR and Sij. Piy: M.j E b xbg, Each block mask M,y fully filled with either or 1, and we skip the computations of Q and P.j V if M. =0.
(Da3) operate at the block level. Specifically, the sparse FlashAttention
2.2 LINEAR ATTENTION
Linear attention methods reduce the complexity of standard attention from ON2d to O(N.). A key idea to decouple the softmax operation by introducing a feature map Φ(-) applied to and (()))(K)
K. Specifically. it replaces the attention weights in standard attention with rowsum (d(Q))(K) This reformulation enables reordering of the matrix multiplications: instead of explicitly computing the attention weights, it first computes ó σ(K) V, and then applies this intermediate result to o(Q):
(QQ)H
H == Z = rowsum(b(k)) E dxl O Ø(Q)Z'
2

---

Published as a conference paper at ICLR 2026
The mapping 6(-) is usually activation function (e.g., ELU + or ReLU CCeeert et al. 2016 Xavier et al. 2011).. This formulation avoids explicitly constructing the N matrices S and achieves linear computational complexity.
3
MOTIVATION AND ANLYSIS
Sparsity - 0%
Sparsity = 45%
Sparsity - 92%
Average Va ue 1/N
Attention Weights
1.2
(45,4%)
(46.5%)
Frequency
0.8-
Marginal
0.6
0.4
Attention Output
PV= =V=
PV
1%%)
0.2
Critical
0.0
[0, 1((200N))
(3I(10ON), 1/N)
(2/N.
Frequency of Attention Weights
Error
0%
Error
Error 33%
Figure 1: The left figure shows typical distribution of attention weights sampled from the Wan2.1 model. The right figure shows the accuracy of sparse attention with different sparsity.
3.1 MOTIVATION SLA
Due to the softmax operator, the attention weights P lie in [0, 1] with each row summing to 1. Furthermore, because of the exponential scaling in softmax, only a small fraction of entries P are relatively large, while the vast majority are close to zero. Figure (left) shows the typical distribution of attention weights sampled from the Wan2.1 model (Wan et. 2025). We highlight two key observations: (1) Only about 8.1% of the weights are larger than the average value 1/N. (2) A considerable proportion of weights are extremely small. In our case, approximately 45% fall below 1/(100N). As shown in Figure | (right), skipping these smallest 45% of weights in sparse attention (i.e.. setting the corresponding entries in M to 0) introduces a relative L1 error of less than 3% compared to the full attention output. In contrast, retaining only the largest 8.1% of weights (sparsity = 92%) leads to a sharp increase in error, reaching about 33%. This explains why existing sparse attention methods struggle achieve sparsity beyond 90%.
The intermediate values between 1/(100N) and 1/N (the yellow column in Figure present a dilemma: omitting them introduces significant accuracy loss, yet computing them with full attention causes a great decrease in sparsity. Fortunately, these values are far less critical than the largest ones. This finding motivates us to categorize the attention weights into three types: critical, marginal, and negligible. For critical weights. we use sparse FlashAttention to compute the output as they dominate the attention distribution; For negligible weights, we skip the computation; For marginal weights, we employ linear attention method to reduce the computational complexity to O(N and enhance the performance sparse attention.
Full Attention
Linear Only
X
Sparse Only X
SLA
(Sparsity 0%)
(Sparsity 100%)
(Sparsity
(Sparsity 95%)
Figure 2: Video generation examples Wan2.1 fine-tuned with full attention, linear attention, sparse attention, and SLA. SLA could achieve a high sparsity of 95% and lossless video quality.
Empirical results. In Figure 2 we present some videos generated by Wan2.1 fine-tuned with different attention methods: using only linear attention, sparse attention with 90% sparsity, and SLA with 95% sparsity. Note that the computational complexity of SLA at 95% sparsity nearly half that of 90% sparse attention, since the cost of linear attention is almost negligible. For example, in the Wan2.1 model, linear attention accounts for less than 0.5% the cost full attention. These empirical results show that SLA significantly outperforms the other two methods in video quality.

---

Published as a conference paper at ICLR 2026
3.2
SEPARATING ATTENTION WEIGHTS: SPARSE FEW, LOW-RANK MANY
Attention Weights
Top-8% Weights
Bottom 92% Weights
Sparse Attention
Low-Rank
(Linear) Attention
10
10
10t*
10
10 10
30
10'
10t4
30
101H
10
Top Top-8%
Bottom- 92%,
Figure 3: Decomposition of attention weights. We sample attention weights from the Wan2.1 model: the left figure shows the full weights, the middle the top 8%, and the right the bottom 92%.
Observation. As shown in Figure 3 full attention weights can be decoupled into two parts: (I) small subset {< 10%) with rank comparable to full attention, and (2) a large subset (> 90%) with very low rank. Since the methods for accelerating attention focus mainly sparsity low-rank structure, this suggests a natural and elegant strategy: apply sparse attention to the first part and low-rank approximation the second.
Previous failures of linear attention are largely due to the high rank full attention weights (Fan et al. 2025), while linear attention is restricted to rank most d. Figure (left) illustrates this with a typical example using the notion of stable rank (Rudelson & Vershynin 2006). We observe that after removing the top values in the attention weights P, the remaining matrix becomes extremely low-rank. This motivates the decomposition of P using the sparse mask M:
P= P ROM
+ O (1 (1- M)
(1)
sparse component
low rank component
Since linear attention is essentially a low-rank version of attention, we are provided with possibility to replace the low-rank component P O (1 - .) with linear attention.
4 SLA
SLA effectively integrates sparse and linear attention within a unified framework, allowing them to complement each other. In particular, we fuse both attention into single efficient GPU kernel. In this section, we introduce the sparse and linear attention components of SLA.
SLA first predicts a compressed attention weights matrix Pe N/b,
(2) where pool(-) is mean pooling operator along the token dimension. For each element of Pa. classify it into three types and record the results in compressed mask Mc N/bxxx///b Specifically, the top k2 positions are marked as critical (labeled 1), the bottom 47% positions as negligible (labeled -1), and the remaining positions marginal (labeled 0). Formally,
Pe (oftma /V)))
Mc[i.j] {1 (top ky %), -1 (bottom k)),, 0 (otherwise)).
We apply different methods according to M
4.1 SPARSE ATTENTION IN SLA
Guided by the mask Mo, sparse FlashAttention used to compute the sparse attention output. For each Q block Q, we iterate over all K, V blocks K,, V,, with j = O,..., N/.,.-- Whenever
Moli.j] = 1, we perform:
S - Q.K.///// /va. Pij OnlineSoftmax (S.)), O +
Here, OnlineSoftmax(() operator (Milakov & Gimelshein 2018) computes the softmax of matrix in block-wise manner (see lines 10-11 of Algorithm I for implementation). The initial value of each O is set to zero. Algorithm [ describes the forward computation of the sparse attention component, and denote the final output of the sparse attention component O=,
4

---

Published as a conference paper at ICLR 2026
4.2 LINEAR ATTENTION IN SLA
Inspired by the idea of low-rank approximation. we replace the low-rank component Po (1 (1-M) in Equation with linear attention introduced in Section 2.2
p{)){(K)T rowsum 6 (Q))(K)T (1-
Specifically, the entries of 0 in Me determine the blocks processed by linear attention. For each query block Qi, we compute the corresponding linear attention output:
Σ rowsum (o(K))), O (QQ H, J:Ma|Ka]=0
H. Σ ((K))
Z.
(5)
o (Q,)Zz
J:M
Here, mentioned in Section 2.2 6(-) denotes the activation function, and H, Rd Z₁ Rdx1 are intermediate results similar to H and Z. Algorithm | describes the forward pass of the linear attention component, and the final output of this component denoted O².
Finally, the overall attention output of SLA is defined as:
0 = o + Proj(O').
(6)
where Proj is learnable linear transformation Rd Rd. Applying this projection O' helps reduce the distribution mismatch between softmax and linear attention. Its computational cost is ((N)), the same as computing O' and negligible compared with the (NV2d) cost of full attention. Specifically, O(Nd²) = 0.004 x O(N2d) when N 32K, d 128 in the Wan2.1-1.33. In this case, 95% % sparsity the sparse attention component means 94. 7% attention complexity reduction.
Insight. Linear attention in SLA does not approximate the output corresponding to marginal at- tention weights, but serves as learnable compensation that enhances the effectiveness of sparse attention. This is because linear attention alone struggles to approximate the output of full atten- tion (Choromanski al., 2020 Qin al., (022). Therefore, we need fine-tune the parameters of the target model, enabling it to adapt to the use of lincar attention.
High-Level Idea
Detailed Algorithm
Attention Weight Prediction
K
K,
K
K
by x d
K
K
KJ d x
K
V1
Q1
0.1
P
V2 V
bke
Softmox( pool(Q) pool K
Sparse FlashAttention
Linear Attention
S = (QıKŞ)
H = x(K2) T V2 o(Ka) V4
Predicted Compressed Attention Weigths
P OnlineSoftmax(5.)
Z sum(o(Ks o(K4)
0.8
0 P,V,
O ó(Q, / (o(Q))))
Attention Computation
O 0 + Proj(O')
O(N3) Attention
0.8
(Critical weights) FlashAttention
0.1
(N) Attention
Skip
(Marginal weights) Linear Attention
(Negligible weights) Skip
Figure 4: Overview of SLA. The left figure illustrates the high-level idea: attention weights classified into three categories and assigned to computations of different complexity. The right figure shows the detailed forward algorithm of SLA using the predicted compressed attention weights.
FINE-TUNING USING SLA
To apply SLA to a diffusion model, we can simply replace the original attention with SLA and fine- tune the model for a few steps dataset consistent with the pretraining data. In this section, describe the forward and backward passes SLA. Morcover, we detail some additional efficiency optimization for SLA in Appendix A.6
5

---

Published as a conference paper at ICLR 2026
Algorithm 1: Forward pass of SLA.
1: Input: Matrices Q, K, V, Q", K E R Nxd block sizes by, bkv, hyper-parameters kit kj-
2: Divide Q,Q" Q to I'm /b blocks (Qr } and {
3: Divide K, V, K
N/buu blocks } (V.) and {K)):
4: h = {hy) = ((K) V ,,) :2 (z/) (rowsum)((5)
}} Precompute for linear attention
P. Sottmax(pool()pool
/vd); ; Initialize M. 0;
6: Mc[i, = if Peli.j] Topk(P: :], k.) Mc[t. [ =
P. [ E (ott.on((P(i.))kka))
7: for =1 11 to 7m do
8: for 1 to T, do
if M.
then S.j K /vd; my max
10:
Púj exp(Sij miy);
11:
4y = e ". mas 4,5-1 rowsum(Ptt) (Pt);
O = diag(e" 1mmm)) 0 + Vs
12:
else if j] = 0 then
13:
H. + H. + h1i Z. Z. 2
14:
end if end for
15:
16: O = =diag(!?) " ) 0
0 = H. H. (Q. L, = =, + log((.r)))
17:
end for
18:
return 0 (0:). O' (
5.1 FORWARD PASS
The formulation of the forward computation was introduced in Section 4 The complete algorithm of the forward pass of SLA is presented in Algorithm [1 It's worth noting that we precompute h = ((K,) V, and 2j = rowsum ( for each pair (K,, V2) (Line 4 in Algorithm I.. This design ensures that, when Me
O, the corresponding operation only involves single matrix addition (Line 13 Algorithm D]. thereby improving efficiency. To simplify the notation, we denote
Q (QQ) and K <(K) in the following.
Algorithm 2: Backward pass of SLA.
I: Input: Q,K, V, Q", K Mc. L. (H.), (Z,), O", O from the forward, dO', dO' Nxd
2: D' rowsum(dO" @ 0"), = rowsum(dO" o O¹), divide D", D' into Tm blocks (D?), (D))
3: for i 1 to T,,, do
dH, (Q. (Q.((( d0:: dz. = --QQQ((Q...)) (T) D
5: dQ, (do
)
end
7: for jorj= = 1 to T., do
8: Initialize dH = 0, dZ - 0;
9: for = 1 to Ti do
10: if Mclijj
then
11:
S, Q:K. /vd: Pú exp(S:j - L:): dV, + dv, +P].0; dP; = do,
12:
dS., = P. 3 (dP,, - D dQ, + dQ, K, dk, + +
13:
else if Mi 0 then
14:
dH dH + dHg dz dZ:
15:
end VyHdH) + (dz)T: dv, KydH:
16:
end for
17:
dk
18:
end for
return dQ (dQ,). (dK.). dV (dV.). dQo (dQ.). dk {dk)):
5.2 BACKWARD PASS
The backward pass computes gradients for both the sparse and linear components, which are also fused into single GPU kernel for efficiency.
Gradient notation. The prefix or d is used to denote gradients, e.g., dOa, dO' are the gradients of O², O with respect some loss function f, respectively.

---

Published as a conference paper at ICLR 2026
Sparse attention gradients. The output gradient dOa is backpropagated to compute dQ, dK, and dV, following the same derivation as in FlashAttention (Dao 2023 Given dO¹, the backward pass is carried out follows:
dP, ij = dO b v
D = rowsum(dO; O O
dS = Psy o (dPi, - DD),
dQ, d dK, ds Qi, dV; Pl
Here, we consider D E TR x1 as a column vector.
Linear attention gradients. The gradient dO' yields dQº, dK* "KK,dV dV through the chain rule:
Q,
Q a
dH;-
dO
D rowsum(dO OO, dz; dk = (dH.) T + (dz.)
D2
Z
O
(8)
(do' (H,) - D
dQ,
dV; = KydH; dH;
Q Z
Here, dk; and dV, obtained by aggregating dH, and dz, Similar to the forward pass, each dH, and dz, is precomputed so that the remaining computation reduces to simple matrix additions. The detailed algorithm provided in Algorithm 2
EXPERIMENT
6.1 SETUP
Model and Datasets. We use the Wan2.1-1.33 model (Wan et al. 2025) for video generation experiments in the main text and LightningDiT (Yao et al. 2025) for image generation experiments in the Appendix A.2 We also conduct experiments
private MM-DiT (Esser et al. (2024) model in the Appendix A.4. For video experiments, we use a private dataset collected from websites such as Pexels (Pexels) and Common Crawl (Common Crawl), consisting of 20,000 5-second videos at 480p resolution for fine-tuning. For image experiments, following Lightning (Yao et al. 2025).
we use the ImageNet (Deng et al.) 2009) dataset at a resolution of 512 512.
Baselines. We compare SLA with state-of-the- art sparse attention methods applicable diffu- sion models, including (1) VSA (Zhang et al. 2025e), (2) VMoBa (Wu et al.) (025), and (3) the training-free SparseAttn (Zhang et al. 2025b) (paarge-F) and (4) a trainable implementation of SpargeAttn (Sparge-T). For VSA and VMoBa, Sparse-T, we implement the method ourselves because there is no official implementation. In addition, design several baselines for ablation studies: (5) Linear Only, which applies only linear attention; (6) Sparse Only, which applies only the sparse attention component of SLA; and (7) L+ which directly sums the attention outputs of the Linear Only and Sparse Only.
their official implementations, while for
Metrics. For video quality, following Zhang et al. (2024a); Yang et al. (0255),, we use four eval- uation dimensions of VBench (Zhang al.) 2024 Imaging Quality (IQ), Overall Consistency (OC), Aesthetic Quality (AQ), Subject Consistency (SC). We also use the Vision Reward (VR) (Xu al. 2024) for human preference evaluation, Aesthetic Video Quality (VA), and Techniqual Video Quality (VT) (Liu et al. 2023). For image quality. following Yao et al.) (2025). we use FID. For attention computation complexity, we use FLCPs s (floating point of operations). For attention eff- ciency, we use S (floating-pon -prinn operations
second) for attention kernel efficiency. Specifi- cally, FLOPS here is (full attention)/ /t, where (--) denotes the operation count and t the attention latency. We use seconds for end-to-end generation latency.
Hyper-parameters.
use training batch size of 64 and fine-tune the Wan2.1 model for 2000 steps. For the activation function o, 5% and k%% is 10%. For block size, we use by = bşv = 64. The pyper-paramedt for image generation tasks are detailed in Appendix A.2
softmax according to ablation experiments. ky %
6.2 EFFECTIVENESS
Table T compares the video generation quality and efficiency of SLA with baseline methods on
Wan Wan2.T-1.3B. T-1.B, fine-tuned separately with SLA, Full Attention, and each baseline. SLA delivers about

---

Published as a conference paper at ICLR 2026
x efficiency gain while maintaining video quality comparable Full Attention. Moreover, compared with the baselines, SLA consistently achieves higher quality even under greater sparsity. For example, 95% (1-5%) sparsity in SLA is actually about 3x more efficient than 85% (1-15%) while still producing better video quality.
Table 1: Quality and efficiency comparison of SLA and other baseline methods.
Quality
Efficiency
Method
VA f
VT f
IQ 1
00 f
AQ f
SC f
VR f
FLOPs
Sparsity
Full Attention
76.78
82.88
62.5
23.3
56.1
93.0
0.059
52.75T
Sparge-F
0.002
0.026
35.7
85.1
-0.216
7.91T
85%
Sparge-T
73.83
77.87
61.9
22.7
55.4
93.1
0.014
7.38T
84%
VMoBa
32.33
35.79
58.0
18.8
46.2
89.9
-0.175
7.91T
85%
VSA
55.37
64.61
60.6
22.4
51.9
83.6
-0.069
5.92T
89%
SLA
76.96
83.92
62.2
23.6
55.9
93.1
0.048
2.74T
95%
Table Ablation results for SLA.
Quality
Efficiency
Method
VA f
vr
1Q f
OC f
AQ f
SC f
VR
PLOPs 4
Sparsity
Full Attention
76.78
82.88
62.5
23.3
56.1
93.0
0.059
52.75T
0%
Linear Only
0.042
0.099
39.5
3.6
28.8
90.7
-0,213
0.10T
100%
Sparse Only
64.00
70.50
57.2
21.8
51.7
88.7
-0.073
7.91T
85%
L+S
29.65
41.15
58.6
18.8
45.3
87.1
-0.105
5.37T
90%
SLA (softmax)
76.96
83.92
62.2
23.6
55.9
93.1
0.048
2.73T
95%
SLA (elu+1)
75.50
81.01
62.8
23.5
55.3
92.9
0.034
2.74T
SLA (hedgehog)
74.59
82.62
61.9
22.5
54.3
93.2
0.035
3.11T
95%
SLA (Top 5%)
76.96
83.92
62.2
23.6
55.9
93.1
0.048
2.73T
95%
SLA (Top 10%)
75.29
82.20
62.5
22.6
55.8
93.5
0.057
5,38T
90%
SLA (Top 20%)
75.81
83.82
62.7
22.4
54.5
92.6
0.059
10.65T
80%
Table 3: Quality of SLA in the zero-shot and limited finetuning settings.
Quality
Method
VA
VT
IQ f
OC
AQ
SC
VR
Full Attention
76.78
82.88
62.5
23.3
56.1
93.0
0.059
SLA (0 step)
41.11
51.79
58.3
21.4
46.7
81.0
-0,1105
SLA (250 steps)
64.46
78.06
59.0
22.8
55.7
88.5
-0.0244
SLA (1000 steps)
74.58
80.09
61.8
23.7
56.1
92.3
0.0429
SLA (2000 steps)
76.96
83.92
62.2
23.6
55.9
93.1
0.0483
6.3 EFFICIENCY
Figure 6 compares the kernel speed and end-to-end latency of SLA on Wan2.1-1.3B with an RTX5090. Note that even VSA in 89% sparsity and VMoBa in 85% sparsity, their generation quality is already worse than SLA, so higher sparsity settings (c.g.. 95%) are not quality-matcher compar- isons. In the forward pass, SLA achieves 13.7x speedup over FlashAttention2 and is 1.93x faster than VSA with 95% sparsity and 3.36x faster than VMoBa with 95% sparsity. In the backward pass, delivers a 6.8x speedup over FlashAttention2. still outperforming VSA and VMoBa. For end-to- end video generation, SLA reduces attention latency from 97s to 11s (8.8x reduction), resulting in a 2.2x x end-to-end speedup. For fine-tuning overhead, we train Wan2.1-1.3B for only 2,000 steps with batch size of 64, which is less than 0.1% 1% of the cost of pretraining (typically 100-108 steps with a batch size of -0"-104) (Wan et al. 2025). The finetuning of SLA requires approximately hours on 8 NVIDIA H200 GPUs.
8

---

Published as a conference paper at ICLR 2026
Attention (Sparsity 07%
(Sparsity 95%)
S+L (Sparsity 90%)
Sparge-T (Sparsity 84%
Linear Only
Sparse Only (Spursity 85%)
Sparge-F (Sprsiity-5%)
VSA (Sparsity 89%)
VMoBa (Sparsity 85%)
(Sparsity 100%)
Figure 5: Video examples using Wan2.1 fine-tuned with SLA and baselines. For Li line Only, Sparse Only, Sparge-F, VSA, and VMoBa, only single frame per prompt is shown, as their video quality is sufficient. The full visible comparison is in Figure in Appendix A.1
FlashAttn
VMoDa (95%)
VSA (95%)
Attention
(85%)
(89%)
(95%)
Original
628
97s
Kernel Speed
Speed
2996
1479
(Sparsity=8%%)
VMoBa
62s
47s
Speed (1 (1)
13.7x
Sopeet (1 /.
1032
1011
(Sparsity
VMoBa
62s
28s
1553
(Sparsity-89%)
VSA
626
268
Speedup
219
226
218
VSA
62s
16s
(Sparsity 95%)
Flash Al
VSA (15%) (9575)
Flash
USA
(1S76)
VSA
(Sparsity 95%)
SLA
(a) Attention GPU Kernel speed comparison on RTX5090.
(b) End-to-end video generation latency comparison.
Figure 6: Attention kernel speed and end-to-end generation latency of SLA and baselines Wan2.1- 1.3B with RTX5090, FlashAttn refers to FlashAttn2, the fastest available version on RTX5090.
In Appendix A.7
compare the efficiency of SLA and FlashAttention on more GPUs, while in we explore LoRA (Hu et al. 2022 as more efficient finetuning paradigm.
Appendix A.9
6.4 ABLATION STUDY
Fusing sparse and linear attention. To evaluate the effectiveness of SLA in integrating sparse and linear attention, we compare SLA with Sparse Only. Linear Only, and S+L on Wan2. I in terms of end-to-end generation quality and efficiency. As shown in Table [ SLA achieves the best generation quality and is more efficient than Sparse Only and S+L, conhrming the effectiveness of our fusion strategy.
Activation function in linear attention. study the effect the activation function & linear attention component of SLA, we evaluate softmax, elu+1, and hedgehog. Results in Table 2 show that softmax generally provides better generation quality and efficiency.
Impact of parameter кą. We vary kn. from 5% 20% and report the results in Table 2 We find that hy = 5% already yields generation quality close to that of full attention. Since ky 5% saves about half and quarter of the computation compared with k₁ 10% and k₁ 20%, offers the best trade-off between efficiency and quality.
Impact of parameters k; by and bku. These parameters appear to have smaller influence com- pared with kn We conduct ablation experiments on the LightningDiT model, and the results are reported in Appendix A.3

---

Published as a conference paper at ICLR 2026
Zero-shot and limited finetuning. Table 3 presents the performance of SLA under zero-shot (0- step finetuning) and limited finetuning budgets. The quality of videos generated by SLA steadily improves finetuning progresses. After only IK finetuning steps, the quality is already close to that full attention, and 2K steps yield the best results.
6.5 VISIBLE EXAMPLES
Figure and Figure show video examples from Wan2.1-1.3B fine-tuned using SLA and baselines. SLA produces videos comparable to full attention even at 95% sparsity, while other methods exhibit noticeable distortions even at sparsity levels below 90%.
7 RELATED WORK
As sequence lengths in generative models (c.g., language and video) grow, the quadratic cost of attention becomes a key bottleneck. Many studies aim to improve efficiency in two main directions:
sparse and linear attention. Most sparse attention methods (Xiao al. 2024b a Jiang et al. 2024
Fu et al. 2024 Xi et al. 2025 Zhang et al. 2025b Ribar et al. 2023) Yang et al, 2025a Xi et al. 2026) speed up inference without training by masking computation at test time.
Gao et al.
2024
Some Zhang et al. 2222e: Wu et al., 2025 Zhang et al. 2026b c) add sparsity during training,
enabling higher sparsity. Linear attention methods (Wang al. 2020: Choromanski et al. 2020)
Katharopoulos et al.) 22200 Qin et al. [2024] Yang et al. 2024 Sun et al. 2023) are mainly studied in language models. For DiT, SANA (Xie al. (2024) and Dig Zhu al.
2025) show linear attention
works for image generation pre-training, but in video generation, existing methods cannot rely on it alone for lossless quality. Another direction is harfware-efficient attention (Dao et al.) 2022] [Do]]
2023 Shah et al.) 2024 Zhang et al. [024cc b 2025c a), which optimizes GPU execution through tiling, kernel fusion, and quantization.
8 CONCLUSION
We propose SLA, trainable attention that unifies sparse and linear attention accelerate Diffusion Transformers. SLA assigns computation according to importance: it computes O(N2) attention for critical weights, O(N) attention for marginal weights, and skips negligible computations. This design enables substantial reductions attention cost while preserving effectiveness. Experiments show that just few fine-tuning steps enable SLA accelerate models effectively. Specifically, SLA achieves about 20 reduction in attention computation, along with a 13.7 13.7x GPU kernel speedup and 2. end-to-end speedup on Wan2.1-15 1-1.BB, all without degrading the quality of video genera- tion.
ETHICS STATEMENT
This work proposes a method for improving the efficiency of Diffusion Transformers. The study does not involve human subjects, personally identifiable information, sensitive data. We believe the proposed method does not raise ethical concerns beyond standard considerations in efficient model design.
ACKNOWLEDGEMENT
This work supported by the Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education China (No. [YB2025XDXM101 the NSFC Projects (Nos. 62550004, 92270001, 62376131 J.Z also supported by the XPlorer Prize.
REFERENCES
V. L. Arlazarov, E. A. Dinic, M. A. Kronod, and A. Faradzev. On economical construction of the transitive closure of an oriented graph. Sovier Mathematics Doklady, 11:1209-1210, 1970.
Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreca Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser,
10

---

Published as a conference paper at ICLR 2026
David Benjamin Belanger, Lucy J Colwell, and Adrian Weller. Rethinking attention with per- formers. In International Conference on Learning Representations, 2020.
Djork-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). In Proceedings of the International Conference on Learning Represeniations (ICLR), 2016.
Common Crawl. Common crawl.
Tri Dao. Flashattention-22: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.0869) 2023.
Tri Dao, Daniel Y Fu, Stefano Ermon, Atri Rudra, and Christopher Re. Flashattention: Fast and memory-efficient exact attention with 10-awareness. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Patrick Esser, Sumith Kulal, A. Blattmann, Rahim Entezari, Jonas Muller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion En- glish, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. ArXiv, 0aw/2403.03206 2024. URL wtt.a/ooaa...roooooooooooooo 2688447798
Qihang Fan, Huaibo Huang, and Ran He. Breaking the low-rank dilemma of linear attention. In CVPR, 2025.
Tianyu Fu, Haofeng Huang, Xuefei Ning, Genghan Zhang, Boju Chen, Tianqi Wu, Hongyi Wang, Zixiao Huang, Shiyao Li, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. Moa: Mixture of sparse attention for automatic large language model compression. arXiv preprint arXiv:2406. 14909, 2024.
Yizhao Gao, Zhichen Zeng, Dayou Du, Shijie Cao, Hayden Kwok-Hay So, Ting Cao, Fan Yang. and Mao Yang. Seerattention: Learning intrinsic sparse attention in your llss. arXiv preprint arXiv:2410. 13276, 2024.
Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention trans- former. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 6185-6194, 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation large language models. In International Con- ference Learning Representations, 2022. URL
net forum?
id-nzevKeeFYE9
Huiqiang Jiang, YUCHENG LI, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhen- hua Han, Amir H. Abdi, Dongsheng Li, Chin- Yew Lin, Yuqing Yang, and Lili Qiu. MInference 1.0: Accelcrating pre-filling for long-context LLMs via dynamic sparse attention. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024.
Youhe Jiang, Fangeheng Fu, Wanru Zhao, Stephan Rabanser, Nicholas Lane, and Binhang Yuan. Cascadia: A cascade serving system for large language models. arXiv preprint rrXiv:2506.04203, 2025a.
Youhe Jiang, Wenshuang Li, You Peng, Jintao Zhang, Ran Yan, Jianfei Chen, Xu Han, Fangcheng Fu, and Binhang Yuan. Hexgen-3: fully disaggregated Ilm serving framework with fine- tine-grained heterogeneous resource autoscaling. 2025b.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers mns: Fast autoregressive transformers with linear attention. In International conference on ma- chine learning, pp. 5156-5165. PMLR, 2020.

---

Published as a conference paper at ICLR 2026
Xingyang Li, Muyang Li, Tianle Cai, Haocheng Xi, Shuo Yang, Yujum Lin, Lvmin Zhang, Songlim Yang. Jinbo Hu, Kelly Peng, et al. Radial attention: o(n' log n) wwwpgggcccooooooooiieeeeeeeeeeee
spar
onllongvvidoogeeerationn arXiv preprint arXiv:2506. 10Xi::250611982222 2025.
Akide Liu, Zeyu Zhang, Zhexin Li, Xuehai Bai, Yizeng Han, Jiasheng Tang, Yuanjie Xing, Jichao Wu, Training-aware
Mingyang Yang, Weihua Chen, et al. Fpsattention: Training aware fp8 and sparsity co-design for fast video diffusion. arXiv preprint urXiv:2506.06445 2025.
Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tiey- ong Zeng, Raymond Chan, and Ying Shan. Evalcrafter: Benchmarking and evaluating large video generation models. arXiv preprint arXiv:2310. 14440, 2023.
Maxim Milakov and Natalia Gimelshein. Online normalizer calculation for softmax. arXiv preprint arXiv: 008602267, 2018.
William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv:2212.09748, 2022.
arXiv preprint
Pexels. Pexels: Free stock photos and videos. httpp:://w..pexellss
Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong. cosformer: Rethinking softmax in attention. In International Confer- ence Learning Representations, 2022. URL https:://ppereeiiee net..eee///eee pent/forum?? forum?id- B18CQrx2Up4
Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, and Yiran Zhong. Lightning attention- 2: A free lunch for handling unlimited sequence lengths in large language models. arXiv preprint arXiv:2401.04653 046558 2024.
Luka Ribar, Ivan Chelombiev, Luke Hudlass-Galley, Charlie Blake, Carlo Luschi, and Douglas Orr. Sparq attention: Bandwidth-effiiient IIm inference. arXiv preprini arXiv:2312.049900 2023.
Mark Rudelson and Roman Vershynin. Sampling from large matrices: an approach through geometric functional analysis, 2006. htttttsss///xiii>
abs http://aammmmmt/5550////
Jay Shah, Ganesh Bikshandi, Ying Zhang. Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024.
Xuan Shen, Chenxia Han, Yufa Zhou, Yanyue Xie, Yifan Gong, Quanyi Wang, Yiwei Wang, Yanzhi Wang, Pu Zhao, and Jiuxiang Gu. Draftattention: Fast video diffusion via low-resolution attention guidance. arXiv preprint arXiv:2505.14700 2025.
Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.086221 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Team Wan. Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng. Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang. Mengyang Feng. Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fang, Tianxing Wang, Tianyi Gui, Tingyu Weng, Tong Shen, Wei Lin, Wei Wang, Wei Wang, Wenmeng Zhou, Wente Wang, Wenting Shen, Wenyuan Yu. Xianzhong Shi, Xiaoming Huang. Xin Xu, Yan Kou, Yangyu Lv, Yifei Li, Yijing Liu, Yiming Wang, Yingya Zhang, Yitong Huang, Yong Li, You Wu, Yu Liu, Yulin Pan, Yun Zheng, Yuntao Hong. Yupeng Shi, Yutong Feng, Zeyinzi Jiang, Zhen Han, Zhi-Fan Wu, and Ziyu Liu. Wan: Open and advanced large-scale video generative models. arXiv preprint 25Xiv:2503.20314 2025.
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.047000 2020.
12

---

Published as a conference paper at ICLR 2026
Jianzong Wu, Liang Hou, Haotian Yang, Xin Tao, Ye Tian, Pengfei Wan, Di Zhang, and Yunhai Tong. Vmoba: Mixture-of-bloce attention for video diffusion models. arXiv preprint arXiv:2506.3385. 2025.
Glorot Xavier, Bordes Antoine, and Bengio Yoshua. Deep sparse rectifier neural networks. In Pro- ceedings of the Fourteenth International Conference Artificial Intelligence and Statistics, pp- 315-323, 2011.
Haocheng Xi, Shuo Yang, Yilong Zhao, Chenfeng Xu, Muyang Li, Xiuyu Li, Yujun Lin, Han Cai, Jintao Zhang, Dacheng Li, et al. Sparse videogen: Accelerating video diffusion transformers with spapoal-temporal sparsity. arXiv preprint arXiv:2502.01771 2025.
Haocheng Xi, Shuo Yang, Yilong Zhao, Muyang Li, Han Cai, Xingyang Li, Yujun Lin, Zhuoyang Zhang, Jintao Zhang, Xiuyu Li, et al. Quant videogen: regressgees long video generation via 2-bit kv-cache quantization. arXiv preprint arXiv:2602.02955 2026.
Chendong Xiang, Jiajun Liu, Jintao Zhang, Xiao Yang, Zhengwei Fang, Shizun Wang, Zijum Wang, Yingtian Zou, Hang Su, and Jun Zhu. -eaaetry-aware rotary position embedding for consistent video world model. arXiv preprint 7XXv:2602.078544. 2026.
Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. Infilm: Training-free long-context extrapolation for Ilms with efficient context memory. In First Workshop on Long-Cont Context Foundation Models@ ICML 2024, 2024a.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming lan- guage models with attention sinks. In The Twelfth International Conference on Learning Represen- tations, 2024b.
Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, et al. Sana: Efficient high-resolution image synthesis with linear diffusion transformers. arXiv preprint arXiv:2410. 10629, 2024.
Jiazheng Xu, Yu Huang, Jiale Cheng, Yuanming Yang, Jiajun Xu, Yuan Wang, Wenbo Duan, Shen Yang, Qunlin Jin, Shurun Li, et al. Visionreward: Fine Fine-grained nulti-dimmessinall human preference learning for image and video generation. arXiv preprint arXiv:2412.21005 2024.
Shuo Yang, Haocheng Xi, Yilong Zhao, Muyang Li, Jintao Zhang, Han Cai, Yujun Lin, Xiuyu Li, Chenfeng Xu, Kelly Peng, et al. Sparse videogen2: Accelerate video generation with sparse attention via semantic-aware permutation. arXiv preprint arXiv:2505. 18875, 2025a.
Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. arXiv preprint arXiv:2412.06460 2024.
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. In The Thirteenth International Conference Learning Representations, 2025b.
Jingfeng Yao, Bin Yang, and Xinggang Wang. Reconstruction generation: Taming optimization dilemma in latent diffusion models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Putlern Recognition, 2025.
Fan Zhang, Shulin Tian, Ziqi Huang, Yu Qiao, and Ziwei Liu. Evaluation agent: Efficient and prompt- able evaluation framework for visual generative models. arXiv preprint arXiv:2412.009655 2024a.
Jintao Zhang, Rundong Su, Chunyu Liu, Jia Wei, Ziteng Wang, Pengle Zhang, Haoxu Wang, Huiqiang Jiang, Haofeng Huang. Chendong Xiang, et al. A survey of efficient attention methods: Hardware- efficient, sparse, compact, and linear attention.
Jintao Zhang. Haofeng Huang, Pengle Zhang, Jia Wei, Jun Zhu, and Jianfei Chen. Sageattention2: Efficient attention with thorough outlier smoothing and per-threat thread int4 quantization. arXiv preprint 1. 10958, 2024b.
13

---

Published as a conference paper at ICLR 2026
Jintao Zhang, Jia Wei, Haofeng Huang, Pengle Zhang, Jun Zhu, and Jianfei Chen. Sageattention: Accurate 8-bit attention for plug-and aad--anddplyyy inference acceleration. arXiv preprint arXiv:2410.02367,, 2024c.
Jintao Zhang, Jia Wei, Pengle Zhang, Xiaoming Xu, Haofeng Huang, Haoxu Wang, Kai Jiang, Jun Zhu, and Jianfei Chen. Sageattention3: Microscaling fp4 attention for inference and an exploration of 8-bit training. arXiv preprint vrXiv:2505.1 2025a.
Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia Wei, Haocheng Xi, Jun Zhu, and Jianfei Chen. Spargeattn: Accurate sparse attention accelerating any model inference. In International Conference on Machine Learning (ICML), 2025b.
Jintao Zhang, Xiaoming Xu, Jia Wei, Haofeng Huang, Pengle Zhang, Chendong Xiang, Jun Zhu, and Jianfei Chen. Sageattention2+:: A more efficient implementation of sageattention2. arXiv preprint arXiv:2505.21136, 2025c.
Jintao Zhang, Kaiwen Zheng, Kai Jiang, Haoxu Wang, Ion Stoica, Joseph E Gonzalez, Jianfei Chen, and Jun Zhu. Turbodiffusion: Accelerating video diffusion models by 100-200 times. arXiv preprint ar arXiv:2512.16093, 2025d.
Jintao Zhang, Marco Chen, Haoxu Wang, Kai Jiang, lon Stoica, Joseph E. Gonzalez, Jianfei Chen, and Jun Zhu. Sagebwd: A trainable low-bit attention. 2026a.
Jintao Zhang, Kai Jiang, Chendong Xiang, Weiqi Feng, Yuezhou Hu, Haocheng Xi, Jianfei Chen, and Jun Zhu. SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine- Tunnin arXiv preprint arXiv:2602.13515, 2026b.
Jintao Zhang, Haoxu Wang, Kai Jiang, Kaiwen Zheng, Youhe Jiang. Ion Stoica, Jianfei Chen, Jun Zhu, and Joseph E. Gonzalez. SLA2: Sparse-Linnat Attention with Learnable Routing and QAT. arXiv preprint arXiv:2602. 12675, 2026c.
Peiyuan Zhang, Yongqi Chen, Haofeng Huang, Will Lin, Zhengzhong Liu, lon Stoica, Eric Xing, and Hao Zhang. Vsa: Faster video diffusion with trainable sparse attention. arXiv preprint arXiv:2505. 13389, 2025e.
Pengle Zhang, Jia Wei, Jintao Zhang, Jun Zhu, and Jianfei Chen. Accurate int8 training through dynamic bloek-level fallback. arXiv preprint arXiv:2503.0800 08040, 2025f.
Lianghui Zhu, Zilong Huang, Bencheng Liao, Jun Hao Liew, Hanshu Yan, Jiashi Feng, and Xinggang Wang. Dig: Scalable and efficient diffusion models with gated linear attention. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 7664-7674, 2025.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 MORE VISIBLE EXAMPLES
Attention (Sparsity = 0%)
(Sparsity 95%)
(Sparsity 90%)
Sparge-T (Sparsity 84%)
Linear Only (Sporsity 100%)
Sparse Only (Sparsity 85%)
Sparge-F (Sparsity-85%) 85%)
VSA (Sparsity 89%)
VMoBa (Sparsity 85%)
Figure 7: Full video examples generated by the Wan2.1 fine-tuned with SLA and baseline methods. The first prompt is "A polar bear is playing guitar". The second prompt "Pacific coast, carmel by the sea ocean and "aves". The third prompt is "a bird building a nest from twigs and leaves".
Figure demonstrates some additional video examples generated by the Wan2.1 model, fine-tuned with SLA and other attention methods. We can find that SLA consistently achieves higher quality under bigger sparsity than baselines.
15

---

Published as a conference paper at ICLR 2026
A.2 EXPERIMENTS FOR IMAGE GENERATION
Experimental setup. As described in Section 6.1 we evaluate SLA and the baselines pretrain- ing task of LightningDiTT (aao al. 2025). Specifically, we use the L,i Lightni inDi T-1 T-1p08/ model. consisting of 1.03B parameters, trained the ImageNet (Deng et al.) 2009) dataset at resolution of 512 x 512. We introduce a new baseline block diagonal as a representative of static pattern methods. It splits the sequence into blocks and computes attention only within each block.
Hyperparameters. All hyperparameters follow (ao et al. 2025 except that we train for 100000 steps with batch size of 128. For SLA, set softmax and use block size of by bbu 64.
Metrics. Following Yao et al. 2025 we adopt FID to assess image quality and FLOPs to measure computational complexity.
The results are summarized in Table At the highest sparsity level, SLA outperforms all other baselines and surpasses full attention the FID metric, confirming the advantage of SLA in preserving image quality. This finding is consistent with the video experiments on Wan2.1 reported in Section 6.2
Results.
Table 4: Quality and efficiency comparison of SLA and other baselines image generation.
Quality
Efficiency
Method
FID 4
FLOPs +
Sparsity 1
Full Attention
31.87
12.88G
0%
SpargeAttn-F
206.11
3.66G
71.57%
SpargeAttn-T
46.05
3.16G
75.45%
VSA (2D)
35.75
3.62G
75.00%
VMоBA (2D)
39.45
3.22G
75.00%
SLA
31.49
1.73G
87.50
Block Diagonal
55.06
3.22G
75.00%
A.3 ABLATION STUDY ON IMAGE GENERATION
In addition the ablation study conducted the Wan2.1-1.3B model in Section 6.4 we further in- vestigate the impact of the parameters k,, by and bav on the Lightning model. 5 varying by and by. leads to only marginal changes in performance, while increasing ky consistently degrades generation quality.
shows that
Table Ablation Study on ki, by and beg-
bg
bkv
k
FTD
64
64
0%
31.49
128
64
0%
31.49
128
32
0%
31.74
64
64
6.25%
32.94
64
64
12.5%
34.04
64
64
25%
37.31
A.4 EXPERIMENTS ON MM-DIT MODEL
Experimental setup. To demonstrate that SLA can be applied to models with different attention layouts, we evaluate SLA and the baseline on finetuning task using a private MM-DiT model.
Hyperparameters. All hyperparameters are the same as those in Section 6.
Metrics. Following Section 6.1
VA, TQ, OC, AQ, and SC measure video quality.
16

---

Published as a conference paper at ICLR 2026
The results are summarized in Table 6 SLA has almost loss in end-to-end metrics compared to full attention, indicating that SLA generalizes well to MM-DiT architectures.
Results.
Table 6: Quality comparison of SLA and baseline MM-DiT model.
Quality
Method
VA f
IQ f
OC 1
AQ f
SC f
Full Attention
57.81
0.58
7.88
55.77
86.89
SLA (ours)
55.18
55.97
8.28
56.11
88.50
A.5 EXPERIMENTS USING TOP-CDF
Because top-CDF Zhang et al. 2025b) is a common alternative for top-k when forming the sparse mask Mc. particularly to improve robustness against the inaccuracies in Pa. compare the two approaches in Table / Our results demonstrate that top-k consistently delivers better video quality, suggesting that SLA not overly sensitive small errors P..
Table 7: Experiments of combining SLA and additional tricks.
Quality
Method
IQ f
OC f
AQ f
SC
VR f
Full Attention
62.5
23.3
56.1
93.0
0.059
SLA (Top-k)
62.2
23.6
55.9
93.1
0.0483
SLA (Top-CDF)
60.8
22.2
53.8
91.2
-0.0263
A.6 ADDITIONAL EFFICIENCY OPTIMIZATION
Since the efficiency of SLA depends heavily on the sparsity pattern, we introduce several comple- mentary optimizations tailored different sparsity levels. These optimizations lead substantial gains in computational efficiency:
Lookup table. When Me is highly sparse (e.g.. sparsity 90%), scanning entire rows or columns to read mask values causes significant memory overhead. To mitigate this, we preprocess the nonzero positions of each row and column and store them in a lookup table. During computation, only the lookup table is accessed, substantially reducing memory traffic.
Pre-aggregation for linear attention. Although Line 13 in Algorithms I and Line 14 Algo- rithm 2 require only single matrix addition, repeatedly performing such additions incurs high over- head when many entries of M, are 0 (e.g.. > 90%). To address this, we precompute the row/column sums E, h, and Σ, 2j. and then subtract the contributions corresponding to Meli.] * 0. In this way, 90% of the additions can be replaced by only 10% subtractions.
Method of Four Russians. When the number of blocks with [ = is neither very small nor
very large (e.g., around 50%), provide efficient implementation for Line 13 in Algorithms and Line 14 in Algorithm 2 Specifically, we adopt the Method of Four Russians (Arlazarov et al. 1970). The key idea is to group h, and 2j into segments of g consecutive blocks and precompute all 29 possible subset sums within each segment. During the forward pass, any subset of g blocks can then be obtained by a single look-up, rather than summing them on the fly. This scheme allows theoretical computation reduction by 1/g-
A.7 EFFICIENCY TEST ON MORE DEVICES
While SLA achieves substantial speedups FlashAttention the RTX5090, further evaluate its performance across broader range GPUs to demonstrate its generality. Table rellorts the speedup of STA over FlashAttention RTX4090, A800, I.20, and H800 95% sparsity. Across all
17

---

Published as a conference paper at ICLR 2026
devices, SLA delivers at least 10x acceleration in attention computation, highlighting its consistent efficiency gains across diverse hardware.
Table 8: Speedup of SLA on more devices.
GPU
Speedup
RTX4090
11.20x 10.76x
A800
L20
13.18x
H800
12.94x
A.8 MEMORY FOOTPRINTS OF SLA
To provide more hardware details for the SLA kemelss, report the memory footprints of SLA ky = 10% in Table 9 including MFU, 1.2 cache I/O utilization, DRAM I/O utilization, and peak memory usage relative to FlashAttention for both the forward and backward passes. The profiling results indicate that SLA is compute-bound workload, with no evident memory bottlenecks.
Table 9: Memory footprints of SLA.
Kernel
MFU
I.2 Cache Util.
DRAM Util.
Memory Usage
SLA Foward
75.5%
63.8% 57.4%
2.9% 1.8%
1.12x
SLA Backward
54.2%
1.15x
A.9 COMBINING SLA WITH LORA
We evaluate the combination of SLA with LoRA (r 8,a 4) explore more efficient fine-tuning strategies. The results, summarized in Table I0 show there is clear performance gap between the LoRA setup and full -all-maraeeter finetuning.
Although more aggressive hyperparameter tuning (e.g., larger ranks) might reduce this gap. the current results indicate that full finetuning a more reliable way for SLA.
Table 10: Experiments of combining SLA with LoRA.
Quality
Method
IQ
OC f
AQ
SC 1
Full Attention
62.5
23.3
56.1
93.0
SLA
62.2
55.9
93.1
SLA + LoRA
85.0
56.3
51.8
22.9
USE OF LARGE LANGUAGE MODELS
We used a language model only for polishing English writing, while all ideas, experiments, results, and interpretations
18


Reviews:
[{"review_id": "46RXW6JkrM", "reviewer": "Reviewer_kaNn", "summary": "This work adopts the linear attention for the intermediate stage between the sparse attention and full (dense) attention.\n\nThis work first adopts the pooling to $Q$ and $K$ to get the sparse mask, then apply the full attention to the important area, skip the unimportant area, and adopt linear attention to those intermediate important area.\n\nThis work adopt the fine-tuning for the new introduced linear attention weights, which I think is the main contribution of this work. This work makes a lot effort on the forward and backward implementation of the linear attention for the insert to the original full attention and sparse attention technique.\n\naccording to the results, this work achieves good results.", "questions": "1. This work may focus on the contribution of the implementation of the forward and backward for the insert of linear attention to the full attention and sparse attention, then I think the novelty is limited.\n\n2. Can the authors provide the similarity metric like PSNR, SSIM, and LPIPS to compare to other works like [1] [2]. Especially, compare to the work [1]. Theoretically, the linear attention is slower than the skip (i.e., the sparse attention) and this work takes a lot of training effort for fine-tuning, authors need to show that the linear attention exactly improve much performance than the pure sparse attention.\n\n3. According to the result in work [1] (Table 1), the 90% sparsity ratio does not lead to much accuracy drop on image quality, while the result in Table 2 of this work show that the 'Sparse Only' result shows a large accuracy drop on metric like 'Image Quality'. Can the author explain this?\n\n4. For this work, which adopts the linear attention for the intermediate stage between full and sparse attention, I would like to ask for the results in low-resolution. I think it may be enough to adopt the sparse attention for acceleration in low-resolution video generation, as there may not be so many 'Marginal' blocks in the low-resolution video generation process.\n\n5. I want to further ask the authors about: the fine-tuning results for the pure sparse attention technique with downsampled attention (i.e., equation (2) in this paper). I think it is quite important ablation to show the effectiveness of the linear attention under the same training effort. I believe if apply same fine-tuning effort to pure sparse attention, the results will be good.\n\n---\n[1] DraftAttention: Fast Video Diffusion via Low-Resolution Attention Guidance\n\n[2] Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This work makes a lot of effort for the implementation of the linear attention - the forward and backward - during the fine-tuning.", "weaknesses": "1. The equation (2) which adopts the mean pooling to identify the attention importance is the same as the work [1] which also use the mean pooling to identify the attention importance for the acceleration of video generation. Thus, I think the novelty is limited to this work.\n\n2. This work does not provide any similarity metric like PSNR, SSIM, and LPIPS in their results, then the comparison to the work [1] is not direct. As the work [1] also use the sparse attention for the acceleration, and achieves 90% sparsity, so the exact performance improvement from the linear attention remains unknown. \n\n3. This work focus more on the new CUDA kernel designs rather than some algorithm-level novelty.\n\n---\n\n[1] DraftAttention: Fast Video Diffusion via Low-Resolution Attention Guidance", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5db6OrVkHk", "reviewer": "Reviewer_fACC", "summary": "This paper addresses the quadratic attention bottleneck in Diffusion Transformers (DiTs), particularly for video generation where sequence lengths are large. Based on the key observation that DiT attention weights can be decomposed into a small high-rank component (a few large weights) and a large, extremely low-rank component (many small weights), the authors propose Sparse-Linear Attention (SLA). SLA is a trainable, hybrid mechanism that classifies attention blocks into three categories: 1)Critical: Computed with $O(N^2)$ sparse attention. 2) Marginal: Computed with $O(N)$ linear attention. 3)Negligible: Skipped entirely.By fusing these operations into an efficient GPU kernel and applying minimal fine-tuning, SLA achieves a 95% reduction in $O(N^2)$ attention computation. This results in a 13.7x kernel speedup and a 2.2x end-to-end video generation speedup on the Wan2.1-1.3B model, all while maintaining generation quality.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The core observation decoupling attention weights into a sparse/high-rank component and a dense/low-rank component is novel and insightful. It clearly explains the limitations of using only sparse or only linear attention and provides a strong foundation for the proposed hybrid approach\n2. The proposed SLA method, which dynamically assigns different computational complexities ($O(N^2)$, $O(N)$, or $O(0)$) based on a block's predicted importance, is an elegant and highly effective solution.\n3. The paper demonstrates truly significant gains. A 95% reduction in expensive $O(N^2)$ computation translating to a 13.7x kernel speedup and a 2.2x end-to-end speedup with no loss in quality is a major contribution for efficient DiTs", "weaknesses": "1. The \"95% sparsity\" figure is confusing. Based on the hyperparameters ($k_h$=5%, $k_l$=10%), this 95% refers to blocks that avoid $O(N^2)$ computation. It does not mean 95% of blocks are skipped. In fact, 85% (100% - 5% - 10%) are still processed by $O(N)$ linear attention. This terminology should be clarified\n2. All latency benchmarks (e.g., 13.7x speedup) are reported on an RTX 5090 GPU. While impressive, providing benchmarks on more widely accessible hardware  would greatly strengthen the paper's practical and immediate value to the research community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hjHQ9G9wzI", "reviewer": "Reviewer_PbE1", "summary": "This paper proposes Sparse–Linear Attention (SLA), a trainable hybrid attention mechanism for Diffusion Transformers (DiTs), particularly targeting video generation models. The core observation is that attention weight matrices can be decomposed into a small set of high-rank, high-magnitude entries and a large set of low-rank, low-magnitude entries. SLA classifies block-wise attention weights into critical, marginal, and negligible categories, applying standard sparse FlashAttention to the critical blocks, linear attention to the marginal blocks, and skipping negligible blocks. The authors implement a fused GPU kernel covering both forward and backward passes and demonstrate—on Wan2.1-1.3B video DiT and LightningDiT image models—that SLA achieves up to 95 % sparsity, 13.7× attention kernel speedup, and 2.2× end-to-end generation speedup with minimal quality degradation.", "questions": "* Fine-tuning sensitivity: How does SLA perform if fine-tuning resources are restricted (fewer steps, smaller batch, or lower-resolution data)? Could you provide a curve illustrating quality vs. fine-tuning budget to justify the “few steps” claim quantitatively?\n* Mask prediction robustness: The compressed mask Mc relies on pooled Q/K projections. How sensitive is SLA to inaccuracies in Pc during inference, especially when encountering distribution shifts (e.g., different video domains)? Have you evaluated adaptive thresholds or confidence measures?\n* Backward pass efficiency: The backward kernel reportedly yields a 6.8× speedup, but details are sparse. Can you share profiling numbers on gradient computation vs. memory traffic? Are there cases where the backward pass becomes the new bottleneck?\nModel generality: Have you attempted SLA on cross-attention or text-conditioned video generation models? If not, what modifications would be necessary to extend SLA to multi-head, multi-source attention patterns?\n* Failure cases: Could you provide examples or statistics of prompts where SLA underperforms full attention (e.g., higher error metrics, visual artifacts)? Understanding the boundaries of applicability would help practitioners adopt the method safely.\n* Hyper-parameter tuning: For practitioners, how should kh and kl be selected for a new model? Do you recommend starting from a certain sparsity target (e.g., 95%) and adjusting based on validation loss, or is there a principled heuristic derived from attention statistics?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Originality: The sparse/linear decomposition of attention weights for DiTs is a compelling insight. Rather than naively combining sparse and linear attention, SLA’s dynamic block classification and fused kernel design show genuine innovation tailored to diffusion workloads, pushing beyond previous sparse-only or linear-only acceleration schemes.\n* Quality: The methodology is carefully engineered: Algorithm 1 and 2 detail the fused forward/backward passes; precomputation, lookup tables, and Method-of-Four-Russians optimizations indicate a strong systems perspective. Experimental evidence spans both video (Wan2.1) and image (LightningDiT) domains, with comprehensive metrics including VBench dimensions, VR, VA/VT, FID, FLOPs, sparsity, and latency.\n* Clarity: The paper is generally well written. Figures 1–6 effectively motivate the decomposition, illustrate quality impacts, and benchmark performance. The block-level notation and discussion of online softmax, pooling, and projection clarify the mechanism.\nSignificance: Attention cost is the dominant bottleneck in DiTs, especially for long video sequences (10K–100K tokens). Demonstrating a 20× reduction in attention FLOPs with negligible quality loss is highly impactful for scalable diffusion-based video generation. The approach could influence both research and production systems.", "weaknesses": "* Dependence on fine-tuning: The method relies on retraining/fine-tuning to adapt models to SLA. While 2K steps are claimed to be minor, the paper provides limited analysis of how SLA behaves in zero-shot replacement or under constrained finetuning budgets (e.g., small datasets, limited steps). A sensitivity study on fine-tuning duration and data diversity would strengthen the practical narrative.\n* Generalization beyond the tested architectures: Experiments focus on Wan2.1-1.3B and LightningDiT. Both employ similar attention layouts. It remains unclear whether SLA’s block classification and kernel fusion generalize to other DiT variants (e.g., different block sizes, mixed precision, cross-attention layers, text-video architectures). More diversity in model families (or a detailed discussion of expected behavior) would be valuable.\n* Ablation breadth: While Table 2 covers fusion, activation, and kh variations, the key hyper-parameter pair (kh, kl) and block size (bq, bkv) are not jointly explored. Similar attention heads with differing dimensions may require different thresholds; without a deeper ablation, reproducibility across setups could be challenging.\n* Evaluation scope: The paper reports strong aggregate metrics but lacks qualitative failure analysis. For instance, where does SLA still degrade outputs (e.g., motion consistency over long horizons, rare events)? Additionally, comparisons to other hybrid approaches (if any exist) or to recent structured sparsity methods in diffusion (e.g., block-diagonal or strided patterns) are absent.\n* Kernel integration specifics: Although Appendix A.3 sketches optimization tricks, concrete profiling (e.g., breakdown of compute vs. memory bottlenecks, impact on backward pass peak memory) is missing. Readers aiming to reimplement SLA might struggle without more hardware-level detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h0LKBH7eIn", "reviewer": "Reviewer_24AN", "summary": "This paper proposed SLA, a sparse-linear attention for efficient diffusion transformer generation. SLA finds that the attention weights can be decoupled into two types: a small fraction of large weights with high rank and the remaining weights with very low rank. SLA further designed a trainable attention method that fuses sparse and linear attention to accelerate diffusion models. Experiments shown that SLA reduces attention computation by 95% without degrading end-to-end generation quality, with 2.2 times speedup for video generation.", "questions": "Please see above weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.SLA designed efficient forward and backward propagation modules, which are very complete and valuable in engineering.\n\n2.It is very reasonable and meaningful for SLA to decompose attention into two computing modes, and achieve advanced performance.\n\n3.SLA has great advantages in efficiency and has achieved state-of-the-art acceleration performance.", "weaknesses": "1.Is the attention mode mentioned in SLA commonly present in existing advanced video generation models? Reporting more models such as CogVideoX's results can further demonstrate the universality of this phenomenon.\n\n2.How is Eq3 calculated specifically? For example, directly calculating the average value or calculating the rank? If it is the average, is there a direct correlation between this and the rank mentioned before?\n\n3.Does the implementation of SLA distinguish between model architectures, such as Wan2.1 using DiT architecture, while other models such as HunyuanVideo and CogVideoX use MM-DiT architecture? Will this lead to differences in attention calculation methods and implementation?\n\n4.It is meaningful to conduct additional supplementary experiments on other architectures such as CogVideoX-2B based on MM-DiT architecture.\n\n5.The paper should report the resources consumed during training, such as the number of GPUs and time. And exploring more efficient fine-tuning paradigms such as LoRA can further enhance the practicality of SLA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)