Radial Attention: $\mathcal{O}(n\log n)$ Sparse Attention with Energy Decay for Long Video Generation
Authors: Xingyang Li
Muyang Li
Tianle Cai
Haocheng Xi
Shuo Yang
Yujun Lin
Lvmin Zhang
Songlin Yang
Jinbo Hu
Kelly Peng
Maneesh Agrawala
Ion Stoica
Kurt Keutzer
Song Han
Venue: NeurIPS 2025 poster ()
Keywords: video generation
sparse attention
diffusion models
Primary area: deep_learning
TL;DR: a sparse attention with $\mathcal O(n \log n)$ complexity for long video generation

Abstract: Recent advances in diffusion models have enabled high-quality video generation, but the additional temporal dimension significantly increases computational costs, making training and inference on long videos prohibitively expensive. In this paper, we identify a phenomenon we term Spatiotemporal Energy Decay in video diffusion models: post-softmax attention scores diminish as spatial and temporal distance between tokens increase, akin to the physical decay of signal or waves over space and time in nature. Motivated by this, we propose Radial Attention, a scalable sparse attention mechanism with $\mathcal{O}(n \log n)$ complexity that translates energy decay into exponentially decaying compute density, which is significantly more efficient than standard $\mathcal{O}(n^2)$ dense attention and more expressive than linear attention. Specifically, Radial Attention employs a simple, static attention mask where each token attends to spatially nearby tokens, with the attention window size shrinking with temporal distance. Moreover, it allows pre-trained video diffusion models to extend their generation length with efficient LoRA-based fine-tuning. Extensive experiments show that \method maintains video quality across Wan2.1-14B, HunyuanVideo, and Mochi 1, achieving up to a 1.9× speedup over the original dense attention. With minimal tuning, it enables video generation up to 4× longer while reducing training costs by up to 4.4× compared to direct fine-tuning and accelerating inference by up to 3.7× compared to dense attention inference. Code is released at https://github.com/mit-han-lab/radial-attention.

Radial Attention: O(n log Sparse Attention with Energy Decay for Long Video Generation
Xingyang Li
Muyang Li
Tianle Cai
Haocheng Xi
Shuo Yang
Yujun Lin Lvmin Zhang
Songlin Yang
Jinbo Hu
Kelly Peng
Maneesh Agrawala
Ion Stoica
Kurt Keutzer Song Han
MIT NVIDIA Princeton UC Berkely
Stanford First Intelligence
https://github com/mit- taii-iittttiiin haa--abbradial-atteenn
"Nothing spreads without loss; every signal, every influence, every attention decays with distance."
Inspired by thermodynamic principles
dgaage Ske mone samuee and Sponic mukle conidenio and conalll: Ze sirat dany refletine sarve exfest of levil lighs, Mayy pralostsane malk
Pre Trained
Dense Attention
Laternc:: 1649s
Radial Atfention (Ours)
Lateney: 876s (1.9k Faster)
PSNR: 27.3
(Default Length)
Pre Trained
Latency:
Latency:
LokA Tuned
Dense Attention Vision Reward: 0.133
Tuning Radial Attention
Reward: 0.134
Tuning Cost: 746 GPU Hours
Lateney: 2895%
(4.4x Fewer) Latency: Faster)
(b) 309 Fromes (4x Extension)
Figure 1: present Radial Attention, sparse attention mcchanism with OO log n) computational complexity. Radial Attention accelerates pre-trained Hunyuan Vidco [1] by 1,9x at its default video length while maintaining comparable video quality. When generating 4x longer videos, it reduces tuning costs by up to 4.4x and speeds up inference by up to 3.7x versus dense attention.
Abstract
Recent advances in diffusion models have enabled high-quality video generation, but the additional temporal dimension significantly increases computational costs, making training and inference on long videos prohibitively expensive. In this paper, we identify a phenomenon we term Spatiotemporul Energy Decay video diffusion models: post-softmax attention scores diminish spatial and temporal distance between tokens increase, akin to the physical decay of signal or waves over space and time in nature. Motivated by this, propose Radial Attention, scalable sparse attention mechanism with O(n log n) complexity that translates energy decay into exponentially decaying compute density, which is significantly more efficient than standard ( 2) dense attention and more expressive than linear attention. Specifically, Radial Attention employs simple, static attention mask where each token attends to spatially nearby tokens, with the attention window size
iiniictees equal contributions.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

shrinking with temporal distance. Moreover, it allows pre-trained video diffusion models to extend their generation length with efficient boRA-based tine-tuning Extensive experiments show that Radial Attention maintains video quality across Wan2.1-14B, Hunyuan Video, and Mochi I, achieving up to a 1.9x speedup over the original dense attention. With minimal tuning. it enables video generation up 4x longer while reducing training costs by up to 4.4x compared to direct fine-tuning and accelerating inference by up to 3.7x compared to dense attention inference. Code is released at https://ittuub.com cwwp://tcooo/mmmoo///aeaaoo waatm/mmittaanntt maaiittha-laaee lalaaaaaaalllattnttioio
Introduction
Diffusion models have achieved remarkable success in generating high-quality images [2, 3]. Recent advances have extended their capabil- ities to video generation, producing visually compelling results [4, 7, !].
Corpputanze
Intermoo Latrayy
Traiing Corks
-
Radall (lyckgg
Eal Mareyyyl
Ratia (pslags)
Radal dhilaga
CIPL than
x
However, such advances incur substantial com- putational costs. Unlike image synthesis, the temporal dimension in video synthesis greatly increases token counts compared to images, and the quadratic scaling of self-attention with con- text length renders training and inference long videos computationally prohibitive, restrict- ing model scalability.
200
300
480
30
130
30
350
800
30
340
327
373
4Trano
Figure 2: Radial Attention reduces the computational complexity of attention from O(n²) to O(n log n). When generating 509-frame 720p vidco with Hun-
yuan Video. it reduces the attention computation by 9x, achieves 3.7x speedup, and saves 4.4x tuning costs.
Several prior works tried to mitigate this challenge using sparse attention. For instance, as illustrated in Figure 3()), Sparse VideoGen (SVG) [8] employs online profiling strategy that classifies each attention head as either spatial or temporal and then applies the corresponding sparse mask. While this can accelerate inference, it poses challenges during training. especially for longer videos. The profiling may misclassify attention heads on unseen data distributions, whose error can be reinforced during optimization, leading to degraded performance. Other approaches replace the softmax attention with linear alternatives [9, 10], but these often require substantial architectural changes, where modest fine-tuning is typically insufficient recover the original video quality.
In physics, it is well known that signals and waves lose energy as they propagate through space and time. Inspired by this principle, observe similar phenomenon in attention: post-softmax attention scores between tokens diminish as their spatial or temporal distance increases (sce Figure 4(b)). We term this phenomenon Spatiotemporal Energy Decay and model the decay as an exponential function of both spatial and temporal distances. Based on this model, we unify spatial and temporal attention heads in SVG [8] into Radial Attention, scalable sparse attention mechanism with On log n) computational complexity (see Figure 2). Radial Attention employs static sparse attention mask to translate the concept of energy decay into computation density decay. The mask design is simple yet effective: each token attends to others at similar spatial locations, while the attention window shrinks exponentially with temporal distance, as illustrated in Figure 3(b).
Moreover, since Radial Attention only prunes unimportant token relations without modifying the underlying softmax attention mechanism. it enables efficient adaptation of pre-trained video diffusion models to longer sequences using lightweight fine-tuning, such as LoRA [1 1]. Compared to full- parameter fine-tuning with dense attention, it achieves better video quality, LoRA focuses on updating parameters most critical for temporal coherence and visual fidelity. The length-extension LoRA is also compatible with existing style LoRAs (see Section 5.2).
When generating videos at the default length, Radial Attention accelerates leading video diffusion models of 1an2. [7], Hunyuan Video [1] by up to 1.9x speedup. When generating 4x longer videos, Radial Attention reduces tuning costs by up to 4.4x and accelerates inference by up to 3.7x without sacrificing quality. Some visual examples on Hunyuan Video can be found in Figure 1.

---

Mask Valne:
Dycaninally Pashead Selest
Gio logge1 Unifiedd Statie Mask
X
K
V
Q
X
K
V
ATule
Head
Accoeraro Iefermcc
Aocelenee Traiing
Viteo Adguution
Accilerato Ieferrnne Anodeone Traking
lorge Vider Adagssttre
(4) SVG
(b) Our Radial Attention
Figure 3: Attention pipelines of SVG [8] and our Radial Attention. Softmax is omitted for clarity. (a) SVG dynamically selects either spatial temporal attention for each head to speed up inference. However, it does not overeome the original modcl's length limitation and cannot be trained on unseen distributions like longer videos. (b) Our Radial Attention uses static mask that unifies spatial and temporal attention with On log computational complexity. This static design enables efficient longer-video adaptation.
2 Related Work
Video diffusion models. Diffusion models have achieved state-of-the-art (SOTA) results in image synthesis [2, 3. 12, 10]. Researchers further extend them to the video domain. Early approaches [13, 14, 15, 16] adapted 2D UNets [2, 17] to handle frame sequences by adding temporal modules. Ever since the advent of Sora [4], the community has largely shifted use DiT [18] as the backbone. Latte [19] first proposed decoupled spatial and temporal attention for modeling video sequences. To better capture long-range dependencies and jointly model apatial-temporal dynamics, recent SOTA models have adopted 3D dense attention [20, 21, 5, 22, 1, 7, 6]. However, dense attention is computationally intensive due to the additional temporal dimension, and its cost scales quadratically with the number of frames, posing substantial challenges for both training and deployment.
Efficient video generation. Many techniques developed accelerate image diffusion models- as timestep distillation [23, 24], caching [25, 26), quantization [27, 28, 29], and distributed infer- ence [30, 31, 32]-also apply to video diffusion. However, video models often rely on 3D dense attention. shifting the bottleneck from feedforward to attention layers. Recent works like SageAt- tention [33, 34, 35, 36] and FlashAttention-33 [37] show that quantizing attention can significantly speed up inference. In large language models (LLMs), sparse attention has been widely applied for acceleration [38, 39, 40, 41, 42, 43, 44, 45, 463. For instance, Long LoRA [39] combines two local sparse attention patterns with shifting to achieve a global receptive field in video understanding. Pow- erAttention [45] restricts attention to power-of-two token distances, yielding O(n log n) complexity. However, these methods ignore the inherent spatial and temporal structure in video data, making them suboptimal for video generation (see Section 5.2). To better exploit this structure, several video-specific sparse attention methods have been proposed [8, 47, 48, 49]. For example, STA [47] uses sliding 3D windows for local attention, and SVG [8] dynamically selects spatio-temporal patterns for cach head. Both improve efficiency but struggle with long videos: STA's fixed receptive field limits long-range dependencies, while SVG's runtime profiling becomes unreliable for unseen long video distributions. In contrast, our Radial Attention employs static On log n) pattern across all heads, accelerating both training and inference and enabling efficient extension to longer videos.
Long video generation. Due to the quadratic cost of dense attention, training and inferernce on long videos remain highly expensive. RIFLEx [50] extends video length by modifying RoPE [51] frequencies to tackle temporal repetition and motion deceleration, allowing 2 extrapolation with the pre-trained models. However, still suffers from poor video quality (e.g., blurring) when generating longer videos. Dalal el al. generate short video segments and stitch them together via test-time training layers [52]. Framepack [53] adopts an autoregressive strategy, generating short clips sequentially based context frames that are encoded into fixed number of tokens. Other approaches replace dense attention with lincar attention [10, 9, 54, 55, 56, 57, 58, 59], offering faster computation and global receptive fields. However, linear attention struggles capture local details [60], often degrading quality. Our Radial Attention strikes a middle ground between O(n²) dense attention and (nn) linear attention, achieving O(n log n) complexity while preserving the visual fidelity. Moreover, it can be efficiently fine-tuned from existing models using LoRA [11], enabling scalable longer-video generation with minimal overhead.
Attention with O(n log n) complexity. Preliminary efforts in this direction include Reformer [61], which approximates dense attention via docality-sensitive hashing; H-Transformer [62], which
3

---

imposes a hierarchical structure on the attention matrix; Multi-resolution attention [63], which recursively refines high-attention regions: Fast Multipole Attention [64], which adapts the classical fast multipole method; and LogSparse Transformer [65] for time-series forecasting, which restricts each token to attend to ((logn) positions per layer. However, these methods are often hardware- unfriendly and exhibit limited scalability. In contrast, our approach employs simple, block-friendly static mask that scales efficiently while maintaining strong modeling capacity.
Preliminary
Diffusion models synthesize vidcos by sampling Gaussian noise XT (0, I) in latent space and progressively denoising it through a neural network to produce a clear latent Xu. which is subsequently decoded into the final video using pre-trained decoder. Compared to images, videos introduce an additional temporal dimension, significantly increasing the number of latent tokens. For instance, generating 5-second 720p video in Hunyuan Video [1] requires approximately I 110K tokens. Excessive latent compression degrades video quality [3], limiting token reduction.
To capture spatiotemporal correlation in video generation, recent models [7, 1, 22, 5] use 3D dense attention, which computes interactions between all token pairs. Given tokens with embedding dimension d, attention is computed as:
T (~v Vd
[QK]
Attention (ariionn(Q,A K, V) = softmax
V,
where Q, K,K.V V E Faxd the query, key, and value matrices. The computation of QK" incurs (n²²) time and memory complexity. While FlashAttention [66, 67] series reduce memory overhead, the quadratic time complexity remains bottleneck, especially for long high-resolution videos. Thus, designing more efficient attention mechanisms is vital for scaling video diffusion models.
To mitigate this computational burden, sparse attention restricts interactions to subset of token pairs. Formally, this is achieved by adding sparsity mask M E (-00,,0) 00000 to the attention logits:
(QKT) M
SparscAttention( Q K, V) softmax
V.
(2)
Vd
Entries oo ignored in the softmax computation. Various schemes have been proposed to construct the mask. Static methods, such as STA [47], use fixed sparsity patterns but are less expressive. In contrast, dynamic schemes like SVG [8] does dynamic sparse pattern based input content to improve fidelity. However, dynamic masking introduces online mask decision overhead and does not apply to training. Can we design static attention pattern that matches the expressiveness of dynamic methods and can also be used in training?
4
Method
The key insight of Radial Attention is that attention scores between tokens decay with increasing spa- tial and temporal distance. This motivates to allocate computation based on the inherent spatiotem- poral correlations. In Section 4.1, we characterize the spatiotemporal energy decay phenomenon in attention. In Section 4.2, we formally define Radial Attention, which translates energy decay into cor- responding compute density reduction, enabling speedup. We also analyze its complexity and approx- imation error, showing that the complexity is O(n log n) and the effectiveness of our mask. Finally, in Section 4.3, we show how to extend pre-trained models to longer videos using Radial Attention.
4.1 Spatiotemporal Energy Decay in Attention
In Figure 4(a), we show two post-softmax attention maps from Hunyuan Video [1]. Following the terminology SVG [8], the left map referred to as spatial attention, where each token primarily attends to nearby tokens within adjacent frames. The right map represents temporal attention, where each token focuses on tokens at the same spatial location across different frames. Figure 4(b) illustrates the attention score distributions of these two maps, along with a third curve of averaged attention scores over multiple heads and diffusion steps. Figure 4(b1), we show the average attention score between tokens at the same spatial location but with increasing temporal distance. In Figure 4(b2),

---

Soat MANREI L
80
(Rigg Deoggna Spoiill te
AilAAttctionn (lane Suppere theoe
(D)-merioo Score Dismbttio
340
Figure 4: (a) Example spatial and temporal attention maps from Hunyuan Video (defined in Section 4.1). (b) Attention score distributions. (b1): Average score between tokens at the same spatial location decreases with temporal distance (b2): Average attention score within a frame decreases with spatial distance. Spatial and Temporal Attention refer to the distributions derived from the corresponding maps in (a). Average means averaging over multiple random maps and diffusion steps. The plots indicate that spatial attention shows a high temporal decay and relatively low spatial decay, while temporal attention exhibits the opposite.
we show the average score between tokens in the same frame as spatial distance increases. In both cases, attention scores exhibit clear decay pattern with increasing distance between the query and key tokens. We refer to this phenomenon as Spatiotemporal Energy Decay. Moreover, regression analysis suggests that this decay closely follows exponential distribution (see Section 5.3).
Specifically, following the notation from Section 3, assume the video latent consists of f frames, each containing s tokens (in total n = fs tokens). Consider a query token located at the ko-th spatial position of the 1n-th frame. The corresponding attention score after applying softmax, denoted by p E [0, 1I". is given by p = softmax to k) Then there exist constants a, 3 > 0 and Cee 0, for each key token spatial position in frame j satisfying
Pjot1 S Crue aajj-ial Blt-kol pios+
(3)
Parameters a and 3 control temporal and spatial decay, respectively. High 3 (strong spatial locality) and low model temporal attention, while high and low 3 capture spatial attention, shown in the empirical plots in Figure 4(b). This motivates our unified sparsity pattern that leverages both spatial and temporal decay in principled manner.
4.2 Radial Attention: Convert the Energy Decay to Compute Density Decay
Radial Attention simulates energy decay through compute density decay to save computation.
Temporal density decay. Along the temporal dimension, Radial Attention applies an exponential decay rule: the compute density between tokens in frame and frame j is (3) [log, (mxx([i)))))) This forms a structured pattern as illustrated in Figure 5(a) with 2 [logg
diagonal bands centered on the main diagonal (band O). Bands above and below the diagonal are indexed as 1, 3. ... and -1, 2, -3,...... respectively. Each band's width doubles relative to the previous one, ensuring that the total computation per band remains bounded by constant. The attention from tokens frame to frame j lies in band sign(j - i) [log, max( jl. 1)]. The central band (band 0) retains 100% compute density, while each successive band moving outward has half the compute density of the preceding one- producing a radial decay effect with progressively lighter colors.
2)
Spatial density decay. As observed in Figure and formalized in Equation 3, most attention energy concentrated on tokens similar spatial locations across frames. We preserve these high-energy interactions, which yield diagonal- like structures within each frame-to-frate temporal decay, the computed diagonal width of these blocks shrinks as the temporal distance between frames increases. Specifically, as shown in Figure 5b)) the diagonal width for attention between frame
attention block. Due to
i and frame j is given by L. 120O max( =J] If it falls below 1, instead of further narrowing the diagonal, we reduce the frequency of diagonals. Specifically, we only retain diagonals in those blocks where i أ mod
2. logg max([i-)l.1]]
0 keep the same amortized attention density decay.
Formal definition. Here we formally define the Radial Attention mask. We construct a 4D attention mask M E {-00, 00' fxexx where each element Mij.a.. 0 indicates that the token at spatial position & in frame i is permitted to attend to the token at position / in frame Conversely, M,, Maj.k.
5

---

Key
Key
Key
Quay
Querry
200 102 1999
Vial Salee
Scall Sihee
--
(a) Compute Density Distribution
(b) Attenntion Mask
Mask used in Hunyuan
Figure 5: (a) The compute density pattern. The attention map is divided into 2 2[ogg (max (f. 2)) bands (here, the number of frames f = 12) based the temporal distance between tokens. The central band has full compute density, while each successive outer hand has half the density of the previous one. Except for band #1, each band also doubles the diagonal width
predecessor. (b) The corresponding attention mask for (a). The compute density is reflected in the compute diagonal width of each frame-to to- to-frame block. When the diagonal width drops below 1, we reduce the frequency of diagonals. We additionally add an attention sink. (c) An example mask used in Hunyuan Video, illustrating the final sparsity pattern in practice.
-oo denotes that attention between the token pair is suppressed. The mask is constructed according to:
0,
if 2 log, max(]e- (=).1)-)1111 < 8 and jk - + 1 2 maxt i 21.37
Mij.k,1
0,
i jl mod [Bogg max(le 21.11) = 0 and k
4
Oc.
otherwise
The final attention mask M E {-,0,0) used in the attention operation of Equation 2 is obtained by flattening frame and spatial indices: Mistk,jt+ = MM.j.e.. For better quality, we incorporate an attention sink [38, 8] the first frame's attention is crucial. Figure 5(c) shows a example mask we use in Hunyuan Video for generating a 253-frame 720p video. This strategy keeps spatial interactions with high temporal proximity while using sparse sampling for distant frames to maintain efficiency.
Relation to SVG. Radial Attention unifies spatial and temporal attention in SVG [8] using a single attention mask. Specifically, the central band (band 0 in Figure 5(a)) of our mask already captures dense spatial interactions, effectively subsuming the spatial attention in SVG. For temporal attention, SVG overlooks temporal decay, allocating unnecessary computation relevance. In contrast, Radial Attention reduces attention to these regions and reallocates the budget toward tokens nearer in time, achieving both improved efficiency and enhanced modeling.
distant frames with low
Complexity analysis. The computational cost of our method is proportional to the number of zeros in the attention mask M, When the number of frames ff is large, we derive the following upper bound:
loga $1 Σ
25²
Doga f1-1
#zeros M <
4
21+1
Σ
logg s s1+1 fs
(5)
2"
central band and sink
[log *J+1
diagonal 2idth
diagonal width
< 4
4snglog, log, s).
(6)
A detailed derivation of Equation 5 can be found in Appendix A. From Equation 6, we find that for long videos (i.e., large f) with fixed resolution s, the computational complexity scales as O(n log n). Empirical results Hunyuan Video, shown in Figure 2, confirm this trend. Notably, our Radial Attention reduces attention computation by 9x compared to dense attention for 4x longer videos.
Error analysis. Following Equation 3, we derive ervor bound for the attention score corresponding to query token at position kn in the 1t frame. P = softmax ( Q1g KT + M denotes the masked attention score. The & attention error of our approximated attention bounded as follows:
8e (3+11)
+ e ca(++))
| 1p-pph p|11 1 Czel
(1-e e- +
= O(CraL min(2/2))))) (7)
1 - e 1 1-e a

---

Table 1: Quantitative results default video length. Under same computation budget, our method consistently outperforms STA and PA PSNR, SSIM, and LPIPS, matches the vidco fidelity SVG, and achieves 1.8x speedup on Hunyuan Video and Wan2. 1 1-14B on single H100 GPU.
Model
Method
PSNR (f)
(S)
LPIPS ())
Vision Reward (t)
PFLOPs
Latency (6)
Speedup
Original
0.141
612
1649
Hunyuan
STA (FA3)
26.7
0.866
0.167
0.132
331
719
2.29x
Videa
0.256
(117 frames)
SVG
27.2
0.895
0.114
0.144
340
867
I 1.90x
Ours
27.3
0.886
0.114
0.139
339
1.88x
Original
0.136
560
16,30
STA (FA3)
22.9
2.01x
Wan2. 1-14B
(69 frames)
PA
22.4
0.790
0.176
0.126
324
978
1.67x
SVG
23.2
0.825
0.202
0.114
324
1.71x
Ours
23,9
0.842
0.163
0.128
323
917
1.77x
Proof details are provided in Appendix A.2. As Equation 7 shows, the error decreases exponentially with larger decay rates a and In Section 5.3, we further empirically compare this error bound that of SVG, showing that Radial Attention achieves smaller errors, thereby validating its effectiveness.
Hardware-friendly block sparsity. To ensure efficient execution modern hardware, attention is computed over 128 x 128 blocks rather than individual 1 x tokens [68, 8, 40, 43, 44, 66].
4.3 Low-Rank Adaptation for Long Videos
Although employ efficient attention mechanism, the pre-trained model was originally trained on short videos. Recent works [50] have explored training-free methods for extending generation to longer videos, but their performance remains limited due to length distribution mismatch. Training di- rectly on long videos, meanwhile, is computationally prohibitive. Radial Attention alleviates this chal- lenge by reducing the training time complexity On logn). Importantly, it preserves critical inter- token relations in the softmax attention, allowing the original pre-trained weights to remain largely intact. Thus, only minimal fine-tuning required. To further minimize training overhead, incorpo- rate low-rank adapters (LoRA) [1] 39] into the attention mechanism. Specifically, LoRA is applied to the query, key, value, and output projections of the attention layers, enabling efficient fine-tuning with significantly reduced memory and computational costs. Empirically, we find that LoRA fine-tuning with Radial Attention not only minimizes overhead but also improves video quality by refining only the most critical weights and attention more effectively. See Section 5.3 for detailed results.
Experiments
5.1 Setups
Models. We benchmark our method on three popular text-to-video diffusion models: Mochi 1 [22], Hunyuan Video [1], and Wan2.1 [7], which contain 10, 13, and 14 billion parameters, respectively. Mochi can generate up to 5-second video with 480p resolution and 162 frames. Hunyuan Video can generate up a 5-second video with 720p resolution and 125 frames. Wan2.1-14P can generate up to 5-second video with 720p and 81 frames.
Benchmarks. We use Vision Reward [69] (higher is better) approximate the human rating of the generated videos. For pre-trained models evaluated at their default video lengths, we further report PSNR and SSIM quantify numerical similarity, and LPIPS [70] assess perceptual differences between the outputs of the original models and the benchmarked methods. For longer-video genera- tion, we additionally metrics of subject consistency, aesthetic quality, and imaging quality. where the original models exhibit notable degradation.
VBBench-lon[[71 to evaluate our fine-tuned models. Specifically, we report
Baselines. We compare Radial Attention against the following methods:
SVG [8]: Accelerates video models with sparse attention by dynamically classifying attention
heads as spatial temporal and applying corresponding masks.
Spatial/eempral: The respective attention masks used in SVG's spatial and temporal heads, as described Section 4.
STA [47]: Applies sliding window attention to capture spatially and temporally local dependencies.
7

---

Table Quantitative results at extended (2x and 4x) video lengths. With minimal fine-tuning, our method maintains the quality regarding Vision Reward and multiple VBench dimensions (Subject Consistency, Aesthetic Quality, and Image Quality) when the length grows. It also achieves high sparsity, reducing training costs by up 4.4x and delivering up 3.7x inference speedup.
Training Time (h) Speedup
Training
Inference Time (s)
Inference Speedup
Vision Reward (t)
VRench
Model
#Faaes
Metbod
Sparsity
A.Q.
I.Q.
125 (1 x)
Original
0.00%
225
0.119
0.959
0.643
0.672
Original
0.00%
797
1.00X
0.122
0.953
0.603
0.611
Fx
0.00%
797
1.00x
0.622
0.614
Spotial
80.5%
16.0
2
335
2.38x
0.054
0.979
0.607
0.670
Temporal
80.7%
16.2
2.78%
338
2.36X
0.104
0.963
0.620
0.658
253 (2x)
Long LoRA
80.6%
16.6
2.71
363
2.20X
0.112
0.958
0.620
0.685
PA [45]
80.4%
16.7
2.69X
334
2.39
0.109
0.967
0.608
0.633
Hunyuan
SANA
12.8
3,52x
285
2,80
-0,205
0.907
0.300
0.442
Video
Full
0.00%
45.0
1.00X
797
1
0.124
0.955
0,616
0.648
Ours
80.8%
16.2
2.78x
339
2.35
0.126
0.968
0.623
Original
0.00%
2895
L,00X
0.054
0.988
0.545
0.451
RIFLEx
0.00%
2895
1,00
0.037
0.989
0.539
0.456
Spotial
88.3%
20.7
4.52
755
3.83
0.112
0.922
0.598
0.664
Temporal
21.1
509 (4x)
Long LoRA
88.4%
20.9
4.48X
8013
3.61
0.130
0.936
0.618
0.689
PA [45]
88.2%
21.8
x
766
3.78
0.128
0.950
0.590
Full
0.00%
93.6
L.00X
2895
1,00X
0.133
0.977
0.590
0.635
Ours
88.3%
21.4
4.37 X
781
3.71
0.134
0.973
0.623
0.672
163 (1
Original
0.00%
112
0.071
0.973
0.623
0.672
Original
0.00%
302
1,00X
0.040
0.937
0.551
0,466
Spatial
76.1%
8.57
L.55%
186
1.62X
0.088
0.935
0.596
0.595
Temporal
76.3%
1,60X
0.075
0.591
331 (2x)
Long LoRA
76.0%
9,07
L.65X
210
1.44X
0.095
0.950
0.596
0.630
PA [45]
77.8%
8.53
1.65
0.101
0.610
SANA
8.22
1.82x
166
1.82
-0,201
0.905
0.334
0.568
Mochi
Full
0.00%
15.0
1.00X
302
1,00X
0.095
0.923
0.610
0.594
Ours
76.4%
8.43
L,78x
185
1.63X
0.110
0.951
0.615
0.602
Original
0.00%
992
1,00
-0.091
0.916
0.383
0.322
Spotial
85.2%
17.4
2.83
382
2,60X
0.091
0.930
0.611
0.585
Temporal
85.4%
17.6
2.80x
393
2,52
0.028
0.931
0.556
0.536
667 (4x)
Long LoRA
86.0%
19.0
2.59X
426
2.33%
0.086
0,944
0.584
0.543
PA [45]
86.5%
17.3
2.84%
381
2.60X
0.107
0.956
0.633
0.650
Full
0.00%
49.2
1.00x
992
1,00
0.099
0.934
0.613
0.613
Ours
85.5%
17.4
2883x
386
2.57
0.113
0.958
0.618
0.638
81 (1x)
Original
0.00%
1630
0.135
0.973
0.623
0.672
W
Original
0.00%
5735
1,00
0.109
0.946
0.598
0.614
-14B
161 (2x)
Full 0.00%
28.0
1.00
5735
1.00x
0.150
0.966
0.590
0.689
14.5
2.01X
0.607
PowerAttention (PA) [45]] A sparse attention mechanism with Onn log n) complexity for LLMs, attending only to tokens at power-of-two distances.
LongLoRA [39]: Uses shifted local attention to efficiently extend the context window of LLMs.
SANA[10]: An efficient diffusion model backbone with linear attention. We replace softmax attention with SANA's for adapting to longer videos.
RIFLEx[50]] Training-free video length extrapolation by adjusting the frequency of RoPE [51].
Implementation details. In terms Radial Attention implementation, we use FlashInfer [72] for inference and Block-Sparse-Attennie Attention [73] with F [67] backend during training. For default-length inference, we evaluate Hunyuan Vidco on 117 frames at 768p resolution (768x1280) and W 69 frames at the same resolution. Following SVG, we apply dense attention during the first 12 steps as a warm-up phase for all models. Additionally, we keep dense attention in the first DiT block to maintain quality. We measure all the latencies with a single NVIDIA H100 GPU.
For longer-video generation, we fine-tune the model with videos that are 2~4x longer than the default length from Open Vid- 1M [74]. Specifically, we sample 2k top-scoring videos in aesthetic and motion scores for each extended length. We use 8 H100 GPUs for training, which takes around 16~21 hours for Hunyuan Video, 8~ 7 hours for Mochi 1, and 15 hours for Wan 2.1. Inference latency for Wan Video and Mochi are evaluated using 8 H100s.
2.1 measured on single H100, while Hunyuan Video See Appendix B for more details.
8

---

Prompt: A close-up shot capures chaster of phump, dony grapes, glasoning under seft studio lighting os they slowly rolote on a sleek, reffactive tuble. The grapes, varying shudes of deep purple and green, showoure their smooth, lout skins Gay deoplets
Orriginal WanZ.1-1 1-1R
Lateney: 1630s
Wanz. 1-14B-Raddi Astsctoo(OUus) Tateney: 917 (LSxfaster) PSNR: 25,0
Prompt: a person. Realsstic, Natural lighting Cazual
Original Wan2.1-14D
Latency: 1630s
WanZ. Radiall Aftention(uurs) Lateney: 9174 (LSxxfaster) PSNR: 29.9
Figure 6: Examples of videos generated by Radial Attention and the original Wan2.1-14h in the default video length. Radial Attention mirrors the video quality of the original model.
Orianal Harraaan ses 509 bassacçk)
HusgasWWarlLlk Detae Alcnice, 565 iuma (40)
Huns www.eaaaraaaaaa Lator Lall-aa Radia Auication (Dan). 500 fransc
Auerage Vision Broond
Acringe Vision Rrnaad
Aceray Visce Recard:
Pronept opon fild desled with rorn occationally clopnng towordh groand Fornery uhra ban are londing their fiolds, walking alordru pendrraal ahoreis fand
and scatioral small farm, The sty a clear olas with fuff whire elonsl Canar poocefialy the aisiance thair headl
Prompe, A gesole, futti shesp skite woul and lorss, dipretaticc cies Acaid dowe alp vatar From cryitel HONY. The sheps' was aimsur Macher rol ahasss revouling trantee consonted cupt dision. host duth with graan grass WldTomers. creaiin scrond and piceersgge fandscape
Figure 7: Visual comparison of Hunyuan Video with 4x length extension (509 frames). LoRA tne-tuned models using Radial Attention achieve higher vision rewards, outperforming dense attention baselines, while achieving a 3.7% speedup and reducing tuning costs by 4.4x.
5.2 Main Results
Training-free inference acceleration. Table presents a quantitative comparison of Radial Attention against three strong sparse attention baselines on Hunyuan Video [1] and Wan2.1-14B [7] at their default generation lengths. Corresponding visual results are provided in Figure 6 and Appendix C.I. Under the same compute budget (measured in PFLOPs), Radial Attention preserves the video quality of dense attention while consistently outperforming STA and PA similarity metrics (PSNR, SSIM, LPIPS), and matches the quality of SVG. While PA shares a similar On log n) complexity with our design, it ignores the spatio-temporal locality inherent in video data, making suboptimal in practice.
Regarding efficiency, we adopt the same system optimization used in SVG [8]. Specifically, on single H100, our Radial Attention achieves 1.9x and 1.8x end-to-end speedups for Hunyuan Video and Wan 2.1, respectively, matching the theoretical compute budget savings (1.8x and 1.7x fewer PFLOPs). Although STA yields slightly higher specdup by using FlashAttention-3 (FA-3) [37], it suffers from noticeably degraded visual quality. Our current implementation uses FA2 [67]. Upgrading to FA3 is orthogonal to our algorithm and is left as future work.
Long video generation. Table provides results for video generation at 2x and 4x the original lengths, with visualizations available in Figure 7 and Appendix C.2. For 1-n2.1114B, only 2x extrapolation is reported due its significantly higher computational and memory costs. To ensure fairness, all sparse attention baselines use similar sparsity ratios.
When generating longer videos, the original models without further tuning exhibit significant quality degradation, especially for 4x video length extension. While RIFLEx improves performance at 2x length extrapolation, its quality deteriorates beyond that, indicating limited extension capability. Spatial and temporal sparse attentions suffer from limited reception fields; on the other hand, LongLoRA and PA, though with global reception field, fail capture spatial-temporal correlations, resulting in degraded quality. Interestingly. PA exhibits a large gain in Vision Reward after fine-tuning, suggesting that its original sparse pattern misaligns with the pre-trained attention distribution. Fine-tuning allows the model to adapt to the imposed attention sparsity, improving alignment and quality. SANA, which replaces softmax attention with linear attention, requires massive retraining and fails under fine-tuning-based video length extension. In contrast, Radial Attention achieves quality on par with LoRA fine-tuned dense attention models. Notably, even slightly improves the Vision Reward over the pre-trained model at the default video length.

---

0.14
Dense Aitn Full FT
Douse Attr-LoRA FT
Kadial Atte+Full F-]
10.
Vaiion Roward (7)
0.12
Radial Atteel oR A FT (Cum)
Attention Score
Attention Score
0.06
0.05
100
250
400
550
700
25
18
12.5
15
17.5
30
#Frames
Temporal Distance
Spatial Distance
(+) LoRA Effectivenesss Results
(b) Regression Analysis
Figure 8: (a) Radial Attention outperforms dense attention in generation quality. When combined with LoRA, it further improve the quality while significantly reducing training costs. (b) We model the attention decay curves using the exponential function y = exp (xp((ace az b). It fits the data well, achieving It² > 0.985.
Thanks to On n) complexity, Radial Attention delivers substantial inference and training speedups over dense attention, as detailed in Table and Figure 2. For instance, when generat- ing 4x longer videos, we can save up to 4.4x training costs and get up to 3.7x inference speedup.
Compatibilityy with existing LoRAs. A key advantage of Radial Attention is its seamless compat- ibility with pre-trained task-spe LoRAs (e.g., artistic style transfer). We observe that Radial Attention is compatible with existing style LoRAs in both the default length settings, and the longer video settings by directly merging the LoRA weights of Radial Attention in long videos and the style LoRAs. Visualizations and further analysis can found Appendix C.3.
5.3 Ablation Study & Analyses
Effectiveness of Low-Rank Adaptation. Figure 8(a) compares Vision Reward between full fine- tuning and LoRA as video length increases. For dense attention, LoRA fine-tuning lags behind full fine-tuning until 4x length extension. However, with our proposed Radial Attention, LoRA fine-tuning matches or even outperforms full fine-tuning, suggesting that Radial Attention not only scales better computationally, but also makes the model easier adapt longer-video generation.
Attention error. We evaluate the average attention output Mean Squared Error (MSE, lower is better) of Radial Attention achieves an MSE of 3.9 x 10 significantly lower than 4.4 x 10-3 3 for SVG and 1.5 x 10 for STA, demonstrating the effectiveness of our mask in preserving attention fidelity.
Van2.1-14B, comparing it to SVG [8] and STA [47]. Radial Attention
Regression results. We perform regression analysis using the model y = exp(-az + b) to fit the average attention decay curves in Figure 4. As illustrated in Figure 8, the fitted curves achieve an R² value of over 0.985, indicating that the exponential functions can well model the decay.
More ablations on Radial Attention design choice. Please refer to Appendix D for more details.
6 Conclusion & Discussion
In this work, we propose Radial Attention, an On log sparse attention for efficient long video generation. We observe Spatiotemporal Energy Decuy in video diffusion models, which motivates a unified attention pattern with sub-quadratic complexity. At default video length, Radial Attention achieves up to a 1.9x speedup with high fidelity. For videos up to 4x longer, Radial Attention preserves quality and delivers up to 4.4x and 3.7x speedups in training and inference. respectively, with minimal LoRA fine-tuning, This work contributes toward scalable, high-quality video generation and offers a foundation for efficient long-range attention in broader sequence modeling tasks.
Limitations. The assumption of exponential decay for attention scores (Equation 3) simplifies the complex spatiotemporal dependencies in natural video data. While aiding theoretical analysis, future work could improve efficiency and performance by more deeply understanding and modeling the underlying data structure. As shown in Equation 6, our method still has quadratic complexity with respect to resolution. Future work should explore more efficient attention mechanisms and pre- tre-training strategies, in NSA [75] and MoBA [76], to better support long, high-resolution videos.
10

---

Acknowledgments
We thank MIT-IBM Watson Al Lab, National Science Foundation, Hyundai, and Amazon for supporting this research.
References
[1] Weijie Kong. Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint xxXiv2412.003600 2024. 1, 3, 4, 7, 9, 24, 26
[2] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NewrlPS, 2020. 2,
[3] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffuusion models. In CVPR, 2022. 2, 3.4
[4] OpenAL. Sora: Video generation models as world simulators, 2024. 2,
[5] Zhuoyi Yang, Jiayan Teng. Wendi Zheng. Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong. Xiaohan Zhang, Guanyu Feng, al. Cogvideox: Text-to-vides diffusion models with an expert transformer. arXiv preprint arXiv:2408. 06072, 2024. 3.4 4
[6] Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Eitan Richardson, Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon, Poriya Panet, Sapir Weissbuch, Victor Kulikov, Yaki Bitterman, Zeev Melumian, and Ofir Bibi. Ltx-video: Realtime video latent diffusion. arXiv preprint arXiv:2501.00100 2024. 3,3
[7] Ang Wang. Baole Ai, Bin Wen, Chaojie Mao, Chen- Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang. Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fang. Tianxing Wang. Tianyi Gui, Tingyu Weng, Tong Shen, Wei Lin, Wei Wang. Wei Wang. Wenmeng Zhou, Wente Wang. Wenting Shen, Wenyuan Yu, Xianzhong Shi, Xiaoming Huang, Xin Xu, Yan Kou, Yangyu Lv, Yifei Li, Yijing Liu, Yiming Wang, Yingya Zhang, Yitong Huang. Yong Li, You Wu, Liu, Yulin Pan, Yun Zheng, Yuntao Hong, Yupeng Shi, Yutong Feng. Zeyinzi Jiang, Zhen Han, Zhi-Fan Wu, and Ziyu Liu. Wan: Open and advanced large-scale video generative models. arXiv preprint 2xXi:2503.20314 2025. 2, 3, 4, 7, 9, 24 26
[8] Haocheng Xi, Shuo Yang, Yilong Zhao, Chenfeng Xu, Muyang Li. Xiuyu Li. Yujun Lin, Han Cai, Jintao Zhang, Dacheng Li, et al. Sparse videogen: Accelcrating video diffusion transformers with spatial-temporal sparsity. In ICML, 2025. 4, 6, 7, 9, 10, 25, 26
[9] Hongjie Wang, Chih Yao Ma, Yen Ben-Cheng Liu, Hou, Tao Xu, Jialiang Wang, Felix Juefei-Xu, Yaqiao Luo, Peizhao Zhang. Tingbo Hou, et al. Lingen: Towards high-resolution minute-length text-to text-to-video generation with linear computational complexity. In CVPR, 2025. 2,3
[10] Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, et al. Sana: Efficient high-resolution image synthesis with linear diffusion transformers. ICLR, 2025. 2,
[11] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. Weizhu Chen, al. Lora: Low-rank adaptation of large language models. In ICLR, 2022. 2,3, 3,7,
[12] Dustin Podell, Zion English, Kyle Laccy, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In ICLR, 2024. 3
[13] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. NeurlPS, 2022.
[14] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis, Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, 2023. 3
[15] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets, arXiv preprint arXiv:2311.15127 2023.
11

---

[16] Yuwei Guo, Ceyuan Yang. Anyi Rao, Zhengyang Liang. Yaohui Wang, Yu Qiao. Maneesh Agrawala. Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. In ICLR, 2024.
[17] Olaf Ronneherger, Philipp Fischer, and Thomas Brox, U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisee Intervention-MICCA/ 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234 241. Springer. 2015. 3
[18] William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023.
[19] Xin Ma. Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03448 2024.
[20] Bin Lin, Yunyang Ge, Xinhua Cheng, Zongjian Li, Bin Zhu, Shaodong Wang. Xianyi He, Yang Ye. Shenghai Yuan, Liuhan Chen, et al. Open-sora plan: Open-source large video generation model. arXiv preprint 241:::2422.00111111 2024. 3
[21] Xiangyu Peng, Zangwei Zheng, Chenhui Shen, Tom Young. Xinying Guo, Binluo Wang. Hang Xu. Hongxin Liu, Mingyan Jiang. Wenjun Li, Yuhui Wang, Anbang Ye, Gang Ren, Qianran Ma. Wanying Liang, Xiang Lian, Xiwen Wu, Yuting Zhong. Zhuangyan Li, Chaoyu Gong, Guojun Lei, Leijun Cheng. Limin Zhang, Minghao Li, Ruijie Zhang, Silan Hu, Shijie Huang, Xiaokang Wang. Yuanheng Zhao, Yuqi Wang, Ziang Wei, and Yang You, Open-sora 2.0: Training commercial-level video generation model in $200k. arXiv preprint arXiv:2503.0000 09642, 2025. 3
[22] Genmo Team. Mochi 1. https: / gi thub , com/ genmoai/modele 2024. 3, 4, 24,
[23] Tianwei Yin, Qiang Zhang, Richard Zhang, William T Freeman, Fredo Durand, Eli Shechtman, and Xun Huang. From slow bidirectional to fast autoregressive video diffusion models. CVPR, 2025.
[24] Jiachen Li, Weixi Feng. Tsu-Jui Fu, Xinyi Wang, S Basu, Wenhu Chen, and William Yang Wang. T2v-turbo: Breaking the quality bottleneck of video consistency model with mixed reward feedback. In Neur!PS, 2024. 3
[25] Xinyin Ma, Gongfan Fang, and Xinchao Wang. Deepcache: Accelerating diffusion models for free. In CVPR, 2024.
[26] Feng Liu, Shiwei Zhang, Xiaofeng Wang, Yujic Wei, Haonan Qiu, Yuzhong Zhao, Yingya Zhang, Qixiang Ye, and Fang Wan. Timestep emhedding tells: It's time to cache for video diffusion model. In CVPR, 2025.
[27] Xiuyu Li. Yijiang Liu, Long Lian, Huanrui Yang. Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. Q-diffusion: Quantizing diffusion models. In ICCV, 2023, 3
[28] Tianchen Zhao, Tongcheng Fang, Enshu Liu, Wan Rui, Widyadewi Soedarmadji, Shiyao Li, Zinan Lin, Guohao Dai, Shengen Yan, Huazhong Yang, et al. Vidit-q: Efficient and accurate quantization of diffusion transformers for image and vidco generation. In ICLR, 2025. 3
[29] Muyang Li, Yujun Lin, Zhekai Zhang, Tianle Cai, Junxian Guo, Xiuyu Li, Enze Xie, Chenlin Meng, Jun- Yan Zhu, and Song Han. Svdquant: Absorbing outliers by low-rank component for 4-bit diffusion models. In ICLR, 2025.
[30] Muyang Li, Tianle Cai, Jiaxin Cao. Qinsheng Zhang. Han Cai, Junjie Bai, Yangqing Jia, Ming- Yu Liu, Kai Li, and Song Han. Distrifusion: Distributed parallel inference for high-resolution diffusion models. In CVPR, 2024. 3
[31] Jiarui Fang, Jinzhe Pan, Jiannan Wang, Aoyu Li, and Xibo Sun. Pipefusion: Patch-level pipeline parallelism for diffusion transformers inference. arXiv preprint arXiv:245..1 244..4445.444 2024. 3
[32] Jiarui Fang, Jinzhe Pan, Xibo Sun, Aoyu Li, and Jiannan Wang. xdit; an inference engine for diffusion transformers (dits) with massive parallelism. arXiv preprint arXiv:2411.01733, 2024.
[33] Jintao Zhang, Jia Wei, Pengle Zhang, Jun Zhu, and Jianfei Chen. Sageattention: Accurate 8-bit attention for plug-and-play inference acceleration. In ICLR, 2025. 3
[34] Jintao Zhang. Haofeng Huang, Pengle Zhang, Jia Wei, Jun Zhu, and Jianfei Chen. Sageattention2: Efficient attention with thorough outlier smoothing and per- thread int4 quantization. In ICML, 2025. 3
12

---

[35] Jintao Zhang, Jia Wei, Pengle Zhang, Xiaoming Xu, Haofeng Huang, Haoxu Wang, Kai Jiang, Jun Zhu, and Jianfei Chen. Sageattention3: Microsealing fp4 attention for inference and an exploration of 8-bit training, arXiv preprint arXiv:2505. 115944 2025. 3
[36] Jintao Zhang, Xiaoming Xu, Jia Wei, Haofeng Huang, Pengle Zhang, Chendong, Xiang, Jun Zhu, and Jianfei Chen. Sageattention2+:: A more efficient implementation of sageattention2. arXiv preprint urXivi2505.21136 2025.
[37] Jay Shah, Ganesh Bikshandi, Ying Zhang. Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-:: Fast and accurate attention with asynchrony and low- precision. NeurIPS, 2024. 3,9
[38] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In /CLR, 2024. 6,6
[39] Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-context large language models. In ICLR, 2024. 3, 7,
[40] Huiqiang Jiang. Yucheng Li, Chengruidong Zhang. Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir Abdi, Dongsheng Li. Chin- Lin, et al. Minference 1.0: Accelerating pre-filling for long-context Ilms via dynamic sparse attention. NewrlPS, 2024. 3,7
[41] Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia Wei, Haocheng Xi, Jun Zhu, and Jianfei Chen. Spargeatin: Accurate sparse attention accelerating any model inference. In ICML. 2025. 3
[42] Jiaming Tang. Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context I1m inference. In ICML, 2024. 3
[43] Ruyi Xu, Guangxuan Xiao, Haofeng Huang, Junxian Guo, and Song Han. Xattention: Block sparse attention with antidiagonal scoring. In ICML, 2025. 3.7
[44] Shang Yang. Junxian Guo, Haotian Tang. Qinghao Hu, Guangxuan Xiao, Jiaming Tang, Yujun Lin, Zhijian Liu, Yao Lu, and Song Han. Lserve: Efficient long-sequence IIm serving with unified sparse attention. In Sys, 2025. 3,7
[45] Lida Chen, Dong Xu, Chenxin An, Xintao Wang. Yikai Zhang, Jiangjie Chen, Zujie Liang, Feng Wei, Jiaqing Liang. Yanghua Xiao, et al. Powerattention: Exponentially scaling of receptive fields for effective sparse attention. arXiv preprint arXiv:2503.08 035888 2025. 3,8
[46] Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. Ducattention: Efficient long-context Ilm inference with retrieval and streaming heads. In ICLR, 2025.
[47] Peiyuan Zhang, Yongqi Chen, Runlong Su, Hangliang Ding, lon Stoica, Zhenghong Liu, and Hao Zhang. Fast video generation with sliding tile attention. arXiv preprint (xii:2552.005500 2025. 3, 4,7, 7, 10, 24, 26
[48] Yifei Xia. Suhan Ling. Fangcheng Fu, Yujie Wang. Huixia Li, Xuefeng Xiao, and Bin Cui. Training free and adaptive sparse attention for efficient long video generation. arXiv preprint arXiv:2502.21073 2025.
[49] Xin Tan, Yuetao Chen, Yimin Jiang, Xing Chen, Kun Yan, Nan Duan, Yibo Zhu, Daxin Jiang, and Hong Xu. Dsv: Exploiting dynamic sparsity to accelerate large-scale video dit training. arXiv preprint arXiv:2502.07590. 2025.
[50] Min Zhao, Guande Hc, Yixiao Chen, Hongzhou Zhu, Chongxuan Li, and Jun Zhu. Riflex: A free lunch for length extrapolation in video diffusion transformers. In ICML, 2025. 3, 7,8
[51] Jianlin Su, Murtadha Ahmed, Yu La, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 2024. 3,
[52] Karan Dalal, Daniel Koceja, Gashon Hussein, Jiarui Xu, Yue Zhao, Youjin Song, Shihao Han, Ka Chun Cheung, Jan Kautz, Carlos Guestrin, et al. One-minute video generation with test-time training. arXiv preprint arXiv:2504.0522 2025. 3
[53] Lymin Zhang and Maneesh Agrawala. Packing input frame context in next-frame prediction models for vidco generation. arXiv preprint 2504 12626, 2025. 3
[54] Songhua Liu, Weihao Yu, Zhenxiong Tan, and Xinchao Wang. Linfusion: 1 gpu, minute, 16k image. arXiv preprint arXiv:2409.02099 020977 2024. 3
[55] Dongchen Han, Xuran Pan. Yizeng Han, Shiji Song, and Gao Huang. Flatten transformer: Vision transformer using focused linear attention. In ICCV, 2023.
13

---

[56] Songhua Liu, Zhenxiong Tan, and Xinchao Wang. Clear: Conv-like linearization revs pre-trained diffusion transformers up. arXiv preprint 1rXiv:2412.1611,, 2024. 3
[57] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are runs: Fast autoregressive transformers with linear attention, In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 5156-5165. PMLR, 2020.
[58] Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. In Forty-first International Conference on Machine Learning, 2024. 3
[59] Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In Forty-first International Conference on Machine Learning, 2024. 3
[60] Han Cai, Junyan Li, Muyan Hu, Chuang Gan, and Song Han. Efficientvit: Lightweight multi-scale attention for high-resolution dense prediction. In ICCV, 2023. 3
[61] Nikita Kitacv, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Learning Representations, 2020. 3
[62] Zhenhai Zhu and Radu Soricut. H-transformer-Idd Id: Fast one-dimensional hicrarchical attention for sequences, 2021. 3
[63] Zhanpeng Zeng, Sourav Pal, Jeffery Kline, Glenn M Fung, and Vikas Singh. Multi resolution analysis (mra) for approximate self-attention, 2022. 4
[64] Yanming Kang, Giang Tran, and Hans De Sterck. Fast multipole attention: A divide-and-come and-conquer attention mechanism long sequences. 2024.
[65] Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in neural information processing systems, 32, 2019.
[66] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficintt exact attention with IO-awareness. In NeurlPS, 2022. 4,7
[67] Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. ICLR, 2024.
4, 8, 9
[68] Muyang Li, Ji Lin, Chenlin Meng. Stefano Ermon, Song Han, and Jun-Yan Zhu. Efficient spatially sparse inference for conditional gans and diffusion models. NeurlPS, 2022. 7
[69] Jiazheng Xu. Yu Huang. Jiale Cheng, Yuamming Yang, Jiajun Xu. Yuan Wang. Wenbo Duan, Shen Yang. Qunlin Jin, Shurun Li, et al. Visionreward: Fine-grained nuuli-dimmnnionall human preference learning for image and video generation. arXiv preprint 2rXiv:24/2.20002 2024. 7, 25
[70] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable
effectiveness of deep features as perceptual metric. In CVPR, 2018. 7
[71] Dian Zheng, Ziqi Huang, Hongbo Liu, Kai Zou, Yinan He, Fan Zhang, Yuanhan Zhang, Jingwen He, Wei-Shi Zheng, Yu Qiao, and Ziwei Liu. Vbench-2.0: Advancing video generation benchmark suite for intrinsic faithfulness. arXiv preprint arXiv:231 17982, 2025. 7
[72] Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang. Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, and Luis Ceze. Flashinfer: Efficient and customizable attention engine for Ilm inference serving. In MLSys, 2025. 8
[73] Junxian Guo, Haotian Tang, Shang Yang, Zhckai Zhang, Zhijian Liu, and Song Han. Block Sparse Attention, https: //ggthub..ttt/// com/mic t- han lab/Block Sparse- Attention, 2024.
[74] Kepan Nan, Rui Xie, Penghao Zhou, Tichan Fan, Zhenheng Yang, Zhijie Chen, Xiang Li, Jian Yang, and Ying Tai. Openvid- Im: A large-scale high-quality dataset for text-to-vidco generation. In ICLR, 2025. 8
[75] Jingyang Yuan, Huazmo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, YX Wei, Lean Wang. Zhiping Xiao, et al. Native sparse attention: Hardware- aligned and natively trainable sparse attention. arXiv preprini arXiv:2502.1100 11089, 2025. 10
[76] Enzhe Lu, Zhejun Jiang. Jingyuan Liu, Yulun Du. Tao Jiang, Chao Hong. Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, et al. Moba: Mixture of block attention for long-context 1lss. arXiv preprint arKiv:2502.13 13189, 2025, 10
14

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See abstract and introduction. We clearly state what we do in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section 6.
15

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: See Section 4 and Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
See Section 5.1 and Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: See Section 5.1 and Appendix B. Our code is released at https:/github..oom/iii walaaaaaaallaattnnniiiii
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: /ttpsscccppbblcc////eeccppcpppcp hwwbbliccguiieee guwdee/ /CodeSubmis ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
17

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Section 5.1 and Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We do not report error bars as each experiment resource-intensive. All reported results are from single runs, but reproducible.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-simma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See Section 5.1 and Appendix B.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
18

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttpsr////eeu../eeeeee cttppp.ccc..ppoooooooooo.ooc httppppullclciis/i/iiae ines?
Answer: [Yes]
Justification:
We read the NeurIPS Code of Ethics. Our paper does not have an ethics issue.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Appendix E.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [Yes]
Justification: We will explicitly specify the usage permission of our code and models with proper licenses.
Guidelines:
19

---

A comprehensive README will be provided when release code and The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. www.coo///aaa.ae com/datasete has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pane se-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
models.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
20

---

Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this paper does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non- non-standard components.
Please refer to our L.M policy (https: bettps//neurrps //////nurs.../// cc/Confferencess22225 for what should or should be described.
21

---

A Derivations and Proofs
A.1 Complexity
In this section, we provide the detailed complexity derivation in Equation 5, which scales as On log n). Since the computational cost of masked attention is proportional to the number of zeros in the attention mask M, we only need derive an O(n log n) upper bound for the latter.
Central battenttnnioon sink. Firstly, recall from Figure 5(a) that we apply dense attention on these frame-to-frame attention blocks within the central band and attention sink. The attention sink refers to the pattern that every token attends to all tokens in the first frame. Using the same notation in Section 4, where is the total number of tokens, 8 is the number of tokens per frame, and f is the number of frames (so n = fs), we define the attention mask for this region as (1) E { (-00,,).xxxxx 0x/xxxxxx
M L..k
0, -00.
ii =
(8)
otherwise
k.j.k indicates whether the k-th token frame allowed to attend to the I-th token in frame j. with denoting allowed attention and -oo indicating disallowed attention. Since the attention sink spans I blocks and the central band includes at most 3/ blocks, the total number of nonzero entries in this region is bounded by:
Here, (1
#zeros in ( 54 fs
Bands with diagonal width 2 1. The second part is those bands with diagonal width 2 1, except the central band. The mask for this region can be defined as (2)) {-00, 0f/xfxxxs;
00 if 2 max((i-5l.1) S and IK 1| + 1 Uooxa
M (2)
M. -00. otherwise
(10)
Thus, since there are at most [log2 sJ bands in this region, the number of zeros in these bands is bounded by:
logg
#zeros in (2)
21+1
sm
2/2"
(11)
area bound For band ir compute density bound hand dor
logg #]
2
(12)
2"
2 [log2 8].
(13)
Bands with diagonal width < 1. The last part is those bands with 21 loss max((t-5111) 1, where reduce the frequency of diagonals. The mask for this region (3) E ( (0x,,,0/xfxxe
is given by:
(3)
0. (j.ke -00.
1 otherwise
li - jl mod glluwg max(lf s1.11
0 and ks = l
(14)
Since there are most (fog, fl - 1) (llog2 s| + 1) bands satisfying this condition, we have the number of zeros in these bands bounded by:
Dogg fl 11
#zeros in (3)
Σ
Dogg #1+1
Y2
(15)
r= logg x]
number of dingonals bound cach dingonal
< (lloga fl
logg 4)
(16)
22

---

Combining Equation 9, Equation 13, and Equation 16 together, we have the aggregate upper bound of the number of zeros in Radial Attention's mask:
of zeros in M < J. [log2 f] S 4s .n log2 8),
(17)
which scales (n log with the number of frames f for long video generation.
A.2 Error Bound
The design of Radial Attention is inspired by the spatial-temporal structure in video. In this section, we formulate this intuition by theoretically bounding the asymptotic approximation error of Radial Attention with respect to the decay speed of the attention value in the spatial and temporal dimensions.
We focus on bounding the approximation error of single row of the attention matrix. We fix reference query token at position ko of frame in, and write the unnormalized row of the attention matrix as
aj exp(Q K {
where Quy refers to the query vector at position kg in frame in. Kjo+t refers to the key vector at position in frame j. and refers the number of tokens per frame.
Assumptions
(A1) Relative exponential decay. To capture the intuition that the closer frames have stronger correlation and each token typically attends tokens in other frames similar spatial positions, we assume there exist Crel > 0 and (a, 3) >> O such that
0 S ajJ J- Blt-ki do: Ag == Oig.
where a characterizes the temporal decay rate and characterizes the spatial decay rate. (A2) Infinite temporal grid finite spatial grid. To conduct asymptotic analysis, we let E Z (temporal) but keep / E (1....,8) (spatial). Extending to Z only enlarges the sums we bound.
Notation
7
Zreep
Σ
Zout = Z Z
JEZ
(,)) M4
Exact and masked softmax rows: Py aj./Z, РJJ1 aرز o) Zuoop. The total variation error be calculated as follows by standard algebraic argument,
Zout
p p|1 2
(1)
Z
Because ag itself is in the sum, ag. Hence
Zout
Zout
<
(2)
0g
Mask geometry For a temporal offset At : |j tol 2 0, define the bandwidth
w At)
s7.
2 loga At.1)
The mask keeps a spatial index iff Kol S w(At) and the frame one of the sub-sampled frames;
otherwise M -o.
Two kinds of approximation errors, therefore, appear:
(i) Spatial tails inside kept frames
23

---

For each At, the discarded spatial part satisfies
Bd ((m()))+1))
Σ
T-
d>w(An)
Because w w(Ai) 2 3/2 when Al 8..
40 (3+11)
T a0 Σ c-a
e-d
(3)
(1 (1-E (1-
d> (At)))
(ii) Frames skipped by the subsampling rule
For At s, only every ((At)= = logg arl 2t//s) frame is kept; the remainder contributes
1 + c T2 20n
2Crel (1 + c) ca(s+1)
c-
I
e At>a
(1 e e1-e-))(1 eaa)
Total variation error
Combine all equations above:
Se 0 (++1)
1 e c-((+))
|- pll & Crel
+
OCCree -iin//a-mi(()/2aat
e1- (1-
1 e a
This characterizes how the decay rates affect the approximation error.
B Additional Implementation Details
In terms our LoRA fine-tuning for longer-video generation, we fine-tune Hunyuan Video [1] and Mochi [22] at global batch size of I with sequence parallelism, and train Wan 2.1 [7] with a global batch size of 8. All tuning experiments are conducted on H100 GPUs. During training, we keep the first two DiT blocks with dense attention. Since there are 60, 48, and 40 blocks for Hunyuan Video, Mochi 1, and 1an2.1-14B, this only incurs negligible overhead. We train Hunyuan Video for 2x and 4x length video generation for 2400 and 1200 steps, respectively. We train Mochi for 5000 steps for both 2x and length video generation. We train Wan2. for 2500 steps for 2x length video generation. The LoRA rank is 128 for all training tasks.
C Visualization of the generated videos
In this section we compare Radial Attention against various baselines in video quality, and list our speedup in both training and inference.
C.1 Default Video Length
We provide visual comparison between the original dense attention, STA [47], and our Radial
Attention on Hunyuan Video [1] and Wan2.1-14B [7]. We conduct experiments under 768p. 117 frames settings for Hunyuan Video, and 768p, 69 frames settings for Wan2.1-14BB As shown in Figure and Figure B, Radial Attention has higher PSNR compared STA [47], effectively maintaining the high fidelity of the original videos.
C.2 Longer-video Length
We provide visual comparison between the aforementioned baselines and Radial Attention on Hunyuan Video [1], Mochi [22], and Wan2.1-14B [7]. We conduct experiments under the default resolution settings, which are 720p for Hunyuan Video and Wan2.1-144 and 480p for Mochi Moreover, generate videos at 4x longer length for Hunyuan Video (21 seconds, 509 frames) and

---

Table A: We compare Radial Attention against another (n log n)
attention baseline, Harmonic Series (HSS).
Radial Attention consistently outperforms across metrics.
Model
Method
PSNR (T)
SSIM (T)
LPIPS (4)
VisionReward (T)
Hunyuan Video (117 frames)
IIS
27.0
0.881
0.119
0.136
Table B: Ablation the number of initial full-attention (warmup) steps for default-length video generation of Wan2.1-14B model. The 12-step warmuup achieves best performance across all metrics.
Model
#Warmup Steps
PSNR (t)
SSIM (1)
LPIPS (L)
12.8
0.486
0.522
18.5
0.267
Wan2. frames)
21.7
0.778
0.183
23.2
0.813
0.151
12 (Ours)
23.6
0.823
0,146
23.5
0.819
0.150
Mochi 1(22 seconds, 667 frames), and 2x longer length for Wan2.1-14B (10 seconds, 161 frames). We use Vision Reward [69] to evaluate the generated videos. Figure C, Figure D, and Figure E demonstrate that Radial Attention achieves the highest average Vision Reward score compared to the baselines, well preserving the video quality even at longer-video settings.
C.3 LoRA Compatibility Visual Results
As illustrated Figure combining our extended-length LoRA with existing style LoRAs preserves visual quality while enabling longer-video generation. We observe that the content style generated by the merged LoRA exhibits subtle differences from the original LoRAs. This discrepancy is primarily attributed to the relatively small dataset used for training the extended-length LoRA, which may introduce a slight style bias that interacts with the style LoRA. We expect that training the mngthhextension LoRA on more comprehensive dataset would help mitigate this issue.
D Ablations on Initial Dense-Attention Layers and Steps
We provide additional ablation studies justify our design choices, including how many dense- attention timesteps and blocks we use to deliver the best video quality with the same computation budget, as well as the design of our On log n) attention mask.
D.1 Ablation on Initial Dense-Attention Steps
For default-length video generation, we follow SVG [8] to apply full attention to the first 25% of timesteps (12 steps) as warmup for all methods. We ablate this setting in Table B. For fair comparison, we match the overall computation of all settings by adjusting the sparsity of our Radial Attention mask. The 12-step warmup achieves the best video quality across all metrics.
For 4x longer video generation, we apply full attention to the first 2 steps as a warmup during inference. The impact of different warmup steps Hunyuan Video is shown in Table C. Computation matched across configurations. Using warmup steps achieves the highest Vision Reward.
D.2 Ablation on Initial Dense-Attention Layers
To better capture global information, we keep the first two layers as full attention during training. Table reports results on Hunyuan Video when using 0, 1, 2, or 3 dense layers, under the same computation budget. Our choice of using 2 full-attention layers delivers the best video quality.
25

---

Table C: Ablation on the number of warmup steps for 4x longer video generation. Two warmup steps yield the best Vision Reward.
Table D: Ablation the number of initial full-
attention (dense) layers during training. Using two cull-attention layers yields the highest Vision Reward.
Model
#Warmup Steps
Vision Reward (f)
Model
#Dnse Layers
Vision Reward (t)
0,154
0.139
0.160
0.156
Hunyuan Video (117 Trames)
Hunyuan Video (117 Irames)
(Ours)
0,163
2 (Ours)
0.163
0.157
0.157
D.3 Comparison with Other On log n) Sparsity Patterns
We additionally conduct an ablation study to validate the effectiveness of the sparsity pattern in our proposed On log n) attention mask. Specifically, compare Radial Attention with the Harmonic Series Decay Attention (HS), which features a computed diagonal width inversely proportional to its distance from the main diagonal. Table A presents quantitative results comparing Radial Attention with HS, demonstrating the superiority of Radial Attention.
E Broader Impacts
Radial Attention significantly reduces computational costs for video diffusion models, enabling longer-video generation with minimal fine-tuning efforts while maintaining quality. This paves the way for high-quality video creation tools for education and creative arts. Since Radial Attention accelerates self-attention to Oín log n.) complexity, it can accelerate video diffusion models and decrease energy consumption, leading to greener Al applications. This also helps the popularization of generative models. However, malicious users can misuse our method to create deepfakes and spread misinformation. The technology may also exacerbate the digital divide between those with and without access the minimal necessary computational resources. To address these concerns, we advocate for responsible deployment, adherence to ethical standards, and the development of effective detection methods. We encourage the research community continue advancing both efficient generation techniques and safeguards to ensure these powerful tools benefit society while minimizing potential harms. We will explicitly specify the usage permission of our code and models with proper licenses.
License
Here, we show all the licenses for used assets. Wan 2.1 [7], Mochi 1 [22], Diffusers, and STA [47] are under Apache-2.0 license. The license of Hunyuan Video [1] is here. SVG [8] and OpenVid-IN, do have explicit license.
26

---

Prompt: shark swinming the ocean, featuring steady and perspective. Realistic. Natural lighting. Mysterious.
Original HunyaanVides meney: Specdup: 1.0x
STA(FA3) PSNR; 29.8
PFLOPs: 331 Larency: 719
Speodup: 2
Radial Atention (Ours)
PFLON: 339 Lavency: 87ts Speedup: 1.9x
Prompt: Martial artists exchanging fluid, powerful strikes in ancient temple courtyard. dust clouds rising in slow motion from every footfall and impact.
Original Video Lateney: Specdup: 1.0x
STA(TA3) PSNR: 23.6
331 Latency: 71% Speedug: 2.3x
Radial Attenion (Ours)
PSNK: 26.1
PFLOPs: 339 Latency; 8766s Speodup:
Prompt: couple in formal evening walks home and gets cought in heavy downpour with umbrellax, surreulism style. Night lighting, Mysterious.
Original HunyumVidee Video PFLOPs: Latency; Speedagy
PSIR: 24.2
PFL:PP: 331 T Farency: 719A
Speedup: 2
(Ours)
PSNR: 25.5
PELOPS: Lateney: 876s Speadup:
Prompt: dancer spinning with explosive energy under sharp spotlight, loose fabric fine dust swirling around her whirhwind motion and emotion.
Original HunyaanVideo PFLOPs: Lateney: 1649s Speedup: 1,0k
STA(FAS) PSNR: 27.2
PELOPs: Latency: Speedup: 2.3x
Radial Attention (Ours)
PSNR; 28.8
PFLOPs: Latency: 876s
Spoedup: 1.9x
Figure A: Comparison of Dense Attention and Radial Attention on Hunyuan Video Text-to- Video generation at the default length (5 seconds, 117 frames, 768p).
27

---

Prompt: A bus stuck in traffic during rush hour. Realistic, Natural lighting Tense.
Original Wan2.1-14B PFLOPs: 560 Latency; 1630s Specdup: 1.0%
STAPA33 PSNR: 19.4
Latency: Spcodupp 2.0
Radial Attemtion (Oun) 22.2
PFLOPs: 323 Latency; 917s Speedup: L88
Prompt: solitary figure stands on windswept cliff. their silhouett framed by dramatic sunset,
wearing a long, flowing coat that billows in the breeze.
Original Wan2.1-148 PFLON: 560 Latency:
Speedup: 1.0s
STAFFA) PSNK: 21.8
PFLOPs: 322
812s
Speodup:
Radial Attesrion (Ouns)
PSNR: 23.6
Latency: 9174 Speedup: 1.8x
Prompt: breathtaking coastal beach spring, with gentle lapping against the golden sand. is depicted vibrant. swirling brushstrokes Van Gogh.
Original Wanz. 1-14B OPc: 560 Lateney: 1630s Speedup: 1,0%
STA(FA3) PSNR: 19.5
PELOPs: 322 Latency: 8:12s
Spoodupp: 3.0x
Radial Attention (Ouns PSNR: 22,1
PFLOPs 323 Latency: 917s
Speedup: 1.8
Prompt: teddy bear is swimming in the Realistic, Natural lighting, Mysterious
Original Wan2.1-148
PFLOP:: 560 Laiency: 1630s Speedup: 1.0%
STA(FA3) PSNR: 18.5
PFLOPss: iteeny:
Speedup: 2.0 0x
Radial Amention (Ours)
PSNR:
PFLOPs: Latency: 9176
Speedug: 1.5-
Figure B: Comparison of Dense Attention and Radial Attention 1-12.1-14B Text-to-Video generation at the default length (4 seconds, 69 frames, 768p).
28

---

Prongt: gentle carions pundl, soft, Haatt large round eyes, degicttdd charating wutereodor puinting: pandia
copy zable is quaint cafe located in the heart of Paris.
Oliginal HansanViden, Visienllenand 8.065, Laiases: 2595,, lnferesse Spoodap, L0k, Traiting Time: GPL Hous
HengaanWeee-RiPLLe VisionRewant 8,55, Latenyy: 2595,, Inference Speedag: Ldt,, Traising Time: DGP. Hours
Ihmnson/Vidooili http://wwwokkkkopiiiiiaalaaaao (Spriallll Victokkeaant
larreey
Infermos Spoodop:
Triining GPC Triining Speckhgr
Serpuu/VMeettRRATTppeeee Head, Vistonkewandd 0.t01, Latency: 774,, Infercnce Speedap: 3.7x, Triiting Time: IPR Hours, Training Speedop: 4.4x
flnnyannViooo LomglaaA,,,,, Vicunitwadd: 0. Lateayy A014, Infermmet Spoodap: J.s., Triming Time: 188 GPC Hluu,, Trinnnng Specdap:: 4.5-
Daryun' Silon LLoAA PhwarAntmiiu, VictnRewand 1.11 1666, Inkonono Speodap Ro, Toiiii Time: 114 GPU Iluun, T'iining Specokg:
HanyyanVidarrr Lobt--Densc
Te0
HungrannVico-LL.o
Latery: Speeda::
Traning
Hours, Traning Speeday:
Prompt: A pichoresque coastal beach the enchanning spring scason, wieerz gentie water lap riystmmically against reft sondy shore, srene capturer the beauy of nature during this vubront time afyear
Odigiaal HurouaViden, VisioRRevad Latency:
Speedap:
MonyanNiito-RFLLEs VisioReeaard 8,08,, Latency: 2895s Inference Speedup: 100s, Traiting Time: GPU Hours
Head, VisonRevasi. 0,191, Latewcy. 755k, Infoeasz Speedap. 3.84, Traning Tine: 160 GPL Hour,, Trainnng Speeday; 4.5
HanyyaaVislo LaRA ToReere Head, Viskzsltewand 0:201, Lasanyy: 774,, Infocnce Specoke: 3.7., Training Time: CPU Hlons, Teaning Spealap: 4,4+
HuwyaanVideott Visisclleward Latescy; 903s, Speedup: Trainits Tine; Hous, Speodag,
Huryuat PowerrAtentin,,
Latexcy
Spentup Training
Training Speedug:
lateray Speciig: Trining
Toining Spockgr
HunyaanVidza LaRA Ralii Atzetio (Oun)) Visunflevani 6,191, Lakeny: 28 Infoenze Specike:: 3.7,, Traning Time: 1?l GPU Hours, Traning Specdap: 4.4
Figure C: Comparison of all baselines and Radial Attention at 4x default length (21 seconds, 509 frames) Text-to-Videe video generation from Humyuan Video. Radial Attention achieves the best Vision Reward score with good visual quality and consistency. In contrast, Original Hunyuan Video and RIFL generate blurred videos with poor visual quality. Temporal Head generates distorting figures. Spatial Head, Long LoRA, and PowerAttention generate temporally inconsistent video backgrounds. Dense Attention generates dynamic videos.
29

---

Pronpe: staple yet detuiled scone set a caay thing room. a person, tikely wildli aged momso uith gestle featurns and searly voyled wher huir, vit coneoortalyy on phooh svoe fuii.
Original Modii VitonRewadd 9044, Lateway 9925, Inforase Spoodag: LOs, Trining Tine: CPU Heuzs
Lasaay:
Speodap
Triniing Spookg:
I-LoRA-TTnppeall VidonRewerd .0604,
255
2,84
Meei L-LougL.oRA, ViiouReward 6.037, Lotecoy: 42ts, Inforoee Speodas; 23H, Trciting Time: 152 Hour,, Toasing Speedap: 2
Mochi I-LoRA-Powce PowserAisssoo ViskesRewaad: 1000, Lataney: 3811, inforese Scoodap: 2.6%, Transng Tirse: 138 GPU Hours Traisita Specoka: 2.8x
Machi I-LoRA-Desse Ancetior, ViscoReward 0:130, Lotency: 992s, Infieenc Spooday: Lox, Trirring Time: 30a GPU Hous, Training Spealop: 100-
Machi 1: LatA Radial Amurton (Durs), Vasrullewnnt a 1st, Laimny- 1okc, Inforrezz Speedap 746,, Touiiig Time: Ieee CPU Elour, Trining Spadig:2k
Prompt: bevadraking cousta beach the vibmant spring mommin unner gondly top the goldes vondy showes. In bluck and white, the some the saroms beandy maiure. lone Rgure is stylis beige winibresker stralls along sdge the mater, casting ovcanionai glances towarda Seagull overicad, silooeetss greenery of hearby Spendop: Toinung
dunes rise blending
Orginal
004 Laaeay
Modii I-LoRA-Sputial Head, ValesRevadd 0.045, Lalancy: 342, Inference Speedup 26% Traiiing Time. 139 GPU Hous, Traiiigg Speaku;; 28c
Mochi (-LoRA-Tenpeal Head, VisonkRevadd 0,00,, Lateno:: 393s, Infoenze Sponlap: 25. Triiting Timz: 141 GPU Hozz, Teannn Specdap: 24H
I+Long Latercy: Spodky:: Trinieg
Towning Speetop
Mochi F-ookePPoeeetee ViskoRRwand 0.007, Lawway 381s, Infe Speedap. 2 Trainin Timee: 134 GPU Rours, Traiiing Speedky: 2,5x
Tiene Attention, VidunRroond 585,, latreey 983s Infennos Spoodap: L0,, Thiring Time llars, Toening Speadop Lifx
LattA Anantion (Dam), Vaisallovant Laimay: 3964, Inforencs Spendup Immnn Haun, Triting Speaker
Figure D: Comparison of all baselines and Radial Attention 4x default length (22 seconds, 667 frames) Text-to- Pect-to-Vie Video video generation from Mochi Radial Attention achieves highest Vision Reward score because it has excellent visual quality and consistency. In contrast, Original Mochi generates blurred videos with poor visual quality. Spatial Head, Temporal Head, Long LoRA, PowerAttention, and Dense Attention generate videos with either inconsistent hackgrounds or inconsistent figures.
30

---

Pronpt: A medtum sized golden reiiener dog is siming poacefully in suntt bachyard, its till wagging gendy, Suddonó) it spring up and saarss ciroles, snil woggiigg excitelly and cars fopping The lush and green. wildhowers scartered aroond.
Original Wa12. ViconRewadd: di Lateryy 5733s, Inference Speadap: 10s, Tranning Time: o GPU Hours
WanD: I: LB--LeDADDDs Aftention, ViskanBevard 0.094, Latencr: 57356, ease Speeday: 1.k,, Traiiing Time: 224 GPU Hour, Takiis Spoodap: 1.8
Lateay: Spoatip
Triinig
Tovenn Spacieg
Pronpt Aright orange cornut black amebvolla. Realiatic Reight lighting Conuol
Origiaa Wont.I-i4lh, Vidrnkeeoodd 14i, Laiezyy 9TM4, Infermer Spacdape Lis Trining Time: ó GPU liure
Wai2.-144 Lee--Lo-----e ViniuRewad 5735%,
Ward: 1-148t-1 aftA Radial Attentkor (Curg), Visiwikoonn 141, strney 76447, Informnce Spordge dt. Thomiigg Thmer (PPU Hoam, Training Speodep
Prompt: visinge buyyle olong tree-lined wearing
aneakery, some coptores the golám bour, wilh nanlight filtering throough leane,, rasing duppled shodons on ground. The fious freely in the freeze, and jogfil mule lights up their fice.
Original Wark 1-14,, Visienltesank 0.165, Laxncy: 5735,, Infoense Specdap: LDd, Trining Tine: GPU Bours
Spaalope
Spectae:
Anentior (Our)) VistoRkroonh Latenyy Spocchge 2.0r, Tooming Time 116 Eloun, Training Speedep:
Prumpt: A solitury figure stunds windswept cliff. their silhouctie framed by drumatic wearing Bowing cont that billows in the breeze. The sky is ablaze with bues orange. pink, and purple, casting warm glaws the scene.
Orgiial War2. 148, VisouRewad. 16t, Latesey: STE5,, laference Speedup: L0s, Taiing Time: 0 GPU Hour
Wanc: 1:4B-L..AA-Deae Alteniion Visketllevrd 0.130, Laiany: $735s, Infeesse Specda:: 1.0s, Traitig Time: 224 GPU Houn. Tratsise Spcodap, 1,0x
Wan2. 1-14B -Lakk---aaa Amention (Oun), Vinumleewan 0.161, Latency: 28474, Infermnce Speadap: 2.0% Triniig Time: 116 GPU Houns Toannu Spondap:: 1.
Figure E: Comparison of all baselines and Radial Attention at 2x default length (10 seconds, 161 frames) Text-to-Vidd Video video generation from Wan2. 1- 14B. Radial Attention achieves the highest Vision Reward score original Wan2. 14B generates blurred videos and Dense Attention generates videos with inconsistent figures.
Desi Altakin 122 (manciter
Dznc
Jamo
"Raial Alientne LallA (Damk. franca Vitte Roward 1PP
Voice Hovasti &
Yinke lovand tugs
Proupt Agsinuuu fookilggga aniienn Aroasks Wiking stansl prominently 66 sppliittnna wlink aveglaanse whlig moders edge serticaaa apgearance fire miinng Mort fursur sinias centt. chorasterzzooo intesse oas Aetiowtkee strd leves ond dbese frrser do strong fratvies
Figure F: Radial Attention LoRA is compatible with existing style LoRAs. On Hunyuan Video, it extends vidco length by while maintaining vision reward comparable that the original- length LoRA video.
31


Reviews:
[{"review_id": "3pqCFP2U4P", "reviewer": "Reviewer_E5WB", "summary": "This paper introduces Aura Attention, a sparse attention mechanism designed to mitigate the quadratic computational cost of generating long videos with diffusion models. The central motivation is a phenomenon the authors call \"Spatiotemporal Energy Decay,\" where attention scores are observed to diminish with increasing distance between tokens in both space and time. Aura Attention implements a static attention mask that mimics this decay by exponentially shrinking the attention window as temporal distance grows, achieving a reported complexity of O(nlogn). This allows pre-trained models to be adapted for longer video generation through efficient LoRA fine-tuning. The authors test their method on several large video models (Hunyuan Video, Wan2.1-14B, Mochi 1) and report substantial improvements in inference speed and training cost while preserving video quality.", "questions": "Please read the weakness of the manuscript provided above.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1.\tThe paper's primary strength lies in its empirical results. The method delivers significant, practical speedups (e.g., up to 3.7x in inference for 4x longer videos) while maintaining video quality across multiple SOTA models. This is a non-trivial engineering achievement.\n2.\tThe concept of \"Spatiotemporal Energy Decay\" is an intuitive and appealing motivation for designing a sparsity pattern. It provides a clear, high-level story for why the proposed attention mask might be effective.\n\nWeakness:\n1.\tThe paper’s scientific foundation rests on the \"Spatiotemporal Energy Decay\" phenomenon, which is modeled as an exponential decay. While this is a compelling narrative, the evidence provided is primarily qualitative, based on visual inspection of the plots in Figure3. For a method presented as \"principled,\" the lack of a rigorous quantitative analysis to validate this core assumption is a significant weakness. This makes the approach feel more like a well-motivated heuristic than a scientifically-validated model.\n2.\tThe paper omits critical ablation studies that would justify its specific implementation. Key decisions feel arbitrary and are not supported by experiments: The model retains full attention for the first few layers and initial denoising timesteps, which is noted as being \"crucial for video quality\". However, there is no analysis of how this parameter was chosen or its impact on the performance/speed trade-off.", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful and constructive reviews. Below, we provide point-by-point responses to each of your comments.\n\n### Quantitative Analysis on \"Spatiotemporal Energy Decay\"\nTo validate the exponential decay modeling of our proposed \"Spatiotemporal Energy Decay\" phenomenon, we conducted a quantitative regression analysis on the average attention score distribution of Wan2.1 14B using 10 diverse prompts from VBench [1]. Specifically, we fit three functions to the distribution: \n- the inverse model $y = \\frac{a}{(x-b)} + c$,\n- the inverse square model $y = \\frac{a}{(x-b)^2} + c$,\n- our proposed exponential decay model $y = \\exp(-ax + b) + c$. \n\nThe regression performance, measured by $R^2$ (higher is better), is summarized in the table below. Our exponential model achieves the highest $R^2$ values in both spatial decay and temporal decay, up to 0.996. This indicates that our proposed decay modeling is an excellent fit to the data.\n\n| Regression Model                   | Spatial Decay $R^2$ ($\\uparrow$)    | Temporal Decay $R^2$ ($\\uparrow$)   |\n|------------------------------------|-------------|--------------|\n| $y = \\frac{a}{(x-b)} + c$          | 0.980       | 0.989        |\n| $y = \\frac{a}{(x-b)^2} + c$        | 0.988       | 0.994        |\n| $y = \\exp(-ax + b) + c$ **(Ours)** | **0.990**   | **0.996**    |\n\n### Ablations on Initial Full-Attention Layers and Steps\nAs requested by the reviewer, we provide additional ablation studies to justify our design choices. We will include them in our manuscript.\n\nFor default-length video generation, we follow SVG [2] to apply full attention to the first 25% of timesteps (12 steps) as a warmup for all methods. We ablate this setting in the table below. For fair comparison, we match the overall computation of all settings by adjusting the sparsity of our Aura Attention mask. The 12-step warmup achieves the best video quality across all metrics.\n\n| #Warmup Steps                | PSNR ($\\uparrow$)  | SSIM ($\\uparrow$) | LPIPS ($\\downarrow$) |\n|-----------------------------|-------|-------|-------|\n| 0                           | 12.8  | 0.486 | 0.522 |\n| 4                           | 18.5  | 0.693 | 0.267 |\n| 8                           | 21.7  | 0.778 | 0.183 |\n| 11                          | 23.2  | 0.813 | 0.151 |\n| **12 (Ours)**               | **23.6**| **0.823**| **0.146**|\n| 13                          | 23.5  | 0.819 | 0.150 |\n\n\nFor 4× longer video generation, we apply full attention to the first 2 steps as a warmup during inference. The table below shows the impact of different warmup steps on HunyuanVideo. Similarly, we match computation of all configurations. Using 2 warmup steps achieves the highest Video Reward.\n\n| #Warmup Steps | Vision Reward ($\\uparrow$) |\n|---------------|----------------------------|\n| 0             | 0.154                      |\n| 1             | 0.160                      |\n| 2 (**Ours**)  | **0.163**                 |\n| 3             | 0.157                      |\n\nTo better capture global information, we keep the first two layers as full attention during training. The table below reports results on HunyuanVideo when using 0, 1, 2, or 3 dense layers, under the same computation budget. Our choice of using 2 full-attention layers delivers the best video quality.\n\n\n| #Dense Layers | Vision Reward ($\\uparrow$) |\n|---------------|----------------------------|\n| 0             | 0.139                      |\n| 1             | 0.156                      |\n| 2 (**Ours**)  | **0.163**                      |\n| 3             | 0.157                      |\n\n\n[1] VBench: Comprehensive Benchmark Suite for Video Generative Models, CVPR 2024\n\n[2] Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity, ICML 2025"}, {"review_id": "ZmTTiVCBLC", "reviewer": "Reviewer_XyA8", "summary": "This paper introduces Aura Attention, a sparse attention mechanism designed for efficient long video generation in diffusion models. The key innovation is inspired by the observation of spatiotemporal energy decay in attention scores, where post-softmax scores decrease with increasing spatial and temporal distance between tokens. Aura Attention translates this decay into exponentially decaying compute density, achieving O(n log n) computational complexity. The method uses a static attention mask that focuses on spatially nearby tokens with a shrinking window size as temporal distance increases. Experiments on models like HunyuanVideo and Wan2.1-14B show up to 1.9× speedup at default lengths and 3.7× speedup for 4× longer videos, while reducing training costs by 4.4× with minimal fine-tuning using LoRA. The approach maintains video quality comparable to full attention baselines.", "questions": "Please refer to the weakness part.", "limitations": "Please refer to the weakness part.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength：\n- The integration of physical energy decay principles into attention mechanism design is innovative, providing a principled way to reduce computation while preserving critical dependencies.\n- Achieves O(n log n) complexity, addressing the quadratic scaling issue of full attention, which is crucial for long video generation.\nEmpirical Validation: Comprehensive experiments across multiple large video models (HunyuanVideo, Wan2.1-14B) demonstrate consistent speedups and quality preservation.\n- Enables longer video generation (up to 4× extension) with minimal fine-tuning, reducing both training costs and inference latency significantly.\n\nWeakness：\n- The complexity still scales quadratically with spatial resolution, which may limit performance for very high-resolution videos.\n- The model relies on a simplified exponential decay function for attention scores, which may not fully capture complex spatiotemporal dependencies in real-world videos.\n- The static mask design might struggle with dynamic scenes requiring long-range temporal dependencies beyond the predefined decay  pattern.\n- Some baselines (e.g., SANA) are not fully optimized, and comparisons may not reflect the latest advancements in linear attention. Discussing more linear attention-based image generation methods here would be better.", "was_revised": true, "final_justification": "Keep my original rating.", "rebuttal": "Thanks for your insightful and constructive reviews. Below, we provide point-by-point responses to each of your comments.\n\n### Complexity with Respect to Spatial Resolution\n\nWhile Aura Attention has a theoretical complexity of $\\mathcal{O}(n^2)$ with respect to spatial resolution, its practical speedup does not degrade as resolution increases for a fixed video length. As discussed in Section 4.2, our mask design guarantees constant sparsity across resolutions for a fixed number of frames. The table below compares the per-step latency of Wan2.1 14B using original full attention versus our Aura Attention for generating 5-second videos. The speedup becomes more significant at higher resolutions, since attention computation scales quadratically with the number of pixels per frame, while most other operations scale linearly.\n\n| Resolution      | Attention Type       | Single-Step Latency (s)     | Speedup |\n|-----------------|----------------------|------------------|--|\n| 720p (1280×720) | Full        | 32.9           | -- |\n|                 | **Our Aura**| **15.4**  | **2.1×** |\n| 1080p (1920×1080)| Full       | 136.6          | -- |\n|                 | **Our Aura**| **53.1**   | **2.6×** |\n| 2K (2160×1440)  | Full        | 437.8          | -- |\n|                 | **Our Aura**| **127.5** | **3.4×** |\n\nMoreover, recent advances in video VAE have achieved remarkable progress in spatial compression, pushing the compression ratio from $8\\times8$ to $16\\times16$ [2] or even $32\\times32$ [3]. These methods can be combined with Aura Attention to mitigate the quadratic cost of spatial resolution.\n\n### Robustness to Complex Spatiotemporal Dependencies\n\nTo evaluate whether our exponential decay model can capture complex spatiotemporal dependencies, we sampled 10 prompts from the LongVILA [4] supervised fine-tuning dataset. These prompts describe complex real-world scenes that demand high spatiotemporal dependencies due to intricate interactions and dynamic changes over time. A sample prompt is displayed below.\n> The woman shows the stickers to the camera, talks about them, places them on the table, and at one point sticks a sticker on her hand.\n\nWe then conducted a quantitative regression analysis on the average attention score distribution of Wan2.1 14B on these prompts. Specifically, we fit three models to the spatial and temporal attention decay curves:\n- inverse model $y = \\frac{a}{(x-b)} + c$\n- inverse square model $y = \\frac{a}{(x-b)^2} + c$\n- our proposed exponential decay model $y = \\exp(-ax + b) + c$. \n\nThe regression results, measured by $R^2$ (higher is better), is summarized in the table below. The attention score distribution nicely fit our exponential decay model, achiving $R^2$ up to 0.992. This means even for complex real-world scenes, our model still holds. See the following response for the video quality.\n\n| Regression Model                   | Spatial Decay $R^2$ ($\\uparrow$)    | Temporal Decay $R^2$ ($\\uparrow$)   |\n|------------------------------------|-------------|--------------|\n| $y = \\frac{a}{(x-b)} + c$          | 0.980       | 0.987        |\n| $y = \\frac{a}{(x-b)^2} + c$        | 0.989       | 0.991        |\n| $y = \\exp(-ax + b) + c$ **(Ours)** | **0.989**   | **0.992**    |\n\n\n### Capability of Capturing Long-Range Dependencies\n\nAs mentioned in Section 5.1 in our paper, we preserve full attention in the first two layers during both training and inference, and apply a 2-step full-attention warmup during inference. This setup helps the model effectively capture global information. Additionally, stacking multiple Aura Attention layers expands the receptive field, allowing information to propagate across distant tokens. Therefore, our method is capable of capturing long-range dependencies even in dynamic scenes.\n\nTo support this, we further evaluate default-length video generation on Wan2.1 14B using the aforementioned complex prompts from LongVILA [4]. As shown in below table, the similarity metrics demonstrate that our approach maintains high video fidelity on both VBench prompts used in our paper and the LongVILA prompts, indicating robust performance across different scenarios.\n\n|Prompt Set| PSNR($\\uparrow$) | SSIM($\\uparrow$) | LPIPS($\\downarrow$) |\n|----------------------|------------------|------------------|---------------------|\n| VBench Prompts            | 23.6             | 0.823            | 0.146               |\n| LongVILA Prompts   | 23.0             | 0.823            | 0.151               |\n\nWe also report the video quality of 4$\\times$ longer video generation on HunyuanVideo using LongVILA prompts in the below table. Aura Attention matches the Vision Reward of full attention, demonstrating strong long-range modeling capabilities.\n\n| Method                  | Vision Reward ($\\uparrow$)|\n|-------------------------|--------------------|\n| Full Attention  | 0.128             |\n| **Aura Attention (Ours)** | **0.128** |\n\n### Additional Linear Attention Baselines\n\nAs suggested by the reviewer, we adapted two more image-based linear attention methods—LinFusion [5] and Focused Linear Attention [6]—for video generation. The table below shows their performance on 4× longer video generation after LoRA fine-tuning on HunyuanVideo, measured by Vision Reward. As mentioned in our paper, due to the substantial architectual discrepency between the linear attention and the original softmax attention, these baselines fail to the learn the weight distributions during the fine-tuning.\n\n\n| Method          | Vision Reward ($\\uparrow$) |\n|-----------------|--------------------|\n| LinFusion       | -0.209             |\n| Focused Linear Attention  | -0.205             |\n| **Aura Attention (Ours)**|**0.134**        |\n\n[1] Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity, ICML 2025\n\n[2] Wan 2.2: Leading AI Video Generation Model\n\n[3] Open-Sora 2.0: Training a Commercial-Level Video Generation Model in \\$200k\n\n[4] LongVILA: Scaling Long-Context Visual Language Models for Long Videos, ICLR 2025\n\n[5] LinFusion: 1 GPU, 1 Minute, 16K Image\n\n[6] FLatten Transformer: Vision Transformer using Focused Linear Attention, ICCV 2023"}, {"review_id": "RwbiqqdFkV", "reviewer": "Reviewer_aPz8", "summary": "This paper introduces Aura Attention, an static attention mask with O(n log n) complexity wrt. number of frames suitable for long video generation. The design of this attention mask is motivated by empirical observations on the attention patterns in HunyuanVideo. Long video generaiton is enabled through LoRA finetuning, since the sparse attention mask reduces the prohibitive cost of training on long videos. Experiments are done on three backbones at the default video length and extended length, showing that Aura Attention speeds up both training and inference while achieving good quality compared to baselines.", "questions": "- Figure 5 shows that Aura Attention enables better video quality for long video generation finetuning. Can the authors give some conceptual or theoretical analysis behind this?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "## Strengths\n- Good writing quality. \n- Strong experiment results on long video generation in tab. 2. A wide range of baselines are adopted in tab. 1, 2. \n- The design of Aura Attention mask is intuitive, innovative, and motivated through empirical observations shown in fig. 3.\n- Complexity and error bound for Aura Attention are provided.\n- Aura Attention enables long video generation finetuning with modest training cost, which is a core challenge for long video generation.\n\n## Weaknesses\n- The complexity of Aura Attention, while $\\mathcal{O}(n \\log{n})$ wrt. number of frames, still have $\\mathcal{O}(n^2)$ wrt. the spatial resolution, which is shown in Eq. 6 and discussed in Sec. 6 as a limitation. This limits the main application of Aura Attention to long video generation, while existing methods (STA, SVG) are either more efficient or have better quality in the short video setting, as shown in Tab. 1.\n- No ablation studies are done. Possible ablation studies include the usage of attention sink, the width of the central band, the width scale between consecutive bands (currently 2x), whether to reduce the frequency of diagonals if the diagonal width falls below 1, etc. Conducting some ablation studies can provide insights into the design space of Aura Attention.", "was_revised": true, "final_justification": "My concerns are addressed in the rebuttal. I maintain my rating of accept, since the paper introduces a novel attention pattern for video generation, demonstrate strong experiment results, and should have a high impact in the field.", "rebuttal": "Thanks for your insightful and constructive reviews. Below, we provide point-by-point responses to each of your comments.\n\n### Complexity with Respect to Spatial Resolution\n\nWhile Aura Attention has a theoretical complexity of $\\mathcal{O}(n^2)$ with respect to spatial resolution, its practical speedup does not degrade as resolution increases for a fixed video length. As discussed in Section 4.2, our mask design guarantees constant sparsity across resolutions for a fixed number of frames. The table below compares the per-step latency of Wan2.1 14B using original full attention versus our Aura Attention for generating 5-second videos. The speedup becomes more significant at higher resolutions, since attention computation scales quadratically with the number of pixels per frame, while most other operations scale linearly.\n\n| Resolution      | Attention Type       | Single-Step Latency (s)     | Speedup |\n|-----------------|----------------------|------------------|--|\n| 720p (1280×720) | Full        | 32.9           | -- |\n|                 | **Our Aura**| **15.4**  | **2.1×** |\n| 1080p (1920×1080)| Full       | 136.6          | -- |\n|                 | **Our Aura**| **53.1**   | **2.6×** |\n| 2K (2160×1440)  | Full        | 437.8          | -- |\n|                 | **Our Aura**| **127.5** | **3.4×** |\n\nMoreover, recent advances in video VAE have achieved remarkable progress in spatial compression, pushing the compression ratio from $8\\times8$ to $16\\times16$ [2] or even $32\\times32$ [3]. These methods can be combined with Aura Attention to mitigate the quadratic cost of spatial resolution.\n\n### Short Video Generation\n\nWe would like to clarify that in Table 1 Aura Attention achieves video quality comparable to SVG [1] and outperforms STA [4] on Hunyuan Video under the same computation budget. On Wan2.1 14B, Aura Attention beats both baselines in quality. Regarding latency, Aura Attention matches SVG on both models. STA uses FlashAttention-3 (FA3) [5] as its backend, providing slightly higher speedup. Aura Attention can reach similar performance by switching the backend from FA2 to FA3, achieving a speedup of 2.27× on Hunyuan Video and 1.98× on Wan2.1 as shown in the below table, on par with STA.\n\n| Model         | Method         | PSNR (↑) | SSIM (↑) | LPIPS (↓) | Vision Reward (↑) | Latency (s) | Speedup |\n|---------------|----------------|----------|----------|-----------|--------------------|--------------|---------|\n| Hunyuan Video | STA (FA3)      | 26.7     | 0.866    | 0.167     | 0.132              | **719**          | **2.29×**   |\n|               | SVG (FA2)      | 27.2     |**0.895**| **0.114**     | **0.144**              | 867          | 1.90×   |\n|               | **Ours (FA2)** | **27.3** |   0.886| **0.114** | 0.139          | 876      | 1.88× |\n|               | **Ours (FA3)** | **27.3** |   0.886| **0.114** | 0.139          | 724      | 2.27× |\n| Wan2.1-14B    | STA (FA3)      | 22.9     | 0.830    | 0.171     | **0.132**              | **812**          | **2.01×**   |\n|               | SVG (FA2)           | 23.2     | 0.825    | 0.202     | 0.114              | 949          | 1.71×   |\n|               | **Ours (FA2)**       | **23.9** | **0.842**| **0.163** | 0.128          | 917      | 1.77× |\n|               | **Ours (FA3)**       | **23.9** | **0.842**| **0.163** | 0.128          | 825      | 1.98× |\n\n\n### Ablation Studies\n\nAs requested by the reviewer, we provide additional ablation studies to justify our design choices. The experiments were conducted on Wan2.1 14B at a resolution of $1280\\times768$ with 117 frames. We will include them in our manuscript.\n\n**Attention Sink**\n\nWe follow SVG [1] to preserve attention to the first frame (attention sink), which retains a significant portion of the attention energy. Quality metrics listed in the table below show that removing the attention sink could result in noticeable degradation of video quality. Here we align the mask sparsity with and without the attention sink for fair comparison.\n\n| Configuration                | PSNR ($\\uparrow$)  | SSIM ($\\uparrow$)  | LPIPS ($\\downarrow$) |\n|------------------------------|-------|-------|-------|\n| Ours w/o Sink       | 21.1  | 0.767 | 0.209 |\n| **Ours**                 | **23.6**  | **0.823** | **0.146** |\n\n**Width of Central Band**\n\nThe table below shows results of different central band widths. A central band width of $d$ means attention between frames $i$ and $j$ where $|i - j| \\leq d$. For fair comparison, we align mask sparsity across all settings. Narrower bands (e.g., $d=0$) are suboptimal as they miss some important interactions between neighboring frames, which is crucial for video quality, while wider bands (e.g., $d=2$) spend too much computation budget on the central areas, failing to capture the critical tokens in the distant frames.\n\n| Central Band Width | PSNR ($\\uparrow$)  | SSIM ($\\uparrow$)  | LPIPS ($\\downarrow$) |\n|------------------------------|-------|-------|-------|\n| 0     | 23.5  | 0.810 | 0.148 |\n| **1 (Ours)** | **23.6**  | **0.823** | **0.146** |\n| 2    | 18.8  | 0.660 | 0.275 |\n\n\n**Width Scale Between Consecutive Bands**\n\nThe table below shows results of different width width scale between consecutive bands. For fair comparison, we also keep mask sparsity consistent across all settings. Our width scale of 2 achieves the best video quality. Smaller scales (e.g., 1.5) reduce attention on the central band, while larger scales (e.g., 3 or 4) limit computation on important tokens in distant frames, making them suboptimal.\n\n| Decay Rate | PSNR(↑) | SSIM(↑) | LPIPS(↓) |\n|------------|---------|---------|----------|\n| 1.5| 23.2    | 0.813   | 0.154    |\n| **2 (Ours)** | **23.6**| **0.823**| **0.146**|\n| 3  | 23.4    | 0.819   | 0.149    |\n| 4  | 21.8    | 0.759   | 0.182    |\n\n**Whether to Reduce the Diagonal Frequency when the Width is below 1**\n\nWe reduce the diagonal frequency when the width is below 1 to maintain $\\mathcal{O}(n\\log n)$ complexity. Otherwise, long videos would incur quadratic computation cost with respect to the number of frames, despite potential quality gains. Therefore, this ablation is not feasible in our design choice. We appreciate your understanding and are open to further discussion on this.\n\n### Why Aura Attention is Better for Longer Video Generation\n\nFor long video generation, the large number of tokens makes dense attention inefficient and hard to focus on key interactions. Aura Attention addresses this by leveraging the Spatiotemporal Energy Decay to preserve only important interactions, improving both learning efficiency and performance.\n\n[1] Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity, ICML 2025\n\n[2] Wan 2.2: Leading AI Video Generation Model\n\n[3] Open-Sora 2.0: Training a Commercial-Level Video Generation Model in \\$200k\n\n[4] Fast Video Generation with Sliding Tile Attention, ICML 2025\n\n[5] FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision"}, {"review_id": "ZBpZJq0Jzy", "reviewer": "Reviewer_oEiS", "summary": "This paper introduces Aura Attention, a novel sparse attention mechanism for video diffusion models that reduces computational complexity from O(n²) to O(n log n) by leveraging spatiotemporal energy decay—an observed phenomenon where attention scores decay exponentially with spatial/temporal distance between tokens. The method uses a static hardware-friendly mask that prioritizes local spatial interactions and exponentially shrinks the attention window with temporal distance. Evaluated on SOTA models (HunyuanVideo, Wan2.1, Mochi 1), Aura Attention achieves:\n\n(1) 1.9× speedup at default video lengths with comparable quality.\n\n(2) 4.4× fewer tuning costs and 3.7× faster inference for 4× longer videos when combined with LoRA fine-tuning.\n\nThe paper includes a theoretical analysis of complexity/error bounds, extensive experiments, and ablation studies.", "questions": "My main concerns are as follows:\n\n1. The details of Fig.3 are vague. This is the key motivation of the paper, but how this figure is obtained is vague: (1) Are the results obtained on a single run or an average of a set of videos? (2) It is obtained on a single topic of content or diverse scenarios.\n\n2. Does Fig.3 also hold for other models like Wan2.1 and Open-Sora?\n\n3. The robustness of the observed Attention Spatiotemporal Energy Decay is unclear: does it still hold for more complex scenarios, e.g., large motion videos and videos with complex content?\n\n4. Typo: Line 261, Line 273", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n+ Identifies and formalizes spatiotemporal energy decay (Fig. 3), grounding the approach in empirical analysis of attention maps. This provides a principled basis for sparsity.\n+ Achieves O(n log n) complexity via a static mask (Fig. 4) with exponentially decaying compute density. This avoids costly dynamic profiling (e.g., SVG) and enables hardware optimization. Demonstrated 9× compute reduction for 509-frame videos (Fig. 5a) and 3.7× inference speedup on H100 GPUs.\n+ Demonstrate acceleration on both inference and training with Lora, making potential applications in practice.\n\nWeakness:\n- The details of Fig.3 are vague. This is the key motivation of the paper, but how this figure is obtained is vague: (1) Are the results obtained on a single run or an average of a set of videos? (2) It is obtained on a single topic of content or diverse scenarios.\n- The robustness of the observed Attention Spatiotemporal Energy Decay is unclear: does it still hold for more complex scenarios, e.g., large motion videos and videos with complex content?", "was_revised": true, "final_justification": "The authors mostly address my concerns. The revision should be polished according to the comments in the rebuttal.", "rebuttal": "Thanks for your insightful and constructive reviews. Below, we provide point-by-point responses to each of your comments.\n\n### How is Figure 3 Obtained?\n\nThe attention scores shown in Figure 3 are averaged across all attention heads, layers, and timesteps, based on 10 diverse prompts from VBench [1]. These prompts cover a range of themes, including animals, sports, people, et cetera.\n\n\n### Does Fig 3. also Hold for Wan2.1 14B or Open-Sora?\n\nYes, both the attention decay phenomenon and our proposed exponential decay model hold for Wan2.1 14B [2] and Open-Sora [3], demonstrating that proposed \"Spatiotemporal Decay\" is universal for leading video generation models.\n\nFor Wan2.1 14B [2], we conducted a quantitative regression analysis on the average attention using 10 diverse prompts from VBench [1]. Specifically, we fitted three functions to the distribution: \n- the inverse model $y = \\frac{a}{(x-b)} + c$, \n- the inverse square model $y = \\frac{a}{(x-b)^2} + c$ \n- our proposed exponential decay model $y = \\exp(-ax + b) + c$.\n\nThe regression results, measured by $R^2$ (higher is better), is summarized in the table below. Our exponential decay model achieves the best fit for both spatial and temporal decay, with $R^2$ values up to 0.996.\n\n| Regression Model                   | Spatial Decay $R^2$ ($\\uparrow$)    | Temporal Decay $R^2$ ($\\uparrow$)   |\n|------------------------------------|-------------|--------------|\n| $y = \\frac{a}{(x-b)} + c$          | 0.980       | 0.989        |\n| $y = \\frac{a}{(x-b)^2} + c$        | 0.988       | 0.994        |\n| $y = \\exp(-ax + b) + c$ **(Ours)** | **0.990**   | **0.996**    |\n\nWe applied the same analysis to Open-Sora 2.0, and again our exponential model achieved the highest $R^2$ values—over 0.992 in both decays.\n\n| Regression Model                   | Spatial Decay $R^2$ ($\\uparrow$)    | Temporal Decay $R^2$ ($\\uparrow$)   |\n|------------------------------------|-------------|--------------|\n| $y = \\frac{a}{(x-b)} + c$          | 0.986       | 0.996        |\n| $y = \\frac{a}{(x-b)^2} + c$        | 0.988       | 0.996        |\n| $y = \\exp(-ax + b) + c$ **(Ours)** | **0.992**   | **0.997**    |\n\nThese results demonstrate that both models follow the exponential Spatialtemporal Energy Decay.\n\n### Robustness of Spatiotemporal Energy Decay in Complex Scenarios\n\nAs requested by the reviewer, we validate our exponential decay model in the complex scenarios. We collected 10 high-motion prompts (e.g. dancing, competitive sports, and culinary action) with detailed and complex descriptions. Here we provide an example prompt.\n\n> breakdancer caught mid-freeze, balancing on one hand while the crowd forms a vibrant, cheering circle, energy radiating through low ambient street lighting.\n\nSimilarly, we average all the attention maps of Wan2.1 14B [2] on these prompts and then perform the regression analysis on the attention scores. The results are reported below. Our exponential decay model remains the best fit, achieving up to 0.997 $R^2$, demonstrating that the decay pattern holds in complex scenarios.\n\n| Regression Model                   | Spatial Decay $R^2$ ($\\uparrow$)    | Temporal Decay $R^2$ ($\\uparrow$)   |\n|------------------------------------|-------------|--------------|\n| $y = \\frac{a}{(x-b)} + c$          | 0.976       | 0.987        |\n| $y = \\frac{a}{(x-b)^2} + c$        | 0.987       | 0.993        |\n| $y = \\exp(-ax + b) + c$ **(Ours)** | **0.989**   | **0.997**    |\n\nWe also evaluated the visual quality of Aura Attention on these complex prompts for default-length generation. As shown in below table, the similarity metrics demonstrate that our approach maintains high video fidelity on both VBench prompts used in our paper and the complex prompts, indicating robustness of our method.\n\n|                     | PSNR ($\\uparrow$) | SSIM ($\\uparrow$) | LPIPS ($\\downarrow$) |\n|---------------------|-------------------|--------------------|----------------------|\n| VBench Prompts            | **23.6**              | 0.823              | **0.146**                |\n| Complex Prompts  | 23.1              | **0.842**              | 0.178                |\n\nWe also compare Aura Attention against the full attention for 4$\\times$ longer video generation of HunyuanVideo after LoRA tuning. It is 4.4× faster in training and 3.7× faster in inference, while also delivering a slightly higher Vision Reward score:\n\n| Method                  | Vision Reward |\n|-------------------------|--------------------|\n| Full Attention | 0.102             |\n| **Our Aura Attention**  | **0.105**             |\n\n### Typos\nThanks for pointing these out! We will correct them in our manuscript.\n\n[1] VBench: Comprehensive Benchmark Suite for Video Generative Models\n\n[2] Wan: Open and Advanced Large-Scale Video Generative Models\n\n[3] Open-Sora 2.0: Training a Commercial-Level Video Generation Model in \\$200k"}]

Author rebuttal:

Meta-review: Summary:

This paper introduces Aura Attention, a novel sparse attention mechanism for video diffusion models, motivated by the empirical observation of "Spatiotemporal Energy Decay" - the tendency for attention scores to decay exponentially with increasing spatial and temporal distance between tokens. The proposed static attention mask achieves O(n log n) complexity with respect to the number of frames, enabling efficient long video generation with minimal quality loss. The method is validated on several state-of-the-art video generation models (HunyuanVideo, Wan2.1-14B, Mochi 1), demonstrating significant speedups in both training and inference, especially for long videos, and supports efficient LoRA-based fine-tuning.

Strengths 
- Principled Motivation: The identification and quantitative validation of the spatiotemporal energy decay phenomenon provides a strong, intuitive basis for the proposed sparsity pattern.
- Technical Innovation: The static, exponentially-decaying attention mask is both hardware-friendly and effective, achieving substantial computational savings without sacrificing video quality.
- Comprehensive Evaluation: Extensive experiments across multiple large-scale models and settings (default and extended video lengths) demonstrate consistent speedups (up to 3.7× inference, 4.4× training) and quality preservation.
- Ablation and Robustness: The authors provided additional ablation studies and quantitative analyses in the rebuttal, addressing concerns about design choices and the generality of the energy decay phenomenon across models and complex scenarios.

Weaknesses
- Initial Lack of Quantitative Analysis: The paper relied on qualitative evidence for the energy decay phenomenon. The rebuttal addressed this with detailed regression analyses across diverse prompts and models, showing strong quantitative support.
- Ablation Studies: Reviewers requested more ablations on mask design choices. The authors provided these in the rebuttal, validating their default settings.
- Quadratic Complexity in Spatial Resolution: While Aura Attention is O(n log n) in frames, it remains quadratic in spatial resolution. The authors clarified that practical speedups are maintained at higher resolutions and discussed how spatial compression techniques can be combined with their method.
- Baselines and Comparisons: Some reviewers suggested including more linear attention baselines. The authors added results for additional baselines in the rebuttal, showing that Aura Attention outperforms them in the video setting.

Overall, the paper received two clear accept ratings and two borderline accept ratings, with all reviewers acknowledging the technical soundness, empirical rigor, and potential impact of the work. ACs concur with the reviewer consensus. 

Suggestions for Camera-Ready:
- Incorporate the additional quantitative analyses and ablation studies from the rebuttal into the main paper.
- Further clarify the details of how attention decay is measured and visualized.
- Address minor presentation issues and typos as noted by reviewers.
Decision: Accept (poster)