Title: Radial Attention: $\mathcal{O}(n\log n)$ Sparse Attention with Energy Decay for Long Video Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: video generation

sparse attention

diffusion models
Abstract: Recent advances in diffusion models have enabled high-quality video generation, but the additional temporal dimension significantly increases computational costs, making training and inference on long videos prohibitively expensive. In this paper, we identify a phenomenon we term Spatiotemporal Energy Decay in video diffusion models: post-softmax attention scores diminish as spatial and temporal distance between tokens increase, akin to the physical decay of signal or waves over space and time in nature. Motivated by this, we propose Radial Attention, a scalable sparse attention mechanism with $\mathcal{O}(n \log n)$ complexity that translates energy decay into exponentially decaying compute density, which is significantly more efficient than standard $\mathcal{O}(n^2)$ dense attention and more expressive than linear attention. Specifically, Radial Attention employs a simple, static attention mask where each token attends to spatially nearby tokens, with the attention window size shrinking with temporal distance. Moreover, it allows pre-trained video diffusion models to extend their generation length with efficient LoRA-based fine-tuning. Extensive experiments show that \method maintains video quality across Wan2.1-14B, HunyuanVideo, and Mochi 1, achieving up to a 1.9× speedup over the original dense attention. With minimal tuning, it enables video generation up to 4× longer while reducing training costs by up to 4.4× compared to direct fine-tuning and accelerating inference by up to 3.7× compared to dense attention inference. Code is released at https://github.com/mit-han-lab/radial-attention.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Aura Attention, a novel sparse attention mechanism for video diffusion models that reduces computational complexity from O(n²) to O(n log n) by leveraging spatiotemporal energy decay—an observed phenomenon where attention scores decay exponentially with spatial/temporal distance between tokens. The method uses a static hardware-friendly mask that prioritizes local spatial interactions and exponentially shrinks the attention window with temporal distance. Evaluated on SOTA models (HunyuanVideo, Wan2.1, Mochi 1), Aura Attention achieves:\n\n(1) 1.9× speedup at default video lengths with comparable quality.\n\n(2) 4.4× fewer tuning costs and 3.7× faster inference for 4× longer videos when combined with LoRA fine-tuning.\n\nThe paper includes a theoretical analysis of complexity/error bounds, extensive experiments, and ablation studies.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n+ Identifies and formalizes spatiotemporal energy decay (Fig. 3), grounding the approach in empirical analysis of attention maps. This provides a principled basis for sparsity.\n+ Achieves O(n log n) complexity via a static mask (Fig. 4) with exponentially decaying compute density. This avoids costly dynamic profiling (e.g., SVG) and enables hardware optimization. Demonstrated 9× compute reduction for 509-frame videos (Fig. 5a) and 3.7× inference speedup on H100 GPUs.\n+ Demonstrate acceleration on both inference and training with Lora, making potential applications in practice.\n\nWeakness:\n- The details of Fig.3 are vague. This is the key motivation of the paper, but how this figure is obtained is vague: (1) Are the results obtained on a single run or an average of a set of videos? (2) It is obtained on a single topic of content or diverse scenarios.\n- The robustness of the observed Attention Spatiotemporal Energy Decay is unclear: does it still hold for more complex scenarios, e.g., large motion videos and videos with complex content?\n\nQUESTIONS:\nMy main concerns are as follows:\n\n1. The details of Fig.3 are vague. This is the key motivation of the paper, but how this figure is obtained is vague: (1) Are the results obtained on a single run or an average of a set of videos? (2) It is obtained on a single topic of content or diverse scenarios.\n\n2. Does Fig.3 also hold for other models like Wan2.1 and Open-Sora?\n\n3. The robustness of the observed Attention Spatiotemporal Energy Decay is unclear: does it still hold for more complex scenarios, e.g., large motion videos and videos with complex content?\n\n4. Typo: Line 261, Line 273\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Aura Attention, an static attention mask with O(n log n) complexity wrt. number of frames suitable for long video generation. The design of this attention mask is motivated by empirical observations on the attention patterns in HunyuanVideo. Long video generaiton is enabled through LoRA finetuning, since the sparse attention mask reduces the prohibitive cost of training on long videos. Experiments are done on three backbones at the default video length and extended length, showing that Aura Attention speeds up both training and inference while achieving good quality compared to baselines.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- Good writing quality. \n- Strong experiment results on long video generation in tab. 2. A wide range of baselines are adopted in tab. 1, 2. \n- The design of Aura Attention mask is intuitive, innovative, and motivated through empirical observations shown in fig. 3.\n- Complexity and error bound for Aura Attention are provided.\n- Aura Attention enables long video generation finetuning with modest training cost, which is a core challenge for long video generation.\n\n## Weaknesses\n- The complexity of Aura Attention, while $\\mathcal{O}(n \\log{n})$ wrt. number of frames, still have $\\mathcal{O}(n^2)$ wrt. the spatial resolution, which is shown in Eq. 6 and discussed in Sec. 6 as a limitation. This limits the main application of Aura Attention to long video generation, while existing methods (STA, SVG) are either more efficient or have better quality in the short video setting, as shown in Tab. 1.\n- No ablation studies are done. Possible ablation studies include the usage of attention sink, the width of the central band, the width scale between consecutive bands (currently 2x), whether to reduce the frequency of diagonals if the diagonal width falls below 1, etc. Conducting some ablation studies can provide insights into the design space of Aura Attention.\n\nQUESTIONS:\n- Figure 5 shows that Aura Attention enables better video quality for long video generation finetuning. Can the authors give some conceptual or theoretical analysis behind this?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Aura Attention, a sparse attention mechanism designed for efficient long video generation in diffusion models. The key innovation is inspired by the observation of spatiotemporal energy decay in attention scores, where post-softmax scores decrease with increasing spatial and temporal distance between tokens. Aura Attention translates this decay into exponentially decaying compute density, achieving O(n log n) computational complexity. The method uses a static attention mask that focuses on spatially nearby tokens with a shrinking window size as temporal distance increases. Experiments on models like HunyuanVideo and Wan2.1-14B show up to 1.9× speedup at default lengths and 3.7× speedup for 4× longer videos, while reducing training costs by 4.4× with minimal fine-tuning using LoRA. The approach maintains video quality comparable to full attention baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrength：\n- The integration of physical energy decay principles into attention mechanism design is innovative, providing a principled way to reduce computation while preserving critical dependencies.\n- Achieves O(n log n) complexity, addressing the quadratic scaling issue of full attention, which is crucial for long video generation.\nEmpirical Validation: Comprehensive experiments across multiple large video models (HunyuanVideo, Wan2.1-14B) demonstrate consistent speedups and quality preservation.\n- Enables longer video generation (up to 4× extension) with minimal fine-tuning, reducing both training costs and inference latency significantly.\n\nWeakness：\n- The complexity still scales quadratically with spatial resolution, which may limit performance for very high-resolution videos.\n- The model relies on a simplified exponential decay function for attention scores, which may not fully capture complex spatiotemporal dependencies in real-world videos.\n- The static mask design might struggle with dynamic scenes requiring long-range temporal dependencies beyond the predefined decay  pattern.\n- Some baselines (e.g., SANA) are not fully optimized, and comparisons may not reflect the latest advancements in linear attention. Discussing more linear attention-based image generation methods here would be better.\n\nQUESTIONS:\nPlease refer to the weakness part.\n\nLIMITATIONS:\nPlease refer to the weakness part.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Aura Attention, a sparse attention mechanism designed to mitigate the quadratic computational cost of generating long videos with diffusion models. The central motivation is a phenomenon the authors call \"Spatiotemporal Energy Decay,\" where attention scores are observed to diminish with increasing distance between tokens in both space and time. Aura Attention implements a static attention mask that mimics this decay by exponentially shrinking the attention window as temporal distance grows, achieving a reported complexity of O(nlogn). This allows pre-trained models to be adapted for longer video generation through efficient LoRA fine-tuning. The authors test their method on several large video models (Hunyuan Video, Wan2.1-14B, Mochi 1) and report substantial improvements in inference speed and training cost while preserving video quality.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1.\tThe paper's primary strength lies in its empirical results. The method delivers significant, practical speedups (e.g., up to 3.7x in inference for 4x longer videos) while maintaining video quality across multiple SOTA models. This is a non-trivial engineering achievement.\n2.\tThe concept of \"Spatiotemporal Energy Decay\" is an intuitive and appealing motivation for designing a sparsity pattern. It provides a clear, high-level story for why the proposed attention mask might be effective.\n\nWeakness:\n1.\tThe paper’s scientific foundation rests on the \"Spatiotemporal Energy Decay\" phenomenon, which is modeled as an exponential decay. While this is a compelling narrative, the evidence provided is primarily qualitative, based on visual inspection of the plots in Figure3. For a method presented as \"principled,\" the lack of a rigorous quantitative analysis to validate this core assumption is a significant weakness. This makes the approach feel more like a well-motivated heuristic than a scientifically-validated model.\n2.\tThe paper omits critical ablation studies that would justify its specific implementation. Key decisions feel arbitrary and are not supported by experiments: The model retains full attention for the first few layers and initial denoising timesteps, which is noted as being \"crucial for video quality\". However, there is no analysis of how this parameter was chosen or its impact on the performance/speed trade-off.\n\nQUESTIONS:\nPlease read the weakness of the manuscript provided above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-12"}