Title: Faster Video Diffusion with Trainable Sparse Attention
Venue: NeurIPS 2025
Accepted: yes
Keywords: Diffusion Model;Video Generation;Efficient Attention;Sparse Attention
Abstract: Scaling video diffusion transformers (DiTs) is limited by their quadratic 3D attention, even though most of the attention mass concentrates on a small subset of positions.  We turn this observation into VSA, a trainable, hardware-efficient sparse attention that replaces full attention at both training and inference. In VSA, a lightweight coarse stage pools tokens into tiles and identifies high-weight critical tokens; a fine stage computes token-level attention only inside those tiles subjecting to block computing layout to ensure hard efficiency. This leads to a single differentiable kernel that trains end-to-end, requires no post-hoc profiling, and sustains 85\% of FlashAttention3 MFU. We perform a large sweep of ablation studies and scaling-law experiments by pretraining DiTs from 60M to 1.4B parameters. VSA reaches a Pareto point that cuts training FLOPS by 2.53$\times$ with no drop in diffusion loss. Retrofitting the open-source Wan2.1-1.3B model speeds up attention time by 6$\times$ and lowers end-to-end generation time from 31s to 18s with comparable quality, while for the 14B model, end-to-end generation time is reduced from 1274s to 576s.
Furthermore, we introduce a preliminary study of Sparse-Distill, the first method to enable  sparse attention and distillation concurrently, achieving 50.9x speed up for Wan-1.3B while maintaining quality.
These results establish trainable sparse attention as a practical alternative to full attention and a key enabler for further scaling of video diffusion models.  Code is available at https://github.com/hao-ai-lab/FastVideo.


Peer reviews:
comments: SUMMARY:
This paper introduces ViSA, a trainable, hardware-efficient sparse attention mechanism for video diffusion transformers. By using a two-stage coarse-to-fine block-sparse design aligned with GPU kernels, VISA achieves faster attention inference without sacrificing generation quality. Extensive experiments demonstrate the effectiveness of ViSA.

STRENGTHS AND WEAKNESSES:
### **Strengths**
1. The trainable sparse attention is novel, which can eliminate train-test mismatch.
2. The block-sparse design is hardware-efficient and achieves real-world speedups compatible with Flash Attention.
3. Extensive experiments, including scaling laws and ablations, validate ViSA’s effectiveness across model sizes and training budgets.

### **Weaknesses**
1. The tile size and sparsity are predefined hyperparameters that require empirical tuning and may not generalize well to other resolutions or multi-scale training and inference.
2. Despite good quantitative results, the lack of qualitative visualizations makes it hard to judge the real impact of the sparse attention mechanism, especially given VBench’s limited coverage.
3. The VBench results reported seem inconsistent, with Wan’s baseline scores much worse than previously published (should be 84.92/80.10/83.96 for Quality/Semantic/Total).

QUESTIONS:
Please see the weaknesses for the main questions. In addition, the anonymous github repo is empty.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper \"Faster Video Diffusion with Trainable Sparse Attention\" introduces VISA (Video Diffusion with Trainable Sparse Attention), a novel method aimed at accelerating video diffusion models by reducing the computational cost of attention mechanisms. The key contribution is a trainable, hardware-efficient sparse attention mechanism that replaces full attention during both training and inference. VISA employs a hierarchical attention approach, consisting of a coarse stage that identifies critical tokens and a fine stage that computes token-level attention within these critical regions. The method is designed to be end-to-end trainable and compatible with modern GPU architectures, achieving significant speedups without compromising generation quality. The authors demonstrate that VISA reduces training FLOPS by 2.53× and inference time by 1.7× compared to full attention, while maintaining comparable performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The paper is technically sound, with a well-supported methodology and comprehensive experiments. The authors provide extensive ablation studies and scaling-law experiments, demonstrating the effectiveness of VISA across different model sizes and training budgets. The results show significant improvements in computational efficiency without a drop in diffusion loss.\n\nWeaknesses: \n- Generalizability: While the experiments demonstrate impressive results on specific datasets and model sizes, it is unclear how well VISA generalizes to other types of video data or different diffusion model architectures. Additional experiments on diverse datasets would strengthen the paper's claims.\n- Computational Overhead During Training: Although VISA reduces overall computational cost, the introduction of the coarse stage adds some overhead. The authors could provide more detailed analysis on the trade-offs between the benefits of sparse attention and the additional complexity introduced during training.\n\nQUESTIONS:\n- Generalization to Other Datasets: How does VISA perform on other types of video data, such as higher-resolution videos or videos with different content characteristics? Can the authors provide additional experiments or analysis to demonstrate the generalizability of their method?\n\n- Training Overhead: Can the authors provide more details on the computational overhead introduced by the coarse stage during training? How does this overhead compare to the overall computational savings achieved by VISA?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of their work by discussing the current constraints, such as the fixed cube size and the dependency on specific hardware architectures. They also acknowledge the potential negative societal impact of their work, such as the risk of generating misleading or harmful content, and emphasize the need for robust detection tools and ethical guidelines. Constructive suggestions for improvement could include exploring methods to make VISA more adaptable to different hardware platforms and conducting additional experiments to validate its generalizability to diverse datasets.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}

comments: SUMMARY:
The paper introduces ViSA, a trainable sparse attention mechanism for video diffusion transformers (DiTs) that reduces the computational bottleneck of quadratic 3D attention. ViSA employs a two-stage approach to identify critical tokens and compute attention efficiently, achieving a 2.53× reduction in training FLOPS and a 6× speedup in attention time with comparable quality.

STRENGTHS AND WEAKNESSES:
Strengths:

1. ViSA proposes a novel trainable sparse attention mechanism for video DiTs, learning critical token positions during training rather than sparsifying only at inference, thus better preserving model performance.

2. The design of ViSA fully takes into account the hardware characteristics of modern GPUs, ensuring compatibility with efficient attention implementations such as FlashAttention through block-sparse computational layouts, thereby translating theoretical computational savings into actual runtime acceleration.

3. The paper includes extensive ablation studies and scaling experiments, demonstrating ViSA's robustness across various configurations (e.g., tile sizes, sparsity levels) and model sizes.

4. This paper is well written and easy to follow.

Weakness: Although ViSA has limitations like fixed cube size and optimal sparsity determination, the authors provide practical solutions like cropping that still significantly enhance video DiT training and inference efficiency.

QUESTIONS:
NA

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-06