Title: VORTA: Efficient Video Diffusion via Routing Sparse Attention
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video Generation

Diffusion Models

Vision Transformers

Efficient Models
Abstract: Video diffusion transformers have achieved remarkable progress in high-quality video generation, but remain computationally expensive due to the quadratic complexity of attention over high-dimensional video sequences.
Recent acceleration methods enhance the efficiency by exploiting the local sparsity of attention scores; yet they often struggle with accelerating the long-range computation.
To address this problem, we propose VORTA, an acceleration framework with two novel components: 1) a sparse attention mechanism that efficiently captures long-range dependencies, and 2) a routing strategy that adaptively replaces full 3D attention with specialized sparse attention variants.
VORTA achieves an end-to-end speedup $1.76\times$ without loss of quality on VBench.
Furthermore, it can seamlessly integrate with various other acceleration methods, such as model caching and step distillation, reaching up to speedup $14.41\times$ with negligible performance degradation.
VORTA demonstrates its efficiency and enhances the practicality of video diffusion transformers in real-world settings.
Codes and weights are available at https://github.com/wenhao728/VORTA.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nVORTA is a plug-in accelerator for Video Diffusion Transformers that adds a tiny router and two sparse attention options to every self-attention block. At each diffusion step, the router, conditioned on timestep and head features, sends a head either to a sliding-tile local window or to a global branch that keeps only a coreset of representative tokens chosen inside space-time buckets, falling back to full attention only when needed. This adaptive routing turns quadratic attention into near-linear cost without retraining the generator. Tested in 720p HunyuanVideo and Wan2.1, VORTA shortens end-to-end inference time and lowers memory usage while matching or slightly improving baseline models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper evaluates VORTA on two demanding 720p text-to-video diffusion transformers and reports end-to end latency, GPU memory, VBench sub-scores, and LPIPS, giving a well-rouded view of speed-quality trade-offs.\n- Experiments show that adaptive routing alone yields about 1.7 times speed-ups without sacrificing VBench quality, and that VORTA composes additively with cached-KV and step-distillation to reach up to 14 times overall acceleration.\n- Ablation studies isolate every design decision: removing either sparse branch, disabling timestep conditioning, replacing bucketed coreset selection with naive pooling, and omitting full-attention fallback. Each variant's impact on speed is quantified.\n- Analysis of attention-score statistics across diffusion steps reveals that global context dominates early iterations while local detail dominates later ones; this empirical insight motivates the router and ilustrates a general principle that may guide future diffusion designs.\n- Implementation details are clearly documented, making reproductions easier.\n\nWeaknesses:\n- All quality evidence comes from a single, learned-metric suite (VBench). Adding a second, independent check, either a brief human-preference study or a public metric like FVD, would make the ‘no quality loss’ claim more robust.\n- The learned router, though lightweight, requires an extra distillation pass and a small video dataset. It would be good addition to the paper if they examine router robustness on out-of-distribution prompts or much longer clips.\n- Memory consumption rises by around 7% or more versus the dense baseline, and the added branches increase implementation complexity, which could hinder deployment on single-GPU or resource-constrained environments.\n- Related-work discussion omits earlier conditional-computation and information-gain routing methods (e.g., ConvNet-AIG, Information Gain Trellis), understating the lineage of adaptive gating ideas and making the novelty of the router appear greater than it is.\n\nQUESTIONS:\n- The router is distilled only to replicate dense attention, yet Table 1 shows a 0.3 -0.4 VBench gain over the baseline. Please analyse the source of this improvement. Showing that the gain is statistically significant and not incidental would raise my rating.\n- VBench is multi-dimensional (scoring frame fidelity, motion, text alignment, etc.) but all dimensions are produced by learned evaluators. A second, independent lens (e.g., a brief human-preference study or a metric like FVD) would strengthen the \"no-quality-loss\" claim.\n- Add a short related-work paragraph on conditional computation, e.g. ConvNet-AIG (Veit et al., 2017) and Information Gain Trellis (Bicici et al., 2024), to acknowledge prior adaptive-routing ideas and highlight what is new (per-head, per-timestep gating for diffusion).\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes VORTA, a sparse attention technique with a router to speed up the inference stage of Video Diffusion Models. This paper proposes three attention variations: full attention, sliding window attention, and coreset attention for each head. They further propose a router to classify each attention head into one of these attention patterns, and apply distillation loss and regularization loss to punish the excessive usage of full attention. VORTA exhibits strong quality and speedup performance on HunyuanVideo and Wan 2.1, demonstrating its effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The design of the attention router, along with the introduction of the distillation and regularization loss term, is innovative and contributes to the novelty of the approach.\n2. The proposed method is flexible and can be effectively integrated with other techniques, such as PAB, to achieve additional speedup.\n\nWeaknesses:\n1. The coreset attention pattern requires further empirical validation. It remains unclear how performance would be affected if alternative attention patterns—such as the temporal attention used in Sparse VideoGen[1]—were employed.\n2. The paper does not provide comparisons against other baselines using the Wan 2.1 model, which limits the assessment of its relative performance.\n3. Refer to the questions section.\n\n[1] Xi, Haocheng, et al. \"Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity.\" arXiv preprint arXiv:2502.01776 (2025).\n\nQUESTIONS:\n1. What's the average attention sparsity during the generation process for sliding window attention and coreset attention?\n2. What is the value of r_core for BCS module? How does this value affect the final performance?\n3. What will the attention maps look like for the 0.2% full attention heads? More visualization results for these attention heads will help to understand the model's behavior.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents VORTA, an acceleration framework for Video Diffusion Transformers (VDiTs) that improves efficiency without sacrificing performance. VORTA introduces a sparse attention mechanism to capture long-range dependencies and a dynamic routing strategy to select optimal attention variants. This approach achieves a 1.76× speedup in video generation and can integrate with other methods, reaching up to a 14.41× speedup.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n- Adaptability: The dynamic routing mechanism is tailored for adaptive sparse scheme for different diffusion configurations and backbones.\n- Compatibility with other compression techniques: VORTA could be with other existing acceleration techniques, like caching and step distillation.\n- Detailed description of implementation details in the Appendix.\n\nWeakness\nMy main concern lies in the lack of clarity regarding the motivation, applicability, and effectiveness of some proposed methods.\n- Bucketed Core set Selection (BCS):\n  - While detailed pseudo-code is provided, it only describes how to determine the core set attention sparse mask. However, it is unclear whether this sparse mask is shared across different conditions and time steps. Additionally, it remains ambiguous whether the BCS is performed online for each attention head.\n  - If BCS is performed online, the overhead analysis appears to be missing. This process involves multiple steps, which could introduce significant overhead. If the BCS sparse mask is determined offline, the paper should justify whether such a sparse mask can generalize across multiple time steps and prompts.\n  - The assumption underlying the design of BCS is that the \"center token is very similar to its neighboring tokens,\" which, as the authors state, \"will not always hold.\" When this assumption does not hold, the claim that \"the remaining tokens could complement the necessary information\" needs proper justification. Furthermore, there is a lack of analysis to validate the assumption behind BCS and to assess the effectiveness of BCS itself.\n- Dynamic Routing:\n  - The motivation for employing training-based dynamic routing requires further justification. The necessity of the routing design should be demonstrated by showing that the optimal compression plan varies significantly across different conditions. Even in such cases, the advantage of introducing an additional training-based router, rather than simply adopting different plans or heuristic rules for different conditions, needs to be addressed.\n  - Additionally, the generalization ability of the optimized router should be thoroughly verified to demonstrate its effectiveness.\n\nQUESTIONS:\nPlease refer to the weakness section.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

comments: SUMMARY:
VORTA accelerates video diffusion models by dynamically selecting and routing sparse attention mechanisms, improving efficiency with minimal impact on generation quality. It employs coreset selection to reduce attention computation and uses a router to adaptively choose the most suitable attention branch, achieving up to 1.76x end-to-end speedup.

STRENGTHS AND WEAKNESSES:
**Strength**:
1. The paper addresses a critical issue in video generation, the computational inefficiency of Video Diffusion Transformers. By introducing VORTA, it significantly reduces the sampling time, making VDiTs more practical for real-world applications.
2. The paper presents a novel coreset attention mechanism and a signal-aware routing strategy that dynamically selects the most appropriate sparse attention variant. The bucketed coreset selection (BCS) strategy is an innovative way to handle long-range dependencies efficiently.
3. The experimental results are convincing, demonstrating the effectiveness of VORTA in improving efficiency while maintaining generation quality. The ablation studies provide insights into the importance of each component and the impact of different design choices.

**Weakness**:
1. The demonstration of the proposed corset selection and router strategy is not clear enough, could you please complement more illustration or description?

QUESTIONS:
Please refer to the weakness.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-05