VORTA: Efficient Video Diffusion via Routing Sparse Attention
Authors: Wenhao Sun
Rong-Cheng Tu
Yifu Ding
Jingyi Liao
Zhao Jin
Shunyu Liu
Dacheng Tao
Venue: NeurIPS 2025 poster ()
Keywords: Video Generation
Diffusion Models
Vision Transformers
Efficient Models
Primary area: applications
TL;DR: VORTA accelerates video diffusion transformers by sparse attention and dynamic routing, achieving up to 14.4× speedup with negligible quality loss.

Abstract: Video diffusion transformers have achieved remarkable progress in high-quality video generation, but remain computationally expensive due to the quadratic complexity of attention over high-dimensional video sequences.
Recent acceleration methods enhance the efficiency by exploiting the local sparsity of attention scores; yet they often struggle with accelerating the long-range computation.
To address this problem, we propose VORTA, an acceleration framework with two novel components: 1) a sparse attention mechanism that efficiently captures long-range dependencies, and 2) a routing strategy that adaptively replaces full 3D attention with specialized sparse attention variants.
VORTA achieves an end-to-end speedup $1.76\times$ without loss of quality on VBench.
Furthermore, it can seamlessly integrate with various other acceleration methods, such as model caching and step distillation, reaching up to speedup $14.41\times$ with negligible performance degradation.
VORTA demonstrates its efficiency and enhances the practicality of video diffusion transformers in real-world settings.
Codes and weights are available at https://github.com/wenhao728/VORTA.

VORTA: Efficient Video Diffusion via Routing Sparse Attention
Wenhao Sun Rong-Cheng Tu
Yifu Ding
Zhao Jin
Jingyi Liao Shunyu Liu
Dacheng Tao
College of Computing and Data Science, Nanyang Technological University, Singapore
{wenhao006, rongcheng tu, m2409547h, zhao jin wtt.:/tuuuuuuu......eeeue sg {jingyi012, shunyu. 1iu, dacheng entu snt.u.du..uggg
HunyuanVidee
avg. latency
original
avg. latency
176% speedup
VORTA PAB
avg. Jalency 444s
235% speedup
avg. latency 72s
speedup
prompt: a,ooyuul Corgi with a fluffy coot and perky ears bounds through sunlit park...
prompt: younn woman with curly hair and a bright smile
wearing a yellow sweater..
Figure I: VORTA enables lossless acceleration video diffusion transformers [16, 43], and remains compatible with other acceleration methods such as PAB [60] and PCD [44] for additional speedups.
Abstract
Video diffusion transformers have achieved remarkable progress in high-quality video generation, but remain computationally expensive due the quadratic com- plexity of attention over high-dimensional video sequences. Recent acceleration methods enhance the efficiency by exploiting the local sparsity of attention scores; yet they often struggle with accelerating the long-range computation. To address this problem, propose VORTA, acceleration framework with two novel components: (1) sparse attention mechanism that efficiently captures long-range dependencies, and (2) routing strategy that adaptively replaces full 3D attention with specialized sparse attention variants. VORTA achieves an end-to-end speedup 1.76x without loss of quality VBench. Furthermore, can scamlessly inte- grate with various other acceleration methods, such as model caching and step distillation, reaching up to speedup 14.41 with negligible performance degrada- tion. VORTA demonstrates its efficiency and enhances the practicality of video
$ Co-corresponding authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

diffusion transformers in real-world settings, Codes and weights are available at https:/ gitbb....... com//eenha7728/VRRZ2
1 Introduction
Video Diffusion Transformers (VDiTs) have demonstrated impressive video generation performance, producing realistic and dynamic content [16, 33, 43, 51J. Despite this progress, VDiTs remain computationally expensive due to the inherently high-dimensional nature of video data, compounded by the quadratic complexity of attention operations. For example, recent Hunyuan Video [16] requires almost 1000 seconds (500 PFLOPS) to generate a 5-second 720p video at 24 frames per second (FPS) on an H100 GPU. To mitigate the high sampling cost, few-step sampling methods [39, 44, 52] leverage distillation on the self-consistency property of the probability flow ODE (PF-ODE) [38]. achieving up to an 8x reduction in sampling steps [56]. Other research [22, 26, 60] introduces intermediate feature caching to accelerate sampling without additional training.
Another promising direction investigates miti- gating the quadratic computational complexity of the self-attention operation by leveraging its inherent sparse structure. The attention score distribution in VDiTs exhibits a dichotomy: lo- attention. Local attentions [1, 53] tend concentrate attention scores on small subset of local keys around each query. Specifically, we observed that the spatially closest 4% of the keys (4,000 out of 108,000) contribute more than 80% the to- tal attention score shown in Figure 2 (left). The remaining distant 96% keys, which domi- nate the computational cost, contribute less than 20%. Approaches [1, 50, 53, 55] to mitigate this inefficiency in attention with local behavior re- strict interactions to nearby tokens and discard distant ones. The second regime, long-range at- tention, distributes interactions across the entire sequence to capture global context and long- range dependencies. In these attention heads, the closest keys account for less than 40% of the total attention score as shown in Figure 2 (right). Directly imposing local attention constraint on these inherently nonlocal heads for acceleration compromises the critical global context modeling necessary for effective video generation [51], Many works [47, 55] resort to dense operation in nonlocal heads, which severely limits the overall acceleration gain. Alternative unstructured or semi-structured attention pruning methods [40, 46. 54] also struggle with quadratic-time sparsity detection or profiling, introducing substantial overhead during the forward pass. Therefore, effectively and efficiently accelerating the long-range attention components in VDiTs remains critical, unresolved challenge.
head
head
head head
head head
head
head
timestep 40 (out of 50)
timestep 10 (out 50)
1.0
cal attentions and
attention recall
0.6
0.4
0.0
2000
4000
0
2000
4000
nearest 4,000 keys out of the total 108,000 keys (=
Figure 2: Attention scores recalled by the nearest keys. (left) Attention scores are predominantly concentrated within local neighborhood. (right) The locality is less pronounced at earlier sampling steps. Results are from the 20th (of 60) layer in Hunyuan Video [16]. Only 8 out of the 24 attention heads are shown for clarity.
We observed that components exhibiting long- range dependencies are associated with high intra-sequence redundancy, whose tokens are highly similar. In this scenario, interactions with such redundant tokens are inefficient, as a few representative tokens can summarize their in- formation. An example is provided in Figure (middle), which depicts an intermediate gener- ation result after the first few sampling steps. While high-level semantic structures, like spa- layouts and object motions, are primarily formed, the tokens have not acquired sufficient distinctiveness (i.e.hi highly redundant). This moti- vates design of token-level sparse attention
generatio at generation at stop 10
ganoration al slop 10 with connent of the preciction
Figure 3: (left) Generation with the complete sam- pling process. (middle) Intermediate generation result. (righr) Intermediate generation result with only core-set predictions.

---

to substitute the full-sequence computation when compact core-set is sufficient to capture all necessary information. We propose a bucketed core-set selection (BCS) strategy remove similar, redundant tokens and retain only the most representative ones for attention computation. As illustrated in Figure 3 (right), the output generated with core-set tokens retains semantic accuracy. Given that the core-set contains fewer tokens (e.g., 50% of the full sequence), the attention cost is reduced quadratically (e.g., to 25% of the original cost of the full sequence).
The implicit relationship between the VDiT's attention behaviors (i.e., local or long-range dependen- cies) and the sampling steps, which correlates with the input signal strength, presents a challenge for the principled integration of sparse attention variants. Several approaches [47, 55] log attention score distributions through a reverse diffusion sampling process and reuse them. However, this is impractical in real-world settings where sampling configurations (e.g., video size, sampling steps, or reverse diffusion solvers) frequently change, rendering the logged patterns obsolete and requiring re-tuning. To overcome this, we introduce VORTA routing mechanism that dynamically selects the optimal sparse attention variant by leveraging the input signal-to-noise level. It provides precise control and adaptable support for diverse diffusion configurations, offering greater flexibility.
To conclude, our contributions are summarized as follows:
We propose novel core-set attention that effectively models long-range dependencies while theoretically reducing the attention cost by 75%.
We introduce VORTA, an approach that integrates multiple sparse attentions into VDiTs for end-to-end acceleration without sacrificing performance. VORTA is compatible with various SDE/ODE solvers and network backhones.
VORTA achieves a 1.76x end-to-end speedup VBench [11]. It is also compatible with other acceleration techniques, achieving overall speedups of up to 2 x with feature caching [60] and 14.41x with consistency distillation [44].
2 Preliminary and related work
Flow matching and diffusion models. Flow Matching (FM) [21, 23] builds upon Continuous Normalizing Flows (CNFs) [5] and has since been integrated with the diffusion paradigm [10, 38], forming the foundation of many recent diffusion models [9, 12, 16, 32, 43, 61]. The core concept of FM involves a time-dependent velocity field (VF) and a time-dependent diffeomorphism x; = /x, + (1 (1-1) ()x,, known as allo.. The VF governs the evolution of the flow through an ordinary differential equation (ODE) dx,
(xx) dí, where the flow x, defines a probability density path Pt. starting from the simple prior po = N (0, and evolving towards the intractable target density P1-
Lipman et al. [21] introduced Conditional Flow Matching (CFM) to optimize the neural network of (xx) (i.e. VDiT' in this context) as follows:
LCFM = (x. )x00))) 1l, (xt) (x1 - x2
Once optimized, it can start from X Pu and follow the ODE trajectory generate samples.
3D self-attention of video diffusion transformers. Recent VDiTs [16, 18, 19, 33, 43, 51] employ 3D attention to capture spatio-temporal dependencies in video data. Given video with F frames at resolution of H x W, the input is flattened into a sequence of length L = x H x W, denoted as H E IR lxxd where each token is d-dimensional vector. The self-attention is formulated as:
altn a = softmax (HW x) (HWv),
(2)
where WQ: W Wy E Rdxd are the linear projection matrices. The attention operation is of complexity dl + O(Ld)) In the context of high-resolution video processing, where the embedding dimension d is significantly smaller than the sequence length L, the overall complexity is dominated by the first term O(L2d), which scales unfavorably with sequence length. Despite employing techniques such as Variational Autoencoders (VAEs) [15, 34] and patchification [31], the sequence length L still reaches up to 100K for 5-second 720p video in Hunyuan Video [16]. Consequently, attention operations dominate the computational cost, accounting for over 75% of the computation cost. Optimizing the attention computation is crucial for the efficiency of VDiTs.
3

---

Video diffusion acceleration. General diffusion acceleration methods include step distillation [24, 28, 35, 36, 39, 44], which reduces the number of sampling steps as few to 8 with sufficient tuning, and feature caching [14, 22, 26, 58, 60), which avoids redundant computation by reusing intermediate features. In the case of VDiTs, the long sequence length leads high attention costs. As result, many studies have focused optimizing attention for long sequences. Some recent approaches [47, 55] apply predefined sparse attention patterns to reduce computational overhead. Others [40, 46, 54] adopt online profiling to dynamically select attention patterns during inference. However, these sparse attention techniques depend on careful hyperparameter tuning for different configurations and/or introduce additional forward-pass complexity that offsets their speed benefits. VORTA addresses these challenges with a flexible design that compatible with various diffusion configurations and model backbones, without incurring any additional inference-tin time overhead.
Acceleration by conditional computing. The concept of conditional computation has emerged as a powerful paradigm to accelerate neural networks by selectively activating only subset of the parameters or operations for given input. Bengio et al. [3] first introduced stochastic neurons in which parameter activation is conditional on their output. Subsequent methods selectively drop or gate layers [2, 27, 29, 37, 41, 45, 49], effectively utilizing a different, sparser network structure for each sample. contrast to these prior efforts, our proposed VORTA integrates operation- and token-level sparsity and uses diffusion temporal dynamics to adaptively route computation.
VORTA: efficient video diffusion via routing sparse attention
This section introduces VORTA accelerate VDiTs. We begin with a taxonomy of attentions in Section 3.1. Next, we present two core components: i) sparse attention variants that speed up specific attentions in Section 3.2: ii) a routing strategy that integrates these sparse attentions into pretrained VDiTs for end-to-end acceleration (Section 3.3).
3.1 Taxonomy of VDTT attentions
The attention sparsity has been briefly discussed in Section 1. We formally categorize attentions in VDiTs into three types:
Local attentions focus on short-range interactions without attending to distant tokens. They are primarily responsible for fine-gres fine-grained details during generation.
Long-range attentions distribute their attention scores across the entire sequence. These atten- tions mainly capture high-level semantic information. including coarse layout and motion. Minor perturbations, such as merging similar tokens, are acceptable.
Pivotal attentions maintain global perceptual field while simultaneously refining local details. Small perturbations to these attentions can result in noticeable quality degradation.
All three types of attention coexist in VDiTs and are not mutually exclusive during the sampling: local attention may transition into nonlocal attention as the diffusion process evolves, and vice versa. We will provide supporting evidence for this taxonomy through experimental results in Figure 9.
3.2 Sparse attentions
Sliding window for local attentions. Sliding window attention 53] proposes restricting
each query token at position i to attend to its lo- cal neighborhood within the range (i- w, i w). where w is the window size. It provides an ef- ficient alternative when the attention distribu- tion is highly localized. However, its zigzag- shaped attention mask introduces computation bubbles in block-wise kernel implementations (e.g., Flash [6. 7J). This problem be- comes more severe in tiree-dimensiona video data, where the number of computation bubbles
sliding window attention
rovvll Sera tile beyest
sliding attention
awneelange dlrppe
Figure 4: Illustration of converting sliding win- dow mask into a sliding tile mask. ID attention mask is shown for simplicity, with both the window size and tile size set to 2.
grows cubically. Zhang et al. [55] proposed sliding tile attention tackle this problem. The ID atten-

---

Bucketed Core-set Selection
splr info furkets
self attention
Bucketed Core-set Selection: the similar dropped
2
5
7
8
9
compune
5
dlop
simulority
Figure 5: Bucketed Core-set Selection (BCS). For clarity, the 2D image is used for illustration; the actual inputs and buckets operate on 3D video data within the latent space. In this example, the top k 4 tokens from each bucket are dropped.
tion mask is defined as: M = {m.,} } = {j E (T(1) w. 7() w]], where r(i) [i/t| -t+ [t/2] the tile center, and the tile size. It can be interpreted as shifting the window of non-center queries to align with the query in the center of the tile, as illustrated in Figure 4. The attention mask for sliding tile attention is block-wise dense and offers greater hardware efficiency. Our implementation employs 3D sliding tile attention to model local interactions of quality attentions with detailed pseudo code Appendix A.I.
Core-set selection for long-range attentions. is observed that many attentions with large perceptual fields are less sensitive to perturbations [40], corresponding to the long-range attentions defined earlier. Based on this inherent token-level sparsity. we introduce core-set attention to drastically accelerate computation by actively pruning redundant tokens during the attention operation. A prototype core-set attention operation is defined as follows:
coreset-attnl (H) = unpool o attn o pool (H),
(3)
where o denotes the composition operator, meaning the connected operators are applied sequentially from right to left. pool(.) and (-pool operations compress long sequence into compact core-set and recover the original sequence length, respectively.
Direct application of standard average pooling often results in significant degradation of generation quality (see Section 4.2), because its core assumption, that all tokens within the pooling kernel sufficiently similar, is often violated. As alternative, using Bucketed Core-set Selection (BCS), as outlined in Figure 5. introduces novel, buketed approach to achieve token-level sparsity. BCS first divides the tokens into buckets. Intra- Intra-bucket similarity is computed exclusively between a designated center token (token '5' in this example) and its neighboring tokens. Tokens with the top-k highest similarity to the center then pruned, and their representational information is merged into the center. Crucially, by eliminating inter-bucket similarity calculations, BCS achieves linear complexity (L) for sequence length L, offering significant computational advantage over ((L?) sequence length pruning methods [4, 40] while maintaining competitive performance. A detailed complexity analysis is provided in Appendix A.
implement the pool(-) operation
After applying the attention operation on the core-set tokens, we retrieve the original sequence length by scattering the center token back to the pruned tokens for the subsequent operations.
3.3 Signal-aware attention routing and detection
Attention router. Accurately identifying the optimal sparse attention variant is the remaining technical hurdle. We observe that attention behavior correlates strongly with the signal-to-noise ratio (SNR) of input features in Figure To address this, we introduce router implemented linear layer with diffusion timestep embedding T E Rd as input, following softmax activation to output the gate values (n) for dynamic attention variant selection:
a (m) = softmaxxTTTia
where ((n)) E Rdx3 denotes the linear projection matrix of the n-th transformer block. The gate values (n) la o02 0g (n) quantify the suitability of full, sliding-window, and core-set
5

---

condition embedd
fall atto. (beacher)
Tull atmn.
sliding atta.
cureset
condition
embed.
softmax
distill. loss
softmax
sliding
atta.
routing between attention branches
(b) attention pattern detection
Figure 6: Overview of VORTA. ID masks are used to represent the attention branches for simplicity. (a) During inference, the router takes the condition embedding as imput and selects the appropriate attention branch. In this example, it activates sliding attention. (b) During pattern detection, all attention branches are activated, and their outputs are aggregated using gate values from the router. The divergence between the full attention output and the routed output is used to update the router.
attention operations, respectively. This lightweight design adds just O. % to the model's parameters, while enabling the router to adaptively modulate gate values based on the current signal-to- -o-iiee ratio (SNR), without directly operating the full set of diffusion tokens.
Inference-time routing strategy. The router activates the attention branch with the highest gate value, as illustrated in Figure 6 (a). It is formalized as a hard selection, defined as follows:
sliding-altn(H'
if o12 (m)
(n) (n) and 02
(n) > 03
(n)
H H
coresct-attn (H)))
if a3 > di and 0g > 02
(n)
(n) (n)
(5)
attn(H(())
otherwise
where ())) represents the input features to the n-th transformer block. attn(-) is the standard full attention. sliding-attn(-) and coreset-attn(() represent the sliding and core-set attentions, respectively, defined in Section 3.2. Some operations, such RoPE and FFN, have been omitted for clarity. Activating sparse attention branches yields substantial speedup attention computation. Although the full attention branch is activated in fewer than 0.2% of cases, it remains important for preserving model performance. We will empirically investigate these design choices in Section 4.2.
Router optimization. Inspired by recent advances in language modeling [49], we adopt self- supervised optimization strategy, as illustrated in Figure (b). In the forward process, the gate values are employed to weight the outputs of the three branches:
(6) introduce a distillation loss Laistin, defined the MSE between the routed output HNN) and the original output (A) of the final block, to ensure that the routed outputs remain close to the original model outputs to preserve the pretrained performance. addition, we adopt the conditional flow matching loss LCFM from the VDiT pretraining stage detailed in Section 2. To promote sparsity in routing decisions, we apply an L.2 regularization term to gate values. The final loss function combines the conditional flow matching loss, the distillation loss, and the regularization term, weighted by hyperparameters Aisstill and Are. respectively:
(n++) a (x) attn H (n)) + 02 (n)
Bliding-attn(H"" + ag (w) corcsot-attn(H"))
LCFM + Adistill Eaistill + Areg Greg:
(7)
where Ldistill MSE
H (N)
H(M) and Creg = Σ l|a (n) 12
(8)
Notably, the original parameters of the VDiTs are always frozen, and only the router is updated. We set Adistill = 20 and Areg = 0.02 to balance the losses at a similar scale in our experiments. detailed analysis of these hyperparameters is provided in Appendix B.I.
4 Experiments
Baselines. This section presents the evaluation of the text-to-video generation task, We evaluate VORTA with two recently open-sourced VDiTs: HunyuanVideo [16] and Wan 2.1 [43]. Our

---

Table 1: Quantitative comparison under standard baseline settings (bf16, 720p. 5s).
caching;
step distillation;
sparse attention.
Models
Type
VBench
Quality
Semantic 1
LPIPS
Latency (8)
Speedup
Mem. (GB)
Hunyuan Video [16]
82.26
83,68
76.60
1043.85
1.00x
47.64
ARaR [40]
S
82.39
83.85
76.56
0.211
790.55
1.32x
78.15
STA [55]
82.33
83.56
77.39
0.201
676.39
1.54x
51.79
PAB [60]
82.40
83.80
76.81
0.186
815.51
1.28x
> 80
VORTA
S
82.59
83,74
77.95
0.185
594.23
1.76x
51.15
VORTA PAB
82.56
83.60
78.38
0.195
444.19
2.35x
PCD [44]
81.17
82.56
75.35
0.564
125.98
8.29
47.64
VORTA & PCD
&
81,49
82,78
76.31
0.575
72.46
14,41
51.15
Wan 2.1 (14B)
82,36
83,05
79.60
1304.82
1.00x
41,77
VORTA
82.85
83.45
80.44
0.222
856.50
1.52
43.97
comparisons include sparse attention acceleration methods: STA [55], which adopts a predefined sparse attention pattern, and ARnR [40], which performs online profiling to determine the sparse attention pattern dynamically. We also compare VORTA against two orthogonal approaches: the caching-based method PAB [60] and the step distillation method PCD [44]. Additionally, we report results for VORTA combined with PAB and PCD, as these methods can be integrated. For the Wan 2.1 [43], we only compare the pretrained baseline and VORTA, since other methods have not yet released code to support this model.
Benchmarks and evaluation metries. Following prior works [40, 60], we evaluate on the standard VBench prompt suite [11], which contains over 900 text prompts across 16 dimensions. The primary performance metric is the aggregated VBench score. We also report the VBench quality and semantic subscores provide more detailed breakdown. To assess the deviation of generation from the pretrained models, we use LPIPS [57] as reference metric. For efficiency analysis, we measure video sampling latency of the VDiTs, excluding the time required for text encoding and VAE decoding. We report the relative speedup Alatency (latency + 1). Additionally, we record peak memory usage, which impacts practical deployment due hardware constraints.
Implementation. We implement VORTA in PyTorch [30], using FlexAttention kernel for sliding attention and FlashAttention [6. 7] kernel for all other attention operations. Video samples are generated in 50 steps, except for PCD [44], which uses 6 steps. The videos 5 seconds long at 720p resolution. Due to out-of of-momory issues with PAB [60] at this resolution, we adopt sequential
CPU offloading [42]. For fair comparison, the latency introduced by model loading and offloading excluded in the experiment results. For router optimization, we use the Mixkit dataset [20]. training for 100 steps with learning rate of 10-2 and batch size of 4. All experiments are conducted on H100 GPUs with 80GB of memory. Additional implementation details are provided in Appendix A.2.
4.1 Main results
Performance. Table 1 presents the quantitative evaluation results of the baseline methods and VORTA. The key observations are: All methods maintain high VBench scores, except PCD, which shows 1-point drop due to aggressive compression of generation steps. ii) VORTA, with and without PAB, ranks first and second in VBench score and LPIPS on Hunyuan Video, respectively. iii) The good performance of VORTA on both the MMDiT-
[9] Hunyuan Video and DiT-based [31] Wan 2.1 demonstrates its generalizability across diffusion backbones. Figures and 7 quantitatively depict the generations produced by ARnR [40], STA [55]. VORTA, VORTA & PAB [60], and VORTA & PCD [44] on Hunyuan Video. All variants maintain high visual quality with vivid appearance and coherent motion, consistent with the guantitative metrics. More qualitative comparisons and complete VBench dimensional scores are provided in Appendix B.4 and Appendix B.5, respectively.
Efficiency. Among the sparse attention approaches, VORTA achieves the highest efficiency on Hunyuan Video, with a 1.76x speedup. For the other line of work, PAB achieves 1.28x speedup on Hunyuan Video, but it requires over 1.7x additional memory, which makes less practical for large models or high-resolution videos. In practice, running PAB on an 80GB GPU necessitates sequential
7

---

Huayuan Video
avg. latency
original
ARnR
avg. lateney 710s
132% speedup
STA
avg. lalency 676s
154% speedup
VORTA
avg. lutency 594s
176% speedup
prompt: a, fbuuff white sheep ... leisurely strills through picturesque meadow.
prompt: Gwen Stacy is reading book
Figure 7: Qualitative comparison of sparse attention methods: ARnR [40], STA [55], and VORTA.
CPU offloading. The latency introduced by parameter loading and offloading nearly offsets the benefits of computation. The step distillation approach, PCD, achieves an 8.29 speedup using only 6 sampling steps, with mild drop performance. While these methods offer different trade-offss in efficiency and performance, both can be effectively integrated with VORTA. When combined with PAB and PCD, VORTA achieves total speedups of 2.35x and 14.41x x, respectively.
Table 2: Quantitative comparison on Wan 2.1 [43] with different sampling configurations.
VBench
Configuration
Latency (8)
(GB)
Subject 1
Consistency
Aesthetic 1
Imaging 1
Wan 2.1 (43)
UniPC (50 step) [59]
91.21
75.49
63.33
63.99
1359.76
41.77
VORTA
90.76
75.09
63.79
63.07
791.02
43.97
Wan 2.1 (14B) [43]
DPM++ step) [25]
89.72
74.54
64.43
63.99
817.00
41.77
VORTA
91.82
Generalization to various schedulers. Unlike sparsity-inducins techniques that rely on offline profiling or hand-crafted heuristics define sparse execution strategy [40, 55], VORTA designed to possess scheduler and step-count generalization, mirroring the flexibility of the baseline dense model. This obviates the need for costly re-profiling or hyperparameter re-tuning when changing sampling configurations. We demonstrate this capability by comparing the performance the Wan 2.1 [43] using both the 50-step UniPC [59] and 30-step DPM++ [25] schedulers.
Table 2 presents the VBench dimensions [11] and efficiency metrics from these experiments, which were conducted B200 GPUs. The consistent, lossless performance of VORTA relative the dense model confirms its ability to generalize efficiently without added cost, an advantage enabled by its lightweight training, which heuristic-based methods like STA [55] and ARnR [40] do not achieve.
Runtime breakdown. Figure 8 further presents the average latency of individual components for the sparse attention methods. Here, "attn." denotes the isolated attention operation latency, whereas "attn. related" includes associated operations such as projections, RoPE, layer normalization, and other computations within the attention module.
Compared to ARnR [40], VORTA demonstrates superior efficiency. To preserve lossless generation, ARnR adopts a more conservative sparsity, resulting in higher latency in its "attn." component. Furthermore, its ((L)) similarity computation introduces additional latency, as evidenced by larger latency in its "attn. related" component. In contrast, VORTA adaptively identifies efficient sparse patterns and allows greater speedups. Its core-set attention also involves similarity computation; however, BCS has only (LL complexity, incurring negligible additional cost.

---

VORTA also outperforms STA [55], which uses predefined sliding attention pattern. The main bottleneck of STA occurs during the first 15 sampling steps (see bottom two bars), when it must expand the window size to capture long-range dependencies. Consequently, these steps do not benefit from attention sparsity. In contrast, VORTA adaptively routes attention branches to accommodate diverse interaction types while maintaining near-constant latency across all diffusion steps.
related
fin
other
Table 3: Evaluation of each VORTA component
on Wan 2.1 (1.3B) [43].
Models
VBecch f
Latency (s)
Speedup
amr
Wan 2.1 (1.3B)
81.20
73.24
1.00x
0urs
Sliding Attn.
80.25
65.14
1.12x
wio Coreset Atm.
79.89 89
66.10
1.11x
(1-15)
w/o Full Attn.
77
$9.34
1.23x
(1-15)
ours
Timestep Cond.
65.00
1.13x
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5 20.0
w/ AP
77.08
57.53
1.27
latency per sampling stop
w/ AP
76.01
$7.64
1.27 x
Figure 8: Runtime breakdown for sparse atten- tion methods on Hunyuan Video [16].
w/ AP12
75.94
57.55
1.27
VORTA
81.06
58.42
1.25
4.2 Ablation study
We conduct ablation studies the Wan 2.1 (1.3B) [43] at its pretrained 480p resolution. All other experimental settings are identical to those used in the main experiment. At 480p, attention operation accounts for smaller portion of the overall latency compared to 720p, resulting in less pronounced acceleration. Nevertheless, we adopt this smaller model and resolution to reduce the computational cost of benchmarking. The results are summarized in Table 3.
Attention branches. We evaluate the contribution of the attention branches individually by remov- ing them. When the sliding attention branch is disabled, the router shifts more selections toward the full attention expert to preserve performance, resulting in an over 10% runtime increase, despite applying the same regularization level. Similarly, removing the core-set attention branch increases the latency by 14%. These results highlight the importance of both sparse branches in improving efficiency through the adaptive selection of attention patterns tailored specific characteristics.
Given that the router selects the full attention branch only 0.2% cases in VORTA (will be detailed in Section 4.3), assess whether retaining is necessary. As shown in Table 3, the removal of the full attention branch leads to a 4-point drop in the VBench score without any additional speedup. This suggests that the full attention branch remains crucial and validates the existence of the pivotal attention patterns defined in Section 3.1.
Timestep condition for signal-aware routing. We also examine the impact of including timestep information in attention routing. Removing the timestep embedding from the router input results in uniform attention branch selection across all diffusion steps, leading to slower 12%. In the absence of timestep conditioning, the router consistently selects the same attention expert. To maintain performance, the model predominantly routes the full attention expert.
Bucketed core-set selection (BCS). We evaluate how much BCS improves performance compared to conventional average pooling. BCS sets the core-set size 50% of the original sequence length. To ensure a fair comparison, we apply the same reduction ratio in the average pooling cases by configuring the pooling kernel sizes such that their product equals compression along different dimensions, we evaluate three pooling kernel configurations: (2, 1, 1), (1, 2, 1). and (1, 1,2), denoted as AP211, AP121, and AP112. respectively. Although average pooling offers slightly lower latency by being free from similarity computations, its performance drops significantly. When adjacent tokens differ significantly, naive merging causes information loss, often resulting in artifacts such as pixelation or blurring in the generation.
To isolate the effect of

---

4.3 Attention patterns in VDiTs.
attn. variant
Figure illlsstrtees the routed attention branch assigned to each head, layer, and sampling step. reveals a clear temporal pattern: earlier time steps tend to use more corset attention, while later time steps increasingly rely on sliding at- tention. Only small fraction (about 0.2%) of attention heads are assigned to the full attention branch, highlighting the sparsity. As intuitively expected, earlier time steps likely focus on con- structing high-level semantics rather than fine- grained details, whereas later time steps empha- size local interactions. Unlike auto-regressive or discriminative models [8, 13], VDiTs ex- hibit weaker layer-wise specialization; attention heads within the same layer are more evenly distributed across branches. A minor tendency is observed in later time steps where sliding attention is more frequently assigned inter- mediate layers, while corset attention is more often assigned the initial and final layers. One plausible explanation is that intermediate layers capture local details, while subsequent layers re- fine the overall representation quality. Step-wise visualizations and results for other models are provided in Appendix B.3.
sliding
coreset
full
45
35
sampling step
25
15
5
30
head index
20
10 layer index
20
30
10
Figure Optimized gate values for Wan 2.1 (14B) [43]. We visualize snapshots at sampling steps 5, 15, 25, 35, and 45 out total 50 steps.
5 Conclusion
In this work, we presented VORTA, an efficient and generalized framework for accelerating diffusion transformers in video generation. VORTA reduces attention overhead by dynamically identifying attention patterns and routing them through appropriate sparse attention mechanisms. To further enhance efficiency, modeling of long-range dependencies. VORTA achieves a 1.76x end-to-end speedup without com- promising generation quality. Moreover, its high compatibility with existing acceleration techniques enables a combined speedup of up to 14.41 x. We believe VORTA offers practical and extensible solution, paving the way for broader adoption and future research in video generation.
introduced bucketed coreset selection (BCS) strategy that improves the
Limitations. VORTA targets the attention mechanism, which accounts for over 75% of the total computation in high-resolution video generation. However, for tasks with short sequence lengths, such as image or low-resolution video generation, the attention overhead becomes less dominant. As result, the potential acceleration achievable limited. Additionally, this work focuses the bidirectional generation paradigm. Other paradigms, such as autoregressive generation, are not directly supported and may require substantial adaptation. Additional discussion on failure cases, limitations, and border impacts is provided in Appendix C.
Acknowledgments and Disclosure of Funding
This project supported by the National Research Foundation, Singapore, under NRF Professor- ship Award No. NRF-P2024-001
10

---

References
[1] Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. CoRR. abs/2004.05150, 2020.
Emmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. CoRR, abs/1511 1..11..000 106297, 2015.
Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, abs/ 308.3432, 2013.
[4] Daniel Bolya and Judy Hoffman. Token merging for fast stable diffusion. In CVPR Workshops. IEEE, 2023.
[5] Tian Qi Chen, Yulia Rubanova, Jesse Beltencourt, and David Duvenaud. Neural ordinary differential equations. In NeurlPS, 2018.
[6] Tri Dao. Flashattention-22 22 Faster attention with better parallelism and work partitioning. ICIR, 2024.
[7] Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with to-awaressss. In NeurlPS, 2022.
[8] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In ICLR, 2024.
[9] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In ICML. PMLR, 2024.
[10] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
[11] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Champaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Vbench: Comprehensive benchmark suite for video generative models. In CVPR. IEEE, 2024.
[12] Haobo Jiang, Mathicu Salzmann, Zheng Dang, Jin Xie, and Jian Yang. SE(3) diffusion model- point cloud registration for robust 6d object pose estimation. In NeurlPS, 2023.
[13] Huiqiang Jiang. Yucheng Li. Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir Abdi, Dongsheng Li, Chin- Yew Lin, Yuqing Yang, and Lili Qiu. Minference 1.0: Accelcrating pre-filling for long-context Iims via dynamic sparse attention. NeurlPS, 2024.
[14] Kumara Kahatapitiya, Haozhe Liu, Sen He, Ding Liu, Menglin Jia, Michael S Ryoo, and Tian Xie. Adaptive caching for faster video generation with diffusion transformers. CoRR, abs/2411. 2024.
[15] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
[16] Weijie Kong. Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Bo Wu, Jianwei Zhang, Kathrina Wu, et al. Hunyuanvideo: A systematic framework for large video generative models. CoRR, abs/2412.03603, 2024.
[17] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Selective attention improves transformer. CoRR, abs/2410.02703, 2024.
[18] Hengjia Li, Haonan Qiu, Shiwei Zhang, Xiang Wang, Yujie Wei, Zekun Li, Yingya Zhang, Boxi Wu, and Deng Cai. Personalvideo: High id-fidelity video customization without dynamic and semantic degradation. CoRR, abs/2411.1700 17048. 2024.
[19] Hengjia Li, Lifan Jiang, Xi Xiao, Tianyang Wang, Hongwei Yi, Boxi Wu, and Deng Cai. Magicid: Hybrid preference optimization for id-consistent and dynamic-preserved video customization. CoRR, abs/2503. 12899, 2025.
[20] Bin Lin, Yunyang Ge, Xinhua Cheng, Zongjian Li, Bin Zhu, Shaodong Wang, Xianyi He, Yang Yc, Shenghai Yuan, Liuhan Chen, Tanghui Jia, Junwu Zhang, Zhenyu Tang, Yatian Pang, Bin She, Cen Yan, Zhiheng Hu, Xiaoyi Dong, Lin Chen, Zhang Pan, Xing Zhou, Shaoling Dong, Yonghong Tian, and Li Yuan. Open-sora plan: Open-source large video generation model. CoRR, abs/2412.00i31,, 2024.
[21] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In ICLR, 2023.

---

[22] Feng Liu, Shiwei Zhang, Xiaofeng Wang, Yujie Wei, Haonan Qiu, Yuzhong Zhao. Yingya Zhang, Qixiang Ye, and Fang Wan. Timestep embedding tells: It's time cache for video diffusion model. CoRR, abs/2411.19108. 2024.
[23] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. ICIR, 2023.
[24] Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, and Qiang Liu, Instaflow: One step is enough for high-quality diffusion-based text-to-image generation. In ICLR, 2024.
[25] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of difflusion probabilistic models, Mach. Intell. Res., 22(4):730-751) 751, 2025.
[26] Zhengyao Lv. Chenyang Si, Junhao Song, Zhenyu Yang, Yu Qiao, Ziwei Liu, and Kwan- Yee K. Wong. Fastercache: Training- free video diffusion model acceleration with high quality, CoRR, abs/2410.1953 2024.
[27] Mason McGill and Pietro Perona. Deciding how to decide: Dynamic routing artificial neural networks. In ICMI,, volume 70 Proceedings of Machine Learning Research, pages 2363-2372. PMIR, 2017.
[28] Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik P. Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In CVPR. IEEE, 2023.
[29]
James O' N'Neil.. An overview of neural network compression. CoRR, abs/2006.03665 2020.
[30]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang. Zachary Vito,o, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjic Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, 2019.
[31] William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV. IEEE, 2023.
[32] Liang Peng, Boxi Wu, Haoran Cheng, Yibo Zhao, and Xiaofei He. Sudo: Enhancing text-to-image diffusion models with supf-sup direct preference optimization. CoRR, abs/2504, 145344 2025.
[33] Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching-Yan Yao Chuang, David Yan, et al. Movie gen: A cast of media foundation models. CoRR, abs/2410.13320, 2024.
[34] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR. IEEE, 2022.
[35] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In ICLR, 2022.
[36] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. In ECCV. Springer, 2024.
[37] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In ICLR. OpenReview.net, 2017.
[38] Yang Song, Jascha Sohl-Dit Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
[39] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In ICML. PMLR, 2023.
[40] Wenhao Sun, Rong-Cheng Tu, Jingyi Liao, Zhao Jin, and Dacheng Tao. Asymmr: Video diffusion transformers acceleration with asymmetric reduction and restoration. CoRR, abs/2412.11706, 2024.
[41] Andreas Veit and Serge J. Belongie. Convolutional networks with adaptive inference graphs. IJCV, 128(3): 730-741, 2020.
[42] Patrick von Platen. Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj. Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: Sfate-of-the diffusion models. https://gitths com
wwt.:/owoo/ggnnneeeeeeeee hhw.hhhggggnggene hwt.c/////fffre/////eeeeoeo 2022.
12

---

[43] Ang Wang. Baole Ai, Bin Wen, Chaojie Mao. Chen- Wei Xie. Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, al. Wan: Open and advanced large-scale video generative models. CoRR, abs/2503.20314. 2025.
[44] Fu- Yun Wang. Zhaoyang Huang. Alexander William Bergman, Dazhong Shen, Peng Gao, Michael Lingelbach, Keqiang Sun. Weikang Bian, Guanglu Song, Yu Liu, Xiaogang Wang, and Hongsheng Li. Phased consistency models. In NewrlPS,
[45] Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S. Davis, Kristen Grauman, and Rogério Schmidt Feris. Blockdrop: Dynamic inference paths in residual networks. In CVPR, pages 8817 8826. Computer Vision Foundation/ / JEEE Computer Society, 2018.
[46] Haocheng Xi. Shuo Yang, Yilong Zhao, Chenfeng Xu, Muyang Li, Xiuyu Li, Yujun Lin, Han Cai. Jintao Zhang, Dacheng Li, Jianfei Chen, lon Stoica, Kurt Keutzer, and Song Han. Sparse videogen: Accelerating video diffusion transformers with spatial-temporal sparsity. CoRR, abs/2502.01776, 2025.
[47] Yifei Xia. Suhan Ling, Fangcheng Fu, Yujie Wang, Huixia Li, Xuefeng Xiao, and Bin Cui. Training- and adaptive sparse attention for efficient long video generation. CoRR, abs/2502.210799 2025.
[48] Guangxuan Xiao, Ji Lin, Mickaël Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post post-training quantization for large language models. In ICML. PMLR, 2023.
[49] Guangxuan Xiao, Jiaming Tang. Jingwei Zuo, Junxian Guo, Shang Yang. Haotian Tang, Yao Fu, and Song Han. Duoattention: Efficient long-context LLM inference with retrieval and streaming heads. CoRR, abs/2410. 10819, 2024.
[50] Guangxuan Xiao, Tianwei Yin, William T. Freeman, Frédo Durand, and Song Han. Fastcomposer: Tuning-free free multi- subject image generation with localized attention. IJCV, 2025.
[51] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang. Jiazheng Xu, Yuanming Yang. Wenyi Hong, Xiaohan Zhang, Guanyu Feng. Da Yin, Xiaotao Gu, Yuxuan Zhang, Weihan Wang, Yean Cheng, Ting Liu, Bin Xu, Yuxiao Dong, and Jie Tang. Cogvideox: Text-to-videe diffusion models with an expert transformer. CoRR, abs/2408.0607 06072, 2024.
[52] Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Frédo Durand, William T. Freeman, and Tacsung Park. Onc-step diffusion with distribution matching distillation. In CVPR. IEEE, 2024.
[53] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big hird: Transformers for longer sequences. In NeurlPS, 2020.
[54] Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia Wei, Haocheng Xi, Jun Zhu, and Jianfei Chen. Spargeattn: Accurate sparse attention accelerating any model inference. CoRR, abs/2502.181 18137, 2025.
[55] Peiyuan Zhang, Yongqi Chen, Runlong Su, Hangliang Ding, Ion Stoica, Zhenghong Liu, and Hao Zhang. Fast video generation with sliding tile attention. CoRR, abs/2502.04507 2025.
[56] Peiyuan Zhang, Runlong Su. Yongqi Chen, Hangliang Ding, William Lin, Kevin Lin, Wei Zhou, You Zhou, Yuzhou Nie, and Hao Zhang, Fastvideo a lightweight framework for accelerating large video diffusion models. https: // //tp://gttttttt h//gghttuu.... com/hao-aa al wat./aaaaa/aaaaaaaa 2025.
[57] Richard Zhang, Phillip Isola, Alexii A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR. IEEE, 2018
[58] Wentian Zhang. Haozhe Liu, Jinheng Xie, Francesco Faccio, Mike Zheng Shou, and Jürgen Schmidhuber. Cross-attention makes inference cumbersome in text-to-image diffusion models. CoRR, abs/2404.027477, 2024.
[59] Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipe: A unified predictor-corrector framework for fast sampling of diffusion models. In NeurlPS, 2023.
[60] Xuanlei Zhao, Xiaolong Jin, Kai Wang, and Yang You. Real-time video generation with pyramid attention broadcast. CORR, abs/2408. 12588, 2024.
[61] Yibo Zhao, Liang Peng. Yang Yang, Zekai Luo, Hengjia Li, Yao Chen, Zheng Yang. Xiaofei He, Wei Zhao, Qinglin Lu, Wei Liu, and Boxi Wu. Local conditional controlling for text-to-image diffusion models. In AAAI, pages 10492-10500. AAAT Press, 2025.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The technical contributions highlighted in the abstract and introduction as follows: (1) VORTA, an end-to-end acceleration framework for Vision DiTs (VDiTs), and (2) Coreset Attention, sparse attention mechanism designed for efficient modeling long-range dependencies. Both contributions are thoroughly detailed and empirically validated in the manuscript.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations of the current method are discussed in the main manuscript and will further elaborated in the appendix and supplementary materials. These materials include suitable application scenarios, compatible methods, and failure cases.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
14

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include formal results such as lemmas, theorems, corollar- ies, proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proofss provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The experimental setup and implementation details are disclosed in the main manuscript. Additional code and supplementary information will provided the appendix to facilitate reproducibility. However, even with identical software environments and fixed random seeds, minor variations results may occur due to differences in hardware conditions.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
15

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: As previously noted, the code and open-sourced dependencies (e.g., model weights) will be included in the appendix to support reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips.co cttpps.ccppbbicc//// /ww.//gi.se////eg/// Cooeeubmi ssiompolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental settings, datasets, pretrained model weights, and related implementation details are provided in the main manuscript and supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The primary evaluation metric in this study, running latency, remains consistent across repeated runs. Minor variations may arise from hardware-level factors, which are difficult to isolate with our available computational resources. The benchmark requires over
16

---

1,000 H100 GPU hours per run and is widely accepted metric for the target task. Due to the exceptionally high computational cost, performing statistical significance tests is not feasible.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative eτror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The experimental settings, datasets, pretrained model weights, and related implementation details are provided in the main manuscript and supplementary materials.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: / /ntrrinur....// wwtppcccc/ccceeccccccccccccccc wtw.:i/citthiiiuuieeeeee csGuidellldlllieeeec lines?
Answer: [Yes]
Justification:
We affirm compliance with the NeurIPS Code of Ethics after thorough review.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
17

---

Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
We provide thorough analysis and claim the potential societal impacts in the appendix.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our acceleration method is not directly related to the risks of misuse. Refer- ences to safeguards for compatible models will be included in the code if available.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: assets (e.g., code, data, models) used in this paper comply with their respective licenses and are credited.
18

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The new code and model weights are released under the CC-BY 4.0 license and are accompanied by a README file included the supplementary materials.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
19

---

Answer:
[NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not non involve LLMs as any important, original, non-stane non-standard components.
Please refer to our LLM policy (https / /tt////eripp////// ccC.onferrences/222555 for what should or should not be described.
20

---

VORTA: Efficient Video Diffusion via Routing Sparse Attention
Appendix
Table of Contents
Methodological and implementation details:
Appendix A.I: Methodological details
Appendix A.2: Implementation details
Additional experimental results and findings:
Appendix B.1: Hyperparameters analysis
Appendix B.2: Runtime analysis
Appendix B.3:
Attention pattern analysis
Appendix B.4: Qualitative comparison
Appendix B.5: VBench dimensional scores
Appendix B.6: Comparison with SVG
Limitations and border impact:
Appendix C.I: Failure cases
Appendix C.2: Border impact
A Methodology and implementation details
A.1 Methodological details
Sliding tile attention. Section 3.2 introduced the sliding tile attention [55], the sparse attention pattern we used to model local interactions. To keep the main text concise, we explained only the 1D case. Here, we present the 3D version of the sliding tile attention mask used our implementation, as detailed in Algorithm 1.
Bucketed coreset selection (BCS). Section 3.2 also introduced the bucketed coreset selection (BCS) method, designed to reduce the computational overhead of modeling long-range interactions. In the naive setting, computing pairwise similarities across an input sequence of length L incurs quadratic complexity of (1.2). Bolya and Hoffman [4], Sun et al. [40] select a subset of tokens (e.g., 25%) 25%) as anchors and computing similarities between these anchors and the remaining tokens. This reduces the number of comparisons but still results in ((3L/4) (I/4)) (IL)) complexity.
In contrast, BCS achieves linear complexity ((L) by employing a bucketing strategy. Each bucket, of size (t, h, w), computes similarities between central token and the remaining (thw - 1) tokens, yielding per-bucket cost of (thw). With (t(ww) such buckets, the total cost becomes (((L/(thw) thw) OLL). No inter-bucket comparisons are performed, and the emperical results in Section 4.1 show that this approach is effective enough in selecting representative subset of tokens for long-range interactions. Compared to global pairwise methods [4, 40), which require (I.2) operations, BCS offers a substantially more efficient (T.) alternative.
A.2 Implementation details
Implementation details for our VORTA. Besides the implementation introduced in Section 4, we also provide the implementation details for VORTA model.
For router optimization, we train on the Mixkit dataset [20] for 100 steps using learning rate of 10-2 and a batch size of 4. Training completes in approximately one day using two H100 GPUs.
21

---

Algorithm 1 3D sliding tiled attention mask
Input: video size (F, H, W), tile size t = (Ir, Ln, Iw), window size w (WF, WH, ww).
1: // Total sequence length for self-attention
2: L 4 F x H W
3: M < -LLXL
4: // The attention mask between the i-th query and the j-th key
5: for + to L do
6:
for j to L do
7:
If Get tile id from token
8:
QF-9H.90 RW
get_tiile tile id 3d(i, v,t)
9:
kr, kHr kiw < get_lile id_ddjj, v,t)
10:
II All queries within the same window share the same tile id as the window center tile id gF, qu, qW <- gee__eeiidoo window feww__eett_rddr _id((rd( , Qu qw, w) Il true if key file is within the local window query tile
11:
12:
13:
m + bool(abs(()) and bool(abs(gH kH) S WH (H//2) (2)
- kF) S w2/2)
m.
15:
m f- m and bool(abs((
kw) S wW /2)
I/ Save the mask
17:
M [] t m
end for
end for Output: sliding tile attention mask M.
19:
The sliding attention branch employs a window size (18, 27, 24), while the coreset attention branch uses a bucket size of (2, 2) with coreset ratio of Teore 0.5.
Regarding video configurations during both training and inference, Hunyuan Video [16] utilizes videos with 117 frames at a resolution of 720 x 1280 (720p), while Wan 2.1 [43] uses 77-frame videos the same resolution. For rendering, Hunyuan Video outputs videos at 24 frames per second (FPS), whereas Wan 2. generates videos at 15 FPS, as specified in their respective repositories.
Implementation details for baseline methods. To evaluate efficiency, the PAB [60] is tested on 720p videos from Hunyuan Video [16], aligning with the setup used in other methods. However, processing at this resolution with PAB requires over 80 GB of GPU memory. To address this limitation, we apply sequential CPU offloading [42]. For a fair comparison, we exclude the latency caused by model loading and offloading from the reported results. Despite this, the end-to-end runtime with CPU offloading exceeds 2000 seconds per video, which substantially slower than the original pretrained model and offers practical efficiency advantage. The original STA kernel supports video generation only at a fixed resolution of 768 x 1280, which exceeds the 720p resolution used by the pretrained model. To ensure consistency in evaluation, reimplemented the kernel using FlexAttention to support 720p video generation.
Evaluation metrics. To evaluate how the generated outputs differ from those of pretrained models, use LPIPS [57] reference metric. However, since the pretrained models do not represent the ground truth, divergence from them does not necessarily indicate degraded performance. Multiple generated outputs may be equally valid, provided they align with the input prompts. LPIPS only used as a comparative reference. The primary evaluation of video quality and prompt alignment should be based on VBench [11] scores.
The improved performance acceleration methods. Interestingly, despite using less computation, VORTA slightly outperforms the original pretrained models in terms of VBench score. Similar findings have been reported other acceleration methods [17, 48). A possible explanation lies in the redundancies present in overparameterizeed models, which can introduce marginal negative effects. By pruning these redundancies, these acceleration methods may contribute to slight performance gains, although they are not intended for this purpose.
22

---

B
Additional experimental results and findings
B.1 Hyperparameters analysis
Wan 2.1 (1.3B)
avg. latency 73.24s
Areg 0.01
avg. latency 67.30s
Areg = 0.02
avg. latency 58.42s
Xreg
0.05
avg. latency 58.06s
prompt: a fluffy orange tabby cat eating from a ceramic bowl decorated with fish patterns...
Figure 10: Qualitative evaluation of varying the regularization weight Areg-
Figure 10 shows the effect of varying the regularization weight A When Areg is small (Aeg = 0.01), the speedup is limited, and most scenarios, the router tends to select full attention. In contrast, with a large Are 0.05, the video exhibits noticeable distortion (e.g.. the cat's head in the final frame). A moderate value of Areg 0.02 achieves a good trade-off between acceleration and output quality.
Figure 11 provides further analysis of alternative pooling strategies in the coreset attention branch. Using average pooling with Feore 50% coreset ratio significantly underperforms compared to BCS pooling, primarily because lacks a selection mechanism. As the coreset ratio Toor increases, the VBench score improves; however, ratio of r'ore 50% is sufficient to achieve strong performance. Higher ratios yield marginal performance gains while introducing additional computational overhead during generation.
23

---

82
C-07
80
Pooling
--03
Beseine the pooting)
76
11.2:3)
(0.1.2)
VORKA BCS toursi
VORLA AVG
55.0
57.5
60.0
62.5
65.0
67.5
70.0 72.5
latency (s)
Figure 11: Effect of varying the coreset size BCS and the kernel size average pooling.
B.2
Runtime analysis
20
30
sampling step
Figure 12: Per-step sampling latency on Hunyuan Video [16]. ARnR [40] yields smaller speedups overall. STA [55] shows no speedup in early timesteps.
Section 4.1 analyzed the average runtime (over diffusion steps) of each component, as shown in Figure 8. Figure 12 further presents the runtime breakdown per diffusion step. Overall, ARnR exhibits relatively high runtime, primarily due to limited acceleration attention operations. Additionally, its periodic global similarity computation, with O(L²) complexity, introduces bottleneck. This is evident from latency spikes every five steps. contrast, STA shows significantly higher runtime during the initial 15 steps due the lack of early-stage acceleration, leading reduced overall efficiency. Our VORTA achieves near-constant runtime across steps, demonstrating stable and efficient performance.
B.3 Attention pattern analysis
Figures 13 and 14 show the router gate values for Hunyuan [16] and Wan 2.1 (14B) [43], respectively, supplementary results for Section 3.1.
B.4
Qualitative comparison
In addition to the qualitative results in Figures and 7 and the quantitative results in Table 1, we present further visualizations Wan 2.1 (14B) in Figure 15. VORTA inherits the strong performance of the pretrained ViTs while offering significant speedup.
B.5 VBench dimensional scores
VBench evaluates the generated videos across 16 dimensions. Due to space constraints, we report only three aggregated scores in Table with the complete set of scores provided in Table 4 for completeness.
B.6 Comparison with SVG
evaluate Spare Video Gen (SVG) [46] on a B200 GPU and compare with our VORTA, as shown in Table 5. The comparison covers four VBench dimensions for quality and latency, well as peak memory usage for efficiency. Since both SVG and VORTA focus on attention sparsity, minor non-attention optimizations (e. ooperator fusion and quantization) are disabled in SVG to ensure

---

Figure 13: Optimized gate values for Hunyuan Video [16]. The 50 sampling steps are grouped into equal intervals, and the averaged gate value within each interval is reported. Each color corresponds distinct attention branch, with color intensity indicating confidence.
fair comparison. We define attention sparsity the ratio of skipped query-key token multiplications to the total number of query-key multiplications in standard dense attention. A higher sparsity indicates better theoretical computational efficiency.
For both Hunyuan and Wan 2.1 models, VBench scores are statistically indistinguishable, indicating significant differences in quality. However, VORTA consistently outperforms SVG in latency and sparsity metrics. Additionally, SVG incurs substantially higher memory usage due to its reliance on online profiling, which limits its applicability in resource-constrained environments.
C Limitations and border impact
C.1 Failure cases
VORTA does modify the pretrained parameters of VDiTs, and therefore its performance inherently depends on the quality of the underlying pretrained model. As illustrated in Figure 16, when the pretrained VDiTs fail generate high-quality videos, resulting in distortions or non-physical outputs, VORTA exhibits similar deficiencies. In some cases, it even produces outputs of lower quality than the original VDiTs. This degradation is attributed computations on erroneous generations, which amplify distortions in the resulting videos.
C.2 Border impact
Generative models pose risks of producing biased, privacy-invasive, harmful content. While method accelerates video generation, it may also propagate such risks. It is imperative that researchers, developers, and platform providers actively assess and mitigate these potential harms to promote responsible use.
25

---

sampling step 1-10 10
samping Step 22
samping step 21
30
35
Tayer Indox
Sycr Iriex
Ryyr Triex
samping step 31-40
sampling Stap 41
10
altn. siiing cortee
ang
corsset
Arl
Tayer Indox
byer Triex
Figure 14: Optimized gate values for Wan 2.1 (14B) [43].
Wan 2.1 (14B)
avg. latency 1305s
VORTA
avg. latency 857s
prompt: black cat lounges sunlit poolside deck, wearing stylish, tiny sunglasses...
prompt: couple stands park during outumn their eyes close they share tender kiss avg. latency 1305s
O
Wan 2.1 (14B)
VORTA
avg. latency 857s
prompt:... the bieycle comes to halt, fallen leaves crunch sofily under the tires
prompt: rolling large snowhall to form the base
Figure 15: Qualitative comparison Wan 2.1 (14B) [43].
26

---

Table 4: Quantitative results for VBench dimensions [11] for Hunyuan Video [16] and Wan (14B)
[43].
Method
Aesthetic Quality
Appearance Style
Background Consistency
Dynamic Degree
Human Action
Imaging Quality
Motion Smoothness
HunyuanVides
62.05
77.88
93.80
91.35
38.89
96.00
63.33
96.98
ARnR
62.58
80.36
93.99
87.54
39.77
93.73
63.21
96.12
+ STA
63.87
79.48
94.47
86.39
39.58
97.00
61.92
96.79
VORTA
62.33
80.63
94.62
92.13
37.50
97.00
63.16
95,84
VORTA & PAB
63.16
80.43
95.22
90.92
36.31
97.50
62.05
96.18
+CC
62.69
76.21
94.44
85.94
31,94
94.00
63.08
96.46
VORTA & PCD
61.43
76.92
94.87
89.33
35.42
98.00
62.42
95.11
Wan 2.1 (14B)
63.33
82.12
94.55
83.26
37.50
99.00
63.99
91.60
VORTA
63.79
83.32
95.28
87.04
39.58
100.00
63.07
92.17
Mutliple Objects
Objects Class
Overall Consistency
Spatial Relationship
Subject Consistency
Temporal Flickering
Temporal
Method
Scene
Style 1
HlanyuanVidov
52.21
84.41
74.30
90,30
98.57
ARnR
52.09
87.49
69.88 88.88
69.21
80.89
90.55
99.28
70.70
STA
56.55
87.82
75.01
64.08
60
88.12
98,40
PAB
56.86
84.97
74.97
63.91
78. 34
90.89
98.61
70.50
VORTA
58,00
89.56
74.62
61.87
92.43
98.47
VORTA PAB
59.17
89.90
75.82
62.99
77.43
92.27
97.96
70.29
PCD
54.04
81.88
75.07
66.03
74.48
90.64
97.37
70.52
VORTA & PCD
51.07
85.05
74.93
67.09
73,70
91.34
97.50
70.69
Wan (14R)
74.62
86.47
75.49
64.70
79.16
91.21
97.69
71.54
VORTA
75.76
88.45
75.09
63.29
80.28
90.76
97.78
70.70
Table 5: Quantitative comparison with SVG [46].
Models
Sparsity (%)
VBench
Lateney
Mem. (GB)
Subject
Consistency
Aesthetic
Imaging
Hunyuan Video [16]
0.00
90,30
74.30
62.05
63.33
1224.22
47.6
+ SVG [46]
80.00
90.59
75.11
62.73
63.17
664
71.38
VORTA
82.65
92.43
74.62
62.33
63.16
568.74
51.15
Wan 2.1 (14B) [43]
0.00
91.21
75.49
63.33
63,99
1359.76
41.77
SVG [46]
75.00
90.42
74.96
63.53
64.45
921.50
65.01
VORTA
81.68
90.76
75.09
63.79
63.07
791.02
43.97
Hunyuan Video
avg. lutency 1044s
VORTA
avg. latency 894s
prompt: as the bicycle comes to fallen leaves crunch softly under the tires
prompt:
horse galloping open field......
issue: unexpected fragment segmentation and distortion
issue:
violation of physical
Figure 16: Failure cases. When the pretrained model exhibits distortions or non non-physical phenomena, VORTA inherits these issues. We refer the reader to the supplementary video for more illustrative example.
27


Reviews:
[{"review_id": "NET1Cn9YKI", "reviewer": "Reviewer_EhBo", "summary": "VORTA accelerates video diffusion models by dynamically selecting and routing sparse attention mechanisms, improving efficiency with minimal impact on generation quality. It employs coreset selection to reduce attention computation and uses a router to adaptively choose the most suitable attention branch, achieving up to 1.76x end-to-end speedup.", "questions": "Please refer to the weakness.", "limitations": "yes", "rating": 5, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**:\n1. The paper addresses a critical issue in video generation, the computational inefficiency of Video Diffusion Transformers. By introducing VORTA, it significantly reduces the sampling time, making VDiTs more practical for real-world applications.\n2. The paper presents a novel coreset attention mechanism and a signal-aware routing strategy that dynamically selects the most appropriate sparse attention variant. The bucketed coreset selection (BCS) strategy is an innovative way to handle long-range dependencies efficiently.\n3. The experimental results are convincing, demonstrating the effectiveness of VORTA in improving efficiency while maintaining generation quality. The ablation studies provide insights into the importance of each component and the impact of different design choices.\n\n**Weakness**:\n1. The demonstration of the proposed corset selection and router strategy is not clear enough, could you please complement more illustration or description?", "was_revised": true, "final_justification": "Due to the interest scope and detailed rebuttal provided by author, I will keep my score.", "rebuttal": "We thank Reviewer **BhBo** for recognizing the practical significance, novelty, and the strength of our experiments and analysis, as well as for the valuable feedback. Below, we address the questions raised.\n\n---\n\n> **Q1**. The demonstration of the proposed corset selection and router strategy is not clear enough, could you please complement more illustrations or descriptions?\n\n**Bucketed Core-set Selection (BCS) and Core-set Attention** (main manuscript Section 3.2)\n\n1. Given an input video token sequence $H$, we divide it into non-overlapping spatial-temporal buckets of size $(t, h, w)$. For a concrete example, we set $t=1$, $h=w=3$, and define the core-set retention ratio as $r_\\mathrm{core} = \\frac{5}{9}$. Each resulting bucket contains $3 \\times 3 = 9$ tokens, which we index from 1 to 9 as shown in the layout below.\n| 1 | 2 | 3 |\n| --- | --- | --- |\n| 4 | 5 | 6 |\n| 7 | 8 | 9 |\n2. Within each bucket, the center token (index 5) is designated as the anchor. This anchor computes the cosine similarity with the remaining 8 tokens in the bucket. The least similar $thw \\cdot (1 - r_\\mathrm{core}) = 9 \\cdot \\frac{4}{9} = 4$ tokens are discarded.\n3. This reduction is motivated by empirical observations that many attention heads in Video DiTs do not rely on fine-grained local details. Thus, semantically redundant tokens can be safely removed without degrading performance.\n4. After pruning, each bucket retains $r_\\mathrm{core}$ of its original tokens, resulting in a compressed sequence termed the ***core-set sequence***. \n5. Full self-attention is then applied to this shorter sequence, significantly reducing computational cost compared to dense attention.\n6. Following the attention step, the pruned tokens are restored by assigning them the attention output of the corresponding anchor token (index 5). This reconstructed sequence is then forwarded to subsequent modules, such as the feed-forward layers in the transformer block.\n\nThe entire process is performed during inference and is independently recomputed for each attention head.\n\n**Signal-aware Attention Routing** (main manuscript Section 3.3)\n\n1. For each layer and attention head, a router takes the time-step embedding as input and projects it through a linear layer to a tensor of shape `(batch_size, 3)`. A softmax is then applied, producing three gate values between 0 and 1 that sum to 1.\n2. **Inference.** Each gate value indicates the relevance of one attention branch for the current head. The branch with the highest gate value is selected for use.\n3. **Training.** The router is trained to learn suitable branch selection by computing a weighted sum of the outputs from all branches, as defined in Equation 6 of the main manuscript. The gated output is supervised using two losses: one to match the output of dense attention ($\\mathcal{L}\\_\\mathrm{distill}$), and another to align with the ground-truth denoising direction ($\\mathcal{L}\\_\\mathrm{CFM}$).\n\n**A detailed component-wise overhead analysis.** Specifically, we report the latency for a single sampling step (i.e., one forward pass of the model) averaged over 10 prompts and 50 time steps in Table 1 below. All layers and attention heads are configured to use the same attention mechanism for benchmarking. We evaluate three strategies: dense (full) attention, sliding window attention, and core-set attention.\n\nTable 1\n\n|  | Attention | Attention related | Router | BCS | Feed-Forward MLP | Others |\n| --- | --- | --- | --- | --- | --- | --- |\n| Dense | 21197.99 ($\\pm 18.52$) | 2912.12 ($\\pm 0.74$) | - | - | 269.91 ($\\pm 0.14$) | 2.99 ($\\pm 0.03$) |\n| Sliding window | 7215.30 ($\\pm 14.78$) | 2913.28 ($\\pm 0.69$) | 87.23 ($\\pm 0.06$) | - | 268.96 ($\\pm 0.13$) | 2.98 ($\\pm 0.03$) |\n| Core-set | 5647.17 ($\\pm 13.40$) | 2911.67 ($\\pm 0.72$) | 87.24 ($\\pm 0.05$) | 862.73 ($\\pm 0.59$) | 267.81 ($\\pm 0.11$) | 3.00 ($\\pm 0.03$) |\n\nNote:\n\n- All latency values in Table 1 are in milliseconds; values in parentheses denote standard deviation.\n- “Attention-related” includes all operations associated with attention computation except the attention itself (e.g., QKV projections, output projections, and QK normalization).\n- “Others” include operations outside the transformer blocks (e.g., patchification, unpatchification, and ROPE initialization).\n- Latency measurements on HunyuanVideo were performed using a B200 GPU, which was the only available device at the time.\n\nThe results indicate that, although the proposed BCS strategy and router module introduce slight additional overhead, this cost is negligible compared to the reduction in attention computation latency.\n\nAn overhead analysis of the entire sampling process, conducted on HunyuanVideo with an H100 GPU, is provided in Figure 7 of the main manuscript. And a theoretical complexity analysis of the BCS is detailed in Appendix A.1 of the main manuscript."}, {"review_id": "RcgCyBSyc8", "reviewer": "Reviewer_m2pg", "summary": "The paper presents VORTA, an acceleration framework for Video Diffusion Transformers (VDiTs) that improves efficiency without sacrificing performance. VORTA introduces a sparse attention mechanism to capture long-range dependencies and a dynamic routing strategy to select optimal attention variants. This approach achieves a 1.76× speedup in video generation and can integrate with other methods, reaching up to a 14.41× speedup.", "questions": "Please refer to the weakness section.", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength\n- Adaptability: The dynamic routing mechanism is tailored for adaptive sparse scheme for different diffusion configurations and backbones.\n- Compatibility with other compression techniques: VORTA could be with other existing acceleration techniques, like caching and step distillation.\n- Detailed description of implementation details in the Appendix.\n\nWeakness\nMy main concern lies in the lack of clarity regarding the motivation, applicability, and effectiveness of some proposed methods.\n- Bucketed Core set Selection (BCS):\n  - While detailed pseudo-code is provided, it only describes how to determine the core set attention sparse mask. However, it is unclear whether this sparse mask is shared across different conditions and time steps. Additionally, it remains ambiguous whether the BCS is performed online for each attention head.\n  - If BCS is performed online, the overhead analysis appears to be missing. This process involves multiple steps, which could introduce significant overhead. If the BCS sparse mask is determined offline, the paper should justify whether such a sparse mask can generalize across multiple time steps and prompts.\n  - The assumption underlying the design of BCS is that the \"center token is very similar to its neighboring tokens,\" which, as the authors state, \"will not always hold.\" When this assumption does not hold, the claim that \"the remaining tokens could complement the necessary information\" needs proper justification. Furthermore, there is a lack of analysis to validate the assumption behind BCS and to assess the effectiveness of BCS itself.\n- Dynamic Routing:\n  - The motivation for employing training-based dynamic routing requires further justification. The necessity of the routing design should be demonstrated by showing that the optimal compression plan varies significantly across different conditions. Even in such cases, the advantage of introducing an additional training-based router, rather than simply adopting different plans or heuristic rules for different conditions, needs to be addressed.\n  - Additionally, the generalization ability of the optimized router should be thoroughly verified to demonstrate its effectiveness.", "was_revised": true, "final_justification": "The author’s response addresses part of my concerns. However, the key issue remains that some assumptions in the paper are not properly justified. Despite multiple rounds of discussion, the author’s explanations are provided without direct observations or empirical evidence, relying instead on intuitive new “assumptions” that lack clear validation. Therefore, I maintain my original stance and lean towards a weak rejection.", "rebuttal": "We sincerely thank the Reviewer **m2pg** for acknowledging the adaptability, extensibility, and clarity of our implementation, as well as for the insightful comments and questions. Below, we provide detailed responses to the raised concerns.\n\n---\n\n> **W1.** It is unclear whether this sparse mask is shared across different conditions and time steps. Additionally, it remains ambiguous whether the BCS is performed online for each attention head.\n\nFor the core-set attention, a sparse mask is dynamically computed for each attention head, varying across conditions and time steps. This design enables the method to adapt flexibly to diverse input patterns.\n\n---\n\n> **W2.** If BCS is performed online, the overhead analysis appears to be missing. This process involves multiple steps, which could introduce significant overhead.\n\nWe present a detailed component-wise overhead analysis in Table 1 below. Specifically, we report the latency for a single sampling step (i.e., one forward pass of the model) averaged over 10 prompts and 50 time steps. All layers and attention heads are configured to use the same attention mechanism for benchmarking. We evaluate three strategies: dense (full) attention, sliding window attention, and core-set attention.\n\nTable 1\n\n|  | Attention | Attention related | Router | BCS | Feed-Forward MLP | Others |\n| --- | --- | --- | --- | --- | --- | --- |\n| Dense | 21197.99 ($\\pm 18.52$) | 2912.12 ($\\pm 0.74$) | - | - | 269.91 ($\\pm 0.14$) | 2.99 ($\\pm 0.03$) |\n| Sliding window | 7215.30 ($\\pm 14.78$) | 2913.28 ($\\pm 0.69$) | 87.23 ($\\pm 0.06$) | - | 268.96 ($\\pm 0.13$) | 2.98 ($\\pm 0.03$) |\n| Core-set | 5647.17 ($\\pm 13.40$) | 2911.67 ($\\pm 0.72$) | 87.24 ($\\pm 0.05$) | 862.73 ($\\pm 0.59$) | 267.81 ($\\pm 0.11$) | 3.00 ($\\pm 0.03$) |\n\nNote:\n\n- All latency values in Table 1 are in milliseconds; values in parentheses denote standard deviation.\n- “Attention-related” includes all operations associated with attention computation except the attention itself (e.g., QKV projections, output projections, and QK normalization).\n- “Others” include operations outside the transformer blocks (e.g., patchification, unpatchification, and ROPE initialization).\n- Latency measurements on HunyuanVideo were performed using a B200 GPU, which was the only available device at the time.\n\nThe results indicate that, although the proposed BCS strategy and router module introduce slight additional overhead, this cost is negligible compared to the reduction in attention computation latency.\n\nAn overhead analysis of the entire sampling process, conducted on HunyuanVideo with an H100 GPU, is provided in Figure 7 of the main manuscript. And a theoretical complexity analysis of the BCS is detailed in Appendix A.1 of the main manuscript. \n\n---\n\n> **W3.** The assumption underlying the design of BCS is that the \"center token is very similar to its neighboring tokens,\" which, as the authors state, \"will not always hold.\" When this assumption does not hold, the claim that \"the remaining tokens could complement the necessary information\" needs proper justification. Furthermore, there is a lack of analysis to validate the assumption behind BCS and to assess the effectiveness of BCS itself.\n\n**Assumption and design of BCS.** In BCS, the design that we select the center token as an anchor to compute similarity scores with surrounding tokens is motivated by the assumption that “the center token is similar to its surrounding tokens”.\n\n**What if this assumption does not hold**\n\n- When fine-grained details are not critical and the model primarily captures high-level semantic features such as contours, the loss introduced by BCS has a negligible impact.\n- When fine-grained details are important, the router is trained to select alternative attention branches by replicating the output of dense attention ($\\mathcal{L}\\_\\text{distill}$, main manuscript Equation 8) and aligning with the ground-truth denoising direction ($\\mathcal{L}\\_\\text{CFM}$, main manuscript Equation 1).\n\n**Experimental results show the practical effectiveness of BCS.** Experiments have shown that VORTA achieves lossless video generation compared to the dense model. Notably, approximately 40% of attention heads adopt the core-set attention strategy (visualized in the main manuscript in Figures 8, 12, and 13). These findings empirically validate the BCS assumption across diverse settings and demonstrate its effectiveness for lossless acceleration.\n\n**BCS is much better than naive pooling.** BCS discards the most similar tokens and retains the least similar ones, which maintains \"the necessary information\". In contrast, common strategies like average pooling do not account for cases where the similarity assumption fails, often resulting in degraded performance, as shown empirically in main manuscript Section 4.3.\n\nThe ablation studies comparing BCS to other pooling methods are presented in Section 4.3 and Table 2 of the main manuscript. A sensitivity analysis is further provided in Appendix B.1 and Figure 10 of the main manuscript.\n\n---\n\n> **W4.** The motivation for employing training-based dynamic routing requires further justification. The necessity of the routing design should be demonstrated by showing that the optimal compression plan varies significantly across different conditions. Even in such cases, the advantage of introducing an additional training-based router, rather than simply adopting different plans or heuristic rules for different conditions, needs to be addressed.\n> \n\n**Our router is more flexible and adaptable than heuristic rules.** Employing heuristic rules to determine the routing strategy is of high complexity. Methods such as ARnR [1] and STA [2] rely on offline profiling and heuristics to select sparse strategies. These approaches typically require multiple sampling runs to determine the strategy for a single configuration (e.g., sampling with UniPC [3] with 50 sampling steps), which increases overhead.\n\nIn practice, dense models are frequently used with varying sampling configurations to balance quality and speed. These configurations include:\n\n1. Adjusting sampling hyperparameters (e.g., modifying the flow shift),\n2. Changing the number of sampling steps (e.g., from 50 to 30), and\n3. Switching the scheduler (e.g., from UniPC [3] to DPM++ [4])\n\nWe aim to preserve this flexibility of dense models. Heuristic methods [1, 2] must re-profile and re-tune their strategy whenever any sampling configuration is modified. This reconfiguration process can take several hours on a single GPU due to the high computational cost of dense sampling. In contrast, our VORTA router generalizes across different sampling settings without requiring any changes or additional overhead, owing to its lightweight training paradigm.\n\n**Quantitative evaluation on diverse sampling configurations.** We conduct additional experiments on Wan, replacing the default UniPC scheduler (50 steps) with the DPM++ scheduler (30 steps). Table 2 below shows the results across four VBench metrics for quality and latency, as well as peak memory usage for efficiency.\n\nTable 2\n\n| Scheduler | Method | Latency (s) ↓ | Memory (GB) ↓ | Subject ↑ | Consistency ↑ | Aesthetic ↑ | Imaging ↑ |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| UniPC (50) | Wan 2.1 | 1359.76 | 41.77 | 91.21 | 75.49 | 63.33 | 63.99 |\n| UniPC (50) | + VORTA | 791.02 | 43.97 | 90.76 | 75.09 | 63.79 | 63.07 |\n| DPM++ (30) | Wan 2.1 | 817.00 | 41.77 | 89.72 | 74.54 | 64.43 | 63.99 |\n| DPM++ (30) | + VORTA | 440.76 | 43.97 | 91.82 | 75.13 | 64.32 | 63.95 |\n\n**Note**: *Latency measurements were performed using a B200 GPU, which was the only available device at the time.*\n\nThe consistent, lossless performance of VORTA relative to the dense model confirms its ability to generalize efficiently without added cost, an advantage enabled by its lightweight training, which heuristic-based methods like STA and ARnR do not achieve.\n\nThese motivations and results will be incorporated in more detail in the next revision.\n\n---\n\n> **W5.** The generalization ability of the optimized router should be thoroughly verified to demonstrate its effectiveness.\n\nThe results in W4 demonstrate that the optimized router generalizes well across different schedulers and sampling steps. \n\n**VBench is out-of-distribution evaluation with respect to the training data.** The training dataset, Mixkit, consists exclusively of real-world videos. In contrast, VBench includes stylized or synthetic content (for example, beach scenes rendered in Van Gogh, oil painting, or Ukiyo-e styles). So, evaluation on VBench (main manuscript Section 4) also confirms its generalization capability to out-of-distribution prompts.\n\n**The OOD claim is statistically significant.** To statistically assess this, we perform a Maximum Mean Discrepancy two-sample test using 400 CLIP embeddings from the training prompts and 400 CLIP embeddings sampled from VBench prompts, testing the null hypothesis ($H_0$) that both samples are drawn from the same distribution. The resulting p-value is 0.0009, which allows us to reject $H_0$ at the 0.1% significance level. This confirms that the VBench prompts represent a distribution shift from the training data, and the performance on VBench prompts highlights the strong generalization capability of our VORTA.\n\n**Reference**\n\n[1] Sun, et al. “AsymRnR: Video Diffusion Transformers Acceleration with Asymmetric Reduction and Restoration.” ICML (2025)\n\n[2] Zhang, et al. “Fast Video Generation with Sliding Tile Attention.” ICML (2025)\n\n[3] Zhao, et al. “UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Models.” NIPS (2023)\n\n[4] Lu, et al. “DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models.” ICLR (2023)"}, {"review_id": "y09UQCimvG", "reviewer": "Reviewer_BsHH", "summary": "This paper proposes VORTA, a sparse attention technique with a router to speed up the inference stage of Video Diffusion Models. This paper proposes three attention variations: full attention, sliding window attention, and coreset attention for each head. They further propose a router to classify each attention head into one of these attention patterns, and apply distillation loss and regularization loss to punish the excessive usage of full attention. VORTA exhibits strong quality and speedup performance on HunyuanVideo and Wan 2.1, demonstrating its effectiveness.", "questions": "1. What's the average attention sparsity during the generation process for sliding window attention and coreset attention?\n2. What is the value of r_core for BCS module? How does this value affect the final performance?\n3. What will the attention maps look like for the 0.2% full attention heads? More visualization results for these attention heads will help to understand the model's behavior.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The design of the attention router, along with the introduction of the distillation and regularization loss term, is innovative and contributes to the novelty of the approach.\n2. The proposed method is flexible and can be effectively integrated with other techniques, such as PAB, to achieve additional speedup.\n\nWeaknesses:\n1. The coreset attention pattern requires further empirical validation. It remains unclear how performance would be affected if alternative attention patterns—such as the temporal attention used in Sparse VideoGen[1]—were employed.\n2. The paper does not provide comparisons against other baselines using the Wan 2.1 model, which limits the assessment of its relative performance.\n3. Refer to the questions section.\n\n[1] Xi, Haocheng, et al. \"Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity.\" arXiv preprint arXiv:2502.01776 (2025).", "was_revised": true, "final_justification": "The authors' rebuttal is clear and solid. They have addressed my concerns, in terms of explanation, experimental validation, and numerical calculation. Therefore I raised my score.", "rebuttal": "We thank the Reviewer **BsHH** for acknowledging the motivation, novelty, and extensibility of our approach. Below, we provide responses to the questions and concerns raised. For clarity, we have reordered the responses to improve readability and interpretation.\n\n---\n\n> **Q2.** What is the value of $r_\\mathrm{core}$ for BCS module? How does this value affect the final performance?\n\n**Explanation of $r_\\mathrm{core}$.** The core-set ratio $r_\\mathrm{core}$ denotes the proportion of tokens retained for attention computation after applying BCS, relative to the full sequence length (main manuscript Section 3.2). For example, given a sequence of 100 tokens, setting $r_\\mathrm{core} = 0.4$ means that 60 tokens are deduplicated during BCS, and only 40 tokens participate in the subsequent attention computation.\n\n**Impact of varying** **$r_\\mathrm{core}$.** Reducing $r_\\mathrm{core}$ leads to more aggressive compression and faster inference, at the potential cost of output quality. When $r_\\mathrm{core} = 1$, no tokens are removed, and the computation is equivalent to standard dense attention. As $r_\\mathrm{core}$ decreases to 0.5, video quality remains nearly unchanged, as indicated by VBench scores and human case study. However, further reductions below 0.5 result in noticeable quality degradation, despite latency improvements. Detailed results are provided in Appendix B.1 and Figure 10 of the main manuscript.\n\nAs a consequence, we use $r_\\mathrm{core} =0.5$ in all experiments (main manuscript Appendix A.2). We will define this variable more explicitly in the next revision.\n\n---\n\n> **Q1.** What's the average attention sparsity during the generation process for sliding window attention and core-set attention?\n\n**Definition of sparsity.** We define attention sparsity as the ratio of skipped query-key token multiplications to the total number of query-key multiplications in standard dense attention. A higher sparsity indicates better theoretical computational efficiency.\n\n**HunyuanVideo’s overall sparsity is 82.65%.** HunyuanVideo generates video tokens with shape (30, 45, 80)\n\n- For sliding window attention, we use a window size of (18, 27, 24). The resulting sparsity is computed as\n    - $$1 - \\frac{(18 \\times 27 \\times 24) \\times (30 \\times 45 \\times 80)}{(30 \\times 45 \\times 80)^2} = 0.892$$\n- For core-set attention with $r_\\mathrm{core} =0.5$. The sparsity is computed as\n    - $$1 - \\frac{(0.5 \\times 30 \\times 45 \\times 80 )^2}{(30 \\times 45 \\times 80)^2} = 0.75$$\n- According to the routing statistics (visualized in the main manuscript Figures 8 and 12), full attention, sliding attention, and core-set attention are used in 0.49%, 56.47%, and 43.04% of the cases, respectively. The resulting overall sparsity is **82.65%**.\n\n**Wan 2.1’s overall sparsity is 81.68%.** Wan 2.1 generates video tokens with shape (20, 45, 80)\n\n- For sliding window attention, we use a window size of (15, 27, 24). The resulting sparsity is computed as\n    - $$1 - \\frac{(15 \\times 27 \\times 24) \\times (20 \\times 45 \\times 80)}{(20 \\times 45 \\times 80)^2} = 0.865$$\n- For core-set attention with $r_\\mathrm{core} =0.5$. The sparsity is computed as\n    - $$1 - \\frac{(0.5 \\times 20 \\times 45 \\times 80 )^2}{(20 \\times 45 \\times 80)^2} = 0.75$$\n- According to the routing statistics (visualized in main manuscript Figure 13), full attention, sliding attention, and core-set attention are used in 0.23%, 59.61%, and 40.16% of the cases, respectively. The resulting overall sparsity is **81.68%**.\n\n---\n\n> **W2.** The paper does not provide comparisons against other baselines using the Wan 2.1 model, which limits the assessment of its relative performance.\n\n**Comparison with SVG on Hunyuan and Wan 2.1.** We evaluate Spare Video Gen (SVG) on a B200 GPU and compare it with our VORTA, as shown in Tables 1 and 2. The comparison covers four VBench dimensions for quality and latency, as well as peak memory usage for efficiency. Since both SVG and VORTA focus on attention sparsity, minor non-attention optimizations (e.g., operator fusion and quantization) are disabled in SVG to ensure a fair comparison.\n\nTable 1\n\n|  | Latency (s) ↓ | Sparsity ↓ | Memory (GB) ↓ | Subject ↑ | Consistency ↑ | Aesthetic ↑ | Imaging ↑ |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| HunyuanVideo | 1224.22 | 0% | 47.64 | 90.30 | 74.30 | 62.05 | 63.33 |\n| + SVG | 664.48 | 80% | 71.38 | 90.59 | **75.11** | **62.73** | 63.17 |\n| + VORTA | **568.74** | **82.65%** | **51.15** | **92.43** | 74.62 | 62.33 | 63.16 |\n\nTable 2\n\n|  | Latency (s) ↓ | Sparsity ↓ | Memory (GB) ↓ | Subject ↑ | Consistency ↑ | Aesthetic ↑ | Imaging ↑ |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| Wan 2.1 | 1359.76 | 0% | 41.77 | 91.21 | 75.49 | 63.33 | 63.99 |\n| + SVG | 921.50 | 75% | 65.01 | 90.42 | 74.96 | 63.53 | **64.45** |\n| + VORTA | **791.02** | **81.68%** | **43.97** | **90.76** | **75.09** | **63.79** | 63.07 |\n\nFor both Hunyuan and Wan 2.1 models, the VBench scores are statistically indistinguishable, indicating no significant differences in quality. However, VORTA consistently outperforms SVG in latency and sparsity metrics. Additionally, SVG incurs substantially higher memory usage due to its reliance on online profiling, which limits its applicability in resource-constrained environments.\n\n---\n\n> **W1.** The coreset attention pattern requires further empirical validation. It remains unclear how performance would be affected if alternative attention patterns—such as the temporal attention used in SVG—were employed.\n\n**VORTA is designed for SOTA foundational models.** Earlier video diffusion models adopted factorized spatio-temporal (2+1D) attention [1, 2, 3]. Subsequent works [4, 5, 6] transitioned to full 3D attention, demonstrating superior generation quality and establishing 3D attention as the mainstream architecture in video foundation models. Our method, VORTA, focuses on optimizing inference efficiency for these 3D attention models. We exclude older and less competitive attention mechanisms from our scope due to their diminished relevance and performance.\n\n**VORTA outperforms other acceleration methods with sparse attention pattern.** Similar to VORTA, SVG also aims to improve inference efficiency via sparse attention. However, as shown in experiment results above, VORTA achieves better efficiency without compromising performance.\n\n**VORTA can integrate with various acceleration methods in a plug-and-play manner.** Additionally, VORTA integrates well with other orthogonal efficiency techniques, such as PAB and PCD, as shown in Section 4 of the main manuscript. Incorporating another sparse attention methods would introduce significant engineering complexity with limited practical benefit. \n\nGiven the constraints of the rebuttal period, we prioritize experiments and analyses with greater impact.\n\n---\n\n> **Q3.** What will the attention maps look like for the 0.2% full attention heads? More visualization results for these attention heads will help to understand the model's behavior.\n\nWe analyzed the attention maps of the full attention heads and identified several notable patterns. Due to the 2025 rebuttal policy, we cannot include figures at this stage, so we describe the key observations textually.\n\nFor queries and keys within the same frame $f$, the attention pattern typically exhibits a strong diagonal, indicating that each token predominantly attends to itself. Additionally, we observe several vertical, non-uniformly spaced slashes, suggesting that certain key positions are globally important across the entire frame. Notably, the positions of these vertical lines vary with different prompts, forming a composite attention pattern resembling “| | |” + “\\”.\n\nWhen querying tokens in frame $f$ with keys in adjacent frames $f \\pm i$, a similar pattern persists. However, as $i$ increases, the overall attention weights decrease. In particular, the vertical components decay more rapidly than the diagonal ones.\n\nWe will include this analysis and the corresponding figure in the next revision to provide additional insights for readers.\n\n---\n\n**Reference**\n\n1. Singer, et al. “Make-A-Video: Text-to-Video Generation without Text-Video Data.” ICLR (2023)\n2. Wang, et al. “LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models.” IJCV (2024)\n3. Ma, et al. “Latte: Latent Diffusion Transformer for Video Generation.” TMLR (2025)\n4. Polyak, et al. “Movie Gen: A Cast of Media Foundation Models.” preprint arXiv: 2410.13720 (2024)\n5. Kong, et al. “HunyuanVideo: A Systematic Framework For Large Video Generative Models.” preprint arXiv: 2412.03603 (2024)\n6. Wang, et al. “Wan: Open and Advanced Large-Scale Video Generative Models.” preprint arXiv: 2503.20314 (2025)"}, {"review_id": "Nvw8BDeawy", "reviewer": "Reviewer_Pmtr", "summary": "VORTA is a plug-in accelerator for Video Diffusion Transformers that adds a tiny router and two sparse attention options to every self-attention block. At each diffusion step, the router, conditioned on timestep and head features, sends a head either to a sliding-tile local window or to a global branch that keeps only a coreset of representative tokens chosen inside space-time buckets, falling back to full attention only when needed. This adaptive routing turns quadratic attention into near-linear cost without retraining the generator. Tested in 720p HunyuanVideo and Wan2.1, VORTA shortens end-to-end inference time and lowers memory usage while matching or slightly improving baseline models.", "questions": "- The router is distilled only to replicate dense attention, yet Table 1 shows a 0.3 -0.4 VBench gain over the baseline. Please analyse the source of this improvement. Showing that the gain is statistically significant and not incidental would raise my rating.\n- VBench is multi-dimensional (scoring frame fidelity, motion, text alignment, etc.) but all dimensions are produced by learned evaluators. A second, independent lens (e.g., a brief human-preference study or a metric like FVD) would strengthen the \"no-quality-loss\" claim.\n- Add a short related-work paragraph on conditional computation, e.g. ConvNet-AIG (Veit et al., 2017) and Information Gain Trellis (Bicici et al., 2024), to acknowledge prior adaptive-routing ideas and highlight what is new (per-head, per-timestep gating for diffusion).", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper evaluates VORTA on two demanding 720p text-to-video diffusion transformers and reports end-to end latency, GPU memory, VBench sub-scores, and LPIPS, giving a well-rouded view of speed-quality trade-offs.\n- Experiments show that adaptive routing alone yields about 1.7 times speed-ups without sacrificing VBench quality, and that VORTA composes additively with cached-KV and step-distillation to reach up to 14 times overall acceleration.\n- Ablation studies isolate every design decision: removing either sparse branch, disabling timestep conditioning, replacing bucketed coreset selection with naive pooling, and omitting full-attention fallback. Each variant's impact on speed is quantified.\n- Analysis of attention-score statistics across diffusion steps reveals that global context dominates early iterations while local detail dominates later ones; this empirical insight motivates the router and ilustrates a general principle that may guide future diffusion designs.\n- Implementation details are clearly documented, making reproductions easier.\n\nWeaknesses:\n- All quality evidence comes from a single, learned-metric suite (VBench). Adding a second, independent check, either a brief human-preference study or a public metric like FVD, would make the ‘no quality loss’ claim more robust.\n- The learned router, though lightweight, requires an extra distillation pass and a small video dataset. It would be good addition to the paper if they examine router robustness on out-of-distribution prompts or much longer clips.\n- Memory consumption rises by around 7% or more versus the dense baseline, and the added branches increase implementation complexity, which could hinder deployment on single-GPU or resource-constrained environments.\n- Related-work discussion omits earlier conditional-computation and information-gain routing methods (e.g., ConvNet-AIG, Information Gain Trellis), understating the lineage of adaptive gating ideas and making the novelty of the router appear greater than it is.", "was_revised": true, "final_justification": "After reviewing the authors' detailed rebuttal, I am increasing my score from a 4 to a 5. The authors have exceeded expectations in addressing my concerns. They provided new experimental evidence (a human study, statistical significance tests) and insightful analysis (attention weight distribution) that have substantially strengthened the paper. My initial reservations about quality evaluation, robustness, and the source of performance gains have been fully resolved. The paper is a solid contribution to the field, and I now recommend acceptance.", "rebuttal": "We thank Reviewer **Pmtr** for acknowledging the clarity of our experiments, analysis, and documentation, and for providing thoughtful and constructive feedback. Your comments have been invaluable in refining our work.\n\n---\n> **Q1.** Table 1 shows a 0.3-0.4 VBench gain over the baseline. Please analyse the source of it.\n\n**Source of Gains on the Aggregated VBench Score.** As shown in Table 3 of the Appendix, the improvement mainly stems from the \"object\" and \"multiple objects\" dimensions. Each generated video is assigned a score for each dimension, measuring the proportion of prompt-specified objects successfully detected in the video. This score reflects the semantic accuracy of the generation. Appendix Table 3 shows the averaged scores across all generated videos.\n\n**Statistical significance test of the gains.** We performed the Wilcoxon signed-rank test (a non-parametric alternative to the t-test for paired samples) on the Hunyuan scores $s_{0i}$ and the VORTA scores $s_{1i}$. \n\n- **Null hypothesis** $H_0$: The distribution of $\\{s\\_{0i} - s\\_{1i}\\}$ is symmetric about $\\mu = 0$.\n- **One-sided alternative hypothesis** $H_1$: The distribution of $s_{0i} - s_{1i}$ is symmetric about $\\mu < 0$.\n\nFor the \"object\" and \"multiple objects\" dimensions, the p-values are 0.003 and 0.03, respectively. Both results allow us to reject $H_0$ at the 5% significance level, indicating statistically significant improvements.\n\n**Further investigation.** Hunyuan Video adopts the MMDiT [5] architecture, where text and vision tokens are concatenated and jointly processed through self-attention. Unlike prior vision diffusion models (e.g., Stable Diffusion) that commonly employ cross-attention for text conditioning, MMDiT relies solely on self-attention. So we analyze how attention weights are distributed across modalities in both dense and core-set attention. On average, the attention weights from vision queries to text keys reached 33% in core-set attention, compared to only 18% in dense attention. This suggests that the core-set selection step reduces visual redundancy and allocates more attention to semantic (textual) cues, thereby enhancing the model’s performance on semantically driven metrics. This gain does not lead to a drop in quality, as fine-grained vision-to-vision attention is less critical in components that employ core-set attention.\n\nBesides the distillation loss, which emphasizes replicating the dense attention outputs, we also employ a conditional flow-matching loss $\\mathcal{L}\\_\\text{CFM}$ to guide learning toward the ground truth (main manuscript Equation 1). The use of core-set selection can be interpreted as a form of unstructured pruning, contributing to the observed performance.\n\n---\n> **Q2, W1.** A second, independent lens (e.g., a brief human-preference study or a metric like FVD) would strengthen the no-quality-loss claim.\n\nFVD is known to exhibit instability and weak correlation with human evaluations [1, 2, 3, 4]. \n\nSo we conducted a human preference study on the generated videos. Specifically, we evaluated 100 sampled videos, where three human evaluators were asked to select the better video, or indicate a tie, between the dense model and VORTA outputs. Results show that 94% of the videos were judged as ties. These results will be incorporated in the next revision.\n\n| Method | Votes |\n| --- | --- |\n| Hunyuan Dense  | 11 (3.7%) |  \n| Tie | 282 (94%) |\n| VORTA | 7 (2.3%) |\n---\n> **W2.** It would be good addition to examine router robustness on out-of-distribution prompts or much longer clips.\n\n**VBench is out-of-distribution evaluation with respect to the training data.** The training dataset, Mixkit, consists exclusively of real-world videos. In contrast, VBench includes prompts for stylized or synthetic content (for example, beach scenes rendered in Van Gogh, oil painting, or Ukiyo-e styles). The model’s stable performance on VBench suggests robustness to out-of-distribution prompts.\n\n**Statistical test of the OOD claim.** We perform a Maximum Mean Discrepancy two-sample test using 400 CLIP embeddings from the training prompts and 400 CLIP embeddings sampled from VBench prompts, testing the null hypothesis ($H_0$) that both samples are drawn from the same distribution. The resulting p-value is 0.0009, which allows us to reject $H_0$ at the 0.1% significance level. This confirms that the VBench prompts represent a distribution shift from the training data, and the performance on VBench prompts highlights the strong generalization capability of our VORTA.\n\nVORTA can be extended to longer video clips, and additional experiments are planned. However, we observed a notable degradation in the performance of the dense model as the number of frames increases. In particular, temporal distortion becomes significant when the frame count exceeds 157. Consequently, we do not exhibit further experiments on longer clips at present.\n\n---\n> **W3.** Memory consumption rises by around 7% or more vs the dense baseline, which could hinder deployment on single-GPU or resource-constrained environments.\n\nWe observe that VORTA reaches a peak memory usage of 51 GB for 5-second 720p videos, which exceeds the capacity of many consumer GPUs. However, this is primarily due to the dense HunyuanVideo model itself, which consumes approximately 47 GB and is unsuitable for resource-constrained devices.\n\nMoreover, **VORTA requests the lowest memory among efficient attention methods** (e.g., ARnR and STA). Caching-based approaches like PAB require over 80 GB of memory, even surpassing the capacity on high-end GPUs like the H100.\n\n---\n> **Q3, W4.** Add a short related-work paragraph on conditional computation, e.g. ConvNet-AIG and Information Gain Trellis, to acknowledge prior adaptive-routing ideas and highlight what is new.\n\nThank you for your suggestion. We will incorporate these references into the related work section to better situate our contributions within the existing literature and enhance the overall coherence of the paper.\n\n---\n**Reference**\n\n1. Singer, et al. “Make-A-Video: Text-to-Video Generation without Text-Video Data.” ICLR (2023)\n2. Wang, et al. “LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models.” IJCV (2024)\n3. Polyak, et al. “Movie Gen: A Cast of Media Foundation Models”. preprint arXiv: 2410.13720 (2024)\n4. Ge, et al. “On the Content Bias in Fréchet Video Distance.” CVPR (2024)\n5. Esser, et al. \"Scaling Rectified Flow Transformers for High-Resolution Image Synthesis.\" ICML (2024)"}]

Author rebuttal:

Meta-review: This paper proposes VORTA, a routing-based sparse attention framework for accelerating video diffusion transformers. The method achieves substantial speedups with minimal quality loss and integrates well with other efficiency techniques. Reviewers generally praised the novelty, thorough experiments, and practical impact, noting clear ablations and overhead analyses. While one reviewer (m2pg) questioned the empirical validation of the coreset selection assumption and found some explanations overly intuitive, three reviewers recommended acceptance after rebuttal, highlighting the detailed responses and strong experimental evidence. Overall, the paper makes a solid and timely contribution. The meta-reviewer encouages the authors further polish the paper according to the reviewers' comments.
Decision: Accept (poster)