Sparse Video-Gen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity
Authors: Haocheng Xi
Shuo Yang
Yilong Zhao
Chenfeng Xu
Muyang Li
Xiuyu Li
Yujun Lin
Han Cai
Jintao Zhang
Dacheng Li
Jianfei Chen
Ion Stoica
Kurt Keutzer
Song Han
Venue: ICML 2025 poster (2025)
Keywords: Sparse Attention
Video Diffusion Transformer
Efficient Inference
Primary area: deep_learning->attention_mechanisms
TL;DR: We propose a sparse attention method to accelerate the inference of video diffusion models.

Abstract: Diffusion Transformers (DiTs) dominate video generation but their high computational cost severely limits real-world applicability, usually requiring tens of minutes to generate a few seconds of video even on high-performance GPUs. This inefficiency primarily arises from the quadratic computational complexity of 3D full attention with respect to the context length. In this paper, we propose a training-free framework termed Sparse VideoGen (SVG) that leverages the inherent sparsity in 3D full attention to boost inference efficiency. We reveal that the attention heads can be dynamically classified into two groups depending on distinct sparse patterns: (1) Spatial Head, where only spatially-related tokens within each frame dominate the attention output, and (2) Temporal Head, where only temporally-related tokens across different frames dominate. Based on this insight, SVG proposes an online profiling strategy to capture the dynamic sparse patterns and predicts the type of attention head. Combined with a novel hardware-efficient tensor layout transformation and customized kernel implementations, SVG achieves up to 2.28$\times$ and 2.33$\times$ end-to-end speedup on CogVideoX-v1.5 and HunyuanVideo, respectively, while preserving generation quality. Our code will be open-sourced upon publication.

Sparse VideoGen: Accelerating Video Diffusion Transformers with Opatial-Tempond Sparsity
Haocheng Xi Shuo Yang" Yilong Zhao Chenfeng Xu Muyang Li Xiuyu Li Yujun Lin Han Cai Dacheng Li Jianfei Chen lon Stoica Kurt Keutzer Song Han
Jintao Zhang
https: /ttes://gittttt
Abstract
CogVidenX-vl.5 720p, 80 frames, Image-to- Viden
Dense Attention mins X VS Ours (4 min:
Donic
Diffusion Transformers (DiTs) dominate video generation but their high computational cost severely limits rorldworld applicability, usually re- quiring tens of minutes to generate few seconds of video even on high-performance GPUs. This inefficiency primarily arises from the quadratic computational complexity of 3D full attention with respect to the context length. In this pa- per, propose training- free framework termed Sparse VideoGen (SVG) that leverages the inher- ent sparsity in 3D full attention to boost inference efficiency. We reveal that the attention heads can dynamically classified into two groups depend- ing on distinct sparse patterns: (1) Spatial Head, where only spatially-related tokens within each frame dominate the attention output, and (2) Tem- poral Head, where only temporally-related tokens across different frames dominate. Based on this insight, SVG proposes an online profiling strat- egy to capture the dynamic sparse patterns and predicts type of attention head. Combined with a novel hardware-efficient tensor layout transformation and customized kernel implemen- tations, SVG achieves up 2.28x end-to-end speedup Cog CogVideoX-v1.55 2.33x on Hunyuan- Video, while preserving generation quality. Our code is open-sourced at https//giitub..com/vve Saapet//SSpareevieGGGieee
Cun PSNN-18-
Minicretee PSNR-225
Primpt: beautiful posing
Hunyuan 720p. 128 frames, Texi-to-Vidoo
X
Datse
Miniorae
"A woodpeckee pocking holes
Figure I. SVG accelerates video generation while maintaining high quality. On Cog VideoX-v1.S5 1.5-12V and Hunyuan-T2V. method achieves a 2.28x and 2.33x speedup with high PSNR. In contrast, MInference (Jiang al., 2024) fails to maintain pixel fidelity (significant blurring in the first example) and temporal coherence (inconsistencies in tree trunk the second example).
1. Introduction
'Equal contribution. University of California, Berkeley. NVIDIA. MIT. Correspondence to: Chenfeng Xu <xuchen- feng@berkkley.cdu>>
Diffusion Transformers (DiTs) (Peebles & Xie, 2023) have recently emerged transformative paradigm for genera- tive tasks, achieving state-of-the-ae results in image genera- tion. This success has been naturally carried over to video generation, with models adapting from a spatial 2D attention to a spatiotemporal 3D) full attention (Arnab et al., 2021;
Proceedings of the 42 International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Sparse VideoGen: Accelerating Video Diffusson Transformers with Spatial-Temporal Sparsity
LayerNorm
QK Norn
RoPE
Lincar
Attention
fore, the computation for both types of heads can be greatly reduced by only calculating the attended tokens.
Context Length
17k
4%
23%
22%
51%
Despite the theoretical speedup, leveraging sparsity for end- to-end acceleration is still challenging. Firstly, sparsity patterns are highly dynamic across different denoising steps and input prompts. It necessitates an online method to iden- tify sparsity patterns without incurring overhead. Secondly, some sparsity patterns are unfriendly hardware accelera- tors. For example, temporal head computes over noncontigu- ous data that cannot be fed to GPU's tensor cores, resulting in significant efficiency degradations (Ye et al., 2023).
45k
2%11%
12%
73%
120k
14%
12%
82%
0%
20%
40%
60%
80%
100%
Figure Attention dominates the computation in video diffu- sion models. For Cog C and -v1.5 with 17k and 45k context length, attention takes 51% % and 73% of the latency, respee- tively. For Hunyuan Vidco with 120k context length, attention can take over 80% amount of the runtime latency.
To tackle these challenges, we propose Sparse VideoGen (SVG), a training-free framework that accelerates video DiTs with the following novel designs: (1) To efficiently identify the best sparsity pattern for each attention head, SVG introduces an online profiling strategy with minimal overhead (~3%).. It randomly samples 1 1% tokens from each attention head and processes sampled tokens with full atten- tion computation and two distinct sparse attentions (spatial head and temporal head). Finally, the sparse pattern with a lower error compared to the full attention is selected for each head. (2) To improve hardware efficiency, SVG proposes a novel layout transformation, which reorders the noncontigu- ous sparsity pattern of temporal head into a compact and hardware-friendle sparsity pattern..
Yang et al., 2024c; Kong et al., 2024), resulting in high- fidelity and temporally consistent outputs. Close-sourced models such as Sora and Kling, and open-soureed models including Wan 2.1 (Wang et al., 2025), CogVideo (Hong et al., 2023), and Hunyuan Video (Kong et al., 2024), have showcased impressive capabilities in applications ranging from animation (Guo et al., 2024; Feng et al., 2024) to physical world simulation (Liu et al., 2024b).
Despite significant advances in generating high-quality videos, the deployment of video generation models remains challenging due to their substantial computation usage. For instance, Hunyuan Video requires almost an hour on NVIDIA A100 GPU to generate only 5-second video, where the 3D full attention accounts for more than 80% of end-to-end runtime (Figure 2). Moreover, due to the quadratic computationnl complexity with respect to the con- text length (Dao al., 2022), the attention can be much more dominant the resolution and number of frames in- crease, shown in Figure 2.
We prototype SVG with customized kernel implementation by Triton (Tillet et al., 2019) and FlashInfer (Ye et al., 2025) and evaluate SVG's accuracy and efficiency on representa- tive open video generative models including Cog VideoX- v v1.5-12V, CogVideoX-v1.5-T
and Hunyuan Video-T2V. SVG delivers significant efficiency improvements, achiev- ing an end-to-end speedup of up to 2.33 x while maintaining high visual quality with a PSNR of up to 29, outperforming all prior methods. Additionally, we show that SVG is com- patible with FPS quantization, enabling additional efficiency gains without compromising quality. We summarize our key contributions as follows:
Fortunately, attention in transformers is well-known for its sparsity, offering a great opportunity to reduce redun- dant computation. For example, in large language models (LLMs), a small portion of the tokens can dominate the attention output (Zhang et al., 2023c; Xiao et al., 2024b; Tang et al., 2024). Therefore, the computation can be dra- matically reduced by only computing the attention among such important tokens, while still maintaining generation accuracy. However, existing methods cannot be directly applied to DiT's (as shown in Table 1), as video data has fundamentally different sparsity patterns from text data.
We conduct in-depth analysis of video DiTs' sparse pat- terns, revealing two inherent sparse attention patterns (temporal head and spatial head) for efficient video gener- ation.
We propose SVG, a training- free sparse attention frame- work comprising an efficient online profiling strategy and an efficient inference system for accurate and efficient video generation.
Our key observation that attention heads in DiTs exhibit inherent sparsity in two categories: Spatial Head and Tem- poral Head, based on their distinct sparsity patterns. As shown in Figure 3, spatial head mainly focuses on tokens that reside within the same frame, which determines the spatial structures of generated videos. In contrast, temporal head attends to tokens at the same spatial location across all frames, contributing to the temporal consistency. There-
SVG delivers significant speedup while maintaining good video generation quality, paving the way for practical applications of video generative models.
2

---

Sparse VideoGen: Accelerating Video Difffusion Transformers with Spatial-Tempan Sparsity
your
Cor
(a) Spatial Head
(b) Spatial Attention Map
(c) Temporal Head
(d) Tempural Attention Map
Text Prompt
Conclation
Tempprrd Comvllatoo
Frame
Frame Visualization of Sputiul and Temporal Correlation. This exumple have frumes und tokens per frame.
Frame
Figure 3. We observe types of attention maps with distinct sparse patterns: spatial map (b) and temporal map (d). Based the attention map, we classify all attention heads into Spatial Head (a) and Temporal Head (c), which contribute to the spatial and temporal consistency of generated videos respectively. As visualized in (e), spatial head primarily focuses all tokens within the same frame (painted as red). In contrast, temporal head attends to tokens at the same position across all frames (painted as green).
2. Related Work
these methods can improve throughput, their output quality often drops below a PSNR of 22. By contrast, our method preserves PSNR above 30, thus substantially outperform- ing previous approaches in maintaining output fidelity.
2.1. Efficient diffusion models
Decreasing the denoising steps. Most diffusion models employ SDEs that require many sampling steps (Song & Ermon, 2019; Ho et al., 2020; Meng et al., 2022). To ad- dress this, DDIM (Song et al., 2020) approximates them with an ODE; subsequent techniques refine ODE paths and solvers (Lu al., 2022a;b; Liu et al., 2022: 2024c) or in- corporate consistency losses (Song et al., 2023; Luo al., 2023). Distillation based methods (Yin et al., 2024a;b) train simpler, stw-step models. However, these require expensive re-training fin-tuning-iipractiia for most video use cases. In contrast, our approach directly uses off-the-sheelf pre-trained models without any additional training.
2.2. Efficient attention methods
Sparse attention in LLMs. Recent research on sparse attention in language models reveals diverse patterns to reduce computational overhead. StreamingLLM (Xiao et al., 2023) and LM-Infinite (Han al., 2023) observe that attention scores often concentrate the first few or local tokens, highlighting temporal locality. H2O (Zhang et al., 2023b), Scissorhands (Liu et al., 2024d) and DoubleSparsity (Yang et al., 2024b) identify a small set of "heavy hitter" tokens dominating overall attention scores. TidalDecode (Yang et al., 2024a) shows that attention patterns across layers are highly correlated, while DuoAttention (Xiao al., 2024a) and MInference (Jiang et al., 2024) demonstrate distinct sparse patterns across different attention heads. However, these methods focus on token-level sparsity and do not leverage the inherent redundancy of video data.
Diffusion model compression. Weight compression through quantization is a common tactic (Li al., 2023: Zhao et al., 2024a; Li* al., 2025), pushing attention mod- ules INT8 (Zhang et al., 2025b) or even INT4/FP8 (Zhang et al., 2024). Other work proposes efficient architectures (Xie et al., 2024; Cai et al., 2024; Chen et al., 2025) or high- compression autoencoders (Chen et al., 2024a) to improve performance. Our Sparse VideoGen is orthogonal to these techniques and can incorporate them for additional gains.
Linear and low-bit attention. Another direction involves linear attention (Cai al., 2023; Xie et al., 2024; Wang et al., 2020; Choromanski al., 2020; Yu et al., 2022; Katharopou- los et al., 2020), which lowers complexity from quadratic to linear, and low-bit attention (Zhang et al., 2025b; 2024), which operates in reduced precision to accelerate attention module. Sparse VideoGin is orthogonal to both approaches:
Efficient system implementation. System-level optimiza- tions focus on dynamic batching (Kodaira al., 2023; Liang et al., 2024), caching (Chen et al., 2024b; Zhao et al., 2024b), or hybrid strategies (Lv et al., 2024; Liu et al., 2024a). While
3

---

Sparse VideoGen: Accelerating Video Diffuusion Transformers with Sparsity
(o) SVG Attention Workflow
(b) Per-head Online Profiling
Spatial Head
MSE
Spatial
Spatial Head
7
x
OR
12
Temporal
Full Attention
Headno
Temporal Head
Temporal Head
Figure 4. Overview of SVG framework. (a) During generation, SVG adaptively classifies each attention head as either spatial head or a temporal head and applies a dedicated sparse attention computation accordingly. (b) This adaptive classification is driven by online profiling strategy, which extracts small portion of Q. denoted Qp. to perform both spatial temporal attention computations. SVG then selects the attention patter that yields the minimal MSE compared to full attention, ensuring accurate classification.
it can be combined with techniques like FP8 attention while still benefiting from the video-specific spatial and temporal sparsity in video diffusion models.
In addition to the spatial and temporal patterns, we observe that the text prompts and the first frame hold significant attention scores for both spatial and temporal head, which aligns with previous investigations (Xiao et al., 2024b; Shen et al., 2024; Su et al., 2025). Therefore, we include these tokens in both the spatial and temporal head.
3. Motivation and Analysis
3.1. 3D Full Attention shows instinet sparsity
3.2. Sparse attention achieves lossless accuracy
We identify that 3D full attention possess inherent sparsity, characterized by different distinct sparse patterns tailored for diffferent functions (Xiao et al., 2024a). We deeply inves- tigate the sparsity nature across various text-to-video and image-to-vider models and identify two types of attention heads based on sparse patterns: Spatial Head and Temporal Head, as shown in Figure. 3.
Furthermore, we find that directly applying sparse patterns to corresponding heads does not hurt the quality of gener- ated videos. We demonstrate this by evaluating Cog Vigei v1.5 and Hunyuan Video VBench (Huang et al., 2023) with sparse attention. We determine the sparse pattern by computing full attention along with two different sparse mechanisms (spatial head and temporal head) for each at- tention head and denoising step. The sparse pattern with the lowest mean squared error (MSE) relative to full attention is chosen for further inference. This approach achieves a PSNR over 29, showing that the right sparse pattern main- tains the high quality of generated videos.
Spatial Head. As illustrated in Figure 3(a-b), spatial head primarily focuses its attention scores spatially-local tokens. This leads to the attention map exhibiting block-wise layout. Since pixels within the same frame are tokenized into contiguous tokens, spatial head attends exclusively to pixels within the same frame and its adjacent frames. This property essential for maintaining the spatial consistency of generated videos. In spatial head, the block size relates to the number tokens per frame.
However, despite its accuracy. this strategy does not provide practical efficiency benefits, as full attention computation is still required to determine the optimal sparse pattern. We will address this issue in See 4.1.
Temporal Head. In contrast, temporal head exhibits slash- wise layout with constant interval (Figure 3(c-d)). Since each frame is tokenized into a fixed number of tokens L, pixels occupying the same spatial position across different frames are arranged at a stride of L. Consequently, temporal head captures information from the token with the same spa- tial position across multiple frames. This pattern is impor- tant for ensuring temporal consistency in video generation's
3.3. Sparse attention promises theoretical speedup
Instead of computing full attention, sparse attention selec- tively processes only the important tokens based on sparsity patterns, leading significant computational savings. We
training data consists of slow-mooe mww-mooionnnoo vidcos, making the temporal head's focus on tokens with the same spatial position in several frames adequate to maintain temporal consistency.
We hypothesize that this occurs because the majority the

---

Sparse VideoGen: Accelerating Video Difffusion Transformers with Spatial-Temporal Sparsity
Algorithm 1 Online Profiling Strategy
Fare I
Freaz
Tokes
Tolesa
Tikea
Tokes
Tikes
Tayour
Raaee
Rrate
France
Qr V, 0: [B, H, S, DJ query, key, value, output Total Token Number E.g., 19k Sampled led Taker Number. B.g-r 32
Phemys
Transfomaatoo
Prumpe
Ter
Tae:
Sample the Indices
t:
Tikes
1aken
Proeppt
Tokre
Frame 2
Frasse
Procpee
Q1 Qli, ", indices, 1]
indi cen somple, indi cas (S, t) (t,))
Tikee
Tokes
Frase
Takee
Frane
Get the atcertion maska mask_spatial aaa_apatial_maskl [t, indices, :] mask_tempozal gen_temperal Iir ;, indices,
lokes 3
Tokee
Pranse
Takre
Prrms 7
(a) Non-contiguous layout
Contignous layout
Compute sampled attention score
Hardware inefficient
Hardware efficient
Shape: 18, H,
fulll mask_attention
K, v, None)
o_spatiel - mask ien K, V, mask spatiel)
Figure 5. Visualization of hardware-effie efffccient layout transforma- tion. (a) Non-contiguous sparsity layout of temporal head, which is hardware inefficient due the contiguous layout required by hardware accelerators. (b) Contiguous layout generated by trans- posing the token-major tensor into frame-major one, which can be efficiently handled by black sparse attention.
o_temparal mask_attention [0_1, V, mask_tomporsl)
Calculate MSE and get best mask
Shaps: [3,
MSR_s tull sp norm () mean (dim= (2, 31) MBE_t full o_tempozal) .norm() mean (dim-(2,
best_mask_conffi (MSE_s NS2_t)
analyze the theoretical computation saving below.
generation quality.
Given a model configuration of hidden dimension H, num- ber of tokens per frame L, and number of total frames N, the total computation (FLOPS) for each full attention is 2 2. (LN)²- H = 4 H. For spatial head, assuming each head only attends to nearby e, frames, the computation reduced (2 11 c,NV, resulting in a sparsity of 5t For temporal head, assuming each token only at- tends e tokens across all the frames, the computation is (2 2 H) L. with sparsity of 7. Since both and are typically much smaller compared to N and L respectively, the sparsity can easily achieve 30%. E.g.. the aforementioned Cog VideoX-v1. ----5V-V achieves sparsity of 31% % for both spatial and temporal head while maintaining an average of 29.99 PSNR.
4.1. Online profiling strategy for sparsity identification
As discussed in Sec 3.1, all attention heads can be clas- sified and sparsified into spatial head and temporal head. However, we find that such sparse patterns can be highly dynamic across different denoising steps and input data. E.g., a certain head can be a spatial head for one prompt while being a temporal head given another. This dynamic nature necessitates an efficient online sparsity identification method, which classifies attention heads on the fly without extra overhead.
To this end, SVG proposes an online profiling strategy. In- stead computing the entire full attention to identify sparse attention, SVG only samples a subset of input rows (x%) and calculates results with both the spatial and temporal sparsity patterns. By choosing the one with the lower MSE compared to full attention, SVG can efficiently approximate the oracle identification method discussed in Sec 3.2. We detail the profiling process in Algorithm 1.
Despite the theoretical speedup, the temporal head can not be directly translated into real speedup since the pattern is hardware-inefficient. We will discuss our practical solution in Sec. 4.2 and prove it can achieve theoretical speedup in Sec. 5.3. Note that we do not include the text prompts and first frame in the theoretical calculation for simplicity, as they are constant and small compared to the remaining part.
To demonstrate the effectiveness of the proposed method, we conduct a sensitivity test on profiling ratio x with CogVideoX-vV.5-IIV As shown in Table 3, profiling only 1% can achieve up to 31.1 PSNR, with only 3% runtime overhead compared to full attention.
4. Methodology
In this section, we introduce SVG, a training-free framework designed to exploit the sparse patterns 3D full attention while addressing practical deployment challenges through careful design. To identify sparse patterns, SVG employs an online profiling strategy (Sec 4.1). To effectively utilize sparsity, SVG introduces hardware-efficient layout trans- formation, which enables real-world hardware acceleration (Sec 4.2). Additionally, by integrating techniques such as customized kernels and quantization (Sec 4.3), SVG signifi- cantly accelerates video generation without compromising
4.2. Hardware-efficient layout transformation
Despite the high sparsity in attention computation, speedups are limited without hardware-efficient sparsity layout (Ye et al., 2023; Zheng et al., 2023). For instance, NVIDIA's Tensor Core, matrix-matrix multiplication accelerator, requires at least 16 contiguous elements along each dimension to use. However, temporal head exhibits a sparse layout of non-contiguous elements with a stride of I. (i.e.,
5

---

Sparse VideoGen: Accelerating Video Diffusson Transformers with Spatial-Tempor Sparsity
Table Quality and efficiency henchmarking results of SVG and other baselines.
Type
Method
Quality
Efficiency
PSNR
S
LPIPS
ImageQual
SubConsist
FLOPS
Latency
Speedup
12V
II (720p. 10s, 80 frames)
70.09%
95.37%
147.87 PFLOPS
528s
1x
DiTFastArto (Spatial-only)
24.591
0.836
0.167
70.44%
95.29%
78.86 PFLOPs
338s
1.56x
Temporal-only
23.839
70.37%
95.13%
70.27 PFLOPs
327s
Minference
22.489
0,743
0.264
58.85%
87.38%
84.89 PFLOPs
357s
1.48X
PAR
23.234
0.842
0.145
69.18%
95.42%
105.88 PFL.OPs
374s
1.41x
Ours
28.165
0,915
0.104
70.41%
95.29%
74.57 PFLOPs
237s
2.23x
T2V
CogVideoX-w1.. 1.5 (720p, 10s, 80 frames)
62.42%
98.66%
147.87 PFLOPs
528s
AiTF Spaatillonly)
23,202
62.22%
78.86
1.56x
Temporal-only
23.804
0.811
0.198
62.
98.53%
70.27 PFT PFLOPs
327s
1.61x
Minference
22.451
0.691
0.304
54.87%
91.52%
84.89 PELOPS
3576
1.48%
PAB
22.486
0.740
0.234
$7.32%
98, 98.76%%
105.88 PFL.Oss
374s
1.41x
29.989
63.01%
98.67%
PHLOPS
2.28x
T2V
Honyuan (720p. 5.33s, 128 frames)
66.11%
93.69%
612.37 PFLOPs
22538
DiTFasiAUn (Spatial only)
21.416
0.646
0.331
67.33%
90.10%
260.48 PFLOPs
1238s
1.82%
Temporal-oaly
25.851
0.857
0.175
62.12%
98.53%
259.10 PFLOPs
1231s
1.83x
Minlerence
23.157
0.823
0.163
63.96%
91.12%
293.87 PFLOPs
1417s
1.59%
29.546
0.907
0.127
65.90%
93.51%
259.79 PFLOPs
11718
1.92x
Ours + FP8
29.452
0.906
0,128
65,70%
93.51%
259.79 PFLOPs
968s
2.33x
number of tokens per frame). This sparsity pattern prevents effective utilization of Tensor Core, thereby constraining overall efficiency.
5. Experiment
5.1. Setup
Models. We evaluate SVG open-sourced state-of-the-an video generation models including
To tackle this, SVG introduces layout transformation strat- egy that transforms the sparsity layout of temporal head into hardware-efficient one. As illustrated in Figure 5, this strategy transposes token-major tensor into frame- major one, which makes the tokens across different frames into a contiguous layout. Such transformation maintains a mathematically equivalent output as attention computa- tion associative (Dao et al., 2022; 2019). We ablate the effectiveness of the proposed method in Sec 5.3.
Cog VogVidexX-v.55-TNN and Hunyuan Video-TTi to generate 720p resolution videos. After 3D VAE, Cog VideoooVv-v.. consumes 11 frames with 4080 tokens per frame in 3D full attention, while Hunyuan Video Video works on 33 frames with 3600 tokens per frame.
Metrics. We assess the quality of the generated videos us- ing the following metrics. We use Peak Signal-to-Noise Ratio (PSNR), Learned Perceptual Image Patch Similarity (L.PIPS) (Zhang et al., 2018), Structural Similarity Index Measure (SSIM) to evaluate the generated video's similar- ity, and use VBench Score (Huang al., 2023) to evaluate the video quality, following common practices in commu- nity (Horé & Ziou, 2010; Zhao et al., 2024b; Li* et al., 2025; Li et al., 2024). Specifically, we report the imaging quality and subject consistency metrics, represented by ImageQaul and SubConsist in our table.
4.3. Other optimizations
Efficient kernel customization. We notice that current implementations of QK-norm and RoPE suffer from per- formance issues, due to limited parallelism small head dimensions (e.g., 64 in Cog TideoX-v1.5) Therefore, we customize those operations with CUDA by sub-warp re- duction implementation, providing up to 5x speedup com- pared to torch implementation (see Table 2). We also use Tri- ton to implement fused online profiling strategy and layout transformation kernels, followed by a block sparse attention kernel with FlashInfer (Ye al., 2025).
Datasets. For Cog VideoX-v1.5, we generate video using the VBench dataset after prompt optimization, as suggested by VogVideoX (Yang et al., 2024c). For Hunyuan Video, we benchmark our method using the prompt in Penguin Video Benchmark released by Hunyuan Video (Kong et al., 2024).
Quantization. We further incorporate FP8 quantization into sparse attention (Zhang et al., 2025b; 2024; Zhao et al., 2024c), which further boosts up to 1.3x throughput with minimal accuracy drop as shown in Table We also cus- tomize an attention kernel that supports both FP8 quantiza- tion and block sparse computation.
Baselines. We compare SVG against sparse attention algorithms DiTFastAttn (Yuan et al., 2024) and MInfer- ence (Jiang et al., 2024). As DiTFastAttn can be considered as spatial head only algorithm, we also manually imple- ment a temporal head only baseline named Temporal-only attention. We also include mache-based DiT acceleration
6

---

Sparse VideoGen: Accelerating Video Diffuusion Transformers with Sparsity
Dense
CogVideoX-va SVG
(a) Prompt: A blue boat is navigating in the occan next to cruise ship.
(b) Prompt: A book on fire with flames coming out of it.
Hunyuan VinyyannVidoo Video
Dense
SVG
Prompt: Several swallows carrying mud to build nests under the caves, low angle shot.
(d) Prompt: Several ducks lying in the mud pit, occasionally preening their feathers leisurcly.
Figure 6. Examples of generated videos by SVG and original implementation CogVideoX-v1.5-12) and Hunyuan Video-T2V. We showcase four different scenarios: (a) minor scene changes, (b) significant scene changes, (c) rare object interactions, and (d) frequent object interactions. SVG produces videos highly consistent with originals in all cases, maintaining high visual quality.
Othes
Nome
Lincar
Amtetiinn
5.2. Quality evaluation
Baseline
Efficient
We evaluate the quality of generated videos by SVG com- pared to baselines and report the results Table 1. Results demonstrate that SVG consistently outperforms all base- line methods in terms of PSNR, SSIM, and LPIPS while achieving higher end-to-end speedup.
Kernel
Sparse
1
Attaation
FPS
1.21x
(SVG)
480 960 1440
1920
2400 ()
Specifically, SVG achieves an average PSNR exceeding 29.55 on Hunyuan Video and 29.99 on Cog Vog T2V, highlighting its exceptional ability to maintain high fidelity and accurately reconstruct fine details. For visual understanding of the video quality generated by SVG, please refer to Figure 6.
Figure 7. The breakdown of end-to-end end runtime of Hunyuan Video when generating a 5.3s, 720p video. SVG effectively reduces the end-to-end inference time from 2253 seconds to 968 seconds through system-algorithm co-design. Each design point contributes considerable improvement, with total 2 speedup.
SVG maintains both spatial and temporal consistency by adaptively applying different sparse patterns, while all other baselines fail. E.g., since the mean- mean-poolin pooninnooing block sparse cannot effectively select slash-wise temporal sparsity (see Figure 3), MInference fails to account for temporal depen- dencies, leading substantial PSNR drop. Besides, PAB skips computation of 3D full attention by reusing results from prior layers, which greatly hurts the quality.
algorithm PAB (Zhao et al., 2024b) as a baseline.
Parameters. For MInference and PAB, we use their official configurations. For SVG, we choose c, 4 frames and 1224 tokens for Cog while c, 10 frames and as 1200 tokens for Hunyuan Video. Such configurations lead to approximately 30% sparsity for both spatial head and temporal head, which is enough for lossless generation in general. We skip the first 25% denoising steps for baselines as they are critical to generation quality, following previous works (Zhao et al., 2024b; Li et al., 2024; Lv et al., 2024; Liu et al., 2024a).
Moreover, SVG is compatible with FP8 attention quanti- zation, incurring only a 0.1 PSNR drop on Hunyuan Video. Such quantization greatly boosts the efficiency by 1.3x. Note that we do not apply FP8 attention quantization on Cog Vog as its head dimension of 64 limits the arithmetic intensity, offering no on-GPU speedups.
Visualizations. We present a comparison of the videos generated by Dense Attention and Sparse VideoGen Ap- pendix B. Additionally, real video samples are available on Google Drive and can be accessed here.

---

Sparse VideoGen: Accelerating Video Diffuusion Transformers with Spatial-Temporal Sparsity
Original
Ours (w/ transform)
Thooretical
Table Inference speedup customized QK-norm and RoPE compared to PyTorch implementation with different number of frames. use the same configuration of Vog i.e. 1080 tokens per frame, 96 attention heads.
3200
Attemnin's Latency
Lateney (mns)
2400
5.5%%
3,66x
2,12x
1.53x
Frame Number
8
9
10
11
1600
1,66x
800
QK-norm
7.44x
7.45x
7.46x
7.47x
RoPE
14.50x
15.23x
15.93x
16.47x
0.17
0.27
0.47
0.67
0.87
1.0
Sparsity
Figure 8. Latency comparison of different implementations of sparse attention. Our hardware-efficient layout transformation optimizes the sparsity pattern of temporal head for better con- tiguity, which is 1.7x faster than naive sparse attention (named original), approaching the theoretical speedup.
Table 3. Sensitivity test online profiling strategy ratios. Profiling just 1% tokens achieves generation quality comparable to the oracle (100%) while introducing only negligible overhead.
Ratios
PSNR 1 SSIM f LPIPS +
CogVideoX-v1.5-11 (720p, 10s, 80 frames)
profiling 0.1%
30.791
0.941
0.0799
5.4. Sensitivity test
profiling 1%
31.118
0.945
0.0757
profiling 5%
31.008
0.944
0.0764
In this section, we conduct a sensitivity analysis on the hy- perparameter choices of SVG, including the online profiling strategy ratios (Sec 4.1) and the sparsity ratios e, and ea (Sec 4.2). Our goal is to demonstrate the robustness of SVG across various efficiency-accuracy trade-offs.
profiling 100%
31.324
0.947
0.0744
5.3. Efficiency evaluation
Online profiling strategy ratios. We evaluate the effec- tiveness of online profiling strategy with different profiling ratios on Cog CogVideoX-v1.5 using a random subset of VBench in Table In our experiments, we choose to profile only of the input rows, which offers comparable generation quality comparable to the oracle profile (100% profiled) with negligible overhead.
To demonstrate the feasibility of SVG, prototype the entire framework with dedicated CUDA kernels based FlashAttention (Dao et al., 2022), FlashInfer (Ye et al., 2025), and Triton (Tillet et al., 2019). We first showcase the end-to-end speedup of SVG compared to baselines on an 1LL00-80GB-HER conduct kernel-level efficiency evaluation. Note that baselines adopt FlashAttentiont (Dao et al., 2022).
with CUDA 12.4. Besides, we also
Generation quality over different sparsity ratios. As dis- cussed Sec 3.3, different sparsity ratio of the spatial head and temporal head can set by choosing different e, and Ct, therefore reaching different trade-offs between efficiency and accuracy. We evaluate the LPIPS of Hunyuan Video over a random subset of VBench with different sparsity ra- tios. As shown in Table 4, SVG consistently achieves decent generation quality across various sparsity ratios. E.g., even with a sparsity of 13%, SVG still achieves 0.154 LPIPS. We leave the adaptive sparsity control for future work.
End-to-end speedup benchmark. We incorporate the end- to-end efficiency metric including FLOPS, latency, and cor- responding speedup into Table 1. SVG consistently out- performs all baselines by achieving an average speedup of 2.28x while maintaining the highest generation quality. We further provide detailed breakdown of end-to-end in- ference time Hunyuan Video in Figure 7 to analyze the speedup. Each design point described in Sec 4 contributes significantly to the speedup, with sparse attention delivering the most substantial improvement of 1.81x.
5.5. Ablation study
Kernel-level efficiency benchmark. We benchmark in- dividual kernel performance including QK-norm, RoPE, and block sparse attention with unit tests in Table Our customized QK-norm and RoPE achieve consistently bet- ter throughput across all frame numbers, with average speedup of 7.4x and 15.5x, respectively. For the sparse attention kernel, we compare the latency of our customized kernel with the theoretical speedup across different spar- sity. As shown in Figure 8, our kernel achieves theoretical speedup, enabling practical benefit from sparse attention.
We conduct the ablation study to evaluate the effectiveness of the proposed hardware-efficient layout transformation (as discussed Sec 4.2). Specifically, we profile the la- tency of the sparse attention kernel with and without the transformation under the Humyuan Video configuration. As shown in Figure 8, the sparse attention with layout transfor- mation closely approaches the theoretical speedup. whereas the original implementation without layout transformation falls short. For example, at a sparsity level of 10%, our method achieves an additional 1.7x speedup compared to
8

---

Sparse VideoGen: Accelerating Video Diffuusin Transformers with Spatial-Temporal Sparsity
Chen, J., Ge, C., Xie, E., Wu, Y., Yao, L., Ren, X., Wang. Z., Luo, P., Lu, H., and Li, Z. Pixart-sigma: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. In European Conference on Computer Vision, pp. 74-91. Springer, 2025.
Table Video quality of Hunyuan Video subset of VBench when varying sparsity ratios. LPIPS decreases as the sparse ratio increases, achieving trade-ofts between efficiency and accuracy.
Sparsity4|
0.13
0.18
0.35
0.43
0.52
LPIPSL 0.154
0.135
0.141
0.129
0.116
Chen, P., Shen, M.. Ye, P., Cao, J., Tu, C., Bouganis, C.- s., Zhao, Y., and Chen, T. \delta-dit: A training-free acceleration method tailored for diffusion transformers. arXiv preprint arXiv:2406. 125, 2024b.
the original approach, achieving 3.63x improvement.
6. Conclusion
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, I.., al. Rethinking attention with performers. arXiv preprint arXiv:2009. 14794, 2020.
We accelerate video diffusion transformers by exploiting sparse attention. We reveal that attention heads have in- herent sparsity patterns and we classify them into spatial head and temporal head. We proposed Sparse VideoGen (SVG), training-free method to utilize these sparsity pat- terns for end-to-end efficiency boosts, including an efficient online profiling algorithm and an efficient inference sys- tem. On representative open video diffusion transformers (CogVideoX-vv 11.5 and Hunyuan Video), SVG demonstrates prominent end-to-end speedup without losing quality.
Dao, T., Gu, A., Eichhorn, M., Rudra, A., and Ré, C. Learn- ing fast algorithms for linear transforms using butterfly factorizations. In International conference on machine learning, pp. 1517-1527. PMLR, 2019.
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. Flashat- tention: Fast and memory-efficient exact attention with io- awareness, 2022. URL https: /ttp/v.axvvvv./////////.//..a../ org/ /bbs/ 2205.14135
Impact Statement
Feng. H., Ding, Z., Xia, Z., Niklaus, S., Abrevaya, V., Black, M. J., and Zhang, X. Explorative inbetweening time and space. In European Conference on Computer Vision, pp. 378-395. Springer, 2024.
This paper presents work that aims to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Guo, Y., Yang, C., Rao, A., Liang, Z., Wang, Y., Qiao, Y.. Agrawala, M., Lin, D., and Dai, B. Animatediff: Animate your personalized rext-to-image diffusion models without specific tuning. In ICLR, 2024.
Acknowledgment
We thank Guangxuan Xiao and Zihao Ye for the visualiza-
tions and kernel designs.
Han, C., Wang, Q. Xiong, W., Chen, Y., Ji, H., and Wang. S. Lm-infinite: Simple on-the-fly length generalization for large language models. arXiv preprint arXiv:2308. 16137 2023.
References
Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučié, M., and Schmid, C. Vivit: A vidco vision transformer. In ICCV, 2021.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. NeurIPS, 2020.
Cai, H., Li, J., Hu, M., Gan, C., and Han, S. Efficientvit: Lightweight multi-scale attention for high-resolution dense prediction. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pp. 17302- 17313,0033
Hong. W., Ding, M., Zheng, W., Liu, X., and Tang, J. Cogvideo: Large-scale pretraining for text-to-video gen- eration via transformers. In ICLR, 2023.
Horé, A. and Ziou, D. Image quality metries: Psnr vs. ssim. In 2010 20th International Conference on Pattern Recognition, pp. 2366-2369, 2010. doi: 1091109/ 2010.579.
Cai. H., Li, M.. Zhang, Q., Liu, M.-Y., and Han, S. Condition-aware neural network for controlled image generation. In CVPR, 2024.
Huang, Z., He, Y., Yu, J., Zhang, F., Si, C., Jiang, Y., Zhang, Y., Wu, T., Jin, Q. Chanpaisit, N., Wang, Y., Chen, X., Wang, L., Lin, D., Qiao, Y., and Liu, Z. Vbench: Com- prehensive benchmark suite for video generative mod- els, 2023. URL https: / /arxiv.... org/ wtgg/bb//////ooooooo/ogo////b/b/ 17982.
Chen, J., Cai, H., Chen, J., Xie, E., Yang, S., Tang, H., Li, M., Lu, Y., and Han, S. Deep compression autoen- coder for efficient high-resolution diffusion models. arXiv preprint XrXiv:2410. 10733, 2024a.
abs/ abs/2311. 2311.
9

---

Sparse VideoGen: Accelerating Video Diffusson Transformers with Spatial-Temporal Sparsity
Jang, H., Li, Y., Zhang, C., Wu, Q., Luo, X., Ahn, S., Han, Z., Abdi, A. H., Li, D., Lin, C.-Y., et al. Minference 1.0: Accelerating pre-filling for long-context Ilms via dynamic sparse attention. arXiv preprint mYXiv:4407.22490, 2024.
Liu, X., Zhang, X., Ma, J., Peng, J., and Liu, Q. Instaflow: One step is enough for high-quality diffusion-based text- to-image generation. In International Conference on Learning Representations, 2024c.
Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are rmns: Fast autoregressive transformers with linear attention. In International conference on ma- chine learning, pp. 5156-5165, PMLR, 2020.
Liu, Z.,, Desai, A., Liao, F., Wang, W., Xie, V., Xu, Z.,, Kyril- lidis, A., and Shrivastava, A. Seissorhands: Exploiting the persistence of importance hypothesis for Imm kv cache compression at test time, Advances in Neural Information Processing Systems, 36, 2024d.
Kodaira, A., Xu, C., Hazama, T., Yoshimoto, T., Ohno, K., Mitsuhori, S., Sugano, S., Cho, H., Liu, Z., and Keutzer, K. Streamdiffusion: A pipeline-level solu- tion for real-time interactive generation. arXiv preprint MrXiv:2312.124911 2023.
Lu, C.,, Zhou, Y., Bao, F., Chen, J., Li, C.,, and Zhu, Dpm- solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. In NeurIPS, 2022a.
Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, J. Dpm- solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:221 1.01095, 2022b.
Kong, W., Tian, Q., Zhang, Z.,, Min, R., Dai, Z.,, Zhou, J., Xiong, J., Li, X., Wu, B., Zhang, J., et al. Hunyuan- video: A systematic framework for large video generative models. arXiv preprint arXiv:24/2.03603 2024.
Luo, S., Tan, Y., Huang, L., Li, J., and Zhao, H. La- tent consistency models: Synthesizing high-resolution images with stw-step inference. arXiv preprint arXiv: 2310.04378, 2023.
Li, M., Cai, T., Cao, J., Zhang, Q. Cai, H., Bai, J., Jia, Y., Liu, M.-Y., Y., Li, K., and Han, S. Distrifusion: Distributed parallel inference for high-resolution diffusion models. In CVPR, 2024.
Lv, Z., Si, C., Song, J., Yang, Z. Qiao, Y., Liu, Z., and Wong, K.-Y. K. Fastercache: Training-free video diffu- sion model acceleration with high quality. 2024.
Li M., Lin Y., Zhang*. Z., Cai, T., Li, X., Guo, J., Xie, E., Meng, C., Zhu, J.-Y., and Han, Svdquant: Absorb- ing outliers by low-rank components for 4-bit diffusion models. In The Thirteenth International Conference on Learning Representations, 2025.
Ma, X., Fang, G., and Wang. X. Deepcache: Accelerating diffusion models for free In CVPR, 2024.
Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.-Y., and
Li, X., Liu, Y., Lian, L., Yang, H., Dong, Z., Kang, D., Zhang, S., and Keutzer, K. Q-diffusion: Quantizing diffusion models. In 2C23.
Ermon, S. SDEdit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2022.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In ICCV, 2023.
Li, Y., Jiang, H., Zhang, C., Wu, Q. Luo, X., Ahn, S., Abdi, A. H., Li, D., Gao, J., Yang, Y., et al. Mmin- ference: Accelerating pre-filling for long-context vlms via modality-aware permutation sparse attention. arXiv preprint 2504. 16083, 2025.
Shen, X., Xiong, Y. Zhao, C., Wu. L., Chen, J., Zhu, C., Liu, Z., Xiao, F., Varadarajan, B., Bordes, F., Liu, Z., Xu, H., Kim, H. J., Soran, B., Krishnamoorthi, R., Elhoseiny, M., and Chandra, Longvu: Spatiotemporal adaptive com- pression for long video-language understanding, 2024. URL https: awwpvvrxvvvv/..////v///... org/ /bs///b//// 22102211.0200 17434.
Liang, F., Kodaira, A., Xu, C.,, Tomizuka, M., Keutzer, K., and Marculescu, D. Looking backward: Streaming video- to-video translation with feature banks. arXiv preprint arXiv:2405. 15757, 2024.
Song, J., Meng. C., and Ermon, S. Denoising diffusion implicit models. In ICLR, 2020.
Liu, F., Zhang, S., Wang, X., Wei, Y., Qiu, H., Zhao, Y., Zhang, Y., Ye. Q.. and Wan, F. Timestep embedding tells: It's time to cache for video diffusion model. arXiv preprint arXiv:2411. 191008, 2024a.
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Liu, H., Yan, W., Zaharia, M., and Abbeel, P. World model on minlion-length video and language with ringattention. arXiv preprint, 2024b.
Song, Y., Dhariwal, P., Chen, M., and Sutskever, I. Consis- tency models. In ICML, 2023.
Su, Z., Shen, W., Li, L., Chen, Z., Wei, H., Yu, H., and Yuan, K. Akvq-vl: Attention-aware kv cache adaptive 2- bit quantization for vision-language models, 2025. URL batps://arxive http:s://arxxv.orrgaa////......
Liu, X., Gong, C., and Liu, Q. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.0300 00003, 2022.
10

---

Sparse VideoGen: Accelerating Video Difffusion Transformers with Spatial-eemporal Sparsity
Tang, J., Zhao, Y., Zhu, K., Xiao, G., Kasikci, B., and Han, S. Quest: Query-aware sparsity for efficient long- context IIm inference, 2024. URL https:/ /arxiv. org/
Ye, Z., Lai, R., Shao, J., Chen, T., and Ceze, L. Sparsetir: Composable abstractions for sparse compilation in deep learning, 2023, URL https:
10774.
Tillet, P., Kung, H.-T., and Cox, D. D. Tri- ton: an intermediate language and compiler for tiled neural network computations. Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, 2019. URL https: http:////..ppssasissssssseeeeeee semanticscholax atgg/CorpustD:1848811
Ye, Z., Chen, L.. Lai, R., Lin, W., Zhang, Y., Wang, S., Chen, T., Kasikci, B., Grover, V., Krishnamurthy, A., and Ceze, L. Flashinfer: Efficient and customizable attention engine for Ilm inference serving, 2025. URL https: / //axxixviv/////////// orgo abs / /2//225111 2501.01005
Yin, T., Gharbi, M., Park, T., Zhang, R., Shechtman, E., Du- rand, F., and Freeman, W. T. Improved distribution match- ing distillation for fast image synthesis. arXiv preprint arXiv:2405 14867, 2024a.
Wang, A., Ai, B., Wen, B., Mao, C., Xie, C.-W., Chen, D., Yu, F., Zhao, H., Yang, J., Zeng. J., et al. Wan: Open and advanced large-scale video generative models. arXiv preprint ar ar.iv:2503.0m 20314, 2025.
Yin, T., Gharbi, M., Zhang, R., Shechtman, E., Durand, F,, Freeman, W. T. and Park, T. One-step diffusion with distribution matching distillation. In CVPR, 2024b.
Wang, S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.0470 2020.
Yu, W., Luo, M., Zhou, P., Si, C., Zhou, Y., Wang, X., Feng, J., and Yan, S. Metaformer is actually what you need for vision. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10819 10829, 2022.
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Ef- ficient streaming language models with attention sinks. arXiv preprint arXiv:2309. 17453, 2023.
Xiao, G., Tang, J., Zuo, J., Guo, J., Yang, S., Tang, H., Fu, Y., and Han, S. Duoattention: Efficient long-context Ilm inference with retrieval and streaming heads. arXiv preprint ar Arxiv:2410. 10819, 2024a.
Yuan, Z., Zhang, H., Lu, P., Ning. X., Zhang, L.. Zhao, T., Yan, S., Dai, G., and Wang. Y. Ditfastatmm: Attention pression for diffusion transformer models, 2024. URL https: / //l/arxivvv/ org/ abs 2400 08552.
Xiao, G., Tian, Y. Chen, B., Han, S., and Lewis, M. Effi- cient streaming language models with attention sinks, 2024b. URL https:: / http//axiv....raaa/////a 2309. 17453.
Zhang, J., Herrmann, C., Hur, J., Cabrera, L. P., Jampani, V., Sun, D., and Yang, M.-H. A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. 2023a.
Xie, E., Chen, J., Chen, J., Cai, H., Tang, H., Lin, Y., Zhang, Z., Li, M., Zhu, L., Lu, Y., et al. Sana: Efficient high- resolution image synthesis with linear diffusion trans- formers. arXiv preprint arXiv:2410. 10629, 2024.
Zhang, J., Huang. H.,, Zhang, P., Wei, J., Zhu, J., and Chen, J. Sageattention2: Efficient attention with thorough out- lier smoothing and per-thread int4 quantization. arXiv preprint arXiv:2411.
2024.
Xu, R., Xiao, G., Huang, H., Guo, J., and Han, S. Xattention: Block sparse attention with antidiagonal scoring, arXiv preprint ar 2r03::2503.1 16428, 2025.
Zhang, J., Wei, J., Zhang, P., Xu, X., Huang, H., Wang, H., Jiang, K., Zhu, J., and Chen, J. Sageattention3: Microscal- ing fp4 attention for inference and exploration of 8-bit training. arXiv preprint arXiv: 1154,, 2025a.
Yang, I., Zhang, 7., Chen, 7., Li, Z.., and Jia, Z. Tidalde- code: Fast and accurate I1m decoding with position per- sistent sparse attention. arXiv preprint 2xXiv:2410.05076. 2024a.
Zhang, J., Wei, J., Zhang, P., Zhu, J., and Chen, Sageatten- tion: Accurate 8-bit attention for plug-and aad--and-ply inference acceleration. In International Conference on Learning Representations (ICLR), 2025b.
Yang, S., Sheng, Y., Gonzalez, J. Post-training sparse attention with double sparsity, 2024b.
E., Stoica, I,, and Zheng, L.
URL https:t/p/// / http/vixvvvv///// org/ abs 2408 07092.
Zhang, J., Xiang, C., Huang, H., Wei, J., Xi, H., Zhu, J., and Chen, J. Spargeattm: Accurate sparse atten- tion accelerating any model inference. arXiv preprint wwX1::2502.1813. 2025c.
Yang, Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu, J. Yang, Y., Hong, W., Zhang, X., Feng, G., et al. Cogvideox: Text-to-video diffusion models with an expert transformer. anXiv preprint 07X0::2408.06072, 2024c.
Zhang, J., Xu, X., Wei, J., Huang, H., Zhang, P., Chendong, X., Zhu, J., and Chen, J. Sageattention2+#:: A more

---

Sparse VideoGen: Accelerating Video Diffusson Transformers with Sparsity
efficient implementation of sageattention2. arXiv preprint arXiv:2505.2/ 2025d.
Zhang. R., Isola, P., Efros, A. A., Shechtman, E., and Wang,
O. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C., et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:34661-34711 2023b.
Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C., Wang, Z., and Chen, B. H2o: Heavy-hitter oracle for efficient generative inference of large language models, 2023c. URL htsps: //arxiv. org/ abs/ 6306 14048.
Zhao, T., Fang, T., Liu, E., Rui, W., Soedarmadji, W., Li, S., Lin, Z., Dai, G., Yan, S., Yang. H., al. Vidit- q: Efficient and accurate quantization of diffusion trans- formers for image and video generation. arXiv preprint NKKv:2406.025444 2024a.
Zhao, X., Jin, X., Wang, K., and You, Y. Real-time video generation with pyramid attention broadcast, 2024b. URL https: /arxiv...riivv.//aa/e/aeeee org/ abs/ 2408 12588.
Zhao, Y., Lin, C.-Y., Zhu, K., Ye, Z., Chen, L., Zheng, S., Ceze, L., Krishnamurthy, A., Chen, T., and Kasikci, B. Atom: Low-bit quantization for efficient and accurate Ilm serving. MI,Sys, 2024c.
Zheng, N., Jiang, H., Zhang, Q., Han, Z., Yang, Y., Ma, L., Yang. F., Zhang, C., Qiu, L., Yang. M., and Zhou, L. Pit: Optimization of dynamic sparse deep learning models via permutation invariant transformation, 2023. URL https: / /arxivvi... org/
12

---

Sparse VideoGen: Accelerating Video Diffussin Transformers with Spatial-Temporal Sparsity
A. A full version of related work
A.1. Efficient Diffusion Models
Diffusion Models function primarily as denoising models that are trained to estimate the gradient of the data distribution (Song & Ermon, 2019; Zhang et al., 2023a). Although these models are capable of generating samples with high quality and diversity, they are known as inefficient. To enhance the efficiency of diffusion models, researchers often focus on three primary approaches: (1) decreasing the number of denoising steps, (2) reducing the model size, and (3) optimizing system implementation for greater efficiency.
Decreasing the denoising steps. The main diffusion models rely on stochastic differential equations (SDEs) that learn to estimate the gradient of the data distribution through Langevin dynamics (Ho et al., 2020; Meng et al., 2022). Consequently, these models generally require numerous sampling steps ( e.g., 1,000). To improve sample efficiency, DDIM (Song et al., 2020) approximates SDE-based diffusion models within an ordinary differential equation (ODE) framework. Expanding on this concept, DPM (Lu et al., 2022a), DPM++ (Lu et al., 2022b), and Rectified Flows (Liu et al., 2022; 2024c) enhance ODE paths and solvers to further reduce the number of denoising steps. Furthermore, Consistency Models (Song al., 2023; Luo et al., 2023) integrate the ODE solver into training using a consistency loss, allowing diffusion models to replicate several denoising operations with fewer iterations. In addition, approaches grounded in distillation (Yin et al., 2024a;b) represent another pivotal strategy. This involves employing a simplified, few-step denoising model to distill a more complex, multi-step denoising model, thereby improving overall efficiency.
Nevertheless, all these approaches necessitate either re-training or fine-tuning the complete models on image or video datasets. For video generation models, this is largely impractical due to the significant computational expense involved, which is prohibitive for the majority of users. In this work, our primary focus is a method to enhance generation speed that requires no additional training.
Diffusion Model Compreesion A common approach to enhancing the efficiency of diffusion models involves compressing their weights through quantization. Q-Diffusion (Li et al., 2023) introduced a W8A8 strategy, implementing quantization in these models. Building on this foundation, ViDiT-Q (Zhao et al., 2024a) proposed a timestep-aware dynamic quantization method that effectively reduces the bit-width to W4A8. Furthermore, SVDQuant (Li* et al., 2025) introduced a -ost-effeetive branch designed address outlier problems in both activations and weights, thus positioning W4A4 as feasible solution for diffusion models. SageAttention (Zhang et al., 2025b) advanced the field by quantizing the attention module INT8 precision via smoothing technique. SageAttention V2 (Zhang et al., 2024; 2025d) and V3 (Zhang et al., 2025a) extended these efforts by pushing the precision boundaries to INT4 hybridized with FP8 or even full FP4. Another common approach is to design efficient diffusion model architectures (Xie et al., 2024; Cai et al., 2024; Chen et al., 2025) and high-compression autoencoders (Chen et al., 2024a) to boost efficiency. Our Sparse VideoGen is orthogonal to these techniques and can utilize them as supplementary methods to enhance efficiency.
Efficient System Implementation In addition to enhancing the efficiency of diffusion models by either retraining the model to decrease the number of denoising steps or compressing the model size, efficiency improvements can also be achieved at the system level. For instance, strategies such dynamic batching are employed in StreamDiffusion (Kodaira et al., 2023) and Stream V2V (Liang et al., 2024) to effectively manage streaming inputs in diffusion models, thereby achieving substantial throughput enhancements. Other approaches include: DeepCache (Ma et al., 2024), which leverages feature caching to modify the UNet Diffusion; A - DiT (Chen et al., 2024b), which implements this mechanism by caching residuals between attention layers in DiT to circumvent redundant computations; and PAB (Zhao et al., 2024b), which caches and broadcasts intermediary features at distinct timestep intervals. FasterCache (Lv et al., 2024) identifies significant redundancy in CFG and enhances the reuse of both conditional and unconditional outputs. Meanwhile, TeaCache (Liu et al., 2024a) recognizes that the similarity in model inputs can be used to forecast output similarity, suggesting improved machine strategy to amplify speed gains.
Despite these advanced methodologies, they often result in the generated output diverging significantly from the original, as indicated by a PSNR falling below 22. In contrast, our method consistently achieves a PSNR exceeding 30, thus ensuring substantially superior output quality compared to these previously mentioned strategies.
13

---

Sparse VideoGen: Accelerating Video Diffusson Transformers with Spatial-Temporal Sparsity
A.2. Efficient Attention
Sparse Attention in LLM Recent studies on sparse attention in language models have identified patterns that reduce computational costs by targeting specific token subsets. StreamingLLM (Xiao et al., 2023) and LM-Infinite (Han et al., 2023) reveal concentration initial and local tokens, highlighting temporal locality in decoding. H2O (Zhang et al., 2023b) and Scissorhands (Liu et al., 2024d) note attention focuses mainly on a few dominant tokens. TidalDecode (Yang et al., 2024a) shows cross-layer attention pattern correlations, aiding in attention sparsity. DuoAltention (Xiao et al., 2024a) and MInference (Jiang et al., 2024) find distinct sparse patterns among attention heads, with varying focus on key tokens and context. MMInference (Li et al., 2025) speedup the vision language models through modality-award permutation. SpargeAttention (Zhang et al., 2025c) and XAttention (Xu al., 2025) propose general sparsity identification algorithms that be applied to all forms of models. Despite their success in LLMs or VLMs, these mechanisms constrained to token-level sparsity and miss the redundancy unique to video data.
Linear and Low-bit Attention Significant advancements have been achieved in enhancing attention efficiency, notably through linear attention (Cai et al., 2023; Xie et al., 2024) and low-bit attention techniques (Zhang et al., 2025b; 2024). Linear attention models, including Linformer (Wang et al., 2020), Performer (Choromanski et al., 2020), MetaFormer (Yu et al., 2022), and LincarAttention (Katharopoulos et al., 2020), reduce the quadratic complexity traditional attention to linear. Low-bit attention approaches decrease computational demands by utilizing lower precision, with SageAttention (Zhang al., 2025b) employing INT8 precision enhance efficiency without notable performance loss.
Sparse VideoGen, sparse attention method, is orthogonal to both linear attention and low-bit attention techniques. Moreover, it can be integrated with low-bit attention methods, such as FP8 attention, to further enhance computational efficiency.
B. Visualization of the generated videos
Video We provide visualization comparison between Dense Attention and Sparse VideoGen on Hunyuan Video and Wan 2.1.
We conduct both Text-to- video generation and Image-to- Video generation under 720p resolution. Results demonstrate that Sparse VideoGen can preserve high pixel-level fidelity, achieving similar generation quality compared with the dense attention.
14

---

Sparse VideoGen: Accelerating Vidco Difflusion Transformers with Sparsity
Dense Attention
Sparse VideoGen
Dense Attention
Sparse VideoGen
Dense Attention
Sparse VideoGen
Dense Attention
Spanse VideoGen
Figure 9, Comparion of Dense Attention and Sparse VideoGen on Hunyuan Vidco Text-to-Videe generation.
15

---

Sparse VideoGen: Accelerating Video Diffuusion Transformers with Spatial-Temponee Sparsity
Dense Attention
Sparse VideoGen
Dense Attention
Sparse VideoGen
Dense Attention
Sparse VideoGen
Dense Attention
Sparse VideoGen
Dense Attention
Sparse VideoGen
Pigure 10. Comparison of Dense Attention and Sparse VideoGen Wan 2.1 Image-to- Vidco generation.
16

---

Sparse VideoGen: Accelerating Video Diffussin Transformers with Spatial-Temporal Sparsity
C. Theoretical Analysis of Sparse Video Attention
Here we provide the theoretical analyses of SVG. Consider video with N frames, each containing tokens. Every token can be encoded by an indices pair (i, where 0 I i N,0 I < F. In the attention map we will flatten these two-dimensional indices into ID vector. Corresponding formula is æ F.
Spatial Head
For spatial head, let 01 as:
denote the threshold for spatial closeness between tokens.
The spatial attention mask is defined
{
if i i2 < a1 otherwise
((2.
For flattened indices x) = i -F + ji and x2 = =1 j2.
T2 F
(x) =
a1
The resulting attention map takes the form of block-banded structures: the attention mask will be on the main diagonal and also on neighboring {aa 1) diagonals.
Temporal Head
For temporal head, let 02 > 0 denote the threshold for temporal closeness between tokens. The temporal attention mask is defined as:
{
1 if ji - jj2l < a2. otherwise
f1((i1.11) (12.12))
For flattened indices 1 = 1 F and 12 = 12. F.FFF. j2.
x2) (ar mod F) ( mod F)] a 3k << 2N 11,111 -x2)
-F 02
The resulting attention map forms 2N slanted diagonals that align along constant column index differences. These diagonals, often referred to as "slashes," correspond to the token positions sharing similar temporal locations across frames. The corresponding points in the attention matrix yield "slash-wise" pattern with width d2.
17


Reviews:
[{"review_id": "NqlyTujtoo", "reviewer": "Reviewer_usfF", "summary": "This paper proposes Sparse VideoGen, a training-free method to optimize and accelerate video diffusion DiTs through an online profiling strategy and hardware-friendly implementation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable suggestions and questions. Below, we address each point raised.\n\n> Q1: Can the attention layers in multi-view diffusion models also be accelerated?\n\nAnswer: Recent multi-view diffusion models such as MVDream [1] and CAT3D [2] have introduced 3D attention mechanisms to improve cross-view consistency and spatial coherence. MVDream takes an initial step in this direction but is limited by a relatively short context length (≤ 4k tokens), which diminishes the practical benefits of attention acceleration.\nCAT3D adopts finer-grained image representations and extends the attention span in 3D. It goes further than MVDream by applying 3D attention over longer sequences and finer feature maps. CAT3D also highlights the critical scalability issue: applying full 3D attention to high-resolution feature maps (e.g., 64×64) leads to prohibitively large sequence lengths—up to 32k tokens—making the approach computationally expensive. As a result, CAT3D restricts full 3D attention to lower-resolution feature maps (32×32 and below), where the overhead remains manageable.\n\nThis design underscores the computational bottlenecks present in naive 3D attention and presents a clear opportunity for approaches like SVG. \nSince SVG can leverage the spatial and temporal redundancy, it could potentially enable efficient 3D full attention at larger resolutions, unlocking performance and scalability gains that current methods avoid due to cost. \nHowever, CAT3D does not open-source their weights. We are willing to validate the effectiveness of SVG on them when the weights are available.\n\n[1] Shi, Yichun, et al. \"Mvdream: Multi-view diffusion for 3d generation.\" arXiv preprint arXiv:2308.16512 (2023).\n\n[2] Gao, Ruiqi, et al. \"Cat3d: Create anything in 3d with multi-view diffusion models.\" arXiv preprint arXiv:2405.10314 (2024).", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "diffusion model acceleration", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\n\n1. The observation that the attention layers in video diffusion can be divided into two types—pure spatial and pure temporal attention—is interesting and insightful.\n\n2. The proposed online profiling strategy is well-defined, as the attention type in DiTs is highly dynamic depending on input data and denoising steps. Such a strategy is necessary to handle various scenarios effectively.\n\n3. The proposed sparse attention acceleration is demonstrated to be efficient, as shown in Table 1.\n\n4. The presentation is clear and easy to follow.\n\nWeaknesses:\n\nI do not have significant concerns. However, I am curious whether the observation that attention layers can be divided into solely spatial or temporal types is applicable to all diffusion models. For example, can the attention layers in multi-view diffusion models also be accelerated using the proposed method?", "other_comments_or_suggestions": "I am not familiar with this field.  However, I think the proposed training-free method is effective in accelerating the large video diffusion model. Therefore, I tend to give a weak accept.", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7GSilOQ2BV", "reviewer": "Reviewer_JiXV", "summary": "This paper addresses the critical challenge of computational inefficiency in Diffusion Transformers (DiTs) for video generation, caused by the quadratic complexity of 3D full attention operations. To tackle this issue, the authors propose Sparse VideoGen (SVG), a training-free inference optimization framework leveraging the inherent sparsity within 3D attention. Specifically, they reveal attention heads can be dynamically classified into two categories: Spatial Heads, dominated by spatially-related tokens within frames, and Temporal Heads, dominated by temporally-related tokens across frames. SVG employs an online profiling strategy to dynamically identify these sparse patterns during inference, coupled with a hardware-efficient tensor layout and customized kernel implementations.\n\n\n###Post-rebuttal\n\nThank the authors for the rebuttal. I have no further questions and will keep my original score, leaning towards accepting the paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments and feedback. We respond to the questions below.\n\n> Experimental Designs Or Analyses 1: The submission currently does not provide actual video demos.\n\nAnswer: As requested by the reviewer, we include actual video demonstrations at the following anonymous link:\nhttps://drive.google.com/drive/folders/1jhEpZ69bKfyZWmoy63iS3FhECNnX-AZU?usp=sharing. Our method achieves **nearly lossless** video quality.\n\n>  Experimental Designs Or Analyses 2: How does the method perform in more challenging scenarios? Such as those involving complex spatial-temporal dynamics.\n\nAnswer: As suggested by the reviewer, we additionally include some representative video results of complex spatial-temporal dynamics in the following anonymous link: https://drive.google.com/drive/folders/1xscj4RcaOE-PeAXq6yiiUFqnTJqS5kaV?usp=sharing. Our method maintains **high video fidelity (PSNR=28.624)** under these challenging cases, such as scenes involving rapid motion and dynamic multi-person interactions.\n\n> Weakness 1: Insufficient clarity regarding the implementation specifics of the proposed online profiling strategy. When this profiling occurs, and does the profiling dynamically vary across different prompts, layers, or diffusion timesteps?\n\nAnswer: Thank you for the valuable feedback. In our implementation, online profiling is performed at **every diffusion step** and **every layer** right before the 3D full-attention computation. This ensures that SVG dynamically adapts to the different prompts and spatial-temporal patterns at each step.\n\nWe would like to emphasize that dynamic online profiling is essential to our method. We find that attention patterns exhibit **substantial variation**, especially across different steps and prompts. Static profiling patterns—whether fixed across layers, steps, or prompts—will lead to **performance degradation**, as demonstrated in Table 1, measured by PSNR, SSIM and LPIPS. The results are averaged over 10 examples. Our SVG uses the dynamic profiling strategy, consistently outperforming all static configurations.\n\nTable 1\n|                        | PSNR (↑) | SSIM (↑) | LPIPS (↓) |\n|------------------------|----------|----------|-----------|\n| Static Across Layers   | 22.365   | 0.846    | 0.187     |\n| Static Across Steps    | 28.007   | 0.866    | 0.131     |\n| Static Across Examples | 26.955   | 0.803    | 0.185     |\n| SVG (Ours)             | **29.062**   | **0.879**   | **0.122**     |\n\nStatic profiling patterns lead to significant quality degradation due to inaccurate identification of attention patterns. To illustrate this, we analyze the variation in attention patterns across different layers, diffusion steps, and data examples (prompts) by computing cosine similarity. Specifically, we employ oracle profiling to identify attention patterns for each head across different configurations. For example, we calculate the variation across different layers by computing a tensor with shape [num_layers, num_heads]. Each element in this tensor is labeled either -1 (indicating a spatial head) or 1 (indicating a temporal head). By calculating the cosine similarity between columns of this tensor, we quantify the variation of attention patterns. As demonstrated in Table 2, attention patterns exhibit **substantial variation**, necessitating a dynamic online profiling method.\n\nTable 2\n|                   | Across Layers | Across Steps | Across prompts |\n|-------------------|---------------|--------------|----------------|\n| Cosine Similarity | 0.2597        | 0.7158       | 0.7238         |", "claims_and_evidence": "The claims presented in the submission are supported by experimental evidence.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are appropriate.", "theoretical_claims": "The paper does not contain explicit theoretical claims, formal proofs, or mathematical theorems that require verification. Therefore, there were no theoretical proofs to check.", "experimental_designs_or_analyses": "Here are two key questions highlighting potential experimental concerns:\n\n**Lack of Real Video Demonstrations**: The submission currently does not provide actual video demos, making it challenging to clearly evaluate the impact of sparse attention acceleration on generated video quality. It is better for authors to provide actual generated video examples (particularly corresponding to experiments shown in Figure 1 and Figure 6)? These demos would clarify how much video quality is affected by your acceleration method.\n\n**Performance on Challenging Scenes**: How does the method perform in more challenging scenarios? Specifically, could the authors clearly demonstrate the performance on difficult cases, such as those involving complex spatial-temporal dynamics (e.g., rapid movements, human interactions such as group dancing, fighting, or dynamic sports scenes)? Providing video demonstrations of challenging scenes—such as a football match, basketball gameplay, rope skipping, fast-paced dancing, or multi-person interactions—would strengthen the claims regarding spatial and temporal consistency.", "relation_to_broader_scientific_literature": "The key contributions of this paper are closely related to the broader literature on efficient inference techniques for generative models, particularly diffusion-based video generation.", "essential_references_not_discussed": "I did not identify any essential related works that are missing from the current citations.", "other_strengths_and_weaknesses": "**Additional Weakness**: Insufficient Clarity in Implementation Details. The manuscript currently lacks clarity regarding the implementation specifics of the proposed online profiling strategy. It is unclear exactly when this profiling occurs—does the profiling need to be computed at every diffusion step, or is it computed just once or intermittently? Moreover, does the profiling dynamically vary across different prompts, layers, or diffusion timesteps? Clarifying and validating this would strengthen the paper. Specifically, could the authors provide experimental evidence or analysis showing the stability or variability of profiling patterns under different conditions (e.g., across prompts, layers, or timesteps)?", "other_comments_or_suggestions": "I have no additional comments or suggestions.", "questions_for_authors": "Please answer the questions in the experimental analyses and Implementation Details", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1QWiynLxmT", "reviewer": "Reviewer_bnzg", "summary": "This paper proposes Sparse VideoGen for efficient (accelerated) video generation, a method which can be applied to existing video generative diffusion models which use diffusion transformers. The paper mainly targets video generative models which operate on the spatio-temporal latents. The key idea is to classify the inherent sparsity in the attention heads into either the spatial head or the temporal head based on their sparsity patterns, and to sparsity the attention operation based on the classified result. The authors notice that the sparsity patterns can be dynamic i.e., the same head can exhibit varying sparsity patterns depending on the prompt or timestep. To handle this, the authors propose an online profiling strategy to determine the inherent sparsity pattern on-the-fly. Combined with the proposed hardware-aware tensor layout transformation and customized kernel implementations of the QK-norm and rotary positional embedding (RoPE), SVG achieves SoTA speedup of existing (spatio-temporal) video generative models.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging our contributions and the insightful comments. We respond to the questions below. We will revise the paper's final versions based on the reviewers' comments.\n\n> How does the performance / efficiency compare to models interleaving spatial attention and temporal attention (2D + 1D Attention)?\n\nAnswer:\nRecent state-of-the-art models [1,2,3] have shifted toward using 3D full attention, which significantly improves performance at the cost of higher computational complexity. In contrast, although interleaved spatial and temporal attention [4,5,6] is computationally efficient, it suffers from **limited modeling capability**—especially in capturing complex spatiotemporal dependencies—leading to lower generation quality.\n\nOur method delivers video **generation quality on par with full 3D attention** models while operating at a **significantly lower computational cost**. Compared to 2D + 1D approaches, SVG achieves not only substantial improvements in visual fidelity but also stronger temporal consistency. In terms of efficiency, SVG sits between 2D + 1D and full 3D attention—achieving a compelling **balance of speed and quality**. For example, as shown in the table below, OpenSora-v1.2—a representative open-sourced model utilizing 2D+1D attention—achieves significantly lower VBench scores than Wan2.1, a 3D model of similar parameter size. By reducing attention redundancy, SVG effectively decreases attention FLOPs by 2.55x while preserving superior generation quality. Thus, SVG presents a practical and efficient alternative for high-quality video generation.\n\n| Model         | Attention Pattern | #Params | Attention FLOPS | VBench |\n|---------------|-------------------|---------|-----------------|--------|\n| OpenSora-v1.2 | 2D+1D             | 1.1B    | 8.4×10^12       | 77.59  |\n| Wan2.1        | 3D                | 1.3B    | 7.4×10^13       | 82.91  |\n| Wan2.1        | SVG               | 1.3B    | 2.9×10^13       | 82.91  |\n\n[1] Kong, Weijie, et al. \"Hunyuanvideo: A systematic framework for large video generative models.\" arXiv preprint arXiv:2412.03603 (2024).\n\n[2] Wang, Ang, et al. \"Wan: Open and Advanced Large-Scale Video Generative Models.\" arXiv preprint arXiv:2503.20314 (2025).\n\n[3] Yang, Zhuoyi, et al. \"Cogvideox: Text-to-video diffusion models with an expert transformer.\" arXiv preprint arXiv:2408.06072 (2024).\n\n[4] Zheng, Zangwei, et al. \"Open-sora: Democratizing efficient video production for all.\" arXiv preprint arXiv:2412.20404 (2024).\n\n[5] Lin, Bin, et al. \"Open-sora plan: Open-source large video generation model.\" arXiv preprint arXiv:2412.00131 (2024).\n\n[6] Ma, Xin, et al. \"Latte: Latent diffusion transformer for video generation.\" arXiv preprint arXiv:2401.03048 (2024).", "claims_and_evidence": "Yes, the claims made in the submission are supported by clear and convincing evidence e.g., lossless accuracy in 3.2 supported by PSNR value, quality and efficiency results in Table 1, and sensitivity test on online profiling strategy ideas in Table 3.", "methods_and_evaluation_criteria": "The proposed method and the evaluation criteria (videos generated from prompts provided in VBench, quality: SPNR, SSIM, LPIPs, VBench scores, and Efficiency: FLOPs, Latency, Speedup) make sense for the problem or application at hand.", "theoretical_claims": "The paper does not propose proofs or theoretical claims that require correctness checks -- the claim that 'sparse attention achieves lossless accuracy' has been sufficiently validated.", "experimental_designs_or_analyses": "The experimental designs and analyses seem sound and valid.", "relation_to_broader_scientific_literature": "As the authors have already mentioned, sparsity in transformers offer a great opportunity to reduce redundant computation e.g., in LLMs. However, the transformer architecture is likely to show largely varying sparsity patterns based on the task / field at hand e.g., video data has fundamentally different sparsity patterns. The findings in this work may be valuable for other fields where transformer networks are prominent e.g., MLLMs.", "essential_references_not_discussed": "I could not find any significant essential reference which was not discussed.", "other_strengths_and_weaknesses": "Strengths \\\nS1. Strong performances in terms of efficiency and quality.  \\\nS2. Comprehensive experiments to validate the efficacy of each contribution made by SVG. \\\nS3. Well written and easy to follow.\n\nWeaknesses\n- I could not find any notable weaknesses from the paper.", "other_comments_or_suggestions": "No other comments or suggestions.", "questions_for_authors": "Once each attention head has been classified and sparsified to spatial / temporal heads, the overall pipeline may resemble video generative models which operate in the spatial domain i.e., pipeline consisting of interleaved spatial attention and temporal attention (e.g., SVD). How does the performance / efficiency compare to such models, which do not use 3D full attention? Such comparison would be valuable in understanding where SVG stands in the overall literature of video generative models.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N8AsPWVg9m", "reviewer": "Reviewer_7rqP", "summary": "The paper presents Sparse Video-Gen (SVG), a training-free framework aimed at accelerating the inference of video diffusion models. Diffusion Transformers (DiTs) are powerful for video generation but suffer from high computational costs due to the quadratic complexity of 3D full attention. The authors reveal two distinct sparse attention patterns in DiTs: Spatial Head and Temporal Head. SVG uses an online profiling strategy to identify these patterns for sparse attention and a novel hardware - efficient tensor layout transformation, along with customized kernel implementations. It achieves significant speedups (up to 2.28x and 2.33x on CogVideoX and HunyuanVideo respectively) while maintaining high-quality video generation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your insightful feedback and the opportunity to discuss our work further. Your comments have been invaluable in refining our approach and clarifications.\n\n> Weakness 1: In-depth technical analysis and underlying mathematical principles or theoretical insights that govern these sparse patterns in the context of video diffusion models.\n\nAnswer:  Here we provide the theoretical analyses of SVG.\nConsider a video with N frames, each containing F tokens. Every token can be encoded by an indices pair $(i, j)$, where $0 \\leq i < N, 0 \\leq j < F$. In the attention map we will flatten these two-dimensional indices into a 1D vector. Corresponding formula is $x = i \\cdot F + j$.\n\n**Spatial Head**:\n\nFor spatial head, let $a_1 > 0$ denote the threshold for spatial closeness between tokens. The spatial attention mask is defined as:\n\n$f_{s}\\left( (i_1, j_1), (i_2, j_2) \\right) = 1 \\text{ if } |i_1 - i_2| < a_1,  \\text{ otherwise } 0$\n\nFor flattened indices $x_1 = i_1 \\cdot F + j_1$ and $x_2 = i_2 \\cdot F + j_2$,  \n$f_{s}(x_1, x_2) = 1$ is equivalent to:\n$| \\lfloor \\frac{x_1}{F} \\rfloor - \\lfloor \\frac{x_2}{F} \\rfloor | < a_1$\n\nThe resulting attention map takes the form of block-banded structures: the attention mask will be on the main diagonal and also on neighboring $\\pm(a_1 - 1)$ diagonals.\n\n**Temporal Head**\n\nFor temporal head, let $a_2 > 0$ denote the threshold for temporal closeness between tokens. The temporal attention mask is defined as:\n\n$$\nf_{t}\\left( (i_1, j_1), (i_2, j_2) \\right) = \n1 \\text{ if } |j_1 - j_2| < a_2,  \\text{ otherwise } 0\n$$\n\nFor flattened indices $x_1 = i_1 \\cdot F + j_1$ and $x_2 = i_2 \\cdot F + j_2$,  \n$f_{t}(x_1, x_2) = 1 \\Leftrightarrow\n| (x_1 \\bmod F) - (x_2 \\bmod F) | < a_2 \\Leftrightarrow\n| x_1 - F \\cdot \\lfloor \\frac{x_1}{F} \\rfloor - \\left(x_2 - F \\cdot \\lfloor \\frac{x_2}{F} \\right) | < a_2 \\Leftrightarrow\n\\exists k, 0 <= k < 2N - 1, |(x_1 - x_2) - k F| < a_2$\n\nThe resulting attention map forms $2N - 1$ slanted diagonals that align along constant column index differences. These diagonals, often referred to as “slashes,” correspond to the token positions sharing similar temporal locations across frames. The corresponding points in the attention matrix yield a “slash-wise” pattern with width $a_2$.\n\n---\n\n\nIn addition, we would like to emphasize that our work also provides other technical depth beyond the proposed online profiling and layout transformation. Our contributions comprise four main aspects. \n\n(1) We identify the spatial and temporal sparse patterns, which unveil the potential for efficient acceleration. \n\n(2) We propose algorithmic innovations, including online profiling strategy and layout transformation that exploit these patterns. \n\n(3) We co-design highly optimized kernels with our algorithm to translate theoretical computation saving to on-hardware speedup. \n\n(4) We present thorough evaluations that SVG can achieve an end-to-end speedup of up to 2.33x while maintaining high visual quality with a PSNR of up to 29.\n \n> Weakness 2: Comprehensive exploration of why these patterns emerge at a fundamental level.\n\nAnswer: In essence, these patterns arise because the pixels in a video often exhibit **high similarity** both within a single frame (spatially) and across consecutive frames (temporally). As similar patches in a video share nearly identical embeddings in the Q/K/V space, the self-attention mechanism naturally assigns high attention scores to these similar patches.\n\nSpatial Head: Within a single frame, neighboring pixels (and patches) often share similar colors, brightness values, and gradual transitions. If a patch in one region of a frame has a high attention score to itself, it tends to exhibit high attention to other patches in the same frame that share similar visual patterns. \n\nTemporal Head: Consecutive frames in a video, especially in static or slowly changing scenes, exhibit high similarity for the same patch location across time. If a particular patch has a strong correlation with itself (i.e., high attention to its own embedding), it will also place high attention on patches in other frames that share similar visual features.\n\nTraditional video compression techniques have similarly leveraged spatial and temporal redundancy to reduce data size and complexity. For example, H264 (used in MP4) uses intra-prediction and discrete cosine transform (DCT) to reduce spatial redundancy and uses inter-prediction and recording only the difference (motion vectors and residuals) between consecutive frames to reduce temporal redundancy. These well-known strategies highlight that video data inherently contains a high degree of spatial and temporal redundancy—precisely the property our sparse videogen approach leverages.\n\n> Weakness 3: No video sample.\n\nVideo samples showcasing the results of our method can be accessed at the following anonymous link:\nhttps://drive.google.com/drive/folders/1jhEpZ69bKfyZWmoy63iS3FhECNnX-AZU?usp=sharing", "claims_and_evidence": "Claims: The main claims are that SVG can accelerate video diffusion models, identify distinct sparse attention patterns, and achieve high-quality video generation at a faster speed.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "This paper do not contain theoretical claims.", "experimental_designs_or_analyses": "The analyses of the experimental results are detailed.", "relation_to_broader_scientific_literature": "The paper is well-related to the broader scientific literature. It builds on previous work in efficient diffusion models, efficient attention and sparse attention. The authors discuss how their work is orthogonal to existing techniques such as decreasing denoising steps, diffusion model compression, and linear/low - bit attention, and can be combined with them for further gains. They also contrast their approach with existing sparse attention methods in LLMs, highlighting that those methods do not leverage the inherent redundancy of video data.", "essential_references_not_discussed": "No essential references were identified as missing from the paper. The authors have covered a wide range of relevant works in the field of video generation, efficient attention, and diffusion models.", "other_strengths_and_weaknesses": "### **Strengths**\n\n1. **Sufficient Model Evaluation**: SVG is evaluated on representative open-sourced video generation models, including CogVideoX and HunyuanVideo. \n\n### **Weaknesses**\n\n1. **Limited Technical Depth**: The paper appears to be mainly an empirical exploration. it lacks in-depth technical analysis beyond the proposed online profiling strategy and layout transformation. For instance, the theoretical analysis of the sparsity patterns mainly focuses on the computational savings and does not delve deeply into the underlying mathematical principles or theoretical insights that govern these sparse patterns in the context of video diffusion models. This makes the technical contribution seem rather shallow in the ICML context.\n2. **Superficial Understanding of Sparsity**: In this paper, while they identify two types of attention heads based on sparse patterns (Spatial Head and Temporal Head), they do not provide a comprehensive exploration of why these patterns emerge at a fundamental level. There is no in-depth discussion on how the nature of video data, such as temporal continuity and spatial structure, precisely relates to these sparse patterns. Without a more thorough understanding of the root causes of sparsity, it is difficult to fully assess the generality and scalability of the proposed method. \n3. **No video sample.** It is weird that a video generation paper do not have videos in the supplementary.\n\n\nOverall, I hold a positive perspective on this paper. Addressing the aforementioned issues will significantly enhance the paper’s quality.", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents Sparse Video-Gen (SVG), a framework for accelerating video diffusion models by leveraging sparse attention patterns in Diffusion Transformers (DiTs). All reviewers acknowledge the method’s effectiveness in speeding up video generation while preserving high quality, with claims supported by detailed experimental results, including significant speedups and maintained PSNR values. The authors addressed the reviewers' initial concerns in their rebuttal adequately, providing additional technical details and video samples. Overall, the paper offers valuable contributions to efficient video generation.
Decision: Accept (poster)