DSA: Efficient Inference For Video Generation Models via Distributed Sparse Attention
Authors: Shenggui Li
Runyu Lu
qiaoling chen
Haiyan Yin
Yueming Lyu
Yonggang Wen
Ivor Tsang
Tianwei Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Distributed System
Diffusion
Inference
Sparsity
Primary area: infrastructure, software libraries, hardware, systems, etc.
TL;DR: DSA introduces a training-free sparse attention with distributed inference for diffusion-based video generation, cutting redundant computation and achieving up to 10.55× faster inference.

Abstract: Diffusion Transformer models have driven the rapid advances in video generation, achieving state-of-the-art quality and flexibility. However, their attention mechanism remains a major performance bottleneck, as its dense computation scales quadratically with the sequence length. To overcome this limitation and reduce the generation latency, we propose DSA, a novel attention mechanism that integrates sparse attention with distributed inference for diffusion-based video generation. By leveraging carefully-designed parallelism strategies and scheduling, DSA significantly reduces redundant computation while preserving global context. Extensive experiments on benchmark datasets demonstrate that, when deployed on 8 GPUs, DSA achieves up to 1.43× inference speedup than the existing distributed method and 10.79× faster than single-GPU inference.

Published as a conference paper at ICLR 2026
DSA: EFFICIENT SERVING FOR VIDEO GENERATION MODELS VIA DISTRIBUTED SPARSE ATTENTION
Shenggui Li
Runyu Lu
Nanyang Technological University shenggui 001 θe. ntu. edu.ueedu.gg
University of Michigan
runyuluêum ch edu
Qiaolin Chen
Haiyan Yin
Nanyang Technological University qiaoling. chen@ntu. edu. sg
CFAR and IHPC, Agency for Science, Technology and Research (A (AAR),,, Singapore yin ha nanaa -starstar... wew.areddusgg...
Yueming Lyu
Yonggang Wen Nanyang Technological University ygwen@ntus edu. sg
CFAR and IHPC, Agency for Science, Technology and Research (A (A*STAR), Singapore Lyu_yuemingea a-star edu. sq
Ivor Tsang
Tianwei Zhang
CFAR and IHPC, Agency for Science,
Nanyang Technological University edu.
Technology and Research (A (A*STAR), Singapore Lvor.tsangea-stte edu.. sq
t
ABSTRACT
Diffusion Transformer models have driven the rapid advances video generation, achieving state-of-the-ard quality and flexibility. However, their attention mech- anism remains a major performance bottleneck, as its dense computation scales quadratically with the sequence length. To overcome this limitation and reduce the generation latency, we propose DSA, a novel attention mechanism that inte- grates sparse attention with distributed inference for diffusion- based video gener- ation. By leveraging carefully-desigee significantly reduces redundant computation while preserving global context. Ex- tensive experiments on benchmark datasets demonstrate that, when deployed on 8 GPUs, DSA achieves up 1.43x inference speedup than the existing distributed method and 10.79x faster than single-GPU inference.
parallelism strategies and scheduling, DSA
INTRODUCTION
Recent advances in generative models have transformed the landscape of digital content creation, introducing unprecedented capabilities in generating sophisticated visual content (Rombach et al., 2021; Ho al., 2020; Song et al., 2020; Ding et al., 2021). This breakthrough has streamlined creative processes across multiple industries, from artistic design to media production. Particu- larly, advanced video generative models have been integrated into professional workflows through proprietary commercial platforms such as Google Veo, Kwai Kling and OpenAl Sora, as well as open-sourced alternatives like Stable Video Diffusion (Blattmann et al., 2023), Mochi (Team, 2024), CogVideo (Hong al., 2023), Hunyuan Video (Kong al., 2024) and Wan (Wan et al., 2025).
In the field of vision generation, diffusson transformer models (DiT') have emerged as a cornerstnne, renowned for their ability to synthesize highly realistic and visually coherent outputs (Peebles & Xie, 2022). By setting new benchmarks in video quality, these models represent a major step forward in computer-generee
content. However, this advantage comes at the cost of prohibitive inference latency due to the substantial computational overhead the attention mechanism. In practice, DiTs often rely full attention across temporal and spatial dimensions (Zheng et al., 2024; Lin al.,

---

Published as a conference paper at ICLR 2026
2024: Hong et al., 2023; Wan al., 2025), which incurs quadratic complexity with respect to the sequence length. This scaling bottleneck poses severe challenges for generating high-resolution, long-duration videos. For instance, producing 5-second, 720p video with Wan2.1-14B (Wan al., 2025) requires approximately 31 minutes. This underscores the inefficiency of current approaches and their prohibirive nature for commercial applications, necessitating further optimization.
Prior projects focus on the transformation from dense attention to sparse attention (Zhang et al., 2025a; Xi et al., 2025; Zhang et al., 2025c:b). Video data inherently exhibit sparsity in the tem- poral and spatial dimension. Therefore, sparse attention methods typically rely on the observation that only a subset of temporal or spatial tokens contribute significantly to the next-step denoising. By dynamically pruning attention maps, these methods achieve notable FLOP reductions without retraining. However, such savings alone are insufficient at scale. Another domain focuses on the system optimization. xDiT (Fang et al., 2024) successfully applies sequence parallelism (Li et al., 2023; Fang & Zhao, 2024; Liu al., 2024; Jacobs et al., 2024) for video generations. By splitting the hidden states along the sequence dimension, sequence parallelism can evenly distribute the com- putation workloads across GPUs, reducing the overall latency. However, this method often achieves sub-linear scaling due to extra communication overhead. One direction for further improvement the integration of sparse attention and distributed inference. MagiAttention (Zewei & Yunpeng, 2025) combines sparse attention and distributed attention. However, it is used for training Large Language Models (LLMs) instead of inference.
Our proposed Distributed Sparse Attention (DSA) bridges this gap by jointly exploiting redundancy in attention maps and the computational capacity of distributed hardware. DSA is built on two key components: mixed parallelism (MP) and dynamic attention scheduling (DAS). At runtime, a lightweight profiler determines the attention pattern for each head, after which the most suitable sequence parallelism strategy is applied. This adaptive choice ensures that both computation and communication overheads are substantially reduced. Furthermore, since the distribution of attention patterns can vary across layers and time steps, DAS dynamically adjusts the execution order to better overlap computation with communication, thereby maximizing the efficiency. Notably, this design achieves super-linear scaling, enabling larger models to run faster than their smaller counterparts under specific configurations.
Overall, our work makes the following key contributions: (1) We analyze the runtime characteristics of advanced DiT models during video generation and identify the computation bottleneck. (2) We propose DSA, a novel training-free attention mechanism which integrates both sparse attention and distributed inference. (3) We conduct extensive experiments to evaluate DSA, demonstrating its ability to reduce end-to-end latency by 1 x while maintaining the video quality.
PRELIMINARIES
2.1 DIFFUSION
Diffusion models are based on stochastic denoising process, where data is gradually corrupted by noise via forward diffusion process and then reconstructed using learned reverse process (Rom- bach al., 2021; Song et al., 2020; Ho et al., 2020). The forward process is defined as:
((110,1)) = (mas 10 (1 7))
where represents the noisy data at timestep i, and a controls the variance schedule. The reverse process is parameterized by a neural network ég, which predicts the noise added at each timestep. The reverse transitions are modeled as:
po(x)-1/2;) M(x--11 t). Ee (2re,)))
where μe and Mg are learned mean and variance functions. The model iteratively refines a noisy sample until it converges the original data distribution.
Diffusion models excel in their ability to handle complex data distributions and produce high- resolution outputs, making them preferred choice for generative tasks. However, their iterative denoising process requires multiple forward passes through the network, resulting in high computa- tional and memory demands.
2

---

Published as a conference paper at ICLR 2026
2.2 DIFFUSION TRANSFORMER
Transformers, originally designed for sequence-to-sequence tasks in natural language process- ing (Vaswani al.. 2017), become a comnestonn of modern AI architectures. Their self-attention mechanism enables effective modeling of long-range dependencies, making them well-suited for diverse tasks, including generative modeling (Liu et al., 2021; Dosovitskiy et al., 2020; Peebles & Xie, 2022). In recent diffusion models, transformers are often employed as the backbone for the de- noising network, where they learn to predict the noise or original data distribution at each timestep.
The transformer architecture relies on self-attention (MHSA) layers and feedforward neural net- works. The self-attention mechanism computes a weighted representation of input tokens by attend- ing to their pairwise relationships:
QKT vdz
Attention(Q, V) soft
)
where Q, K, and represent the query, key, and value matrices, and dj the dimensionality of the keys. The attention module can capture global context efficiently, which is critical for vision tasks.
2.3 SPARSE ATTENTION
Sparse attention exploits the fact that only subset of tokens-either within frames across contriont significantly to the output, allowing many attention computations to be skipped. Broadly, sparse attention be categorized into static and dynamic patterns. Static sparse attention relies on predefined attention masks, typically designed based on observed runtime characteristics of the model. Because the computation pattern is fixed in advance, it enables the use of high- performance kernels. In contrast, dynamic sparse attention determines the sparse patterns on the fly during inference, usually by approximating query-key interactions. While static patterns offer efficiency through predictable computation, dynamic patterns provide greater adaptivity.
Examples of static sparse attention include MInference (Jiang et al., 2024). STA (Zhang et al., 2025c), and SVG (Xi et al., 2025). Among them, SVG achieves the best performance, as it pre- serves the original video generation quality without degradation. In contrast, dynamic sparse atten- tion is exemplified by SpargeAttention (Zhang et al., 2025a), which pools query and key tokens and computes cosine similarities to identify critical attention blocks in an online manner, skipping the unimportant ones. SpargeAttention is versatile and can be applied to large language models, image generation models, and video generation models. However, its performance lags behind state-of- the-art static methods such as SVG, particularly in maintaining video quality.
2.4 SEQUENCE PARALLELISM
Traditional parallelism strategies, including data, tensor and pipeline parallelism (Li et al., 2020; Zheng et al., 2022; Rasley et al., 2020; Narayanan et al., 2021), do not scale well when the sequence length becomes extremely large. Sequence parallelism (SP) partitions the input along the sequence dimension across devices to distribute both memmory and compute burden for attention over long sequences. There are mainly three categories of sequence parallelism:
SP-Ring (Li al., 2023; Liu al., 2024): The sequence data is partitioned into chunks and distributed across devices in a ring layout. During the attention operation, the key and value embeddings circulated among devices in a ring fashion via pecr-to-peer (P2P) communication, which is often overlapped with computation to improve efficiency.
SP-Ulysses (Jacobs et al., 2023): The input is also partitioned along the sequence dimension. Through an all-to-all communication step, these chunks are redistributed so that each GPU holds the full sequence for subset of attention heads. Local attention is computed independently for each head, after which the outputs are redistributed to restore the original sequence partitioning.
SP-Unified (Fang& & Zhao, 2024; Gu et al., 20i4):This is a hybrid sequence parallelism, combin- ing the strengths of Ulysses and Ring while mitigating their respective limitations. Devices are organized into two-dimensional grid (mesh): Ulysses is applied along dimension (rows), while Ring is applied along the other (columns). Redistribution via all-to-all and send-receive communication ensures proper transfer of data between sequence partitions and head slices.
3

---

Published as a conference paper at ICLR 2026
3 CHALLENGES AND MOTIVATION
Deploying transformer model for video generation has the following challenges.
Attention is the computational bottleneck. A high-resolution video is typically flattened into a long sequence of vision tokens. Taking Wan-2.1-14B as an example, a 5-second 720p video corresponds to approximately 302k tokens per imput channel, with total of 16 channels. As the attention module scales quadratically with the sequence length, it accounts for substantial fraction of inference time, evidenced by the breakdown of inference execution time for Wan2.1-14B model with flash attention (Dao, 2024) in Figure 1. This overhead becomes even more pronounced when scaling to longer durations or higher resolutions.
1.00
Exocution Time Purcentage
FFN
80
Others
Ganeration Time
20
Ideal Linear Scaling
w3n-1.38
wan-148
humyuan-136
2
2
Number of GPUs
Figure/ 1: Execution time breakdown of 720p 5-second video generation of different models H100 GPUs
Figure 2: Weak scaling of a 720p 5-second video using Wan2.1-14B showing sub-linear decrease in generation time
H100 GPUs,
2. Sparse attention is not scalable. Since high-resolution videos lead to long sequences of vision tokens for inference, it is natural to adopt parallel inference strategies such as sequence parallelism reduce per-device computational overhead. However, existing sparse attention methods not designed for multi-GPU inference and thus fail to scale efficiently. Sequence parallelism partitions the token sequence into sub-chunks, with each device responsible for subset of the query, key, and value embeddings. A key challenge arises because existing sparse attention methods (Xi et al., 2025: Zhang al.. 2025a:e) require access the full-sequence query and key to determine the sparse attention pattern. Under ring-style sequence parallelism (Li et al., 2023; Liu et al., 2024), this leads to excessive communication overhead as devices must exchange full embeddings. Ulysses- style sequence parallelism (Jacobs al., 2024) alleviates this by gathering embeddings via all-to- all communication, but still incurs significant overhead since tokens outside the sparse mask are redundantly transferred.
Meanwhile, existing sparse attention methods fail to consider additional complexities when scaling sparse attention to distributed settings. For instance, attention sinks (Xiao et al., 2024b) can be observed in video generation models (Xi et al., 2025). When distributing the sequence over GPUs, only 1 GPU holds attention sink tokens, while these tokens need be attended by all other GPUs.
3. Sequence parallelism is sub-linearly scalable. Sequence parallelism is an effective approach for handling sequence training and inference. However, it introduces additional communication overhead since query, key, and value embeddings must be exchanged across devices. Consequently, the scaling efficiency becomes sub-linear, meaning adding reduction in latency. As shown in Figure 2, when generating 5-second 720p video using 1a 14B on H100 GPUs, the inference time only reduces from 1837.9s to 287.9, when scaling from 1 to 8 GPUs, equivalent to scaling efficiency of 79.7%. Thus, sequence parallelism only trades the overall throughput for a single request latency. This limitation raises concerns about the cost- effectiveness of sequence parallelism in commercial model-as-a-servias (MaaS) deployments.
GPUs does yield a proportional
4 DISTRIBUTED SPARSE ATTENTION
To address the above challenges, introduce Distributed Sparse Attention (DSA), methodology that integrates sparse attention with distributed inference for efficient video generation. In contrast to computing full-attention with sequence parallelism, DSA selectively matches the sparse pattern and
4

---

Published as a conference paper at ICLR 2026
Takema
Tokama
Attention
Attention
Sink
Smk
(a) Spatial attention pattern
(b) Temporal attention pattern
Figure 3: Attention patterns which are unique to video generation models (Xi et al., 2025). In the spatial attention pattern, query tokens primarily attend to key tokens within the same frame in adjacent frames, reflecting spatial locality. In contrast, the temporal attention pattern involves query tokens attending key tokens at the same spatial location but across different frames. Both patterns exhibit the attention sink pattern (Xiao et al., 2024b), which all query tokens attend to the first few key tokens, which often the text tokens in video generation.
parallel strategy, leading to significant lower computational overhead. During the communication of query, key and value embeddings, DSA also filter out the unimportant tokens but only trans- fer critical tokens to the target device, reducing the overall communication overhead. As result, DSA achieves sparse computation and super-linear scalability while preserving the video generation quality, leading to significant reduction in the generation latency and deployment cost.
4.1 SPARSITY PATTERN MATCHINN
Existing methods, such SVG (Xi al., 2025), adopt static sparsity patterns for video generation models to achieve training-free free inference acceleration. These static patterns are effective because video generation models exhibit distinct attention patterns, specifically spatial sparsity and temporal sparsity. Similar patterns have been observed Large Language Models (Xiao et al., 2025). SVG pre-defines spatial and temporal sparse masks and matches attention heads to of these masks through sampling, achieving video generation without quality loss.
While adopt the same static pattern strategy, challenge arises in distributed inference: video data is split into sub-sequences, rendering the previous matching strategy-which relies on full sequences-ined innereceeiieffeetiiee....... To address this limitation, we employ local pattern matching with majority voting. We create pre-defined attention masks for local query and key sub-sequences, ensuring mask locations align with the corresponding query and key positions. Subsequently, we perform all all-gather operations to aggregate local pattern matching decisions and vote on the final sparsity pattern.
4.2 MIXED PARALLELISM
Existing models, including Hunyuan-Vide Video (Kong et al., 2024), Wan (Wan et al., 2025), and Step- Video (Ma et al., 2025), have adopted unified sequence parallelism (USP) (Fang & Zhao, 2024) as their default parallelization strategy. USP combines ring-style (Li et al., 2023; Liu al., 2024) and Ulysses-style (Jacobs et al., 2023) sequence parallelism approaches. Specifically, it first performs all-to-all operations gather sub-sequences, then executes ring-style attention to exchange key- value embeddings for self-attention computation. This design allows USP to degenerate to ring-style sequence parallelism when the Ulysses degree is 1, and vice versa.
However, this hybrid design is primarily optimized for cross-node communication. In contrast, model deployment is typically confined to a single node, since video generation models generally range from several billion around 20 billion parameters. Furthermore, USP fails to account for the attention patterns inherent in video generation models. As demonstrated in prior work, attention maps in video generation models exhibit sparsity, particularly in the form of temporal and spatial sparse attention patterns illustrated in Figure Current approaches lack specialized designs that leverage these distinct attention patterns to reduce the computational and communication overhead.
5

---

Published as a conference paper at ICLR 2026
To address this limitation, propose Mixed Parallelism (MP). As shown in Figure 3, the spatial and temporal patterns show distintive features: the spatial sparsity occurs as the tokens are attending to its spatially close tokens in the same frame in the nearby frames while the temporal sparsity shows that the tokens are attending other tokens at the same spatial location but across different frames. Thus, it can be wiser to apply distinct parallel strategy to each sparsity pattern.
Kay & Volue dirculated
Substequence
Bend key value to adjacent
Bub Sub-seqqeence
Device
Device
Sub-eequence
Sub-eequence
Sub-eequence
Sub-eequance
Device
Sub-sequance
sebunneee
Devics 4
Device
(a) Ring
(b) Partial-ring
Figure 4: Comparison between the original ring-style attention (a) and partial-ring attention (b). The typical ring attention will transfer the key and value embeddings from one device to others, resulting in N times of data transfer. By leveraging the spatial attention pattern, partial-ring only transfers the embeddings to the adjacent neighbors, keeping the number of data transfer to 2.
Spatial Sequence Parallel. This parallel strategy is applied to spatial sparse patterns. Given N devices, each video sequence is partitioned into N chunks of sub-sequences. Since query tokens primarily attend to spatially proximate key tokens in spatial sparsity, we can simplify sparse attention to local and adjacent computation only. However, this approach introduces two key complexities:
The first frame contains attention sink tokens that require global attention. Specifically, tokens in the first frame located on the first device must be attended by all other query tokens across devices.
Attention sink tokens:
Variable spatial proximity ranges: The range of spatially proximate tokens varies across different attention heads. In some cases, spatial tokens located on adjacent devices also require attention.
To address these challenges, we broadcast attention sink tokens from the first device to all others and perform partial-ring communication for adjacent spatial tokens, as illustrated in Figure 4b. We compute attention outputs in chunks using online softmax (Dao, 2024) and overlap communication with computation. Since we only attend to spatially adjacent tokens, our approach performs send- receive operations only twice (one clockwise and one counterclockwise). compared to the N iterations required by typical ring attention. This design significantly reduces communication costs as the number of GPUs increases. Moreover, by incorporating adjacent spatial tokens rather than relying solely on local computation, better preserve video generation quality.
Temporal Sequence Parallel. For temporal sparsity, the challenge is more complex due to re- peated diagonal attention patterns that require query tokens on one device to attend to key tokens distributed across all devices. This necessitates the use of sequence parallelism. To achieve higher computational efficiency on modern accelerators such as GPUs, we perform layout transformations on temporal sparsity patterns to enable blockwise computation.
While ring-style sequence parallelism processes only subsequences per transmission and cannot perform global layout transformations, Ulyssesstyle sequence parallelism (Jacobs et al., 2024) is ideally suited for this scenario. Each device initially stores a subsequence of the input with shape S/N, H, D). where B is the batch size, S is the full sequence length, H is the number of atten- tion heads, and is the head dimension. An all-to-all exchange is first performed so that each device reconstructs full sequences with shape [B, S, HI/N DJ. With the complete sequence available lo- cally, we can then apply a sparse attention pattern independently to the subset of heads assigned to each device. After the attention computation, second all- all-to-all operation restores the tensor layout to [B, S/N, H, D). While the total communication volume remains the same as in conventional Ulysses sequence parallelism, the use of sparse attention greatly reduces the computational cost.

---

Published as a conference paper at ICLR 2026
4.3
DYNAMIC ATTENTION SCHEDULING
Terspraal
Spatial Sparse Attention
Spotial Sporso Attention
Spatial Sparse Attendion
Sporoe
Nalve Scheduling
Brozdcaatt Bine
Sand -ae---e
Sand-Racalive
NISAI
NIDAI
Time Saved
Sparial Sparse Attention
Spotial Sperso Attention
Sperso Attention
Spetial Sparse Atteriin
Optimized Scheduling
Broadcast Aftention
Ping
Ring
Spatial- Prominent Schedule.
Sponss
Sparse
Spame Atlestion
Temponal Spare Amsnntoon
Scheduling
Rirg Racahve
Wing Recaive
Groedcast Bink
Time
Spotial Atlandion
Sportal Attention
Spatal Attentionn
Temponi Spance
Optimized Scheduling
Ring Sand-
Ring
AUDAII
Sand-
AIDAR
Receive
(b) Temporal-Promee Promminet Schedule.
Figure 5: Dynamic attention scheduling. The green boxes represent spatial attention and blue boxes represent temporal attention.
Diffusion models exhibit dynamic behavior in their attention computation across different layers and denoising steps when processing various prompts. Consequently, the ratio between spatial sparse heads and temporal sparse heads fluctuates dynamically throughout the inference process. To en- hance performance, we propose dynamic attention scheduling, which efficiently coordinates com- putation and communication operations. Figure shows the mechanism of our proposed solution.
Spatial-dominant Schedule. When spatially sparse heads comprise the majority of attention heads, we interleave spatial attention computation with temporal attention computation. The key optimiza- tion is to hide the communication overhead of all-to-all operations through this interleaving strategy.
Temporal-dominan Schedule. When temporally sparse heads are dominant, we adopt a different approach. First, we compute the local attention for spatial heads while overlapping this computation with all-to-all communication. During the subsequent Ulysses attention computation, we perform partial-ring communication gather spatial tokens, which are then concatenated into larger tensor. Finally, we exccute spatial attention computation while simultancously overlapping it with temporal all-to-all communication.
EVALUATION
5.1 EXPERIMENT SETUP
We evaluated DSA on three state-of-the-art video generation models: 1-n2.1-1.3B, 1an2.1-14B, and Hunyuan- Video. We employed VBench (Huang et al., 2024) as our primary benchmark for assess- ing video quality. Since the original prompts in VBench are concise and limited in complexity, refined them using GPi-4-mini. From VBench's 's comprehensive evaluation framework, we selected four critical dimensions: overall consistency, subject consistency, spatial relationship, and tempo- ral style, which together provide a holistic assessment video generation quality. Additionally, conducted Trame-to-frame comparisons using traditional image quality metrics, including Peak

---

Published as a conference paper at ICLR 2026
Generated Vidco Quality
Model
Method
Overall Consistency
Subject Consistency
Spatial Relationship
Temporal
PSNR T
SSIM T
LPIPS
Style T
Dense
0.168
0.922
0.819
0.156
Sparge
31.39
0.704
0.175
0.166
0.909
0.713
0.152
Wan2.1-1.3B
SVG
34.74
0.832
0.073
0.168
0.921
0.825
0.154
DSA (Ours)
34.67
0.833
0.073
0.166
0.922
0.824
0.152
0,170
0.927
0.798
0.163
Sparge
30.79
0.641
0.189
0,161
0.892
0.701
0.155
W-n2.1-14B
SVG
33.03
0.781
0.109
0,170
0.925
0.804
0.166
(Ours)
0.775
0.103
0.171
0,922
0.804
0.165
Dense
0.165
0.940
0.614
0.158
Sparge
32.19
0.762
0.141
0,160
0.930
0.584
0.143
Hunyuan-vidoo
SVG
33.32
0.810
0.120
0.168
0.938
0.637
0.152
DSA (Ours)
33.40
0.804
0.121
0.167
0.940
0.633
0.149
Table 1: Video quality evaluation on VBench
Signal-to-Noiss Ratio (PSNR). Structural Similarity Index Measure (SSIM), and Learned Percep- tual Image Patch Similarity (LPIPS) (Zhang et al., 2018). These evaluation metrics comprehensively cover both image quality and spatial-temporal coherence the video level.
We compared DSA with both sparse attention and distributed inference approaches. For video quality evaluation, we selected SparseAttention (Zhang al., 2025a), and Sparse-Video-Ge (SVG) (Xi et al., 2025). Ring/Ulysses Sequence parallelism is not used for quality evaluation as achieves the same performance as the full attention baseline. For system performance, we compared the generation latency for both sparse attention and distributed methods including SVG (Xi et al., 2025) and SP-Unified (Fang& & Zhao, 2024; Gu al., 2024). As USP enables different combinations of ring and Ulysses attention, we only kept the best results in Table 2.
Dense
Dense
DSA
DSA
(a) Prompt: A vihrant orange-and-white clownfish darts through sunlit coral reef, weaving gracefully among swaying anemones and colorful corals.
(b) Prompt: fluffy panda joyfully swings back and forth on brightly colored playground swing set.
Figure 6: Visualization the generated outputs from Wan2.1-14B
5.2 VIDEO QUALITY EVALUATION
We evaluated the quality of the videos generated by different methods, and the results are summa- rized in Table 1. We set the sparsity level to 75% for both SVG and DSA, while using similarity threshold of 0.6 and a CDF threshold of 0.98 for Sparse Attention. We not report results for the USP method, as it is lossless and produces identical results to dense attention.
According to the quantitative evaluation metrics, DSA consistently achieves performance comparable to SVG, while outperforming MInference and SparseAttentiin across both the Wan and Hunyuan models. This indicates that DSA effectively preserves the fidelity and coherence of generated video sequences despite its use of sparse attention mechanism.
Figure presents two randomly selected prompts along with the videos generated by each method. For both methods, show frames sampled from the beginning, middle, and end of each video. Visual inspection indicates that the frames produced by DSA closely resemble those generated using dense attention, preserving high visual fidelity and temporal coherence. Additional frames for more diverse prompts are provided in the Appendix A.1, and full video examples are included in the supplementary materials.
8

---

Published as a conference paper at ICLR 2026
Model
Method
System Performauce
of GPUs
Generation time
Speedup
Dense
400.34
Wan2.1-1.3B
SVG
USP
310,
1.29
DSA (Ours)
54,11
7.43
Dense
1889
Wan2,1-14B
SVG USP
1221.34
1.55
251.26
7.52
(Ouns)
175
Dense
17901.3
Hunyuan video-
SVG
1340.40
USP
284.7
6.29
DEA (Ouss)
189.38
9.45
Table 2: Latency and speed of different models when generating a 720p 5-second video.
Model
Strategy
Generation time
Naive Schedule
188.92 Spatial Only
Wan2.1-141
Dynamic Schedule
180.47
175
Table 3: Generation latency when adopting different strategies for attention.
5.3 SYSTEM PERFORMANCE E EVALUATION
We also investigated the system performance of DSA. Unlike large language models, which typi- cally emphasize metrics such as time to first token (TTFT) and time per output token (TPOT), video generation models place a higher priority on overall generation latency, as the total runtime spans the scale of minutes rather than seconds. As shown in Table 2, DSA significantly outperforms the single-GPU method and can achieve up to 10.79x speedup. This translates to super-linear scaling on Wan- 14B and Hunyuan-13B as the speedup is greater than the proportional increase in the num- ber of GPUs, demonstrating promising cost-effectiveness in large-scale deployment. Compared to the distributed unified sequence parallelism, DSA can still achieve 43% improvement on Wan-14B. However, it is noted that DSA still scales sub-linearly for Wan-1.3B, suggesting that the computation sharding hurts the hardware utilization and reduces the computation efficiency.
5.4 ABLATION STUDIES
Scheduling Strategies. We evaluated the impact of different scheduling strategies on DSA. Un- der naïve scheduling. spatial and temporal attention are executed sequentially without overlap. In contrast, Dynamic Attention Scheduling reorders execution based on the spatial-temporal ratio and incorporates computation-commuiice
overlap. As shown Table 3, this dynamic strategy re- duces latency by 4.7%. We further examined spatial-only strategy, where all attention heads adopt the spatial pattern. This configuration decreases generation latency to 175 seconds an 8% im- provement over naïve scheduling whille while incurring negligible impact video quality (results put in the appendix).
Sparsity Selection In DSA, since computation for spatial and temporal patterns is decoupled, can assign different sparsity levels to each, unlike SVG, which enforces a uniform sparsity level across both. To evaluate this flexibility, dimension and used Wan2.1-14BB to generate videos under varying sparsity settings. Specifically, we experimented with sparsity levels of 80%, 90%, and 95% for both spatial and temporal dimensions, and assessed their impact on video quality. As shown in Table 4, setting spatial sparsity too high degrades performance: when spatial sparsity is increased from 80% 95% with temporal sparsity fixed at 95%, the overall consistency score drops from 0.179 to 0.174. However, very high temporal sparsity tends to yield comparable performance. For example, a temporal sparsity of 95% produces results similar to those at lower spatial sparsity levels of 90% or 80%. This reveals that temporal attention patterns are generally more sparse than the spatial patterns. This is because the number of frames is generally smaller than the size of tokens in single frame. Consequently, for given query token, the number of key tokens at the same spatial location but across different temporal locations much smaller than the number of key tokens located within the same or adjacent frames.
sampled 20 prompts from each VBench evaluation

---

Published as a conference paper at ICLR 2026
Overall Consistency 1
Subject Consistency
Spatial Relationship
Temporal
Model
Spatial Sparsity
Temporal Sparsity
Style
95%
0.174
0.916
0,941
0.135
95%
0.176
0.918
0.957
0.135
80%
0.177
0.918
0.957
0.134
95%
0.178
0.915
0,952
0.138
Wan2,1-148
90%
90%
0.178
0.919
0,948
0.135
80%
0.178
0.920
0,950
0,134
95%
0.179
0.917
0,944
0,138
80%
90%
0,179
0.919
0,948
0.136
80%
0.177
0.921
0.956
0.135
Table Sensitivity sparsity levels for spatial and temporal attention respectively.
6
RELATED WORK
Diffusion models have been accelerated through several largely orthogonal approaches. One line of work focuses on sparse attention. Although methods such as BigBird (Zaheer et al., 2020), StreamingL.LMM (Xiao et al., 2024b), DuoAttention (Xiao al., 2024a), and Native Sparse Atten- tion (Yuan et al., 2025) demonstrate strong performance in large language models, they rely on language-specific attention patterns and do not transfer effectively to diffusion models. More re- cently, SpargeAttentes
(Zhang et al., 2025a) dynamically detects sparsity and implements an effi- cient kernel for acceleration. SVG (Xi al., 2025) and STA (Zhang al., 2025c) extend sparse attention to diffusion models by applying static sparsity patterns, among which SVG achieves the best generation quality.
Another line of work focuses system-level optimization, including DistriFusion (Li et al., 2024) and PipeFusion (Fang al., 2025). DistriFusion leverages stale latents and patch parallelism to partition images across devices for parallel computation, while PipeFusion extends this approach with pipeline parallelism to further reduce latency and improve hardware utilization. However, these methods primarily target image generation. USP (Fang & Zhao, 2024) instead proposes a lossless distributed inference framework that combines Ring Attention (Li et al., 2023; Liu et al., 2024) and Ulysses (Jacobs et al., 2023) to improve scalability.
third line of work explores caching mechanisms that reuse intermediate activations based on the similarity of latent representations across denoising steps. PAB (Zhao et al., 2025) and DiTFas- tAttn (Yuan et al., 2024) use static reuse, while AdaCache (Kahatapitiya et al., 2024) adapts caching based feature variance and TaylorSeer (Liu al., 2025) predicts feature evolution via Taylor expansion. These methods are complementary to sparse attention and distributed inference.
In contrast to prior work, method jointly considers both the sparsity characteristics of atten- tion patterns and distributed inference strategies. By aligming sparsity-aware computation with dis- tributed execution, our approach improves both computational efficiency and communication effi- ciency. Furthermore, our method is orthogonal to caching-base techniques and can be seamlessly combined with them for additional acceleration.
7 CONCLUSION
In conclusion, we introduce DSA, a novel attention mechanism that integrates sparse attention with distributed inference for diffusion based video generation. By selecting suitable parallel strategy for distinct sparse patterns, DSA substantially reduces computation and communication overhead. Ex- periments demonstrate that DSA achieves significant efficiency gains: up to 10.79x faster inference compared to the single-GPU dense attention inference while preserving the vidco quality.
This work explores parallelization strategies for spatial and temporal attention patterns, without yet addressing sparse patterns that may emerge in future models. Although dynamic attention scheduling overlaps computation and communication, its multiple kernel launches can degrade performance. As future work, plan to incorporate adaptive sparse patterns and fuse com- comemmmmmmiiation into efficient CUDA kernels using libraries such as TileLink (Zheng et al., 2025b) and Distribuitt (Zheng al., 2025a).
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We thank the anonymous reviewers for their valuable feedback and constructive suggestions. We are grateful to our collaborators and colleagues for insightful discussions and support throughout this project. Shenggui Li is generously supported by the A S ACIS scholarship.

---

Published as a conference paper at ICLR 2026
REFERENCES
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rom- bach. Stable video diffusion: Scaling latent video diffusion models to large datasets, 2023. URL https: //apivv org/ abs 221111111110 15127.
Tri Dao. FlashAttention-2:: Faster attention with better parallelism and work partitioning. In Inter- national Conference on Learning Representations (ICLR), 2024.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang. Cogview: mastering text-to-image generation via trans- formers. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS '21, Red Hook, NY, USA, 2021. Curran Associates Inc. ISBN 9781713845393.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ArXiv, abs/2010.11929, Corpus ID:: 225039882.
2020. URL https: /aaii////p..a/ semanticschee
.org/
Jiarui Fang and Shangchun Zhao. A unified sequence parallelism approach for long context gener- ative ai. arXiv preprint arXiv:2405.07719 2024.
Jiarui Fang, Jinzhe Pan, Xibo Sun, Aoyu Li, and Jiannan Wang. xdit: an inference engine for diffusion transformers (dits) with massive parallelism. arXiv preprint arXiv:2411.01738 2024.
Jiarui Fang, Jinzhe Pan, Aoyu Li, Xibo Sun, and WANG Jiannan. Pipefusion: Patch-level pipeline parallelism for diffusion transformers inference. In The Thirty-ninth Annual Conference on Neu- ral Information Processing Systems, 2025. URL https: opwnrevieeeeeeeeeeeeeee net / forum?id= 5xwyxupsLL.
Diandian Gu, Peng Sun, Qinghao Hu, Ting Huang. Xun Chen, Yingtong Xiong, Guoteng Wang, Qiaoling Chen, Shangchun Zhao, Jiarui Fang, Yonggang Wen, Tianwei Zhang, Xin Jin, and Xu- anzhe Liu. Loongtrain: Efficient training of long- sequene Ilms with head-context parallelism, 2024. URL https:/ arxiv. org/ abs/2405..2 18488
Jonathan Ho, Ajay Jain, and P. Abbeel. Denoising diffusion probabilistic models. ArXiv, abs/2006. 11239, 2020. URL https://api.. wwwanticscholeeeee.e prrg/CorpusDD: 219955663.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pre- training for text-to-video generation via transformers. In The Eleventh International Confer- ence on Learning Representations, 2023. URL httss///pppeeeieeee wwtpeeiiwwwwweeeeeeeeeeme http://wwwwee/fouuu.ne/e forum?id= rB6TpjAuSRy.
Ziqi Huang, Yinan Hc, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianx- ing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang. Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogni- tion, 2024.
Sam Adé Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Leon Song, Samyam Ra- jbhandari, and Yuxiong He. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models. ArXiv, abs/2309.14509, 2023. URL https:
/ //api. www.maanissccooaa.orgooooo
Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Reza Yazdani Aminad- abi, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. System optimizations for enabling training of extreme long sequence transformer models. In Proceedings of the
43rd ACM Symposium on Principles of Distributed Computing, PODC '24, pp. 121-130. New York, NY, USA, 2024, Association for Computing Machinery. ISBN 9798400706684, doi: 10.1145/3662155 188 3662286. URL http:: //toss//ddoii /oii 1144/66. 366628
12

---

Published as a conference paper at ICLR 2026
Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin- Yew Lin, Yuqing Yang, and Lili Qiu. MInference 1.0: Accelerating pre-filling for long-context LLMs via dynamic sparse attention. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024. URL https:/ openreview.net forum? 1==fBBCAA
Kumara Kahatapitiya, Haozhe Liu, Sen He, Ding Liu, Menglin Jia, Michael S. Ryoo, and Tian Xie. Adaptive caching for faster video generation with diffusion transformers. ArXiv, abs/2411 abs/2411.023977, 2024. URL https:// / api. semant icscholar. org
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03605, 2024.
Muyang Li, Tianle Cai, Jiaxin Cao, Qinsheng Zhang. Han Cai, Junjie Bai, Yangqing Jia, Ming- Yu Liu, Kai Li, and Song Han. Distrifusion: Distributed parallel inference for high-resolution diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, and Soumith Chintala. Pytorch distributed: Experi- ences on accelerating data parallel training, 2020. URL https: //appp:://axxv.../// org/ abs//00///22 15704.
Shenggui Li, Fuzhao Xue, Chaitanya Baranwal, Yongbin Li, and Yang You. Sequence paral- lelism: Long sequence training from system perspective. Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 2391-2404, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: ht https:://aali htttp:://aclanthhlooyy... wrtp://g..rgg////// wwtp////..22..1.....oo..aaaa 134/.
134. URL
Bin Lin, Yunyang Ge, Xinhua Cheng, Zongjian Li, Bin Zhu, Shaodong Wang, Xianyi He, Yang Yc, Shenghai Yuan, Liuhan Chen, et al. Open-sora plan: Open-soure large video generation model. arXiv preprint 11 2024.
Hao Liu, Matei Zaharia, and Pieter Abbeel. Ringattention with blockwise transformers for near- infinite context. In The Twelfth International Conference Learning Representations, 2024. URL /tttps://openrvviiie
net/ wtwpre/ewweeeeeeeoooorreee....dd hiwrummidddissHpHHHsss
Jiacheng Liu, Chang Zou, Yuanhuiyi Lyu, Junjie Chen, and Linfeng Zhang. From reusing to fore- casting: Accelerating diffusion models with taylorseers. arXiv preprint arXiv:2503.0692. 2025.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang. Stephen Lin, and Bain- ing Guo. Swin transformer: Hierarchical vision transformer using shifted windows. 202/ IEEE/CVF International Conference Computer Vision (ICCV), pp. 9992 10002, 2021. URL httt:p////allseeenneiiiicocco
https: /ttp///ppi.. www.nteeetttteee www.mmantntccccatam wwwoooo///oooooooooooooo :
232352874.
Guoqing Ma, Haoyang Huang, Kun Yan, Liangyu Chen, Nan Duan, Shengming Yin, Changyi Wan, Ranchen Ming, Xiaoniu Song, Xing Chen, Yu Zhou, Deshan Sun, Deyu Zhou, Jian Zhou, Kai- jun Tan, Kang An, Mei Chen, Wei Ji, Qiling Wu, Wen Sun, Xin Han, Yanan Wei, Zheng Ge, Aojie Li, Bin Wang, Bizhu Huang, Bo Wang, Brian Li, Changxing Miao, Chen Xu, Chenfei Wu, Chenguang Yu, Dapeng Shi, Dingyuan Hu, Enle Liu, Gang Yu, Ge Yang, Guanzhe Huang, Gulin Yan, Haiyang Feng, Hao Nie. Haonan Jia, Hanpeng Hu, Hanqi Chen, Haolong Yan, Heng Wang, Hongcheng Guo, Huilin Xiong, Huixin Xiong, Jíahao Gong, Jianchang Wu, Jiaoren Wu, Jie Wu, Jie Yang, Jiashuai Liu, Jiashuo Li, Jingyang Zhang, Junjing Guo, Junzhe Lin, Kaixiang Li, Lei Liu, Lei Xia, Liang Zhao, Liguo Tan, Liwen Huang, Liying Shi, Ming Li, Mingliang Li, Muhua Cheng, Na Wang, Qiaohui Chen, Qinglin He. Qiuyan Liang, Quan Sun, Ran Sun, Rui Wang, Shaoliang Pang, Shiliang Yang. Sitong Liu, Siqi Liu, Shuli Gao. Tiancheng Cao, Tianyu Wang. Weipeng Ming, Wenqing He, Xu Zhao, Xuelin Zhang, Xianfang Zeng, Xiaojia Liu, Xuan Yang, Yaqi Dai, Yanbo Yu, Yang Li. Yineng Deng, Yingming Wang. Yilei Wang, Yuanwei Lu, Yu Chen, Yu Luo, Yuchu Luo, Yuhe Yin, Yuheng Feng, Yuxiang Yang. Zecheng
13

---

Published as a conference paper at ICLR 2026
Tang, Zekai Zhang, Zidong Yang, Binxing Jiao, Jiansheng Chen, Jing Li, Shuchang Zhou, Xi- angyu Zhang, Xinhao Zhang, Yibo Zhu, Heung- Yeung Shum, and Daxin Jiang. Step-video-12v technical report: The practice, challenges, and future of video foundation model, 2025. URL https: / /awxiv.xvvv//. org/
. 0248.
Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vi- jay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. Efficient large-scale language model training on gpu clus- ters using megatron-Im. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC '21, New York, NY, USA, 2021. Associa- tion for Computing Machinery. ISBN 9781450384421. doi: 10.1145//458811734776000 URL
https: htttp::/ddi.orgg///// http///...11/335588811 3476234772000 3206
William S. Peebles and Saining Xie. Scalable diffusion models with transformers. 2023 JEEE/CVF
International Conference Computer Vision (ICCV), pp. 4172-4182, 2022. URL https:
/ //app... semant wwwcchhllr.rggeeeoeoco Corpus DD DD 254854389.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed:
System opti- mizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '20, pp. 3505-3506, New York, NY, USA, 2020. Association for Computing Machin- ery. ISBN 9781450379984. doi: 10. 00.1145/3394886.40667 URL https::// /doi http:/oooooooo.///////// 1145/3394483 3406703.
Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. 2022 IEEE/CVE Conference on Com- puler Vision and Pattern Recognition (CVPR), pp. 10674-10685, 2021. URL pattps://api.. remanticschon
org/ /ttg///ooooooooo Corpus 245335280.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ArXiv, abs/2010.02502,, 2020. URL /ctps://aptt semanticscholar: httpss//pg....mmntcccooooo.... Cttpg//orpusscc ID: 222140788.
Genmo Team. Mochi 1. https:::////tctt https://github.... com ctt.:///acccooggoooaaoaoaaaaaaaa httimmaaammodeess...mmm 2024.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Neural Information Processing Systems, 2017. URL https: / api. www.ntseeaettaaaeeeeeeee
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang. Jianyuan Zeng. Jiayu Wang, Jingfeng Zhang. Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fang, Tianxing Wang. Tianyi Gui, Tingyu Weng, Tong Shen, Wei Lin, Wei Wang. Wei Wang. Wenmeng Zhou, Wente Wang. Wenting Shen, Wenyuan Yu, Xianzhong Shi, Xiaoming Huang, Xin Xu, Yan Kou, Yangyu Lv, Yifei Li, Yijing Liu, Yiming Wang, Yingya Zhang, Yitong Huang, Yong Li, You Wu, Yu Liu, Yulin Pan, Yun Zheng, Yuntao Hong, Yupeng Shi, Yutong Feng, Zeyinzi Jiang, Zhen Han, Zhi-Fan Wu, and Ziyu Liu. Wan: Open and advanced large-scale video generative models. arXiv preprint 2503. 203144 2025.
Haocheng Xi, Shuo Yang, Yilong Zhao, Chenfeng Xu, Muyang Li, Xiuyu Li, Yujun Lin, Han Cai, Jintao Zhang, Dacheng Li, Jianfei Chen, lon Stoica, Kurt Keutzer, and Song Han. Sparse video- gen: Accelerating video diffusion transformers with spatial-temporal sparsity. In Forty-second International Conference on Machine Learning, 2025. URL pestps://openreiieee forum??d-u8CA300
Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. Duoattention: Efficient long-context Ilm inference with retrieval and streaming heads. Xrviv, 2024a.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Tweifth International Conference on Learning Rep- resentations, 2024b. URL httph::t////ete /ttp:////////eeeeeeeeeeeee / www.:r/wwwweeeeeeeeeeeeaaaaa
net forum? d= 2VV..
14

---

Published as a conference paper at ICLR 2026
Guangxuan Xiao, Jiaming Tang. Jingwei Zuo, junxian guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. Duoattention: Efficient long-context LLM inference with retrieval and streaming heads. In The Thirteenth International Conference on Learning Representations, 2025. URL https:/ httes///ppeeeeee /wwpeeeeeeeeee.eeo net forum? d=cF u 7xUm.
Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Yuxing Wei, Lean Wang. Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. Native sparse attention: Hardware-aligned and natively trainable sparse attention. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 23078 23097, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 881-888889775555 251-0. doi: 10. 111111111/2225555 wat.0025..clllooooo acl-long. 126. owg.oogg////// acl wocclloooe long 1126/.
URL https: http///alaaaaahhcc/e
Zhihang Yuan, Hanling Zhang, Lu Pu, Xuefei Ning, Linfeng Zhang, Tianchen Zhao, Shengen Yan, Guohao Dai, and Yu Wang. DiTFastattn: Attention compression for diffusion transformer models. In The eighth Annual Conference on Neural Information Processing Systems, 2024. URL
https: wpwpreview.eeeeee////e/e. net/ Eorum? ii= 1eeiidd5511@@p 51H0pk0y3t.
Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: transformers for longer sequences. In Proceedings of the 34th International Conference on Neural Informa- tion Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Tao Zewei and Huang Yunpeng. Magiattention: distributed attention towards linear scalability for ultra-long context, heterogeneous mask training. https:/ / /tggt/gtttb///////// com/ SandAI SandAI-orgg//////// MagiAttention/ 2025.
Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia Wei, Haocheng Xi, Jun Zhu, and Jianfei Chen. Spargeattn: Accurate sparse attention accelerating any model inference. International Confer- ence on Machine Learning (ICML), 2025a.
Peiyuan Zhang, Yongqi Chen, Haofeng Huang, Will Lin, Zhengzhong Liu, lon Stoica, Eric Xing, and Hao Zhang. Vsa: Faster video diffusion with trainable sparse attention, 2025b. URL https: 13389.
// //arxivv.om www.xiv.org/bbbao wwt.ooorg//bbaaeeeeee
Peiyuan Zhang. Yongqi Chen, Runlong Su, Hangliang Ding, Ion Stoica, Zhengzhong Liu, and Hao Zhang. Fast video generation with sliding tile attention. In Forty-second International Conference on Machine Learning, 2025c. URL https: http:///pp///eeeee/ee/e/ net/forumiid /
U74MOXPF.Jd.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as a perceptual metric. In CVPR, 2018
Xuanlei Zhao, Xiaolong Jin, Kai Wang, and Yang You. Real-time video generation with pyramid attention broadcast. In The Thirteenth International Conference on Learning Representations, 2025. URL https: http//ppnrevveeeee https://ppenzeview.ctt net/ hett//oorumiid-hDBBOB
Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric Xing. Joseph E. Gonzalez, and Ion Stoica.
Alpa: Automating inter- and Intra-Operator parallelism for distributed deep learning. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pp. 559- 578, Carlsbad, CA, July 2022. USENIX Association. ISBN 978-1-939113-22-1 URL https: httpeenceeeoodde///eeeeeeee
//ww /www. www.uu.................. org/
/heeg/henggggge -iaaminn.
Size Zheng, Wenlei Bao, Qi Hou, Xuegui Zheng, Jin Fang. Chenhui Huang, Tianqi Li, Haojie Duanmu, Renze Chen, Ruifan Xu, Yifan Guo, Ningxin Zheng, Ziheng Jiang. Xinyi Di, Dongyang Wang, Jianxi Ye, Haibin Lin, Li-Wen Chang. Liqiang Lu, Yun Liang, Jidong Zhai, and Xin Liu. distribbb::: Programming overlapping kernels on distributed systems with the triton compiler, 2025a. URL https: / //axxiv.... org/ alos 2504 19442.
15

---

Published as a conference paper at ICLR 2026
Size Zheng, Jin Fang, Xuegui Zheng, Qi Hou, Wenlei Bao, Ningxin Zheng, Ziheng Jiang. Dongyang Wang, Jianxi Ye, Haibin Lin, Li-Wen Chang, and Xin Liu. Tilelink: Generating efficient compute-commmuiicttio overlapping kernels using tile-centric primitives, 2025b. https: //aaxxi..rrgabb//2222..... 2503 /ttbb///2533223333 20313.
Zangwei Zheng, Xiangyu Peng. Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, 2024. URL https: / /arxiv....... org/ abs /2tpp///1222244.
16

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 VISUAL COMPARISON BETWEEN DSA AND BASELINES
The figures below present videos generated using full attention, SVG (Xi et al., 2025), and our proposed DSA method, respectively. We employ Wan2.1-14B Wan et al. (2025) to generate the videos and extract one frame every 10 frames to illustrate both spatial and temporal consistency.
Dense
DSA
Figure 7: Prompt: A cheerful, fluffy panda strums guitar beside crackling campfire, with snow- capped mountains rising in the background.
Dense
SVG
DSA
Figure& 8: Prompt: A camera soars through surreal fantasy landscapes- floating islands, crystalline spires, bioluminescent forests, cascading waterfalls, and shifting, star-lit sky.
Dense
Figure 9: Prompt: Digital-art video of a whimsical hybrid creature: a raccoon with textured turtle shell and subtle reptilian markings, rendered with detailed fur and shell textures, soft cinematic lighting, and gentle, playful animation.
17

---

Published as a conference paper at ICLR 2026
Dense
5VG
DSA
Figure 10: Prompt: Extreme slow- dlow-motion close-up of a vibrant turquoise water splash with fine droplet detail on transparent background (alpha channel included).
Dense
DSA
Smooth, cinematic aerial panoramic drone shot sweeping over vivid fantasy realm of floating islands, crystalline lakes, mist-shrouded forests, and towering ancient spires bathed in warm golden-hour light.
Figure 1I: Prompt:
Dense
Figure 12: Prompt: Studio portrait of happy dog facing the camera, wearing bright yellow turtleneck, centered in frame with soft studio lighting against dark background.
18

---

Published as a conference paper at ICLR 2026
Dense
SVG
DSA
Figure 13: Prompt: Tightly cropped macro slow-motion close-up of roasted coffee beans cascading into an empty bowl, highlighting surface texture, sheen, and the graceful motion of individual beans.
Dense
SVG
DSA
Figure 14: Prompt: Two pandas in a cozy study animatedly discuss an academic paper, pointing at charts, flipping pages, and jotting notes cluttered desk.
Dense
SVG
DSA
Figure 15: Prompt: Vibrant ink droplets swirl and diffuse through water, forming dreamy, abstract cloud-like color formations.
19


Reviews:
[{"review_id": "n9whF2TP9S", "reviewer": "Reviewer_FLsG", "summary": "This paper proposed distributed sparse attention (DSA) for DiT-based video generation model and achieved $1.43 \\times$ speed up than the unified sequence parallelism (USP).", "questions": "1. In Table 2, the results of USP + SVG can be provided.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. By combining sparse attention with distributed strategies, the video generation model achieves improved multi-GPU efficiency while preserving the original generation quality as much as possible.\n\n2. Evaluations were conducted on several mainstream models, including Wan2.1-1.3B, Wan2.1-14B, and Hunyuan-Video, covering video quality metrics (PSNR, SSIM, LPIPS, VBench) as well as system performance metrics (latency and speedup).", "weaknesses": "1. The overall approach resembles a combination of the SVG method and the USP method, with optimized attention scheduling. \n\ntypo: Line 152, \"xx tokens\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jweCbT1WgU", "reviewer": "Reviewer_3sqw", "summary": "The paper presents an interesting approach to improving the efficiency of video generation models using distributed sparse attention. However, the lack of supplementary materials, incomplete evaluation, and inconsistent reporting of results hinder the ability to fully assess the effectiveness of the proposed method. Addressing these issues will significantly enhance the paper's quality and credibility.", "questions": "no", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper presents an interesting approach to improving the efficiency of video generation models using distributed sparse attention.", "weaknesses": "1. The authors have not provided the MP4 files for the proposed method and comparison methods in the supplementary materials. This makes it very difficult to assess the actual visual quality and temporal consistency of the generated videos.\n\n2. Even without the MP4 files, the authors could have provided code or other means to reproduce the results and evaluate the visual quality.\n\n3. The authors have not tested the full range of metrics from VBench or VBench2.0. Comprehensive evaluation is crucial to understand the strengths and weaknesses of the proposed method.\n\n4. Table 2 includes timing results for USP, but Table 1 lacks corresponding quality metrics for USP. This inconsistency makes it difficult to compare the proposed method with USP comprehensively.\n\n5. The related work section could benefit from a more comprehensive review of pre-trained models for video generation acceleration and sparse/linear attention methods.\n\nIf the authors address the above concerns effectively, particularly by providing supplementary materials and a more comprehensive evaluation, I would be willing to reconsider my assessment and potentially give a more positive score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3nCx3wkrJa", "reviewer": "Reviewer_HyRK", "summary": "This paper proposes Dynamic Selective Attention (DSA) to accelerate video generation.\n\nDSA adopts the dynamic sparse patterns based on the fixed spatial and temporal sparse patterns.\n\nThis work mainly focus on the contribution on the system level scheduling design.\n\nThe experimental results show that this work achieves good results.", "questions": "1. Please provide the details of the dynamic sparse pattern generation method.\n\n2. Please provide the overhead (like latency) brought by the dynamic sparse pattern generation.\n\n3. Please provide the comparison to dynamic sparse pattern works in video generation like [1].\n\n---\n[1] DraftAttention: Fast Video Diffusion via Low-Resolution Attention Guidance", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This work shows how to schedule the attention computation for sparse attention.", "weaknesses": "1. This work does not provide the detailed dynamic sparse pattern online generation methods. In Section 4.1, this work only shows that they adopted the spatial and temporal kernels from the work [1], which shows the limited novelty. The dynamic pattern looks like the token-importance based pruning, which is not novel.\n\n2. This work claims dynamic sparse pattern, while it compares to the fixed sparse pattern works like [1]. This work does not compare to dynamic sparse pattern works like [2] \n\n\n---\n[1] Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity\n\n[2] DraftAttention: Fast Video Diffusion via Low-Resolution Attention Guidance", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vj0OSIynLJ", "reviewer": "Reviewer_RRN5", "summary": "This paper proposes DSA, a system that combines sparse attention mechanisms with distributed inference to accelerate video generation using Diffusion Transformer models. The approach introduces two key components: (1) Mixed Parallelism (MP) that applies different parallelism strategies (spatial sequence parallel vs. temporal sequence parallel) based on attention patterns, and (2) Dynamic Attention Scheduling (DAS) that optimizes computation-communication overlap. Experiments on Wan and Hunyuan-Video models show up to 10.79× speedup over single-GPU inference while maintaining video quality.", "questions": "See weakness", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper makes a solid contribution by integrating sparse attention with distributed inference for video generation. The key insight of matching different parallelism strategies to spatial versus temporal attention patterns is well-motivated and novel. The experimental evaluation covers multiple models with comprehensive quality and system performance metrics, demonstrating impressive super-linear scaling for larger models. The training-free nature makes it immediately applicable to existing models.", "weaknesses": "1.The paper only reports PSNR, SSIM, and LPIPS metrics without perceptual quality metrics like VBench scores that evaluate specific video generation dimensions such as subject consistency, temporal style, spatial relationships, and overall consistency. These metrics are crucial for assessing whether sparse patterns preserve the semantic and temporal coherence of generated videos.\n\n2.The evaluation does not compare against other sparse attention methods for video generation such as DiTFastAttn, MInference applied to video models, or cache-based methods like PAB. \n\n3.Critical design decisions lack ablation studies. The paper does not validate whether the mixed parallelism strategy outperforms a uniform strategy. The impact of dynamic scheduling versus naive scheduling is mentioned briefly but not thoroughly evaluated across different models and workloads. There is no sensitivity analysis on the sparsity hyperparameters (cs and ct) to show robustness across different efficiency-accuracy trade-offs.\n\n4.The paper provides only two visual examples in Figure 6. More extensive qualitative comparisons across diverse scenarios (minor vs. significant scene changes, rare vs. frequent object interactions) would strengthen the quality claims, especially given the method's high PSNR but potential for subtle temporal artifacts.\n\n5.While the paper claims hardware efficiency through layout transformation, it does not provide detailed kernel-level benchmarks comparing the proposed approach against naive sparse attention implementations across different sparsity levels.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)