Fast Video Generation with Sliding Tile Attention
Authors: Peiyuan Zhang
Yongqi Chen
Runlong Su
Hangliang Ding
Ion Stoica
Zhengzhong Liu
Hao Zhang
Venue: ICML 2025 poster (2025)
Keywords: Diffusion model
attention
machine learning system.
Primary area: deep_learning->generative_models_and_autoencoders

Abstract: Diffusion Transformers (DiTs) with 3D full attention power state-of-the-art video generation, but suffer from prohibitive compute cost -- when generating just a 5-second 720P video, attention alone takes 800 out of 950 seconds of total inference time. This paper introduces sliding tile attention (STA) to address this challenge. STA leverages the 
observation that attention scores in pretrained video diffusion models predominantly concentrate within localized 3D windows. By sliding and attending over local spatial-temporal region, STA eliminates redundancy from full attention. Unlike traditional token-wise sliding window attention (SWA), STA operates tile-by-tile with a novel hardware-aware sliding window design, preserving expressiveness while being \emph{hardware-efficient}. With careful kernel-level optimizations, STA offers the first efficient 2D/3D sliding-window-like attention implementation, achieving 58.79\% MFU -- 7.17× faster than prior art methods. On the leading video DiT model, Hunyuan, it accelerates attention by 1.6–10x over FlashAttention-3, yielding a 1.36–3.53× end-to-end speedup with no or minimum quality loss.

Fast Video Generation with SLIDING TILE ATTENTION
Peiyuan Zhang' Yongqi Chen
Runlong Su Hangliang Ding
Ion Stoica Zhengzhong Liu
Hao Zhang
Abstract
Diffusion Transformers (DiTs) with 3D full atten- tion power state-of-the-art video generation, but suffer from prohibitive compute cost when gen- erating just a 5-second 720P video, attention alone takes 800 out of 945 seconds of total inference time. This paper introduces sliding tile attention (STA) to address this challenge. STA leverages the observation that attention scores in pretrained video diffusion models predominantly concen- trate within localized 3D windows. By sliding and attending over the local spatial-temporal region, STA eliminates redundancy from full attention. Unlike traditional token-wise wise sliding window at- tention (SWA), STA operates tile-by-tile with novel hardware-aware sliding window design, preserving expressiveness while being hardware- efficient. With careful kernel-level optimiza- tions, STA offers the first efficient 2D/3D sliding- window-lik like attention implementation, achieving 58.79% MFU. Precisely, STA accelerates atten- tion by 2.8-17x over FlashAttention-2 (FA2) and 1.6-10x over FlashAttentionn33 (FA3). On the leading video DiT, Hunyuan STA reduces end-to-end latency from 945s (FA3) to 501s with- out quality degradation, requiring no training. En- abling finetuning further lowers latency to 268s with only a 0.09% drop on VBench. We make our codebase public at https://github.com//// com/hao-ai- lad/FastVideo.
FA3
256.59
NATTEN
313.92
21.44
CLEAR
307.44
Tiled
208
NATTEN
25.38
10.11x
Actual
Theoretical
100 150
200
250
300
350
(a) FLOPS distribution
(b) Attention kernel latency(ms)
Figure I. (a) Generating 5s 720P clip in Hunyuan involves pro- cessing 115K tokens, making attention the dominant cost. (b) Attention latency comparison: existing methods fail to trans- late FLOP reduction into wall-clock speedup: STA is hardware- efficient and achieves proportional speedup with sparsity.
synthesizing long-duration, visually coherent outputs (Pee- bles & Xie, 2023; OpenAI, 2024). Central to their success is 3D attention mechanism, which models spatial and tem- poral dependencies by flatterning video frames as a unified sequence of visual tokens (Yang et al., 2024b; Genmo-Team 2024; Hunyuan Vidiaaniiioo---a Tideo-Team, 2025). However, this design introduces significant computational overhead due to the quadratic complexity of attention, making training and infer- ence prohibitively slow video resolutions and durations increase. As illustrated in Figure 1, attention computation dominates the overall inference cost. Even with FlashAt- tention 3 (FA3) (Shah et al., 2024) and a high-end H100 GPU, Hunyuan Video (Hunyuan Video-Teeaae 2025) still re- quires 16 minutes to generate a 5-seconds 720p video. This bottleneck severely limits the practical deployment of DiTs.
Video data inherently exhibit high redundancy adjacent frames exhibit minimal differences, and spatially close pix- els tend to have stronger correlations. This redundancy sug- gests that treating every token independently in 3D attention may be unnecessarily expensive. In this paper, we hypothe- size that such redundancies are carried by 3D full attention in pretrained video diffusion models, which, if properly exploited, can drastically accelerate inference. To verify this, we visualize the attention scores of Hunyuan Video in Figure 2.
1. Introduction
Diffusion Transformers (DiTs) have emerged as the leading architecture for high-resolution video generation, capable of
"Co-second authorship 'University of California, San Diego University of Michigan, Ann Arbor 'Tsinghua University University of California, Berkeley 'Mohamed bin Zayed Uni- versity of Artificial Intelligence. Correspondence to: Hao Zhang <haozhang @ edu>.
The results reveal an intriguing 3D locality pat- tern: queries assign significantly higher attention scores to spatially and temporally nearby keys. To quantify this effect, we compute attention recall, measuring the fraction of total
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Fast Video Generation with Sliding Tile Attention
Frame 0
Frame 20
To overcome this, develop SLIDING TILE ATTEN- TION (STA), hardware-aware attention mechanism that rethinks sliding window computation via system-algorihhm co-design. We define tile as a contiguous group of tokens forming a by the block size in FlashAttention. Instead of sliding over contiguous tokens, STA operates tile-by-tile, enabling effi- cient memory access and parallelism while effectively pre- serving the 3D locality. Inspired by FA3, STA adopts consumer-produucer paradigm, where producer warpgroups asynchromously load data from HBM to SRAM while con- sumer warpgroups compute attention. Because STA slides over tiles instead of individual tokens, eliminates the need for explicit attention masking at computation, significant overhead observed in other SWA implementations. The sparse attention mask is managed entirely by the producer warpgroups; hence, the computation on consumer wrap groups remains dense and hardware-efficient. As result, STA is the first higher-order biding-windoww attention to achieve wallcock speedups proportional to sparsity (Fig- ure (b)).
cube, with its size determined
Frame
Frame 60
Figure 2. Visualization of attention locality. The green point means the query point and the magma-colored regions indicate areas of high attention values in response to the query. Instead of attending the entire image, the query's attention forms concentrated local hotspot.
attention scores concentrated within a local window, As shown in Figure left, despite training with full 3D atten- average, local window covering only 15.52% of the total token space accounts for 70% of the total attention score.
tion, Hunyuan Video exhibits strong locality:
Besides efficient computation, selecting optimal window sizes is crucial to preserving generation quality. We find that different attention heads exhibit specialized locality pat- terns some heads focus on finer details in small area, yet others capture broader context at a larger window - which we term as head specialization. Importantly, this head spe- cialization remains agonistic to prompts, as evidenced in Figure 3. Based on this property, we develop a simple yet effective method to automatically configure the optimal window size per head via profiling, striking balance be- tween efficiency and quality. With STA, Hunyuan Video can generate a 5-second 720P video in 501s with no or mini- mal quality loss in a plug-and-plas manner. In comparison, Hunyuan Video with Ataetinttntionn22 takes 1496s, while FlashAttention-3 takes 945s. STA achieves end-to-end speedup of 2.98x over FlashAttention-2 and 1.89x over FlashAttention-33. Additionally, by fine-tuning diffusion models under more radical attention sparsity, we unlock even greater efficiency, delivering -353x end-to-end speedup compared with FlashAttention-3.
Recall Average
Recail Std Acress Prompts
1.0
0.8
2agi AcToes Prompls
Recall Awersge
20
0.6
Layer
0.6
30
30
0.0
0,4
-0.2
10
0.0
Head
Figure 3. Left: Fraction of attention scores within (12, 24, 24) local window across diffussin steps and 10 different prompts. Most heads show high recall, indicating local attention pattern. Right: Despite the different recall across heads, the standard deviation across prompts remains low.
This observation seemingly suggests that sliding window attention (SWA) is an ideal alternative to retain attention expressiveness while reducing computational cost. How- ever, existing 2D or 3D SWA implementations, such as NATTEN (Hassani et al., 2023) and CLEAR (Liu et al., 2024), fail translate FLOP reductions into proportional wall-clock speedups, as shown in Figure 1b. Their ineffi- ciency arises because higher-order (2D/3D) sliding window attention creates a highly irregular attention mask, wast- ing many computations and generating significant masking overhead. The computation pattern for SWA is inherently GPU-unfriendly, resulting in poor hardware utilization (see $2.2).
This paper makes the following contributions: (1) We iden- tify and quantify 3D locality and head specialization in state- of-the-art video DiTs, revealing substantial redundancy in full 3D attention. (2) We introduce SLIDING TILE ATTEN- TION, tile-based sliding window attention mechanism. Our optimized kernel achieves minimum overhead compared to FlashAttention 3 with an MFU of 58.79%. (3) STA accel- crates attention by > 10x and end-to-end video generation by up to x with no or minimum quality loss.
2

---

Fast Vidco Generation with Sliding Tile Attention
2. Problem
ing SWA high-dimensional data fails to fully utilize GPU computation. Existing 2D/3D SWA kernlls, such Tiled NATTEN (Hassani et al., 2023), shift window centers at im- age and video boundaries to ensure each query attends to a constant number of keys. It also improves kernel efficiency through input tiling and kernel fusion, but its performance suffers from problems described next.
In this section, we provide background on why sliding win- dow attention (SWA) inefficient in figh-dimensional set- tings, particularly for Video Diffussin Transformers (Pee- bles & Xie, 2023). For clarity, our notation assumes cubic window, tile, and video sizes unless stated otherwise, though our approach extends to non-cubic configurations.
2.2. Inefficiency of Sliding Window Attention
2.1. Attention in Video DiT's
Implementing 2D/3D SWA with FlashAttention requires defining its attention mask. As discussed in $2.1, FlashAt- tention calculates and applies masks at the block level. Based different intra-block masking strategies, we cat- egorize attention blocks into three types: dense (with all attention scores retained), empty (mask out all values), and mixed (with some scores removed), shown in Figure 4. Empty blocks can be entirely skipped during computation. Mixed blocks, while sparser than dense blocks, introduce significant overhead. First, they require full computation for the entire block before applying masks to retain or dis- card attention scores, introducing unnecessary computations. Second, to determine which position to retain or discard, the attention kernel needs to calculate the value of the mask based on the SWA pattern and the block's 's position relative to the entire attention mask. The mask calculation introduces substantial overhead - in the case of calculating the simple causal mask, FlexAttention (Dong et al., 2024) reports a 15% overhead. We show in $4.1 that the mask overhead dominates the attention latency for more complex masking patterns such as sliding windows. In SWA, each query at- tends to a distinct set of keys, resulting in a zigzag pattern in the attention map and generating numerous mixed blocks, as shown in Figure (a). Although the state-of-the art sliding window attention implementation, Tiled NATTEN, aims to reorder the inputs increase the number of dense blocks, a significant portion of blocks remains as mixed blocks.
State-of-the-art Video DiTs employ 3D full attention to mix signals across tokens, allowing each token to attend to any other token. Given a video latent of shape (I., I,, L)) (often encoded via a VAE), this is achieved by flattening the 3D data into a sequence of length 1 and applying full bidirectional attention. However, as the sequence length grows cubically with L, even small increase in resolution or duration leads to a significant computational burden. As result, applying 3D attention to high-resolution, long- duration videos becomes prohibitively expensive.
Formally, let Q,K, V E Xd represent the query, key, and value of input sequences for a single attention head, where N L3, and d is the dimension of each head. Let M { {-o 03* NNN represents the attention mask. The attention operation is defined as:
S QKT VTx
A= S=Softmax$S + M), O = AV (I)
A naive attention implementation constructs the full A,A,M M E NNN on GPU HBM, leading to both ((N3) memory overhead and excessive data transfers between HBM and SRAM. FlashAttention mitigates this issue by tiling input sequences into smaller blocks. Through online softmax, each GPU SM loads a block of queries, keys, and values into SRAM, performs computation, and writes the final result to HBM, avoiding the materialization of A and S. To reduce computation cost, we can apply attention mask to control sparsity, with the mask computed on-chip per block to avoid the (N2)) memory cost of a global mask. This sparsity can reduce latency by skipping masked-out attention regions. However, as we show in the next section, this is not always effective.
In summary. SWA beyond ID sequences introduces two major inefficiencies: (1) mixed blocks do not reduce FL.PPs due to sparsity, and (2) they incur additional mask evaluation overhead, making them slower than dense blocks. Efficient sparse attention in 3D should minimize mixed blocks while maintaining locality. This insight drives the development of our novel STA method, which significantly reduces mixed blocks while preserving the locality property of SWA.
Sliding window attention (SWA) is a widely used sparse attention method that reduces computation costs while pre- serving locality. In SWA, a query attends only to keys within a fixed window, and stacking multiple attention lay- naturally expands the receptive field beyond the window size. SWA has been extensively studied in natural language processing (Beltagy et al., 2020; Jiang et al., 2023). As motivated in $1, ofate-of-thh-ar art video diffusion models ex- hibit a strong 3D locality pattern. making them a natural candidate for applying 3D SWA. However, directly apply-
2.3. Alternative Methods for 3D Attention
Other than sparsifying the attention maps, some approaches accelerate video diffusion by decomposing the 3D attention into alternating spatial and temporal components (Singer et al., 2022; Wang et al., 2023; Chen et al., 2023; Ma et al., 2024). However, these methods have been largely super- seded by full 3D attention in state-of-ther video mod-
3

---

Fast Video Generation with Sliding Tile Attention
192
Empr Blcck
80
95
112
160
178
Key Tokens Idy,
200
100
400
300
Tokens ld
200
300
400
500
( NATTEN
(b) NATTEN
(c) STA
Figure The attention map of NATTEN, Tiled NATTEN, and STA. plot an image size 24x24 and a 12x local window. The tile size is set to 4 x4. (a) NATTEN creates many mixed blocks that are very inefficient for Flash Attention computation. (b) Tiled NATTEN increases the number of dense blocks, but the mixed blocks persist. (c) STA completely climinates the mixed block, making the computation extremely friendly for GPU. Note that mainly show STA's 's application in 3D scenarios for video generation in this paper. but for better illustration, we present the 2D scenario in this plot.
Table 1. Ratio of dense and mixed blocks for tiled NATTEN and STA with tilc size (4,4,4) and vidco size 448,48,48). STA generate only dense blocks, which is computationally friendly than mixed blocks on GPU.
els (Zheng et al., 2024; Lin et al., 2024; Genmo-Team, 2024; Hunyuan Video-Team, 2025). We hypothesize that this is because alternating spatial and temporal attention fails to capture interactions between tokens that are offset in both spatial and temporal dimensions. For instance, query at (1, 1) attend keys at (1, X, X) or (X, 1, 1), but not at (2, 2, 2), even though they spatially close. This disrupts the 3D locality pattern, which we have shown as key characteristic of video diffusion models.
Attention
Window Size
Dense Block
Mixed Block
Tiled NATTEN
(11,11,11)
0.06%
7.17%
STA STA
(12, 12, 12)
1.56%
0.0%
(20, 20, 20)
7.23%
0.0%
arranging queries in tile with consecutive token indices, we form dense FlashAttention blocks. This design allows each query tile to attend densely to key tiles within the window, eliminating mixed blocks and improving compute efficiency. We illustrate STA attention mask in Figure 4 (c).
3. Methods
3.1. ST.DIIG TILE ATTENTION
In vanilla sliding window attention, each query attends local window centered around it, resulting in different queries attending to distinct key groups. This lack of shared attention key groups is the root cause of irregularities in SWA's attention map, creating mixed blocks. We propose SLIDING TILE ATTENTION (STA), novel sliding window attention variant that exclusively operates on dense blocks and empty blocks. As shown in Figure 5, STA organizes queries and keys into tiles. All queries in the same tile attend to the same set of keys within their common local window, ensuring more structured attention pattern. By setting the tile area equal to the block size in PlashAttention and
Formally, for 3D STA, given video of dimension (L, L, L) and FlashAttention block size of (B, B), STA sets the tile size T such that B = T It further assumes that both the video size L and window size W are integer multiples of T. The video is partitioned into non-overlapping tiles of size (T, T, T), and flattened into 1D sequence in way that tokens within the same tile have consecutive sequence indices. Conceptually, STA slides the window with step size of (T, T,Y). For each step, it computes attention be- tween the central query tiles and all key tiles within the window, producing (T] dense attention blocks without

---

Fast Video Generation with Sliding Tile Attention
Intuitively, each query tile will only attend to its local win- dow in STA, which has (W) tiles of keys, creating the same number of blocks in the attention map. We apply The- orem 3.1 and Theorem 3.2 to calculate the ratio of different blocks and report them in Table 1.
Kernel-level optimization. STA can be efficiently imple- mented with FlexAttention, which provides enough func- tionality to skip all empty blocks and avoid adding unnec- essary intra-block masks on the dense blocks. We can fur- ther optimize the sparse attention masks by disaggregat- ing the inter-block mask logic from the compute kernels, Thus, we implement our attention kernels based on Thun- derKittens (Spector al., 2024) and FlashAttention3 (Shah et al., 2024). Our implementation splits the threadblock into compute warpgroups and data warpgroups, and the inter-block mask is completely managed by the data warp- groups. Each compute warpgroup is responsible for calcu- lating one query block, which always resides the SRAM (Split-Q (Dao, 2024)). The data warpgroup responsible for asynchronously loading the KV blocks from HBM to SRAM. For each block of query, the data warpgroup needs to decide which key and value blocks the query block will attend to in STA and only load those blocks. Since the data warpgroups are asynchronous, the overhead of calculating the inter-block mask in STA and deciding which data to load can be hidden with overlapping. On the other hand, the compute worker is completely oblivious of the sparse attention pattern. It performs attention computation with the key value blocks in shared memory loaded by data workers, and once all data is consumed in the circular cache, the computation is finished.
Figure 5. 2D SLIDING TILE ATTENTION with tile size (2, 2) and window size (6. 6). After attending to all the key tiles, each query tile will generate nine 4x4 dense blocks in the attention map. We showcase 2D STA for better illustration. 3D STA can be inferred similarly.
mixed blocks.
To demonstrate STA superiority in creating a GPU-friendly compute pattern, we give the following formula to quanti- tatively measure the different types of blocks in 3D Tiled NATTEN and 3D STA.
Theorem 3.1. Consider a tiled NATTEN configuration with tile size TT), window size (W, W, W), and video size (I, L, L). Let the FA block size be (B, B), where B = T Ignoring boundary effects, the number of dense blocks is given by:
W +1
-.00)
3
I. 3 (
Neeuue = maxx
-
2T
T
3.2. Applying STA to Video Diffusion Model
The number of mixed blocks in tiled NATTEN is:
We can either apply STA to directly replace the 3D attention in pretrained video DiTs without training, or with small amount of training which enables even greater sparsity.
Nmix (2
W 2T
3
L 3
+1
Ndense-
T
Training-free. As illustrated in Fig. 3, video diffusion mod- els exhibit a pronounced 3D locality and head specialization pattern. Different transformer heads have different levels of locality, but their pattern is largely consistent across differ- ent prompts. We can exploit this property to search for the optimal window size for each head on a very small number of prompts, and expect the search result to work well on other prompts. We develop simple heuristics to find such configuration in Algorithm I and decide the final configura- tion by averaging the mask-search loss across 16 prompts. In practice, we keep full attention for the initial To timesteps (Li et al., 2024a; Zhao et al., 2024; Lv et al., 2024), and then apply STA for the rest timesteps.
Intuitively, for a block to be dense in NATTEN, the window size should be at least twice the size of the tile size, such that the left-most query in the tile can attend to the right-most query. On the other hand, the left-most query in a tile can still attend keys that are mixed blocks.
W- tiles further left, creating 2T
Theorem 3.2. With the same notation, if W is an integer multiple T, the number of dense blocks in SLIDING TILE ATTENTION is:
W T
T
3
Saeuse
Finetuning. Beyond searching for the optimal mask per attention head without tuning. we can fix a window size with a high sparsity and fine-tune the model to adapt. Since
All remaining blocks are empty and there are no mixed blocks.
5

---

Fast Video Generation with Sliding Tile Attention
HunyuanVides 15 mins 45
STA-tf-1.880 8 mins 21
STA-1-2.43x
mins s
A-DIT-1.366 11
Prompt: Tour of an art gallery with many beautiful works of art in different
Figure 6. Qualitative example of 720P 5-second videos. While fine-tuning introduces minor shifts in the output distribution of STA-t-2.43x. the model still preserves high video generation quality. Videos generated by A-DIT are generally less sharp than those generated by the original Hunyuan Video and STA.
STA follows the 3D locality property, this adaptation can be learned efficiently with minimal training overhead (in experiments, hours compared to the pretrain cost of video diffusion models). Although each attention layer is restricted to a local window, the receptive field expands through stacked transformer lay- ers, enabling the Diffusion Transformer to generate globally coherent videos in the end.
Algorithm STA Mask Search
H100, which is minimal
Input:
Transformer model M, Total steps T, Mask pattern list P, Keep first To timestep full attn
Output: Dictionary dict that stores selected mask pattern for each head
Initialize dict
for t = To + 1 to do
for each layer head combination (1, h) in M do
We three different loss terms during finetuning. The attention distillation loss directly supervises the intermediate attention patterns of our STA to match the original dense attention behaviors:
O с- + output of original (t, h)
Initialize minimum _lsss Initialize best-pattern <- null for each in P do
Mask head h for layer / using mask pattern O' + (attn output of M after masking) loss 4- (S, O')
Lattn N
Σ 12) (s,,t,)) S (x.,
where f(() and f denote the intermediate attention outputs from the i-th transformer layer of our sliding tile model and the original attention teacher. This loss ensures each sparse attention layer to approximate its corresponding dense at- tention teacher. We also add a final layer loss to align the final output of the student and teacher:
if loss < minimum loss then minimum loss loss
best-pattern p
Record best for (t, h) in dict return dict
Liinal [ c) Fo(s,,t,))
(3)
6

---

Fast Vidco Generation with Sliding Tile Attention
Additionally, we employ a data loss following the flow matching formulation (Esser et al., 2024; Lipman et al., 2022):
Win
Tie
Loss
STA-tf-1.89x
5.0%
03.0%
12.0%
HunyuanVideo
Edata zo) -
(4)
STA-tf-1.89k
Vs-
06.5%
23.5%
A-DiT-1.366
where 20 represents the VAF latent of the input frame, dy is the noised latent at diffusion step t, and e denotes the text embedding.
STA 2.
70 0%
19.0%
11.0%
A D-DiT-1.8x
0%
20% 40%
60%
80%
100%
The complete objective combines these terms:
Figure 7. Human evaluation on 200 prompts from MovieGen Bench (Polyak et al., 2024). STA achieves 1.89x end-to-end speedup while maintaining performance comparable to the original Hunyuan Video. Additionally, STA consistently outperforms A- DiT across different inference budgets.
min E Pexppte)..-.000...ee...... + final + >Lattn] (5)
The detailed training setup can be found in Appendix B.
4. Experiments
We evaluate STT on Hunyuan a state-of-the-art open video DiT comparable to many proprietary ones!. We gen- erate Hunyuan outputs with 117 frames at a 1280x768 reso- lution. After VAE compression and tokenization, this cor- responds latent video of shape (30, 48, 80). Beyond video, we also apply STA the leading image diffusion model, FLUX (Black-Forest, 2023), to demonstrate its ef- fectiveness in 2D. We evaluate both efficiency and video quality. STA kernel's efficiency is measured using standard metrics such as MFU and latency, as detailed in $4.1. For end-to-end speedup on DiT, we report measured wall-clock latency, excluding time spent on VAE and text encoder. For generated video quality, find existing automated met- rics are often unreliable. Following Polyak et al. (2024), we emphasize human evaluation and present the results in $4.2. For completeness, we also report automated metrics, including VBench (Huang et al., 2024), SSIM, PSNR, and CD-FVD (Ge al., 2024). We provide an example in Figure 6. with additional qualitative results available in Appendix Section G.
results for lower sparsity setting (56%) provided in ta- ble 7. Since STA builds FA3 and ThunderKittens, use ThunderKittens' FA3 as the baseline and report the rel- ative speedup of all sparse attention kernels. To quantify efficiency, We introduce kernel efficiency, defined as the ratio of a sparse kernel's 's MFU to that of full attention. This metric captures how well sparse kernels translate theoretical FLOP reductions into actual latency improvements.
The results highlight the inefficiency of existing methods. Despite reducing TFLOPs to 15.65, CLEAR incurs a 0.86x slowdown. Similarly, NATTEN variants, despite reaching 0.91 sparsity, still suffers from inefficiency: its vanilla ver- sion slows down by 0.85x, while its optimized tiled variant in Altenttnn achieves only a modest 1.27x speedup. Among existing methods, Swin (Liu et al., 2021a) is the only kernel with MFU exceeding 40% and kernel efficiency above 60%. However, Swin is not sliding-window window-bass based attention, and we argue its efficiency comes at the cost of expressiveness in $4.4.
Baseline methods. We compare STA to other sparse or window attention designed for image or video, including CLEAR (Liu et al., 2024), NATTEN (Hassani et al., 2023), and Swin (Liu et al., 2021b). Also, we adapt the caching- based method, A-DiT (Chen et al., 2024), for evaluation on Hunyuan Further details on the baseline methods and their implementations can be found in Appendix C.
Compared to Tiled NATTEN, one of the most optimized slid- ing window attention implementations, the key algorithmic difference in SLIDING TILE ATTENTION is changing the sliding unit from token to tile. Despite its simplicity, this modification significantly improves efficiency. To ensure a direct comparison with tiled NATTEN, we also implement STA in Flexnttention STA improves MFU from 8.20% to 41.03%. Further, with our optimized kernel for asyn- chronous data loading and inter-block mask management in ThunderKittens, STA achieves a 10.45 x speedup over full attention. Additionally, we evaluate STA with 58.33 % sparsity, where it achieves 2.37x speedup. This efficiency gain enables a significantly larger window size while still outperforming NATTEN. To our knowledge, STA is the window
4.1. Efficiency of SLIDING TILE ATTENTION
We benchmark the efficiency of various attention algorithms assuming generating 720P 5s videos using Hunyuan, shown in Table 2. The configuration ensures that all sparse ker- nels maintain approximately 90% sparsity, with additional
We skip evaluating other open models (Lin et al., 2024; Zheng al., 2024; Ma et al., 2024) due their significantly lower overall quality compared to Hunyuan.
first sliding-window sparse attention that achieves both 3D locality and hardware efficiency.
7

---

Fast Video Generation with Sliding Tile Attention
Table Forward speed of sparse attention kemelss in setup aligned with Hunyuan inference configuration (hf16, 720P, 5s, 115.2K seq len, diead 128, # heads 24). Config controls the window size of each sparse attention.
Methods
Implementation
Config
Sparsity
TFLOPS
(atency(ms)
MFU
Kernel Efficienccy
Speedup
FA 3
Thunderkittens
0.00%
164.03
265.28
62.49%
100.00%
1.00x
FA 3
CUDA
0.00%
164.03
256.59
64.61% %
103.39%
1.03x
CLEAR
FlexAttention
r=16
90.46%
15.65
307.44
5.15%
8,24%
0.86x
NATTEN
FlexAttention
w=(19,25,25)
89.69%
16.91
313.92
5,44%
8.71%
0.85x
Tiled NATTEN
CUDA
w=(19,25,25)
89.69%
16.91
458.36
3.73%
5.97%
0.58x
Tiled NATTEN
FlexAttention
(=911225,25)
89.69%
16.91
208.36
8,20%
13.12%
x.27
Swin
FlexAttention
w=22333,322
87.42%
20.64
47.90
43.55%
69.69%
5.54x
STA
FlexAttention
(=118,24,24
91.00%
14.76
36.36
41.03%
65.66%
7.30x
STA
ThunderKittens
w=(30,40,40)
58.33%
68.35
111.73
61.82%
98.93%
2.37x
STA
ThunderKittens
w=(18,24,24)
91.00%
14.76
25.38
58.79%
94.09%
x0.45
4.2. Human Evaluations
Table 3. Training- free performance with varying sampling steps.
We assess human preference across five models that achieve the best quality performance: (1) Hunyuan Video; (2) STA-if- 1.89:: Hunyuan Video with 1.89x speedup via training- free mask search, (3) STA-1-2.43x: Hunyuan Video with 2.43x speedup via finetuning with STA, (4-5) two variants of A- DiT (1.36x and 1.8x speedup). Other baselines such as CLEAR or Swin are either prohibitively slow or produce subpar quality. Following MovieGen (Polyak et al., 2024), we randomly sample 200 prompts from the MovieGen Bench and conduct pairwise comparisons between these models. Evaluators select the video with higher overall quality mark both as tic.
D-DiT shows consistently worse quality compared STA
Model
SSIM
PSNR
CD-FVD : Lateney
Speedup
steps = 50
A-DIT
72.86
18.09
122.74
693s
1.36x
STA
87.67
28.76
66.12
501s
1.89x
steps = 25 D-DIT
77.91
19.86
196.25
352s
1.34x
STA
88.96
28.99
76.34
250s
1.89x
steps =
A-DIT 83.19
21.20
201 224
144s
1.32x
STA
87.84
27.14
84.80
105s
1.76x
In Figure 7, STA-t-2.43x decisively outperforms D-DIT- 1.8x, achieving a dominant 70.0% win rate versus 1 11.0%, despite a greater speedup. Similarly, STA-tf-1.89x sur- passes A-DiT-1.36x with a 66.5% win rate against 10.0%. Compared the original HunyuanVideo, STA maintains competitive quality, with STA-tf-1.89, achieving 83.0% tie rate-indicating near-parity in most cases. Though it has 7.0 percentage point lower win rate than its loss rate, this tradeoff' comes with a 1.89x speedup, demonstrating strong quality preservation alongside significant efficiency gains. These results establish STA as achieving a superior quality-efficiency tradeoff compared A-DiT.
inference respectively.
Our training-fret STA consistently outperforms A-DIT even with much higher speedup. At steps, STA achieves substantial improvements in all metrics, with 14.81 higher SSIM (87.67 vs. 72.86) and 10.67 higher PSNR (28.76 vs. 18.09). A-DiT's CD-FVD score is 56.62 higher than STA (122.74 vs. 66.12, where lower is better). This gap grows to I 119.9 at 25 steps and 116.44 at 10 steps. Qual- itatively, A-DiT consistently produces visually degraded outputs across all sampling steps, exhibiting compromised structural similarity and diminished fine details, while STA maintains high fidelity to the original model.
4.3. Training-free Results
4.4. Finetuning Results
In Table 3, we evaluate mask-search STA and A-DiT on VBench prompts, testing robustness across different sam- pling steps. For each diffusion step count, we report SSIM, PSNR, and CD-FVD, using Humyuan Video's outputs at the same step count as the reference. We set the scheduler shift to 17 for 10-step inference and 7 for 25- and 50-step infer- ence following Hunyuan's default settings. We keep full attn for the first 12,6,3 steps for 50-step, 25-step, and 10-step
tine-tuning on new data introduces slight distribution shifts, meaning the same prompt may yield different, yet high- quality, video variants. Consequently. similarity metries like PSNR become less suitable, and we instead rely on VBench(Huang et al., 2024), a comprehensive benchmark for video generation. We first examine the impact of di- rectly replacing full attention with sparse attention, with- out tuning, to evaluate how well each algorithm approx-
8

---

Fast Video Generation with Sliding Tile Attention
Table Performance on VBench across different sparse attention patterns. STA achieves both high-quality video generation and significant speedup, while CLEAR and Tiled NATTEN suffer from efficiency issues and Swin suffers from quality degradation.
Bennh Quality
VBench Semantic
VBench Total
Methods
Config
Attn Sparsity
PFLOPS
Lateney
Speedup
FA2
85.34% 85.34%
72.17% 72.17%
82.71 %
0.00% 0.00%
574.16 574.16
1496s 945s
0.63 x
FA3
82.71% %%
1.00
w.o training
CLEAR
r=32
84,41 %
74.20%
82.37%
56.23%
280.90
2567s
0.37 x
Tiled NATTEN
w=3,0,4,,41
84.61% %
75.00%
82.69%
58.33%
269.92
1858s
0.51 X
Swin
w-(48,64,64)
80.91%
71.35%
79.00%
55.81%
283.11
762s
1.24
Swin
w=330,40,40)
78.84%
72.28%
77.53%
76.49%
175.20
497s
1.90
STA
w=(30,40,40)
84.63%
73.83%
82.46%
58.33%
269.92
527s
1.79
STA
w=118,24,24)
81.47%
77.03%
80.58%
91.00%
99.54
268s
3.53
w. training
Swin
w=330,40,40)
77.50%
67.39%
75.48%
55,81%
283.08
497s
1.90
STA
w= W3030,24,40)
85.37%
73.52%
83.00%
75.00%
182.99
388s
2.44 x
STA
w=(18,24,24
84.76%
74.05%
82.62%
91.00%
99.54
268s
3.53 x
imates full 3D attention. In Table 4, CLEAR and Tiled NATTEN retain reasonable video quality (VBench scores of 82.37% and 82.68%, respectively) compared to full at- tention (8)..1%) However, despite sparsifying attention, these methods paradoxically increase end-to-end inference latency. Swin presents the opposite challenge: while it achieves moderate speedup (1.24x-1.90%) its rigid, non- overlapping window partitions prevent local queries and keys from attending each other if they fall into separate windows, violating the 3D locality property. This results in degraded video quality, and crucially, fine-tuming with Swin attention not only fails to recover performance but further lowers the VBench score. In contrast, STA addresses both quality and efficiency limitations. With a window configu- ration of =33.3,,, it achieves 91.00% attention sparsity, yielding 5.76x FLOPs reduction and x actual la- tency reduction. 2 Importantly, this efficiency gain comes with minimal quality tradeoff: STA maintains an 80.58% VBench score in the training-free setting and improves to 82.62% with fine-tuning.
2021a; Arar et al., 2022; Yang et al., 2024a) can decompose the softmax operation in quadratic attention using kernel or gate functions to achieve linear complexity. However, these methods have not yet been successful in video DiTs. Concurrent to our work, quantized attention (Zhang et al., 2024; 2025a;b) and sparse attention (Zhang et al., 2025c; Xi et al., 2025) are proposed with different design for video DiTs. Another major bottleneck in diffusion models the large number of diffusion steps. Several techniques have been proposed to accelerate sampling without sacrificing quality, including DDIM (Song et al., 2020) and faster ODE and PDE solvers using numerical methods (Song & Ermon, 2019; Lu et al., 2022a;b; Jolicoeur-Martinne New methods have also emerged to further reduce the num- ber of sampling steps, such consistency distillation (Kim et al., 2023; Song al., 2023; Salimans et al., 2024; Xie et al., 2024), adversarial distillation (Sauer al., 2023), and other distillation approaches (Li et al., 2024b; Yin et al., 2023; 2024). STA is largely complementary to these meth- ods.
et al., 2021).
4.5. Results on Image Super-Resolute
6. Conclusion and Future Work
We also apply STA to speed up image superresolution with SDEdit (Meng et al., 2022). We find that FLUX with STA achieves comparable generation quality to CLEAR while offering significantly higher efficiency. Experiments for can be found in Appendix
We introduce SLIDING TILE ATTENTION to accelerate video diffusion models, with optimized kernel for high- order sliding- wwwdww--idwke attention, enabling efficient GPU execution while preserving the locality property. Experi- ments demonstrate that SLIDING TILE ATTENTION accel- erates video generation with minimal or no quality loss. Conceptually, our method is orthogonal other accelera- tion techniques, such caching and consistency distillation. We plan to explore their combined effectiveness for further efficiency gains in future work.
5. Related Work
We review additional related work in diffusion accelera- tion. Linear attention methods (Wang et al., 2020; Liu et al.,
Other memory-bound operations, such as LayerNorm and modulation, likely contribute to inference overhead, preventing the full FLOPs reduction from translating directly into speedup.
9

---

Fast Video Generation with Sliding Tile Attention
Acknowledgements
ating optimized attention kemels, 2024. URL https:
/ / arriivv www./r/g//////////////// /2020. 05496.
We would like to thank Will Lin and Wei Zhou for their open- source contribution in VastVideo. The work is supported by UCSD HDSI, Nvidia, and a faculty research award from Google.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel. F., et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
Impact Statement
Ge, S., Mahapatra, A., Parmar, G., Zhu, J.-Y., and Huang, J.-B. On the content bias in frechet video distance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7277-7288, June 2024.
Our work addresses computational bottlenecks in Diffusion Transformers by introducing efficient attention kernels that reduce video generation time while maintaining output qual- ity. The improved efficiency makes video generation more practical for researchers and developers working with lim- ited computing resources, potentially benefiting Al-driven video applications across creative industries, education, and so on. While faster video generation could potentially en- able misuse, existing content detection and watermarking techniques can help mitigate such risks.
Genmo-Team. Mochi 1. https://github.com genmoai /modelss s, 2024.
Hassani, A., Walton, S., Li, J., Li, S., and Shi, H. Neighbor- hood attention transformer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Overall, the benefits of more efficient video generation sig- nificantly outweigh potential concerns, representing mean- ingful step toward accessible video Al systems.
Huang, Z., He, Y., Yu, J., Zhang, F., Si, C., Jiang, Y., Zhang, Y., Wu, T., Jin, Q., Chanpaisit, N., et al. Vhench: Compre- hensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21807-21818, 2024.
References
Arar, M., Shamir, A., and Bermano, A. H. Learned queries for efficient local attention. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern
Hunyuan Video- -eam. Hunyuanvideo: A systematic frame- work for large video generative models, 2025. URL 03603.
Recognition, pp. 10841-10852, 2022.
https: : /ttaaaavvrgggass////a/aeaaaaaeea
Beltagy, I,, Peters, M. E., and Cohan, A. Longformer:
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-
The long-document transformer, 2020. URL https:
//arxiv. org/ abs/2004. 05150.
A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023. URL https:
Black-Forest.
https://githie com/ plack-forest-ler /flux, 2023.
Flux.
/ //trxiv.orgg/eeee/... www.obs////.a..oo.ooooo....... 22100.6655..
Chen, H., Xia, M., He, Y., Zhang. Y., Cun, X., Yang, S., Xing, J., Liu, Y., Chen, Q. Wang, X., Weng, C., and Shan, Y. Videocrafterl: Open diffusion models for high-quality video generation, 2023. URL peattss:/aaxiv...o abs abs/2310. 19512.
Jolicoeur-Martineau, A., Li, K., Tiché-Taillefer, R., Kach- man, T., and Mitliagkas, I. Gotta go fast when gen- erating data with score-based models. arXiv preprint arXiv:2105. /4080, 2021.
Kim, D., Lai, C.-H., Liao, W.-H., Murata, N., Takida, Y., Uesaka, T., He, Y., Mitsufujii Y., and Ermon, Consis- tency trajectory models: Learning probability flow ode trajectory of diffusion. arXiv preprint arXiv:2310.02279, 2023.
Chen, P., Shen, M., Ye, P., Cao, J., Tu, C., Bouganis, C.- S., Zhao, Y., and Chen, T. Delta-dit: A training-free acceleration method tailored for diffusion transformers. CoRR, abs/2406.01 125, 2024. URL https: //doi. org/10. 010110.445500 400..08550/ra00 arw.vv. 2406. 01125.
Li, D., Shao, R., Xie, A., Xing, E. P., Ma, X., Stoica, I., Gonzalez, J. E.,, and Zhang, H. Distflashatm:: Distributed eefori-efficient attention for long-contex, llms training. In First Conference on Language Modeling, 2024a.
Dao, T. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth In- ternational Conference on Learning Representations, 2024. URL httpt:tt/////tttt https:///penreviiew met/forunt i id=m7n2Xyh9Ecc.
Li, Feng, W., Fu, T.-J., Wang, X., Basu, S., Chen, W., and Wang, W. Y. T2y-turbo: Breaking the quality bottleneck of video consistency model with mixed reward feedback. arXiv preprint arXiv:2405. 205.0 2024b.
Dong, J., Feng, B., Guessous, D., Liang, Y., and He, H. Flex attention: A programming model for gener-
10

---

Fast Video Generation with Sliding Tile Attention
Lin, B., Ge, Y., Cheng, X., Li, Z., Zhu, B., Wang, S., He, X., Ye, Y., Yuan, S., Chen, L.,, al. Open-sora plan: Open-source large video generation model. arXiv preprint arXiv:2412.00133 2024.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 4195-4205, 2023.
Lin, T.-Y., Maire, M., Belongie. S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zimick, C. L.., and Dollár, P. Microsoft coco: Common objects in con- text, 2015. URL https:////// http//aaxivv... org/ harrggabs/4455 /1055. 0312.
Polyak, A., Zohar, A., Brown, A., Tjandra, A., Sinha, A., Lec, A., Vyas, A., Shi, B., Ma, C.-Y., Chuang, C.-Y., et al. Movie gen: cast of media foundation models. arXiv preprint arXiv:2410.1372 2024.
Salimans, T., Mensink, T.,, Heek, J., and Hoogeboom, E. Multistep distillation of diffusion models via moment matching. arXiv preprint 2xiv:2246.004003. 2024.
Lipman, Y. Chen, R. T., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. arXiv preprint wxXv:2210.007447 2022.
Sauer, A., Lorenz, D., Blattmann. A., and Rombach, R. Adversarial diffusion distillation. arXiv preprint arXiv:2311.10422, 2023.
Liu, S., Tan, Z., and Wang. X. Clear: Conv-like linearization revs pre-trained diffusion transformers up, 2024. URL https: rxiv. org/
wwt./bs/////oooooooro/ooo/ssaaer abs 2412 16112.
Shah, J., Bikshandi, G., Zhang. Y., Thakkar, V., Ramani, P., and Dao, Flashattention-3: Fast and accurate attention with asynchrony and low-precision, 2024, URL https: / www.iiv.ogg/ataie / 2077 08608.
Liu, Z., Lin, Y., Cao, Y. Hu, H., Wei, Y., Zhang, Z.,, Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103. 14030, 2021a.
Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., Gafni, O., Parikh, D., Gupta, S., and Taigman, Y. Make-a-video: Text-to-vidco generation without text-video data, 2022. URL https: / //arxiv. wwwxiiv.org//ee abs 2209 14792.
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows, 2021b. URL https: //arxiv. org/ abs/ 2103.111.. 14030.
Song, J., Meng, C., and Ermon, Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020.
Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, J. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-57877 2022a.
Song, Y. and Ermon, S. Generative modeling by estimat- ing gradients of the data distribution. arXiv preprint arXiv: 1907.9560000000 2019.
Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, J. Dpm- solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint 1..00922 2022b.
Song, Y., Dhariwal, P., Chen, M., and Sutskever, I. Consis- tency models. arXiv preprint arXiv:2303.0 2023.
Spector, B. F., Arora, S., Singhal, A., Fu, D. Y., and Ré C. Thunderkittens: Simple, fast, and adorable ker- nels, 2024. htttts://///tttttttettttt 20399.
Lv, Z., Si, C., Song, J., Yang, Z. Qiao, Y., Liu, Z., and Wong, K.-Y. K. Fastercache: freee video diffu- sion model acceleration with high quality. arXiv preprint arXiv:24100 10. 19355, 2024.
Wang. S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768. 2020.
Ma, X., Wang, Y., Jia, G., Chen, X., Liu, Z., Li, Y.-F,, Chen, C., and Qiao, Y. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03048 2024.
Wang. Y., Chen, X., Ma, X., Zhou, S., Huang. Z., Wang, Y., Yang, C., He, Y., Yu, J., Yang, P., al. Lavie: High- quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309. 1033 2023.
Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.-Y., and Ermon, S. Sdedit: Guided image synthesis and edit- ing with stochastic differential equations, 2022. URL attps://arxxt org/ abs 2108.1001.0 01073.
Xi, H., Yang, S., Zhao, Y., Xu, C., Li, M., Li, x., Lin, Y., Cai, H., Zhang, J., Li, D., et al. Sparse videogen: Acceler- ating video diffusion transformers with ppatial-temporal sparsity. arXiv preprint arXiv:2502.01771 2025.
OpenAI. Sora, 2024. URL https: http///pp/aaccc//e//// com/ index/sora/. Accessed: [2024].

---

Fast Video Generation with Sliding Tile Attention
Xie, Q. Liao, Z.,, Deng, Z., Tang, S., Lu, H., al. Mlem: Multistep consistency distillation of latent dif- fusion model. arXiv preprint 74Xi:22406005768, 2024.
Yang, S., Wang, B., Shen, Y., Panda, R., and Kim, Y. Gated linear attention transformers with hardware-efffiient train- ing. arXiv preprint X11:2441.00000,, 2024a.
Yang. Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu, J., Yang, Y., Hong, W., Zhang, X., Feng, G., et al. Cogvideox: Text-to-video diffusion models with expert transformer. arXiv preprint wrXiv:2488.06072 2024b.
Yin, T., Gharbi, M., Zhang, R., Shechtman, E., Durand, F., Freeman, W. T., and Park, T. One-step diffusion with distribution matching distillation. arXiv preprint arXiv:2311.18828 2023.
Yin, T., Gharbi, M., Zhang, R., Shechtman, E., Durand, F., Freeman, W. T., and Park, T. One-step diffusion with distribution matching distillation. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6613-6623, 2024.
Zhang. J.. Wei, J., Huang, H., Zhang, P., Zhu, J., and Chen, Sageattention: Accurate 8-bit attention for plug-and-ple inference acceleration. arXiv preprint 2100::2410002367, 2024.
Zhang. J., Huang, H., Zhang, P., Wei, J., Zhu, J., and Chen, Sageattention2: Efficient attention with thorough outlier smoothing and per-thread int4 quantization. In Interna- tional Conference Machine Learning (ICML), 2025a.
Zhang. J., Wei, J., Zhang, P., Xu, X., Huang, H., Wang, H., Jiang, K., Zhu, J., and Chen, Sageattention3: Microscal- ing fp4 attention for inference and exploration of 8-bit training, arXiv preprint arXiv:2505. 1594, 2025b.
Zhang, J., Xiang, C., Huang, H., Wei, J., Xi, H., Zhu, J., and
Chen, J. Spargeattn: Accurate sparse attention accelerat- ing any model inference. In International Conference Machine Learning (ICML), 2025c.
Zhao, X., Jin, X., Wang, K., and You, Real-time video generation with pyramid attention broadcast. arXiv preprint ar 4008 12588, 2024.
Zheng, Z., Peng. X., Yang. T., Shen, C., S., Liu, H., Zhou, Y., Li, T., and You, Y. Open-sora: Democratizing efficient video production for all, March 2024. URL https: //github. www.cooa.iiceeeeeee won//pcaaitecc/ppeeeeeee Sora.
12

---

Fast Vidco Generation with Sliding Tile Attention
A. Further Details of SLIDING TILE
Algorithm 3 Mask Definition of 3D STA
ATTENTION
Input: Query coordinates (gt. 9h: gh. Iw key coordinates
(ks, kn, kw), video size (Ia, Lay Jow), kernel size
Mask of 3D NATTEN Algorithm 2 defines the attention mask in 3D NATTEN. First, it computes the window cen- ter for each query token. If the query is near the video edges, the center shifts inward to stay within bounds. Next, determines the query's 's attention window within the spa- tiotemporal neighborhood. Finally, the mask constructed by enforeing spatiotemporal constraints on query-key dis- tances.
(We, W,, Ww,), tile size (T,, Tha Tw)
Compute QK coordinates in:
It,üle + qt/ /Ti
Qh, + 9h/ /Th
qu t qu TTw
Ke,alle t ke//T,
ti kin/ /in//Tx.
Re cule kw/ /Tw
Compute window size in tiles:
Wi.tile + W
Algorithm 2 Mask Definition of 3D NATTEN
WA Wn//Th
Input: Query coordinates (ge: gh: Aw), Key coordinates
W Ww,tile W Ww//TT. TTw
(Ki, kig kw), Video size (I,, Lis Lw), Window size
Compute window center:
(We, Wi. Ww)
9et f max min qt,üle (L.//Ti 1)
We.
Wiitee
Compute window center:
Get + max (min (gt, Le- - 1
Wr
We
9ch + max
min qh,tile: (Ln/ /Th 1)
Wh,tk 2
Wh.sle 2
2
qoh max (min [h, Li
W,
W4
2
9ow f max
min qwitilo (Li (TT 1)
W ale 2
W 2
gew 4 max (min (th. Lx-
W
W
Compute masks:
2
Compute masks:
consttontt +- 1get kt,tilel <
Wesdte
<- 19ct - kiel S
We
hori constraint + 1ach kn,itee
W lillle
2
hori.constraint < 19ch kipl < WA
2
vert constraint +-
Igow kine
vert.constraint <
19o - kwl 5 2
return tome.ccnstraint www.ootrainthooroooootttrteeeeee conosstrainn/vertcooossttiiii cont.cont
2
return time constraint A www..ooonstriitt....ooo AererraintAveett.onnsraaantt constraint
B. Finetuning Details
Mask of 3D STA Algorithm 3 defines the mask for STA, introducing a tile-based coordinate framework that differs from 3D NATTEN. First, query and key coordinates are mapped tile coordinates, where each QK pair is assigned tile ID, with queries and keys in the same tile sharing the same ID. STA also computes the window center within tile coordinates, ensuring queries remain within valid bounds. Finally, neighboring keys are selected based on their tile distance from the query's 's window center.
We train on 2,000 synthetically generated videos from Hun- yuan at a resolution of 1280x768 with 117 frames. The prompts are sourced from the Mixkit dataset (Lin et al., 2024). To reduce memory usage and accelerate training, we precompute VAR-encoded latents and text encoder states. Training involves fine-tuning for 1,600 steps with a batch size of 2 and a learning rate of 2e-5. We optimize using the loss function from Eq. (5) with coefficients a = 1, B = 0.5, and 7 = 0.5. To prevent overfitting on a single guidance scale, we alternate between guidance scales of 1 and at odd and even steps. The entire process runs 8 H100 GPUs with FSDP and context parallelism for training (8 hours) and sequence parallelism for inference.
Tiling in STA Figure& 8 illustrates STA's token tiling and or- dering mechanism in 2D scenario, which extends naturally to 3D. Unlike conventional approaches that flatten 2D/3D data into ID sequences using zigzag pattern, STA orga- nizes tokens into tiles, ensuring that tokens within a tile maintain neighboring sequence IDs. This ordering strat- egy preserves locality, so when a tile attends to another tile, the resulting attention map forms a dense block, as all participating sequence IDs remain consecutive.
C. Further Details of Baselines
Swin Transformer (Liu et al.. 2021b) introduces hierarchi- cal vision transformer with a shifted window-based attention mechanism. Instead of computing self-attention globally, it partitions the image into non-overlapping windows and ap- plies attention locally, improving computational efficiency. A key innovation is the alternating window partitioning strat- egy: one layer uses standard window partitioning, while the next shifts the windows to enable cross-window connections
Visialization of 2D SWA In Figure 9, we illustrate how query tokens attend to its window key tokens. In 2D-SWA, the window slides token by token. For each window, SWA calculates the attention between the center q with all keys within the window.
13

---

Fast Video Generation with Sliding Tile Attention
18
12
19
20
Table Training-fri performance Wan 2.1
10
12
13
14
15
16
17
18
4
15
22
24
19
20
21
23
24
25
26
27
7
15
18
25
26
27
Model
SSIM f
PSNR 1
Latency
Speedup
20
30
35
35
56
28
30
16
STA
85.81
24.42
730s
1.60x
56
80
S1
62
63
SS
56
64
65
73
74
75
00
58
08
A-DiT implementation is unavailable, we reimplemented its method based the paper to accelerate video generation. Given speedup budget, vary N₂ NN, and to pick the best hyperparameters, ensuring a fair evaluation of its effee- tiveness. For the 50-step 1.36x speedup, we set N 24,
73
75
76
77
78
79
80
S1
61
62
63
70
72
79
80
S1
Figure Left: Conventional zigzag flattening strategy. Right:
STA' sequence flattening strategy. The plot given assuming a (9, 9) image with (3, 3) tile size.
3, and = 24. For 1.8x speedup, we set No 28, N = 6, and = 24.
D. Results on Wan 2.1
Following Table 3, we evaluate the effectiveness of STA Wan 2.1. Despite being applied to different model, STA achieves comparable performance across key evalua- tion metrics while preserving the same sparsity level. Wan 2.1 is evaluated on videos with the same resolution but a shorter length of 69 frames, resulting in a reduced end-to- end speedup.
E. Results on Image Super-Resolution
Table 6. Image superresolution results with FLUX (Black-Forest, 2023) 1000 captions randomly sampled from COCO-2014 (Lin ct al., 2015) validation datasct.
Figure 2D Sliding Window Attention visualization.
Methods
SSIM
PSNR
Sparsity
Latency
Speedup
and better information exchange. Swin attention is typically used in a train-from-ssratch setting. A limitation of this approach is that it disrupts local connectivity within single attention layer. Tokens in adjacent regions may not attend to each other if they fall into separate windows. In this paper, we apply Swin attention to Hunyuan Video and shift the window every other layer accordingly.
IK +2K
CLEAR 1=16
0.9291
28.1142
96.12%
138
1.54x
CLEAR r=32
0.9443
29.6722
85.94
15s
1.33x
STA w=(88.72)
0,9357
29.1086
81.25%
14s
1.43x
2K-4K
CLEAR
0,9394
29,0463
98.98%
67s
2.90x
CLEAR ==32
0,9455
30.0742
96.08%
92s
2.11x
STA w=488,7
0,9470
30.1939
95.31
57%
3.40x
CLEAR (Liu al., 2024) achieves linear attention by re- placing the original full attention with circular window- based attention mechanism where each query token only attends to key-value tokens within radius maintaining the same scaled dot-product attention formula but restricting its computation to local windows. The authors implement CLEAR with FlexAttention.
F. More Experiment Results
F.1. Kernel Performance
We additionally benchmark various sparse attention kernels at sparsity level of around 56% and present the results in Table 7. With lower sparsity, sparse kernels generally have a higher MFU, but the findings in Table 2 remain unchanged.
A-DIT (Chen et al., 2024) optimizes inference speed by caching feature offsets instead full feature maps. em- ploys staged caching strategy: residuals from later DiT blocks are stored for early-step sampling, while residuals from earlier blocks are cached for later steps. The key pa- rameters in A-DiT include the residual cache interval N, the number of cached blocks Ne, and the timestep boundary which determines the cache position, Since the official
F.2. Detailed VBench Results
In Tables 8 and 9, we present detailed comparisons of VBench scores across key dimensions originally summa- rized Table 4. Our analysis reveals that STA surpasses swin attention in video quality metries such as Imaging Qual-
14

---

Fast Video Generation with Sliding Tile Attention
Table Speedup with sparse attention kernels H100.
Methods
Implementation
Config
Sparsity
TFLOPS
Latency(ms)
MFU
Kernel Efficiency
Speedup
FA
ThunderKittens
0.00%
164.03
265.28
62.49%
100.00%
1.00x
FA 3
CUDA
0.00%
164.03
256.59
64.61% %
103.39%
1.03x
CLEAR
RexAAtention
r=32
56.23%
71.80
675.05
10.75%
17.20%
0.39x
NATTEN
HexxAttention
w=(30,41,411
56.22%
71.81
804.62
9.02%
14.43%
0.33x
Tiled NATTEN
CUDA
111) 57.68%
69.41
173.57
4.04%
6.47%
0.15x
Tiled NATTEN
FlexAttention
1=130,441,41
56.22%
71.81
409.89
17.70%
28.33%
0.65x
Swin
FlexAttention
wx4(8,64,64)
55.81%
72.49
127.51
57.46%
91.95%
2.08x
STA
FlexAttention
w=:30440,40
58.33%
68.35
174.17
39.66%
63.46%
1.52x
STA
ThunderKittens
W (30,400440
58.33%
68.35
111.73
61.82%
98.93%
2.37x
ity and Multiple Objects, while achieving comparable or superior scores CLEAR and Tiled NATTEN, For training- free models, observe systematic degradation in quality- related metrics (e.g., temporal flickering, motion smooth- ness) as sparsity increases in the STA attention mechanism. Conversely, semantic-aligner dimensions- including Ap- pearance Style, Color, and Spatial Relationships- improve under higher sparsity regimes, phenomenon driven by the text embeddings" amplified role in attention computation when spatial-tempont attention is sparsified. Furthermore, the trained STA demonstrates significant gains in video qual- ity metrics over its untrained counterpart, while maintaining semantic coherence at comparable levels which underscores the efficacy of training in refining low-level visual fidelity without compromising text-vise video alignment.
G. Qualitative Examples
We show qualitatively show videos generated by the original Hunyuan Video, STA, and A-DiT in Figure 10 and Figure 11. While fine-tuning introduces minor shifts in the output distribution of STA-1-2.43x, the model still preserves high video generation quality. Videos generated by A-DiT are generally less sharp than those generated by the original Hunyuan Video and STA. More demos are available at htts://fast-video..ii github..o....b........
15

---

Fast Video Generation with Sliding Tile Attention
Table Model Performance Comparison Part 1
Appearance Style 18.43%
Subject Consistency
Background
Temporal Flickering
Motion
Dynamic
Aesthetic Quality
Imaging Quality
Overall Consistency
Model
Consistency
Smoothness
Degree
FA3
94.22%
96.74%
99.21%
99.15%
75.00%
64.63%
67.97%
25.96%
w.o training
CLEAR
18.73%
93.63%
96.51%
98.99%
99.01%
68.06%
63.75%
68.35%
26.23%
Tiled NATTEN
18.79%
94.59%
96.61%
98.75%
98 85%
70.83%
63.79%
68. 16%
26.53%
Swin (=8.6664,64)
20.85%
91.74%
95.48%
98.67%
97.77%
77.78%
51.01%
62.22%
25.27%
Swin ==330,40,44
20.62%
90.33%
93.09%
98.78%
96.53%
75.00%
48.10%
61.89%
25.62%
STA w= (30,440,40)
18.79%
94.75%
96.50%
98.82%
98.83%
69.44%
64.18%
68.39%
26.47%
STA (18,24,224)
21.25%
89.66%
91.64%
98.46%
97.27%
83.33%
59.75%
64.23%
26.61 %
training
Swin (30,440,40
20.07%
89.78%
94.93%
98.86%
96.64 96.64%
70.83%
44.91%
55.99%
26.00%
STA (30.24.40
18.90%
94.90%
97.60%
99.68%
99.23%
73.61%
63.77%
66.21%
26.58%
STA (18,2,,244
18.90%
94.64%
96.76%
99.22%
99.11%
69.44%
64.52%
66.67%
26.09%
Table Model Performance Comparison Part 2
Object Classification
Multiple Objects
Human
Spatial Relationship
Quality
Semantic
Final
Model
Color
Scene
Action
Score
Score
Score
FA3
85.76%
70.12%
90.00%
88.66%
71.28%
35.25%
85.34%
72.17%
82.71%
training
CLEAR
88. 13%%
77.97%
88.00%
91.10%
77.49%
32.85%
84.41%
74.20%
82.37%
Tiled NATTEN
83.54%
72.18%
94.00%
92.28%
81.21%
37.94%
84.61%
75.00%
82.69%
Swin (44...64,64)
78.16%
58.54%
87.00%
93.68%
77.45%
37.79%
80.91%
71.35%
79.00%
Swin (30,440,40
79. 79.19%
60.44%
88.000 28.00%
93.68%
77.24%
35.54%
78.84%
72.28%
77.53%
STA w=330,40,40)
80.54%
71.19%
93.00%
89.81%
79.25%
36.77%
84.63%
73.83%
82.47%
STA w=118244,24)
88. 13%
75.46%
91.00%
91.61%
82.52%
42.15%
81 11.47%
77.03%
80.58%
w. training
Swin w= (30,40,40)
77.14%
48.86%
73.00%
87.00%
63.38%
39.03%
77.50%
67.39%
75.48%
STA w= -= 30,24,40)
91.77%
68.45%
86.00%
89.59%
72.76%
39.53%
85.37%
73.52%
83.00%
STA w= w=w:==11200000 (18,24.24)
92.96%
74.16%
93.00%
84.50%
73.41%
38.23%
84.76%
74.05%
82.62%
16

---

Fast Video Generation with Sliding Tile Attention
HunyuanVideo 15 mins 45
STA-tf -1-8--1189x mins 21
STA-t-2.43x 6 mins 29
A-DiT-1.36x 11 mins 34 s
Prompt: An astronaut walking between stone buildings.
HunyuanVide mins 45
STA-E S1A-8-1189% 8 mins 21 s
STA-I-2.43x
6 mins 29 s
A-DIT-1.36x 11 mins 34
Prompt: mysterious ancient temple hidden in the jungle.
Figure Qualitative comparisons. While fine-tuning introduces minor shifts in the output distribution of STA-t-2.43x, the model still preserves high vidco generation quality. Vidcos generated by D-DiT are generally less sharp than those generated by the original Hunyuan Video and STA.
17

---

Fast Video Generation with Sliding Tile Attention
HunyuanVideo
15 mins 45 s
STA-tf-1.889
8 mins 21
STA-1-2.43x
66 mins 29
A-DIT-1.36x
11 mins 34
Prompt: A tranquil island retreat features swaying palm trees and hammocks strung between them, inviting guests relax and enjoy the serene beauty of the surroundings.
HunyuanVides mins s
STA- 1.889x mins
STA-t-2.43x mins
A-D1T-1.36x
mins 34
Prompt: People gather on peaceful beach at sunset, a bonfire crackling as they sit around, enjoying the warmth and the sight of the sun dipping below the horizon.
Figure 11. Qualitative comparisons. While fine-tuning introduces minor shifts in the output distribution of STA-t-2.43x. the model still preserves high video generation quality. Videos generated by A-DiT are generally less sharp than those generated by the original Hunyuan Video and STA.
18


Reviews:
[{"review_id": "surAWAo8m9", "reviewer": "Reviewer_3sP9", "summary": "This paper introduces Sliding Tile Attention (STA), a novel attention mechanism designed to accelerate video generation using Diffusion Transformers (DiTs). The key idea is to leverage the observation that attention scores in pretrained video diffusion models are predominantly concentrated within localized 3D windows, thus eliminating redundancy from full attention. STA operates tile-by-tile with a hardware-aware sliding window design, preserving expressiveness while being hardware-efficient. The paper claims that STA achieves significant speedups (1.36–3.53× end-to-end) with minimal or no quality loss compared to existing methods like FlashAttention-3.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate Reviewer 3sP9's insightful question regarding the constraints on input resolutions and aspect ratios for Sliding Tile Attention (STA). Below, we clarify and expand upon these points:\n\n> *I think STA should have restrictions on input resolution, length, and aspect ratio. The authors should clarify this key issue.*\n\nIndeed, STA requires input video latents to have dimensions that are integer multiples of the tile size to fully achieve the promised efficiency gains from sparsity. We briefly mentioned this constraint in line 214 of our submission. Specifically, in our current implementation, we use a tile size of (6, 8, 8), meaning the optimal latent dimensions are multiples of these numbers, i.e., (a × 6, b × 8, c × 8) where a, b, and c are positive integers.\n\n\nFor video latents whose dimensions do not strictly satisfy this requirement, we suggest two practical approaches to address this limitation:\n\n**Padding with Masking**:\n\nWhen generating a video with a dimension slightly different from multiples of the tile size, we can introduce padding tokens to match the tile-aligned dimension (which is a standard practice that is also used in FlashAttention kernel when the input sequence length is not an integer multiple of block size.) For instance, to generate a video with dimensions (29, 45, 76), we would pad the latents to size (30, 48, 80).  In practice, padding tokens are then masked during attention computation to avoid contaminating the attention. \n\n**Cropping**:\n\nAlternatively, one can generate a slightly larger latent (e.g.,(30, 48, 80)) and subsequently crop the result to the desired dimensions.\n\n\nAlthough both methods introduce computational overhead due to processing padding tokens or larger dimensions, this cost remains minimal compared to STA’s efficiency gains. To thoroughly address your question, we conducted additional experiments under multiple scenarios and resolutions. Below, we show the kernel benchmark results of applying STA to a (29, 45, 76) latent on RTX 4090 by padding to (30, 48, 80):\n| Methods   | Implementation | Config        | Sparsity | TFLOPS | Latency(ms) | MFU    | Kernel Efficiency | Speedup |\n|-----------|----------------|---------------|----------|--------|-------------|--------|-------------------|---------|\n| Full Attn | FA2            | -             | 0.00%    | 141.22 | 825.46      | 51.84% | 100.00%           | 1.00×   |\n| Full Attn | Flex Attn      | -             | 0.00%    | 141.22 | 873.78      | 48.98% | 94.47%            | 0.94×   |\n| STA       | Flex Attn      | w=(29,24,24)| 85.00%   | 21.18  | 149.77      | 42.86% | 82.68%            | 5.51×   |\n| STA       | Flex Attn      | w=(18,24,24)| 89.55%   | 14.76  | 89.76       | 49.83% | 96.12%            | 9.20×   |\n\n\nOur evaluations suggest this padding overhead is negligible compared to the substantial latency reduction achieved by STA.\nWe also present end-to-end results applying STA to Wan 2.1 at a different resolution (93 frames × 1024 × 1024) in a training-free setup, further demonstrating STA is applicable to varying input dimensions:\n | Steps | Attn  Sparsity                  | SSIM ↑ | PSNR ↑ | Latency | Speedup |\n|------------|-----------------------------------|--------|--------|---------|---------|\n| 50 steps full attn|   0.0%    |  –|   –   | 1996s    | 1.00×   |\n| 18 steps full + 32 steps sta|   50.31%    | 77.41 | 20.16  | 1322s    | 1.51×   |\n\n\nWe will explicitly clarify these practical constraints and solutions in the revised manuscript to better inform readers.\nWe hope the additional experimental results address your concern and clarify the constraints involved. If our response resolves your reservations, we would be grateful if you would consider raising your score. We also welcome any further questions or suggestions you may have.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence. The authors provide extensive experimental results, including efficiency metrics (MFU and latency), human evaluations, and automated metrics (VBench, SSIM, PSNR, CD-FVD). The evidence demonstrates that STA achieves significant speedups with minimal quality loss. The paper also includes detailed comparisons with baseline methods like CLEAR, NATTEN, and Swin, showing that STA outperforms these methods in both efficiency and quality.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the problem of accelerating video generation with DiTs. The STA mechanism is designed to address the computational overhead of 3D full attention by leveraging localized attention patterns. The evaluation criteria include both efficiency metrics  and quality metrics.The methods are well-motivated and the evaluation is comprehensive.", "theoretical_claims": "The theoretical claims in the paper are correct. The authors provide a clear formulation of the STA mechanism, including the tiling strategy and the attention mask definition. The theorems (3.1 and 3.2) are used to quantify the number of dense and mixed blocks in different attention configurations, which helps in understanding the efficiency gains of STA.", "experimental_designs_or_analyses": "The experimental designs and analyses are sound and valid. The authors conducted extensive experiments to evaluate the efficiency and quality of STA. They benchmarked the efficiency of various attention algorithms, including STA, against baseline methods using metrics like MFU and latency. Human evaluations were performed on a large set of prompts to assess the quality of generated videos.", "relation_to_broader_scientific_literature": "NAN", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "None", "questions_for_authors": "I think STA should have restrictions on input resolution, length, and aspect ratio. The authors should clarify this key issue.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UHFg7C5Mve", "reviewer": "Reviewer_C6tW", "summary": "This paper introduces sliding tile attention (STA) that operates tile-by-tile with a novel hardware-aware sliding window design, preserving expressiveness while being hardware-efficient. STA achieves 1.36–3.53× end-to-end speedup with no or minimum quality loss.\n\n## update after rebuttal\"\nThanks for the authors' response. The authors have addressed most of my concerns. My final score is 4 \"accept\".", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer C6tW for their insightful suggestions and valuable questions. Below we address your comments and strengthen our paper.\n\n> *Are the results generalizable to other models than Huanyuan-Video?*\n\n\n\n\nAs suggested, we further validated STA's generalizability beyond Huanyuan-Video. In our original manuscript, we demonstrated the effectiveness of STA on image-generation models like FLUX. Here, we additionally apply STA to Wan 2.1 in a training-free setup.\n\n\nResolution: 93X1024X1024\n\n\n| Steps  |  Sparsity                  | SSIM ↑ | PSNR ↑ | Latency | Speedup |\n|------------|-----------------------------------|--------|--------|---------|---------|\n| 50 steps full attn|   0.0%    |  –|   –   | 1996s    | 1.00×   |\n| 18 steps full + 32 steps STA|   50.31%    | 77.41 | 20.16  | 1322s    | 1.51×   |\n| 25 steps full attn|   0.0%    |  –|   –   | 998s    | 1.00×   |\n | 9 steps full + 16 steps STAl |  49.83%    | 79.3% | 22.29  | 661s    | 1.51×   |\n | 10  steps full attn|   0.0%    |  –|   –   | 402s    | 1.00×   |\n| 4 steps full + 6 steps STA |  46.61%    | 82.7% | 22.25  | 277s    | 1.45×   |\n\n\n\n\nResolution: 93X768X1280\n\n\n\n\n| Steps                    | Sparsity     | SSIM ↑ | PSNR ↑ | Latency | Speedup |\n|--------------------------|--------------|--------|--------|---------|---------|\n| 50 steps full attn       | 0.0%         | –      | –      | 1839s   | 1.00×   |\n| 18 steps full + 32 STA   | 52.71%       | 80.97 | 22.09  | 1241s   | 1.48×   |\n| 25 steps full attn       | 0.0%         | –      | –      | 920s    | 1.00×   |\n| 9 steps full + 16 STA    | 52.36%       | 79.48  | 22.3   | 621s      | 1.48×   |\n| 10 steps full attn       | 0.0%         | –      | –      | 378s    | 1.00×   |\n| 4 steps full + 6 STA     | 49.13%       | 79.49  | 22.73  | 258s    | 1.46×   |\n\n\nThe above results confirm the generalizability of STA beyond Huanyuan-Video. On Wan 2.1, STA achieves consistent speed-ups across different step settings and resolutions without compromising perceptual quality.\n\n\n\n> *Are the optimization hardware dependent?* \n\nSTA itself is not hardware dependent, and we designed our method to be easily deployable across GPU architectures:\n\n\n**A100/RTX 4090**\n\n\nUsing the FlexAttention backend, STA achieves substantial speedups with minimal overhead, as demonstrated in the table below:\n\n\nRTX 4090:\n\n\n| Methods   | Implementation | Config        | Sparsity | TFLOPS | Latency(ms) | MFU    | Kernel Efficiency | Speedup |\n|-----------|----------------|---------------|----------|--------|-------------|--------|-------------------|---------|\n| Full Attn | FA2            | -             | 0.00%    | 164.03 | 958.68      | 51.85% | 100.00%           | 1.00×   |\n| Full Attn | Flex Attn      | -             | 0.00%    | 164.03 | 984.16      | 50.51% | 97.41%            | 0.97×   |\n| STA       | Flex Attn      | wt=(18,24,24) | 91.00%   | 14.76  | 896.74      | 49.89% | 96.22%            | 10.69×  |\n\n\n\n\nA100:\n| Methods | Implementation | Config | Sparsity | TFLOPS | Latency(ms) | MFU | Kernel Efficiency | Speedup |\n|-----------|----------------|---------------|----------|--------|-------------|--------|-------------------|---------|\n| Full Attn | FA2 | - | 0.00% | 164.03 | 697.61 | 75.36% | 100.00% | 1.00× | \n| Full Attn | Flex Attn | - | 0.00% | 164.03 | 999.03 |52.63% | 69.83% | 0.70× |\n| STA | Flex Attn | wt=(18,24,24) | 91.00% | 14.76 | 89.97 | 52.59% | 69.78% | 7.75× |\n\n\n\n\n**On H100 GPUs**\n\n\nWe further leverage Tensor Memory Accelerator (TMA) capabilities to achieve even better performance. Specifically, STA implemented in TK on H100 achieves a 1.43× improvement compared to FlexAttention implementation, showcasing additional optimization potential with modern hardware.\n\n\nIn summary, STA does not inherently depend on specific GPU architectures. While advanced hardware features like TMA can further boost STA's performance, using FlexAttention alone provides straightforward and efficient speedups across other GPUs.\n\n> *Extend the current relative work section with more literatures on attention based speed-up that are more related to the paper, instead of other types of speed-up.*\n\nWe appreciate your suggestion to enhance the related work section of our manuscript. Should the paper be accepted and additional space permit, we plan to incorporate discussions on the following areas:​ 1. Efficient Vision Transformers (ViTs) with a focus on attention mechanisms such as Swin Transformer and EfficientViT. 2. Efficient attention mechanisms in large language models such as  StreamingLLM and H2O.​ 3. Attention quantization techniques such as SageAttention.​", "claims_and_evidence": "Yes, claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, proposed methods and evaluation criteria make sense for the problem", "theoretical_claims": "Yes, overall looks correct", "experimental_designs_or_analyses": "Yes, the experiments overall are sound", "relation_to_broader_scientific_literature": "Video generation model is running with very high latency. Reducing latency is a very significant topic to the area. This paper introduces some innovations along the direction.", "essential_references_not_discussed": "I don't see any specific missing pieces but it would be helpful to extend the current relative work section with more literatures on attention based speed-up that are more related to the paper, instead of other types of speed-up.", "other_strengths_and_weaknesses": "The paper overall is well written and easy to follow. Experiments are sound, contribution of the paper is clear. The topic is very important to the community. Overall it's a good paper.", "other_comments_or_suggestions": "Extend the current relative work section with more literatures on attention based speed-up that are more related to the paper, instead of other types of speed-up.\n\nIt would be great to present results on more models / hardware types to show case the generalizability of the work.", "questions_for_authors": "Are the results generalizable to other models than Huanyuan-Video? E.g., StepVideo, Wan, CogVideoX, LTX. Can you please provide some analysis on those models? \n\nAre the optimization hardware dependent? It would be good to report speed-ups on other types of GPUs.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vSVt7ijDUZ", "reviewer": "Reviewer_nmHo", "summary": "This paper introduces sliding tile attention (STA) to address prohibitive compute cost in attention calculation. The authors observed that attention scores in pretrained video diffusion models predominantly concentrate within localized 3D windows. The proposed STA can eliminate redundancy from full attention by sliding and attending over local spatial-temporal region.  STA operates tile-by-tile with a novel hardware-aware sliding window design, preserving expressiveness while being hardware-efficient. Experimental results verify the effectiveness of the proposed method in achieving diffusion acceleration.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer nmHo for their constructive feedback. Below, we address specific comments one by one.\n\n> *More training details about STA w. Training are missing.*\n\nWe have provided the primary details of STA training, including datasets, prompts, learning rate, and hardware specifications, in Appendix Section B and briefly mentioned them in line 265 of the manuscript. If there are specific additional details you would like us to include beyond these, we would be happy to provide them. We will also open-source all of our **training code, configurations, and scripts** to facilitate reproducibility. \n\n> *No demo videos are provided in the supplementary files*\n\nThanks for raise this concern. We provide an anonymous link to 20 sampled videos from our human evaluation experiments here: https://drive.google.com/drive/u/4/folders/1kRDt4ahiYQj1zk593FE6Hg2CgNsJcdIf\n\n> *For video generation, only HunyuanVideo is used to conduct experiments.*\n\nFollowing your valuable suggestion, we further validated STA on Wan2.1 in a training-free setup, as shown in the results below. Together with FLUX's result in Table 5 of our manuscript, we believe they provide broader evidence of STA's generalizability.\n\n\nResolution: 93X1024X1024\n\n\n| Steps  |  Sparsity                  | SSIM ↑ | PSNR ↑ | Latency | Speedup |\n|------------|-----------------------------------|--------|--------|---------|---------|\n| 50 steps full attn|   0.0%    |  –|   –   | 1996s    | 1.00×   |\n| 18 steps full + 32 steps STA|   50.31%    | 77.41 | 20.16  | 1322s    | 1.51×   |\n| 25 steps full attn|   0.0%    |  –|   –   | 998s    | 1.00×   |\n | 9 steps full + 16 steps STAl |  49.83%    | 79.3% | 22.29  | 661s    | 1.51×   |\n | 10  steps full attn|   0.0%    |  –|   –   | 402s    | 1.00×   |\n| 4 steps full + 6 steps STA |  46.61%    | 82.7% | 22.25  | 277s    | 1.45×   |\n\n\n\n\nResolution: 93X768X1280\n\n\n\n\n| Steps                    | Sparsity     | SSIM ↑ | PSNR ↑ | Latency | Speedup |\n|--------------------------|--------------|--------|--------|---------|---------|\n| 50 steps full attn       | 0.0%         | –      | –      | 1839s   | 1.00×   |\n| 18 steps full + 32 STA   | 52.71%       | 80.97 | 22.09  | 1241s   | 1.48×   |\n| 25 steps full attn       | 0.0%         | –      | –      | 920s    | 1.00×   |\n| 9 steps full + 16 STA    | 52.36%       | 79.48  | 22.3   | 621s      | 1.48×   |\n| 10 steps full attn       | 0.0%         | –      | –      | 378s    | 1.00×   |\n| 4 steps full + 6 STA     | 49.13%       | 79.49  | 22.73  | 258s    | 1.46×   |\n\n\nThe results confirm STA's applicability across different model architectures. \n\nWe truly appreciate your thoughtful review. If our response has addressed your concerns, we would be grateful if you might consider a higher score. If there are any remaining points we could further clarify or improve, we would be sincerely thankful for your guidance.", "claims_and_evidence": "This paper claims that \"attention scores in pretrained video diffusion models predominantly concentrate within localized 3D windows.\". The claims are supported by experimental results.", "methods_and_evaluation_criteria": "This paper introduces sliding tile attention (STA) to address prohibitive compute cost in attention calculation. Both qualitative and quantitative results verify the effectiveness of the proposed method.", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes. STA operates tile-by-tile with a novel hardware-aware sliding window design, preserving expressiveness while being hardware-efficient.", "relation_to_broader_scientific_literature": "This paper introduces sliding tile attention (STA) to address prohibitive compute cost in attention calculation. The authors observed that attention scores in pretrained video diffusion models predominantly concentrate within localized 3D windows. The proposed STA can eliminate redundancy from full attention by sliding and attending over local spatial-temporal region.  STA operates tile-by-tile with a novel hardware-aware sliding window design, preserving expressiveness while being hardware-efficient. Experimental results verify the effectiveness of the proposed method in achieving diffusion acceleration.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Pros:\n\n1.\tThe authors observed that attention scores in pretrained video diffusion models predominantly concentrate within localized 3D windows.\n2.\tA novel hardware-aware sliding window design is proposed , preserving expressiveness while being hardware-efficient\n3.\tExtensive experimental results verify the effectiveness of the proposed method.\n\nCons:\n\n1.\tMore training details about STA w. Training are missing.\n\n2.\tNo demo videos are provided in the supplementary files, which makes it hard to judge the effectiveness of the proposed method.\n\n3.\tFor video generation, only HunyuanVideo is used to conduct experiments. More other architectures such as CogVideoX and Mochi should be discussed and compored.", "other_comments_or_suggestions": "Please refer to Weaknesses for more details", "questions_for_authors": "Please refer to Weaknesses for more details", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xhqhlrj9Wa", "reviewer": "Reviewer_BHCc", "summary": "this paper addresses the problem of slow speeds in video generation. the paper proposes a method called sliding tile attention that is designed to address the challenge. the proposed method learns to do sliding and attending over local spatial and temporal region, allowing  the  reduction of redundancy in computing full attention. the resulting method is much faster than prior methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer BHCc for their supportive feedback and genuine interest in our work. \n\n> *Will the code be released to ensure reproducibility*\n\nWe appreciate your emphasis on reproducibility. We confirm that all the code, configurations, and scripts required to reproduce our experiments will be fully open-sourced. This will ensure the community can easily replicate and build upon our results.\n\n> *Will the proposed method have the potential to be applied to 3D or 4D generation?*\n\nWe appreciate your insightful question regarding STA's potential applicability beyond standard video generation. Videos are already a type of 3D data (time × height × width). We believe STA naturally generalizes to other forms of 3D data such as 3D object generation (e.g., voxel grids) or even 4D data generation (e.g., dynamic point clouds or time-evolving 3D structures), so long as those data conform to some degree of data locality.", "claims_and_evidence": "the claims are supported by experimental evaluations.", "methods_and_evaluation_criteria": "the proposed method and evaluation metrics make sense", "theoretical_claims": "the theorems are technically sound", "experimental_designs_or_analyses": "the evaluation is thorough.", "relation_to_broader_scientific_literature": "i believe the resulting method will have a huge impact on many researchers working on diffusion models. i see the merit of the proposed method and the potential impact this paper could have", "essential_references_not_discussed": "references are adequate", "other_strengths_and_weaknesses": "i enjoyed reading this paper and i believe this will be useful for many applications. however, I would still like to request some clarifications\n\n1. will the code be released to ensure reproducibility\n\n2. will the proposed method have the potential to be applied to 3D or 4D generation? (I know 3D and 4D gen is beyond the scope of this paper, so I'm not asking for any experiments or comparisons, but I'm genuinely curious about the impact that this method could potentially have", "other_comments_or_suggestions": "n/a", "questions_for_authors": "please see the comments in #Other Strengths And Weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces Sliding Tile Attention (STA), a novel attention mechanism to accelerate video generation in Diffusion Transformers (DiTs) by leveraging localized 3D attention patterns. STA achieves significant end-to-end speedups (1.36–3.53×) with minimal or no quality loss, supported by comprehensive experiments on efficiency and quality metrics. The method is hardware-efficient, with a tile-by-tile sliding window design, and shows generalizability across models like Hunyuan and Wan 2.1.

Strengths
* significant contribution: STA addresses a critical bottleneck in video generation, reducing attention computation costs with a novel, hardware-aware approach.
* robust evaluation: extensive experiments validate efficiency (MFU, latency) and quality (SSIM, PSNR, human evaluations), with clear comparisons to baselines like FlashAttention-3.
* generalizability: additional results on Wan 2.1 and FLUX demonstrate applicability beyond the primary model, addressing reviewer concerns.
* theoretical soundness: Theorems and formulations are clear and correct, enhancing understanding of efficiency gains.

Weaknesses
* limited model diversity: while generalizability is shown, some reviewers noted the need for broader testing across architectures like CogVideoX or Mochi.
* supplementary material: Lack of demo videos or initial supplementary material was a concern, though addressed in rebuttals with a provided link.
* training details: some reviewers requested more specifics on STA training, which authors partially addressed by referencing the appendix and promising open-source code.
* input constraints: STA’s reliance on tile-aligned dimensions requires padding or cropping, which introduces minor overhead, though mitigated in practice.

Overall, the paper presents a significant, well-evaluated contribution to video generation efficiency with broad applicability. The rebuttal effectively resolves reviewer concerns, and the promise of open-source code ensures reproducibility.
Decision: Accept (poster)