Sparse VideoGen2: Accelerate Video Generation with  Sparse Attention via Semantic-Aware Permutation
Authors: Shuo Yang
Haocheng Xi
Yilong Zhao
Muyang Li
Jintao Zhang
Han Cai
Yujun Lin
Xiuyu Li
Chenfeng Xu
Kelly Peng
Jianfei Chen
Song Han
Kurt Keutzer
Ion Stoica
Venue: NeurIPS 2025 spotlight ()
Keywords: video diffusion
attention acceleration
Primary area: deep_learning
TL;DR: We propose a method to speedup video diffusion generation through efficient attention.

Abstract: Diffusion Transformers (DiTs) are essential for video generation but suffer from significant latency due to the quadratic complexity of attention. 
By computing only critical tokens, sparse attention reduces computational costs and offers a promising acceleration approach. 
However, we identify that existing methods fail to approach optimal generation quality under the same computation budget for two reasons: 
(1) Inaccurate critical token identification: current methods cluster tokens based on position rather than semantics, leading to imprecise aggregated representations. 
(2) Excessive computation waste: critical tokens are scattered among non-critical ones, leading to wasted computation on GPUs, which are optimized for processing contiguous tokens.
In this paper, we propose SVG2, a training-free framework that maximizes identification accuracy and minimizes computation waste, achieving a Pareto frontier trade-off between generation quality and efficiency. 
The core of SVG2 is semantic-aware permutation, which clusters and reorders tokens based on semantic similarity using k-means. This approach ensures both a precise cluster representation, improving identification accuracy, and a densified layout of critical tokens, enabling efficient computation without padding. 
Additionally, SVG2 integrates Top-p dynamic budget control and customized kernel implementations, achieving up to $2.30\times$ and $1.89\times$ speedup while maintaining a PSNR of up to $30$ and $26$ on HunyuanVideo and Wan 2.1, respectively. Our code is open-sourced at https://github.com/svg-project/Sparse-VideoGen.

Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation
Shuo Yang"
Haocheng Xi
Yilong Zhao
Muyang Li
Jintao Zhang Kelly Peng Ion Stoica
Han Cai
Yujun Lin
Xiuyu Li
Chenfeng Xu
Jiaffei Chen
Song Han
Kurt Keutzer
University of California, Berkeley
MIT
NVIDIA Stanford University
Hunywan 720P, Text-to-Video
Dense Attention Latency 30 min
Spurse
Lasenev
Caaurry surda
Wan 2.1 720P, Vidco
Latescy
Sporse VideoClen PSNR 25 808 Lanency la
Figure 1: SVG2 accelerates video generation while maintaining high quality. On single H1000, for Hunyuan- Video and Wan 2.1. SVG2 achieves up to 2.30 and end-to-end speedup, with PSNR up and
Abstract
Diffusion Transformers (DiTs) are essential for video generation but suffer from significant latency due the quadratic complexity of attention. By computing only critical tokens, sparse attention reduces computational costs and offers promising acceleration approach. However, we identify that existing methods fail to approach optimal generation quality under the same computation budget for two reasons: (1) Inaccurate critical token identification: current methods cluster tokens based on position rather than semantics, leading to imprecise aggregated representations. (2) Excessive computation waste: critical tokens are scattered among non-critical ones, leading to wasted computation on GPUs, which are optimized for processing contiguous tokens. In this paper, we propose SVG2, training-free framework that maximizes identification accuracy and minimizes computation waste. achieving
Pareto frontier trade-off between generation quality and efficiency. The core
"Equal contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

SVG2 is semantic-awan permutation, which clusters and reorders tokens based on semantic similarity using k-means. This approach ensures both precise cluster representation, improving identification accuracy, and a densified layout of critical tokens, enabling efficient computation without padding. Additionally, SVG2 integrates top-p dynamic budget control and customized kernel implementations, achieving up to 2.30 and 1.89x speedup while maintaining a PSNR of up to 30 and 26 on Hunyuan Video and Wan 2.1, respectively. Our code is open-sourced at https://gttub..com/sgggg prop://ssssooooesppsssseee VideoGen.
Introduction
Diffusion Transformers (DiTs) have demonstrated significant efficacy in generative tasks, particularly excelling in generating high-quality images and videos [1, 2, 3. However, the computa- tional efficiency of DiT's remains a major bottleneck, primarily
Ourr
SVG
SpargeAtte
29
26
Raduriaa
23
due to the quadratic computational complexity introduced by 3D spatio-temporal attention mechanisms [4]. For instance, generating just a five-second video using Hunyuan Video on an NVIDIA A100 GPU takes nearly an hour, where the 3D attention accounts for more than 80% of end-to-end runtime. This inefficiency severely limits the practical deployment of DiT-based generative models.
17
Deusity
Figure 2: Trade-off curves between gen- eration quality (PSNR) and efficiency (density). SVG2 consistently surpasses existing methods given the same den- sity, achieving a Pareto frontier.
To mitigate the quadratic computational complexity, previous studies have observed that self-attention mechanisms are nat- urally sparse, where only small portion of computations significantly influence the final out- put [5, 6, 7]. Therefore, the computational costs can be dramatically reduced (up to 8x) with negligible degradation in generation quality by only computing the critical tokens [4, 8). To ef- fectively identify these critical tokens, existing approaches introduce an identification step where activations from each token are used to estimate attention scores [9. 10]. Tokens with the highest scores are then selected as critical and processed by the following sparse attention. To minimize overhead, the identification typically processed the block granularity, treating consecutive tokens as an aggregated token, which are selected or ignored as whole [11, 12].
However, observe that given the same computational budget (i.e., the number of selected critical tokens), existing sparse attention methods significantly fall behind the oracle generation quality, where the critical tokens are selected assuming the attention scores are known in advance rather than estimated ($ 3.2). We identify that this performance gap arises from two primary challenges:
1. Inaccurate identification: existing block-wise identification methods are ineffective in precisely identifying critical tokens. Because tokens are clustered into blocks based on positions rather than semantic similarities, tokens within the same block may have dramatically different activations in the latent space. Consequently, the aggregated activations become less representative [11], leading to inaccurate estimations of attention scores and thus incorrect identification of critical tokens. E.g., widely used techniques such as mean pooling [9] and max pooling [5] are prone to inaccuracies, particularly when applied to distinct tokens.
2. Computation waste: existing methods cause computation waste even if critical tokens could be perfectly identified. This is because of the mismatch between sparse computation and hardware specifications [13]. For instance, tensor cores on NVIDIA GPUs require a minimum matrix multiplication shape of 16 16 8 [14), which necessitates batch size of 16 tokens. Thus, even only a subset of a block of 16 tokens are critical, the entire block must still be computed to utilize tensor cores, causing computation waste. Our empirical evaluations show that up to 80% of computation can be wasted on non-critical tokens ($ 3.2).
To bridge this gap, we propose SVG2, training-free free sparse attention approach specifically designed to accelerate video generation for DiT-based models, achieving a Pareto frontier trade-off between generation quality and computational efficiency (as shown in 5.4). Our key insight is to leverage semantic-aware permutation to maximize the accuracy of critical token identification and minimize the computation waste of sparse computation. Specifically, semantic-awer permutation clusters

---

tokens into blocks according to the semantics of activations rather than positions, Consequently, tokens within each block exhibit closely aligned activations, ensuring more accurate aggregated representations and thereby significantly improving identification accuracy. Additionally, semantic- aware permutation densifies sparse computations by consolidating scattered critical tokens into compact, dense blocks. Due to their semantic similarity, tokens in a single block tend to be either all critical or all non-critical. This property ensures that computation is not wasted on blocks containing mix of critical and non-critical tokens, thus improving computational efficiency.
To integrate semantic-awan aware permutation into an end-to-end framework, SVG2 introduces three key techniques. First, to implement eemantic-warr permutation, SVG2 applies k-means clustering on the Query, Key, and Value vectors of each head and layer before the identification step. The resulting clusters are then permuted so that tokens within the same cluster are grouped together, ensuring semantically coherent blocks. Second, to enable dynamic allocation of the computational budget, SVG2 adopts a Top-p critical token selection strategy inspired by Tactic and Twilight [11, 15]. Specifically, SVG2 uses the centroids of clusters to approximate attention scores for each cluster, selecting tokens with the highest estimated scores until their cumulative sum reaches p. This approach enables dynamic budget allocation without manual adjustments. Third, to support dynamic block sizes for sparse attention, SVG2 introduces customized kernel implementation. This is essential because the clusters formed by semantic-awee permutation naturally vary in size, and existing block sparse attention kernels, which are limited to fixed block sizes, cannot efficiently handle such variability.
We prototype SVG2 based on an open-sourced video generation framework [4] and customize kernels with FlashInfer [16]. We evaluate SVG2's ' quality and efficiency on representative video generative models including Hunyuan Video [1] and Wan 2.1 [2]. Results demonstrate that SVG2 consistently achieves a Pareto frontier, delivering superior generation quality at any given computational budget. Specifically, SVG2 delivers significant efficiency improvements, achieving end-to-end speedup of up to x and 1.84x speedup while maintaining high visual quality with a PSNR of up to 30 and 26 on Hunyuan Video and Wan2. outperforming all prior methods.
2 Related Work
Sparse Attention for Video DiT's Sparse attention mechanisms for accelerating DiTs fall into two categories: static and dynamic, depending on whether to select critical tokens dynamically during runtime or statically offline. Static methods [8, 4] predefine sparse patterns offline, such as identifying recent tokens as critical [4]. These methods lack adaptability to diverse sparsity patterns, leading to suboptimal performance. Dynamic methods [9, 10, 17, 18, 19, 20, 21, 22] determine sparse patterns at runtime, selecting critical tokens through an additional identification step. However, existing dynamic methods fail to achieve both high identification accuracy and low computation waste. In contrast, SVG2 consistently achieves superior performance under the same computation budget.
Sparse Attention for Large Language Models (LLMs). Sparse attention for LLMs falls into two categories: memory-efficien and compute- -efficiieef.. Memory-efficiennt methods [6. 5, 7, 23, 24] reduce memory load to accelerate decoding but are ineffective for compute-bound DiT-based video generation. compute-efficieen methods [25, 26, 27, 12, 28, 29] focus on processing only critical tokens, yet cannot directly optimize video DìTs due to unique sparse patterns of video data. Notably, MMInference [12] introduces a modality- permutation for multi-modal LLMs. This permutation rase-based, designed to permute inter-modality tokens.
Linear Attention for Diffusion Models I. Linear attention [30, 31, 32] and state space models [33, 34] have gained attention in video generation, where the long-context problem makes attention module dominate the latency. Matten [35] uses the mamba block to capture global information and uses attention for local information. LinGen [36] adopts a combination of Mamba2 and Swin attention block for 1-minute video generation. M4V [37] proposes an MM-DiM block to overcome the adjacency of mamba in capturing complex spatial-temporal dynamics. Block- Block-wise SSM scanning scheme [38] achieves long-term memory ability in video generation. TTT [39] proposes to use RNNs and designs a new TTT block for minute-long video generation. SANA [40, 41, 42] and DC-AE [43, 44, 45] reduce the generation overhead by using Linear Attention and Deep-Compressed Auto-Encoders This line of research reduces the complexity of the attention from quadratic to linear, making it efficient in long video generation.
3

---

Long Video Generation and Caching- Acceleration Minute-level long video generation poses new challenges in the field of video generation in both quality and efficiency. Caus [46] and Self-Forcing [47] distill a bidirectional diffusion model to autoregressive one to leverage the efficiency of KV cache. LongLive [48] achieves multi-prompt minute-level video generation with KV cache refreshing. Framepack [49] compressed the token number within frames to reduce the sequence length. RifleX [50] and Freelong [51] extend the video generation length by context extrapolation in training- free manner. RadialAttention [52]. VMOBA [53], Mixture-of-Conter [54], and VSA [55] adopt sparse attention in long-context fine-tuning to reduce the training overhead. Caching-bai methods [56, 57, 58, 59, 60] optimized the efficiency by utilizing the redundancy between timesteps and classifier free guidance (CFG). These lines of research is orthogonal to our sparse attention
method, and can be integrated to achieve higher speedup.
3 Motivation
3.1 Attention in DiTs is Inherently Sparse
Attention operation in DiTs costly. During each denoising step, DiTs transform the input activations with hidden dimension d into Query (Q), Key (K), and Value (V) tensors, followed by self-attention operation produce the final output O [61]]
QKT
dim fiim=-1] -1)
0 O=PxV, softmax
where the attention score captures the relationship among tokens However, computing P has a quadratic complexity relative to the sequence length. State-of-the-arr DiTs typically process thousands of tokens per frame across multiple frames, creating a significant performance bottleneck. E.g., generating 33-frame video using Hunyuan Video-T2V-13BB requires over 80% of the total end-to-end time to be spent on the attention alone [4, 8].
Attention operation in DiT's is highly sparse. Fortunately, attention is inherently sparse, where only a small subset of computations significantly contributes the final output. This sparsity arises from the characteristics of the softmax function, where a few largest values in Q x K dominate the attention score P. which in turn dictates the final weighted output P
[6, 11].
To quantitatively assess this sparsity, we collect attention maps from Wan2.1-12V- 14B video generation and visu- alize the average recall of attention scores under varying computational budgets, defined by the number of critical K tokens. Specifically, the critical tokens are selected following an oracle policy, where tokens are ranked in descending order based on their attention scores. This approach illustrates the upper bound of achievable recall under a constrained computational budget.
Oracle Policy
SVG
Sparge
Alteniion Recall
0.9
0.8
0.7
0.6
0.5
12.5
25
37.5
50
Density
As depicted in Figure 3, the attention is highly sparse, where only 13% of the computations (i.c., the percentage of attention map retained in the sparse attention) are suffi- cient to achieve an attention recall of 95%, maintaining a neas-lossless PSNR of 27 while providing up to 2x theo- retical end-to-end speedup. This observation highlights an opportunity to leverage the trade-off between generation quality and computational efficiency.
Figure 3: Comparison of attention recall ver- sus density (i.c., number of sparse compu- tation normalized by total computation) for the oracle policy, SVG, and SpargeAttention. Notably. the significant gap between the or- acle policy and existing methods highlights the potential for improvement.
3.2 Existing Sparse Attention Fails Match the Oracle Policy
Despite the potential of sparsity in reducing computational cost, directly adopting the oracle policy is impractical. This because identifying critical tokens requires calculating the full attention scores P by Q x K, thus providing no actual speedup. To achieve practical efficiency, state-of-the-ar approaches [9, 10] implement a coarse-grained identification strategy. Specifically, they cluster consecutive tokens into large blocks and caleulate attention scores at the block level, providing an

---

Atlention 5%
Recalli
Compuic Budget: 68%
Compute Budget 92%
Compute Budget 28%
Effective Compute: 27.9% X
Effective Compute: 26.4% X
Fffective Compute: 86 6%
Compute
Block
Critical
Token
(a) OriginalAteation Map (b) Inaecurate Identification
(c) Computation Waste
(d)
Semantic-neare
Pemmutation
Figure 4: Illustration of how existing methods fall short due to the inaccurate identification and computation waste, assuming computation unit of 4 x block. (a) Original attention map of demonstration example. (b) Position-based clustering groups distinct tokens within the clustering, causing the imprecise representation mean-pont mean-pooling or max-pooling. Therefore, blocks with smaller number critical tokens are ignored, causing lower recall of attention scores. (c) Due to the scattered layout of critical tokens, even if achieving high attention recall, each compute block processes both critical and non -ritiiai tokens, thus causing computation waste and decreasing effective compute critical tokens. (d) Semantic-ave aware permutation clusters and reorders similar tokens into contiguous layout, thus achieving high attention recall while minimizing computation waste.
approximation of the original P. This approach significantly reduces the identification overhead, with less than 1% computation compared to the full attention when using a block size of 128.
However, existing coarse paars-graiine approaches reduce identification accuracy and lead to computation waste. As illustrated Figure 3, existing sparse attention mechanisms consistently fall significantly short of achieving the attention recall of the oracle policy, regardless of the computation budget. This performance gap arises primarily from two key factors:
Position-based clustering leads to inaccurate identification. Existing methods reduce identification overhead by clustering consecutive tokens into blocks. For instance. SpargeAttention [9] groups every 128 query tokens and 64 key tokens, using mean pooling to create a single representation for each block, which then used to approximate P. However, this position-basee baated clustering does not guarantee semantic similarity among tokens. Tokens within the same block can exhibit vastly different activations in the latent space. For example, two physically close objects in a video frame, such as an apple and a cake, may have no semantic relationship. This variability within block degrades the quality of the aggregated block-wise representation, leading to reduced identification accuracy. We illustrate this issue in Figure 4 and provide quantitative analysis of the identification accuracy in $ 5.5. To address this problem, we propose using semantic-aware clustering instead of dosition-baees clustering, detailed in $ 4.1.
Scattered critical tokens cause computation waste. Even if all critical tokens are perfectly identified, existing sparse attention mechanisms cannot achieve the theoretical computational savings promised by the oracle policy due to mismatch between scattered sparse computations and hardware specifications. Since the criticality of tokens is determined by semantics, critical tokens are naturally scattered across the tensor rather than being contiguous. However, modern ML accelerators, such as NVIDIA GPU tensor cores, are optimized for dense matrix multiplication, which requires contiguous input dimensions [14, 62]. As a result, scattered critical tokens must padded with non-critical tokens to maintain a contiguous layout, leading to significant computation waste. This issue is visualized in Figure 4, and we further quantify the computation waste in Š 5.5. To approach the performance of the oracle policy, an automatic permutation is required to rearrange scattered critical tokens into a dense layout to minimize computation waste, as detailed $ 4.1.
4 Methodology
In this section, we introduce SVG2, a training-fres sparse attention framework designed use semantic-wware permutation to achieve a Pareto frontier trade-off between the generation quality and computational efficiency for video DiTs. We visualize the workflow of SVG2 in Figure 5. At the core of SVG2 is semantic-aware permutation, which aims to maximize identification accuracy of critical tokens and minimize computation waste ($ 4.1.. To dynamic select and adjust the computation budget, SVG2 proposes centroid- based top-p selection, which enables practical deployment ($ 4.2).
5

---

Semantic-aware Permutation
k-mcans Clustering
othrr
Key Centroids
Query Centmis
Top-P
Selection
Aftend
<1% Overhead
Centroid-bused
Original Attention Map
(b) Top-p Selection
(c) Permuied Atiention Map
Figure 5: Overview of SVG2. (() Original attention map of demonstration example, with various colors representing various semanties. Only tokens with similar semantics attend each other, having high attention scores thus selected as critical tokens. (b) After k- k-means clustering, semantic-similat tokens (i.e.. similar colors) are grouped into the same cluster, with the query and key centroids to precisely represent the Bluster-leve semantics. These centroids are then used to approximate the attention score for accurate identification of critical tokens. (c) Combined with Top-p selection, critical tokens can be dynamically identified in contiguous layout.
Additionally, SVG2 investigates several system -allort-gg co-designs, such fast k-means and attention kernel, significantly accelerating video generation ($ 4.3).
4.1 Semantic-Aware Permutation with k-means Clustering
As discussed in $ 3.2, existing sparse attention mechanisms suffer from inaccurate identification due to the position-based clustering. To this end, SVG2 proposes using semantic similarity to cluster rather than position, by performing k-means on the activations of the input tokens.
Specifically, for each attention head and transformer layer, k-means is independently applied to query tokens (Q E I xd) and key tokens (K E IR xx),, where N, and N₂ represent the number of tokens in Q and K, creating C, query clusters Q...... Qc and C₁. key clusters KI.... Ko., This approach enables tokens within each cluster to share similar semantics, improving the precision of centroid representation for better identification, as detailed in $ 4.2.
Furthermore, to densify the sparse computation of scattered critical tokens, SVG2 performs semantic- aware permutation based on the k-means clustering. Although the semantically similar tokens are logically clustered, they physically scattered in the tensors, resulting in substantial computational waste as described in $ 3.2. address this, SVG2 permutes tokens within each cluster into contiguous layout. Such cluster-wise contiguous layout can be efficiently computed by the underlying ML accelerators, thus reducing computation waste. We detail the permutation algorithm and the mathematical equivalence for the attention output in the following formulations. Assuming my R.N,NN, and my IR X be the permutation matrices such that 7T and n I, the permuted tokens are then Q' Q, K' K,K, and V TT V. where K and V share the same permutation F to guarantee the output equivalence. The permuted attention output O' is:
O' = Attention (Q', K', V') = =my softmax (T, (TT()(TA
TBV
Vd
QKT
[QKT
-TT, Fy softmax
(T T softmax
V-O
Vd
Vd
4.2 Centroid-Based Top-p Selection
Despite the semantic-coherent clusters provided by the semantic-aware permutation, it remains impractical to deploy SVG2 without addressing two critical challenges: (1) how to effectively estimate the criticality of clusters, and (2) how to dynamically determine the number selected critical clusters (i.e., the number of critical tokens) satisfy arbitrary accuracy requirements.
Crross cross-category centroids arise from SVG2's independent clustering, which enables a many-to many-to-many ma-y map where multiple query clusters share important key clusters.

---

Accurate and efficient estimation of criticality. To estimate the criticality of each cluster, SVG2 introduces a centroid-based estimation of attention scores P. Specifically, approximates the criticality of each token by estimating its P using the centroids of its cluster, mimicing the oracle policy defined 3.1. As formulated in Equation 1, this approach calculates the pre-softmax scores S using the centroids of each cluster. These scores are then weighted by the number of tokens within the cluster (i.e.. the size of the cluster), to generate an approximate attention score P in Equation 2, providing estimation of the actual P.
[Kg] exp(St))
Sky = centroid(Qi)) c (1)
P3
KKl exp(Sit)
ddz
Since tokens within the same cluster already share similar semantics, the centroids can serve as highly accurate representations of the actual activations, ensuring the reliability of such estimation. Further- more, because the typical number of clusters (i.e., C and C2)) is less than 1024, the computational overhead for this approximation is negligible compared to the full attention calculation, typically accounting for less than 1% of the total computational cost.
Dynamic adjustment of computation budget. To dynamically adjust the number of critical tokens instead of pre-defined constant, SVG2 employs a Top-p selection strategy based on the approximated P'. SVG2 first sorts all potential clusters descending order according their corresponding P', then selects clusters sequentially until the accumulated p' reaches a predefined target.
4.3 Efficient System-Algorithm Co-design
Fast k-means with centroid cache. While k-means clustering is essential to semantic-aware permutation, its iterative process can introduce substantial latency if the number of iterations is large before convergence. For example, with the state-of oftte-of-the-rrt GPU implementation of k-means++ [63], it takes more than 100 iterations to converge, consuming 50% or even comparable time the attention computation. Fortunately, DiTs are known to be similar between consecutive denoising steps [59, 64], enabling reusing the centroids from the previous step as the fast initialization for k-means in the next step. Based on this observation, SVG2 implements centroids cache, which automatically caches and reuses centroids between consecutive steps. This technique reduces the runtime of k-means by up to 76x, evaluated in 5.3.
Efficient sparse attention kernel for varied block-sizes. While existing efficient attention imple- mentations (e.g., FlashAttention [65], FlexAttention [66], and FlashInfer [16]) support block-wise sparse computation, they only support static block size (e.g., 128 128). However, the sizes of clusters after semantic-aware permutation are naturally dynamic and diverse, causing computation waste with the static block size. For example, SVG2 could generate a query cluster with 128 tokens with key cluster with 32 tokens. Such 128 x 32 computation needs to be padded into 128 x 128 to use existing kernels, which causes 75% computation waste. To address this, SVG2 implements a customized attention kernel that accepts dynamic block-sizes as input.
Our dynamic block-sparse attention kernel supports both FA2 (A100) and FA3 (H100), combining sparse loading and dense computation. For FA3, we use wgmma (m64n64k16) for dense compute to maximize hardware efficiency. For query tokens, we load contiguous tokens from the same cluster, which are naturally contiguous in memory after permutation. For key/value tokens, which may be scattered in global memory due to varying cluster sizes, SAPAttn uses per-token address offsets to perform sparse loading and stores them in shared memory in a contiguous layout. This enables efficient use of MMA instructions without the need for expensive key/value padding, leading to over 85% of the theoretical maximum performance, where the upper bound is estimated by multiplying the sparsity density with the runtime of the dense FlashAttention-3. We evaluate the kernel efficiency in 5.3.
7

---

Original Alteniios Map
(b) Aller Permunaionn (c) Recoverol after Top-p (a) Original Attention Map
(b) Penmutation
(6) Rosoverel afte Top-p
Figure Visualization attention maps from different attention heads Wan2.1 when generating videos from VBench [67]. (a) Original attention maps with diverse sparse patterns, assuming critical tokens highlighted red. (b) Permuted attention maps. After semantic-aware permutation, critical tokens are permuted into a contiguous layout based on the k-means clustering, enabling efficient block-wise computation without waste. (c) Recovered attention maps after applying centroid-based top-p selection and undoing the permutation. The high similarity between original and recovered attention maps demonstrates the effectiveness SVG2.
Experiment
5.1
Setup
Models. We evaluate SVG2 on open-sourced state-of-the-art video generation DiT models including 1an2..-12V/T2V-444 [2], and Vinoouaniiee-222-1133 [1] to generate videos with 720p resolution. After being tokenized by 3D-VAE, Wan2. generates 21 frames with 3600 tokens per frame, while Hunyuan Video processes 33 frames with 3600 tokens per frame.
Metrics. We assess the similarity of generated video compared to full attention using the following metrics: Peak Signal-to-Noise Ratio (PSNR), Learned Perceptual Image Patch Similarity (LPIPS), and Structural Similarity Index Measure (SSIM). We VBench [67] evaluate the video quality. To quantify the efficiency of sparse attention mechanisms (i.e., computational budget), we use density, which is defined as the sparse attention computation divided by the full attention computation. To assess end-to-end efficiency, we use the total amount of computation (i.e., FLOPs) needed for generating videos.
Datasets.
For text-to-video generation, we adopt the prompt in Penguin Benchmark after prompt optimization provided by VBench team. For image-to- vivvi generation, adopt the prompt-image pairs provided by VBench [67] and crop images to 16: 9 ratios for 720p resolution.
Baselines. We compare SVG2 against state-of-the-arr sparse attention algorithms including static method Sparse VideoGen (SVG) [4], and dynamic methods Sparge Attention [9] and XAttention [10]. Note that we skip the evaluation of XAttention on Wan2.1 as it is not supported yet. For SVG, Sparge/ttention, and XAttention, we use their official configurations.
Implementations. We prototype SVG2 as an end-to-end framework with customized kernels from FlashInfer [16] and benchmark NVIDIA H100 GPU with CUDA 12.8. For SVG2, choose C =100 and C₁ = 500, and explain the choice in $ D. To showcase the trade-off between generation quality and efficiency, we evaluate on various accuracy target (i.e., attention score recall) as detailed in $ 5.4. We also sample single data point for detailed comparison as shown in Table 1. We conduct experiments with sparse attention skipped during the first 30% of denoising steps for all methods, as these steps are critical for generation quality. following previous work [64, 68, 56, 59]. For experiment results without warmup, please check Table Appendix.
100
5.2 Quality Evaluation
We first qualitatively showcase the effectiveness of our proposed method by showing the visualization of attention maps. As shown in Figure 6, we collect attention maps from different attention heads when running Wan2.1 on prompts from VBench. Despite the diversity of the sparse patterns (i.e., different columns in Figure 6), semantic-aware permutation effectively densifies critical tokens into contiguous layout, which enables efficient computation without waste. Furthermore, by applying
8

---

Table 1:
Quality and efficiency benchmarking results of SVG2 and baselines. Warmup steps is set to 30%.
Model
Config
PSNR f
SSIM
LPIPS
VRench
Density
FLOP 1
Speedup
Wan 2.1
148, 720P, Image-to-Video
0.841
100%
526.76 PFLOPs
1x
SpargeAttn
21.181
0.665
0.333
38.99%
366.80 PFT LOPs
1.47
SVG
24.059
0.813
0.174
0.836
30.25%
343.88 PFLOPs
1.56x
Ours
26.562
0.861
0.138
0.838
31.28%
346.59 PFLOPs
1.68x
Ours-
24.510
0.836
301.62 PELOPs
1.84x
Wan 2.1
148, 720P, Text-to- Video
0.846
100%
658.46 PFLOPs
1x
SpargeAttn
20.519
0.623
0.343
0.820
42.03%
468 PFLOPs
1.44x
SVG
22.989
0,785
0.199
0.837
30.25%
429.86 PFLOPs
1.58x
Ours
25. 80.808
0.854
0.138
0.842
29.51%
427.43 PFLOPs
1,0x
Turbo
PELOPs
Hunyuan
138, 720R, Text-to- Video
0,850
100%
612.38 PFI LOPs
1x
SpargeArtn
27.892
0.884
0,151
42.62%
399.16 PFLOPs
1,53x
XAtiention
28.892
0.898
0.120
0,839
39.32%
386. 90 PFLOPs
1.56x
SVG
29.157
0.120
0.845
86%
351.75 PELOPs
1.91
FPS
381.75 PELOPs
Ours
30.452
0.910
0.117
0.852
25.45%
335.36 PELOPs
2,30x
Ours + FP8
30.389
0.908
0.118
0.851
25.45%
335.36 PFI .OPs
2.55x
w/ centroids cache
w/o centroids cache
FlashInfer (Static)
Ours (Dynamic)
Theoretical
27
step
(Expereive)
9000
26
U stop
1.72x
1.88x
Density
76x smaller
I step stop stop
6750
25
GFLOPS
step
24
2 step
step
stpp.
2250
10
23
5
10
15
20
25
30
35
(20,100)
(20,200) (00,500) (100. 100) 200)(100 500)
Latency (ms)
Cluster, K. Cluster)
(a) Efficiency evaluation of k-meuns
(b) Efficiency evaluation of customized attention
Figure 7: Efficiency evaluation for fast k-means with centroids cache and customized attention kernel.
centroid-based top-p selection and then undoing the permutation, the permuted attention map is recovered into the original layout, which shows high similarity to the original attention map.
To quantitatively assess the generation quality, evaluate the quality of videos generated by SVG2, when compared to baselines and report the results Table 1. SVG2 consistently outperforms all baseline methods in terms of PSNR, SSIM, and LPIPS, while still maintaining the highest speedup. Specifically, SVG2 achieves an average PSNR of 26.5 on Wan2. and 30,4 on Hunyuan Video, demonstrating its effectiveness generating highly consistent and smooth videos.
Due to space limitations, the full results of VBench can be found at Table 3 and Table in the appendix.
5.3 Efficiency Evaluation
Efficient k-means with centroids cache. To demonstrate the effectiveness of centroids cache in improving efficiency of k-means, we compare the density achieved by SVG2 to reach the 90% attention recall, when varying different number of execution time (i.c., number of k-means iterations). We use widely-used algorithm k-means++ [63]. Since the k-means quality directly determine the accuracy of critical token identification, it also determines the achieved density. The lower the density is, the better the k-means is. As shown in Figure 7(a), enabling centroids cache reduces the end-to-end latency of k-means by 76x when reaching comparable or lower density. This demonstrates the effectiveness of centroids cache, which greatly reduce the initialization time.
Efficient attention kernel with dynamic block-sizes. To showcase the efficiency of our customized attention kernels with dynamic block-sizes, we evaluate the computation FL.OPs of our implementa-

---

tion compared with ofate-ofthhe-at attention library. FlashInfer [16]. We vary different combination of hyper-parameters (i.e., number of clusters C and C+)) and apply centroid- based top-p selection to generate the practical workloads of dynamic block sizes. We fix the attention recall as 90%. As shown in Figure 7(b), our customized kernels achieve an average of 1.48x computation reduction. On practical setup with C2 = 100, C2 500, ours achieves 1.88x reduction of computation waste.
End-to-end speedup evaluation. To showcase the end-to-end speedup of SVG2, we incorporate several efficiency metrics, including density, FLOPs, and end-to-end speedup into Table 1. SVG2 achieves an average speedup of 1.82x while maintaining the highest generation quality. We also include a SVG2-Turbe Turbo to showcase the efficiency potential, which maintains a similar generation quality as baselines but achieves much higher speedup. Specifically, LVG2-Turbo achieves 2.5x smaller density compared to SVG while achieving an even better PSNR of 23.7. 7. Such results can be cross-validated with the sensitivity evaluation in $ 5.4.
5.4 Sensitivity Test on Quality- Trade-off
To validate the effectiveness of SVG2, we conduct a comprehensive evaluation on Wan2.1-12V 1-12 comparing it against baseline methods across wide range of computational budgets (i.e., density). As shown in Figure 2. SVG2 consistently achieves better generation quality at any given density, trade-off. Notably, SVG2 reduces density
positioning it the Pareto frontier of the by up to 2.3x while maintaining the same PSNR.
5.5 Ablation Study on Semantic-Aware Permutation
Effectiveness on improving identification accuracy. To assess the effectiveness semantic-awan permutation in improving the accuracy of critical token identification, we measure attention recall by comparing semantic-aware permutation-enabled and semantic-aware permutation- disabled configurations across varying computational bud- gets. Both methods meca-ppoling and maintain the same cluster size for consistency. As shown in Figure 8, semantic-aware permutation consistently achieves higher attention recall, indicating more accurate identification of critical tokens. This improvement is attributed to the semantic-coherest clusters generated by semantic-aware permutation, which offer precise representations.
w/ Kmeans Permutation
wlo Kimeas Penmutation
Attention Rocall
0.9
0.8
0.7
0.6
Density
Figure 8: Attention recall across various den- sities. Enabling permutation consistently sur- passes disahling permutation.
Effectiveness on reducing computation waste. We further investigate the impact of semantic-aware permutation on reducing computational waste. Specifically, we use the same set of critical tokens selected by centroid-based top-p selection. For the semantic-aware permutation-enablec configuration, we feed the contiguous layout generated after permutation into GPUs, while for the semantic-awan permutation-disablee configuration, we use the seattered layout before permutation. As shown in our results, enabling semantic-awan permutation reduces computational overhead by an average of 36%, while maintaining the same set of critical tokens.
Conclusion & Limitation
In this paper, we proposed SVG2, training- free sparse attention approach for accelerating DiT-based video generation. By clustering tokens based on semantic similarity. SVG2 accurately identifies critical tokens. By permuting critical tokens into a contiguous layout, SVG2 effectively reduces the computation waste. Comprehensive evaluations show that SVG2 achieves a superior trade-off between generation quality and efficiency, making video generation more efficient and practical. The major limitation of this paper lies in the lack of discussion and evaluation on whether the proposed methods can be extended to attention mechanisms other than DiTs.
References
[1] Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai. Jin Zhou. Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, al. Hunyuanvideo: A systematic framework for large video generative models. arXiv
10

---

preprint arXi::2412.03603 2024. 2,
[2] Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen. Feiwu Yu, Haiming Zhao, Jianxiao Yang. Jianyuan Zeng, Jiayu Wang. Jingfeng Zhang, Jingren Zhou, Jinkai Wang. Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fang. Tianxing Wang. Tianyi Gui, Tingyu Weng, Tong Shen, Wei Lin, Wei Wang. Wei Wang. Wenmeng Zhou, Wente Wang. Wenting Shen, Wenyuan Yu. Xianzhong Shi. Xiaoming Huang, Xin Xu. Yan Kou, Yangyu Lv. Yifei Li, Yijing Liu, Yiming Wang. Yingya Zhang, Yitong Huang, Yong Li, You Wu, Yu Liu, Yulin Pan, Yun Zheng. Yuntao Hong. Yupeng Shi, Yutong Feng, Zeyinzi Jiang. Zhen Han, Zhi-Fan Wu, and Ziyu Liu. Wam: Open and advanced large-scale vidco generative models, arXiv preprint ar 2ii::2503.203 14, 2025. 2, 3,8
[3] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang. Wenyi Hong, Xiaohan Zhang. Guanyu Feng. et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408, 06072, 2024. 2
[4] Haocheng Xi, Shuo Yang, Yilong Zhao, Chenfeng Xu, Muyang Li, Xiuyu Li, Yujun Lin, Han Cai, Jintao Zhang, Dacheng Li, et al. Sparse videogen: Accelerating video diffusion transformers with spatial-temporal sparsity. In ICML, 2025. 2, 2,8
[5] Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikei, and Song Han. Quest: Query-aware sparsity for efficient long-context IIm inference. In ICMI,, 2024. 2,
[6] Zhenyu Zhang, Ying Sheng. Tinyy Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian. Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. H2o: Heavy-hitter oracle for efficient generative inference of large language models, 2023. 2, 3,4
[7] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In ICLR, 2024. 2.3
[8] Peiyuan Zhang, Yongqi Chen, Runlong Su, Hangliang Ding, Ion Stoica, Zhenghong Liu, and Hao Zhang. Fast video generation with sliding tile attention. arXiv preprint arXiv:2502.05507, 2025. 2,
[9] Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia Wei, Haocheng Xi. Jun Zhu, and Jianfei Chen. Spargeattn: Accurate sparse attention accelerating any model inference. In ICML, 2025. 2, 3, 4, 8
[10] Ruyi Xu, Guangxuan Xiao, Haofeng Huang, Junxian Guo, and Song Han. Xattention: Block sparse attention with antidiagonal scoring. In ICMI., 2025. 2,3,488 8
[11] Kan Zhu, Tian Tang, Qinyu Xu, Yile Gu, Zhichen Zeng, Rohan Kadekodi, Liangyu Zhao, Ang Li, Arvind Krishnamurthy, and Baris Kasikei. Tactic: Adaptive sparse attention with clustering and distribution fitting for long-contest Ilms, 2025. 2,3,4
[12] Yucheng Li, Huiqiang Jiang. Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Amir H. Abdi, Dongsheng Li, Jianfeng Gao, Yuging Yang, and Lili Qiu. Mminference: Accelerating pre-filling for long-context vlms via modality-aware permutation sparse attention, 2025. 2,
[13] Ningxin Zheng, Huiqiang Jiang, Quanlu Zhang. Zhenhua Han, Yuqing Yang. Lingxiao Ma, Fan Yang. Chengruidong Zhang, Lili Qiu, Mao Yang, and Lidong Zhou. Pit: Optimization of dynamic sparse deep learning models via permutation invariant transformation, 2023.
[14] NVIDIA. Nvidia a100 tensor core gpu. https:/ http:://www.vviia.. cww Dolutions/Dattee Ciira-CCeeeer/ee wtttter/a////pff////e /nwidia-iiiaiiiii a100 letoo--ttaaheee------eeeeeee pdf, 2021. 2,5
en zz/ reb
[15] Chaofan Lin, Jiaming Tang, Shuo Yang. Hanshuo Wang, Tian Tang, Boyu Tian, Ion Stoica, Song Han, and Mingyu Gao. Twilight: Adaptive attention sparsity with hierarchical top-p pruning, 2025.
[16] Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang. Tianqi Chen, Baris Kasikei, Vinod Grover, Arvind Krishnamurthy, and Luis Ceze. Flashinfer: Efficient and customizable attention engine for 1l inference serving, 2025. 3, 7, 8, 10
[17] Yifci Xia, Suhan Ling, Fangeheng Fu, Yujic Wang, Huixia Li, Xuefeng Xiao, and Bin Cui. Training-free and adaptive sparse attention for efficient long video generation, 2025. 3
[18] Tianchen Zhao, Ke Hong. Xinhao Yang. Xuefeng Xiao, Huixia Li, Feng Ling, Ruiqi Xie, Siqi Chen, Hongyu Zhu, Yichong Zhang, and Yu Wang. Paroattention: Pattern- aware reordering for efficient sparse and quantized attention in visual generation models, 2025.

---

[19] Xin Tan, Yuetao Chen, Yimin Jiang. Xing Chen. Kun Yan, Nan Duan. Yibo Zhu, Daxin Jiang, and Hong Xu. Dsv: Exploiting dynamic sparsity to accelerate large-scale video dit training. 2025. 3
[20] Akide Liu, Zeyu Zhang, Zhexin Li, Xuchai Bai, Yizeng Han, Jiasheng Tang. Yuanjie Xing, Jichao Wu, Mingyang Yang. Weihua Chen, Jiahao He, Yuanyu He, Fan Wang, Gholamreza Haffari, and Bohan Zhuang. Fpsattention: Training-aware fp8 and sparsity co-design for fast video diffusion, 2025. 3
[21] Wenhao Sun, Rong-Cheng Tu, Yifu Ding, Zhao Jin, Jingyi Liao, Shunyu Liu, and Dacheng Tao. Vorta; Efficient video diffusion via routing sparse attention, 2025. 3
[22] Yuechen Zhang, Jinbo Xing. Bin Xia, Shaoteng Liu, Bohao Peng, Xin Tao, Pengfei Wan, Eric Lo, and Jiaya Jia. Training-fie free efficient video generation via dynamic token carving. 2025.
[23] Shuo Yang, Ying Sheng, Joseph E. Gonzalez, Ton Stoica, and I Lianmin Zheng. Post-training sparse attention with double sparsity, 2024. 3
[24] Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. Duoattention: Efficient long-context Ilm inferenee with retrieval and streaming heads, 2024. 3
[25] Huiqiang Jiang, Yucheng Li, Chengruidong Zhang. Qianhui Wa, Xufang Luo, Surin Ahn, Zhenhua Han, Amir Abdi, Dongsheng Li, Yew Lin. Minference 1.0: Accelerating pre- pre-filling for long-context Ilms via dynamic sparse attention. NeurIPS, 2024. 3
[26] Xunhao Lai, Jianqiao Lu, Yao Luo, Yiyuan Ma, and Xun Zhou. Flexprefill: A context-aware sparse attention mechanism for efficient long-sequence inference, 2025. 3
[27] Yizhao Gao, Zhichen Zeng, Dayou Du, Shijie Cao, Peiyuan Zhou, Jiaxing Qi. Junjic Lai, Hayden Kwok- Hay So, Ting Cao. Fan Yang, and Mao Yang. Seerattention: Learning intrinsic sparse attention in your llms, 2025.
[28] Chaojun Xiao, Pengle Zhang. Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. Inflim: Training-fiee free long-contess extrapolation for Ilms with an efficient context memory, 2024. 3
[29] Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Zero-shot extreme length generalization for large language models, 2024.
[30] Angelos Katharopoulos, Apoory Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mns: Fast autoregressive transformers with linear attention, 2020. 3
[31] Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardwarc-efficien training, 2024. 3
[32] Songlin Yang, Jan Kautz, and Ali Hatamizadeh, Gated delta networks: Improving mamha2 with delta rule, 2025. 3
[33] Albert Gu and Tri Dao. Mamba: Lincar-time sequence modeling with selective state spaces, 2024, 3
[34] Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, 2024. 3
[35] Yu Gao, Jiancheng Huang, Xiaopeng Sun, Zequn Jic, Yujic Zhong, and Lin Ma. Matten: Video generation with mamba-attention, 2024. 3
[36] Hongjie Wang, Chih-Yao Ma. Yen-Cheng Liu, Ji Hou, Tao Xu, Jialiang Wang. Felix Juefei-Xu, Yaqiao Luo, Peizhao Zhang, Tingbo Hou, Peter Vajda, Niraj K. Jha, and Xiaoliang Dai. Lingen: Towards high- nighressouution minute-length tox--tovvidoo generation with linear computational complexity, 2025. 3
[37] Jiancheng Huang, Gengwei Zhang, Zequn Jie, Siyu Jiao, Yinlong Qian, Ling Chen, Yunchao Wei, and Lin Ma. M4v: Multi-modal mamba for text-to-video generation, 2025. 3
[38] Ryan Po, Yotam Nitzan, Richard Zhang, Berlin Chen, Tri Dao, Eli Shechtman, Gordon Wetzstein, and Xun Huang. Long-context state-space video world models, 2025.
[39] Karan Dalal, Daniel Koceja, Gashon Hussein, Jiarui Xu, Yue Zhao, Youjin Song. Shihao Han, Ka Chun Cheung, Jan Kautz, Carlos Guestrin, Tatsunori Hashimoto, Sanmi Koyejo. Yejin Choi, Yu Sun, and Xiaolong Wang. One-minute video generation with test-time training, 2025. 3
12

---

[40] Enze Xie, Junsong Chen, Yuyang Zhao, Jincheng Yu, Ligeng Zhu, Chengyue Wu, Yujun Lin, Zhekai Zhang, Muyang Li, Junyu Chen, Han Cai, Bingchen Liu, Daquan Zhou, and Song Han. Sana 1.5: Efficient scaling of training-time and inference-time compute in linear diffusion transformer, 2025.
[41] Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng 7hu, Yao Lu, and Song Han. Sana: Efficient high-resolution image synthesis with linear diffusion transformers, 2024. 3
[42] Junsong Chen, Yuyang Zhao, Jincheng Yu, Ruihang Chu, Junyu Chen, Shuai Yang. Xianbang Wang, Yicheng Pan, Daquan Zhou, Huan Ling, Haozhe Liu, Hongwei Yi, Hao Zhang, Muyang Li, Yukang Chen, Han Cai, Sanja Fidler, Ping Luo, Song Han, and Enze Xie. Sana-vidco: Efficient video generation with block linear diffusion transformer, 2025.
[43] Junyu Chen, Dongyun Zou, Wenkun He, Junsong Chen, Enze Xie, Song Han, and Han Cai. De-ae 1.5: Accelerating diffusion model convergence with structured latent space, 2025. 3
[44] Wenkun He, Yuchao Gu, Junyu Chen, Dongyun Zou, Yujun Lin, Zhekai Zhang, Haocheng Xi, Muyang Li, Ligeng Zhu, Jincheng Yu, Junsong Chen, Enze Xie, Song Han, and Han Cai. De-gen: Post- fost-training diffusion acceleration with deeply compressed latent space, 2025. 3
[45] Junyu Chen, Han Cai, Junsong Chen, Enze Xie, Shang Yang. Haotian Tang, Muyang Li, Yao Lu, and Song Han. Deep compression autoencoder for efficient high-resolution diffusion models, 2025. 3
[46] Tianwei Yin, Qiang Zhang, Richard Zhang, William T Freeman, Fredo Durand, Eli Shechtman, and Xun Huang. From slow bidirectional to fast autoregressive video diffusion models. CVPR, 2025. 4
[47] Xun Huang, Zhengqi Li. Guande He. Mingyuan Zhou, and Eli Shechtman. Self forcing: Bridging the train-test gap in autoregressive video diffusion, 2025.
[48] Shuai Yang. Wei Huang, Ruihang Chu, Yicheng Xiao, Yuyang Zhao, Xianhang Wang, Muyang Li, Enze Xie, Yingeong Chen, Yao Lu, Song Han, and Yukang Chen. Longlive: Real-time interactive long video generation, 2025. 4
[49] Lvmin Zhang and Maneesh Agrawala. Packing input frame context in next-frame prediction models for video generation, 2025. 4
[50] Min Zhao, Guande Hc, Yixiao Chen, Hongzhou Zhu, Chongxuan Li, and Jun Zhu. Riflex: A free lunch for length extrapolation in video diffusion transformers, 2025. 4
[51] Yu Lu and Yi Yang. Freelong+++: Training-free long video generation via multi-band spectralfusion, 2025.
[52] Xingyang Li. Muyang Li. Tianle Cai, Haocheng Xi, Shuo Yang. Yujun Lin, Lvmin Zhang, Songlin Yang. Jinbo Hu, Kelly Peng, Mancesh Agrawala, Ion Stoica, Kurt Keutzer, and Song Han. Radial attention: o(n log n) sparse attention with energy decay for long video generation, 2025.
[53] Jianzong Wu, Liang Hou, Haotian Yang. Xin Tao, Ye Tian, Pengfei Wan, Zhang, and Yunhai Tong. Vmoba: Mixture-of-block attention for video diffusion models, 2025. 4
[54] Shengqu Cai, Ceyuan Yang. Lvmin Zhang, Yuwei Guo, Junfei Xiao, Ziyan Yang. Yinghao Xu, Zhenheng Yang, Alan Yuille, Leonidas Guibas. Maneesh Agrawala, Lu Jiang, and Gordon Wetzstein. Mixture of contexts for long video generation, 2025. 4
[55] Peiyuan Zhang, Yonggi Chen, Haofeng Huang, Will Lin, Zhengzhong Liu, Ion Stoica, Eric Xing, and Hao Zhang. Vsa: Faster video diffusion with trainable sparse attention, 2025. 4
[56] Zhengyao Lv, Chenyang Si. Junhao Song, Zhenyu Yang. Yu Qiao, Ziwei Liu, and Kwan-Yer K. Wong. Fastercache: Training- free video diffusion model acceleration with high quality. 2024. 4, 8
[57] Jiazi Bu, Pengyang Ling, Yujie Zhou, Yibin Wang. Yuhang Zang. Tong Wu, Dahua Lin, and Jiaqi Wang. Dicache: Let diffusion model determine its own cache, 2025.
[58] Huanpeng Chu, Wei Wu, Guanyu Fen, and Yutao Zhang. Omnicache: A trajectory-oriented global perspective on training-free cache reuse for diffusion transformer models, 2025. 4
[59] Feng Liu, Shiwei Zhang. Xiaofeng Wang. Yujie Wei, Haonan Qiu, Yuzhong Zhao, Yingya Zhang, Qixiang Ye, and Fang Wan. Timestep embedding tells: It's time to cache for video diffusion model. In CVPR, 2025.
4, 7, 8
13

---

[60] Xin Zhou, Dingkang Liang, Kaijin Chen, Tianrui Feng, Xiwu Chen, Hongkai Lin, Yikang Ding, Feiyang Tan, Hengshuang Zhao, and Xiang Bai. Less is enough: Training-free video diffusion acceleration via auniieeadaptivv caching, 2025. 4
[61] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention you need. NeurlPS, 2017. 4
[62] Zihao Ye, Ruihang Lai, Junru Shao, Tianqi Chen, and Luis Ceze. Sparsetir: Composable abstractions for sparse compilation deep leamning 2023."
[63] David Arthur and Sergei Vassilvitskii. k-means++: the advantages of careful seeding. In Proceedings of Eighteenth Annual LCM-SIAM Symposium on Discrete Algorithms, SODA '07, page 1027-1035, USA. 2007. Society for Industrial and Applied Mathematics. 7,9
[64] Xuanlei Zhao, Xiaolong Jin, Kai Wang, and Yang You. Real-time video generation with pyramid attention broadcast, 2025. 7,
[65] Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Dao. Flashattention-:: Fast and accurate attention with asynchrony and low-precision. NeurlPS, 2024."
[66] Juechu Dong, Boyuan Feng, Driss Guessous, Yanbo Liang, Horace He. Flex attention: A programming model for generating optimized attention kernels, 2024. 7
[67] Ziqi Huang, Yinan He, Jiashuo Yu, Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Champaisit, Yaohui Wang. Xinyuan Chen, Limin Wang. Dahua Lin, Yu Qiao, and Ziwei Liu. Vbench: Comprehensive benchmark suite for video generative models, 2023.
[68] Muyang Li, Tianle Cai, Jiaxin Cao, Qinsheng Zhang, Han Cai, Junjie Bai, Yangqing Jia, Ming- Yu Liu, Kai Li, and Song Han. Distrifusion: Distributed parallel inference for high-resolution diffusion models. In CVPR, 2024. 8
14

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our claims are empirically verified by $ 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the applicability limitation in 6.
15

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We provide the theoretical proof for the correctness of semantic permutation
in $ 4.1.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
All experiment details are included in 5.
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All the algorithms and implementation details are mentioned in 4 and $ 5. Our code and scripts will be released upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: /ttpsscccppbbpcc////eeccppcpppcp wwwubblccguuideee guwdee/ /CodeSubmii ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
17

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
All experiment hyperparameters are included in
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We test over multiple random seeds and the results are highly deterministic and reproducible. So we don't include error bar in the figure.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split. initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative еттог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The experiments are all conducted on common NVIDIA GPUs described in
$ 5.
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: //nurrip/........////// cc/public csuuidelinees
Answer:
[Yes]
Justification:
The research conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
This paper introduces a method to accelerate video generation models, so does not have direct societal impact. However, the use of video generation models may have social impact, which can be amplified by this technique.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
19

---

Answer: [Yes]
Justification: We will explicitl specify the usage permission of our method with proper licenses.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: This paper uses open-source cltasetts and models in $ 5. The usage respects all the original licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of derived asset (if has changed) should be provided. this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
20

---

14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this paper does involve LLMs any important, original or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: / httpr/pssscc//oofeennneeeeeee//e for what should or should not be described.
21

---

A Visualization of the Generated Videos
We provide visualization comparison between SVG2 and Dense Attention Hunyuan Video and Wan 2.1. Results in Figure 9 and Figure 10 demonstrate that SVG2 can preserve high pixel-level fidelity, achieving similar generation quality compared with the dense attention. Real video samples are provided in the supplementary materials.
Dense Attention
SVG2
Dense Attention
SVG2
Dense Attention
SVG2
Dense Attention
SVG2
Dense Attention
SVG2
Dense Attention
SVG2
Dense Attention
SVG2
Figure 9: Comparion of Dense Attention SVG2 on Hunyuan Video and Wan 2.1 Text-to- Video generation.
22

---

Dense Attention
SVG2
Dense Attention
SVG2
Dense Attention
SVG2
Dense Attention
SVG2
Dense Attention
SVG2
Dense Attention
SVG2
Dense Attention
SVG2
Dense Attention
SVG2
Figure 10: Comparison Dense Attention SVG2 Wan Image-to- Emage-to-Video Video generation.
23

---

B Performance Comparison in Warmup-free Setting
We present the performance comparison between SVG2 and the baseline without warmup steps. We find that our method consistently offers better quality under warmup-fres setting.
Table 2: Quality and efficiency benchmarking results of SVG2 and baselines. Warmup steps is set to 0%.
Model
Config
PSNR
SSIM
LPIPS
YBench
Deusity
FLOP
Attn Speedup
Speedup
Wan 2.1
148, 7201, Imsge do Video
0.41
100%
526.76 PFLOPs
1x
1x
2.2xx
18.276
0.615
0.317
0.822
29.34%
262.10 PILOPs
255x
2.10x
Wan 2.1
110, 720r, Teer so Video
0.851
100%
658.46 PTLOPs
1x
1x
SVG
13.294
0,407
0312
0.849
29.54%
328.56 PFLOPs
2.28x
1.89
Hony 130. 720P, Teor-to Video
0.820
100%
612.38 PFLOPs
1x
1x
SVG
12.298
0,492
0.483
0.808
29 86%
240.05 PFLOPs
3.45x
2.48x
Ours
19,879
0.735
0.260
0.816
28.94%
235.16 PFLOPs
4.0x
2.69%
C VBench Results
We provide the full VBench results of SVG2 and baselines Table and Table 4. These results clearly shows that SVG2 outperforms all other baselines.
Table 3: VBench result of SVG2. Warmup steps is 0%.
Model
Config
SubConsis
BackConsis
MotionSmooth
AcsQual
ImagQual
Average
7208 Image-to- Video
0.946
0.979
0.709
SVG
0.916
0.935
0,976
0.591
0.698
0.823
0.936
0.946
0.977
0.597
0,700
Wan 2.1
/48, 720P, Text-to- Video
0.970
0.970
0.992
0.612
0.708
0.851
SVG
0.963
0.969
0,991
0.612
0.708
0.849
Ours
0.971
0.970
0.992
0.624
0.707
0.852
Hunyuan
13R, 720P Text-to Video
0.888
0.938
0.994
0.594
0.685
0.820
0.867
0.594
0.656
Ours
0. 0.888
0.935
0.994
0.589
0.675
0.816
Table 4: VBench result SVG2. Warmup steps 30%.
Model
Config
SubConsis
BackConsis
MotionSmooth
AesQual
ImagQuall
Average
Wan 2.1
14B, 720P Image-to-Video
0.946
0.956
0.979
0.618
0.709
0.841
0.941
0.948
0.606
0.709
0.836
Ours
0.943
0.951
0.977
0.606
0.709
0.838
Wan
14B, 720P, Text-to-Video
0.956
0.968
0.983
0.613
0.713
0.846
SpargeAtto
0.927
0.948
0,978
0.567
0.684
0.820
0.947
0.960
0.980
0.597
0.703
0.837
Ours
0.954
0.965
01.822
0.602
0.709
0.842
Hunyuan
13B, 720P, Text-to- Video
0.915
0.941
0.993
0.648
0.753
0.850
XAttention
0.912
0.924
0.992
0.631
0.739
0.839
SVG
0.914
0.928
0.993
0.652
0.739
0.845
Ours
0.917
0.946
0.993
0.657
0.751
0.852
24

---

Throughout Densityy Row: [200]
1000, 2000,
Trrocchput
Throughout
Sparse HNww200
370
NMw-222
300
303
200
Row-220 11500
250 2000
Sparse F32
Rom-2000 4000
Denoz
Cal MM
200
0.4
0.0
3.0
0.2
0.5
Olock Daralyy
Dluck Dusalyy
Figure 11: Efficiency evaluation for attention kernel. We fix the number of query clusters and vary the number of key clusters.
Throughout Row.
300. Diocks Col:
Throughout
300
300
353
Throughpi L20g
300
Spare: 1A2
x
Spane (A3
2 3320
202
Spurae FA2
Sparac 1A2
R0e-100 3000
2 1200
Spanse 1A3
Spanaa 7A3
Rou-to 3320
100m=00 2200
sparce M0w02000 3000
sparse was
Now02000 2500
202
Sparce Rose 100 3000
Sparse (&X)
Row-2000 1250
Sparis
Roumm00 1000
Spanso
Derse
303
04
0.0
0.5
140
0.2
04
0.6
0.5
3.0
Block Danety
Decicty
Figure 12: Efficiency evaluation of our attention kernel, where we fix the number of key clusters and vary the number of query clusters.
D Ablation on the Number of Clusters
D.1 Effect on the Sparse Attention Kernel
Our sparse attention kernel fully compatible with both FlashAttention-2 and FlashAttention-3. We implemented Sparse VideoGen2 on both FA2 and FA3 backends, achieving substantial speedups on A100 and H100 GPUs. Under the Wan 2.1 setting with sequence length 74,256 on H100, benchmark kernel performance by comparing sparse and dense attention across varying densities and cluster configurations. In each experiment, we control either C, or Ct. fixing one while varying the other. Results in Figure I and Figure 12 demonstrate that the kernel's performance will drastically decrease when C, is larger than 200. However, the performance is nearly identical as we increase C to as large as 4000. This suggests us to adopt a larger Ci. than Cg-
D.2 End-to-end Latency-quality Trade-off
We further varied the Q/K cluster counts and measured both PSNR and end-to-end efficiency. Our results Table 5 show that setting Q = 100 and K 500 provides the best balance between generation quality and efficiency. While increasing the number of clusters generally improves quality, efficiency can degrade due hardware layout constraints. In particular, tensor cores require fixed input sizes (e.g., 64 for m64n64k16) fully utilize computation, meaning that each Q cluster
25

---

Cq
Ck
PSNR
SSIM
LPIPS
Speedup
100
250
25.497
0.801
0.182
1,90x
100
1000
26.276
0.825
0.159
1.71x
50
500
22.561
0.742
0.258
1.90x
200
500
26.213
0.820
0.157
1.78x
400
500
26.488
0.868
0.132
1,25x
100
500
26.128
0.816
0.169
1.89x
Table 5: Performance comparison across different QC and KC settings.
must contain at least 64 tokens on average. Cluster counts beyond Q = 100 or = 550 lead to underutilization, reducing efficiency despite potential quality gains.
D.3 Ablation Permutation
We performed ablation studies to investigate whether the query and key representations can adopt the same clustering strategy. Specifically, evaluated three variants: applying Q clustering permutation TQ to both Q and K, applying K clustering permutation *x to both, and using clustering based on hidden states before QKV linear layer (i.c., shared QK embedding) for both sides иT. As shown in Table 6, all three variants led to worse PSNR even with more computation budget (i.e., density) compared clustering Q and independently.
Table 6: Permutation comparison with corresponding Density and PSNR values
Permutation used by Q Permutation used by K
Density
PSNR
QQ
πK
31.28%
26.562
TQ
IQQ
38.23%
22.439
TK
38.58%
22.183
TS
87.27%
26.495
To further understand this, we compared the permutations Q and clustering and found that the permutation patterns differ substantially. Specifically, we calculate the Adjusted Rand Index value between Q clusters and clusters, and the average ARI is 0.345, which not very high. Therefore, clustering Q and K independently is necessary for preserving the expressiveness of attention.
E
Performance Gap between Hunyuan Video and Wan 2.1
E.1
Quality Difference
In our experiments, we find that the quality performance (c.g., PSNR, SSIM, LPIPS) on Wan 2.1 is generally lower than Hunyuan uan Video across all methods. The reason is that Hunyuan is relatively robust against precision variance while Wan2.1 highly sensitive. For instance, when evaluating the same dense attention using different backends (FlexAttention, FlashAttention, Torch SDPA). Wan2.1 exhibited PSNR low as 27-28. However, Hunyuan Video exhibits 33-34 PSNR despite no setup changes. Therefore, it is natural that SVG2 achieves a lower PSNR on Wan compared with Hunyuan Video, due to its sensitivity to numerical changes. These differences are largely model- specific and reflect varying sensitivity to low-level numerical behaviors, which do not correlate with the performance of the methodology.
Video
E.2 Speedup Difference
We also find that the speedup result on Wan 2.1 is generally lower than Hunyuan Video (1.89x versus 2.30x). The difference end-to-end speedup between Hunyuan Video and Wan primarily stems from their varying attention cost ratios, which are mainly due to different context lengths and model architectures. Specifically, Hunyuan Video's context length 118k, while Wanx's 's context length 75k. Hunyuan Video has 2 parts in its layers: Self Attention and Feed-Forwanr Network, while Wanx
Video
26

---

has an additional cross- aatssttttnniinn block. Therefore, the attention proportion in Hunyuan Video will be larger than WanX. Since our method primarily accelerates the attention module via SVG2, the overall speedup naturally scales with its contribution to total runtime. We will revise Table to separate attention-level and end-to-end speedups to improve clarity explicitly.
27


Reviews:
[{"review_id": "h7KjnwJyRf", "reviewer": "Reviewer_QGvt", "summary": "This paper introduces SAPAttn, a training-free sparse attention framework designed to accelerate video generation based on Diffusion Transformers (DiTs). The paper's core contribution lies in its semantic-aware permutation, top-p dynamic budget control, and customized kernel implementations, which together achieve a Pareto frontier trade-off between generation quality and efficiency.", "questions": "see weakness", "limitations": "see weakness", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\nThe proposal of semantic-aware permutation is novel. By clustering tokens based on semantic similarity using k-means and reordering them, the method improves the identification accuracy of critical tokens and reduces computation waste. \n\n\nWeaknesses and Suggestions\n1. In Figure 5(b), some key centroids are shown in both red and blue. The authors need to explain the rationale behind these cross-category centroids. Are they determined based on specific semantic associations, or is it a random occurrence? A detailed theoretical analysis or experimental verification should be provided to address this confusion.\n2. The paper should explicitly clarify whether critical tokens are equivalent to cluster centers. This would help readers better understand the core idea of the proposed method.\n3. In Table 1, HunyuanVideo achieves a 2.30× speedup at a density of 25.45%, while Wanx only achieves a 1.89× speedup at a density of 12.87%. The authors should provide a detailed analysis of the reasons for this difference, considering factors such as computational efficiency and model architecture differences.\n4. The paper mentions using VBench for evaluation but does not test the main benchmark metrics of VBench. These main benchmark metrics are crucial for comparing the performance of text-to-video models. The lack of such tests makes it difficult to fully assess the performance advantages of SAPAttn. The authors should supplement the evaluation with VBench main benchmark metrics and report the results in detail.\n5. As far as I know, Flash Attention does not support attention masks and only supports the original full attention mechanism. The complex attention mechanisms proposed in this paper may not be compatible with Flash Attention v3. However, the paper only provides FLOPs comparisons instead of direct time comparisons, which may limit the assessment of acceleration effects. The authors should consider using Flash Attention v3 as the baseline and directly comparing the total time for video generation to more accurately reflect the actual acceleration performance of SAPAttn.\n6. However, the completeness and usability of the code and data cannot be verified during the review process. The authors should provide the code and data, for example, detailed full data of each video in VBench evaluation to the reviewer, and usable after release to facilitate further research by other researchers.", "was_revised": true, "final_justification": "My concerns regarding the details have been fully addressed.", "rebuttal": "Dear Reviewer QGvt,\n\nThanks for your insightful reviews. Below, we provide point-by-point responses to each of your comments.\n\n> Weakness 1: In Figure 5(b), some key centroids are shown in both red and blue. The authors need to explain the rationale behind these cross-category centroids. Are they determined based on specific semantic associations, or is it a random occurrence? A detailed theoretical analysis or experimental verification should be provided to address this confusion.\n\n**Reply**: The appearance of shared key centroids is an expected outcome of sparse attention. In SAPAttn, query and key tokens are **clustered independently**, where each query cluster selects its key/value clusters based on centroid-level attention scores. Since this selection is done independently for each query cluster, it is natural for the same key cluster to be selected by multiple query clusters, forming a many-to-many mapping. For example,  one key/value cluster that encodes semantically important information can be selected by all the other query clusters. This behavior is clearly shown in Figure 6, where certain key clusters are densely attended by multiple query clusters in the “after permutation” attention maps. \n\nThis phenomenon is also well observed by attention sink [1] in LLMs, where the first key/value tokens are highly attended to by most query tokens.  Therefore, such flexible many-to-many mapping widely exists and is important for high-quality generation, which is inherently supported by SAPAttn’s design.\n\n[1] Xiao, Guangxuan, et al. \"Efficient streaming language models with attention sinks.\" arXiv preprint arXiv:2309.17453 (2023).\n\n> Weakness 2: The paper should explicitly clarify whether critical tokens are equivalent to cluster centers. This would help readers better understand the core idea of the proposed method.\n\n**Reply**: Thanks for the suggestion, and we will incorporate this discussion into the revised manuscript. \n\nWe respectfully clarify that critical tokens and cluster centers serve different roles in our method. Cluster centers are computed during semantic-aware permutation via k-means and act as representatives of semantically similar token clusters. In contrast, critical tokens are dynamically selected based on attention scores, and can be approximated by a subset of cluster centers. \n\nWhile critical tokens often reside within high-attention clusters, they are not equivalent to the whole cluster centers — they form a subset guided by the importance derived from centroid-based estimation.\n\n> Weakness 3: In Table 1, HunyuanVideo achieves a 2.30× speedup at a density of 25.45%, while Wanx only achieves a 1.89× speedup at a density of 12.87%. The authors should provide a detailed analysis of the reasons for this difference, considering factors such as computational efficiency and model architecture differences.\n\n\n**Reply**: Thank you for the valuable comment. The difference in end-to-end speedup between HunyuanVideo and Wan primarily stems from their varying attention cost ratios, which are mainly due to different context lengths and model architectures.\n\nSpecifically, HunyuanVideo’s context length is 118k, while Wanx‘s context length is 75k. HunyuanVideo has 2 parts in its layers: Self Attention and Feed-Forward Network, while Wanx has an additional cross attention block.\n\nTherefore, the attention proportion in HunyuanVideo will be larger than WanX. Since our method primarily accelerates the attention module via SAP, the overall speedup naturally scales with its contribution to total runtime. We will revise Table 1 to separate attention-level and end-to-end speedups to improve clarity explicitly.\n\n\n> Weakness 4: The paper mentions using VBench for evaluation but does not test the main benchmark metrics of VBench. These main benchmark metrics are crucial for comparing the performance of text-to-video models. The lack of such tests makes it difficult to fully assess the performance advantages of SAPAttn. The authors should supplement the evaluation with VBench main benchmark metrics and report the results in detail.\n\n**Reply**: We evaluate VBench main benchmark metrics on Wan 2.1 T2V and HunyuanVideo T2V models. We find that SAPAttn consistently performs better than baselines on VBench, demonstrating the effectiveness of our method.\n\n|    Wan 2.1     | SubConsis | BackConsis | MotionSmooth | AesQual | ImagQual  |\n|---------|-----------|------------|--------------|---------|-----------|\n| Dense   | 0.956     | 0.968      | 0.983        | 0.613   | 0.713     |\n| Sparge  | 0.927     | 0.948      | 0.978        | 0.567   | 0.684     |\n| SVG    | 0.947     | 0.96       | 0.98         | 0.597   | 0.703     |\n| SAPAttn | 0.954     | 0.965      | 0.982        | 0.602   | 0.709     |\n\n\n|   HunyuanVideo      | SubConsis | BackConsis | MotionSmooth | AesQual | ImagQual  |\n|---------|-----------|------------|--------------|---------|-----------|\n| Dense   | 0.915     | 0.941      | 0.993        | 0.648   | 0.753     |\n| XAttn   | 0.912     | 0.924      | 0.992        | 0.631   | 0.739     |\n| SVG    | 0.914     | 0.928      | 0.993        | 0.652   | 0.739     |\n| SAPAttn | 0.917     | 0.946      | 0.993        | 0.657   | 0.751     |\n\n\n> Weakness 5: As far as I know, Flash Attention does not support attention masks and only supports the original full attention mechanism. The complex attention mechanisms proposed in this paper may not be compatible with Flash Attention v3. However, the paper only provides FLOPs comparisons instead of direct time comparisons, which may limit the assessment of acceleration effects. The authors should consider using Flash Attention v3 as the baseline and directly comparing the total time for video generation to more accurately reflect the actual acceleration performance of SAPAttn.\n\n**Reply**: We appreciate the reviewer's valuable comment on the compatibility of SAPAttn with Flash Attention v3 (FA3). We want to clarify that SAPAttn is fully compatible with FA3. Specifically, we implemented SAPAttn in both FA2 and FA3 versions, ensuring substantial speedups on both A100 and H100 GPUs. The table below shows the latency under the Wan 2.1 setting with sequence length 74256 on H100, comparing sparse and dense attention using both FA2 and FA3 across various densities. \n\n| Density      | 0.1  | 0.2  | 0.3   | 0.4   | 0.5   | 1.0 (original FA2/FA3)  |\n|--------------|------|------|-------|-------|-------|-------------------------|\n| SAP-FA2 (ms) | 35.5 | 71.2 | 107.1 | 141.3 | 176.9 | 317.7                   |\n| SAP-FA3 (ms) | 24.4 | 47.6 | 70.9  | 95.4  | 117.7 | 195.6                   |\n\n> Weakness 6: However, the completeness and usability of the code and data cannot be verified during the review process. The authors should provide the code and data, for example, detailed full data of each video in VBench evaluation to the reviewer, and usable after release to facilitate further research by other researchers.\n\n**Reply**: Due to the policy, we are unable to share anonymous links during the review process. However, the core component—dynamic block sparse attention—has already been open-sourced and integrated into the FlashInfer library. The remaining parts of the system, including code and evaluation data (e.g., full video metrics for VBench), will be released shortly to support both research reproducibility and practical adoption."}, {"review_id": "jsFkmvxiig", "reviewer": "Reviewer_PvVz", "summary": "This paper proposes a sparse attention via semantic-aware permutation to accelerate video diffusion transformers, which clusters and reorders tokens based on semantic similarity using k-means, providing cluster representations for efficient computation. However, the demonstration of this paper is poor, which makes it confusing about how it achieves this goal, including the implementation details about k-means and the attention mechanism based on cluster tokens.", "questions": "The implementation details can be refined to facilitate the understanding of the proposed method.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n- The proposed method is training-free and can be seamlessly transferred to pre-trained DIT-based video diffusion models.\n- From the provided quantitative results, the proposed method achieves better content preservation under the same speedup.\n\n**Weaknesses:**\n- The paper claims that it employs k-means for token clustering and uses a centroid cache for fast convergence. However, it is unclear how to determine the optimal number of centroids, as the number of areas with different semantics can vary significantly in different videos. A limited number of centroids may cluster tokens with different semantics together, while a large number of centroids may limit acceleration performance.\n- It claims that it permutes tokens within each cluster into a contiguous layout for efficient computation. Does this technique only support the proposed method? It would be beneficial to transfer this technique to other methods and verify its performance in the same setting. This technique seems to be more of an engineering trick.\n- How is sparse attention performed based on the clustered tokens? Is it through token merging or attention only within the same cluster?\n- How does the top-k approach ensure semantic-aware clustering? There is no visualization of the clustered tokens using top-k.", "was_revised": true, "final_justification": "My coconers have been fully adressed.", "rebuttal": "Dear Reviewer PvVz,\n\nThanks for your insightful reviews. Below, we provide point-by-point responses to each of your comments.\n\n> Weakness 1: The paper claims that it employs k-means for token clustering and uses a centroid cache for fast convergence. However, it is unclear how to determine the optimal number of centroids, as the number of areas with different semantics can vary significantly in different videos. A limited number of centroids may cluster tokens with different semantics together, while a large number of centroids may limit acceleration performance.\n\n**Reply**: Thanks for the insightful question. In our response below, we use QC to denote the number of query centroids and KC to denote the number of key centroids. We empirically select KC and QC by a comprehensive ablation study of both efficiency and accuracy.\n\n**Rationale for the choice of QC:**\nWith KC fixed at 500, we analyze the effect of varying QC. As shown in Table 1, increasing QC from 50 to 100 yields a notable improvement in generation quality. However, further increasing QC to 200 or 400 leads to a sharp drop in computational efficiency. This degradation stems from constraints in the sparse attention kernel due to the hardware layout requirements, where the tensor core requires a fixed layout (e.g., input size 64 for m64n64k16) to saturate computation. As shown in Table 2, we observe a clear latency increase when QC > 100, and therefore, we choose QC = 100 as the optimal balance point.\n\n\n**Rationale for the choice of the KC:**\nWith QC fixed at 100, we investigate the impact of varying KC. Compared to KC = 250, KC = 500 yields significantly improved visual quality while only being slightly slower. Compared to KC = 1000, though KC = 1000 achieves slightly higher visual quality, KC = 500 is faster in terms of end-to-end latency. Both of them lie on the Pareto frontier with respect to the trade-off between generation quality and computational efficiency. In our case, we select KC = 500 for efficiency.\n\nTable 1: Generation quality and speedup when varying QC and KC.\n| QC  | KC   | PSNR   | SSIM  | LPIPS | Speedup  |\n|-----|------|--------|-------|-------|----------|\n| 100 | 250  | 25.497 | 0.801 | 0.182 | 1.90x    |\n| 100 | 1000 | 26.276 | 0.825 | 0.159 | 1.71x    |\n| 50  | 500  | 22.561 | 0.742 | 0.258 | 1.90x    |\n| 200 | 500  | 26.213 | 0.82  | 0.157 | 1.78x    |\n| 400 | 500  | 26.488 | 0.868 | 0.132 | 1.25x    |\n| **100** | **500**  | **26.128** | **0.816** | **0.169** | **1.89x**    |\n\n\nTable 2: Sparse attention kernel's latency when varying QC.\n Latency (ms)   | QC = 50 | QC = 100 | QC = 200 | QC = 400  |\n|----------------|---------|----------|----------|-----------|\n| Sparsity = 0.3 | 69.48   | 70.86    | 79.3     | 107.95    |\n| Sparsity = 0.1 | 23.49   | 24.4     | 28.59    | 34.7      |\n\n\n\n> Weakness 2: It claims that it permutes tokens within each cluster into a contiguous layout for efficient computation. Does this technique only support the proposed method? It would be beneficial to transfer this technique to other methods and verify its performance in the same setting. This technique seems to be more of an engineering trick.\n\nReply: We respectfully disagree with the opinion that dynamic permutation is an engineering trick. Instead, it is a core component of **an algorithm-hardware co-design** that is tightly integrated with our token-level kmeans identification. Our key insight is that critical tokens can be **highly scattered** across the attention map, which fundamentally motivates the need for permutation. This scattering only occurs with token-level identification, and reordering these tokens into contiguous layouts is necessary for efficient execution on modern hardware.\n\nIn contrast, prior block-level identification methods[1-2] (e.g., mean or max pooling) inherently produce block-contiguous layouts but suffer from coarse, inaccurate selection and significant computation waste. These methods do not benefit from permutation, as their sparsity patterns are already aligned for compute. Our approach, by contrast, leverages token-level semantic clustering via k-means to enable dynamic permutation and unlock substantial speedups. Thus, permutation is not a general-purpose trick but a mechanism **deeply co-designed** with SAPAttn’s core method.\n\n[1] Xu, Ruyi, et al. \"Xattention: Block sparse attention with antidiagonal scoring.\" arXiv preprint arXiv:2503.16428 (2025).\n\n[2] Zhang, Jintao, et al. \"Spargeattn: Accurate sparse attention accelerating any model inference.\" arXiv preprint arXiv:2502.18137 (2025).\n\n> Weakness 3: How is sparse attention performed based on the clustered tokens? Is it through token merging or attention only within the same cluster?\n\nReply: We would like to clarify that SAPAttn neither merges tokens nor performs attention only within the same cluster. In SAPAttn, query and key tokens are clustered independently using k-means. Based on the centroids of clusters, we get an estimation of the attention map and apply top-p selection for the estimated attention scores to determine, for each query cluster, which key clusters are critical. For each selected query-key cluster pair, every query token in the query cluster attends to all key tokens in the corresponding key cluster.\n\n> Weakness 4: How does the top-k approach ensure semantic-aware clustering? There is no visualization of the clustered tokens using top-k.\n\n**Reply**: Thank you for the comment. We would like to clarify that our method uses a top-p selection strategy, where the number of critical tokens is determined dynamically based on their importance, rather than using a fixed budget as in top-k selection. This allows for dynamic, content-adaptive selection of critical tokens based on cumulative attention scores. In Figure 6 (b), we provide a visualization of the attention map after semantic-aware permutation, where the tokens are clustered together through permutation. This clearly shows that semantically similar tokens are grouped together, making the attention map very condensed, thus being hardware-friendly.\n\n> Question 1: The implementation details can be refined to facilitate the understanding of the proposed method.\n\n**Reply**: Thank you for the suggestion. To provide a comprehensive clarification, we summarize the algorithmic workflow and our implementation below:\n\n**Algorithm-level clarification:**\n\nWe summarize the key steps of SAPAttn as follows:\n\n1. Independent Q/K clustering:\nQuery and key tokens are clustered separately using k-means into 100 and 500 clusters, respectively. To accelerate convergence, we use the centroids from the previous diffusion step as initialization. After clustering, tokens within each query/key cluster are permuted to be contiguous in memory.\n\n\n2. Centroid-level attention estimation and top-p selection:\nEach query cluster estimates attention scores to all key clusters via centroid-to-centroid computation, weighted by key cluster sizes. The top key clusters covering a cumulative top-p mass are selected, forming a sparse attention pattern.\n\n\n3. Sparse attention execution:\nEach token attends only to tokens in the selected key clusters, enabling token-level sparsity while preserving semantic coverage.\n\n\n**Implementation-level clarification:**\n\nOur dynamic block-sparse attention kernel supports both FA2 (A100) and FA3 (H100). For FA3, we use wgmma (m64n64k16) for dense compute, with sparse per-token key/value loading via offset indexing and rematerialization in shared memory—achieving high throughput without padding overhead."}, {"review_id": "oV4BIdx8Ez", "reviewer": "Reviewer_31M2", "summary": "The paper presents a method for accelerating video generation with sparse attention via semantic-aware permutation. It details techniques such as k-means clustering for semantic-aware permutation and centroid-based top-p selection for critical token identification. Experiments show that its pipeline achieves significant speedup while maintaining high-quality generation results, outperforming other sota methods.", "questions": "1. In Table 1, why do the performance metrics (like PSNR) vary significantly across different models(Hunyuan vs Wan2.1)? What underlying principles explain these performance differences?\n2. In the benchmarking results for the Hunyuan model in Table 1, the SVG[1]  also includes tests on the Hunyuan model. Specifically, the SVG+FP8 model achieves a speedup of 2.33x while maintaining similar quality metrics. This result appears comparable to the speedup metric of the author's model. Why did the authors choose not to include this result in this comparisons?\n3. Building upon Question 1: The paper mentions \"the lack of discussion and evaluation on whether the proposed methods can be extended to attention mechanisms other than DiTs.\" Given the noticeable differences in the method's effectiveness across models in Table 1, under what circumstances might the proposed methods fail? In other words, what are the capability boundaries and generality of the method?\n4. In Section 4.3, regarding \"customized attention kernels,\" the original text states, \"By gathering blocks with various sizes on on-chip memory from global memory, SAPAttn is able to utilize the accelerators.\" The implementation details here remain unclear. Could the authors provide a detailed explanation of the implementation?\n5. In the XAttention row of Table 1, there seems to be an extra \"s\" in the FLOPs column.\n\n\nReference:\n\n[1] Sparse videogen: Accelerating video diffusion transformers with spatial-temporal sparsity. In ICML, 2025.", "limitations": "Please refer to the Questions section.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Clear Structure and Readability. The paper is well-structured with a logical flow, making it easy to follow. The coherent organization of sections—from problem definition to methodology and experiments—facilitates smooth comprehension.\n2. Methodology is simple yet effective. By leveraging the k-means++ algorithm for Semantic-Aware Permutation and combining it with Centroid-Based Top-P Selection for critical token identification, the approach balances simplicity and efficacy. The approach is straightforward without complex theories, but works really well—showing how simple, practical solutions can make a big difference.\n3. It maintains high generation quality while delivering over 2×+ end-to-end speedup.\n\nWeaknesses:\n\nI do not find obvious weaknesses, but there are some questions. Please refer to the Questions section.", "was_revised": true, "final_justification": "Given that the authors have addressed all my concerns and demonstrated the robustness of the proposed pipeline—even in its FP8 version—I am revising my score to 5, recommending acceptance.", "rebuttal": "Dear Reviewer 31M2,\n\nWe thank the reviewer for the acknowledgment of our novel contributions and insightful questions. Below we respond to the questions.\n\n> Question 1: In Table 1, why do the performance metrics (like PSNR) vary significantly across different models(Hunyuan vs Wan2.1)? What underlying principles explain these performance differences?\n\n**Reply**: The performance variation (e.g., PSNR differences) across models ( Hunyuan and Wan2.1) arises from inherent numerical stability. In our experiments, we found that Hunyuan is relatively robust against precision variance while Wan2.1 is highly sensitive.\nFor instance, when evaluating the same dense attention using different backends (FlexAttention, FlashAttention, Torch SDPA), Wan2.1 exhibited PSNR as low as 27–28. However, HunyuanVideo exhibits ~33-34 PSNR despite no setup changes.\n\nTherefore, it is natural that SAPAttn achieves a lower PSNR on Wan compared with HunyuanVideo, due to its **sensitivity to numerical changes**. These differences are largely model-specific and reflect varying sensitivity to low-level numerical behaviors, which do not correlate with the performance of the methodology.\n\n> Question 2: In the benchmarking results for the Hunyuan model in Table 1, the SVG[1] also includes tests on the Hunyuan model. Specifically, the SVG+FP8 model achieves a speedup of 2.33x while maintaining similar quality metrics. This result appears comparable to the speedup metric of the author's model. Why did the authors choose not to include this result in these comparisons?\n\n**Reply**: We apply FP8 quantization in combination with SAPAttn and evaluate its impact on the Wan 2.1 I2V model. Compared with the version without FP8, our approach achieves a significant speedup with nearly lossless quality.\n\n|                | PSNR    | LPIPS  | Speedup  |\n|----------------|---------|--------|----------|\n| SAPAttn        | 26.562  | 0.138  | 1.58x    |\n| SAPAttn + FP8  | 26.545  | 0.141  | 1.76x    |\n\n\n> Question 3: Building upon Question 1: The paper mentions \"the lack of discussion and evaluation on whether the proposed methods can be extended to attention mechanisms other than DiTs.\" Given the noticeable differences in the method's effectiveness across models in Table 1, under what circumstances might the proposed methods fail? In other words, what are the capability boundaries and generality of the method?\n\n**Reply**: SAPAttn primarily accelerates attention computation, which dominates runtime in long-context scenarios due to its quadratic complexity. Its benefits are thus most pronounced when **context length is large**. \nIn addition, SAPAttn introduces a centroid cache to accelerate the k-means clustering process. which is especially effective in **iterative generation** tasks (e.g., diffusion or auto-regressive decoding) where context is reused across steps. In these cases, the centroid cache amortizes the clustering cost and contributes substantially to speedup. In settings where the context is used only once (e.g., prefill-only inference in LLMs), the lack of reuse limits the performance gains. \nThese observations outline the capability boundaries of SAPAttn: **it offers the greatest benefits in long-context, multi-forward generation tasks where attention dominates the runtime and centroid reuse is possible.**\n\n\n> Question 4: In Section 4.3, regarding \"customized attention kernels,\" the original text states, \"By gathering blocks with various sizes on on-chip memory from global memory, SAPAttn is able to utilize the accelerators.\" The implementation details here remain unclear. Could the authors provide a detailed explanation of the implementation?\n\n**Reply**: Our sparse attention with variable block sizes is built on FlashAttention3 (FA3), combining sparse loading and dense computation. To maximize hardware efficiency, SAPAttn uses the `wgmma` with an `m64n64k16` shape. For query tokens, we load contiguous tokens from the same cluster, which are naturally contiguous in memory after permutation. For key/value tokens, which may be scattered in global memory due to varying cluster sizes, SAPAttn uses per-token address offsets to perform sparse loading and stores them in shared memory in a contiguous layout. This enables efficient use of MMA instructions without the need for expensive key/value padding, leading to over 85% of the theoretical maximum performance, where the upper bound is estimated by multiplying the sparsity density with the runtime of the dense FlashAttention-3.\n\n> Question 5: In the XAttention row of Table 1, there seems to be an extra \"s\" in the FLOPs column.\n\n**Reply**: Thanks for pointing that out. We will remove it in our camera-ready version."}, {"review_id": "mtWVM0spVS", "reviewer": "Reviewer_8YJY", "summary": "Authors propose a method \"SAPAttn\" for accelerating inference speed of DiTs. The idea is quite simple: rearrange Q/K into smaller groups via k-means, and compute the attention with only the group mass. Video generation experiments are done with Wan2.1 and HunyuanVideo, with acceleration ratio of 1.89~2.30 without obvious quality deterioration.", "questions": "1) How much gain can we get if the proposed method is applied to DiT based image generation model? This is not discussed in the manuscript.\n2) The contribution of the work lies mostly in its custom implementation of \"sparse attention\", will the code be released? what's the schedule?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Most parts of the manuscript are clearly written. The idea is not hard to follow, novelty is good as respected to the standard of NeurIPS, experiments and results are also solid. Overall it's a pretty good work. Some minor concerns:\n1) The clustering of Q and K are done independently, in Line266 it says Cq=100 and Ck=500. Here lacks ablation study to clarify why Q/K clustering needs to be done independently, and how to determine the value of group numbers for each.\n2) Line269, it says \"We skip performing sparse attention on the first 30% denoising steps\", but didn't mention how many steps are actually used in the experiments.\n3) Line132~133, it says \"13% of the computations are sufficient to achieve an attention recall of 95%\", here needs clarification of the exact definitions of \"computations\" and \"attention recall\", and how the numbers are concluded.", "was_revised": true, "final_justification": "All my questions are addressed properly by the authors, I'd recommend to include the result on image generation task in the final manuscript. I raised my final score to 5.", "rebuttal": "Dear Reviewer 8YJY,\n\nThank you for your valuable questions. Below, we address each point raised.\n\n> Weakness 1: The clustering of Q and K are done independently, in Line266 it says Cq=100 and Ck=500. Here lacks ablation study to clarify why Q/K clustering needs to be done independently, and how to determine the value of group numbers for each.\n\n**Reply**: We conducted ablation experiments to test whether Q and K can share the same clustering strategy. Specifically, we evaluated three variants: applying Q clustering permutation ($\\pi_Q$) to both Q and K, applying K clustering permutation ($\\pi_K$) to both, and using clustering based on hidden states before QKV linear layer (i.e., shared QK embedding) for both sides ($\\pi_S$). As shown in the table below, all three variants led to **worse PSNR** even with more computation budget (i.e., density) compared to clustering Q and K independently.\n\n| Permutation used by Q  | Permutation used by K  | Density  | PSNR    |\n|------------------------|-------------------------|----------|---------|\n| $\\pi_Q$                | $\\pi_K$                 | 31.28    | 26.562  |\n| $\\pi_Q$                | $\\pi_Q$                 | 38.23    | 22.439  |\n| $\\pi_K$                | $\\pi_K$                 | 38.58    | 22.183  |\n| $\\pi_S$                | $\\pi_S$                 | 87.27    | 26.495  |\n\nTo further understand this, we compared the permutations of Q and K clustering and found that the permutation patterns differ substantially. Specifically, we calculate the Adjusted Rand Index (ARI) [1] value between Q clusters and K clusters, and the average ARI is 0.345, which is not very high. Therefore, clustering Q and K independently is necessary for preserving the expressiveness of attention.\n\nAs for the number of clusters, we conducted ablations in the table below, varying the Q/K cluster counts and reported both PSNR and end-to-end efficiency. We found that setting Q = 100 and K = 500 strikes a **sweet balance** between quality and efficiency. While increasing the number of clusters improves the generation quality, the efficiency may decrease due to the hardware layout requirements. As tensor core requires a fixed layout (e.g., input size 64 for m64n64k16) to **saturate computation**, each Q cluster must contain more than 64 tokens on average. Cluster counts beyond Q=100 or K=500 begin to hurt efficiency due to underutilized computation.\n\n| QC  | KC   | PSNR   | SSIM  | LPIPS | Speedup  |\n|-----|------|--------|-------|-------|----------|\n| 100 | 250  | 25.497 | 0.801 | 0.182 | 1.90x    |\n| 100 | 1000 | 26.276 | 0.825 | 0.159 | 1.71x    |\n| 50  | 500  | 22.561 | 0.742 | 0.258 | 1.90x    |\n| 200 | 500  | 26.213 | 0.82  | 0.157 | 1.78x    |\n| 400 | 500  | 26.488 | 0.868 | 0.132 | 1.25x    |\n| 100 | 500  | 26.128 | 0.816 | 0.169 | 1.89x    |\n\n\n[1] https://en.wikipedia.org/wiki/Rand_index\n\n\n> Weakness 2: Line269, it says \"We skip performing sparse attention on the first 30% denoising steps\", but didn't mention how many steps are actually used in the experiments.\n\n**Reply**: Thank you for pointing this out. Our experiments use a total of 50 denoising steps, with the first 15 steps (30%) skipped for sparse attention as a warm-up phase.\n\n> Weakness 3: Line132~133, it says \"13% of the computations are sufficient to achieve an attention recall of 95%\", here needs clarification of the exact definitions of \"computations\" and \"attention recall\", and how the numbers are concluded.\n\n**Reply**: Thank you for the suggestion. The “13% of the computations” refers to **attention density**, i.e., the percentage of attention map retained in the sparse attention. “Attention recall” measures the proportion of the total attention score (after softmax) captured by the selected sparse tokens.\n\nIn this case, retaining just 13% of token pairs preserves 95% of the total attention weight, indicating that our selection effectively focuses on the **most important attention regions**. We will revise the text to make these definitions explicit.\n\n> Question 1: How much gain can we get if the proposed method is applied to DiT based image generation model? This is not discussed in the manuscript.\n\n**Reply**: Thank you for raising this valuable point. We experimented with applying SAPAttn to the DiT-based image model Flux.1-dev, using a top-p threshold of 0.9 and a warmup ratio of 15%. We observed a PSNR of 25.3, indicating that SAPAttn preserves image generation quality.\nIn terms of acceleration, SAPAttn achieved an end-to-end speedup of 1.05x. The gain is modest because Flux.1-dev operates at a context length of 4096, where attention accounts for only 15.6% of total runtime. As such, SAPAttn yields more substantial benefits in models with **longer context** lengths. This suggests that SAPAttn is particularly effective for video generation models or other settings where attention becomes a dominant computational bottleneck.\n\n> Question 2: The contribution of the work lies mostly in its custom implementation of \"sparse attention\", will the code be released? what's the schedule?\n\n**Reply**: Yes. We have implemented both FlashAttention v2 (for A100) and FlashAttention v3 (for H100) compatible versions of dynamic block sparse attention. This core component has already been open-sourced and integrated into the FlashInfer library. The remaining components of our system will also be open-sourced shortly to facilitate further research and practical adoption."}]

Author rebuttal:

Meta-review: This paper proposed SAPAttn, a training-free framework that maximizes identification accuracy and minimizes computation waste, achieving a Pareto frontier trade-off between generation quality and efficiency for video diffusion transformers. 

Paper got 2 Borderline accept, 2 Accept rating.

Before rebuttal, reviewers thought the strengths of this paper are:
1) method is novel (Reviewer QGvt, 8YJY )
2) method shows improvement. (Reviewer QGvt, PvVz, 31M2, 8YJY)
3) proposed method is training-free. (Reviewer PvVz)
4) paper is well written (Reviewer 31M2, 8YJY)

Weaknesses are:
1) need more experiments (Reviewer QGvt)
2) more clarification and reason for some of the design and experiments (Reviewer QGvt, PvVz, 31M2, 8YJY)

After the rebuttal, all the reviewers thought authors addressed their concerns and gave accept or borderline accept rating. Given these AC decided to accept this paper.
Decision: Accept (spotlight)