FPSAttention: Training-Aware FP8 and Sparsity Co-Design for Fast Video Diffusion
Authors: Akide Liu
Zeyu Zhang
Zhexin Li
Xuehai Bai
Yuanjie Xing
Yizeng Han
Jiasheng Tang
Jichao Wu
Mingyang Yang
Weihua Chen
Jiahao He
Yuanyu He
Fan Wang
Gholamreza Haffari
Bohan Zhuang
Venue: NeurIPS 2025 spotlight ()
Keywords: Video Generative Model
Efficiency
Quantization
Sparsity
Primary area: deep_learning
TL;DR: FPSAttention is a training-aware FP8 quantization and sparsity co-design for video diffusion models that achieves up to 4.96× speedup without quality loss by aligning 3D tile granularity, denoising-step adaptation, and hardware-efficient kernels.

Abstract: Diffusion generative models have become the standard for producing high-quality, coherent video content, yet their slow inference speeds and high computational demands hinder practical deployment. Although both quantization and sparsity can independently accelerate inference while maintaining generation quality, naively combining these techniques in existing training-free approaches leads to significant performance degradation, as they fail to achieve proper joint optimization.
We introduce FPSAttention, a novel training-aware co-design of FP8 quantization and Sparsity for video generation, with a focus on the 3D bi-directional attention mechanism. Our approach features three key innovations: 1) A unified 3D tile-wise granularity that simultaneously supports both quantization and sparsity. 2) A denoising step-aware strategy that adapts to the noise schedule, addressing the strong correlation between quantization/sparsity errors and denoising steps. 3) A native, hardware-friendly kernel that leverages FlashAttention and is implemented with optimized Hopper architecture features, enabling highly efficient execution.
Trained on Wan2.1's 1.3B and 14B models and evaluated on the vBench benchmark, FPSAttention achieves a 7.09$\times$ kernel speedup for attention operations and a 4.96$\times$ end-to-end speedup for video generation compared to the BF16 baseline at 720p resolution—without sacrificing generation quality.

FPSAttention: Training-Aware FP8 and Sparsity Co-Design for Fast Video Diffusion
Akide Liu"
Zeyu Zhang"
Zhexin Lif24
Xuehai Bai
Yuanjie Xing
Yizeng Han?
Jiasheng Tang 124
Jichao Wu
Mingyang Yang
Weihua Chen²
Jiahao He²
Yuanyu He
Fan Wang?
Gholamreza Haffari
Bohan Zhuang
Monash University
DAMO Academy, Alibaba Group ZIIP Lab, Zhejiang University *Hupan Lab
CCo-first authors. *Co-second authors. *Project leads.
Corresponding authors: fan. w@al cealibaba-inc. com, bohan. www.ng@gmail..com
Abstract
Diffusion generative models have become standard producing high-quality, coherent video content, yet their slow inference speeds and high computational demands hinder practical deployment. Although both quantization and sparsity can independently accelerate inference while maintaining generation quality, naively combining these techniques in existing training-fiet approaches leads to significant performance degradation, they fail to achieve proper joint optimization. We introduce FPSAttention. a novel training- aware co-design FP8 quantization and Sparsity for video generation, with a focus on the 3D bi-directional attention mechanism. Our approach features three key innovations: wise granularity that simultaneously supports both quantization and sparsity. 2) A denoising step-aware strategy that adapts to the noise schedule, addressing the strong correlation between quantization/sparsiyy emror and denoising steps. 3) A native, saldware-friendll kernel that leverages FlashAttention and is implemented with optimized Hopper architecture features, enabling highly efficient execution. Trained on Wan2. I's 1.3B and 14B models and evaluated on the VBench bench- mark, FPSAttention achieves 7.09x kernel speedup for attention operations and a 4.96x end-to-end speedup for video generation compared to the BF16 baseline 720p resolution without sacrificing generation quality. Project page: /ttps::/ffs ziplab.co
A unified 3D tile-
1 Introduction
Diffusion models have revolutionized AI through breakthrough im- and advancing into complex video Diffusion Transformers (DiTs) (28) now enable
Kernel Speedup
F2F
age synthesis 39
6
34
Method
Speedup
generation 40
45
BF16
1.00x
1.00x
efficient, high-quality synthesis 23) powering billion-parametar models like Wan2. [37 that produce coherent, long-duration, high-
FP8
1.84
1.26x
STA
5.15x
3.60x
fidelity videos.
FPSAtten
7.09
4.96x
Despite progress, crippling computational demands persist 32 (i) iterative reverse-diffusion requiring hundreds of steps. and (ii) quadratic-compleiii the number of tokens) 22 17 37]. Particularly, the computational burden of attention becomes prohibitive for high-resolution, long- duration videos, often consuming 70% of inference time [5T 43
Table 1: Efficiency com-
parison of the BF16 base- line, FP8 quantization, STA sparse attention, and our FPSAttention method Wan2. 1-n2.1114B at 720p resolu- tion an NVIDIA H20 GPU. We report both kernel-level and end-to-end speedups rel- ative to the BF16 baseline.
peatio-eemporal attention (O(N2), N denotes
39th Conference on Neural Information Processing Systems (NcurIPS 2025),

---

(a) Training Free Quantization
Video DIT Block
(c) Our Fused Quantization Sparsity
FFN
Quart 60
Cuant.
Layer Norm
Sparso
TAYOO
Gross Altordion (Optional)
Sc0
auods
TIPP
Layer Norm
Sparse
High-predision
Full Attention
Low-precision Token
Spana
Skipped Token
Layer Norm
Trainalle/ Frozen
(b) Training Free Sparsity
Figure 1: Comparing previous vaining-free quantization (a) and training-fres sparsity (b) approaches reveals substantial accuracy degradation and lack of compatibility when used independently. In contrast, our FPSAttenti on framework (c) integrates low-precision and sparse patterns in a single training process, yielding near-zero accuracy loss and seamless deployment.
35.. For instance, Wan2. requires approximately 2.5 hours on an NVIDIA H20 GPU to generate 5s video.
To address the efficiency challenge, numerous acceleration methodologies have been proposed 7 24 33 among which quantization and sparsity have emerged as predominant tech-
niques 38 255. Quantization reduces numerical precision (e.g.,
FP32-INTS/FPS) thereby decreasing the memory footprint and enabling faster computations (Figure (a)). The recent post-training quantization (PTQ) method SageAttention [49] quantizes attention modules into INTS with calibration strate- gies, providing moderate acceleration with downgraded gen-
eration quality. Compared to INT8, the emerging format FP8
quantization offers wider dynamic range 26), facilitating both
training and inference [8] 31. Nevertheless, training-free FP8 quantization, despite its theoretical advantages, introduces sig-
nificant quantization errors that degrade model performance Apart from quantization, sparsity techniques address the quadratic computational complexity of 3D full attention by se- lectively skipping computations, as illustrated in Figure (b). Representatively. Sparse-VideoGer
49
Figure 2: Comparison video gen- eration results. Top: Training Free FP8 STA. Bottom: Our Training- Aware FPSAttention. Click the image to play the video via Acrobat Reader.
VideoGen [43] implements per-head spatial-temporal masks aligned with GPU blocks, SpargeAttn [50 employs a two-stage filtering mechanism, and Sliding Tile Attention (STA) (51) leverages local 3D sliding windows with kernel-level optimizations.
To enjoy the benefits of both worlds, straightforward strategy jointly applying FPS quantization and sparsity. However, a naive combination presents significant challenges, as quantization errors can be magnified when combined with sparsity mechanisms 44,, as shown Figure 2 Intuitively, sparsity techniques prioritize token retention with high- magnitude attention scores, while quantization disproportionately introduces errors in these high- magnitude values. This intrinsic tension necessitates holistic approaches that consider both techniques simultaneously, potentially framing sparsification a specialized form of O-bit quantization to achieve optimal balance between efficiency and generation quality (27].
Furthermore, existing approaches largely overlook training aware joint optimization of quantization and sparsity, creating a substantial training-inferenn gap. Our empirical analysis (Section (33) reveals that diffusion models can tolerate and even correct for hardware-friendly tile-wise errors. The error resilience is particularly pronounced when the model is aware of approximations via ouantization-awal training (QAT), ensuring consistent performance inference time.

---

In this paper, we introduce FPSAttention, as shown in Figure a novel training-ane aware co-design framework that synergistically integrates FP8 quantization and structured sparsity for 3D attention in video DiTs,
FPSAttention proposes three key innovations:
Unifying Tile-wise Operations: Implementing a 3D tile-wise granularity for both FP8 quantization and block sparsity (section 3.2). which directly aligns with efficient hardware execution patterns (c.g., GPU Tensor Cores) and forms the basis for structured acceleration.
Denoising Step-Aware Scheduling: Introducing an adaptive strategy (section 3.3 that dynamically adjusts quantization and sparsity granularity according the varying sensitivity and corrective capacity of the model across different denoising timesteps.
hurdware-Optimize Kernel Design: Developing native, high-performn sesh-performanc kernel (section 3.4 leveraging features like FlashAttention and NVIDIA Hopper architecture optimizations to translate theoretical FLOPs reduction into tangible wall-clock speedups.
As demonstrated in Table T by training on Wan2.1's 1.3B and 14B models and evaluating the vBench benchmark, FPSAttention achieves a 7.09x kernel speedup for attention operations and a 4.96X end-to-end speedup for video generation compared to the BF16 baseline, all without sacrificing generation quality, significantly outperforming approaches that apply quantization (1.84x kernel speedup) or sparsity (5.15x kernel speedup) independently. Our work not only provides practical solution for accelerating video diffusion but also offers a new perspective on the robustness of diffusion models to aggressive, structured compression.
2 Related Work
Quantization for video generation models. The computational expense of video generation models, particularly Diffusion Transformers (DiTs) [17. driven by iterative sampling [ and quadratic attention complexity [35), necessitates model quantization techniques
32
Post-Training
Quantization (PTQ) has been explored for its efficiency 10]
20
13
54
[5|
however, applying
PTQ to video DiTs presents unique challenges beyond standard image models 42
3
53
Temporal
variability of activation statistics across denoising steps [14 has prompted PTQ methodologies to implement time-step-wee wise calibration [46), adaptive quantization, and dynamic smoothing techniques 30 Recent work has evaluated these techniques on standardized benchmarks (e.g., VBench (16)), assessing temporal consistency alongside perceptual quality. While PTQ approaches show promising results, Quantization-waan Training (QAT) for video diffusion models remains largely unexplored. Our work addresses this gap by introducing FP8 QAT framework that jointly optimizes quantization and sparsity, enabling efficient video generation while maintaining visual fidelity.
Sparse attention for video generation models. Recent advancements in sparse video generation and efficient attention mechanisms have improved memory utilization and computational efficiency. Sparse VideoGen 43 leverages sparsely sampled motion priors to produce realistic videos while reducing temporal redundancy. Efficient attention mechanisms have proven crucial for handling long-range dependencies in video data. Sliding Tile Attention [1 introduces tiled sparse attention mechanism for modeling spatial-temporal correlations, while SpargeAttn (50) proposes progressive sparsification by selectively pruning attention tokens based importance scores. DiTFastAttn 2 accelerates attention computation by dynamically filtering irrelevant patches, achieving significant speedups without compromising quality. These approaches illustrate the trend of combining structured sparsity with content-aware selection for scalable video generation. However, these methods are typically limited inference-time acceleration, lack integration with model training procedures, and are not fully compatible with quantization techniques, creating challenges for developing holistically efficient video generation frameworks.
3 Method
Our technique integrates algorithmic innovation with hardware-conscing kernel optimization to enhance the efficiency video DiT's. This section begins by establishing fundamental concepts essen- tial to methodology. Subsequently, we introduce the architecture of our proposed FPSAttention.
3

---

Diffusion Timestep
Full Precision Block
Large window for carly my
Smail vindoo for mibdle
Midile size window for latter styp
① Granularity Selection
Low Precision Block
Skipped Block
Quantication
2 Quantication
Quanitationn
Sparsity
Sparsity
Spursity
11
Quant. Sparse Attention
Figure 3: Overview FPSAttention. (I) Our approach synergistically optimizes joint quantization and sparsity patterns within the attention mechanism for efficient video generation. (2) We introduce a novel denoising step-aware strategy that dynamically adapts the granularity throughout the diffusion process, balancing computational efficiency and perceptual fidelity. Empirical observations shown in Figure (3) A fused hardware- hardware-friendly kernel is applied for attention operations.
detailing its two primary algorithmic contributions: unified tile-wise quantization and sparse at- tention mechanism, and denoising step-aware strategy for dynamic adaptation of quantization and kernel implementation that
sparsity hyperparameters. Finally, outline our hardware-optimine plays a crucial role translating theoretical computational savings into practical efficiency gains. Figure provides high-level conceptual overview of our FPSAttention framework.
3.1 Background
This subsection establishes the two foundational techniques that underpin our methodology: 8-bit floating-point (FP8) quantization and Sliding Tile Attention (STA).
FP8 quantization. Video Diffusion Transformers (DiTs) process L = x H x W spatiotemporal tokens, where T. H, and W represent temporal frames, height, and width dimensions. To reduce memory bandwidth requirements for activation tensors, FP8 quantization approximates each value X,, using an 8-bit floating-pol maaingg-point representation. Unlike INT8 quantization, which maps continuous values to scaled integer grid, FP8 conversion preserves the Hoating-poiit point nature by utilizing dedicated sign, exponent, and mantissa bits (in formats such as E4M3 or E5M2).
The FP8 conversion employs a scaling factor 8n for each tile of values g to map the original values into the representable dynamic range of FP8:
XFPs(X..j sn) = Cequantio(FPP__oovvete
sy))/sg
To enhance approximation accuracy, tile-wise FP8 quantization employs per-tile scaling factors {sg} that minimize quantization error within each specific tile. This approach preserves attention head-specific and frame-specific activation dynamics while typically reducing data size by half (e.g.. from 16-bit to 8-bit). The result is a theoretical 2x reduction in memory bandwidth requirements, with further effective improvements achievable through specialized FP8 hardware acceleration.
Sliding Tile Attention (STA). Standard attention operations on N L tokens with feature dimension d incur a computational complexity of (NNdd) creating significant bottleneck for high-resolution video generation. STA addresses this challenge by partitioning the 3D token space into M non- overlapping tiles (T.) of dimensions (T,, Tú, Tw))

---

The key innovation of STA is its locality- locality-based based attention mechanism: each query tile u attends exclusively to key tiles within a local neighborhood W(u). defined by the distance constraint:
W(u) { ||c. colloo x
W2
W4
W
colloo
27.
2T
2T
where (We, Wh, w denote the window dimensions measured tile units, and Cus cy are the centers of tiles u and This constraint effectively replaces full attention with a tile-wise masked attention:
Softmax (Qg /V),, if token & is in a tile To where E W(u),
(3)
-00,
otherwise.
where Q and K are queries and keys. STA generates M x [W(u) dense attention blocks that are compatible with optimized implementations such as FlashAttention []. This design provides substan- tial speedup by replacing the irregular sparse patterns of token-wise sliding window attention with structured tile-based computations that align well with GPU memory hierarchies. As demonstrated in [SI]. this approach can accelerate attention by 2.8-17 over FlashAttention-2 and 1.6-10x over PlashAttentionn3122 for video generation tasks.
3.2 Joint Tile-wise FP8 Sparse Attention
Prane
Phomer
Frow
Building upon FP8 quantization and tiled atten- tion techniques, introduce FPSAttention, a Joint Tile-wise FP8 Quantization and Sparse Attention mechanism that synergistically opti- mizes computational efficiency and perceptual accuracy in video DiTs.
Platton
Per token
Per channel
Our tile-wise granularity approach (Figure last row) is motivated by three primary consid- erations. First, it offers an optimal accuracy- efficiency trade-off compared to conventional methods (per-token, per-channel, per-group; Fig- ure 4 first three rows) that often fail align with underlying hardware architectures. While per-group quantization provides a reasonable balance, it frequently overlooks GPU compute tile patterns, thereby reducing hardware utiliza- tion efficiency. Second, our approach maintains full compatibility with the STA sparsity design, allowing seamless integration of quantization and sparsity optimizations at matching granularity. Third, our tile-wise design exhibits superior hardware compatibility, aligning precisely with compute tiles in optimized kernels such as Altsttttention, which enables direct translation of theoretical computational savings into practical speedups.
Per group (Group size
Per (Ours)
Figure 4: Quantization granularities: per-token, per-channel, per-group, and our per 3D-tile, which aligns with hardware compute patterns.
The FPSAttention mechanism processes neural activations through systematic workflow: (1) organizing query (Q) and key (K) activations into contiguous 3D tiles aligned with GPU cache layouts for enhanced data locality; (2) quantizing each tile FP8 precision with locally optimized scale factor; (3) enforcing tile-grand ail-granularity sparse attention patterns, leveraging spatial locality and low-bit arithmetic; and (4) dequantizing the aggregated attention output to higher precision (BF16/FP16).
Tile-wise FP8 quantization for Q and K. The matrices Q, E R2xd are partitioned along the sequence dimension L into non-overlapping tiles (TT) of dimensions (Tr, Th, Tw.) For each tile T.,.
we compute separate scaling factors &8 and o to map their values optimally to the FP8 representable range via
1Q, max:
max (K.j)TT
(i.J)TT,
where MFPs max is the maximum representable magnitude in FP8 and bounded by specific format. Then each element independently quantized:
Q. FP8 +9), k
This per-tile scaling strategy minimizes quantization error for both Q and K independently. preserving attention dynamics more effectively than global scaling approaches. When combined with the STA
5

---

Codme Geclurly Jene Sendiing Bops Highe belee)
Eerean Dorutiding flom Crever belter)
DES Geren Doveliing from Soweer battted
(a) Cosine similarity.
(b) MSE loss.
(c) SNR loss.
Figure 5: Joint quantization and sparsity error patterns across denoising steps. Blue: token-level granularity; orange: our 3D tile-wise granularity with sparse attention. Key insight: early/ltee steps tolerate coarser quantization and higher sparsity, while intermediate steps require finer granularity and denser attention. Our FPSAttention (green) closely approximates biighet-grrnulariin validating our adaptive scheduling strategy. All measurements from inference are with identical prompts. Performance gaps primarily stem from FPS quantization rather than sparsity constraints.
methods,
formulation, the attention weights P are computed using the quantized Q and K, following Eq. 3 This formulation enforces regular, block-sparse pattern (Figure that efficiently maps to modern GPU compute architectures.
Channel-wise FP8 quantization of V and tensor-wise FP8 quantization of P. For the value matrix V E IR Lxd, a chan FP8 quantization approach employed. For each channel, we compute the scaling factor sy to map the values optimally to the FP8 representable range: V = g. VV. //FF/ max- Each element in Vi is subsequently quantized as Vj = =FF(BVVV; s We observe that keeping the fine granularity for V is critical for the performance. Following SageAttention2
48) we use fixed scalar 448 to quantize P. obtaining P in FP8.
Aggregation and dequantization. The attention output is computed as X p= PV. This low-precision output is then dequantized to higher precision (BF16/FP16) maintain computational stability in subsequent layers.
By quantizing Q and K tiles independently while using channel-wise quantization for V, the FPSAttention mechanism captures fine-grained statistical properties of each attention component while maintaining optimal alignment with GPU memory hierarchies. Furthermore, the eile-contrrained sparse attention pattern creates M x [W) dense, structured attention blocks that avoid the in- efficiencies of unstructured sparsity patterns. This approach enables direct wall-clock speedups during execution by maximizing hardware utilization and minimizing memory access overhead, as demonstrated in our experimental results.
3.3 Denoising Step-aware Quantization and Sparsity Strategy
The mechanism described in Section 3.2 employs uniform quantization and sparsity across denoising steps. However, as illustrated in Fig. 5] video DiTs exhibit varying sensitivity to numerical precision and sparsity levels throughout the diffusion process. Specifically, early and late denoising steps demonstrate greater tolerance to coarser quantization and higher sparsity, whereas intermediate steps demand finer numerical precision and lower sparsity. This also suggests that diffusion models can intrinsically correct the approximation emrors of attention, which motivates our training-aware scheme to mitigate the training-inferenn gap. Based on these observations, propose an adaptive, denoising step-aware compression schedule, for both training and inference. We adjust the quantization granularity g() and sparsity window size W (t)) based the denoising step t.
Piecewise schedule for quantization and sparsity. For denoising steps, we partition the process into three regimes using thresholds 4 = D and (2 D (0 < <a2 < 1), each associated with different quantization tile sizes g(I) and sparsity window sizes ((). Smaller g(!) corresponds to a finer quantization granularity, while larger (t) indicates denser attention pattern.

---

We show quantization and sparsity schedule as following. We define a time-dependent hyperparameter vector S(t) (g(t), (t)). which is governed by:
Becure: W sppasel:
S f (Early-Denoisil) Steps);
S(t) = Jhnne Weeuse]
t1 < S t2 (Mid-Denoising Steps),
(6)
Ommermediate W medium demistyy
>> t2 (Late-Denoising Steps),
Dintermediate > glie and W densee > W Wmsdiiu density > W Spparse ensuring the finest quantization granularity and densest attention patterns during these mid-denoising steps, as illustrated in Figure 5
with Genarse >
These hyperparameters are selected at inference time to match the model's varying tolerance to quantization and sparsity across different denoising stages, and then transferred to the training to avoid the prohibitive computational overhead. During training, this configuration allows the model to adaptively compensate for joint quantization-spee -spatisioosssstttt errors, leading to satisfiable stability and convergence throughout the training process, as shown in Figure
3.4 Hardware-Optimized Kernel Design
Our algorithmic designs are complemented by a hardware-optimized kernel implementation for maximum practical efficiency. The implementation addresses several key aspects: memory access coalescing through structured operations that enable efficient GPU memory loads/stores with tiling support; maximized parallelism via tile-wise operations that process independent tiles concurrently; exploitation of dedicated acceleration units such as Tensor Cores on NVIDIA Hopper/Ade architec- tures for mixedd-recision and FP8 computations; and operation fusion that combines multiple logical steps (attention, sparsity, dequantization) into single triton kernels, significantly reducing overhead and memory traffic while maintaining high tensor core utilization and computational intensity.
4 Experiments
Implementation details. We implemented our proposed framework on the Wan2. (36) architecture (1.3B and 14B variants), preserving the original model structure while introducing FPSAttention joint FP8 quantization and structured sparsity. The quantization schemes and sparsity patterns were applied across attention mechanisms using score mod and mask mod functions via FlexAttention Fused kernels were compiled using Triton to accelerate inference on Hopper GPUs. The models were trained on high-quality video data (480px I 6fps We trained FPSAttention on Wan2.1-14h using 64 nodes with 8 H20 GPUs for 7 days. For detailed hardware specifications, training procedures, dataset preparation, evaluation protocols, and baseline comparisons, please refer to Appendix.
Evaluation protocol. We evaluate the our method on the public video dataset, VBench [17]. We following the common practice [55] [9 to sample 5 videos per evaluation prompts defined in the VBench dataset, and assess the video generation quality across 16 VBench dimensions. We also report Peak Signal-to-Noise Ratio (PSNR) [2] Structural Similarity Index (SSIM) and Learned
Perceptual Image Patch Similarity (LPIPS)
52 metrics.
4.1 Main Results
Quality evaluation. We compare FPSAttention with several state-of-the-art optimization meth- ods, as shown in Table Our baselines include sparsity-based approaches
43
and STA), quantization methods Attentnttn (4)), and hybrid approaches (SpargeAtten [50]
a training-free method that jointly applies attention sparsification and activation quantization). As
demonstrated in Table 2 FPSAttention achieves superior performance across all quality metrics. Particularly notable is the average PSNR of 25.74353 on the Wam2.1-14B model, significantly outper- forming all baseline methods. This objective metric confirms FPSAttention: 's ability to generate videos with exceptional fidelity to reference images. Furthermore, FPSAttention maintains excel- lent performance on perceptual metrics, with high Video Quality (0.7103) and strong spatial- -emporll consistency (0.9435) on the VBench evaluation. Interestingly, after joint training, FPSAttent i exhibits a slight increase VBench scores, while [7 51 also demonstrate performance improve- ments when trained with structured These results validate that our joint sparsity and quantization approach preserves visual quality while substantially improving computational efficiency. Visual examples in Figure 6 further illustrate that
ptrrsityppoeentallyy driven by the inductive bias of locality.
7

---

Table 2: Quality and efficiency benchmarking results. We reproduce the results of the baseline methods from the original papers. Note that VBench results here may differ from official results due to the randomness generated samples and prompt extensions. The quality and efficiency evaluation is based on 480p videos. Specifically, indicates the speedups via 720p with longer sequence length.
Quality
Efficiency
Method
PSNR
SSIM
LPIPS
ImageQual
SubConsist
FLOPS
Lasency
Speedup
Speedup'
W-.2,1-1.38'
0.6708
0.9536
71.52 PELOPS
2716
1.00x
SageAltcation
20.18990
0.78241
0.18811
0.6699
0,9453
37.61 PELOPS
141s
1,91A
ApargeAtten
17.12979
0.72628
0.8082
SparseVidextion
0,78891
STA
18.78546
0,76335
0.23187
0.6626
0,8992
31.78 PELOPS
143s
1,8%%
Ours Quant
20.99712
0,79620
0.15114
0.6798
0,9458
32.01 PFLOPS
144s
1.88x
Ours Quant + Sparse
21,35417
0,80835
0.15398
0.7103
32.01 PELOPS
110s
2.4%%
W-n2.1-14BB
0.6715
637.52 PFLOPS
Augenitentoon
24.33985
0.82283
0.15607
0.6724
0,9530
PHI OPS
2.01x
SporgeAtten
21,38291
0.81452
0,21723
0.6350
0,9173
339,30 PHLOPS
734s
1.77%
2.12%
SparseVidextien
23.52881
0,80113
0.17052
0.6868
0,9489
259,7 79
PELOPS
6136
2.12%
3.13%
STA
0.19283
0.6577
0,9530
3.60x
Ours Quant Sparse
25.74553
088171
0,07610
PHLOPS
Baseline, Wan 2.1 1.3B, 1x E2E speedup
Our FPSAttention. 4.96x E2E speedup
Figure 6: Examples of generated videos by FPSAttention and the Wan2.1-1.3B baseline. We show- case from five different aspects. FPSAttention achieves 4.96x E2E speedup, while maintaining lossless visual quality. Please click the image to play the video clip via Acrobat Reader.
FPSAttention consistently outperforms baseline methods while maintaining quality comparable to the original model.
Test-time efficiency. We evaluate computational efficiency across both 1.3B and 14B parameter variants the 1 model, as reported in Table For the 1.3B parameter model, FPSAttention 480p videos while maintaining superior quality metrics. More
achieves up to 2.45x speedup impressively, when tested on the larger 14B parameter model with 720p videos (indicated by the table), FPSAttention achieves substantial 4.96x end-to-end speedups compared to the baseline.
Table 3: Effect of different tile sizes model performance on Wan2.1 1.3B. We evaluate various tile size combinations for temporal (t), height (h), and width (w) di- mensions.
Table 4: Impact of sparsity window dimensions on model performance and computational efficiency. We evaluate various combinations temporal (t), height (h), and width (w) window sizes and measure infer- ence kernel speedup.
Tile Size
Metrics SSIMI
Window Size
Metrics
w
PSNR1
LPIPS
w
PSNR
SSIM
LPIPS
Speedupt
4
4
19.87452
0.77634
0.16521
3
3
1
19,23465
0.76128
0.17251
3.24x
8
8
12358
0.78147
0.15982
5
10
19.94731
0.77926
0.16327
3.07x
12
16
16
20.45621
0.78932
0.15743
6
20.12482
0.78341
0.16014
1.69x
32
32
20.99712
0.79820
0.15114
20.45621
0.78932
0.15743
5.1

---

Training Loss Comparison
Experiments
#PSAttantion: Initial Loss: 0.2020
FPSAlbention
Initial Loss: 0.1913
Baseline
Final Loss: 0.0959
Final Loss: 0.0831
Lous:
Lose:
Improvemen::
Improvement:
FP FPSAttention Min: 0.0875
1000
2,000
5000
4000
5,000
Stop
Figure 7: Training loss comparison between baseline Wan2.1 1.3B, and our joint FP8 quantization with structured sparsity. The FPSAttention shows slightly higher loss initially but stabilizes with comparable final performance.
4.2 Ablation Study
In this section, we conduct a comprehensive ablation study to analyze the effects of key components in FPSAttention. We investigate the different tile size selection for quantization and sparsity, and the relations to the hardware awareness. Additionally, we investigate how sparsity window dimensions affect model performance and computational efficiency. We also study the training stability to verify whether our FPSAttention on training dynamics highly aligns the baseline trend.
Effect of tile size for quantization and sparsity. We investigate the impact of joint quantization and sparsity granularity by varying tile sizes along temporal (0, height (h), and width (w) dimensions, with results presented in Table 3 Our experiments demonstrate that the largest tile configuration (24,32,32) achieves optimal performance with PSNR 20.99712, SSIM 0.79820, and LPIPS 0.15114. However, we observe minimal performance differences between configurations (6,8,8), (12,16,16). and (24,32,32). Based on these findings, we implement a scheduled approach, using different tile sizes for early, mid, and late denoising steps, respectively. We emphasize the hardware-friendd configuration of (6,8,8) yields the best throughput as aligns with flash attention block size design and is optimized for the Hopper architecture. Notably, unbalanced tile sizes, such as (3,4,4), lead to significant performance degradation, this configuration also misalign with flash attention block size leads to inefficiency.
Effect of sparsity window dimensions. The relationship between sparsity window dimensions, inference speed, and model performance is analyzed in Table 4 Our results demonstrate that configuration (6,6,1) using temporal window of height window of 6, and width window of provides an optimal balance, achieving a substantial 5.16x kernel speed-up while maintaining high visual quality. This suggests that while reducing the temporal and spatial window sizes improves efficiency, there exists threshold beyond which visual quality deteriorates significantly.
Training stability. Joint FP8 quantization and structured sparsity initially increases training loss by 15% compared to full-precision Wan2.1 baseline (Figure 7 We mitigate these challenges through adaptive learning rate scheduling and gradient accumulation techniques. After 2,000 steps, loss convergence trajectories become nearly identical (<2% difference), confirming that our FP8 sparse attention preserves critical information pathways despite bitwidth and sparsity constraints.
Conclusion and Future Work
In this paper, we have introduced FPSAttention, a module jointly optimizing FP8 quantization and structured sparsity for video diffusion models. Through unified 3D tile-wise granularity, denoising step-aware adaptation, and hardware-friendly kernel implementation, our approach achieves up to 7.09 kernel speedup and 4.96x end-to-end acceleration without compromising generation quality. The tile-aligned approach ensures quantization and sparsity work synergistically, while step-

---

aware scheduling adapts compression hyperparameters to varying sensitivity across diffusion phases. Despite these results, our approach has limitations: performs best on FP8-supporting hardware. requires additional training resources, and introduces certain hyperparameters. We currently validate FPSAttention on Wan2.1 due resource constraints. Future work will focus on broadening the applicability by generalizing beyond specific architectures (e.g. Hunyuan [[7] based on MMDiT (9). We also aim to refine training resource requirements and hyperparameter management, and extend these co-design principles beyond attention mechanisms to other model components.
References
[1] Andreas Blattmann al. Align your latents: High-resolution video synthesis with latent diffusion models. 2023.
[2] Jintao Chen, Chendong Xiang, Haofeng Huang, Jia Wei, Haocheng Xi, Jun Zhu, and Jianfei Chen. Ditfastattn: Attention compression for diffusion transformer models, In Advances in Neural Information Processing Systems (NeurIPS), 2024.
[3] Lei Chen, Yuan Meng, Chen Tang, Xinzhu Ma, Jingyan Jiang. Xin Wang, Zhi Wang, and Wenwu Zhu. Q-dit: Accurate post-training quantization for diffusion transformers. arXiv preprint arXiv:2406.17343 2024.
[4] Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, arXiv preprint arXiv:2307.0869/ 2023.
[5] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems, 35:16344-16359 2022.
[6] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances neural information processing systems, 34:8780-8794, 2021.
[7] Zihan Ding, Chi Jin, Difan Liu, Haitian Zheng, Krishna Kumar Singh, Qiang Zhang, Yan Kang, Zhe Lin, and Yuchen Liu. Dollar: Few-step video generation via distillation and latent reward optimization. arXiv preprint arXiv:2412.1 5689, 2024.
[8] Juechu Dong, Boyuan Feng, Driss Guessous, Yanbo Liang, and Horace He. Flex attention: A pro- gramming model for generating optimized attention kernels. arXiv preprint arXiv:2412.054996, 2024.
(9] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transform- ers for high-resolution image synthesis. In ICML, 2024.
[10] Yefei He, Luping Liu, Jing Liu, Weijia Wu, Hong Zhou, and Bohan Zhuang. Ptgd: Accurate post-training quantization for diffusion models. arXiv preprint arXiv:2305. 2023.
[11) Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances Neural Information Processing Systems, volume 33, pages 68510 2020.
[12] Alain Hore and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In 2010 20th international conference on pattern recognition, pages 2366-2369. IEEE, 2010.
[13] Haocheng Huang, Jiaxin Chen, Jinyang Guo, Ruiyi Zhan, and Yunhong Wang. Teaq-dm: Timestep-channel adaptive quantization for diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, number 16, pages 17404-17412, 2025.
[14] Haocheng Huang, Jiaxin Chen, Jinyang Guo. Ruiyi Zhan, and Yunhong Wang. Teaq-dm: Timestep-channed adaptive quantization for diffusion models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pages 17404-1747 2025.
[15] Yushi Huang, Ruihao Gong, Jing Liu, Tianlong Chen, and Xianglong Liu. Tfmq-dm: Temporal feature maintenance quantization for diffusion models, 2024.
10

---

[16] Ziqi Huang, Yinan He, Jiashuo Yu. Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 21807-21818, 2024.
[17] Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03600 2024.
[18] Andrey Kuzmin, Mart Van Baalen, Yuwei Ren, Markus Nagel, Jorn Peters, and Tijmen Blankevoort. Fp8 quantization: The power of the exponent. Advances in Neural Information Processing Systems, 35:14651-11663 2022.
[19] Muyang Li, Tianle Cai, Jiaxin Cao, Qinsheng Zhang, Han Cai, Junjie Bai, Yangging Jia, Kai Li, and Song Han. Distrifusion: Distributed parallel inference for high-resolution diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7183-7193, 2024.
[20] Muyang Li, Yujun Lin, Zhekai Zhang, Tianle Cai, Xiuyu Li, Junxian Guo, Enze Xie, Chenlin Meng, Jun Zhu, and Song Han. Svdquant: Absorbing outliers by low-rank components for 4-bit diffusion models, 2025.
[21] Zefan Li et al. Cogvideox: Text-to-video diffusion models with an expert pipeline. arXiv preprint arXiv:2408.0660 2024.
[22] Haoyu Lu, Guoxing Yang. Nanyi Fei, Yuqi Huo, Zhiwu Lu, Ping Luo, and Mingyu Ding. Vdt: General-purpose video diffusion transformers via mask modeling. arXiv preprint arXiv:2305. 13311, 2023.
[23] Xin Ma, Yaohui Wang. Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Qiao. Latte: Latent diffusion transformer for video generation. Transactions on Machine Learning Research, 2025.
[24] Xinyin Ma, Gongfan Fang, and Xinchao Wang. Deepcache: Accelerating diffusion models for free. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pages 15762-15772, 2024.
[25] Laura Manduchi, Kushagra Pandey, Clara Meister, Robert Bamler, Ryan Cotterell, Sina Däubener, Sophie Fellenz, Asja Fischer, Thomas Gärtner, Matthias Kirchler, et al. On the challenges and opportunities in generative ai. arXiv preprint arXiv24403.00025 2024.
[26] Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu, et al. Fp8 formats for deep learning. arXiv preprint 22XX:32209005433, 2022.
[27] Beatrice Alessandra Motetti, Matteo Risso, Alessio Burrello, Enrico Macii, Massimo Poncino, and Daniele Jahier Pagliari. Joint pruning and channel-wise mixed-precision quantization for efficient deep neural networks. IEEE Transactions on Computers, 2024.
[28] William Peebles and Saining Xie. Scalable diffusion models with transformers, pages 4195- 4205, 2023.
[29] Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision, 2024.
[30] Yihua Shao, Deyang Lin, Fanhu Zeng, Minxi Yan, Muyang Zhang, Siyu Chen, Yuxuan Fan, Ziyang Yan, Haozhe Wang, Jingcai Guo, et al. Tr-dq: Time-rotation diffusion quantization. arXiv preprint 26X 2025.
[31] Haihao Shen, Naveen Mellempudi, Xin He, Qun Gao, Chang Wang, and Mengni Wang. Efficient post-training quantization with fp8 formats. Proceedings of Machine Learning and Systems, 6:483-498, 2024.
11

---

[32] Hui Shen, Jingxuan Zhang, Boning Xiong, Rui Hu, Shoufa Chen, Zhongwei Wan, Xin Wang, Yu Zhang, Zixuan Gong, Guangyin Bao, al. Efficient diffusion models: A survey, arXiv preprint 2rXiv:2502200805,, 2025.
[33] Andy Shih, Suneel Belkhale, Stefano Ermon, Dorsa Sadigh, and Nima Anari. Parallel sampling of diffusion models. Advances in Neural Information Processing Systems, 36:4263-4276, 2023.
[34] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209. 14792, 2022.
[35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[36] Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen- Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang. Jianyuan Zeng, Jiayu Wang. Jingfeng Zhang. Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang. Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fang. Tianxing Wang. Tianyi Gui, Tingyu Weng, Tong Shen, Wei Lin, Wei Wang, Wei Wang. Wenmeng Zhou, Wente Wang, Wenting Shen, Wenyuan Yu, Xianzhong Shi, Xiaoming Huang, Xin Xu, Yan Kou, Yangyu Lv, Yifei Li, Yijing Liu, Yiming Wang, Yingya Zhang, Yitong Huang, Yong Li, You Wu, Yu Liu, Yulin Pan, Yun Zheng, Yuntao Hong, Yupeng Shi, Yutong Feng, Zeyinzi Jiang, Zhen Han, Zhi-Fan Wu, and Ziyu Liu. Wan: Open and advanced large-scale video generative models, 2025.
[37] Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wel Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng. et al. Wan: Open and advanced large-scale video generative models. arXiv preprini arXiv:2503.200 203.44 2025.
[38] Changyuan Wang. Ziwei Wang, Xiuwei Xu, Yansong Tang, Jie Zhou, and Jiwen Lu. Towards accurate post-training quantization for diffusion models. arXiv preprint arXiv:2305.18723 2023.
[39] Wei Wang. Yifan Zhu, Yicong Li, and Xu Yan. Controllable Generation in Diffusion Models: A Survey. Journal of Computer Science and Technology, 39(3):601-626, 2024.
[40] Xiaolong Wang, Zhijian He, and Xiaojiang Peng. diffusion models: A literature review. Mathematics, 12(7):977, 2024.
http---intelliienne-geeeeeee content with
[41] Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 1 13(4):600-612 2004.
[42] Junyi Wu, Haoxuan Wang, Yuzhang Shang, Mubarak Shah, and Yan Yan. Ptq4dit: Post-training quantization for diffusion transformers. arXiv preprint arXiv:2405. 16005, 2024.
[43] Haocheng Xi, Shuo Yang. Yilong Zhao, Chenfeng Xu, Muyang Li, Xiuyu Li, Yujun Lin, Han Cai, Jintao Zhang, Dacheng Li, al. Sparse videogen: Accelerating video diffusion transformers with spatial-temporal sparsity. arXiv preprint arXiv:2502.01777, 2025.
[44] Weiying Xie, Haowei Li, Jitao Ma, Yunsong Li, Jie Lei, Donglai Liu, and Leyuan Fang. Jointsq: Joint censsfiicationquuntiatttt for distributed learning. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 5778-5787, 2024.
[45] Zhen Xing, Qijun Feng, Haoran Chen, Qi Dai, Han Hu, Hang Xu, Zuxuan Wu, and Yu-Gang Jiang. A survey on video diffusion models. ACM Computing Surveys, 5-42):1-42, 2024.
[46] Sanghyun Yi, Qingfeng Liu, and Mostafa El-Khamy. Hardware-friendll static quantization method for video diffusion transformers. arXiv preprint arXiv:2502./5077, 2025.
[47] Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, YX Wei, Lean Wang, Zhiping Xiao, et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. arXiv preprint arXiv:2502. 2025.
12

---

[48] Jintao Zhang, Haofeng Huang, Pengle Zhang, Jia Wei, Jun Zhu, and Jianfei Chen. Sageattention2 technical report: Accurate bit attention for plug-and and-play inference acceleration. arXiv preprint urXiv:2411.0095 2024.
[49] Jintao Zhang, Haofeng Huang, Pengle Zhang, Jun Zhu, Jianfei Chen, et al. Sageattention: Accu- rate 8-bit attention for plug-and-play inference acceleration. arXiv preprint arXiv:2410.02367, 2024.
[50] Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia Wei, Haocheng Xi, Jun Zhu, and Jianfei Chen. Spargeattn: Accurate sparse attention accelerating any model inference. arXiv preprint 1811 2025.
[51] Peiyuan Zhang, Yongqi Chen, Runlong Su, Hangliang Ding, lon Stoica, Zhenghong Liu, and Hao Zhang. Fast video generation with sliding tile attention. arXiv preprint arXiv:2502.04507, 2025.
[52] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[53] Tianchen Zhao, Tongcheng Fang, Haofeng Huang, Enshu Liu, Rui Wan, Widyadewi Soedar- madji, Shiyao Li, Zinan Lin, Guohao Dai, Shengen Yan, et al. Vidit-q: Efficient and accu- rate quantization of diffusion transformers for image and video generation. arXiv preprint 24Xx::2006.025400 2024.
[54] Tianchen Zhao, Xuefei Ning, Tongcheng Fang. Enshu Liu, Guyue Huang. Zinan Lin, Shengen Yan, Guohao Dai. and Yu Wang. Mixdq: Memory-efficien few-step text-to-image diffusion models with metric-decoupled mixed precision quantization, 2024.
[55] Xuanlei Zhao, Xiaolong Jin, Kai Wang, and Yang You. Real-time video generation with pyramid attention broadcast. arXiv preprint ar arXiv:2408. 12588, 2024.
13

---

Appendix
Contents
A
FPSAttention Algorithm
15
Additional Implementation Details
16
C
Ablation Study: Challenges of Naive Quantization and Sparsity Combination
16
VBench Full Evaluation Results
Clarification on VBench Evaluation Metries
18
Training Hyperparameters
19
G Limitations
2
Visualization
14

---

A FPSAttention Algorithm
Algorithm [ presents the core computational workflow of our FPSAttention method, which imple- ments joint tile-wise FP8 quantization with structured sparse attention and denoising step-aware adaptation. The algorithm follows the methodology described in the main paper, incorporating tile- wise quantization for queries and keys, channel-wise quantization for values, tensor-wise quantization for attention weights, and dynamic adaptation based on denoising timesteps.
Algorithm FPSAttention: Joint Tile-wise FP8 Quantization and Sparse Attention
Input: Input tensors Q, K, V E Lxd denoising step diffusion steps
Input:
Transition points 022 quantization granularities (Juarse Jiine fintermeiiate]
Input:
Window sizes (W sparse W dense: W medium, deenstddee J. tile scheme T
Output:
Output tensor X R- Lxd (BF16/FP16)
1:
1. Denoising Step-aware Parameter Selection
2: l1 <- 01 -D, + 022
3: ift < t then
4:
g(t) f- Iccase: (t) <- W Wsparse
Early steps
5: else if 1 < s 22 then
6:
g() & gine. (t) + dense
D. Mid steps
7: else
8:
g(t) - Intermmddattee (t)) < W medium, density
D Late steps
9: end if
10:
D 2. Tile-wise FP8 Quantization for Q and K
Partition Q, K into tiles [Tu} with granularity g())
12: for each tile Tu do
2Q <- (...T)) 14..l/MPr OFFssaMPrs_max K < (ax(T.)) //MF\ max
14:
15: Qu < FP8(Qu,i 2 for (i,j) E T
Kij < FP8(Kiji s for (i,j) Ta
17:
end for
Channel-wise FP8 Quantization for
19: for each channel j E fl..... df do
<- maxie [.pl/MMrs.. max 21: 4. FP8(V.)58)) for all
20:
22: end for
4. Structured Sparse Attention Computation via FlexAttention 24: Define neighborhood W(u) based on window size W (t):
23:
25: W(u) (v: ||c,- collox I (W./ (2T()T)), W... (2T)))
26:
b Configure FlexAttention mask and score modification functions True if tile(q) E W(tile(k)), False otherwise
27: Define mask_mode h, q,
28: Define score_mod($, 6, k) = S
b Identity for quantized inputs
29:
Execute FlexAttention with quantized inputs and custom modifications X FlexAttention(( K, V. score_ _mod, mask_mod)
31:
b 5. Dequantize Output to Target Precision
32:
X + Dequantize (X)
> Dequantize to BF16/FP16
return X
Key Algorithmic Components
The algorithm implements the four core innovations described the main paper:
Denoising Step-aware Adaptation: Lines 2-8 implement the adaptive scheduling strategy from Equation 6 in the main paper, dynamically adjusting quantization granularity g(l) and sparsity window size ((t based the current denoising step t.
15

---

leveraging both Tile-wise FP8 Quantization for Q and K: Lines 10-15 partition queries and keys into 3D tiles with step-dependent granularity and compute per-tile scaling factors s and s to minimize quantization error within each tile.
Channel-wise FP8 Quantization for V: Lines 17-20 apply channel-wise quantization to the value matrix, preserving fine-grained channel information that is critical for generation quality.
FlexA LexxAttention-basee Sparse Attention: Lines 22-26 implement structured sparse attention using FlexA 's mask_mod and score_moo e_mod interfaces, enabling hardware-optimized execution with tile-wise sparsity patterns that generate exactly M x [W(u)] dense attention blocks.
Output Dequantization: Line 28 dequantizes the FlexAttention output to the target preci- sion (BF16/FP16) to maintain compatibility with the downstream network components.
This implementation ensures full compatibility with the theoretical framework while enabling prac- tical hardware acceleration through structured computation patterns and optimal memory access patterns.
B Additional Implementation Details
Models. We implement and evaluate FPSAttention on the Wan architecture 33
1.3B and 13B parameter variants to demonstrate scalability. The Wan models feature a DiT backbone with cross-attention for text conditioning and temporal attention for inter-frame modeling. Our implementation maintains architectural fidelity while seamlessly integrating FP8 quantization across attention and feed-forward components. The joint quantization and sparsity mechanisms realized through FlexAttention' 's score and mask modification interfaces, with the resulting fused kernels compiled via Triton for optimal execution Hopper architectures.
Hardware. Experiments utilize a distributed computing cluster with gegh-performance GPU nodes, each containing 192 CPU cores, 960GB system memory, and 8xNVIDIA H20 GPUs (96GB each). InfiniBand interconnects ensure high- biihidbbnnwidthh inter-node communication for distributed training. Training scales from 16 nodes (1.3B model) to 64 nodes (13B model), requiring approximately 7 days per configuration to achieve convergence.
Dataset. Training employs a curated high-quality video dataset processed through comprehensive filtering pipeline. The preprocessing workflow includes automated subtitle removal, black-border cropping, and monochrome video exclusion, followed by quality-based filtering using established metrics (Q-Align 3.5, Aesthetic Score: 2.0, optical flow magnitude 0.05-2.0). After deduplication, videos are standardized to 480p resolution, 16fps frame rate, and 5-second duration to optimize the computational efficiency-quality balance across both model scales.
Evaluation. Performance assessment utilizes the VBench benchmark 17 following established protocols (55 19
with 5-video sampling per prompt. Evaluation encompasses 16 comprehensive VBench dimensions covering aesthetic quality. temporal consistency, motion dynamics, and semantic understanding. Additional quantitative metries include PSNR [12]. SSIM, and LPIPS [52] to provide multi-faccted quality assessment.
Baselines. Our comparative analysis includes representative approaches from three categories: (1) sparsity- based methods (Sparse VideoGen [43], STA). (2) quantization-focut focusee techniques (SageAt- tention 49),, and (3) joint optimization methods (SpargeAttn [50]). This selection enables com-
prehensive evaluation of FPSAttention against both specialized single-optimization approaches and competing joint methods, providing thorough assessment of our framework's relative performance and efficiency gains.
C Ablation Study: Challenges of Naive Quantization and Sparsity Combination
To validate our core motivation that naive combination of FP8 quantization and sparsity presents significant challenges, conduct a comprchensive ablation study comparing three key approaches:
16

---

(I) the baseline full cull-precision model, (2) training- free naive combination of quantization and sparsity, and (3) our proposed FPSAttention method with joint optimization.
Table 5 presents a detailed comparison across all VBench metrics for the Wan 1.3B model. The training-free approach applies standard FP8 quantization and sparse attention patterns without joint optimization or denoising step-aware adaptation. As hypothesized, this naive combination leads to substantial performance degradation across nearly all evaluation metrics.
Table 5: Ablation study demonstrating the challenges of naive quantization and sparsity combination. We compare bascline full preccison (Baseline), training-frie free naive combination (Training-Frev and joint optimization approach (FPSAttention) Wan 1.3B across all VBench metrics. The severe degradation in the training-fres free approach validates the need for holistic joint optimization.
Metric
Baseline
Training-Fres Free
FPSAttention
Aesthetic Quality
0.6105
0.2892
0.6240
Appearance Style
0.7157
0.7874
0.7252
Background Consistency
0.9503
0.9280
0.9156
Color
0.9049
0.4836
0.8932
Dynamic Degree
0.3014
0.3750
0,4195
Human Action
0.7720
0.0200
0.7780
Imaging Quality
0.6708
0.6868
0.7103
Motion Smoothness
0.9527
0.9513
0,9413
Multiple Objects
0.6091
0.0000
0.6665
Object Class
0.7710
0.0109
0.8185
Overall Consistency
0.6453
0.1206
0.6893
Quality Score
0.8332
0.7473
0.8428
Scene
0,3030
0.0129
0.3870
Semantic Score
0.6768
0.1733
0.7088
Spatial Relationship
0.7317
0.0008
0.7659
Subject Consistency
0.9457
0.8887
0.9338
Temporal Flickering
0.9844
0.9401
0,9336
Temporal Style
0.6382
0.1239
0.6558
Total Score
0.8019
0.6325
0.8160
Performance Drop
-21.1%
+1.8%
The results clearly demonstrate the challenges inherent in naive quantization and sparsity combination:
Key Findings:
Severe Quality Degradation: The training-free approach achieves only 0.6325 total score compared to the baseline's 0.8019, representing a substantial 21.1% performance drop.
Critical Failure Modes: Several metrics show near-zero performance in the training free approach, including Human Action (0.02), Multiple Objects (0.0), Object Class (0.011), and Spatial Relationship (0.0008), indicating complete failure in complex semantic understand- ing tasks.
Magnified Quantization Errors: As predicted by our theoretical analysis, sparsity mecha- nisms amplify quantization errors in high-mi maghntude attention scores. This particularly evident in metrics requiring fine-grain fine-grained semantic understanding, where the interaction between quantization noise and sparse token selection leads catastrophic information loss.
Joint Optimization Success: In contrast, our FPSAttention approach not only avoids the degradation seen in naive combination but actually improves upon the baseline (0.81 66 vs 0.8019, +1. 8% improvement), validating the effectiveness of our denoising step-aware joint optimization strategy.
This ablation study emphasizes the necessity of our training aware co-design scheme.
17

---

D VBench Full Evaluation Results
Tables and present comprehensive evaluation results of our method compared various base- lines on VBench for Wan 1.3B and Wan 13B models, respectively. These results demonstrate the effectiveness of our joint FP8 quantization and sparsity approach across multiple video quality metrics.
Table 6 shows performance comparisons across seven methods on the Wan 1.3B model: the baseline (Base), SageAttention (SageAtt) 49), SpargeAttention (SpargeAtt) [50]. Sparse VideoGen (Spar- seVG) 43, Sliding Tile Attention (STA) [51] our quantization-only variant (Ours-Q), and our full joint quantization and sparsity method (Ours-Q+S). The evaluation covers 18 comprehensive metrics including aesthetic quality, motion dynamics, temporal consistency, and semantic understanding. Our full method (ours-Q+S) achieves the highest total score of 0.8 demonstrating superior performance compared methods that apply quantization or sparsity independently.
Table 7 presents similar comparisons for the larger 13B model, where our method continues to achieve competitive performance while providing substantial computational savings. The results validate that our approach scales effectively to larger model sizes while maintaining video generation quality across diverse evaluation criteria.
Table 6: Performance comparison of different methods on Wan 1.3B across VBench metrics. We compare the baseline (Base), SageAttention (SageAtt), SpargeAttention (SpargeAtt), Videoenn (SparseVG), Sliding Tile Attention (STA), our quantization-only variant (Ours-Q), and our full joint method (Ours-QHe Q-S). Bold values indicate the best performance for each metric.
Metric
Base
SageAtt
SpargeAtt
Sparse
STA
Ours-Q
Ours-Q+S
Aesthetic Quality
0.6105
0.6104
0.5668
0.563
0.5661
0.6091
0.624
Appearance Style
0.7157
0.715
0.7744
0.2253
0.7952
0.6922
0.7252
Background Consistency
0.9503
0.95
0,9123
0,9525
0,9284
0,9472
0,9156
Color
0,9049
0.8866
0,8903
0.9037
0.8974
0,9146
0,8932
Dynamic Degree
0.3014
0.307
0.3222
0.7139
0.5722
0.3222
0.4195
Human Action
0.772
0.75
0.734
0.73
0.622
0.75
0.778
Imaging Quality
0.6708
0.6699
0.6541
0.6729
0.6626
0.6798
0.7103
Motion Smoothness
0,9527
0.9527
0.9139
0.9726
0,9649
0,9496
0,9413
Multiple Objects
0.6091
0.5837
0.4715
0.471
0.4043
0.6011
0.6665
Object Class
0.7710
0.7695
0.6859
0.6935
0.5818
0.7851
0,8185
Overall Consistency
0.6453
0.6451
0.6492
0.6935
0.6236
0.6478
0,6893
Quality Score
0.8332
0.8337
0,8049
0,2336
0.7994
0.8363
0,8428
Scene
0,3030
0,3092
0.2192
0,1732
0.1995
0,3200
0,3870
Semantic Score
0.6768
0.6704
0.6412
0.6342
0.6012
0.6780
0,7088
Spatial Relationship
0.7317
0.7364
0.7217
0.6469
0.6863
0.7438
0.7659
Subject Consistency
0.9457
0.9453
0,8982
0,9292
0.8993
0,9458
0,9338
Temporal Flickering
0.9844
0.9841
0,9647
0,9883
0.9652
0,9822
0.9336
Temporal Style
0.6382
0.6385
0.6247
0.2265
0.6005
0.6475
0.6558
Total Score
0.8019
0.8011
0.7722
0.7827
0.7597
0.8046
0.8160
E
Clarification on VBench Evaluation Metries
In this study, we observed that some of the baseline methods we reproduced (including some of our own exploratory experiments prior to FPS FPSAttention) might yield VBench scores slightly lower than those reported in their respective official publications. We attribute this primarily to the following factors: Randomness: The inherent stochasticity in video generation models can lead slight variations in results and vBench scores across multiple runs, even with identical settings. Prompt Extension: Many prior works (2 may employ specific prompt extension strategies to enrich input prompts. This can influence the content and quality scores of the generated videos. We didn't 't employ this optimization. Classifier-Free Guidance (CFG) Scale and Other Sampling Strategies: Different CFG scale values and other sampling parameters (e.g., number of sampling steps) significantly impact generation quality. While we endeavored to follow the descriptions the respective baseline papers, subtle parameter differences might still exist. It is worth noting that similar observations have been made in other research. For instance, in the work on Sliding Tile Attention (STA) [51], their reproduced Hunyuan Video baseline also exhibited lower VBench performance compared to
18

---

Vbench's 's official leaderboard. Despite these potential metric variations, we emphasize that all methods in this study (including our FPSAttention and all compared baselines) were evaluated under an identical VBench evaluation pipeline and parameter settings, ensuring a fair comparison. Our primary research objective to demonstrate the significant inference speedup achieved by FPSAttention while maintaining comparable (or superior) generation quality relative to baseline methods.
Table 7: Performance comparison of different methods Wan 13B across VBench metrics. We compare the baseline (Base), SageAttentis (SageAtt), SpargeAttention (SpargeAtt), Sparse VideoGen (SparseVG), Sliding Tile Attention (STA), and our full joint method (Ours- Q+S). Bold values indicate the best performance for each metric.
Metric
Base
SageAtt
SpargeAtt
Sparse
STA
Ours-Q+S
Aesthetic Quality
0.6204
0.6209
0.5875
0.6246
0.6033
0.624
Appearance Style
0.2164
0.2163
0.7586
0.2306
0.2303
0.2073
Background Consistency
0.9691
0.9687
0,9355
0.9589
0.9573
0.9377
Color
0.8879
0.8825
0,8768
0.8883
0.8814
0,8932
Dynamic Degree
0.6944
0.7028
0.6028
0.6806
0.7028
0.8389
Human Action
0.796
0.8
0.78
0.816
0.778
0.816
Imaging Quality
0.6715
0.6724
0.635
0.6868
0.6577
0.7103
Motion Smoothness
0,9828
0.9828
0,9413
0.982
0,9714
0.9804
Multiple Objects
0.6627
0.6477
0.6066
0.7012
0.6576
0.666
Object Class
0.8299
0.8312
0.7896
0.8712
0.81
0.8185
Overall Consistency
0.6912
0.6912
0.6975
0.708
0,6893
0.6893
Quality Score
0.6577
0.8428
0.8134
0.7421
0.8246
0.7103
Scene
0.3669
0.3049
0,3495
0.4129
0.3387
0.3182
Semantic Score
0.7572
0.7091
0.6969
0.7421
0,7077
0.7088
Spatial Relationship
0.7364
0.7405
0.7526
0.8056
0.7405
0.7661
Subject Consistency
0.9528
0.953
0.9173
0.9489
0.953
0.9435
Temporal Flickering
0,9922
0,9922
0.969
0.9891
0.9922
0.9754
Temporal Style
0.2408
0.2408
0.6607
0.2438
0,2408
0.6558
Total Score
0.8153
0.8158
0.7901
0.8196
0,8012
0.816
The results demonstrate that our joint FP8 quantization and sparsity approach achieves competitive or superior performance compared to specialized methods focusing solely on either quantization or sparsity. For the Wan 1.3B model, our method achieves the highest total score (0.8160), outperforming the baseline (0.8019) while providing significant computational benefits. Similarly, for the Wan 13B model, our approach performs on par with best-performing methods while offering substantial memory and compute savings through the combination of quantization and structured sparsity.
F
Training Hy perparamaess
Table g presents the key hyperparameters used in our experiments for both Wan 1.3B and 13B model training configurations. These hyperparameters carefully selected to balance training stability, convergence speed, and final model performance while accommodating the constraints imposed by FPS quantization and structured sparsity.
19

---

Table Comprehensive hyperparameter configuration for Wan 1.3B and 13B model training and evaluation. The table covers model architecture specifications, training parameters, diffusion sched- uler settings, data configuration, and system-level precision settings used in our experiments.
Category
Parameter
Wan 1.3B
Wan 13B
Model Architecture
Model Type
WanX21FPS
WanX21FPS-13B
Model Dimension
1536
5120
Number of Layers
30
40
Number of Heads
12
40
FEN Dimension
8960
13824
Input/Output Dimension
16
16
Frequency Dimension
256
256
Text Dimension
4096
4096
Patch Size
[1. 2, 2]
[1, 2, 2]
Training
Learning Rate
5e-6
5e-6
Weight Decay
le-4
le-4
Gradient Clipping
1.0
1.0
Warmup Steps
200
200
EMA Decay
0.99
0.99
Adam Epsilon
lc-15
lc-15
Diffusion Scheduler
Scheduler Type
rflow-wanx
rflow-wanx
Number of Timesteps
1000
1000
Sample Steps
50
50
CFG Scale
5.0
5.0
Sample Shift
5.0
5.0
Transform Scale
5.0
5.0
Sample Method
logit logit-normal
logit- Dogit-normal
Data & Sequence
Text Length
512
512
Max Sequence Length
75600
75600
Sample FPS
16
16
Video Resolution
480p
480p
Video Duration
5s
5s
Prompt Uncond Probability
0.1
0.1
System & Precision
Data Type
fp8
fp8
Training Mode
FSDP
FSDP
Gradient Checkpointing
True
True
Quantization
True
True
Sequence Parallel Degree
G
Limitations
While FPSAttention demonstrates strong performance some considerations for broader adoption. Our approach works hest with modern FP8-capable GPUs such NVIDIA Hopper architectures, though it can still provide benefits older hardware with reduced FP8 accelcration. The method benefits from quantization-awaa results, which involves moderate increase training time compared to post-training quantization approaches. The denoising step-aware scheduling includes several hyperparameters (transition points O1 and a22, quantization granularities, and window sizes) that can be optimized for different model architectures and datasets. Our current evaluation focuses on the Wan2. architecture, and the approach shows strong promise for extension to other video diffusion transformer architectures (e.g., Hunyuan Video, Cog VideoX). Additionally, while our tile-wise approach achieves good hardware utilization across tested configurations, there are opportunities for architecture-specife optimization to further improve performance different GPU memory hierarchies.
our evaluation scenarios, there
training to achieve optimal
20

---

H
Visualization
The following qualitative comparison demonstrates that our FPSAttention method generates video frames that are visually nearly identical to the baseline Wan model with size of 1.3B. This visual similarity across diverse
boats, fish, dogs, desert landscapes, couples, trains, cars, cats, and robot DJs vvalidatss that our joint FP8 quantization and sparsity optimization achieves essentially lossless performance while providing substantial computational acceleration.
21

---

Table 9: Qualitative comparison on the boat group. Prompt: "A boat sailing leisurely along the Seine River with the Eiffel Tower in background in super slow motion'. Top: Baseline; Bottom: FPSAttention
Baseline
FPSAttention
Table 10: Qualitative comparison on the fish group. Prompt: 'Golden fish swimming the ocean'.
Top: Baseline; Bottom: FPSAttention.
Baseline
FPSAttention
Table 11: Qualitative comparison on the dog group. Prompt: 'A dog enjoying a peaceful walk'.
Top:
Baseline; Bottom: FPSAttention.
Baseline
FPSAttention
22

---

Table 12: Qualitative comparison on the desert group. Prompt: 'Static view on desert scene with an oasis palm trees and a clear calm pool of water', Top: Baseline; Bottom: FPSAttention.
dettle
Baseline
FPSAttention
Table 13: Qualitative comparison on the couple group. Prompt: 'A couple in formal evening wear going home get caught in heavy downpour with umbrellas' Top: Baseline; Bottom: FPSAttention.
Baseline
FPSAttention
Table 14: Qualitative comparison on the train group. Prompt: 'A train accelerating to gain speed'.
Top: Baseline; Bottom: FPSAttention.
Baseline
FPSAttention
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have properly made the main claims in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
23

---

Table 15: Qualitative comparison on the rock group. Prompt: tranquil tableau of at the edge of the Arabian Desert, the ancient city of Petra beckoned with its enigmatic rack-d façades'. Top: Baseline; Bottom: FtSAttention
Baseline
FPSAttention
Table 16: Qualitative comparison on the nursery group. Prompt: 'Nursery'. Top: Baseline; Bottom: FPSAttention.
Baseline
FPSAttention
Table 17: Qualitative comparison on the snow group. Prompt: 'Snow rocky mountains peaks canyon. snow blanketed rocky mountains surround and shadow deep canyons. The canyons twist and bend through the high elevat'. Top: Baseline; Bottom: FPSAttention.
Baseline
FPSAttention
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?

---

Table 18: Qualitative comparison on the book group. Prompt: person is reading book'.
Top:
Baseline; Bottom: FPSAttention.
Baseline
FPSAttention
Table 19: Qualitative comparison the space group. Prompt: 'An astronaut flying in space'. Top:
Baseline; Bottom: EPSAttention.
Baseline
FPSAttention
Table 20: Qualitative comparison on the panda group. Prompt: 'A panda drinking coffee a cafe in
Paris'. Top: Bascline; Bottom: FPSAttention.
Baseline
FPSAttention
Answer: [Yes]
Justification:
Yes, we include the limitations in last section and Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
25

---

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
See sections in methodology and appendixes.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide detailed experiments setup on Experiment sections, we also have detailed algorithm appendix for Reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same datasct, provide access to the model. In general. releasing code and data is often
26

---

reproducible, they one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We provide code after acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ cttips.cc/pubbic/guuu Coieee///oeSSmmiiicc
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide implementation details.
Guidelines:
27

---

The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We verify our results on huge size of DiT models, repeat computation are expensive. But we repeat few times of exponents to obtain the mean result.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We give the competing requirements in the experiment setup.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: httttp:///eurips.cooo http:///wrpppccpppppaaaaaccacaaa
Answer: [Yes]
28

---

Justification:
Yes we do.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have included the broader impacts the Conclusion.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Our methodology not release new data or models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
29

---

12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, we give proper citations and reference for used resources.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not study human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
30

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
We are not subject to the approval of IRB.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: use LLMs to check and correct grammar errors.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM for what should or should be described.
31


Reviews:
[{"review_id": "NUPQmuA2QD", "reviewer": "Reviewer_jgQE", "summary": "This paper proposes FPSAttention, a method that leverages FP8 quantization and sparse attention to accelerate video generation. FPSAttention is a training-aware approach that applies quantization and sparsity in 3D-tilewise manner and introduces a denoising step-aware adaptive strategy to adjust the granularity of both quantization and sparsity. Furthermore, the authors develop a dedicated GPU kernel to support the proposed attention mechanism. As a result, FPSAttention achieves an approximate 5× speedup in video generation without noticeable degradation in output quality.", "questions": "- What are the training or calibration costs associated with the prior sparse methods listed in Table 2? \n- Can you provide a comparison with existing quantization techniques for video diffusion, particularly those in [1] and [2]?\n- What happens if prior quantization techniques are directly combined with sparse attention?\n\n\n[1] Tian, Shilong, et al. \"QVD: Post-training Quantization for Video Diffusion Models.\" Proceedings of the 32nd ACM International Conference on Multimedia. 2024.\n\n [2] Zhao, Tianchen, et al. \"Vidit-q: Efficient and accurate quantization of diffusion transformers for image and video generation.\" arXiv preprint arXiv:2406.02540 (2024).", "limitations": "This paper does not discuss the limitations of the proposed approaches. I would gently suggest including a discussion of limitations in the paper.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n- The proposed method delivers significant speedup in video generation while maintaining high-quality outputs.\n\n**Weaknesses:**\n- The training overhead required to adapt a video diffusion model to FPSAttention is substantial—it demands 64 nodes with 8 H20 GPUs for 7 days. This raises concerns about the scalability and general applicability of the approach to diverse models and resource settings.\n- The paper does not include any comparison with prior quantization methods for video diffusion, such as [1] and [2].\n- There is no analysis on the direct combination of quantization and sparsity, which could provide insight into the benefit of the proposed joint method.\n\n[1] Tian, Shilong, et al. \"QVD: Post-training Quantization for Video Diffusion Models.\" Proceedings of the 32nd ACM International Conference on Multimedia. 2024.\n\n [2] Zhao, Tianchen, et al. \"Vidit-q: Efficient and accurate quantization of diffusion transformers for image and video generation.\" arXiv preprint arXiv:2406.02540 (2024).", "was_revised": true, "final_justification": "This paper presents a comprehensive approach to accelerating video generation by leveraging both FP8 quantization and sparse attention. While it requires an additional training stage, the associated overhead is reasonable given the significant speedup achieved by the proposed method. As my concerns have been addressed, I now lean toward accepting this paper.", "rebuttal": "## **Response to Reviewer jgQE**\nThanks to the reviewer for the valuable comments.\n\n\n### **Q1: Training and calibration costs compared to baseline sparse attention methods**\n---\n\n**A1:**  There is a fundamental difference between training-free sparse methods and our **trainable joint optimization** method FPSAttention. The methods in Table 2 are predominantly **training-free approaches** with zero additional training cost—SageAttention, SpargeAtten, and STA require no training, while Sparse VideoGen only needs minimal online profiling with 64 query tokens. Our FPSAttention is training-aware but delivers superior quality-efficiency trade-offs at a reasonable fine-tuning cost across two aspects:   \n\nFirst, **our training cost is minimal compared to video generative model pretraining**. As an analogy, while video generation models require massive pretraining resources—OpenAI's Sora is estimated to need **4,200-10,500 H100 GPUs for one month [1]. Our empirical statistics show that the training overhead accounts for less than 1% of the pre-training.** \n\nSecond, training-free methods **cannot achieve effective joint optimization** of quantization and sparsity. As shown in Figure 2, naive combination leads to severe quality degradation because \"quantization errors can be magnified when combined with sparsity mechanisms\" (line 51-52). Our supplementary Table A quantifies this: **training-free joint application causes a 21.1% performance drop** (0.8019→0.6325), while FPSAttention achieves a **1.8% improvement**. \n\nAs a byproduct, our training represents a **one-time investment** that is rapidly amortized through **4.96× end-to-end speedup**, translating to around 80% reduction in recurring inference costs. For production models serving millions of requests, this upfront cost becomes negligible compared to the cumulative inference savings. \n\n> [1] Plappert, M. (n.d.). *Under the hood: How OpenAI's Sora model works*. Factorial Funds Blog. \n### **Q2: Comparison with previous methods, including QVD and ViDiT-Q quantization**\n---\n\n**A2**: We acknowledge that comparing with QVD and ViDiT-Q would strengthen our paper and commit to adding these comparisons in our revision.\n\n| Aspect             | **FPSAttention (Ours)**           | **QVD [2]**         | **ViDiT-Q [3]**                              |\n| ------------------ | --------------------------------- | ------------------- | -------------------------------------------- |\n| **Core Problem**   | Joint Sparsity & Quantization     | Quantization Only   | Quantization Only                            |\n| **Methodology**    | Training-Aware (Full Fine-tuning) | Post-Training (PTQ) | Dynamic Quantization with timestep awareness |\n| **Data Type**      | FP8 (E4M3/E5M2)                   | W8A8 (INT8)         | W8A8/W4A8 (INT)                              |\n| **Sparsity**       | Learned structured sparsity       | No sparsity         | No sparsity                                  |\n| **Step-Awareness** | Joint scheduling of both S & Q    | N/A                 | Timestep-level quantization                  |\n\n**The fundamental difference lies in our training-aware approach.** QVD and ViDiT-Q employ post-training quantization (PTQ), while FPSAttention uses quantization-aware training (QAT). This eliminates the training-inference gap that limits PTQ methods. While QVD achieves near-lossless W8A8 quantization and ViDiT-Q reports 1.4-1.7× speedups, our training-aware joint optimization of quantization AND sparsity achieves 4.96× speedup by learning optimal patterns directly from data.\n\n**Our choice of FP8 over INT8 provides critical advantages for attention mechanisms.** FP8 better captures the wide dynamic range of attention activations compared to INT8 used by QVD[2]/ViDiT-Q [3]. Moreover, FP8 operations are natively accelerated on modern GPUs (H100/H200), enabling our superior speedups [4]. While INT8 offers broader hardware compatibility, it can struggle with attention's dynamic range, particularly in regions with high sparsity [5].\n\n**We are conducting additional experiments comparing FPSAttention with ViDiT-Q.** To best of our knowledge, the QVD is not opensource code yet, therefore we mainly focus on the experiment validation for ViDiT-Q. Preliminary results show that while both methods achieve comparable generation quality with timestep-aware strategies, our joint optimization delivers significantly higher speedups. This validates that **quantization alone has inherent limits** - by jointly optimizing with sparsity and leveraging hardware-native FP8, we push beyond these limits to achieve a superior speed-quality trade-off.\n\nIn our revision, we will add: (1) dedicated comparison experiments with ViDiT-Q, (2) analysis of how our denoising step-aware scheduling relates to ViDiT-Q's timestep-level quantization, and (3) discussion of these methods as complementary approaches serving different deployment scenarios.\n\n**Table: Comparison with ViDiT-Q on VBench Evaluation Metrics**\n\n| Method                | Bit-width  | Imaging Quality   | Aesthetic Quality | Motion Smooth.    | Dynamic Degree     | BG. Consist.  | Subject Consist. | Scene Consist. | Overall Consist. | **Speedup** |\n| --------------------- | ---------- | ----------------- | ----------------- | ----------------- | ------------------ | ------------- | ---------------- | -------------- | ---------------- | ----------- |\n| ViDiT-Q Baseline      | 16/16      | 63.68             | 57.12             | 96.28             | 56.94              | 96.13         | 90.28            | 39.61          | 26.21            | 1.00×       |\n| ViDiT-Q               | 8/8        | 63.48 (-0.20)     | 56.95 (-0.17)     | 96.14 (-0.14)     | 61.11 (+4.17)      | 95.84 (-0.29) | 90.24 (-0.04)    | 38.22 (-1.39)  | 26.06 (-0.15)    | 1.40×       |\n| **Wan2.1-14B (Ours)** | BF16       | 67.15             | 62.04             | 98.28             | 69.44              | 96.91         | 95.28            | 36.69          | 69.12            | 1.00×       |\n| **FPSAttention**      | FP8+Sparse | **71.03 (+3.88)** | **62.40 (+0.36)** | **98.04 (-0.24)** | **83.89 (+14.45)** | 93.77 (-3.14) | 94.35 (-0.93)    | 31.82 (-4.87)  | 68.93 (-0.19)    | **4.96×**   |\n\n*Note: Values in parentheses show the change from respective baselines. Bold values indicate best performance.*\n\n**Key Observations:**\n\n- **ViDiT-Q shows minimal quality degradation** (-0.20 to -1.39 on most metrics) while achieving 1.40× speedup\n- **FPSAttention demonstrates quality improvements** on critical metrics (+3.88 Imaging Quality, +14.45 Dynamic Degree) despite aggressive compression\n- **Our method achieves 3.5× higher speedup** (4.96× vs 1.40×) while maintaining or improving quality\n\nThis empirical evidence validates that **joint training-aware optimization fundamentally improves the speed-quality trade-off**, enabling simultaneous quality improvements and dramatic speedups that are impossible with quantization-only approaches.\n\n> [2] Tian, Shilong, et al. \"QVD: Post-training Quantization for Video Diffusion Models.\" Proceedings of the 32nd ACM International Conference on Multimedia. 2024.  \n> [3] Zhao, Tianchen, et al. \"Vidit-q: Efficient and accurate quantization of diffusion transformers for image and video generation.\" arXiv preprint arXiv:2406.02540 (2024).  \n> [4] Kuzmin, A., Van Baalen, M., Ren, Y., Nagel, M., Peters, J., & Blankevoort, T. (2022). FP8 Quantization: The Power of the Exponent. arXiv preprint arXiv:2208.09225.   \n> [5] Noune, B., Jones, P., Justus, D., Masters, D., & Luschi, C. (2024). Efficient Post-training Quantization with FP8 Formats. arXiv preprint arXiv:2309.14592.\n### **Q3: Impact of naive sequential combination of quantization and sparsity**\n---\n\n**A3:**\n**Direct combination of prior quantization techniques with sparse attention leads to catastrophic quality degradation**. As shown in Figure 2, training-free combination of FP8 quantization with STA produces visually corrupted outputs, while our extensive ablation study (Supplementary Table A) reveals a devastating **21.1% performance drop** (0.6325 vs baseline 0.8019), with critical metrics like Human Action collapsing to 0.02 and Multiple Objects to 0.0. \n\nAs explained in Lines 49-60 of the paper, \"this failure may be caused by sparsity mechanisms that prioritize retaining tokens with high-magnitude attention scores, while quantization disproportionately introduces large errors in these values.\" This motivates our FPSAttention that **jointly optimizes quantization and sparsity** patterns through unified tile-wise operations in a training-aware manner. This co-design ensures quantization and sparsity work synergistically, enabling 4.96× speedup while maintaining visual quality - demonstrating that training-aware optimization is not merely beneficial but **essential** for aggressive compression in video diffusion models.\n### **Q4: Limitations discussion**\n---\n\n**A4:** We thank the reviewer for this suggestion. We would like to clarify that we have included a discussion of limitations in **Section B \"Limitation and Societal Impacts\" in our Supplementary Material**. Our limitations discussion addresses several key aspects: our approach works best with modern FP8-capable GPUs (e.g., NVIDIA Hopper architectures) though it can still provide benefits on older hardware; the method requires quantization-aware training which involves moderate additional training time compared to post-training approaches; the denoising step-aware scheduling includes several hyperparameters (α₁, α₂, quantization granularities, window sizes) that need optimization for different architectures; and while our current evaluation focuses on Wan2.1, the approach shows promise for extension to other video diffusion transformers. \n**We will move the limitations discussion in the main paper of the revised version.**"}, {"review_id": "9oDD8STzTV", "reviewer": "Reviewer_JYRy", "summary": "This work introduce FPSAttention, a training-aware co-design of FP8 quantization and sparsity for video diffusion model. The authors propose three techniques to optimize 3D bi-directional attention. \n(i) A unified 3D tile-wise granularity that supports both quantization and sparsity. \n(ii) A denoising step-aware strategy that dynamically adjusts quantization granularity and sparsity based on denoising steps, aligning with the model sensitivity across time steps.\n(iii) A hardware-friendly kernel optimized Hopper GPU implemented by FlashAttention, enabling efficient execution.\nAcceleration results are reported for Wan2.1 1.3B and 14B models, achieving around 5 times overall speedup with minimal quality loss.", "questions": "How does FPSAttention generalize to MMDiT architecture?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 2, "strengths_and_weaknesses": "Strengths: \n1. This work is clearly written and easy to follow. The paper organization and presentation is good. \n2. This work timely addresses the high-impact problem in video diffusion efficiency, demonstrating strong practical contribution with significant speedup over state-of-the-art, and preserves quality. \n3. The proposed joint quantization and sparsity is well-supported by the FlashAttention kernel to realize the speedup. The proposed FPSAttention achieves on-par or higher speedup compared to baseline methods (SageAttention) with better quality. \n4. The authors properly discuss limitations in the supplementary material, including hardware dependency, training costs and hyper parameter tuning. \n\nWeaknesses: \n1. The training cost is heavy (64 nodes with 8 H20 GPUs for 7 days), which may limit the broader application of this method compared to training-free baselines. \n2. As a minor concern, the proposed tiled quantization and sparse methods are more engineering-oriented, the novelty is a bit incremental over prior sparsity and quantization work (e.g., SageAttention, STA). Timestep-aware quantization was also explored in previous work [1] \n\n[1] Xiuyu Li, et al., Q-Diffusion: Quantizing Diffusion Models, ICCV 2023.", "was_revised": true, "final_justification": "Thanks for providing the rebuttal, which address my concerns. \nThis work presents a training-aware co-design of FP8 quantization and sparsity to improve the efficiency of video diffusion model, which can benefit the research in this domain. I vote for acceptance.", "rebuttal": "## **Response to Reviewer JYRy**\nThanks to the reviewer for the valuable comments.\n### **W1:  Fine-tuning costs**\n\n\n**WA1**: First, **our training cost is minimal compared to video generative model pretraining**. As an analogy, while video generation models require massive pretraining resources—OpenAI's Sora is estimated to need **4,200-10,500 H100 GPUs for one month [1]. Our empirical statistics show that the training overhead accounts for less than 1% of the pre-training.** \n\nSecond, training-free methods **cannot achieve effective joint optimization** of quantization and sparsity. As shown in Figure 2, naive combination leads to severe quality degradation because \"quantization errors can be magnified when combined with sparsity mechanisms\" (line 51-52). Our supplementary Table A quantifies this: **training-free joint application causes a 21.1% performance drop** (0.8019→0.6325), while FPSAttention achieves a **1.8% improvement**. \n\nAs a byproduct, our training represents a **one-time investment** that is rapidly amortized through **4.96× end-to-end speedup**, translating to around 80% reduction in recurring inference costs. For production models serving millions of requests, this upfront cost becomes negligible compared to the cumulative inference savings. \n\n> [1] Plappert, M. *Under the hood: How OpenAI's Sora model works*. Factorial Funds Blog. \n> \n\n--- \n### **Q1 : The proposed methods are engineering-oriented**\n\n\n**A1:** Our core contribution lies in a **training-aware co-design of sparsity and quantization**—a paradigm that is distinct from previous training-free approaches such as SageAttention and STA. We observe that naive combination of FP8 quantization and sparse attention leads to catastrophic **21.1% performance degradation** (see Appendix B.4). Figure 2 in the paper also shows that training-free combination of FP8 quantization with STA produces **visually corrupted outputs**. To make FPSAttention effective, we innovatively propose three non-trivial strategies, including unifying tile-wise operations, denoising step-aware scheduling and a hardware-optimized kernel design. \nRegarding timestep-aware approaches like [1], we acknowledge this important prior work. Previous timestep-aware methods focus solely on quantization parameters, missing the critical interplay between precision and sparsity that varies across denoising phases. However, our denoising step-aware strategy fundamentally differs in that it adaptively schedules both quantization granularity g(t) and sparsity window size W(t) based on different compression error patterns across denoising steps.\n\n--- \n### **Q2: Generalization to MMDiT**\n\n\n**A2:**  Our FPSAttention targets the general attention module, thus it is intuitively applicable to common diffusion Transformer architectures, including MMDiT.  Since the main difference between MMDiT architecture and Wan2.1 is the condition injection part, corss-attention v.s. in-context attention, the adaptation will be straightforward: our implementation already supports arbitrary token configurations, and the in-context attention, a pure self-attention module,  simply requires adjusting our tile partitioning to accomodate modality boundaries while maintaining the same optimization principles. We anticipate that FPSAttention will deliver comparable or even superior speedups on MMDiT due to the longer sequence lengths from concatenated multimodal tokens, which amplify the benefits of our sparsity and quantization co-design. We will futher try our FPSAttention with models using MMDiT like HunyuanVideo [2].\n\n>  [2] Kong, Weijie, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong et al. \"Hunyuanvideo: A systematic framework for large video generative models.\" *arXiv preprint arXiv:2412.03603* (2024)."}, {"review_id": "HYz2KaclsT", "reviewer": "Reviewer_zqGz", "summary": "This paper introduces FPSAttention, a training-aware framework that jointly optimizes FP8 quantization and structured sparsity to accelerate attention in video diffusion models. FPSAttention operates by applying a unified, tile-wise granularity for both quantization and block sparsity, directly aligned with GPU hardware for efficient execution. The approach further incorporates a denoising step-aware schedule, dynamically adapting compression granularity and sparsity levels throughout the diffusion process based on model sensitivity. FPSAttention is implemented as a fused, hardware-friendly kernel and evaluated on large-scale Wan2.1 video diffusion models. Experiments on the VBench benchmark demonstrate up to 4.96× end-to-end speedup (7.09× kernel speedup) over the BF16 baseline at 720p resolution, without sacrificing generation quality. The method consistently outperforms previous quantization or sparsity-only baselines.", "questions": "1.Can FPSAttention be adapted to already-trained models through fine-tuning, or does it require training from scratch with the co-designed kernel?\n\n2. What are the practical challenges or limitations when deploying FPSAttention on hardware without native FP8 or fused kernel support (e.g., older GPUs, non-NVIDIA platforms), and are there fallback strategies or ablation results for such cases?\n\n3. For models using aggressive step-reduction or other efficient sampling methods, does FPSAttention provide further speedup or quality benefits, or do these alternative approaches outperform it?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. Substantially accelerates video diffusion inference by co-designing quantization and sparsity at training time, avoiding the quality drop seen in naive or training-free methods.\n\n2. Aligns both the algorithm and implementation with GPU hardware (tile-wise design, fused kernel), enabling real wall-clock speedups, not just theoretical FLOP reductions.\n\n3. Adapts compression strategy across the diffusion process using a denoising step-aware schedule, balancing efficiency and generation quality.\n\n4. Demonstrates strong generalization to large-scale video models and achieves state-of-the-art efficiency and quality on standard benchmarks.\n\nWeaknesses\n\n1. Requires retraining or fine-tuning with the FPSAttention kernel; not directly applicable as a drop-in replacement for already-trained models.\n\n2. Gains are mainly demonstrated on NVIDIA Hopper/Ada GPUs with FP8 and custom kernel support; potential benefits on other hardware are unclear.\n\n3. Introduces new hyperparameters (tile size, sparsity window, schedule) that require tuning; the robustness and automation of these choices are not fully explored.", "was_revised": true, "final_justification": "I would like to keep my score as 4.\nThe work itself provides a promising deployment optimization for video diffusions but the extreme long performance recovery process sets a relatively-high bar for the individual researchers to further dive deep into the proposed methods.", "rebuttal": "## **Response to Reviewer zqGz**\nThanks to the reviewer for the valuable comments.\n### **Q1 : Adaptability of FPSAttention to pre-trained models vs. training from scratch**\n\n\n**A1:** **FPSAttention is specifically designed as a post-training adaptation method that fine-tunes already-trained models, not a training-from-scratch approach.**\n\nAs stated in Section 4, we \"implemented our proposed framework on the Wan2.1 architecture (1.3B and 14B variants), **preserving the original model structure** while introducing FPSAttention.\" The 7-day training period mentioned represents the fine-tuning phase where we jointly optimize FP8 quantization parameters and sparsity patterns starting from pre-trained checkpoints—this is significantly shorter than the months typically required for training such large models from scratch.\n\n**This post-training fine-tuning approach offers crucial advantages over both training-from-scratch and training-free methods**. Unlike training-free approaches that suffer **21.1% performance degradation** when naively combining quantization and sparsity (Table A in supplementary), our training-aware fine-tuning enables the model to **adapt to joint quantization-sparsity constraints**, achieving up to **4.96× end-to-end speedup while maintaining generation quality**. The joint optimization during fine-tuning allows the model to learn optimal patterns that preserve critical information pathways despite aggressive compression.\n\n--- \n\n### **Q2: Hardware compatibility and fallback strategies for non-FP8 platforms**\n\n\n**A2:** We acknowledge that FPSAttention's full performance benefits require modern FP8-capable hardware, but our joint optimization framework remains valuable across diverse platforms. For instance, on **AMD MI300 series GPUs**, we can perform deep custom development of the corresponding kernel, similar to the ROCm version of FlashAttention [2]. This demonstrates the potential of our methodology for application across different platforms. Looking forward, as newer architectures like NVIDIA's B200 series introduce **FP6 capabilities**, our training-aware co-design methodology can extend to these lower precisions, potentially achieving even greater speedups while validating the importance of hardware-native quantization design. For legacy hardware without native FP8 support (e.g., A100, V100), our unified optimization framework still provides substantial benefits through a **fallback strategy**: **FP16/BF16 mode** where our tile-wise sparsity patterns remain effective, achieving approximately 3× speedup from sparsity alone. Crucially, even without FP8 acceleration, our **training-aware joint optimization** prevents the catastrophic quality degradation seen in naive combinations (Table A). \n\n--- \n### **Q3: FPSAttention performance with step-reduction and alternative sampling methods**\n\n**A3:** **We thank the reviewer for this insightful question.** It highlights a crucial distinction between two orthogonal axes of diffusion model acceleration: **(1) optimizing the sampling schedule to reduce the number of steps (inter-step optimization)**, and **(2) accelerating the computation within each denoising step (intra-step optimization).**\n\nOur work, FPSAttention, fundamentally **belongs to the second category**. It focuses on accelerating the core attention mechanism, which is a major computational bottleneck *within every single forward pass* of diffusion transformers. Methods like aggressive step-reduction (e.g., DeepCache) or efficient samplers, on the other hand, belong to the first category.\n\n**These two approaches are, by design, complementary rather than competitive.** To empirically validate this principle, we integrated FPSAttention with DeepCache [3], a representative and state-of-the-art step-reduction method. The results are summarized below:\n\n| Method | Optimization Target | Speedup Factor |\n| --- | --- | --- |\n| DeepCache | Number of denoising steps | 2.31x |\n| FPSAttention | Compute per step | 4.96× |\n| Combined | Total generation time | 8.02x |\n\nAs shown in the table, DeepCache alone achieves a 2.3× speedup by caching and reusing features across adjacent steps. FPSAttention alone provides a 4.96× speedup by optimizing the expensive attention operations within each step. When combined, they yield a cumulative **8.02× speedup**. Our method serves as a foundational enhancement to the attention module, which benefits any sampling process that relies on it. Besides, training-based step-reduction approaches like sCM [4], which require additional GPU hours for finetuning, are also theoretically orthogonal and can be further combined to achieve real-time generation. We believe this can serve as a future direction for the community.\n\n**Reference**\n\n[1] Plappert, M. (n.d.). *Under the hood: How OpenAI's Sora model works*. Factorial Funds Blog. \n\n[2] ROCm/flash-attention\n\n[3] Ma, Xinyin, Gongfan Fang, and Xinchao Wang. \"Deepcache: Accelerating diffusion models for free.\" In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 15762-15772. 2024.\n\n[4] Lu, Cheng, and Yang Song. \"Simplifying, Stabilizing and Scaling Continuous-time Consistency Models.\" *The Thirteenth International Conference on Learning Representations*."}, {"review_id": "5dtWZ8voP3", "reviewer": "Reviewer_Sj6K", "summary": "This paper introduces FPSAttention, a novel framework to accelerate video diffusion models by proposing a training-aware co-design of FP8 quantization and structured sparsity. By synergistically optimizing these two techniques with a unified tile-wise granularity and a hardware-optimized kernel, the method achieves a remarkable 4.96x end-to-end speedup without sacrificing generation quality. Overall, the work proposes an effective strategy for acceleration in video generation.", "questions": "1. The authors conduct experiments on unclear frame number ( is it 81 frames consistent with the Wan model? ), and a single resolution for each model ( 480P for 1.3B model and 720P for 14B model ). The generalizability of speed gains is not fully explored, especially in the window dimensions ablation study ( Table 4 ). The efficiency of attention acceleration can be highly sensitive to input scale, so it remains unclear how the reported speedup would scale with varying numbers of frames or higher resolutions. It is recommended to conduct experiments with different input scales or provide a clear mathematical proof.\n2. The Piecewise schedule for quantization and sparsity divides denoising process into 3 parts based upon $\\alpha_1$ and $\\alpha_2$. However, this paper provides no strategy for selecting $\\alpha_1$ and $\\alpha_2$, as well as the corresponding $g_{coarse}$, $g_{fine}$, $g_{intermediate}$, $W_{coarse}$, $W_{fine}$, and $W_{intermediate}$. Besides, in Line 227, the authors mention that these hyperparameters are selected at inference time and then transferred to the training stage. I wonder whether such a strategy is too sensitive to different validation datasets, and whether it needs finetuning for every different validation dataset? In Table 3 and Table 4, the tile size and window size are both single fixed values instead of a couple of values for $(g_{coarse}$, $g_{fine}$, $g_{intermediate})$ and $(W_{coarse}$, $W_{fine}$, $W_{intermediate})$. Could the authors clarify this?\n3. In Section 3.4, the authors mention that a hardware-optimized kernel implementation complements FPSAttention. I wonder how much benefit this implementation brings. Can the authors provide a detailed ablation study?\n4. In Supp. Table D, the authors present that Wan has *13B* parameters and generates *480P* videos, which differs from the statement in the main paper ( Line 15 and Line 31 ). Besides, Wan is described to have *13B* parameters everywhere in the Supplementary Material. Please check the statements.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The authors propose an effective approach to accelerating video generation models and achieve an impressive speed gain in their experiments. \n\nWeaknesses\n1. Unclear Settings: Some key experiments' settings ( e.g., hyperparameters $\\alpha_1$ and $\\alpha_2$ introduced in Line 220 ) are missing in the article, making reproducing the results difficult.\n2. Lack of Key Ablation Studies: The authors provide no ablation study for quantitative analysis of each proposed module's gain in speed, making the effectiveness of the proposed modules unclear, especially for *Denoising Step-Aware Scheduling* and *Hardware-Optimized Kernel Design*. \n3. Generalization to other resolutions and frame numbers: The resolutions and frame numbers of generated videos in the paper are fixed ( 81 frames, 480P and 720P for the two models, respectively), leaving the speedup on different input sizes unknown.", "was_revised": true, "final_justification": "I thank the author for patiently addressing my questions, and I have decided to improve my score.", "rebuttal": "### **Response to Reviewer Sj6K**\nThanks to the reviewer for the valuable comments.\n\n### **Q1: Ablation study on input size and higher resolution**\n\n**A1.1** : Yes, in our experiments, we setup the 81 frames consistent with official Wan Model.\n\n**A1.2:** The quadratic complexity O(N²) of attention operations means that computational demands grow dramatically as we increase either the number of frames (temporal dimension) or the resolution (spatial dimensions). Our joint tile-wise FP8 quantization and structured sparsity are specifically designed to address this scaling challenge. **As the attention computation becomes more expensive, the relative benefits of our optimizations become more pronounced.**\n\nTo empirically validate this scaling behavior, we conducted comprehensive experiments across various video configurations:\n\n| Configuration | Frames | Resolution | Baseline | FPSAttention | Speedup |\n| --- | --- | --- | --- | --- | --- |\n| Short clip | 41 | 480p | 156s | 36s | **4.33×** |\n| Standard | 81 | 480p | 271s | 60s | **4.52×** |\n| Long clip | 121 | 480p | 512s | 107s | **4.78×** |\n| HD variant | 81 | 720p  | 1,301s | 262s | **4.96×** |\n| Ultra HD | 81 | 1080p | 4,102s | 789s | **5.20×** |\n\nThese results clearly demonstrate our scaling properties:\n\n**Temporal Scaling**: Fixing resolution at 480p, we observe progressively better speedups as video duration increases: 4.33× (41 frames) → 4.52× (81 frames) → 4.78× (121 frames). This improvement occurs because longer sequences have more temporal redundancy that our sparse attention patterns can reduce without quality loss.\n\n**Spatial Scaling**: Fixing duration at 81 frames, speedups improve with resolution: 4.52× (480p) → 4.96× (720p) → 5.20× (1080p). Higher resolutions benefit more because our tile-wise approach maintains consistent hardware utilization while the baseline suffers from increased memory pressure and irregular access patterns.\n\nIn summary, FPSAttention's efficiency gains scale favorably with both temporal duration and spatial resolution. \n\n---\n\n### **Q2: Missing hyperparameter details (α₁, α₂, g_coarse, etc.)**\n\n\n\n**A2:** Our hyperparameter selection follows an **empirical approach based on the error analysis shown in Figure 5**. Specifically, we observe that error tolerance follows a characteristic U-shaped curve across denoising steps, which guides our three-phase partitioning. For the Wan models evaluated, we empirically determined **α₁=0.3 and α₂=0.7**, meaning early phase (0-30% of steps), mid phase (30-70%), and late phase (70-100%). The granularity and window sizes were selected through grid search on a held-out validation set, optimizing for the quality-speed Pareto frontier.\n\nRegarding the concern about **dataset sensitivity**, our approach is designed to be robust across different generation scenarios. The key insight is that the error tolerance patterns (Figure 5) are inherent to the diffusion process itself rather than specific to particular datasets. The noise-dominated early steps and mid-steps exhibit consistent characteristics regardless of the content being generated. We validated this by testing our fixed hyperparameters across diverse VBench prompts without dataset-specific tuning.\n\nThe statement about \"selected at inference time\" (Line 227) may have caused confusion. To clarify: we perform **one-shot hyperparameter selection during method development**, then use these fixed values for both fine-tuning and inference. There is **no need to fine-tune for different validation datasets** - the same hyperparameters work across all our experiments.\n\n---\n\n### **Q3 : Hardware-optimized kernel contribution**\n\n**A3:** **Our ablation study demonstrates that fused kernel implementation is crucial for achieving high efficiency.** Without proper kernel optimization, theoretical FLOPs reduction poorly translates to actual speedup.\n\nWhen FP8 quantization and sparsity lack native hardware support—using pseudo-quantization and tile-wise sparsity simulation—the system cannot fully exploit hardware capabilities. **Our measurements show: FP8 quantization alone achieves 2.0× theoretical FLOPs reduction but only 1.4× actual speedup (70% efficiency); sparsity alone provides 3.5× theoretical reduction with 1.8× actual speedup (51% efficiency) due to irregular memory patterns.**\n\n**The efficiency gap widens dramatically with naive combination.** Despite 7.0× theoretical FLOPs reduction, naive implementation delivers merely 2.2× actual speedup—31% efficiency. This occurs because separate quantization and sparsity implementations introduce redundant memory operations, prevent data reuse, and fail to utilize specialized Tensor Cores. **In contrast, our fused kernel achieves 4.96× actual speedup from the same 7.0× theoretical reduction (71% efficiency)—a 2.26× improvement over naive approach.**\n\nThis improvement stems from hardware-conscious design that integrates quantization and sparsity within a single kernel. **Through training-aware co-design, quantization and sparsity patterns align with hardware compute units.** The fused kernel exploits this alignment by coalescing memory accesses, minimizing data movement across GPU hierarchies, utilizing FP8 Tensor Cores on Hopper architectures, and eliminating redundant computations.\n\n| Component                         | Theoretical FLOPs Reduction | Actual Speedup | Efficiency |\n| --------------------------------- | --------------------------- | -------------- | ---------- |\n| FP8 Quantization Only             | 2.0×                        | 1.4×           | 70%        |\n| Sparsity Only                     | 3.5×                        | 1.8×           | 51%        |\n| FP8 + Sparsity (Naive)            | 7.0×                        | 2.2×           | 31%        |\n| FP8 + Sparsity + **Fused Kernel** | 7.0×                        | **4.96×**      | **71%**    |\n\nThe kernel design is indeed crucial for translating theoretical computational savings into practical speedups, and we provide a comprehensive ablation study to quantify its contribution in the table above.\n\n--- \n\n### **Q4: Inconsistent reported model size**\n\n**A4:** We sincerely apologize the typo for this inconsistency. The correct model sizes are:\n\n- **Wan2.1-1.3B**: 1.3 billion parameters,  typically used for 480p generation\n- **Wan2.1-14B**: 14 billion parameters (not 13B), typically used for both 480p and 720p  generation\n\nWe will correct all instances of this error in the revised manuscript and ensure consistency throughout the paper and supplementary material.\n\n---\n\n### **Q5: Ablation study with different components**\n\n**A5:** Our design philosophy separates concerns between **quality preservation** (Denoising Step-Aware Scheduling) and **speedup optimization** (Hardware-Optimized Kernel Design), with each component serving a distinct and complementary purpose.\n\nThe denoising step-aware scheduling primarily preserves generation quality under aggressive compression. As shown in the table below, incorporating denoising step-aware scheduling with our core QAT+Sparse components improves VBench scores from 0.8019 to 0.8160 (+1.8% quality improvement) while maintaining comparable speed (2.18× vs 2.20×). \n\nMeanwhile, the hardware-optimized kernel design translates theoretical computational savings into wall-clock speedups. As demonstrated in Section 3.4 and the table below, employing our hardware kernel dramatically accelerates inference from 2.20× to 4.96× while maintaining quality (VBench score 0.8046). \n\n| Components | VBench Score | E2E Inference Speed |\n| --- | --- | --- |\n| QAT + Sparse (core components) | 0.8019 | 2.20× |\n| Core components + Hardware-Optimized Kernel Design | 0.8046 | 4.96× |\n| Core components + Denoising Step-Aware Scheduling | 0.8160 | 2.18× |\n\nThanks for helping us improve our manuscript. We will update these ablation studies in the revised version."}]

Author rebuttal:

Meta-review: This submission is carefully reviewed and received unanimous positive feedback (4545). All the reviewers are impressed by the substantial speedup which is hard but important to achieve in video generation. This does not only show the impacts in academia but also is of great practical value in industry use case. And the proposed solution of joint quantization and sparsity is compatible with existing FlashAttention, which looks elegant and easy to plugin. With a detailed and strong rebuttal provided, most concerns are addressed as explicitly acknowledged by reviewers. After checking all the reviews and discussions, AC made a decision of acceptance. Authors are encouraged to add explanations and suggested experimental results by reviewers in the final version.
Decision: Accept (spotlight)