Title: FPSAttention: Training-Aware FP8 and Sparsity Co-Design for Fast Video Diffusion
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video Generative Model

Efficiency

Quantization

Sparsity
Abstract: Diffusion generative models have become the standard for producing high-quality, coherent video content, yet their slow inference speeds and high computational demands hinder practical deployment. Although both quantization and sparsity can independently accelerate inference while maintaining generation quality, naively combining these techniques in existing training-free approaches leads to significant performance degradation, as they fail to achieve proper joint optimization.
We introduce FPSAttention, a novel training-aware co-design of FP8 quantization and Sparsity for video generation, with a focus on the 3D bi-directional attention mechanism. Our approach features three key innovations: 1) A unified 3D tile-wise granularity that simultaneously supports both quantization and sparsity. 2) A denoising step-aware strategy that adapts to the noise schedule, addressing the strong correlation between quantization/sparsity errors and denoising steps. 3) A native, hardware-friendly kernel that leverages FlashAttention and is implemented with optimized Hopper architecture features, enabling highly efficient execution.
Trained on Wan2.1's 1.3B and 14B models and evaluated on the vBench benchmark, FPSAttention achieves a 7.09$\times$ kernel speedup for attention operations and a 4.96$\times$ end-to-end speedup for video generation compared to the BF16 baseline at 720p resolution—without sacrificing generation quality.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces FPSAttention, a novel framework to accelerate video diffusion models by proposing a training-aware co-design of FP8 quantization and structured sparsity. By synergistically optimizing these two techniques with a unified tile-wise granularity and a hardware-optimized kernel, the method achieves a remarkable 4.96x end-to-end speedup without sacrificing generation quality. Overall, the work proposes an effective strategy for acceleration in video generation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The authors propose an effective approach to accelerating video generation models and achieve an impressive speed gain in their experiments. \n\nWeaknesses\n1. Unclear Settings: Some key experiments' settings ( e.g., hyperparameters $\\alpha_1$ and $\\alpha_2$ introduced in Line 220 ) are missing in the article, making reproducing the results difficult.\n2. Lack of Key Ablation Studies: The authors provide no ablation study for quantitative analysis of each proposed module's gain in speed, making the effectiveness of the proposed modules unclear, especially for *Denoising Step-Aware Scheduling* and *Hardware-Optimized Kernel Design*. \n3. Generalization to other resolutions and frame numbers: The resolutions and frame numbers of generated videos in the paper are fixed ( 81 frames, 480P and 720P for the two models, respectively), leaving the speedup on different input sizes unknown.\n\nQUESTIONS:\n1. The authors conduct experiments on unclear frame number ( is it 81 frames consistent with the Wan model? ), and a single resolution for each model ( 480P for 1.3B model and 720P for 14B model ). The generalizability of speed gains is not fully explored, especially in the window dimensions ablation study ( Table 4 ). The efficiency of attention acceleration can be highly sensitive to input scale, so it remains unclear how the reported speedup would scale with varying numbers of frames or higher resolutions. It is recommended to conduct experiments with different input scales or provide a clear mathematical proof.\n2. The Piecewise schedule for quantization and sparsity divides denoising process into 3 parts based upon $\\alpha_1$ and $\\alpha_2$. However, this paper provides no strategy for selecting $\\alpha_1$ and $\\alpha_2$, as well as the corresponding $g_{coarse}$, $g_{fine}$, $g_{intermediate}$, $W_{coarse}$, $W_{fine}$, and $W_{intermediate}$. Besides, in Line 227, the authors mention that these hyperparameters are selected at inference time and then transferred to the training stage. I wonder whether such a strategy is too sensitive to different validation datasets, and whether it needs finetuning for every different validation dataset? In Table 3 and Table 4, the tile size and window size are both single fixed values instead of a couple of values for $(g_{coarse}$, $g_{fine}$, $g_{intermediate})$ and $(W_{coarse}$, $W_{fine}$, $W_{intermediate})$. Could the authors clarify this?\n3. In Section 3.4, the authors mention that a hardware-optimized kernel implementation complements FPSAttention. I wonder how much benefit this implementation brings. Can the authors provide a detailed ablation study?\n4. In Supp. Table D, the authors present that Wan has *13B* parameters and generates *480P* videos, which differs from the statement in the main paper ( Line 15 and Line 31 ). Besides, Wan is described to have *13B* parameters everywhere in the Supplementary Material. Please check the statements.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces FPSAttention, a training-aware framework that jointly optimizes FP8 quantization and structured sparsity to accelerate attention in video diffusion models. FPSAttention operates by applying a unified, tile-wise granularity for both quantization and block sparsity, directly aligned with GPU hardware for efficient execution. The approach further incorporates a denoising step-aware schedule, dynamically adapting compression granularity and sparsity levels throughout the diffusion process based on model sensitivity. FPSAttention is implemented as a fused, hardware-friendly kernel and evaluated on large-scale Wan2.1 video diffusion models. Experiments on the VBench benchmark demonstrate up to 4.96× end-to-end speedup (7.09× kernel speedup) over the BF16 baseline at 720p resolution, without sacrificing generation quality. The method consistently outperforms previous quantization or sparsity-only baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. Substantially accelerates video diffusion inference by co-designing quantization and sparsity at training time, avoiding the quality drop seen in naive or training-free methods.\n\n2. Aligns both the algorithm and implementation with GPU hardware (tile-wise design, fused kernel), enabling real wall-clock speedups, not just theoretical FLOP reductions.\n\n3. Adapts compression strategy across the diffusion process using a denoising step-aware schedule, balancing efficiency and generation quality.\n\n4. Demonstrates strong generalization to large-scale video models and achieves state-of-the-art efficiency and quality on standard benchmarks.\n\nWeaknesses\n\n1. Requires retraining or fine-tuning with the FPSAttention kernel; not directly applicable as a drop-in replacement for already-trained models.\n\n2. Gains are mainly demonstrated on NVIDIA Hopper/Ada GPUs with FP8 and custom kernel support; potential benefits on other hardware are unclear.\n\n3. Introduces new hyperparameters (tile size, sparsity window, schedule) that require tuning; the robustness and automation of these choices are not fully explored.\n\nQUESTIONS:\n1.Can FPSAttention be adapted to already-trained models through fine-tuning, or does it require training from scratch with the co-designed kernel?\n\n2. What are the practical challenges or limitations when deploying FPSAttention on hardware without native FP8 or fused kernel support (e.g., older GPUs, non-NVIDIA platforms), and are there fallback strategies or ablation results for such cases?\n\n3. For models using aggressive step-reduction or other efficient sampling methods, does FPSAttention provide further speedup or quality benefits, or do these alternative approaches outperform it?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduce FPSAttention, a training-aware co-design of FP8 quantization and sparsity for video diffusion model. The authors propose three techniques to optimize 3D bi-directional attention. \n(i) A unified 3D tile-wise granularity that supports both quantization and sparsity. \n(ii) A denoising step-aware strategy that dynamically adjusts quantization granularity and sparsity based on denoising steps, aligning with the model sensitivity across time steps.\n(iii) A hardware-friendly kernel optimized Hopper GPU implemented by FlashAttention, enabling efficient execution.\nAcceleration results are reported for Wan2.1 1.3B and 14B models, achieving around 5 times overall speedup with minimal quality loss.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n1. This work is clearly written and easy to follow. The paper organization and presentation is good. \n2. This work timely addresses the high-impact problem in video diffusion efficiency, demonstrating strong practical contribution with significant speedup over state-of-the-art, and preserves quality. \n3. The proposed joint quantization and sparsity is well-supported by the FlashAttention kernel to realize the speedup. The proposed FPSAttention achieves on-par or higher speedup compared to baseline methods (SageAttention) with better quality. \n4. The authors properly discuss limitations in the supplementary material, including hardware dependency, training costs and hyper parameter tuning. \n\nWeaknesses: \n1. The training cost is heavy (64 nodes with 8 H20 GPUs for 7 days), which may limit the broader application of this method compared to training-free baselines. \n2. As a minor concern, the proposed tiled quantization and sparse methods are more engineering-oriented, the novelty is a bit incremental over prior sparsity and quantization work (e.g., SageAttention, STA). Timestep-aware quantization was also explored in previous work [1] \n\n[1] Xiuyu Li, et al., Q-Diffusion: Quantizing Diffusion Models, ICCV 2023.\n\nQUESTIONS:\nHow does FPSAttention generalize to MMDiT architecture?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes FPSAttention, a method that leverages FP8 quantization and sparse attention to accelerate video generation. FPSAttention is a training-aware approach that applies quantization and sparsity in 3D-tilewise manner and introduces a denoising step-aware adaptive strategy to adjust the granularity of both quantization and sparsity. Furthermore, the authors develop a dedicated GPU kernel to support the proposed attention mechanism. As a result, FPSAttention achieves an approximate 5× speedup in video generation without noticeable degradation in output quality.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- The proposed method delivers significant speedup in video generation while maintaining high-quality outputs.\n\n**Weaknesses:**\n- The training overhead required to adapt a video diffusion model to FPSAttention is substantial—it demands 64 nodes with 8 H20 GPUs for 7 days. This raises concerns about the scalability and general applicability of the approach to diverse models and resource settings.\n- The paper does not include any comparison with prior quantization methods for video diffusion, such as [1] and [2].\n- There is no analysis on the direct combination of quantization and sparsity, which could provide insight into the benefit of the proposed joint method.\n\n[1] Tian, Shilong, et al. \"QVD: Post-training Quantization for Video Diffusion Models.\" Proceedings of the 32nd ACM International Conference on Multimedia. 2024.\n\n [2] Zhao, Tianchen, et al. \"Vidit-q: Efficient and accurate quantization of diffusion transformers for image and video generation.\" arXiv preprint arXiv:2406.02540 (2024).\n\nQUESTIONS:\n- What are the training or calibration costs associated with the prior sparse methods listed in Table 2? \n- Can you provide a comparison with existing quantization techniques for video diffusion, particularly those in [1] and [2]?\n- What happens if prior quantization techniques are directly combined with sparse attention?\n\n\n[1] Tian, Shilong, et al. \"QVD: Post-training Quantization for Video Diffusion Models.\" Proceedings of the 32nd ACM International Conference on Multimedia. 2024.\n\n [2] Zhao, Tianchen, et al. \"Vidit-q: Efficient and accurate quantization of diffusion transformers for image and video generation.\" arXiv preprint arXiv:2406.02540 (2024).\n\nLIMITATIONS:\nThis paper does not discuss the limitations of the proposed approaches. I would gently suggest including a discussion of limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}