Title: $\text{S}^2$Q-VDiT: Accurate Quantized Video Diffusion Transformer with Salient Data and Sparse Token Distillation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Diffusion Transformer

Model Quantization
Abstract: Diffusion transformers have emerged as the mainstream paradigm for video generation models. However, the use of up to billions of parameters incurs significant computational costs. Quantization offers a promising solution by reducing memory usage and accelerating inference. Nonetheless, we observe that the joint modeling of spatial and temporal information in video diffusion models (V-DMs) leads to extremely long token sequences, which introduces high calibration variance and learning challenges. To address these issues, we propose **$S^2$Q-VDiT**, a post-training quantization framework for V-DMs that leverages **S**alient data and **S**parse token distillation. During the calibration phase, we identify that quantization performance is highly sensitive to the choice of calibration data. To mitigate this, we introduce *Hessian-aware Salient Data Selection*, which constructs high-quality calibration datasets by considering both diffusion and quantization characteristics unique to V-DMs. To tackle the learning challenges, we further analyze the sparse attention patterns inherent in V-DMs. Based on this observation, we propose *Attention-guided Sparse Token Distillation*, which exploits token-wise attention distributions to emphasize tokens that are more influential to the model's output. Under W4A6 quantization, $S^2$Q-VDiT achieves lossless performance while delivering $3.9\times$ model compression and $1.3\times$ inference acceleration. Code will be available at https://github.com/wlfeng0509/s2q-vdit.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes S$^2$Q-VDiT, a post-training quantization framework for video diffusion models, which leverages salient data selection and sparse token distillation to improve quantization performance. Specifically, the authors introduce a Hessian-aware salient data selection method to construct a high-quality calibration set, and propose attention-guided sparse token distillation to retain tokens that are more influential to the model’s output. Comprehensive experiments on the VBench benchmark demonstrate the effectiveness of the proposed techniques.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper is generally well written and easy to follow.\n\n2. Both the Hessian-aware salient data selection and attention-guided sparse token distillation components are well motivated. The authors provide insightful observations and analyses that support the design of their proposed methods.\n\n3. The experimental evaluation on VBench is thorough and shows promising performance improvements over existing baselines.\n\nWeaknesses:\n1. In Figure 5, it would be helpful to include frame indices for better interpretation of temporal consistency.\n\n2. The abbreviation “FP” likely refers to “full precision,” but this should be explicitly defined in the text for clarity.\n\n3. The mathematical derivation from Equation (4) to Equation (6) is somewhat condensed and would benefit from additional explanation or intermediate steps to improve readability and reproducibility.\n\n4. The computation of the Hessian matrix for dataset samples and attention scores for each token may introduce significant computational overhead, especially for large-scale video diffusion models. A discussion of the method’s scalability and computational cost would be valuable.\n\nQUESTIONS:\n1. In the W4A6 setting, why are activations quantized to 6 bits while weights use only 4 bits? Some rationale or empirical justification would help clarify this design choice.\n\n2. In the attention modules, do the authors distinguish between cross-attention and self-attention mechanisms?\n\nLIMITATIONS:\nWhile the proposed methods show strong empirical performance, they may introduce notable computational overhead. In particular, computing the Hessian matrix for salient data selection and attention scores for sparse token distillation could be costly, especially in large-scale or resource-constrained settings.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces S²Q-VDiT, a post-training quantization (PTQ) framework designed specifically for video diffusion transformers (V-DMs). The core contribution addresses two major challenges in quantizing V-DMs: the high variance in quantization performance due to limited calibration data, and the inefficiency of uniform token treatment during optimization in the presence of long token sequences and sparse attention patterns. To tackle these, the authors propose two key components: Hessian-aware Salient Data Selection (SDS) and Attention-guided Sparse Token Distillation (STD).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper identifies and addresses crucial challenges specific to post-training quantization of Video Diffusion Models (V-DMs), particularly the high variance from limited calibration data and inefficient token treatment in long sequences.\n\n2. The proposed Hessian-aware Salient Data Selection (SDS) and Attention-guided Sparse Token Distillation (STD) are well-motivated and technically sound. SDS combines diffusion informativeness and quantization sensitivity effectively. STD leverages the empirically observed sparse attention patterns in V-DMs for more efficient optimization.\n\n3. The methods are thoroughly evaluated on large-scale V-DMs ranging from 2B to 13B parameters, demonstrating consistent state-of-the-art performance against various baselines. The evaluation uses the VBench benchmark suite, which covers multiple quality dimensions including Imaging Quality, Aesthetic Quality, Motion Smoothness, Dynamic Degree, and various consistencies.\n\nWeaknesses:\n\n1. While the paper mentions that QAT requires extensive training time and large datasets, a more direct comparison or discussion on scenarios where QAT might still be preferred could strengthen the argument for PTQ.\n\n2. Although the paper highlights a 1.3x inference acceleration , the detailed latency numbers in Table 3 show a more modest improvement, especially for the Hunyuan Video model (191.3s for FP vs. 175.2s for S²c-VDIT). This translates to approximately a 1.09x acceleration for Hunyuan Video and 1.28x for Cog VideoX-5B (259.2s for FP vs. 203.2s for S²-VDIT). Given the significant model compression (3.9x), the observed inference acceleration is not as substantial as might be expected, which could make the quantization less convincing for applications prioritizing runtime speed above all else.\n\nQUESTIONS:\nplease refer to the weaknesses part.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces S2Q-VDiT, a post-training quantization (PTQ) framework designed to address the performance degradation of video diffusion models (V-DMs) under low-bit quantization. Due to the long token sequences and limited calibration budget inherent in V-DMs, existing PTQ approaches often struggle with unstable performance. To tackle this, S2Q-VDiT introduces two main components. The first is Hessian-aware Salient Data Selection, which scores and selects calibration samples based on their contribution to the denoising process and their sensitivity to quantization, using an efficient approximation of the Hessian. The second is Attention-guided Sparse Token Distillation, which leverages the sparsity of attention maps to prioritize important tokens during quantization loss computation, assigning higher weights to those with stronger semantic relevance.\nThe method is evaluated on large-scale video diffusion models, including CogVideoX and HunyuanVideo, under W4A6 and W4A4 quantization settings. Results show that S2Q-VDiT maintains high generation quality while achieving up to 3.9× model compression and 1.3× inference speedup. Additionally, the proposed techniques can be integrated into other PTQ frameworks and require only a small number of calibration samples, with minimal one-time overhead during deployment.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n* The paper is easy to follow, with a clear structure that helps the reader understand the challenges being addressed and the proposed solutions.\n* The authors introduce a Hessian-aware salient data selection method, offering a novel and principled approach for selecting calibration data, which improves the post-training quantization process compared to traditional random sampling.\n* The method effectively leverages the sparse attention patterns inherent in video diffusion models (VDMs), emphasizing salient tokens to enhance PTQ performance.\n* Extensive experiments are conducted across multiple models and quantization settings to validate the robustness of the proposed approach. The method is compared with various PTQ baselines using multiple evaluation metrics, showing strong performance in generation quality. Furthermore, it outperforms other calibration data selection strategies across all reported metrics.\n\n Weaknesses:\n* It seems that the calculation of the Hessian approximation introduces non-negligible computational overhead, which can become significant as the number of calibration samples or timesteps increases.\n* The paper lacks a discussion or ablation study on the effect of calibration data size. It does not investigate how performance varies with different numbers of calibration samples, nor does it explore the trade-off between calibration data size and quantization quality.\n\nQUESTIONS:\nPlease refer to the weakness part.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a post training quantization method for video diffusion transformer. The paper addresses the issues in PTQ from two perspectives regarding the long sequence characteristics of video DiTs. For limited calibration data caused by long sequences, the paper proposes Hessian-aware Salient Data Selection to select key calibration data. Aiming at the naturally occurring sparse attention phenomenon in V-DiTs,  Attention-guided Sparse Token Distillation is proposed to focus more on key tokens to improve the performance during calibration process. Massive experiments are conducted on multiple popular V-DiTs and achieved SOTA performance.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n1. The paper focuses on the problems caused by long token sequences in video diffusion models during PTQ process, which is a novel and valuable perspective.  And the paper writing is clear and easy to understand, with good definitions and solutions to the problems from both calibration data and optimization process perspective.\n2. The method proposed in the paper has a good theoretical basis and a large number of model analyses and references, ensuring the effectiveness of the methods.\n3. Compared with existing methods, the paper has significant improvements in metrics and visual effects, proving the effectiveness of the method. And random seeds experiments are provided in the appendix to demonstrate the robustness of the method.\n4. The paper conducted extensive quantitative and ablation experiments on multiple large-scale models, demonstrating the generality and effectiveness of the method.\n\n##Weaknesses:\n1. The paper provides evaluation results under VBench and Evalcraft benchmark, but providing more low-level metrics such as PSNR and SSIM will more fully demonstrate the effectiveness of the method.\n\n2. Large video diffusion models such as HunyuanVideo support video generation at multiple resolutions. The paper provides acceleration effects under union experiment setting. But providing different resolutions results, especially at higher resolutions, will further illustrates the potential of the method for acceleration.\n\nQUESTIONS:\n1.Providing more low-level metrics for video evaluation like PSNR and SSIM would further strength the superiority of the proposed methods.\n\n2.Providing more acceleration results under different resolution of HunyuanVideo would be better for verifying the acceleration effect in different scenarios.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}