Q-VDiT: Towards Accurate Quantization and Distillation of Video-Generation Diffusion Transformers
Authors: Weilun Feng
Chuanguang Yang
Haotong Qin
Xiangqi Li
Yu Wang
Zhulin An
Libo Huang
Boyu Diao
Zixiang Zhao
Yongjun Xu
Michele Magno
Venue: ICML 2025 poster (2025)
Keywords: Diffusion Model
Model Quantization
Video Generation
Primary area: deep_learning->generative_models_and_autoencoders

Abstract: Diffusion transformers (DiT) have demonstrated exceptional performance in video generation. However, their large number of parameters and high computational complexity limit their deployment on edge devices. Quantization can reduce storage requirements and accelerate inference by lowering the bit-width of model parameters.
Yet, existing quantization methods for image generation models do not generalize well to video generation tasks. We identify two primary challenges: the loss of information during quantization and the misalignment between optimization objectives and the unique requirements of video generation. To address these challenges, we present **Q-VDiT**, a quantization framework specifically designed for video DiT models. From the quantization perspective, we propose the *Token aware Quantization Estimator* (TQE), which compensates for quantization errors in both the token and feature dimensions. From the optimization perspective, we introduce *Temporal Maintenance Distillation* (TMD), which preserves the spatiotemporal correlations between frames and enables the optimization of each frame with respect to the overall video context. Our W3A6 Q-VDiT achieves a scene consistency score of 23.40, setting a new benchmark and outperforming the current state-of-the-art quantization methods by **1.9$\times$**.

Q-VDiT: Towards Accurate Quantization and Distillation of Video-Generation Diffusion Transformers
Weilun Feng"
Chuanguang Yang
Haotong Qin
Xiangqi Li Yu Wang' Zhulin An Libo Huang'
Boyu Diao
Zixiang Zhao?
Yongjun Xu
Michele Magno
Abstract
Diffusion transformers (DiT) have demonstrated exceptional performance in video generation. However, their large number of parameters and high computational complexity limit their deploy- ment on edge devices. Quantization storage requirements and accelerate inference by lowering the bit-width of model parameters. Yet, existing quantization methods for image genera- tion models do not generalize well to video gener- ation tasks. We identify two primary challenges: the loss of information during quantization and the misalignment between optimization objectives and the unique requirements of video generation. To address these challenges, we present Q-VDiT, a quantization framework specifically designed for video DiT models. From the quantization perspective, we propose the Token-aware Quanti- zation Estimator (TQE), which compensates for quantization errors in both the token and feature dimensions. From the optimization perspective, we introduce Temporal Maintenance Distillation (TMD), which preserves the spatiotemporal cor- relations between frames and enables the opti- mization of each frame with respect to the overall video context. Our W3A6 Q-VDiT achieves a consistency of 23.40, setting new bench- mark and outperforming current state-of-the-art quantization methods by 1.9x. Code will be avail- able at http://ggiiubbcoommmmmbbbttttttt
Moteon Smooth.
Anethatic
Dynamic
reduce
Imaging Quality
Combt.
Overal Conabd.
Subject Combt.
FP16
QIDIT
PT04DIT
Scare
VIDIT-Q
VDIT((Uur)
Figure I. Evaluation on VBench of different quantization methods under W3A6 setting.
of generative tasks, including image generation (Ho et al., 2020; Rombach et al., 2022; Dhariwal & Nichol, 2021; Feng et al., 2024; Yang et al., 2025), image super-resolution (Lin et al., 2025; Wang et al., 2024b; Wu et al., 2024b), and video generation (HPC-AI, 2024; Ma et al., 2024). Diffusion Transformers (DiT) (Peebles & Xic, 2023) have emerged as prominent architecture for generation tasks, leverag- ing their ability capture long-range dependencies and scale to large parameter spaces. However, up billions of parameters (Black-Forest-Ladde 2024) and the increased computational complexity pose significant challenges for their deployment on edge devices.
1. Introduction
Diffusion models (DMs) (Ho et al., 2020; Song et al., 2020) have demonstrated remarkable success across wide range
As an effective model compression technique, quantization reduces the bit-width of parameters and enhances inference speed by utilizing integer operations (Gholami et al., 2022). This approach has been extensively applied to compress CNN-based (Pilipović et al., 2018; Ding et al., 2024) and Transformer-bast (Chitty-Venkata tures. However, existing quantization methods for Diffusion Transformers (DiT) have primarily focused on image gener- ation tasks, with limited exploration in the context of video
'Institute of Computing Technology, Chinese Academy of Sciences Universsity of Chinese Academy of Sciences "ETH Zurich. Correspondence #Zhulin An <anzhulin Cat.uuiiiiitttcccc.... "Chuanguang Yang <yynnchuanguanne cattngguuun@@tttt.cc..m @ict.ac.en>>
Venkata et al., 2023) architec-
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Q-VDiT: Towards Accurate Quantization and Distillation of Vidco-Generaaion Diffusion Transformers
Token-aware Quantization Estimator (TQE)
Temporal Maintenance Distillation (TMD)
Quantized Tokens IR N
TMD
Feature Dimension E R
HIKLDE D9)1
Output Tokens
B E IR auc
Intra-frome Distribution
Frame 1
Frame
a E Rd
Frame 2
Frame 2
Frame t
Frame
Quantized Module
TQE
Temporal Dimension Aggression
T (MOQ(0)) XW (000() as
Temporal Dimension
M RM
Spacial Dimension
Token Dimension E B
Input Tokens
Full-Precision Tokens
Quantized Tokens
Figure Overview of proposed Q-VDIT. The framework includes Token-and aware Quantization Estimator (TQE) forward process and Temporal Maintenance Distillation (TMD) for optimization. The middle part denotes the quantized forward process. & denotes matrix multiplication, @ denotes token-wise multiplication.
generation. Directly applying existing quantization meth- ods (Li et al., 2023; Ashkboos et al., 2024; Chen et al., 2024; Wu et al., 2024a) to video-generation DiT leads to significant performance degradation (Zhao et al., 2024).
feature dimensions. From optimization perspective, relying solely on Mean Squared Error (MSE) for optimization ne- glects the inter-frame information and fails to capture the overall temporal dynamics of the video. We construct the similarity distribution between frames in the full-precision (FP) model as prior knowledge. Using KI. divergence, we align the temporal distributions the quantized model with the FP model, enabling the quantized model to maintain se- mantic and temporal coherence across frames. We compare the quantized model's performance in Fig. 1.
Compared to image generation, video generation models require modeling additional temporal dimensions across multiple frames, significantly increasing the information density relative to image generation tasks. Quantization in- troduces information loss (Qin et al., 2020; Liu et al., 2020; Feng et al., 2025), which can lead to considerable perfor- mance degradation in video generation models, given their higher information density (Zhao et al., 2024). Furthermore, video generation involves strong semantic and temporal correlations between frames. Existing optimization meth- ods (Wu al., 2024a; Hc et al., 2023) typically Mean Squared Error (MSE) to align model outputs directly, which fails to account for these correlations. As result, these approaches do not calibrate the quantization process from the perspective of the entire video, leading to degradation in video quality.
The main contributions are summarized as follows:
We theoretically prove that the quantization error of weights carries less information entropy than the origi- nal weights. To address this, we introduce the Token- aware Quantization Estimator (TQE), which employs a small number of additional parameters to perform low- rank approximation of the quantization errors from two orthogonal token and feature dimensions.
To address the aforementioned challenges, propose Q-VDiT (Quantization of Video-Generation Diffusion Transformers). a novel framework specifically designed for quantizing video diffusion transformers. Q-VDiT comprises two key components: Token-aware Quantization Estimator (TQE) and Temporal Maintenance Distillation (TMD). An overview of Q-VDiT is illustrated in Fig. 2. From quantiza- tion perspective, quantizing weights introduces significant information loss, which is primary factor contributing to the degradation of video quality. To mitigate this, we employ a small number of parameters to estimate the low-rank quan- tization error of the weights from two orthogonal tokens and
We identify that using MSE alone fails to capture the inter-frame optimization information in video gener- ation. To address this, we propose Temporal Main- tenance Distillation (TMD), which models the inter- frame distribution to ensure that the optimization of each frame considers the overall distribution of vidco characteristics across all frames.
We propose Q-VDiT, framework specifically de- signed for quantizing video difffusion transformers, Ex- tensive experiments on generative benchmarks show that QDVDiT significantly outperforms current SOTA post-training quantization methods.
2

---

Q-VDIT: Towards Accurate Quantization and Distillation of Vidco-Generation Diffusion Transformers
2. Related Work
Weight Distribution
Octout
2.1. Diffusion Model
1.00
Diffusion models (Ho et al., 2020; Rombach et al., 2022)
X=445707
0.75
perform a forward sampling process by gradually adding noise to the data distribution Xo ((r). In DDPM, the forward noise addition process of the diffusion model is Markov chain, taking the form:
Denstty
0.25
1.25
Quaribed
0.5 20080ww
20.75
Value
T ((xIT)X0)) (x) = II a X xx-1s
Weight Error Distmbuttin
0.25
20=42245
1.25
Quriess-Toe
g(xx|x x1- = (x1s (ar B,1),
Qurmannn hry
Dergg
00.75
where 0₁ 1 Br, Be is time-related schedule. Diffu- sion models generate high-quality images by applying denoising process to randomly sampled Gaussian noise XT ~ (0, I). taking the form:
80.50
TOP
0.5
0.00
-15
-1.0
-0.5
0.0
0.5 1.0 1.5
Value
Outout Value
(((x) = Ao (II),
(2)
Figure An illustration of TQE in Q-VDiT
accelerate the inference. For a floating vector Xy, the quan- tization process can be formulated as
where Po.t and 3, are outputed by the diffusion model.
X Q(x 2) =clip( x/ +,,0, - 1),
2.2. Diffusion Quantization
(3)
For diffusion model quantization, methods such as Q- DM (Li et al., 2024b), BinaryDM (Zheng et al., 2024b), BiDM (Zheng et al., 2024a), and TerDiT (Lu et al., 2024) use quantization-awarc training to maintain model perfor- mance under 1-2 bits. However, these approaches require extensive additional training time, often lasting several days. For more efficient quantization, approaches like Q- Diffusion (Li et al., 2023), PTQ4DM (Shang et al., 2023), PTQ-D (He et al., 2024), TFMQ-DM (Huang et al., 2024a), QuEST (Wang et al., 2024a), EfficientDM (He et al., 2023), and MixDQ (Zhao et al., 2025) explore quantization from the perspectives quantization error, temporal features, and calibration data, particularly for Unet-based diffusion mod- els. Similarly, Q-DiT (Chen et al., 2024), PTQ4DiT (Wu et al., 2024a), SVDQuant (Li et al., 2024a), and ViDiT- Q (Zhao et al., 2024) focus on the quantization of diffusion transformers, considering their unique data distributions and computational characteristics. However, existing quantiza- tion methods primarily focus on image generation tasks, with limited exploration into the more challenging domain of video generation. Therefore, this paper focuses on opti- mizing the quantization performance of video geeerrrnneraioon diffusion transformers.
u-1
2
where X, indicates quantized vector in integer, 1-1 is round fuction and clip(-) is function that clamps values into the range of [0, 2 -1], s is a scale factor and is quantization zero point. / and u are the lower and upper bounds of quan- tization thresholds, respectively. They are determined by x, and the target bit-width. Reversely, in order to restore the low-bit integer quantization vector X, to the full precision representation, the dequantization process is formulated as
ス = Q(xx) = (xx, 2)s,
(4)
where & is the dequantized vector used for forward process.
3.2. Token-aware Quantization Estimator
For Diffusion Transformers (DiTs) (Peebles & Xie, 2023; Ma et al., 2024), the latent representation of the generation target is denoted as Z E R=xd, where n is the number of tokens and d is the hidden dimension. In image genera- tion (Peebles Xie, 2023), n corresponds to the spatial token count s. However, for video generation (Ma al., 2024), n = s where is the spatial token number and t is the temporal token number, representing frames. This means that video DiT's (V-DiTs) contain significantly more tokens than image DiTs (I-DiTs), greatly enhancing their ex- pressive capacity. However, quantization, particularly low- bit quantization, can result in substantial information loss, which is especially critical for V-DiTs (Zhao et al., 2024).
3. Methods
3.1. Model Quantization
Model quantization maps model weights and activations to low bit integer values to reduce memory footprint and
3

---

Q-VDIT: Towards Accurate Quantization and Distillation of Vidco-Generation Diffusion Transformers
Therefore, our goal is retain as much of the model's 's infor- mation as possible using a minimal number of parameters, thereby preserving the expressive capability of V-DiTs,
To address this issue, we propose Token-aware Quantiza- tion Estimator (TQE) estimate overall quantization error across both the token and feature dimensions. The illus- tration of TQE is in Fig. 3. In the token dimension, as information at different token positions tends to be highly concentrated (Zhao et al., 2024), we pre-scale the quan- tized activations to estimate the quantization information loss effectively. Additionally, given the unique temporal differences across frames in video generation (Ma et al., 2024; HPC-AI, 2024), we selectively scale the temporal tokens of individual frames to ensure balanced information distribution across frames. Formally, we reformulate Eq. (7) as:
Proposition 3.1. Given a L layer model [( the quantization process for weight is equivalent applying perturbation A to the original weight:
11 ((w))- - 1 W.+ +.),
(5)
where W; stands for i-th layer weight.
Therefore, our goal is to express the quantization error in terms of a set of efficient parameters.
Å Anttitt == (M. ( (((X))(54AAAA))))) M MER R
Theorem 3.2. Given any layer weight W, the correspond- ing quantization error A; has less information entropy com- pared with original weight W,:
XW R Q(X)((W) T + +
(8)
represents the spatial token number, and denotes the temporal token number. We define f: := x where i E [1.2.-.. .t. To enable M to account for the activation quantization error, we jointly consider the weight of the token's 's salient measurement and the dissimilarity introduced by activation quantization when initializing M. The initialization of M is defined as:
where X E Raxdi.,
H(A.) S H(W.),
(6)
where H(-) denotes the information entropy calculation.
Theorem 3.2 indicates that although the dimension of A remains unchanged, its information entropy is lower than that of the original weight Wi. Therefore, compared to the original weight dimensions, the quantization error can be estimated in a lower-rank space and represented using fewer parameters. To achieve this, we use two liw-dimensional vector parameters, a and 5, to represent the quantization er- rors through low-rank estimation. For matrix multiplication with weight W E douttx and input X E IR' Ruxdi., we approximate the operation during quantization as follows:
exp(1 p(X
11+ t. Q(X)
[se (1tt)fftaa))))
11 E. exp|1 - pl X ls
(())
Σ T== fit1 [X
W/
E | 1 |X trall
Mi 7
A Xa. XWT ~ XQ(W) + +
wi
(7)
(9) where p(:-.) computes the similarity between two se- quences, and 7); serves the weighting factor for the activa- tion quantization error of frame (. To ensure value balance, the weighting factor is normalized by the salient measure- ment w; of the token sequences in frame t.
where a E Rdden and 3 E Rduuee We define Á E Rax1 to represent the low-rank estimated quantization error, with 3 aligning the output dimension. By employing this approach, the additional parameters required to approximate the quan- tization error reduced from the original weight size of (dout X din) to only (dous + d,n), significantly lowering the parameter overhead. Furthermore, the computational efficiency of the model is preserved after quantization.
Overall, we initialize M using Eq. (9), set a with Kaiming initialization (He et al., 2015), and initialize 33 to zero follow- ing common practices (Hu et al., 2021; He et al., 2023). The modified quantization forward process is then implemented using Eq. 8, replacing the original forward computation. Leveraging the LoRunner Kernel (Li et al., 2024a), this re- placement process introduces negligible latency. Details can be found in the Appendix Sec. F.
But for latent representation of the video generation target, Z E x the information is distributed across both the external token dimension and the internal feature dimension. When activations are quantized, the degree of information loss varies across tokens (He et al., 2023; Ashkboos et al., 2024). Consequently, егтor estimation based solely on the feature dimension fails to account for the variation in quanti- zation information loss among tokens, leading to suboptimal quantization performance.
3.3. Temporal Maintenance Distillation
For diffusion transformer optimization, existing meth- ods (Wu et al., 2024a; He et al., 2023) optimize the training parameters of the quantization model by minimizing the
4

---

Q-VDIT: Towards Accurate Quantization and Distillation of Vidco-Generation Diffusion Transformers
In this approach, the optimization of each frame in the quan- tized model only considers the information gap between it and the corresponding frame in the FP model. However, the information between different frames is not indepen- dent in a complete video (Huang et al., 2024b; Ma et al., 2024). Therefore, using Eq. (11) alone to optimize a video generation quantized model insufficient. It neglects inter-frame information dependencies, which may lead to sub-optimal optimization results (Yang et al., 2022) and fail to ensure the coherence of video content.
FP Model
Q-Model w/o TMD
Q-Model W TMD
stperdl Feitton Magne
Srappe Hidrya Mutro
brazzor Indiran Metre
temporal
temporal
temmpral 0.231
To directly perceive the inter-frame information during opti- mization, we introduce Temporal Maintenance Distillation as illustrated Fig. We calculate the relationship distri- bution between token sequences from different frames, al- lowing for direct interaction of information between frames. We use KL divergence minimize the gap between them. Formally, we compute the temporal relation between the i-th and j-th frames as follows:
T = M[Spftt:ff.a (.I.-, i,j E 11,,....
Figure 4. An illustration of TMD in Q-DDIT We have enlarged the upper left and lower right comerr additionally.
(13)
output enror between the full- full-precision (FP) model and the quantization model by
where Tup presents the relationship between i-th frame and j-th frame. Thus, we can formulate the temporal relation distribution for the i-th frame by
Crask
(OQO)))))))) (10)
where (Fp(X; W) E Raxd represents the output of the full-precision model, which serves as ground truth, while ((X): Q(W)) e Rnxd denotes the quantized output. To simplify the notation,
D, = softmax(conca( T,,,,, TT.)), (14)
redefine the outputs of both models as FP and S2, where FP and o represent the restored image information required for each frame in the generated video. Accordingly, Eq. (10) is rewritten as:
where D; presents the temporal relation distribution be- tween i-th frame and whole video frames, and we use softmax formulate probability distribution. For S and 5 from the full-precision model and quantized model, can formulate FP and D& correspondingly. Thus, present Temporal Maintenance Distillation by
Ctask |FP,, ,
(11)
|SFP FP U+tlft
Ltemporat
KL(D ,D)),
(15)
In contrast to image generation which involves the image information of a single frame, video generation requires S to capture the image information across multiple frames. For the quantization model that needs be optimized, the gradient for each frame i in 2 derived from Eq. (11) is given by:
From Eq. (15), we can derive the gradient for single frame [++:ftto by
0L Ltemporal
Ttrepporal
OT a
QTQ iJ
1 +l:fi++ad
==1
[++1:f++t
OCrask -P
Aremporal 2T
UTQ J.t
0 U++l:fitoil
as
j,
[J+1iff+eel
(12)
(16)
5

---

Q-VDIT: Towards Accurate Quantization and Distillation of Vidco-Generation Diffusion Transformers
Table Performance of text-to-video generation VRench evaluation benchmark suite. The bit-width "16" represents FP16 without
quantization. Bold: the best result. Underline: the second- best result.
Bit-width
Imaging Quality
Aesthetic Quality
Motion
Dynamic Degree
BG.
Subject Consist.
Scene Consist.
Overall
Method
(W/A)
Smooth.
Consist.
Consist.
16/16
63.68
57.12
96.28
56.94
96.13
90.28
39.61
26.21
Q-DiT
4/6
55.99
52.99
95.94
51.39
94.85
86.40
32.67
24.50
PTQ4DiT
4/6
55.58
53.31
94.57
56.94
93.70
86.90
33.65
23.48
SmoothQuant
4/6
54.16
52.20
94.83
55.56
93.55
87.08
31.40
22.54
Quarot
4/6
53.79
51.95
93.26
51.39
93.15
85.26
32.77
22.89
EfficientDM
4/6
55.63
53.92
95.19
51.39
95.10
86.58
35.90
23.90
SVDQuant
4/6
55.23
53.38
95,90
47.22
95.70
87.76
32.19
23.18
ViDiT-Q
4/6
55.63
53.68
96.13
56.94
95.38
86.94
32.70
24.53
Q-VDiT
4/6
57.49
55.18
96.25
68.06
95.72
87.78
38.66
25.02
Q-DiT
3/8
36.23
31.97
86.77
13.89
95.86
90,09
0.08
10.29
PTQ4DiT
3/8
46.98
40.88
93.63
16.67
92.38
84.57
7.63
18.04
SmoothQuant
3/8
36.86
33.24
91.32
27.78
93.16
82.42
1.02
11.03
Quarot
3/8
35.94
32.99
90.76
27.78
93.65
83.44
2.51
11.14
EfficientDM
3/8
49.97
43.78
95,49
31.94
95.92
89.14
13.99
16.61
SVDQuant
3/8
43.54
32.51
90.52
23.61
94.28
83.68
4.65
10.43
ViDiT-Q
3/8
50.62
42.83
95.84
31.94
96.35
88.06
15.19
18.53
Q-VDiT
3/8
54.94
49.94
97.11
50.00
95.96
90.14
25.07
22.39
Q-DiT
3/6
37.47
32.58
86.02
13.89
94.84
88.72
0.07
10.57
PTQ4DIT
3/6
45.31
40.20
93.43
19.45
91.59
83.15
7.85
16.55
SmoothQuant
3/6
36.88
33.57
91.18
33.33
93.02
81.99
1.24
11.01
Quarot
3/6
36.02
32.54
91.15
23.61
93.86
82.14
2.26
11.06
ElficientDM
3/6
48.82
42.11
95.04
34.72
95.17
88.17
12.04
15.85
SVDQuant
3/6
43.67
32.32
90.28
23.61
94.18
83.09
3.56
10.30
ViDiT-Q
3/6
48.76
42.70
95.51
37.50
95.34
86.86
11.99
18.38
Q-VDIT
3/6
53.60
49.66
96.98
55,56
95.41
89.06
23.40
22.58
For any
Stremporat 8T4
we can calculate it using the chain rule
frames. This enables direct interaction between frames and optimizes the temporal coherence across the entire video. Ltemporal effectively compensates for the lack of alignment that arises when focusing solely on single-frame optimiza- tions. Through this approach, the video information from both the FP network and the quantized network is enhanced by capturing the overall temporal representations of the video. In summary, the overall optimization objective can be reformulated as follows:
@Lremporal
Ctemporal
TD t.k
OD
QT
1=1
i
DD
FP
D + Σ
FP
D
(17)
kgj
Σ D PP D D.P.
From Eq. (17), any correlation between i-th and j-th frame @ is numerically affected by all frames which ensures the perception of overall video information. For any sequences v; and Vi stands for i-th and j-th frame used in Eq. (13),
Liotal Lsask + Ltemporal
(19)
4. Experiments
the gradient for i-th frame 0T.,, can derived by Zv:
4.1. Experimental and Evaluation Settings
Ivsllll,lIv; (V: V3)
Ve
Will
Ov;
| 11
(18)
Experimental Settings: Following previous work ViDIT- Q (Zhao et al., 2024), we apply our Q-VDiT to Open- SORA (HPC-AI, 2024) and Latte (Ma et al., 2024) for video generation task. We mainly focus on harder settings of W4A6 (4-bit weight quantization and 6-bit activation quantization), W3A8, and W3A6. We follow the exper-
= Uo+t:ft
By applying Eq. (17) and Eq. (18) to Eq. (16), the optimiza- tion of any single frame Q
[[+ 1:f =s.] is jointly guided by all
6

---

Q-VDIT: Towards Accurate Quantization and Distillation of Vidco-Generation Diffusion Transformers
Table Performance of text-to-video generation on OpenSORA prompt set. Bold: the best result. Underline: the second-best result.
Bit-width
VQA- Aesthetic
VQA- Technical
A FLOW
Method
CLIPSIM
CLIP-Temp
Warping Error (L)
(W/A)
Score.())
16/16
0.1797
0.9986
66.91
53,49
0.016
Q-DiT
4/6
0.1757
0.9987
57.11
38.12
0.913
0.015
PTQ4DIT
4/6
0.1777
0.9978
62.37
37.68
0.287
0.022
SmoothQuant
4/6
0.1781
0.9981
55.66
22.37
1.076
0.019
Quarot
4/6
0.1778
0.9980
56.24
30.75
1.013
0.018
MfficientDM
4/6
0,1753
0.9986
62.43
43.85
0.649
0.021
SVDQuant
4/6
0.1780
0.9983
64.84
36.24
0,956
0.015
ViDiT-Q
4/6
0.1782
0.9985
54.66
49.78
0.306
0.012
Q-VDiT
4/6
0.1784
0.9989
67.05
53.75
0.281
0.013
Q-DiT
3/8
0.1707
0.9955
22.31
3.84
1.167
0.018
PTQ4DIT
3/8
0.1764
0.9952
35.21
8.56
1.389
0.039
SmoothQuant
3/8
0.1758
0.9965
16.55
1.08
1.227
0.064
Quarot
3/8
0.1755
0.9961
26.59
5.66
1.048
0.038
EfficientDM
3/8
0.1762
0,9987
45.67
28.42
1.306
0.017
SVDQuant
3/8
0.1735
0.9986
42.13
18.96
1.457
0.019
ViDiT-Q
3/8
0.1733
0.9985
41.00
13.48
1.619
0.020
Q-VDIT
3/8
0.1766
0.9988
54.92
61.59
0.839
0.011
Q-DiT
3/6
0.1681
0.9961
11.90
0.69
1.412
0.028
PTQ4DIT
3/6
0.1765
0.9947
26.35
5.42
1.185
0.042
SmoothQuant
3/6
0.1768
0.9959
18.07
1.33
1.286
0.068
Quarot
3/6
0.1762
0.9950
25.89
3.95
1.084
0.042
EfficientDM
3/6
0.1747
0.9982
43.54
29.58
1.217
0.020
SVDQuant
3/6
0.1712
0.9975
40.75
16.51
1.656
0.026
ViDiT-Q
3/6
0.1716
0.9984
39.82
10.26
1.695
0.020
Q-VDIT
3/6
0.1785
0.9986
53.53
59.10
0.914
0.012
imental settings in ViDiT-Q (Zhao et al., 2024). For the Open-Sora (HPC-AI, 2024) model, we use the setting in Appendix Sec. D.I for benchmark evaluation and use 10 prompts provided by OpenSora prompt sets to generate 10 videos for multi-aspects metrics evaluation. More experi- ments and details be found in the Sec. 4.3, Appendix Sec. B, and E.
Appendix Sec. D.
Compared Methods: We compare different base- line PTQ methods. For LLM baseline, we compare SmoothQuant (Xiao et al., 2023) and Quarot (Ashkboos et al., 2024). For DM baseline, we mainly compare Effi- CientDM (He et al., 2023) and SVDQuant (Li et al., 2024a). For DiT baseline, 2024), PTQ4DiT (Wu et al., 2024a), and ViDiT-Q (Zhao et al., 2024). It is worth mentioning that only ViDiT-Q and Q-DiT have conducted video-generation task evaluation. We rerun all compared methods for fair comparison.
mainly compare Q-DiT (Chen et al.,
Evaluation Settings: The evaluation contains two settings like ViDiT-Q (Zhao et al., 2024). We first evaluate the quan- tized model on VBench (Huang et al., 2024b) to provide comprehensive results for benehmark evaluation. align with ViDiT-Q, we select major dimensions from Vbench. We select representative metries and measure them Open- SORA prompt sets for multi-aspects metrics evaluation like ViDiT-Q. Following EvalCrafter (Liu et al., 2024), we select CLIPSIM, CLIP-Temp. Warping Error to measure consistency, and DOVER (Wu et al., 2023) video quality as- sessment (VQA) metrics to evaluate the generation quality, Flow-score are used for evaluating the temporal consistency. For Open-SORA model, use 100-step DDIM with CFG scale of 4.0. For Latte, we adopt the class-com conditindt Latte model trained on UCF-101 and use the 20-step DDIM solver with CFG scale of 7.0. More details can be found in
4.2. Main Results
Benchmark evaluation: benchmark evaluation, we use vbench (Huang et al., 2024b) to evaluate quantization video diffusion transformer from three key video aspects among 8 details dimensions. As present in Tab. 1, our proposed Q- Q-VDiT achieves notable improvement across dimensions under W4A6, W3A8, and W3A6 settings compared with all existing quantization methods. Especially under the hardest 3-bit weight settings, Q-VDiT exceeds the optimal results of existing methods by large margin. Like in W3A6 setting,

---

Q-VDiT: Towards Accurate Quantization and Distillation of Vidco-Generratin Diffusion Transformers
Table Higher bit setting performance of text-to-video generation OpenSORA prompt
Bit-width
VQA- Aesthetic
VQA-
FLOW Score.())
Method
CLIPSIM
CLIP-Temp
Warping Error (L)
(W/A)
Technical
16/16
0.1797
0.9986
66.91
53.49
0.016
Q-Diffusion
8/8
0.1781
0.9987
51.68
38.27
0.328
0.024
Q-DiT
8/8
0.1788
0.9977
61.03
34.97
0.473
0.017
PTQ4DIT
8/8
0.1836
0.9991
54.56
53.33
0.440
0.018
SmoothQuant
8/8
0.1951
0,9986
59.78
51.53
0.331
0.019
Quarot
8/8
0. 1.1949
0.9976
58.73
52.28
0.215
0.020
ViDiT-Q
8/8
0.1950
0.9991
60.70
54.64
0.089
0.016
Q-VDiT
8/8
0. 0.1950
0.9987
64.43
56.40
0.099
0.015
Q-DiT
6/6
0.1710
0.9943
11.04
1,869
41.10
0.024
PTQ4DiT
6/6
0.1799
0.9976
59.97
43.89
0.997
0.019
SmoothQuant
6/6
0.1807
0.9985
56.45
48.21
29.26
0.020
Quarot
6/6
0. 0.1820
0.9975
61.47
53.06
0.146
0.023
ViDiT-Q
6/6
0.1791
0.9984
64.45
51.58
0.625
0,016
Q-VDiT
6/6
0.1798
0.9984
67.31
54.03
0.061
0.019
Q-DiT
4/8
0. 0.1687
0.9833
0.007
0.018
3.013
0,018
PTQ4DiT
4/8
0.1735
0.9973
2.210
0,318
0.108
0.024
SmoothQuant
4/8
0.1832
0.9983
31.96
22.85
0.415
0.021
Quarot
4/8
0.1817
0.9965
47.36
33.13
0.326
0.020
VidiT-Q
4/8
0.1809
0.9989
60.62
49.38
0.153
0.012
Q-VDIT
4/8
0.1811
0.9989
71.32
55.56
0.147
0.012
Table 4. Ablation studies of Q-VDiT techniques on W3A8 Open-Sora model.
Bit-width
Imaging Quality
Aesthetic Quality
Motion Smooth.
Dynamic Degree
BG
Subject Consist.
Scene Consist.
Overall
Method
(W/A)
Consist.
Consist.
16/16
63.68
57.12
96.28
56.94
96.13
90.28
39.61
26.21
PTQ4DIT
3/6
45.31
40.20
93.43
19.45
91.59
83.15
7.85
16.55
+ +TQE (w/o M)
3/6
51.36
47.69
96.14
47.22
94.86
88.21
21.94
21.65
+ TQQE (w M)
3/6
52.35
48.97
96.77
51.39
95.12
88.86
22.38
22.00
+ +MMD
3/6
53.46
48.70
96.00
52.78
95.24
89.01
22.87
22.31
Q-VDiT
3/6
53.60
49.66
96.98
55.56
95.41
89.06
23.40
22.58
Q-VDiT improves SOTA scene consistency from 12.04 to 23.40 which is almost doubled.
Table 5, Efficiency comparisons of different methods across GPU memory and time under W8A8 setting.
Multi-aspects metrics evaluation: In multi-aspects met- rics evaluation, Q-VDiT also achieves notable improve- ment across all metrics in Tab. Q-VDiT achieves almost lossless performance under W4A6 setting and greatly im- proves existing quantization methods under the hardest 3-bit weight settings. In W3A6 setting, Q-VDiT improve VQA- Technical from current SOTA 29.58 to 59.10, CLIPSIM from 0.1 768 to 0.1785 by a great margin.
GPU Memory
GPU Time (hours)
VQA-
Method
(MB)
Aesthetic
ViDiT-Q
16600
12.5
60.70
EfficienDDM
19460
12.6
61.25
PTQ4DIT
17650
12.8
54.56
Q-VDIT
18770
12.9
64.43
4.3. A Additional Results on Higher Bit Settings
methods under higher bit settings. On W4A8 setting, Q- VDIT achieves 71.32 VQA- Aesthetic which is even higher than the FP model. This further proves our Q-VDiT's supe- riority at different bits. Notably, Q-VDiT achieves the best VQA metrics under W4A8 setting which even surpass the full-precision performance.
Tab. shows that Q-VDiT achieves notable improvement under relatively harder settings with 3-4bit weight quanti- zation. Tab. 3, we present more bit settings on Open- Sora (HPC-AI, 2024) model quantization. can seen that our VDT still outperforms the current quantization
8

---

Q-VDIT: Towards Accurate Quantization and Distillation of Vidco-Generation Diffusion Transformers
PTQ4DIT
==
EP
80
Y- 100
y=200
60
PTQ4DiT
Q-DIT
Imaging Quality
Modion
Dynamic Degrea
Quality
Smooth.
Concist.
Consist.
Consist.
Figure 6. Ablation study on different y in TMD.
ViDIT-Q
mance compared with baseline PTQ4DÎT. This shows that our TMD not sensitive to hyperparameter selection. In our practice, we use 7 100 for balanced choice.
Q-VDIT
4.6. Training Resource Cost
In Tab. 5, we present the training cost of GPU memory cost and time cost between different PTQ methods under W8A8 setting. Compared no-calibration method ViDiT-Q, we only bring 3% additional time cost and less compared with other calibration methods. Yet we achieve the best video quality metrics.
Figure 5. Visualization of different frames in single video.
4.4. Qualitative Comparison
In Fig. 5, we present different frames in single video under W3A6 with the same prompt to qualitatively compare video generation ability. We evenly sample 4 frames from the complete video. All other comparison methods may not even produce clear images, and some may not even produce meaningful images. Our proposed Q-VDiT not only pro- duces clear and meaningful images but also has significant coherent motion changes between different frames. This indicates that Q-VDiT can still produce good and meaning- ful complete videos when other methods fail. More results can be found in Appendix Sec. G.
5. Conclusion
In this paper, we have proposed Q-VDiT, quantization method tailored specifically for video Diffusion Transform- ers. To address severe model quantization information loss, we have proposed Token-aware Quantization Estimator to compensate for quantization errors from both token and feature dimensions. To maintain the spatiotemporal correla- tion between different frames in videos, we have proposed Temporal Maintenance Distillation to optimize each frame from the perspective of the overall video. Our extensive experiments have demonstrated the superiority of Q-VDIT over baseline and other previous quantization methods.
4.5. Ablation Study
Different techniques of Q-VDiT: In Tab. compared different proposed techniques used in Q-VDi.. All proposed techniques can improve the performance of quantized model. We also compare TQE without M. After introducing token- aware M, the effect of TQE is further enhanced which proves the necessity of considering both token and feature dimensions. By combining TQE and TMD, our Q-VDiT achieves the best results across all 8 dimensions.
A Acknowledgements. This work is partially supported by the National Natural Science Foundation of China under Grant Number 62476264 and 62406312, 12, the Postdoctoral Fellowship Program and China Postdoctoral Science Foun- dation under Grant Number BX20240385 (China National Postdoctoral Program for Innovative Talents), the Beijing Natural Science Foundation under Grant Number 4244098, the Science Foundation of the Chinese Academy of Sci- ences, and the Swiss National Science Foundation (SNSF) project 200021E.22 19943 Neuromorphic Attention Models for Event Data (NAMED).
Different hyperparameters used in TMD: In Fig. 6, we compare different used in Eq. (19) and PTQ baseline method PTQ4DiT (Wu et al., 2024a). We conduct experi- ment on W3A8 Open-Sora model. can be seen that all different y used TMD can notably enhance model perfor-

---

Q-VDIT: Towards Accurate Quantization and Distillation of Vidco-Generation Diffusion Transformers
Impact Statement
on Artificial Intelligence, volume 39, pp. 16595-16603, 2025.
This paper presents work whose goal to advance the field of efficient cedeo-generation diffusion transformers. There are few potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., and Keutzer, K. A survey of quantization methods for effi- cient neural network inference. In Low-Power Computer Vision, pp. 291-326. Chapman and Hal//CRC 2022.
References
He, K., Zhang, X., Ren, S., and Sun, J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE inter- national conference on computer vision, pp. 1036-1034, 2015.
Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Cameron, P., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J. Quarot: Outlier-frec 4-bit inference in ro- tated llms. arXiv preprint arXiv:2404.00456, 2024.
Flask-Forest.Labs. Flux.1, 2024. URL https://
He, Y., Liu, J., Wu, W., Zhou, H.,, and Zhuang, B. Effi- cientdm: Efficient quantization-awan fine-tuning of low- bit diffusion models. arXiv preprint arXiv:2310.03270. 2023.
blackforest ai/. 2, 7.
Blattannn, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S. W., Fidler, S., and Kreis, K. Align your latents: High- resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22563-22575, 2023.
He, Y., Liu, L., Liu, Wu, W., Zhou, H., and Zhuang, B. Ptgd: Accurate post-training quantization for diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Chen, L., Meng, Y., Tang, C., Ma, X., Jiang, J., Wang, X., Wang, Z., and Zhu, W. Q-dit: Accurate post-training quantization for diffusion transformers. arXiv preprint arXiv:2406.17333, 2024.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6851, 2020.
Chitty-Venkaie K. T.,, Mittal, S., Emani, M., Vishwanath, V., and Somani, A. K. A survey of techniques for optimizing transformer inference. Journal of Systems Architecture, pp. 102990, 2023.
HPC-AL. Open-sora, 2024. URL https://githan
com/hpcalitech/Oee
Sora.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, I.., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint (Xiv:2106.00966: 2021.
Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances in neural information processing systems,
2021.
Huang, Y., Gong, R., Liu, J., Chen, T., and Liu, X. Tfmq-dm: Temporal feature maintenance quantization for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7362- 7371, 2024a.
Ding, Y., Feng, W., Chen, C., Guo, J., and Liu, X. Reg- ptq: Regression-speeializee post-training quantization for fully quantized object detector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 161 16174-16184, 2024.
Fang, Y. Zhu, H., Zeng, Y., Ma, K., and Wang, Z. Percep- tual quality assessment of smartphone photography. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 3677-3686, 2020.
Huang, 7., He, Y., Yu, Zhang, F., Si, C., Jiang, Y., Zhang, Y., Wu, T., Jin, Q., Chanpaisit, N., et al. Vbench: Compre- hensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21807-21818, 2024b.
Feng, W., Yang, C., An, 7.., Huang, I.., Diao, B., Wang, F., and Xu, Y. Relational diffusion distillation for effi- cient image generation. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 205-213, 2024.
Ke, J., Wang, Q., Wang, Y., Milanfar, P., and Yang. F. Musiq: Multi-scale image quality transformer. In Proceedings of the IEEE/CVF international conference computer vision, pp. 5148-5157, 2021.
Feng. W. Qin, H., Yang, C., An. Z., Huang. L., Diao, B., Wang, F,, Tao, R., Xu, Y., and Magno, M. Mpq- dm: Mixed precision quantization for extremely low bit diffusion models. In Proceedings of the AAAI Conference
LAION-AL https: /ttps://githube wttp://oo////AAAAAA... aesthetic--redioooo: Accessed: 2022-04-
Aesthetic-predictor,, 2022.
URL
16.
10

---

Q-VDIT: Towards Accurate Quantization and Distillation of Vidco-Generation Diffusion Transformers
Li, M., Lin, Y., Zhang, Z., Cai, T., Li, X., Guo, J., Xie, E., Meng, C., Zhu, J.-Y., and Han, S. Svdqunat: Absorb- ing outliers by low-rank components for 4-bit diffusion models. arXiv preprint arXiv:2411.05000 2024a.
IEEE/CVF conference on computer vision and pattern recognition, pp- 2250-2259, 2020.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Li, X., Liu, Y., Lian, L.,, Yang, H., Dong. Z., Kang, D., Zhang, S., and Keutzer, K. Q-diffusion: Quantizing dif- fusion models. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 17535-17545, 2023.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Li, Y., Xu, S., Cao, X., Sun, X., and Zhang, B. Q-dm: An efficient low-bit quantized diffusion model. Advances in Neural Information Processing Systems, 36, 2024b.
Lin, X., He, J., Chen, Z., Lyu, Z., Dai, B., Yu, F., Qiao, Y., Ouyang, W., and Dong, C. Diffbir: Toward blind image restoration with generative diffusion prior. In European Conference on Computer Vision, pp. 4404448.55 Springer, 2025.
Shang, Y., Yuan, Z., Xie, B., Wu, B., and Yan, Y. Post- training quantization on diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 1972-1981, 2023.
Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02550,, 2020.
Liu, Y., Cun, X., Liu, X., Wang, X., Zhang. Y.. Chen, H., Liu, Y., Zeng, T., Chan, R., and Shan, Y. Evalcrafter: Benchmarking and evaluating large video generation models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22139- 22149, 2024.
Soomro, K. Ucf101: A dataset 101 human actions classes from videos in the wild, arXiv preprint arXiv: 1212.040 2012.
Teed, Z. and Deng, J. Raft: Recurrent all-pairs field trans- forms for optical flow. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part Il 16, pp. 402-419. Springer, 2020.
Liu, Z., Shen, Z., Savvides, M., and Cheng, K.-T. Reactnet: Towards precise binary neural network with generalized activation functions. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16. pp. 159-159. Springer, 2020.
Wang. H., Shang, Y., Yuan, Z., Wu, J., and Yan, Y. Quest: Low-bit diffusion model quantization via efficient selec- tive finetuning. arXiv preprint urXiv2240..03666 2024a.
Lu, X., Zhou, A., Lin, Z., Liu, Q. Xu, Y., Zhang, R., Wen, Y., Ren, S., Gao, P., Yan, J., et al. Terdit: Ternary diffusion models with transformers. arXiv preprint arXiv:2405. 14854, 2024.
Wang. J., Yue, Z., Zhou. S., Chan, K. C., and Loy, C. C. Exploiting diffusion prior for real-world image super- resolution. International Journal of Computer Vision, pp. 1-21, 2024b.
Ma, X., Wang, Y., Jia, G., Chen, X., Liu, Z., Li, Y.-F, Chen, C., and Qiao, Y. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.0304 .03048, 2024.
Wang, Y., He, Yi, Yu, J., Ma, X., Li, X., Chen, G., Chen, X., Wang, Y., et al. Internvid: A large- scale video-text dataset for multimodal understanding and generation. arXiv preprint urXiv:2307.0663 069442 2023.
Peebles. W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 4195-4205, 2023.
Wu, H., Zhang, E., Liao, L., Chen, C., Hou, J., Wang, A., Sun, W., Yan, Q., and Lin, W. Exploring video quality assessment on user generated contents from aesthetic and technical perspectives. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 20144- 20154, 2023.
Pilipović, R., Bulić, P., and Risojević, V. Compression of convolutional neural networks: A short survey. In 2018 17th International Symposium INFOTEH-IATIRRIA (INFOTEH), pp. 1-6. IEEE, 2018
Qin, H., Gong, R., Liu, X., Shen, M., Wei, Z., Yu, F., and
Wu, J., Wang, H., Shang, Y., Shah, M., and Yan, Y. Ptq4dit: Post- training quantization for diffusion trans- formers. arXiv preprint arXiv:2405. 16005, 2024a.
Song, J. Forward and backward information retention for accurate binary neural networks. In Proceedings of the

---

Q-VDIT: Towards Accurate Quantization and Distillation of Vidco-Generation Diffusion Transformers
Wu, R., Yang, T., Sun, L.., Zhang, Z.,, Li, S., and Zhang. L. Seesr: Towards semantics-aware real-world image super- resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 25456- 25467, 2024b.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099. PMLR, 2023.
Yang. C., Zhou, H., An, Z., Jiang, X., Xu, Y., and Zhang, Q. Cross- Cross-image relational knowledge distillation for se- mantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
pp. 12319-12328, 2022.
Yang, H., Yang, C., Wang, Q. An, 7., Feng, W., Huang, I.., and Xu, Y. Multi-party collaborative attention control for image customization. arXiv preprint arXiv:2505.01428, 2025.
Zhao, T., Fang, T., Liu, E., Rui, W., Soedarmadji, W., Li, S., Lin, Z., Dai, G., Yan, S., Yang. H., et al. Vidit- q: Efficient and accurate quantization of diffusion trans- formers for image and video generation. arXiv preprint arXiv:2406.000 02540, 2024.
Zhao. T., Ning. X., Fang, T., Liu, E., Huang, G., Lin, Z., Yan, S., Dai, G., and Wang, Y. Mixdq: Memory- efficient few-step text-to-image diffusion models with metric-decoupled mixed precision quantization. Eu- ropean Conference on Computer Vision, pp. 285-302. Springer, 2025.
Zheng, X., Liu, X., Bian, Y., Ma, X., Zhang, Y., Wang, J., Guo, J., and Qin, Bidm: Pushing the limit of quantization for diffusion models. arXiv preprint arXiv:2412.05926, 2024a.
Zheng, X., Qin, H., Ma, X., Zhang, M., Hao, H., Wang, J., Zhao, Z., Guo, J., and Liu, X. Binarydm: Towards accurate binarization of diffusion model. arXiv preprint arXiv:2404.05662, 2024b.
12

---

Q-VDIT: Towards Accurate Quantization and Distillation of Vidco-Generation Diffusion Transformers
A. Proof of Theorem 3.2 Lemma A.l. For any surjection f + Y, we have
H(Y) s H(()
(20)
Proof of Lemma A.I. For any surjection
X Y, we have p(y) ((x)=y)(x))))) Thus, we have
H(Y)
E
p()) logp(y)
NEY
Σ p(y)=p(a)
p(y)logp(y)
E
p(y) p(y)
p(u)+p(a)
Σ
p(u) logg())
Σ
Σ
p(x) log x= (w)
Σ p(a))
p())=(())
p())+(())
x=f-1(01
(21)
E
p(x) ((z)logp(1))
Σ
p(x) log
=F(x((())) p(2)
Σ
=((()))=((x)
p((()))t(ar)
S - Σ
logg()
Σ
log
p(((t)))p(z)
r(f(a)))paa)
H(X)
Therefore, Lemma A.1 holds.
Proof of Theorem 3.2.
To simplify the proof, we omit in the quantization process of Eq. (3) and Eq. (4). Therefore, for any weight quantization error A, we have
H(A) H(W Q(W))
H(W ss W
$ 1)
H(ss W
s W
(22)
W W 1)
Hl
where (
denotes only truncating the decimal part of W. For any item W, and W, in W, have
W
Wi
W;
Wa
if W. = W
W.
W
W;
W
if W. + W, and W.
W
W
W
(23)
W:
W.
*
W
WJ 1),
if W, * W and W.
W
Obviously, for mapping g W
W
W g surjection. Therefore, using Lemma A.1, we have
H(A) Hi W W 1) S H(W)
(24)
Therefore, Theorem 3.2 holds.
13

---

Q-VDIT: Towards Accurate Quantization and Distillation of Vidco-Generaaion Diffusion Transformers
Table 6. Performance of text-to-viden generation UCF-101 Dataset.
Bit-width
VQA- Aesthetic
VQA- Technical
A FLOW
Temp.
Method
FVD(L)
FVD-FPI6())
CLIPSIM
CLIP-T
(W/A)
Score.
Flick.
16/16
99.90
0.00
0.1970
0.9963
36.33
91.23
3.37
96.22
Naive-PTQ
4/6
197.85
244.06
0.1702
0.9921
4.98
0.36
66.51
63.72
ViDiT-Q
4/6
96.72
80.59
0.1940
0.9968
20.25
31.73
3.29
94.86
Q-VDiT
4/6
95.51
79.17
0.1944
0.9971
23.62
35.76
2.70
95.46
Naive-PTQ
3/8
220.83
251.74
0.1698
0.9913
3.23
0.17
69.06
61.42
ViDiT-Q
3/8
99.74
84.28
0.1925
0.9957
17.32
20.56
4.34
92.51
Q-VDiT
3/8
97.07
80.14
0.1935
0.9966
21.74
30.23
3.08
94.37
Naive-PTQ
3/6
221.09
252.37
0.1693
0.9910
3.16
0.16
69.85
61.14
ViDiT-Q
3/6
100.05
84.79
0.1923
0.9955
16.83
19.61
4.42
92.08
Q-VDiT
3/6
98.15
82.09
0.1934
0.9963
19.79
25.81
3.46
93.13
B. Implementation Details
We follow the setup used in ViDiT-Q (Zhao et al., 2024) for different layer bits. We only quantize linear weight and use channel-wise quantization for weight, and dynamic token-wise quantization for activation like ViDiT-Q. We apply the same quantization setting for different methods for fair comparison. We follow the quantization scheme used PTQ4DiT (Wu al.. 2024a) and EfficientDM (He al., 2023) perform post-training calibration which fine-tunes the quantization parameters. For post-training quantization, calibration parameters, use batch size of 4, learning rate of 1e-6 for weight quantization parameters, and le-5 for TQE parameters. We apply the same setting for other post-training-bass methods. For calibration dataset, we use 10 prompts provided by Open-Sora (HPC-AI, 2024) and uniformly select 50 steps as used in ViDiT-Q (Zhao et al., 2024).
calibrate 5k iters for 6-8 bit, 10k iters for 4-bit, and 15k iters for 3-bit. For
C. Experimental Settings
For the Latte (Ma al., 2024) Model, due the lack of ground-truth videos for prompt-only datasets, also report FVD-FP16 which chooses the 1p16-generated video as ground-truth. All metrics are evaluated 101 prompts (1 for each class) for UCF-101 (Soomro, 2012).
D. Detailed Description of Selected Evaluation Metrics
D.1. Benchmark Evaluation
Following VBench (Huang et al., 2024b) and previous work ViDiT-Q (Zhao et al., 2024), we select dimensions from three key aspects in nito-ggeneration task.
1. Frame-wise Quality. In this aspect, we assess the quality of each individual frame without taking temporal quality into concern.
Imaging Quality assesses distortion (e.g., over-exposure, noise) presented in the generated frames using the MUSIQ (Ke al., 2021) image quality predictor trained the SPAQ (Fang al., 2020) dataset.
Aesthetic Quality evaluates the artistic and beauty value perceived by humans towards each video frame using the LAION aesthetic predictor (LAION-AI, 2022).
2. Temporal Quality, In this aspect, we assess the cross-frame temporal consistency and dynamics.
Dynamic Degree evaluates the degree of dynamics (i.e., whether it contains large motions) generated by each model.
Motion Smoothness evaluates whether the motion in the generated video is smooth, and follows the physical law of the real world.
Subject Consistency assesses whether the subject's appearance remains consistent throughout the whole video. Background Consistency evaluate the temporal consistency of the background scenes by calculating CLIP (Rad- ford al., 2021) feature similarity across frames.
14

---

Q-VDIT: Towards Accurate Quantization and Distillation of Vidco-Generation Diffusion Transformers
3. Semantics. In this aspect, we evaluate the video's adherence to the text prompt given by the user. consistency.
Scene evaluates whether the synthesized video is consistent with the intended scene described by the text prompt.
Overall Consistency further use overall video-text consistency computed by ViCLIP (Wang et al., 2023) on general text prompts as an aiding metric to reflect both semantics and style consistency.
We use three different prompts set provided by the official github repository of VBench to generate videos. We generate one video for each prompt for evaluation same as ViDiT-Q (Zhao et al., 2024).
overall consistency.txt: includes 93 prompts, used to evaluate overall consistency, aesthetic quality and imaging quality.
subject consistency.txt: includes 72 prompts, used to evaluate subject consistency, dynamic degree, and motion smoothness.
scene.txt: includes 86 prompts, used to evaluate scene and background consistency.
D.2. Multi-aspects Metrics Evaluation
CLIPSIM and CLIP-Temp: CLIPSIM computes the image-text CLIP similarity for all frames in the generated vidcos and we report the averaged results. This quantifies the similarity between input text prompts and generated videos. CLIP-Temp computes the CLIP similarity of each two consecutive frames of the generated videos and then gets the averages on each two frames. This quantifies the semantics consistency of generated videos. We use the CLIP-VIT-B/S3 (Wang et al., 2023) model compute CLIPSIM and CLIP-Temp. We use the implementation from EvalCrafter (Liu et al., 2024) to compute these two metrics.
DOVER'S VQA: VQA-Technical measures common distortions like noise, blur, and over-exposure. VQA-Aesthetic reflects aesthetic aspects such as the layout, the richness and harmony of colors, the photo-realism, naturalness, and artistic quality of the frames. We use the Dover (Wu et al., 2023) method to compute these two metrics.
FLOW Score: Flow score wae proposed in (Liu et al., 2024) to measure the general motion information of the video. We use RAFT (Teed & Deng, 2020), to extract the dense flows of the video in every two frames and we calculate the average flow on these frames to obtain the average flow score of each generated video. In practice and in previous work (Zhao et al., 2024) finds, some poorly performing methods can cause the generated video to crash, resulting in abnormally high FLOW Score. Thus, the difference Flow Score between the quantized Model and the FP Model is used as A FLOW Score for better comparison.
Warping error first obtain the optical flow of each two frames using the pre-trained optical flow estimation network (Teed & Deng, 2020). Then calculate the pixel-wise differences between the warped image and the predicted image. We calculate the warp differences on every two frames and calculate the final score using the average of all the pairs.
Warping Error:
FVD and FVD-FP16: FVD measures the similarity between the distributions of features extracted from real and generated videos. We employ one randomly selected video per label from the UCF-101 dataset (101 videos in total) (Soomro, 2012) as the reference ground-truie videos for FVD evaluation. We follow (Zhao et al., 2024; Blattmann et al., 2023) to use a pretrained 13D model to extract features from the videos. Lower FVD scores indicate higher quality and more realistic video generation. However, due to relatively smaller video size (e.g. 101 videos in our case), employing FVD to evaluate video generation models faces several limitations. Small sample size cannot adequately represent either the diversity of the entire dataset or the complexity and nuances of video generation, leading to inaccurate and unstable results. To mitigate the limitations above, we propose enhanced metric, FVD-FPI6, for assessing the semantic loss in videos generated by quantized models relative to those produced by pre-quantized models. Specifically, we utilize 101 videos generated by the FP16 model as ground-truth reference videos. The FVD-FP16 has significantly higher correlation with human perception.
Temporal Flickering: Temporal flickering measures temporal consistency at local and feghfrreyeency details of gen- erated videos. Then, we calculate the average MAE (mean absolute difference) value between each frame. We use the implementation in VBench (Huang et al., 2024b) to calculate temporal flickering.
15

---

Q-VDIT: Towards Accurate Quantization and Distillation of Vidco-Generation Diffusion Transformers
E. Additional Results on Latte Model
Tab. 6, we present different quantization results about Latte (Ma et al., 2024) model on UCF-101 (Soomro, 2012) dataset. Compared main baseline ViDiT-Q, our Q-VDiT still outperforms across all selected metries under these harder bit settings. Under the hardest W3A6 setting, Q-VDiT still achieves VQA-Acsthetic of 19.79 and VQA-Technical of 25.81 which surpass the current method ViDiT-Q of 16.83 and 19.61.
Table 7. The illustration of Q-DDiT's hardware resource savings.
Method
Bit- Bit-Width (W/A)
Memory Cost
Latency Cost
VQA-Aesthetic
VQA-Technical
16/16
1.00x
1.00x
66.91
53.49
ViDiT-Q
4/8
2.42x
1.38x
60.62
49.38
Q-VDiT
4/8
2.40x
1.35x
71.32
55.56
F. Latency Analysis of TQE
Introduced Token-aware Quantization Estimator (TQE) in Sec. 3.2 will bring extra computation cost in the actual quantization process. This technique is equivalent LoRA (Hu et al., 2021) module with rank 1. SVDQuant (Li al., 2024a) finds that the main bottleneck comes from memory access. Thus, SVDQuant proposes LORUNNER Kernel which fuses the down projection with the quantization kernel and the up projection with the quantization computation kernel, the low-rank branch can share the activations with the low-bit branch, eliminating the extra memory access and also halving the number of kernel calls. As a result, the low-rank branch adds only 5% latency with rank = 16, making it nearly cost-free. In our practice, TQE is only equivalent to rank = 1 which brings less latency burden. Tab. 7, we present the memory cost and inference latency on W4A8 Open-Sora model. Compared to the full full-precision model, our Q-VDiT can still bring 2.40 memory saving and x inference acceleration.
G. More Qualitative Results
In the following pages, we visualize more generative video comparisons across different quantization methods. For better comparison, we uniformly sample 8 frames each video.
16

---

Q-VDIT: Towards Accurate Quantization and Distillation of Vidco-Generation Diffusion Transformers
EP
Q-DiT
Vidit-Q
Q-VDiT
Figure 7. The qualitative results with prompt "A soaring drone footage captures the majestic bcauty of coastal cliff, its red and yellow stratified rock faces rich in color and against the vibrant turquoise of the sea. Scabirds can be seen taking flight around the cliff's precipices. As the drone slowly moves from different angles, the changing sunlight casts shifting shadows that highlight the rugged textures of the cliff and the surrounding calm sea. The water gently laps at the rock base and the greenery that clings to the top of the cliff, and the scene gives a sense of peaceful isolation the fringes of the ocean. The video captures the essence pristine natural beauty untouched by human structures.".
EP
Q-DiT
VidiT-Q
Q-VDiT
Figure 8. The qualitative results with prompt "The video captures the majestic heauty of a waterfall cascading down a cliff into a serene lake. The waterfall, with its powerful flow, is the central focus of the video. The surrounding landscape is lush and green, with trees and foliage adding to the natural beauty of the scene. The camera angle provides a bird's eye view of the waterfall, allowing viewers to appreciate the full height and grandeur of the waterfall. The video is a stunning representation of nature's 's power and beauty.".
17

---

Q-VDIT: Towards Accurate Quantization and Distillation of Vidco-Generation Diffusion Transformers
EP
Q-DiT
VidiT-Q
Q-VDiT
Figure 9. The qualitative results with prompt "The vibrant beauty of sunflower field, The sunflowers, with their bright yellow petals and dark brown centers, are in full bloom, creating stunning contrast against the green leaves and stems. The sunflowers arranged in neat rows, creating sense order and symmetry. The sun is shining brightly, casting a warm glow on the flowers and highlighting their intricate details. The video is shot from low angle, looking up at the sunflowers, which adds sense of grandeur and the scene. The sunflowers are the main focus of the video, with no other objects or people present. The video is a celebration of nature's beauty and the simple joy of sunny day in the countryside.".
FP
Q-DiT
ViDiT-Q
Q-VDiT
Figure 10. The qualitative results with prompt "A serene underwater scene featuring a sea turtle swimming through coral reef. The turtle, with its aecnish-browe shell, is the main focus of the video, swimming gracefully towards the right side of frame. The coral reef, teeming with life, is visible in the background, providing vibrant and colorful backdrop to the turtle's journey. Several small fish, darting around the turtle, add sense of movement and dynamism the seene. The video is shot from slightly elevated angle, providing comprehensive view of the turtle's 's surroundings. The overall style of the video is calm and peaceful, capturing the beauty and tranquility of the underwater world."
18

---

Q-VDIT: Towards Accurate Quantization and Distillation of Vidco-Generation Diffusion Transformers
EP
Q-DiT
VidiT-Q
Q-VDiT
Figure 11. The qualitative results with prompt "A vibrant underwater scene. A group of blue fish, with yellow fins, are swimming around a coral reef. The coral reef is a mix of brown and green, providing a natural habitat for the fish. The water is deep blue, indicating a depth of around 30 feet. The fish are swimming in circular pattern around the coral reef, indicating sense of motion and activity. The overall seene is beautiful representation marine life.".
FP
Q-DIT
ViDiT-Q
Q-VDiT
Figure 12. The qualitative results with prompt "A bustling city street at night, filled with the glow of car headlights and the ambient light of streetlights. The scene is a blur of motion, with cars speeding by and pedestrians navigating the crosswalks. The cityscape is mix of towering buildings and illuminated signs, creating vibrant and dynamic atmosphere. The perspective of the video is from high angle, providing bird's eye view of the street and its surroundings. The overall style of the video is dynamic and energetic, capturing the essence urban life at night.".
19

---

Q-VDIT: Towards Accurate Quantization and Distillation of Vidco-Generation Diffusion Transformers
EP
Q-DiT
vidiT-Q
Q-VDIT
Figure 1.3. The qualitative results with prompt "A snowy forest landscape with dirt road running through it. The road is flanked by trees covered in snow, and the ground is also covered snow. The sun shining, creating bright and serene atmosphere. The road appears to be empty, and there beauty of the snowy forest and the peacefulness of the road.".
people animals visible in the video. The style of the video is natural landscape shot, with focus the
FP
Q-DIT
ViDiT-Q
Q-VDiT
Figure 14. The qualitative results with prompt "The dynamic movement of tall, wispy grasses swaying the wind. The sky above filled with clouds, creating dramatic backdrop. The sunlight pierces through the clouds, casting a warm glow on the scene. The grasses are mix of green and brown, indicating change in scasons. The overall style of the video is naturalistic, capturing the beauty of the landscape in a realistic manner. The focus is on the grasses and their movement, with the sky serving as secondary element, The video does contain any human or animal elements.".
20

---

Q-VDIT: Towards Accurate Quantization and Distillation of Vidco-Generation Diffusion Transformers
FP
Q-DiT
VidiT-Q
Q-VDIT
Figure 15. The qualitative results with prompt "A serene night scene in forested The first frame shows tranquil lake reflecting the star-fillee filled sky above. The second frame reveals a bcautiful sunset, casting a warm glow over the landscape. The third frame showcases the night sky, filled with stars and vibrant Milky Way galaxy. The video is time-lapse, capturing the transition from day to night, with the lake and forest serving as a constant backdrop. The style of the video is naturalistic, emphasizing the beauty of the night sky and the peacefulness of the forest.".
21


Reviews:
[{"review_id": "DGH9ClUw6z", "reviewer": "Reviewer_paDY", "summary": "Diffusion transformers (DiT) are powerful for video generation but face deployment challenges due to large parameter sizes and high computational complexity. To tackle the issues of information loss and mismatched objectives during quantization, the authors propose Q-VDiT, introducing the Token aware Quantization Estimator (TQE) to mitigate quantization errors and the Temporal Maintenance Distillation (TMD) to preserve spatiotemporal correlations across frames. Their W3A6 Q-VDiT achieves a scene consistency score of 23.40, surpassing current state-of-the-art quantization methods by 1.9.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our manuscript and providing valuable suggestions. Here are our responses to some of the concerns you raised:\n> Q1: Original model’s qualitative results.\n\nWe apologize for our negligence and we have released all original videos in [[https://anonymous.4open.science/r/Generated_videos]](https://anonymous.4open.science/r/Generated_videos-EB77). We have also shown more generated critical video frames under W3A6 setting comparison with FP model in Appendix Sec. H. In W3A6 setting, current methods cannot even produce meaningful videos. Our method is significantly closer to FP model in terms of generation quality than existing methods.\n\n> Q2: Overall quality improvement.\n\nWe have reported the same higher bit setting (e.g. W6A6) with ViDiT-Q in Appendix Tab. 5 and our method can achieve lossless performance compared with FP model. Since the performance gap is minor in higher bit settings, we further explore the performance improvement on lower bit settings (e.g., W3A6). Under this low bit setting, we have achieved state-of-the-art and **significantly outperform existing methods and existing methods can hardly generate reasonable videos under low bit setting.** We show more visual comparison in Fig. 5 and Appendix Sec. H.\n\n**We further conducted W4A6 quantization experiments on larger models HunyuanVideo and CogVideoX, and demonstrated the generated video effects in [[https://anonymous.4open.science/r/Generated_videos]](https://anonymous.4open.science/r/Generated_videos-EB77)**. The visual quality of our method is closer to the full precision model. We obtained better results than baseline method ViDiT-Q.\n\nWe also present quantitative comparisons in the table below：\n\n|Model|Method|Imaging Quality($\\uparrow$)|Aesthetic Quality($\\uparrow$)|Overall Consistency($\\uparrow$)|\n|-|-|-|-|-|\n|CogVideoX|FP|61.80|58.88|26.46|\n|CogVideoX|ViDiT-Q|46.03|45.31|21.65|\n|CogVideoX|**Ours**|**52.13**|**49.86**|**23.75**|\n|Hunyuan|FP|62.30|62.49|26.85|\n|Hunyuan|ViDiT-Q|52.28|55.25|24.81|\n|Hunyuan|**Ours**|**57.42**|**57.04**|**25.49**|\n\nOur method is closer to the FP model in terms of metrics and visual effects, and has a significant improvement compared to the baseline method ViDiT-Q.\n\n> Q3: Additional lower-level metrics (e.g., FVD).\n\nWe have reported FVD on UCF-101 dataset in Appendix Tab. 6. We also report FVD on OpenSORA prompt set used in Tab. 1 and Tab. 2:\n\n|Method|Bit|FVD($\\downarrow$)|VQA-Aesthetic($\\uparrow$)|\n|-|-|-|-|\n|FP|16|101.9|66.91|\n|ViDiT-Q|W4A6|132.6|54.66|\n|**Ours**|W4A6|**103.6**|**67.05**|\n|ViDiT-Q|W3A6|251.8|39.82|\n|**Ours**|W3A6|**191.1**|**53.53**|\n\nCompared to baseline method ViDiT-Q, our method achieved lower FVD, demonstrating the consistent superiority of our method on both high and low level metrics.\n\n> Q4: Inference speed comparison.\n\nWe have reported the inference efficiency of W4A8 model compared with FP model in Appendix Tab. 7. We also display the data here.\n|Method|Memory Cost($\\downarrow$)|Latency Cost($\\downarrow$)|VQA-Aesthetic($\\uparrow$)|VQA-Technical($\\uparrow$)|\n|-|-|-|-|-|\n|FP|10.9G|51.3s|66.91|53.49|\n|**Ours**|**4.5G (2.4$\\times$)**|**38.0s (1.35$\\times$)**|**71.32**|**55.56**|\n\nCompared to FP model, our method achieves complete lossless performance while bringing 2.4$\\times$ reduction in memory cost and 1.35$\\times$ inference acceleration. In the revised version, we will modify our layout to make this more prominent.\n\n> Q5: Motivation for temporal distillation.\n\nWe quantitatively compare our distillation method with temporal differences:\n\n|Method|VQA-Aesthetic($\\uparrow$)|VQA-Technical($\\uparrow$)|\n|-|-|-|\n|No distillation|45.67|38.42|\n|Temporal differences|46.15|53.81|\n|**Ours**|**54.92**|**61.59**|\n\nAs we discussed in Sec. 3.3, our motivation is **to perceive the inter-frame information of the whole video in distillation process. We hope to consider the overall information of the video while optimizing single frame information.** But direct MSE calculates different frame information separately as shown in Eq. 12. Although temporal differences can improve performance to some extent, our method achieved better results. **Concatenating all frames together can directly model the information of different frames in the whole video, which actually includes temporal differences between frames.** Our method can directly model the relationship matrix between all frames instead of simply the gap between two frames, so it can capture the global optimization information of the video well and achieve better performance.", "claims_and_evidence": "Yes. The claims are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Make lots of sense but missing some love level comparisons.", "theoretical_claims": "Yes, I have checked the proofs for theoretical claim regarding all equations.", "experimental_designs_or_analyses": "Yes, I verified that the experimental designs and analyses were sound. The main findings rely on high-level metrics.", "relation_to_broader_scientific_literature": "The paper makes a valuable contribution to the field of video generation by addressing the issue of large, slow models, and demonstrating how quantization can significantly speed up inference.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\n\nThe idea is both good and novel.\n\nThe quantitative results are notably strong.\n\nThe writing is easy to follow.\n\nWeaknesses:\n\nThe supplementary material does not show the original model’s qualitative results, and overall quality does not seem improved compared to other methods.\n\nAlthough the high-level metrics are strong, the presented video exhibits noticeable blurriness; reporting additional lower-level metrics (e.g., FVD) could provide a more comprehensive evaluation.\n\nInference speed comparisons are missing, making it difficult to assess practical efficiency.\n\nThe motivation for concatenating each frame with all frames for temporal distillation is unclear, especially given potential alternatives (e.g., temporal differences).", "other_comments_or_suggestions": "NA", "questions_for_authors": "Please see above parts", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QmaBYKr4sN", "reviewer": "Reviewer_WtXR", "summary": "The paper introduces Q-VDiT, a quantization framework for video DiT to reduce computational complexity while preserving video quality. It addresses two key challenges: quantization error compensation through a Token-aware Quantization Estimator (TQE) and spatiotemporal consistency via Temporal Maintenance Distillation (TMD).\n\n## update after rebuttal\nThanks to the authors for addressing my concerns and providing additional results. I will keep my score. The videos on Hunyuan/opensora look great.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review of our work. Here are our responses to your concerns:\n> Q1: Statement of our Q-VDiT.\n\nWe apologize for the misunderstanding caused by our statement. **We absolutely do not deny ViDiT-Q's contribution. ViDiT-Q is the first method to explore quantization for video generation models and an important baseline method for our paper. We greatly appreciate the contribution of ViDiT-Q.** What the statement wants to emphasize is that we hope to consider the correlation between different frames in the video information during the optimization process and further improve the quality of the quantized video generation model. We will modify our wording in the revised version to avoid this ambiguity.\n\n> Q2: Quantitative analysis on quantization error and information entropy introduced in TQE.\n\nWe add quantitative experiments on W3A6 model last layer weight's  quantization error and information entropy mentioned in the proposed TQE (Sec. 3.2)：\n    \n|Method|Quantization Error($\\downarrow$)|Weight Entropy($\\uparrow$)|VQA-Aesthetic($\\uparrow$)|\n|-|-|-|-|\n|FP|-|6.98|66.91|\n|ViDiT-Q|73.7|4.46|39.82|\n|**Ours**|**56.0**|**5.49**|**53.53**|\n\n**Consistent with our claim, our method indeed reduces quantization errors and improves the information entropy of the quantized weights.** From quantitative metrics, TQE has indeed improved the performance of the model by reducing quantization error and increasing entropy. Theorem 3.2 states that the quantization error of weights has lower information entropy compared to the original weights. This is also proved by the quantitative results. \n\n> Q3: Explanation of quantization settings.\n\nNaturally, lower bit quantization brings more memory saving and acceleration for real-world deployment, but often faces more severe performance degradation, which is a harder situation under exploration. Since existing methods like ViDiT-Q have achieved almost lossless performance at higher bits (e.g., W4A8), we want to further explore the performance improvement at lower bits. Compared to W4 quantization, W3 usually faces severe performance degradation, which is commonly discovered in LLM quantization [1][2]. So we chose lower bit quantization settings (e.g., W3A6) in Tab. 1 and Tab. 2, under which existing methods can hardly generate reasonable videos as shown in Fig. 5. Our method has greatly improved in terms of metrics and visual effects compared to existing methods. **We want to note that we also reported the same higher bit settings as ViDiT-Q in Appendix Tab. 5**, and our method still has improvement compared to existing methods and **achieves lossless**. \n\n[1].GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.\n\n[2].Quarot: Outlier-free 4-bit inference in rotated llms.\n\n> Q4: Performance gap between W3A8 and W4A8.\n\nCompared to W4, the significant loss of weight information in W3 (only 2\\**3=8 representation candidates for a single value) leads to a serious performance degradation as we discussed in Q3. This also highlights the importance of our proposed TQE in compensating for the weight quantization errors. **Our method outperforms current quantization methods in both W3 and W4 (Tab. 1 and Tab. 2)**.\n\n> Q5: Motion dynamics before and after quantization.\n\nExisting methods like ViDiT-Q have also found that the quantized model exhibits a certain degree of dynamic decline. We can also see from the metrics Motion Smoothness and Dynamic Degree in Tab. 1 that this is particularly severe after W3 quantization. **Our method has the same level of motion dynamics as the full precision model in terms of metrics**, which can be seen from the Motion Smoothness and Dynamic Degree metrics in Tab. 1. And our method also shows significant improvement in dynamics compared to existing methods. More generated videos in [[https://anonymous.4open.science/r/Generated_videos]](https://anonymous.4open.science/r/Generated_videos-EB77) also prove the dynamic retention ability of our method at the same level as the FP model.\n\n> Q6: Evaluation on larger video generation models.\n\nWe add W4A6 quantization experiment on larger SOTA models CogVideoX-5B and HunyuanVideo-13B:\n\n|Model|Method|Imaging Quality($\\uparrow$)|Aesthetic Quality($\\uparrow$)|Overall Consistency($\\uparrow$)|\n|-|-|-|-|-|\n|CogVideoX|FP|61.80|58.88|26.46|\n|CogVideoX|ViDiT-Q|46.03|45.31|21.65|\n|CogVideoX|**Ours**|**52.13**|**49.86**|**23.75**|\n|Hunyuan|FP|62.30|62.49|26.85|\n|Hunyuan|ViDiT-Q|52.28|55.25|24.81|\n|Hunyuan|**Ours**|**57.42**|**57.04**|**25.49**|\n\n**We also provide more generated video comparisons in [[https://anonymous.4open.science/r/Generated_videos]](https://anonymous.4open.science/r/Generated_videos-EB77). Our method is closer to the FP model in terms of metrics and visual effects, and has notable improvement compared to the baseline method ViDiT-Q.**", "claims_and_evidence": "The statement that existing approaches fail to calibrate quantization from a video-wide perspective, leading to degraded video quality is too strong. ViDiT-Q does incorporate video tokens and reports results on video datasets, making it inaccurate to claim that it does not consider the entire video. The distinction should focus on how Q-VDiT improves over ViDiT-Q rather than dismissing prior work outright.", "methods_and_evaluation_criteria": "The Token-aware Quantization Estimator (TQE) is introduced to approximate quantization errors across two orthogonal dimensions: token and feature space. However, its quantization error reduction is not directly evaluated. While Figure 3 provides an example and Table 3 includes an ablation study, the effectiveness of TQE in minimizing quantization error remains unclear. A more explicit quantitative evaluation of the error reduction would strengthen the argument.", "theoretical_claims": "Proposition 3.1 and Theorem 3.2 establish that quantized weights retain less information entropy than the original weights. However, the link between this theoretical result and the practical benefits of TQE is not clearly articulated. It remains unclear which specific error terms TQE reduces and by how much. Providing quantitative results on TQE’s impact on entropy reduction would help substantiate its contribution.", "experimental_designs_or_analyses": "- The paper adopts a different quantization setting from ViDiT-Q but lacks a clear justification. The claim that \"we mainly focus on harder settings\" does not sufficiently explain the reasoning behind the choice. Further insights into why these settings are chosen, particularly regarding their relevance to real-world deployment and model robustness, would improve the clarity of the paper.\n- The paper does not explicitly discuss the performance gap between W3A8 and W4A8 (in ViDiT-Q). Given the significant differences in VBench scores between this paper and ViDiT-Q, a more detailed comparison is necessary. Clarifying whether W3A8 introduces substantial quality degradation compared to W4A8 would help readers interpret the reported results.", "relation_to_broader_scientific_literature": "not related", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "n/a", "other_comments_or_suggestions": "n/a", "questions_for_authors": "- why the generated videos exhibit low motion dynamics? Do we observe this before using quantization?\n- The paper does not evaluate on larger video generation models, such as Hunyuan (https://github.com/Tencent/HunyuanVideo), why? The results could be more meaningful on these SOTA models.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EwIry0rzHH", "reviewer": "Reviewer_ZxGk", "summary": "This paper addresses the issue of information loss and misalignment of optimization objectives that arise when applying existing quantization methods to video generation models. Current quantization techniques, which are primarily designed for image generation models, may not be directly suitable for video generation due to the temporal dependencies inherent in videos. To address this challenge, the paper proposes a Token-aware Quantization Estimator (TQE) and a Token-aware Quantization Estimator (TQE). The former aims to reduce information loss by considering token-wise importance during quantization, while the latter focuses on minimizing the temporal distribution discrepancy between the full-precision model and the quantized model. Experimental results demonstrate that the proposed approach effectively mitigates information loss compared to existing quantization methods when applied to video generation models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your high recognition of our work and the valuable suggestions you provided. Our response is as follows:\n> Q1: Quantitative analysis on quantization error.\n\nWe add quantitative experiments on W3A6 model last layer weight's  quantization error and information entropy mentioned in the proposed TQE (Sec. 3.2)：\n    \n|Method|Quantization Error($\\downarrow$)|Entropy($\\uparrow$)|VQA-Aesthetic($\\uparrow$)|\n|-|-|-|-|\n|FP|-|6.98|66.91|\n|ViDiT-Q|73.7|4.46|39.82|\n|**Ours**|**56.0**|**5.49**|**53.53**|\n\n**Consistent with our claim, our method indeed reduces quantization errors and improves the information entropy of the quantized weights.** From quantitative metrics, TQE has indeed improved the performance of the model by reducing quantization error and increasing entropy. Theorem 3.2 states that the quantization error of weights has lower information entropy compared to the original weights. This is also proved by the quantitative results.\n\n> Q2: Actual inference efficiency and memory consumption.\n\nWe have reported the inference efficiency of W4A8 model compared with FP model in Appendix Tab. 7. We also display the data here.\n|Method|Memory Cost($\\downarrow$)|Latency Cost($\\downarrow$)|VQA-Aesthetic($\\uparrow$)|VQA-Technical($\\uparrow$)|\n|-|-|-|-|-|\n|FP|10.9G|51.3s|66.91|53.49|\n|**Ours**|**4.5G (2.4$\\times$)**|**38.0s (1.35$\\times$)**|**71.32**|**55.56**|\n\nCompared to FP model, our method achieves complete lossless performance while bringing 2.4$\\times$ reduction in memory cost and 1.35$\\times$ inference acceleration. In the revised version, we will modify our layout to make this more prominent.\n\n> Q3: In Tab. 4, why ViDiT-Q has training cost?\n\n\nViDiT-Q requires calculating quantization sensitivity between layers to allocate different bit-widths, which incurs additional time consumption. This is also reported in their paper. We will modify our wording to avoid ambiguity.\n\n> Q4: Fig.2 misses some notations.\n\nWe will fix the problem in the final version.\n\n> Q5: Similarity function used in Eq. 9.\n\nWe use cosine similarity.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "yes, the formulas and proofs seem correct.", "experimental_designs_or_analyses": "yes, I have reviewed all the experiments presented in the paper, which are all necessary. However, the analysis of the experiments is not sufficiently detailed.\n\n1.\tAlthough the proposed method outperforms state-of-the-art methods in terms of final generation metrics, it remains unclear whether the improvements genuinely stem from the reduction in quantization error or from other factors. Further analysis is needed to verify the source of the performance gains.\n2.\tIn the experiment, the paper mainly compares the video generation performance at different bit-widths but does not provide a thorough analysis of the acceleration effect or memory savings. This lack of evaluation makes it difficult to assess the actual efficiency improvements brought by the proposed method.\n3.\tIn Table.4, as ViDiT-Q is a calibration-free method, why does it still have training cost? In this table, it would be better to show the memory consumption of the full-precision and the fp16 models, which would help demonstrate the efficiency of the proposed method.", "relation_to_broader_scientific_literature": "Current quantization methods are primarily designed for image generation models, whereas the method proposed in this paper introduces a quantization approach specifically for compressing and accelerating video generation models.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n1.\tThe writing in the paper is relatively clear, making it easy to follow the motivation and the proposed method.\n2.\tThe proposed method outperforms the state-of-the-art methods on public datasets. Additionally, the ablation study validates the effectiveness of each component.\n\n Weaknesses:\nplease see the comments and questions in different sections.", "other_comments_or_suggestions": "In Figure 2, some symbols or notations appear to be missing, which may affect the clarity and completeness of the illustration. It would be helpful to ensure all necessary symbols are properly displayed.", "questions_for_authors": "What similarity function is used in Eq. 9", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QVaX0qwzjg", "reviewer": "Reviewer_Xj8f", "summary": "This paper proposed a quantization method named Q-VDiT tailored specifically for video Diffusion Transformers. The proposed Q-VDiT aims to address severe model quantization information loss in video models. Specifically, the authors proposed Token-aware Quantization Estimator to compensate for quantization errors from both token and feature dimensions. Temporal Maintenance Distillation is used to optimize each frame from the perspective of the overall video. Extensive experiments demonstrate the superiority of the proposed Q-VDiT over baseline and other previous quantization methods.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments on our paper. Regarding the concerns, we provide the following responses.\n> Q1: Optimization details.\n\nSorry for the misunderstanding, **we have reported optimization details including training data in Appendix Sec. B. and show the training cost in Tab. 4**. We will make this more prominent in revised version.\n> Q2: Original videos.\n\nWe apologize for our negligence and we have released all original videos in [[https://anonymous.4open.science/r/Generated_videos]](https://anonymous.4open.science/r/Generated_videos-EB77). **We also have shown more generated critical video frames under W3A6 setting comparison with FP model in Appendix Sec. H. In W3A6 setting, current methods cannot even produce meaningful videos. Our method is significantly closer to FP in terms of generation quality than existing methods.**\n> Q3: Some temporal flicking.\n\nThe OpenSora model we experimented in paper inevitably has some temporary flicking. But our method is significantly better than existing methods in terms of metrics (see Tab. 1 and Tab. 2) and closer to the FP model in terms of visual effects (see Fig. 5, Appendix Sec. H). **We further conducted W4A6 quantization experiments on larger models HunyuanVideo and CogVideoX, and demonstrated the generated video effects in [[https://anonymous.4open.science/r/Generated_videos]](https://anonymous.4open.science/r/Generated_videos-EB77). The videos we generate do not have issues such as temporary flicking and the visual quality is closer to the full precision model. We also has notable improvement compared to the baseline method ViDiT-Q.**\n\nWe also present quantitative comparisons in the table below：\n\n|Model|Method|Imaging Quality($\\uparrow$)|Aesthetic Quality($\\uparrow$)|Overall Consistency($\\uparrow$)|\n|-|-|-|-|-|\n|CogVideoX|FP|61.80|58.88|26.46|\n|CogVideoX|ViDiT-Q|46.03|45.31|21.65|\n|CogVideoX|**Ours**|**52.13**|**49.86**|**23.75**|\n|Hunyuan|FP|62.30|62.49|26.85|\n|Hunyuan|ViDiT-Q|52.28|55.25|24.81|\n|Hunyuan|**Ours**|**57.42**|**57.04**|**25.49**|\n\nOur method is closer to the FP model in terms of metrics and visual effects, and has a significant improvement compared to the baseline method ViDiT-Q.\n\n> Q4: Performance of our quantized model compared with FP model.\n\n**1. Some performance gap under lower bit setting.** We have reported the same higher bit setting (e.g. W6A6) with ViDiT-Q in Appendix Tab. 5 and our method can achieve lossless performance compared with FP model. Since the performance gap is minor in higher bit settings, we further explore the performance improvement on lower bit settings (e.g., W3A6). Under this low bit setting, we have achieved state-of-the-art and **significantly outperform existing methods and existing methods can hardly generate reasonable videos under low bit setting.** We show more visual comparison in Fig. 5 and Appendix Sec. H.\n\n**2. Choice for mainly focus on lower bit.** Naturally, lower bit quantization brings more memory saving and acceleration for real-world deployment, but often faces more severe performance degradation which is a harder situation under exploration. Since existing methods like ViDiT-Q have achieved almost lossless performance at higher bits (e.g., W4A8), we want to further explore the performance improvement at lower bits. Compared to W4 quantization, W3 usually faces severe performance degradation which is commonly discovered in LLM quantization [1][2]. So we chose lower bit quantization settings (e.g., W3A6) in Tab. 1 and Tab. 2, under which existing methods can hardly generate reasonable videos as shown in Fig. 5. Our method has greatly improved in terms of metrics and visual effects compared to existing methods. **We want to note that we also reported same higher bit settings as ViDiT-Q in Appendix Tab. 5**, and our method still has improvement compared to existing methods and **achieves lossless**. \n\n[1].GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.\n\n[2].Quarot: Outlier-free 4-bit inference in rotated llms.\n\n**3. Main contribution.** We would like to further highlight our contribution by investigating the limitations of current quantization methods in video generation models and solving them from both quantization (Sec. 3.2) and optimization (Sec. 3.3) perspectives. With almost the same calibration cost with current methods (Tab. 4), our method brings significant relatively performance improvement at lower bit compared to existing methods in terms of metrics (Tab. 1 and Tab. 2) and visual effects (Fig. 5 and Appendix Sec. H) while maintaining lossless performance at higher bit (Appendix Tab. 5). Meanwhile, our method can bring 2.4x reduction in memory cost and 1.35x actual acceleration for inference (Appendix Tab. 7) with no extra burden compared with baseline ViDiT-Q.", "claims_and_evidence": "This paper claims that \"Quantization can reduce storage requirements and accelerate inference by lowering the bit-width of model parameters. Yet, existing quantization methods for image generation models do not generalize well to video generation tasks.\". The claims are supported by experimental results.", "methods_and_evaluation_criteria": "The authors proposed Token-aware Quantization Estimator to compensate for quantization errors from both token and feature dimensions. Temporal Maintenance Distillation is used to optimize each frame from the perspective of the overall video. Extensive experiments demonstrate the superiority of the proposed Q-VDiT over baseline and other previous quantization methods.", "theoretical_claims": "Yes. Proof of Theorem 3.2 is reasonable.", "experimental_designs_or_analyses": "Yes. Quantization Estimator to compensate for quantization errors from both token and feature dimensions. Temporal Maintenance Distillation is used to optimize each frame from the perspective of the overall video.", "relation_to_broader_scientific_literature": "This paper proposed a quantization method named Q-VDiT tailored specifically for video Diffusion Transformers. The proposed Q-VDiT aims to address severe model quantization information loss in video models. Specifically, the authors proposed Token-aware Quantization Estimator to compensate for quantization errors from both token and feature dimensions. Temporal Maintenance Distillation is used to optimize each frame from the perspective of the overall video. Extensive experiments demonstrate the superiority of the proposed Q-VDiT over baseline and other previous quantization methods.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Pros:\n\n1.\ta quantization method named Q-VDiT tailored specifically for video Diffusion Transformers is proposed.\n2.\tToken-aware Quantization Estimator is proposed to compensate for quantization errors from both token and feature dimensions.\n3.\tTemporal Maintenance Distillation is used to optimize each frame from the perspective of the overall video\n4.\tExperimental results verify the effectiveness of the proposed method.\n\nCons:\n\n1.\tMore optimization details of Temporal Maintenance Distillation should be included such as training data and resources.\n2.\tThe video demos generated by the proposed method, included in the supplementary material, show observed artifacts and temporal flicking, the authors should give some deeper explanations.\n3.\tThe original videos generated by the open-sora model are not displayed, and it seems that the proposed  Quantization method can not preserve the performance of the original models.", "other_comments_or_suggestions": "My major concern is the unsatisfied video generation performance.", "questions_for_authors": "Please refer to Weaknesses for more details.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a quantization framework designed specific for video diffusion transformers. The proposed method, Q-VDiT, is designed to address two challenges, the loss of information during quantization and the misalignment between optimization objectives and the unique requirements of video generation. It introduces Token aware Quantization Estimator (TQE) and Temporal Maintenance Distillation (TMD) to solve them respectively. Experiments show superiority of the new method again baselines in visual quality and quantitative performance in both high and low bitwidth settings.

The reviewers appreciate its clear presentation and strong empirical performance. The rebuttal addressed multiple concerns in the initial reviews, including the lack of low level metrics and analysis, lack of the original videos and speed comparison, and provided additional experiments on larger video generative models. It also explained the remaining artefacts observed in the generated videos. Although the reviewers did not participate in the post-rebuttal discussion, most of their concerns appear to have been sufficiently addressed.
Decision: Accept (poster)