VETA-DiT: Variance-Equalized and Temporally Adaptive Quantization for Efficient 4-bit Diffusion Transformers
Authors: QinkaiXu
yijin liu
YangChen
Lin Yang
Li Li
Yuxiang Fu
Venue: NeurIPS 2025 poster ()
Keywords: Diffusion Transformer
Post-training Quantization
Image and Video Generation
Efficient Inference
Primary area: deep_learning

Abstract: Diffusion Transformers (DiTs) have recently demonstrated remarkable performance in visual generation tasks, surpassing traditional U-Net-based diffusion models by significantly improving image and video generation quality and scalability. However, the large model size and iterative denoising process introduce substantial computational and memory overhead, limiting their deployment in real-world applications. Post-training quantization (PTQ) is a promising solution that compresses models and accelerates inference by converting weights and activations to low-bit representations. Despite its potential, PTQ faces significant challenges when applied to DiTs, often resulting in severe degradation of generative quality. To address these issues, we propose VETA-DiT (**V**ariance-**E**qualized and **T**emporal **A**daptation for **Di**ffusion **T**ransformers), a dedicated quantization framework for DiTs. Our method first analyzes the sources of quantization error from the perspective of inter-channel variance and introduces a Karhunen–Loève Transform enhanced alignment to equalize variance across channels, facilitating effective quantization under low bit-widths. Furthermore, to handle the temporal variation of activation distributions inherent in the iterative denoising steps of DiTs, we design an incoherence-aware adaptive method that identifies and properly calibrates timesteps with high quantization difficulty. We validate VETA-DiT on extensive image and video generation tasks, preserving acceptable visual quality under the more aggressive W4A4 configuration. Specifically, VETA-DiT reduces FID by 33.65 on the DiT-XL/2 model and by 45.76 on the PixArt-$\Sigma$ model compared to the baseline under W4A4, demonstrating its strong quantization capability and generative performance. Code is available at: https://github.com/xululi0223/VETA-DiT.

ViTA-DiT: Variance-Equalized and Temporally Adaptive Quantization for Efficient 4-bit Diffusion Transformers
Qinkai Xu Yijin Liu" Yang Chen Lin Yang Li Li! Yuxiang Fu'
Nanjing University
(qinkaixu, yijinliu, yangchen nju @ ssmail.jj..edu... [linyang. lili, yuxiangful @nju.edu... edu.en
Equal Contribution. Corresppnning Author.
Abstract
Diffusion Transformers (DiTs) have recently demonstrated remarkable perfor- mance in visual generation tasks, surpassing traditional U-Net-based diffusion models by significantly improving image and video generation quality and scal- ability. However, the large model size and iterative denoising process introduce substantial computational and memory overhead, limiting their deployment in real- world applications. Post-training quantization (PTQ) a promising solution that compresses models and accelerates inference by converting weights and activations to low-bit representations. Despite its potential, PTQ faces significant challenges when applied to DiTs, often resulting in severe degradation of generative quality. To address these issues, we propose VETA-DiT (Variance- Euuiice-e and Temporal Adaptation for Diffusion Transformers). dedicated quantization framework for DiTs. Our method first analyzes the sources of quantization error from the per- spective of inter-channel variance and introduces Karhunen-Locks Transform enhanced alignment to equalize variance across channels, facilitating effective quantization under low bit-widths. Furthermore, to handle the temporal variation activation distributions inherent in the iterative denoising steps DiT's we de- sign incoherence-awar adaptive method that identifies and properly calibrates timesteps with high quantization difficulty. We validate VETA-DiT on extensive im- age and video generation tasks, preserving acceptable visual quality under the more aggressive W4A4 configuration. Specifically, VETA-DiT reduces FID by 33.65 on the DiT-XL/2 model and by 45.76 on the PixArt- model compared the base- line under W4A4, demonstrating its strong quantization capability and generative performance. Code is available https//gihbb.ookuuuu0003///E/ DiT
Introduction
Recently, Diffusion Transformers (DiTs) 33
have emerged as the dominant backbone architecture for diffusion models, replacing the U-Net structures [36]. They have been widely adopted in various notable example is OpenAl's SoRA which has attracted significant attention for its remarkable generation quality. Recent studies
generation tasks due their superior performance 8 30
32
51 further demonstrate the impressive capability and scalability of DiTs across modalities.
Despite their success, DiTs face two major limitations: the inherently lengthy itcrative denoising process and the growing model size, both of which lead substantial computational and memory demands. These issues hinder the deployment of DiTs in resource-conee constrained environments and limit their applicability in real-time scenarios. For instance, generating 1024x 1024 resolution image with DiT's can take up to 10 seconds even on a NVIDIA A100 GPU.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

KLT Enhanced Hadlamard Transform for Variance Alignment
outlee
350
caT
hard to quantize
enhanced
220
Hadamard Transform
Hadamard Mantt
Karhunen-Lode
Transforme Matrix
easy to quantize
Calcualte Incoherence- aware scores
synthetical activation for calibration
375
CAT
350
52010
Incoherence-Aven Adaptation for Temporally Varying Data Distributions
53 330
250
Figure 1: (Left) Overview of VETA-DiT. The Hadamard matrix is enhanced by K-L transform to adapt to data distributions and reduce quantization difficulty. Calibration data fed into the KLT synthesized via an incoherence-une aware adaptive strategy to account for temporal variance across different timesteps. (Right) Quantization performance under W4A8 and W4A4, where points closer to the top-left corner indicate better performance. Please refer to Section 4.2 for detailed results.
Model quantization 31119 19] has been widely explored as an effective technique to accelerate infer- ence by reducing memory footprint and computational overhead. It achieves this by compressing model weights and activations from floating-pont point to low-bit bit integer representations. However, the complex data distributions of weights and activations in DiTs pose significant challenges for existing quantization techniques, especially when targeting ultra-low bit-width without severely degrading performance. We identify two key challenges that hinder low-bit quantization of DiTs: (1) the pres- ence extreme outliers in specific channels results in significant inter-channel variance. Traditional using simple Hadamard transforms 49
approaches such applying smoothing factor 46 45
fail to effectively align this variance, making low-bit quantization inaccurate; (2) the inherent iterative denoising nature of DiTs causes large variations in activation distributions across different timesteps, making it difficult to strike balance between quantization effectiveness and computational efficiency,
To address these challenges, propose novel quantization method tailored for DiTs, termed VETA-DÍT. We first analyze the limitations of directly applying the Hadamard transform from the perspective of inter-channel variance, and introduce KLT enhanced Hadamard transform to achieve better variance alignment, enabling acceptable performance even under low-bit scenarios. To handle the temporal variation in activation distributions, we design incoherence-aware adaptive strategy to identify the challenging timesteps for quantization. We then construct a synthetical calibration set that ensures both quantization accuracy and efficiency.
Our contributions are summarized as follows:
1. We analyze the limitations of directly applying the Hadamard transform in DiTs and propose K-L conssform-enhancee Hadamard method to effectively align inter-channel variance.
2. We investigate the quantization difficulty caused by temporal activation variation, and pro- pose an incoherence-awar importance scoring strategy to construct a synthetical calibration set that captures representative distributions across different timesteps.
3. We conduct extensive experiments on both image and video generation tasks with multiple DIT models, and push the quantization precision W4A4, demonstrating effectiveness of VETA-DiT in achieving high quantization performance without sacrificing visual fidelity.
2 Background and Related Works
2.1 Diffusion Transformer
Diffusion Models (DMs) 16 have attracted significant attention due to their powerful generative capabilities in visual domains such as image B3 and video [29) synthesis. DMs simulate

---

forward process that progressively adds Gaussian noise to the original data via Markov chain [5]. perturbing the data into a distribution close to standard Gaussian. learned denoising network then employed in the reverse process to iteratively reconstruct high-quality samples. The denoising process typically parameterized by a deep neural networkeg. which predicts the noise at each timestep. Given sample drawn from standard Gaussian noise 2T ~ (O, T), the model iteratively denoises it through:
po(I(-1(20)) = 11 f).
po 1 (xe Vat
1 - co,,), 3. 1 - -
VI-
Okty
The architecture of the noise prediction network plays a crucial role in determining the performance of diffusion models. DiT is a representative method that integrates Transformer architectures into diffusion models. Its core building block consists of multiple Transformer layers, each composed of
a Multi-Head Belf-Attention (MHSA) mechanism and a Ferd-Fo Network (FFN) 42 Specifically, MHSA and FFN are formulated as follows, respectively:
QK'
T V
M K, V) = Softmaal dd
(3)
FFN LayerNormiX + MLP(X)).
To incorporate conditional information (e.g., class labels), each Transformer block employs MLPs to project a condition vector с E Rda into scale and shift parameters, which are then injected into the hidden state Z E Raxdu via adaptive LayerNorm (adaLN) (3)
(r. = MLPs(c), adaLN = (N + (1+7)++
(5)
Although DiT achieves superior generation quality and expressive power compared to traditional architectures, its inference phase incurs high computational and memory costs due to the deeply stacked Transformer layers and iterative denoising process.
2.2 Model Quantization
Model quantization
19 is a widely adopted technique for reducing model size and accelerating
inference. Quantization methods are typically categorized into Duantization-Aaal Training (QAT) 50
and Post-Training Quantization (PTQ) [TC
46
49
45
5 20 QAT integrates the
quantization process into the training pipeline, often preserving model performance but requiring extensive computational resources and retraining time. contrast, PTQ statically analyzes pretrained models using a small calibration dataset to estimate quantization parameters, offering advantages in deployment speed and computational efficiency.
A typical example is symmetric linear quantization, which can be expressed as:
Fint clamp(1 T z, Pmiin Pmox),
(6)
where r and Wint denote the original and quantized data, respectively; is the scaling factor, is the zero point, 1-1 denotes the rounding operation, and the clamp function constrains the quantized values within [pmin Pmoz)
Although PTQ techniques have demonstrated promising results on architectures such CNNs 21 22 44 and ViT's (23 44 27], directly applying them to DiT' remains challenging. On one hand, DiTs often exhibit significant inter-channel distribution imbalance, making it difficult for unified quantization range to accommodate all channels effectively. On the other hand, the stepwise sampling mechanism in diffusion processes introduces strong temporal dynamics the activation distributions, necessitating quantization strategies that are adaptive to the timestep. To address these challenges, several studies have explored efficient PTQ methods for diffusion models. For instance, Q-Diffusion 21 and PTQ4DM (39] analyze activation variance across time steps to improve temporal robustness, while Q-DiT and PTQ4DiT incorporate joint camporal-channe adaptive quantization mechanisms. ViDiT-Q 49 proposes a unified approach that accounts for both temporal dependency and inter-channel imbalance by leveraging Hadamard rotations and asymmetric formats to enhance quantization effectiveness. However, they suffer from challenges under lower bit-width quantization settings. Recently, SVDQuant (0) employs a high-precision low-rank branch to take in the weight outliers with singular value decomposition, which is orthogonal to our method.
characteristics into the design of
3

---

2000
2000
100
200
Figure 2: (Left) Activation distribution in the DIT model, showing extreme outliers in certain
channels. Distributions after the Hadamard (Middle) and K-L-enhanced Hadamard transforms (Right), respectively. While the Hadamard transform helps reduce variance imbalance, it has limitations; the K-L-enhanced method further improves inter-channel variance alignment.
Method
3.1
KLT Enhanced Hadamard Transform for Variance Alignment
We begin by analyzing the intermediate activations of various layers in the DiT-XL/2 model, focusing on selected linear layers to investigate their activation distribution characteristics, As shown in Figure 2 (Left), the activations exhibit distribution pattern similar those observed in LLMs and DMs 13 [3]] certain channels contain outlier values with extremely large magnitudes. In some layers, the magnitude of the outliers can be up 100 times larger than the rest of the activations. These outliers significantly increase the variance within individual channels, resulting in substantial variance discrepancies across different channels [ 24), Since per-token quantization methods assume uniform variance across channels, such discrepancies introduce considerable quantization errors, which are particularly problematic in hardware deployment.
Previous works in LLM quantization have proposed using the Hadamard transform to mitigate the influence of outliers and reduce inter-channel variance. Formally, given an input tensor X gnxm where is the number of samples and m is the number of channels, the Hadamard transform is defined as: Z XH, where H Rm the Hadamard matrix. The Hadamard matrix H is an orthogonal matrix whose entries are drawn from
+
Let zj denote the j-th column of V
vm
Z, i.e., the j-th transformed channel. Its variance is:
1
Var(z,)
n
Utilizing the linearity of expectation and the properties of variance and covariance, we derive:
Var(z,) Σ Var(
Σ Cou(xx, kgl
(8)
This expression shows that the variance of the transformed channel consists of two components: weighted sum of the original channel variances and a cross-covariance term. Since the cross- covariance term and the corresponding Hadamard weight vector h. vary across channels j. the resulting variances Vzr(z,) cannot assumed to be numerically similar in most cases. As result, the transformed channels have unequal variances, limiting the effectiveness of the Hadamard transform for variance equalization in quantization. As shown in Figure 2 (Middle), the activations after the Hadamard transform alone still exhibit nonnegligible inter-channel variance, which adversely affects the quantization performance.
To achieve better variance alignment, we seek a transformation that ensures uniform average energy across all transformed channels. Let 1g. V Var(x,, ) be the variance of the k-th channel, and define the energy vector v = [Vi, U2.... um Our goal is to find linear transformation T E m such that the transformed tensor Y XT satisfies:
m
Var(y,)
uk
Vj.
m

---

Max Aboolute Valu
Timestop
Figure 3: Boxplot of the maximum absolute activation values across channels at different timesteps in DiT, illustrating significant temporal variance that affects the effectiveness and efficiency of quantization methods.
That is, all output channels have the same average variance, maximizing the quantization range and minimizing quantization error.
This can be achieved using Karhunen-Loève Transform (KLT), which constructs orthogonal basis from the eigenvectors of the input covariance matrix []. The KLT distributes total variance across orthogonal directions, minimizing redundancy and maximizing statistical independence. It has also been applied improve quantization performance in state space models [48]. By combining KLT with the structured and computationally efficient Hadamard transform, choose composite transformation called the KLT enhanced Hadamard transform (KLT-H): TKLT-H = KH, where K is the KLT matrix composed of eigenvectors of the input covariance matrix. Since both K and H are orthogonal, the composite transformation T TKLT- remains orthogonal. Its inverse simply KITHH which enables computational consistency by ensuring that the quantized model produces mathematically equivalent outputs to the original model, as illustrated by XXTKTTH)TKK KLT-H W) XW. After transformation, the variance of each channel becomes:
m
Var(y,) =
Ag
(10)
where A the eigenvalues of the input covariance matrix. For normalized Hadamard matrices, 2 1. leading to: kj = m.
m
Var(y,)
Ak
(11)
which indicates that the variance becomes equalized across different channels. After the KLT enhanced Hadamard transform, the inter-channe variance is further reduced, as illustrated in Figure 2 (Right). Thus, the transformed tensor achieves variance alignment across channels, significantly reducing quantization error and enhancing the robustness of diffusion model inference under low-bit quantization.
3.2 Incoherence-Aware Adaptation for Temporally Varying Data Distributions
DiT models generate images through an iterative denoising process. As shown in Figure 3 we observe that the activation distributions different timesteps during the denoising process vary significantly. Previous quantization methods [26] 49) typically apply the same transformation matrix H all timesteps within layer. While this approach ensures computational efficiency. it often leads to severe performance degradation under low-bit quantization. Unfortunately. the K-L transformation matrix is highly sensitive the distribution characteristics of the calibration dataset, and temporal variance in activation distributions undermine its effectiveness.
straightforward solution is to compute dedicated K-L transformation matrix for each timestep. However, this would incur substantial storage and computation overhead, counteracting the efficiency benefits of quantization.
To address this challenge, we propose an incoherence-aware, adapiiee strategy that improves the performance of quantized models while preserving computational efficiency. The concept of
5

---

incoherence, adapted from [ 41]. is used to measure the degree of anomaly in activation data. Specifically, matrix X mmx" said be j--incoherent if for any and j:
|X
X | Xeyl S
(12)
Vmn
where -|= denotes the Frobenius norm. A higher incoherence value indicates that certain channel activations deviate significantly from the global mean, suggesting skewed distributions and outliers that complicate the quantization process.
During quantization, assign each timestep importance St, which guides the KLT to pay more attention to activation data with highly skewed distributions. The score is defined as:
max X
St Incoherence X (0)
(13)
XX
F/ ///m
where (+) denote the activation data sampled at timestep However, since incoherence values can differ drastically across timesteps, directly using 8; may cause certain timesteps to dominate the calibration statistics. To mitigate this, normalize the scores such that the sum of weights across all timesteps equals 1, ensuring a balanced contribution. The normalized form is given by:
(14)
1==1 &
where >> is control parameter. When 1, the weighting is linear; higher values of emphasize timesteps with extreme incoherence more strongly.
In practice, we apply an entropy- based regularization and softmax-based weighting formulation, which together enhance stability and differentiability:
a₁ = exp(Incoherence((()))
(15)
Σ A== exp(nccoherence(((())))
We construct a synthetical calibration dataset X by combining activations across all timesteps with their respective weights:
X Σ a. (0)
(16)
In this way, the synthetical calibration dataset reflects activation distribution characteristics across multiple timesteps. This enables accurate variance alignment, reduces quantization error, and improves the robustness of the quantized model. We apply the proposed method to the linear layer analyzed in Figure 3 where the incoherence-base
importance scores across timesteps are computed. As shown in Figure 4 (Left), these scores exhibit strong positive correlation with the inter-channel variance each timestep. indicating that the method effectively prioritizes timesteps with higher quantization difficulty. We then perform KLT enhanced Hadamard transforms using calibration datasets constructed from either randomly sampled timestep activations or the incoherence-aware ag- gregation. The resulting channel-wise variance distributions, shown in Figure [ (Right), demonstrate that the incoherence-awee aware approach effectively adapts temporal variance and significantly reduces ihter-channel variance.
4 Experiments
4.1
Experimental Settings
Image Generation. We first evaluate our proposed VETA-DiT framework on the image generation task. Following the original evaluation settings of DiT [33]. we use the pretrained DiT-XL/2 model to generate images with resolutions of 256x256 on the ImageNet dataset [37]. The DDIM-
33
40
is employed during generation, with sampling steps set to 50 and 100. To further demonstrate
the generality of VETA-DiT on diverse generation tasks, we also integrate it into the Pix model
for prompt-based image generation on the COCO dataset (25). A DPM-solver 28 with 20 steps
used, the classifier-fiee free guidance (CFG) scale is set to 4.5. We use FID 15] spatial FID(sFID),

---

371
(Left) Incoherence-aware importance scores acToss timesteps, showing a positive cor- relation with inter-channel variance at each timestep. (Right) Channel-wise variance distributions obtained by applying the K-L enhanced Hadamard transform using calibration datasets constructed from either randomly sampled timestep activations or incoherence-wer weighted aggregated activations. The mncoherence-awan adaptation effectively accommodates temporal variation.
Figure 4:
Inception Score 38 and Precision to evaluate generation fidelity. Additionally, for the COCO task, we include ClipScore [4] to assess text-image alignment, and ImageReward [47 to estimate human preference under complex prompt conditions.
Video Generation. We further evaluate the effectiveness of VETA-DiT in the video generation task by integrating into the STDiT3 model from the Open-Sora [T7. Videos are generated using 50-step DDIM-solver with a CFG scale of 4.0. We conduct comprehensive evaluations on the VBench benchmark to obtain detailed quantitative results. Following 49 we select 8 major evaluation dimensions from VBench [8]
4.2
Quantization Performance
We conduct a comprehensive evaluation of our proposed VETA-DiT and compare it against several dateeoff-hhe-ar post-training quantization (PTQ) approaches specifically designed for DiTs under various experimental settings. In particular, we consider three representative baselines: PTQ4DiT 45). Q-DiT (5) and ViDiT-Q [49]. To ensure fair and consistent comparisons, re-implement and adapt these methods based on their official open-source repositories to support different network architectures and task scenarios. For ViDiT-Q, we intentionally exclude the of its original mixed-precision quantization strategy in our experiments, as it tends to quantize the majority of linear layers INT8, which could obscure the performance differences among these methods. Further experimental details can be found in Appendix Section A
Image Generation Results. Table I presents the quantitative results of DiT-XL/2 the ImageNet 256 x 256 resolution task. Specificallyy, PTQ4DiT and Q-DiT achieve acceptable performance under the W4A8 setting, benefiting from their inter-channel importance balancing and dynamic group- wise quantization strategies. Our proposed VETA-DiT method further narrows the performance gap with the full-precision (FP) model under the same setting, achieving a best FID of 5.22 and an Inception Score 152.52, indicating qugher-quality image generation. When the quantization setting is further reduced W4A4, PTQ4DiT and Q-DiT fail to produce meaningful images due to the limitations of their methodologies. Remarkably, VETA-DiT remains effective in generating visually acceptable images even under this low-precision setting, reducing FID and sFID by 33.65 and 16.68, respectively, compared to the second-best method. This demonstrates the effectiveness of our proposed variance-balame without the use of mixed-pon sexxed-precision quantization, which is incompatible with many existing hardware platforms, suffers from severe performance degradation. We further evaluate VETA-DiT on the COCO dataset using the PixArt-E model for text-to- text-to-image generation. As shown in Figure 5 the results are consistent with those observed on ImageNet: ViDiT-Q fails to generate acceptable outputs under W4A8 due to the lack of mixed- -eecci--reiiioon support. Although Q-DiT performs reasonably well at W4A8, struggles with handling inter-channel imbalances, leading to poor-quality generated images. In contrast, VETA-DiT consistently delivers superior performance across both bit- bit-width settings. Additional randomly generated images are provided in Appendix Figure T0 visual comparison.
strategy and temporal difference-awee aware adaptation. ViDiT-Q.
for
7

---

Table 1: Performance comparison of DiT-X1.22 on the ImageNet 256x256.
Model
Bit-width
Method
IS(1)
FID(U)
SFID(L)
Precision(()
W16A16
FP
90.32
12.25
19.28
0.65
PTQ4DIT
66.85
17.34
20.81
0.58
Q-DiT
77.07
16.16
19.24
0.61
DIT-XL/2 steps=100
W4A8
ViDiT-Q
37.46
49.65
31.87
0.41
Ours
75.90
15.22
18.66
0.62
cfg=1.0
PTQ4DiT
2.08
304.62
129.68
0.07
Q-DiT
1.93
256.13
429.47
0.01
W4A4
ViDiT-Q
25.11
72.23
44.57
0.32
Ours
64.91
22.85
20.50
0.57
W16A16
FP
168.08
4.56
17.72
0.79
PTQ4DIT
138.54
7.33
22.40
0.74
Q-DiT
154.13
5.34
17.62
0.76
DiT-XL/2
W4A8
ViDiT-Q
86.81
20.87
25,40
0.56
steps=100
Ours
152.53
5.29
17.50
0.77
cfg=1.5
PTQ4DiT
2.42
274.59
117.25
0.08
Q-DiT
2.14
243.05
410.74
0.02
W4A4
ViDiT-Q
51.96
40.87
35.58
0.45
Ours
137.98
7.22
18.90
0.74
W16A16
FP
88.07
13.38
19.07
0.65
PTQ4DiT
61.08
23.41
22.29
0.54
Q-DiT
76.12
17.42
18.92
0.61
DIT-XL/2
W4A8
ViDiT-Q
30.79
58.48
39.57
0.36
steps=50
Ours
75.05
17.30
18.36
0.63
cfg=1.0
PTQ4DiT
2.05
298.57
126.64
0.08
Q-DiT
1.81
262.58
421.08
0.01
W4A4
ViDiT-Q
18.13
84.40
57.46
0.28
Ours
64.15
24.24
21.98
0.56
W16A16
FP
164.61
4.86
17.65
0.80
PTQ4DiT
129.06
8.89
22.73
0.71
Q-DiT
148.83
5.77
17.57
0.76
DiT-XL/2
W4A8
ViDiT-Q
81.65
22.65
26.45
0.56
steps=50 cfg=1.5
Ours
147.12
5.53
17.65
0.77
PTQ4DIT
2.33
281.43
119.45
0.08
Q-DiT
2.02
249.98
404.09
0.01
W4A4
ViDiT-Q
48.34
44.15
37.90
0.43
Ours
135.66
7.90
19.11
0.73
"(WxYb)' indicates that the weights and activations quantized x-bit and bit, respectively.
Bit-width
Method
IS(t)
FID(J)
sFID(J)
CLIP(t)
IR(t)
W16A16
FP
38.27
59.74
294.91
0.27
0.89
Q-DiT
38.34
63.36
299.88
0.26
0.93
W4A8
ViDiT-Q
18.81
125.32
325.12
0.22
-0.55
FP16
Q-DIT
Ours
37.62
60.46
297.20
0.26
0.90
Q-DiT 18.27
107.98
312.25
0.24
-0.43
W4A4
ViDiT-Q
9.70
227.80
332.97
0.20
-1.52
Ours
35.56
65.22
305.36
0.26
0.88
VIDIT-Q
Figure 5: (Left) Text-to-image generation performance of the PixArt- model the COCO dataset. (Right) Generated images comparison of W4A4 quantization.

---

Video Generation Results.. As shown in Table 2 similar to the image generation task, existing quantization methods achieve satisfactory performance under the W4A8 setting. However, due to their limited ability to balance inter-channel variance, they struggle under the more challenging W4A4 setting. In contrast, our method consistently outperforms these baselines across various metries, demonstrating its effectiveness in preserving both the visual quality and temporal consistency of generated videos. Appendix Figure [4] [5 presents a visual comparison video for visual evaluation. We also conduct additional experiments to further evaluate the video generation performance of the quantized models; see Appendix D.3 for details.
randomly generated
Table 2: Performance comparison of Open-Sora on the VBench evaluation benchmark.
Subject Consist.
BG. Consist.
Motion
Dynamic Aesthetic Imaging Quality
Scene Consist.
Overall Consist.
Bit-width
Method
Smooth,
Degree
Quality
W16A16
FP 0.947
0.965
0.983
0.680
0.575
0.519
0,454
0.274
Q-DiT
0,951
0.962
0.985
0.600
0,569
0.497
0.451
0.273
W4A8
ViDiT-Q
0.920
0.963
0.982
0.520
0.536
0,495
0.316
0.269
Ours
0.950
0.961
0.985
0.600
0.571
0.509
0.506
0.273
Q-DiT 0.934
0.955
0.983
0,440
0.508
0.452
0.311
0.252
W4A4
ViDiT-Q
0.897
0.962
0.978
0.600
0.499
0.468
0.333
0.254
Ours
0.939
0,959
0,984
0.480
0.546
0,499
0.431
0.269
4.3
Ablation Studies
To assess the effectiveness of each component in our proposed framework, conduct ablation experiments under the challenging W4A4 setting. These experiments are carried out using the DiT-XL/2 model the ImageNet 256 256 dataset, with 50 sampling steps using the DDIM-solver. Detailed quantitative results are provided in Table We begin our evaluation with a baseline that employs simple group-wise linear quantization strategy. Under the W4A4 configuration, the baseline performs poorly across all metrics. Next, we incorporate a K-L enhanced Hadamard transform for variance alignment, which significantly improves the quality of generated images, achieving FID of 48.32, and an IS of 43.68. Building upon this, we further introduce the incoherence-awant temporal adaptation method, which enhances the ability of the Hadamard transform adapt across different timesteps. This results in further reduction of the FID to 7.90. These findings demonstrate the individual effectiveness of each proposed component and highlight how their integration contributes to pushing our VETA-DiT method toward state-of of-the-arr performance under W4A4 settings, enabling the generation of visually plausible and semantically coherent images.
Table 3: Ablation study on ImageNet x56x255 with W4A4.
Method
IS
FID
sFID
Precision
FP
164.61
4.86
17.64
0.80
Baseline
1.84
260.47
409.15
0.01
+ K-L enhanced Hadamard Transform
43.68
48.32
40.19
0.36
Incoherence-avee adaption
135.66
7.90
19.10
0.72
Conclusion
This paper presents VETA-DIT, a post-training quantization framework designed for Diffusion Transformers (DiTs) reduce inference cost while preserving generation quality. We address two major challenges: large inter-channel variance due to outliers and significant activation distribution shifts across denoising timesteps. To this end, we introduce KLT-enhanced Hadamard transform for variance alignment and an incoherence-avee aware adaptive calibration method to handle temporal dynamics. Experiments on image and video generation tasks show that VETA-DiT achieves perfor- mance close to the full-precision model under W4A8, while still delivering acceptable visual quality under the W4A4 setting. Our approach advances the deployment of DiT's in resource-constraing scenarios.

---

6
Acknowledgement
This work was supported in part by the National Key Research and Development Program of China under Grant No. 2023Y222806802 and Grant No. 202IYFB3600100 104, in part by the Joint Funds of the National Nature Science Foundation of China under Grant No. U21 and in part by Suzhou's "Jiebang Guashuai" Project for Key Core Technologies under Grant No. SYG2024134, Lin Yang would like to thank the support from NSFC (No. 62306138). JiangsuNSF (No. BK20230784), and the Inovation Program of State Key Laboratory for Novel Software Technology at Nanjing University (No. ZZKT2024B15, ZZKT2025B25), The authors are grateful for the help from the Interdisciplinary Research Center for Future Intelligent Chips (Chip-X) and Yachen Foundation.
References
[1] Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. In The Twelfth International Conference on Learning Representations, 2024.
[2] Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated Ilms. Advances in Neural Information Processing Systems, 37:100213- 100240, 2024.
[3] Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantiza- tion of large language models with guarantees. Advances in Neural Information Processing Systems, 36:4396 4429, 2023.
[4] Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. In European Conference on Computer Vision, pages 74-91, 2024.
[5] Lei Chen, Yuan Meng, Chen Tang, Xinzhu Ma, Jingyan Jiang, Xin Wang, Zhi Wang, and Wenwu Zhu. Q-dit: Accurate post-training quantization for diffusion transformers. arXiv preprint arXiv:2406. 17344, 2024.
[6] R Dony et al. Karhunen-loeve transform. The transform and data compression handbook, 1(1-34):2,, 2001.
[7] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, G Heigold, S Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2020.
[8] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, al. Scaling rectified flow trans- formers for high-resolution image synthesis. In Forty-first international conference on machine learning. 2024.
[9] Shibo Feng, Chunyan Miao, Zhong Zhang, and Peilin Zhao. Latent diffusion transformer for probabilistic time series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 11979-11987, 2024.
[10] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan-Adrian Alistarh. Optq: Accurate post- training quantization for generative pre-trained transformers. In 11th International Conference on Learning Representations, 2023.
[11] Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer a strong image synthesizer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2114-33173, 2023.
[12] Yefei He, Jing Liu, Weijia Wu, Hong Zhou, and Bohan Zhuang. Efficientdm: Efficient quantization-aware fine-tuning of low-bit diffusion models. In 12th International Conference on Learning Representations, 2024.
10

---

[13] Yefei He, Luping Liu. Jing Liu, Weijia Wu, Hong Zhou, and Bohan Zhuang. Ptqd: Accurate post-training quantization for diffusion models. In Advances in Neural Information Processing Systems 2023, 2023.
[14] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7514-7528, 2021.
[15] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems. 30, 2017.
[16] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proceedings of the 34th International Conference on Neural Information Processing Systems, 2020.
pages
[17] HPC-AL. Open-Sora.
2024.
[18] Ziqi Huang. Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang. Xinyuan Chen, Limin Wang. Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[19] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmeti-onn inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2704-2713, 2018.
[20] Muyang Li, Yujun Lin, Zhekai Zhang, Tianle Cai, Junxian Guo, Xiuyu Li, Enze Xie, Chenlin Meng, Jun- Yan Zhu, and Song Han. Svdquant: Absorbing outliers by low-rank component for 4- bit diffusion models. In The Thirteenth International Conference on Learning Representations, 2025.
[21] Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. Q-diffusion: Quantizing diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17535-17545, 2023.
[22] Yuhang Li, Ruihao Gong. Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Brecq: Pushing the limit of post-training quantization by block reconstruction. In International Conference on Learning Representations, 2021.
[23] Zhikai Li, Junrui Xiao, Lianwei Yang, and Qingyi Gu. Repq-vit: Scale reparamcterization for post-training quantization of vision transformers. In Proceedings the Conference Computer Vision, pages 17227-17236. 2023.
the IEEE/CVF International
[24] Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song, Zhenan Sun, and Ying Wei. Duquant: Distributing outliers via dual transformation makes stronger quantized 1lms. In The Thirth-eighth Annual Conference on Neural Information Processing Systems, 2024.
[25] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part 13, pages 740-755, 2014.
[26) Wenxuan Liu and Sai Qian Zhang. Hq-dit: Efficient diffusion transformer with fp4 hybrid quantization. arXiv preprint arXiv:2405. 19751, 2024.
[27] Yijiang Liu, Huanrui Yang, Zhen Dong, Kurt Keutzer, Li Du, and Shanghang Zhang. Noisyquant: Noisy bias-enhanced post-training activation quantization for vision transformers. In Pro- ceedings of the IEFF/VF Conference on Computer Vision and Pattern Recognition, pages 20321-20330, 2023.
11

---

[28] Cheng Lu, Yuhao Zhou, Fan Bao. Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: fast ode solver for diffusion probabilistic model sampling in around 10 steps. In Proceedings of the 36th International Conference on Neural Information Processing Systems, pages 5775-5787, 2022.
[29] Xin Ma, Yaohui Wang, Xinyuan Chen, Gengyun Jia, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. Transactions on Machine Learning Research, 2025.
[30] Shentong Mo, Enze Xie, Ruihang Chu, HONG Lanqing, Matthias Nießner, and Zhenguo Li. Dit-3d: Exploring plain diffusion transformers for 3d shape generation. In Thirty-seventh Conference Neural Information Processing Systems, 2023.
[31] Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart Van Baalen, and Tijmen Blankevoort. A white paper on neural network quantization. arXiv preprini arXiv:2/06.08295, 2021.
[32] OpenAI. Video generation models as world simulators. luttps://openi..com hww.pppeaaa...omimiemmmeeeeemeee index/ hatpre--gnneratttooommell-l----e www.aaswwwwoddddmlallloooooooooa simmlatorrs/////// 2024.
[33] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195-4205, 2023.
[34] Weiming Ren, Huan Yang. Ge Zhang, Cong Wei, Xinrun Du Wenhao Huang, and Wenhu Chen. Consisti2v: Enhancing visual consistency for image-to-video generation. Transactions on Machine Learning Research, 2024.
[35] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[36] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings. part III 18, pages 234 241, 2015.
[37] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy. Aditya Khosla, Michael Bernstein, Alexander C. Berg. and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJC),, 115(3):211-255) 2015.
[38] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
[39] Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan. Post-training quantization on diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1972-1981. 2023.
[40] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
[41] Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip-#:: Even better Ilm quantization with hadamard incoherence and lattice codebooks. In Forty-first International Conference on Machine Learning, 2024.
[42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[43] Haoxuan Wang, Yuzhang Shang, Zhihang Yuan, Junyi Wu, Junchi Yan, and Yan Yan. Quest: Low-bit diffusion model quantization via efficient selective finetuning. arXiv preprint arXiv:2402.03666, 2024.
12

---

[44] Xiuying Wei, Ruihao Gong, Yuhang Li, Xianglong Liu, and Fengwei Yu. Qdrop: Randomly dropping quantization for extremely low-bit post-training quantization. International Confer- ence on Learning Representations, 2022.
[45] Junyi Wu, Haoxuan Wang. Yuzhang Shang, Mubarak Shah, and Yan Yan. Ptq4dit: Post-training quantization for diffusion transformers. In Advances in neural information processing systems, 2024.
[46] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models.
In International Conference on Machine Learning, pages 38087-38099, 2023.
[47] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: learning and evaluating human preferences for text-to- generation. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 15903-1599 2023.
[48] Zukang Xu, Yuxuan Yue, Xing Hu, Zhihang Yuan, Zixu Jiang, Zhixuan Chen, Jiangyong Yu, Chen Xu, Sifan Zhou, and Dawei Yang. Mambaquant: Quantizing the mamba family with variance aligned rotation methods. arXiv preprint arXiv:2501. 13484, 2025.
[49] Tianchen Zhao, Tongcheng Fang, Haofeng Huang, Enshu Liu, Rui Wan, Widyadewi Soedar- madji, Shiyao Li, Zinan Lin, Guohao Dai, Shengen Yan, et al. Vidit-q: Efficient and accu- rate quantization of diffusion transformers for image and video generation. arXiv preprint arXiv:2406. 025400 2024.
[50] Xingyu Zheng, Xianglong Liu, Haotong Qin, Xudong Ma, Mingyuan Zhang, Haojic Hao, Jiakai Wang. Zixiang Zhao, Jinyang Guo, and Michele Magno. Binarydm: Accurate weight binarization for efficient diffusion models. arXiv preprint Trxiv:2404.05665 2024.
[51] Zheng Zhu, Xiaofeng Wang. Wangbo Zhao, Chen Min, Nianchen Deng, Min Dou, Yuqi Wang, Botian Shi, Kai Wang. Chi Zhang, et al. Is sora world simulator? comprehensive survey on general world models and beyond. arXiv preprint arXiv:2405.03520, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We state the contributions in the abstract and introduction sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the main limitations in Appendix t
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
Please refer to Appendix C for related assumptions and proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Implementation details are provided in Appendix A We have released our code at anonymous repository: http://mooyyyoouu.oooooocccce///
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
We have released our code at
anonymous
repository:
http://womoooooooooooosseee////a Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
ssiicyolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer Section 4.1 and Appendix for experimental settings. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Our study used a fixed seed for all quantization operatipons. following stan- dards in post-training quantization, and thus did report statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
As mentioned in Appendix A all experiments are done on Nvidia A800 GPUs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://ps.//.rrrc..c.///o//////a
Answer: [Yes]
Justification: We confirm the research conducted in thje paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: discuss broader impacts of our work in Appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This work poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all datasets and models utilized in experiments.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This work does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core methods of our research do not involve the of LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy hattps://neurige: /2teearrpp.cc///ooeeeeeee//////2 for what should or should not be described.
20

---

A Additional Experimental Details
In our implementation, we randomly selected 16 classes from the ImageNet dataset perform image generation and calibrate the DiT-XL/2 model. For the PixArt-E model, we used the sample prompts provided in the official PixArt-E codebase for calibration. In video generation tasks, we calibrated the model using text-to-video prompts from the Open-Sora implementation.
For weight quantization, we adopted static quantization with group size of 128. Additionally, applied second-order information from the Hessian matrix, as used in GPTQ, to adjust the weights. This process is performed offline and introduces additional overhead during inference. For activation quantization, we followed the same dynamic quantization strategy as Q-DiT, using group conducted
size of 128 balance quantization accuracy and inference cost. All experiments Nvidia A800 GPUs.
For all baseline methods, used the same calibration set, random noise, and classifier-free guidance, and quantized all Transformer layers in the DiT architecture.
In the ViDiT-Q implementation, we collected the maximum activation values of the layers to be quantized in order to determine the smooth factor. We set the hyperparameter a 0.5 to control the smooth factor. The original ViDiT-Q applies seeeedpreeision quantization to both weights and activations, which we found leads use higher precision the linear layers of the FFN module, To ensure fair comparison, we removed mixed-precision support in our implementation of ViDiT-Q and used the same precision settings as all other methods.
B Detailed Algorithm for VETA-DiT
Algorithm 1: KLT-H Based Quantization with Incoherence-Aware Temporal Sampling
Input: Activation tensor E 1txxxm, weight matrix W gmxd
Output: Quantized activation tensor X., quantized weight matrix W
Step 1: Incoherence-awann sampling of activations
2 Initialize an importance vector a of length t;
for each time step do
Extract the activation matrix (4.) at time 4;; St, of (()); Compute important score Or, based on st;; Store Or, in o;;
Compute the incoherence
8 Step 2: Generate sampling distribution
Apply softmax to obtain probability vector p:
10 Sample multiple time steps based on p: Concatenate the sampled activation matrices form X xampledi
Step Construct KLT-H transformation matrix
13 Compute the covariance matrix of sampled and derive its eigenvectors to form the KLT matrix K;
14 Construct the Hadamard matrix H:
15 Form the composite orthogonal transform: TKLTH K. H:
Step 4: Transform activation and weight using shared transform
17 Transform activation: X' = TKLTH;
18 Transform weight: W' = TKLFH W:
This ensures that the computation (X W) is equivalent (X', W'), preserving functional correctness;
20 Step 5: Quantize the transformed activation and weight Quantize X' using an dynamic activation quantization method, producing Xgi
22 Quantize W using a static weight quantization method, producing W
return Quantized activation X. and quantized weight W
21

---

The VETA-DiT pipeline is detailed in Algorithm The algorithm consists of five key steps. Step 1, we compute the importance scores for each diffusion time step using incoherence-aware sampling. Time steps with higher quantization difficulty are assigned larger weights according to Equation [3 14 allowing the transformation matrix to adapt to the temporally varying activation distributions. Step 2, after obtaining the importance scores, we enhance the numerical stability of the sampling process by applying Equation (13) (6) to generate a composite calibration set, which includes activations sampled in proportion to their importance. Step 3, we calculate the eigenvectors and eigenvalues the covariance matrix of the incoherence-aware calibration set to construct the Karhunen- Loève Transform (KLT) matrix, which is then combined with randomly generated Hadamard matrix, to form the KLT-enhanced Hadamard transformation matrix. Step 4, we apply the transformation matrix T KLT-H to the activation tensor and its inverse (the transpose of the matrix due to orthogonality) to the weight tensor, aligning the inter-channel variance of both activations and weights while ensuring the correctness of the computation. Step 5, apply dynamic quantization to the transformed activations and static quantization to the transformed weights, resulting in low-bit quantized activations and weights for efficient inference.
C Proofs
This section provides detailed derivation demonstrating that the Hadamard transform enhanced by the K-L transform can achieve variance equalization across channels. Recall that the variance of the j-th transformed channel can be expressed Equation 7 Expanding the squared summation, obtain:
m
m
Var(z,)
zugahuyhy
(17)
Interchanging the summation order and grouping terms:
Var(z,) = m
hashus
Σ
(18)
k= 1=1
2=1
We define the sample covariance matrix C E Ruxm as:
(19)
Then the variance of the j-th transformed channel becomes:
m
m
Var(z;)
Calhashu = h2 Chiy
(20)
k=1=1
where hy I is the j-th column of the orthogonal matrix H. To further interpret this expression, recall that:
The diagonal elements of C are Ckk = Vxr(xx). The off-diagonal elements are Cki = Cou(xx: x²) for h: + I.
Thus, can split the summation in Equation 20 into diagonal and off-diagonal parts to derive Equation g which shows even after applying an orthogonal transformation like the Hadamard transform, the resulting channel variances may still differ substantially due to the interaction between original variances and covariances. Hence, Hadamard transform alone does not guarantee uniform variance across channels.
Karhunen-LLoève Transform (KITT), which uses the eigenvectors of the covariance matrix C to form an orthogonal basis. Let C = UAUT, where U contains the orthonormal eigenvectors and A is a diagonal matrix of eigenvalues X₁, >g,... Am- Define K UT as the KLT matrix. The KLT transform is then XK, with covariance:
C
(XK) (XK) K CK TTCK- = A. n
(XK)
(21)
22

---

Thus, the transformed features are decorrelated and their variances are exactly the eigenvalues Ap....,Am
To combine the statistical decorrelation of KLT with the computational efficiency Hadamard, we obtain the transformed tensor Y = XKК XT
This combines the fast computation of the Hadamard transform with the statistical decorrelation property of the KLT. The covariance matrix of the transformed output Y is:
Cy By=Ly H K CKH
AH.
(22)
π
Then, the variance of the j-th channel in Y is:
m
Var(y,) h Ahj Σ Nahiy.
(23)
For normalized Hadamard matrix, ha 1 for all k, so we can obtain the Equation 11
D Additional Empirical Results
D.1 Validation of KLT-H Transformation Effectiveness
We further analyze the effectiveness of the proposed KLT-H transformation in terms of incoherence reduction for both activations and weights. Specifically, we compute the normalized incoherence values across all layers of the DiT-XL/2 Transformer backbone under three settings: without transfor- mation (original), with Hadamard transformation, and with our proposed KLT-H transformation. As shown in Figure and Figure 7 the KLT-H consistently achieves lower incoherence than the other two settings, indicating its superior ability to decorrelate channels and better align with the intrinsic data structure. These improvements hold consistently for both weights (Figure 6 and activations (Figure 7), further validating the generality of our approach across different components of the model.
Original
Hadamard
enhanced Hadamard
Normall Incaherence
0.2
Layer
Figure 6: Normalized incoherence values of weights across all layers in the DiT-XL/2 Transformer backbone under different transformations. The KLT-H transformation consistently achieves lower incoherence, indicating its effecctivneess in improving weight channel alignment.
In addition to the layer-wise results, we also compute the average incoherence and standard deviation across all layers. summarized in Table 4 the KITT-H transformation yields the lowest average incoherence and the smallest standard deviation, further demonstrating its effectiveness and robustness
23

---

Original Hademardd enhanced Hadamerd
Inceherence
40
60
80 100
Layer
Figure 7: Normalized incoherence values of activations across all layers in the DIT-XL/2 Trans- former backbone under different transformations, The KLT-H transformation significantly reduces incoherence compared to both the original and silamarrdtransfommee activations, demonstrating improved inter-channel decorrelation.
in suppressing inter-channel redundancy. These results provide strong empirical evidence supporting the use of KLT-H as principled and practical enhancement for quantization under temporally varying data distributions.
Table 4: Average and standard deviation of incoherence values across all layers of the DiT-X1/2 Transformer under different transformations.
Method
Weight-Incoherence
Activation-Incoherrnne
Original
19.91 + 13.26
49.29 + 44.05
Hadamard
8.56 5.56
19.22 11.80
KLT enhanced Hadamard
7.42 4.40
17.13 + 10.74
D.2 Evaluation Incoherence-Aware Adaptation Strategy
To further validate the effectiveness of proposed incoherence-ane aware adaptation strategy, conduct additional experiments comparing three methods: (1) the standard RTN baseline without any transformation, (2) the KLTT- transform constructed using randomly selected samples, and (3) incoherence-awwre KLT-H transform. For each method, we compute the reconstruction error (mean squared error, MSE) between the output of each linear layer and the corresponding full-precision output.
As shown in Figure 8 under the W4A8 setting, the incoherence-aware KLT-H transform consistently achieves lower reconstruction emrors across the majority layers compared the other two baselines. Similarly. Figure presents the results under the W4A4 setting, where the advantage of our method becomes even more pronounced due to the more aggressive quantization constraints. These results highlight the robustness of the incoherence-aware adaptation handling temporally varying data distributions under low-bit quantization.
D.3 Evaluation of Temporal and Textual Consistency in Video Generation
To comprehensively evaluate the generative capability of the quantized models, we additionally adopt three metrics from different perspectives. Specifically, CLIPSIM and CLIP-Tene used to measure

---

Random-Sampled
Incoberenceeeaart
0.6
Normalized
0.4
0.2
20
40
60
80
100
Layer
Figure 8: Normalized layer-wise MSE reconstruction error comparison under W4A8 quantization.
RTN coneessea
Random-Sampled
Incoherence
0.8
Normel MSE
0.0
20
60
100
Layer
Figure 9: Normalized layer-wise MSE reconstruction enror comparison under W4A4 quantization.
text-video alignment and temporal semantic consistency, respectively, while Temporal Flickering is employed to assess temporal smoothness. As shown in Table our method demonstrates consistently strong performance under both W4A8 and W4A4 quantization settings, outperforming other PTQ methods in terms of both textual alignment and temporal consistency.
E Limitations and Broader Impacts
In this work, we present effective post-training quantization framework that promotes the broader deployment and applicability of Diffusion Transformers (DiTs). By leveraging variance equalization and low-bit data representation, our method significantly reduces the computational and memory overhead, thereby improving the usability of DiT's in resource-constrained settings. VETA-DiT relies on a carefully designed incoherence-awaan sampling strategy that assigns higher importance to timesteps with higher quantization difficulty. However, applicability to other domains or modalitics- audio and 3D data requires further investigation. In future work, plan
25

---

Table 5: Comparison of video generation quality under W4A8 and W4A4 settings using CLIPSIM, CLIP-Temp, and Temporal Flickering.
Bid-Width
Method
CLIPSIM
CLIP-Temp
Temporal Flick
W16A16
0.2217
0.9980
0.9782
Q-DiT 0.2198
0,9973
0.9798
W4A8
ViDiT-Q
0.2200
0.9975
0.9820
Ours
0.2204
0.9982
0.9804
Q-DiT 0.2161
0.9952
0.9702
W4A4
ViDiT-Q 0.2156
0.9953
0.9762
Ours 0.2176
0.9974
0.9770
to explore hardware acceleration of VETA-DiT to enable real-time inference. terms of broader impacts, this work advances efficient generative modeling by improving quantization techniques for diffusion models. Nonetheless, as with many optimization techniques that facilitate model deployment in low-resource scenarios, there is risk of misuse in surveillance, deepfakes, other sensitive content generation without proper regulation. We encourage the community to adopt responsible practices that align with ethical AI development principles.
F
Additional Visualization Results
In this section, we present additional qualitative results to further demonstrate the effectiveness of our proposed TETA-DiT. Figure IO, II shows randomly generated ImageNet images using the DiT-XL/2 model under W4A8 and W4A4 quantization settings. Figure 12 also provides examples of images generated by the Art--Smaa model. addition, frame sample from randomly generated video shown in Figure 14 15 These results qualitatively highlight the strong generation capability of VETA-DiT under both moderate and aggressive quantization.
G Inference overhead
The KLT-enhanced Hadamard matrix is defined TKLT-H KH. where K is a data-driven KLT matrix, precomputed offline using the calibration set, and is a random Hadamard matrix.
During inference, this transformation can be fused into model weights, incurring no additional cost for most layers. For layers like out-proj and down-proj. online transformation necessary to preserve computational invariance. We evaluated the associated overhead on DiT-XL/2 via implementing custom CUDA kernel on NVIDIA A100 GPU, targeting matrix multiplication operations within these layers.
Table 6: Comparison of latency, speedup, and memory reduction.
Width
Method
Latency (ms)
Speedup
Memory (MB) / Reduction
W16A16
FP
6.167 0.045
38.53/ /-
W4A88
ViDiT-Q
3.879 = 0.012
1.59x
18.68 2.06x
W4A4
Ours
2.960 = 0.006
2.08
9.68 3.98x
W4A4+ Online Transform
Ours 3.163 = 0.007
1.95x
10.31 3,74x
As shown in Table 6 due the availability of efficient Hadamard transform implementations, the online transform introduces only 6.8% overhead, while still achieving 1.95x acceleration and 3.74x memory savings over FP16.
26

---

PTQ4DIT(W4A8)
VIDITQ(W4A8)
Q-DIT
thr wit
og
Ours(W4AB)
of
Full-Precision
Figure 10: Random images generated under W4A8 quantization using different PTQ methods with the DiT-XL/2 model on ImageNet 256x
27

---

PTQ4DIT(W4A4)
VIDITQ(WAA4)
Q-DIT(WW44)
Ours (W4A4)
of
Full-Precision
Figure 11: Random images generated under W4A4 quantization using different PTQ methods with the DiT-XL/2 model on ImageNet 256x256.
28

---

VIDITO(mixed-presiiionn
VIDITQ(W4AB)
Q-DIT(W4A8)
Ours(W4A8)
Full-Prrcision
Figure 12: Random images generated under W4A8 quantization using different PTQ methods with the Pixart-E model on COCO. * indicates that ViDiT-Q uses mixed- -recisii quantization, where most linear layers are quantized with higher bit-wi.ths.
29

---

VIDITQ(W4A4)
Q-DIT(WA4A)
Ours (W4A4)
Full-Precision
Figure 13: Random images generated under W4A4 quantization using different PTQ methods with the P model on COCO.
30

---

VIITT (WAAB)
Q-DIT (W4A8)
(W4AB)
Ful-Precision
Figure 14: Random video generated under W4A8 quantization using different PTQ methods with the Open-Sora 1.2 model on VBench.
31

---

VIDITG (W1M)
Q-DIT (W4A4)
Ours (W4A4)
Full ful-Precision
Figure 15: Random video generated under W4A4 quantization using different PTQ methods with the Open-Sora 1.2 model on VBench.
32


Reviews:
[{"review_id": "BvP5NE46Gu", "reviewer": "Reviewer_iU2r", "summary": "This paper introduces VETA-DIT, a 4-bit quantization framework for Diffusion Transformers (DiTs) that tackles activation variance imbalance and temporal dynamics. It uses a KLT-enhanced Hadamard transform to equalize variance and a temporally adaptive strategy to handle shifting statistics across diffusion timesteps. This dual approach allows VETA-DIT to maintain high-quality generation in aggressive W4A4 settings, making powerful DiT models practical for resource-constrained devices.", "questions": "1. How much computational overhead does the KLT-enhanced Hadamard transform introduce during inference?\n2. How sensitive is the final model performance to the specific method used for calculating the timestep importance scores?\n3. Since the KLT is data-dependent, how well does the learned transform generalize to datasets with different statistical properties?\n4. What are the anticipated challenges in applying this framework to even lower bit-rates, such as 3-bit or 2-bit quantization?\n\nI will consider raising the score if the authors can address my concerns.", "limitations": "Yes.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. It introduces an innovative KLT-enhanced Hadamard transform to effectively mitigate the challenging issue of channel-wise variance imbalance in activations.\n2. The proposed temporally adaptive strategy intelligently handles the dynamic statistical shifts of activations across different diffusion timesteps.\n3. VETA-DIT achieves state-of-the-art performance, preserving high generation quality even in the highly aggressive W4A4 quantization setting.\n\nWeaknesses:\n1. Its temporally adaptive strategy relies on the quality of the calibration dataset, which could limit its effectiveness when dealing with extreme or out-of-distribution data.", "was_revised": true, "final_justification": "I appreciate the author's effort in the rebuttal and believe it has addressed my main concerns. I am willing to increase my score.", "rebuttal": "Thanks for your time in dealing with our work. We will answer the question and discuss point by point as follows. We hope that our response satisfactorily addresses the issues you raised. Please feel free to let us know if you have any additional concerns or questions.\n\n**W1 & Q3**: Generalization concern with calibration data.\n\n>KLT is indeed data-dependent. We address this concern by employing an incoherence-aware adaptation strategy to construct the calibration set for guiding the KLT process. Details regarding the calibration set composition and settings are provided in Appendix. A of our paper. Specifically:\n>- For the DiT-XL/2 model, we randomly select 16 classes from ImageNet for calibration, and perform generation across all 1000 classes.\n>- For PixArt-$ \\Sigma$ and Open-Sora models, we adopt prompts provided in their official toolkits for calibration and conduct generation on COCO and VBench datasets, which are not used during calibration.\n>\n>We empirically validate that this strategy ensures the learned KLT transform generalizes across different data distributions in Section 4.2 of our paper. To further verify its generalization, we evaluate the variance-equalization effect both within and outside the calibration set:\n>\n>|Dataset Scope|Var(Before Transform)|Var(After Transform)|\n>|---|---|---|\n>|In-Distribution|0.2328 ± 0.0006|0.0885 ± 0.0001|\n>|Out-of-Distribution|0.3275 ± 0.0035|0.1009 ± 0.0003|\n>\n>The demonstrate show that even on out-of-distribution data, **our method significantly reduces variance, indicating its strong generalization capability**.\n\n**Q1**: Inference overhead from KLT-Hadamard transform.\n\n>The KLT-enhanced Hadamard matrix is defined as  $ \\mathbf{T}_{ \\text{KLT-H}} = \\mathbf{KH}$, where $ \\mathbf{K}$ is a data-driven KLT matrix, precomputed offline using the calibration set, and  $\\mathbf{H}$ is a random Hadamard matrix.\n>\n>During inference, this transformation can be **fused into model weights**, incurring no additional cost for most layers. For layers like out-proj and down-proj, online transformation is necessary to preserve computational invariance [1]. We evaluated the associated overhead on DiT-XL/2 via implementing a custom CUDA kernel on NVIDIA A100 GPU, targeting matrix multiplication operations within these layers.\n>\n>|Bit-Width|Method|Latency (ms)|Speedup|Memory (MB)|Reduction|\n>|---|---|---|---|---|---|\n>|W16A16|FP|6.167 ± 0.045|-|38.53|-|\n>|W4A8|ViDiT-Q|3.879 ± 0.012| 1.59$ \\times$|18.68|2.06$\\times$|\n>|W4A4|Ours|2.960 ± 0.006|**2.08$\\times$**|9.68|**3.98$\\times$**|\n>|W4A4 + Online Transform|Ours|3.163 ± 0.007|**1.95$\\times$**|10.31|**3.74$\\times$**|\n>\n>Due to the availability of efficient Hadamard transform implementations, the online transform introduces only **6.8% overhead**, while still achieving **1.95$\\times$ acceleration** and **3.74$\\times$ memory savings** over FP16. Compared to the W4A8 implementation in ViDiT-Q, which adopts higher activation precision to preserve generation quality, our approach provides **1.23$\\times$ speedup** and **1.81$\\times$ memory reduction**, highlighting its superior efficiency. We will add the results to the revised manuscript to provide a comprehensive view of runtime and memory efficiency.\n>\n>[1] SliceGPT: Compress Large Language Models by Deleting Rows and Columns. ICLR 2024.\n\n**Q2**: Sensitivity to timestep importance estimation.\n\n>Section 4.3 presents our evaluation of the sensitivity to different timestep-wise importance scoring strategies. Table 3 shows that assigning uniform scores across timesteps significantly degrades performance (FID increases from 4.86 to 48.32), leading to severe degradation in image quality. In contrast, applying our incoherence-aware adaptation reduces FID to 7.90, **substantially mitigating performance loss**.\n>\n>Additionally, Appendix D.2 demonstrates that our strategy **consistently outperforms random timestep sampling in reducing quantization error**.\n\n**Q4**: Challenges in extending to lower bit-widths.\n\n>We conducted generation experiments on DiT-XL/2 under the challenging W3A3 setting. We summarize the results in the table below:\n>\n>|Bit-Width|Method|IS $\\uparrow$|FID $\\downarrow$|sFID$\\downarrow$|Precision$\\uparrow$|\n>|---|---|---|---|---|---|\n>|W16A16|FP|164.61|4.86|17.65|0.80|\n>|W3A3|Q-DiT|1.65|266.97|402.47|0.01|\n>|W3A3|Ours|45.83|55.42|34.42|0.32|\n>|W3A3(Mixed Precision)|**Ours**|**118.62**|**13.71**|**22.25**|**0.66**|\n>\n>Although our method significantly outperforms Q-DiT, performance degradation is still observed under such extreme quantization. To mitigate this, we maintain 8-bit precision in sensitive layers, **achieving better results in mixed-precision settings**.\n>\n>Looking forward, as bit-widths decrease further (e.g., to 2-bit), offline KLT becomes insufficient due to severely limited data representation. **Online KLT may be required** to dynamically equalize variance during inference, but this would incur non-trivial computational cost, which we plan to explore in future work."}, {"review_id": "CWTwO60KOc", "reviewer": "Reviewer_CJnV", "summary": "This paper introduces VETA-DiT, a post-training quantization framework tailored for Diffusion Transformers (DiTs) to enable efficient low-bit inference (W4A4 and W4A8) while maintaining high generative quality. 1) To address inter-channel variance caused by outlier activations, the authors propose a Karhunen–Loève Transform (KLT) enhanced Hadamard transformation that equalizes channel variance and improves quantization robustness. 2) To handle the temporal variation in activation distributions during the iterative denoising process, they design an incoherence-aware adaptive calibration method that constructs a synthetic calibration set by prioritizing timesteps with high quantization difficulty. Experiments on image and video generation benchmarks demonstrate that VETA-DiT outperforms prior PTQ methods and remains effective under aggressive 4-bit settings without requiring model retraining.", "questions": "See weaknesses. The main concerns are 1) missing efficiency comparison, 2) effectiveness on distilled smaller models or larger-scale foundation models, and 3) overhead analysis.", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Proposes a novel KLT-enhanced Hadamard transform that effectively equalizes inter-channel variance and improves quantization performance under low-bit settings.\n\n2. The writing is clear and easy to follow.\n\n3. Demonstrates strong empirical performance across multiple datasets (ImageNet, COCO, VBench) and models (DiT-XL/2, PixArt-Σ, STDiT3), especially under challenging W4A4 configurations.\n\nWeaknesses:\n\n1. Efficiency metrics are missing. The authors only report accuracy results, despite claiming improved runtime and efficiency as core motivations. Without latency and memory comparisons—e.g., against FP16 on TorchScript or TensorRT—the work lacks self-contained evidence of practical gains.\n\n2. Diffusion timesteps can be distilled to a small number. It is unclear whether the proposed method remains effective in such scenarios, such as with DMD [1,2]. How much improvement does it bring in terms of both accuracy and efficiency when applied on top of already distilled, smaller models?\n\n3. Although multiple models are evaluated, the core approach is mainly benchmarked on relatively small-scale models except OpenSora (should also clarify how many parameters in models). The scalability of the method to larger models remains unclear. E.g., how does it perform on models such as Wan2.1-T2V-1.3B or Wan2.1-VACE-14B?\n\n4. The method requires computing eigenvectors of covariance matrices, which may introduce non-negligible overhead during calibration for large models. It would be helpful to quantify this preprocessing cost.\n\n[1] Improved Distribution Matching Distillation for Fast Image Synthesis\n[2] One-step Diffusion with Distribution Matching Distillation", "was_revised": true, "final_justification": "The authors have addressed most of my concerns. I'll increase the score.", "rebuttal": "We sincerely thank the reviewer for providing valuable feedback. We detail our response below point by point. Please kindly let us know whether you have any further concerns.\n\n**W1 & Q1**: Efficiency metrics.\n\n> We have extended the manuscript to include latency and memory comparisons. Our implementation of VETA-DiT is built upon CUDA 12.4 with CUTLASS to support INT4 matrix multiplication on Tensor Cores. We evaluated the performance of the linear layers on an NVIDIA A100 GPU, which are the main bottlenecks in DiT [1, 2].\n>\n>Most KLT matrices are fused into model weights to eliminate runtime overhead. For layers such as out-proj and down-proj, we apply online transformations using fast Hadamard kernels. Even with online quantization and KLT, our method still achieves **1.95$ \\times$ speedup** and **3.74$\\times$ memory savings** compared to FP16. Compared to the W4A8 implementation in ViDiT-Q, which adopts higher activation precision to preserve generation quality, our approach provides **1.23$\\times$ speedup** and **1.81$\\times$ memory reduction**, highlighting its superior efficiency:\n>\n>|Bit-Width|Method|Latency (ms)|Speedup|Memory (MB)|Reduction|\n>|---|---|---|---|---|---|\n>|W16A16|FP|6.167 ± 0.045|-|38.53|-|\n>|W4A8|ViDiT-Q|3.879 ± 0.012| 1.59$\\times$|18.68|2.06$\\times$|\n>|W4A4|Ours|2.960 ± 0.006|**2.08$\\times$**|9.68|**3.98$\\times$**|\n>|W4A4 + Online Transform|Ours|3.163 ± 0.007|**1.95$\\times$**|10.31|**3.74$\\times$**|\n>\n>We will add the results to the revised manuscript to provide a comprehensive view of runtime and memory efficiency.\n>\n>[1] Solving Oscillation Problem in Post-Training Quantization Through a Theoretical Perspective. CVPR 2023.\n>\n>[2] PTQ4DiT: Post-training Quantization for Diffusion Transformers. NeurIPS 2024.\n\n**W2 & Q2**: Effectiveness with distilled timesteps.\n>While methods such as DMD reduce the number of denoising steps, **temporal variation in activation statistics still exists**, which is central to our approach. We analyze activations from randomly selected Transformer blocks and observe notable differences in mean, max, and incoherence across timesteps:\n>\n>| Timestep | Mean      | Max   | Incoherence |\n>| -------- | --------- | ----- | ----------- |\n>| t1       | -0.000235 | 4.200 | 5.680       |\n>| t2       | -0.000419 | 4.375 | 5.871       |\n>| t3       | -0.000179 | 4.956 | 4.316       |\n>| t4       | -0.000573 | 5.363 | 7.242       |\n>\n>Our incoherence-aware adaptation is designed to capture and address such temporal variation, regardless of the total number of steps. We quantized the Transformer blocks in the DMD2-SDXL-4step model and evaluated on COCO. Our method **achieves better results than Q-DiT** under lower bit-widths:\n>\n>| Bit-Width | Method | IS $ \\uparrow$    | FID $\\downarrow$   | sFID $\\downarrow$   | CLIP $\\uparrow$ | IR $\\uparrow$   |\n>| --------- | ------ | ----- | ----- | ------ | ---- | ---- |\n>| W16A16    | FP     | 42.39 | 54.64 | 263.93 | 0.27 | 0.89 |\n>| W4A8      | Q-DiT  | **41.67** | 56.80 | 274.34 | **0.27** | **0.90** |\n>| W4A8      | Ours   | 41.40 | **54.89** | **264.86** | 0.26 | 0.88 |\n>| W4A4      | Q-DiT  | 32.93 | 74.19 | 293.94 | 0.23 | 0.43 |\n>| W4A4      | Ours   | **40.14** | **58.39** | **277.15** | **0.25** | **0.87** |\n>\n>Our approach can be effectively combined with timestep distillation techniques like DMD without additional runtime cost, thus improving efficiency.\n\n**W3 & Q2**: Scalability to larger models.\n\n>The Open-Sora v1.2 model contains **1.1B parameters**, which is comparable in scale to Wan2.1-T2V-1.3B. For even larger models like Wan2.1-VACE-14B, which adopt deeper Transformer stacks, the architecture remains consistent across layers.\n>\n>It has been observed that **larger models are typically more robust to PTQ degradation** [3]. We conducted a block-wise MSE analysis on Wan2.1-14B, and our method shows **significantly lower quantization error compared to RTN and Q-DiT**:\n>\n>| Bit-Width | MSE(RTN)     | MSE(Q-DiT)   | MSE(Ours)    |\n>| --------- | ------- | ------- | ------- |\n>| W4A8      | 0.02546 | 0.00766 | **0.00374** |\n>| W4A4      | 0.07424 | 0.01530 | **0.00714** |\n>\n>[3] Scaling Laws for Precision. ICLR 2025.\n\n**W4 & Q3**: Calibration cost of KLT.\n\n>The KLT calibration is a **one-time offline process** and **does not incurs inference-time overhead**. We report the calibration time on NVIDIA A100 GPU:\n>- DiT-XL/2: **~35 minutes**\n>- Open-Sora v1.2 (1.1B): **~2.5 hours**"}, {"review_id": "3DrzXJocwC", "reviewer": "Reviewer_eHfZ", "summary": "This paper proposes VETA-DiT, a post-training quantization framework for Diffusion Transformers (DiTs) under W4A4 setting. The method addresses two major challenges, large inter-channel variance due to outliers and timestep-dependent activation shifts during iterative denoising. To mitigate these, the authors introduce a KLT enhanced Hadamard transform for better variance alignment and an incoherence-aware temporal adaptation strategy. VETA-DiT outperforms existing PTQ baselines across image and video generation tasks, achieving high visual quality under low-bit quantization settings without retraining.", "questions": "1. The combination of KLT with the Hadamard transform appears similar to the method proposed in MambaQuant (ICLR 2025). Can the authors clearly articulate how their usage is different in either formulation, application, or motivation specific to DiTs?\n\n2. While the paper addresses temporal variance in image generation (across denoising steps), it is unclear whether similar temporal adaptation is used or needed in video synthesis. Do video generation tasks show similar incoherence trends, and is the adaptation equally effective?\n\n3. How does your incoherence-aware adaptation with synthetic data generation compare to previous methods built upon KLT-Hadamard transforms, in terms of effectiveness and novelty?\n\n4. What is the actual runtime improvement or slowdown introduced by your method, including the overhead of the KLT-H computation and calibration?\n\n5. You mention disabling mixed-precision in ViDiT-Q for fair comparison. Could you elaborate on the implications of this choice, and whether similar mixed-precision designs could benefit VETA-DiT as well?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The paper is well-structured, with clear visualizations and an accessible algorithm description, aiding reproducibility and understanding.\n2. This paper integrates the Karhunen–Loève Transform (KLT) with the Hadamard transform in the context of DiT for the first time, and demonstrates outstanding generative quality in low-precision settings.\n3. The incoherence-aware temporal adaptation and synthetic data generation strategy are straightforward and well-validated through diverse empirical experiments.\n4. The proposed scheme is applied to large-scale networks for both image and video generation, demonstrating its generality and robustness.\n\nWeaknesses\n1. The combination of KLT with the Hadamard transform has already been presented in MambaQuant (ICLR 2025), which predates this NeurIPS submission. Thus, the novelty of this paper is somewhat reduced, although its contribution to the DiT domain remains valid and meaningful.\n2. As a result, the main contribution of this work lies in the design of the incoherence-aware adaptation. While the idea is interesting, comparisons with alternative temporal calibration or sampling strategies are insufficient.\n3. Since the method combines KLT with the Hadamard transform, the computational overhead is likely higher than using Hadamard alone. The paper should provide practical validation of inference speedups on real hardware.\n4. While the paper analyzes the time-varying activation distributions for image generation, it gives limited attention to similar temporal characteristics in video synthesis tasks.", "was_revised": true, "final_justification": "During the rebuttal process, the authors provided additional supporting evidence. I think the novelty is somewhat limited, but the paper is above the decision boundary.", "rebuttal": "We greatly appreciate the reviewer's constructive comments on our paper. We will respond to the reviewer's feedback with detailed explanations for each point.\n\n**W1 & Q1**: Difference from MambaQuant in the use of KLT.\n\n>We also noticed that MambaQuant applies KLT to state space models, as described in line 147 of our paper. MambaQuant focuses on Mamba-based state space models and addresses the numerical instability caused by outlier amplification in PScan operations. In contrast, our work is dedicated to Transformer-based diffusion models (DiTs), which exhibit a **naturally large variance across activation dimensions** and **significant temporal variation** due to iterative denoising.\n>\n>We introduce KLT to equalize the intrinsic variance distribution across channels and time steps in DiTs. However, applying a single, time-step-independent KLT matrix across all denoising steps, as MambaQuant, leads to a **severe degradation in generation quality** due to the significant temporal variation in DiT activations. On the other hand, using fully time-step-specific KLT matrices incurs excessive load/offload overhead during inference. To address this trade-off, we propose an incoherence-aware adaptation strategy that enables **efficient and temporally-sensitive KLT usage** in DiTs, effectively mitigating both representational and computational inefficiencies.\n>\n>We will include a more explicit discussion of the differences between our approach and MambaQuant in the revised manuscript.\n\n**W2**: Effectiveness compared to alternative temporal sampling strategies.\n>Prior methods, such as ViDiT-Q [1], adopt a shared balancing mask $s$ across all denoising steps, which fails to account for the dynamically changing data distributions inherent in DiTs, effectively corresponding to uniform timestep sampling without adaptation. Currently, there is a lack of viable strategies specifically designed to handle this temporal variation. \n>\n>Therefore, we provide a comprehensive evaluation of our incoherence-aware adaptation in Table 3 and Appendix D.2 of our paper, assessing both its impact on **final generation quality and its effectiveness in reducing quantization error during inference**. The results demonstrate that **our method significantly outperforms uniform and random sampling strategies**.\n>\n>[1] ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generation. ICLR 2025.\n\n**W3 & Q4**: Runtime and calibration overhead.\n\n>The calibration process is conducted **once offline and does not affect inference latency**. On NVIDIA A100 GPUs, calibrating DiT-XL/2 takes approximately **35 minutes**, and calibrating Open-Sora v1.2 (1.1B parameters) takes around **2.5 hours**.\n>\n>During inference, we use CUTLASS INT4 kernels optimized for Tensor Cores (CUDA 12.4) and **fuse most KLT matrices into the model weights**. Only a few blocks (e.g., out-proj, down-proj) require lightweight online transforms, implemented via fast Hadamard kernels commonly available in existing libraries. We still achieve **1.95$ \\times$ speedup** and **3.74$\\times$ memory savings** over FP16. Compared to the W4A8 implementation in ViDiT-Q, which adopts higher activation precision to preserve generation quality, our approach provides **1.23$\\times$ speedup** and **1.81$\\times$ memory reduction**, demonstrating the hardware efficiency of our method:\n>\n>|Bit-Width|Method|Latency (ms)|Speedup|Memory (MB)|Reduction|\n>|---|---|---|---|---|---|\n>|W16A16|FP|6.167 ± 0.045|-|38.53|-|\n>|W4A8|ViDiT-Q|3.879 ± 0.012| 1.59$ \\times$|18.68|2.06$\\times$|\n>|W4A4|Ours|2.960 ± 0.006|**2.08$\\times$**|9.68|**3.98$\\times$**|\n>|W4A4 + Online Transform|Ours|3.163 ± 0.007|**1.95$\\times$**|10.31|**3.74$\\times$**|\n>\n>These results confirm that our method maintains efficiency while incorporating KLT-H adaptations. We will add the results to the revised manuscript to provide a comprehensive view of runtime and memory efficiency.\n\n**W4 & Q2**: Temporal incoherence in video generation.\n\n>Temporal incoherence also arises in video synthesis. We analyzed activation statistics at different denoising steps in Open-Sora and observed **similar patterns of variation** as in image generation. The table below presents varying incoherence values across timesteps, confirming the presence of temporal drift in diffusion-based video generation:\n>\n>| Timestep | Mean     | Max   | Incoherence |\n>| -------- | -------- | ----- | ----------- |\n>| t1       | -0.00414 | 3.201 | 10.821      |\n>| t5       | -0.00140 | 3.262 | 11.109      |\n>| t10      | 0.00042  | 3.283 | 13.344      |\n>| t15      | 0.00073  | 3.408 | 11.008      |\n>| t20      | 0.00108  | 3.773 | 12.742      |\n>| t25      | 0.00117  | 4.191 | 14.078      |\n>| t30      | 0.00267  | 5.391 | 15.906      |\n>\n>As shown in Table 2 of our paper, our incoherence-aware adaptation continues to deliver **substantial improvements in video generation quality**, confirming its effectiveness in modeling temporal variations in both image and video diffusion models.\n\n**Q3**: Effectiveness and novelty of incoherence-aware adaptation.\n\n>MambaQuant applies a shared KLT transform across all time steps. However, these assumptions break down in DiTs, where iterative denoising introduces **highly dynamic and non-stationary activation patterns**.\n>\n>Our incoherence-aware adaptation departs from this assumption by designing a temporally-sensitive calibration strategy specifically tailored for DiTs. Rather than relying on handcrafted or static transforms, we leverage synthetic data generation to adaptively construct calibration sets that capture evolving activation statistics across time steps—**a capability absent in prior KLT-Hadamard quantization methods**.\n>\n>Moreover, our approach precomputes transform matrices offline and fuses them into model weights wherever possible, minimizing runtime overhead without sacrificing temporal sensitivity.\n>\n>This combination of **temporal modeling**, **data-driven calibration**, and **runtime efficiency** distinguishes our method both in novelty and practical utility.\n\n**Q5**: Mixed-precision considerations in comparison with ViDiT-Q.\n\n>In the official implementation of ViDiT-Q, certain Linear layers are retained in FP16, which substantially **reduces quantization-induced degradation but also limits efficiency gains**, especially since linear layers are bottlenecks in DiTs [1, 2]. For a fair comparison, we disabled mixed-precision in ViDiT-Q.\n>\n>To test whether our method can benefit similarly, we evaluated W3A3 quantization on DiT-XL/2 with and without mixed precision. Although W3A3 is a highly aggressive setting, **our method benefits significantly from a mixed-precision strategy**:\n>\n>| Bit-Width              | IS $\\uparrow$    | FID $\\downarrow$  | sFID  $\\downarrow$ | Precision $\\uparrow$ |\n>| ---------------------- | ------ | ----- | ----- | --------- |\n>| W16A16                 | 164.61 | 4.86  | 17.65 | 0.80      |\n>| W3A3                   | 45.83  | 55.42 | 34.42 | 0.32      |\n>| W3A3(Mixed Precision) | **118.62** | **13.71** | **22.25** | **0.66**      |\n>\n>This indicates that VETA-DiT is also compatible with hybrid-precision schemes. However, as in ViDiT-Q, these approaches reduce theoretical efficiency gains and often require specialized CUDA kernels or hardware to realize end-to-end improvements.\n>\n>[1] Solving Oscillation Problem in Post-Training Quantization Through a Theoretical Perspective. CVPR 2023.\n>\n>[2] PTQ4DiT: Post-training Quantization for Diffusion Transformers. NeurIPS 2024."}, {"review_id": "pL6rb8FXRg", "reviewer": "Reviewer_5kuo", "summary": "This paper identifies two key challenges in quantizing DiT models: (1) large inter-channel variance caused by outliers, and (2) significant shifts in activation distributions across denoising timesteps. To address these issues, the authors propose a KLT-enhanced Hadamard transform for variance alignment and an incoherence-aware adaptive calibration method to manage temporal dynamics. Experimental results demonstrate strong performance compared to existing baselines.", "questions": "Q1: Can you provide more details on the practical implementation aspects of your method, especially regarding hardware compatibility or deployment on resource-constrained devices?\n\nQ2: Given the clear performance differences observed when quantizing from W4A8 to W4A4, do you anticipate similar trends at lower bit-widths? What challenges do you foresee when pushing the quantization limits further?", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n* **S1**: The paper is clearly written and easy to follow. The presentation is organized and the explanations are generally accessible, which helps the reader understand the main ideas without much difficulty.\n* **S2**: The proposed method is simple and straightforward, making it easy to implement. This practical aspect increases its usability and potential for adoption in real-world applications.\n* **S3**: The experimental results are strong and well-supported. The model is evaluated across multiple datasets, tasks, and metrics, showing its effectiveness and generality. Notably, the clearer performance gap observed when further quantizing from W4A8 to W4A4 provides additional evidence of the proposed method’s effectiveness.\n\n## Weaknesses:\n* **W1**: For a quantization method like this, I would expect an analysis of hardware resource usage. Unfortunately, the paper does not include such an evaluation.\n* **W2**: The paper could be further strengthened by evaluating the proposed method at even lower bit settings, such as W2A8 or W4A2.", "was_revised": true, "final_justification": "The authors have addressed my main concerns. I think it is reasonable to keep the current score as accept.", "rebuttal": "Thank you sincerely for your thoughtful and positive feedback on our work. Below, we have provided a detailed explanation for your remaining concern as follows. Please do not hesitate to let us know if you have any further questions.\n\n**W1 & Q1**: Hardware resource usage evaluation.\n\n>We thank the reviewer for pointing this out. We have updated the manuscript to include an evaluation of hardware resource usage. In DiTs, linear layers dominate both computation and memory consumption [1, 2]. To address this, we implement our INT4 matrix multiplications using CUTLASS on Tensor Cores (CUDA 12.4) and evaluate our custom kernels on NVIDIA A100 GPU.\n>\n> We fuse most KLT matrices into model weights during offline calibration, thereby eliminating the need for runtime transformation. For layers that require runtime KLT to preserve mathematical equivalence [3], we use fast online Hadamard kernels, which are widely available and efficient.\n>\n>Despite incorporating both online quantization and runtime KLT transforms, we still observe **1.95$ \\times$ speedup** and **3.74$\\times$ memory savings** over FP16. Compared to the W4A8 implementation in ViDiT-Q, which adopts higher activation precision to preserve generation quality, our approach provides **1.23$\\times$ speedup** and **1.81$\\times$ memory reduction**, demonstrating the hardware efficiency of our method:\n>\n>|Bit-Width|Method|Latency (ms)|Speedup|Memory (MB)|Reduction|\n>|---|---|---|---|---|---|\n>|W16A16|FP|6.167 ± 0.045|-|38.53|-|\n>|W4A8|ViDiT-Q|3.879 ± 0.012| 1.59$\\times$|18.68|2.06$\\times$|\n>|W4A4|Ours|2.960 ± 0.006|**2.08$\\times$**|9.68|**3.98$\\times$**|\n>|W4A4 + Online Transform|Ours|3.163 ± 0.007|**1.95$\\times$**|10.31|**3.74$\\times$**|\n>\n>We believe this provides a comprehensive view of both computational and memory efficiency. We will add the results to the revised manuscript to provide a comprehensive view of runtime and memory efficiency.\n>\n>[1] Solving Oscillation Problem in Post-Training Quantization Through a Theoretical Perspective. CVPR 2023.\n>\n>[2] PTQ4DiT: Post-training Quantization for Diffusion Transformers. NeurIPS 2024.\n>\n>[3] SliceGPT: Compress Large Language Models by Deleting Rows and Columns. ICLR 2024.\n\n**W2 & Q2**: Evaluation under lower bit settings.\n\n>We conducted additional experiments on DiT-XL/2 under the W3A3 quantization setting, which is already highly challenging. As shown below, our method significantly outperforms Q-DiT, but performance degradation is still notable under such low bit-widths. Introducing mixed-precision, by keeping some sensitive layers in 8-bit, yields substantial quality improvements.\n>\n>| Bit-Width             | Method | IS $ \\uparrow$    | FID $\\downarrow$    | sFID $\\downarrow$   | Precision  $\\uparrow$|\n>| --------------------- | ------ | ------ | ------ | ------ | --------- |\n>| W16A16                | FP     | 164.61 | 4.86   | 17.65  | 0.80      |\n>| W3A3                  | Q-DiT  | 1.65   | 266.97 | 402.47 | 0.01      |\n>| W3A3                  | Ours   | 45.83  | 55.42  | 34.42  | 0.32      |\n>| W3A3(Mixed Precision)| **Ours**   | **111.62** | **13.71**  | **22.25**  | **0.66**      |\n>\n>At even lower bit-widths, quantizing becomes extremely lossy due to limited representational range. In such cases, offline KLT-based variance equalization becomes insufficient, and **online KLT computation may be required to maintain generative fidelity**. However, this would introduce non-negligible runtime cost. We will explore this trade-off in future work."}]

Author rebuttal:

Meta-review: This paper introduces VETA-DiT, a novel PTQ framework to enable efficient 4-bit inference for DiTs. The paper's primary strengths, as noted by all reviewers, are its clear motivation, technical novelty, and strong empirical results on both image and video generation tasks under aggressive W4A4 settings. Initial weaknesses raised by reviewers (CJnV, eHfZ, 5kuo) were significant, focusing on the critical omission of hardware efficiency metrics, questions about novelty relative to prior work (eHfZ), and scalability (iU2r, CJnV). However, the authors provided an exemplary and thorough rebuttal, adding comprehensive end-to-end latency and memory benchmarks that demonstrated substantial speedups over FP16 and prior art. They also successfully clarified the novelty of their temporally adaptive approach and provided evidence of scalability. This discussion was highly effective, convincing reviewers CJnV, iU2r, and eHfZ to raise their scores. I recommend acceptance.
Decision: Accept (poster)