PTQ4DiT: Post-training Quantization for Diffusion Transformers
Authors: Junyi Wu
Haoxuan Wang
Yuzhang Shang
Mubarak Shah
Yan Yan
Venue: NeurIPS 2024 poster (2024)
Keywords: Model Quantization; Diffusion Transformer; Efficiency
Primary area: generative_models
TL;DR: We propose a novel Post-training Quantization method for Diffusion Transformers to enhance the inference efficiency.

Abstract: The recent introduction of Diffusion Transformers (DiTs) has demonstrated exceptional capabilities in image generation by using a different backbone architecture, departing from traditional U-Nets and embracing the scalable nature of transformers. Despite their advanced capabilities, the wide deployment of DiTs, particularly for real-time applications, is currently hampered by considerable computational demands at the inference stage. Post-training Quantization (PTQ) has emerged as a fast and data-efficient solution that can significantly reduce computation and memory footprint by using low-bit weights and activations. However, its applicability to DiTs has not yet been explored and faces non-trivial difficulties due to the unique design of DiTs. In this paper, we propose PTQ4DiT, a specifically designed PTQ method for DiTs. We discover two primary quantization challenges inherent in DiTs, notably the presence of salient channels with extreme magnitudes and the temporal variability in distributions of salient activation over multiple timesteps. To tackle these challenges, we propose Channel-wise Salience Balancing (CSB) and Spearmen's $\rho$-guided Salience Calibration (SSC). CSB leverages the complementarity property of channel magnitudes to redistribute the extremes, alleviating quantization errors for both activations and weights. SSC extends this approach by dynamically adjusting the balanced salience to capture the temporal variations in activation. Additionally, to eliminate extra computational costs caused by PTQ4DiT during inference, we design an offline re-parameterization strategy for DiTs. Experiments demonstrate that our PTQ4DiT successfully quantizes DiTs to 8-bit precision (W8A8) while preserving comparable generation ability and further enables effective quantization to 4-bit weight precision (W4A8) for the first time.

PTQ4DiT: Post-training Quantization for Diffusion Transformers
Junyi 3 Haoxuan Wang Yuzhang Shang? Mubarak Shah? Yan Yan
University of Illinois Chicago Iillioois Institute of Technology University of Central Florida
Abstract
The recent introduction of Diffusion Transformers (DiTs) has demonstrated excep- tional capabilities in image generation by using a different backbone architecture, departing from traditional U-Nets and embracing the scalable nature of transform- ers. Despite their advanced capabilities, the wide deployment of DiTs, particularly for real-time applications, is currently hampered by considerable computational demands at the inference stage. Post-training Quantization (PTQ) has emerged as a fast and data-efficient solution that can significantly reduce computation and mem- ory footprint by using low-bit weights and activations. However, its applicability to DiTs has not yet been explored and faces non-trivial difficulties due to the unique design of DiTs. In this paper, we propose PTQ4DiT, a specifically designed PTQ method for DiTs. We discover two primary quantization challenges inherent in DiT's, notably the presence of salient channels with extreme magnitudes and the temporal variability in distributions of salient activation over multiple timesteps. To tackle these challenges, we propose Channel-wise Salience Balancing (CSB) and Spearman's 's p-guided Salience Calibration (SSC). CSB leverages the comple- mentarity property of channel magnitudes to redistribute the extremes, alleviating quantization errors for both activations and weights. SSC extends this approach by dynamically adjusting the balanced salience to capture the temporal varia- tions in activation. Additionally, to eliminate extra computational costs caused by PTQ4DiT during inference, we design an offline re-paaameterization strategy for DiTs. Experiments demonstrate that our PTQ4DiT successfully quantizes DiT's 8-bit precision (W8A8) while preserving comparable generation ability and further enables effective quantization to 4-bit weight precision (W4A8) for the first time.
Introduction
Diffusion models have spearheaded recent breakthroughs in generation tasks [59, 7J. In the past, these models were based convolutional U-Nets [40] as their backbone architectures [46, 17, 9, 39]. How- ever, recent work [2, 60, 30] has revealed that the U-Net inductive bias not essential for the success of diffusion models and even limits their scalability. Among this trend, Diffusion Transformers (DiT's) [37] have demonstrated exceptional capabilities in image generation by using a different backbone architecture. Different from U-Nets that carefully design downsampling and upsampling blocks with skip-connections, DiT's are constructed by repeatedly and sequentially stacking transformer blocks [49]. This architectural choice inherits the scaling property of transformers [5, 48, 58, 31], facilitating more flexible parameter expansion for enhanced performance. With their versatility and scalability, DiTs have been successfully integrated into advanced frameworks like Sora [4], demonstrating their
potential as a leading architecture for future generative models [14, 6, 30, 65].
'Equal Contribution. 'Coreesponding Author. Work done during Junyi Wu's visit CRCV, UCF.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

WBA8
W4AB
Salient Activation Channels
Actvation Magnitude
14
o O-Diffusion
Q.Diffusion
PTQD
25
PTQD
Rapa
Repq"
20-
Ours
Channel
10
Weight Asssmaae
Complementarity
6-
Chamma
5
Salient Weight Channels
160 180 200 220 240 260
100
150
200
250
IS
I5
Figure 1: (Left) Illustration of salient channels in activation and weight. Note that salient acti- vation channels exhibit variations over different timesteps (e.g., t2, fz.), posing non-trivial quantization challenges. To mitigate the overall quantization difficulty, our method leverages the complementarity (activation and weight channels do not have extreme magnitude simultaneously) to redistribute channel salience between weights and activations across various timesteps. (Right) Quantization performance on W8A8 and W4A8, employing FID (lower is better) and IS (higher is better) metrics on ImageNet 2 [41]. The circle size indicates the model size.
Nonetheless, the widespread adoption of Diffusion Transformers currently constrained by their massive amount of parameters and computational complexity. DiTs consist of large number of repeated transformer blocks and employ a lengthy iterative image sampling process, demanding high computational costs during inference. For instance, generating 512x512 resolution image using DìTs can take more than 20 seconds and 10 Gflops on an NVIDIA RTX A6000 GPU. This substantial requirement makes them unacceptable impractical for real-time applications, especially considering the potential for increased model sizes and feature resolutions.
Model quantization [33, 32, 28] is a prominent technique for accelerating deep learning models because its high compression rate and significant reduction in inference time. This technique trans- forms model weights and activations into low-bit formats, which directly reduces the computational burden and memory usage. Among various methods, Post-training Quantization (PTQ) stands out as leading approach since it circumvents the need to re-train the original model [62, 44, 18, 63, 22]. Practically, PTQ requires only a small dataset for fast calibration, thus is highly suitable for quantizing DiTs, whose re- re-training process involves extensive data and computational resources [14, 6].
However, quantizing DiTs in a post-training manner is non-trivial due to the complex distribution patterns in weights and activations. We discover two major challenges that impede the effective quantization of DiTs: The emergence of salient channels, channels with extreme magnitudes, in both weights and activations of linear layers within DiT blocks. When low-bit representations are used for these salient channels, pronounced errors compared to the full-precision (FP) counterparts observed, incurring fundamental difficulty for quantization. 2 The extreme magnitudes within salient activation channels significantly vary as the inference proceeds across multiple timesteps. This dynamic behavior further complicates the quantization of salient channels, quantization strategies optimized for one timestep may fail to generalize to other timesteps. Such inconsistency, especially in salient channels that dominate the activation signals, result in significant deviations from the full-precision distribution, leading to degradation in the generation ability of quantized models.
Targeting these two challenges, we propose novel Post-training Quantization method specifically for Diffusion Transformers, termed PTQ4DIT. To address the quantization difficulty associated with salient channels, we propose Channel-wis. Salience Balancing (CSB). CSB capitalizes on an interesting observation of the salient channels that extreme values do not coincide in the same channel of activation and weight within the same layer, shown in Figure I (Left). Leveraging this complementarity property, CSB facilitates the redistribution of extreme magnitudes between activations and weights minimize the overall channel salience. Concretely, we introduce Salience Balancing Matrices, derived from the statistical properties of activation and weight distributions, to channel-wise transform both activations and weights. This transformation achieves equilibrium in their salient channels, effectively mitigating the quantization difficulty of the balanced distributions.
Recognizing the variability in activations over different timesteps, we further extend the concept of channel salience along the temporal dimension and propose Spearman's 's p-guided Salience Calibration (SSC). This method refines the Salience Balancing Matrices to comprchensively evaluate activation

---

DiT Block
X
Maythale
X
til & WW
Linear Layer f
Activation from Vanous Timestops
CSB
Channel
Channel
Pointwise
Integer Matrix Multipicaaiin
Feedlonward
high correlation with W
high quant. error, less focus SSC
adaLN
Quantizer
Quantizer
Uww Agrina
X
W
correlation with
quant. error, more focus
Mulli Head
Sell-Atention
11.01
Magobaste
X
Mapeoatt
&
adaLN
MLPPs
X
W
Impul
Conditioing
Figure 2: (Left) Overview of the Diffusion Transformer (DiT) Block [37]. (Middle) Illustration of the linear layer in Multi-Head Self-Attention (MHSA) and Pointwise Feedforward (PF) modules. which incorporates our proposed Channel-wise Salience Balancing (CSB) and Spearman's p-guided Salience Calibration (SSC) to address quantization difficulties for both activation X and weight W. Appendix A depicts detailed structures of the MHSA and PF modules with adjusted linear layers. (Right) Illustration of CSB and SSC in PTQ4DiT. CSB redistributes salient channels between weights and activations from various timesteps reduce overall quantization errors. SSC calibrates the activation salience across multiple timesteps via selective aggregation, with more focus on timesteps where quantization errors be significantly reduced by CSB.
salience over timesteps, with more emphasis timesteps where the complementarity between salient activation and weight channels is more significant. Furthermore, we design a re-parameterization scheme that offline absorb these Salience Balancing Matrices into adjacent layers, thus avoiding additional computation overhead at the inference stage.
While the performance of mainstream PTQ methods degrades on DiTs, our PTQ4DiT achieves comparable performance to the FP counterpart with 8-bit weight and activation (W8A8). addition, PTQ4DiT can generate high-quality images with further reduced weight precision at 4-bit (W4A8). To the best our knowledge, PTQ4DiT is the first method for effective DiT quantization.
2 Backgrounds and Related Works
2.1 Diffusion Transformers
Although generative models built upon U-Nets have made great advancements the last few years, transformer-like architectures are increasingly attracting attention [39, 7, 59]. The recently explored Diffusion Transformers (DiTs) [37] have achieved state-of-the-art performance in image generation. Encouragingly, DiTs exhibit remarkable scalability in model size and data representation, positioning them as a promising backbone for wide range of generative applications [4, 30, 65].
DiTs consist of n blocks, each containing a Multi-Head Self-Attention (MHSA) and a Pointwise Feedforward (PF) module [49, 11, 37], both preceded by their respective adaptive Layer Norm (adaLN) [38]. We illustrate the DiT Block structure in Figure 2 (Left). These blocks sequentially process the noised latent and conditional information, which are both represented
tokens in lower-dimensional latent space [39]. In each block, conditional input E Rddi is converted to scale and shift parameters (7, 3 E Rdin), which regressed through MLPs then injected into the noised latent E Ruxddi via adaLN:
(v,3) MLPs(c), adaLN(() LL(Z) + (1+7)+
(1)
where LN(-) is the standard Layer Norm [1]. These adaLN modules dynamically adjust the layer normalization before each MHSA and PF module, enhancing DiTs' adaptability to varying conditions and improving the generation quality.
Despite their effectiveness, DiTs require extensive computational resources to generate high-quality images, which impedes their real-world deployment. In this paper, we devise model quantization method for DiTs that reduces both time and memory consumption without necessitating re-training the original models, offering robust and practical solution for enhancing the efficiency of DiTs.

---

Activation
Weight
Max Abs. Value
12
Error
Max Abs. Valuee
Error
Salient Channel
Salient Channel
1000
Channel Index
Channel Index
Figure 3: Illustration of maximal absolute magnitudes of activation (left) and weight (right) channels DiT linear layer, alongside their corresponding quantization Error (MSE). Channels with greater maximal absolute values tend to incur larger errors, presenting fundamental quantization difficulty.
2.2 Model Quantization
Model quantization is compression technique that improves the inference efficiency of deep learning models by transforming full-pue cull-prcisson tensors into b-bit integer approximations, leading to direct computational acceleration and memory saving [33, 62, 8, 28, 19, 64]. Formally, the quantization process can be defined as:
(xx) clamp(1
X, 0, 1);
where denotes the full- cell-sreeisoon tensor, [-] is the round-to- nearest operator [32], and the clamp function restricts the quantized value within the range of [0, 2 1]. Here, 8 and À are quantization parameters subject to optimization. Among various quantization methods, Post-training Quantization (PTQ) is dominant approach for large quantized models, it circumvents the substantial resources required for model re-training [20, 52, 25, 44, 15]. PTQ employs small calibration dataset to optimize quantization parameters, which aims to reduce the performance gap between the quantized models and their full-precision counterparts with minimal data and computational expenses.
PTQ has been effectively applied to wide range of neural networks, including CNNs [20, 52, 25], Language Transformers [8, 57, 24, 23, 27], Vision Transformers [62, 13, 21, 291, and U-Net-based Diffusion models [44, 18, 51, 50]. Despite its demonstrated success, PTQ's applicability to Diffusion Transformers (DiTs) remains unexplored, presenting a significant open challenge within the research community. To bridge this gap, our work delves into the unique challenges of quantizing DiTs and introduces the first PTQ method for DiTs that can fruitfully preserve their generation performance.
Diffusion Transformer Quantization Challenges
Diffusion Transformers (DiTs) diverge from conventional generative or discriminative models [39, 11] through their unique design. Specifically, DiTs are constructed with a series large transformer blocks and operate under multi multi-timestep paradigm progressively transform pure noise into images. Our analysis reveals complex distribution patterns and temporal dynamics in the inference process of DiTs, identifying two primary challenges that prevent effective DiT quantization.
Pronounced Quantization Error in Salient Channels. The first challenge lies in systematic quantization errors in DiT's linear layers. As shown in Figure 3, activation and weight channels with significantly high absolute values are prone to substantial errors after quantization. We term these as salient channels, characterized by extreme values that greatly exceed the typical range of magnitudes. Upon uniform quantization (Eq. (2)), it is often necessary to truncate these extreme values order to maintain the precision of the broader set of standard channels. This compromise can result in notable deviations from the original full-proe precision distribution as the sampling process proceeds, especially given DiT's layered architecture and repetitive inference paradigm.
Temporal Variation in Salient Activation. Another challenge of DiT quantization arises from temporal variations in the magnitudes of salient activation channels. Rather than static inputs, DiT's operate across sequence of timesteps to generate high-quality images from random noise. Consequently, activation distributions can vary drastically within the inference process, which is particularly evident salient channels that dominate the signal. Figure 4 demonstrates that distribution of maximal absolute values in activation channels exhibits significant variations over different timesteps. This temporal variability introduces non-trivial difficulty to quantization optimization: Quantization parameters effective for salient activation channels at one timestep may

---

40
Max Abs Value
30
4
250
240
230
220
210
200
190
180
170
160
150
140
130
120
110
100
90
80
70
60
20
10
Timestep
Figure 4: Boxplot of maximal absolute magnitudes of activation chanmels in a linear layer within DiT over different timesteps, which exhibit significant temporal variations.
not be suitable at other timesteps. Such discrepancies can exacerbate quantization errors, cumulatively impairing the generation quality. Therefore, for accurate quantization, it is imperative to capture the evolving trait of salient channels throughout the entire denoising procedure.
PTQ4DIT
To overcome the identified challenges, propose Channel-wise Salience Balancing (CSB) and Spearman's p-guided Salience Calibration (SSC) our PTQ4DiT in Sections 4.1 and 4.2, respectively. Subsequently, devise re-parameterization scheme in Section 4.3, eliminating extra computational demands of PTQ4DiT during inference while maintaining the mathematical equivalence.
4.1 Channel-wise Salience Balancing
A linear layer {t; W) within MHSA and PF modules typically takes a token sequence X E RR as input and performs linear transformation with its weight matrix W E Rd xdout, formulated as W, where n the sequence length, and d and dout denote the input and output
f(X: W) dimensions, respectively. As discussed in Section 3, both the activation X and the weight matrix W exhibit salient channels that possess elements with significantly greater absolute magnitudes, which lead to large post-quantization errors.
Fortunately, large values do not coincide in the same channels of activation and weight, these extremes do not amplify each other, as observed in Figure 3. This property suggests the feasibility of complementarily redistributing the large magnitudes in salient channels between activation and weight, thereby alleviating quantization difficulties for both. Inspired by previous works on large model compression [54, 45, 23], we propose Channel wise Salience Balancing (CSB), which employs diagonal Salience Balancing Matrices B and B W to adjust the channel-wise distribution of activation and weight, as expressed by:
XB W
To address the quantization difficulties, need to achieve balanced distributions in X and W. which requires B and BW to capture the characteristies of salient channels. Considering that the quantization error is significantly influenced by the range of distributions [33, 57, 26], we measure the salience 8 of an activation or weight channel as the maximal absolute value among its elements:
X,, max (Xx)), (W;)) = max( W s)), where j 1,,,...., din.
(4)
Here, j is the channel index. Consequently, the balanced salience ă, representing the equilibrium between activation and weight channels, can be quantified using the geometric mean. Specifically, for the j-th channel, the balanced salience calculated follows:
8X,, W,) (s(X,) :(W,))
(5)
Building these concepts, proceed to construct the Salience Balancing Matrices, which modulate the salience of activations and weights with the guidance of s:
3(X1, W1) 5(X2, We)
((.... W s(Xdm))
B diag(
(6)
s(X))
s(x2)
X1, Wi)
(X2, W₂
X.i.. Wdin)
W diag(
(7)
s(W1)
8 s(w))
8( s(W
5

---

Following these, the balancing transformation defined by Eq. (3) will result in complementary redistribution of channel salience between activations and weights. Specifically, for each channel j. we have s s(X;) = s( W, = ; W),, thereby alleviating the quantization difficulties, as demonstrated by the reduction in overall channel salience:
(xxx) so (W) I max(s (X), so (W)).
(8)
Here, we characterize the overall salience 8, of activations or weights using the maximum salience across channels, e.g., (x() max(s(X;) (s(x)) s(x2),.... ..... s (Xd...), )), which reflects the distribution range of elements that are quantized collectively under certain granularity.
4.2 Spearman's p-guided Salience Calibration
Diffusion Transformers (DiTs) utilize an iterative denoising process for image sampling [37]. Under this sequential paradigm, the linear layer f receives inputs from an activation sequence (1:2)) (x x ,.... x which encompasses T timesteps. Targeting a certain timestep I, the salience of all activation and weight channels can be evaluated using Eq. (4):
s(X (1) = (=(x()) (x.)))
)), S ((W)= - (( 1). (W2). (W 4...).
While s( () remains consistent, we find that {s(X (s(x )/= exhibits significant temporal variations during the process of transforming purely random noise into high-quality images, as demonstrated in Figure 4. These fluctuations diminish the effectiveness of our CSB since quantization errors can be exacerbated by the biased estimation of activation salience among timesteps, resulting in degraded generation quality of the quantized models.
To accurately gauge the activation channel salience under multi-timestep scenarios, we propose Spearman's p-guided Salience Calibration (SSC). This offers a comprehensive evaluation of activation salience, with enhanced focus allocated to the timesteps where the complementarity property is more significant, facilitating effective salience balancing between activation and weight channels. Essentially, the lower the correlation between activation salience s (((()) and weight salience s (W), the greater reduction effect in overall channel salience (Eq. (8)). The intuition of SSC is visualized in Figure (Right). Mathematically, we formulate the Spearman's 's p-calibrated Temporal Salience sp by selectively aggregating the activation salience along timesteps:
s (1-T) (m 72. ... ..))
(s(X (1),, s(X) (2)) s s(X (T)) E Rdi",
(10)
where weighting factors {me )== are derived from a normalized exponential form of inverse Spear- man's p statistic [47, 55, 56]:
T expl-pls(X (W))
(11)
expl->(s(X (7)))))))))))))) sW))]
Here, p(-.)) computes the correlation between two sequences, and 1)t serves as the weighting factor for activation salience at timestep t. In this method, π inversely reflects the correlation coefficient p(s(X()) (t)), ), s( s(W)), thereby prioritizing timesteps where there is higher degree of complementarity in salience between activations and weights. Subsequently, we utilize sp for activation salience in Eqs. (5), (6), and (7), yielding refined Salience Balancing Matrices, denoted as B and BW By applying SSC, we calibrate the activation salience within CSB to strategically account for the temporal variations during the denoising process. Appendix B presents the full Algorithm for PTQ4DiT.
4.3 Re- Ce-Paarameterization
Before quantization, we estimate B- and BW on a small calibration dataset generated from multiple timesteps. Then, we incorporate these matrices into the linear layers within MHSA and PF modules [37] to alleviate the quantization difficulty. Given that B- and BW are mutual inverses, this incorporation maintains mathematical equivalence the original linear layer
X W (XB:) (BWW)
X W.
(12)
The proof is provided in Appendix C. Furthermore, we design a re- pa-aammtttrriationn scheme for DiTs, allowing for obtaining X and W without extra computational burden during inference. Specifically,

---

RepQ* (W4A8)
Q-Diffuusin (W4A8)
PTQ4D (W4A8)
Figure 5: Random samples generated by PTQ4DÎT and two strong baselines: RepQ* [21] and Q-Diffusion [18]. with W4A8 quantization on ImageNet 512x512 and 256x256. Our method can produce high-quality images with finer details. Appendix E. presents more visualization results.
update the weight matrix of linear layer to W offline and seamlessly integrate B into the preceding linear transformation operations. This integration includes adaptations to adaL.N [38, 37] and matrix multiplications within attention mechanisms [49]. Appendix A discusses these adaptations.
Post-adaLN, For linear layers following the adaLN module, we integrate B by adjusting the scale and shift parameters (y, 3 E Rd:") within adaLN:
X adaLN(()) (Z) LN(Z) (B 7) B where 7 7 B B 3-3B1 BB
(13)
Equivalently, we fuse Bx into the MI.ss responsible for regressing these parameters, thus avoiding additional computation overhead at inference time. Detailed derivations provided in Appendix D.
Post- Bos-Matrix-MMnn Murtriii-ttiiiiiiiiinnn For linear layers after matrix multiplication, the effeet PTQ4DiT can be realized by directly absorbing the Salience Balancing Matrices into the preceding de-quantization functions associated with the matrix multiplication [12, 53, 61].
Experiments
5.1 Experimental Settings
Our experimental setup is similar to the original study Diffusion Transformers (DiTs) [37]. We evaluate PTQ4DiT on the ImageNet dataset [41], using pre-trained class-conditional DiT-XL/2 models [37] at image resolutions of 256x and 512x512. The DDPM solver [17] with 250 sampling steps is employed for the generation process. To further assess the robustness of our method, we conduct additional experiments with reduced sampling steps of 100 and 50.
For fair benchmarking, all methods utilize uniform quantizers for all activations and weights, with channel-wise quantization for weights and tensor-wise for activations. unless specified otherwise. To construct the calibration set, we uniformly select 25 timesteps for 256-resolution experiments and 10 timesteps for 512-resolution experiments, generating 32 samples at each selected timestep. The optimization of quantization parameters follows the implementation from Q-Diffusion [18]. Our code based on PyTorch [36], and all experiments are conducted on NVIDIA RTX A6000 GPUs.
To comprehensively assess generated image quality, we employ four metrics: Fréchet Inception Distance (FID) [16]. spatial FID (sFID) [42, 34], Inception Score (IS) [42, 3], and Precision, all computed using the ADM toolkit [10]. For all methods under evaluation, including the full-precision (FP) models, we sample 10,000 images for ImageNet 256x256, and 5,000 for ImageNet 512x512. consistent with conventions from prior studies [35, 44].
5.2 Quantization Performance
We present comprehensive assessment of our PTQ4DiT against prevalent baseline methods in various settings. Our evaluation focuses on mainstream Post-training Quantization (PTQ) methods that widely used and adaptable to DiTs, including PTQ4DM [44], Q-Diffusion [18], and PTQD [15].
7

---

Table 1: Performance comparison ImageNet 256x256. '(W/A) indicates that the precision of weights and activations are W and A bits, respectively.
Timesteps
wit-width (W/A)
Method
(MB)
FID +
sFID 4
IS f
Precision
32/32
2575.42
4.53
17.93
278.50
0.8231
PTQ4DM
645.72
21.65
100.14
134.22
0.6342
Q-Diffusion
645.72
5.57
18.22
227.50
0.7612
8/8
PTQD
645.72
5.69
18.42
224.26
0.7594
RepQ
645.72
4.51
18.01
264.68
0.8076
250
Ours
645.72
4.63
17.72
274.86
0.8299
PTQ4DM
323.79
72.58
52.39
35.79
0.2642
O-Diffusion
323.79
15.31
26.04
134.71
0,6194
4/8
PTQD
323.79
16.45
22.29
130.45
0.6111
RepQ
323.79
23.21
28.58
104.28
0.4640
Ours
323.79
7.09
23.23
201.91
0.7217
32/32
FP
2575.42
5,00
19.02
274.78
0.8149
PTQ4DM
645.72
15.36
79.31
172.37
0.6926
Q-Diffusion
645.72
7.93
19.46
202.84
0.7299
8/8
PTQD
645.72
8.12
19.64
199.00
0.7295
RepQ"
645.72
5.20
19.87
254.70
0.7429
100
Ours
645.72
4.73
17.83
277.27
0.8270
PTQ4DM
323.79
89.78
57.20
26.02
0.2146
O-Diffusion
323.79
54.95
36.13
42.80
0.3846
4/8
PTQD
323.79
55.96
37.24
42.87
0.3948
RepQ
323.79
26.64
29.42
91.39
0.4347
Ours
323.79
7.75
22.01
190.38
0.7292
32/32
FP
2575.42
6.02
21.77
246.24
0.7812
PTQ4DM
645.72
17.52
84.28
154.08
0.6574
O-Diffusion
645.72
14.61
27.57
153.01
0.6601
8/8
PTQD
645.72
15.21
27.52
151.60
0.6578
RepQ"
645.72
7.17
23.67
224.83
0.7496
50
Ours
645.72
5.45
19.50
250.68
0.7882
PTQ4DM
323.79
102.52
58.66
19.29
0.1710
O-Diffusion
323.79
22.89
29.49
109.22
0.5752
4/8
PTQD
323.79
25.62
29.77
104.28
0.5667
RepQ
323.79
31.39
30.77
80.64
0.4091
Ours
323.79
9.17
24.29
179.95
0.7052
Table 2: Performance on ImageNet 512x512 with W4A8.
We reimplement these methods to suit the unique structure of DiTs. Considering the architectural simi- larity between DīTss and ViT' [11], analysis also includes RepQ- ViT [21], the state-of-the-art PTQ method initially designed for ViTs.
Timesteps
Method
FID +.
sFID +
IS
Precision f
FP
8.39
36.25
257.06
0.8426
PTQ4DM
68.43
57.76
35.16
0.4712
250
QDiffusion
58.81
56.75
31.29
0.4878
PTQD
87.53
74.55
34,40
0.5144
RepQ*
59.65
73.71
33.19
0.3676
We enhance RepQ-ViT (denoted
Ours
17.55
46.92
123,49
0.7592
RepQ*) by extending the cal- ibration set to integrate temporal dynamics and customizing its ad- vanced channel-wise and log quantizers specifically for DiTs.
FP
9.06
37.58
239.03
0,8300
PTQ4DM
70.63
57.73
33.82
0.4574
100
QDiffusion
62.05
$7.02
29.52
0.4786
PTQD
81.17
66.58
35.67
0.5166
RepQ*
62.70
73.29
31.44
0.3606
Tables and 2 report the outcomes on large-scale classccon image generation for ImageNet 256x256 and 512x512, respec- tively. Table demonstrates the effectiveness of PTQ4DiT across various quantization settings and timesteps. Notably, our finding re- veals that b-t precision (W8A8), PTQ4DiT closely matches the generative capabilities of the
Ours
19.00
50.71
121.35
0.7514
FP
11.28
41.70
213.86
0.8100
PTQ4DM
71.69
$9.10
33.77
0.4604
QDiffusion
53.49
50.27
38.99
0.5430
PTQD
73.45
59.14
39.63
0.5508
RepQ*
65,92
74.19
30.92
0.3542
Ours
19.71
52.27
118.32

---

FP models, whereas most base- line methods experience signif- icant performance losses. At the more stringent 4-bit weight precision (W4A8), all baseline methods exhibit more consider- able degradation. For instance, under 250 timesteps, PTQ4DM [44] sees drastic FID increase of 68.05. In contrast, our PTQ4DiT only incurs slight in- crease of 2.56. This resilience re- mains evident the number of timesteps decreases, underscor- ing the robustness of PTQ4DIT in essurce--limtee environments. Moreover, PTQ4DIT markedly outperforms mainstream methods at the higher 512x512 resolution, further validating its superiority, For example, using 250 timesteps, PTQ4DiT substantially lowers FID by 41.26 and sFID by 9.83 over the second-best method, Q-Diffusion. Figure depicts the efficiency-vs-efficae trade-off on W8A8 across various timestep configurations. Our PTQ4DiT achieves comparable performance levels to FP models but with considerably reduced computational costs, offering a viable alternative for high-quality image generation. Figures 5, 8, and also present randomly generated images for visual comparisons, highlighting PTQ4DiT's ability to produce images of superior quality.
ImageNet 256 x 256
ImageNet 512 512
14
Ours
Ours
O-Diffusion
35
O-Diffusion
12
30
20
- 250 steps
-
250 steps
steps
10
steps
30 steps
50 steps
18
20
22
24
26
36
38
40
42
44
46
48
sFID
sFID
Figure 6: Quantization performance on W8A8. The circle size represents the computational load (in Gflops).
5.3 Ablation Study
To verify the efficacy of CSB and SSC, conduct an ablative study the challenging W4A8 quantization. Experiments are performed on ImageNet 256 using 250 sampling timesteps. Three method variants considered in our ablation: (i) Baseline, which applies basic linear quantization on DiTs, (ii) Baseline + CSB, which integrates CSB in the linear layers within MHSA and PF modules, where the Salience Balancing Matrices B and B W estimated based distributions the midpoint timestep 2: and (iii) Baseline CSB + SSC, which is the complete PTQ4DËT. Results detailed in Table 3 indicate that each proposed component improves the performance, validating their effectiveness. Particularly, CSB enhances upon the Baseline by large margin, decreasing FID by 14.37 and sFID by 2.35, suggesting its critical role in alleviating the severe quantization difficulties inherent in DiTs. Note that with the addition of CSB, our method surpasses Q-Diffusion [18], leading PTQ method for diffusion models. Moreover, integrating SSC further boosts our PTQ4DiT towards state-of-the-at performance, facilitating high-quality image generation at W4A8 precision, as shown in Figure 5.
T.
Table 3: Ablation study on ImageNet 256x256 with W4A8.
Method
Size (MB)
FID 4
sFID 4
IS f
Precision
FP
2575.42
4.53
17.93
278.50
0.8231
Q-Diffusion
323.79
15.31
26.04
134.71
0.6194
Baseline
323.79
22.54
27.31
105.55
0.4791
CSB
323.79
8.17
24.96
187.94
0.7183
CSB + SSC (Ours)
323.79
7.09
23.23
201.91
0.7217
Conclusion
This paper proposes PTQ4DIT, a novel Post-training Quantization (PTQ) method for Diffusion Transformers (DiTs). Our analysis identifies the primary challenges in effective DiT quantization: the pronounced quantization errors incurred by salient channels with extreme magnitudes and the temporal variability in salient activation. To address these challenges, we design Channel-wise Salience Balancing (CSB) and Spearman's 's p-guided Salience Calibration (SSC). Specifically, CSB utilizes the complementarity nature salient channels to redistribute the extremes within activations and weights toward the balanced salience. SSC dynamically adjusts salience evaluations across different timesteps, prioritizing timesteps where salient activation and weight channels exhibit significant complementarity, thereby mitigating overall quantization difficulties. To avoid extra

---

computational costs of PTQ4DiT, we also devise re-parameterizztion strategy for efficient inference. Experiments show that our PTQ4DiT can effectively quantize DiTs 8-bit precision (W8A8) and further advance to 4-bit weight (W4A8) while maintaining high-quality image generation capabilities.
Acknowledgements. This research is supported by NSF IIS-2309073 and ECCS-2123521. This article solely reflects the opinions and conclusions of authors and not funding agencies.
References
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization, arXiv preprint 1rXiv:1607.064500 2016.
[2] Fan Bao, Chongxuan Li, Yue Cao, and Jun Zhu. All worth words: vit backbone for score-based diffusion models. In NeurlPSW, 2022.
[3] Shane Barratt and Rishi Sharma. A note on the inception score. arXiv preprint arXivv1880.0011 2018.
[4] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing. David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024.
[5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[6] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao. Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In ICLR, 2024.
[7] Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor lonescu, and Mubarak Shah. Diffusion models in vision: A survey. IEEE TPAMI, 2023.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3, int8 (): 8-bit matrix multiplication for transformers at scale. In NeurlPS, 2022.
[9] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In NeurIPS, volume 34, pages 8780-8794, 2021.
[10] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In NeurlPS, volume 34, pages 9780-8794, 2021.
[11] Alexey Dosovitskiy, Lucas Beyer. Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[12] Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dhar- mendra S Modha. Learned step size quantization. In ICLR, 2020.
[13] Natalia Frumkin, Dibakar Gope, and Diana Marculescu. Jumping through local minima: Quantization in the loss landscape of vision transformers. In ICCV, pages 16978-16988, 2023.
[14] Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer a strong image synthesizer. In ICCV, pages 23164 22113, 2023.
[15] Yefei He, Luping Liu, Jing Liu, Weijia Wu, Hong Zhou, and Bohan Zhuang. Ptqd: Accurate post-training quantization for diffusion models. In NeurIPS, volume 36, 2023.
[16] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. In NewrIPS, volume 30, 2017.
[17] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, volume 33, pages 6840-6851, 2020.
[18] Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang. Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. Q-diffusion: Quantizing diffusion models. In ICCV, pages 17535-17545, 2023.
[19] Yanjing Li, Sheng Xu, Xianbin Cao, Xiao Sun, and Baochang Zhang. Q-dm: An efficient low-bit quantized diffusion model. In NeurIPS, volume 36, 2023.
10

---

[20] Yuhang Li, Ruihao Gong. Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Brecq: Pushing the limit of post-training quantization by block reconstruction. In ICI.R, 2021.
[21] Zhikai Li, Junrui Xiao, Lianwei Yang, and Qingyi Gu. Repq-vit: Scale reparameterization for post-training quantization of vision transformers. In ICCV, pages 17227-17236, 2023.
[22] Haokun Lin, Haoli Bai, Zhili Liu, Lu Hou, Muyi Sun, Linqi Song, Ying Wei, and Zhenan Sun. Mope-clip: Structured pruning for efficient vision-language models with module-wise pruning error metric. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27370-27380, 2024.
[23] Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song, Zhenan Sun, and Ying Wei. Duquant: Distributing outliers via dual transformation makes stronger quantized 1lms. arXiv preprint arXiv:2406. 01721, 2024.
[24] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang. and Song Han. Awq: Activation- aware weight quantization for IIm compression and acceleration. In MLSys, 2024.
[25] Jiawei Liu, Lin Niu, Zhihang Yuan, Dawei Yang, Xinggang Wang, and Wenyu Liu. Pd-quant: Post-training quantization based on prediction difference metric. In CVPR, pages 24427-24437, 2023.
[26] Jing Liu, Ruihao Gong, Xiuying Wei, Zhiwei Dong, Jianfei Cai, and Bohan Zhuang. Qllm: Accurate and efficient low-bitwidth quantization for large language models. In ICLR, 2024.
[27] Ruikang Liu, Haoli Bai, Haokun Lin, Yuening Li, Han Gao, Zhengzhuo Xu, Lu Hou, Jun Yao, and Chun Yuan. Intactkv: Improving large language model quantization by keeping pivot tokens intact. arXiv preprint arXiv:2403.011 01241, 2024.
[28] Shin-Yang Liu, Zechun Liu, and Kwang-Ting Cheng. Oscillation-free quantization for low-bit vision transformers. In ICML, 2023.
[29] Yijiang Liu, Huanrui Yang, Zhen Dong, Kurt Keutzer, Li Du, and Shanghang Zhang. Noisyquant: Noisy bias-enhanced post-training activation quantization for vision transformers. In CVPR, pages 20321 203330, 2023.
[30] Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, et al. Sora: A review background, technology, limitations, and opportunities of large vision models. arXiv preprint arXiv:2402. 17177, 2024.
[31] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[32] Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. In ICML, pages 7197-7206,
2020.
[33] Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart Van Baalen, and Tijmen Blankevoort. A white paper on neural network quantization. arXiv preprins arXiv:2106.0829 2021.
[34] Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W Battaglia. Generating images with sparse representations. arXiv preprint arXiv:2103.0388 2021.
[35] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, pages 8162-8171. PMLR, 2021.
[36] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, perfoppeformance deep learning library. In NeurIPS, volume 32, 2019.
[37] William Peebles and Saining Xie. Scalable diffusion models with transformers. In CVPR, pages
4195 4205, 2023.
[38] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In AAAI, 2018.
[39] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pages 10684-10695, 2022.
[40] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, pages 234-241. Springer, 2015.
11

---

[41] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 11::211-252, 2015.
[42] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung. Alec Radford, and Xi Chen. Improved techniques for training gans, In NeurlPS, volume 29, 2016.
[43] Yuzhang Shang, Zhihang Yuan, Qiang Wu, and Zhen Dong. Pb-llm: Partially binarized large language models. In ICLR, 2024.
[44] Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan. Post-training quantization on diffusion models. In CVPR, pages 1972-1981, 2023.
[45] Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantiza- tion for large language models. In ICLR, 2024.
[46] Jascha Sohl-Diekstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsu- pervised learning using nonequilibrium thermodynamics. In ICML, pages 2256-2263 PMLR, 2015.
[47] Charles Spearman. The proof and measurement of association between two things. 1961.
[48]
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers and distillation through attention. In ICML, 2021.
[49] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
[50] Changyuan Wang, Ziwei Wang, Xiuwei Xu, Yansong Tang, Jie Zhou, and Jiwen Lu. Towards accurate post-training quantization for diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16026-16035, 2024.
[51] Haoxuan Wang, Yuzhang Shang, Zhihang Yuan, Junyi Wu, and Yan Yan. Quest: Low-bit diffusion model quantization via efficient selective finetuning. arXiv preprint arXiv:2402.03666, 2024.
[52] Xiuying Wei, Ruihao Gong, Yuhang Li, Xianglong Liu, and Fengwei Yu. Qdrop: Randomly dropping quantization for extremely low-bit post-training quantization. In ICLR, 2022.
[53] Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling. In EMNLP, 2023.
[54] Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer language models. In NeurlPS, 2022.
[55] Junyi Wu, Bin Duan, Weitai Kang, Hao Tang, and Yan Yan. Token transformation matters: Towards faithful post-hoc explanation for vision transformer. In CVPR, 2024.
[56] Junyi Wu, Weitai Kang, Hao Tang, Yuan Hong, and Yan Yan. On the faithfulness of vision transformer explanations. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 10936-10945, 2024.
[57] Guangxuan Xiao, Ji Lin, Mickael Seznee, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In ICML, pages 38087-38099. PMLR, 2023.
[58] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. In NeurlPS, volume 34, pages 12077-12090. 2021.
[59] Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong. Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys. 56(4):1-39, 2023.
[60] Xiulong Yang, Sheng-Min Shih, Yinlin Fu, Xiaoting Zhao, and Shihao Ji. Your vit secretly a hybrid discriminnatvv--eeeraaii diffusion model. arXiv preprint arXin:228.00799), 2022.
12

---

[61] Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. Asvd: Activation-awa aware singular value decomposition for compressing large language models. arXiv preprini arXiv:2312.0588// 2023.
[62] Zhihang Yuan, Chenhao Xue, Yiqi Chen, Qiang Wu, and Guangyu Sun. Ptq4vit: Post-training quantization for vision transformers with twin uniform quantization. In ECCV, pages 191-207,
2022.
[63] Aozhong Zhang, Naigang Wang. Yanxia Deng, Xin Li, Zi Yang. and Penghang Yin. Magr: Weight magnitude reduction for enhancing post-training quantization. In NeurIPS, 2024.
[64] Aozhong Zhang. Zi Yang, Naigang Wang, Yingyong Qin, Jack Xin, Xin Li, and Penghang Yin. Comq: A backpropagation-free algorithm for post-training quantization, arXiv preprint arXiv:2403.07134 134, 2024.
[65] Zheng Zhu, Xiaofeng Wang, Wangbo Zhao, Chen Min, Nianchen Deng, Min Dou, Yuqi Wang. Botian Shi, Kai Wang, Chi Zhang, al. Is sora world simulator? comprehensive survey on general world models and beyond. arXiv preprint arXiv:2405.03520, 2024.
13

---

A
Structures of MHSA and PF with Adjusted Linear Layers
Integrating p
offline
DiT Block
xBB
Integrating x offline
FC2
scale
02
GELU
Linear Layer f
Pointwise
FC1
Feedforward
Integer Matrix Multiplication
12,82
adaLN
Projection2
Quantizer
Quantizer
scale
W
Softmax
Multi-Head
Self-Attention
CSB ssc
n,
adaLN
K
MLPs
Input
Projection1
W
Tokens
Conditioning
Figure 7: Illustration of structures of the MHSA and PF modules within DiT Blocks [37]. Our proposed CSB and SSC are embedded in their linear layers, including Projectionl, Projection2, and FC1. CSB and SSC collectively mitigate the quantization difficulties by transforming both activations and weights using Salience Balancing Matrices, B" and B To prevent extra computational burdens at inference time, B is absorbed into the weight matrix of the linear layer f. Meanwhile, B is integrated offline into the MLPs layer prior to adaLN modules for Projection and FC1, and into the preceding matrix multiplication operation for Projection2.
The Multi-Head Self-Attention (MHSA) and Pointwise Feedforward (PF) modules are essential for processing input tokens and conditional information in DiT Blocks [37]. As depicted in Figure Channel-wise Salience Balancing (CSB) and Spearman's 's p-guided Salience
7, we incorporate Calibration (SSC) techniques into the linear layers within both modules. These techniques are designed to mitigate the quantization difficulties by dynamically adjusting the salience of activations and weights via Salience Balancing Matrices. Through the adjustments, CSB and SSC allow for more uniform distributions of activation and weight magnitudes across salient channels, reducing the impact extreme values and enhancing the overall stability of the quantization process.
To eliminate additional computational demands during inference, the Salience Balancing Matrices, BW and B are pre-integrated p p
the DiT Blocks. Specifically, we replace the weight matrix of the linear layer with W BW W and integrate B- into the preceding linear transformations. For Projection and FCI, B is absorbed into the MLPs before the adaLN modules, while for Projection2, it can be absorbed within the matrix multiplication [12, 53, 43]. Derivations of the integration are provided in Appendix
B PTQ4DIT Pipeline
This section provides a comprehensive description of the PTQ4DiT pipeline, detailed in Algorithm 1. The PTQ4DiT designed to enhance the performance of quantized DiTs by addressing quantization challenges through sophisticated salience estimation and balancing strategies.
The full algorithm mainly consists of five steps: The pipeline begins with estimating activation and weight salience for the pre-trained model using calibration dataset. Following the estimation, we employ Spearman's 's p-guided Salience Calibration compute correlation coefficients between activation salience and weight salience, which helps determine the weighting factors for each timestep. These factors are crucial for computing a temporally adjusted salience, which aims to minimize quantization emror that typically occur due misalignment in salience peaks across the timesteps. & The Channel-wise Salience Balaneing step follows, wherein Salience Balancing Matrices are constructed to redistribute the activation and weight values channel Specifically, for each
14

---

Algorithm Post-Training Quantization for Diffusion Transformers (PTQ4DiT)
Input: Pre-trained DiT model, Activation sequence ( from calibration dataset
2: Output: Quantized DiT model with low-bit activations and weights
3: ! Preparation:
4: Estimate activation salience s(X(1) each timestep
Using Eq. (9)
5: Estimate weight salience s( s(W)
> Using Eq. (9)
6: @ Spearman's ' p-guided Salience Calibration:
7: Compute correlation coefficients ()))(
8: Compute weighting factors {TT 12"
b. Using Eq. (11)
9: Compute temporal salience s, (x
Using Eq. (10)
S Channel-wise Salience Balancing:
11: Compute balanced salience sp {x (1:T), Wj) for each channel j
b Using Eqs. (5), (14)
12: Construct refined Salience Balancing Matrices B and BW
Using Eqs. (6), (7)
13: ! Re-Parameterization:
14: Integrate BW into the weight matrix of linear layers offline
By W = BWW W
15: Integrate B into the MLPs before adaLN modules offline
Using Eqs. (13), (20)
16: ① Quantization:
BB W without extra computational demand during inference 18: Perform quantization on balanced activation X and weight W
17: Obtain X XB and W = BD
channel j. the balanced salience (X, (1:T)
W,) is given by:
(1:2) W = (sp (X (1:2))
(x(11)))) . ,(W,))
(14)
where sp(X(11)) is the j-th element of s (x(11))) Then, we formulate the refined Salience Balancing Matrices Bº and B,W based on 5 (x, This step is pivotal in aligning the activation and weight distributions, thereby minimizing the overall quantization difficulty. 4 In the Re-Parameterization phase, these balancing matrices are integrated into the pre-trained model, ensuring that no additional computational cost required during inference. This integration maintains computational efficiency while retaining the benefits of our salience balancing technique. ① Finally, we perform quantization on the model with balanced activations and weights, setting the stage for the deployment of efficient and effective quantized DiTs in resource-conee constraieee environments.
(I:T), W,) and s, sx.(((())) as detailed in Eq. (15).
C Proof of Mathematical Equivalence
In this section, we provide detailed proof demonstrating that our PTQ4DiT maintains mathematical equivalence to the original linear layers. This proof ensures that the balancing operation does not alter the original computational outcomes of the full-precision models.
In PTQ4DIT, introduce the Salience Balancing Matrices BB and BW, which are diagonal matrices intended to balance the salience across activation and weight channels. We verify the inverse relationship of B and BW mentioned in Section 4.3:
B
B W
X (1:2)
W1)
(1:T) don
Waih),
(X
(1:2)
Wi)
(x((11 (1:T)
dim WWan)
diag
diag(
sp (x
8p din (1:
s(1))
s(W den)
(1:T) W1) (x((1))) Wi)
(x (1:T)
W Wd...
5p 5x((((
(1:T) W
dia
diag
dim
(x(1:2)
s(W1)
(1:T))
s(Wd..)
Sp
din
(sp (x(1:)) (())))
(xp (1:T)
s(W 3)
=
diag(
sx
(1:TT) s (i)
spl did
(x((x( (1:T)) (Wa...)
(15)
15

---

where I denotes the identity matrix. Therefore, we can derive the mathematical equivalence:
X W (XB:) (Br W) (BA W = X.W
(16)
D Derivations of Post-adaLN Integration
This section details the integration of the Salience Balancing Matrix B into the MLPs before the adaptive Layer Norm (adaLN) modules [37], aimed at eliminating extra computational overhead at the inference stage. Recall that the initial formulation of adaLN on the input latent noise Z R given by:
X = aZ) LZ (1 + 7) + B,
(17)
where y, B Rd are scale and shift parameters, respectively, regressed by MLPs based on the conditional input с E 1ddi;
(T MLPs(c) = c. (W,, Wa) + (by, bg).
(18)
Here, Wy, Wa weight matrices, and by, by bias terms. In PTQ4DiT, X is obtained by applying B the output of adaL as follows:
X XB LN(Z) (B² +B 3B
(19)
which echos with Eq. (13). To avoid additional matrix multiplications yB and 3BX pre-absorb Bx in MLPs' weights and biases offline, expressed as:
p. we can
W 7, W B, ) (bx. (3) = Bb, b,
(20)
Thus, the re-parameterized MLPs can directly produce the adjusted scale and shift parameters:
MLPs(c)
<W, Wg) + (by, bp)
p W WaB (byB, baBx) = =cc. (c.(W,, Wa) + (by: bB)) B
=c. CW,BB
p
(21)
-7,,3) B
= (y p. xB
This allows for obtaining X without extra computational burden at the inference stage.
E Additional Visualization Results
Figures 8 and supplement visualization results of PTQ4DIT W8A8 quantization, compared with baseline PTQ methods and the full- puel-irecision (FP) counterpart, on ImageNet 512x512 and 256 226. Our method generates results that closely mirror those of the FP models, presenting finer details and richer semantic content than the baseline approaches.
Limitations and Broader Impacts
This work introduces a pioneering solution facilitating the broad deployment of Diffusion Transform- ers (DiTs) through Post-training Quantization. Our method substantially lowers computational and memory demands, thereby improving the accessibility of DiTs. Currently, our research concentrates on visual generation. For future work, we plan to extend our methodology to other generative models across various modalities, such as audio and 3D. However, there remains an inherent risk that these generative models could be utilized to produce disinformation. While our study contributes to the widespread application of DiTs, it does not address such ethical risks. We recognize the importance of developing safeguards and encourage further research into strategies that can prevent the misuse of these powerful generative technologies.
16

---

RepQ* (W8A8)
Q-Diffusion (W8A8)
PTQD (W8A8)
PTQ4DIT (W8A8)
Full-Precision
Figure 8: Random samples generated by different PTQ methods with W8A8 quantization, alongside
the full-precision DiTs [37], on ImageNet 512x512.
17

---

RepQ* (W8A8)
Q-Diffusion (W8A8)
PTQD (W8A8)
PTQ4DIT (W8A8)
Full-Precision
Figure 9: Random samples generated by different PTQ methods with W8A8 quantization, alongside the full-precision DiTs [37], on ImageNet 256
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims in the abstract and introduction accurately reflect the contributions the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
See Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
19

---

Justification:
See Appendices C and D.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification:
The code will be released publicly.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips. wttpccccpppccccceeececceccccccct Sttp://///.iieeeeeeeuuubiiiiiiii /CodeSubmic ssisnnoillll onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
21

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttp///eerpp../// cc
Answer: [Yes]
Justification:
The research conducted in this paper adheres the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Appendix F.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
22

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All external assets such as datasets and code libraries used in the research are properly credited, and their licenses are respected and clearly mentioned.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
23

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "vh89s5TMyt", "reviewer": "Reviewer_N2VF", "summary": "This paper proposes a new PTQ method for DiTs. It develops a Channel-wise Salient Balancing method to suppress the outliers of linear layers in transformer blocks when applying activation quantization. Besides, it designs the Spearmen’s ρ-guided Salience Calibration to tackle the timestep dimension’s variety. It can improve the performance of the quantized models, comparing to directly applying the conventional quantization methods for Unet-based diffusion models, ViTs or traditional convolution networks to DiTs.", "questions": "1. Please see the weaknesses.\n2. Did the authors quantize the post-softmax layer? \n4. The SOTA DiT models can achieve a FID around 3 on ImageNet, why are the results reported in this paper not aligned with the origin paper? Why did the authors choose different settings?", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+This paper proposes a new method for DiT quantization, which achieves relatively well performance on W8A8 and W4A8.", "weaknesses": "-Lacks of experiments. This paper does not report the W8A8 results of ImageNet 512x512. Besides, it only conducts the experiment on one DiT model. The proposed method should be validated on more architectures.\n-Lack of novelty. The idea of Re-Parameterization activation range has already been used in PTQ of ViT and LLM, such as Rep-Q [1] or outlier suppression+ [2]. \n-Lack of direct evidence. It lacks direct evidence to display the effectiveness of the proposed method, such as the visualization of activation before and after CSB and SSC to verify it actually suppress the salience channel. \n-Lack of motivation. This paper proposes several modules, but there is no direct evidence to prove that these modules can solve the problems. For example, in RepQ’s, it chooses the mean value of activation range to scale the distribution, but this paper uses the equilibrium between weight and activation without explanation. Besides, it also not explains why using inverse Spear-man’s ρ statistic as the weigh.\n[1]Li Z, Xiao J, Yang L, et al. Repq-vit: Scale reparameterization for post-training quantization of vision transformers. In ICCV, 2023.\n[2]Wei X, Zhang Y, Li Y, et al. Outlier Suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling. In EMNLP, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Weakness 1: More experiments**\n**a. W8A8 ImageNet 512x512**\n\nWe evaluate PTQ4DiT on ImageNet 512x512 with W8A8, compared against strong Diffusion PTQ methods, including Q-Diffusion and PTQD:\n|Timesteps|Method|FID↓|sFID↓|IS↑|Precision↑|\n|---|---|---|---|---|---|\n|250|FP|8.39|36.25|257.06|0.8426|\n||Q-Diffusion|39.91|47.91|60.78|0.644|\n||PTQD|80.04|70.11|45.67|0.5636|\n||**Ours**|**11.45**|**38.34**|**196.21**|**0.8266**|\n|100|FP|9.06|37.58|239.03|0.83|\n||Q-Diffusion|38.77|46.77|63.82|0.6504|\n||PTQD|90.77|79.16| 42.47|0.5392|\n||**Ours**|**13.10**|**39.92**|**173.37**|**0.7866**|\n|50|FP|11.28|41.70|213.86|0.81|\n||Q-Diffusion|37.51|**44.46**|69.55|0.642|\n||PTQD|85.28|74.75|46.42|0.5458|\n||**Ours**|**16.56**|45.52|**170.20**|**0.7944**|\n\nPTQ4DiT consistently outperforms mainstream methods across various timesteps. Figure 8 provides the images generated by PTQ4DiT in this setting, which closely mirror the FP model.\n\n**b. New DiT model**\n\nPlease refer to **General Response 1: Generality of PTQ4DiT**.\n## **Weakness 2: Novelty discussion**\n**a. The general concept of re-parameterization**\n\nThe fundamental idea of re-parameterization in quantization is to absorb the extra affine transformation into adjacent layers, thereby enabling the construction of quantization-friendly distributions without incurring extra inference costs.\n\n**b. Existing works that use re-parameterization**\n\nSeveral quantization methods benefit from re-parameterization. For example, RepQ-ViT [1] uses scale re-parameterization to avoid the extra costs of channel-wise and $log\\sqrt2$ quantizers for post-LayerNorm and post-Softmax activations in ViTs. OS+ [2] shifts and scales activations to address the asymmetry and concentration issues in LLMs and re-parameterizes them to avoid extra costs.\n\n**c. Unique innovation and contribution of PTQ4DiT**\n\nWhile PTQ4DiT shares the general concept of scale re-parameterization, its unique innovation lie in 3 aspects:\n\n**(I)** We explore the quantization of **DiTs**, while existing works focus on **ViTs** or **LLMs**.\n\n**(II)** We delve into the complementarity of **activations and weights**, while existing works [1,2] focus solely on **activations**. Specifically, we find that DiTs exhibit extreme values in both activations and weights, yet these extreme values do not simultaneously occur in the same activation and weight channels. We leverage this property to redistribute salience between activation and weight, alleviating quantization errors for both.\n\n**(III)** Unlike the **static** re-parameterization in existing works [1,2], PTQ4DiT **dynamically** adapts to the temporal variability of channel salience, a special characteristic of DiTs.\n\n[1] RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformers. ICCV 2023.\n\n[2] Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling. EMNLP 2023.\n\n## **Weakness 3&4: Direct evidence and Motivation**\n**a. Direct evidence**\n\nPlease refer to **General Response 2: Experiment on direct evidence**.\n\n**b. Motivation of equilibrium**\n\nUnlike ViTs and LLMs which exhibit outliers **only** in activations, extreme values are presented in **both** activations and weights in DiTs. Fortunately, they do not coincide in the same activation and weight channels. Such complementarity property inspired us to seek a balance between activations and weights, where their extreme values can be mitigated. This balance is the essence of **equilibrium**. To realize this, we adopt geometric mean as it is suitable for the multiplicative relationship between activations and weights.\n\n**c. Motivation of inverse ρ**\n\nThe motivation of inverse Spearman's ρ stems from the need to balance salient weights with salient activations from different timesteps, which have various degrees of complementarity.\n\nSpearman's ρ measures the rank correlation between two variables (the statistical dependence between their rankings). A high ρ suggests that two variables exhibit higher values simultaneously, which **inversely** reflects the complementarity. In our context, the high ρ implies that the same channel tends to have extreme values in both activations and weights, hindering the salience balancing.\n\nTo counteract this, our SSC inversely weights timesteps with high ρ between activation and weight salience, thereby prioritizing timesteps with more significant complementarity. This approach helps distribute the extreme values more evenly across channels and timesteps, improving overall quantization performance.\n## **Question 2: Post-softmax layer**\nWe quantize the post-softmax layer.\n## **Question 3: Generation setting**\n**a. Clarification on FID-10K**\n\nThe origin DiT paper [1] achieves a 2.27 **FID-50K** on ImageNet 256x256.\n\nIn our work, the only difference is that we use **FID-10K**, following pioneering studies on diffusion such as [2,3,4]. Specifically, the work on IDDPM by OpenAI [3] indicates that while FID-10K may result in a slight increase, it significantly reduces computational resource demands.\n\nThus, we use FID-10K to facilitate more experiments for comprehensive evaluation. For fair comparisons, all baseline methods and our PTQ4DiT are evaluated using the same metrics.\n\n**b. Experiment on 50K samples**\n\nWe perform an additional experiment generating 50K samples on ImageNet 256x256 as in [1], evaluating W8A8 quantization. The results indicate that PTQ4DiT effectively recovers the performance and closely matches the FP model:\n|Method|FID-50K↓|sFID-50K↓|\n|---|---|---|\n|FP|2.27|4.60|\n|PTQ4DiT|2.31|4.82|\n\nIn this setting, generating 50K samples takes around 167 hours on an NVIDIA RTX A6000, which is less feasible for extensive comparisons.\n\n[1] Scalable diffusion models with transformers. CVPR 2023.\n\n[2] Diffusion Models Beat GANs on Image Synthesis. NIPS 2021.\n\n[3] Improved denoising diffusion probabilistic models. ICML 2021.\n\n[4] Post-training Quantization on Diffusion Models. CVPR 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pvuyxgzvcI", "reviewer": "Reviewer_USPH", "summary": "This paper proposes the first post-training quantization (PTQ) method for Diffusion Transformer (DiT). It addresses the presence of salient channels with extreme magnitudes and the temporal variability in the distributions of salient activations over multiple timesteps. Experimental results demonstrate comparable performance in low-bit scenarios.", "questions": "Please refer to the weaknesses part.", "limitations": "Yes. The authors have addressed the limitations and broader impacts.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-organized and includes clear illustrations.\n2. It presents the first post-training quantization method for Diffusion Transformers.\n3. In the W8A8 scenario, PTQ4DiT achieves lossless performance compared to full precision.\n4. The paper also includes theoretical analysis.", "weaknesses": "1. In line 158, the author proposes an important property: large values do not coincide in the same channels of activation and weight. This property is crucial for the proposed method but is only briefly illustrated by a sketch. We believe that such an important property should be demonstrated with statistical experiments.\n2. In line 170, the author attempts to use the geometric mean to balance activation and weight channels. Is this approach well-designed? Could another type of mean be used?\n3. PTQ4DiT is designed for Diffusion Transformers and is tested on traditional DiT. However, many new DiT-based models, such as pixart-alpha [1], pixart-sigma [2], SD3 [3], and Lumina [4], have emerged. To verify the generality of the method, it should be tested on a broader range of DiT-based models.\n4. This method appears to work only with linear layers.\n\n[1] PixArt-$\\alpha$: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis, ICLR24.\n[2] PixArt-\\Sigma: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation, arXiv.\n[3] Scaling Rectified Flow Transformers for High-Resolution Image Synthesis, arXiv.\n[4] Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers, arXiv.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your acknowledgement of our work and your constructive feedbacks and suggestions.\n## **Response to Weakness 1: Statistical experiments for the important property**\nWe validate the important complementarity property by additional statistical experiments on the well-established Jaccard similarity [1].\n\n**a. Jaccard similarity**\n\nTo quantitatively demonstrate the **complementarity** property (large values do not coincide in the same channels of activation and weight), we measure the **Jaccard similarity** [1] of salient activation and weight channels:\n$$\nJaccard(S_A,S_W)=\\frac{|S_A\\cap S_W|}{|S_A\\cup S_W|}\\in[0,1],\n$$\nwhere $S_A$ is the set of indices of salient activation channels, $S_W$ is the set of indices of salient weight channels, and $|S|$ calculates the number of elements in set S. In this way, a lower $Jaccard(S_A, S_W)$ reflects stronger complementarity.\n\n**b. Detecting salient channels**\n\nTo accurately construct the sets $S_A$ and $S_W$, we utilize a robust statistical outlier detector, **Interquartile Range (IQR)** [2], to identify the salient channels among all channels. This method identifies data points that lie significantly outside the middle 50% of the distribution as outliers. We perform IQR detection on channels' maximal absolute values to identify the salient channels.\n\n**c. Statistical experiment**\n\nWe randomly select 100 samples for ImageNet 256x256 generation with total timestep T = 250. We evaluate the average Jaccard similarities at t = $\\frac{1}{4}T, \\frac{1}{2}T,$ and $\\frac{3}{4}T$. The results are averaged over 100 samples and linear layers in DiTs:\n|Timestep t|$\\frac{1}{4}T$|$\\frac{1}{2}T$|$\\frac{3}{4}T$|\n|---|---|---|---|\n|**$$Jaccard(S_A, S_W)$$**|0.0675|0.1364|0.1006|\n\nOur finding are:\n\n**(I)** We obtain relatively low Jaccard similarities (note that $Jaccard \\in [0, 1]$), suggesting a significant complementarity between salient activation and weight channels. The complementarity property motivates our Channel-wise Salience Balancing (CSB) to redistribute extreme values among non-overlapping salient channels.\n\n**(II)** Different timesteps t exhibit various Jaccard similarities, aligning with the key observation in our paper: the temporal variation in salient channels. Such variability further indicates the necessity of our proposed Salience Calibration method (SSC) for various timesteps.\n\n[1] Distance between sets. Nature 1971.\n\n[2] Outlier detection: Methods, models, and classification. ACM Computing Surveys (CSUR) 2020.\n\n\n## **Response to Weakness 2: Is the geometric mean well-designed?**\n**a. The effectiveness of geometric mean**\n\nThe geometric mean was selected due to its suitability for the multiplicative relationship between activations and weights (Eq. (3) in our paper) and its ability to balance and minimize the influence of extremely large values, which is critical in our quantization method.\n\n**b. The uniqueness of geometric mean**\n\nMathematically, alternative means could include the arithmetic, quadratic, or harmonic mean. However, interestingly, we found that the geometric mean is uniquely capable of ensuring the mathematical equivalence of the balancing transformation (as expressed by Eqs. (12), (15), (16) in our paper). Here, we provide a brief demonstration:\n\n$$\n\\quad\\frac{mean(s(X_j), s(W_j))}{s(X_j)}\\cdot \\frac{mean(s(X_j), s(W_j))}{s(W_j)} = 1\n$$\n$$\n\\Leftrightarrow\nmean(s(X_j), s(W_j)) = (s(X_j) \\cdot s(W_j))^{\\frac{1}{2}}\n$$\n\nThis essential property offers the feasibility of reparameterization, thereby eliminating extra computation overhead of the balancing transformation during inference.\n\n## **Response to Weakness 3: Generality of PTQ4DiT**\nPlease refer to **General Response 1: Generality of PTQ4DiT**.\n\n## **Response to Weakness 4: Work only with linear layers**\n**a. Extending PTQ4DiT to other structures**\n\nAlthough our method is derived from the quantization of linear layers, the underlying idea of balancing transformation can be seamlessly extended to other structures, such as convolutional layers. This can be achieved by reformulating the convolution operation into a matrix multiplication between activations and weights [1], which is common in the practical implementation of convolution [2].\n\n**b. The reason for focusing on linear layers**\n\nThe linear layers in Transformers incur significant computational and memory overhead due to the large matrix multiplications [3, 4, 5, 6], representing the primary efficiency bottleneck in Transformer models including Diffusion Transformers (DiTs).\n\n[1] Solving Oscillation Problem in Post-Training Quantization Through a Theoretical Perspective. CVPR 2023.\n\n[2] Optimizing hardware accelerated general matrix-matrix multiplication for cnns on fpgas. Transactions on Circuits and Systems 2020.\n\n[3] PTQ4ViT: Post-training quantization for vision transformers with twin uniform quantization. ECCV 2022.\n\n[4] RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformers. ICCV 2023.\n\n[5] OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models. ICLR 2024.\n\n[6] QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models. ICLR 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PJraSMRUOj", "reviewer": "Reviewer_jrKh", "summary": "- The paper introduces PTQ4DiT, a post-training quantization method for Diffusion Transformers. This approach can facilitate the widespread deployment of DiTs. By investigating the distribution of activation and weight of DiTs, PTQ4DiT designs the Channel-wise Balancing and timestep-aware Salience Calibration. Through these, PTQ4DiT effectively quantizes DiTs to W4A8 and reduce computational costs while maintaining image generation quality.", "questions": "- Is the 'Spearman’s ρ-guided Salience Calibration (SSC)' method capable of generalizing across a different timesteps and solvers? It seems that the calibration process depends on the number of timesteps. Does this imply that recalibration is necessary when employing SSC with different models, timesteps, or solvers?\n- In the context of reparameterization, the paper specifically addresses the 'Post-adaLN' and 'Post-Matrix-Multiplication' scenarios. However, within the Feed-Forward Network (FFN) layers of DiTs, there are instances where two consecutive linear layers do not conform to either of these cases. It raises the question of how to approach reparameterization in such circumstances.\"\n- Some unclear details about the quantization process:\n  - Are all layers in the DiTs quantized? e.g., time embedding linear layers, attention matmuls. \n  - In Line 237, the authors state that \" the optimization of quantization parameters follows the implementation of Q-Diffusion\". Does this mean that the quantization process involves gradient-based optimization of scaling factors, and Adaround of zero-points? What is the cost of the PTQ process. \n- The statement “the lower the correlation between activation salience s(X(t)) and weight salience s(W), the greater the reduction effect in overall channel salience”  in Sec. 4.2 should be more clearly explained. What is the definition of \"correlation salience\"?", "limitations": "- The authors have discussed the limitations in the appendix.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper identifies the two key challenges associated with quantizing DiTs and provides simple but effective methodology for effective DiT quantization.\n- The description and illustration of the paper is clear and easy to follow. Detailed description of the reparamterization process of the quantization parameters are provide", "weaknesses": "- **Effectiveness of Salience Calibration:** The methodology sections 4.1 and 4.3 of the paper  resembles existing work on quantization [1]. The primary novel contribution of this paper is the 'Salience Calibration' technique, which addresses the specific challenge of activation timestep-variance in DiTs. It is critical to verify its importance. However, the evidence supporting the effectiveness of the Salience Calibration (SSC) is currently confined to Table 3, which presents results for ImageNet with 256x256 resolution, using a W4A8 configuration. The improvement observed with SSC is only moderate when compared to the CSB method. To more effectively emphasize the paper's unique contribution, additional evidence demonstrating the effectiveness of CSB would be beneficial.\"\n- **Overheads for Salience Calibration:** The 'Salience Calibration' technique introduces a timestep-wise correction coefficient to refine the activation $S(X(t))$ and weight $S(W)$ distributions, resulting in the adjusted binary matrices $B_{\\rho}^x$ and $B_{\\rho}^w$. This refinement implies that $B_{\\rho}^w$ may differ for each timestep t. However, the weights are reparameterized offline prior to quantization. Introducing a time-varying $B_{\\rho}^w$ could lead to the generation of distinct quantized integer (INT) weights for each timestep, which may complicate efforts to reduce the model's weight memory cost. This could involve either storing multiple sets of INT weights or repeatedly offloading weights for each timestep.\n\n[1] Xiao, Guangxuan, et al. \"Smoothquant: Accurate and efficient post-training quantization for large language models.\" International Conference on Machine Learning. PMLR, 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Response to Weakness 1: Effectiveness of SSC**\nWe clarify the innovation of CSB and provide additional evidence of the efficacy of SSC.\n\n**a. Innovation of CSB**\n\nWhile CSB shares the general concept of distribution re-scaling and re-parameterization with Smoothquant, it has unique innovation. Our CSB is designed for quantizing DiTs, whereas existing methods, such as Smoothquant, focus on quantizing LLMs. The quantization of DiTs presents unique challenges due to the extreme values in **both activations and weights**. In contrast, Smoothquant identifies outliers **only in activations** within LLMs and addresses this by migrating outliers from activations to weights. Fortunately, the extreme values do not occur in the same channels of activation and weight in DiTs. Our CSB is then derived from this complementarity to balance the salience between activation and weight, alleviating quantization errors for both.\n\n**b. More SSC ablation**\n\nTo validate the importance of SSC, we conduct additional experiment on the challenging ImageNet 512x512 with W4A8 using 50 timesteps:\n| Method | FID ↓ | sFID ↓ | IS ↑ | Precision↑|\n|---|---|---|---|---|\n|Baseline|56.94|74.55|39.16|0.4178|\n|+CSB|25.45|58.05|90.73|0.6390|\n|+CSB+SSC|19.71|52.27|118.32|0.7336|\n\nThe results demonstrate the significant impact of SSC, as it further enhances the benefits introduced by CSB.\n\n**c. Direct evidence of SSC's effect**\n\nPlease see **General Response 2: Experiment on direct evidence**.\n\n\n## **Response to Weakness 2: Overheads**\nWe clarify that $B_\\rho^X$ and $B_\\rho^W$ are not time-varying. As defined in Eq. (10), $s_\\rho(X^{(1:T)})$ aggregates activation saliences across all timesteps. Therefore, the resulting refined salience is a function of the number of timesteps $T$ and does not depend on any specific timestep $t$. We then formulate $B_\\rho^X$ and $B_\\rho^W$ based on $s_\\rho(X^{(1:T)})$ and $s(W)$, which are not time-varying.\n\nThus, we only perform reparameterization **once before quantization, without additional memory cost for model weights**.\n\n\n## **Response to Question 1: Recalibration**\nWhile SSC is orthogonal to solvers and models, it does depend on the number of timesteps ($T$ in Eq. (10)) due to its design of aggregating channel salience across T. Encouragingly, SSC is capable of generalizing across different T without recalibration. To verify its generalizability, we conduct two sets of experiments on ImageNet 256x256 with W8A8:\n\n**(I) Downsampling**: We calibrate and quantize the model using PTQ4DiT with 250 timesteps and evaluate this model using 100 and 50 timesteps.\n\n**(II) Upsampling**: We calibrate and quantize the model using PTQ4DiT with 50 timesteps and evaluate this model using 100 and 250 timesteps.\n\n|Calibration Timesteps|Evaluation Timesteps|FID↓|sFID↓|IS↑|Precision↑|\n|---|---|---|---|---|---|\n|250|250|4.63 |17.72|274.86|0.8299|\n|100|100|4.73|17.83|277.27|0.8270|\n|50|50|5.45|19.50|250.68|0.7882|\n|**250**|**100**|4.86|17.76|269.93|0.8221|\n|**250**|**50**|5.47|19.32|249.01|0.7899|\n|**50**|**100**|4.99|18.12|261.84|0.8239|\n|**50**|**250**|4.73|17.89| 266.04|0.8312|\n\nThe results show that both downsampling and upsampling does not significantly effect the performance, demonstrating the generalizability of SSC. We conjecture that this is due to two factors:\n\n**(I) Inherent generalizability of DiTs.** Previous studies [1,2,3] indicated that diffusion models possess strong ability to generalize across timesteps. Consequently, models trained with 250 timesteps can be directly applied to 100 and 50 timesteps with acceptable performance degradation. This interpolation ability is also evidenced by the performance of FP DiTs with 100 and 50 timesteps in Tables 1 and 2 of our paper (note that the original DiT work [4] only released the FP model for 250 timesteps).\n\n**(II) Effect of PTQ.** PTQ is formulated as a numerical problem and does not re-train the original models, which will not significantly affect the inherent interpolation ability if the quantization error remains low enough.\n\n[1] Diffusion Models Beat GANs on Image Synthesis. NIPS 2021.\n\n[2] Improved denoising diffusion probabilistic models. ICML 2021.\n\n[3] Post-training quantization on diffusion models. CVPR 2023.\n\n[4] Scalable diffusion models with transformers. CVPR 2023.\n\n## **Response to Question 2: Reparameterization**\nWe address 3 types of linear layers exhibiting significant channel salience: FC1, Projection1, and Projection2. We design reparameterization strategies for these layers. Specifically, FC1 follows the 'Post-adaLN' scenario. Thus, the Balancing Matrices can be integrated to the adaLN before Pointwise Feedforward. We further absorb them into MLPs regressing $\\gamma_2$ and $\\beta_2$, a process detailed in Appendix D.\n\nWe illustrate the integration strategies in Figure 7 and discuss them in the caption. We apologize for causing any confusion.\n\n## **Response to Question 3: Quantization details**\n**a. Are all layers in the DiTs quantized?**\n\nWe quantize all layers in the DiT models. The time embedding linear layers and attention matmuls are quantized.\n\n**b. Quantization optimization**\n\nFollowing Q-Diffusion, we involve gradient-based optimization for the quantization parameters, which has similar PTQ cost as that of Q-Diffusion (about 19 hours on a single NVIDIA A6000 GPU) since we do not introduce additional parameters.\n\n## **Response to Question 4: Explaining the statement**\nCorrelation salience refers to the degree of alignment between activation salience $s(X^{(t)})$ and weight salience $s(W)$. A low correlation indicates that channels with extreme values in activations are not the same as those in weights, and vice versa.\n\nLeveraging this complementarity, we propose SSC to prioritize timesteps with lower correlation between activation and weight saliences. This approach helps distribute the quantization impact more evenly across channels and timesteps, improving overall quantization performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y30BX5pa1z", "reviewer": "Reviewer_a8C3", "summary": "This paper presents PTQ4DiT, a quantization method designed for diffusion transformers. The method focuses on addressing quantization challenges due to extreme magnitudes in salient channels and the temporal variability of activations across multiple timesteps. To combat these issues, it incorporates techniques like Channel-wise Salience Balancing (CSB) and Spearmen’s ρ-guided Salience Calibration (SSC). These strategies help in redistributing magnitudes to minimize quantization errors and adapt dynamically across different timesteps, significantly enhancing the performance of quantized DiTs without re-training the original models.", "questions": "see weaknesses", "limitations": "This work reasonably discussed the limitations and future work", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method incorporates temporal information into the calibration process for salience balancing.\n- The experimental results are robust, covering a wide range of scenarios and effectively demonstrating the method's efficacy across different settings.", "weaknesses": "- The classifier-free guidance scales used for sampling are not specified, which could impact the reproducibility and evaluation of the model's performance.\n- Under the W4A8 setting, the method exhibits significant degradation in performance, suggesting limitations in its effectiveness at lower bit-widths.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Response to Weakness 1: Classifier-free guidance scales**\nThank you for pointing out this important concern. We set classifier-free guidance scales as 1.5 in all the experiments in our paper, following the origin DiT work [1].\n\n[1] Scalable diffusion models with transformers. In CVPR, 2023.\n\n\n## **Response to Weakness 2: W4A8 quantization**\nThanks for the valuable feedback. Here, we would like to emphasize the challenging W4A8 quantization and the contribution of our PTQ4DiT.\n\n**a. Challenge of W4A8 quantization**\n\nPrior studies [1, 2] have revealed that W4A8 quantization presents non-trivial challenges due to the severe information loss in low-bit model weights and have attempted to mitigate the degradation of W4A8 U-Net-based diffusion models. However, we observed that these methods result in substantial performance loss when applied to W4A8 Diffusion Transformers (DiTs), indicating the significant difficulty of low-bit quantization for DiT architectures.\n\n**b. Contribution of PTQ4DiT**\n\nThis is the first time a PTQ method has enabled high-quality generation at W4A8 bit-widths for DiTs, paving a promising path for future research in this field. Specifically, our work delves into DiT quantization and identifies two key challenges (Section 3). Then, we develop PTQ4DiT to address these challenges. Encouragingly, we find that PTQ4DiT consistently shows significant performance improvements over mainstream methods, as detailed in Tables 1 and 2 of our paper. Figure 5 further demonstrates that PTQ4DiT facilitates high-quality image generation despite the difficulty inherent in the lower bit-width of W4A8.\n\n[1] Q-Diffusion: Quantizing Diffusion Models. In ICCV, 2023.\n\n[2] PTQD: Accurate Post-Training Quantization for Diffusion Models. In NIPS, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# **General Response by Authors**
We express our gratitude to all the reviewers for dedicating their time and providing valuable comments. They acknowledged that our work is novel (N2VF, USPH), effective for DiT quantization (USPH, jrKh, a8C3) and well-written (USPH, jrKh). However, the reviewers also raised constructive concerns about the method's generality and statistical evidence to support our observations. To further enhance our paper, we added the corresponding experiments with analysis and presented them as follows.

## **General Response 1: Generality of PTQ4DiT**
To verify the generality of PTQ4DiT, we extend our experiment to include PixArt-α [1], an advanced Diffusion Transformer model facilitating text-to-image generation. Consistent with the literature convention [1, 2, 3], we adopt the CLIP score as our metric and perform text-to-image generation on the COCO validation dataset.
|Bit-width|Method|CLIP Score↑|
|---|---|---|
|FP|PixArt-α|31.5305|
|W8A8|PTQ4DiT|31.5368|
|W4A8|PTQ4DiT|31.5077|

The results demonstrate that PTQ4DiT significantly recovers the generation ability and delivers comparable performance to FP PixArt-α, suggesting the general efficacy of our method for quantizing Transformer-based diffusion models.

[1] PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis. ICLR 2024.

[2] Clipscore: A reference-free evaluation metric for image captioning. EMNLP 2021.

[3] Q-Diffusion: Quantizing Diffusion Models. ICCV 2023.


## **General Response 2: Experiments on direct evidence**
We analyze the effects of our CSB and SSC on channel salience, providing direct quantitative evidence to support their efficacy.

**Experiment design.** For an in-depth evaluation, we design a statistical experiment assessing the percentage of salient channels before and after CSB and SSC. To accurately identify the salient channels, we adopt a robust statistical outlier detector, Interquartile Range (IQR) [1]. This method identifies data points that lie significantly outside the middle 50% of the distribution as outliers. We perform IQR detection on maximal absolute values of channels to identify the salient channels.

**Experiment result.** We assess the percentage of salient channels detected by IQR. For a comprehensive assessment, we average the results over 100 random samples on ImageNet 256x256 generation and across all DiT layers:

| Model | Salient Activation Channel | Salient Weight Channel |
| --- | --- | --- |
| Original DiT | 5.37% | 6.10% |
| + CSB | 3.53% | 3.41% |
| **+ CSB + SSC (PTQ4DiT)** | **1.25%** | **1.32%** |

The results demonstrate a significant reduction in the percentage of salient channels when applying CSB and SSC, highlighting the efficacy of our proposed method in suppressing salient channels.

[1] Outlier detection: Methods, models, and classification. In ACM Computing Surveys (CSUR), 2020.

***For each reviewer's individual concerns, we would like to address them in the responses separately.***

Meta-review: All reviewers agree on the quality of this paper, I recommend acceptance
Decision: Accept (poster)