EfficientDM: Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Models
Authors: Yefei He
Jing Liu
Weijia Wu
Hong Zhou
Bohan Zhuang
Venue: ICLR 2024 spotlight (2024)
Keywords: Diffusion Models
Model Quantization
Model Compression
Efficient Models
Primary area: generative models

Abstract: Diffusion models have demonstrated remarkable capabilities in image synthesis and related generative tasks. Nevertheless, their practicality for low-latency real-world applications is constrained by substantial computational costs and latency issues. Quantization is a dominant way to compress and accelerate diffusion models, where post-training quantization (PTQ) and quantization-aware training (QAT) are two main approaches, each bearing its own properties. While PTQ exhibits efficiency in terms of both time and data usage, it may lead to diminished performance in low bit-width settings. On the other hand, QAT can help alleviate performance degradation but comes with substantial demands on computational and data resources. To capitalize on the advantages while avoiding their respective drawbacks, we introduce a data-free, quantization-aware and parameter-efficient fine-tuning framework for low-bit diffusion models, dubbed EfficientDM, to achieve QAT-level performance with PTQ-like efficiency. Specifically, we propose a quantization-aware variant of the low-rank adapter (QALoRA) that can be merged with model weights and jointly quantized to low bit-width. The fine-tuning process distills the denoising capabilities of the full-precision model into its quantized counterpart, eliminating the requirement for training data. To further enhance performance, we introduce scale-aware optimization to address ineffective learning of QALoRA due to variations in weight quantization scales across different layers. We also employ temporal learned step-size quantization to handle notable variations in activation distributions across denoising steps. Extensive experimental results demonstrate that our method significantly outperforms previous PTQ-based diffusion models while maintaining similar time and data efficiency. Specifically, there is only a marginal $0.05$ sFID increase when quantizing both weights and activations of LDM-4 to 4-bit on ImageNet $256\times256$. Compared to QAT-based methods, our EfficientDM also boasts a $16.2\times$ faster quantization speed with comparable generation quality, rendering it a compelling choice for practical applications.

Published as a conference paper at ICLR 2024
EFFICIENTDM: EFFICIENT QUANTIZATION-AWANE FINE- TUNNIG OF LOw-BIT DIFFUSION MODELS
Yefei He!
Jing Liu?
Weijia Wu!
Hong Zhou
Bohan Zhuang?*
'Zhejiang University, China ZZIP Lab, Monash University, Australia
ABSTRACT
Diffusion models have demonstrated remarkable capabilities in image synthe- sis and related generative tasks. Nevertheless, their practicality for low-latency real-world applications is constrained by substantial computational costs and la- tency issues. Quantization is a dominant way to compress and accelerate dif- fusion models, where post-training quantization (PTQ) and quantization-aware training (QAT) are two main approaches, each bearing its own properties. While PTQ exhibits efficiency in terms of both time and data usage, it may lead to di- minished performance in low bit-width settings. On the other hand, QAT can help alleviate performance degradation but comes with substantial demands on computational and data resources. To capitalize on the advantages while avoid- ing their respective drawbacks, we introduce a data-free, quantization-awee -awari and parameter-efficient fine-tuning framework for low-bit diffusion models, dubbed EfficientDM, to achieve QAT-level performance with PTQ-like efficiency. Specifi- cally, propose quantization-aww -ware variant of the low-rank adapter (QALoRA) that can be merged with model weights and jointly quantized to low bit-width. The fine-tuning process distills the denoising capabilities of the full-precision model into its quantized counterpart, eliminating the requirement for training data. To further enhance performance, address ineffective learning of QALoRA due to variations in weight quantiza- tion scales across different layers. We also employ temporal learned step-size quantization to handle notable variations in activation distributions across denois- ing steps. Extensive experimental results demonstrate that our method signifi- cantly outperforms previous PTQ-based diffusion models while maintaining sim- ilar time and data efficiency. Specifically, there only a marginal 0.05 sFID increase when quantizing both weights and activations of LDM-4 to 4-bit on ImageNet 256 x 256. Compared to QAT-based methods, our EfficienDDM also boasts 16.2x faster quantization speed with comparable generation quality, rendering a compelling choice for practical applications. Code is available at https//gghhbbboooom///ssBiiieete
introduce scale-aware LoRA optimization
INTRODUCTION
Diffusion models (DM) (Ho et al., 2022b; Dhariwal Nichol, 2021; Rombach al., 2022a; Ho et al., 2022a) have demonstrated remarkable capabilities in image generation and related tasks. Nonetheless, the iterative denoising process and the substantial computational overhead of the de- noising model limit the efficiency of DM-based image generation. To expedite the image generation process, numerous methods (Bao et al., 2022: Song et al., 2021; Liu et al., 2022; Lu et al., 2022) have been explored to reduce the number of denoising iterations, effeetively reducing the previously quired thousands of iterations to mere dozens. However, the significant volume of parameters within the denoising model still demands a substantial computational burden for each denoising step, sulting in considerable latency, hindering the practical application of DM in real-world settings with latency and computational resource constraints.
"Corresoonding author. Email: zhouhong wwwtzj...dd...omm bohan www..aagggngmm..ccomm

---

Published as a conference paper at ICLR 2024
Model quantization, which compresses weights and activations from 32-bit floating-por point values into lower-bit fixed-point formats, alleviating both memory and computational burdens. This ef- fect can be increasingly pronounced as the bit-width decreases. For instance, leveraging Nvidia's CUTLASS (Kerr et al., 2017) implementation, an 8-hit model's inference speed can be 2.03x faster than that of full-precision (FP) model, and the acceleration ratio reaches 3.34x for a 4-bit model. Therefore, it possesses substantial potential for significantly compressing and accelerating diffu- sion models, making them highly suitable for deployment on resource-constrained devices such as mobile phones.
Nevertheless, the challenges associated with low-bit quantization for diffusion models have not re- ceived adequate attention. Typically, model quantization can be executed through two predominant approaches: post-training quantization (PTQ) and training-aware quantization (QAT). PTQ cali- brates the quantization parameters with small calibration dataset, which is time- and data-efficient. However, they introduce substantial quantization errors low bit-width. As illustrated in Figure 1, when quantizing both weights and activations to 4-bit with the PTQ method (He et al., 2023a), diffu- sion models fail to maintain their denoising capabilittes. In contrast, QAT methods (Krishnamoorthi, 2018; Esser al., 2019)
recover performance losses lower bit-width by fine-tuning the whole model. However, this approach requires significantly more time and computing resources compared to PTQ method (He al., 2023a), evidenced by 2.6x increase in GPU memory consumption (31.4GB vs. 11.7GB) and a 18.9x longer execution time (54.5 GPU hours vs. 2.88 GPU hours) when fine-tuning LDM-4 (Rombach al., 2022a) on ImageNet 256 256. Moreover, in some cases, it may be challenging or even impossible to obtain the original training dataset due to privacy or copyright concerns.
In this paper, we introduce a data-free and parameter- efficient fine-tuning framework for low-bit diffusion mod- els. denoted as EfficientDM, which demonstrates the ca- pability to achieve QAT-level performance while uphold- ing PTQ-level efficiency during fine-tuning in terms of data and time. The foundation of our approach lies in a quantization-awarc form of the low-rank adapter, as de- picted in Figure 2b. This variant enables the joint quanti- zation of LoRA weights with model weights, thereby ob- viating additional storage and calculations. Compared to previous QAT method (So et al., 2023), our fine-tuning process is executed in a data-free manner, accomplished by minimizing the mean squared error (MSE) between the estimated noise of full-precision denoising model and its quantized counterpart, as illustrated in Figure 2c, thus eliminating the need for the original training dataset. Due to quantization, the relationship between full-precision LoRA weights and quantized updated model weights be- comes step-like with step size, i.e. the quantization scale parameter. Effective updates are mostly observed in lay- ers with small scales, while other layers with larger scales do not benefit. To address this, we introduce scale-aware LoRA optimization that adaptively adjusts the gradient scales of LoRA weights in different layers ensure an ef- fective optimization. Furthermore, we extend the learned step size quantization (LSQ) method (Esser al., 2019) into the denoising temporal domain for activations, effectively mitigating quantization errors due to varying activation distributions across time steps.
PTQ
100
FID
Ours
QAT
0.5
50
Time Cost (h)
Figure 1: An overview of the efficiency-vs-quaiiy tradeoff across various quantization approachs. Data is collected on LDM-8 (Rombach et al., 2022a) with 4-bit weights and activations on LSUN-Churches. The GPU memory consumption is visual- ized by circle size.
In summary, our contributions are as follows:
We introduce EfficientDM, an efficient fine-tuning framework for low-bit diffusion models which achieve QAT performance with the efficiency of PTQ. The framework is rooted in the quantization-aware form of low-rank adapters (QALORA) and distills the denoising capabilities of full-precision models into their quantized counterparts.
We propose scale-aware LoRA optimization to alleviate the ineffective learning of QALoRA re- sulting from substantial variations in weight quantization scales across different layers. We also
2

---

Published as a conference paper at ICLR 2024
Weights
LoRA
Activations
Weights
LoRA Activations
Q-Model
Qu(
MSF
(w(
Activations
FP-Model
Weights
Eq. (2)
FP Matrix Multiplication
INT Matrix Multiplication
Quantized Data/Model
Backward
IP Data/Moee!
Output
Grad
Output
Grad
(a) QloRA layer
(b) QALORA layer
EfficientDM fine-tuning framework
Figure 2: An overview of the proposed EfficientDM fine-tuning framework. Here, Sw and 82 repre- sent the learnable quantization scales for weights and activations, respectively. Compared to QLoRA layer, both updated weights and activations in our QALoRA A are quantized to enable efficient bit- wise operations during inference. Eine-tuning is performed by minimizing the mean squared error between the estimated noises of FP and quantized models.
introduce TALSQ. an extension of activation learned step size quantization within the temporal domain to tackle the variation activation distributions across denoising steps.
Extensive experiments CIFAR-10, LSUN and ImageNet demonstrate that our EfficientDM reaches a new state-of-the-art performance for low-bit quantization of diffusion models.
2 RELATED WORK
Model quantization. Quantization is a widely employed technique for compressing and accelerat- ing neural networks. Depending whether fine-tuning of the model is necessitated, this method can be categorized into two approaches: post-training quantization (PTQ) (Nagel et al., 2020; Li al., 2021; Wei et al., 2022; Lin al., 2022) and quantization-aware training (QAT) (Esser al., 2019; Gong et al., 2019; Nagel et al., 2022; Louizos et al., 2019; Jacob et al., 2018; Zhuang et al., 2018; He al., 2023b). PTQ does involve fine-tuning the model's weights and only requires small dataset to calibrate the quantization parameters. This approach is fast and data-efficient but may result in suboptimal performance especially when employing low bit-widths. Recent advance- ments in reconstruction-bant based PTQ methods (Li et al., 2021; Wei et al., 2022) utilize second-order emor analysis and gradient descent algorithms to optimize quantization parameters. Such methods demonstrate robust performance, which can perform well even at 4-bit on image classification tasks. On the other hand, QAT entails fine-tuning the model weights to achieve low-bit quantization while minimizing performance degradation. Nevertheless, the substantial data prerequisites and high com- putational overhead render QAT much slower than PTQ. For instance, when applied to ResNet- 18 model for the ImageNet classification task, it is reported to be 240 slower than the PTQ method (Li et al., 2021). This hindrance poses challenges to its widespread adoption, particularly for models with massive parameters.
Given the demand to compress diffusion models and accelerate their sampling, quantization is con- sidered an effective approach. Currently, most quantization work on diffusion models has focused on PTQ (Shang et al., 2023; Li et al., 2023; He et al., 2023a). Since diffusion models can gener- samples from random Gaussian noise, these methods conducted in data-free manner, with the calibration sets collected from the denoising process. For example, PTQ4DM (Shang et al., 2023) and Q-Diffusion (Li al., 2023) apply reconstruction-basse PTQ methods to diffusion mod- els. PTQD (He et al., 2023a) further decomposes quantization noise and fuses it with diffusion noise. Nevertheless, these methods experience significant performance degradation at 4-bit or lower bit-widths. Recent work TDQ (So et al., 2023) fine-tunes diffusion models in a QAT manner and employs additional MLP modules to estimate quantization scales for each step. However, requires the original dataset and training the entire quantized model with these extra MLP modules, incurring significant costs. In contrast, our approach introduces only a few trainable quantization scales per layer and can attain QAT-level performance with PTQ-level efficiency.
Parameter-efficient fine-tuning, Parameter Crfumeter-Effcieen Fine- Tine-Tuning (PEFT) has emerged as a potent alternative to full fine-tuning, which tunes small subset of parameters while keeping the vast ma-
3

---

Published as a conference paper at ICLR 2024
jority frozen, ease storage burden of large pretrained models. Noteworthy approaches within this realm include Prefix Tuning (Li & Liang. 2021; Liu et al., 2021), Prompt Tuning (Lester et al., 2021; Wang et al., 2022), and Low-rank adapters (LoRA) (Hu et al., 2022; Zhang et al., 2022), which have found extensive utility within large language models (LLMMs).
Among these PEFT methods, LoRA (Hu et al., 2022) keeps the pre-trained weights fixed and learns a set of low-rank decomposition matrices as updates to the weights. This method substantially reduces the number of trainable parameters, and does not impose additional computational overhead due to re-parameterization. QLoRA A (Dettmers et al., 2023) combines this technique with quantized LLMs. further curtailing memory requirements while mitigating performance degradation brought by quantization. However, its full-precision LoRA weights cannot be integrated with quantized model weights, leading to extra storage and computational costs for deployment. In contrast, we propose to merge LoRA weights with full-precision model weights and jointly quantize them to the target bit-width, enabling their seamless integration without incurring any additional overhead for deployment. Additionally, we identify and address the issue of ineffective learning of LoRA weights stemming from the variations in weight quantization scales, further enhancing the performance of low-bit diffusion models.
BACKGROUND
3.1 DIFFUSION MODELS
Diffusion models (Song et al., 2021; Ho et al., 2020) are a class of generative models that iteratively introduce noise to real data X, through a forward process and generate high-quality samples via re- verse denoising process. Generally, the forward process is a Markov chain, which can be formulated as:
((x/|xx-1) = Nxxs; /I - -,,,,x,,,,,
(I)
where 3g governs the magnitude of the introduced noise. The reverse process adheres to a similar form, where diffusion models approximate the distribution of ((x))1 via variational inference by learning a Gaussian distribution:
Po px 1x (), Ex(,,
(2)
where 14 and Eg two neural networks. When these neural networks are subjected to quan- tization, the resulting estimated statistics of the Gaussian distribution can be inaccurate, thereby compromising the efficacy of the denoising process and the overall quality of the generated sam- ples. Our study is focused on introducing an efficient and powerful fine-tuning framework to narrow or even eliminate the performance gap between low-bit bit diffusion models and their full- precisonn counterparts.
3.2 MODEL QUANTIZATION
Model quantization represents model parameters and activations with low- www-precision integer values reduce memory footprint and accelerate the inference. Given floating-poin vector x, it can be uniformly quantized as follows:
x= (x, = clip(( 1,
Here, 1-1 is the round operation, is the trainable quantization scale, and and u are the lower and upper bound of quantization thresholds, which are determined by the target bit-width. To overcome the non-differentiabiiityy of the round operation during QAT, straight-through estimator (STE) is widely adopted to estimate the gradient:
Oc
OL
1iss
0x
0%
where 1 represents an indicator function that takes on the value the element is in the set A, and the value O vice versa.
4

---

Published as a conference paper at ICLR 2024
4 EFFICIENT QUANTIZATION-NWARE FINE-TUNING OF DIFFUSION MODELS
In this section, we propose efficient fine-tuning framework for diffusion models with both weights and activations quantized, possessing the efficiency of PTQ and the accuracy of QAT. The proposed framework, dubbed EfficienDMM, is depicted in Figure 2. It consists of quantization-aww rank adapter and a noise distillation strategy, delivering parameter-effiiient and data-free fine-tuning. Moreover, is also equipped with scale-aware LoRA optimization and a temporal-awar quantizer for improved performance. We elaborate each design as follows.
low-
Quantization-aware low-rank adapter. Low-rank adapter (LoRA) fine-tuning constrains the up- date of the model parameters to possess a low intrinsic rank, denoted as r. Given pretrained linear module Y x WWo, where E TRxxein and Wo
XCout, with representing the batch size, Cin and Cout representing the number of input and output channels, respectively, LoRA fixes the original weights Wo and introduces updates as follows:
Y = XW + XBA,
(5)
where B E X and A Rexe are the learnable two low-rank matrices with r min( (Ciz, Cout).
Nevertheless, this approach incurs limitations when both weights and activations are quantized, denoted by Wo and X, respectively. In this case, the inner product between Wo and X can be efficiently implemented with bit-wise operations, whereas the operations involving BA and X are computationally expensive during inference as BA is full-precision and has the same size as Wo-
To address this, we propose Quantization-awar Low-rank Adapter (QALoRA), where the LoRA weights are first merged with FP model weights and then jointly quantized to the target bit-width, as depicted Figure 2b. Formally, the QALoRA is defined as follows:
Qu (X, 82) 2r (Wo + BA, Sw)) = XW,
where Sw denotes the channel-wise quantization scale for weights and s is the layer-wise quanti- zation scale for activations. After the fine-tuning process, only quantized updated model weights W need to be saved. Notably, our approach can be readily integrated with QLoRA (Dettmers et al., 2023) by substituting Wo with Wo to further reduce memory footprint.
Data-free fine-tuning for diffusion models. Diffusion models require access to large and diverse datasets for effective training. Obtaining such datasets be challenging due their sheer size, privacy concerns, or copyright restrictions. To alleviate the dependency on the original dataset, we propose data-free fine-tuming approach that distills the denoising capabilities of full-precision model into its quantized counterpart. Specifically, we input the same noise x, to both FP and quan- tized denoising models at denoising step í and minimize the mean squared error (MSE) between their denoising results:
C, = 1| (x,, 1) - (x,( 1)11
where [x, t) and p t) denote the denoising results of the FP and quantized models for the denoising step L, respectively. The input data x, is obtained by denoising random Gaussian noise XT ~ (N 1) with FP model iteratively for T - L steps, as illustrated in Figure 2c.
To facilitate the training of QALoRA, we further address the following technical challenges:
Variation of weight quantization scales across layers. As demonstrated in Eq. (6), due to the quantization process, the relationship between full-precision LoRA weights BA and quantized up- dated weights W follows a step function, where the step size exactly equal to the quantization scale s,.. and W, serves as fixed offset. This relationship is visually presented in Figure 3a. Con- sequently, while the full-precision LoRA weights are continuously optimized during the fine-tuning process, they need to be large enough to update the quantized model weights, otherwise they will be diminished by the round operation within the quantization process, as referred to Eq. (3) and (6). As shown in Figure 3b and "Scale-agnostic training" in Figure 3c, fine-tuning the LoRA weights with limited iterations only yields effective updates for a few layers with relatively small scales. For other layers, their quantization scales are too substantial for LoRA weights to take effect due to the round operation. Alternative approaches, such as directly amplifying the learning rate, may impede the convergence process.
5

---

Published as a conference paper at ICLR 2024
W= Q. (W. + BA, s,)
stop size
Mean Scale Valuee
0.015
Max: 0.017514
0.000337
Mean Absolute Vieight Update
0.125
0.150
Sow ng nontott trawiing
Scarc unar Marnig
0.025
0.010
0.100
0.000
W
0.075
-0.025
0.005
0.050
0.050
-0.05
0.000
100
150 Layer idb
200
250
0.000
50
100
150 idx
200
(b)
(c)
Figure 3: The motivation and effect of scale-aware LoRA optimization. Data is collected from
the 4-bit LDM-4 model. (a): Due to step-like relationship between BA and W, BA needs to be large enough to update model weights. Data is collected from the first channel in the 12th layer. (b): Significant disparity in weight quantization scales across layers. (c): Mean absolute value of quantized weight updates (W-Wo) )) for each layer. Most of them are zero under scalc-agnostic training. indicating full- cull-precision LoRA weights are too small update quantized model weights. The proposed scale-aware LoRA optimization facilitates a more equitable distribution of quantized weight updates across layers.
To facilitate the optimization of LoRA weights, consider the ratio of VBA
R
Sw
should be roughly consistent in each layer, where 5 represents the averaged weight quantization scale across channels. This can be achieved by simply multiplying the gradient of LoRA weights by this average weight quantization scale during the back-propagation phase. As shown in Fig- ure 3c, optimizing LoRA in the scale-aware approach enables effective training across the majority of layers.
Variation of activation distribution across steps. Previous research on diffusion models (Shang al., 2023; Li et al., 2023; So et al., 2023) has identified pronounced variability in activation distri- butions at different time steps, which is also presented in Appendix C. This variability poses substan- tial challenges the quantization of diffusion models. Notably, existing methods proposed to ad- dress this issue aim to either find a set of quantization parameters applicable to all time steps (Shang al., 2023) or employ additional MLP module to estimate quantization parameters for each indi- vidual time step and layer (So et al., 2023), which can be either suboptimal or cumbersome.
Inspired by LSQ (Esser et al., 2019), a technique where quantization scales are optimized along- side other trainable parameters through the gradient descent algorithm, we allocate temporal-aware quantization scales for activations and optimize them individually for each step, which we refer to as Temporal Activation LSQ (TALSQ):
S, = { 82 ......, T-1)
where T the number denoising steps for the fine-tuning. It is noteworthy that recent advance- ments efficient samplers have significantly reduced the number of sampling steps. Therefore, TALSQ introduces only a few trainable parameters for a single layer, which is negligible even com- pared LoRA weights (which generally have thousands of parameters per layer). After fine-tuning, we interpolate the learned temporal quantization scales to deal with the gap of sampling steps be- tween fine-tuning and inference.
5 EXPERIMENTS
5.1 IMPLEMENTATION DETAILS
Models and metrics. To verify the effectiveness of the proposed method, we evaluate it with two widely adopted network structures: DDIM (Song et al., 2021) and LDM (Rombach et al., 2022b).
For experiments with DDIM, we evaluate it on CIFAR-10 dataset (Krizhevsky & Hinton, 2009). Experiments with LDM are conducted two standard benchmarks: ImageNet (Deng et al., 2009)
and LSUN (Yu et al., 2015). The performance diffusion models is evaluated with Inception Score (IS), Fréchet Inception Distance (FID) (Heusel al., 2017). For experiments ImageNet datasets,

---

Published as a conference paper at ICLR 2024
also report sFID (Salimans et al., 2016) and Precision for reference. Results are obtained by sampling 50, 000 images and evaluating them with ADM's TensorFlow evaluation suite (Dhariwal & Nichol, 2021). The rank of the adapter is set to 32 in experiments. For the proposed Ef- ficientDM framework, we fine-tune LoRA weights and quantization parameters for 16K iterations with batchsize of 4 LDM models and 64 on DDIM models, respectively. The number of de- noising steps for the fine-tuning is set to 100. We employ Adam (Kingma & Ba, 2014) optimizers with learning rate of 5e
Quantization settings. The notion 'WzAy' is employed to represent the bit-widths of weights "W" and activations 'A'. We use channel-wise quantization for weights and layer-wise quantization for activations, as is a common practice. The input embedding and output layers in the model employ fixed W8A8 quantization, whereas other convolutional and fully-connected layers are quantized to the target bit-width with a QALoRA module.
5.2
MAIN RESULTS
5.2.1
EVALUATION UNCONDITIONAL GENERATION
We begin our evaluation by performing unconditional generation on CIFAR-10 dataset and compare results against well-established quantization methods, including both PTQ-based (Shang al., 2023; Li et al., 2023) and QAT-based (Esser et al., 2019; So et al., 2023) contenders. The results are presented in Table Recent PTQ-based method (Li et al., 2023) performs well under W8A8 pre- cision but exhibits performance degradation under W4A8 precision. Recent QAT-based method (So al., 2023) surpasses all PTQ-based approaches under W4A8 precision but necessitates access to the original training dataset and incurs a substantial 17.8x increase in training time. In contrast, our proposed EfficientDM demonstrates superior performance even compared to previous QAT methods under both W8A8 and W4A8 precisions, while incurring significantly lower time costs. Notably, under W4A8 precision, we achieve an FID of 3.80, outperforming QAT-based method TDQ (So et al., 2023) by 0.33.
Additional experimental results on LSUN dataset can found Appendix A.
Table 1: Performance comparisons of quantized diffusion models on CIFAR-10 32 x 32. Results are obtained by DDIM sampler with 100 steps.
Bit-width (WI)
GPU Time (hours)
Model Size (MB)
Method
Training
1St
FID,
FP
32/32
50K
136.4
9.12
4.14
PTQ4DM
878
0
0.95
34,26
9.31
14.18
O-Diffusion
8/8
0
0.95
34.26
9.48
3.75
LSQ
8/8
50K
13.89
34,26
9.62
3.87
TDQ
8/8
50K
16.99
34.30
9.58
3.77
Ours
8/8
0
0.97
34.30
9.38
3.75
PTQ4DM
4/8
0
0.95
17.22
9.31
10.12
Q-Diffusion
4/8
0
0.95
17.22
9.12
4.93
LSQ
4/8
50K
13.89
17.22
9.38
4,53
TDQ
4/8
50K
17.26
9.59
Ours
4/8
0.97
17.26
9.41
3.80
5.2.2 EVALUATION OF CONDITIONAT. GENERATION ON IMAGENET 256
256
We also evaluate the performance of our method over LDM-4 on large-scale ImageNet 256 x 256 dataset for class-conditional image generation, as presented Table 2. Experiments are conducted with three distinct samplers. including DDIM (Song et al., 2021), PLMS (Liu et al., 2022) and DPM- Solver (Lu et al., 2022). Importantly, we offer a comprehensive evaluation based on four metries (IS, FID, sFID, and Precision), which should be collectively considered to ensure robust assessment of image quality. W8A8 quantization, recent PTQ-based methods (Li et al., 2023; He et al., 2023a) exhibit performance levels with negligible degradation. However, when the bit-width drops to W4A8, performance degradation becomes evident for them. For instance, when employing the DDIM sampler, Q-Diffusion experiences decline of 27.8 in IS and decrease of 2.6% in precision. The introduction of EfficientDM yields a significant recovery in performance, with a merely 2.4 IS decrease compared with FP method. As the bit-width decreased, the performance enhancements

---

Published as a conference paper at ICLR 2024
of EfficientDM became more pronouneed. In the case of W4A4 quantization, none of the diffusion models quantized with PTQ-based methods able to denoise images, regardless of the sampler used. In contrast, models quantized by our approach manage to maintain exceptionally low FID of 6.17 and a sFID of 7.75 for the DDIM sampler, respectively. Leveraging the capabilities of MMf push the quantization of diffusion model weights to 2-bit for the first time, resulting in marginal increase of less than 1 in the sFID metric when using the DDIM sampler. Moreover, we present the real-time speed up of EfficientDM over LDM-4 in Appendix B.
Table 2: Performance comparisons of fully-quantized LDM-4 models on ImageNet denotes failed image generation.
'N/A'
Bit-width (W/A)
Precision
Sampler
Method
1S1
FID,
sFID)
(%)
HP
32/32
364.73
11.28
7.70
93.66
O-Diffusion
878
350.93
10.60
9.29
92.46
PTQD
8/8
359.78
10.05
9.01
93.00
Ours
362.34
11.38
8.04
93.77
Difflusioon
478
336.80
9.29
9.29
91.06
PTQD
4/8
344.72
8.74
7.98
91.69
LDM-4 DDIM (20 stops)
Ours
353.83
9.93
7.34
93.10
Q-iffuusion
474
N/A
N/A
N/A
N/A
PTQD
444
N/A
N/A
N/A
N/A
Ours
250.90
6.17
7.75
86.02
Q Diffusion
278
49.08
43.36
17.15
43.18
FTQD
2/8
53.36
39.37
15.14
45.89
Ours
175.03
7.60
8.12
78.90
FP
32/32
379.19
11.71
6.08
94,22
Q Diffluuion
8/8
373.49
11.25
5.75
93.84
FTQD
374.50
11.05
5.45
94,00
Ours
8/8
376.06
11.78
6.21
94.27
Diffusion
4/8
358.13
9.67
S.74
92.71
PTQD
357.66
9.24
5.42
92.46
LDM-4 PLMS (20 steps)
Ours
4/8
367.22
10.26
5.11
93.44
Diffusion
474
N/A
N/A
N/A
N/A
PTQD
N/A
Ours
4/4
185.18
10.95
11.65
75.61
Q Diiffusion
2/8
56.73
34.71
12.00
49.
PTQD
58.08
31.87
10.32
50.73
Ours
2/8
182.63
6.51
6.99
79.43
FP
32/32
373.12
11.44
6.83
93.67
Q-Diflusion
365.64
T0.78
6.15
92.39
PTQD
8/8
368.04
10.55
6.19
92.99
Ours
8/8
370.22
11.21
6.83
93.50
O-Diflusion
351.00
9.36
6.36
91.30
PTQD
4/8
354.94
8,88
6.73
92.03
LDM-4 DPM-Solver (20 steps)
Ours
4/8
359.16
9.82
6.92
92.67
Dillusion
N/A
N/A
PTQD
4/4
N/A
N/A
N/A
N/A
Ours
4/4
223.40
7.54
9.47
80.06
Q-Diflusion
50.12
39.08
13.75
44.67
PTQD
2/8
51.51
38.32
12.90
46.48
Ours
2/8
165.76
8.55
9.71
76.76
5.3 ABLATION STUDY
To assess the efficacy of each proposed component, we conduct comprehensive ablation study on the ImageNet 256 x 256 dataset, employing the LDM-4 model with a DDIM sampler, as presented in Table We initiate the evaluation with baseline PTQD (He al., 2023a), which is rooted in the reconstruction- based PTQ method (Li et al., 2021). However, it failed to denoise images when operating under W4A4 bit-width, yielding an exceedingly high FID score of 259.73. This result underscores the inadequacy of the PTQ method in this particular low bit-width. In sharp contrast, our proposed efficient fine-tuning method QALoRA showcases significant performance improvement with FID of 11.42, without imposing additional time costs. By incorporating scale- aware LoRA optimization, we achieve an FID reduction of 1.87 and a sFID reduction of 10.95, demonstrating that the LoRA weights are effectively trained to update the quantized weights. By further introducing TALSQ that learns quantization parameters for each denoising step, our method achieves remarkable sFID of 7.75, putting it in contention with even full-precision models.
Moreover, we conduct a comparative efficiency analysis of PTQ, QAT and our approach across data, GPU memory, and time consumption, as demonstrated in Table 4. The PTQ method (He al.,
8

---

Published as a conference paper at ICLR 2024
Table 3: The effect of different components proposed in the paper. The experiment is conducted over LDM-4 model on ImageNet 256 x 256.
Method
Bit-width (W/A)
Time Costs (GPU hours)
1St
FIDL
sFID1
Precision
(%)
FP
32/32
364.73
11.28
7.71
93.66
PTQD
4/4
2.88
2.16
259.73
329.01
0.00
QALoRA
4/4
2,60
156.15
11.42
24.18
78.36
+scale-aware LoRA optimization
4/4
172.96
9.55
13.23
81.87
+TALSQ
4/4
3.05
250.90
6.17
7.75
86.02
2023a) emerges as the most efficient in terms of data, time and memory consumption. However, cannot maintain denoising capabilities at W4A4 precision. On the other hand, QAT method (Park al., 2022), despite achieving lower FID, necessitates the original training data and incurs a 2.2x increase in memory resources (16950 MB vs. 7538 MB), and 16.8x increase in time consumption (18.5 GPU hours vs. 1.1 GPU hours) during the quantization of the LDM-8 model on the LSUN- Churches dataset. Comparatively, without the need for training data and incurs less memory and time consumption compared to the QAT method. Specifically, when quantizing DDIM models CIFAR-10 dataset, our method attains a FID that is comparable to that of the QAT method (10.48 vs. 7.30), while enjoying similar time efficiency the PTQ method (0.97 GPU hours vs. 0.95 GPU hours), rendering it practical choice for wealdworld applications. Additional ablation experiments can be found in Appendix D.
proposed approach offers compelling alternative. It operates
Table 4: Efficiency comparisons of various quantization methods across training data, GPU memory and time. 'OOM' denotes out-of-memory on RTX3090 GPU. We employ DDIM models for CIFAR- 10 dataset and LDM models for other datasets. Both weights and activations of models are quantized to 4-bit.
GPU Memory
GPU Time (hours)
Method
Dataset
Training data
FID
(MB)
CIFAR-10 LSUN-Churches
4334
0.95
181.05
PTQ (He et al., 2023a)
7538
1.10
321.90
ImageNet 256
2,88
259.73
CIFAR-10
50K
9974
13.89
7.30
QAT (Esser et al. 2019)
LSUN-Churches ImageNet 256 x 256
126K
16950
18.50
9.08
1.2M
OOM
CIFAR 10 LSUN-Churches
9554
0.97
10.48
Ours
1.14
ImageNet 256 x 256
19842
3.05
6.17
CONCLUSION
In this paper, we have proposed EfficientDM, an efficient data-free fine-tuning framework for low- bit diffusion models. To commence, we have introduced a quantization-awan variant low-rank adapters, which can be jointly quantized with model weights, enabling both efficient fine-tuning and low-bit inference. To mitigate the reliance on original datasets, we have employed a distilling frame- work that transfers the noise estimation capabilities of full-precision model to its quantized coun- terpart. To address ineffective learning arising from variations in weight quantization parameters across layers, we have introduced scale-aware LoRA optimization, which adaptively adjusts the gra- dient scales of LoRA weights in different layers. Moreover, we have introduced TALSQ to address the activation distribution variations across steps. Our extensive experiments have demonstrated the superiority of EfficientDM over previous post-training quantized diffusion models. Notably, even when quantizing both weights and activations of LDM-4 4-bit on ImageNet 256 256, where previous methods failed to denoise, EfficientDM exhibited only marginal 0.05 increase in sFID. Moreover, our method has demonstrated superior efficiency compared to QAT-hased approaches with minor performance gap, making it practical choice for real-world applications.
Limitations and future work. While EfficientDM can attain QAT-level performance using PTQ- level data and time efficiency, it employs gradient descent algorithm for optimizing QALoRA parameters. Compared to PTQ-based methods, this places higher demands on GPU memory, partic- ularly with diffusion models featuring massive parameters. To address this, memory-effiiien opti- mization methods can be integrated. Moreover, efficient diffusion models for tasks such as video or 3D generation still under explored.

---

Published as a conference paper at ICLR 2024
Acknowledgement This work was supported by National Key Research and Development Program of China (2022FF33602001,
REFERENCES
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: an analytic estimate of the optimal reverse variance diffusion probabilistic models. ICIR, 2022.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 2013.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. hierarchical image database. In CVPR, 2009.
Imagenet: A large-scale
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. In NeurlPS, 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In NeurlPS, 2021.
Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmen- dra Modha. Learned step size quantization. ICLR, 2019.
Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differentiable soft quantization: Bridging full- full-precision and low-bit neural networks. In ICCV, 2019.
Yefei He, Luping Liu, Jing Liu, Weijia Wu, Hong Zhou, and Bohan Zhuang. Ptqd: Accurate post- training quantization for diffusion models. arXiv preprint 2.Xi::3305.10657: 2023a.
Yefei He, Zhenyu Lou, Luoming Zhang, Weijia Wu, Bohan Zhuang, and Hong Zhou. Bivit: Ex- tremely compressed binary vision transformer. In ICCV, 2023b.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In NeurlPS, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
Jonathan HO, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02300: 2022a.
Jonathan Ho, Tim Salimans, Alexey A. Gritsenko, William Chan, Mohammad Norouzi, and David J.
Fleet. Video diffusion models. In NeurIPS, 2022b.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICIR, 2022.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient antger-arithmmetic-onl inference. In CVPR. 2018.
Andrew Kerr, Duane Merrill, Julien Demouth, and John Tran. Cutlass: Fast linear algebra in cuda
c++. NVIDIA Developer Blog. 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14Xiv:1412.6988) 2014.
Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint arXiv: 2018
10

---

Published as a conference paper at ICLR 2024
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech Report, 2009.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficien prompt tuning. arXiv preprint arXiv:2104. 08691, 2021.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint XX1::2001.00190 2021.
Xiuyu Li, Long Lian, Yijiang Liu, Huanrui Yang, Zhen Dong, Daniel Kang. Shanghang Zhang, and
Kurt Keutzer. Q-diffusion: Quantizing diffusion models. In ICCV, 2023.
Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. BRECQ: pushing the limit of post-training quantization by block reconstruction. In ICLR, 2021.
Yang Lin, Tianyu Zhang, Peiqin Sun, Zheng Li, and Shuchang Zhou. Fq-vit: Post-training quanti- zation for fully quantized vision transformer. In IJCAI, 2022.
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models manifolds. In ICLR, 2022.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P- tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602, 2021.
Christos Louizos, Matthias Reisser, Tijmen Blankevoort, Efstratios Gavves, and Max Welling. Re- laxed quantization for discretized neural networks. In ICLR, 2019.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In NeurlPS, 2022.
Markus Nagel, Rana Ali Amjad, Mart van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. In JCML, 2020.
Markus Nagel, Marios Fournarakis, Yelysei Bondarenko, and Tijmen Blankevoort. Overcoming oscillations in quantization-awarr training. In ICML, 2022.
Sein Park, Junhyuk So, Juncheol Shin, and Eunhyeok Park. Nipq: Noise injection pseudo quantiza- tion for automated dnn optimization. arXiv preprint arXiv:2206.00820, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022a.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022b.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NeurIPS, 2016.
Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan. Post-training quantization on diffusion models. In CVPR, 2023.
Junhyuk So, Jungwon Lee, Dachyun Ahn, Hyungjun Kim, and Eunhyeok Park. Temporal dynamic quantization for diffusion models. arXiv preprint arXiv:2306.023/6 2023.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
Changyuan Wang. Ziwei Wang, Xiuwei Xu. Yansong Tang, Jie Zhou, and Jiwen Lu. Towards accurate data-free quantization for diffusion models. arXiv preprint arXiv:2305. 18723, 2023.
Zhen Wang, Rameswar Panda, Leonid Karlinsky, Rogerio Feris, Huan Sun, and Yoon Kim. Multi- task prompt tuning enables parameter-efficient transfer learning. In ICLR, 2022.

---

Published as a conference paper at ICLR 2024
Xiuying Wei, Ruihao Gong, Yuhang Li, Xianglong Liu, and Fengwei Yu. Qdrop: Randomly drop- ping quantization for extremely low-bit post pestinraining quantization. ICLR, 2022.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint urXiv:/506.03360 2015.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficien fine-tuning. ICLR, 2022.
Bohan Zhuang, Chunhua Shen, Mingkui Tan, Lingqiao Liu, and Ian Reid. Towards effective low- bitwidth convolutional neural networks. In CVPR, 2018.
12

---

Published as a conference paper at ICLR 2024
Appendix
A EVALUATION OF UNCONDITIONAL GENERATION ON LSUN
A.1 EVALUATION WITH 100 DENOISING STEPS
In this section, we expand our evaluation of unconditional image generation using LDM mod- els (Rombach et al., 2022a) high-resolution L.SUN-eedrooms and LSUN-Churches datasets, as illustrated in Table A. The full-precision LDM models are 7.6x larger in size compared to DDIM (Song et al., 2021) models. Therefore most methods focus on PTQ. Under W8A8 quan- tization, our approach surpasses all previous methods, including the QAT-based LSQ (Esser et al., 2019), demonstrating superior performance in terms of FID. Under W6A6 quantization, existing PTQ-based methods all exhibit significant performance degradation. For instance, on the LSUN- Bedrooms dataset, the state-of-the-an PTQ-based method ADP-DM (Wang et al., 2023) experiences substantial FID increase of 6.45, whereas our method reduces FID increase to mere 0.19. Un- der W4A4 quantization, while all other PTQ-based methods fail to generate meaningful images, approach still achieves a FID of 10.60 PTQ4DM (Shang et al., 2023) at W6A6 bit-width. Although LSQ. a QAT-based method, attains a lower FID at W4A4 and W6A6 bit-widths, it demands the original training dataset and consumes considerably more time and computing resources. In contrast, our approach achieves a FID that is only 0.07 higher than LSQ W6A6 bit-width, striking a better balance between performance and efficiency.
surpassing the performance of
Table A: Performance comparisons of unconditional image generation on LSUN datasets. 'N/A denotes failed image generation.
LSUN- BeN-rudrsusss
LSUN-Churches
LDM-4 (steps
100)
LDM-8 (steps = 100)
Bitwidth
Model Size
Bitwidth
Model Size
Method
1ST
FIDI
Method
1st
FID
(W/A)
(MB)
(W/A)
(MB)
FP
32732
1045.4
2.29
3.43
FP
32/32
IT25.2
2.70
4.08
PTOADM
8/8
261.67
2.21
4.75
PIQUDM
8/8
281.85
2.52
5.54
Q Q-Diffusion
8/8
261.67
2.19
4.67
O-Diffusion
8/8
281.85
2.53
4.87
ADP-DM
8/8
261.65
2.35
3.88
ADP-DM
8/8
281.85
2.69
4.02
LSQ
261.67
2.18
3.23
LSQ
8/8
281.85
2.68
4.06
261.69
2.27
2.98
Ours
8/8
281.89
2.71
PTO4DM
196.33
2.08
11,10
PTQ4DM
6/6
211.53
2.46
11.05
Q-Diffusion
6/6
196.33
2.11
10.10
Q-Diffusion
6/6
211.53
2.47
10.90
ADP-DM
6/6
196.33
2.27
9.88
ADP-DM
6/6
211.53
2.67
6.90
LSQ
196.33
2.16
3.55
LSQ
6/6
211.53
2.66
5.04
Ours
196.36
2.28
3.62
Ours
6/6
211.57
2,82
6.29
PTQ4DM
130.99
N/A
NIA
PTQ4DM
4/4
141.20
N/A
Q-Diffusion
130.99
N/A
N/A
Q-Diffusion
4/4
141.20
N/A
N/A
ADP-DM
4/4
130.99
N/A
N/A
ADP-DM
4/4
141.20
N/A
N/A
LSQQ
4/4
130.99
2.11
6.17
LSQ
4/4
141.20
2.63
9.08
Ours
4/4
131.02
2.27
10.60
Ours
4/4
141.24
2.81
14.34
A.2 EVALUATION WITH 20 DENOISING STEPS
In this section, we perform experiments on -SUN-Bedrooms with 20 steps to validate the effec- tiveness of interpolated quantization scales, as shown in Table B. Notably, our approach introduces only a slight FID increase of 0.01 and 0.64 for W8A8 and W6A6 bit-widths, respectively, which demonstrates the effectivvness the interpolated quantization scales in the fewer-step regime.
B DEPLOYMENT EFFICIENCY
In this section, we evaluated the latency of matrix multiplication and convolution operations in both quantized and full-precision diffusion models, utilizing an RTX3090 GPU and the CUTLASS (Kem et al.. 2017) implementation, quantized to 8-bit, we observe a 8.74x reduction model size and a 2.03x reduction in latency compared to their full-precision counterparts. Furthermore, with weights and activations quantized
demonstrated in Table C. When both weights and activations
13

---

Published as a conference paper at ICLR 2024
Table B: Performance of unconditional image generation on LSUN-Bedrooms with 20 steps. 'N/A' denotes failed image generation.
Bitwidth
Model Size
Method
ISt
FID
(W/A)
(MB)
FP
32/32
1045.4
2.21
9.74
Q-Diffusion
261.67
2.13
Ours
8/8
261.69
2.20
9.75
Q Diffusiion
6/6
196.33
3.78
149.39
Ours
6/6
196.36
2.27
10.39
Q-Diffusion
4/4
130.99
N/A
N/A
Ours
4/4
131.02
2.24
16.35
to 4-bit, we observe a reduction in model size by 6.58x and an increased speedup 3.34x. In the case of W2A8 quantization, the model size is further reduced by 10.57x. However, there is currently no hardware support for W2A8 precision on Nvidia's ' GPUs. Therefore, the calculations still conducted under W8A8 precision.
Table C: Comparisons of time cost across various bitwidth configurations on ImageNet 256 256.
Bitwidth (W/A)
Model Size
Precision (%)
Time
Model
ISt
FID4
sFID.
(MB)
(ms)
32/32
1603.68
364.73
11.28
7.70
93.66
436.8
8/8
427.65
362.34
11.38
8.04
93.77
214.4
LDM- DDIM (20 steps)
4/4
243.61
250.90
6.17
7.75
86.02
130.4
2/8
151.60
175.03
8.12
78.90
214.4
C VARIATION OF ACTIVATION DISTRIBUTIONS ACROSS STEPS
In this section, we present the value ranges activations across various time steps in the LDM-4 model pretrained on ImageNet 256 x 256 dataset, illustrated in Figure A. Previous PTQ meth- ods for diffusion models employ a single set of quantization parameters for activations across all time steps. However, this approach result in significant quantization errors when dealing with steps with distinct activation distributions (such as step 0 in Figure A), thereby leading to subop- timal denoising performance. In contrast, proposed TALSQ addresses this issue by assigning different quantization parameters to individual time steps, a scheme that can be optimized during the fine-tuning process. We further visualize the learned quantization scales and the variance of the activations in Figures B and C. Layers exhibiting significant shifts in activation distribution show considerable variation in the learned scales across different steps (see Figure Ba), and vice versa in the remaining subfigures.
2-
Value
19
15
11
Step
Figure A: Ranges of model output across various steps. Results are obtained by LDM-4 model on ImageNet 256 256 dataset.
14

---

Published as a conference paper at ICLR 2024
0.5
0.180
0.044
0.4
0.33
Activattinn quantDationn Scale
0.175
Actvatton variacce
0.30
Activation variance
10.170
0.3
-0.23
0.038
0.:05
0.2
0.20
0.036
00.15
0.034
0.155
0.032
0.150
19181716151413111
Step
Stop
(a) www.el.inut..lloc....o.. wttp:ptt.ooooss.......... Jayers.
(b) modellinpuutbblcck..44. o.tto.
Figure B: The variance of activations and the learned quantization scales of intermediate layers across different steps. Caption denotes the layer name. Data is collected by W4A4 LDM-4 model on ImageNet 256 256.
0.072
0.1660
0.024
Adtvation veriance
0.070
Activation variance
0.068
0.1660
0188
0.1658
0.066
0.1814
0.1656
0.064
0.1812
0.014
0.1654
1911116111419121113
191817161514131221
Stop
Step
model.input blocks. 1.0.in laiers2s
model.input bhlocksssttthkkss... layersa
Figure C: The variance of activations and the learned quantization scales of intermediate layers across different steps. Caption denotes the layer name. Data collected by W4A4 LDM-4 model on -EUN-Bedrooms.
D ADDITIONAL ABLATION STUDY
D.I ABLATION STUDY OF TALSQ
In this section, compare the performance of TALSQ with dynamic quantization and time-step- wise post-training quantization. The experiments are conducted over ImageNet 256 x 256 with W4A4 LDM-4 models. As shown Table D, the naive Min-Max dynamic quantization fails to denoise images, even with QALoRA weights fine-tuned. Time-step-wise quantization, obtained through post-training techniques with calibration sets, outperform non-time-step-win quantization by incorporating temporal information. Utilizing TALSQ further improves performance, resulting in reduction of 1.64 FID and 2.03 sFID, showcasing the superiority of the learned temporal quantization scales.
D.2 SELECTION RANK
In this section, we conduct ablation experiments to explore the impact of the rank parameter (r) LoRA on its performance, as outlined in Table E. When the rank is set below 32, the model experiences a substantial decrease in performance following the fine-tuning process. However, when the rank is set at 32, 2.39% model parameters are trainable, resulting in a 4.61 decrease in FID. Increasing the rank further 64 leads to 2.27% increase in trainable parameters, with only marginal 0.09 reduction in FID. Consequently, we set r = 32 as the default value for all experiments.
15

---

Published as a conference paper at ICLR 2024
Table D: Performance comparisons of various time-step-wise quantization scheme on ImageNet 256 x 256.
Bit-width (WIA)
Method
ISt
FID
sFID)
Precisiont
haseline
172.96
9.55
13.23
81.87
OALLoAsscale-awar LoRA optimization)
baseline dynamic quantization
4/4
N/A
N/A
N/A
N/A
baseline PTQ time-ster step--steeiee quantization
224.02
7.81
9.78
82.42
baseline TALSQ
250.90
6.17
7.75
86.02
Table E: Performance comparisons under different rank LoRA. Data is collected over 6-bit LDM- 8 model on LSUN-Churches
r=8
r=16
r'=32
r=64
FID
10.90
10.53
8.06
6.29
6.20
Trainable Parameters (%)
0.61
1.20
2.39
4.66
E GRADIENT ANALYSIS OF QALORA
In this section, we further analyze the gradient of QALoRA, elucidating the reasons behind its ineffective learning. Referring to Eq. (6) and considering the impact of STE (Bengio et al., 2013), the gradient of BA can be expressed as:
OY OY
OW
XT.
(A)
ABA OW 0(BA)
The key insight that the gradient magnitudes of LoRA weights remain unaffected by the quanti- zation scale, which is exactly the step size of the quantization step-like function. Consequently, in layers with larger quantization scales, the minor updates to LoRA weights will be diminished by the round operation.
F ADDITIONAL VISUALIZATION RESULTS
(a) Q-Diffusion
(b) Ours
Figure D: Samples generated by W4A4 LDM model on ImageNet 256 x 256.
16

---

Published as a conference paper at ICLR 2024
(a) Q-Diffusion
(b) Ours
Figure E: Randomly generated samples by W4A8 LDM model on ImageNet 256 x 256.
(a) Q-Diffusion
(b) Ours
Figure F: Randomly generated samples by W8A8 L.D model on ImageNet 256 256.
(a) Q-Diffusion
(b) Ours
Figure G: Randomly generated samples by W4A4 LDM model on LSUN-Bedrooms.
17

---

Published as a conference paper at ICLR 2024
(a) Q-Diffusion
(b) Ours
Figure H: Randomly generated samples by W6A6 LDM model on LSUN-Bedrooms.
(a) Q-Diffusion
(b) Ours
Figure Randomly generated samples by W8A8 LDM model on LSUN-Bedrooms.
18

---

Published as a conference paper at ICLR 2024
(a) ImageNet 256 256 samples
(b) samples
Figure J: Samples generated by full-precision LDM models.
FP
(b) Ours (W2A32)
Figure K: Randomly generated samples by FP and W2A32 Stable Diffusion model with prompt "A cozy cabin nestled in a snowy forest with smoke rising from the chimney".
(a) FP
(b) Ours (W2A32)
Figure L: Randomly generated samples by FP and W2A32 Stable Diffusion model with prompt "a painting of a virus monster playing guitar".
19

---

Published as a conference paper at ICLR 2024
(a) FP
(b) Ours (W2A32)
Figure M: Randomly generated samples by FP and W2A32 Stable Diffusion model with prompt "a magical fairy tale castle on a hilltop surrounded by a mystical forest".
EfficientDM Finetuning
Figure N: Visualization of samples generated by PTQD (the leftmost column) and EfficientDM dur- ing fine-tuning process. Results are obtained by 4-bit LDM model on PTQD fails to denoise images when both weights and activations are quantized to 4-bit, while Effi cientDM recovers its strong capability to generate high-quality images through the proposed efficient fine-tuning framework.
dataset.
20


Reviews:
[{"review_id": "xwL7I36Hg3", "reviewer": "Reviewer_ec6g", "summary": "This paper proposes a quantization-aware variant of low rank adapter and a data-free training scheme  for fine-tuning quantized diffusion models. It introduces scale-aware techniques to optimize the weight quantization parameters. For activation quantization, this paper employs a separate activation quantization step-size parameter for each denoising time step. With tuning the low rank weight parameter adapters, this method can achieve  image generation performance comparable to QAT based methods with much lower fine-tuning cost. It firstly achieves FID score as low as 6.17 on conditional image generation on ImageNet 256x256 dataset with 4bit-weight, 4bit-activation diffusion model.", "questions": "* In Table 2, the FID score of W4A4 model (6.17) is much lower than FP model (11.28) and W8A8 model (11.38) is comparable to FP model. Is there any possible explanation for that? And in the paper \"High-Resolution Image Synthesis with Latent Diffusion Models\", FID score of conditional generation on ImageNet 256x256 is 3.60, which is much lower than 11.28, why is there a gap? In the Appendix.A Table.A, the unconditional image generation on LSUN dataset, the FID score of W4A4 model is much worse than the FP model, how to explain the gap in these two set of experiments?\n\n* Table 3 shows the ablation study results. Does the **QALoRA** use LSQ algorithm to fine-tune the low rank adapter parameters and quantization step-size parameters?  \n\n* Are there any results on using LSQ method on quantized diffusion models on dataset other than Cifar10?\n\n* In **Data-free fine-tuning for diffusion models** part, is $\\mathbf{x}_t$ in Eq(7) sampled from Gaussion noise with an FP model?\n\n* In **Variation of activation distribution across steps** part, it proposes to assign a separate step size parameter for activation quantization in each denoising time step, and the results shown in Table 2 are obtained from 20-step sampling. Is the total time steps fixed to 20 for the fine-tuning. Is the data-free fine-tuning in Eq(7) fixed for 20 steps?\n\n* In Sec3.2 Eq(3), the quantization scheme has three parameters, $l, u, s$, are they all trainable? If so, is it optimized with LSQ [2] or LSQ+ [3] algorithm?\n\n\n[1] High-Resolution Image Synthesis with Latent Diffusion Models.\n\n[2] Learned step size quantization.\n\n[3] LSQ+: Improving low-bit quantization through learnable offsets and better initialization.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* This paper is the first to achieve very good image generation performance with W4A4 diffusion models and W2A8 diffusion models.\n\n* This paper introduces **low rank adapter** and **distillation loss** to fine-tune quantized diffusion models and achieve good results with relatively low cost than QAT methods.", "weaknesses": "* The experimental results listed in this paper are confusing and do not align well. The effectiveness is not very well proved. \n\n* This paper proposes **TLSQ**, which is quite similar to TDQ in [1]. TDQ is applicable to diffusion models with both continuous time and arbitrary discrete time steps . The paper should clarify the number of time steps used in TLSQ and discuss the settings.  \n\n[1] Temporal dynamic quantization for diffusion models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WMVclxuBq0", "reviewer": "Reviewer_5Wot", "summary": "This paper introduces a data-free fine-tuning framework tailored for low-bit diffusion models. The key approach involves freezing the pretrained diffusion model and fine-tuning a set of quantization-aware LoRA variants (QALoRA) by employing knowledge distillation to capture the denoising capabilities in the full-precision model. The paper also introduces two techniques, namely scale-aware optimization and learned step-size quantization, to address challenges related to ineffective learning of QALoRA and variations in activation distributions. Extensive experiments highlight that EfficientDM achieves performance levels comparable to QAT methods while preserving the data and time efficiency advantages of PTQ methods.", "questions": "See weaknesses", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1.\tAchieving QAT-level performance with PTQ-level efficiency is significant and promising for low-bit diffusion models.\n2.\tThe idea of the QALoRA is novel. Compared to QLoRA, it avoids extra floating-point calculations during inference.\n3.\tThe results are encouraging and demonstrate the strong performance of EfficientDM under various bit-widths.\n4.\tThe paper is well-organized and easy to follow. The supplementary material provides additional experimental results and comprehensive visualization results, which enhance the overall credibility of the work.", "weaknesses": "1.\tIt would be beneficial to evaluate EfficientDM over recent text-to-image diffusion models, such as Stable Diffusion.\n2.\tRecent work TDQ [1] also introduces a quantization method that adjusts the quantization scale at various denoising steps. The differences should be discussed.\n3.\tFormulating the gradient of LoRA weights can help elucidate the reasons for ineffective learning of QALoRA.\n4.\tFigure 2: the notation for scale-aware optimization is inconsistent with Eq. (8), please fix it.\n\n[1] So, Junhyuk, et al. “Temporal Dynamic Quantization for Diffusion Models.” arxiv 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iEjIvf2S6p", "reviewer": "Reviewer_BLfX", "summary": "This paper presents a data-free and efficient Quantization-Aware Training (QAT) method for diffusion models. For efficient QAT, it introduces the Quantization-Aware Low-Rank Adapter (QALoRA), which combines LoRA and QAT. The paper extends the LSQ QAT method a little to the Temporal LSQ method, which learns different scale factors for different time steps to handle the activation distribution difference across steps. The experimental of image diffusion and latent diffusion models on CIFAR-10, LSUN, and ImageNet demonstrates that this method can significantly outperform previous PTQ methods when doing W4A4 and W2 quantization.", "questions": "- How to get $\\bf{x}_t$ in equation (7) is not described properly. Do the authors sample $\\bf{x}_T \\sim \\mathcal{N}$, and run several solver steps using the FP model to get $\\bf{x}_t$, or otherwise? If it is the case, the equation (7) is not written properly.\n- The proposed Temporal LSQ (TLSQ) method uses a different activation quantization scale for different time steps. Can the authors show the learned scales and analyze how the scale factors of certain layers change w.r.t. the time steps on different datasets? \n- Can the authors compare TLSQ with deciding the time-step-wise activation quantization scale using some calibration data or even run-time dynamic quantization? This can help illustrate the necessity of LSQ.\n- The paper mentioned that \"we interpolate the learned temporal quantization scales to deal with the gap of sampling steps between fine-tuning and inference\". I found steps=100 experiments on CIFAR-10 and LSUN in Table 1 and Appendix A, I wonder if the authors experimented with using a different schedule with fewer steps? Does this scale-deciding technique work well in the fewer-step regime?\n- Is QALoRA applied for all the weights, including the convolutions and the attention layers?\n- For the good of future efficient diffusion, can the authors discuss more relevant limitations and raise questions worth future studying? The current discussion is not specific.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Applying QAT to diffusion models to achieve better quantization performance is reasonable.\n- The proposed QALoRA is data-free and efficient (cost about 10 GPU hours).\n- The experimental results are promising.\n- Actual speedup with CUTLASS is reported.", "weaknesses": "There exist some formulas and details that are not clear enough. Some additional ablation and analysis experiments are needed to make the overall method more convincing. Check the questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wRdBzZoupu", "reviewer": "Reviewer_SYoS", "summary": "The paper proposes a new quantization scheme for diffusion models. In particular, the paper notes that post-training quantization (PTQ) may be efficient but brings relatively low performance compared to quantization-aware training (QAT). Conversely, QAT brings higher performance but requires heavy computational resources. To combine the advantages of these two main quantization approaches, the paper introduces a quantization counterpart of low-rank adapter (LoRA). The paper also proposes to quantize the model in a data-free manner through distillation from the original full-precision model.", "questions": "Please refer to the weaknesses section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a new quantization method that brings substantial efficiency improvement without incurring extra computational overhead and performance degradation.\n\n- The paper demonstrates strong performance.\n\n- The paper is clearly written.", "weaknesses": "- Comparisons: What happens if other quantization models also employ LoRA and distillation, which are common techniques to use?\n\n- Novelty concern: I think the paper presents a combination of existing works (scale-aware optimization from LSQ, common distillation technique, and LoRA). Can the authors clarify the difference in contribution from the combination of existing works? If there is a difference, how does the performance differ compared to the combination?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)