BitsFusion: 1.99 bits Weight Quantization of Diffusion Model
Authors: Yang Sui
Yanyu Li
Anil Kag
Yerlan Idelbayev
Junli Cao
Ju Hu
Dhritiman Sagar
Bo Yuan
Sergey Tulyakov
Jian Ren
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion
Quantization
Stable Diffusion
Low bit
Primary area: diffusion_based_models
TL;DR: We propose BitsFusion, which quantizes the text-to-image model into 1.99 bits.

Abstract: Diffusion-based image generation models have achieved great success in recent years by showing the capability of synthesizing high-quality content. However, these models contain a huge number of parameters, resulting in a significantly large model size. Saving and transferring them is a major bottleneck for various applications, especially those running on resource-constrained devices. In this work, we develop a novel weight quantization method that quantizes the UNet from Stable Diffusion v1.5 to $1.99$ bits, achieving a model with $7.9\times$ smaller size while exhibiting even better generation quality than the original one. Our approach includes several novel techniques, such as assigning optimal bits to each layer, initializing the quantized model for better performance, and improving the training strategy to dramatically reduce quantization error. Furthermore, we extensively evaluate our quantized model across various benchmark datasets and through human evaluation to demonstrate its superior generation quality.

BitsFusion: 1.99 bits Weight Quantization of Diffusion Model
Yang Sui
Yanyu Li'
Anil Kag"
Yerlan Idelbayev'
Junli Cao Ju Hu
Dhritiman Sagar
Bo Yuan²
Sergey Tulyakov
Jian Ren 1.*
SSnap Inc.
Rutgeer University
Project Page: hispseecccccciiiu..oooiiiiiiiiii
Figure 1: Top: Images generated from full-precision Stable Diffusion v1.5. Bottom: Images generated from BitsFusion, where the weights of UNet are quantized into 1.99 bits, achieving 7.9x smaller storage than the one from Stable Diffusion v1.5. All the images are synthesized under the setting of using PNDM sampler [49] with 50 sampling steps and random seed as 1024. Prompts and more generations are provided in App. M.
Abstract
biffusion-basee image generation models have achieved great success in recent years by showing the capability of synthesizing high-quality content. However, these models contain huge number of parameters, resulting in a significantly large model size. Saving and transferring them a major bottleneck for various applications, especially those running on resource-constrained devices. In this work, we develop novel weight quantization method that quantizes the UNet from Stable Diffusion v1.5 to 1.99 bits, achieving model with 7.9x smaller size while exhibiting even better generation quality than the original one. Our approach includes several novel techniques, such assigning optimal bits to each layer, initializing the quantized model for better performance, and improving the training strategy dramatically reduce quantization error. Furthermore, extensively evaluate our quantized model across various benchmark datasets and through human evaluation to demonstrate its superior generation quality.
Introduction
Recent efforts developing diffusion-basee image generation models [77, 31, 79, 21, 80] have demonstrated remarkable results in synthesizing high-fidelity and photo-realistic images, leading various applications such as content creation and editing [68, 67, 61, 71, 90, 88, 50, 40], video generation [20, 75, 3, 1, 16, 57, 15], and 3D asset synthesis [87, 44, 64, 74, 65), among others.
Work done during internship at Snap Inc. "Correspndingg author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

However, Diffusion Models (DMs) come with the drawback of a large number of parameters, e.g., millions or even billions, causing significant burdens for transferring and storing due to the bulky model size, especially on resource-constrainee hardware such as mobile and wearable devices.
Existing studies have explored reducing the model size of large-scale text-to-image diffusion models by designing efficient architectures and network pruning [41, 92, 32]. These approaches usually require significant amounts of training due the changes made to the pre-trained networks. Another promising direction for model storage reduction is quantization [12, 30], where floating-point weights are converted low-bit fixed- exed-point representations, thereby saving computation memory and storage.
There have been emerging efforts on compressing the DMs through quantization [73, 38, 39]. However, these approaches still face several major challenges, especially when quantizing large-scale text-to-image diffusion models like Stable Diffusion v1.5 (SD-v1.5) [70]. First, many of these methods are developed relatively small-scale DMs trained on constrained datasets. For example, models trained on CIFAR-10 10 require modest storage of around 100 MB 39]. In contrast, SD-v1.55 necessitates 3.44 GB of storage in full-precision format. Adapting these methods to SD-v1.5 remains to be challenging problem. Second, current arts mainly focus on quantizing weights to 4 bits. How to quantize the model extremely low bit is not well studied. Third, there is a lack of fair and extensive evaluation of how quantization methods perform on large-scale DMs, i.e., SD-v1.5.
To tackle the above challenges, this work proposes BitsFusion, quantization aware training frame- work that employs a series of novel techniques to compress the weights of pre-trained large-scale DMs into extremely low bits (i... 1.99 bits), achieving even befter performance (i.e., higher image quality and better text-image alignment). Consequently, we compress the 1.72 GB UNet (FP16)' of SDDv11.5 into 219 MB model, achieving 7.9x compression ratio. Specifically, our contributions can be summarized into the following four dimensions:
Quantization for DMs. We propose effective approach for quantizing DMs in a mixed-precision manner. First, we thoroughly analyze the appropriate metrics to understand the quantization error in the quantized DMs (Sec. 3.2). Second, based on the analysis, we quantize different layers into different bits according to their quantization error (Sec. 3.3).
Initialization for Quantized DMs. We introduce several techniques initialize the quantized model to improve performance, including time embedding pre-computing and caching, adding balance integer, and alternating optimization for scaling factor initialization (Sec. 4.1).
Improved Training Pipeline for Quantized DMs. We improve the training pipeline for the quantized model with the proposed two-stage training approach (Sec. 4.2). In the first stage, use the full-precision model as a teacher to train the quantized model through distillation. Our distillation loss forces the quantized model to learn both the predicted noise and the intermediate features from the teacher network. Furthermore, we adjust the distribution of time step sampling during training, such that the time steps causing larger quantization errors are sampled more frequently. In the second stage, we fine-tune the model using vanilla noise prediction [21].
Extensive Quantitative Evaluation. For the first time in the literature, we conduct extensive quantitative analysis to compare the performance of the quantized model against the original SD-v1.5. We include results on various benchmark datasets, i.e., TIFA [25], GenEval [13], CLIP score [66] and FID [19] on MS-COCO 2014 validation set [46]. Additionally, we perform human evaluation on PartiPrompts [86]. Our 1.99-bit weights quantized model consistently outperforms the full-precision model across various evaluations, demonstrating the effectiveness of our approach.
2 Related Works
To enhance model efficiency in terms of storage and computational costs, quantization [11, 59, 58, 43, 36, 60, 48, 84, 45, 53] is adopted for diffusion models [73, 38, 18, 76, 81, 83, 51, 85, 4, 82, 7, 93, 27, 17, 39, 91] with primarily two types: post-training quantization (PTQ) and quantization-awe awaaee training (QAT). PTQ does not require a full training loop: instead, it utilizes limited calibration dataset to adjust the quantization parameters. For example, PTQ4DM [73] calibrates the quantization parameters to minimize the quantization error of DMs. Q-Diffusion [38] minimizes the quantization error via the block-wise reconstruction [42]. PTQD [18] integrates quantization noise into the
For SD-v1.5, we measure the generation quality using the FP32 format. However, since SD-v1.5 1.5 FP16 has similar performance to SD-v1.5 FP32, use SD-v1.5 FP16 to calculate compression ratio.

---

stochastic noise inherent in the sampling steps of DMs. TDQ [76] optimizes scaling factors for activations across different time steps, applicable to both PTQ and QAT strategies. TFMQ [27] focuses on reconstructing time embedding and projection layers to prevent over-fitting. However, PTQ often results in performance degradation compared to QAT, partieularly when aiming for extremely low-bit DMs. In contrast, QAT involves training the full weights to minimize the quantization error, thereby achieving higher performance compared to PTQ. For instance, EfficientDM [17], inspired by LoRA [24], introduces quantization-wwar low-rank adapter to update the LoRA weights, avoiding training entire weights. Q-DM [39] employs normalization and smoothing operation on attention features through proposed Q-attention blocks, enhancing quantization performance. Nevertheless, existing works primarily study 4 bits and above quantization on small-scale DMs trained on constrained datasets. In this paper, we focus on quantizing large-scale Stable Diffusion to extremely low bits and extensively evaluating the performance across different benchmark datasets.
3 Mixed Precision Quantization for Diffusion Models
In this section, we first go through the formulations of weight quantization and generative diffusion models. We then determine the mixed-precision strategy, assigning optimized bit widths to different layers to reduce the overall quantization error. Specifically, first analyze the quantization error of each layer in the diffusion model and conclude sensitivity properties. Then, based on the analysis, we assign appropriate bits each layer by jointly considering parameter efficiency (i.e., size savings).
3.1 Preliminaries
Quantization is a popular and commonly used technique to reduce model size. While many quantization forms exist, we focus on uniform quantization, where full-precision values mapped into discrete integer values as follows:
P Clip(1 01e 1 0, -1),
(1)
where Orp denotes the foottig-poiit weights, Own is the quantized integer weights, s the scaling factor, 12 is the zero point, and b is the quantization bit-width. [-1 denotes the nearest rounding operation and Clip(-) denotes the clipping operation that constrains Oiat within the target range. Following the common settings [38, 17]. we apply the channel quantization and set 8 bits for the first and last convolutional layer of the UNet.
Stable Diffusion. Denoising diffusion probabilistic models [77, 21] learn to predict real data distribution x ~ Paata by reversing the ODE flow. Specifically, given a noisy data sample z, 0,x + dre (a, and o, are SNR schedules and c is the added ground-trune noise), and a quantized denoising model E parameterized by Oint and s, the learning objective can be formulated as follows,
(2) where is the sampled time step and c is the input condition (e.g., text embedding). Note that during the training of quantized model, optimize Ozp and s by backpropagating Lom.. via Straight- Through Estimator (STE) [2] and quantize the weights to the integers for deployment. Here, for the notation simplicity, we directly use Ornt to represent the optimized weights in the quantized models.
Ceuu Ex,x ]||E E (1, 2,, c))
The latent diffusion model [70] such as Stable Diffusion conducts the denoising process the latent space encoded by variational autoencoder (VAE) [34, 69], where the diffusion model is the UNet [9]. This work mainly studies the quantization for the UNet model, given it the major bottleneck for the storage and runtime of the Stable Diffusion [41]. During the inference time, classifier-free guidance (CFG) [22] is usually applied to improve the generation,
Eo e (t. 2,, c) w.e... st. 2,, c) - -w--1) 1) . (t, z,, 2).
(3)
where w 2 and x 24, Ø) denotes the generation conditioned on the null text prompt Ø.
3.2 Per-Layer Quantization Error Analysis
We first perform a per-layer sensitivity analysis for the diffusion model. Specifically, given a pre-trained full-precision diffusion model, we quantize each layer to 1, 2, and 3 bits while freezing others at full-precision, and performing quantization-awee aware training (QAT)
Obtaining Quantized Models.
3

---

RB conv shortcut
RB conv shortcul
SD-v1.5
CA toq
CA tok
CA
CA tok
CA tok
RB conv
CA tok
(a) Left most column shows the images synthesized by SD-v1.5 FP32 and other columns show images generated by the quantized models, where only one layer is quantized (e.g.. CA toq denotes the cross- cross-attention layer for Query projection is quantized and RB conv shortcut denotes the Convolution Shotcut layer in Residual Block is quantized. The quantized layers follow the same order of highlighted layers in (b) and (c). from left to right.
Quantizing the layers impact both the image quality (as in RB cony shortcut) and text-image alignment (e.g., the teddy bear disappears after quantizing some CA tok layers).
noo CA tok
CA tok
RB cone sbortcut
tog
CA
R6 comr
RB cony shortcut
to
-A.tok
Quinttatton lro
RS conv shortout
CA tok
CA
conv shortcut
(b) MSE value by quantizing layers in SD-v1.5.
(c) CLIP score drop by quantizing layers in 11.5
Figure 2: 1-bit quantization error analysis for all the layers from the UNet of SD-v1.5.
respectively. For instance, for the SD-v1.5 UNet with 256 layers (excluding time embedding, the first and last layers), we get total of 768 quantized candidates. We perform QAT over each candidate on a pre-defined training sub dataset, and validate the incurred quantization error of each candidate by comparing it against the full- full-precision model (more details in App. B).
Measuring Quantization Errors. To find the appropriate way to interpret the quantization error, we analyze four metrics: Hean-Squared, Error (MSE) that quantifies the pixel-level discrepancies between images (generations from floating and the quantized model in our case), LPIPS [89] that assesses human-like perceptual similarity judgments, PSNR [23] that measures image quality by comparing the maximum possible power signal with the power of a corrupted noise, and CLIP score [66] that evaluates the correlation between an image and its language description. After collecting the scores (examples in Fig. 2b and Fig. 2c, full metrics are listed in App. F), we further measure the consistency of them by calculating the Pearson correlation [8] for different metrics under the same bit widths (in Tab. 1), and different bit widths under the same metric (in Tab. 2). With these empirical results, draw the following two main observations.
Observation 1: MSE, PSNR, and LPIPS show strong correlation and they correlate well with the visual perception of image quality.
Tab. 1 shows that MSE is highly correlated with PSNR and LPIPS under the same bit width. Additionally, we observe a similar trend of per-layer quantization error under different bit widths, as in Tab. 2. As for visual qualities in Fig. 2a and 2b, we can see that higher MSE errors lead to severe image quality degradation, e.g., the highlighted RB conv shortcut. Therefore, the MSE metric effectively reflects quality degradations incurred by quantization, and it unnecessary to incorporate PSNR and LPIPS further.
Observation 2: After low-hit quantization, changes in CLIP score not consistently correlated with MSE across different layers. Although some layers show smaller MSE, they may experience larger semantic degradation, reflected in larger CLIP score changes.
We notice that, after quantization, the CLIP score changes for all layers only have a weak correlation with MSE, illustrated in Tab. 1. Some layers display smaller MSE but larger changes in CLIP score.

---

Table 1: Pearson correlation (absolute value) of quantization eтrоr between different metrics (e.g. MSE vs. PSNR denotes the correlation between two metrics) when quantizing individual layers to 1. 2, and 3 bits. CS denotes CLIP Score.
Table 2: Pearson correlation (absolute value) of quantization error between different bit pairs (e.g. I vs. 2 denotes the correlation between the two bit widths) for single metric when quantiz- ing individual layers 1, 2, and bits.
MSE rs. PSNR MSE vs. LPIPS MSE vs. CS
MSE
PSNR
LPIPS
CLIP Score
bit
0,870
0.984
0.733
vs. 2 bit
0.929
0,954
0,943
0.504
bit
0.882
0.989
0.473
vs. 3 bit
0.766
0.843
0,802
0.344
3 bit
0.869
0.991
0,535
2 vs, 3 bit
0.887
0.923
0.895
0.428
For example, in Fig. 2b, the MSE of CA tok layer (5t, highlighted layer (green) from left to right) is less than that of RB conv layer (6,, highlighted layer (orange) from left to right), yet the changes in CLIP score are the opposite. As observed in the first row of Fig. 2a, compared to RB com layer, quantizing this CA tok layer changes the image content from "a teddy bear" to "a person", which diverges from the text prompt A leddy bear on a skateboard in Times Square, doing tricks on cardboard box ramp. This occurs because MSE measures only the difference between which does not capture the semantic degradation. In contrast, the CLIP score reflects the quantization error in terms of semantic information between the text and image. Thus, we employ the CLIP score as complementary metric to represent the quantization error.
images,
3.3 Deciding the Optimal Precision
With the above observations, we then develop the strategy for bit-width assignments. We select quantitative metrics, along with the number of parameters of each layer the
MSE and CLIP indicator of size savings.
Assigning bits based on MSE. Intuitively, layers with more parameters and lower quantization error are better candidates for extremely low-bit quantization, as the overall bit widths of the model can be significantly reduced. According to this, propose a layer size-aware sensitivity score S. For the ies layer, its sensitivity score for the b-bits (b E (1, 2, 3)) is defined as S. = Mr,NN, where M denotes the MSE error, N is the total number of parameters of the layer, and r E [0. 1] denotes the parameter size factor. To determine the bit width (i.e., b*) for each layer, we define sensitivity threshold as S., and the deh layer is assigned to bū -bits, where b = min < So). The remaining layers are 4 bits.
Assigning bits based on CLIP score. For the layers with a high CLIP score dropping after quantiza- tion, instead of assigning bits based on sensitivity score as discussed above, we directly assign higher bits to those layers. Therefore, the quantized model can produce content that aligns with the semantic information of the prompt. We provide the detailed mixed-precision algorithm in Alg. 1 of App. B.
4 Training Extreme Low-bit Diffusion Model
With the bits of each layer decided, then train the quantized model with series of techniques to improve performance. The overview of our approach is illustrated in Fig. 3.
4.1 Initializing the Low-bit Diffusion Model
Time Embedding Pre-computing and Caching. During the inference time of a diffusion model, time step is transformed into embedding through projection layers to be incorporated into the diffusion model. As mentioned by existing works [27], the quantization of the projection layers can lead to large quantization errors. However, the embedding from each time step č is always the same, suggesting that we can actually pre-compute the embedding offline and load cached values during inference, instead of computing the embedding every time. Furthermore, the storage size of the time embedding is 25.6x smaller than the projection layers. Therefore, we pre-compute the time embedding and save the model without the project layers. More details are provided in App. C.
Adding Balance Integer. general, weight distributions in deep neural networks are observed as symmetric around zero [94]. To validate the assumption on SD-v1.5, we analyze its weight
5

---

Freese
Trainable
Stable Dilfasion
UNet ااо-
Prompt: A cate
Tokenbzer/ Embedding
EO:o
Removed Time Projection Layers
Pre-computed and Cached
Amalysis
Time Features
L- Betaiie, f
Cleat L Pror.
Staged Training
(mooee
Precision Recipe
-0000
Pronupt: dog
Prodicted Neise
Tokrnizer Embedding
000-000 Quantised
E
Efim...
Staze-H1
Quantteed UNet
Initialization
Training Stage
Inference Stage
Figure 3: Overview of the training and inference pipeline for the propused BitsFusion. Left: We analyze the quantization error for each layer in SD-v1.5 (Sec. 3.2) and derive the recipe (Sec. 3.3) to assign different bit widths to different layers. We then initialize the quantized UNet by adding balance integer, pre-computing and caching the time embedding, and alternately optimizing the scaling factor (Sec. 4.1). Middle: During the Stage-I training, we freeze the teacher model (i.e., SD-v1.5) and optimize the quantized UNet through CFG-aware quantization distillation and feature distillation losses, along with sampling time steps by considering quantization errors (Sec. 4.2). During the Stage-Il training, fine-tune the previous model with the noise prediction. Right: For the inference stage, using the pre-cached time features, our model processes text prompts and generates high-quality images.
distribution for the layers under full precision by calculating the skewness of weights. Notably, the skewness of more than 97% of the layers ranges between 0-0.5, 0.5], indicating that the weight distributions are symmetric almost all layers. Further details are provided in App. D.
However, existing works on diffusion model quantization overlook the symmetric property [38, 73, 39], as they perform relatively higher bits quantization, e.g., 8 bits. This will hurt the model performance at extremely low bit levels. For example, in 1-bit quantization, the possible most symmetric integer outcomes can only be 1] or {-1, 0}. Similarly, for 2-bit quantization, the most balanced mapping integers can be either {-2. 2-22-14 -1, 0, 1] or (-1. 0, 1, 2}. significantly disrupting the numbers under low-bit quantization
symmetric property. The absence of a single value among can have a significant impact. To tackle this, we leverage the bit balance strategy [37, 56] to initialize the model. Specifically, we introduce an additional value to balance the original quantization values. Namely, in a 1-bit model, we adjust the candidate integer set from {0, 1} to {-1, 0, 1) achieving more balanced distribution. By doing so, we treat the balanced n-bits weights as 1og(2" + 1)-bits.
Scaling Factor Initialization via Alternating Optimization. Initializing scaling factors is an important step in quantization. Existing QAT works typically employ the Min-Max initialization strategy [17, 52] to ensure the outliers are adequately represented and preserved. However, such method faces challenges in extremely low-bit quantization settings like 1-bit, since the distribution of the full-precision weights is overlooked, leading to large quantization error and the increased difficulty to converge. Therefore, we aim to minimize the &g error between the quantized weights and full- full-precision weights with the optimization objective as:
min ||s (0in: I2) 0rpl1
(4)
Nevertheless, considering the rounding operation, calculating an exact closed-fome solution is not straightforward [29]. Inspired by the Lloyd-Max algorithm [28, 54], we use an optimization method on scaling factor to minimize the initialization emror of our quantized diffusion model as follows:
00, |m - -1)) (0im..
0 Qinc (Otp, s1 s²
(.) Iz)=
where Q.ne(.) denotes the integer mapping quantization operation that converts the full-precision weights to integer as Eq. (1), and represents the iterative step. The optimization is done for 10 steps.

---

4.2 Two-Stage Training Pipeline
With the prexissoe model initialized, we introduce the two-stage training pipeline. Stage-I, we train the quantized model using the -reeciiin model as the teacher through distillation loss. In Stage-II, we fine-tune the model from the previous stage using noise prediction [21, 80].
CFG-aware Quantization Distillation. Similar to existing works [11], we fine-tune the quantized diffusion model to improve the performance. Here both the weights and scaling factors are opti- mized. Additionally, we notice that training the quantized model in a distillation fashion using the full-precision model yields better performance than training directly with vanilla noise prediction. Furthermore, during distillation, it is crucial for the quantized model to be aware of CFG, i.e., text dropping is applied during distillation. Specifically, our training objective is as follows:
Chol
Oene 8 = xx,xx |||@o,, (1, 21, c) - s 2i, c) , Ø if U00 1] < p else c,
(6)
where P controls the text dropping probability during training and is set as 0.1
Feature Distillation. To further improve the generation quality of the quantized model, we distill the full-precision model at a more fine-grained level through feature distillation [32] as follows:
Bx,x [l (t, (t, c) Fe (t, Rr, c)]]
(7)
where (-) denotes the operation for getting features from the Down and Up blocks in UNet. We then have the overall distillation loss Cáist in Stage-I as follows:
Les a X
(8)
where x is empirically set as 0.01 to balance the magnitude of the two loss functions.
Quantization Error-aware Time Step Sampling. The training of diffusion models requires sampling differ-
ent time steps in each optimization iteration. We ex- plore how to adjust the strategy for time step sampling such that the quantization error in each time step can be effectively reduced during training. We first train a 1. 1.99-bit quantized model with Eq. (8). Then, we cal- culate the difference of the predicted latent features be- tween the quantized model and the full-precision model
Sampling Frequency
50-
3=3, 8=1 Quanlization
0F1 1.5, 6=1
0.012
Quanlization Error
40
0.008
30
20-
0.004
as Bx, 11 &e ||Eo,, (t,, 2,, c) (t, ze, c))l where t e 10, 9999] and āe is the noise scheduler (detailed derivation in App. E). The evaluation is conducted on a dataset with 128 image-text pairs. Fig. 4 shows the quan- tization error does not distribute equally across all time steps. Notably, the quantization error keeps increasing the time steps approach t = 999.
0
200
400
600
800
1000
0.000
Time Steps
Figure 4: More time steps are sampled towards where larger quantization error occurs.
To mitigate the quantization error prevalent near the time steps 999, we propose sampling strategy by utilizing a distribution specifically tailored to sample more time steps exhibiting the largest quantization errors, thereby enhancing performance. To achieve this goal, we leverage the Beta distribution. Specifically, time steps are sampled according to ~ Beta(a, (a, 3), as shown in Fig. 4. We empirically set 3.0 and 3 = 1.0 for the best performance. Combining the strategy of time steps sampling with Eq. (8), we conduct the Stage-I training.
Fine-tuning with Noise Prediction. After getting the model trained with the distillation loss in Stage-I, we then fine-tune it with noise prediction, as in Eq. (2), in Stage-II. We apply a text dropping with probability as 10% and modify the distribution of time step sampling based on the quantization error, as introduced above. The reason we leverage two-stage fine-tuning, instead of combining Stage-I and Sla is that we observe more stabilized training results.
5 Experiments
Implementation Details. We develop our code using diffusers library² and train the models with AdamW optimizer [33] and constant learning rate as le-05 internal dataset. For Stage-I,
7

---

GenEval Scores
CLPP Scores
TIFA Scores
0.80
0.44
0.310
0.78
0.305
Oum-I (1.99 bits)
SDw11.5 (32 bits)
0.78
Ours- (1.5 bits)
SD.v1.5 (32 bits)
0.38
SD-v1.5 (32 bits)
Ours (1 99 bits)
0.300
(1.3 19 bits)
0.74
bis)
0.32
Ours bits)
25
9.5
4.5
6.5 7.5 9.5 CFG Scales
4.5
6.5 CFG Scales
8.5
CFG Scales
(a) CLIP score on MS-COCO.
(b) Evaluation on TIFA.
(c) Evaluation on GenEval.
Figure 5: Comparison between our 1.99-bits model vs. SD-v1.5 various evaluation metrics with CFG scales ranging from 2.5 to 9.5. Ours-I denotes the model with Stage-I training and Ours-II denotes the model with Stage-Il training.
Table 3: Comparison with existing quantization meth- ods, including LSQ [11]. J. Q- Diffusion [38], EfficientDM [17], and Apple-MBP [62]. The CLIP score is measured on 1K PartiPrompts.
Table Analysis of our proposed methods measured under vari- CFG scales, ie.,, 3.5, 5.5, 7.5, and 9.5. We LSQ [11] the basic QAT method, which involves the training of weights and scaling factors of uniformly 2-bit quantized UNet. Then, we gradually introduce each proposed technique to evaluate their effectiveness. CLIP scores measured on 1K PartiPrompts.
Method
Bittwidth
5.5
Average
Method
Bit-width CLIP score
SD-v1.5
32
0.3110 0.3159 0.3175 0.3180 0.3156
SD-v1.5
32
0.3175
QAT-Basc Bas-
2
0,2679
0.2793 0.2849 0.2868
0.2797
LSQ
2
0,2849
-Balance
2.32
0,2990
0.3059 0,3080 0,3086
0.3054
+0.0257
O-Diffusion
0.3137
Altematinn Opt.
2.32
0,3061 0.3108 0.3117 0.3115
0.3100
+0.0046
EfficienDDM
2
0.2918
Mixed/Caching
1.99
0.3055 0.3129 0.3142 0.3145
0,3118
+0.0018
Apple-MBP
2
0,3023
Feaat Dist.
1.99
0,3086 0.3147 0.3167 0.3169
0.3142
+0.0024
+Time Sampling
0,3098 0.3159 0.3181
0.3184
0.3156
+0.0014
Ours
1.99
0,3212
+ -iine-tuning
1.99
0,3163 0.3192 0,3212 0.3205
0.3183
+0.0027
we use 8 NVIDIA A100 GPUs with total batch size of 256 to train the quantized model for 20K iterations. For Stage-II, we use 32 NVIDIA A100 GPUs with a total batch size of 1024 to train the quantized model for 50K iterations. During inference, sampling steps to generate images for comparison. Other sampling approaches (e.g., DDIM [78] and DPMSolver [55]) lead the same conclusion (App. K).
adopt the PNDM scheduler [49] with 50
Evaluation Metrics. We conduct evaluation on CLIP Score and FID on MS-COCO [47], TIFA [26].
GenEval [14], and human evaluation PartiPrompts [86]. We adopt ViT-B/32 model [10] in CLIP score and the Mask2Former (Sxin-S-8x2) [5] GenEval. App. provides details for the metrics.
5.1 Main Results
Comparison with SD-v1.5. Our quantized 1.99-bits UNet consistently outperforms the full full-precision
model across all metrics.
30K MS-COCO 2014 Validation Set. For the CLIP score, as demonstrated Fig. 5a, attributed to the proposed mixed-precision recipe with the introduced initialization techniques and advanced training schemes in Stage-I, our 1.99-bits UNet, with a storage size of 219MB, achieves perfor- mance comparable to the original SD-v1.5. Following Stage-ll training, our model surpasses the performance of the original SD-v1.5. With CFG seales ranging from 2.5 to 9.5, our model yields 0.002 ~ 0.003 higher CLIP scores.
TIFA. As shown in Fig. 5b, our 1.99-bits model with Stage-I training performs comparably to the SD-v1.5. With the Stage-Il training, model achieves better metrics over the 1.5.
GenEval. We show the comparison results for GenEval in Fig. 5c (detailed comparisons of
GenEval score presented in Appn. L). Our model outperforms SD-v1.5 for all CFG scales.
Human Evaluation. With the question: Given a prompt, which image has better aesthetics and image-text alignment? More users prefer the images generated by our quantized model over SD- v1.5, with the ratio as 54.4%. The results are shown in Fig. 6. We provide a detailed comparison in App. J.
Comparison with Other Quantization Approaches. Additionally, we conduct the experiments by comparing our approach with other works including LSQ [11], Q-Diffusion [38], EfficientDM [17],

---

Table 5: Analysis of 7) in the mixed- Table Anlysis of À in dis- precision strategy.
Table Analysis of a in
tillation loss.
time step aware sampling.
0.1
0.2
0.3 0.4
0.5
0.1
0.01
1.3
2.0
3.0
CLIP score 0.3155 0.3173 0.3162 0.3181 0.3171
0.3168
0.3164 0.3159 0.3181
CLI score 0.3169 0.3173 0.3181
and Apple-MBP [62], as shown in Tab. 3. Our model achieves a higher CLIP score compared with all other works and better performance than SD-v1.5.
5.2 Ablation Analysis
Here we perform extensive analysis for our proposed method. We mainly evaluate different experi- mental settings using the CLIP score measured on 1K PartiPrompts [86].
Analysis of the Proposed Techniques. We adopt the LSQ [11] the basic QAT method to update the weights and scaling factors of uniform 2-bit UNet with Min-Max initialization. Results are presented in Tab. 4 with the following details:
+Balance. By adding balance integer, a 2-bit model that typically represents 4 integer values can now represent 5 integers, becoming 2.32-bit model by 1og(4 + 1) bits. The average CLIP score has significantly increased from 0.2797 to 0.3054.
Alternating Opt. By further utilizing the scaling factor initialization via alternating optimization, the average CLIP score of the 2.32-bit model increases to 0.3100.
+Mixed/Caching, By leveraging time embedding pre-computing and caching, we minimize the storage requirements for time embedding and projection layers by only retaining the calculated features. This significantly reduces the averaged bits. Combined with our mixed precision strategy. this approach reduces the average bits from 2.32 to 1.99 bits and can even improve the performance,
mixed-precision
i.e., CLIP score improved from 0.3100 to 0.3118.
+Feat Dist. By incorporating the feature distillation loss, i.e., Eq. (7), the model can learn more fine-grained information from the teacher model, improving CLIP score from 0.3118 to 0.3142.
+ Time Sampling. By employing quantization error-aware sampling strategy at various time
steps, the model focuses more on the time step near t = 999. With this sampling strategy, our 1.99-bits model performs very closely to, or even outperforms, the original SD-v1.5.
+Fine-tuning, By continuing with Stage-Il training that incorporates noise prediction, our 1.99- bits model consistently outperforms the SD-v1.5 across various guidance scales, improving the CLIP score to 0.3183.
Effect of 1 in Precis-oe Strategy. Tab. 5 illustrates the impact of the parameter size factor 7) (as discussed Sec. 3.3) determining the optimal mixed precision strategy. We generate six different mixed precision recipes with different v with 20K training iterations for comparisons. Initially, we explore the mixed precision strategy determined with and without the parameter size factor. Setting 0 results in N = 1, indicating that the mixed precision is determined without considering the impact of parameter size. The results show that neglecting the parameter size significantly degrades performance. Further, empirically choose 1) 0.3 in our experiments after comparing different values of 1).
Effect of A of Distillation Loss. Tab. 6 illustrates the impact of the balance factor X for loss functions Eq. (8). We empirically choose x 0.01 in our experiments after comparing the performance.
Effect of a in Time Step-aware Sampling Strategy. Tab. 7 illustrates the impact of the for different Beta sampling distribution. As analyzed in Sec. 4.2, the quantization error increases near I = 999. To increase sampling probability near this time step, Beta distribution requires a > 1 with 1. A larger a enhances the sampling probability near 999. Compared to 1.5 and = 2.0, = 3.0 concentrates more on later time steps and achieves the best performance. We choose a 3.0 in our experiments.
Analysis for Different Schedulers. One advantage of our training- quantization approach is that quantized model consistently outperforms SD-v1.5 for various sampling approaches. We conduct extensive evaluations on TIFA to show we achieve better performance than SD-v1.5 for using both DDIM [78] and DPMSolver [55] perform the sampling. More details are shown in App. K.
FID Results. As stated in SDXL [63] and PickScore [35], FID may not honestly reflect the actual performance of the model in practice. FID measures the average distance between generated images

---

and reference real images, which is largely influenced by the training datasets. Also, FID does not capture the human preference which is the crucial metric for evaluating text-to-image synthesis. We present FID results evaluated on the 30K MS-COCO 2014 validation set in Fig. 7. Our Stage-I model has a similar FID as SDDvv1.5. However, as training progresses, although our Stage-Il model preferred by users, its FID score is higher than both Stage-I and SD-v1.5.
60
54.41
30
Win Win Rate (%)
45.59
25
40
FID (30K)
20
15
20
10
SD-v1.5 (32 bits)
Ours-I (1.99 bits)
Ours-Il (1.99 bits)
Ours (1.99 bits)
SD-v1.5 bits)
2.5
6.5 CFG Scales
Figure 6: Overall human evaluation com- parisons between SD-v1.55 and BitsFusion. Notably, BitsFusion, is favored 54.41 % of the time over SD-v1.5.
Figure 7: FID results evaluated 30K MS-COCO 2014 validation set.
6 Conclusion
To enhance the storage efficiency of the large-scale diffusion models, we introduce an advanced weight quantization framework. BitsFusion, which effectively compresses the weights of UNet from SD-v1.5 to 1.99 bits, achieving a 7.9x smaller model size. BitsFusion even outperforms SD-v1.5 in terms of generation quality. Specifically, we first conduct a comprehensive analysis to understand the impact of each layer during quantization and establish mixed-pree precisson strategy. Second, we propose series of effective techniques initialize the quantized model. Third, during the training stage, we enforce the quantized model to learn the full-precision SD-v1.5 by using distillation losses with the adjusted distribution of time step sampling. Finally, we fine-tune the previous quantized model through vanilla noise prediction. Our extensive evaluations on TIFA, GenEval, CLIP score, and human evaluation consistently demonstrate advantage of BitsFusion over full-precision SD-v1.5 15.
References
[1] Omer Bar- Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michaeli, et al. Lumiere: A space-time diffusion model for video generation. arXiv preprint arXiv:2401.12944 2024.
[2] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 1308.3432, 2013.
3
[3] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22563-22575, 2023.
[4] Hanwen Chang, Haihao Shen, Yiyang Cai, Xinyu Ye, Zhenzhong Xu. Wenhua Cheng, Kaokao Lv, Weiwei Zhang, Yintong Lu. and Heng Guo. Effective guantization for diffusion models on cpus. arXiv preprint arXiv:2311. 166133,,,. 2023. 2
[5] Bowen Cheng, Ishan Misra, Alexander Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IFFF/CVE conference on computer vision and pattern recognition, pages 1290-1299, 2022.
30
10

---

[6] Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-imaae generation models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3043-3054, 2023. 30
[7] Huanpeng Chu, Wei Wu, Chengjie Zang. and Kun Yuan. Qued: Quantization noise correction for diffusion models. arXiv preprint arXiv:2403. 19140, 2024. 2
[8] Israel Cohen, Yiteng Huang, Jingdong Chen, Jacob Benesty, Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. Pearson correlation coefficient. Noise reduction in speech processing, pages 1 1-4, 2009.
[9] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021. 3
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020. 8
[11] Steven K Esser, Jeffrey I. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dhar- mendra S Modha. Learned step size quantization. In International Conference on Learning Representations, 2019. 2, 7, 8,
[12] Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. In Low-Power Com- puter Vision, pages 291-326. Chapman and Hall/CRC, 2022. 2
[13] Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36, 2024. 2
[14] Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36, 2024. 8, 30
[15] Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu video: Factorizing text- to-video generation by explicit image conditioning. arXiv preprint arXiv:2311.10700) 2023.
[16] Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models. arXiv preprint arXiv:2312.06662 2023. 1
[17] Yefei He, Jing Liu, Weijia Wu, Hong Zhou, and Bohan Zhuang. Efficientdm: Efficient quantization-awar fine-tuning of low-bit diffusion models. In The Twelfth International Conference on Learning Representations, 2023. 2, 3, 6,
[18) Yefei He, Luping Liu, Jing Liu, Weijia Wu, Hong Zhou, and Bohan Zhuang. Ptqd: Accurate post-training quantization for diffusion models. Advances Neural Information Processing Systems, 36, 2024. 2
[19] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[20] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303. 2022.
[21] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 18668406881,, 2020. 1, 2, 7,

---

[22] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. 12X15
arXiv preprint
2022.
[23] Alain Hore and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In 2010 20th international conference on pattern recognition, pages 2366-2369. IEEE, 2010.
[24] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
[25] Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. arXiv preprint arXiv:2303.11897, 2023. 2
[26] Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In Proceedings of the IEEE/CVF International Conference on Computer
Vision, pages 20406-20417, 2023. 8, 30
[27] Yushi Huang, Ruihao Gong, Jing Liu, Tianlong Chen, and Xianglong Liu. Tfmqq-m: Temporal feature maintenance quantization for diffusion models. arXiv preprint arX1v:2311.16503, 2023.
2, 3, 5, 18
[28] Kyuyeon Hwang and Wonyong Sung. Fixed-point feedforward deep neural network design using weights+ 1, 0, and- 1. In 2014 IEEE Workshop on Signal Processing Systems (SiPS),
pages 1-6. IEEE, 2014. 6
[29] Yerlan Idelbayev, Pavlo Molchanov, Maying Shen, Hongxu Yin, Miguel A Carreira-Perpinán, and Jose Alvarez. Optimal quantization using scaled codebook. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12095-12104, 2021. 6
[30] Qing Jin, Jian Ren, Richard Zhuang, Sumant Hanumante, Zhengang Li, Zhiyu Chen, Yanzhi Wang, Kaiyuan Yang, and Sergey Tulyakov. F8net: Fixed-point 8-bit only multiplication for network quantization. arXiv preprint arXiv:2202.05239, 2022. 2
[31] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. arXiv preprint arXiv:2206.00364, 2022. 1
[32] Bo-Kyeong Kim, Hyoung-K Koyu Song, Thibault Castells, and Shinkook Choi. On architectural compression of to-image diffusion models, arXiv preprint 1757922305.15798, 2023. 2. 7
[33] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. 7
[34] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint
132.6:1312.6114, 2013. 3
[35] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for exxt-to-image generation. Advances in Neural Information Processing Systems, 36, 2024. 9
[36] Junghyup Lee, Dohyung Kim, and Bumsub Ham. Network quantization with element-wiss gradient scaling. In Proceedings of the IEEF/CVE conference computer vision and pattern recognition, pages 6448-6457, 2021. 2
[37] Fengfu Li, Bin Liu, Xiaoxing Wang. Bo Zhang, and Junchi Yan. Ternary weight networks. arXiv preprint arXiv: 1605.04711, 2016. 6
[38] Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. Q-diffusion: Quantizing diffusion models. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 17535-17545, 2023. 2, 3, 6,
12

---

[39] Yanjing Li, Sheng Xu, Xianbin Cao, Xiao Sun, and Baochang Zhang. Q-dm: An efficient low-bit quantized diffusion model. Advances in Neural Information Processing Systems, 36, 2024. 2, 3,
[40] Yanyu Li, Xian Liu, Anil Kag, Ju Hu, Yerlan Idelbayev, Dhritiman Sagar, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Textcraftor: Your text encoder can be image quality controller. arXiv preprint arXiv:2403. 18978, 2024. 1
[41] Yanyu Li, Huan Wang, Qing Jin, Ju Hu, Pavlo Chemerys, Yun Fu, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Snapfusion: Text-to-image diffusion model on mobile devices within two seconds. Advances in Neural Information Processing Systems, 36, 2024. 2,3
[42] Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang. Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Brecq: Pushing the limit post-training quantization by block reconstruction. In International Conference on Learning Representations, 2020. 2
[43] Yuhang Li, Ruihao Gong. Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Breeq: Pushing the limit of post-training quantization by block reconstruction. arXiv preprint 2rXiv:2102.05426, 2021.
[44] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. arXiv preprint arXiv:2211.1044
2022.
[45] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang. Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-awar weight quantization for on-device IIm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024.
[46] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Purt V 13, pages 740-755. Springer, 2014.
[47] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Purt V 13, pages 740-755. Springer, 2014. 8, 30
[48] Jing Liu, Bohan Zhuang, Peng Chen, Chunhua Shen, Jianfei Cai, and Mingkui Tan. Single- patghe bit sharing for automatic loss-aware model compression. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10): 12259-124772 2023. 2
[49] Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint arXiv:2202.09778. 2022. 1,8 8
[50] Xian Liu, Jian Ren, Aliaksandr Siarohin, Ivan Skorokhodov, Yanyu Li, Dahua Lin, Xihui Liu, Ziwei Liu, and Sergey Tulyakov. Hyperhuman: reaaettr human generation with latent structural diffusion. arXiv preprint arXhv:2310.08579 2023.
[51] Xuewen Liu, Zhikai Li, Junrui Xiao, and Qingyi Gu. Enhanced distribution alignment for post-training quantization of diffusion models. arXiv preprint arXiv:2401.04585, 2024. 2
[52] Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Data-free quan- tization aware training for large language models. arXiv preprint arXiv:2305.1788 2023.
[53] Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant quantization with learned rotations. arXiv preprint 2XXv:2405.116004 2024.
[54] Stuart Lloyd. Least squares quantization pem. IEFE transactions on information theory, 129-177,, 1982.
13

---

[55] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022. 8, 9, 31,
[56] Shuming Ma, Hongyu Wang. Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, and Furu Wei. The era of 1-bit Ilms: All large language models are in 1.58 bits. arXiv preprint arXiv:2402. 17764, 2024.
[57] Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, Ekaterina Deyneka, Tsai-Shien Chen, Anil Kag, Yuwei Fang. Aleksei Stoliar, Elisa Ricci, Jian Ren, et al. Snap video: Scaled spatiotemporal transformers for text-to-video synthesis. arXiv preprint arXiv:2402. (e0e22422.17777, 14797, 2024.
[58] Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. In International Conference on Machine Learning, pages 7197-7206. PMLR, 2020.
[59] Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1325-1334, 2019.
[60] Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart Van Baalen, and Tijmen Blankevoort. A white paper on neural network quantization. arXiv preprint
10Xiv:200600899 2021. 2
[61] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112. 2021.
[62] Atila Orhon, Michael Siracusa, and Aseem Wadhwa. Stable diffusion with core ml on apple silicon, 2022. 8,9
[63] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference Learning Representations, 2023. 9
[64] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 2022.
[65] Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin- Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, et al. Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors. arXiv preprint arXiv:2306. /743,, 2023.
[66] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In International conference on machine learning.
pages 8748 8763. PMLR, 2021. 2, 4, 18
[67] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 2rXiv:2004006/25, 2022.
[68] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.
[69] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pages 1278-1288 PMLR, 2014. 3
[70] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 10684-10695, 2022. 2,3
14

---

[71] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee. Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: to--ge- diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1-10, 2022. 1
[72] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, al. Photorealistic text-to to-image diffusion models with deep language understanding, Advances in neural information processing systems. 14:36799-36494 2022. 30
[73] Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan. Post-training quantization on diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1972-1981, 2023. 2, 6
[74] Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. In The Twelfth International Conference on Learning
Representations, 2023. 1
[75] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209, 14792, 2022. 1
[76] Junhyuk So, Jungwon Lee, Dachyun Ahn, Hyungjun Kim, and Eunhyeok Park. Temporal dynamic quantization for diffusion models, Advances Neural Information Processing Systems, 36, 2024. 2, 3
[77] Jascha vised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-2265. PMLR, 2015. 1,3
Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper-
[78] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 31,
[79] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[80] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 13466, 2020. 7.7
[81] Siao Tang, Xin Wang, Hong Chen, Chaoyu Guan, Zewen Wu, Yansong Tang, and Wenwu Zhu. Post-training quantization with progressive calibration and activation relaxing for text-to-image diffusion models. arXiv preprint arXiv:2311.00 12022211.00333 1.0622, 2023.
[82] Changyuan Wang, Ziwei Wang, Xiuwei Xu, Yansong Tang, Jie Zhou, and Jiwen Lu. Towards accurate data-free quantization for diffusion models. arXiv preprint 2rXiv:2305.187233 2023.
[83] Haoxuan Wang, Yuzhang Shang, Zhihang Yuan, Junyi Wu, and Yan Yan. Quest: Low-bit diffusion model quantization via efficient selective finetuning. arXiv preprint arXiv:2402.03666, 2024.7
[84] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087-38099, PMLR, 2023.
[85] Yuewei Yang, Xiaoliang Dai, Jialiang Wang, Peizhao Zhang, and Hongbo Zhang. Efficient quantization strategies for latent diffusion models. arXiv preprint arXiv:2312.05431, 2023. 2
[86] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfci Yang, Bureu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. Transactions on Machine Learning Research, 2022. 2, 8, 9, 30
[87] Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojeic, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3d shape generation. arXiv preprint arXiv:2210.0697. 2022. 1
15

---

[88] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to- image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
[89] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as perceptual metric. In CVPR, 2018.
[90] Zhixing Zhang, Ligong Han, Arnab Ghosh, Dimitris N Metaxas, and Jian Ren. Sine: Single image editing with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6027-6037, 2023.
[91] Tianchen Zhao, Xuefei Ning, Tongcheng Fang. Enshu Liu, Guyue Huang, Zinan Lin, Shengen Yan, Guohao Dai, and Yu Wang. Mixdq: Memory-efficiees few-step text-to-image diffusion models with metric-decoupled mixed precision quantization. arXiv preprint a7Xi: 2024.7
[92] Yang Zhao, Yanwu Xu, Zhisheng Xiao, and Tingbo Hou. Mobilediffusion: Subsecond text-to- image generation on mobile devices. arXiv preprint arXiv:2311.16567 2023. 2
[93] Xingyu Zheng, Haotong Qin, Xudong Ma. Mingyuan Zhang, Haojie Hao, Jiakai Wang, Zixiang Zhao, Jinyang Guo, and Xianglong Liu. Binarydm: Towards accurate binarization of diffusion model. arXiv preprint
006622664..6,,, 2024. 2
[94] Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained ternary quantization. In International Conference on Learning Representations, 2016.
16

---

Appendix
Table of Contents
A I Limitations
18
B More details for Mixed-Precision Algorithm
18
More Details for Time Embedding Pre-computing and Caching
D Analysis of Symmetric Weight Distribution
20
E More Details for Quantization Error Across Different Time Steps
20
F Detailed Metrics for Quantization Error by Quantizing Different Layers
21
More Visualization Quantization Error by Quantizing Different Layers
H 1.99 Bits Mixed Precision Recipe
26
Details for Evaluation Metrics
30
J Human Evaluation
30
J.1 Analysis on Categories
30
J.2 Analysis on Challenges
30
K Evaluation Different Schedulers
31
L. Detailed GenEval Results
32
M More Comparisons
33
M.1 Prompts
33
M.2 Additional Image Comparisons
33
17

---

A Limitations
In this work, we study the storage size reduction of the UNet in Stable Diffusion v1.5 through weight quantization. The compression of VAE and CLIP text encoder [66] is also an interesting direction, which is not explored in this work. Additionally, our weight quantization techniques could be extended to the activations quantization, as a future exploration.
B More details for Mixed-Precision Algorithm
In Sec. 3, we analyze the per-layer quantization error and develop the mixed-precision strategy. Here, we provide the detailed algorithm as outlined Alg. The inputs include: a pre-defined candidate set of bit- bit-width b E {1, 2, 3), the full-precision 31- D, the total number of layers I. (except for the time embedding, time projection, the first and last convolutional layers), the training dataset X, the number of training iterations T, the number of evaluation images for calculating metrics K, the bit threshold S., the parameter size factor 1,, and the number of parameters of the ich layer N,.
In the first stage, we aim to obtain quantized models by quantizing each individual layer. Given the full-precision SD-v1.5 UNet D, we consecutively perform the quantization on every single layer to 1, 2. or 3 bits individually, while maintaining the remaining layers at FP32 format. Notice, to align with our experiments, we add the balance integer and initialize the scaling factor with our alternating optimization. For each quantized model, the weights and scaling factors are fine-tuned using quantization-awan training to minimize the quantization error by learning the predicted noise of the SD-v1.5. We obtain quantized models Di,,, i = 1, ... L, b = 1,
In the second stage, we measure the quantization error of each layer by calculating various metrics from comparing images generated by the quantized model Di, with those from the unquantized SD-v1.5 D. Specifically, we generate K 100 baseline images Ia from the full-precision SD-v1.5 model with PartiPrompts. Then, for each quantized model Dj,,, we use identical prompts and seed to generate corresponding images I, We calculate the quantization error by measuring the metrics including MSE, CLIP score, PSNR, and LPIPS using these images and prompts.
In the third stage, we collect the mixed-precision recipe. We first compute sensitivity score for each layer, factoring in both the MSE and the parameter size adjusted by For the izh layer, its sensitivity score for the b-bits (b E (1,2,3) is defined as S. = M,,,NN where M denotes the MSE error, N is the total number of parameters of the layer, and 17 E [0, 1] denotes the parameter size factor. To determine the bit width (i.e., b²) for each layer, we define a sensitivity threshold as S., and the ieh layer is assigned to bū- where bj min(b]S.,6 < S.). The remaining layers are bits if they fail to meet the threshold. After determining the initial bits based on the MSE error, we refine this recipe by considering the degradation in the CLIP score associated with each bit-width. We simply consider the CLIP score change at 3 bits. We assign layers with the highest 10%, 5%, 2% CLIP score drop with 1, 2, 3 more bits, respectively.
The final output is a mixed preccison recipe {b?}, i == 1,2, ... L, specifying the bit-width for each layer. Then, we set the first and last convolutional layers as 8 bits and pre-computing and caching the time embedding and projection layers.
C More Details for Time Embedding Pre-computing and Caching
In Sec. 4.1, we introduce "Time Embedding Pre-computing and Caching". Here, we provide more details for the algorithm. In the Stable Diffusion model, the time step 1. ,999] is transformed into a time embedding emb, through the equation emb, = e(t). where e(t) denotes the time embedding layer and emb, E IRdhe, In SD-v1.5, die = 1280. Then, for each ResBlock, denoted as R2 for i = 1, 2, .N. where N, is total number of ResBlocks with time projection layers, the emb; is encoded by time projection layers rz(.) by F,,, = r; (emb,)). Notice that rı(-) and e(-) are both linear layers. Finally, F,,, is applied to the intermediate activations of each R₃ via addition operation, effectively incorporating temporal information into the Stable Diffusson model.
As observed before [27]. time embedding and projection layers exhibit considerable sensitivity to quantization during PTQ on DM. To address this problem, existing work specifically pays attention to reconstructing layers related to time embedding [27]. In this study, propose more effective
18

---

Algorithm
Algorithm
Input: Candidate bits set 6 E {1, 2,3, SD-v1.5 model D. number of total layers (except for the time embedding. time projection, first and last convolutional layers), dataset X. training iterations T, number of evaluation images K, threshold So. parameter size factor number of parameters of the ish layer N;. 12. ,
Output: Mixed precision recipe (bī J
1: 1: Obtaining the quantized models.
2: for = 1 to 3 do
3: for i to I. do
4: Quantize the i-th layer to bits via Eq. (1) and proposed initialization methods Sec. 4.1 get model Da,s;
5:
for == to do
6:
Updating weights and scaling factors by minimizing the quantization error using quantization-aware training D.,s with data X:
end for 8: end for
9: end for
10: 2: Calculating quantization error metrics.
11: Generating images La via SD-v1.5;
for
3 do for 1 to do
13:
14:
Generating K images 4, via quantized model Da.;; Calculating MSE, M,, via images L.b and La; Calculating PSNR, Pt, via images Ti and Ta;
15:
16:
Calculating LPIPS, T,, via images I, and Ta;
18:
Calculating CL.P score drop, Ci,s via images T, and prompts; 19: end for
20: end for
Deciding the optimal precision.
22: Calculating sensitivity score S.,, = Mc
23: for = to do
24: bj + 4;
25: for b 1
26: S, < S. then
27:
Assign the i-th layer with bits with by <- by
28: end if
29: end for
30: end
31: Calculating score
score drop, Ci,3 and its Peb percentile Cy;
32: for = to I. do
33:
if C,, > Cgo then
34:
by <- +
35:
end if
36:
if C.,3 > Cas then
37:
D + b, + 1;
38:
end if
39:
if C..3 > Cas then
40:
bj + bç + 1;
41:
end if
42:
end for
method. We observe that 1) during the inference stage, for each time step t, the emb; and consequently F., remain constant. 2) In the Stable Diffusion model. the shape of Fr, are considerably smaller compared to time embedding and projection layers. Specifically, in SD-v1.5, F;, is with the dimension in {320, 640, 1280} which largely smaller than time projection layers W,, E Dx1200 where D E {320, 640, 1280]. Therefore, introduce efficient and lossless method named Time Embedding Pre-computing and Caching. Specifically, for total Tiat inference time steps, we opt to store only Tint time features, rather than retaining the original time embedding layers e(-) and the time projection layers in the i-th ResBlock r.(.).
The inference time steps are set as 50 less in most Stable Diffusion models. This method signif- icantly reduces more than 1280/50 25.6x storage requirements and entire computational costs in terms of time-related layers. Given that the storage size of the pre-computed Fú, is substantially
19

---

smaller than that of the original linear layers, this approach effectively diminishes the average bit of our quantized model without any performance degradation.
D Analysis of Symmetric Weight Distribution
In Sec. 4.1, introduce " "Adding Balance Integer" by assuming the weight distribution in Stable Diffusion is symmetric. Here, we provide more analysis for the assumption. To verify the weight distribution is symmetric around zero in SD-v1.5, measure the skewness of the weight distribution of each layer. Lower skewness indicates more symmetric weight distribution. As illustrated in Fig. 8, 97% of layers exhibiting skewness between [-0.5, 0.5), this suggests that most layers in SD-v1.5 have symmetric weight distributions.
Skewness
0.5
0.0 Kand
-0.5
10
50
70
110 130 150 160 170 160 190 200 210 250 Layers
Figure 8: Skewness of weight distribution of each layer in SD-v1.5. Lower skewness represents the weight distribution more symmetric. 97% layers are with skewness between [-0.5, 0.5], indicating that most layers have symmetric weight distribution in SD-v1.5.
E More Details for Quantization Error Across Different Time Steps
In Sec. 4.2, we introduce the "Quantization Error-aware Time Step Sampling" method. Here, we provide more details for measuring the quantization error from the predicted latent instead of the predicted noise. During the inference stage, the actual denoising step requires the scaling operation on the predicted noise in diffusion models. Therefore, directly calculating the quantization error via noise prediction is not accurate. Instead, we calculate the quantization error in the latent feature space. We derive the relationship of quantization error calculated from the predicted latent and noise as follows:
E Ex,x
[1|20,, (1,21, c) 20 (1, Be, c)]13],
v1 at Eo., (1, (6,20,
1
σa Eom. (1, Ze, c)
)
Ex,
21
2.
Vau
Vaa
Va
Ex,x
[
|| (1, 2z, c) -0 (1, Zes c)l{?
(9)
where ā; is the noise scheduler in [21].
20

---

Detailed Metrics for Quantization Error by Quantizing Different Layers
In Sec. 3.2, we calculate the various metrics for representing the quantization error when quantizing different layers. Here, we provide detailed metrics when quantizing each layer of SD-v1.5 to 1, 2, and bits.
Quantization Error (MSE)
6000
5000
4000
2000
1000
0
10
20
30
40
50
60
70
80
90
100
110
120
130
140
150
160
170
180
190
200
210
220
230
240
250
Layers
(a) MSE value caused by the 1-bit quantized layers in SD-v1.5.
Quantization Error (MSE)
3000
2500
2000
1500
1000
500
70
110
130 Layers
150
170
190
210
230
250
(b) MSE value caused by the 2-bit quantized layers in SD-v1.5.
Quantization Error (MSE)
2000
1500
500
20
40
90 100
110
130
150
160
170
180
190
200
210
230
240
250
Layers
(c) MSE value caused by the 3-bit quantized layers in SD-v1.5.
Figure 9: MSE value caused by the quantized layers in SD-v1.5..
21

---

Quantization Error (CLIP score)
0.04
0.03
0.01
0.00
10
20
30
40
50
70
80
90
100
110
120
130
140
150
160
170
180
190
200
210
220
230
240
250
Layers
(a) CLIP degradation caused by the 1-hit quantized layers in SD-v1.5.
0.02
Quantization Error (CLIP score)
0.01
0.00
50
100
120
210
250
Layers
(b) CLIP score degradation caused by 2-bit quantized layers in SD-v1.5.
Quantization Error (CLIP score)
0.01
0.00
10
20
30
40
50
70
90
100
110
120
130 Layers
140
150
160
170
180
190
200
210
220
230
240
250
(c) CLIP score degradation caused by the 3-bit quantized layers in SD-v1.5.
Figure 10: CL.PP score degradation caused by quantized layers in SD-v1.5.
22

---

Quantization Error (LPIPS)
0.3
0.2
0.1
0.0
10
20
30
50
60
70
80
100
110
120 130
140
150
160
170
180
190
200
210
220
230
240
250
Layers
(a) LPIPS value of the 1-bit quantized layers in SD-v1.5.
Quantization Error (LPIPS)
0.3
0.0
130 160 Layers
250
(b) LPIPS value of 2-bit quantized layers in SD-v1.5.
0.35
Quantization Error (LPIPS)
0.30
0.25-
0.20
0.15
0.10
0.05
0.00
10
20
30
40
50
70
90
100
110
120
130 Layers
140
150
160
170
180
190
200
210
220
230
240
250
(c) LPIPS value of the 3-bit quantized layers in SD-v1.5.
Figure 11: LPIPS value of quantized layers SD-v1.5.
23

---

Quantization Error (PSNR)
30
20
15
10
5
0
10
20
30
40
50
60
70
80
90
100
110
120 130
140
150 160
170
180
190
200
210
220
230
240
250
Layers
(a) PSNR value of the 1-bit quantized layers in SD-v1.5.
Quantization Error (PSNR)
15
10
0
20
30
40
50
60
70
80
90
100
110
130 Layers
120
140
150
160
170
180
190
200
210
220
230
240
250
(b) PSNR value of the 2-bit quantized layers in SD-v1.5.
Quantization Error (PSNR)
40
35
20
15
10
5
20
50
70
100
120 130
150
180
200
230 250
Layers
(c) PSNR value of the 3-bit quantized layers in SD-v1.5.
Figure 12: PSNR value of quantized layers in SD-v1.5.
24

---

G More Visualization for Quantization Error by Quantizing Different Layers
In Sec. 3.2, show the images for representing the quantization enror when quantizing different lay- ers. Here, we provide more visualization for demonstrating the different quantization errors caused by quantizing different layers bit. The quantized layers from left to right correspond the annotated layers at the bottom: SD-v1.5 w/o quantization, Down. 0 0 attn2. toq. Down owwn..00. 0.0. attn2. tok, Down. 0. attn2 tov, Down. 1. a.t.2. tok, Down 1 attn2. tok, Down. r.r.s. convi, Up. res. 2. convshortcut, Up.3. atttn........................... tok, Up. res 2. convshortcut.
Figure 13: Quantization demonstrated in generated images (via PartiPrompts) after performing 1-bit quantization on different individual layers.
25

---

H 1.99 Bits Mixed Precision Recipe
We provide 1.99 bits recipe in experiments. During the training and inference stage, add balancing integer to the n-bits values, resulting in log(2" + 1) bits. We calculate the average bits by Σ. 10 N where by is the calculated bit-width in the ith layer, N; is the number of weights of the ish layer, Nez is the number of parameters for pre-cached time features, and Nw is the total number of weights in linear and convolutional layers. We calculate the model size by integrating
N..
all other parameters as 32 bits. The index and name of each layer are listed:
down_ blocks. 0. a.t.eetiions... 00.rro.inr. 6
2 down_blocks
3 Crtckts...ateee atttenttoos www.ssorrerr atttm1 taattiito_q:
www.wttttttttttssoooom hown_blocks.o
/rg.o//nooooooooiiooooooooooooon 7
7 www._blocks.
8
down
10 org.geageoogggiiiioooooaaaaaaaaa
11 1e
12 oeu.tbbboccssooaaaaarrrttoooooea
13 oot.
14 0ot.opaooo...ttiiaareeeeaaaaaraa orteeeaeeeeeeoioio.oooooooooooon to_q:
15 out.aaaoooocctttaaaaaa.aaoooo.aa to_k: 4
16 oat.aaaaoooo...................n
17 orae
18 ora.opaoee...atttaaaaaaaaaaaaaaa
5
19 or
20 ort.opaeeo.ttttiaai.......aaaaar
21 ott.pp/eeeeffttttroooo..o/aaearr
22 ott.opaeeeeaaaaatacaec..oeeaeaaa
23 ott.op/wee..cttaaaaa......aa.aaa
24 ort.
25 doum
26 dom._blookk.OOrremettt.......2
27 dow._plocks...rreettt....../oo/
28 down_
29 dom.._bbocctt...ooooaaaaaeesesss
30 dbm._bboecs...attrttttt.....ppp.
31 ouw.bbac.....tttttaao......aaaaa
32 2ag.
3
33 Stg.apppee....aattccccooc..ceeea
4
34 oet.lpaeeee....atcccooooooooocrr
35 ort.ohauoos..taaaaarr.c.oeeeoerr
36 oct.l/alee...tcaaccooooooccoocra
37 ort.opaeeeeaaaaaaaaaaaoaaoaaaaar
38 ott. out
39 oaa.aaa...aaaa.aaaatraa.....a.aa 0.oojj 2
40 o
41 or
42 oor.
43 or to_k: 2
to_q: 2
44 45 org.aaaeeoeott.ii.....a.....a.on
46 htt.eaaeeeeiiiiiiiiiaaaaaaaaaaan 3
47 or 4
48
6
49 org.oa/noo.ooaa..aaaaaaaaaaaaaan or
50 o otg.op/lee....ccaaaoooo.acooaaaa
52 o
53 dom._block.....ctttttooo.......
54
26

---

55 down_blocks ott.:odd....rreeett....oooomm 3
56 wtw.:ss....eeeet...ooommmm noon_shhotcuu:: 7
57 dow..bboc......rrssss......oo.o
58 dow..pblcae....rree........... 2
59 www....booc.......oooaaaaaaee.p.
4
60 oow.:.aucccass22....r.rr........
61 De
to 3
62 ott.ppa.....a.aaaaaaa....aaaaaar
to__tt 2
63 ott.aaao......aaaaaaa.oooooa.oar to_v:
64 65 org.aaaeseoaaaraaaooooooooooaoaa
to out.0:
66 or
4
67 org.aaaaa..ota..a..a.....aaraaar 4
68 org.aaaae...ttaiiiiaaeooororraor
69 hrg.aeaeeee.iii..i...a..nanaaann
70 hrt.aaaaaooooiiiaaaaaaaaaaaaaaaa
71 2rt.
72 oom..bobccas/2atttttttoa....ppp.
73 ort.alaooooccc2acttooooooaotoerr to_q: 4
74 |aa.al/oee22222cctttccoccaocaaaa to_k: 2
76 oal.aaac................a.aaaaar 3
77 ora.eaagooooegara.a.oo....o..oor 4
78 orw.ooalooootttiiioaooooooooooin
79 ort. 4
80 ort.aaaeeeecciiiiooooo.oooooooen
81 ool.ooaoee..eeeiii..aaaaaaaaaaaa projj:
82 otg.00ae......aaaa.r...........r
83 obt..bbbecc....ttttto.cooo.p..p.
84 obw._
85 dow..aplcckkee.rreeees......aoa
86 oo
87 www.. 2
88 www..ppoooeee2..resstttt.....o..
89 dww.:aa.ooo.ee.22ooooaaaaaaoo...
90 dww..a.oocat....eaattttoo...oo. 1
91 dow..blocaa....reaatto....oorr 1
92 dow._bbocke...rrrset....... 1
93 dbt._bboookee33rrreeaaaaa.....o
94 wpw.bbllcckk...rrettttto...ooooo 2
95 wtw.bbickk....rrettt.......////
96 otw.://ccc.....eeret.....oo.ooar 1
97
1
98 wp..boock.....rentts.........
99 htt.bb/cc.....rrrrt....ooooooooo
1
100
2.
101 og.
102 wp..block.....remtta2.....o
103 wgg.abccct.e..mppappprrr.....oo.
104 upb.acccss....tttttttt.....pppp.
105 wtt.apaccc..taaata.ooooooooaoaaa
106 htt.:p/ccc....aaaaaaaaaaooaaaaaa
107 or
108 a
109 ogg.apae.....aaaaaaaaa...aaaaaoa
110 obgpapahhhheo.iaaaaarooaoaaaaaar
111 htt.ppaccec.aaaaaaaaaa..aaaaaaaa
112 hbg.a
113 htt.ppaeh.....aaataaooooooooococ proj: 2
115 opb..ca.....a.ccmmtcsss..pppppp.
116 og..Doeas....ccrrrrt............
117
httploormmeeobocceeeeeeeeeeeeeee blocks.0. attm1......... to_q:
118 aabloccss..attmtiiooss...aaaaee. transformeerrrrrrr...oooo blw.ooorrmee.e..b...oooooo blocks. 0. attni. aettii.toov:: 2
htwpa/forreeeboocceeeeeee blocks. attni. 2
119 up
27

---

120 ott.:eatcc...t..aaaaoaoooooo.ro.
121 o
t:
122 /ttpa//eeeiiiaaaaaaaa.o.ororrrrr 4
123 hrt.a/ac.....aaaaaaaaa..oo.oooor
124 ott.apaeeeeceeaiiioooooooaaaeaar
125 ott.ppahw.aaattaaaaaearaooaarrar
1
126 Ab
127 ugg.boeee_l.atttriiaaa..ppppppp_
128 gg.
wlo.a//sooooooeoooooooooo. 129 ogppocacslattttttooosssaaaarrrrr
130 og
131 htt.op/eeooattaaaaaaaaaaaaaaaaaa
132 aag_apaeeeeeaaaaaaaaaeeaoeeooeoa
133 org.:paeee....caaaaaaaaaaamaamaa
1
134 oag.apahe..aaaaaaaaaaaaaaaaaaaaa
135 ottpapacccaatataaaaaaaaaaaaaaaaa
136 org.:paceet...aaaaaaaaaaeaaaaaan
137 Sbt.ceaeeeaaaaattoooeeeteetttrra
138 org.:aas....tttttt....oooaaaaaan
139 wp..ooocts...atrttttio.......p.p
140 wg.boocts-..remmetss.....o11
141 wp.boocte...remmete...ccoo.
142 ogg.accca...reettttc...ooooo...o 3
143 wg.bbocks...remett...comm
144 wp.blocks....remmtt......o
145 c.g.bocces...eenntt.....ooooo.re
146 1p.b.octs....reeeess.....om
147 r
148 wgl.
149 upbbboccss...uppppeeees.......
150 op
151
2
152 ogg.opachh...ttttaaaaaaaaaaeooor
to_k: 2
153
154 ott.aeace.a....aaaaaoa.oooooooor
155 hgg.ghaghhhcc.caaaooooooooooooar 1
156 acg.apaeeec...aaaaaaaaaaaaaaaaar
157 arg
158 ort.o
159 ot 1
160
161 o.p.cccaaaa.aaattooo............
162 ogb
164 ott.ap/a..aaaaaaaaaaaaaaa.oaaaan
165 166 ott.pppp....................ooaa
167 orge
168 htt.aaa................a.oo.ooar
169 otg.pppp.........aaee.e...o.ooar
170 htt.a//ee...iiiii.i....aooa.ooaa
171 htt.p//oooooooiiiaaaaaoaoooooaoa
173 up 174 p.b.icccaaa.a.ttrriia...........
www.cko.....n............ to_q:
176 2 177 otg.ppaceeaccccccaaooaaaaaaaaaaa
2
178 htt.u//ccb..ctaatt.ooooooooaoaoo
179 ara orttepaceeaa2atttcccmomooeettood
180
181 up.blocka. orgp://c.cccttt..aa.....aaaaaaaa
182 2p.blocks.2aattenee atteatioosn baaloceeeeccacaaaoooaaaeeeeeeeen ora.ataaoooeeeeaaaooooooooo..oo.
183 up.blocks.2 2.tocktii.attetiioon.... transformer wlw.klloook.o.. Of
184 up_blocks. attentions 2, ttw.siorrrerrrrmrrrrmmmmmr. www..alcc......ff......o.. 2:
atteentions
Transformer_blockee
28

---

185 up_blocks .. att.ntionns 2. proj
186 up_blocks resnets. 0.
187 up_blocks
188 2..bloccss..rreeeetes conv2:
wtw.::eeeeeeee....oosooooooooo.. cony_shorttutt:
189
r. 1
190 up.blocks.2
conv2:
191 up.blocks. w.raccss...reemtta...... 1onv. shortcut: co......ooottttuttt
192 2. 2
193 2. rewmeeta
194
195 2.
196 ot
197
198 oag.:hhhhhcc.caaaaaaooooooooooon
199
200 ort.ppat.....tttiiiaaoaaaaaaaaar
201 org.aeaiei.gaattaaa...aaeeeeeeer
202 otg.apachc....atraaooooooommaora
203 ocg.ahhhhc.cctatttoooeooooooooon
204 @e0. 0:
205 ohg.apaeeecacaaaataaaeoeoeeaaeaa
206 ogg._pa..a....taaaaaaaaaaaaaaaar
207 op
208 ugg
209 otg.uuaceeetsttttatttaoooooooaaa to_k:
to_q:
210 ott.ppacee...tttttaaaoa.aaaarrrr
211 2ct.bbaessscttttteeo.rrr/oooooor
to
212 ott.ahas...tttttiooo...aaaaaaaer to_outto0 .0:
213 33t.acacec..ttttaooooo.tttooaoon
214 obg.ahahhhcaaaaaaaaaaaaoaaaaaaar
215 2b
216 [bg.ahachhccaaaaiaaaaaaaaaaaaaea
217 htg.phach.ccciiiiooaooaaoooooooa
proj:: 2
218 ott.oeessssttttttaaaooootttttttr
219 opb.acassB...tttrttta....ppp.po:
220 up ogg.bocce..B.attrrrrt.....pp....
opg wttttl wtt.ttt.tttttt..a.......... to_q:
222 Araecaas._aatttttto.o.22aaasrrsr a.ttl to_k:
223 up.
224 srg.acacces..atttrrrraaaaaaaaaaa
ogg.apaee...ttttaaaec.oooooooorr 0:
226 hag.llacseeattttcccaaaaaeeetttnn
227 oa
228 up_ o.g.aoaces...ttttaati...........
229 2aa.aaate...ctaaa.aaaaaa22ataaae
3
230 up
bo..llocck....atttttiioosss obt.a//oss....a.aooooo.oo..o....
231 sg..blocc.....attetiiiioo.......
232 up- up.blocks. ob
233 3l
234 opb.
235 sp.blocks...rretttt......m.
236 1p..bockk....rrenttt....cmm
237 up_ up bop.blocks...reemttt....mmm
238 bpp.blockk...reeentts2...co.
239 bpp.llckk.....reeeeee.....m
240 itp.bloccs...rretsss...mmm
241 mal_Dhooc...ttttttt.s........p..
242 aag.hhoc..ccttttta.ooooooaaaerre att.n..ttt................ at..:
243 ala.bcccc..rrrrtto.ooooooorrroor
wwwp::llcccck..tttt.... to_k: 1
244 otl.bbocc...cccn.....o.......a.. otwp://oorrrrbbbooo..oo.ooo
attnii.toov:
246 www.:kkatteetiinss...ooo atteratcioss......attfo......... totout..ooott....
245 mid_block.
247 sid_block. bllcck.attentions. wtw.ssorrer...looce......
248 hdd_block.attt attentions 0. transformer attn2. to_v:
249 mid_block, attentions 0, www.:oon........ooooooooooomm baansforrerrbbockee attn2 to_out.0:
29

---

250 mid_block. attentions. O.transfrrmrrrblooss .nt.00 o.proj: 2
251 mid_block. attentions..... trwnsformer_bbeeeeeeeoo bloc__lllb....oo ff net. 2:
252 mid_block attentions O.proj_out:
253 wlock..renete... conv1:
254 mid_
255 www.bllock.remett...ccoo
256
conv_in:
conv_out: 8
I Details for Evaluation Metrics
In Sec. measure the performance various metries such TIFA, GenEval, CLIP score and FID. Here, we provide more details for these metrics.
TIFA Score. TIFA v1.0 [26] aims to measure the faithfulness of generated images. It includes various 4K text prompts sampled from the MS-COCO captions [47], DrawBench [72], PartiPrompts [86], and PaintSkill [6], associated with pre-generated set of question-answer pairs, resulting in 25K questions covering 4.5K diverse elements. Image faithfulness is measured by determining if the VQA model can accurately answer the questions from the generated images.
EvaEval Score. GenEval [14] measures the consistency between the generated images and the description, including 6 different tasks: single object, two object, counting, colors, position, color attribution. All text prompts are generated from task-specific templates filled with: randomly sampled object names from MS-COCO [47]. colors from Berlin- Kay basic color theory, numbers with 2, 3, 4, and relative positions from "above", "below", "to the left of",, or "to the right of". We adopt the pre-trained object detection model Mask2Former (Swin-S-8x) [5] for evaluation.
CLIP score and FID. CLIP score measures measure the similarity between text prompts and corresponding generated images. FID is used to evaluate the quality of generated images by measuring the distance between the distributions In the main experiments, evaluation are measured based on MS-COCO 2014 validation set with 30K image-caption pairs [47]. We adopt ViT-B/32 model to evaluate the CLIP score our experiments.
features extracted from generated images and target images.
Human Evaluation
In Sec. 5, provide the human evaluation results. Here, provide more detailed human evaluation with category and challenge comparisons on PartiPrompts (P2), comparing Stable Diffusion v1.5 and BitsFusion, with the question: Given a prompt, which image has better aesthetics and image-text alignment? Our model is selected 888 times out of 1632 comparisons, indicating general preference SD-v1.5, which is chosen 744 times, demonstrating more appealing and accurate generated images.
J.1 Analysis on Categories
Illustrations, People, and Arts. Our model significantly outperforms SD-v1.5 in generating illustra- tions (77 wins out of 124), images of people (101 out of 174), and arts (45 out of 65).
Outdoor and Indoor Scenes. Our model also shows strength in generating both outdoor (73 out of 131) and indoor scenes (23 out of 40), suggesting better environmental rendering capabilities.
J.2 Analysis on Challenges
Complex and Fine-grained Detail: Our model excels in generating images with complex details (73 of 113) and fine-graines details (173 out of 312), suggesting advanced capabilities in maintaining detail at varying complexity levels.
Imagination and Style & Format: Our model also shows strong performance in tasks requiring imaginative (92 out of 149) and stylistic diversity (1 (118 out 204), highlighting its flexibility and creative handling of artistic elements.
30

---

Ours
SD-v1.5
Produce Plants
52
48
Animals
51.9
45.1
Indoor Scenes
57.5
42.5
Illustrations
62.1
37.9
People
Abstract
51
49
Outdoor Scenes
Artifacts
54.7
45.3
World Knowledge
44.9
55.
Food & Beverage
Vehicles
54.8
45.2
Arts
69.2
Figure 14: Human evaluation across particular categories.
Ours
SD-v1.5
Imagination
61.7
38.3
Simple Detail
46.1
53.9
Complex
64.6
35.4
Fine-grained Detall
55.4
44.6
Writing & Symbols
57.1
42.9
Basic
49.1
50.9
Properties & Positioning
57.1
42.9
Quantity
56.7
43.3
Style Format
57.8
42.2
Linguistic Structures
55.7
44.3
Perspective-
51.4
Figure 15: Human evaluation across particular challenges.
The strong performance in imaginative and artistic categories presents an opportunity to target applications in creative industries, such as digital and entertainment, where these capabilities can be particularly valuable.
K Evaluation on Different Schedulers
In the main experiments in Sec. 5, we leverage the PNDM scheduler to generate images. Here, we measured the performance of different schedulers, such DDIM [78] and DPMSolver [55], to demonstrate the generality and effectiveness of BitsFusion. We set 50 inference steps and fix
31

---

the random seed as 1024. As shown in Fig. 16, BissFusionconsistenlly outperforms SD-v1.5 with different schedulers.
0.82
0.82
TIFA Scores
TIFA Scores
0.80
0.78
0.78
0.76
0.76
SD-v1.5 1. (32 bits)
SD-v1.5 5 (32 bits)
0.74
Ours-ll (1.99 bits)
0.74
Ours-Il (1.99 bits)
3.5
4.5 5.5
7.5 8.5 CFG Scales
6.5
2.5
7.5 8.5 9.5 CFG Scales
5.5 6.5
(a) DDIM
(b) DPMSolver
Figure 16: TIFA scores comparisons between SD-v1.5 and BitsFusion, with different schedulers. schfeuuessseduulerrrs Left: TIFA scores measured with DDIM [78] scheduler. Right: TIFA score measured with DPMSolver [55] scheduler.
L Detailed GenEval Results
In Sec. 5, we provide the overall GenEval results. Here, we provide detailed GenEval results for further comparisons illustrated in Tab. 8.
Table 8: Detailed GenEval with different CFG scales.
Method
Overall
Single
Two
Counting
Colors
Position
Color Attribution
Guidance Scale 2.5
SD-v1.5
0.3589
0,9350
0.2626
0,2775
0.6043
0.0340
0,0400
Ours-I
0,9075
0.5426
Ours-Il
0.4024
0.8975
0.3859
0,2750
0.6979
0.0560
0.1020
Guidance Scale = 3.5
SD
0.3879
0,9400
0.3010
0.3275
(.6787
0,0300
0.1500
Ours-I
0,3650
0.9500
0,2808
0.2575
0.6277
0.0280
0.0460
Ours-Il
0.4370
0.9350
0.4727
0.3125
0.7340
0.0600
0.1080
Guidance Scale 4.5
SD-v1.5
0.4056
0,9700
0.3010
0.3200
0,7426
0.0340
0.0660
Ours-I
0.3851
0.9500
0.3091
0.3100
0.6574
0.0340
0.0500
Ours-Il
0.4516
0.9575
0.4788
0.3450
0.7723
0.0520
0.1040
Guidance Scale
5.5
SD-v1.5
0.4094
0,9750
0.3111
0.3325
0.7319
0.0400
0.0660
Ours-I
0.4039
0.9675
0.3232
0.3425
0,7000
0.0300
0.0600
Ours-11
0.7979
Guidance Scule 6.5
SD-v1.5
0.4224
0,9800
0.3293
0.3725
0.7447
0.0400
0.0680
Ours-I
0.4161
0.9675
0.3414
0.3350
0.7425
0.0360
0.0740
Ours-Il
0.9750
0.4990
0.3275
0.7957
0.0540
0.1160
Guiddacee
SD-v1.5
0.4262
0.9775
0.3313
0,3850
0.7596
0.0440
0.0600
Ours-l
0.4226
0,9775
0.3495
0,36110
0.7447
0.0360
0.0680
Ours-Il
0.4682
0.9800
0.5091
0.3300
0.8085
0.0680
0.1140
Guidance Scale
8.5
SD-v1.5
0.4271
0.9825
0.3273
0.7745
0.0320
0.0540
Ours-I
0.4269
0,9800
0.3616
0.3475
0.7702
0.0400
0.0620
Ours-Il
0.4747
0,9700
0.5111
0,3675
0,8213
0.0620
0.1160
Guidance Scale 9.5
SD-v1.5
0.4260
0.9825
0.3556
0.3825
0.7553
0.0280
0.0520
Ours-I
0.4190
0.9825
0.3495
0.3450
0.7447
0.0300
0.0620
Ours-Il
0.4736
0,9700
0.5192
0,3625
0.8277
0.0560
0.1060
32

---

M More Comparisons
We provide the prompts for the images featured in the Fig. 1. Additionally, we provide more generated images for the comparison.
M.1 Prompts
Prompts of Fig. from left to right are:
portrait of anthropomorphic cyberpunk amoking cigar, cyberpunk!, fantasy, elegant, digital painting, artatation, concept art, matte, sharp focus, illustration, art by josan Gonzalez
2. Pirate ship trapped in cosmic maelstrom nebula, rendered in cosmic beach whirlpool engine, volumetric lighting, spectacular, ambient lights, light pollution, cinematic atmosphere, art nouveau style, illustration art artwork by SemseiJaye, intricate detail.
3. tropical island, k, high resolution, detailed charcoal drawing, beautiful hd, art nouveau, concept art, colourful, the style vadym meller
4. anthropomorphic art of for wearing white suit, white cowboy hat, and sunglasses, smoking cigar, texas inspired clothing by artgerm, victo ngai, ryohei hase, artstation. highly detailed digital painting, smooth, global illumination, fantasy art by greg rutkousky, karl spitzweg
5. painting of lantina elder woman by Leonardo da Vinci details, smooth, aharp focus, illustration, realistic, cinematic, artatation, award winning, rgb unreal engine, octane render, cinematic light, macro, depth of field, blur, red light and clouds from the back, highly detailed epic cinematic concept art ca render made in Maya, Blender and Photoshop, octane render, excellent composition, dynamic dramatic cinematic lighting, aesthetic, very inspirational, arthouse.
6. panda mad scientist mixing sparkling chemicals, high-contras painting
7. An astronaut riding horse on the moon, oil painting Van Gogh.
red dragon dressed tuxedo playing chess. chess pieces are fashioned after robots.
M.2 Additional Image Comparisons
We provide more images for further comparisons. For each set of two rows, the top row displays images generated using the full-precision Stable Diffusion v1.5, while the bottom row features images generated from BitsFusion, where the weights of UNet are quantized into 1.99 bits and the model size is 7.9x smaller than the one from SD-v1.5. All the images are synthesized under the setting of using PNDM sampler with 50 sampling steps and random seed as 1024.
33

---

Figure 17: Top: Images generated from full- call-precision Stable Diffusion v1.5. Bottom: Images generated from BitsFusion. Prompts from left to right are: A person standing the desert, desert waves, gossip illustration, half red, half blue, abstract image of sand, clear style, trendy illustration, outdoor, top view, clear style, precision art, ultra high definition image; b; A detailed oil painting of an old sea captain, steering his ship through a storm. Saltwater is splashing against his weathered face, determination in his eyes. Twirling malevolent clouds are seen above and stern waves threaten to submerge the ship while seagulls dive and twirl through the chaotic landscape. Thunder and lights embark in the distance, illuminating the scene with an eerie green glow; in mists peers up from the cobble stone street at the imposing and dark gothic buildings surrounding it. an fashined lamp shines nearby. oil painting-; d: A deep forest clearing with a mirrored pond reflecting a galaxy-filled night sky, e: a handsome 24 years old boy in the middle with sky color background wearing eye glasses, it's super detailed with anime style, it's a portrait with delicated eyes and nice looking face; I: A dog that has been meditating all the time.
solitary figure shrouded
34

---

PIINT
Pn
Figure 18: Top: Images generated from full-precision Stable Diffusion v1.5. Bottom: Images generated from BitsFusion. Prompts from left right are: small cactus with a happy face in the Sahara desert:; b. A middle-aged woman of Asian descent, her dark hair streaked with silver, appears fractured and splintered, intricately embedded within a sea of broken porcelain. The porcelain glistens with splatter paint patterns in harmonious blend of glossy and matte blues, greens, oranges, and reds, capturing her dance in a surreal juxtaposition of movement and stillness. Her skin tone, light hue like the porcelain, adds an almost mystical quality to her form.; c: A high contrast portrait photo of a fluffy hamster wearing an orange beanie and sunglasses holding sign that says "Let's PAINT!"; d: An extreme close-up up of an gray-haired man with a beard in his 60s, he is deep in thought pondering the history of the universe as he sits cafe Paris, his eyes focus on people offscreen they walk he sits mostly motionless, he is dressed in wool coul suil coat with button-down shirt, he wears a brown beret and glasses and has a very professorial appearance, and the end he offers subtle closed-mouth smile as if he found the answer to the mystery of life, the lighting is very cinematic with the golden light and the Parisian streets and city in the background, depth of field, cinematic 35mm film.; poster of mechanical cat, techical Schematics viewed from front and side view on light white blueprint paper, illustartion drafting style, illustation, typography, conceptual art, dark fantasy steampunk, cinematic, dark fantasy; f: I want to supplement vitamin c, please help paint related food.
35

---

Top: Images generated from full-precision Stable Diffusion v1.5. Bottom: Images generated from BitsFusion. Prompts from left to right are: a: new cyborg with cybertronic gadgets and vr helmet, hard surface, beautiful colours, sharp textures, shiny shapes, acid screen, biotechnology. tim hildebrands, bruce pennington, donato giancola, larry elmore, masterpiece, trending on artstation, featured on pixiv, cinematic composition, dramatic pose, beautiful lighting. sharp, details, hyper detailed, hd. hdr. 4 k, 8 k;; b: portrait of teenage aphrodite, freckles, curly copper colored hair, smiling kindly, wearing an embroidered white linen dress with lace neckline, intricate, elegant, mother of pearl jewelry, glowing lights, highly detailed, digital painting, artstation, concept art, smooth, sharp focus, illustration, art by wlop, mucha, artgerm, and greg Rutkowski; c: portrait of dystopian cute dog wearing an outfit inspired by the handmaid V/ s tale ( 2 0 ( ). intricate, headshot, highly detailed, digital puinting, artstation, concept art, sharp focus, cinematic lighting, digital painting, art by arigerm and greg rutkowki,, ulphonse mucha, cgsociely; d: Portrait of man by Greg Rutkowski, symmetrical face, a marine with helmet, using a VR Headsel, Kubric Stare, crooked smile, he's wearing tacitcal gear, highly detailed portrait, scifi, digital painting, artstation, book cover, cyberpunk, concept art, smooth, sharp foccus ilustration, Artstation HQ; e: Film still of female Saul Goodman wearing a catmaid outfit. from Red Dead Redemption 2 (2018 video game), trending on artstation, artstationHID, artstationHQ: oil paining of robotic humanoid, intricate mechanisms, highly detailed, professional digital painting, Unreal Engine 5, Photorealism, HD quality, 8k resolution, cinema 4d, 3D, cinematic, professional photography, art by artgerm and greg rutkowski and alphonse mucha and loish and WLOP
Figure 19:
36

---

HAAYY
b
d
Figure 20: Top: Images generated from full- full-precision Stable Diffusion v1.5. Bottom: Images generated from BitsFusion. Prompts from left to right are: a: anthropomorphic tetracontagon head in opal edgy darknimite mudskipper, intricate, elegant, highly detailed animal monster, digital painting, artstation, concept art, smooth, sharp focus, illustration, art by artgerm, bob eggleton, michael whelan, stephen hickman, richard corben, wayne barlowe, trending on artstation and greg rutkowski and alphonse mucha, 8 k; b: background shows moon, many light effects, particle, lights, gems, symmetrical!!! centered portrait dark witch, large cloak, fantasy forest landscape, dragon scales, fantasy magic, undercut hairstyle, short purple black fade hair, dark light night, intricate, elegant, sharp focus, digital painting, concept arl, malte, art by wlop and artgerm and greg rutkowski and alphonse mucha, masterpiece; designer and musician, long haired attractive androgynous fluffy humanoid character design, sharp focus, weirdcore voidpunk digital art by artgerm, akihiko yoshida, louis wain, simon stalenhag, wlop, noah bradley, furaffinity, artstation hd. trending on deviantart; d: concept art of ruins of victorian city burning down by j. leyendecker, wlop, ruins, dramatic, octane render, epic painting, extremely detailed, 8 k; e: hyperrealistic Gerald Gallego as a killer clown from outer space, trending on artstation, portrait, sharp focus, illustration, art by artgerm and greg rutkowski and magali Villeneuve; f: low angle photo of a squirrel dj wearing on- ear headphones and colored sunglasses, stadning a dj table playing techno music dance club, hyperrealistic, highly detailed, intricate, smoke, colored lights, concept art, digital art, oil painting, character design by charlie bowater, ross tran, arigerm, makoto shinkai, wlop
seahorse fursona, autistic bisexual graphic
Images generated from cull-precision Stable Diffusion v1.5. Bottom: Images generated from BitsFusion. Prompts from left right are: a: a photograph of an ostrich wearing fedora and singing soulfully into a microphone; b: a pirate ship landing on the moon; e: a pumpkin with candle in it; d: a rabbit wearing a black tophat and monocle; e: a red sports car on the road; f: a robot cooking in the kitchen.
Figure 21: Top:
37

---

Figure 22: Top: Images generated from full-prees precisieciinn Stable Diffusion v1.5. Bottom: Images generated from BitsFusion. Prompts from left to right are: a: a baby daikon radish a tutu, b. baby penguin wearing blue hat, red gloves, green shirt, and yellow pants; woman with long black hair and dark skin; d: an emoji of baby penguin wearing a blue hat, red gloves, green shirt, and yellow pants; e: a blue sports car the road; f: a butterfly.
Figure 23: Top: Images generated from full-preet cell-preeisoon Stable Diffusion v1.5. Bottom: Images generated from BitsFusion. Prompts from left to right are: a: Helmet of a forgotten Deity, clowing corals, extremly detailed digital painting, in the style of Fenghua Zhong and Ruan Jia and jeremy lipking and Peter Mohrbacher, mystical colors, rim light, beautiful lighting, 8k, stunning scene, raytracing, octane, trending artstation; b: Jeff Bezos as female amazon warrior, closeup, D&D, fantasy, intricate, elegant, highly detailed, digital painting, arlstation, concept art, malte, sharp focus, illustration, hearthstone, art by Artgerm and Greg Rutkowski and Alphonse Mucha; Portrait of draconic humanoid, HD, illustration, epic, D&D,,fantayy painting, artstation, concept art, smooth, sharp focus, illustration, art by artgerm and greg rutkowski and alphonse mucha, monster hunter illustrations art book; d: [St. Georges slaying car adorned with checkered flag. Soviet Propaganda!!! poster!!!, elegant, highly detailed, digital painting, artstation. concept art, matte, sharp focus, illustration, octane render, unreal engine, photography]: e: fire breathing dragon
fantasy, intricate, elegant, highly detailed, digital
medieval hobbit home, ornate, beautiful, atmosphere, vibe, mist, smoke, chimney, rain, wet, pristine, puddles, waterfall, clear stream, bridge, forest, flowers, concept art illustration, color page, 4 k, lone mapping, doll, akihiko yoshida, james jean, andrei riabovitchev. marc simonetti, yoshitaka amano, digital illustration, greg rutowski, volumetric lighting, sunbeams, particles; f: portrait of well- dell-dressed raccoon, oil painting in the style of Rembrandt
38

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We propose BitsFusion that quantizes the UNet from Stable Diffusion v1.5 to 1.99 bits, achieving a model with 7.9x smaller size while even better generation quality than the original one.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We list the limitations in Section "Limitations" in Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
39

---

Answer:
[NA]
Justification:
We do not include theory assumptions and proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the details of the quantization strategy, quantization error met- rics, mixed precision recipe and implementation setting in the Method, Experiments, and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
40

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We plan to release our code and trained models to facilitate the research efforts towards extreme low-bits quantization.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / httppsscccppbccccccceecce herbblicggiidee CodeeSbbm ssi aasonnooicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide detailed experimental settings in the Experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Similar to other Stable Diffusion and guantization works, we report the results large-scale datasets without the error bar.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
41

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the experimental resources the Experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: / /ttp///eerrp//e//ee/eee/eeeeeeee hwwppuli//ttisssuieeeeeeoo.eeeee
Answer: [Yes]
Justification:
We conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We list the broader impacts Section "Broader Impacts" in Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
42

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not have such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We claim and cite the asset in the paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
43

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooell as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our research does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our research does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
44


Reviews:
[{"review_id": "qnqaCC2LEp", "reviewer": "Reviewer_WzcG", "summary": "This paper proposes a novel weight quantization method called \"BitsFusion\" for compressing the Stable Diffusion v1.5 model. The primary goal is to address the issue of large model sizes, which hinder the deployment of diffusion models on resource-constrained devices. The BitsFusion framework quantizes the UNet component of Stable Diffusion v1.5 to 1.99 bits, achieving a 7.9× reduction in model size while improving image generation quality. The approach involves mixed-precision quantization, novel initialization techniques, and an improved two-stage training pipeline. Extensive experiments on various benchmark datasets demonstrate that the quantized model outperforms the original full-precision model in terms of generation quality and text-image alignment. My detailed comments are as follows.", "questions": "Please check the weakness section.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The BitsFusion framework is an innovative approach to compressing the UNet component of Stable Diffusion v1.5 to 1.99 bits, achieving a significant reduction in model size while enhancing image quality by using mixed-precision quantization, novel initialization techniques, and an improved two-stage training pipeline.\n2. The paper provides a thorough analysis of quantization errors and develops a mixed-precision strategy based on this analysis, which contributes to the theoretical understanding of low-bit quantization for large-scale diffusion models.\n3. The results demonstrate the effectiveness of the proposed framework. The quantized model consistently outperforms the original full-precision model across various metrics, including CLIP score, TIFA score, and GenEval score. The model's ability to achieve a 7.9× reduction in size while maintaining or improving performance highlights its practical potential for deployment on resource-constrained devices.\n4. The manuscript is well-written and easy to understand, with clear explanations of the methodology and experimental procedures. Sufficient experimental details are provided, ensuring that the results can be reproduced by other researchers, which enhances the paper's credibility and utility.", "weaknesses": "1. The two-stage training pipeline, while effective, may introduce additional computational complexity and training time, which could be a drawback for some applications. However, the paper does not explicitly discuss the additional computational complexity and training time it introduces. To provide a more comprehensive evaluation of this method, it is recommended that the authors discuss the potential additional computational overhead introduced by the two-stage training pipeline and its impact on training time.\n2. The study demonstrates that there is a high correlation between certain metrics, such as MSE and PSNR, and uses these correlations to validate the effectiveness of their quantization methods. However, correlation does not imply causation. The study does not delve into why these metrics are correlated or explain the intrinsic mechanisms that cause certain layers to be more sensitive to quantization (e.g., why quantization affects some layers more than others, and what intrinsic properties of these layers make them more prone to quantization-induced errors). Understanding these mechanisms is crucial for improving the quantization process, as knowing why certain layers are sensitive can lead to more targeted and effective quantization strategies.\n3. The claim that the 1.99-bit quantized model outperforms the full-precision model in all evaluated metrics is a significant overgeneralization. This assertion, based on specific experimental settings and datasets, may not fully represent the diversity encountered in real-world applications. Primarily, the evaluation on the MS-COCO 2014 validation set, though extensive, does not cover a wide variety of prompts, limiting the assessment of the model's ability to handle different types of inputs (e.g., complex scenes, abstract concepts, specific artistic styles). Additionally, the tests do not include various real-world conditions such as different lighting, backgrounds, or object complexities. Therefore, the claim that the 1.99-bit quantized model outperforms the full-precision model in all evaluated metrics is premature. A more robust evaluation including varied datasets, prompts, and real-world conditions, such as [A-E],  is necessary to substantiate such a broad claim.\n4. The discussion regarding model compression techniques is insufficient. It would be better for the authors to present more model quantization methods [F-J].\n\n[A] Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations. International Journal of Computer Vision. IJCV. 2017.\n[B]  Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models. ICCV. 2015.\n[C]  The Open Images Dataset V4: Unified Image Classification, Object Detection, and Visual Relationship Detection at Scale. IJCV. 2020.\n[D]  Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts. CVPR. 2021.\n[E]  Scene Parsing through ADE20K Dataset. CVPR. 2017.\n[F] Binary quantized network training with sharpness-aware minimization. Scientific Computing 2023.\n[G] Network Quantization with Element-Wise Gradient Scaling. CVPR 2021.\n[H] Single-path bit sharing for automatic loss-aware model compression. TPAMI 2023.\n[I] Generative Data Free Model Quantization with Knowledge Matching for Classification. TCSVT 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. About training computation.** \n\nA1. Thanks for the suggestions. For the stage-I training, we use 8 NVIDIA A100 GPUs with a total batch size of 256 to train the quantized model for 20K iterations. The training time is within 40 hours. \nFor the stage-II training, we use 32 NVIDIA A100 GPUs with a total batch size of 1024 to train the quantized model for 50K iterations. The total time is within 100 hours.\n\nNote that the stage-II training aims to further improve the performance of the quantized model so that it is better than the full-precision SDv1.5. If reducing the training cost would be the goal, we can remove stage-II training. Stage-I training is able to achieve similar performance as the full-precision SDv1.5 (Fig. 5 of the main paper).\n\nAdditionally, we compare training costs between our model and SDv1.5 from scratch (https://huggingface.co/runwayml/stable-diffusion-v1-5). We use the total trained samples to measure the training costs as the training hours depend on the actual hardware used. As in the following table, our stage-I training only requires 0.16% training cost compared with training the SDv1.5, and stage-II training requires 1.6% training cost compared with SDv1.5.\n\n>| Stage | A100 | Batch size | Gradient accumulation | Total batch size | 256x256 iters (K) | 512x512 iterations (K) | Total training samples (M) | Percentage of total training samples compared to SD-v1.5 |\n| :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- |\n| Stable Diffusion v1.5 | 256 | 4 | 2 | 2048 | 237 | 1304 | 3155.97 | 100% |\n| Stage-I | 8 | 8 | 4 | 256 | \\- | 20 | 5.12 | 0.16% |\n| Stage-II | 32 | 8 | 4 | 1024 | \\- | 50 | 51.2 | 1.6% |\n\n\n---\n\n**Q2.1. Why certain metrics are correlated.**\n\nA2.1. Thanks for the insightful comments! MSE, PSNR, and LPIPS are highly correlated since they measure the changes of image quality. Namely, they calculate the perceptual similarity between two images that align with human perception. More specifically, MSE quantifies the pixel-wise difference between images. Similarly, PSNR, derived from MSE, evaluates the peak error in images via $PSNR = 10 * log_{10}(Max^2 / MSE)$. LPIPS has a different calculation as MSE and PSNR. Yet, it still focuses on significant perceptual differences beyond pixel-level comparisons. Given that perceptual changes often accompany measurable pixel-wise changes, LPIPS correlates with PSNR and MSE.\n\nOn the other hand, the CLIP score measures the alignment between the image and prompt, leading to a low correlation with MSE, PSNR, and LPIPS metrics.\n\n---\n\n**Q2.2. Why quantization affects some layers more than others.**\n\nA2.2. We notice that cross-attention layers and convolutional shortcut layers are more sensitive to low-bit quantization than other layers. Cross-attention layers combine textual and image information. Therefore, any quantization-induced errors for cross-attention layers might greatly change the semantics and content of the generative images, leading to substantial semantic shifts and the loss of important content. For example, in Line 147~149 and Fig. 2 of the main paper, quantizing a cross-attention key layer changes the image content from \"a teddy bear\" to \"a person\". Convolutional shortcut layers link different layers and scales of features across the UNet. They facilitate the information flow between layers, enabling the network to learn complex patterns. Quantization of these shortcut connections disrupts the well-established connectivity of features, leading to a breakdown in the coherence of the learned representations (as in Fig. 2).\n\n\n---\n\n**Q3. Some evaluation datasets, like MS-COCO 2014, may not fully represent the diversity in real-world.** \n\nA3. We agree with the reviewer that most of the datasets used for evaluating text-to-image models have their limitations. In this paper, we evaluate our model on most of the *commonly used benchmark datasets* to get relatively accurate and fair comparisons. Specifically, we use Parti Prompts, TIFA, GenEval scores, and Human evaluation. These benchmark datasets cover a wide variety of prompts and are commonly adopted in evaluating text-to-image models in the literature, like SDXL. Additionally, we would like to kindly clarify that our human preference evaluation includes complex scenes, abstract concepts, and specific artistic styles (as in Fig. 13 and Fig. 14).\n\nRegarding our claim that our quantized model outperforms the full-precision SDv1.5 across all the evaluations, we intend to mention that our BitsFusion can outperform full-precision SDv1.5 across commonly used metrics evaluated in the paper. Based on the feedback from the reviewer, we will modify our writing on Line 61 as our model outperforms the full-precision model *across various evaluation metrics used in the paper*. We hope it can make the paper more rigorous.\n\nWe thank the reviewer for suggesting new metrics. We will discuss them in the revised paper. These metrics have not been widely adopted for evaluating text-to-image generation. For instance, Visual Genome is for visual question answering [A], Flickr30k Entities is for evaluating image description [B], Open Images Dataset V4 is for classification, detection, and visual relationship detection [C], and ADE20K is for understanding scene parsing  [E]. Adapting these metrics to evaluate text-to-image models requires a series of experiments to understand how to use them and correlate them with human perception. We take this as a future work.\n\nDuring the rebuttal, we run the suggested evaluation metric from Conceptual 12M with the CLIP score [D]. By randomly selecting 1K prompts (captions) and using the same CFG scale of 7.5, our 1.99 bits model outperforms full-precision SD-v1.5 (CLIP scores of 0.3084 *vs.* 0.3067).\n\n---\n\n**Q4. More related methods.** \n\nA4. Thanks for suggesting these papers, which are indeed insightful and relevant! We will discuss these papers in the revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eSPgMZjt5m", "reviewer": "Reviewer_tjXw", "summary": "This paper demonstrates a quantized diffusion model called BitFusion, which successfully quantize the Stable Diffusion (SD) v1.5 to 1.99 bits with 7.9x smaller size.  They first analyse the SD model in a layer perspective and assign the optimal bit based on the analysis. Then they propose a training pipeline to perform quantization-aware training (QAT) based on the analysis result. The experiment result shows that the quantized diffusion model BitFusion outperforms the full-precision SD v1.5 model.", "questions": "NA", "limitations": "The authors honestly point out their limitation: not quantizing VAE and text encoder and not quantizing activation. For the first limitation, it is quite common that diffusion model quantization research doesn't quantize VAE and text encoder according to the literature. But for the second limitation, the authors leave it for future work and doesn't address it. Activation quantization is recommended, as the fact that activation usually has higher quantization difficulty.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The result is significant. BitFusion successfully compressed the SD model while maintaining its performance.\n\n2. The authors successfully combine novel quantization techniques and existing quantization techniques, showing an effective way to quantize SD model to extreme low bit.\n\n3. The paper provides valuable observations about different layers' behaviour under quantization.", "weaknesses": "1. The analysis and experiment according to scaling factor initialization is not comprehensive. The authors propose alternating optimization for scaling factor initialization which is based on the MSE between quantized weight and full-precision weight. However, quantized weight initialization is well-studied and there are existing weight initialization technique that optimize the same objective. Nevertheless, there are some advanced initialization technique like adaround [1]. Those techniques are adapted by other Diffusion Model quantization research [2]. An experiment to compare the alternating Opt. and those initialization techniques is necessary.\n\n2. Stable Diffusion 1.5 is no longer the SOTA in Diffusion text-to-image model. Diffusion transformer (DiT) is the new trend in diffusion model research. The conclusion on UNet-based diffusion model like SD 1.5 might not be precise for the DiT diffusion model.\n\n3. FID is widely adapted by most of the diffusion model quantization research. Although the authors provide human evaluation and justify that FID is not accurate compared with human evaluation, the FID comparison between BitFusion and other baseline models should be included in the main body of the paper.\n\n4. In this paper [3], the authors propose a 1.58-bit LLM quantization approach comparable to Bit Fusion at model size reduction. However, they can perform activation quantization to 8-bit while Bit Fusion doesn't. \n\n\n[1] Nagel, Markus, et al. \"Up or down? adaptive rounding for post-training quantization.\" International Conference on Machine Learning. PMLR, 2020.\n\n[2] Li, Xiuyu, et al. \"Q-diffusion: Quantizing diffusion models.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[3] Ma, Shuming, et al. \"The era of 1-bit llms: All large language models are in 1.58 bits.\" arXiv preprint arXiv:2402.17764 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Comparison with adaround \\[1\\], which is used in Diffusion Model quantization research \\[2\\]**.\n\nA1. Thank the reviewer for suggesting the adaround, which is a relevant method \\[1\\]. However, we would like to kindly emphasize that adaround \\[1\\] focuses on *post-training quantization* by proposing a better weight-rounding operation, while our scaling factor initialization is used for *training-aware quantization*. These two technologies are applied during different quantization pipelines. As highlighted by Reviewer WzcG, our scaling factor initialization is novel.\n\nIndeed, as mentioned by the reviewer, the Q-Diffusion \\[2\\] uses the adaround, implemented by MinMax and Brute-force MSE searching. As analyzed in the paper (Line 198\\~201), MinMax is able to maintain the outlier weights but overlook the overall quantization error in the extreme low-bit setting, thus sacrificing the performance of the quantized model. To further prove this, we have conducted an ablation analysis in Table 4 of the main paper. With the alternating optimization (indicated by the row labeled “+Alternating Opt.”), the CLIP score is improved from 0.3054 (indicated by the row labeled “+Balance”, which uses MinMax as the default initialization method) to 0.3100. \n\nAs for the brute-force MSE, it takes lots of time to find the ideal scaling factor than our initialization approach. Following the same setting, we measure the CLIP score where the brute-force MSE in Q-Diffusion \\[2\\] can achieve a score of 0.3098, while our alternating optimization achieves 0.3100. However, the brute-force MSE requires 13 minutes to initialize the scaling factor by traversing 80 different scaling factors in each layer. In contrast, our alternating optimization significantly reduces the initialization time, which only needs 5 minutes with 10 optimization iterations.\n\nIn fact, we have compared our approach with Q-Diffusion \\[2\\]. As shown in Table 3 of the main paper, our 1.99 bits model can achieve a 0.3175 CLIP score, even outperforming the 4 bits Q-Diffusion model which has a CLIP score of 0.3137 evaluated on the PartiPrompts.\n\nThanks again for recommending the comparison with adaround. We will revise the manuscript to include the above discussion.\n\n---\n\n**Q2. Quantization of the diffusion transformer (DiT).**\n\nA2. We agree with the reviewer that besides UNet, DiT is another promising architecture used as the backbone for the diffusion model. This work targets the quantization of UNet for several reasons.\n\n- First, UNet is a widely adopted architecture for diffusion models. Besides the text-to-image generation, there are numerous applications like ControlNet and IP-Adapter that are built upon the UNet-based diffusion models. However, there is no previous effort to study the extremely low-bit quantization for such an architecture. Therefore, it is important to understand how to quantize the UNet to low-bits.  \n- Second, UNet is more efficient than DiT during inference, especially on resource-constrained devices. Using quantization to compress the efficient architecture has important practical values, as agreed by Reviewer WzcG and W2xs. Even if we apply the quantization to DiT models, how to deploy such models on resource-constrained devices is unclear, *e.g.*,  we are not able to run DiT-based text-to-image models on the iPhone.  \n- Third, although there is quite some work studying the improvement of DiT-based diffusion models, the UNet-based model still shows competitive performance for text-to-image generation. For instance, the open-sourced Kolors ([https://github.com/Kwai-Kolors/Kolors/tree/master?tab=readme-ov-file\\#-evaluation](https://github.com/Kwai-Kolors/Kolors/tree/master?tab=readme-ov-file\\#-evaluation)), which is built upon SDXL, achieves the SOTA results for text-to-image generation.\n\nWith all that being said, we believe that our approach can be applied to quantize DiT models, especially considering the quantization error analysis and training pipelines proposed in this paper are generic methods.\n\nThanks again for suggesting the quantization of DiT, which is an interesting topic that we would like to explore by using our proposed methods.\n\n---  \n**Q3. Adding the FID comparison to the main paper.**\n\nA3. Thanks for the suggestion. We will move the FID comparison from the Appendix to the main paper.\n\n---\n\n**Q4. About the activation quantization for the 1.58-bit LLM quantization approach.**   \n\nA4. This work mainly targets the size reduction of the diffusion model. Therefore, we did not show the activation quantization. Nevertheless, 8-bit activation quantization can be directly applied to our approach. Based on 1.99 bits weight quantization, we further apply 8-bit activation quantization to all layers except for the first and last convolutional layer and train the corresponding scaling factors. The experimental settings are the same as those described in Section 5 of the main paper. With a CFG scale of 7.5, our stage-I model and stage-II model can achieve CLIP scores of 0.3172 and 0.3193 on the PartiPrompts, respectively. For the TIFA score, our stage-I model and stage-II model can achieve 0.786 and 0.805 with a CFG scale of 7.5. With two-stage training, our BitsFusion model with a 1.99-bit weight and 8-bit activation quantization can still outperform the full-precision SD-v1.5.\n\n>| Methods | Weight Bit-width | Activation Bit-width | CLIP score | TIFA score |\n| :---- | :---- | :---- | :---- | :---- |\n| Stable Diffusion v1.5 | 32 | 32 | 0.3175 | 0.788 |\n| Stage-I | 1.99 | 8 | 0.3172 | 0.786 |\n| Stage-II | 1.99 | 8 | 0.3193 | 0.805 |\n\n---\n\n\\[1\\] Nagel, Markus, et al. \"Up or down? adaptive rounding for post-training quantization.\" International Conference on Machine Learning. PMLR, 2020\\.   \n\\[2\\] Li, Xiuyu, et al. \"Q-diffusion: Quantizing diffusion models.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023\\.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qaqgHdpZdT", "reviewer": "Reviewer_W2xs", "summary": "The paper presents BitsFusion, a novel method for weight quantization of diffusion models, specifically applied to the UNet architecture in Stable Diffusion v1.5. The approach quantizes weights to 1.99 bits, achieving a model size reduction of 7.9 times while enhancing or maintaining image generation quality. The experiment evaluates the quantized model across various benchmarks, including MS-COCO, TIFA, GenEval, and human evaluations, and demonstrates superior performance of the quantized model compared to full-precision Stable Diffusion v1.5.", "questions": "What's the total resource consumption for this entire process? For instance, how many GPUs with what memory capacity were utilized, and for how many hours was the training conducted? How does this compare to the resources required for a baseline that trains the original SD v1.5?", "limitations": "See weakness.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper is well-organized and presents its content in an accessible manner. The organization of the paper is clear.\n\n2.This paper mainly focuses on quantizing large-scale models like Stable Diffusion v1.5 with fewer bits than 4, which has significant implications for the industry and has achieved satisfactory results.\n\n3.The paper methodologically addresses the challenges associated with low-bits quantization in SD and provides a robust solution through the BitsFusion framework. The experiments demonstrate the effectiveness of BitsFusion in achieving superior performance compared to existing methods on various large-scale datasets.", "weaknesses": "1.The motivation of the paper seems to be heuristic and lacks necessary theoretical analysis, but the impact of this method in the industry is worth looking forward to.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. About the motivation of this paper.**\n\nA1. The overall motivation for performing quantization on the diffusion model is to reduce significant burdens for transferring and storing the model due to its large size. To this end, we propose several methods with corresponding motivations for model quantization:\n\nMixed-precision Strategy: \n- Mixed-precision Strategy: Various layers exhibit different sensitivities during quantization. To achieve a lower quantization error, it is crucial to employ a mixed-precision approach where more sensitive layers are quantized at higher bit-widths to retain essential information, while less sensitive layers can be quantized at lower bit-widths. In Section 3.2, we conduct the quantization error of layers for different bit-widths via appropriate metrics and obtain valuable observations to guide our mixed-precision approach.\n\nInitialization Schemes: \n- Time Embedding Pre-computing and Caching: During the inference, the embedding for a fixed time step is always the same (Line 177\\~179). Therefore, we can pre-compute the embedding offline and load cached values during inference, instead of computing the embedding every time. \n- Adding Balance Integer: We observe that weight distributions in the diffusion model are symmetric around zero. However, the existing quantization works disrupt this symmetric property (Line 182\\~189). Therefore, we introduce an additional value to balance the original values to maintain this property.\n- Scaling Factor Initialization via Alternating Optimization: Min-Max initialization leads to a large quantization error and the increased difficulty to converge in extremely low-bit quantization settings like 1-bit (Line 198\\~201). Therefore, we propose using an alternating optimization method to improve scaling factor initialization. \n\nTraining Schemes:\n- CFG-aware Quantization Distillation: Distillation is an effective approach to recover the performance degradation caused by quantization. Additionally, considering that CFG enhances the performance of the diffusion model, we leverage CFG-aware distillation to effectively recover the quantization error (Line 217\\~218).\n- Feature Distillation: To further improve the generation quality of the quantized model, we distill the full-precision model at a more fine-grained level (Line 220\\~221). Therefore, we adopt the feature distillation. \n- Quantization Error-aware Time Step Sampling: We analyze the quantization error during various time steps during training. As shown in Fig. 4 in the main paper, the quantization error does not distribute equally across all time steps in the quantized model (Line 230\\~240). Therefore, we propose a quantization error-aware time step sampling based on Beta distribution to reduce the overall quantization error. \n\nWe believe that our comprehensive analysis and the proposed methods will contribute significantly to the field of quantization on diffusion models.\n\n---\n\n**Q2. About training computation.**\n\nA2. Thank you for the suggestions. Here we provide more details for our two-stage training pipeline. For the stage-I training, we use 8 NVIDIA A100 GPUs with 40GB memory, and a total batch size of 256 to train the quantized model for 20K iterations. The total training time is within 40 hours. For the stage-II training, we use 32 NVIDIA A100 GPUs with 40GB memory, and a total batch size of 1024 to train the quantized model for 50K iterations. The total training time is within 100 hours.\n\nNote that the stage-II training aims to further improve the performance of the quantized model such that it is better than the full-precision Stable Diffusion v1.5. If reducing the training cost would be the goal, we can remove stage-II training and only use stage-I training. Stage-I training is able to achieve similar performance as the full-precision Stable Diffusion v1.5, as shown in Fig. 5 of the main paper.\n\nTo better understand the training cost of our model, we show the training cost comparison between our model and Stable Diffusion v1.5 (the training cost is obtained from the public resource https://huggingface.co/runwayml/stable-diffusion-v1-5). We use the total trained samples to compare the training costs as the training hours depending on the actual hardware used. As shown in the following table, our stage-I training only requires 0.16% training cost compared with training the Stable Diffusion v1.5 from scratch, and stage-II training requires 1.6% training cost compared with training the Stable Diffusion v1.5 from scratch. For the training time, based on the public information (Huggingface), SD-v1.5 is trained for 30-60 days on 256 GPUs, while our model takes 5~6 days on at most 32 GPUs.\n\n>| Stage | A100 | Batch size | Gradient accumulation | Total batch size | 256x256 iters (K) | 512x512 iterations (K) | Total training samples (M) | Percentage of total training samples compared to SD-v1.5 |\n| :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- |\n| Stable Diffusion v1.5 | 256 | 4 | 2 | 2048 | 237 | 1304 | 3155.97 | 100% |\n| Stage-I | 8 | 8 | 4 | 256 | \\- | 20 | 5.12 | 0.16% |\n| Stage-II | 32 | 8 | 4 | 1024 | \\- | 50 | 51.2 | 1.6% |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XQv2PhFaYr", "reviewer": "Reviewer_Fq1C", "summary": "This paper proposes a weight quantization method to quantize the UNet of SDv1.5 to 1.99 bits while maintaining model performance comparable to the floating-point model. The approach includes a series of techniques such as bit-width allocation for mixed-precision quantization, low-bit diffusion model initialization, and a two-stage training pipeline. Extensive experiments demonstrate the effectiveness of the proposed method. Results show that the quantized diffusion model can even outperform the floating-point SDv1.5.", "questions": "1. The setting of activation is not mentioned in the paper. Are the activations kept in floating point?\n2. What is the dataset used for stage-II training? Suppose the training set of the corresponding dataset is used. In that case, claiming the model outperformed the floating-point SDv1.5 model may be inappropriate since the SDv1.5 model is evaluated under a zero-shot setting.\n3. The average bits uses $\\log(2^n + 1)$ bits for calculation. Is it necessary to incorporate an optimal encoding method such as Huffman encoding to reach this average bit-width?", "limitations": "See weaknesses and questions.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow.\n2. The per-layer sensitivity analysis is comprehensive, considering both quality and contextual information, which is helpful for future research.\n3. The effectiveness of the proposed approach is validated across multiple datasets and evaluation metrics.\n4. The generated results look impressive.", "weaknesses": "1. It is well-known that QAT-based approaches can achieve superior performance at the expense of additional training effort. While this work involves training the weights via QAT, it does not include comparisons with other QAT-based methods such as Q-DM and TDQ.\n2. It is unclear how the sensitivity threshold in Section 3 is determined and whether this hyperparameter generalizes to other models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Comparison with QAT-based approaches such as Q-DM and TDQ.**\n\nA1. Thanks for the suggestion. We have compared the QAT-based approach LSQ and EfficientDM in Table 3 of the main paper. Here, we provide more results for Q-DM and TDQ on PartiPrompts with a CFG scale of 7.5. Compared to Q-DM with 2-bit weight quantization, our BitsFusion achieves a CLIP score of 0.3212, which is 0.228 higher than Q-DM. For a fair comparison with TDQ, which optimizes activation quantization through dynamic quantization, we also apply 8-bit activation quantization to our 1.99-bit BitsFusion. As shown in the table below, our BitsFusion achieves a CLIP score of 0.3193, surpassing TDQ with a score of 0.2907.\n\n>| Methods | Weight Bit-width | Activation Bit-width | CLIP score |\n| :---- | :---- | :---- | :---- |\n| Stable Diffusion v1.5 | 32 | 32 | 0.3175 |\n| Q-DM | 2 | 32 | 0.2984 |\n| BitsFusion | 1.99 | 32 | 0.3212 |\n| TDQ | 2 | 8 | 0.2907 |\n| BitsFusion | 1.99 | 8 | 0.3193 |\n\n---\n\n**Q2. How to determine the sensitivity threshold in Section 3.** \n\nA2. Thanks for the comment. We empirically set the sensitivity threshold to achieve an average of 1.99 bits. The sensitivity threshold only affects the average bits. It can be adjusted as needed for other models with different bit requirements, thus it is generalized to other models.\n\nWe would like to kindly note here the sensitive threshold is not a hyperparameter that influences the model training. Thanks for mentioning this. We will modify our manuscript accordingly to emphasize this distinction.\n\n---\n**Q3. Activation quantization.** \n\nA3. Thanks for the valuable question. Since this paper is focusing on reducing the model size, we explore the weight quantization on the diffusion model and keep activation as floating point value. However, 8-bit activation quantization can be directly applied to our approach. Based on 1.99 bits weight quantization, we further apply 8-bit activation quantization to all layers except for the first and last convolutional layer and train the corresponding scaling factors. The experimental settings are the same as those described in Section 5 of the main paper. With a CFG scale of 7.5, our stage-I model and stage-II model can achieve CLIP scores of 0.3172 and 0.3193 on the PartiPrompts, respectively. For the TIFA score, our stage-I model and stage-II model can achieve 0.786 and 0.805 with a CFG scale of 7.5. With two-stage training, our BitsFusion model with a 1.99-bit weight and 8-bit activation quantization can still outperform the full-precision SDv1.5.\n\n>| Methods | Weight Bit-width | Activation Bit-width | CLIP score | TIFA |\n| :---- | :---- | :---- | :---- | :---- |\n| Stable Diffusion v1.5 | 32 | 32 | 0.3175 | 0.788 |\n| Stage-I | 1.99 | 8 | 0.3172 | 0.786 |\n| Stage-II | 1.99 | 8 | 0.3193 | 0.805 |\n\n---\n**Q4. Datasets used in the stage-II training.**\n\nA4. Thanks for the question. Our training data consists of synthetic data and real images, and there is no overlap between the training data and evaluation data. To clarify, our model performs *zero-shot evaluation*, which is the same as SDv1.5. Specifically, we perform different evaluations to verify the effectiveness of BitsFusion, such as CLIP scores on the MS-COCO dataset and PartiPrompts, GenEval score, and TIFA score. PartiPrompts provides prompts from different categories. GenEval provides prompts to evaluate compositional image properties such as object co-occurrence, position, count, and color. TIFA provides various prompts and uses the VQA model to measure the text-to-image model if it can accurately answer the questions. Evaluating CLIP scores, TIFA score, and GenEval score do not include any reference images, and we do not have the same prompts from these datasets in the training. \n\n---\n**Q5. Is it necessary to incorporate an optimal encoding method such as Huffman encoding to reach the averaged bit-width?** \n\nA5. Thanks for the comment. We calculate the average bits according to Section I in the Appendix with $\\frac{\\sum_{i} log \\(2^{b_{i}^*}+1\\)* N_i + 16*N_{tf}}{N_w}$, where $b_{i}^{\\ast}$ is the calculated bit-width in the $i_{th}$ layer, $N_i$ is the number of weights of the $i_{th}$ layer, $N_{tf}$ is the number of parameters for pre-cached time features, and $N_w$ is the total number of weights in linear and convolutional layers.  We do not need the Huffman encoding to reach this average bit-width. However, as suggested by the reviewer, Huffman encoding is a valuable tool to further reduce our storage size.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their valuable suggestions and feedback. We appreciate the reviewers acknowledge the strengths of this paper, including:

- **addressing the challenges** for low-bits quantization of Stable Diffusion (Reviewer W2xs);
- **thorough, comprehensive, and valuable** analysis and observations of the quantization errors for each layer, contributing to the **theoretical understanding** of low-bit quantization for large-scale diffusion models and helpful for **future research** (Reviewer WzcG, Fq1C, tjXw);
- **innovative approaches** (e.g., innovative initialization techniques and improved two-stage pipeline) to compress the UNet to 1.99 bits (Reviewer WzcG, tjXw, W2xs);
- **significant results** for model size reduction and **superior performance** that the quantized model outperforms Stable Diffusion v1.5 across **various metrics and datasets** (Reviewer WzcG, tjXw, W2xs, Fq1C);
the quantized model has **practical potential** on resource-constrained devices and **significant implications** for the industry (Reviewer WzcG, W2xs);
- **a well-organized, well-written, and clear paper**, with experimental details provided, enhancing the paper's **credibility and utility** and ensuring the results can be **reproduced** (Reviewer W2xs, Fq1C, WzcG).

In the following, we provide detailed responses to all concerns from reviewers.

Meta-review: This paper proposes a novel weight quantization method called "BitsFusion" for compressing the Stable Diffusion v1.5 model. The primary goal is to address the issue of large model sizes, which hinder the deployment of diffusion models on resource-constrained devices. The BitsFusion framework quantizes the UNet component of Stable Diffusion v1.5 to 1.99 bits, achieving a 7.9× reduction in model size while improving image generation quality. The results demonstrate the effectiveness of the proposed framework.
Decision: Accept (poster)