De-biasing Diffusion: Data-Free FP8 Quantization of Text-to-Image Models with Billions of Parameters

Authors: Yaniv Blumenfeld, Daniel Soudry
Primary area: generative models
Keywords: Diffusion, Quantization, Floating-Point, Data-Free, Text-To-Image

Abstract:
Diffusion neural networks have become the go-to solution for tasks involving automatic image generation, but the generation process is expensive in terms of memory, energy, and computational cost. Several works have aimed to reduce the cost via quantization to 8 bits or less. Despite that, half-precision (FP16) is still the default mode for large text-to-image generation models --- where reducing numerical precision becomes more challenging. In this work, we show that the reduction of quantization bias can be more important than the reduction in quantization (mean square) error. We propose a data-free method for model quantization, with the goal of producing images that are indistinguishable from those generated by large, full-precision diffusion models. We show that simple methods like stochastic rounding can decrease the quantization bias and improve image generation quality with little to no cost. To close the remaining gap between full-precision and quantized models, we suggest a feasible method for partial stochastic-rounding of weights. When using the MS-COCO dataset as the baseline, we show our quantization methods achieve as good FID scores as the full-precision model. Moreover, our methods decrease the quantization-induced distortion of the images generated by the full-precision model, with the distortion decreasing with the number of diffusion steps.

Decision: reject
Reviewer scores: 6, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730431343426

thread_type:
official_review

<thread_object>
creator: reviewer_6a3r

<content>
summary:
This manuscript proposes a FP-based Diffusion Model quantization scheme. Specifically, the paper uses Stochastic Rounding, originally used for training, to assist the quantization calibration process. The technique uses PSNR as a measure to try and achieve calibration where images produced by a quantized denoising U-Net/Transformer are very similar to that of the full precision model. This approach is applied to SDXL and Flux.1.

strengths:
- Floating point quantization for diffusion models is positive direction.
- The paper is handling the larger models, SDXL and Flux.1. 
- The approach is generally intuitive and easy to understand.
- This paper builds on top of PTQD [1] in adjusting the denoising schedule, which is good as not a lot of diffusion quantization paper adopt its approach.
- Figure 2 is effective.

weaknesses:
- The primary similarity metric used by the authors for calibration is PSNR. They cite the SDXL [2] paper which states that FID has limitations due to Mean-Opinion-Scores. This is usually because the generates images are compared to real images (FID-GT). However, [3] propose FID-FP32, a variant of FID which compares the distribution of generated images from the quantized model, against those of the full precision model (which is treated as the baseline). If FID-32 is 0, the distribution of images from the quantized model is the same as that of the full precision model. Since this metric discards the weaknesses of the original FID, it is arguably a better metric for the objective than PSNR for what the authors aim to achieve, yet this paper does not seem to be aware of it.
- Most of the experimental results generate too few images (100 or 1k, e.g., Fig 1, Fig 3, Tab 2) to provide trustworthy and meaningful results. 
- Line 463-464 "the minimal recommended size" this is incorrect. See https://huggingface.co/docs/diffusers/en/using-diffusers/sdxl which states "anything below 512x512 is not likely to work".
- The paper only considers W8 precision, whereas other methods consider W4 [3].

questions:
See above weaknesses

References:

[1] He, Yefei, et al. "Ptqd: Accurate post-training quantization for diffusion models." Advances in Neural Information Processing Systems 36 (2024).

[2] Podell, Dustin, et al. "Sdxl: Improving latent diffusion models for high-resolution image synthesis." arXiv preprint arXiv:2307.01952 (2023).

[3] Tang, Siao, et al. "Post-training quantization with progressive calibration and activation relaxing for text-to-image diffusion models." arXiv preprint arXiv:2311.06322 (2023).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730550011956

thread_type:
official_review

<thread_object>
creator: reviewer_xmw8

<content>
summary:
This work proposes stochastic rounding of weights and activations to mitigate the bias incurred by quantization in the iterative denoising process. The introduced approach is evaluated on fp8 quantization of modern diffusion models - SDXL and FLUX.

strengths:
* Bias-variance decomposition of quantization error makes sense and provides some useful insights about the possible solution to reduce the quantization error, 

* The superiority of the proposed stochastic rounding for weights and activations compared to naive rounding-to-nearest is validated on several modern architectures and inference setups.

weaknesses:
* The overall novelty is limited. Stochastic rounding is an established technique in various applications [1, 2] and the current work adopts it for diffusion models.

* The evaluation setup is not very compelling. Unless the drop of generation quality is very significant, metrics such as FID do not reflect it. PSNR/SSIM between the generations produced by the original model and quantized one may be not a proper measure, as a compressed model can generate samples different from the teacher, yet of high quality. I would suggest adding user preference study at least for some cases as a reliable assessment. Specifically, given two images an assesor should tell, whether one image is better than other (or both are equal in terms of quality). 

* WSR(p=4) increases amount of memory used for model storage by 50% relative to fp8. I would suggest to check whether mixed fp8/fp16 quantization is still inferior in terms of quality. 

**Minor**

Line 84: PTDQ -> PTQD

---
[1] Gupta, Suyog, et al. "Deep learning with limited numerical precision." International conference on machine learning. PMLR, 2015.
[2] Wang, Naigang, et al. "Training deep neural networks with 8-bit floating point numbers." Advances in neural information processing systems 31 (2018).

questions:
* The paper focuses on fp8 quantization, where performance drops relative to uncompressed model are not very pronounced. Would be interesting to see whether the conclusions are valid for the case of more aggressive quantization (say W4/A8) - i.e one should focus on reducing the bias, rather than the variance term. I suggest having at least one experiments with stronger compression to showcase the difference between RTN performance and SR. 

* Noticeable decrease of FID metric relative to uncompressed model given high PSNR (small difference between the output of original and quantized model) looks suspicious. Do you have any explanation for this phenomenon? Do you adopt the same random seed for generation with compressed and original model? How robust is the FID value to the change of seed? 

* What is the sampler used for image generation? Is it the default sampler in diffusers config or DDIM?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730634233055

thread_type:
official_review

<thread_object>
creator: reviewer_76dq

<content>
summary:
The paper presents the study of FP8 quantization (in various special forms) for diffusion models in data-free setting; it overviews various techniques to overcome quantization error/bias of aforementioned schemes, and has actionable recommendations for practical use. The presentation of the paper, including overall writing, and other evidence (proofs, plots, and supporting information) is very informative and is one of the strengths of the paper.

strengths:
The paper studies the effect of fp8 quantization on diffusion models and presents various observations and recommendations for practical use. The authors convincingly show that main issue in FP8 quantization can be attributed to quantization bias and as such, propose to de-bias it with stochastic rounding schemes. Such a rounding scheme can be easily implemented for activations, but for weights it will be counter-productive; as such authors propose wight-stochastic-rounding (WSR) scheme that maintains FP8 weight copy + p-bit mantissa and certain uniform mixing scheme.  Authors  provide comparisons in terms FID/PSNR for every proposed de-biasing technique indicating that suggesting methods have merit and singling out M4E3 version of FP8 scheme as most suitable candidate for deployment.

The main strengths of the paper is thoughtfulness of the investigation and presentation: for every claim/equation in the paper there is sufficient empirical or theoretical explanation given in main paper or appendix.

weaknesses:
There are only few weaknesses of the paper, and most of them were mentioned by authors in the manuscript:
1. Practical applicability of some of the proposed schemes. In the current form, only the stochastic rounding of activations can be deployed on devices; any other schemes need significant effort in software/hardware support. SR itself gives measurable improvements, but not as good as SR+WSR (based on table results)
2. Assumptions on #diffusion steps used throughout manuscript (i.e., steps > 50) are unrealistic in my opinion simply due to the nature of discussion: if we are talking about efficiency and on-device deployment, such models are most probably will be deployed in few step regime
3. p=4 for WSR scheme might be too big (in my opinion) for any practical deployment despite the shown advantage: wsr with p=4 means 12bit storage for every weight tensor, as such, there might be alternative allocation or scheme for same 12bit per weight storage. I think, for any WSR benefits, authors should try coming up with less taxing scheme on additional storage.
3. User study. I think, having user study on perceived quality difference across proposed techniques would significantly strengthen the conclusions of the paper.

questions:
Please see weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730681032145

thread_type:
official_review

<thread_object>
creator: reviewer_j51z

<content>
summary:
In this paper, the authors introduce a de-biasing approach for quantizing large-scale text-to-image diffusion models to 8 bits. They identify and mathematically analyze the bias in the existing quantization methods, which negatively impacts quantization performance, especially for long denoising sequences. To mitigate this, they use simple stochastic rounding to reduce bias. Beyond applying stochastic rounding to activations, they extend this method to weights, introducing the concept of Stochastic Weight Rounding (WSR). Experiments on SDXL and FLUX demonstrate the effectiveness of the proposed approach.

strengths:
* The authors evaluate their method directly on FLUX, a state-of-the-art diffusion model, making the method practical for real-world use.
* In addition to FID, the authors report PSNR and SSIM metrics to compare their generated images with those from 16-bit models, showcasing the robustness of the proposed approach.
* The proposed method is supported by strong mathematical analysis.

weaknesses:
* As noted in lines 407–417, using stochastic weights requires storing additional $p$ bits. Therefore, for a fair comparison, I suggest that the authors compare their method to W$(8+p)$A$8$ baselines. It is possible that the observed improvement primarily results from the additional weight bits.
* Writing and Presentation:
  * In Figures 1, 2, and 3, there are too many curve lines in each figure, making them difficult to interpret. Additionally, some colors are similar, and some lines overlap, making it hard to distinguish between them.
  * The authors should include visual comparisons within the main text to better showcase their results.
  * To enhance clarity, the authors should consider adding a figure that provides an overview of their main method. Currently, it is hard to grasp the key ideas from the pure text.
  * Line 131: Consider replacing "U-Net" with a different term, as FLUX uses a diffusion transformer.
  * Figure 3: The left figure lacks a label for the y-axis.
  * Line 169: The quotation marks for M$x$E$y$ are incorrect.

questions:
* The concept of stochastic weights is unclear to me. Are the weights stored using $8+p$ bits while computations are still performed with 8-bit precision?
* In the paper, the authors apply their methods only to the 50-step FLUX.1-dev model. I wonder if the proposed method is also effective on the 4-step FLUX.1-schnell model.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732093047694

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal: General Comment

comment:
We thank the reviewers for their helpful and insightful comments.

All the reviewers acknowledged the benefits of quantizing the activations to 8bit (with SR) for its significant memory reduction. It is important to clarify that to benefit from this memory reduction in existing computation engines, both weights and activations must be quantized to a similar format. The main goal of WSR (as explained in lines 407-416) is to enable us to enjoy this benefit. Since the memory needed for the weights is much smaller in comparison to the activations memory, a direct reduction in weight memory is a secondary concern. Following the reviews, we understand that this claim may not have been sufficiently convincing and would like to add some real-world numbers to back it up.

The SDXL model has approximately 3 Billion parameters, which require a memory of 6GB for half-precision models, or 3GB for models with 8-bit weights. When running 8-bit SDXL models on H100 (80GB), A100 (40GB), or even RTX4090 (24GB), the amount of memory reserved for weights only amounts to a small percentage (3.75%, 7.5% or 12.5%, respectively) of the overall, available memory. The rest of the memory will be reserved for activations-memory, as the batch size may be increased to utilize all available resources. The weight/activation ratio is less drastic for FLUX models, but is still largely dominated by activations, at least for the commercial GPUs. (Flux-dev requires ~24GB with half-precision, and ~16GB with 8-bit quantization)

For reviewers who may still be concerned about the additional memory cost (e.g., for users running larger models or using smaller GPUs), we also added (in their individual comments, for now) results with more conservative implementations of WSR, showing that we can save around 750MB (SDXL) while maintaining the generated images’ quality. Other changes to the paper include the addition of FID-FP32 measurements (Appendix E and Figure 5d), presentation changes, and adding a new figure (Figure 14) describing our methods for an easier understanding of the compute-flow in its entirety.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a data-free FP8 quantization method for text-to-image diffusion models, emphasizing the importance of reducing quantization bias through stochastic rounding techniques like Stochastic Weight Rounding, evaluated on models such as SDXL and FLUX.
Its strengths include addressing a relevant cost issue in diffusion models and applying to advanced models with multiple evaluation metrics. However, weaknesses are evident. The novelty of stochastic rounding is limited, and there are concerns with the evaluation setup (insufficient image numbers in some experiments) and practical implementation (hardware support and memory usage). Overall, the significant weaknesses in novelty and evaluation prevent a convincing demonstration of superiority and practicality, leading to a reject decision.