DGQ: Distribution-Aware Group Quantization for Text-to-Image Diffusion Models

Authors: Hyogon Ryu, NaHyeon Park, Hyunjung Shim
Primary area: generative models
Keywords: Diffusion Models, Model Quantization, Model Compression, Efficient Models

Abstract:
Despite the widespread use of text-to-image diffusion models across various tasks, their computational and memory demands limit practical applications. 
To mitigate this issue, quantization of diffusion models has been explored. It reduces memory usage and computational costs by compressing weights and activations into lower-bit formats. 
However, existing methods often struggle to preserve both image quality and text-image alignment, particularly in lower-bit($<$ 8bits) quantization.
In this paper, we analyze the challenges associated with quantizing text-to-image diffusion models from a distributional perspective. Our analysis reveals that activation outliers play a crucial role in determining image quality. 
Additionally, we identify distinctive patterns in cross-attention scores, which significantly affects text-image alignment.
To address these challenges, we propose Distribution-aware Group Quantization (DGQ), a method that identifies and adaptively handles pixel-wise and channel-wise outliers to preserve image quality. Furthermore, DGQ applies prompt-specific logarithmic quantization scales to maintain text-image alignment. 
Our method demonstrates remarkable performance on datasets such as MS-COCO and PartiPrompts. We are the first to successfully achieve low-bit quantization of text-to-image diffusion models without requiring additional fine-tuning of weight quantization parameters. Code is available at \link{https://github.com/ugonfor/DGQ}.

Decision: accept (poster)
Reviewer scores: 6, 8, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1730104040749

thread_type:
official_review

<thread_object>
creator: reviewer_1kqn

<content>
summary:
This paper presents an analysis of feature distributions within layers and cross-attention scores in text-to-image (T2I) diffusion models. Building on these analysis, the authors introduce Distribution-aware Group Quantization (DGQ), a method specifically designed to address the distinct distribution patterns in T2I diffusion models. The proposed approach is validated across multiple datasets, demonstrating its effectiveness.

strengths:
- The paper's motivation is clear and straightforward.

- The authors effectively identify and address outlier issues present in prior methods, providing a targeted method.

- The analysis of attention scores for <start> tokens is convincing, and the proposed method appears to address these challenges well.

weaknesses:
- The authors validate the proposed method only on SDv1.4. These limited experiments restrict the ability to assess the generality of their approach. Additional analyses and experiments on other publicly available checkpoints (e.g., SD v2+, EDM, Imagen, etc) would strengthen the evaluation.

- Lines 127-129: "To the best of our knowledge, we are the first to achieve low-bit quantization (< 8-bit) on text-to-image diffusion models." However, [A] has previously proposed low-bit quantization for T2I diffusion models. While [A] and the submitted paper are concurrent works, a discussion of recent works would need for the future work.

- While the analysis and proposed method have novel elements, the approach relies heavily on existing methods (Group Quantization, logarithmic quantization). This reliance may worsen the novelty of the method in contrast to the insightful analysis.

[A] BitsFusion: 1.99 bits Weight Quantization of Diffusion Model, NeurIPS 2024

questions:
The major concerns are in the weakness section, therefore I currently feel that this paper is borderline.
I have some other questions in below:

- The authors analyze the <start> token's attention score distribution; however, I am curious about the distributions for other special tokens such as <end> or <pad> tokens and a comparison discussion with <start> would find meaningful insight within the T2I diffusion model.

- As I understand it, the proposed method (Attention-aware quantization) applies full precision to the <start> token while quantizing other components to lower precision. Consequently, in Table 2, I would expect slight differences in memory req and BOPs compared to the baselines. However, the current paper shows identical values for these metrics. Could you clarify the difference?

- The evaluation includes FID and IS metrics for assessing image qualtiy. I think other image quality assesments (IQA, [A, B]) against full-precision and baseline models would provide practical insights and could enhance the evaluation for the both datasets.

[A] MANIQA: Multi-dimension Attention Network for No-Reference Image Quality Assessment, CVPRW 2022
[B] PromptIQA: Boosting the Performance and Generalization for No-Reference Image Quality Assessment via Prompts, ECCV 2024
</content>
</thread_object>

</thread>
<thread>
creation date:
1730265157355

thread_type:
official_review

<thread_object>
creator: reviewer_s9vr

<content>
summary:
The paper proposes a novel quantization method for diffusion models, named DGQ, which efficient reduce memory usage and computational costs compared to existing methods without additional finetuning. This approach aims to achieve high-quality generation while maintaining performance metrics. Experiments are conducted on MS-COCO and PartiPrompts using IS, FID and CLIP evaluation metrics.

strengths:
1.It is interesting that the author finds activation outliers play a crucial role in determining image quality.
2.It is interesting that maintain text-image alignment in quantizing text-to-image diffusion models without requiring additional fine-tuning of weight quantization parameters
3.The logic of the work is clear, and the exploratory part of the experiments is rich in content

weaknesses:
1.The conclusions in Figure 5(b) should provide statistical results.
2.This work presents several new matrix computations, including $D_d$. However, the authors fail to address the extra time overhead that these computations may incur.
3.In Table 2, the proposed method with groups set to 16 underperforms compared to groups set to 8 on some metrics (e.g., IS and FID). It would be beneficial to analyze the reasons behind this occurrence.

questions:
Does excluding the <start> token mean dropping and zeroing it? Why does the author think that the peak near the value of 1 on the far right of Figure 5(a) corresponds to the <start> token, while other regions do not?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730385845223

thread_type:
official_review

<thread_object>
creator: reviewer_hfso

<content>
summary:
This paper proposes Distribution-aware Group Quantization (DGQ), which identifies and adaptively manages pixel-wise and channel-wise outliers to preserve image quality. Additionally, a prompt-specific logarithmic quantization strategy is introduced to enhance text-image alignment. Experiments conducted on the MS-COCO dataset demonstrate the effectiveness of DGQ.

strengths:
This paper exhibits several strengths:

1. The analysis of key characteristics necessary for maintaining model performance during the quantization process is scientific and persuasive.

2. The proposed group-wise quantization strategy, based on the distribution of activations, is both promising and innovative.

weaknesses:
This paper exhibits several Weaknesses:

1.	The figures are confusing and difficult to understand.

     a)   How are issues like "drops outlier," "high quantization error," and "high overhead" reflected in Fig. 3? What specific phenomenon indicates "drops outlier"? 

    b)    What does the horizontal axis in Fig. 4(b) represent? In channel-wise boxplot, why is the horizontal axis range of case1 from 0 to 900, while for case 2 it ranges from 1100 to 1190? What do "specific pixel" and "specific channel" refer to?

2.	The distinctions between group-wise quantization, layer-wise quantization, and channel-wise quantization are somewhat abstract and challenging to grasp. It would be beneficial for the authors to include a figure that illustrates the differences in their principles.

3.	There is a lack of qualitative and quantitative comparisons with the latest research, such as MixDQ[1].

4.	The base model used is outdated and singular. It is recommended to include experiments with more recent Text-to-Image models (e.g., SDXL, PixArt, SD3, or Flux) to validate the generalization of DGQ. 


[1] MixDQ: Memory-Efficient Few-Step Text-to-Image Diffusion Models with Metric-Decoupled Mixed Precision Quantization

questions:
1.	How can the quantized model in Table 2 outperform the original model? What mechanisms allow the quantization process to enhance performance?

2.	The meaning of W8A8 should be explained upon its first mention. The article lacks clarity on the notation WXAX, and similarly, the meaning of WXAXGX in the supplementary materials is not explained.

3.	It is recommended to add brackets around the feature indices in Formula 9 for improved clarity, such as A_{[:,0]} V_{[0,:]}.

4.	Why does reducing the bits of activation result in significant performance loss?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730453596434

thread_type:
official_review

<thread_object>
creator: reviewer_83s4

<content>
summary:
This paper deeply explores the unique distribution patterns of activations and cross-attention scores in text-to-image diffusion models. Based on the analysis of patterns, the authors propose a method called Distribution-aware Group Quantization, which is composed of outlier-preserving group quantization for activations and a new quantizer for cross-attention scores. Experimental results show that the proposed method outperforms the state-of-the-art methods and achieves under 8-bit quantization for the first time.

strengths:
1. This paper is well organized and the writing is good.
2. The in-depth analysis of the activation and cross-attention in the text-to-image diffusion models is interesting and may benefit the community.
3. The experiments are extensive and solid, showing the effectiveness of the proposed approach. The improvement is significant.
4. The idea is reasonable and easy to follow.

weaknesses:
1. Typos from line.326 to line.327: these lines contain repeated sentences.
2. The number of generated samples to compute FID and IS is not provided from line.403 to line.410.
3. The baseline model SD 1.4 may be out-of-date now. It would be better if results on more advanced models like SDXL were provided.

questions:
Please refer to the weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730828252569

thread_type:
official_review

<thread_object>
creator: reviewer_qna8

<content>
summary:
This paper addresses the challenges of quantizing text-to-image diffusion models, which suffer from high memory and computational costs. The authors propose a method called Distribution-aware Group Quantization (DGQ), which handles activation outliers and applies prompt-specific quantization scales to preserve image quality and text-image alignment.

strengths:
1. The writing is clear and easy to understand.

2. The experiments are thorough, and the analysis of outliers value offers valuable insights.

weaknesses:
1. The evaluation of inference performance is lacking.

2. In line 417, it states, “we set all attention score quantizer bits to match the activation bits.” Could this direct approach negatively impact performance? What is the FLOP cost of setting the attention score quantizer bits to 16?

3. As shown in Figure A.3, there is still a significant performance drop when activation quantization bits are below 8.

questions:
Refering to Weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732423818579

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear reviewers, 

We sincerely thank the reviewers for their praise regarding the writing of our paper(**qNA8, 83s4, 1KQn**). We also appreciate the recognition of our analysis and experiments by all reviewers (**qNA8, 83s4, Hfso, s9vr, 1KQn**), and their evaluation that our proposed method and ideas are reasonable, clear, promising, innovative, or effective(**83s4, HFso, s9vr**).

---
Furthermore, we would like to thank you once again for taking the time to review our paper and for providing constructive and valuable comments to improve it. We have updated the manuscript by addressing the points you raised. The revised parts are highlighted in blue. The main changes are as follows:

1. Added experimental results for SDXL-Turbo and corrected a mistake in the BOPs calculation for SDv1.4 (Table 2).
2. Included evaluation results using IQA and a human preference reward model (Appendix C).
3. Added a discussion on Quantization-Aware Training (Appendix D.2).
4. Provided an explanation of Quantization Granularity (Appendix E).
5. Added qualitative results on the impact of the bit-width of the attention score quantizer (Appendix F).
6. Included statistical results to support the conclusion of Figure 5(b) (Appendix G).
7. Added full visualization of Figure 4(b) (Appendix H).
8. Provided qualitative results for the SDXL experiments (Appendix I).
9. Made minor revisions, including correction of typos.

We are currently conducting experiments on additional aspects that we were not able to include yet and plan to integrate them into the final version. Thank you.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733297180051

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Rebuttal

comment:
Dear Reviewers and AC,

We sincerely appreciate your valuable time and effort in reviewing our paper.

As the rebuttal period is coming to an end, we summarize the major concerns raised during the review and outline how they were addressed, respectively.

---

**Experiments on an advanced model:** 

 - Reviewers **83s4**, **Hfso**, and **1KQn** recommended conducting experiments on advanced diffusion models. In response, we conducted additional experiments on SDXL-turbo. As a result, **our method significantly outperformed the existing baseline on SDXL-turbo**.



**Comparison with other methods(MixDQ, BitFusion):** 

- Reviewers **Hfso** and **1KQn** asked about comparisons with MixDQ[1] and BitFusion[2], respectively. However, since these methods are based on mixed-precision (MixDQ) and quantization-aware training (BitFusion), **we believe they are not suitable for a direct apples-to-apples comparison with our fixed-precision, post-training quantization approach.** We compared our method with a strong baseline (TFMQ-DM[3]) under the same settings as our method. The results show that our approach achieves significantly higher performance.



**Additional Overheads:** 

- There were comments about computational overheads incurred during $D_d$ operations (**s9vr**) and when processing the `<start>` token in full precision (**1KQn**). We clarified that **the $D_d$ operation is not performed during inference and thus it introduces no overhead**, and that the **overhead from processing the `<start>` token is only 0.0237 TBOPs, which is negligible**. Additionally, we are currently working on implementing the custom kernel ( as **qNA8**'s request ) and will add a comparison of actual inference performance to the final version. Furthermore, we will make our code publicly available.



**Evaluation with Additional Metrics:** 

- Reviewer **1KQn** recommended evaluating image quality using IQA methods. In response, we evaluated the image quality using IQA methods (MANIQA[4]) and a human preference reward model (ImageReward[5]). The results show that **our method outperforms the existing baseline in these metrics as well**.



**Details of Figures:** 

- Reviewers **Hfso** and **s9vr** requested the further information about some Figures (Figure 3, Figure 4(b), Figure 5(b)). In response, we have added detailed explanations in the appendix. Specifically, to better explain Figure 3, we included additional information about quantization granularity. For Figure 4(b), we included entire figure to the appendix. For Figure 5(b), R4 requested statistical results to support the conclusion. We have conducted additional experiments, confirming that the distribution of cross-attention indeed changes more dynamically.

---

During the rebuttal phase, we addressed most of the weaknesses and questions raised by the reviewers. As a result, reviewer **1KQn** raised their score from 5 to 8. Reviewers **qNA8** and **83s4** maintained their scores (6 and 8), both indicating a acceptance. Although two reviewers(**Hfso** and **s9vr**) did not respond, we would like to emphasize that we have clearly resolved the issues they raised.



We would like to highlight once again the contribution of our work. All reviewer recognized **our analysis and experiments** (**qNA8, 83s4, Hfso, s9vr, 1KQn**), and evaluated that **our proposed method and ideas are reasonable, clear, promising, innovative, or effective**(**83s4, HFso, s9vr**). Furthermore, it's **the first work to successfully achieve low-bit quantization of text-to-image diffusion models without requiring additional fine-tuning**.

---

We would like to express our sincere gratitude to all reviewers and AC for their efforts and time in reviewing our paper.

Best regards,
The Authors

---

[1] Zhao, Tianchen, et al. "MixDQ: Memory-Efficient Few-Step Text-to-Image Diffusion Models with Metric-Decoupled Mixed Precision Quantization." *arXiv preprint arXiv:2405.17873* (2024).

[2] Sui, Yang, et al. "BitsFusion: 1.99 bits Weight Quantization of Diffusion Model." *arXiv preprint arXiv:2406.04333* (2024).

[3] Huang, Yushi, et al. "Tfmq-dm: Temporal feature maintenance quantization for diffusion models." *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.

[4] Yang, Sidi, et al. "Maniqa: Multi-dimension attention network for no-reference image quality assessment." *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2022.

[5] Xu, Jiazheng, et al. "Imagereward: Learning and evaluating human preferences for text-to-image generation." *Advances in Neural Information Processing Systems* 36 (2024).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper discovered that the activation outlier and cross-attention scores plays an important role in image quality and alignment with texts in T2I models, and then proposed an effective approach (DGQ) to solve the quality degradation issue when quantizing diffusion models. All reviewers are consistently positive about this work (with two clear accept and three borderline accept) by appreciating the solid analysis and effectiveness of solution. Most concerns focus on a few aspects of evaluation and clarity of figures, which are addressed via detailed rebuttal from authors. While some reviewers did not respond, the AC helped go over the rebuttal and agreed that the proposed concern is mostly addressed. Overall, this is a good work by solving a known issue in model quantization with a simple yet effective training-free approach and of great value to community to push forward this direction. Considering all reviews and discussions, a decision of acceptance is made and authors are suggested to include all the suggestions in the final revised version.