DGQ: Distribution-Aware Group Quantization for Text-to-Image Diffusion Models
Authors: Hyogon Ryu
NaHyeon Park
Hyunjung Shim
Venue: ICLR 2025 Poster (2025)
Keywords: Diffusion Models
Model Quantization
Model Compression
Efficient Models
Primary area: generative models
TL;DR: We propose a distribution-aware group quantization method to preserve both image quality and text-to-image alignment in T2I diffusion models.

Abstract: Despite the widespread use of text-to-image diffusion models across various tasks, their computational and memory demands limit practical applications. 
To mitigate this issue, quantization of diffusion models has been explored. It reduces memory usage and computational costs by compressing weights and activations into lower-bit formats. 
However, existing methods often struggle to preserve both image quality and text-image alignment, particularly in lower-bit($<$ 8bits) quantization.
In this paper, we analyze the challenges associated with quantizing text-to-image diffusion models from a distributional perspective. Our analysis reveals that activation outliers play a crucial role in determining image quality. 
Additionally, we identify distinctive patterns in cross-attention scores, which significantly affects text-image alignment.
To address these challenges, we propose Distribution-aware Group Quantization (DGQ), a method that identifies and adaptively handles pixel-wise and channel-wise outliers to preserve image quality. Furthermore, DGQ applies prompt-specific logarithmic quantization scales to maintain text-image alignment. 
Our method demonstrates remarkable performance on datasets such as MS-COCO and PartiPrompts. We are the first to successfully achieve low-bit quantization of text-to-image diffusion models without requiring additional fine-tuning of weight quantization parameters. Code is available at \link{https://github.com/ugonfor/DGQ}.

Published as a conference paper at ICLR 2025
DGQ: DISTRIBUTION-AWARY GROUP QUANTIZATION FOR TEXT-TO-IMAGE DIFFUSION MODELS
Hyogon Ryu
NaHyeon Park Hyunjung Shim'
Korea Advanced Institute of Science and Technology (KAIST)
(hyogon.ryu) julialg, @ateshim@@kaissee
ABSTRACT
Despite the widespread use of text-to-image diffusion models across various tasks, their computational and memory demands limit practical applications. To miti- gate this issue, quantization of diffusion models has been explored. It reduces memory usage and computational costs by compressing weights and activations into lower-bit formats. However, existing methods often struggle to preserve both image quality and text-image alignment, particularly in lower-bit(<< 8bits) quan- tization. In this paper, we analyze the challenges associated with quantizing text- to-image diffusion models from a distributional perspective. Our analysis reveals that activation outliers play crucial role in determining image quality. Addi- tionally, we identify distinctive patterns in cross-attention scores, which signif- icantly affects text-image alignment. To address these challenges, propose Distribution-wware Group Quantization (DGQ), method that identifies and adap- tively handles pixel-wise and channel-wise outliers to preserve image quality. Fur- thermore, DGQ applies poopp-sppeiffi logarithmic quantization scales to main- tain text-image alignment. Our method demonstrates remarkable performance on datasets such as MS-COCO and PartiPrompts. We are the first to successfully achieve low-bit quantization of text-to-image diffusion models without requir- ing additional fine-tuning of weight quantization parameters. Code is available at httpp://ggtuub.cooo/goooooo/////
INTRODUCTION
Diffusion models (Sohl-Dickstein et al., 2015) have recently become key component of modern text-to-image models (Rombach et al., 2022; Ramesh al., 2022: Li al., 2023b; Podell et al., 2024), enabling the generation of high-quality images from natural language prompts. However, they often require a high computational workload, driven by iterative computations and significant memory costs. (Figure 1). This has limited their practicality in weal-world applications, particularly on resource-constrainee devices or in real-time generation (Kim et al., 2023; Ulhaq et al., 2022).
To reduce excessive computing resource us- age, model quantization has gained signifi- cant attention. It involves compressing weights and activations from floating-point formats lower-bit representations, thereby reduc- ing both memory usage and computational requirements. Numerous approaches (Shang et al.. 2023; Li et al., 2023a; He et al., 2023; Huang et al., 2024;
Model Size
BOPs
34,700T
3,438MB
2,555T
492MB
694T
198MB
al., 2024; He et al., 2024) have been proposed to quantize dif- fusion models while minimizing image qual- ity degradation. However, these methods often fail to maintain accurate text-image alignment, which is crucial for text-to-image models (Figure 2). While reducing the degradation in text-image
Text
U-Net
Image
Text
U-Net step
U-Net
Image Decoder
Encoder
Decoder
Encoder
50 steps
Figure 1:
Memory requirements and computa- tional cost of Stable diffusion v1.4.
"Hunjung Shim is corresponding author.

---

Published as a conference paper at ICLR 2025
Full Prec.
TFMQ-DM
DGQ
MS-COCO & FTD Scure (6-hit Activation)
Q-difitaion
TFMQ-DM
100
Full
DGQ
Prec
0.5
0.38
"Aphoto of the room with a yellow walls and white clock"
CLIP Score
MS-COCO CLIP & FID Score B-bit Activation)
Q-diftusion
DID Stotes
IEMQ-DM
Full
Prec.
DGQ
"A photo of'a dog"
0.73
0.7%
0.553
0.58
0.79
8.500
CLIP Score
Figure 2: The impact of DGQ. (a) Two types of performance degradation in text-to-image diffusion model quantization. DGQ preserves both text-image alignment (as shown above) and image quality (as shown below) significantly better than TFMQ-DM, Each model is quantized to the 8-bits setting (both weight and activation). (b) Performance comparison with other methods.
Layer-wise quantization with minmax
Layer-wise quantization with MSE
Chunnel-wise quantization with minmax
Group-wise quantization with minmax
Preserves outlier
Drops outlee Low quantization error Law overhead
Preserves outlier Low qumtization error High overhead
Preserves outlier
ligh quantization error
Low quatiizatoon eroor Low overhead
Low overhead
value
channel
Figure 3: Comparison of quantization strategies. We show layer-wise, channel-wise and group- wise quantization methods. Minmax and MSE (mean-squared error) are the most common strate- gies for calibrating the quantization scale, but both approaches struggle to effectively quantize the activation. The gray dotted lines represent the quantized values. Unlike layer-wise quantization, in channel-wise quantization, the quantized values are adapted to each channel. In group-wise quanti- zation, the quantized values adapted to groups, such as outliers or other channels. More detailed information about quantization granularity can be found in Appendix E
alignment after quantization has been less explored, it is essential considering the application sce- narios of text-to-image models. Most recently, Mixdq (Zhao et al., 2024) and PCR (Tang et al., 2023) have tried to quantize text-to-image diffusion models. These methods evaluated the sensitiv- ity of each layer (Zhao et al., 2024) or timesteps (Tang al., 2023) based on both image quality and text-image alignments, and allocate higher bit precision to more sensitive components, Both ap- proaches, however, rely on mixed precision which presents challenges for hardware implementation. Additionally, they didn't 't focus on bower-bit(below 8-bits) setting.
This paper investigates the quantization of text-to-image diffusion models, enabling hardware- friendly bower-bit precision quantization. While existing methods primarily focus on maintaining high image quality, our approach aims preserve both high image quality and text-image align- ment after quantization. We analyze the distribution of activations and identify key characteristics necessary for maintaining model performance during the quantization process. Firstly, we observe the presence of outliers in activations (C1), and recognize their critical role in preserving image quality. We also find that existing method using layer-wise quantization fail to retain these outliers. As result, their generation performance is largely degraded though quantization errors successfully minimized (Figure 3).
We also observed that the attention scores in the cross-attention layers form distinctive distribution (C2). As noted in many prior quantization studies on ViT (Lin et al., 2022; Li al., 2023c), atten- tion scores typically exhibit simple log-normal distribution. However, analysis revealed that,
2

---

Published as a conference paper at ICLR 2025
unlike self-attention, cross-attention exhibits two distinct peaks each corresponding <start> token and the remaining tokens, respectively. Existing methods disregard this distribution and em- play a uniform quantizer, resulting in inadequate quantization of smaller values logarithmic scale. Moreover, due to the presence of the <start> token, the quantization error for the remain- ing tokens becomes significant, leading degradation of text-image alignment. However, simply removing the <st art token significantly degrades image fidelity. Thus, we treat the <start> token separately, enabling us achieve high level of text-image alignment without compromising image quality.
Based on these findings, we propose Distribution-awee Group Quantization (DGQ) to address two key challenges in diffusion model quantization: (C1) outliers in activations and (C2) distinctive pat- terns in cross-attention, each having critical impact on image quality and text-image alignment. For C1, we introduce the outlier-preserving group quantization, which categorizes outliers into pixel- wise and channel-wise outliers. Then, for each outlier type, we form a group and apply customized quantization with group-wise scale parameters. To address C2, apply logarithmic quantization to the attention scores, except for <start> token, and use different quantization scales based on the input prompt. The attention score corresponding to <start> token is handled separately and maintained in full precision.
We tested method various datasets, including MS-COCK (Lin al., 2014) and Par- tiPrompts (Yu et al., 2022), and confirmed its superior performance in generating high-quality and text-aligned images. Our method achieved reduction of 1.29 in FID score compared to full pre- cision and an almost identical CLIP score (a decrease of only 0,001) on MS-COCO dataset, while saving 33.7% in bit operations (from 694 TBOPs 43.4 TBOPs).
To the best of our knowledge, we are the first to achieve low-bit quantization (< 8-bit) on text- to-image diffusion models (e.g., Stable Diffusion (Rombach et al., 2022)) without additional fine- tuning of weight quantization parameters.
In summary, our contributions are as follows:
We identify that text-to-image difffusion models exhibit unique patterns in activations and cross- attention scores, which lead to performance degradation in existing quantization methods.
We propose Distribution-awwe -wware Group Quantization (DGQ). It consists of outlier-preserving group quantization for activations and a customized quantizer for attention scores.
Our outtier-present preserving group quantization significantly enhances image quality after quantization, particularly in lower-bit settings. Meanwhile, customized quantizer for attentions facilitates high text-image fidelity.
Extensive experiments demonstrate our method outperforms existing approaches. On the MS- COCO dataset, we achieve an FID score of 13.15, which is even lower than the score for full precision. Furthermore, we are the first to achieve lower-bit quantization (under 8-bit) in text-to- image diffusion models without any additional fine-tuning.
RELATED WORK
Diffusion models can successfully generate high-quality images through an iterative denoising pro- cess. Combined with pre-trained language models, diffusion models have shown outstanding perfor- mance in text-to-image generations. The release of large diffusion models such as Imagen (Saharia et al., 2022), Midjourney', DALL-E2 (Ramesh et al., 2022), GLIDE (Nichol et al., 2021), Stable Diffusion (Rombach et al., 2022), and FLUX² has further accelerated advancements in the field of generative AI. However, the high memory and computational costs of these large diffusion models present challenges for practical use.
Model quantization is technique to reduce model size and improve inference speed by lowering the bit precision of the model's 's weights and activations. There are two primary approaches to model quantization: post-training quantization (PTQ) (Lin et al., 2024; Li et al., 2023a; Huang al., 2024; Tang et al., 2023; Lin et al., 2022; He et al., 2024; Shang et al., 2023; Nagel et al., 2020; Wei et al.,
'nttss//midjoummc.cco/// hhtp:://gthhuhcccm//llttfffsssss
3

---

Published as a conference paper at ICLR 2025
2022: Li et al., 2021) quantization-awie -ware training (QAT) (Bondarenko et al., 2021; Esser al., 2019; Jung et al., 2019). PTQ applies the quantization process after the model has been fully trained, requiring additional training. In contrast, QAT incorporates the quantization process during train- ing. It allows the model to adjust and maintain performance at lower precision. However, since QAT requires additional training time and resources compared PTQ. PTQ is often more suitable for quantizing large foundation models. Recent works have a quantization-awe aware fine-tuning (He et al., 2023; Wang al., 2024; Ryu al., 2024; Kim al., 2024; Dettmers et al., 2024) approach that slightly modifies QAT from-scratch and performs fine-tuning after PTQ. However, this ultimately requires additional computational cost. Our method has the advantage of not requiring such fine- tuning.
There have been studies on the quantization of diffusion models. These studies propose methods for quantization that take into account the timestep of diffusion models. Specifically, Q-Diffusion (Li al., 2023a) constructs calibration dataset by considering activation diversity across timesteps, while TFMQ-DM (Huang al., 2024) employs differently structured reconstruction block to better preserve temporal features. However, these studies focus solely on timestep-related character- istics without considering the text-condition. More recently, MixDQ (Zhao al., 2024) measured layer-wise sensitivity with respect to the text condition and quantized through mixed precision. PCR (Tang et al., 2023) introduced dynamic bit-precision mechanism depending the timestep. However, both methods rely on mixed precision, making practical implementation challenging. A similar approach ours, QuEST (Wang al., 2024), highlights the importance of activation out- liers and varies the quantization of weights. Different from these studies, our approach focuses on activation quantization, and both methods can be applied simultancously.
3 METHOD
In this section, we provide an overview of hardware-friendly quantization techniques and analyze the challenges of applying existing methods text-to-image diffusion models. To address these challenges, we introduce our approach, Distribution Group Quantization (DGQ). DGQ is specifically designed to preserve the unique characteristics of activations and cross-attention scores. As a result, it maintains high image quality even at lower-bit settings and significantly improves -it-tt--mmae alignment.
3.1 PRELIMINARY: HARDWARE-FRIENDLL QUANTIZATION
We briefly introduce two hardware-efficient quantization methods: linear (uniform) quantization and logarithmic (log) quantization.
Linear Quantization. Linear quantization maps Seating-point values to discrete integer levels uni- formly across the value range. The quantization and de de-quantization processes are defined as:
Zgg clamp (1= + z, 2,,02"
Idy
(xg 2) R2 x.
J, Zg. Jdg are the motting-point input, quantized input, de-quantized input, respectively. s, = and the quantization parameter(scale factor, zero-point, bit-width). This method is popular due to its simplicity and compatibility with standard hardware operations.
Logarithmic quantization. Log quantization utilizes logarithmic scale to handle values with wide dynamic range, particularly effective for exponential distribution. The quantization and de- quantization processes defined as:
Z4g = clamp (L logz 0,)
Idg s. 2 x.
This method benefits from efficient hardware implementation using bit-shifting operations.
3.2 ANALYZING CHARACTERISTICS OF TEXT-TO-IMAGE DIFFUSTON MODELS
To effectively quantize text-to- text-to-image models, we focus on analyzing them, particularly from a distri- butional perspective. Specifically, examine the activations and attention scores, which expect
4

---

Published as a conference paper at ICLR 2025
Original
Random
Outlier dropping
Pixel-wise boxplot
Channel-wise bexplot
"A mountain with snow-capped peaks rising high into the sky."
Casel: Outliers specific pixel
Maaliai Case2:
cat, eating a fish, with big smile its face."
Outhers on specific channel
(a)
(b)
Figure 4: Characteristies of activation outliers. (a) Comparison of dropping random values and dropping outlier values. (b) Two types of outliers are identified. These outliers often appear in spe-
cific channels or specific pixels. We provide full activation matrix visualization in Appendix H
to significantly impact on image quality and text-to-image alignment, repectively. Based on our investigation, identified several key characteristics.
We analyzed the importance of individual activation by examining the activation distribution of the diffusion model. Our analysis revealed that outliers (i.e., small fractions of activa- tions with either very high very low values) play a critical role in image generation. In Figure 4(a), we create three different images using the same text prompt with different activation manipulation. The first image represents the original generation (without any ac- tivations dropped), while the second and third images show exam- ples where single activation from each layer was set to zero. As illustrated in the figure, dropping random activations (that are not outliers) had minimal impact on the output image. However, drop- ping outlier activations resulted images with drastically different shapes and noticeably lower quality (Table 1). This indicates that certain activations, specifically the outliers, are crucial for image generation. Overall, our findings reveal that outliers significantly impact model performance, con- sistent with observations made in studies on large language models (Lin et al., 2024) and Vision Transformers (Darcet al., 2023). Unlike these studies, our work primarily focuses on the effects of activations, particularly the role of activation outliers.
Activation outliers play a crucial role in image quality.
Table 1: Comparison of dropping values.We used the original samples as ref- erence images and evaluated them on a randomly sam- pled set of 1,000 MS-COCO prompts.
Methods
PSNR1
LPIPS]
random drop
17.81
0.295
outlier drop
Outliers appear on a few specific channels or pixels. Then, where do outliers occur? We fur- ther trace the occurrence of outliers along various spatial dimensions. As shown in Figure 4(b), we confirmed that the outliers tend to appear in specific channels or pixels, rather than being evenly distributed. Furthermore, the locations of these outliers vary across different layers. This pattern per- sists even when the seeds and prompts are changed, suggesting that it is distinctive characteristic of text-to-imen text-to-image diffusion models. We speculate that this results from specific architectural choices and long pretraining without activation regularization (Bondarenko et al., 2021).
Cross-attention score corresponding to <start> token make distinct peak. Since attention scores are computed using the Softmax function, they typically follow a logarithmic distribution:
Attention Score Q, K); Softmax
where sij Q; K2.
Σ exp Pig!
Q E R E R i 1, 1, ny and = 1,2, Tk. my and nu denotes number of tokens, and d is the feature size. Since and k, are normally distributed in self-attention, dy is also nor-
va
mally distributed, and the exponentials exp
#i
follow a log-normal distribution. Consequently.
5

---

Published as a conference paper at ICLR 2025
Cross-atteation
<start> token
Self-attention
Cross-attention
Max: 2-0:3u
Max: 2-1400
Peak near
"An astronaut on the moon riding bike"
Excluding <start> token.
Max: -0.99
Max: 0.0
"two shoeps in field"
(a)
(b)
Figure 5: Characteristics of cross-attention scores. (a) The <start> token causes a peak near 1.0(Left). Background pixels tend to have high attention scores for the <start> token (Right). (b) Unlike self attentttnnio, maximum values of cross-attention scores change more dynamically.
in log scale, self-attention scores are approximated by a normal distribution. However, when ex- amining the distribution of cross-attention scores, distinct patterns emerge, where they clearly differ from that of self-attention scores. As shown in Figure 5(a), the first notable difference is the presence of peak near the <start (also referred as <bos>) token. To find the cause of this pattern, we analyze the attention score of the <start> token. We find that the background pixels tend to have high attention scores (almost close to 1.0) for the <start> token. Moreover, we empirically confirmed the role of these high attention scores of the <start> token. Adjusting attention scores to various levels (e.g., dropping to zero and clamping the second highest token's attention score) affects overall image details. (See Appendix A).
The distribution of cross-attention scores highly dependent on the input prompts. Addition- ally, analyzed the attention scores except for <start> token and observed that the distribution of the cross-attention scores are also different from those self-attention. An image input has fixed pixel size and locality, resulting in consistently widespread attention scores and a similar distribu- tion range across the input prompts. contrast, the number of text tokens relevant to specific pixels varies, causing cross-atted aatsettttntion scores to either concentrate disperse. As shown in Figure 5(b), this variation in distribution depends on the input prompts (Statistical results are in Appendix G). The high variation makes it difficult for static quantizer preserve large values, resulting in informa- tion loss in the content at the pixel level, which is important for text-image alignment.
3.3 DISTRIBUTION-AWIH
GROUP QUANTIZATION
Based on our findings on activation and cross-attention scores, we develop a novel quantization method tailored text-to-image diffusion models. Specifically, we suggest (1) outlier-peeservng group quantization for handling outliers in activations and (2) attention-aware quantization for ad- dressing patterns in cross attention.
Outlier- cellier-preserving group quantization. To maintain high image quality during quantization, it essential to preserve outliers while minimizing quantization error. However, most existing ap- proaches fail to meet both of these criteria at the same time. As shown in Figure 3, when outliers are preserved, the overall quantization error becomes excessively large. Conversely, if the quantization scale is optimized wise quantization can mitigate this issue, it introduces significant computational overhead and is ineffective at handling lexel-level outliers.
minimize mean-squared eиror, outliers often ignored. Although channel-
We propose outlier-preserving group quantization, that effectively reduces computational overhead while maintaining image quality. Our approach identifies the most efficient dimension for applying group quantization in each layer. After considering the activation range for vectors along the selected dimension, group them accordingly and create customized quantizer for each group. Specifi-

---

Published as a conference paper at ICLR 2025
cally, for each layer, identify the outlier type by examining the activation range across channels or pixels. Since outliers appear in specific channels or pixels, the difference in activation range in the dimensions, where outliers occur, is larger than in other dimensions. We determine the optimal dimension d E {channel, pixel) and apply quantization in a way that preserves critical outlier infor- mation. To achieve this, we define metric Da that measures the variability of activation values in each dimension:
Da= = max max min a
max
+ (max min min and
min Lid
(4)
min represent the maximum and minimum values of the i-th vector in dimension where a,'d and aid d, respectively. i is the index of the d-th dimension. This metric indicates the magnitude of the differences in activation value ranges across the channel and pixel dimensions
Then, the dimension d", where Da is large, is selected as the dimension to be used for grouping:
d" arg max Dd--
At the optimal dimension d², we divide the activation values into K groups, based on their range using K-means clustering. For each group, quantization scale 8% and zero-point 2 are calculated as:
max min
8k
minx,
(6)
2b
where k {l,.., Kj. X represents the activation belonging to the k-th group, and b denotes the number of quantization bits. Each group quantizes its values using the same quantizer. This approach adjusts the quantization scale according to the distribution range of the activation values within each group. This minimizes quantization errors and preserves outliers, effectively preserving image quality.
Finally, we consider the variance of activation that changes with the timestep of the diffusion model, we consider the same process for each timestep as described previous studies (He et al., 2023; Huang et al., 2024). The overall quantization scale S and zero-point Z for each layer are as follows:
S {su, $1, ST
{20, 21, .... 2T
Note that T represents the total denoising step. For Stable Diffusion v1.4, when T is set to 25 steps and 16 groups used, the additional memory occupied by the quantization parameter is 25 x 16 x 3008(byte) = 2.29MB. This is negligible(-~ 0.1%) compared to the UNet's memory requirement of 3,438 MB.
Attention-aware quantization. Cross-attention plays a crucial role in aligning text and images, as integrates text conditions into the image generation model (Zhao et al., 2024; Wang et al., 2024). As discussed in Section 3.2, the distribution of attention scores has several clearly different patterns from other activations. However, existing methods naively use uniform quantizer for handling these attentions. They therefore fail to preserve this distribution and leading to text-image alignment degradation. To address this problem, first, we apply a logarithmic quantizer to both self and cross- attentions. In this way, can preserve the small values in log scale while uniform quantizer cannot. Secondly, for cross-attention, we separate the forward path for the attention scores corresponding to the <start> token and the others. Then, we maintain the attention scores of <start> token and apply the quantizer to attention scores the others. Since the Softmax operation is normally performed in full precision, no additional dequantization is needed, making it efficient to implement in hardware. Third, since the range of cross-attention scores varies depending on the input prompt, employ dynamic quantization, which adjusts the quantization scale to the maximum value of the attention score excluding <start> token in inference-time Our quantization process can be expressed as the multiplication between quantized attention score Å R", X and quantized value E RBoxdd xd That is,
clamp (1-1883
A
] 0,,20 -1). where max max(A)).))
Ag [.1:
(8)
[.0) 8.2 A-aal,,,
Áv
|A1.0) V Vo..,, 2.2
[1:d]]
where A is the full precision attention score. nig n&. nhe are the number of tokens of query, key, and value, respectively.
7

---

Published as a conference paper at ICLR 2025
Model
Method
(its(WA))
Model Size
BOPs
MS-COCO
PurtiPrompts
FID
CLIP
Full Precision
32/32
3,438MB
823T
36,52
14.44
0.298
0.293
Q-Diff
871MB
51.41
27.65
26.12
0.273
0.275
TFMQ
878
871MB
51.4T
32.79
18.85
0.286
0.286
DGQ (#groups=8)
8/8
871MB
51.4T
35.38
13.26
0.297
0.292
DGQ (#groups=16)
871MB
51.4T
35.22
13:15
0.297
0.292
Q-Diff
8/6
871MB
38.61
4.12
221.76
0.080
0.120
TFMQ
38.6T
175.16
DGQ (#gmups=8)
8/6
871MB
38.6T
22.65
37.76
0.268
0.277
SD v1.4
DGQ (#grups==16 16)
38.6T
31.36
Q-Diff
478
436MB
25.7T
26.52
28.06
0.269
0.271
TEMQ
436MB
25.71
30.85
19.98
0.281
0.281
DGQ (#grops==8)
478
436MB
25.7T
33.91
13.28
0.294
0,289
DGQ (#groups=10)
4/8
436MB
25.7T
33.56
13.74
0.294
0,288
Q-Diff
436MB
19.3T
242.75
0.072
0,108
TFMQ
4/6
436MB
19.3T
5.24
229.64
0.127
0.155
DGQ (#group==8)
19.3T
51.94
DGQ (#gnups=16)
4/6
436MB
19.3T
22.17
43.66
0.263
0.274
Full Precision
32/32
10,269MB
6,9271
35.97
21.25
0.308
0.309
TFMQ
8/8
2,567MB
433T
12.24
1.6.69
0.067
0.069
DGQ (#groups=8)
8/8
2,567MBB
433T
34.79
22.46
0.299
0.294
TFMQ
2,567MBB
325T
163.02
0.025
SDXL Turbo
DGQ (Mgrups=88
8/6
2,567MBB
325T
28.56
34,31
0.251
0.223
(4 steps)
TFMQ
1,284MB
216T
13.00
109.56
0.068
0.069
DGQ (#groups=8)
4/8
1,284MB
2161
28.33
29.22
0.289
0.291
TFMQ
4/6
1,284MB
162T
1.99
270.45
0.022
0.049
DGQ (#group==8)
1,284MB
162T
22.93
45.00
0.245
0.226
Table 2: Quantitative Comparison. Results of different quantization methods MS-COCO and
PartiPrompts datasets.
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
Datasets, models and evaluation metrics. The dataset used for calibration during quantization was generated using 64 captions from the MS-COCO Dataset (Lin al., 2014). Similar to the ap- proach taken in Tang et al. (2023), we evaluated prompt generalization performance using the Par- tiPrompts (Yu et al., 2022) dataset, which differs from the calibration dataset. For the ext-tt--mage model, used Stable Diffusion v1.4. We measured FID (Heusel et al., 2017) and IS (Salimans et al., 2016) scores to evaluate image quality, and the CLIP score to evaluate text-image alignment. For main results (Table 2), we compute the FID and IS using 30K samples. For the ablation study (Table 3), we use 10K samples. Additionally, to evaluate computational cost, we measured BOPs (BOPs FLOPs bb -b -b)), where by and bg each stand for bits of weight and activation, respectively.
Baseline and implementation details. We use two sfate-of-the-art methods, O-Diffusion (Li et al., 2023a) and TFMQ-DM, (Huang al., 2024), as basclines for comparison. To ensure a fair evaluation, we employ the diffusers' library for both the baselines and our method. Unless specified otherwise, we apply 25 inference steps for computational efficiency. should be noted that Q-Diffusion and TFMQ-DM set the attention score's quantizer bits to 16 bits to avoid text- ext-imaae alignment degra- dation. However, in our implementation, to ensure a fair comparison, we set all attention score's quantizer bits to match the activation bits.
Weight quantization. Since our method focuses on activation quantization, we evaluated effec- tiveness by applying the same quantization methods to the weights of both the baseline and our method. Following previous studies Huang et al. (2024); Li et al. (2023a); Shang et al. (2023); Tang al. (2023), used BRECQ (Li et al., 2021) and Adaround (Nagel et al., 2020) for weight quan- tization. Block reconstruction were applied to both transformer and residual blocks. The calibration dataset used for reconstruction the same as that used for activation quantization. We collect the intermediate output with 64 captions from the MS-COCO dataset.
8

---

Published as a conference paper at ICLR 2025
Full Prec.
TFM
DGQ (W8A8)
Full Prec.
TFMQ-DM(W8AB)
DGQ (W8A8)
"A smiling teddy bear"
"A sports running Murs"
Full Prec.
TFMQ-DM(W4A6
DGQ (W46)
Full Prec.
TFMQ-DM((4A6) DGQ (W4A6)
"A basket full of apples"
"A mountain with snow-capped peaks rising high into the sky.""
Figure 6: Qualitative Comparison. Images in the top row generated with the W8A8 setting, and images the bottom row were generated with the W4A6 setting. WXAY represents weights and activations with X and Y bits, respectively.
4.2 MAIN RESULTS
We conducted experiments on the MS-COCO and PartiPrompts datasets, with the results presented in Table 2. First, on the MS-COCO dataset, all results indicate that our DGQ significantly out- performs existing methods. Specifically, with 8-bit activations, DGQ consistently delivers the best performance, regardless of the weight bit width. The FID scores were 13.15 and 13.28 for the W8A8 and W4A8 settings, respectively, surpassing the performance of the full-precision model (14.44). For text-image alignment, the CLIP score experienced only minimal drops (less than 0.005).
For the 6-bit activation setting, bascline methods failed to generate images, whereas DGQ success- fully produced images of acceptable quality. Although there was inevitable performance degradation compared to the full-precision model, our results showed significant improvement over the baseline methods. These improvements in the W4A6 setting) and the CLIP score (increasing from 0.155 to 0.274 in the W4A6 setting).
evident in both the FID score (reducing from over 200 to 43.66
Our method outperformed all previous approaches when evaluated the PartiPrompts dataset, which designed to assess prompt generalization performance. Despite the PartiPrompts dataset includes different types of prompts compared to the captions in MS-COCO, we achieved a high CLIP score, suggesting our successful preservation of text-to-image alignment. The qualitative re- sults can be seen in Figure
4.3 ABLATION STUDY
We performed an ablation study to analyze the impact of cach component of our proposed method.
The experimental results are presented in Table 3.
Effects of each component, In this section, we analyze the effect of each component of DGQ. We examined the impact of Outier-preserving Group Quantization and Attention-aware Quantization individually. and the results are shown in Table 3(a). For Outlier Outlier-preserveee preserving Group Quantization, group size of was used. For Attention-aware Quantization, we applied Log Quantizer, sep- arating the <start> token, and utilized dynamic quantization. Each component contributed to improvements in image quality and text-image alignment, with the performance boost from Outlier- preserving Group Quantization being particularly significant. The best performance achieved when both techniques were applied together.
Effects of grouping strategy. We investigated the effects of group size and dimension selection in outlier-preserving group quantization. As shown in Table 3(b), the best image quality was achieved when dimension selection was applied and the group size set to 2. Meanwhile, the best text- image alignment performance occurred when dimension selection was applied with a group size of 8. This suggests that increasing the group size does not always lead to better performance, and

---

Published as a conference paper at ICLR 2025
Dim.
# Groups
181
FID.
CLIP
(its(W/A)
Outlier
Attention
1S1
FID.
CLIP
X
1
31.34
18.83
0.286
8/8
X
X
31.34
18.83
0.286
X
2
32.45
16.74
0.290
8/8
X
32.76
17.00
0.296
X
4
33.01
16.54
0.294
8/8
X
V
31.18
18.24
0.287
X
8
32.61
16.79
0.294
8/8
V
33.61
14,40
0.297
x
16
32.07
16.91
0.294
8/6
X
X
6,45
176.44
0.144
2
32,94
16.24
0.292
8/6
x
18.20
50.04
0.250
4
33.03
16.64
0.295
8/6
7.17
207
8 32.64
16.99
0.295
8/6
21.88
40.15
0.267
16
32,42
17.15
0,295
(a) Effects of each component
(h) Effects of grouping strategy
(WA))
Quantizer
Seperate <atart>
1S
FID1
CLIP
8/8
Linear
X
31.34
18.83
0.286
8/8
Linear
V
31.44
18.78
0.285
8/8
Log (w/ dynamic quant.)
X
30.91
18.37
0.287
8/8
Log (w/ dynamic quant.)
V
31.18
18.24
0.287
8/8
Log (w/o dynamic quant.)
X
24.51
31.12
0.265
8/8
Log (w/ dynamic quant.)
V
24.60
28.79
0.271
8/6
Linear
X
18.20
50.04
0.249
8/6
Linear
17.64
49.98
0.249
8/6
Log (w/ dynamic quant.)
X
21.36
41.28
0.268
8/6
Log (w/ dynamic quant.)
V
21.88
40.15
0.267
(c) Effects of attention quantization.
Table 3: Ablation study. In (c), Due the low-quality image under the 8/6-bit setting, evaluation
was not possible. Therefore, we applied outlier-preserving group quantization beforehand to analyze the impact of attention-awar quantization.
there exists an optimal group size. We interpret this phenomenon as being influenced by the 8-bit environment used in the experiments, where the quantization scale is already sufficiently small. As result, increasing the group size further does not significantly enhance the precision of the activation representation. The drop in performance may be attributed to overfitting to the calibration dataset as the group size increases. In 8-bit settings, the best solution was to use 2 groups with dimension selection. However, we anticipated that larger group size would be more effective in lower-bit settings. Therefore, we set the default group sizes to and 16. As shown in Table 2, the results confirmed that in lower-bit settings, group size of 16 outperformed group size of 8 in both image quality and text-image alignment. More ablation study on 6-bit setting can be found in Appendix. B
Effects of attention-aware quantization. Table 3(c) illustrates the impact of each component of attention-aware quantization. The best performance was observed when applying all com- ponents(Log quantizer, separating <start> token, and dynamic quantization). Separating the <start> token resulted in a slight reduction in the FID score. For Log quantizer without dynamic quantization, the quantization scale determined using the running-minmax minmax method. It calculates the min-max range through the exponential moving average of multiple batches (Krishnamoorthi, 2018). This method is commonly used to calibrate the log quantizer in existing vision transformer quantization approaches (Li et al., 2023c; Li & Gu, 2023), but we find that it is not suitable for the cross-attention diffusion models. The log quantizer with dynamic quantization outperforms linear quantizer; in particular, on a 6-bit activation setting, it outperforms the linear quantizer by large margin.
CONCLUSION
In this work, we propose Distribution-awan Group Quantization (DGQ) for text-to-image diffusion models. We identify the crucial role of outliers in image quality and preserve them by grouping channels or pixels based on activation distribution. Furthermore, we uncover unique patterns in cross-attention scores and apply prompt-specific logarithmic quantization. DGQ outperforms exist- ing methods and, for the first time, enables low-bit quantization of text-to-image diffusion models without additional fine-tuning. By reducing computational costs while preserving both image quality and text-image alignment, our approach broadens the deployment of diffusion models in real-world applications, including edge devices.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We thank all CVML members for valuable feedbacks. This work was supported by Institute of In- formation & communications Technology Planning & Evaluation (IITP) grant funded by the Korea governmen((MSTT) (RS-2019-11190055 Artificial Intelligence Graduate School Program(KAIST) No.2021-0-02068 Artificial Intelligence Innovation Hub, No. RS-2024-00457882 National AI Re- search Lab Project), IITP grant funded by the Korea government(MSIT) and KEIT grant funded by the Korea governmen((MOTTE) (No. 2022-0-00680, 2022-0-01045), and Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the MSIP (NRF- 2022R1A2C301 1554, RS- RBS20023000219019,
REFERENCES
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Understanding and overcoming the challenges of efficient transformer quantization. arXiv preprint vrXiv:2109.12944 2021.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint vrXiv:2309.16566 2023.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024.
Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmen- dra Modha. Learned step size quantization. arXiv preprint arXiv:1902.0811 00000000881,.... 2019.
Yefei He, Jing Liu, Weijia Wu, Hong Zhou, and Bohan Zhuang. Efficientdm: Efficient quantization- aware fine-tuning of low-bit diffusion models. arXiv preprint arXiv:2310.032700 2023.
Yefei He, Luping Liu, Jing Liu, Weijia Wu, Hong Zhou, and Bohan Zhuang. Ptqd: Accurate post- training quantization for diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Yushi Huang, Ruihao Gong, Jing Liu, Tianlong Chen, and Xianglong Liu. Tfmq-dm Temporal fea- ture maintenance quantization for diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7362-7371, 2024.
Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning to quantize deep networks by optimizing quantization intervals with task loss. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 4350-4359, 2019.
Bo-Kyeong Kim, Hyoung-Kyy Kyu Song, Thibault Castells, and Shinkook Choi. Bk-sdm: A lightweight, fast, and cheap version of stable diffusion. arXiv preprint 1.XX::23005.5799 2023. URL https: //arxiv. www. abs /aoobs//3330 30..355.11700
Jeonghoon Kim, Jung Hyun Lee, Sungdong Kim, Joonsuk Park, Kang Min Yoo, Se Jung Kwon, and Dongsoo Lee. Memory-effiiient fine-tuning of compressed large language models via sub-4-bit integer quantization. Advances in Neural Information Processing Systems, 36, 2024.
Raghuraman Krishnamoorthi, Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint arXiv:1806.000 2018.
Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. Q-diffusion: Quantizing diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 17535-17545. October 2023a.
Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang. Fengwei Yu, Wei Wang, and Shi Gu. Breeq: Pushing the limit post-training quantization by block reconstruction. arXiv preprint ar2iv:2 2021.
11

---

Published as a conference paper at ICLR 2025
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. CVPR, 2023b.
Zhikai Li and Qingyi Gu. I-vit: Integer-only quantization for efficient vision transformer infer- ence. In Proceedings of IEEE/CVF International Conference on Computer Vision, pp. 17065- 17075, 2023.
Zhikai Li, Junrui Xiao, Lianwei Yang, and Qingyi Gu. Repq-vit: Scale reparameterization for post- training quantization of vision transformers. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pp. 17227-17236, 2023c.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for Ilm compression and acceleration. In MLSys, 2024.
Tsung-V Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer VisVon-ECCV 2014: 13th European Conference, Zurich, Switzelland, September 6- 12, 2014, Pro- ceedings, Part V 13, pp. 740-755. Springer, 2014.
Yang Lin, Tianyu Zhang, Peiqin Sun, Zheng Li, and Shuchang Zhou, Fq-vit: Post-training quantiza- tion for fully quantized vision transformer. In Proceedings the Thirty-Firs International Joint Conference on Artificial Intelligence, IJCAI-22, pp. 1173-1179, 2022.
Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. In International Conference on Machine Learning, pp. 7197-7206. PMLR, 2020.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arxiv:2112.10741 2021.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference Learning Representations, 2024. URL https: // /ttp:////peeeeeeeeeeemme net / wtwprrewwwweeeeeetrrrti....m
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Hyogon Ryu, Seohyun Lim, and Hyunjung Shim. Memory-efficient fine-tuning for quantized diffu- sion model. In European Conference Computer Vision, pp.
Springer, 2024.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- fion processing systemx, 35:36479-36494, 2022.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan. Post-training quantization diffusion models. Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pp. 1972-1981, 2023.
Junhyuk So, Jungwon Lee, Dachyun Ahn, Hyungjun Kim, and Eunhyeok Park. Temporal dynamic quantization for diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
12

---

Published as a conference paper at ICLR 2025
Jascha Sohl-Dickstein Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing. pp. 2256-2265 PMLR, 2015.
Yang Sui, Yanyu Li, Anil Kag. Yerlan Idelbayev, Junli Cao, Ju Hu, Dhritiman Sagar, Bo Yuan, Sergey Tulyakov, and Jian Ren. Bitsfusion: 1.99 bits weight quantization of diffusion model. arXiv preprint arXiv:2406.0433 2024.
Siao Tang, Xin Wang. Hong Chen, Chaoyu Guan, Zewen Wu, Yansong Tang, and Wenwu Zhu. Post-training quantization with progressive calibration and activation relaxing for text-to-image diffusion models. arXiv preprint (rXiv:2311.063: 2023.
Anwaar Ulhaq, Naveed Akhtar, and Ganna Pogrebna. Efficient diffusion models for vision: A sur- vey. arXiv preprint arX.022210.09292, 2022.
Haoxuan Wang, Yuzhang Shang. Zhihang Yuan, Junyi Wu, and Yan Yan. Quest: Low-bit diffusion model quantization via efficient selective finetuning, arXiv preprint arXiv:2402.03666 2024.
Xiuying Wei, Ruihao Gong, Yuhang Li, Xianglong Liu, and Fengwei Yu.
Qdrop: Ran- domly dropping quantization for extremely low-bit post-training quantization. arXiv preprint arXiv:2203.05770, 2022.
Jiazheng Xu, Xiao Liu, Yuchen WU, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Sidi Yang. Tianhe Wu, Shuwei Shi, Shanshan Lao, Yuan Gong, Mingdeng Cao, Jiahao Wang, and Yujiu Yang. Maniqa: multi-dimension attention network for no-reference image quality assess- ment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 1191-1200, 2022.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang. Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2(3):5, 2022.
Tianchen Zhao, Xuefei Ning, Tongcheng Fang, Enshu Liu, Guyue Huang, Zinan Lin, Shengen Yan, Guohao Dai, and Yu Wang. Mixdq: Memory-efficient few-step text-to- to-image diffusion models with decrre-d mixed precision quantization. arXiv preprint arXiv:2405.17873 2024.
Xingyu Zheng, Haotong Qin, Xudong Ma, Mingyuan Zhang, Haojie Hao, Jiakai Wang, Zixiang Zhao, Jinyang Guo, and Xianglong Liu. Binarydm: Towards accurate binarization of diffusion model. arXiv preprint arXiv:2404.000 .05622, 2024.
Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A survey on model compression for large language models. arXiv preprint arXiv:2308. 076303,300., 2023.
13

---

Published as a conference paper at ICLR 2025
A EFFECTS OF THE ATTENTION SCORE CORRESPONDING TO <START> TOKEN.
We analyze the effects of the attention corresponding to <start> token. For that, we adjust the attention scores, and compare the sampled images. We change the attention score of <start> token two ways, clamping and dropping. clamping sets the attention score to the maximum value of attention score corresponding to the other tokens(except <start> token), and dropping sets it O. Clamping is used check whether this can be excluded when determining the quantization scale of the quantizer, and dropping is used to check whether this can be excluded altogether. We confirmed that the <start> token doesn't change the main contents of the images, but affects on the details of the images. Therefore, in order to maintain full precision output as much as possible, <st aart>> token should be preserved.
Original
Clamped
Dropped
"A cat riding a bike"
"A monkey playing a guitar"
"A virus monster playing guitar"
"A painting of a car"
Figure A.l: Analysis on <start> token.
14

---

Published as a conference paper at ICLR 2025
B FURTHER ABLATION STUDY ON GROUPING STRATEGY
We conduct an ablation study to assess the impact of different grouping strategies on 6-bits setting. As shown Table A.I, increasing the number of groups generally improves model performance. Applying dimension selection consistently yields better results, and, unlike the 8-bit setting, more groups consistently improve model performance.
Table A.1: Effects of grouping strategy on 8/6 setting.
Dim.
Groups
1st
FID
CLIP
6.45
176.44
0.144
X
8.44
136.14
0.190
X
4
11.88
86.10
0.223
X
8
14.06
70.35
0.232
16
15.20
64.85
0.235
9.69
117.11
0.201
4
14.91
66.33
0.236
17.70
52.11
0.243
16
18.46
49.05
0.248
C EVALUATION ON VARIOUS METRICS
Considering the widespread usage image quality assessment (IQA) models and human preference reward models, we evaluate our methods on the IQA model MANIQA (Yang et al., 2022) and the human preference model ImageReward (Xu et al., 2024). The evaluation is conducted on 30K samples from the MS-COCC dataset. As shown in Table A.2, in almost all cases, DGQ significantly outperforms the baseline. With 8-bit activation settings, TFMQ achieves slightly higher performance with MANIQA, but the ImageReward model, DGQ achieves better results in all cases.
Table A.2: Quantitative comparison. MANIQA is the image quality assessment model and Im- ageReward is the human preference reward model.
Method
Bits(W/A)
MANIQAT
ImageReward
Full Precision
32/32
0.5525
0.1120
TFMQ
8/8
0.5359
-0.0375
DGQ (#grouuss=))
8/8
0.5340
0.0668
TFMQ
8/6
0.3795
-1.8643
DGQ #ggrouss88)
8/6
0.4187
-0.2724
TFMQ
4/8
0.5265
-0.1222
DGQ (fgrouss=8)
4/8
0.5223
-0.0256
TFMQ
4/6
0.3834
-2.0587
DGQ(#groups=
4/6
0.4282
-0.4630
D DISCUSSION
D.1 LIMITATION AND FUTURE DIRECTIONS
We summarize the current limitation and potential future works.
Combining with advanced weight quantization methods. Since methods concentrate the activation quantization, it would be able combined with other advanced weight quantization techniques such as EfficientDM, QuEST or the other quantization-aware training methods.
15

---

Published as a conference paper at ICLR 2025
More effective quantizer for attention scores. For cross- ross-attentiion the distribution range are deeply depends on the user input prompts. In DGQ, because of hardware- constraint, adjust the quantization scale the maximum value of remaining attention scores, but would be more effective methods such as using lookup table or reparameterization.
analysis reveals that
D.2 DIFFERENCE BETWEEN POST-TRAINING QUANTIZATION AND QUANTIZATION-AWALL TRAINING
Some other quantization methods (Zheng et al., 2024; Sui et al., 2024) achieve extremely low-bit quantization. However, they are based on Quantzation-Aware Training (QAT), which is a com- pletely different setting from ours (i.e., Post- Trainirainnn Quantization). BitFusion (Sui al., 2024) requires huge dataset and significant computational cost to obtain a quantized model. In contrast, DGQ (Ours) a model generated through PTQ(Post-Training Quantization) that does not require dataset, requires only 64 prompts, and has minimal computational cost. Specifically, according to the BitFusion paper, their model was trained for 50K iterations with a batch size of 1024 using an internal dataset, utilizing 32 NVIDIA A100 GPUs. On the other hand, DGQ used only 64 sample prompts during the activation quantization process and was completed in about 20 minutes on just one RTX A6000 (based Stable Diffusion v1.4 with 25 steps).
Generally, models quantized through QAT have better performance compared to models quantized through PTQ. However, due to the need for huge training dataset and high training costs, QAT is not practical. Therefore, recent quantization research for the large foundation models has been focused PTQ(please refer to the survey paper (Zhu al., 2023)). DGQ is the first method to achieve low-bit quantization of text-to-image diffusion models without any additional fine-tuning (i.e., PTQ).
E QUANTIZATION GRANULARITY
Quantization in deep learning models involves reducing the precision of weights and activations to lower bit-width representations, thereby enhancing computational efficiency and reducing memory consumption. The granularity of quantization tratiizzatiothhatt -hat is, the level at which quantization parameters are spplied--sgniffcantle impacts the trade-off between model accuracy and computational perfor- mance. The primary types of quantization granularity are layer-wise, group-wise, and channel-wiss quantization.
Layer-wise Quantization
Group-witt wise Quantization
Channel- Quantization
b
b: batch index, c: channel index, pixel index, group
Figure A.2: Illustration of Quantization Granularity. Different quantizers are applied to each color.
As shown in Figure A.2, layer-wise quantization applies a single scale and zero-point to all weights or activations within entire layer, simplifying implementation but potentially reducing accuracy due to its coarse approach. Group-wist quantization divides the weights or activations within layer into multiple groups, assigning separate quantization parameters to each group. This method offers a balance between efficiency and precision, capturing more detail than laser-wise quantization with- the full complexity of channel-wiss quantization. Channel-wise quantization assigns individual
16

---

Published as a conference paper at ICLR 2025
quantization parameters to each channel, providing the most precise representation of weight and ac- tivation distributions. While this ine-grained approach often yields higher model accuracy, comes with increased computational and memory overhead due the need to store and process multiple sets quantization parameters.
F IMPACT OF ATTENTION SCORE QUANTIZER BIT-WIDTH ON IMAGE-TEXT ALIGNMENT.
To analyze the effect of the attention score quantizer's bit-width, we adjust its bit-width while keep- ing the other layers full precision. As shown in Figure A.3, using the linear quantizer (employed in the baselines Q-Diffusion and TFMQ-DM) causes slight changes in the image content, and at the 6-bit setting, the image becomes misaligned with the text prompt. However, with attention-aware quantization, the image is successfully preserved, matching the quality of the full-precision images even the 6-bit setting.
16bits
8bits
6bits
Linear Quantization
Full Precision
Attention-aware Quantization
"A photo of a cat and a dog"
Figure A.3: Qualitative comparison of attention score quantizer
G STATISTICS FOR ATTENTION SCORE DISTRIBUTION
To provide detailed information about the attention distribution, calculated the statis- ties of attention scores and compared the distinct patterns between self-attention and cross-attention. We conducted experiments on the PartiPrompts dataset and collected the maximum value of each layer's 's attention scores. We transformed the maximum values using base-2 logarithm (log2) and calculated the statistics.
As shown in Figure 5(b), the maximum values of cross-attention scores vary more dynamically than those of self-attention. According to the statisties of the maximum attention scores, the standard
Statistic
Value
Std of cross-attention
0.826
Std of self-attention
0.334
Mean ratio of each layer's 's attention std
3,210
Table A.3: Statistics for attention score distribution.
17

---

Published as a conference paper at ICLR 2025
deviation of cross-attention is much larger than that of self-attention. The first two of Table A.3 present the standard deviation(std) all maximum attention scores for each attention type. For more accurate comparison, and computed the mean of the ratios of these standard deviations. This confirms that the standard deviation of cross-attention is, average, more than three times larger than that of self self-attention.
calculated the standard deviation for each transformer layer
18

---

Published as a conference paper at ICLR 2025
H FULL VISUALIZATION OF ACTIVATION MATRIX
To better illustrate that outliers occur at a specific pixel (case 1) or channel (case 2), we visual- ized only the values around the indexes where outliers occur in Figure 4(b). Figure A.4 shows the visualization of full activation matrix.
Dooplet Loansie down Modd CONTOOTEENT O.COT
THAC 1220
Value
channel index
Deep Dowrple than biebe dww.omrleee..e.. www.mr---cceeee Yacomes:
Value
Outliers on the specific pixel
pixel index
(a) Casel: Outliers on the specific pixel
esig coompee 1p, EWCES www.rnanssnnnooo anstoma MLLD 1 aes PaM 12W0
Outliers on specific channels
Value
channel index
bode attertione Innetomer bicks Bartn 254
Value
4000
pixel index
(b) Case2: Outliers on specific channels
Figure A.4: Visualization of full activation matrix in Figure 4(b)
19

---

Published as a conference paper at ICLR 2025
I ADDITIONAL QUALITATIVE RESULTS
we provide more random samples from quantized models obtained using DGQ and TFMQ-DM. Results are shown in the figures below.
Full Precision
DGQ(WBA8G8)
DGQ(WBA8G16)
TFMQ(W8A8)
Full Precision
DGQ(WBA6GB)
DGQ(W8A6G16)
TFMQ(W8A6)
Figure A.5: Additional qualitative results with the 8-bit weight, SDv14. we randomly sampled the captions from the MS-COCO and generate images with them. WXAYGZ represents weights and activations with X and Y bits and a group size of Z.
20

---

Published as a conference paper at ICLR 2025
Full Precision
DGQ(W4A8G8)
DGQ(W4A8G16)
TFMQ(W448)
Full Precision
DGQ(W4A6G8)
DGQ(WAA6G16)
TFMQ(W446)
Figure A.6: Additional qualitative results with the 4-bit weight, SDv1.4 randomly sampled the captions from the MS-COCO and generate images with them. WXAYGZ represents weights and activations with X and Y bits and a group size of Z.
21

---

Published as a conference paper at ICLR 2025
Full Precision
DGQ(W8A8G8)
TFMQW8A8)
DGQ(W8A68)
TFMQ(W8A6)
Figure A.7: Additional qualitative results with the 8-bit weight, SDXL-Turbo. we randomly sampled the captions from the MS-COCO and generate images with them. WXAYGZ represents weights and activations with X and bits and a group size of Z.
22

---

Published as a conference paper at ICLR 2025
Full Precision
DGQ(W4A888
TFMQWAA8
DGQ(W4A688)
TFMQ(WAA6
Figure A.8: Additional qualitative results with the 4-bit weight, SDXL-Turbo. we randomly sampled the captions from the MS-COCO and generate images with them. WXAYGZ represents weights and activations with X and Y bits and a group size of Z.
23


Reviews:
[{"review_id": "XIQumgUgVb", "reviewer": "Reviewer_qNA8", "summary": "This paper addresses the challenges of quantizing text-to-image diffusion models, which suffer from high memory and computational costs. The authors propose a method called Distribution-aware Group Quantization (DGQ), which handles activation outliers and applies prompt-specific quantization scales to preserve image quality and text-image alignment.", "questions": "Refering to Weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The writing is clear and easy to understand.\n\n2. The experiments are thorough, and the analysis of outliers value offers valuable insights.", "weaknesses": "1. The evaluation of inference performance is lacking.\n\n2. In line 417, it states, “we set all attention score quantizer bits to match the activation bits.” Could this direct approach negatively impact performance? What is the FLOP cost of setting the attention score quantizer bits to 16?\n\n3. As shown in Figure A.3, there is still a significant performance drop when activation quantization bits are below 8.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wQseQC513P", "reviewer": "Reviewer_83s4", "summary": "This paper deeply explores the unique distribution patterns of activations and cross-attention scores in text-to-image diffusion models. Based on the analysis of patterns, the authors propose a method called Distribution-aware Group Quantization, which is composed of outlier-preserving group quantization for activations and a new quantizer for cross-attention scores. Experimental results show that the proposed method outperforms the state-of-the-art methods and achieves under 8-bit quantization for the first time.", "questions": "Please refer to the weakness part.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. This paper is well organized and the writing is good.\n2. The in-depth analysis of the activation and cross-attention in the text-to-image diffusion models is interesting and may benefit the community.\n3. The experiments are extensive and solid, showing the effectiveness of the proposed approach. The improvement is significant.\n4. The idea is reasonable and easy to follow.", "weaknesses": "1. Typos from line.326 to line.327: these lines contain repeated sentences.\n2. The number of generated samples to compute FID and IS is not provided from line.403 to line.410.\n3. The baseline model SD 1.4 may be out-of-date now. It would be better if results on more advanced models like SDXL were provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hDpVaqMF1s", "reviewer": "Reviewer_Hfso", "summary": "This paper proposes Distribution-aware Group Quantization (DGQ), which identifies and adaptively manages pixel-wise and channel-wise outliers to preserve image quality. Additionally, a prompt-specific logarithmic quantization strategy is introduced to enhance text-image alignment. Experiments conducted on the MS-COCO dataset demonstrate the effectiveness of DGQ.", "questions": "1.\tHow can the quantized model in Table 2 outperform the original model? What mechanisms allow the quantization process to enhance performance?\n\n2.\tThe meaning of W8A8 should be explained upon its first mention. The article lacks clarity on the notation WXAX, and similarly, the meaning of WXAXGX in the supplementary materials is not explained.\n\n3.\tIt is recommended to add brackets around the feature indices in Formula 9 for improved clarity, such as A_{[:,0]} V_{[0,:]}.\n\n4.\tWhy does reducing the bits of activation result in significant performance loss?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper exhibits several strengths:\n\n1. The analysis of key characteristics necessary for maintaining model performance during the quantization process is scientific and persuasive.\n\n2. The proposed group-wise quantization strategy, based on the distribution of activations, is both promising and innovative.", "weaknesses": "This paper exhibits several Weaknesses:\n\n1.\tThe figures are confusing and difficult to understand.\n\n     a)   How are issues like \"drops outlier,\" \"high quantization error,\" and \"high overhead\" reflected in Fig. 3? What specific phenomenon indicates \"drops outlier\"? \n\n    b)    What does the horizontal axis in Fig. 4(b) represent? In channel-wise boxplot, why is the horizontal axis range of case1 from 0 to 900, while for case 2 it ranges from 1100 to 1190? What do \"specific pixel\" and \"specific channel\" refer to?\n\n2.\tThe distinctions between group-wise quantization, layer-wise quantization, and channel-wise quantization are somewhat abstract and challenging to grasp. It would be beneficial for the authors to include a figure that illustrates the differences in their principles.\n\n3.\tThere is a lack of qualitative and quantitative comparisons with the latest research, such as MixDQ[1].\n\n4.\tThe base model used is outdated and singular. It is recommended to include experiments with more recent Text-to-Image models (e.g., SDXL, PixArt, SD3, or Flux) to validate the generalization of DGQ. \n\n\n[1] MixDQ: Memory-Efficient Few-Step Text-to-Image Diffusion Models with Metric-Decoupled Mixed Precision Quantization", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AhXjqxDJKe", "reviewer": "Reviewer_s9vr", "summary": "The paper proposes a novel quantization method for diffusion models, named DGQ, which efficient reduce memory usage and computational costs compared to existing methods without additional finetuning. This approach aims to achieve high-quality generation while maintaining performance metrics. Experiments are conducted on MS-COCO and PartiPrompts using IS, FID and CLIP evaluation metrics.", "questions": "Does excluding the <start> token mean dropping and zeroing it? Why does the author think that the peak near the value of 1 on the far right of Figure 5(a) corresponds to the <start> token, while other regions do not?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.It is interesting that the author finds activation outliers play a crucial role in determining image quality.\n2.It is interesting that maintain text-image alignment in quantizing text-to-image diffusion models without requiring additional fine-tuning of weight quantization parameters\n3.The logic of the work is clear, and the exploratory part of the experiments is rich in content", "weaknesses": "1.The conclusions in Figure 5(b) should provide statistical results.\n2.This work presents several new matrix computations, including $D_d$. However, the authors fail to address the extra time overhead that these computations may incur.\n3.In Table 2, the proposed method with groups set to 16 underperforms compared to groups set to 8 on some metrics (e.g., IS and FID). It would be beneficial to analyze the reasons behind this occurrence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vuz3Wno7vx", "reviewer": "Reviewer_1KQn", "summary": "This paper presents an analysis of feature distributions within layers and cross-attention scores in text-to-image (T2I) diffusion models. Building on these analysis, the authors introduce Distribution-aware Group Quantization (DGQ), a method specifically designed to address the distinct distribution patterns in T2I diffusion models. The proposed approach is validated across multiple datasets, demonstrating its effectiveness.", "questions": "The major concerns are in the weakness section, therefore I currently feel that this paper is borderline.\nI have some other questions in below:\n\n- The authors analyze the <start> token's attention score distribution; however, I am curious about the distributions for other special tokens such as <end> or <pad> tokens and a comparison discussion with <start> would find meaningful insight within the T2I diffusion model.\n\n- As I understand it, the proposed method (Attention-aware quantization) applies full precision to the <start> token while quantizing other components to lower precision. Consequently, in Table 2, I would expect slight differences in memory req and BOPs compared to the baselines. However, the current paper shows identical values for these metrics. Could you clarify the difference?\n\n- The evaluation includes FID and IS metrics for assessing image qualtiy. I think other image quality assesments (IQA, [A, B]) against full-precision and baseline models would provide practical insights and could enhance the evaluation for the both datasets.\n\n[A] MANIQA: Multi-dimension Attention Network for No-Reference Image Quality Assessment, CVPRW 2022\n[B] PromptIQA: Boosting the Performance and Generalization for No-Reference Image Quality Assessment via Prompts, ECCV 2024", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The paper's motivation is clear and straightforward.\n\n- The authors effectively identify and address outlier issues present in prior methods, providing a targeted method.\n\n- The analysis of attention scores for <start> tokens is convincing, and the proposed method appears to address these challenges well.", "weaknesses": "- The authors validate the proposed method only on SDv1.4. These limited experiments restrict the ability to assess the generality of their approach. Additional analyses and experiments on other publicly available checkpoints (e.g., SD v2+, EDM, Imagen, etc) would strengthen the evaluation.\n\n- Lines 127-129: \"To the best of our knowledge, we are the first to achieve low-bit quantization (< 8-bit) on text-to-image diffusion models.\" However, [A] has previously proposed low-bit quantization for T2I diffusion models. While [A] and the submitted paper are concurrent works, a discussion of recent works would need for the future work.\n\n- While the analysis and proposed method have novel elements, the approach relies heavily on existing methods (Group Quantization, logarithmic quantization). This reliance may worsen the novelty of the method in contrast to the insightful analysis.\n\n[A] BitsFusion: 1.99 bits Weight Quantization of Diffusion Model, NeurIPS 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)