KOALA: Empirical Lessons Toward Memory-Efficient and Fast Diffusion Models for Text-to-Image Synthesis
Authors: Youngwan Lee
Kwanyong Park
Yoorhim Cho
Yong-Ju Lee
Sung Ju Hwang
Venue: NeurIPS 2024 poster (2024)
Keywords: text-to-image synthesis
diffusion model
Primary area: diffusion_based_models
TL;DR: We present empirical lessons for building an efficient text-to-image model

Abstract: As text-to-image (T2I) synthesis models increase in size, they demand higher inference costs due to the need for more expensive GPUs with larger memory, which makes it challenging to reproduce these models in addition to the restricted access to training datasets. Our study aims to reduce these inference costs and explores how far the generative capabilities of T2I models can be extended using only publicly available datasets and open-source models. To this end, by using the de facto standard text-to-image model, Stable Diffusion XL (SDXL), we present three key practices in building an efficient T2I model: (1) Knowledge distillation: we explore how to effectively distill the generation capability of SDXL into an efficient U-Net and find that self-attention is the most crucial part. (2) Data: despite fewer samples, high-resolution images with rich captions are more crucial than a larger number of low-resolution images with short captions. (3) Teacher: Step-distilled Teacher allows T2I models to reduce the noising steps. Based on these findings, we build two types of efficient text-to-image models, called KOALA-Turbo & -Lightning, with two compact U-Nets (1B & 700M), reducing the model size up to 54% and 69% of the SDXL U-Net. In particular, the KOALA-Lightning-700M is 4 times faster than SDXL while still maintaining satisfactory generation quality. Moreover, unlike SDXL, our KOALA models can generate 1024px high-resolution images on consumer-grade GPUs with 8GB of VRAMs (3060Ti). We believe that our KOALA models will have a significant practical impact, serving as cost-effective alternatives to SDXL for academic researchers and general users in resource-constrained environments.

KOALA: Empirical Lessons Toward Memory-Effcient and Fast Diffusion Models for Text-to-Image Synthesis
Youngwan Lee Kwanyong Park' Yoorhim Cho Yong-Ju Lee' Sung Ju Hwang2+
EElectoonics and Telecommunications Research Institute (ETRI), South Korea
Korea Advanced Institute of Science and Technology (KAIST), South Korea
"Sungkyunwwan University, South Korea
'DeepAuto.ai, South Korea
project page: https: y/yonggwnllee github.io//OAn
Abstract
As text-to-image (T2I) synthesis models increase in size, they demand higher inference costs due to the need for more expensive GPUs with larger memory, which makes it challenging to reproduce these models in addition to the restricted access to training datasets. Our study aims to reduce these inference costs and explores how far the generative capabilities of T21 models can be extended using only publicly available datasets and open open-source models. To this end, by using the present
de facto standard text-to-image model, Stable Diffusion XL (SDXL), three key practices in building an efficient T2I model: (1) Knowledge distillation: we explore how to effectively distill the generation capability of SDXL into an efficient U-Net and find that self-attention is the most crucial part. (2) Data: despite fewer samples, high-resolution images with rich captions are more crucial than larger number of lesouusonution images with short captions. (3) Teacher: Step- distilled Teacher allows T2I models to reduce the noising steps. Based these findings, we build two types of efficient text-to-image models, called KOALAA LA- Turbo &-Lightning, with size up to 54% and 69% of the SDXL. U-Net. In particular, the KOALA- LOALA-Lightning. 700M is 4x faster than SDXL while still maintaining satisfactory generation quality. Moreover, unlike SDXL, our KOALA models can generate 1024px high- resolution images consumer-grade GPUs with 8GB of VRAMs (3060Ti). We believe that our KOALA models will have a significant practical impact, serving as cost-effective alternatives to SDXL for academic researchers and general users resource- constree-onstteanee environments.
compact U-Nets (1B & 700M), reducing the model
1 Introduction
Since Stable Diffusion XL [31] (SDXL) has become the de facto standard model for text-to- image (T2I) synthesis due to its ability generate high-resolution images and its open-source nature, many models for specific downstream tasks [7; 47; 3; 58; 56) now leverage SDXL as their backbone. However, the model's massive computational demands and large size necessitate expensive hardware, thus incurring significant costs in terms of training and inference, Moreover, recent T2I works [5; 6; 39; 1] do release their training datasets, making challenging for the open-source community to reproduce their performance due to internal or commercial restrictions.
To alleviate the computation burden, previous works have resorted quantization [45], hardware- aware optimization [8], denoising step reduction [38; 29; 39; 23], and architectural model optimiza- tion [21]. In particular, the denoising step reduction (i.e., atep-disiillation) and architectural model compression methods adopt the knowledge distillation (KD) scheme [14; 11] by allowing the model
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Figure 1: Samples by KOALA-Lightning-000 with 1024² resolution and 10 denoising steps, generated in 0.65 seconds NVIDIA 4090 GPU. The prompts and more qualitative comparisons are illustrated in App. C.
to mimic the output of the SDMs a teacher model. For the architectural model compression, BK-SDM [21] exploits KD when compressing the most heavy-weight part, U-Net, in SDM-v1.4 [35]. BK-SDM builds compressed U-Net by simply removing some blocks, which allows the compressed U-Net to mimic the last features at each stage and the predicted noise from the teacher model during the pre-training phase. However, the compression method proposed by BK-SDM achieves limited compression rate (33% in Tab. 2) when applied to the larger SDXL than SDM-v1.4 .4, and the strategy for feature distillation in U-Net has not yet been fully explored.
In this work, our goal is to build an efficient T2I model based on knowledge distillation in the latent diffusion model (KOALA) by exploring how far we can push the performance using only open-source data and models alone. To this end, we first design two compressed U-Nets, KOALA- IB and KOALA- 700M, using not only block removal but also layer-wise removal to reduce the model size of the SDXL U-Net by up to 54% and 69% (vs. BK's method: 33%) in Tab. 2. Then, we explore how to enhance the generative capability of the compact U-Net based on three empirical findings; (1) self-attention based knowledge distillation: we investigate how to effectively distill SDXL teacher model and find essential factory for feature-level KD. Specifically, we found that self-attention features are the most crucial for distillation since elf-aatention-bbsee KD allows models to learn more discriminative representations between objects attributes. (2) Data: When performing KD-training, high-resolution images and longer captions are more critical, even with fewer samples, than a larger number of low-resolution images with short captions. (3) Teacher model: The teacher model with higher performance improves the capability of the student model, and step-distilled teachers allow the student model to reduce the denoising steps, which results in further speed-up.
Based on these findings, we train efficient text-to-image synthesis models on publicly available LAION-POP [40] by using two types of distilled teacher models, SDXLL-Turbo [39] and SDXL- Lighting [23], with 512px and 1024px resolutions, respectively. We observe that our KD method consistently outperforms the BK [21] method both U-Net and Diffusion Transformer backbones in Tabs. 7 and In addition, DDALL--ighhning70000 using the BK method, at 3x faster speed. Furthermore, (OALL-Lightting-- speed and 3x model efficiency than SDXL-Base while exhibiting satisfactory generation quality. Lastly, to validate its practical impact, we perform inference analysis on a variety of consumer- grade GPUs with different memory sizes (e.g.. 8GB. 11GB, and 24GB), and the results show that SDXL models cannot be mounted on an 8GB GPU, whereas our KOALA models can operate on it, demonstrating that our KOALA models are cost-effective alternatives for practitioners in resource-constrained environments.
outperforms SSD-IB [10], which trained
achieves 4x faster
'nttps://civitai.comm

---

Iuare Sarmoc Bloce
Hansformet Blocks
Hammbern Blocks
Transfurme
Self-attention-based knowledge distillation (Sec. 3.1)
swath
Papth 30
Septn
Teacher SOXL Base (2.568)
Ieter-a
Predicued me:e
040012201128
Student KOALA-18 (1.168)
KOALA-700M
(782M)
DW--3
DW--
WP-3
UP-3
Figure 2: Overview of AnOwledee-Distil... tion in LAtent diffusion model based on SDXL and architecture of KOALA. We omit skip connections for simplicity. We perform feature distillation in transformer blocks using self-attention layers.
Our main contributions are as follows:
We design two efficient denoising U-Net with model sizes (1.13B/782M) that are more than twice compact SDXL's U-Net (2.56B).
2. We perform comprchensive analysis to build efficient T2I models, drawing three key lessons: self-attention distillation, the impact of data characteristics, and the influence of the teacher model.
We conduct systematic analysis inference on consumer-grade GPUs, highlighting that our KOALA models operate efficiently an 8 GB GPU, unlike other state-of- teftthh-rtt models.
2 In-depth Analysis: Stable Diffusion XL
Table SDXL-BBse-1.00 model budget. Latency is measured un- der the image scale of 1024 FP16- precision, and 25 denoising steps in
Table 2: U-Net Comparison. Tx means Transformer. SDM- v2.0 [36] uses 768 resolution, while SDXL and KOALA mod- els use 1024 resolution. CKPT means the trained checkpoint file.
NVIDIA 4090 GPU (24GB).
U-Net
SDM-v2.0
SDXL-110
BK-SDXL
KOALA-IE
KOALA-700M
SDXL
Text Enc.
VAE Dec.
U-Net
Parram.
865M
2.567M
1,717M
1.161M
782M
CKPT size
346GB
10.3GB
6.8GB
4.4GB
3.0BB
#Param.
817M 0.008
83M
2,567M
Tx blocks
[1. 1,
2, 101
10.
[0, 10,2,6
10. 2.
Latency
0.002
3.133
Mid block
V
Latency
1.13s
3.13s
2.42s
1.60s
1.25s
SDXL [31], the latest version of the SDM series [35; 36; 34], exerts a significant influence on both the academic community and the industry due to its unprecedented 1024² high-resoluties image quality and open source resources. It has several key improvement points from the previous SDM-v2.0 [36], e.g., multiple sizes- & roo-conditioning, an improved VAE, a much larger U-Net, and an ad hoc style of refinement module, which leads to significantly improved generation quality. However, the significant enlargement of U-Net in model size results in increased computational costs and significant memory (or storage) requirements, hampering the accessibility of SDXL. Thus, we investigate U-Net in SDXL. to design a more lightweight U-Net for knowledge distillation. We dissect the components of SDXL, quantifying its size and latency during the denoising phase, as detailed in Tab. 1. The enlarged U-Net (2.56B) is the primary cause of the increasing SDXL model size (vs. SDM-v2.0 (865 M)). Furthermore, the latency of U-Net is the main inference time bottleneck in SDXL. Therefore, it is necessary to reduce U-Net's model budget for better efficiency.
The SDXL's U-Net varies in the number of transformer blocks for each stage, unlike SDM-v2.0, which employs transformer block for each stage (see Tab. 2). At the highest feature levels (e.g.. DN-1&UP-3 Fig. 2), SDXL uses only residual blocks without transformer blocks, instead distributing more transformer blocks to lower-level features. So, in Fig. 13, we analyze the parameter distribution of each stage in the U-Net. Most parameters (83%) are concentrated on the transformers with ten blocks in the lowest feature map (e.g., g., 32 of DW-3, Mid, UP-1 in Fig. 2), making the main

---

parameter bottleneck. Thus, it is essential to address this bottleneck when designing an efficient U-Net architecture.
Three lessons for building an efficient text-to-image model
In this section, we introduce three empirical lessons to realize an efficient text-to-image synthesis; first, we design a lightweight U-Net architecture and perform a comprehensive analysis with the proposed efficient U-Net find nnowlege-disttllation (KD) strategies in See. 3.1.2. Secondly, we investigate the training data characteristics that affect image generation quality in Sec. 3.2. Finally, we explore how different teacher models influence the student model in Sec. 3.3. For these empirical studies, we adopt two evaluation metrics, Human Preference Score (HPSv2) [55] and CompBench [17]. instead of FID. Recently, several works [2; 55; 31] have claimed that FID [13] not closely correlated with visual fidelity. HPSv2 [55] is for visual aesthetics metric, which allows us to evaluate visual quality in terms of more specific types. As an image-text alignment metric, Compbench [17] is a more comprehensive benchmark for evaluating the compositional text-to-image generation capability than the single CLIPP score [12]. We report average scores for HPS and Comphench, respectively.
3.1 Lesson. Self-attention based Knowledge Distillation with Efficient U-Net Architecture
3.1.1 Efficient U-Net architecture
A prior strategy for compressing the text-to-image model is to remove pair of residual and trans- former blocks at each stage, namely block removal [21; 10]. While this method may be sufficient for compressing shallow U-Nets like in SDM-v1.4 [35], it shows limited effectiveness for recent, more complex U-Nets. For cases of SDXL [31], the compression rate is reduced only from 2.5B to 1.7B, as shown in Tab. 2. To address this limitation, we investigate a new approach for compressing these heavier U-Nets to achieve more efficient text-to-image generation.
Transformer layer-wise removal is the core of efficient U-Net architecture design. According to the discussion Sec. 2, the majority of parameters are concentrated in the transformer blocks the lowest feature levels. Each block comprises multiple consecutive transformer layers, specifically ten layers per block in SDXL (see Fig. 2). We address this computational bottleneck by reducing the number of transformer layers (i.e., depth), a strategy term layer-wise removal.
Using this removal strategy as the core, we instantiate two compressed U-Net variants: KOALA-IB and KOALA-700M. First, we apply the prior block-removal strategy [21] to the heavy U-Net of SDXL. We note that in the decoder part (e.g., UP-1 UP-3), we remain more blocks than in the encoder because the decoder part plays a more important role in knowledge distillation, which is addressed Sec. 3.1 and Tab. 3b. On this removed backbone, we then adopt layer-wise removal at different ratios. Specifically, we reduce the transformer layers at the lowest feature level (i.e., DW-3, Mid and UP-1 in Fig. 2) from 10 to 5 for KOALA-
and to 6 for KOALA-IB For KOALA-700M, we also removed the Mid block. An overview of the compressed U-Nets is presented in Tab. and Fig. 2. Our KOALA-IB model has 16 parameters, making it half the size of SDXL (2.56B). KOALA-700M, with 782M parameters, is comparable in size to SDM-v2.0 (865M).
3.1.2 Exploring Knowledge Distillation for SDXL
Table 3: Analysis of feature level knowl-
Prior work [21] that attempts to distill an early se-
ries of stable diffusion (i.e,, SDM-v1.4 [35]) di-
edge distillation of U-Net in SDXL [31].
rectly follows traditional knowledge distillation liter- ature [37; 11]. The compressed student U-Net model Sş is jointly trained to learn the target task and mimic the pre-trained U-Net of SDM-v 1.4 teacher net- work. Here, the target task is the reverse denoising process [16], and we denote the corresponding learn- ing signal as Lusk. Besides the task loss, the com- pressed student model is trained to match the output of the pre-trained U-Net at both output and feature levels. Laut and Lieat represent the knowledge distillation (KD) loss at the output- and feature-level, respectively. For designing the feature-level
Type IIPSv2
Loc.
IPPSv2
Baseline
25.53
Bascline
25.53
SA
26.74
DW-2
25.32
CA
26.11
DW-3
25.57
Res
26.27
Mid
25.66
FFN 26.48
UP-1
26.52
LF
26.63
UP-2
26.05
(a) Distillation type
(b) Distill stage
KD-loss, BK-SDM [21] simply considers only the last feature (LF) map of the teacher f-) and

---

big hippoopoannus and small rablit."
2012
MU
UF2
LF (BK)
SA (Ours)
(a) Generated Image
(b) Analysis
(c) Self-attention visualization
Figure 3: Analysis on self-attention maps of distilled student U-Nets. (a) Generated images of LF- and SA-based distilled models, which are BK-SDM [21] and our proposal, respectively. In BK- SDM's result, a rabbit is depicted like a hippopotamus (i.e., appearance leakage). (b) Visualization of PCA analysis results on self-attention maps of UP-1 stage. (c) Representative visualization of self-attention map from different U-Net stages. Red boxes denote the query patches. Note that from the MID stage, the SA-based model attends the rabbit more discriminatively than the LF model, demonstrating that attention-baser KD allows to generate objects more distinctly.
student network f-()) at each stage as follows:
LteakD
min Eg E.,e... Σ - f,
Se
where and e denote given diffusion timestep and text embeddings as conditions. Thus, the feature distillation approach for text-to-image diffusion models has not been sufficiently explored, leaving room for further investigation.
In this section, we extensively explore feature distillation strategies to distill the knowledge from the U-Net SDXL effectively to our efficient U-Net, KOALA-IB. We start from bascline trained only by Cask and add LtaaKD without LowKD to validate the effect of feature distillation. More training details are described in Sec. 4.1 and App. A.
Self-attention-based knowledge distillation transfers discriminative image representation.
With the increasing complexity of U-Net and its stage, relying solely on the last feature (LF) as BK [21] may not be sufficient to mimic the intricate behavior of the teacher U-Net. Thus, we revisit which features provide the richest guidance for effective knowledge distillation. We focus on key intermediate features from each stage: outputs from the self-attention (SA), cross-attention (CA), and feedforward net (FFN) in the transformer block, well outputs from convolutional residual block (Res) and LF. Tab. 3a summarizes the experimental results. While all types of features help obtain higher performance over the naïve baseline with only the task loss, distilling self-attention features achieves the most performance gain. Considering the prior studies [22; 46; 49] which suggest that SA plays vital role in capturing semantic affinities and the overall structure of images, the results emphasize that such information is crucial for the distillation process.
To understand the effects more clearly, we illustrate representative example in the Fig. 3. To reason about how the distilled student U-Net captures self-similarity, we perform a PCA analysis [19; 50] on self-attention maps. Specifically, apply PCA on self-attention maps from SA- and LF-based models and show the top three principal components each principal component distinctly represents individual objects (i.e.. unique color assignments to each object). This indicates that the SA-based model effectively distinguishes different objects in modeling self-similarity, which plays crucial role in accurately rendering the distinct appearance of each object. In contrast, the LF LF-based model exhibits less distinction between objects, resulting in appearance leakage between them (e.g., small hippo with rabbit ears). More PCA analyses are detailed in Fig. 10.
Fig. 3-(b). Interestingly,
the SA-based model,
Self-attention at the decoder has larger impact on the quality of generated images.
We further explore the role and significance each self-attention stage. To this end, we first visualize the self-attention map in Fig. 3-(c). The self-attention maps initially capture general contextual
5

---

Table 4: Training Data comparison. AR and ACL mean aver- Table 5: Teacher model comparison.
age resolution and average caption length, respectively. synCap We use KOALA-700M as a student means synthetic captions by LLaVA-v1.5 [24]
model.
ID
Data
Himgs
AR
ACL
HPSv2
CompBench
Teacher model
Step
HPSv2
CompBeach
(a) L-66N-A-6+ (43)
580
27.43
0.3791
SDXL-Base-I.0
27.79
0.4290
(b) (a) synCap
SM
560 x 676
72
27.61
0.4168
Turbo
10
27.88
0.4470
(c) LAION-POP [40]
491K
1274 1457
81
27.79
0.4290
SDXI-Lightnin
28.13
0.4538
information (e.g., DW 2W-22DW-3) and gradually focus localized semantics (e.g., MID). In the decoder, self-attentions increasingly correlate with higher-level semantics (e.g., object) to accurately model appearances and structures. Notably, in this stage, the SA-based model attends corresponding object regions (given the query patch, red box) more discriminatively than the LF-based model, which results in improved compositional image generation performance.
In addition, we ablate the significance of each self-attention stage in the distillation process. Specifi- cally, we adopt SA-based loss at single stage alongside the task loss. As shown in Tab. 3b, the results align with the above understanding: distilling self-attention knowledge within the decoder stages significantly enhances generation quality. In comparison, the impact of self-atte self-attention solely within the encoder stages is less pronounced. Consequently, we opt to retain more SA layers within the decoder (see Fig. 2).
In summary, we train our efficient KOALA U-Nets using the following objectives: Lusk + LouIKD LreaKD- We apply our proposed self-attention-baseee knowledge distillation (KD) methods to LieaKD- Further analyses on featKD, including how to locate features and combine different types of features, are provided in App. B.I.
3.2 Lesson 2. Data: the impact of sample size, image resolution, and caption length
We investigate various data factors-such image resolution, caption length, and the number of samples-that impact the quality of the final text-to-image model. To ensure reproducibility, we design three data variants using spen-source data. (i) LAION-Aesthetic-e includes large volume of image-text pairs (8,483,623 with images filtered for high aesthetics. Most images are low-resolution (average 580 x 676). and the corresponding captions are brief (average length of 13 words). (ii) rescription-augmentte LAION-A-6H is designed to demonstrate the impact of detailed descriptions. For each image in LAION-A-66 (LMM) to generate detailed descriptions. These synthesized captions, referred to as synCap, convey significantly more semantic information and are longer (e.g., an average length of 72 words; more details synCap in App. A.I). This data source is denoted in the second row of the table. (iii) LAION-POP [40] features high-resolution images (average 1274 x 1457) and descriptive captions (average length of 81 words), although the dataset size is relatively small (e.g., 491,567 samples). The descriptions are generated by LMM, CogVML [54] and LLaVA-v1.5.
(LAION-A-6+) [43]
we use a large multimodal model [24]
We train 1OAALA-700M models using the same training recipes for each data source. From the results summarized in Tab. 4, we make several observations. First, detailed captions significantly boost performance, enabling the model to learn detailed correspondences between images and text (See (a) and (b) in the Compbench score). Second, high-resolution images, which convey complex image structures, are valuable source for training T21 models. Despite having fewer samples, LAION-POP further boosts overall performance. Based on these findings, we opt to use LAION-POP as the main training data, as it features high-resolution images and descriptive captions.
3.3 Lesson 3. The influence of Teacher model
Following the tremendous success of SDXL. [31], recent large-scale text-to-image models have adopted its U-Net backbone. SDrboo [39] and SDXII-Lightning [23] are notable examples, enabling high-quality image generation in low-step regime through progressive distillation [38]. This section investigates whether our distillation framework can effectively exploit these diverse models. To this end, we leverage SDXL-Base and its variants. SDXL-Turbo and SDXL-Lightning, as teacher models, transferring their knowledge into KOALA-700M. We apply the former two lessons and more training details are described in App. A.2.

---

Step: 2
Step: 4
Step: 6
S Step: 8
Baseline
(+) 14
2
lesson
SOXL
"A cat riding motorcycle. :
SDXL Tubbo
"A pink teddy bear yellow surfboard the
SDXL SDXL- Lightring
cat riding a motorcycle."
"A alpaca made of colorful building blocks."
Figure 5: Denoising process of different teacher models.
Figure 6: Qualitative analysis proposed lessons.
As shown in Tab. all KOALA-700M models distilled from different teacher models demonstrate decent image generation capabilities. This highlights the generality of our knowledge distillation framework. More interestingly, when using SDXL-Turbo and SDXL-Lightnig as teach- ers, KOALA-700M, models exhibit comparable or even better image quality than when SDXLL-Base is used as the teacher, despite requiring fewer denoising steps (e.g., 10 vs. 25). Note that the KD framework or noise schedule (Euler discrete schedule [20], the same SDXL-Base) for the different KOALA models does not require specific modifications. Thus, KOALA models seamlessly inherit ability to illustrate realistic structures and details images, even in the short-step regime, from their step- distilled teachers (See nnd and 3rd rows in Fig. 5). This results in robust performance across diverse range of denoising steps (See Fig. 4). In contrast, when SDXL is used as the teacher model, KOALA models struggle to depict realistic structures in few steps, leading to flawed features (e.g., missing handles and lights on a motorcycle in Fig. 5). For more efficient text-to-image synthesis, leverage step-distileed teachers, enabling KOALA models to generate high-quality images in just a few steps.
27.3
HP5
Teacher model
SDX0 Base-1.
Turoe
50a14 ghining
Figure 4: Teacher model comparisons across denoising steps with KOALA- 700M.
Therefore, combining all these lessons, we build KnOwledge-distilAtiionnn models, called KOALA. As discussed in each section, the three proposed lessons complement each other. Both image quality and image-text alignment improve progressively each lesson is added, as shown in Fig. 6. In particular, we build two types of KOALA models: KOALA- -uubbo with 512px and KOALA-Lightming with 1024px using two KOALA U-Net (IB&700M), respectively.
LAtent diffusion
4 Experiments
4.1
Implementation details
Dataset. As the datasets used for state-of-the-art methods in Tab. 6 are proprictary or not released, we opt for the publicly accessible only LAION dataset [44] to ensure the reproducibility of our work. From the data recipe in Sec. 3.2, we finally use LAION-POP [40] for KOALA models in Tab. More details of the dataset used described in App. A.1
Training. According to the recipe about the Teacher model in Sec. 3.3, we use SDXL-Turbo [39] and SDXI--Lghtning [23] teacher models, building two types of KOALA models, KOALA-Turbo and KOALA--Lightning. Since SDXI--Turbo and Lightning are based on SDXL-Base model [31], we the same two text encoders, OpenCLIP ViT-bigG [18] and CLIP ViT-L [33] and only replace the
7

---

Table 6: Performance comparison to state-of-the-art models. We measure latency and memory usage with a bath size of I on NVIDIA 4090 GPU. We obtain HPSv2 and Compbench scores of all models on the same GPU and library environment by using their official weights. We highlight the best value in green, and the second-best value in blue. The full scores of HPSv2 and Compbench are shown in Tab. 10.
Model
Resolution
Steps
Latency (8)
U-Net Param.
Memory
HPSv2
CompBench
SDM-v2.0 [36]
768
25
1.236
0.86B
5.6GB
25.86
0.3672
SDXL-Base-LO [31]
10242
25
3.229
2.56B
11.9GB
30.82
0.4445
SDXL-TTubo [39]
512
0.245
2.56B
8.5GB
29.93
0,4489
SDXL-Lightning [23]
10242
0.719
2,56B
11.7GB
32.18
0,4445
Pixart-o: [5]
10242
25
3.722
0.6B
17.3GB
32.06
0,3880
Pixart-E [6]
1024a
3.976
0.6B
17.3GB
31.75
0.4612
SSD-IB [10]
10242
2.094
9,4GB
31.43
0,4497
SSD-Vega [10]
10242
25
1.490
0.74B
8.2GB
32.17
0.4461
5122
10
0.194
0.78B
4,9GB
29.98
0.4555
KOALA-Turbo-I1
512
10
0.238
1.16B
5.7GB
29.84
0.4560
KOALA- DOALAA--ighttnnng-7000
1024
10
0.655
0.78B
8,3GB
31.50
0.4505
KOALLA-Lightting-I1 10242
10
0.790
1.16B
9.1GB
31.71
0,4590
Table 7: Comparison to BK [21]. All models are
Table 8: KD feature types in Diffusion Transformer (Pixart-E [6]).
trained for 50K iterations same BK-SDM.
KD method
Backhone
HPSv2
CompBench
KD Type.
HPSv2
CompBench
BK [21]
BK-Small
26.72
0.3237
SA CA FFN
25.16
0.4281
Ours
BK-Small
26.86
0.3417
24.94
0.4279
BK [21]
KOALA-IB
27.01
0.3599
24.80
0.4191
Ours
KOALA-IB
27.15
0.3712
LF BK [21]
21.62
0.3527
original U-Net with efficient KOALA U-Net. Our U-Nets are initialized with the teacher's U-Net weights at the exact block location. Using our self-attention-basee KD method in Sec. 3.1.2, we train our KOALA models the LAION-POR POPN-POP dataset using four NVIDIA A100 (80GB) GPUs with 512" and 1024" resolutions for KOALA-Turbo and KOALA-Lightring, respectively. Inference. We use Euler discrete scheduler [20] as the same sampler in SDXL [31]. All KOALA models generate images with 10 denoising steps, FP16, and cfg-sale [15] of 3.5. Please see further details of training and inference in App. A.2 and App. A.3, respectively.
4.2 Main results
vs. SDXL models: Compared to SDXL-Base- 1.0 [31], our KOALA-LighhnnngTOON-- models achieve better performance in terms of HPSv2 and CompBench while showing about 5x and 4x faster speed, respectively. Compared to SDXL-Turbo [39] and SDXL-Lightning [23], our KOALA- Turbo and Lighi-Lightningg models show comparable or inferior HPSv2 scores but achieve higher CompBench scores with up to 3x smaller model sizes and 1.7x lower memory usage. vs. Pixart: KOALA-Lightning models fall short in HPSv2 and CompBench. Especially, Pixart-N, [6] achieves the best CompBench and the second-best HPSv2 scores. This result is attributed to data quality, as Pixart-Y collects high-quality internal data consisting of 33 million images above IK resolution and uses synthetic longer captions (with an average length of 184 words). However, our KOALA- Lightning-700M shows 6x faster speed and 2x better memory efficiency. vs. SSD: Note that due to the difference in training datasets, we cannot make a direct comparison with SSD models, which are trained by BK [21]'s KD method. Except for HPSv2 of SSD- [10], KOALA-Lighting models show better HPSv2 and CompBench scores while achieving up to 3x faster speed. More qualitative comparisons in App. C support the quantitative results.
4.3 Discussion
Comparison with BK-SDM. For fair comparison to BK-SDM [21], we train our KOALA U-Net backbones with their distillation method under the same data setup (See training details in A.2). As shown in Tab. 7, our KD method consistently achieves higher HPSv2 and CompBench scores than the BK-SDM [21] when using different U-Net backbones. These results demonstrate two main implications as follows: 1) the proposed distillation of the self-attention layer is more helpful for

---

toala-lightting-7000 koala-llghtninng118
sdxt-lightning Pixart-Alpha
koata-turbo-18
sdxi-turbo
SSD-18
Cooonn La0ag 99 (GB)
Latency (sec)
1ge
155
32
12
50
SSD-1B
2-
Pixart
2.00
SDXL Models
Pixart
0.2
OOM
OOM
GPU-8GB (3060TI)
GPU-11GB (2080Ti)
GPU-24GB (4090)
Figure 7: Latency and Memory comparison on across different consumer-grade GPUs. We run cach model with the denoising steps in Tab, and FP16. For fair comparison, use the official pre-trained weights and inference code in the Hugginface without any other tricks such torch. compile quantization. Note that only our KOALA models and ISD-Vege can run types of GPUs.
Table 9: Synergy effect with step-distilled method, PCM [53]. We conduct step-distillation training using PCM with our KOALA backbones and compare with PCM-SDXL- Basse.
Method
Teacher
Student
#Step
Param. (B)
Memory (GB)
Latency
HPS
CompBench
PCM [53]
SDXL-Base
SDXL-Base
2
2.56
12.1
0.345
29.99
0.4169
PCM [53]
SDXL-Buse
KOALA-700M
2
0.78
8.2
0,222
28.78
0.3930
PCM [53]
SDXL-Base
KOALA-IB
1.16
9.0
0,235
29.04
0.4055
Visual aesthetics than simply distilling the last layer feature by BK [21]. 2) our self-attention--asse KD approach allows the model to learn more discriminative representations between objects or attributes so that it can follow prompts faithfully (as shown in Sec. 3.1.2 and Fig. 3). More qualitative comparisons demonstrated in Fig. 19.
Applicability of self-attention based KD Diffusion Transformer. To validate the generality of our self-attention distillation method, we also apply it to diffusion transformer (DiT) based T21 model, Pixart-E [6]. To this end, We compress the DiT-XL. [30] backbone and build DiT-M by reducing the number of layers from 28 to 14 with the same hidden dimension (see more training details App. A.4.). Following our KD strategy, we conduct an ablation study by simply changing the distillation location due to DiT's architectural simplicity. which consists of only transformer blocks without resolution changes. Tab. illustrates that distilling the self-attention feature outperforms other features while using the last features proposed in BK [21] shows the worst performance, demonstrating that the self-ateention layer is still the most crucial part for diffusion transformer.
Synergy Effect with Miep-isttlllttinn Method, PCM [53]. Since diotdditilllation methods [23; 39; 53] and our KD approach are orthogonal, applying our KOALA backbones to the step-distillation methods could yield synergistic effects, leading to further speed improvements. To verify the synergy effect between the step-distillation method (e.g., PCM [53]) and our KOALA backbones. we conduct distilllltt training using PCM with our KOALA backbones, and the results are presented in Tab. 9. Thanks to their efficiency, our KOALA backbones allow PCM to achieve additional speed-up with only slight performance drop compared to using the SDXL-Base backbone. Furthermore, we provide qualitative comparisons between PCM-KOALA models and PCM-SDXL- Base in Fig. 20, demonstrating that the generated images achieve visual quality comparable those of SDXL-Base.
2PCM the only work that officially provides step-distillation training code pectps://gitain
cattchuub.co//uuuu.. nom/G-U-D htt-:/aasedd-oossssteee----ee-ee wwwssoooonnssooooooooooo

---

4.4 Model budget comparison on consumer-grade GPUs
We further validate the efficiency of our model by measuring its inference speed and memory usage on variety of consumer-grade GPUs with different memory sizes, such as 8GB (3060Ti), 11GB (2080Ti), and 24GB (4090), because the GPU environment varies for individual practitioners. On the GPU-8GB. all SDXL models can't 't fit, while only KOALA models and SSD- [10] can run. KOALA-Lightning- 700M consumes comparable GPU memory but shows 2x faster than SSD-Vega Vega. On the EPU-11GB, SDXL models run, but KOALA-Lightnnn-7000
still runs at approximately 5x faster speed than SDXL-Base [31]. It is noted that Pixart-o: [5] & - Σ [6] cannot operate on GPUs with 8GB and IGB of memory due their higher memory usage, but they can albeit at the slowest speed. is worth noting that our KOALA models can operate efficiently on all types of GPUs, highlighting the versatility and accessibility of our approach. Furthermore, our KOALA-Lighhing-O000 is the best alternative for high-resolution image generation that can replace SDXL models in resource-constraiined GPU environments.
GPU-24GB,
Limitations
While our KOALA models generate images with decent aesthetic quality, such as photo- realistic or 3D-art renderings, they still show limitations in synthesizing legible texts in the generated image as shown in Fig. 8 (Left). Also, models have difficulty in generating plex prompts with multiple attributed or object relationships, as shown Fig. (Right). Addi- tionally, since SDXL is the de facto T2I model, have tried to compress the SDXL U-Net by addressing its bottlenecks. However, this ap- proach is somewhat specific to the SDXL U-Net and heuristic. This limitation arises because the SDXL U-Net has a complex and heterogeneous architecture, comprising both convolutional and transformer blocks, which hinders the formulation of more general compression principle. More detailed investigations and examples are described in App. D.
TheaRE peace
LET PrquF
art character of cuto litty hulding sign thys "het be peace"
brown big dog wearing sunglasses standing on the left and a white small calt wearing helmet stting on the right
Figure 8: Failure
of KOALA-700M
Conclusion
In this work, we have explored how to build memory-efficiett and fast T2I models, designing compact denoising U-Nets and presenting three critical lessons for boosting the performance of the efficient T2I models: 1) the importance of self-attention in knowledge distillation, 2) data characteristics, and 3) the influence of teacher models. Thanks to these empirical insights, our KOALA-Lightningg 700M model substantially reduces the model size (69%4) and the latency (79%4) of SDXL-Base while exhibiting satisfactory generation quality. We hope that our KOALA models can serve as cost-effective alternatives for practitioners in limited GPU environments and that our lessons benefit the open-souree community in their attempts to improve the efficiency of T2I models.
Additionally, since we have identified the potential of applying Diffusion Transformer (DiT) models [30; 5; 6] in Tab. 8 due to their architectural simplicity compared to the U-Net in SDXL, we plan further explore more general model compression methods for DIT, as in the language model literature [9; 28), and KD techniques based on our self-attention distillation.
self-attention-basse based KD to
7 Acknowledgments
This work was supported by Institute of Information & communications Technology Planning & Eval- uation (ITTP) grant funded by the Korea government (MSIT) (No. RS-2022-00187233. Development of Large Korean Language Model Technology for Efficient Pre-training, 45%). (No. 2022-0-00871, Development of AI Autonomy and Knowledge Enhancement for AT Agent Collaboration, 45%) and (No.2019-0-00075. Artificial Intelligence Graduate School (rogram(KAIST), 10%).
10

---

References
[1] Betker, J., Goh, G., Jing. L.., Brooks, T., Wang. J., Li, I.., Ouyang. I.., Zhuang, J., Lee, J., Guo, Y., Manassra, W., Dhariwal, P., Chu, C., Jiao, Y., and Ramesh, A. Improving image generation with better captions. https: h :/cdnn.//dn/////// http://///c....aaiii...mmmaaaaa. www.coo//aaaaae pwtppppppppallll-
1--3..pp.. pdf, 2023. 1, 15
[2] Betzalel, E., Penso, C., Navon, A., and Fetaya, A study on the evaluation of generative models. arXiv preprint arXiv:2206. 10935, 2022. 4
[3] Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V., Letts, A., et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.1512) 2023.
[4] Bohan, O. B.
Sdxl-vae-fp16-fix.
sdx1 -aa--aaee-p----- fix, 2023. 16, 17
[5] Chen, J., Yu, J., Ge, C., Yao, L., Xie, E., Wu, Y., Wang. Z., Kwok, J., Luo, P., Lu, H., and Li, Z. Pixart-o: Fast training of diffusion transformer for photorcalistic text-to-image synthesis, 2023. 1.8, 10, 15, 16
[6] Chen, J., Ge, C., Xie, E., Wu, Y., Yao, L., Ren, X., Wang, Z., Luo, P., Lu, H., and Li, Z. Pixart-o: Weak-to to-sooo training of diffusion transformer for 4k text-to-image generation, 2024. 1, 8, 10, 16, 17
[7] Chen, J., Huang. Y., Lv, T., Cui, L., Chen, Q., and Wei, F. Textdiffuser: Diffusion models as text painters. NewrIPS, 2024. 1
[8] Chen, Y-H.,, Sarokin, R., Lee, J., Tang, J., Chang, C.-L., Kulik, A., and Grundmann, M. Speed all you need: On-device acceleration of large diffusion models via gpu-aware optimizations. In WPPR-Wor 2023.
[9] Gromov, A., Tirumala, K., Shapourian, H., Glorioso, P., and Roberts, D. A. The unreasonable ineffectiveness of the deeper layers. arXiv preprint (tdi::2403.788877 2024. 10
[10] Gupta, Y., Jaddipal, V. V., Prabhala, H., Paul, S., and Platen, P. V. Progressive knowledge distillation of stable diffusion xl using layer level loss, 2024. 2, 4, 8, 10, 16, 21, 26
[11] Heo, B., Kim, J., Yun, S., Park, H., Kwak, N., and Choi, J. Y. A comprehensive overhaul of feature distillation. In ICCV, 2019. 1,
[12] Hessel, J., Holtzman, A., Forbes, M., Bras, R. L., and Choi, Y. Clipscore: A reference-frre evaluation metric for image captioning. In EMNLP, 2021. 4
[13] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, Gans trained by a two time-scale update rule converge local nash equilibrium. In NeurlPS, 2017. 4
[14] Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in neural network. arXiv preprint arXiv: 1503.000000 02531, 2015.
[15] Ho, J. and Salimans, Classifier-free diffusion guidance. arXiv preprint arXiv
2022. 8, 17
[16] Ho, J., Jain, A., and Abbeel, Denoising diffusion probabilistic models. In NeurlPS, 2020. 4, 16, 17
[17] Huang, K., Sun, K., Xie, E., Li, Z., and Liu, X. T2i-compbench: A comprehensive benchmark for woen-world compositional text-to-image generation. NewrlPS, 2023. 4, 16
[18] Ilharco, G., Wortsman, M., Carlini, N., Taori, R., Dave, A., Shankar, V., Namkoong, H., Miller, J., Hajishirzi, H., Farhadi, A., and Schmidt, L. Openclip. https://github. com/ mlf wwwudatiinn/opppeeeeoeo clip, 2021. URL httpp://ddo ottg:1...22///eeddd555533333 16
[19] Jolliffe, T. and Cadima, J. Principal component analysis: review and recent developments. Philosophical transactions of the royal society A: Mathematical, Physical and Engineering Sciences, 37442065),20150222,, 2016. 5, 19

---

[20] Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion-basee generative models. NeurIPS, 2022. 7. 17
[21] Kim, B.-K., Song, H.-K., Castells, T., and Choi, S. On architectural compression of text-to- image diffusion models. arXiv preprint 1.Xi::2305.15790 2023. 2, 4, 5, 8, 9, 16, 17, 18, 21
[22] Kolkin, N., Salavon, J., and Shakhnarovich, G. Style transfer by relaxed optimal transport and self-similarity. In CVPR, 2019. 5
[23] Lin, S., Wang, A., and Yang, X. Sdxl-lightning: Progressive adversarial diffusion distillation. arXiv preprini arXiv:2402. / 3929, 2024. 1, 6, 7, 8, 9, 16, 21, 26
[24] Liu, H., Li, C., Li, Y., and Lee, J. Improved baselines with visual instruction tuning, 2023. 6,
15
[25] Liu, H., Li, C., Wu, Q. and Lee, Y. Visual instruction tuning, 2023. 15
[26]
Loshchilov, and Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv: 2017. 16
[27] Luo, Y., Ren, X., Zheng, Z., Jiang, Z., Jiang, X., and You, Y. Came: Confidence-guided adaptive memory efficient optimization. arXiv preprint arXiv:2307.0m 02047, 2023. 17
[28] Men, X., Xu, M., Zhang, Q.. Wang, B., Lin, H., Lu, Y., Han, X., and Chen, W. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.03853, 2024. 10
[29] Meng, C., Rombach, R., Gao, R., Kingma, D.,, Ermon, S., Ho, and Salimans, T. On distillation of guided diffusion models. In CVPR, 2023.
[30] Peebles, W. and Xie, S. Scalable diffusion models with transformers. In ICCV, 2023. 9, 10, 17
[31] Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01953 2023. 1,3, 4, 6, 7, 8, 10, 16, 17, 19, 29,21,22 26
[32] Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T. Müller, J., Penna, J., and Rombach, R. Stabilityai: Sdxl-base-1.00 1.0. stable-diffusion xl-base-1....e----aa-eeeeee 0, 2023. 17
[33] Radford, A., Kim, W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., al. Learning transferable visual models from natural language supervision. ICML, 2021. 7, 16
[34] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, High-resolution image synthesis with latent diffusion models. In CVPR, 2022. 3
[35] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. Stable-diftusion--11.4 https:// gttgggtthuuu..e
ciffuuiii..........ooo..oooo 2022. 2, 3, 4, 16
[36] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. Stable-diffusion-v220 com
httpp://eet //g:: http://thu...o//taaaiiiiaaee
2022. 3, 8, 16
[37] Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., and Bengio, Y. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6555 2014. 4
[38] Salimans, T. and Ho, J. Progressive distillation for fast sampling of diffusion models. In ICLR, 2022. 1,6
[39] Sauer, A., Lorenz, D., Blattmann, A., and Rombach, R. Adversarial diffusion distillation, 2023. 2, 6, 7, 8, 9, 16, 21,
[40] Schuhmann, C. and Bevan, P. Laion pop: 600,000 high-resolution images with detailed descriptions. 17,21
http:///aaaaceatttttttt//a////// -n-pop. 2023. 2, 6, 7, 15,
12

---

[41] Schuhmann, C.. Beaumont, R., Vencu, R., Gordon, C., Wightman, R.. Cherti, M., Coombes, T. Katta, A., Mullis, C., Wortsman, M., et Laion-aestheics v2. https://laion. ai/blog/ laion aesthetis 2022. 15
[42] Schuhmann, C.. Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C.. Wortsman, M., et Laion-aesthetiic v2 6.5+. hestps:/huuginngtee co/ /od/atasets//ae CctrisstppsSchuuaaanaaaaaaa
5pl..issss 2022. 16
[43] Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., al. Laion-aesthetics v2 6+. https: //huggingfacc. cantaastts/Christtphhhhhhaa
http://ooooeeeeeteeessssosseee c_pluussis__pu,,,..eeeepe 2022. 6, 15, 16
[44] Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T. Katta, A., Mullis, C.. Wortsman, M., et al. Laion-5b: An open large-scale dataset for training next generation image-text models. In NeurlPS, 2022. 7
[45] Shang. Y., Yuan, Z., Xie, B., Wu, B., and Yan, Y. Post-training quantization on diffusion models. In CVPR, 2023.
[46] Shechtman, E. and Irani, M. Matching local self-similarities across images and videos. In CVPR, 2007. 5
[47] Shi, Y.. Wang, P., Ye, J., Long, M., Li, K., and Yang, X. Mydream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308. 2023.
[48] Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. arXiv preprint arXiv:2010.0225 02502, 2020.
[49] Tumanyan, N., Bar-Tal, O., Bagon, S., and Dekel, T. Splicing vit features for semantic appearance transfer. In CVPR, 2022. 5
[50] Tumanyan, N., Geyer, M., Bagon, S., and Dekel, T. Plug-and-pla diffusion features for text-driven image-to-image translation. In CVPR, 2023.
[51] von Platen, P., Patil, S., Lozhkov, A., Cuenca, P., Lambert, N., Rasul, K., Davaadorj, M., and Wolf, T. Diffusers: State-of ofate-of-the-ar diffusion models. https: / hon//uggnngfac//////////o/oo diffusers, 2022. 16
[52] von Platen, P., Patil, S., Lozhkov, A., Cuenca, Lambert, N., Rasul, K., Davaadorj, M., and Wolf, T. Diffusers: State-of-the-art diffusion models. https: : /tt/gtttttbbbt// wtcchuu...oooooaeoeeeeeeeeeee com com/huugingfaaee diffusers/blob wwt.o///ma/////eeeeeeeeeeeeeeeee wwwpaaaaaeieaaaaaaeeeaaaa http://exx__ttiiineeeeeeeeee 2023. 16
text_to_iiaaeeeeeeeoooott sdxl. py,
[53] Wang, F.Y.,, Huang, Z., Bergman, A. W., Shen, D., Gao, P., Lingelbach, M., Sun, K., Bian, W., Song, G., Liu, Y., Li, H., and Wang, X. Phased consistency model. In NeurIPS, 2024. 9
[54] Wang, W. Lv, Q., Yu, W., Hong, W., J., Wang, Y., Ji, J., Yang, Z.. Zhao, L., Song, X.. Xu, J., Xu, B., Li, J., Dong, Y., Ding, M., and Tang, J. Cogylm: Visual expert for pretrained language models, 2023. 6, 15
[55] Wu, X., Hao, Y., Sun, K., Chen, Y., Zhu, F., Zhao, R., and Li, H. Human preference score v2: A solid benchmark for evaluating human preferences text-to-image synthesis. arXiv preprint 2xX6:2230600944 2023. 4, 16, 19
[56] Yang, L., Yu, Z., Meng, C., Xu, M., Ermon, S., and Cui, B. Mastering text-to-image diffusion: Recaptioning, planning, and generating with multimodal 11ms. arXiv preprint arXiv:2401. 20122411110000 11708, 2024.
[57] Yu, J., Xu, Y., Koh, J. Y., Luong, T., Baid, G., Wang. Z., Vasudevan, V., Ku, A., Yang. Y., Ayan, B. K., et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.
2022. 15
[58] Zhang, S., Wang, J., Zhang, Y., Zhao, K., Yuan, H., Qin, Z., Wang, X., Zhao, D., and Zhou, J. 12vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models. arXiv preprint 2rX1v.231..04145, 2023. 1
13

---

Appendix
Contents
Introduction
2 In-depth Analysis: Stable Diffusion XL
3 Three lessons for building an efficient text-to-image model
3.1 Lesson. I: Self-attention based Knowledge Distillation with Efficient U-Net Archi- tecture
3.1.1
Efficient U-Net architecture
3.1.2
Exploring Knowledge Distillation for SDXL
3.2
Lesson 2. Data: the impact of sample size, image resolution, and caption length
3.3
Lesson 3. The influence of Teacher model
Experiments
4.1
Implementation details
4.2
Main results
4.3
Discussion
4.4
Model budget comparison consumer-grade GPUs
10
5
Limitations
10
Conclusion
10
7 Acknowledgments
10
A
Implementation details
15
A.I Data
15
A.2 Training
16
A.3 Inference
17
A.4 Knowledge Distillation for Diffusion Transformer
17
A.5 Detailed formulation of training objectives
17
Additional Analysis
18
B.1
Self-attention based Feature-level Knowledge distillation
18
B.2 Attention visualization for Tab. 1a and Tab. 11b
19
B.3 Feature cosine similarity analysis for Tab. 11c
19
B.4
Implementation details SA-bottom
19
Qualitative results
20
C.1 Represenative prompts in Fig. 1
20
C.2 Comparison to other methods
21
14

---

C.3 Comparison to BK-SDM
21
D Failure cases
21
Societal Impacts
21
A Implementation details
A.1 Data
LAION-Aesthetiics V2 6+ [41; 43] includes some imperfections; thus, we conduct careful data preprocessing. We first filtered out trivial imperfections such as blank text and corrupted images, resulting in 8,483,623 image-text pairs. Despite this, we observed that the text prompts in the LAION dataset are notably brief, which could hinder learning the accurate image-text correspondence.
Instruction:
"Describe the image and its style in very detailed manner."
LLaVA:
"The image is panoramic view a city skyline with Statue of Liberty in the foreground. The statue holding torch and is surrounded body of water_"
LAION caption: "Why Visit New York.?"
city skyline with the Statue of Liberty in the foreground. The statue is holding torch and is surrounded by body of water. The city skyline is filled with skyscrapers, and there are several boats visible in the water. The scene is captured during sunset, giving it warm and serene atmosphere."
LLaVA caption (Pixart-a''s style): "The image is panoramic
LLavA SynCap (Ours): "Why Visit New york.?, ?, The image is a panoramic view of city skyline with the Statue of Liberty in the foreground. The statue holding a torch and surrounded by a body water. The city skyline filled with skyscrapers, and there several boats visible in the water. The scene is captured during sunset, giving it a warm and serene atmosphere."
Figure 9: Sythesizing captions by LLaVA-1.5 [24]. append the synthesized captions by LLaVA to the original ones, leveraging the existing contextual information such proper nouns, e.g., New York.
Synthesized captions, synCap. As some work [57; 1; 5] show that making caption data richer in information improves its generation quality, refining (cleaning) the training caption data and synthesizing more detailed captions corresponding each paired image by large multimodal model, e,g. LLaVA [25; 24J would improve the image-text alignment capability. To do this, we utilize LLa [24] to synthesize captions corresponding to images LAION-Aesthetics V2 6+. As shown in Fig. 9, when we input the instruction to LLaVA to describe the details of the input image, we can get highly augmented captions. It is worth noting that contrary to Pixart-a [5], which replaces original captions with synthesized ones, our approach appends augmented captions to the original ones, leveraging the contextual richness of existing proper nouns (e.g., New York).
LAION-POP [40] has a rather smaller number of images (e.g., 491,567) but it has images with a higher resolution (1274 x 1457) and longer average prompt length (81) which is also generated by LMM models, VML [54] and LLaVA-v1.5 We can download the dataset the Huggingface repository We train the main models, KOALA-Turbo and KOALA- Lrighnnggg,,,ng, LAION-POP dataset Tab. and Tab. 10.
15

---

Table 10: Quantitative comparison to state-of-the-art models with HPSv2 [55] (Left) for visual aesthetics and with BeenCh [17] (Right) for Image-text alignment.
Model
aPtram.
HPSv2
Attribtic
Object Rehaisonhhp
Comples
Average
11-Net
Asimo
Paintings
Photo
Cencept-ont
Aserage
Color
Shape
Texmure
Syenial
Non-qurial
SDM4.v2.0000
0.368
25.2
0.4922
0.1342
U.3672
1oxi-BBse-110 L0 [11)
10.50 08.08
29.02
10:76
50.83
0.6210
0.5909
0.197
0.3133
04000
0.3645
-Tarhe [19]
31.45
17
30.00
50:05
0.5157
06181
0.163
0.3135
0.3968
0.4480
1PXII-Lighningg [33]
2.568
33.6
32.42
52.18
0.5106
0.5816
0.2133
0.3080
0.3954
0.4445
Picant-lppaa [5]
0.68
30.50
32.07
$2.06
04618
0.563
0.5108
0.192
0.3072
0.3991
0.3850
Pixart-siqua [6]
0.68
33.13
30.64
31.59
06112
0.2538
0.3091
04302
0.4612
SSD- 1B [10)
32.90
31.78
52.18
0.5313
0,5914
0.2129
0.3174
04108
0.4497
SSID>Negs [10]
0.748
33.56
12.53
29.65
32:95
0.6455
0.5102
0.6044
0.209
0.3139
0,031
0.4461
KOALA--Trree 111
0.78B
28.57
30.20
8.631
0.1844
0.3141
0.4266
KOALA- Turbo-1B
1.168
31.51
28:21
29.84
0.3192
3.6244
0.1882
0.3148
0,4550
KOALLA-LLccbnnnng- 700M
0.788
32.26
30.09
31.76
31.87
31.50
0.5953
0.3102
0,4585
KO.L.A-Lighhnnnng11
1.168
32.52
30.54
31.90
31.71
0.6706
0.3345
0,5M00
0.2177
0.3114
0.4261
0.4590
A.2 Training
Common training protocol. First, we describe a common training protocol for all experiments in our work. We base framework the officially released SDXL- 1.0 and Diffusers
library [51; 52]. We mainly replace computationally burdened SDXL's U-Net with our efficient U-Net. We keep the same two text encoders, OpenCLIP ViT-bigG [18] and CLIP ViT-L [33], used in SDXL. For VAE, we use sdxe-vae- fp16--fp------ f-16-fix [4], which enables us to use FP16 precision for VAE computation. We initialize the weights of our U-Net with the teacher's U-Net weights at the same block location. We freeze the text encoders, VAF, and the teacher U-Net of SDXL and only fine-tune our U-Net. When training, use discrete-time diffusion schedule [16], size- and crop-conditioning as in SDXL [31], Adam W optimizer [26], a batch size of 128, a constant learning rate of 10-5, and FP16 precision.
For the ablation study on the knowledge distillation strategies in Tabs. 3a, 3b and 11, following the common training protocol except for batch size and training iteration, for fast verification, train our KOALA models for 30k iterations with a batch size of 32 and 1024 x 1024 resolution on LAION-Aestheties V2 6+ [43] dataset using one NVIDIA A100 (80GB) GPU.
For the ablation study in Lesson 2. Data in Tab. 4, following our common training protocol, we train all cases, e.g., (a), (b), and (c) on each dataset (a) LAION-Aesthetie V2 6+ (b) LAION-Aesthetics- V2-6+ with SyПCAP and (c) LAION-POP with a batch size of 128 and 1024 x 1024 resolution for 100K iterations using 4 NVIDIA A100 (80GB) GPUs.
For the ablation study in Lesson 3. Teacher in Tab. 4, following our common training protocol, we train all cases on the LAION-POP dataset for 100K iterations using 4 NVIDIA A100 (80GB) GPUs. In particular, for using SDXL-Turbo as a Teacher model, SDXL-Turbo was originally trained with 512 x 512 resolution, so we perform KD- -raiiiin using the SDXL-Turo teacher with 512 512 resolution. We use the officially released checkpoint in Hugginface. In contrast, for using SDXL-Baae and SDXIL-Lightnin as Teacher models, 1024 x 1024 resolution. For the SDXI.-L.ghtniing teacher model, we use the officially released 4-step unet-checkpoint in Hugginface.
we follow their original papers with
For the main results in Tab. 6, following our common training protocol, we finally train KOALA- Turbo and KOALA-Lightning equipped with two KOALA U-Net backbones with a batch size of 128 for 500K iterations on LAION-POP dataset using 4 NVIDIA A100 (80GB) GPUs.
For a fair comparison to our counterpart BK [21] in Tab. 7, we train SDM-Small proposed in BK- SDM [21] with our saff-attention-baseee KD using SDM-v1.4 [35] as Teacher model. following the BK-SDM training recipe for 50K iteration with a batch size of 256 on LAION-Aesthetics V2 6.5+ [42]. On the other hand, train our KOALA-IB IB U-Net with the BK method and compare it with our KD method under the same training setup such as the same SDXL-Base-I 1.0 Teacher model, following the common training protocol except for 50K training iterations.
16

---

Table 11: Analysis of feature level knowledge distillation of U-Net in SDXL [31]. SA, CA, and FFN denote self-attention, cross-attention, and ferw-forward net in the transformer block. Res is a convolutional residual block and LF denotes the last feature (same in BK [21]). For the ablation study, we train our KOALA-IB IB as student U-Net for 30K iterations with a batch size of 32.
Distill type
HPSv2
Distill
HPSv2
SA loc.
HPSv2
Comhination
HPSv2
SD-loss
25.53
SD-loss
25.53
SA-bottom
26.74
Buscline (SA only)
26.74
SA
26.74
DW-2
25.32
SA-inter
26.58
SA + LF DW-I UP-3
26.98
CA
26.11
DW-3
25.57
SA-up
26.48
SA + Res at DW-1 & UP-3
26.94
Res
26.27
Mid
25.66
SA + LF all
26.83
FEN
26.48
UP-1
26.52
SA Res all
26.80
LF
26.63
UP-2
26.05
AACAA-Res+PNN-Lalll
26.39
(a) Distillation type
(b) Distill stage
(c) SA location.
(d) Combination.
A.3 Inference
When generating samples, we also generate images with 1024 x 1024 and 512 512 for KOALA- Lightning and KOALA-Turbo, FP16-precision and sdxl-vae-fpi6-f1 [4] for VAB-decoder. Note that in the SDXL original paper [31], authors used DDIM sampler [48] to generate samples in the figures while the diffuser's 's official SDXL code [32] used Euler discrete scheduler [20] as the default scheduler. Therefore, we also use the Euler discrete scheduler for generating samples. For KOALA- Lighting and KOALA-Turbo, we infer with 10 denoising steps. we set classifier-free guidance [15] to 3.5. For measuring latency and memory usage in fair conditions, we construct the same software environments across machines with different GPUs. Specifically, use Pytorch==v2. 1.2 and for a fair comparison, we don't 't use any speed-up tricks such as torch. compile and quantization.
A.4 Knowledge Distillation for Diffusion Transformer
Following our KD strategies, we first compress Diffusion Transformer (DiT [30]) backbone, DiT-XL, in Pixart-E [6] by reducing the number of 28 transformers layers to 14 based on our finding in Tab. 11c, building DiT-M. Specifically, we select the bottom layers, e.g., from 0 to 14-th layers and remove the upper layers, e.g., from 14-th to 27-th layers. We maintain the same embedding dimension size of 1152 for DiT-M as in DiT-XI., resulting in a model size of approximately 313M for DiT-M (compared to 1M for DiT-XL). Then, we initialize the weights of DiT-M from the corresponding layers in DiT-XL. For training, we optimize the same objective as ours: Lask + LouKD + LEaaKD- We conduct the ablation study Tab. by changing the feature location from the teacher model. Using the training recipe, codebase and MixArt-Sigmas -eeraa-XL------ XL-2--122---- checkpoint in Pixart-E, we train DiT- M with 512 512 resolution, batch size of 192, multi-scale augmentation, CAME optimizer [27], a constant learning rate of 2e-5, and FP16 precision for 50 epochs on LAION-POP [40] dataset.
A.5 Detailed formulation of training objectives
We detail the two objectives, the Lusk and Loat which are omitted in the main paper. First, the target task loss Lask to learn reverse denoising process [16] summarized as:
Ltask
min E E,,,eee - eso (2t, t, c)l
(2)
So
where 6 is the ground-truth sampled Gaussian noise at timestep t, e is text embedding as condition, and (-) denotes the predicted noise from student U-Net model, respectively. Second, the output- level knowledge distillation (KD) loss is formulated as:
LOOKD
min E. B.,ett.eleen, (x,.ellee,,,,,,,,,, c), - es, (z, c) So
(3)
where en, (.) denotes the predicted noise from each U-Net in the teacher model.
17

---

big hippoppotamus and small rabbit."
LF (8K)
SA (Ours)
(a) Generated Image
(b) PCA Analysis
"A kitten and a dog sat sidl by side"
UWA2
DW3
MD
UP2
(BK)
SA (Ours)
(a) Generated Image
(b) PCA Analysis
Figure 10: Extended analysis on self-attention maps of distilled student U-Nets. (a) Generated images of LF- and SA-based distilled models, which are BK-SDM [21] and our proposal, respec- tively. In BK-SDM's result, a rabbit or dog is depicted like hippopotamus or cat, repectively (i.e., appearance leakage). (b) Visualization of PCA analysis results. Note that from the UP-1 stage, the SA-based model attends to the corresponding object (i.e., rabbit or dog) more discriminatively than the model, demonstrating that self-attention-bat KD allows to generate objects more distinctly.
B Additional Analysis
B.1
Self-attention based Feature-level Knowledge distillation
In this section, we further perform analyses for how to effectively distill feature information from the Teacher model.
Which SA's location is effective in the transformer blocks? At the lowest feature level, the depth of
the transformer blocks is 6 for KOALA-IB we need decide which locations to distill from the 10
transformer blocks of teacher U-Net. We assume three cases for each series transformer blocks; (1)
SA-bottom: { E 2, 4,5)). (2) SA-interleave: {ft E {1,3, 5, 7, 10}}, and (3) SA-up: ( 1 E (6. 8, 9, 10}) where 1 is the number of block. Tab. 1 shows that SA-bottor
performs the best while SA-up performs the worst. This result suggests that the features of the early blocks are more significant for distillation. A more empirical analysis is described in App. B.3.
Therefore, we adopt the SA-bottom strategy in all experiments.
Which combination is the best? In SDXL's U-Net, as shown in Fig. 2, there are transformer blocks at the highest feature levels (e.g., DW-1&UP-3); consequently, self-attention features cannot be distilled at this stage. Thus, we try two options: the residual block (Res at DW-1&UP-33 and the last feature (LF at DW-1&UP-3) as BK-SDM [21]. To this end, we perform SA-based feature distillation at every stage except for DW-1 and UP-3, where we use the above two options, respectively. In addition, we try additional combinations: SA+LF all, SA+Res all, and SAFCA+Ree+FFN-LL all where all means all stages. Tab. 1id demonstrates that adding more feature distillations
18

---

1.0
DW
Feature Cosine Similarity
MID
0.9
0.85
0.8
0.0
0.2
0.8
1.0
Normalized Layer Index
Figure 11: Feature cosine similarity analysis. We plot the cross-layer cosine similarity against the normalized layer indexes of transformer block.
to the SA-absent stage (e.g., DW-12UP-33 consistently boots performance, and especially LF DW1&UP3 shows the best. Interestingly, both +LF all and +Res all are worse than the ones at only DW-1&UP-3 and SA+CA+Res+FFN-LF all is also not better, demonstrating that the SA features not complementary to the other features.
B.2 Attention visualization for Tab. 1la and Tab. 11b
In Section 4.3 of the main paper, we provide empirical evidence demonstrating the paramount importance of self-attention features in the distillation process. Our findings particularly highlight the sigmificant impact of specific self-attention (SA) stages (e.g., UP-1&UP-2) on enhancing performance. To support these results, we extensively analyze self-attention maps in the main paper. To complete the analysis, we expand our Principal Component Analysis [19] (PCA) on self-attention maps to encompass all layers in Fig. 10.
As elaborated in the main paper, self-attention begins by capturing broad contextual information (e.g., DW-2&DWV-3 and then progressively attends to localized semantic details (e.g. MID). Within the decoder, self-attentions are increasingly aligned with higher-level semantic elements UP-1&UP-2), such as objects, for facilitating more accurate representation of appearances and structures. Notably, this stage, the SA-based model focuses more on specific object regions than the LF-based model. This leads to marked improvement in compositional image generation performance.
B.3 Feature cosine similarity analysis for Tab. 11c
KOALA models compress the computationally intensive transformer blocks in the lowest feature levels (i.e., DW-3&M d&UP-1 stages). Specifically, we reduce the depth of these transformer blocks from to for KOALA-700M and to 6 for KOALA-IB, For this purpose, we demonstrate that distilling knowledge from the consecutive bottom layers of transformer blocks is simple yet effective strategy (see third finding (F3) in the main paper).
To delve deeper into the rationale behind this strategy, we conducted thorough feature analysis of the original SDXL model [31]. In particular, we investigate the evolution of the features within the transformer blocks. We compute the cross-layer cosine similarity between the output features of each block and those of its predecessors. lower similarity score indicates a significant contribution of the current block, whereas a higher score implies marginal contribution.
For this analysis, leverage the diverse domain of prompts in the HPSv2 dataset [55]. We compute the cross-layer cosine similarity for each stage (DW&Mid&UP) and average these values across all prompts. The results are illustrated Fig. 11. For all stages, transformer blocks exhibit a tendency of feature saturations: While early transformer blocks generally show a significant contribution, later blocks have less impact. This motivates us distill the learned knowledge of consecutive bottom layers of transformer blocks for minimal performance degradation.
B.4 Implementation details of SA- SA-bottom
Fig. 12 illustrates how to choose transformer blocks when distilling self-attention (SA) features DW3 & MID UP1 as described in App. B.I and Tab. 1cc. In Fig, 12, the Transformer blocks (yellow) with
19

---

Transformer blocks (depth=10)
Transformer blocks (depth=6)
Figure 12: SA-bottom illustration in Tab. Ilc.
1000
SDXL (2.578)
KOALA-1B (1.168)
800
KOALA-700M (0.78B)
Parameter
600
DWL,8 R
DW2
DW2, T
DW3 R
DW3
MID R
MID. 7
UP1 T
UP2 R
UP2,1
UP3 R
Stage
Figure 13: Dissection of U-Net in SDXL. DWi and UPi indicate i-th stage of the down and the up block, and and T denote the Residual block and Transformer block, respectively.
a depth of 10 from the original SDXL's U-Net teacher model, and the Transformer blocks (blue) with depth of 6 is from our KOALA-IB's U-Net student model. For SA-bottom in Tab. 11c, we perform feature distillation by selecting consecutive blocks from the teacher model's transformer blocks, starting with the first one, and comparing to each transformer's self-attention (SA) features from the student model's transformer blocks.
C Qualitative results
C.1 Represenative prompts in Fig.
We use the following prompts for Fig. From left-top to right-bottom:
A 4k DSLR photo raccoon wearing an astronaut suit, photorealistic.
A koala making latte art.
A highly detailed zoomed-in digital painting of a cat dressed as a witch wearing a wizard hat in haunted house, artstation.
Cartoon of a cute hedgehog with tangled fur, standing character, looking surprised and awkward standing in dirty puddle, dark circles under its eyes due to lack of sleep, depicted with comic exaggeration, spotlight effect highlighting unkempt spines, use of vivid colors, high-definition digital rendering.
20

---

A photorealistic render of an origami white and tan mini Bernadoodle dog standing in surrealistic field under the moonlit setting.
Peter Pan aged 60 years old, with black background.
A teddy bear wearing a sunglasses and cape is standing on the rock. DSLR photo.
A photograph of sloth wearing headphones and speaking into high-end microphone in recording studio.
More qualitative results are illustrated in Figs. 14 to 16.
C.2 Comparison to other methods
We compare our KOALA-Light LOALA-Lightning with SDXL-Base-L.0 [31], SDXL-Lightning [23], SSD-IB [10] and Vega [10] using 1024 x 1024 resolution in Fig. 17. In addition, we compare our KOALA- Turbo with SDXL-Turo [39] using 512 512 resolution in Fig. 18.
C.3 Comparison to BK-SDM
In addition to the quantitative comparisons the main paper, we also provide qualitative comparison with BK-SDM [21]. As illustrated in Fig. 19, BK-SDM occasionally overlooks specific attributes or objects mentioned in the text prompt and generates structurally invalid images. On the contrary, our proposed model consistently generates images with enhanced adherence to the text, showcasing superior ability to capture the intended details accurately.
D Failure cases
Fig. 21 illustrates that the KOALA-Lightning-ID model faces challenges in rendering legible text (the first row), accurately depicting human hands (the 2nd row), and complex compositional prompts with multiple attributes (the third row). We conjecture that these limitations may stem from the dataset, LAION-POP dataset [40], we used train, whose images don't 't have enough of those styles.
Rendering long legible text. We have observed that the model has difficulty in synthesizing long- legible texts in the generated image. For example, shown in Fig. 21 (Ist-row), it renders unintended letters and sometimes doesn't generate correct characters. Complex prompt with multiple attributes. When attempting to compose an image using prompts that include various attributes of an object or scene, KOALA sometimes generates instances that do not perfectly follow the intended description. human hands details. While we have confirmed that the model excels representing human faces, it still struggles to render human hands. This may be because we haven'l 't learned enough about the structure of the hand itself, as human hands are more often seen in conjunction with other objects or situations than in isolation.
E Societal Impacts
The text-to-image generation model, KOALA models developed in this study, has the potential to significantly advance the field of visual content creation by enabling the automated generation of diverse and creative images from textual deseriptions. This innovation has numerous applications across various industries, including entertainment, education, advertising, and more. However, is crucial to acknowledge and address the potential risks associated with the misuse of such technology, particularly concerning the generation of Not Safe For Work (NSFW) content.
To mitigate the risks associated with NSFW content, our model leverages the NSFW content detection capabilities provided by Huggingface and the transformers library. By integrating these tools, we ensure that any potentially harmful, violent, adult content generated by KOALA is identified and filtered out before reaching the end-users. Specifically, the NSFW score is calculated for each generated image, and images with scores exceeding predefined threshold automatically discarded. This approach helps maintain the ethical and responsible use of our technology, promoting a safer and more positive user experience.
The adoption of such filtering mechanisms is essential to prevent the spread of inappropriate content and to adhere to ethical standards in AI development. By implementing robust NSFW detection and
21

---

filtering strategies, we demonstrate our commitment to addressing broader societal concerns and promoting the responsible use of AI Al-generated content.
In conclusion, while the KOALA model offers significant benefits and opportunities for innovation, recognize the importance of proactive measures to prevent its potential misuse. Our integration of NSFW content detection serves as a crucial safeguard, ensuring that our contributions to the field align with ethical guidelines and societal values.
22

---

The underground, the gnomes are digging diamonds and gold, the cave is brightly with magic lights, the gnomes magic light is everywhere, the gnomes have dimonds, photorealistic, dslr photo
magical book with glowing floating above open pages
A 4K dsir photo a hedgehog sitting in small boat in the middle of pond. There are a few leaves in the background.
Portrait photo of beautiful female cyborg from 1920, trending artstation
Figure 14: Qualitative results of KOALA-Lightning-.. models with 1024 resolution.
23

---

A 4k dsir photo raccoon wearing sutr photorealistic.
koala making latte art.
An baby owl tree, 3d animation.
A highly detailed zoomed-in digital painting of cat dressed witch wearing wizard hat haunted house, artstation.
Figure 15: Qualitative results of KOALA-Lightningg 1B models with 1024 resolution.
24

---

Cartoon a cute hedgehog with tangled fur, standing character, looking surprised and awkward standing dirty puddle, dark circles under its eyes due lack sleep, depicted comic exaggeration, spotlight effect highlighting mnkemptssiness use vivid colors, high-definition digital rendering.
cute fluffy sentient allen from planet the Andromeda galaxy, the alien large innocent eyes digitigrade, high detail.
A photograph of a sloth wearing headphones and speaking into a high-end microphone a recording studio.
Iridescent crystal cave, flying lizards creatures from hell
Figure 16: Qualitative results of KOALA-Lightning. 1B models with 1024 resolution.
25

---

SDXL-Base110
SOXL-Lighanin
5SD-18
Segmind-Vega
KOALA-Lighoring
A 3d art animation of cute baby raccoon walking on Mars, wearing astronaut suit, with many stars in the sky.
A cute magical flying dog, fantasy art, golden color, high quality, highly detailed, elegant, sharp focus, concept art, character concepts, dig ital painting, mystery, adventure.
A photograph of a sloth wearing headphones and speaking into high-end microphone in a recording studio.
teddybear skateboard Times Square.
A bird known for its distinctive blue and orange plumage. The kingfisher perched branch, its body angled slightly the left poised take flight at any
close-up photo of person. The subject is She blue coat with gray dress undernett. She has blue eyes and blond hair, and wears pair of earrings. Behind are blurred city buildings and streets.
Figure 17:
Qualitative comparison with state-of-the-art SDXL models: SDXL-Base-1.00 [31], SDXL-Lighhting [23], SSD-1B [10] and SSD-Vega [10] with 1024 resolution.
26

---

SDXL-Turbo KOALA-Turbo-18
TULL--Turbo-100
SDXL-Turbo
Koala-Turbo-18
KOALA-Turba-700m
Drone view of waves crashing against rugged clifss along Big Sur's Garay Po Point beach. The crashing blue waters create white-tipped waves, while the golden light of the setting sun illuminates the rocky shore.
P Pirate ship trapped in a cosmic meelstrom nebula.
A leddybear on skateboard Times Square.
baby raccoon walking on wearing an astronaut suit with many stars the sky.
Oil painting of black hole and astronaut.
A cute magical flying dog, fantasy art. golden color, high quality, highly detailed, elegant, sharp focus, concept art, character concepts, digital painting, mystery, adventure.
origami eagle flying through living room.
illustration of robotic wolf, wearing sunglasses and hat, cold color, raining, dark, mist smoke, extremely detailed, photorealistic.
A high-contrast photo of panda riding a horse. The panda is wearing wizard hat.
photograph a sloth wearing headphones and speaking into high-end microphone in recording studio.
Figure 18: Qualitative comparison of SDXL--Turhe [39] and our KOALA- Turboo models with 5122 resolution.
27

---

BK-SDXL-700M
KOALA-700M
BK-SDXL-700M
KOALA-700M
squarr neal circuler carister
recianguler hung sink
dog chasing ball and having fun the park
A boat calling on lake and birds are flying above
liue book
blue
Around
Figure 19: Qualitative comparison between BK-Base-700M KOALA-700M (ours). These models are trained with the same training recipe, such as the LAION-A+6 dataset and SDX-Base-LL 1.0 teacher model.
28

---

2-Step PCM-KOALA-TOOK
4-Step
POM-SDXL
PCM-KOALA-18
PCM-SDXL
PCM-KOALA-TOOM
PCM-KDALA-1B
4K dsir photo of raccoon wearing astronaul suit, photorealistic.
A highly detailed zoomed-in digital painting of a cat dressed witch wearing wizard hat haunted house, artstation.
magical glowing runes floating open pages
dsir photo of hedgehog sitting a small boat the middle of pond. There are few leaves background.
Figure 20: Qualitative comparison between PCM-SDXL and our PCM-KOALA models with 10242.
29

---

BOCGR
chriasm ealbaas
VEYD DEP LARNiNIg LSONEYION
VERON
A portrait photo of kangaroo wearing orange hoodie blue sunglasses standing on the grass front
Frog sitting in 1950s diner wearing
Beautiful pixei art of Wizard white speech saying "Dilfusion"
green sign that "Very Deep Learning edge of the Grand Canyon. uffy white clouds are in the sky.
jacket top hat. on the table is giant burger small sign that says "frogyy fridays"
Sydney pera House holding a sign on the chest that says "Welcome Friends?
up handpalm waler.
clase-up photo hand full of maple leaves.
photorealistic image young girl blowing bubbles in park, colorful flowers and a big blue sky in the background. Shot from a close-up angle capture the sense of playfulness and innocence
photo lady with her hand raised in greeting.
A wombat in yellow beach chair, while sipping martini, The wombat wearing a white panama hat and floral Haweiian shirt, Out-of-focus palm trees in the background. DSLR photograph. Wide-angle view.
baby penguin wearing blue hat, red gloves, and purple sweater. Running ice land
A photo of one black handbag and two green wallet the wooden floor
A brown big dog wearing sunglasses standing on the left and white smal cat wearing heimet sitting on the right
of KOALA-LLghhting-IB.. KOALA-Lightning-ID model faces challenges in complex scenarios, such as rendering legible text (Ist row), accurately depicting human hands (2nd row), and complex compositional prompts with multiple attributes (3rd row).
Figure 21: Failure
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We made sure that the claims made in the abstract and introduction accurately reflect our contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of our method in Sec. 5, including failure cases generated by our KOALA and our simple and specific compression method.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
31

---

Answer:
[NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide detailed instructions of how to reproduce all of the experiments, hyperparameters, and the models and datasets used, all of which are open-source.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
32

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
We already provided where we can download the training datasets and evalua- tion benchmarks. However, we cannot provide the training code and checkpoint weights due to the internal policy our company. After getting permission from our company, we will release the trained weights in Huggingface to generate images.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (uttps://nips.. cc/
Seaiiees/ooeeuubi
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips http:/ppwc.g...ee/eeeppppo www./o/eemmiiiiioooooooooooo
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We include details for dataset preparation and experimental settings in App. A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: In our study, we compared various models using quantitative evaluations, specifically focusing HPSv2 and Compbench. The evaluation of HPSv2 required gen- erating 2,400 images, while Compbench required 24,000 images, resulting in significant inference cost that made repeated inferences challenging. Despite this, we believe that our performance evaluations are reliable as the performance deviations are expected to be minimal based on the sufficient number of test images used in our evaluations. Furthermore, we did not cherry-pick any images during the quantitative evaluation due to the sheer volume of images generated and evaluated. This extensive evaluation approach ensures that the reported performance metrics are representative and robust.
33

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the relevant details in App. A and Sec. 4.4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https://neurs www.rps.....p/.. ccps.cc////e cc/public/Eeeeecccc
Answer: [Yes]
Justification: We have thoroughly reviewed the ethics guidelines and we have made sure to preserve anonymity for the review process.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
34

---

Answer: [Yes]
Justification:
We already addressed the broader impacts in App. E.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: As text-to-imagee generation model, KOALA, addresses the potential risks of generating Not Safe For Work (NSFW) content, which includes harmful, violent, or adult imagery. To mitigate these risks, KOALA integrates NSFW content detection capabilities provided by Huggingface and the transformers library. By calculating the NSFW score for each generated image and filtering out those that exceed predefined threshold, effectively prevents the creation and dissemination of inappropriate content. This approach ensures the ethical and responsible use of AI technology, aligning our contributions with societal values and ethical guidelines.
model
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
35

---

Justification:
we have clearly cited the teacher models used for training KOALA: SDXL- Base, SDXL-Turbo, and LDXh-Lightning the main paper and Appendix.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi whoodede com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
Justification:
We didn't provide new assets except for main paper.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We have not used crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
36

---

Answer:
[NA]
Justification:
We have not used crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
37


Reviews:
[{"review_id": "uj2fWkmlO9", "reviewer": "Reviewer_dMSC", "summary": "1. The authors propose two different designs for an efficient denoising U-Net based on SDXL.\n\n2. The authors present three empirical lessons for developing efficient U-Net designs.", "questions": "Please refer to the \"Weaknesses\" section.", "limitations": "The authors adequately address the limitations in section 5.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors present empirical findings to justify their design considerations and provide further analyses to examine the effects of those choices.\n\n2. The authors conduct a systematic analysis with extensive quantitative and qualitative experiments, comparing their results with different baselines.", "weaknesses": "1. The proposed method is specific to the SDXL U-Net and is based on heuristics, rather than presenting a generalizable approach. This makes the paper resemble a technical report more than an academic paper. It would be beneficial if the authors could further clarify their novelty and contributions in terms of academic research.\n\n2. The authors present some failure cases of their methods, including rendering long legible text, complex prompts with multiple attributes, and human hand details. Do the teacher models and other baseline models also suffer from the same problems? Were any of these failure cases aggravated in the proposed model to some extent? If so, what could be the reasons?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for thoroughly reviewing our work and for your insightful and helpful suggestions for improving our paper. We provide our response in the following. We have made every effort to address all your concerns. Please don’t hesitate to let us know if your concerns/questions are still not clearly resolved. We are open to active discussion and will do our best to address your concerns during the discussion period.\n\n---\n\n> **W1-1. specific to the SDXL U-Net and heuristics, not a generalizable approach**\n\n&rarr; Please refer to `G1` section in the general response. \n\n> **W1-2. novelty and contributions to the academic community**\n\n&rarr; **`Novelty`**: We believe that our self-attention-based distillation approach for text-to-image model compression is novel, supported by two key points.\n- To the best of our knowledge, our approach is the **first** to identify that self-attention is the most crucial element for feature distillation in T2I model compression.\n- The self-attention-based distillation approach can be **generally** applied across various architectural designs of diffusion models (e.g., U-Net-based and transformer-based) that consist of transformer blocks with self-attention layers.\n\n&rarr; **`Contribution`**: We believe that we have made two contributions to the academic community from the perspective of **practical impact**.\n\n- **An efficient T2I Backbone for Consumer-Grade GPUs**: \n    - Many downstream tasks, such as personalized image generation [1] and image-editing [2], leverage SDXL-Base as a de facto backbone due to its open-source model and superior performance. However, the SDXL model barely works on GPUs with more than 11GB of VRAM due to its substantial model cost, and its slower latency may cause delays in the outcomes of new research using it. To overcome this limitation, our efficient KOALA T2I backbones can serve as a cost-effective alternative, facilitating further downstream research, especially for those working in limited-resource environments. Furthermore, our efficient KOALA models can assist practitioners and designers by enabling rapid ideation and supporting their creativity with low latency in a consumer-grade GPU environment.\n\n- **An effective recipe of distillation for T2I model compression**: \n    - We presented three key lessons in our paper: i) self-attention-based distillation strategy, ii) data considerations, and iii) the influence of the teacher model. Additionally, through an in-depth analysis of each lesson, we offer general insights into popular U-Net-based diffusion models, such as the role and computational burden of each layer. We strongly believe that the lessons and insights we provide will benefit researchers and developers striving to make T2I models more efficient. \n    - In the large language models (LLMs) field, larger models continue to emerge rapidly, and there have been many efforts to compress these models. Similarly, new and larger T2I models have also been developed. However, unlike in the LLM field, there have been relatively few attempts to compress T2I models due to factors such as the high cost of training (i.e., healing) and data scarcity (i.e., high-resolution and expensive copyright). In this context, we believe that our lessons can be particularly helpful for the community, especially for those working with resource constraints.\n    - It is important to note that research providing crucial **recipes** for effective training, alongside proposing entirely novel methods, has historically played a significant role in revolutionizing their respective fields and boosting overall performance. For instance, Improved GANs [3] for GANs, MoCo-v3 [4] for self-supervised learning, DeiT [5] for transformers, and ConvNext [6] for CNNs have all made substantial impacts. Similarly, we hope our work brings valuable insights and benefits to the research community.\n\n\n> **W2. Discussion on failure cases**\n\n&rarr; Please refer to `G2` section in the general response. \n\n---\n[1] Ruiz et al. DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation. ICLR 2023  \n[2] Zhang et al. Adding Conditional Control to Text-to-Image Diffusion Models. ICCV 2023  \n[3] Salimans et al. Improved Techniques for Training GANs, NeurIPS 2016  \n[4] Chen et al. An Empirical Study of Training Self-Supervised Vision Transformers. ICCV 2021  \n[5] Touvron et al. Training data-efficient image transformers & distillation through attention. ICML 2021  \n[6] Xie et al. A ConvNet for the 2020s. CVPR 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "suAT86Wcf8", "reviewer": "Reviewer_C2ZB", "summary": "This paper presents a set of empirical guidelines to use when distilling Stable Diffusion XL (SDXL) when computational/data resources are limited. The presented guidelines focus on 1) identifying which transformer blocks to drop, 2) what features are the best to distill, and 3) which publicly-available datasets are the best to use. The paper presents analyses for all guidelines, including ablations and comparisons to what other popular distillation approaches do. Insights are extracted from these analyses and used to motivate the guidelines. Performance results are presented showing that the guidelines result in distilled models that are quite competitive with the parent/other distilled models yet are much more computationally efficient.", "questions": "1) Sec 3.1.1. mentions both \"block removal\" as well as \"layer-wise removal\". However, the text describes layer-wise removal as \"reducing the number of transformer layers\". I found this vernacular confusing. It is common practice to let a \"layer\" mean a \"block\", so both approaches sounds as if they are doing the same thing: removing blocks. I think (?) when the text refers to \"block removal\" (108) it means \"removal of 100% of the blocks in a sequence\" but when it refers to \"layer-wise removal\" (line 117) it means \"removal of *less* than 100% of the blocks of the sequence\". Is this understanding correct? If so, then I recommend re-writing this section to make it more clear that \"block removal\" is the special case of \"layer-wise removal\" when all blocks are removed.\n\n2) Table 10d shows that adding DW-1 and UP-3 feature distillation to SA + LF further boosts performance. But the formal paper stops at suggesting SA + LF3 is optimal. Why?\n\n3) Nit: The only time I see the \"KOALA\" acronym explained is implicitly in a couple of the figure captions. Adding it somewhere in the introduction would be nice.\n\n4) Nit: Line 199 uses the phrase \"in the second row of the table\" before any table has been presented to the reader.", "limitations": "The paper presents the commonly suggested limitations of stable diffusion -- it doesn't generate text well and has problems with multiplicity.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main topic of this paper -- how to more easily generate cheaper SDXL distillations -- is of much relevance to virtually all associations. This paper does a thorough job of exploring many distillation aspects. The analyses it reports give good insights into what aspects are most important. For example, the feature distillation ablation section shows that self-attention are the best feature to distill. The authors then use those results to motivate removing fewer transformer decoder blocks than encoder ones. Fig 3c is compelling and really drives home the primary role of self attention in distillation. The suggested recipe for removing blocks goes beyond just a basic approach of removing all blocks in a stage, as some other distillation approaches suggest. I also thought the insights into the various LAION datasets was interesting and that the conclusion of using few high res images with detailed prompts is better than more low res with with short (or detailed) prompts is useful. Further, the performance results from the distilled models are quite competitive with other variants.", "weaknesses": "The guidelines this paper presents are very focused on specifically SDXL. It is unclear if any of them would apply to other transformer-based models. Whatever the next version after SDXL is created, it is unclear how relevant these guidelines would be.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for thoroughly reviewing our work and for your insightful and helpful suggestions for improving our paper. We provide our response in the following. We have made every effort to address all your concerns. Please don’t hesitate to let us know if your concerns/questions are still not clearly resolved. We are open to active discussion and will do our best to address your concerns during the discussion period.\n\n---\n\n> **W1. focused on specifically to SDXL & would be appled to transformer-based models** \n\n&rarr; Please refer to `G1` section in the general response. \n\n> **Q1.  Confusion Between the Terms `Block` and `Layer`**\n\n&rarr; The SDXL-Base architecture consists of convolutional residual blocks and transformer blocks, each of which is composed of multiple transformer layers. A potential point of confusion is the distinction between transformer `blocks` and transformer `layers`. Our intention is to use ‘block’ as a higher-level concept, referring to a group of ‘layers.’ Therefore, we will reiterate this definition in the manuscript to avoid any misunderstanding. Thank you for your comment. ”\n  \n\n> **Q2. SA+LF combination**\n\n&rarr; We apologize for the confusion. The final feature combination for the distillation training is self-attention (SA) from all transformer layers plus the last feature (LF) from the highest feature level (DW1+UP3), as shown in `Fig. 2` and `Tab. 10d`. Using only SA features already shows superior performance compared to using only LF in BK-SDM. However, we have found that adding the last features at DW-1 and UP-3, where there are no self-attention layers, further boosts performance. To explain the cause of this confusion, due to space constraints in the manuscript, we moved the optimal feature combination to the appendix along with `Tab. 10(d)` and missed mentioning SA+LF in the main text. We will clarify this and update the manuscript. We appreciate your comment once again.\n\n\n\n> **Q3. \"KOALA\" acronym**\n\n&rarr; Thank you for your kind advice. As you noted, the explanation of the \"KOALA\" acronym is either implicitly included in the figure captions or found later in the paper (lines 237-238). We will refine and move the explanation to the introduction section in the final version.\n\n\n> **Q4. Tab.4**\n\n&rarr; We intended to use the phrase “in the second row of the table” to indicate that the second data source is demonstrated in the second row of Table 4. However, as the reviewer noted, the current demonstrations might cause confusion for readers. Therefore, we will revise the notation (to ensure consistency of data source notation both in the table and the document) and clarify the phrasing in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f0bJMtILKs", "reviewer": "Reviewer_EitJ", "summary": "This paper presents KOALA, a pair of efficient text-to-image synthesis models that reduce computational and memory requirements compared to the base model. This paper achieves this through three key innovations: knowledge distillation into a compact U-Net, the strategic use of high-resolution images and detailed captions in training data, and the employment of step-distilled teachers to enhance the learning process. The resulting models, KOALA-Turbo and KOALA-Lightning, demonstrate faster inference times and the ability to generate high-quality images on consumer-grade GPUs.", "questions": "In the \"Lesson 3\" section, it is puzzling that SDXL-Base, as a teacher model, performs the worst. Intuitively, a better teacher model should have greater potential to distill a superior student model. I am curious if the results presented in the paper could be attributed to a suboptimal distillation approach.", "limitations": "The authors have discussed both limitations and potential negative social impacts.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper is easy to follow.\n2.\tThis paper offers a practical solution for generating high-quality images with reduced computational requirements.", "weaknesses": "1.\tAlthough KOALA achieves good visual quality, the extent of degradation in text rendering is unclear as it has not been compared with a baseline in terms of text rendering capabilities.\n2.\tFor each baseline, KOALA requires a carefully designed pruning network architecture followed by retraining, which entails a significant acceleration cost compared to training-free methods. However, this paper does not discuss or compare with training-free methods, nor does it compare with step distillation approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for thoroughly reviewing our work and for your insightful and helpful suggestions for improving our paper. We provide our response in the following. We have made every effort to address all your concerns. Please don’t hesitate to let us know if your concerns/questions are still not clearly resolved. We are open to active discussion and will do our best to address your concerns during the discussion period.\n\n---\n\n> **W1. extent of degradation in text rendering & quantitative comparison**\n\n&rarr; Please refer to `G2` section in the general response \n\n> **W2. Comparison with training-free methods**\n\n&rarr; Thank you for your valuable insight. To the best of our knowledge, very recent training-free acceleration methods for diffusion models leverage the redundancy of features over the denoising steps. These methods (e.g., DeepCache [1], Cache Me If You Can [2]) cache and retrieve features of previous steps for efficient inference of the current step. In this rebuttal, we compare our KOALA model with **DeepCache**, which has an official implementation available. Using their code, we measured the inference cost, memory footprint, latency, and performance in terms of HPSv2 and Compbench scores. The results are summarized in the table below. Compared to the original SDXL-Base, DeepCache achieves faster inference speed at the cost of performance and increased memory usage. We speculate that the increased memory usage of DeepCache is due to the caching and retrieval process of features, which requires additional memory. In contrast, when compressing the same SDXL-Base model, our KOALA model reduces both memory usage and latency while achieving superior performance compared to DeepCache. Additionally, due to the step-distilled teacher and longer training, our KOALA model is capable of generating more faithful images with fewer diffusion steps, resulting in lower latency. Furthermore, the training-free method is **orthogonal** to our approach, we expect that combining the training-free method with ours could exhibit a synergy effect, suggesting a promising future direction.\n\n*Note: KOALA-SDXL-Base and KOALA-Lightning denote models trained with the SDXL-Base teacher (1st row of Tab. 5) for 100K iterations and with the SDXL-Lightning teacher (11th row of Tab. 6) for 500K iterations, respectively.\n|Method|Backbone|#Step|Memory(GB)|Latency|HPSv2|CompBench|\n|--|--|--|--|--|--|--|\n|SDXL-Base-1.0|SDXL-Base|25|11.9|3.229|30.82|0.4445|\n|DeepCache [1]|SDXL-Base|25|14.6|1.453|26.32|0.4094|\n|KOALA-SDXL-Base|KOALA-700M|25|8.3|1.424|27.79|0.4290|\n|KOALA-Lightning|KOALA-700M|10|**8.3**|**0.655**|**31.50**|**0.4505**|\n\n> **W3. Comparison with step-distillation methods**\n\n&rarr; We would like to clarify that we have already compared the step-distillation methods, such as SDXL-Turbo and SDXL-Lightning, in `Sec. 4.2` and `Tab. 6` of our manuscript. Below, we re-summarize a comparison with step-distillation methods, including the more recent work, PCM [3], in the following table. We observe that, compared to SDXL-Lightning and PCM at a $1024^2$ resolution, our KOALA-Lightning-700M demonstrates competitive performance while achieving lower latency and memory usage with a model size that is $3\\times$ smaller.\n\n*Table note: SDXL-Turbo generates images with $512^2$ resolution while the other models use $1024^2$ resolution.\n\n|Method|Backbone|Res.|#Step|Param.(B)|Memory(GB)|Latency|HPS| CompBench|\n|--|--|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\n|SDXL-Tubo|SDXL-Base|512|8|2.56|5.6|0.245|29.93|0.4489|\n|SDXL-Lightning|SDXL-Base|1024|8|2.56|11.7|0.719|**32.18**|0.4445|\n|PCM [3]|SDXL-Base|1024|8|2.56|12.1|0.884|30.01|0.4360|\n|**KOALA-Lightning (ours)**|**KOALA-700M**|1024|10|**0.78**|**8.3**|**0.655**|31.5|**0.4505**|\n\n&rarr; Furthermore, we have verified the synergy effect between the step-distillation method (e.g., PCM) and our KOALA backbones during this rebuttal period. We performed the step-distillation training using PCM with our KOALA backbones and obtained results as shown in the table below.  Due to their efficiency, our KOALA backbones enable PCM to further boost speed-up with only a slight performance drop compared to PCM with the SDXL-Base backbone. Additionally, we presented qualitative comparisons of our PCM-KOALA models with PCM-SDXL-Base in `Fig. 1` of the attached `general_response.pdf` file (Please see the PDF file).\n\n*Table note: SDXL-Turbo and SDXL-Lightning do not provide training code, while PCM releases the official training code, allowing us to train PCM with our KOALA backbones.  \n\n|Method|Teacher|Student|#Step|Param.(B)|Memory(GB)|Latency|HPS|CompBench|\n|--|--|--|:--:|:--:|:--:|:--:|:--:|:--:|\n|PCM [3]|SDXL-Base|SDXL-Base|2|2.56|12.1|0.345|**29.99**|**0.4169**|\n|PCM [3]|SDXL-Base|**KOALA-700M**|2|**0.78**|**8.2**|**0.222**|28.78|0.3930|\n|PCM [3]|SDXL-Base|**KOALA-1B**|2|1.16|9.0|0.235 |29.04|0.4055|\n\n> **Q1. a better teacher model should have greater potential to distill a superior student model.**\n\n&rarr; We would like to clarify that in the main state-of-the-art comparison table (e.g., `Tab. 6`) of our manuscript, SDXL-Lightning shows better or comparable performance to SDXL-Base on HPSv2 and CompBench, respectively. Thus, SDXL-Lightning could be considered a better teacher model. Your statement that _“a better teacher model should have greater potential to distill a superior student model”_ aligns with our findings. Indeed, we observed that the KOALA models distilled from SDXL-Lightning achieve better performance than those distilled from SDXL-Base as shown in `Tab 3`.\n\n---\n[1] Ma et al. DeepCache: Accelerating Diffusion Models for Free. CVPR 2024.  \n[2] Wimbauer et al. Cache Me if You Can: Accelerating Diffusion Models through Block Caching. CVPR 2024.  \n[3] Wang et al. Phased Consistency Model. Arxiv (2405.18407) 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KQ31EuTjfC", "reviewer": "Reviewer_9zYS", "summary": "The paper at hand performs a series of ablation studies evaluating the impact of three design decisions on the quality of Text to Image Generation models. Specifically, the authors look at the target of knowledge distillation, training data as well as choice of teacher for distillation. In the end, the authors present a model named KOALA, building upon the insights gained from those ablations. The model is smaller and more efficient than the comparisons presented in the paper.", "questions": "It would be great, if the authors could provide a short precise explanation how the proposed distillation loss works.", "limitations": "The limitations are discussed in the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Improving the efficiency of Text To Image Generation models is of key importance to the research and industrial community, as the size an inference speed of current state of the art models is often a limiting factor for real world applications. \n\nThe paper shows positive results and a good set of comparisons to prior art. \n\nThe paper touches important aspects, such as distillation objectives as well as data selection.", "weaknesses": "My main concern about the paper is its approachability. The presentation of the paper is subpar. Specifically, the paper is very verbose and repetitive, but then on the other hand imprecise and lacking important details. \n\nThe paper could be half its size and get more information across. For example, the paper has an almost two page discussion on knowledge based distillation, but I could not a single precise definition how the proposed approach actually performs. \n\nSimilarly, the entire discussion on the U-Net architecture could be a single table. This would make it much more approachable and easier to understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for thoroughly reviewing our work and for your insightful and helpful suggestions for improving our paper. We provide our response in the following. We have made every effort to address all your concerns. Please don’t hesitate to let us know if your concerns/questions are still not clearly resolved. We are open to active discussion and will do our best to address your concerns during the discussion period.\n\n---\n\n> **W1 & Q1. lacking important details & single precise definition of how the proposed approach actually performs**\n\n&rarr; We apologize for any ambiguity regarding how the proposed distillation method performs. The main reason we explained the distillation part in almost two pages is that we needed to address two main aspects simultaneously: 1) pruning the SDXL-Base model and building efficient KOALA U-Nets, and 2) exploring the effective feature distillation strategy. Furthermore, our key finding for the distillation is that self-attention is the most crucial part. To provide sufficient evidence, we conducted in-depth qualitative (refer to `Fig. 3`) and quantitative (refer to `Tab. 3` and `Tab. 10`) analyses, which took up considerable space in the manuscript.\n\nIn addition, as you suggested, we introduce a brief summary of self-attention-based distillation:\n-  For every transformer block composed of transformer layers at each stage, we allow the student model to mimic the self-attention features from the teacher model (refer to `Fig. 2`).\n- At the highest feature level (e.g., DW-1 & UP-3), since there are no transformer blocks, we compare only the last output features at that stage (refer to `Fig. 2` & `Tab. 10d`).\n\nSince we don’t have enough space to address all of our findings (e.g., four findings) for the effective distillation strategy, we kept only two points in `Sec. 3.1.2` and moved the other findings to Appendix `B`, along with `Table 10`. As a result, it may cause some details to be missing. \n\nLet us describe more details of four findings for our distillation strategy:\n\n1) Among different feature types for distillation, self-attention shows the best performance (`Tab. 3a`) and represents more discriminative features (`Fig. 3` & `Fig. 10` in Appendix).\n2) Self-attention at the decoder (UP-1 to UP-3) has a larger impact on the quality of generated images than the encoder (DW-1 to DW-3). Therefore, we keep more transformer blocks in the decoder part when pruning SDXL-Base’s U-Net (`Tab. 3b`).\n3) When comparing self-attention features, the features of the early layers in the transformer block, which is composed of multiple layers, are more significant for distillation (`Tab. 10c`). This is supported by the feature cosine similarity analysis, which shows that early layers exhibit more distinct feature discrimination (`Sec. B3` & `Fig. 11`).\n4) The combination of the last output features (LF) at the highest feature level and the self-attention features at the other stages shows the best results (`Tab. 10d`).\n\n> **W2. the entire discussion on the U-Net architecture could be a single table.**\n\n&rarr; Thank you for your suggestion. As you pointed out, we will consolidate the U-Net section into a single, more concise and clear table to make it more approachable and easier to understand.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General response
Thank you for thoroughly reviewing our work and for your insightful and helpful suggestions. We have made every effort to address all your concerns. Please let us know if any questions remain unresolved. We are open to discussion and will do our best to address your concerns during the discussion period.

In this general response, we have addressed the common concerns reviewers raised.

> **G1. specific to the SDXL U-Net and heuristics, not a generalizable approach by reviewers** `dMSC` & `C2ZB`

&rarr; We respectfully clarify that although our method has been mainly explored with SDXL, the most popular T2I model at the time of submission, we have already applied our self-attention-based distillation scheme to a transformer-based method, Pixart-$\Sigma$ [1], the only model providing training code at the time of submission. As shown in `Tab. 9` of the manuscript (we include the table below for reference), self-attention-based distillation exhibited superior performance compared to other feature types. These results align with those obtained on U-Net-based diffusion models, demonstrating that our self-attention-based distillation scheme has been proven to be a **general** approach across various architectural designs of diffusion models (e.g., U-Net-based and transformer-based). Given these findings, we can also expect that our self-attention-based distillation can be applied to more recent transformer-based methods with self-attention operations, such as SD3 [2]. Consequently, our approach can serve as a crucial baseline for further T2I model compression research.

Additionally, we would like to explain that the reason we designed an SDXL-tailored method is largely due to the heterogeneous structure of SDXL’s U-Net. It consists of irregularly combined convolutional and transformer layers, unlike pure transformer-based methods such as Pixart [1,3], and SD3 [2], which build a uniform stack of transformer layers. However, since our method is primarily based on in-depth analysis and understanding of each layer of SDXL, we believe our findings will benefit popular SDXL-like T2I models and their applications.


|KD type in Pixart-$\Sigma$|HPSv2|CompBench|
|--|:--:|:--:|
|**SA**|**25.16**|**0.4281**|
|CA|24.94|0.4279|
|FF|24.80|0.4191|
|LF in BK|21.62|0.3527|

> **G2. Discussion on failure cases (e.g., text rendering) by reviewers** `dMSC` & `EitJ`

&rarr; While recent text-to-image diffusion models can generate images of unprecedented quality in general, they still exhibit limited performance in certain specialized cases. These cases include rendering long and legible text, handling complex prompts with multiple attributes, and generating intricate structures such as detailed human hands. The authors of SDXL-Base [4] acknowledge these limitations in their paper, and other baseline models we considered also share these limitations, as shown in `Fig. 2` of the attached `general_response.pdf`. Due to the nature of knowledge distillation, our distilled models inevitably inherit these limitations. Among these models, our KOALA model shows inferior text rendering quality compared to the teacher models, as shown in the table below. We conjecture that the main cause of this inferior performance originates from the **training dataset**(LAION-POP) we used. Although this dataset includes images with high visual aesthetics, the corresponding text prompts rarely describe the text within the images. This leads to difficulties in complex text rendering.

As reviewer `EitJ` suggested, we quantitatively evaluate the text rendering capability on the MARIO-Eval benchmark [5]. Specifically, we utilize existing OCR tools to detect and recognize text regions in the generated images and measure the performance. Using 5,414 test prompt sets in the benchmark, we generated images with rendered text for each model and compared performance using the OCR metrics, as shown in the table below. Our KOALA models show inferior performance compared to SDXL-Base teacher models. In addition, SDXL models achieve much lower performance compared to the specialized T2I model for text rendering, TextDiffuser [5].

|Metrics|TextDiffuser [5]|SDXL-Base|SDXL-Lightning|KOALA-Lightning-700M|KOALA-Lightning-1B|
|--|--|--|--|--|--|
|OCR(accuracy)|0.5609|0.0181|0.0169|0.0123|0.0153|
|OCR(precision)|0.7846|0.0223|0.0228|0.0012|0.0024|
|OCR(recall)|0.7802|0.0463| 0.044|0.0033|0.0065|
|OCR(F-measure)|0.7824|0.0301|0.0301|0.0018|0.0035|

The limitations that our model has inherited remain an open question in the community. To address these limitations, several specialized models have been recently proposed by designing specialized data and architecture. For example, TextDiffuser [5] aims to improve the capability of rendering long, legible text by constructing text-rendered images with OCR annotation, while Paragraph-Diffusion [6] attempts to handle more complex prompts faithfully. Exploring the synergy between these specialized models and our distillation framework could be an interesting direction for future work.

--- 
[1] Chen et al. PixArt-$\Sigma$: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation. Arxiv 2024  
[2] Esser et al. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. ICML 2024  
[3] Chen et al. PixArt-$\alpha$: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis. ICLR 2024  
[4] Podell et al. SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis. ICLR 2024  
[5] Chen et al. TextDiffuser: Diffusion Models as Text Painters. NeurIPS 2023  
[6] Wu et al. Paragraph-to-Image Generation with Information-Enriched Diffusion Model, arXiv 2023  


---
> **Figures in the attached pdf file**

&darr; Please refer to the PDF file.

Meta-review: The paper received Weak Accept, Accept, Weak Accept, and Borderline Accept.

The paper presents a detailed empirical study on how to more effectively distill a text-to-image model such as StableDiffusionXL into a more efficient model by carefully evaluating what to drop, what to tune, and how to best leverage data during the distillation phase. Reviewers are generally convinced of the immediate applicability of the method and potential impact but are worried about longer term impact since most of the paper is very specific to a single model. However there is a small section that seems to have been overlooked at first with DiT diffusion transformers on sigma-pixart. Since the general consensus is to accept this paper by the reviewers I agree with the recommendation but also suggest authors to improve their manuscript by emphasizing what are recommendations from their work that can apply generally.
Decision: Accept (poster)