Title: On Architectural Compression of Text-to-Image Diffusion Models
Venue: NeurIPS 2023
Accepted: no
Keywords: Stable Diffusion

Block Removal

Knowledge Distillation

Network Compression

Text-to-Image Generation
Abstract: Exceptional text-to-image (T2I) generation results of Stable Diffusion models (SDMs) come with substantial computational demands. To resolve this issue, recent research on efficient SDMs has prioritized reducing the number of sampling steps and utilizing network quantization. Orthogonal to these directions, this study highlights the power of classical architectural compression for general-purpose T2I synthesis by introducing a block-removed knowledge-distilled SDM (BK-SDM). We eliminate several residual and attention blocks from the U-Net of SDMs, obtaining over a 30% reduction in the number of parameters, MACs per sampling step, and latency. We conduct distillation-based pretraining with only 0.22M LAION pairs (fewer than 0.1% of the full training pairs) on a single A100 GPU. Despite being trained with limited resources, our compact models can imitate the original SDM by benefiting from transferred knowledge and achieve competitive results against larger multi-billion parameter models on the zero-shot MS-COCO benchmark. Moreover, we demonstrate the applicability of our lightweight pretrained models in personalized generation with DreamBooth finetuning. 


Abstract

Exceptional text-to-image (T2I) generation results of Stable Diffusion models (SDMs) come with substantial computational demands. To resolve this issue, recent research on efficient SDMs has prioritized reducing the number of sampling steps and utilizing network quantization. Orthogonal to these directions, this study highlights the power of classical architectural compression for general-purpose T2I synthesis by introducing a block-removed knowledge-distilled SDM (BK-SDM). We eliminate several residual and attention blocks from the U-Net of SDMs, obtaining over a 30% reduction in the number of parameters, MACs per sampling step, and latency. We conduct distillation-based pretraining with only 0.22M LAION pairs (fewer than 0.1% of the full training pairs) on a single A100 GPU. Despite being trained with limited resources, our compact models can imitate the original SDM by benefiting from transferred knowledge and achieve competitive results against larger multi-billion parameter models on the zero-shot MS-COCO benchmark. Moreover, we demonstrate the applicability of our lightweight pretrained models in personalized generation with DreamBooth finetuning.

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

1 Introduction

Large diffusion models [44, 51, 38, 47] have showcased groundbreaking results in text-to-image (T2I) synthesis tasks, which aim to create photorealistic images from textual descriptions. Stable Diffusion models (SDMs) [46, 47] are one of the most renowned open-source models, and their exceptional capability has begun to be leveraged as a backbone in several text-guided vision applications, e.g., text-driven image editing [2, 23] and 3D object creation [67], text-to-video generation [1, 68], and subject-driven [50, 25] and controllable [37, 71] T2I.

SDMs are T2I-specialized latent diffusion models (LDMs) [47], which employ diffusion operations [17, 59, 30] in a latent space to improve compute efficiency. Within a SDM, a U-Net [49, 6] conducts an iterative sampling procedure to gradually eliminate noise from random latents and is assisted by a text encoder [42] and an image decoder [9, 64] to produce text-aligned images. This inference process still involves excessive computational requirements (see Figure 2), which often hinder the utilization of SDMs despite their rapidly growing usage.

To alleviate this issue, numerous approaches toward efficient SDMs have been introduced.

Meng et al. [35, 34] reduce the number of denoising steps by distilling a pretrained diffusion model to guide an identically architectured model with fewer sampling steps. Li et al. [28], Hou and Asghar [19], Shen et al. [57] employ post-training quantization techniques, and Chen et al. [4] enhance the implementation of SDMs for better compatibility with GPUs. However, the removal of architectural elements in diffusion models has not been investigated in spite of the established efficacy of structured pruning across discriminative models [26, 69] and generative adversarial networks (GANs) [31, 24].

This study unlocks the immense potential of classical architectural compression in attaining smaller and faster diffusion models. We eliminate multiple residual and attention blocks from the U-Net of a SDM and pretrain it with feature-level knowledge distillation (KD) [48, 13] for general-purpose T2I synthesis. Despite being trained with only 0.22M LAION pairs (less than 0.1% of the entire training pairs) [55] on a single A100 GPU, our compact models can mimic the original SDM by leveraging transferred knowledge. On the popular zero-shot MS-COCO benchmark [29], our work achieves a FID [15] score of 15.76 with 0.76B parameters and 16.98 with 0.66B parameters, which are on par with multi-billion parameter models [43, 7, 8]. Furthermore, we present the practical application of our lightweight pretrained models in customized T2I with DreamBooth finetuning [50].

Our contributions are summarized as follows:

To the best of our knowledge, this is the first study to architecturally compress large-scale diffusion models. Our work is orthogonal to prior directions for efficient diffusion, e.g., enabling less sampling steps and employing quantization, and can be readily integrated with them.

We compress SDMs by removing architectural blocks from the U-Net and achieve more than 30% reduction in model size and inference speed. We also introduce an interesting finding on the minor role of innermost blocks.

We demonstrate the advantage of distillation-based pretraining, which allows us to attain competitive zero-shot T2I results even with very limited training resources.

We highlight the capability of our light pretrained backbones in customized generation. Our models can lower the finetuning cost by 30% while retaining 97% scores of the original SDM.

2 Related work

Large T2I diffusion models. By gradually removing noise from corrupted data, diffusion-based 64 generative models [18, 59, 6] enable high-fidelity synthesis with broad mode coverage. Integrating 65

66

these merits with the advancement of pretrained language models [42, 41, 5] has significantly

improved the quality of T2I synthesis. In GLIDE [38] and Imagen [51], a text-conditional diffusion 67

Teacher

Original

SDM

Dn

ua

Dn

Dn

Mid

Up

Up

Up

4×64×64

Latent

R

4×64×64

Noise

320 × 64×64

320×32×32

640×16×16

1280×8×8

1280×8×8

1280×8×8

1280× 16×16

1280×32×32

640×64 ×64

320×64 ×64

320×64×64 320×32×32 640×16×16 1280×8×8 1280×8×8 1280×16×16

1280×32×32

640× 64 ×64

320 ×64 ×64

4×64×64

Latent

Dn

Dn

Dn

R

Dn

Up

Up

Up

Up

4×64×64

Noise

uu

ToBeRemoved

Distillation

Student

Compact

BK-SDM

Mid

Dn

Up

Mid-,

Up Stages

Down,

Time Emb

Feature →

1280

Block

(time-embedding

conditioned)

ResBlock

Text Emb-

77 × 768

Feature -→

Block

Attn

(text-embedding

conditioned)

CrossAttentionBlock

Concatenation

Channel

Conv (spatial

sizepreserved)

Dn

Conv

Down-sizing

upConv

Up-sizing

model generates a 64×64 image, which is upsampled via super-resolution modules. In DALL·E-2 [44], 68 a text-conditional prior network produces an image embedding, which is transformed into a 64×64 69 image via a diffusion decoder and further upscaled into higher resolutions. SDMs [46, 47] perform 70 the diffusion modeling in a 64×64 latent space constructed through a pixel-space autoencoder. We use 71 SDM as our baseline because of its open-access and gaining popularity over numerous downstream 72 tasks [2, 67, 1, 50]. 73

Efficient diffusion models. Several studies have addressed the slow sampling process of diffusion 74 models. Diffusion-tailored distillation approaches [35, 34, 52] progressively transfer knowledge from 75 a pretrained diffusion model to a fewer-step model with the same architecture. Fast high-order solvers 76 [32, 33, 73] for diffusion ordinary differential equations boost the sampling speed. Orthogonal 77 to these directions for less sampling steps, our network compression approach reduces per-step 78 computation and can be easily integrated with them. Leveraging quantization techniques [28, 19, 57] 79 and implementation optimizations [4] has been applied for SDMs and also can be combined with our 80 models for further efficiency gains. 81

82

83

84

85

86

87

88

89

90

91

92

Distillation-based compression. KD enhances the performance of small-size models by exploiting output-level [16, 39] and feature-level [48, 13, 70] information of large source models. Although this classical distillation has been actively used toward efficient GANs [27, 45, 31, 22, 72], its power has not been explored for structurally compressed diffusion models. Distillation-based pretraining enables small yet capable general-purpose language models [54, 61, 21] and vision transformers [63, 11]. Beyond such models, we show that its success can be extended to diffusion models with iterative sampling steps. Concurrently with our study, a recently released small SDM without paper evidence [40] similarly utilizes KD pretraining for a block-eliminated architecture, but it relies on significantly more training resources along with multi-stage distillation. In contrast, our lightest model achieves further reduced computation, and we show that competitive results can be obtained even with much less data and single-stage distillation.

93

3 BK-SDM: block-removed knowledge-distilled SDM

We compress the U-Net [49] of a SDM [46, 47], which is the most compute-heavy component (see 94 Figure 2). Conditioned on the text and time-step embeddings, the U-Net performs multiple denoising 95 steps on latent representations. At each denoising step, the U-Net produces the noise residual to 96 compute the latent for the next step (see the top part of Figure 3). We reduce this per-step computation 97 by exploiting block-level elimination and feature distillation. 98

3.1 Compressed U-Net architecture 99

The proposed models are referred to as: 100

101

BK-SDM-Base (0.76B parameters) obtained with Section 3.1.1 (fewer blocks in outer stages).

102

◦

BK-SDM-Small (0.66B) with Section 3.1.1 (fewer blocks) and Section 3.1.2 (mid-stage removal).

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

3.1.1 Fewer blocks in the down and up stages

Our design philosophy is closely aligned with that of DistilBERT [54] which halves the number of layers for improved computational efficiency and initializes the compact model with the original weights by benefiting from the shared dimensionality. In the original U-Net, each stage with a common spatial size consists of multiple blocks, and most stages contain pairs of residual (R) [12] and cross-attention (A) [65, 20] blocks. We hypothesize the existence of some unnecessary pairs and use the following removal strategies, as shown in Figure 3.

For the down stages, we maintain the first R-A pairs while eliminating the second pairs, because the first pairs process the changed spatial information and would be more important than the second pairs. This design choice does not harm the dimensionality of the original U-Net, enabling the use of the corresponding pretrained weights for initialization [54].

For the up stages, while adhering to the aforementioned scheme, we retain the third R-A pairs. This allows us to utilize the output feature maps at the end of each down stage and the corresponding skip connections between the down and up stages. The same process is applied to the innermost down and up stages that contain only R blocks.

3.1.2 Removal of the entire mid-stage

Surprisingly, removing the entire mid-stage from the original U-Net (marked with red in Figure 3) does not noticeably degrade the generation quality for many text prompts while effectively reducing the number of parameters (see Table 1 and Figure 4). This observation is consistent with the minor role of inner layers in the U-Net generator of GANs [24].

Integrating the mid-stage removal with fewer blocks in Section 3.1.1 further decreases computational 123 burdens (Table 3) at the cost of a slight decline in performance (Table 2). Therefore, we offer 124 this mid-stage elimination as an option, depending on the priority between compute efficiency and 125 generation quality. 126

127

128

129

130

3.2 Distillation-based pretraining

For general-purpose T2I generation, we train the compact U-Net to mimic the behavior of the original U-Net. Following Rombach et al. [47], we use the pretrained-and-frozen encoders to obtain the inputs of the U-Net.

Given the latent representation z of an image and its paired text embedding y , the task loss for the 131 reverse denoising process [18, 47] is computed as: 132

where ϵ ∼ N (0 , I ) and t ∼ Uniform(1 , T ) denote the noise and time step sampled from the diffusion 133 process, respectively, and ϵ S ( ◦ ) indicates the output of our compact U-Net student. For brevity, we 134 omit the subscripts of E z,ϵ,y,t [ ◦ ] in the following notations. 135

SDM-v1.4

Mid-Stage Removal

Candles and flowers

A room is furnished

Two stuffed bears are

neatly placed on a

with couches,rugs,

dressed in astronaut

table

and fancy paintings

suits

The compact student is also trained to imitate the outputs of the original U-Net teacher, ϵ T ( ◦ ) ,with 136 the following output-level KD objective [16]: 137

A key to our approach is the utilization of feature-level KD [48, 13] that provides abundant guidance 138 for the student's training: 139

where f l T ( ◦ ) and f l S ( ◦ ) represent the feature maps of the l -th layer in a predefined set of distilled layers from the teacher and the student, respectively. While learnable regressors (e.g., 1×1 convolutions to match the number of channels) have been commonly used in existing studies [58, 45, 48], our approach circumvents this requirement. By applying distillation at the end of each stage in both models, we ensure that the dimensionality of the feature maps already matches, thus eliminating the need for additional regressors.

The final objective is formalized as below, and we simply set the loss weights λ OutKD and λ FeatKD as 1. Without any hyperparameter tuning, our approach is effective in empirical validation.

3.3 Application: faster and smaller personalized SDMs

To emphasize the benefit of our lightweight pretrained SDMs, we use a popular finetuning scenario for personalized generation. DreamBooth [50] enables T2I diffusion models to create contents about a particular subject using just a few input images. Our compact models not only accelerate inference speed but also reduce finetuning cost. Moreover, they produce high-quality images based on the inherited capability of the original SDM.

4 Experimental setup

4.1 Datasets and evaluation metrics

Pretraining. We train our compact SDM with only 0.22M image-text pairs from LAION-Aesthetics V2 6.5+ [55, 56], which are significantly fewer than the original training data used for SDM-v1.4 [46] (i.e., 600M pairs of LAION-Aesthetics V2 5+ [55] for the resumed training).

Zero-shot T2I evaluation. Following the popular protocol [43, 47, 51] to assess general-purpose T2I with pretrained models, we use 30K prompts from the MS-COCO validation split [29] and compare the generated images to the whole validation set. We compute Fréchet Inception Distance (FID) [15] and Inception Score (IS) [53] to assess visual quality. Moreover, we measure CLIP score [42, 14] with CLIP-ViT-g/14 model to assess text-image correspondence.

Finetuning for personalized generation. We use the DreamBooth dataset [50] that covers 30 subjects, each of which is associated with 25 prompts and 4 ∼ 6 images. Through individual finetuning for each subject, 30 personalized models are obtained. For evaluation, we follow the protocol of Ruiz et al. [50] based on four synthesized images per subject and per prompt. We consider CLIP-I and DINO scores to measure how well subject details are maintained in generated images (i.e., subject fidelity) and CLIP-T scores to measure text-image alignment (i.e., text fidelity). We use ViT-S/16 embeddings [3] for DINO scores and CLIP-ViT-g/14 embeddings for CLIP-I and CLIP-T.

4.2 Implementation

Weuse the released version v1.4 of SDM [46] as our compression target. We remark that our approach is also applicable to other versions in v1.1-v1.5 with the same architecture and to SDM-v2 with a similarly designed architecture.

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

† and ‡ : FID from [62] and [47], respectively. ⋆ and ♯ : IS from [8] and [47], respectively. DF and AR: diffusion and autoregressive models. ↓ and ↑ : lower and higher values are better.

We adjust the codes in Diffusers library [66] for pretraining our models and those in PEFT library 175 [60] for DreamBooth-finetuning, both of which adopt the training process of DDPM [18] in latent 176 spaces. We use a single NVIDIA A100 80G GPU for 50K-iteration pretraining with a constant 177 learning rate of 5e-5. For DreamBooth, we use a single NVIDIA GeForce RTX 3090 GPU to finetune 178 each personalized model for 800 iterations with a constant learning rate of 1e-6. 179

Following the default inference setup, we use PNDM scheduler [30] for zero-shot T2I generation 180 and DPM-Solver [32, 33] for DreamBooth results. For compute efficiency, we always opt for 25 181 denoising steps of the U-Net at the inference phase. The classifier-free guidance scale [17, 51] is set 182 to the default value of 7.5, except the analysis in Figure 7. 183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

5 Results

5.1 General-purpose T2I generation

5.1.1 Main results

Table 2 shows the zero-shot T2I results on 30K samples from the MS-COCO 256×256 validation set. Despite being trained with only 0.22M samples and having fewer than 1B parameters, our compressed models demonstrate competitive performance on par with previous large pretrained models. Despite the absence of a paper support, we include the model [40] that is identical in structure to BK-SDM-Base for comparison. This model benefits from far more training resources, i.e., two-stage KD relied on two teachers (SDM-v1.4 and v1.5) and a much larger volume of data with significantly longer iterations.

Figure 5 depicts synthesized images of different models with some MS-COCO captions. Our compressed models inherit the superior ability of SDM and produce more photorealistic images compared to the AR-based [8] and GAN-based [74, 62] baselines. Noticeably, the same latent code results in a shared visual style between the original and our compact SDMs (4th-6th columns in Figure 5), similar to the observation in transfer learning for GANs [36].

Table 3 summarizes how the computational reduction for each sampling step of the U-Net impacts the overall compute of the entire SDM. The per-step reduction effectively decreases MACs and inference time by more than 30% as well as the number of parameters.

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

CogView2

LAFITE

GALIP-CC12M

SDM-v1.4

BK-SDM-Base

BK-SDM-Small

(Tao, 2023)

(Rombach, 2022)

(Ours)

(Ours)

A bowl that has

vegetables inside

of it.

A cat that is

staring at the

camera.

There aremany

decorative

umbrellas

hanging up.

A man staring

ahead at the

camera witha

neutral

expression.

5.1.2 Ablation study

Table 4 presents the ablation study with the zero-shot MS-COCO benchmark dataset. The common default settings for the models N1-N7 involve the usage of fewer blocks in the down and up stages (Section 3.1.1) and the denoising task loss (Eq. 1). All the models are drawn at the 50K-th training iteration. We made the following observations.

N1 vs. N2. Importing the pretrained weights for initialization clearly improves the performance of block-removed SDMs. Transferring knowledge from well-trained models, a popularized practice in machine learning, is also beneficial for T2I generation with SDMs.

N2 vs. N3 vs. N4. Exploiting output-level KD (Eq. 2) effectively boosts the generation quality compared to using only the denoising task loss. Leveraging feature-level KD (Eq. 3) further improves the performance by offering sufficient guidance over multiple stages in the student.

N4 vs. N5. An increased batch size leads to a better IS and CLIP score but with a minor drop in FID. We opt for a batch size of 256 based on the premise that more samples per batch would enhance the model's understanding ability.

N6 and N7. Despite slight performance drop, the models N6 and N7 with the mid-stage removal have fewer parameters (0.66B) than N4 and N5 (0.76B), offering improved compute efficiency.

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

FID↓

IS↑

CLIPScore↑

0.3

35

Ours-Base,With KD,Bat 64

SDMv1.4

Ours-Base, Without KD, Bat 64

35

0.28

￥。

30

Ours-Base,WithKD,Bat 256

Ours-Small,WithKD,Bat256

30

25

0.26

25

20

0.24

20

15

Ours-Base,Without KD,Bat 64

Ours-Base, With KD, Bat 64

SDMv1.4

0.22

SDMv1.4

Ours-Base,Without KD,Bat 64

Ours-Base, With KD, Bat 64

15

10

Ours-Base,With KD,Bat 256

Ours-Small, With KD, Bat 256

0.2

Ours-Small,WithKD,Bat 256

Ours-Base, With KD, Bat 256

0

10K

20K

30K

40K

50K

0

10K

20K

30K

40K

50K

0

10K

20K

30K

40K

50K

Iteration

Iteration

Iteration

5.1.3 Impact of distillation on pretraining phase

We further analyze the merits of transferred knowledge via distillation, with the models from the pretrained weight initialization. Figure 6 shows zero-shot T2I performance over training iterations. Compared to the absence of KD (indicated with green), distillation (purple and pink) accelerates the training process and leads to improved generation scores, demonstrating the benefits of providing sufficient hints for training guidance. Notably, our small-size model trained with KD (yellow) outperforms the bigger base-size model without KD (green). Additionally, while the best FID score is observed early on for our models, IS and CLIP score exhibit ongoing improvement, implying that judging models solely with FID may be suboptimal.

Figure 7 shows the trade-off curves from different classifier-free guidance scales [17, 51] { 2 . 0 , 2 . 5 , 3 . 0 , 3 . 5 , 4 . 5 , 5 . 5 , 6 . 5 , 7 . 5 , 8 . 5 , 9 . 5 } . For the analysis, we use 5K samples from the MSCOCO validation set and our base-size models from the 50K-th iteration. Higher guidance scales lead to better text-aligned images at the cost of less diversity. Compared to the baseline trained only with the denoising task loss, distillation-based pretraining leads to much better trade-off curves.

5.2 Personalized T2I with DreamBooth

Table 5 compares the results of DreamBooth finetuning [50] with different pretrained models. BK233 SDM-Small can preserve over 97% performance of the original SDM with the reduced finetuning 234 time and number of parameters. Figure 8 depicts that our models can accurately capture the subject 235 details and generate various scenes. Over the models pretrained with a batch size of 64, we observe 236 the impact of KD pretraining on personalized synthesis. The baselines without KD fail to generate 237 the subjects entirely or cannot maintain the identity details. 238

43

43

39

WithKD,Bat64

WithKD,Bat256

39

WithKD,Bat64

(5K)

35

FID (5K)

35

FID

31

31

27

27

23

23

19

19

0.19 0.210.23 0.25 0.27 0.29

17

20

23

26

29

32

CLIP Score

IS

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

Input

SDM-v1.4

BK-SDM-Base

BK-SDM-Small

BK-SDM-Base

WithoutKD&

WithoutKD&

Subject

(Rombach,2022)

(Ours)

(Ours)

Pretrained Init.

op[Λ]

in the jungle

A[V]stuffed

animal with the

the background

A[V]backpack

in the snow

A[V]sneaker

on top of green

grass with

sunflowers

6 Conclusion and discussion

This study uncovers the potential of architectural compression for general-purpose text-to-image synthesis with a renowned model, Stable Diffusion. Our block-removed lightweight models are effective for zero-shot generation, achieving competitive performance against large-scale baselines. Distillation is a key aspect of our method, leading to effective pretraining even under very constrained resources. Moreover, our smaller and faster pretrained models are successfully applied in personalized generation. Our work is orthogonal to previous directions for efficient diffusion models, e.g., enabling fewer sampling steps, and can be readily combined with them. We hope our study can facilitate future research on structural compression of large diffusion models.

Limitations and future works. Our compact models inherit the capability of the source model for high-fidelity image generation, but they have shortcomings such as inaccurate generation of full-body human appearance. While we show that distillation pretraining is powerful even with very limited resources, increasing the volume of data and analyzing its effects would be promising.

Negative social impacts. Because recent large generative models are capable of creating high-quality plausible content, they also involve potential risks of malicious use. To avoid causing unintended social bias, researchers should take steps to ensure the appropriateness of training data. Moreover, the release of resulting models should be accompanied by strong and reliable safeguards.

256

257

258

References

A. Blattmann, R. Rombach, H. Ling, T. Dockhorn, S. W. Kim, S. Fidler, and K. Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR , 2023.

T. Brooks, A. Holynski, and A. A. Efros. Instructpix2pix: Learning to follow image editing instructions. 259 In CVPR , 2023. 260

M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in 261 self-supervised vision transformers. In ICCV , 2021. 262

Y.-H. Chen, R. Sarokin, J. Lee, J. Tang, C.-L. Chang, A. Kulik, and M. Grundmann. Speed is all you need: 263 On-device acceleration of large diffusion models via gpu-aware optimizations. In CVPR Workshop , 2023. 264

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers 265 for language understanding. In NAACL , 2019. 266

P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. In NeurIPS , 2021. 267

M. Ding, Z. Yang, W. Hong, W. Zheng, C. Zhou, D. Yin, J. Lin, X. Zou, Z. Shao, H. Yang, et al. Cogview: 268 Mastering text-to-image generation via transformers. In NeurIPS , 2021. 269

M. Ding, W. Zheng, W. Hong, and J. Tang. Cogview2: Faster and better text-to-image generation via 270 hierarchical transformers. In NeurIPS , 2022. 271

P. Esser, R. Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis. In CVPR , 272 2021. 273

O. Gafni, A. Polyak, O. Ashual, S. Sheynin, D. Parikh, and Y. Taigman. Make-a-scene: Scene-based 274 text-to-image generation with human priors. In ECCV , 2022. 275

Z. Hao, J. Guo, D. Jia, K. Han, Y. Tang, C. Zhang, H. Hu, and Y. Wang. Learning efficient vision 276 transformers via fine-grained manifold distillation. In NeurIPS , 2022. 277

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR , 2016. 278

B. Heo, J. Kim, S. Yun, H. Park, N. Kwak, and J. Y. Choi. A comprehensive overhaul of feature distillation. 279 In ICCV , 2019. 280

J. Hessel, A. Holtzman, M. Forbes, R. Le Bras, and Y. Choi. CLIPScore: A reference-free evaluation 281 metric for image captioning. In EMNLP , 2021. 282

M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale 283 update rule converge to a local nash equilibrium. In NeurIPS , 2017. 284

G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. In NeurIPS Workshop , 285 2014. 286

J. Ho and T. Salimans. Classifier-free diffusion guidance. In NeurIPS Workshop , 2021. 287

J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In NeurIPS , 2020. 288

J. Hou and Z. Asghar. World's first on-device demonstration of stable diffusion on an android phone. 289 https://www.qualcomm.com/news , 2023. 290

A. Jaegle, F. Gimeno, A. Brock, O. Vinyals, A. Zisserman, and J. Carreira. Perceiver: General perception 291 with iterative attention. In ICML , 2021. 292

X. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu. Tinybert: Distilling bert for 293 natural language understanding. In Findings of EMNLP , 2020. 294

Q. Jin, J. Ren, O. J. Woodford, J. Wang, G. Yuan, Y. Wang, and S. Tulyakov. Teachers do more than teach: 295 Compressing image-to-image models. In CVPR , 2021. 296

B. Kawar, S. Zada, O. Lang, O. Tov, H. Chang, T. Dekel, I. Mosseri, and M. Irani. Imagic: Text-based real 297 image editing with diffusion models. In CVPR , 2023. 298

B.-K. Kim, S. Choi, and H. Park. Cut inner layers: A structured pruning strategy for efficient u-net gans. 299 In ICML Workshop , 2022. 300

N. Kumari, B. Zhang, R. Zhang, E. Shechtman, and J.-Y. Zhu. Multi-concept customization of text-to-image 301 diffusion. In CVPR , 2023. 302

H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf. Pruning filters for efficient convnets. In ICLR , 303 2017. 304

M. Li, J. Lin, Y. Ding, Z. Liu, J.-Y. Zhu, and S. Han. Gan compression: Efficient architectures for 305 interactive conditional gans. In CVPR , 2020. 306

X. Li, L. Lian, Y. Liu, H. Yang, Z. Dong, D. Kang, S. Zhang, and K. Keutzer. Q-diffusion: Quantizing 307 diffusion models. arXiv preprint arXiv:2302.04304 , 2023. 308

T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft 309 coco: Common objects in context. In ECCV , 2014. 310

L. Liu, Y. Ren, Z. Lin, and Z. Zhao. Pseudo numerical methods for diffusion models on manifolds. In 311 ICLR , 2022. 312

Y. Liu, Z. Shu, Y. Li, Z. Lin, F. Perazzi, and S.-Y . Kung. Content-aware gan compression. In CVPR , 2021. 313

C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic 314 model sampling in around 10 steps. In NeurIPS , 2022. 315

C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu. Dpm-solver++: Fast solver for guided sampling of 316 diffusion probabilistic models. arXiv preprint arXiv:2211.01095 , 2022. 317

C. Meng, R. Gao, D. P. Kingma, S. Ermon, J. Ho, and T. Salimans. On distillation of guided diffusion 318 models. In NeurIPS Workshop , 2022. 319

C. Meng, R. Gao, D. P. Kingma, S. Ermon, J. Ho, and T. Salimans. On distillation of guided diffusion 320 models. In CVPR , 2023. 321

S. Mo, M. Cho, and J. Shin. Freeze the discriminator: a simple baseline for fine-tuning gans. In CVPR 322 Workshop , 2020. 323

C. Mou, X. Wang, L. Xie, J. Zhang, Z. Qi, Y. Shan, and X. Qie. T2i-adapter: Learning adapters to dig out 324 more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.08453 , 2023. 325

A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen. Glide: 326 Towards photorealistic image generation and editing with text-guided diffusion models. In ICML , 2022. 327

W. Park, D. Kim, Y. Lu, and M. Cho. Relational knowledge distillation. In CVPR , 2019. 328

J. Pinkney. Small stable diffusion. https://huggingface.co/OFA-Sys/ 329 small-stable-diffusion-v0 , 2023. 330

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised 331 multitask learners. OpenAI blog , 2019. 332

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, 333 J. Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021. 334

A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot 335 text-to-image generation. In ICML , 2020. 336

A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation 337 with clip latents. arXiv preprint arXiv:2204.06125 , 2022. 338

Y. Ren, J. Wu, X. Xiao, and J. Yang. Online multi-granularity distillation for gan compression. In ICCV , 339 2021. 340

R. Rombach and P. Esser. Stable diffusion v1-4. https://huggingface.co/CompVis/ 341 stable-diffusion-v1-4 , 2022. 342

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with 343 latent diffusion models. In CVPR , 2022. 344

A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio. Fitnets: Hints for thin deep 345 nets. In ICLR , 2015. 346

O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. 347 In MICCAI , 2015. 348

N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman. Dreambooth: Fine tuning 349 text-to-image diffusion models for subject-driven generation. In CVPR , 2023. 350

C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, 351 B. Karagol Ayan, T. Salimans, et al. Photorealistic text-to-image diffusion models with deep language 352 understanding. In NeurIPS , 2022. 353

T. Salimans and J. Ho. Progressive distillation for fast sampling of diffusion models. In ICLR , 2022. 354

T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for 355 training gans. In NeurIPS , 2016. 356

V. Sanh, L. Debut, J. Chaumond, and T. Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper 357 and lighter. In NeurIPS Workshop , 2019. 358

C. Schuhmann and R. Beaumont. Laion-aesthetics. https://laion.ai/blog/laion-aesthetics , 359 2022. 360

C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, 361 C. Mullis, M. Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation 362 image-text models. In NeurIPS Workshop , 2022. 363

H. Shen, P. Cheng, X. Ye, W. Cheng, and H. Abidi. Accelerate stable diffusion with intel neural compressor. 364 https://medium.com/intel-analytics-software , 2022. 365

C. Shu, Y. Liu, J. Gao, Z. Yan, and C. Shen. Channel-wise knowledge distillation for dense prediction. In 366 ICCV , 2021. 367

J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. In ICLR , 2021. 368

L. D. Y. B. S. P. Sourab Mangrulkar, Sylvain Gugger. Peft: State-of-the-art parameter-efficient fine-tuning 369 methods. https://github.com/huggingface/peft , 2022. 370

Z. Sun, H. Yu, X. Song, R. Liu, Y. Yang, and D. Zhou. Mobilebert: a compact task-agnostic bert for 371 resource-limited devices. In ACL , 2020. 372

M. Tao, B.-K. Bao, H. Tang, and C. Xu. Galip: Generative adversarial clips for text-to-image synthesis. In 373 CVPR , 2023. 374

H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jegou. Training data-efficient image 375 transformers amp; distillation through attention. In ICML , 2021. 376

A. Van Den Oord, O. Vinyals, et al. Neural discrete representation learning. In NeurIPS , 2017. 377

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. 378 Attention is all you need. In NeurIPS , 2017. 379

P. von Platen, S. Patil, A. Lozhkov, P. Cuenca, N. Lambert, K. Rasul, M. Davaadorj, and T. Wolf. Diffusers: 380 State-of-the-art diffusion models. https://github.com/huggingface/diffusers , 2022. 381

H. Wang, X. Du, J. Li, R. A. Yeh, and G. Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d 382 diffusion models for 3d generation. In CVPR , 2023. 383

J. Z. Wu, Y. Ge, X. Wang, W. Lei, Y. Gu, W. Hsu, Y. Shan, X. Qie, and M. Z. Shou. Tune-a-video: One-shot 384 tuning of image diffusion models for text-to-video generation. arXiv preprint arXiv:2212.11565 , 2022. 385

Z. Xie, L. Zhu, L. Zhao, B. Tao, L. Liu, and W. Tao. Localization-aware channel pruning for object 386 detection. Neurocomputing , 403:400-408, 2020. 387

S. Zagoruyko and N. Komodakis. Paying more attention to attention: Improving the performance of 388 convolutional neural networks via attention transfer. In ICLR , 2017. 389

L. Zhang and M. Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint 390 arXiv:2302.05543 , 2023. 391

L. Zhang, X. Chen, X. Tu, P. Wan, N. Xu, and K. Ma. Wavelet knowledge distillation: Towards efficient 392 image-to-image translation. In CVPR , 2022. 393

Q. Zhang and Y. Chen. Fast sampling of diffusion models with exponential integrator. In ICLR , 2023. 394

Y. Zhou, R. Zhang, C. Chen, C. Li, C. Tensmeyer, T. Yu, J. Gu, J. Xu, and T. Sun. Towards language-free 395 training for text-to-image generation. In CVPR , 2022. 396

L. Zhu. Thop: Pytorch-opcounter. https://github.com/Lyken17/pytorch-OpCounter , 2018. 397

Peer reviews:
comments: SUMMARY:
This paper proposes a comprehensive stable diffusion compression pipeline, including architecture compression and knowledge distillation. The authors apply this technique to general text-to-image generation and subject-driven image generation tasks, showing competitive performance as compared with the original stable diffusion while reducing parameter numbers and training dataset size.

STRENGTHS:
1. The approach proposed in this paper is straightforward.
2. The writing of this paper is easy to understand.

WEAKNESSES:
1. The most important issue is that this paper is just a marginal improvement based on stable diffusion, so it cannot be generalized to other text-to-image synthesis frameworks, which severally narrows its application scope.
2. Reducing the SD parameter number from 1B to 0.76B or 0.66B is not so significant that this approach may not bring leaping progress in industrial applications.
3. The output and feature distillation are just classical distillation strategies that are widely adopted in other domains, e.g., image classification or GAN-based image generation. So here is a lack of innovation.

QUESTIONS:
1. This paper saying "this is the first study to architecturally compress large-scale diffusion models" may not be right. The paper "Scalable Diffusion Models with Transformers" discussed this issue in December 2022. Did the authors ignore this work?
Please refer to Peebles, William, and Saining Xie. "Scalable Diffusion Models with Transformers" arXiv preprint arXiv:2212.09748 (2022).
2. Is there any theoretical analysis or intuitive explanation of Sections 3.1.1 and 3.1.2, which only simply propose to reduce blocks in the down and up stages and remove the mid-stage?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies model compression for stable diffusion models (1.4 specifically). Specifically, they study 1. removing entire layers from the diffusion U-Net, 2. fine-tuning from a small subset of LAION to recover the loss, and 3. knowledge distillation from intermediate feature layers. They show that they can get a significant (~30-40%) reduction in both parameter count and inference latency at a modest output quality degradation, at a modest training cost (60-300 hours on a single GPU) that individuals can have access to.\n\nSTRENGTHS:\nThe paper works on a timely topic, especially given that Stable Diffusion has widespread use in the open source community, and reducing the inference cost will not only have a broad impact on usability for these models, but also the environment. The paper is easy to understand, and seems to be first to evaluate structured pruning (broadly) on Stable Diffusion to the best of my knowledge. The analysis includes several different metrics and ablation studies that are useful to the community.\n\nWEAKNESSES:\nI found it hard to intuitively understand why the layers were specifically chosen to remove. One of the interesting results the authors show in this paper is that removing the mid-layer without any additional fine tuning already results in decent quality (competitive with some of the smaller models that are fine-tuned). Then, a natural question that might come up then is if we were to remove every layer and run an analysis of how much cost (in quality loss) the removal of each layer ends up with, would we then be able to confirm that the layers that were removed are in fact the best layers to remove in terms of their quality cost? This would be much cheaper of an experiment to run than fine-tuning the entire removed architecture, and would be very helpful to know if the \"unsupervised\" structured pruning is a good indicator for the end performance after fine-tuning. \n\nIt would at least be interesting if there could be an analysis of how much the fine-tuning (and KD) adds (in terms of quality) to an 'unsupervised' baseline, whether that means running an ablation specifically with the mid results removed, or adding an 'unsupervised' baseline with all layers removed (the orange and red in Figure 3). \n\nThis is very much concurrent work (and does not affect my score), but it could be worth adding https://arxiv.org/abs/2305.10924 to the citations.\n\nQUESTIONS:\n1. How costly is the addition of KD in terms of training times? The supplementary mentions training times but not with respect to some of these design decisions (as far as I could find)\n\nLIMITATIONS:\nThe limitations are adequately addressed. It might also be interesting if the authors can comment whether model compression itself could have risks in society, for example if people prefer worse but slightly faster models over better models, then that could potentially exacerbate the bias that these models have (and of the models people use in practice).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an technique to compress the network structure of a diffusion model. Specifically, the method combines a static block pruning strategy and a knowledge distillation retraining to learn compact diffusion model. Experiments on text to image generation and customization shows the effectiveness of the approach.\n\nSTRENGTHS:\n1. The paper shows that conventional compression techniques leveraged in CNNs/GANs can also be used for diffusion model.\n2. Some experiments are conducted to show the effectiveness of the method.\n\nWEAKNESSES:\n1. The paper is not novel at all. Pruning and distillation has long been used for vision network compression since the era of CNNs and later GANs. Nothing surprising comes out from this paper, where the author just simply combine two techniques and makes it work. Block pruning and output + feature distillation is developed far long ago and naively combines them show little academic value.\n2. The paper lacks insight significantly. There's no theoretical proof nor ablation study to judge the design choice. For example, why shall we remove the second pairs of R-A instead of the first pair of A-R in down blocks? Why shall we retain the third R-A pair in the up blocks? Why can't we do selective removal where some blocks only remove R and some blocks only remove A? The pruning design choice is totally ad-hoc where nothing systematic and insightful comes out. Due to this, the method seems to only be applicable on SD-v1.4. I see no way that how this method can be applied to SD-v2.1 / SDXL or any other new SDs where we simply add one/two more modules of R/A into a block.   \n3. The use of distillation-based pre-training sounds weird. Training/fine-tuning with distillation to recover the performance is a more appropriate wording.\n4. In Sec 4.1, the paper says that the BK-SDM is only fine-tuned on 0.22M image-text pairs. Why only fine-tuned on such smaller scale of dataset? How about fine-tuning on 2.2M or 22M? Would the quality get improved? How about 22k or even fewer? Will the quality stay the same? No judgement is presented here as well.\n5. In Figure 5, the paper should compare more diffusion method instead of showing the first 3 column of GAN-based approach. Since the speed of the network falls into the scale of diffusion model, it does not make much sense to compare GANs.\n\nQUESTIONS:\nPlease see the Weakness section.\n\nLIMITATIONS:\nYes, the author addressed both.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
This submission proposes a light-weighted network for stable-diffusion-based text-to-image generation.
The proposed network is named BK-SDM, which became light-weighted thanks to removing unnecessary layers
and knowledge distillation.
An interesting finding is that removal of the entire mid-stage (the most low-resolution part of the u-net) does not seriously harm
generation quality.
As a result, 30% computational cost reduction was achieved while retaining 97% scores of generated image quality.

STRENGTHS:
- Light-weight diffusion models are, despite their importance,  not explored. I appreciate pioneering this direction.
- Training using a small subset (0.22M image-text pairs from LAION) is a nice point that is easier to reproduce, while keeping the generation quality.

WEAKNESSES:
-  Methodological novelty is limited because layer removing and knowledge distillation are well-used technics, while they were not done in diffusion. I did not find special challenges in applying them to diffusion models.

- Trades-off between speed and quality are not aggressively investigated; the only presented model is the  30% faster / 97% worse version. For example, what happens when the model is 90% faster? Is it possible to achieve any speed-up with 100% quality? Perhaps too aggressive speed-up results in the uselessly worse generation quality, but trying and clarifying it would be a useful contribution for a paper.

QUESTIONS:
"0.22M image-text pairs from LAION-Aesthetics157 V2 6.5+": is this a randomly sampled version of the dataset? Or any systematic image reduction (for example, smallifying the caption vocabulary size) was done?

LIMITATIONS:
Limitations and potential social impacts are properly discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07

comments: SUMMARY:
This work explores traditional model compression for diffusion models, aiming to mitigate their considerable computational overhead. By employing block removal and knowledge distillation, the authors are able to reduce parameters by over 30% with only minimal performance loss with the 0.22M LAION dataset. The paper further showcases its applicability for personalized generation tasks.

STRENGTHS:
1. The proposed hand-crafted compression strategy, that leverages block-removing knowledge distillation, 
simple, but effective.  
2.  This work demonstrates the effectiveness of their proposed method on different model sizes, different tasks, including text-image generation and personalized text-image generation. 
The paper provides robust evidence of the approach's versatility across varying model sizes and tasks, which include both text-image and personalized text-image generation.

WEAKNESSES:
1. The main concern lies in the hand-crafted block removal process from the original DDPM. It may be more reasonable to calculate an importance score for each block and remove them accordingly. The simplicity of the current approach, while appreciated, seems to compromise on the degree of novelty. Please further clarify this.

2. A comparison of different sampling steps is missing. Will the model compression influence the trade-off between fidelity and sampling efficiency? 


3. The proposed methods seem to be general for the diffusion model.  Could the authors elaborate on the reason not to include experiments on general image generation?

QUESTIONS:
My key questions stem from innovation in weakness. If it is solved, I am willing to improve my score.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07