Transferring Pretrained Diffusion Probabilistic Models
Authors: Fuming You
Zhou Zhao
Venue: Submitted to ICLR 2023 (2023)
Keywords: transfer learning
diffusion probabilistic models
cross-attention
fine-tuning

Abstract: Diffusion Probabilistic Models (DPMs) achieve impressive performance in visual generative tasks recently. However, the success of DPMs heavily relies on large amounts of data and optimization steps, which limits the application of DPMs to small datasets and limited computational resources. In this paper, we investigate transfer learning in DPMs to leverage the DPMs pretrained on large-scale datasets for generation with limited data. Firstly, we show that previous methods like training from scratch or determining the transferable parts is not suitable for the DPM due to its U-Net based denoising architecture with the external  denoising timestep input. To address it, we present a condition-based tuning approach to take full advantages of existing pretrained models. Concretely, we obtain the semantic embeddings of condition images by the pretrained CLIP model, and then inject these semantic informations to the pretrained DPM via a ''Attention-NonLinear'' (ANL) module. The adaptation to a new task can be achieved by only tuning the ANL module inserted into the pretrained DPM hierarchically. To further enhance the diversity of generated images, we introduce a masked sampling strategy based on the condition mechanism. Extensive experiments validate the effectiveness and efficiency of our proposed tuning approach in generative task transfer and data augmentation for semi-supervised learning. 

Under review as a conference paper at ICLR 2023
TRANSFERRING PRETRAINED DIFFUSION BILISTIC MODELS
PROBA-
Anonymous authors Paper under double-blind review
ABSTRACT
Diffusion Probabilistic Models (DPMs) achieve impressive performance in visual generative tasks recently. However, success of DPMs heavily relies on large amounts of data and optimization steps, which limits the application of DPMs small datasets and limited computational resources. In this paper, we investigate transfer learning in DPMs to leverage the DPMs pretrained on large-scale datasets for generation with limited data. Firstly, we show that previous methods like training from scratch determining the transferable parts is not suitable for the DPM due to its U-Net based denoising architecture with the external denoising timestep imput. To address full advantages of existing pretrained models. Concretely, we obtain the semantic embeddings of condition images by the pretrained CLIP model, and then inject these semantic informations to the pretrained DPM via a "Attention-None innnaa (ANL.) module. The adaptation to new task can achieved by only tuning the ANL module inserted into the pretrained DPM hierarchically. To further enhance the diversity of generated images, we introduce a masked sampling strategy based the condition mechanism. Extensive experiments validate the effectiveness and efficiency of our proposed tuning approach in generative task transfer and data augmentation for semi-supervised learning.
present condition-based tuning approach take
CelebA
Flowers
StanfordCCrrs
0.5K
IK
1,5K
2K
lteration
2.5k
3
3,5K
4K
Figure 1: Generated images by tuning
diffusion model Rombach et 2022 pretrained
KeNzeeKKrihevsk et al. (2012) with our proposed tuning approach (batch sizc=32). We illustrate the generated images from iteration 500 to 4000 with the interval of 500 on three datasets: CelebA Liu et al (2015), Flow- ers Nilsback & Zisserman (2008) and StanfordCars Krause al. (013).

---

Under review as a conference paper at ICLR 2023
INTRODUCTION
Recently, diffusion probabilistic models (DPMs) (Ho et al. 2020: Song ct al. 2020b) have been demonstrated the increasing power of generating complex and high-quality images by introducing a hierarchy of denoising autoencoders. DPMs adopt a diffusion process to gradually inject noise to the data distribution, and then learn the reverse process to generate images by Markov model. However, the success of diffusion models heavily depends on huge data and computation costs. For instance, training DPMs often needs large-scale datasets (e.g., ImageNet (Krizhevsky et al. (0122), and takes hundreds of GPU days in extreme cases (c.g., 150-1000 V100 GPU days in (Dhariwal & Nichol 2021)). Considering that it is usually hard to collect adequate training data for a specific domain (Long et al., 2015), we propose to leverage the released powerful DPMs pretrained on large- scale datasets to facilitate downstream generative tasks.
This transfer learning paradigm is thoroughly investigated on discriminative tasks in many fields like computer vision (Bengio) 2012; Yosinski et al. 2014; Zamir et al., 2018) and natural language pro- cessing (Devlin et al. 2018 Mozafari et al. 2019: Peng et al. (019)) For computer vision, Yosinski jet al. (014)) observed that the representations learned by deep convolutional neural networks tran- sit from general (e.g., Gabor filters and color blobs) to specific, which facilitates the "pretraining and fine-tuning" learning paradigm. For natural language processing, reusing the pretrained BERT model achieves impressive performance on multiple downstream tasks (Houlsby et al. (019))
However, few efforts has been paid to investigate transfer learning in the generative tasks, especially for the DPMs. (Zhao et al.) (2020) is recent work, which explores the transferability of generative adversarial networks (GANs). Motivated by the insights from (Yosinski et al.] 2014)) they proposed to preserve the low-level layers that capturing properties of generic patterns, and fine-tune the high- level layers that are associated with the semantic aspects of data. However, the insight of determining the transferable parts is not suitable for DPMs since DPMs often adopt U-Net based denoising architecture with the external denoising timestep input. DPMs generate images by progressively predicting the statisties of the gaussian noise corresponding to a specific timestep, so it hard to say which part learns the general or task-specific representations.
We have also considered other two baselines: training from seratch and tuning the whole pretrained DPM ("Tuning-All"), Based on empirical observations, we find training from scratch is infeasi- ble for learning with limited data and optimization steps, and Tuning-All is likely to overfit small training sets, resulting in slow convergence.
In this work, we present a condition-based tuning approach to achieve fast adaptation from pre- trained DPMs to new datasets. To make the adaptation procedure efficient, we propose to take maximum advantages of pretrained models. Firstly, we leverage the vision-text pretrained model CLIP (Radford et al.) 2021) to obtain the embedding of input images. It is worth noticing that the CLIP embedding is highly associated with the semantic informations of images. To make use of CLIP embeddings, we consider "Attention-Nonl.inee Linear" module (ANL), which injects the exter- nal conditions (i.e., the CLIP embedding) to each block of the pretrained DPM. The ANL module consists of cross-attention and non-linear mapping module. Note that the ANL module is in- serted into each block of DPM, fusing the CLIP embedding and adjusting the DPM hierarchically. Therefore, transferring pretrained DPM to new dataset can be achieved by fine-tuning the ANL modules only and freezing the pretrained CLIP model and DPM.
Introducing the trainable ANL modules into pretrained DPMs enjoys the following advantages: (1) This tuning approach preserves the main structures of pretrained DPMs, but is able to achieve effective transfer since the ANL module hierarchically adjusts each block of the pretrained DPM. (2) Compared to training from scratch and tuning the whole DPM, our approach introduces less trainable parameters and faster adaptation, which is beneficial for learning with limited data and training resources. (3) Based on the CLIP semantic embeddings, our approach can be extended to the language modality like simple text-to-image synthesis.
In summary, we have the following contributions:
1. We investigate transfer learning in recent DPMs, and uncover that previous methods like training from scratch determining the transferable parts are not efficient. To the best of our knowledge, we are at the very early attempt to leverage pretrained DPMs for generation with limited data and training resources.
2

---

Under review as a conference paper at ICLR 2023
2. We present pretrained DPM via an "Attention-Nonl.iee less parameters needed to be tuned and converges faster than the baselines.
tuning approach, that injects the CLIP embeddings to the (ANL) module. Our tuning approach has
3. We experimentally show that our tuning approach achieves fast adaptation with few train- ing data and optimization steps, and outperforms previous generative transfer learning ap- proaches. Further, also evaluate the potential of our method in data augmentation for the data-scarce scenarios like semi-supervised learning.
BACKGROUND
Diffusion Probabilistic Models (DPMs). DPMs construct a forward process that gradually adds noise data distribution p(xo) and learn to reverse this process for generation. Based on the Markov forward process with a forward noise schedule Br E 1), 1,... ,T, denoising diffusion prob- abilistic models (DDPMs) formulate it as:
(x1:r100)) II (x)((1111) (xx]/ar11) -N(x; - 1,
where I is the identity matrix. The reverse (generation) process is defined as
T
po(Xu:)) p(xx) II po (x(
pe(xi-1(x)) -Xxxx t), (x(x,, t)), (2)
where Eo (x,,, t) is often fixed constant. The reverse (generation) process can be learned by optimizing a variational bound on negative log-likelihood
L E, DKI (K(((((x/xx))l((xx))) + Σ DKI Xx, (po - log po
(3)
Let at 1 Bt and or = I x(x0; e) be reparameterized Va,xo vT are for e E & (N( I). By parameterizing 140 with the predicted noise c0. we have
LDDPM = (xx,r- [|le ca(va,xo + -I a,c, t)
(4)
Recently, RRobbac et al. 2022) proposed to perform diffusion models on the latent code of images rather than the original ones, and named it latent diffusion models (LDMs). Specifically, LDMs firstly train an autoencoder to convert the original images to a low-dimensional latent space that is latent
perceptually equivalent to the original data space. Let & be the encoder that encodes code 2, D be the decoder that reconstructs & from the latent code z: & (x)()),, we can write the learning objective of LDMs
LLDM (x) [||e c0 t)]
Since LDMs achieve state-of of-the-art performance on image synthesis and are capable for high- resolution synthesis with less inference costs, we build our method upon the LDM pretrained on the large-scale dataset ImageNet (Krizhevsky et al. 2017) with the resolution of 256 256. To evaluate the performance of generative models, we use the Frechet Inception Distance (FID) metric (Heusel et al. 2017). which estimating the realism and variation of generated images.
3
METHOD
In this section, we elaborate how to train DPM with limited data. Considering that training a DPM from scratch usually needs large amounts of data and training resources (Rombach et al. 2022). propose transfer DPM pretrained on large-scale datasets to new task with limited samples from distinct categories and domains. Specifically, given a pretrained DPM, we preserve the main structures, and inject externl semantic conditions via "Attention-NonLinge Hence, via this method, the number of parameters that need to tuned is reduced, alleviating the overfit issue when learning with limited data. Here, introduce tuning approach in detail:
(ANL) module.
3

---

Under review as a conference paper at ICLR 2023
Thisk Transfer
Conv Blocks
Cross Attention
Non-Lineur
condition
Randam Initialzzed
Trainable
CLIP Model
Preteinedd argosseale Datrects
(b) Tuning
Figure 2: (Best viewed in color.) An overview of three approaches for task transfer with the U-Net based diffusion probabilistic model (DPM): (a) random initializing the DPM and learning from scratch, (b) tuning the whole DPM pretrained on large-scale datasets, (c) the proposed tuning approach, which takes advantages of pretrained models and only adapts the "Attention-Noni Linear" module to achieve fast and effective transfer. Taking the latent diffusion model pretrained on ImageNet with the resolution of 256 as an example: the number of trainable parameters is 400M for (a) and (b), 169M for (c) our tuning approach.
3.1 CONDITION DPMS WITH CLIP EMBEDDING
Many approaches have been proposed for conditional DPMs like conditioning with classi- fier (Nichol & Dhariwal 2021 ) conditioning during sampling with reference image (Choi et al. 2021 ) learning conditional score (Song et al. 2020b and injecting external conditions during learning (Rombach et al., 2022 Giannone et al. 2022). The condition mechanism is usually recog- nized technique for conditional generation like class-conditional generation and text-to-image synthesis. Recently, (Giannone et al. (2022) reveal that conditioning a DPM with rich and expres- sive information promotes learning. To achieve this, Giannone al. (2022) introduce an additional trainable vision transformer for learning the conditional representations. However, both training the vision transformer and DPM may suffer from overfitting when limited target data are provided. Fol- lowing the criteria that take full advantages of pretrained models, we propose to directly obtain the conditional representations by the vision-language pretrained model CLIP (Radford et al., 2021):
c= ((x),
where e is pretrained CLIP model, is the semantic embedding corresponding the image input x. We take the LDMs as an example, the learning objective can be written as:
T (= [lle (2,, 1, c)]
c= <()(
CLIP is a powerful pretrained model that learning directly from 400 million text-image pairs. Com- pared to (Giannone al., )022), approach shows the following advantages: (1) Using the CLIP embedding fastens the learning process since we get rid of training an additional embedding net- work. (2) Thanks the shared text-image embedding space, we can achieve simple text-to-image generation even without training with explicit text-image pairs.
3.2 PATTENTIN-NONLINEAARR MODULES FOR EFFECTIVE TUNING
Given the semantic embedding e, we present an "Attention-NonLLnnaar (ANL) module that injects the rich semantic information to the pretrained DPM. The ANL module consists a cross-attention and a non-linear mapping module. In practice, DPMs usually adopt the UNet backbone, in which residual convolutional blocks are stacked hierarchically. In order to adapt each block of pretrained UNet, we add the ANL module after each residual convolutional block. Based on this, we present an efficient and effective tuning approach for transferring pretrained DPMs: tuning the ANL modules only, and freezing the pretrained residual convolutional blocks. Here, we elaborate each component of the ANL module by taking the pretrained LDM as an example.
The cross-attention module is used to fuse the intermediate representations of the pretrained LDM and the semantic embeddings of input images:
Attention K) softmax QKT
Q= -- h,(2,), K= - V w c,
VV, VT
(8)
where hx(2t) E N.."" is the intermediate representation of i-th residual convolutional block, c E N.d, is the CLIP embedding of input and independent of the timestep. w:(0) Rdd.dd w(() (1) e
de and w((0) Rededde are the learnable mapping matrices Vaswani al. 2017). Recalling that
4

---

Under review as a conference paper at ICLR 2023
hope to preserve the main structures of pretrained DPMs, and adapt them with tuning a specific module, we propose a non-linear adapter that mapping the semantic-fune fused representations to match the pretrained LDM. Therefore, the ANL module can be written
( c) f ((Attention(h,(2)) (2), c)):
where ANL.)) is the ANI. module parameterized by fANL. f is the non-linear mapping module con- sisting two linear layers and activation layer, Attention() is the cross-attention module defined in Eq. (8). fusing the intermediate representation of the pretrained LDM (z,) and the semantic CLIP embedding of the image input o(x). Here, we present our learning objective:
min V [lle - [|l--<(=e,0 c,(=,,t,c)13 c)
(10)
AANI
To better illustrate the insight of our tuning approach, we discuss three baselines here: (i) training from scratch, (ii) tuning the whole pretrained DPM, and (iii) determining the transferable part of the pretrained DPM. Training DPM from scratch usually needs large amounts of training samples and takes long iterative process to converge (Nichol & Dhariwal (002). Training a DPM from scratch usually needs large amounts of data, and fails to generate meaningful images with limited training samples and optimization steps (Sec. 4.2) Tunnng the whole model is able to adapt to a new task, but still introduces lots of parameters to be trained. The third method takes from (Zhao al. (020), which freezes the transferable part that extracts the general features and fine-tunes the remaining part that mainly learns the task-specific features achieve task transfer. However, this paradigm is infeasible for DPMs. Because the timestep is an external input for DPM, and the output of DPM is the predicted mean and variance of gaussian noise. Hence, it not reasonable to indicate which part is general or task-specific. An overview of baselines (i), (ii) and our approach is illustrated in Fig, 2
3.3 MASKED SAMPLING AND UNCONDITIONAL SAMPLING
In the sampling stage, our model takes an image as the condition input for conditional generation. Since the generation process is guided by the CLIP embedding, the generated images may be over- constrained by the training data. To ensure the diversity of generated images, we present a simple masked sampling strategy to mask part of the condition image. Concretely, given a mask rate å e where bs is the batch size, ch is the number of
10,1) and condition images X E bhe channels, he and wi are the height and width, respectively, we generate the mask matrix M E Rhe,wi by masking a matrix with all values of It can be easily implemented by identifying the coordinates of upper left corner (rand(0, (1 - 5)he), r0 (1 8) ()wr)) and lower right corner (ōhe + rOnd(0, (1 - he), Swi + rand(0, (1 - w)wi)), where rand() is an integer random function. Then, we obtain the masked images X' by repeating the mask matrix to match the size of inputs. By forwarding the masked images to the CLIP model, we obtain the perturbed semantic representations for diverse conditional sampling.
Another extension of our work is unconditional generation. Recalling that the key insight of our work is the efficient tuning approach based on the ANL module, which can be generalized to the unconditional generative models. Specifically, we replace the cross-attention module with the self- attention module since unconditional generative models take no condition input. In other words, we tune the self-attention layers and non-linear mapping layers to achieve task transfer. By this way, the semantic encoder (c.g., CLIP) is discarded. Compared the aforementioned conditional models, unconditional models take more optimization steps, but show more flexibility since they have no need of the condition input during sampling.
EXPERIMENTS
In this section, we evaluate the proposed techniques in natural image generation and its poten- tial application on semi-supervised learning. Specifically, following the protocols in (Zhao et al., 2020), we transfer the latent diffusion model (LDM) pretrained on the large-scale dataset Ima- geNet (Krizhevsky et al. 2012) to three smaller datasets: CelebA (Liu et al. 2015) (202.599), Flowers (Nilsback & Zisserman 2008) (8, 189), StanfordCars (Krause et al. 2013) (7,350) and their variants: CelebA- 1k, Flowers- 1k, StandfordCars- 1k and the extreme scenario with 25 training sam- ples. (Zhao et al. 2020) also conducts experiments on the Cathedral dataset, which is not released now. Further, we take our approach as a data augmentation technique, and demonstrate its effective- ness in semi-supervised learning (StandfordCars (Krause et al. 2013)).
5

---

Under review as a conference paper at ICLR 2023
Table 1:
FID scores (L) of GAN-based and DPM- approaches after 60k iterations. "failed" indicates training or mode collapse. "*" means the results are obtained on CelebA -0kk rather than CelebA fuul..
Mathod/Target
CelebA
Flowers
Cars
CelebA-Ik
Flowers- Ik
Cars-Ik
TransferGAN
18.69
failed
failed
ScratchGAN
16.51
29.65
11.77
20.75
58.18
39.97
Zhao et al. 2020
9.90
16.76
10.10
19.77
43.05
35.88
SifttchDiff
*46.97
failed
failed
failed
failed
failed
Aul
*17.8
15.75
13.30
24.60
29.79
29.54
Ours (unconditional)
$12.77
12.31
16.89
15.30
22,43
22.76
Ours (conditional)
*8.76
9.97
11.15
14.34
21.87
15.70
Table 2: Classification accuracy (%) StandfordCars with semi-supervised learning evaluation protocols in
the unified benchmark @iang et al. 2022).
Label Set/Method
ERM
Pscudo Label
UDA
Fixmatch
Self Tuning
Flexmatch
Avg
4 labels per category
37.1 40.3
40.4
42.7
44.8
59.8
55.3
46.68
+ augmentation (ours)
43.0
43.9
46.5
59.6
56.1
48.23
4.1 EVALUATION PROTOCOLS AND IMPLEMENTATION DETAILS
To demonstrate the effectiveness of proposed tuning approach in natural image generation, com- pare our approach with three GAN-based baselines: TransferGAN (Wang et al.] 2018), Scratch- GAN, (Zhao et al., 2020). and two DPMs-based baselines: ScratchDiff, Tuning- All. TransferGAN initializes with the pretrained model (Mescheder et al. 2018) and fine-tunes the whole parameters. ScratchGAN trains GP-GAN (Mescheder et al. 2018) from scratch. (Zhao al. 2020) investigates the transferable parts of pretrained GANs, and finettunes them with further proposed adaptive filter modulation. As for the DPM-based methods, the insight of SeratchDiff and Tuning-All are similar to ScratchGAN and TransferGAN, respectively. For implementation, we adopt the latent diffusion model (LDM) (Rombach et al.) (022) pretrained on ImageNet with the resolution of 256 256. The hyper-parameters in LDM are retained. To obtain the semantic embeddings of inputs, we use the pretrained CLIP model (Radford et al. (02I) with VIT-B/32 backbone (Dosovitskiy et al.) 2020). The base learning rate is set to 2.0c-6 and will be scaled up by the batch size and the number of GPUs. For sampling, we use the DDIM sampler (Song et al. 2020a) with 200 sampling timesteps. We use RRX 3090 GPUs with batch size of 8. Due to the fewer training parameters, our method can also be implemented on a single RTX 2080 Ti GPU with 11GB memory.
To validate the potential applications in data-scarce scenarios, take the semi-supervised learning (SSL) as an example, which aims at learning a great recognition model with annotating few training samples. We randomly sample four labels per category, and then generate additional 10 samples for each labeled sample. The generated images will be involved into the labeled sample set. We evaluate it with the empirical risk minimization (ERM) baseline and five state-of-the-a SSL meth- ods: Pseudo Label (Lee et al. 2013), UDA (Xie et al. 2020), FixMatch (Sohn et al. 2020), Self Tuning (Wang et 2021) and Flexmatch (Zhang et 2021
4.2 QUANTITATIVE RESULTS
Quality of Generated Images. We compare the quality of generated images by different approaches
quantitatively, as measured by the widely used FID score (Heusel et al. (017). For training with the full dataset, the FID score is computed w.r.t the training set, as is standard practice (Ho et al. 2020). For training with subset of the dataset, we compute the FID score on the full dataset. From Table we have the following findings: (1) For the limited training data scenarios, fine-tuning the pretrained
GP-GAN leads to mode collapse, while training the LDM from scratch fails to generate meaningful images with limited training steps. (2) Tuning the whole pretrained LDM is able to transfer the learned knowledge to a new task, and is comparable to the previous state-of-thed art (Zhao et al. 2020) the limited data scenarios (i.e., CelebA-1k, Flowers- Ik and StandfordCars- 1k). (3) Our proposed tuning approach achieves the best FID scores on most cases, significantly outperforms other DPM-based methods with much less training parameters.

---

images are Under review as a conference paper at ICLR 2023
CelebA
Flowers
StanfordCars
10k
25
full
full
1k
Figure 3: Images generated by our proposed tuning approach. We illustrate the results on three datasets: CelehA, Flowers and StandfordCars with different training sample sizes: full, 10k, Ik and 25. In the top three rows, the condition images come from the training set (i.e., seen for the model). In the next three rows, the condition images are unseen the model. Note that for Flowers-full and StandfordCars-ful generated by the seen conditions.
Ablation Study, Our conditional model leverages the CLIP embeddings achieve fast task transfer. To verify the effectiveness of this technique, we show the performance of unconditional model in Table We can observe that the unconditional model performs slightly worse than the conditional ones since it lacks the semantic guidance. However, the unconditional model shows superiority on flexibility since has no need of condition input during sampling. This ablation study reveals that the pre-trained encoder (e.g., CLIP) is not necessary for transferring pre-trained DPMs, but it can provide noticeable boost. It is worth noticing that both unconditional and conditional mod- els perform better than the baseline Tuning-All, demonstrating the efficacy of our proposed tuning approach.
Tuning DPMs for Data Augmentation. In this part, we empirically demonstrate the potential of our tuning approach data augmentation. From Table 2 the augmented samples generated by our tuming approach improves the classification accuracy that the augmented samples brings 3.2% and 2.6% improvement on the ERM and Pscudo Label, respectively. Compared to other SSL methods, the learning strategies of these two methods simple, implying the effectiveness the high-quality generated samples data augmentation.
five SSL methods. It is worth noticing
4.3 QUALITATIVE RESULTS
Image Synthesis with Different Sample Sizes. In Fig. we show the generated images by our tun- ing method with different training sample sizes. We can observe that our method effectively transfers the pretrained DPM generate with limited data. For CelebA-10k, Flowers-full and StandfordCars- full, the adapted model is able to generate high-quality images for both seen and unseen conditions. For small training size (e.g., CelebA-1k), 1k), the quality and reality of generated images are still com- petitive. As for learning with extremely limited data (e.g., 25) and sampling with unseen conditions, the generated images are slightly blurred, and the model conditioned with different images tends to generate similar images. This is because that the tuned model inevitably suffers from over-fitting when training with extremely limited data. However, when generating with seen conditions, our tuning approach is still effective with extremely limited training data. Another interesting finding that the images generated with unseen conditions on CelebA usually focus on the faces and have blank backgrounds. More generated images provided in Appendix C

---

Under review as a conference paper at ICLR 2023
Image Synthesis with Few Tuning Steps. To validate the efficiency of our proposed tuning ap- proach, we show the generated images with different tuning steps from 500 to 4000 with the interval of 500. From Fig. I
the coarse-to-fin adaptation process during tuning the pretrained LDM. In the beginning, our method learns the general outlines and shapes, and then supplements the details continually. From Fig. show the comparison between Tuning-All and our approach. We can find with less trainable parameters, our approach shows faster and better adaptation. It is worth noting that our tuning approach shows impressive generation quality with very limited iter- ations, which provides a convincing path for transferring pretrained DPMs to new dataset with limited data and training resources.
ITl Bunin
(0.5K: 33,83)
(IK: 45.38)
(L.5K: 34.25)
(2K: 33.86)
(2.5K: 27.76)
(3K: 30.26)
Smn
(0.5K: 32.86)
27,47)
28.27)
(2:97))7)
22 22.74)
(3K:21.76)
Figure 4: Generated images by Tuning-All All and approach with few tuning steps. To better validate approach, the Tuning-All also utilizes the CLIP embeddings here. The first number under the image is the number of iterations, the one after is FID score.
Simple Text-to-Image Synthesis. Recalling that leverage the pretrained CLIP model obtain the semantic embeddings of condition images, our method can achieve simple text-to-image synthe- sis since the CLIP embedding space is share by both vision and language modalities. We illustrate some banguage-prompeee
examples on the StandfordCars dataset in Fig. 5 We can observe that our method is able to generate suitable images w.r.t the given prompt texts like "a red car", "A black car driving on the road." and "There is a white car in front of the house". However, when given the prompt texts like "a truck", "a green car" or "A car driving the rain.", the model fails generate the corresponding images.
prompt red car
prompt A black driving the road.
prompt text: There is a white front of the house.
prompt text (failure case): A driving in the rain.
Figure 5: Illustration of simple text-to-image synthesis the StandfordCars dataset.
Masked Sampling, We set the mask rate = 0.5 (Sec. 3.3 and show the sampling results in Fig. 66]
We can find that masked sampling promote the diversity of generated images when given the same condition image. Note that the mask rate is trade-off between quality and diversity. When increasing the value of mask rate 8, the diversity of generated images will be improved while the quality will be impaired. We find 8 0.5 is great choice for balancing the diversity and quality. We provide detailed quantitative results Appendix A
condition
sampled
Figure 6: The results of diverse sampling.
8

---

Under review as a conference paper at ICLR 2023
5
RELATED WORKS
Transfer Learning and Few-Shot Image Generation. Transfer learning focuses on reusing learned knowledges from previous tasks for the learning of new tasks (Pan & Yang 2009). For GAN- based image generation, TransferGAN (Wang et al.) 2018) propose to fine-tune the whole model pretrained on source domain with limited target data. Zhha et al. (2020) follow the insights of transfer learning in discriminative tasks, and proposed to freeze the generally-appliabble low-level filters of convolutional neural networks.
Another similar topic is few-shot image generation (Hong et al., 2020; Noguchi & Harada 2019; Ojha et al. 2021, Liu al., 2020). Despite that few works are proposed, most of them focus on GANs, and infeasible for DPMs since they share totally different architectures and learning objec- tives. For DPM-based few-shot image generation, [Sinha al. (202T) uses a learned diffusion-basee prior over the latent representations to improve generation. (Sinha et al. (021) mainly addresses minor manipulations like generating faces with the blond hair or adding the beard for male face.
Recently, concurrent work (Giannone et al. 2022) propose few-shot diffusion models by intro- ducing a trainable vision transformer to condition diffusion models. It is worth noticing that (Gian- none et al. 2022) trains additional vision transformer and the whole diffusion models, while our method propose to take full advantages of pretrained models and only tunes the ANL modules.
Diffusion Probabilistic Models. Diffusion probabilistic models (DPMs) have been demonstrated the effectiveness in high-quality image synthesis (Ho et al., 2020). The DPM is first proposed by (Sohl-Dickstein et al.) (015) to generate by reversing diffusion process using Markov chain with discrete timesteps. DDPM (Ho et al. 2020) leverages noise prediction networks to parame- terize the mean of the Markov chain, which different timesteps share the same parameters. To improve DDPM, various approaches are proposed from different aspects, such as learning on latent space (Pandey et al.] [0022] Rombach et al., 2022). faster sampling (Song et al.) 2020al Liu et al.) 2022]. Song et al. (2020b) propose that the DPM with infinitesimal timesteps can be representation by stochastic differential equations (SDEs). To improve the score-based diffusion models, [Song al. (2021) further propose seelihood-basede training scheme, and lots of efforts are paid for faster sampling like designing faster SDEs solvers (Jlicoeur-Martineal et al. 2021: Popov et al., 2021
or analytically estimate the optimal reverse variance (Bao et al.) 2022a
6 LIMITATIONS AND SOCIETAL IMPACT
Limitations. While our proposed tuning approach significantly reduces optimization steps and the demand for training data, it fails to generate high-quality images with unseen conditions when ex- tremely limited data (c.g.. 25) are provided. Another limitation is that although our approach reduces the training parameters from 400M (conditional L.DM pretrained on ImageNet with the resolution of 256x 256) to 169M, 169M trainable parameters are still little bit large. The parameters mainly come from the non-lineat mapping module which consists two fully-connected layers and an acti- vation layer. Further reducing the parameters that need tuning is direction for future works.
Societal Impact. Our paper presents general transfer learning approach for DPMs, so it may help expand any impact that generative models had on the broader world. For natural image generation, generative models may encounter biased, deliberate synthesis and copyright infringement issucs. Our method enables transferring pretrained DPM to specific task using much less data and computing resources. We hope that our approach can be used to achieve various creative and positive generations, such as synthesizing early prototypes for artists, or augmenting and extending datasets for the data-scarce scenarios (Antoniou et al. 2017).
7 DISCUSSIONS
We introduce a new tuning approach for leveraging pretrained DPMs to generate high-quality natu- ral images with limited training data and optimization steps. The advances in DPMs such as faster sampling algorithms can be used to further improve our tuned model since our method are orthog- onal to them. To the best of our knowledge, learning in diffusion probabilistic models. We hope our work paves the way for small laboratories and companies as well as individual researchers to train task-specific generative models.
are at the very early attempt to investigate transfer

---

Under review as a conference paper at ICLR 2023
REFERENCES
Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative ad adversarial networks. arXiv preprint 1rXiv:1711.043400 2017.
Fan Bao, Chongxuan Li, Jiacheng Sun, Jun Zhu, and Zhang. Estimating the optimal covariance with imperfect mean in diffusion probabilistic models. arXiv preprint arXiv:2206.07388 2022a.
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. arXiv preprint arXiv:2201.0633 2022b.
Yoshua Bengio. Deep learning of representations for unsupervised and transfer learning. In Pro- ceedings of ICML workshop on unsupervised and transfer learning, pp. 17-36. JMLR Workshop and Conference Proceedings, 2012.
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Con- ditioning method for denoising diffusion probabilistic models. arXiv preprini 0rX1:2108..02988, 2021.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 2018.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems, 118780-8794, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 2020.
Giorgio Giannone, Didrik Nielsen, and Ole Winther. Few-shot diffusion models. arXiv preprint arXiv:2205.15463. 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6855 2020.
Yan Hong. Li Niu, Jianfu Zhang, Jing Liang, and Liqing Zhang. Deltagan: Towards diverse few-shot
image generation with sample-specific delta. arXiv preprint arXiv:2009.08753 2020.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799. PMLR, 2019.
Junguang Jiang, Yang Shu, Jianmin Wang, and Mingsheng Long. Transferability in deep learning: A survey, 2022.
Alexia Jolicoeur-Martineau, Ke Li, Rémi Piché-Taillefer, Tal Kachman, and loannis Mitliagkas. Gotta go fast when generating data with score-based models. arXiv preprint arXiv:2 14080, 2021.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision work- shops, pp. 554-561, 2013.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
10

---

Under review as a conference paper at ICLR 2023
Dong-Hyun Lee al. Pseudo-label: The simple and efficient supervipe learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, pp. 896, 2013.
Bingchen Liu, Yizhe Zhu, Kunpeng Song, and Ahmed Elgammal. Towards faster and stabilized gan training for high-fidelity few-shot image synthesis. In International Conference on Learning Representations, 2020.
Luping Liu, Yi Ren, Zhijic Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint arXiv:2202.09778, 2022.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. arXiv preprint arXiv: S.Xiv:/502.02991,, 2015.
Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2837-2844 2021.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In International conference on machine learning, pp. 3481-3490. PMLR, 2018.
Marzich Mozafari, Reza Farahbakhsh, and Noel Crespi. A bert-based transfer learning approach for hate speech detection in online social media. In International Conference on Complex Networks and Their Applications, pp. 928-99989999 2889900. Springer, 2019.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffuusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171, PMLR, 2021.
Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. Diffusion models for adversarial purification. arXiv preprint arXiv:2205.07460 2022.
Maria-Elema Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. 2008 Sixth Indian Conference on Computer Vision, Graphics Image Processing, pp. 722-729. IEEE, 2008
Atsuhiro Noguchi and Tatsuya Harada. Image generation from small datasets via batch statistics adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 2750-2758, 2019.
Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexei A Efros, Yong Jae Lee, Eli Shechtman, and Richard Zhang. Few-shot image generation via cross-domain correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10743-10752, 2021.
Sinno Jialin Pan and Qiang Yang. survey on transfer learning. IEEE Transactions on knowledge 2009.
and data engineering,
Kushagra Pandey, Avideep Mukherjee, Piyush Rai, and Abhishek Kumar. Diffusevae: Effi- cient, controllable and high-fidelity generation from low-dimensional latents. arXiv preprint arXiv:2201.00308 2022.
Yifan Peng, Shankai Yan, and Zhiyong Lu. Transfer learning in biomedical natural language processing: an evaluation of bert and elmo on ten benchmarking datasets. arXiv preprint arXiv: 2019.
Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, Mikhail Kudinov, and Jiansheng Wei. Diffusion- based voice conversion with fast maximum likelihood sampling scheme. arXiv preprint 109, /3821, 2021.
11

---

Under review as a conference paper at ICLR 2023
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising dif- fusion models for multivariate probabilistic time series forecasting. In International Conference on Machine Learning. pp. 8857-8868. PMLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Negar Rostamzadeh, Emily Denton, and Linda Petrini. Ethics and creativity in computer vision. arXiv preprint 0r.iv:2112.03111 2021.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Hiroshi Sasaki, Chris Willcocks, and Toby P Breckon. Unit-ddpm: Unpaired image translation with denoising diffusion probabilistic models. arXiv preprint arXiv:2104.05333 2021.
Abhishek Sinha, Jiaming Song, Chenlin Meng, and Stefano Ermon. D2c: Diffusion-decoding mod- els for few-shot conditional generation. Advances in Neural Information Processing Systems, 34: 12533-12548, 2021.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing. pp. 2256-2265. PMLR, 2015.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-super learning with consistency and confidence. Advances in neural information processing systems, 33:596-608, 2020.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint wrXii:2010.02502. 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 1. 13456, 2020b.
Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score- based diffusion models. Advances in Neural Information Processing Systems, 34:1415-1428,, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Ximei Wang, Jinghan Gao, Mingsheng Long, and Jianmin Wang. Self-tuning for data-efficient deep learning. International Conference on Machine Learning, pp. 177738-1778... PMLR, 2021.
Yaxing Wang, Chenshen Wu, Luis Herranz, Joost van de Weijer, Abel Garcia, and Bog- dan Raducanu. Transferring gans: generating images from limited data. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 218-234, 2018
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. Advances in Neural Information Processing Systems, 33:6256-6268 2020.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in neural information processing systems, 27, 2014.
12

---

Under review as a conference paper at ICLR 2023
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. Proceedings of the IEEE con- ference computer vision and pattern recognition, pp- 3712-3722, 2018
Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo la- beling, Advances in Neural Information Processing Systems, 34:18408-18411 18419, 2021.
Miaoyun Zhao, Yulai Cong, and Lawrence Carin. On leveraging pretrained gans for generation with limited data. In International Conference on Machine Learning, pp. 11340-11311 PMLR, 2020.
Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. Egsde: Unpaired image-to- image translation via guidey stochastic differential equations. arXiv preprint arXiv:2207.06633. 2022.
13

---

Under review as a conference paper at ICLR 2023
A RESULTS OF DIFFERENT MASK RATE
In Sec. 3.3 we propose a simple masked sampling approach for conditional diverse sampling. In- tuitively speaking, the masked sampling will increase the diversity of generated samples, but hurt the quality. To verify it, we use the FID score to evaluate the quality, and the Inception Score (IS) evaluate the diversity, respectively. The results are shown in Table Since the StandfordCars is single-class dataset, the inception score will always keep small value. From Table [3] we can find the FID and IS scores increase by enlarging the mask rate, revealing that there exists trade-off between the quality and diversity of generated images.
Table 3:
The FID and IS scores of the generated images with different mask rate. The evaluated model is trained StandfordCars dataset with 4k tuning iterations.
mask rate
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
IS + FID 4
3.18
3.14
3.10
3.18
3.25
3.46
3.56
3.87
4.12
4.21
22.74
23.23
22.96
24.08
25.08
27.28
35.08
44.20
55.86
68.38
B GENERALIZE TO OTHER DPMs
In this paper, focus natural image generation with latent diffusion models (LDM). Here, we provide intuitions for other DPMs like DDPM. Unlike the pre-trained LDM, the pre-trained DDPM does not contain the ANL module, thus may introduce more optimization steps since the ANL module will be randomly initialized. However, since LDM inserts the ANL module into each residual convolutional block of U-Net, is not convenient further reduce the number of layers that needed to be tuned. For DDPM, we suggest to discover more efficient insert pattern further reduce the tuning parameters. Another interesting direction to extend our method to other generative tasks like text-to-image generation, text-to-speech synthesis and protein structure generation. For example, stable diffusion model (Rombach et al. 2022) has achieved impressive performance text-to-image generation, and our method can be used to transfer it to specific domain.
C ADDITIONAL SAMPLES
In Fig. we have shown some images sampled by different models. Here, we provide more gener- ated images with different conditions in Fig. 10 12 and 16. To avoid the pdf file being too large, we compressed these images to 96 dpi.
14

---

Under review as a conference paper at ICLR 2023
WWW
RYA
20
21
ONS
Figure 7: The 256 sampled images on CelebA-10k with seen condition images.
15

---

Under review as a conference paper at ICLR 2023
2
Figure The 256x256 sampled images on CelebA-10k with unseen condition images.
16

---

Under review as a conference paper at ICLR 2023
booun
ii
Zoom
OTH
Figure 9: The 256x256 sampled images on CelebA-Ik with seen condition images.
17

---

Under review as a conference paper at ICLR 2023
Figure 10: The 256x256 sampled images on CelebA-Ik with unseen condition images.
18

---

Under review as a conference paper at ICLR 2023
Figure 11: The 256x256 sampled images on Flowers-full.
19

---

Under review as a conference paper at ICLR 2023
Figure 12: The 256x256 sampled images on Flowers- Ik with seen condition images.
20

---

Under review as a conference paper at ICLR 2023
Figure 13: The 256x256 sampled images on Flowers- Ik with unseen condition images.
21

---

Under review as a conference paper at ICLR 2023
INO
Figure 14: The 256x256 sampled images on Cars-full.
22

---

Under review as a conference paper at ICLR 2023
Figure 15: The 256x256 sampled images on Cars- Ik with seen condition images.
23

---

Under review as a conference paper at ICLR 2023
Figure 16: The 256x256 sampled images on Cars-Ik with unseen condition images.
24


Reviews:
[{"review_id": "zjjkqeFu5U", "reviewer": "Reviewer_YaAB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an efficient way of fine-tuning diffusion models on new datasets. Instead of training the entire network or training from scratch, the authors add a attention non-linear block which is the only learnable part which is learned during fine-tuning. The attention non-linear block is a cross attention layer followed by a 2 layer NN. Results are shown on a few datasets where the method shows better results.", "summary_of_the_review": "The merits of the paper outweigh the weaknesses, so I am recommending the paper for acceptance. Would have given a 7 if there was an option, but its better than a 6, so I am marking it as an 8.\n\nAfter reading reviews of other reviewers about concerns around novelty, I am updating my rating to 6.", "strength_and_weaknesses": "Strengths:\n1) The method is simple to implement and shows strong results over good baselines.\n2) It is all the more important given that new latent diffusion models like 'stable diffusion' are publicly available which are trained on hundreds of millions of images and fine-tuning on newer datasets with less data will become even more important. The paper proposes a neat trick to address this problem.\n\nWeaknesses:\nThe cross attention based additional architecture isn't very novel and there have been several attempts around this framework traditional deep learning tasks.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "sIt6bV_1HB", "reviewer": "Reviewer_wtTC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on exploration of transfer learning in diffusion models, technically, they propose an Attention-NonLinear (ANL) module to facilitate the conditioning process of CLIP embedding. To avoid the diffusion model overfits into the clip embedding of the images, they also generated an approximate clip embedding from masked images. \n\nThe experimental results show that the adaptation can be achieved by few-training data and steps, and outperforms related baselines. Also, the generated samples can further boost the performance of semi-supervised learning.", "summary_of_the_review": "This paper is well motivated, while lacking in some in-depth analysis around the proposed contributions. Some details are missing to hinder the reproducibility of the paper. ", "strength_and_weaknesses": "Pros:\n\n- The paper is well written and easy to follow.\n\n- The proposed method is well motivated and convincing in real-life scenarios, as the training of diffusion models is really time-consuming.\n\n- The experimental result is solid compared with related baselines.\n\n\nCons:\n\n-\"We investigate transfer learning in recent DPMs, and uncover that previous methods like training from scratch or determining the transferable parts are not effective.\", I am doubtful about this statement in contribution and the failed result in Table 1, maybe running DDPM instead of LDM cannot fail? Failure on LDM is expected, as you cannot update the Encoder-Decoder in LDM in your fine-tuning, any sensitive parameter updating inside will greatly affect the sampling quality of LDM.\n\n- ANL module is not novel, already been proposed in latent diffusion models, etc.\n\n- Contribution 2 is not clear, efficient on what? speed? parameter number? convergence? \n\n- Lacking in-depth analysis of ANL modules, which layer should we insert the ANL module? Should we insert it into all layers by default? Maybe we can find a more efficient inserting pattern?\n\n- Lacking in analysis of overfitting problem, the models will meet the problem of catastrophic forgetting so that it fails to generate samples from old distribution.\n\n- In Figure 2, your model's trainable parameter number is 169M for few images(1k)? This should inevitably lead to an overfitting problem, as mentioned above. \n\n- In sampling, you still need the CLIP embedding, where is the clip-embedding comes from? How masked sampling performs to increase the diversity of the generated samples (the effect of Fig6 can be quantified by Inception Score as function of mask rate.)? As CLIP embedding is a too-fine-grained condition so that the sampling might only mesmerize the training data. \n\n- The exposition of masked sampling is too abrupt, the motivation is missing there.\n\n- FID/IS of generated samples should be compared before and after deploying the masking, I think there will be trade-off issue between FID and IS. \n\n- Details about the \"tuning DPMs for data augmentation\" is missing, how to obtain the condition of CLIP embedding during this process of sampling? How many samples are used in this experiment?\n\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_Ygv-PLEGS4", "reviewer": "Reviewer_2H8p", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces an efficient method to finetune pre-trained LDM to small data. The problem is formulated as an image-to-image generation task. A conditional image is first encoded by the CLIP image encoder and then injected into the LDM by the cross attention mechanism. To avoid overfitting, only the parameters of attention and non-linear layers are finetuned. Experiments show the effectiveness both quantitatively and qualitatively.\n", "summary_of_the_review": "This paper proposes an efficient fintuning method to adapt pre-trained LDM to small datasets. The method is well-motivated and the results are promising, but some details and ablations are missing. See the above comments.", "strength_and_weaknesses": "Strengths\n\n1. It proposes to transfer pretrained LDMs to small data and use limited training resources.\n\n2. The CLIP embeddings are used as the conditions of LDMs and a portion of model parameters (Attention-NonLinear) are finetuned to be efficient.\n\n3. Experiments demonstrate better FID scores than finetuning all the parameters and the GAN related methods.\n\nWeaknesses\n\n1. Some equation types exist. For example, the right of Equation 1 and the left of Equation 3 have one parenthesis missing, respectively. The Gaussian distribution in Equation 1 and 2 have inconsistent forms. Both d_c and d^c appear in the descriptions under Equation 8.\n\n2. Table 2 uses the semi-supervised setting. It generates 10 additional samples for each labeled sample. Can we assume the 10 generated samples have the sample labels as their original sample? Since the generation is conditioned on the original sample, it’s very likely that the generated variants still keep the label. If so, it could be just supervised learning with data augmentations.\n\n3. Section 3.3 mentions using masked images as conditional input. Are they used in training, generation, or both? If used in training, any ablation study of delta? Figure 6 only shows illustrations of masking used in generation.\n\n4. I feel the task is defined clearly. The experiments use the LDM pre-trained on ImageNet. Is the pre-training task a noise-to-image generation task? In the finetuning, the task is an image-to-image generation task due to the image conditioning. This change is not described clearly in the paper.\n\n5. Are there two factors of randomness in generating images, i.e., the random masking of the conditional image and the random Gaussian noises? I guess Figures 3 and 5 use random Gaussian noises but no image masking, and Figure 6 uses fixed Gaussian noises but random conditional image mask, right?\n\n6. CLIP embedding and Attention-NonLinear are two components of the proposed transfer learning method. What if removing the CLIP embedding part, i.e., only finetuning the NonLinear part of the LDM? With CLIP, it is an image-to-image generation formulation since it generates one image conditioned on an image. It’s unclear for me how important the CLIP embedding is in the proposed method. Which one really matters in the low-data regime: the image-to-image generation formulation help (compared to the pure noise-to-image generation) or the CLIP embedding? Maybe also try a different embedder, e.g, a pre-trained vision transformer.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZUMsQ_kJZP", "reviewer": "Reviewer_uRV2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This submission presents a method for fine-tuning large generative models on small datasets efficiently. The proposed approach used CLIP to encode images from the small datasets into a semantic latents space that is then used to condition the frozen pre-trained model via a cross-attention module. The approach is evaluated on several standard benchmarks datasets outperforming the baselines where models are trained from scratch or where all the parameters of the pre-trained model are fine-tuned. ", "summary_of_the_review": "This paper proposes an approach for fine-tuning large pre-trained generative models on small datasets in an efficient way. I believe this problem is interesting for the community and deserves exposure. The proposed approach is technically correct and the empirical evaluation is sound. However, I believe there’s a couple of major baselines that need to be run in order to better understand the contributions (see weaknesses). \n", "strength_and_weaknesses": "Strengths:\n- The problem of fine-tuning large pre-trained models on small datasets in an efficient way is interesting and deserves exposure in the community.\n- The proposed approach is technically correct and the empirical results support the main claims in the paper.\n\nWeaknesses:\n- There’s one important baseline missing. That is figuring out how important the semantic latent c is for this approach to work. In other words, would this work with latents that do not come from big models like CLIP? I suggest the authors run the following experiments to better understand this phenomenon: - 1) Use a CLIP visual encoder with random weights and fine tune it on the small dataset. - 2) Let the latent c of each example be non-parametric (eg. a nn.Embedding in pytorch). In this way, one will let the fine-tuning approach to backprop all the way to these non-parametric latents.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject