Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an efficient way of fine-tuning diffusion models on new datasets. Instead of training the entire network or training from scratch, the authors add a attention non-linear block which is the only learnable part which is learned during fine-tuning. The attention non-linear block is a cross attention layer followed by a 2 layer NN. Results are shown on a few datasets where the method shows better results.
strength_and_weaknesses: Strengths:
1) The method is simple to implement and shows strong results over good baselines.
2) It is all the more important given that new latent diffusion models like 'stable diffusion' are publicly available which are trained on hundreds of millions of images and fine-tuning on newer datasets with less data will become even more important. The paper proposes a neat trick to address this problem.

Weaknesses:
The cross attention based additional architecture isn't very novel and there have been several attempts around this framework traditional deep learning tasks.

clarity,_quality,_novelty_and_reproducibility: The method is simple and clearly explained. It barely meets the technical novelty bar, but is very relevant to the work going on in the field today. The authors do not mention if they will release the code, so it is not clear how reproducible would be the method, but it does look simple enough for someone to try it out.
summary_of_the_review: The merits of the paper outweigh the weaknesses, so I am recommending the paper for acceptance. Would have given a 7 if there was an option, but its better than a 6, so I am marking it as an 8.

After reading reviews of other reviewers about concerns around novelty, I am updating my rating to 6.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces an efficient method to finetune pre-trained LDM to small data. The problem is formulated as an image-to-image generation task. A conditional image is first encoded by the CLIP image encoder and then injected into the LDM by the cross attention mechanism. To avoid overfitting, only the parameters of attention and non-linear layers are finetuned. Experiments show the effectiveness both quantitatively and qualitatively.

strength_and_weaknesses: Strengths

1. It proposes to transfer pretrained LDMs to small data and use limited training resources.

2. The CLIP embeddings are used as the conditions of LDMs and a portion of model parameters (Attention-NonLinear) are finetuned to be efficient.

3. Experiments demonstrate better FID scores than finetuning all the parameters and the GAN related methods.

Weaknesses

1. Some equation types exist. For example, the right of Equation 1 and the left of Equation 3 have one parenthesis missing, respectively. The Gaussian distribution in Equation 1 and 2 have inconsistent forms. Both d_c and d^c appear in the descriptions under Equation 8.

2. Table 2 uses the semi-supervised setting. It generates 10 additional samples for each labeled sample. Can we assume the 10 generated samples have the sample labels as their original sample? Since the generation is conditioned on the original sample, it’s very likely that the generated variants still keep the label. If so, it could be just supervised learning with data augmentations.

3. Section 3.3 mentions using masked images as conditional input. Are they used in training, generation, or both? If used in training, any ablation study of delta? Figure 6 only shows illustrations of masking used in generation.

4. I feel the task is defined clearly. The experiments use the LDM pre-trained on ImageNet. Is the pre-training task a noise-to-image generation task? In the finetuning, the task is an image-to-image generation task due to the image conditioning. This change is not described clearly in the paper.

5. Are there two factors of randomness in generating images, i.e., the random masking of the conditional image and the random Gaussian noises? I guess Figures 3 and 5 use random Gaussian noises but no image masking, and Figure 6 uses fixed Gaussian noises but random conditional image mask, right?

6. CLIP embedding and Attention-NonLinear are two components of the proposed transfer learning method. What if removing the CLIP embedding part, i.e., only finetuning the NonLinear part of the LDM? With CLIP, it is an image-to-image generation formulation since it generates one image conditioned on an image. It’s unclear for me how important the CLIP embedding is in the proposed method. Which one really matters in the low-data regime: the image-to-image generation formulation help (compared to the pure noise-to-image generation) or the CLIP embedding? Maybe also try a different embedder, e.g, a pre-trained vision transformer.

clarity,_quality,_novelty_and_reproducibility: 1. The paper is well-motivated by the fact that training DPMs requires big data and much compute resources. Efficiently finetuning pre-trained LDMs on small data is interesting and novel. 
2. The paper clearly presents the technical components, i.e., CLIP embedding conditioning and Attention-NonLinear finetuning. But it doesn't define the task clearly, a noise-to-image generation or image-to-image translation.
3. No code is provided and some training configurations are unclear. Not sure about reproducibility.
summary_of_the_review: This paper proposes an efficient fintuning method to adapt pre-trained LDM to small datasets. The method is well-motivated and the results are promising, but some details and ablations are missing. See the above comments.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This submission presents a method for fine-tuning large generative models on small datasets efficiently. The proposed approach used CLIP to encode images from the small datasets into a semantic latents space that is then used to condition the frozen pre-trained model via a cross-attention module. The approach is evaluated on several standard benchmarks datasets outperforming the baselines where models are trained from scratch or where all the parameters of the pre-trained model are fine-tuned. 
strength_and_weaknesses: Strengths:
- The problem of fine-tuning large pre-trained models on small datasets in an efficient way is interesting and deserves exposure in the community.
- The proposed approach is technically correct and the empirical results support the main claims in the paper.

Weaknesses:
- There’s one important baseline missing. That is figuring out how important the semantic latent c is for this approach to work. In other words, would this work with latents that do not come from big models like CLIP? I suggest the authors run the following experiments to better understand this phenomenon: - 1) Use a CLIP visual encoder with random weights and fine tune it on the small dataset. - 2) Let the latent c of each example be non-parametric (eg. a nn.Embedding in pytorch). In this way, one will let the fine-tuning approach to backprop all the way to these non-parametric latents.  
clarity,_quality,_novelty_and_reproducibility: - The paper is clearly written and easy to understand. I believe the proposed approach is novel and technically sound. The details to reproduce the experiments in the submission are provided in the paper and all the datasets are publicly availalble. 
summary_of_the_review: This paper proposes an approach for fine-tuning large pre-trained generative models on small datasets in an efficient way. I believe this problem is interesting for the community and deserves exposure. The proposed approach is technically correct and the empirical evaluation is sound. However, I believe there’s a couple of major baselines that need to be run in order to better understand the contributions (see weaknesses).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on exploration of transfer learning in diffusion models, technically, they propose an Attention-NonLinear (ANL) module to facilitate the conditioning process of CLIP embedding. To avoid the diffusion model overfits into the clip embedding of the images, they also generated an approximate clip embedding from masked images. 

The experimental results show that the adaptation can be achieved by few-training data and steps, and outperforms related baselines. Also, the generated samples can further boost the performance of semi-supervised learning.
strength_and_weaknesses: Pros:

- The paper is well written and easy to follow.

- The proposed method is well motivated and convincing in real-life scenarios, as the training of diffusion models is really time-consuming.

- The experimental result is solid compared with related baselines.


Cons:

-"We investigate transfer learning in recent DPMs, and uncover that previous methods like training from scratch or determining the transferable parts are not effective.", I am doubtful about this statement in contribution and the failed result in Table 1, maybe running DDPM instead of LDM cannot fail? Failure on LDM is expected, as you cannot update the Encoder-Decoder in LDM in your fine-tuning, any sensitive parameter updating inside will greatly affect the sampling quality of LDM.

- ANL module is not novel, already been proposed in latent diffusion models, etc.

- Contribution 2 is not clear, efficient on what? speed? parameter number? convergence? 

- Lacking in-depth analysis of ANL modules, which layer should we insert the ANL module? Should we insert it into all layers by default? Maybe we can find a more efficient inserting pattern?

- Lacking in analysis of overfitting problem, the models will meet the problem of catastrophic forgetting so that it fails to generate samples from old distribution.

- In Figure 2, your model's trainable parameter number is 169M for few images(1k)? This should inevitably lead to an overfitting problem, as mentioned above. 

- In sampling, you still need the CLIP embedding, where is the clip-embedding comes from? How masked sampling performs to increase the diversity of the generated samples (the effect of Fig6 can be quantified by Inception Score as function of mask rate.)? As CLIP embedding is a too-fine-grained condition so that the sampling might only mesmerize the training data. 

- The exposition of masked sampling is too abrupt, the motivation is missing there.

- FID/IS of generated samples should be compared before and after deploying the masking, I think there will be trade-off issue between FID and IS. 

- Details about the "tuning DPMs for data augmentation" is missing, how to obtain the condition of CLIP embedding during this process of sampling? How many samples are used in this experiment?





clarity,_quality,_novelty_and_reproducibility: 
Clarity: 9/10

Quality: 7/10

Novelty: 6/10

Reproducibility: 5/10

summary_of_the_review: This paper is well motivated, while lacking in some in-depth analysis around the proposed contributions. Some details are missing to hinder the reproducibility of the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduced a method to fine tune large generative models on small datasets. In particular, the authors used CLIP to encode images from the small datasets into a semantic latent space, and frozen pre-trained model via a cross-attention module. The authors have done basic experiments to evaluate the performance of the proposed algorithm. 



justification_for_why_not_higher_score: Reviewers share some major concerns about this paper. In particular, they thought fine-tuning a subset of parameters is nothing new and has already been a technique that is widely known in the field. Also, the proposed algorithm is heavily relied on the LDM architecture, which makes it unclear about the generalization ability on standard diffusion models or other models, rather than only LDM. At last, though there might lack relevant comparison works in the literature, the authors only designed quite weak baselines for the comparison experiments.
Decision: Reject