Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This work proposed a transfer learning method for GANs in the limited data domain. It borrow ideas from IMLE (to overcome mode-collapse) and conditional GAN (to improve training stability and generation quality), by introducing data instance prior (plays a similar role to that of  label information in conditional GAN) and knowledge distillation techniques, the model is claimed to be effective in preventing mode collapse and discriminator overfitting.

Though the main idea makes sense to some extent, the writing is a little weak, especially the equations and some statements that are not correctly verified in the experiments, making it difficult to go through the paper. 

Detailed concerns are listed below.

1. Eq 3 is not correct, the sample process should be placed under “Expectation” rather than “minimize”; also, there is no information about which parameter is going to be optimized here. What’s the relation ship between x and x~, are they independently sampled from the target dataset? 

2. Figure 1 is confusing. It shows that the adversarial loss is depend on the projection loss, but there is no equation to show the relationship between these two losses.

3. There are many network components in the proposed method, it is not easy to guess the objective related to the real/fake score in an adversarial manner, a clear equation for the adversarial training is necessary. Also, a detailed training process is necessary, e.g. how to sample, when to optimize the generator, discriminator, and the embedding networks?

4. How to make sure that “enforcing feature Df(G(z|C(x))) to be similar to Demb(C(x)) ensures that for each real sample, there exists”? Apparently, the constrain is made on some latent space and there is no statement to show that close in latent space is equivalent to close in image space.

5. When the data is large scale, doing clustering is inhibitive.

6. On Table 3, the results are not state of the art on CIFAR10 and fall behand the recent works on generation with limited data, e.g. discriminator augmentations (DA) [1],  DiffAugment [2], then what’s the advantage of the proposed method when compare to these methods?

7. An ablation study is suggested. It’s not easy to see the contribution of each part (e.g. knowledge distillation, covering real data modes) to the final performance.

8. It claims that the proposed method can prevent discriminator overfitting, but in the experiments, it is not shown.  

9. What’s the motivation to do interpolation on the data instance prior? How to make sure that the interpolation on the prior (Eq.4) is smooth? 

10. On semantic diffusion for image manipulation. It is not clear how to obtain the results shown in Figure 4/9 on custom editing, e.g. there is only one input image, how to compute and exchange the C(x)? Also, the effects of manipulating high-level semantics and fine-grained details are not observed/discussed in the experiments.


[1] Karras T, Aittala M, Hellsten J, et al. Training generative adversarial networks with limited data[J]. arXiv preprint arXiv:2006.06676, 2020.
[2] Zhao S, Liu Z, Lin J, et al. Differentiable augmentation for data-efficient gan training[J]. arXiv preprint arXiv:2006.10738, 2020.

review_title: A transfer learning method for GANs in the limited data domain 

final_score: rating: 6: Marginally above acceptance threshold
review_content: This submission deals with transfer learning for training GANs with limited label data. The challenge is that training with limited data can result in mode collapse. This submission proposes to use data priors for each instance of the target distribution, transformed through knowledge from a source domain, as conditional information in GAN to ensure mode coverage of the target data distribution. A pre-trained feature extractor is used to provide the information to condition the GAN. A range of experiments is performed with the features extracted from VGG16, SIMCLR. They show consistent improvements in the image quality and diversity, measured via FID and precision-recall,  for few-shot, limited data, and even large scale data settings. 

Strong points
- The idea of creating useful guides based on pre-trained features for conditional GANs makes sense.
The experiments are extensive and they show improved quality and diversity for a wide range of settings.

Correctness
- The idea and the reported experiments make sense. 

Reproducibility
- The code has been provided with the code for pre-training and other details included. 

More suggestions and comments
- IMLE description in (2) is vague. It seems that a gaussian sample z is taken, but on the other hand, for a given network G, z is optimized to match x. The restrictions on z is unclear.
It seems that hinge-loss is used for GAN based on (1)? It would be important if the authors could comment on the choice of the divergence/loss in this setting. One may wonder that the limited data and mode-collapse could be better handled with Wasserstein distance. An ablation study would be very useful to clarify the role of the distance.

review_title:   

final_score: rating: 7: Good paper, accept
review_content: This paper illustrates how they train GANs with small sample sizes with the help of Transfer Learning. The paper tackled a very specific problem: what should we do with a small sample training size if we want to train a GAN. The authors have supported their arguments by a proof in Data In Prior and experiment results. They illustrated well in both aspects.  

Here are my point of views:
Transfer learning is a good way to help GANs when sample size is limited. but I have two concerns over this paper:
1. The datasets are very popular in the filed of GAN, however, for the Anime one, I am just curious how the VGG pretrained network can also help.  

2.  As for the experiment, it lacks of comparison with results that transfer learning is not applied. 

Generally, the paper is good and it can help data augmentation in other applications.
review_title: GAN with Transfer Learning

final_score: rating: 6: Marginally above acceptance threshold
review_content: ##########################################################################

Summary:

The paper focuses on  improving the performance of training generative adversarial networks (GANs) with limited target data. With the low diversity and quality when traing GANs with few data,  the paper proposes to use data instance prior to reduce the overfitting. Specially, taking the target sample as input, the data prior is extracted by a pre-trained network /  self-supervised model  , and then mapped into the embedding by  both G_emb and D_emb. The former acts as the class embedding, and the latter is the image embedding combined with the discriminator.     Authors also extend the proposed method into the large dataset, and provide the cluster method or a Gaussian Mixture Model.  The quantitative and qualitative result support the proposed method

##########################################################################

Pros:
+ The idea of utilizing the informative data prior to help train GANs is interesting.  Authors leverage the pre-trained model to extract data prior, and combine it with conditional GANs. Basically, SNGAN is considered in this peper, which uses the projection loss instead of cross-entropy loss to perform conditional image generation. Based on form of the projection loss, authors are able to avoid the problem the label of target data, and directly use the extracted embedding to conduct image generation.

+ For  large dataset, authors also provide a simple and effective method to get the semantic embedding.

+ For experiment, the paper uses current SOTA methods (BigGAN, SNGAN and StyleGAN2) and a series of datasets to evaluate the proposed method. Besides, all latest methods (to my best knowledge)  is compared to the proposed method, even the unpublished papers (DiffAugment),  which indicates that the proposed method is effective and convincing .

##########################################################################

 Cons:
- For me, the paper is so clear to understand, and miss a few information. 
  (1) Does the final objective contains E.q 2 ?  From the description above E.q 3 and the architecture , it seems to contain two losses.

  (2) Is E.q 2 used at inference time which is combined to the Inference section? I fail to connect the inference section to the whole paper.

  (3) If E.q 2 is utilized in this paper, what is different to BSA?  From my point, BSA assigns the noise and the real sample pair, and optimize  the input noise as well as the parameter of the batchnorm,  but fails to consider the adversarial loss,  which results in generating blur image. In this paper, authors  additionally consider the adversarial loss, and improve the reality of the synthesized image. 


- With conditional GANs is selected in this paper, I am wondering how to combine to the StyleGAN2, although the description is provided in Appendix. To be honest, I am not sure it works well when combining StyleGAN2 with project loss.

- What is the goal of both G_emb and D_emb? Do them just map the extracted embedding  to the same dimension required by conditional GANs?  authors mention that  it is   on-linearity  or   linear transformation matrices for different GANs frameworks, and  varying for  varying GANs architectures.  What is  linear transformation matrices?  what is the role to design    both G_emb and D_emb?

- In table 1, the result when combining FreezeD on Flower is low. Could authors explain it?

Minor comments: 
(1) Is  TransferGAN cited correctly in section of Baselines and Datasets  in page 6 ?  I think it is the one [1], which is first paper to perform transfer learning for GANs with limited data.

(2) What is the x and hat of x in E.q 3. 

(3) In abstract,  authors mention ' Previous works have addressed training inlow data setting by leveraging transfer learning and data augmentation techniques with limited success'. Is it  'with limited success' or 'with limited data'? 

[1] Transferring gans:  generating images from limited data.


review_title: Data Instance Prior for Transfer Learning in GANs

Meta-review: The paper proposes to use a feature extractor (encoder) $C(x)$, pre-trained with label supervision or contrastive learning on a large image dataset, to both regularize the discriminator's last feature layer $D_f(x)$ and encode the data $x$ itself as the conditional input of the generator $G(z|G_{emb}(C(x)))$. The main purpose is to help the training of GANs when there is a limited number of images in the target domain. A clear concern of this approach is that to generate a fake image, one will need to first sample a true image, making the model unattractive if the training dataset size is large (need to store the whole training dataset even after training). To mitigate this issue, the authors propose to fit up to 200k randomly sampled $G_{emb}(C(x))$ with a GMM with 1k components. To validate the practice of requiring a GMM (a shallow generative model) to help a GAN (a deep generative model) to generate, the authors have done a rich set of experiments under state-of-the-art GAN architectures or training methods (SNGAN, BigGAN, StyleGAN2, DiffAugment) to illustrate the efficacy of the proposed data instance prior and its compatibility with the state-of-the-art methods in a variety of settings. In the AC's opinion, the paper is missing references to 1) related work that combines VAE (or some other type of auto-encoder) and GAN, which often helps stabilize the GAN training [1,2,3], 2) VAE with a VampPrior [4], and 3) more broadly speaking, empirical Bayes related methods where the prior model is learned from the observed data (see [5] and the references therein). The potential advantages of using a VAE rather than a GMM to help a GAN to generate include: 1) there is no need to store 1k GMM components, which may require a large amount of memory; 2) there is no need to subsample the training set; and 3) the VAE and GAN can be jointly trained. The AC recommend the authors to discuss the connections to these related work in their future submission.

[1] Larsen, Anders Boesen Lindbo, et al. "Autoencoding beyond pixels using a learned similarity metric." International conference on machine learning. PMLR, 2016.

[2] Zhang, Hao, et al. "Variational Hetero-Encoder Randomized GANs for Joint Image-Text Modeling." International Conference on Learning Representations. 2019.

[3] Tran, Ngoc-Trung, Tuan-Anh Bui, and Ngai-Man Cheung. "Dist-gan: An improved gan using distance constraints." Proceedings of the European Conference on Computer Vision (ECCV). 2018.

[4] Tomczak, Jakub, and Max Welling. "VAE with a VampPrior." International Conference on Artificial Intelligence and Statistics. PMLR, 2018.

[5] Pang, Bo, Tian Han, Erik Nijkamp, Song-Chun Zhu, and Ying Nian Wu. "Learning Latent Space Energy-Based Prior Model." Advances in Neural Information Processing Systems 33 (2020).

Decision: Reject