Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims to achieve a better understanding of GAN fine-tuning. A synthetic experiment demonstrates that pretrained discriminators improve the quality of the initial gradients, and pretrained generators help with improving mode coverage. Transfer learning experiments on real image datasets reveal that pretraining primarily improves mode coverage rather than sample fidelity, and that datasets containing a diverse set of images are best suited for transfer.
main_review: **Strengths:**  
- Well written and easy to understand.  
- Transfer learning for GANs is an important subject. Better pretrained models could save much training time and enable users with less computational resources to achieve good results.  
- Simple synthetic experiment gives good insight into the roles that generator and discriminator play during transfer learning.  

**Weaknesses:**  
- One of the main conclusions of this work is that diverse source datasets (i.e., ImageNet) result in the best performance during transfer learning. However, most of the target datasets contain classes which overlap with ImageNet classes in some form, so it is not surprising that this model transfers well. It would be good to see how ImageNet transfers to datasets it does not overlap much, such as those from the medical imaging domain (one option is the BreCaHAD dataset from [1]). 
- An aspect that I think is missing from this study is the level of convergence of the source model. That is to say, is it better to take a checkpoint that it still in the middle of training, or one that has completely converged? What is the optimal time to take a checkpoint?  
- No confidence intervals on most experimental results. This is understandable due to the excessive compute budget required to train each of these models, but it still makes it difficult to understand the significance of the results.  

**Other Questions and Comments:**  
- In Table 2, row Grumpy Cat, column ImageNet, the P and R values seem very off compared to values from other models, and they do not match the values shown in Figure 10.  
-In Figure 9, why does ImageNet pretrained models often decrease very fast and then get worse? Is the model collapsing because the discriminator is quickly overfitting? Shouldn't the adaptive data augmentation prevent this kind of catastrophic collapse?  
- In Figure 10, FFHQ, why does the model trained from scratch diverge? I would have expected StyleGAN2 to perform very well on FFHQ since this is the dataset it was designed for.  
- Many lines are missing from plots in Figure 10 and 11. Why is this?
- If ImageNet checkpoint was not an option to use, would the proposed checkpoint selection method still have selected the best choice of the remaining options?

[1] A. Aksac, D. J. Demetrick, T. Ozyer, and R. Alhajj. BreCaHAD: A dataset for breast cancer histopathological annotation and diagnosis. BMC Research Notes, 12, 2019.  
summary_of_the_review: Paper provides useful insights about the role of the generator and discriminator in GAN transfer learning, as well as the behaviour of transfer learning itself (i.e., improved coverage but not fidelity).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the problem: when, why, and which pretrained GANs are useful. 

main_review: Pros:
(1). This kind of practical paper has significance for the entire community. This paper makes some conclusions, which could promote the development of adapting pre-trained GANs to different domains. 
(2). Part 2.1 “high-level intuition” is very interesting.
(3). The writing is clear and easy to understand.

Cons:
(1). It would be better to add a “related works” section to compare this paper with related papers; 
(2). In the introduction part, the author claimed that “some conclusions from Wang are not confirmed for modern architectures”. It would be more clear to summarize which conclusion is not confirmed in the related work part. 
(3). It would be better to list the distance between source datasets and target datasets in Table 1. 

summary_of_the_review: In general, this paper has conducted a series of experiments with the stylegan2, and got some conclusions, which are helpful for subsequent papers and the community. The analysis and experiments are not very comprehensive. In many ways, [1] is more comprehensive than this one, although [1] is from a few years ago.

[1]. Transferring GANs: generating images from limited data
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper performs a large-scale study of transfer learning in GANs. It proposes a way to understand the relevance of a pre-trained generator and discriminator, as well as heuristics to select good source/initialization dataset and even a training snapshot. All of this is very valuable for the practitioners.
main_review: While some aspects of the work have been hinted at in prior art, to the best of my knowledge no one has properly studied the questions so eloquently put forward in the title. These are important questions and I for one have wanted to know the answers for some time! The paper does a good job building intuition through toy examples, large-scale transfer test (Table 2), and some quite novel tests (Figs 6 & 7). Overall this feels like important work that should to be accepted. I will champion this paper.

A possible error:
- p.3 second to last para: "... have significant negative correlations with the finetuned GAN quality". I was very surprised to read this and in fact had to backtrack twice. I suspect this statement is strictly false. Unless I'm mistaken, the negative correlation is with Wasserstein distance, but small distance implies high quality, and thus there is a strong POSITIVE correlation with quality. Right?

Some minor complaints/suggestions/questions:
- [citation] is not a noun. It's silent when reading the sentence.
- Personally I found the contribution list at the end of Sec 1 completely unnecessary. We have just read all the same items... (twice if you count the abstract)
- Thumb up/down in Figure 1 may not be the best way to convey the message.
- p5: ImageNet pretraining is always useful: This is an important finding, but LSUN Dog and FFHQ were also uniformly beneficial. ImageNet was just a bit better. Please consider rewording. There is no magic in ImageNet, and it's very likely that some higher quality diverse dataset would almost certainly be even better.
- Table 2: "Random" on the top row was initially mysterious. You mean it's random initialized, while the text up to this point has been talking about training from scratch. Consider a clarification, maybe in the caption.
- Fig 7: What exactly does the class change probability mean? Probability during which timespan? Between init and current, or perhaps current and some previous timestep. (the answer isn't too important, just make sure to be specific). 
- Q: The method in Sec 4 is used to choose a particular snapshot. This is nice. Now, if I understand correctly, you're using it for selecting the best snapshot per source dataset -- did you consider using the same heuristic for also selecting the best source dataset/initialization dataset? If it doesn't work, why?



summary_of_the_review: This paper answers important questions in GAN transfer learning, and therefore makes a solid contribution.
review_title: null

Meta-review: This paper empirically studies when, why, and which pretrained GANs are useful.  
All the reviewers are positive about this work, that they all consider very valuable for practitioners and the community.  
First building intuition through toy examples, authors conduct a large-scale study of transfer learning in GANs (with the stylegan2). They propose a way to understand the relevance of a pre-trained generator and discriminator, as well as heuristics to select good initialization.  
Overall, this paper makes a solid contribution that should to be accepted.
Decision: Accept (Poster)