When, Why, and Which Pretrained GANs Are Useful?
Authors: Timofey Grigoryev
Andrey Voynov
Artem Babenko
Venue: ICLR 2022 Poster (2022)
Keywords: GAN
pretraining

Abstract: The literature has proposed several methods to finetune pretrained GANs on new datasets, which typically results in higher performance compared to training from scratch, especially in the limited-data regime. However, despite the apparent empirical benefits of GAN pretraining, its inner mechanisms were not analyzed in-depth, and understanding of its role is not entirely clear. Moreover, the essential practical details, e.g., selecting a proper pretrained GAN checkpoint, currently do not have rigorous grounding and are typically determined by trial and error. 

This work aims to dissect the process of GAN finetuning. First, we show that initializing the GAN training process by a pretrained checkpoint primarily affects the model's coverage rather than the fidelity of individual samples. Second, we explicitly describe how pretrained generators and discriminators contribute to the finetuning process and explain the previous evidence on the importance of pretraining both of them. Finally, as an immediate practical benefit of our analysis, we describe a simple recipe to choose an appropriate GAN checkpoint that is the most suitable for finetuning to a particular target task. Importantly, for most of the target tasks, Imagenet-pretrained GAN, despite having poor visual quality, appears to be an excellent starting point for finetuning, resembling the typical pretraining scenario of discriminative computer vision models.

Published as a conference paper at ICLR 2022
WHEN, WHY, AND WHICH PRETRAINED GANS ARE USEFUL?
Timofey Grigoryev
Andrey Voynov Yandex an woy.oovyyyyyad........
Yandex
grigorev. seggorew.tt@phystech.eeee
Artem Babenko
Yandex artem. www..nnkkoohhhtecchhccoo edu
ABSTRACT
The literature has proposed several methods to finetune pretrained GANs on new datasets, which typically results in higher performance compared to training from scratch, especially in the limited-date regime. However, despite the apparent em- pirical benefits of GAN pretraining, its inner mechanisms were not analyzed in- depth, and understanding of its role is not entirely clear. Moreover, the essential practical details, e.g., selecting a proper pretrained GAN checkpoint, currently do not have rigorous grounding and are typically determined by trial and error.
This work aims to dissect the process of GAN finetuning. First, we show that initializing the GAN training process by pretrained checkpoint primarily affects the model's coverage rather than the fidelity of individual samples. Second, we explicitly describe how pretrained generators and discriminators contribute to the finetuning process and explain the previous evidence on the importance of pre- training both of them. Finally, as an immediate practical benefit of our analysis, we describe a simple recipe to choose an appropriate GAN checkpoint that is the most suitable for finetuning a particular target task. Importantly, for most of the target tasks, Imagenet- Imagenet-pretraimed GAN, despite having poor visual quality, appears to be excellent starting point for finetuning, resembling the typical pretraining scenario of discriminative computer vision models.
INTRODUCTION
These days, generative adversarial networks (GANs) (Goodfellow al. 2014) successfully ap-
proximate the high-dimensional distributions of real images. The exceptional quality of the state-of- the-art GANs (Karras et al. 2020b Brock al. 2019) makes them a key ingredient in applications,
including semantic editing (Isola et al. [20 Z hu et al.
2018
Shen et al. 2020 Voynov & Babenko,
2020), image processing (Pan al. 2020 Ledig et al.,
2017 Menon et al. 2020). video generation
Wang et al. 2018a), producing high-quality synthetics (Zhang al.) 202 Voynov et al. 2020).
To extend the success of GANs the limited-data regime, common to use pretraining. i.e., to initialize the optimization process by the GAN checkpoint pretrained on some large dataset. A line
of works (Wang et al. 20188 Noguchi Harada 2019 Zhao et al. 2020 Mo et al. 22200: Wang iet al., 2020 Li et al. 2020 investigate different methods transfer GANs
datasets and report significant advantages compared to training from scratch both in terms of generative quality and convergence speed. However, the empirical success of GAN pretraining in-depth, and its reasons are not entirely understood. From the practical standpoint, it is unclear how choose a proper pretrained checkpoint or if one should initialize both generator and discriminator or only one of them. To the best of our knowledge, the only work that systematically studies the benefits of pretraining /Wang al. (2018b). However, the experiments in (Wang et al. 2018b) were performed with currently outdated models, and we observed that some conclusions from Wang
not investigated
"Iddccttss equal contribution

---

Published as a conference paper at ICLR 2022
et al. (2018b) not confirmed for modern architectures like StyleGAN2 KKrrras et al. 2020b). In
particular, unlike the prior results, it appears that for state-of-the-an art GANs, it is beneficial to transfer from sparse and diverse sources rather than dense and less diverse ones.
In this work, we thoroughly investigate the process of GAN finetuning. First, we demonstrate that starting the GAN training from the pretrained checkpoint can significantly influence the diversity of the finetuned model, while the fidelity of individual samples is less affected. Second, we dissect the mechanisms of how pretrained generators and discriminators contribute to the higher cover- age of finetuned GANs. In a nutshell, we show that a proper pretrained generator produces sam- ples in the neighborhood of many modes of the target distribution, while proper pretrained dis- criminator serves as gradient field that guides the samples to the closest mode, which together result in smaller risk of mode missing. This result explains the evidence from the literature that it is beneficial to initialize both generator and discriminator when finetuning GANs. Finally, we investigate different ways choose a suitable pretrained GAN checkpoint for a given target dataset. Interestingly, for most of the tasks, imagenett-reeraaneet models appear to be the optimal initializers, which mirrors the pretraining of discriminative models, where Imagenet-basee initial- ization is de-facto standard (Donahue et al., 2014; Long al., 2013 He et al. 2020 Chen et al., 2020a). Our conclusions are confirmed by experiments with the state-of -hhee-rrt StyleGAN2 (Karras) al. 2020b), chosen due its practical importance and variety of open-sourced checkpoints, which can be used as pretrained sources. The code and pretrained models are available online at /tts///itthus con/ commyannexx rwweereee wawarccggaa-traasee ht https::
The main contributions of our analysis are the following:
We show that initializing the GAN training by the pretrained checkpoint can significantly affect the coverage and has much less influence the realism of individual samples.
2. We explain why it is important to initialize both generator and discriminator by describing their roles in the finetuning process.
3. We describe a simple automatic approach to choose a pretrained checkpoint that is the most suitable for a given generation task.
2 ANALYSIS
This section aims to explain the success of the GAN finetuning process compared to training from scratch. First, we formulate the understanding of this process speculatively and then confirm this understanding by experiments on synthetic data and real images.
2.1 HIGH--LEVEL INTUITION
Let us consider pretrained generator G and discriminator D that are used to initialize the GAN training on a new data from a distribution Pargett Throughout the paper, we show that a discriminator initialization is "responsible for" an initial gradient field, and generator initialization is "responsi- ble for" target data modes coverage. patterns. Intuitively, the proper discriminator initialization guarantees that generated samples will move towards "correct" data regions. On the other hand, the proper pretrained generator guarantees that the samples will be sufficiently diverse at the initialization, and once guided by this vector field, they will cover all target distribution. Below, we confirm the validity of this intuition.
the overall idea with different initialization
2.2 SYNTHETIC EXPERIMENT
We start by considering the simplest synthetic data presented on Figure Our goal is to train a GAN on the target distribution, a mixture of ten Gaussians arranged in circle. We explore three options to initialize the GAN training process. First, we start from random initialization. The second and the third options initialize training by GANs pretrained on the two different source distributions. The first source distribution corresponds to wide ring around the target points, having high coverage and low precision w.r.t. target data. The second source distribution is formed by three Gaussians that share their centers with three target ones but have slightly higher variance. This source distribution has high precision and relatively low coverage w.r.. target data. Then we train two source GANs from scratch to fit the first and the second source distributions and employ these checkpoints to initialize GAN training on the target data. The results of GAN training for the three options are presented Figure 2 which shows the advantage of pretraining from the more diverse model,
2

---

Published as a conference paper at ICLR 2022
Generator
Generator
Generator
Discriminator
Discriminator
Discriminator
G(z) VD
Ptarget
Ptarget
Ptarget
Figure 1: Different G/D initialization patterns: the red dots denote pretrained generator samples, the arrows denote a pretrained discriminator gradient field, the blue distribution is the target. From left to right: bad discriminators will lead good initial samples out of the target distribution; bad generators will drop some of the modes even being guided by good discriminators; both proper G/D D serve as an optimal initialization for transfer a new task.
Sourced imit
Sounce- init
Random
Scrrowd
Source-ll
Giretatal
Genertted
3.9
16.5
12.8
Figure 2: Impact of GAN pretraining for synthetic data. 1) source and target distributions. 2- 3) GANs pretrained on two source distributions. 4-6): GANs trained on the target distribution, initialized by the two source checkpoints and randomly. Each plot also reports the Wasserstein-1 distance between the generated and the target distributions.
which results in a higher number of covered modes. The details the generation of the synthetic provided in the appendix.
Dissecting the contributions from G and D. Here, we continue with the synthetic example from above and take closer look the roles that the pretrained generator and discriminator play when finetuning GANs. Our goal is to highlight the importance of (1) the initial coverage of the target distribution by the pretrained generator and (2) the quality of the gradient field from the pretrained discriminator. We quantify the former by the established recall measure (Kynkäänniemi et al. 2019) computed in the two-dimensional dataspace with k=5 for 1000 randomly picked samples from the target distribution and the same number of samples produced by the pretrained generator. To evaluate the quality of the discriminator gradient field, we use a protocol described in (Sinha et al.) 2020)) Namely, assume that the "golden" ground truth gradients would guide each sample towards closest Gaussian center from the target distribution. Then we compute the similarity between the vector field D.D provided by the pretrained discriminator and the vector field of "golden" gradients. Specifically, we evaluate the cosine similarity between these vector fields, computed for the generated samples.
Given these two measures, we consider a series of different starting generator'diseriminaaoo check- points (G,, i i 1, N. The details the choice of the starting checkpoints are provided in the appendix. Then we use each pair (G;, D:) as initialization of GAN training on the target distribution of ten Gaussians described above. Additionally, for all starting G./De recall and the discriminator gradients field similarity to the "golden" gradients. The overall quality of GAN finetuning measured as the Wasserstein-1 distance between the target distribution and the distribution produced by the finetuned generator. The scatter plots of recall, the similarity of gradient fields, and Wasserstein-1 distance are provided in Figure 3 As can be seen, both the re- call and gradient similarity have significant negative correlations with the Wi-distance between the ground-truth distribution and the distribution of the finetuned GAN. Furthermore, for the same level of recall, the higher values of the gradient similarity correspond to lower Wasserstein distances. Al- ternatively, for the same value of gradient similarity, higher recall of the source generator typically corresponds to the lower Wasserstein distance. We also note that the role of the pretrained generator more important since, for high recall values, the influence from the discriminator is not significant (see Figure left).
evaluate the
3

---

Published as a conference paper at ICLR 2022
This synthetic experiment does not rigorously prove the existence of a causal relationship between the recall or gradient similarity and the quality of the finetuned GANs since demonstrates only correlations of them. However, in the experimental section, we show that these correlations can be successfully exploited to choose the optimal pretraining checkpoint, even for the state-of-the-art GAN architectures.
W ---llstangee
Wisdiaance
Recall
-22
-08
-66
-44
-22
50
-10
-05
-06
-44
-02
10
00
02
04
2.5
0.5
10
V similarity
VD similarity
Recall
Figure 3: Scatter plots of the pretrained generator quality (Recall) and the pretrained discriminator quality (VD similarity) vs the quality of finetuned GAN (Widdistance). Each point represents result of GAN finetuning, which started from particular pair of pretrained discriminator and generator. The color indicates the Wi-distance between the final generator distribution and the target distribution. The Pearson correlation of the final Wi-distance equal -0.84 for the Recall, and -0.73 for the gradient similarity.
3 LARGE-SCALE EXPERIMENTS
3.1 EXPLORING PRETRAINING FOR STYLEGAN2
In this section, we confirm the conclusions from the previous sections experimentally with the state- of-the-art StyleGAN2 architecture (Karras et al. 2020b). If not stated otherwise, we always work with the image resolution x 256.
Datasets. We work with six standard datasets established in the GAN literature. We also include two datasets satellite images to investigate the pretraining behavior beyond the domain of natural images. As potential pretrained sources, we use the StyleGAN2 models trained on these datasets. Table the list of datasets and the FID values (Heusel et al., 2017) the source checkpoints. We also experimented with four smaller datasets verify conclusions in the medium-shot and few-shot regimes. The details on the datasets are provided in the appendix.
Experimental setup. Here, describe the details of experimental protocol for both the pre- training of the source checkpoints and the subsequent training on the target datasets. We always the official PyTorch implementation of StyleGAN2-ADA (Karras et al.) 2020a) provided by the a We use the "stylegan2" configuration in the ADA implementation with the default hy- perparameters (same for all datasets). Training is performed on eight Tesla V100 GPUs and takes approximately three hours per 1M real images shown to the discriminator.
Pretraining of source checkpoints. We pretrain one checkpoint on the Imagenet for 50M real images shown to the discriminator and seven checkpoints on other source datasets from Table for 25M images. larger number of optimization steps for the Imagenet used since this dataset is more challenging and requires more training epochs to converge. For the large LSUN datasets (Cat, Dog, Bedroom), we use 10 first images to preserve memory. For use ADA due to its smaller size. Then, always use checkpoints with the best FID for further transferring to target datasets for each source dataset.
we
Training on target datasets. For each source checkpoint, perform transfer learning all datasets from Table [ We use the default transfer lcarning settings from the StyleGAN2-ADA implementation (faster adaptive data augmentation (ADA) adjustment rate, if applicable, and no Gema warmup). ADA is disabled for the datasets containing more than 50K images and enabled for others with default hyperparameters. In these experiments, train for 25M real images shown
https:://gt http://cco//wwaaa////yeeeeeeee// aaatyoyoo pytorch

---

Published as a conference paper at ICLR 2022
to the discriminator. Each transfer experiment is performed with three independent runs, and the metrics are reported for the run corresponding to the median best FID (Heusel et al. (017).
Metrics. In the experiments, evaluate the performance via the four following metrics, (1) Frechet Inception Distance (FID) (Heusel et al. 2017), which quantifies the discrepancy between the distributions of real and fake im- ages, represented by deep embeddings. Both distributions are approximated by Gaussians, and the Wasserstein distance between them is computed. (2) Precision (Kynkäänniemi et al. 2019), which measures the realism of fake im- ages, assuming that the visual quality of par- ticular fake is high if it belongs to the neigh- borhood of some real images in the embedding space. (3) Recall (Kynkäänniemi et al., 2019), which quantifies GAN diversity, measuring the rate of real images that belong to the neighbor- hood of some fake images in the embedding space. (4) Convergence rate equals a number of real images that were shown to the discrim- inator at the moment when the generator FID for the first time exceeded the optimal FID by at most 5%. Intuitively, this metric quantifies how fast the learning process reaches plateau. FID computed based on the image embeddings extracted by the Inception modef") Precision and Recall use the embeddings provided by the VGG-16 mode Precision and Recall are al- ways computed with k= neighbors. For FID calculation, we always use all real images and 50K generated samples. For Precision/Recall calculation, we use the first 200K real images (or less, if the real dataset is smaller) and 50K generated samples.
Dataset
Number of images
FID
Datasets for pretraining
Imagenet
1 281 1 137
49.8
LSUN-Cat
1 1000 000
7.8
LSUN-Dog
000000
15.0
LSUN-Beerrod
0000000
3.3
LSUN-Church
126
3.2
Laalliip-Ladsesspsss
2 608
26.6
Satellite- Bullldi-B
280 741
12.4
FFHQ
70000
5.5
Additional target datasets
CIFAR-10
50 000
Grumpy Cat
100
Flowers
189
Simpsons
41 866
BreCaHAD
3 253
Table The datasets used in our experiments. All images are resized to 256 x 256 resolution. The last column reports the FID values of the source checkpoints trained with the random initialization.
Results. The metric values for all datasets are reported in Table where each cell corresponds to a particular source-target pair. For the best (in terms of FID) checkpoint obtained for each source- target transfer, we report the FID value (top row each cell), Precision and Recall (the second and the third rows in each cell), and the convergence rate measured in millions of images (bottom row in each cell). We highlight the sources that provide the best FID for each target dataset or differ from the best one by at most 5%. We additionally present the curves of FID, Precision, and Recall values for several target datasets on Figure and Figure 10 in the appendix.
We describe the key observations from Table below:
In terms of FIID, a pretraining based on diverse source (e.g., Imagenet or L.SUN Dog) superior to training from scratch on all datasets in our experiments.
The choice of the source checkpoint significantly influences the coverage of the finetuned model, and the Recall values vary considerably for different sources, especially for smaller target datasets. For instance, on the Flowers dataset, their variability exceeds ten percent. In contrast, the Precision values are less affected by pretraining, and their typical variability is about 2-3%. Figure Preports the standard deviations of Precision/Recal computed over different sources and highlights that Recall has higher variability compared to Precision, despite the latter having higher absolute values.
Pretraining considerably speeds up the optimization compared to the training from scratch.
httee:///oiaaa-----ccccmmmmiaaaa http::///hhbiiiiiaadddcecaaaaaar met pr
http:://wwllaaaa...oooo..oo..aas netricssvggl6.pp
5

---

Published as a conference paper at ICLR 2022
L.Bedroom
L.Church
S.Buildings
Landscapes
Imagenet
From scratch
FFHQ
L.CaL
L.Dog
5.7
5.52
6.08
5.18
8.49
6.57
4.87
5.52
FFHQ
0.782
0.793
0.767
0.793
0.798
0.797
0.784
0.795
R
0.417 5
0.419
0.455
0.459
0.318
0.355
0.457
0.472
8
7
11
14
16
12
13
L.Bedroom
2.71
3.01
2.8
2.63
3.77
4.3
2.57
3.3 +0.2
0.651
O. 0.663
0.657
0.667
0.643
0.679
0.668
0.469
0.447
0,485
0.471
0.314
0.344
0.475
0.459
24
25
25
25
21
24
25
23
7.62
7.68
7.72
6.65
9.09
9.95
7.12
7.85
L.Cat
0.68
0.68
0.669
0.687
0.703
0.67
0.688
0.684
R
0.402
0,381
0,402
0. 0,409
0,273
0.303
0.39
0.368
22
25
25
18
25
25
21
20
L.Church
3.09
3.11
3.28
2.97
3,99
6.79
3.0
3.16
0.689
0.7
0.69
0.677
0.692
0.63
0.699
0.682
R
0.54
0.497
0.496
0.543
0.414
0.322
0.528
0.554
23
22
23
21
22
5
23
25
14.8
15.1
13,9
15.6
18.4
18.4
14.4
15.02
L.Dog
0.74
0.747
0.757
0.738
0.753
0.754
0.743
0.758
R
0.363
0.334
0.359
0,36
0.237
0,256
0,365
0,349
C
22
22
24
25
25
25
24
24
S.Buildings
11.1
11.5
11.77
11.2
12.1
16.7
10.72
12.36
0.347
0.337
0.316
0.34
0.333
0.281
0.319
0.348
R
0.549
0.53
0.52
0.555
0.526
0.413
0.574
0.507
C
24
20
21
16
25
25
25
19
S.Landscaper
25.3
26.1
24.3
25.3
23.8
28.2
21.0
26.6
P
0.756
0.762
0,762
0.73
0.759
0.769
0,719
0.737
0.249
0.191
0.2
0.291 5
0.282
0.136
0.393 2
0.214
23
21
18
14
25
CIFAR-10
8.6 +0.5
8,29
7.6
8.62
7.11
10.4
9.22
6.2 20.5
9.33
0.79
0.764
0.75
0,759
0.769
0.783
0.759
0.761
0.781
0.493
0.45
0.479
0.502
0.525
0.397
0.401
0.559
0.455
15
19
20
9.47
9.79
9.4
9,88
8,88
11.8
9.07
8.31
10.73
Flowers
0.786
0.776
0.795
0.773
0.773
0.772
0.809
0.773
0.78
0.251
0.215
0.194
0.226
0.269
0.14
0.153
0.282
0.271
16
15
25
Grumpy Cat
11.9
12.3
14.0
16.1
12.6
28.5
16.3
14.7
15.34
0.999
0.996
0.997
0.995
0.998
0.868
0. 0.999
0,999
0.997
0.06
0.02
0.03
0.0217
0.04
0.01
0.045
0.05
0.0175
24
25
24
25
25
25
25
25
Simpsons
7.87
8.0 +0.4
8.12
7.93
7.67
10.0
9.98
8.28
8.42
0.432
0,423
0.431
0.436
0.416
0.404
0.418
0,427
0.42
0.406
0.349
0.353
0.384
0.395
0.219
0.173
0.364
0.335
25
24
21
25
25
25
BreCaHAD
26.31
23.12
24.80
25.40
25.36
23.81
21.84
22.73
23.72
0.694
0.679
0.696
0.709
0.692
0.730
0.712
0.703
0.705
0.385
0.417
0.462
0.412
0.439
0.337
0.473
0.483
0.434
Table Metrics computed for the best-FID checkpoint for different source and target datasets. Each row corresponds a particular target dataset, and each column corresponds to a particular source model used to initialize the training. For each target dataset, we highlight (by orange) the sources that provide the smallest FID which FID differs from the best one by most 5%. each cell, report from to bottom: FID, Precision, Recall, and convergence rate measured in millions of images (lower better). In purpose make the table casier to read, we report std only once exceeds 5% which happens rarely. The typical values vary around 0.1.

---

Published as a conference paper at ICLR 2022
0.06
0.04
0.02
0.00
LBedroon
Lar
LChurch
Logg
Builings
St Staddsaper
CIFAR-10 10
Flovers
Grapp Cat
Simpoons
BeCaHAD
Figure 4: Standard deviations of Reciision/Recaal values for each target dataset computed over dif- ferent sources. Due to the symmetric nature of the quantities, the table reports the standard deviation for precision and recall computed over an equal number of real and generated samples (minimum between dataset size and 50K)
Overall, despite having poor quality (FID=49.8), the Imagenet- pretrained unconditional StyleGAN2 model appears to be superior GAN initialization that typically leads to more efficient optimization compared to alternatives. This result contradicts that it is beneficial transfer from dense and less diverse sources rather than sparse and diverse ones, like Imagenet. We attribute this inconsistency to the fact that (Wang et al. 2018b) experimented with the WGAN-GP models, which are significantly inferior to the current state-of-the-an ones.
observations (Wang al. 2018b) showing
3.2 ANALYSIS
In this section, we perform several additional experiments that illustrate the benefits of pretraining.
Pretraining improves the mode coverage for real data. Here, we consider the Flowers dataset and assume that each of its 102 labeled classes corresponds to different distribution modes. To assign the generator samples to the closest mode, we train a 102-way flowers classifier via finetuning the linear head of the Imagenet-pres preetriind ResNet-50 real labeled images from Flowers. Then apply this classifier to generated images from eleven consecutive generator snapshots from the GAN training process the interval from 0 to 200 kimgs taken every 20 kimgs. This pipeline allows for tracking the number covered and missed modes during the training process. Figure 5 how the number of covered modes changes when GAN is trained from scratch the checkpoints pretrained on FFHQ and Imagenet. In this experiment, we consider mode being "covered" if it contains at least ten samples from the generated dataset of size 10 100. One can see that Imagenet, being the most diverse source, initially covers more modes of the target data and faster discovers the others. FFHQ also provides coverage improvement but misses more modes compared to Imagenet even after training for 200 kimgs. With random initialization, the training process covers only third of modes after training for 200 kimgs. On the right Figure we show samples drawn the mode, which is poorly covered by the GAN trained from FFHQ initialization and well-covered by its Imagenet counterpart.
(left) demonstrates
100
Covered Modes Count
Imagenet-init
FFHQ-init
Random-inil
1K images
Figure 5: Left: Number modes covered by the generator snapshots during the training process from three different initializations. Right: samples of the 65-th class of the Flowers dataset, which is well-covered by the GAN trained from the Imagenet initialization and poorly covered by the GAN trained from the FFHQ initialization. Top: real images; Middle: FFHQ; Bottom: Imagenet.

---

Published as conference paper at ICLR 2022
0.8
Imageres aid
Rardomtinnll
A1PPPS
0.4
0.2
1M images
Figure 6: Evolution of the generated samples with different source initializations. Left: average disstddis between images generated by the consecutive generator snapshots for the same la- tent code. Right: images generated with the same latent code evolving during training: top start from FFHQ, middle row: start from Imagenet, bottom row: start from random initialization.
0.30
Imagenet-init
0.7
FRHCQiii
Class change probabiiity
0.5-
Rondom-énit
0120
0.5
0.15
0.4
0.11
Imagenet-init
0.3
FFHQ-init
0.05
Random-init
0.2
0.00
L.PIIS-trajectory length
images
Figure 7: Left: distribution of samples trajectories lengths with Flowers as target dataset. Right: generated class change probability for individual latents during the training.
Pretraining provides more gradual image evolution. The observations above imply that it ben- eficial to initialize training by the checkpoint with higher recall so that the target data is originally better "covered" by the source model. We conjecture that transferring from model with higher recall makes it easier cover separate modes in the target distribution since, in this case, generated samples can slowly drift to the closest samples of the target domain without abrupt changes to cover previously missing modes. To validate this intuition, we consider fixed batch of 64 random la- tent codes & and a sequence of the generator states G G obtained during the training. Then quantify the difference between consecutive images computed the perceptual LPIPS distance Zhang et al. (2018) (PIPS(G,(:). (z)). Figure 6 shows the dynamics of the distances for Flowers the target dataset and Imagenet, FFHQ, and random initializations. Since the Imagenet source initially has higher coverage of the target data, its samples need to transform less, which results in higher performance and faster convergence.
Figure 6 indicates more gradual sample evolution when GAN training starts from a pretraining checkpoint. Here we additionally report the distributions of samples' trajectories' lengths quantified by LPIPS. Namely, for fixed z and sequence of the generator snapshots G GT...., GN obtained during training, we calculate the length of the trajectory a sum Σ, LPIPS(G) G2)) Figure 7 (left) presents the length distributions for three initializations and Flowers the target dataset.
Finally, to track the dynamics of mode coverage of the target dataset, we obtain the class assignments of the generated samples Gz(2).... (z(2) with classifier pretrained on the Flowers dataset. Then for the samples (z(z), (z) generated with the consequent checkpoints, we calculate the probability that the sample changes its class assignment by averaging over 256 latent codes. That is, we evaluate the probability that flower class of a sample Gz(z) differs from class of sample Gz)1(z). The probabilities of the class change for different source checkpoints are presented in Figure right. Importantly, training from pretrained sources demonstrates higher class persistence
8

---

Published as a conference paper at ICLR 2022
GAN Model (source, target)
Inverted Domain
Best FID
LPIPS-eeror
F-error
Random, t: FFHQ
CelcbA-HQ
5.35
0.22
0.186
8:
Imagenet, t: FFHQ
CelebA-HQ
4.86
0.22
0.174
Random, FFHQ
FFHQ
5.35
0.25
0.180
8: Imagenet, t: FFHQ
FFHQ
4.86
0.25
0.168
Random, t: L.Beedrom
L.Bedroom
2.97
0.47
0. 0.123
Imagenet, t: L.Bedroom
L.Bedroom
2.56
0.44
0.115
Table 3: Reconstruction errors for GAN models with different source and target datasets.
of individual samples. This indicates that the imagenet-pretraing generator initially covers the target dataset well enough and requires fewer mode-changing sample hops during training.
Pretraining beneficial for downstream tasks. Here, we focus on the task of inverting a real image given pretrained generator, which is necessary for semantic editing. We employ the recent GAN inversion approach (Tov et al. 2021) and train the encoders that map real images into the latent space of generators approximating FFHQ and Bedroom distributions. For both FFHQ and Bedroom, we consider the best generators that were trained from scratch and the Imagenet-based pretraining. Table reports the reconstruction errors quantified by the LPIPS measure and / F-error. The details of the metrics computation are provided in the appendix. Overall, Table 3 confirms that higher GAN recall provided by pretraining allows for the more accurate inversion of real images.
CHOOSING PROPER PRETRAINED CHECKPOINT
This section describes simple recipe to select the most appropriate pretrained checkpoint to initial- ize GAN training for particular target dataset. To this end, we consider a set of natural proxy met- rics that quantify the similarity between two distributions. Each metric is computed in two regimes. In the first regime, measure the distance between the source dataset, consisting of real images used to pretrain the GAN checkpoint, and the target dataset of real images. the generated images from pretrained checkpoints instead of the source dataset. The second regime is more practical since does not require the source dataset. As natural proxy metrics, we consider FID, KID (Binkowski et al. 2018), Precision, and Recall measures.
the second regime,
To estimate the reliability of each metric, we calculate the number of target datasets for which this metric does not correctly predict the optimal starting checkpoint. We consider a starting checkpoint optimal if it provides the lowest FID score or its FID score differs from the lowest by most 5%. The quality for all met- rics presented in Table 4 which shows that FID or Recall can be used as rough guide to select a pretrained source in both regimes. On the other hand, Precision is entirely unreliable. This observation is consistent with our findings from Section 2 that imply that Recall can serve as a predictive measure of finetuning quality.
Regime/Metrie
FID
KID
Precision
Recall
Real Source
3
Generated Source
3
Table The number of target datasets for which the metrics fail to identify the best source (with up to 5% best FID deviation).
CONCLUSION
Transferring pretrained models to new datasets and tasks is workhorse of modern ML. In this paper, we investigate its success in the context of GAN finetuning. First, we demonstrate that transfer from pretrained checkpoints can improve the model coverage, which crucial for GANs exhibiting mode-seeking behavior. Second, we explain that it is beneficial to use both pretrained generators and discriminators for optimal finetuning performance. This implies that the GAN studies should open-source discriminator checkpoints as well rather than the generators only. Finally, we show that the recall measure can guide the choice of a checkpoint for transfer and highlight the advantages of Imagenet-basse based pretraining, which is not currently common in the GAN community. We open- source the StyleGAN2 checkpoints pretrained on the Imagenet of different resolutions for reuse in future research.

---

Published as a conference paper at ICLR 2022
REFERENCES
Mikolaj Binkkwski,, Danica J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying
MMD GANs. In International Conference on Learning Representations, 2018. URL https:
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In International Conference Learning Representations, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0429) 2020b.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In Inter- national conference on machine learning, pp. 647-655. PMLR, 2014.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, 2014.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637. 2017.
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexci A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference computer vision and pattern recognition, 2017.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. NeurlPS, 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8110-8119, 2020b.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In Advances in Neural Information Processing Systems, pp. 1929-39333 2019.
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. -eatoste sin- gle image super-resolution using a generative adversarial network. Proceedings of the IEEE conference on computer vision and pallern recognition, 2017.
Yijun Li, Richard Zhang, Jingwan Lu, and Eli Shechtman. Few-shot image generation with elastic weight consolidation. arXiv preprint arXiv:2012.027800, 2020.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431-3440, 2015.
Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. Pulse: Self- supervised photo upsampling via latent space exploration of generative models. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 2437-2445, 2020.
10

---

Published as a conference paper at ICLR 2022
Sangwoo Mo, Minsu Cho, and Jinwoo Shin. Freeze discriminator: A simple baseline for fine-tuning gans. arXiv preprint arXiv:2002. 10964 2020.
Atsuhiro Noguchi and Tatsuya Harada. Image generation from small datasets via batch statistics adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2750-2758, 2019.
Xingang Pan, Xiaohang Zhan, Bo Dai, Dahua Lin, Chen Change Loy, and Ping Luo. Exploiting deep generative prior for versatile image restoration and manipulation. In European Conference on Computer Vision, pp. 262-277. Springer, 2020.
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
Samarth Sinha, Zhengli Zhao, Anirudh Goyal ALIAS PARTH GOYAL, Colin Raffel, and Au- gustus Odena. Top-k training of gans: Improving gan performance by throwing away bad samples. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 14638 14649. Curran As- sociates, Inc., 2020. URL httppsss/pprooceddgggsss.. neurips www.eearriis.ccppppe///2200 wctpppppr//2////11.....e a851bd0d44
331 0bd Paper pdf
Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing encoder for stylegan image manipulation. arXiv preprint arXiv:2102.02766. 2021.
Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions in the gan latent space. In International Conference on Machine Learning, pp. 9786-9796. PMLR, 2020.
Andrey Voynov, Stanislav Morozov, and Artem Babenko. Big gans are watching you: To- wards unsupervised object segmentation with off-the-shelf generative models. arXiv preprint arXiv:2006.04988 049888 2020.
Ting-Chun Chun Wang, Ming- Yu Liu, Jun- Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catan- zaro. Video-to-video synthesis. In Advances in Neural Information Processing Systems, 2018a.
Yaxing Wang, Chenshen Wu, Luis Herranz, Joost van de Weijer, Abel Gonzalez-Garcia, and Bog- dan Raducanu. Transferring gans: generating images from limited data. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 218-234, 2018b.
Yaxing Wang, Abel Garzzaa David Berga, Luis Herranz, Fahad Shahbaz Khan, and Joost van de Weijer. Minegan: effective knowledge transfer from gans to target domains with few im- ages. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9332-9341, 2020.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. arXiv preprint arXiv:2104.064900 2021.
Miaoyun Zhao, Yulai Cong, and Lawrence Carin. On leveraging pretrained gans for limited-data generation. ICML, 2020.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, 2018
11

---

Published as a conference paper at ICLR 2022
APPENDIX
DATASETS
Here we provide the details for the used datasets. Table reports reports the size, original image resolution (which was always resized to 256 x 256 in experiments), number of samples used for training, we report pairwise distances between source and target datasets for different metrics. Figure illustrates samples from each dataset. As for
and URL for each of the datasets. In Tables 6 9
BreCaHAD, we generate dataset of 256 x 256 crops of the original dataset images with the code provided in StyleGAN-ADA repository.
Dataset
Size
Original Resolution
Samples Used
CIFAR-10
50.000
x
50.000
FFH
70.000
1024 x 1024
70.00
Flowers
189
varies
8 189
Grumpy-Ca
100
256 x 256
100
Imagenet
1281 137
varies
281 137
LSUN Bedroom"
3 0.3 042
256 x 256
000 000
LSUN Ca
657 266
256 x 256
0000 000
LSUN Church
126
256 x 256
127
LSUN Dog
5 817
256 x 256
1000 000
Satellite- Satellite-uildingg
280741
300 x 300
280 741
2688
1800 x 1200
2608
Simpsons
41 16666
varies
41 866
BreCaHAD
3253
256 x 256
3253
Table 5: Datasets information.
LEARNING CURVES
On Figure and Figure I0 we present the learning curves from Table 2 in the main text. To make the plots readable, for each target dataset, report only the curves corresponding training from scratch, training from the Imagenet checkpoint, and from two checkpoints that perform best among the as representative subset of sources.
SYNTHETIC DATA DETAILS
Here we provide the details for the experiment described Section 2.2 The synthetic target data is formed by 10 Gaussians with centers on the circle of radius 20 and = 0.25. Source-I (blue) is distribution formed as sum of a uniform distribution on zero-centered circle of a radius 20 and the zero-centered Gaussians with 4. Source-Il (green) is formed by 3 Gaussians with centers that coincide with the consequent centers of three Gaussians of the original data and a 0.5. We use the standard GAN loss (Goodfellow et al.) 2014) and perform 5000 generator training steps with 4 discriminator steps for every generator step. We use batch size 64 and Adam optimizers with learning rate 0.0002 and 31, 82 0.5, 0.999. The generator has 64-dimensional latent space
http:tteh///e.es.ooooootaaaaaaaa htm
http::/ww.yy..0//2
12

---

Published as a conference paper at ICLR 2022
Imagenet
FFHQ
L.Bdroom
L.Cat
L.Church
L.Dog
S. Landscapes S. S.Buldiiggs
CIFAR-14 -00
Flowers
Grumpy Cat
Simpsons
BreCaHAD
Figure 8: Samples for each of the target and source datasets.
13

---

Published as a conference paper at ICLR 2022
L.B
L.Ca
L.Ch
L..oog
S.B
S.L.
F
0
244.0
194.8
240.5
178.8
256.1
233.3
150.8
L.B
244.0
165.0
182.8
162.4
233.3
236.7
143.4
L.Ca
194.8
165.0
200.8
97.6
206.9
185.9
104.1
L.Ch
240.5
182.8
200.8
0
167.0
199.8
232.5
140.4
L.Dog
178.8
162.4
97.6
167.0
200.0
182.3
63.9
S.B
256.1
233.3
206.9
199.8
200.0
0
172.2
177.5
S.L
233.3
236.7
185.9
232.5
182.3
172.2
145.3
C
197.2
188.1
120.9
192.3
102.2
202.1
185.3
85.4
F1
257.7
254.7
235.4
243.8
215.9
285.4
261.4
192.8
GC
293.1
260.8
188.4
259.2
259.3
341.4
334.5
264.4
S
252.5
225.2
199.4
218.8
195.9
217.7
244.3
167.6
BCH
347.8
345.7
319.7
356.0
303.8
351.2
245.4
280.4
Table 6: FID distances between source and target datasets. Underlined cell in a row corresponds to a source domain that is closest to fixed target. Datasets names are shortened as: L.Bdr (LSUN Bedroom), L.Cat (LSUN Cat), L.Chr (LSUN Church), L.Dog (LSUN Dog). S.Bld (Satel- lite Buildings), S.Lnd (Satellite Landscapes), Imgn (Imagenet), C-10 (CIFAR-10), Flw (Flowers), GC (Grumpy Cat), S (Simpsons), BCH (BreCaHAD).
L.B
L.Ca
L.Ch
L.Dog
S.B
S.I.
F
0
0.237
0.169
0.213
0.116
0.230
0.165
0.116
L.B
0.237
0
0.161
0.193
0.124
0.249
0.200
0.126
L.Ca
0.168
0.161
0.185
0.080
0.189
0.129
0.105
L.Ch
0.213
0.193
0.185
0
0.114
0.202
0.185
0.096
L.Dog
0.116
0.125
0.079
0.113
0.155
0.095
0.027
S.B
0.229
0.248
0.189
0.202
0.
0
O. 129
0.179
S.L.
0.165
0.200
0.130
0.185
0.095
0.129
0
0.109
C
0.137
0.149
0.092
0.144
0.048
0.170
0.117
0.060
Fl
0.227
0.260
0.211
0.230
0.157
0.277
0.212
0.153
GC
0.260
0.283
0.113
0.276
0.196
0.332
0.249
0.195
S
0.265
0.276
0.215
0.244
0.178
0.247
0.227
0.179
BCH
0.335
0.374
0.316
0.375
0.267
0.349
0.205
0.273
Table 7: KID distances between source and target datasets computed. Highlighted cell in row corresponds to a source domain that is closest to a fixed target.
and consists of six consequent linear layers, all but the last followed by batch-norms and ReLU- activations. The intermediate layers' sizes are 64, 128, 128, 128, 64. The discriminator is formed by a sequence five linear layers, each but the last followed by the ReLU-activation. The intermediate layers' sizes are 64, 128, 128, 64.
The starting checkpoints for the Dissecting Contributions experiments are taken from the interme- diate checkpoints of the GAN training for Source-I. We take every 50-th checkpoint, gathering 100 in total. We perform fine-tuning to the target distribution with the same parameters as above except the number of steps equals 1000.
LONGER TRAINING
In this series of experiments, run GAN training for source checkpoint being either Imagenet- pretrained or randomly initialized for two times higher number of steps (50 million real images shown to the discriminator). The results are presented in Table I0 Generally, Imagenet-pretraining almost always either improves GAN quality or performs equally to the random initialization while speeding up convergence by large margin.
14

---

Published as a conference paper at ICLR 2022
F
L.B
L.Ca
L.Ch
L.Dog
S.B
S.L
F
0.000
0.014
0.000
0.057
0.001
0.000
0.005
L.B
0.333
0.333
0.235
0.337
0.307
0.058
0.021
L.Ca
0.448
0.598
0.253
0.384
0.619
0.229
0.094
L.Ch
0.027
0.050
0.007
0.058
0.208
0.016
0.003
L.Dog
0.539
0.679
0.591
0.350
0.726
0.265
0.144
S.B
0.000
0.000
0.000
0.000
0.000
0.000
0.000
S.L.
0.007
0.014
0.007
0.042
0.002
0.705
0.016
C
0.000
0.000
0.000
0.000
0.000
0.001
0.000
0.000
Fl
0.006
0.000
0,001
0.000
0.000
0.002
0.012
0.003
GC
0.000
0.000
0.001
0.000
0.000
0.000
0.000
0.000
S
0.000
0.000
0.000
0.012
0.000
0.046
0.000
0.000
BCH
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
Table The Precision values computed for the targets datasets w.r.t the source datasets.
F
L.B
L.Ca
L.Ch
L.Dog
S.B
S.L
F
0.333
0.448
0.027
0.539
0.000
0.007
0.737
L.B
0.000
0.598
0.050
0.679
0.000
0.014
0.124
L.Ca
0.014
0.333
0.007
0.591
0.000
0.007
0.218
L.Ch
0.000
0.235
0.253
0.350
0.000
0.042
0.303
L.Dog
0.057
0.337
0.384
0.058
0.000
0.002
0.325
S.B
0.001
0.307
0.619
0.208
0.726
0.705
0.533
S.L.
0.000
0.058
0.229
0.016
0.265
0.000
0.378
C
0.001
0.053
0.240
0.006
0.340
0.000
0.003
0.718
Fi
0.001
0.183
0.249
0.010
0.410
0.000
0.017
0.708
GC
0.000
0.020
0.790
0.000
0.970
0.000
0.000
0.000
S
0.013
0.324
0.328
0.060
0.379
0.000
0.045
0.294
BCH
0.001
0.203
0.428
0.053
0.546
0.006
0.553
0.789
Table The Recall values computed for targets datasets w.r.t. the source datasets.
15

---

Published as a conference paper at ICLR 2022
From Scratch
Imagenet
pretraining
Dataset
Step (M)
FID
Precision
Recall
Step (M)
FID
Precision
Recall
L.Bedroom
50
2.50
0.663
0.485
50
2.33
0.691
0.483
L.Cat
42
6.87
0.686
0.394
48
6.35
0.712
0.385
L.Church
36
3.01
0.705
0.547
12
3.00
0.693
0.523
L.Dog
40
12.7
0.751
0.384
45
12.8
0.753
0.382
S.Buildings
35
11.9
0.363
0.498
14
10.9
0.304
0.591
S.Landscapes
25
27.4
0.737
0.214
1
21.1
0.721
0.393
Table 10: Number of real images shown the discriminator (step) for the checkpoint with the best FID value, this value and corresponding precison and recall values for long-tern trainings with two initialization options.
Source Model
FID
Precision
Recall
Steps Convergance
Imagenet
8.31
0.77
0.28
9
Imagenet (half)
8.54
0.81
0.22
25
FFHQ
9.47
0.79
0.25
22
FFHQ (half)
9.5
0.77
0.27
25
Table 11: Finetuning Flowers from converged source checkpoint and from checkpoint that passes two times fewer steps.
TRANSFER FROM AN EARLIER EPOCH
This experiment verifies if it is important transfer from well-converged nearly-optimal source checkpoint, or is sufficient to start from a roughly stabilized checkpoint from the intermediate step of the optimization process. To address the question, with Imagenet and FFHQ as source domains, and Flowers as a target domain. As pretrained check- points, consider the best-FID checkpoint and checkpoint that passed two times fewer steps. The results for these runs are presented in Table 11 Overall, the choice between two options has only a marginal impact the transfer quality, and training to initialize the finetuning process.
perform series of additional experiments
the source checkpoint from the middle of
DETAILS OF EXPERIMENTS ON THE GAN INVERSION
We take the e4e generator inversion approach proposed by Tov et al. 2021 and train an encoder that maps real data the GAN latent space. This scheme is known to be capable of mapping real images to the GAN latent space preserving all generator properties such as latent attributes manipulations. We follow the original author's implementation and train an independent encoder model for each generator. For a generator G we receive an encoder E which is trained to satisfy G(E =x each real data sample x. We evaluate the encoders with the average I.PIP--dsttane (Zhang et al. 2018) between a test set real samples and their inversions equal Err.pp T.PISS( G G(F(r))) We also report the average distance between an original image and its reconstruction features of pretrained features extractor F' which is equal Ex- [F(x) F(G(E(z)))ll The lower these quantities the better reconstruction quality is. Following TTov et al. 2021). for FFHQ-target generators, train the encoder the FFHQ dataset and evaluate it the Celeba-HQ dataset and on FFHQ itself. As for SeUN-oedoo we split the original data into a train and a test subset in the proportion 9 and train e4e the train set and evaluate the test set. As the feature extractor F, for FFHQ we use a Face-ID pretrained model, same as in (Tov et al. 202T). and MoCo-v2 (Chen) al. 2020b) model for LSUN-Bedroom
16

---

Published as a conference paper at ICLR 2022
FFHQ
14
1.0
1.0
0.8
Precision
FID
0.6
0.6
0.4
0.4
0.2
10
15
20
25
0.0
5
10
15
20
25
0.0
10
15
20
25
1M images
1M images
1M images
LSUN-Bedroom
LSUN- Bedroom
1.0
0.8
Precision
FID
5
0.6
Recall
0.6
0.4
0.0
0.0
15
20
25
5
10
15
20
25
10
15
20
25
1M images
images
1M images
LSUN-Cat
1.0
13
0.8
0.8-
Precision
FID
11
0.6
0.6
0.2
02
0.0
0.0
10
15
20
25
1M images
1M images
1M images
LSUN-Church
1.0
1.0
0.8
0.8
Precision
HDD
0.6
0.4
3
0.2
0.2
0.0
0.0
1M images
1M images
1M images
LSUN-Dog
30
1.0
1.0
0.8
Precision
HDD
0.6
0.6
18
0.4
0.4
14
0.2
100
0.0
5
10
15
20
0.0
1M images
1M images
1M images
Buillite-Buildings
30
1.0
Precision
22
0.6
Recall
0.6
0.4
0.2
10
0.0
0.0
15
20
25
5
10
15
20
25
10
15
20
25
1M images
images
1M images
From Scratch
FFHQ
LSUN-Cat
LSUN-Dog
Imagenet
LSUN-Church
Figure 9: Learning curves for different target and sources datasets, part 1.
17

---

Published as a conference paper at ICLR 2022
Langliip-L
40
1.0
1.0
0.8
Precision
0.6
Recall
0.6
28-
0.4
0.4
24
0.2
200
0.0
0.0
10
15
20
25
5
10
15
20
25
10
15
20
25
1M images
1M images
1M images
CIFAR-10
15
1.0
0.8
Precision
FID
11
0.6
0.6
0.4
0.4
15
20
25
0.0
5
10
15
20
25
0
10
15
20
25
1M images
images
1M images
Flowers
1.0
13
0.8
Precision
FID
11
0.6
0.6
0.2
0.0
10
15
20
25
0.0
1M images
1M images
1M images
Grumpy-Cat
30
1.0
1.0
26
0.8
0.8
Precision
0.6
0.4
14
0.2
022
10
0.0
0.0
1M images
1M images
IM images
Simpsons
16
1.0
1.0
0.8
Precision
HDD
0.6
Recall
0.6
10
0.4
0.4
0.2
0.0
10
20
0.0
IM
images
1M images
1M images
BreCaHIA
40
1.0
Precision
32
0.6
Recall
0.6
0.4
0.2
20
0.0
15
20
25
5
10
15
20
25
0
10
15
20
25
1M images
images
1M images
Satellite-Lnndscaps Figure 10: Learning curves for different target and sources datasets, part 2.
From Scratch Imagenet
FEHQ
LSUN-Dog
BeUN-Bedroom
LSUN-Cat
18


Reviews:
[{"review_id": "QJnP1qSHsOT", "reviewer": "Reviewer_BE8y", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**  \n- Well written and easy to understand.  \n- Transfer learning for GANs is an important subject. Better pretrained models could save much training time and enable users with less computational resources to achieve good results.  \n- Simple synthetic experiment gives good insight into the roles that generator and discriminator play during transfer learning.  \n\n**Weaknesses:**  \n- One of the main conclusions of this work is that diverse source datasets (i.e., ImageNet) result in the best performance during transfer learning. However, most of the target datasets contain classes which overlap with ImageNet classes in some form, so it is not surprising that this model transfers well. It would be good to see how ImageNet transfers to datasets it does not overlap much, such as those from the medical imaging domain (one option is the BreCaHAD dataset from [1]). \n- An aspect that I think is missing from this study is the level of convergence of the source model. That is to say, is it better to take a checkpoint that it still in the middle of training, or one that has completely converged? What is the optimal time to take a checkpoint?  \n- No confidence intervals on most experimental results. This is understandable due to the excessive compute budget required to train each of these models, but it still makes it difficult to understand the significance of the results.  \n\n**Other Questions and Comments:**  \n- In Table 2, row Grumpy Cat, column ImageNet, the P and R values seem very off compared to values from other models, and they do not match the values shown in Figure 10.  \n-In Figure 9, why does ImageNet pretrained models often decrease very fast and then get worse? Is the model collapsing because the discriminator is quickly overfitting? Shouldn't the adaptive data augmentation prevent this kind of catastrophic collapse?  \n- In Figure 10, FFHQ, why does the model trained from scratch diverge? I would have expected StyleGAN2 to perform very well on FFHQ since this is the dataset it was designed for.  \n- Many lines are missing from plots in Figure 10 and 11. Why is this?\n- If ImageNet checkpoint was not an option to use, would the proposed checkpoint selection method still have selected the best choice of the remaining options?\n\n[1] A. Aksac, D. J. Demetrick, T. Ozyer, and R. Alhajj. BreCaHAD: A dataset for breast cancer histopathological annotation and diagnosis. BMC Research Notes, 12, 2019.  ", "summary_of_the_paper": "This paper aims to achieve a better understanding of GAN fine-tuning. A synthetic experiment demonstrates that pretrained discriminators improve the quality of the initial gradients, and pretrained generators help with improving mode coverage. Transfer learning experiments on real image datasets reveal that pretraining primarily improves mode coverage rather than sample fidelity, and that datasets containing a diverse set of images are best suited for transfer.", "summary_of_the_review": "Paper provides useful insights about the role of the generator and discriminator in GAN transfer learning, as well as the behaviour of transfer learning itself (i.e., improved coverage but not fidelity).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3CJ84EFCZGj", "reviewer": "Reviewer_iExS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n(1). This kind of practical paper has significance for the entire community. This paper makes some conclusions, which could promote the development of adapting pre-trained GANs to different domains. \n(2). Part 2.1 “high-level intuition” is very interesting.\n(3). The writing is clear and easy to understand.\n\nCons:\n(1). It would be better to add a “related works” section to compare this paper with related papers; \n(2). In the introduction part, the author claimed that “some conclusions from Wang are not confirmed for modern architectures”. It would be more clear to summarize which conclusion is not confirmed in the related work part. \n(3). It would be better to list the distance between source datasets and target datasets in Table 1. \n", "summary_of_the_paper": "This paper studies the problem: when, why, and which pretrained GANs are useful. \n", "summary_of_the_review": "In general, this paper has conducted a series of experiments with the stylegan2, and got some conclusions, which are helpful for subsequent papers and the community. The analysis and experiments are not very comprehensive. In many ways, [1] is more comprehensive than this one, although [1] is from a few years ago.\n\n[1]. Transferring GANs: generating images from limited data\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0ndPqhPKvNP", "reviewer": "Reviewer_8amM", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While some aspects of the work have been hinted at in prior art, to the best of my knowledge no one has properly studied the questions so eloquently put forward in the title. These are important questions and I for one have wanted to know the answers for some time! The paper does a good job building intuition through toy examples, large-scale transfer test (Table 2), and some quite novel tests (Figs 6 & 7). Overall this feels like important work that should to be accepted. I will champion this paper.\n\nA possible error:\n- p.3 second to last para: \"... have significant negative correlations with the finetuned GAN quality\". I was very surprised to read this and in fact had to backtrack twice. I suspect this statement is strictly false. Unless I'm mistaken, the negative correlation is with Wasserstein distance, but small distance implies high quality, and thus there is a strong POSITIVE correlation with quality. Right?\n\nSome minor complaints/suggestions/questions:\n- [citation] is not a noun. It's silent when reading the sentence.\n- Personally I found the contribution list at the end of Sec 1 completely unnecessary. We have just read all the same items... (twice if you count the abstract)\n- Thumb up/down in Figure 1 may not be the best way to convey the message.\n- p5: ImageNet pretraining is always useful: This is an important finding, but LSUN Dog and FFHQ were also uniformly beneficial. ImageNet was just a bit better. Please consider rewording. There is no magic in ImageNet, and it's very likely that some higher quality diverse dataset would almost certainly be even better.\n- Table 2: \"Random\" on the top row was initially mysterious. You mean it's random initialized, while the text up to this point has been talking about training from scratch. Consider a clarification, maybe in the caption.\n- Fig 7: What exactly does the class change probability mean? Probability during which timespan? Between init and current, or perhaps current and some previous timestep. (the answer isn't too important, just make sure to be specific). \n- Q: The method in Sec 4 is used to choose a particular snapshot. This is nice. Now, if I understand correctly, you're using it for selecting the best snapshot per source dataset -- did you consider using the same heuristic for also selecting the best source dataset/initialization dataset? If it doesn't work, why?\n\n\n", "summary_of_the_paper": "This paper performs a large-scale study of transfer learning in GANs. It proposes a way to understand the relevance of a pre-trained generator and discriminator, as well as heuristics to select good source/initialization dataset and even a training snapshot. All of this is very valuable for the practitioners.", "summary_of_the_review": "This paper answers important questions in GAN transfer learning, and therefore makes a solid contribution.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper empirically studies when, why, and which pretrained GANs are useful.  
All the reviewers are positive about this work, that they all consider very valuable for practitioners and the community.  
First building intuition through toy examples, authors conduct a large-scale study of transfer learning in GANs (with the stylegan2). They propose a way to understand the relevance of a pre-trained generator and discriminator, as well as heuristics to select good initialization.  
Overall, this paper makes a solid contribution that should to be accepted.
Decision: Accept (Poster)