When, Why, and Which Pretrained GANs Are Useful?

1 INTRODUCTION . These days , generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) can successfully approximate the high-dimensional distributions of real images . The exceptional quality of the state-ofthe-art GANs ( Karras et al. , 2020b ; Brock et al. , 2019 ) makes them a key ingredient in applications , including semantic editing ( Isola et al. , 2017 ; Zhu et al. , 2018 ; Shen et al. , 2020 ; Voynov & Babenko , 2020 ) , image processing ( Pan et al. , 2020 ; Ledig et al. , 2017 ; Menon et al. , 2020 ) , video generation ( Wang et al. , 2018a ) , producing high-quality synthetics ( Zhang et al. , 2021 ; Voynov et al. , 2020 ) . To extend the success of GANs to the limited-data regime , it is common to use pretraining , i.e. , to initialize the optimization process by the GAN checkpoint pretrained on some large dataset . A line of works ( Wang et al. , 2018b ; Noguchi & Harada , 2019 ; Zhao et al. , 2020 ; Mo et al. , 2020 ; Wang et al. , 2020 ; Li et al. , 2020 ) investigate different methods to transfer GANs to new datasets and report significant advantages compared to training from scratch both in terms of generative quality and convergence speed . However , the empirical success of GAN pretraining was not investigated in-depth , and its reasons are not entirely understood . From the practical standpoint , it is unclear how to choose a proper pretrained checkpoint or if one should initialize both generator and discriminator or only one of them . To the best of our knowledge , the only work that systematically studies the benefits of pretraining is Wang et al . ( 2018b ) . However , the experiments in ( Wang et al. , 2018b ) were performed with currently outdated models , and we observed that some conclusions from Wang et al . ( 2018b ) are not confirmed for modern architectures like StyleGAN2 ( Karras et al. , 2020b ) . In particular , unlike the prior results , it appears that for state-of-the-art GANs , it is beneficial to transfer from sparse and diverse rather than dense and less diverse sources . In this work , we thoroughly investigate the process of GAN finetuning . First , we demonstrate that starting the GAN training from the pretrained checkpoint can significantly influence the diversity of the finetuned model , while the fidelity of individual samples is less affected . Second , we dissect the mechanisms of how pretrained generators and discriminators contribute to the higher coverage of finetuned GANs . In a nutshell , we show that a proper pretrained generator produces samples in the neighborhood of many modes of the target distribution , while a proper pretrained discriminator serves as a gradient field that guides the samples to the closest mode , which together result in a smaller risk of mode missing . This result explains the evidence from the literature that it is beneficial to initialize both generator and discriminator when finetuning GANs . Finally , we investigate different ways to choose a suitable pretrained GAN checkpoint for a given target dataset . Interestingly , for most of the tasks , Imagenet-pretrained models appear to be the optimal initializers , which mirrors the pretraining of discriminative models , where Imagenet-based initialization is de-facto standard ( Donahue et al. , 2014 ; Long et al. , 2015 ; He et al. , 2020 ; Chen et al. , 2020a ) . Our conclusions are confirmed by experiments with the state-of-the-art StyleGAN2 ( Karras et al. , 2020b ) , chosen due to its practical importance and a variety of open-sourced checkpoints , which can be used as pretrained sources . The main contributions of our analysis are the following : 1 . We show that initializing the GAN training by the pretrained checkpoint can significantly affect the coverage and has much less influence on the realism of individual samples . 2 . We explain why it is important to initialize both generator and discriminator by describing their roles in the finetuning process . 3 . We describe a simple automatic approach to choose a pretrained checkpoint that is the most suitable for a given generation task . 2 ANALYSIS . This section aims to explain the success of the GAN finetuning process compared to training from scratch . First , we formulate the understanding of this process speculatively and then confirm this understanding by experiments on synthetic data and real images . 2.1 HIGH-LEVEL INTUITION . Let us consider a pretrained generator G and discriminator D that are used to initialize the GAN training on a new data from a distribution ptarget . Throughout the paper , we show that a discriminator initialization is “ responsible for ” an initial gradient field , and a generator initialization is “ responsible for ” a target data modes coverage . Figure 1 illustrates the overall idea with different initialization patterns . Intuitively , the proper discriminator initialization guarantees that generated samples will move towards “ correct ” data regions . On the other hand , the proper pretrained generator guarantees that the samples will be sufficiently diverse at the initialization , and once guided by this vector field , they will cover all target distribution . Below , we confirm the validity of this intuition . 2.2 SYNTHETIC EXPERIMENT . We start by considering the simplest synthetic data presented on Figure 2 . Our goal is to train a GAN on the target distribution , a mixture of ten Gaussians arranged in a circle . We explore three options to initialize the GAN training process . First , we start from random initialization . The second and the third options initialize training by GANs pretrained on the two different source distributions . The first source distribution corresponds to a wide ring around the target points , having high coverage and low precision w.r.t . target data . The second source distribution is formed by three Gaussians that share their centers with three target ones but have a slightly higher variance . This source distribution has high precision and relatively low coverage w.r.t . target data . Then we train two source GANs from scratch to fit the first and the second source distributions and employ these checkpoints to initialize GAN training on the target data . The results of GAN training for the three options are presented on Figure 2 , which shows the advantage of pretraining from the more diverse model , which results in a higher number of covered modes . The details of the generation of the synthetic are provided in the appendix . Dissecting the contributions from G and D. Here , we continue with the synthetic example from above and take a closer look at the roles that the pretrained generator and discriminator play when finetuning GANs . Our goal is to highlight the importance of ( 1 ) the initial coverage of the target distribution by the pretrained generator and ( 2 ) the quality of the gradient field from the pretrained discriminator . We quantify the former by the established recall measure ( Kynkäänniemi et al. , 2019 ) computed in the two-dimensional dataspace with k=5 for 1000 randomly picked samples from the target distribution and the same number of samples produced by the pretrained generator . To evaluate the quality of the discriminator gradient field , we use a protocol described in ( Sinha et al. , 2020 ) . Namely , we assume that the “ golden ” groundtruth gradients would guide each sample towards the closest Gaussian center from the target distribution . Then we compute the similarity between the vector field ∇xD provided by the pretrained discriminator and the vector field of “ golden ” gradients . Specifically , we evaluate the cosine similarity between these vector fields , computed for the generated samples . Given these two measures , we consider a series of different starting generator/discriminator checkpoints ( Gi , Di ) , i = 1 , . . . , N . The details on the choice of the starting checkpoints are provided in the appendix . Then we use each pair ( Gi , Di ) as initialization of GAN training on the target distribution of ten Gaussians described above . Additionally , for all starting Gi/Di , we evaluate the recall and the discriminator gradients field similarity to the “ golden ” gradients . The overall quality of GAN finetuning is measured as the Wasserstein-1 distance between the target distribution and the distribution produced by the finetuned generator . The scatter plots of recall , the similarity of gradient fields , and Wasserstein-1 distance are provided in Figure 3 . As can be seen , both the recall and gradient similarity have significant negative correlations with the W1-distance between the ground-truth distribution and the distribution of the finetuned GAN . Furthermore , for the same level of recall , the higher values of the gradient similarity correspond to lower Wasserstein distances . Alternatively , for the same value of gradient similarity , higher recall of the source generator typically corresponds to the lower Wasserstein distance . We also note that the role of the pretrained generator is more important since , for high recall values , the influence from the discriminator is not significant ( see Figure 3 , left ) . This synthetic experiment does not rigorously prove the existence of a causal relationship between the recall or gradient similarity and the quality of the finetuned GANs since it demonstrates only correlations of them . However , in the experimental section , we show that these correlations can be successfully exploited to choose the optimal pretraining checkpoint , even for the state-of-the-art GAN architectures . 3 LARGE-SCALE EXPERIMENTS . 3.1 EXPLORING PRETRAINING FOR STYLEGAN2 . In this section , we confirm the conclusions from the previous sections experimentally with the stateof-the-art StyleGAN2 architecture ( Karras et al. , 2020b ) . If not stated otherwise , we always work with the image resolution 256× 256 . Datasets . We work with six standard datasets established in the GAN literature . We also include two datasets of satellite images to investigate the pretraining behavior beyond the domain of natural images . As potential pretrained sources , we use the StyleGAN2 models trained on these datasets . Table 1 reports the list of datasets and the FID values ( Heusel et al. , 2017 ) of the source checkpoints . We also experimented with four smaller datasets to verify our conclusions in the medium-shot and few-shot regimes . The details on the datasets are provided in the appendix . Experimental setup . Here , we describe the details of our experimental protocol for both the pretraining of the source checkpoints and the subsequent training on the target datasets . We always use the official PyTorch implementation of StyleGAN2-ADA ( Karras et al. , 2020a ) provided by the authors1 . We use the “ stylegan2 ” configuration in the ADA implementation with the default hyperparameters ( same for all datasets ) . Training is performed on eight Tesla V100 GPUs and takes approximately three hours per 1M real images shown to the discriminator . Pretraining of source checkpoints . We pretrain one checkpoint on the Imagenet for 50M real images shown to the discriminator and seven checkpoints on other source datasets from Table 1 for 25M images . A larger number of optimization steps for the Imagenet is used since this dataset is more challenging and requires more training epochs to converge . For the large LSUN datasets ( Cat , Dog , Bedroom ) , we use 106 first images to preserve memory . For Satellite-Landscapes , we use ADA due to its smaller size . Then , we always use checkpoints with the best FID for further transferring to target datasets for each source dataset . Training on target datasets . For each source checkpoint , we perform transfer learning to all datasets from Table 1 . We use the default transfer learning settings from the StyleGAN2-ADA implementation ( faster adaptive data augmentation ( ADA ) adjustment rate , if applicable , and no Gema warmup ) . ADA is disabled for the datasets containing more than 50K images and enabled 1https : //github.com/NVlabs/stylegan2-ada-pytorch for others with default hyperparameters . In these experiments , we train for 25M real images shown to the discriminator . Each transfer experiment is performed with three independent runs , and the metrics are reported for the run corresponding to the median best FID ( Heusel et al. , 2017 ) . Metrics . In the experiments , we evaluate the performance via the four following metrics . ( 1 ) Frechet Inception Distance ( FID ) ( Heusel et al. , 2017 ) , which quantifies the discrepancy between the distributions of real and fake images , represented by deep embeddings . Both distributions are approximated by Gaussians , and the Wasserstein distance between them is computed . ( 2 ) Precision ( Kynkäänniemi et al. , 2019 ) , which measures the realism of fake images , assuming that the visual quality of a particular fake is high if it belongs to the neighborhood of some real images in the embedding space . ( 3 ) Recall ( Kynkäänniemi et al. , 2019 ) , which quantifies GAN diversity , measuring the rate of real images that belong to the neighborhood of some fake images in the embedding space . ( 4 ) Convergence rate equals a number of real images that were shown to the discriminator at the moment when the generator FID for the first time exceeded the optimal FID by at most 5 % . Intuitively , this metric quantifies how fast the learning process reaches a plateau . FID is computed based on the image embeddings extracted by the InceptionV3 model2 . Precision and Recall use the embeddings provided by the VGG-16 model3 . Precision and Recall are always computed with k=5 neighbors . For FID calculation , we always use all real images and 50.000 generated samples . For Precision/Recall calculation , we use the first 200K real images ( or less , if the real dataset is smaller ) and the same amount of generated samples . Results . The metric values for all datasets are reported in Table 2 , where each cell corresponds to a particular source-target pair . For the best ( in terms of FID ) checkpoint obtained for each sourcetarget transfer , we report the FID value ( top row in each cell ) , Precision and Recall ( the second and the third rows in each cell ) , and the convergence rate measured in millions of images ( bottom row in each cell ) . We highlight the sources that provide the best FID for each target dataset or differ from the best one by at most 5 % . We additionally present the curves of FID , Precision , and Recall values for several target datasets on Figure 9 and Figure 10 in the appendix . We describe the key observations from Table 2 below : • In terms of FID , a pretraining based on a diverse source ( e.g. , Imagenet or LSUN Dog ) is superior to training from scratch on all datasets in our experiments . • The choice of the source checkpoint significantly influences the coverage of the finetuned model , and the Recall values vary considerably for different sources , especially for smaller target datasets . For instance , on the Flowers dataset , their variability exceeds ten percent . In contrast , the Precision values are less affected by pretraining , and their typical variability is about 2−3 % . Figure 4 reports the standard deviations of Precision/Recall computed over different sources and highlights that Recall has higher variability compared to Precision , despite the latter having higher absolute values . • Pretraining considerably speeds up the optimization compared to the training from scratch . 2https : //nvlabs-fi-cdn.nvidia.com/stylegan2-ada-pytorch/pretrained/ metrics/inception-2015-12-05.pt 3https : //nvlabs-fi-cdn.nvidia.com/stylegan2-ada-pytorch/pretrained/ metrics/vgg16.pt Overall , despite having poor quality ( FID=49.2 ) , the Imagenet-pretrained unconditional StyleGAN2 model appears to be a superior GAN initialization that typically leads to more efficient optimization compared to alternatives . This result contradicts the observations in ( Wang et al. , 2018b ) showing that it is beneficial to transfer from dense and less diverse sources rather than sparse and diverse ones , like Imagenet . We attribute this inconsistency to the fact that ( Wang et al. , 2018b ) experimented with the WGAN-GP models , which are significantly inferior to the current state-of-the-art ones .

Summary: This paper performs a large-scale study of transfer learning in GANs. It proposes a way to understand the relevance of a pre-trained generator and discriminator, as well as heuristics to select good source/initialization dataset and even a training snapshot. All of this is very valuable for the practitioners.