When, Why, and Which Pretrained GANs Are Useful?

1 INTRODUCTION . These days , generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) can successfully approximate the high-dimensional distributions of real images . The exceptional quality of the state-ofthe-art GANs ( Karras et al. , 2020b ; Brock et al. , 2019 ) makes them a key ingredient in applications , including semantic editing ( Isola et al. , 2017 ; Zhu et al. , 2018 ; Shen et al. , 2020 ; Voynov & Babenko , 2020 ) , image processing ( Pan et al. , 2020 ; Ledig et al. , 2017 ; Menon et al. , 2020 ) , video generation ( Wang et al. , 2018a ) , producing high-quality synthetics ( Zhang et al. , 2021 ; Voynov et al. , 2020 ) . To extend the success of GANs to the limited-data regime , it is common to use pretraining , i.e. , to initialize the optimization process by the GAN checkpoint pretrained on some large dataset . A line of works ( Wang et al. , 2018b ; Noguchi & Harada , 2019 ; Zhao et al. , 2020 ; Mo et al. , 2020 ; Wang et al. , 2020 ; Li et al. , 2020 ) investigate different methods to transfer GANs to new datasets and report significant advantages compared to training from scratch both in terms of generative quality and convergence speed . However , the empirical success of GAN pretraining was not investigated in-depth , and its reasons are not entirely understood . From the practical standpoint , it is unclear how to choose a proper pretrained checkpoint or if one should initialize both generator and discriminator or only one of them . To the best of our knowledge , the only work that systematically studies the benefits of pretraining is Wang et al . ( 2018b ) . However , the experiments in ( Wang et al. , 2018b ) were performed with currently outdated models , and we observed that some conclusions from Wang et al . ( 2018b ) are not confirmed for modern architectures like StyleGAN2 ( Karras et al. , 2020b ) . In particular , unlike the prior results , it appears that for state-of-the-art GANs , it is beneficial to transfer from sparse and diverse rather than dense and less diverse sources . In this work , we thoroughly investigate the process of GAN finetuning . First , we demonstrate that starting the GAN training from the pretrained checkpoint can significantly influence the diversity of the finetuned model , while the fidelity of individual samples is less affected . Second , we dissect the mechanisms of how pretrained generators and discriminators contribute to the higher coverage of finetuned GANs . In a nutshell , we show that a proper pretrained generator produces samples in the neighborhood of many modes of the target distribution , while a proper pretrained discriminator serves as a gradient field that guides the samples to the closest mode , which together result in a smaller risk of mode missing . This result explains the evidence from the literature that it is beneficial to initialize both generator and discriminator when finetuning GANs . Finally , we investigate different ways to choose a suitable pretrained GAN checkpoint for a given target dataset . Interestingly , for most of the tasks , Imagenet-pretrained models appear to be the optimal initializers , which mirrors the pretraining of discriminative models , where Imagenet-based initialization is de-facto standard ( Donahue et al. , 2014 ; Long et al. , 2015 ; He et al. , 2020 ; Chen et al. , 2020a ) . Our conclusions are confirmed by experiments with the state-of-the-art StyleGAN2 ( Karras et al. , 2020b ) , chosen due to its practical importance and a variety of open-sourced checkpoints , which can be used as pretrained sources . The main contributions of our analysis are the following : 1 . We show that initializing the GAN training by the pretrained checkpoint can significantly affect the coverage and has much less influence on the realism of individual samples . 2 . We explain why it is important to initialize both generator and discriminator by describing their roles in the finetuning process . 3 . We describe a simple automatic approach to choose a pretrained checkpoint that is the most suitable for a given generation task . 2 ANALYSIS . This section aims to explain the success of the GAN finetuning process compared to training from scratch . First , we formulate the understanding of this process speculatively and then confirm this understanding by experiments on synthetic data and real images . 2.1 HIGH-LEVEL INTUITION . Let us consider a pretrained generator G and discriminator D that are used to initialize the GAN training on a new data from a distribution ptarget . Throughout the paper , we show that a discriminator initialization is “ responsible for ” an initial gradient field , and a generator initialization is “ responsible for ” a target data modes coverage . Figure 1 illustrates the overall idea with different initialization patterns . Intuitively , the proper discriminator initialization guarantees that generated samples will move towards “ correct ” data regions . On the other hand , the proper pretrained generator guarantees that the samples will be sufficiently diverse at the initialization , and once guided by this vector field , they will cover all target distribution . Below , we confirm the validity of this intuition . 2.2 SYNTHETIC EXPERIMENT . We start by considering the simplest synthetic data presented on Figure 2 . Our goal is to train a GAN on the target distribution , a mixture of ten Gaussians arranged in a circle . We explore three options to initialize the GAN training process . First , we start from random initialization . The second and the third options initialize training by GANs pretrained on the two different source distributions . The first source distribution corresponds to a wide ring around the target points , having high coverage and low precision w.r.t . target data . The second source distribution is formed by three Gaussians that share their centers with three target ones but have a slightly higher variance . This source distribution has high precision and relatively low coverage w.r.t . target data . Then we train two source GANs from scratch to fit the first and the second source distributions and employ these checkpoints to initialize GAN training on the target data . The results of GAN training for the three options are presented on Figure 2 , which shows the advantage of pretraining from the more diverse model , which results in a higher number of covered modes . The details of the generation of the synthetic are provided in the appendix . Dissecting the contributions from G and D. Here , we continue with the synthetic example from above and take a closer look at the roles that the pretrained generator and discriminator play when finetuning GANs . Our goal is to highlight the importance of ( 1 ) the initial coverage of the target distribution by the pretrained generator and ( 2 ) the quality of the gradient field from the pretrained discriminator . We quantify the former by the established recall measure ( Kynkäänniemi et al. , 2019 ) computed in the two-dimensional dataspace with k=5 for 1000 randomly picked samples from the target distribution and the same number of samples produced by the pretrained generator . To evaluate the quality of the discriminator gradient field , we use a protocol described in ( Sinha et al. , 2020 ) . Namely , we assume that the “ golden ” groundtruth gradients would guide each sample towards the closest Gaussian center from the target distribution . Then we compute the similarity between the vector field ∇xD provided by the pretrained discriminator and the vector field of “ golden ” gradients . Specifically , we evaluate the cosine similarity between these vector fields , computed for the generated samples . Given these two measures , we consider a series of different starting generator/discriminator checkpoints ( Gi , Di ) , i = 1 , . . . , N . The details on the choice of the starting checkpoints are provided in the appendix . Then we use each pair ( Gi , Di ) as initialization of GAN training on the target distribution of ten Gaussians described above . Additionally , for all starting Gi/Di , we evaluate the recall and the discriminator gradients field similarity to the “ golden ” gradients . The overall quality of GAN finetuning is measured as the Wasserstein-1 distance between the target distribution and the distribution produced by the finetuned generator . The scatter plots of recall , the similarity of gradient fields , and Wasserstein-1 distance are provided in Figure 3 . As can be seen , both the recall and gradient similarity have significant negative correlations with the W1-distance between the ground-truth distribution and the distribution of the finetuned GAN . Furthermore , for the same level of recall , the higher values of the gradient similarity correspond to lower Wasserstein distances . Alternatively , for the same value of gradient similarity , higher recall of the source generator typically corresponds to the lower Wasserstein distance . We also note that the role of the pretrained generator is more important since , for high recall values , the influence from the discriminator is not significant ( see Figure 3 , left ) . This synthetic experiment does not rigorously prove the existence of a causal relationship between the recall or gradient similarity and the quality of the finetuned GANs since it demonstrates only correlations of them . However , in the experimental section , we show that these correlations can be successfully exploited to choose the optimal pretraining checkpoint , even for the state-of-the-art GAN architectures . 3 LARGE-SCALE EXPERIMENTS . 3.1 EXPLORING PRETRAINING FOR STYLEGAN2 . In this section , we confirm the conclusions from the previous sections experimentally with the stateof-the-art StyleGAN2 architecture ( Karras et al. , 2020b ) . If not stated otherwise , we always work with the image resolution 256× 256 . Datasets . We work with six standard datasets established in the GAN literature . We also include two datasets of satellite images to investigate the pretraining behavior beyond the domain of natural images . As potential pretrained sources , we use the StyleGAN2 models trained on these datasets . Table 1 reports the list of datasets and the FID values ( Heusel et al. , 2017 ) of the source checkpoints . We also experimented with four smaller datasets to verify our conclusions in the medium-shot and few-shot regimes . The details on the datasets are provided in the appendix . Experimental setup . Here , we describe the details of our experimental protocol for both the pretraining of the source checkpoints and the subsequent training on the target datasets . We always use the official PyTorch implementation of StyleGAN2-ADA ( Karras et al. , 2020a ) provided by the authors1 . We use the “ stylegan2 ” configuration in the ADA implementation with the default hyperparameters ( same for all datasets ) . Training is performed on eight Tesla V100 GPUs and takes approximately three hours per 1M real images shown to the discriminator . Pretraining of source checkpoints . We pretrain one checkpoint on the Imagenet for 50M real images shown to the discriminator and seven checkpoints on other source datasets from Table 1 for 25M images . A larger number of optimization steps for the Imagenet is used since this dataset is more challenging and requires more training epochs to converge . For the large LSUN datasets ( Cat , Dog , Bedroom ) , we use 106 first images to preserve memory . For Satellite-Landscapes , we use ADA due to its smaller size . Then , we always use checkpoints with the best FID for further transferring to target datasets for each source dataset . Training on target datasets . For each source checkpoint , we perform transfer learning to all datasets from Table 1 . We use the default transfer learning settings from the StyleGAN2-ADA implementation ( faster adaptive data augmentation ( ADA ) adjustment rate , if applicable , and no Gema warmup ) . ADA is disabled for the datasets containing more than 50K images and enabled 1https : //github.com/NVlabs/stylegan2-ada-pytorch for others with default hyperparameters . In these experiments , we train for 25M real images shown to the discriminator . Each transfer experiment is performed with three independent runs , and the metrics are reported for the run corresponding to the median best FID ( Heusel et al. , 2017 ) . Metrics . In the experiments , we evaluate the performance via the four following metrics . ( 1 ) Frechet Inception Distance ( FID ) ( Heusel et al. , 2017 ) , which quantifies the discrepancy between the distributions of real and fake images , represented by deep embeddings . Both distributions are approximated by Gaussians , and the Wasserstein distance between them is computed . ( 2 ) Precision ( Kynkäänniemi et al. , 2019 ) , which measures the realism of fake images , assuming that the visual quality of a particular fake is high if it belongs to the neighborhood of some real images in the embedding space . ( 3 ) Recall ( Kynkäänniemi et al. , 2019 ) , which quantifies GAN diversity , measuring the rate of real images that belong to the neighborhood of some fake images in the embedding space . ( 4 ) Convergence rate equals a number of real images that were shown to the discriminator at the moment when the generator FID for the first time exceeded the optimal FID by at most 5 % . Intuitively , this metric quantifies how fast the learning process reaches a plateau . FID is computed based on the image embeddings extracted by the InceptionV3 model2 . Precision and Recall use the embeddings provided by the VGG-16 model3 . Precision and Recall are always computed with k=5 neighbors . For FID calculation , we always use all real images and 50.000 generated samples . For Precision/Recall calculation , we use the first 200K real images ( or less , if the real dataset is smaller ) and the same amount of generated samples . Results . The metric values for all datasets are reported in Table 2 , where each cell corresponds to a particular source-target pair . For the best ( in terms of FID ) checkpoint obtained for each sourcetarget transfer , we report the FID value ( top row in each cell ) , Precision and Recall ( the second and the third rows in each cell ) , and the convergence rate measured in millions of images ( bottom row in each cell ) . We highlight the sources that provide the best FID for each target dataset or differ from the best one by at most 5 % . We additionally present the curves of FID , Precision , and Recall values for several target datasets on Figure 9 and Figure 10 in the appendix . We describe the key observations from Table 2 below : • In terms of FID , a pretraining based on a diverse source ( e.g. , Imagenet or LSUN Dog ) is superior to training from scratch on all datasets in our experiments . • The choice of the source checkpoint significantly influences the coverage of the finetuned model , and the Recall values vary considerably for different sources , especially for smaller target datasets . For instance , on the Flowers dataset , their variability exceeds ten percent . In contrast , the Precision values are less affected by pretraining , and their typical variability is about 2−3 % . Figure 4 reports the standard deviations of Precision/Recall computed over different sources and highlights that Recall has higher variability compared to Precision , despite the latter having higher absolute values . • Pretraining considerably speeds up the optimization compared to the training from scratch . 2https : //nvlabs-fi-cdn.nvidia.com/stylegan2-ada-pytorch/pretrained/ metrics/inception-2015-12-05.pt 3https : //nvlabs-fi-cdn.nvidia.com/stylegan2-ada-pytorch/pretrained/ metrics/vgg16.pt Overall , despite having poor quality ( FID=49.2 ) , the Imagenet-pretrained unconditional StyleGAN2 model appears to be a superior GAN initialization that typically leads to more efficient optimization compared to alternatives . This result contradicts the observations in ( Wang et al. , 2018b ) showing that it is beneficial to transfer from dense and less diverse sources rather than sparse and diverse ones , like Imagenet . We attribute this inconsistency to the fact that ( Wang et al. , 2018b ) experimented with the WGAN-GP models , which are significantly inferior to the current state-of-the-art ones .

Summary: This paper aims to achieve a better understanding of GAN fine-tuning. A synthetic experiment demonstrates that pretrained discriminators improve the quality of the initial gradients, and pretrained generators help with improving mode coverage. Transfer learning experiments on real image datasets reveal that pretraining primarily improves mode coverage rather than sample fidelity, and that datasets containing a diverse set of images are best suited for transfer.