Projected GANs Converge Faster

1.5 5 10 15 20 25 . 1 Introduction . A Generative Adversarial Network ( GAN ) consists of a generator and a discriminator . For image synthesis , the generator ’ s task is to generate an RGB image ; the discriminator aims to distinguish real from fake samples . On closer inspection , the discriminator ’ s task is two-fold : First , it projects the real and fake samples into a meaningful space , i.e. , it learns a representation of the input space . Second , it discriminates based on this representation . Unfortunately , training the discriminator jointly with the generator is a notoriously hard task . While discriminator regularization techniques help to balance the adversarial game [ 31 ] , standard regularization methods like gradient penalties [ 36 ] are susceptible to hyperparameter choices [ 26 ] and can lead to a substantial decrease in performance [ 4 ] . In this paper , we explore the utility of pretrained representations to improve and stabilize GAN training . Using pretrained representations has become ubiquitous in computer vision [ 29 , 30 , 48 ] 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . and natural language processing [ 18 , 45 , 47 ] . While combining pretrained perceptual networks [ 58 ] with GANs for image-to-image translation has led to impressive results [ 14 , 49 , 59 , 64 ] , this idea has not yet materialized for unconditional noise-to-image synthesis . Indeed , we confirm that a naïve application of this idea does not lead to state-of-the-art results ( Section 4 ) as strong pretrained features enable the discriminator to dominate the two-player game , resulting in vanishing gradients for the generator [ 2 ] . In this work , we demonstrate how these challenges can be overcome and identify two key components for exploiting the full potential of pretrained perceptual feature spaces for GAN training : feature pyramids to enable multi-scale feedback with multiple discriminators and random projections to better utilize deeper layers of the pretrained network . We conduct extensive experiments on small and large datasets with a resolution of up to 10242 pixels . Across all datasets , we demonstrate state-of-the-art image synthesis results at significantly reduced training time ( Fig . 1 ) . We also find that Projected GANs increase data efficiency and avoid the need for additional regularization , rendering expensive hyperparameter sweeps unnecessary . Code , models , and supplementary videos can be found on the project page https : //sites.google.com/view/ projected-gan . 2 Related Work . We categorize related work into two main areas : pretraining for GANs and discriminator design . Pretrained Models for GAN Training . Work on leveraging pretrained representations for GANs can be divided into two categories : First , transferring parts of a GAN to a new dataset [ 15 , 38 , 65 , 71 ] and , second , using pretrained models to control and improve GANs . The latter is advantageous as pretraining does not need to be adversarial . Our work falls into this second category . Pretrained models can be used as a guiding mechanism to disentangle causal generative factors [ 54 ] , for text-driven image manipulation [ 44 ] , matching the generator activations to inverted classifiers [ 19 , 56 ] , or to generate images via gradient ascent in the latent space of a generator [ 41 ] . The non-adversarial approach of [ 53 ] learns generative models with moment matching in pretrained models ; however , the results remain far from competitive to standard GANs . An established method is the combination of adversarial and perceptual losses [ 21 ] . Commonly , the losses are combined additively [ 10 , 14 , 32 , 52 , 64 ] . Additive combination , however , is only possible if a reconstruction target is available , e.g. , in paired image-toimage translation settings [ 74 ] . Instead of providing the pretrained network with a reconstruction target , Sungatullina et al . [ 59 ] propose to optimize an adversarial loss on frozen VGG features [ 58 ] . They show that their approach improves CycleGAN [ 74 ] on image translation tasks . In a similar vein , [ 49 ] recently proposed a different perceptual discriminator . They utilize a pretrained VGG and connect its features with the prediction of a pretrained segmentation network . The combined features are fed into multiple discriminators at different scales . The two last approaches are specific to the image-toimage translation task . We demonstrate that these methods do not work well for the more challenging unconditional setting where the entire image content is synthesized from a random latent code . Discriminator Design . Much work on GANs focuses on novel generator architectures [ 4,26,27,69 ] , while the discriminator often remains close to a vanilla convolutional neural network or mirrors the generator . Notable exceptions are [ 55,70 ] which utilize an encoder-decoder discriminator architecture . However , in contrast to us , they neither use pretrained features nor random projections . A different line of work considers a setup with multiple discriminators , applied to either the generated RGB image [ 8 , 13 ] or low-dimensional projections thereof [ 1 , 40 ] . The use of several discriminators promises improved sample diversity , training speed , and training stability . However , these approaches are not utilized in current state-of-the-art systems because of diminishing returns compared to the increased computational effort . Providing multi-scale feedback with one or multiple discriminators has been helpful for both image synthesis [ 23 , 24 ] and image-to-image translation [ 43 , 64 ] . While these works interpolate the RGB image at different resolutions , our findings indicate the importance of multi-scale feature maps , showing parallels to the success of pyramid networks for object detection [ 34 ] . Lastly , to prevent overfitting of the discriminator , differentiable augmentation methods have recently been proposed [ 25 , 63 , 72 , 73 ] . We find that adopting these strategies helps exploit the full potential of pretrained representations for GAN training . 3 Projected GANs . GANs aim to model the distribution of a given training dataset . A generator G maps latent vectors z sampled from a simple distribution Pz ( typically a normal distribution ) to corresponding generated samples G ( z ) . The discriminator D then aims to distinguish real samples x ∼ Px from the generated samples G ( z ) ∼ PG ( z ) . This basic idea results in the following minimax objective min G max D ( Ex [ logD ( x ) ] + Ez [ log ( 1−D ( G ( z ) ) ) ] ) ( 1 ) We introduce a set of feature projectors { Pl } which map real and generated images to the discriminator ’ s input space . Projected GAN training can thus be formulated as follows min G max { Dl } ∑ l∈L ( Ex [ logDl ( Pl ( x ) ) ] + Ez [ log ( 1−Dl ( Pl ( G ( z ) ) ) ) ) ] ) ( 2 ) where { Dl } is a set of independent discriminators operating on different feature projections . Note that we keep { Pl } fixed in ( 2 ) and only optimize the parameters of G and { Dl } . The feature projectors { Pl } should satisfy two necessary conditions : they should be differentiable and provide sufficient statistics of their inputs , i.e. , they should preserve important information . Moreover , we aim to find feature projectors { Pl } which turn the ( difficult to optimize ) objective in ( 1 ) into an objective more amenable to gradient-based optimization . We now show that a projected GAN indeed matches the distribution in the projected feature space , before specifying the details of our feature projectors . 3.1 Consistency . The projected GAN objective in ( 2 ) no longer optimizes directly to match the true distribution PT . To understand the training properties under ideal conditions , we consider a more generalized form of the consistency theorem of [ 40 ] : Theorem 1 . Let PT denote the density of the true data distribution and PG the density of the distribution the Generator G produces . Let Pl ◦ T and Pl ◦G be the functional composition of the differentiable and fixed function Pl and the true/generated data distribution , and y be the transformed input to the discriminator . For a fixed G , the optimal discriminators are given by D∗l , G ( y ) = PPl◦T ( y ) PPl◦T ( y ) + PPl◦G ( y ) for all l ∈ L. In this case , the optimal G under ( 2 ) is achieved iff PPl◦T = PPl◦G for all l ∈ L. A proof of the theorem is provided in the appendix . From the theorem , we conclude that a feature projector Pl with its associated discriminator Dl encourages the generator to match the true distribution along the marginal through Pl . Therefore , at convergence , G matches the generated and true distributions in feature space . The theorem also holds when using stochastic data augmentations [ 25 ] before the deterministic projections Pl . 3.2 Model Overview . Projecting to and training in pretrained feature spaces opens up a realm of new questions which we address below . This section will provide an overview of the general system and is followed by extensive ablations of each design choice . As our feature projections affect the discriminator , we focus on Pl and Dl in this section and postpone the discussion of generator architectures to Section 5 . Multi-Scale Discriminators . We obtain features from four layers Ll of a pretrained feature network F at resolutions ( L1 = 642 , L2 = 322 , L3 = 162 , L4 = 82 ) . We associate a separate discriminator Dl with the features at layer Ll , respectively . Each discriminator Dl uses a simple convolutional architecture with spectral normalization [ 37 ] at each convolutional layer . We observe better performance if all discriminators output logits at the same resolution ( 42 ) . Accordingly , we use fewer down-sampling blocks for lower resolution inputs . Following common practice , we sum all logits for computing the overall loss . For the generator pass , we sum the losses of all discriminators . More complex strategies [ 1 , 13 ] did not improve performance in our experiments . Random Projections . We observe that features at deeper layers are significantly harder to cover , as evidenced by our experiments in Section 4 . We hypothesize that a discriminator can focus on a subset of the feature space while wholly disregarding other parts . This problem might be especially prominent in the deeper , more semantic layers . Therefore , we propose two different strategies to dilute prominent features , encouraging the discriminator to utilize all available information equally . Common to both strategies is that they mix features using differentiable random projections which are fixed , i.e. , after random initialization , the parameters of these layers are not trained . Cross-Channel Mixing ( CCM ) . Empirically , we found two properties to be desirable : ( i ) the random projection should be information preserving to leverage the full representational power of F , and ( ii ) it should not be trivially invertible . The easiest way to mix across channels is a 1×1 convolution . A 1×1 convolution with an equal number of output and input channels is a generalization of a permutation [ 28 ] and consequently preserves information about its input . In practice , we find that more output channels lead to better performance as the mapping remains injective and therefore information preserving . Kingma et al . [ 28 ] initialize their convolutional layers as a random rotation matrix as a good starting point for optimization . We do not find this to improve GAN performance ( see Appendix ) , arguably since it violates ( ii ) . We therefore randomly initialize the weights of the convolutional layer via Kaiming initialization [ 16 ] . Note that we do not add any activation functions . We apply this random projection at each of the four scales and feed the transformed feature to the discriminator as depicted in Fig . 2 . Cross-Scale Mixing ( CSM ) . To encourage feature mixing across scales , CSM extends CCM with random 3×3 convolutions and bilinear upsampling , yielding a U-Net [ 50 ] architecture , see Fig . 3 . However , our CSM block is simpler than a vanilla U-Net [ 50 ] : we only use a single convolutional layer at each scale . As for CCM , we utilize Kaiming initialization for all weights . Pretrained Feature Networks . We ablate over varying feature networks . First , we investigate different versions of EfficientNets , which allow for direct control over model size versus performance . EfficientNets are image classification models trained on ImageNet [ 7 ] and designed to provide favorable accuracy-compute tradeoffs . Second , we use ResNets of varying sizes . To analyze the dependency on ImageNet features ( Section 4.3 ) , we also consider R50-CLIP [ 46 ] , a ResNet optimized with a contrastive language-image objective on a dataset of 400 million ( image , text ) pairs . Lastly , we utilize a vision transformer architecture ( ViTBase ) [ 9 ] and its efficient follow-up ( DeiT-small distilled ) [ 62 ] . We do not choose an inception network [ 60 ] to avoid strong correlations with the evaluation metric FID [ 17 ] . In the appendix , we also evaluate several other neural and non-neural metrics to rule out correlations . These additional metrics reflect the rankings obtained by FID . In the following , we conduct a systematic ablation study to analyze the importance and best configuration of each component in our Projected GAN model , before comparing it to the state-of-the-art .

Summary: The papers proposes a technique for faster training of GANs while achieving state-of-the-art FID scores on several datasets. The main idea is to use pre-trained features for the discriminator. Four improvements are recommended: 1) Multi-scale discriminators, 2) cross-channel mixing 3) cross-scale mixing 4) choice of pre-trained network.  FID scores are shown for several datasets, CLEVR, FFHQ, Cityscapes, Bedroom and Church.