On Self Modulation for Generative Adversarial Networks

1 INTRODUCTION . Generative Adversarial Networks ( GANs ) are a powerful class of generative models successfully applied to a variety of tasks such as image generation ( Zhang et al. , 2017 ; Miyato et al. , 2018 ; Karras et al. , 2017 ) , learned compression ( Tschannen et al. , 2018 ) , super-resolution ( Ledig et al. , 2017 ) , inpainting ( Pathak et al. , 2016 ) , and domain transfer ( Isola et al. , 2016 ; Zhu et al. , 2017 ) . Training GANs is a notoriously challenging task ( Goodfellow et al. , 2014 ; Arjovsky et al. , 2017 ; Lucic et al. , 2018 ) as one is searching in a high-dimensional parameter space for a Nash equilibrium of a non-convex game . As a practical remedy one applies ( usually a variant of ) stochastic gradient descent , which can be unstable and lack guarantees Salimans et al . ( 2016 ) . As a result , one of the main research challenges is to stabilize GAN training . Several approaches have been proposed , including varying the underlying divergence between the model and data distributions ( Arjovsky et al. , 2017 ; Mao et al. , 2016 ) , regularization and normalization schemes ( Gulrajani et al. , 2017 ; Miyato et al. , 2018 ) , optimization schedules ( Karras et al. , 2017 ) , and specific neural architectures ( Radford et al. , 2016 ; Zhang et al. , 2018 ) . A particularly successful approach is based on conditional generation ; where the generator ( and possibly discriminator ) are given side information , for example class labels Mirza & Osindero ( 2014 ) ; Odena et al . ( 2017 ) ; Miyato & Koyama ( 2018 ) . In fact , state-of-the-art conditional GANs inject side information via conditional batch normalization ( CBN ) layers ( De Vries et al. , 2017 ; Miyato & Koyama , 2018 ; Zhang et al. , 2018 ) . While this approach does help , a major drawback is that it requires external information , such as labels or embeddings , which is not always available . In this work we show that GANs benefit from self-modulation layers in the generator . Our approach is motivated by Feature-wise Linear Modulation in supervised learning ( Perez et al. , 2018 ; De Vries et al. , 2017 ) , with one key difference : instead of conditioning on external information , we condition on the generator ’ s own input . As self-modulation requires a simple change which is easily applicable to all popular generator architectures , we believe that is a useful addition to the GAN toolbox . Summary of contributions . We provide a simple yet effective technique that can added universally to yield better GANs . We demonstrate empirically that for a wide variety of settings ( loss ∗Work done at Google . 1Code at https : //github.com/google/compare_gan functions , regularizers and normalizers , neural architectures , and optimization settings ) that the proposed approach yields between a 5 % and 35 % improvement in sample quality . When using fixed hyperparameters settings our approach outperforms the baseline in 86 % ( 124/144 ) of cases . Further , we show that self-modulation still helps even if label information is available . Finally , we discuss the effects of this method in light of recently proposed diagnostic tools , generator conditioning ( Odena et al. , 2018 ) and precision/recall for generative models ( Sajjadi et al. , 2018 ) . 2 SELF-MODULATION FOR GENERATIVE ADVERSARIAL NETWORKS . Several recent works observe that conditioning the generative process on side information ( such as labels or class embeddings ) leads to improved models ( Mirza & Osindero , 2014 ; Odena et al. , 2017 ; Miyato & Koyama , 2018 ) . Two major approaches to conditioning on side information s have emerged : ( 1 ) Directly concatenate the side information swith the noise vector z ( Mirza & Osindero , 2014 ) , i.e . z′ = [ s , z ] . ( 2 ) Condition the hidden layers directly on s , which is usually instantiated via conditional batch normalization ( De Vries et al. , 2017 ; Miyato & Koyama , 2018 ) . Despite the success of conditional approaches , two concerns arise . The first is practical ; side information is often unavailable . The second is conceptual ; unsupervised models , such as GANs , seek to model data without labels . Including them side-steps the challenge and value of unsupervised learning . We propose self-modulating layers for the generator network . In these layers the hidden activations are modulated as a function of latent vector z . In particular , we apply modulation in a feature-wise fashion which allows the model to re-weight the feature maps as a function of the input . This is also motivated by the FiLM layer for supervised models ( Perez et al. , 2018 ; De Vries et al. , 2017 ) in which a similar mechanism is used to condition a supervised network on side information . Batch normalization ( Ioffe & Szegedy , 2015 ) can improve the training of deep neural nets , and it is widely used in both discriminative and generative modeling ( Szegedy et al. , 2015 ; Radford et al. , 2016 ; Miyato et al. , 2018 ) . It is thus present in most modern networks , and provides a convenient entry point for self-modulation . Therefore , we present our method in the context of its application via batch normalization . In batch normalization the activations of a layer , h , are transformed as h′ ` = γ h ` − µ σ + β , ( 1 ) where µ and σ2 are the estimated mean and variances of the features across the data , and γ and β are learnable scale and shift parameters . Self-modulation for unconditional ( without side information ) generation . In this case the proposed method replaces the non-adaptive parameters β and γ with input-dependent β ( z ) and γ ( z ) , respectively . These are parametrized by a neural network applied to the generator ’ s input ( Figure 1 ) . In particular , for layer ` , we compute h′ ` = γ ` ( z ) h ` − µ σ + β ` ( z ) ( 2 ) In general , it suffices that γ ` ( · ) and β ` ( · ) are differentiable . In this work , we use a small onehidden layer feed-forward network ( MLP ) with ReLU activation applied to the generator input z . Specifically , given parameter matrices U ( ` ) and V ( ` ) , and a bias vector b ( ` ) , we compute γ ` ( z ) = V ( ` ) max ( 0 , U ( ` ) z + b ( ` ) ) . We do the same for β ( z ) with independent parameters . Self-modulation for conditional ( with side information ) generation . Having access to side information proved to be useful for conditional generation . The use of labels in the generator ( and possibly discriminator ) was introduced by Mirza & Osindero ( 2014 ) and later adapted by Odena et al . ( 2017 ) ; Miyato & Koyama ( 2018 ) . In case that side information is available ( e.g . class labels y ) , it can be readily incorporated into the proposed method . This can be achieved by simply composing the information y with the input z ∈ Rd via some learnable function g , i.e . z′ = g ( y , z ) . In this work we opt for the simplest option and instantiate g as a bi-linear interaction between z and two trainable embedding functions E , E′ : Y → Rd of the class label y , as z′ = z + E ( y ) + z E′ ( y ) . ( 3 ) This conditionally composed z′ can be directly used in Equation 1 . Despite its simplicity , we demonstrate that it outperforms the standard conditional models . Discussion . Table 1 summarizes recent techniques for generator conditioning . While we choose to implement this approach via batch normalization , it can also operate independently by removing the normalization part in the Equation 1 . We made this pragmatic choice due to the fact that such conditioning is common ( Radford et al. , 2016 ; Miyato et al. , 2018 ; Miyato & Koyama , 2018 ) . The second question is whether one benefits from more complex modulation architectures , such as using an attention network ( Vaswani et al. , 2017 ) whereby β and γ could be made dependent on all upstream activations , or constraining the elements in γ to ( 0 , 1 ) which would yield a similar gating mechanism to an LSTM cell ( Hochreiter & Schmidhuber , 1997 ) . Based on initial experiments we concluded that this additional complexity does not yield a substantial increase in performance . 3 EXPERIMENTS . We perform a large-scale study of self-modulation to demonstrate that this method yields robust improvements in a variety of settings . We consider loss functions , architectures , discriminator regularization/normalization strategies , and a variety of hyperparameter settings collected from recent studies ( Radford et al. , 2016 ; Gulrajani et al. , 2017 ; Miyato et al. , 2018 ; Lucic et al. , 2018 ; Kurach et al. , 2018 ) . We study both unconditional ( without labels ) and conditional ( with labels ) generation . Finally , we analyze the results through the lens of the condition number of the generator ’ s Jacobian as suggested by Odena et al . ( 2018 ) , and precision and recall as defined in Sajjadi et al . ( 2018 ) . 3.1 EXPERIMENTAL SETTINGS . Loss functions . We consider two loss functions . The first one is the non-saturating loss proposed in Goodfellow et al . ( 2014 ) : VD ( G , D ) = Ex∼Pd ( x ) [ log σ ( D ( x ) ) ] + Ez∼P ( z ) [ log ( 1− σ ( D ( G ( z ) ) ) ) ] VG ( G , D ) = −Ez∼P ( z ) [ log σ ( D ( G ( z ) ) ) ] The second one is the hinge loss used in Miyato et al . ( 2018 ) : VD ( G , D ) = Ex∼Pd ( x ) [ min ( 0 , −1 +D ( x ) ) ] + Ez∼P ( z ) [ min ( 0 , −1−D ( G ( z ) ) ) ] VG ( G , D ) = −Ez∼P ( z ) [ D ( G ( z ) ) ] Controlling the Lipschitz constant of the discriminator . The discriminator ’ s Lipschitz constant is a central quantity analyzed in the GAN literature ( Miyato et al. , 2018 ; Zhou et al. , 2018 ) . We consider two state-of-the-art techniques : gradient penalty ( Gulrajani et al. , 2017 ) , and spectral normalization ( Miyato et al. , 2018 ) . Without normalization and regularization the models can perform poorly on some datasets . For the gradient penalty regularizer we consider regularization strength λ ∈ { 1 , 10 } . Network architecture . We use two popular architecture types : one based on DCGAN ( Radford et al. , 2016 ) , and another from Miyato et al . ( 2018 ) which incorporates residual connections ( He et al. , 2016 ) . The details can be found in the appendix . Optimization hyper-parameters . We train all models for 100k generator steps with the Adam optimizer ( Kingma & Ba , 2014 ) ( We also perform a subset of the studies with 500K steps and discuss it in . We test two popular settings of the Adam hyperparameters ( β1 , β2 ) : ( 0.5 , 0.999 ) and ( 0 , 0.9 ) . Previous studies find that multiple discriminator steps per generator step can help the training ( Goodfellow et al. , 2014 ; Salimans et al. , 2016 ) , thus we also consider both 1 and 2 discriminator steps per generator step2 . In total , this amounts to three different sets of hyperparameters for ( β1 , β2 , disc iter ) : ( 0 , 0.9 , 1 ) , ( 0 , 0.9 , 2 ) , ( 0.5 , 0.999 , 1 ) . We fix the learning rate to 0.0002 as in Miyato et al . ( 2018 ) . All models are trained with batch size of 64 on a single nVidia P100 GPU . We report the best performing model attained during the training period ; although the results follow the same pattern if the final model is report . Datasets . We consider four datasets : CIFAR10 , CELEBA-HQ , LSUN-BEDROOM , and IMAGENET . The LSUN-BEDROOM dataset ( Yu et al. , 2015 ) contains around 3M images . We partition the images randomly into a test set containing 30588 images and a train set containing the rest . CELEBAHQ contains 30k images ( Karras et al. , 2017 ) . We use the 128 × 128 × 3 version obtained by running the code provided by the authors3 . We use 3000 examples as the test set and the remaining examples as the training set . CIFAR10 contains 70K images ( 32 × 32 × 3 ) , partitioned into 60000 training instances and 10000 testing instances . Finally , we evaluate our method on IMAGENET , which contains 1.3M training images and 50K test images . We re-size the images to 128× 128× 3 as done in Miyato & Koyama ( 2018 ) and Zhang et al . ( 2018 ) . Metrics . Quantitative evaluation of generative models remains one of the most challenging tasks . This is particularly true in the context of implicit generative models where likelihood can not be effectively evaluated . Nevertheless , two quantitative measures have recently emerged : The Inception Score and the Frechet Inception Distance . While both of these scores have some drawbacks , they correlate well with scores assigned by human annotators and are somewhat robust . Inception Score ( IS ) ( Salimans et al. , 2016 ) posits that that the conditional label distribution p ( y|x ) of samples containing meaningful objects should have low entropy , while the marginal label distribution p ( y ) should have high entropy . Formally , IS ( G ) = exp ( Ex∼G [ dKL ( p ( y|x ) , p ( y ) ] ) . The score is computed using an Inception classifier ( Szegedy et al. , 2015 ) . Drawbacks of applying IS to model comparison are discussed in Barratt & Sharma ( 2018 ) . An alternative score , the Frechet Inception Distance ( FID ) , requires no labeled data ( Heusel et al. , 2017 ) . The real and generated samples are first embedded into a feature space ( using a specific layer of InceptionNet ) . Then , a multivariate Gaussian is fit each dataset and the distance is computed as FID ( x , g ) = ||µx − µg||22 + Tr ( Σx + Σg − 2 ( ΣxΣg ) 1 2 ) , where µ and Σ denote the empirical mean and covariance and subscripts x and g denote the true and generated data , respectively . FID was shown to be robust to various manipulations and sensitive to mode dropping ( Heusel et al. , 2017 ) . 2We also experimented with 5 steps which didn ’ t outperform the 2 step setting . 3Available at https : //github.com/tkarras/progressive_growing_of_gans .

Summary: The paper examines an architectural feature in GAN generators -- self-modulation -- and presents empirical evidence supporting the claim that it helps improve modeling performance. The self-modulation mechanism itself is implemented via FiLM layers applied to all convolutional blocks in the generator and whose scaling and shifting parameters are predicted as a function of the noise vector z. Performance is measured in terms of Fréchet Inception Distance (FID) for models trained with and without self-modulation on a fairly comprehensive range of model architectures (DCGAN-based, ResNet-based), discriminator regularization techniques (gradient penalty, spectral normalization), and datasets (CIFAR10, CelebA-HQ, LSUN-Bedroom, ImageNet). The takeaway is that self-modulation is an architectural feature that helps improve modeling performance by a significant margin in most settings. An ablation study is also performed on the location where self-modulation is applied, showing that it is beneficial across all locations but has more impact towards the later layers of the generator.