On Self Modulation for Generative Adversarial Networks

1 INTRODUCTION . Generative Adversarial Networks ( GANs ) are a powerful class of generative models successfully applied to a variety of tasks such as image generation ( Zhang et al. , 2017 ; Miyato et al. , 2018 ; Karras et al. , 2017 ) , learned compression ( Tschannen et al. , 2018 ) , super-resolution ( Ledig et al. , 2017 ) , inpainting ( Pathak et al. , 2016 ) , and domain transfer ( Isola et al. , 2016 ; Zhu et al. , 2017 ) . Training GANs is a notoriously challenging task ( Goodfellow et al. , 2014 ; Arjovsky et al. , 2017 ; Lucic et al. , 2018 ) as one is searching in a high-dimensional parameter space for a Nash equilibrium of a non-convex game . As a practical remedy one applies ( usually a variant of ) stochastic gradient descent , which can be unstable and lack guarantees Salimans et al . ( 2016 ) . As a result , one of the main research challenges is to stabilize GAN training . Several approaches have been proposed , including varying the underlying divergence between the model and data distributions ( Arjovsky et al. , 2017 ; Mao et al. , 2016 ) , regularization and normalization schemes ( Gulrajani et al. , 2017 ; Miyato et al. , 2018 ) , optimization schedules ( Karras et al. , 2017 ) , and specific neural architectures ( Radford et al. , 2016 ; Zhang et al. , 2018 ) . A particularly successful approach is based on conditional generation ; where the generator ( and possibly discriminator ) are given side information , for example class labels Mirza & Osindero ( 2014 ) ; Odena et al . ( 2017 ) ; Miyato & Koyama ( 2018 ) . In fact , state-of-the-art conditional GANs inject side information via conditional batch normalization ( CBN ) layers ( De Vries et al. , 2017 ; Miyato & Koyama , 2018 ; Zhang et al. , 2018 ) . While this approach does help , a major drawback is that it requires external information , such as labels or embeddings , which is not always available . In this work we show that GANs benefit from self-modulation layers in the generator . Our approach is motivated by Feature-wise Linear Modulation in supervised learning ( Perez et al. , 2018 ; De Vries et al. , 2017 ) , with one key difference : instead of conditioning on external information , we condition on the generator ’ s own input . As self-modulation requires a simple change which is easily applicable to all popular generator architectures , we believe that is a useful addition to the GAN toolbox . Summary of contributions . We provide a simple yet effective technique that can added universally to yield better GANs . We demonstrate empirically that for a wide variety of settings ( loss ∗Work done at Google . 1Code at https : //github.com/google/compare_gan functions , regularizers and normalizers , neural architectures , and optimization settings ) that the proposed approach yields between a 5 % and 35 % improvement in sample quality . When using fixed hyperparameters settings our approach outperforms the baseline in 86 % ( 124/144 ) of cases . Further , we show that self-modulation still helps even if label information is available . Finally , we discuss the effects of this method in light of recently proposed diagnostic tools , generator conditioning ( Odena et al. , 2018 ) and precision/recall for generative models ( Sajjadi et al. , 2018 ) . 2 SELF-MODULATION FOR GENERATIVE ADVERSARIAL NETWORKS . Several recent works observe that conditioning the generative process on side information ( such as labels or class embeddings ) leads to improved models ( Mirza & Osindero , 2014 ; Odena et al. , 2017 ; Miyato & Koyama , 2018 ) . Two major approaches to conditioning on side information s have emerged : ( 1 ) Directly concatenate the side information swith the noise vector z ( Mirza & Osindero , 2014 ) , i.e . z′ = [ s , z ] . ( 2 ) Condition the hidden layers directly on s , which is usually instantiated via conditional batch normalization ( De Vries et al. , 2017 ; Miyato & Koyama , 2018 ) . Despite the success of conditional approaches , two concerns arise . The first is practical ; side information is often unavailable . The second is conceptual ; unsupervised models , such as GANs , seek to model data without labels . Including them side-steps the challenge and value of unsupervised learning . We propose self-modulating layers for the generator network . In these layers the hidden activations are modulated as a function of latent vector z . In particular , we apply modulation in a feature-wise fashion which allows the model to re-weight the feature maps as a function of the input . This is also motivated by the FiLM layer for supervised models ( Perez et al. , 2018 ; De Vries et al. , 2017 ) in which a similar mechanism is used to condition a supervised network on side information . Batch normalization ( Ioffe & Szegedy , 2015 ) can improve the training of deep neural nets , and it is widely used in both discriminative and generative modeling ( Szegedy et al. , 2015 ; Radford et al. , 2016 ; Miyato et al. , 2018 ) . It is thus present in most modern networks , and provides a convenient entry point for self-modulation . Therefore , we present our method in the context of its application via batch normalization . In batch normalization the activations of a layer , h , are transformed as h′ ` = γ h ` − µ σ + β , ( 1 ) where µ and σ2 are the estimated mean and variances of the features across the data , and γ and β are learnable scale and shift parameters . Self-modulation for unconditional ( without side information ) generation . In this case the proposed method replaces the non-adaptive parameters β and γ with input-dependent β ( z ) and γ ( z ) , respectively . These are parametrized by a neural network applied to the generator ’ s input ( Figure 1 ) . In particular , for layer ` , we compute h′ ` = γ ` ( z ) h ` − µ σ + β ` ( z ) ( 2 ) In general , it suffices that γ ` ( · ) and β ` ( · ) are differentiable . In this work , we use a small onehidden layer feed-forward network ( MLP ) with ReLU activation applied to the generator input z . Specifically , given parameter matrices U ( ` ) and V ( ` ) , and a bias vector b ( ` ) , we compute γ ` ( z ) = V ( ` ) max ( 0 , U ( ` ) z + b ( ` ) ) . We do the same for β ( z ) with independent parameters . Self-modulation for conditional ( with side information ) generation . Having access to side information proved to be useful for conditional generation . The use of labels in the generator ( and possibly discriminator ) was introduced by Mirza & Osindero ( 2014 ) and later adapted by Odena et al . ( 2017 ) ; Miyato & Koyama ( 2018 ) . In case that side information is available ( e.g . class labels y ) , it can be readily incorporated into the proposed method . This can be achieved by simply composing the information y with the input z ∈ Rd via some learnable function g , i.e . z′ = g ( y , z ) . In this work we opt for the simplest option and instantiate g as a bi-linear interaction between z and two trainable embedding functions E , E′ : Y → Rd of the class label y , as z′ = z + E ( y ) + z E′ ( y ) . ( 3 ) This conditionally composed z′ can be directly used in Equation 1 . Despite its simplicity , we demonstrate that it outperforms the standard conditional models . Discussion . Table 1 summarizes recent techniques for generator conditioning . While we choose to implement this approach via batch normalization , it can also operate independently by removing the normalization part in the Equation 1 . We made this pragmatic choice due to the fact that such conditioning is common ( Radford et al. , 2016 ; Miyato et al. , 2018 ; Miyato & Koyama , 2018 ) . The second question is whether one benefits from more complex modulation architectures , such as using an attention network ( Vaswani et al. , 2017 ) whereby β and γ could be made dependent on all upstream activations , or constraining the elements in γ to ( 0 , 1 ) which would yield a similar gating mechanism to an LSTM cell ( Hochreiter & Schmidhuber , 1997 ) . Based on initial experiments we concluded that this additional complexity does not yield a substantial increase in performance . 3 EXPERIMENTS . We perform a large-scale study of self-modulation to demonstrate that this method yields robust improvements in a variety of settings . We consider loss functions , architectures , discriminator regularization/normalization strategies , and a variety of hyperparameter settings collected from recent studies ( Radford et al. , 2016 ; Gulrajani et al. , 2017 ; Miyato et al. , 2018 ; Lucic et al. , 2018 ; Kurach et al. , 2018 ) . We study both unconditional ( without labels ) and conditional ( with labels ) generation . Finally , we analyze the results through the lens of the condition number of the generator ’ s Jacobian as suggested by Odena et al . ( 2018 ) , and precision and recall as defined in Sajjadi et al . ( 2018 ) . 3.1 EXPERIMENTAL SETTINGS . Loss functions . We consider two loss functions . The first one is the non-saturating loss proposed in Goodfellow et al . ( 2014 ) : VD ( G , D ) = Ex∼Pd ( x ) [ log σ ( D ( x ) ) ] + Ez∼P ( z ) [ log ( 1− σ ( D ( G ( z ) ) ) ) ] VG ( G , D ) = −Ez∼P ( z ) [ log σ ( D ( G ( z ) ) ) ] The second one is the hinge loss used in Miyato et al . ( 2018 ) : VD ( G , D ) = Ex∼Pd ( x ) [ min ( 0 , −1 +D ( x ) ) ] + Ez∼P ( z ) [ min ( 0 , −1−D ( G ( z ) ) ) ] VG ( G , D ) = −Ez∼P ( z ) [ D ( G ( z ) ) ] Controlling the Lipschitz constant of the discriminator . The discriminator ’ s Lipschitz constant is a central quantity analyzed in the GAN literature ( Miyato et al. , 2018 ; Zhou et al. , 2018 ) . We consider two state-of-the-art techniques : gradient penalty ( Gulrajani et al. , 2017 ) , and spectral normalization ( Miyato et al. , 2018 ) . Without normalization and regularization the models can perform poorly on some datasets . For the gradient penalty regularizer we consider regularization strength λ ∈ { 1 , 10 } . Network architecture . We use two popular architecture types : one based on DCGAN ( Radford et al. , 2016 ) , and another from Miyato et al . ( 2018 ) which incorporates residual connections ( He et al. , 2016 ) . The details can be found in the appendix . Optimization hyper-parameters . We train all models for 100k generator steps with the Adam optimizer ( Kingma & Ba , 2014 ) ( We also perform a subset of the studies with 500K steps and discuss it in . We test two popular settings of the Adam hyperparameters ( β1 , β2 ) : ( 0.5 , 0.999 ) and ( 0 , 0.9 ) . Previous studies find that multiple discriminator steps per generator step can help the training ( Goodfellow et al. , 2014 ; Salimans et al. , 2016 ) , thus we also consider both 1 and 2 discriminator steps per generator step2 . In total , this amounts to three different sets of hyperparameters for ( β1 , β2 , disc iter ) : ( 0 , 0.9 , 1 ) , ( 0 , 0.9 , 2 ) , ( 0.5 , 0.999 , 1 ) . We fix the learning rate to 0.0002 as in Miyato et al . ( 2018 ) . All models are trained with batch size of 64 on a single nVidia P100 GPU . We report the best performing model attained during the training period ; although the results follow the same pattern if the final model is report . Datasets . We consider four datasets : CIFAR10 , CELEBA-HQ , LSUN-BEDROOM , and IMAGENET . The LSUN-BEDROOM dataset ( Yu et al. , 2015 ) contains around 3M images . We partition the images randomly into a test set containing 30588 images and a train set containing the rest . CELEBAHQ contains 30k images ( Karras et al. , 2017 ) . We use the 128 × 128 × 3 version obtained by running the code provided by the authors3 . We use 3000 examples as the test set and the remaining examples as the training set . CIFAR10 contains 70K images ( 32 × 32 × 3 ) , partitioned into 60000 training instances and 10000 testing instances . Finally , we evaluate our method on IMAGENET , which contains 1.3M training images and 50K test images . We re-size the images to 128× 128× 3 as done in Miyato & Koyama ( 2018 ) and Zhang et al . ( 2018 ) . Metrics . Quantitative evaluation of generative models remains one of the most challenging tasks . This is particularly true in the context of implicit generative models where likelihood can not be effectively evaluated . Nevertheless , two quantitative measures have recently emerged : The Inception Score and the Frechet Inception Distance . While both of these scores have some drawbacks , they correlate well with scores assigned by human annotators and are somewhat robust . Inception Score ( IS ) ( Salimans et al. , 2016 ) posits that that the conditional label distribution p ( y|x ) of samples containing meaningful objects should have low entropy , while the marginal label distribution p ( y ) should have high entropy . Formally , IS ( G ) = exp ( Ex∼G [ dKL ( p ( y|x ) , p ( y ) ] ) . The score is computed using an Inception classifier ( Szegedy et al. , 2015 ) . Drawbacks of applying IS to model comparison are discussed in Barratt & Sharma ( 2018 ) . An alternative score , the Frechet Inception Distance ( FID ) , requires no labeled data ( Heusel et al. , 2017 ) . The real and generated samples are first embedded into a feature space ( using a specific layer of InceptionNet ) . Then , a multivariate Gaussian is fit each dataset and the distance is computed as FID ( x , g ) = ||µx − µg||22 + Tr ( Σx + Σg − 2 ( ΣxΣg ) 1 2 ) , where µ and Σ denote the empirical mean and covariance and subscripts x and g denote the true and generated data , respectively . FID was shown to be robust to various manipulations and sensitive to mode dropping ( Heusel et al. , 2017 ) . 2We also experimented with 5 steps which didn ’ t outperform the 2 step setting . 3Available at https : //github.com/tkarras/progressive_growing_of_gans .

Summary: The manuscript proposes a modification of generators in GANs which improves performance under two popular metrics for multiple architectures, loss, benchmarks, regularizers, and hyperparameter settings. Using the conditional batch normalization mechanism, the input noise vector is allowed to modulate layers of the generator. As this modulation only depends on the noise vector, this technique does not require additional annotations. In addition to the extensive experimentation on different settings showing performance improvements, the authors also present an ablation study, that shows the impact of the method when applied to different layers.