LOGAN:  Latent Optimisation for Generative Adversarial Networks

1 INTRODUCTION . Generative Adversarial Nets ( GANs ) are implicit generative models that can be trained to match a given data distribution . GANs were originally proposed and demonstrated for images by Goodfellow et al . ( 2014 ) . As the field of generative modelling has advanced , GANs have remained at the frontier , generating high-fidelity images at large scale ( Brock et al. , 2018 ) . However , despite growing insights into the dynamics of GAN training , most recent advances in large-scale image generation come from architectural improvements ( Radford et al. , 2015 ; Zhang et al. , 2019 ) , or regularisation focusing on particular parts of the model ( Miyato et al. , 2018 ; Miyato & Koyama , 2018 ) . Inspired by the compressed sensing GAN ( CS-GAN ; Wu et al. , 2019 ) , we further exploit the benefit of latent optimisation in adversarial games using natural gradient descent to optimise the latent variable z at each step of training , presenting a scalable and easy to implement approach to improve the dynamical interaction between the discriminator and the generator . For clarity , we unify these approaches as latent optimised GANs ( LOGAN ) . To summarise our contributions : 1 . We present a novel analysis of latent optimisation in GANs from the perspective of differentiable games and stochastic approximation ( Balduzzi et al. , 2018 ; Heusel et al. , 2017 ) , arguing that latent optimisation can improve the dynamics of adversarial training . 2 . Motivated by this analysis , we improve latent optimisation by taking advantage of efficient second-order updates . 3 . Our algorithm improves the state-of-the-art BigGAN-deep model ( Brock et al. , 2018 ) by a significant margin , without introducing any architectural change or additional parameters , resulting in higher quality images and more diverse samples ( Figure 1 and 2 ) . 2 BACKGROUND . 2.1 NOTATION . We use θD and θG to denote the vectors representing parameters of the generator and discriminator . We use x for images , and z for the latent source generating an image . The prime ′ is used to denote a variable after one update step , e.g. , θ′D = θD − α ∂f ( z ; θD , θG ) ∂θD . p ( x ) and p ( z ) denote the data distribution and source distribution respectively . Ep ( x ) [ f ( x ) ] indicates taking the expectation of function f ( x ) over the distribution p ( x ) . 2.2 GENERATIVE ADVERSARIAL NETS . A GAN consists of a generator that generates image x = G ( z ; θG ) from a latent source z ∼ p ( z ) , and a discriminator that scores the generated images asD ( x ; θD ) ( Goodfellow et al. , 2014 ) . Training GANs involves an adversarial game : while the discriminator tries to distinguish generated samples x = G ( z ; θG ) from data x ∼ p ( x ) , the generator tries to fool the discriminator . This procedure can be summarised as the following min-max game : min θD max θG Ez∼p ( x ) [ hD ( D ( x ; θD ) ) ] + Ez∼p ( z ) [ hG ( D ( G ( z ; θG ) ; θD ) ) ] ( 1 ) The exact form of h ( · ) depends on the choice of loss function ( Goodfellow et al. , 2014 ; Arjovsky et al. , 2017 ; Nowozin et al. , 2016 ) . To simplify our presentation and analysis , we use the Wasserstein loss ( Arjovsky et al. , 2017 ) , so that hD ( t ) = −t and hG ( t ) = t. Our experiments with BigGANdeep uses the hinge loss ( Lim & Ye , 2017 ; Tran et al. , 2017 ) , which is identical to this form in its linear regime . Our analysis can be generalised to other losses as in previous theoretical work ( e.g. , Arora et al . 2017 ) . To simplify notation , we abbreviate f ( z ; θD , θG ) = D ( G ( z ; θG ) ; θD ) , which may be further simplified as f ( z ) when the explicit dependency on θD and θG can be omitted . Training GANs requires carefully balancing updates toD andG , and is sensitive to both architecture and algorithm choices ( Salimans et al. , 2016 ; Radford et al. , 2015 ) . A recent milestone is BigGAN ( and BigGAN-deep , Brock et al . 2018 ) , which pushed the boundary of high fidelity image generation by scaling up GANs to an unprecedented level . BigGANs use an architecture based on residual blocks ( He et al. , 2016 ) , in combination with regularisation mechanisms and self-attention ( Saxe et al. , 2014 ; Miyato et al. , 2018 ; Zhang et al. , 2019 ) . Here we aim to improve the adversarial dynamics during training . We focus on the second term in eq . 1 which is at the heart of the min-max game , with adversarial losses for D and G , which can be written as L ( z ) = [ LD ( z ) , LG ( z ) ] T = [ f ( z ) , −f ( z ) ] T ( 2 ) Computing the gradients with respect to θD and θG obtains the following gradient , which can not be expressed as the gradient of any single function ( Balduzzi et al. , 2018 ) : g = [ ∂LD ( z ) ∂θD , ∂LG ( z ) ∂θG ] T = [ ∂f ( z ) ∂θD , −∂f ( z ) ∂θG ] T ( 3 ) The fact that g is not the gradient of a function implies that gradient updates in GANs can exhibit cycling behaviour which can slow down or prevent convergence . In Balduzzi et al . ( 2018 ) , vector fields of this form are referred to as the simultaneous gradient . Although many GAN models use alternating update rules ( e.g. , Goodfellow et al . 2014 ; Brock et al . 2018 ) , following the gradient with respect to θD and θG alternatively in each step , they can still suffer from cycling , so we use the simpler simultaneous gradient ( eq . 3 ) for our analysis . 2.3 LATENT OPTIMISED GANS . Inspired by compressed sensing ( Candes et al. , 2006 ; Donoho , 2006 ) , Wu et al . ( 2019 ) introduced latent optimisation for GANs . We call this type of model latent-optimised GANs ( LOGAN ) . Latent optimization has been shown to improve the stability of training as well as the final performance for medium-sized models such as DCGANs and Spectral Normalised GANs ( Radford et al. , 2015 ; Miyato et al. , 2018 ) . Latent optimisation exploits knowledge from D to guide updates of z . Intuitively , the gradient∇zf ( z ) = ∂f ( z ) ∂z points in the direction that satisfies the discriminatorD , which implies better samples . Therefore , instead of using the randomly sampled z ∼ p ( z ) , Wu et al . ( 2019 ) uses the optimised latent ∆z = α ∂f ( z ) ∂z z′ = z + ∆z ( 4 ) in eq . 1 for training 1 . The general algorithm is summarised in Algorithm 1 and illustrated in Figure 3 a . We develop the natural gradient descent form of latent update in Section 4 . 1We use a single step of gradient-based optimisation during training , and justify this choice in section 3 . Algorithm 1 Latent Optimised GANs with Automatic Differentiation Input : data distribution p ( x ) , latent distribution p ( z ) , D ( · ; θD ) , G ( · ; θG ) , learning rate α , batch size N repeat Initialise discriminator and generator parameters θD , θG for i = 1 to N do Sample z ∼ p ( z ) , x ∼ p ( x ) Compute the gradient ∂D ( G ( z ) ) ∂z and use it to obtain ∆z from eq . 4 ( GD ) or eq . 12 ( NGD ) Optimise the latent z′ ← [ z + ∆z ] , [ · ] indicates clipping the value between −1 and 1 Compute generator loss L ( i ) G = −D ( G ( z′ ) ) Compute discriminator loss L ( i ) D = D ( G ( z ′ ) ) −D ( x ) end for Compute batch losses LG = 1N ∑N i=1 L ( i ) G and LD = 1 N ∑N i=1 L ( i ) D Update θD and θG with the gradients ∂LD∂θD , ∂LG ∂θG until reaches the maximum training steps 3 ANALYSIS OF THE ALGORITHM . To understand how latent optimisation improves GAN training , we analyse LOGAN as a 2-player differentiable game following Balduzzi et al . ( 2018 ) ; Gemp & Mahadevan ( 2018 ) ; Letcher et al . ( 2019 ) . The appendix provides a complementary analysis that relates LOGAN to unrolled GANs ( Metz et al. , 2016 ) and stochastic approximation ( Heusel et al. , 2017 ; Borkar , 1997 ) . 3.1 THE SYMPLECTIC GRADIENT ADJUSTMENT ( SGA ) . An important problem with gradient-based optimization in GANs is that the vector-field generated by the losses of the discriminator and generator is not a gradient vector field . It follows that gradient descent is not guaranteed to find a local optimum and can cycle , which can slow down convergence or lead to phenomena like mode collapse and mode hopping . Balduzzi et al . ( 2018 ) ; Gemp & Mahadevan ( 2018 ) proposed Symplectic Gradient Adjustment ( SGA ) to improve the dynamics of gradient-based methods in adversarial games . For a game with gradient g ( eq . 3 ) , define the Hessian as the second order derivatives with respect to the parameters , H = ∇θg . SGA uses the adjusted gradient g∗ = g + λAT g where λ is a positive constant ( 5 ) and A = 12 ( H − H T ) is the anti-symmetric component of the Hessian . Applying SGA to GANs yields the adjusted updates ( see Appendix B.1 for details ) : g∗ = [ ∂f ( z ) ∂θD + λ ( ∂2f ( z ) ∂θG ∂θD ) T ∂f ( z ) ∂θG , −∂f ( z ) ∂θG + λ ( ∂2f ( z ) ∂θD ∂θG ) T ∂f ( z ) ∂θD ] T ( 6 ) Compared with g in eq . 3 , the adjusted gradient g∗ has second-order terms reflecting the interactions between D and G. SGA has been shown to significantly improve GAN training in basic examples ( Balduzzi et al. , 2018 ) , allowing faster and more robust convergence to stable fixed points ( local Nash equilibria ) . Unfortunately , SGA is expensive to scale because computing the second-order derivatives with respect to all parameters is expensive . Explicitly computing the gradients for the discriminator and generator at z′ after one step of latent optimisation ( eq . 4 ) obtains [ dLD dθD , dLG dθG ] T = [ ∂f ( z′ ) ∂θD + ( ∂∆z ∂θD ) T ∂f ( z′ ) ∂∆z , −∂f ( z ′ ) ∂θG − ( ∂∆z ∂θG ) T ∂f ( z′ ) ∂∆z ] T ( 7 ) = [ ∂f ( z′ ) ∂θD + α ( ∂2f ( z ) ∂z∂θD ) T ∂f ( z′ ) ∂z′ , −∂f ( z ′ ) ∂θG − α ( ∂2f ( z ) ∂z∂θG ) T ∂f ( z′ ) ∂z′ ] T ( 8 ) In both equations , the first terms represent how f ( z′ ) depends on the parameters directly and the second terms represent how f ( z′ ) depends on the parameters via the optimised latent source . For the second equality , we substitute ∆z = α ∂f ( z ) ∂z as the gradient-based update of z and use ∂f ( z′ ) ∂∆z = ∂f ( z′ ) ∂z′ . The original GAN ’ s gradient ( eq . 3 ) does not include any second-order term , since ∆z = 0 without latent optimisation . In LOGAN , these extra terms are computed by automatic differentiation when back-propagating through the latent optimisation process ( see Algorithm 1 ) . The SGA updates in eq . 6 and the LOGAN updates in eq . 8 are strikingly similar , suggesting that the latent step used by LOGAN reduces the negative effects of cycling by introducing a symplectic gradient adjustment into the optimization procedure . The role of the latent step can be formalized in terms of a third player , whose goal is to help the generator , see appendix B for details . Crucially , latent optimisation approximates SGA using only second-order derivatives with respect to the latent z and parameters of the discriminator and generator separately . The second order terms involving parameters of both the discriminator and the generator – which are extremely expensive to compute – are not used . For latent z ’ s with dimensions typically used in GANs ( e.g. , 128–256 , orders of magnitude less than the number of parameters ) , these can be computed efficiently . In short , latent optimisation efficiently couples the gradients of the discriminator and generator , as prescribed by SGA , but using the much lower-dimensional latent source z which makes the adjustment scalable . An important consequence of reducing the rotational aspect of GAN dynamics is that it is possible to use larger step sizes during training which suggests using stronger optimisers to fully take advantage of latent optimisation . Latent optimisation can improve GAN training dynamics further by allowing larger single steps ∆z towards the direction of ∂f ( z ) ∂z without overshooting .

Summary: The paper proposes a novel training scheme for GANs, which leads to improved scores w.r.t. state-of-the-art. The idea is to update the sampled latent code in a direction improving the inner maximization in the min-max problem. The work considers both, the gradient direction and a direction motivated by the natural gradient which is shown to yield excellent performance. The overall scheme is motivated as an approximation to the (prohibitively expensive) symplectic gradient adjustment method.