The Unusual Effectiveness of Averaging in GAN Training

1 INTRODUCTION . Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) are two-player zero-sum games . They are known to be notoriously hard to train , unstable , and often don ’ t converge . There has been a lot of recent work to improve the stability of GANs by addressing various underlying issues . Searching for a more stable function family ( Radford et al. , 2015 ) and utilizing better objectives ( Arjovsky et al. , 2017 ; Gulrajani et al. , 2017 ) have been explored for training and instability issues . For tackling non-convergence , regularizing objective functions ( Mescheder et al. , 2017 ; Salimans et al. , 2016 ) , two time-scale techniques ( Heusel et al. , 2017 ) and extra-gradient ( optimistic ) methods ( Daskalakis et al. , 2018 ; Mertikopoulos et al. , 2019 ) have been studied . One reason for non-convergence is cycling around an optimal solution ( Mertikopoulos et al. , 2018 ; Papadimitriou & Piliouras , 2018 ) , or even slow outward spiraling ( Bailey & Piliouras , 2018 ) . There might be multiple reasons for this cycling behavior : ( i ) eigenvalues of the Jacobian of the gradient vector field have zero real part and a large imaginary part ( Mescheder et al. , 2017 ) . ( ii ) Gradients of the discriminator may not move the optimization in the right direction and may need to be updated many times to do so ( Arjovsky et al. , 2017 ) . ( iii ) Sampling noise causes the discriminator to move in different directions at each iteration , as a result of which the generator fluctuates . ( iv ) Even if the gradients of the discriminator point in the right direction , the learning rate of the generator might be too large . When this overshooting happens , the gradients of the discriminator lead in the wrong direction due to over-confidence ( Mescheder et al. , 2018 ) . ∗ Corresponding Author : yasin001 @ e.ntu.edu.sg † Joint last authors 1 The code is available at https : //github.com/yasinyazici/EMA_GAN In this work , we explore in detail simple strategies for tackling the cycling behavior without influencing the adversarial game . Our strategies average generator parameters over time outside the training loop . Averaging generator and discriminator parameters is known to be an optimal solution for convex-concave min-max games ( Freund & Schapire , 1999 ) . However , no such guarantees are known ( even for bilinear games ) if we apply exponential discounting . Our contributions are the following : ( i ) We show theoretically that although EMA does not converge to equilibrium , even in bilinear games , it nevertheless helps to stabilize cyclic behavior by shrinking its amplitude . In non-bilinear settings it preserves the stability of locally stable fixed points . ( ii ) We demonstrate that both averaging techniques consistently improve results for several different datasets , network architectures , and GAN objectives . ( i ) We compare it with several other methods that try to alleviate the cycling or non-convergence problem and demonstrate its unusual effectiveness . 2 RELATED WORK . There have been attempts to improve stability and alleviate cycling issues arising in GANs . Salimans et al . ( 2016 ) use historical averaging of both generator and discriminator parameters as a regularization term in the objective function : each model deviating from its time average is penalized to improve stability . Ge et al . ( 2018 ) update the discriminator by using a mixture of historical generators . Heusel et al . ( 2017 ) use two time scale update rules to converge to a local equilibrium . Mescheder et al . ( 2017 ) identify the presence of eigenvalues of the Jacobian of the gradient vector with zero real part and large imaginary part as a reason for non-convergence . They address this issue by including squared gradients of the players to the objective . Extra-gradient ( optimistic ) methods help with convergence ( Daskalakis et al. , 2018 ; Mertikopoulos et al. , 2019 ) . Some of the above stated methods such as Salimans et al . ( 2016 ) ; Mescheder et al . ( 2017 ) influence the min-max game by introducing regularization terms into the objective function . Regularizing the adversarial objective can potentially change the optimal solution of the original objective . Our method on the other hand does not change the game dynamics , by keeping an average over iterates outside of the training loop . Averaging at maximum likelihood objective : Uniform Average and Exponential Moving Average over model parameters has been used for a long time ( Polyak & Juditsky , 1992 ; van den Oord et al. , 2017 ; Athiwaratkun et al. , 2018 ; Izmailov et al. , 2018 ) . However we would like to emphasize the difference between convex/non-convex optimization and saddle point optimization . GANs are an instance of the latter approach . There is no formal connection between the two approaches , so we can not extrapolate our understanding of averaging at maximum likelihood objectives to saddle point objectives . Averaging at minimax objective : Gidel et al . ( 2018 ) used uniform averaging similar to our work , however we show its shortcomings and how exponential decay averaging performs better . While this strategy has been used in ( Karras et al. , 2017 ) recently , their paper lacks any insights or a detailed analysis of this approach . 3 METHOD . In case of convex-concave min-max games , it is known that the average of generator/discriminator parameters is an optimal solution ( Freund & Schapire , 1999 ) , but there is no such guarantee for a non-convex/concave setting . Moving Average ( MA ) over parameters θ is an efficient implementation of uniform averaging without saving iterates at each time point : θ ( t ) MA = t− 1 t θ ( t−1 ) MA + 1 t θ ( t ) ( 1 ) When the generator iterates reaches a stationary distribution , Eq.1 approximates its mean . However there are two practical difficulties with this equation . First we do not know at which time point the iterates reach a stationary distribution . Second , and more importantly , iterates may not stay in the same distribution after a while . Averaging over samples coming from different distributions can produce worse results , which we have also observed in our own experiments . Because of the issues stated above , we use Exponential Moving Average ( EMA ) : θ ( t ) EMA = βθ ( t−1 ) EMA + ( 1− β ) θ ( t ) ( 2 ) where θ ( 0 ) EMA = θ ( 0 ) . EMA has an effective time window over iterates , where early iterates fade at a rate depending on the β value . As β approaches 1 , averaging effectively computes longer time windows . Both EMA and MA offer performance gains , with EMA typically being more robust ( for more details see Section 5 ) . Averaging methods are simple to implement and have minimal computation overhead . As they operate outside of the training loop , they do not influence optimal points of the game . Hyperparameter search for EMA can be done with a single training run by simultaneously keeping track of multiple averages with different β parameters . 4 WHY DOES AVERAGING WORK ? . 4.1 GLOBAL ANALYSIS FOR SIMPLE BILINEAR GAMES . As we will show experimentally , parameter averaging provides significant benefits in terms of GAN training across a wide range of different setups . Although one can not hope to completely justify this experimental success theoretically in general non-convex-concave settings , it is at least tempting to venture some informed conjectures about the root causes of this phenomenon . To do so , we focus on the simplest possible class of saddle point problems , the class of planar bilinear problems ( i.e . zero-sum games such as Matching Pennies ) . This is a toy , albeit classic , pedagogical example used to explain phenomena arising in GANs . Specifically , similar to Goodfellow ( 2017 ) , we also move to a continuous time model to simplify the mathematical analysis . In terms of the moving average method , it is well known that time-averaging suffices to lead to convergence to equilibria ( Freund & Schapire , 1999 ) . In fact , it has recently been established that the smooth ( continuous-time ) analogues of first order methods such as online gradient descent ( follow-the-regularized leader ) in bilinear zero-sum games are recurrent ( i.e . effectively periodic ) with trajectories cycling back into themselves ( Mertikopoulos et al. , 2018 ) . As a result , the timeaverage of the system converges fast , at a O ( 1/T ) rate , to the solution of the saddle point problem ( Mertikopoulos et al. , 2018 ) . These systems have connections to physics ( Bailey & Piliouras , 2019 ) . In contrast , not much is known about the behavior of EMA methods even in the simplest case of zero-sum games . As we will show by analyzing the simplest case of a bilinear saddle problem maxxminy xy , even in this toy case EMA does not lead to convergent behavior , but instead reduces the size of the oscillations , leading to more stable behavior whilst enabling exploration in a close neighborhood around the equilibrium . In this case the gradient descent dynamics are as follows : { dx dt = y dy dt = −x } ⇔ { dx dt = ∂H ∂y dy dt = − ∂H ∂x } This corresponds to a standard Hamiltonian system whose energy function is equal to H = x 2+y2 2 . All trajectories are cycles centered at 0 . In fact , the solution to the system has the form of a periodic orbit ( x ( t ) , y ( t ) ) = ( a cos ( t ) , a sin ( t ) ) . These connections have recently been exploited by Balduzzi et al . ( 2018 ) to design training algorithms for GANs ( see also Bailey & Piliouras ( 2019 ) ) . Here we take a different approach . For simplicity let ’ s assume a = 1 . The Exponential Moving Average method now corresponds to an integral of the form C · ∫ T 0 βT−tθ ( t ) dt where θ ( t ) = ( cos ( t ) , sin ( t ) ) and C a normalizing term such that the EMA of a constant function is equal to that constant ( i.e . C = 1/ ∫ T 0 βT−tdt ) . Some simple algebra implies that C · ∫ T 0 βT−t sin ( t ) dt = − ln ( β ) 1+ln2 ( β ) βT−cos ( T ) −ln ( β ) sin ( T ) 1−βT . As the terms β T vanish exponentially fast , this function reduces to ln ( β ) 1+ln2 ( β ) ( cos ( T ) + ln ( β ) sin ( T ) ) . This is a periodic function , similarly to the components of θ ( t ) , but its amplitude is significantly reduced , and as a result it is more concentrated around its mean value ( which is the same as its corresponding θ ( t ) component , i.e . zero ) . For β very close to 1 , e.g . β = 0.999 this periodic function can be further approximated by ln ( β ) cos ( T ) . In other words , no matter how close to 1 we choose the discounting factor β , even in the toy case of a planar bilinear saddle problem ( i.e . the Matching Pennies game ) EMA has a residual non-vanishing periodic component , alas of very small amplitude around the solution of the saddle point problem .

Summary: This paper tries to adapt the concept of averaging, well known is the game literature, to GAN training. In a simple min-max example the iterates obtained by gradient method do not converge to the equilibrium of the game but their average does. This work first provides intuitions on the potential benefits of exponential moving average (EMA) on a simple illustrative example and explore the effect of averaging on GAN.