Sample weighting as an explanation for mode collapse in generative adversarial networks

1 INTRODUCTION . Generative adversarial networks have come a long way since their introduction ( Goodfellow et al. , 2014 ) and are currently state of the art for some tasks , such as generating images . A combination of deep learning developments , GAN specific advances and vast improvements in data sets and computational resources have enabled GANs to generate high resolution images that require some effort to distinguish from real photos ( Zhang et al. , 2018 ; Brock et al. , 2018 ; Karras et al. , 2018 ) . GANs use two competing networks : a generator G that maps input noise to samples mimicking real data , and a discriminator D that outputs estimated probabilities of samples being real rather than generated by G. We summarize their cost functions , JD and JG , for the minimax and nonsaturating formulations introduced in Goodfellow et al . ( 2014 ) . We denote samples from real data and noise distributions by x and z and omit the proper expectation value formalism : JDMM ( x , z ) = JDNS ( x , z ) = − log ( Dp ( x ) ) − log ( 1−Dp ( G ( z ) ) ) JGMM ( z ) = log ( 1−Dp ( G ( z ) ) ) JGNS ( z ) = − log ( Dp ( G ( z ) ) ( 1 ) For clarity , we use subscripts to distinguish between the discriminator ’ s pre-activation logit output Dl and the probability representation Dp : Dp ≡ ( 1 + exp ( −Dl ) ) −1 ( 2 ) Both formulations have the same cost function for D , representing the cross entropy between probability estimates and ground truth . In the minimax formulation ( MM-GAN ) , G is simply trained to maximize D ’ s cost . Ideally , G matches its outputs to the real data distribution while also achieving meaningful generalization , but many failure modes are observed in practice . NS-GAN uses a modified cost for G that is non-saturating when D distinguishes real and generated data with very high confidence , such that G ’ s gradients do not vanish . ( Supplementary : C ) Various publications establish what the different cost functions optimize in terms of the JensenShannon and reverse Kullback-Leibler divergences between real and generated data : JGMM ⇔ 2 · DJS ( Goodfellow et al. , 2014 ) JGMM + JGNS ⇔ DRKL ( Huszár , 2016 ) JGNS ⇔ DRKL − 2 · DJS ( Arjovsky & Bottou , 2017 ) ( 3 ) Huszár ( 2015 ) and Arjovsky & Bottou ( 2017 ) have suggested NS-GAN ’ s divergence as an explanation for the ubiquitous mode dropping and mode collapsing problems with GANs ( Metz et al. , 2016 ; Salimans et al. , 2016 ; Srivastava et al. , 2017 ) . While MM-GAN seems promising in terms of its Jensen-Shannon divergence , the formulation has largely been ignored because the saturating cost causes training to break down . A variety of other GAN formulations have been introduced , such as WGAN-GP ( Arjovsky et al. , 2017 ; Gulrajani et al. , 2017 ) , LS-GAN ( Mao et al. , 2016 ) and Hinge-GAN ( Miyato et al. , 2018 ) . Lucic et al . ( 2018 ) finds that different cost formulations tend to get similar results given sufficient parameter tuning , including various forms of regularization . Despite the questionable form of NSGAN in terms of divergences , it is widely used and can produce very impressive results , such as in the improved StyleGAN ( Karras et al. , 2019 ) . 2 THEORY 2.1 MM-GAN SATURATION The parameters of a network are typically trained with some form of gradient descent on a cost function . We find the expressions for D ’ s and G ’ s gradients with respect to their parameters , φ and θ : ( Supplementary : F ) ∇φJDMM , NS = + ∂Dl ( G ( z ) , φ ) ∂φ ·Dp ( G ( z ) , φ ) + ∂Dl ( x , φ ) ∂φ · ( 1−Dp ( x , φ ) ) ∇θJGMM = − ∂Dl ( G ( z , θ ) ) ∂θ ·Dp ( G ( z , θ ) ) ∇θJGNS = − ∂Dl ( G ( z , θ ) ) ∂θ · ( 1−Dp ( G ( z , θ ) ) ) ( 4 ) We emphasize the two kinds of scaling factors for these gradients in red and blue : they are plotted in figure 2 . The discriminator ’ s scaling factors decrease as it minimizes its cost , approaching 0 towards the optima for both the real and the generated data term . The minimax formulation JG = −JD is suitable for adversarial training in terms of the generator ’ s optimum , but the unchanged scaling factor means that G ’ s gradients increase towards and decrease away from its optimum . The saturation effect described in Goodfellow et al . ( 2014 ) is that limDp ( G ( z ) ) →0∇θJGMM = 0 , such that G stops training when D is highly confident that its samples are fake . More generally , the scaling factor makes JG concave with respect to Dl , which interacts poorly with common optimization methods ( see section 2.4 ) . As∇θJGMM and∇θJGNS are the same aside from their scaling factors , the different behaviors of the two formulations must follow from these . NS-GAN ’ s scaling factor avoids saturation , but gives rise to a different , more subtle mode dropping tendency ( see section 2.3 ) . 2.2 NON-SATURATION AND SAMPLE WEIGHTING . As can be seen from eq 4 , the NS-GAN and MM-GAN gradients are parallel for a single sample , but with different magnitudes . Stochastic gradient descent estimates the gradient of the cost over the entire input distribution by using a number of samples ( a minibatch ) . We can express the NS-GAN minibatch gradient in terms of the MM-GAN gradient : ∇θJbatchGNS = N−1∑ i=0 ∇θJ sampleGMM ( zi ) [ 1−Dp ( G ( zi ) ) Dp ( G ( zi ) ) ] ( 5 ) Due to the bracketed factor , NS-GAN rescales the contribution from each sample relative to MMGAN , implicitly emphasizing samples with smaller values of Dp . Seeing as saturation is caused by the gradient ’ s vanishing magnitude , this additional effect on the gradient ’ s direction is questionable . The exact ratio of the minibatch gradient magnitudes for NS-GAN and MM-GAN depends on ∂/∂θ ( Dl ( G ( z ) ) ) for each sample and has no convenient expression . We can approximate it by replacing Dp ( G ( zi ) ) in eq 5 with its mean over the minibatch , Dp = 1N ∑N−1 i=0 Dp ( G ( zi ) ) . This allows us to formulate a form of non-saturation for MM-GAN that mimicks NS-GAN : ∇θJbatchGMM-nsat = 1−Dp Dp N−1∑ i=0 ∇θJ sampleGMM ( zi ) ( 6 ) We refer to the formulation with this generator gradient as MM-nsat . The relative weights of samples in each batch are as for MM-GAN , while the gradient magnitude approximates that of NS-GAN . Note , however , that the relative weights of samples may be disturbed across batches , such as when the minibatch size is small and Dp fluctuates . Despite different theoretical motivation , MM-nsat is very closely related to importance weighted NS-GAN ( Hu et al. , 2017 ) . ( Supplementary : D ) 2.3 SAMPLE WEIGHTING AND MODE DROPPING . If we use r ( x ) and g ( x ) to denote the density of real and generated samples at a point x in data space , the optimal discriminator ( i.e . the function that minimizes JDMM , NS ) is given by : Doptp ( x ) = r ( x ) r ( x ) + g ( x ) ( Goodfellow et al. , 2014 ) ( 7 ) This expression for the optimial discriminator assumes idealized conditions : that D is optimized for a fixed G , with unlimited capacity and without having to estimate the underlying real and generated data distributions by finite sampling . While Dopt is not realized in practice ( Sinn & Rawat , 2018 ) , we use it to form some intuitions about D ’ s behaviors . Suppose we have real data with two disjunct , equiprobable modes , and that one of these modes , O , is overrepresented in generated data . For convergence , G would need to shift probability mass from O to the underrepresented mode , U . However , the minibatches used to update G will have more samples from O , simply because they are generated more often . For a strong discriminator , these samples will also tend towards smaller values of Dp ( G ( z ) ) , due to equation 7 . This in turn causes them to be weighted differently by NS-GAN and MM-GAN , because Dp ( G ( z ) ) appears in their scaling factors ( eq 4 ) . We refer to the first effect as over- and underrepresentation ( of generated samples relative to real samples ) and the second effect as up- and downweighting ( governed on the scaling factor ) . The fundamental problem with NS-GAN can be seen by looking at how these two effects interact . The NS-GAN generator has a scaling factor 1−Dp ( G ( z ) ) , which combines overepresentation with upweighting and underrepresentation with downweighting , allowing an overrpresented mode O to dominate U ’ s contributions to the parameter updates . If parameter updates overwhelmingly based on gradients from O have an adverse effect on G ’ s ability to generate samples from U , this mode may become increasingly underrepresented , making O yet more dominant . MM-GAN ’ s scaling factor has the reverse behavior , pairing overrepresentation with downweighting and underrepresentation with upweighting , such that the two effects combine in a stabilizing rather than destabilizing way . For the extreme example of a nearly dropped or newly discovered mode , r ( x ) g ( x ) such that we expect Dp ( x ) ≈ 1 for a strong discriminator . NS-GAN ’ s sample weighting disregards gradients from such samples , whereas MM-GAN ’ s sample weighting emphasizes them . This difference between MM-GAN and NS-GAN in terms of scaling factors reflects the different divergences they have been shown to optimize ( eq 3 ) . Huszár ( 2015 ) and Arjovsky et al . ( 2017 ) both connect the mode dropping tendencies of NS-GAN to its reverse Kullback-Leibler divergence , which strongly penalizes G for generating samples outside of the real data modes , but not for dropping modes . A variety of attempts to address mode dropping , mode collapse and mode hopping in GANs seem unaware that this is reasonable behavior for NS-GAN given its divergence : mode dropping minimizes DRKL and maximizes DJS . 2.4 MM-GAN INTERACTION WITH ADAM GANs are generally trained with the Adam optimizer ( Kingma & Ba , 2014 ) , as was recommended by Radford et al . ( 2015 ) while introducing the DCGAN architecture . The parameter update step for Adam is given by : ∆θt = −α m̂t√ v̂t + mt = β1mt−1 + ( 1− β1 ) gt , m̂t = mt 1− βt1 vt = β2vt−1 + ( 1− β2 ) g2t , v̂t = vt 1− βt2 ( 8 ) Here , gt is the gradient at timestep t , and m̂t and v̂t are the first and second order exponential moving averages of the parameterwise gradients , bias-corrected to account for zero-initialization . There are four hyperparamters : is a small constant primarily for numerical stability , α is the learning rate and β1 and β2 determine the effective memories of the moving averages . The fraction m̂t√ v̂t resembles unit normalization of a vector , and for a constant gradient gt = g ( such that the moving averages are trivial ) update steps depend only on the sign of g , if |g| . ∆θt = −α sgn ( g ) 1 + |g| ( 9 ) However , this normalization does not necessarily address MM-GAN ’ s saturation problem , due to the training dynamics . Doptl = ±∞ where the real and generated data do not overlap ( eq 7 ) , such that if D can cleanly separate real from generated samples , we expect it to further decrease its loss by inflating its outputs . Supposing that D approaches this optimum linearly , i.e . Dtl ( G ( z ) ) = at , we get : Dtp ( G ( z ) ) ≈ exp ( at ) ( 10 ) Dp also appears as the MM-GAN scaling factor ( eq 4 ) , such that G will be optimized with gradients of the form gt = g0 exp ( at ) . For reasonable values of a , β1 and β2 , the update step size for each of G ’ s parameters can be approximated : ( Supplementary : G ) ∆θt ≈ g0C exp ( ( a− 1 2 log ( β2 ) ) t ) ( 11 ) Given the commonly used β2 = 0.999 , parameter updates will vanish exponentially if a < −0.0005 , given that |Dl| increases fast enough and conforms reasonably well to our simplified model . If D learns to distinguish the real and generated data manifolds before they meaningfully intersect , this interaction between D , G and Adam threatens to freeze parameter updates altogether . ( Supplementary : L )

Summary: This paper reexamines the original (MM) and the non-saturating (NS) GAN objective.  The authors show that the gradients of the respective objectives just differ from a scaling factor depending on the discriminator's output for generated samples. While the scaling factor for the MM gradient is responsible for the well known vanishing gradient if the discriminator is optimal, the scaling factor of the NS gradient counteracts this saturation effect. However, on the other side, the NS scaling factor introduces a mode dropping effect and the inability of the learning dynamci to discover new modes. The authors show additionally that the NS minibatch gradient is the weighted sum of the single sample MM gradients with respective scaling factors as weights. These scaling factors avoid saturating, however, alter the direction of the resulting minibatch gradient. To counteract the change of the gradient direction the authors propose to summarize the sample scaling factors into one scalar for the batch gradient which preserves the non-saturating behaviour of the NS and the gradient direction of the MM objective. The new GAN objective is called MM-nsat.  Additionally the authors discuss the non-saturating effect of the ADAM beta2 parameter for the MM-GAN Generator.