SAMPLE WEIGHTING AS AN EXPLANATION FOR MODE COLLAPSE IN GENERATIVE ADVERSARIAL NETWORKS
Year: 2020

Abstract:
Generative adversarial networks were introduced with a logistic MiniMax cost formulation, which normally fails to train due to saturation, and a Non-Saturating reformulation. While addressing the saturation problem, NS-GAN also inverts the generator’s sample weighting, implicitly shifting emphasis from higher-scoring to lower-scoring samples when updating parameters. We present both theory and empirical results suggesting that this makes NS-GAN prone to mode dropping. We design MM-nsat, which preserves MM-GAN sample weighting while avoiding saturation by rescaling the MM-GAN minibatch gradient such that its magnitude approximates NS-GAN’s gradient magnitude. MM-nsat has qualitatively different training dynamics, and on MNIST and CIFAR-10 it is stronger in terms of mode coverage, stability and FID. While the empirical results for MM-nsat are promising and favorable also in comparison with the LS-GAN and Hinge-GAN formulations, our main contribution is to show how and why NS-GAN’s sample weighting causes mode dropping and training collapse. Figure 1: Median Fréchet Inception Distance during training for ten runs on MNIST, CIFAR-10, CAT 128 and FFHQ 512, using very simple convolutional GANs. The shaded areas show minimum and maximum value during training for the cost formulations. MM-nsat is best overall, suffers less from gradual mode dropping and trains reliably on the more challenging datasets.

heading: 1 INTRODUCTION
text: Generative adversarial networks have come a long way since their introduction (Goodfellow et al., 2014) and are currently state of the art for some tasks, such as generating images. A combination of deep learning developments, GAN specific advances and vast improvements in data sets and computational resources have enabled GANs to generate high resolution images that require some effort to distinguish from real photos (Zhang et al., 2018; Brock et al., 2018; Karras et al., 2018).
GANs use two competing networks: a generator G that maps input noise to samples mimicking real data, and a discriminator D that outputs estimated probabilities of samples being real rather than generated by G. We summarize their cost functions, JD and JG, for the minimax and nonsaturating formulations introduced in Goodfellow et al. (2014). We denote samples from real data and noise distributions by x and z and omit the proper expectation value formalism:
JDMM(x, z) = JDNS(x, z) = − log(Dp(x))− log(1−Dp(G(z))) JGMM(z) = log(1−Dp(G(z))) JGNS(z) = − log(Dp(G(z))
(1)
For clarity, we use subscripts to distinguish between the discriminator’s pre-activation logit output Dl and the probability representation Dp:
Dp ≡ (1 + exp(−Dl))−1 (2)
Both formulations have the same cost function for D, representing the cross entropy between probability estimates and ground truth. In the minimax formulation (MM-GAN), G is simply trained to maximize D’s cost. Ideally, G matches its outputs to the real data distribution while also achieving meaningful generalization, but many failure modes are observed in practice. NS-GAN uses a modified cost for G that is non-saturating when D distinguishes real and generated data with very high confidence, such that G’s gradients do not vanish. (Supplementary: C)
Various publications establish what the different cost functions optimize in terms of the JensenShannon and reverse Kullback-Leibler divergences between real and generated data:
JGMM ⇔ 2 · DJS (Goodfellow et al., 2014) JGMM + JGNS ⇔ DRKL (Huszár, 2016)
JGNS ⇔ DRKL − 2 · DJS (Arjovsky & Bottou, 2017) (3)
Huszár (2015) and Arjovsky & Bottou (2017) have suggested NS-GAN’s divergence as an explanation for the ubiquitous mode dropping and mode collapsing problems with GANs (Metz et al., 2016; Salimans et al., 2016; Srivastava et al., 2017). While MM-GAN seems promising in terms of its Jensen-Shannon divergence, the formulation has largely been ignored because the saturating cost causes training to break down.
A variety of other GAN formulations have been introduced, such as WGAN-GP (Arjovsky et al., 2017; Gulrajani et al., 2017), LS-GAN (Mao et al., 2016) and Hinge-GAN (Miyato et al., 2018). Lucic et al. (2018) finds that different cost formulations tend to get similar results given sufficient parameter tuning, including various forms of regularization. Despite the questionable form of NSGAN in terms of divergences, it is widely used and can produce very impressive results, such as in the improved StyleGAN (Karras et al., 2019).
2 THEORY
2.1 MM-GAN SATURATION
The parameters of a network are typically trained with some form of gradient descent on a cost function. We find the expressions for D’s and G’s gradients with respect to their parameters, φ and θ: (Supplementary: F)
∇φJDMM,NS = + ∂Dl(G(z), φ)
∂φ ·Dp(G(z), φ)
+ ∂Dl(x, φ)
∂φ · (1−Dp(x, φ))
∇θJGMM = − ∂Dl(G(z, θ))
∂θ ·Dp(G(z, θ))
∇θJGNS = − ∂Dl(G(z, θ))
∂θ · (1−Dp(G(z, θ)))
(4)
We emphasize the two kinds of scaling factors for these gradients in red and blue: they are plotted in figure 2. The discriminator’s scaling factors decrease as it minimizes its cost, approaching 0 towards the optima for both the real and the generated data term.
The minimax formulation JG = −JD is suitable for adversarial training in terms of the generator’s optimum, but the unchanged scaling factor means that G’s gradients increase towards and decrease away from its optimum. The saturation effect described in Goodfellow et al. (2014) is that limDp(G(z))→0∇θJGMM = 0, such that G stops training when D is highly confident that its samples are fake. More generally, the scaling factor makes JG concave with respect to Dl, which interacts poorly with common optimization methods (see section 2.4).
As∇θJGMM and∇θJGNS are the same aside from their scaling factors, the different behaviors of the two formulations must follow from these. NS-GAN’s scaling factor avoids saturation, but gives rise to a different, more subtle mode dropping tendency (see section 2.3).

heading: 2.2 NON-SATURATION AND SAMPLE WEIGHTING
text: As can be seen from eq 4, the NS-GAN and MM-GAN gradients are parallel for a single sample, but with different magnitudes. Stochastic gradient descent estimates the gradient of the cost over the entire input distribution by using a number of samples (a minibatch). We can express the NS-GAN minibatch gradient in terms of the MM-GAN gradient:
∇θJbatchGNS = N−1∑ i=0 ∇θJ sampleGMM (zi) [ 1−Dp(G(zi)) Dp(G(zi)) ] (5)
Due to the bracketed factor, NS-GAN rescales the contribution from each sample relative to MMGAN, implicitly emphasizing samples with smaller values of Dp. Seeing as saturation is caused by the gradient’s vanishing magnitude, this additional effect on the gradient’s direction is questionable.
The exact ratio of the minibatch gradient magnitudes for NS-GAN and MM-GAN depends on ∂/∂θ(Dl(G(z))) for each sample and has no convenient expression. We can approximate it by replacing Dp(G(zi)) in eq 5 with its mean over the minibatch, Dp = 1N ∑N−1 i=0 Dp(G(zi)). This allows us to formulate a form of non-saturation for MM-GAN that mimicks NS-GAN:
∇θJbatchGMM-nsat = 1−Dp Dp N−1∑ i=0 ∇θJ sampleGMM (zi) (6)
We refer to the formulation with this generator gradient as MM-nsat. The relative weights of samples in each batch are as for MM-GAN, while the gradient magnitude approximates that of NS-GAN. Note, however, that the relative weights of samples may be disturbed across batches, such as when the minibatch size is small and Dp fluctuates. Despite different theoretical motivation, MM-nsat is very closely related to importance weighted NS-GAN (Hu et al., 2017). (Supplementary: D)

heading: 2.3 SAMPLE WEIGHTING AND MODE DROPPING
text: If we use r(x) and g(x) to denote the density of real and generated samples at a point x in data space, the optimal discriminator (i.e. the function that minimizes JDMM,NS ) is given by:
Doptp (x) = r(x)
r(x) + g(x) (Goodfellow et al., 2014) (7)
This expression for the optimial discriminator assumes idealized conditions: that D is optimized for a fixed G, with unlimited capacity and without having to estimate the underlying real and generated data distributions by finite sampling. While Dopt is not realized in practice (Sinn & Rawat, 2018), we use it to form some intuitions about D’s behaviors.
Suppose we have real data with two disjunct, equiprobable modes, and that one of these modes, O, is overrepresented in generated data. For convergence, G would need to shift probability mass from O to the underrepresented mode, U. However, the minibatches used to update G will have more samples from O, simply because they are generated more often. For a strong discriminator, these samples will also tend towards smaller values of Dp(G(z)), due to equation 7. This in turn
causes them to be weighted differently by NS-GAN and MM-GAN, because Dp(G(z)) appears in their scaling factors (eq 4). We refer to the first effect as over- and underrepresentation (of generated samples relative to real samples) and the second effect as up- and downweighting (governed on the scaling factor).
The fundamental problem with NS-GAN can be seen by looking at how these two effects interact. The NS-GAN generator has a scaling factor 1−Dp(G(z)), which combines overepresentation with upweighting and underrepresentation with downweighting, allowing an overrpresented mode O to dominate U’s contributions to the parameter updates. If parameter updates overwhelmingly based on gradients from O have an adverse effect on G’s ability to generate samples from U, this mode may become increasingly underrepresented, making O yet more dominant.
MM-GAN’s scaling factor has the reverse behavior, pairing overrepresentation with downweighting and underrepresentation with upweighting, such that the two effects combine in a stabilizing rather than destabilizing way. For the extreme example of a nearly dropped or newly discovered mode, r(x) g(x) such that we expect Dp(x) ≈ 1 for a strong discriminator. NS-GAN’s sample weighting disregards gradients from such samples, whereas MM-GAN’s sample weighting emphasizes them.
This difference between MM-GAN and NS-GAN in terms of scaling factors reflects the different divergences they have been shown to optimize (eq 3). Huszár (2015) and Arjovsky et al. (2017) both connect the mode dropping tendencies of NS-GAN to its reverse Kullback-Leibler divergence, which strongly penalizes G for generating samples outside of the real data modes, but not for dropping modes. A variety of attempts to address mode dropping, mode collapse and mode hopping in GANs seem unaware that this is reasonable behavior for NS-GAN given its divergence: mode dropping minimizes DRKL and maximizes DJS.
2.4 MM-GAN INTERACTION WITH ADAM
GANs are generally trained with the Adam optimizer (Kingma & Ba, 2014), as was recommended by Radford et al. (2015) while introducing the DCGAN architecture. The parameter update step for Adam is given by:
∆θt = −α m̂t√ v̂t +
mt = β1mt−1 + (1− β1)gt, m̂t = mt
1− βt1 vt = β2vt−1 + (1− β2)g2t , v̂t =
vt 1− βt2
(8)
Here, gt is the gradient at timestep t, and m̂t and v̂t are the first and second order exponential moving averages of the parameterwise gradients, bias-corrected to account for zero-initialization. There are four hyperparamters: is a small constant primarily for numerical stability, α is
the learning rate and β1 and β2 determine the effective memories of the moving averages.
The fraction m̂t√ v̂t resembles unit normalization of a vector, and for a constant gradient gt = g (such that the moving averages are trivial) update steps depend only on the sign of g, if |g| .
∆θt = −α sgn(g) 1 + |g|
(9)
However, this normalization does not necessarily address MM-GAN’s saturation problem, due to the training dynamics. Doptl = ±∞ where the real and generated data do not overlap (eq 7), such that if D can cleanly separate real from generated samples, we expect it to further decrease its loss by inflating its outputs. Supposing that D approaches this optimum linearly, i.e. Dtl (G(z)) = at, we get: Dtp(G(z)) ≈ exp(at) (10) Dp also appears as the MM-GAN scaling factor (eq 4), such that G will be optimized with gradients of the form gt = g0 exp(at). For reasonable values of a, β1 and β2, the update step size for each of
G’s parameters can be approximated: (Supplementary: G)
∆θt ≈ g0C exp((a− 1
2 log(β2))t) (11)
Given the commonly used β2 = 0.999, parameter updates will vanish exponentially if a < −0.0005, given that |Dl| increases fast enough and conforms reasonably well to our simplified model. If D learns to distinguish the real and generated data manifolds before they meaningfully intersect, this interaction between D, G and Adam threatens to freeze parameter updates altogether. (Supplementary: L)

heading: 3 METHOD

heading: 3.1 COST FUNCTION MODIFICATIONS
text: In addition to our novel non-saturating version of MM-GAN (MM-nsat: eq 6), we normalize the gradient magnitudes of the original cost functions (MM-unit, NS-unit). Motivated by our model for MM-GAN saturation in section 2.4, we also test if modifying the Adam β2 parameter for G gives the expected results (MM β2 = 0.99). (Supplementary: H)
MM-nsat is the only of these cost functions that is interesting in its own right: the others are used to demonstrate various behaviors and highlight the roles of sample weighting as opposed to gradient magnitude. Note that the prefixes MM and NS always correspond to the sample weighting used by the cost function. Furthermore, gradient magnitudes are matched for MM-unit & NS-unit and approximately matched for MM-nsat & NS-GAN.

heading: 3.2 EVALUATION
text: Evaluating a generator is generally difficult. In addition to visual inspection, we use Frechét Inception Distance (Heusel et al., 2017), which we find informative also on MNIST, in spite of the feature extraction network being trained on natural images. Due to our focus on the mode dropping problem, we also use an unusual metric for datasets with class labels: we compute the Jensen-Shannon divergence between class distributions in real and generated data. For G, we estimate its class distribution by drawing samples and using a pre-trained classification network to label them. Note that this metric only sees class distributions and is not sensitivite to the mode coverage or collapse within any given class. (Supplementary: I)
DJSCD ≡ DJS(class frequenciesdataset||class frequenciesgenerator) (12)

heading: 3.3 EXPERIMENTS
text: For the majority of our experiments we use very simple networks without tuning their hyperparameters: either fully connected networks (FC) with a fixed number of hidden units, or strided convolutional networks with kernel size 3 and doubling the number of filters when the width and height is halved (Conv). We use ReLU activations, except for the final layer, where G uses tanh activation with real data normalized to [−1, 1], while D uses sigmoid activation to map Dl to Dp. We make use of how fully connected networks are harder to train than convolutional ones (ThanhTung et al., 2018) and how training grows increasingly fragile for deeper networks and higher resolution datasets to find illustrative test cases. We make use of batch normalization (Ioffe & Szegedy, 2015) (bn), zero centred real data gradient penalty (Mescheder, 2018) (sgp) and spectral normalization (Miyato et al., 2018) (sn) only where explicitly mentioned. In addition to the simpler networks, we test the full DCGAN (Radford et al., 2015) and StyleGAN (Karras et al., 2018) architectures.
We primarily use the MNIST (LeCun & Cortes, 2010) and CIFAR-10 (Krizhevsky et al., 2009) datasets to investigate the differences between cost functions. We run additional experiments using the CAT (Zhang et al., 2008; Jolicoeur-Martineau, 2018) and FFHQ (Karras et al., 2018) datasets to study behaviors for higher resolution images.

heading: 4 RESULTS AND DISCUSSION

heading: 4.1 QUALITATIVE PRELIMINARIES
text: Training GANs on a ring of Gaussians has been used to study both the mode dropping tendency of GANs (Metz et al., 2016; Srivastava et al., 2017) and how to address it. MM-GAN tends to do very well on such problems, as shown in figure 4. For this problem, it is easy to generate samples indistinguishable from real ones, limiting problems with saturation. MM-GAN is more mode-covering in practice as suggested by its divergence (Huszár, 2015; Arjovsky et al., 2017). (Supplementary: S)
Using MNIST and weak, fully-connected networks, we replicate the well-known failure mode of MM-GAN that motivates the NS-GAN reformulation. In figure 5, we show the gradient magnitudes early in training, in particular a super-exponentially vanishing gradient for MM-GAN after roughly 3 epochs that halts training altogether. Additionally, we show that reducing β2 only for G’s optimizer (effectively giving it a shorter memory for the second order momentum) stabilizes the training process, as suggested by our theory on the interaction of MM-GAN and Adam. (Supplementary: L)
For the same setup, we show samples at the end of prolonged training for NS and MM-nsat cost functions in figure 6. We see that our version of minimax non-saturation trains well. The difference in terms of mode coverage is visually striking and corresponds well with our numerical evaluations for the same generators in table 1.
For MNIST, NS-GAN’s mode collapse can mostly be addressed by early stopping or regularization. In figure 7 we show samples from training on the more challenging CAT 1282 dataset. We find that MM-nsat trains well and generates fairly realistic samples, whereas for NS-GAN, catastrophical mode collapse occurs before G learns to produce reasonable samples. More rigorous and quantitiative results are presented in the supplementary material, particularly U and V.

heading: 4.2 QUANTITATIVE EVALUATION
text: Figure 8 shows more comprehensive results for MNIST and CIFAR-10. We plot metrics throughout training to emphasize the dynamics for MM-GAN and NS-GAN, as well as variants of the same cost functions. There are two key results: that simple stabilization techniques allow us to train GANs with
MM-GAN sample weighting, without the saturation issues that plague unmodified MM-GAN; and that their behavior is qualitatively different from those using NS-GAN sample weighting. Gradient magnitudes mostly influence training stability. Additionally, the plots on the right hand side show that the performance gain for minimax variants is reflected by more correct class distributions in generated data. In some cases, such as for 3-layer fully connected networks on MNIST, there is a notable gap between DJSCD values, even while FID is similar.
The importance of sample weighting is best demonstrated by comparing MM-unit and NS-unit. By construction, these variants have the same gradient magnitude, such that the only difference between the formulations is whether high- or low-scoring samples are emphasized when finding the total gradient for a minibatch of samples. The results show that this difference is crucial for the training dynamics.
Aside from stability, there are no clear differences between the variants with minimax sample weighting. While MM-nsat and MM-unit avoid saturation in very different ways, their results are highly similar, to each other and to the results for unmodified MM-GAN when it does not saturate. This suggests that the approximation we make in equation 6 when introducing Dp is of limited importance, and that MM-nsat is more faithful to the original, logistic minimax formulation than NS-GAN.
The main issue with NS-GAN for these datasets is its strong tendency to deteriorate in terms of both FID and DJSCD as training progresses. There is an early stopping point where NS is competitive with MM-nsat in some cases, but not all. The results for CIFAR-10 are the most extreme: NS and NS-unit consistently undergo catastrophical mode collapse.
Overall, these results validate theory from the literature showing that MM-GAN and NS-GAN correspond to different divergences and the importance of sample weighting as described in section 2.3. We present a variety of additional results in the supplementary material (M through W), both testing the validity of our theory in different experimental settings and showing results for higher dimensional datasets.

heading: 5 CONCLUSION AND FURTHER WORK
text: Based on a more thorough, theoretical analysis of the differences between the formulations introduced in Goodfellow et al. (2014), we have designed a form of non-saturation for the minimax cost function for GANs that rescales the minibatch gradient. This corrects the training difficulties for the original minimax GAN, without the side-effects inherent to the NS-GAN reformulation. Running experiments, we have shown that our new stabilization has qualitatively different behavior, in particular better mode coverage as indicated both by our theory and by previous works showing which divergences are optimized by NS-GAN and MM-GAN. Perhaps most importantly, our work is an important correction to the view that NS-GAN is just MM-GAN corrected for its saturation problems, as the name might suggest.
We have shown promising results with MM-nsat (our gradient rescaled version of MM-GAN), but it is primarily designed for demonstration purposes. Results with simpler networks on MNIST and CIFAR-10 are much stronger for MM-nsat compared to NS-GAN, Hinge-GAN and LS-GAN, and experiments on higher resolution images from the CAT and FFHQ datasets show that MM-nsat has less issues with catastrophic mode collapse during the early stages of training, greatly reducing the stability issues that GANs tend to suffer from.
Interactions with discriminator regularization are unclear and important for more advanced applications (see R). Performance when combining MM-nsat with various designs that directly address mode dropping remains to be determined, for instance the class conditioning (Mirza & Osindero, 2014) used in BigGAN (Brock et al., 2018) and the minibatch standard deviation used in StyleGAN (Karras et al., 2018). Unlike the normalizations and regularizations that mask the underlying issues with NS-GAN and often introduce extra hyperpameters, MM-nsat changes only the cost function.
While we have focused on variants of MM-GAN and NS-GAN, with the convenience that these all work with the same discriminator cost function, we expect analysis in terms of sample weighting to apply much more generally, explaining training dynamics and tradeoffs between individual sample quality and overall sample diversity also for other GAN formulations.

heading: A TABLE OF COST FUNCTIONS
text: Refer to table 2 for an overview of the cost functions used in this paper. Note that sample emphasis is not directly comparable between formulations with different costs for the discriminator.
B INFORMAL AND EXTENDED SUMMARY
We include an extended summary of the most central points from our work. This presentation is intended to be less technical and somewhat more accessible to a wider audience.
When training a classifier, the standard cross entropy loss has stronger gradients for samples with greater error. This means that correcting misclassifications is emphasized over further increasing the confidence level of correctly classified samples. Training usually runs for multiple epochs, giving the classifier repeated opportunities to readjust if its cost increases for any particular sample.
Like for classifiers, every term in the NS-GAN cost functions for the discriminator and generator emphasizes the samples which are furthest from their optima. While this seems reasonable, it does not necessarily interact well with adversarial training.
To minimize its cost when training on equal amounts of real and generated data, the discriminator Dp(x) approximates P (y is real|y = x). This means that Dp ≈ 1 for real data that G does not generate and Dp ≈ 0 for generated data outside of the real data manifold. More generally, Dp(x) approximates the density of real samples at x relative to the sum of densities of real and generated samples.
The generator cost function includes a real data term which is often omitted: without any gradient with respect to G’s parameters it does not influence the training. This makes it difficult for G to act on dropped modes, because its updates only see D’s gradients at the points in data space where G generates samples.
The degree to which NS-GAN’s generator cost emphasizes samples further from their optimum is quantified by the scaling factor given in equation 4. NS-GAN weights contributions from each generated sample by a factor 1 − Dp, such that low-scoring (“unrealistic”) samples contribute the most to the gradient.
MM-GAN’s generator instead uses the scaling factor Dp, emphasizing high-scoring (“realistic”) samples. MM-GAN saturation follows directly from this scaling factor: for highly unrealistic generated samples, Dp → 0, and this weighting causes G’s gradient vanish. Adam does not work around this problem with the usual hyperparameter settings.
In either case, G’s parameters are updated by sampling the gradient of D with respect to G’s parameters at points x where samples happen to be generated. This means that the updates see most contributions from regions where the density of generated samples is high.
The main problem with NS-GAN is how this sampling frequency interacts with its scaling factor. Regions on the real data manifold where the density of generated samples is too low tend towards small values of 1−Dp: these generated samples are considered realistic by D and thus have small
error for G and are weighted down by the scaling factor. Since this region in data space is also sampled rarely, due to the G’s low density, its contribution to G’s parameter updates will be small. At the same time, regions with too high density of generated samples have large scaling factors and are sampled frequently. In a situation where different modes disagree on the best configuration of G’s parameters, undersampled modes will be massively outvoted.
This would not be a problem if D’s gradients were able to push the density of generated samples away from oversampled modes and discover undersampled or entirely dropped modes. However, we have no guarantee that this sort of action at a distance will work in practice: the literature instead shows that mode collapse is a recurring problem, likely because D’s gradients fail to be informative across longer distances in data space.
Sample weighting also explains why MM-GAN behaves much better in terms of mode coverage. Since MM-GAN uses the opposite scaling factor, Dp, it boosts the contributions to parameter updates from undersampled modes. The MM-GAN scaling factor partially cancels against the sampling bias effect, making the parameter update less dependent on a single, dominant mode. While a generated sample in an undersampled mode is not itself an error, the very high value of Dp for this generated sample reflects a large discrepancy between generated and real densities at this point. MM-GAN’s sample weighting makes it better able to address errors of this sort.
The main problem with MM-GAN is that its gradient decreases when the cost is increases. This makes it very difficult to optimize, also with Adam. NS-GAN, the established solution to this problem, changes both sample weighting and gradient magnitudes. The problem can instead be addressed with a very simple rescaling of the minibatch gradient, as shown in eq 10. We call the resulting formulation MM-nsat as it combines minimax sample weighting with the same form of non-saturation as used in Goodfellow’s NS-GAN. The key difference is that the NS-GAN non-saturation is implicitly applied to each sample, disturbing sample weights, while the MM-nsat rescaling is explicitly applied to the minibatch gradient as a whole.
Our experimental results validate that this minibatch gradient rescaling corrects the saturation problem, and comparing NS-GAN and MM-nsat allows us to show that sample weighting is highly important for training dynamics. Mode coverage and general stability is greatly improved for MM-nsat relative to NS-GAN, which sometimes suffers from gradual mode dropping throughout training, and sometimes from catastrophical mode collapse early in training. Note, however, that strong models already tend to include various designs to counteract the problems caused by NS-GAN’s unfortunate sample weighting, limiting the benifits of simply replacing NS-GAN with MM-nsat.

heading: C GOODFELLOW’S MOTIVATION OF NS-GAN
text: We include Goodfellow et al. (2014)’s motivation for NS-GAN for easy reference: “In practice, equation 1 [minimax cost] may not provide sufficient gradient for G to learn well. Early in learning, when G is poor, D can reject samples with high confidence because they are clearly different from the training data. In this case, log(1 − D(G(z))) saturates. Rather than training G to minimize log(1 − D(G(z))) we can train G to maximize logD(G(z)). This objective function results in the same fixed point of the dynamics of G and D but provides much stronger gradients early in learning.”
D IMPORTANCE WEIGHTING
Hu et al. (2017) studies the parallels between generative adversarial networks and variational autoencoders. They transfer the idea of importance weighting from VAE to GANs, arriving at an update rule for the generator (Hu et al. (2017): equation 22) which reweights G’s NS-GAN gradients for each sample in the minibatch with a normalized weighting factor. Closely related ideas are seen in Hjelm et al. (2018) and Che et al. (2017). The expression for the unnormalized weighting factor is given as:
wi = qrφ0(y|xi) qφ0(y|xi)
(13)
Here, qφ0 and q r φ0 represent the discriminator and its reverse respectively, i.e. qrφ0(y|x) = qφ0(1 − y|x), where y represents the true label of a sample x. In our notation, this is corresponds to the
following, as given explicitly in Hjelm et al. (2018):
wi = Dp(G(zi))
1−Dp(G(zi)) (14)
Comparing this weighting factor to the bracketed factor in our eq 5, it is clear that it perfectly cancels against the reweighting of NS-GAN relative to MM-GAN. In other words, importance weighted NSGAN actually recovers the same original MM-GAN sample weighting that is used by MM-nsat, such that our approach of avoiding the NS-GAN sample reweighting leads us to rediscover this update rule.
The remaining difference is that of the gradient magnitude. When reweighting the NS-GAN gradients, Hu et al. (2017) and Hjelm et al. (2018) choose to normalize the weights over the minibatch: omitting to do this would make their importance weighting also recover the MM-GAN gradient magnitude, and thus reintroduce the MM-GAN saturation problem. They normalize by calculating the actual sum of the reweighting factors, which is slightly different from our approximation where we replace Dp with its mean over the minibatch (eq 6). Note that neither method is exact unless the samplewise gradients in the minibatch have the same magnitude. As seen by for instance comparing MM-nsat and MM-unit, which have highly similar performance despite massively different gradient magnitudes, these diferences should have little effect on training dynamics.
The primary difference between importance weighthing for NS-GAN and our design of MM-nsat is the logical steps and the theoretical motivation. In our work, the direct relationship to the original MM-GAN update rule is made explicit, whereas importance weighting goes the cirucuitous route of first taking the heuristic NS-GAN formulation in place the original MM-GAN formulation, then reweighting it to arrive at an importance weighted NS-GAN formulation, which our work shows to actually be just a non-saturating version of MM-GAN.

heading: E CONCURRENT WORK
text: A concurrent work recently made available as a pre-print, Sinha et al. (2020), makes use of a simple design which is directly comparable to our method. When updatingG, they ignore the gradients arising from the lowest scoring generated samples, where the fraction of the samples ignored increases throughout training. This effectively shifts weighting from low-scoring to high-scoring samples, which is the same overall effect as is achieved by using our MM-nsat gradient. Their method is much simpler and less principled as to exactly how low- and high-scoring samples are weighted, but it is also more easily applied to other GAN cost formulations.
Similar to our findings, Sinha et al. (2020) observe an increase in mode coverage and overall quality. By investigating the cosine similarity between low- and high-scoring samples, they observe that optimization conflicts between these, as we suggest in section 2.3. Their analysis in terms of pushing samples towards and away from modes is somewhat at odds with ours, where we note that lowscoring samples can be high-quality, but lie in modes that are significantly oversampled by G. The discriminator score is not a direct measure of the sample’s quality, but rather of relative densities of real and generated data, such that we should not expect D’s gradients to necessarily point towards the centres of the real data modes.

heading: F GRADIENTS DERIVED FROM COST FUNCTIONS
text: Finding the MM-GAN gradient for a single sample is a matter of straightforward calculus. The expression for NS-GAN’s gradient can be found using the same approach. First note that the relationship between Dp and Dl is given by eq 2, such that:
∂Dp ∂Dl = D2p exp(−Dl) (15)
exp(−Dl) = 1−Dp Dp
(16)
We take JG = −JD (eq 1) as the starting point, as defines the minimax formulation. Note that the first term does not depend on G or θ, such that it can be omitted in the G cost function:
JGMM = log(Dp(x)) + log(1−Dp(G(z, θ))
∇θJGMM = − ∂Dp(G(z, θ)) ∂θ · 1 1−Dp(G(z, θ))
= −∂Dl(G(z, θ)) ∂θ · ∂Dp ∂Dl · 1 1−Dp(G(z, θ))
= −∂Dl(G(z, θ)) ∂θ
· D2p exp(−Dl)
1−Dp
= −∂Dl(G(z, θ)) ∂θ · Dp(1−Dp) (1−Dp)
= −∂Dl(G(z, θ)) ∂θ ·Dp(G(z, θ))
(17)
It might seem counterintuitive to mix Dl and Dp when expressing the gradients, instead of using for instance the expression on line 2 of equation 17. However, the non-linear behavior of ∂Dp/∂θ makes interpretation difficult. As Dp approaches either 0 or 1, increasingly small changes in Dp reflect the same change in odds ratios. It is less clear from the gradient in terms of only Dp that the MM-GAN gradient saturates as Dp approaches 0, or that it does not diverge as Dp approaches 1. However, for the scaling factor itself, we find the Dp formulation most intuitive. The interpretation suggested by eq 7 is particularly convenient.

heading: G MM-GAN INTERACTION WITH ADAM
text: We include additional details for the approximation of the Adam update step. We combine equation 8 and gt = g0 exp(at). For simplicity, we assume that sgn(gt) does not depend on t: if the sign of the gradient changes, mt is reduced and vt unaffected. This can contribute to making ∆θt vanish, but cannot prevent it.
This allows us to express the momentum as a sum of contributions from each previous time step i.
mt = i=t∑ i=1 βt−i1 (1− β1)g0 exp(ai) (18)
For reasonably large values of t and assuming a 6= log(β1), we can approximate this sum by a definite integral:
mt ≈ ∫ i=t i=0 (1− β1)βt−i1 g0 exp(ai)di
= 1− β1
a− log(β1) g0(exp(at)− βt1)
(19)
Using this approximation (similarly also for vt), assuming to be negligible and packing assorted time independent constants including −α into C, we get the update step:
∆θt ≈ g0C √
1− βt2 1− βt1 exp(at)− exp(log(β1)t)√ | exp(2at)− exp(log(β2)t)|
(20)
Having already assumed t to be large, we omit the first fraction (bias corrections) for simplicity:
∆θt ≈ g0C exp(at)− exp(log(β1)t)√ | exp(2at)− exp(log(β2)t)|
(21)
For sufficiently large values of t, behavior is determined by which of the terms in the numerator and denominator have the largest constant in the exponent (i.e. slowest decay). We enumerate all the
four possible cases:
a > log(β1) ∧ a > 1
2 log(β2)→ ∆θt ∝ 1
a < log(β1) ∧ a < 1
2 log(β2)→ ∆θt ∝ ( β1√ β2 )t
a > log(β1) ∧ a < 1
2 log(β2)→ ∆θt ∝ e(a− 1 2 log(β2))t
a < log(β1) ∧ a > 1
2 log(β2)→ ∆θt ∝ e(log(β1)−a))t
(22)
Using hyperparameters such that β1 > √ β2 is highly unusual. It is required to satisfy the inequalities for the last case and means that the base is greater than one in the second case. In both cases, we get exploding update steps, which is generally undesirable. Assuming β1 < √ β2, the first three cases show intended behavior: update steps are normalized for constant or even exploding gradients, while they diminish for vanishing gradients, enabling convergence. The third case, a < 12 log(β2), is discussed in the main text. In the second case, where a < log(β1), we get a different kind of vanishing behavior governed by the relationship between β1 and β2.
Additional results and discussion for this problem is given in L.
H IMPLEMENTATION DETAILS
For consistency with the theoretical framework and to allow testing of magnitude normalized gradients, we modify GANs by rescaling the generator gradient. This amounts to calculating the minibatch gradient explicitly and multiplying it by a factor R calculated as given in algorithms 1 and 2 before passing it on to the optimizer. (Code repository: J)
Algorithm 1 Non-saturating minimax rescaling factor (MM-nsat) Require: G,Dl: generator and logit valued discriminator functions Require: N : the minibatch size Require: z0, . . . , zN−1: N generator inputs Require: R: a small constant, prevents zero-division overflow such that Rmax = −1R
1: Dp = 1 N ∑i=N−1 i=0 (1 + exp (−Dl(G(zi)))−1 2: R← 1−Dp R+Dp
Algorithm 2 Unit rescaling factor (MM-unit, NS-unit) Require: ∇θ: the gradient of the cost function with respect to G’s parameters Require: Nθ: the number of parameters of G Require: R: a small constant, prevents zero-division overflow such that Rmax = Nθ −1R
1: |∇θ| = ( ∑i=Nθ−1 i=0 ∇2θi) 1 2 2: R← Nθ R+|∇θ|
However, for proper behavior with penalty terms as well as convenience, we strongly recommend implementing MM-nsat by multiplying the minimax cost by a factor R which depends on D(G(z)) for the entire batch and disabling back-propagation through this scaling factor. In TensorFlow, a straightforward implementation is:
cost_mm-nsat = tf.stop_gradient(R) * cost_mm (23)
The rescaling factor is constant across the minibatch and can be applied either samplewise or to the total cost.
For MM-unit and NS-unit, the rescaling factors unfortunately cannot be calculated without first computing the gradient for the unmodified cost functions.
Applying gradient rescaling directly does not interact well with penalty terms. When the MM-GAN cost saturates, the gradient will typically be dominated by the penalties, causing both MM-nsat and MM-unit rescaling to inflate their effect. Using the approach above, rescale the adversarial minimax cost and leave penalty terms unchanged.
Parallelization may raise some issues when calculating the rescaling factor R, because proper minimax behavior requires R to be calculated across all parallels and a naive approach will introduce some overhead. Ideally, D(G(z)) and gradients from parallels should be aggregated in the same step, in order to rescale gradients before running them through the optimizer. Correctly implemented, the overhead should be unnoticeable for multi-GPU training.
We note that for very small batch sizes, or where R for other reasons vary significantly between subsequent batches, MM-nsat will have some NS-like behaviors, because weighting between samples in different batches is disturbed by the difference between their values of R. However, not even unmodified MM-GAN weights properly across batches, due to updates to D and optimizer corrections. It is possible to smooth R across batches using for instance an exponential running mean, at the risk of reintroducing saturation issues.
In algorithm 2, we make the unintuitive choice of normalizing the gradient not to unity, but to Nθ, the number of network parameters. Due to cancellation effects in Adam’s parameter update step, this time-independent scaling factor is only relevant when gt is small and non-negligible. Because Adam’s ∆θt is calculated for each parameter independently, this choice of normalization prevents introducing an inadvertent relationship between the number of parameters of a network and whether updates vanish due to as shown in eq 9.

heading: I CLASS DISTRIBUTION DIVERGENCE
text: Implementing a class distribution divergence requires us to have labels for the real data and some way to determine which class generated samples belong to. This limits its applicability to simpler problems where we can train strong classifiers. An additional complication is that we are applying this classifier to generated data, where it may not generalize well. The classifier is calibrated for real data samples and its behavior is not well defined outside of this manifold. In particular, a standard classifier is compelled by design to assign a class to all samples, even when they are so poor that they cannot meaningfully be said to belong to any class at all. It is conceivable that a generator producing only noise-like samples will be assigned functionally random classes and achieve a very good DJSCD. Furthermore, DJSCD is not sensitive to the degree of coverage or collapse within any given class.
In practice, we find that a poor generator invariably scores badly in terms of DJSCD, with nearly all its samples being assigned to the same class, and that FID and DJSCD are strongly correlated. Nonetheless, combining DJSCD with a standard generator evaluation method such as FID is recommended, and for poor generators, DJSCD may be misleading. Similarly, the class divergence serves as a control for our use of FID for MNIST, which is significantly removed from the natural images FID’s Inception network in trained on.
For interpretation, note that we use a normalized Jensen-Shannon divergence, such that DJS ∈ [0, 1]. The lower bound corresponds to perfectly matched distributions and increasingly mode dropping generators will approach the upper bound. When the real data distribution is non-zero for all classes, (DJSCD)max < 1. For the case where there are ten evenly distributed classes in real data, which includes MNIST and CIFAR-10, we get (DJSCD)max ≈ 0.758. For classification, we train a simple, convolutional network for MNIST and a ResNet (He et al., 2015) based convolution model with batch normalization (Ioffe & Szegedy, 2015) for CIFAR-10 with approximately 99% and 89% validation accuracy (see section J). For MNIST-1K, we apply the MNIST classifier to each channel separately, treating each ordered triplet of classifications as its own class. In line with the established standard for FID, we use 50000 samples for estimating DJSCD for a generator.

heading: J CODE AND NETWORKS
text: We have made code for Python 2.7 and TensorFlow available in a GitHub-repository: to maintain anonymity, see uploaded supplementary material instead. Qualitatively reproducing the MNIST and toy experiments in the main paper is straightforward, but other datasets as well as running the evaluation metrics involves additional setup (see repository for details).
The repository includes code for training classifiers for DJSCD for MNIST and CIFAR-10. Step by step setup for the CAT and FFHQ datasets is not included, nor is the forked version of StyleGAN for training it with MM-nsat.

heading: K GRADIENT NORM AND DIRECTION FOR MM-NSAT
text: Figure 9 shows how the gradients obtained by the NS-GAN and MM-nsat generator costs differ during training, both in terms of norm and direction. These results illustrate the theory from section 2.2 and give an impression of the quality of the approximation used for MM-nsat (eq 6), in order to renormalize its gradient magnitude of MM-GAN to match with that of NS-GAN.
For 4-layer convolutional networks, FID improves for both formulations with very similar quantitative results for roughly 200 epochs. During this period, gradients are also the most similar in terms of norm and direction. Afterwards, cosine similarity drops, indicating that gradients are becoming increasingly different for the purpose of optimization, and NS-GAN and MM-nsat diverge in terms of FID due to NS-GAN increasingly dropping modes.
For 4-layer fully connected networks, cosine similarity drops much earlier, and FID for NS-GAN falls behind very early in training, reflefcting mode dropping (see DJSCD for FC-4 in figure 8). During the later stages of training, gradient norms diverge, particularly for the case where we train with the NS-GAN gradient. Note, however, that this is a stage of training where NS-GAN shows very pathological behavior, with more than 34 of generated samples representing the digit 1.
For interpretation of the cosine similarity, note that gradients are high dimensional vectors, such that near-orthogonality is unexceptional. Gradient magnitudes are not perfectly matched for NS-GAN and MM-nsat, particularly not later in training where values of Dp become more extreme and NSGAN starts falling off in terms of FID. When training with the MM-nsat gradient, its norm relative to the NS-GAN gradient are matched to within an order of magnitude and allows MM-nsat to train without any apparent issues.
Figure 10 shows the same kind of plots as figure 9, but comparing the original MM-GAN with MMnsat instead. As the gradients obtained by these formulations are parallel by construction, the cosine similarity is almost exactly 1.0 at all times. We use differently scaled y-axes for the relative gradient magnitudes, as these approach 1016 when MM-GAN falls into its saturating failure mode.
The primary difference in gradient magnitude between MM-GAN and MM-nsat is a relatively brief spike very early in training, corresponding well with the the observation by Goodfellow et al. (2014) cited in section C. This spike is both taller and wider in the FC-4 setting, likely reflecting G having more difficulty learning to generate somewhat realistic samples. During the rest of the training, the magnitude of MM-nsat relative to MM-GAN increases slowly and consistently.

heading: L ADDITIONAL SATURATION RESULTS AND DISCUSSION
text: When D learns its task faster than G, Dp(G(z)) will begin to approach 0, such that both D’s and G’s gradients diminish. The effect on updates after Adam’s normalization step depends on whether the first or second order momentum diminishes faster. The usual case is that β2 > β1, such that the second order momentum has a longer effective memory. This partially disables normalization of updates.
For D, this works as intended: as the cost approaches its minimum, the optimizer sees the gradients becoming smaller and reduces its step size, allowing proper convergence. The problem when optimizing MM-GAN with Adam is that this same behavior triggers for the ill-behaved G cost function when it is failing at its task. While the update steps go in the right direction, the diminishing step size effectively causes parameters to get stuck in a poor configuration. Note that this problem does not really lie with Adam: it is the MM-GAN cost function that has a shortcoming that Adam cannot address since it only sees the decreasing gradients and has no way of knowing that this actually means that the MM-GAN cost is increasing. The role of the gradient rescaling in equation 6 is to make it so that G’s gradients and cost increase and decrease in parallel, such that the optimizer behaves properly.
L.1 THE ADAM β2 HYPERPARAMETER
Here, we provide additional empirical results (figure 11). In particular, we investigate the importance of how quickly D and G’s updates vanish, relative to each other, by adjusting the β2 parameters of D and G (our default is β2 = 0.999). After all, the theory in 2.4 assumes a gradient that is simply exponential in time and does not model the interactions between D and G when the generated data gradients begin to vanish.
A fairly straightforward pattern emerges: for the standard case, where D and G have the same value of β2, there are multiple U-shaped dips, where gradients alternate between diminishing and recovering, until the gradient finally vanishes for good. This still holds when when we reduce β2 for both optimizers, effectively maintaining their balance. However, both setups where β2 is lower for G than for D avoids vanishing updates entirely, whereas when we reduce the value of β2 only for D, training breaks down earlier and more decisively. Finally, we show that we can avoid vanishing gradients by simply reinitializing G’s optimizer at regular intervals: this resets both the first and second order momenta. This stabilizes the training process, which is somewhat surprising seeing as we are effectively crippling the optimizer’s memory. However, this effect is well explained by our hypothesis of large, lingering contributions to the second order momentum disabling normalization.
L.2 LINEAR MODEL FOR Dl(G(z))
Figure 12 shows more detailed diagnostics for two runs where MM-GAN with standard Adam settings falls into its saturating failure mode. In addition to the gradients, we also plot the following: Dl, the mean of the discriminator’s logit outputs, both for real and generated data; and |∆θ| and |∆φ|, the norm of the parameter update for G and D at each timestep.
In section 2.4, we assume thatDl(G(z)) behaves as a linear function of training steps. Note that this simple model is cannot fully capture the training dynamics for MM-GAN, but is meant to explain why Adam does not reliably address MM-GAN saturation despite its built-in gradient normalization.
The actual behavior of Dl(G(z)) in figure 12 is more complicated than we assume in our model. Particularly, |Dl| grows much faster early on than it does towards the end, where it eventually tapers off. However, for Dl(G(z)) = −20, Dp(G(z)) ≈ 2 ∗ 10−9, whereas G only produces a total of 6 ∗ 107 samples during the 1000 epochs we use to train the networks. The behavior of Dl past this point is of limited interest.
We find an early, critical period most relevant for understanding the interaction between Adam and the saturating MM-GAN cost, characterized by Dl becoming monotonic for each of real and generated data. In figure 12, these critical periods start around 0.4 and 2.9 epochs for Conv-4 and FC-4 respectively, without any clearly defined endpoint. Within a single epoch, the step size for G’s updates drops by 3 to 4 orders of magnitude, and G(z) ceases to change during training, as shown in figure 13.
The exception to the above is the sudden spike after 100 epochs for FC-4. Ideally, this spike would be the first step towards the training process recovering, as the size of G’s gradients and updates increases. However, while gradients are larger after this spike than before, the update step size actually plummets by an additional 4 orders of magnitude. The unusually large contribution to the second order momentum lingers much longer than the increase in the first order momentum, effectively disabling Adam normalization for a long while. This is a very illustrative example of the problematic interaction between the saturating MM-GAN cost and Adam.
L.3 ADDITIONAL COMMENTS
Stabilizing MM-GAN by use of the β2 parameter is not recommended: we intend these experiments to demonstrate the effects discussed in section 2.4. The setting where only G’s β2 is reduced to 0.99 suffers the least from noisy updates and performs well in some settings, but all these variants are less stable than the more principled MM-nsat version. However, it should be noted that instead of changing the MM-GAN cost to work with Adam, as we do for MM-nsat, it is possible to modifiy the optimization procedure to account for MM-GAN’s peculiarly shaped cost function.
NS-GAN owes some of its success to how D and G have different scaling factors. Only D’s gradients vanish as Dp(G(z)) → 0, and only G’s gradients vanish as Dp(G(z)) → 1. This gives the
training process a limited form of self-stabilization: update step sizes (roughly equivalent to learning rates) increase for whichever of the networks is doing worse. Since MM-nsat has the same gradient magnitudes as NS-GAN, this behavior carries over.

heading: M CIFAR-10 SPECTRAL NORMALIZATION BASELINE
text: For the CIFAR-10 Conv-4-sn results at the bottom of figure 8, we use exceedingly simple networks as described in section 3.3. In order to compare MM-nsat against a benchmark from the literature, we the copy the settings from Miyato et al. (2018), i.e. the network architectures from table 3 and the hyperparameter settings C from table 1.
The main differences between this setup and our Conv-4-sn settings are:
• Batch normalization in G • 4x4-kernels (not 3x3) for 2-stride convolutions in D and G • 3 additional 3x3-kernel, 1-stride convolutions in D • Spectral normalization also for the final layer of D • Leaky ReLU in D • Approximately 10 times as many network parameters • Doubled learning rate
Additionally, for comparing FID directly against values tabulated in Miyato et al. (2018), note the following:
• Likely different batch size and training epochs • We use all real and 50k generated images to estimate FID, instead of 10k and 5k
Across five runs, we obtain median FID values of 26.7 for NS-GAN and 25.5 for MM-nsat, using the same set of hyperparameters as Miyato et al. (2018) optimized for their NS-GAN results, reporting a value of 29.3. As in other experiments, we again see that DJSCD favors MM-nsat, even at the points during training where FID values are closely matched. Unlike for our networks, we see no clear tendency for the NS-GAN FID to increase towards the end of training, possibly due to higher quality of generated samples and spectral normalization also in the final layer of D combining to pull Dp values closer to 0.5 throughout training. While there is significant overlap between the cost functions, MM-nsat shows slightly better performance overall and converges faster.

heading: N LINEAR COMBINATIONS OF MM-GAN AND NS-GAN
text: Motivated by theory connecting GAN divergences and mode dropping (section 2.3) and Huszár (2016) who suggests adding together the NS-GAN and MM-GAN cost functions in order to get rid of the subtracted Jensen-Shannon divergence for NS-GAN (eq 3), we run experiments using linear combinations of the cost function. Huszár’s suggestion changes the scaling factor (1 − Dp) → 1, which is an improvement in terms of the effect discussed in section 2.3, since it reduces the emphasis on overrepresented modes. However, if Dp(G(z)) is close to zero during training, as often happens with GANs since D has an easier task than G, there will be little difference in practice.
We use the following linear interpolations of the MM-GAN and NS-GAN cost functions, parametrized by a which determines the weight of each term:
∇θJGNS+MM(a) = (1− a) · ∇θJGNS + a · ∇θJGMM ∇θJGNS+MM-nsat(a) = (1− a) · ∇θJGNS + a · ∇θJGMM-nsat
(24)
Due to the MM-GAN and NS-GAN scaling factors (see fig 2), we expect linear combinations of MM-GAN and NS-GAN to be dominated by the non-saturating term, since Dp(G(z)) tends to be small when training GANs unless D is strongly regularized. On the other hand, using our modified MM-nsat approximately balances the gradient norm for each term, such that the behavior of linear combinations should correspond to the weighting of the NS-GAN and MM-nsat terms.
Figure 15 shows results using linear combinations of NS-GAN, MM-GAN and MM-nsat for G’s cost. Results for NS-GAN and MM-nsat are as shown in the main text in figure 8. The behavior for NS-GAN and MM-nsat interpolations is most straightforward, with numerical results falling inbetween the pure NS-GAN and MM-nsat variants with roughly the degree of separation suggested by their weighting constants. Since NS-GAN and MM-nsat have approximately equal gradient magnitudes for all values of Dp(G(z)), the contributions from each term are predictable.
For NS-GAN and MM-GAN interpolations, behavior is less consistent and more similar to NSGAN. This is reasonable, seeing as NS-GAN and MM-GAN have scaling factors 1 − Dp and Dp respectively, such that the two terms are effectively weighted both explicitly by the weighting constant and implicitly by whatever the values of Dp(G(z)) happen to be during training. Note that if D completely fails to discriminate between real and generated samples, it can minimize its cost by making Dp(x) = Dp(G(z)) = 0.5, such that NS-GAN’s scaling factor will almost always be larger than that of MM-GAN.
For the FC-4 results, note that even a 910 weighting for MM-GAN still gives behavior qualitatively similar to NS-GAN: the early stopping metrics are much better than for pure NS-GAN and the mode collapsing behavior is delayed, but the final metrics are worse both in terms of FID and DJSCD even than for the 13 -weighted MM-nsat. Generally, Huszár’s suggestion of adding together the NS-GAN and MM-GAN leads to somewhat improved results, but does not come close to stability and mode coverage achieved by MM-nsat. Compare also to the divergences shown in eq 3.

heading: O SCALING FACTOR EXPERIMENTS
text: In section 2.1, we show that the gradients for the MM-GAN and NS-GAN generators have different scaling factors:
∇θJGMM = − ∂Dl(G(z, θ))
∂θ ·Dp(G(z, θ))
∇θJGNS = − ∂Dl(G(z, θ))
∂θ · (1−Dp(G(z, θ)))
(25)
In section 2.3, we suggest that the different emphasis this places on under- and oversampled modes is the reason for the more zero-avoiding and mode-covering behaviors of NS-GAN and MM-GAN. To test this hypothesis, we can modify the scaling factors and see whether we get the expected change in behavior. The linear interpolations used in section N probe the same effect in a less direct manner.
While we expect the importance of the scaling factors to depend on the relative weights they assign to different scoring samples, it is not clear which relationships are most crucial. The difference between MM-GAN and NS-GAN is most pronounced for low- and high-scoring samples, but the important difference might very well be the relative weights of two samples with different, low scores.
A simple way to modify the NS-GAN gradient is to add a constant a to the scaling factor:
∇θJGNS-add-a = − ∂Dl(G(z, θ))
∂θ · (1−Dp(G(z, θ)) + a) (26)
Which is obtained by the following single-sample cost function:
JGNS-add-a = JGNS − a ∗Dl (27)
Renormalizing the total gradient magnitude in the same way as for MM-nsat, we get the following cost function:
∇θJbatchGNS-add-a = 1−Dp
a+ 1−Dp N−1∑ i=0 ∇θ(JGNS(zi)− a ∗Dl(zi)) (28)
And similarly for MM-GAN:
∇θJbatchGMM-nsat-add-a = 1−Dp a+Dp N−1∑ i=0 ∇θ(JGMM(zi)− a ∗Dl(zi)) (29)
These cost functions are only reasonable for a ≥ 0, such that we avoid negative scaling factors for samples. For increasingly large values of a, sample weights effectively become uniform across the whole range of possible scores.
Since these modifications only allow us to make MM and NS less extreme and more similar to each other, we also try a different approach, introducing a exponentiation parameter c for the scaling factor:
∇θJGNS-exp-c = − ∂Dl(G(z, θ))
∂θ · (1−Dp(G(z, θ)))c (30)
The general solution of this differential equation can be expressed in terms of the hypergeometric function, which does not lend itself to efficient computation. However, specific values of c give rise to useful cost functions. As before, we consistently renormalize the gradient magnitude to that of NS-GAN:
JbatchGNS-exp-2 = 1
1−Dp N−1∑ i=0 ∇θ(JGNS(zi)−Dp(zi)) (31)
JbatchGMM-nsat-exp-2 = 1−Dp Dp 2 N−1∑ i=0 ∇θ(JGMM(zi) +Dp(zi)) (32)
JbatchGNS-exp-0.5 = (1−Dp) 1 2 N−1∑ i=0 ∇θ(2 log( √ e−Dl + √ e−Dl + 1)) (33)
JbatchGMM-nsat-exp-0.5 = 1−Dp Dp 1 2 N−1∑ i=0 ∇θ(−2 log( √ eDl + √ eDl + 1)) (34)
Renormalizing the overall gradient magnitude goes a long way towards stabilizing the adversarial training dynamics for these cost functions. However, overemphasis on oversampled modes tends to accelerate catastrophical mode collapse. Furthermode, expressions such as log( √ eDl)) are numerically unstable for large values of |Dl|. Results using these variant cost functions are shown in figure 16. The add-variants have very clean behavior, all falling in between NS and MM-nsat in terms of FID and DJSCD, in the same order as suggested by their more uniform scaling factors. The exp-variants are more erratic, with NS-exp-2 having major stability issues and MM-nsat-exp-2 falling off later in training for CIFAR-10 Conv-4. Aside from these points, results correspond with our theoretical expectations.
Perhaps the most striking result is the strong performance of the MM-nsat-exp-2 cost function. This strange variant is designed simply to have a more extreme version of the minimax scaling factor, which we expect to further temper the the mode-dropping mechanism described in section 2.3. Indeed, it generally improves performance relative MM-nsat.
Finally, we note that all the MM-nsat-add variants tend towards stronger mode collapse than the MM-nsat-exp variants, regardless of the choice of parameter. We suggest the following explanation for this behavior: Consider two generated samples, both such that Dp(G(z)) is close to 0. With NS sample weighting or with MM-nsat-add variants, the relative weights of these samples will be close to 1, simply because a small Dp is negligible relative to the additive constants. With MM sample weighting and MM-nsat-exp variants, on the other hand, relative weights have a strong dependence the exact values of Dp for each sample and may be orders of magnitude apart.

heading: P HINGE-GAN AND LS-GAN
text: To supplement our results for NS-GAN and MM-GAN variants in section 4.2, we choose to run experiments with the LS-GAN (Mao et al., 2016) and Hinge-GAN (Miyato et al., 2018) formulations. As these only require the cost functions themselves to be modified, they can be compared directly to NS-GAN and MM-nsat. Note that we do not adjust network architectures or hyperparameters from the default settings we use across all experiments, such that results are more indicative of general stability than best-case performance.
The widely used WGAN (Arjovsky et al., 2017) formulation requires additional regularization to prevent D’s outputs from diverging, most commonly a 1-centred gradient penalty on interpolations between real and generated data (Gulrajani et al., 2017). This makes it difficult to draw fair comparisons. Hinge-GAN is very similar to WGAN, but simply clips D’s cost outside of the (0, 1) interval and is known to produce strong results in for instance BigGAN (Brock et al., 2018).
P.1 MNIST AND CIFAR-10 RESULTS
Figure 17 shows results, as in figure 8 in the main text, but with a different set of cost functions. Figure 1 includes a subset of these plots. For the bottom figure, note the use of spectral normalization and a much smaller interval of FID values than the other plots. As for NS-GAN and MM-nsat, we have not adjusted networks
Generally speaking, Hinge-GAN performs better than NS-GAN but worse than MM-nsat and suffers from some of the same gradual mode collapse issues as NS-GAN. Where Hinge-GAN and MM-nsat are most similar in terms of FID, MM-nsat tends towards better class balance. LS-GAN performs remarkably well for MNIST Conv-4, but is otherwise unimpressive and suffers the most from stability issues.
P.2 GRADUAL MODE COLLAPSE FOR HINGE-GAN AND LS-GAN
Interestingly, the results in figure 17 suggest that the behavior of gradual mode collapse seen for NS-GAN and explained in section 2.3 is not specific to NS-GAN. This is most clear for HingeGAN, which in a qualitative sense follows the same trajectory as NS-GAN in terms of both FID and DJSCD, but with better numerical results overall. For LS-GAN, we see similar tendencies, but its behavior is much more erratic overall, including very poor early stopping results for MNIST FC-4 and very strong results for CIFAR-10 Conv-4, even beating MM-nsat.

heading: Q SAMPLE WEIGHTING AND MODE DROPPING FOR OTHER GAN FORMULATIONS
text: In section 2.3, we suggest that the mode dropping tendencies of NS-GAN can be understood as a consequence of how overrepresentation and upweighting interact, allowing generated samples from a small region of the data space to dominate G’s parameter updates. In this section, we discuss the wider implications of this effect.
Note in particular the following figures showing relevant empirical results, with additional discussion in the subsections:
• MM-GAN and NS-GAN variants: fig 8 • MM-GAN and NS-GAN linear combinations: fig 15 • MM-GAN and NS-GAN scaling factor modifications: fig 16 • Hinge-GAN and LS-GAN comparisons: fig 17 and fig 21
The key result from these experiments is that the relationship between upweighting of overrepresented samples and gradual mode dropping is robust across a variety of changes in sample weighting, both implicitly (linear combinations) and explicitly (scaling factor modifications). Increasingly upweighting scaling factors give increasingly mode dropping behavior. Furthermore, Hinge-GAN and LS-GAN suffer from mode dropping behavior which is similar to that of NS-GAN in a qualitative sense: this is consistent for Hinge-GAN, while LS-GAN is more erratic and suffers from stability issues.
In section 2.3 we only discuss the scaling factors of NS-GAN and MM-GAN, which respectively upweight and downweight overrepresented samples. To understand the effect of scaling factors in general, it is instructive to consider the linear combination 12NS + 1 2MM in figure 15. Its scaling factor is simply 12 (1 − Dp(G(z))) + 1 2Dp(G(z)) = 1 2 : in other words, it neither upweights or downweights samples.
Despite its uniform sample weighting, the linear combination 12NS + 1 2MM still has mode dropping tendencies, though it is less blatant than for NS-GAN. This is expected from our theory in section 2.3: MM-GAN downweights overrepresented samples to compensate for their overrepresentation. Uniform weighting, while better than NS-GAN upweighting, does not avoid the mode dropping effect, because overrepresentation on its own is enough to cause gradual mode dropping.
While it is difficult to make general claims about how sample weighting relates to mode dropping for other GAN formulations in general, we can consider Hinge-GAN which has simple cost functions: its generator has the same uniform sample weighting as the linear combination 12NS + 1 2MM discussed above, and its discriminator also has clipped, uniform sample weighting, unlike the erroremphasizing cross entropy loss for MM-GAN and NS-GAN. While the Hinge-GAN discriminator will behave differently during training, it does not give Hinge-GAN any mechanism for downweighting overrepresented samples, explaining the mode dropping behavior seen in our results.
MM-GAN’s form of downweighting is highly unusual. Note that we mean overrepresentation of generated samples relative to real samples: MM-GAN downweights realistic samples that are generated too often. However, it also downweights highly unrealistic samples, since these will be overrepresented by virtue of falling in regions where the density of real samples is negligible. This downweighting is a liability early in training, because it tends to cause saturation problems when D learns to assign very low values of Dp to G’s unrealistic samples. Note results in section O, which indicate that strong downweighting is necessary to stabilize the overrepresentation effect.
This means that cost functions which properly downweight overrepresented samples will generally fail to train unless care is taken to decouple overall gradient magnitude (determining saturation) from sample weighting (determining mode dropping).

heading: R DISCRIMINATOR REGULARIZATION
text: Figure 18 shows results for MM-nsat and NS cost functions when combined with zero-centred real data gradient penalty (Mescheder, 2018). See also results for spectral normalized networks at the bottom in figures 8 and 17, where results are much more similar than in the unregularized case: this still holds true with different cost functions and a different form of regularization. This is essentially a special case of an effect shown in Qin et al. (2018), that all strongly regularized cost functions degenerate into the same behavior.
The key result is how strong regularization makes the performance of MM-nsat and NS (and other cost functions) much more similar. Some form of regularization is often used to obtain strong results, since the introduction of various forms of gradient penalties such as the one used in WGAN-GP (Gulrajani et al., 2017) and spectral normalization (Miyato et al., 2018).
Discriminator regularization has many effects, such as reducing the ability ofD to learn quickly and smoothing the landscape of D’s outputs. It is difficult to fully understand how it affects the training process for GANs. Generally speaking, limiting the Lipschitz value of D (as is done explicitly by spectral normalization and partially by zero-centred real data gradient penalty) flattens the shape of D and ultimately limits the difference between the outputs for real and generated data. It follows that assuming thatD ≈ Dopt as given by eq 7 is less reasonable for a regularized discriminator, such that the effect discussed in section 2.3 will be less pronounced.
As discussed in section 2.1, the difference between MM-GAN and NS-GAN is only in the scaling factors, Dp and 1−Dp respectively. Strong regularization effectively squeezesDp towards 0.5: this makes Dp and 1−Dp more similar, and thus also MM-nsat and NS, as seen in fig 18. For the runs we have plotted, regularization vastly improves the results for NS, but the benefits for MM-nsat are much more limited. The common default value of the weighting for the gradient penalty, λ = 101, is actually somewhat harmful for MM-nsat compared to the unregularized case, and looking only at values for this regularization gives the misleading impression that NS is stronger than MM-nsat.
We consider a more thorough discussion of this problem out of scope for this work, but suggest that it might explain the fairly similar results for MM-nsat and NS when applied to the StyleGAN model (see W). Absent a better understanding of the interactions between discriminator regularization and cost functions, hyperparameters for regularization terms should be tested extensively when applying MM-nsat.

heading: S RING OF GAUSSIANS
text: The ring of Gaussians toy problem has a number of degrees of freedom, both for the dataset (number of modes, number of standard deviations of separation) and the model (architecture, batch size, training iterations). There are settings where both MM-GAN and NS-GAN cover or drop modes and significant variation between individual runs. We do not present a thorough analysis of this problem: in the main text, we use settings chosen to obtain the same qualitative NS-GAN behavior as shown in Metz et al. (2016); Srivastava et al. (2017).
To give some impression of the diversity of possible behaviors and the usual differences between NS-GAN and MM-GAN, we include results for a more challenging toy problem in figure 19 and table 3. In this case, where NS-GAN drops only some modes, we again find that MM-GAN has qualitatively better mode coverage. Furthermore, frequencies of samples from each mode is much better aligned with real data for MM-GAN than for NS-GAN. Interestingly, while NS-GAN places particular emphasis on avoiding generated samples outside of the real data manifold, more samples fall outside of the real data modes for NS-GAN than for MM-GAN.

heading: T MNIST-1K
text: We run tests on the Stacked MNIST dataset (Metz et al., 2016) (also known as MNIST-1K), in order to get a more demanding and multi-modal training task where we can still classify generated samples. The samples in this dataset are obtained by combining three samples from MNIST to represent each of the three color channels for an RGB image. We find a variety of network architectures difficult to train for any of our cost functions and resort to regularizing D. We show results in figure 20: in both cases, differences between linear combinations are minor. For the most heavily regularized model, there is no clear trend. Removing spectral normalization and relaxing the zero centred gradient penalty improves the models and makes them slightly more different, recovering the usual ordering in terms of DJS.
U CAT 1282
We run tests with CAT 1282 (Zhang et al., 2008), chosen as a reasonably difficult dataset which is still amenable to simple network architectures. FID during training for two architectures is shown in figure 21, emphasizing training stability. Figures 22a and 22b show samples for DCGAN with spectral normalization for D and self attention: these are shown cropped in the main text in figure 7. Figures 22c, 22d and 22e show samples for other network architectures. Note that samples are JPEG compressed.
We find that MM-unit and MM-nsat perform much better than NS in this setting. In particular, we find that the failure mode of NS cannot be addressed by early stopping in these experiments, unlike for MNIST, where NS tends to produce better results early on and mode dropping is mostly due to extended training (see figure 8). For CAT 1282, catastrophical mode collapse tends to happen before G has learned to produce reasonable samples. MM-nsat is much less susceptible to this failure mode than NS. As usual, results are most similar with regularized discriminator.
Figure 22a: Early stopping samples from best NS run for CAT 1282 using DCGAN with spectral normalization in the discriminator and self-attention (as in main text).
Figure 22b: Early stopping samples from worst MM-nsat run for CAT 1282 using DCGAN with spectral normalization in the discriminator and self-attention (as in main text).
Figure 22c: Final CAT 1282 samples for Conv-6 networks for 4 random runs. Left: NS. Right: MM-nsat.
Figure 22d: Final CAT 1282 samples for Conv-6-sn networks for 4 random runs. Left: NS. Right: MM-nsat.
Figure 22e: Final CAT 1282 samples for DCGAN-bn-sn networks for 4 random runs. Left: NS. Right: MM-nsat.

heading: V FFHQ AT VARIOUS RESOLUTIONS
text: Figure 23 shows results for training simple convolutional GANs on the FFHQ dataset downsampled to various resolutions, comparing NS-GAN and MM-nsat. The differences between the two cost functions are generally as seen in other experiments. Aside from the decent early stopping results for NS-GAN even at high resolutions the and relatively strong performance of MM-nsat even at 5122 resolution with very unsophisticated network architectures, these results only replicate previously discussed effects. Figure 24 shows samples from training at 5122, showcasing the mode collapsing behavior that causes FID to increase. Note that samples are JPEG compressed.
Figure 23: FID, an estimated distance between generated and real data based on Inception activations. For various resolutions using the FFHQ dataset, we plot the median values for three runs during training. The shaded area indicates maximum and minimum values. Results for FFHQ 10242 has been left out: neither MM-nsat or NS-GAN achieve FID values meaningfully better than randomly initialized networks. For resolutions from 322 to 5122, training collapse is universal for NS-GAN while MM-nsat is stable and achieves similar FID for low resolutions and much better FID for high resolutions.

heading: W STYLEGAN
text: We run experiments training StyleGAN (Karras et al., 2018), replacing the traditional NS loss used in the original implementation with our MM-nsat. Due to available resources, we train each model using only 4 GPUs. We make no other adjustments. We show results in table 4 and figures 25a and 25b.
The primary result from these tests is that MM-nsat performs reasonably well, albeit worse than NS-GAN, as a drop-in replacement in a sophisticated, state of the art implementation. In the full resolution case, which is carefully optimized by the original authors (for instance learning rate adjustments throughout training and a specific value of truncation for the latent input for G), NS achieves much better FID and MM-nsat samples seem to have more artifacts. For the lower resolution case, which is a somewhat more fair comparison, FID values are closely matched: NS gets the best final value, whereas MM-nsat has the best early stopping value.
Notably, StyleGAN uses zero centred gradient penalty and minibatch standard deviation features in D (see R). Both of these can serve to limit the failure modes we have shown for NS-GAN. For StyleGAN, coverage is limited, particularly for the latent truncation settings that achieve the best FID (Kynkäänniemi et al., 2019), and visual quality of samples is the first priority.
Implementing MM-nsat for StyleGAN requires some minor modifications, which essentially consist of accumulating values of Dp from parallels in the same way the original implementation does for gradients and rescaling the total gradient before passing it back to the parallels to update parameters for each network clone.

Review summaries:
This work proposes that many common issues with GAN methods are based on the weighting of the samples given to the generator’s objective function. They focus on a study of the original GAN objective proposed in Goodfellow et al. where the generator’s objective is the negative of the discriminators objective. The GAN community quickly observed that when the discriminator outperforms the generator with this objective, the saturating nature of the sigmoid function causes the gradients to vanish for the generator’s objective. For this reason, a new objective (NS-GAN) was proposed which modifies the generator’s objective to alleviate this gradient vanishing issue.  The authors argue that this modified objective is to blame for a number of common issues with GAN methods -- most notably the mode-dropping issue. The authors present theory that backs up these claims and they propose a new generator training objective which re-weights the gradients of the generator objective to have the same average magnitude as NS-GAN but have the same relative magnitudes of the original GAN objective. 

This paper reexamines the original (MM) and the non-saturating (NS) GAN objective.  The authors show that the gradients of the respective objectives just differ from a scaling factor depending on the discriminator's output for generated samples. While the scaling factor for the MM gradient is responsible for the well known vanishing gradient if the discriminator is optimal, the scaling factor of the NS gradient counteracts this saturation effect. However, on the other side, the NS scaling factor introduces a mode dropping effect and the inability of the learning dynamci to discover new modes. The authors show additionally that the NS minibatch gradient is the weighted sum of the single sample MM gradients with respective scaling factors as weights. These scaling factors avoid saturating, however, alter the direction of the resulting minibatch gradient. To counteract the change of the gradient direction the authors propose to summarize the sample scaling factors into one scalar for the batch gradient which preserves the non-saturating behaviour of the NS and the gradient direction of the MM objective. The new GAN objective is called MM-nsat.  Additionally the authors discuss the non-saturating effect of the ADAM beta2 parameter for the MM-GAN Generator. 

This paper proposes an explanation for mode collapse in the original GAN with the log -D objective for the generator (dubbed the non-saturating GAN or NS-GAN for short). The paper takes the approach of comparing the gradient of the generator objective for the original GAN with cross-entropy loss (dubbed the minimax GAN or MM-GAN for short) and the log -D variant. The key observation is that the difference between the gradient of the generator objective of MM-GAN and NS-GAN is that MM-GAN has a factor of D_p(G(z,\theta)), whereas NS-GAN has a factor of 1-D_p(G(z,\theta)), where D_p(G(z,\theta)) is the output of the discriminator on a sample generated from z. Hence, the terms in the MM-GAN gradient that appear fake to the discriminator are downweighted, whereas they are upweighted in the NS-GAN gradient. Because the samples from modes that are already overrepresented are likely declared fake by the discriminator, the contribution to the generator gradient is dominated by these samples in NS-GAN.