Sample weighting as an explanation for mode collapse in generative adversarial networks
Authors: Aksel Wilhelm Wold Eide
Eilif Solberg
Ingebjørg Kåsen
Venue:  (2021)
Keywords: GAN
generative adversarial networks
generative model
image synthesis
sample weighting
importance weighting
cost function
loss
mode collapse
mode dropping
coverage
divergence
FID
training dynamics
NS-GAN
MM-GAN
non-saturating
minimax

Abstract: Generative adversarial networks were introduced with a logistic MiniMax cost formulation, which normally fails to train due to saturation, and a Non-Saturating reformulation. While addressing the saturation problem, NS-GAN also inverts the generator's sample weighting, implicitly shifting emphasis from higher-scoring to lower-scoring samples when updating parameters. We present both theory and empirical results suggesting that this makes NS-GAN prone to mode dropping. We design MM-nsat, which preserves MM-GAN sample weighting while avoiding saturation by rescaling the MM-GAN minibatch gradient such that its magnitude approximates NS-GAN's gradient magnitude. MM-nsat has qualitatively different training dynamics, and on MNIST and CIFAR-10 it is stronger in terms of mode coverage, stability and FID. While the empirical results for MM-nsat are promising and favorable also in comparison with the LS-GAN and Hinge-GAN formulations, our main contribution is to show how and why NS-GAN's sample weighting causes mode dropping and training collapse.

Under review as a conference paper at ICLR 2021
SAMPLE WEIGHTING AS AN EXPLANATION FOR MODE COLLAPSE IN GENERATIVE ADVERSARIAL NETWORKS
Anonymous authors
Paper under double-blind review
ABSTRACT
Generative adversarial networks were introduced with a logistic MiniMax cost formulation, which normally fails train due to saturation, and Non-Saturating reformulation. While addressing the saturation problem, NS-GAN also inverts the generator's sample weighting, implicitly shifting emphasis from higher-scoring lower-scoring samples when updating parameters. We present both theory and empirical results suggesting that this makes NS-GAN prone to mode dropping. We design MM-nsat, which preserves MM-GAN sample weighting while avoid- ing saturation by rescaling the MM-GAN minibatch gradient such that its mag- nitude approximates NS-GAN's gradient magnitude. MM-nsat has qualitatively different training dynamics, and on MNIST and CIFAR-10 10 it is stronger in terms of mode coverage, stability and FID. While the empirical results for MM-nsat are promising and favorable also in comparison with the LS-GAN and Hinge-GAN formulations, main contribution is to show how and why NS-GAN's sample weighting causes mode dropping and training collapse.
HS
194-nst
linge
linge
307
P
301
Training spocts (batch 128, siamples-6tsoe)
Training spochs (batch 28-128, #samples-s00oe)
Hing
4 10P
10
2p
10r
2013
ain sdn sin
1500
6* 100
203 zin rón sin
1204
Treining spochs (batch 22-128, #tamples-6444)
"raining epochs (batch 520-64 #samples-02443)
Figure 1: Median Fréchet Inception Distance during training for ten runs on MNIST, CIFAR-10, CAT 128 and FFHQ 5122, using very simple convolutional GANs. The shaded areas show mini- mum and maximum value during training for the cost formulations. MM-nsat best overall, suffers less from gradual mode dropping and trains reliably the more challenging datasets.
INTRODUCTION
Generative adversarial networks have come long way since their introduction (Goodfellow et al., 2014) and are currently state of the art for some tasks, such as generating images. A combination of deep learning developments, GAN specific advances and vast improvements in data sets and computational resources have enabled GANs to generate high resolution images that require some effort distinguish from real photos (Zhang et al., 2018; Brock al., 2018; Karras et al., 2018).

---

Under review as a conference paper at ICLR 2021
GANs use two competing networks: generator G that maps imput noise samples mimicking real data, and a discriminator D that outputs estimated probabilities of samples being real rather than generated by G. We summarize their cost functions, Jn and Jg. for the minimax and non- saturating formulations introduced Goodfellow et al. (2014). We denote samples from real data and noise distributions by and and omit the proper expectation value formalism:
Daou (I,2) 2) (x, 2) log (o))) log(1- Dy(G(z)))
JGsau (z) = log(1 - (G(()=))
(1)
JG (z) log(()(((()))
For clarity, we use subscripts distinguish between the discriminator's pre-activation logit output Dr and the probability representation Dy:
D₂ = (1 exp(
(2)
Both formulations have the same cost function for D, representing the cross entropy between prob- ability estimates and ground truth. In the minimax formulation (MM-GAN), G is simply trained to maximize D's cost. Ideally, G matches its outputs to the real data distribution while also achieving meaningful generalization, but many failure modes are observed in practice. NS-GAN uses a mod- ified cost for G that is non-saturating when D distinguishes real and generated data with very high confidence, such that G's gradients do not vanish. (Supplementary:
Various publications establish what the different cost functions optimize in terms of the Jensen- Shannon and reverse Kullback-Leible divergences between real and generated data:
JGHN + 2 2Dss
(Goodfellow al., 2014)
JGsa Joss + D
(Huszár, 2016)
(3)
Jans - D - 2. (Arjovsky & Bottou, 2017)
Huszár (2015) and Arjovsky & Bottou (2017) have suggested NS-GAN's divergence as an expla- nation for the ubiquitous mode dropping and mode collapsing problems with GANs (Metz al., 2016; Salimans et al., 2016; Srivastava et al., 2017). While MM-GAN seems promising in terms of its Pensen-Shannon divergence, the formulation has largely been ignored because the saturating cost causes training to break down.
variety of other GAN formulations have been introduced, such as WGAN-GP (Arjovsky et al., 2017; Guirajani et al., 2017), LS-GAN (Mao et al., 2016) and Hinge-GAN (Miyato et al., 2018). Lucic et al. (2018) finds that different cost formulations tend to get similar results given sufficient parameter tuning. including various forms of regularization. Despite the questionable form of NS- GAN terms of divergences, it is widely used and can produce very impressive results, such as in the improved StyleGAN (Karras et al., 2019).
2
THEORY
2.1 MM-GAN SATURATION
The parameters a network are typically trained with some form of gradient descent cost function. We find the expressions for D's and G's gradicnts with respect to their parameters, & and d: (Supplementary: F)
Scaling facter as funciin dorniaawtttscomm
V HJ DAoaNs aDi(G(=).)) ()
1-d)
.D. ()(())),))
(D, Φ)
Figure 2: Scaling factors as a func- tion of the discriminator output, at (4) a scale emphasizing asymptotic be- haviors. MM-GAN's scaling factor causes G's gradient to vanish when (G())) - O, which corresponds to the maximum value of its cost.
+ Db
(1 D,(x,)))
(D((G(z,0)) 0)) D,,(((z,))
VoJasa
00 VoJam (D,(G(,,)) 0))
(1 (G
00
2

---

Under review as a conference paper at ICLR 2021
We emphasize the two kinds of factors for these gradients in red and blue: they plotted in figure 2. The discriminator's 's scaling factors decrease as it minimizes its cost, approaching towards the optima for both the real and the generated data term.
The minimax formulation Jc JD is suitable for adversarial training in terms of the gener- ator's optimum, but the unchanged scaling factor means that G's gradients increase towards and decrease away from its optimum. The saturation effect described in Goodfellow et al. (2014) is that Limp, (G()))+0 VaJGua O, such that G stops training when D is highly confident that its samples are fake. More generally, the scaling factor makes Ja concave with respect to D,, which interacts poorly with common optimization methods (see section 2.4).
As VaJGG and V VaJGss are the same aside from their scaling factors, the different behaviors of the two formulations must follow from these. NS-GAN's scaling factor avoids saturation, but gives rise to different, more subtle mode dropping tendency (see section 2.3).
2.2 NON-SATURATION AND SAMPLE WEIGHTING
As can be seen from eq 4, the NS-GAN and MM-GAN gradients are parallel for a single sample, but with different magnitudes. Stochastic gradient descent estimates the gradient of the cost over the entire input distribution by using a number of samples (a minibatch). We can express the NS-GAN minibatch gradient in terms of the MM-GAN gradient:
N-1
[ [
1 (GEa)) Dp(G(=t))
hhatch Σ
V
V Gras
ysumple
(zs)
(5)
Due to the bracketed factor, NS-GAN rescales the contribution from each sample relative MM- GAN, implicitly emphasizing samples with smaller values of Dr. Seeing as saturation is caused by the gradient's 's vanishing magnitude, this additional effect on the gradient's direction questionable.
The exact ratio of the minibatch gradient magnitudes for NS-GAN and MM-GAN depends on D/00(D((G(z))) for each sample and has no convenient expression. We can approximate it by replacing D₂ (G(z)) in eq 5 with its mean over the minibatch, D,, N Σ Ei allows us to formulate form of non-saturation for MM-GAN that mimicks NS-GAN:
N-1 D, (G(+)). This
LJ
batch Grastmai
D-Dp
Dp
N N-1
Σ i=D
sample Gsm
(z:)
(6)
Dp
We refer to the formulation with this generator gradient as MM-nsat. The relative weights of sam- ples in each batch are for MM-GAN, while the gradient magnitude approximates that of NS-GAN. Note, however, that the relative weights of samples may be disturbed across batches, such as when the minibatch size is small and Dp fluctuates. Despite different theoretical motivation, MM-nsat is very closely related to importance weighted NS-GAN (Hu et al., 2017). (Supplementary: D)
2.3 SAMPLE WEIGHTING AND MODE DROPPING
If we use r(z) and g(x) to denote the density of real and generated samples at a point in data space, the optimal discriminator (i.e. the function that minimizes JDAANs is given by:
r(x) (Goodfellow et al., 2014) PpP((a)) r(x) g(x)
This expression for the optimial discriminator assumes idealized conditions: that D is optimized for a fixed G. with unlimited capacity and without having to estimate the underlying real and generated data distributions by finite sampling. While D is not realized in practice (Sinn & Rawat, 2018), we use it to form some intuitions about D's behaviors.
Suppose we have real data with two disjunct, equiprobable modes, and that one of these modes, O, is overrepresented in generated data. For convergence, G would need to shift probability mass from O to the underrepresented mode, U. However, the minibatches used to update G will have more samples from O simply because they are generated more often. For a strong discriminator, these samples will also tend towards smaller values of (G(()))), due equation 7. This in turn
3

---

Under review as a conference paper at ICLR 2021
causes them to be weighted differently by NS-GAN and MM-GAN, because Dp(G(z)) appears in their scaling factors (eq 4). We refer to the first effect as over- and underrepresentation (of generated samples relative real samples) and the second effect as up- and downweighting (governed on the scaling factor).
The fundamental problem with NS-GAM can be seen by looking at how these two effects interact. The NS-GAN generator has a scaling factor 1 (G(())) which combines overepresentation with upweighting and underrepresentation with downweighting. allowing an overrpresented mode C to dominate U's contributions to the parameter updates. If parameter updates overwhelmingly based on gradients from O have an adverse effect on G's ability generate samples from U, this mode may become increasingly underrepresented, making O yet more dominant.
MM-GAN's scaling factor has the reverse behavior, pairing overrepresentation with downweighting and underrepresentaatonn with upweighting, such that the two effects combine in a stabilizing rather than destabilizing way. For the extreme example of nearly dropped or newly discovered mode, r(x) >> g(r) such that we expect (z)z) & for strong discriminator. NS-GAN's sample weight- ing disregards gradients from such samples, whereas MM-GAN's sample weighting emphasizes them.
This difference between MM-GAN and NS-GAN in terms of scaling factors reflects the different divergences they have been shown to optimize (eq 3). Huszár (2015) and Arjovsky et al. (2017) both connect the mode dropping tendencies of NS-GAN to its reverse Kullback-Leibe which strongly penalizes G for generating samples outside of the real data modes, but not for drop- ping modes. A variety of attempts to address mode dropping, mode collapse and mode hopping in GANs seem unaware that this is reasonable behavior for NS-GAN given its divergence: mode dropping minimizes D and maximizes DJS-
divergence,
2.4 MM-GAN INTERACTION WITH ADAM
GANs are generally trained with the Adam optimizer (Kingma & Ba, 2014), as was recommended by Radford et al. (2015) while introducing the DCGAN architecture. The parameter update step for Adam given by:
numoica
12"
A0,-
V 6
m 31 3/ + -1))))
mu 1 88
m
(8)
570
32000
10.00
2000
Figure 3: Update step size for an arti- ficial gradient 9r = (x.000))001)) with 31 0.99, Bn 0.999 and 1. In green and red, simulated and approxi- mated update magnitudes, using equa- tions 8 and 11. Updates vanish even while 9: >>
= -2222-1 + (1-32)g7 0: 1 8
Here, Si is the gradient at timestep L, and ri and G. are the first and second order exponential moving averages of the parameterwise gradients, bias-corrected to account for ccro-initialization. There are four hyperparamters: e is small constant primarily for numerical stability, is the learning rate and B1 and 32 determine the effective memories of the moving averages.
The fraction du resembles unit normalization of vector, and for constant gradient 9a =g (such Vir that the moving averages are trivial) update steps depend only on the sign of g. if [g] >>>>> E.
Ad, = sgn gg)
(9)
However, this normalization does not necessarily address MM-GAN's saturation problem, due to the training dynamics. D #oo where the real and generated data do not overlap (eq 7), such that if D can cleanly separate real from generated samples, we expect it to further decrease its loss by inflating its outputs. Supposing that D approaches this optimum linearly, i.e. Di(G(=)) at, we get:
(²())) As exp(at)
(10)
Dp also appears as the MM-GAN scaling factor (eq 4), such that G will be optimized with gradients
of the form gr go exp(at). For reasonable values of a, 3r and 82, the update step size for each of
4

---

Under review as a conference paper at ICLR 2021
G's parameters can be approximated: (Supplementary: G)
A = go exx(xx((a
2 log(52))t)
(11)
Given the commonly used B2 0.999, parameter updates will vanish exponentially i < 00.0005 given that Di] increases fast enough and conforms reasonably well to our simplified model. D learns to distinguish the real and generated data manifolds before they meaningfully intersect, this interaction between D, G and Adam threatens to freeze parameter updates altogether. (Supplemen- tary: L)
METHOD
3.1
COST FUNCTION MODIFICATIONS
In addition to our novel non-saturating version of MM-GAN (MM-nsat: eq 6), we normalize the gradient magnitudes of the original cost functions (MM-unit, NS-unit). Motivated by our model for MM-GAN saturation in section 2.4, we also test modifying the Adam 32 parameter for gives the expected results (MM B3 0.99). (Supplementary: H)
MM-nsat is the only of these cost functions that is interesting in its own right: the others are used to demonstrate various behaviors and highlight the roles of sample weighting opposed to gradient magnitude. Note that the prefixes MM and NS always correspond to the sample weighting used by the cost function. Furthermore, gradient magnitudes matched for MM-unit & NS-unit and approximately matched for MM-nsat & NS-GAN.
3.2 EVALUATION
Evaluating a generator generally difficult. addition visual inspection, we use Frechét Incep- tion Distance (Heusel et al., 2017), which we find informative also on MNIST, in spite of the feature extraction network being trained on natural images. Due our focus on the mode dropping prob- lem, we also use an unusual metric for datasets with class labels: compute the Jensen-Shannon divergence between class distributions in real and generated data. For G, we estimate its class distribution by drawing samples and using pre- traitrained classification network to label them. Note that this metric only sees class distributions and not sensitivite to the mode coverage or collapse within any given class. (Supplementary: I)
CD Dis(class frequencies class frequencies generator)
(12)
3.3 EXPERIMENTS
For the majority of our experiments we use very simple networks without tuning their hyperpa- rameters: either fully connected networks (FC) with fixed number of hidden units, or strided convolutional networks with kernel size and doubling the number of filters when the width and height halved (Conv). We use ReLU activations, except for the final layer, where G uses tanh activation with real data normalized to [-1, 1], while D uses sigmoid activation to map D; to Dp-
We make use of how fully connected networks are harder train than convolutional ones (Thanh- 'Tung et al., 2018) 8) and how training grows increasingly fragile for deeper networks and higher reso- lution datasets to find illustrative test cases. We make use of batch normalization (loffe & Szegedy, 2015) (bn), zero centred real data gradient penalty (Mescheder, 2018) (sgp) and spectral normaliza- tion (Miyato et al., 2018) (sn) only where explicitly mentioned. In addition to the simpler networks, test the full DCGAN (Radford et al., 2015) and StyleGAN (Karras et al., 2018) architectures.
We primarily use the MNIST (LeCun & Cortes, 2010) and CIFAR-10 (Krizhevsky et al., 2009) datasets to investigate the differences between cost functions. We run additional experiments using the CAT (Zhang et al., 2008; Volicoeur-Martinca Martineau, 2018) and FFHQ (Karras et al., 2018) datasets to study behaviors for higher resolution images.
5

---

Under review as a conference paper at ICLR 2021
Real and generated samples for ring of Gaussians tay data
MNIST FC early training gradients
P0al xp
65-
50
Nél-redt
-53
12-4
-1.8
Reat compins KS CAN
Net CAN
-38
2.0
-10 0.5
0.0
20
20
Figure 4: Real and generated samples on a classic toy 2D mixture of Gaussians dataset: a sample is here a coordinate pair. In blue, NS- GAN samples exhibiting mode dropping be- havior, here in the process of hopping between modes. In red, samples using the unmodified MM-GAN cost function. MM-GAN trains well this toy problem despite its saturating cost function and shows reasonable coverage of all modes.
Training epochs (batch d2z-1188 #samples-600000)
Figure 5: Gradient magnitudes in the early phase of training on MNIST with 4-layer fully connected networks. Top two: NS-GAN work- ing properly and unmodified MM-GAN in its saturating failure mode, ultimately freezing pa- rameters. Bottom two: MM-nsat, and MM- GAN with G's Adam parameter reduced B2 0.999 + 0.99, both avoiding saturation.
4 RESULTS AND DISCUSSION
4.1 QUALITATIVE PRELIMINARIES
Training GANs on a ring of Gaussians has been used to study both the mode dropping tendency of GANs (Metz et al., 2016; Srivastava et al., 2017) and how to address it. MM-GAN tends to do very well on such problems, as shown in figure 4. For this problem, it is easy to generate samples indistinguishable from real ones, limiting problems with saturation. MM-GAN more mode-covering in practice suggested by its divergence (Huszár, 2015; Arjovsky et al., 2017). (Supplementary: S)
Using MNIST and weak, fully-connected networks, replicate the well-known failure mode of MM-GAN that motivates the NS-GAN reformulation. In figure 5, we show the gradient magnitudes early in training, in particular a super-expomentially vanishing gradient for MM-GAN after roughly 3 epochs that halts training altogether. Additionally, we show that reducing 3g only for G's opti- mizer (effectively giving it shorter memory for the second order momentum) stabilizes the training process, as suggested by our theory on the interaction of MM-GAN and Adam. (Supplementary: I.)
For the same setup, we show samples the end of prolonged training for NS and MM-nsat cost functions in figure 6. We see that our version of minimax non-saturation trains well. The difference in terms of mode coverage is visually striking and corresponds well with our numerical evaluations for the same generators in table 1.
For MNIST, NS-GAN's mode collapse can mostly be addressed by early stopping or regulariza- tion. In figure 7 we show samples from training on the more challenging CAT 128² dataset. We find that MM-nsat trains well and generates fairly realistic samples, whereas for NS-GAN, catas- trophical mode collapse occurs before G learns to produce reasonable samples. More rigorous and quantitiative results are presented the supplementary material, particularly U and V.
4.2 QUANTITATIVE EVALUATION
Figure shows more comprehensive results for MNIST and CIFAR-10. We plot metrics throughout training to emphasize the dynamics for MM-GAN and NS-GAN, as well as variants of the same cost functions. There are two key results: that simple stabilization techniques allow us train GANs with

---

Under review as a conference paper at ICLR 2021
7
7
9
7
9
7
9
7
7
9
9
/
NS-GAN: FID
152. Dis CD
0.56
Early stopping NS samples (best run)
1
1
7
9
7
7
\
4
2
I
6
7
2
/
5
2
0
9
O
4
O
8
2
6
4
7
l
5
6
6
4
6
/
g
/
1
/
4
3
6
7
7
4
MM-nsat: FID = 49.7, DisCD = 0.15
Early stopping MM-asat samples (worst run)
Figure Generated samples after 1000 epochs for MNIST with 4-layer fully connected net- works. MM-nsat strikingly better in terms of mode coverage.
Figure 7: Samples from training runs for CAT 128 using the DCGAN architecture with spec- tral normalization in the discriminator and self- attention. All NS runs collapse in terms of di- versity, compared to only one of the MM-nsat runs. We show the least favorable comparison for MM-nsat, i.e. the run with latest collapse for NS and earliest collapse for MM-nsat, For each of these runs, we hand pick the best look- ing batch of samples generated during training.
Table 1: Class distributions for (see fig 6).
Cost
Class frequency in %
0
3
4
5
NS
0
MM-nsat
5
45
3
4
9
5
7
12
3
8
MM-GAN sample weighting, without the saturation issues that plague unmodified MM-GAN; and that their behavior is qualitatively different from those using NS-GAN sample weighting. Gradient magnitudes mostly influence training stability. Additionally, the plots on the right hand side show that the performance gain for minimax variants is reflected by more correct class distributions in generated data. In some cases, such as for 3-layer fully connected networks on MNIST, there is notable gap between DD values, even while FID is similar.
The importance of sample weighting is best demonstrated by comparing MM-unit and NS-unit. By construction, these variants have the same gradient magnitude, such that the only difference between the formulations is whether high- or low-scoring samples are emphasized when finding the total gradient for minibatch of samples. The results show that this difference crucial for the training dynamics.
Aside from stability, there are no clear differences between the variants with minimax sample weighting. While MM-nsat and MM-unit avoid saturation in very different ways, their results are highly similar, to each other and the results for unmodified MM-GAN when it does not saturate. This suggests that the approximation make in equation when introducing D, is of limited importance, and that MM-nsat more faithful to the original, logistic minimax formulation than NS-GAN.
The main issue with NS-GAN for these datasets is its strong tendency to deteriorate in terms of both FID and CD training progresses. There is early stopping point where NS is competitive with Mst in some cases, but not all. The results for CIFAR- 10 are the most extreme: NS and NS-unit consistently undergo catastrophical mode collapse.

---

Under review as a conference paper at ICLR 2021
MNIST DyCO ovrnng tminkig
Pa
137
106
130
200
MNSTTFC DyCD during trinking
105-
HS unit
NO
NV-uni
HMM - 0.99
VM - 0.99
W04-rrstt
NV noor
107-
20-1.
z60
1900
260
400
abo
3000
Training epachs (barch S:20-18, 128, #samples--B0 (00000
Training apachs (batch size-128, #iampis-60000)
durig traniing
106
NS
HS unit
No
writ
20-1
NV-uuit
HMM - 0.99
VM - 0.99
0
30a-
H84-rrst
NV yoor
20-2
300-
10-3
260
800
1900
260
400
alo
3000
Training epochs (barch #20-128, Samplee--00000
Training epachs (batch size-128. samp kam0ps--6000)
DyCO ouring training
105
NS
NS unit
NO
unit
NV-unt
HM jo 0.99
VM 2000
NM yoor
10"-
10-1
260
600
800
1900
260
400
810
30000
Training epochs (barch 128, *sam ssamples- 50000)
Training opachs (batch size-128. samples-50000
8x
10F
traning
NS
NS unit
vet
No
HM Du
MM 2i 0.00
0 5xx 300
NY moor
10-1
4x 300
200
<b2
600 450
1000
260
400
600
350
3000
Training opachs (batch 1ize-125, 0 #a00p
Training opochs (batch size-128. zamp 50000)
Figure 8: Left: FID, an estimated distance between generated and real data based on Inception activations. Right: DDsCD, the Shasnnn- divergence between class distributions in generated and real data. For various networks using the MNIST and CIFAR-10 10 datasets, we plot the median values for ten runs during training. The shaded area indicates maximum and minimum values. NS and MM are the traditional cost functions; variants use normalized gradients, reduced B2 for G's Adam optimizer and the non-saturating rescaling from eq 6.
8

---

Under review as a conference paper at ICLR 2021
Overall, these results validate theory from the literature showing that MM-GAN and NS-GAN cor- respond to different divergences and the importance of sample weighting as described in section 2.3. We present variety of additional results in the supplementary material (M through W), both testing the validity of our theory in different experimental settings and showing results for higher dimensional datasets.
CONCLUSION AND FURTHER WORK
Based on thorough, theoretical analysis of the differences between the formulations intro- duced in Goodfellow et al. (2014), we have designed a form of non-saturation for the minimax cost function for GANs that rescales the minibatch gradient. This corrects the training difficulties for the original minimax GAN, without the side-effects inherent to the NS-GAN reformulation. Run- ning experiments, we have shown that our new stabilization has qualitatively different behavior, in particular better mode coverage as indicated both by our theory and by previous works showing which divergences are optimized by NS-GAN and MM-GAN, Perhaps most importantly, our work is an important correction to the view that NS-GAN is just MM-GAN corrected for its saturation problems, as the name might suggest.
We have shown promising results with MM-nsat (our gradient rescaled version of MM-GAN), but is primarily designed for demonstration purposes. Results with simpler networks on MNIST and CIFAR- 10 are much stronger for MM-nsat compared to NS-GA NS-GAN, Hinge-GAN and LS-GAN and experiments on higher resolution images from the CAT and FFHQ datasets show that MM-nsat has less issues with catastrophic mode collapse during the early stages of training, greatly reducing the stability issues that GANs tend to suffer from.
Interactions with discriminator regularization
unclear and important for more advanced appli- cations (see R). Performance when combining MM-nsat with various designs that directly address mode dropping remains be determined, for instance the class conditioning (Mirza & Osindero, 2014) used in BigGAN (Brock et al., 2018) and the minibatch standard deviation used in StyleGAN (Karras et al., 2018). Unlike the normalizations and regularizations that mask the underlying issues with NS-GAN and often introduce extra hyperpameters, MM-nsat changes only the cost function.
While we have focused on variants of MM-GAN and NS-GAN, with the convenience that these all work with the same discriminator cost function, we expect analysis in terms of sample weighting to apply much more generally, explaining training dynamics and tradeoffs between individual sample quality and overall sample diversity also for other GAN formulations.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Martín Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. CoRR, abs// 00462,, 2017. URL http: http///axxvv.o//// hatp//ab//// 2abs/1700 /1001. C4862.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp- 214-223, International Convention Cen- tre, Sydney, Australia, 06-11 Aug 2017. PMI.R. URL http: ppwcceddoeeeeeeo nga. mmr.. m.r.ppre prrss/v70/ arjovsky17a, htm. 2, 2,4,633 6.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. CoRR, abs/1809.1 1096, 2018. URL http: http:///rxiv org/ httg//bb/18800 11096. 1, 1,
Tong Che, Yanran Li, Ruixiang Zhang. R. Devon Hjelm, Wenjie Li, Yangqiu Song, and Yoshua Bengio. Maximum-likelihoodd augmented discrete generative adversarial networks. CoRR, abs/1702.07983, 2017.
URL http: /ttp:a/axvvvv.//////ee///aa
Ian Goodfellow, Jean Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Ghahramani, Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.). Advances in Neural Information Processing Systems 27, pp. 2672-2680. Curran Associates, Inc., 2014. URL http:/ / /ppapprs..... www.:npss..ccoooo wttpppppccppppp///22..22 generatives wdversariaa
pdf. 1, 2. 9, 12, 20
Ishaan Guirajani. Faruk Ahmed, Martín Arjovsky, Vincent Dumoulin, and Aaron C. Courville. Improved training of wasserstein gans. CoRR, abs/1704.00028, 2017. URL http:/ /ttp:://araa /arxiv org abs/1144. 00029. 2,31, 34
Kaiming Нe, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1512.03385, 2015. URL httt:://t///tleeee www.awwwww...aaaeaooooooooooeeee wwt/bb////////o//e/ aa///bb//11/ /1512.111.. 03385. 17
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Günter Klambauer, and Sepp Hochreiter. Gans trained by a two time-seale update rule converge to nash equilibrium. CoRR, abs/1706.08500, 2017. URL http:////// /ttp:///axvvvvv/e org/ wwtp////////////////////ae/e/e// a250/170...80000 5
Devon Hjelm, Athul Paul Jacob, Tong Che, Adam Trischler, Kyunghyun Cho, Yoshua Bengio. Boundary- seeking generative adversarial networks, 2018. 13, 14
Zhiting Hu, Zichao Yang, Ruslan Salakhutdinov, and Eric Xing. On unifying deep generative models. CoRR, abs/1706.00555, 2017. URL http::tt///. http:///xxvvv//r/e /abs/ /10.. 111.117760...0.... 3, 13, 14
Ferenc Huszár. An alternative update rule for generative adversarial networks. vo/an-ale wwwatterntiiiie. www.ettttttaatteeeeeeee------eee Accessed: 2019-11-07. 2, 26
netwokss http::ttttttt..tt. /ttpppp:://www..ee www.w.i..eene.....mmm 2016.
Ferenc Huszár. How (not) to train your generative model: Scheduled sampling, likelihood, adversary? 2015. 2, 6,
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. CoRR, abs/1502.03167, 2015. URL http: arxiv. org/ 031.00003311111.... 03167. 5, 17
Alexia Malicceur-Martinene
The relativistic discriminator: a key element missing from standard GAN. CoRR, 11/11/117000111111
abs/1807.00734, 2018. URL http:://axii..rrreee/
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. CoRR, abs/1811..0494, 2018. URL wwttttpp//aaate httl:t//xax/..c.oo wwtp///////aaaeaaeeee..asaa aalaab///81/ /1112. 04948. 1, 9,
Tero Karras, Samuli Laine, Miika Aittla,, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan, 2019. 2
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/ 2014.
URL http:: arxiv. o abs/ 1412. 6980. 4
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar- 10 (canadian institute for advanced rescarch). 2009.
URL http:://t/wttt
wow.ntoroooooooooo... edu/ -kriz ci sirar. html. 5
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. CoRR, ahs/1994.0699) 2019. 46
10

---

Under review as a conference paper at ICLR 2021
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http::/yaannnn.ttt lecun. com/ wxxdb/mnist/.
Mario Lucic, Karol Kurach, Marcin Michalski. Sylvain Gelly, and Olivier Bousquet. Are gans created equal? large-scale study.
Bengio, H. Wallach, H. Larochelle, K. Grauman, Cesa-Bianchi and R. Garnett (eds.), Advances Neural Information Procesing Systems 31, pp. 700-709. Curran Associates, Inc., 2018. URL http: / /pppprs......p//pp/./....e/eee.. pepers.nnipe. nips. large- wewesscaleessseee study. study.pp..2
-e03550-aee---e gans creeteddeeeduaeeeeeeee
Xudong Mao, Qing Li, Haoran Xie, Raymond Y. K. Lau, and Zhen Wang. Multi-class generative adversarial networks with the 1.2 loss function. CoRR, ahs/1611.04076 2016. URL http::::////w...ooggpgeeoee 1611.80 04076. 2, 12 31
Lars M. Mescheder. On the convergence propertics of GAN training. CoRR, http: / hwwp:/org/aaaooe.ooe
2018. URL
64406. 5,
Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled generative adversarial networks. 2016. 2, 6, 36,
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. CoRR, abs/1411. 2014. URL http:// // arxiv www.iii..rgab////eee /wt////sss11.......// 1784.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. CoRR, abs/ N0s.18022,5957, 2018. URL http:://axxiii htcp:://aaxlv.orggbb/////eeeeeee 05957. 5, 12, 25, 31, 34
Yipeng Qin, Niloy J. Mitra, and Peter Wonka. Do GAN loss functions really matter? CoRR, ahs/ 811,09677 2018. URL http: a rxf wwwgg/ba/////// aba//811/111 09567. 34
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation Icamning with deep convolutional generative adversarial networks. CoRR, ab3/1511.06434, 2015. URL 06434. 5,5
absp://ass/lseee 11511
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans, 2016. 2
Samarth Sinha, Zhengli Zhao, Anirudh Goyal, Colin Raffel, and Augustus Odena. Top-k training of gans: Improving gan performance by throwing away bad samples, 2020. 14
Mathicu Sinn and Ambrish Rawat. Non-parametric estimation of jensen-shannon divergence in generative adversarial network training. In Amos J. Storkey and Fernando Pérez- (cds.), International Conference on Artificial Intelligence and Statistics, AISTATS 2018, 9-11 April 2018, Playa Blanca, Lanzarote, Canary Islands, Spain, volume 84 of Proceedings of Machine Learning Research, pp. 642-651, PMLR, 2018. URL http:/ / http://ccccceeeeccccccccc
mlr press v84 inn Ba. htmi.
Akash Srivastava, Lazar Valkov, Chris Russell, Michael U. Gutmann, and Charles Sutton. Veegan: Reducing mode collapse in gans using implicit variational learning. 2017. 2, 36
Hoang Thanh-Tung Truyen Tran, and Svetha Venkatesh. On catastrophic forgetting and mode collapse in generative adversarial networks. CoRR, ahs/1807.04415, 2018. URL. httpttt://axxi..rrg../.
1807.
04015. 5
Han Zhang, Goodfellow, Dimitris N. Metaxas, and Augustus Odena. Self-attention generative adversarial networks. CoRR, abs/1885.088311 2018. URL http::tttt/aaeeeee http::///awwwwe.eaeaaeeeee http://aaa1185..83333 08318.
Weiwei Zhang, Jian Sun, and Xiaoou Tang. Cat head detection how to effectively exploit shape and texture features. volume 5305, pp. 802-816, 10 2008. doi: 0-77997778---4-88869-44 59. 5,
11

---

Under review as a conference paper at ICLR 2021
Table 2: Overview of GAN cost formulations.
Cost
Reference
D cost
G sample emphasis
G gradient magnitude
MM-GAN
Goodfellow et al. (2014)
Cross-entropy Cross-entropy
High-scoring
Saturating
MM-nsat
ours
High-scoring
R. Non-saturating
MM-unit
ours
Cross-entropy
High-scoring
Unit-normalized
MM 82 = 0.99
ours
Cross-entropy
High-scoring
Saturating, adj. Adam
NS-GAN
Goodfellow et al. (2014)
Cross-entropy
Low-scoring
Non-saturating
NS-unit
ours
Cross-entropy
Low-scoring
unit-normalized
LS-GAN
Mao et al. (2016)
Quadratic
?
Hinge-GAN
Miyato (2018)
Clipped linear
Uniform
SUPPLEMENTARY MATERIAL
TABLE OF COST FUNCTIONS
Refer to table for an overview of the cost functions used this paper. Note that sample emphasis is not directly comparable between formulations with different costs for the discriminator.
B INFORMAL AND EXTENDED SUMMARY
We include extended summary of the most central points from our work. This presentation is intended to be less technical and somewhat more accessible to a wider audience.
When training a classifier, the standard cross entropy loss has stronger gradients for samples with greater error. This means that correcting misclassifications is emphasized over further increasing the confidence level of correctly classified samples. Training usually runs for multiple epochs, giving the classifier repeated opportunities to readjust if its cost increases for any particular sample.
Like for classifiers, every term in the NS-GAN cost functions for the discriminator and generator emphasizes the samples which are furthest from their optima. While this seems reasonable, it does not necessarily interact well with adversarial training.
To minimize its cost when training on equal amounts of real and generated data, the discriminator Dp(#) approximates P(y is really x)). This means that D, & for real data that G does generate and D, z 0 for generated data outside of the real data manifold. More generally, (x) samples.
approximates the density of real samples relative to the of densities of real and generated
The generator cost function includes real data term which is often omitted: without any gradient with respect to G's parameters it does not influence the training. This makes it difficult for G to act on dropped modes, because its updates only see D's gradients the points data space where G generates samples.
The degree which NS-GAN's generator cost emphasizes samples further from their optimuum is quantified by the scaling factor given in equation 4. NS-GAN weights contributions from each generated sample by a factor Dp, such that low-scoring ("unrealistic") samples contribute the most to the gradient.
MM-GAN's generator instead uses the scaling factor Dp, emphasizing high-scoring ("realistic") samples. MM-GAN saturation follows directly from this scaling factor: for highly unrealistic gen- crated samples, Dp > O, and this weighting causes G's gradient vanish. Adam does not work around this problem with the usual hyperparameter settings.
In either case, G's parameters are updated by sampling the gradient of D with respect to G's pa- rameters points where samples happen to be generated. This means that the updates see most contributions from regions where the density of generated samples is high.
The main problem with NS-GAN is how this sampling frequency interacts with its scaling factor. Regions on the real data manifold where the density of generated samples is too low tend towards small values of D, these generated samples considered realistic by D and thus have small
12

---

Under review as a conference paper at ICLR 2021
error for G' and are weighted down by the scaling factor. Since this region in data space is also sampled rarely, due to the G's low density, its contribution to G's parameter updates will be small. At the same time, regions with too high density of generated samples have large scaling factors and are sampled frequently. In a situation where different modes disagree on the hest configuration of G's parameters, undersampled modes will be massively outvoted.
This would not be a problem if D's gradients were able to push the density of generated samples away from oversampled modes and discover undersampled or entirely dropped modes. However, we have no guarantee that this sort of action at a distance will work in practice: the literature instead shows that mode collapse is a recurring problem, likely because D's gradients fail be informative across longer distances in data space.
Sample weighting also explains why MM-GAN behaves much better in terms of mode coverage. Since MM-GAN uses the opposite scaling factor, Dz. it boosts the contributions to parameter up- dates from undersampled modes. The MM-GAN scaling factor partially cancels against the sam- pling bias effect, making the parameter update less dependent a single, dominant mode. While a generated sample in an undersampled mode not itself an error, the very high value of Dp for this generated sample reflects a large discrepancy between generated and real densities at this point. MM-GAN's ' sample weighting makes it better able to address errors of this sort.
The main problem with MM-GAN is that its gradient decreases when the cost is increases. This makes it very difficult to optimize, also with Adam. NS-GAN, the established solution to this prob- lem, changes both sample weighting and gradient magnitudes. The problem with a very simple rescaling of the minibatch gradient, as shown in eq 10. We call the resulting for- mulation MM-nsat it combines minimax sample weighting with the same form of non-saturation as used in Goodfellow's s NS-GAN, The key difference is that the NS-GAN non-saturation is implic- itly applied each sample, disturbing sample weights, while the MM-nsat rescaling is explicitly applied to the minibatch gradient as a whole.
instead be addressed
Our experimental results validate that this minibatch gradient rescaling corrects the saturation prob- lem, and comparing NS-GAN and MM-nsat allows us to show that sample weighting is highly im- portant for training dynamics. Mode coverage and general stability greatly improved for MM-nsat relative to NS-GAN, which sometimes suffers from gradual mode dropping throughout training, and sometimes from catastrophical mode collapse early in training. Note, however, that strong models already tend to include various designs to counteract the problems caused by NS-GAN's unfortunate sample weighting, limiting the benifits of simply replacing NS-GAN with MM-nsat.
C GOODFELLOW'S MOTIVATION OF NS-GAN
We include Goodfellow et al. (2014)'s motivation for NS-GAN for easy reference: "In practice, equation| I [minimax cost] may not provide sufficient gradient for to learn well. Early in learning, when G is poor, D can reject samples with high confidence because they are clearly different from the training data. In this case, log(1 - D(G(2)) saturates. Rather than training G to minimize log(1 D(G())) we can train G to maximize log D(G(=)). This objective function results in the same fixed point of the dynamics of G and D but provides much stronger gradients early in learning."
D IMPORTANCE WEIGHTING
Hu al. (2017) studies the parallels between generative adversarial networks and variational autoen- coders. They transfer the idea of importance weighting from VAE to GANs, arriving at an update rule for the generator (Hu et al. (2017): equation 22) which reweights G's NS-GAN gradients for each sample the minibatch with normalized weighting factor. Closely related ideas are seen in Hjelm et al. (2018) and Che et al. (2017). The expression for the unnormalized weighting factor is given as:
1% (y (x:)
w; Loo (y) xx)
(13)
Here, Joo and %% represent the discriminator and its reverse respectively, i.e. Lo (v/x) goo vlx), where y represents the true label of a sample x. In our notation, this is corresponds to the
13

---

Under review as a conference paper at ICLR 2021
following, as given explicitly in Hjelm et al. (2018):
W; Dj Dp(G(z))
(14)
(,(G())))
Comparing this weighting factor to the bracketed factor in our eq 5, it clear that it perfectly cancels against the reweighting of NS-GAA relative to MM-GAN. In other words, importance weighted NS- GAN actually recovers the same original MM-GAN sample weighting that is used by MM- -sat,, such that our approach of avoiding the NS-GAN sample reweighting leads us to rediscover this update rule.
The remaining difference is that of the gradient magnitude. When reweighting the NS-GAN gradi- ents, Hu al. (2017) and Hjelm et al. (2018) choose normalize the weights over the minibatch: omitting to do this would make their importance weighting also recover the MM-GAN gradient magnitude, and thus reintroduce the MM-GAN saturation problem. They normalize by calculating the actual sum of the reweighting factors, which is slightly different from our approximation where we replace D, with its mean over the minibatch (eq 6). Note that neither method is exact unless the samplewise gradients in the minibatch have the same magnitude. As seen by for instance comparing MM-nsat and MM-unit, which have highly similar performance despite massively different gradient magnitudes, these diferences should have little effect on training dynamies.
The primary difference between importance weighthing for NS-GAN and our design of MM-nsat is the logical steps and the theoretical motivation. In work. the direct relationship to the original MM-GAN update rule is made explicit, whereas importance weighting goes the cirucuitous route of first taking the heuristic NS US-GAN formulation in place the original MM-GAN formulation, then reweighting it to arrive at an importance weighted NS- DS-GAN formulation, which our work shows to actually be just non-saturating version of MM-GAN.
CONCURRENT WORK
A concurrent work recently made available as a pre-print, Sinha al. (2020), makes use of simple design which is directly comparable to our method. When updating G. they ignore the gradients aris- ing from the lowest scoring generated samples, where the fraction of the samples ignored increases throughout training. This effectively shifts weighting from low-scoring to high-scoring samples, which the same overall effect as achieved by using our MM-nsat gradient. Their method is much simpler and less principled as to exactly how low- and high-scoring samples are weighted, but it is also more easily applied other GAN cost formulations.
Similar to our findings, Sinha et al. (2020) observe an increase in mode coverage and overall quality. By investigating the cosine similarity between low- and high high-scoring samples, they observe that optimization conflicts between these, as we suggest in section 2.3. Their analysis in terms of pushing samples towards and away from modes is somewhat at odds with ours, where we note that low- scoring samples can be high-quality, but lie in modes that are significantly oversampled by G. The discriminator score is not a direct measure of the sample's quality, but rather of relative densities of real and generated data, such that we should not expect D's gradients to necessarily point towards the centres the real data modes.
F GRADIENTS DERIVED FROM COST FUNCTIONS
Finding the MM-GAN gradient for a single sample is a matter straightforward calculus. The expression for NS-GANT gradient can be found using the same approach. First note that the rela- tionship between Dp and Dj is given by eq 2, such that:
aDp = DJ eexp(-D) aD;
(15)
exp(-Di) (D) = 1-Dg Dp
(16)
We take JGG = - Jp (eq 1) as the starting point, as defines the minimax formulation. Note that the first term does not depend G 0, such that it can be omitted in the G cost function:
14

---

Under review as a conference paper at ICLR 2021
JGrm (zg(D((x)) + log Dp(G(z,0))
V iDp(G(2,0)
00
Dp(G(2,,))
aDi(G(z, 0))
00
0Di
1- -D2(G(2,0))
aDi(G(z, 0))
D3 epeepp-1 (Di)
(17)
00
1-Dp Dp
aDi(G(z, 0)) Dp(1 Dp)
00 (1 Dp)
aDi(G(z, 0)) D,(G(zz,0)
00
It might seem counterintuitive to mix Dj and Dp when expressing the gradients, instead of using for instance the expression on line of equation 17. However, the non-linear behavior of aD,/06 makes interpretation difficult. As Dp approaches either 0 or 1, increasingly small changes in Dp reflect the same change odds ratios.
less clear from the gradient terms of only D, that the MM-GAN gradient saturates D2 approaches 0, or that it does not diverge as Dp approaches 1. However, for the scaling factor itself, we find the D, formulation most intuitive. The interpretation suggested by eq is particularly convenient.
MM-GAN INTERACTION WITH ADAM
We include additional details for the approximation of the Adam update step. We combine equation 8 and 9r Go exp(at). For simplicity, we assume that sgn(gt) does not depend on if the sign of the gradient changes, my is reduced and v, unaffected. This can contribute to making Ad, vanish, but cannot prevent it.
This allows us to express the momentum as a sum of contributions from each previous time step i.
my Σ 3 -B19g0 exp(ai)
(18)
For reasonably large values of and assuming a * log(31), we can approximate this sum by definite integral:
i=t
R
(1 - 31)3 9u exp(ai)di
1=0
(19)
31
0 log( 90 (0(exp(al) - 84
Using this approximation (similarly also for V,), assuming to be negligible and packing assorted time independent constants including aa into C, we get the update step:
Ad, C /1 B/ 1- B (2xp)2al) - exp(log(32)1))
exp(al) exp(oo((()))
(20)
Having already assumed to be large, we omit the first fraction (bias corrections) for simplicity:
A0, ~ C exp(at) - exx(o))))
(21)
exp(2al) exp(log(33())
For sufficiently large values of behavior is determined by which of the terms in the numerator and denominator have the largest constant in the exponent (i.e. slowest decay). We enumerate all the
15

---

Under review as a conference paper at ICLR 2021
four possible cases:
a> e>log(31) A log( + 20, x
B1
<< log(3)) A log( 7 A
2
(22)
a log(Bi) Aa < log((2) Au, cla log(&z))
a< log(())
log (32) Adl ((og(()))a))+
Using hyperparameters such that 81 > vB2 is highly unusual. It is required satisfy the inequal- ities for the last case and means that the base is greater than one in the second case, In both cases, get exploding update steps, which is generally undesirable.
Assuming 31 < vB2, the first three cases show intended behavior: update steps are normalized for constant or exploding gradients, while they diminish for vanishing gradients, enabling convergence. The third case, a < 1 log(32), is discussed in the main text. In the second case, where (S)( get different kind of vanishing behavior governed the relationship between B1 and B2.
2
Additional results and discussion for this problem is given in L.
IMPLEMENTATION DETAILS
For consistency with the theoretical framework and allow testing of magnitude normalized gradi- ents, we modify GANs by rescaling the generator gradient. This amounts to calculating the mini- batch gradient explicitly and multiplying it by a factor R calculated as given in algorithms and before passing it on to the optimizer. (Code repository: J)
Algorithm Non Non-saturating minimax rescaling factor (MM-nsat)
Require: G, Dy: generator and logit valued discriminator functions
Require: N: the minibatch size
Require: 20.-.-, 2N-1: N generator inputs
Require: ex: small constant, prevents zero-division overflow such that Rmax E
D N 2: -i=N-1 + exp (xp())((G())))
2: + 1-D,
En+Dp
Algorithm 2 Unit rescaling factor (MM-unit, NS-unit)
Require: Vo: the gradient of the cost function with respect to G's parameters
Require: No: the number of parameters of G
Require: <R: a small constant, prevents zero-division overflow such that Rmaa N
1: V 0
( No 42 7
2: VR+
However, for proper behavior with penalty terms as well as convenience, we strongly recommend implementing MM-nsat by multiplying the minimax cost by factor which depends on (GG(z)) for the entire batch and disabling back-propagation through this scaling factor. In TensorFlow, straightforward implementation
cost mm nsat tf. stop_gradieettttt.eee (R)
cost_mm mm
(23)
The rescaling factor is constant across the minibatch and can be applied either samplewise or to the
total cost.
16

---

Under review as a conference paper at ICLR 2021
For MM-unit and NS-unit, the rescaling factors unfortunately cannot be calculated without first computing the gradient for the unmodified cost functions.
Applying gradient rescaling directly does not interact well with penalty terms. When the MM-GAN cost saturates, the gradient will typically be dominated by the penalties, causing both MM-nsat and MM-unit rescaling inflate their effect. Using the approach above, rescale the adversarial minimax cost and leave penalty terms unchanged.
Parallelization may raise some issues when calculating the rescaling factor R. because proper min- imax behavior requires R to be calculated across all parallels and a naive approach will introduce some overhead. Ideally, D(G(=)) and gradients from parallels should be aggregated in the same step, in order to rescale gradients before running them through the optimizer. Correctly imple- mented, the overhead should be unnoticeable for multi-GPU training.
We note that for very small batch sizes, or where R for other reasons vary significantly between subsequent batches, MM-nsat will have some NS-like behaviors, because weighting between sam- ples in different batches is disturbed by the difference between their values of R. However, not even unmodified MM-GAN weights properly across batches, due to updates to and optimizer corree- tions. It is possible to smooth R across batches using for instance an exponential running mean, at the risk of reintroducing saturation issues.
In algorithm 2, we make the unintuitive choice of normalizing the gradient not to unity, but to No, the number of network parameters. Due to cancellation effects in Adam's 's parameter update step, this time-independent scaling factor is only relevant when 9t is small and non-negligible. Because Adam's ' Ad, is calculated for each parameter independently, this choice of normalization prevents introducing inadvertent relationship between the number of parameters of a network and whether updates vanish due to as shown in eq 9.
I CLASS DISTRIBUTION DIVERGENCE
Implementing a class distribution divergence requires us to have labels for the real data and some way to determine which class generated samples belong to. This limits its applicability to simpler problems where we can train strong classifiers. An additional complication is that we are applying this classifier to generated data, where it may not generalize well. The classifier is calibrated for real data samples and its behavior is not well defined outside of this manifold. In particular, a standard classifier is compelled by design to assign class to all samples, even when they are poor that they cannot meaningfully be said to belong to any class at all. It is conceivable that a generator producing only noise-like samples will be assigned functionally random classes and achieve very good DD. Furthermore, DjsCD CD is not sensitive to the degree of coverage or collapse within any given class.
In practice we find that a poor generator invariably scores badly in terms of DısCD, with nearly all its samples being assigned to the same class, and that FID and DjsCD are strongly correlated. Nonetheless, combining Djs CD with standard generator evaluation method such as FID is recom- mended, and for poor generators, DD may be misleading. Similarly, the class divergence serves as a control for our use of FID for MNIST, which is significantly removed from the natural images FID's Inception network in trained on.
For interpretation, note that we use normalized The lower bound corresponds to perfectly matched distributions and increasingly mode dropping generators will approach the upper bound. When the real data distribution is non-7ero for all classes, (D) max < 1. For the case where there are ten evenly distributed classes in real data, which includes MNIST and CIFAR-10, we get (D) max 0.758.
divergence, such that Djs E [0, 1].
For classification, we train a simple, convolutional network for MNIST and a ResNet (He et al., 2015) based convolution model with batch normalization (loffe & Szegedy, 2015) for CIFAR-10 with approximately 99% and 89% validation accuracy (see section J). For MNIST-IK, we apply the MNIST classifier to each channel separately, treating each ordered triplet of classifications as its own class. In line with the established standard for FID, we use 50000 samples for estimating DD for generator.
17

---

Under review as a conference paper at ICLR 2021
J CODE AND NETWORKS
We have made code for Python 2.7 and TensorFlow available in a GitHubbrepository: to maintain anonymity, see uploaded supplementary material instead. Qualitatively reproducing the MNIST and toy experiments in the main paper is straightforward, but other datasets as well as running the evaluation metries involves additional setup (see repository for details).
The repository includes code for training classifiers for DjsCD for MNIST and CIFAR-10. Step by step setup for the CAT and FFHQ datasets is not included, nor the forked version of StyleGAN for training it with MM-nsat.
18

---

Under review as a conference paper at ICLR 2021
K GRADIENT NORM AND DIRECTION FOR MM-NSAT
Figure shows how the gradients obtained by the NS-GAN and MM-nsat generator costs differ during training, both in terms of norm and direction. These results illustrate the theory from section 2.2 and give an impression of the quality of the approximation used for MM-nsat (eq 6), in order to renormalize its gradient magnitude of MM-GAN match with that of NS-GAN.
For 4-ayer convolutional networks, FID improves for both formulations with very similar quantita- tive results for roughly 200 epochs. During this period, gradients are also the most similar in terms of norm and direction. Afterwards, cosine similarity drops, indicating that gradients are becoming increasingly different for the purpose of optimization, and NS-GAN and MM-nsat diverge in terms of FID due to NS-GAN increasingly dropping modes.
For 4-layer fully connected networks, cosine similarity drops much earlier, and FID for NS-GAN falls behind very early in training, reflefeting mode dropping (see DjcCD for FC-4 in figure 8). During the later stages of training, gradient norms diverge, particularly for the case where we train with the NS-GAN gradient. Note, however, that this is a stage of training where NS-GAN shows very pathological behavior, with more than of generated samples representing the digit 1.
For interpretation of the cosine similarity, note that gradients are high dimensional vectors, such that near-orthogonality unexceptional. Gradient magnitudes are not perfectly matched for NS-GAN and MM-nsat, particularly not later in training where values of Dp become more extreme and NS- GAN starts falling off in terms of FID. When training with the MM-nsat gradient, its norm relative to the WS-GAN gradient are matched to within an order of magnitude and allows MM-nsat to train without any apparent issues.
MNIST Conv-4 NS-GAN: digg nosticccs
MNIST Conc-d MMAsaat: diagnestics
10a
10
FID
Rl 1it raal / [Rlc
R10
Rls nat- / [Rle usl
(al MM auz. 0/g
nillaily(/ Me. ret. V/ ucl
-2.5
2.0
2.0
102
102
FID
1.0
1.0
10.
0.5
10¹
0.5
200
400
600
RD0
1000
X62
400
630
800
5000
Training epochs (batch size=126, samppees=8000)
Training epachs (batch size=128, #sampies=500001
MNSST FC-L NS-CAMe: sagnastics
MMIST FC-4 Mik-nsan: diagnostics
fem (99. uol
Bl.. vu nexe 18,. enty
(Vja was.
2.5
25
1.0
1.0
10/-
10*-
200 4013
600
3000
203
4010
830
800
3000
Training epochs (batch size=11222 128, #samples= B2000)
Training epachs (batch size 128, #sam plee=bcce00
Figure 9: FID during training with additional diagnostics for 4-layer convolutional (top) and fully- connected (bottom) networks on MNIST, using (left) NS-GAN and (right) MM-nsat for the param- eter updates. For each run, calculate both the NS-GAN and MM-nsat gradients at each iteration and plot their relative norms (green) and cosine similarity (orange). We plot the running means, with the actual, noisy values indicated by the shaded areas.
19

---

Under review as a conference paper at ICLR 2021
Figure 10 shows the same kind of plots figure 9, but comparing the original MM-GAN with MM- nsat instead. As the gradients obtained by these formulations are parallel by construction, the cosine similarity is almost exactly 1.0 at all times. We use differently scaled y-axes for the relative gradient magnitudes, as these approach 10 when MM-GAN falls into its saturating failure mode.
The primary difference in gradient magnitude between MM-GAN and MM-nsat is a relatively brief spike very early in training, corresponding well with the the observation by Goodfellow et al. (2014) cited in section C. This spike is both taller and wider in the FC-4 setting, likely reflecting G having more difficulty learning to generate somewhat realistic samples. During the rest of the training, the magnitude of MM-nsa relative MM- MM-GAN increases slowly and consistently.
MNIST Conv MM GAN; diagnostics
MNIST Conv rist:: diegnostics
10
Ry we -earl / Mouuel
10ah
F0 su surl 10 uol
millant((j -0 and. Vc:us)
sim similart(VVk en rad. Vo.vel
101
1044
10'-
10P
10"
109
10a
10a
104
101
101
200
400
600
1000
10
200
400
600
800
1000
Training epochs (botch sizc-128. #0aplles-00000)
Training epochs (batch sizc-128, #samples-80000)
MNIST Conv 4 MM GAN: diagnostics
10a
FID
Resell Roowel
Cosine smilantt//kk...
10a
101-
R10
100
200
1000
Training opochs (batch sizc-128. #5ampies=60000)
FC-4 MM-GAN: diagnoctics
diagnostics
10a
10
PIU
Rlous]]
10t3
HD
[Vl:: [Nocaol
107
Casire smilanity/Sk.. ont To us)
Cosine similarlyP Ly/Ve
vl)
109
10a
10a
10?
10a
307
105-
10*
301
102
10-1
1000
1000
rraining epochs ibatch size 288 # #samples-uu00)
Training epochs (batch size- 128, #samples -bmplcss-suu)) b0000)
Figure 10: FID during training with additional diagnostics for 4-layer convolutional (top and middle) and fully-connected (bottom) networks MNIST, using (left) MM-GAN and (right) MM-nsat for the parameter updates. For each run, we calculate both the MM-GAN and MM-nsat gradients at each iteration and plot their relative norms (green) and cosine similarity (orange). We plot the running means, with the actual, noisy values indicated by the shaded areas. Since both Conv-4 MM-GAN produces both good and bad results frequently, we show diagnosties for both cases.
20

---

Under review as a conference paper at ICLR 2021
L ADDITIONAL SATURATION RESULTS AND DISCUSSION
MNIST FC-4 early training gradients
MM G: B2 - 0.999, D: 8: 0.999
[W/1 #8
10-5 6
1V/=] #
MM G: B3 0.99. D: B3 0.99
10
MM 0.99. D.
min !ill
10
10
MM G: B: - 0.999, D: B2 0.99
6
MM G: reset optimizer
10-6
Training epochs (batch size=128, #samples=60000)
Figure 11: Gradient magnitudes in the early phase of training on MNIST with 4-layer fully con- nected networks: as in fig 5, but including additional tests for MM-GAN with modified B3 hy- perparameters for D's and G's optimizers. As suggested by our model in section 2.4, MM-GAN saturation when using Adam depends B2 values causing gradients to vanish. In particular, lower value of B2 for G than disrupts the feedback mechanism that otherwise brings training to a halt.
When D learns its task faster than G, (G(=)) will begin to approach 0, such that both D's and G's gradients diminish. The effect on updates after Adam's normalization step depends on whether the first or second order momentum diminishes faster. The usual case is that 32 > ,B,, such that the second order momentum has longer effective memory. This partially disables normalization of updates.
For this works as intended: as the cost approaches its minimum, the optimizer sees the gradients becoming smaller and reduces its step size, allowing proper convergence. The problem when opti- mizing MM-GAN with Adam is that this same behavior triggers for the ill-behaved G cost function when it is failing at its task. While the update steps go in the right direction, the diminishing step size effectively causes parameters to get stuck in a poor configuration. Note that this problem does not really lie with Adam: it is the MM-GAN cost function that has shortcoming that Adam cannot address since it only sees the decreasing gradients and has no way of knowing that this actually means that the MM-GAN cost is increasing. The role of the gradient rescaling in equation 6 is to make it so that G's gradients and cost increase and decrease in parallel, such that the optimizer behaves properly.
21

---

Under review as a conference paper at ICLR 2021
L.1 THE ADAM B2 HYPERPARAMETER
Here, we provide additional empirical results (figure 11). In particular, we investigate the importance of how quickly D and G's updates vanish, relative to each other, by adjusting the 32 parameters of D and G (our default is 32 = 0.999). After all, the theory in 2.4 assumes a gradient that simply exponential in time and does not model the interactions between D and G when the generated data gradients begin to vanish.
A fairly straightforward pattern emerges: for the standard case, where D and G have the same value of 32, there are multiple U-shaped dips, where gradients alternate between diminishing and recovering, until the gradient finally vanishes for good. This still holds when when we reduce 32 for both optimizers, effectively maintaining their balance. However, both setups where 3g is lower for G than for D avoids vanishing updates entirely, whereas when we reduce the value of (2 only for D, training breaks down earlier and more decisively. Finally, we show that we can avoid vanishing gradients by simply reinitializing G's optimizer regular intervals: this resets both the first and second order momenta. This stabilizes the training process, which is somewhat surprising seeing as we are effectively crippling the optimizer's memory. However, this effect is well explained by our hypothesis of large, lingering contributions to the second order momentum disabling normalization.
L.2 LINEAR MODEL FOR Di(G(z))
Figure 12 shows more detailed diagnostics for runs where MM-GAN with standard Adam set- tings falls into its saturating failure mode. In addition to the gradients, we also plot the following: Di, the mean of the discriminator's logit outputs, both for real and generated data; and |A0] and |Δø), the norm of the parameter update for G and D at each timestep.
In section 2.4, we assume that Dr)G(z)) behaves as a linear function of training steps. Note that this simple model is cannot fully capture the training dynamics for MM-GAN, but is meant to explain why Adam does not reliably address MM-GAN saturation despite its built-in gradient normalization.
The actual behavior of Di(G(z)) in figure 12 is more complicated than we assume in our model. Particularly, D grows much faster early on than does towards the end. where it eventually tapers off. However, for Dz() = - 20, Dp (p(((z) R 2 * 10-9, whereas G only produces a total of 6 * 10? samples during the 1000 epochs we use to train the networks. The behavior of D: past this point is of limited interest.
We find an early, critical period most relevant for understanding the interaction between Adam and the saturating MM-GAN cost, characterized by D; becoming monotonic for each of real and generated data. In figure 12, these critical periods start around 0.4 and 2.9 epochs for Conv-4 and FC-4 respectively, without any clearly defined endpoint. Within a single epoch, the step size for G's updates drops by 3 to 4 orders of magnitude, and G(z) ceases to change during training, as shown in figure 13.
The exception to the above is the sudden spike after 100 epochs for FC-4. Ideally, this spike would be the first step towards the training process recovering, as the size of G's gradients and updates increases. However, while gradients are larger after this spike than before, the update step size actually plummets by an additional 4 orders of magnitude. The unusually large contribution to the second order momentum lingers much longer than the increase in the first order momentum, effectively disabling Adam normalization for a long while. This is a very illustrative example of the problematic interaction between the saturating MM-GAN cost and Adam.
L.3 ADDITIONAL COMMENTS
Stabilizing MM-GAN by use of the 32 parameter not recommended: we intend these experiments to demonstrate the effects discussed in section 2.4. The setting where only G's Bg is reduced to 0.99 suffers the least from noisy updates and performs well in some settings, but all these variants are less stable than the more principled MM-nsat version. However, it should be noted that instead of changing the MM-GAN cost to work with Adam, as we do for MM-nsat, it is possible to modifiy the optimization procedure to account for MM-GAN's peculiarly shaped cost function.
NS- NS-GAN owes some of its success to how and G have different scaling factors. Only D's gra- dients vanish (G)))) -> 0, and only G's gradients vanish D,(((=)) + 1. This gives the
22

---

Under review as a conference paper at ICLR 2021
training process limited form of self-stabilization: update step sizes (roughly equivalent to learning rates) increase for whichever of the networks is doing worse. Since MM-nsat has the same gradient magnitudes NS-GAN, this behavior carries over.
MNIST Conv-4 MH--CNN: diegnostics
MNIST RC-4 MIM-GAN d diagnostics
Mu] M
XX Grea samplesi
[P6]]
T (real samples]
10-2-
A
Mal 04
D Igenersted samples)
40
10-2
[6d]/ Ad
(generated somples)
10
1401/ x0
10
[bp] 05
10-1
10-4
10-11
10-11
10-14
-20
10-54-
-20
-40
20
400
60
23
400
Training epochs (batch size=128. #0000)
Traiiiin epochs (batch size=126. #samples=60000
MNIST Canu-d MMM-AN:: diagnastics
NNIST FC-4 NM-GAN diagnostics
10a
[bb] a
Or tpenersted samples!
:real semplesi
20
107
[Va]/
X (generated semples)
frew samples]
15
10-
/401/ 49
|Ap 1
1U
[N0 / ed
[Aa] / 44
10
10
5
10-1
00
10-1
10
-5
10-10
-10
10-4
-10
10
-00
10
-15
10-14 14
20
-70
10.0
10.0
Training epochs (batch size=128, #semples=E0000)
Training epochs (batch size=128 128, #samples= 50000)
MM--AN: diagrostics
diagnostics
10.0
109
(real samples)
107
Mal) 300
samples]
[Po] :4
(gencrated mplest
7.5
Pool 3:p
(generated samples)
160 / =0
[60] / 25
10-7
/ 4a
5,0
10-
|Aa
10
255
10-4
0.0
20
-2.5
10-1
20-10
10-39
-10
-13
10-13
-0.00
20-12
0.2
0.4
0.6
1.0
2.6
3.3
3.2
3.4
Training epochs (batch s 2e-128, #samples-cu0pu)
training epachs (batch size- 128, #samples- b0000)
Figure 12: Gradient magnitudes (logarithmic) and mean logit outputs (linear) from during training for MNIST, using convolutional (left) and fully connected (right) networks. Top to bottom, plots show the same training run, increasingly closed up on the critical period where training breaks down due to gradients vanishing. We plot running means, with actual values indicated by shaded areas.
23

---

Under review as a conference paper at ICLR 2021
Conv-4: 0 epochs
FC-4: 0 epochs
Conv-4: 50 epochs
FC-4: 50 epochs
Conv-4: 200 epochs
FC-4: 200 epochs
Conv-4: 500 epochs
FC-4: 500 epochs
Conv-4: 1000 epochs
FC-4: 1000 epochs
Figure 13: MNIST samples for 4-layer convolutional (left) and fully connected networks (right), using the saturating MM-GAN cost. Top to bottom show samples during training for a single run: we reuse the same set of latents z to generate samples each time. Cross-reference with figure 12, showing diagnostics for the same runs. These samples show the behavior of the MM-GAN generator when it falls into its saturating failure mode: G's s outputs effectively cease change due to negligible parameter updates.
24

---

Under review as a conference paper at ICLR 2021
M CIFAR-10 SPECTRAL NORMALIZATION BASELINE
C3AA0300 D: Cans.-ar, Com-dlir:: Mediun during
CIAAI-1 Cane-aan, Canuolann: Median
M6
KG
Velesst
10-1
200
1BC
60d
250
1000
200
400 600
030
3000
Training epochs (batch 252e-256, #samples-5000)
Training epochs (betch sice-256. Psamples-20000)
Figure 14: Left: FID, an estimated distance between generated and real data based on Inception activations. Right: DDsCD, the Jensen- Shannno divergence between class distributions in generated and real data. Copying the CNN settings from Miyato al. (2018) for the CIFAR- dataset, we plot the median values for ten runs during training. The shaded area indicates maximum and minimum values.
For the CIFAR- 10 Conv-4-sn results the bottom of figure 8, exceedingly simple networks as described section 3.3. In order to compare MM-nsat against a benchmark from the literature, the copy the settings from Miyato et al. (2018), i.e. the network architectures from table 3 and the hyperparameter settings C from table 1.
The main differences between this setup and our Conv-4-sn settings are:
Batch normalization in G
4x4-kemnee (not 3x3) for 2-stride convolutions in D and G
3 additional 3x3-kernel, 1-stride convolutions in D
Spectral normalization also for the final layer of D
Leaky ReLU in D
Approximately 10 times many network parameters
Doubled learning rate
Additionally, for comparing FID directly against values tabulated in Miyato et al. (2018), note the following:
Likely different batch size and training epochs
We use all real and 50k generated images to estimate FID, instead of 10k and 5k
Across five runs, we obtain median FID values of 26. for NS-GAN and 25.5 for MM-nsat, using same set of hyperparameters
Miyato et al. (2018) optimized for their NS-GAN results, reporting a value of 29.3. As in other experiments, we again see that DJ favors MM-nsat, even at the points during training where FID values closely matched. Unlike for networks, clear tendency for the NS-GAN FID to increase towards the end of training, possibly due to higher quality of generated samples and spectral normalization also in the final layer of D combining to pull D, values closer to 0.5 throughout training. While there is significant overlap between the cost functions, MM-nsat shows slightly better performance overall and converges faster.
25

---

Under review as a conference paper at ICLR 2021
N LINEAR COMBINATIONS OF MM-GAN AND NS-GAN
Motivated by theory connecting GAN divergences and mode dropping (section 2.3) and Huszár (2016) who suggests adding together the NS-GAN and MM-GAN cost functions in order to get rid of the subtracted Jensen-Shannon divergence for NS-GAN (eq 3), we run experiments using linear combinations of the cost function. Huszár's ' suggestion changes the scaling factor (1- D2) + 1, which is an improvement in terms of the effect discussed in section 2.3, since it reduces the emphasis on overrepresented modes. However, if Dp(G( (G())) is close to zero during training, as often happens with GANs since D has an easier task than G, there will be little difference in practice.
We use the following linear interpolations of the MM-GAN and NS-GAN cost functions, parametrized by a which determines the weight of each term:
V Gosssss (a) = (1 a) VoJGrs + a VoJGrn
(24)
VoJGNs Gis MNaa (a) = (1 (1-a) VoJGrs a VeJ
Due to the MM-GAN and NS-GAN scaling factors (see fig 2), expect linear combinations of MM-GAN and NS-GAN to be dominated by the non-saturating term, since (z))(z)) tends to be small when training GANs unless D is strongly regularized. On the other hand, using our modified MM-nsat approximately balances the gradient norm for each term, such that the behavior of linear combinations should correspond the weighting of the NS-GAN and MM-nsat terms.
Figure 15 shows results using linear combinations of NS-GAN, MM-GAN and MM-nsat for G's cost. Results for NS-GAN and MM-nsat are shown in the main text in figure 8. The behavior for NS-GAN and MM-nsat interpolations is most straightforward, with numerical results falling inbetween the pure NS-GAN and MM-nsat variants with roughly the degree of separation suggested by their weighting constants. Since NS-GAN and MM-nsat have approximately equal gradient magnitudes for all values of Dp)G)=)), the contributions from each term are predictable.
For NS-GAN and MM-GAN interpolations, behavior is less consistent and more similar to NS- GAN. This is reasonable, seeing as NS-GAN and MM-GAN have scaling factors Dp and Dp respectively, such that the two terms are effectively weighted both explicitly by the weighting con- stant and implicitly by whatever the values of (G())) happen to be during training. Note that D completely fails to discriminate between real and generated samples, it can minimize its cost by making (x) = (G(=)) = 0.5, such that NS-GAN's scaling factor will almost always be larger than that of MM-GAN.
9 weighting for MM-GAN still gives behavior qualitatively For the FC-4 results, note that even a 10 similar to NS-GAN: the early stopping metrics much better than for pure NS-GAN and the mode collapsing behavior is delayed, but the final metrics are worse both in terms of FID and DjsCD even than for the 3 -weighted MM-nsat. Generally, Huszár's suggestion of adding together the NS-GAN and MM-GAN leads to somewhat improved results, but does not come close to stability and mode coverage achieved by MM-nsat. Compare also the divergences shown in eq 3.
26

---

Under review as a conference paper at ICLR 2021
Seed
during trsining
306
HBI
1/46
[NS- (HeM-rook
21d- 2000 maat
JNS- [MM
10"-
NMN-mor
200
600
800
1000
260
400
600
3000
Training epochs (bacch 128, #5amples-00000)
Training epochs (batch size-128, 55000))-60000)
HNIST corv
105
HB1
JIM
101
By
16
(NS 1HMt-rout
20-3
211 200 maat
JnS- matt
181-mmet
NMN-mor
30a
30
200
100
600
800
1000
200
450
000
81.
3000
Training epochs (bacch $20- 128, #samples-60000)
Training epochs (batch size-128, # sampkcs-60000)
OFAA Mediar F.D
HBI Lun
JAM
NS
[4S (eet-neat
[nG 10tteot
(MS-
3440 neat
Pyco
Inss -aMM noor
HBt
10a-
10-1-
200
00C
800
1000
200
400
600
810
3000
Training epochs (bacch s20-128. #samples-50000)
Training epochs (batch size-128. $aampes-55000)
FAR Cone Mortinn EI's ding
6x
HBI
MM
JIM
NS
[as [MMtett
5x 304
2400 nest
Pyco
Inss jatM
30-1-
4x
200
102
600 800
1000
200
400 000
000
5000
Training epachs (batch d2ze-128. #samples-50000
Training opochs (batch size-128 wwwpplss---ppppp........ 50000000
Figure 15: Left: FID, an estimated distance between generated and real data based on Inception divergence between class distributions in generated
activations. Right: DD, the and real data. For various networks using the MNIST and CIFAR-10 10 datasets, we plot the median values for ten runs during training. The shaded area indicates maximum and minimum values. Costs are the traditional MM-GAN and NS-GAN (Goodfellow et al., 2014), as well as our MM-nsat from eq 6 and linear combinations as given in eq 24.
27

---

Under review as a conference paper at ICLR 2021
O SCALING FACTOR EXPERIMENTS
In section 2.1, we show that the gradients for the MM-GAN and NS-GAN generators have different scaling factors:
V VaJGsm @Di(G(Z,0). D,(G(z,0)
00
(25)
VoJass aDi(G(=, 0)) (1 Dp(G(z,0))
00
In section 2.3, we suggest that the different emphasis this places on under- and oversampled modes is the reason for the more zero-avoiding and mode-covering behaviors NS-GAN and MM-GAN. To test this hypothesis, we can modify the scaling factors and see whether we get the expected change in behavior. The linear interpolations used in section N probe the same effect a less direct manner.
While we expect the importance of the scaling factors to depend on the relative weights they assign different scoring samples, not clear which relationships are most crucial. The difference between MM-GAN and NS-GAN is most pronounced for low- and high-scoring samples, but the important difference might very well be the relative weights of two samples with different, low scores.
simple way to modify the NS-GAN gradient to add a constant a to the scaling factor:
GsJGss @Di(G(2.0) (1 Dp + aa)
(26)
00
Which is obtained by the following single-sample cost function:
Ja JGss * Dj
(27)
Renormalizing the total gradient magnitude in the same way as for MM-nsat, we get the following cost function:
D- N-1 a a Dp
V J batch Crss Crs
(z) - a D((z))
(28)
1
And similarly for MM-GAN:
Dr --1
batch Gantimanation
Σ + Dp 1=0
V (2i) a Di(z:))
(29)
These cost functions are only reasonable for a 2 O, such that we avoid negative scaling factors for samples. For increasingly large values of a, sample weights effectively become uniform across the whole range of possible scores.
Since these modifications only allow make MM and NS less extreme and more similar each other, we also try a different approach, introducing exponentiation parameter for the scaling factor:
V JoNsas = @D, .1 (1- DD((p(G()))))
(30)
a0
The general solution of this differential equation can be expressed in terms of the hypergeometric function, which does not lend itself to efficient computation. However, specific values of e give rise to useful cost functions. As before, we consistently renormalize the gradient magnitude to that of NS-GAN:
Gssespp
Vo(Jona Dp(x))
(31)
1 Dp
i=0
1- 1-D, D
N-1
jbatch
Vo(Jorn + D2(2s))
(32)
Dp
N-1
Thutch Gyseu
= -1 =(1-Dp D
Vo(2log De Di -e-D++1)
(33)
i=0
28

---

Under review as a conference paper at ICLR 2021
Dp N-1
Jhatch -ap00
Vol (log2 VeDi + V + +
(34)
Dj
Renormalizing the overall gradient magnitude goes long way towards stabilizing the adversarial training dynamics for these cost functions. However, overemphasis on oversampled modes tends to accelerate catastrophical mode collapse. Furthermode, expressions such as log( (veii) are numeri- cally unstable for large values of Di]-
Results using these variant cost functions are shown in figure 16. The add-variants have very clean behavior, all falling in between NS and MM-nsat in terms of FID and DisCD, the same order as suggested by their more uniform scaling factors. The exp-variants are more erratic, with NS-exp-2 having major stability issues and MM-nsat-exp-2 falling off later in training for CIFAR-10 Conv-4. Aside from these points, results correspond with our theoretical expectations.
Perhaps the most striking result the strong performance the MM- -xa---xx---p222 cost function. This strange variant is designed simply to have more extreme version of the minimax scaling factor, which we expect further temper the the mode-dropping mechanism described in section 2.3. Indeed, it generally improves performance relative MM-nsat.
Finally, we note that the MM-nsat-add variants tend towards stronger mode collapse than MM-nsat-exp variants, regardless of the choice of parameter. We suggest the following explanation for this behavior: Consider two generated samples, both such that (G))) is close to O. With NS sample weighting with MM-nsat-aut add variants, the relative weights of these samples will be close to 1, simply because a small Dp is negligible relative to the additive constants. With MM sample weighting and MM-nsat-exp variants, the other hand, relative weights have strong dependence the exact values of Dp for each sample and may be orders of magnitude apart.
29

---

Under review as a conference paper at ICLR 2021
Cony-l Nedien D,CD during traning
10
MS-eap-0
NS exp-2
M0-watt-0
NS-usd--..
add 1.0
HS 1-00000-
fg
nest aop
MMA most wnd 0.5
MMA root add 1.0
307
epochs (banch 128, (amplle--80000))
apachs
samples-60000)
training
CIAR-20 Canud: Median byCD during irsming
20a
Mi-eap-0
XG-eop-C55
MS exp
IS- -S--at---3
NS- N5-2ski-2.
NS ado
H5 100-----
s96-natt
104 aop-
M84 cop-2
MM5 rost and
10"-
M1A root udd 1.0
10-1
Vd4 noxt aui 1.0
Training epachs (banch 128, Saampees-5000e 50000)
apachs (batch sita-128. samples-500000)
AI-A0-1D Com--hun: Madian 0,C0 during traning
6 200
Mi-eap-0
MS exp
NS
NS adts- 5
Hos 2i---
NS edes
NG K6-1000-.4
J94-nsat
10-14
nozt
MM5 most aodd-----dd--..
MMA riatt and 1.0
144 nowt add 1.5
Training (batch siza-15,,
apachs (batch sita-128. 3300pes--0000)
Figure 16: Left: FID, an estimated distance between generated and real data based Inception activations. Right: DjsCD, the Sensen-Shannon divergence between class distributions in generated and real data. For various networks using the MNIST and CIFAR-10 10 datasets, plot the median values for ten runs during training. The shaded area indicates maximum and minimum values. Mod- ified cost functions behave predicted by their scaling factors according theory in section 2.3. In particular, MM-nsat-exp-2. using scaling factor with even more emphasis on underrepresented modes, achieves better values of FID and Dis CD than the more standard cost functions.
30

---

Under review as a conference paper at ICLR 2021
P HINGE-GAN AND LS-GAN
To supplement our results for NS-GAN and MM-GAN variants in section 4.2, we choose to run ex- periments with the LS-GAN (Mao al., 2016) and Hinge-GAN (Miyato et al., 2018) formulations. As these only require the cost functions themselves to be modified, they can be compared directly to NS-GAN and MM-nsat. Note that do not adjust network architectures or hyperparameters from the default settings we use across all experiments, such that results are more indicative of general stability than best-case performance.
The widely used WGAN (Arjovsky et al., 2017) formulation requires additional regularization to prevent D's outputs from diverging, most commonly 1-centred gradient penalty on interpolations between real and generated data (Gulrajani et al., 2017). This makes it difficult to draw fair compar- isons. Hinge-GAN very similar to WGAN, but simply clips D's cost outside of the (0, 1) interval and is known produce strong results in for instance BigGAN (Brock et al., 2018).
P.1 MNIST AND CIFAR-10 10 RESULTS
Figure 17 shows results, as in figure 8 in the main text, but with different set of cost functions. Figure includes subset of these plots, For the bottom figure, note the use of spectral normalization and much smaller interval of FID values than the other plots. As for NS-GAN and MM-nsat, we have not adjusted networks
Generally speaking, HingeeGGAN performs better than NS-GAN but worse than MM-nsat and suffers from some of the same gradual mode collapse issues as NS-GAN, Where Hinge-GAN and MM-nsat most similar in terms of FID, MM-nsat tends towards better class balance. LS-GAN performs markably well for MNIST Conv-4, but is otherwise unimpressive and suffers the most from stability issues.
P.2 GRADUAL MODE COLLAPSE FOR HINGE-GAN AND LS-GAN
Interestingly, the results in figure 17 suggest that the behavior of gradual mode collapse seen for NS-GAN and explained section 2.3 not specific to NS-GAN. This most clear for Hinge- GAN, which in qualitative sense follows the same trajectory NS-GAN in terms of both FID and DjcCD, but with better numerical results overall. For LS-GAN, we see similar tendencies, but its behavior is much more erratic overall, including very poor early stopping results for MNIST FC-4 and very strong results for CIFAR-10 Conv-4, even beating MM-nsat.
31

---

Under review as a conference paper at ICLR 2021
MNIST DyCO ovring sminkng
13"
cent
130
200
DyCD during trinking
105
no4t rtrge
L5
107-
zó0
1900
260
400
ato
Training epachs (barch S:20-28, 128, #samples--B0 p100--6000))
Training spachs (batch size-128, kaa0pss-00000)
durig traniing
106
594 noat Herge
20-1
20-2
100
10-3-
z60
800
1000
260
400
alo
3000
Training epochs (barch #20-128, Samplees-00000
Training spachs (batch size-128. samp
DyCO ouring yriiiigg
105
UM4 noat Herge
HN
10"-
10-1
260
800
1000
260
400
3000
Training epochs (barch s220 128, #sam 50000)
Training opochs (batch size-128. samp
8x
10F
noat Herge
HN--
Hing
5xx 30
10-1
4x 100
200
<b2
850
1000
260
460
600
350
3000
Training opachs (batch size-125, 4 #camples-0000)
Training opochs (batch size-128. samp 50000
Figure 17: Left: FID, an estimated distance between generated and real data based on Inception activations. Right: DDsCD, the Jensen-Shannon divergence between class distributions in generated and real data. For various networks using the MNIST and CIFAR- 10 datasets, we plot the median values for ten runs during training. The shaded area indicates maximum and minimum values. NS is Goodfellow's non-saturating cost; MM-nsat is from eq 6; Hinge and LS are alternative formulations for comparison, which use different costs for both D and G.
32

---

Under review as a conference paper at ICLR 2021
Q SAMPLE WEIGHTING AND MODE DROPPING FOR OTHER GAN FORMULATIONS
In section 2.3, we suggest that the mode dropping tendencies of NS-GAN can be understood as consequence of how overrepresentation and upweighting interact, allowing generated samples from a small region of the data space to dominate G's parameter updates. In this section, we discuss the wider implications of this effect.
Note in particular the following figures showing relevant empirical results, with additional discus- sion in the subsections:
MM-GAN and NS-GAN variants: fig 8
MM-GAN and NS-GAN linear combinations: fig 15
MM-GAN and NS-GAN scaling factor modifications: fig 16
Hinge-GAN and LS- LS-GAN comparisons: fig 17 and fig 21
The key result from these experiments is that the relationship between upweighting of overrepre- sented samples and gradual mode dropping is robust across a variety of changes in sample weight- ing, both implicitly (linear combinations) and explicitly (scaling factor modifications). Increasingly upweighting scaling factors give increasingly mode dropping behavior. Furthermore, Hinge-GAN and LS-GAN suffer from mode dropping behavior which similar to that NS-GAN in a qual- itative sense: this is consistent for Hinge-GAN, while LS-GAN is more erratic and suffers from stability issues.
In section 2.3 only discuss the scaling factors of NS-GAN and MM-GAN, which respectively upweight and downweight overrepresented samples. To understand the effect of scaling factors in general, it is instructive consider the linear combination JN + MM in figure 15. Its scaling factor is simply {1 (G(()))) ()((())) downweights samples.
f: in other words, it neither upweights or
Despite its uniform sample weighting, the linear combination N + MM still has mode dropping tendencies, though it is less blatant than for NS-GAN. This is expected from our theory in section 2.3: MM-GAN downweights overrepresented samples compensate for their overrepresentation. Uniform weighting, while better than NS-GAN upweighting, does not avoid the mode dropping effect, because overrepresentation on own is enough to cause gradual mode dropping.
While it is difficult to make general claims about how sample weighting relates to mode dropping for other GAN formulations general, we can consider Hinge-GAN which has simple cost func- tions: its generator has the same uniform sample weighting the linear combination N + MM discussed above, and its discriminator also has clipped, uniform sample weighting, unlike the error- emphasizing cross entropy loss for MM-GAN and NS-GAN, While the Hinge-GAN discriminator will behave differently during training, it does not give Hinge-GAN any mechanism for downweight- ing overrepresented samples, explaining the mode dropping behavior seen in our results.
MM-GAN's form of downweighting highly unusual. Note that we mean overrepresentation of generated samples relative to real samples: MM-GAN downweights realistic samples that are gen- erated too often. However, also downweights highly unrealistic samples, since these will be overrepresented by virtue of falling in regions where the density of real samples is negligible. This downweighting is a liability early in training, because it tends to cause saturation problems when D learns to assign very low values of Dp G's unrealistic samples. Note results in section O, which indicate that strong downweighting is necessary to stabilize the overrepresentation effect.
This means that cost functions which properly downweight overrepresented samples will generally fail to train unless care is taken to decouple overall gradient magnitude (determining saturation) from sample weighting (determining mode dropping).
33

---

Under review as a conference paper at ICLR 2021
R
DISCRIMINATOR REGULARIZATION
CIFAR-10 Conv-4 varying weight for zero-centred real data gradient penalty
MM-nsat: À 10
10
10
MM-nsat: A 10
10-1
MM-nsat: À 10
DjCD
MM-nsat: À 101
MM-nsat: X 10a
MM-nsat: A 10
NS: =
NS: A =
NS: A
NS: ==
10
= 10
NS: 10
NS:
10
6x10'
10a
Figure 18: Scatter plot Dj FID for 4-layer convolutional networks CIFAR- 10, using MM- nsat and NS cost functions. We regularize with zero-centred real data gradient penalty, sweeping across different values of the weighting constant À used in D's cost function, and train for 1000 epochs. Each dot represents a single run, while the larger circles show the mean value for the twenty different runs for each value of X. The dotted lines indicate each for increasing values of X. With very weak regularization, MM-nsat is far stronger than NS, but for increasingly strong regularizations, the two costs converge in terms of performance, Their best results (A = 10%) are similar and match weakly regularized MM-nsat in terms of FID, but with improved class balance. Performance progressively degrades with stronger regularization.
function's trajectory
Figure 18 shows results for MM-nsat and NS cost functions when combined with zero-centred real data gradient penalty (Mescheder, 2018). See also results for spectral normalized networks the bottom in figures 8 and 17, where results are much more similar than in the unregularized case: this still holds true with different cost functions and different form of regularization. This is essentially a special case of an effect shown in Qin et al. (2018), that all strongly regularized cost functions degenerate into the same behavior.
The key result is how strong regularization makes the performance of MM-nsat and NS (and other cost functions) much more similar. Some form of regularization is often used to obtain strong results, since the introduction of various forms of gradient penalties such as the one used in WGAN-GP (Gulrajani al., 2017) and spectral normalization (Miyato et al., 2018).
Discriminator regularization has many effects, such reducing the ability of D to learn quickly and smoothing the landscape D's outputs. difficult to fully understand how affects the training process for GANs. Generally speaking, limiting the Lipschitz value of D (as is done explicitly by spectral normalization and partially by zero-centred real data gradient penalty) flattens the shape of D and ultimately limits the difference between the outputs for real and generated data. It follows that assuming that D ze De as given by eq is less reasonable for a regularized discriminator, such that the effect discussed in section 2.3 will be less pronounced.
34

---

Under review as a conference paper at ICLR 2021
As discussed in section 2.1, the difference between MM-GAN and NS-GAN is only in the scaling factors, D2 and - D respectively. Strong regularization effectively squeezes Dp towards 0.5: this makes D, and D, more similar, and thus also MM-nsat and NS, we have plotted, regularization vastly improves the results for NS, but the benefits for MM-nsat are much more limited. The common default value of the weighting for the gradient penalty, is actually somewhat harmful for MM-nsat compared to the unregularized case, and looking only at values for this regularization gives the misleading impression that NS is stronger than MM-nsat.
in fig 18. For the runs
10',
We consider a more thorough discussion of this problem out of scope for this work, but suggest that might explain the fairly similar results for MM-nsat and NS when applied the StyleGAN model (see W). Absent a better understanding of the interactions between discriminator regularization and cost functions, hyperparameters for regularization terms should be tested extensively when applying MM-nsat.
35

---

Under review as a conference paper at ICLR 2021
S RING OF GAUSSIANS
The ring of Gaussians toy problem has a number of degrees of freedom, both for the dataset (number of modes, number of standard deviations of separation) and the model (architecture, batch size, training iterations). There are settings where both MM-GAN and NS-GAN cover or drop modes and significant variation between individual runs. We do not present thorough analysis of this problem: in the main text, we use settings chosen to obtain the same qualitative NS-GAN behavior as shown in Metz et (2016); Srivastava et al. (2017).
To give some impression of the diversity of possible behaviors and the usual differences between NS- -S-GAN and MM-GAN, we include results for more challenging toy problem figure 19 and table 3. In this case, where NS-GAN drops only some modes, we again find that MM-GAN has qualitatively better mode coverage. Furthermore, frequencies samples from each mode is much better aligned with real data for MM-GAN than for NS-GAN. Interestingly, while NS-GAN places particular emphasis on avoiding generated samples outside the real data manifold, more samples fall outside of the real data modes for NS-GAN than for MM-GAN.
and generated samples toy data: spiral Gaussians
1.0
0.0
Real samples
NS-GAN
MM-GAN
-1.0
-1.0
-0.5
0.5
1.0
Figure 19: Real and generated samples on modified 2D mixture of Gaussians: modes ordered along expanding spiral from the origin, making in excess of two revolutions. Density of modes increases along this spiral. In blue, NS-GAN, In red, samples using the unmodified MM-GAN cost function. MM-GAN trains well on this toy problem despite its saturating cost function and shows decent coverage of all modes. Cross-referent with table 3.
Table 3: Densities for each mode for generator with 3-yeyer fully connected networks. Samples classified as belonging to the closest real data mode if is within standard deviations of any mode, otherwise as belonging to no mode.
Mode frequency in %
Mode
None
2
4
11
Cost NS MM
4.3
1.4
8.6
12
1.0
0.6
0.6
0.0
0.0
25
27
1.2
2.5
6.8
7.8
14
Data
2.0
3.1
4.3
5.4
6.6
7.7
8.9 10
13
36

---

Under review as a conference paper at ICLR 2021
MNITT Com &
N6556V combination
VIAST sgp NO510M combinatiin
0289
2012
6.3W
0349
3200
0080
20
3.00
Figure 20: Scatterplots D1s FID with trendlines for 20 training runs, using linear combinations of the NS and MM-nsat costs (eg 24), on the MNIST-IK dataset created by combining triplets of samples from MNIST into 3-channel RGB images. Left: zero centred gradient penalty and spectral normalization. Right: no spectral normalization and relaxed zero centred gradient penalty.
MNIST-IK
We tests the Stacked MNIST dataset (Metz al., 2016) (also known as MNIST-IK), in order to get a more demanding and multi-modal training task where we can still classify generated sam- ples. The samples in this dataset obtained by combining three samples from MNIST represent each of the three color channels for an RGB image. We find a variety of network architectures dif- ficult to train for any of our cost functions and resort regularizing D. We show results in figure 20: in both cases, differences between linear combinations are minor. For the most heavily regular- ized model, there is clear trend. Removing spectral normalization and relaxing the zero centred gradient penalty improves the models and makes them slightly more different, recovering the usual ordering terms of DJs-
37

---

Under review as a conference paper at ICLR 2021
CAT 122 Caned: Median curng
CAT 120 DCC-ón
Medi HDD
M6
KC
most 1krge
firge
2x 107
2 1n
200
Training epochs
esamples-6440)
Training size-04. esamples-Gaae
Figure 21: FID during training for ten runs on CAT 128 with different network architectures. Left: 6-layer convolutional networks. Right: DCGAN with batch normalization as well as spectral normalization for D. For both architectures, only MM-nsat trains reasonably well, while NS GANN Hinge-GAN and LS-GAN all have major stability issues due early, catastrophical mode collapse.
CAT 128
We rum tests with CAT 128 (Zhang al., 2008), chosen as a reasonably difficult dataset which is still amenable to simple network architectures. FID during training for two architectures is shown figure 21, emphasizing training stability. Figures 22a and 22b show samples for DCGAN with spectral normalization for D and self attention: these are shown cropped in the main text in figure Figures 22c, 22d and 22c show samples for other network architectures. Note that samples are JPEG compressed.
We find that MM-unit and MM-nsat perform much better than find that the failure mode of NS cannot be addressed by early stopping in these experiments, unlike for MNIST, where NS tends to produce better results early on and mode dropping is mostly due to extended training (see figure 8). For CAT 128?, catastrophical mode collapse tends to happen before G has learned to produce reasonable samples. MM-nsat much less susceptible to this failure mode than NS. As usual, results most similar with regularized discriminator.
this setting. particular, we
38

---

Under review as a conference paper at ICLR 2021
Figure 22a: Early stopping samples from best NS run for CAT 1282 using DCGAN with spectral normalization in the discriminator and self-attention (as in main text).
39

---

Under review as a conference paper at ICLR 2021
Figure 22b: Early stopping samples from worst MM-nsat run for CAT 1282 using DCGAN with spectral normalization in the discriminator and self-attention (as in main text).
40

---

Under review as a conference paper at ICLR 2021
Figure 22c: Final CAT 1 samples for 6 networks for 4 random runs. Left: NS. Right: MM-nsat.
41

---

Under review as a conference paper at ICLR 2021
Figure 22d: Final CAT 128 samples for Conv-6-sn networks for 4 random runs Left: NS. Right: MM-nsat.
42

---

Under review as a conference paper at ICLR 2021
O
Figure 22e: Final CAT 1282 samples for DCGAN-bn bn-sn networks for 4 random runs. Left: NS.
Right: MM-nsat.
43

---

Under review as a conference paper at ICLR 2021
V FFHQ AT VARIOUS RESOLUTIONS
Figure 23 shows results for training simple convolutional GANs on the FFHQ dataset downsampled various resolutions, comparing NS-GAN and MM-nsat. The differences between the two cost functions are generally as seen in other experiments. Aside from the decent early stopping results for NS-GAN even high resolutions the and relatively strong performance of MM-nsat even at 512" resolution with very unsophisticated network architectures, these results only replicate previously discussed effects. Figure 24 shows samples from training at 512?, showeasing the mode collapsing behavior that causes FID to increase. Note that samples are JPEG compressed.
riig 32lcame 6 Medann FID during masning
mnig Median duinng treining
f 101-
Training epochs (satch tize=120, #samples:09944)) mio Caraed ITD during traniin
Training epochs (baithh size=120. *samples-69-co)
mio Camu-T: Medwnn rID ouring tranting
Gv 10
g
2x 10
6x
Training epochs (satch sibe=128, #ramples-04440)
Training epachs (batch tize=126, Mrampled-t94400
mo Camud:: Median during truniig
6x 36
3x
2xx30 304
6x
400 Training apachs (batch sibem64, #samples-53440)
Figure 23: FID, an estimated distance between generated and real data based on Inception activa- tions. For various resolutions using the FFHQ dataset, we plot the median values for three runs during training. The shaded area indicates maximum and minimum values. Results for FFHQ 1024 has been left out: neither MM-nsat or NS-GAN achieve FID values meaningfully better than randomly initialized networks. For resolutions from 32 to 5122, training collapse is universal for NS-GAN while MM-nsat is stable and achieves similar FID for low resolutions and much better FID for high resolutions.
44

---

Under review as a conference paper at ICLR 2021
(a) NS-GAN: 200 epochs (pre-collapse)
(b) NS-GAN: 250 epochs (post-collapse)
(c) NS-GAN: 1000 epochs (final samples)
(d) MM-nsat: 1000 epochs (final samples)
Figure 24: FFHQ 512" samples for 9-layer convolutional networks. Subfigures (a-c) show NS- GAN's early stopping performance, its abrupt, catastrophical mode collapse and its final, nonsensi- cal samples. Subfigure (d) shows final samples for MM-nsat, which fall well short of photo-realism, but are of higher quality than the NS-GAN early stopping samples and fairly good given the exceed- ingly simple network architecture.
45

---

Under review as a conference paper at ICLR 2021
Table 4: FFHQ FID values using StyleGAN, comparing results from retraining the original imple- mentation on 4 GPUs at full and lowered resolution, with results obtained by replacing NS loss with MM-nsat without any additional tuning. Cross-reference with figure 25 showing samples.
Settings
Final FID
Best FID
1024º NS
4,2887
3,9354
1024² MM-nsat
6,1292
5,9294
256 NS
5,6232
5,4288
256 MM-nsat
6.2375
5.3844
W. STYLEGAN
We run experiments training StyleGAN (Karras et al., 2018), replacing the traditional NS loss used in the original implementation with our MM-nsat. Due to available resources, we train each model using only 4 GPUs. We make other adjustments. We show results in table and figures 25a and 25b.
The primary result from these tests is that MM-nsat performs reasonably well, albeit worse than NS-GAN, as a drop-in replacement in a sophisticated, state of the art implementation. resolution case, which is carefully optimized by the original authors (for instance learning rate ad- justments throughout training and specific value of truncation for the latent input for G), NS achieves much better FID and MM-nsat samples seem have more artifacts. For the lower resolu- tion case, which is a somewhat more fair comparison, FID values are closely matched: NS gets the best final value, whereas MM- has the best early stopping value.
the full
Notably, StyleGAN uses zero centred gradient penalty and minibatch standard deviation features in (see R). Both of these can serve limit the failure modes we have shown for NS-GAN, For StyleGAN, coverage is limited, particularly for the latent truncation settings that achieve the best FID (Kynkäänniemi et al., 2019), and visual quality of samples the first priority.
Implementing MM-nsat for StyleGAN requires some minor modifications, which essentially consist of accumulating values of Dp from parallels in the same way the original implementation does for gradients and rescaling the total gradient before passing it back to the parallels to update parameters for each network clone.
46

---

Under review as a conference paper at ICLR 2021
(a) 1024 NS: FID 4.2887 (b) 1024 MM-nsat: FID 6.1292
Figure 25a: FFHQ samples using StyleGAN. Samples selected at random. Overall, MM- MM-ssat sam- ples are somewhat less convincing, especially when viewed in full size.
47

---

Under review as a conference paper at ICLR 2021
(a) 256 NS: FID 5.6232
(b) 256 MM-nsat: FID 6.2375
Figure 25b: Lower resolution FFHQ samples using StyleGAN. Samples selected at random. We find no obvious differences in visual quality.
48


Reviews:
[{"review_id": "sZDukTplxQn", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis work proposes that many common issues with GAN methods are based on the weighting of the samples given to the generator’s objective function. They focus on a study of the original GAN objective proposed in Goodfellow et al. where the generator’s objective is the negative of the discriminators objective. The GAN community quickly observed that when the discriminator outperforms the generator with this objective, the saturating nature of the sigmoid function causes the gradients to vanish for the generator’s objective. For this reason, a new objective (NS-GAN) was proposed which modifies the generator’s objective to alleviate this gradient vanishing issue.  The authors argue that this modified objective is to blame for a number of common issues with GAN methods -- most notably the mode-dropping issue. The authors present theory that backs up these claims and they propose a new generator training objective which re-weights the gradients of the generator objective to have the same average magnitude as NS-GAN but have the same relative magnitudes of the original GAN objective. \n\nThe authors demonstrate the impact of their new loss function in a series of quantitative and qualitative settings. \n\n\nStrong areas:\n\nI am a very big fan of work that questions standard assumptions that are taken almost as fact within our community. When GANs were originally proposed, most researchers saw the NS-GAN objective as a strict improvement over the MM-GAN objective and moved on. This work does a great job to demonstrate that NS-GAN is most definitely not “superior” to MM-GAN and may possibly be a worse choice of objective function. This change may seem insignificant on the surface and the authors here clearly show that it has an impact -- one large enough to consider if it should be used at all. To aid in their claims, the authors provide clear and concise explanations backed up by easy to understand theory. Particularly interesting to me was the argument that the popular Adam optimizer does not alleviate the saturation issues found with MM-GAN even though my own intuition on the optimizer tells us that it should help with issues like this. \n\nWeaknesses:\n\nThe empirical results presented appear promising. The proposed approaches (MM-Unit and MM-NSAT) considerably outperform the NS-GAN objectives. My biggest issues are with the quality of the baselines. I am not an expert in the GAN field, but from inspecting the Conv-4 model with spectral normalization on CIFAR10, it appears like the best performing model achieves an FID of approx 42 and the worst model gets around 48 (figure 8, bottom right). While this difference is notable and the error-bars indicate it is statistically significant, I am concerned because these numbers seem to considerably underperform prior work. The original paper on spectral normalization for GANs reports an FID of 29.3 for standard CNNs. This model uses the NS-GAN objective (to my knowledge) so, I am confused as to why the authors did not simply replicate their setup -- especially since the standard CNN model proposed in the original spectral norm paper can be trained with reasonable compute and the authors have released code. If I am misunderstanding something about the experiments, please do let me know, but I am confused by this choice.\n\nI also took some issues with the D-JS-CD score proposed to score the class distribution of generated samples. There have been a number of proposed metrics like this such as IS and classifier score (https://arxiv.org/abs/1905.10887). I understand that this score (unlike classifier score) does not rely on a conditional model, but if a new score is to be proposed and it is used to convince the reader that a new method performs well, then it should be applied to some baseline models. Pre-trained models from prior GAN papers could be used to obtain these scores. \n\nAs a researcher from a different field, my main concern about the experimental results is that the results are quite far from the current state-of-the-field. State of the art results are by no means required for publication in this venue, but the baseline models presented here perform much worse than they have been shown to in prior work. Since a near 20-point increase in FID can be achieved with a few tweaks to the NS-GAN objective (SN-GAN), I am left wondering if the presented improvement from the MM-NSAT objective will vanish once those improvements are applied or if it will still hold. Since this is not shown, then I am uncertain of the significance of the observations made in this work. \n\nSome more nit-picky issues:\n\nThe text in the figures is too small and near impossible to read. I would present all of the results in Figure 8 in a table instead. There is no information gained by seeing loss curves. A table would save much more space and allow the readers to more easily compare this work to other works. In Figure 7, I find “best” and “worst” picks by qualitative methods to be somewhat unconvincing. You should show the highest and lowest FID or just show random samples.\n\nMy recommendation:\n\nI am not an active member of the GAN community so I am more than willing to accept if my recommendation goes against more senior folks who work in that field. \n\nI found this to be an interesting work that provided a non-trivial insight, backed it up with clear and easy-to-follow theory and demonstrated their observations held on some medium-scale experiments. My biggest issues come from my reservations about the experimental results. The baseline NS-GAN with spectral normalization presented in this work greatly underperforms previously published methods that use the same objective. The difference between the presented baseline and the proposed method is smaller than the difference in performance between the presented baseline and previously published methods with the baseline objective. \n\nThese discrepancies give me sufficient doubt where I am not certain that the insights of this work provide a sufficient improvement when combined with architectural improvements and proper parameter tuning. \n\nFor these reasons, I am advocating against acceptance of this work but making it clear that I think this paper is borderline.\n\nIf the experimental setup was more in line with prior work and the same trend in results held, then I would be more likely to recommend acceptance of this paper. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "kZS0cBafc4f", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper reexamines the original (MM) and the non-saturating (NS) GAN objective.  The authors show that the gradients of the respective objectives just differ from a scaling factor depending on the discriminator's output for generated samples. While the scaling factor for the MM gradient is responsible for the well known vanishing gradient if the discriminator is optimal, the scaling factor of the NS gradient counteracts this saturation effect. However, on the other side, the NS scaling factor introduces a mode dropping effect and the inability of the learning dynamci to discover new modes. The authors show additionally that the NS minibatch gradient is the weighted sum of the single sample MM gradients with respective scaling factors as weights. These scaling factors avoid saturating, however, alter the direction of the resulting minibatch gradient. To counteract the change of the gradient direction the authors propose to summarize the sample scaling factors into one scalar for the batch gradient which preserves the non-saturating behaviour of the NS and the gradient direction of the MM objective. The new GAN objective is called MM-nsat.  Additionally the authors discuss the non-saturating effect of the ADAM beta2 parameter for the MM-GAN Generator. \n\nExperiments: A ring of Gaussians experiment shows the mode dropping effect of the NS-GAN. Training a 4-layer network on MNIST shows demonstrates the vanishing gradient effect on MM-GAN and the counteracting effect with a smaller ADAM beta2 parameter applied to the Generator. Also on MNIST MM-nsat was compared with NS showing a lower FID and Jensen-Shannon divergence between real and generated class distributions for MM-nsat. On the Cat 128x128 dataset mode collapse was visually shown for NS. \nOn MNIST, CIFAR10 and Cat 128x128 MM-nsat was compared with NS, Hinge and LS-GAN outperforming all of them based on the FID.\n\nPros: The theoretical insight why NS-GAN suffers from mode collapse is novel and interesting. The experiments are convincing and extensive.\n\nCons: The proposed objective is not novel [1][2], however, it is derived directly from the original MM-GAN objective and is better theoretically motivated.  \n\nFor completeness, it would be interesting to see the experiments in section K with the pair MM-nsat/MM as well. \n\n[1] R Devon Hjelm, Athul Paul Jacob, Tong Che, Adam Trischler, Kyunghyun Cho, and Yoshua Bengio. Boundaryseeking generative adversarial networks, 2018\n[2] Zhiting Hu, Zichao Yang, Ruslan Salakhutdinov, and Eric P. Xing. On unifying deep generative models. CoRR,\nabs/1706.00550, 2017. URL http://arxiv.org/abs/1706.00550\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "RoKA3yx7pPf", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present a simple estimator which approximates the gradient of the \"true\" generator loss in GANs using the gradients from the more commonly used non-saturating generator loss. They present results on toy data and small image datasets like CIFAR to show that the method leads to stabler training and does not suffer from mode collapse as badly.\n\nHad this paper appeared at ICLR in, say, 2016 or 2017, I think it would have been a welcome addition to the literature on \"tricks to improve GAN training\", which was fairly small at that point. Since then, the field has absolutely exploded, to a degree that I find somewhat baffling, as GANs have no practical applications beyond generating pretty pictures. In addition, recent advances to VAE training (building on some of these GAN tricks like spectral normalization) have massively improved the quality of VAE samples, so the need for GANs continues to diminish. Nevertheless, it is not my job to judge whether a given paper is trendy or not - only to judge whether it is technically correct and up to the standards of publication. On that front, I think it is a decent paper - the method is quite simple, but the results on both mixtures of Gaussians and CIFAR and other image datasets do seem like an improvement. I would have appreciated experiments on a larger dataset like CIFAR or high-res CelebA. I also would have appreciated a more rigorous comparison against the plethora of GAN training tricks that have appeared in the last 5 years - for instance, competitive gradient descent (Schaefer and Anandkumar, 2019) to name just one. As it is, I worry this paper will disappear in a flood of other similar papers without more rigorous comparisons - but then, that may be for the field to judge after it is published.\n\nA few stylistic points:\n* Please increase the size of the axis labels in Fig 1. They are unreadable unless zoomed in very far on a screen.", "extra_scores": {}, "extra_text": {}}, {"review_id": "J9xS09WqEpP", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an explanation for mode collapse in the original GAN with the log -D objective for the generator (dubbed the non-saturating GAN or NS-GAN for short). The paper takes the approach of comparing the gradient of the generator objective for the original GAN with cross-entropy loss (dubbed the minimax GAN or MM-GAN for short) and the log -D variant. The key observation is that the difference between the gradient of the generator objective of MM-GAN and NS-GAN is that MM-GAN has a factor of D_p(G(z,\\theta)), whereas NS-GAN has a factor of 1-D_p(G(z,\\theta)), where D_p(G(z,\\theta)) is the output of the discriminator on a sample generated from z. Hence, the terms in the MM-GAN gradient that appear fake to the discriminator are downweighted, whereas they are upweighted in the NS-GAN gradient. Because the samples from modes that are already overrepresented are likely declared fake by the discriminator, the contribution to the generator gradient is dominated by these samples in NS-GAN. \n\nStrengths:\nThis observation is very nice, intuitive and simple and is new to my knowledge. It sheds light on the weaknesses of the log -D variant of GANs.  \n\nWeaknesses:\nPaper title is broader than what the paper shows - the proposes explanation only applies to GANs with the log -D generator objective and does not apply to other GAN variants, e.g.: original GAN (with cross-entropy generator objective), WGAN, LSGAN etc.\n\nThe argument is not presented clearly, and sometimes observations with no obvious logical relationship to the claim are mentioned. At other times, it is unclear what the point is. For example:\n\nOn pg. 3, in the first paragraph under sect. 2.3, it is mentioned that \"the minibatch used to update G will have more samples from O since they are generated more often\". It is unclear how the number of samples from O in the minibatch could cause a difference between MM-GAN and NS-GAN - this observation is true for both MM-GAN and NS-GAN!\n\nIn the next paragraph, the paper mentioned that the generator gradient \"is only locally informative\" - again this is true for all GANs. How is this relevant for the argument made in the paper?\n\nThen it mentioned that \"generated samples give rise to conflicting gradients\" - it is unclear what conflicting gradients mean. In what sense are the gradients \"conflicting\"?\n\nIn the next paragraph, the paper mentioned \"NS-GAN struggles to discover new modes: g(x) ≈ 0 ⇒ 1 − D_p(x) ≈ 0\" - the connection between the claim about the difficulty of discovering new modes and the equations should be explained more clearly. Also, the implication in the equations would not be true if r(x) ≈ 0. \n\nAlso, for eq. (7), technically the optimal discriminator is only uniquely defined at the real data points (see Sinn & Rawat, AISTATS 2018) because the GAN is trained on a finite sample. The optimal discriminator result in (Goodfellow et al., 2014) assumes access to the true data distribution, or equivalently an infinite stream of samples from the distribution. So technically the paper's claim on the discovery of new modes cannot be justified by \"g(x) ≈ 0 ⇒ 1 − D_p(x) ≈ 0\", because D_p(x) could take on any value at locations other than the data points. Similarly, on pg. 4, in the paragraph below eq. (9), because D_l^{opt} could take on arbitrary values at positions other than the data points, the claim that D_{opt} = ±\\infty is inaccurate - it is in fact not uniquely defined for points that are not real data points. Though because this is a common mistake in the literature, I'd be fine with the addition of a note that explains this caveat before presenting the theoretical argument (without insisting on a fundamental solution to this issue). \n\nSect. 2.4 on \"MM-GAN Interaction with ADAM\" is not very mathematically rigorous and relies primarily on an assumption that the value of the logits of the discriminator approaches the optimum linearly. It's unclear if this assumption is actually justified, since the gradient of the cross-entropy loss w.r.t the logits tapers off on the extreme ends of the logits, so the logits should approach the optimum more and more slowly as they become larger in magnitude. \n\nAlso, in eq. 11, the left size is vector graphics, whereas the right side is a scalar. \n\nUnder sect. 3.1, the JSD between class predictions assume each class contains only one mode and cannot detect intra-class mode collapse. This caveat should be prominently posted. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I think we did learn something new from this paper, and I think the reviewers all seem to agree with this.
The observation you make about the objective seems correct and interesting (though reviewers and ACs do sometime miss errors), but I have the following complaints that keep me from recommending acceptance:

1. The theory seems right, but in practice, all sorts of GANs with all sorts of objective functions experience "mode collapse", 
so it doesn't seem like the issue you point out w/ the NS-GAN objective can be the whole story. 
However, we generally don't ask of a paper that it tells the whole story all in one go...

2. I do think the experiments are somewhat poorly done (compared to those for say the median paper about GANs that gets accepted to one of these conferences). Moreover, many people have made similar experimental claims to the ones that are in this paper that haven't held up on more complicated data sets, so I tend to apply more scrutiny to such claims when they're only evaluated on smaller tasks.

3. There have, as R3 points out, been a huge number of papers proposing tricks for training GANs, and some of them work really well. 
What I'm missing from this paper is an exploration of the relationship between your observation and those (mostly ad-hoc) tricks. 
Does your observation explain why those tricks are necessary? 
Does it explain why some existing trick works as well as it does?
If your observation is totally orthogonal to existing tricks, can you get much better performance on a challenging data set by using it?
I don't feel like I got satisfactory answers to those questions.

All this being said, the paper was borderline, and I think if you dealt with some of the complaints above you would have a pretty good shot of getting a revised version accepted at another major machine learning conference.
Decision: Reject