Abstract:
Adaptive methods are widely used for training generative adversarial networks (GAN). While there has been some work to pinpoint the marginal value of adaptive methods in minimization problems, it remains unclear why it is still the method of choice for GAN training. This paper formally studies how adaptive methods help performance in GANs. First, we dissect Adam—the most popular adaptive method for GAN training—by comparing with SGDA the direction and the norm of its update vector. We empirically show that SGDA with the same vector norm as Adam reaches similar or even better performance than the latter. This empirical study encourages us to consider normalized stochastic gradient descent ascent (nSGDA) as a simpler alternative to Adam. We then propose a synthetic theoretical framework to understand why nSGDA yields better performance than SGDA for GANs. In that situation, we prove that a GAN trained with nSGDA provably recovers all the modes of the true distribution. In contrast, the same networks trained with SGDA (and any learning rate configuration) suffers from mode collapsing. The critical insight in our analysis is that normalizing the gradients forces the discriminator and generator to update at the same pace. We empirically show the competitive performance of nSGDA on real-world datasets.

heading: 1 INTRODUCTION
text: It is commonly accepted that adaptive algorithms are required to train modern neural network architectures in various deep learning tasks. This includes minimization problems that arise in natural language processing (Vaswani et al., 2017) and fMRI (Zbontar et al., 2018) or min-max problems such as generative adversarial network (GAN) training (Goodfellow et al., 2014). Indeed, it has been empirically observed that Adam (Kingma & Ba, 2014) yields a solution with better generalization than stochastic gradient descent (SGD) in these problems (Choi et al., 2019). Several works have attempted to explain this phenomenon in the minimization case. Common explanations are that adaptive methods train faster (Zhou et al., 2018), escape faster very flat saddle-point like plateaus (Orvieto et al., 2021) or deal better with heavy-tailed stochastic gradients (Zhang et al., 2019b). However, much less is known regarding min-max problems such as GANs. In this paper, we investigate why GANs trained with adaptive methods outperform those trained using stochastic gradient descent ascent with momentum (SGDA).
Some prior works attribute this outperformance to the superior convergence speed of adaptive methods. For instance, Liu et al. (2019) show that a variant of Optimistic Gradient Descent (Daskalakis et al., 2017) converges faster than SGDA for a class of non-convex non-concave min-max problems. However, contrary to the minimization setting, convergence to a stationary point is not guaranteed and not even a requirement to ensure a satisfactory GAN performance. Indeed, Mescheder et al. (2018) empirically shows that popular architectures such as Wasserstein GANs (WGANs) (Arjovsky et al., 2017) do not always converge, and yet produce realistic images. We support this observation through the following experiment. We train a DCGAN (Radford et al., 2015) using Adam –the most popular adaptive method– and set up the generator (G) and discriminator (D) step-sizes respectively as ηD, ηG. Note that D is usually trained faster than G i.e. ηD ≥ ηG. Figure 1(a) displays the GAN convergence measured by the ratio of gradient norms, and the GAN’s performance measured in FID score (Heusel et al., 2017). We observe that when ηD/ηG is close to 1, the algorithm does not converge and yet, the model produces high-quality solutions. On the other hand, when ηD/ηG 1, the model converges to an equilibrium –a similar statement has been proved by Jin et al. (2020) and Fiez & Ratliff (2020) in the case of SGDA. However, the GAN produces low-quality solutions at
this equilibrium. Thus, simply comparing the convergence speed of adaptive methods and SGDA cannot explain the GAN’s performance obtained with adaptive methods. This observation motivates the central question in this paper: What factors explain that Adam produces better quality solutions than SGDA when training GANs?
To address this question, we dissect Adam following the approach by Agarwal et al. (2020). They frame a generic optimizer’s update as W (t+1) = W (t) − ηa(t)G(t), where W (t) ∈ Rd is the iterate, G(t) ∈ Rd such that ‖G(t)‖2 = 1 is the optimizer’s direction and a(t) ≥ 0 is the optimizer’s magnitude. Therefore, a first step in our paper is to understand whether Adam outperforms SGDA maintly due to its direction or to its magnitude. As detailed in Section 2, we train a GAN using i) AdaLR, an algorithm that updates in the direction of SGDA but with the magnitude of Adam ii) AdaDir which uses the direction of Adam but the magnitude of SGDA. We empirically show that not only, AdaLR significantly outperforms AdaDir, SGDA, and Adam itself. This observation encourages us to conclude that:
Adam produces higher quality solutions relative to SGDA in GANs mainly due to its adaptive magnitude and not to its adaptive direction.
In Section 2, we empirically analyze the adaptive magnitude of AdaLR and observe that it stays approximately constant throughout training. This observation eventually encourages the study of AdaLR with a constant step-size. Such algorithm actually corresponds to normalized SGDA (nSGDA).Compared to SGDA, nSGDA has the same direction but differs in magnitude since we divide the gradient by its norm. Intuitively, this normalization forces D and G to be updated by vectors with constant magnitudes no matter how different the norms of D’s and G’s gradients are.
Motivated by the aforementioned observations, this paper studies the performance of GANs trained with nSGDA. We believe that this is a first step to formally understand the role of adaptive methods in GANs. Our contributions are divided as follows:
– In Section 3, we experimentally confirm that nSDGA consistently competes with Adam and outperforms SGDA when using different GAN architectures on a wide range of datasets.
– In Section 4, we provide a theoretical explanation on why GANs trained with nSGDA outperform those trained with SGDA. More precisely, we devise a data generation problem where the target distribution D is made of multiple modes. The model trained with nSGDA provably recovers all the modes in the target distribution while the SGDA’s one fails to do it under any step-size configuration: We prove that even when SGDA converges to a locally optimal min-max equilibrium, the model still suffers from mode collapsing and fails to learn recover the modes separately.
The key insight of our theoretical analysis is that no matter how the step-sizes are prescribed, D and G necessarily update at very different speeds when we use SGDA. Therefore, i) either D updates its weights too fast, thus learns a weighted average of the modes of D. This makes G learn this
weighted average of modes ii) or D does not update its weights fast enough and thus, G aligns its weights with those of D. This forces D to converge to a locally optimal min-max equilibrium that classifies any instance as ”fake”. On the other hand, by normalizing the gradients as done in SGDA, we force D and G to update at the same speed throughout training. Thus, whenever D learns a mode of the distribution, G learns it right after, which makes both of them learn all the modes of the distribution separately.
Our paper advocates for the use of balanced updates in GAN training i.e. the ratio of D vs G updates should remain close to constant. To our knowledge, we are the first to theoretically show the importance of these balanced updates. This insight contrasts with the related work that analyzes GANs, and more generally zero-sum differentiable games, in the regime where D is updated much faster than G i.e. ηD/ηG 1 (Fiez & Ratliff, 2020; Jin et al., 2020; Fiez et al., 2020). RELATED WORK Adaptive methods in games optimization. Several works designed adaptive algorithms and analyzed their convergence to show their benefits relative to SGDA. For variational inequality problems, Gasnikov et al. (2019); Antonakopoulos et al. (2019); Bach & Levy (2019); Antonakopoulos et al. (2020) propose adaptive algorithms that reach optimal convergence rates under regularity assumptions. For a class of non-convex non-concave min-max problems, Liu et al. (2019); Barazandeh et al. (2021) design algorithms that converge faster than SGDA. On the other hand, Heusel et al. (2017) show that Adam locally converge to a Nash equilibrium in the regime where the step-size of the discriminator is much larger than the one of the generator. Our work differs from these papers as we analyze Adam and do not focus on the convergence properties but rather on the fit of the trained model on the true (and not empirical) data distribution. Besides, contrary to some of the aforementioned papers, our work is not in the two-time scale learning-rates regime which do not correspond to what is mostly done in practice.
Importance of balanced updates in GANs. The importance of balanced updates for GANs has been noticed in the literature. The most popular GAN architectures (Radford et al., 2015; Arjovsky et al., 2017; Brock et al., 2018) set the step-sizes such that ηD/ηG is constant. On the other hand, Berthelot et al. (2017) introduced a control variable to ensure that the updates between G and D are at the same speed. In this work, we do not modify the GAN objective loss to enforce balancedness. Instead, we empirically and theoretically investigates how Adam and nSGDA enforce balanced updates.
Statistical results in GANs. Early works investigate whether GANs memorize the training data or actually learn the distribution (Arora et al., 2017; 2018; Dumoulin et al., 2016). Zhang et al. (2017); Bai et al. (2018) then show that for specific GANs, the model learn some distributions with nonexponential sample complexity (Liang, 2017; Feizi et al., 2017). Recently, Li & Dou (2020); AllenZhu & Li (2021) further characterized the distributions learned by the generator. On the other hand, some works attempted to explain GAN performance through the optimization lens. Lei et al. (2020); Balaji et al. (2021) show that GAN models trained with SGDA converge to a global saddle point when the generator is one-layer neural network and the discriminator is a specific quadratic/linear function. Our contribution significantly differs from these two works as i) we construct a setting where SGDA converges to a locally optimal min-max equilibrium and yet suffer from mode collapse. Conversely, nSGDA does not necessarily converge and yet, recovers the true distribution ii) our setting is more challenging since we need at least a degree-3 discriminator to learn the distribution – see Section 4 for a justification.
Normalized gradient descent. Introduced by Nesterov (1984), normalized gradient descent has been widely used in the minimization setting. Indeed, it has been observed that normalizing out the gradient improves the ’slow crawling’ problem of gradient descent and avoids the iterates to be stuck in flat regions – such as spurious local minima or saddle points – (Hazan et al., 2015; Levy, 2016; Murray et al., 2019). Normalized gradient descent or its variants outperform the non-normalized counterparts in multi-agent coordination (Cortés, 2006) and deep learning tasks (You et al., 2017; 2019; Cutkosky & Mehta, 2020; Liu et al., 2021). Our work rather considers the min-max setting and shows that nSGDA performs better than SGDA as it forces the discriminator and generator to update at the same rate.

heading: 2 FROM ADAM TO NSGDA
text: Generative adversarial networks. Given a training set sampled from some target distribution D, a GAN learns to generate new data from this distribution. The architecture is constituted of two
Algorithm 1 Generic second-moment adaptive optimizer
Input: initial pointsW(0),V(0), step-size schedules {(η(t)G , η (t) D )} , hyperparameters {β1, β2, ε}.
Initialize M(0)W,1, M (0) W,2, M (0) V,1 and M (0) V,2 to zero. for t = 0 . . . T − 1 do Receive stochastic gradients g(t)W ,g (t) V evaluated atW(t) and V(t).
Update accumulators for Y ∈ {W,V}, ` ∈ [2]: M(t+1)Y,` = β`M (t) Y,` + g (t) Y . Compute gradient oracles for Y ∈ {V,W}: A(t+1)Y = M (t+1) Y,1 / √ M (t+1) Y,2 +ε. Update: W(t+1) =W(t) + η(t)D A (t+1) W , V(t+1) = V(t) − η (t) G A (t+1) V .
return W(T ),V(T ).
networks: the generator maps points in the latent space Dz to sample candidates of the desired distribution; the discriminator evaluates these samples by comparing them to samples from D. More formally, the generator is a mappingGV : Rk → Rd where V is some parameter set. Generally, the latent variables are sampled from the normal distribution. On the other hand, the discriminator is a mapping DW : Rd → R whereW is some parameter set. In this section, one can think of these parameter sets as made of matrices and vectors. To train the model, we consider the WGAN-GP problem formulation (Gulrajani et al., 2017) (where D̂ = D + (1− )Dz for > 0),
min V max W
Ez∼Dz [DW(GV(z))]−Ex∼D[DW(x)]+λEy∼D̂[(‖∇WDW‖2−1) 2] := f(V,W). (1)
Adaptive methods. In this paper, we particularly focus on second-moment-based adaptive optimizers to solve (1). Figure 1 captures the usual formulations of Adam (Kingma & Ba, 2014), Adagrad (Duchi et al., 2011) and RMSprop (Tieleman et al., 2012) up to some scaling conventions that can be absorbed into {(η(k)G , η (k) D )}. Note that all operations on vectors in Figure 1 are entry-wise and g2 denotes the entry-wise power on vector g.
The exponential window parameters β1, β2 ∈ [0, 1) respectively denote the first- and second-order momentum parameters. A rule of thumb is to set them as β1 = 0.5 and β2 = 0.9. As we aim to understand the role of adaptivity, an important baseline is stochastic gradient descent ascent with momentum (SGDA) which consists in the update of Figure 1when the denominator √ MY,2 + is omitted. In what follows, we refer to SGDA as the SGDA with momentum update.
Optimizers as autorefalg:adaptive are adaptive in that they keep updating step-sizes while training the model. Therefore, two types of schedules are involved: the explicit step size schedules {(η(t)G , η (t) D )} that the practitioner manually sets up and the implicit step size schedules induced by the optimizers. The contributions of these two schedules overlap and it remains unclear how they contribute to the superior performance of adaptive methods relative to SGDA in GANs.
Method dissection using grafting. To decouple the explicit and implicit schedules, we adopt the step size grafting approach proposed by Agarwal et al. (2020) and described as follows. At each iteration, we compute stochastic gradients, pass them to two optimizers A1,A2 and make a grafted step which combines the magnitude of A1’s step and direction of A2’s step. Since our goal is to understand the benefits of adaptivity, it is natural to consider the grafting approach with Adam and SGDA. We define AdaLR, an optimizer that updates in the SGDA direction with Adam magnitude:
W(t+1) =W(t) + η(t)D ‖A (t) W ‖2
M (t) W,1
‖M(t)W,1‖2 + ε , V(t+1) = V(t) − η(t)G ‖A (t) V ‖2
M (t) V,1
‖M(t)V,1‖2 + ε , (2)
and AdaDir which updates in the Adam direction with SGDA magnitude
W(t+1) =W(t) + η(t)D ‖M (t) W,1‖2
A (t) W
‖A(t)W ‖2 + ε , V(t+1) = V(t) − η(t)G ‖M (t) V,1‖2
A (t) V
‖A(t)V ‖2 + ε . (3)
Note that two implementations are possible for AdaLR and AdaDir. In the layer-wise version, Y(t) is a single parameter group (typically a layer in a neural network) and therefore, the update is applied to each group. In the global version, Y(t) contains all of the model’s weights. Note that in Figure 2, we set up AdaLR and AdaDir with the layer-wise version –the global AdaLR and AdaDir perform approximately as well as their layer-wise counterparts.
We train a WGAN-GP (Arjovsky et al., 2017) on CIFAR-10 with AdaLR, AdaDir and Adam. Figure 2(a) shows the GAN performance measured in FID score (Heusel et al., 2017) obtained by the three trained models. We observe that AdaDir does not generate samples from the desired distribution as its FID score is high. On the other hand, AdaLR performs slightly better than Adam. Therefore, we deduce that the success of adaptive methods in GANs may be explained by the implicit step-size schedule induced by the algorithm.
From AdaLR to normalized SGDA. The previous experiment hints that AdaLR which combines Adam’s magnitude and SGDA direction performs better than Adam. We here take a closer look at this algorithm. In particular, we investigate the fluctuations of Adam’s magnitude during training. Figures 2(b),2(c) show that this magnitude barely varies for both generator and discriminator.
Therefore, an update with SGDA direction and constant step-size seems to be a valid approximation of Adam in GAN training. Such an algorithm corresponds to normalized SGDA (nSGDA),
W(t+1) =W(t) + η(t)D M
(t) W,1
‖M(t)W,1‖2+ε , V(t+1) = V(t) − η(t)G
M (t) V,1
‖M(t)V,1‖2+ε . (4)
Similarly to AdaLR and AdaDir, we also define two versions of nSGDA: layer-wise nSGDA (lnSGDA) and global nSGDA (g-nSGDA). We use both of these in the numerical experiments. As a comparison to (4), the SGDA (with momentum) update is:
W(t+1) =W(t) + η(t)D M (t) W,1, V (t+1) = V(t) − η(t)G M (t) V,1. (5)

heading: 3 NUMERICAL PERFORMANCE OF NSGDA
text: Section 2 indicates that nSGDA may work as well as Adam in GAN training. In this section, we empirically verify this hypothesis through an extensive numerical study. To evaluate the proposed algorithm, we conducted extensive experiments on CIFAR-10 (Krizhevsky et al., 2009), LSUN Churches (Yu et al., 2016), STL-10 (Coates et al., 2011) and Celeba-HQ (Liu et al., 2015). Similarly to above, we choose the Fréchet Inception distance (FID) (Heusel et al., 2017) to quantitatively assess the performance of the model. In all our experiments, 50k samples are randomly generated for each model to compute the FID. As for the architectures, we choose Resnets (He et al., 2016) from Gidel et al. (2018) and set up the WGAN-GP loss (Gulrajani et al., 2017). Note that for each optimizer, we grid-search over stepsizes to find the best one in terms of FID. Due to limited computational resources, we trained the models for 100 epochs in the case of CIFAR-10 and Celeba-HQ
and for 50 epochs for LSUN and STL-10. We apply a linear decay learning rate scheduling during training. All the results are averaged over 10 seeds.
nSGDA competes with Adam. Figure 4 shows the performance of l-nSGDA, g-nSGDA, Adam, and SGD on different datasets. We first observe that l-nSGDA and g-nSGDA compete with Adam
in all these datasets. On the other hand, SGD struggles to produce good quality images compared to the other methods as expected. It is worth reminding that the nSGDA methods and SGD have the same direction but differ in their magnitude. Therefore, this experiment confirms our hypothesis that adaptive methods outperform SGDA thanks to their implicit step-size schedule. Lastly, we highlight that l-nSGDA and g-nSGDA perform almost as well. This suggests that a global step-size adaptivity is enough to perform well in GANs. Similar observations hold for DCGAN (see Appendix A).
Influence of batch size. While the nSGDA methods compete with Adam in GAN, it remains unclear whether such performance is proper to the algorithm or tied to the stochastic noise of the gradients. Figure 3 shows the performance of the model trained with l-nSGDA, g-nSGDA, Adam, and SGD and using a wide range of batch sizes. We remark that the nSGDA methods perform better than Adam when the batch size is small. However, as the batch size increases, their performance seriously deteriorates. On the other hand, the performance of Adam seems to be less sensitive to the change of batch size. As expected, SGD overall performs poorly. nSGDA performs well in a relevant batchsize regime, as many GAN architectures such as DCGAN, WGAN-GP, SNGAN (Miyato et al., 2018), SAGAN (Zhang et al., 2019a) requires small batch sizes for optimal performances.

heading: 4 WHY DOES NSGDA PERFORM BETTER THAN SGDA IN GANS?
text: In Section 2, we numerically observed that Adam outperforms SGDA mainly thanks to its adaptive magnitude (implicit step-size schedule). In Section 3, we observed that nSGDA methods – which differs from SGDA only by adaptive magnitude – compete with Adam and significantly surpass SGDA. To the best of our knowledge, there is no theoretical result that demonstrates the importance of adaptive magnitude in GANs performance. Thus, we set the following question:
From a theoretical perspective, for what distribution learning problem using GANs does nSGDA perform better comparing to SGDA?
We devise a simple data generation problem where the target distribution consists of two modes, described as vectors u1, u2, that are slightly correlated (See Assumption 1). We show that using standard GANs’ training objective, with high probability, SGDA – with any reasonable 1 step-size configuration – only learn distributions that suffer from mode collapse i.e. u1, u2 always show up in the learned distribution simultaneously. Conversely, nSGDA learns the two modes separately. Notations. We set the GAN 1-sample loss L(t)V,W(X, z) = log(D (t) W (X)) + log(1−D (t) W (G (t) V (z))). g (t) Y = ∇YL (t) V,W(X, z) is the 1-sample stochastic gradient (without momentum).

heading: 4.1 SETTINGS
text: In this section, we consider the classical GAN problem formulation.
min V max W EX∼D[log(DW(X))] + Ez∼Dz [log(1−DW(GV(z)))]. (6)
In the following paragraphs, we describe each element of our formulation.
Data distribution D. The data generation problem consists in having a training dataset with points sampled from a target distribution D. Using this dataset, our goal is to train a model that generates samples with the same statistics as D. The latter is defined as follows.
1Here reasonable simply means that the learning rates are bounded to prevent the training to blow up.
Let u1 and u2 two vectors in Rd such that ‖ui‖2 = 1, for i ∈ {1, 2}. Each sample from D consists of an input data X generated as:
1. Sample s = (s1, s2) from the distribution S defined as si ∈ [0, 1], P[si = 1] ≥ 1/2 for i ∈ {1, 2}, such that ‖s‖0 ≥ 1. Note that s1 and s2 are not necessarily independent.
2. Define data-point X = s1u1 + s2u2 ∈ Rd.
The distribution D generates points that are a linear combination of two modes u1 and u2. In what follows, we make the following assumptions on the coefficients (si) and modes (ui). Assumption 1 (Data structure). Let γ = 1polylog(d) . The coefficients s1, s2 and modes u1, u2 of the distribution D respect one of the following conditions:
1. Correlated Modes: 〈u1, u2〉 = γ and the generated data point is either X = u1 or X = u2. 2. Correlated Coefficients: P[s1 = s2 = 1] = γ and the modes are orthogonal, i.e., 〈u1, u2〉 = 0.
Assumption 1 captures some of the realistic structure of images. Case 1 models the setting where we have two pure modes (e.g., two different types of cats) that are correlated. Case 2 corresponds to two (roughly) orthogonal modes (e.g., vertical and horizontal edges), that may sometimes be mixed together (images containing object corners in the example above).
Learner models. To learn the true distribution D, we use a linear generator GV defined as
GV(z) = V z = mG∑ i=1 vizi, (7)
where V = [v>1 , v > 2 , · · · , v>mG ] ∈ R mG×d is the weight matrix and z ∈ {0, 1}mG .We set V = {V }. Intuitively, GV outputs linear combinations of modes (vi). We assume that GV samples from the latent distribution Dz defined for z ∈ {0, 1}mG , ‖z‖0 ≥ 1 as:
∀i, j ∈ [mG], Pr[zi = 1] = Θ ( 1
mG
) , Pr[zi = zj = 1] =
1
m2Gpolylog(d) (8)
First, remark that vectors sampled from Dz are binary-valued. Although usual latent distributions in GANs are either Gaussian or uniform, the distribution Dz should be rather seen as modelling the weights’ distribution of a hidden layer of a deep generator. Indeed, Allen-Zhu & Li (2021) argue that the distributions of these hidden layers are sparse, non-negative, and non-positively correlated. Besides, in (8), Pr[zi = zj = 1] = 1m2Gpolylog(d)
ensures that that the output of the generator is only made of one mode with probability 1−o(1) and thus avoid creating weighted averages of the two or more vi’s (which might cause mode collapsing). To assess the distribution learned by GV , we also train a non-linear neural network as discriminator DW
DW(X) = sigmoid a ∑ i∈[mD] σ(〈wi, X〉) + λb  , σ(z) =  z3 if |z| ≤ Λ 3Λ2z − 2Λ3 if z > Λ 3Λ2z + 2Λ3 otherwise , (9)
where W = [w>1 , . . . , w > mD ] ∈ R mD×d and a, b ∈ R are the trainable parameters of DW , λ > 0 is a fixed scaling factor (specified below) and Λ = d0.2. For simplicity, we setW = {W,a, b}. Besides, σ(·) is the truncated degree-3 activation function—it is thus made Lipschitz, which is only needed in the proof to deal with the case where the generator is trained much faster than the discriminator. Note that this latter case is uncommon in practice.
Lastly, we choose a cubic activation as it is the smallest polynomial degree for the discriminator that is sufficient: as pointed out in Li & Dou (2020), with linear or quadratic activations, the generator can fool the discriminator by only matching the first and second moments of D. By doing so, the generator cannot recover the modes u1, u2 but only weighted averages of them even at the global optimal solution.
Algorithms. We solve the training problem (6) using SGDA and nSGDA with momentum = 0. For simplicity, we also set the batch-size to 1. At each step t, we sample X ∼ D and z ∼ Dz and define SGDA’s update as
W(t+1) =W(t) + ηDg(t)W , V (t+1) = V(t) − ηGg(t)V , (SGDA)
where ηD, ηG > 0 are constant step-sizes. On the other hand, nSGDA is defined by the update rule
W(t+1) =W(t) + ηD g
(t) W
‖g(t)W ‖2 , V(t+1) = V(t) − ηG
g (t) V
‖g(t)V ‖2 . (nSGDA)
Compared to (4), (nSGDA) is a global nSGDA update (without momentum) (i.e. we do not consider layer-wise normalization). Indeed, ‖g(t)Y ‖2 in the update refers to the sum of norms of the gradients with respect to a, b,W . We now detail how to set parameters in (SGDA) and (nSGDA). Parametrization 4.1. When running SGDA and nSGDA on (1), we set the parameters as
– Initialization: b(0) = 0, a(0) ∼ N (
0, 1mDpolylog(d)
) , w
(0) i ∼ N ( 0, 1dI ) , v (0) j ∼ N ( 0, 1d2 I ) for i ∈ [mD], j ∈ [mG].
– Number of iterations: we run SGDA for t ≤ T0 iterations where T0 is the first iteration such that ‖∇E[LV(T0),W(T0)(X, z)]‖2 ≤ 1/poly(d). . For nSGDA, we run for T1 = Θ̃ ( 1 ηD ) iterations.
– Step-sizes: For SGDA, ηD, ηG ∈ (0, 1poly(d) ). For nSGDA, ηD ∈ (0, 1 poly(d) ], ηG = ηD polylog(d) .
– Over-parametrization: For SGDA, mD,mG = polylog(d) are arbitrarily chosen i.e. mD may be larger than mG or the opposite. For nSGDA, we set mD = log(d) and mG = log(d) log log d.
Parametrization 4.1 corresponds to usual initialization and optimization hyper-parameters. Regarding initialization, the discriminator’s weights are sampled from a standard normal and its bias is set to zero. The weights of the generator are initialized from a normal with variance 1/d2 (instead of the 1/d in standard normal). Such a choice is explained as follows. In practice, the target X ∼ D is a 1D image, thus has entries in [0, 1]d and norm O( √ d). Yet, we sample the initial generator’s weights from N (0, Id/d) in this case. In our case, since ‖ui‖2 = 1, the target X = s1u1 + s2u2 has norm O(1). Therefore, we scale down the variance in the normal distribution by a factor of 1/d to match the configuration encountered in practice. Therefore, we also set λ = 1√
dpolylog(d) in (9)
to ensure that the weights and the bias in the discriminator learn at the same speed.
Regarding the number of iterations, our theorem holds when running SGDA for any (polynomially) possible number of iterations –after T0 steps, the gradient becomes inverse polynomially small and SGDA essentially stops updating the parameters. Lastly, we allow any step-size configuration for SGDA i.e. larger, smaller, or equal step-size for D compared to G. To our knowledge, this setting has never been studied in GAN performance. Note that our choice of step-sizes for nSGDA matches with the one used in practice i.e. ηD slightly larger than ηG.

heading: 4.2 MAIN RESULTS
text: We state our main results on the performance of models trained using (SGDA) and (nSGDA). We show that nSGDA is able to learn the two modes of the distribution D while SGDA is not. Theorem 4.1 (SGDA suffers from mode collapse). Let T0, ηG, ηD and the initialization as defined in Parametrization 4.1. Let t be such that t ≤ T0. Run SGDA for t iterations with step-sizes ηG, ηD. Then, with probability at least 1− o(1), for all z ∈ {0, 1}mG , we have:
G (t) V (z) = α (t)(z)(u1 + u2) + ξ (t)(z), where α(t)(z) ∈ R and ξ(t)(z) ∈ Rd,
such that for all ` ∈ [2], |〈ξ(t)(z), u`〉| = o(1)‖ξ(t)(z)‖2 for every z ∈ {0, 1}mG . In the specific case where ηG = √ dηD polylog(d) , the model mode collapses i.e. ‖ξ (T0)(z)‖2 = o(α(T0)(z)).
Theorem 4.1 indicates that when using SGDA with any step-size configuration, the generator either does not learn the modes at all – when α(t)(z) = 0, G(t)V (z) = ξ (t)(z) – or learns an average of the
modes – when α(t)(z) 6= 0, G(t)V (z) ≈ α(t)(z)(u1 + u2). We emphasize that the theorem holds for any time t ≤ T0 which is the iteration where SGDA converges to an approximate first-order locally optimal min-max equilibrium. Conversely, nSGDA succeeds to learn the two modes separately. Theorem 4.2 (nSGDA recovers modes separately). Let T1, ηG, ηD and the initialization as defined in Parametrization 4.1. Run nSGDA for T1 iterations with step-sizes ηG, ηD. Then, the generator learns both modes u1, u2 i.e.,
Pr z∼Dz (∥∥ G(T1)V (z) ‖G(T1)V (z)‖2 − u` ∥∥ 2 = o(1) ) = Ω̃(1) , for ` = 1, 2 . (10)
Theorem 4.2 indicates that when we train a GAN with nSGDA in the regime where the discriminator updates slightly faster than the generator (as done in practice), the generator successfully learns the distribution containing the direction of both modes.

heading: 4.3 WHY DOES SGDA SUFFER FROM MODE COLLAPSE?
text: We now sketch the reason why SGDA suffers from mode collapse using Figure 5. Figure 5(a) displays the relative update speed η‖g (t) Y ‖2
‖Y(t)‖2 , Figure 5(b) the correlation 〈w (t) i ,u`〉 ‖w(t)i ‖2 between D’s neuron
and mode u` and Figure 5(c) the correlation 〈v(t)j ,u`〉 ‖v(t)j ‖2 between G’s neuron and mode u`.
We focus on the case where the discriminator’s step-size is set so it updates slightly faster than the generator at the beginning – as displayed in iterations 1–5 in Figure 5(a). The key observation is that in the early stages, the discriminator’s update is approximately
w (t+1) i ≈ w (t) i + ηD〈w (t) i , X〉 2X, for i ∈ [mD]. (11)
With high probability, there exists at least a neuron i such that 〈w(t)i , X〉 > 0. Thus, (11) implies that 〈w(t)i , X〉 is an increasing sequence. As t increases, w (t) i gradually grows its correlation with one of the modes u` (iterations 1-20 in Figure 5(b)) and D’s gradient norm thus increases. Therefore, after a first phase where D updates slightly faster than G, D’s update speed becomes significantly larger than G’s one (iterations 5–20 in Figure 5(a)). Thus, D learns the first mode after 20 iterations in Figure 5(b).
However, one of the goal of D (from the training objective in (6)) is to maximize its average correlation with the target distribution D. Since G does not catch up and D is made of two modes showing up with equal probability, the optimal solution forD is to have each of itswi = αi(u1+u2) to maximize the average correlation. Therefore, after iteration 20, D learns the second mode and eventually gets this optimal solution (which is a mode collapse) at iteration 40. Then, D’s update speed starts dropping as we see in Figure 5(a) which helps G to catch up and grow its update speed. However, since D already learnt a weighted average of the modes, it can only teach G to learn this average and thus mode collapses as we see in Figure 5(c).
On the other hand, nSGDA ensures thatG andD always learn at the same speed, so thatG can learn one mode immediately when D learns (such as at iteration 25 in (b) in Figure (5)), which avoids mode collapse.

heading: 5 CONCLUSION
text: Our work offers a complementary view to several works in the min-max optimization literature where the discriminator is much faster than the generator to converge to an equilibrium. Here, instead, we advocate the use of balanced updates to ensure that the GAN performs well.
Our work is a first step towards understanding how adaptive methods improve the GAN performance. Our empirical observations and theorems heavily rely on the fact that the batch size is small. However, nSGDA methods seem to not work well for large batch sizes and may not be suitable to some large-scale GANs such as BigGAN (Brock et al., 2018). It would be interesting to understand why adaptive methods are crucial in this case. Another interesting direction would be to improve the training of nSGDA in the batch setting.

Review summaries:
This paper studies how adaptive methods help performance in GANs. The study empirically finds that SGDA with the same vector norm as Adam reaches similar better performance. Based on this observation, normalized SGDA (nSGDA) is proposed as a simpler alternative to Adam. nSGDA is evaluated on several datasets and the results demonstrate that nSGDA is more stable than SGDA. 

In this manuscript the authors investigate the effect of normalized and unormalized gradient updates on the convergence of GANs. In a first experiment a DCGAN model is trained with Adam and it is shown that the best models evaluated with the FID have balanced learning rates but haven't converged while models trained with extremly unbalanced learning rates converge while having high FIDs i.e. didn't learn the training distribution. In the second experiment it is shown that an adaptive gradient magnitude helps to train a good model compared to an adaptive gradient direction. In a next step using the WGAN-GP model on CIFAR-10, LSUN Churches, STL-10, and CelebA-HQ datasets the optimizers Adam, normalized gradient updates and unnormalized SGDA were compared. The normalized optimizers including Adam outperformed SGDA. Finally it was shown that SGDA is more sensitive to the batch-size. In a final experiment on a small toy dataset it was shown that SGDA suffers from mode collapse while normalized gradient updates guide the generator to learn the modes of the dataset. 

This paper dissects the success of Adam in GAN training by swap the gradient direction/magnitude with that of SGDA. By doing so, authors found the Adam produces higher quality solutions relative to SGDA in GANs mainly due to its adaptive magnitude and not to its adaptive direction. Inspired by this observation, the combination of direction of SGDA and magnitude of Adam yields the use of normalized SGDA (nSGDA) in GAN training, which consistently competes with Adam.