Adam is no better than normalized SGD:  Dissecting how adaptivity improves GAN performance
Authors: Samy Jelassi
Arthur Mensch
Gauthier Gidel
Yuanzhi Li
Venue: ICLR 2022 Submitted (2022)
Keywords: Generative adversarial networks
non-convex optimization

Abstract:  Adaptive methods are widely used for training generative adversarial networks (GAN). While there has been some work to pinpoint the marginal value of adaptive methods in minimization problems, it remains unclear why it is still the method of choice for GAN training. This paper formally studies how adaptive methods help performance in GANs.  First, we dissect Adam---the most popular adaptive method for GAN training---by comparing with SGDA the direction and the norm of its update vector. We empirically show that SGDA with the same vector norm as Adam reaches similar or even better performance than the latter. This empirical study encourages us to consider normalized stochastic gradient descent ascent (nSGDA) as a simpler alternative to Adam. We then propose a synthetic theoretical framework to understand why nSGDA yields better performance than SGDA for GANs.  In that situation, we prove that a GAN trained with nSGDA provably recovers all the modes of the true distribution. In contrast, the same networks trained with SGDA (and any learning rate configuration) suffers from mode collapsing. The critical insight in our analysis is that normalizing the gradients forces the discriminator and generator to update at the same pace. We  empirically show the competitive performance of nSGDA on real-world datasets.
     

Under review as a conference paper at ICLR 2022
ADAM IS NO BETTER THAN NORMALIZED SGD: DISSECTING HOW ADAPTIVE METHODS IMPROVE GANS PERFORMANCE
Anonymous authors Paper under double-bling brindd review
ABSTRACT
Adaptive methods are widely used for training generative adversarial networks (GAN). While there has been some work to pinpoint the marginal value of adap- tive methods in minimization problems, it remains unclear why it is still the method of choice for GAN training. This paper formally studies how adaptive methods help performance in GANs. First, we dissect Adam-the most popular adaptive method for GAN training- by comparing with SGDA the direction and the norm of its update vector. We empirically show that SGDA with the same vector norm as Adam reaches similar or even better performance than the lat- ter. This empirical study encourages us to consider normalized stochastic gradient descent ascent (nSGDA) as simpler alternative to Adam. We then propose a synthetic theoretical framework to understand why nSGDA yields better perfor- mance than SGDA for GANs. In that situation, we prove that a GAN trained with nSGDA provably recovers all the modes of the true distribution. In contrast, the same networks trained with SGDA (and any learning rate configuration) suffers from mode collapsing. The critical insight in our analysis is that normalizing the gradients forces the discriminator and generator update the same pace. We empirically show the competitive performance of nSGDA on real-world datasets.
INTRODUCTION
It commonly accepted that adaptive algorithms are required train modern neural network chitectures in various deep learning tasks. This includes minimization problems that arise in natural language processing (Vaswani et al., 2017) and fMRI (Zbontar al., 2018) or min-max problems such as generative adversarial network (GAN) training (Goodfellow et al., 2014). Indeed, it has been empirically observed that Adam (Kingma & Ba, 2014) yields a solution with better generalization than stochastic gradient descent (SGD) in these problems (Choi et al., 2019). Several works have attempted explain this phenomenon in the minimization case. Common explanations are that adaptive methods train faster (Zhou et al., 2018), escape faster very flat saddle-point like plateaus (Orvieto et al., 2021) or deal better with heavy-tailed stochastic gradients (Zhang et al., 2019b). However, much less is known regarding min-max problems such as GANs. In this paper, we inves- tigate why GANs trained with adaptive methods outperform those trained using stochastic gradient descent ascent with momentum (SGDA).
Some prior works attribute this outperformance to the superior convergence speed of adaptive meth- ods. For instance, Liu et al. (2019) show that variant of Optimistic Gradient Descent (Daskalakis et al., 2017) converges faster than SGDA for a class of non-convex non-concave min-max problems. However, contrary the minimization setting, convergence and not even a requirement to ensure a satisfactory GAN performance. Indeed, Mescheder et al. (2018) empirically shows that popular architectures such as Wasserstein GANs (WGANs) (Arjovsky et al.. 2017) do not always converge, and yet produce realistic images. We support this observation through the following experiment. We train DCGAN (Radford et al., 2015) using Adam -the most popular adaptive method- and set up the generator (G) and discriminator (D) step-sizes respectively as TiD, TJ. Note that D is usually trained faster than G i.e. 7D 2 TJ. Figure 1(a) displays the GAN convergence measured by the ratio of gradient norms, and the GAN's performance measured in FID score (Heusel et al., 2017). We observe that when P// is close to 1, the algorithm does not converge and yet, the model produces high-quality solutions. On the other hand, when np /// >>> 1, the model converges to an equilibrium -a similar statement has been proved by Jin et al. (2020) and Fiez. & Ratliff (2020) in the case of SGDA. However, the GAN produces low-quality solutions
stationary point is guaranteed

---

Under review as a conference paper at ICLR 2022
10"
10
Gradient ratio
10
Gradient ratio
10'
10"
10
100
10
10
Discriminator (0.4 (0.41) k
10
Generator (1.38)
10"
10
20
40
60
80
100
100 203
300
FID
Epochs
(a) Each circle corresponds to a specific step-size configuration no /nc. The pest-igr mod- els have step-size ratios between 10 and 1, and do not converge. As TD / increases, the models perform worse but get closer to an equilibrium.
(b) shows that during training, the gradient ratio of a perf-orrnnnnnng GAN approximately stays con- stant to 1. We also display the images produced by the model during training.
Gradient ratio against FID score (a) and number of epochs (b) obtained with DCGAN CIFAR- 10. This ratio is equal to l (4) ||a/ la/llgrad (0) ||2 + llgrad //2//llrad D0 where grad (resp. grad and grads (0) (resp. grad are the current and initial gradients of G (resp. D). Note that || ll refers the sum of all the parameters norm a network. For the plots, the models are trained for 100 epochs using batch-size
Figure 1:
(0))
64. For (b), the results are averaged over 5 seeds.
this equilibrium. Thus, simply comparing the convergence speed of adaptive methods and SGDA cannor explain the GAN's performance obtained with adaptive methods. This observation motivates the central question in this paper: What factors explain that Adam produces better quality solutions than SGDA when training GANs?
To address this question, we dissect Adam following the approach by Agarwal et (2020). They frame generic optimizer's update as ((t-)) W(() ya where W E Rd is the iterate, g(t) Rd such that [| [|2 magnitude. Therefore, first step in our paper is to understand whether Adam outperforms SGDA maintly due its direction or its magnitude. As detailed in Section 2, we train GAN using i) AdaLR, an algorithm that updates in the direction of SGDA but with the magnitude of Adam ii) AdaDir which uses the direction of Adam but magnitude of SGDA. We empirically show that not only, AdaLR significantly outperforms AdaDir, SGDA, and Adam itself. This observation encourages us to conclude that:
the optimizer's direction and (t)) > 0 is the optimizer's
Adam produces higher quality solutions relative to SGDA GANs mainly due to its adaptive magnitude and not to its adaptive direction.
In Section 2, we empirically analyze the adaptive magnitude of AdaLR and observe that it stays approximately constant throughout training. This observation eventually encourages the study of AdaLR with constant step-size. Such algorithm actually corresponds to normalized SGDA (nS- GDA). to SGDA, nSGDA has the same direction but differs in magnitude since we divide the gradient by its norm. Intuitively, this normalization forces D) and G to be updated by vectors with constant magnitudes no matter how different the norms of D's and G's gradicnts are.
Motivated by the aforementioned observations, this paper studies the performance of GANs trained with nSGDA. We believe that this is a first step to formally understand the role of adaptive methods in GANs. Our contributions divided as follows:
In Section 3. we experimentally confirm that nSDGA consistently competes with Adam and out- performs SGDA when using different GAN architectures on wide range of datasets.
In Section 4, we provide a theoretical explanation on why GANs trained with nSGDA outperform those trained with SGDA. More precisely, we devise data generation problem where the target distribution D is made of multiple modes. The model trained with nSGDA provably recovers all the modes in the target distribution while the SGDA's figuration: We prove that even when SGDA converges to locally optimal min-max equilibrium, the model still suffers from mode collapsing and fails learn recover the modes separately.
fails do it under any step-size
The key insight of our theoretical analysis is that no matter how the step-sizes prescribed, D and G necessarily update at very different speeds when we use SGDA. Therefore, i) either D updates its weights too fast, thus learns weighted average of the modes of D. This makes G learn this
2

---

Under review as a conference paper at ICLR 2022
weighted average of modes ii) or D does not update its weights fast enough and thus, G aligns its weights with those of D. This forces D to converge to a locally optimal min-max equilibrium that classifies any instance as "fake". On the other hand, by normalizing the gradients done in SGDA, we force D and G to update at the same speed throughout training. Thus, whenever D learns a mode of the distribution, G learns it right after, which makes both of them learn all the modes of the distribution separately.
Our paper advocates for the use of balanced updates in GAN training i.e. the ratio of D vs G updates should remain close to constant. To our knowledge, we are the first to theoretically show the importance of these balanced updates. This insight contrasts with the related work that analyzes GANs, and more generally zero-sum differentiable games, in the regime where D is updated much faster than G i.e ///1G/ >> 1 (Fiez& & Ratliff, 2020; Jin et al., 2020; Fiez et al., 2020). RELATED WORK
Adaptive methods in games optimization. Several works designed adaptive algorithms and ana- lyzed their convergence to show their benefits relative to SGDA. For variational inequality problems, Gasnikov et al. (2019); Antonakopoulos et al. (2019); Bach & Levy (2019); Antonakopoulos et al. (2020) propose adaptive algorithms that reach optimal convergence rates under regularity assump- tions. For a class of non-convex non-concave min-max problems, Liu et al. (2019); Barazandeh et al. (2021) design algorithms that converge faster than SGDA. On the other hand, Heusel et al. (2017) show that Adam locally converge to a Nash equilibrium in the regime where the step-size of the discriminator is much larger than the one of the generator. Our work differs from these papers as we analyze Adam and do not focus on the convergence properties but rather on the fit of the trained model on the true (and not empirical) data distribution. Besides, contrary to some of the aforemen- tioned papers, our work is not in the two-time scale learning- maaiin regime which do not correspond to what is mostly done in practice.
Importance of balanced updates in GANs. The importance of balanced updates for GANs has been noticed in the literature. The most popular GAN architectures (Radford et al., 2015; Arjovsky et al., 2017: 7; Brock et al., 2018) set the step-sizes such that ?D/NG is constant. On the other hand, Berthelot al. (2017) introduced a control variable ensure that the updates between G and D are at the same speed. In this work, we do not modify the GAN objective loss to enforce balancedness. Instead, we empirically and theoretically investigates how Adam and nSGDA enforce balanced up- dates.
Statistical results in GANs. Early works investigate whether GANs memorize the training data or actually learn the distribution (Arora et al., 2017; 2018; Dumoulin et al., 2016). Zhang et al. (2017); Bai et al. (2018) then show that for specific GANs, the model learn some distributions with non- exponential sample complexity (Liang, 2017: Feizi et al., 2017). Recently, Li & Dou (2020); Allen- Zhu & Li (2021) 1) further characterized the distributions learned by the generator. On the other hand, some works attempted to explain GAN performance through the optimization lens. Lei et al. (2020); Balaji et al. (2021) show that GAN models trained with SGDA converge to a global saddle point when the generator is one-layer neural network and the discriminator is specific quadratic/linear function. Our contribution significantly differs from these two works as i) we construct a setting where SGDA converges to locally optimal min-max equilibrium and yet suffer from mode collapse. Conversely, nSGDA does not necessarily converge and yet, recovers the true distribution ii) our setting is more challenging since we need at least a degree-3 discriminator learn the distribution see Section 4 for a justification.
Normalized gradient descent. Introduced by Nesterov (1984), normalized gradient descent has been widely used in the minimization setting. Indeed, it has been observed that normalizing out the gradient improves the 'slow crawling' problem of gradient descent and avoids the iterates to be stuck in flat regions- such as spurious local minima or saddle points (Hazan et al., 2015; Levy, 2016; Murray et al., 2019). Normalized gradient descent or its variants outperform the nor-mozml counterparts in multi-agent coordination (Cortés, 2006) and deep learning tasks (You et al., 2017; 2019; Cutkosky& & Mehta, 2020; Liu et al., 2021). Our work rather considers the min-max setting and shows that nSGDA performs better than SGDA as it forces the discriminator and generator to update at the same rate.
FROM ADAM TO NSGDA
Generative adversarial networks. Given a training set sampled from some target distribution D, GAN learns to generate new data from this distribution. The architecture is constituted of two
3

---

Under review as a conference paper at ICLR 2022
Algorithm 1 Generic second moment adaptive optimizer
Input: initial points W((), v(O), step-size schedules (in&) )} hyperparameters {31, 82, Initialize M W. M (0)
M and My
(0)
W:2
v22 zero.
for /. = . T do
(2) Receive stochastic gradients Bw By evaluated at W(() and V((1)
Compute gradient oracles for E (V, W): AJ (t+1) y. B.M VIE + gy
Update accumulators for y/ E (W, V), E [2]: M (++1)
(t)
-555 //M +e.
Update: W((+1) W - (1) A Aw (1+1) return W p(T).
1D
V(++) v(t) (2)
no Av (t+1)
networks: the generator maps points in the latent space D3 to sample candidates of the desired distribution; the discriminator evaluates these samples by comparing them to samples from D.
More formally, the generator is mapping Gy: Rt + I2 where V is some parameter set. Generally, the latent variables are sampled from the normal distribution. On the other hand, the discriminator is mapping Dw: Rd - IR where W is some parameter set. In this section, one can think of these parameter sets as made of matrices and vectors. To train the model, we consider the WGAN-GP problem formulation (Gulrajani et al., 2017) (where D = cD + (1 e)D; for e > 0),
min max EZ-D, [Dw(Gv(2)) -Ez- + AE yy Dwlla 1) = f(V,W) (1) V W
Adaptive methods. In this paper, we particularly focus on second-moment-basee adaptive optimiz- solve (I). Figure 1 captures the usual formulations of Adam (Kingma & Ba, 2014), Adagrad (Duchi et al., 2011) and RMSprop (Tieleman et al., 2012) up to some scaling conventions that can be absorbed into {(Ta) g² denotes the entry-wise power vector g-
(k) "DD () Note that all operations on vectors in Figure are entry-wise and
The exponential window parameters 31, 32 E [o, 1) respectively denote the first- and second-order momentum parameters. A rule of thumb is to set them as 81 = 0.5 and B2 = 0.9. As we aim to understand the role of adaptivity, an important baseline is stochastic gradient descent ascent with momentum (SGDA) which consists in the update of Figure I when the denominator / + omitted. In what follows, we refer to SGDA as the SGDA with momentum update.
Optimizers as addapiv are adaptive in that they keep updating step-sizes while train- ing the model. Therefore, two types of schedules are involved: the explicit step size schedules {(TG
(t) ( that the practitioner manually sets up and the implicit step size schedules induced by
the optimizers. The contributions of these two schedules overlap and it remains unclear how they contribute to the superior performance of adaptive methods relative to SGDA in GANs.
Method dissection using grafting. To decouple the explicit and implicit schedules, adopt the step size grafting approach proposed by Agarwal et al. (2020) and described as follows. At each iteration, we compute stochastic gradients, pass them to two optimizers A1, A2 and make grafted step which combines the magnitude of A 's step and direction of AAs's step. Since our goal is to understand the benefits of adaptivity, it is natural to consider the grafting approach with Adam and SGDA. We define AdaLR, an optimizer that updates in the SGDA direction with Adam magnitude:
(1) M
M
(0) W(+++ = w(t) + +
W.1
p(++)) = plu
NGG
1A
(2)
|M w.
+e
M
villa
+e"
and AdaDir which updates in the Adam direction with SGDA magnitude
A | v
w(++1) = wio + +TD (6) M Wallz
D(++) = ylu)
ni |M
vill2
(3)
2 + W
+
Note that two implementations are possible for AdaLR and AdaDir. In the layer-wise version, p)()) a single parameter group (typically a layer in neural network) and therefore, the update is applied to each group. In the global version, yxt)) contains all of the model's weights. Note that in Figure 2, we set up AdaLR and AdaDir with the layer-wis- wise version -the global AdaLR and AdaDir perform approximately as well as their layer-wise counterparts.
4

---

Under review as a conference paper at ICLR 2022
130
Discrimiaato
Adem 124.58)
AdeLR 122 (22.54) AdaDDi (303.11)
24
24
27
310
2.0
20
-33
18
-55
16
1e2
14
Ted
-22
Foramoters
20
100
Number of epochs
(b)
(c)
(a)
Figure 2: (a) FID score against the number of epochs of a Resnet WGAN-GP trained on CIFAR-10 with Adam, AdaLR, and AdaDir. AdaLR performs slightly better than Adam while AdaDir performs very poorly. (b)-(c) displays the fluctuations of AdaLR's adaptive magnitude. We plot the ratio || (t) lla/ y |l for cach discriminator's 8 (b) and generator's 's (c) parameters. At early stages, this ratio barely increases and remains constant after steps. We train for 100 epochs using batch-size 64 and results averaged over seeds.
We train a WGAN-GP (Arjovsky al., 2017) on CIFAR-10 with AdaLR, AdaDir and Adam. Fig- ure 2(a) shows the GAN performance measured in FID score (Heusel et al., 2017) obtained by the three trained models. We observe that AdaDir does not generate samples from the desired distri- bution as its FID score is high. On the other hand, AdaLR performs slightly better than Adam. Therefore, we deduce that the success of adaptive methods GANs may be explained by the im- plicit size schedule induced by the algorithm.
From AdaLR to normalized SGDA. The previous experiment hints that AdaLR which combines Adam's 's magnitude and SGDA direction performs better than Adam. We here take closer look at this algorithm. In particular, we investigate the fluctuations of Adam's magnitude during training. Figures 2(b),2(c) show that this magnitude barely varies for both generator and discriminator.
Therefore, an update with SGDA direction and constant step-size seems to be a valid approximation of Adam in GAN training. Such an algorithm corresponds normalized SGDA (nSGDA),
W = py(t) + +nD
(2)
[MW 1ill-==
M
((++1) = ((t)
GG
(e)
JIM laae llate
M
(4)
Similarly to AdaLR and AdaDir, we also define two versions of nSGDA: layer-wise nSGDA (I- nSGDA) and global nSGDA (g-nSGDA), We use both of these in the numerical experiments. As comparison to (4), the SGDA (with momentum) update is:
W(++1 Mp(() DD
(t) M
((++1) Vl0)
(4)
M
(5)
NUMERICAL PERFORMANCE OF NSGDA
Section indicates that nSGDA may work well as Adam in GAN training. In this section, we empiri- cally verify this hypothesis through an extensive nu- merical study. To evaluate the proposed algorithm, we conducted extensive experiments on CIFAR-10 (Krizhevsky et al., 2009). LSUN Churches (Yu et al., 2016), STL-10 (Coates et al., 2011) and Celeba-HQ (Liu al., 2015). Similarly above, choose the Fréchet Inception distance (FID) (Heusel et al., 2017) quantitatively assess the performance of the model. In all our experiments, 50k samples are randomly generated for each model compute the FID. As for the architectures, we choose Resnets (He al., 2016) from Gidel al. (2018) and set up the WGAN-GP loss (Gulrajani et al., 2017). Note that for each optimizer, we grid-search over stepsizes to find the best one in terms of FID. Due to limited computational resources, we trained the models for 100 epochs in the case of CIFAR- 10 and Celeba-HO and for 50 epochs for LSUN and STL-10. We apply a linear decay learning rate scheduling during training. All the results are averaged over 10 seeds.
90
HiSODA
g-nSDDA
Adam
80
SGDA
40
20
36
128
256
512
Batch size
Figure 3: FID
of Resnet WGAN-GP model trained with I-nSGDA, g g-nSGDA Adam, and SGDA against the batch size in the CIFAR-I0 dataset. At small batch sizes, the best perform- ing models those trained with nSGDA meth- ods. As the batch size increases, the performance of nSGDA methods worsens and Adam performs better. Lastly, models trained with nSGDA con- sistently outperform those trained with SGDA.
nSGDA competes with Adam. Figure 4 shows the performance I-nSGDA, g nSGAA Adam, and SGD on different datasets. We first observe that I-nSGDA and g-nSGDA compete with Adam
5

---

Under review as a conference paper at ICLR 2022
CIFAR-10
LSUN-CHURCHES
STL-10
CELEBA-HO
92.5
45.0
60
18
90.0
42.5
27.5
60
25.0
55
Adam
knsGDA
gnsspa
SGDA
Adam AnSGDA
ansso
Adam
insGDA
onSGGD
SoDA
Adam
AnSGDA
SODA
(a) CIFAR-I0
(b) LSUN Churches
(c) STL-
(d) Celeba-HQ
Figure 4: FID scores obtained when training Resnet WGAN-GP using Adam, I-nSGDA, g-nSGDA, and SGD on different datasets. In these datasets, I-nSGDA, g-nSGDA and Adam perform approximately as well. SGDA performs much worse. The models are trained with batch-size 64.
in all these datasets. On the other hand, SGD struggles to produce good quality images compared to the other methods as expected. It is worth reminding that the nSGDA methods and SGD have the same direction but differ their magnitude. Therefore, this experiment confirms our hypothesis that adaptive methods outperform SGDA thanks to their implicit step-size schedule. Lastly, we highlight that I-nSGDA and g-nSGDA perform almost as well. This suggests that global step-size adaptivity is enough to perform well in GANs. Similar observations hold for DCGAN (see Appendix A).
Influence of batch size. While the nSGDA methods compete with Adam in GAN, remains unclear whether such performance is proper to the algorithm or tied to the stochastic noise of the gradients. Figure shows the performance of the model trained with I-nSGDA, g-nSGDA, Adam, and SGD and using a wide range of batch sizes. We remark that the nSGDA methods perform better than Adam when the batch size is small. However, as the batch size increases, their performance seriously deteriorates. On the other hand, the performance of Adam seems to be less sensitive to the change of batch size. As expected. SGD overall performs poorly, nSGDA performs well in a relevant batch- size regime, as many GAN architectures such as DCGAN, WGAN-GP, SNGAN (Miyato et al., 2018), SAGAN (Zhang et al., 2019a) requires small batch sizes for optimal performances.
4 WHY DOES NSGDA PERFORM BETTER THAN SGDA IN GANS?
In Section 2, we numerically observed that Adam outperforms SGDA mainly thanks to its adaptive magnitude (implicit step-size schedule). In Section 3, we observed that nSGDA methods which differs from SGDA only by adaptive magnitude - compete with Adam and significantly surpass SGDA. To the best of our knowledge, there is no theoretical result that demonstrates the importance of adaptive magnitude in GANs performance. Thus, we set the following question:
From theoretical perspective, for what distribution learning problem using GANs does nSGDA perform better comparing to SGDA?
We devise simple data generation problem where the target distribution consists of modes, described as vectors U1, Ug. that are slightly correlated (See Assumption 1), We show that using standard GANs" training objective, with high probability. SGDA with any reasonable step-size configuration only learn distributions that suffer from mode collapse i.e. U1, always show up in the learned distribution simultaneously. Conversely, nSGDA learns the two modes separately.
Notations. We set the GAN 1 1-sample loss ( (X, 2) = log log(D (X)) (X)) + log(1 - D (G (z))).
(2) 1. is the 1-sample stochastic gradient (without momentum).
g)
4.1 SETTINGS
In this section, we consider the classical GAN problem formulation.
min Ex-D Dog (X)()) Ez-D, [lo((1 Dw
(6)
In the following paragraphs, describe each element of our formulation.
Data distribution D. The data generation problem consists having a training dataset with points sampled from a target distribution D. Using this dataset, our goal is to train a model that generates samples with the same statistics as D. The latter is defined as follows.
Here reasonable simply means that the learning rates bounded prevent the training to blow up.

---

Under review as a conference paper at ICLR 2022
Let un and U2 two vectors in Ied such that ||ui||2 = 1, for E (1,2).
Each sample from D consists of an input data X generated as:
1.
Sample
($1, $2) from the distribution S defined as s, 10, Pls: = = 2 1/2 for E {1, 2), such that ||s|lo 2 1. Note that $1 and 82 are not necessarily independent.
Define data-point X 81 u + 821/ E R.d.
The distribution D generates points that are linear combination of two modes i and u2- In what follows, we make the following assumptions on the coefficients (s:) and modes (us).
Assumption (Data structure). Let polylog(z] The coefficients 81, and modes u1, of the distribution DD respect one of the following conditions:
Correlated Modes: (11, U22 = 7 and the generated data point either X = u1 or X = u2-
2. Correlated Coefficients: P[s1 82 = and the modes are orthogonal, i.e., (u1, U2) 0.
Assumption 1 captures some of the realistic structure of images. Case 1 models the setting where we have two pure modes (c.g., two different types of cats) that are correlated. Case 2 corresponds to two (roughly) orthogonal modes (e.g., vertical and horizontal edges), that may sometimes be mixed together (images containing object corners in the example above).
Learner models. To learn the true distribution D. we use a linear generator Gy defined
Gv(z)= = V= 2
where == [u] 12 ..... TT R is the weight matrix and {0,1) We set == {V}. Intuitively, Gy outputs linear combinations modes (v:). We assume that Gy samples from the latent distribution Dj defined for (0, 1) mc ||=1|o as:
1
Vi,j E [ma], Pr[z- 1] +
Pr[z, 111==22-11
(8)
mG
mopolyyog(d)
First, remark that vectors sampled from Dz are binary-valued. Although usual latent distributions in GANs are either Gaussian uniform, the distribution D= should be rather seen modelling the weights' distribution of a hidden layer of a deep generator. Indeed, Allen-Zhu & Li (2021) argue that the distributions of these hidden layers are sparse, non-negative, and non-positivelly correlated. Besides. in (8), Pr[z: = 2j = 1] polylooo(()) ensures that that the output of the generator is only
made of one mode with probability 1 1- 1-o(1) and thus avoid creating weighted averages of the two or more U 's (which might cause mode collapsing). To assess the distribution learned by Gy, also train a non-linear neural network as discriminator Dy
2
if | S A if > otherwise
Dw (X) sigmoid
a
Σ p]
alwi, X)) + Ab
(())
3A
2A3
(9)
3AA + 2A3
where W = [w]] ..... w "npp E Rd and a, b E R are the trainable parameters of Dw. À > 0 is a fixed scaling factor (specified below) and A = 2 For simplicity, we set W = (W, a. b).
Besides, a(-) is the truncated degree-3 activation function- thus made Lipschitz, which is only needed in the proof to deal with the case where the generator is trained much faster than the discriminator. Note that this latter case uncommon practice.
Lastly, we choose cubic activation as it is the smallest polynomial degree for the discriminator that sufficient: as pointed out in Li & Dou (2020), with linear or quadratic activations, the generator can fool the discriminator by only matching the first and second moments of D. By doing the generator cannot recover the modes U1, U2 but only weighted averages of them even at the global optimal solution.
We solve the training problem (6) using SGDA and nSGDA with momentun 0. For simplicity, also set the batch-size to 1. At each step t, we sample X D and D2 and define SGDA's update as
Algorithms.
w(++1)= w(t MDBW,
((++) = v(t) 1ogy
(SGDA)
7

---

Under review as a conference paper at ICLR 2022
Relalive uptete
10a
Disct intraltr Gienet senerater
Mode Made to
20"4
Correlation
06
0-1
2014
0.2
02
42
20 Iterations
40
0.0
leerations
Generator's
(C)
5/
V5
Generato's neurons
(d)
Figure 5: Learning process of SGDA (a,b,c) and learned modes by nSGDA (d) with mp 1000. We see that in the nSGDA model, each neuron solely learns one of the modes.
mg
5, d
where DID 1G > O are constant step-sizes, On the other hand, nSGDA is defined by the update rule
W(+++) = Wx(t +
W
D((+I) = v(t GG
(nSGDA)
gw
By 1|
Compared to (4), (nSGDA) is a global nSGDA update (without momentum) (i.e. we do not consider layer-wise normalization). Indeed, IBy ||2 in the update refers to the sum of norms of the gradients with respect to b, W. We now detail how to set parameters in (SGDA) and (nSGDA). Parametrization 4.1. When running SGDA and nSGDA on (I), we set the parameters as
(0, Initialization: ((0) = O, (l)) ~ ~N( NN00 mppolylog(d)
wi (0, of ~ (O,
for i [mb], j E [ma]-
Number of iterations: run SGDA for t S To iterations where To is the first iteration such that VVL wer (X, =)]l22 N 1/poly(d) For nSGDA, we run for Ti = - iterations.
SSte--iizs: For SGDA, nD, TlG E (0, poly(a)) ). For nSGDA, TID E (0, polytayl GG polylog(d)
Over-parametriaation: For SGDA, m.D. polylog(d) are arbitrarily chosen i.e. mp may be larger than mg or the opposite. For nSGDA, we set mp = (og(d) and mg = log(d) log log d.
Parametrization 4.1 corresponds to usual initialization and optimization hyper-parameters. Regard- ing initialization, the discriminator's weights are sampled from a standard normal and its bias is set to zero. The weights of the generator are initialized from a normal with variance 1/d (instead of the 1/d in standard normal). Such a choice is explained as follows. In practice, the target ~ D is a ID image, thus has entries in [o, 1jd and norm O(vd). Yet, we sample the initial generator's weights from (0, La/d) in this case. In our case, since ||w:|la = 1, the target X 8181 + 82122 has norm (1). Therefore, we scale down the variance in the normal distribution by factor of 1/d to match the configuration encountered in practice. Therefore, we also set A vdpolylog(a) (9)
to ensure that the weights and the bias in the discriminator learn at the same speed.
Regarding the number of iterations, our theorem holds when running SGDA for any (polynomially) possible number of iterations -after To steps, the gradient becomes inverse polynomially small and SGDA essentially stops updating the parameters. Lastly, we allow any step-size configuration for SGDA i.c. larger, smaller, or equal step-size for D compared to G. To our knowledge, this setting has never been studied in GAN performance. Note that our choice of step-sizes for nSGDA matches with the one used in practice i.e. hu slightly larger than nJ.
4.2 MAIN RESULTS
We state main results on the performance of models trained using (SGDA) and (nSGDA). We show that nSGDA is able to learn the two modes of the distribution D while SGDA is not.
dD and the initialization as defined in Parametrization 4.1. Let be such that t < To. Run SGDA for iterations with step-sizes nG. np- Then, with probability at least 1 for all (0, 13 we have:
Theorem 4.1 (SGDA suffers from mode collapse). Let To. TD
G (z) = (=)(() + u2) (2), where (3) E R and (2) E Rdd
such that for all é E [2). (x), uz) of o(1)]l{() for every = E 10,1)mm,
In the specific case where nG = Vánp
the model mode collapses i.e. (T) ||2 = o(a (=)).
polylog(d)'
Theorem 4.1 indicates that when using SGDA with any step-size configuration, the generator either does not learn the modes all when a (1) (2) 0, c (z) (z) or learns an average of the
8

---

Under review as a conference paper at ICLR 2022
(2) (0 u2). We emphasize that the theorem holds for any time /. x To which is the iteration where SGDA converges to an approximate first-order locally optimal min-max equilibrium. Conversely, nSGDA succeeds to learn the two modes separately.
modes when a (z) + O,
Theorem 4.2 (nSGDA recovers modes separately). Let T1, IJ, RD and the initialization as defined in Parametrization 4. I. Run nSGDA for Ti iterations with step-sizes nJ2 VJD- Then, the generator learns both modes u1, u2 i.e.
(I (+) (I uell2= = = =2(1), for == 1,2.
Pr DD
(10)
Theorem 4.2 indicates that when we train GAN with nSGDA in the regime where the discriminator updates slightly faster than the generator (as done practice), the generator successfully learns the distribution containing the direction of both modes.
4.3 WIYY DOES SGDA SUFFER FROM MODE COLLAPSE?
We now sketch the reason why SGDA suffers from mode collapse using Figure 5. Figure 5(a)
nig ll2
(w and between D's neuron
displays the relative update speed
Figure 5(b) the correlation
and mode ue and Figure 5(c) the correlation
between G's neuron and mode ug-
We focus on the case where the discriminator's step-size is set so it updates slightly faster than the generator at the beginning as displayed iterations 1-5 in Figure 5(a). The key observation is that in the early stages, the discriminator's update is approximately
(1+1) = (1) + no(u X)" w
for í E [mp].
(11)
With high probability, there exists at least a neuron such that (u(",x) .X) > O. Thus, (11) implies that (w,"), (t) X) is an increasing sequence. As increases, w(() gradually grows its correlation with one of the modes uy (iterations 1-20 in Figure 5(b)) and D's gradient norm thus increases. Therefore, after a first phase where D updates slightly faster than G, D's update speed becomes significantly larger than G's one (iterations 5-20 in Figure 5(a)). Thus, D learns the first mode after 20 iterations in Figure 5(b).
However, one of the goal of D (from the training objective in (6)) is to maximize its average correlation with the target distribution D. Since G does not catch up and is made of two modes showing up with equal probability, the optimal solution for D is to have each its w; = a; a;(ui +ug) to maximize the average correlation. Therefore, after iteration 20, learns the second mode and eventually gets this optimal solution (which is mode collapse) at iteration 40. Then, D's update speed starts dropping as we see in Figure 5(a) which helps to catch up and grow its update speed. However, since D already learnt a weighted average of the modes, it can only teach G to learn this average and thus mode collapses as we see in Figure 5(c).
On the other hand, nSGDA ensures that G and D always learn at the same speed, so that G can learn one mode immediately when D learns (such as at iteration 25 in (b) in Figure (5)), which avoids mode collapse.
5 CONCLUSION
Our work offers a complementary view to several works in the min-max optimization literature where the discriminator is much faster than the generator to converge to an equilibrium. Here, instead, we advocate the use of balanced updates to ensure that the GAN performs well.
Our work first step towards understanding how adaptive methods improve the GAN perfor- mance. Our empirical observations and theorems heavily rely on the fact that the batch size is small. However, nSGDA methods seem to not work well for large batch sizes and may not be suitable to some large-scale GANs such as BigGAN (Brock et al., 2018). It would be interesting to understand why adaptive methods are crucial in this case. Another interesting direction would be to improve the training of nSGDA in the batch setting.
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
Naman Agarwal, Rohan Anil, Elad Hazan, Tomer Koren, and Cyril Zhang. Disentangling adaptive gradient methods from learning rates. arXiv preprint arXiv:2002. 1803, 2020.
Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. CoRR, abs/2012.09816, 2020. URL htttt:::// https:/t://aatt http:s://axxvv..rrr./// abs/2012 09816.
Zeyuan Allen-Zhu and Yuanzhi Li. Forward super-resolution: How can gans learn hierarchical generative models for real-world distributions. arXiv preprint arXiv:2106.0261 2021.
Kimon Antonakopoulos, Veronica Belmega, and Panayotis Mertikopoulos. An adaptive mirror- prox method for variational inequalities with singular operators. Advances in Neural Information Processing Systems, 32:8455-844 8465, 2019.
Kimon Antonakopoulos, Veronica Belmega, and Panayotis Mertikopoulos. Adaptive extra- gradient methods for min-max optimization and games. arXiv preprint wrXiv:2010.12/00 2020.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference machine learning, pp. 214-223. PMLR, 2017.
Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets (gans). In International Conference on Machine Learning, pp. 224- 232. PMLR, 2017.
Sanjeev Arora, Andrej Risteski, and Yi Zhang. Do gans learn the distribution? some theory and empirics. In International Conference on Learning Representations, 2018
Francis Bach and Kfir Y Levy. A universal algorithm for variational inequalities adaptive to smooth- ness and noise. Conference on Learning Theory. pp. 164-194. PMLR, 2019.
Yu Bai, Tengyu Ma, and Andrej Risteski. Approximability of discriminators implies diversity in gans. arXiv preprint arXiv: 1806. 10586, 2018
Yogesh Balaji, Mohammadmahdi Sajedi, Neha Mukund Kalibhat, Mucong Ding, Dominik Stöger, Mahdi Soltanolkotabi, and Soheil Feizi. Understanding overparameetrrizatinn in generative ad- versarial networks. arXiv preprint 1xXi::204.0056//0 2021.
Babak Barazandeh, Davoud Ataee Tarzanagh, and George Michailidis. Solving class of non- convex min-max games using adaptive momentum methods. In ICASSP 2021-2021 IEEE In- ternational Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 3625-3629, IEEE, 2021.
David Berthelot, Thomas Schumm, and Luke Metz. Began: Boundary equilibrium generative ad- versarial networks. arXiv preprint arXiv:1703.107117 2017.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1096, 2018
Dami Choi, Christopher J Shallue, Zachary Nado, Jachoon Lee, Chris J Maddison, and George Dahl. On empirical comparisons of optimizers for deep learning. arXiv preprint 1900v:1910.05446, 2019.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelli- gence and statistics, pp. 215-223. JMLR Workshop and Conference Proceedings, 2011.
Jorge Cortés. Finite-time convergent gradient flows with applications to network consensus. Auto- matica, 1291100993300000 2006.
Ashok Cutkosky and Harsh Mehta. Momentum improves normalized sgd. International Confer- on Machine Learning, pp. 2260-2266 PMLR, 2020.
10

---

Under review as a conference paper at ICLR 2022
Constantinos Daskalakis, Andrew Ilyas, Vasilis Syrgkanis, and Haoyang Zeng. Training gans with optimism. arXiv preprint arXtv:1111.00141, 2017.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning reseurch, 12(7), 2011.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Ar- jovsky, and Aaron Courville. Adversarially learned inference. arXiv preprint arXiv:1606.0071 2016.
Soheil Feizi, Farzan Farnia, Tony Ginart, and David Tse. Understanding gans: the lag setting. arXiv preprint arXiv:1710. 1700,09330,,799 2017.
Tanner Fiez and Lillian Ratliff. Gradient descent-ascent provably converges to strict local minmax equilibria with a finite timescale separation. arXiv preprint wiXiv:200..48220 2020.
Tanner Fiez, Benjamin Chasnov, and Lillian Ratliff. Implicit learning dynamics in stackelberg games: Equilibria characterization, convergence analysis, and empirical study, In International Conference Machine Learning, pp- 1133-3144, PMLR, 2020.
AV Gasnikov, PE Dvurechensky, FS Stonyakin, and AA Titov. adaptive proximal method for variational inequalities. Computational Mathematics and Mathematical Physics, 59(5):836-88) 841, 2019.
Gauthier Gidel, Hugo Berard, Gaëtan Vignoud, Pascal Vincent, and Simon Lacoste-Julien. A variational inequality perspective on generative adversarial networks. arXiv preprint arXiv:/802.10S5/ 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans. arXiv preprint arXiv:1704.0003: 2017.
Elad Hazan, Kfir Y Levy, and Shai Shalev-Shwartz. Beyond convexity: Stochastic quasi-convex optimization. arXiv preprint arXhv:1507.02030. 2015.
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Chi Jin, Praneeth Netrapalli, and Michael Jordan. What local optimality in nonconvex-noncconcae minimax optimization? In International Conference on Machine Learning, pp. 4880-4889. PMLR, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Qi Lei, Jason Lee, Alex Dimakis, and Constantinos Daskalakis. Sgd learns one-layer networks in wgans. In International Conference on Machine Learning, pp. 5799-5808. PMLR, 2020.
Kfir Y Levy. The power of normalization: Faster evasion of saddle points. arXiv:1611.04831. 2016.
arXiv preprint
Yuanzhi Li and Zehao Dou. Making method of moments great again?-hov can gans learn distribu- tions. arXiv preprint arXiv:2003.0403: 2020.

---

Under review as a conference paper at ICLR 2022
Tengyuan Liang. How well can generative adversarial networks learn densities: A nonparametric view. arXiv preprint arXiv: 2017.
Mingrui Liu, Youssef Mroueh, Jerret Ross, Wei Zhang, Xiaodong Cui, Payel Das, and Tianbao Yang. Towards better understanding of adaptive gradient algorithms in generative adversarial 2019.
nets. arXiv preprint arXiv:
Yang Liu, Jeremy Bernstein, Markus Meister, and Yisong Yue. Learning by turning: Neural archi- tecture aware optimisation. arXiv preprint arXiv:2102.07227, 2021.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In International conference on machine learning, pp. 3481-3490. PMLR,
2018.
Takeru Miyato, Toshiki Kataoka, Masamori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05990 059570200555..... 2018.
Ryan Murray, Brian Swenson, and Soummya Kar. Revisiting normalized gradient descent: Fast evasion of saddle points. IEEE Transactions on Automatic Control, (4(11))4818-444 2019.
Y.E. Nesterov. Minimization methods for nonsmooth convex and quasiconvex functions. Econ. Mat. Met., 20:519-531, 1984.
Antonio Orvicto, Jonas Kohler, Dario Pavllo, Thomas Hofmann, and Aurelien Lucchi. Vanishing curvature and the power of adaptive methods in randomly initialized deep networks, 2021.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks, arXiv preprint arXiv:1511.06434 2015.
Tijmen Tieleman, Geoffrey Hinton, et al. Lecture 6.5-rmsprop: Divide the gradient by running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26-31, 2012.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv: 777.03888888 2017.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint 1904
2019.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop, 2016.
Jure Zhontar, Florian Knoll, Anuroop Sriram, Tullie Murrell, Zhengnan Huang, Matthew J Muckley, Aaron Defazio, Ruben Stern, Patricia Johnson, Mary Bruno, et al. fastmri: An open dataset and benchmarks for accelerated mri. arXiv preprint vrXiv:181.00880 2018.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In International conference on machine learning, pp. 7354-7363. PMLR, 2019a.
Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models? arXiv preprint arXiv:1912.03194, 2019b.
Pengchuan Zhang, Qiang Liu, Dengyong Zhou, Tao Xu, and Xiaodong He. On the discrimination- generalization tradeoff in gans. arXiv preprint 171 2017.
12

---

Under review as a conference paper at ICLR 2022
Dongruo Zhou, Jinghui Chen, Yuan Cao, Yiqi Tang, Ziyan Yang, and Quanquan Gu. On the convergence of adaptive gradient methods for nonconvex optimization. arXiv preprint arXiv:1808.05677,
2018. 2018.
13


Reviews:
[{"review_id": "gniKIURmJXi", "reviewer": "Reviewer_5J1v", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Carefully adjusted update steps in Reinforcement Learning (actor/critic models) and GAN training are vital to reach convergence. E.g. [1] proposed a control variable to level out discriminator and generator. [2] has proven that under some assumptions competing agents converge to an equilibrium if the learning speeds are carefully adjusted. [3] based on the results of [2] have shown that the results are also valid in the GAN context where the slower agent is the generator and the faster agent the discriminator. To reach convergence a faster leader, critic, discriminator guides a slower follower, generator such that the follower, generator is fast enough to follow but in the same time is not too fast to disturb the leader [4]. My main concern with this paper is, that having normalized gradient updates is probably a good starting point to select optimal update steps but according to the research mentioned above it's more likely normalized gradient updates still need to be scaled (for both agents separately).\nAs the learning speeds of the agents depend not only on the learning rates, e.g. the complexity of the architecture is important as well, one has to be careful to select the correct settings. E.g. in the DCGAN architecture the generator has more parameters and needs a higher learning rate compared to the discriminator to converge cf. Section A5 in [3]. However, in the first experiment the authors train the DCGAN generator with a smaller learning rate which probably leads not to optimal FIDs. I suggest to run additional experiments with higher learning rates for the generator and add them to the results shown in Figure 1. In section 4 i'm not sure if i can see a theoretical proof of theorems 4.1 and 4.2.\n\n[1] Berthelot et al., BEGAN: Boundary Equilibrium Generative Adversarial Networks,  https://arxiv.org/abs/1703.10717 balance\n[2] V. S. Borkar. Stochastic approximation with two time scales. Systems & Control Letters,\n29(5):291–294, 1997\n[3] Heusel et al. GANs trained by a two time-scale update rule converge to a local nash equilibrium. Advances in\nneural information processing systems, 30, 2017.\n[4] M. W. Hirsch. Convergent activation dynamics in continuous time networks. Neural Networks,\n2(5):331–349, 1989.\n[5] Tanner Fiez and Lillian Ratliff. Gradient descent-ascent provably converges to strict local minmax\nequilibria with a finite timescale separation. arXiv preprint arXiv:2009.14820, 2020.", "summary_of_the_paper": "In this manuscript the authors investigate the effect of normalized and unormalized gradient updates on the convergence of GANs. In a first experiment a DCGAN model is trained with Adam and it is shown that the best models evaluated with the FID have balanced learning rates but haven't converged while models trained with extremly unbalanced learning rates converge while having high FIDs i.e. didn't learn the training distribution.\nIn the second experiment it is shown that an adaptive gradient magnitude helps to train a good model compared to an adaptive gradient direction. In a next step using the WGAN-GP model on CIFAR-10, LSUN Churches, STL-10, and CelebA-HQ datasets the optimizers Adam, normalized gradient updates and unnormalized SGDA were compared. The normalized optimizers including Adam outperformed SGDA. Finally it was shown that SGDA is more sensitive to the batch-size. In a final experiment on a small toy dataset it was shown that SGDA suffers from mode collapse while normalized gradient updates guide the generator to learn the modes of the dataset.\n", "summary_of_the_review": "The importance of balanced updates of the generator and discriminator in GAN training is already well known. Using normalized gradient update steps for convergence is plausible, however not enough as shown in [2,3,4,5].  \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZpNbCBCIgVv", "reviewer": "Reviewer_5fFW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. This paper finds that the adaptive magnitude of Adam is critical to the performance of GANs. Personally, I think the analysis of the optimizers for GANs is an interesting topic. \n2. In the theoretical aspect, the analysis explains the success of nSGDA.\n3. The results show that nSGDA achieves a similar performance compared to Adam using WGAN-GP.\n\nWeaknesses:\n1. Although the paper finds that the adaptive magnitude of Adam helps the performance in GANs, the proposed nSGDA is still inferior to Adam. The study does not analyze why Adam is superior to nSGDA. I would expect the study presents how to further improve nSGDA or even presents methods that outperform Adam.\n2. The study uses a ResNet WGAN-GP as the network backbone. I would expect to use more recent GANs, such as BigGAN and StyleGAN.\n3. For the optimizer, the current experiments are not comprehensive enough. I think the proposed optimizer should be evaluated over different network architectures, different resolutions, and different GANs. \n", "summary_of_the_paper": "This paper studies how adaptive methods help performance in GANs. The study empirically finds that SGDA with the same vector norm as Adam reaches similar better performance. Based on this observation, normalized SGDA (nSGDA) is proposed as a simpler alternative to Adam. nSGDA is evaluated on several datasets and the results demonstrate that nSGDA is more stable than SGDA. ", "summary_of_the_review": "Although the topic and findings of this paper are interesting, I think the contributions of the current version are limited, especially for the performance of nSGDA and the experiments. \n\n******************************\nPost Rebuttal Comments:\n\nI thank the authors for their efforts in addressing my concerns. However, I still have the major concern about the performance of the proposed method over commonly used GANs such as BigGAN and StyleGAN. The authors respond that simpler with similar performances is better. However, there is no result to show similar performance over BigGAN or StyleGAN. Thus I keep my original rating. \n\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8nkCw1Sm2rR", "reviewer": "Reviewer_QjHh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents a great study that dissects the success of Adam in GAN training, which covers insightful observations and comprehensive analysis both empirically and theoretically. The findings of this paper, such as \"it is the adaptive magnitude of Adam that matters\" and \"the learning objective of GAN converges does not imply the generator synthesizes high-quality samples\" will definitely encourage more exploration along this direction, helping us better understand the optimization process of similar non-convex non-concave min-max problems.\n \nMinor concerns:\n1. The space is overly adjusted. Related work should be included in a separate section.\n2. I suggest avoiding representing both distribution and distriminator as D, which hurts the readability significantly.\n3. typos, e.g. autorefalg:adaptive?\n4. While the finding of nSGDA is indeed interesting and insightful, it performs comparably with Adam. Can authors discuss the potential advantages of nSGDA over Adam?\n", "summary_of_the_paper": "This paper dissects the success of Adam in GAN training by swap the gradient direction/magnitude with that of SGDA. By doing so, authors found the Adam produces higher quality solutions relative to SGDA in GANs mainly due to its adaptive magnitude and not to its adaptive direction. Inspired by this observation, the combination of direction of SGDA and magnitude of Adam yields the use of normalized SGDA (nSGDA) in GAN training, which consistently competes with Adam.", "summary_of_the_review": "I enjoy reading this paper. The findings are insightful with comprehensive empirical and theoretical analysis.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies how adaptive methods help train GANs to achieve better FID scores. It empirically shows that the adaptive magnitude in ADAM is the reason for ADAM's wide adoption for GAN training. The paper receives three reviews: one ranked the paper "accept, good paper" and two ranked the paper "marginally below the acceptance threshold". The supportive reviewer likes the findings in the paper interesting but does not provide enough explanation on the significance of the findings. On the other hand, the negative reviewers raise several concerns, including the GAN architectures used in the paper are outdated and the achieved performance gain is not major. As the paper focuses on performance instead of convergence, the meta-reviewer feels it would be better to include results on SOTA GAN architectures. The provided rebuttal does not lead to any review score change. Consolidating the review and rebuttal, the meta-reviewer feels the paper needed to be improved to meet the bar and would not recommend its acceptance.
Decision: Reject