Dissecting adaptive methods in GANs
Authors: Samy Jelassi
David Dobre
Arthur Mensch
Yuanzhi Li
Gauthier Gidel
Venue: Submitted to ICLR 2023 (2023)
Keywords: deep learning theory
generative adversarial networks
adaptive methods

Abstract: Adaptive methods are a crucial component widely used for training generative adversarial networks (GANs). While there has been some work to pinpoint the “marginal value of adaptive methods” in standard tasks, it remains unclear why they are still critical for GAN training. In this paper, we formally study how adaptive methods help train GANs; inspired by the grafting method proposed in (Agarwal et al. 2021), we separate the magnitude and direction components of the Adam updates, and graft them to the direction and magnitude of SGDA updates respectively. By considering an update rule with the magnitude of the Adam update and the normalized direction of SGD, we empirically show that the adaptive magnitude of Adam is key for GAN training. This motivates us to have a closer look at the class of normalized stochastic gradient descent ascent (nSGDA) methods in the context of GAN training. We propose a synthetic theoretical framework to compare the performance of nSGDA and SGDA for GAN training with neural networks. We prove that in that setting, GANs trained with nSGDA recover all the modes of the true distribution, whereas the same networks trained with SGDA (and any learning rate configuration) suffer from mode collapse. The critical insight in our analysis is that normalizing the gradients forces the discriminator and generator to be updated at the same pace. We also experimentally show that for several datasets, Adam's performance can be recovered with nSGDA methods.

Under review as a conference paper at ICLR 2023
DISSECTING ADAPTIVE METHODS IN GANS
Anonymous authors
Paper under double-blind review
ABSTRACT
Adaptive methods are a crucial component widely used for training generative adversarial networks (GANs). While there has been some work to pinpoint the "marginal value of adaptive methods" in standard tasks, it remains unclear why they are still critical for GAN training. In this paper, we formally study how adaptive methods help train GANs; inspired by the grafting method proposed in Agarwal et al. (2020), we separate the magnitude and direction components of the Adam updates, and graft them to the direction and magnitude of SGDA updates respectively. By considering an update rule with the magnitude of the Adam update and the normalized direction of SGD, we empirically show that the adaptive magnitude of Adam is key for GAN training. This motivates us to have closer look at the class of normalized stochastic gradient descent ascent (nSGDA) methods in the context of GAN training. We propose synthetic theoretical framework to compare the performance of nSGDA and SGDA for GAN training with neural networks. We prove that in that setting, GANs trained with nSGDA recover all the modes of the true distribution, whereas the same networks trained with SGDA (and any learning rate configuration) suffer from mode collapse. The critical insight in our analysis is that normalizing the gradients forces the discriminator and generator be updated the same pace. We also experimentally show that for several datasets, Adam's 's performance can be recovered with nSGDA methods.
INTRODUCTION
Adaptive algorithms have become key component in training modern neural network architectures in various deep learning tasks. Minimization problems that arise in natural language processing (Vaswani et al., 2017), fMRI (Zbontar et al., 2018), or min-max problems such as generative adversarial networks (GANs) (Goodfellow et al., 2014) almost exclusively use adaptive methods, and it has been empirically observed that Adam (Kingma & Ba, 2014) yields a solution with better generalization than stochastic gradient descent (SGD) in such problems (Choi et al., 2019). Several works have attempted to explain this phenomenon in the minimization setting. Common explanations are that adaptive methods train faster (Zhou et al., 2018), escape flat "saddle-point like plateaus faster (Orvieto et al., 2021), or handle heavy-tailed stochastic gradients better (Zhang al., 2019; Gorbunov et al., 2022). However, much less is known about why adaptive methods are so critical for solving min-max problems such as GANs.
Several previous works attribute this performance to the superior convergence speed of adaptive methods. For instance, Liu al. (2019) show that an adaptive variant of Optimistic Gradient Descent (Daskalakis et al., 2017) converges faster than stochastic gradient descent ascent (SGDA) for a class of non-convex, non-concave min-max problems. However, contrary the minimization setting, convergence to a stationary point is not required to obtain satisfactory GAN performance. Mescheder al. (2018) empirically shows that popular architectures such as Wasserstein GANs (WGANs) (Arjovsky et al., 2017) do not always converge, and yet they produce realistic images. We support this observation with our own experiments in Section Our findings motivate the central question in this paper: what factors of Adam contribute to better quality solutions over SGDA when training GANs?
In this paper, we investigate why GANs trained with adaptive methods outperform those trained using SGDA. Directly analyzing Adam is challenging due to the highly non-linear nature of its gradient oracle and its path-dependers update rule. Inspired by the grafting approach in (Agarwal et al., 2020), disentangle the adaptive magnitude and direction of Adam and show evidence that algorithm using the adaptive magnitude of Adam and the direction of SGDA (which we call Ada-nSGDA) recovers the performance of Adam in GANs. Our contributions are as follows:

---

Under review as a conference paper at ICLR 2023
In Section 2, present the Ada-nSGDA algorithm. We emprically show that the adaptive magnitude in Ada-nSGDA stays within a constant range and does not heavily fluctuate which motivates the focus normalized SGDA (nSGDA) which only contains the direction of SGDA.
In Section 3, we prove that for a synthetic dataset consisting of two modes, a model trained with SGDA suffers from mode collapse (producing only single type of output), while model trained with nSGDA does not. This provides an explanation for why GANs trained with nSGDA outperform those trained with SGDA.
In Section 4, we empirically confirm that nSGDA mostly recovers the performance of Ada-nSGDA when using different GAN architectures on a wide range of datasets.
Our key theoretical insight is that when using SGDA and any step-size configuration, either the generator G' or discriminator D updates much faster than the other. By normalizing the gradients as done in nSGDA, D and G are forced to update at the same speed throughout training. The consequence is that whenever D learns a mode of the distribution, G learns right after, which makes both of them learn all the modes of the distribution separately at the same pace.
1.1 RELATED WORK
Adaptive methods in games optimization. Several works designed adaptive algorithms and analyzed their convergence to show their benefits relative to SGDA e.g. in variational inequality problems, Gasnikov et al. (2019); Antonakopoulos et al. (2019): Bach & Levy (2019); Antonakopoulos et al. (2020); Liu et al. (2019); Barazandeh et al. (2021). Heusel et al. (2017) show that Adam locally converges to a Nash equilibrium in the regime where the step-size of the discriminator is much larger than the one of the generator. Our work differs as we do not focus on the convergence properties of Adam, but rather on the fit of the trained model to the true (and not empirical) data distribution.
Statistical results in GANs. Early works studied whether GANs memorize the training data or actually learn the distribution (Arora et al., 2017: Liang, 2017; Feizi et al., 2017: Zhang et al., 2017; Arora al., 2018; Bai al., 2018; Dumoulin et al., 2016). Some works explained GAN performance through the lens of optimization. Lei et al. (2020); Balaji et al. (2021) show that GANs trained with SGDA converge to global saddle point when the generator one-layer neural network and the discriminator is a specific quadratic/linear function. Our contribution differs as i) we construct setting where SGDA converges to a locally optimal min-max equilibrium but still suffers from mode collapse, and ii) we have more challenging setting since need at least degree-3 discriminator to Iearn the distribution, which is discussed in Section 3.
Normalized gradient descent. Introduced by Nesterov (1984), normalized gradient descent has been widely used in minimization problems. Normalizing the gradient remedies the issue of iterates being stuck in flat regions such spurious local minima saddle points (Hazan al., 2015; Levy, 2016). Normalized gradient descent methods outperforms their non-normalized counterparts in multi- agent coordination (Cortés, 2006) and deep learning tasks (Cutkosky & Mehta, 2020). Our work considers the min-max setting and shows that nSGDA outperforms SGDA as it forces discriminator and generator to update at same rate.
1.2 BACKGROUND
Generative adversarial networks. Given training set sampled from some target distribution D, GAN learns to generate new data from this distribution. The architecture is comprised of two networks: generator that maps points in the latent space D, to samples of the desired distribution, and discriminator which evaluates these samples by comparing them to samples from DD More formally, the generator is mapping Gy: I - ddd and the discriminator is mapping Dw Rd R, where V and W are their corresponding parameter sets. To find the optimal parameters of these two networks, one must solve min-max optimization problem of the form
min max Ex=pes., (og(((n))))l Ez-p, [log(1 Dw(Gr(=))) == J), (GAN)
where Pdata is the distribution of the training set, Pz the latent distribution, Gy the generator and Dw the discriminator. Contrary to minimization problems where convergence to local minimum is required for high generalization, we empirically verify that most of the well well-performing GANs do not converge to stationary point.
2

---

Under review as a conference paper at ICLR 2023
10
10
10a
Gradient ratio
10
10
Gradient
10"
10"
Discriminator (0.41)
10000
Generator (1.38)
10"
20
40
60
80
100
100
200
300
Epochs
FID
(a) Each circle corresponds to a specific step-size configuration 'Io/ 11///-- The best-performing mod- els have step- step-size ratios between 10 and 1, and do not converge. As TjD /no increases, the models perform worse but get closer to an equilibrium.
(b) shows that during training, the gradient ra- tio of a well perfoornrn GAN approximately stays constant to 1. We also display the im- ages produced by the model during training.
Gradient ratio against FID score (a) and number epochs (b) obtained with DCGAN on CIFAR-10. This ratio is equal to ligrad | ||2 + llgrad laal:llllllllee grad (resp. grad are the current and initial gradients of G (resp. D). Note that || -ll2 refers to the sum of all the parameters norm in a network. For all the plots, the models are trained for 100 epochs using batch-size 64. For (b), the results are averaged over 5 seeds.
Figure 1:
|lgrad"!la//llrraa [la//lgrad (2) ll2. where grad
(9) (resp. grad and
Convergence and performance are decorrelated in GANs. We support this observation through the following experiment. We train a DCGAN (Radford et al., 2015) using Adam and set up the step-sizes for G and D as TJD, NG. respectively. Note that D is usually trained faster than G i.e. dn 2 TJ. Figure la displays the GAN convergence measured by the ratio of gradient norms and the GAN's performance measured in FID score (Heusel et al., 2017). We observe that when T/D /DD/Ic is close to 1, the algorithm does not converge, but the model produces high-quality samples. other hand, when 7ID /ng > 1, the model converges an equilibrium: similar statement has been proved by Jin et al. (2020) and Fiez & Ratliff (2020) in the case SGDA. However, the trained GAN produces low-quality solutions at this equilibrium, simply comparing the convergence speed of adaptive methods and SGDA cannot explain the performance obtained with adaptive methods.
the
SGDA and adaptive methods. The most simple algorithm solve the min-max (GAN) is SGDA, which is defined as follows:
1y(++) w + M M W.D.
((++1)
v(0 M (t)
where W.,: M are the first-order momentum gradients as defined in Algorithm While this method has been used in the first GANs (Radford et al., 2015), most modern GANs are trained with adaptive methods such as Adam (Kingma & Ba, 2014). The definition of this algorithm for game optimizations given Algorithm The hyperparameters 31, 32 E [0, 1) control the weighting of the exponential moving average of the first and second-order moments. In many deep-learning tasks, practitioners have found that setting 82 = 0.9 works for most problem settings. Additionally, has been empirically observed that having no momentum (i.c., 31 R 0) is optimal for many popular GAN architectures (Karras et al., 2020; Brock et al., 2018). Therefore, we only consider the case where 31 O.
Optimizers such as Adam (Algorithm 1) are adaptive since they use step-size for each parameter that (!) is different than the magnitude of the gradient gy for that parameter up to some constant (such the global learning rate), and this step-size updates while training the model. There are three components that makes the adaptive update differ from the standard SGDA update: 1) the adaptive normalization
by Igg ||2. 2) the change of direction from g /1gg) ||2 to (1) /I|A (4) ||2 and 3) adaptive scaling by
||A y Il2. In summary, the steps from the standard the adaptive update, are:
Dormalization
(*) gy / 1| change of direction
(*) / l|2 adaptive scaling
(
(2)
x1/1l (2
x|IA 12
The three components are entangled and it remains unclear how they contribute to the superior performance of adaptive methods relative to SGDA in GANs.
3

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Adam (Kingma & Ba, 2014) for games. All operations on vectors are element-wise.
Input: initial points WXO), y(0) step-size schedules ((16)) ,nb)); hyperparameters (2,, Initialize M W.1 M w.2. M and 2 zero.
for = ..... do
Receive stochastic gradients Sw: gy evaluated Wi( and v(e)
(1++1 y1 3. (0) yi + By and M322" B2M (1) (1)
Update for y' E V):
y.2 By
Compute gradient oracles for Y (V. A (++1)_ M //1 M ((111)
(e+1) Update: W = (p(t)) + (0) Aw
p((+1) = 1itt) 16 A (1+1)
(1)
NSGDA AS MODEL TO ANALYZE ADAM IN GANS
In this section, we show that normalized stochastic gradient descent-ascent (nSGDA) is suitable proxy to study the learning dynamics of Adam.
To decouple the normalization, change of direction, and adaptive scaling in Adam, we adopt the grafting approach proposed by Agarwal et al. (2020). At each iteration, we compute stochastic gradients, pass them to two optimizers A1, and make grafted step that combines the magnitude of Aiiss 's step and direction of A2's step. We focus on the optimizer defined by grafting the Adam magnitude onto the SGDA direction, which corresponds to omitting the change of direction in (2):
w(+++)= = -p(t) + ADDIA 12
By
D(++1) = -V()) - 1
(3)
lla +
ligy 12
+
where A AW w are the Adam gradient oracles as in Algorithm and gy Iw the stochastic gradients. We refer to this algorithm Ada-nSGDA (combining the Adam magnitude and SGDA direction). There are two natural implementations for nSDGA. In the layer-wise version, y is single parameter group (typically a layer in a neural network), and the updates are applied to each group. In the global version, yot) contains all of the model's weights.
In Fig. 2a, we see that Ada-nSGDA and Adam appear to have similar learning dynamics in terms of the FID score. Both Adam and Ada-nSGDA significantly outperform SGDA as well as AdaDir, which is the alternate case of (3) where we instead graft the magnitude of the SGDA update to the direction of the Adam update. AdaDir diverged after a single step so we omit it in Fig. 2. These results show that the adaptive scaling and normalization components are sufficient to recover the performance of Adam, suggesting that Ada-nSGDA is a valid proxy for Adam
A natural question that arises is how the total adaptive magnitude varies during training. We empirically investigate this by tracking the tuser-wise adaptive magnitudes of the Adam gradient oracles when training GAN with Ada-nSGDA, and summarize our key findings here (see Section 4 for complete experimental details). We first train a WGAN-GP (Arjovsky et al., 2017) model, and find that the adaptive magnitude is bound within constant range, and that all the layers have approximately the same adaptive magnitude (Fig. 2 (b,c)). This suggests that the adaptive scaling component is constant (in expectation) and motivates the use of nSGDA, corresponding to Ada- nSGDA with constant adaptive scaling factor. We then train WGAN-GP model with nSGDA and we find that nSGDA mostly recovers the FID score obtained by Ada-nSGDA (Fig. 2a).
We also validate this observation for more complicated GAN architectures by repeating this study on StyleGAN2 (Karras et al., 2019). We find that the adaptive magnitudes also vary within a constant range, but each layer has its own constant scaling factor. Thus, training StyleGAN2 with nSGDA and a global normalization fails, but training with nSGDA with a different constant step-size for each layer yields performance that mostly recovers that of Ada-nSGDA (Fig 5). These results suggest that the schedule of the adaptive scaling is not central in the success of Ada-nSGDA in GANs. Instead, adaptive methods are successful because they normalize the gradients for each layer, which allows for more balanced updates between G and as we will show in Section 3. We conduct more experiments in Section 4 and in Appendix A.
4

---

Under review as a conference paper at ICLR 2023
:34
WSANGP- CIFAR1O
Busriminttrr
Oskimizer
24
24
Ads 153DA
12
22
IS0DA
2.0
20
200
-.8
10
-.6
16
102
1.4
123
-4
50
-12
128
-2
10000
200003
30000
40000
7.564
10
-1.0
20
-10
sdep
(a)
(b)
(c)
Figure 2: (a) shows the FID training curve for a WGAN-GP ResNet, averaged over seeds. We see that Ada- nSGDA and nSGDA have very similar performance to Adam for a WGAN-GP. (b, c) displays the fluctuations of Ada-nSGDA adaptive magnitude. We plot the ratio |A ||// || for each of the generator's 's (b) and discriminator's 's (c) layers. At early stages, this ratio barely increases and remains constant after 10 steps.
WHY DOES NSGDA PERFORM BETTER THAN SGDA IN GANS?
In Section 2, we empirically showed that the most important component of the adaptive magnitude is the normalization, and that nSGDA (an algorithm consisting of this component alone) is sufficient to recover most of the performance of Ada-nSGDA (and by extension, Adam). Our goal to construct dataset and model where we can prove that model trained with nSGDA generates samples from the true training distribution while SGDA fails. To this end, we consider a dataset where the underlying distribution consists of two modes, defined vectors U1 Ug2 that are slightly correlated (see Assumption 1) and consider the standard GANs' training objective. We show that a GAN trained with SGDA using any reasonable' step-size configuration suffers from mode collapse (Theorem 3. 3)1 it only outputs samples from a single mode which is a weighted average of in and u2- Conversely, ASGDA-trainee GANs learn the two modes separately (Theorem 3.2).
Notation We set the GAN I-sample loss (4) (X, 2) = log(DW + log(1 - DW (:))).
We denote gy w, as the I-sample stochastic gradient. We use the asymptotic complexity notations when defining the different constants e.g. poly(d) refers to any polynomial in the dimension d, polylog(d) to any polynomial in log(d), and o(1) to a constant < d. We denote x 6 for vectors and b in Rd if there is positive scaling factor > such that laa-blle o(1b6l2))
(t)
3.1 SETTING
In this section, we present the setting to sketch our main results in Theorem 3.1 and Theorem 3.2. We first define the distributions for the training set and latent samples, and specify our GAN model and the algorithms we analyze to solve (GAN). Note that for many assumptions and theorems below, we present informal statements which are sufficient to capture the main insights. The precise statements can be found in Appendix B.
Our synthetic theoretical framework considers bimodal data distribution with two correlated modes:
Assumption 1 (Pdata structure). Let polylog(d)- We assume that the modes are correlated. This means that (ui, u2) = 7 > 0 and the generated data point X is either X U1 or X ug.
Next, we define the latent distribution Ps that Gy samples from and maps Pdata- Each sample from Ps consists of a data-point that is a binary-valued vector z E [0,1)me, where my is the number of neurons in Gy. and has non-zero support, i.e. || 2 1. Although the typical choice of latent distributions in GANs is either Gaussian or uniform, we choose pz to be a binary distribution because it models the weights' distribution of hidden layer of deep generator; Allen-Zhu & Li (2021) argue that the distributions of these hidden layers are sparse, non-negative, and non-pisiyively correlated. We now make the following assumptions on the coefficients of 2:
Assumption 2 (p, structure). Let Pz. We assume that with probability 1 1- o(1), there is only one non-zero entry in z. The probability that the entry i E [ma] is non-zero is Pr[z; = 1] = O (11/m)).
In Assumption 2, the output of Gy is only made of one mode with probability o(1). This avoids summing two of the generator's 's neurons, which may cause mode collapse.
R Reasonable simply means that the learning rates are hounded prevent the training from diverging.
5

---

Under review as a conference paper at ICLR 2023
To learn the target distribution Pdatas linear generator Gy with me neurons and a non-linear neural network with M&D neurons:
ma
mp
3 b
Gv(2) = V2
U
Dw(X) sigmoid a
W X)
(4)
where V |0
mg E Rmaxd 2 E {0, 1m W [uo] W " E Rmoxd and
a. b E R. Intuitively, Gy outputs linear combinations of the modes U,- We choose cubic activation as it is the smallest monomial degree for the discriminator's non-linearity that is sufficient for the generator recover the modes u1, U2.
We now state the SGDA and nSGDA algorithms used to solve the GAN training problem (GAN). For simplicity, we set the batch-size 1. The resultant update rules for SGDA and nSGDA are:
SGDA: at each step > 0, sample X ~ Pdata and ~ Pa and update as
W(++1) With nDgw v(++) = v(t) =\()) nog
(5)
nSGDA: at each step & 0, sample Pdato and ~ Ps and update
W(++1) w)()) + 1jD I
((++1) (()) 1lG
(6)
11a
Compared to the versions of SGDA and Ada-nSGDA that introduced in Section 2, we have the same algorithms except that we set 31 = O and omit in (5) and (6). Note that since there is only one layer in the neural networks nSGDA are actually the same. Lastly, we detail how to set the optimization parameters for SGDA and nSGDA in (5) and (6).
study in this paper, the global-wise and layer-wise versions of
Parametrization 3.1 (Informal). When running SGDA and nSGDA (GAN),
Initialization: ((0) 0, and (!)),
w ( E [mD]), o o [ma]) are initialized with
Gaussian with small variance.
Number of iterations: we SGDA for 1. S To iterations where To is the first iteration such that the algorithm converges to approximate first order local minimum. For nSGDA, we run for T = =0(1/m) iterations.
Step-sizes: For SGDA, TD. 7G E (0, poly(a)) can be arbitrary. For nSGDA, an E (0, poly(a) | and ng is slightly smaller than n)D
1
Over-vverrae over-parametrization:: For SGDA, mDD ma polylog(d) are arbitrarily chosen i.e. mp may be larger than mg or the opposite. For RSGDA, we mp llo(d) and my 2 log(d).
Our theorem holds when running SGDA for any (polynomially) possible number of iterations; after To steps, the gradient becomes inverse polynomially small and SGDA essentially stops updating the parameters. Additionally, or equal step-size for compared to G. Note that our choice of step-sizes for nSGDA is the one used in practice, i.e. dn slightly larger than GJ.
setting allows any step-size configuration for SGDA i.c. larger, smaller,
3.2 MAIN RESULTS
We state our main results the performance of models trained using SGDA (5) and nSGDA (6). We show that nSGDA learns the modes the distribution Pdota while SGDA does not.
Theorem 3.1 (Informal). Consider a training dataset and a latent distribution as described above and let Assumption I and Assumption 2 hold. Let To. GJ. Tp and the initialization be as defined in
i & Dou (2020) show that when using linear or quadratic activations, the generator fool the discriminator by only matching the first and second moments of Pdata-
In the nSGDA algorithm defined in (3), the step-sizes were time-dependent. Here, assume for simplicity that the step-sizes np. nG > 0 are constant.

---

Under review as a conference paper at ICLR 2023
Mode Mode iy
Relatve upo0te
10a
30
Correletion
10"
96
9.4
10-4
10
0.2
0.0
02
02
Iterationns
Iterations
Generator
5% 1/4 V4
Generators neurons
(a) Relative gradients
(b) weight correlation
(c) learned SGDA
(d) learned nSGDA
Figure 3: (a) shows the relative gradient updates for SGDA. D first updates its weights while G does not move
until iteration 20, then C moves its weights. (b) shows the correlation for one neuron of D (with maximal correlation u2 initialization) with the modes u1, ug during the learning process of SGDA. (c, d) shows the correlations of neurons of G with modes when trained with SGDA and nSGDA respectively. This shows that for SGDA (c), the model ultimately learns the weighted average ú1 + ug For nSGDA, we see from (d) that one of the neurons (V4) is highly correlated with u1 and another one (Va) is correlated with u2.
Parametrization 3.1. Let be such that t S Tg. Run SGDA on (GAN) for t iterations with step-sizes 1G. Then, with probability at least 1 o(1), the generator outputs for all z {0, m
(z)
u1 + u2 (z)
if 2 nG
otherwise
where (2) E Id is some vector that is not correlated to any of the modes. Formally, V4 E [2]. uz) = o(1 for all = E {0, 1mm
formal proof can be found in Appendix G. Theorem 3.1 indicates that when training with SGDA and any step-size configuration, the generator either does not learn the modes at all (G (z) (=))
or learns an average of the modes (G (z) 21 + u2). The theorem holds for any time S To which is the iteration where SGDA converges to an approximate first-order locally optimal min-max equilibrium. Conversely, nSGDA succeeds learning the two modes separately:
Theorem 3.2 (Informal). Consider a training dataset and a latent distribution as described above and let Assumption 1 and Assumption hold. Let Tı, JGG, in and the initialization as defined in Purametrization 3.1. Run nSGDA on (GAN) for T2 iterations with step-sizes T1 IDD- Then, the generator learns both modes U1, for E {1,2}.
Pr 2-pp CG (2) x uz] is non-negligible.
(8)
A formal proof can be found in Appendix I. Theorem 3.2 indicates that when we train GAN with nSGDA in the regime where the discriminator updates slightly faster than the generator (as done in practice), the generator successfully learns the distribution containing the direction of both modes.
We implement the setting introduced in Subsection 3.1 and validate Theorem 3.1 and Theorem 3.2 (t)
in Fig. 3. Fig. 3a displays the relative update speed 7/ n|lgy> /l2/ (|]()]) where y corresponds to the parameters of either D or G. Fig. 3b shows the correlation (w! (t) uue//lw"" between one of D's neurons and a mode uy and Fig. 3c the correlation (uf²) ue>//lu/ [1lu) ||2 between G's neurons and up. We discuss the interpretation of these plots the next section.
WHY DOES SGDA SUFFER FROM MODE COLLAPSE AND NSGDA LEARN THE MODES?
We now explain why SGDA suffers from mode collapse, which corresponds to the case where Vi WG. Our explanation relies the interpretation of Figs. 3a, 3b, and 3c, and on the updates around initialization that are defined as followed. There exists E [mp)] such that D's update is
2
10 (1)
= w.
+ DDD
Σ (uu 1=1
(9) un) u
(9)
Thus, the weights of D receive gradients directed by un and ug On the other hand, the weights of G at early stages receive gradients directed by w
(t+1) ~ vi (t) + nG Σ
(o mu 50
(10)
3
7

---

Under review as a conference paper at ICLR 2023
We observe that the learning process in Figs. 3a & 3b has three distinct phases. In the first phase (iterations 1-20), D learns one of the modes (u1 or (2) of Pdota (Fig. 3b) and G barely updates its weights (Fig. 3a). In the second phase (iterations 20-40), D learns the weighted average un + u2 (Fig. 3b) while G starts moving its weights (Fig. 3a). In the final phase (iterations 40+), G learns 01 + U2 (Fig. 3c) from D. In more detail, the learning process is described as follows:
(0) At initialization, w and Vi are small. Assume w.L.ogg. that (w: (0) u2) > (w) u1).
(0)
Phase 1:
Because of the (w, u42 in front of U2 in (9), the parameter wi gradually grows its correlation
with U2 (Fig. 3b) and D's gradient norm thus increases (Fig. 3a). While |/06 < 14j, have that (Fig. 3a).
o U2 (0)
Phase 2: D has learned Ug. Because of the sigmoid in the gradient of wp (that was negligible during Phase 1) and (in, u2) = > w/") now mainly receives updates with direction u2. Since G did not update its weights yet, the min-max problem (GAN) is approximately just a minimization problem with respect D's parameters. Since the optimum of such problem is the weighted average "1 + Ug. wy (0) slowly converges to this optimum. Meanwhile, u/ start to receive some significant signal (Fig. 3a) but mainly learn the direction "1 + U2 (Fig. 3c), because w is aligning with this direction.
The parameters of G only receive gradient directed by in + u2. The norm of its relative updates stay large and D only changes its last layer terms (slope and bias b).
Phase 3:
In contrast to SGDA, nSGDA ensures that G and D always learn at the same speed with the updates:
(wl (+) ,x)xx and ot
En(w)) (4) (8)
2/ 2
(++1) W = =w:
+ +ND
(e+1) = u/ + nG 11 (00( 10 w 1l12
(11)
(2) 12
No matter how large (w, X) is, G still learns at the same speed with D. There is a tight window (iteration 25, Fig. 3b) where only one neuron of D is aligned with u₁. This is when G' can also learn to generate U1 by "'ctching up" to D at that point, which avoids mode collapse.
4 NUMERICAL PERFORMANCE OF NSGDA
In Section 2, we presented the Ada-nSGDA algorithm (3) which corresponds to "grafting" the Adam magnitude onto the SGDA direction. In Section 3, we construct dataset and GAN model where we prove that a GAN trained with nSGDA can generate examples from the true training distribution, while GAN trained with SGDA fails due to mode collapse. We now provide more experiments comparing nSGDA and Ada-nSGDA with Adam on real GANs and datasets.
We train ResNet WGAN with gradient penalty on CIFAR-10 10 (Krizhevsky et al., 2009) and STL-10 (Coates et al., 201 1) with Adam, Ada-nSDGA, SGDA, and nSGDA with fixed learning rate as done in Section 3. We use the default architectures and training parameters specified in Gulrajani et al. (2017) (AGP 10, Tldis 5. learning rate decayed linearly to 0 over 100k steps). We also train a StyleGAN2 model (Karras et al., 2020) on FFHQ (Karras et al., 2019) and LSUN Churches (Yu et al., 2016) (both resized to 128 128 pixels) with Adam, Ada-nSGDA, SGDA, and nSGDA. We use the recommended StyleGAN2 hyperparameter configuration for this resolution (batch size = 32, y 0.1024, map depth 2, channel multiplier 16384). We use the Fréchet Inception distance (FID) (Heusel et al., 2017)(o to quantitatively assess the performance of the model. For each optimizer, we conduct coarse log-space sweep over step sizes and optimize for FID. We train the WGAN-GP models for 2880 thousand images (kimgs) on CIFAR- 10 and STL-10 (45k steps with a batch size of 64), and the StyleGAN2 models for 2600 kimgs on FFHQ and LSUN Churches. We average our results over 5 seeds for the WGAN-GP ResNets, and over 3 seeds for the StyleGAN2 models due to the computational cost associated with training GANs.
WGAN-GP Figures 4a and 4b validates the conclusions WGAN-GP from Section 2. We find that both Ada-nSGDA and nSGDA mostly recover the performance of Adam, with nSGDA obtaining final FID of ~2-3 points lower than Ada-nSGDA. As discussed in Section 2, such performance is possible because the adaptive magnitude stays within a constant range. In contrast, models trained with SGDA consistently perform significantly worse, with final FID scores 4x larger than Adam.
8

---

Under review as a conference paper at ICLR 2023
CPARED
90
SkylRGANG mmo
StyleGAMD LSUN Churches 12B
85
320
73
75
200
30
30
Aughh
Augus
20
20
sonA
HEGDA
Alaunsonal sont
Iean neo esobs sp00
Optimizer
Optimizer
(d) Figure 4: (a, b) are the final FID scores (5 seeds) for a ResNet WGAN-GP model trained for 45k steps on CIFAR- 10 and STL-10 respectively. (c. d) are the final FID scores (3 seeds) for a StyleGAN2 model trained for 2600kimgs on FFHQ and LSUN Churches respectively. We use the same constant layer scaling in (d) for nSGDA as that (c), which was found by tracking the layer-wiss wise adaptive step-sizes.
(a)
(b)
(c)
Dscrim Hator
StyleGAN22 PTHQ 1288128
400
Optimmiat
Adam fete wicin nGGDA
Funrga
200
.00
FLavolght
12-1
500
1000
1500
2000
2500
step
Training Step
(a)
(b)
Figure 5: (a) shows the FID curve for the StyleGAN2 model trained for 2600 kimgs on FFHQ. (b) shows the fluctuations of the Ada-nSGDA adaptive magnitude for each layer over an arbitrary slice of 500 training steps for the Discriminator. The layers grouped by common types, e.g. Conv weights and biases, etc.). We find that although the magnitude for each layer fluctuates, the fluctuations bounded to some fixed range for each layer. We show similar behaviour for the Generator in Appendix.
StyleGAN2 Figures 4c and 4d show the final FID scores when training a StyleGAN2. We find that Ada-nSGDA recovers most of the performance of Adam, but one difference with WGAN-GP that nSGDA does not work if we use the same global learning rate for each layer. As discussed in Section 2, nSGDA with a different (but constant) step-size for each layer does work, and is able to mostly recover Ada-nSGDA's performance (Fig. 5a). To choose the scaling for each layer, we train StyleGAN2 with Ada-nSGDA on FFHQ-128 128, track the layer-wise adaptive magnitudes, and take the mean of these magnitudes over the training run (for each layer). Figure 5b shows that the fluctuations for each layer are bound to constant range, validating step-sizes. Additionally, the same scaling obtained from training FFHQ seems to work for different datasets; we used it to train StyleGAN2 with nSGDA on LSUN Churches- 128 and recovered similar performance to training on this dataset with Ada-nSGDA (Fig. 4d).
assumption of constant
5 CONCLUSION
Our work addresses the question of which mechanisms in adaptive methods are critical for training GANs, and why they outperform non-adaptive methods. We empirically show that Ada-nSGDA, an algorithm composed of the adaptive magnitude of Adam and the direction of SGD, recovers most of the performance of Adam. We further decompose the adaptive magnitude into two components: normalization, and adaptive step-size. We then show that the adaptive step size is roughly constant (bounded fluctuations) for multiple architectures and datasets. This empirically indicates that the normalization component of the adaptive magnitude is the key mechanism of Ada-nSGDA, and motivates the study of nSGDA; we verify that it too recovers the performance of Ada-nSGDA. Having shown that nSGDA is good proxy for a key mechanism for adaptive methods, we then construct setting where we proved that nSGDA -thanks to its balanced updates- recovers the modes of the true distribution while SGDA fails to do it. The key insight from our theoretical analysis is that the ratio of the update of D and G must be close to I during training in order to recover the modes of the distribution. This matches the experimental setting with nSGDA, as we find that global norm of the parameter updates for both D and G are almost equal for optimal choices of learning rates.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Naman Agarwal, Rohan Anil, Elad Hazan, Tomer Koren, and Cyril Zhang. Disentangling adaptive gradient methods from learning rates, arXiv preprint arXiv:2002.11801, 2020.
Zeyuan Allen-ZIn Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. CoRR, abs/2012.09816, 2020. URL abs/2012. 09816.
Zeyuan Allen-Zie -hu and Yuanzhi Li. Forward super-resolution: How can gans learn hierarchical generative models for real-world distributions. arXiv preprini arXiv:2106.02669, 2021.
Kimon Antonakopoulos, Veronica Belmega, and Panayotis Mertikopoulos. An adaptive mirror-prox method for variational inequalities with singular operators. Advances in Neural Information Processing Systems, 32:8455-8465 2019.
Kimon Antonakopoulos, E Veronica Belmega, and Panayotis Mertikopoulos. Adaptive extra-gradient methods for min-max optimization and games. arXiv preprint arXiv:2010. 12/00, 2020.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 214-223. PMLR, 2017.
Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets (gans). In International Conference on Machine Learning, pp. 224-232. PMLR, 2017.
Sanjeev Arora, Andrej Risteski, and Yi Zhang. Do gans learn the distribution? some theory and empirics. In International Conference on Learning Representations, 2018.
Francis Bach and Kfir Y Levy, A universal algorithm for variational inequalities adaptive to smooth- ness and noise. In Conference on Learning Theory, pp. 164-194. PMLR, 2019.
Yu Bai, Tengyu Ma, and Andrej Risteski. Approximability of discriminators implies diversity in gans. arXiv preprint arXiv: 1806. 10586, 2018
Yogesh Balaji, Mohammadmahdi Sajedi, Neha Mukund Kalibhat, Mucong Ding, Dominik Stöger, Mahdi Soltanolkotabi, and Soheil Feizi. Understanding overparameterrizaion generative adver- sarial networks. arXiv preprint arXiv:2104.05605, 2021.
Babak Barazandeh, Davoud Ataee Tarzanagh, and George Michailidis. Solving a class of non-convex min-max games using adaptive momentum methods. In ICASSP 2021 2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 3625-3629 IEEE, 2021.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809. 1096, 2018.
Dami Choi, Christopher Shallue, Zachary Nado, Jaehoon Lee, Chris J Maddison, and George Dahl. On empirical comparisons of optimizers for deep learning. arXiv preprint 4111::191. 2019.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. Proceedings of the fourteerah international conference on artificial intelligence and statistics, pp. 215-223. JMLR Workshop and Conference Proceedings, 2011.
Jorge Cortés. Finite-time convergent gradient flows with applications to network consensus. Auto- matica, 42 19033-2000 2006.
Ashok Cutkosky and Harsh Mehta. Momentum improves normalized sgd. In International Conference Machine Learning, pp. 2260-2268. PMLR, 2020.
Constantinos Daskalakis, Andrew Ilyas, Vasilis Syrgkanis, and Haoyang Zeng. Training gans with optimism. arXiv preprint /r1iv: 1.00111,, 2017.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Arjovsky, and Aaron Courville. Adversarially leamned inference. arXiv preprint arxiv: 1606 00704, 2016.
10

---

Under review as a conference paper at ICLR 2023
Soheil Feizi, Farzan Farnia, Tony Ginart, and David Tse. Understanding gans: the lag setting. arXiv 2017.
preprint 1710v
Tanner Fiez and Lillian Ratliff. Gradient descent-ascent provably converges to strict local minmax equilibria with a finite timescale separation. arXiv preprint (KKi::2009.18820, 2020.
AV Gasnikov, PE Dvurechensky, FS Stonyakin, and AA Titov. adaptive proximal method for variational inequalities. Computational Mathematics and Mathematical Physics, 59(5):836-841, 2019.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural information processing systems, 27, 2014.
Eduard Gorbunov, Marina Danilova, David Dobre, Pavel Dvurechensky, Alexander Gasnikov, and Gauthier Gidel. Clipped stochastic methods for variational inequalities with heavy-tailed noise. 2022.
arXiv preprint
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of wasserstein gans. arXiv preprint (0X,00000400025 2017.
Elad Hazan, Kfir Y Levy, and Shai Shalev-Shwartz. Beyond convexity: Stochastic quasi-convex optimization. arXiv preprint prXiv:1507.02030 2015.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Chi Jin, Praneeth Netrapalli, and Michael Jordan. What is local optimality nonconvex-nonconee minimax optimization? International Conference on Machine Learning, pp. 4880-4889. PMLR, 2020.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1110-8119, 2020.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 141.6994,,, 2014.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Qi Lei, Jason Lee, Alex Dimakis, and Constantinos Daskalakis. Sgd learns one-layer networks in wgans. In International Conference on Machine Learning, pp. 5799-5808. PMLR, 2020.
Kfir Y Levy. The power of normalization: Faster evasion of saddle points. arXiv:1611.04831 2016.
arXiv preprint
Yuanzhi Li and Zehao Dou. Making method of moments great again?- can gans learn distributions. arXiv preprint arXiv:2003.0400 040033 2020.
Tengyuan Liang. How well can generative adversarial networks learn densities: A nonparametric view. arXiv preprint arXiv: 2017.
Mingrui Liu, Youssef Mroueh, Jerret Ross, Wei Zhang, Xiaodong Cui, Payel Das, and Tianbao Yang. Towards better understanding of adaptive gradient algorithms generative adversarial nets. arXiv preprint arXiv: / /940, 2019.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In International conference on machine learning. pp. 3481-3490. PMLR, 2018.
11

---

Under review as a conference paper at ICLR 2023
Y.E. Nesterov. Minimization methods for nonsmooth convex and quasiconvex functions. Econ. Mat.
Met., 20:519-531, 01 1984.
Antonio Orvieto, Jonas Kohler, Dario Pavilo, Thomas Hofmann, and Aurelien Lucchi. Vanishing curvature and the power of adaptive methods in randomly initialized deep networks, 2021.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint
2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop, 2016.
Jure Zbontar, Florian Knoll, Anuroop Sriram, Tullie Murrell, Zhengnan Huang, Matthew Muckley,
Aaron Defazio, Ruben Stern, Patricia Johnson, Mary Bruno, al. fastmri: An open dataset and benchmarks for accelerated mri. arXiv preprint arXiv: 11/////118880 1.8839,, 2018.
Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models? arXiv preprint arXhv:1912.03194. 2019.
Pengchuan Zhang, Qiang Liu, Dengyong Zhou, Tao Xu, and Xiaodong He. On the discrimination- generalization tradeoff in gans. arXiv preprint arXiv:1711.02771 2017.
Dongruo Zhou, Jinghui Chen, Yuan Cao, Yiqi Tang, Ziyan Yang, and Quanquan Gu. On the conver- gence of adaptive gradient methods for nonconvex optimization. arXiv preprint arXiv:1808.05671 2018.
12


Reviews:
[{"review_id": "e2RWLC3HyM", "reviewer": "Reviewer_GLnC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates why there is a huge difference in performance between the Adam optimizer and the SGDA optimizer for training GANs. \nTo do so, it proposes to consider separately, in Adam, the effects of the adaptive magnitude of gradients and those of the direction of the gradients. Therefore, the authors conduct experiments with normalized SGDA (nSGDA) and introduce variants by combining either the direction of nSGDA and the adaptive magnitudes of Adam (Ada-nSGDA), or the direction of Adam and the magnitudes of SGDA (AdaDir).\nThey build a synthetic setting on which they develop proof that nSGDA recovers the target distribution while SGDA cannot, and use this experiment to draw some insight into the training dynamics.\nFinally, they show empirically that only Adam, Ada-nSGDA, and in some cases nSGDA achieve good results, and conclude that the balancing of magnitudes of updates is the key.", "summary_of_the_review": "It was a very interesting read. To this day, the training dynamics of GANs are still obscure and cause problems (for instance on few data, or conversely, to scale to very large datasets or long training time). In that sense, this paper tackles an important question and there is a strong possibility that the findings could in the future be refined to be of practical use.\n\nI also appreciated the inclusion of StyleGAN-2 in the empirical studies. It is important for this kind of study to check if their findings could be at least partially transferred to more modern architectures and training setups.", "strength_and_weaknesses": "Strengths:\n- The paper tries to answer a challenging question and shed lights on some standard practices with GANs. A better understanding the training dynamics of GANs is essential for improving the current state-of-the-art, and this work brings some valuable insights in that regard.\n- The analysis encompasses both theoretical insights and solid empirical evidence, including results with StyleGAN2 that diverge from the theoretical assumptions.\n\nWeaknesses (and questions):\n- To the question \"Why does nSGDA perform better than SGDA in GANs?\" (Section 3) the authors answer by building a scenario where they can prove that nSGDA succeeds while SGDA fails. The demonstration is compelling but not entirely rigorous as it seems likely one could very well build a scenario where the opposite happens.\n- For the StyleGAN-2 plots in Figure 4c and 4d, while the authors mention that \"nSGDA was not able to recover the performance of Adam\", the actual numbers still seem relevant to the discussion. Can the authors provide them and add them to the figure?\n- As mentioned by the authors, StyleGAN-2 is vastly different from the other GANs, which makes the interpretation of the results difficult. It could be insightful to apply the analysis to BigGAN [1] or FastGAN [2] which might be closer to the theoretical framework.\n- The authors suggest multiple times throughout the paper that what is important is that the ratio of the magnitudes stays in range. This should be discussed in a condensed form in the conclusion. Also, could it become an actionable diagnosis tool for monitoring GAN training?\n\n---\n\n[1] Andrew Brock, Jeff Donahue, Karen Simonyan. Large Scale GAN Training for High Fidelity Natural Image Synthesis. ICLR 2019\n\n[2] Bingchen Liu, Yizhe Zhu, Kunpeng Song, Ahmed Elgammal. Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesis. ICLR 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "S09k36F2IK6", "reviewer": "Reviewer_3Jvo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies why and how adaptive learning methods help training of GANs. The authors postulate that the adaptive magnitudes of gradients of Adam is a key. This postulate leads them to propose, with the step-size grafting approach by Agarwal et al., two algorithms, Ada-nSGDA and nSGDA, the former combining the Adam magnitude and SGDA direction, whereas the latter omitting the Adam magnitude from Ada-nSGDA. Theoretical analysis is then done on behaviors of SGDA and nSGDA under an idealistic setting, showing that SGDA fails to learn the modes (Theorem 3.1) and that nSGDA succeeds under reasonable conditions on the learning rates (Theorem 3.2). Numerical experiments with WGAN-GP and StyleGAN2 were conducted to empirically confirm the key observation that the adaptive magnitudes is a key to success of Adam in GAN training.", "summary_of_the_review": "Although this paper provides an interesting working hypothesis on why adaptive learning methods are helpful in training GANs, I feel that the arguments in this paper are not convincing enough, as detailed in the Weaknesses section. I would therefore evaluate this paper as being slightly negative.", "strength_and_weaknesses": "### Strength\n- This paper provides an interesting working hypothesis about why and how adaptive learning methods are useful in training of GANs.\n- Theoretical results were obtained for learning behaviors of SGDA and nSGDA, even though under a quite idealistic setting.\n### Weaknesses\n1. In the last paragraph, the authors claimed that nSGDA, which is explained as \"Ada-nSGDA where we omit the adaptive magnitudes\" (I guess that it is nothing other than SGDA with gradient normalized to be of unit norm), performs as well as Adam. Regarding this claim:\n  - If normalizing the gradients to be of unit norm be sufficient for good performance, how can one justify the key claim in this paper that the adaptive magnitudes are essential for good performance?\n  - The theoretical development in Section 3 is based on SGDA and nSGDA. One can then argue that the theoretical consideration in this paper fails to distinguish between constant-norm gradients (nSGDA) and the adaptive magnitudes (Ada-nSGDA).\n  - The same criticism can also be raised against the discussion on the StyleGAN2 experiments in page 9. It might suggest, as the authors are claiming throughout this paper, importance of the adaptive magnitudes in Ada-nSGDA, but this claim is not well supported by the theoretical argument in this paper.\n2. The discussion is inconsistent in many other places as well. For example:\n  - On page 8, line 12, one sees reference to Fig. 3 (b) while discussing behaviors of nSGDA. Figure 3 (b) shows, however, results for SGDA, not for nSGDA, so that the reference here does not make sense.\n  - On page 8, lines 34 and 44, the authors claim that Figures 4 (a)-(d) show the FID *curves during training*, whereas these figures show the final FID scores.\n  - On page 8, line 46, the authors claimed that nSGDA did not perform as well as Adam. Why did not they show the results in Figure 4 (c), (d) in order to support the claim?\n### Minor points\n- Page 2, line 2: An extra space after the word \"direction\".\n- Page 2, line 13: What \"its counterpart\" refers to is unclear.\n- Page 2, line 46: The domain of the generator $G_{\\mathcal{V}}$ is written as $\\mathbb{R}^k$, but it should be the latent space $\\mathcal{D}_z$.\n- Page 2, line 46: The range of the discriminator $D_{\\mathcal{W}}$ should not be $\\mathbb{R}$ but $(0,1)$.\n- Page 2, line 49: The operator $:=$ in equation (GAN) should be $=:$.\n- Page 2, lines 49, 50: Is $p_{\\mathrm{data}}$ different from $\\mathcal{D}$?\n- Page 3, Figure 1 caption: The definition of gradient ratio should be put in the main text.\n- Page 3, Figure 1 caption: I did not understand what \"all the parameters norm\" means. Should not $\\\\|\\cdot\\\\|_2$ be squared?\n- Page 3, line 11: most simple $\\to$ simplest\n- Page 3, line 17: The first sentence is somehow duplicate of the statement in line 14.\n- Page 7, line 13: (to $\\to$ in) the next section.\n- Page 7, line 17: as follow(ed $\\to$ s).\n- Page 7, line 29: in front of ($u_2$ $\\to$ $u_l$)\n- Page 9, line 1: What does \"the adaptive magnitude the ratio\" mean? Also, $\\\\|\\mathbf{A}_\\mathcal{Y}^{(t)}\\\\|_2$ is itself not a ratio.\n- Page 9, line 25: the same perform(ance)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "LkwII4ocVo", "reviewer": "Reviewer_WMSK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work investigates the Adam optimizer for GAN training. The authors analyze the optimization steps by separating the magnitude and the direction of weight updates. They consider nSGDA, a normalized version of the standard stochastic gradient descent ascent (SGDA), and Ada-nSGDA, a combination of the magnitude of Adam and the direction of nSGDA, as alternatives to analysis Adam. The paper is composed of three parts. \n1. The authors use Ada-nSGDA to show that the adaptive magnitude is essential in optimization empirically.\n2. A theoretical study shows that under some assumptions, nSGDA can perfectly recover the distribution but SGDA can not. Then they analyze the optimization process of GAN empirically.\n3. They empirically confirm that Ada-nSGDA and nSGDA perform similarly to Adam using two GAN architectures and three datasets.", "summary_of_the_review": "This paper works on an interesting problem of figuring out how Adam works for GAN training. The authors conduct rich empirical and theoretical studies. However, those studies are not convincing enough to make general conclusions. The contribution of this work is limited.", "strength_and_weaknesses": "Strength\n- The paper is well-organized and easy to understand.\n- The paper includes abundant analyses, both empirically and theoretically.\n- The code is provided for reproduction.\n\nWeakness\n- The contribution is limited as it is well-known that adaptive mechanisms are useful in SGD-based optimization. Also, the claimed key insight about GAN training does not very beneficial to the community.\n- The claims or conclusions are not well-supported by the empirical and theoretical studies.\n    - Section 2:  As a study of optimization methods, empirical study on only one architecture and dataset is not general enough.\n    - Section 3: While the authors claim adaptive magnitude is essential to Adam, the conclusions about nSGDA, which use neither magnitude nor direction from Adam, seem not really relevant to Adam.\n    - Section 3: The assumptions are so restricted that the conclusion is not general. Does SGDA always mode collapse? Does nSGDA never mode collapse? Does nSGDA always better than SGDA?\n- Some sections are not conclusive.\n    - Section 2: The authors claim that the critical component of Adam is the adaptive magnitude because of the comparison between Ada-nSGDA and SGDA. However, if we look at the comparison between Ada-nSGDA and nSGDA, the adaptive magnitude seems not important.\n    - Section 4: Similar to above, nSGDA works better than Ada-nSGDA.\n- Some minor problems.\n    - Section 1, 2nd paragraph: SGDA not defined yet\n    - Fig 2 (b), (c) the order of generator and discriminator is not consistent with the caption.\n    - Evaluated methods are described repeatedly in the 2nd and 3rd paragraphs in Section 4.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "96xooDHAGl", "reviewer": "Reviewer_Vs2s", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors study the underlying reasons of adaptive methods’ success in GAN training. They decouple the factors of Adam, namely adaptive magnitude and direction, and graft the adaptive magnitude to the normalized direction of SGDA to obtain Ada-nSGDA. By doing so, they manage to compare the eﬀects of magnitude and direction in a systematic way. In order to prove the suitability of normalized directions, they also show the superiority of nSGDA to SGDA in terms of mode collapse. Lastly, they present empirical results to conﬁrm Ada-nSGDA’s compatibility to Adam with regards of GAN’s performance using various GAN architectures and datasets. Taking everything into consideration, they conclude that the adaptive magnitude of Adam is the main reason for the success of adaptive methods in GANs.", "summary_of_the_review": "In this paper, the authors investigate the eﬀects of the adaptive methods’ components which are adaptive magnitude and direction to the GAN performance. As the nature of Adam optimizer makes the disentanglement to these components hard, the authors ﬁrst present nSGDAs which have compatible performance with Adam and that they outperform SGDAs. Once they obtain this suitable method for directions, they graft the adaptive magnitude of Adam to the direction of nSGDA (Ada-nSGDA) so that they can analyze the eﬀects of magnitude and direction of adaptive methods in a systematic way. Even though the idea is valuable and the authors clearly show the outperformance of nSGDAs to SGDAs both theoretically and experimentally, I believe they should show the performance improvements of Ada-nSGDAs over nSGDAs in order to conclude that the underlying reason of adaptive methods’ success is adaptive magnitude instead of direction. Their experimental results seem lacking in this context.\n", "strength_and_weaknesses": "Strengths:\n\n1. They follow a systematic and theoretically sensible approach to show the importance of adaptive magnitude.\n\n\n2. The proofs of the theorems seem plausible.\n\n\n3. Some of their empirical results are compatible to the theory.\n \n Weaknesses:\n\n1. In Figure 4(a) and (b), we should expect to see the superiority of Ada-nSGDA to nSGDA in terms of FID since the main objective of the paper is to show the necessity of the adaptive magnitude. The authors already explained this result is possible because of nSGDA’s competence. If nSGDA is already enough to recover Adam’s performance, how can one elucidate the importance of adaptive magnitude?\n\n2. For StyleGAN2, the authors state that nSGDA cannot recover the performance of Adam. However, they did not show the FID scores of nSGDA in Figure 4(c) and (d). We should be able to visually compare nSGDA with the other three. Besides, if normalized directions are not enough in this case, can we conclude that Ada-nSGDA’s success is the result of adaptive magnitude? The authors did not discuss these results in terms of adaptive magnitude.\n\n3. Even though the objective is to emphasize the importance of adaptive magnitude, the authors put a lot more eﬀort to prove and show the superiority of nSGDA over SGDA and its compatibility to Adam than to compare Ada-nSGDA to nSGDA. As an example, in Appendix A.2 Figure 12, it would be good to see the results of Ada-nSGDA compared with layer-wise and global nSGDAs which would also support the conclusion. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject