Adam is no better than normalized SGD:  Dissecting how adaptivity improves GAN performance

1 INTRODUCTION . It is commonly accepted that adaptive algorithms are required to train modern neural network architectures in various deep learning tasks . This includes minimization problems that arise in natural language processing ( Vaswani et al. , 2017 ) and fMRI ( Zbontar et al. , 2018 ) or min-max problems such as generative adversarial network ( GAN ) training ( Goodfellow et al. , 2014 ) . Indeed , it has been empirically observed that Adam ( Kingma & Ba , 2014 ) yields a solution with better generalization than stochastic gradient descent ( SGD ) in these problems ( Choi et al. , 2019 ) . Several works have attempted to explain this phenomenon in the minimization case . Common explanations are that adaptive methods train faster ( Zhou et al. , 2018 ) , escape faster very flat saddle-point like plateaus ( Orvieto et al. , 2021 ) or deal better with heavy-tailed stochastic gradients ( Zhang et al. , 2019b ) . However , much less is known regarding min-max problems such as GANs . In this paper , we investigate why GANs trained with adaptive methods outperform those trained using stochastic gradient descent ascent with momentum ( SGDA ) . Some prior works attribute this outperformance to the superior convergence speed of adaptive methods . For instance , Liu et al . ( 2019 ) show that a variant of Optimistic Gradient Descent ( Daskalakis et al. , 2017 ) converges faster than SGDA for a class of non-convex non-concave min-max problems . However , contrary to the minimization setting , convergence to a stationary point is not guaranteed and not even a requirement to ensure a satisfactory GAN performance . Indeed , Mescheder et al . ( 2018 ) empirically shows that popular architectures such as Wasserstein GANs ( WGANs ) ( Arjovsky et al. , 2017 ) do not always converge , and yet produce realistic images . We support this observation through the following experiment . We train a DCGAN ( Radford et al. , 2015 ) using Adam –the most popular adaptive method– and set up the generator ( G ) and discriminator ( D ) step-sizes respectively as ηD , ηG . Note that D is usually trained faster than G i.e . ηD ≥ ηG . Figure 1 ( a ) displays the GAN convergence measured by the ratio of gradient norms , and the GAN ’ s performance measured in FID score ( Heusel et al. , 2017 ) . We observe that when ηD/ηG is close to 1 , the algorithm does not converge and yet , the model produces high-quality solutions . On the other hand , when ηD/ηG 1 , the model converges to an equilibrium –a similar statement has been proved by Jin et al . ( 2020 ) and Fiez & Ratliff ( 2020 ) in the case of SGDA . However , the GAN produces low-quality solutions at this equilibrium . Thus , simply comparing the convergence speed of adaptive methods and SGDA can not explain the GAN ’ s performance obtained with adaptive methods . This observation motivates the central question in this paper : What factors explain that Adam produces better quality solutions than SGDA when training GANs ? To address this question , we dissect Adam following the approach by Agarwal et al . ( 2020 ) . They frame a generic optimizer ’ s update as W ( t+1 ) = W ( t ) − ηa ( t ) G ( t ) , where W ( t ) ∈ Rd is the iterate , G ( t ) ∈ Rd such that ‖G ( t ) ‖2 = 1 is the optimizer ’ s direction and a ( t ) ≥ 0 is the optimizer ’ s magnitude . Therefore , a first step in our paper is to understand whether Adam outperforms SGDA maintly due to its direction or to its magnitude . As detailed in Section 2 , we train a GAN using i ) AdaLR , an algorithm that updates in the direction of SGDA but with the magnitude of Adam ii ) AdaDir which uses the direction of Adam but the magnitude of SGDA . We empirically show that not only , AdaLR significantly outperforms AdaDir , SGDA , and Adam itself . This observation encourages us to conclude that : Adam produces higher quality solutions relative to SGDA in GANs mainly due to its adaptive magnitude and not to its adaptive direction . In Section 2 , we empirically analyze the adaptive magnitude of AdaLR and observe that it stays approximately constant throughout training . This observation eventually encourages the study of AdaLR with a constant step-size . Such algorithm actually corresponds to normalized SGDA ( nSGDA ) .Compared to SGDA , nSGDA has the same direction but differs in magnitude since we divide the gradient by its norm . Intuitively , this normalization forces D and G to be updated by vectors with constant magnitudes no matter how different the norms of D ’ s and G ’ s gradients are . Motivated by the aforementioned observations , this paper studies the performance of GANs trained with nSGDA . We believe that this is a first step to formally understand the role of adaptive methods in GANs . Our contributions are divided as follows : – In Section 3 , we experimentally confirm that nSDGA consistently competes with Adam and outperforms SGDA when using different GAN architectures on a wide range of datasets . – In Section 4 , we provide a theoretical explanation on why GANs trained with nSGDA outperform those trained with SGDA . More precisely , we devise a data generation problem where the target distribution D is made of multiple modes . The model trained with nSGDA provably recovers all the modes in the target distribution while the SGDA ’ s one fails to do it under any step-size configuration : We prove that even when SGDA converges to a locally optimal min-max equilibrium , the model still suffers from mode collapsing and fails to learn recover the modes separately . The key insight of our theoretical analysis is that no matter how the step-sizes are prescribed , D and G necessarily update at very different speeds when we use SGDA . Therefore , i ) either D updates its weights too fast , thus learns a weighted average of the modes of D. This makes G learn this weighted average of modes ii ) or D does not update its weights fast enough and thus , G aligns its weights with those of D. This forces D to converge to a locally optimal min-max equilibrium that classifies any instance as ” fake ” . On the other hand , by normalizing the gradients as done in SGDA , we force D and G to update at the same speed throughout training . Thus , whenever D learns a mode of the distribution , G learns it right after , which makes both of them learn all the modes of the distribution separately . Our paper advocates for the use of balanced updates in GAN training i.e . the ratio of D vs G updates should remain close to constant . To our knowledge , we are the first to theoretically show the importance of these balanced updates . This insight contrasts with the related work that analyzes GANs , and more generally zero-sum differentiable games , in the regime where D is updated much faster than G i.e . ηD/ηG 1 ( Fiez & Ratliff , 2020 ; Jin et al. , 2020 ; Fiez et al. , 2020 ) . RELATED WORK Adaptive methods in games optimization . Several works designed adaptive algorithms and analyzed their convergence to show their benefits relative to SGDA . For variational inequality problems , Gasnikov et al . ( 2019 ) ; Antonakopoulos et al . ( 2019 ) ; Bach & Levy ( 2019 ) ; Antonakopoulos et al . ( 2020 ) propose adaptive algorithms that reach optimal convergence rates under regularity assumptions . For a class of non-convex non-concave min-max problems , Liu et al . ( 2019 ) ; Barazandeh et al . ( 2021 ) design algorithms that converge faster than SGDA . On the other hand , Heusel et al . ( 2017 ) show that Adam locally converge to a Nash equilibrium in the regime where the step-size of the discriminator is much larger than the one of the generator . Our work differs from these papers as we analyze Adam and do not focus on the convergence properties but rather on the fit of the trained model on the true ( and not empirical ) data distribution . Besides , contrary to some of the aforementioned papers , our work is not in the two-time scale learning-rates regime which do not correspond to what is mostly done in practice . Importance of balanced updates in GANs . The importance of balanced updates for GANs has been noticed in the literature . The most popular GAN architectures ( Radford et al. , 2015 ; Arjovsky et al. , 2017 ; Brock et al. , 2018 ) set the step-sizes such that ηD/ηG is constant . On the other hand , Berthelot et al . ( 2017 ) introduced a control variable to ensure that the updates between G and D are at the same speed . In this work , we do not modify the GAN objective loss to enforce balancedness . Instead , we empirically and theoretically investigates how Adam and nSGDA enforce balanced updates . Statistical results in GANs . Early works investigate whether GANs memorize the training data or actually learn the distribution ( Arora et al. , 2017 ; 2018 ; Dumoulin et al. , 2016 ) . Zhang et al . ( 2017 ) ; Bai et al . ( 2018 ) then show that for specific GANs , the model learn some distributions with nonexponential sample complexity ( Liang , 2017 ; Feizi et al. , 2017 ) . Recently , Li & Dou ( 2020 ) ; AllenZhu & Li ( 2021 ) further characterized the distributions learned by the generator . On the other hand , some works attempted to explain GAN performance through the optimization lens . Lei et al . ( 2020 ) ; Balaji et al . ( 2021 ) show that GAN models trained with SGDA converge to a global saddle point when the generator is one-layer neural network and the discriminator is a specific quadratic/linear function . Our contribution significantly differs from these two works as i ) we construct a setting where SGDA converges to a locally optimal min-max equilibrium and yet suffer from mode collapse . Conversely , nSGDA does not necessarily converge and yet , recovers the true distribution ii ) our setting is more challenging since we need at least a degree-3 discriminator to learn the distribution – see Section 4 for a justification . Normalized gradient descent . Introduced by Nesterov ( 1984 ) , normalized gradient descent has been widely used in the minimization setting . Indeed , it has been observed that normalizing out the gradient improves the ’ slow crawling ’ problem of gradient descent and avoids the iterates to be stuck in flat regions – such as spurious local minima or saddle points – ( Hazan et al. , 2015 ; Levy , 2016 ; Murray et al. , 2019 ) . Normalized gradient descent or its variants outperform the non-normalized counterparts in multi-agent coordination ( Cortés , 2006 ) and deep learning tasks ( You et al. , 2017 ; 2019 ; Cutkosky & Mehta , 2020 ; Liu et al. , 2021 ) . Our work rather considers the min-max setting and shows that nSGDA performs better than SGDA as it forces the discriminator and generator to update at the same rate .

Summary: In this manuscript the authors investigate the effect of normalized and unormalized gradient updates on the convergence of GANs. In a first experiment a DCGAN model is trained with Adam and it is shown that the best models evaluated with the FID have balanced learning rates but haven't converged while models trained with extremly unbalanced learning rates converge while having high FIDs i.e. didn't learn the training distribution. In the second experiment it is shown that an adaptive gradient magnitude helps to train a good model compared to an adaptive gradient direction. In a next step using the WGAN-GP model on CIFAR-10, LSUN Churches, STL-10, and CelebA-HQ datasets the optimizers Adam, normalized gradient updates and unnormalized SGDA were compared. The normalized optimizers including Adam outperformed SGDA. Finally it was shown that SGDA is more sensitive to the batch-size. In a final experiment on a small toy dataset it was shown that SGDA suffers from mode collapse while normalized gradient updates guide the generator to learn the modes of the dataset.