On the One-sided Convergence of Adam-type Algorithms in Non-convex Non-concave Min-max Optimization

1 INTRODUCTION . As one of the most popular optimizers in supervised deep learning tasks like natural language processing ( Chowdhury , 2003 ) as well as the main workhorse of generative adversarial network training ( Goodfellow et al. , 2014 ) , Adam-type methods are widely used because of their minimal need for learning rate tuning and their coordinate-wise adaptivity on local geometry . Starting from AdaGrad ( Duchi et al. , 2011 ) , adaptive gradient methods have evolved into a variety of different Adam-type algorithms , such as Adam ( Kingma & Ba , 2015 ) , RMSprop , AMSGrad ( Reddi et al. , 2018 ) and AdaDelta ( Zeiler , 2012 ) . In supervised learning , adaptive gradient methods and Adamtype algorithms play important roles . Especially in the field of NLP ( natural language processing ) , Adam-type algorithms are the goto optimizer . Multiple NLP experiments show that sparse Adam outperforms other non-adaptive algorithms like Stochastic Gradient Descent ( SGD ) not only on the solution performance , but also on both the training and testing error ’ s convergence rates . It ’ s worth mentioned that the most popular pre-training language model BERT ( Devlin et al. , 2018 ) also uses Adam as its optimizer , which shows the power of Adam-type algorithms . Also , Adam-type algorithms are very effective in min-max optimization . As a direct and widely used application of min-max optimization , generative adversarial networks ( GANs ) are notorious for the training difficulty . Training by SGD will easily diverge to nowhere or converge to a limiting cycle , both of which will lead to an ill-performing solution , while Adam optimizer , as the default optimizer for GANs ( Hsieh et al. , 2020 ) , can obtain better performance . The reason why these two optimizers have so much difference in GAN ’ s training has long been an open problem . Traditionally , the training performance of min-max optimization is measured according to its first-order convergence , which means the norm of the gradient , but is it really true in GAN ’ s training ? After training GAN on two relatively simple datasets , MNIST and Fashion-MNIST , we can find that , in a practical training process of GAN , Adam optimizer does not perfectly converge since the norm of discriminator ’ s gradient remains quite high through out the training process . Instead , it only has a one-sided convergence as the norm of generator ’ s gradient actually converges to 0 . This paper thus aims to explain this phenomenon by bridging the gap between theory and practice . On one hand , we understand under which conditions Adam-type optimization algorithms have provable convergence for min-max optimization . Towards this end , a recent work ( Liu et al. , 2020 ) designs two algorithms , Optimistic Stochastic Gradient ( OSG ) and Optimistic AdaGrad ( OAdaGrad ) for solving a class of non-convex non-concave min-max problems and gives theoretical guarantee on their convergence . ( Liu et al. , 2020 ) also proposes an open problem on the convergence proof of Adam-type algorithms , which is solved by this paper . On the other hand , we find that the MVI condition needed for our convergence proof does not practically hold for GANs . Instead , we propose the much milder one-sided MVI condition , which tends to hold practically and under which we provide the theoretical guarantee of the one-sided convergence of Adam-type algorithms . Despite some theoretical guarantee made on the convergence of Adam-type algorithms on convex concave or non-convex concave min-max optimization , in the non-convex non-concave setting which is most general , there is no theoretical guarantee on convergence . Comparatively speaking , proving the convergence of Adam-type algorithms is much more difficult since they use an empirical version of Momentum . Although it has been shown to perform well in practice , it is actually difficult to analyze theoretically . Even in the standard convex setting , proving the convergence of Adam-type algorithms ( Reddi et al. , 2018 ; Zou et al. , 2021 ) is much harder than other adaptive algorithms such as AdaGrad ( Duchi et al. , 2011 ) . Actually , the original version of Adam is known not to converge in convex settings . Therefore , to formally analyze the convergence of Adam-type algorithms in min-max optimization , we also consider a “ theoretically correct ” version of Adam , which is an analog of AMSGrad ( Reddi et al. , 2018 ) . In this paper , there are three main contributions . ( 1 ) We analyze Extra Gradient AMSGrad , which is an Adam-type algorithm used for solving non-convex non-concave min-max optimization problems as well as GAN ’ s training . We prove that , under the assumption of standard MVI condition , the Extra Gradient AMSGrad algorithm provably converges to a ε-stationary point with O ( dε−2 ) complexity in deterministic setting and O ( dε−4 ) complexity in stochastic setting . ( 2 ) Although the standard MVI condition above is a much milder assumption than convexity , we empirically show that MVI condition does not hold for GAN ’ s objective functions in reality . Instead , the one-sided MVI condition proposed by us tends to hold , which is the mildest assumption ever used in all the convergence proofs for min-max optimization . Under the the one-sided MVI condition , we modify the algorithm above by using dual rate decay , and theoretically prove its convergence rate . ( 3 ) We conduct empirical experiments on GAN ’ s training by the Extra Gradient AMSGrad algorithm and the Extra Gradient AMSGrad with dual rate decay analyzed by us . We show that they have much better performance than the Stochastic Gradient Descent Ascent ( SGDA ) algorithm . Also , we empirically verify that our new one-sided MVI condition is indeed satisfied during GAN ’ s training while the previously proposed standard MVI condition is not , which makes the one-sided MVI condition much closer to reality than the standard version . After achieving all these results , we are eventually able to understand the one-sided convergence of Adam-type algorithms in min-max optimization as well as in GAN ’ s training . 2 BACKGROUND AND RELATED WORKS . In this section , we will introduce the background knowledge as well as related works on the following three fields : adaptive gradient methods , min-max optimization , and the convergence properties of multiple algorithms for min-max optimization problems . 2.1 ADAPTIVE GRADIENT METHODS AND ADAM-TYPE METHODS . We consider the simplest 1-dimensional unconstrained minimization problem : min x∈D⊆R f ( x ) . where f : D → R is a continuously differentiable function . As one of the most dominant algorithms on the optimization problem above , Stochastic Gradient Descent ( SGD ) was originally proposed by ( Goodfellow et al. , 2016 ) , which has been both empirically and theoretically proved effective , especially when facing large datasets and complicated models . To further improve the performance of SGD , several adaptive variants of SGD have been proposed , such as RMSprop , Adam ( Kingma & Ba , 2015 ) , AdaGrad ( Duchi et al. , 2011 ) , AMSGrad ( Reddi et al. , 2018 ) and AdaDelta ( Zeiler , 2012 ) . Distinguished from the vanilla gradient descent or its stochastic version SGD , adaptive gradient methods use a coordinate-wise scaling of the updating direction and each iteration relies on the history information of past gradients . In AdaGrad , we use arithmetic average when adopting history gradient information of each iteration while in Adam , RMSprop etc. , we use exponential moving average instead because its believed that the more current gradient information is more important . Although adaptive gradient methods and momentum based methods are two different routes on optimization , they are combined perfectly in Adam . Now we introduce the family of adaptive gradient methods and Adam-type , and all of them have the following form : mt+1 = ht∇f ( xt ) + rt ·mt , vt+1 = pt ( ∇f ( xt ) ) 2 + qt · vt xt+1 = xt − λt · mt+1√ vt+1 + ε . [ Adaptive ] Here , f is the objective function to minimize . h , r , p , q are scalars depending on t , λt is the learning rate of the t-th iteration and ε > 0 is a small constant used to protect the denominator from being close to 0 . From the formula above , we see that the momentummt is the weighted sum of the past gradients and vt is the weighted sum of the past squared gradients . When h = 1 , r = 0 , mt+1 = ∇f ( xt ) is just the current gradient . We start with the original Adam . vt+1 = αtvt + ( 1− αt ) ( ∇f ( xt ) ) 2 , mt+1 = βtmt + ( 1− βt ) ∇f ( xt ) xt+1 = xt − λ · mt+1√ vt+1 + ε . [ Adam ] As we can see , Adam is a combination of adaptive gradient method and momentum method . Here , the momentum term is empirical , meaning that it does not coincide with acceleration techniques that are theoretically sound , which creates extra difficult for the analysis . In Adam , we have ht + rt = pt + qt = 1 . When the αt = α , βt = β remains constant , there is a bias correction step where vt+1 ← vt+11−αt and mt+1 ← mt+1 1−βt . However , we may practically ignore this bias correction step since 11−αt and 1 1−βt rapidly approach to 1 . As one of the variants of Adam , AMSGrad has the following formulation : v̂t+1 = αtvt + ( 1− αt ) ( ∇f ( xt ) ) 2 , vt+1 = max ( vt , v̂t+1 ) mt+1 = βtvt + ( 1− βt ) ∇f ( xt ) , xt+1 = xt − λ · mt+1√ vt+1 + ε . [ AMSGrad ] As we can see , their difference is that the velocity term vt keeps increasing in AMSGrad . After showing the details of these traditional adaptive gradient methods and Adam-type methods , we introduce their convergence properties as well as their further variants . Reddi et al . ( 2018 ) shows that Adam does not converge in some settings where large gradient information is rarely encountered and it will die out quickly because of the “ short memory ” property of the exponential moving average . However , under some conditions , the convergence proofs of adaptive gradient methods have been obtained . Basu et al . ( 2018 ) proved the convergence rate of RMSprop and Adam when using deterministic gradients instead of stochastic gradients . Li & Orabona ( 2018 ) analyzed the convergence rate of AdaGrad under both convex and non-convex settings . All the papers above provide theoretical guarantee for the convergence of different types of adaptive gradient descent . After that , Chen et al . ( 2019 ) extends Adam to a broader class of Adam-type algorithms and provides its convergence analysis for non-convex optimization problems . In order to combine the fast convergence of adaptive methods and better generalization with momentum based methods , a number of new algorithms are proposed , such as SC-AdaGrad / SC-RMSprop ( Mukkamala & Hein , 2017 ) , AdamW ( Loshchilov & Hutter , 2019 ) , AdaBound ( Luo et al. , 2019 ) etc .. 2.2 MIN-MAX OPTIMIZATION . In the min-max optimization problem ( or saddle point problem ) , we have to solve : min x∈X max y∈Y φ ( x , y ) , [ SP ] where X ⊆ Rn1 , Y ⊆ Rn2 , and φ : X × Y → R is the objective function . When φ is convex on x and concave on y , we call it a convex-concave min-max optimization . Otherwise , it ’ s a more general non-convex non-concave min-max optimization . For the brevity , we denote z = ( x , y ) and Z = X × Y ⊆ Rn1+n2 . We also introduce our gradient vector field : V ( z ) = ( −∇xφ ( x , y ) , ∇yφ ( x , y ) ) , which are the update directions on both sides . The goal of [ SP ] is to find a tuple z∗ = ( x∗ , y∗ ) such that φ ( x∗ , y ) 6 φ ( x∗ , y∗ ) 6 φ ( x , y∗ ) holds for ∀x ∈ X , y ∈ Y , which is called the solution of [ SP ] . If the inequality above only holds in the local neighbourhood of z∗ , then z∗ can only be called a local solution . Notice that the necessary condition of being a solution ( or even a local solution ) is to be a stationary point of φ , which means V ( z∗ ) = 0 . Furthermore , if V is C1 , any local solution of [ SP ] must be stable , which means ∇2xxφ ( x∗ , y∗ ) 0 and ∇2yyφ ( x∗ , y∗ ) 0 . Next , we will introduce several commonly-used algorithms which are designed to solve [ SP ] . Stochastic Gradient Descent Ascent ( SGDA ) This is a simple extension of Stochastic Gradient Descent ( SGD ) algorithm for minimization problems ( Johnsen , 1959 ) . In the t-th iteration : zt+1 = zt + γt · V ( zt ; ωt ) , [ SGDA ] where ω1 , ω2 , · · · are the independent and identically distributed sequence of noises . V ( z , ω ) can be treated as a query to the stochastic first-order oracle ( SFO ) . In each iteration of SGDA , we need to query SFO once . Notice that , we simultaneously update x , y in each iteration of SGDA . Therefore , if we alternate the updates of x and y , we obtain a variant of SGDA , which is named as the alternating stochastic gradient descent ascent ( AltSGDA ) algorithm . Different from original SGDA , we have to make two queries to SFO in each iteration . One for zt = ( xt , yt ) , and the other for the intermediate step ( xt+1 , yt ) . Since original SGDA is not going to work even in the convex-concave setting ( such as minx maxy f ( x , y ) = xy ) , so researchers propose the following “ theoretically correct modification ” . Stochastic Extra-gradient ( SEG ) This is a different algorithm with the above SGDA , and it is originally proposed for solving the convex-concave setting of min-max optimization problems by Korpelevich ( 1976 ) . Given zt as a base , we take a virtual gradient descent ascent step and obtain a z̃t , which can be treated as the shadow of zt . Then we use the gradient at z′t as the update direction of zt . This process can be described as : z′t = zt + γt · V ( zt ; ω ( 1 ) t ) zt+1 = zt + γt · V ( z′t ; ω ( 2 ) t ) . [ SEG ] In each iteration , we need to make two queries to the SFO . One for the base zt and the other for the shadow z′t . However , in the first step of [ SEG ] , we can use the gradient at the previous shadow z ′ t−1 so that we only have to make only one query in each iteration and remember the query ’ s result of the previous step . This algorithm is called Optimistic Gradient or Popov ’ s Extra-gradient ( Popov , 1980 ) which can be described as : z′t = zt + γt · V ( z′t−1 ; ωt−1 ) zt+1 = zt + γt · V ( z′t ; ωt ) . [ OG ] As a widely used algorithm , it has been applied in multiple works ( Daskalakis et al. , 2018 ; Mertikopoulos et al. , 2019 ) . Under some mild assumptions , convergence rates are proved by many theoretical works and we will summarize them in the next section .

Summary: This manuscript developed several algorithms (e.g., AMSGrad-EG, AMSGrad-EG-DRD) for nonconvex-nonconcave min-max optimization. The convergence result of AMSGrad-EG-DRD is shown under the one-sided MVI condition. Polynomial-time complexity results are established. Some toy experiments are conducted for GAN on MNIST and fashion-MNIST datasets.