Towards Better Understanding of Adaptive Gradient Algorithms in Generative Adversarial Nets

( − 2 1−α ) , where α characterizes the growth rate of the cumulative stochastic gradient and 0 ≤ α ≤ 1/2 . To the best of our knowledge , this is the first work for establishing adaptive complexity in nonconvex non-concave min-max optimization . Empirically , our experiments show that indeed adaptive gradient algorithms outperform their non-adaptive counterparts in GAN training . Moreover , this observation can be explained by the slow growth rate of the cumulative stochastic gradient , as observed empirically . 1 INTRODUCTION . Adaptive gradient algorithms ( Duchi et al. , 2011 ; Tieleman & Hinton , 2012 ; Kingma & Ba , 2014 ; Reddi et al. , 2019 ) are very popular in training deep neural networks due to their computational efficiency and minimal need for hyper-parameter tuning ( Kingma & Ba , 2014 ) . For example , Adagrad ( Duchi et al. , 2011 ) automatically adjusts the learning rate for each dimension of the model parameter according to the information of history gradients , while its computational cost is almost the same as Stochastic Gradient Descent ( SGD ) . However , in supervised deep learning ( for example , image classification tasks using a deep convolutional neural network ) , there is not enough evidence showing that adaptive gradient methods converge faster than its non-adaptive counterpart ( i.e. , SGD ) on benchmark datasets . For example , it is argued in ( Wilson et al. , 2017 ) that adaptive gradient methods often find a solution with worse performance than SGD . Specifically , Wilson et al . ( 2017 ) observed that Adagrad has slower convergence than SGD in terms of both training and testing error , while using VGG ( Simonyan & Zisserman , 2014 ) on CIFAR10 data . GANs ( Goodfellow et al. , 2014 ) are a popular class of generative models . In a nutshell , they consist of a generator and a discriminator , both of which are defined by deep neural networks . The generator and the discriminator are trained under an adversarial cost , corresponding to a non-convex non-concave min-max problem . GANs are known to be notoriously difficult to train . In practice , Adam ( Kingma & Ba , 2014 ) is the defacto optimizer used for GAN training . The common optimization strategy is to alternatively update the discriminator and the generator ( Arjovsky et al. , ∗Correspondence to mingrui-liu @ uiowa.edu 2017 ; Gulrajani et al. , 2017 ) . Using Adam is important in GAN training , since replacing it with non-adaptive methods ( e.g . SGD ) would significantly deteriorate the performance . This paper studies and attempts to answer the following question : Why do adaptive gradient methods outperform their non-adaptive counterparts in GAN training ? We analyze a variant of Optimistic Stochastic Gradient ( OSG ) in ( Daskalakis & Panageas , 2018 ) and propose an adaptive variant named Optimistic Adagrad ( OAdagrad ) for solving a class of nonconvex non-concave min-max problems . Both of them are shown to enjoy state-of-the-art complexities . We further prove that the convergence rate of OAdagrad to an -first-order stationary point depends on the growth rate of the cumulative stochastic gradient . In our experiments , we observed an interesting phenomenon while using adaptive gradient methods for training GANs : the cumulative stochastic gradient grows at a slow rate . This observation is in line with the prediction of our theory suggesting improved convergence rate for OAdagrad in GAN training , when the growth rate of the cumulative stochastic gradient is slow . Since GAN is a min-max optimization problem in nature , our problem of interest is to solve the following stochastic optimization problem : min u∈U max v∈V F ( u , v ) : = Eξ∼D [ f ( u , v ; ξ ) ] , ( 1 ) where U , V are closed and convex sets , F ( u , v ) is possibly non-convex in u and non-concave in v. ξ is a random variable following an unknown distribution D. In GAN training , u , v represent the parameters of generator and discriminator respectively . The ideal goal for solving ( 1 ) is to find a saddle point ( u∗ , v∗ ) ∈ U × V such that F ( u∗ , v ) ≤ F ( u∗ , v∗ ) ≤ F ( u , v∗ ) for ∀u ∈ U , ∀v ∈ V . To achieve this goal , the typical assumption usually made is that the objective function is convexconcave . When F ( u , v ) is convex in u and concave in v , non-asymptotic guarantee in terms of the duality gap is well established by a series of work ( Nemirovski & Yudin , 1978 ; Nemirovski , 2004 ; Nesterov , 2007 ; Nemirovski et al. , 2009 ; Juditsky et al. , 2011 ) . However , when F ( u , v ) is non-convex in u and non-concave in v , finding the saddle point is NP-hard in general . Instead , we focus on finding the first-order stationary point provided that the objective function is smooth . I.e . we aim to find ( u , v ) ∈ U × V such that ∇uF ( u , v ) = 0 , ∇vF ( u , v ) = 0 . Note that this is a necessary condition for finding the ( local ) saddle point . Related Work . Several works designed iterative first-order deterministic ( Dang & Lan , 2015 ) and stochastic ( Iusem et al. , 2017 ; Lin et al. , 2018 ) algorithms for achieving the -first-order stationary point with non-asymptotic guarantee . The goal is to find x such that ‖T ( x ) ‖ ≤ or E [ ‖T ( x ) ‖ ] ≤ , where the first-order oracle is defined as T ( x ) = [ ∇uF ( u , v ) , −∇vF ( u , v ) ] > with x = ( u , v ) and the first-order stochastic oracle is the noisy observation of T , i.e . T ( x ; ξ ) = [ ∇uF ( u , v ; ξ ) , −∇vF ( u , v ; ξ ) ] > . For instance , Dang & Lan ( 2015 ) focuses on the deterministic setting . On the other hand , ( Iusem et al. , 2017 ) develops a stochastic extra-gradient algorithm that enjoys O ( −4 ) iteration complexity . The extra-gradient method requires two stochastic first-order oracles in one iteration , which can be computationally expensive in deep learning applications such as GANs . The inexact proximal point method developed in ( Lin et al. , 2018 ) has iteration complexity O ( −6 ) for finding an -first-order stationary point 1 . To avoid the cost of an additional oracle call in extragradient step , several studies ( Chiang et al. , 2012 ; Rakhlin & Sridharan , 2013 ; Daskalakis et al. , 2017 ; Gidel et al. , 2018 ; Xu et al. , 2019 ) proposed single-call variants of the extragradient algorithm . Some of them focus on the convex setting ( e.g . ( Chiang et al. , 2012 ; Rakhlin & Sridharan , 2013 ) ) , while others focus on the non-convex setting ( Xu et al. , 2019 ) . The closest to our work is the work by ( Daskalakis et al. , 2017 ; Gidel et al. , 2018 ) , where the min-max setting and GAN training are considered . However , the convergence of those algorithms is only shown for a class of bilinear problems in ( Daskalakis et al. , 2017 ) and for monotone variational inequalities in ( Gidel et al. , 2018 ) . Hence a big gap remains between the specific settings studied in ( Daskalakis et al. , 2017 ; Gidel et al. , 2018 ) and more general non-convex 1The result in ( Lin et al. , 2018 ) assumes the first-order oracle T is a weakly-monotone operator , which is milder than the Lipschitz-continuity assumption as assumed in Iusem et al . ( 2017 ) . However , simply applying the Lipschitz-continuity condition in their proof does not change their iteration complexity . non-concave min-max problems . Table 1 provides a complete overview of our results and existing results . It is hard to give justice to the large body of work on min-max optimization , so we refer the interested reader to Appendix B that gives a comprehensive survey of related previous methods that are not covered in this Table . Our main goal is to design stochastic first-order algorithms with low iteration complexity , low periteration cost and suitable for a general class of non-convex non-concave min-max problems . The main tool we use in our analysis is variational inequality . Let T : Rd 7→ Rd be an operator and X ⊂ Rd is a closed convex set . The Stampacchia Variational Inequality ( SVI ) problem ( Hartman & Stampacchia , 1966 ) is defined by the operator T and X and denoted by SVI ( T , X ) . It consists of finding x∗ ∈ X such that 〈T ( x∗ ) , x− x∗〉 ≥ 0 for ∀x ∈ X . A similar one is Minty Variational Inequality ( MVI ) problem ( Minty et al. , 1962 ) denoted by MVI ( T , X ) , which consists of finding x∗ such that 〈T ( x ) , x− x∗〉 ≥ 0 for ∀x ∈ X . Min-max optimization is closely related to variational inequalities . The corresponding SVI and MVI for the min-max problem are defined through T ( x ) = [ ∇uF ( u , v ) , −∇vF ( u , v ) ] > with x = ( u , v ) . Our main contributions are summarized as follows : • Following ( Daskalakis et al. , 2017 ) , we extend optimistic stochastic gradient ( OSG ) analysis beyond the bilinear and unconstrained case , by assuming the Lipschitz continuity of the operator T and the existence of a solution for the variational inequality MVI ( T , X ) . These conditions were considered in the analysis of the stochastic extragradient algorithm in ( Iusem et al. , 2017 ) . We analyze a variant of Optimistic Stochastic Gradient ( OSG ) under these conditions , inspired by the analysis of ( Iusem et al. , 2017 ) . We show that OSG achieves state-of-the-art iteration complexityO ( 1/ 4 ) for finding an -first-order stationary point . Note that our OSG variant only requires invoking one stochastic first-order oracle 3Note that the pseudo-monotonicity assumption used by ( Iusem et al. , 2017 ) can also be replaced by our MVI assumption in their proof . The main difference between our OSG and the stochastic extragradient method in ( Iusem et al. , 2017 ) is the number of stochastic gradient calculations in each iteration . 4Here the bilinear game is defined as minu∈Rp maxv∈Rq u > Av+u > a+bv > , where the smallest singular value of A ∈ Rp×q is positive , a ∈ Rp×1 , b ∈ R1×q . 5Here n , L , µ denote the number of components in the finite sum structure , Lipschitz constant and strongmonotonicity parameter of the operator of variational inequality respectively . while enjoying the state-of-the-art iteration complexity achieved by stochastic extragradient method ( Iusem et al. , 2017 ) . • Under the same conditions , we design an adaptive gradient algorithm named Optimistic Adagrad ( OAdagrad ) , and show that it enjoys better adaptive complexity O ( − 2 1−α ) , where α characterizes the growth rate of cumulative stochastic gradient and 0 ≤ α ≤ 1/2 . Similar to Adagrad ( Duchi et al. , 2011 ) , our main innovation is in considering variable metrics according to the geometry of the data in order to achieve potentially faster convergence rate for a class of nonconvex-nonconcave min-max games . Note that this adaptive complexity improves upon the non-adaptive one ( i.e. , O ( 1/ 4 ) ) achieved by OSG . To the best of our knowledge , we establish the first known adaptive complexity for adaptive gradient algorithms in a class of non-convex non-concave min-max problems . • We demonstrate the effectiveness of our algorithms in GAN training on CIFAR10 data . Empirical results identify an important reason behind why adaptive gradient methods behave well in GANs , which is due to the fact that the cumulative stochastic gradient grows in a slow rate . We also show that OAdagrad outperforms Simultaneous Adam in sample quality in ImageNet generation using self-attention GANs ( Zhang et al. , 2018 ) . This confirms the superiority of OAdagrad in min-max optimization .

Summary: This paper proposes two methods named OSG and OAdagrad for solving stochastic non-convex non-concave min-max problems. In theoretical analyses, a convergence rate of $O(\epsilon^{-4})$ and a much better rate are provided for OSG and OAdagrad, respectively, to find the $\epsilon$-accurate first-order stationary point. Finally, the superior performance of the proposed method is empirically verified on training generative adversarial networks (GANs).