GENERALIZED ADAPTIVE MOMENT ESTIMATION

Adaptive gradient methods have experienced great success in training deep neural networks ( DNNs ) . The basic idea of the methods is to track and properly make use of the first and/or second moments of the gradient for model-parameter updates over iterations for the purpose of removing the need for manual interference . In this work , we propose a new adaptive gradient method , referred to as generalized adaptive moment estimation ( Game ) . From a high level perspective , the new method introduces two more parameters w.r.t . AMSGrad ( S. J. Reddi & Kumar ( 2018 ) ) and one more parameter w.r.t . PAdam ( Chen & Gu ( 2018 ) ) to enlarge the parameterselection space for performance enhancement while reducing the memory cost per iteration compared to AMSGrad and PAdam . The saved memory space amounts to the number of model parameters , which is significant for large-scale DNNs . Our motivation for introducing additional parameters in Game is to provide algorithmic flexibility to facilitate a reduction of the performance gap between training and validation datasets when training a DNN . Convergence analysis is provided for applying Game to solve both convex optimization and smooth nonconvex optimization . Empirical studies for training four convolutional neural networks over MNIST and CIFAR10 show that under proper parameter selection , Game produces promising validation performance as compared to AMSGrad and PAdam . 1 INTRODUCTION . Stochastic gradient descent ( SGD ) and its variants have become the mainstream training methods in machine learning ( ML ) due to its simplicity and effectiveness . In general , SGD is known to work reasonably well regardless of their problem structure if the learning rate is set properly in a dynamical manner over training iterations . Intuitively speaking , if optimization problems admit certain structural properties ( e.g. , gradients magnitudes not balanced across the parameter set ) , advanced gradient descent methods exploiting the structural properties would be likely to boost optimization performance . In 2011 , Duchi et al . firstly proposed to track the second moment of gradients and then scale each gradient coordinate using the tracked information before updating the model parameters , which is referred to as AdaGrad ( Duchi et al . ( 2011 ) ) . From a conceptual point of view , AdaGrad adaptively adjusts “ individual learning rates ” for all the model parameters , allowing for the parameters to be updated on a roughly equal scale . It is found that AdaGrad converges significantly faster than SGD when the gradients are sparse . Its performance deteriorates when the gradients are dense due to a rapid decay of the learning rates . Since the pioneering work of AdaGrad , various adaptive gradient methods have been proposed on tracking and using the first and/or second moment of the gradients for effective parameter updates . The methods include , for example , RMSProp ( Tieleman & Hinton ( 2012 ) ) , Adam ( Kingma & Ba ( 2017 ) ) , and NAdam ( Dozat ( 2016 ) ) . The main difference of the above methods from AdaGrad is that the first and/or second moment of the gradients are tracked via exponential moving averaging to enhance the importance of the most recent gradients . Despite the wide usage of Adam in ML community , Reddit et al . have recently shown that the method does not even converge for a class of specially constructed convex functions ( S. J. Reddi & Kumar ( 2018 ) ) due to the non-monotonicity of the “ individual learning rates ” being multiplied to the gradients . To fix the convergence issue of Adam , the authors proposed a so-called AMSGrad method by additionally tracking the maximum value of the second moment of gradients over iterations ( i.e. , vector v̂ of Alg . 1 in Table 1 ) . Later on , Chen and Guo generalized AMSGrad by introducing one free parameter ( i.e. , p of Alg . 2 in Table 1 ) , referred to as PAdam ( Chen & Gu ( 2018 ) ) . The authors ’ primary motivation is to improve the generalization performance by tuning the parameter . Both AMSGrad and PAdam require additional memory in comparison to Adam for storing v̂ . While machine learning has seen rapid advances in algorithm development , theoretical convergence analysis has also made remarkable progress recently . The work of ( Zhou et al . ( 2018 ) ) extends the original analysis for PAdam in ( Chen & Gu ( 2018 ) ) from convex optimization to smooth nonconvex optimization . In another recent article ( Chen et al . ( 2018 ) ) , the authors also considered smooth nonconvex optimization and provided convergence analysis for a class of Adam-related algorithms including AMSGrad and Adam . The convergence bounds developed in the above two articles differ in their analysis approach and step-size selection . From a high level point of view , analysis on nonconvex optimization is highly valuable in practice as training deep neural networks ( DNNs ) is well known to be a nonconvex optimization problem . An improved convergence analysis on existing algorithms would provide insights on designing more advanced adaptive gradient methods . In this paper , we make two main contributions . Firstly , we propose a new adaptive gradient method named generalized adaptive moment estimation ( Game ) . As shown in Table 1 , Game tracks only two variables ( m , v̂ ) 1 over iterations as compared to AMSGrad and PAdam , which track three variables ( m , v , v̂ ) . We emphasize that since the dimension of v is the same as the number of model parameters , the memory saved by Game can be remarkable for modern large-scale neural networks , which usually have millions of parameters . Furthermore , Game introduces two parameters ( i.e. , ( p , q ) of Alg . 3 in Table 1 ) in comparison to PAdam which introduces one parameter ( i.e. , p of Alg . 2 in Table 1 ) to further enhance its flexibility . The parameter q enables Game to track information of the qth moment of gradient magnitude rather than only the second moment of the gradients . By doing so , the parameter provides one more degree of freedom for the method to balance the tradeoff between convergence speed on training data and generalization ability on validation data . Secondly , in our theoretical convergence analysis for Game , we manage to remove the condition on the relationship between β1 and β2 while β1 ≤ β2 and β1 ≤ βp2 are required in ( S. J. Reddi & Kumar ( 2018 ) ) for AMSGrad and ( Zhou et al . ( 2018 ) ) for PAdam , respectively . We provide convergence analysis for both convex optimization and smooth nonconvex optimization . The results for nonconvex case are briefly summarized in Table 1 with the analysis results from ( Zhou et al . ( 2018 ) ) as a reference . Finally , our experimental results on training four convolutional neural networks ( CNNs ) for MNIST and CIFAR10 suggest that with a proper setup of ( p , q ) , Game produces promising validation performance in comparison to AMSGrad and PAdam . 2 PROBLEM SETUP . Before introducing the problem , we describe the notation used in the paper , which are basically in line with that of ( S. J. Reddi & Kumar ( 2018 ) ) and ( Chen & Gu ( 2018 ) ) for consistency . We denote 1v is not tracked but just computed and used at each iteration . scalars by lower-case letters , vectors by bold lower-case letters , and matrices by bold upper-case letters . Following convention , we denote the lp ( p ≥ 1 ) norm of a vector x ∈ Rd by ‖x‖p = ( ∑d i=1 |xi|p ) 1/p . As p → ∞ , the l∞ norm takes a special form ‖x‖∞ = maxdi=1 |xi| . We let g1 : t , i = [ g1 , i , g2 , i , · · · , gt , i ] , where gt , i is the ith element of a vector gt ∈ Rd . We use E [ · ] to denote the expectation operation . Given two sequences { at } and { bt } , the notation at = O ( bt ) indicates that the magnitude of at is proportional to that of bt for t ≥ 0 . Formally , we reconsider solving the stochastic smooth nonconvex optimization studied in ( Ghadimi & Lan ( 2013 ; 2016 ) ; Zhou et al . ( 2018 ) ) min x∈Rd f ( x ) = Eξ [ f ( x ; ξ ) ] , ( 1 ) where x represents the parameters of a model in a vector form and ξ is a random variable . Due to randomness of ξ , one can only obtain an unbiased noisy gradient ∇f ( x ; ξ ) , satisfying ∇f ( x ) = Eξ [ ∇f ( x ; ξ ) ] . In practice , the variable ξ often represents the random mini-batch state . For the above situation , we denote the function realization at iteration t as ft ( x ; ξt ) , where ξt is a realization of ξ . Solving ( 1 ) is asymptotically equivalent to addressing the following optimization lim T→∞ min x∈Rd T∑ t=1 ft ( x ; ξt ) . ( 2 ) We will consider both the two formulations ( 1 ) and ( 2 ) in the reminder of the paper . 3 GENERALIZED ADAPTIVE MOMENT ESTIMATION ( GAME ) . In this section , we present our new adaptive gradient method for solving ( 1 ) . Our algorithm Game is motivated by making two observations about AMSGrad and PAdam in Table 1 . Firstly , both the two methods have to track the parameter v̂ in addition to v. In particular , the parameter v̂t at iteration t is computed as vt = β2vt−1 + ( 1− β2 ) g2t ( 3 ) v̂t = max ( vt , v̂t−1 ) . ( 4 ) It is seen from ( 3 ) - ( 4 ) that vt is a function of the squared gradients { g2j } tj=1 while v̂t tracks the maximum values of { vj } tj=1 up to iteration t. From a high level perspective , it feels redundant to keep both vt and v̂t in AMSGrad and PAdam as both parameters are related to the second moment of the gradients . One natural question is if it is sufficient to keep only one parameter about the second moment of the gradients without sacrificing convergence speed . Less memory usage is always desirable in designing an algorithm for simplicity and applicability . Secondly , it is clear from Table 1 that PAdam is designed by replacing the diagonal matrix V̂ − 1 2 t with V̂ −pt at iteration t in AMSGrad , where p ∈ [ 0 , 12 ] . One can easily show that p = 0 corresponds to SGD while p = 12 leads to AMSGrad . The parameter p establishes a smooth connection between SGD and AMSGrad , allowing the resulting method PAdam to carry the advantages of both methods . The empirical study in ( Chen & Gu ( 2018 ) ) suggests that when p is properly chosen in the range [ 0 , 12 ] , PAdam shows better generalization performance than AMSGrad , which is as expected due to the fact SGD usually produces good generalization performance . To summarize , the above technique of introducing an additional parameter to an existing algorithm adds one more degree of freedom to enhance its applicability . One can find many examples in applied mathematics that use similar techniques for knowledge expansion . For instance , the extension from l2 norm to lp norm and generalization from Cauchy-Schwarz inequality to Hölder ’ s inequality . Based on the above two observations , we propose a new adaptive gradient method named Game as summarized in Table 1 . Basically , the new method only tracks two parameters ( m , v̂ ) , where v̂t at iteration t is computed as vt = β2v̂t−1 + ( 1− β2 ) |gt|q ( 5 ) v̂t = max ( vt , v̂t−1 ) , ( 6 ) where q > 0 is our newly introduced parameter and |gt|q denotes element-wise qth power of |gt| . It is clear from ( 5 ) - ( 6 ) that vt is only a temporary parameter at iteration t and is not required to be stored in the memory for next iteration . Therefore , it is safe to say that Game saves roughly 33 percent memory per iteration as compared to AMSGrad and PAdam by removing the need for tracking { vt } . We now study the impact of saved memory by Game when training large-scale neural networks . Note that the dimension of v is actually the number of model parameters . That is , Game manages to save a memory space which is equivalent to the size of the neural network to be trained . It is known that state-of-the-art neural networks for challenging tasks ( e.g. , ImageNet competition ( Russakovsky et al . ( 2015 ) ) ) tend to be extremely deep and consist of millions of parameters ( Simonyan & Zisserman ( 2016 ) ) . When applying Game to train those large-scale networks , the memory saved by the new training method becomes remarkable with AMSGrad as a reference . Next we let q = 2 and p = 12 for Game to make a fair comparison w.r.t . AMSGrad . In this case , one can easily show that v̂t in ( 6 ) is either greater than or equal to the one in ( 4 ) . As a result , Game would always have either equal or smaller effective learning rate αtV̂ − 12 t than AMSGrad . The above property implies that Game is more conservative than AMSGrad and PAdam due to the update reformulation ( 5 ) - ( 6 ) . If needed , one can adjust the parameters { αt } to larger values to make Game more aggressive . Finally we consider the new scalar parameter q introduced in ( 5 ) . From an algebraic point of view , introduction of parameter q allows v̂ in ( 5 ) - ( 6 ) to track information of the qth moment of the gradient magnitude over iterations . As q decreases , small gradients would be amplified while large gradients would be suppressed , leading to a decreasing dynamic range of v̂ . When q → 0 , it is not difficult to show that Game approaches SGD . Therefore , the parameter q has a similar effect as the parameter p of PAdam . We point out that when β2 > 0 , we can not merge p and q into one parameter via certain reformulation of the updating expressions of Game . That is , p and q play different roles in Game . Our purpose of introducing q in addition to p is to enlarge the parameter-selection space of Game and improve generalization performance with proper parameter setup .

Summary: The authors proposed a generalized adaptive moment estimation method(Game). Compared to the existing methods AMSGrad and PAdam, the new method Game tracks only two parameters in iteration and hence saves memory. Besides, they introduced a additional tuning parameter $q$ to track the q-th moment of the gradient and allow more flexibility. The authors also provided the theoretical convergence analysis of Game for convex optimization and smooth nonconvex optimization. Their experiment shows Game may produce better performance than AMSGrad and PAdam with a little bit sacrifice of convergence speed. Game is a promising alternative method for training large-scale neural network.