Padam: Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks

1 INTRODUCTION . Stochastic gradient descent ( SGD ) is now one of the most dominant approaches for training deep neural networks ( Goodfellow et al. , 2016 ) . In each iteration , SGD only performs one parameter update on a mini-batch of training examples . SGD is simple and has been proved to be efficient , especially for tasks on large datasets . In recent years , adaptive variants of SGD have emerged and shown their successes for their convenient automatic learning rate adjustment mechanism . Adagrad ( Duchi et al. , 2011 ) is probably the first along this line of research , and significantly outperforms vanilla SGD in the sparse gradient scenario . Despite the first success , Adagrad was later found to demonstrate degraded performance especially in cases where the loss function is nonconvex or the gradient is dense . Many variants of Adagrad , such as RMSprop ( Hinton et al. , 2012 ) , Adam ( Kingma & Ba , 2015 ) , Adadelta ( Zeiler , 2012 ) , Nadam ( Dozat , 2016 ) , were then proposed to address these challenges by adopting exponential moving average rather than the arithmetic average used in Adagrad . This change largely mitigates the rapid decay of learning rate in Adagrad and hence makes this family of algorithms , especially Adam , particularly popular on various tasks . Recently , it has also been observed ( Reddi et al. , 2018 ) that Adam does not converge in some settings where rarely encountered large gradient information quickly dies out due to the “ short momery ” problem of exponential moving average . To address this issue , Amsgrad ( Reddi et al. , 2018 ) has been proposed to keep an extra “ long term memory ” variable to preserve the past gradient information and to correct the potential convergence issue in Adam . On the other hand , people recently found that for largely over-parameterized neural networks , e.g. , more complex modern convolutional neural network ( CNN ) architectures such as VGGNet ( He et al. , 2016 ) , ResNet ( He et al. , 2016 ) , Wide ResNet ( Zagoruyko & Komodakis , 2016 ) , DenseNet ( Huang et al. , 2017 ) , training with Adam or its variants typically generalizes worse than SGD with momentum , even when the training performance is better . In particular , people found that carefullytuned SGD , combined with proper momentum , weight decay and appropriate learning rate decay strategies , can significantly outperform adaptive gradient algorithms eventually ( Wilson et al. , 2017 ) . As a result , even though adaptive gradient methods are relatively easy to tune and converge faster at the early stage , recent advances in designing neural network structures are all reporting their performances by training their models with SGD-momentum ( He et al. , 2016 ; Zagoruyko & Komodakis , 2016 ; Huang et al. , 2017 ; Simonyan & Zisserman , 2014 ; Ren et al. , 2015 ; Xie et al. , 2017 ; Howard et al. , 2017 ) . Moreover , it is difficult to apply the same learning rate decay strategies that work well in SGD with momentum to adaptive gradient methods , since adaptive gradient methods usually require a much smaller base learning rate that will soon die out after several rounds of decay . We refer to it as the “ small learning rate dilemma ” ( see more details in Section 3 ) . With all these observations , a natural question is : Can we take the best from both Adam and SGD-momentum , i.e. , design an algorithm that not only enjoys the fast convergence rate as Adam , but also generalizes as well as SGD-momentum ? In this paper , we answer this question affirmatively . We close the generalization gap of adaptive gradient methods by presenting a new algorithm , called partially adaptive momentum estimation ( Padam ) method , which unifies Adam/Amsgrad with SGD-momentum to achieve the best of both worlds . In particular , we make the following contributions : • We propose a new algorithm , Padam , which unifies Adam/Amsgrad and SGD with momentum by a partially adaptive parameter . We show that Adam/Amsgrad can be seen as a special fully adaptive instance of Padam . The intuition behind our algorithm is that , by controlling the degree of adaptiveness , the learning rate in Padam does not need to be as small as other adaptive gradient methods to prevent the gradient update from exploding . This resolves the “ small learning rate dilemma ” for adaptive gradient methods and allows for faster convergence , hence closing the gap of generalization . • We show that the Padam ’ s performance is also theoretically guaranteed . We provide a convergence analysis of Padam in the convex setting , based on the analysis of Kingma & Ba ( 2015 ) ; Reddi et al . ( 2018 ) , and prove a data-dependent regret bound . • We also provide thorough experiments about our proposed Padam method on training modern deep neural architectures . We empirically show that Padam achieves the fastest convergence speed while generalizing as well as SGD with momentum . These results suggest that practitioners should pick up adaptive gradient methods once again for faster training of deep neural networks . Notation : Scalars are denoted by lower case letters , vectors by lower case bold face letters , and matrices by upper case bold face letters . For a vector θ ∈ Rd , we denote the ` 2 norm of θ by ‖θ‖2 = √∑d i=1 θ 2 i , the ` ∞ norm of θ by ‖θ‖∞ = maxdi=1 |θi| . For a sequence of vectors { θj } tj=1 , we denote by θj , i the i-th element in θj . We also denote θ1 : t , i = [ θ1 , i , θ2 , i , . . . , θt , i ] > . With slightly abuse of notation , for any two vectors a and b , we denote a2 as the element-wise square , ap as the element-wise power operation , a/b as the element-wise division and max ( a , b ) as the element-wise maximum . We denote by diag ( a ) a diagonal matrix with diagonal entries a1 , . . . , ad . Let Sd++ be the set of all positive definite d × d matrices . We denote by ΠX , A ( b ) the projection of b onto a convex set X , i.e. , argmina∈X ‖A1/2 ( a− b ) ‖2 for b ∈ Rd , A ∈ Sd++ . Given two sequences { an } and { bn } , we write an = O ( bn ) if there exists a constant 0 < C < +∞ such that an ≤ C bn . We use notation Õ ( · ) to hide logarithmic factors . We write an = o ( bn ) if an/bn → 0 as n→∞ . 2 REVIEW OF ADAPTIVE GRADIENT METHODS . Various adaptive gradient methods have been proposed in order to achieve better performance on various stochastic optimization tasks . Adagrad ( Duchi et al. , 2011 ) is among the first methods with adaptive learning rate for each individual dimension , which motivates the research on adaptive gradient methods in the machine learning community . In detail , Adagrad1 adopts the following 1The formula here is equivalent to the one from the original paper ( Duchi et al. , 2011 ) after simple manipulations . update form : θt+1 = θt − αt gt√ vt , where vt = 1 t t∑ j=1 g2j ( Adagrad ) Here gt stands for the stochastic gradient ∇ft ( θt ) , and αt = α/ √ t is the step size ( a.k.a. , learning rate ) . Adagrad is proved to enjoy a huge gain in terms of convergence especially in sparse gradient situations . Empirical studies also show a performance gain even for non-sparse gradient settings . RMSprop ( Hinton et al. , 2012 ) follows the idea of adaptive learning rate and it changes the arithmetic averages used for vt in Adagrad to exponential moving averages . Even though RMSprop is an empirical method with no theoretical guarantee , the outstanding empirical performance of RMSprop raised people ’ s interests in exponential moving average variants of Adagrad . Adam ( Kingma & Ba , 2015 ) 2 is the most popular exponential moving average variant of Adagrad . It combines the idea of RMSprop and momentum acceleration , and takes the following update form : θt+1 = θt − αt mt√ vt , where mt = β1mt−1 + ( 1− β1 ) gt , vt = β2vt−1 + ( 1− β2 ) g2t ( Adam ) Adam also requires αt = α/ √ t for the sake of convergence analysis . In practice , any decaying step size or even constant step size works well for Adam . Note that if we choose β1 = 0 , Adam basically reduces to RMSprop . Reddi et al . ( 2018 ) identified a non-convergence issue in Adam . Specifically , Adam does not collect long-term memory of past gradients and therefore the effective learning rate could be increasing in some cases . They proposed a modified algorithm namely Amsgrad . More specifically , Amsgrad adopts an additional step to ensure the decay of the effective learning rate αt/ √ v̂t , and its key update formula is as follows : θt+1 = θt − αt mt√ v̂t , where v̂t = max ( v̂t−1 , vt ) ( Amsgrad ) The definitions of mt and vt are the same as Adam . Note that the modified v̂t ensures the convergence of Amsgrad . Reddi et al . ( 2018 ) also corrected some mistakes in the original proof of Adam and proved an O ( 1/ √ T ) convergence rate of Amsgrad for convex optimization . Note that all the theoretical guarantees on adaptive gradient methods ( Adagrad , Adam , Amsgrad ) are only proved for convex functions . 3 THE PROPOSED ALGORITHM . In this section , we propose a new algorithm that not only inherits the O ( 1/ √ T ) convergence rate from Adam/Amsgrad , but also has comparable or even better generalization performance than SGD with momentum . Specifically , we introduce the a partial adaptive parameter p to control the level of adaptivity of the optimization procedure . The proposed algorithm is displayed in in Algorithm 1 . Algorithm 1 Partially adaptive momentum estimation method ( Padam ) input : initial point θ1 ∈ X ; step sizes { αt } ; momentum parameters { β1t } , β2 ; partially adaptive parameter p ∈ ( 0 , 1/2 ] set m0 = 0 , v0 = 0 , v̂0 = 0 for t = 1 , . . . , T do gt = ∇ft ( θt ) mt = β1tmt−1 + ( 1− β1t ) gt vt = β2vt−1 + ( 1− β2 ) g2t v̂t = max ( v̂t−1 , vt ) θt+1 = ΠX , diag ( v̂pt ) ( θt − αt ·mt/v̂pt ) end for In Algorithm 1 , gt denotes the stochastic gradient and v̂t can be seen as a moving average over the second order momentum of the stochastic gradients . As we can see from Algorithm 1 , the key 2Here for simplicity and consistency , we ignore the bias correction step in the original paper of Adam . Yet adding the bias correction step will not affect the argument in the paper . difference between Padam and Amsgrad ( Reddi et al. , 2018 ) is that : while mt is still the first order momentum as in Adam/Amsgrad , it is now “ partially adapted ” by the second order momentum , i.e. , θt+1 = θt − αt mt v̂pt , where v̂t = max ( v̂t−1 , vt ) ( Padam ) We call p ∈ ( 0 , 1/2 ] the partially adaptive parameter . Note that 1/2 is the largest possible value for p and a larger p will result in non-convergence in the proof ( see the details of the proof in the appendix ) . When p→ 0 , Algorithm 1 reduces to SGD with momentum3 and when p = 1/2 , Algorithm 1 is exactly Amsgrad . Therefore , Padam indeed unifies Amsgrad and SGD with momentum . Now the question is , what value for p should we choose ? Or in another way , is p = 1/2 the best choice ? The answer is negative . The intuition behind this is simple : it is very likely that Adam/Amsgrad is “ over-adaptive ” . One notable fact that people found when using adaptive gradient methods to train deep neural networks is that the learning rate needs to be much smaller than that of SGD-momentum ( Keskar & Socher , 2017 ; Wilson et al. , 2017 ) . For many tasks , the base learning rate for SGD-momentum is usually set to be 0.1 while that of Adam is usually set to be 0.001 . In fact , the key reason that prohibits Adam from adopting a more aggressive learning rate is the large adaptive term 1/ √ v̂t . The existence of such a large adaptive term makes the effective learning rate ( αt/ √ v̂t ) easily explode with a larger αt . Moreover , the learning rate decaying strategy used in modern deep neural network training makes things worse . More specifically , after several rounds of decaying , the learning rates of the adaptive gradient methods are too small to make any significant progress in the training process . We call it “ small learning rate dilemma ” . This explains the relatively weak performances of adaptive gradient methods at the later stage of the training process , where the non-adaptive gradient methods like SGD start to outperform them . The above discussion suggests that we should consider using Padam with a proper adaptive parameter p , which will enable us to adopt a larger learning rate to avoid the “ small learning rate dilemma ” . And we will show in our experiments ( Section 5 ) that Padam can adopt an equally large base learning rate as SGD with momentum . Figure 1 shows the comparison of test error performances under the different partial adaptive parameter p for ResNet on both CIFAR-10 and CIFAR-100 datasets . We can observe that a larger p will lead to fast convergence at early stages and worse generalization performance later , while a smaller p behaves more like SGD with momentum : slow in early stages but finally catch up . With a proper choice of p ( e.g. , 1/8 in this case ) , Padam can obtain the best of both worlds . Note that besides Algorithm 1 , our partially adaptive idea can also be applied to other adaptive gradient methods such as Adagrad , Adadelta , RMSprop , AdaMax ( Kingma & Ba , 2015 ) . For the sake of conciseness , we do not list the partially adaptive versions for other adaptive gradient methods here . We also would like to comment that Padam is totally different from the p-norm generalized version of Adam in Kingma & Ba ( 2015 ) , which induces AdaMax method when p → ∞ . In their 3The only difference between Padam with p = 0 and SGD with momentum is an extra constant factor ( 1 − β1 ) , which can be moved into the learning rate such that the update rules for these two algorithms are identical . case , p-norm is used to generalize 2-norm of their current and past gradients while keeping the scale of adaptation unchanged . In sharp contrast , we intentionally change ( reduce ) the scale of the adaptive term in Padam to get better generalization performance . Finally , note that in Algorithm 1 we remove the bias correction step used in the original Adam paper following Reddi et al . ( 2018 ) . Nevertheless , our arguments and theory are applicable to the bias correction version as well .

Summary: The idea is simple and promising: generalize AMSgrad and momentum by hyperparameterizing the p=1/2 in denominator of ADAM term to be within [0,1/2], with 0 being momentum case.  It was good to see the experiments use non-MNIST data (e.g. ImageNet, Cifar) and reasonable CNN models (ResNet, VGG).  However, the experimental evaluation is not convincing that this approach will lead to significant improvements in optimizing such modern models in practice.