Convergence Guarantees for RMSProp and ADAM in Non-Convex Optimization and an Empirical Comparison to Nesterov Acceleration

Next we design experiments to empirically study the convergence and generalization properties of RMSProp and ADAM against Nesterov ’ s Accelerated Gradient method on a variety of common autoencoder setups and on VGG-9 with CIFAR-10 . Through these experiments we demonstrate the interesting sensitivity that ADAM has to its momentum parameter β1 . We show that at very high values of the momentum parameter ( β1 = 0.99 ) ADAM outperforms a carefully tuned NAG on most of our experiments , in terms of getting lower training and test losses . On the other hand , NAG can sometimes do better when ADAM ’ s β1 is set to the most commonly used value : β1 = 0.9 , indicating the importance of tuning the hyperparameters of ADAM to get better generalization performance . We also report experiments on different autoencoders to demonstrate that NAG has better abilities in terms of reducing the gradient norms , and it also produces iterates which exhibit an increasing trend for the minimum eigenvalue of the Hessian of the loss function at the iterates . 1 INTRODUCTION . Many optimization questions arising in machine learning can be cast as a finite sum optimization problem of the form : minx f ( x ) where f ( x ) = 1k ∑k i=1 fi ( x ) . Most neural network problems also fall under a similar structure where each function fi is typically non-convex . A well-studied algorithm to solve such problems is Stochastic Gradient Descent ( SGD ) , which uses updates of the form : xt+1 : = xt − α∇f̃it ( xt ) , where α is a step size , and f̃it is a function chosen randomly from { f1 , f2 , . . . , fk } at time t. Often in neural networks , “ momentum ” is added to the SGD update to yield a two-step update process given as : vt+1 = µvt−α∇f̃it ( xt ) followed by xt+1 = xt+vt+1 . This algorithm is typically called the Heavy-Ball ( HB ) method ( or sometimes classical momentum ) , with µ > 0 called the momentum parameter ( Polyak , 1987 ) . In the context of neural nets , another variant of SGD that is popular is Nesterov ’ s Accelerated Gradient ( NAG ) , which can also be thought of as a momentum method ( Sutskever et al. , 2013 ) , and has updates of the form vt+1 = µvt − α∇f̃it ( xt + µvt ) followed by xt+1 = xt + vt+1 ( see Algorithm 1 for more details ) . Momentum methods like HB and NAG have been shown to have superior convergence properties compared to gradient descent in the deterministic setting both for convex and non-convex functions ( Nesterov , 1983 ; Polyak , 1987 ; Zavriev & Kostyuk , 1993 ; Ochs , 2016 ; O ’ Neill & Wright , 2017 ; Jin et al. , 2017 ) . While ( to the best of our knowledge ) there is no clear theoretical justification in the stochastic case of the benefits of NAG and HB over regular SGD in general ( Yuan et al. , 2016 ; Kidambi et al. , 2018 ; Wiegerinck et al. , 1994 ; Orr & Leen , 1994 ; Yang et al. , 2016 ; Gadat et al. , 2018 ) , unless considering specialized function classes ( Loizou & Richtárik , 2017 ) ; in practice , these momentum methods , and in particular NAG , have been repeatedly shown to have good convergence and generalization on a range of neural net problems ( Sutskever et al. , 2013 ; Lucas et al. , 2018 ; Kidambi et al. , 2018 ) . The performance of NAG ( as well as HB and SGD ) , however , are typically quite sensitive to the selection of its hyper-parameters : step size , momentum and batch size ( Sutskever et al. , 2013 ) . Thus , “ adaptive gradient ” algorithms such as RMSProp ( Algorithm 2 ) ( Tieleman & Hinton , 2012 ) and ADAM ( Algorithm 3 ) ( Kingma & Ba , 2014 ) have become very popular for optimizing deep neural networks ( Melis et al. , 2017 ; Xu et al. , 2015 ; Denkowski & Neubig , 2017 ; Gregor et al. , 2015 ; Radford et al. , 2015 ; Bahar et al. , 2017 ; Kiros et al. , 2015 ) . The reason for their widespread popularity seems to be the fact that they are believed to be easier to tune than SGD , NAG or HB . Adaptive gradient methods use as their update direction a vector which is the image under a linear transformation ( often called the “ diagonal pre-conditioner ” ) constructed out of the history of the gradients , of a linear combination of all the gradients seen till now . It is generally believed that this “ pre-conditioning ” makes these algorithms much less sensitive to the selection of its hyperparameters . A precursor to these RMSProp and ADAM was outlined in Duchi et al . ( 2011 ) . Despite their widespread use in neural net problems , adaptive gradients methods like RMSProp and ADAM lack theoretical justifications in the non-convex setting - even with exact/deterministic gradients ( Bernstein et al. , 2018 ) . Further , there are also important motivations to study the behavior of these algorithms in the deterministic setting because of usecases where the amount of noise is controlled during optimization , either by using larger batches ( Martens & Grosse , 2015 ; De et al. , 2017 ; Babanezhad et al. , 2015 ) or by employing variance-reducing techniques ( Johnson & Zhang , 2013 ; Defazio et al. , 2014 ) . Further , works like Wilson et al . ( 2017 ) and Keskar & Socher ( 2017 ) have shown cases where SGD ( no momentum ) and HB ( classical momentum ) generalize much better than RMSProp and ADAM with stochastic gradients . Wilson et al . ( 2017 ) also show that ADAM generalizes poorly for large enough nets and that RMSProp generalizes better than ADAM on a couple of neural network tasks ( most notably in the character-level language modeling task ) . But in general it ’ s not clear and no heuristics are known to the best of our knowledge to decide whether these insights about relative performances ( generalization or training ) between algorithms hold for other models or carry over to the full-batch setting . A summary of our contributions In this work we try to shed some light on the above described open questions about adaptive gradient methods in the following two ways . • To the best of our knowledge , this work gives the first convergence guarantees for adaptive gradient based standard neural-net training heuristics . Specifically we show run-time bounds for deterministic RMSProp and ADAM to reach approximate criticality on smooth non-convex functions , as well as for stochastic RMSProp under an additional assumption . Recently , Reddi et al . ( 2018 ) have shown in the setting of online convex optimization that there are certain sequences of convex functions where ADAM and RMSprop fail to converge to asymptotically zero average regret . We contrast our findings with Theorem 3 in Reddi et al . ( 2018 ) . Their counterexample for ADAM is constructed in the stochastic optimization framework and is incomparable to our result about deterministic ADAM . Our proof of convergence to approximate critical points establishes a key conceptual point that for adaptive gradient algorithms one can not transfer intuitions about convergence from online setups to their more common use case in offline setups . • Our second contribution is empirical investigation into adaptive gradient methods , where our goals are different from what our theoretical results are probing . We test the convergence and generalization properties of RMSProp and ADAM and we compare their performance against NAG on a variety of autoencoder experiments on MNIST data , in both full and mini-batch settings . In the full-batch setting , we demonstrate that ADAM with very high values of the momentum parameter ( β1 = 0.99 ) matches or outperforms carefully tuned NAG and RMSProp , in terms of getting lower training and test losses . We show that as the autoencoder size keeps increasing , RMSProp fails to generalize pretty soon . In the mini-batch experiments we see exactly the same behaviour for large enough nets . We further validate this behavior on an image classification task on CIFAR-10 using a VGG-9 convolutional neural network , the results to which we present in the Appendix E. We note that recently it has been shown by Lucas et al . ( 2018 ) , that there are problems where NAG generalizes better than ADAM even after tuning β1 . In contrast our experiments reveal controlled setups where tuning ADAM ’ s β1 closer to 1 than usual practice helps close the generalization gap with NAG and HB which exists at standard values of β1 . Remark . Much after this work was completed we came to know of a related paper ( Li & Orabona , 2018 ) which analyzes convergence rates of a modification of AdaGrad ( not RMSPRop or ADAM ) . After the initial version of our work was made public , a few other analysis of adaptive gradient methods have also appeared like Chen et al . ( 2018 ) , Zhou et al . ( 2018 ) and Zaheer et al . ( 2018 ) . 2 NOTATIONS AND PSEUDOCODES . Firstly we define the smoothness property that we assume in our proofs for all our non-convex objectives . This is a standard assumption used in the optimization literature . Definition 1 . L−smoothness If f : Rd → R is at least once differentiable then we call it L−smooth for some L > 0 if for all x , y ∈ Rd the following inequality holds , f ( y ) ≤ f ( x ) + 〈∇f ( x ) , y − x〉+ L2 ‖y − x‖ 2 . We need one more definition that of square-root of diagonal matrices , Definition 2 . Square root of the Penrose inverse If v ∈ Rd and V = diag ( v ) then we define , V − 1 2 : = ∑ i∈Support ( v ) 1√ vi eie T i , where { ei } { i=1 , ... , d } is the standard basis of Rd Now we list out the pseudocodes used for NAG , RMSProp and ADAM in theory and experiments , Nesterov ’ s Accelerated Gradient ( NAG ) Algorithm . Algorithm 1 NAG 1 : Input : A step size α , momentum µ ∈ [ 0 , 1 ) , and an initial starting point x1 ∈ Rd , and we are given query access to a ( possibly noisy ) oracle for gradients of f : Rd → R. 2 : function NAG ( x1 , α , µ ) 3 : Initialize : v1 = 0 4 : for t = 1 , 2 , . . . do 5 : vt+1 = µvt +∇f ( xt ) 6 : xt+1 = xt − α ( ∇f ( xt ) + µvt+1 ) 7 : end for 8 : end function RMSProp Algorithm . Algorithm 2 RMSProp 1 : Input : A constant vector Rd 3 ξ1d ≥ 0 , parameter β2 ∈ [ 0 , 1 ) , step size α , initial starting point x1 ∈ Rd , and we are given query access to a ( possibly noisy ) oracle for gradients of f : Rd → R. 2 : function RMSPROP ( x1 , β2 , α , ξ ) 3 : Initialize : v0 = 0 4 : for t = 1 , 2 , . . . do 5 : gt = ∇f ( xt ) 6 : vt = β2vt−1 + ( 1− β2 ) ( g2t + ξ1d ) 7 : Vt = diag ( vt ) 8 : xt+1 = xt − αV − 12 t gt 9 : end for 10 : end function

Summary: This paper present a convergence analysis of the popular methods RMSProp and ADAM in the case of smooth non-convex functions. In particular it was shown that the above adaptive gradient algorithms are guaranteed to reach critical points for smooth non-convex objectives and bounds on the running time are provided. An empirical investigation is also presented with main focus on the comparison of the adaptive gradient methods and the Nesterov accelerated gradient algorithm (NAG).