Adaptive norms for deep learning with regularized Newton methods

1 INTRODUCTION . We consider finite-sum optimization problems of the form min w∈Rd [ L ( w ) : = n∑ i=1 ` ( f ( w , xi , yi ) ) ] , ( 1 ) which typically arise in neural network training , e.g . for empirical risk minimization over a set of data points ( xi , yi ) ∈ Rin × Rout , i = 1 , . . . , n. Here , ` : Rout × Rout → R+ is a convex loss function and f : Rin × Rd → Rout represents the neural network mapping parameterized by the concatenation of the weight layers w ∈ Rd , which is non-convex due to its multiplicative nature and potentially non-linear activation functions . We assume that L is lower bounded and twice differentiable , i.e . L ∈ C2 ( Rd , R ) and consider finding a first- and second-order stationary point w̄ for which ‖∇L ( w̄ ) ‖ ≤ g and λmin ( ∇2L ( w̄ ) ) ≥ − H . In the era of deep neural networks , stochastic gradient descent ( SGD ) is one of the most widely used training algorithms ( Bottou , 2010 ) . What makes SGD so attractive is its simplicity and per-iteration cost that is independent of the size of the training set ( n ) and scale linearly in the dimensionality ( d ) . However , gradient descent is known to be inadequate to optimize functions that are ill-conditioned ( Nesterov , 2013 ; Shalev-Shwartz et al. , 2017 ) and thus adaptive gradient methods that employ dynamic , coordinate-wise learning rates based on past gradients—including Adagrad ( Duchi et al. , 2011 ) , RMSprop ( Tieleman & Hinton , 2012 ) and Adam ( Kingma & Ba , 2014 ) —have become a popular alternative , often providing significant speed-ups over SGD . From a theoretical perspective , Newton methods provide stronger convergence guarantees by appropriately transforming the gradient in ill-conditioned regions according to second-order derivatives . It is precisely this Hessian information that allows regularized Newton methods to enjoy superlinear local convergence as well as to provably escape saddle points ( Conn et al. , 2000 ) . While second-order algorithms have a long-standing history even in the realm of neural network training ( Hagan & Menhaj , 1994 ; Becker et al. , 1988 ) , they were mostly considered as too computationally and memory expensive for practical applications . Yet , the seminal work of Martens ( 2010 ) renewed interest for their use in deep learning by proposing efficient Hessian-free methods that only access second-order information via matrix-vector products which can be computed at the cost of an additional backpropagation ( Pearlmutter , 1994 ; Schraudolph , 2002 ) . Among the class of regularized Newton methods , trust region ( Conn et al. , 2000 ) and cubic regularization algorithms ( Cartis et al. , 2011 ) are the most principled approaches in the sense that they yield the strongest convergence guarantees . Recently , stochastic extensions have emerged ( Xu et al. , 2017b ; Yao et al. , 2018 ; Kohler & Lucchi , 2017 ; Gratton et al. , 2017 ) , which suggest their applicability for deep learning . We here propose a simple modification to make TR methods even more suitable for neural network training . Particularly , we build upon the following alternative view on adaptive gradient methods : While gradient descent can be interpreted as a spherically constrained first-order TR method , preconditioned gradient methods—such as Adagrad—can be seen as first-order TR methods with ellipsoidal trust region constraint . This observation is particularly interesting since spherical constraints are blind to the underlying geometry of the problem , but ellipsoids can adapt to local landscape characteristics , thereby allowing for more suitable steps in regions that are ill-conditioned . We will leverage this analogy and investigate the use of the Adagrad and RMSProp preconditioning matrices as ellipsoidal trust region shapes within a stochastic second-order TR algorithm ( Xu et al. , 2017a ; Yao et al. , 2018 ) . Since no ellipsoid fits all objective functions , our main contribution lies in the identification of adequate matrix-induced constraints that lead to provable convergence and significant practical speed-ups for the specific case of deep learning . On the whole , our contribution is threefold : • We provide a new perspective on adaptive gradient methods that contributes to a better understanding of their inner-workings . • We investigate the first application of ellipsoidal TR methods for deep learning . We show that the RMSProp matrix can directly be applied as constraint inducing norm in second-order TR algorithms while preserving all convergence guarantees ( Theorem 1 ) . • Finally , we provide an experimental benchmark across different real-world datasets and architectures ( Section 5 ) . We compare second-order methods also to adaptive gradient methods and show results in terms of backpropagations , epochs , and wall-clock time ; a comparison we were not able to find in the literature . Our main empirical results demonstrate that ellipsoidal constraints prove to be a very effective modification of the trust region method in the sense that they constantly outperform the spherical TR method , both in terms of number of backprogations and asymptotic loss value on a variety of tasks . 2 RELATED WORK . First-order methods The prototypical method for optimizing Eq . ( 1 ) is SGD ( Robbins & Monro , 1951 ) . The practical success of SGD in non-convex optimization is unquestioned and theoretical explanations of this phenomenon are starting to appear . Recent findings suggest the ability of this method to escape saddle points and reach local minima in polynomial time , but they either need to artificially add noise to the iterates ( Ge et al. , 2015 ; Lee et al. , 2016 ) or make an assumption on the inherent noise of SGD ( Daneshmand et al. , 2018 ) . For neural networks , a recent line of research proclaims the effectiveness of SGD , but the results come at the cost of strong assumptions such as heavy over-parametrization and Gaussian inputs ( Du et al. , 2017 ; Brutzkus & Globerson , 2017 ; Li & Yuan , 2017 ; Du & Lee , 2018 ; Allen-Zhu et al. , 2018 ) . Adaptive gradient methods ( Duchi et al. , 2011 ; Tieleman & Hinton , 2012 ; Kingma & Ba , 2014 ) build on the intuition that larger ( smaller ) learning rates for smaller ( larger ) gradient components balance their respective influences and thereby the methods behave as if optimizing a more isotropic surface . Such approaches have first been suggested for neural nets by LeCun et al . ( 2012 ) and convergence guarantees are starting to appear ( Ward et al. , 2018 ; Li & Orabona , 2018 ) . However , these are not superior to the O ( −2g ) worst-case complexity of standard gradient descent ( Cartis et al. , 2012b ) . Regularized Newton methods The most principled class of regularized Newton methods are trust region ( TR ) and adaptive cubic regularization algorithms ( ARC ) ( Conn et al. , 2000 ; Cartis et al. , 2011 ) , which repeatedly optimize a local Taylor model of the objective while making sure that the step does not travel too far such that the model stays accurate . While the former finds first-order stationary points within O ( −2g ) , ARC only takes at most O ( −3/2 g ) . However , simple modifications to the TR framework allow these methods to obtain the same accelerated rate ( Curtis et al. , 2017 ) . Both methods take at most O ( −3H ) iterations to find an H approximate second-order stationary point ( Cartis et al. , 2012a ) . These rates are optimal for second-order Lipschitz continuous functions ( Carmon et al. , 2017 ; Cartis et al. , 2012a ) and they can be retained even when only sub-sampled gradient and Hessian information is used ( Kohler & Lucchi , 2017 ; Yao et al. , 2018 ; Xu et al. , 2017b ; Blanchet et al. , 2016 ; Liu et al. , 2018 ) . Furthermore , the involved Hessian information can be computed solely based on Hessian-vector products , which are implementable very efficiently ( Pearlmutter , 1994 ) . This makes these methods particularly attractive for deep learning , but the empirical evidence of their applicability is rather limited . We are only aware of the works of Liu et al . ( 2018 ) and Xu et al . ( 2017a ) , which report promising first results but are by no means fully encompassing . Gauss-Newton methods An interesting line of research proposes to replace the Hessian by ( approximations of ) the generalized-Gauss-Newton matrix ( GGN ) within a Levenberg-Marquardt framework1 ( LeCun et al. , 2012 ; Martens , 2010 ; Martens & Grosse , 2015 ) . As the GGN matrix is always positive semidefinite , these methods can not leverage negative curvature to escape saddles and hence , there exist no second-order convergence guarantees . Furthermore , there are cases in neural networks where the Hessian is better conditioned than the GGN matrix ( Mizutani & Dreyfus , 2008 ) . Nevertheless , the above works report promising preliminary results , most notably Grosse & Martens ( 2016 ) find that K-FAC can be faster than SGD on a small convnet . On the other hand , recent findings report performance at best comparable to SGD on the much larger ResNet architecture ( Ma et al. , 2019 ) . Moreover , Xu et al . ( 2017a ) reports many cases where TR and GGN algorithms perform similarly . This line of work can be seen as complementary to our approach since it is straightforward to replace the Hessian in the TR framework with the GGN matrix . Furthermore , the preconditioners used in Martens ( 2010 ) and Chapelle & Erhan ( 2011 ) , namely diagonal estimates of the empirical Fisher and Fisher matrix , respectively , can directly be used as matrix norms in our ellipsoidal TR framework . 3 AN ALTERNATIVE VIEW ON ADAPTIVE GRADIENT METHODS . Adaptively preconditioned gradient methods update iterates as wt+1 = wt − ηtA−1/2t gt , where gt is a stochastic estimate of ∇L ( wt ) and At is a positive definite symmetric pre-conditioning matrix . In Adagrad , Aada , t is the un-centered second moment matrix of the past gradients computed as Aada , t : = GtG ᵀ t + I , ( 2 ) where > 0 , I is the d× d identity matrix and Gt = [ g1 , g2 , . . . , gt ] . Building up on the intuition that past gradients might become obsolete in quickly changing non-convex landscapes , RMSprop ( and Adam ) introduce an exponential weight decay leading to the preconditioning matrix Arms , t : = ( ( 1− β ) Gt diag ( βt , . . . , β0 ) Gᵀt ) + I , ( 3 ) where β ∈ ( 0 , 1 ) . In order to save computational efforts , the diagonal versions diag ( Aada ) and diag ( Arms ) are more commonly applied in practice , which in turn gives rise to coordinate-wise adaptive stepsizes that are enlarged ( reduced ) in coordinates that have seen past gradient components with a smaller ( larger ) magnitude . 3.1 ADAPTIVE PRECONDITIONING AS ELLIPSOIDAL TRUST REGION . Starting from the fact that adaptive methods employ coordinate-wise stepsizes , one can take a principled view on these methods . Namely , their update steps arise from minimizing a first-order Taylor model of the function L within an ellipsoidal search space around the current iterate wt , where the diameter of the ellipsoid along a particular coordinate is implicitly given by ηt and ‖gt‖A−1t . Correspondingly , vanilla ( S ) GD optimizes the same first-order model within a spherical constraint . Figure 1 ( top ) illustrates this effect by showing not only the iterates of GD and Adagrad but also the implicit trust regions within which the local models were optimized at each step.2 It is well known that GD struggles to progress towards the minimizer of quadratics along lowcurvature directions ( see e.g. , Goh ( 2017 ) ) . While this effect is negligible for well-conditioned objectives ( Fig . 1 , left ) , it leads to a drastic slow-down when the problem is ill-conditioned ( Fig . 1 , 1This algorithm is a simplified TR method , initially tailored for non-linear least squares problems ( Nocedal & Wright , 2006 ) 2We only plot every other trust region . Since the models are linear , the minimizer is always on the boundary . center ) . Particularly , once the method has reached the bottom of the valley , it struggles to make progress along the horizontal axis . Here is precisely where the advantage of adaptive stepsize methods comes into play . As illustrated by the dashed lines , Adagrad ’ s search space is damped along the direction of high curvature ( vertical axis ) and elongated along the low curvature direction ( horizontal axis ) . This allows the method to move further horizontally early on to enter the valley with a smaller distance to the optimizer w∗ along the low curvature direction which accelerates convergence . Let us now formally establish the result that allows us to re-interpret adaptive gradient methods from the trust region perspective introduced above . Lemma 1 ( Preconditioned gradient methods as TR ) . A preconditioned gradient step wt+1 −wt = st : = −ηtA−1t gt ( 4 ) with stepsize ηt > 0 , symmetric positive definite preconditioner At ∈ Rd×d and gt 6= 0 minimizes a first-order model around wt ∈ Rd in an ellipsoid given by At in the sense that st : = arg min s∈Rd [ m1t ( s ) = L ( wt ) + sᵀgt ] , s.t . ‖s‖At ≤ ηt‖gt‖A−1t . ( 5 ) Corollary 1 ( Rmsprop ) . The step srms , t : = −ηtA−1/2rms , tgt minimizes a first-order Taylor model around wt in an ellipsoid given by A 1/2 rms , t ( Eq . 3 ) in the sense that srms , t : = arg min s∈Rd [ m1t ( s ) = L ( wt ) + sᵀgt ] , s.t . ‖s‖ A 1/2 rms , t ≤ ηt‖gt‖A−1/2rms , t . ( 6 ) Equivalent results can be established for Adam using gadam , t : = ( 1−β ) ∑t k=0 β t−kgt as well as for Adagrad by replacing the matrix Aada into the constraint in Eq . ( 6 ) . Of course , the update procedure in Eq . ( 5 ) is merely a reinterpretation of the original preconditioned update , and thus the employed trust region radii are defined implicitly by the current gradient and stepsize .

Summary: Authors propose a new perspective on adaptive gradient methods. Main contribution is a trust region based algorithm they call "Stochastic Ellipsoidal Trust Region Method" thats flexible to include both full, and diagonal matrix as the preconditioning matrix.  Authors also mention that the preconditioners are generally diagonally dominant in practice, and may only require diagonal matrix (leaves full matrix for future work).