Ellipsoidal Trust Region Methods for Neural Network Training

1 INTRODUCTION . We consider finite-sum optimization problems of the form min w∈Rd [ L ( w ) : = n∑ i=1 ` ( f ( w , xi , yi ) ) ] , ( 1 ) which typically arise in neural network training , e.g . for empirical risk minimization over a set of data points ( xi , yi ) ∈ Rin × Rout , i = 1 , . . . , n. Here , ` : Rout × Rout → R+ is a convex loss function and f : Rin × Rd → Rout represents the neural network mapping parameterized by the concatenation of layers w ∈ Rd , which is non-convex due to its multiplicative nature and potentially non-linear activation functions . We assume that L is lower bounded and twice differentiable , i.e . L ∈ C2 ( Rd , R ) and consider finding a first- and second-order stationary point w̄ for which ‖∇L ( w̄ ) ‖ ≤ g and λmin ( ∇2L ( w̄ ) ) ≥ − H . Non-convex optimization problems are ubiquitous in machine learning . Among the most prominent examples are present-day deep neural networks , that have achieved outstanding results on core tasks such as collaborative filtering ( Wang et al. , 2015 ) , sentence classification ( Kim , 2014 ) and image classification ( Krizhevsky et al. , 2012 ) . In the era of big data and deep neural networks , stochastic gradient descent ( SGD ) is one of the most widely used training algorithms ( Bottou , 2010 ) . What makes SGD so attractive is its simplicity and per-iteration cost that are independent of the size of the training set ( n ) and scale linearly in the dimensionality ( d ) . However , gradient descent is known to be inadequate to optimize functions that are ill-conditioned ( Nesterov , 2013 ; Shalev-Shwartz et al. , 2017 ) and thus adaptive gradient methods that employ dynamic , coordinate-wise learning rates based on past gradients—including Adagrad ( Duchi et al. , 2011 ) , RMSprop ( Tieleman & Hinton , 2012 ) and Adam ( Kingma & Ba , 2014 ) —have become a popular alternative , often providing significant speed-ups over SGD . Yet , there exist no theoretical proofs that these methods are faster than gradient descent ( Li & Orabona , 2018 ) . From a theoretical perspective , Newton methods provide stronger convergence guarantees by appropriately transforming the gradient in ill-conditioned regions according to second-order derivatives . It is precisely this Hessian information that allows regularized Newton methods to enjoy superlinear local convergence as well as to provably escape saddle points ( Conn et al. , 2000 ) . While second-order algorithms have a long-standing history even in the realm of neural network training ( Hagan & Menhaj , 1994 ; Becker et al. , 1988 ) , they were mostly considered as too computationally and memory expensive for practical applications . Yet , the seminal work of Martens ( 2010 ) renewed interest for their use in deep learning by proposing efficient Hessian-free methods that only access second-order information via matrix-vector products which can be computed at the cost of an additional backpropagation ( Pearlmutter , 1994 ; Schraudolph , 2002 ) . Among the class of regularized Newton methods , trust region ( Conn et al. , 2000 ) and cubic regularization algorithms ( Cartis et al. , 2011 ) are the most principled approaches in the sense that they yield the strongest convergence guarantees . Recently , stochastic extensions have emerged ( Xu et al. , 2017b ; Yao et al. , 2018 ; Kohler & Lucchi , 2017 ; Gratton et al. , 2017 ) , which suggest their applicability for deep learning . We here propose a simple modification to make TR methods even more suitable for neural network training . Particularly , we build upon the following alternative view on adaptive gradient methods : While gradient descent can be interpreted as a spherically constrained first-order TR method , preconditioned gradient methods—such as Adagrad—can be seen as first-order TR methods with ellipsoidal trust region constraint . This observation is particularly interesting since spherical constraints are blind to the underlying geometry of the problem , but ellipsoids can adapt to local landscape characteristics , thereby allowing for more suitable steps in regions that are ill-conditioned . We will leverage this analogy and investigate the use of the Adagrad and RMSProp preconditioning matrices as ellipsoidal trust region shapes within a stochastic second-order TR algorithm ( Xu et al. , 2017a ; Yao et al. , 2018 ) . Since no ellipsoid fits all objective functions , our main contribution lies in the identification of adequate matrix-induced constraints that lead to provable convergence and significant practical speed-ups for the specific case of deep learning . On the whole , our contribution is threefold : • We provide a new perspective on adaptive gradient methods that contributes to a better understanding of their inner-workings . Furthermore , we empirically find that many neural network problems exhibit diagonally dominated Hessian matrices which suggests the effectivity of diagonal preconditioning . ( Section 3 ) • We investigate the first application of ellipsoidal TR methods for deep learning . In Theorem 1 we show that the RMSProp matrix can directly be applied as constraint inducing norm in second-order TR algorithms while preserving all convergence guarantees ( Theorem 2 ) . • Finally , we provide an experimental benchmark across different real-world datasets and architectures . We also compare against adaptive gradient methods and show results in terms of backprogations , epochs , and wall-clock time ; a comparison we were not able to find in the literature . ( Section 5 ) Our main empirical results demonstrate that ellipsoidal constraints prove to be a very effective modification of the trust region method in the sense that they constantly outperform the spherical TR method , both in terms of number of backprogations and asymptotic loss value on a variety of tasks . 2 RELATED WORK . First-order methods The prototypical method for optimizing Eq . ( 1 ) is SGD ( Robbins & Monro , 1951 ) . While the practical success of SGD in non-convex optimization is unquestioned , the theoretical foundation of this phenomenon is still rather limited . Recent findings suggest the ability of this method to escape saddle points and reach local minima in polynomial time for general non-convex problems , but they either need to artificially add noise to the iterates ( Ge et al. , 2015 ; Lee et al. , 2016 ) or make an assumption on the inherent noise of vanilla SGD ( Daneshmand et al. , 2018 ) . For neural network training , a recent line of research proclaims the effectiveness of SGD , but the results usually come at the cost of fairly strong assumptions such as heavy overparametrization and Gaussian inputs ( Du et al. , 2017 ; Brutzkus & Globerson , 2017 ; Li & Yuan , 2017 ; Du & Lee , 2018 ; Allen-Zhu et al. , 2018 ) . Adaptive gradient methods ( Duchi et al. , 2011 ; Tieleman & Hinton , 2012 ; Kingma & Ba , 2014 ) build on the intuition that larger learning rates for smaller gradient components and smaller learning rates for larger gradient components balance their respective influences and thereby make the methods behave as if they were optimizing a more isotropic surface . Such approaches have first been suggested for neural networks by LeCun et al . ( 2012 ) . Recently , convergence guarantees for such methods are starting to appear ( Ward et al. , 2018 ; Li & Orabona , 2018 ) . However , these are not superior to the O ( −2g ) worst-case complexity of standard gradient descent ( Cartis et al. , 2012b ) . Regularized Newton methods The most principled class of regularized Newton methods are trust region ( TR ) and adaptive cubic regularization algorithms ( ARC ) ( Conn et al. , 2000 ; Cartis et al. , 2011 ) , which repeatedly optimize a local Taylor model of the objective while making sure that the step does not travel too far such that the model stays accurate . While the former finds first-order stationary points within O ( −2g ) , ARC only takes at most O ( −3/2 g ) . However , simple modifications to the TR framework allow these methods to obtain the same accelerated rate ( Curtis et al. , 2017 ) . Both methods take at most O ( −3H ) iterations to find an H approximate second-order stationary point ( Cartis et al. , 2012a ) . These rates are optimal for second-order Lipschitz continuous functions ( Carmon et al. , 2017 ; Cartis et al. , 2012a ) and they can be retained even when only sub-sampled gradient and Hessian information is used ( Kohler & Lucchi , 2017 ; Yao et al. , 2018 ; Xu et al. , 2017b ; Blanchet et al. , 2016 ; Liu et al. , 2018 ; Cartis & Scheinberg , 2017 ) . Furthermore , the involved Hessian information can be computed solely based on Hessian-vector products , which are implementable efficiently for neural networks ( Pearlmutter , 1994 ) . This makes these methods particularly attractive for deep learning , but the empirical evidence of their applicability is so far very limited . We are only aware of the works of Liu et al . ( 2018 ) and Xu et al . ( 2017a ) , which report promising first results but these are by no means fully encompassing . Gauss-Newton methods An interesting line of research proposes to replace the Hessian by ( approximations of ) the generalized-Gauss-Newton matrix ( GGN ) within a Levenberg-Marquardt framework1 ( LeCun et al. , 2012 ; Martens , 2010 ; Martens & Grosse , 2015 ) . These methods have been termed hessian-free since only access to GGN-vector products is required . As the GGN matrix is always positive semidefinite , they can not leverage negative curvature to escape saddles and hence , there exist no second-order convergence guarantees . Furthermore , there are cases in neural network training where the Hessian is better conditioned than the GGN matrix ( Mizutani & Dreyfus , 2008 ) . Nevertheless , the above works report promising preliminary results , most notably Grosse & Martens ( 2016 ) report that K-FAC can be faster than SGD on a small convnet . On the other hand , recent findings report performance at best comparable to SGD on the much larger ResNet architecture ( Ma et al. , 2019 ) . Moreover , Xu et al . ( 2017a ) reports many cases where TR and GGN algorithms perform similarly . This line of work is to be seen as complementary to our approach since it is straight forward to replace the Hessian in the TR framework with the GGN matrix . Furthermore , the preconditioners used in Martens ( 2010 ) and Chapelle & Erhan ( 2011 ) , namely diagonal estimates of the empirical Fisher and Fisher matrix , respectively , can directly be used as matrix norms in our ellipsoidal TR framework . 3 AN ALTERNATIVE VIEW ON ADAPTIVE GRADIENT METHODS . Adaptively preconditioned gradient methods update iterates as wt+1 = wt − ηtA−1/2t gt , where gt is a stochastic estimate of ∇L ( wt ) and At is a positive definite symmetric pre-conditioning matrix . In Adagrad , Aada , t is the un-centered second moment matrix of the past gradients computed as Aada , t : = GtG ᵀ t + I , ( 2 ) where > 0 , I is the d× d identity matrix and Gt = [ g1 , g2 , . . . , gt ] . Building up on the intuition that past gradients might become obsolete in quickly changing non-convex landscapes , RMSprop ( and Adam ) introduce an exponential weight decay leading to the preconditioning matrix Arms , t : = ( ( 1− β ) Gt diag ( βt , . . . , β0 ) Gᵀt ) + I , ( 3 ) where β ∈ ( 0 , 1 ) . In order to save computational efforts , the diagonal versions diag ( Aada ) and diag ( Arms ) are more commonly applied in practice , which in turn gives rise to coordinate-wise adaptive stepsizes that are enlarged ( reduced ) in coordinates that have seen past gradient components with a smaller ( larger ) magnitude . In that way , the optimization methods can account for gradients of potentially different scales arising from e.g . different layers of the networks .

Summary: In this paper, the authors investigate the use of ellipsoidal trust region constraints for second order optimization. The authors first show that adaptive gradient methods can be viewed as first-order trust region methods with ellipsoid constraints. The authors then show that the preconditioning matrix of RMSProp and Adam can be used as norm inducing matrices for second order trust region methods. This ellipsoidal trust region method is empirically compared with first order gradient methods and spherical second order trust region methods.