L-SR1 Adaptive Regularization by Cubics for Deep Learning

1 INTRODUCTION . Most deep learning problems involve minimization of the empirical risk of estimation min Θ f ( x ; Θ ) , ( 1 ) where Θ ∈ Rn is the set of weights and f is some scalar-valued loss function . To solve ( 1 ) , various optimization approaches have been implemented , which we describe below . Throughout this paper , we write f ( Θ ) and f ( x ; Θ ) interchangeably . Gradient and adaptive gradient methods are widely used for training deep neural networks ( DNN ) for their computational efficiency . The most common approach is Stochastic Gradient Descent ( SGD ) which , despite its simplicity , performs well over a wide range of applications . However , in a sparse training data setting , SGD performs poorly due to limited training speed ( Luo et al . ( 2019 ) ) . To address this problem , adaptive methods such as AdaGrad ( Duchi et al . ( 2011 ) ) , AdaDelta ( Zeiler ( 2012 ) ) , RMSProp ( Hinton et al . ( 2012 ) ) and Adam ( Kingma & Ba ( 2014 ) ) have been proposed . These methods take the root mean square of the past gradients to influence the current step . Amongst all of these adaptive methods , Adam is arguably the most widely used in a deep learning setting due to it rapid training speed . Newton ’ s method has the potential to exploit curvature information from the second-order derivative ( Hessian ) matrix ( see e.g. , Gould et al . ( 2000 ) ) . Generally , the iterates are defined by Θk+1 = Θk − αk∇2f ( Θk ) −1∇f ( Θk ) , where αk > 0 is a steplength defined by a linesearch criterion ( Nocedal & Wright ( 2006 ) ) . In a DNN setting , we know that the number of parameters ( n ) of the network can be of the order of millions . Thus storing the Hessian which takes O ( n2 ) memory , becomes impractical . In addition , the inversion of the Hessian matrix , which takes O ( n3 ) operations , is also impractical . Even though Newton ’ s method achieves convergence in fewer steps , the method becomes computationally intractable to use on large-scale DNNs . Quasi-Newton methods are alternatives to Newton methods . They compute Hessian approximations , Bk+1 , that satisfy the secant condition given by yk = Bk+1sk , where sk = Θk+1 −Θk and yk = ∇f ( Θk+1 ) − ∇f ( Θk ) . The most commonly used quasi-Newton method , including in the realm of deep learning , is the limited-memory BFGS update , or L-BFGS ( see e.g. , Liu & Nocedal ( 1989 ) ) , where the Hessian approximation is given by Bk+1 = Bk + yky > k y > k sk − BkskskB > k s > k Bksk . ( 2 ) The generic L-BFGS quasi-Newton update scheme is described in Algorithm 1 , and numerous variants of L-BFGS exist ( see Goldfarb et al . ( 2020 ) ; Moritz et al . ( 2016 ) ; Gower et al . ( 2016 ) ) . One advantage of using an L-BFGS update is that the Hessian approximation can be guaranteed to be definite , which is highly suitable in line-search settings because the update sk is guaranteed to be a descent direction , meaning there is some step length along this direction that results in a decrease in the objective function ( see Nocedal & Wright ( 2006 ) , Algorithm 6.1 ) . However , because the L-BFGS update is positive definite , it does not readily detect directions of negative curvature for avoiding saddle points . In contrast , the Symmetric-Rank One ( SR1 ) quasi-Newton update is not guarateed to be positive definite and can result in ascent directions for line-search methods . However , in trust-region settings where indefinite Hessian approximations are an advantage because they capture directions of negative curvature , the limited-memory SR1 ( L-SR1 ) has been shown to outperform L-BFGS in DNNs for classification ( see Erway et al . ( 2020 ) ) . We discuss this in more detail in Section 2 but in the context of Adaptive Regularization using Cubics . Algorithm 1 L-BFGS Quasi-Newton Method with Line Search Require : Initial weights Θ0 , batch size d , learning rate α , dataset D , loss function f ( Θ ) . for k = 0 , 1 , 2 , . . . do Sample mini-batch of size d : Dk ⊆ D Perform the forward backward pass over the current mini-batch Compute the limited memory approximation Bk using ( 2 ) Compute step sk = αB−1k ∇Θf ( Θk ) , where α is the line-search step length end for 2 L-SR1 ADAPTIVE REGULARIZATION USING CUBICS METHOD . We begin by discussing the L-SR1 update and the adaptive regularizion using cubics methods for large-scale optimization . Unlike the BFGS update ( 2 ) , which is a rank-two update , the SR1 update is a rank-one update , which is given by Bk+1 = Bk + 1 s > k ( yk −Bksk ) ( yk −Bksk ) ( yk −Bksk ) > ( 3 ) ( see Khalfan et al . ( 1993 ) ) . As previously mentioned , Bk+1 in ( 3 ) is not guaranteed to be definite . However , it can be shown that the SR1 matrices can converge to the true Hessian ( see Conn et al . ( 1991 ) for details ) . We note that the pair ( sk , yk ) is accepted only when |s > k ( yk − Bksk ) | > ε‖yk −Bksk‖22 , for some constant ε > 0 ( see Nocedal & Wright ( 2006 ) , Sec . 6.2 , for details ) . The SR1 update can be defined recursively as Bk+1 = B0 + k∑ j=0 1 s > j ( yj −Bjsj ) ( yj −Bjsj ) ( yj −Bjsj ) > . ( 4 ) In limited-memory SR1 ( L-SR1 ) settings , only the last m n pairs of ( sj , yj ) are stored and used . If Sk+1 = [ s0 s1 · · · sk ] and Yk+1 = [ y0 y1 · · · yk ] , then Bk+1 admits a compact representation of the form Bk+1 = B0 + [ Ψk+1 ] [ Mk+1 ] [ Ψ > k+1 ] , ( 5 ) where Ψk+1 = Yk+1−B0Sk+1 and Mk+1 = ( Dk+1+Lk+1+L > k+1−S > k+1B0Sk+1 ) −1 , ( 6 ) where Lk+1 is the strictly lower triangular part , Uk+1 is the strictly upper triangular part , and Dk+1 is the diagonal part of S > k+1Yk+1 = Lk+1 + Dk+1 + Uk+1 ( see Byrd et al . ( 1994 ) for further details ) . Because of the compact representation of Bk+1 , its partial eigendecomposition can be computed ( see Burdakov et al . ( 2017 ) ) . In particular , if we compute the QR decomposition of Ψk+1 , then we can write Bk+1 = B0 = U‖Λ̂k+1U > ‖ , where U‖ ∈ R n× ( k+1 ) has orthonormal columns and Λ̂ ∈ R ( k+1 ) × ( k+1 ) is a diagonal matrix . If B0 = δkI ( see e.g. , Lemma 2.4 in Erway et al . ( 2020 ) ) , where 0 < δk < δmax is some scalar and I is the identity matrix , then we obtain the eigendecomposition Bk+1 = Uk+1Λk+1U > k+1 , where Uk+1 = [ U‖ U⊥ ] , with U⊥ ∈ Rn× ( n− ( k+1 ) ) and U > k+1Uk+1 = I . Here , ( Λk+1 ) i = δk + λ̂i for i ≤ k+ 1 , where λ̂i is the ith diagonal in Λ̂k+1 , and ( Λ ) i = δk for i > k + 1 . Since the SR1 Hessian approximation can be indefinite , some safeguard must be implemented to ensure that the resulting search direction sk is a descent direction . One such safeguard is to use a “ regularization ” term . The Adaptive Regularization using Cubics ( ARCs ) method ( Griewank ( 1981 ) ; Cartis et al . ( 2011 ) ) can be viewed as an alternative to line-search and trust-region methods . At each iteration , an approximate global minimizer of a local ( cubic ) model , min s∈Rn mk ( s ) ≡ g > k s + 1 2 s > Bks + µk 3 ( Φk ( s ) ) 3 , ( 7 ) is determined , where gk = ∇f ( Θk ) , µk > 0 is a regularization parameter , and Φk is a function ( norm ) that regularizes s. Typically , the Euclidean norm is used . In this work , we propose an alternative “ shape-changing ” norm that allows us to solve each subproblem ( 7 ) exactly . This shape-changing norm was proposed in Burdakov et al . ( 2017 ) , and it is based on the partial eigendecomposition of Bk . Specifically , if Bk = UkΛkU > k is the eigendecomposition of Bk , then we can define the norm ‖s‖Uk def = ‖U > k s‖3 . Applying a change of basis with s̄ = U > k s and ḡk = U > k gk , we can redefine the cubic subproblem as min s̄∈Rn m̄k ( s ) = ḡ > k s̄ + 1 2 s̄ > Λks̄ + µk 3 ‖s̄‖33 . ( 8 ) With this change of basis , we can find a closed-form solution of ( 8 ) easily . The proposed Adaptive Regularization using Cubics with L-SR1 ( ARCSLSR1 ) algorithm is given in Algorithm 2 . 2.1 CONTRIBUTIONS . The main contributions of this paper are as follows : 1 . L-SR1 quasi-Newton methods : The most commonly used quasi-Newton approach is the L-BFGS method . In this work , we use the LSR1 update to better model potentially indefinite Hessians of the non-convex loss function . 2 . Adaptive Regularization using Cubics ( ARCs ) : Given that the quasi-Newton approximation is allowed to be indefinite , we use an Adaptive Regularized using Cubics approach to safeguard each search direction . 3 . Shape-changing regularizer : We use a shape-changing norm to define the cubic regularization term , which allows us to compute the closed form solution to the cubic subproblem ( 7 ) . 4 . Computational complexity : Let m be the number of previous iterates and gradients stored in memory . The proposed LSR1 ARC approach is comparable to L-BFGS in terms of storage and compute complexity ( see Table 1 ) . Algorithm 2 Limited-Memory Symmetric Rank-1 Adaptive Regularization using Cubics 1 : Given : Θ0 , γ2 ≥ γ1 , 1 > η2 ≥ η1 > 0 , and σ0 > 0 2 : for k = 0 , 1 , 2 . . . do 3 : Obtain Sk = [ s0 · · · sk ] , Yk = [ y0 · · · yk ] 4 : Solve the generalized eigenproblem S > k Yku = Λ̂S > k Sku and let δk = min { λ̂i } 5 : Compute Ψk = Yk − δkSk 6 : Perform QR decomposition of Ψ = QR 7 : Compute the eigendecomposition RMR > = PΛP > 8 : Assign U‖ = QP and U > ‖ = P > Q > 9 : Define C‖ = diag ( c1 , . . . , cm ) , where ci = 2 λi+ √ λ2i +4µ|ḡi| and ḡ‖ = U > ‖ g 10 : Compute α∗ = 2 δk+ √ δ2k+4µ‖g⊥‖ where g⊥ = g −U‖ḡ‖

Summary: The paper describes a limited memory quasi-Newton method based on SR1 updating using a variant on the adaptive regularized cubic (ARC) approach to globalization.  The algorithm is applied to training deep neural networks for image classification and autoencoding, and compared to L-BFGS and various SGD variants.  The authors claim the main contributions to be the different optimizer ingredients (L-SR1, ARC safeguarding, and the particular shape-changing regularizer) as well as computational complexity similar to L-BFGS.