A fast quasi-Newton-type method for large-scale stochastic optimisation

1 INTRODUCTION . In learning algorithms we often face the classical and hard problem of stochastic optimisation where we need to minimise some non-convex cost function f ( x ) min x∈Rd f ( x ) , ( 1 ) when we only have access to noisy evaluations of the function and its gradients . We take a particular interest in situations where the number of data and/or the number of unknowns d is very large . The importance of this problem has been increasing for quite some time now . The reason is simple : many important applied problems ask for its solution , including most of the supervised machine learning algorithms when applied to large-scale settings . There are two important situations where the non-convex stochastic optimisation problem arise . Firstly , for large-scale problems it is often prohibitive to evaluate the cost function and its gradient on the entire dataset . Instead , it is divided into several mini-batches via subsampling , making the problem stochastic . This situation arise in most applications of deep learning . Secondly , when randomised algorithms are used to approximately compute the cost function and its gradients the result is always stochastic . Our contributions are : 1 . A new stochastic line search algorithm allowing for adaptive step-lengths akin to what is done in the corresponding state-of-the-art deterministic optimisation algorithms . This is enabled via a stochastic formulation of the first Wolfe condition . 2 . We provide a new and efficient way of incorporating second-order ( curvature ) information into the stochastic optimiser via a direct least-squares approach conceptually similar to the popular limited memory quasi-Newton methods . 3 . To facilitate a fast and numerically robust implementation we have derived tailored updating of a small dimension Cholesky factor given the new measurement pair ( with dimension equal to the memory length ) . 4 . We support our new developments by establishing several theoretical properties of the resulting algorithm . The performance is also demonstrated on real-world benchmark problems which shows improved convergence properties over current state-of-the-art methods . 2 RELATED WORK . Due to its importance , the stochastic optimisation problem is rather well studied by now . The first stochastic optimisation algorithm was introduced by Robbins & Monro ( 1951 ) . It makes use of first- order information only , motivating the name stochastic gradient ( SG ) , which is the contemporary term ( Bottou et al. , 2018 ) for these algorithms , originally referred to as stochastic approximation . Interestingly most SG algorithms are not descent methods since the stochastic nature of the update can easily produce a new iterate corresponding to an increase in the cost function . Instead , they are Markov chain methods in that their update rule defines a Markov chain . The basic first-order SG algorithms have recently been significantly improved by the introduction of various noise reduction techniques , see e.g . ( Johnson & Zhang , 2013 ; Schmidt et al. , 2013 ; Konečný & Richtárik , 2017 ; Defazio et al. , 2014 ) . The well-known drawback of all first-order methods is the lack of curvature information . Analogously to the deterministic setting , there is a lot to be gained in extracting and using second-order information that is maintained in the form of the Hessian matrix . The standard quasi-Newton method is the BFGS method , named after its inventors ( Broyden , 1967 ; Fletcher , 1970 ; Goldfarb , 1970 ; Shanno , 1970 ) . In its basic form , this algorithm does not scale to the large-scale settings we are interested in . The idea of only making use of the most recent iterates and gradients in forming the inverse Hessian approximation was suggested by Nocedal ( 1980 ) and Liu & Nocedal ( 1989 ) . The resulting L-BFGS method is computationally cheaper with a significantly reduced memory footprint . Due to its simplicity and good performance , this has become one of the most commonly used second-order methods for large-scale problems . Our developments makes use of the same trick underlying L-BFGS , but it is carefully tailored to the stochastic setting . Over the past decade we have witnessed increasing capabilities of these so-called stochastic quasiNewton methods , the category to which our developments belong . The work by Schraudolph et al . ( 2007 ) developed modifications of BFGS and its limited memory version . There has also been a series of papers approximating the inverse Hessian with a diagonal matrix , see e.g . Bordes et al . ( 2009 ) and Duchi et al . ( 2011 ) . The idea of exploiting regularisation together with BFGS was successfully introduced by Mokhtari & Ribeiro ( 2014 ) . Later Mokhtari & Ribeiro ( 2015 ) also developed a stochastic L-BFGS algorithm without regularisation . The idea of replacing the stochastic gradient difference in the BFGS update with a subsampled Hessian-vector product was recently introduced by Byrd et al . ( 2016 ) , and Wang et al . ( 2017 ) derived a damped L-BFGS method . Over the past five years we have also seen quite a lot of fruitful activity in combining the stochastic quasi-Newton algorithms with various first-order noise reduction methods ( Moritz et al. , 2016 ; Gower et al. , 2016 ) . A thorough and forward-looking overview of SG and its use within a modern machine learning context is provided by Bottou et al . ( 2018 ) . It also includes interesting accounts of possible improvements along the lines of first-order noise reduction and second-order methods . It is interesting—and perhaps somewhat surprising—to note that it is only very recently that stochastic line search algorithms have started to become available . One nice example is the approach proposed by Mahsereci & Hennig ( 2017 ) which uses the framework of Gaussian processes and Bayesian optimisation . The step length is chosen that best satisfies a probabilistic measure combining reduction in the cost function with satisfaction of the Wolfe conditions . Conceptually more similar to our procedure is the line search proposed by Bollapragada et al . ( 2018 ) , which is tailored for problems that are using sampled mini-batches , as is common practice within deep learning . 3 STOCHASTIC LINE SEARCH . In deterministic line search algorithms we first compute a search direction pk and then decide how far to move along that direction according to xk+1 = xk + αkpk , ( 2 ) where αk > 0 is referred to as the step length . The search direction is of the form pk = −Hkgk , ( 3 ) where Hk denotes an approximation of the inverse Hessian matrix . The question of how far to move in this direction can be framed as the following scalar minimisation problem min α f ( xk + αpk ) , α > 0 . ( 4 ) The most common way of dealing with this problem is to settle for a possibly sup-optimal solution that guarantees at least a sufficient decrease . Such at solution can be obtained by selecting a step length αk that satisfies the following inequality f ( xk + αpk ) ≤ f ( xk ) + cαkgTk pk , ( 5 ) where c ∈ ( 0 , 1 ) . The above condition is known as the first Wolfe condition ( Wolfe , 1969 ; 1971 ) or the Armijo condition ( Armijo , 1966 ) . While deterministic line search algorithms have been well established for a long time , their stochastic counterparts are still to a large extent missing . Consider the case when the measurements of the function and its gradient are given by f̂k = f ( xk ) + ek , ĝk = gk + vk , ( 6 ) where gk , ∇f ( x ) |x=xk , and ek ∈ R and vk ∈ Rd×1 denote noise on the function and gradient evaluations , respectively . Furthermore we assume that E [ ek ] = b , Cov [ ek ] = σ 2 f , E [ vk ] = 0 , Cov [ vk ] = σ 2 gI . ( 7a ) The challenge is that since f̂k and ĝk are random variables it is not obvious how to select a step length αk that—in some sense—guarantees a descent direction . We explore the idea of requiring equation 5 to be fulfilled in expectation when the exact quantities are replaced with their stochastic counterparts , E [ f̂ ( xk + αp̂k ) − f̂ ( xk ) − cαkĝTk p̂k ] ≤ 0 , ( 8 ) where p̂k = −Hkĝk . This is certainly one way in which we can reason about the Wolfe condition in the stochastic setting we are interested in . Although satisfaction of equation 8 does not leave any guarantees when considering a single measurement , it still serves as an important property that could be exploited to provide robustness for the entire optimisation procedure . To motivate our proposed algorithm we hence start by establishing the following results . lemma 1 ( Stochastic Wolfe condition 1 ) Assume that i ) the gradient estimates are unbiased , ii ) the cost function estimates are possibly biased , and iii ) a descent direction is ensured in expectation E [ p̂Tk ĝk ] < 0 . Then ( for small α : s ) E [ f̂ ( xk + αp̂k ) ] ≤ E [ f̂ ( xk ) + cαkĝ T k p̂k ] , where 0 < c < c̄ = pTkgk pTkgk − σ2g Tr ( H ) . ( 9 ) Proof 1 See Appendix A.1 . lemma 2 There exists a step length αk > 0 such that E [ f̂ ( xk + αkp̂k ) − f̂ ( xk ) ] < 0 . Proof 2 See Appendix A.2 Relying upon these results we propose a line search based on the idea of repeatedly decreasing the step length αk until the stochastic version of the first Wolfe condition is satisfied . Pseudo-code for this procedure is given in Algorithm 1 . An input to this algorithm is the search direction p̂k , which can be computed using any preferred method . The step length is initially set to be the minimum of the `` natural '' step length 1 and the iteration dependent value ξ/k . In this way the initial step length is kept at 1 until k > ξ , a point after which it is decreased at the rate 1/k . Then we check whether the new point xk + αp̂k satisfies the stochastic version of the first Wolfe condition . If this is not the case , we decrease αk with the scale factor ρ . This is repeated until the condition is met , unless we hit an upper bound max { 0 , τ − k } on the number of backtracking iterations , where τ > k is a positive integer . With this restriction the decrease of the step length is limited , and when k ≥ τ we use the initial step length no matter if the Wolfe condition is satisfied or not . The purpose of ξ is to facilitate convergence by reducing the step length as the minima is approached . The motivation of τ comes from arguing that the backtracking loop does not contribute as much when the step length is small , and hence it is reasonable to provide a limit on its reduction . Algorithm 1 Stochastic backtracking line search Require : Iteration index k , spatial point xk , search direction p̂k , scale factor ρ ∈ ( 0 , 1 ) , reduction limit ξ ≥ 1 , backtracking limit τ > 0 . 1 : Set the initial step length αk = min { 1 , ξ/k } 2 : Set i = 1 3 : while f̂ ( xk + αp̂k ) > f̂ ( xk ) + cαkĝTk p̂k and i ≤ max { 0 , τ − k } do 4 : Reduce the step length αk ← ραk 5 : Set i← i+ 1 6 : end while 7 : Update xk+1 = xk + αkp̂k

Summary: The authors present an interesting variation of the standard QN methods. Their main point of departure from LBFGS/SR1 is in constructing a simpler Hessian inverse approximation. Recall that SR1 and LBFGS updates all satisfy the secant equation for each of the `m` previous gradient differences stored in memory. The authors choose to get "close" to satisfying the equations by solving an l_2 penalization of the secant equations.