Learning the Step-size Policy for the Limited-Memory Broyden-Fletcher-Goldfarb-Shanno Algorithm

1 INTRODUCTION . Consider the unconstrained optimization problem minimize x f ( x ) ( 1 ) where f : Rn → R is an objective function that is differentiable for all x ∈ Rn , with n being the number of decision variables forming x . Let ∇xf ( x0 ) be the gradient of f ( x ) evaluated at some x0 ∈ Rn . A general quasi-Newton algorithm for solving this problem iterates xk+1 = xk − tkHkgk ( 2 ) for an initial x0 ∈ Rn until a given stop criterion is met . At the k-th iteration , gk = ∇xf ( xk ) is the gradient , Hk is a positive-definite matrix satisfying the secant equation ( Nocedal and Wright , 2006 , p. 137 ) and tk is the step size . In this paper , we develop a policy that learns to suitably determine step sizes tk when the product Hkgk is calculated by the Limited-Memory Broyden–Fletcher–Goldfarb–Shanno ( L-BFGS ) algorithm ( Liu and Nocedal , 1989 ) . The main contributions of the paper are : 1 . We propose a neural network architecture defining this policy taking as input local information of the current iterate . In contrast with more standard strategies , this policy is tuning-free and avoids re-evaluations of the objective function and gradients at each step . The training procedure is formulated as a stochastic optimization problem and can be performed by easily applying backpropagation through time ( TBPTT ) . 2 . Training classifiers in the MNIST database ( LeCun et al. , 1998 ) , our approach is competitive against heuristically tuned optimization procedures . Our tests show that the proposed policy is not only able to outperform competitors such as ADAM and RMSprop in wall-clock time and optimal/final value , but also performs better than L-BFGS with backtracking line searches , which is the gold standard , and with constant step sizes , which is the baseline . 3 . According to subsequent experiments on CIFAR-10 ( Krizhevsky et al. , 2009 ) , the proposed policy can generalize to different classes of problems after a few additional training steps on examples from these classes . This indicates that learning may be transferable between distinct types of tasks , allowing to explore transfer learning strategies . This result is a step towards the development of optimization methods that frees the designer from tuning control parameters as it will be motivated in Section 2 . The remaining parts of this paper are organized as follows : Section 3 presents the classical L-BFGS algorithm and discuss some methodologies to determine step sizes ; Section 4 contains the architecture for the proposed policy and also discussions on how it was implemented ; Section 5 describes the training procedure ; and , finally , Section 6 presents experiments using classifiers to operate on MNIST and CIFAR-10 databases . The notation is mainly standard . Scalars are plain lower-case letters , vectors are bold lower-case , and matrices are bold upper-case . The clip function is defined as clipul ( y ) : = min ( u , max ( l , y ) ) . 2 MOTIVATION . Most algorithms used in artificial intelligence and statistics are based on optimization theory , which has widely collaborated for the success of machine learning applications in the last decades . However , this two-way bridge seems not to be currently leveraging its full potential in the other sense , that is , to learn how to automate optimization procedures . Indeed , performing satisfactory optimization , or solving learning problems , still relies upon the appropriate tuning of parameters of the chosen algorithm , which are often grouped with other hyper-parameters of the learning task . Despite the existence of several methodologies to obtain good values for these parameters ( Bengio , 2000 ; Bergstra et al. , 2011 ; Bergstra and Bengio , 2012 ; Snoek et al. , 2015 ; Daniel et al. , 2016 ; Dong et al. , 2018 ) , the search for tuning-free algorithms that perform better than heuristically designed ones is of great interest among practitioner and theoreticians . Indeed , besides the generally-desirable faster convergence , the ready-to-use nature of such algorithms allows the user to focus his attention on other problem-level hyper-parameters while the optimization procedure is automatically performed , resulting in better time and effort allocation . As recent advancements of machine learning have helped automatize the solution of numberless problems , optimization theory should equally benefit from these , balancing the bridge flows . From a wider viewpoint , most optimization problem requires the user to select an algorithm and tune it to some extent . Although intuition and knowledge about the problem can speed-up these processes , trial-and-error methodologies are often employed which can be a time-consuming and inefficient task . With that in mind , the concept of Learned optimizers has been gathering attention in the last few years and , basically , refers to optimization policies and routines that were learned by looking at instances of optimization problems , here called tasks . This idea was introduced by Li and Malik ( 2016 ) and Andrychowicz et al . ( 2016 ) building upon previous results of “ learning to learn ” or “ meta-learning ” ( Thrun and Pratt , 1998 ; Hochreiter et al. , 2001 ) . In the former , the authors presented an optimization policy based on a neural network trained by reinforcement learning and taking as input the history of gradient vectors at previous iterations . The latter adopts a long short-term memory ( LSTM ) to achieve a similar task , but the learning is done by truncated backpropagation through time after unrolling the proposed optimizer for a certain number of steps . Subsequently , it was shown in Metz et al . ( 2019 ) how multilayer perceptrons ( MLP ) , adequately trained using a combined gradient estimation method , can perform faster in wall-clock time compared to current algorithms of choice . Also within this scenario , in Xu et al . ( 2019 ) a reinforcement learning-based methodology to auto-learn an adaptive learning rate is presented . Following this same fashion , in this present paper , instead of completely learning an optimizer from data , we propose a mixture of these ideas into a classical optimization procedure . Thus , the resulting optimizer , composed by a combination of L-BFGS and the proposed policy , will be learned in a constrained domain that assures valuable mathematical properties . The idea is to leverage both frameworks , inheriting the theoretical aspects assured by optimization theory while learning a policy to rule out the hand-design of parameters . Algorithm 1 : L-BFGS algorithm Input : si = xi+1 − xi , yi = gi+1 − gi and ρi = 1/ ( sTi yi ) for all i ∈ k −m , . . . , k − 1 ; and current gradient gk , Result : update direction dk = −Hkgk 1 q ← gk ; 2 for i = k − 1 , . . . , k −m do 3 αi ← ρisTi q ; 4 q ← q − αiyi ; 5 end 6 γ = |sTk−1yk−1|/ ( yTk−1yk−1 ) ; 7 r ← γq ; 8 for i = k −m , . . . , k − 1 do 9 β ← ρiyTi r ; 10 r ← r + si ( αi − β ) ; 11 end 12 dk ← −r ; 3 L-BFGS ALGORITHM . The L-BFGS algorithm was originally presented in Liu and Nocedal ( 1989 ) and is here transcribed into Algorithm 1 . It is a quasi-Newton method derived from the BFGS algorithm ( Nocedal and Wright , 2006 ) lowering space complexity from quadratic to linear in the problem dimension at the expense of precision . This algorithm calculates a descending direction in the search space taking into account an estimate of the inverse hessian matrix of f ( x ) , given by Hk . This matrix is not explicitly constructed but rather the product dk : = −Hkgk is obtained from the past m values of xk and gk , which have to be stored . This property makes it often the algorithm of choice for large-scale deterministic non-linear optimization problems . If f ( x ) is convex in x , this algorithm is guaranteed to provide a descending update direction , but the same does not apply for non-convex objective functions . However , a simple way to circumvent this is by removing iterations i in lines 2 and 8 of Algorithm 1 such that ρi ≤ 0 ( Nocedal and Wright , 2006 , p. 537 ) , which is used in this paper . A matter of great relevance within this scope is how to choose an appropriate step size tk to apply the update rule in Eq . ( 2 ) . To the best of our knowledge , there does not seem to exist a consensus on how to choose tk in a general way for non-convex objective functions . The scaling factor γ in lines 6-7 of Algorithm 1 is known to assure that the step size tk = 1 is accepted in most iterations in the convex optimization context , but not always . We will refer to a constant step-size policy that outputs tk = 1 as the baseline L-BFGS . However , a line search ( LS ) procedure is often combined with L-BFGS to assure its convergence . Ideally , this should be performed by solving tk = arg mint > 0 f ( xk + tdk ) but this exact approach is often too expensive to be adopted , motivating the use of inexact ones . An example is the backtracking line search ( BTLS ) , which takes an initial length tk for the step size and shrinks it repeatedly until the so-called sufficient decrease Wolfe Condition f ( xk + tkdk ) ≤ f ( xk ) + c1tkg T k dk is fulfilled , where c1 ∈ ( 0 , 1 ) is a control parameter to be tuned . Another parameter that has to be designed is the contraction factor c2 ∈ ( 0 , 1 ) that shrinks the step size , i.e. , tk ← c2tk , see Nocedal and Wright ( 2006 , p. 37 ) . This method assures convergence to a localminima at the cost of re-evaluating the objective function several times per iteration . This is a price that the user is , in some cases , willing to pay , but for large-dimensional problems this procedure is likely to become the bottle-neck of the optimization task . It is important to highlight that the method to be presented may also apply to other optimization algorithms that deeply rely on line searches to perform well . However , this paper focus on L-BFGS as it is often the algorithm of choice in large-scale deterministic optimization . In the context of stochastic optimization , many modified versions of Algorithm 1 together with methodologies for choosing tk are available ( Moritz et al. , 2016 ; Zhou et al. , 2017 ; Bollapragada et al. , 2018 ; Wills and Schön , 2019 ) , but for sake of simplicity , our work will deal exclusively with deterministic non-linear optimization problems . 4 LEARNED POLICY FOR SELECTING STEP SIZES . Recalling the definition of sk and yk in Algorithm 1 , our policy is defined as tk = π ( dk , gk , sk−1 , yk−1 ; θ ) and selects an adequate step size for L-BFGS but neither relying on any parameter tuning nor requiring additional evaluations of the objective function . Instead , its parameters that are represented by θ should be learned from data . Let us , from now on , de- fine this policy combined with Algorithm 1 as the L-BFGS-π approach . The architecture of the policy π ( dk , gk , sk−1 , yk−1 ; θ ) is shown in Fig . 1 . To allow the policy to be independent from the problem size n , only the inner products between its inputs are used . These values define u0 = dotln ( dk , gk , sk−1 , yk−1 ) where dotln ( · ) returns the component-wise application of f ( x ) = ln ( min ( x , ) ) to the elements of X = [ dk gk sk−1 yk−1 ] T [ dk gk sk−1 yk−1 ] but with the superdiagonal entries having their signs reversed . We have chosen = 10−8 to avoid imaginaryvalued entries . The vector u0 is the input to two parallel input layers , which are fully connected linear layers that transport information in u0 to another vector space Rnh ( in our tests , we adopted nh = 6 ) . Their outputs , as usual , are defined as u1 = W01u0 + b01 and u2 = W02u0 + b02 . The logarithm operation was adopted to let the linear layers evaluate products and divisions between powers of the inputs by simply summing and subtracting them . Moreover , as the output is positive , working in the logarithmic vector space allows us to use a wider range of numerical values . Subsequently , let us define the normalized vectors ū1 = u1/‖u2‖ and ū2 = u2/‖u2‖ to calculate the scalar projection of ū1 onto ū2 and clip the result to some interval [ τm , τM ] , yielding the log-step size τk = clip τM τm ( ūT2 ū1 ) = : p ( u1 , u2 ) ( 3 ) Finally , the selected step size is obtained as tk = eτk . To geometrically interpret this , we sketch three different scenarios in Fig . 2 . The dashed lines represent orthogonal axes spanned by some arbitrary ū2 and the gray strip represents the interval [ τm , τM ] along the direction of ū2 whence τk should be taken . When the Linear Layer 1 maps u0 into u′1 , the scalar projection of ū ′ 1 onto ū2 is beyond the maximal τM , so τk is clipped to it . In the same way , for ū′′′1 the step size will be the minimal one tk = eτm whereas for the intermediate ū′′1 we have τk ∈ ( τm , τM ) . The two layers , jointly trained , should learn how to position ū1 and ū2 in the lifted space to represent important directional information of dk and gk by looking at similar optimization tasks , being thus able to produced suitable step sizes . ū2 ū′1 ū′′′1 ū′′1 [ τm , τM ] for example , in the sufficient decrease Wolfe condition for backtracking line search , which makes our policy comparable to them in the sense that π ( · ; θ ) does not require additional information to operate . However , notice that the clip function is not suitable for training given that it is non-differentiable and gradients can not be backpropagated through it . Fortunately , the clip operation ( 3 ) can be cast as a convex optimization problem τk = arg min τ∈R ‖u2τ − u1‖2 ( 4 ) s.t . τm ≤ τ ≤ τM ( 5 ) allowing τk to be calculated by a convex optimization layer , defined here as a CVX Layer , ( Agrawal et al. , 2019 ) . This last layer can output the solution to a parameter-dependent convex optimization problem . For the special case where a solution is not differentiable with respect to the input ( e.g. , in our case when an inequality constraint is active ) , the automatic differentiation procedure delivers an heuristic quantity that can be employed as a gradient . The use of a CVX Layer is therefore convenient for training our policy but , on the other hand , using Eq . ( 3 ) in its place when applying the already-trained policy significantly speeds up the step-size evaluation , compared to solving ( 4 ) . It is important to remark that this policy is defined as independent from both the memory length m of Algorithm 1 and the problem dimension n. Additionally , the lower and upper limits for the log-step size are τm and τM , respectively , and can also be learned . In this work , however , we chose τm = −3 and τM = 0 , letting tk ∈ [ 0.0497 , 1 ] . This interval is comprehensive enough to let our method be compared in a fair way to backtracking line searches . Moreover , when we allowed τM to be learned in our tests it converged to values that were very close to τM = 0 , indicating that 1 was already an adequate upper limit for the step size .

Summary: The paper studies a problem of learning step-size policy for L-BFGS algorithm. This paper falls into a general category of meta-learning algorithms that try to derive a data-driven approach to learn one of the parameters of the learning algorithm. In this case, it is the learning rate of L-BFGS. The paper is very similar in nature to the papers of Ravi & Larochelle, MAML and Andrychowicz.