Accelerating First-Order Optimization Algorithms

1 INTRODUCTION . The majority of existing optimization algorithms are first-order methods — use first derivatives of the function to minimize — , based on the gradient descent technique . These techniques ( e.g . the back propagation of error ) are used to find a matrix of weights that meets the error criterion . Adam ( Adaptive Moment estimation ) ( Kingma & Ba , 2014 ) is probably the most popular ( Goodfellow et al. , 2016 ; Isola et al. , 2017 ) . However , Adam ( while possessing convergence proof issues ) has recently been proven unable to converge to the optimal solution for a simple convex optimization setting ( Reddi et al. , 2018 ) . AMSGrad addresses this issue by endowing Adam and other adaptive stochastic methods with a long-term memory . Unfortunately , Adam can still outperform AMSGrad in some cases 1 . Therefore , instead of looking for the absolute best optimizer , finding a way to accelerate the existent would be of greater use . In this perspective , we propose a simple technique aiming at improving the empirical performance ( find a lower minimum for the loss function ) of any firstorder optimization algorithm , while preserving their property . We will refer to the solution when applied to an existing algorithm A as AA for ( Accelerated Algorithm ) . For example , for AMSGrad and Adam , the modified versions will be mentioned as AAMSGrad and AAdam . The solution is based on the variation of the direction of the current gradient with respect to the direction of the previous update . We conducted several tests on problems where the shape of the loss is simple ( has a convex form ) like Logistic regression , but also with non-trivial neural network architectures such as Multi-layer perceptron , deep Convolutional Neural Networks and Long Short-Term Memory . We used MNIST , CIFAR-10 and IMDB movie reviews Maas et al . ( 2011 ) datasets to validate the solution . Our experiments are limited to SGD , AdaGrad , Adam and AMSGrad . 2 RELATED WORK . 2.1 HOW GRADIENT DESCENT METHOD WORKS ? . Let f ( x ) be a function parameterized by a model ’ s parameters x ∈ Rn , sufficiently differentiable of which one seeks a minimum . The gradient method builds a sequence that should — in principle — work towards finding the minimum . For this , we start from any value x0 ( i.e . a random value ) and 1https : //fdlm.github.io/post/amsgrad/ we construct the recurrent sequence by : xt+1 = xt − α · ∇ft ( xt ) ( 1 ) where α is the learning rate . For adaptive methods like Adam , the learning rate varies for each parameter . This method is ensured to converge even if the input sample is not linearly separable , to a minimum of the error function for a well-chosen learning rate . There exist several variants of this method in two basic categories : first-order and second-order methods . While first-order methods use first derivatives of the function to minimize , second-order methods make use of the estimation of the Hessian matrix ( second derivative matrix of the loss function with respect to its parameters ) ( Schaul et al. , 2013 ; LeCun et al. , 1993 ) . The latter determines the optimal learning rates ( or step size ) to take for solving quadratic problems . While such approach provides additional information useful for optimization , computing accurate second-order derivatives is too computationally expensive for large models . Furthermore , the computed value usually equals to a bad approximation of the Hessian . In this paper , we will only focus on first-order techniques . 2.2 EXISTING FIRST ORDER OPTIMIZATION ALGORITHMS . Adam is a first-order gradient-based algorithm of stochastic objective functions , based on adaptive estimates of lower-order moments . The first moment , once normalized by the second moment , gives the direction of the update . Adam updates are directly estimated using a running average of the first and second moment of the gradient . It computes adaptive learning rates for each parameter . In addition to storing an exponentially decaying average of past squared gradients vt , Adam also keeps an exponentially decaying average of past gradients mt , similar to momentum . It has two main components : a momentum component and an adaptive learning rate component . Adam has a biascorrection feature that helps slightly outperform previous adaptive learning rate methods towards the end of optimization when gradients become sparser ( Ruder , 2016 ) . To our knowledge , Adam is one of the latest state-of-the-art optimization algorithms used by many practitioners of machine learning . There exist several techniques to improve Adam such as fixing the weight decay ( Loshchilov & Hutter , 2017 ) , using the sign of the gradient in distributing learning cases ( Bernstein et al. , 2018 ) , or switching from Adam to SGD ( Keskar & Socher , 2017 ) . However , it has been recently proven that Adam is unable to converge to the optimal solution for a simple convex optimization setting ( Reddi et al. , 2018 ) . A more recent algorithm ( AMSGrad ) fixes this problem by endowing the techniques with a long-term memory . The main difference between AMSGrad and ADAM is that AMSGrad maintains the maximum of all vt ( exponentially decaying average of past squared gradients ) until the present time step and uses this maximum value for normalizing the running average of the gradient . The original paper claims that AMSGrad either performs similarly , or better than Adam on some commonly used problems in machine learning . The update rule of AMSGrad is as follows : xt+1 = xt − α√ v̂t + m̂t mt = β1 ·mt−1 + ( 1− β1 ) · ∇ft ( xt ) vt = β2 · vt−1 + ( 1− β2 ) · ∇ft ( xt ) 2 m̂t = mt 1− βt1 v̂t = max ( v̂t−1 , vt 1− βt2 ) . Recall that an optimizer aims to look for parameters that minimize a function , knowing that we do not have any knowledge on what this function looks like . If we knew the shape of the function to minimize , it would then be easy to take accurate steps that lead to the minimum . While we can ’ t plan for the shape , each time we take a step ( using any of the optimizers ) we can know if we passed a minimum by computing the product of the current and the past gradient , and check whether it is negative . This informs us on the curvature of the loss surface and is — as far as we know — the only accurate information available in real-time . The proposed method exploits this knowledge to improve the convergence of an optimizer . None of the existing techniques uses the variation of the direction of the gradient as information to compute the next update . The technique can be seen as a momentum technique with a condition that specifies when to add momentum . 3 A METHOD TO ACCELERATE FIRST-ORDER OPTIMIZATION ALGORITHMS . 3.1 NOTATION . ‖Mi‖2 denotes the l2-norm of ith row of M. The projection operation ∏ F , A ( y ) for A ∈ Sd+ ( the set of all positive definite d ∗ d matrices ) is defined as arg minx∈F ∥∥A1/2 ( x− y ) ∥∥ for y ∈ Rd . F has bounded diameter D∞ if ‖x− y‖∞ ≤ D∞ forall x , y ∈ F . All operations applied on vectors or matrices are element-wise . 3.2 INTUITION AND PSEUDO-CODE . The intuition behind proposed solution is presented in this section . The pseudo-code of our the method applied to the generic adaptive method setup proposed by Reddi et al . ( 2018 ) is illustrated in algorithm 1 . For Adam , ψt ( g1 , ... , gt ) = ( 1− β2 ) ∑t i=1 β t−i 2 g 2 i . For AMRSgrad , ψt ( g1 , ... , gt ) = ( 1− β2 ) diag ( ∑t i=1 β t−i 2 g 2 i ) . Algorithm 1 Accelerated - Generic Adaptive Method Setup Input : x1 ∈ F , step size ( αt > 0 ) Tt=1 , sequence of functions ( ϕt , ψt ) Tt=1 t← 1 S ← threshold repeat gt =∇ft ( xt ) Vt = ψt ( g1 , ... , gt ) if gt ·mt−1 > 0 and | mt−1 − gt | > S then gmt = gt +mt mt = β1 ·mt−1 + ( 1− β1 ) ġmt else mt = β1 ·mt−1 + ( 1− β1 ) ġt x̂t+1 = xt − αtmtV −1/2t xt+1 = ∏ F , √ Vt ( x̂t+1 ) t← t + 1 until t > T Lets take the famous example of the ball rolling down a hill . If we consider that our objective is to bring that ball ( parameters of our model ) to the lowest possible elevation of the hill ( cost function ) , what the method does is to push the ball with a force higher than the one produced by any optimizer , if the ball is still rolling in the same direction . This is done by adding the sum between the current computed gradient ( gt ) and the previous gradient ( gt−1 ) or the previous moment ( mt−1 ) to the step currently being computed instead of only gt . The ball will gain more speed as it continues to go in the same direction and lose its current speed as soon as it will pass over a minimum . To avoid the oscillation of the ball over a minimum , a constraint was added concerning the difference between the past gradient and the current one . This difference should be more than a threshold ( S < | gt−1−gt | ) in order for the acceleration to be effective when updates became smaller . The higher the threshold , the closer we will get to the original algorithm . However , S should not be too small ( see section 5.2 ) . Another solution could be to stop accelerating the ball and let the original optimizer take full control of the training once the direction changes for the first time . This solution may also work , but would converge slower . In this paper , we will only focus on the case where the gradient gt is replaced by gt + mt−1 or gt + gt−1 when computing mt in Adam and AAmsGrad ( see algorithm 1 for details ) or gt in other optimizers ( SGD and AdaGrad for example ) if the direction does not change . That change we made to the optimizer do not alter its convergence since the quantity Γt ( Reddi et al. , 2018 ) — which essentially measures the change in the inverse of learning rate of the adaptive method with respect to time — will keep its property . Algorithm 2 Accelerated - Generic Gradient Descent Method Input : x1 ∈ F , step size ( α > 0 ) t← 1 S ← threshold repeat gt =∇ft ( xt ) new g = gt if gt · gt−1 > 0 and | gt−1 − gt | > S then new g = gt + gt−1 xt+1 = xt − αt · new g t← t + 1 until t > T 3.3 CONVERGENCE ANALYSIS . We assume that if we are able to prove that modifying one optimizer with the proposed method does not alter its convergence , then the same applies for the other optimizers . Thus , we only analyse the convergence of one deterministic optimization method AGD ( Accelerated Gradient Descent ) and one non-deterministic method ( AAMSGrad ) . For deterministic methods such as GD , the convergence analysis consists in finding an upper bound of the difference between the function value at the t-th iteration and the minimal value ( f ( xt ) − f ( x∗ ) ) , where f is the function to minimize , x∗ is the optimal solution and xt is the solution found by the algorithm at time t. Theorem 1 . Let f : Rd → R be a L-Lipschitz convex function and x∗ = argminxf ( x ) . Then , GD with step-size α ≤ 1/L satisfies the following : T−1∑ t=0 ( f ( xt+1 ) − f ( x∗ ) ) ≤ L 2 [ ‖x0 − x∗‖2 − ‖xT − x∗‖2 ] f ( xT ) − f ( x∗ ) ≤ L 2T ‖x0 − x∗‖2 ( 2 )

Summary: The authors present an acceleration technique for first-order optimization algorithms by comparing the directions of gradients in consecutive steps, which works for SGD, Adam, and AMSGrad.  Empirically it seems to work well with some standard evaluations with CNN for MNIST and CIFAR10 and LSTM for IMDB, beating the non-accelerated versions in convergence speed. However there are some issues with the parameter choice and proofs. Below are my specific comments: