A new accelerated gradient method inspired by continuous-time perspective

1 Introduction . Optimization is a core component of statistic and machine learning problems . Recently , gradientbased algorithms are widely used in such optimization problems due to its simplicity and efficiency for large-scale situations . For solving convex optimization problem min x∈Rd F ( x ) , where F ( x ) is convex and sufficiently smooth , a classical first-order method is gradient descent . We assume that f ( x ) = ∇F ( x ) satisfies L-Lipschitz condition , that is , there exists constant L such that ∥f ( x ) − f ( y ) ∥ ≤ L∥x− y∥ , ∀x , y . Under these conditions , gradient descent achieves a convergence rate of O ( n−1 ) , i.e. , ∥F ( xn ) − F ( x∗ ) ∥ decreases to zero at a rate of O ( n−1 ) , where xn denotes the nth iteration and x∗ denotes the minimum point of F ( x ) in Rd . Nesterov ’ s accelerated method ( Nesterov , 1983 ) is a more efficient first-order algorithm than gradient descent , of which we will use the following form : starting with x0 = x1 , yn = xn + n− 3 n ( xn − xn−1 ) , xn+1 = yn − sf ( yn ) ( 1.1 ) for n ≥ 1 . It is shown that under abovementioned conditions , Nesterov ’ s accelerated method converges at a rate of O ( n−2 ) . Accelerated gradient method has been successful in training deep and recurrent neural networks ( Sutskever et al. , 2013 ) and is widely used in problems with machine learning background to avoid sophisticated second-order methods ( Cotter et al. , 2011 ; Hu et al. , 2009 ; Ji & Ye , 2009 ) . To provide more theorical understanding , an important research topic of Nesterov ’ s accelerated method is to find an explanation of the acceleration . On this topic , Nesterov ’ s method was studied via a continuous-time perspective ( Su et al. , 2014 ) . They considered a curve x ( t ) , introduced the ansatz xn ≈ x ( n √ s ) and substituted it to ( 1.1 ) . Letting s → 0 , they obtained the following differential equation . ẍ+ 3 t ẋ+ f ( x ) = 0 . ( 1.2 ) The differential equation was used as a tool for analyzing and generalizing Nesterov ’ s scheme . Furthermore , this idea has been studied from different directions . A class of accelerated methods have been generated in continuous-time ( Wibisono et al. , 2016 ) . ODE ( 1.2 ) can also be discretized directly using Runge-Kutta method to achieve acceleration ( Zhang et al. , 2018 ) . Although many results have been achieved , the process of obtaining the differential equation ( 1.2 ) has not been rigorous , and the method is still time-consuming for large-scale problems . In this work , we give the precise order of the iterations of Nesterov ’ s accelerated method converging to solution of the differential equation ( 1.2 ) with initial conditions x ( 0 ) = x0 , ẋ ( 0 ) = 0 ( 1.3 ) as step size s goes to zero . Inspired from this perspective , we present a new accelerated method to make this convergence faster . As we expected , iterations of the new method are closer to the solution x ( t ) of differential equation ( 1.2 ) than original Nesterov ’ s method . Moreover , we find the new method is more stable than original Nesterov ’ s method when step size is large . Based on abovementioned observations , we try to take advantage of the new method in more practical problems . We apply the new method to matrix completion problem . We combine the new method with proximal operator ( Parikh & Boyd , 2014 ) into a new algorithm , which we call modified FISTA . We find that the new method performs better than FISTA ( Beck & Teboulle , 2009 ) and acclerated proximal gradient method ( Parikh & Boyd , 2014 ) because it can work with larger step sizes . This paper is organized as follows . In section 2 , we prove that iterations of Nesterov ’ s accelerated method converge to solution of the differential equation ( 1.2 ) . In section 3 , we present a new method to make the convergence faster and show its better stablity through two simple examples . In section 4 , we apply the new method to matrix completion problem . 2 A strict analysis of the relation between Nesterov ’ s method and its continuous-time limit . We refer to x ( t ) as the solution of differential equation ( 1.2 ) with initial conditions ( 1.3 ) . Existance and uniqueness of such solutions have been proved ( Su et al. , 2014 ) . In this section , We give the order of the iterations of Nesterov ’ s accelerated method converging to x ( t ) as step sizes go to zero . For convenience , we substitute the first equation in Nesterov ’ s method ( 1.1 ) to the second one to get xn+1 = xn + n− 3 n ( xn − xn−1 ) − s · f ( xn + n− 3 n ( xn − xn−1 ) ) . We write s = h2 and rewrite the above recurrence relation as xn+1 = xn + n− 3 n ( xn − xn−1 ) − h2 · f ( xn + n− 3 n ( xn − xn−1 ) ) . ( 2.1 ) Inspired by the ansatz xn ≈ x ( n √ s ) ( Su et al. , 2014 ) , we consider the convergence between xn and x ( nh ) . More precisely , we show that for fixed time t , xn converges to x ( t ) as h goes to zero , where n = th . 2.1 Truncation error Firstly , we consider the following ‘ truncation error ’ : L [ x ( t ) ; h ] =x ( t+ h ) − 2t− 3h t x ( t ) + t− 3h t x ( t− h ) + h2f ( x ( t ) + t− 3h t ( x ( t ) − x ( t− h ) ) ) . ( 2.2 ) ( 2.2 ) is obtained from ( 2.1 ) by replacing xn+1 , xn , xn−1 with x ( t+h ) , x ( t ) , x ( t−h ) and substituting the relation n = th . Our first result is the order of truncation error L [ x ( t ) ; h ] . Theorem 1 . Assume f satisfies L-Lipschitz condition , and solution x ( t ) of the derived differential equation ( 1.2 ) has a continuous third derivative . For fixed time t , the truncation error ( 2.2 ) satisfies L [ x ( t ) ; h ] = O ( h3 ) . Theorem 1 shows the size of error caused by a single iteration when the starting point is just on x ( t ) . Then we have to add up these errors to prove the convergence proporty we need . 2.2 Convergence theorem We now come to the convergence theorem . In this theorem , we give the precise order of the iterations of Nesterov ’ s method converging to solution of the derived differential equation . Theorem 2 . Under conditions in Theorem 1 , for fixed time t , xt/h converges to x ( t ) as h goes to zero at a rate of O ( h ln 1h ) if x0 = x ( 0 ) and x1 = x ( h ) . Theorem 2 coincides with derivation of ODE ( 1.2 ) ( Su et al. , 2014 ) . 3 New accelerated method . 3.1 Derivation of the new method and analysis of truncation error Inspired from the continuous-time perspective and our proof of the convergence from iterations of Nesterov ’ s method to its continuous-time limit , we present a new method to make this convergence faster . Precisely , the new method has a higher truncation order . We need one more step in our scheme than in Nesterov ’ s method to achieve higher truncation order in the following analysis , so we consider a recurrence relation with form 4∑ i=1 ( αi + βi n + γi n2 ) xn+2−i = −sf ( xn + n− 3 n ( xn − xn−1 ) ) , ( 3.1 ) where { αi } , { βi } and { γi } are to be determined . Now we expand x ( t− h ) to first order . Calculation shows that f ( x ( t ) + t− 3h t ( x ( t ) − x ( t− h ) ) ) =− hx ( 3 ) ( t ) − ( 3h t + 1 ) x ( 2 ) ( t ) + ( 3h t2 − 3 t ) x ( 1 ) ( t ) +O ( h2 ) . Substitute this expansion to truncation error L [ x ( t ) ; h ] = 4∑ i=1 ( αi + βih t + γih 2 t2 ) x ( t+ ( 2− i ) h ) + h2f ( x ( t ) + t− 3h t ( x ( t ) − x ( t− h ) ) ) , and choose parameters appropriately to eliminate low-order terms , we get the following recurrence relation xn+1 = 10n2 + 9n+ 6 4n2 + 8n xn − 4n2 + 3 2n2 + 4n xn−1 + 2n− 1 4n+ 8 xn−2 − n 2n+ 4 sf ( 2n− 3 n xn − n− 3 n xn−1 ) . ( 3.2 ) Here we rewrite this scheme as Algorithm 1 . Algorithm 1 The new method ( 3.2 ) Input : step size s Initial value : X2 = X1 = X0 . ( k− 1 ) th iteration ( k ≥ 2 ) . Compute Yk = 10k2 + 9k + 6 4k2 + 8k Xk − 4k2 + 3 2k2 + 4k Xk−1 + 2k − 1 4k + 8 Xk−2 , Zk = 2k − 3 k Xk − k − 3 k Xk−1 , Xk+1 = X − ( Yk − ks 2k + 4 f ( Zk ) ) . For truncation order of this new method , we have the following theorem . The abovementioned procedure is presented in Appendix A.4 detailedly , as proof of Theorem 3 . Theorem 3 . If f has continuous second order derivative , the first and second derivative are bounded , and x ( t ) has continuous fourth derivative , then for fixed t , truncation error of ( 3.2 ) satisfies L [ x ( tn ) ; h ] = O ( h4 ) . The convergence of the new method and x ( t ) can be proved similar to Theorem 2 . 3.2 Advantage of the new method Since the new method has a truncation error of higher order than original Neaterov ’ s method , the iterations of the new method converge to the differential equation ( 1.2 ) when those of original Nesterov ’ s method diverge . In another word , the new method is more stable for large step size . We present two numerical results in Figure 1 to confirm it . Quadratic . F ( x ) = xTAx is a strongly convex function , in which x ∈ R2 and A is a 2× 2 matrix . Linear regression . F ( x ) = n∑ i=1 ( yi −wTi x ) 2 , where n is the number of samples and ( wi , yi ) is the ith sample . In these examples , iterations of the new method converge to the minimum point , while those of original Nesterov ’ s method diverge , which confirms that the new method is more stable for large step size . 3.3 Absolute stability of Nesterov ’ s method and the new method In this subsection , we explain the better stability of the new method with absolute stability theory . Firstly , recall the scheme of our new method xn+1 = 10n2 + 9n+ 6 4n2 + 8n xn − 4n2 + 3 2n2 + 4n xn−1 + 2n− 1 4n+ 8 xn−2 − n 2n+ 4 sf ( 2n− 3 n xn − n− 3 n xn−1 ) . We use linear approximation f ( xn + n− 3 n ( xn − xn−1 ) ) = ∇F ( xn + n− 3 n ( xn − xn−1 ) ) ≈ ∇2F · ( xn + n− 3 n ( xn − xn−1 ) ) , and the characteristic equation of this finite scheme is approximately λ3− ( 10n2 + 9n+ 6 4n2 + 8n − s · ∇2F · 2n 2 − 3n 2n2 + 4n ) λ2+ ( 4n2 + 3 2n2 + 4n − s · ∇2F · n 2 − 3n 2n2 + 4n ) λ− 2n− 1 4n+ 8 = 0 . For large n , we can ignore the high order terms and the characteristic equation becomes λ3 − ( 5 2 − s · ∇2F · ) λ2 + ( 2− s 2 · ∇2F ) λ− 1 2 = 0 . According to the absolute stability theory , the numerical stability of Nesterov ’ s scheme with respect to accumulated roundoff error is equivalent to this : all the roots of the characteristic equation lie in the unit circle ( Leader , 2004 ) . Noticing that the left hand of the equation can be factorized to ( λ− 1 2 ) ( λ2 − ( 2− s · ∇2F ) λ+ 1 ) , the largest modulu of the roots is 1 when 0 ≤ s · ∇2F ≤ 4 , and the absolutely stable region of the new method is s · ∇2F ∈ [ 0 , 4 ] . When s·∇2F lies in the absoletely stable region , the related theory guarantees that the error caused by every iteration will not be magnified as the iteration number increases . To make the analysis more precise , we should consider the difference of the scheme between iterations caused by different n. We define the transfer matrix Pn = ( 10n2+9n+6 4n2+8n − s · ∇ 2F · 2n 2−3n 2n2+4n ) − ( 4n2+3 2n2+4n − s · ∇ 2F · n 2−3n 2n2+4n ) 2n−1 4n+8 1 0 0 0 1 0 and Qn = PnPn−1 · · ·P1 . Error analysis shows that if the largest modulu of eigenvalues of Qn goes to zero , then error caused by iterations will be eliminated as the iteration number increases . Figure 2 presents the largest module of eigenvalues of Qn for different values of s · ∇2F . From the experiment we can see that the above condition is satisfied . We then apply the same method to Nesterov ’ s method discussed in ( Su et al. , 2014 ) and conclude that the absolutely stable region of Nesterov ’ s method is [ 0 , 43 ] . According to the above analysis , the absolutely stable region of the new method is four times as large as Nesterov ’ s method , so the new method is more stable , and we can choose larger step sizes to achieve faster convergence .

Summary: Review:  This paper refines the the truncation error analysis for discretizing the ODE to obtain accelerated optimization method.  The truncation results include higher order term. Built upon the analysis, the authors propose a new method which is claimed to be more stable for large step size and converges faster. Numerical evidence on matrix completion problem is provided.