Escape saddle points by a simple gradient-descent based algorithm

1 Introduction . Nonconvex optimization is a central research area in optimization theory , since lots of modern machine learning problems can be formulated in models with nonconvex loss functions , including deep neural networks , principal component analysis , tensor decomposition , etc . In general , finding a global minimum of a nonconvex function is NP-hard in the worst case . Instead , many theoretical works focus on finding a local minimum instead of a global one , because recent works ( both empirical and theoretical ) suggested that local minima are nearly as good as global minima for a significant amount of well-studied machine learning problems ; see e.g . [ 4 , 12 , 14 , 15 , 17 , 18 ] . On the other hand , saddle points are major obstacles for solving these problems , not only because they are ubiquitous in high-dimensional settings where the directions for escaping may be few ( see e.g . [ 5 , 8 , 11 ] ) , but also saddle points can correspond to highly suboptimal solutions ( see e.g . [ 19 , 28 ] ) . Hence , one of the most important topics in nonconvex optimization is to escape saddle points . Specifically , we consider a twice-differentiable function f : Rn ! R such that • f is ` -smooth : krf ( x1 ) rf ( x2 ) k ` kx1 x2k 8x1 , x2 2 Rn , • f is ⇢-Hessian Lipschitz : kH ( x1 ) H ( x2 ) k ⇢kx1 x2k 8x1 , x2 2 Rn ; here H is the Hessian of f . The goal is to find an ✏-approximate second-order stationary point x✏:1 krf ( x✏ ) k ✏ , min ( H ( x✏ ) ) p ⇢✏ . ( 1 ) ⇤Corresponding author . Email : tongyangli @ pku.edu.cn 1We can ask for an ( ✏1 , ✏2 ) -approx . second-order stationary point s.t . krf ( x ) k ✏1 and min ( r2f ( x ) ) ✏2 in general . The scaling in ( 1 ) was adopted as a standard in literature [ 1 , 6 , 10 , 20 , 21 , 22 , 26 , 29 , 30 , 31 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In other words , at any ✏-approx . second-order stationary point x✏ , the gradient is small with norm being at most ✏ and the Hessian is close to be positive semi-definite with all its eigenvalues p⇢✏ . Algorithms for escaping saddle points are mainly evaluated from two aspects . On the one hand , considering the enormous dimensions of machine learning models in practice , dimension-free or almost dimension-free ( i.e. , having poly ( log n ) dependence ) algorithms are highly preferred . On the other hand , recent empirical discoveries in machine learning suggests that it is often feasible to tackle difficult real-world problems using simple algorithms , which can be implemented and maintained more easily in practice . On the contrary , algorithms with nested loops often suffer from significant overheads in large scales , or introduce concerns with the setting of hyperparameters and numerical stability ( see e.g . [ 1 , 7 ] ) , making them relatively hard to find practical implementations . It is then natural to explore simple gradient-based algorithms for escaping from saddle points . The reason we do not assume access to Hessians is because its construction takes ⌦ ( n2 ) cost in general , which is computationally infeasible when the dimension is large . A seminal work along this line was by Ge et al . [ 12 ] , which found an ✏-approximate second-order stationary point satisfying ( 1 ) using only gradients in O ( poly ( n , 1/✏ ) ) iterations . This is later improved to be almost dimension-free Õ ( log4 n/✏2 ) in the follow-up work [ 20 ] ,2 and the perturbed accelerated gradient descent algorithm [ 22 ] based on Nesterov ’ s accelerated gradient descent [ 27 ] takes Õ ( log6 n/✏1.75 ) iterations . However , these results still suffer from a significant overhead in terms of log n. On the other direction , Refs . [ 3 , 25 , 30 ] demonstrate that an ✏-approximate second-order stationary point can be find using gradients in Õ ( log n/✏1.75 ) iterations . Their results are based on previous works [ 1 , 6 ] using Hessian-vector products and the observation that the Hessian-vector product can be approximated via the difference of two gradient queries . Hence , their implementations contain nested-loop structures with relatively large numbers of hyperparameters . It has been an open question whether it is possible to keep both the merits of using only first-order information as well as being close to dimension-free using a simple , gradient-based algorithm without a nested-loop structure [ 23 ] . This paper answers this question in the affirmative . Contributions . Our main contribution is a simple , single-loop , and robust gradient-based algorithm that can find an ✏-approximate second-order stationary point of a smooth , Hessian Lipschitz function f : Rn ! R. Compared to previous works [ 3 , 25 , 30 ] exploiting the idea of gradient-based Hessian power method , our algorithm has a single-looped , simpler structure and better numerical stability . Compared to the previous state-of-the-art results with single-looped structures by [ 22 ] and [ 20 , 21 ] using Õ ( log6 n/✏1.75 ) or Õ ( log4 n/✏2 ) iterations , our algorithm achieves a polynomial speedup in log n : Theorem 1 ( informal ) . Our single-looped algorithm finds an ✏-approximate second-order stationary point in Õ ( log n/✏1.75 ) iterations . Technically , our work is inspired by the perturbed gradient descent ( PGD ) algorithm in [ 20 , 21 ] and the perturbed accelerated gradient descent ( PAGD ) algorithm in [ 22 ] . Specifically , PGD applies gradient descents iteratively until it reaches a point with small gradient , which can be a potential saddle point . Then PGD generates a uniform perturbation in a small ball centered at that point and then continues the GD procedure . It is demonstrated that , with an appropriate choice of the perturbation radius , PGD can shake the point off from the neighborhood of the saddle point and converge to a second-order stationary point with high probability . The PAGD in [ 22 ] adopts a similar perturbation idea , but the GD is replaced by Nesterov ’ s AGD [ 27 ] . Our algorithm is built upon PGD and PAGD but with one main modification regarding the perturbation idea : it is more efficient to add a perturbation in the negative curvature direction nearby the saddle point , rather than the uniform perturbation in PGD and PAGD , which is a compromise since we generally can not access the Hessian at the saddle due to its high computational cost . Our key observation lies in the fact that we do not have to compute the entire Hessian to detect the negative curvature . Instead , in a small neighborhood of a saddle point , gradients can be viewed as Hessianvector products plus some bounded deviation . In particular , GD near the saddle with learning rate 1/ ` is approximately the same as the power method of the matrix ( I H/ ` ) . As a result , the most negative eigenvalues stand out in GD because they have leading exponents in the power method , and thus it approximately moves along the direction of the most negative curvature nearby the sad- 2The Õ notation omits poly-logarithmic terms , i.e. , Õ ( g ) = O ( g poly ( log g ) ) . dle point . Following this approach , we can escape the saddle points more rapidly than previous algorithms : for a constant ✏ , PGD and PAGD take O ( log n ) iterations to decrease the function value by ⌦ ( 1/ log3 n ) and ⌦ ( 1/ log5 n ) with high probability , respectively ; on the contrary , we can first take O ( log n ) iterations to specify a negative curvature direction , and then add a larger perturbation in this direction to decrease the function value by ⌦ ( 1 ) . See Proposition 3 and Proposition 5 . After escaping the saddle point , similar to PGD and PAGD , we switch back to GD and AGD iterations , which are efficient to decrease the function value when the gradient is large [ 20 , 21 , 22 ] . Our algorithm is also applicable to the stochastic setting where we can only access stochastic gradients , and the stochasticity is not under the control of our algorithm . We further assume that the stochastic gradients are Lipschitz ( or equivalently , the underlying functions are gradient-Lipschitz , see Assumption 2 ) , which is also adopted in most of the existing works ; see e.g . [ 9 , 20 , 21 , 35 ] . We demonstrate that a simple extended version of our algorithm takes O ( log2 n ) iterations to detect a negative curvature direction using only stochastic gradients , and then obtain an ⌦ ( 1 ) function value decrease with high probability . On the contrary , the perturbed stochastic gradient descent ( PSGD ) algorithm in [ 20 , 21 ] , the stochastic version of PGD , takes O ( log10 n ) iterations to decrease the function value by ⌦ ( 1/ log5 n ) with high probability . Theorem 2 ( informal ) . In the stochastic setting , our algorithm finds an ✏-approximate second-order stationary point using Õ ( log2 n/✏4 ) iterations via stochastic gradients . Our results are summarized in Table 1 . Although the underlying dynamics in [ 3 , 25 , 30 ] and our algorithm have similarity , the main focus of our work is different . Specifically , Refs . [ 3 , 25 , 30 ] mainly aim at using novel techniques to reduce the iteration complexity for finding a second-order stationary point , whereas our work mainly focuses on reducing the number of loops and hyper-parameters of negative curvature finding methods while preserving their advantage in iteration complexity , since a much simpler structure accords with empirical observations and enables wider applications . Moreover , the choice of perturbation in [ 3 ] is based on the Chebyshev approximation theory , which may require additional nested-looped structures to boost the success probability . In the stochastic setting , there are also other results studying nonconvex optimization [ 16 , 24 , 32 , 37 , 13 , 33 , 36 ] from different perspectives than escaping saddle points , which are incomparable to our results . It is worth highlighting that our gradient-descent based algorithm enjoys the following nice features : • Simplicity : Some of the previous algorithms have nested-loop structures with the concern of practical impact when setting the hyperparameters . In contrast , our algorithm based on negative curvature finding only contains a single loop with two components : gradient descent ( including AGD or SGD ) and perturbation . As mentioned above , such simple structure is preferred in machine learning , which increases the possibility of our algorithm to find real-world applications . • Numerical stability : Our algorithm contains an additional renormalization step at each iteration when escaping from saddle points . Although in theoretical aspect a renormalization step does not affect the output and the complexity of our algorithm , when finding negative curvature near saddle points it enables us to sample gradients in a larger region , which makes our algorithm more numerically stable against floating point error and other errors . The introduction of renormalization step is enabled by the simple structure of our algorithm , which may not be feasible for more complicated algorithms [ 3 , 25 , 30 ] . • Robustness : Our algorithm is robust against adversarial attacks when evaluating the value of the gradient . Specifically , when analyzing the performance of our algorithm near saddle points , we essentially view the deflation from pure quadratic geometry as an external noise . Hence , the effectiveness of our algorithm is unaffected under external attacks as long as the adversary is bounded by deflations from quadratic landscape . Finally , we perform numerical experiments that support our polynomial speedup in log n. We perform our negative curvature finding algorithms using GD or SGD in various landscapes and general classes of nonconvex functions , and use comparative studies to show that our Algorithm 1 and Algorithm 3 achieve a higher probability of escaping saddle points using much fewer iterations than PGD and PSGD ( typically less than 1/3 times of the iteration number of PGD and 1/2 times of the iteration number of PSGD , respectively ) . Moreover , we perform numerical experiments benchmarking the solution quality and iteration complexity of our algorithm against accelerated methods . Compared to PAGD [ 22 ] and even advanced optimization algorithms such as NEON+ [ 30 ] , Algorithm 2 possesses better solution quality and iteration complexity in various landscapes given by more general nonconvex functions . With fewer iterations compared to PAGD and NEON+ ( typically less than 1/3 times of the iteration number of PAGD and 1/2 times of the iteration number of NEON+ , respectively ) , our Algorithm 2 achieves a higher probability of escaping from saddle points . Open questions . This work leaves a couple of natural open questions for future investigation : • Can we achieve the polynomial speedup in log n for more advanced stochastic optimization algorithms with complexity Õ ( poly ( log n ) /✏3.5 ) [ 2 , 3 , 10 , 29 , 31 ] or Õ ( poly ( log n ) /✏3 ) [ 9 , 34 ] ? • How is the performance of our algorithms for escaping saddle points in real-world applications , such as tensor decomposition [ 12 , 17 ] , matrix completion [ 14 ] , etc. ? Broader impact . This work focuses on the theory of nonconvex optimization , and as far as we see , we do not anticipate its potential negative societal impact . Nevertheless , it might have a positive impact for researchers who are interested in understanding the theoretical underpinnings of ( stochastic ) gradient descent methods for machine learning applications . Organization . In Section 2 , we introduce our gradient-based Hessian power method algorithm for negative curvature finding , and present how our algorithms provide polynomial speedup in log n for both PGD and PAGD . In Section 3 , we present the stochastic version of our negative curvature finding algorithm using stochastic gradients and demonstrate its polynomial speedup in log n for PSGD . Numerical experiments are presented in Section 4 . We provide detailed proofs and additional numerical experiments in the supplementary material .

Summary: In this work, the authors study the escaping saddle points problem for nonconvex optimization. Previously Jin et al. proposed a perturbed accelerated gradient descent (AGD) method which finds $\epsilon$-second-order stationary points in $O(\log^6 n/\epsilon^{1.75})$ number of iterations, where $n$ is the dimension. By utilizing a refined perturbed negative curvature estimator, the authors successfully improves the dependence of dimension from $\log^6 n$ to $\log n$. Experiment results suggest that their algorithm performs well in practice.