High Probability Bounds for a Class of Nonconvex Algorithms with AdaGrad Stepsize

√ T ) with high probability without the knowledge of smoothness and variance . We use a particular version of Freedman ’ s concentration bound for martingale difference sequences [ Kakade & Tewari , 2008 ] which enables us to achieve the best-known dependence of log ( 1/δ ) on the probability margin δ . We present our analysis in a modular way and obtain a complementary O ( 1/T ) convergence rate in the deterministic setting for the same class of algorithms . To the best of our knowledge , this is the first high probability result for AdaGrad with a truly adaptive scheme , i.e. , completely oblivious to the knowledge of smoothness and uniform variance bound , which simultaneously has best-known dependence on probability margin δ . 1 INTRODUCTION . Adaptive gradient methods are a staple of machine learning ( ML ) in solving core problems such as min x∈Rd f ( x ) : = Ez∼D [ f ( x ; z ) ] , ( P ) where the objective f ( x ) is possibly non-convex , and D is a probability distribution from which the random vector z is drawn . Problem ( P ) captures for instance empirical risk minimization or finite-sum minimization ( Shalev-Shwartz & Ben-David , 2014 ) problems , where z represents the mini-batches and D corresponds to the distribution governing the data or its sampling strategy . Within the context of large-scale problems , including streaming data , computing full gradients is extremely costly , if not impossible . Hence , stochastic iterative methods are the main optimizer choice in these scenarios . The so-called adaptive methods such as AdaGrad ( Duchi et al. , 2011 ) , Adam ( Kingma & Ba , 2014 ) and AmsGrad ( Reddi et al. , 2018 ) have witnessed a surge of interest both theoretically and practically due to their off-the-shelf performance . For instance , adaptive optimization methods are known to show superior performance in various learning tasks such as machine translation ( Zhang et al. , 2020 ; Vaswani et al. , 2017 ) . From a theoretical point of view , existing literature provides a quite comprehensive understanding regarding the expected behaviour of existing adaptive learning methods . Nevertheless , these results do not capture the behaviour of adaptive methods within a single/few runs , which is related to the probabilistic nature of these methods . While there exist high probability analysis of vanilla SGD for non-convex problems ( Ghadimi & Lan , 2013 ) , adaptive methods have received limited attention in this context . Our main goal in this paper is to understand the probabilistic convergence properties of adaptive algorithms , specifically AdaGrad , while focusing on their problem parameter adaptation capabilities in the non-convex setting . In this manuscript , adaptivity refers to the ability of an algorithm to ensure convergence without requiring the knowledge of quantities such as smoothness modulus or variance of noise . Studies along this direction largely exist for the convex objectives ; for instance , Levy et al . ( 2018 ) shows that AdaGrad can ( implicitly ) exploit smoothness and adapt to the magnitude of noise in the gradients when f ( x ) is convex in ( P ) . This alternative perspective to adaptivity is crucial because most existing analysis , both for classical and adaptive methods , assume to have access to smoothness constant , bound on gradients ( Reddi et al. , 2018 ) and even noise variance ( Ghadimi & Lan , 2013 ) . In practice , it is difficult , if not impossible , to compute or even estimate such quantities . For this purpose , in the setting of ( P ) we study a class of adaptive gradient methods that enable us to handle noisy gradient feedback without requiring the knowledge of the objective ’ s smoothness modulus , nor a bound on gradient norms neither noise variance . We summarize our contributions as follows : 1 . We provide a modular and simplified high probability analysis for AdaGrad-type adaptive methods . 2 . We present the first optimal high probability convergence result of the original AdaGrad algorithm for non-convex smooth problems . Concretely , ( a ) we analyze a fully adaptive step-size that is oblivious to Lipschitz constant L and the noise level σ , ( b ) our analysis demonstrates the best known dependence of log ( 1/δ ) on the probability margin δ . 3 . We present new adaptive versions of AdaGrad extensions that include averaging and momentum primitives , and prove similar high probability bounds for these methods as well . Concretely , we present a general adaptive template and provide complementary convergence results for different step-size regimes which individually recovers standard AdaGrad , AdaGrad with averaging and adaptive RSAG ( Ghadimi & Lan , 2016 ) . In the next section , we will provide a broad overview of related work with an emphasis on the recent developments . Section 3 formalizes the problem setting and states our blanket assumptions . Section 4 introduces the building blocks of our proposed proof technique while proving convergence results for AdaGrad . We generalize the convergence results of AdaGrad for a class of nonconvex , adaptive algorithms in Section 5 . Finally , we present concluding remarks in the last section . 2 RELATED WORK . Adaptive methods for stochastic optimization As an extended version of the online ( projected ) GD ( Zinkevich , 2003 ) , AdaGrad ( Duchi et al. , 2011 ) is the pioneering work behind most of the contemporary adaptive optimization algorithms Adam , AmsGrad and RmsProp ( Tieleman & Hinton , 2012 ) to name a few . Simply put , such AdaGrad-type methods compute step-sizes on-the-fly by accumulating gradient information and achieve adaptive regret bounds as a function of gradient history ( see also ( Tran & Phong , 2019 ; Alacaoglu et al. , 2020b ; Luo et al. , 2019 ; Huang et al. , 2019 ) ) . Universality , adaptive methods and acceleration We call an algorithm universal if it achieves optimal convergence rates under different settings , without any modifications . For convex minimization problems , Levy et al . ( 2018 ) showed that AdaGrad attains a rate of O ( 1/T + σ/ √ T ) by implicitly adapting to smoothness and noise levels in the feedback ; here T is the number of noisy gradient queries and σ is the noise variance . They also proposed an accelerated AdaGrad variant with scalar step-size . The latter result was extended for compactly constrained problems by ( Kavis et al. , 2019 ) , by devising accelerated Mirror-Prox algorithm . Very recently , Ene et al . ( 2021 ) have further generalized the latter results by designing and analyzing a novel adaptive and accelerated AdaGrad version with per-coordinate step-sizes . Convergence properties of these universal ( and some accelerated ) adaptive algorithms under smooth , non-convex losses are unknown to date . Adaptive methods for nonconvex optimization Following the popularity of neural networks , adaptive methods have attracted massive attention due to their favorable performance in training and their ease of tuning . The literature is quite vast , which is impossible to cover exhaustively here . Within the representative subset ( Chen et al. , 2019 ; Zaheer et al. , 2018 ; Li & Orabona , 2019 ; Zou et al. , 2019 ; Defossez et al. , 2020 ; Alacaoglu et al. , 2020a ; Chen et al. , 2020 ) . The majority of the theoretical results on adaptive methods for nonconvex problems focus on their in expectation performance . High probability results Ghadimi & Lan ( 2013 ) are the first to analyze SGD in the non-convex regime and provide tight convergence bounds . Nevertheless , their method requires a prior knowledge of the smoothness modulus and noise variance . In the context of adaptive methods , Li & Orabona ( 2020 ) considers delayed AdaGrad ( with lag-one-behind step-size ) for smooth , non-convex losses under subgaussian noise and proved O ( σ √ log ( T/δ ) / √ T ) rate . Under similar conditions , Zhou et al . ( 2018 ) proves convergence of order O ( ( σ2 log ( 1/δ ) ) /T + 1/ √ T ) for AdaGrad . However , both works require the knowledge of smoothness to set the step-size . Moreover , Ward et al . ( 2019 ) guarantees that AdaGrad with scalar step-size convergences at O ( ( 1/δ ) log ( T ) / √ T ) rate with high probability . Although their framework is oblivious to smoothness constant , their dependence of probability margin is far from optimal . More recently , under heavy-tailed noise having bounded pth moment for p ∈ ( 1 , 2 ) , Cutkosky & Mehta ( 2021 ) proves a rate of O ( log ( T/δ ) /T ( p−1 ) / ( 3p−2 ) ) for clipped normalized SGD with momentum ; nevertheless their method requires the knowledge of ( a bound on ) the behavior of the heavy tails . 3 SETUP AND PRELIMINARIES . As we stated in the introduction , we consider the unconstrained minimization setting min x∈Rd f ( x ) : = Ez∼D [ f ( x ; z ) ] , where the differentiable function f : Rd → R is a smooth and ( possibly ) nonconvex function . We are interested is finding a first-order -stationary point satisfying ‖∇f ( xt ) ‖2 ≤ , where ‖·‖ denotes the Euclidean norm for the sake of simplicity . As the standard measure of convergence , we will quantify the performance of algorithms with respect to average gradient norm , 1T ∑T t=1 ‖∇f ( xt ) ‖2 . It immediately implies convergence in minimum gradient norm , mint∈ [ T ] ‖∇f ( xt ) ‖2 . Moreover , note that if we are able to bound 1T ∑T t=1 ‖∇f ( xt ) ‖2 , then by choosing to output a solution x̄T which is chosen uniformly at random from the set of query points { x1 , . . . , xT } , then we ensure that E‖∇f ( x̄T ) ‖2 : = 1T ∑T t=1 ‖∇f ( xt ) ‖2 is bounded . A function is called G-Lipschitz continuous if it satisfies |f ( x ) − f ( y ) | ≤ G‖x− y‖ , ∀x , y ∈ dom ( f ) , ( 1 ) which immediately implies that ‖∇f ( x ) ‖ ≤ G , ∀x ∈ dom ( f ) . ( 2 ) A differentiable function is called L-smooth if it has L-Lipschitz gradient ‖∇f ( x ) −∇f ( y ) ‖ ≤ L‖x− y‖ , ∀x , y ∈ dom ( ∇f ) . ( 3 ) An equivalent characterization of smoothness is sometimes referred to as the “ descent lemma ” ( Ward et al. , 2019 ; Beck , 2017 ) , i.e. , |f ( x ) − f ( y ) − 〈∇f ( y ) , x− y〉| ≤ L 2 2 ‖x− y‖2 . ( 4 ) Assumptions on oracle model : We denote stochastic gradients with ∇̃f ( x ) = ∇f ( x ; z ) , for some random vector drawn from distribution D. Since our template embraces single-call algorithms , we use this shorthand notation for simplicity . An oracle is called ( conditionally ) unbiased if E [ ∇̃f ( x ) |x ] = ∇f ( x ) , ∀x ∈ dom ( ∇f ) . ( 5 ) Gradient estimates generated by a first-order oracle are said to have bounded variance if they satisfy E [ ‖∇̃f ( x ) −∇f ( x ) ‖2|x ] ≤ σ2 , ∀ ∈ dom ( ∇f ) . ( 6 ) Finally , we assume that the stochastic gradient are bounded almost surely , i.e. , ‖∇̃f ( x ) ‖ ≤ G̃ , ∀x ∈ dom ( ∇f ) . ( 7 ) Remark 1 . Bounded variance assumption ( 6 ) is standard in the analysis of stochastic methods ( Lan , 2020 ) . Similarly , for the analysis of adaptive methods in the nonconvex realm , it is very common to assume bounded stochastic gradients ( see ( Zaheer et al. , 2018 ; Zhou et al. , 2018 ; Chen et al. , 2019 ; Li & Orabona , 2020 ) and references therein ) .

Summary: This paper provides an analysis of adaptive learning rate scheme in stochastic non-convex settings. Under assumptions of smoothness and bounded stochastic gradients (a light tailed condition), it is shown that the sum of the gradient norms of the iterates of the non-convex algorithm is small. In order to accomplish this, the analysis must deal with the standard difficulty for adaptive learning rates, which is that the learning rate depends on the current iterate and so makes it harder to apply standard martingale inequalities. The final results include a dependence on the smoothness constant $L$ and variance $\sigma$ which are not explicitly used in the algorithm.