Convergence Rates of Stochastic Gradient Descent under Infinite Noise Variance

1 Introduction . We consider the unconstrained minimization problem minimize x∈Rn f ( x ) , ( 1.1 ) using the stochastic gradient descent ( SGD ) algorithm . Initialized at x0 ∈ Rn , the SGD algorithm is given by the iterations , xt+1 = xt − γt+1 ( ∇f ( xt ) + ξt+1 ( xt ) ) , t = 0 , 1 , 2 , ... ( 1.2 ) where { γt } t∈N+ denotes the step-size sequence , and { ξt } t∈N+ is a martingale difference sequence adapted to a filtration { Ft } t∈N , characterizing the noise in the gradient ( the sequence { xt } t∈N is also adapted to the same filtration , if we assume x0 is F0-measurable ) . Our focus is on the case where the noise is state dependent , and its variance is infinite , i.e. , E [ ‖ξt‖22 ] =∞ . ∗Work partially conducted while affiliated with the Vector Institute . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Many problems in modern statistical learning can be written in the form ( 1.1 ) , where f ( x ) typically corresponds to the population risk , that is , f ( x ) : = Ez∼ν [ ` ( x , z ) ] for a given loss function ` and an unknown data distribution ν . In practice , one observes independent and identically distributed ( i.i.d . ) samples zi ∼ ν for i ∈ [ n ] , and estimates the population gradient ∇f ( x ) with a noisy gradient at each iteration , which is based on an empirical average over a subset of the samples { zi } i∈ [ n ] . Due to its simplicity , superior generalization performance , and well-understood theoretical guarantees , SGD has been the method of choice for minimization problems arising in statistical machine learning . Starting from the pioneering works of Robbins and Monro [ 1951 ] , Chung [ 1954 ] , Sacks [ 1958 ] , Fabian [ 1968 ] , Ruppert [ 1988 ] , Shapiro [ 1989 ] , Polyak and Juditsky [ 1992 ] , theoretical properties of the SGD algorithm and its variants have been receiving a growing attention under different scenarios . Recent works , for example Tripuraneni et al . [ 2018 ] , Su and Zhu [ 2018 ] , Duchi and Ruan [ 2021 ] , Toulis and Airoldi [ 2017 ] , Fang et al . [ 2018 ] , Anastasiou et al . [ 2019 ] , Yu et al . [ 2020 ] established convergence rates for SGD in various settings . By building on the analysis of Polyak and Juditsky [ 1992 ] to prove a central limit theorem ( CLT ) for the Polyak-Ruppert averaging , these works led to novel methodologies to compute confidence intervals using SGD . However , a recurring assumption in this line of work is the finite noise variance , which may be violated frequently in modern frameworks . Heavy-tailed behavior in statistical methodology may naturally arise from the underlying model , or through the iterative optimization algorithm used during model training . In robust statistics , one often encounters heavy-tailed noise behavior in data , which in conjunction with standard loss functions leads to infinite noise variance in SGD . Very recently , heavy-tailed behavior is shown to emerge from the multiplicative noise in SGD , when the step-size is large and/or the batch-size is small [ Hodgkinson and Mahoney , 2021 , Gürbüzbalaban et al. , 2021 ] . On the other hand , there is strong empirical evidence in modern machine learning that the gradient noise often exhibits a heavy-tailed behavior , which indicates an infinite variance . For example , this is observed in fully connected and convolutional neural networks [ Şimşekli et al. , 2019 , Gürbüzbalaban and Hu , 2021 ] as well as recurrent neural networks [ Zhang et al. , 2020 ] . Thus , understanding the behavior of SGD under infinite noise variance becomes extremely important for at least two reasons . A computational complexity reason : modern machine learning and robust statistics frameworks lead to heavy-tailed behavior in SGD ; thus , understanding the performance of this algorithm in terms of precise convergence rates as well as the required conditions on the step-size sequence as a function of the ‘ heaviness ’ of the tail become crucial in this setup . A statistical reason : many inference methods that rely on Polyak-Ruppert averaging utilize a CLT which holds under finite noise variance ( see e.g . online bootstrap and variance estimation approaches [ Fang et al. , 2018 , Su and Zhu , 2018 , Chen et al. , 2020 ] ) . Using the same methodology in the aforementioned modern frameworks ( under heavy-tailed noise ) will ultimately result in incorrect confidence intervals , jeopardizing the statistical procedure . Thus , establishing the limit distribution in this setting is of great importance . In this work , we study the behavior of the SGD algorithm with diminishing step-sizes for a class of strongly convex problems when the noise variance is infinite . We establish the convergence rates of the SGD iterates towards the global minimum , and identify a sufficient condition on the Hessian of f , which interpolates between the positive semi-definite cone and the cone of diagonally dominant matrices ( with non-negative diagonal entries ) . We further study the Polyak-Ruppert averaging of the SGD iterates , and show that the limit distribution is a multivariate α-stable distribution . We illustrate our theory on linear regression and generalized linear models , demonstrating how to verify the conditions of our theorems . Perhaps surprisingly , our results show that even under heavy-tailed noise with infinite variance , SGD with diminishing step-sizes can converge to the global optimum without requiring any modification neither to the loss function nor to the algorithm itself , as opposed to the conventional techniques used in robust statistics [ Huber , 2004 ] . Finally , we argue that our work has potential implications in constructing confidence intervals in the infinite noise variance setting . 2 Preliminaries and Technical Background . Notational Conventions . By N , N+ and R we denote the set of non-negative integers , positive integers , and real numbers , respectively . For m ∈ N+ , we define [ m ] = { 1 , . . . , m } . We use italic letters ( e.g . x , ξ ) to denote scalars and scalar-valued functions , sign ( x ) to denote the sign of x , bold face italic letters ( e.g . x , ξ ) to denote vectors and vector-valued functions , and bold face upper case letters ( e.g . A ) to denote matrices . We use |x| and ‖x‖p to denote the 2-norm and p-norm of a vector x ; ‖A‖ and ‖A‖p the operator 2-norm and operator p-norm of a matrix A . The transpose of a matrix A and a vector x ( viewed as a matrix with 1 column ) are denoted by AT and xT . If { Ai } i∈N is a sequence of matrices and k > ` , the empty product ∏ ` i=k Ai is understood to be the identity matrix I . For two sequences of real numbers { at } t∈N , { bt } t∈N , we write at = O ( bt ) if lim supt→∞ |at|/|bt| < ∞ , at = o ( bt ) if lim supt→∞ |at|/|bt| = 0 , at = Θ ( bt ) if both at = O ( bt ) and bt = O ( at ) hold , and at bt if limt→∞ |at|/|bt| exists and is in ( 0 , ∞ ) . If at = O ( bttε ) for any ε > 0 , we say at = Õ ( bt ) . Sufficiently large or sufficiently small positive constants whose values do not matter are written as C , C0 , C1 , . . . , sometimes without prior introduction . IfX1 , X2 , . . . is a sequence of random vectors taking value in Rn and µ is a probability measure on Rn , we write Xt D−−−→ t→∞ µ if { Xt } t∈N+ converges in distribution ( also called ‘ weak convergence ’ ) to µ. Stochastic Approximation . In the SGD recursion ( 1.2 ) , we can replace ∇f with an arbitrary continuous functionR : Rn → Rn , and consider the same iterations that stochastically approximate the zero x∗ ofR , xt+1 = xt − γt+1 ( R ( xt ) + ξt+1 ( xt ) ) . ( 2.1 ) This is called the stochastic approximation process [ Robbins and Monro , 1951 ] , which is a predecessor of stochastic gradient descent ( SGD ) and describes a larger family of iterative algorithms ( see e.g . [ Kushner and Yin , 2003 , Chapters 2 and 3 ] ) . Theoretical investigation of the recursion ( 2.1 ) has been active ever since its invention , especially under finite noise variance assumption : Robbins and Monro [ 1951 ] prove that the recursion ( 2.1 ) can lead to the L2 convergence limt→∞ E [ |xt − x∗|2 ] = 0 ; Chung [ 1954 ] further calculates an exact convergence rate ( see ( 3.6 ) in Section 3 ) ; Blum [ 1954 ] presents an elegant proof that the convergence of xt to x∗ can hold almost surely . The asymptotic distribution of ( 2.1 ) can be attributed to [ Chung , 1954 , Theorem 6 ] , which states that the expression γ −1/2 t ( xt − x∗ ) converges weakly to a normal distribution . In their seminal works , Polyak and Juditsky [ 1992 ] and Ruppert [ 1988 ] independently introduce the concept of ‘ averaging the iterates ’ , xt = x0 + . . .+ xt−1 t , showing the striking result that √ t ( xt−x∗ ) converges weakly to a fixed normal distribution regardless of the choice of the step-size { γt } t∈N+ as long as it satisfies mild conditions . Recently , optimization algorithms that can handle heavy-tailed noise sequence { ξt } t∈N+ have been proposed [ Davis et al. , 2019 , Nazin et al. , 2019 , Gorbunov et al. , 2020 ] ; however , they still rely on a uniformly bounded variance assumption , hence do not cover our setting . Compared with the copious collection of theoretical studies on stochastic approximation algorithms with finite variance as mentioned above , papers that study stochastic approximation under infinite noise variance are extremely scarce ; we shall summarize only a few papers known to us . Krasulina [ 1969 ] is the first to consider such problems , proving almost sure and Lp convergence for the one-dimensional stochastic approximation process without variance . The weak convergence of the iterates ( without averaging ) t1/α ( xt − x∗ ) is also considered by Krasulina [ 1969 ] , but only for the fastest-decaying step-size γt = 1/t . Goodsell and Hanson [ 1976 ] discuss how xt → x∗ in probability can imply xt → x∗ almost surely , when no finite variance is assumed , and Li [ 1994 ] provides a necessary and sufficient condition for almost sure convergence of xt → x∗ , stating that faster-decaying step-size γt = o ( t−1/p ) is required when moments of lower orders E [ |ξt|p ] are not in place . Anantharam and Borkar [ 2012 ] show that although step-size that decays slower than t−1/p can not yield almost sure convergence , Lp convergence can still hold under what they call the ‘ stability assumption ’ , but their analysis technique provides no convergence rate . Recently , Şimşekli et al . [ 2019 ] and Zhang et al . [ 2020 ] considered SGD with heavy-tailed noise ξt having uniformly bounded p-th order moments . Besides not being able to handle state-dependent noise due to this uniform moment condition , Şimşekli et al . [ 2019 ] imposed further conditions on R = ∇f such as global Hölder continuity for a non-convex f , whereas Zhang et al . [ 2020 ] modified SGD with ‘ gradient clipping ’ , in order to be able to compensate the effects of the heavy-tailed noise . Finally , we shall mention that a class of stochastic recursions similar to ( 2.1 ) have been considered in the dynamical systems theory [ Mirek , 2011 , Buraczewski et al. , 2012 , 2016 ] , for which generalized central limit theorems with α-stable limits have been established . However , such techniques typically requireR to be ( asymptotically ) linear and the step-sizes to be constant as they heavily rely on the theory of time-homogeneous Markov processes . Hence , their approach does not readily generalize to the setting of our interest , i.e. , non-linearR and diminishing step-sizes , where the latter is crucial for ensuring convergence towards the global optimum . Stable Distributions . In probability theory , a random variable X is stable if its distribution is non-degenerate and satisfies the following property : Let X1 and X2 be independent copies of X . Then , for any constants a , b > 0 , the random variable aX1 + bX2 has the same distribution as cX+d for some constants c > 0 and d ( see e.g . [ Samorodnitsky and Taqqu , 1994 ] ) . The stable distribution is also referred to as the α-stable distribution , first proposed by Lévy [ 1937 ] , where α ∈ ( 0 , 2 ] denoting the stability parameter . The case α = 2 corresponds to the normal distribution , and the variance under this distribution is undefined for any α < 2 . The multivariate α-stable distribution dates back to Feldheim [ 1937 ] , which is a multivariate generalization of the univariate α-stable distribution , which is also uniquely characterized by its characteristic function . In particular , an Rn-valued random vector X has a multivariate α-stable distribution , denoted as X ∼ S ( α , Λ , δ ) if the joint characteristic function ofX is given by E [ exp ( iuTX ) ] = exp { − ∫ s∈S2 ( |uTs|α + iν ( uTs , α ) ) Λ ( ds ) + iuTδ } , for any u ∈ Rn , and 0 < α 6 2 . Here , α is the tail-index , Λ is a finite measure on S2 known as the spectral measure , δ ∈ Rn is a shift vector , and ν ( y , α ) : = − sign ( y ) tan ( πα/2 ) |y|α for α 6= 1 and ν ( y , α ) : = ( 2/π ) y log |y| for α = 1 for any y ∈ R , and S2 denotes the unit sphere in Rn ; i.e . S2 = { s ∈ Rn : |s| = 1 } . Stable distribution also emerges as the limit distribution in the Generalized Central Limit Theorem ( GCLT ) [ Gnedenko and Kolmogorov , 1954 ] , which states that for a sequence of i.i.d . random variables whose distributions have a power-law tail with index 0 < α < 2 , the normalized sum converges to an α-stable distribution as the number of summands go to∞ . Domains of Normal Attraction of Stable Distributions . LetX1 , X2 , . . . , Xt be an i.i.d . sequence of random vectors in Rn with a common distribution function F ( x ) . If there exists some constant a > 0 and a sequence bt ∈ Rn such that X1 + · · ·+Xt at1/α − bt D−−−→ t→∞ µ , ( 2.2 ) then F ( x ) is said to belong to the domain of normal attraction of the law µ , and α is the characteristic exponent of µ [ Gnedenko and Kolmogorov , 1954 , page 181 ] . If µ is an α-stable distribution , then we say F ( x ) belongs to the domain of normal attraction of an α-stable distribution . For example , the Pareto distribution belongs to the domain of normal attraction of an α-stable law . In Section C in the supplementary document , we provide more details as well as a sufficient and necessary conditions for being in the domain of normal attraction of an α-stable law .

Summary: This paper concerns the convergence rate of SGD with heavy-tail noise. In particular, the considered gradient noise consists of a finite-variance part (corresponds to the multiplicative noise in SGD for least square) and an infinite-variance part (corresponds to the additive noise in SGD for least square). The main results can be viewed as a law of large number and a central limit theorem in the absence of finite-variance. First the authors show that when the loss is "strongly convex" and "smooth" in the $p$-PD sense and the heavy-tail part noise has finite $p$-moment, SGD (with decaying stepsize) iterates converges to the correct optimum, and the convergence rate is justified in the p-norm sense. Moreover, if the heavy-tail noise is the domain of normal attraction of some stable distribution, then the properly normalized SGD iterates also converges to the stable distribution. Finally, the heavy-tailed noise model is justified from two linear model examples.