The Heavy-Tail Phenomenon in SGD

1 INTRODUCTION . The learning problem in neural networks can be expressed as an instance of the well-known population risk minimization problem in statistics , given as follows : minx∈Rd F ( x ) : = Ez∼D [ f ( x , z ) ] , ( 1.1 ) where z ∈ Rp denotes a random data point , D is a probability distribution on Rp that denotes the law of the data points , x ∈ Rd denotes the parameters of the neural network to be optimized , and f : Rd × Rp 7→ R+ denotes a measurable cost function , which is often non-convex in x . While this problem can not be attacked directly since D is typically unknown , if we have access to a training dataset S = { z1 , . . . , zn } with n independent and identically distributed ( i.i.d . ) observations , i.e. , zi ∼i.i.d . D for i = 1 , . . . , n , we can use the empirical risk minimization strategy , which aims at solving the following optimization problem ( Shalev-Shwartz & Ben-David , 2014 ) : minx∈Rd f ( x ) : = f ( x , S ) : = ( 1/n ) ∑n i=1 f ( i ) ( x ) , ( 1.2 ) where f ( i ) denotes the cost induced by the data point zi . The stochastic gradient descent ( SGD ) algorithm has been one of the most popular algorithms for addressing this problem : xk = xk−1 − η∇f̃k ( xk−1 ) , where ∇f̃k ( x ) : = ( 1/b ) ∑ i∈Ωk ∇f ( i ) ( x ) . ( 1.3 ) Here , k denotes the iterations , η > 0 is the stepsize ( also called the learning-rate ) , ∇f̃ is the stochastic gradient , b is the batch-size , and Ωk ⊂ { 1 , . . . , n } is a random subset with |Ωk| = b for all k. Even though the practical success of SGD has been proven in many domains , the theory for its generalization properties is still in an early phase . Among others , one peculiar property of SGD that has not been theoretically well-grounded is that , depending on the choice of η and b , the algorithm can exhibit significantly different behaviors in terms of the performance on unseen test data . A common perspective over this phenomenon is based on the ‘ flat minima ’ argument that dates back to Hochreiter & Schmidhuber ( 1997 ) , and associates the performance with the ‘ sharpness ’ or ‘ flatness ’ of the minimizers found by SGD , where these notions are often characterized by the magnitude of the eigenvalues of the Hessian , larger values corresponding to sharper local minima ( Keskar et al. , 2016 ) . Recently , Jastrzębski et al . ( 2017 ) focused on this phenomenon as well and empirically illustrated that the performance of SGD on unseen test data is mainly determined by the stepsize η and the batch-size b , i.e. , larger η/b yields better generalization . Revisiting the flat-minima argument , they concluded that the ratio η/b determines the flatness of the minima found by SGD ; hence the difference in generalization . In the same context , Şimşekli et al . ( 2019b ) focused on the statistical properties of the gradient noise ( ∇f̃k ( x ) −∇f ( x ) ) and illustrated that under an isotropic model , the gradient noise exhibits a heavy-tailed behavior , which was also confirmed in follow-up studies ( Zhang et al. , 2019 ) . Based on this observation and a metastability argument ( Pavlyukevich , 2007 ) , they showed that SGD will ‘ prefer ’ wider basins under the heavy-tailed noise assumption , without an explicit mention of the cause of the heavy-tailed behavior . In another recent study , Martin & Mahoney ( 2019 ) introduced a new approach for investigating the generalization properties of deep neural networks by invoking results from heavy-tailed random matrix theory . They empirically showed that the eigenvalues of the weight matrices in different layers exhibit a heavy-tailed behavior , which is an indication that the weight matrices themselves exhibit heavy tails as well ( Ben Arous & Guionnet , 2008 ) . Accordingly , they fitted a power law distribution to the empirical spectral density of individual layers and illustrated that heavier-tailed weight matrices indicate better generalization . Very recently , Şimşekli et al . ( 2020 ) formalized this argument in a mathematically rigorous framework and showed that such a heavy-tailed behavior diminishes the ‘ effective dimension ’ of the problem , which in turn results in improved generalization . While these studies form an important initial step towards establishing the connection between heavy tails and generalization , the originating cause of the observed heavy-tailed behavior is yet to be understood . In this paper , we argue that these three seemingly unrelated perspectives for generalization are deeply linked to each other . We claim that , depending on the choice of the algorithm parameters η and b , the dimension d , and the curvature of f ( to be precised in Section 3 ) , SGD exhibits a ‘ heavy-tail phenomenon ’ , meaning that the law of the iterates converges to a heavy-tailed distribution . We rigorously prove that , this phenomenon is not specific to deep learning and in fact it can be observed even in surprisingly simple settings : we show that when f is chosen as a simple quadratic function and the data points are i.i.d . from an isotropic Gaussian distribution , the iterates can still converge to a heavy-tailed distribution with arbitrarily heavy tails , hence with infinite variance . We summarize our contributions as follows : 1 . When f is a quadratic , we prove that : ( i ) the tails become monotonically heavier for increasing curvature , increasing η , or decreasing b , hence relating the heavy-tails to the ratio η/b and the curvature , ( ii ) the law of the iterates converges exponentially fast towards the stationary distribution in the Wasserstein metric , ( iii ) there exists a higher-order moment ( e.g. , variance ) of the iterates that diverges at most polynomially-fast , depending on the heaviness of the tails at stationarity . 2 . We support our theory with experiments conducted on both synthetic data and neural networks . Our experimental results confirm our theory on synthetic setups and also illustrate that the heavy-tail phenomenon is also observed in fully connected multi-layer neural networks . To the best of our knowledge , these results are the first of their kind to rigorously characterize the empirically observed heavy-tailed behavior of SGD with respect to η , b , d , and the curvature , with explicit convergence rates . 1 2 TECHNICAL BACKGROUND . Heavy-tailed distributions with a power-law decay . In probability theory , a real-valued random variable X is said to be heavy-tailed if the right tail or the left tail of the distribution decays slower 1We note that in a concurrent work , which very recently appeared on arXiv , Hodgkinson & Mahoney ( 2020 ) showed that heavy tails with power laws arise in more general Lipschitz stochastic optimization algorithms that are contracting on average for strongly convex objectives near infinity with positive probability . Our Theorem 1 and Lemma 14 are more refined as we focus on the special case of SGD with Gaussian data , where we are able to provide constants which explicitly determine the tail index as an expectation over data and SGD parameters ( see also eqn . ( 3.6 ) ) . Due to the generality of their framework , ( Hodgkinson & Mahoney , 2020 , Thm 1 ) is more implicit and it can not provide such a characterization of these constants , however it can be applied to other algorithms beyond SGD . All our other results ( including Theorem 2 – monotonicity of the tail-index and Corollary 9 – central limit theorem for the ergodic averages ) are all specific to SGD and can not be obtained under the framework of Hodgkinson & Mahoney ( 2020 ) . We encourage the readers to refer to ( Hodgkinson & Mahoney , 2020 ) for the treatment of more general stochastic recursions . than any exponential distribution . We say X has heavy ( right ) tail if limx→∞ P ( X ≥ x ) ecx =∞ for any c > 0 . 2 Similarly , an Rd-valued random vector X has heavy tail if uTX has heavy right tail for some vector u ∈ Sd−1 , where Sd−1 : = { u ∈ Rd : ‖u‖ = 1 } is the unit sphere in Rd . Heavy tail distributions include α-stable distributions , Pareto distribution , log-normal distribution and the Weilbull distribution . One important class of the heavy-tailed distributions is the distributions with power-law decay , which is the focus of our paper . That is , P ( X ≥ x ) ∼ c0x−α as x→∞ for some c0 > 0 and α > 0 , where α > 0 is known as the tail-index , which determines the tail thickness of the distribution . Similarly , we say that the random vector X has power-law decay with tail-index α if for some u ∈ Sd−1 , we have P ( uTX ≥ x ) ∼ c0x−α , for some c0 , α > 0 . Stable distributions . The class of α-stable distributions are an important subclass of heavy-tailed distributions with a power-law decay , which appears as the limiting distribution of the generalized CLT for a sum of i.i.d . random variables with infinite variance ( Lévy , 1937 ) . A random variable X follows a symmetric α-stable distribution denoted as X ∼ SαS ( σ ) if its characteristic function takes the form : E [ eitX ] = exp ( −σα|t|α ) , t ∈ R , where σ > 0 is the scale parameter that measures the spread of X around 0 , and α ∈ ( 0 , 2 ] is known as the tail-index , and SαS becomes heavier-tailed as α gets smaller . The probability density function of a symmetric α-stable distribution , α ∈ ( 0 , 2 ] , does not yield closed-form expression in general except for a few special cases . When α = 1 and α = 2 , SαS reduces to the Cauchy and the Gaussian distributions , respectively . When 0 < α < 2 , α-stable distributions have their moments being finite only up to the order α in the sense that E [ |X|p ] < ∞ if and only if p < α , which implies infinite variance . Wasserstein metric . For any p ≥ 1 , definePp ( Rd ) as the space consisting of all the Borel probability measures ν on Rd with the finite p-th moment ( based on the Euclidean norm ) . For any two Borel probability measures ν1 , ν2 ∈ Pp ( Rd ) , we define the standard p-Wasserstein metric ( Villani , 2009 ) : Wp ( ν1 , ν2 ) : = ( inf E [ ‖Z1 − Z2‖p ] ) 1/p , where the infimum is taken over all joint distributions of the random variables Z1 , Z2 with marginal distributions ν1 , ν2 respectively . 3 SETUP AND MAIN THEORETICAL RESULTS . Before stating our theoretical results in detail , let us informally motivate our main method of analysis . Suppose that the initial SGD iterate x0 is in the domain of attraction3 of a local minimum x ? of f and the function f is smooth and well-approximated by a quadratic function in this basin . Under this assumption , by considering a first-order Taylor approximation of ∇f ( i ) ( x ) around x ? , we have ∇f ( i ) ( x ) ≈ ∇f ( i ) ( x ? ) +∇2f ( i ) ( x ? ) ( x− x ? ) . By using this approximation , we can approximate the SGD recursion ( 1.3 ) as follows : xk ≈ xk−1 − ( η/b ) ∑ i∈Ωk ∇2f ( i ) ( x ? ) xk−1 + ( η/b ) ∑ i∈Ωk ( ∇2f ( i ) ( x ? ) x ? −∇f ( i ) ( x ? ) ) = : ( I − ( η/b ) Hk ) xk−1 + qk , ( 3.1 ) where I denotes the identity matrix of appropriate size . Here , our main observation is that the SGD recursion can be approximated by a linear stochastic recursion , which gives us access to the tools from implicit renewal theory for investigating its statistical properties ( Kesten , 1973 ; Goldie , 1991 ) . In a renewal theoretic context , the object of interest would be the matrix ( I − ηbHk ) , whose statistical properties determine the behavior of xk : depending on the moments of this matrix , xk can have heavy or light tails , or might even diverge . In this study , we focus on the tail behavior of the SGD dynamics by analyzing it through the lens of implicit renewal theory . As , the recursion ( 3.1 ) is obtained by a quadratic approximation of the component functions f ( i ) , which arises naturally in linear regression , we will consider a simplified setting and rigorously study this dynamics in the case of linear regression . We would like to underline that , in our analysis , the Taylor approximation ( 3.1 ) is not crucial . Indeed , we can easily extend our theory to more general non-linear recursions by imposing strict statistical assumptions on the loss function ( which can be chosen non-convex ) and the data distribution ( Mirek , 2011 ) . Unfortunately , such assumptions would either be trivially false for deep learning problems , 2A real-valued random variable X has heavy ( left ) tail if limx→∞ P ( X ≤ −x ) ec|x| =∞ for any c > 0 . 3We say x0 is in the domain of attraction of a local minimum x ? , if gradient descent iterations to minimize f started at x0 with sufficiently small stepsize converge to x ? as the number of iterations goes to infinity . e.g. , ( Mirek , 2011 ) or can not be verified in practice , e.g. , ( Hodgkinson & Mahoney , 2020 ) . In order to be able to provide explicit results with clear assumptions , we limit our scope to quadratic optimization , which turns out to be already fairly technical . We leave the analysis of the general case as a natural next step of our work . We now consider the case when f is a quadratic , which arises in linear regression : min x∈Rd F ( x ) : = ( 1/2 ) E ( a , y ) ∼D [ ( aTx− y ) 2 ] , ( 3.2 ) where the data ( a , y ) comes from an unknown distribution D with support Rd × R. Assume we have access to i.i.d . samples ( ai , yi ) from the distribution D where ∇f ( i ) ( x ) = ai ( aTi x − yi ) is an unbiased estimator of the true gradient ∇F ( x ) . The curvature , i.e . the value of second partial derivatives , of this objective around a minimum is determined by the Hessian matrix E ( aaT ) which depends on the distribution of a . In this setting , SGD with batch size b leads to the iterations xk = Mkxk−1 + qk with Mk : = I − ( η/b ) Hk , Hk : = ∑ i∈Ωk aia T i , qk : = ( η/b ) ∑ i∈Ωk aiyi , ( 3.3 ) where Ωk : = { b ( k − 1 ) + 1 , b ( k − 1 ) + 2 , . . . , bk } with |Ωk| = b . Here , for simplicity , we assume that we are in the one-pass regime ( also called the streaming setting ( Frostig et al. , 2015 ; Jain et al. , 2017 ) ) where each sample is used only once without being recycled . Our purpose in this paper is to show that heavy tails can arise in SGD even in simple settings such as when the input data ai is Gaussian , without the necessity to have a heavy-tailed input data4 . Consequently , we make the following assumptions on the data throughout the paper : ( A1 ) ai ∼ N ( 0 , σ2Id ) are i.i.d . ( A2 ) yi are i.i.d . with a continuous density whose support is R with all the moments finite . Assumption ( A2 ) would be satisfied in many cases , for instance when yi is normally distributed on R. Note that by Assumption ( A1 ) , the matrices Mk = I − ηbHk are i.i.d . and the Hessian matrix of the objective ( 3.2 ) satisfies E ( aaT ) = σ2Id where the value of σ2 determines the curvature around a minimum ; smaller ( larger ) σ2 implies the objective will grow slower ( faster ) around the minimum and the minimum will be flatter ( sharper ) ( see e.g . Dinh et al . ( 2017 ) ) . We introduce h ( s ) : = limk→∞ ( E‖MkMk−1 . . .M1‖s ) 1/k , ( 3.4 ) which arises in stochastic matrix recursions ( see e.g . Buraczewski et al . ( 2014 ) ) where ‖ · ‖ denotes the matrix 2-norm ( i.e . largest singular value of a matrix ) . Since E‖Mk‖s < ∞ for all k and s > 0 , we have h ( s ) < ∞ . Let us also define Πk : = MkMk−1 . . .M1 and ρ : = limk→∞ ( 2k ) −1 log ( largest eigenvalue of ΠTk Πk ) . ( 3.5 ) The latter quantity is called the top Lyapunov exponent of the stochastic recursion ( 3.3 ) . Furthermore , if ρ exists and is negative , it can be shown that a stationary distribution of the recursion ( 3.3 ) exists . In the Appendix ( see Lemma 14 ) , we show that under our assumptions , ρ = E log ‖ ( I − ( η/b ) H ) e1‖ , h ( s ) = E [ ‖ ( I − ( η/b ) H ) e1‖s ] for ρ < 0 , ( 3.6 ) where H is a matrix with the same distribution as Hk , and e1 is the first basis vector . In the following , we show that the limit density has a polynomial tail with a tail-index given precisely by α , the unique critical value such that h ( α ) = 1 . The result builds on adapting the techniques developed in stochastic matrix recursions ( Alsmeyer & Mentemeier , 2012 ; Buraczewski et al. , 2016 ) to our setting . Our result shows that even in the simplest setting when the input data is Gaussian without any heavy tail , SGD iterates can lead to a heavy-tailed stationary distribution with an infinite variance . To our knowledge , this is the first time such a phenomenon is proven in the linear regression setting . Theorem 1 . Consider the SGD iterations ( 3.3 ) . If ρ < 0 , then SGD iterations admit a unique stationary distribution x∞ which satisfy limt→∞ t αP ( uTx∞ > t ) = eα ( u ) , u ∈ Sd−1 , ( 3.7 ) for some positive and continuous function eα on the unit sphere Sd−1 , where α is the unique positive value such that h ( α ) = 1 . 4Note that if the input data is heavy-tailed , the stationary distribution of SGD automatically becomes heavytailed , see Buraczewski et al . ( 2012 ) for details . In our context , the challenge is to identify the occurrence of the heavy tails when the distribution of the input data is light-tailed , such as a simple Gaussian distribution . As Martin & Mahoney ( 2019 ) ; Şimşekli et al . ( 2020 ) provide both numerical and theoretical evidence showing that the tail-index of the density of the network weights is closely related to the generalization performance , where smaller tail-index indicates better generalization , a natural question of practical importance is how the tail-index depends on the parameters of the problem including the batch size , dimension and the stepsize . We prove that larger batch sizes lead to a lighter tail ( i.e . larger α ) , which links the heavy tails to the observation that smaller b yields improved generalization in a variety of settings in deep learning ( Keskar et al. , 2016 ; Panigrahi et al. , 2019 ; Martin & Mahoney , 2019 ) . We also prove that smaller stepsizes lead to larger α , hence lighter tails , which agrees with the fact that the existing literature for linear regression often choose η small enough to guarantee that variance of the iterates stay bounded ( Dieuleveut et al. , 2017b ; Jain et al. , 2017 ) . Theorem 2 . The tail-index α is strictly increasing in batch size b and strictly decreasing in stepsize η and variance σ2 provided that α ≥ 1 . Moreover , the tail-index α is strictly decreasing in dimension d. Next result characterizes the tail-index α depending on the choice of the batch size b , the variance σ2 , which determines the curvature around the minimum and the stepsize ; in particular we show that if the stepsize exceeds an explicit threshold , the stationary distribution will become heavy tailed with an infinite variance . Proposition 3 . Let ηcrit = 2bσ2 ( d+b+1 ) . The following holds : ( i ) There exists ηmax > ηcrit such that for any ηcrit < η < ηmax , Theorem 1 holds with tail index 0 < α < 2 . ( ii ) If η = ηcrit , Theorem 1 holds with tail index α = 2 . ( iii ) If η ∈ ( 0 , ηcrit ) , then Theorem 1 holds with tail index α > 2 . Relation to first exit times . Proposition 3 implies that , for fixed η and b , the tail-index α will be decreasing with increasing σ . Combined with the first-exit-time analyses of Şimşekli et al . ( 2019b ) ; Nguyen et al . ( 2019 ) , which state that the escape probability from a basin becomes higher for smaller α , our result implies that the probability of SGD escaping from a basin gets larger with increasing curvature ; hence providing an alternative view for the argument that SGD prefers flat minima . Three regimes for stepsize . Theorems 1-2 and Proposition 3 identify three regimes : ( I ) convergence to a limit with a finite variance if ρ < 0 and α > 2 ; ( II ) convergence to a heavy-tailed limit with infinite variance if ρ < 0 and α < 2 ; ( III ) ρ > 0 when convergence can not be guaranteed . For Gaussian input , if the stepsize is small enough , smaller than ηcrit , by Proposition 3 , ρ < 0 and α > 2 , therefore regime ( I ) applies . As we increase the stepsize , there is a critical stepsize level ηcrit for which η > ηcrit leads to α < 2 as long as η < ηmax where ηmax is the maximum allowed stepsize for ensuring convergence ( corresponds to ρ = 0 ) . A similar behavior with three ( learning rate ) stepsize regimes was reported in Lewkowycz et al . ( 2020 ) and derived analytically for one hidden layer linear networks with a large width . The large stepsize choices that avoids divergence , so called the catapult phase for the stepsize , yielded the best generalization performance empirically , driving the iterates to a flatter minima in practice . We suspect that the catapult phase in Lewkowycz et al . ( 2020 ) corresponds to regime ( II ) in our case , where the iterates are heavy-tailed , which might cause convergence to flatter minima as the first-exit-time discussions suggest ( Şimşekli et al. , 2019a ) . Moment Bounds and Convergence Speed . Theorem 1 is of asymptotic nature which characterizes the stationary distribution x∞ of SGD iterations with a tail-index α . Next , we provide non-asymptotic moment bounds for xk at each k-th iterate , and also for the limit x∞ . Theorem 4 . ( i ) If the tail-index α ≤ 1 , then for any p ∈ ( 0 , α ) , we have h ( p ) < 1 and E‖xk‖p ≤ ( h ( p ) ) kE‖x0‖p + 1− ( h ( p ) ) k 1− h ( p ) E‖q1‖p . ( 3.8 ) ( ii ) If the tail-index α > 1 , then for any p ∈ ( 1 , α ) , we have h ( p ) < 1 and for any > 0 such that ( 1 + ) h ( p ) < 1 , we have E‖xk‖p ≤ ( ( 1 + ) h ( p ) ) kE‖x0‖p + 1− ( ( 1 + ) h ( p ) ) k 1− ( 1 + ) h ( p ) ( 1 + ) p p−1 − ( 1 + ) ( ( 1 + ) 1 p−1 − 1 ) p E‖q1‖p . ( 3.9 ) Theorem 4 shows that when p < α the upper bound on the p-th moment of the iterates converges exponentially to the p-the moment of q1 when α ≤ 1 and a neighborhood of the p-moment of q1 when α > 1 , where q1 is defined in ( 3.3 ) . By letting k →∞ and applying Fatou ’ s lemma , we can also characterize the moments of the stationary distribution . Corollary 5 . ( i ) If the tail-index α ≤ 1 , then for any p ∈ ( 0 , α ) , E‖x∞‖p ≤ 11−h ( p ) E‖q1‖ p , where h ( p ) < 1 . ( ii ) If the tail-index α > 1 , then for any p ∈ ( 1 , α ) , we have h ( p ) < 1 and for any > 0 such that ( 1 + ) h ( p ) < 1 , we have E‖x∞‖p ≤ 11− ( 1+ ) h ( p ) ( 1+ ) p p−1− ( 1+ ) ( ( 1+ ) 1 p−1−1 ) p E‖q1‖p . Next , we will study the speed of convergence of the k-th iterate xk to its stationary distribution x∞ in the Wasserstein metricWp for any 1 ≤ p < α. Theorem 6 . Assume α > 1 . Let νk , ν∞ denote the probability laws of xk and x∞ respectively . Then Wp ( νk , ν∞ ) ≤ ( h ( p ) ) k/pWp ( ν0 , ν∞ ) , for any 1 ≤ p < α , where the convergence rate ( h ( p ) ) 1/p ∈ ( 0 , 1 ) . Theorem 6 shows that in case α < 2 the convergence to a heavy tailed distribution occurs relatively fast , i.e . with a linear convergence in the p-Wasserstein metric . We can also characterize the constant h ( p ) in Theorem 6 which controls the convergence rate as follows : Corollary 7 . When η < ηcrit = 2bσ2 ( d+b+1 ) , the tail-index α > 2 , W2 ( νk , ν∞ ) ≤ ( 1− 2ησ2 ( 1− η/ηcrit ) ) k/2W2 ( ν0 , ν∞ ) . ( 3.10 ) Theorem 6 works for any p < α . At the critical p = α , Theorem 1 indicates that E‖x∞‖α = ∞ , and therefore has E‖xk‖α →∞ as k →∞ , 5 which serves as an evidence that the tail gets heavier as the number of iterates k increases . By adapting the proof of Theorem 4 , we have the following result stating that the moments of the iterates of order α go to infinity but this speed can only be polynomially fast . Proposition 8 . Given the tail-index α , we have E‖x∞‖α =∞ . Moreover , E‖xk‖α = O ( k ) if α ≤ 1 , and E‖xk‖α = O ( kα ) if α > 1 . It may be possible to leverage recent results on the concentration of products of i.i.d . random matrices ( Huang et al. , 2020 ; Henriksen & Ward , 2020 ) to study the tail of xk for finite k , which can be a future research direction . Extension to non-Gaussian data . Our main purpose in Theorem 1 is to show that heavy tails can arise even in the simplest setting when the input is Gaussian . However , Proposition 1 extends naturally if the input ai is not necessarily Gaussian . For example , if we assume that the distribution of ai has support of Rd and has a finite second moment , it can be checked that our proof technique for Theorem 1 will be still applicable and Theorem 1 will hold with h ( s ) defined by ( 3.4 ) . The only difference is that when input is not Gaussian , the explicit formula ( 3.6 ) for h ( s ) will not hold as an equality but it will become an inequality , i.e . h ( s ) ≤ h̃ ( s ) : = E [ ‖ ( I − ( η/b ) H ) e1‖s ] , ( 3.11 ) where h ( s ) is defined by ( 3.4 ) . This inequality is just a consequence of sub-multiplicativity of matrix products appearing in ( 3.4 ) . If α̃ is such that h̃ ( α ) = 1 , then by ( 3.11 ) , α̃ is a lower bound on the tail index α that satisfies h ( α ) = 1 where h is defined as in ( 3.4 ) . In other words , when the input is not Gaussian , we have α̃ ≤ α and therefore α̃ serves as a lower bound on the tail index . Furthermore , Theorem 2 will also apply in the sense that α̃ . α̃ will be strictly increasing in batch size b and strictly increasing in stepsize η and variance σ2 provided that α̃ ≥ 1 . Extension to non-quadratic optimization . We note that extending our results beyond quadratic optimization is possible , if the gradients have asymptotic linear growth . For example , consider the cost F ( x ) = E [ ` ( aTx− y ) ] with loss function ` . The choice of ` ( z ) = ‖z‖2/2 is the standard linear regression setting where the gradient of F is an affine function of x and in this case ‖∇F ( x ) − Σx‖ is bounded if we choose Σ = E [ aaT ] . Theorem 1 will hold as long as there exists a matrix Σ such that ‖∇F ( x ) − Σx‖ stays bounded even if the function ` is not a quadratic function , the proof is straightforward and would be based on verifying that the conditions of ( Mirek , 2011 , Theorem 1.4 ) hold in the setting of Theorem 1 . This type of optimization problems arises for instance in robust regression where the objective is F ( x ) = E [ ( aTx− b ) 2 ] +λg ( x ) with a penalty function g ( x ) whose gradient is bounded and a tunable parameter λ . The boundedness of the gradient of g ( x ) results in at-most linear growth of g ( x ) and allows robustness to outliers where the parameter λ can be used to adjust the robustness level desired . Examples for the choice of g ( x ) include the smoothly clipped 5Otherwise , one can construct a subsequence xnk that is bounded in the space L α converging to x∞ which would be a contradiction . absolute deviation ( SCAD ) penalty ( Loh & Wainwright ( 2015 ) ) , Huber loss ( Huber , 1992 ) or the exponential squared loss when g ( x ) = 1− exp ( −‖x‖2/c ) where c is a tuning parameter . Generalized Central Limit Theorem for Ergodic Averages . When α > 2 , by Corollary 5 , second moment of the iterates xk are finite , in which case central limit theorem ( CLT ) says that if the cumulative sum of the iterates SK = ∑K k=1 xk is scaled properly , the resulting distribution is Gaussian . In the case where α < 2 , the variance of the iterates is not finite ; however in this case , we derive the following generalized CLT ( GCLT ) which says that if the iterates are properly scaled , the limit will be an α-stable distribution . This is stated in a more precise manner as follows . Corollary 9 . Assume ρ < 0 so that Theorem 1 holds . Then , we have the following : ( i ) If α ∈ ( 0 , 1 ) ∪ ( 1 , 2 ) , then there is a sequence dK = dK ( α ) and a function Cα : Sd−1 7→ C such that as K → ∞ the random variables K− 1α ( SK − dK ) converge in law to the α-stable random variable with characteristic function Υα ( tv ) = exp ( tαCα ( v ) ) , for t > 0 and v ∈ Sd−1 . ( ii ) If α = 1 , then there are functions ξ , τ : ( 0 , ∞ ) 7→ R andC1 : Sd−1 7→ C such that asK →∞ the random variables K−1SK −Kξ ( K−1 ) converge in law to the random variable with characteristic function Υ1 ( tv ) = exp ( tC1 ( v ) + it〈v , τ ( t ) 〉 ) , for t > 0 and v ∈ Sd−1 . ( iii ) If α = 2 , then there is a sequence dK = dK ( 2 ) and a function C2 : Sd−1 7→ R such that as K → ∞ the random variables ( K logK ) − 12 ( SK − dK ) converge in law to the random variable with characteristic function Υ2 ( tv ) = exp ( t2C2 ( v ) ) , for t > 0 and v ∈ Sd−1 . ( iv ) If α ∈ ( 0 , 1 ) , then dK = 0 , and if α ∈ ( 1 , 2 ] , then dK = Kx̄ , where x̄ = ∫ Rd xν∞ ( dx ) . In addition to its evident theoretical interest , Corollary 9 has also an important practical implication : estimating the tail-index of a generic heavy-tailed distribution is a challenging problem ( see e.g . Clauset et al . ( 2009 ) ; Goldstein et al . ( 2004 ) ; Bauke ( 2007 ) ) ; however , for the specific case of α-stable distributions , accurate and computationally efficient estimators , which do not require the knowledge of the functions Cα , τ , ξ , have been proposed ( Mohammadi et al. , 2015 ) . Thanks to Corollary 9 , we will be able to use such estimators in our numerical experiments , as we will detail in the next section . We finally note that the gradient noise in SGD is actually both multiplicative and additive ( Dieuleveut et al. , 2017b ; a ) ; a fact that is often discarded for simplifying the mathematical analysis . In the linear regression setting , we have shown that the multiplicative noise Mk is the main source of heavy-tails , where a deterministic Mk would not lead to heavy tails.6 In the light of our theory , in Appendix A , we discuss in detail the recently proposed stochastic differential equation ( SDE ) representations of SGD in continuous-time and argue that , compared to classical SDEs driven by a Brownian motion ( e.g. , ( Jastrzębski et al. , 2017 ; Cheng et al. , 2019 ) , SDEs driven by heavy-tailed α-stable Lévy processes ( e.g. , ( Şimşekli et al. , 2019b ) ) are more adequate when α < 2 .

Summary: The main theme of this work is to study conditions under which SGD iterations result in random variables with heavy-tail random distributions. Specifically they focus on the step size, batch size and problem dimension. First they show theoretical results showing how the tail-index of the distribution generated by SGD depends on the chosen step size, batch size and problem dimension.