The Heavy-Tail Phenomenon in SGD

1 INTRODUCTION . The learning problem in neural networks can be expressed as an instance of the well-known population risk minimization problem in statistics , given as follows : minx∈Rd F ( x ) : = Ez∼D [ f ( x , z ) ] , ( 1.1 ) where z ∈ Rp denotes a random data point , D is a probability distribution on Rp that denotes the law of the data points , x ∈ Rd denotes the parameters of the neural network to be optimized , and f : Rd × Rp 7→ R+ denotes a measurable cost function , which is often non-convex in x . While this problem can not be attacked directly since D is typically unknown , if we have access to a training dataset S = { z1 , . . . , zn } with n independent and identically distributed ( i.i.d . ) observations , i.e. , zi ∼i.i.d . D for i = 1 , . . . , n , we can use the empirical risk minimization strategy , which aims at solving the following optimization problem ( Shalev-Shwartz & Ben-David , 2014 ) : minx∈Rd f ( x ) : = f ( x , S ) : = ( 1/n ) ∑n i=1 f ( i ) ( x ) , ( 1.2 ) where f ( i ) denotes the cost induced by the data point zi . The stochastic gradient descent ( SGD ) algorithm has been one of the most popular algorithms for addressing this problem : xk = xk−1 − η∇f̃k ( xk−1 ) , where ∇f̃k ( x ) : = ( 1/b ) ∑ i∈Ωk ∇f ( i ) ( x ) . ( 1.3 ) Here , k denotes the iterations , η > 0 is the stepsize ( also called the learning-rate ) , ∇f̃ is the stochastic gradient , b is the batch-size , and Ωk ⊂ { 1 , . . . , n } is a random subset with |Ωk| = b for all k. Even though the practical success of SGD has been proven in many domains , the theory for its generalization properties is still in an early phase . Among others , one peculiar property of SGD that has not been theoretically well-grounded is that , depending on the choice of η and b , the algorithm can exhibit significantly different behaviors in terms of the performance on unseen test data . A common perspective over this phenomenon is based on the ‘ flat minima ’ argument that dates back to Hochreiter & Schmidhuber ( 1997 ) , and associates the performance with the ‘ sharpness ’ or ‘ flatness ’ of the minimizers found by SGD , where these notions are often characterized by the magnitude of the eigenvalues of the Hessian , larger values corresponding to sharper local minima ( Keskar et al. , 2016 ) . Recently , Jastrzębski et al . ( 2017 ) focused on this phenomenon as well and empirically illustrated that the performance of SGD on unseen test data is mainly determined by the stepsize η and the batch-size b , i.e. , larger η/b yields better generalization . Revisiting the flat-minima argument , they concluded that the ratio η/b determines the flatness of the minima found by SGD ; hence the difference in generalization . In the same context , Şimşekli et al . ( 2019b ) focused on the statistical properties of the gradient noise ( ∇f̃k ( x ) −∇f ( x ) ) and illustrated that under an isotropic model , the gradient noise exhibits a heavy-tailed behavior , which was also confirmed in follow-up studies ( Zhang et al. , 2019 ) . Based on this observation and a metastability argument ( Pavlyukevich , 2007 ) , they showed that SGD will ‘ prefer ’ wider basins under the heavy-tailed noise assumption , without an explicit mention of the cause of the heavy-tailed behavior . In another recent study , Martin & Mahoney ( 2019 ) introduced a new approach for investigating the generalization properties of deep neural networks by invoking results from heavy-tailed random matrix theory . They empirically showed that the eigenvalues of the weight matrices in different layers exhibit a heavy-tailed behavior , which is an indication that the weight matrices themselves exhibit heavy tails as well ( Ben Arous & Guionnet , 2008 ) . Accordingly , they fitted a power law distribution to the empirical spectral density of individual layers and illustrated that heavier-tailed weight matrices indicate better generalization . Very recently , Şimşekli et al . ( 2020 ) formalized this argument in a mathematically rigorous framework and showed that such a heavy-tailed behavior diminishes the ‘ effective dimension ’ of the problem , which in turn results in improved generalization . While these studies form an important initial step towards establishing the connection between heavy tails and generalization , the originating cause of the observed heavy-tailed behavior is yet to be understood . In this paper , we argue that these three seemingly unrelated perspectives for generalization are deeply linked to each other . We claim that , depending on the choice of the algorithm parameters η and b , the dimension d , and the curvature of f ( to be precised in Section 3 ) , SGD exhibits a ‘ heavy-tail phenomenon ’ , meaning that the law of the iterates converges to a heavy-tailed distribution . We rigorously prove that , this phenomenon is not specific to deep learning and in fact it can be observed even in surprisingly simple settings : we show that when f is chosen as a simple quadratic function and the data points are i.i.d . from an isotropic Gaussian distribution , the iterates can still converge to a heavy-tailed distribution with arbitrarily heavy tails , hence with infinite variance . We summarize our contributions as follows : 1 . When f is a quadratic , we prove that : ( i ) the tails become monotonically heavier for increasing curvature , increasing η , or decreasing b , hence relating the heavy-tails to the ratio η/b and the curvature , ( ii ) the law of the iterates converges exponentially fast towards the stationary distribution in the Wasserstein metric , ( iii ) there exists a higher-order moment ( e.g. , variance ) of the iterates that diverges at most polynomially-fast , depending on the heaviness of the tails at stationarity . 2 . We support our theory with experiments conducted on both synthetic data and neural networks . Our experimental results confirm our theory on synthetic setups and also illustrate that the heavy-tail phenomenon is also observed in fully connected multi-layer neural networks . To the best of our knowledge , these results are the first of their kind to rigorously characterize the empirically observed heavy-tailed behavior of SGD with respect to η , b , d , and the curvature , with explicit convergence rates . 1 2 TECHNICAL BACKGROUND . Heavy-tailed distributions with a power-law decay . In probability theory , a real-valued random variable X is said to be heavy-tailed if the right tail or the left tail of the distribution decays slower 1We note that in a concurrent work , which very recently appeared on arXiv , Hodgkinson & Mahoney ( 2020 ) showed that heavy tails with power laws arise in more general Lipschitz stochastic optimization algorithms that are contracting on average for strongly convex objectives near infinity with positive probability . Our Theorem 1 and Lemma 14 are more refined as we focus on the special case of SGD with Gaussian data , where we are able to provide constants which explicitly determine the tail index as an expectation over data and SGD parameters ( see also eqn . ( 3.6 ) ) . Due to the generality of their framework , ( Hodgkinson & Mahoney , 2020 , Thm 1 ) is more implicit and it can not provide such a characterization of these constants , however it can be applied to other algorithms beyond SGD . All our other results ( including Theorem 2 – monotonicity of the tail-index and Corollary 9 – central limit theorem for the ergodic averages ) are all specific to SGD and can not be obtained under the framework of Hodgkinson & Mahoney ( 2020 ) . We encourage the readers to refer to ( Hodgkinson & Mahoney , 2020 ) for the treatment of more general stochastic recursions . than any exponential distribution . We say X has heavy ( right ) tail if limx→∞ P ( X ≥ x ) ecx =∞ for any c > 0 . 2 Similarly , an Rd-valued random vector X has heavy tail if uTX has heavy right tail for some vector u ∈ Sd−1 , where Sd−1 : = { u ∈ Rd : ‖u‖ = 1 } is the unit sphere in Rd . Heavy tail distributions include α-stable distributions , Pareto distribution , log-normal distribution and the Weilbull distribution . One important class of the heavy-tailed distributions is the distributions with power-law decay , which is the focus of our paper . That is , P ( X ≥ x ) ∼ c0x−α as x→∞ for some c0 > 0 and α > 0 , where α > 0 is known as the tail-index , which determines the tail thickness of the distribution . Similarly , we say that the random vector X has power-law decay with tail-index α if for some u ∈ Sd−1 , we have P ( uTX ≥ x ) ∼ c0x−α , for some c0 , α > 0 . Stable distributions . The class of α-stable distributions are an important subclass of heavy-tailed distributions with a power-law decay , which appears as the limiting distribution of the generalized CLT for a sum of i.i.d . random variables with infinite variance ( Lévy , 1937 ) . A random variable X follows a symmetric α-stable distribution denoted as X ∼ SαS ( σ ) if its characteristic function takes the form : E [ eitX ] = exp ( −σα|t|α ) , t ∈ R , where σ > 0 is the scale parameter that measures the spread of X around 0 , and α ∈ ( 0 , 2 ] is known as the tail-index , and SαS becomes heavier-tailed as α gets smaller . The probability density function of a symmetric α-stable distribution , α ∈ ( 0 , 2 ] , does not yield closed-form expression in general except for a few special cases . When α = 1 and α = 2 , SαS reduces to the Cauchy and the Gaussian distributions , respectively . When 0 < α < 2 , α-stable distributions have their moments being finite only up to the order α in the sense that E [ |X|p ] < ∞ if and only if p < α , which implies infinite variance . Wasserstein metric . For any p ≥ 1 , definePp ( Rd ) as the space consisting of all the Borel probability measures ν on Rd with the finite p-th moment ( based on the Euclidean norm ) . For any two Borel probability measures ν1 , ν2 ∈ Pp ( Rd ) , we define the standard p-Wasserstein metric ( Villani , 2009 ) : Wp ( ν1 , ν2 ) : = ( inf E [ ‖Z1 − Z2‖p ] ) 1/p , where the infimum is taken over all joint distributions of the random variables Z1 , Z2 with marginal distributions ν1 , ν2 respectively . 3 SETUP AND MAIN THEORETICAL RESULTS . Before stating our theoretical results in detail , let us informally motivate our main method of analysis . Suppose that the initial SGD iterate x0 is in the domain of attraction3 of a local minimum x ? of f and the function f is smooth and well-approximated by a quadratic function in this basin . Under this assumption , by considering a first-order Taylor approximation of ∇f ( i ) ( x ) around x ? , we have ∇f ( i ) ( x ) ≈ ∇f ( i ) ( x ? ) +∇2f ( i ) ( x ? ) ( x− x ? ) . By using this approximation , we can approximate the SGD recursion ( 1.3 ) as follows : xk ≈ xk−1 − ( η/b ) ∑ i∈Ωk ∇2f ( i ) ( x ? ) xk−1 + ( η/b ) ∑ i∈Ωk ( ∇2f ( i ) ( x ? ) x ? −∇f ( i ) ( x ? ) ) = : ( I − ( η/b ) Hk ) xk−1 + qk , ( 3.1 ) where I denotes the identity matrix of appropriate size . Here , our main observation is that the SGD recursion can be approximated by a linear stochastic recursion , which gives us access to the tools from implicit renewal theory for investigating its statistical properties ( Kesten , 1973 ; Goldie , 1991 ) . In a renewal theoretic context , the object of interest would be the matrix ( I − ηbHk ) , whose statistical properties determine the behavior of xk : depending on the moments of this matrix , xk can have heavy or light tails , or might even diverge . In this study , we focus on the tail behavior of the SGD dynamics by analyzing it through the lens of implicit renewal theory . As , the recursion ( 3.1 ) is obtained by a quadratic approximation of the component functions f ( i ) , which arises naturally in linear regression , we will consider a simplified setting and rigorously study this dynamics in the case of linear regression . We would like to underline that , in our analysis , the Taylor approximation ( 3.1 ) is not crucial . Indeed , we can easily extend our theory to more general non-linear recursions by imposing strict statistical assumptions on the loss function ( which can be chosen non-convex ) and the data distribution ( Mirek , 2011 ) . Unfortunately , such assumptions would either be trivially false for deep learning problems , 2A real-valued random variable X has heavy ( left ) tail if limx→∞ P ( X ≤ −x ) ec|x| =∞ for any c > 0 . 3We say x0 is in the domain of attraction of a local minimum x ? , if gradient descent iterations to minimize f started at x0 with sufficiently small stepsize converge to x ? as the number of iterations goes to infinity . e.g. , ( Mirek , 2011 ) or can not be verified in practice , e.g. , ( Hodgkinson & Mahoney , 2020 ) . In order to be able to provide explicit results with clear assumptions , we limit our scope to quadratic optimization , which turns out to be already fairly technical . We leave the analysis of the general case as a natural next step of our work . We now consider the case when f is a quadratic , which arises in linear regression : min x∈Rd F ( x ) : = ( 1/2 ) E ( a , y ) ∼D [ ( aTx− y ) 2 ] , ( 3.2 ) where the data ( a , y ) comes from an unknown distribution D with support Rd × R. Assume we have access to i.i.d . samples ( ai , yi ) from the distribution D where ∇f ( i ) ( x ) = ai ( aTi x − yi ) is an unbiased estimator of the true gradient ∇F ( x ) . The curvature , i.e . the value of second partial derivatives , of this objective around a minimum is determined by the Hessian matrix E ( aaT ) which depends on the distribution of a . In this setting , SGD with batch size b leads to the iterations xk = Mkxk−1 + qk with Mk : = I − ( η/b ) Hk , Hk : = ∑ i∈Ωk aia T i , qk : = ( η/b ) ∑ i∈Ωk aiyi , ( 3.3 ) where Ωk : = { b ( k − 1 ) + 1 , b ( k − 1 ) + 2 , . . . , bk } with |Ωk| = b . Here , for simplicity , we assume that we are in the one-pass regime ( also called the streaming setting ( Frostig et al. , 2015 ; Jain et al. , 2017 ) ) where each sample is used only once without being recycled . Our purpose in this paper is to show that heavy tails can arise in SGD even in simple settings such as when the input data ai is Gaussian , without the necessity to have a heavy-tailed input data4 . Consequently , we make the following assumptions on the data throughout the paper : ( A1 ) ai ∼ N ( 0 , σ2Id ) are i.i.d . ( A2 ) yi are i.i.d . with a continuous density whose support is R with all the moments finite . Assumption ( A2 ) would be satisfied in many cases , for instance when yi is normally distributed on R. Note that by Assumption ( A1 ) , the matrices Mk = I − ηbHk are i.i.d . and the Hessian matrix of the objective ( 3.2 ) satisfies E ( aaT ) = σ2Id where the value of σ2 determines the curvature around a minimum ; smaller ( larger ) σ2 implies the objective will grow slower ( faster ) around the minimum and the minimum will be flatter ( sharper ) ( see e.g . Dinh et al . ( 2017 ) ) . We introduce h ( s ) : = limk→∞ ( E‖MkMk−1 . . .M1‖s ) 1/k , ( 3.4 ) which arises in stochastic matrix recursions ( see e.g . Buraczewski et al . ( 2014 ) ) where ‖ · ‖ denotes the matrix 2-norm ( i.e . largest singular value of a matrix ) . Since E‖Mk‖s < ∞ for all k and s > 0 , we have h ( s ) < ∞ . Let us also define Πk : = MkMk−1 . . .M1 and ρ : = limk→∞ ( 2k ) −1 log ( largest eigenvalue of ΠTk Πk ) . ( 3.5 ) The latter quantity is called the top Lyapunov exponent of the stochastic recursion ( 3.3 ) . Furthermore , if ρ exists and is negative , it can be shown that a stationary distribution of the recursion ( 3.3 ) exists . In the Appendix ( see Lemma 14 ) , we show that under our assumptions , ρ = E log ‖ ( I − ( η/b ) H ) e1‖ , h ( s ) = E [ ‖ ( I − ( η/b ) H ) e1‖s ] for ρ < 0 , ( 3.6 ) where H is a matrix with the same distribution as Hk , and e1 is the first basis vector . In the following , we show that the limit density has a polynomial tail with a tail-index given precisely by α , the unique critical value such that h ( α ) = 1 . The result builds on adapting the techniques developed in stochastic matrix recursions ( Alsmeyer & Mentemeier , 2012 ; Buraczewski et al. , 2016 ) to our setting . Our result shows that even in the simplest setting when the input data is Gaussian without any heavy tail , SGD iterates can lead to a heavy-tailed stationary distribution with an infinite variance . To our knowledge , this is the first time such a phenomenon is proven in the linear regression setting . Theorem 1 . Consider the SGD iterations ( 3.3 ) . If ρ < 0 , then SGD iterations admit a unique stationary distribution x∞ which satisfy limt→∞ t αP ( uTx∞ > t ) = eα ( u ) , u ∈ Sd−1 , ( 3.7 ) for some positive and continuous function eα on the unit sphere Sd−1 , where α is the unique positive value such that h ( α ) = 1 . 4Note that if the input data is heavy-tailed , the stationary distribution of SGD automatically becomes heavytailed , see Buraczewski et al . ( 2012 ) for details . In our context , the challenge is to identify the occurrence of the heavy tails when the distribution of the input data is light-tailed , such as a simple Gaussian distribution . As Martin & Mahoney ( 2019 ) ; Şimşekli et al . ( 2020 ) provide both numerical and theoretical evidence showing that the tail-index of the density of the network weights is closely related to the generalization performance , where smaller tail-index indicates better generalization , a natural question of practical importance is how the tail-index depends on the parameters of the problem including the batch size , dimension and the stepsize . We prove that larger batch sizes lead to a lighter tail ( i.e . larger α ) , which links the heavy tails to the observation that smaller b yields improved generalization in a variety of settings in deep learning ( Keskar et al. , 2016 ; Panigrahi et al. , 2019 ; Martin & Mahoney , 2019 ) . We also prove that smaller stepsizes lead to larger α , hence lighter tails , which agrees with the fact that the existing literature for linear regression often choose η small enough to guarantee that variance of the iterates stay bounded ( Dieuleveut et al. , 2017b ; Jain et al. , 2017 ) . Theorem 2 . The tail-index α is strictly increasing in batch size b and strictly decreasing in stepsize η and variance σ2 provided that α ≥ 1 . Moreover , the tail-index α is strictly decreasing in dimension d. Next result characterizes the tail-index α depending on the choice of the batch size b , the variance σ2 , which determines the curvature around the minimum and the stepsize ; in particular we show that if the stepsize exceeds an explicit threshold , the stationary distribution will become heavy tailed with an infinite variance . Proposition 3 . Let ηcrit = 2bσ2 ( d+b+1 ) . The following holds : ( i ) There exists ηmax > ηcrit such that for any ηcrit < η < ηmax , Theorem 1 holds with tail index 0 < α < 2 . ( ii ) If η = ηcrit , Theorem 1 holds with tail index α = 2 . ( iii ) If η ∈ ( 0 , ηcrit ) , then Theorem 1 holds with tail index α > 2 . Relation to first exit times . Proposition 3 implies that , for fixed η and b , the tail-index α will be decreasing with increasing σ . Combined with the first-exit-time analyses of Şimşekli et al . ( 2019b ) ; Nguyen et al . ( 2019 ) , which state that the escape probability from a basin becomes higher for smaller α , our result implies that the probability of SGD escaping from a basin gets larger with increasing curvature ; hence providing an alternative view for the argument that SGD prefers flat minima . Three regimes for stepsize . Theorems 1-2 and Proposition 3 identify three regimes : ( I ) convergence to a limit with a finite variance if ρ < 0 and α > 2 ; ( II ) convergence to a heavy-tailed limit with infinite variance if ρ < 0 and α < 2 ; ( III ) ρ > 0 when convergence can not be guaranteed . For Gaussian input , if the stepsize is small enough , smaller than ηcrit , by Proposition 3 , ρ < 0 and α > 2 , therefore regime ( I ) applies . As we increase the stepsize , there is a critical stepsize level ηcrit for which η > ηcrit leads to α < 2 as long as η < ηmax where ηmax is the maximum allowed stepsize for ensuring convergence ( corresponds to ρ = 0 ) . A similar behavior with three ( learning rate ) stepsize regimes was reported in Lewkowycz et al . ( 2020 ) and derived analytically for one hidden layer linear networks with a large width . The large stepsize choices that avoids divergence , so called the catapult phase for the stepsize , yielded the best generalization performance empirically , driving the iterates to a flatter minima in practice . We suspect that the catapult phase in Lewkowycz et al . ( 2020 ) corresponds to regime ( II ) in our case , where the iterates are heavy-tailed , which might cause convergence to flatter minima as the first-exit-time discussions suggest ( Şimşekli et al. , 2019a ) . Moment Bounds and Convergence Speed . Theorem 1 is of asymptotic nature which characterizes the stationary distribution x∞ of SGD iterations with a tail-index α . Next , we provide non-asymptotic moment bounds for xk at each k-th iterate , and also for the limit x∞ . Theorem 4 . ( i ) If the tail-index α ≤ 1 , then for any p ∈ ( 0 , α ) , we have h ( p ) < 1 and E‖xk‖p ≤ ( h ( p ) ) kE‖x0‖p + 1− ( h ( p ) ) k 1− h ( p ) E‖q1‖p . ( 3.8 ) ( ii ) If the tail-index α > 1 , then for any p ∈ ( 1 , α ) , we have h ( p ) < 1 and for any > 0 such that ( 1 + ) h ( p ) < 1 , we have E‖xk‖p ≤ ( ( 1 + ) h ( p ) ) kE‖x0‖p + 1− ( ( 1 + ) h ( p ) ) k 1− ( 1 + ) h ( p ) ( 1 + ) p p−1 − ( 1 + ) ( ( 1 + ) 1 p−1 − 1 ) p E‖q1‖p . ( 3.9 ) Theorem 4 shows that when p < α the upper bound on the p-th moment of the iterates converges exponentially to the p-the moment of q1 when α ≤ 1 and a neighborhood of the p-moment of q1 when α > 1 , where q1 is defined in ( 3.3 ) . By letting k →∞ and applying Fatou ’ s lemma , we can also characterize the moments of the stationary distribution . Corollary 5 . ( i ) If the tail-index α ≤ 1 , then for any p ∈ ( 0 , α ) , E‖x∞‖p ≤ 11−h ( p ) E‖q1‖ p , where h ( p ) < 1 . ( ii ) If the tail-index α > 1 , then for any p ∈ ( 1 , α ) , we have h ( p ) < 1 and for any > 0 such that ( 1 + ) h ( p ) < 1 , we have E‖x∞‖p ≤ 11− ( 1+ ) h ( p ) ( 1+ ) p p−1− ( 1+ ) ( ( 1+ ) 1 p−1−1 ) p E‖q1‖p . Next , we will study the speed of convergence of the k-th iterate xk to its stationary distribution x∞ in the Wasserstein metricWp for any 1 ≤ p < α. Theorem 6 . Assume α > 1 . Let νk , ν∞ denote the probability laws of xk and x∞ respectively . Then Wp ( νk , ν∞ ) ≤ ( h ( p ) ) k/pWp ( ν0 , ν∞ ) , for any 1 ≤ p < α , where the convergence rate ( h ( p ) ) 1/p ∈ ( 0 , 1 ) . Theorem 6 shows that in case α < 2 the convergence to a heavy tailed distribution occurs relatively fast , i.e . with a linear convergence in the p-Wasserstein metric . We can also characterize the constant h ( p ) in Theorem 6 which controls the convergence rate as follows : Corollary 7 . When η < ηcrit = 2bσ2 ( d+b+1 ) , the tail-index α > 2 , W2 ( νk , ν∞ ) ≤ ( 1− 2ησ2 ( 1− η/ηcrit ) ) k/2W2 ( ν0 , ν∞ ) . ( 3.10 ) Theorem 6 works for any p < α . At the critical p = α , Theorem 1 indicates that E‖x∞‖α = ∞ , and therefore has E‖xk‖α →∞ as k →∞ , 5 which serves as an evidence that the tail gets heavier as the number of iterates k increases . By adapting the proof of Theorem 4 , we have the following result stating that the moments of the iterates of order α go to infinity but this speed can only be polynomially fast . Proposition 8 . Given the tail-index α , we have E‖x∞‖α =∞ . Moreover , E‖xk‖α = O ( k ) if α ≤ 1 , and E‖xk‖α = O ( kα ) if α > 1 . It may be possible to leverage recent results on the concentration of products of i.i.d . random matrices ( Huang et al. , 2020 ; Henriksen & Ward , 2020 ) to study the tail of xk for finite k , which can be a future research direction . Extension to non-Gaussian data . Our main purpose in Theorem 1 is to show that heavy tails can arise even in the simplest setting when the input is Gaussian . However , Proposition 1 extends naturally if the input ai is not necessarily Gaussian . For example , if we assume that the distribution of ai has support of Rd and has a finite second moment , it can be checked that our proof technique for Theorem 1 will be still applicable and Theorem 1 will hold with h ( s ) defined by ( 3.4 ) . The only difference is that when input is not Gaussian , the explicit formula ( 3.6 ) for h ( s ) will not hold as an equality but it will become an inequality , i.e . h ( s ) ≤ h̃ ( s ) : = E [ ‖ ( I − ( η/b ) H ) e1‖s ] , ( 3.11 ) where h ( s ) is defined by ( 3.4 ) . This inequality is just a consequence of sub-multiplicativity of matrix products appearing in ( 3.4 ) . If α̃ is such that h̃ ( α ) = 1 , then by ( 3.11 ) , α̃ is a lower bound on the tail index α that satisfies h ( α ) = 1 where h is defined as in ( 3.4 ) . In other words , when the input is not Gaussian , we have α̃ ≤ α and therefore α̃ serves as a lower bound on the tail index . Furthermore , Theorem 2 will also apply in the sense that α̃ . α̃ will be strictly increasing in batch size b and strictly increasing in stepsize η and variance σ2 provided that α̃ ≥ 1 . Extension to non-quadratic optimization . We note that extending our results beyond quadratic optimization is possible , if the gradients have asymptotic linear growth . For example , consider the cost F ( x ) = E [ ` ( aTx− y ) ] with loss function ` . The choice of ` ( z ) = ‖z‖2/2 is the standard linear regression setting where the gradient of F is an affine function of x and in this case ‖∇F ( x ) − Σx‖ is bounded if we choose Σ = E [ aaT ] . Theorem 1 will hold as long as there exists a matrix Σ such that ‖∇F ( x ) − Σx‖ stays bounded even if the function ` is not a quadratic function , the proof is straightforward and would be based on verifying that the conditions of ( Mirek , 2011 , Theorem 1.4 ) hold in the setting of Theorem 1 . This type of optimization problems arises for instance in robust regression where the objective is F ( x ) = E [ ( aTx− b ) 2 ] +λg ( x ) with a penalty function g ( x ) whose gradient is bounded and a tunable parameter λ . The boundedness of the gradient of g ( x ) results in at-most linear growth of g ( x ) and allows robustness to outliers where the parameter λ can be used to adjust the robustness level desired . Examples for the choice of g ( x ) include the smoothly clipped 5Otherwise , one can construct a subsequence xnk that is bounded in the space L α converging to x∞ which would be a contradiction . absolute deviation ( SCAD ) penalty ( Loh & Wainwright ( 2015 ) ) , Huber loss ( Huber , 1992 ) or the exponential squared loss when g ( x ) = 1− exp ( −‖x‖2/c ) where c is a tuning parameter . Generalized Central Limit Theorem for Ergodic Averages . When α > 2 , by Corollary 5 , second moment of the iterates xk are finite , in which case central limit theorem ( CLT ) says that if the cumulative sum of the iterates SK = ∑K k=1 xk is scaled properly , the resulting distribution is Gaussian . In the case where α < 2 , the variance of the iterates is not finite ; however in this case , we derive the following generalized CLT ( GCLT ) which says that if the iterates are properly scaled , the limit will be an α-stable distribution . This is stated in a more precise manner as follows . Corollary 9 . Assume ρ < 0 so that Theorem 1 holds . Then , we have the following : ( i ) If α ∈ ( 0 , 1 ) ∪ ( 1 , 2 ) , then there is a sequence dK = dK ( α ) and a function Cα : Sd−1 7→ C such that as K → ∞ the random variables K− 1α ( SK − dK ) converge in law to the α-stable random variable with characteristic function Υα ( tv ) = exp ( tαCα ( v ) ) , for t > 0 and v ∈ Sd−1 . ( ii ) If α = 1 , then there are functions ξ , τ : ( 0 , ∞ ) 7→ R andC1 : Sd−1 7→ C such that asK →∞ the random variables K−1SK −Kξ ( K−1 ) converge in law to the random variable with characteristic function Υ1 ( tv ) = exp ( tC1 ( v ) + it〈v , τ ( t ) 〉 ) , for t > 0 and v ∈ Sd−1 . ( iii ) If α = 2 , then there is a sequence dK = dK ( 2 ) and a function C2 : Sd−1 7→ R such that as K → ∞ the random variables ( K logK ) − 12 ( SK − dK ) converge in law to the random variable with characteristic function Υ2 ( tv ) = exp ( t2C2 ( v ) ) , for t > 0 and v ∈ Sd−1 . ( iv ) If α ∈ ( 0 , 1 ) , then dK = 0 , and if α ∈ ( 1 , 2 ] , then dK = Kx̄ , where x̄ = ∫ Rd xν∞ ( dx ) . In addition to its evident theoretical interest , Corollary 9 has also an important practical implication : estimating the tail-index of a generic heavy-tailed distribution is a challenging problem ( see e.g . Clauset et al . ( 2009 ) ; Goldstein et al . ( 2004 ) ; Bauke ( 2007 ) ) ; however , for the specific case of α-stable distributions , accurate and computationally efficient estimators , which do not require the knowledge of the functions Cα , τ , ξ , have been proposed ( Mohammadi et al. , 2015 ) . Thanks to Corollary 9 , we will be able to use such estimators in our numerical experiments , as we will detail in the next section . We finally note that the gradient noise in SGD is actually both multiplicative and additive ( Dieuleveut et al. , 2017b ; a ) ; a fact that is often discarded for simplifying the mathematical analysis . In the linear regression setting , we have shown that the multiplicative noise Mk is the main source of heavy-tails , where a deterministic Mk would not lead to heavy tails.6 In the light of our theory , in Appendix A , we discuss in detail the recently proposed stochastic differential equation ( SDE ) representations of SGD in continuous-time and argue that , compared to classical SDEs driven by a Brownian motion ( e.g. , ( Jastrzębski et al. , 2017 ; Cheng et al. , 2019 ) , SDEs driven by heavy-tailed α-stable Lévy processes ( e.g. , ( Şimşekli et al. , 2019b ) ) are more adequate when α < 2 .

Summary: This paper studies the relations between the heavy tail phenomenon of SGD and the ‘flatness’ of the local minimum found by SGD and the ratio of the step size $\eta$ to the batch size $b$ for the quadratic and convex problem. They show that depending on the curvature, the step size, and the batch size, the iterates can converge to a heavy-tailed random variable.  They conduct experiments on both synthetic data and fully connected neural networks, and illustrate that the results would also apply to more general settings and hence provide new insights about the behavior of SGD in deep learning.