Title: Emergence of heavy tails in homogenized stochastic gradient descent
Venue: NeurIPS 2024
Accepted: yes
Keywords: stochastic gradient descent

heavy tails

gradient noise

Pearson diffusion

tail index
Abstract: It has repeatedly been observed that loss minimization by stochastic gradient descent (SGD) leads to heavy-tailed distributions of neural network parameters. Here, we analyze a continuous diffusion approximation of SGD, called homogenized stochastic gradient descent (hSGD), and show in a regularized linear regression framework that it leads to an asymptotically heavy-tailed parameter distribution, even though local gradient noise is Gaussian. We give explicit upper and lower bounds on the tail-index of the resulting parameter distribution and validate these bounds in numerical experiments. Moreover, the explicit form of these bounds enables us to quantify the interplay between optimization hyperparameters and the tail-index. Doing so, we contribute to the ongoing discussion on links between heavy tails and the generalization performance of neural networks as well as the ability of SGD to avoid suboptimal local minima.

Emergence of heavy tails in homogenized stochastic gradient descent

Abstract

It has repeatedly been observed that loss minimization by stochastic gradient descent (SGD) leads to heavy-tailed distributions of neural network parameters. Here, we analyze a continuous diffusion approximation of SGD, called homogenized stochastic gradient descent, and show in a regularized linear regression framework that it leads to an asymptotically heavy-tailed parameter distribution, even though local gradient noise is Gaussian. We give explicit upper and lower bounds on the tail-index of the resulting parameter distribution and validate these bounds in numerical experiments. Moreover, the explicit form of these bounds enables us to quantify the interplay between optimization hyperparameters and the tail-index. Doing so, we contribute to the ongoing discussion on links between heavy tails and the generalization performance of neural networks as well as the ability of SGD to avoid suboptimal local minima.

1 Introduction

Stochastic gradient descent (SGD) is the cornerstone of optimization in modern deep learning (cf. [Bottou et al., 2018]). In contrast to deterministic methods, it introduces stochasticity to the optimization procedure and therefore has to be analyzed from a probabilistic viewpoint. For instance, it has been observed by Martin and Mahoney [2019], Simsekli et al. [2019], Hodgkinson and Mahoney [2021], Gurbuzbalaban et al. [2021] and others, that the distributions of neural network parameters under loss minimization by SGD are typically heavy-tailed . This heavy-tailed behavior has been linked to the generalization performance of neural networks: Simsekli et al. [2019] give evidence that the extreme realizations of heavy-tailed random variables allows SGD to escape local minima of the loss landscape, and Hodgkinson and Mahoney [2021] argue for a negative correlation between the parameter distributions's tail-index and the network's generalization performance. 2 For these reasons, it is important to understand the origin and effects of heavy-tailed behavior of neural network parameters in SGD. An important step in this direction has been taken in [Gurbuzbalaban et al., 2021], where the tail behavior of SGD iterates is characterized in dependence on optimization parameters, dimension and Hessian curvature at the loss minimum. One limitation of [Gurbuzbalaban et al., 2021] is that this link is described only qualitatively, but not quantitatively. Here, we provide an alternative approach through analyzing homogenized stochastic gradient descent, a diffusion approximation of SGD introduced in [Paquette et al., 2022b, Mori et al., 2022]. Leveraging Itô calculus for diffusion processes, we are able to provide more precise bounds and estimates of the tail behavior of SGD iterates, which we subsequently validate in numerical experiments.

1.1 Our contribution

Our contribution to the analysis of heavy-tailed phenomena in SGD can be summarized as follows:

We introduce a new method, namely comparison results in convex stochastic order for homogenized stochastic gradient descent. These comparison results, given in Section 3 allow us to link SGD to the well-studied class of Pearson Diffusions (cf. [Forman and Sørensen, 2008]) and then to obtain bounds for their tail-index.

Contrary to [Gurbuzbalaban et al., 2021], who describe the tail-index only implicitly (observing phase-transitions between different regimes) our tail-index bounds are fully explicit. Moreover, their explicit form is validated in numerical experiments in Section 4.

Our results suggest (skew) Studentt -distributions as surrogate for parameter distributions in neural networks under SGD, in contrast to the earlier work of [Gurbuzbalaban et al., 2021] where α -stable distributions have been suggested. This proposal is validated by numerical experiments and statistical test in Section 4.

Finally, our results challenge the claim that the 'observed heavy-tailed behavior of SGD in practice cannot be accurately represented by an SDE driven by a Brownian motion' put forward in [Simsekli et al., 2020]. Our modeling approach is based on hSGD - an SDE driven by Brownian motion - which asymptotically exhibits heavy-tailed behavior with a tail-index that, in experiments, closely matches the empirical tail index of SGD iterates on real data.

2 Background

2.1 Empirical risk minimization

The general framework for training deep neural networks is to solve the problem of empirical risk minimization

where L i denotes the loss induced by the data point a i ∈ R d with label/response b i ∈ R , given the model's parameter vector x ∈ R d . For our theoretical and numerical analysis of heavy-tailed phenomena we focus on the specific case of regularized linear regression. Hence, as in [Gurbuzbalaban et al., 2021], we assume a quadratic structure of L i ( x ) , setting

Including a regularization term weighted by δ ≥ 0 , we arrive at the objective function

which is the loss function of ridge regression (cf. [Hastie et al., 2009]). We arrange the training data into a design matrix A ∈ R n × d and label vector b ∈ R n , whose i -th row are given by a i and b i respectively, allowing the write ( δ -ERM) as s

with gradient given by ∇ L reg ( x ) = 1 n ( A glyph[latticetop] ( Ax -b ) + δx ) .

2.2 Stochastic gradient descent

The standard approach to solve the problem of empirical risk minimization in deep learning is to use stochastic gradient descent (SGD) or any of its generalizations involving momentum, adaptive learning rates, gradient rescaling, etc. (cf. [Goodfellow et al., 2016, Bottou et al., 2018]). As a first step, we consider plain SGD with constant learning rate γ , which can be written in recursive form as

where ∇ L reg Ω k ( x k ) = 1 B ∑ i ∈ Ω k L reg i ( x ) and Ω k is a batch of size B glyph[greaterorequalslant] 1 sampled uniformly and independently from { 1 , · · · , n } . It will be convenient to rewrite (SGD) as

where the gradient noise is given by

Note that the gradient noise is unbiased (i.e. E ε ( x ) = 0 ) with covariance matrix given by 3

The theoretical properties of SGD can now be either analysed directly through the stochastic recurrence (1) (cf. [Bottou et al., 2018]) or through a continuous diffusion approximation, known in the general case as stochastic modified equation , cf. [Mandt et al., 2016, Li et al., 2017]. This approximation is obtained by recognizing (1) as the Euler-Maruyama approximation (in the small learning-rate regime) of the stochastic differential equation (SDE)

driven by a d -dimensional Brownian motion ( W t ) t ≥ 0 ; cf. Thm. 1 in [Li et al., 2017]. A common further simplification is to assume that the covariance matrix C ( x ) is constant, yielding the OrnsteinUhlenbeck-approximation (also known as Langevin equation) of SGD, cf. [Mandt et al., 2016, Li et al., 2017].

2.3 Homogenized Stochastic Gradient Descent

Our analysis of SGD is based on homogenized stochastic gradient descent (hSGD), introduced concurrently in [Paquette et al., 2022a] and [Mori et al., 2022], which is another approximation of (SME). In contrast to the Ornstein-Uhlenbeck-approximation where the covariance matrix of gradient noise is assumed constant, hSGD uses the more elaborate 'decoupling approximation'

see [Paquette et al., 2022a] and [Mori et al., 2022] for a derivation. Hence, in our notation, hSGD for penalized empirical risk minimization is given by 4

In the regime where n and d are simultaneously large, and under certain assumptions on the distribution of the data A and b , [Paquette et al., 2022a] provide approximation guarantees of the following form: For any given T > 0 and D > 0 , there is a C > 0 , such that

for quadratic statistics R : R d → R and when n ≥ d glyph[epsilon1] for some glyph[epsilon1] > 0 ; cf. Thm. 1.3 in [Paquette et al., 2022a] for details. Further empirical evidence for the approximation quality of hSGD with respect to SGD can is also given in [Paquette et al., 2022a, Mori et al., 2022], altogether providing a sufficient basis for analyzing the properties of SGD through hSGD.

Furthermore, the stochastic differential equation (hSGD) can be simplified by using the reduced singular value decomposition (SVD) of the design matrix A . In detail, let r = rank ( A ) glyph[lessorequalslant] d , and let A = P Σ Q glyph[latticetop] be the reduced SVD of A , where Q is d -byr and satisfies Q glyph[latticetop] Q = I r , P is n -byr and satisfies P glyph[latticetop] P = I r , and

is the diagonal matrix of non-zero singular values of A . We distinguish the following two cases of hSGD:

Underparametrized hSGD : Ax = b has no exact solution,

Overparametrized hSGD : Ax = b has an exact solution,

and impose the following assumption:

Assumption 2.1. In the overparametrized case, we require δ > 0 , i.e. the loss function must be regularized.

It is easily verified that x ∗ = Q Σ -1 P glyph[latticetop] b is the unique global minimum of the unregularized loss in the underparametrized case and the global minimum of smallest norm in the overparametrized case. We set Y t = ( Y i t ) r i =1 = Q glyph[latticetop] X t -Q glyph[latticetop] x ∗ and obtain the following system of SDEs 5 for the 'centered principal components' ( Y 1 t , . . . , Y r t ) of (hSGD)

with

glyph[negationslash]

and ( B t ) t ≥ 0 a r -dimensional Brownian motion, obtained as an orthogonal transformation B t = Q glyph[latticetop] W t of the d -dimensional Brownian motion ( W t ) t ≥ 0 . Note that PP glyph[latticetop] b is the projection of b onto the column space of A . Thus, in the overparametrized case, PP glyph[latticetop] b = b and hence β = 0 , whereas in the underparametrized case PP glyph[latticetop] b = b and hence β > 0 . Here, our main objective is to use (hSGD) to study the distributional properties, in particular the tail behavior, of SGD iterates.

2.4 Heavy-Tailed Distributions

We collect some relevant definitions related to heavy-tailed distributions and their tail index (cf. [Resnick, 2007, Foss et al., 2011]).

Definition 2.2 (See Def. 1.1 in Foss et al. [2011]) . A distribution function F ( z ) is said to be heavy-tailed (at the right end) if and only if

A real-valued random variable is said to be heavy-tailed if its distribution function is heavy-tailed.

Definition 2.3. An R d -valued random vector X is heavy-tailed if u T X is heavy-tailed for some vector u ∈ S d -1 := { u ∈ R d : | u | = 1 } .

Definition 2.4. The tail-index of an R d -valued random vector X is defined as

In particular, a finite tail-index η < ∞ implies heavy-tailedness of X , and lower values of η signify increased heaviness of tails and more extremal behavior. A tail index of η < 2 , for example, implies infinite variance and η < 1 implies non-existence of even the mean of X . Examples of heavytailed distributions are the lognormal distribution, the Studentt -distribution, the Pareto (power-law) distribution, and α -stable distributions.

Finally, we introduce a definition related to the asymptotic behavior of stochastic processes.

Definition 2.5. Let X = ( X t ) t ≥ 0 be a stochastic process. The asymptotic tail-index of X is defined as

2.5 Pearson Diffusions

To analyze its tail behavior, we perform a further rescaling of (4) by setting, for i ∈ { 1 , . . . , r } ,

This recasts the system (4) to

with | Z t | 2 = ∑ r i =1 ( Z i t ) 2 . These SDEs now have a clear structural resemblance to the system of independent one-dimensional SDEs

with the only difference given by the coupling of (7) through the | Z t | 2 -term in the diffusion coefficient. 6 The components of (8) are independent Pearson diffusions . Pearson diffusions are a flexible class of SDEs with a unified theory for statistical inference and with stationary distributions known as Pearson distributions (cf. [Forman and Sørensen, 2008]). In more detail, we obtain from [Forman and Sørensen, 2008] the following properties:

Underparametrized hSGD ( β > 0 ) : ˆ Z i t is R -valued and the stationary distribution of ˆ Z i t is called Pearson's type IV distribution (or skew Student t -distribution) and has the unnormalized density

with i

Overparametrized hSGD ( β = 0 ) : ˆ Z i t is (0 , ∞ ) -valued and the stationary distribution of ˆ Z i t is called Pearson's type V distribution (or inverse Gamma distribution) and has the unnormalized density

with ν i = φ -1 i +1 .

In both cases, the stationary distribution is heavy-tailed with tail-index given by ν i , thus providing a first connection between the SDE-approach and the emergence of heavy-tails. This connection will be quantified and made rigorous in Section 3.

2.6 Comparison to existing literature

We compare our approach to studying the distributional properties of SGD through (hSGD) with other continuous-time approximations: The Ornstein-Uhlenbeck-approximation uses (SME) under the additional assumption that the covariance matrix C ( x ) is constant. Thus, gradient noise is approximated by Gaussian noise and the Gaussian noise enters (SME) additively . The α -stable Ornstein-Uhlenbeck-approximation of [Gurbuzbalaban et al., 2021] instead presumes (based on a generalized central limit theorem) that gradient noise is non-Gaussian and follows an α -stable law. Moreover, the noise is assumed state-independent, and therefore also enters additively. In (hSGD), gradient noise is locally (i.e., conditionally on the state X t ) Gaussian, but state-dependent . The diffusion term in (7) reveals that the noise enters the SDE both multiplicatively (through the | Z t | 2 -term) and additively (through the constant χ ). Moreover, χ = 0 in the overparametrized case, such that we observe a phase transition from a mix of additive and multiplicative noise in the underparametrized case, to purely mulitiplicative noise in the overparametrized case. We note that the importance of multiplicative noise in models of SGD dynamics is discussed in great detail in [Hodgkinson and Mahoney, 2021]. We provide a summary of the comparison of these approaches in Table 1

for all p ≥ 1 .

Note that finiteness of the expectations does not need to be assumed, i.e., the inequalities also hold if one of the expectations takes the value + ∞ . Comparison results for SDEs generally require two conditions (cf. [Bergenthum and Rüschendorf, 2007]): An ordering between the drift- and diffusioncoefficients of the two SDEs, and the 'propagation-of-order'-property for one of the processes. Comparing (7) and (8), we see that the drift coefficients are identical, while the diffusion coefficients satisfy the required ordering condition 2 θ i φ i ( | z | 2 + χ ) ≥ 2 θ i φ i ( z 2 i + χ ) for any z ∈ R r . The propagation-of-order property of ˆ Z and the full proof of Theorem 3.1 are provided in Supplement A.3.

3.2 Upper and lower bounds for the asymptotic tail index

Since the process ( Z t ) t ≥ 0 is a linear transformation of the hSGD process ( X t ) t ≥ 0 , it is clear that the tail behaviour of their marginal distributions - in particular the finiteness of p -moments - is identical. Hence, an application of Thm. 3.1 provides an upper bound on the asymptotic tail index of (hSGD):

Theorem 3.2. The asymptotic tail index η of (hSGD) has the upper bound

Under conditions on the learning rate γ , a complementary lower bound can be derived from existing results on moment stability of SDEs, see Thm. 5.2 in [Li et al., 2019] and Supplement A.5 for details:

Theorem 3.3. Suppose that the learning rate γ satisfies

then the asymptotic tail index η of (hSGD) has the lower bound

3.3 Wasserstein convergence

Theorems 3.2 and 3.3 are results on the asymptotic tail index, raising the question how fast convergence to the stationary distribution takes place. The next result shows that, under a suitable assumption on the learning rate, convergence takes place exponentially fast in 2 -Wasserstein distance:

3 Theoretical results

3.1 Moment comparison

Our first result shows that the decoupled Pearson diffusions (8) are lower bounds, in convex stochastic order 7 , to the coupled hSGD process (7). In particular, a comparison result for moments holds.

Theorem 3.1. For i = 1 , · · · , d , let ( Z i t ) t glyph[greaterorequalslant] 0 be the components of the rescaled (hSGD) from (7) and ( ˆ Z i t ) t glyph[greaterorequalslant] 0 be the independent Pearson diffusion from (8) . Then for any t glyph[greaterorequalslant] 0 and convex function g : R → R it holds that

In particular this implies the ordering of p -moments

Theorem 3.4. Suppose that the learning rate γ satisfies

has a unique positive solution ρ ∗ > 0 , and the marginal distribution π t of the hSGD process X t converges in 2 -Wasserstein distance W 2 to its unique invariant distribution π . Moreover, there exists C > 0 , such that

We remark that if the conditions of Theorem 3.4 are satisfied, then the asymptotic tail-index η is necessarily greater than two, such that second moments and in particular the 2 -Wasserstein distance are well-defined and finite.

3.4 Discussion of theoretical results

We compare our results to Gurbuzbalaban et al. [2021], who analyse the distributional properties of SGD directly through the stochastic recurrence (1) under the assumption of an isotropic Gaussian data distribution. In our setting, the data distribution is arbitrary, since all results are given conditional on the data matrix A . On the other hand, we analyse SGD only through its diffusion approximation (hSGD) rather than directly. However, in contrast to [Gurbuzbalaban et al., 2021], we obtain the quantitative and explicit tail-index bounds (13) and (14), whereas Gurbuzbalaban et al. [2021] only describe the tail index through an implicit equation and derive qualitative results on its behaviour.

Parameter Dependency. Some interesting observations can be made when we consider the dependency of η on several meta-parameters of the stochastic gradient descent procedure:

Corollary 3.5. The upper and lower bounds of the tail-index are increasing in the regularization parameter δ and batch size B , and are decreasing in the learning rate γ and the first singular value λ 1 of the data matrix A .

This result agrees with Theorem 4 in [Gurbuzbalaban et al., 2021], obtained under the assumption of an isotropic data distribution a i ∼ N (0 , σ 2 I d ) , in all aspects, except the dependency on dimension d . While Gurbuzbalaban et al. [2021] report decreasing dependency on d , our tail-index bounds do not explicitly depend on dimension d . Nevertheless, the two results can be reconciled as follows: Under the assumptions in [Gurbuzbalaban et al., 2021], the data matrix A = ( a i ) is random with E ( A T A ) = σ 2 I d , and the product matrix W := A T A follows the so-called Wishart ensemble (cf. [Wishart, 1928]). Moreover, from Theorem 1.1 in [Johnstone, 2001] it follows that for large d the maximum eigenvalue of W is

where the ratio r = d n -1 < 1 and the distribution function of the random variable Ψ is the well-known Tracy-Widom distribution of order 1 (cf. [Craig and Harold, 1996]). From (15), we can calculate the average of λ 2 1 as

and λ 2 1 fluctuates around this expectation over a narrow region of width O ( d 1 3 ) . Substituting λ 2 1 by its expectation in (13) and (14) we can now see that η ∗ and η ∗ increase in both variance σ 2 and d , consistent with [Gurbuzbalaban et al., 2021].

Distributional properties. From Theorem 3.1 we see that the skew Studentt distribution provides an asymptotic lower bound in convex order for the marginal distribution of hSGD. Empirically (see Section 4) we see that skewness is negligible and furthermore, that the Studentt -distribution not only provides a lower bound, but in fact a very good fit to the parameter distribution of SGD in general, surpassing the fit of the α -stable distribution proposed in [Gurbuzbalaban et al., 2021]. For this reason, we propose to use the Studentt -distribution, rather than α -stable distribution, as a proxy for the parameter distribution in SGD.

Then the equation

4 Experiments

Based on the upper and lower bounds in Theorems 3.2 and 3.3, we present some experiments to illustrate the tail behavior of SGD and the factors influencing the tail index. The procedure of our experiments contains the following steps.

Given [data | b ] , we transform the data to be on a similar scale by the linear scaling

Let K be the iteration number of SGD. We apply (SGD) to solve (ERM). The final state x K ∈ R d is a random vector.

Repeat the second step 1000 times for different initial points and obtain 1000 different samples of x K .

For further distributional analysis we project x K via y = q glyph[latticetop] 1 x K on the dominant direction, given by the first right singular vector q 1 of A . Then we utilize the 1000 samples to obtain the empirical complementary cumulative distribution function (ccdf) of y .

4.1 Datasets

Synthetic data. We first validate our results in the same synthetic setup used in [Gurbuzbalaban et al., 2021]. All data points are drawn from isotropic Gaussian distributions, precisely, the i -th row of X ∈ R n × d contains χ i ∈ R d ∼ N (0 , I d ) . Then given x ∈ R d ∼ N (0 , 3 I d ) we draw the response vextor b ∈ R n with components b i ∼ N ( χ i x, 3) . We set the number n of the synthetic data to be 2000 through our experiments.

Real data. In our second setup we conduct our experiments on the handwritten digits dataset from the Scikit-learn python package (cf. [Pedregosa et al., 2011]) using a random feature model proposed in [Rahimi and Recht, 2007] and a three-layer neural network. The digits dataset contains n = 1797 images of handwritten digits in a 8 × 8 pixel format. The pixels are stacked into vectors of length n 0 = 8 2 = 64 resulting in a raw data matrix Y ∈ R n × n 0 and the class label b i = { 0 , 1 , · · · , 9 } is used as response vector. For the random feature model, we choose a dimension d and draw a random weight matrix W ∈ R n 0 × d having standard Gaussian entries. The feature matrix W ∈ R n × d is given by

where σ ( · ) is a rescaled ReLu activation function. The neural-network model uses 64 neurons in each hidden layer and sigmoid activation functions. The precise parameter values used for the figures are reported in Tables 4 and 5 in the supplement.

4.2 Empirical results

To verify the heavy-tailed behavior of y as well as our tail-index bounds from Theorems 3.2 and 3.3 and the distributional approximation suggested by (9), we use MLE-estimation to fit our centered data as

where t ( ν ) denotes a Studentt -distribution with parameter ν and κ is a fitted scaling factor. 8 The QQ-plots in Figures 1, 2 (a)-(c) show that the Studentt -distribution provides a very good fit to the empirical data, validating our use of Pearson diffusions to approximate SGD. In comparison, it can be seen in Figure 1, 2 (d)-(f) that the fitted α -stable distribution overestimates the heaviness of tails, in particular for the random feature model on real data. We complement these figures by KolmogorovSmirnov tests (cf. Chapter 4.4 in [Corder and Foreman, 2014]) testing for the goodness-of-fit of the Studentt -distribution and the α -stable distribution respectively; see Tables 2, 3 for detailed results. In all three settings, the hypothesis of a Studentt -distribution is accepted, while the α -stable distribution is rejected.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

3

3

4

2

2

N

1

N

1

N

2

les

0

les

0

les

Quantile

-1

antile

ueno

-1

Quantile

0

-2

-2

2

-3

3

-4

-3

-2

-1

0

1

2

3

E-

-2

-1

0

1

2

3

2

0

2

4

Quantilesoft-distribution

Quantilesoft-distribution

Quantilesoft-distribution

0.3

3

0.3

0.2

2

0.2

N

1

N

0.1

N

0.1

lantile

0

S

0.0

S

D

0.0

-1

C

0.1

0.1

-2

0.2

0.2

-3

0.3

0.3

-3

-2

-1

0

1

2

3

-0.3-0.2-0.1

0.0

0.1

0.2

0.3

0.3

-0.2

-0.1

0.0

0.1

0.2

0.3

Quantilesofα-stabledistribution

Quantilesofα-stabledistribution

Quantilesofα-stabledistribution

100

100

100

F(x)

10-1

F(x)

10-1

F(x)

10-1

t(n*)

t(n*)

60l

t(n*)

10-2

10-2

10-2

Z

Z

Z

t(n*)

t(n*)

t(n*)

10-3

10-3

10-3

100

10-1

100

10-1

logx

Moreover, in Figure 1 (g)-(i) we plot (in doubly logarithmic coordinates) the empirical ccdf of the SGD iterates z , together with the ccdf of the Studentt -distribution parametrized by our lower and upper bound η ∗ and η ∗ . It can be seen that the empirical ccfd, including its tail, is nicely sandwiched between upper and lower bound, validating Theorems 3.2 and 3.3. Additionally, we once more confirm the heavy-tailed behavior of SGD iterates as already observed in [Simsekli et al., 2019, Hodgkinson and Mahoney, 2021, Gurbuzbalaban et al., 2021].

(a)

(b)

(c)

(d)

(e)

(f)

2

2

2

N

1

N

1

N

1

0

0

0

1

-1

-1

-2

-2

2

2

1

0

1

2

2

1

0

1

2

-2

-1

0

1

2

Quantilesoft-distribution

3

2

3

2

2

1

N

1

N

N

1

htiles

0

Quantile

les

0

les

0

uer

lant

eno

1

-1

eno

-1

-2

-2

-2

-3

-3

-3

Quantiles ofα-stabledistribution

2

1

0

1

2

3

-2

Quantiles of α-stabledistribution

-1

Q

1

2

-3

Quantiles ofα-stable distribution

-2

1

0

1

2

3

5 Conclusion and Limitations

This study delves into the phenomenon of heavy tails emerging in the parameters of homogenized stochastic gradient descent applied to regularized linear regression. By establishing a connection between hSGD and Pearson diffusions, we have been able to derive both explicit upper and lower bounds for the tail index of the parameter distribution. Our results reveal that heavy tails can emerge even in the presence of locally Gaussian gradient noise and provide insights into the influence of optimization hyperparameters on the tail index. However, it is essential to recognize that our analysis relies on the approximation of SGD by hSGD and is limited to the setting of linear regression with quadratic loss. Another limitation (see (14)) is that the tail-index of hSGD is lower-bounded by one, and thus hGSD can not be used to analyse 'ultra-heavy tails' with tail-index η ≤ 1 . Future work will be devoted to extending our results to non-linear models and to providing a tighter connection between the behaviour of hSGD and the discrete-time SGD algorithm.

Acknowledgments and Disclosure of Funding

Zhe Jiao's research is supported by the National Natural Science Foundation of China (12272297). Martin Keller-Ressel acknowledges support from the Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI) in Lepizig/Dresden, Germany.

References

Anderson. A secular equation for the eigenvalues of a diagonal matrix perturbation. Linear algebra and its applications , 246:49-70, 1996.

Azagra. Global and fine approximation of convex functions. Proceedings of the London Mathematical Society , 107(4):799-824, 2013.

Beesack. Comparison theorems and integral inequalities for Volterra integral equations. Proceedings of the American Mathematical Society , 20(1):61-66, 1969.

Benjamin and U. Simsekli. Generalization bounds for heavy-tailed SDEs through the fractional Fokker-Planck equation. In ICML , 2024.

Bergenthum and L. Rüschendorf. Comparison of semimartingales and Lévy processes. The Annals of Probability , 35(1):228-254, 2007.

Bottou, E. F. Curtis, and J. Nocedal. Optimization methods for large-scale machine learning. SIAM Review , 60(2):223-311, 2018.

G.W. Corder and D.I. Foreman. Nonparametric Statistics: A Step-by-Step Approach . Wiley, 2014.

T. Craig and W. Harold. On orthogonal and symplectic matrix ensembles. Communications in Mathematical Physics , 177:727-754, 1996.

Cuchiero, M. Keller-Ressel, and J. Teichmann. Polynomial processes and their applications to mathematical finance. Finance and Stochastics , 16:711-740, 2012.

Filipovi´ c and M. Larsson. Polynomial diffusions and applications in finance. Finance and Stochastics , 20:931-972, 2016.

Forman and M. Sørensen. The Pearson diffusions: a class of statistically tractable diffusion processes. Scandinavian Journal of Statistics , 35:438-465, 2008.

Foss, D. Korshunov, S. Zachary, et al. An introduction to heavy-tailed and subexponential distributions , volume 6. Springer, 2011.

Friesen, P. Jin, and B. Rüdiger. Stochastic equation and exponential ergodicity in Wasserstein distances for affine processes. The Annals of Applied Probability , 30(5):2165-2195, 2020.

Goodfellow, Y. Bengio, and A. Courville. Deep learning . MIT Press, 2016.

Gurbuzbalaban, U. Simsekli, and L. Zhu. The heavy-tail phenomenon in SGD. In International Conference on Machine Learning , pages 3964-3975, 2021.

Hastie, R. Tibshirani, and J. Friedman. The Elements of Statistical Learning . Springer, 2009.

Hiriart-Urruty and C. Lemaréchal. Convex analysis and minimization algorithms I: Fundamentals . Springer, 1996.

Hodgkinson and M. Mahoney. Multiplicative noise and heavy tails in stochastic optimization. In International Conference on Machine Learning , pages 4262-4274, 2021.

Horn and C. Johnson. Matrix analysis . Cambridge university press, 2012.

M. Johnstone. On the distribution of the largest eigenvalue in principal components analysis. The Annals of Statistics , 29(2):295-327, 2001.

Karatzas and S. Shreve. Brownian motion and stochastic calculus . Springer, 2012.

Karatzas and S. Shreve. Brownian motion and stochastic calculus , volume 113. springer, 2014.

E. Kloeden and E. Platen. Numerical Solution of Stochastic Differential Equations . Springer, 1999.

Li, T. Cheng, and W. E. Stochastic modified equations and adaptive stochastic gradient algorithms. In International Conference on Machine Learning , pages 2101-2110. PMLR, 2017.

Li, X. Mao, and G. Yin. Explicit numerical approximations for stochastic differential equations in finite and infinite horizons: truncation methods, convergence in p -th moment and stability. IMA journal of Numerical Analysis , 39:847-892, 2019.

Mandt, M. Hoffman, and D. A. Blei. A variational analysis of stochastic gradient algorithms. In International Conference on Learning Representations , 2016.

Martin and M. Mahoney. Traditional and heavy tailed self regularization in neural network models. In International Conference on Machine Learning , pages 4284-4293, 2019.

Mori, Z. Li, K. Liu, and M. Ueda. Power-law escape rate of SGD. In International Conference on Machine Learning , pages 15959-15975, 2022.

Müller and D. Stoyan. Comparison Methods for Stochastic Models and Risks . Wiley, 2002.

Oksendal. Stochastic differential equations: an introduction with applications . Springer Science &Business Media, 2013.

Paquette, E. Paquette, B. Adlam, and J. Pennington. Homogenization of SGD in high-dimensions: Exact dynamics and generalization properties. Advances in Neural Information Processing Systems , 35:35984-35999, 2022a.

Paquette, E. Paquette, B. Adlam, and J. Pennington. Implicit regularization or implicit conditioning? exact risk trajectories of SGD in high dimensions. Advances in Neural Information Processing Systems , 35:35984-35999, 2022b.

Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, Thirion B., O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and É. Duchesnay. Scikit-learn: Machine learning in python. Journal of Machine Learning Research , 12(85):2825-2830, 2011.

Rahimi and B. Recht. Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems , 2007.

Raj, L. Zhu, M. Gurbuzbalaban, and U. Simsekli. Algorithmic stability of heavy-tailed SGD with general loss functions. In International Conference on Machine Learning , pages 28578-28597. PMLR, 2023.

I. Resnick. Heavy-tail phenomena: probabilistic and statistical modeling . Springer, 2007.

Shaked and J. G. Shanthikumar. Stochastic orders . Springer, 2007.

Simsekli, L. Sagun, and M. Gurbuzbalaban. A tail-index analysis of stochastic gradient noise in deep neural networks. In International Conference on Machine Learning , pages 5287-5837, 2019.

Simsekli, O. Sener, G. Deligiannidis, and M. A. Erdogdu. Hausdorff dimension, heavy tails, and generalization in neural networks. In Advances in Neural Information Processing Systems , pages 5138-5151, 2020.

Strassen. The existence of probability measures with given marginals. The Annals of Mathematical Statistics , 36:432-439, 1965.

Wishart. The generalized product moment distribution in samples from a normal multivariate population. Biometrika , 20A(1/2):32-52, 1928.

A Supplementary material

A.1 Derivation of covariance matrix

Consider the minibatch stochastic gradient

where B is the batchsize and the random set Ω k = { i 1 , · · · , i B } consists of B independently identically distributed random integers sampled uniformly from { 1 , 2 , · · · , n } .

Let ∇ ˜ L k ( x ) = 1 B ∑ i ∈ Ω k ∇ L i ( x ) . It can be rewritten as

where the random variable s i = l if l -multiple i 's are sampled in Ω k , with 0 glyph[lessorequalslant] l glyph[lessorequalslant] B . The probability of s i = l is given by the multinomial distribution P (s i = l ) = C l B ( 1 n ) l (1 -1 n ) B -l . Moreover, we have

We can also compute

and

Combining (16) with (17) gives

A.2 Transformation of hSGD

By multiplying both sides of hSGD by Q glyph[latticetop] we obtain

Due to and

(18) can be reformulated as

Let B t := Q glyph[latticetop] W t , which is an r -dimensional Brownian motion, due to Q glyph[latticetop] Q = I r . From (19) it follows that Y t = Q glyph[latticetop] X t -Q glyph[latticetop] x ∗ satisfies

with

Reading (20) component by component, we obtain (4).

A.3 Proof of Theorem 3.1

We write the SDEs (7) and (8) in the form

where

While the drift coefficients are identical, the diffusion coefficients satisfy the inequality σ i ( z ) ≥ ˆ σ i ( z ) for all z ∈ R r and i = 1 , . . . , r . Note that all coefficients are Lipschitz continuous and of bounded growth, such that the standard assumptions for uniqueness and existence of strong SDE solutions are satisfied. Moreover, the SDEs for ˆ Z i t are decoupled and each is a Markov diffusion with generator given by

where x denotes the scalar state variable of ˆ Z i . Let C l P ( R ) denote the subspace of C l -functions for which all derivatives up to order l have polynomial growth. Suppose that g ∈ C l P ( R ) . From Theorem 4.8.6 in [Kloeden and Platen, 1999] the backward functional

satisfies the backward Kolmogorov equation

with ∂ t G i continuous and G i ( t, · ) ∈ C l P ( R ) for each t ∈ [0 , T ] . We now provide a Lemma showing the propagation-of-order property of ˆ Z :

Lemma A.1. If g ∈ C l P ( R ) is convex, so is G i ( t, · ) for all t ∈ [0 , T ] and i = 1 , . . . , r .

Proof. For better readability we suppress the superscript and subscript i in the SDE

and consider its Euler-Maruyama approximation

with t j = j T -t K + t , j = { 0 , 1 , · · · , K } and ∆ t j = T -t K := ∆ . Using Theorem 9.7.4 in [Kloeden and Platen, 1999] we have

Let A be a transition operator given by

with W ∼ N (0 , ∆) . We will show that A satisfies the convex-ordering property

for any convex function h ( · ) . Let S 1 , S 2 be random vectors which are independent of W and satisfy E h ( S 1 ) glyph[lessorequalslant] E h ( S 2 ) . Due to Strassen's theorem in [Strassen, 1965], we can also assume that E ( S 2 | S 1 ) = S 1 . It follows from conditional Jensen's inequality that

Here, the linearity of b ( · ) implies E ( b ( S 2 ) | S 1 ) = b ( S 1 ) . Note that the function f ( x ) = √ x 2 + χ is convex. Similarly, σ ( · ) is convex. Using conditional Jensen's inequality again gives

Due to

with µ = E ( S 1 +∆ b ( S 1 )) , by Theorem 3.4.7 in [Müller and Stoyan, 2002], (25) implies that

Combined with (24) we have proved the convex-ordering property (23).

By the Markov property of the Euler-Maruyama approximation we have

Let Z be a Bernoulli random variable which takes the value z 1 ∈ R with probability p ∈ (0 , 1) and the value z 1 ∈ R with probability 1 -p . Then E ( Z ) = p z 1 +(1 -p) z 2 . Then we have

Using the convex-ordering property (23) of the operator A we obtain

due to g is convex. Take expectation on both sides of (26) gives

which means G K ( t, · ) is convex. The approximation property (22) implies the convexity of G ( t, · ) .

Next, we need a technical result that shows that each process G ( z, Z i ) is of 'class (D) '. 9

i t t ∈ [0 ,T ] Lemma A.2. For each i = 1 , . . . , r , the process G i ( t, Z i t ) t ∈ [0 ,T ] is of class (D) .

Proof. Since the solution to (8) is a polynomial process (see example 3.6 in [Cuchiero et al., 2012]), it follows from Theorem 3.1 in [Filipovi´ c and Larsson, 2016] that

where with

and P( Z i t ) = (0 , 1 , Z i t , ( Z i t ) 2 , · · · , ( Z i t ) p ) T . Then there is a constant C T that depends on T such that

Let τ n be a localizing sequence for G ( t, y t ) . Then we have which implies

Taking F 0 -condition on both sides of (27) gives

Here, the last inequality holds based on Lemma 2.17 in [Cuchiero et al., 2012]. Thus, we complete the proof of this lemma.

We are now prepared to give the proof of Theorem 3.1.

Proof. Let g be a convex function and assume for now that g ∈ C 2 P ( R ) . Define the local martingale

. Using Itô's formula in the first step and (21) in the second step, we have

By G i ( t, · ) ∈ C 2 P ( R ) and Lemma A.1 we obtain ∂ xx G i ( s, · ) glyph[greaterorequalslant] 0 for all i ∈ { 1 , . . . , d } . Thus, due to the ordering of σ 2 i and ˆ σ 2 i , the first term in the right hand side of (28) is nonnegative. Since L is

a continuous local martingale with zero initial data, it follows that G i ( t, Z t ) -G i (0 , Z 0 ) is a local submartingale.

Let τ n be a localizing sequence for G i ( t, Z t ) . For all t ∈ [0 , T ] , we have

Since G i ( t, Z t ) is a process of class (D) or locally L p -bounded, p > 1 , it follows that G i ( t ∧ τ n , Z t ∧ τ n ) -G i (0 , Z 0 ) is uniformly integrable. Combining almost-sure convergence with the uniformly integrable property, it implies that the convergence (29) also takes place in L 1 , and therefore, G i ( t, Z t ) -G i (0 , Z 0 ) is a submartingale. By taking expectations on both sides of (28) and using the fact that Z 0 = ˆ Z 0 , we obtain the comparison result

for all convex g ∈ C 2 P ( R ) .

Now let g be arbitrary convex function on R . From Theorem 3.1.4 in [Hiriart-Urruty and Lemaréchal, 1996] we can find, for each n ∈ N a convex Lipschitz function ˜ g n such that ˜ g n = g in [ -n, n ] and ˜ g n ≤ g in R \ [ -n, n ] . By [Azagra, 2013] we can find further smooth convex functions g n ∈ C ∞ Lip ( R ) such that ˜ g n -1 n ≤ g n ≤ ˜ g n on all of R . It follows that the sequence g n converges pointwise to g from below. We observe that C ∞ Lip ( R ) ⊂ C 2 P ( R ) and equation (11) now follows from (30) by monotone convergence. Finally, equation (12) follows by choosing the convex function g ( z i ) = | z i | p .

A.4 Proof of Theorem 3.2 (upper bound)

From X t = QY t + x ∗ , the triangle inequality and the unitary invariance of the Euclidean norm, it follows that | Y t | ≤ | X t | + | x ∗ | . Thus, we have

Now, let p > ν 1 . By Theorem 3.1, Fatou's Lemma, and the properties of the distribution (9) or (10)

Together with (31) this implies that also

and it follows from (5) that the tail index satisfies η ≤ p for all p > ν 1 . Finally, the parameter ν 1 in the limit distribution of ˆ Z 1 is given by ν 1 = 1 + φ -1 1 , where φ 1 can be found in (6). Thus, we obtain Theorem 3.2.

A.5 Proof of Theorem 3.3 (lower bound)

For better readability, we rewrite (hSGD) in the form

with

Our goal is to show that

for all ρ ∈ (0 , η ∗ ) , where C 1 is a positive constant and

Under condition (33) it follows directly from Theorem 5.2 in [Li et al., 2019] that the solution X t of the SDE (32)) satisfies

with C 2 a positive constant, showing Theorem 3.3.

In order to show (33), let

denote the Rayleigh-quotient of A T A . From Chapter 1 in [Horn and Johnson, 2012] we have that the range of M ( x ) is equal to the line segment [ λ 2 r , λ 2 1 ] , i.e.,

Evaluating the condition (33), we have

and

With | √ A T A | 2 = tr( A T A ) and the positive constant ρ given below, we obtain

where

Set

Note that due to the assumption γ < ¯ γ we have ϑ > 2 . We claim that

for all ρ ∈ [2 , ϑ ) . First, note that m ↦→ q ( m,ρ ) is concave for any ρ ∈ [2 , ϑ ) , such that its minimum must be attained at one of the boundary values m ∈ { λ 2 r , λ 2 1 } . Second, note that ρ ↦→ q ( m,ρ ) is strictly decreasing for any m ∈ (0 , ∞ ) , such that for (37) it is sufficient to show

Using the assumption γ < ¯ γ we obtain

For δ = 0 the right hand side vanishes and (38) is shown. Differentiation shows that the right hand side is increasing in δ , such that (38) holds for all δ ≥ 0 . Altogether, we have shown that the right hand side of (35) is strictly negative. Thus, the SDE (32) satisfies the Assumption 5.1 in [Li et al., 2019]. Based on Theorem 5.2 in Li et al. [2019], the solution X t of the SDE (32) satisfies

for all ρ ∈ [2 , ϑ ) . Therefore, the lower bound, denoted by η ∗ , for the asymptotic tail index of X t is

A.6 Wasserstein convergence

Lemma A.3. Let Z and ˜ Z be two strong solutions of (7) with possibly different initial conditions Z 0 , ˜ Z 0 ∈ R r . Suppose that

has a unique positive solution ρ ∗ > 0 and there exist constants C, C ′ independent of Z 0 , ˜ Z 0 , such that and

Proof. We set µ = ( µ 1 , . . . , µ r ) , Θ = diag ( θ 1 , . . . , θ r ) , ψ = (2 φ 1 θ 1 , . . . , 2 φ r θ r ) , and transform Z into V t := e Θ t ( Z t -µ ) . Applying Ito's formula, we see that V can be written as

Then the equation

The same representation holds for ˜ V in relation to ˜ Z . Setting d ( z, z ′ ) = √ | z | 2 + χ -√ | z ′ | 2 + χ , we estimate

for each i = 1 . . . r . Using Ito isometry and the Lipschitz property d ( z, z ′ ) ≤ | z -z ′ | , we obtain

Introducing D t = ( D 1 t , . . . , D r t ) , where D i t = E [ ∣ ∣ ∣ Z i t -˜ Z i t ∣ ∣ ∣ 2 ] and M = ψ 1 glyph[latticetop] = ( ψ i ) i,j , where 1 = (1 , . . . , 1) , we can combine these inequalities into the vector-valued inequality

Now, consider the comparison equality

Differentiation shows that

Applying the comparison result of Beesack [1969], we obtain

Hence,

, where ρ ∗ is the smallest Eigenvalue of Θ -2 M .

Now, M = ψ 1 glyph[latticetop] , i.e., Θ -2 M can be considered a rank-one perturbation of the diagonal matrix Θ . By [Anderson, 1996], the Eigenvalues ρ 1 , . . . , ρ r of Θ -2 M are solutions of the secular equation

Moreover, they interlace the diagonal values of Θ , i.e., we have ρ ∗ = ρ 1 < θ 1 < ρ 2 < · · · < ρ r < θ r . Therefore, all Eigenvalues of Θ -2 M are positive, except for ρ ∗ which may be either positive or negative. On ( -∞ , θ 1 ) the function F is strictly decreasing from 1 to -∞ , such that its root ρ ∗ satisfies ρ ∗ > 0 if and only F (0) > 0 . Rewriting this condition in terms of (6) yields (39); doing the same for the secular equation (42) yields (40). This completes the proof for the estimate of E [ | Z s -Z ′ s | ] 2 ; the proof for E [ | Z s | 2 ] is completely analogous.

We are now prepared for the proof of Theorem 3.4, which uses some key ideas from [Friesen et al., 2020]:

Proof. Let ( Z t ) t ≥ 0 be the unique strong solution of (7) and denote by p t ( z, dζ ) its Markov transition kernel. Moreover, for any Borel measure µ on R r set

Note that P t + s = P t P s = P s P t by the Markov property of Z . Denote by P 2 the set of all Borel measures µ on R r with ∫ | z | 2 µ ( dz ) < ∞ . From Lemma A.3 we see that under condition (39) P t maps P 2 into P 2 for any t ≥ 0 . Moreover, the contraction estimate in Lemma A.3 implies that

with δ z , δ z ′ the Dirac measures in z and z ′ respectively. Using the convexity of the 2 -Wasserstein distance (cf. Sec. A.2 in [Friesen et al., 2020]), it now follows that

for any µ, ν in P 2 .

Let µ ∈ P 2 . For any n, k ∈ N 0 , we have

which shows that ( P n µ ) n ∈ N 0 is a Cauchy sequence in ( P 2 , W 2 ) . In particular there exists a limit π ∈ P 2 such that lim n →∞ W 2 ( P n µ, π ) = 0 . Next, we show that π is an invariant measure for Z . Indeed, for any h > 0 and k ∈ N , we can estimate

where the right hand side tends to zero as k →∞ . Finally, we show that the invariant measure π is unique. Suppose that there is another invariant measure π ′ ∈ P 2 . Then

which tends to zero as n →∞ . Together, this shows that under the conditions of Lemma A.3, Z converges in W 2 -distance to its unique invariant distribution π , and hence completes the proof of Theorem 3.4.

A.7 Parameter Values

A.8 Experimental configuration

The computing device that we use for calculating our examples includes a single Intel Core i7-10710U CPU with 16GB memory. Our code is available at: https://github.com/zhezhejiao/hSGD .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: see the Section 1.1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: see the Sections 2.6 and 3.4.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: See the Supplement A.1-A.6 for all the proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The experiments in the paper are reproducible; code will be released.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: See the Supplement A.8.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See the Supplement A.7.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: see the Table 2.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See the Supplement A.8.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research in this paper conform with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: See the Supplement A.8.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: See the Supplement A.8.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyses the emergence of heavy-tails in homogenized SGD applied on linear models with quadratic loss. Following other works, the authors model SGD through homogenized SGD and show theoretically and empirically that such an equation converges naturally to heavy-tailed distributions, despite the fact that the noise appearing in the SDE is Gaussian. The proof technique is based on known properties of Student-t-distributions, which appear in the study of similar equations, and a comparison result for moments. This allows the authors to derive analytic upper and lower bounds of the tail index that they were able to successfully confront with numerical experiments. These numerical experiments suggest that Student-t-distributions may better described the distribution of SGD iterates than previously used stable distributions.\n\nSTRENGTHS:\n- Showing that heavy tails can arise even in the presence of Gaussian-type noise is an original contribution, which may illuminate some aspects of the presence of heavy tails in machine learning.\n\n- As opposed to previous works, analytical bounds are derived on the tail-index. These bounds are directly computable from the optimization hyperparameters.\n\n - The authors confirmed previous experimental findings regarding the emergence of heavy tails in SGD. Moreover, their results suggest that Student-t-distributions could be used to model SGD noise, as opposed to previously used $\\alpha$-stable distributions.\n\nWEAKNESSES:\n- The paper has no conclusion, it could improve the paper to add a few lines of conclusion to discuss limitations and future works.\n\n - As it is explained in Section 2.1, the analysis is entirely limited to the case of linear regression with quadratic loss. However, this is never mentioned neither in the title, abstract, or introduction (including the claimed contributions) of the paper. The claim that the paper analyses ``heavy-tailed phenomena in SGD'' may therefore be too strong. It should be mentioned that all results were obtained for linear regression.\n\n - A weakness related to the previous point is that there should be more discussion on whether this analysis can extend to more complex settings, ie, outside of the linear regression settings, instead of just saying that all models are approximately quadratic near a local minimum. It is understandable that it is hard to extend the theoretical analysis to this setting, but  it would largely enhance the paper to perform similar experiments as in Figure 1, but with a simple non-linear model, like a 2-layers neural network.\n\nQUESTIONS:\n- In the equation just after line 93, what is $D$?\n\n - Can you elaborate on how big the right-hand side of Equation (12) can be in practice? In particular, the ratio $n/\\gamma$ (number of data points over learning rate) seems to be huge. What are the conditions for the tail index to be small (ie, the distribution has very heavy tails)? In particular, it would be beneficial to present the analytical values found in Table 3 in the main part of the paper (they are only presented in the appendix).\n\n - Theorem 3.4: in order for the 2-Wasserstein distance to make sense, isn't the fact that the tail index is greater than 2 necessary? Does that follow from the assumptions of this theorem?\n\n - In Figure 1, you compare the Student-t-distribution and $\\alpha$-stable distributions to the experimental data. How was the tail-index $\\alpha$ (used for the $\\alpha$-stable distributions) estimated?\n\n**Other remarks**\n\n - In the introduction, you mention that some authors argue for a negative correlation between the tail-index and the generalization error. However, a lot of recent works have argued that different behaviors can happen in many settings, see in particular https://arxiv.org/abs/2006.09313, https://arxiv.org/abs/2402.07723 and https://arxiv.org/abs/2301.11885.\n\nLIMITATIONS:\nSeveral limitations were discussed in the paper. However, it should be clear from the abstract and introduction that the analysis is restricted to linear models. Moreover, a discussion on the potential extensions of the theoretical (and empirical) result to vanilla SGD and non-linear models would be very interesting.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper builds up on the theory of heavy tails in Stochastic Gradient Descent (SGD), clearing some aspects of the distribution of the parameters learned during training. Leveraging a combination of the works on homogeneized SGD and diffusion, the author(s) are able to shed light on the role of noise and the distributional properties of parameters. \nMore in depth, they relate homogeneized SGD to a \"coupled\" Pearson diffusion, that can be put in comparison (convex stochastic order) with the standard one. Thanks to this, an explicit description of heavy-tailedness via the asymptotic tail index is obtained. One side of the bound is automatic, the other requires an assumption on the learning rate. After establishing the asymptotic behavior, the speed of convergence to it is explored; subject to an assumption on the learning rate, the author(s) prove that it can happen exponentially fast in Wasserstein distance. To validate their arguments, statistical tests for the parameter distribution are proposed in one case.\n\nSTRENGTHS:\nThe overall exposition is clear and to the point. \n- On a related note, the structure is pedagogical: my impression is that any reader at any level could take something from spending time on the paper. This is important. \n- The observation that gaussian noise is sufficient to have a heavy-tailed distribution is very valuable. \n- Asymptotic results are paired with practical convergence at the cost of assumptions that are not absurd. \n- The work connects recent works on the distribution of SGD parameters and well-developed theory of diffusions. \n- The experiment proposed to validate one of the results is a theoretically grounded suggestion for modelling parameter distribution, and not just a heavy-tailed educated guess. \n- The coupled-Pearson diffusion link sheds a new light on the interpretation of the dynamics. \n- The idea of using convex stochastic order is simple and effective to obtain just what is needed. Such point is probably the part I appreciated the most. \n- In general, a nice and useful read.\n\nWEAKNESSES:\n(more of a subtlety) The convex stochastic order bound works for $p\\geq 1$ only, Theorem 3.2 does not explicitly state this. One might ask: what if we have a distribution with asymptotic index less than one? Then Theorem 3.1 does not work, but the truth is that for both cases of the Pearson diffusion we have that the tail index is $\\nu_i\\geq 1$ . Maybe this should be made more explicit? Can you give more clarifications? \n- The diffusion approximation is backed on an approximation of the covariance done to match the moments. Crucially, it makes the problem tractable. The works about hSGD, as the author(s) mention themselves in lines 91-97 are subject to \"certain assumptions\" and \"further empirical evidence\". This justifies their paper, to the extent that they trust the same setting, as it does not qualify for a full mathematical proof. For the same reason, the present submission misses a set of assumptions, while proving theorems. In general, I agree with this exploratory perspective, but maybe it needs to be mentioned more explicitly and not just briefly stated (related to the note on Limitations just below).\n- A proper, explicit discussion of limitations is missing. Section 2.6 is a comparison with literature, section 3.4 is a reconciliation with a specific work. As a reader -- and this is a personal opinion -- I would be very happy to hear why and when your work might/does not work.  \n- The experiments show only one of the two scenarios (inverse Gamma type distribution is not shown). \n- Figure 1, subfigures (g)-(h)-(i) should show upper and lower bounds on the ccdf but have some inconsistency. I understand that this might be due to the experimental setting, but Figure (g) clearly has the lower bound (blue) above the lower upper bound (green), with the true distribution even below. Maybe you can clarify this aspect?\n- Theorem 3.4 is not backed by experiments. \n- Overall, I would be glad to see more validation of the theory. \n\n\n#### Typos\nThese are not weaknesses, but I am adding them here since there is no dedicated box. Please note that some are very pedantic, but I want to be constructive and give you all that I saw. \n- The notation for $L, L_{\\Omega_k}, L^{\\mathrm{reg}}, L^{\\mathrm{reg}}_{\\Omega_k}$ is possibly confusing: if I look at equations (SGD), (1) and (2) it looks like there is something off, but then checking the original formulation of the works cited it is possible to understand why this is the case. Maybe a clarification sentence would be helpful to the reader. \n- Related to the point above, If we also take into account appendix A.1, there we also need to take care of $L, f$. Maybe too many redundant symbols do not help an inexperienced reader. \n- (line 74-75) The standard notation for vectors is to take them as columns, the fact that $C(x) = \\mathbb{E}[\\epsilon(x)^\\top\\epsilon(x)]$ on the top of my head makes me think it is a scalar. At the same time, I understand that this is a matter of canonical choice. \n- (line 100) The matrix $Q$ is stated to be \"$r$-by-$d$\" but according to $A = P\\Sigma Q^\\top$ with $\\Sigma$ in $r$-by-$r$ it should be the opposite. \n- (line 113) is the power in the covariance $\\Sigma$ appearing in $\\alpha$ right? I have not thouroughly checked this. \n- (line 121, Def. 2.2) $\\limsup_{t\\to\\infty}$ is taken, but there is no $t$, also $F(x)$ appears in the equation. I believe you wanted to write the limit supremum of $z\\to\\infty$, $1 - F(z)$ at the  numerator, and $e^{-sz}$ at the denominator? \n- (line 125, Def. 2.4) you take the modulus of a vector in the definition of tail index. Later in equation (6) you take the norm of $Z_t$, where $Z_t$ is a vector. In one way or the other I would like to point out the inconsistency of notation for norms of vectors. \n- (Figure 1, subfigs. g, h, i and caption) in the legend, you have $t(\\eta_i)$, in the caption, you use the notation $\\eta_*, \\eta^*$, the two are intuitively bijected, but maybe you can fix the legend to be consistent with the paper?  \n- secondary, but related to the point above: the legend in the subfigures is very small and redundantly repeated. If you have space left from the submission page limit, and time in these busy times, could you please build a common legend in a separate subfigure? I am guessing here, but it looks like this is Matplotlib, and there are quick ways to extract a legend as a png from a figure, and add it to the text. This is more of an aesthetic aspect, please take it as a random idea rather than a request. \n- (lines 419-420) The sterile comment is that the \"class (D)\" TeX typesetting is wrong, there is a specific command for quotations. The constructive one is that you may want to remove the hat from the $\\hat{Z}_t^i$ in line 419 \n- (lines 446-447) the function $\\tilde{g}_n$ is bounded by $g$ in the same way in the two intervals. \n- (between line 454-455) in the last inequality, you are missing an $\\mathbb{E}$.\n\nQUESTIONS:\nWhat ensures that the solutions are strong? \n- (more of a curiosity) Do you have any comments on a possible regime/example in which the lower bounds on the learning rate $\\gamma$ obtained in Theorems 3.3-3.4 give a nice interpretation? \n- In the experiments, you analyze the first eigen-component of $\\mathbf{Q}$ via the projected quantity $\\mathbf{q}_1^\\top \\mathbf{x}_K$, what happens to the analogues for $\\mathbf{q}_2, \\ldots$? Do you observe that the theory works less? Maybe it would be interesting to see other experiments. \n- Can you somehow validate Theorem 3.4 with an experiment? The parameter $\\rho_{\\star}$ is the solution of an equation, you have an explicit bound on the learning rate that if verified would make you expect to have a certain exponential convergence, do you see it in a simulation? \n\nThe paper is technically solid, and of interest to the community. Subject to fixing the typos and potentially extending the experimental section, I believe it is very good. Therefore, count my score as \"waiting for engagement\".  Dear author(s), please do. I will be glad to raise my scores.\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides another perspective of the emergence of heavy tails in SGD to the recent literature. Unlike the previous literature, the paper assumes that the SGD can be adequately approximated by homogenized SGD, which is a Brownian-motion driven SDE with a given state-dependent diffusion term. \n\nThe paper restricts the study to the quadratic loss. \n\nThe tail-index bounds are fully explicit. \n\nThe paper introduces a comparison method based on convex stochastic order for homogenized SGD. This allows linking SGD to Pearson diffusions in the literature and then obtain bounds for the tail-index. The results suggest that one can use skew student-$t$-distributions as proxy for parameter distributions in neural networks under SGD, in contrast to the $\\alpha$-stable distributions that are commonly used in the literature.\n\nSTRENGTHS:\n(1) The idea of using homogenized SGD as a proxy and then obtain upper and lower bounds for the asymptotic tail-index is new.\n\n(2) The upper and lower bounds are explicit.\n\n(3) It provides a new perspective to a recently popular topic in the literature that is the emergence of heavy tails in SGD.\n\n(4) The analysis is rigorous, and the paper comes up with some comparison method and compares the homogenized SGD with Pearson diffusion.\n\n(5) Wasserstein convergence is also obtained.\n\nWEAKNESSES:\nThere are a few weaknesses of the paper.\n\n(1) Unlike Gurbuzbalaban et al. (2021) in the literature, the paper does not tackle the tail-index of SGD directly, but only the tail-index of homogenized SGD, which is an approximation of the SGD.\n\n(2) Even though an approximation of the SGD is used, and only the quadratic loss is considered, the results for the tail-index only concern upper and lower bounds. Despite being semi-explicit in Gurbuzbalaban et al. (2021), the tail-index in Gurbuzbalaban et al. (2021) is exact. \n\n(3) Since only the upper and lower bounds are obtained for the tail-index, it is natural to ask whether the analysis can be extended beyond the quadratic loss. This is a very reasonable question because upper and lower bounds should be used when it is impossible to obtain the precise tail-index. One expects that it is impossible to obtain the precise tail-index for the non-quadratic loss. But since comparison method is used in this paper, it is natural to ask whether it is possible to compare the non-quadratic loss with quadratic loss to obtain upper and lower bounds for the tail-index beyond the quadratic loss. In my view, that will make the paper much stronger.\n\nQUESTIONS:\n(1) Your definition in Definition 2.2. for heavy-tailedness  basically says a random variable is heavy-tailed if its tail is heavier than any exponential distribution. But your definition for tail-index in Definition 2.4. is about polynomial decay. I think there is some disconnect here. For example, consider $1-F(x)=e^{-\\sqrt{x}}$ for any $x\\geq 0$. Then, according to your definition, the tail-index is infinity even though it is heavy-tailed. That makes me wonder whether it is better to change your definition of heavy-tailedness. For example, can you say it is heavy-tailed if the tail-index is finite?\n\n(2) The analysis is for quadratic loss. Since your main results are the upper and lower bounds for the asymptotic tail-index,  would it be possible to extend your analysis beyond the quadratic loss?\n\n(3) It seems you did not provide any discussions on the theoretical result Theorem 3.4. For example, it might be worth mentioning how the convergence rate $\\rho_{\\ast}$ depends on the model parameters.\n\n(4) You description that Gurbuzbalaban et al. [2021] only describe a phase transition of the asymptotic tail-index $\\eta$ from $\\eta<2$ to $\\eta>2$, without giving quantitative estimates of $\\eta$ is not that accurate. There, they obtained a semi-explicit formula for the tail-index $\\eta$, and when the input data is Gaussian, the tail-index $\\eta$ becomes even more explicit, without the necessity to rely on upper and lower bounds.\n\n(5) In your equations (SME) and (hSGD), the $d$-dimensional Brownian motion is denoted as $W_{t}$. But later, in equation (3), $B_{t}$ is used to denote the $d$-dimensional Brownian motion. If they are the same, it is better to use the same notation. If they are different,\nyou should make the relation between $W_{t}$ and $B_{t}$ more transparent.\n\nLIMITATIONS:\nI did not see many discussions on the limitations, which should be mentioned in the summary. In fact, there is not even a conclusion section of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript studies homogenized SGD (hSGD) to characterize the tail behavior of SGD iterates for solving the ridge regression problem. By comparing the homogenized diffusion with a known diffusion process called Pearson diffusion, the authors provide a lower bound on the tail index of the iterates of homogenized SGD. They analyze how hyperparameters contribute to this lower bound, offering a qualitative analysis of hyperparameter choices and their impact on the distributional limit of SGD iterates.\n\nOverall, the paper makes a solid contribution and offers a more in-depth analysis of the heavy-tail phenomena in SGD compared to existing work. However, there are a couple of important points that need clarification:\n\n1. Is it the case that the marginals of hSGD and SGD have the same distribution? Theorem 1.3 in [1] suggests this holds for certain quadratic statistics, explained through the concentration of measure. However, if this comparison relies on the concentration of measure, the overall distributions of the two random variables can differ significantly despite overlapping in certain statistics. Therefore, I suggest the authors clarify this aspect.\n\n2. The comparison between the diffusion in (6) and (7) may be somewhat loose, especially given that the ratio between their interaction terms can be high. While the authors use experiments to suggest otherwise, the theoretical analysis does not clearly establish this. Could the authors extend their discussion on how tight or loose this comparison is in high dimensions?\n\n[1] Paquette, C., Paquette, E., Adlam, B., & Pennington, J. (2022). Homogenization of SGD in high-dimensions: Exact dynamics and generalization properties.\n\nSTRENGTHS:\nSee the Summary part\n\nWEAKNESSES:\nSee the Summary part\n\nQUESTIONS:\nSee the Summary part\n\nLIMITATIONS:\nI am not convinced that the theoretical results presented in this paper are directly applicable to analyzing the iterates of SGD. Therefore, I am currently leaning towards suggesting a Borderline Reject. I look forward to receiving further clarification, which may lead to a reconsideration of my evaluation.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}