From Spikes to Heavy Tails: Unveiling the Spectral Evolution of Neural Networks
Authors: Vignesh Kothapalli
Tianyu Pang
Shenyang Deng
Zongmin Liu
Yaoqing Yang
Venue: Accepted by TMLR ()

Abstract: Training strategies for modern deep neural networks (NNs) tend to induce a heavy-tailed (HT) empirical spectral density (ESD) in the layer weights. While previous efforts have shown that the HT phenomenon correlates with good generalization in large NNs, a theoretical explanation of its occurrence is still lacking. Especially, understanding the conditions which lead to this phenomenon can shed light on the interplay between generalization and weight spectra. Our work aims to bridge this gap by presenting a simple, rich setting to model the emergence of HT ESD. In particular, we present a theory-informed setup for ‘crafting’ heavy tails in the ESD of two-layer NNs and present a systematic analysis of the HT ESD emergence without any gradient noise. This is the first work to analyze a noise-free setting, and we also incorporate optimizer (GD/Adam) dependent (large) learning rates into the HT ESD analysis. Our results highlight the role of learning rates on the Bulk+Spike and HT shape of the ESDs in the early phase of training, which can facilitate generalization in the two-layer NN. These observations shed light on the behavior of large-scale NNs, albeit in a much simpler setting.

Published in Transactions on Machine Learning Research (08/2025)
From Spikes to Heavy Tails: Unveiling the Spectral Evolution of Neural Networks
Vignesh Kothapalli
1k2115@mmu.edv
Courant Institute of Mathematical Sciences
New York University
Tianyu Pang
Kangupanng@220mmail.ccm
Department Computer Science
Dartmouth College
Shenyang Deng
shenyang edu
Department of Computer Science Dartmouth College
Zongmin Liu
zongminl@sstanfrrd.dd edu
Department of Neurology and Neurological Sciences Stanford University
Yaoqing Yang
Department of Computer Science Dartmouth College
Reviewed OpenReview:
net/ formff id= BJHB
Abstract
Training strategies for modern deep neural networks (NNs) tend to induce heavy-tailed (HT) empirical spectral density (ESD) in the layer weights. While previous efforts have shown that the HT phenomenon correlates with good generalization in large NNs, a thoo- retical explanation
occurrence is still lacking. Especially, understanding the conditions which lead to this phenomenon can shed light on the interplay between generalization and weight spectra. Our work aims to bridge this gap by presenting a simple, rich setting to present theory-informed setup for
model the emergence of HT ESD. In particular, 'crafting' heavy tails in the ESD of two-layer NNs and present systematic analysis of the HT ESD emergence without any gradient noise. This is the first work to analyze noise-frec setting, and we also incorporate optimizer (GD/Ad (GD/Adam) dependent (large) learning rates into the HT ESD analysis. Our results highlight. the role learning rales on the Sulke and HT shape of the ESDs in the early phase training, which can facilitate generalization in the two-layer NN. These observations shed light on the behavior of large- scale NNs, albeit in much simpler setting. The code and documentation is available at: https::/githhn thub. com
1 Introduction
Training NNs on real-world data, which often has low intrinsic dimensionality WWightt & Ma 2022: Hastic et al. 2005), can commonly lead to spiked covariance structures in the ESD of weight matrices |Ba el al. 2023 Lee et al. 2024). From a classical statistics viewpoint, spiked models capture low-rank signal directions emerging from noisy high-dimensional data (Anderson al. 22100) Arous & Guionnot, 2008 Ding & Yang. 2021). offering a natural framework study spectral evolution during training. Recent work by Martin & Mahoney (202aa] showed that the ESD of weight matrices evolves through a sequence of phases to exhibit

---

Published in Transactions on Machine Learning Research (08/2025)
IM5M Mea53
Inful MertD
I-t Madlt
0.0006
0.0006
Seatt
logns (ESO)
0.0004
0.0004
0.0002
10-1
0.0002
10-1
0.0000
0.0000
0
5000
10000
15000
10000
20000
30000
A
logu
(a. GD 17 = 2000
(b) GD 17 = 2000
(c) FB-Adam 77 7= 0.5
(d) FB-Adam 17 = 0.5
Figure 1: Emergence of HT spectra after 10 GD GD/FB-Adss Fdam steps. Both linear and log scales are shown. For the purpose of illustrating clear HT shapes, we did not choose a highly HT spectrum. Highly HT spectra can also be generated without gradient noise and are discussed in Appendix A
heavy-tailed distribution- without explicit regularization. While the Bulk+ shaped covariance
matrices have been widely analyzed (Baik et al. 2005 Anderson et al. 2010 Ba et al. 2023), the emergence of heavy-tailed ESDs has drawn increasing attention due to their strong correlation with generalization performance in modern deep learning tasks across vision and language domains. The shape of HT ESDs has
also been successfully leveraged to assess the quality of pre-trained NNs (Martin & Mahoney
2020 Martin
et al. 2021 Martin & Mahoney 202Tb) (including Large Language Models (Yang et al. 2023 and design
layer-wisc learning rate schedulers (Zhou et al. 2023 Liu ct al. 2024
Towards a practical systematic analysis of HT ESDs.
Owing to the wide range of applications
described above, many efforts (Simsekli et al.] 2019 2020a b Gurbuzbalaban et al. 2021: Hodgkinson & Mahoney 2021 Hodgkinson et al., 2022 Dandi et al.] 2024a] have tried to understand the underlying mechanisms of the HT phenomenon. In particular, they have focused on the limiting distributions of the weight values by modeling the stochastic gradient noise as stochastic processes. The underlying idea is that when the weight values lend to an HT distribution, then the resulting ESD also exhibits a similar transition towards beavy-tailedness (see Arous & Guionnet (2008). Although fundamentally connected to the HT ESDs, these insights apply to the limiting/ momiting//coveegennn coovergeeccee phases of training and do not directly explain the HT ESD emergence after finite training steps as observed by the empirical studies of Martin & Mahoney (2021a); Martin et al. (202T). Furthermore, it is currently unclear if such emergence is limited to large-scale NNs trained until convergence and if it is even possible simulate such behavior in small-scale NNs during the early phases of training.
A Tangible Setup. In this paper, we present an alternative setup that is amenable to discrete-step analysis and is rich enough to study the evolution of ESDs in two-layer NNs without any gradient noise. Our setup does not rely on continuous time approximations to SGD or limiting distribution analysis but instead employs a Stacher-Student setting (Arous et al.) 2021 Bietti et al. 2022 Ba et al., 2022: Dandi et al.) 2023 Ba et al. 2023 Mousavi-Hossseini et al.] 2023b) to study the effects of finile optimizer steps on the weight matrix ESD (Figurel 4). By training a two- layer
NN (Student) to learn single-index model (Teacher) using vanilla Gradient Descent (GD) and Full-batch Adam (FB-Adam) optimizers, we show that:
"The ESD of the hidden layer weight matris exhibits heavy luils after malliple sleps of CD GD/FB-Ade FB/FBBAAdam with (sufficiently) large learning rotes".
Intuition. Before diving into the details, we present the reader with a mental model of the ESD evolution in this setup. At initialization, the hidden layer weight matrix ESD of the student NN can be characterized by the Marchencko-Pasee to result in an outlier singular value (i.e. a 'spike") in the ESD and results in a 'Bulk+Spike' shape. The singular vector corresponding to this spike tends to align with the target direction of the teacher model and results improved generalization (Ba et al.) [2022 Dandi et al. (002).. By continuing to train the student NN with such large n, our work shows that the interactions between the spike and the bulk gradually lead to Bulk-Deccyy and finally lead to HT ESD.
distribution (i.c. random-like). The first step with a larye n has been shown
2

---

Published in Transactions on Machine Learning Research (08/2025)
Our theory is primarily aimed at identifying the large 17 for Adam, which can result in a spike after the one- step update. Especially, we formulate the scale of (depending on hidden layer width and data dimension) which can lead to Bulk+SSpike ESD after the one-step update. Such a result has been formulated and studicd for one-step GD by recent works (Ba et al. 2022 Dandi et al. 2023) but none of the previous works have extended the analysis to FB-Adam (which is closer to the practical scenarios where Adam widely used).
Toward understanding the shape transitions of ESD from the Bulk+Spiike to an HT, we vary 1) from small to large values and observe that the distance between the spike and the bulk (i.e spectral gap) determines the HT emergence during the early phases of training. For instance, 7) 1 leads to HT ESDs just after t= 10 steps, whereas f = 0.1 requires very long training up to t = 10000 steps (sec also Table 3 Table 4 Thus highlighting the necessity to study critical learning rates for one-step optimizer updates. We also analyze the correlations between the heaviness of tails (as measured by power-law fits) in the ESD and the generalization of the student NN after multiple steps. We showcase empirical results on the existence of a range of 77 (depending on the optimizer) that can result in an HT ESD as well as improve the gencralization of the student NN. In particular, "f in such suitable range results in HT ESDs whose power-law fit (as measured by the hill-climbing approach (Yang et al.) (023)) lie in the range of (2, 2.5) and lead to strong generalization. This result supports the observations of Martin & Mahoney (2021a): Martin et al. (0221 for
well-trained deep NNs where the truncated power-law fit of the ESDs lies within range of (2,4).
To summarize, our main contributions are as follows:
We present gradient-noise setting to study the emergence of HT ESD in the hidden layer weight matrix of two-layer NNs during the initial phases of training. To the best of our knowledge, this is the first work to study the early evolution phases of the ESD across discrete training steps with large learning rates. In this setup, we also present doeper connections between feature learning after the first step (via the spike in the weight matrix ESD) and the HT emergence after finite steps.
We theoretically establish the scale of the learning rate which one step of FB-Adam results in spike in the hidden layer weight matrix ESD. This result can be of independent interest to the community since prior results have been established only for GD.
We empirically analyze the evolution of the hidden layer weight matrix ESD into an HT distribution during training. Interestingly, we show that for a certain range of optimizer-dependent learning rates, the two-layer NN can exhibit HT ESDs and generalize well.
Overall, our paper makes multiple novel theoretical and empirical contribut ions centered around the emer- gence of HT ESDs and their correlations with the good generalization of NNs.
2 Related Work
The heavy-tailed phenomenon. Heavy tails in machine learning have been observed and studied in various forms. The most prominent empirical results are from series of works by Martin & Mahoney| (2020 2021abb bi; Martin et al. (2021), which propose heavy-tailed self regularization (HT-SR) theory of deep NNs. In particular, Martin & Mahoney (2021a) proposed a '5 + phase model corresponding to the ESD evolution, and aims to model the HT-SR effect during training. The phases are as follows: (1) Random- like, (2) Bleeding-out, (3) Splkes, (4) Bulk-Decay, (5) Hcavy-Tailed (HT), and the extreme case of (6) Rank-Collapse. Their observations from extensive empirical analysis showcased a correlation between the power law fits of weight matrix ESDs and generalization. Although it is still unclear if HT ESDs are necessary for generalization, Martin et al. (2021) Yang et al. (2023) have shown that the shape metrics of the ESD can be effectively leveraged to identify well-trained NNs (and can be extended to largo-scale models such as LLMs). From theoretical perspective, previous eflorts have primarily locused on stochastic optimization settings and proposed generalization bounds based on the tail indices of the stochastic noise (Simsckli et al. 2019 2020b a Gurbuzbalaban et al. 22021 Hodgkinson & Mahoney 2021 Hodgkinson et al.,
2022 Raj al. 2023 Nguyen et al. 2019 Barsbey et al. 2021). In particular, the earlier work by [Simsekli
et al. 2020aa employed continuous-time time approximation of SGD via Feller-processes and studied the role of the
3

---

Published in Transactions on Machine Learning Research (08/2025)
Hausdorff Dimension generalization. More recently, Hodgkinson et al. (002)) extended this analysis to discrete-time settings. In concurrent works, Gurbuzbalaban et al. (2021) studied the heavy-tailed stationary distributions of discrete Markov processes as an approximation to SGD iterates in the infinite data regime, while Hodgkinson & Mahoney (2021) analyzed the role of multiplicative noise in such settings. While these efforts have emphasized the role of the learning rate/batch size ratio in determining the tail index of the iterates, a formal study on the role of these hyper-parameters still lacking. More importantly, none of these efforts have focused on the evolution of the ESD itself. Recent work by Waan et al. (2023) empirically studied the emergence of HT-ESDs when training shallow neural networks to learn multi-index models with Adam. They showcased that Adam can facilitate the emergence of HT-ESDs, but do not theoretically characterize the learning rates required to observe such behavior. Furthermore, they do not present a systematic study of the correlation between learning rales and tail indices for HT ESD. Our work aims to bridge these gaps and analyzes the fine-grained ESD evolution from a Marchenko-Pastuue (MP) fit (i.e. random initialization) - "Bulk+ -Bulk-Spike" - "Bulk-Decay" - HT distributions and the correlations with generalization.
Feature learning and large learning rates. Learning single-indez models using two-layer NNs under the Peacher-Student sctup has provided rich insights into the sample complexity (Damian et al. 2023
Mousavi- et al.
2023a
Zweig et al.
2023
Damian el al.
2024
2022 Abbe el al. 2023) and Training
dynamics (Bietti et al.) 2022 Wang et al.
2023
Cui et al. 2024
Moniri et al. 2023 of NNs. In particular,
the study of feature learning in such two- layer NNs focuses on the factors (such as optimizers (Abbe et al.
2023).
loss landscapes (Damian et al.
2023
representations (Nichani et al.
2023) and learning rates (Dandi
et al.
2023)) that facilitate sample efficient learning beyond the kernel regime [Louart et al. 2018 Gerace
eet al.
2020 Mei & Montanari 2022: Hu & Lu 2022 Goldt et al., 2022 Liu et al. [202Ip. Recently, [a et al.
(2022) p analyzed the first step of GD update in the high dimensional setting and formalized the scale of ry
required to go beyond the random feature regime. In particular, such large 7j is necessary for two-layer
NN to learn the hidden direction of a single-index model after one step of GD (see also (Dandi et al. 2023)
This first large update was shown to result in an outlier singular value (i.e. a 'spike') in the ESD of the hidden layer weight matrix. Our work presents the first result. for such large in the case of FB-Adam and can be of wider interest in the feature learning context. Such a characterization is important since Dandi ctt al. (2025) have shown in the asymptotic limit of input dimension d + 00 (and the proportional scaling regime) that the spikes in the weight matrix ESD tend to affect the ESD shape of feature mairix after a single step with large 77. Beyond the one-step analysis, Dandi et al. (024b analyzed the role of two-pass GD in learning single-index models with large information exponents (Arous et al. 2021). However, the effects of multiple passes over the data on the ESD and generalization are yet to be fully understood. Our multi-step analysis empirically highlights the importance of such feature learning after the first step for the emergence of HT ESD. Thus, showcasing the underexplored connections between these two areas of research.
3 Preliminaries and Setup
Notation. For n E N, we denote [m] = (1, ..... We use O(-) to denote the standard big-O notation and the subscript Oa(-) to denote the asymptotic limit of d + xc Formally, for two sequences of real numbers za and yd, va = Oa(ya) represents lima- |ara| I Cilyal for some constant C1. Similarly, Ia = Oa.(Ua)) denotes that the asymptotic inequality almost surely holds under a probability measure P. The definitions can be extended to the standard 2(-), or notations analogously (Graham & Knuth 1989. For two sequences
of real numbers zd and yd. Id Ye represents |ya < |raal I Cilyal, for constants C1, C2 > 0 (Wang)
ct al. 2021 Moniri et al. 2023). For a real matrix B = (Brj) (xm))xm E Raxm B represents an element- p-power transformation such that B = ) @ is the matrix Hadamard product, sign(.) denotes the element wise sign function. ||-l|2 denotes the lg norm for vectors and the operator norm for matrices. I1-I1F denotes the Frobenius norm. Oaxd: 1xd E R represent the all-zero and all-ones matrices.
Dataset. We sample n data points (-1,.... ,,) from the isotropic Gaussian x; ~ NN Ia), Vi E [n] as our input data. For a given 2; E Rd, we use a single-index teacher model F* R2d + R to generate the corresponding scalar label y E IR as follows:
y: =
===F'(m FF(m)))
E = (8 mx) + Ex.
(1)
4

---

Published in Transactions on Machine Learning Research (08/2025)
Here E (the d' 1-ddimnnsioonal sphere in R4) is the target direction, d2 : - IR is the target non-lineor link function, and & ~ (0, n²) is the independent additive label noise. We represent X E R y E IR" the imput matrix and the label vector, respectively.
Learning. We consider a two-layer fully-connected NN with activation a R + R as our student model f() Rd R. For an input o; IR,, its prediction is formulated as:
1 TJo (tawa)) f(æ,) = Vh
(2)
Here W E Rixd a E IR are the first and second layer weights, respectively, with entries sampled i.id follows (Woleg NO 1), [a], NNO0,), [b],j [d].
3.1 Training Procedure
We employ the following "wwo-stage training procedure (Ba et al., 2022 Moniri et al., 2023; Cui et al., 2024; Dandi et al. 2023; Wang et al.] 2023) the student network. In the first stage, fix the last layer weights E I and apply optimizer update(s) (GD/FB-Adam) only for the first layer W. In the second stage, we perform ridge regression on the last layer using a hold-out dataset of the same size to calculate the ideal
value of a (Ba et al. 2022).
Optimizer updates for the first layer. In this phase, we fix the last layer weights a to its value at initialization and perform GD/FB-Adam update(s) on W minimize the mean-squared error Rif. X, y) E (3:- ( f(x))) The update to W using GD is given by:
W+1 = W. - Gr
(3)
where W, denotes the weights W at step t and G. V w. R(f, represents the full-batch gradient. Next, to formulate the updates using FB-Adam, let:
M41 M, (1 Gr
V 32 V + (1 - G3.
(4)
Here Mj V. E Rdxd represent the first and second order moving averages of the gradient respectively, with base values Mo- = Opxd, Vo = =0xx (Kingma & Ba 2014). (81, 32) E IR are the decay factors. Considering G. = (Ver2 + eelxxd o Me+1, we formulate the FB-Adam update
W+1 -
G
(5)
For the remainder of this paper, we use the overloaded term 'oplimizer updale' to represent either FB-Adam or GD update. Specific choices of the optimizer will be mentioned explicitly.
reggessennnsioon on final layer. Similar the setup of Ba et al. (2022): ; Moniri al. (2023): Wang
et al. (202); Ba et al. (002)), we consider hold-out training dataset X E Raxa y RD sampled in the fashion X, y learn the last layer weights. Formally, after t optimizer updates to the first layer
to obtain We, we calculate the post-activation features Z, = Vh (ta w.x') and solve the following ridgo-regression problem:
à arg min 1 g Z, a Ti
||al
(6)
2
Here À > is the regularization constant. The solution à is now used the last-layer weight vector for our student network f(-) and consider the resulting regression loss setup allows us to measure the impact of updates to W after L. sleps on the hold-out dataset's 's regression
training loss. Formally, this
loss. Finally, for a test sample x E Rd, the student network prediction is given as: 9 = J aTo va Wax).
These predictions on the test data are used for computing the test loss using the mean squared error.
1 We choose the subscript G. for notational consistency between FB- and updates.
5

---

Published in Transactions on Machine Learning Research (08/2025)
10"
10
Aeal
10gn (E30)
10g (PSD)
logis (ES0)
30
10-4
10ga
logu: (h)
loguo
(a) GD = 0.1
(b) GD 17 = 2000
(c) FB-Adam 17
(d) FB-Adam rj = 10
Figure 2: ESD W Wi for GD/FB-Adam with varying Y3 and 2000, 1000, 1500, softplus, tanh, Pe 0.3, À 0.01.
3.2 Alignment and HT Metrics
Alignment between W, To quantify the "extent" of feature learning in our student network during training, we measure the aligmment. between the first principal component of W (denoted as ui) and the target direction 8* (Ba et al. 2022 Wang et al. 2023 as: sim W, B**) 3*
Kernel Target Alignment (KTA). In addition to analyzing W. we also consider the alignment between the Conjugate Kernel (CK) (Wang et al. 2023 Lee et al. 2018 Matthews et al. 2018 Fan & Wang 2020) based on hidden layer activations and the target outputs. Formally, consider the hidden layer activations of
the holdout data Z VAo
va WX and define CK as: K Z Raxn The KTA (Cristianini
ct al.
2001 between K and yy E Rax" is given as:
(K yy
KTA
(K, vy ||
Σ 1
Kig (v.))
K
lyy
Power-law fits (PL_Alpta Alpha Hill, PL Alpha_aaaK KS) To quantify the heaviness of the tails in the ESD, we measure PL FL__AlphaHHip _ill (hoou et al. 2023), and PL_Alpha KS, which refer to the power-law exponents that
are fit to the ESD of W, W, using the Hill estimator Hill 1975) and based on the Kolmogorov-Smimon
statistic respectively MMrrtin et al. 2021 Clauset et al. 2009).
4 Splke Phenomenon after One Optimizer Step
In this section, present empirical and theoretical results for the scale of n for FB-Adam that results in the "Bulk+Spikke ESD and facilitates feature learning in the student network after the first update. The discussion on the first optimizer update and is essential, as we show in Scction that the Bulk+Spiie ESD after the first update facilitates the emergence of HT ESDs.
4.1 The Sulk+Spike ESD by Scaling ?
Setup. We consider the two-layer NN f(-) of width 1500, o tanh and train it on a datasct of size 2000, with input dimension d 1000. We choose a, softplus as the larget link function and set the label noise to P. 0.3, with À 0.01. The test data consists of 200 samples.
FB-Adam needs much smaller than GD to exhibit Bulk+SSiike ESD. By varying 7, we train the student network f()) for one step using CD/FB-Adam. Observe from Figure 2 that after one GD update with 77 = 0.1, the ESD of W,T W1 remains largely unchanged from that of W Wo (i.e ESD at random initialization). On the other hand, Figure 2b illustrates that for 17 = 2000, the ESD of W. W, exhibits a spike. However, FB-Adam exhibits a spike the ESD of W, W, after the first step even with n (see Figure 2ch. Finally, for 7 10, the ESD tends towards seemingly bimodal distribution (see Figure 2d

---

Published in Transactions on Machine Learning Research (08/2025)
0.95
2.5
3.0
0.040
0.5
10 A8AA
0.035
0.90
2.0
SinDE,R R')
0.030
0.85
1.5
0.025
0.2
1.0
0.1
0.80
0.5
0.015
0.0
-2
10gus (n)
Ing
logu
(a) GD loss
(b) FB-Adam loss
(c) KTA
(d) sim (W, ,8")
Figure 3: Losses, KTA, sim (W, for ʃ()) trained with one-step of GD, FB-Adam. Here n = 2000, 1000,
h 1500, o+ = softplus, o = tanh, Pe 0.3, À 0.01.
Impact on losses, KTA, and sim(W, B*). As the choice of optimizer affects the scale of leading to a Bulk+SSpike ESD of WI we vary 7 across {0.001, 0.1,, 10, 100, 1000, 2000, 3000} and plot the means and standard deviations of losses, KTA and sim(W.8' across 5 runs in Figure 3 In the case of GD, observe that 17 = 2000 is the threshold for the: reduction of train and test losses (Figure Ba), an increase in KTA (Figure 3c), and an increase in sim(W, (Figure 3d.. Thus, implying that the occurrence of spike leads to better generalization after one step (as also verified by Ba et al. (2022) ). In the case of increase in KTA/ (TA/sim(W,FF) 8*)) For
FB-Adam, 0.01 does not improve generalization and we do not 0.01 S n I 1, generalization improves, and KTA/Sim W, 3, values increase. For 17 > 1, generalization degrades and there is a slight reversal in the trend for KTA. Note that due to the two-phase training strategy, the large magnitude of the first step update with > can lead to lower training loss (since the last layer is computed based on a closed-form solution), but lead to higher test loss. These observations hint at a sweet spot for 7) beyond which FB-Adam leads to poor test performance (sce also Appendix F.1). Towards understanding these observations, we focus the following pressing question: why does n 0.1 suffice for one-step of FB-Adam to exhibit spike in the ESD, whereas one-step GD requires 17 = 2000 ? Specifically, how large should v) be for FB-Adam to exhibit spike in the ESD?
4.2 Theoretical Results for Scaling 7) with FB-Adam
To thcorctically answer the above questions, we start by formulating the first step gradient Go as:
1 Go = nvd Lh ay
Vh aaTo ( (Va (/awx))) od (Vamax X.
(8)
Here d + R is the derivative of the activation function a acting element-wise on XW, /vd. Based on equation equation 4 let P1 1/1/2 elaxd. Considering the FB-Adam epsilon hyper-parameter e z 0 [Kunstner et al. 2022), the scaled sign update (Bernstein et al. 2018) can be given as:
1- 31 v1 - B2 sign(Go).
= M1
(9)
Theorem 4.1 Given the two-stage training procedure with large n, d, such that = d, and large (fized) h, assume the teacher F is Lipssccch with [|F* Oa(1), and a normalized 'student' activation a, which has boundded first three derivatives almost surely and satisfies E[o((2] O, 0,E[20(2)] # 0, for 1)), then the matria norm bounds for the one-step FB-Adam update can be given as:
|Go]1 @ap(Vhd), |G -ap((hhd).
(10)
Appendix B presents the proof. The sketch of the proof is as follows: First, we show that Go does not contain values that are exactly equal to almost surely. This allows us to obtain Go g
Oup Vhd).
Next, we leverage a rank approximation of Go (in the operator norm (Ba al. 2022)
show that
sign (A) (Go). Finally, we obtain the lower-bound of |Goll tue( (hh) to prove the theorem
2hh normalized activation is not practical limitation and is solely required for the proof BB al. 2022
7

---

Published in Transactions on Machine Learning Research (08/2025)
Corollary 4.2 Under the assumptions of Theorem 4.1 have the following learning rate scales
1 0
[W Wolle [Wlll
(11)
n n= 2 (1/Vh)
| Woll2 [Woll2
where Woll2 O..r(Va
Woll, (vhhv)
Theorem 4.1 shows that the spectral and Frobenius norms of Go scale similarly and that the top singular value contributes the most the Frobenius norm d, h increase. As consequence, Corollary 1.2 indicates that 17 (1) (which is independent of h. d) is sufficient for n/oo to result in 'spike' in the ESD of W, W1. This explains our empirical observations above where even "f was sufficient for the ESD to transition into a Bulk+Sppike shape.
Remark. We note that similar result for in the case of GD was previously established by Ba el al. (022 However, they employ a mean-fieet initialization of the two-layer NN f()) and differs from our WTK--base initialization (Wang et al., 2023). Nonetheless, we show that our results can be extended to such a setup as well and the adjusted can indeed explain the Sulk+Spike ESD after one step update (Appendix presents a comprehensive discussion).
5 HT Phenomenon after Multiple Optimizer Steps
In this section, we empirically analyze the ESD evolution from the "Bulk+Spiko" shape into an HT distri- bution. This is followed by correlation analysis with generalization of the two-layer NN.
5.1 Transitioning from Sulk+Spike to HT ESD
Spike after one step facilitates the emergence of HT ESD. We begin by employing the same setup as Section 4 and show in Table [ that the spike in the ESD after one step facilitates the emergence of НT ESD. We observe that after just 10 FB-Adam steps with " requires 10000. This indicates that the relative position of the spike from the bulk (which is captured by the spectral gap in this sctup) plays key role in determining the step complexity of the HT ESD emergence 3 Finally, for 0.01, which does not exhibit an HT ESD even after t = 10000, we observed that a significantly long amount of training for L = 10 steps can lead to HT. Note that v 0.01 tends to fall within the (0(1/Vh), @(1)) range, considering the scaling constants for θ This leads to a scenario where a spike does not emerge after one step but the magnitude of gradient updates over 10 steps can lead HT ESD (see Table 5). Given this observation, extremely small y) might not be able to exhibit such changes to the ESD even after t 10t steps (see Corollary 5.3 in Wang et al. (2023) for related formal result). We
the HT ESD emerges. However, 1/ 0.1
extend the analysis to wider range of in Appendix E (see Table
intermediate stage. By considering Adam with 17 0.2 and the same setup Section 4 we illustrate in Figure
Bulk-Decay
4 that the spike emerges after one-step (Figure 1a) and gradually decays the Bulk (Figure 4b Figure 4c toward a HT distribution (Figure 4dh. Since practical settings employ deep NNs and train with stochastic gradient noise on complex datasets, the ESD evolution is much more muanced. Especially the Bleeding-out phase and the presence of multiple spikes before the onset of Bulk- Decay. Nonetheless, we extended our results from the student-teacher setup to more practical settings, such as VGG, and ResNet models, and validated our HT ESD findings in those settings as well (see Section 5.3 Overall, we emphasize that our setup with two-layer NNs and single-index models opens up the possibility of approximating such dynamics while being subject to theoretical treatment.
Thhoreticcl analysis of the spectral gap and its role on step complexity for HT ESD emergence is out of scope this paper and can be valuahle direction for future research.
8

---

Published in Transactions on Machine Learning Research (08/2025)
t
t 100
10000
10
10a
valus
Agg
10gu (ESD)
Iosu (ESD)
l0g 10gix (550)
7 = 0.01
10
10
2.5
3.5
2.5
3.5
2.5
3.5
legun MA
loguo
logio
3.0
3.0
10P
Austccoo
(ES0)
Ioqu (E50/ (ES0)
logio (F50)
7) = 0.1
10
10-1
3.0
IoguitA)
logu (AA
log CAL
3.00
10"
10"
1b0 1bq. (850)
(ES0)
log (F50)
10-4
7) 1
10
10gu 1h))
logu (A))
loguo (ki)
Table 1: Evolution ESD over steps t E {1, 100, 10000} with 17 E {0.01, 0.1, 13 for FB-Adam.
0.0006
0.0008
0,0008
Medl
0.0005
0.0006
0.0006
0.0006
0.0004
0.0004
0.0004
0.0004
0.0002
0.0002
0.0002
0.0002
0,0000
0.0000
5000 7500 10000
0.0000
5000
10000
15000
0.0000
5000
10000
15000
(a) Bulk+Spike
(b) Bulk-Decay (onset)
(c) Bulk-Decay (progress)
(d) Heavy Tails (onset)
Figure 4: Evolution of ESD (linear-linaar scale) from Marchenko-Pastur fit (at initialization 0: red
bulk) to (a) Sulk- (t 1), (b) the onset of Bulk-Decay (t 5), (c) Continued decay of the bulk
(t 10), (d) onset of heavy tails (t = 50), with FB-Adam optimizer and 0.2.
5.2 Impact on Generalization
Setup. We consider the two-layer student NN f(.) of width h = 1500, a = tanh and train it on a dataset of size n = 8000 for 10 steps using GD CD/FB-AA AD/PB-Adam. We choose a sample dimension = 1000, a = softplus and P. = 0.3. The test dataset has 200 samples.
Correlations between ESD and losses. We plot the means and standard deviations across runs for the train/test losses, KTA, PL_ PL_Alpha_Hir Hill and PL Alphapaa__R__ KS (see Section 3.2 of W Wio after 10 GD/FB-Ade Adam updates by varying 7 across (0.001, 0.01, 0.1, 0.1, value of PL_ Al__apha_aa Hill / PL Alpha_ppaaa_aa KS indicates heavier-tailed spectrum Observe that for baseline GD experiments with 1) 2 1000, the reduction in train and test losses are correlated with an increase in KTA and a decrease in PL_ Awp.apppaaaaaa Hill and PL_ PL_Alpha_KS. Additional experiments are presented in Appendix F.2
1, 10, 100, 1000, 2000, 3000 in Figure 5 A lower
In the case of FB-Adam, a much clearer correlation between the training loss, KTA, PL_Alpha_Hi Mallpa_aHllll and PL PL_Alpha_Hr can be observed. Especially, there seems to be a region of benign learning rates (0.01 S n S 1)
4The effects of the power-law exponent estimation approaches discussed Appendix G

---

Published in Transactions on Machine Learning Research (08/2025)
1,2
Gdarlaan
2.4
1.0
0.05
Alptt H41
T_Alpha KS
2.0
0.9
0.04
0.8
o2 (188
0.07
0.7
1.6
0
log
logu: (1)
10g (1)
log
(a) loss
(b) KTA
(c) PL Al_hapaa_H111 Hill
(d) PL PL_Alpha_KS
Figure 5: Losses, KTA, PL_ Alppa__ill, PL_Alpha_ KS after 10 steps of GD, FB-Adam, with n = 8000, 1000, h 1500, o+ = softplus, a = tanh, Pe 0.3, À 0.01.
for which, the PL PL_A1pppp Alpha_Hill Hill estimates lie in the range of (2, 2.5) and a decrease in the estimate (resulting in a 'heavier' tailed ESD) improves generalization. For S 17 S 100, although we observe similar values of PL_Alpha_ Hill, the ESDs of Wio differ in the scale of the singular values, and the spike seems to have large inftuence on the estimation of PL_ A__Alpha__Hiie Hill (see Figure 24 in Appendix F). However, the PL_ Alpha_aa_K KS captures the monotonically decreasing trend for this range of 1). Finally, for extremely large >> 100, we observe much smaller estimates of Al_Alpha__i _atlll_llll__a (< 1.8) but these extremely heavier tails do not correlate with better generalization. We also note that these benign r ranges vary based on the choice of the activation functions. In particular, when o = o, = tanh, we observed that the range can be reduced by an order of magnitude (Appondix F.5). Overall, these observations support the conclusions of Martin & Mahoney (2021a); Martin et al. (2021) which state that well-trained deep NNs do not exhibit extreme HT ESDs but rather whose power-law estimates lie within suitable range. For instance, range of (2, 4) for the truncated power-law fit estimates.
A note on the suitable range of tail index values Previous works such as Simsekli et al. (0200a Hodgkinson et al.] (2022) studied the limiting distribution of the weight matrix values by modeling the stochastic gradient noise using Levy/Felles processes. In particular, they showed that smaller tail-index values lead to smaller generalization errors. The underlying idea is that when the weight values tend to an HT distribution, then the resulting ESD also an HT (sce Arous & Guionnet (2008)). Given this explanation, we note that the theoretical results by [Simsekli al. (2020a); Hoddkinson shown to hold in practice by their numerical experiments, where the tail index tends to lie in suitable range (depending on the PL alpha fit approaches) and represents heavy-tailed noise for good generalization. Even though our setting differs significantly from these previous works and avoids gradient noise, the suitability of such a range seems to hold. In particular, for our setup the PL_ PL_Alpha_ Aalpa_aillll11a11a1ppppppppppll1 estimates lie in the range of (2, 2.5) where a lower value implies a 'heavier' tailed ESD, and improves generalization of student network.
al. 22022 were
Remark. Since practical training approaches employ techniques such as weight normalization (WN) and learning rate schedules, we present a preliminary analysis of their role in the ESD evolution and genoralization in Appendix [.4. We employ a WN technique (Huang al. 2023) after each update and observed that it leads to relatively heavier-tailed spectra (i... lower PL PL_Alpha__ Hill) while exciibtting similar correlations with generalization as discussed above. On the other hand, by employing schodulers such as torch. optim StepLR, we showcase a fine-grained manipulation of the ESD evolution depending on the decay rate ?.
5.3 Applicability of Insights to Deeper NNs
We verify our findings full-batch training and optimizer-dependennt learning rates for the emergence of HT ESD with FB-Adam in practical deep NNs. We choose the VGG11 and ResNet18 models to train the CIFAR10, MNIST and SVHN datascts. We avoid (very) large NNs and datasets due to GPU memory constraints for full-batch training and instead focus on the paper's s main theme. Figure iillstraaes the ESDs of selected layers in the deeper NNs and indicates the emergence of HT ESDs during the early phases of training (i.c just after 10 steps) (See Appendix F.6 for further illustrations). The main takcaway from these results is the observation that such emergence in practical NNs is not limited training till convergence with mini-balches (Martin & Mahoney 2021a). Furthermore, since the power-law fits have been used to determine
10

---

Published in Transactions on Machine Learning Research (08/2025)
10P
Juned
10
10
(ES0)
togui (ESO)
login
loginiES0)
10
10m4
10-34
-5.0
-2.5
0.0
logui
logua
(a) VGG11-I4 (CIFARIO)
(b) VGG11-L7 (MNIST) (c) ResNet18-L5 (CIFAR10) (d) ResNet18-L10 (SVHN)
Figure 6: ESDs VGGI1, ResNet18 after 10 steps with FB-Adam(1 = 0.01) on MNIST, CIEARIO, SVHN.
if layer is under-trained or over-trained Martin al. (2021) Zhou et al. (023), our results highlight that such connections hold more muance than considered by previous efforts. Thus opening up wide range of
avenues to explore the interplay between the ESDs and training convergence (see also Appendix
6 Conclusion
This paper presents different angle to study the emergence of HT ESDs during NN training. Unlike existing studics that mostly focused on the mini-batch settings, we show that full-batch GD or Adam can still lead to HT ESDs in the weight matrices after only a few optimizer updates with large Our paper also connects with several ongoing studies in this field. In particular, our study analyzes the '5++' phase model (Martin & Mahoney 2021a] of ESD evolution and sheds light on the transitions from a Marchenko-Pasteee (MP) fit (i.e. random mitialization) + "Bulk+Spikee - "Bulk- Deelk-" + HT distributions. We also showcase that the "Bulk-Decay" ESD can be treated bulk (see also Appendix F.3). Our paper also presents several surprising phenomena: (1) the emergence of the HT spectra seems to require only single spike aligned with the teacher model; (2) the emergence of the HT spectra can appear early during training, way before the NN reaches low training loss; (3) several factors, such as weight. normalization and learning rate scheduling, can all contribute to the emergence of HT ESDs. Overall, our work connects the feature learning aspect of the bulk + spike ESD (Section 4 with the early emergence of HT ESDs (Section 5) and presents a unique perspective to study the correlations between HT ESDs and generalization in NNs (especially with FB-Adam).
intermediate state generated from diffusing the spike into the main
Acknowledgments
This work is supported by DOE under Award Number DE-SC0022584 and Dartmouth College.
References
Emmanuel Abbe, Enric Boix Adsera, and Thoodor Misiakiewicz. Sgd learning on ncural networks: leap complexity and saddle-to-saddle dynamics. In The Thirty Sixth Annual Conference on Learning Theory, pp. 2552-2625 2623. PMLR, 2023.
Greg W Anderson, Alice Guionnet, and Ofer Zeitouni. An introduction to random matrices. Number 118. Cambridge university press, 2010.
Gérard Ben Arous and Alice Guionnet. The spectrum of heavy tailed random matrices. Communications in Mathematical Physics, 278(3):715 751, 2008.
Gerard Ben Arous, Reza Gheissari, and Aukosh Jagannath. Online stochastic gradient descent non-convex losses from high-dinensional inference. Journal of Machine Learning Reseorch, 22(106):1 51, 2021.
Jimmy Ba, Murat A Erdogdu, Taiji Suzuki, Zhichao Wang, Denny Wu, and Greg Yang. High-dimensional asymptotics of feature learning: How one gradient step improves the representation. Advances in Neural Information Processing Systems, 35:37932-3794m 2022.
11

---

Published in Transactions on Machine Learning Research (08/2025)
Jimmy Ba, Murat A Erdogdu, Taiji Suzuki, Zhichao Wang, and Denny Wu. Learning in the presence of low-dinensional structure: A spiked random matrix perspective. In Thärty-seuenth Conference on Neural Information Processing Systems, 2023.
Jinho Baik, Gérard Ben Arous, and Sandrine Péché. Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices. Annals of Probubility,, pp. 1643 1697, 2005.
Mclih Barsbey, Milad Sefidgaran, Murat A Erdogdu, Gacl Richard, and Umut Simsekli. Heavy tails in sgd and compressibility of overparametrized neural nelworks. Advances in neural information processing systems, 34:29364 29378,, 2021.
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Avizadenesheli, and Animashree Anandkumar. signsgd: Com- pressed optimisation for non-convex problems. In International Conference Machine Learning, pp. 560-569. PMLR, 2018.
Alberto Bietti, Joan Bruna, Clayton Sanford, and Min Jae Song. Learning single-index models with shallow neural networks. Advonces in Neural Information Processing Systems, 35:9768 9783, 2022.
Aaron Clauset, Cosma Rohilla Shalizi, and Mark EJ Newman. Power-law distributions in empirical data. SIAM review, 51(4):661-70) 703, 2009.
Nello Cristianini, John Shawe-Taylor, Andre Elissoeff, and Jaz Kandola. On kernel-target alignment. Ad- vances in neural information processing systems, 14, 2001.
Hugo Cui, Luca Pesce, Yatin Dandi, Florent Krzakala, Yue M Lu, Lenka Zdcborová, and Bruno Loureiro. Asymptotics of feature learning in two-layer networks after one gradient-step. arXiv preprint arXiv:2402.04980 2024.
Alex Damian, Eshaan Nichani, Rong Ge, and Jason D Lee. Smoothing the landscape boosts the signal for sgd: Optimal sample complexity for learning single index models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Alex Damian, Loucas Pillaud-Vivien, Jason D Lee, and Joan Bruna. The computational complexity of learning gaussian single-index models. arXiv preprinl arxi::2403.0055 055529 2024.
Alexandru Damian, Jason Lee, and Mahdi Soltanolkotabi. Neural networks can learn representations with gradient descent. In Conference on Learning Theory, pp. 5412-5452. PMLR, 2022.
Yatin Dandi, Florent Krzakala, Bruno Loureiro, Luca Pesce, and Ludovic Stephan. How two-layer neural networks learn, one (giant) step at a time. In NewrIPS 2023 Workshop on Mathematics of Modern Machine Learning, 2023.
Yatin Dandi, Luca Pesce, Hugo Cui, Florent Krzakala, Yue M Lu, and Bruno Loureiro. A random matrix theory perspective on the spectrum of learned features and asymptotic generalization capabilities. arXiv preprint Xrv::22410. 18938, 2024a.
Yatin Dandi, Emanuele Troiani, Luca Arnaboldi, Luca Pesce, Lenka Zdeborová, and Florent Krzakala. The benefits of reusing batches for gradient descent in two-layer networks: Breaking the curse of information and leap exponents. XrXiv preprinl 0322200 2024b.
Yatin Dandi, Luca Pesce, Hugo Cui, Florent Krzakala, Yue Lu, and Bruno Loureiro. A random matrix theory perspective on the spectrum of learned features and asymptotic generalization capabilities. In The 28th International Conference on Artificial Intelligence and Statistics, 2025.
Xiucai Ding and Fan Yang. Spiked separable covariance matrices and principal components. The Annals of Slatistics, 1912:11113 1138, 2021.
Ahmed El Alaoui and Michael I Jordan. Detection limits in the high-dimensional spiked rectangular model. In Conference On Learning Theory, pp. 410-438. PMTR, 2018.
12

---

Published in Transactions on Machine Learning Research (08/2025)
Zhou Fan and Zhichao Wang. Spectra of the conjugate kernel and neural tangent kernel for linear-width neural networks. Advances in neural information processing systems, 33:7710-777 2020.
Jürgen Forster, Matthias Krause, Satyanarayana V Lokam, Rustam Mubarakzjanov, Niels Schmitt, and Hans Ulrich Simon. Relations between communication complexity, linear arrangements, and computational complexity. In International Conference on Foundations of Software Technology and Theoretical Computer Science, pp. 171-182, Springer, 2001.
Matan Gavish and David I. Donoho. Optimal shrinkage of singular values. IEEE Transactions on Information Theory, (11:)2137211222 2017.
Rong Ge, Sham M Kakade, Rahul Kidambi, and Praneeth Netrapalli. The step decay schedule: A near optimal, geometrically decaying learning rate procedure for least squares. Advances in newral information processing systems, 32, 2019.
Federica Gerace, Bruno Loureiro, Florent Krzakala, Mare Mézard, and Lenka Zdeborová. Generalisation error in learning with random features and the hidden manifold model. In International Conference on Machine Learning, pp. 3452 3462. PMLR, 2020.
Sebastian Goldt, Bruno Loureiro, Galen Reeves, Florent Krzakala, Mare Mézard, and Lenka Zdeborová.
The gaussian equivalence of generative models for learning with shallow neural networks. In Mathematical and Scientific Machine Learning, pp. 426-471, PMLR, 2022.
Ronald L Graham and Donald E Knuth. 0. patashník, conerete mathematics. In A Foundation for Compuler Science, volume 104, 1989.
Mert Gurbuzbalaban, Umut Simsekli, and Lingjiong Zhu. The heavy-tail phenomenon in sgd. In International Conference on Machine Learning, pp. 3964 3975. PMLR, 2021.
Trevor Hastie, Robert Tibshirani, Jerome Friedman, and James Franklin. The elements of statistical learning: data mining, inference and prediction. The Mathemalical Intelligencer, 27(2):83 85, 2005.
Bruce M Hill. A simple general approach to inference about the tail of a distribution. The annals of statistics, pp. 1163 1174, 1975.
Liam Hodgkinson and Michael Mahoney. Multiplicative noise and heavy tails in stochastic optimization. In International Conference on Machine Learning, pp. 4262-4274. PMLR, 2021.
Liam Hodgkinson, Umut Simsekli, Rajiv Khanna, and Michael Mahoney. Generalization bounds using lower tail exponents in stochastic optimizers. In International Conference on Machine Learning, pp. 8774-8795. PMLR, 2022.
Hong Hu and Yue M Lu. Universality laws for high-dimensional learning with random features. IEEE Transactions on Information Theory, 69(3):1932-199 1964, 2022.
Lei Huang, Jie Qin, Yi Zhou, Fan Zhu, Li Liu, and Ling Shao. Normalization techniques in training dnns:
Methodology, analysis and application. IEFE transactions on pattern analysis and machine intelligence, 45(8):10173 10196, 2023.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXio preprint 1r 6980, 2014.
Frederik Kunstner, Jacques Chen, Jonathan Wilder Lavington, and Mark Schmidt. Noise is not the main factor behind the gap between sgd and adam on transformers, but sign descent might be. In The Fleventh International Conference on Learning Representations, 2022.
Itamar D. Landau, Gabriel C. Mcl, and Surya Ganguli. Singular vectors of sums of rectangular random matrices and optimal estimation of high-rank signals: The extensive spike model. Phys. Rev. E, 108: 054129 29, Nov 2023. doi:
13

---

Published in Transactions on Machine Learning Research (08/2025)
Jachoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeftrey Pennington, and Jascha Sohl- Dickstein. Deep neural networks as gaussian processes. In International Conference on Learaing Repre- sentations, 2018.
Jason D Lee, Kazusato Oko, Taiji Suzuki, and Denny Wu. Neural network learns low-dimensional polynomials with sgd near the information-theorciic limit. Advances in Neural Information Processing Systems, 37: 58716 58756, 2024.
Fanghui Liu, Xiaolin Huang, Yudong Chen, and Johan AK Suykens. Random features for kernel approxima- tion: A survey on algorithms, theory, and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):7128 7148, 2021.
Zihang Liu, Yuanzhe Hu, Tianyu Pang, Yefan Zhou, Pu Ren, and Yaoqing Yang. Model balancing helps low-data training and fine-tuning. arXiv preprint orXiv:24100 121178 2024.
Cosme Louart, Zhenyu Liao, and Romain Couillet. A random matrix approach to neural networks. The Annals of Applied Probability, 28(2):1190-122 1248, 2018.
Charles H Martin and Michael W Mahoney. Heavy-tailed universality predicts trends in test accuracies for very large pre-trained deep neural networks. In Proceedings of the 2020 STAM International Conference on Data Mining, pp. 505 513. SIAM, 2020.
Charles H Martin and Michael W Mahoney. Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning. The Journal of Machine Learning Research, 22 (1):7479 7551, 2021a.
Charles H Martin and Michael W Mahoney. Post-mortem deep learning contest: a simpson's paradox and the complementary roles of scale metrics versus shape metries. arXiv preprint 1rXix:2106.0034,, 2021b.
Charles H Martin, Tongsu Peng, and Michael W Mahoney. Predicting trends in the quality of state-of- thrt--rrttt--tttt neural networks without access to training or testing data. Nature Communications, 12(1): 2021.
Alexander G de G Matthews, Jiri Hron, Mark Rowland, Richard E Turner, and Zoubin Ghahramani. Gaus- sian process behaviour in wide deep neural networks. In International Conference on Learning Represen- tations, 2018.
Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 75(4):667 766, 2022.
James A Mingo and Roland Speicher. Free probability and random matrices, volume 35. Springer, 2017.
Behrad Moniri, Donghwan Lee, Hamed Hassani, and Edgar Dobriban. A theory of non-linear feature learning with one gradient step in two-layer neural networks. arXiv preprint arXiv:2310.07811 2023.
P.A.P. Moran. An introduction probability theory. 1984.
Alireza Mousavi-Hosseini,, Sejun Park, Manuela Girotti, Joannis Mitliagkas, and Murat A Erdogdu. Neu- ral networks efficiently learn low-dimensional representations with SGD. In The Eleventh International Conference Learning Represendations, 2023a.
Alireza Mousavi-Hosseinii Denny Wu, Taiji Sumuki, and Murat A Erdogdu. Gradient-based fea learning under structured data. In Thirth-seventh Conference on Neural Information Processing Systems, 2023b.
Thanh Huy Nguyen, Umut Simsekli, Mert Gurbuzbalaban, and Gaël Richard. First exit time analysis of stochastic gradient descent under heavy-tailed gradient noisc. Advances in neural information processing systems, 32, 2019.
Eshaan Nichani, Alex Damian, and Jason D Loc. Provable guarantees for nonlincar feature learning three-layer neural networks. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pp. 10828-10875, 2023.
14

---

Published in Transactions on Machine Learning Research (08/2025)
Anant Raj, Lingjiong Zhu, Mert Gurbuzbalaban, and Umut Simsekli. Algorithmic stability of heavy-tailed sgd with general loss functions. In International Conference on Machine Learning, pp. 28578-28597. PMLR, 2023.
Andrey A Shabalin and Andrew B Nobel. Reconstruction of low-rank matrix in the presence of gaussian noisc. Journal of Multivariate Analysis, 118:67-76, 2013.
Unut Simsekli, Levent Sagun, and Mert Gurbuzbalaban. tail-index analysis of stochastic gradient noise in deep neural networks. In International Conference on Machine Learning, pp. 5827-5837. PMLR, 2019.
Umut Simsekli, Ozan Sener, George Deligiannidis, and Murat A Erdogdu. Hausdorff dimension, heavy tails, and generalization in neural networks. Advances in Neural Information Processing Systems, 33:5138 5151, 2020a.
Umut Simsekli, Lingjiong Zhu, Yee Whye Teh, and Mert Gurbuzbalaban. Fractional underdamped langevin dynamics: Retargeting sgd with momentum under heavy-tailed gradient noise. In International conference on machine learning, pp. 8970 8980. PMLR, 2020b.
Charles M Stein. Estimation of the mean of multivariate normal distribution. The annals of Statistics, pp. 1135 1151, 1981.
Emanuele Troiani, Vittorio Erba, Florent Krzakala, Antoine Maillard, and Lenka Zdeborová. Optimal denoising of rotationally invariant rectangular matrices. In Mathematical and Scientific Machine Leorning. pp. 97-112. PMLR, 2022.
Hongjian Wang, Mert Gurbuzbalaban, Lingjiong Zhu, Umut Simsekli, and Murat A Erdogdu. Conver- gence rates of stochastic gradient descent under infinite noise variance. Advances in Neural Information Processing Systems, 34:18866-18877, 2021.
Zhichao Wang, Andrew William Engel, Anand Sarwate, Ioana Dumitriu, and Tony Chiang, Spectral evolu- tion and invariance in linear-width neural networks. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
John Wright and Yi Ma. High-DDmeessinue Data Analysis with Low- Low-Dimensional Models: Principles, Com- putation, and Applications. Cambridge University Press, 2022.
Yaoqing Yang, Ryan Theisen, Liam Hodgkinson, Joseph E Gonzalez, Kannan Ramchandran, Charles H Martin, and Michael W Mahoney. Test accuracy generalization gap: Model selection in nlp without accessing training or testing data. In Proceedings of the 29th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 3011-3021, 2023.
Yefan Zhou, Tianyu Pang, Kegin Liu, Charles H. Martin, Michael W. Mahoney, and Yaoqing Yang. Tem- perature balancing, layer-wise weight analysis, and neural network training. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Vladimir Antonovich Zorich and Octavio Paniagua. Mathematical analysis II, volume 220. Springer, 2016.
Aaron Zweig, Loucas Pillaud- Vivien, and Joan Bruna. On single-index models beyond gaussian data. In seirth-seventhh Conference on Neural Information Processing Systems, 2023.
15

---

Published in Transactions on Machine Learning Research (08/2025)
A Generating Very Heavy-Tailed Tailed Spectra without Gradient Noise
Linear-Linear plot without truncation
Linear-Linear plot with truncation
Log-Log plot
Inttll
0.0005
Intil sep10
Intill step
0.0003
0.0004
10
logue (ESD)
ESD
0.0002
0.0003
0.0002
0.0001
10
0.0000
0.0000
0.0
0.5
1.0
1.5
25000 50000
75000
2
1e7
A
log (A.)
GD 17
5000
(b) 17
5000
GD 17 = 5000
Lincar-Lincar plot without truncation
Linear-Lincar plot with truncation
Log-Log plat
0.0005
Intill
0,0005
sle20
slep10
0.00D4
10"
0.0004
logue (ESD)
E50
0.0003
ESO
0.0003
0.0002
0.0002
0.0001
0.0001
10-2
0.0000
0,0000
20000
40000
60000
20000
40000
60000
2
log1o (Ai)
(d) Adam 77
(e) FB-Adam 7
(f) FB-Adam 7=
Figure 7: Emergence of HT spectra after 10 CD/FB-Adam steps with weight normalization. Here n 4000, d 500, h 3000, a, softplus, a tanh, Pe 0.3. PL @__Alpha_Kee KS for GD (first row) is 1.58, and PL_ Alpha KS for FB-Adam (second row) is 1.59.
In this section, we present two examples of highly HT ESD, shown in Figure 7 If one only observes the linear-linear plot (first column), the shapes can be invisible due to large eigenvalues. For example, see Figure 7a After truncation, they become clearer (middle column). To quantitatively verify our observations, we adopted the Weight Watcher (Martin et al.) 2021) Python APIs to fit a power-law (PL) distribution to these ESDs. We find that both ESDs have a PL coefficient PL PL_Alpha_K, _apaaasaaaaaaa smaller than 2. As discussed in Martin & Mahoney 2021a Table 3), this indicates that these ESDs have entered the very HT regime. More importantly, these ESDs are generated without gradient noise after only 10 GD/FB-Adam steps.
B Proof of Theorem 4.1
In this section, prove Theorem 4.1 and conduct numerical simulations for empirical verification. We begin by stating and discussing the main assumptions of the theorem.
Assumption B.1 Gaussian Initialization. The entries of the weights are sampled independently Wolp fi.dd N(O0,) 1) and [a]; Ltd N1001),Vie [h], e [d].
Assumption B.2 Normalized Activation. The nonlinearr activation a has A bouunded first three deriva- tives almost surely. In addition, a satisfies B[a(2)] = 0, [[)) # O, for 2 ~ No 1).
be the tanh function and 2 NN0, 1). Since (2)) = tanh(2) -= it is easy to check that a(z) has bobuudd first three derivatives and it an odd function satisfying (2) (-22), we have:
Discussion. Consider a: R
c=4e-s
[p(c) =
( (=)))(2)d== = 1 a(=)p(=)dz + L-d o(())(=)dz= 0.
(12)
16

---

Published in Transactions on Machine Learning Research (08/2025)
where p(z) = 1 e 2. Next, observe that V25"
ztanh(3 (z) z(E² - = 6 e= 2 0,
(13)
where 2z) holds only when Let E and expand the expectation [x(=)[20(2]] as follows:
U-e [p(a)[20(<3) 20 =0())p(2)d== = / zo(z)p(2)dz L =0(=)((=)dz f z0(2)p(2)d2.
0+e
Jote
(14)
From equation 13 the first two terms are 0, ie::
L 20())((2)d2 0
20()) 0,
(15)
whereas for - O, the third term can be bounded as:
0+z J
Jo =a(2)p(=)d=2
(16)
By combining the above results, a satisfies: Rpte) >0= Bxxa) *
Assumption B.3 Teacher-Student Setup. The target labels are generated by the single inder teacher model y, = F* (xx) + E,, where æ; Li.d. (), is i..dd Gaussion noise with mean 0 and voriance pa: and the teacher F is d -Lpisschht with [| La = Oa(1).
Discussion. Recall that our teacher model is given by F"(x;) (B* æ;), where 8 E Edd-1 22 E R E [n], and (=) = log(1 + + is the softplus function. Note that the derivative oz(2) Ite= < 1 is bounded, and g x; - æ1ل122 = || l22 ||= æ3اl2 Væ;, 2tj Rd This gives us:
c²
| T x.) 0 21)i|2 S || x; 3 STTa,le 22112 s 11 1| 2z112 Vz,, By RR
(17)
which implies that F* is a 11,3"ll- function. Next, we consider B æ, for x ~ N(O, Ia), which implies NO, (| and bound a. (2) by < (2) Do. (2), where Io. (2) is:
go. (=) = 1, 2+1, 1, 2 0.
(18)
Based on these results, we calculate | 1 as follows.
F 12.- = / oo()))du < / gr. (2)2
(19)
210 dz the Gaussian measure. Further expansion of the upper bound gives: where du 11
90. (z)
(2)
21 TZ dz
R
JR
V2T
19-11,
0
1
2
2I 1 dz +
(2² + + 1)
e
21 1
dz
2T 18*1/2
/2T ||3111
000
1 + 2
11,8 112 +
|z| 22T
e
21013 dz
|| 112
2
= 1 + 2
11,3 2
|| 1/2
< 20.
Since B¹ E gd-1 we get: 0 <
F 27
<
1+
|18 11 +
2 ||
1,y and |F* = =Oa(1).
17

---

Published in Transactions on Machine Learning Research (08/2025)
B.1 Norms of One-Step Update Matrix
We begin by formulating the full-batch gradient for the first slep (Go) as follows:
(
(awwox) oo' (
Go nyd
Vh
ay
Vh
a
Va
Go H1 Vhd
1
ay 0o1 Va (aaWor x
ay
X
+
Vhd
(20)
aaTo (1 Vd
Vawa od' ( X.
n hvd
Here utilized the orthogonal decomposition of the activation function: d' in + d' (z) to the second equality, Due to Stein's lemma (Stein 1981) O for 2 ~ N).
know that B[zo(z]] = E [o'(2)] M4, and hence E (of
Lemma B.4 (Baa et al. 2022)): ): Given Assumptions B.1 B. and B.3 let Go = (Wo - Wi) and ==
1.
Vhd ay X. Then there crists constant c, such that for sufficienlly large n.
n
P P All S
log Vn
||Goll2| 2 1 ne ne- -cooo mn
(21)
This Jemma implies that Go can be approximated by a rank one matrix A under the operator norm. Now, to analyze the FB-Adam update, recall from equation that Go = 1-8, (ggn(Go). Observe that the essence VI-B2
of the first step FB-Adam update lies in the sign matrix sign(Go) Based the expansion of Go, we leverage the rank-1 approximation matrix A to state the following lemmaa.
Lemma B.5 Given Go = , (W1- - Wo) and rank-1 1 matria ==
14 avhd
wayyy X, then for sufficiently large n: (22)
(A) sign(Go))lz = 0, (Ag sign(Go) with high probability
Let Umin min:so,so [A]yp Since our analysis is based on large (fixed) h, from Lemma B.4 we almost surely have:
Proof of Lemma B.5
Vō > 0, 3k
0, Vn > k, |Goo A 8
(23)
48 0, 3k
Vn k, |.o A I min (Vh, Vdz | - All S min (vh, vdjo
Considering d /2.mn { (vh.va) gives us:
Golli - [A] < |.o Al <
Omin
(24)
2
which implies 3k > 0, such that Vn > A:
sign Sggn(G = sign([A]]
(25)
Thus, (i) - sign (Go) O, sign( (A) = sign(Go). Next, we show that every entry of the matrix is not exactly almost surely.
Proposition B.6 Let A A==
aayT T X Raxd, then Ar, + V, [M, E [d] almost surely.
Via
Definition B.7 Given two measurable spaces (2, M,v,, we say v is absolutely continuous with respect to p if and only if
= 0, MBEM.
We denote it p.
18

---

Published in Transactions on Machine Learning Research (08/2025)
Lemma B.8 [Moran 1984) Given two measurable space (R", B(R"), mm), (RR)), L"), where m" is the Gaussian measure, L is the Lebesgue measure, we have m" & L..
By Radon-Nikodyym Theorem Moran 1984 we define the Gaussian measure using the Lebesgue integral:
{ &aL", AA e B(R)),
(E) dL"; m" m=())= : dm" = dmm==m
(26)
where + is probability density function of m" respect to L" In the problem we consider, there are two groups of Gaussian measures: a and (X11, X12, Xnd: En)...,u)) which induce ((R), m) and (IRnd+n B (R m respectively. Here at is the ith element a; Xig is the element of X; & is
the Gaussian noise random variable.
Proof of Proposition B.6 Consider A := n Vhd
PL
ay X Bixdd if we can prove the following:
ViE hh],j E 1(,, = 0) = 0,
we can further have p. Ar, = 0) S E Σラ= P(A., = 0) = 0, which means Ar, + 0, Vi E [M].j
0, [d] almost surely. So goal is prove Vi E [b].j E [d), P( P(Aij 0) 0. Given R y X E Ridd
notice that
{A., = 0) + (a 0) (1) X)5 0}.
Since a and y T X are independent, we aim to prove Vi E [M]. E [d], m ad+ (( (1y = 0) and
m(a; = 0)
We first show Vj Id),
(IP(((TT)) X)₂ 0) = 0. Consider y = o. (X E, where o,, is Softplus function
and 9 = (b). ba) T to get:
(y X),
Σ 2=1
X
I
exp
Σ
by
+1 + E1
(27)
It is easy to observe that (y X)j can be written a function:
y X), = (X1, ,1 Xnd E1,-.. ....... En))
(28)
We can easily verify fy Bdd+a + R is continuously differentiable of the first order, and we denote it fj C (RRndtn)) Considering the set.
Mi {(X,,.... _Xid. Xndd Si, :En) E Rnd+n
(29)
18,(,1, Xia, End:E1, ".. :nn = 0)
due to J3 E (Rudtan) and rank(Df,) = rank( axn
... 0/1 ) = 1, by Implicil Function Theorem (Zoiic & Paniagua 2016 we have Mi is C (nd ind+n- n di sub-manifold. Therefore (nd+n(Mi) = MidCnain =0.
Of
af. ax1s " "Xaa' 2 86 .
0f. af
By Lemma
B.8 m ndtn < Lndnn Then mdd-n (M1) = 0, we get Vj E [d] mnd+n Xx); = =0) = 0.
Observe that since any single point set is zero-measure set for Lebesgue mcasure L and m < L), we get Vi e [h].m(a 0) = 0.
Since have proved Vi E [k].j E [d], andt*((yT X), 0) = 0 and m(a; = 0) 0, we get Vi E [h].j E [d),P( P(A.. 0) = 0. Thus proving the proposition.
Lemma B.9 Forster et al. 2001) Let M (-1,+1]axi and M' E RRxxd such that sign (Maj)
sign (M) for all i E [h], E [d]. Then the following holds;
hdd
rank (M')
>
(30)
MM
19

---

Published in Transactions on Machine Learning Research (08/202))
From Lemma B.5 and Proposition B.6
it is clear that sign(Go) almost surely only contains {-1,1}. Now,
by combining Lemma B.9 and Lemma B.4 for sufficiently large n, almost surely leads to:
hd
hd
rank(A) 1
12
(31)
sign A)
Go))
Therefore, we have:
Go
-.ap(l hd)
(32)
Additionally:
Go
Go
31
sign Go)
= 1-81 Vhd
| ((hid).
(33)
/1 B2
v1 B2
Finally, by combined equations
33
and 32
we get:
Go = Dar(Vhdd,
Gu F
= -arr(vin
(34)
Thus proving the theorem.
B.2 Numerical Simulations
We consider multiple sets of n, h, d (see Table 2 for one-step FB-Adam and plot the Frobenius norm and spectral norm of Go. Figure shows lincar relationship of the norms with vhd, which validatos the results in our theorem:
Go O.. (vh)),
Go Osp(Vhd).
Linear coeft 3.16
Unear coett: 2.82
Frobenius norm
Spectral norm
4890
2230
2033
Yhd
(hd
(a) Frobenius norm
(b) Spectral norm
Figure 8: Plots of
G
Go
with varying n, d, h and 81 0.9, B2 0.999.
Index
Optimizer
0 1000
750
500
FB-Adam
2
3000
2250
1500
PB-Adem
4000
3000
2000
PB-Adam
4 5000
3750
2500
FB-Adam
5 6000
4500
3000
FB-Adam
7
8000
6000
4000
FB- Adam
Table 2: Parameters for Figure
20

---

Published in Transactions on Machine Learning Research (08/2025)
G,75 8
0.75
0.8
overlap
0.50 g
0.4
0.2
0.2
0.00
0.00
0.0
0.0
1000
1000
1000
1000
750
750
250
000
U.M
200000
1000
250 V.M
500
1000
250
500
U_M
250 250
v.W
750 1000
250
500
V_M
(a) O(U,, M
(b) O(V,, Vlr
(c) OU UM)))
(d) O(Viio VMa)
Figure 9: Overlaps after one step (a), (b) and 10 steps (c), (d) with FB-Adam11 = 1).
C Singular Vector Alignments of Weights and Optimizer Updates
In this section, present singular-vector perspective the emergence of HT ESD after multiple update steps. By considering an 'update' matrix M, E Rixdd we formulate the weight updates as:
W++1 W, Mr,
(35)
where M, is the optimizer update matrix based on GD (M, Gr) or FB-Adam (M, = - We
leverage methods from the rich literature on signal recovery in spiked matrix models (Shabalin & Nobel
2013 Landau et al. 2023 El Alaoui & Jordan 2018 Gavish & Donoho 2017 Troiani et al. 2022 to analyze
the role of singular vectors of M, in transforming the ESD of W, W, gradually into a HT distribution. Let b = min(h,d) d), we consider the SVD of W W,, M, as:
W+11 = UW..i Vi
W. = Uw, Sw. Vw M, = UM, SM, VM.
(36)
where Uw441 UW. UM. Baxbb, Sw Sw,, SM,. E bx and Vw Vw, VM, E gxxd
Definition C.1 Landau et al. defined as: O(J, Q) = (J Q)
2023 ). The 'overlaps' between two singular vector matrices J, Q E Raxb is E R
Overlaps during training. From the finite rank spiked matrix model, we know that the singular values Sw, are non-linear transformations (also termed as 'inflations' (Landau et al. 2023) of S and Uw, Vw, are rotated variants of UM. VMA, respectively. Formally, let $1 52 > denote the singular values of Wi, and let 81 2 82 2 86 denote the singular values of Mo. Let új E RR Reh,0 û, E .Rd represent the left and right singular vectors of Wi corresponding to singular value 8y. Similarly, let us E Rh U E Rd represent the left and right singular vectors of My corresponding to singular value $g. Owing to the rotational invariant nature of the Gaussian matrix Wo, the alignment. values IF
(6 )² Vj. k E {1,.... bb can [ú, ux) E
be computed solcly based on $j, 8% (scc Landau al. (2023) ; Mingo & Speicher (2017)). ). In this one-step context, we show that the large v) (Irom Section 4b leads outlier aligument values in the overlap plots.
In particular, observe from Figure 9a Figure 9b that (û, u,) (6) v1)² (i.e. alignments of top singular vectors of W,,M) MM) have high values which are close to and correlate with the presence of spike in the ESD (Table 4 with / = 1, = 1. Furthermore, Figure 9c Figure 9d show a decay of overlap values across the diagonal that tends to correlate with the HT ESD emergence (Table with 7 = 1, 10. Similar observations for GD are presented in Appendix
D Discussions on Mean-Field Initialization
Our setup employs the widely studicd NTK initialization Wang et al. 2023 for the two--ayer NNs. Alter- natively, previous studies have also focused on mean-fiekd-based initialization (Ba et al. 2022 Moniri el al., 2023) to analyze the role of one-step optimizer updates. The mean-field initialization for a two-layer NN can be formulated as:
f(x.)
To (Wa.). Vh
(37)
21

---

Published in Transactions on Machine Learning Research (08/2025)
Here W E Raxd Rh are the first and second layer weights respectively, with entries sampled Wolay Lid. ~ (O 2), [al; lid. NO, #), Vi E [b],j E [d]. Notice the change in scale of the entries and Oar(Vh).
the 1 scaling factor in equation 37 In this setup: [Woll2 = (1. vi
This section provides additional results and discussions for the mean-field initialization. We claim that it is straightforward to extend the conclusions regarding the scale of for one-step FB-Adam to this setting. In particular, with large 7) and multiple optimizer steps, we obscrve the emergence of HT ESDs. Additionally, the alignments of singular vectors of the weight and corresponding optimizer update matrices also remain potential contributor to the emergence of HT ESDs, consistent with the discussion in the main paper.
D.1
Scaling !/ and Alignment of Singular Vectors for FB-Adam
A note on notation from Ba et al. (2022): In our setup, we denote Go = 1) ( Wo), whereas Ba
et al. 2022) consider Go uyt (W1- Wo). Thus, in the mean-field setting, the learning rates we obtain are
simply the scaled versions of theirs by a factor of vh. To this end, Ba al. (2022) showed that 1j = e(h) (scaling adjusted to our notation) is (sufficiently) large for the GD update Go (see Figure [0]. One can also verify that the results of Theorem 4.1 for FB-Adam can be scaled and extended to this sctting.
Corollary D.1 Under the assumptions of Theorem we obtain the following scaling for n in the mean- field initialization selling:
n 0(1/Va)
W W 1Woll
(38)
7 θ (1/Vhd)
|| Woll2 x |Woll2
where Wollz = Os.((VI) Woll Oap(Vh).
10a
kital
109
(ES0)
logoo (ESO]
Ingnn (FS0)
loghn loghn (ES0)
10-1
-1.0
-0.5
0.0
0.5
-1. -0.5
0.0
0.5
-1.0
-0.5
0.0
0.5
2.0
2
Ingu:
Iogaati (hi)
(A]]
lng (A))
(a) 17 = 0.1
(b) 1)
(c) 17 100
(d) 1)
2000
Figure 10: Evolution of ESD of W W after one step optimizer update in mean-field setting. Here 2000, = 1000, h 1500, a. softplus tanh, Pe 0.3.
10a
cxps
log (F50)
Ing (E50) (E50)
10-k
-0.5
0.0
loqw
(a) = 0.00005
(b) 1 = 0.005
(c) 1 = 0.04
(d) = 0.5
T W after one step FB-Adam update mean-field sotting. Here 2000, Figure 11: Evolution of ESD of W d 1000, h 1500, a. sof softplus, a tanh, Pe 0.3.
22

---

Published in Transactions on Machine Learning Research (08/2025)
1:00
0.010g
0.07
0.50
0.75 9
0.50 gg
0.75 9
0.005
0.01
0.25
0.25
0.000
0.00
0.00
1000
1000
1000
1000
150
750
250 U. W
250
200
250 V.M
500
200
250 U.M
500
250
250 V.M
500
1000 0
1000
U.W
1501
v.W 1000 0
(a) OUUw. Uss
(b) O(Vee Vadg)
(c) OUU,, Uni
(d) O(Wi, Vo4o)
Figure 12: Overlaps after one FB-Adam update with vj = 0.04 in mean-field setting.
100
100
100
10P
kgg ve (F52)
logh (F50)
10-4
10-4
-1.0
-0.5
0.0
0.5
2
6
logto D.)
Iugoo (1))
log
logos (A))
(a) 0.00005
(b) 1 0.005
(c) 1 0.04
(d)
0.5
Figure 13: Evolution of ESD of W W after 10 FB-Adam updates in mean-field setting. Here 2000, d = 1000, h = 1500, a. = softplus, a tanh, Pe = 0.3.
0.020
1.00
00155
0.02
0.75
g
0010
0.50
0.50 gg
0.005 3
0.25
0.25
0.000
0.00
0.00
0.00
1000
1000
1000
750
750
750
750
250 y00 750, 000
500
250 von 150,
500
250 U_W
500
250
500
U_W
U_M
V.
V.M
159
1000
U_M
v_W
750 1000
V.M
(a) 0.00005
(b) 1 0.00005
(c) = 0.04
(d) 17 0.04
OU Umg))
O VWio: VM,)
OU M,))
O O(V VMO))
Figure 14: Overlaps between singular vectors after 10 FB-Adam updates with 1 0.00005 (plots (a), (b)) and 17 = 0.04 (plots (c), (d)) in the mean-field setting.
Singular Vector Overlaps after one-step FB-Adam update. Following the main paper, we compute the following overlap metries: OUUw, UM,), O(W,, VMA,), O(Uw, UM,). O(W,, VMO.) For FB-Adam with n 0.04 (which is a large 1 in this setting), Figure 12 shows the outliers for O(Uw, UM.), O(V VMa) corresponding to the spike in Figure 11c. Thus aligning with the results in Section
Heavy-Tailed Phenomenon after Multiple Steps With Mean-Field Initialization. Similar to the experiments in Section 5 we employ the mean-field initialization and apply 10 FB-Adam updates with various to compute OUUw,o UM,), O(Vwi.: VM,)). Notice that for small 7) = 0.00005, the ESD expected to remain largely unchangod and the overlap plots illustrato random alignment values. However, for r = 0.04, the outlien liers emerge in the overlap matrices in Figure 14 and correlate with the HT ESD in Figure 13
Note on the kernel lower bound. Without loss of generality, consider a dataset X for which we have kernel bound for the lest loss. Ba et al. 2022) have shown that the Bulk + Spike ESD after the first step with GD will result in test loss that is much better than this kernel bound (For example, with r = 100 in Figure 10 Informally, our results imply that, by choosing 7 for which a bulk + spike ESD occurs with FB- Adam, (with that same X), then the same feature learning behaviour (via increased KTA and sim(W,3) represents that we are indeed better than the kernel bound. Nonetheless, we defer a rigorous characterization of this behaviour to future work (Appendix Hb.
23

---

Published in Transactions on Machine Learning Research (08/2025)
E ESD evolution over long training periods
In the main text, we claimed and validated that the ESD of the hidden layer weight matrix exhibits heavy tails after multiple steps of GD/ FB-Adam with sufficiently large In the following ESD plots, we further justify our claim and illustrate that very long training t = 10000 with small does not result. HT ESDs.
E.1 ESD evolution with GD
10
100
10000
iniut StutD
Six1
(CS0)
100 E500)
login
17
0.1
10
10
10
3.0
2.5
3.5
3.0
2.0
3.5
logua
109
10
10
20
Aepto
slmp210000
lnggs (F50)
logn (ESD)
I0gu (EEU/E50) (ESD)
10
10
10
3.0
2.5
3.5
2.5
2.0
3.0
logio
logio (hi)
logae Chi)
Ingin (A)
10"
H934
10"
107
20a
e0 (F52))
(CS0)
o (250)
loghs
7/
10
2.0
2.5
3.0
3.5
2.0
2.5
3.0
3.5
2.0
2.5
5.0
3.5
2.0
2,5
3.0
3.5
logus CAA
loggo (ix)
loqu: (A,)
loguo (AO
inist Seuto
log (FS0)
logoo (CSO)
I00 (ESO)
logio
7
100
10
20
10
3.0
2.5
3.5
3.0
2.0
logua
logu:
1.0
10
10a
30a
stptt
19x32000
Hep020000
logns (F50)
(E50) (ESD)
Ingne
losu
2000
10
10-1
10
10
logio
log (h))
logae CA.)
logio/A)
Table 3: Evolution of ESD over different step times t (1, 10, 100, 10000) with different " E {0.1, 1, 10, 100, 2000} for GD optimizer. Through this grid, highlight the critical n for the occurrence of spike at. / and the effects of longer training on the emergence of HT ESD.
24

---

Published in Transactions on Machine Learning Research (08/2025)
E.2
ESD evolution with FB-Adam
1
t 10
100
10000
5XS2
Agdt
logto (150
Iogu losw (ESD)
logia l0gi (650)
77 = 0.01
10
10
10
10
3.5
2.5
3.5
3.0
2.5
3.5
logic
logu
loguu
loguu
10a
30"
10a
10"
exple
map200
depliccio
Vodas (ESD)
(ESD)
100 (ESO)
10 (FSD)
77 = 0.1
10-1
10
10
2.5
3.0
2.0
2.5
3.0
3.5
Iogie (A))
loguo (A))
logno (A))
logio (AL)
20
HCW
10a
logu (ES0)
Vogu (ESD)
1S (ESD)
loow
7
10-1
10-1
logie (A))
logio (A.)
(Ai))
10a:
20
10*
10P
dxcl
logu (ESD)
loga (E50)
10 (F50)
10
10
20-4
1.
looz:
loguo (A,)
loguo
logu:
10a
2.0
100
3."
logta (CSD)
log. logu (ESD)
loguo (053)
log (ggu/E50) (FSD)
10-1
17 = 100
10-1
1011.
6
2
8
10
10
2.5
5.0
7.5
10.0
logu: (A))
loqu IA))
10g (A.)
loquo (ki)
Table 4: Evolution ESD over different step times t E {1,10, 100, 10000} with different 1 E (0.01, 0.1, 1, 10, 100) for FB-Adam optimizer. Through this grid, we highlight the critical n for the occur- rence of spike at = 1 and the effects of longer training on the HT emergence.
25

---

Published in Transactions on Machine Learning Research (08/2025)
106
intal
sepp
xastooocco
log (ESO)
logua LESD)
0.01
10-
2.0
2.5
3.0
logua
Table 5: ESD after (1, 100) steps of FB- FB-Adam with n 0.01. The plot showcases scenario where spike does not appear after the first step but results in HT ESD after extremely long training.
F Additional Experiments
Hyperparameters: In most of our experiments, we follow a consistent setup with n = 2000, n lest = 200, = 1000, 1500, 0.01, Ps = 0.3, a, = softplus, a = tanh. Additionally, we explicitly mention the parameter changes wherever applicable in our experiments.
F.1 One-Step Optimizer Updates
In this section, we present additional experiments for ono-stop optimizer updates. Figure 15 leverages the same experimental setup as Section 4 and illustrates the ESD of W W after the first step of and FB-Adam. Based on the results obtained in the main text, 77 = 2000 is an extremely large learning rate for FB-Adam, which results in clear bimodal distribution. Note that the tendency towards such distribution was already observed with 7) 10 in Figure 2 (Section 4 Based on the same setup as Section C Figures 16
17 18 represent the overlaps of singular vectors after one-step update and showcase the presence of outliers for sufficienlly large n, Finally, we illustrate the role of sample sizes on the losses, KTA and sim W, B*) in
Figure
19
(for n = 4000) and in Figure 20 (for n 8000).
10p
10
100
10
(egm//550)
lngio (550)
le50
20m
20"4
10
10
2.0
2.5
3.0
3.5
2.0
2,5
3.0
3.5
10
12
logas (A))
logio (1)
log (A)
log (A))
(a) GD
(b) GD 10
(c) FB-Adam rj
(d) FB-Adam 2000
Figure 15: Evolution of ESD of W W after one step optimizer update with varying lcarning rates. Here = 2000, d = 1000, h = 1500, a. = softplus, o = tanh, Pe 0.3.
0015
0.015
0.02
0.02
0.01
0.005
0.01
1000
1000
1000
1000
230
759,000 000 U.W
0
250
300
200 v.w
750,000
250
500
V.M
250 U.W 750,000
250 U.M
500
250
v.W 750,000 0
250 V.M
500
(a) (Uw.. UMA))
(b) O(Vw.. VML
(c) OUw,, UnA)
(d) O(V,, VMO.)
Figure Overlaps between singular vectors after one step CD update with n 0.1. Here n 2000, d 1000, h 1500, a, softplus, a tanh, Pe 0.3.
26

---

Published in Transactions on Machine Learning Research (08/2025)
0015
0.03
0.110
0.02
0.6
0.0056
0.01
0.4
0.4
0.2
0.000
0.00
0.0
1000
1000
1000
1000
150
750
750
750
230
1900
U.M
200
1000
250
V.M
200
U_W
1000
U.M
230
v.W 1000 V.M
(a) OU U
(b) O(Vwe, Vmg))
(c) OlU,, M
(d) O(W Vaoo)
Figure 17: Overlaps between singular vectors after one step FB-Adam update with = 0.1. Here n 2000, 1000, h 1500, a. softplus, a tanh, Pe = 0.3.
0.010
0.015
0.8
ovatlep
0.0058
0.01
0.2
0.000
0.00
0.0
0.0
150
750
750
150
250 750 U.W
500
230 V.W
500
0
250 500
500
250 500 v.W
500
1000
U.M
75% 1000
V.M
U.W
750,
1000
U.M
750
1000
V_M
(a) OUUwo Un4
(b) O(Vwe Vmgg)
(c) O(U UMA)
(d) O(V Val.)
Figure 18: Overlaps between singular vectors after one step GD update with d = 1000, h = 1500, ax = softplus, tanh, Pe = 0.3.
2000. Here 2000,
1.6
GD.train
"B "B-ddmmtrainn
0.05
C
r0-Adam
0.6
F5-ccam
6D
1.4
Adamitest
0.04
sim g*)
0.4
1.2
0.03
0.2
0.8
0.02
0.6
0.01
0.0
-2
logo0 (n)
logso (n)
logs0 (g)
(a) loss
(b) KTA
(c) sim W, W,8* 8*)
Figure 19: Train/test losses, KTA, sim( (W for f(-) trained with one-step of GD, FB-Adam. Here n 4000, d 1000, h 1500, a. = softplus, a tanh, Pe = 0.3.
GDitrain
M5-Adam
0.6
PS-ccam
GD:test
0.06
1.1
7B Adamitest
1.0
0.04
sim(W,P
0.4
0.9
0.02
0.0
-2
0
2
-2
0
2
-2
0
2
log:o (n)
log:o (n)
log (n)
(a) loss
(b) KTA
(c) sim W. WV)
Figure 20: Train /test losses, KTA, sim(W,8) for f(-) trained with one-step of GD, FB-Adam. Here 8000, d = 1000, h = 1500, a. softplus, a tanh, Pe = 0.3.
27

---

Published in Transactions on Machine Learning Research (08/2025)
F.2 10 Step Optimizer Updates
In this section, we present additional experiments for 10 oplimizer updates. Figure 21 presents the losses, ESD of W W and the overlaps of singular vectors after 10 steps with GD(1) = 2000). Notice that the prominent outlier values in Figures [T8c 18d after the onc-stop update have now reduced significantly. Thus illustrating the varying spread values even for the left and right singular vector overlaps. The role of y on the losses, KTA and the FSD metric PL PL_Alpha_HAr Hill are illustrated
23 We illustrate the ESD of W W after 10 steps with 8000 and learning rates r chosen from {1, 10, 100, 1000} in Figure 24 Observe that as lj increases, the spike tends to move far away from the bulk and significantly distorts the shape of the bulk only for f 1000. Finally, in Figures 25 26 we illustrate the role of label noise increasing from Pe = 0.3 (Figure 25 to Pe 0.7 (Figure 26). Although the ESDs look the same in both cases, note that the outlier (max) values of the overlap matrices for Pe 0.7 have relatively larger values compared to the Pe = 0.3 case.
Figures 22
10P
1,05
Mayi2
1.00
0.375
0.100
overlpp
0.95
0.050
0.01
0.075
0.90
0.000
0.00
0.85
150
1000
150
1000
30859
2.0
2.5
logutAA
3.0
3,5
4.0
300
750 1000
500
U_M
250 v.W
750 1000
250 V.M
(a) losses
(b) ESD of W W
(c) OU Umo
(d) O ((Vioo VASa)
Figure 21: Losses, ESD, and Overlaps between singular vectors after 10 steps of GD(1 = 2000). Here π 2000, d = 1000, = 1500, a. = softplus, o = tanh, Pr: 0.3.
144 main Feddardeu
2.4
30
0.04
2.2
220
Alpha
0.02
42
(a) loss
(b) KTA
(c) PL AI__Apha_Hii Hill
(d) PL PL_Alpha_KS
Figure 22: Train/test losses, KTA, PL Alpha Hill, PL Alphap KS for f(-) trained with 10 steps of GD, Ad-A Here 2000, d = 1000, h 1500, softplus, tanh, Pe 0.3.
1.4
0.06 Fliaaa
72
00
1.2
Ligha, HII
2,2
20
0.04
1.0
2.0
0.8
1.8
0.6
-2
loque (n)
l001e (n)
logoo (n)
loquo (n)
(a) loss
(b) KTA
(c) PL PL_Alpha_Hin Halha_Hilll
(d) PL PL_Alpha_KS _aS
Figure 23: Train losses, KTA, PL_Alpha_Hiile _ill, PL PL_Alpha_Ka KS for f(.) trained with 10 steps of GD, FB-Adam Here n 4000, d 1000, 1500, a. softplus, o = tanh, Pe = 0.3.
28

---

Published in Transactions on Machine Learning Research (08/2025)
Fibal
lagie
logi login (ES0)
10-1
10
(A))
AA)
(A))
111
(b) 17 10
(c) 17 100
(d) 7 1000
Vog (ES0)
logne (E50)
loguo (ES0)
10-3
30-1
10
4.5 5.0
logie
(A)
(e) 77
(f) 17 10
(g) 1 100
77
1000
Figure 24: Evolution of ESD of W W after 10 steps of FB- Adam updates with 8000, d 1000, h 1500, X 0.01, Pe 0.3. The first row compares the initial and final ESDs. The second row illustrates solely the final ESD of W W (i.c. W1)) for better visualizations of the shape.
1.00
10P
dipuz
0.35
(ES0)
0.4
overaap
overlap
0.90
0.2
0.80
1000
1000
10
250 U.M
500
250 v.W
250 V.M
500
logco PA)
750 1000
750 1000
(a) losses
(b) ESD of W W
(c) OUUwio UM,)
(d) O O(Vio. VMS
Figure 25: Losses, ESD, and Overlaps between singular vectors after 10 FB-Adam(1 = 1) steps for n 8000, 1000, 1500, 0.01, Pe = 0.3.
1.35
1.30
(630)
0.5
0.8
0.8
0.6
overlap
1.25
0.2
0.2
0.4
1.20
10-1
0.0
0.0
1000
1000
10
250 500 J.W
250
500
250 v.W
250
500
Stet>
logoo CAA
750,
750
(a. losses
(b) ESD W W
(c) OUw Usa))
(d) O(V VML
Figure 26: Losses, ESD, and Overlaps between singular vectors after 10 FB-Adam(y) = 1) steps for n 8000, 1000, 1500, 0.01, P. 0.7.
29

---

Published in Transactions on Machine Learning Research (08/2025)
F.3 Spike movement with GD
During our experiments with GD(1) = 2000), we observed surprising transition in the position of the spike relative to the bulk of the ESD. Particularly between steps 5 and 6, the spike in the ESD of W W which represents the largest singular value, reduces in value by an order of magnitudo. Additionally, this reduction seems to be correlated with the reduction in maximum overlap values from max(O(Uws. UM,)) to max (axUOO UM,)) (see Figure 277. To understand this behavior, we emphasize the spike in the FSD of W3 W; in Figure 27a and the large overlap value (black dot) in Figure 27b Let úwa uM, åwe lh represent the left singular vectors corresponding to the largest singular values in Ws, Ms, W6 respectively. The large overlap value (black dot) in Figure 27b intuitively represents high degree of overlap/alignmeet between úw,, uMg. As result of obtaining We by W. = Ws + Ms, the singular vector êw, seems to be rotated from dw; in such a way that its alignment with ūM, is reduced (see Figure 27d One can intuitively think of this process as the 'diffusion' /'spredd' of the overlap between üM, and all left singular vectors of WB.
0.8
lagin 1elee
overlpp
10.1
10:1
0.1
0.0
1000
1000
750
(A))
250 500
U.W
750
1000
250
500
U.M
2.0
2.5
3.0
(A)
35
4,0
U W
500
750 1000
250
500
U.M
(a) ESD W Ws
(b) OUU,, UMS)
(c) ESD W. We
(d) O(Uw. UMs)
Figure 27: Phase transition in ESD of W W between steps 5, 6 when updated using GD (1) 2000). Here 2000, d 1000, h 1500, o, softplus, = tanh, Pe 0.3.
F.4 On Generalization, Learning Rate Schedules and Weight Normalization
Role of sample size (n). By fixing d = 1000, h 1500 and n 1, we vary the size of the training datasct n as per the set {500, 2000, 4000, 80003. In this setting, observe from Figure 28a that the train loss and test loss improve significantly for
8000, while the network overfits for smaller n. Furthermore, for 2000, observed that WT Wio exhibits heavy-tailed ESD with PL PL_Alpha paipphaaallll=__aaaaaaaaaallllll Hill= = 1.8 (Figure 22c), which is less than the n = 8000 case of x 1.9 (see Figure 5c). The key difference in the latter case is that the spike in the FSD is consumed by the bulk, unlike the former where the outlier singular value is almost an order of magnitude away from the bulk.
Role of regularization constant (A). By considering a sample size of n = 8000, and fixing d 1000, h 1500, 1 = as before, can observe from Figure 28b that lower training and test loss is achieved by A 10-3 and À = 10-4, but with large generalization gap (i.c. the difference between train and test loss). Additionally, observe that X = 10-2 reasonably balances the test loss and generalization gap. Since the regression procedure does not modify the first layer weights, the choice of X can affect the interpretation of heavy tails leading to good/bad generalization.
Role of label noise (pe). Figure 28c illustrates a consistent increase losses for an increase in the additive Gaussian label noise Pe from (0.1, 0.5, 0.7). However, the ESD of W Wio alone does not provide the complete picture to reflect this difficulty in learning. Instead, we observe noticeable differences in the distribution of values (especially outliers) along the diagonal of overlap matrices OUUn.. Um,); O(V VM,) for P. = 0.3 (Figure 25c 25d and Pe 0.7 (Figure 26c 26d Especially, the outliers in the former case had smaller values (almost 0.5x) than the latter.
Role of learning rate scheduling. As a natural extension of seleeting a large learning rate at initial- ization, we analyze the role of employing learning rate schedules |Ge et al. 2019 on the losses and ESD of W T W. We consider the simple torch. torch.optim. wtt.:opptetp...o..tttttt StepLR scheduler with varying decay factors (7) per step. A
30

---

Published in Transactions on Machine Learning Research (08/2025)
1.5
1.4
1.00
1.0
0.95
0.8
20
0.85
0.7
0.8
10
stop 10 0.0
2.5
7.5
10.0
stop
stop
stop
(a) vary 73
(b) vary A
(c) vary Pe
(d) vary StepLR (7)
Figure 28: FB-Adam Irain/tes /test loss across 10 steps with varying n, A, Pe. and StepLR(7). (a) Varying dataset
size n (b) Varying regularization parameter À for obtaining the second-layer weights, (c) Varying the std.der (pe) of the additive Gaussian label noise, (d) Varying the ? paramcter which controls the docay of 17. The bold lines indicate train loss and the dashed lines indicate test loss.
10
10
10
loglo 19500
logle (E50)
logne (ES0)
10-1
10-1
10-4
10-4
2.0
2.5
3.0
3.5
1.5 2.0
2.5 3.0
3.5
4.0
2.0
2.5
3.0
3.5
logus (A)
logio
V.)
logie
loguo (A))
(a) GD 0.1
(b) GD 77 2000
(c) FB-Adam 77
0.1
(d) FB-Adam
Figure 29: Evolution of FSD after 64 steps of GD, FB-Adam updates with varying 1y and weight normalization. Here n 2000, d = 1000, h 1500, softplus, tanh, Pe 0.3.
1.2 staln
0.014
1.1
per:
0.012
1.0
0.010
Jona
2.0
33
0.008
'1.0
20
0.8
0.006
1.6
0.7
0.004
-2
2
-2
0
-2
0
Iog
Ingi:(t)
logul()
(ggw(()
(a.) loss
(b) KTA
(c) PL Ad_phaa__H11 Hill
(d) PL PL_hlpha_K
Figure 30: Train/test losses, KTA, PL_ PL_Alpha Hill, PL Alpha_ppaK__H. KS for f(-) trained with 10 steps of GD, FB-Adam with weight normalization, Here = 8000, d = 1000, h = 1500, a, softplus, = tanh, 0.3.
smaller 7 indicates a faster decay in the learning rate 1) per step. We obscrve from Figure 28d that such fast decays (with 0.2 and = 0.4) quickly turns large learning rate to smaller one and lead to stable loss curves. However, the trends are relatively unstable for y = 0.6 and y = 0.8 in the early steps.
Role of weight normalization (WN). As mentioned in the main text, we employ a weight normalization vhdW (11
technique (Huang et al. 2023) after each optimizer update to W as follows: W
W W++1
W
W, M, to ensure that Wellly 1 is always Vhd, before the forward pass. Observe from Figure
29 that
after 64 steps of GD/FB-Adam updates with varying 7, the large 7 cases lead to HT ESDs which spread
wider range of singular values than the non- noN-WN cases (scc Table 4. Furthermore, Figure 30 conveys that in the case of FB-Adam for 0.01 < v S 100, the PL A__Appaa_Hill metric and the mean estimates of PL_Alpha tend to exhibit HT ESDs and generalize well. Notice that this range is much broader than the non- WN case in the main text (Figure 5) Additionally, note that the PL_Alpha_Hil _illl estimates for 10 S r S 100 are relatively lower than the non- WN case. Finally, by employing learning rate schedules, we can control the presence of the spike while nudging the bulk of the ESD toward HT distribution (Figure 31
31

---

Published in Transactions on Machine Learning Research (08/2025)
Mextt
Statt
100x (E30)
logas (550)
logns (ESO)
10
10
10
10gua
logu:
log (A)
(a) StepLR(y=0..) 0.2)
(b) StepLR = 0.4)
(c) StepLR = 0.6)
(d) StepLR(y= 0.8)
Figure 31: Evolution of ESD of W W after 10 sleps of FB-Adam(1) 1) with weight normalization and varying decay rates for StepLR schedule. The decay factor (?) is applied after every step. Here n 2000, d 1000, h 1500, a. softplus, a tanh, Pe 0.3.
F.5 Effect of using the same activation function for teacher and student
In our main paper, we adopt different activation functions for the teacher and student models: softplus for teacher and a tanh for student, Although this distinction introduces some learning challenges for the two-layer NN - making it harder to minimize both training and test loss - it does not affect the key theoretical results of this work, such as the scaling of v) for a single-step FB-Adam update, provided that both activation functions meet the fundamental assumptions. In this section, we use tanh activation for both models to examine whether more consistent teacher-student sctup yiclds any notable differences. We use the same setup as Section 5.2
Correlations between ESD and losses. Similar to the observations in the main text (see Figure for different activation functions, note that for GD with 17 2 1000, the KTA increases and PL Alpha_Hill mean estimates of PL Alpha KS reduces. More importantly, note that the test loss values are relatively smaller (i.e R 0.85 in Figure 5 and R 0.2 in this setup). Tn the case of FB-Adam, we observe a surprising shift in the trend of loss values for 1) 2 0.1. In essence, the range for which HT ESDs emerge and lead to good gencralization has now shrunk to much smaller range. Espocially, the PL PL_Alpha latha_lillllll value of = 2 which resulted in good generalization in Figure 5 (relative to losses of the full range of 1)), correlates with poorer generalization in this setup. Overall, there seem to be non-trivial dependencies on the choice of actívation functions to determine the correlations between HT ESDs and generalization.
Hill
0.1
LET E
0.3
40
occedd rBideTcist
2.2
0.5
0.2
2.0
Aphh
2/1.8
TEvan
0.1
-2
-2
-2
logiz (r)
log (r)
logu (g)
(a) loss
(b) KTA
(c) PL Alpha Hill
(d) PL_ Alpha_KS
Figure 32: Losses, KTA, PL Alppa_Hill PL PL_Alpha_Ke KS for f(-) trained with 10 steps of GD, FB-Adam, with = 8000,0000000000 d = 1000, h = 1500, a. = tanh, o tanh, P. = 0.3, A = 0.01.
F.6
Applicability of our results on deeper NNs
Beyond bwo-layer NNs trained on synthetic data, we train VGG11, ResNet18 with FB-Adam on MNIST, CIFAR10, SVHN with a constant 7 = 0.01 for 10 steps (epochs) to validate our claims. In Figure 33 Figure Figure 35 below, we illustrate that the layers of VGG11 can indeed exhibit HT ESDs even without stochastic gradient noise during the early phases of training. See Figure 36 and Figure 37 for ResNet18 plots with CIFAR10, SVHN datasets.
32

---

Published in Transactions on Machine Learning Research (08/2025)
10c
Inttal Ctaii
10a
Inttlll tnainred
Intal trained
10gho (ESD)
logao (ESD)
logso (ESD)
10-1
10
10-1
10
-2
-2
-1
-7.5
-5.0
-2.5
0.0
logio (A,)
logue (A))
log (A))
(a) VGG11-Layer2
(b) VGG11-Layer4
(c) VGGIL-Layer7
Figure 33: layerwise ESDs of VGG11 after 10 steps with FB-Adam and 1 = 0.01 on MNIST.
10c
inibal tslned
100
inttlal tranned
intil trained
log (ESD)
logio LESD)
10
10
0ogoo
10
10-2
-2
-2
-1
100
-5
l0g2o (A:)
log2c (Ai)
log (A)
(a) VGG11-Layer2
(b) VGG11-Layer4
(c) VGGI1-Layer7
Figure 34: layerwise ESDs of VGG11 after 10 steps with FB-Adam and v 0.01 on CIFAR10.
10c
intial traihed
10a
intial tranedd
intal
trained
to (ESD)
1og (ESD)
io (ESD)
10-1
10
10-1
10
10 -2
-2
-2
-1
0
-7.5
-5.0
-2.5
0.0
2.5
log:o IA.)
logac (A))
log (A))
(a) VGG11-Layer2
(b) VGG11-Layerl
(c) VCG11-Layer?
Figure 35: layerwise ESDs of VGG11 after 10 steps with FB-Adam and r = 0.01 on SVHN.
G A note on estimators of power-law exponents
We found that the estimation of power-law (PI.) exponents, such as PL PL_Alpha_He Hill and PL_Alpha__s KS, can be sensitive the scale of singular values the weight matrix. However, Figure 5c 5d illustrates a similar trend for both estimates. We believe this does not affect our qualitative interpretation of the relationship between HT, learning rates, and generalization.
Additionally, we highlight particularly interesting observation. Considering FB-Adam based updates with = after - 10 steps and after / = 100 steps in Table notice that the spike tends to get closer to the bulk in the latter (i.e as training progresses). We have observed a similar behavior for GD (see Appendix F.3) where the spike tends merge with the bulk. Thus, the estimation of PL exponents should be relatively less affected by such outlier spikes as training progresses. Furthermore, in Figure 5d] the PL PI_Alpha_ KS is affected by the relative position of the spike from the bulk, especially when the gap is negligible or there is
33

---

Published in Transactions on Machine Learning Research (08/2025)
Iribil Irai wer
iital traimed
109.
Intal Irainee
10c
logho (ESD)
logho (ESD)
(ESD)
10-1
10-1
10-
10
-2
0
-8
-6
-4
-2
0
log (A:)
logae (A)
log (A)
(a) ResNNet88-Layrr
(b) Layeer
(c) ResNeet8-Layer10
Figure 36: layerwise ESDs ReaNet18 after 10 steps with FB- Ad-Adam and 7j = 0.01 on CIFAR10.
inttll beined
intall trained
10
Intal trcinod
10
logno Jogro (ESD)
logio pogio (ESO)
10-1
10
11 log M))
8
-6
-4
2
logie (A)
log (A)
(a) ResNet18-Layer
(b) ResNeet1-Layert
(c) ResNee18-Layer10
Figure 37: layerwise ESDs of ReaNet18 after 10 steps with FB-Adam and r 0.01 on SVHN.
gap (see Table 4 for ESD visualization with 0.01). Therefore, the trends with PI. alpha estimates being the same apply to the 1j 2 0.1 values, where the spike is sufficiently far from the bulk and the estimates tend to exhibit a non-increasing trend.
On related note, some recent papers have adopted both fitting methods explore the relationship between HT and generalization and leveraged it improve model training. For example, Martin Mahoney (021aa) used PL PL__AlL Alpha_ KS to study the relationship between HT and generalization to propose the HT-SR theory, whereas Zhou et al. (2023 proposed layer-wise 7 selection technique based on layer-wise PL Ad_Alpha_Hal Hill estimates of the ESD 's during training.
H Limitations and Future work
Currently, our analysis on FB-Adam updates does not theoretically analyze the role of large learning rates on the training and test losses after one or more steps. such rigorous characterization requires new techniques to analyze the regression loss beyond the Gaussian equivalence assumption (Ba et al.) 2022). Furthermore, the techniques used to calculate the PL PL_A1L Alpha of ESDs are subject to bias (in the case of the hill estimator)
and relatively larger variance the case of the KS variant. Similar issues have been discussed in previous works Yang al. 2023 Zhou al. 2023), and analyzing multiple approaches can provide complete picture of the heaviness of the tails.
To this end, we discuss the following potential future efforts.
Generalization with FB-Adam. Recent papers employing a similar setup have analyzed the feature matrix Z to rigorously characterize the training and test errors after a one-step GD update (Moniri et al. 2023 Cui et al. 2024). In a similar spirit, the results from our work can be utilized to theoretically explore the spectral properties of Z with FB-Adam. Additionally, since the spectral properties of Z are tightly linked the ESD of W, W. the fundamental question the necessity of HT ESDs for generalization can be studied.
34

---

Published in Transactions on Machine Learning Research (08/2025)
Spectral gap and step complexity. In Section we observed that the number of steps required for HT FSD emergence depends on the spectral gap (i.e the distance between the spike and the bulk in our setup) after the first step. While our work establishes the necessity of the spike, further analysis of the relationship between this spectral gap and the step complexity for HT ESD emergence can lead to novel insights.
HT phenomenon and singular vector overlaps. By presenting qualitative results that indicate HT- like distributions along the diagonals of overlap matrices, we aim bring singular vectors into the picture for future HT phenomenon studies. Particularly, the qualitative differences in the overlap matrices of left and right singular vectors remain to be explored. Furthermore, metries to quantify the 'spread' of the on oofoff- diagonal overlap values can present holistic picture of the interactions between the singular spaces of the weights and optimizer updates.
Towards analysis with deeper NNs and the '5 + 1 phase model. Our work showcased how the '5 + I' phase model of the ESDs can be studied under simpler two-layer NN settings and reasonably explain the practical NN ESD dynamics. A valuable direction of research to extend this analysis to deeper NNs (Nichani al.
2023
with multi-index models teacher networks.
Designing novel techniques to improve generalization. Recently, Zhou et al. (2023) employed the layer-wise PL_ PL_Alpha_HHil Hill metric to design layer-wise learning rate scheduler based on SGD. Based on our observations on the varying effects of for GD/FB-Adan Adam and the dynamics of ESD evolution, there is immense potential to further improve such schedulers. Furthermore, new regularization techniques to balance the PL S___lpha__Baa _atallallllll1 metrics across layers during training can lead to NNs that satisfy the ESD shape metric criteria be considered as 'well-trained" by model selection approaches (Martin al.j 2021 Yang el. al. 2023). The intriguing consequences of such technique on the convergence rates and sample complexities lead to new directions of rescarch.
35


Reviews:
[{"review_id": "KpNolc9G7Q", "reviewer": "Reviewer_WboY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe theoretical investigation of the training process of artificial neural networks (ANNs) is an active research area. Because of the non-convex high-dimensional nature of the ANNs, research problems in this area are highly challenging. The paper documents significant efforts towards this direction.\n\nWeaknesses and my concerns:\n1. I see this sentence from page 2 of the manuscript \"For instance, η = 1 leads to HT ESDs just after t = 10 steps, whereas η = 0.1 requires very long training up to t = 10000 steps (see also Table 3, Table 4).\" Frankly, this raises concerns regarding the overall reliability of the research. Intuitively, one would expect that a smaller learning rate explores the parameter space more finely, though more slowly. Roughly speaking, after ten steps with η = 0.1, one should arrive at approximately the same point as with a single step using η = 1. If this is not the case, there could be other reasons, for example because of complicated landscape of the parameter space, gradient explosion, or gradient vanishing. However, the approach taken in this paper appears to treat the observed phenomenon in an overly mystified manner, placing undue reverence on the step size as though it were a source of magic. I do not think this is science.\n\n2. The training process in Section 3.1 is confusing. When one wants to match function f in (2), to function F* in (1) through training, ideally the activation functions should be consistent, or otherwise the behavior could be very unclear, misting the phenomenon observed. On the other hand, conducting intensive training on the weight matrix $W$ while keeping the coefficient vector $a$ randomly specified would introduce significant uncertainty into the results. Analyzing the outcome of such training resembles, to a considerable extent, the study of a dice roll.\n\n3. Since the matrix G0-tilde has all of its entries either C or -C almost surely (where C is given in (9)), the Frobenius norm of G0-tilde is almost surely equal to sqrt(C*h*d). Making the estimation in (10) appears pointless.\n\n4. The Discussion on page 18 is not consistent. We see that at the beginning of the section, vector beta could be any d-dimensional real vector, while at the end, it is restrictied to have unit norm.\n\n5. The proof of Lemma B.5 has a problem. While the cited Lemma B.4 is a results with probability, it is used to prove an almost sure inequality.\n\n6. The proof of Proposition B.6 confuses me. If the objective is to ensure that the $A_{ij}$ entries are nonzero, one could simply exclude zero entries when generating the vector $a$, and disregard sample points that make $y \\cdot X = 0$. Introducing such an exclusion step would also contribute to improved numerical stability since the marginal cases could also be removed. Relying on the assumption that these cases will not occur, merely by employing random sampling, does not appear to be a sound approach.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the training process of two-layer artifical neural networks through gradient-based methods. Using theoretical and empirical approaches, the paper shows that a heavy-tailed empirical spectral density can be achieved through restricting the learning rates within a specific scope, so as to achieve generalization ability.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "See above section.", "broader_impact_concerns": "This paper studies the performance of a class of deep learning algorithms with numerical and theoretical approaches. I have no Broader Impact Concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "prrchgUIiD", "reviewer": "Reviewer_fwJm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "On the whole, I do not think this paper presents a clear story, and not all of its claims are well-supported. So as to not repeat myself, I elaborate on these concerns under the changes I request below. In brief, I think that substantial rewriting and additional experiments would be required before the paper could be accepted. \n\nThe paper leaves open a large question: for a single index model, what is the possible benefit from moving away from the single-spike phase? Nor does it answer the question of why the particular tail index range they observe is linked to good generalization performance yields that good performance. There is also the fact that their theoretical analysis only captures the emergence of the spike. \n\nWith this, I would also like to object to the author's assertion that \"[their] study tightens the connection between ESDs and feature learning, explaining why ESD-based training methods can improve the generalizability of large and deep models.\" Because the paper does not theoretically link heavy-tailed ESDs to generalization, I think this claim is unjustified.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a series of empirical and theoretical results on the emergence of heavy tails in the singular value distributions of deep neural network weights. Throughout, the focus is on single-hidden-layer networks learning a single-index target function using full-batch gradient descent or Adam. On the theory side, the main result is a characterization of the learning rate required to produce an outlier in the hidden layer weight spectrum after one step of full-batch Adam, extending an earlier result of Ba et al.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The introduction and related works sections require substantial re-writing to clarify the phenomena of interest for a general machine learning audience. Most simply, it is not clear quite what question the paper primarily aims to address, as the theoretical results on outlier emergence need not be framed in terms of Martin & Mahoney's ideas. In that vein, the authors must make sure to adequately define terms from past works when they are first used (for instance: Why is this referred to as \"self-regularization\"? What are the \"5+1 phases\"?). \n\n- In the vein of better motivating the study: I appreciate the authors' attempt at the end of page 6 to set up Theorem 4.1, but this setup begs the question of what would happen if one simply trained for more steps. That is, I would hope for a comparison of how GD and Adam compare if one trains for many steps, to see if the spectral evolution qualitatively differs or if it just differs in the speed with which certain spectral structure develops. \n\n- The authors should note in the statement of Theorem 4.1 that because they are considering only a single step of gradient descent, Adam with their parameters becomes equivalent to sign-GD (see Bernstein et al 2018, \"signSGD: Compressed Optimisation for Non-Convex Problems\"). \n\n- The contributions of the paper relative to Wang et al., \"Spectral Evolution and Invariance in Linear-width Neural Networks\" (NeurIPS 2023) should be clarified. That work also presents empirical results for Adam, and the authors of the present manuscript should explicitly detail how their findings differ. Wang et al. also showed that heavy-tailed weight spectra are not sufficient for good generalization performance. Moreover, I think the work of Wang et al. should be acknowledged when the authors make claims like \"Overall, by connecting the HT phenomenon with feature learning, we hope to promote further research into these deeply connected characteristics of NNs\" as such connections are already made there.\n\n- Recent work by Dandi et al. (https://arxiv.org/abs/2410.18938) that includes partial theoretical characterization of changes in the bulk after one large gradient step is not cited. \n\n- Some more analysis of the massive (~10-fold discrepancy) in estimated tail exponent for the two estimators of tail index shown in Figure 5c-d (and elsewhere) is required. I am a bit puzzled by the use of minimizing the KS statistic to fit the scaling parameter $\\alpha$, as the (cited) classic review of Clauset et al recommends minimizing the KS statistic to fit the minimum value of the range over which power-law behavior applies, not to fit $\\alpha$. Please visualize somehow the quality of the resulting fits. \n\n- Given that Adam is an adaptive method, is it really fair to plot together the GD and Adam results on the same axis of learning rate? \n\n- The results on singular vector overlaps are not well-integrated into the results, and are not compelling. To me it does not make sense to view the weight matrix $W\\_{t}$ as 'noise' on top of the update $M\\_t$ after multiple steps in training. Certainly after multiple steps of training the cited results on denoising for additive Gaussian noise do not apply. What is the motivation to do so, i.e., what do we learn from this? \n\nMinor comments: \n\nThere are various small typographical errors throughout the paper. Here are a few examples:\n\n- In the first paragraph of Section 2, the authors write \"feller-processes\". This should read \"Feller processes\", as Feller is a name. There is inconsistent capitalization elsewhere, e.g. \"lebesgue\" rather than Lebesgue or \"gaussian\" rather than Gaussian. \n\n- In the paragraph below eq. (6), I believe there is a missing transpose on $\\hat{\\mathbf{a}}$ in the expression for $\\hat{y}$.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8T1ozhmx3x", "reviewer": "Reviewer_2epa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: The paper presents ample discussion on the topic, and discusses many related works in satisfactory fashion. It is well written, and illustrated with several experimental results. The significance and limitations of the claims and observations are well delineated. The results are novel, although limited in scope theoretically (see below), and should be of interest to the community. I have briefly skimmed, but did not carefully examined, the proof.\n\nWeakness: The main technical novelty of the paper consists in an extension of Proposition 2 (Ba et al, 22) to Adam, but is restricted to the limit $\\epsilon=0$. While many experiments are insightful and interesting, none of them (to the best of my reading) conclusively changes or bolsters existing conclusions (such as the correlation between heavy tails and generalization), while they do bring further light and insights.\n\nIn summary, the paper can be of interest to the TMLR readership. Although I have not checked the proofs in detail, the results seem sound, to the best of my reading. I am thus in favor of acceptance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors consider the problem of characterizing the spectrum of the weights of a two-layer network with extensive width, with a single (or small number) of gradient-descent or Adam steps. \n\nTheoretically, in the context of learning a single-index teacher model, they study the scaling of the Frobenius and operator norm of the Adam update, in the limit of where the associated parameter vanishes $\\epsilon=0$, and argue that a learning rate $\\eta=\\Theta(1)$ suffices for a spike to emerge.\n\nEmpirically, they illustrate their finding with several numerical experiments. These experiments support that Adam seems to necessitate smaller learning rates to exhibit a spike, when compared to GD. The authors further evaluate different generalization and heavy-tail metrics, and discuss how they correlate, or fail to. Experiments for more sophisticated practical networks are also presented.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I have a few comments and questions, *none of which is critical to my evaluation*:\n\n- To the best of my reading, it is not specified if the empirical experiments are also run in the $\\epsilon=0$ limit, of if this is only to aid the analytical study. Overall, the significance of this approximation is insufficiently discussed : do the authors expect the theoretical results to still hold for $\\epsilon$ small but non-zero?\n- The authors do not discuss whether a single Adam step allows to beat the kernel lower bound, a key finding of (Ba et al, '22) for GD, which is a main inspiration of the work. While I am aware the authors allude to this as a future direction in Appendix G, I believe the inclusion of even just an experiment would strengthen the interest of the paper to some readers. \n- Is the last term in square brackets in (8) supposed to be transposed ? The dimensions do not seem to match, is it a typo?", "broader_impact_concerns": "I do not have any ethical concern regarding this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)