Spectral Evolution and Invariance in Linear-width Neural Networks
Authors: Zhichao Wang
Andrew William Engel
Anand Sarwate
Ioana Dumitriu
Tony Chiang
Venue: NeurIPS 2023 poster (2023)
Keywords: Random matrix theory
Heavy tails
Feature learning
Linear-width neural networks
Spike phase transition
TL;DR: Spectral Evolution of Linear-width Neural Networks

Abstract: We investigate the spectral properties of linear-width feed-forward neural networks, where the sample size is asymptotically proportional to network width. Empirically, we show that the spectra of weight in this high dimensional regime are invariant when trained by gradient descent for small constant learning rates; we provide a theoretical justification for this observation and prove the invariance of the bulk spectra for both conjugate and neural tangent kernels. We demonstrate similar characteristics when training with stochastic gradient descent with small learning rates. When the learning rate is large, we exhibit the emergence of an outlier whose corresponding eigenvector is aligned with the training data structure. We also show that after adaptive gradient training, where a lower test error and feature learning emerge, both weight and kernel matrices exhibit heavy tail behavior. Simple examples are provided to explain when heavy tails can have better generalizations. We exhibit different spectral properties such as invariant bulk, spike, and heavy-tailed distribution from a two-layer neural network using different training strategies, and then correlate them to the feature learning. Analogous phenomena also appear when we train conventional neural networks with real-world data. We conclude that monitoring the evolution of the spectra during training is an essential step toward understanding the training dynamics and feature learning.

Spectral Evolution and Invariance in Linear-width Neural Networks
Zhichao Wang
Andrew Engel
University of California San Diego zhw0360ucsd, edu
Pacific Northwest National Laboratory andrew... aagel@pnnl.. gov
Anand Sarwate
Ioana Dumitriu
Rutgers, The State University of New Jersey ads221@so0 rutgers.
University of California San Diego
idumitriuQuced, edu
Tony Chiang
Pacific Northwest National Laboratory
University Washington
University of Texas at El Paso
tony. chi ahhinnn.nnn. gov
Abstract
We investigate the spectral properties of linear-width feod-forward neural networks, where the sample size is asymptotically proportional to network width. Empirically, we show that the spectra of weight this high dimensional regime are invariant when trained by gradient descent for small constant learning rates; we provide theoretical justification for this observation and prove the invariance of the bulk spectra for both conjugate and neural tangent kernels. We demonstrate similar characteristics when training with stochastic gradient descent with small learning rates. When the learning rate is large, exhibit the emergence of outlier whose corresponding eigenvector is aligned with the training data structure. We also show that after adaptive gradient training, where lower test errоr and feature learning emerge, both weight and kernel matrices exhibit heavy tail behavior. Simple exam- ples are provided to explain when heavy tails can have better generalizations. We exhibit different spectral properties such as invariant bulk, spike, and heavy-tailed distribution from a two-layer neural network using different training strategies, and then correlate them to the feature learning. Analogous phenomena also appear when we train conventional neural networks with real-world data. We conclude that monitoring the evolution of the spectra during training is essential step toward understanding the training dynamics and feature learning.
1 Introduction
Deep learning theory has made insightful connections between the behavior of neural networks (NNs) and kernel machines through asymptotic analyses of the so-called kernel regime [5
X3
48
40
12
84). When the neural network (NN) is infinitely wide, the behavior of NN coincides with a kernel machine, and the training process, as well as the generalization performance of this altra-wide NN, can be fully described. The performance of finite-width NNs, however, does not correspond to this theory, as NNs optimized with gradient-basee methods perform better than infinitely wide networks in many circumstances 47 38
51 25 32 76 42 This gap heavily relies on the task complexity. data distribution, architecture the NN and the training strategy [1] We consider more realistic
34
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

setting, a linear-width regime (LWR), when the sample size n, the input feature dimension d. and the width of the hidden layer approach infinity comparable rates. Under the LWR, we aim to empirically study this theoretical gap in generalization and spectral properties by training various NNs with different optimization tools.
The ultra-wide NN (h >> n, fixed d) stays close to the kernel machine induced by initial NN, throughout the gradient-baseh training processes [88 28 27 [T]. There are two kernels commonly studied in theory: the conjugate kernel (CK) and the neural tangent kernel (NTK), CK (or the
equivalent Gaussian process kernel) is the Gram matrix of the last hidden layer, which represents training only the last layer of the network 48 [5 61]; by contrast, the NTK is the Gram matrix of the Jacobian of the NN for all trainable parameters, which governs the gradient flow of NN 40 (88 4 In most theoretical results, these kernels remain fixed throughout training, which leads to kernel
gradient descent with the initial kernel 40
16 whereas practice the spectra of the weight matrix,
CK, and NTK of the NN change while learning the features from the training data [58 59 30 19, 68]
In this paper, under the LWR, we experimentally and theoretically explore the following question:
How do the spectra of weight and kernel matrices of the NN evolve during the training process?
This question is crucial extend our understanding beyond the kernel regime and will help us analyze the generalization of the NN in instances when it performs better than the kernel machine. For this case, the spectral properties of the trained NN could be entirely different from the initial kernel [55 10 78]. Also, various spectral properties of weight and kernel matrices can reveal different features learned by different training procedures [82]. Understanding the dynamics of the spectral properties may aid in finding better approaches to training and tuning sypper-araamter for NNs. From a theoretical perspective, random matrix theory (RMT) can be further exploited to study and elucidate the NN training under the proportional limit in high dimensions 45
57
74
61
Our main findings/contributiinn
follows.
We find a simple scenario that exhibits different spectral properties for both weight and kernel matrices through different training procedures. With the kernel regime benchmark, we compare how NN generalizes with different spectral evolutions of weight and kernel matrices in NN.
The spectra of NNs trained with full batch gradient descent (GD) are globally invariant, indicating that the NN is still close to kernel machine; we prove the global convergence of GD and the invariance of the limiting spectra for both weight and kernel matrices in this scenario.
We observe a phase transition of the alignment and the emergence of a spike outside the bulk of the spectrum when the learning rate exceeds some threshold. The strong alignment of the spike with the teacher model when step sizes are large confirms that the NN is indeed learning germane features from data. This observation justifies the theoretical result of (6) in an ideal two-stage training process.
The evolution towards heavy-tailed spectra is also discovered by using adaptive methods. Our experiments rule out a causal relationship between the occurrence of heavy-tailed spectrum for the weight matrices and a good generalization. This complements the work of [60] 63 [6) where the authors had observed a strong correlation between the two; while at the same time, we provide simple examples of when heavy-tailed spectra exhibit feature learning and better generalizations.
For more details on how our results fit into existing literature, please see Appendix
2 Notation and Preliminaries
Throughout this paper, || denotes the E2 norm for vectors, l2 - b2 is the operator norm for matrices, while -| / the Frobenius norm, and @ represents the Hadamard product between matrices. (.) represents little-o in probability as d + co.
2

---

Neural Tangent Kernel Parameterization. initialization without bias term: for 2
Consider L-layer fully connected feedforward NN at
ho
I vd
10) ( ch
(Wehh( (1
(I)
VTEE
and - for the L-th layer, and v = vh,....,tu [h]] E R is the last-layer weight. Let no = d. Denote all trainable parameters by 0 = [vec 1)...... (ec(W v] IR where each parameter's initial value is independently sampled from some distribution and p is the total number of parameters. Let the training dataset be (X, y) = ([=1...... Lnl: y) [Rdxn x R2'=; the output of this NN with respect to this dataset is fo (X) (x1) ..... (xx)). We call the above parameterization the NTK parameterization. The loss function for training is a mean squared error (MSE)
where the input vector is æ E Rd, We Ru is the weight matrix
((
y fo (x)
We focus on the NTK parameterization and consider the kernel machine (6) induced by the initial NTK of the NN. We aim to seek the cases when the NN outperforms this kernel during the training process. For this purpose, we adopt different optimizers of training this NN obtain different testing performances and spectral properties of trained weights and empirical kernels.
Training Processes of NNs. NNs are usually trained by gradient- baaedd methods such as full-batch gradient descent (GD), mini-batch stochastic gradient descent (SGD), Adaptive Gradients (AdaGrad), and Adam 433 We can represent GD by
0,+1 0. -V
where 7) is the learning rate and VoL(O,) is the gradient of the training loss w.r.t. trainable parameters at step /. 2 O. We will prove the global convergence of GD in some special (overparameterizee) cases ensuring the convergence
NN that interpolates the data. We will also show the hyper- parameters (e.g. learning rate 17) affect the spectral properties of NNs during training.
Conjugate Kernel and Neural Tangent Kernel When L 2, let n1 and no be the widths of the output and input layer. The CK is denned as
K-C x R
(4)
where X1 == 1 WX/Vd). We can view the NN function of all training parameters
Vh and input data X. The neural tangent kernel (NTK) is related to the gradient of this neural network function with respect to 0, which is the Gram matrix of the Jacobian of the neural network function with respect 0, KNN = (o ( (Vofo(X)), Specifically, the empirical NTK of two-layer NN can be explicitly writter
1 oo (Va NTN 1 x
(
WX
diag(v)²o' (va WX) + K-CK
(5)
In this paper, we are interested in comparing the spectral distributions for these three matrices (weight, CK, and NTK) initialization and the end of training.
Lazy Training. Lazy training 21 can be viewed as a linear approximation of the NN, i.e. fe(x) ~ fea (x) + (0 - 0o) Vefe. (x), defined by minimum-norn interpolation Ô == arg min
[Ile 0 : (0- 0o) VoJo. X) = y fo, (X) Then, lazy training also represents a kernel
machine
f(x) = fee (x) + (y fe. (X) (XXX)) XXXXXX)'XXXX,, (X
In this work, we only consider empirical conjugate and neural tangent kernels of finite-widh width NNs. "Here we train both layers, so we have two parts in NTK expression. we only train the first-hidden layer, we can simply remove the second CK part. In the following, we further introduce more empirical results for practical NNs in Section 5 and two-layer NNs with Gaussian dataset in Section 3 For general formuaa of the empirical NTK, see [3 30
3

---

where f(x) is the unregularized regression prediction on test data a E R", the kernel ( is the initial K-NTK on training data, and K (X, .²) = (Vose, (Vofo, The asymptotic performance of f(x) has been analyzed by [ under the LWR. We view this regime mark: [T]] prove that NN through gradient flow is close to lazy training if h > n; [8] shows NN can go beyond lazy training under non propprrptti regime.
bench-
3 Case Study for Linear-width NNs
In this section, we investigate a two-layer NN with synthetic data. This setting is promising for future theoretical studies by virtue of RMT. We will showcase the evolution of its spectral properties over training. two-layer NN in (1 is defined by
fo(x) = vh Σ w (;o(w[//Vd))
(7)
At initialization, we assume that the first hidden-layer W [wi,... wa) E Bixd is composed of independent standard normal random vectors.
Assumption 3.1 (Linear-width regime (LWR)). Assume that म + 71 and 3 + 72 as + 0o where the aspect ratios y1, 72 E (0, oo) are two fixed constants.
LWR stands as pivotal setting grounded in high-dimensional statistics 1 61 It offers valuable insights especially when addressing world datasets. This is in contrast to the infinite-widih regime, in which we are already in the asymptotic limit for width at first. Hence, LWR is better approximation of real-world datasets and practical neural networks compared with the inftnite-width regime.
Assumption 3.2 (Activation function). Suppose that the activation function (z) nonlinear and A₂ -LpLishhiitz with |or (z)). (z)).(""(z)) S X; for all x E R. Moreover, E[a(()) = 0 for 2 ~ ~N0 1).
Though the LWR is somewhat impractical, it is still more aligned with deployed models than the wndth regime (h >> n, fixed d). As kernel machine, the infinite-width NN has been studied 84] This infinite-width limit is special, however, NNs may generally evolve beyond the kernel regime and achieve superior performance (3155) 10 78).
extensively 440 28
Optimization
Learning rate 1)
R2 score
Test error
Spectra
Case I
GD
5.0
0,63582
0.36381
Invariant Bulk
Case 2
SGD
0.1
0.60605
0.36879
Invariant Bulk
Case 3
SGD
22.0
0.76081
0.23791
Bulk+spike
Case 4
Adam
0.092
0.78829
0.21071
Heavy tail
Lazy regime
0.68092
0.3185
Four models with the same architecture (n = 2000, h = 1500, d = 1000, and a is normalized tanh), but different choices of initial learning rates and optimizers listed in Table The training label noise o. 0.3 and the teacher model is defined by (g) with o" normalized softplus and 0.2. We observe that simply choosing an optimizer and learning rate can affect the shapes of the final spectra and the performance of the NN. as measured by Ite scores and test errors.
Table 1:
Assumption 3.3 (Synthetic dataset and teacher model). Training data is X = [xi,..... F.] E dxn where a, f.d.d. NN(0, Ia). The training labels y = [Vi,..., yn] are defined by yi (x;) + Ei, for i [n], where fr Rd + IR is the teacher model, and e; is centered sub-Gaussian noise with variance o.
One of the simplest nonlinear teacher models
generate is the single-index model, namely f*(x) = o" (x (3) for a fixed vector ,3 with || = 1 and nonlinear function a"; the hidden feature simply / E Rdd. In general, we can consider a multiple-inde model
1
Σ
0" (x (3))
(8)
k
i=1

---

0.5
0.20
plot
Initia Trained
Q-Q plct
Inttal
Q-Q plot
Intial Trined
Trained
0.4
Trained
10-
0.15
Traine
100
0.4
0.10
Initial
0.2
Initial
Initial
0.2
0.05
0.1
2 Weight spectra
2
10
12
0.0
2.5
5.0
10.0 12.5 15.0
(b) CK spectra
(c) CK spectra
Figure 1: Different spectral behaviors in Table I (a) The initial and trained spectra of W in Case 1. The
spectrum is invariant based on the Q-Q subplot. (b) The initial and trained spectra of K Case 3. There
an outlier (orange arrow) in the spectrum after training. (c) The imitial and trained spectra K Case 4. We refer to Appendix [B.2] for other spectra of weight, CK, and NTK matrices in Case 1-4, where analogous phenomena hold for other matrices.
where B3 are orthogonal unit vectors. We will specifically consider mixture of single-index and quadratic models as our teacher model this section:
(xxx= 0z B) |
for some nonlinear target o", signal 3 and constant - Following the above assumptions and constructions, we show different spectral properties (Figure I for this two-layer NN using different training procedures (Table Figure exhhibits three types of spectra after training: unchanged bulk distribution, bulk with one spike, and heavy tail spectra. Putting things together, we can see close relationships between the spectra and the generalization of the NN. These different spectral properties actually reveal disparate features learned via different training strategics.
The advantage of this toy model is that we can easily extract the spectral behaviors over training and then compare them with the kernel machine. We use lazy training defined in O as our benchmark to assist us in determining whether a NN outperforms the associated kernel machine. Table compares the test errors and R² scores for different optimization cases and the lazy training. By tuning the hyper-parameters, we can find specific situations where NN outperforms the lazy training (see also Figure (0(c) Appendix B.2).
From Figures I [a). 12 and 13 Appendix x.xxxB2.22 B.2, one can observe the spectral distributions of the weight, CK and NTK matrices remain invariant and static during training in Cases 1&2, which indicates both cases still belong to the lazy regime. This spectral invariance impedes further feature learning during the training process. The emergence of the outlier in Figures [b) and of Appendix B.2 however, shows the improvement over lazy training and potential feature learning via the training process, where the spectra possibly inherit the structures teacher models (see Section (4.2). Comparing with Case 2, Case of Table jsuggests the importance of the large learning rate regime for training NNs As remark, our spectral results Case are consistent with the observations in through RM'T hypothesis testing, where the majority of trained weight matrices remain random, and the learned feature may be contained in the largest singular value (outlier) and associated vector only. From Figures [Tc) and 16 I6 Appendix B.2] Case further exhibits more spikes and heavy tails in the trained spectra, which thoroughly goes beyond the realm of initial kernel machine. Notably, this phenomenon is not unique Adam since heavy tails also occur with AdaGrad in Figure 2 in Appendix B.6 Although all of these cases have the same identical initialization, different methods of optimization eventually lead to various training trajectories and evolutions of the spectra of the weight and kernel matrices. To acquire feature learning, Cases 3&:4 cause weights to deviate far from initialization. In the following Section provide more refined analyses of spikes and heavy tails in terms of feature learning.
52
64
55
15
79
prove the invariance of the bulk distributions and
'Her, the norm term of æ in 9 is designed to make the teacher model more complicated to be learned. All empirical results still hold when O.
5

---

Different Spectral Behaviors in NNs
We now further explore the spectral behaviors in different cases of Table | by clarifying how the spectra evolve through different training processes and how this evolution may affect the NN. we study the training processes case-by-case: invariant bulk, spikes outside the bulk, and heavy-tailed distribution. Additional experiments are exhibited in Appendix
Following Figure [
4.1 Invariant Bulk Distributions
In Figure aa) (also Figures T2 and 13 in Appendix B.2). we observe the bulk distributions of weight and kernel matrices in Cases 1&2 remain globally unchanged (invariant) over the training process. under the LWR, this is also empirically verified by Figures I0 (b):(b) in Appendix [B2 In this section, by investigating the global convergence GD, we prove this invariant-bulk phenomenon under certain assumptions.
For simplicity, we focus on analyzing the training process of the first-hidden layer with the second layer fixed. Denote fo (X) by fw(X) (X) in this case. At any time t E N, consider the gradient steps:
W ++1 = W.
t.W.)
(10)
Denote the CK and NTK at gradient step t E N by K. = (W (W,)), and K NTK := diag(u,) description of the changes the weight, CK, and NTK at the early phase of the training (after any finite steps) as follows.
x tot ( W,
( X x respectively. First, we present an elaborate Vd
Lemma 4.1 (Early phase). Under Assumptions 3.1 3.2 and 3.3 we further assume that ||oll S 1 and f is a X Liischitr.functin function. Given any fixed t E N and learning rate 17 = ((),, after t gradient [W - Woll K KOK CK| and || K are all less than C n.
steps, the changes
with probability at least 4n exp (-cn), ). for some positive constants c, which only depend on step t and parameters 7) 71, 72, No, dz.
Lemma 4.1 shows Va W. Woll. K - -CCK-KKKI and K K are asymptotically
vanishing for any fixed time f. Therefore, all the eigenvalueseiggnvvetoss are asymptotically un- changed at the early phase of the training (see Corollary [C. Appendix (C). Now we aim to analyze the spectra at the end of the training process (TO). In this case, although we are unable to show the invariance for each eigenvalue, we can verify the invariance of the limiting bulk distributions for KCK and K.CK for all
By 81 Theorem 2.9], the smallest eigenvalue of 0 NTK has asymptotic lower bound:
Amin NIN 2
Ge
The (1-op((1)) oz)(()))
(11)
where a, := [[ and 7 is the k-th Hermite coefficient of o'. Hence, can claim there exists some constant > 0 only dependent a such that Amin (IKNTK) > 4a² with high probability, Note that a is not vanishing since a is nonlinear. With this lower bound, we obtain the following global convergence for (10) and norm control of W as n/d 7 and h/d-
Theorem 4.2 (Global convergence). Under the same assumptions of Lemma 4.1 we further assume V: 's are independent and centered random variables in the second layer. For any a'n n min{ 2 (1+ and all E N, there exists some 7 such that, when 72 2 y", the gradient steps
10 will satisfy
1702)
((W,) - 1
((Wo),
(12)
2n
W. -Will + ((W,) s (Ww))
(13)
W - W. w.ll < 48( 48(Wo)
(14)
0=0

---

with high probability, as n/d + - and h/d - 72. Here, training loss (W) := ly fw (X)
We apply the techniques and results by 669 70 to obtain Theorem 4.2 Notice that, unlike Lemma 4.1 the largest learning rate we can choose is of order (()). As a byproduct, the Frobenius norm in 13 implies the following corollary for the invariance of limiting bulk distribution.
Corollary 4.3. Under the same assumptions of Theoremm 4.2 for all t E N. with high probability, there exists some constant |KTTX Ko p are all less than R with high probability. This implies the limiting empirical
such that the changes Va IW. Wollp, | -Ko*ll
r and
spectra of h w, W. K.t and K.r are the same as the limiting spectra of h W. o Wa. KOO and KN respectively, almost surely as n/d - 711 and h/d - 72-
Corollary 4.3 is empirically validated by Figure 29 Appendix In addition, based on Figure 3] in Appendix one can further extend Corollary 4.3 to the SGD training process. The total path is O(hh) in 13 and (14), which is negligible compared with the Frobenius norm of initial weight matrix (which is of order (h)). Thus, gradient descent iterates (1O) remain close to initialization
and small perturbation of NTK ensures the smallest eigenvalue (11 of NTK is always lower bounded
away from zero. Theorem 4.2 however, does not require that the NTK stays unchanged all the time.
Moreover, Corollary 4.3 only shows the invariance of the bulk distribution, while the emergence of outliers cannot be excluded from this result. Though we have global convergence in general, we may still move out of the kernel regime. Global convergence cannot explain when NN in LWR outperforms the kernel regime (Figure [)).)). Notice that [I] Theorem 5.4] is not directly applicable to show that NN is still close to lazy training under the LWR. It requires deeper analysis to claim whether a NN still belongs to the kernel regime or already goes beyond in our case. As we will show in Section 4.2 this also relies the magnitude of the learning rate for GD/SGD.
4.2 Aligmments for Spiked Models
The outliers appear in the spectra of the trained weight, CK, and NTK matrices (Figure 1 in Appendix[B2. when NNs are optimized with large learning rates. The outlier is especially clear for the NTK matrix (Figure [4fb)). Heuristically. this indicates that the NN learning the feature from the teacher model fr. In Figure ([a). a). Figures| and 27 in Appendix D we empirically exhibit these phenomena for W, K CK and K NIK respectively through different training processes.
2.30
Viii r13h:
rillal right
wetght OX
2.00
2.00
-2.06
32
0.2
-2.00
Talnod
0.0
Sigaal tww.wate mane
(a) 3 vs. ui of weight.
(b) Amax (W,W WT /d)
(c) Amax (KCK)
(d) Alignments
Figure 2: (a) Alignment between teacher feature 3 and first PC of the trained/imitial weights in Case 3 of Table (b)-(d) Transitions Amax WWW. /d), >max and alignments (10 11,3] and ly T w/l/ ||y||| where ui and ei are the first singular vectors of W, and either K. KN respectively) when increasing the learning rate y) while training the NN with SGD. the green region, the largest eigenvalues are attached to the bulk (black horizontal lines) and the alignments are weak; in the orange one, outliers become apparent and the alignments hecome stronger. For different 7,, we train the same NN with the same datasct until the training loss is less than 10 Here, "Ir" in the r-axis represents varying lcarning rates.
Spikes of Weight Matrices. The differences between Cases 2&:3 empirically validate the benefits of training with large learning rates [52 64
15 D. Inspired by (6. we consider the alignment between the leading right singular vector u of W and the signal /3 in the teacher model defined by (9). For Case 3, notable alignment appearing in Figure ( after training suggests that W
[5
is capturing the feature /3 during training. Although this does not ensure NN will entirely beat the optimal kernel lower bound, this alignment reveals non-negligible feature selection [10] via large- stepsize training. This dynamical alignment along the task-relevant direction may further interpret the generalization of the NN. We also observe similar phenomena for the adaptive optimization in
Figure 27 in Appendix B.7
7

---

5GD
025
0.10
0.35
(a) Evolution of KTA.
(b) Alignment evolution. (c) Heavy-tailed initial CK. (d) Bad example.
Figure 3: (a) Evolution KTA of CK defined by 15 with respect to training labels for Cases 1, 38c4 in Table
We normalize the epoch scales (æ-axis) for better observations. Heavy-tailed phenomena: (b) Evolutions of PC angles 0, between feature subspace U of (8) and top 100 eigenspace of W; W, during training with Adam (solid line). SGD (dashed line) and GD (dash-dot). For the first PC 01, Figure in Appendix B.4 (c) The CK spectra two initializations for W: standard Gaussian and Cauchy distributions. (d) Weight spectra initial and after SGD training. After training the weight reveals a heavy tail, but generalizes not as well as former examples (test loss 1.47504; R score -0.48).
Transitions of the Spike as a Function of Learning Rate. From Case 2 to Case we observe the emergence of outliers in the trained spectra when increasing the learning rate 7). This indicates transition of the emergence of the spike outside the bulk distribution. Figure [ analogously to the well-known BBP transition by Baik, Ben Arous, and Péché in [9] from the RMT community, shows there is a threshold (yellow region) for learning rate: the outliers only appear when r/ exceeds this threshold. We fix the same NN and dataset for all trials of training. The flat black lines in Figures (bb) and (c) are the right edges of the limiting spectra at initialization. Figure ((d) records the angles between 33 and the leading eigenvector W, Wa We/d, and y and the leading eigenvectors of K-C and K after training for different yy. Similarly with [10), when v) is sufficiently large (orange region), we obtain significant alignments which suggest potential feature learning. These transitions of leading eigenvalue and eigenvector alignment have been proved for We by 66 for a different scenariq
Spikes of Kernel Matrices. The alignment of the kernel matrix with the training labels y is defined by 122 by Kerne Target Alignment (KTA) as follows: when kernel K is either CK or NTK,
(K y y)
KTA-
(15)
K
|
Analogously to 110 78, Figure (aa depicts the evolution of KTA of CK in several cases. Based on
Figure [ d), when the spike appears outside the bulk (Case 3), its corresponding (leading) eigenvector 01 of kernel matrix naturally dominates the alignment with (Figure [5] in Appendix [..)) which regarded as a kernel rotation during training in (68. Notice that this is not the common situation in Cases 1&2 of Table [ (and cf. Figure I1 in Appendix B.2). On the other hand, KTA measures the alignment between y and the full eigenbasis of the kernel. These kernel alignments improve the speed of the convergence of training dynamics but may hurt or boost the generalization of the NNs 168, (78 [0]. Figure indicates that Case with heavy-tailed spectra after training has a larger KTA than the other cases. In this case, the emergence of heavy tail in the spectrum is closely related to better generalization the NN and more significant feature learning.
4.3
Phenomenon of Heavy-tailed Spectra
Next, we analyze the heavy-tailed spectra of weight and kernel matrices in Figure C(.. found strong correlation between the heavy-tailed spectra of trained state-of-the-arr the-art models with better generalization (Figure 7 in Appendix [.I)). Heavy-tailed spectra can be viewed as an extreme of "bulk+spikes", where a fraction of the eigenvalues move out of the initial bulk. In RMT, heavy- tailed spectra generally appear when the entries of the matrix are highly correlated [59 This could heuristically explain ntived phenomena in the spectra since the entries of well-trained W should be strongly correlated. Unlike [58, 59 we focus on the heavy-tailed phenomena for both weight and kernel matrices a simpler model 7 and provide a connection between feature learning and heavy-tailed spectra, which opens important avenue for further theoretical analysis.
58 59
WWe apply NTK parameterization for our neural networks and train both layers until convergence, while 6 considers the mean-field initialization and early stage of training dynamics of GD for the first layer.
8

---

1000
1400
Tranned
nwieed httal
Q-9 plot
Ind
1200-
Q-Q plot
Q-0 piot
500
10
1000
100
10
400
.6'
12'
200
Indis!
Intill
2500
4000
2000
3030
9 1000 1300
5200 3500
(a) GD
(b) SGD
(c) Adam
Figure 4: Different NTK spectra for small-CNN model on CIFAR-2, The subplots are Q-Q plots for the comparison between initial and trained spectra. Test accuracies: (a) 79%, (b) 84%, (c) 86.4%.
Heavy Tails and Generalization. We emphasize that heavy tails not sufficient for good generalization, in general, [60] [3]. Figures 3 (d) exhibit NNs with heavy-tailed weights but in the absence of good performance initialization. In fact, it is the alignments between the features learned from the heavy-tailed part and the features in the teacher model that finally determine the generalization error of NNs.
More precisely, we provide an example of when heavy tails indicate better generalizations. Consider the multiple-index teacher model (8) with k 5 feature directions B,. and train NNs Z with GD, SGD, and Adam to get invariant bulk, bulk with one spike and heavy tails, respectively. after training. In Figure 3 we present the evolutions of the principle angles 6; between feature subspace U span ( (B, Fi=1 and top 100 eigenspace of W. W i during different training processes. This eigenspace with respect to the top 100 eigenvalues of W W. corresponds to the heavy-tail part of the spectrum in W, W. when training NNs with Adam (solid lines in Figure b)). Comparing with GD and SGD training processes, we observe strong aligmments between feature space U and eigenspace w.r.t heavy tails Adam case Figure bb) which explains why Adam case (NNs with heavy-tailed spectra) generalizes better than the other two cases. For more examples, see Figures] 17 and20 Appendix [..4] This concludes that NNs with heavy-tailed spectra can generalize better only when the teacher features from data are aligned with the heavy-tailed part of spectra. If the feature dimension in the teacher model is high (i.e. the teacher model is more complicated and intrinsically high-dimensional), then we expect to get heavy-tailed weight spectrum of well-trained NN where the heavy-tailed part learns all the features in the teacher modes. This example explains why we can use the heavy tails to discriminate well-trained and poorly-trained large models [60 63 86
Discussions and Future Directions
We empirically investigated how the spectra of W, K-C and KNN evolve under the LWR for an idcalized student- teacher setting. Our work implies that understanding the relationship between feature learning and training processes requires understanding the evolution of the spectra of both weight and kernel matrices. In particular, we show that different training processes affect the eigenstructure of weight and kernel matrices. Since evolution is sensitive to feature learning, can link feature learning and different training dynamics by studying the spectra of these matrices.
While synthetic data is easier to analyze theoretically, we also investigate these spectral properties on real-world data and more complicated tasks in the following. In practice, people mainly focus on analyzing spectra of the weight matrices in fully connected layers; we choose to also focus on the spectral properties of general kernel matrices induced by the NNs, which contain abundant information [19 55 78).
First, we show the spectra of K NTK before and after training for binary classification on CIFAR-2 through small CNNs in Figure 4 Similarly with Case 1, Figure 4a (especially in the Q-Q subplot) manifests the invariant spectral distribution of NTK through GD training while SGD exhibits heavier tail in NTK spectrum in Figure bb.. This phenomenon is more evident when trained by Adam in Figure with improved accuracy. data in Section can be extended to real-world data and on more practical architectures. We note that there a lack of the emergence of spikes after training because spikes already exist in the initial NTK spectrum for this complicated neural architecture real-world datasets. Figure 4 aa) also indicates
Juggesst that our observations on synthetic

---

that the spectral invariance of NTK through training will impede the feature learning and the NN does not generalize well in this training process.
We also investigate the spectral pronerties on the pre-trained model, BERT from 26 with fine-tuning on Sentiment 40 dataset of tweet from [36]. We fine-tune the BERT model for binary classifier on Sentiment 140 and capture the evolution of CK spectra, rather than the NTK due to the size of BERT, in Figure 5 (see also Figure in Appendix B.
1e6
first
0.8
0.8
aligemrr
0.6
0.6
second
0.4
0.2
0.0
10
20
20
opech
opoch
(a) Eigenvalues
(b) Alignments
Figure 5: We SGD for fine-tuning the BERT model. The training accuracy is 95.90% and the test accuracy 84%. (a) The evolution first and second eigenvalues of empirical CK during fine-tuning, (b) The alignments of training labels with first and second eigenvectors of CK during fine-tuning, See Figure for the spectra of CK at different epochs.
A heavy-tailed CK spectrum with several spikes already exists in this pre-trained model. Unlike Figure 4 (and cases in Table [ where the first spike NTK becomes larger than at random initialization after training, in Figure aa.) the leading eigenvalue first decreases and then increases. Moreover, similarly to Figure d), our Figure b)b) shows that the aligmment of the first eigenvector of the CK and training labels becomes more apparent through fine-tuning with the leading eigenvalue decrease. Heuristically, this process seems to unlearn the features in the pre-trained model and, remarkably, learn new features on the new dataset in only few epochs of fine-tuning (see Figure 7 Appendix I. We believe that the evolutions of the kernel matrices and some spectral metrics are crucial for understanding feature learning through fine-tuning (2] more comprehensive exploration of the evolutionary spectral properties of "foundation models" may help shed further light on these phenomena.
Limitations. Although LWR has garnered significant attention in recent years, e.g., 777 50 17
87 [23), we recognize the limitations inherent in LWR. Our LWR is more realistic compared with infinite-width neural networks and is one of the ways to approximate finite but very large neural networks with very large datasets, but there are more sophisticated regimes for NNs. We leave this for future theoretical work. The NTK parameterization is another limitation of this work. We expect to apply our spectral analysis for other parameterizations of NNs with more real-world datasets. See the discussion at the beginning of Appendix B
Acknowledgement
Z.W., A.E., I.D., and T.C. were partially supported by the Mathematics for Artificial Reasoning in Science (MARS) initiative via the Laboratory Directed Research and Development (LDRD) Program at Pacific Northwest National Laboratory (PNNL). A.S. and T.C. were also partially supported by the Statistical Inference Generates kNowledge for Artificial Learners (SIGNAL) program PNNL. PNNL is a multi-program national laboratory operated for the U.S. Department of Energy (DOE) by Battelle Memorial Institute under Contract No. DEE-ACO5-76RLO-180 Z.W. would like to thank Denny Wu and Libin Zhu for their valuable suggestions and comments.
10

---

References
[1] Ben Adlam and Jeffrey Pennington. The neural tangent kernel in high dimensions: Triple descent and multi-scale theory of generalization. In International Conference Machine Learning, pages 74 84. PMLR, 2020.
[2] Zeyuan Allen-Zhte Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparame- terized neural networks, going beyond two layers. Advances neural information processing systems, 32, 2019.
[3] Maksym Andriushchenko, Aditya Vardhan Varr,, Loucas Pillaud-Vivien, and Nicolas Flammar- ion. Sgd with large step sizes learns sparse features. In International Conference on Machine Learning, pages 903-9255 PMLR, 2023.
[4] Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances in Neural Information Processing Systems, 32, 2019.
[5] Alexander Atanasov, Blake Bordelon, and Cengiz Pehlevan. Neural networks as kernel learners: The silent alignment effect. In International Conference on Learning Representations, 2022.
[6] Jimmy Ba, Murat A Erdogdu, Taiji Suzuki, Zhichao Wang, Denny Wu, and Greg Yang. High- dimensional asymptotics of feature learning: How one gradient step improves the representation. Advances in Neural Information Processing Systems, 35:37932-37946, 2022.
[7] Yu Bai and Jason D. Lee. Beyond linearization: On quadratic and higher-order approximation of wide neural networks. In International Conference on Learning Representations, 2020.
[8] Zhidong Bai and Jack W Silverstein. Spectral analysis of large dimensional random matrices, volume 20. Springer, 2010.
[9] Jinho Baik, Gérard Ben Arous, and Sandrine Péché. Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices. The Annals of Probability, 3343) 2005.
[10] Aristide Baratin, Thomas George, César Laurent, R Devon Hjelm, Guillaume Lajoie, Pascal Vincent, and Simon Lacoste-Julien. Implicit regularization via neural feature alignment. In International Conference on Artificial Intelligence and Statistics, pages 2269-2277. PMLR, 2021.
[11] Peter I. Bartlett, Andrea Montanari, and Alexander Rakhlin. Deep learning: a statistical viewpoint. Acta numerica, 30:87-201, 2021.
[12] Mikhail Belkin, Siyuan Ma, and Soumik Mandal. To understand deep learning we need to understand kernel learning. In International Conference Machine Learning, pages 541-549. PMLR, 2018
[13] Lucas Benigni and Sandrine Péché. Higenvalue distribution of some nonlinear models of random matrices. Electronic Journal of Probability, 26:1-37, 2021.
[14] Lucas Benigni and Sandrine Péché. Largest eigenvalues of the conjugate kernel of ssngle-layered neural networks. arXiv preprint 2XXi:2201.004533 2022.
[15] Gaspard Beugnot, Julien Mairal, and Alessandro Rudi. On the benefits of large learning rates for kernel methods. In Conference on Learning Theory, pages 282- PMLR, 2022.
[16] Alberto Bietti and Julien Mairal. On the inductive bias of neural tangent kernels. In Advances in Neural Information Processing Systems, pages 12873-12884, 2019.
[17] David Bosch, Ashkan Panahi, and Babak Hassibi. Precise asymptotic analysis of deep random feature models. arXiv preprint 2.02:22002002100 2023.
[18] Sourav Chatterjee, Convergence of gradient descent for deep neural networks. arXiv preprint 20X1::200..16462,22 2022.

---

[19] Shuxiao Chen, Hangfeng He, and Weijie Su. Label-aware neural tangent kernel: Toward better generalization and local elasticity. Advances Neural Information Processing Systems, 33, 2020.
[20] Lenaic Chizat and Francis Bach. On the global convergence of gradient descent for over- parameterized models using optimal transport. In Advances in neural information processing systems, pages 3036-3046, 2018.
[21] Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable program- ming. Advances in Neural Information Processing Systems, 32, 2019.
[22] Nello Cristianini, John Shawe- Shawe-Taylor, Andre Elisseeff, and Jaz Kandola. On kernel-target alignment. Advances in neural information processing systems, 14, 2001.
[23] Hugo Cui, Florent Krzakala, and Lenka Zdeborova. Bayes-optimal learning of deep random networks of extensive-width. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 6468 6521. PMLR, 23-29 Jul 2023.
[24] Alexandru Damian, Jason Lee, and Mahdi Soltanolkotabi. Neural networks can learn represen- tations with gradient descent. In Conference on Learning Theory, pages 5413-5452. PMLR, 2022.
[25] Amit Daniely and Eran Malach. Learning parities with neural networks, Advances in Neural Information Processing Systems, 33:20356-2085 20365, 2020.
[26] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0480) 2018.
[27] Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International conference on machine learning, pages 1675-1685. PMLR, 2019.
[28] Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. In International Conference on Learning Representations, 2018.
[29] Ethan Dyer and Guy Gur-Ari. Asymptotics of wide networks from feynman diagrams. In International Conference on Learning Representations, 2020.
[30] Zhou Fan and Zhichao Wang. Spectra of the conjugate kernel and neural tangent kernel for linear-width neural networks. Advances in neural information processing systems, 33, 2020.
[31] Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli. Deep learning versus kernel learning: empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. Advances in Neural Information Processing Systems, 33:5850-5861, 2020.
[32] Mario Geiger, Stefano Spigler, Arthur Jacot, and Matthicu Wyart. Disentangling feature and lazy training in deep neural networks. Journal of Statistical Mechanics: Theory and Experiment, 2020(11): 11:10113333,,, 13301, 2020.
[33] Federica Gerace, Bruno Loureiro, Florent Krzakala, Marc Mézard, and Lenka Zdeborová. Generalisation error in learning with random features and the hidden manifold model. In International Conference on Machine Learning, pages 3462. PMLR, 2020.
[34] Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. When do neural networks outperform kernel methods? Advances in Neural Information Processing Systems, 33:14820-148300 2020.
[35] Behrooz Ghorbani, Song Mei, Theodor Misiakiewiez, and Andrea Montanari. Linearized two-layers neural networks in high dimension. The Annals of Statistics, 1922):1029-1054 2021.
12

---

[36] Alec Go, Richa Bhayani, and Lei Huang. Twitter sentiment classification using distant supervi- sion. CS224N project report. Stanford, 1(12):2009. 2009.
[37] Boris Hanin and Mihai Nica. Products of many large random matrices and gradients in deep neural networks. Communications in Mathematical Physics, 376(1):287-322, 2020.
[38] Wei Hu, Lechao Xiao, Ben Adlam, and Jeffrey Pennington. The surprising simplicity of the early-time learning dynamics of neural networks. Advances Neural Information Processing 3ystems,33:171I 33:17116-17128, 2020.
[39] Jiaoyang Huang and Horng-Tzer Yau. Dynamics deep neural networks and neural tangent hierarchy. In International Conference Machine Learning, pages 4542-455/ PMLR, 2020.
[40] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018
[41] Stanislaw Jastrzebski, Maciej Szymczak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho, and Krzysztof Geras. The break-even point on optimization trajectories of deep neural networks. In International Conference on Learning Representations, 2020.
[42] Stefani Karp, Ezra Winston, Yuanzhi Li, and Aarti Singh. Local signal adaptivity: Provable feature learning in neural networks beyond kernels. Advances in Neural Information Processing Systems, 34, 2021.
[43] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69880 2014.
[44] Dmitry Kopitkov and Vadim Indelman. Neural spectrum alignment: Empirical study. In International Conference on Artificial Neural Networks, pages 168-179. Springer, 2020.
[45] Yann Le Cun, Ido Kanter, and Sara Solla. Eigenvalues of covariance matrices: Application to neural-network learning. Physical Review Letters, 66(18):2396, 1991.
[46] Guillaume Leclerc and Aleksander Madry. The two regimes of deep network training. arXiv preprint 203000000002.0766, 2020.
[47] Jaehoon Lee, Samuel Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein. Finite versus infinite neural networks: an empirical study. Advances in Neural Information Processing Systems, 11151116111722 2020.
[48] Jaehoon Lee, Jascha Sohl-dickstein, Jeffrey Pennington, Roman Novak, Sam Schoenholz, and Yasaman Bahri. Deep neural networks as gaussian processes. In International Conference on Learning Representations, 2018
[49] Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The large learning rate phase of deep learning: the catapult mechanism. arXiv preprint arXiv:2003.02218, 2020.
[50] Qianyi Li and Haim Sompolinsky. Statistical mechanics of deep linear neural networks: The backpropagating kernel renormalization. Physical Review X, 11(3):03105, 2021.
[51] Yuanzhi Li, Tengyu Ma, and Hongyang Zhang. Learning over aver-parametrized two-layer neural networks beyond ntk. In Conference on learning theory, pages 2613 1682.. PMLR, 2020.
[52] Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. In Advances in Neural Information Processing Systems, pages 11674-1168. 11685, 2019.
[53] Zhenyu Liao, Romain Couillet, and Michael W Mahoney. A random matrix analysis of random fourier features: beyond the gaussian kernel, a precise phase transition, and the corresponding double descent. Advances in Neural Information Processing Systems, 11-113999-19500 2020.
13

---

[54] Chaoyue Liu, Libin Zhu, and Mikhail Belkin. Loss landscapes and optimization in over- parameterized non-linear systems and neural networks. Applied and Computational Harmonic Analysis, 2022.
[55] Philip Long. Properties of the after kernel. arXiv preprint arXiv:2105./0583 2021.
[56] Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Evolution of neural tangent kernels under benign and adversarial training. Advances in Neural Information Processing Systems, 35:1 1642-11657, 2022.
[57] Cosme Louart, Zhenyu Liao, and Romain Couillet. A random matrix approach to neural networks. The Annals of Applied Probability, 28(2):1 190-1248, 2018.
[58] Michael Mahoney and Charles Martin. Traditional and heavy tailed self regularization neural network models. In International Conference on Machine Learning, pages 4284 4293. PMLR, 2019.
[59] Charles H Martin and Michael W Mahoney. Implicit ceff-regularization in deep neural networks: Evidence from random matrix theory and implications for learning. Journal of Machine Learning Research, 22(165):1-3,, 2021.
[60] Charles H Martin, Tongsu Serena Peng, and Michael W Mahoney. Predicting trends in the quality of state-of-the-e art neural networks without access to training or testing data. Nature Communications, 12(1):1-13, 2021.
[61] Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 75(4))667-766, 2022.
[62] Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences, 115(33):E7665- E7671, 2018.
[63] Xuran Meng and Jianfeng Yao. Impact of classification difficulty the weight matrices spectra in deep learning and application to carly-stopping. Journal of Machine Learning Research, 24(28):1-40, 2023.
[64] Preetum Nakkiran. Learning rate annealing can provably help generalization, even for convex problems. OPT2020 Workshop, 2020.
[65] Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 1995.
[66] Quynh Nguyen. On the proof of global convergence of gradient descent for deep relu networks with linear widths. In International Conference on Machine Learning, pages 8056-8062. PMLR, 2021.
[67] Guillermo Ortiz-Jiménez, Apostolos Modas, Seyed-Mohsee Moosavi, and Pascal Frossard. Neural anisotropy directions. Advances in Neural Information Processing Systems, 33:17896- 17906. 2020.
[68] Guillermo Ortiz-Jiméne, Seyed- linearized neural networks actually say about generalization? Advances Neural Information Processing Systems, 34, 2021.
Seyed-Mohsen Mohsen Moosavi-Dezfood and Pascal Frossard. What can
[69] Samet Oymak, Zalan Fabian, Mingehen Li, and Mahdi Soltanolkotabi. Generalization guaran- tees for neural networks via harnessing the low-rank structure of the jacobian. arXiv preprint arXiv: 05392, 2019.
[70] Samet Oymak and Mahdi Soltanolkotabi. Overparameterrized nonlinear learning: Gradient descent takes the shortest path? In International Conference on Machine Learning, pages 4951-4960. PMLR, 2019,
14

---

[71] Samet Oymak and Mahdi Soltanolkotabi. Toward moderate overparameterization:: Global convergence guarantees for training shallow neural networks. IEEE Journal on Selected Areas in Information Theory, 1(1):44-005, 2020.
[72] Jeffrey Pennington and Yasaman Bahri. Geometry of neural network loss surfaces via random matrix theory. In International Conference on Machine Learning, pages 7798-2806. PMLR, 2017.
[73] Jeffrey Pennington and Pratik Worah. Nonlinear random matrix theory for deep learning. Advances in neural information processing systems, 30, 2017.
[74] Jeffrey Pennington and Pratik Worah. The spectrum of the fisher information matrix of single-hidden-llyer neural network. Advances in neural information processing systems, 31, 2018.
[75] Bartlomiej Polaczyk and Jacek Cyranka. Improved overparametrization bounds for global convergence of sgd for shallow neural networks. Transactions on Machine Learning Research, 2022.
[76] Maria Refinetti, Sebastian Goldt, Florent Krzakala, and Lenka Zdeborová. Classifying high- dimensional gaussian mixtures: Where kernel methods fail and neural networks succeed. In International Conference on Machine Learning, pages 3936-8947 PMLR, 2021.
[77] Dominik Schröder, Hugo Cui, Daniil Dmitriev, and Bruno Loureiro. Deterministic equivalent and error universality of deep random features learning. In Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 00285-30320. PMLR, 23-29 Jul 2023.
[78] Haozhe Shan and Blake Bordelon. A theory of neural tangent kernel alignment and its influence on training. arXiv preprint arXiv:2105. 14301, 2021.
[79] Matthias Thamm, Max Staats, and Bernd Rosenow. Random matrix analysis of deep neural network weight matrices. Physical Review E, 106(5):054124, 2022.
[80] Roman Vershynin. High-dimensional probability: An introduction with applications data science, volume 47. Cambridge university press, 2018
[81] Zhichao Wang and Yizhe Zhu. Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networks. arXiv preprint arXiv:2109.09330 2021.
[82] Alexander Wei, Wei Hu, and Jacob Steinhardt. More than a toy: Random matrix models predict how real-world neural representations generalize. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 13549-235585 PMLR, 17-23 Jul 2022.
[83] Christopher Williams. Computing with infinite networks. Advances in neural information processing systems, 1996.
[84] Greg Yang. Wide feedforward or recurrent neural networks of any architecture are gaussian processes. Advances in Neural Information Processing Systems, 32, 2019.
[85] Greg Yang and Edward J Hu. Tensor programs iv: Feature learning in infinite-widhh width neural networks. In International Conference on Machine Learning, pages 11727-11737 PMLR, 2021.
[86] Yaoqing Yang, Ryan Theisen, Liam Hodgkinson, Joseph E Gonzalez, Kannan Ramchandran, Charles H Martin, and Michael W Mahoney. Evaluating natural language processing models with generalization metrics that do not need access to any training or testing data. arXiv preprint arXiv:2202.02842. 2022.
[87] Jacob A Zavatone-Verr William L Tong, and Cengiz Pehlevan. Contrasting random and learned features in deep bayesian linear regression. Physical Review E, 105(6):064118) 2022.
[88] Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Gradient descent optimizes over- parameterized deep relu networks. Machine learning, 109:467-492, 2020.
15

---

A Additional Related Work
Nonlinear Random Matrix Theory and Random Feature Regression. The limiting spectrum of CK with random input dataset has been investigated by [73 13:: whereas [57] (30] studied the spectrum of CK with more general input data. This spectrum is actually a deformed Martenko-Pastur
distribution [30], which becomes a deformed semicircular distribution [8]] when h >> n. The largest
eigenvalue of the CK matrix has been studied in [4], and the spectrum of the NTK was analyzed
in [1 30) As an application, random feature ridge regression was fully determined by the limiting
spectra of CK or NTK: [57 33 53
I. All of these results belong LWR.
Global Convergence of GD for Ultra Wide NNs. A recent line of work has shown the global convergences of the learning dynamics of gradient-based methods in certain overparameterizze regime, e.g. 128 27 70 66 54 75 [8] We refer to Table 1 of [75 as a summary of these recent results. Most of the theorems in the literature require > n, which implies that the NTK is almost static during training. while 770 66) can consider LWR under some specific assumptions. Recently, [8] established a new criterion for the convergence of GD which results in the global convergence of general NNs with finite width h and d 2 n.
Beyond NTK Regime. Under the proportional limit, the initial kernel regression can only learn a linear component of the target 35 Thus, it is reasonable to consider the cases beyond the NTK regime. To this end, [2 39] considered the dynamics of NTK throughout training while 12 7 have shown a second-order approximation of NTK, outperforming the initial kernel. In addition, there are many theoretical works analyzing when NN outperforms the initial kernels in some specific settings: (5T) proved two-layer ReLU NN that is shown to beat any kernel method; [42] verified a two-layer CNN with some simple dataset can outperform the initial NTK for image classifications; (6) showed a NN can escape the kernel regime by only taking one specific large gradient step; [24] showed a specific gradient-basse training can even learn polynomials with oow-diensiional latent representation.
Evolution of NTK and Alignment in NNs. The feature learning can be characterized by the evolution of the kernel during training (3 68 55 5 (56.. Specifically, (55) studied the hard-margin SVM for "after kernels" which are the CK and NTK matrices of trained NNs. One of the effective ways of depicting how the kernels evolve during training is to capture the evolution of kernel alignment [10 78 5 (56). Kernel alignments between kernels and training labels essentially reveal how the NN accelerates training (78). Also, several papers showed that the top eigenfunctions of the kernel align with the target function learned by the NN
67 68). This becomes an efficient way of analyzing how NNs learn features through a particular gradient-based optimization.
Large Learning Rate Regime. As mentioned earlier, the large learning rate may contribute to feature learning. The benefits of large-learning-rate training have been studied from different aspects
[2 64 15 3). Specifically, (46) observed that training dynamics with large learning rates differ from the small learning rate regime. where the latter regime exhibits monotone and fast convergence of training loss but may not generalize well on test data. At the early phase of training, 4T showed using lower learning rates may result in finding a region of the loss surface with worse conditioning of kernel and Hessian matrices. In (55), the after kernels of NNs trained with larger learning rates generalize better and stay more stable. 49 raised a ''catapult mechanism", where gradient descent dynamies converge to flatter minima for extremely large learning rates. There transition as a function of the learning rate, from lazy training to the catapult regime. Section 4.2 illustrates similar transition our situations.
Haivy-tailed Phenomenon. The heavy-tailed phenomenon has appeared in many places in deep learning theory; [58 59) observed that many state-of-the-art pre-trained models obtain heavy-tailed weight spectra. More precisely, these spectra have a "5+1" phase transition which relates to different degrees of regularization of the NN. With this heavy-tailed seuf-regularization theory, (60] further showed how to distinguish well-trained and poorly trained models by a power-law- tion. (63) classified trained weight spectra into three types: Marčenko- Pastur law, bulk with (few) outliers, and heavy-tailed spectra. We extend this classification to both weight and kernel matrices in [igure Additionally, similarly to the discussion 4.3 [X
approxima-
showed that the difficulty of the classification problem is related to the emergence of heavy-tailed spectra in weight matrices. This
16

---

heavy-tailed phenomenon can be used to construct metrics for evaluating the generalization of NNs 60 [6], and early stopping of NNs avoid over-fitting [63]
B Additional Empirical Results
There are different parameterizations for NNs at initialization. The orders of the output of NN are distinct in different cases 25 88). This affects the size of stable and non-trivial gradient steps. The distance of trainable parameters from initialization determines whether the NN learns any features from the training data (6, Figure 2]. The performance of networks with different initializations indicates whether the NN belongs to the kernel regime or not [85]. Unlike the NTK parameterization,
the mean-field parameterization 662 20 and maximal update parameterization learning.
85
tend to be feature
For all NNs in our experiments, we apply normalized and centered nonlinear activation function such that Assumption 3.2 holds (R][ for N(O, 1)) because we can exclude large but trivial spike in the initial spectra of kernel matrices. In all architectures of NNs we considered, we [T] with standard Gaussian initialization. Specifically, all entries of W and in 7 at initialization are i.i.d. standard Gaussian random variables. For all experiments on synthetic datasets, we use standard Gaussian random matrices generate the training data X. In addition, consider the training label noise defined in Assumption 3.3 as E ~ N(O, IO..
remove the bias term of each layer and apply the NTK parameterization
B.1 Further Discussions on Real-world Data Experiments
For the first experiment in Section 5 fix the small-CNN architecture, which is similar to the VGG model, and CIFAR-2 dataset, and vary the methods of optimization of training. Corresponding to Figure 4 the training and test accuracy histories for three cases are shown in Figure 6 Here. in Figure 6(),, we used GD with learning rate 8 x 10-3; we used SGD with learning rate TO-3 batch size 32 and momentum 0.2 in F Figure ((c) employs the same learning rate and batch size as 6 (b) but employs Adam optimization.
0.80
0.75
0.9
0.70
0.8
3 0.65
0.60
0.55
train
0.6
train
0.6
0.50
0.5
100
200 spoth
300
400
0
50
100
150 epoch
200
250
300
20
60
80
100
epoch
(a)
GD
(b) SGD
(c) Adam
Figure 6: Training/test accuracy epochs for small-CNN model CIFAR-2 with different optimizers.
In the experiment of the transformer language model in Section 5 we fine-tuned the BERT model with SGD for a binary classification the Sentiment]4g dataset. We apply this transformer and fine- tuning to sentiment analysis for social media data. For fine-tuning, the learning rate 0.003, the batch extract the spectral properties
size is 64 and the momentum 0.8. The purpose of this experiment of pre-trained models and the evolution of the CK spectra over fine-tuning. Combining Figure 5 the following Figure F exhibits the evolution of the CK spectrum during fine-tuning. Similarly with Case 4 in Table I the CK spectrum of this pre- pre-trained model (red histogram in Figure possesses heavy-tailed distribution, which suggests this transformer has received adequate training. From (c), we observe the bulk distribution first shrinks then extends during fine-tuning This similar to evolution the first eigenvalue of in 3 Accompanied by this spectra evolution, there is a rapid transformation of the features through fine-tuning, linking the features in pre-trained model with features in the new dataset. We expect that further spectral analysis will elucidate the feature learning in this kind of transformer (82).
Figure a to
17

---

100
100
100
Intaa,, test acc: 49.3
Inttaa, test acc: 49.3
Initial, test acc: 48.3 84,4
81.875
83.225
Q-O plot
Q-O plot
Q-O plot
10a
sepoch
12-epoch
10a
10-3
20-3
16a Inibill
20
16a Initial
107
Initial
CK spestra
SKeeprt
CK spectra
Figure The spectra CK the BERT model on Sentimentl dataset at epoch 1, and 12.
B.2 Additional Results for Cases in Table
To complement the findings in Figure I data and two-layer NNs. In this section, we will always use the same architecture and dataset as the typical examples in Table
and Section now present additional results synthetic
Norms of the Change. Based on Figure the trajectories of the weight and kernel matrices are quite different among all cases in Table I Hence, for all cases in Table we record the changes in the weight and NTK matrices in both Frobenius norm and operator norm Figure
1 Wo- WWll, KTK
w Wells Vd
K. NTK
F. and || K K, NTK
at every epoch through training. The changes in Figures a) and (b) are much smaller than in the last case. Figure has significant changes both norms after training, which is consistent with the heavy-tailed phenomenon in Figure c). The global optima of the last case is far from the initialization.
Following the settings of Theorem 4.2 in Figure 9 we compute the differences between initial W. and final W, in Frobenius norm, operator norm and 2, co-norm. Empirically, Figure 9 shows
Wo W WolFF,
I l.l.
W W.ll2,0 +(1)
(16)
Vd
as n Co with n/d - 71 and N/d 72, where is the final time for GD. Here, the entry-wise 2-00 matrix norm is defined
M 12,000 = max
m
ISEN
for any matrix M E R Xxd with the i-th row m. E Rd and Sis N. Notice that
||
M S |
(17)
Similar observations for CK and NTK in both Frobenius norm and operator norm are also apparent in Figure 9 which empirically verifies the invariance of the spectra after training. Here, we fix the aspect ratios and let n grow to keep the NNs residing in LWR. For different n's, we repeat the experiments 10 times for average. In each experiment, in Figure a)), the test losses are almost the same for different n 's. Figures bb)((d) empirically validate Corollary 4.3 Moreover, the observation that A W o W Willg ll and Va IIW W.
train the NN until it converges. As shown
(1) may suggest that Va (Wo W.) )) is low-rank perturbation. That is, training in LWR may be transferring some low-rank structures to the weight spectrum. This low-rank perturbation can help us better understand the spectral evolution during training. Notice that these norms of the change are different from ultra- wide NN 28). Similar phenomena can be also observed in Figures [0' and (b). Analogous result with different and a* is exhibited in Figure 30 in Appendix C In addition, cases when NNs can outperform lazy training as defined by (6). In these experiments, we compare the performances of GD, and SGD with small or large learning rates, and lazy training as n + oo. Each time, we take 10 trials to average. We observe that SGD with large learning rate (green line) can asymptotically outperform lazy training.
Figure (0(c) further investigates
18

---

15.0
12.5
150
10.0
operator norm NTK Frobenlus norm
NTK
operater NTX trobenius norm
norme
110
NTK Frobenius narm Weight aperatornom
nore
100
7.5
Weight operater norm
Weight operator norm
5.0
Weght Froberius
5-
Weight frobenius nor
50
Weight Frobenius
2.5
0.0
2000
4000
6000
8000
1000
2000 opoch
3000
4000
0
10 epoch
15
20
epoch
(a) Case
(b) Case 2
(c) Case 4
Figure 8: The evolution of the changes in probenoo//robenuus norms of the weight/CK/NTK matrices through different training processes. Each case corresponds the case Table Case 3 exhibited in Figure c) below.
25-
19-
200
100
230
138
(a) Train/test losses
(b) Change in weight
(c) Change in CK
(d) Change in NTK
Figure 9: Performances NNs and changes different norms for weight kernels, when d/n 0.6 and N/n = 1.2 are fixed as n is growing. The activation is normalized tanh and the teacher model is (=) = o "(3 æ) where o" is normalized softplus. We average over 10 trials in each case. All these curves are almost flat, which indicates these values are not growing with ?/ and 72- Here, the second figure from left, we normalized all weights W with V7
to observe T6.
Case 1. Comparing with Figure 15 Figure shows no aligmments with training data in GD training. This corresponds to the performances Table T The performance of Case not as good as the prediction risks in Figure since Figure 11 suggests that no feature learning appears after GD training. Gradient descent requires the weights to converge to some global minima close to initialization, thereby offering guarantees for lower generalization errors. Next, Figure 12 further presents more results on GD training and indicates more evidence of kernel regime in Case This shows that, from spectral point of view, the NTK is invariant/static through training. Based on Figures aa) and 12 we can empirically verify Corollary C.3 stated Appendix C Globally, the spectra of W, K KCK and KINT are not changing over training as n//d -> 71 and N/d - 72- The Marčenko-Paasee Pastur law; the initial spectrum of NTK under
initial spectrum of weight Wo converges proportional limit has been studied by 1 30). Figure (2 demonstrates the global convergence for
GD under the proportional regime, as proved in Theorem 4.2 We can observe this global convergence
even for SGD, Case in Table although we do not have proof for it.
Case 2. As a complement, Figure 13 exhibits the spectra of W. K-CK and K NTK for Case 2 in T The phenomena are similar to Case 1. This observation provides evidence that all results and conjectures in Section 4.1 can be extended to SGD training with sufficiently small learning rates, which is subject future work. Analogously to Theorem 4.2 conjecture that the global convergence when training both layers of NN with SGD still holds in this proportional limit. The proof strategy for global convergence, in this case, can again follow 669
70 Once we have the invariant
global spectra in Corollary 4.3) we can apply the nonlinear RMT limiting spectra under LWR.
73
57
13
30
to characterize the
Next, in Figures 14 and 15| we present spectral properties for Case 3 in Table where a spike detaches from the bulk after large-step-size training. Notice that Figures ( and [4[a) imply that the bulk spectra for weight and CK remain unchanged over training despite the emergence of spikes. This is not true for NTK by observing Figures[4[] and (c). The Frobenius norm of NTK changes significantly during training and is not O(1) anymore; the spectra of the first component
Case 3.
19

---

2.2
107
0.8
2.0
1.8
0.6
1.6
100 OKsl
500 brge
550 small
0.2
1.2
Lazy training F125
W
MTX knwwr tound
0.0
Sample Dimersion
Semple Dimension
(a) Change in weight.
(b) Change in CK.
(c) Prediction risks vs. 71-
Figure 10: Change between initial Wo and final step W, operator Frobenius norm and oc)- norm when d/n 0.5, N/n 0.8 are fixed as n - oc. We average. (h) Change of K in operator norm and Frobenius norm. (c) Prediction risks for lazy training defined by (6). GD with I/ = (1) (red), SGD with ((1) (blue dol) and
NNs by SGD with ry = 2.5
trials to
and 722 2.5. The black dashed stands for the kernel lower bound given by nonlinear part of teacher model.
(green), m
Inttal
Trained
Singlar voctor
0.05
0.05
0.05
Pf
0.00
-0.05
-0.05
00.0
butil Trained
Signal the teacher modal
Traniig labas
Training label
Figure 11: Alignment with leading PCs of trained weight, CK and NTK matrices in Case of Table
of the NTK shrinks after training (Figure 14),,, which indicates NNs converge flatter minima. This resembles the catapult phase in [49 for extremely large learning rates. Figure 15(a) shows the convergence rate for SGD in Case Empirically, observe that the training loss will not monotonically decrease when using a larger learning rate than Case 3, which may be analogous to catapult phases from 44
Case Additional results for Case 4 Table are shown in Figure Unlike the strong alignments in Case 3 (Figure 15). Figures [6 (e)-(() do not exhibit strong alignments for the leading singular vectors or eigenvectors. This may due the heavy tails present after training, with the other large spikes detaching from the bulks are also important for generalization. A similar phenomenon can be also seen in Figures 23 and 25 in Appendix B.6 where we have comparable performances to Case in Table
Heavy tails are essentially power laws. To measure how "heavy" the spectrum is, 58
provide
estimates on the power law of W. Consider the empirical spectrum of W as ((x) x. for large x and some positive constant a. The spectrum with heavier tail has a smaller value of Figure 17(a) shows how a evolves through training in Case 4 of Table [ As decreases, heavy tail in the spectrum of the weight matrix emerges Figure Figure [7]a) we introduce two
metrics show this evolution: Weighted Alpha â σλ and Log a-norm log [E X²
where xy S ... S Y₁ are the eigenvalues W T Remarkably, Figure a7 indicates the spectra change dramatically at the early stage of training, which matches the observation from fine-tuning via BERT on real-world data in Figure in Section These metrics are applied to measure the tails in pre- trained models [60].
B.3 Additional Results for the Emergence of A Spike
Figure we show the training dynamics for SGD training with a larger learning rate in the example of Figure Figure((bd)) b-dd). Here we consider 24 which belongs to the orange region Figure bb-d), where the spike and eigenvector alignment emerge. Figure T8 presents the details of the training dynamics of the NN in this case: the largest eigenvalues of CK and
As complement to section 4.2
20

---

Q-Q plot
Intisl
10
0.5
0.6
Trained
1.25
Trained
10
0.4
1.00
lssee
10
0.0
R2 Sccnn
10
0.2
10-5
100
0.00
0
2000
4000
6000
8000
(a) Weight spectra
NTK spectra
spech
Figure 12:
Performances of Case Table
(a) The initial trained spectra the first-hidden layer
(c) The initial and trained spectra of empirical NTK matrix defined by (5) Q-Q subplot shows these two spectra are almost the same. Training and test losses epochs for GD (right).
Q-Q plot
Initial Trained 1.5
Q-Q plat
Initial
Q-Q plot
Initial
Trained
0.6
Trained
Triired
2.0-
0.4
1.0
10
1.5*
S Initial
Initial
Initial
0.2
0.5-
0.0
3
4
0.0
0.0
2.5
5.0
7.5
10.0
12.5
15.0
0.0
0
6
(a) Weight spectra
(b) NTK spectra
(c) CK spectra
(a) The initial and trained spectra of the first-hidden layer W. The initial and trained spectra of empirical NTK defined by (5). (c) The initial and trained spectra of empirical CK defined by (4)
Figure 13: Spectral properties for Case 2 in Table 1
NTK both increase and the losses first increase and then drop. Figure we empirically justify that the phase transitions we presented in section 4.2 for SGD can be also extended to full-batch GD) cases. We can also observe phase transitions for test losses and R² scores when we are gradually increasing the learning rates. Parallel to these, a spike also appears outside the bulk distribution, which corresponds to feature alignments Figure (f))
B.4 Multiple-index Examples for Heavy-Tailed Spectra in Section 4.3
Figures 17(b) and (c) are additional results for Figure 3b)) in Section In this experiment, we consider o ReLU, n 5000, h 2500 and d 1000 for NN Comparing with the teacher model p used in Table we employ the multiple-index teacher model (8) with k = 5 and o² = a. We trained this student-teacher model using GD (17 = 15), SGD (n = 7.25 and batch size 8), and Adam
0.007 and batch size 16) for training this NN, respectively. Similarly with Figure []] correspondingly, we observe invariant spectrum, bulk with one spike, and heavy tails after training respectively. Heuristically, to learn this f*, the weight W of NN should gradually align with the feature space U spanned by B,'s. Hence, to study feature learning, we can apply principle angles measure the alignment between W and U. Consider the eigen-decomposition of W W.W,- Σd 'aaaas with A 2 λ 2 ... 2 Ad. Figure 3 shows the heavy-tailed part (the eigenspace
span (v; H aligned with after training. which shows how features are learned in the heavy-tailed spectra. Remarkably, the test errors for training processes with SGD and Adam are even smaller than || and || where P denotes the orthogonal projection onto the nonlinear part of the function w.r.t. Gaussian measure. Thus, we experimentally showed that NNs with heavy-tailed spectra can obtain feature learning and generalize better than the other two cases. Another example is exhibited in Figure 20 In this case, k = 5 and there are five leading outlier eigenvalues in the spectrum the trained weight matrix, along with a heavy-tailed bulk. Interestingly, Figure 20 justifies that the eigenspace of these five leading outliers is strongly aligned with features B for S i This indicates that heavy-tailed spectra with large spikes may have correlation with feature learning and good generalizations.
21

---

1.50
plot
ndial Trained
1.25
Q-Q plot
Intia Trained
0.6
Trained
Traine
200
1.00
norms
NTK Frobenius norm
0.75
10 Initial
Weight operator norm
0.50
20
Weight Frobenlus
0.25
0.0
0.00
0.0
2.5
10.0
20
spoch
(a) Spectra of weights.
(b) Spectra of the first part in NTKs.
(c) Norms of change vs. epochs.
Figure 14: Additional performance for Case in Table (a) The initial and trained weight spectra. Notice there is one outlier after training, while the bulk remains invariant. This analogous the behavior spectra in Figure b). (b) The spectra of the first part in (5) at initialization and after training. The orange arrow points out the outlier of the spectrum. (c) The changes W. Woll. I Woll K KNNK
KNTTK and
NTT at each epoch throughout the training process.
8.075
2.000
2.025
0,025
0,000
0.000:
10-0
0.,05
-0.025
10.
Intal
1ital
-15
Tranzed
Palted
40
60
80
#20ch
Tathing Ibbe
(a) Losses epochs.
(b) y/ of CK.
(b) vs. NTK.
Figure 15: (a) Training/test losses and R² scores each epoch of training in Case 3. (b) Alignment between training labels y and first PC of trained/initial CK. (c) Alignment between training labels and the first PC of trained/initial NTK. We the setting Case of Table There strong alignments between kernels training labels as stated Section
B.5 Training Only the First Hidden Layer
We now present additional results when only training the first layer of NN with Adam. This result resembles Figure 29 in the next section. In section 4.1 Corollary 4.3 shows that under LWR with sufficiently large width h. the limiting spectra of W W,, K CC and K-TTK are essentially the same as those of the corresponding initial matrices if train only the first layer with GD (see (TO).. Figure 21 further investigates these phenomena when training only the first layer with Adam. In when
particular, the Q-Q plots show the invariant spectra of weight, CK, and NTK matrices the training loss is approaching zero. Here, in Figure 21(c), we only consider the first component of NTK since the gradient is only taken with respect W Another observation is that the smallest eigenvalues of the initial and trained NTK are both bounded away from zero. This is crucial for the proof of the global convergence shown in Appendix C
B.6 Adaptive Gradients
Inspired by Case 4 in Table we show the spectral performances of adaptive gradient (AdaGrad) The performance of this method matches Case 4 in Table where we can also easily observe heavy tails and detaching spikes after training, especially in Q-Q subplots. This suggests that adaptive optimization more likely to yield heavy-tailed distributions in trained NNs. Besides, analogously to Figure [6] there is no strong alignment in the single leading PC of the weight or kernel matrices after training Figure 23
in Figures 22 and 23
B.7 Different Global Minima and Alignments
In this section, to distinguish the different alignments Case 3&4 of Table [ we introduce fol- lowing two simulations with slightly different optimizers to get quite different spectra and alignments among leading PCs after training. In the first experiment, Figures 24 and 25 we first take Adam with large stepsizes for few steps and then mall-steppiie SGD for convergence. In this scenario,
22

---

0.5
Q-O plot
hitil
Q-Q p ot
Inibal Trained
brain
0.6
Trained
0.4
Trained
300
loccars M
score
Initial
0.2
Initaa
0.1
0.0
8
10
0.0
10
35
0
10
epoch
(a) Spectra of weights.
(b) Spectra of NTKs.
(c) Losses vs. epochs.
0.10
Inttal Trained
Intal
Triied
Tre Vegulte vector
0.05
0.05
0.05
of
0.00
0.00
0.00
-0.05
-0.05
-0.05
0.10
-0.10
-0.05
0.00
0.05
0.10
Signal in teacher model
Triniing labels
Traininn labels
(d) First PC in trained & initial weights.
(c) First PC in trained & initial CKs.
(f) First PC in trained & initial
NTKs.
I (a) The initial and trained weight spectra. Notice that there are several outliers after training, while the bulk has heavier tail. (b) The spectra of the NTK (5) initialization and after training. (c) The test/training losses and R score (green line) at each epoch throughout training process. (d) A Alignment between the leading PC of the weight matrix and the signal /3 in the teacher model before (red) and after (blue) training. (c) Alignment between the leading PC of the CK matrix and the training labels y before/after training. (I) Alignment between the leading PC of the NTK matrix and before/after training.
Figure 16: Additional performance for Case 4 in Table
0.35
Intill
0.8
0.30
Trained
125
4189
0.7
Adam 9
9
0.20
0.6
CD 6
10
0.15
59
0.10
25
0.00
(a) Evolution of metrics.
(b) Evolution of PC angles.
(c) Weight spectra.
Figure 17: (a) The evolution of power a, weighted Alpha and Log a-norm (several metrics of power law tails: see (60] during the training process Case of [able (b) Evolutions the first PC angle between
feature subspace U = span (8,n (3,) of the multiple-index, model (8} and the eigenspuce spanned by top 100 of eigenvectors of W. W. during training with Adam (blue solid line), SGD red dashed line) and GD (green dash-dot). The final test error is 0.33865 and the R² score is -0.71065 for GD. The test error is 0. 10814 and the R2 score is 0.45373 for SGD, where one spike emerges in the weight spectrum after training. The test ervor is 0.08672 and the It? score is 0.56195 for Adam. (c) Initial and trained spectra for weight matrices when training with Adam (blue solid line in (b)). Heavy tail emerges in this case.
can get heavy-tailed distributions after training, and the phenomena are essentially the same as Case 4 of Table [ There is no strong alignment for the first leading cigenvector, while useful features may be learned by a few top eigenvectors in the heavy-tailed spectra after training. In the second experiment, we directly apply Adam with small initial learning rate for training. In contrast, the results of this case, presented in Figures and 27 are similar to Case 3 in Table 1 The test loss and R² score are close to previous examples, though slightly worse. As explained in the previous section, because there is only one spike appearing outside the bulk after training in the second case, the leading PC is highly aligned with the training dataset structure after training and the feature learning mainly stems from the outliers in this situation. This interprets the strongly anisotropic structures in trained spectra of NNs in the second case 67 68 78). These two different spectral properties reveal significant differences between the global minima of these two training processes and different evolutions of the
23

---

2401 come
502 00565
10
12-
15
cpech
120ch
c2ech
(a) Losses.
(b) Amax (CK).
(c) Amax (NTK)
Figure 18: The training dynamic when training neural networks with SGD and learning 24 in the example of Figure (b-c). The learning rate we chose here above threshold we showed Figure
We use the same architecture, dataset, and teacher model as in Section our paper. The batch size Is 32. (a) The evolution of the training and test emros during training. (b) The evolution of the largest eigenvalue of the CK matrix. (c) The evolution of the largest cigenvalue of the NTK matrix. This regime corresponds to the catapult phenomenon 49
in
rital Zained
trained
0.6
3.55
2
3.50
-3.05
38-1-
12
00
0.30
0.75
0.10
verning potne
Weight spectrum
wegge
Losses learning rates.
(b) Weight spectra with small
Weight aligaments with small
learning rate.
learning rate.
0.7
ril
Zained
Nottrre wecher
3.00
0.69
-2.35
6.1
101
0.0
-5.30
-2.25
0.00
3.05
0.30
(d) 122 vS learning rates.
(e) Weight spectra with a large learning rate.
(f) Weight alignments with a large learning rate.
Figure 19: Grid Grid search different learning rates when training NNs with full-hatch GD in same setting as Case in Table [ (a) Final test losses when varying learning rates. (d) H² scores when varying learning rates. For all these learning rates, we did not observe heavy-tailed spectra. (b-c) present the spectral behaviors for the smallest learning rate used in (d). (e-f) present the spectral behaviors for the largest learning which still ensures the convergence of GD. In this case, analogously to the SGD in Scction 4.2 observe an outlier in the trained weight matrix and strong alignment with the spike.
spectra in NNs. Remarkably, based on the different spectral behaviors in trained weight and kernel matrices, these two experiments exhibit disparate features learned by distinct training procedures. Hence, analyzing the spectral properties in trained kernel matrices beneficial for clarifying what features our NNs have learned during the training processes.
C Proofs of Results in Section 4.1
C.1
GD Analysis at Early Phase
From
10,, the GD process with learning rate 17 > 0 can be written by
W i+1
W, n. GG, where
(18)
G. nV dh ( T (o @ d'w,.///V/ X
(19)
Vh
for f N, where E Ix", Following [6] Appendix B), in this section we prove the control for gradient step G, For simplicity, denote fx( == fe, (X)
1 v (W x //vd) for t = N.
vh

---

0.14
Q-Q plot
0.10
5.0
0.08
2.5-
0.6
2.5 5.0 Initial
7.5
0.06
0.4
average angle,
principle angle
second angle
0.02
third angle
fourth angle
0.00
40 Weight spectra
60
500
epoch
Figure 20: (Left) Initial and trained spectra for weight matrices when training with Adam. Five leading spikes emerge in this case. (Right) Evolutions of the angles between the first four PCs of W W. and feature subspace U span [3, V of the multiple-index model (8) during training with Adam. Here le = 5. The final test error 0.33865 and R score 00.71065 for GD. The test error is 0.01681 and R score is 0.9154 for Adam.
0.25
Q-Q plot
Q-Q plot
2.0
Q-Q
Initial
Trained
Trained
Trained
Trained
Trained
Iraind
0.20
1.5
0.15
1.0-
10
0.10
Initial
Initial
Initial
1
0.05
0.0
2.5
5.0
7.5
10.0
12.5
(a) Weight spectra
(b) CK spectra
(c) NTK spectra
Figure 21: Additional spectral performance when training the NN by Adam with small learning rate = 0.001, where 2000. h 3000, = 1000, 0.3 and batch size is 100. In this simulation, we only train the first hidden layer W.. .. The activation function normalized softplus and the target function is normalized tanh. The final test loss is 0.36219 and A² score is around 63. 73.70%
Lemma C.1. Under the same assumptions as Lemma 4.
we have
((ooWox/vall CvT < -o-
IP (lall 2 CVv) < on
for some constants C, > 0 only depending de Xo, n. and 72.
Proof. Due to 30 Lemma D.4.], we can directly obtain that
P ((orwax\vaal C' VT < 2e
Here the fact that both Wo and X are i.i.d. Gaussian random matrices. Then by Assump-
tion 3.1 conclude that control a (W, XX/V)) Recall that Assumption 3.3 implies that
x'() + Hence, by Lipschitz Gaussian concentration inequality [S Theorem 5.2.2), each entry of f* (X) has independent sub-Gaussian coordinates, whence
get [Xf < C.n
with probability at least 1 2ne -en for some constants e, C > 0. On the other hand, [e]: = E; are
i...d. centered sub-Gaussian noises with variance oZ. By 80 Theorem 3.1.1], we have
P (|||) S / 2 2 exp
Cn
K
where the constant K is the sub- Gaussian defined by K max; |eill Hence, combining all things together, we obtain the second inequality of this lemma.
25

---

0.25
0.10
Q-Q plat
Inibal Trained
Q-0 plat
Inibal Trained
Q-9 plot
Intial Trsined
0.6
Talred
0.20
Taired
500
0.08
500
0.15
0.06
0
0.2
0.05
0.02
0.0
10
35
20
0.00
10
20
30
40 0
0.00
20
40
60
80
100
(a] Weight spectra
(b) CK spectra
[c] NTK spectra
Figure 22: Additional performance for AdaGrad with learning rate 1j 0.5, where 2000, h 1500, d 1000, de 0.3 and small batch size is 8. Activation is normalized softplus and target normalized tanh. The final loss 0.23555 and R score around 0.76249. The black lines in the Q-Q subplots the line
Z.
0.10
Intial
0.3
titial
0.3
Initial
Mngular vettor
Tained
T Rained
Tained
0.05
Pc NIK
0.00
0.0
0.0
-0.05
0.1
0.2
-0.05
0.00
Signal
Training
Training
(a) First in trained initial weights.
(b) First PC in trained initial CKs. (C) First in trained & initial
NTKs.
Figure 23: Alignment between the Ieading PC of the weight/kemel matrices and the signal f or the training labels y before/after training for experiment Figure [2 Analogously to Case 4 in Appendix B there is no strong alignment in the leading component of weight/kernel matrices.
0.5
0.20
Q-Q plot
Initial Trained
Q-Q plot
Initial
Q-Q plot
Initial Trained
Taiind
10
Tained
0.15
Tained
0.3
0.4
Intttal
0.2
Initial
10
Initial
5
0.2
0.05
15
10
1'5
26
25
30
10 15 [c) CK spactra
20
25
(a) Weig spoctra
[b] NTK spectra
Figure 24: Additional performance for Adam with learning rate = 0.09 and 4 epochs. then SGD with learning rate !! x 10 and 100 epochs. where 2000, / 1500, d 1000, a, = 0.3 and the batch size 32. We train the NN until the training loss less than 10 10 The activation is normalized softplus and the target is normalized tanh. The final test loss is 0.22511 and R² score is around 0.77462.
Lemma C.2. Under the assumptions of Lemma given any fixed N and learning rate (1), the weight matrix after gradient steps W.
defined in (18) satisfies
C I exp (-cn),
I - Woll 2
(20)
Vn
for some positive constants c, C only depending n. de, Xo, "/2 and 72-
Proof. Denote (I) = (xx) - #iæ which the nonlinear part of and μn = E[za Thus, E(o1(2) 0 for ~ (0, 1). Based on this, can further decompose the gradient G₁ into
G.
111
(y f((X))
X
nv dh (v(y f((x)) oo w.x// x/va)) XT.
(21)
nV dh
A
n
26

---

0.10
0.10
Intiaa Trained
0.10
Intial Trained
0.05
0.05
CK
Pc of NTK
Top steplee
0.00
0.00
0.00
-0.05
-0.10
nital 7ained
-0.30
00.0
0.05
0.05
Sigral in the teacher mocal
Taikinn labels
Traniing label
(a) First PC in trained weight.
(b) First PC in trained CK.
(c) First PC in trained NTK.
Figure 25: Alignment between leading of the weight/kemel matrices and signal 3 or training labels y before/after training for the experiment in Figure 24 This is analogous to Case in Appendix B.2
0.5
0.6
Trained
Q-Q plot
Trained
0.4
Trained
20
Q-Q plot
Inibal Traihed
0.15
Trained
Q-Q plot
Inital
Trained
10.
0.3
0.4-
Initial
Initial
Initial
0.2
0.05
0.0
8
10
0
10
15
20
0.00
10 (c) CK spectra
15
(a) Weight spectra
(b) MTK spectra
Figure 26:
Additional performance for Adam with learning rate n 0.002 and 700 epochs, where normalized softplus and the
2000, 1500, d 1000, a 0.3 and batch size 64. The activation target is normalized tanh. final test loss 0.23954 and R² score is around 0. 7.76777 orange arrows show the positions of the outliers. Spectra behaviors in this case differ from Figure 24
At first, consider t = 0 and bound the spectral norm of W, By assumption, we know ||/||| S Vh. Due to Corollary 7.3.3 [80]. have
n )
IP
1+
2 exp (-en).
(22)
Therefore, by
21 we can control A and Bd separately. Notice that, as rank-one matrix,
P1 X (llvl + [lfo(x))l)
|
|| | = |
/n dd vn
vh
P1 X
||>|||
Vh vn [|y/| +
||
oW /owox//aal)
Vd
Vh
Hence, by Lemma C.1 and 22 can easily claim that || s C/Vn with probability at least - e en for some constants e, > O. On the other hand, since v (y - fe(X)) is rank-one and o' o' 11 with [a'(x)) Ao, similarly obtain
B F
<
e (y f(X (
X
|X||
nv dh
XX|| y ll,o(x)))
||||||
max 12
a (W X Vd)
ny hd
f1 A
X [
|
|
a
vn
Vd
Vh
Vn
Vh
As A we can apply Lemma
C.1 and 22 again to conclude 20 for 1.
For general I, we apply induction. We assume that after the 1-th gradient step with v O(1), Eq. (20) holds for some constants C, c > 0. Following ( Lemma 16), we now show that the similar piobbbboyayylity statement also holds for W
(+1 (for some different constants c', C'). Firstly,
27

---

0.10
nital
0.075
0.050
Trained
0.050
Trained
Top shipler
0.000
0.000
-0.05
-0.025
0.000
-0.10
0.10 -0.075
0.10
0.05
0.00
Sigral in teacher modil
Trakiin labels
Training labels
(a) First PC in trained & initial weights.
(b) First PC in trained & initial CKs.
(c) First PC in trained & initial
NTKs.
Figure 27: Alignment between the leading PC of the weight/kemell matrices and the signal 33 or the training labels y before/after training for the experiment in Figure 26 We can observe strong alignments, in this case,
comparing with Figure [5 because of outliers above Figure 26 These kernel alignments induce anisotropic
structures the kernel matrices during training (78).
following the same argument as
71 Setion 6.6.1], we know that
(x)] - (x
+ (X) - fo(X))l A
|
11.o(
|m||
IW - Woll
(23)
dd
Note that W. =o
with high probability by the induction hypothesis. Hence, by Lemma C.I and 22). we have (X) s Cvn with high probability, Indeed, the difference between
fx( and fo(X)
significantly negligible comparing with the initial value fx(X). Similarly with
Ao. A satisfies
(llll) +Ix(())II) 10
A
S ||
Analogously Bd, we have
11 +
1 1 vd vh vn
B
(lell) + (x)
n
Thus, Lemma C.I
22 and (3
ensure that
C"
C"
P
(11')
exp (-dn), PP [P||B
2
Vnl
S exp (-p))))),
for constants é', C > 0. Since W
1+1 WolF < |W Woll + || +
1 by
induction hypothesis, we can conclude that (20) holds for the (t + 1)-th step with some constants C, O, which are different from the constants at the t-th step.
As corollary,
17 also deduce the following norm bounds:
exp (-cn), (IW. Woll2,00
C S exp (-en). Vn
IP W W.
Vn
Lemma C.2 and the above bounds empirically verified by Figure 28 a) for t 3. Not only upper bounds, this simulation also shows that at early phase [W Woll. [W, Woll and |W, Woll2,0 are all of the same ((1/vn) order.
As remark, from the bound of the second term of (23 can deduce that the change of the output of the NN satisfies
[f(()) - fo(x)) Vn'
0, any x ~ N' I) and any finite time t. In other words, when 77 (11), the change of the output of the NN the early phase (i.e. t + negligible and its order is O
for some t-dependent constant C' >
28

---

C.2 Proof of Lemma 4.1
In this section, we complete the proof of Lemma 4.1 We first mention the empirical validation of Lemma 4.1 Figure 28 Notice that the changes Frobenius norm for W and KCK are exactly (n/v/n) and ((1/n), respectively. The operator norm of K matches with Lemma 4.1 while the Frobenius norm of the change decays slower than the rate (1/nn. Additionally, in the simulation, we use v No I), which indicates that our assumption for Lemma 4.1 can be weakened.
Weight differences
10"1
CK differences
NTK differences
811m))
10-1
(X--OKj)
10-3
(14 Wil-
10-2
(W/ Wil..
10
10
JRTXs - 1/KK
1X10 3x 107
2x100
3''10
2x107
3w10
Sample
Sample
Sample
(a) Early phase for Wt.
(b) Early phase for K
(c) Early phase for K
Figure 28: Empirical validations for Lemma and Lemma C.2 at f Here a. = 0.2, activation a a normalized ReLU and the target function a is normalized tanh. Fix d/n = 0.6 and N/n = 1.2 as n is increasing. At each dimension, we take 25 trials to average. Norms of the changes for KG K (c) Norms of the changes for KNIK K
(a) Norms of the changes for W Wa. (b)
Proof of Lemma 4.7 Lemma C.2 directly validates the control of this result, we now present estimates for CK and NTK. Based on Section 6.6.1]. we have
W.
Woll p- By virtue of
W. Welle.
(24)
Vd
We apply the mean value theorem to obtain this incquality. Recall the operator norm bound for Gaussian random matrix X in (2)) We know
|X/Vdl| S 1 + /ni with high probability as
n/d + 71 Hence, with the help of Lemma C.2 we can claim
[W xx /Vd) - o (W,x/V/ /Va)
s CC vn)/vm.
with probability at least - exp (-cn), for any fixed finite [n]. Similarly, we can control the change in the Frobenius norm as follows:
X2 d
(W, X /Va) a < (w.x/v/)
|x| W. w < C1 + /T)//n, (25)
with probability at least 1 exp (-en). Therefore, can control the change in the CK matrix in the Frobenius norm by the following inequalities:
K-K MOKK-KKK
(WW,X/Val (o -WW.X//VD-oWxxx//)) + 1 (ww. (/o(
h 1
< h -owp///))oooo///// ( / //
x σ W (wWx//V)--oo o
Therefore, by 24 25 and Lemma C.I
we can claim that there exist constants c, C > 0 such that upper bounded by C/n the LWR. F
with probability at least exp cn).
K-C
K
Now we consider the change in the NTK matrix during training. Since the empirical NTK can be decomposed into two parts, one which is exactly the CK, it suffices to consider the change of the first part of the empirical NTK. Recall that
K, : =
-
x x 7
1///
( (
diag(u:)20
( w
(
29

---

Following the notation in 771. we denote (WW, =
E Raxhd with /vd E Raxd Hence, K; = J(W.)JJ and
J(w,) = Th (iia(oo(X""::/V))))) w./vdj/x
w:
K, Koll [[ J J(Wo))
S 2 (W -
J(W a)ll + - JWW
(26)
0 By 71 Lemma 6.6], we know o Koxk TT is upper bounded by some constant C > with high probability. Then, apply the inequalities from Lemma 6.5 of [71 obtain
IJ(W.) J(W
( ((W T diag(u)²
-OW
x x
h
d
(d -(WWX/// diag(v) Vh
d(WoX/Va) I d' d' o' dW /Va
x x2./V V
max 1E(m
x [Vdl
max sE|n]
h
< 2 | h
I. - ooll max (Ein)
- le./va] vall).
(27)
where the last inequality is due to the mean value theorem, the uniform bound on o", and the assumption the second layer v. Notice that Gaussian random vectors satisfy
max iE(n]
x 11
2ne
(28)
as n/d + 71 and h/d + 72. Thus, with
22 and Lemma C.2
we obtain
P [15(W.)) - J(W ol C1a(
I 4ne
where constant C relies on the number steps t. Hence, by (2)). we can finally bound in norm the difference between the initial and the trained NTK matrices at the early phase (t is finite).
Corollary C.3. For any fixed t E N, E [d] and [n], denote Xt, v and ph the i-th, and k-th eigenvalues of W. W. KCC and K respectively. Then, under the assumptions of Lemma 4.1 we have
IN 271, 1/ - 2l lite -12l +
f. almost surely LWR. Consequently, the eigenvalues of 1 W W., KCK and KNNI are the same as corresponding the eigenvalues of initial W Wo. K and K respectively.
This corollary is a direct outcome of Weyl's incquality from Theorem A.46 [8]. Consequently, this corollary concludes that for any fixed 2 O, almost surely, the limiting spectra of W, W., KCK and K-NTK are the same as those of W W. K-C and K in LWR. This corollary claims that not only does the bulk of distributions stay identical to the initialization, but also that any cigenvalues stay the same as at the initialization. This shows that the smallest eigenvalue of K-NTK has the same lower bound NTK in the early phase of training.
C.3 Global Convergence for GD Under LWR
In this section, we study the final stage of 10 as training loss is approaching zero and prove Theorem 4.2 Figure 29 shows that the spectra are unchanged globally, even after training this case. In Corollary 4.3 we confirm this observation for the weight, CK, and NTK matrices via Frobenius norm control. In the simulation, the second layer is initialized
(O, I), which is more general
than our assumption on in Theorem 4.2
30

---

Q-Q plot
Initial Trained
2.5
Q-Q plot
Initial Trained
Q-Q plot
Initial
Trained
2.0
Trained
Trained
10
0.3
1.00
1.5
0
0.75
0
Tnltial
1.0
0.50
0.1-
0.5
0:25
0.0
0.0
0.00
10 NTK spectra
Weight spectra
(b) CK spectra
Figure 29: The initial and trained spectra (until training loss less than 10 when using GD only for first layer (h 3000, 2000, d 1000): (a) Weight spectra. (b) K- spectra. (c) K spectra. The final R
score is 0.55964 and the test loss is 0.44724. The activation is normalized ReLU, and the target Sigmoid.
Proof of Theorem 4.2 Recall the Jacobian matrix J(W) defined in the proof of Lemma 4.1 and
the definition of a based (11) in Section 3 Denote the event
A
X < 2 + vn) vd, max | s 2d, m (J(W o)) 2 20:
iE ml
28 and Theorem 2.9 of [8], we have P (A) 2 2n 2ne - n n-nn//// 7/3 for some By
constant > 0 and all large n in LWR. In the following, conditionally on event A, we will apply
Theorem 6.10 of 71 to obtain the global convergence. Conditionally on A, Lemma 6.6 of implies
[[((W) S A |0 x x/Va
2X
V
(29)
for any W. Define 3 = 2A (1 + MT) Moreover, in terms of
27
we can verify the Lipschitz
property for the Jacobian matrix as follows: conditionally on A,
2 Vh
W W
(30)
for any W, W E hxd Therefore, conditionally on A. J(W) is a L-Lipschitz function with respect to W where == 28 To complete the proof, it suffices to investigate the smallest singular value of J (W) when W is in the vicinity of W o. Recall e(W) || fw(()]l. Notice that for any unit vector u E R", we have uT, fw,(X) vt 2: [w X Vd)u, where w the E
vh
o for i E [N]. Consider event B /o s CVn for some universal i-th row of Wo
constant C > O. Lemma C.I proves P (B) > 1 cm By the assumption of v, know each entry v; is a sub-Gaussian random variable with sub-Gaussian norm at most 1. Then, according to Hoeffding's inequality, conditionally on the event B, we have
(1
P
m X/Vajuu 21) 2exp (-cl²,
for every 2 0 and some constant c > Let = 2/72. Considering an 1 nett N of the unit sphere gn-1, we can get
(I.w.. Z Vn) S SP <2(2max MEN
(X) 2 v < 29" (-en) S 2e e'n
(31)
for some constant é > Hence, based on Lemma C.I and (31). we can obtain E(Wo) S Covn with high probability for some universal constant Co > 0. Let us denote this event as C: (E(W)) S Coyn). Define R = 4e(W,))( o)/ar For any W in a ball of radius R centered at W.- have I W [| < R and ([] -[WW))l o)ll < LR, conditionally on event A. Thus, by (30). on event A C, the smallest singular value miin ( (J(W)) of the Jacobian matrix J (W) can be bounded by
T minn (J (J())) 2 amin Fmin(J[WW (J(W [I((W) - J(W
83 ((W))
8C8
71
2 2a - LR > 2a
> 2a
Vh
72
31

---

for some universal constant C' > 0 and sufficiently large n. d, h. Notice that here constants C, 3, and do not rely on 72. Therefore, there exists sufficiently large ?" O such that for all 72 2 y", we
have 2a 8CB
21 2 In other words, when h is sufficiently large but still in the same order as
V
72
m (J(W)) 2 conditionally with (29) and (30), conditionally on C A. all the assumptions of Theorem 6.10 by [7]] are satisfied
and d, for all W W I R. we have min
C COA Combining
when W Woln I R. Therefore, when the learning rate S _ min {1, LR we can get
4ez
(12 14) for all N, conditionally on C E A. Both events A and C occur with high probability
and only depend on initialization Wo. X and y. Hence we complete the proof of this theorem.
Therefore, it suffices to require ?) S n²/3² to conclude that [2). 13 and (14) hold with high probability. This completes the proof. Moreover, (4) further shows that for all t N
Notice that since 1/2 2 7" is sufficiently large, T.R 2 2CB V
a
Wo Well s S CVn+ o1,(
(32)
where we again apply Lemma C.1 in the following way:
((W)) CV
for some constant C > 0 only depending on 7 72, oz and o².
(i Rec (Ac.
64-
63
66.
Sample Dimersioon
Sample Dimeciioe
Sample Dome
(a) Changes for Vd W.
(b) Changes for CK.
(c) Changes for NTK.
Figure 30: Mcasuring the change for the weight, CK, and NTK matrices when training NN with (T). We fix d/n = 1.2 and h/n = 0.6 when n is increasing. Here, o is normalized ReLU and the target is normalized tanh. The largest
6400 and the learning rate 17 3.0 for all training processes. We train each neural network until the training losses approach zero. Each experiment repeats 4 times. In (a), consider the changes Va IW.
Woll va
I[W
Wollp- and Va IW, W Wollg.
As a corollary, 14 controls the deviation of the final step weight from the initial weight. This is empirically shown in Figure a))a), which shows that Va IW W oll Va [, [Wl-Wollr- and IW W. oo are ((1) when trainable parameters are convergent. This implies that the final W, still close to the initial weight Wo. o. even after training. Consequently, with this observation. we can prove Corollary 4.3 in the following.
ve
Proof of Corollary 4.3 Based on (32.. we know Ve IW - Well < Co holds with high probabil-
ity for some universal constant Co > 0. Conditionally this event, can then estimate changes in
CK and NTK after training. The method is analogous to Lemma 4.1 For CK, we employ Lemma
and 25 to get
KCK KCK
F
2
h //x//// /Va) OW +
212(1 + n1)
2
W - 20 2C
vn W. - Well
h
h
21 + Cn (2r(1 + vo)Co + CVT)- = Oar(1).
22
32

---

Hence, this shows control of the change for the CK matrix after training, compared with the initial CK.
Let us denote u E IR as the i-th row of Wt, and æy as the j-th column of X. Additionally, by
Assumption 3.2.
we know that
(o'(z) - of (v)) S Aa y/.
(33)
for any y E R. For NTK. by modifying (27 one can deduce that
[ o(Wo)) -
Σ [[](t) J (20
1=1
91 1
x 2 1:
diag (d'wwt//Vdd-d
F
(10) )
2
h
||- o..(()
diag
h
i=1
(10) X1 + \n1)
(wr mo?)a;)) +1ar(1)
h
( X2 (1 + vm)) W; - W Woll 7 + oa.(11) - 12 + C + od,F(1),
h
where (i) is because of 80 Exercise 6.3.3] and the assumption on v, (ii) is due to (2). (tii) is due to the definition of Frobenius norm and (33). and (iv) is due to (80) Exercise 6.3.3] and (22). As result, from 26 we can finally conclude that | KCK
OI, n/d - m and
h/dd + 72-
As for the limiting spectra of weight and kernel matrices, since we know that
W. Wolle, KCCK K ||K.TKK K 1 Oap(1),
we can automatically apply Corollary A.41 of [8] This directly implies that the limiting empirical spectra of 1/ W, W., K-C and KN are the same as the limiting spectra of π, W KNT respectively, as n/d + 7i and h/d + 72 (see Figure 29
I Wo. K-K and
Further Simulations for Changes in Norms. The simulation of Figure 30 empirically coin- cides with the norm bounds in Theorem 4.2 for different norms. Because of (7), it suffices to only consider the Frobenius norm of the change for each matrix. As remark, Theorem 4.2 requires "2 to be larger than some threshold y* to ensure norms of the change throughout train-
ing. However, Figure 30 indicates Theorem 4.2 still holds even when * is small i.e. the width h is not very large. Here in Figure 30 Y2
2 = 0.6. Figure 31 suggests that similar re- hold for SGD when training the first layer. This is also akin to Figure 21 Moreover, we further conjecture that similar results to Theorem 4.2 and Corollary 4.3 will hold even when training both layers 33. Denote the second layer at step t
sults to Theorem 4.2 and Corollary 4.3
Woll. W. Woll V IIW Wall,, K. KNTKI ||. and ||u poll/||e0) are all @(1) in LWR. Since voll = O(Vh), we observe that |,,,-val - = ((vh) h)
by v, Then, indicated by Figure 32
I|W
va
| K ||. K KCK
CK
K.
in this case. Meanwhile, unlike Corol- p is still upper bounded by constant. One possible explanation is that in this case the change in the second layer v, is much more significant than in the first-layer weight Wt. hence the NTTK matrix may change a lot in this general case.
lary 4.3 Figure cannot verify that K K ||
Similarly, Figure 33 shows norms of the change when training the NN with SGD for both layers. Here we use the same batch size 128 and learning rate 7) = 1.0 for all experiments when varying the dimension but fixing the aspect ratios 71 and 722 All the observations are similar to the results of Corollary 4.3 except the Frobenius norm of the change for NTK. Figure 33(c) indicates that
KIK KNIK
will not O(1) anymore, which fluctuates more this case.
33

---

22
023
180
075
6.0
0.2
033
399
600
18500 2008
6000
15009
Sam
(a) Changes for W.
(h) Changes for CK.
(c) Changes for NTK.
Vd
Figure 31: Measuring the change for weight W., K and K matrices when training NN with 10 for the first layer W. using SGD with batch size 200. We fix d/n 0.6 and h/n 1.2 when is increasing. Here a is normalized softplus and the target is normalized tanh. The largest n 11000, dg = 0.3 and the learning rate is 7 = 3.6 for all training processes. We train each neural network until the training losses approach zero. Each experiment repeats 15 times.
66
PS PS--91
68
(a) Changes for W.
(b) Changes for CK.
(c) Changes for NTK.
Figure 32: Mcasuring the change for W., K-CK the second layer v. and NTK when training NN with 33 for both layer W. and es using GD. We fix d/n = 0.5 and h/n = 0.8 when n is increasing. Here, o is normalized softplus and the target is normalized tanh. The largest is 11000, ar = 0.3 and the learning rate is 3.6 for all training processes. We train each neural network until the training losses approach Each experiment repeats 15 times.
68.
1976 1Tk
200
200
2208
3000
2006
Sangee cenanioe
sangle Dinar So0e
Aingle ceneresae
(a) Changes for
// W.
(b) Changes CK.
(c) Changes for NTK.
Figure 33: Measuring the change for W. KCK the second layer and K when training NN with SGD for both layers W, and v,, We fix d/n 0.6 and h/n 1.2 when increasing. Here, is normalized ReLU the target is normalized tanh. a, and the learning rate is 7) 1.0 for training processes. We train each neural network until the training losses approach zero. Each experiment repeats 12 times.
34


Reviews:
[{"review_id": "PN2EhUnhnx", "reviewer": "Reviewer_jctx", "summary": "Considerable attention in deep learning theory has been given to the analysis of neural networks in the kernel regime, which neural networks approach as they approach infinite width. While this analysis has led to insights, there is an important gap between the theorized generalization performance of neural networks in this infinite width regime and the practical performance of finite-width neural networks, which typically perform better. \n\nThis paper studies simple, two-layer neural networks in the more realistic *linear-width regime* (LWR), in which sample size, input feature dimension, and the width of the layer approach infinity at comparable rates. In several analyses, the authors target the following question: *How do the spectra of the NN's weight and kernel matrices evolve during the training process?* The authors compare to the kernel regime as a benchmark and analyze changes in spectra that occur through training with different gradient descent algorithms . \n\nFor their first simple experiment (Section 3), they find that the spectra of the weight, conjugate kernel (CK) and neural tangent kernel (NTK) matrices remain invariant during training for gradient descent (GD) and stochastic gradient descent (SGD) with a low learning rate. This is consistent with the Lazy Training (LT), and indicates that the network is not outperforming a kernel machine. However, for SGD with a large learning rate, a spike in the spectral distribution emerges after training, which indicates improvement over lazy training. They note that this spike is consistent with previously published results that demonstrate that learned features may be indicated by an outlier in the spectra with a large singular value.\n\nThey classify the spectral distributions into three categories: invariant bulk, spikes outside the bulk, and heavy-tailed distributions. They prove that the invariant bulk phenomenon is expected under certain assumptions. They then empirically analyze the spike phenomenon, and demonstrate that as the learning rate increases, there is a threshold at which the spike emerges, which corresponds with greater alignment and suggests feature learning. They then provide an explanation for the relationship between heavy tailed spectra and better generalization performance. Lastly, they perform some analyses on CNNs trained on more a more natural dataset, CIFAR-2, as well as BERT with fine-tuning on Sentiment140. ", "questions": "- What can we understand about neural networks, generally speaking, by looking at the spectra of their weight matrices and kernels? Why use this approach, rather than analyzing other properties of the weights and network structure?\n- How do the results found here advance our theoretical understanding of how deep networks learn?\n- How can these results be used in practice? \n", "limitations": "The authors note the limitation of the FWR. They do not state limitations of the spectral approach to analyzing networks. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The authors tackle an important question, which is to understand how the structure of neural networks change through the training process. To get a handle on it, they analyze the spectra of the weight and kernel matrices of simple neural networks on controlled datasets. They find some interesting phenomena that relate the spectral properties to learning rate and generalization performance.", "weaknesses": "The paper would benefit from more effort put towards making clear the meaning and significance of the experiments and results. Why look at the spectra? What is the significance of the three classes of spectral distributions that are observed? (The answer to the latter question is in the paper, but it is somewhat buried. This should be put front and center and made very clear). \n\nThe figures and captions in the paper could be clearer. Oftentimes, variables are used as labels. The reader forgets which variables correspond to what. It would be better to use English names, or more descriptive properties. \n\nThe specific primary findings should be stated more clearly. The statements provided in the introduction are general, and don't provide much insight into the conclusions. It takes work for the reader to tease out these conclusions. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive evaluation and thoughtful feedback. Per the reviewer's suggestion, we will provide more detailed captions for the figures to help readers. We will also clearly state our primary findings and conclusions in the introduction. We have summarized this work's main contributions and insights in the global rebuttal for clarification. In the following, we address the technical comments and questions. \n\n1. **Q: Why look at the spectra? Theoretical understanding?** The spectral properties of the neural networks are a key component of understanding the explainability and dynamics of feature learning in deep neural networks. We believe that a spectral analysis of weight and kernel matrices can be seen as one of the first, fundamental steps in understanding the performance of neural networks. \n\n      There are many previous works that use the spectral properties of either the weight or kernel matrices of neural networks (there are many references [28, 55-57,60,70,71,82]). Intuitively, we can think of the eigendecomposition of the CK and NTK as describing the variance of the underlying feature space, akin to PCA. The emergence of an outlying eigenvalue tells us that the total explained variance of that feature space is becoming concentrated along one direction. In our alignment experiments (see Figures 1(b) and 2(a)), we show that this direction corresponds to the target space of the neural network. This alignment could be thought of as an operant definition of feature learning itself. \n\n     While there are alternative approaches to the spectra, this analysis is simple and theoretically motivated. Furthermore, in Figure 2(b-d), the emergence of the outlier and strong alignment are highly related to the choice of learning rates, which indicates that the spectral properties we studied here help us understand how we choose the correct optimizer and hyperparameters for efficient training. In conclusion, our empirical results of spectral evolution provide a promising way for future theoretical understanding of how deep networks learn features during training.\n \n2. **Q: Applications in practice?** Based on our empirical results, we emphasize that understanding the spectral properties of the weight and kernel matrices is essential to understanding feature learning. For example, in Figure 5, we computed the top two principal components of the CK matrix in the BERT model before and after fine-tuning, and we can see how the model forgets the pre-trained features and learn new features in the fine-tuning process. In addition, our results indicate that choosing optimizers and hyperparameters affects the spectral behavior and generalization of neural networks. Finally, [55-57] found a correlation between the occurrence of heavy tail spectra and generalization. Our experiments rule out that this relationship is causal. Our example in Figure 18 empirically shows that we need to analyze additional spectral properties, e.g., how well features align with the eigenspace in the heavy-tailed part, to test if the neural network generalizes well or not.\n\nWe are happy to clarify any concerns or answer any questions that may come up during the discussion period."}, {"review_id": "catjHL7HHl", "reviewer": "Reviewer_m9fj", "summary": "This paper analyzes the spectral properties of feedforward neural networks under a student-teacher setting. A linear-width regime is considered, where the sample size and network width grow comparably with the input feature dimension. The paper shows that the spectra of weight and kernel matrices are invariant under training, and that outliers occur in large step size regimes. Heavy-tailed spectra are also discussed with their relation to generalization. Evidence from both synthetic and real-world datasets is shown.", "questions": "- In the proposed LWR, how should one understand that the input feature dimension $d$ goes to infinity? This feels somewhat counterintuitive, compared with e.g. the infinite-width regime.\n- To what extent does the observed spectral invariance depend on the LWR? I would assume this does not occur under the infinite-width regime.", "limitations": "n/a, theoretical work", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Proposed LWR as a novel perspective for analyzing feedforward networks, and presented an interesting analysis of NN spectral invariance, potentially providing insights into NN feature learning.\n- Evidence from both synthetic and real-world datasets enhance the argument.\n- The presentation of the paper is clear.", "weaknesses": "- The proposed LWR feels under-explained. Please see \"Questions\". Otherwise I think this is a solid papar.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive evaluation and thoughtful feedback. In the following, we address some of the technical comments and questions.\n \n1. **Q: Why feature dimension goes to infinity?** The LWR, where the input feature dimension $d$ goes to infinity proportionally to the sample size $n$, is a classical setting in high-dimensional statistics, and provides important insights for real-world datasets. This is in contrast to the infinite-width regime, in which we are already in the asymptotic limit for width at first.  \n    We believe the LWR is a better approximation of real-world datasets and practical neural networks compared with the infinite-width regime since the dimension $d$ in real datasets is very large and we should allow the dimension of the feature space to grow with the sample size. In real networks, $d$ is not infinite but can depend on $n$: with a larger $n$ we may be able to use higher-dimensional features. For instance, the feature dimension of the ImageNet dataset is usually cropped to $d=224\\times 224$ for learning.  \n    \n2. **Q: How depends on the LWR?** Our theory of spectral invariance depends on the LWR, but it can be extended to the infinite-width regime easily. In fact, in the infinite-width regime, we can have sharper estimates of the changes of the weight matrix and each neuron as long as the width $h$ is much larger than sample size $n$. The main difficulty of our theory is to extend the result of the spectral invariance to LWR which is a more realistic regime for neural networks. Besides, although we can show that each neuron is close to the initialization under the infinite-width regime, we cannot get a limiting eigenvalue distribution of the weight matrix when $d$ is fixed and $h\\to\\infty$. Only when we consider the LWR, we can claim that the limiting eigenvalue distribution of the weight matrix is an invariant Marchenko–Pastur distribution through GD training with a small learning rate.\n\nWe would be happy to clarify any concerns or answer any questions that may come up during the discussion period.\n "}, {"review_id": "0Z7zcEPqHc", "reviewer": "Reviewer_DCEt", "summary": "The paper carries out an empirical study of the spectral properties of finite-width neural network kernels after training, and compares them to their infinite-width counterparts to assess if there is an improvement. They study these changes with respect to various hyperparameters such as learning rate and optimizer. The results suggest that for large learning rates, where the linear dynamics is not a valid approximation, they show that feature learning emerges by inspecting the distribution of the final spectra. They identify two different spectral property that distinguish feature learning from lazy regime: emergence of low-rank spikes in the spectrum and heavy-tailness. They explore when these properties emerge and provide theoretical arguments for these observations.", "questions": "1. It is hard to parse Figure 1, first panel shows weight spectrum for GD with LR 5, second panel shows CK spectrum for SGD with LR 22 and last one shows CK spectrum for ADAM. Something like CK spectrum for Case 2, 3, 4 would be more helpful.\n\n2. In Figure 4, is there a reason why GD does not show the same properties as SGD? With large enough LR, could GD also show the same heavy-tailness? Also, it is not clear how adaptive optimizer qualitatively differs from SGD.", "limitations": "A limitations section is missing.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and easy to follow. The experiments are complimented well with theoretical arguments, and the discussion is sound. The Supplementary Material also includes nicely organized discussion of previous works and additional experiments and discussion.\n\nIn summary, they introduce the well-known fact that the small-learning rates make a NN with NTK parameterization stuck around its initialization and derive theoretical results to show that the bulk of the spectrum remain invariant under lazy training. In my opinion, the main contribution is that they find two different ways feature learning can happen: emergence of low-rank outliers in the spectrum with large learning rates with GD optimizers and heavy-tailed spectrum with adaptive optimizers.\n\nFurthermore, for GD/SGD, they make a theoretical connection for how low-rank outliers emerge with a phase transition in RMT (ref. [9] in main paper.)\n\nFinally, the relation of heavy-tailed spectrum to generalization has been discussed.", "weaknesses": "1. While the paper brings valuable insights, it is highly restricted by the 2-layer neural networks and synthetic dataset with a particular target. It would be useful to perform experiments on simple real datasets such as MNIST to show that spikes related to number of classes also emerge in the spectrum. For example, in neural collapse phenomenon, one can see that weight matrices of last layer acquire low rank structure that can be tracked to the number of classes.\n\n2. Such experiments can also shed light on how full-batch vs. stochastic GD differ in terms of the after-training spectrum. Currently, I do not think there are any insights related to how SGD helps feature learning since GD experiment does not have a large LR analogue.\n\n3. Is there an intuitive explanation how adaptive optimizers cause heavy-tailed spectrum in CK? More discussion around this phenomenon could be very helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive evaluation and thoughtful feedback. In the following, we address the technical comments and questions.\n \n1. **Q: Real datasets such as MNIST?** We will present experiments of the MNIST dataset and show that the spikes in the kernel matrix are related to the different classes of the dataset. In the attachment of our global rebuttal, we present the top principal components of pre-trained transformers before and after fine-tuning. Similarly, with the neural collapse phenomenon, we can see that these principal components correlate to the different classes after fine-tuning training. This should be consistent with the MNIST experiments the reviewer describes in the question.\n    \n2. We thank the reviewer's suggestion for additional experiments to compare full-batch GD and SGD. In the attachment of the global rebuttal, we present additional simulations for GD training with different learning rates and the maximal learning rate we can use in GD training. Especially for GD training with very large learning rates, we can observe the emergence of an outlier in the spectra after training, but the performance is not as good as SGD training with large learning rates. And in this case, the training dynamic with GD is unstable. \n\n   Besides, we also run a grid search for the learning rate when training with GD/SGD, and we do not figure the same heavy-tailed spectra as adaptive methods showed. We can observe the emergence of heavy tails if we apply sufficiently large learning rates for GD at the early stage of training, but we must adjust the learning rate later to ensure the convergence of the training loss. This is the heuristic reason we need adaptive optimizers to obtain heavy-tailed spectra in weight and CK matrices after training.\n\n3. So far, we do not have a sufficient understanding of what causes a heavy-tailed spectrum after training. But we believe that the heavy-tailed spectrum in CK is due to the heavy-tailed spectrum of the weight matrix. And its heavy-tailed spectrum indicates that the weight matrix is moving far away from the initialization which possibly leads to good feature learning. Moreover, a large learning rate at initialization is important to feature learning, and adaptive optimizers will then guarantee convergence to a global minimum. Our experiments in Figure 18 provide more insights into the adaptive optimizers, heavy-tailed distribution, and how this heavy-tailed distribution related to the multiple-index teacher model. We will move this part to the main part for more discussion.\n\n4. In Figure 1, we only presented part of the spectral results for different cases. In Appendix B.2, we additionally present all the spectra of the weight matrices, CK, and NTK matrices before and after training in Case 1-4 separately. We will mention this in the caption of Figure  \n\n5. We will present a section about limitations in the conclusion part of our paper for completeness.\n\nWe would be happy to clarify any concerns or answer any questions that may come up during the discussion period.\n\n"}, {"review_id": "3rRzTVItET", "reviewer": "Reviewer_C57a", "summary": "This paper examines the spectrum of the conjugate kernel and neural tangent kernel of real networks before and after training. Based on analyzing the spectrum in some experiments, several observations about how the spectum evolves during training are made. In particular, there seems to be different phases depending on the optimizer used and learning rate which effect how the spectrum changes from initialization. Understanding this spectrum-evolution phenomenon may lead to a better understanding of how neural networks train, and shed light on important questions like understanding differences between ADAM vs GD vs SGD.", "questions": "* Is there any possibility to describe how the outlier eigenvalues are evolving in time during training? Particularly when there is only one outlier eigenvalue, this would be an interesing bit of theory to try and understand.\n* Is there any relationship between the spectrum of the CK/NTK and the spectrum of the Hessian?", "limitations": "* One limitation that always comes up in these limiting type things is the question of how large real neural networks have to be for the theory to actually work. Some discussion or experiments specifically addressing this (e.g. showing the error in the predictions as a function of network size) could help explicitly address this.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The review of existing literature is really well done (there is even a more detailed review in the appendices). This puts the current work in context quite nicely and makes this paper a very accessible place to start learning about these spectral questions.\n* The experimental plots are nicely done and clearly showcase the behaviour being discussed (the Q-Q plots also help a lot with the understanding). There are also numerous other experiments in the appendix. This again makes this paper a great resource for these kinds of experiments. \n* I am not an expert on this area, but to me the way the different phases of spectral evolution are laid out (i.e. what they call \"Invariant Bluk\" vs \"Bulk+Spike\" vs \"Heavy tail\") is new to me. (I have heard of this outlier thing happening before for the Hessian...I am not sure how related that is). To me this paper looks like the first step towards building a phase diagram of how choices about optimizer/trianing effect the spectrum which would be a really useful thing to have.\n* I found the conclusion about learning rate (e.g. Figure 2) the most impressive conclusion here, and to me this may be the most actionable of all the various phenomenon observed here. This is a nice conclusion to draw and could pave the way for more results in this direction.", "weaknesses": "* One might complain that the theory developed here is a bit weak, in that the theoretical results are all inequalities which are (probably) not sharp. However, since the paper is mainly empirical I actually think these theoretical results are reasonable here.\n* I am not an expert on this area, so I was not able to tell how much of the results here were expected/known already. Hopefully at least one other reviewer is able to give that context. \n* It is not clear how robust the experimental results are: in the given plots the phenomenon seem to be happening as described, but maybe there could be some extra discussion about \"edge cases\" between the various phases and discussion about when things break down (e.g. when N is too small, or there are not enough examples etc)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive evaluation and thoughtful feedback. In the following, we address the technical comments and questions.\n\n1. Our experiments are robust to dimensions as long as the width $h$, sample size $n$, and feature dimension $d$ are large. Anecdotally, when these dimensions are around several hundred is enough. As in random matrix theory, when the aspect ratios are fixed and the dimensions are large enough, the eigenvalue distribution is close to the limiting law, and it is very stable. We will present additional experiments for different aspect ratios to show that our phenomena are stable for different aspect ratios when dimensions are large enough. \n\n2. We thank the reviewer for asking about the evolution of the outlier through training. We have not checked this evolution in the paper since we only present the emergence of the outliers after training. In the attachment of the global rebuttal, we present the evolution of the outlier during the training process when there is only one outlier eigenvalue. There are some interesting phenomena in this evolution. We observe the training error first increases and then decreases. We believe this regime corresponds to the catapult phenomenon in reference [47].  And the outlier for the kernel matrix first progressively increases and then oscillates around some value before convergence. We agree this will be an interesting theoretical direction to further explore.\n\n3. We thank the reviewer for asking the question related to Hessian. In fact, NTK has a close relationship with the Hessian with square loss function. There are two parts of the Hessian and one of these parts is the Fisher information matrix which has the same non-zero eigenvalues as the NTK matrix. This is another motivation for analyzing the spectrum of the NTK matrix through training. We can have a better understanding of the spectrum of the Hessian matrix and the loss landscape, e.g., the relationship between sharpness and generalization, during different training processes. \n\n4. We agree that our theory requires asymptotic limits. However, as classical results in random matrix theory, the eigenvalue distributions of random matrices will be very close to the limiting law even when the dimension is several hundred. We believe that there will be some **non-asymptotic** results for our empirical observations, i.e. our empirical results should hold practical neural networks whose widths and sample size are large but finite. Additional discussion and experiments (as dimensions are gradually growing) will be added to explain this problem.\n\nWe would be happy to clarify any concerns or answer any questions that may come up during the discussion period.\n\n\n "}, {"review_id": "TDOmCDT6hw", "reviewer": "Reviewer_zoyL", "summary": "This paper studies gradient descent training in single-hidden-layer neural networks in the linear-width regime (i.e., that in which the input dimension, hidden layer width, and number of training datapoints together tend proportionally to infinity). Its primary result is that the bulk spectra of the conjugate and tangent kernels do not change over training in this regime, but at large learning rates an outlier eigenvalue can emerge. ", "questions": "- In the Introduction, it would be nice to mention recent work on Gaussian equivalence for deep random feature models in the linear-width regime by Schröder et al., \"Deterministic equivalent and error universality of deep random features learning\" (ICML 2023) and Bosch et al., \"Precise Asymptotic Analysis of Deep Random Feature Models\" (2023). \n\n- The linear-width regime for deep Bayesian neural networks has recently been the subject of study in the community of researchers using tools from statistical physics to study deep learning, see e.g., Li and Somplinsky, \"Statistical mechanics of deep linear neural networks: The backpropagating kernel renormalization\" (PRX 2021), Zavatone-Veth et al., \"Contrasting random and learned features in deep Bayesian linear regression (PRE 2022), and Cui et al., \"Optimal learning of deep random networks of extensive-width\" (ICML 2023). I think it would be useful to at least comment upon the Gaussian equivalence results of Cui et al. as a point of reference for how different inference procedures behave in the extensive-width regime. \n\n- In Figure 2 (b-d) and elsewhere, it would be better to label the abscissa with $\\eta$ or \"learning rate\" rather than \"lr,\" for the sake of consistency with the text.\n\n- In Figure 1 and elsewhere, the histograms should be plotted using a log-scaled ordinate. As it stands, the outlier in 1b is nearly invisible. \n\n- Why use the single-index + quadratic model in eq. 8 rather than a (simpler) single-index model? \n\n- The BERT experiments in Figure 5 strike me as mostly decorative, as the setting is not commensurate with the earlier portions of the paper. What insights would the authors argue can be transferred from their MLP experiments to this setting? ", "limitations": "The paper does not provide adequate discussion either of the relevance of the linear-width setting for practice or of the limitations of their experiments. ", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The topic of how kernels evolve outside of the lazy regime is timely, and the RMT-inspired setup and results presented should be of interest to the community. The theoretical results appear correct, though I've not checked the proofs line-by-line. ", "weaknesses": "The manuscript presents a series of interesting observations, but to my taste it sacrifices depth for the sake of breadth. The results on heavy-tailed spectra are thought-provoking, but I think focusing on the emergence of spikes in the spectra would make for a stronger manuscript. If the authors could prove the existence of a BBP-like transition, they would have a strong paper. The current comment on Lines 247-248 comparing to [6] is not very satisfying. At the very least, a more detailed empirical investigation of this phenomenon would result in a more convincing story. \n\nIn this vein, the experiments are not very systematic, and as a result cannot clearly disentangle which changes in the setup are required to produce a given phenomenon. For instance, Table 1 contains various settings of batch size and learning rate, but nowhere are these parameters swept jointly over some reasonable range. I have in mind something like the parameters sweeps for learning single-index models performed in recent work by Atanasov et al., \"The Onset of Variance-Limited Behavior for Networks in the Lazy and Rich Regimes\" (ICLR 2023; note also that this relevant recent work is not cited). The goal here would be to establish under precisely what conditions a spike emerges. As it stands, the authors can make only vague statements. \n\nThe transition to using Adam for the heavy-tailed weight experiments is even more drastic: is it possible to observe this phenomenon using a non-adaptive optimizer? I think dissecting this phenomenon is more properly the subject of a separate paper, as it is currently not well-integrated with the other results. The discussion in Lines 286-291 is also rather heuristic, and I don't think the authors provided sufficient evidence for their claim that heavy-tailed spectra naturally emerge in feature-learning networks trained on complex tasks. Moreover, how does \"[t]his example explain[] why we can use the heavy tails to discriminate well-trained and poorly-trained large models?\" This requires elaboration, and more empirical evidence. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful feedback and additional references on random feature models and deep Bayesian networks under LWR. We will add these references and provide additional comments and comparisons. In the following, we address the comments and questions.\n\n1. The goal of this paper is to demonstrate empirically that this BBP-like transition we observed appears in neural networks under LWR, both for the weight and kernel matrices. While proving this theorem in particular setups is our ultimate goal, this paper is an empirical study that can help inform our and others' approaches to rigorously proving such a transition. We are familiar with reference [6] where the authors considered a two-stage training process and proved the BBP-like transition for the weight matrix, in a much simpler setting than the one considered here. We will provide a detailed comparison with this result in Section 4.2.\n\n2. Thanks for your suggestions on improving the readability of our captions and figures. We will change things accordingly. We'd like to point out that our experiments **are** systematic, as follows: we fixed the synthetic dataset, the teacher model, and the two-layer neural networks under the linear width regime; then, we used different optimizers (GD/SGD/Adam) to train the same network. For each optimizer, we tuned the hyperparameters, e.g., learning rate, to observe different spectral behaviors and accuracy after training. For example, Figure 2(b-d) presents how the spikes and eigenvector alignments emerge when we use different learning rates. Although we do not have a theoretical threshold for the learning rate when the spike emerges, we demonstrated these transitions and threshold empirically. In the attachment to the general rebuttal, we show how we tune the hyperparameter (a grid search for learning rate) for GD when presenting Table 1. In the revision, we will add more explanations on how to choose the parameters in the experiments in Table 1 of our paper. \n\n3. In the heavy-tailed section, our main message is two-fold. In references [55-57], the authors explored a correlation between the heavy tails in the spectrum and the good performance of the NN. They left causality as an open question. In our paper, we rule out general causality: Figure 3 (c-d) shows that there are cases when both the weight and CK matrices appear to exhibit power law in their spectrum, but the neural network does not perform well. Second, we argue that even if not causal, the relationship between heavy-tailed spectra and good performance can be subtle: we demonstrate an example where a neural network with heavy tails generalizes well on the test dataset in Figure 3 (b) and Figure 18. Additionally, we exhibit a case when neural networks benefit from heavy tails: When the teacher model is a multiple-index model with a high intrinsic dimension and the heavy-tailed part is really aligned with the multiple-index directions well, the neural network with heavy tails will generalize well (see Figure 18).  We will move the experiment of Figure 18 to Section 4.3 and clarify our explanations.\n    \n4. The reason we did not choose the log scale is that we want to show that initial spectra are Marchenko–Pastur-type distributions that one could not see in a log scale. We will adopt the reviewer's suggestion, though, and use log-scaled histograms to highlight outliers.\n\n5. The reason we chose the teacher model as a single-index + quadratic in Eq. (8) is to make it a little more complicated to learn. However, the performances and the spectral properties are similar to the case when the teacher model is just a single-index model. To make the paper more consistent and clearer, we will only present the cases when the teachers are single-index and multiple-index models in our reversion.\n\n6. The BERT example is important to the point we are trying to make, as other reviewers have noticed. The application of a transfer BERT model onto Twitter140 relies on a mathematical trick allowing us to pretend that this model is a 2-layer feed-forward (FF) network. We can think of any FF network as a composition of two functions, $f(g(x)) $, where $f$ is a classification head and $g$ is the composition of all the hidden layers; in this instance, studying the CK matrix for MLP is equivalent to studying the output of $g$. We can, in principle, do this for any architecture of neural networks. From these experiments, we want to highlight that understanding the spectral evolution of the conjugate kernel matrices for the large language model is crucial for analyzing the pre-trained model and fine-tuning process. First, in our experiments, spikes and heavy tails emerge in the spectrum of conjugate kernel---which is analogous to our toy models with synthetic datasets. Second, Figure 5 indicates that the evolution of the spikes in the conjugate kernel is closely related to the feature learning in the fine-tuning process. In the general rebuttal, we provide additional figures showing how these spikes forget the pre-trained feature and learn new features efficiently, quantitatively corroborating what Chen et al. have qualitatively shown in their preprint [CZZ23].\n\n7. We will provide an adequate discussion of the advantages and limitations of the linear-width setting. We will especially mention how the linear-width regime has been applied to study the Gaussian equivalence for deep random feature models and deep Bayesian networks. Our linear-width regime is more realistic compared with infinite-width neural networks and it has the theoretical benefit that random matrix theory can be applied in this setting. This linear-width regime is one of the ways to approximate finite but very large neural networks with very large datasets.\n\n [CZZ23] Chen, L., Zaharia, M. and Zou, J., 2023. How is ChatGPT's behavior changing over time?  arXiv:2307.09009.\n\nWe are happy to clarify any concerns or answer any questions that may come up during the discussion period."}]

Author rebuttal:
**Overall Summary:**

We would like to thank all reviewers for their evaluation of our work and their helpful comments. The key contribution of our work is to describe how the spectral evolution of both weight and kernel matrices changes during different training procedures. We focus on a simple two-layer neural network under a linear-width regime (LWR). These empirical results open a promising way to understand the training dynamic and feature learning of neural networks via random matrix theory. We believe our results will be impactful for both the random matrix theory and optimization communities by providing experimental evidence that can inform future theoretical studies. We briefly summarize our main results:
 
1. We observe the invariant spectra of weight and CK matrices through training with GD/SGD and small learning rates, which indicates the performance of this kind of training is still close to the kernel regime. We theoretically justify this observation for invariant weight and kernel matrices under certain assumptions by using the global convergence of GD under LWR.
 
2. We observe a strong alignment between the eigenvector corresponding to the outlying largest eigenvalue and the teacher model when training with a large learning rate. We demonstrate how to find the threshold for a large enough learning rate by showing a BBP-like transition in the spectra of the weight and kernel matrices when increasing the learning rate. The importance of this BBP-like transition is critical, as it is known to be indicative of feature learning, as explained in Response 1 to **Reviewer jctx**. This will also help us understand how to choose the learning rate to attain feature learning.
    
3. Our experiments rule out a causal relationship between the occurrence of a heavy-tailed spectrum for the weight matrices and a good generalization.  This complements the work of Mahoney et. al. [55-57] in reference where the authors had observed a strong correlation between the two; our work can be considered a limitation of the phenomena underlying that trend, while at the same time, through the example in Figure 18, we confirm the existence of a relationship and heuristically explain why this neural network benefits from heavy tails. 

4. We investigate the properties of weight and kernel matrices of larger models to demonstrate the phenomena observed in our toy examples. Therefore, our spectral analysis has the potential for high impact as we can investigate feature learning and training dynamics for different optimization algorithms using models used in applications.

5. Given **Reviewer zoyL**'s comments, we want to emphasize the fact that our BERT experiment is motivated by a connection to the spectral analysis we do on the feed-forward neural networks. Please see item 6. in our response to **Reviewer zoyL**. To reinforce our point here, we are including two PCA plots in Figure 3 of the attachment which are part of a related project we are working on. Due to authorship restrictions, these plots cannot appear in our revision, but we hope they can help inform our responses regarding spectral theory and feature learning. 

**Additional Experiments in the Attachment:**
1. Following the experiments in Figure 2 in our paper, we present the training dynamics of SGD training with a learning rate larger than the BBP-like threshold in Figure 2. We present the evolution of the largest eigenvalues (outliers) of CK and NTK matrices respectively during this training process. There are two phases of this evolution: at the early stage, the largest eigenvalues progressively grow, and then these outliers start oscillating before convergence.
2. Following the experiments in Table 1 of our paper, we further present the GD training with a grid search for all possible learning rates which ensures global convergence after training. We did not observe heavy tails for all these learning rates. However, analogously to the SGD case in Figure 2, the emergence of an outlier also appears when gradually increasing the learning rates. We then separately present the spectral behaviors of weight matrices when using a small learning rate and the maximal learning rate for GD training.   
3. Following the BERT experiment in our paper, we further present the alignment between the top two principal components of the CK matrix of a large language model before and after fine-tuning.


We would be happy to clarify any concerns or answer any questions that may come up during the discussion period.

Meta-review: This paper analyzes the NTK spectra of single-hidden-layer neural networks in the regime in which the number of samples is proportional to the width. A number of interesting empirical observations are obtained, and among other things it is argued that the bulk spectrum remains invariant but a spike can emerge for large learning rates.

The reviewers generally found the topic relevant and the findings to be interesting to the community. The breadth of the investigation is probably the paper's main strength and weakness. The number of phenomena identified and explored paints a rich landscape worthy of significant further study, with many interesting observations already identified here; on the other hand, no single topic was explored in sufficient depth to tell a complete story about it. This tension led to some mixed perspectives in the reviews.

There is no directive that a NeurIPS paper must present a deep dive into a very narrow topic, and the question is whether the broader approach undertaken here yields a paper that has non-trivial and novel insights that will interest the community and inspire future work. It seems the answer to that question is almost certainly yes and I therefore recommend acceptance.
Decision: Accept (poster)