Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra
Authors: Roman Worschech
Bernd Rosenow
Venue: ICLR 2025 Spotlight (2025)
Keywords: Statistical mechanics
neural scaling laws
Primary area: learning theory

Abstract: Neural scaling laws describe how the performance of deep neural networks scales with key factors such as training data size, model complexity, and training time, often following power-law behaviors over multiple orders of magnitude. Despite their empirical observation, the theoretical understanding of these scaling laws remains limited. In this work, we employ techniques from statistical mechanics to analyze one-pass stochastic gradient descent within a student-teacher framework, where both the student and teacher are two-layer neural networks. Our study primarily focuses on the generalization error and its behavior in response to data covariance matrices that exhibit power-law spectra.
For linear activation functions, we derive analytical expressions for the generalization error, exploring different learning regimes and identifying conditions under which power-law scaling emerges. Additionally, we extend our analysis to non-linear activation functions in the feature learning regime, investigating how power-law spectra in the data covariance matrix impact learning dynamics. Importantly, we find that the length of the symmetric plateau depends on the number of distinct eigenvalues of the data covariance matrix and the number of hidden units, demonstrating how these plateaus behave under various configurations. In addition, our results reveal a transition from exponential to power-law convergence in the specialized phase when the data covariance matrix possesses a power-law spectrum. This work contributes to the theoretical understanding of neural scaling laws and provides insights into optimizing learning performance in practical scenarios involving complex data structures.

Published as a conference paper at ICLR 2025
ANALYZING NEURAL SCALING LAWS IN TWO-LAYER NETWORKS WITH POWER-LAW DATA SPECTRA
Roman Worschech 1.2- Bernd Rosenow!
Institut für Theoretische Physik, Universität Leipzig, Brüderstraße 16, 04103 Leipzig, Germany 2 Max Planck Institute for Mathematics the Sciences, Inselstraße 22, 04103 Leipzig, Germany
ABSTRACT
Neural scaling laws describe how the performance of deep neural networks scales with key factors such as training data size, model complexity, and training time, of- ten following power-law behaviors over multiple orders of magnitude. Despite their empirical observation, the theoretical understanding of these scaling laws remains limited. In this work, we employ techniques from statistical mechanics to analyze one-pass stochastic gradient descent within student-teacher framework, where both the student and teacher are two-layer neural networks. Our study primarily focuses on the generalization error and its behavior response to data covariance matrices that exhibit power-law spectra. For linear activation functions, we derive analytical expressions for the generalization error, exploring different learning regimes and identifying conditions under which power-law scaling emerges. Ad- ditionally, we extend our analysis to non-limen -onearnear activation functions in the feature learning regime, investigating how power-law spectra in the data covariance matrix impact learning dynamics. Importantly, we find that the length of the symmetric plateau depends the number of distinct eigenvalues of the data covariance matrix and the number of hidden units, demonstrating how these plateaus behave under various configurations. In addition, our results reveal transition from exponen- tial to power-law convergence in the specialized phase when the data covariance matrix possesses power-law spectrum. This work contributes to the theoretical understanding of neural scaling laws and provides insights into optimizing learning performance practical scenarios involving complex data structures.
INTRODUCTION
Recent empirical studies have revealed that the performance of state-of-the-art deep neural networks, trained on large-scale real-world data, can be predicted by simple phenomenological functions Hestness al. Malooyy et al. (2022); Hoffmann et al. (2022); Porian et al. (2024). Specifically, the network's error decreases a power-law fashion with respect to the number of training examples, model size, or training time, spanning many orders of magnitude. This observed phenomenon is encapsulated by neural scaling laws, which describe how model performance varies key scaling factors change. Interestingly, the performance improvement due to one scaling factor is often limited by another, suggesting the presence of bottleneck effects Kaplan et al. (2020). Understanding these scaling laws theoretically is crucial for practical applications such as optimizing architectural design and selecting appropriate hyperparameters. However, the fundamental reasons behind the emergence of neural scaling laws have mainly been explored for linear Lin et al. (2024) and random feature models Bahri et (2024), and more comprehensive theory is still absent.
Scope of Study. In this work, we employ techniques from statistical mechanics to analyze one-pass stochastic gradient descent within -tuchnt-teacher framework. Both networks are two-layer neural networks: the student has K hidden neurons, the teacher has M, and we train only the student's s input-to-hidden weights, realizing a so-called committee machine Biehl Schwarze (1995). We begin our analysis with linear activation functions for both networks and then extend it to non-linear activation functions, focusing on the feature learning regime where the student weights undergo
"Corresponding author: r. wwwsshechhh... wechhchhwebb....

---

Published as a conference paper at ICLR 2025
1
L=1
L=4
L
10
10-3
10
500
1500
2500
3500
Figure 1: Generalization error as a function of a for K= M n 0.1, = 0.01, and N 1024, with varying numbers of distinct eigenvalues L. As L increases, the plateau length decreases until it disappears. Additionally, with increasing L, the convergence of the asymptotic generalization error slows down, transitioning from exponential power-law scaling in the early asymptotic phase.
significant changes during training. Our primary focus is on analyzing the generalization error eg by introducing order parameters that elucidate the relationships between the student and teacher weights. Despite the diversity of datasets across various learning domains, a critical commonality is that their feature-feature covariance matrices often exhibit power-law spectra Maloney et al. (2022. To model realistic data, therefore utilize maussian-dditributee inputs with covariance matrices that display power-law spectra.
White Noise vs. Power-Law Spectra. The student-teacher setup with isotropic input data has been extensively studied and is well-understood the literature [Sadd & Solla (1995). the realizable scenario where K = M, the generalization error typically undergoes three distinct phases: rapid learning phase, a plateau phase, and an exponentially decaying phase with time a. Introducing a power-law spectrum in the covariance matrix leads observable changes in the plateau's height and duration, along with a slowdown in the convergence towards zero generalization error. Notably, as the number of distinct eigenvalues L in the data covariance spectrum increases, the plateau shortens, and the convergence to perfect learning becomes progressively slower, as depicted in Figure [ This observation indicates potential transition from exponential decay power-law scaling the generalization eтror over time. Identifying and understanding this transition is critical focus of our investigation. Our main contributions are:
For linear activation functions, we derive exact analytical expression for the generalization error as a function of training time a and the power-law exponent 3 of the covariance matrix. We characterize different learning regimes for the generalization error and analyze the conditions under which power-law scaling emerges.
In addition, for linear activation functions, we demonstrate a scaling law in the number of trainable student parameters, effectively reducing the input dimension of the network. This power-law different from the power-law characterizing the training time dependence.
We derive an analytical formula for the dependence the plateau length on the number of distinct eigenvalues and the power-law exponent 3 of the covariance matrix, illustrating how these plateaus behave under different configurations.
We investigate the asymptotic learning regime for non-linear activation functions and find that, in the realizable case with M K, the convergence to perfect learning shifts from exponential power-law regime when the data covariance matrix has a power-law spectrum.
2
RELATED WORK
Theory of Neural Scaling Laws for Linear Activation Functions. Previous studies on neural scaling laws have primarily focused random feature models or linear (ridge) regression with
2

---

Published as a conference paper at ICLR 2025
power-law features Wei et al. (2022). In particular, Maloney et al. (2022); Paquette al. (2024)
and Atanasov et al. (024) ana analyzed random feature models for linear features and ridge regression, employing techniques from random matrix theory. Bahri al. (2024) examined random feature models for kernel ridge regression within a student-teacher framework using techniques from statistical mechanics. In their analysis, either the number of parameters the training dataset size was considered infinite, leading to scaling laws in the test loss with respect to the remaining finite quantity. [Bordelon et al. (2024b) studied random feature models with randomly projected features and momentum, trained using gradient flow. Using a dynamical mean field theory approach, they derived a "bottleneck scaling" where only one of time, dataset size, or model size is finite while the other two quantities approach infinity. Additionally, Hutter (2021) investigated a binary toy model and found non-trivial scaling laws with respect to the number of training examples.
Bordelon & Pehlevan 2022) studied one-pass stochastic gradient descent for random feature models, deriving a scaling law for the test error over time in the small learning rate regime. Similarly, Lin al (024) investigated infiniteddimensionall linear regression under one-pass stochastic gradient descent, providing insights through statistical learning theory framework. They derived upper and lower bounds for the test error, demonstrating scaling laws with respect to the number of parameters and dataset size under different scaling exponents.
Building upon the work of Lin et al. (2024) and [Bordelon & Pehlevan (2022). we also consider one-pass stochastic gradient descent. However, our study extends to both linear and non-linear neural networks where we train the weights used in the pre-activations (i.e., feature learning), and use fixed hidden-to-output connections. Unlike Bordelon & Pehlevan (2022). we extend the analysis for linear activation functions to general learning rates and varying numbers of input neurons. Additionally, we derive upper and lower bounds for the time interval over which the generalization error exhibits power-law behavior. A significant difference from previous works our focus on feature learning, where all pre-activation weights are trainable. In this regime, certain groups of student weights, organized by student vectors, begin to imitate teacher vectors during the late training phase, leading to specialization.
Other theoretical studies have explored different aspects scaling laws. Some have focused on learnable network skills abilities that drive the decay of the loss Arora & Goyal (2023) Michaud
ct al.] (2023;; ; Caballero et al. (2023); Nam et al.] (2024). Others have compared the influence of
synthetic data with real Jain Jain al. 2024) or investigated model collapse phenomena Dohmatob
cet al. (h024b)) a). Further works studying correlated and realistic input data are Goldt et al. 2020);
Loureiro al. (202T): Cagnetta al. (2024); Cagnetta Cagnetta & Wyart (2024).
Statistical Mechanics Approach. Analytical studies using the statistical mechanics framework for
online learning have traditionally focused on uncorrelated input data or white noise. [aad & Solla (1993) first introduced differential equations for two-layer neural networks trained via stochastic gradient descent on such data. Building upon this, Yoshida & Okada (2019) recently expanded these models to include
correlated input patterns, deriving a set of closed-form differential equations. Their research primarily involved numerically solving these equations for covariance matrices with up to two distinct eigenvalues, exploring how the magnitudes of the eigenvalues affect the plateau's 's length and height. In our study, we extend this hierarchy of differential equations to investigate the dynamics of order parameters for data covariance matrices with power-law spectra, considering distinct eigenvalues.
3 SETUP
Dataset. We consider a student network trained on outputs generated by a teacher network, using p input examples E" R.N, where μ I....,p ,p. Each input &" is drawn from a correlated Gaussian distribution N (0, E), with covariance matrix Σ E NNN, Although the covariance matrix generally has N eigenvalues, we assume it has only I. distinct eigenvalues, each occurring with multiplicity N/L. L, where 1 < L < N and N/L L is an integer. The eigenvalues follow a power-law distribution:
X4
A
(1)
11+8
3

---

Published as a conference paper at ICLR 2025
where B >> is the power-law exponent of the covariance matrix, A+ = Yı is the largest eigenvalue, and I E {I,..., L}. We choose X such that the total variance satisfies Σí= ( Ai) N, ensuring that the pre-activations of the hidden neurons remain of order one in our setup.
Student-Teaches Setup. The student is soft committee machine two-layer neural network with an input layer of N neurons, hidden layer of K neurons, and an output layer with a single neuron. In the statistical mechanics framework, we represent the weights between the input layer and the hidden layer as vectors. Specifically, the connection between the input layer and the i-th hidden neuron is represented by the student vector J; E TRAN Thus, we have K student vectors J, each representing the weights connecting the entire input layer to of the hidden neurons. The pre-activation received by the i-th hidden neuron is defined as 2; VN 5" J,. The overall output of the student is given by
K Σ (m))
(J,E) K
where g(;;) is the activation function, and the output weights are set to VM/K. In this setup, we train the student vectors J, and keep the hidden-to-output weights fixed. The teacher network has the same architecture but with M hidden neurons, and its weights are characterized by the teacher vectors Bn E R.W. The pre-activations for the teacher are yn = 5" Bn. and its
overall output is <(B, = E (Vn). We initialize the student and teacher vectors from normal distributions: Ja ~ (0, gJ) and Ba ~ (0 1). where oî the variance of the student weights and a E {I,...,NN Nj. To quantify the discrepancy between the student's 's output and the teacher's output, we use the squared loss function e = [ - r² Our main focus is the generalization error Eg = (E(E)) which measures the typical error of the student on new inputs. Throughout this work.
we consider the error function as our non-linear activation function g(z) erf x
Transition from Microscopic Macroscopic Formalism. Rather than computing expectation values directly over the input distribution, we consider higher-order pre-activations defined as (!) E" (Σ)' B./VN, suggested in Yoshida & Okada 2019). Here, J. (1) 4 (Σ)' V and yn
(Σ)' denotes the I-th power of the covariance matrix, and we define (Σ) = I. In the thermodynamic ae-aaiaivaiinss become Gaussian random variables with zero mean (k)
limit N + co, these higher-order
and covariances given by: (x. (A) _ J,(E)'J, = Q (x) yn (9) J.(E) == n!!)) and
N
2V
(9) ym B. B... = Tim. where p = k 1. The higher-order order parameters ?.-
N
R!!) and T'lh. capture the relationships between the student and teacher weights at different levels. By expressing the generalization error as function of these order parameters, we transition from microscopic view - focused on individual weight components - to a macroscopic perspective centered on the relationships between student and teacher vectors without detailing their exact components.
Dynamical Equations. During the learning process, we update the student vectors J; using stochastic gradient descent after each presentation of an input example:
J
Jut1- - J = - (J, ("),
where 17 is the learning rate. In the thermodynamic limit, as p. - oo while maintaining a finite ratio P/N, Yoshida & Okada (2019) derived a set of hierarchical differential equations describing the dynamics of the order parameters under stochastic gradient descent. Applying these findings to our specific setup, obtain the following differential equations:
dR ! Fi
R ((+1)) Q(((t)
da kF
dQ()
n7
da K" F2
() Q0) ((+1), Q((+))
1t ((())))))
(4)
from Eq. 3 to Eq. (4) \/. The functions F1, Fz, and F3 defined in Appendix A The transition represents shift from discrete-time updates indexed by μ to a continuuus-tinn framework where serves as continuous time variable.
where 14 N
At this stage, the differential equations are not closed because the left-hand sides of Eqs. 4 involve derivatives of the 1-th order parameters, while the right-hand sides depend the next higher-order
4

---

Published as a conference paper at ICLR 2025
parameters R(++1) and Q(++)) To close the system of equations, employ the Cayley Hamiiton theorem, which states that every square matrix satisfies its own characteristic equation. Specifically,
for the covariance matrix Σ, the minimal polynomial is given by P(E) : I t Xx =
Ei= C/ Eh O, where e₁ are the coefficients of the polynomial, and X the distinct eigenvalues of Σ. Consequently, we can express the highest-order order parameters in terms of lower-order ones: R(L) - -i RR Q = --11 aQ()), and T(L) = - -i= cT By substituting these expressions back into the differential equations, we close the system, resulting in (KM+ +*²) coupled differential equations. Further details on the derivation of these differential equations are provided in Appendix
4
LINEAR ACTIVATION FUNCTION
4.1 SOLUTION OF ORDER PARAMETERS
For the linear activation function, a significant simplification occurs: the generalization error becomes independent of the sizes of the student and teacher networks. Specifically, we replace the student and teacher vectors with their weighted sums, effectively acting as single resultant vectors. By defining B- 1 Σn B., the student effectively learns this combined teacher vector. Consequently, we focus the case where K = M 1. In this scenario, the generalization error simplifies to Eg 3 (Q(1) 2R() + T()), which depends only on the first-order order parameters. Therefore, our main interest lies in solving the dynamics of these first-order parameters. Since we have only one student and one teacher vector, we represent the order parameters in vector form R (R) ((") ..., R( Q = (Q("), Q()) ...., "...)Q((=)))" and T = (I((), TT)),.... T(I--))" derive the following dynamical equation:
VM
Using this setup and notation, along with Eq.
R
A
OLXL 2 + nU)
R Q
u 77
dor Q
17
22
2T
+
(5)
where = (T""),
T(()) U = -ue, and e2 = (0, 1, 1,.
The matrix A1 e
R4XL is defined in Appendix B.1 From Eq. 5). we observe that the differential equations governing the higher-order student-teacher order parameters R can be solved independently of the student- student parameters Q. Therefore, to understand the dynamical behavior of R(a), determine the eigenvalues of A1, and for the asymptotic solution, we require its inverse. Additionally, the solution for the student-student order parameters ((a)) depends R(a:) and the spectrum of A + UU. In Appendix B.I we derive an expression for the generalization error averaged over the teacher and initial student entries Bg and Jo:
need to
(1+ 2L
(Eg),g.B.
Σ by 5e exp(eeepp20 (xp(--2nRa)),
(6)
where Å are the eigenvalues of A + +U, by E (W-)) T() and W contains the eigenvec- tors of A, - n This equation generally requires numerical evaluation. However, in the regime of small learning rates 7), where we retain terms up to O(1)) in Eq. (5). we can determine the spectra of all involved matrices analytically and solve the differential equations. The solutions for the first-order order parameters are then given by
J.... (R Jg.B.. = 1 I
Σ
d exp (-TiAaa),
2 L L b
(Q()) =1 o3
d exp (-2n)
Σ
A exp (-nApa),
(7)
and the generalization error becomes
1+03 1.
(Eg) Jg. 7-
da explp -11))a))
(8)
2h
k=
5

---

Here, Å are the distinct eigenvalues of the data covariance matrix as defined in Eq. Figure compares the generalization error obtained from the exact solution in Eq. (6) with the small learning rate approximation in Eq. (8). We observe that the generalization error without approximations consistently lies above the small learning rate solution. This discrepancy arises from the fluctuations Published as a conference paper at ICLR 2025
10-1
10
0 2
10 3
Simulation
small 17 limit no approximation
Eq. (9)
10-3
10
10c
10
10a
10
10
10a
10a
10a
a
Figure 2: Generatlizatinn error <g for linear activation function. Left: eg evaluated using Eq. (
(blue) and Eq. (6) (orange) for N 128. K M L, o3 1, 3 1, and 7 1. Right: Eg evaluated using Eq. (9) (dashed orange) compared to simulations experiments averaged over 15 random initializations (solid blue), with N L 1024, 3 0.75, 'j 0.01, and a_ 0.01. The dashed vertical lines indicate the time window where the generalization error scales as fg x 193.
in the stochastic gradient descent trajectory, which become more pronounced at larger learning rates.
4.2 SCALING WITH TIME
To evaluate the the right-hand side of Eq. { employ the Fuler-Maclaurin approximation, which allows us to approximate the sum by an integral. In Appendix B.2 we derive the following approximation for the generalization error:
A_ + (21) Tio
3
21d4a - L
B - [
2n
(c,(a)) 12.B. n0 2L
r
(9)
1+3
[+B3*
where ((8, x)) is the incomplete gamma function. This expression reveals that the generalization error
exhibits a power-law scaling within the time window 24 < a
11+ 2n
I0(1+37
3
I+9
In
this regime, the generalization error scales as (g(a) x a I+B aligning with the results Bordelon & Pehlevan (2022) and Lin et al. (2024) for the random feature model. The right panel of Figure illustrates our analytical prediction from Eq. (9), alongside the generalization error observed in student neural network trained on Gaussian input data with power-law spectrum. Additional numerical analyses are provided in Appendix
B.2 B.2
4.3 FEATURE SCALING
Students typically learn directions associated with the largest eigenvalues of the data covariance matrix more rapidly Advani et al. (2020). To model this behavior, we assume the student can learn at most N₁ < N' distinct eigenvalues of the data covariance matrix. Consequently, only the first Ni entries of the student vector are trainable, while the remaining N Ni entries remain fixed at their initial random values. Our objective explores more eigendirections of the data covariance matrix. We first consider diagonal covariance matrix Σ, where each student vector entry Js independently converges to the corresponding teacher entry Bg at rate proportional to an eigenvalue λg- (see Appendix [B.)).
examine how the generalization error scales as the student
Figure(3 disurayss displays the generalization error as a function the generalization error approaches limiting asymptotic value eaymmp In Appendix B.3 we derive the following expression for the expected generalization error in this model:
for various values of Nj. We observe that
Nr
1
((g) Bg 9- 2L
A exp (-21)b00) +
(10)
R= N++1

---

Published as a conference paper at ICLR 2025
10-1
10-2
N:-50
10-3
55
N; 150
N: 100
Ni 100
N
200
Ni =00
10a
16a
102
10a 10
10a
10
10a
10a
a
Figure 3: Generalization error Eg for different trainable input dimensions Ni of the student network.
Left: cg function of for various M, with L. N 256, K M 1, a_ 0.01, 7) 0.05, and / = 1. The student network trained on synthetic data and the teacher's outputs. Right: Cg as a .. 0.01, and r - 0.05. The student network is
function of a, with L = N 1024, K M trained on the CIFAR-5m dataset Nakkiran et al. 2 using the teacher's outputs. We estimate the scaling exponent
0.3 for this dataset. For the theoretical predictions, the empirical data spectrum is used to evaluate Eq. (IT). Both plots compare the simulation results (solid curves) to the theoretical
prediction from Eq. (TI) (black dashed lines). For both plots, the generalization error averaged over 50 random initializations of the student and teacher vectors.
Using the Euler-Maclaurin formula, we approximate the sums by integrals and find:
1 1
X+
1
1
(cg (Ni, aN,,)))., sg 7-+0
+ (cg(a)))2g.n
(11)
BL
No
La
From this, derive the asymptotic generalization when L Nf. we find a power-law scaling of the asymptotic generalization error with respect to the number of learned features: Cg.ssymp N A similar scaling result for feature scaling is
Eg,asymp
1+0² A
31 N. +) Thus,
presented in Maloney et al. (2022) for random feature models. However, our scaling exponent for the dataset size (parameterized by differs from that for the number of features. In Appendix [B.3] we
analyze the student network trained with non- non-diagonal data covariance matrix. In this setting, we
find the same power-law exponent fg.ssymp
5
NON-LINEAR ACTIVATION FUNCTION
5.1 PLATEAU
As discussed in the introduction and illustrated in Fig. both the length and height of the plateau are influenced by the number of distinct eigenvalues in the data covariance matrix. Specifically, as the number of distinct eigenvalues increases, the plateau becomes shorter and can eventually disappear. In this section, we present our findings that explain the underlying causes of this behavior for the case where committee machine trained via stochastic gradient descent with randomly initialized student vectors. We adopt their heuristically derived formula for our setup, which takes the form
M. BBehl et al. (1996) derived a formula to estimate the plateau length ap for a soft
Tesc D- (D- (03) + (N)
O.p
000
(12)
where D is constant of order ((1) that depends on the variances at initialization and during the plateau phase, On an arbitrary starting point on the plateau, and Tese represents the escape time from the plateau. Our goal is to show how the escape time Texe is modified when the dataset has power-law spectrum. However, there is not a single plateau or plateau length. As shown numerically by Biehl al. (1996), multiple plateaus can exist, and their number depends on factors such as the network sizes K and M, as well as hyperparameters like the learning rate 7j. To investigate how the plateau lengths depend on the number of distinct eigenvalues, we focus on the differential equations for the error function activation up to order O (n). This corresponds to the small learning rate regime, although the associated plateau behavior occur for intermediate learning rates well.
7

---

Published as a conference paper at ICLR 2025
70
0.8
0.0-0:3455
0.6
0.18
530
0.6
0.344b
1000
0.0071
0.4
0.00711
0.2
0.0
0.0069
550
1000
1500
2000
4000
6000
2000
4000
6000
1000
2000
3000
Figure 4: Symmetric plateau for non-linear activation function. Left and center: Plateau behavior of the order parameters for L 10, N 7000, aj 0.01, 1 0.1, and M = K 4, using one random initialization of the student and teacher vectors. We solve the differential equations in the small learning rate regime, retaining terms up to (1)). The insets display the higher-order order parameters at the plateau. For the student-teacher order parameters, we observe M distinet plateau heights, while the student-student order parameters exhibit a single plateau height with minor
statistical deviations in the matrix entries Q!! The dashed horizontal lines in the insets correspond to the plateau heights predicted by Eq. (T3). Right: Corresponding generalization eπror Eg for the same
setup. The vertical dashed lines indicate the estimated plateau length based on Eqs. 112) and 15)
5.1.1 PLATEAU HEIGHT
In contrast to isotropic input correlations, the higher- order order parameters in our setup are no longer self-averaging, resulting in more complex learning dynamics. For the teacher-teacher order parameters, we find the expectation value (I = 6nm N Tr (E') and the variance Var (Iam)
(1)m Ex= X2' In Appendix C.2 we show that the teacher-teacher order parameters T!ll with are only self-averaging when L/N + 0 in the thermodynamic limit N - co, while for finite ratio L/N their variance does not decrease with increasing input dimension N. Consequently, in the latter case, the plateau height and length can fluctuate between different initializations, even in the thermodynamic limit. Figure displays the first-order order parameters as functions of a. For the teachn order parameters, we observe M distinct plateau heights, while the student-student order parameters exhibit single plateau height. In Appendix
1 1> N2
we discuss that these unique plateau heights are determined by the sum of off-diagonal elements d( -EM-1 T( for each row of the teacher-teacher matrix. To simplify the analysis, assume that all diagonal elements are equal to T(),, and all off diagonal elements are given by Tilli 1 D where D(!) represents the average sum of off-diagonal entries. This approximation captures the general behavior of the plateaus. By considering the stationary solutions to Eqs. (4. we find the fixed points for =
R
Q
1
(13)
MT TW DUy
(1
-1
M
MTO) T0
(1
D0
Expressions for the fixed points of higher-order order parameters are provided in Appendix C.3
5.1.2 ESCAPE FROM THE PLATEAU
To escape from the plateau, the symmetry in each order I of the order parameters must be broken. To
model this symmetry breaking, we introduce parameters 5()) and (()), which indicate the onset of specialization for the student-teacher and student- studentt-tudentt order parameters, respectively. Specifically,
we use the parametrization !!!h) R(' + (()) (1 - ) and Q = Q(0 d.y + C 8zz). To study the onset of specialization, we introduce small perturbation parameters (1)) (11) (9) and (4) to represent deviations from the plateau values: H!()) R² + (1), s(t) 5 + s!),
Q Q- + (²), and C C- + c(O), where 50 R and ( Q (0) Therefore. instead analyzing the dynamics the order parameters directly, we focus on the dynamics of these perturbative parameters and linearize the differential equations given in Eq. 44.
C.4 we demonstrate that, due to the structure of the leading eigenvectors of the dynamical system, we can set c!!) (0())
In Appendix
T04 R. (7(() + (M- 1)s(O) and s(() ( This
27
8

---

Published as a conference paper at ICLR 2025
0.010
M=2-
7500
0.009
L5
50000
0.009
M M=
2500
5000
0.008
250
100
50000
0.008
20
50
500
0.005
0.020
0.035
M
0.007
1/L
0.007
0.006
0.006
0.005
2000
4000
6000
8000
10000
0.005
500
15000
30000 50000
70000
a
a
Figure 5: Plateau behavior the generalization enror from simulations for a random initialization of student and teacher vectors. Left: eg for various student-teacher sizes for I. 4, N 1024,
0.1, as 0.01, 3 1. The inset shows the plateau length vs. M averaged over 15
initializations. Right: Eg for different L with = . 6, N 500, = 0.01, Ø, = 10 10-6 0.25. The inset shows the plateau length vs. 1/L, averaged over 10 initializations. The vertical lines represent measured plateau lengths.
allows us obtain reduced dynamical differential equation of the form dr da A,
(14)
where r = [ [()),, and A, E R is defined in Appendix C.4
After solving these
differential equations, we find that the escape from the plateau follows Eg Eg X e Teso where the escape time Tesc is given by
V(M 1)T(I D()) M (D +M++1)TU) M)
Tesc 2m
(15)
T(2)
p M-I +T())
For large L, show that (2) L. Therefore, for large M and L, the escape time scales as M
Tese yp. This behavior illustrated in Figure where we train student network with synthetic
input data, Additional numerical results for the plateau length are provided in Appendix C.5
5.2 ASYMPTOTIC SOLUTION
In this subsection, investigate how the generalization error converges its asymptotic value. To this end, we consider the typical teacher configuration where (T 8mm (!), as this configuration effectively captures the scaling behavior of the generalization error. For the asymptotic fixed points of the order parameters, we find R T and Q3 T 6ty. To model the convergence towards the asymptotic solution, we again distinguish between diagonal and off-diagonal entries, parametrizing the order parameters R!!") R 500 (1 6mm) and Q Q() + (1) &g)) similar to the plateau case. We then linearize the dynamical equations for small perturbations around the fixed points, setting (()) (V)) + ((() Sil) T + ,(1), Qi") T + gh,, and
C (1) c!!), and retain terms up O (1)) This yields the following linearized equation
dæ
= a.A. A Assym
(16)
de
where C;
2/3 (1, q(- (1) c(i-1)) 1) = 3 and Asaym E 1244X4L is defined in Ap-
pendix C.6
After solving Eq. (T)) we find for the generalization error
(1) (2+v3)Azo (2V30 4v30 3 - 1) Uk,L422 6 (M 11) (1)
(1)
(1)
Cg
g
0k,L+2
6
(2)
aa 13)
(2v3r 4v 3M 1) 0,31+2 6 (M - 1) 12) 2
(2)
(2)
(17)

---

Published as a conference paper at ICLR 2025
10
10
ODE (q.(1
1
10
10
10-3
8=0.5 8=
500
1000
1.5
10
B=1
10a
104
0 500
1000
1500 2000 2500 3000
α
α
Figure 6: Scaling behavior of the generalization error <g in the asymptotic regime for non-linear activation function. Left: Eg as function of a for K .M 40, 1 0.01, as 10-6 and = N 512 for simulations averaged over 10 different initializations. Right: obtained from equation Eq. 17 (orange) comparison the solution of differential equations of ((1) (blue) for K M = 2, 0.25, 3 = =1 and L = 9.
(1) and uk are two groups of eigenvec- where 0 are the eigenvalues of the data covariance matrix, 1/
(2)
/3) Ag. and the coefficients 9k and tors corresponding to the eigenvalues (2 + v3) D and (2
(1)
g depend on the initial conditions, detailed in Appendix C.6 The weighted sum of exponentials results in a slowdown of the convergence of the generalization error, similar to the linear activation function case. Figure illlustrates the generalization error during the late phase of training for different 3. In Appendices [ and F. we investigate the scaling law through numerical analyses and simulation experiments for the ever-parameterized regime K M and the ReLU activation function, well as experiments on additional activations. In all configurations, we observe the previously derived
scaling &g xx consistent with the linear activation function.
CONCLUSION
We have provided theoretical analysis of neural scaling laws within student-teacher framework using statistical mechanics. By deriving analytical expressions for the generalization error, we demonstrated how power-law spectra in the data covariance matrix influence learning dynamics across different regimes. For linear activation functions, we have established the conditions under which power-law scaling for the generalization error with a emerges and computed the power-law exponent for the scaling of the generalization етгоr with the number of student parameters. For non-linear activations, we presented an analytical formula for the plateau length, revealing its dependence on the number of distinct eigenvalues and the covariance matrix's 's power-law exponent. In addition, we found that the convergence to perfect learning transitions from exponential decay to power-law scaling when the data covariance matrix exhibits power-law spectrum. This high- lights the significant impact of data correlations on learning dynamics and generalization performance.
Limitations. In Appendix F we extended our analysis to include trainable second-layer weights. We found that for the error function activation, the scaling exponent did not change, whereas for the ReI.U activation, there was an improvement in the scaling exponent. Numerical investigations of the differential equations reveal that with the error function activation, the weights converge to the configuration the soft committee machine. In contrast, with the ReL.U activation, the weights reach a distinct fixed point leading to improved scaling. This finding provides foundation for future theoretical investigations into how architectural choices influence scaling laws.
Note added: After completion of this work, became aware of the preprint Bordelon al. (204a) which also studies neural scaling laws in the feature learning regime.
ACKNOWLEDGMENTS
This work supported by the IMPRS of MPI MiS Leipzig.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Madhu S. Advani, Andrew M. Saxe, and Haim Sompolinsky. High-dimensional dynamics of
generalization error in neural networks. Neural Networks, 132:428 446, 2020. ISSN 0893-6080. doi:
URL https: htttpp://www.com http///ww...cccieeeetttttt
com/ http://ccooee/eteeeee//////33333
Tom M Apostol. An elementary view of euler's summation formula. The American Mathematical Monthly, 106(5))409-418, 1999.
Sanjeev Arora and Anirudh Goyal. A theory for emergence of complex skills in language models. arXiv preprint arXhv:2307.15936, 2023.
Alexander B Atanasov, Jacob A Zavatone- Vetth, and Cengiz Pehlevan. Scaling and renormalization in high-dimensional regression. arXiv preprint arXiv:2405.00592 2024.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural
scaling laws. Proceedings the National Academy of Sciences, 121(27)xc2311878122 2024.
M Biehl and H Schwarze. Learning by on-line gradient descent. Journal of Physics A: Mathematical and General, 28(3):633, feb 1995, doi: 10. 108800055-44022//00001 URL http::tts////e //attss:/dx..o doi org/10. 1088 0000000005-4440-38
Michael Biehl, Peter Riegler, and Christian Wöhler. Transient dynamics of on-line learning in two- layered neural networks. Journal of Physics A: Mathematical and General, 29(16):4769, aug 1996. doi: 10. 0000008/0000-4400/29110000
http:/xxxodooooooo/.o 0ttrg//11.118/033000 URL /0
4470/
29 29/16/005
Blake Bordelon and Cengiz Pehlevan. Learning curves for SGD on structured features. In Interna- tional Conference on Learning Representations, 2022. URL http:ttt///eeeeeviiiie net/
forum?id-WP
Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. How feature learning can improve neural scaling laws. arXiv preprint arXiv:2409. /7858, 2024a.
Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. A dynamical model of neural scaling laws. In Forty-first International Conference on Machine Learning, 2024b.
Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws. In
The Eleventh International Conference on Learning Representations, 2023. URL https:/ ///
openreview n forum www.foorrrssssssoo id==skkjveq102
Francesco Cagnetta and Matthieu Wyart. Towards theory of how the structure of language is acquired by deep neural networks. arXiv preprint arXiv:2406.0000 2024.
Francesco Cagnetta, Leonardo Petrini, Umberto M. Tomasini, Alessandro Favero, and Matthieu Wyart. How deep neural networks learn compositional data: The random hierarchy model. Phys. Rev. X, 14:031001, Jul 2024. doi: 10. Re.. 14.031001. 1e...o...ev.....100001 URL wttphhtt///tnimt aps
org/doi/10. http:///....11//yyyee.eeee
14. 031001
Elvis Dohmatob, Yunzhen Feng, and Julia Kempe. Model collapse demystified: The case of regression. arXiv preprint arXiv:2402.07712. 2024a.
Elvis Dohmatob, Yunzhen Feng, Pu Yang, Francois Charton, and Julia Kempe. A tale of tails: Model collapse a change of scaling laws. In Forty-first International Conference Machine Learning, 2024b. URL htt.:htttps://peereeeee
net. for://ooum? id= shWi
Sebastian Goldt, Mare Mézard, Florent Krzakala, and Lenka Zdeborová. Modeling the influence of data structure learning in neural networks: The hidden manifold model. Phys. Rev. X, 10: 041044, Dec 2020. doi: 10.1 1011100/yyeeex.10..o0 10.001144 URL https:///iikk www.kaas..................m..... org/
103./hyyey.. 10. 041044
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint 17 1712.00409, 2017.

---

Published as a conference paper at ICLR 2025
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Au- relia Guy, Simon Osindero, Karén Simonyan, Erich Elsen, Oriol Vinyals, Jack Rae, and Laurent Sifre. An empirical analysis of compute-optinal large language model training. In S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Ad- vances in Neural Information Processing Systems, volume 35, pp. 30016-30030. Curran Asso- ciates, Inc., /ttp:/tt..//ppprreee....oo.....s
URL
Marcus Hutter. Learning curve theory. arXiv preprint
2021.
Ayush Jain, Andrea Montanari, and Eren Sasoglu. Scaling laws for learning with real and surrogate data. arXiv preprint arXiv:2402.04377 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint vrXiv:2011.08361 2020.
Licong Lin, Jingfeng Wu, Sham M Kakade, Peter I. Bartlett, and Jason Lee. Scaling laws in linear regression: Compute, parameters, and data, arXiv preprint arXiv:2406 (XX::2406.08866,, 2024.
Bruno Loureiro, Cedric Gerbelot, Hugo Cui, Sebastian Goldt, Florent Krzakala, Marc Mezard, and Lenka Zdeborová. Learning curves of generic features maps for realistic datasets with teacher-student model. Advances in Neural Information Processing Systems, 34:18137-18151, 2021.
Alexander Maloney, Daniel A Roberts, and James Sully. A solvable model of neural scaling laws. arXiv preprini arXiv:2210.1685 2022.
Eric Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. In Thirty-seventh Conference Neural Information Processing Systems, 2023. URL
https:/ http:////ppeeeeeeewneo Serpennereee..eet/oouuuuttooo. wttp:/oooommimmooooeoooo
Preetum Nakkiran, Behnam Neyshabur, and Hanie Sedghi. The deep bootstrap framework: Good
online learners are good offline generalizers. In International Conference on Learning Representa- fions, 2021. / wttp://wwwweeeeoommdddddddeeeeee
Yoonsoo Nam, Nayara Fonseca, Seok Hyeong Lee, and Ard Louis. An exactly solvable model for emergence and scaling laws. arXiv preprint arXiv:2404. /7563, 2024.
Elliot Paquette, Courtney Paquette, Lechao Xiao, and Jeffrey Pennington. 4+ 3 phases of compute- optimal neural scaling laws. arXiv preprinl arXiv:2405.15074 2024.
Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, and Yair Carmon. Resolving discrepancies in compute-optimal scaling of language models. arXiv preprint 241. 2024.
Frederieke Richert, Roman Worschech, and Bernd Rosenow. Soft mode in the dynamics of over- realizable online learning for soft committee machines. Physical Review E, 105(5). 2022.
Grant Rotskoff and Eric Vanden-Eijnden Trainability and accuracy of artificial neural networks: An interacting particle system approach. Communications on Pure and Applied Mathematics, 75: 1889-1935, 09 2022. doi: 10. 10.1002/cpa.22074.
David Saad and Sara A. Solla. On-line learning in soft committee machines. Phys. Rev. E, 52: 4225 4243, Oet 1995. doi: 10. 10.110//yy doi /00 1wt.11113//////.//.. 52. 4225
URL inttps:///1in aps...ppp... o
Michiel Straat and Michael Bichl. On-line learning dynamics of relu neural networks using statistical physics techniques. arXiv preprint arXiv: 2019.
12

---

Published as a conference paper at ICLR 2025
Alexander Wei, Wei Hu, and Jacob Steinhardt. More than toy: Random matrix models predict how real-world neural representations generalize. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 13549-23588 PMLR, 17-23 Jul 2022. URL wtt.htttss///oceeeeiiii wei22a. html
ml www.mm....r..vv.....mmm 627
Roman Worschech and Bernd Rosenow. Asymptotic generalization errors in the online learning of random feature models. Phys. Rev. Res., 6:L022049, Jun 2024. doi: 10. Reeeeeh 6.L022049.
URL https:
03 http://.............oeeeeeeevee.
6
L022049
Yuki Yoshida and Masato Okada. Data-dependence of plateau phenomenon in learning with neural net-
work statistical mechanical analysis. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL httpp/////ppppeessss..eeooooooo cctpaperccpppereeeeeeeeeee
fi f1ee/pperr20111eeeeeeeaaeeceeeee -appeaaaa3---aaPe--------- pdf
13

---

Published as a conference paper at ICLR 2025
A
DIFFERENTIAL EQUATIONS
From the stochastic gradient descent given in Eq. (3. Yoshida & Okada (2019) derived the following differential equations, generalization error, and covariance matrices in the thermodynamic limit N + co. Following their work for our setup, one finds
!1) dQ
M
M K
M Σ m=1
K
bj
Σ
2jj
pm)
13( 2j
xxR +
1a ( .y
la la xxx
da
K
K
K
m==1
k=1
K.K
K.M
K2
M(+1
K2
M
22),
Σ
(24, yh Mm)
K
2
Σ
La T. Th yrc)
k,I=1
k,n=1
d.R
M
K
(a yn 1), 4mm)
Σ
Ja( vn
(111) ,,,5)))
(18)
do
m==1
with V1 =
Σ X [s 22)23) = (d(2))22((25)) and Ia(21, 22, 23. 21)
(2))g (0)))(2)99)))) (z)a))
In this setting, the generalization enror becomes
M
K
M
Eg
Σ
T2 (1)
(1) (1) 12(2)) 25 1-=1
Σ
12 w2
(1)
(19)
2
mm==1
where [2(21, 22) = (0(2))9(22)). Thereby, the 12, Ia and I4 are integrals over the generalized pre-activations x7 and (h Thereby, 2, are normally distributed variables and stand for either s; or (1) Therefore, the integrals 12, /3. and I, are multivariate Gaussian expectation values that are determined by the expectation values and covariance matrix of the generalized pre-activations. 12 is two-dimensional Gaussian integral. For example, for I2 (x( x _z on obtains the following covariance matrix
(1)
C (i.j)
(20)
The resulting elements of the covariance matrices depend the higher-order order parameters. I3 is a three-dimensional Gaussian integral, and an example of the covariance matrix is
Q
(
R!ll
CO (I.j.)=
Q (I+1)
21+1)
(l++1)
(21)
R
R!(+))
T(l))
jn
(1) (1) for [z(xx()) .2) depend on the first-order. Furthermore, Ia a cir-dimensional Gaussian integral and depends, for (1) (1) (I)
( Note that I3 depends on higher-order I as compared to In and T4, which only
example, on the following covariance matrix for (zx) .. ym
R
R
R
Q
(1)
())
R (1)
(i.j.,
R(1)
R(1)
T(
T(1
(22)
in
in
R(1)
R
(())
(())
The specific differential equations for the linear and error function activation are provided in their corresponding subsections.
14

---

Published as a conference paper at ICLR 2025
B LINEAR ACTIVATION
B.1 SOLUTION OF ORDER PARAMETERS
The linear activation function leads to the following teacher output ((B
VM 2 n=l
BB.. This VN"
makes possible rewrite the output as ((B,E)
1
M
c
Σ .. B, Bn= <B VM
VM
VN
VN
vN
where we have defined new teacher vector B v
Σm=1 B Since has the same statistical
properties as one random teacher vector B, it makes no difference whether we consider the case M > 1 and define B or M = 1. The same argument also applies to the linear student network. Therefore, in the following, we analyze the case M For K = M 1, the generalization error becomes
Eg = (20)
-R + T()).
(23)
The differential equations for the order parameters
dR()) = 17 [1(+)) - -R((+1)
da
dQ
2n [R(++)) - qu++i]] 2 [T()) + Q()) 22(1)
(24)
da
with 0 11 L, 2 and for the last component
dR(L-1)
L-1
Σ
(T(W)
(T(N)
-
da - 17
ch
dQ(i-1) 2n
L-1
Σ
Ck (
(
- Q(11) + T(T
[T()) + Q()) -2R
(25)
da
Thereby, we have exploited the minimal polynomial for the order parameters for the T-th order, e.g. ( - (8), with the coefficients of the minimal polynomial Gę- The set of coupled linear differential equations given by Eqs.
24 and (25) can be written in the following form
R Q
A -2A
OLXL 2A + nTU
R Q
dex
= 11
2T
+
1
(26)
where u (T(1), T(2) T(L))
U uez
with e2 = (0, 1, 0, ..., 0) and
0
0
0
0
-1
:
A
0
0
0
0
0
U = U=ue,
(27)
-1
0
.Co
C2
CL-2
CL-1
Therefore, the differential equations for the order parameters are
dR
nA R m
(28)
da
dQ
n1 + A + n
(29)
da
15

---

Published as a conference paper at ICLR 2025
with A, -2A, 2n, A, = 2A1 + n
Thus, we can solve differential equations for the student-teache order parameters independent from the student-student and find
R (a) = e Ry + e A
(30)
where Ro are the student-teacher order parameters at initialization and we set Ro = 0 which is achieved on average. Before inserting this result into the differential equation for the student-student order parameters, we evaluate Eq. (30). For this, we need to find the eigenvalues of the matrix A and evaluate A u.
First, start with the eigenvalues. In order to find the determinant of A₁ XIZ, apply the Laplace expansion with respect to the last row of A 222 and evaluate the determinants of L different L. 1 L 1 smaller matrices. The resulting sub-matrices triangular, and their determinant is, therefore, simply given by the product of the diagonal entries. After applying the Laplace expansion, we find
L-2
det (A1 - AIL) Σ
(-
ca(11)
(CL-1 (1-1))((-2)--
i=0
Σ c-
(31)
with 1. Since ..., are the coefficients of the minimal polynomial for the distinct eigenval- ues of the data covariance matrix, we now know the roots of Eq. (31). Therefore, the eigenvalues of A are given by the negative eigenvalues of the distinct eigenvalues of the data covariance matrix
AA = - for s I I L. By applying the matrix A, on a potential eigenvector A Ub = Ug we find the following conditions for the eigenvector entries
Uk = (- Uk,1,
(32) choose 1k,1 for all eigenvectors. The
obtained by recursive method. Furthermore, cigenvector matrix V, for which an cigenvector gives each column, is given by the transpose of the Vandermonde matrix
d
2
V
(33)
X J-11
Second, we evaluate all matrix products given in Eq. 30). Since all eigenvalues are strictly negative, the student-teacher order parameters converge lima-+x R (a) A 'u that we are going to
evaluate using the eigenvector matrix. We insert the eigendecomposition A VA.V- into and find for the entries of the student-teacher order the asymptotic solution with A2,5j -ōkj A parameters
lim ( Ra)Ja,o.b = Σv. Vij Σ
A2,j Σ
(V (v- Jo..B.
a-+00
Σ VigA2.33 Σ (V- (ui)
Σ 2
(i-1)
1
Σ (V-1) (ui).e,o.Be
L
L
Σ (0-2) Σ (v-1) (ui)Je.o.B.
(34)
16

---

Published as a conference paper at ICLR 2025
and further, evaluate
(V- (2)) Ja, Σ (V-1), (111)) Jo.
Σ
(V-1) Σ
a
1
Σ (V )1 Na1
Σ Ne Σ (V-1));
V1.a
L A
(35)
where we have used Et= (V-1), matrix with its inverse. Thus, we find
jt Vi,a ò3.a obtained by the definition of the product between
L A1 = (T(1-1))Saoo.B
lim (Ra) a=000
(36)
L
Note that R; = R and therefore lim (R J...... = ((((i)) Bo. Next, want evaluate AAa AAiiAA'
A Iu and define
Vis exp (-n d5k Σ Au [(v-)) im (Um))
F; =
m
1 -x,- Σ Vis exp (-1dja)
Σ (v mm (Um)
1
Σ
X exp (-ndyar)
T,
1
Σ
exp (-11Aaa)
T,
(37)
which leads to = = = A = =T F. Thus, obtain for the expectation value of (())
(R()) No.0. = (R2)
=1 Σ
exp (-11Aaa) Aa.
(38)
As next step, insert the result given by Eq. (30 for RR)) into the differential equations for the student-student order parameters given by Eq. (29 in order to obtain a new expression
dQ = 1) Ae A u + + + 1
(39)
da
In order to simplify the differential equation, we evaluate ATA The inverse of A, can be obtained analytically, where find
17

---

Published as a conference paper at ICLR 2025
8
A
(40)
and obtain for the matrix-vectto product - -A u 2(n 1) u. We can further simplify
m/u A3A u = (17 2) u and insert this result in Eq. (39) order to obtain
dQ da = nA Aia A, u n/2 1012-nn--- +
(41)
The solution of this differential equation is given by
Q (a) enA Qo + c (A1 A4) (cldAI-AAia- -II AsAT'u
(42)
+ (2-n)e n)e A (I e - u
(43)
with Qo are the student-student order parameters at initialization. Note that the initial value Qo does not vanish in the thermodynamic limit and also not zero on average contrast to Ro- By the definition of the matrices, we find A A₁ 1/2 Ag making it straightforward to evaluate their 21,. Furthermore, in order estimate (2 -1) A u, we need know
relations (A, A44)- the inverse of A, for which we find
20g
2-
200
200
20p
20g
0
0
(T
0
0
A
2(n-2)
(44)
0
(
0
0
2(n 2)
11
0
0
2(v-2)
Thus, we obtain for the product (2 - (2-11) A, 4 -T. Therefore, we can simplify Eq. (43) to
(a) = + Qo (enA.n - Ae T
nA
(45)
Using Eq. (33 and taking the expectation value over initial student and teacher entries B, and Jo. obtain for the generalization error
(1+03) 2L
(g n.
Σ by d exp( exp(-2n)xa)
(46)
where À are the eigenvalues of A + n,, = Σ (W-1) T(, and W contains the eigenvec- tors of A + n,. Although U is rank-1 matrix, standard perturbation methods applicable to find the eigenvalues of the shifted matrix A, + nUU because U may have a large eigenvalue, making it unsuitable small perturbation.
18

---

Published as a conference paper at ICLR 2025
B.1.1 SMALL LEARNING RATES
For small learning rates, can approximate A4 2A, and immediately find for the first order student-student order parameters
1Q NJ.o.. = (Q2)
1 E exp (-2mmma) do 2 Σ exp ( (-n Aa = =1+ +1+
1
Expp
(47)
L
where we have exploit that JQ.) = (T) J. After inserting Eqs. 23), we obtain for the generalization error
38
and
47
into Eq.
1
B. o3)
Σ Xa exp(-227da0)) 2L
(48)
leading to Eq. 8 in the main text.
B.2 SCALING WITH TIME
In order to evaluate the sum for the generalization given in Eq- Maclaurin approximation. The formula for the approximation is given by (cf. Apostol 1999))
448). we consider the Euler-
[
Σ 1(0) [ [(x) dx + [(n) + {(m)
Bah (2k)
(f(2 - ((2k-1((m))
(49)
2
mm
where Bog is the kth Bernoulli number. For the difference of the sum from the integral, one can find
Σ1 S f(x) dr f(m) + ((n)
+I" f((x))(((x)) dx
m
(50)
[(m) + ((n) 1 (n) ('(mm [" (x) Pa(x)
dr,
6
21
31
where Pa (x) are the periodized Bernoulli polynomials of order k. These are defined as:
Pa(x)) B& (z:
[x]))
where (z) are the Bernoulli polynomials and | denotes the floor function. In our case, the function and its derivative are
(1+03) d+
2n144a exp
a(k,a,
k0+1
k:41
f'(k) (1 + 0 (1+ exp
d+
21144a k-+1
221
3 3+1
L
12843
k342
We want to approximate the sum by an integral and, therefore, estimate their maximal difference. Due to the exponential pre-factor in all functions, the maximal deviation of the integral from the sum obtained at initialization a = 0 or for -> 0. Therefore, we make the following ansatz for lim,-
I" f(x) do f(m) + f(m) [ ((()))(()) dr.
Σ 5(
(51)
Our goal is express the term f() dr for - To do this, we consider the integral over the interval [k: + 1), where is integer. This allows us handle the floor function more easily, as it is constant over each interval. Within each interval, one can simplify the expression for Jr ++1 (z) dx, which be written as:
L
lim [ (x)
A (1 (1+ d
19

---

Published as a conference paper at ICLR 2025
10a
ex:0-
2w10
sum
integral corrections
integral without carrections
4x
integral 0th order corrections
integral + 0th order corrections
integral upper bound
10
10
10
10
10a
10
Figure 7: Generalization error evaluated by Eq.
48
(full blue) and Eq.
449
including different
order of function derivatives (dashed) for N = 128
K = M 1, 0
and B = 0.5. The 0.001, the difference Left: For 7 2A
integral without corrections (orange) is given by Eq. 54)
between the sum and its integral representation is well represented by Eq. 50 including corrections up to Oth order in the derivative of f. Right: For = 10 and
2nA Re 103, the difference
between the sum and its integral representation is quantified by the upper bound given in Eq, (52) (red dashed). However, for 2qA,, including derivatives up Oth order Eq. (50) are suffictent to quantify the deviation of the sum from its integral representation.
Substituting the expression for f'(x) and taking the limit as 17 - O, A becomes:
1
A ( + 1) + 21-+1 +
3(k 1) 3K8
To estimate the total correction across the entire interval from 1 to L, sum this expression from to for large L. For the first term, we obtain
L-1
1 (k 1)8H1 5 + (((3+1)- 2
1
2 k-1
2
where C(B + 1) is the Riemann zeta function. The second term is straightforward and sums
-
[(( 11.
2 K
k=1
For the difference between the third and fourth terms, we find
4-1
Σ
1(k+1)s
3k3
3
Combining all these results, the sum of the integral correction term over all intervals from 1 to L 1 is:
L-1
1
A=1
After inserting the start and end point of the into the function f f(m 1) and f (n L) for
7 + O, we obtain an upper bound for the error
Σ 1 f(x)dx do (1+03) A+
1 B
1,+1 + ((3+1)
(52)
2L
This "worst case" upper bound works excellent for moderate input sizes N ~ O (10*) as well. The difference between the sum and its integral representation beyond the small learning rate limit can be estimated as well. If we consider a learning rate larger than 1) > 1 2A_a then the difference
between the and its integral representation well quantified by including derivatives of f up
20

---

Published as a conference paper at ICLR 2025
15101
2x20
10-1
sum
Sum
Integral without carrections
integral without carrections
ex107
integral with only G1
integral with only G1
10
10
10a
10
a
Figure 8: Generalization error evaluated by Eq.
(48)
(full blue), Eq. (54) (dashed orange) and (55) and = 0.01. Left: For 3 = 1. Right: For
(dashed green) for N = 128, K = M = 1, of 0.01.
Oth order the right-hand side of Eq. (50). Especially f(m 1) plays the most important role, and we find
E5 f(i) [ f(x) de < 1 f(1) 1
(1+03) x+ exp (-22))a)).
(53)
2L
Since we consider 7j a the difference decreases exponentially in time. For smaller learning rates 2A-a higher order must be included up to the "worst case" bound given in Eq. 52) equivalently
However, for the condition 17 244a occurred.
2n no meaningful learning has
In the right panel of Figure we have chosen = 10 2A 10-2 in order to test our
approximation of the sum We find an excellent agreement. However, such learning rate is very untypical since no learning would occur over many time orders. Typically, the learning rates that we use in our simulations are larger of order O (1003) to O (1). Therefore, including Oth order derivatives of order to estimate the difference between the sum and integral representation enough to consider.
Since the corrections, including (tth order derivatives of ʃ, decrease exponentially with we use the integral in order to approximate the generalization error 6 (I; ox) R I f(x) da for m = 1 to This integral can be solved analytically, and obtain
(); RJ (1+ x (2m) a) गेन I (1+
2m
(13
B T 2ndrar)
2T.
L
-
(54)
where I (s, .. is the incomplete gamma function.
Next, aim elarify under which conditions the generalization error scales power-law in a. For the second gamma function within the brackets in Eq. (54). we notice that its argument ++8 larger than for the first gamma function. Thus, we can introduce a scaled time variable
by ā 2m and insert this into the second gamma function I' ( 21124a = 1 (
decreasing exponentially fast with â. Therefore, we can neglect the second gamma function compared to the first since both operate on different time scales, presented in Figure Thus, further simplify
(1 (1+03) 2+ (2n)4aa)
148
3
2
Cy (L; ~
T
(55)
2L
1
1+
L
valid for 24A4 < a. Furthermore, from the previous discussion based on empirical observations, we know that no meaningful learning happens for a < 2n
The incomplete gamma function given in
Eq. (55) can further be approximated by
2(8,2) R T(s) =
(56)
21

---

Published as a conference paper at ICLR 2025
10a
10
10-2
Eg
w
10-2
10-3
Simulation
Simulation
Eq. (9)
Eq. (9)
am
10
100
10
10a
10
10
10
10a
102
10
10
a
Figure 9: Generalization error as a function of a obtained by simulation experiments for the linear activation function for K = .M 1. We test our time window given in Eq. (58) (vertical lines) and
power-law exponent cg x a (black dashed descending line) for and a, = 0.01. Left: A setup configuration for N A setup configuration for N = L = 1024. 5 = and = 0.01, where we obtain 0.2 < < 17740. in the right panel.
256, 3 0.75 and r 0.001, where obtain 3.8 a < 12850. Right:
Note that we have placed the lower bound
for z < 1, 5 3+I and 2 =
2r L As last approximation, we use I(s, 2) F(s) valid for
r(s)
> order find for the generalization error
Cg(L; (1+ X (2n)+
1-
3 1 1+B) B r
(57)
2L
1+ 3
From Eq. (57). we obtain power-law scaling for the generalization error cg X QTTZ. For which range of can this scaling law be observed? Clearly, the lower bound for is 2n < as already discussed. The upper bound can be estimated from the approximation [(s, 2) [(s).
which valid for the condition A < [s)). order to quantify the notation "much less than",
we demand that g is at least one order of magnitude smaller than [(ss,, i.e. _~ To (ss). Using this inequality together with the definition of and s, we obtain the following time condition
1+d
11+8 < 2nd (0(1+8) expect power-law of the <g in
T
Thus, we estimate the following time window for which we
143
I,+P (3 3
B
#
<a
(58)
2m
2mA+
10 B)
(1
We observe that the power-law range is extended with increasing number of distinct eigenvalues L and covariance matrix power-law exponent B.. In Figure we test our upper and lower bound for simulation experiments and find very good agreement.
B.2.1 GENERAL LEARNING RATE
Here, we reconsider Eq. 45 and want to evaluate the student-student order parameters not in the small learning rate limit. For this, we have to understand how the eigenvectors of A4 influence T. We call the eigenvector matrix of A simply Va which contains all eigenvectors as its columns and introduce the eigendecomposition Ara Vye"Aaa with A4 containing the eigenvalues of A on its diagonal. To find the eigenvalues of A4 either analytically numerically is no longer easy. However, we can pretend to know the eigenvalues of A, and call them X1,6 for < k < L. This makes it possible to find the structure of the eigenvector matrix.
Here, we present a more general solution for the matrix B = aA, + cU and call the eigenvalues of B simply AB, and define the eigenvectors BUB A UB, for k {1,..., L.). Note that for a = 2 and ε = 1), we can reproduce B (a = 2, 1) = A,, For each AB,k, the corresponding k-th eigenvector has the following structure
AB.k a
1 1
UBI1k 1, UB.2k
UB.3k
( \B,R),
a
22

---

Published as a conference paper at ICLR 2025
L= 10
L=
104
L= 10
L 20
7=
Lxb. 1x x [AR.a]
10t
L- 10
L- 30
L-125
10100
x Aa.
10
L=
L-
10
L=
L- 125
100
10-1
10
10
10
10
10a
Figure 10: Coefficients by defined by Eq.
62
multiplied with λB for 1.0. Left: 3 0.1.
Right: = 1.0.
1.0.
L-2
X-1 B.k
1--1ao----x NB, T
B,ik aL-2( a)
The eigenvectors have an interesting property. Like for the companion matrix A₁ and its Vandermonde eigenvector matrix V1, the first 1 entries of the eigenvalue equation BUB, "B. give zero by construction (BUB. & - AB (B),) 0 for l I....,, L - 1. Only the last entry = L of the eigenvalue equation provides information about the eigenvalues of B
L
BBOB - AB. = Σ CB..AABA
j=1
This is exactly the minimal polynomial for B where have identified
(-1)4t1
CB,0 = aCp,
CB.L
a!.-
(59)
c
(-1)+41 CB. w-1
L-1-i
acj te
Σ 1=0
(60)
the shifted coefficients of B compared the coefficients Cj of A₁ for j 1,..., 1. Since have assumed that λB is an eigenvalue of B, the right-hand side of Eq. has to be zero, and the CB. numerically calculate
indeed the coefficients. Since we know the shifted coefficients, the corresponding eigenvalues as the roots of the new minimal polynomial.
Most important is the second entry of the eigenvector matrix V4 since this will have an influence on the generalization error. Next, we evaluate V 'T
(cg) = 01)
1
Σ k=
bedB exp((22)
(61)
2L
with by = Ef (V=) and λB. are the eigenvalues of B for = 2 and = Thus, in order calculate the generalization error, we have find an expression for V-1 We know that B shows some properties of companion matrix. Therefore, we perform a similarity transformation B SB.S Thereby, Bg has again a companion matrix structure similar A1, but it has cB.; for its last row entry for 0, ...., L given by Eq. (60). For the transformation matrix S, we obtain
a
0
0
1
0
0
2
0
S
(a
7))
0
-n3
T2
0
-T--1
T
-TL-3
( (-1)
1-
a2
23

---

Published as a conference paper at ICLR 2025
8=1
8=1
8m
8=0.5
Lxbx x AA
10
Lxbx x AA
104
8
0.0.11
10a
10a
10-1
10
10a
10-
10
10
k
10a
LXb. x [As.el
10"
--1
- 0.5
8 0.1
0.01
10
10
k
Figure 11: Coefficients by defined by Eq.
62 multiplied with AB and L for L = 40 (solid) and eigenvalues of the data covariance matrix (dashed). Upper left: n 0.1. Upper right: 7) 0.5. Center: = 1.0.
10a
L= 150
1 125
1-100
1- 100
10.
50
1 75
[As.el
Ma.
25
L 20
10a
L
1 10
10a
10-1
10-2
10a
101
102
10a
10a
102
10a
150
1- 125
L- 125
10a
10a
1
10-1
1 75
50
25
10
1=
24 21-
10
20
2
L= 10
10-1
L= 10
10-5
10-
10
10a
10
10a
10a
10a
k
Figure 12: Spectrum of matrix B for (upper figures) and corresponding difference with the
spectra of the data covariance matrix. Left: = 1 and 3 = 0.1. Right: = 0.5 and 3 = 1.0.
24

---

Published as a conference paper at ICLR 2025
n limit no approximation
small n limit no approximation
10-1.
-2
10-2.
10
10a
10a
10"
10t
a
small limit no approximation
10-22
10
10
10a
10a
a
Figure 13: Generalization error evaluated by Eq.
8 (blue) and Eq. (6 (orange) for N = 128
K = M 03 and B 1. Upper left: 7) O.T. Upper Right: 7j = 0.5. Bottom: v) 1.0.
having a triangular structure. The entries of the inverse of triangular matrix can be calculated by this simple relation
S
Su
S.j
Su
for
S k=
Note that the eigenvector matrix VB2 is again Vandermonde matrix, meaning that know the entries and their inverse. Therefore, we obtain
bk Σ (SSVB2)+ T(O kl
(62)
In practice, we no longer have to directly invert any matrix, which allows us to consider higher values of T. and more different covariance matrix power-law exponents 33. Calculating the inverse of VB is numerically very unstable and just possible for small numbers distinct eigenvalues L.. Figure FigureT0coompann 10 for various L. For small learning rates, the differences in the spectra are very small and increase with increasing learning rates until the trend of power-law decay is destroyed. Note that by 7. and λB, - aA if we just consider the differential equations up to order (()). Figure 11 shows the same behavior but for different consistent power-law scaling. Figure 12 shows the spectrum of X and its difference from A We also observe in the spectra that the pure power-law behavior perturbed. However, this effect seems to just increase the generalization error without changing its scaling, shown in Figure 13
by multiplied with λg, with the eigenvalues of the data covariance matrix A
The smaller & becomes, the larger the divination from
B.3 FEATURE SCALING
Instead of considering the statistical mechanics approach, we study directly the stochastic gradient descent here
Ju+1 - J - (J g.
(63)
25

---

Published as a conference paper at ICLR 2025
From this difference equation, can derive Langevin equation for the weight dynamics in the thermodynamic limit N + oo (see Rotskoff & Vanden Einieen- (2022): Worschech & Rosenow (2024)). ). For the continuous equation describing the trajectories of the weights, we obtain
dJ
da - Nc +
(64)
VN
where is random vector describing the path fluctuations with (7) = 0 and (7x (a) 7i (a')) Cijó (ax - a²).
For small learning rates proper scaling of the learning rate with the network size, neglect path fluctuations and approximate the stochastic process by its mean path. This leads to the following differential equation
dJ
= -1/Vcc da 7-10
=- (J (J-
(65)
For diagonal covariance matrix Σn 6kI the solution of Equation (65)
J& = cxp (-))aa) (JL B2) B₁
(66)
for E (I....,, .} and initial weight component J.. Thus, the individual weights are learned exponentially fast, and each component Jk converges thereby the component of the teacher. For the first-order order parameters, we find
N
JA Au Ba
((1)
(exp(-n (Jh -Ba) + BB) d BR
(67)
N
N
k=
k=1
Q())
JR
1
Σ
N
(exp(-ndba) (J2 -B)) BB)² dx
(68)
N
and for their corresponding expectation value
N Σ
(R (R Dg.B = =1
d exp(
(69)
N
Amm
1 + 03
2 N
(Q =1
Σ d exp(-27)2a)
Σ
d exp( nn)aa)
(70)
N
N 1=1
For the expectation value of the generalization error, we obtain
1+
(e.)
As expx(222111
(71)
2N
of As (exp(-2nAua))
(72)
2L
k=1
where have exploit that E N as assumed for setup (see Section 3). Note that Equation
(72) that we derived from the approximated Langevin equation, is the same as the generalization error derived from the statistical mechanies approach in the small learning rate limit 17 0 given by
Equation 8 Therefore, neglecting the higher order of the learning rate in the differential equations equivalent neglecting path fluctuations of the stochastic gradient descent.
Next, in order to model how the generalization error scales as more and more feature directions are explored, we assume that N; components of the student are already learned. Thereby, the other N N components are kept fixed and random. Here, we want to investigate the generalization error as a function of N₁. For this, we make the ansatz:
B for k 1.... N J
Jx =
(73)
else
26

---

Published as a conference paper at ICLR 2025
Therefore, obtain two different parts for the order parameters
R
Jk Bk
1
B
Σ
ARJA Bk
N
N
k=1
k=1
N=++1
N₁
N
Q
Σ
J.JUJ
X B
Σ
(74)
N
N
k=1
N=1
k= N
and their expectation values become
Ni
A
N
N₂
(Q
)J.
AB
(75)
k=N++1
Next, we insert Equation
75 into the expression of the generalization error and obtain
Ni
(eg)de..BB = 1
2
L
km
1 + oj
Ak:
(76)
2
where we have exploit that Σx 2 N and NY E N. Next, we use the definition of the eigenvalues
X -+ (2) (=) and define the partial S(M) + EL ( which leads to
1 1 o3 S(N)
Eg =
(77)
2
In order to approximate the sum, we use the replace the sum with integral. For 3 O, we find
formula defined in Equation (49) and
d_
S(N)) ~
di
i8+1
N:
x
(78)
N 8
N
Thus, we obtain
1+
A+
1
1
(cg.amym)) ~
(79)
2
LB
The zeroth-order error of this approximation, which we consider as the worst-case, is geg = 1+03
M + Li+z
4
If we not assume that N, eigenvalues are already converged, then we can make the ansatz
Jx for
1,
N
J, = 1 else
(80)
27

---

Published as a conference paper at ICLR 2025
where JR given in Equation 66). parameters
Therefore, again obtain two different parts for the order
R
1
Σ
N
N:
N
1
+
Σ
Ax
N
N=
1
d Texp (-n2Ra) (Jh- Bb) Bu] Be A
1
E
N
N=N+11
Q
1
JaJR
N
N
N
1
Σ deJReh
N
N=
1
A Texp -n) (Jh BB) BB]
Σ
deckth
(81)
N
k=N++1
and their expectation values become
N: exp (-n))aa);
(R
(R
Jg..B.
dB
k=1
1 03
M
N; r exp (-mAAaa
(Q(D)))g.BB
dE exp (-2m)aa)
N
N
k=1
k=1
A
Σ
A
(82)
N
Note that Equation reduces Equation 75) expression of the generalization error and obtain
for + oc. Next, insert Equation
82 into the
1 + oj
Ni
Jg.
Σ
A exp (22
A
(83)
2L
Note that Equation 833 is basically a combination of Equation (6) and Equation 72 If we set N₁ N in Equation ( then can reproduce Equation (72). and if we let a - 30 Equation 83 then we reach the asymptotic generalization error given by Equation (6)). Therefore, we can
repeat the analysis of this and the previous subsection separately and finally obtain the generalization
enror for small learning rates
1
A
1
1
1
Eg (N)
2
L
3
N
1,8
(2m\4a)
ITZ
2m -I -1 [ (1+ 2m 2nd4a)
B
)]
(84)
1+ B
1+
L
Next, consider general Σ, which is no longer diagonal. The solution of the Langevin equation is j = (() - + B.
(85)
where we have introduced the eigendecomposition = WDW and expressed the weights in the eigenbasis
w J, W B. Therefore, we obtain for the order parameters
((1) JJ
ΣJ N
JWDWW J
N
R() = J ΣB
J WDW B
jTDB
(86)
N
N
N
28

---

Published as a conference paper at ICLR 2025
10a
101
10
10
10
cifar5m
10
synthelic
10a
10
10a
16a
Figure 14: Left: Spectra of the feature-feature covariance matrix of CIFAR-5m data set vs. Eq. for 3 = 0.3 and N L 1024.
Note that the expectation values for the order parameters and, therefore, for the generalization error still the same.
We use the parametrization from Eq. (85) to obtain the results shown in the right panel of Figure Thereby, test our prediction for the generalization error from Eq. (84) to student network
trained on CIFAR-5M images using approximately 10 (see Nakkiran et al. (202)) for details on the dataset). We use only the first channel of the images resulting in total input dimension of N = 1024 after flattening. To approximate the true covariance matrix Σ, we numerically estimate the feature-feature covariance matrix based the input examples from the training dataset. During training, we update only the first entries of J, while resetting the remaining entries their initial values after each iteration. Based on the spectra of the feature-feature covariance matrix depicted in Figure we estimate 0.3 and use this spectrum evaluate Eq. (85).
B.3.1 DEPENDENCE ON STUDENT SIZE FOR GENERAL COVARIANCE EIGENBASIS
In this Subsection, consider the generalization weights and N N, random weights. Thereby, we no longer train the weights of the student the eigenbasis of the data-covariance matrix. In order distinguish between trainable and untrainable components, we decompose the student vector J in two parts: the trainable components J and the non-trainable components J. For the covariance matrix we identify the following structure in block matrix form
evaluated for student with N, trainable
S
Ecross S
Σ
(87)
Σ
where
E R NiN represents the submatrix of the covariance matrix, acting exclusively on the
subspace spanned by the first Nj components the vector J, denoted as J.
Σ (NN-N))(M-N, refers to the submatrix corresponding to the subspace of the remain- ing N - N₁ components of J, denoted as ...
Exoos (N-N.) represents the cross-covariance matrix part, describing the interac- tions between the subspace of the first N₁ components and the complementary subspace of the last N Ni components
The evolution of the student vector J is governed by the differential equation for small learning rates
N
dJ,
Eak (B - Jk)-
(88)
da
The solution for the trainable components j(a at time is given by
J(a) = - Eh jo + 11 10 E E (B-j) (1- -1Eo)
(89)
29

---

Published as a conference paper at ICLR 2025
Ni
N 150
N, 100
N. = 200
10-1
10-2
100
10
10
10
10*
Figure 13: cy as a function a for different trainable input dimensions N₁ of the student vector with L = N 256, K = M = 1, Ø. 0.01, 1) 0.05, and / 1. Here, train the student outside the eigenbasis of the data covariance matrix. We compare results based on simulations (solid curves) the theoretical prediction from Eq. (93) (black dashed lines). The student network is trained synthetic data, where we average over 300 different initializations of student and teacher vectors.
where j is the initial condition for the trainable components, and the non-trainable components kept at their random initial values. Moreover, B and B are the components of the teacher vector equivalent to the trainable and non-trainable parts of the student vector, respectively. At stationarity (a >> oo), the trainable components become
B B ( (B-J).
(90)
For the first-order order parameters, we find
ROO
J. ΣB
JJBB + j Eeoos B + J -T
B f EB)
cross
Q EJ E.J
N (JTJ SJ +
(
(91)
Taking the expectation over and j yields
(R()) p..: N (Tr(E) + Tr T Σ --enos))) cross
(Q B... (TT(E) + (1 03 TO H-1 Eenos)) + +TTr(EE)
20 [IT 5-15 Ecross)
))
(92)
Finally, we obtain for the generalization error
of Tr
(ggasyyp) B. 2 Tr D + T3) N
ΣT
T $-1 Egnoss Cross
(93)
Note that for diagonal covariance matrices E; S, Eq. (93) reduces to Eq. 110) for
Figure 15 presents the generalization error obtained from simulations for various Ni as a function
of In this comparison, the asymptotic solution derived from Eq. (93) aligns closely with the simulation results, demonstrating excellent agreement.
Next, we compare the numerical solution of Eq. (93) with our theoretical prediction for training the student vector in the eigenbasis of the data covariance matrix, given by Eq. (79). The results are presented in Figure [6] For a fixed number of trainable parameters, Nr, the generalization error is consistently lower when the student is trained in the eigenbasis of the data covariance matrix. This expected, as training in the eigenbasis aligns with the directions of the largest Nj eigenvalues, leading to more efficient learning process. Consequently, the overall generalization enror reduced compared training outside the eigenbasis. Under the condition N > N/8 observe the same
30

---

Published as a conference paper at ICLR 2025
10
10-2
Eg, asymp
Eg. as.asymp
10-3
10-2
8= 0.3
8 ==
= 0.75
10"
==15
102
10a
N
N
Figure 16: agasymp as a function of N; for different 3 and N I, 1024. The student is trained with synthetic data. We compare the numerical solution of Eq. (93) (dashed) with theoretical prediction for training the student vector in the eigenbasis of the data covariance matrix, as given by Eq. (79) (solid). The numerical solution is averaged over 100 different covariance matrices.
power-law scaling for the generalization eπrоr, Gg,ssymp X N₁ more general configurations, we find that the scaling behavior of the generalization error remains consistent across both setups. The data covariance matrices are generated by W WAWT, where Ag A with eigenvalues defined by Eq. ① and W is a random orthogonal matrix with zero mean and variance which scales as (W N
in both scenarios. Additionally, for
C NON-LINEAR ACTIVATION
C.I DIFFERENTIAL EQUATIONS
Throughout this work, we consider the error function as our non-linear activation function (z)
erf x For this activation function, one can solve the integrals I2, I3 and 14 given Section A
v
analytically. Following Saad & Solla (1955) Yoshida & Okada 2019). one finds
Cie
12(C) arcsin
(94)
π
V(1+ Cn) + C2a)
1
C2a(1 + C11) -11) -12
IC(C') क
(95)
v(1 + C11)(1 + C33)
C7
1 Ci1
4
Ao
L(C)- n² AA
aresin
(96)
VAIVAz
where
A4 = (1 + C1)(( + Cz)) -c
and
Ag = A4Ca + C2 CisCua C22) + C2
4 = A4(1 + Cas) + C2a(1 + Cn) - Cia(1 + C22) + C12 Cza.
A2 = A4 + -
C..) C14 + C22) + 221221440o5
depending on the precise covariance matrix and its entries Cj as discussed in Section After evaluating all necessary covariance matrices, obtain for K M
31

---

Published as a conference paper at ICLR 2025
M
T 11)+1)
(1 + Qi+) R/y Rem
M 11
ijn (1+ 1 111-11 Qi)
20j
nm
do
MT
1 + Qii
v(1 Qss) (1 Tmm) Rim
(1+ Q+: (1 221) 0z
m
(97)
dQuk ik da
2m
R((+1) (1+ (1+ - ((1+11) Rim
M ((+1)
Qki (1 QQ) 1
km
MT
1+ Qii
+ Qūi) (1+ Tmmm)
R
V (1 Q1) + Qii) O2
(1+11) (1 + Qkk) - -(1+11) Rk
M Q (1+1) (1 QKk) - 0/ 11+11 Qui
1
+
Σ
1+ Qkk
V(1 + Qak) (1 + Tmmm R
(1+ QKk) (1 + Qúi) - 2%
+0 (1²).
(98)
where we have omitted higher-order terms in 7) for notational simplicity. Similarly, we have also omitted the superseripts for the first-order parameters, meaning that all instances of Rin Trm and
Qi without superscripts correspond to their first-order forms: R2(1) T(() and Q Note that the differential equations are closed given Equation 98 For the last component L In the same
has to
apply the Gayyey-Hamilton theorem for the order parameters as discussed in Section notation, the generalization error becomes
Eg
1
Σ aresin
Q
Σ
aresin
(1) T
mm
7
1 Q V Q
+ T (:(1)
R2(1)
2
Σ
aresin
in
f.R
Q
(1)
V 1
T
(99)
which just depends on the first-order order parameters. This system is gonna be analyzed in the following.
C.2 FLUCTUATIONS IN ORDER PARAMETERS
For isotropic input data, the order parameters are self-averaging, i.e., as increases, the fluctuations of the order parameters decrease, and in the thermodynamic limit N - one can replace these order parameters by their expectation values (Saad Solla 1995). However, for structured input data, fluctuations in the order parameters can still be present. In order to show this, we calculate the relative variance of the teacher-teacher order parameters as a measure for fluctuations.
For the expectation value and variance of the teacher-teacher order parameters (())) Br. (Σ) B.n we obtain
N
(T()) nm
8um N
B..Bn
)²) Ba.Bm
(1 + 6mm)
N
((ll) (TT()) Bn, Bm
Bm.
(100)
N2
where we have exploit that B,, and B,, are independently distributed random vectors with B N/ (0. for each entry of the vector B,. Note that the variance for n m is twice as large as for n * m. After substituting Eq. (T) into Eq. (100). we find
X
(T() B. 5mm
M(++8)
km
(TT()) (Trlli Bm )²) Bm (1 6mm
21
(101)
NL
6211+0]
32

---

Published as a conference paper at ICLR 2025
Next, we discuss the dependence of X+ L and N. Due to the normalization condition
N E Lk=l d L, we can infer Ei= d L Σ E and thus x Ei= 20rD
A
O (L) since Σ go-a a convergent sum. Thus, we find
X
A
[ N
O(L-)), WO+5)
0
(102)
T
NL
22()+5)
Therefore, for the relative variance for n = m, we obtain
(Tlh (Tm)) B.
2
TTi 2 B..
(103)
N
B,
This result indicates that we need to distinguish two different scenarios when taking the ther- modynamic limit N oo: i) when t + 0, the relative variance vanishes, and the order parameters become aelraaiggg quantities, and ii) the case where & = O (1) stays finite, and the teacher-teacher overlaps are not self- self-averaging. We note that the case of isotropic inputs is recovered for T. = 1, where the relative variance vanishes with 1/N.
For the case = 0, we find Elen pC(+))) = Ek 1 = L which is no longer converging. Therefore, we obtain
(IIM) (TTim) Bin )²)
B.
(104)
(T / ) B,,
N
which decreases with 1/N.
C.3 PLATEAU HEIGHT
Here, we consider the differential equations given in Eq. (98) for the higher-order order parameters up to the first order in the learning rate O (11) As already mentioned in the main text, the order parameters are no longer self-averaging, i.e. we cannot replace the random variables Tı()) by their expectation value in the thermodynamic limit N + co. However, without any assumptions the teacher-teacher order parameters, we find approximately the following fixed point for =
M
R
(L)
Qa(i
Σ
M T
(T T:l
+ (1 T -11
M
MT T
(1
+
+
T
(105)
and for l +
T(())
1 M
T,(()
Rim
(1)
R3
Qa(1)
Σ
M
MT(()
(1+
T -1)
M
MTQ.
[
THe +d!
THU +
THE
+
TH
(106)
with ( = >M> T.(1) Thus, for the student-teacher order parameters, we obtain M different
plateau heights for each order 1 depending on the sum over the off-diagonal entries of the higher-order
teacher-teacher order parameters (1) and 7:(1) This approximation is exact if all 7-(0) D for
m and T.. (1)) and all plateau heights are the same R R)) In Figure 17 compare
the numerically found generalization error by evaluating the differential equations given in Eq. (98)
up O (1) with the generalization error based on the approximately found fixed points given Eq.
(105). For small L, we find very good agreement between the true plateau and the approximation.
In order proceed, make the ansatz for the off-diagonal entries TT(I) D N Σn M (0)
33

---

Published as a conference paper at ICLR 2025
1
106
1000
1750
Figure 17: Comparison of the generalization error plateau evaluated by numerical solutions of the differential equations given in Eq. (98) (solid) and approximated fixed point given by Eq. (T05) (dashed) for 7000, K M 4, O. 0.01, 3 = 0.25 and rj = 0.1. We solve the differential equations up to O (17).
for n * m. This approach preserves the statistical properties of the sum represented by a single parameter D Moreover, for 1 0, the teacher-teacher order parameters still self-averaging in the thermodynamic limit, and we can assume (T()) = and T = 0 for large N. Thus in summary, we assume T 6um and T(U = 8mm + (1 6mm ) ( for 1 * 0. After these assumptions, we find new stationary points for our differential equations for
Re
Q)(1)
1
(107)
MT 1 Day (1 T(
M
MT() T
(1 + イ - -1)
D
and +
R
T()) D
MT( - T
(1 (1+ -
Q 10)
T(1) D
MTU:
(1 + T(oT)) -1
T())
R
T())
M
MTU T04DD
(1 T 1
T())
Q 10
(108)
T())
MTO(
(1 ( -1
T04
As one can see in Eqs. 107). and (08). we end up in one plateau height for the order parameters
0 Q ()) and R R 10 In addition Figure 4 given in the main text, provide the plateau behavior for higher order-order parameters in Figure T8 and compare our newly obtained stationary solutions given in Eq. (108) with the solutions of the differential equations given in Eq.. 988 up
to O (7)) We observe that the student-teacher order parameters defined in given in Eq. (108) yield an approximation for the mean value the M groups of order parameters. For the student-student order parameters, we observe a small systematic error which appears to be small compared to the magnitude of Q=!")
34

---

Published as a conference paper at ICLR 2025
6.
4.75
20
4.74
250
230
500
750
1000
10
1000
2000
3000
4000
5000
6000
7000
1000
2000
3000
4000
5000
6000
7000
500
500
130
400
(0)
300
250
300
230
750
1909
200
200
100
100
-100
1000 2000 3000 4000
5000
6000
7000
1000
2000 3000 4000 5000 6000 7000
a
Figure 18: Plateau behavior of the order parameters with L 10, N = 7000, Ø. = 0.01, 7 = 0.1 and M K 4 for random initialization of student and teacher vectors. We solve the differential equations for the small learning rate case where we consider terms up to O (n). The inset shows the higher-order order parameters at the plateau. For the student-teacher order parameters, obtain M different plateau heights. For the student-student order parameters, we observe one plateau height with small statistical deviations for the particular matrix entry Q The dashed horizontal lines in the insets show the plateau heights according to Eq. (108).
Next, we insert Eq. (107) into the expression of the generalization eтror and obtain
1
D( + T())
D(1)
E
M arcsin
MTO) + +M/
+ (M - 1) aresin
+1)+1)
2M aresin
(TO) +1
(DW M M (M M)
MT
(Di +27 20 +(1 071
(M 1)2 DDD
T()) T()) +
+ arcsin
(109)
for the plateau height. The right panel in Figure 4 shows an example for the estimated plateau height given by Equation (109) against the numerical solution of the differential equations.
C.4 PLATEAU ESCAPE
In this subsection, we want to present the escape from the plateau. The found stationary equations given by Eqs. (107) and (108) unstable such that after certain time the plateau, the generalization error will eventually escape from it. In order to escape from the plateau, the unique solution of the fixed points must be broken for each / certain time. We want to study the dynamics in the vicinity of the fixed point and clarify how the generalization error leaves it. For this, introduce parameters (1)) and (t)) indicating the onset of specialization for the student vectors towards one teacher vector. Therefore, parameterized the order parameters by R! R( + (1 8,m). Q = Q 6,5 + cO (1 - ō:z). Moreover, to study the escape from the plateau, we introduce small perturbation parameters p!) (1) (1)) and (()) modeling the repelling characteristic of the unstable
35

---

Published as a conference paper at ICLR 2025
fixed point. Thus parametrized the order parameters by their stationary solution and small
perturbation R()) R + r(() (()) S + +s, =Q + and (1)) C + cl0) with S- = R. and C (t) = Q0 Next, we insert this parametrization into the differential equations given by Eq. 98 up O (1)) In order study the dynamics in the vicinity of the fixed point,
linearized the dynamical equations in (r((), s(0, g(), ,e()) around zero.
In the following, we present the differential equations, eigenvalues, and eigenvectors for the specific case where D = 0 and T = 1 for notational simplicity. The full system is too large to display in its entirety. However, we provide insights throughout on how these results generalize. After reducing the system of differential equations, we conclude by presenting the full solution for the general case.
After first-order Taylor expansion in (r((), s(1) q( c00)) T.
we find the following linearized equation
d
2M
Ap
(110)
da
q
TM 2M +
with ri p(i-1), 5; s(i qi =
qi--1), c (i--)) and Ap
A + B with Ap E R 1444L
The individual matrices can be written as Kronecker products A U =ue and
G 8 Ab, B= H @ U with
M -
0
M
G
2
M
-2
(M 1)
2 1)
(111)
2M
2M-1
M
-2
2M-I (M - 1) 2( 1)
2M
-2M(M-1) 1]
2M +3M-2
(M-1W2M-1
(2M+1)(2M) 1)
(2M+1)(2M
VMM /2M 2
M(22++))
-2M
M(2M
2M +3M-2
(M- 1y/2
H
12M (2M+1)2M-)
(2M+1)2MM1
(2M 1)v2M
MM(24+1)
(112)
-2/77
-2
M+2
2(M-1)
2M+1V22-
(2M+1)/MM
2M++
2M
-2VM
-2(M11)MM
M+2
2
2M+1)/2M-1
(2M+1VV2M-
2M+1
2M+1
We obtain the following eigenvalues and eigenvectors for G
2/M
M
AG,1 = M,
VG.1 = 1, V2M -I I V2M
(113)
AG, 2M,
VG,2 (0,0, 1,),,
(114)
AG.3 0,
VG,3 1, M
1
0.0)" vG,1
(0.0,1,
1
(115)
M
and for H
M
AH =
4M² TT'
VH,
1, M - 1 0,0
(116)
T
2M
2/M
2
An.2
VH2
1,1,
(117)
2M +
V2M - V2M -1
M3/2./210 -1
12M + 3M 2
AH,3 = O,
VH3
1,1,0, 2M2 3M +
VHH
(0,0,1
22M²
3M +
(118)
36

---

Published as a conference paper at ICLR 2025
Since A is of block matrix structure expressed by Kronecker product, obtain for its spectrum A = A and corresponding eigenvectors a = aG @ VA, for which we multiply each eigenvalue of A, with each of G. The same also applies for the cigenvalues and -vectors of B: AB A4 and corresponding eigenvectors dg = DG @ DA,- 'he eigenvalues of A were already studied in Subsection [.. and are the negative eigenvalues of the data covariance matrix - Xx with cigenvectors summarized by the matrix V (cf. Eq. (33)). Since A possesses L eigenvalues and -vectors, obtain multiple groups of different eigenvalues and -vectors for A and B. The eigenvalues and -vectors of U are also already known. We have one eigenvector u with eigenvalue Au T and L, - 1 eigenvectors e, with zero eigenvalue for = 1,3,4..., 4..., L. Thereby e: is the Ith unit vector. In the following, the superscript for the eigenvalues and -vectors indicates the corresponding group.
For A, the first two groups of eigenvalue combinations A A, = -M X with eigenvector (1) AAK (1%, ok 20% /2M-1 plateau attractive. Their corresponding eigenvalues are negative and their directions are against the breaking of order parameter symmetry. The latter fact be seen that the first two entries of the eigenvectors and the last two are the same. This would drive the dynamics in the direction corresponding to y!!)) y()) and q()) = c!!) which is exactly the plateau condition. The third
JM 20k /2M-1
VM
and A (2) 22 with v U'A (0, 0. Uk. vx)
group of eigenvalue combinations X (3) 0 with eigenvectors A.
(3) A,
(Us. 2k 0.0) and
A.A 0,,0, Uk, -th -
are neither attractive nor repelling. However, their directions indicate symmetry breaking in the order parameters, at least for group y(() + s(0) q()) # cl)
(4)
(1) O, with total of 4L distinct eigenvectors. However, For the matrix B, we observe that AB the significant impact comes from the directions associated with non-zero eigenvalues. These
eigenvalues play a crucial role in influencing the spectrum of A, particularly when viewed as non-negligible perturbation of A. The second eigenvalue of B is λB.2
with eigenvector (T
4M2-1
UB.
(u, 14.-Ukp M- 0,0)
For the third one, obtain λB,3
2MT
2M+1 with eigenvector
2/M BB, u. u, V2M-I U,
V2M-I"/
In summary, obtain two important directions for the escape of the plateau. The first corre- sponds to the eigenvectors v a and UB.2 and the second one is in the direction of UB,3 and A Note that these directions are also present for the sum A B resulting in Ap. Therefore, we make as a first ansatz (() = c(l) since this condition is fulfilled for all important eigendirections. Moreover, we can make the following second ansatz (²)) (0) 2/2 second ansatz fulfilled by both eigendirections as well. For D
(3)
(r-(0) + (M - 1)) The and general 7.0 find
with similar steps the relation g = c( Tu R² (r-1) + (M - 1))
27
Next, we re-parametrize the dynamical equations under the condition (()) 2R (5) (1-0)) + (M- - 1)s(0) and find
c(
2 TM
2M
rr)
Ars
(119)
da
2M
+
with A,s = A+ B and A, E 222xx2 The matrices A and B are re-defined as follows:
= G & A
B
(120)
with re-defined G and H
5M
4M 7M +37
(M - 1)
H (2M + +1) (2M-
1M2-7M+3
2M 2 6.M +
G=
(M - 1]
(121)
M-1
37

---

Published as a conference paper at ICLR 2025
0.012
M =
32
0.010
M - 8
M 40
M = 10
0.008
0.006
0.004
0.002
100
4000
8000
12000
16000
a
Figure 19: Plateau behavior of the generalization error eg from simulations with a single random
initialization of student and teacher vectors. Parameters: M, = 0.01, a_ = 10-6, and = 0.25. The results are shown for varying student-teacher sizes with L N 512.
For A, the eigenvectors Aa given by da DG MA where AA - and UA 2y for k (I,...,,, L). The corresponding groups of cigenvalues are A, -MAx with eigenvectors (1)
AA (Uk, vk) The second group is given by A 0 and the corresponding eigenvectors are (2)
0AA (Uk, 0%
For B, the first eigenvalue AB,1
(2M (2M-1)(2M+1) with the corresponding eigenvector 2MT(3)
MT
Un u. w-1) The second eigenvalue is An,2 2M+T with eigenvector Un (u, 2 Furthermore, we have multiple eigenvectors for the eigenvalue zero. For E (1,3,4..... we have AB,m = 0, with corresponding cigenvectors UB.m = (em, M-1 Similarly, for E 4), AB.. = O, with cigenvectors VB.n = (en, Note that all eigenvalues were already encountered for the larger system verifying our analytical ansatz. Moreover, the new eigenvectors are the first two entries of the eigenvectors of the large original system.
Due to the special structure of A, the eigenvector UB,1 are also eigenvectors of A, both associated with the eigenvalue 0, meaning AUB.m = 0 and AUg, = 0. Among the eigenvalues of Ars, 2L MT(: of them zero. The first non-zero eigenvalue is AA... (2M-1)) (2MM1) which is larger than zero indicating a repelling character for direction UB,1 and v For the eigenvector UB.2, we have AUB,2 = M (Aiu, Aru) Therefore, UB.2 is an eigenvector of A, provided that u is an eigenvector of A₁. For the product, we find A u with u. (T ,.., T((+1)) T.
A (v + Un.2) ( = Ag (wy, vg) M (u+, ) and this group eigenvalues is therefore negative.
Finally, we obtain one important eigendirection showing an eigenvalue larger than zero This (2)
direction corresponds to UB,1 and U Therefore, we make the following last ansatz s()) M-1 in order to reduce the system for second time. Note that the exact same relation also holds for D # 0 and general (1).
(
For the final form of the differential equations, return to the case where D * 0 and T & 1, as the expressions are now more manageable to display and no longer excessively large. The final re-definition of the dynamical system yields
dr
ng, Ür,
(122)
da
(D (T())
with 9r
2
Ú
üej and ù =
ạ V(M- D (())0
(T())
((
(2)
M-1 .... (L) M-1
(
Note that we define A, G for the main
M-I.
38

---

Published as a conference paper at ICLR 2025
text. Since U is rank-1 matrix, we obtain L zero eigenvalues and one eigenvalue
D
A, = T
(123)
M
larger than zero. Thus, Ar drives the escape from the plateau. We can solve the differential equation directly and find for the first-order perturbation parameter
r (1) = e 1 (1)
(124)
where To (an) and ag denotes an arbitrary time the plateau. For the escape of the generalization error within our re-defined dynamical system, we find
((M 1)r + M DD (DO) +TO))
(1) (125)
Eg Eg
THne TO
8 (M - 1) T (DC44T0))
4M2
where we have introduced the escape time
1
Tesc
1g
π VMM 1)T()) D() + M (DW) (M 1)T()) + M)
(126)
2n
((1)) M-T
Diai
(DL) T(1)
X² Σt ा ~ I M² n..
x2 T. for large L. The
Furthermore, we can approximate ((2) Tr(E²)
same applies to D22), For large M and L, we find Tesc Figure D presents the plateau scaling observed in simulation experiments, confirming the predicted
M²2 In addition, Figure 19 highlights the plateau dependence relationship for the escape time Tesc nI" under different configuration.
C.5 NUMERICALLY ESTIMATED PLATEAU LENGTH
In this subsection, we demonstrate how to combine the analytically derived formula from Eq. 12 with our calculated escape time (Eq. (15) to estimate the plateau length. The plateau escape is described by the equation
aP au = D in + In (N)).
(127)
where D is constant of order (()), dependent on the variance initialization and the plateau; on is an arbitrary starting point on the plateau; and Tesc is the escape time from the plateau. To estimate the constant D, we interpret the results of Bichl et al. (996) and find D In ( (.). where B represents the deviation of the order parameter responsible for the plateau escape at ap from its value at ag- Thereby, e a proportionality constant between the fluctuations at the plateau and at initialization. In our case, the order parameter that drives the plateau escape is the first-order student-teacher order parameter (cf. Subsection C.4b. Thus, we define B = R (ax) R()) (a Additionally, to estimate
we set (ap) = cH following the definition of the escape time for the generalization error. Next, we estimate op , where the plateau variance op derived from numerical simulations of (1) the plateau.
For the example shown in Figure 4 we set on = 300 and find ap & 0.000279, with e RJ 0.0279 (since 5_ 0.01 is given), B 0.33, and = 2.47. For the escape time, we find Fesc 239 by averaging the diagonal terms of (1) to obtain (1) and using the averaged sum of the off-diagonal entries in order to estimate ( and ( Finally, we obtain ap ag = 2751.
This procedure provides valuable insight into how the plateau length behaves with respect to various parameters. Figure husshws the generalization error based on the solution of the differential equations and presents additional examples for the plateau length estimation.
39

---

Published as a conference paper at ICLR 2025
M=2
M=4
M
M=
M
0.0071
0.008
0.0070
0.006
0.0069
0.002
0.0067
L=2
L
L-4 8
0.000
0.0066
1000
2000 3000
5000 6000
7000
2000
40000
6000
8000
10000
Figure 20: Plateau phase of the generalization error evaluated by numerical solutions of the differential equations for one random initialization (solid) and plateau length estimations given by Eq. 127 (verticle lines) for N 7000, K M σ. 0.01, 3 0.25 and 0.1. The black verticle line indicates the arbitrarily chosen plateau start 500, and the colored verticle lines show ap for different L. We retain terms up to (1)) for the differential equations.
C.6 ASYMPTOTIC SOLUTION
Here, we want to investigate how the generalization eπror converges to its asymptotic value in more detail. For this, consider the typical teacher configuration (T 8mm since this configuration already captures the scaling behavior of the generalization error. For the asymptotic fixed points of the order parameters, we find 12 T
6,.. and Q -T 6eg. Here, we distinguish again between diagonal and off- off-diagonal entries for R!!h 8 (1)) 8mm) and Q 6 + (*) (1 - -iy) as for the plateau case. Furthermore, we linearized the dynamical equations for small perturbation around its fixed point π! = T + +, ,0) 5()) = T + s(0, Q T + (0) and ( (!)) + c(t)
We find the following dynamic equations
d
2v3
A
(128)
da
3
q
with p(--1), 8; (i-1) 9 (1-1), C2 d- and A.
Á B gC and
"
2v3( (4-+5(M-1)
13rM
The individual matrices can be written as Kronecker products À
x
A
B=H
Č F U with
//
0
0
v
33
0
3
G
H
8
(129)
-2
v/
2
v3
v3
1
v3
2
v3
2
v3
4
0
0
0
0
0
0
F
f1
fa
f1 fa
(130)
-2
1
2
f
fs
fi f4
fo fo
V3
2
2/3
15(M - 1)
15
where f
V6 + M
22
/45 + 5(M - 11,
S4
.s
5
5
4(M - 2) 3M/2 15M + 26, fo
Thereby, A₁ and U₁ are the same matrices as for the
5
40

---

Published as a conference paper at ICLR 2025
linear activation function case. Therefore, the linearized version of the dynamical equation for the non-linear activation function resembles the dynamical equation for the linear activation. However, we encounter an additional "perturbation" by B, whereas C describes the influence of higher-order terms in the learning learning rate. Moreover, compared to the linear case, the differential equation has more variables due to correlations between different student and teacher vectors. In order to analyze the behavior of the dynamical system, we need to determine the eigenvalues and eigenvectors of all sub-matrices. Here, we analyze the system for first order in the learning rate O (7) neglecting the contribution by gC.
The eigenvalues of G are AG = 2 v3 with eigenvector v0.1 (0, AG = 2 v3 with eigenvector UG2 (0,0, 1)", λG,3 1- 1/3 with eigenvector UG (1, - 1, -2) AG.1 + } 33 with eigenvector vG,4 (1, 21) The eigenvalues of A₁ were already studied in Subsection B.1 and are the negative eigenvalues of the data covariance matrix -ÅØ. with eigenvec- tors 8g summarized by the matrix V (cf. Eq. (33)). Since Á is of block matrix structure expressed by a Kronecker product, we obtain for its spectrum A = AGAA, and corresponding eigenvectors v Ug @ VA, for which multiply each eigenvalue of A with each of G. Thus, we obtain four different groups of eigenvalues for Á and in total 4T eigenvalues. The first group is obtained by multiplying the first eigenvalue of G with all eigenvalues of A leading to X *,,k =- (2 (2-v3) X
with eigenvector v (1)
A. (0,0, va, -o²) With the same procedure, we obtain for the other
groups (2) - (2 + v3) d with eigenvector e AA (0, 0, 2k; Txx), A (3)
(2)
A.A = (1 vv) X
with eigenvector v A.k
- 313 Å with eigenvector A. (Uk, Uk. 20k. 2vx)² The upper index for the eigenvalues and -vectors indicates the corresponding group.
(3) = (ok: - Uk. 20x; - -2x) and A (4)
(4)
The eigenvalues of H are
XH =
1 -
v/3 3
with eigenvector UHH
12
T
L ( (v43 2, 2 9
(V43 +4)
AH.2
1
+ 12
with eigenvector UH2
3
(1, v3 (V43 + 4) 2, 2 (V43 + 4) and AH.3 with eigenvectors Un.3 (,, 2/3
and 0H14 = (0, For the matrix U, we have just one eigenvalue distinct from zero Au. (22) with eigenvector u since U has rank one. The remaining eigenvectors are given by ve.1 = e, for / E (1, ..., L) and especially + 2. Thus, we obtain two different eigenvalues distinct from zero and 4L - 2 zero eigenvalues for B. The eigenvalues distinct from zero are
AB.1
((2) and Un.1 u, 9 43 + u, 2u,
2v3
(V43 + 4)
u and
AB.
1 3
+
43 (2) and Ug2 u, v3 (V43 4) u, 2u, 2v3
43 4) 2 Then,
12
have two eigenvectors with zero eigenvalue UB u, 0,0, 2/3 and U8 (0, u, 0,
Further eigenvectors have the structure e @ UH.1, ej @ UH22 ej & UHS and e @ HH
Strietly speaking, none of the eigenvectors of Á and B are the same and we cannot calculate the spectrum of their sum directly. However, we notice that B has just two eigenvalues distinct from zero and that its corresponding eigenvectors UB and vg are of the same structure as the last two
groups of À namely i and (1) For each of the vectors, the third and the fourth components are twice as large as the first and the second one. Therefore, only the eigenvalues of the last two groups of Á are influenced by adding the matrix B leading to the eigenvectors of Ag with the structure
(3)
2 w (3) 2,,,z 2w0
(3)
(4) and v. (() = 2 w
(4) 22 2w. (1) with vectors 2(()
w , 26 and unf that has to be determined. Moreover, since the other eigenvalues of B are A.
(3) (4)
zero, the eigenvalues of the first and second group (1) and (2) remain the same for Ag as for A. However, the corresponding eigenvectors are no longer analytically determinable and we have to rely on numerical solutions. All these claims for the spectra and eigenvectors are in excellent agreement
A.A
41

---

Published as a conference paper at ICLR 2025
1=3
10
10-9
10
10
10
10
Simfustion
10
Simluation
Theery
Theery
1000 1500 2000 2300
3000
1000
1000
Figure 21: Left: Generalization emror function of obtained via the solution of differential
equations of the order parameters up to ((n) and K = = 2, 0.25, O. = 0.01, , N = 9000
and B = 1. Center and Right: Comparison of the first-order Taylor expansion of the generalization error based on Eq. 131) for numerically obtained
q00) and c( (Simulation) with Eq. (322
(Theory) where the initial conditions (()) (oo), (ao) and (ax) are obtained from simulations. Thereby, we choose ag = 1000 and parameterize a -pp- Center: I. Right:
== 9.
with numerical experiments.
Next, we Taylor expand the generalization error up to first order in the small perturbation parameters p()), s(1), ((1) and c(!), We find
Eg = Gπ
(2v3g -1 + 3 1) c()) 6 (M - 1))
(131)
From this expansion, we observe that the eigen-directions v A and v A_..k do not contribute to the generalization error in first-order since their components cancel out. After inserting the expressions for the first and second groups of eigenvectors, obtain
Eg
(1) (2-v3)Ako (2V30 ()) - 4v30 - (1)
U4,L44 (M
2k
(
(2)
e (2+v3)Anaa (2v3u - + +3 02,3422 - 6 6 - 11) 1) uk,
(2)
(2)
+ Jb
(132)
where 2 E R4L are the eigenvectors of A, to the eigenvalues - -(2 v x
(2)
and - -(2+ v3) dp. respectively. Thereby, (1) = Σthh ((vi)))))
(2)
Tt. 9k
Eth (((12)
where v and (2) containing the first and second group of eigenvec-
(1) and Uk respectively and æ = (r(an), s(ao), ((ao), c(ao)) E 4 is some reference
(2)
tors Uk
point at arbitrary chosen ag in the asymptotic phase. The asymptotic convergence is governed by the smaller group of eigenvalues (2 /3) Au-
Figure compares the generalization error derived from the first-order Taylor expansion in Eq. [31 where we obtain the parameters r, r. s, e by solutions of the differential equations and our theoretical results based on Eq. (I32p. For both, we use the same imitial conditions, with ag = 1000 as in the numerical solutions. The comparison shows excellent agreement. The small discrepancies between the graphs arise from the arbitrariness of (gg and the chosen initial conditions. Similar to linear activation functions, we observe a slowdown in convergence towards perfect generalization as increases, which eventually leads to transition from exponential convergence to power-law scaling. We solve Eq. (32) using the Julia programming language; however, we encounter limitations in increasing L due to constraints in numerical precision (see Section G)
Rather than solving the differential equations for the order parameters, we next solve the differential equations for the student weight vectors directly. As for the linear activation function, we approximate
42

---

Published as a conference paper at ICLR 2025
10-1
10-2
0.5 0.5
8= 1.25
10
0.75
8=15 1.5
8=1
a
100
10
10
10
10*
10
Figure 22: Generalization emor as a function of a obtained via the solution of differential equations
given in Eq. (13 for K M 2, K N 1024, 0.001, and Ø. 1. We find the same
-0
scaling exponent as experiments and for the linear activation function a =+8 We average over 50
seeds of student and teacher initializations.
J. de
-1 V Jeg and obtain
Bin (1 (1+Q!!)) -J.
dJ:
24
M
da
MR
( 01!1!
mm=1
V(1 Q.!h)) (1+thhh)
(R!
Jj 1 Q.11
(1+ - -JQQ.
Σ
(133)
-!!h (1 +
(0.))
(1)
The results for the solution of Eq. 1533 are given in Figure
22
We find scaling exponent of
Cg X a consistent results obtained from simulations.
D OVER-PARAMETRIZED SCENARIO K > M
Here we present numerical results for the over- peer-parametrized scenario > M. The left panel of Figure 23 shows the scaling of the plateau length of T(2), As for the realizable scenario, find linear dependency on the inverse of (22) predicted by Eq. 155) Note that T(2) L for large L. The right panel Figure shows simulation results for a student-teacher setup for > M, where we train student network based the outputs of teacher and synthetic input data with power-law covariance matrix. We obtain that the scaling law in the asymptotic phase is not altered by areepparametrization. Moreover, increasing the learning leads to curve collapse, as shown the inset. Thereby, we use the same initial student and teacher weights as before and choose the learning rate such that the ratio # is constant. For this example, we choose 7 = 0.005, 7 = 0.01 and = = 0.02.
with the student size K for fixed M
The phenomenon of curve collapse can be understood as follows: for constant teacher size M, an increase in the student size K results in scaling of the student output, which is proportional to " (cf Eq. (2)). Given that the output weights remain constant, this scaling reduces the overall learning rate.
VMM
For the werrpaaameerrized scenario, (Richert et al. 2022) demonstrated that the asymptotic gen- eralization error obeys a power-law scaling Co & even for isotropic input data. This can be characterized as an "intrinsic" power-law since depends on the architectural configuration. We
find that the asymptotic power-law < is independent of K > M,, meaning that the plwer-law determined by the data covariance matrix is not altered by the intrinsic plwer-law -2, This likely occurs because the power law exponent is consistently smaller than the intrinsic power law exponent 8
B4I 2.
43

---

Published as a conference paper at ICLR 2025
10-?
L=1
K=2 K=8
10
K=4
100
10-3
1000
10
700
10
10
3.30
0.30
5.5
10
200
400
600
800 1000 1200
100
10
102
10
10*
a
Figure 23: Left: Generalization error as a function of a for the over-paramerrized case obtained via the solution of differential equations of the order parameters up O(1) for K 4, M 2, 1j 0.1, OJ = 0.01, N = 10 and 3 = 1. The inset shows the measured scaling of the plateau length with the inverse (2) for this single initialization, where we find a linear dependency. Right: Generalization obtained for the training of student-teacher setup for = 0.01, ._ 0.01, N = I, = 1024 and 3 = 1.5. The inset shows the curve collapse for the scaling of the learning constant. For this example, we choose 7)K-2 = 0.005, = 0.01 and T=== = 0.02. We average the generalization error over random seeds.
error function of for the over-vovveae parametrized
with K. We choose the learning rate such that the ratio & is
E RELU ACTIVATION FUNCTION
For the ReLU activation function, (Stratt Bichl 2019) derived an expression for the generalization eπror analytically and analyzed the learning dynamics for isotropic input samples. The analytical expression for the generalization error for correlated inputs becomes
M
Q
(1)
(o!!"
Rg.RLLU
2M
4
+
2m
Qij arcsin
Vo
00
M
R
1
(T() (R()) + RD arcsin
M
2T
i.n
M
T
T((
+
1
V
I mm
(1))(((
-(Irl))
+ Tom aresin
2M
4
2m
(1) T (I)
m,
(134)
again for the soft committee machine in the realizable case K M. In the following, we solve the differential equations for the student weights rather than for the order parameters numerically. As for the error function and linear activation function, we approximate dJi de -n JCg, and find
J. Q (1)
(p(2)) 8m
dJ, de
11
M
B aresin
RD Q Tum
B +
Σ
M
2TQ
2x
(1)
4
m=1
is
Q (1) 0 ())
J.y
Q (2) (1)
J
+ J 4
aresin
(135)
2x QTQ
2m
and solve this equation numerically. The top left panel of Figure shows the generalization error for the ReLU activation function for different L. As for the error function, the plateau length decreases as the number distinct eigenvalues L and the plateau length is inversely proportional to (?) oc However, formal proof has to be provided.
44

---

Published as a conference paper at ICLR 2025
L=1 1 =
10-1
L = 2
L -
10-1
L = 3 L
1002
W
10-2
8 0.5
8 = 1.25
10-3
8 0.75
B- 1.5
8=1
a
500
1000
1500
2000
2500
10
10
10
10a
10
10
10-1
10-2
10-3
0= 1.25
10-4
0.7.
= 3
B=1
100
10
10
10a
104
a
Figure 24: Generalization eπrоr function of a for the ReLU activation function obtained via the
solution of differential equations given in Eq. (135) (top) and experiments (bottom) for K = M 2. Top Left: Plateau behavior for 0.01, σ__ 10 N 10 and 3 1. Top Right: Asymptotic behavior of eg for K = N = 1024, 1/ = 0.001, and .ı = 1. We find the same scaling exponent as for the error function o πîş. We average over 25 seeds of student and teacher initializations. Bottom: We train student based the outputs of teacher network and synthetic input data for r 0.01,
σ, = 10-2, N L = 2048. We obtain the same scaling exponent as for the error function riz.
Moreover, we observe slow-down in the convergence of the asymptotic generalization error. The top right panel of Figure 24 shows the solution of the differential equations for T. N. We find the same scaling exponent for the error function शー These numerical findings are supported by simulations. The bottom panel of Figure 24 illustrates the results for training a student network by teacher and synthetic correlated input data with covariance matrix possessing a power-law spectra. For numerical simplicity, we solve the differential equations for diagonal covariance matrices. Figure 27 presents simulation experiments where student is trained on the outputs of teacher using synthetic input data for various activation functions, with K = M = 2. For all setups, we obtain the same scaling exponent for the asymptotic phase as for the error function a ifr. However, the time window for which the scaling law is achieved varies across different activation functions.
F GENERALIZED PERCEPTRON
In this section, aim investigate the numerical implications of training both layers of the student model on the scaling of the asymptotic generalization error. To this end, we examine a fully trained two-layer student model, where K which we refer to as the generalized perceptron. The teacher model in this scenario is single-layer perceptron. Our analysis will concentrate on the error function and ReLU activation.
We now redefine the student output to
=J. (z),
((B.)) = (v),
(136)
45

---

From Equation (37). identify the Published as a conference paper at ICLR 2025
10a
10-1
10
10-3
Eg
0 a
10-4
0.5
8= 1.25
== 072
B 0
1=55
10-5
P 1.25, a-004
8=1
1.5, a
10-4
10
100
10
102
103
10
10
102
10a
10*
10
a
a
Figure 25: Generalization enror as a function of a for the generalized perceptron for K N 1024, 0.001, and øı = 0.1. We average over 50 seeds of student and teacher initializations. Left: Solution for error function activation obtained from the normal distribution co . (0, 0.1). We find the same scaling exponent for the soft committee machine a T Right: Solution for the ReLU activation function obtained via Eq. (38). We initialize the output weight by co 0.5. The scaling exponent differs from that of the soft committee machine.
Eq. (137). We choose the initial output weight
where is now an additional trainable parameter. Next, we approximate the dynamics of the weights by dJ
Jeg and
de da
ss -1 VoE and obtain for the error function
dJ
2nc a (1 + Q())
R(), - B + Q())
J
~
E
da 1-20
(RO))² + (1+ (()) QQ1)
(2Q(
de
N
Q(1)
V11+ + (1)) )
1(2)
caresin
arcsin
(137)
da y-00
1 Q(3)
and ReLU activation function
dJ 2 -
Je
B
B
R(O)
arcsin
da 7-1
2
4
2T
VQm
1
T()) J - R((1) B
R(1)
Q B R())
2m
(QQITT( - 2
- - (R(W))
Q(1)
de
Q(1
(((I)
1
(Quraa (RECI)) + 12() aresin
Ra)
23 da n
-
+
2
4
2T
(Q
(138)
Note that the hidden-to-output weight operates on different time scale compared to the student vector, primarily due to the additional factor N. Figure 25 illustrates the solutions to the ordinary differential equations (ODEs) under consideration. Our analysis reveals that, for the error function activation, there is significant deviation from the predicted scaling law. Conversely, for the ReLU activation function, we observe an improvement in the scaling law when training the second layer weights.
One potential reason for the improved scaling could be the shifting of fixed points for ReLU activation, which is missing in the error function following fixed point for the error function:
J B and c" ==
(139)
which corresponds the same weight configuration that of perceptron student. Furthermore, in more complex architectures where K = M 1, the fixed point equation continues to yield the soft committee machine with oj and Jí B; for K = M and 1,.., K when the teacher output
46

---

Published as a conference paper at ICLR 2025
100
ReLU
10-1
A
0g
10
Reau
10
101
102
10a
104
105
10
10
102
103
104
10
a
a
Figure 26: Output weight and first-order order parameters as a function for both error function and ReLu activation based on the solution of Eq. (137 and Eq. (138). respectively. We show the solution for one random initialization for 7 = 1, N= 1024 and 3 Left: Output weight with initial value co = 0.5. Right: Rescaled first-order order parameters based on initial student vector coming from the normal distribution with standard deviation σ. = 0.1.
weights are one. However, for the ReLU activation, the fixed point that corresponds to the soft committee machine is no longer applicable and becomes shifted as shown in Figure 26 The precise value of this shifted fixed point necessitates numerical evaluation of the implicit equations
B c²
(R())*
J
and e" (Q(1)*
(140)
with et * in general. As result, the student network can establish a new weight configuration for the ReLU activation function more rapidly than it retrieves the "old" weight configuration for the error function that corresponds to the soft committee machine.
G REMARKS ON NUMERICAL SOLUTIONS
Evaluating large number of distinct eigenvalues becomes computationally challenging as L increases. The expectation value of the teacher-teacher order parameters is given by (Tith) = 5pm NV Tr(E)). In this context, the highest order trace term the differential equations L 1, and for large L. we can approximate Tr(E') as -I-I(_
π (A+) -- Since X seales with L for large values of L, the expectation values increase in a super-exponential manner with L. This growth also applies to the standard deviation of the off-diagonal entries of Ti(L-1) further complicating numerical evaluations as L grows.
As result, numerical investigations are restricted to small values of L. This limitation applies to solving the differential equations, evaluating fixed points, and analyzing the generalization error in the asymptotic phase. For instance, to evaluate Eq. (17) and generate Figures I0 through [3 we utilized Julia, high-level scripting language, with arbitrary precision arithmetic.
47

---

Published as a conference paper at ICLR 2025
10
10
10
10-2
Eg
6g
10-3
10-3
0.0.
B=
8=
1=15
B=1
8'
8-1
a Tiz
100
10a
10a
10a
10a
100
10
10a
10a
10a
a
10-1
10-1
10
10
10
10
10-4
0.5
1.5
10-4
8=
8=15
8-1 a
8-1
a
100
10
10²
10a
104
100
10
10?
10a
10
a
a
Figure 27: Generalization error as function of a for different acitvation functions based on simulations for K .M 2. We train student based on the outputs of teacher network and synthetic input data for 77 = 0.01,
10-2 N L. For all setups, we obtain the same
scaling exponent as for the error function a 1-s We average over ten different student and teacher initializations. Upper left: elu activation with N = L = 1024. Upper right: tanh activation
N L 1024. Bottom left: gelu activation with N L 2048. Bottom right: swish activation with N L 2048.
48


Reviews:
[{"review_id": "243eGtVGn7", "reviewer": "Reviewer_fBBP", "summary": "This paper studies the learning dynamics of committee machines trained on random data with power law covariance structure. They utilize a hierarchy of order parameters which they can analytically close for linear activation functions. They obtain scaling laws with training observations $\\alpha$ (also time for online learning) and model size $N$ when trained on spectra with $L$ distinct eigenvalues. The authors show a number of interesting effects including the disappearance of learning plateaus when $L$ increases and a transition from exponential convergence (for isotropic covariates) and power law convergence for large $L$. The authors demonstrate their derived theory is accurate for linear networks. They also examine the specialization transition from their hierarchy of ODEs and argue that the escape time scales inversely with $L$.", "questions": "1. Are most power laws derived under the assumption of random teacher vectors $T$? Can analysis be performed for a fixed realization of the teacher $T$?\n2.  In Figure 3, how is the CIFAR-5M plot made? Are the true target labels from the dataset used or is an artificial teacher network used? If the target is generated from a synthetic teacher in this Figure, do the authors think they can use their theory to predict the learning curves for the true labels?\n3. Could some of the plots like Figure 6 be plotted on log-log scale to see that the power law exponent depends on $\\beta$?\n4. In the linear network case, does the committee machine *learn features*? The neural tangent kernel for the committee machine with linear activations would be constant over training. \n5. How sensitive are the solutions in the committee machine to the initialization of the student weights?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This paper studies an important problem, namely the theoretical origin of neural scaling laws, and studies this in nonlinear two layer committee machines. For linear activations, they can obtain very precise learning curves in terms of the spectrum. They obtain power law exponents for both training time $\\alpha$ and number of student features $N$. For nonlinear networks, they predict escape times for the specialization transition in terms of $M,L$.", "weaknesses": "While the paper introduces a very promising approach of utilizing a hierarchy of order parameters to deal with power law structured covariates, most of the closed form theoretical predictions require restricting to the linear activation case. However, these results are also of interest. There are some remaining questions and issues, which if answered/addressed, could cause me to increase my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MqnnDMyvYU", "reviewer": "Reviewer_WSqh", "summary": "In this paper, the authors investigate neural scaling laws in a two-layer student-teacher network, where the data spectrum is generated to have L distinct eigenvalues following a power-law distribution. The authors use one-pass stochastic gradient descent with MSE loss. First, the authors derive analytical expressions for the generalization error in the case of a linear activation function and establish a condition for power-law scaling. Then, for non-linear activation functions, where plateaus may emerge, they derive expressions to predict both the height and width of these plateaus and investigate the asymptotic solution. For large $L$ and the number of hidden units $M$, the width exhibits an elegant scaling law of $\\sim M^{2}/L$", "questions": "- In Sec. 4.3 the authors model some learning behavior by training only $N_{l}$ of the student vector. Could the authors clarify the intended learning behavior they are simulating? Specifically, what motivates the use of $N_l$ in this context? Could you clarify the content of the statement \"each entry of the student\nvector directly corresponds to an eigenvalue\"? This should be clear to the reader without referring to the appendices.\n\n- Fig. 5: To enable a quantitative comparison with the derived expressions, could the authors include the width of the plateau (Eqs. (12) and (15)) in the figure?\n\n- Can the authors give more intuition about the underlying cause for the existence of the plateau, and for the scaling law of the escape time? It is a bit hard to infer it from the analytical derivation, and a heuristic/intuitive/hand-wavy explanation for this it would be helpful.\n\n- There is a bit of missing discussion regarding the generalization of the results. Are they specific to this particular setup, or do we expect some of these results to apply in other scenarios? For instance, how would the findings change i?\n\n- Line 209: To avoid confusion, consider renaming $\\alpha=p/N$, since it is already used as the time in the rest of the paper.\n\n- Around line 222: Have the authors mistaken the minimal polynomial with the characteristic polynomial?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "The paper is well-written overall. The arguments are well-founded, the text is concise and clear, and the authors skillfully focus on essential points in the main text, leaving detailed calculations for the appendix. All of the derived analytical results are verified by numerical simulations. The contribution, particularly the investigation of the properties of the plateau, appears to be quite solid as well.", "weaknesses": "The main problem with the manuscript is that the setting is quite narrow - exact degeneracy of the covariance eigenvalue, 2nd layer neurons are all untrainable and *identical*, only one nonlinear activation tested, etc. It is not clear how robust these results are to to even slight variation of the setting. IMHO, This is the main point that would determine the impact of this work.\n\nClearly, it is very difficult to extend the analytical results beyond the setting described by the authors, but it would strengthen the manuscript considerably if the authors investigated numerically the robustness of their results. Since the networks are relatively small, this should be easy to do. Concretely, to what extent do the conclusions hold when:\n  - $K\\neq M$?\n - The weights of the 2nd layer are not identical?\n- The covariance is only almost degenerate?\n- Other (and more common) nonlinear activations are used?\n\n(even a subset of these would be useful, but as I wrote above, the experiments are not challenging)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JIrwzWCqCa", "reviewer": "Reviewer_5FqQ", "summary": "The authors analyze the dynamics of learning a two-layer neural network with single-pass SGD, in the limit of large data dimension, for data with a power-law covariance spectrum. For linear regression, they derive an expression for the rate of decay of the error with time, which matches previous results in the literature in related settings. In non-linear cases, they analytically determine the length of the plateau in the learning dynamics, and in particular how it depends on the number of distinct eigenvalues in the spectrum.", "questions": "- l.190 Is a squareroot missing, i.e. should it be $\\xi^\\mu (\\sigma)^{l/2}J_i$ ? It does not seem consistent with l. 194 otherwise.\n\n- l.223 Shouldn't each term in parenthesis in the characteristic polynomial be to the power $N/L$ ? I might be missing something, further clarification would be helpful.\n\nMinor comments and recommendation.\n\n- Plotting the predicted slope for the power law (l.314) and the associated window of validity in Fig. 2 (right) would be a very compelling illustration of the theory.\n\n- I believe the scaling of $\\tau_{esc}\\sim M^2/L$  to be one of the most interesting results of the work. I would be curious to see a plot where the escape time is experimentally measured, alongside this predicted rate, to illustrate and support this finding, but this is just a recommendation.\n\n- In the continuation of the last point, could the authors provide more intuition why the length of the plateau decreases (increases) with $L$ ($M$)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is very well written, results are clearly exposed and connected to related works. Abundant and clear numerical experiments are provided to support the main results.\n\nThe question explored is interesting. To the best of my reading, the main technical contributions are\n- Establishing the rate of decay of linear regression for one-pass SGD, which happens to match previous full-batch results, e.g. Bordelon and Pehlevan (2022), Bahri et al (2024). The model and training are much simpler than in those papers, but I believe this particular case has not yet been covered in the literature, although I am not completely familiar with it.\n- Generalizing the escape time analysis of Biehl (1996) to structured data, again to the best of my understanding of the literature. In particular, the result that the length of the plateau decreases with the number of eigenvalues is an interesting one.\n\nI have a number of concerns, which I detail in the following sections. Overall, I think the paper is sound, although I have not checked the derivations in detail, and am overall in favor of acceptance if my concerns are addressed by the authors.", "weaknesses": "I have a number of observations and questions. I regroup my main concerns in this section, and leave the more minor points for the next.\n\n- l.191 I have strong doubts about this statement. I believe preactivations are always Gaussian, as linear combination of the Gaussian inputs. The Gaussian Equivalence principle is needed when discussing the post-activations. Please correct me if I'm wrong.\n\n- l. 417. Is a $l$ missing somewhere, why does the variance not depend on it ? Furthermore, it seems to me the variance scales as $1/N$, which does imply self-averaging. I do not understand the author's claim that higher-order overlaps do not self-average, which is linked to the result that $M$ distinct plateaus are present for the dynamics of e.g. the $R$ overlap. Further clarification would be very helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)