Dynamical Decoupling of Generalization and Overfitting in Large Two-Layer Networks
Authors: Andrea Montanari
Pierfrancesco Urbani
Venue: NeurIPS 2025 oral ()
Keywords: Overfitting; feature learning; dynamical mean field theory; generalization;
Primary area: theory
TL;DR: Large neural networks first learn low dimensional feature representation then overfit the data and revert to a kernel regime.

Abstract: Understanding the inductive bias and generalization properties of large overparametrized machine learning models requires to characterize the dynamics of the training algorithm.  We study the learning dynamics of large two-layer neural networks via dynamical mean field theory, a well established technique of non-equilibrium statistical physics. We show that, for large network width $m$,
and large number of samples per input dimension $n/d$, the training dynamics exhibits a separation of timescales which implies:
$(i)$ The emergence of a slow time scale associated with the growth in Gaussian/Rademacher complexity of the network;
$(ii)$ Inductive bias towards small complexity if the initialization has small enough complexity;
$(iii)$ A dynamical decoupling between feature learning and overfitting regimes; $(iv)$ A non-monotone behavior of the test error, associated  `feature unlearning' regime at large times.

Dynamical Decoupling of Generalization and Overfitting in Large Two-Layer Networks
Andrea Montanari Department of Statistics and Department of Mathematics, Stanford University
Pierfrancesco Urbani Université Paris-Sackas CNRS, CEA, Institut de Physique Théorique, 91191, Gif-Sur- Yvete,, France
Abstract
Understanding the inductive bias and generalization properties of large over- parametrized machine learning models requires to characterize the dynamics of the training algorithm. We study the learning dynamics of large two-layer neural networks via dynamical mean field theory, a well established technique of non- equilibrium statistical physics. We show that, for large network width m, and large number of samples per imput dimension n/d. /d, the training dynamics exhibits a separation of timescales which implies: (i) The emergence of a slow time scale associated with the growth in Gaussian/Rademace Rademacher complexity of the network; (ii) Inductive bias towards small complexity if the initialization has small enough complexity; (iii) A dynamical decoupling between feature learning and overfit- ting regimes; (iv) A non-monotone behavior the test error, associated 'feature unlearning' regime large times.
Introduction
Machine learning (ML) models are trained using stochastic gradient descent (SGD), of its variants to minimize the eтror on training data (empirical risk function). Classically, their good behavior on unseen data is explained by the fact that model complexity is kept small by regularization techniques: these models do not 'overfit.' Traditional ML theory decouples the analysis of the model from the optimization algorithm, which assumed to converge an approximate global minimizer [7].
In contrast, in modern ML, the empirical risk is highly non-convex, the number of parameters is comparable with the number of training samples, and the model complexity only weakly controlled. As consequence, there can be many assignments of the model parameters (many global empirical risk minimizers) that perfectly interpolate the data -even when these are noisy. While all of these interpolators are indistinguishable on the training data, they behave very differently (and some of them very poorly) on test data. has been hypothesized that models trained by SGD generalize well to test data because the algorithm selects a near global minimizer with low complexity, although mechanistic understanding this process is lacking. For this reason, the generalization properties cannot be decoupled from the training dynamics.
Several striking consequences of this lack of decoupling are documented in the literature (and have long been familiar to practitioners): (i) Test error after training is observed to depend strongly on
the initial weights distribution 28 (ii) Test error depends strongly on the optimization algorithm
(SGD, RMSProp, ADAM, to name a few), even when these algorithms achieve the same train error [55]: (ti)) Careful choice of the hyperparameters in the optimization algorithm is crucial [34 (59). and the optimal choice is often different from the one that minimizes train error; (iv) Models learned by training for shorter time have smaller complexity and generalize better 44 111
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

0.7
MF Feat. learning
Feat. learning
Overfit /eneftt/unlarr
5
0.6
Thain/Ten 'Test error
0.5
W2nd || y m
2nd laycr norm
0.4
0.3
Train
2
0.2
generaliz error
|[Wand x1
11
0.1
1mm (m)
x
to (m)
0.0
0
10
10a
10a
t
Figure 1: Three dynamical regimes of learning in a two-layer neural networks, with m hidden neurons. Training data comprises points dimensions distributed according single index model. We assume m, all large with n/mdd = (here a = 0.3). Blue: test error. Purple: train error. Red: & norm of second-layer weights proxy for model complexity).
These observations have motivated broad effort to encapsulate the effect of the dynamics 'implicit regularization' [88] 3 15 56): the algorithm selects an empirical risk minimizer that also minimizes a specific notion of model complexity. While this implicit regularization hypothesis has been fruitful, can only be validated if we can precisely understand the training dynamics.
In this work leverage tools from theoretical physics directly analyze the training dynamics and derive quantitative predictions on the implicit bias of neural network training, in a simple setting. This allows us capture feature learning and lazyloverfitting regimes within the same unified picture. We discover a time-scale separation in the training dynamics, between an early stage in which the model learns the relevant features representation of the data, and late stage of training that characterized by overfitting, feature 'unlearning,' and hence test error that increases with training. While the regularizing effect of early stopping has been an important object of study (for simpler models) in the past 44 11 61 [7), our work is the first to point out time-scale separation between feature learning (on a faster timescale) and overfitting (on slower time scale), thus reconciling the feature learning and neural tangent theories of learning.
We study two-layer fully connected neural networks f(::0) IRd R. i.c.
f(x; 0) = m Σ a oo(,,,
1
(1.1)
where = (a, W), where W = (WI..... Wm) E ddxm and a (aI...., am) I are, respectively, first- and second-layer weights. For convenience, we fix the normalization ||w,|| and assume that a does not depend on m. We apply model (I.I to a supervised learning task. We are given i.i.d. data (y,, a..), i n, with Y E R a response variable and a; E IRd feature vector, and try to learn model f(::0) to predict the response yeew corresponding to new input B.ew. We use gradient flow (GF) to minimize the empirical risk under square loss, namely
(tt)
n PoV POV (0()))
(,) = 2n Σ (I - (Ix;0))²
1
(1.2)

---

Here Pe is a projection matrix that guarantees that (t) E g at all times. The factor n/d is introduced for convenience and simply amounts to rescaling of time. We will typically initialize the training by setting (w,)ism rid Unif()) and = for all i S m, and study the dependence of the training dynamies on three key parameters:
Network width: m.
Overparametrization ratio: a md
n
Initialization scale: ag
Alongside the train error, we will be interested the test error at time t, i... (B((()) = E[
( 3/2, and the generalization error o ()) (..(((t))
Model 1.1 is much simpler than state-of-the-art architectures 52 but is rich enough to investigate
several general questions, which we summarize below:
When the network sufficiently overparametrized (a small) and dg is large, neural tangent kernel
(NTK) theory predicts that GF converges to an interpolator 30 22 16
Q1. For which region of a, dg does convergence take place, heyond NTK theory?
Q2. Does the selected model provide good generalization not [27 37
In contrast, when ag small, gradient-based algorithms can learn non-lineat -ww-dimessional representation of the data [3] 2 In these results, the difference between train and test error (generalization error) negligible: the model does not overfit.
Q3. Can we reconcile this Ceauur--erminngno--vveitttngg behavior with the lazy- training/overfiting regime described previously?
In the early phase training, the generalization error vanishes. However, training longer times can be beneficial, despite leading to overfitting.
Q4. When does the test error start increasing with training time? When should stop training?
Finally, scaling with the network size is crucial:
Q5. How does the generalization error depend on network size and number of iterations?
Q6. Does overfitting start earlier for larger networks or later?
In Section 2 we will present our analysis using theoretical physics techniques. Section 3 presents rigorous results confirming the picture emerging from this analysis. Finally, Section we discuss how our results address the above questions.
2 Main results: Dynamical mean field theory
We study the dynamics of model [..] under the simplest data distribution in which genuine non-linear learning is required to efficiently learn good prediction rule, the so called k-index model. Namely, we assume æ;; ~ N(0, Ia) and that depends on a low-dimensional projection U w;
y: P 2.)) t E N(O,
(2.1)
where the noise 6 is independent of 2; U E [Bd is an orthogonal matrix (U = Ix) and (p: 2 + R is a nonlinear function, (p) oo for g standard Gaussian.
An important aspect of this data distribution is that (for large d) it presents the largest possible gap between linear/kemel learning, which requires sample size to be superpolynomial in d [27] [8].. and nonlincar/neural network learning which only requires n (dd) (generically, for constant k). When the dimension d becomes large, discovering the latent features U x is crucial for learning and requires nonlinear processing of the labels yı [5 1
Our main focus will be on the simplest case, namely k = with & generic function (in particular [[p(G))) o for G N( 1), which corresponds information exponent equal to one according to the classification of [... Some of our results apply to k-index models for general fixed & (in particular, the rigorous results Section T.. We defer future work a more complete analysis the DMFT for k > 2.

---

We discover separation of time scales at large m (or large n/d), for sufficiently small initialization ag: feature learning takes place on a fast time scale, followed by overfitiin/reveesall kernel learning. This scenario is summarized in Figure | which plots numerical evaluations of our theoretical predictions at k = 1, T > 0 data distribution, the limit n, d, m -> oc at overparametrization ratio = 0.3.
More precisely, we observe three regimes (helow W 2nd a/ is the vector of second-layer weights in model (I.I):
(i) Mean field feature learning. t - O(1). The network learns the low-dimensional features U w;; the train error and test error decrease while their difference (generalization error) is negligible; the second layer weights remain small 22lll11 O(1)
(it) feature learning. 1 < t < m. The train error decreases slowly; the generalization error increases is small, i.e. (0(()) Hr, (@()) o(1); the test error can evolve non-monotonically, but remains approximately constant. Recond-layer weights become large < Wan |1 < m..
(iti) Overfitting and feature unlearning. t 2 m. Train error and test error diverge significantly, i.e. ((())) (..((())) becomes of order one. At the end of this regime, the train error converges to 0, i.e. the neural network interpolates the noisy data. The test error instead grows, and its limit value is the one of (data independent) kernel method: in other words, the model unlearns the low-dimensional structure. Finally, the second weights grow to W Wolll1 X /m, which indeed is the scale required for interpolation.
In this section we outline our results based on 'dynamical mean field theory' (DMFT). The next section will present rigorous results that are proven independently.
2.1 Technique
Our DMFT analysis is based on the following two steps:
Step 1: We leverage techniques from theoretical physics to derive approximate asymptotic characterization of the gradient flow dynamies (1.2) in the limit n, -> oo, with n/d σ.. This characterization consists of set of quantities (here p-lim denotes limit in probability, and we use the superscripts n to emphasize the dependence of the right-hand side n, d)
equations for the following asymptotic
Cig (21) p-lim (w, (11), (12))
n,d-+o
(2.2)
v1(() = ---lim U (1), ax(l) n,d-+00
p-lim (t).
n,d0+0
A rigorous derivation of the DMFT in a setting that includes two-layer networks is given in 13
However, the asymptotically exact DMFT characterization of 13 is rather complex to integrate numerically or study analytically. In order cireumvent this problem, we DMFT that is is asymptotically exact for a well-defined Gaussian version of the original model. Namely, we observe that the empirical risk of Eq. (1.2) takes the form
1 2n
(
F(O)
(2.3)
where F:; (Sd-1y" "R + IR" is stochastic process with i.i.d. components (. yt - [an; 0).
We replace these by Gaussian processes with matching mean and covariance, and study the DMFT for gradient flow with respect to the associated risk A (0).
The Gaussian approximation comes with an error which we show analytically is vanishing on time
scales of order one (indeed on these time scales we correctly recover the mean field theory of [38] 14 and we demonstrate empirically to be small on larger time scales (see for instance example Fig. The curves in Fig. [ were obtained by solving numerically the DMFT equations, see Appendix for details.
Step 2: We study this DMFT, with special attention to the large network limit m - and large sample size a - oo, with a a/m fixed, for a generic single index model (k = 1). We obtain separation of time scales in the dynamics, corresponding distinct learning regimes.

---

0.20
Thain ettor
0.10
01.55
10-1 10P 10
10 10P 10-1
10P 10 10
Figure 2: Evolution of second-layer weights (left) and frain error (right) when fitting pure noise data. Here we use mean field initialization, 1(2) = (9/10) + (1/6)=², a 0.4 and T 0.6. Symbols: SGD results actual 2-layer networks with d = 200, = amd (averaged over 10 simulations). Continuous viridis lines: Numerical solution of the DMFT equations. Note that the second layer weights given in terms of scalar quantity the result of the statistically symmetric initialization.
The analysis of the DMFT equations in the double limit m,! perturbation theory 9 29 Making this type of analysis rigorous is notoriously challenging and we proceed by combination of numerical solutions and analytical derivations.
is an example of singular
In the following, will first consider the simplest possible setting, pure noise data, and subsequently consider the single-index model. The structure of the activation function and target nonlinearity will be encoded in the functions
h(g)) == (Go(G)) == =
where G1. G9 are standard jointly Gaussian with Gg) = g. The relation between +p
and h, & is conveniently expressed in terms of the expansions in Hermite polynomials (xx) =
E 120 (x <px)) Ex20 (e)) which corresponds to the analytic expansion h(g) E Exz0 " (()) Ex20 saJug".
As mentioned above, assume throughout n, + co, with n/d a (0, oo), with the limit
m, a - oo taken afterwards. To further simplify our analysis, we assume a symmetric initialization whereby (0) an is independent of and (0) we use 'with high probability' for 'with probability converging to one as n, oc.'
m) aid Unif(Sd-1). Throughout,
In Section we present rigorous results that do not require either of these simplifying assumptions.
2.2 Training pure noise
We begin by the in which the data is pure noise: y: =; 720, A by-now-classic experiment [60] showed that deep learning models have sufficient capacity to achieve vanishing training error even when actual labels are replaced by random ones: they 'interpolate pure noise."
The ability of a model Fe -[f(-:0) 0 E θ) to interpolate pure noise is intimately connected to its Gaussian complexity ([Fe:n) = Suppee (g. f(X; 0))/n (53) (where g ~ N(O, I,) independent 0) (f 0) i n). Indeed, interpolation impossible unless ((Fe:n) 2 Viceversa, ((Fein) < ensures good generalization.
By a theorem of 77 for the network I.I.. 9(Fo: n) S La lal/mmlli vd/n (with L,, depending uniquely on σ). This means that, in order to interpolate noise, the average magnitude of second layer weights must be ||a/m|]i 2 T Vn/d
(/2
However, complexity bounds do not have implications on the convergence of GF to an interpolator.
Figure 2 compares the DMFT predictions to simulations using SGD to train actual two layer networks. this figure we initialize a(0) 1, and let a(t) evolve with GF alongside the first layer weigths. We observe that the theory describes well the empirical results, despite the Gaussian
5

---

Teiin/Ta eteor
ThainyTTtt.o
0.8
12
0.6
0.0
0.4
Figure 3: Train/test error (right) when fitting data from single index model. We set h(2) ((z) = (9/10)= /2/ T 0.3 and
0.3. Lines correspond predictions from the DMFT (continuous: train error; dashed: test error). Black continuous line is the m - oo value. Right: Same data plotted versus
approximation in our DMFT and the difference between SGD and GF. We also observe that second- layer weights remain roughly constant until large time t (m), which appears to increase with m. Roughly the same time, train error starts to decrease and converges to zero.
the appendix, we will make precise the above picture of the evolution of a(t). Here, we consider simplified setting in which a(t) = Yvm with independent of m, not evolving with training. Note that ((Fe; n) x 7/ ya and hence such a network can interpolate pure noise y is larger than threshold depending GF converges to vanishing train error with high probability if > Yor(a, m)), and converges to strictly positive training error if Yar(a, m)r. The threshold Yor m) converges limit (a)(a) E (0, 1) as m + xc
In Section G.1
Our DMFT predicts sharp phase transition. For E (0, 1),
rephrasing of the same phenomenon states that limm,d-roo 2 (0()t) (t; and
for 7 < Var (a)T, lim lim Ctr(t; m. 7o) = (7) > {o-
(2.4)
1000 m-
for 7 2 Yar (a)T.
Informally (a) is the minimum complexity for a very large network to interpolate noise via gradient flow. The functions (a), (x) will play an important role below.
We will next consider training data from single-index model. The initial scale of second- layer weights (0)/m]]] plays crucial role and we will separately analyze lazy and mean field initializations.
2.3 Training on data with latent structure: lazy initialization
We initialize (0) = /oymm, and let a(t) evolve according to GF alongside first-layer weights. DMFT predicts the emergence of three dynamical regimes large m and large (with n/d - a). For an illustration, we refer to Fig. 3
First dynamical regime: L 0(1/m). Second layer weights do not change significantly 7()
To om (1), while first layer- weights move by (l) (0)) = @(//mm). Because the weights at(t) are of order /m, even an ((1//m) change in the w, leads to a significant decrease in test error and train error.
Train and test error are close to each other. Namely, the following limits are well defined
lim (e) etr sp, 7o. m,
( = Cts :2, 7o. m, a) lim mid
(2.5)
50000000
with lim etr (i/m;; sp, 7o. m, a) = limm ets (f/m; s,, Yo, m, ax) =: dlll (i: s, To. a).
For large scaled time i, the error ettl (î; sp, 70. converges to the error of the best linear approximation to fa. This dynamical regime follows the qualitative predictions NTK theory, and essentially linear in the weights wi, but the time is too short for the model to overfit the data.

---

Second dynamical regime: ((1). Second layer weights do not change significantly: 7(t)
To 0m((), while first layer weights change significantly l|w,(t) w, ((1. However they change orthogonally to the latent subspace U and hence the test error does not change: no actual learning takes place in this regime, but the model starts to overfit the data.
More formally, train and test error have well defined limits as the network width diverges:
etr (t; P, To, a) ;= lim etr (t; ip Yo. m, a): Es (t; 4, To, a) == lim ets(t; 9;; 7o, m, a). (2.6)
m-+00
m=-30
However, the scaling function es value achieved the end of the first dynamical regime. Namely
ei (t: 9, 7o, a)) for the test error is constant in time and equal to the
222 ess (t: Yo, a) lim e" (fi a0,a) a) E1000
1 (72 |||| 2
[V) k'(0)
(b(1)) H(()))
(2.7)
Since the w 's move orthogonally to the latent space, their dynamics is equivalent (for large m) to the one in the pure noise setting, modulo redefinition of h. The right plot in Fig. Billustrates this.
Third dynamical regime: = (m)) The qualitative properties of this regime depend whether or not To is larger than an interpolation threshold ( (. T). which generalizes the threshold (a) = (a, 0, introduced in the pure noise case. Because the dynamics of weights w; in the subspace orthogonal to is equivalent to dynamies in pure noise, we expect the interpolation threshold Yrn (a, ip, T) to be given in terms of pure noise threshold (ar( as follows:
[V)11) 1/2 (G)(
(a, (9, = (+² + ||11
(2.8)
h'(0)
For to Tar(a, p,, r), interpolation is achieved during the second dynamical regime, no further evolution takes place.
For 7o < 7ar (a, s. r), monvtaivval evolution takes place for t = O(m). Introducing the rescaled time 2 (0, o0), we obtain, as m + DC,
(2.9) Further, for large values of the rescaled time z + oo, (z) grows to Ta, (p R Ta 4,, T), while (z) decreases to O. In other words, interpolation is achieved on this third regime.
1x3 (mzz) = 7 (z) + om (1), etr(m)) = eir (z) + om (1), ets(mz) = (2) + om (1).
Further the test error (2) increases from ets (1,9,, Thr T) whereby ckze (-..) is given by Eq.
(4;9,,00 70,a) a) egg Tar a), with Tar
2.4 Training on data with latent structure: mean field initialization
We initialize a()) og. independent of m and let second layer weights evolve. Note that initialization the network's Rademacher complexity is small, namely of order a d/n a aam Our DMFT analyisis predicts two dynamical regimes for large m. We will refer to them as 'first and "third regime' for consistency with other settings see Sec G.2 of the appendix). For an illustration, we refer to Figs. [ 5
First dynamical regime: t O(1). Both first and second layer weights change by order one: a(t) = ag + (11) and [t),(t) w; (;)ll ((1). and as a consequence test and train error decrease significantly. In this regime, the two errors remain close to each other and their evolution is well captured by the mean field theory of 38 as specialized to the case of spherically invariant distributions [10 2
Namely, limm a(t) = aml (t), limm +o (t) = e) (t), and DMFT reduces to a system of k: + 1 ordinary differential equations for the + scalar variables (amml (t), wmth (t))
m (t) ar mf (m)
(x) Vepa (1)) m1 (t)h ( (1)))))=((()))
(2.10)
0.am mil (t) = ap((="((t)
mil (t)
where Qu := I To As mentioned above, train and test error coincide in the large width limit
lim eir(() lim ets = eTm( (t).
m m=000
7

---

Train Tost
044
0.2-
107
10"
10' 102 10P
10P
101
10
107
Figure 4: Training dynamics under single-index model. We set h(g) - F( (9/10)g / т 0.3 and a 0.3, under mean field initialization. Left: second-layer weights. Right: train and test error. Symbols are empirical results for SGD with actual two-layer neural networks with d = 200, n amd (averaged over 10 simulations). Lines correspond to predictions from the DMFT (on the right, continuous: train error; dashed: test error).
In the case k = and ((=) = h(2), have that mml em = is fixed point of Eq. (2.10). and indeed the only fixed point with umn > O. If N' (0) > 0, then, we have (a (t), 0 (t)) (1, 1) as - 0o, and therefore test and train eтror converge the Bayes error e
An explicit formula for e (1) is given in Appendix G.2.1
7 /2. This significantly smaller than the test error achieved with lazy initialization. The separation between lazy and mean-field initialization is expected because feature learning takes place in the mean field regime.
Third dynamical regime: (m)). Computing the local stability of DMFT solutions around the mean field asymptotics (see Appendix G.2.2) suggests that the latter breaks down for = ((m). For t 2 m, we observe that the second layer weights grow to achieve a(t) x /m, the projection onto the latent space decreases to v(t) x 1/ I/m and train and test error diverge, eventually achieving etr(t) 0 and test error significantly larger than the Bayes error achieved earlier. We refer this phenomenon as 'featuee unlearning."
Denoting by (m(;;c) c) the time at which a(t) c/m (for small constant), we expect the existence of window size wi(m) such that
lim a affo(m; c) + z w(m))
(z), lim etryts (fo(m;;) + w(()) (r//1s (=),
Vm
(2.11)
where y (2), e (2), (2) are scaling functions describing the dynamics on this timescale. We expect to(m;e) = to(cmm- + o(m), and w(m) S to(m; c). but our numerical solutions are not sufficient to determine the precise scaling. On the other hand, it appears that at large times, the complexity converges close the interpolation threshold:
lim 2-000
(2) = (ar s,, T) RS (a, p,
(2.
Finally, the evolution train and test error for a(t) x vm appears to match the behavior at fixed second-layer weights. Namely, we define two functions
fr/tts (7) = lim (t) 7).
(2.13)
m-+00
We observe that the limit curves (y, -td (7)). (7, (1)),, match closely asymptotic train and test error obtained by fixing a(t) = m and letting second-layer weight evolve. This confirms the hypothesis that 7t) is slow variable, while others converge as if y was fixed.
3 Lower bounding the overfitting timescale
In this section we rigorously establish two results that confirm elements scenario outlined the previous sections. We emphasize that the result presented here are non-asymptotic, i.c. hold at finite

---

01
Figure 5: Left: second layer weights on the scale m as function of t/m. Curves appear to collapse on a master curve. The red arrow denotes P&F and the curves appear to converge to that limit. Center: the projection of the first layer weights model as a function time on timescales order m. Right: difference between test and train error function of the second layer weights on the scale vm. The finite curve are approaching a scaling curve which coincides with the one obtained by evaluating the same quantity but with a lazy initialization and fixed second layer weights.
the latent space in the single index
n, m, d modulo unspecified absolute constants. Further, we do not assume symmetric initialization of the weights. Throughout this section setting, it is more convenient to rescale time defining f to. Hence. instead of the flow (1.2). we study
@ = m V
(3.1)
For a = (1) the parametrizations & and are equivalent.
The first result of this section implies that (under mean field initialization) overfitting cannot take place on times of order one.
Theorem 1. Under the GF dynamics 1.2 and the data distribution in the introduction (with k arbitrary), ,. further assume |o lillup |lo 1s S L. (0)) 1upliup S L, ||a(0)]] S ao for some og 2 and that the w0))), < m are independent of the data {Vi,, : S n). Finally assume n 2 d m. Then, there exist universal constants Co. C₁ and the following holds for all f 2 0,
||a())) S +
+ E, 01 = CoL(T + Vic L)
(3.2)
(a((), ( (a(()) w(D) s (T + +1 + d
(3.3)
Under mean field initialization, is a fixed constant and hence on is also bounded, whence the generalization error in Eq. (3.3) small as long as i (((/d))//) (equivalently, for a fixed,
i = oommm//))
By itself, this result implies a separation timescales between learning and overfitting, thus confirm- ing the picture developed within DMFT, but falls short of characterizing the overfitting timescale.
The second result implies that, up to time-scale of order one, the dynamics is closely tracked by the mean field equations (2.10). Since the (0) at initialization are not necessarily all equal, these are generalized as
m a
0:000 (1) mll (0) (
(ip(O (0)
mit
(D) (0), 1z (0)))
m
m
0:am ml (t) (( (t)
0 mll
(i) (i), 1 (f)))
(3.4)
m
j=1
The mean field prediction for test error is the same for training error and given by
ets (t) = - 111 - an (1)p(17)) (0))
m
Σ a (1)am (1) le((oTT (1) of (i))
m
2m
J=1
Theorem 2. Under the the GF dynamics and the data distribution in the introduction (with
k arbitrary), further assume that ||o co. Ilsp'lluip < L. ||o|]ooo. I|o''loo. l|o luip S. L. Further

---

assume (a)(0) S for all m, (an,(0))ssm rid Unif(S ). Then for any 8 > 0 there exist
constants co er, C depending on L, T. 6, k such that, letting Ti = co coloog m) A (log Bogn/d)/2 /3, the following happens with probability at least - 2 2exp(-cidd
1
d' V
1/2-6
sup
(a:(0)
mii (0) + +1/0(()) u. (1ll) SC
-
(3.5)
d
i=1
sup 1
P(a(t). (oc(),W(O)) - Sc
1 m
v
d'
1/2-6
(3.6)
d
Remark 3.1. While the analysis in the previous section requires m - oo after n, d + oc, nei- ther Theorem 3.1 nor Theorem 3.2 make the assumption. In particular, Eq. (33 implies that the generalization error is small for = o((n/d)//3) irrespective of m.
Similarly, Eqs. 3.5), (3.6) imply that the mean field theory of [38 14 45 of the system for times = of (log m) A (logn/d) n//))/3).
captures well the evolution
1/3
4 Discussion
We conclude by highlighting a few qualitative conclusions of our work, and how they address questions raised in Section the following remarks, we consider a = n//nd as constant.
Interpolation mechanism. In the current setting, the neural model complexity is proportional to [a(())l = (()++ (1). We observe two alternative scenarios. If the complexity at initialization large enough To > (a)r,) then the gradient flow rapidly converges to near interpolator without significant change in 7(). If instead, To Thr (a)T, then (()) grows to reach the interpolation threshold at which point the training error converges to O.
Adiabatic evolution of model complexity. In the latter case, the complexity 7()) evolves on a slower time scale than other degrees of freedom. The dynamics on shorter timescales is well approximated by the one at fixed yy (given by the current value (t)). The generalization error becomes of order one only when (t) is of order one.
Decoupling of learning and overfitting. When To 0m (1), the fact that (t) acts slow variable implics a large-m decoupling between learning (which takes place on faster timescales, as long as ytt) om
(m))) (1)), and overfitting (which takes place on slower timescales, when >(t) Pm (1)). This has several implications for the questions outlined in the introduction.
Q3: Lazy initialization (0) x / m leads to poor generalization because the feature-leaming phase is skipped either partially altogether.
Q2: Training until interpolation is generally suboptimal.
Q4: The optimal tradeoff is obtained at the end of the first phase.
Q5, Q6: Further, fixed overerparametrriation 1t/md= a, overfitting starts later for larger models.
Overfitting and feature unlearning. The above description points at non-monotonicity of the model quality, which improves on short time scales, and deteriorates at larger time scales. Recipro- cally, early stopping acts as a regularization. While this phenomenon is well understood for linear models [2]] our analysis provides an analogous (quantitative) scenario for training neural network models. In particular, it clarifies the underlying mechanism: in the same dynamical regime in which network complexity grows (T(t) becomes of order one), and training error becomes negligible, the low-dimensional latent features are 'unlearned' (v(t) becomes of order 1/vm). We expect that these findings also allow to understand the beneficial effect of regularization on the second layer.
Acknowledgments
This work was supported by the NSF through award DMS-2031883, the Simons Foundation through Award 8 116639 for the Collaboration on the Theoretical Foundations of Deep Learning, and the ONR grant 1-900-------2799. This work was supported by the French government under the France 2030 program (PhOM- Graduate School of Physics) with reference ANR-11-IDEX-00
10

---

References
[1] Emmanuel Abbe, Enric Boix Adsera, and Theodor Misiakiewicz. The merged-staircase property: necessary and nearly sufficient condition for sgd learning of sparse functions two-layer neural networks. In Conference on Learning Theory, pages 4782 4887. PMLR, 2022.
[2] Luca Arnaboldi, Ludovic Stephan, Florent Krzakala, and Bruno Loureiro. From high- dimensional and mean-field dynamics to dimensionless odes: unifying approach sgd in two-layers networks. In The Thirty Sixth Annual Conference on Learning Theory, pages 1227. PMLR, 2023.
[3] Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems, 32, 2019.
[4] Gerard Ben Arous, Reza Gheissari, and Aukosh Jagannath. Online stochastic gradient descent on non-convex losses from high-dimensional inference. Journal of Machine Learning Research, 22(106): 2021.
[5] Jimmy Ba, Murat A Erdogdu, Taiji Suzuki, Zhichao Wang, Denny Wu, and Greg Yang. High- dimensional asymptotics of feature learning: How gradient step improves the representation. Advances in Neural Information Processing Systems, 35:37932-37946, 2022.
[6] Boaz Barak, Benjamin Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang. Hidden progress deep learning: SGD learns parities near the computational limit. Advances in Neural Information Processing Systems, 35:21750-21764, 21764, 2022.
[7] Peter Bartlett. For valid generalization the size of the weights is more important than the size of the network. Advances in neural information processing systems, 9, 1996.
[8] Gérard Ben Arous, Amir Dembo, and Alice Guionnet. Cugliandolo-kurchan equations for dynamics of spin-glasses. Probability theory and related fields, 136(4):619-660, 2006.
[9] Nils Berglund. Perturbation theory of dynamical systems. arXiv preprint math/0/1/ 2001.
[10] Raphaël Berthier, Andrea Montanari, and Kangjie Zhou. Learning time-scales in two-layers neural networks. Foundations of Computational Mathematics, pages 1-84, 2024.
[11] Christopher M Bishop. Regularization and complexity control feed-forward networks. In Proceedings International Conference on Artificial Neural Networks ICANN' '95, pages 141-148, 1995.
[12] Blake Bordelon and Cengiz Pehlevan. Self-consistent dynamical field theory of kernel evolution in wide neural networks. Advances in Neural Information Processing Systems, 35:32240-322 32256, 2022.
[13] Michael Celentano, Chen Cheng, and Andrea Montanari. The high-dimensional asymptotics of first order methods with random data. 20X1v:2112.0077222201..
[14] Lenaic Chizat and Francis Bach. the global convergence of gradient descent for over- parameterized models using optimal transport. Advances in neural information processing systems, 31, 2018
[15] Lenaic Chizat and Francis Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference on learning theory, pages 1305-1338 PMLR, 2020.
[16] Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable program- ming. Advances in neural information processing systems, 32, 2019.
[17] Andrea Crisanti, Heinz Horner, and HJ J Sommers. The spherical p-spin interaction spin-glass model: the dynamics. Zeitschrift für Physik Condensed Matter, 92:257-271, 1993.
[18] Leticia F Cugliandolo. Recent applications of dynamical mean-field methods. Annual Review of Condensed Matter Physics, 15, 2023.

---

[19] Leticia F Cugliandolo and David S Dean. Full dynamical solution for a spherical spin- spin-glass model. Journal of Physics A: Mathematical and General, 28(15):4213, 1995.
[20] Leticia F Cugliandolo and Jorge Kurchan. Analytical solution of the off-equilibrium dynamics of long-range spin-glass model. Physical Review Letters, 71(1):173, 1993.
[21] Alexandru Damian, Jason Lee, and Mahdi Soltanolkotabi. Neural networks can learn represen- tations with gradient descent. Conference on Learning Theory, pages 5413-5452. PMIR, 2022.
[22] Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International conference on machine learning, pages 1675-1685. PMLR, 2019.
[23] Giampaolo Folena, Silvio Franz, and Federico Ricci- Terii-Tersenhi. Rethinking mean-field glassy dynamics and its relation with the energy landscape: The surprising case of the spherical mixed p-spin model. Physical Review X, 10(3):031045, 2020.
[24] Jerome Friedman, Trevor Hastie, and Robert Tibshirani. Additive logistic regression: a statistical view of boosting (with discussion and a rejoinder by the authors). The annals of statistics, 28(2):337-407, 2000.
[25] Yan V Fyodorov. A spin glass model for reconstructing nonlinearly encrypted signals corrupted by noise. Journal of Statistical Physics, 175:789-818, 2019.
[26] Yan V Fyodorov and Rashel Tublin. Optimization landscape in the simplest constrained random least-square problem. Journal of Physics A: Mathematical and Theoretical, 55(24)2244008, 2022.
[27] Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Linearized two-layers neural networks in high dimension. The Annals of Statistics, 49(2), 2021.
[28] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256. JMLR Workshop and Conference Proceedings, 2010.
[29] Mark Holmes. Introduction to Perturbation Methods, Springer, 2013.
[30] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
[31] Persia Jana Kamali and Pierfrancesco Urbani. Dynamical mean field theory for models of confluent tissues and beyond. SciPost Physics, 15(5):219, 2023.
[32] Persia Jana Kamali and Pierfrancesco Urbani. Stochastic gradient descent outperforms gradient descent in recovering high-dimensional signal in glassy energy landscape. arXiv preprint arXiv:2309.04788. 2023.
[33] Jaron Kent-Dobias. On the topology of solutions random continuous constraint satisfaction problems. arXiv preprint arXiv:2409.1278) 2024.
[34] Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. Advances in neural information processing systems, 32, 2019.
[35] Stefano Sarao Mannelli, Florent Krzakala, Pierfrancesco Urbani, and Lenka Zdeborova. Passed spurious: Descent algorithms and local minima in spiked matrix-tensor models. In interna- tional conference on machine learning, pages 4333-4342 PMLR, 2019.
[36] Andreas Maurer. A vector-contraction inequality for Rademacher complexities. In Algorithmic Learning Theory: 27th International Conference, pages 3-17. Springer, 2016.
12

---

[37] Song Mei. Theodor Misiakiewicz, and Andrea Montanari. Generalization error of random feature and kernel methods: hypercontractivtty and kemel matrix concentration. Applied and Computational Harmonic Analysis, 59:3-84, 2022.
[38] Song Mei, Andrea Montanari, and Miia-MMinhaa Nguyen. A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences, 115(33):E7665- E7671, 2018.
[39] Marc Mézard, Giorgio Parisi, and Miguel Angel Virasoro. Spin glass theory and beyond, volume 9. World Scientific, 1987.
[40] Francesca Mignacco, Florent Krzakala, Pierfrancesco Urbani, and Lenka Zdeborová. Dynamical mean-field theory for stochastic gradient descent in gaussian mixture classification. Advances in Neural Information Processing Systems, 33:9540-9550, 2020.
[41] Francesca Mignacco and Pierfrancesco Urbani. The effective noise of stochastic gradient descent. Journal of Statistical Mechanics: Theory and Experiment, 2022(8);083055, 2022.
[42] Andrea Montanari and Eliran Subag. Solving overparametrized systems of random equations: I. model and algorithms for approximate solutions. arXiv:2306. 13326, 2023.
[43] Andrea Montanari and Eliran Subag. On Smale's ' 17th problem over the reals. arXiv:2405.01735, 2024.
[44] Nelson Morgan and Hervé Bourlard. Generalization and parameter estimation in feedforward nets: Some experiments. Advances in neural information processing systems, 2, 1989.
[45] Grant Rotskoff and Eric Vanden- Trainability and accuracy of artificial neural networks: An interacting particle system approach. Communications on Pure and Applied Mathematics, 1599):188911955, 2022.
[46] Mark Sellke. The threshold energy of low temperature Langevin dynamics for pure spherical spin glasses. Communications on Pure and Applied Mathematics, 77(11):4065 4099, 2024.
[47] Shai Shalev-Shwartz and Shai Ben-David, Understanding machine learning: From theory to algorithms. Cambridge University Press, 2014.
[48] Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent separable data. The Journal of Machine Learning Research, 19(1):2822-283 -2788, 2018.
[49] Eliran Subag. Concentration for the zero set of random polynomial systems. arXiv preprint arXiv:2303.11924, 2023.
[50] Michel Talagrand. Mean field models for spin glasses: Volume I: Basic exumples, volume 54. Springer Science & Business Media, 2010.
[51] Pierfrancesco Urbani. A continuous constraint satisfaction problem for the rigidity transition in confluent tissues. Journal of Physics A: Mathematical and Theoretical, 56(1 I): 115003, 2023.
[52] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances Neural Infor- mation Processing Systems (NeurIPS), volume 30, pages 1998-6008. Curran Associates, Inc., 2017.
[53] Roman Vershynin. High-dimennsioal probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018
[54] Nikhil Vyas, Yamini Bansal, and Preetum Nakkiran. Limitations of the ntk for understanding generalization in deep learning. arXiv preprint arXiv:2206. 10012, 2022.
[55] Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. Advances neural informa- tion processing systems, 30, 2017.
13

---

[56] Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparameerrized models. In Conference on Learning Theory, pages 3635-3673. PMLR, 2020.
[57] Yuan Yao, Lorenzo Rosasco, and Andrea Caponnetto. On early stopping in gradient descent learning. Constructive Approximation, 26(2): 315, 2007.
[58] Gilad Yehudai and Ohad Shamir. On the power and limitations of random features for under- standing neural networks. Advances in neural information processing systems, 32, 2019.
[59] Kaichao You, Mingsheng Long, Jianmin Wang, and Michael Jordan. How does learning rate decay help modern neural networks? arXiv preprint arXiv: 1908.008,,, 2019.
[60] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107 115, 2021.
[61] Tong Zhang and Bin Yu. Boosting with early stopping: Convergence and consistency. Annals of Statistics, pages 1538-1579, 2005.
[62] Jean Zinn-Justin. Quantum field theory and critical phenomena. Oxford University Press, 2021.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification: We conduct theoretical analysis that is described by the abstract and that answers the questions detailed in the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the introduction section we discuss how the contribution compares to previous literature and limitations related the use of non-rigorous mathematical techniques.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [Yes]
Justification: We conduct theoretical analysis of training dynamics. The method that we use is non-rigorous but well established in theoretical physies. We show that the method correctly reproduces observations and it is checked against simulations. We prove two theorems that support our analysis.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We detail the numerical simulations in the appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
16

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [NA]
Justification: Our paper is theoretical nature and simulations are fairly standard and only play a support role.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The theoretical results and figures are detailed with the corresponding settings that we used to produce them.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes].
Justification:
The paper contains all the details about the numerical simulations we used.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [NA]-
Justification: There are extensive complex experiments we have performed. The paper is theoretical nature and aims at understanding simple yet paradigmatic models.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethies
Answer: [Yes]
Justification:
We conform with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA].
Justification: Our work is theoretical in nature and aims understanding neural network models rather to extend their use in technological applications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification: Our work is theoretical in nature and aims at understanding neural network models rather to extend their use in technological applications.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA].
Justification:
We do not use existing datasets or codes.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
19

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]-
Justification:
We do not produce any new asset. Our study is purely theoretical in nature.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA].
Justification: Our work theoretical in nature and aims at understanding neural network models rather to extend their use in technological applications. We do not perform experi- ments with humans.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification:
We do not conduct experiments with humans.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]-
Justification:
We do not use LLM.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (ttps///eeuiss.cccoommeeresessss for what should or should not be described.
21

---

A Setting
We recall for reference some basic definitions and notations. We consider the 2-layer network defined by
f(x: W) = o (w;;
(A.1)
Throughout, we assume an offset to be subtracted so that (G) O, for G ~ N(O, 1). The network input is d-dimensional real vector and the output is scalar variable. The parameters of the network are the weights of the first layer collected in the matrix W defined as
W
W2
W
E gmxd
w; IR
(A.2)
mm
We will assume that ||u,; 1|2 = The weights of the second layer are instead (01,..., am)) and are real, possibly unbounded, variables.
We consider dataset of points independent and identically distributed (vi, a;) where n; ~ N(O, Ia), and the labels ys are generated according to the following k-index models:
y: = < x;) +
(A.3)
Therefore, labels depend on the projection of the covariates on fixed subspace U E BRdk with U =
Ix (there is no loss of generality in assuming U orthogonal). Efficient learning requires to estimate this subspace. Since we consider learning with square loss, we assume
1|+112 E(P(UT=a x.,)) ) B
where g ~ N(O, 1x). We refer the case yp the 'pure noise case' 'pure noise data'.
We now discuss the covariance structure of the network given by Eq. (A.I). For two sets of weights (a1, W) and (az, W2) we have
B(f(;;), W f(; 02 W2)
a ((w),,, W2.))
(A.4)
m
The average in the rhs of Eq. A.4) over the data distribution while the function h(g) is defined as
h(g) ((o(G))
(A.5)
for (G1, G2) centered jointly Gaussian with E(G?) 1, E(G:G2) =q.
Furthermore we have that:
Σ 11
a U w:).
(A.6)
where @ is given by
((v) (v):=BBol((a) VI- loPP)))(())
(A.7)
for G N(O, 1 independent of Go N(O,
We consider Gaussian process (a, W), 8 with the covariance function defined above and define the empirical risk under Gaussian approximation as
1
(a m.
(a, (a, W) -2- =2²²
(A.8)
2n
i=1
1
fa W) 49 - =|
2n
22

---

where - (f.( i.dd copies of the above processes. We will also write y = 60 + E.
n),, sp¹ : n). E =
(E, : 1 n) are vectors containing
Given model with estimated parameters à, W. the test error is given by
1 B ( W) 40 -5-)) -c)²
(a, W)
(A.9)
2 1 ([ à W) (0U a:) - =)
2
where the expectation in the first line over a triple (f9, 40, independent the data, and in the second line with respect to The two expectations coincide because they depend uniquely the second moments of these processes.
We are interested studying the gradient flow dynamics in the random landscape R., W)
a(t) V (a(()))) W()),
Ti
d
(A.10)
(t) = V S (alt), W (t) - (t((),, Vi = 1........ m
n
d
The Lagrange multipliers are added to enforce the spherical constraint (1)l² 1. While we consider the case of normalized first-layer weights, our approach can be generalized unconstrained weights or to include weight decay (ridge regularization). As explained in the main text, we will replace this by gradient flow in the Gaussian model Φ (a, W). We refer to Section K for discussion of DMFT in the original non-Gaussian model.
In our analysis we will always consider the proportional asymptotics
n, d + 00, d + a E (0)
n1
(A.11)
We typically index sequences and limits by n, but it is understood that d = d(n) + 0o as well. After d + do proportionally, will consider the large network asymptotics oo at fixed a = a/m.
In the following we will drop the superscript g and write, for instance (a, W) instead of (a, W) whenever clear from the context. All of our analytical predictions (except for Section obtained within the Gaussian model.
B Technique
Notice that each fitting error F. y
(I,..., . n is random function of the model parameters 0. The randomness due to the randomness in æ; and the noise E,. The empirical risk
in Eq. 1.2 can be rewritten
(,)
1 F0) 2m
F(0) (F),
(B.1)
Our key approximation consists in replacing the i.i.d. random functions (Fi)s by i.i.d. Gaussian processes (FF") )is, with matching mean and covariance. While DMFT equations have been recently proven without recurring to this approximation (see [13 and appendices), their structure is simpler in the Gaussian case, which allows us to carry out the large-m analysis.
Computing the covariance of F(- () is a straightforward exercise. We assume for simplicity that an intercept is subtracted so that E[a(G)) = 0, B[(((G) 0 and otherwise these functions are generic (G, G and so on will denote standard Gaussian vectors). We then have
1 0))( m²
(a ho.T 222)).
(B.2)
B[f(x:0)] m² (a, (()
(B.3)
Recall that (a. W) where T W (wi,..., wm) E dxxm are the first layer weights Finally, h : I R, @ R R encode the activations and the target function &, with h applied entrywise to the matrix W! W2.
23

---

The covariance of (0) fi(;; 0) is easily computed from the above, and this defines completely the corresponding Gaussian process (F)) We denote the associated risk function (0) =
| ( (0)ll//n.
Let us emphasize that the cost function 2 () remains highly non- non-trivial despite the fact that the functions F; are replaced by Gaussian processes. Near-minima of high high-dimensional Gaussian processes have a very rich structure, which is a central theme in spin glass theory [39 50]. Additional
layers of complexity arise here for two reasons. First, Lx (0) is sum of squares of Gaussians and, second, the underlying Gaussian process has a significantly more intricate covariance than in standard spin glasses (where typically depends only on the inner product (0) 02)). Recent work explored the
simpler case in which (-) is Gaussian process with covariance E (0:) ) (02)) = ( 02))
depending uniquely on the inner product [25 26 51
49
42
33 Gradient descent dynamics on 43
these models has been recently studied via DMFT 31
32
: our work builds on these advances.
DMFT was leveraged before to address other questions in high dimennniiiii statistics and ML B35 [2].
We refer to [8 [3] for mathematical results on the DMFT approach.
While x (0) has non-trivial structure, methods from statistical physics can be brought bear to derive an asymptotic characterization. Namely, define the functions
(t,, t2) (t.) wy((2)) vt = U (t),,, (t).
(B.4)
These functions are random (because of the random initialization and the randomness in F and depend on n, d. However, as n..d + oo with n/d - a. they converge to non-random limits (C., (11, tz)) (t)()))mmm (a)(t)s<ssm that are the unique solution of set of coupled integro- differential equations, see the appendices. We refer to these as to the DMFT equations.
Our main focus is on the behavior of the solutions of these equations for large m and, at first sight, the complexity of the DMFT increases with m. An important simplification arises when choosing symmetric initial condition ao( = an for all i S m, and (w;(0) Wid Unif() Namely, the solution of the DMFT equations is symmetric under permutations of the neurons: Ca((1, (2) = Ca(l1, (2) for i S m and Ciy (2) = C,, 12) for i + j S m, while (t) vt). (t) = a(t for S m. We then have a reduction to a set of integro- differeenii equations on k + functions, that depend parametrically
m.
We use two approaches to study these equations (see appendix):
(a) Numerical integration for increasing values of m under different initial conditions. Asymptotics as m - xo (at fixed a = -a/m) via singular perturbation theory 9 29
For (b), a specific dynamical regime is identified by a scaling of the time variable, which in our case will take the form = t (m) f for certain fixed function f (m) and f = O(1) a scaled time, The asymptotics of DMFT quantities in that regime takes the form
lim u (t4 (m) l,m,a a - m 02-000
= v. (f:a).
(B.5)
C Dynamical Mean Field Theory (DMFT)
In this section we state the results of Dynamical Mean Field Theory (DMFT). We will outline heuristic derivation in Section [ We first introduce the general DMFT equations in Section C.T the corresponding predictions for certain observable of interest in Section C.2 These are set of O(m²) integro-differential equations in as many unknown functions.
We then specialize these equations to the case of a symmetric initialization, in which w;( ~ Unif (Sd-()) and (0) = ag for all i S m, see Section C.3 In this case, the dynamics is characterized by set of k + 3 equations which are stated in Sections and C.5
C.1 General DMFT equations
Let of (1), wo (t), u² (t) the the solution of Eq A. when the dynamics is initialized at non-random a. 00,5. 8 S and possibly random, w (0) such that (ur² (0), unt (0)) + CO for i, $ n, UT (Top(( (0) > of for S n. While random, the wnl (0) are assumed here to be independent of the random processes fd, 60,
24

---

f, s 2 0 consider the quantities
(1,)) == (ww(() un (8)) (7(( = U (()
(C.1)
Then DMFT predicts that these quantities have a well defined non non-random limit as n, + co,
Cig(1, 8)
lim (L,)s n.d-000
ve(l)
(1), a;(1) n.d-+0
lim
lim d)(),, n,d+oc
(C.2)
where the limits are understood to hold in almost sure sense. These limits are the unique solution of a set of integro-differential equations in the unknowns (Cig(t, 8), Riy(t, v,(t), ax((): i,j m), which we next state as three sets: (1) Dynamical equations; (2) Equations for auxiliary functions; (3) Boundary conditions. Before that, we mention some constraints that need to be satisfied by the solution of these equations.
(0) Constraints. The functions Ct)(t,s), Riy (t, s) satisfy:
C+(t,t) = v0 st,
(C.3)
=) = Cj((s, 0A S 11, Riy(t,8)= =0 Vo <L<..
(C.4)
(C.5)
The first condition in particular implies the following useful relation:
(t,,
(aCis(t,t) aC
lim f-
+
(C.6)
dt
Ot
Ot
We refer to the property constraint.'
(C.5) (and similar ones for R functions appearing below) as 'causality
(1) Dynamical
equations. These equations determine the dynamics of (Cis(t, s), (t, s), a,(()),(())))))))))))))))))))))) i,j s m). and involve the auxiliary functions (memory kernels) MG(t, (t, s). (t, s) and (Lagrange multipliers) (t) (the last equations assume implicitly
la > 1o):
(t)(t) a
m { R)(l,))
(( ds m
1
(C.7)
dt
( Ca(t,s)
1 -
Σ
s)
m
m
(t:(t)
(t)), + (() I. R)(t,s) 1 m /
Σ
M (t, (s)
dt
m
(C.8)
(tartt))))) = vi(ta) (b)
+ 05((6)) f"h (taltas)
a
(C.9)
Ila
Mf 8) Cis(s,t)) ds
m M (ta: 8) Rgilts, s)ds,
m
@Ri((tatts)
lo
(1( (ta to) + S1yō(ta tb) Σ
M s) Rı,(s, ds.
Øt.
(C.10)
We point out that the 5(A, 1b) in the last equation (together with Eq. C.5 has to be interpreted as while, for > 0, Ray (t + 8, 8, + (1).
follows: R2g (1, (') = 0 for
Equations
C.9 and (C.10) can also written terms of an effective stochastic process in Rm; wt) [w)[ < m). This is defined as the solution of the following set of ODEs (for
25

---

i E fl....,m. m)):
(t)
(1) + aa, (V. v(()) I Ra(t,
(C.11)
di
M s) (s) ds (t) bi(t),
(C.12)
m
(0) N( 1),
(C.13)
where (m.(t): : - m²) is a centered Gaussian process with covariance
(lm((tn, = - L')
(C.14)
Define b(t) = (b:(1) m). The solution of Eqs. C.9 and (C.10) can be written as
Ciy(t,t') = lim E
[wa (t,(t)uy(t))
(C.15)
b-00
(E]
(1j(1,
lim b-0
(C.
(b')
In fact the stochastic process of Eq. (C.11 is expected to describe the limit distribution of the second- layer weights (t). Namely, for S d, define w;; = W W(t),, et), E Rm be a vector containing the i-th coordinate of each neuron. Then, for any fixed and any T.
(t) :0 Tt (w (t) : 0
(C.17)
Here 4 denotes convergence in distribution as n, d + co, ((
The memory kernels M and M are defined by (2) Equations for auxiliary functions.
(t, = @ m My (t, Calt, (Ci (t))
+ Ca(t, (C;((( (( (s))
(C.18)
m
where the functions R. and CA satisfy the symmetry properties Ct(t, Ca(s,t) and (t, 0 for t < s, and are the unique solution
[_ 18(t 18(t- () En(t,
( dss 8),
(C.19)
[ 15(t 15(t-
- 8) [x( /' Ec((,s) -0,
Cals,t)
where
Ec(t,s) = 72 + 11 + m² Σ (()(C)))))
(((()(v(()))))
Σ (s))
(C.20)
Ex(t,s) = (t)) (t)(t)
The Lagrange multipliers (t) have to be fixed to enforce the constraint Ca,t, t) = which follows from w. E Sd-1 The corresponding equations are
a km
(ta) Ve(ox((a))) 1/ Ra(la, 8) ds
v1(1a)
(C.21)
1
1
Cigls,
M (t)
26

---

(3) Boundary conditions. The dynamical equations C.7 to C..0
can be integrated from a set of
initial conditions that reflect initial conditions of the GF dynamics:
(0) vg:
ag af Vi.j ml.....m).
vi (1. (1....))) m)
C, 0) = C/²y
(C.22)
Riy (0,0) 0
Vi,j {L....,m. m}
C.2 Expressions for train and test error
The asymptotics of many quantities of interest can expressed in terms of the solutions of the DMFT equations stated in the last section. particular, the train error Ln (W a(t) and test error (W (t), (t)) at time have well defined limits under the proportional asymptoties:
lim (W() a()))(( = etr(t),
lim 28 (()))))tt)) = ets(t) 20000
(C.23)
n 00
The functions (t) (t)(t) are given by
etr(t) - (t,
1
(C.24)
ets(t) = 2 72 + k
1121
Σ
((C)(t,t)
2 Σ ((,((t))
(C.25)
m
1==1
More generally, Ca 8) gives the asymptotics of the correlation of residuals:
lim
(t) (()) sCalt,s)
(C.26)
(t) y (a(t), W(t).
(C.27)
where we recall that y = 00 e.
C.3 Symmetric initialization and solutions
As anticipated, we consider the uninformative initialization (²)(0) ~ Unif and a (0) = do for all m. This results in the following initialization for the DMFT equations of
(0) of 0
Vi {L....,m, m.)
Ci 0) C/ 0
Vi * j, E (1....,m) m)
(C.28)
(0, C. =1
W (1.....m).
This initialization is invariant under permutations of the m neurons. Since the DMFT equations of Sec- tion C are equivariant under such permutations, their solution is also invariant under permutations. This means that takes the form:
Ctt(t Ca(t,t)) ifi=j. CClt,t') if fi
if j. Rt,
Ralt,t))
Ris(t,t')=
(C.29)
if j.
vx(t) = vtt,
px(t) = v(t),
as(t) a(t) Vi.
(C.30)
As consequence, the memory kernels Eq. (C.18)
take the form
(t = (M)(),t) ifi - (t,t') if
Mf(t,t' [Mf(t, ifi (M if
(C.31)
We will refer to the reduced DMFT under symmetry as the SymmDMFT.
27

---

C.4 DMFT equations for symmetric initialization (SymmDMFT)
Dynamical equations. Substituting the ansats of the previous section in the equations of Section
we obtain the following equations for the functions a(t), v(t), Ca(l, Co(,, Ra((, s),
Ro s):
da
(1) p(v(l)) /. ds(l,s)
a
(C.32)
dt
M(Calt,s) + m- MC.((,,)) ds m
Ra(t,s)a(s) m
m
m / Ca(l,))a(s)
1 ()Caa(t))))Ral(o)))+ m - ()) ds,
m
di (1) - - v(1)o(1) (0)=0)())(0)--- + _ vQ(((()))()) L. Ra(l,s)ds
(C.33)
-1 1 / M (1, (m (m-1)) (1, o(s)ds,
- (t) (V(((())) / s)
=
(C.34)
m / (c) (t. ds
1 /'
do,
O,C.(t,t) - (t) T m ((())() / R)(
(C.35)
m / [c. (t, 2 (t, s)] ds
1
m //' [M (L.)RR,)) [M 8) (m 22) (1, (1, Ro s) ds.
1
0,Ra(t,t) = v(t) Ra + 5(t-t)
(C.36)
-1
m /
M (1)) (1, Ra(s,) + (m (m- (m-1)a/h (t, (o)(( (3) ds,
a.Ro(t,t)) (ot)) 1
m l [M (t. st) (
(t, Ra(s,t)
(C.37)
+(m - 2))M (t, ((,s)))(), ds.
(2) Equations for auxiliary functions.
The memory kernels (x) (t, s), (1) (t, (t, and M (t,s).
( (1, given by:
O ( a(()(ss)) m { (CC0U////AR((NNNCCCCCC/:C)/CCC/1
(C.38)
M m HBtta//ASMMMKCccccaacaaaatcttaaa s)]
(C.39)
(( (t, o((tals(Cattss)(((tttt))
(C.40)
m
a M (t,s) c(t
(C.41)
28

---

Further, Ca(t, s), Ra(t, s) are given by the same equations CC.19 where Dc, En are simplified as follows:
(c(t, = - a(t) - a(s))(u(s)) + a(t)a(s) h(Calt,s))
m- (( m En(t,s) a(t)a(s) CCalt Ra) 8) m (Co))(())((C((tss)))))) m
(C.42)
Finally, the Lagrange multipliers are determined by
a (V((0(0), L: s)
(()) m
m / M (t. (t.s)Ca(tt,)) + (m (t, s) ds
(C.43)
M (t, 8) Ra,,t, + (m- 1) (t, 9) Ro (t, ds.
(3) Boundary conditions.
As anticipated the SymmDMFT is initialized as
e (0))
CO,
CO 0) 0.
(C.44)
C.5 Expressions for train and test error under symmetric initialization The general expression for train and test error given in Section C.2 specialize to:
ctr(t) =
2 Ca(,,t),
(C.45)
(l(()) = 2 + ||<11 -2(())(((()))))))))))))))))))))) + 1 a + m (())((C66)))
(C.46)
D Numerical integration of the DMFT equations
D.1 Integration technique
using a standard Euler discretization. Namely, we discretize time on an equi-spaced grid == {0, 27,,... and approximate derivatives by differences and integrals by sums on this grid. As an example, Eq. CC.2 is replaced by
We integrate the SyFmDMFT equations (C.32 to CC377
all + 7) a(l) ((v(t))
Σ
Ra(t,s)
(D.1)
77
m
aET
a
Σ s)(ts))((s)
1 hhCC(lt,s) (---n(oo((ss)))
m 1 ()(Ca(t,s))Raattt
sET.stt
Σ
Calt. s) s)a(s)
()) n.
m
SET
The discretization of Eq. (C.10) deserves additional clarification because of the delta-function. For t. 2 tb, ta, to E Nr, we compute
(ty + n, tb) (ta, = -i to) _ō 1ta
Σ
MR 5) (1)())t))))
m 1=1
29

---

with boundary condition
(4y (b) = 0 Vi,j S m.
Of course, the solution of this system of difference equation does not coincide with the solution of
the original equations (C.2 to C.37). and in this section we will write a(t; Colt, 8;1) and so on
to emphasize the distinction.
Equations (C.42) can be directly interpreted as determining (t 5) and En(t, s) on the grid t, E T. Finally, discretize Eq. (C.19)
Σ t + ER(t, s)n) Rs(
-
"
sET
(D.2)
Σ [ + ER((, 8)]] Cs(s,() + EEC((, .ET
0.
sET
Note that we dropped the integration limits here, since they are enforced by the causality constraints implying [t, 8) 0, RA (t, 0 for / < Defining the matrices ΣR (En(t, 8) 1, T), and similarly for Ec, C., RA, we can rewrite (D.) as
I En RA I 17 I MNJ CA + E 0. We truncate these matrices (which are infinite) to maximum time T (c.g... ΣR (ER(t, t, s E T,s, & T)) and solve these equations by matrix inversion:
(D.3)
(D.4)
RA (I++TER) 71
(D.5)
= - +I+
Ec
(D.6)
We denote by a(t;n), n). v(t; m). Colt, s), Calt, s; m). (t, s;n), Ra(t, n), the functions obtained via the Euler integration scheme. We will assume that this solution is interpolated continuously for i, & T. For instance, for j E N a, b E [0, 1), we let
Ca((i 1)n,in:)) (D.7) Cal(t + (a)t ( b)m: m) =(1 a) - b) Calin, + a(1 - -
(1 - alb Ca((i 1)n.jnn)) ab Ca((i 1)n, (i + 1)::
Finally, while we described the discretization procedure for the SymmDMFT, the discussion above applies verbatimly for the full DMFT of Section C.I
The DMFT equations and their symmetric specialization have a causal structure which means that they can be integrated by progressively by increasing T. Furthermore there is no self-consistency condition the integration scheme at variance with the non-Gaussian settings, see for example (40). This simplification allows to investigate the long time behavior of the dynamics in numerical, rather efficient, way.
D.2 Accuracy of the numerical integration scheme
The discretization of DMFT is expected to converge to the actual solution with errors of order r). Namely, we expect
Ca(t, Ca(t,t) O()).
(D.8)
and similarly for the other functions. We refer to [3] for related examples in which the convergence was proved rigorously, and to [31] for an empirical study in closely related model.
In order to test the accuracy of our approach, and the correctness the DMFT equations, we simulated the gradient descent (GD) dynamics for the Gaussian model. Namely, we generate realizations of the process (a, W) = (f (a, W) S n) with the prescribed covariance (A.4), and the vector
_ (4p) : < n.) with same covariance as in Eq. (A.6) (see Section D.4 We define
2, (a, W)
via Eq. (A.8) and implement the following GD iteration
a Td (t + Man) aa at)
TGD
V
Ln (").))
(D.9)
wil (t + Mmn) Pse-1 (t)
MoD
V
u. R. (a (a), W
30

---

where Pga-i is the projector to the unit sphere, i.e. (z) >/||>]| if a & O and (0) = 0. Note that the trajectories of Eq. (.9) depend on the sample size n (and hence the dimension d = d,,) and the stepsize M/a To emphasize this dependence, we also use the notation a" (t;7)))))) W" T)a)
We expect the GD trajectories defined by Eq. (D.) approach the GF trajectories defined by Eq. (A.10)
as 1/ppp -> uniformly in n, Namely,
lim lim W (t; Rw)) - (t)ll = 0, MGu -0 n,d++00
(D.10)
lim sim a1: (aw) - a a" (t) 0,
(D.11)
Man -0 n,d++20
where the limits are understood to hold in probability for any fixed t. Informally, for fixed small 7/D GD dynamics is good approximation to GF dynamics, irrespective of the dimension.
We generate several realizations of the processes f", 60,, and of the gradient descent trajectories (D.9). We average observables interest over these realizations and compare these with the Euler discretization of the DMFT equations. For instance, consider the correlation functions Ci s). Then we can compare:
City (t,, 8; Nan) = E un (t; TeaD), w/ (s; (an)) where the expectation is taken with respect to the GD process D.9
(t, r)), the solution of the Euler discretization of the DMFT, described in the previous section.
Some results of this comparison are presented in the next subsection. This comparison allows us to gauge two types of systematic effects:
1. The effect of finite d. Indeed, the DMFT equations characterize the n, d - oc limit of the GD dynamics (D.9).
2. The non-zero stepsize 17. Note that the effect of discretization introduced in the DMFT equa- tions are different from the ones in the gradient descent (D.).. Therefore the disagreement between the two is measure of the nonzero-n effects.
To clarify further the last point, we emphasize that, despite the notation, City (t,, s; n) is not the n, d + 20 limit of Cij (t, s)
We note in passing that it is possible to derive DMFT equations for GD, hence characterizing lim, +00 (t, s; (KD)). Similar characterizations were obtained for related (simpler) models in 32). We defer the analysis of GD with large stepsizes to future work.
40
D.3 Testing the numerical accuracy
Figures and we present examples of the numerical comparison described in the previous section, under two different settings, as described below.
Setting 1. We assume pure noise data with r 1 and train network with m 4 neurons and covariance structure given by h(2) = 2/10 + 2²/2. We simulate GD trajectories, according to Eq. (D.9) with d = 100, 150. and correspondingly evaluate the Euler discretization of DMFT, cf. Section D.I for ã n/d 1.5.
We choose an initialization that is not symmetric and therefore we have to use the full DMFT equations of Section C.I More precisely, we initialize second layer weights follows:
(0) (0) 1 ag(0) as a4( --1
(D.12)
The weights of the first layer are instead initialized by generating two random vectors y1 3/ ~ Unif(Sd 1), and setting
wi (0) w²) 91
w2 (.) y
(D.13)
This initialization results in initializing the DMFT equations with
Cn 0) = C2, 0) = Ca, 0) = Ca, 0) = 1,
Cis(0, 0) (0, 1,
(D.14)
C12(0, 0) = Ci, 0) = C2a 0) = (3, =.
31

---

(4)
DMFT
0.5
1.5
0.45
0.5
9e
DMPT
DMPT
DMPT
0.4
0.2
(xr()) Simmiattous
61 DXFFT
0.5
(:) Simelktioos
ex(d)/2 DMFT
0.9
(y(1//2 Simelations 6,f0
0.3
Calt.0
0.25
0.7
05(0)
-0.5
eg(0)/
-1
0.15
0.4
10
20
90
0.1
-1.5
-2
30
Figure 6: Comparison between discretized DMFT and GD dynamics for the Gaussian model (labeled as 'Simulations'). GD results are averaged over N 10 realizations of the Gaussian process, under Setting described in the main text. Left frame: Second layer for DMFT and GD with TJDD 17 = 0.1. Right frame: Train error and correlation function for DMFT with few values of n7. and for GD.
DMPT 06
DMPT
0.4
0.5
(1)
DMPT
0.2
DMPT
(iy Stmula
0.5
(f) - Simeletion
ait - DMFT
0.4
esf0) DMFT
0.9
ax(d) Simelktions
ed( DMFT
8
Cait.0)
0.7
0.8
0..
oy(0) -
0.6
-1
ex())
<a(0)
0.2
0.5
0.4
0.3
-1.5
101
201
30
00
50
0.1
-2
2.5
200
411
70
900
100
2
14
Figure 7: Comparison between discretized DMFT and GD dynamics (labeled as 'Simulations'.. GD results are averaged over N 10 realizations of the Gaussian process, under Setting 2 described in the main text. Results for GD are averaged over N = 10* samples.
Both for the discretized DMFT and for GD for several values of the stepsize. The results of this analysis are plotted in Fig. 0
Sefting We consider again pure noise with t 1, network with m 4, input dimension d = 100 and sample size n 150. We use hidden neurons with the same covariance structure as in the Setting
However, we change the initialization with respect to Setting First layer are initialized indepen- dently and uniformly at random. It follows that
(0 = 5,g Vi.j = 1,... 1....,4
(D.15)
Second layer weights are initialized according to
1 a (1 = -1, a2(0) a2(
2
a1(0)
(D.16)
3
We stepsize 17 0.1.
32

---

D.4
Construction of the Gaussian process (
The Gaussian process f can be constructed as follows. Define sequence of independent Gaussian tensors ((k) E ( 2 1, with entries is.....
(J(h)
:ig S d) ridd N(0, 1). We then let
(a, W)
d1
Ch
(8) 11 Wi,h Wi,6a
(D.17)
is easy to check that this stochastic process has the prescribed covariance, with
M(=)
(D.18)
has long as the series above has radius of convergence larger than 1. An analogous construction holds for yo..
E Dynamical regimes: General preliminaries
In the next two sections, we will study the SyFmDMFT equations of Section C.4 and characterize different dynamical regimes the large network limit. From a technical viewpoint, we develop singular perturbation theory of the DMFT equations as m - co for fixed overparametrization ratio
= a
While singular perturbation theory classical domain of mathematics 9 1229. making this type of
analysis rigorous is notoriously challenging. We will proceed heuristically as follows: (f) Hypothesize
a certain asymptotic behavior DMFT solution a specific time-scale; (di) Check consistency with the DMFT equations; (iii) Check that this behavior is observed in the numerical solution of the DMFT equations.
More precisely, a specific dynamical regime identified by a scaling of the time variable, which in our case will take the form t (m)) for certain fixed function t4 (m) and scaled time of order one. The asymptotics of DMFT quantities in that regime takes the form (for instance)
lim m-+00 Cy (m)
Co (o (m)) (m)) 3; m, m
(f, a)),
(E.1)
where ey (m), co (L, 8; ax) are two fixed functions, the limit is understood to hold at fixed i, 8, a (0, oo), and we made explicit the dependence of C, on m, a. More concisely, we will often write the above formula as
Co (t E, tu (m) m;m,a ( c# (m)co 8; (m))(m)),
(E.2)
m
and we will typically use f, s instead of i, for the dummy variables. The behavior of the DMFT equations depends a crucial way in the initialization of the second layer weights:
In Section we will consider the case of a 'lazy initialization,' i.e. we will assume a(0) 7oym for some constant "o (0, oo) independent of
In Section we will consider the 'mean field initialization' i.e. assume a(0) = Ag to be constant and independent of m.
Dynamical regimes: Lazy initialization
As anticipated, in this section we study dynamical regimes under lazy initialization. In subsection
F.1 we will consider the case of pure noise data and in subsection F2 the k-index model.
Throughout this section, let (t) a(t)/ /m (in particular, 7 %
33

---

0
-02
m11
2
CCCalt, 0) 1)
-0.4
2
-0.6
m 211
212
2
-0.8
m 214
215
216
C (1,
-1
0.1
1
10
100 1000
Figure 8: Training of pure noise data: first dynamical regime. Rescaled correlation function () in the first dynamical regime function of the scaled time Im for model initialized with a lazy scaling and fixed second layer weights. Different curves correspond to the numerical integration of the SymmDMFT equations various values of m. They appear converge the scaling solution the large m limit described by Eqs. (F.6). Here 0.5, h(2) (3/10)= and T 1.
F.1
Pure noise model
Under the pure noise model, we have ④ = 0. Further, the variable v({) is not defined and can be dropped (equivalently, we can set (t) = 0).
We identify three dynamical regimes:
1. t = O(1/m): 7ft = 70 + om (1), train error decreases, and the network approximates the null function (Section F.I.II.
2. @(1): >(t) Yo + om (1), first-layer weights move significantly and train error converges to a limit e. (70) S.octionF..1.2 .. If To is larger than the interpolation threshold, then train error vanishes in this regime.
3. (m): This regime emerges only io is smaller than the interpolation threshold. (We discuss the identification of the interpolation transition of gradient flow in Section F.1.3)
If this the case, (t) grows on the time scale +(m) until it crosses the interpolation threshold. At that point the train error vanishes (Section F.1.4).
Since in the first regimes 7!)) does not change appreciably, the dynamics in these time scales is essentially equivalent to the one of a network which second-layer weights are fixed and do not evolve by GF. In Section F.I.1 and F.2.2 we first consider this case.
We note that the pure noise model is unchanged if we rescale
cr, 7o cYo- More precisely, this results rescaling of the risk by c² and hence of time by the same factor. As a consequence quantities of interest often depend on 7, uniquely through their ratio 7/T.
F.1. First dynamical regime: t ((1mm)
We first consider the case in which the (scaled) second layer weights are not updated and fixed to their initialization, i.e. r(t) = 70
It is possible to check that, up to higher-order terms, the SymmDMFT equations are solved by functions of the form (the first equation holds in weak sense, i.e. after integrating against a test function)
RA (t/ s/m) m ōōt 3) (m))
Calt/m,s/m) (t, + mm)
(F.I)
34

---

s/lt/m,s/m) 1 C. (t, 8) + om (1/mm (F.2)
Rolt/m,s/m) = m R (t, 8) + 0m (1/m)
Rall/m,8/m = 0(1 - 8) + (1)
Call/m,8/m) = =1 m
Clu (1,8) + 0m (1/m)
(F.3)
v(t/m)= = (t) + om (1) where C.. C.a Coi, ا and Ro are suitable functions independent of m. Here and below, we use the notation x(t 1(t 0).
(F.4)
Note that Eq. order 1/m.
H3 implies that on this dynamical regime the weights of the first layer change by
Plugging the asymptotic form in Eqs. F.1 to (F44 into the SymmDMFT equations and matching the leading orders for large m, obtain that the functions C Cal Co vhi and R must satisfy
nd (t)
- (10) (t,t),
0, 1 (t,t') (0)
a,C, (1,1")
(1+
(F.5)
0, (1,t") = (40) = -- - (8)h() (t, st.
These are a set of ordinary differential equations that can be solved explicitly. We get
R (t, 1') (t - t') ce a (0)(t-e))
(t, 20 (()zz 1
(F.6)
for > t, Ch (t, = -1 +e and (0) (Col(t,, (t',t)++1)
(t,t') + - (0041)) (0)())))
1
2207/1 + - --22386(00)))
fort > t
2
In particular, Eqs. F.6) imply
Cb: (1, = -1, 1+000
lim
(F.7)
R (t, -1. t t00
lim
Recalling Eq.
F.2 we conclude that
lim lim m Co(t/mmt//m) -1, t-x00 m0000
(F.8)
or, using the interpretation of C..
lim lim lim m (w:(t/m), w. (t/m)) = - Vi
(F.9)
In other words, at the end of this dynamical regime, the first-layer weights form a regular simplex, with center wtt/m) == m -1 Em wt(t/m) satisfying [|t]tmm]|| om (1).
Hence, at the end of the first dynamical regime, the first-layer weights are such that the linear component of the activation function is removed. In other words, for f a large constant, we have
m
(l:ia(t/m), W (t/m)))
70
oe nl
(w, (t,m)))) err,
(F.10)
m
where afl (w) is Gaussian process with covariance structure given by h(z)) zh'(0), and err is small in mean square.
Notice also that this is achieved by a ((1 /m) change each of the first layer weights. Indeed, by Eq. FF3 we have
2
lim [[) wt(t/m)ll2 2Ca(0.t/m)) #+000
C4 (0, + om (1/m)
(F.11)
m
35

---

10
10
Cytt.0)
-0.8
0.4
11.2
01
10
100
(1.()
10
Figure 9: Training with pure noise data under lazy initialization: second dynamical regime ((1). Left panel: First-layer weights correlation function Ca(t, 0) measuring the inner product between neurons at time 0 and time t. plotted versus for several values of m, and compared with the large m-assymptotics C22. Right panel: training error etr To: m plotted versus for several values of m, and compared with the large-m asymptoties in this regime e (t. 1). Notice the two-steps decrease of the training error, corresponding to the two regimes = (11mm) and = @(1). Inset: Same curves plotted versus tm, and compared with the asymptotic prediction ( 1) in the first dynamical regime. For both panels use 0.5, h(=) (3/10)= 22 70 and 1.
Equations regime: we get the expression:
F.1 to F.4 can be used to compute the behavior of the train error in this dynamical
lim etr(t/m) - (t). m=+00
(F.12)
Using Eqs.
F.6
v (1)
(7++))(11) (1)
(F.13)
In particular, the train error the end of this dynamical regime is
lim lim (tr(t/m) = lim e (t) [72 1 t00 m=
(F.14)
t+00
This is in agreement with (F.10). Indeed, note that
(a, W)
| - ft W)) +
1 - la w
(F.15)
2n
2n
Training in this timescale attempts to minimize (a, w)ll without fitting the noise.
This picture confirmed by the fact that Eqs. (F.6) depend on h only through h' (0). This means that the dynamics on timescales of order 1/m is controlled by the linear part of the covariance structure of the hidden layer.
In Fig we test the correctness of the asymtotic ansatz of Eqs. to F.4 Namely, we compare the results of numerical integration of the SymmDMFT equations for various values of m, with the prediction of Eqs. (F.6.. The match excellent.
So far we assumed that second-layer weights are not optimized and 7(1) = 70- What happens if drop this constraint? It can be checked that the form given in Eqs. 1ET--1441 still solves the SymmDMFT equations when a(t) allowed to evolve, and (t/m)) = % + om (1) for all fixed t (0, oc). In other words, second layer weights do not change significantly during this dynamical regime.
F.1.2 Second dynamical regime: f = +(1)
The second dynamical regime arises when / = (1). Recall from the previous subsection that, for L == om (1). the train error remains close (for large m/) to the plateau characterized at the end of the first dynamical regime, see Eq. (F.14). When of order one, the first layer weights start changing by an amount of order one well, and the model starts to fit the noise.
36

---

As before, we begin by considering the simplified setting in which r(t) is fixed and not optimized by GF. We claim that the SymmDMFT equations are solved by the following ansatz, up to lower order terms as m -> 0C:
v(t) = n (1) + om (1), Ca(t,t') = (t, + 0m (1),
(F.16)
(F.17)
(a(t,t) (1, 1') + om (1); Co(t,t)) 1 C (t, t')
(F.18)
0m mm (1/m)
(t, Om(1/m),
(F.19)
Ro,t,t)) - Ri (t. t') + om (1/m)=--- - Rd (t, + om (1/m).
(F.20)
Here C. R. Co R and غار are certain functiins independent of m. Equations 1.9
20
state in particular that C (t, (t, t') and Rt, are left with the task of determining Cj (t,t', (t, By substituting Eqs. (F.16) to (F.20) into
R (t, f'), and the therefore we
the SymmDMFT equations and matching leading order terms, we get set two integral-differential equations for Ot, ('), (t, ("), which we next state.
We first define
(1, s) := 70 (LF,))) R((0) (1, (1, (t,ss) := (t, (z) s).
(F21)
then we define 12 and C222 the solution of
5(t 1) = l(d [8(t -5-) 8) + (t, R (s, t's
(F.22)
0 L/¹ [8( s) + M (t, s)] C% (s, ds + L'" (f)
We next define the asymptotic form for the memory kernlss
Mh (t, s) = | [ (t. (C s)) + +
(F:23)
(t, a (t,
and we have defined
h(=) h(2) - 1 (0)=
(F.24)
The equations for n Ca2 and R are then given by
(t) -- //- [M S) + M (t, (t, Rt. ds,
(F:25)
(t,t) 8(t t') / (t)
(F.26)
a,C%(t,tt - -10 (
s) [' st) (f', ds
(F.27)
(As before, in the second and last equation, understood that t 2 t', and the last equation is understood to hold in weak sense.)
Given the constraints on Co., R,g, have the following constraints C22 R
(t, 1,
(F.28)
(t, = C²22 (s, (t, =0
(F.29)
VI S.
(F.30)
In particular, the last condition, together with Eq. F226 implies R t) 1.
37

---

0.05
10/7
dag
0.835
0.1
0.005
0.15
10.2
1000
5/0 0.202
10--2
0
11.26
113
0.30
10
10-4
1001
600
10-4
0.05
300
10"4
111
0.3
0..5
100
0.001
0.01
OGF
Figure 10: Training with pure noise data under lazy initialization: algorithmic interpolation threshold. Left Panel. We plot the train error as a function of time for different values of a at To 10/7. The train error has an exponential decay to zero for a below the interpolation threshold. Right Panel. We plot the time for GF to converge near-zero training error as a function of a, for various values of To. computed using the theory of Section F.1.2) The divergence of fa signals the phase transition for GF interpolation OGg- Inset: Trel versus in linear scale. Main panel: 4a versus OFF a (with the fitted value of ar)). Here we use h(2) (3/10)z + 22/2.
The evolution of the the train error in this second dynamical regime is given by lim (tr(t) = e (0) m m=000
(F.31)
oe (0) = ------0--0 ((,)),
(F.32)
where we have made explicit the dependence on the initialization of second-layer weights 70-
Note that Eq. F.22) implies (h2 (0, and Eq. F22 yields E 0) 72
(1) 7
/ (0). Therefore
1 - 72 To ( lim e (o)
lim (1, 70)
(F.33
In other words, this second dynamical regime captures the decrease of the training error which starts at the plateau reached in the first regime, cf. Eq. (F144. which coincides with the long time extrapolation of the first dynamical regime.
This second dynamical regime is fully non-linear and depends on the entire covariance function h. Further, the first order weights move by an amount [|,,,(() (.)] O1), as follows from the fact that (t, 0) < strictly.
In order to confirm the ansatz (6 to (:20), we compared the solution of the full Sym- mDMFT equations, with the solution of the asymptotic equations (F25). (F27 An example of such
a comparison presented in Fig. 9 the agreement is excellent.
The treatment above assumed the constraint (t) = "oo However, as in the first dynamical regime, we let second layer weights evolve, they do not change appreciably. Namely, the asymptotic form given in Eqs. (F.16) to (F.20) still solves the SymmDMFT SymmDMFFequations when a(t) is allowed to evolve. We have (t) %o + om (1) this timescale.
F.1.3
The algorithmic interpolation transition
For the discussion in this section, we denote by etr(t, y0. m.)a) the train eтror as function of t. where we emphasized the dependence on the initial condition Yo, on the number of neurons m, and on the overparametrization ratio We further assume that second layer weigths are not evolved and therefore (t) = %o for all t. We define the asymptotic train error achieved by GF as
tr (70, m. ax) = lim (tr(t, To, m, a)
(F.34)
= lim lim La W (()).
(F.35)
1000 -000
38

---

Again, in this definition a, m is kept fixed and does not evolve with time.
Notice that it is in principle possible that lim,- La (t,)) is strictly smaller than et m, a:) if we let f,, diverge with at sufficiently fast rate. However, based on results on related models in spin-glass theory we expect this not to be the case as long as t, is polynomial in Explicitly, we expect that, for any sequence t,
tn S n2
lim Bn (t.)) = otr (Y0. (70. m.a). a) n
(F.36)
Using the reduced equations for / = (1) timescale, i.e. Eqs. F.25 to F.27), we can also define
10 tr,oc 00, (70, a) := lim tr (t, 70. a) t-00
J2
(F.37)
lim lim lim (a W(I). 7 720 100
et (7o, m, a) coincides with ii (10, ar). This amounts to asking whether there exists dynamical regime with timescale (()) diverging with m at which etr (m)), Y0, m, a) starts diverging significantly from the value at the end of the second dynami- cal regime namely cl22 tr, (7o, a). If cr, (10: a) = then course limm Cr (t(m), Yoo m, a) =
natural question is whether the large m limit of et
as well.
however c (70. a.) > 0, then the answer depends upon whether the second layer weights are evolved with GF:
In the constrained setting in which second-layer weights do not evolve, we observe (from numerical solutions of SymmDMFT) ) that
lim Ctr (10, m, a) fr, (Yo. a). m m-000
(F.38)
In the next section we will see that (t) evolves with GF then the train error achieved on a diverging timescale t = (m)) is strictly smaller than e/e (7o, a) and vanishes for large enough t.
Note that etr, ot (70, m, a)) and cf (To: ax) also depend on the noise variance ?2. However, because of the invariance under rescaling discussed the beginning of this section (adding 7 as an argument):
ef (10, a, 72) 72 of (10/T r, a 7² 1).
(F.39)
and similarly for etr, (To, m, at. Because of this relation, we can think that 72 is fixed throughout, e.g. 22 1.
We expect ( m, a)), tr (1o. a)) to be non-increasing in Yo. and define the thresholds
(a, m) = inf { (To etr m, a) = =]. Yor (a) = inf (10 6 (10. a) 0).
(F.40)
(F.41)
(These definitions need be modified if 7 r to eo m, a) non-monotone.)
Of course, Eq. F.38 implies
lim TGF(a, m) (a).
(F.42)
The numerical solution of the SymmDMFT equations imply that the curve (ā)(a) is monotone increasing with as also suggested by the Gaussian complexity bound (see Section 2.2 in the main text). Hence we can invert it to get threshold (Tu): the two descriptions are equivalent.
In order determine a (7)), we adopt procedure already implemented The procedure is based on the observation (from numerical solutions) that when ef (7o, a) = 0, etr
BI for simpler model.
ei2 (t, 7o. a) exp(-t/tmaaa; + o(t)) for some (a) > 0 which diverges as 1 Car-
Define grid of values of a, Ao = {ai, 02, ... ......... .x., which are expected be smaller than agr (roo).
39

---

1.45
14
17
1,35
170
1.3
1.25
210
241
1.15
1.1
1.05
0.95
0.001
0.01
0.1
10
100
t/m
Figure 11: Training with pure noise data under lazy initialization: second layer weights in the third dynamical regime. Evolution of the (rescaled) weights of the second layer as a function of t/m. Here 2.5 and %o 1, 0.3, and covariance structure for the neurons given by f(z)) (9/10)z 22
2. For each value Ao. integrate numerically the reduced equations F225 to F.27 Verify that e for which this happens.
(t, Yo: a)) appear to converge to with / > oc. Let A C Ao be the subset of values
3. For each E A, define (a threshold value (we use E 10-
inf(t cf, Yo. az) <.) where is small
4. Estimate parameters (1)() c, by fitting the relation (ai((;;)) ~ clea az)
Figure 10 illustrates the calculation of og (10) for three values of 7o. In the inset we plot les for three values of 7o as a function of a. In the main panel, we demonstrate the divergence of hei when (atr a)) vanishes. In practice, observe suggesting this is the universal exponent for the divergence of tyel-
(
fit well the data across variety of settings,
F.1. Third dynamical regime: t ((m)
In the first two dynamical regimes, the large-m behavior did not depend on whether we would let second layer evolve with GF kept them fixed, i.e. >()) 7o-
In contrast, the behavior timescales diverging with depends significantly on the dynamics of second-layer weights.
If second layer weights are fixed, no significant further evolution takes place. In particular, the training error does not decrease significantly below the value reached at the end of the second dynamical regime, i.c. etr,o (70, a). This is stated formally in Eq. (F33).
If second layer weights evolve according to GF, then the dynamics on time-scales diverging with m can be non-trivial and depends on the second-layer weights initialization 70- If to > (a)) then GR reaches vanishing training error during the second dynamical regime, and no further evolution takes place.
However, if yo < (a), second layer weights start evolving when t O(m), thus giving rise to third dynamical regime. This is the object of the present subsection.
In Fig. 11 left frame, we plot the rescaled second layer weights 7()) (as predicted by numerical integration of the SymmDMFT equation) as a function of time for several values of m. Here, obviously, we do not constrain 7(1) 70).
We observe that >()) changes only when / = (m). Indeed, when plotted against l/m, curves obtained for different values of m collapse onto each other. This suggests that, for = o(m) y(t) (0) + om (1) (recall that 7(0) " by definition). Further, the curve collapse suggests that,
40

---

for any fixed E o0):
lim rfm,, 70) = (E, r0),
(F.43
where we have made explicit the dependence on 7o. Of course, the "/o (a) fits in this framework with 7Th (2, 70) = "Yo identically.
We next consider the evolution of the train error. Fig. 12 left frame, we plot the train error (again, predicted by numerical integration of the SymmDMF equation) a function of time for several values of m.
Again, when plotted as a function of t/m,, curves for different values of m reach plateau, and collapse below the plateau. This suggests the following limit behavior, which is consistent with Eq. (F.4)
lim ēir(L m,, 7o: m) - e (1, 70). m=400
(F:44)
(Here use éir(f m. 7)) to denote the train enror when second-laye layer weights evolve, in contrast with etr(f m, 70) which we used for the setting in which second-laye weights are constrained.)
Matching the present dynamical regime (t (m)) with previous one @(1), cf. Section F.1.2) implies that
lim Tr (f, 70) lim ef 70) oo (7o) f-20
(F.45)
t-+00
In other words, the function 2t describes the decrease of the train error below the level off (7)) achieved during the second dynamical regime.
In order to characterize the scaling function etr in Fig. 12 right frame, we plot parametrically the the train error for different values of m as function of the second layer weights ((t.. We also plot the curve (7, fr, (r))))))))))) (7)). This plot is consistent with the following behavior as m - co. first regimes (corresponding to = o(m) the train error has a drop that becomes vertical the m + oo limit, implying that (t) does not evolve while the train error decreases until reaches cr (70). In the last regime (corresponding to f = (mm)), 7(t) increases together with the decrease of the train error efr (t, 7o). Remarkably, they follow the curve (7,, el2 (7)).
In order to describe the last regime, we point out that + 7 / is monotone increasing. Therefore we can re-pa re-parametrize time by the value of the second layer weights. Namely, define 5/ the inverse function, so that
i i= 5- (i, 7o).
(F.46)
Using this reparametrization of time, the behavior in Fig. 12 can be formalized as
lim
eir (t, Yo. m) eir (f (5, no). 70) =: =(7; 70)
(F.47)
The collapse on finite m curves in Fig. right frame, onto the curve (y, (7)) suggests that
e(7, 70)
122 (7)
(F.48)
In other words, the dynamics on timescales of order is adiabatic: at each increase of >(t) timescales of order m, the train error relaxes to the the value it would have had the second layer weights would have been fixed in time at the corresponding value of 7.
A remarkable consequence of Eq. FF48) is that that
lim
b3 (i)
lim yor(a, m) (GP(a). m-+000
(F.49)
81000
In words, in the large network limit, the norm of second-layer weights at the end of training is asymptotically the minimum norm that allows for interpolation.
F22 Multi-index model
In this section we generalize the computations of Section F.1 to the case in which the dataset has structure produced via a k-index model. The weights of the second layer are set to a(I) = 71) /m and evolve with GF. The initialization scale (0) Yo is fixed and independent of m.
As in the pure noise case, we identify three dynamical regimes:
41

---

1.2
1.5
R
0.8
072
214
1o (T) 3/
0.2
f/m
Figure 12: Training with pure noise data under lazy initialization: third dynamical regime. Left frame: Train error on timescales order m. Right frame: GF trajectories the plane 7 (second layer weights) - etr (train error). Black dots represent pairs (7 tr. (7)). where err oo (n) is the train error achieved at the end of the first dynamical regime, cf. Section F.1.3 The data has been produced from the same model in Fig.
1.1
0.95
0.8
U.7
Calt.0)
0.4
111
0.8
711
(1.00
10 10F 10P 10 30
10
10
10' 10
30
tm
Figure 13: SymmDMFT predictions and large network scaling for lazy training in a single index model. Left: Projection v(t) of the first layer weights onto the latent direction on timescales of the order 1/m. The result for m + oo, الو has been obtained by integrating analytically Eq. (F.55). Right: The behavior of Ca(4, 0) on timescales i = ((1), compared with the scaling theory for m + xc, namely Cle. In both cases with h(z) (²z) (9/10)z 22/2, T 0.3 and 0.3,
yo =1
1. = O(L/m): r(t) To om (1), (t) w,)l))l @(1/vm)) On this scale the
network only learns linear approximation of the target. Test and train error remain close to each other (Section F.2.1).
2. ((1): ((1) = "o + om, [1w,(t) - (0 = O(). Test error does not change but train error decreases significantly (Section F.2.2)
3. (mm) This regime only emerges if Yu is below certain interpolation threshold, i.e. To < (a, s. r). In this regime y(t grows until the threshold, and train error decreases to
0 while test error decreases to 0 (Section F2.5
F2211 First dynamical regime: = ((1mm)
On this timescale, the SymmDMFT eunuatioeeuuttionss are solved, up to higher order terms, by the following ansatz:
Calt/m,s/m) + om (1),
Ra(t/m, = O(t 8) + om (1), (F.50)
_ct (t, 8) Om(m=1), Ro(t/m,s/mi = m C.lt/m,s/m) m
1 1 Rt, (t, 8) + om mm(mm -1),
(F.51)
42

---

Ra(t/m,s/m) -(t-
- s) + 0m (1),
Ca(t/m,s/m) = Dt 8) + mm))); (F.52)
a(l/m)/m== = 7o + +0 om (1),
u(t/m)= = .m (1) + 0m 1/2),
(F.53)
with
De (t, 8) = ||pl1 70 -VV()=))
od -o) (o (s)) + 70 (b(1) + (O)) (t.s))
(F.54)
In particular, Eq. (.50) implies ||un;(0) (t/mmm)]l Om (1): weights of the first layer change by small amount.
The scaling functions defined in Eqs.
F.50
F53
satisfy a set of equations that can be derived
directly from the SymmDMFT eqquttiinn
0,0" (t) - ona V (0)) (t),
0,C% (t,t') = 0% (V) h') (1 (t,
(F.55)
R. (t, -a-ihh(00) (0) (1 + R s)s
Note that
(t, 2 lim a,C, (t,
(F:56)
dt
The solution of Eqs.
F55 implies that
VΦ(0)
lim v (t)
70
(F.57)
lim (t,
Furthermore, on this timescale, the train and test error coincide and are given by
1 t2t
lim etr (tr(t/m)= = lim ets(l/m) m-
2 (t,
(F.58)
m=
The corresponding asymptotic value is given by
(72 + +1/111 h2(0)) |V( + +
lim lim (t(mmm)= =
bi 0s
(F.59)
t-000 100
2
where
h(2) h(2)) h0)
(F.60)
The interpretation of this dynamical regime is analogous to the one of the same regime in the pure- noise setting, confirmed by Eq. (F.59) the network learns the linear component of the data distribution.
In the left panel of Fig. 13 we test the scaling theory in this dynamical regime, as given by Eqs. (F,50) [.53p. We plot the solution of the SymmDMFT equations, versus Êm,, for increasing values of m the curve collapse well on their conjectured m + co limit.
F.2.2 Second namiical regime: t = @(1)
We next consider t = @(1). One can show that the SymmDMFT equations are solved, up to higher order terms m oc, by the following ansatz
Ca(t, s) C (t, s) + (1),
Ra(t, s) Rt s) + om (1),
&) 1 Ct,
om vm(mm (m-
Rt,,,)) - (t, 3) + mm(mm
(F.61)
T1
m
v(t) = Vm an + mmmmm 11/2))
p(t) (²)( + om (1),
with 7(1) %o + om (1) and
0 (t, s) -C
(t, -t, + 2
(t, = -
(F.62)
43

---

In other words, on this time scale first layer weights move by order one l|w;(t) (0)l== but in linear subspace that orthogonal to the latent space. Second layer weights do not move appreciably. As consequence, no additional learning takes place in this regime, but the model begins to overfit the data.
Note that the above scaling form is compatible with the long time limit of the previous dynamical regime.
In order to define the equations for the functions on the right-hand side of Eq. F.61 we define R and C to be the solution of
5(t - 12 = [e lō(t - +)++ IR, s)] R ts))))ss
(F.63)
/- 1ō(1 - 8) >ソ (1, x) (s, t') + /'" (E((,))
where
(1,)) 70 (M (1, 8)) k'())) R2 (1,5),
(F.64)
(t, = 7" ||o122 2 (o ) + n (t.)
Define the following memory kernels
(l,, = (CNNANCCCCCCH++++CC00N0)0)0C0)))
(t,
(F.65)
(t, 8) = anâh'(Cy' (t, C) (t, 8),
(t, = (t,
Substituting the ansatz (F66) into the SymmDMFT equations, and using Eqs. F.62 we obtain the following equations for (
(1, (') P ('), (1)
/. (t,
(t,t' +
v wnd (zo
S' ds M 3t (t, s)] ds (t,))
(F.66)
(t) ds,
(t,t) = - v (t,
(F.67)
L.¹ [M (t, RR (s, M (t. s) (s, ds, ds,
(t) a ) /
/'' [M [M (t + M (t, s) (t, ds
Lf' M MM RZ (1, s]] ds.
(F.68)
Finally, the train and test errors converge to well defined limits for t fixed and m + oo:
etr(t, 10) (t, 7o) + om (1), Cts efs 70) + om (1)
(F.69)
where
etr (t, 70) -
22
= 12 (t, 2
(F.70)
Note that, using Eqs. (F62 on the first layer weights), we get
F.64.. and the fact that (t, t) = (because of the unit norm constraint
23 70) = [ 7² + 1 27o 2i) + v (b(1) H'(0) + ((00))
(F.71)
44

---

Teiinn etror
Thain/TTtt.o
0.8
12
0.6
0.4
Figure 14: SymmDMFT predictions and large network scaling for lazy training in a single index model: train and test error. Left frame: train and test error on the time scale ((1/m) for several values of m, together with the asymptotic prediction as m - 00 on this time scale et (f. 70) = es (f, 7a)- Right: train and test error on the time scale = ( for several values of m, together with the asymptotic prediction
on this time scale eir (t. 70). Here To h(2) ((2) 2 2² //, 0.3, 0.3.
Using Eq. (F57). we obtain that the asymptotic test error in this dynamical regime is constant and equal to the test error achieved at the end of the previous regime, namely et (t, 7o) (s cf.
eel
Eq. (F.59). As anticipated, no learning takes place on this timescale.
The predictions of Eqs. (F61) are tested in the right panel of Fig. 13 We plot the correlation function C) for several values of m, as obtained by solving the SymmDMFT equation.. We compare these results with the - oo prediction C (t, 0) obtained by solving Eqs. (F.66) to (F68). We observe collapse of finite m curves on the large asymptotics supporting our conclusions.
In Fig. 14 we plot the behavior of the train and test error both on timescales t ((1/m) (left frame, plotting (t, 7o), (t, 70) versus tm) and
((1) (right frame, plotting (tr yo). ets(t, 70) versus tm). We the solutions of SymmDMFT equations at increasing values of m with the theory scaling theory presented the previous section (for ④1 (11/m). ) left frame) and in this section (for t == (1), right frame). As anticipated, we observe the following:
On the time scale common limiting curve ett (ł, 70) ets (i, (o) which converges, for large i, the positive limiting value fs characterized in the previous section.
(m) (left panel), test and train error collapse (as m + oc)
ell
On the time scale @(1) (right panel), test and train error collapse (as m - oc) on two distinct limiting curves. The first one is constant and equal to ts The second one decreases from E,, 0 and is predicted by the asymptotic theory in this section, cf. Eq. (F.00).
Note that, the example Fig. the initialization "o sufficiently large that the train error de- creases to zero on the time scale ( namely 7o > (ar s0, for suitable threshold Yar (a, 92, T). As we will see in the next section, third dynamical regime emerges when "o Yor (a, $2; T).
F.2.3 The algorithmic interpolation threshold
The asymptotic theory within the second dynamical regime, described in Section F22.2 turns out to be equivalent the one in the pure-noise model, Section F.1.21 up change of variables. Namely, defining
C, s) Ch (t, ||
(F.72)
with initial condition Co(0, = -1 reduce the equations of Section F2.2 to the ones of Section F.1.2 with noise level replaced by
12 22 -111112 (+)())1
(F.73)
h'(0)
45

---

K.la) - ---2-
0.25
1,13)
-6 0.05 0.95
-0.05
(.(
2/ +/ -.35
1011
0.2
18.00
0.15
(m) Tool
10-2
(m)
10-3
0.05
22/2
10-4
Ralg)
-0.05
276
Figure 15: The asymptotic behavior of the test error as a function of m for different h(2) (()). We observe that soon as h(z) contains 2² term, the NTK limit for m - oo is approached from below (left panel). Furthermore the speed of the convergence to the limiting value depends crucially on whether a z² monomial is present in the Taylor expansion h(z) (right panel). The data has been produced with = 0.3 and 0.6.
The interpretation of this reduction simple. On the time scale t = (1), the first layer weights move orthogonally to the latent subspace spanned by U. Hence, the dynamics this timescale not affected by the signal and only attempts to fit the labels noise. The noise inflated as per Eq. (F73 because the network is not able to fit beyond the linear part of the target distribution.
As corollary of the above equivalence, the interpolation threshold of the k-index model coincides with with the interpolation threshold on pure noise data with noise level given by Eq. (F73). Using the extended notation (a, sp, indicate the dependence the underlying data distribution (which is parametrized by 9,, T), we can write the stated relation as
Var p), (7² (+² 11112
(Q)1)) 1/2 Tar (a, 0,
(F.74)
(0)))
(Here we used the invariance under rescaling in the pure noise model, which implies (a, 0, 0?) TYa 1).)
F.2.4 Dependence on
Within NTK theory, it is normally assumed that optimal models are achieved at very large network sizes m + 30. Empirical results contradicting this expectation have been put forward in [54). but no theoretical analysis was provided either in (54) or in subsequent work. We can use the SymmDMFT theory to fill this gap and study the dependence
test error on the number of neurons
under lazy initialization. We choose "to (Gr s,, -), and therefore vanishing training error is reached during the second dynamical regime, i.e. for ((1): this is therefore the last dynamical regime. Throughout this regime, we have 7(t) To + om (1).
Recalling that ets (t, Yo: m. a:) is the test error at time in this setting, as predicted by SymmDMFT we consider the limit
e (70, m, a) = lim ets(t, 70, m, a).
(F:75)
to000
We note that, for % > (a p, 7), we expect
lim (10. Cts,
1el
(F.76)
to be given by Eq. F.59).
In Fig. 15 we plot the SymmDMFT prediction for (70. m, a) as function of m for several choices of h (we use h + here). The limit - oo of these curves matches c8 expected.
However we empirically observe that e (10, m, a)) approaches 10,00 two qualitatively different ways:
dlt
46

---

4.5
4.5
3.5
2.0
2.5
1.5
0.6
0.5
20214
10
10-10-9 0.0001 0.01
4/mm
Figure 16: Train and test error on different timescales when training on single index data and lazy initialization. Train error (solid curves) and error (dashed curves) for model trained on single index data with /(2) (9/10)z + 2 ((2). The noise level is T = 2.5 and initialization a(0) = /o. To Yār
Tar(ar (a, (.T)) Left panel: timescales of order one. The grey dashed line corresponds to the scaling solution for panel: same data plotted versus t/ m, explore timescales of order m. The arrows show scaling appearing and curves collapsing on master curve.
when the second layer does not evolve with GF. Right
In the cases consider that have h²(0) * 0, ciss is approached from below and eoo (7o, m, a)) is non-monotome We also observe that, for the values of m we consider, the approach to the asymptotic value is compatible with a rate m 1/2 e (10: m, a) = glet to 1
oc.
In the cases we consider that have h (0) * O, then ts is approached from above as oc, and e'o (7o, m, a) typically monotone. In this case the approach the limiting behavior is compatible with rate m e (7o, m, ts O(m=l).
The first scenario is the generic one, and similar what is observed in [54 for actual neural networks. An intuitive explanation is that aat fimite m- the projection of neurons onto the latent space || ( /m) sufficient for the network to partially learn the quadratic component of the target function. In order to establish on more solid grounds these empirical observations one should study the 1/m corrections the scaling theory developed here. This is left for future work.
F.2.5 Third dynamical regime: / (mm)
As for the pure noise case, beyond the time scale (1), we distinguish two situations. Tu > Yáp (a, i,, T), then vanishing training error is reached within the second dynamical regime T), GF dynamics develops an additional regime for ((m). this
t= (1). to < section, study this third regime.
In Figure 16 we plot the SymmDMFT predictions for train and test emrors function of time for several values of m, for a setting with "No < (a, sp, T). ) In particular, in Fig 16 left we plot train and test error
function of t. The curves for the train error for increasing value of m collapse on limit curve given by el2 (t,, 70) characterized Section F.2.2 In other words, the dynamics on this timescales follows the scaling theory of Section F.2.2 However in this case "Yo < TG (p, 7),
whence by definition li2 tr 0. This correspond to the limit curve in Fig. 16 having a strictly
positive asymptote.
Figure right shows train and test error plotted against t/m. We observe that curves training error curves collapse common limit, that decreases from e/ 0, while test emoo curves increase above the plateau elz] This suggests the following limit behavior
lim (mrm 7o, m) = ex (i, 70)
(F:77)
lim (mf, No. m)
cts 70)
m 000
47

---

15
01
4.5
3.5
R0
1/w
2.5
122
1.5
0.5
1.3
1/m
Figure 17: Training a two layer network in the same setting of Figure 16 Left panel: second layer weights on the timescale of order m. The black arrow corresponds to the interpolation threshold for a model, ( 7) obtained by fitting the relaxation time as function of the weights of an lazy initialized model for To > VCF (a, T). The second layer weights, at finite m develop plateau at long time. In the inset we show the approach of this plateaus to the limiting value given by Yār y0, r). Right panel: parametric plot of the train error function of the scaled weights of the second layer. The dashed gray dashed line corresponds to the extrapolated train error for an network with second layer weights fixed to the corresponding value the m + oo (as extracted from the numerical integration of the scaling theory).
In order to further explore the GF dynamics in this regime, in Fig. 17 we plot the evolution of the second layer rescaled weights against t/m. The curves for increasing values of m collapse on master curve, suggesting the existence of limit
lim r(mi, 70) - ((, m-+00
(F.78)
The limit curve (t, 70) increases from to to a limit value:
lim (f. n (70). (+00
(F.79)
As in Section F.2.5 we consider the inverse function of L 71, 70). denoted by 7 7- (7, (7,70), 70).
In Fig. 17 right we plot the train error as a function of the second layer weights 7(1. Again, for increasing values of m the curves collapse master curve which is given by
E(y, 20) = c (5-1(77 (7),70),70 7o), 70)
(F.80)
We then also plot in Fig rrihhtht the asymptotic value of the train error for network initialized with second layer weights blocked an initialization scale y, call it et. (7).
The curves c(7, 70) appear to have vertical segment (corresponding to L. o(m) in which the training error decreases, while 7l) = 7o 770++0n Om (1) is nearly unchanged, and continuously decreasing segment in which (() increases while (t, 70) decreases to 0 (corresponding to t = ((m)). In the second phase, the curves appear to converge to etr,
(7) as m + Dc. This suggests
s(7, 70) = te V7 27200 70
(F.81)
In other words the dynamics timescales of order m is adiabatic also in the multi index case. For a small change of the second layer weights on a scale order /m, the train error relaxes to its asymptotic value on timescales of order one. This graph suggests that the limit value of x() coincides with the critical value for interpolation. Namely recalling the definition (F79p for the asymptotic value of ((), have
(10) = THP (a, P)
(F.82)
where the interpolation threshold in the multi-index model &0 (a, sp, is related to the interpolation threshold in the pure noise model via Eq. (.74).
48

---

G Dynamical regimes: Mean field initialization
In this section we assume the initialization of the weights of the second layer is kept of order one. To be definite, we set (() = du, independent of m. This corresponds to the mean field initialization
studied in 38 14 45
Specializing to the data distribution considered here, earlier work characterized the dynamics up to time 7, under few settings (which prove equivalent in this regime):
One-pass SGD, with stepsize < 1/d and therefore time horizons such that T < d/n n (the latter inequality follows from T S. ns for one-pass SGD). In this case, the dynamies is characterized by a set of ODEs for for the projections of the weights on the latent space and inner products between weights.
Gradient flow in the population risk, which admits the same characterization and corresponds to the limit
of the above.
The limit of the above regimes for large width m - oc. This characterized by partial differential equation for the distribution of projections of first layer weights onto the latent space, provided T S Co log m, for oo a sufficiently small constant.
We refer to 21
10 for few pointers to this literature. In all of these settings, the train error remains close to the test error. In contrast, the analysis presented here allow us to explore the overfitting regime.
Section G. we will focus on a pure noise data distribution, while Section G.2 index model. As in the case of lazy initializations, we consider first the limit n, d + oc at n/md = a and fixed (hence characterized by SymmDMFT) and subsequently study dynamical regimes emerging as m + oo at n /mmd= = a fixed.
considers multi-
G.1 Pure noise model
Under the pure noise model, we have & 3 0. We identify three distinct dynamical regimes:
O():: a(l) au + (l,(). (1), etr (t) 72/2 + Om (1), and ((x,(() - w,(O)]l = on (1). In words, the weights change minimally and the train error remains close to the one of the null network f(æ; 0) = 0 (Section G.I.I).
==((ym): (t) @(1), etr 72/2 + om (1), and (() w: (0)] (1). Namely, weights change but the train error does not change significantly. (Section G.1.2).
t= =+(m). In this regime att) /m>(tt/) + om (1), and therefore the network com- plexity becomes large enough for it to fit the noise. The dynamics on this timescale is closely related the one under lazy initialization, studied in Section F.1.4. In particular,
7() converge to the interpolation threshold (a, T) if f > oo (after m oo). (Section G.I.3).
G.1.1 First dynamical regime: O(1)
In this dynamical regime, the CinamDMMFeqquttions are solved by the following scaling ansatz
Calt, 8) = 1 + om (1)
Ra 8) = =O(t 8) + om (1),
(G.1)
(t.(t, 8) = C.all (t, + om (1)
mR. 8) = =Rr (1, 8) + om (1),
(G.2)
all) = ap + 0m 0m,()
u(() Om (1).
(G.3)
Furthermore we have
R (t, 8) = ö(t -) (1) (t, = -2 om (1),
(G.4)
Plugging the scaling ansatz in the SymmDMFT, we obtain equations determining the scaling functions Calll Rum Defining
Po = (0)
(G.5)
49

---

-0.1
0.2
-02
-0.3
-0.4
Cult.rim
- 122
CL
-0.6
-0.0
0.7
0.9
Figure 18: Training on pure noise data under mean-field initialization: / = (1) regime. We plot (t, 0) and (o,t, t) as given by solving the SymmDMFT equations for different values m and compare them with the asymptotic solution of Section G.I.1 Here we use = 0.6, 0.3 and f(z)) (9/10)z 2 /6. Note that the vertical axis is multiplidd by factor m, in agreement with the prediction of Eq. G.2
have
nolt-a)) - -1]0(( 0(1 - 8),
1e (1,
Carll (t,
27
1 (x Po)
2pol
72 - e poll-e)
(G.6)
Po
Po.
Po
72
Pn
72
pood' pot
Po
Po
particular
72 PO
lim C (1,1)
P0
lim Omil (t,t')
72
PO
7
P
(G.7)
Po
Po
72
lim
Cm (t,
t- -
Po
The equations (G.4) imply that the train error constant in this regime and equal to
er
om (m)
(G.8)
In other words, in this regime both first and second layer weights change minimally and the resulting error remains close the one to the null function f(z; = We will see that this regime is
significantly more interesting for the case of data with signal, see Section G.2 We note in passing
that the limit value (w,, w)) 7--po for + corresponds minimizing the empirical risk under
the linear approximation in which (22) ) is replaced by Vh((0)z.
The above predictions are tested in Fig. 18 where we plot Co t) and Co (t, 0) for different values of m and check their approach to the scaling functions C (t, 0) and Cuill (t, t).
G.1.2 Second dynamical regime: (v/m)
We now consider the case in which time scales vm. The following asymptotic forms can be checked to solve the SymmDMFT equations, up higher order terms, for suitable choices of the scaling functions on the right-hand side:
Caltym, m, sym)
C (t, s) + om (1)
Ralt/m, sym) = R s) + om (1),
(G.9)
50

---

1.4
0.
0.2
40 ml
211
110
-0.4
211
0.8
015
2114
-0.0
0.4
0.7
10.31
Cr
1/
1/ym
0.4
-02
2.5
-0.4
-0.6
1.5 0
-0.8
1000
f/
f//m
Figure 19: Training on pure noise data under mean-field initialization: O m regime, under the same setting as in Fig. 18 We plot the solutions of the SymmDMFT equations for several values of m as function 1/ Vm. compare these to the m + oo scaling theory of Section i.e. to numerical solutions of Eqs. G.14 to (G.17.
(ittmmss/m)
mma Ctt(tt (t, mm Ro(tVm,smm) -R
pmf2 (t, 8) + om (m(mm=))
(G.10)
VmRAllym, 8/m) = 8(1 8) + om (1) /mv(ty/m 2 (t) + pm (1)
CallVm,8m = -7² + 0m (1), (G.11)
a(tvm) = a प. (t) + om (1) GG.12
Plugging this scaling ansatz into the SymmDMFT equations we get the constraints
Rea (t, 8) (1, s):
22
(G.13)
CT -CT (t, ah (t))2
We also obtain that the following equations must be satisfied by Cir (t, t'), R (t, t'), (t), (t)
O,Caf (t, m22 (t) (t, ar (t) S a (s) (t, Rup (t, 8) C) (t (t',s) ds
(G.14)
+ am22m12 (8) 1" (s) (C'(CT (1(()) Pe ds,
R (t, (4 = 5(t (/4 cumii
(8) Pil (t,t')
(G.15)
+ ar 2 (m (1)
m. (1, 8)) (1, (),8) R) R2 (s, d'
= 1.r2 (t)= Tond
Mari (1) 1a (s)"" (t, R 8) 3) s)] ds
(G.16)
51

---

+ aar (m))) /'" ( (t,CC - h'(o] R (t, 8) ds,
dama at² J (s) [N'(CT s)) - (0()) 1 ds
(G.17)
dt
with initial conditions given by
Ca ()
Re (0+, 0) =1
md2 00) 40
(G.18)
We test these predictions Fig. We plot several quantities the solution of the Sym- mDMFT equations for increasing values of and compare them with the solution of the asymptotic equations (G.14) to (..17). We observe convergence to the predicted asymptotic behavior.
Equations (G.14 14 to (G.17) can be further simplified. The right-hand side of Eq. (G.17) is positive.
Therefore emin( is monotone increasing function. Define the time change
ē(t) = Tva L. (s) ds,
(G.19)
and the corresponding time-changed scaling functions
D (1)
=(E(t)) a (1)
(G.20)
TR (f((t, E(t') C. (t, t'),
Pe (i(1), [(t') = 122 (1,t').
Equations (G.14) to G.17 imply that these time-changed function functions satisfy
0, 0, (1,1') Dmd (T) (1, { ii(Ca (t,))) Ri (t. S) (C', 8)
(G.21)
+ ff
(1,, &),
O, (t, (t, ) 8(t (1) + S. (((((((t,
(G.22)
(t) = /-¹ hu" in((oT(t (t, s) (t, 8) (t, (t,
+ [ Rt, (G.23)
where again h(2) h(2)) h' (0)))z
Equations G.21
G.23 are independent of the dynamics of the second layer weights. These equations are nothing but the DMFT equations describing gradient descent dynamics of the celebrated
spherical mixed p-spin glass model 17 20 g 23), whose definition we recall next. Consider random cost function H (a:) indexed i. E S which is a centered Gaussian process with covariance structure given by
((H =dip)))))dd(aaa) di((a, 9)).
(G.24)
Define the gradient flow dynamics
a(t) -P PP (+))
(G.25)
where Pe(e) is the projector orthogonal to xt)). Then the high-dimensional asymptotics of this
dynamies is characterized by Eqs. (G.21
G.23 In particular lim (()))) ČT (1, 6')
almost surely.
particularly interesting quantity is the asymptotic energy value in the mixed p-spin model:
lim lim t d (x(()))
(G.26)
The DMFT analysis for this problem implies that
8 = lim ( N'(OT 8)) R (t, 8) ds
Jo
(G.27)
52

---

10
5
20
40
90
100
120
140
Figure 20: Evolution of second layer weights, predicted by the numerical solution of Eq. (G.17).
Here we use l=(=) (9/10)= 10 23 //6. a = 0.3 and r = 0.6. The straight line is just a guide to the eyes to the prediction Eq. G.29
For h(z) = 2 2, we have the explicit expression
20 99 46
h
6 -2ck
(G.28)
An explicit expression for for general covariance structure is an unknown [23
The asymptotic energy has an interesting interpretation for the dynamics of two-layer networks -within the SymmDMFT theory. Eq. (G.28) implies that
a m12 (t)
lim E-+00
Vo = A
(G.29)
t
In Fig. 20 we test the prediction of Eq. (G.99 by integrating numerically Eqs. (G.14)
G.17 and
plotting the prediction for the second-layer weigths (t). We observe that at large t, a am 2
Agt,
with Asc given by Eq. (G.29)as as predicted.
We also note that (t, t) 72 also in this timescale, and hence the train error does not change significantly, Namely. for any constant t, we have
eu(lym) 2 + (m).
(G.30)
If we use heuristically Eq. G.22 and Eq. (G.12) beyond the Vt time scale, we obtain
a(t) == am (t/ (/)m =o Ase mm
(G.31)
This suggests that a(t) becomes of order /m on timescale of order m. When this happens, the network complexity is large enough allow for interpolation, and hence expect the dynamics to change. Indeed new dynamical regime emerges for ((m), as we will study next.
G.1.3 Third dynamical regime: / - (m))
As anticipated, an additional regime arises on timescales of order m. In Figure 21 we plot the evolution of the weights of the second layer as function of t/m for increasing values of the width m. The different curves collapse suggesting the following limit to exist
a(lm) Vm
lim +00
7 MES (1).
(G.32)
The limit curve appears to grows linearly at small mi Aool + o(t), where A is the coefficient computed in the previous section, ef. Eq. (.29p. Hence, this third dynamical regime matches directly with the previous one. As can seen from the right plot, there appear to be finite
limit lim m() (t)
53

---

o/0
0.3
0.2
0.1
10
f/m
4/mo
Figure 21: Evolution of the second layer weigths when training on pure noise data under mean field initialization for (mm). Rescaled second layer weights a(t) m as function of t/ t/m. We plot solutions of the SymmDMFT equations for the setting of Fig. 18
0.07
0.2
0.06
100
0.15
0.04
8
212
0.1
215
214
0.02
8.01
-1.01
-0.02
0.001
0.01
c/m
c//l
Figure 22: Train error and Lagrange multiplier v(() on timescales of order m under mean field initialization for pure noise data. Solutions of the SymmDMFT equations for the setting of Fig. Finite m curves accumulate on master curves suggesting the existence of scaling functions.
We now turn to the analysis the train error. Recall that on the previous timescales, the train error stays approximately constant, and equal to the train error of the null network, namely etr( (trtt/m) 72/2 + Om (1) for any fixed l. Fig. 22 we plot both the train error and the Lagrange multiplier as a function of l/m. Again, m grows, these curve converge to limit curves. This suggests the existence the following limits
lim etr(tm) = e et) m-+00
(G.33)
lim v(t)) = (t). m0000
(G.34)
Note that in this case, differently from the lazy initialization setting, the corresponding scaling function do not depend on the initialization parameter dg.
In order to characterize the limits in Eqs. p G.34).
we proceed as Sec. F. Namely, in Fig. 23 we plot the train error and the Lagrange multiplier v as function of the rescaled second layer weights 7 = att)/ /m. We also plot the asymptotic value of train eттor and Lagrange multiplier under the constrained GF dynamics in which second layer weigths fixed to a(t) 7 and
do not evolve with time: Ix2 (7) := lim,,soc (t 7) and (n)) == lim 2 (t, 7). These are
computed by integration of the scaling theory in Section F.1.2
The good collapse on these curves suggests to consider the the following construction, analogous to Sec. F.1.4 Define the inverse function of (t),, denoted by (11 Then, define
m83 (7) c <m? m(3 -1()),
(G.35)
y (7) = m0 ((r ((7))
54

---

(cc
0.06
0.2
0.35
0.04
0.15
0.2
01212
0.15
0.05
10 10
10
-11314
(22(1)
-0.06
0.3
0.6
0.3
Figure 23: Train error, rescaled second-layer weights and the Lagrange multiplier v(() on timescales of order m under mean field initialization. Left Panel: parametric plot of the train eπrоr as function of the weights of the second layer on the scale vm, namely 7 = a(t) mm. The inset shows the same data logarithmic scale. Right Panel: same plot for the Lagrange multiplier v. Data is the same as in Fig. 18
Figure 23 suggests that
m83 (7) = 20 y (7) = 10 (7))
(G.36)
(G.37)
(G.36) imply that on timescales of order m the dynamics is adiabatic. For each incremental change of the scale /m, all one-time quantities relax to the asymptotic value which turns out be the same as a constrained model with a(t)/ (tt///m- = fixed.
Equations G.36).
The consequence of Eqs. (.36))(G.36 is that
lim (1) (G, (+20
(G.38)
where (a, 7) corresponds the interpolation value of the initialization scale of lazy model.
G.2 Multi-index model
In this section we consider the case which the dataset distributed according to multi-index model. For time scales beyond t O(1), we will assume that h(2) (f(2). This simplifies the asymptotics for t large but order one.
We identify two dynamical regimes emerging as m + oo:
O1)): a(t) O(1) but is not constant. Also, the projection v(t) of first layer weights onto the latent space evolve as well as do train and test error. We further have (tr()) ets (t) om (1): there is overfitting. This evolution is captured the mean field theory 38 14 which we recover as m - co limit of SymmDMFT.
t = (mm): att) = @(ym), v(t) decreases towards and train and test error diverge. In this dynamical regime the network unlearns a large extent the latent structure of the data and overfit it.
G.2.1 First dynamical regime: ((1)
For timescales of order one, the SymmDMFT equations are solved, up subleading terms as m + xc, by the following ansatz
Ca(t, Caati (t, 8) + 0m 0m((): (1),
Colt,8) (t, + om (1), (t,
(G.39)
Ra(t,s) s) Ri (t, 8) + 0m (1),
mR,(t, /e (t, 5) + om (1)
(G.40)
e(t) mlll (t) + pm (1)
a(t) = a (t).
(G.41)
55

---

013
5.5
IN -00
3.5
0.3
1.5
11.1
Figure 24: Gradient flow dynamics under mean field initialization in the first dynamical regime
t (1). for data distributed according a single index model. Curves are numerical solutions of the SymmDMFT equations: we plot (t) and a(t) for different values of m and compare them to the mean field predictions. Data distributed according to single index model with (2(2)) (0()) (=2) (9/10)z + 2 /6 with 0.6 and 0.3.
0.8
0.8
01.7
0.7
0.6
0.6
0.5
m
Thin; Tist
Tex
0.3
0.3
0.2
0.2
0.1
0.1
10
10
Figure 25: Evolution of the train and test error on different timescales under mean field initialization a0)) 1. The train (solid curves) and test (dashed curves) errors function of time t (left panel) and scaled time t/m (right panel). Curves are numerical solutions of the SymmDMFT equations for h(z) (9/10)= 2²/ /6, p(z) h(2), 0.6 and a 0.3. The arrow on the right panel corresponds to the asymptotic test error for a model with second layer weights fixed to the corresponding interpolation threshold.
The corresponding scaling equations then given by
R. (t,t') mit (t) (t, t') ma (t) (t, t)) (Rat (t,t') + R (t,t')
0, 0. (t,t')
mi (t)C (t, + ( (t)), mil ( (t) aa mi (t) h' (t,t))
will (I) ax (V (1)), (1))amm (1) aa m (1) () (C (1, (1, 1)
0, Cmll (t, t') = mil (t) (t, a (VG(( (t)), u
(t (t')) (t) - aa mi (t) (t.t))
o R (t, t') = - "M (f) (t, 8(t (t),
mil (t) - nn (1)o (1) aV (1))a (1) aa (1)2 (C".!! (1, 1)) (1),
0 mil (t) (6(1"") (t)) nd (t) (t,t)))
(G.42)
These equations are solved by setting:
Craat (t,t') mm (t), mi (t"))
(G.43)
56

---

03
40
mntott 11
25
10
-2
Figure 26: Finite width corrections. The 1//m corrections to the second layer weights and the projection on the latent space of the single index model on timescales order Dashed lines are obtained by integrating numerically Eqs. (G.56) to (G.59) determining the limits m + co. Here, ((z) hu(2)= (9/10)z + 2d /6 with 0.6 and a 0.3.
with u (t), amt (t) the solution of
0. (t) aa maamm (Tx 1 "0 (t) (1) (V)("""(()) a (1) (1))))))))))) (1) (1))
0.amil (t) (+6() (amm((() (t)) mm (1)
(G.44)
with initial conditions given by w (0) 0 and mn (0) ag.
Equations (G.44) coincide with the mean field theory of 38 14 45). when the latter specialized
to the multi-index model studied here, under symmetric initializations [10] (See also (2). Using the
ansatz of Eqs. (G.39) to (G.41) in the formulas for training and test enror (C.45). (C.66) get
lim etr(t) lim ets emil(() m-
(G.45)
m--20
with
(t) 1
1 2a mil (t) (t)) + (t)
(G.46)
particularly simple case is the one in which k (single index model) and و (whence & = h). For class of such activations with h h²(0) > O, we have am (t), wm (t) + 1 as t - co and therefore
T
lim
mll
(G.47)
In other words, neurons align perfectly with latent direction, the generalization error vanishes, and and train and test error converge for large constant to the Bayes error 72/2.
Fig. 24 we compare the solution Eqs. (G.44) with the numerical integrations of the Sym- mDMFT equations for range of values of m. As m increases, the SymmDMFT solutions converge to the asymptotic predictions 1m (t), (t), confirming the above ansatz.
Similarly, Fig. left panel we compute mDMFT equations and compare the results the asymptotic prediction provided by Eq. (G.66). We observe that aas predicted- train and test error match on an increasingly long time interval. At certain point, they diverge: will next characterize the timescale which this happens.
train and test error by solving Sym-
G.2.2 Escape from the mean field dynamical regime
In order to understand which time scale the dynamics diverges from mean field theory described above, we will study small deviations from this theory. We expect that these deviations will diverge with time. Characterizing this divergence will allow to determine time scale which we exit the mean field regime.
We focus on the case of single index model 1, with & h, and set a()) = We believe that the qualitative conclusions obtained in this case apply more generally. We also assume h to be such
57

---

that the long time asymptotics of mean field dynamical solutions is
lim a mft (t) = 11 t-00
lim ym (t) =
(G.48)
t-000
As mentioned in the previous section, this holds for broad class of activations. other words, for time large and yet of order one, the neurons are very well aligned.
We next study the corrections to the mean field solution. We claim that such corrections are of order
1/m m and define the functions @(t), ū()), dots _R,((,,)) ('), via
m(a(t) amm (t) = ä(t) + (1),
(G.49)
m(u(t) ymm (t)) = =6(t) + om))),
(G.50)
m(Ca(t, f') Cx (t,t') = Ca(t,t') + (1),
(G.51)
m(Coll, - (l, = C,((,,') (') + om (1),
(G.52)
m(Ra(l, (1, = Ro(l, 1') + om (1), m(Bo(t,t') (t,t)) = tt, + o. (1),
(G.53)
(G.54)
m(p(t)- - ~nth (f)) i(() (l)
(G.55)
Substituting the above form into the SymmDMFT equations and matching the next-to-leading order in m we can obtain the equations for the 1 1/m corrections. It turns out that equations for a, 0, C.
and i' decouple from the equations for C2. Rs and R.- Given that we are interested in the former quantities we only report the corresponding equations:
dā(t) dt
=a (1 (4))((() axp(v (1) 1/"
2/ (1,s))
(1,5) ds (1) [1(1 h(C (L)
(G.56)
! (1,()s)) (1, ( (1, ard (I)'(( (1, (1),,
S. C (t),)))((())) [C (t, R) (t. 8) + N( (f,, ()) Rame (t. s)] ds,
dī(t)
nM (L)T(1) =())um (1) axp(1 (1))(() + (u""(""
()))((()))
(G.57)
dt
ax' (u (1))am (1) / M (1)) (1, ds - 1-" [i/ (1, - M MZ00 (1, Cmail (s)
- [ M 8) m0 (8) Mo0 (t, (((()) ds,
i(t) a' (u (t))ū(t)a mL (t) + asp (ا-0 (t)) (t)ū(t)amm(( (())))(()))) (t)
(G.58)
(v" (t))e (t)ā(t) ag¹ (v (t)) (t)) J (t.
M (M (t, M (0)
R.o (t, s) (t, s)] ds
M (1, m (0) (1, (),(,,)) s)) ds
[M (1) (1, + MC. (t, Rt. (1, 8) ds,
(t,(t,t)
" (1).((11)) B())CTTT (1,1') + ou3" (v (1))6(1)um))) (1))am))))
(G.59)
01
+ tap'l (t) (u (1))))((())))))))
- (1))"")) (1)" /' (1,
-I6' [M (t, (f, + M (t, 8) 3) (t', 5) - 20
(t, s)
(t's) ds
58

---

L/ M (t + s)) (t',s) s)] ds
st
L. [M (1, RYT (1,s) C R5T (1) do.
Here, we used the following auxiliary functions
Il (t, a (t)am)
(t)ami(s) (s) [t' (C (t, s)) Fd (t. + CC. (t, R (t, s)]
(G.60)
(t, (t, [r² + (1) am (t)p((""" (t)) a am' (s) (s)) ammh (t)a" (s)h(C.
(t,
(G.61)
M (t. s) = aa (t)aat))s (1)o(t - s) + C (t, (C (t, R (t.
(G.62)
12.0 (1,8) a<aa' (1))2 (1,8))6(t1-) -8))
(G.63)
M (t, 22a (t)a(t))'(CC) (t,t)) + (ath (C) (t.)(tt o(t- s) s)
(G.64)
- an tr (t)a (s) (s) (t,s)h" (C (t, 5))
(G.65)
+ aa mil (t)a m (s) (s) (t,s)h (Cam (t, B st,
(G.66)
M (t aa (t)a oif (s) (t, (t, (t,s)) (t,
(G.67)
MCD0 (1,8) oxm (4)a (5) (1, ()"((C
(G.68)
Note that Eqs. G.56 to (G.59) are a set of four integral-differentia equations for the four func- tions ā(t), E(t), t(t), (t,t, ('). The original SymmDMFT equations involve three other functions: C)(t,t), t"), [t, t')? We also remark that: (i) These equations are linear in the unknowns ä(t), B(t), E(t), Co t'); (ii) They can be integrated numerically with the same strategy used to integrate the SymmDMFTequatii
In Fig. 26wwe we plot the deviations from the mean field limit m(a(t) aml (t) and m(v(t) - (t)) as function of time f, as obtained by solving the SymmDMFT equations for several values of m. We also plot the predicted limits ä()), i()), which are obtained by integrating Eqs. (G.56) to (G.59
As m gets large, the finite-m curves appear to converge to the predictions ä(l), o()).
In Figure 27 we plot the result of integrating Eqs. (G.56) to (G.59 over wider time window. We observe that ö, â, i and Ct,t, t) diverge linearly with t.
This suggests the following asymptotics for these corrections
a(t)
a(t) Ux
lim (-300 t a.:
lim 11000
(G.69)
t
(tt)
Ct,t t)
lim
t 1/+
lim
(G.70)
The values of the constant a., V., v. and c. can be obtained analytically by using the above ansatz in Eqs. (G.56) G.59). We obtain that they solve the following linear equations
0 = (1))), + @(1)a.
(G.71)
0 = ((()), + +22(1)a
(G.72)
0 = -'((1, - 411 - (1)((1))) - a((((1))) a. + 200(1))²"1),,
(G.73)
1
0 2 vic- 20,01 + 401a7²
(G.74)
where
U1 == lim (v(t) 1)t, (+00
(G.75)
1/1 := lim (t)t.
(G.76)
The asymptotic linear behavior predicted by Eqs. G.69..
G.70 with the coefficients determined by Eqs. (G.77) (G.7I) is plotted in Fig. 27 We observe good agreement with the numerical integration
of Eqs. (G.66) to G.59).
59

---

10000
a(t)
D(()
1000
(t..t)
7(1)
100
1
10
100
1000
Figure 27: Finite width corrections to dynamical observables under mean field initialization. The 1/m corrections v(f), a(t), (t, t) and Ett function of time extracted from the numerical integration of the corresponding equations. The dashed lines are the asymptotic predictions for xo which show that the divergence of all quantities is linear with time. Here, ((z) = l(()) (9/10)= + = with r = 0.6 and a = 0.3.
11.8
0.7
D.4
0.5
0.4
0.2
0.2
10-1 10-2
10
10 102 0.001 0.01
0.1
100
1/mm
1/m0
Figure 28: econd layer weights and projection of the first layer weigths onto the latent structure of the data for gradient flow under mean field initialization on timescales of order m.
Left: rescaled second layer weights a(t)/m as a function of the rescaled time t/m. The arrow on the right points at the threshold Yo (a, ;p, r) for interpolation under gradient flow, see Section F.2.3 Right: projection of the first layer weights on the latent space in the single index model function of rescaled time t/ t/m. Here, ((2)= = h22) = (9/10) + 2 with T = 0.6 and a = 0.3. v = 1/7 in
F.57.
60

---

2
10000
m 2
m =
1000
m 27
m 24
2
100
m 211
ML 213
214
0.1
0.01
0.1
a/
Figure 29: Parametric plot of the rescaled projection onto the latent direction uy m against rescaled second layer weights ? a/ Vm. Same data as in Fig. 28 Dashed line is v. 1/7.
0.8
0.8
0.6
0.5
0.4
0.4
0.2
0.1
0.1
01001
01.
Figure 30: Train and test error of gradient flow under mean field initialization, for increasing values of m. Left: train error as a function of rescaled weights a(t) /mm Dashed line is the Bayes error 72 //. Curves are traversed in time from top to bottom. Right: test error versus train error. Curves traversed in time from right to left. Here ($=) h(2) (9/10)= 23 /6 with r 0.6 and a 0.3.
0.35
0.35
0.25
111
0.25
80
111
80
11
80
0.15
0.15
0.05
0.05
Lagy
a/y/m
Figure 31: The difference between test and train error for the single index data. Left panel: the difference between test and train plotted as function of /m/m and compared to what is obtained from a model with fixed second layer weights initialized with Lazy scaling. Right panel: the difference between and train error timescales of order m. Here, ((=) h(z) (9/10)s 23 /6 with 0.6 and a = 0.3.
61

---

The above analysis implies that (considering to be definite second layer weights, and projection of first layer weigths onto the latent direction), for m 1, > 1.
(tt) = am1 (t) + a,t + o(t) + (ttt,m). (t) u (t) + m
(G.77)
m
(u,t (tt) + - (t,
(G.78)
m
where mimm- An/v(t (t, m) O. If we neglect the error terms, and assume that this expression holds for larger than O(1) in m, then it indicates that a(t), v(t) differ significantly from the mean field prediction when t/m becomes of order one. We expect therefore a third dynamical regime for t = ((m), which will be the object of the next section.
G.2.3 Second dynamical regime: (mm) and beyond
As pointed out at the end of the previous section, we expect a third dynamical regime when = (m). By this time, the stability calculation in the previous section indicates that second layer weights become of order /m. Figure 28 confirms this, and shows that, in the same regime v(t) becomes small. fact, numerical solution of the SymmDMFT equations are consistent with a(t) = O /m). utt) = ((1 and a(t)u(t)= = 1 for t = (mm).
For small constant denote by to(m;; e) the time at which a(to(m;c)) c/m. We then expect that the following exists
lim a(la(m;;) + 0 w(()) =
(G.79)
vm
lim cffo(m;;) (m(m)) xt)) m=+20
(G.80)
provided w(m) is suitable function (with (m)) (tt((m; c))). The stability analysis in the previous section suggests that to (m; c) S ft, (c)m + o(m). Our numerical solutions do not cover a large enough range of values of m to verify this ansatz, and determine the scaling of w(m) with m. On the other hand, they indicate that indeed (o(m; c) = ((m).
Since the second layer weights become of order vm in this dynamical regime, train and test error start to differ significantly. We expect
mff (0), lim (tr(to(m; c) + 0 ((m)) = etr m=00
(G.81)
lim (rtooo(m c) + 0 (m)) = css
mf3 (0)
(G.82)
This picture is confirmed by Fig. 30 which reports train and test error as predicted by numerical solutions of the SymmDMFT equations for increasing values of m. On the left, we plot the train error as a function of the rescaled second layer weights y = a/ /m. We observe that curves for different values of m decrease until they reach the Bayes error 22, On this phase however different curves do not collapse corresponding to the fact that 7 vanishes. In the second phase, y grows to be of order one and correspondingly the train error decreases below the Bayes error: this is the third dynamical regime. Overfitting takes place at this point.
In the right frame of Fig. 30 we plot test error versus train error. We observe, again, the two phases emerging for large m. In the first phase train error and test error are closely matched. In the second phase, train error decreases and test error correspondingly increases. Again, this takes place when t= (mm).
Finally, in Fig. 31 we repeat similar plots for the generalization error (difference between test and train error).
When l/m is large, the train error vanishes. We observe from Figure 28 left frame that, as - 0o, rescaled second layer weights reach finite limit that is close to the interpolation threshold characterized in Section F.2.3 Namely
lim + (0) Tal (a, sp,
(G.83)
'We note that solving the SymmDMFT equations accurately enough to capture these corrections requires either to very fine discretization, higher-order integration method.
62

---

1000
1000
100
100
215
706
40
1010
0.4
0.6
0.1
2.18
1000
1.17
1.16
1.15
2.18
1.11
10
1.09
1.08
10
100 1000 0.1
Figure 32: The interpolation transition for pure noise data and network with second layer weights that do not evolve with time, fixed at a = 1, see Section [H] The noise level fixed to t = and we considered h(=)= = (3/10)= + =2/2. Top left panel: relaxation time ((rate for convergence to vanishing error) for different values of m. Top right panel: logarithmic plot of the relaxation time. The value of the algorithmic threshold for different values of is fitting parameter. Bottom left panel: values of the algorithmic thresholds as a function m. Bottom right panel: the relaxation time extracted from the scaling limit of the SymmDMFT equations in the m - limit. The algorithmic threshold is this case Tge (DO) = 1.18 which fits well the behavior plotted in the left bottom plot.
H Dynamics under mean field initialization for n/d a fixed
H.1 Interpolation threshold at fixed a(t) =
In this section, we consider alternative scaling in the large width limit. As before, we use the SymmDMFT equations, and therefore study the limit n, + oc with n/d + a. In the previous sections studied the large width limit
with a/m fixed. In that setting interpolation is only possible when the network complexity scales, i.c. second-layer -lyer weights are a = @(ym)
Here instead we keep a(t) = and do not let evolve second-layer weights with GF. We consider pure noise data, and show that interpolation takes place if a [ar(m), while the train error remains bounded away from zero for a > Oka (m). As expected from Gaussian complexity considerations, the threshold Oar (m)) has finite limit bounded cannot interpolate pure noise data.
co. In particular, for any 0, network with
As thorough in Sec we fix a and integrate numerically the SymmDMFT equutions for finite but increasing values of m. We fix the initialization scale dg and the noise level and change only
We observe that for small enough the train error decreases exponentially fast zero. Namely, recalling that e (t: a) := lim,, L, (a(t), (t)), we have that
< <m) + ctr(t: a)) expf-t/te (a, m) + o(t)}
(H.1)
However, the relaxation time time the (a, m) increases as a t (er(()). Concretely, we define ba, m, c) as the infimum time such that (t; a) S c, where e some small constant. In practice,
we set c 10 The results are plotted as function of a for several values of m in Fig. 32 top left. plot.
63

---

10
50
2000
10000
Bt 1000
71
0.01
0.4
0.3
0.2
0.001
50
02)
1/o0
4/60
Figure 33: heck of the convergence of the numerical solution of the SymmDMFT for ā fixed to the scaling solution for m + oc. The left panel shows the behavior of the train error while the right panel shows the behavior of the correlation Ca(t, 0). Both panels refer to model where the teacher pure noise with hx2) = (3/10)z + 2² /2.
and the student is made of of neurons whose covariance structure is given by
For each value of m the relaxation time appears to diverge at the critical point @ar(m) as an inverse power of (ar(() a, namely:
L(m, c) = (āt CE, m. c) (Tarr(m) ब) (1 o(1))
a t (m)
(1))
(H.2)
The exponent v appears to be independent of m. We fit this form to our data and extract the interpolation thresholds (m)) In Fig. 32 top right, we plot La, m, c)/m function of the gap to this threshold. This plot confirms the form (H.2). with exponent v R 2. Also, the fact that different curves superimpose indicate that L(m, c) (c)m.
The estimated interpolation thresholds (m) plotted function of m in the bottom left of Fig. 32
These data are consistent with the existence of finite limit
Oraa (Dc) lim Clez (m).
(H.3)
and numerically (or((c)) 1.18.
In the next subsection, we derive equations describing the m + 0o limit for a = 0(1), a 0(1)
fixed. Studying these equations yields further support to Eq. (H.))
H.2 Infinite width limit at fixed at
In order to study the limit when xc. As we have seen previously. the relaxation time of the train error is proportional to m. This clearly visible in Fig. top/left. This suggests that for m + oc, dynamics takes place on timescales of order m. Therefore we propose the following asymptotic ansatz
oo fixed a. we discuss the limit of the SymmDMFT equations
(tm, sm) = (t, + om (1),
(ommm = Ct 3) om (1),
(H.4)
Ra(tm, sm) = (t, + (m,, mv(tm) (t) om (1).
Ca(tm, sm) CT 8) on (1),
(H.5)
(H.6)
which defines set of functions, Reg Ög feg. č and Da We now describe the equations that these scaling functions satisfy satisfy. First we define Č% and R the solution of
8(t- - 1² = 1/ [0( (((t-s)
+)+ ()
ER((,)) s)] (x,
(H.7)
0 / (t(t 8) +ni(t, 8)
( () ss + 1 dt (t ds,
64

---

where
En(t,s) (' +
(H.8)
( (t,
Then we define the limit memory kernels:
( 8)
T (t,s) = aC*att, s) (t,
(H.9)
/ (t, a [[Ca(t,))) shh" (0% R (t, 8) + (t (C. (t,)s) NE = (t)
(t, (t,
Substituting the above ansatz the SymmDMFT equations and matching the leading order terms, we get the following equations that determine Rł. ČT Rg Öo and ia:
S/' (t.))( + M (t.s) s)] ds
(t, = -- (t)C
- -fo"
+ M (t. (R .)] ds.
(H.10)
O,Cg(1,)) - S [M (1, + M/h (1) (,((',
- 1/' [ R s)] M
(H.11)
(t,)) -7 (t) 8(t-)) - do (t, (s,)),
(H.12)
R, = (T) (t. - 1:' (t. NT (t, (s,
+M (t, (s,
(s, t') ds.
(H.13)
05 (1) = [ ds M/ (1, Ca(, + 1/ (t, O) 8)] ds
L/ [M (t, st.s) + M (t. 8) R. (t, $) ds.
(H.14)
These are to be solved with boundary condition
Cg (0) 0, (0, 1,
(0, (0, = 0, R -0)0+ 1.
(H.15)
(H.16)
The scaling behavior the train error is then given by
lim etr(() _ 2 =: (T(
(H.
=-
In order to test the accuracy of the asymptotic analysis developed in this sections, we solved numerically the SymmDMFT equations for increasing values m and compare the results the numerical integration of Eqs. (0.10). (H.14) presented in this section. Some results of this comparison
are presented Fig. 33 which shows good agreement between finite-m curves and m > oc limit.
The solution of Eqs. HH. (H.14) provides another route to estimate the large-m interpolation threshold (oc( at fixed at)) = 1. Namely, we solve the equations numerically and extract the
(ā, 00, c). which is defined analogously above. We then fit the divergence of tā oc, c) at (oc( according to Eq. H.2).. We obtain (x)(()))
1.18, in agreement with the threshold obtained by extrapolating the finite-m thresholds aGr(m). the bottom right plot of Fig. 32 we plot
(a, 2o, as function of (oo( a. This confirms the behavior of Eq. (H.2) with ~ 2.
We conclude by emphasizing that, throughout this section a(l) = 1 and = were fixed. If we generalize to arbitrary a(l) = ag and arbitrary 7 > O, the threshold oer (m) will of course on these quantities through the ratio ao/T.
65

---

I Details about SGD simulations
In this appendix we provide some details about the numerical simulations with stochastic gradient descent (SGD) presented in Figures
We generate data according to the pure noise model y; = E; (Fig. - 3s - + ex (Fig.
i S n. We learn the two-layer network of Eq. [.II. see below for the class definition.
class Net (nn Module):
def --init_- (self, a, m, d)
super ()__.iitt__(
self.m
self.lini an Liner m, biasFaaaa)F
self.linl.wer wea...eight.dee data = (1/np. sqrt (d)) (d)))tttr) *torch randn ((m, ,))
self.limi nn. Linear (m,1, bias=False)
self.lin2 weight data [0]]]
self Myact()
self ()
def fsrlard x ):
x1 self act act (self. lx)) return self.lin2 (x1)/ /self.m def project(self... epsilon):
row mow_norms torch.nore (self lim1 weight data, dim=1, keepdim= True)
row_norms torch. clamp (rom_norms, min=epsilon)
self lini weight data self limi weight data/
As shown in this code, the initialization
(no, W o) PB (a, Wo):
(1.1)
ag (ag ............ ao): (Wo.s) N(O, 1/d).
(1.2)
where P8 projects first layer weights to the unit ball:
Ps a. (wi, w.,) ((a())
w1
Wm
(1.3)
We use the standard SGD iteration without weight decay and constant stepsize 17, and batch size
0++1 = 0 1/ (O))))
Es(
Σ (y f(x;;
(I.4)
2
iES
0 (e(0++1)
(I.5)
The optimizer is defined in the code below
optimizer optim. SCD (net. parameters 1r=lr, momentum=O.,
veight_decay=0. lambda_step = lambda epoch:
scheduler torch optim 1r_ tt__cheduler LambdaLR (optinzzer,, 1r 1r_lambda= lambda_step)
In the simulations of Figures 2 and we use batch size = 100 and step size = 0.1. Each symbol
reports the average of Nsim 10 simulations.
J Lower bounding the overfitting timescale
Throughout this appendix we use denote the rescaled time f introduced in Section
J.1 Proof of Theorem 3.1
By computing the derivative 0, (a(t). W()), we get
d Iao(1)]
(ys ((x; a(), (1)))(w((())))))
66

---

n
$ 22 (a(() W(I)
o ( x2)2 a.)
1=
< 4LV 28, (a(0), W(0))
+ (w;(1)=x;)3)
72
< 100 2. (a))) W(O),,
where, for n 2 d, the last inequality holds with probability at least 1 - -exp(-ccr (for some universal 0) by standard upper bounds on the norm of random matrices [53] Further
V (a(),,(()), (()) y
a: X Xw,)
|| + ao ps
max
(Xw:(()))]
rlgg| + | XU) + 0g max
|o (x,(())]
2 | + L XUI| + (0) + o X l.p + (o)))
where (a) it is understood that is applied entrywise to Xw; E N(O, I,), and yp is applied row-wise to XU Ra By using standard concentration on the norm of random matrices, also with probability - exp( exp(-en), we have (for I n))
R and in (6) we have g ~
V 2 (a()), (()) C(T + v + LoL).
Summarizing the above bounds, we have
d ax((t] s a1,
dt
which implies the first claim by integration.
To prove the second claim, we consider the following sets of parameters a (a) C (m) (which will also prove useful in the next section)
|alli
(m) (a. W) E R mxdd
< a, |m: ||2 Vi < m
(J.1)
m
W (a) = {(a W) Rm pad
||a S a, |15 1 Vi < m
(J.2)
The second claim follows in turn if we prove that there exists universal constant C such that
(a, W) (a, w) S C(L C(L' d
sup (a. WWW.W
(J.3)
This is standard estimate, that we reproduce for the readers' convenience.
We begin by bounding the expectation of the supremum by symmetrization and contraction inequali- ties. Letting (Ei)ssn aid Unif((+1.-11) -13),, we have
1 T2
E
sup (a.W)
Ln a. W) (a, W)
SE E
sup (a.W)eWm.dde) n
Ec f(z;; a, W))
1
< 2E
sup
Enrf(x W) +E
sup (a. W...)
Eaf W)
Wa.W).W... ((()
n
1=1
=: 2E
E
67

---

We begin by bounding Ej:
m aj 1
=E
sup (e.
Eayoo (w)
m71 n
j=1
Sa E sup
Litio x;))
ax:)
|| =1 71
(a)
S a LI
sup
Σ (1 + lw:1 1200 xx
|| 7
c1 + 1o 1az }
Sa LI
1=1
<Ca <Ca L(L+(L+)V n
where in (a) we applied the contraction incquality of (36 to the function (t) y:o(t/(lbsl + 1)). We next bound term F.z:
Fs =
sup (a,W)
m
1,1=1
n
S a E
sup
(wo as a x;)
80, The 1d-1
n2
CL. a E
Σ Law T x
sup
WEB
< a
where inequality (b) follows by applying the contraction inequality of (36 to = which is C'p-Liischitz because (Lips ||0 S L. Summarizing, we proved that
E sup (a.W)eW...de
(a, - Iaa w/ s
+
/
(J.4)
In order to complete the proof of Eq. J.3 we will show that the supremum concentrates around its expectation. For fixed (a, W) E Wm,((a).
we have
[(x; a, W) - f(=x: a, W) < La æ - x 12
(e(UTa)) az) + (UTx') < T.I x 12ll2-
We write (x W) to emphasize the dependence of the risk on X Letting r(m;a, W) = Pp x) - fx; a, W), we have
(X;a, W)
(e. + r(r; Vn, r(a;; W),
n
C
V (X: W)
s
(|e + La)La.
n
Hence
V x 3 (X; a, w)l < C
L La
||l|| vil
Vn
C'
La(Lā+
Vn
68

---

where the last inequality holds on an event that has probability at least 1-e"". Defining Z (ā) ==
(a, W.la.. a W) ((a W)), Borell inequality yields
IP Z n.m. (T) EZn.d ( Bt) S 2 2e-ni= t
B C" La +))
Together with Eq. J.4). we thus obtain that the following holds with probability 1- 1-2e-t_ E
(a, W B(a s C(L2a + + C(L +
E sup
(a. (a.W)eW...ll
This yields the desired claim.
J.2 Proof Theorem 3.2
We introduce the notations:
- (a, W) :=
m
V w, P,
Rw W) V Laa.
(J.5)
01
2h.e W)
(a, (a. W) Vo,
(J.6)
We begin by establishing a uniform convergence lemma.
Lemma J.1. Under the data distribution of Section A assume || S L and the activation activattio,function to be bounded differentiable with Lipschitz continuous first derivative || I|o'lloos Ilo' S L. Then there exists a universal constant C1, and a constant eg > 0 dependent on L, T, o such that, with probability at least exp((-nco).
max (a, W) < C(T V d
sup (a, WiEW
log(ne/d),
(J.7)
&mm
sup (a. WwW.W (T)
max
In W) C SS
(J.8)
n
Proof. Gradient with respect to wy. By a concentration argument, it is sufficient to consider the expected supremum. Writing the formula for V V., Hn and using standard symmetrization argument, we get
E sup (a. (a.W)eW...dtn)
19 w)ll (a, (a.
sup
(u, Mh W))
1
< 2 Esup w. 17
Eigoo' (wTTa, xx a. + 2 sup
Σ w. w.w.aa
Elo (wTT an)o x. x,
=: Bi + B2.
where the &, are i.i.d. Radamacher random variables and in the last two lines is understood that the supremum is over ||w]], [
|||||| < 1. Consider the second term in the last expression. Defining
7(xx) =
= x + M(12> 1se M). and 77x)) = I - (x)), we have
B₂ = E
sup
fo (w) 2.) (w)) xx Tux x;
w. (I)
n
< 2a
sup
Eoo Gio'(w) T x, 22) (w) x. 2,)n(u TT)))
w. w.E 1
+ 2
sup
0 (T.towooooo w m xx))
w. w.B.ueB
17
1=1
=: B2,1 + BB
69

---

Further defining @tt1, t2. (s) = (f1) (t2)m(ts))) (which is M-Lisshi for M 2 1), we have
B2,1 = a E
sup
Σ (wo(") T Di, w an,
Bs; u x.))
(I.9)
w v. u (1)
Using the contraction inequality of 36 we get
B2 S Ma
sup (LB
Σ
2s + E sup
Σ 1=1
23 + E sup
Σ
E D3
weB4(1)
MEB4(1) t=1
1=1
S CL² Māy
Next consider B2.2:
1 77 x;)l
22 S 2 sup
HEB
< 20L2 sup MEB((I)
ET T x.)) + 2
sup
Σ c 7 T x;)
WEB n 1=1
S CCL M2 CL
n
where the last inequality holds because w, is Gaussian with variance ||u]|² 2 and using again the contraction inequality. Collecting various terms and optimizing over M 2 1, we obtain
d +e be-N2/44
B2 s CCCC M
S sCL'av log(n/d).
n
The proof of Eq. (.7 is completed by bounding B₁ along the same lines.
Gradient with respect to ag. Writing Va, En and using symmetrization, we get
E
sup
Un. W)|
sup (a. (e.W)eWm..dm
ea, W)
(a, (W..)eW..eft
n
m
2E sup W.u 71
Ey +2 sup a.W.u
aj
(o m
5=1
=: D1 + D2.
Consider term D2, and define the L Lipsshhtt function (tt,, t2) == o
1
D2 < 2a E sup max
S xz
W. u Jm
< 2
sup w w.WEBd(1)
Σ
( TouT 2ú TTx)))
d
< CT. CI. a
Term D, is controlled analogously, yielding the proof of Eq. (.8).
We next prove some continuity properties of the population risk B. is useful recall the form:
a(a, W) //4 ( + ||,11 1
(U (.) 2m
aa,h(ww...)
(J.10)
70

---

Lemma J.2. Under the data distribution of Section A assume 11o/1lo S L that and a are bounded differentiable with Lipschitz continuous first derivative, ||o I|o' loo I L. l|sell zo; V oc I. p I I, T. 2 1. Then, there exists absolute constant C such that for any (a, W), (a, W) (a):
adl m
Vw W) w. (a, WIll S CLL
(1 a
max L0, wyll 55m
(I.11)
CL2
(a W) 0n, Baa W)
(1
max 55m
Wy w/l
(J.12)
m
and
CL²
(1 + a) - ael + CL22 ag
V.. R0 W) V v R(a, W) v
all
(I.13)
m
m
CL3
0o (ā W) 0n 2(a W)
v m
ā all
(J.14)
Proof. As preliminary remark, the assumptions on ye, a imply similar smoothness properties of P, h. In particular, recall that h(q) (GG) for (G1, Gg) jointly Gaussian, cen- tered with unit variance and covariance E[G1, Ggl 1, whence its k-th derivative is h (* E(a(A) (Gr (8) (G,)] (whenever ((k) Therefore, the assumptions imply that || 1/l ||k Ipppp L2, Similarly, || V up S CLS.
Proof of Eq. I.III. Differentiating Eq. (.10)
m V Vw, la, W) = UU wi) +
a h' W(w]ww)). wy)wj
(J.15)
3=1
Therefore
m
Raa W) Vm w)|l VV((TTm)) Vē(UT w;)1
lael
|راl
|| (w) )ū h' [w wz w
m
j=1
<CL² w well + ã max
h1 N' i ) - h' [w] )
5mm
Further, by the above smoothness properties of h,
|N ( it; - h (w) wy ) < CT21( - + well
Substituting above, this yields the claim J.II. Proof of Eq. (J.12.. We proceed analogously to the previous point. Namely
m
mdn,a(a. W) = -U (()) +
aj
[u w
(J.16)
m
1=1
whence
m H(a W) - 02, Hl W) S ( wx) - -(TT@wT) we)l
Σ
/ T w.;) - -hww wy)
+
3=1
5 CCL w. well + CCLL
wrll + || - -w;ll).
which implies immediately Eq. (J.12
Proof of Eq. J.I3). Recalling Eq.
0.15). we have
1
m V w. 2 ( W) - V., laa w/ < V will le ael +
|| NowT)w || Tāāā= d;a,)
71

---

SCL ael CL2 (12
l all1 + CLaa ār ael
which proves the desired claim.
Proof of Eq. J.13). Recalling Eq. J.16p. have
m R(a W) da la, W) -
Σ
h(w,
w,)1 (1,))1-100 lāg - azl
m
j=1
CL
ā
a
m
Using the last lemma and triangle inequality we get the following.
Corollary J.3. Under the assumptions of Lemma J.2 there exists absolute constant C such that,
for all (a, W), (a, W) Wm. (a))
CL (ā, W) - V w. (a, WIll v V w.
CL2
max
(1 + ④
max | - wyyl
(1 + a ā
alloo
m
35m
m
CT2²
CT22
max Pmm
Ou A (ā. W) Dap L (a a, W)
v
(1 + a max تx,
wwil +
||ā allxc.
55m
We next consider at),, (t) that follows GF with respect to the empirical risk, as per Eq. A.10). which we rewrite as
a(t) mv Gn (a)), W(t)),
(I.17)
(t) = -mP.. 1 m. V.,, L (a(t), (()) vi= 1......,
and denote by ag (t), W. Wo(t) the GF with respect to population risk:
(C) =
-m 77 a R an (1) W (o)(),
(J.18)
(t) = mP w. V., A (as (t), Wo(t) =i= 1......, m.
Lemma J.4. Under the data distribution of Section A there exists constant = co)),, cu = (6)
depending uniquely on 8 > and an absolute constant C such that the following holds. Assume p,, a to be bounded, differentiable with Lipschitz continuous first derivative ||y]|oo. 1/p'l " [ <
L. ||o | 1|o' Ilui S L, Further assume n n/d 2 exp(cgL2). L 2 L. Let (a(t). W (t)).
(t)()). Wo Wo(()), be defined as above, with W W())= Wo(O) and a(0) (0) such that [0)]] |ao(0))l S ag. Define
T.(m,) inf {t: (Ila(t)looc v [lau(t)llo) 2 (c.L-* log mn) Vog (GLL-200 log ne
1/3
d
(J.19)
Then
(
1/2-5
sup (mmc)
A(t) & C(L2 +(L
Att == max (t)(t) - w2((t)l + ||ā(t) a(t) 25m
(J.20)
Proof. We will prove that the desired bound holds on the bigh-probability event of Lemma J.1 where by we
L-2 lo ne/ m//3 Throughout the proof, we use co, Ct. C' to denote constants that might change from line to line, with dependence on the parameters of the problem as per the statement of the lemma.
We start by noting that, letting v, -m L, W) and Vo, -nn/d)V wo. R (ao, W. U. and
P = I - -ww the projector orthogonal to w.
P W.. U.
PL.. Lo.. vo.1ll < P (v. vo,i [[(P.
P W vook
72

---

< lle vo + ammaa - wu, wo llpplllo// Po,ill
< lle wo + 2/|2; - wo pl..pl
Hence, comparing the evolution of (t) and wo,r(t), we get
d
dt (1,(t) wo. S ml V (a(l)) W()) V ((o(()) Wo(O)]
+m| V w. 8 (t)(t), W (()))))l -|,))))1-111,0)))) -0 w.,(t)
D; + D2 ll, w)))(())l
Since we are working on the event of Lemma 1.1 and using Corollary
J.3
we get, for 1 S T. (m,
D1 S m V (a(t))) (t)) Vu. R(a(t), (t))l
m V.. w. (aa(t), (()) w. A0 (t)()) W Wo(!))|
S C(L'T 2) / log(ne/d) CL a²)
max 75m
w (t) wo,
CT 1 a)) ao (1)l
Further
D2 = 1a: U w;) 1
ayh (w) wj)wi
S Ca( L22 - aL')
Collecting all the terms, and using 2 1, we get
dt d [w:)(() (1)l sCa(L'a + ,yy + log(ne/d) +1) (I.21)
We next consider the evolution of second-layer weights:
d
dt (+((t) (t)ll S ml mal (a(t), (()) (t), Wo(t))l
S ml Ra En (a(t). ((t) (a((), (())
+ mm| Lt (a(t), W(t)) 0n, 20 (an Wo(())]
<C(T,a+ + x2)
C + a)
max |m,(1) wo, CT.21Ia(() a0( m5m
S sC(L'++T))
+ (1+))+)))(
n
Using the last bound together with Eq. (J.21) we get
d A(t) S Ca(L'a+ 7) W d
lo CL
n
whence the claim follows by Gromwall inequality for sufficiently small c₁.
We finally need a lemma from approximating GF in the population risk by the mean field dynamics.
Lemma J.5 (Corollary 1 and Proposition 3 10 Let at((). Wo(t) be GF with respect to the population risk (118)) with initialization (ag.. S ao and (wo.; (0) Dimm ~ Unif(Sd-1)) Recall
that at (f), mmll (t) is the solution of the ODEs 3
4
with initialization (o(((() (0) (0), wnth (t) O.
Under the assumptions of Theorem 3.2. for any & O there exists constants Co: e₁ depending uniquely
on L, and an absolute constant C such that letting To (m) ((co/s) logm) m)213 the following
happens with probability at least - 2 2exp(-c1dd
77 (1as(0) a. (1)) + [lx;(() (0]1 Cm {
(2
1
sup (mT
Val
(J.22)
sup (m))
B(a(t), (t)) - ets(t) s cCm²
1
val
1
(J.23)
v
73

---

Proof of Theorem 3.2. Throughout the proof L, are assumed to be fixed, and constants C, co....
depend on them and can change from line to line. We will further work on the high probability events of Theorem 3.1, Lemma J.4, and Lemma J.5 By Theorem 3.1, for all f. S (m)) we have ||a(()]]>> S c2ol 2m) 1/3 (where the constant C2 can be made sufficiently small, by eventually reducing c1). An analogous of of Theorem 3.1 for the population risk implies [t) I C2 (2og 2m)' as well for all I T (m)) Hence we can apply Lemma J.4 and Lemma 1.5 which yields the claim.
K
Dynamical mean field theory for non-Gaussian model
The DMFT equations for GF in the original non-Gaussian model can be derived from the general theory of [3
Given a (positive semi-definite) kernel R20 Rz0 Rmxm (t,z) + Q(t, s), we write
(P, Q) if is a centered Gaussian process with values in R" and covariance ([=(()=(s)]] (t..s).
The DMFT equations can be interpreted as set of fixed point equations for the functions Cij, Ruj. Ui.
We define the deterministic processes at)), (t) and stochastic processes w= (t) = (vo (1) i < m). rt)) (r) m), the solution of
(t,(t) m (t) ((()))((()))())))))))
E dt
(K.1)
(K.2)
m
duf(1
(t)
"Σ/ 1=1
1 m
Σ
L
Mt,(t, (s) ds
(K.3)
dt
Mis *)u + mt),,
7~ GP(O, C²),
r:(1) = Σ 1=1
Rall, (a(l, +(())
E E~ GPCO (K.4)
m
E(t) = y
a((t))(r((t)).
(K.5)
Here, in the first equation, (OO.0 & N(O. Ix) are independent of 77. In the second equation, y = (rro) + with (To. ~(0, @ N(O, 72) independent of &.
(0.,)~~N(O...0)) N(0.
@n(()) (5)
(ty(t, 8) - (1) 8(t - 8) + æ
(a)t)
(K.6)
Mg((++) = -- at
m B Sa( Or((t)
E o (m:
(K.7)
m
Dro."
(t, s) = (t(t)a;(s)
E ((s)
(K.8)
m²
Sty(t) = - (t) E(t)o" + a(t)a((t) (100))
(K.9)
m
and
Ct,(t,)) = E (s))
(K.10)
(t, = E dwl (t)
(K.
Ong(()
In solving the above, the random functions 2ne) (t) of the following linear ODEs:
Byy (5) and dr (8
(for / > 8) are defined to be solutions
d (w) dt (s) On,
(t) Ow, (t)
m
{ Mt,(t,t))
(t) dt, (s)
m Σ/
(K.
1=1
74

---

0r,(1)
Org Ra Ra(l,
(5, + Eggō(1' -s) de'
(K.13)
(5)(8)
1,==1
(r,(t)
Ra (()()))))()) (n(l))))
1
Σ
(I Stg((())
(rg(5) ds (K.14)
Oroj
m
m
Oro.j
1-=1
with boundary condition
Ow (t)
dij for the first equation.
(t)
L Derivation of the dynamical mean field theory equations
The study of the dynamics in such high-dimensional limit can be done via dynamical mean field theory (DMFT) [18]. The theoretical technology that we will employ is an evolution of the one first derived in 31 32 to study gradient flow and stochastic gradient descent
models that very much related to the Gaussian process we are discussing here [11 42 33]. We remark that the formalism considered here be used study both the single index model and the pure noise case. To obtain the pure noise model, one can set h2 = & = O. Furthermore, the extension to multi-index models can be also done easily the same lines.
The analysis of Eqs. (A.10) be done by recasting them into path integral representation. We follow the same procedure presented in [31]. Eqs.(A.10) can be packed into a dynamical partition function
7dyn / DaDā f DW DW exp Ala, ā, W. W7]
(L..1)
where the path measure Da(t) Da(t) is implicitly defined. The action A reads
ddall)) dt
0. (t)
(wn(t)), d r01(1) dt
8Rh
A
(t)
+
dt +
+ dv;(t)w;(())
dt
ow (1)
(L.2)
Eq. L.2 can be rewritten by introducing Grassmann variables 62
Call (tu, 0.) a supertime
coordinate, with 0g a Grassmann variable. Define, with a slight abuse of notation
(a) = wr((a) ) + 00,00 or (ā) ataa + (ta) 1 15 m.
(L.3)
Eq.
I..2
can be written
4
Kiy (v + Σ +J==1 a.b
/
Riylä, (a,(a)))) (I(a))
2
0.h
(L.4)
The first terms of the sum describe the kinetic terms of the dynamical equations motion. The last term instead contains interaction between the weights of the network. The empirical risk S, depends the training dataset. We are interested in understanding the behavior of the dynamics of gradient flow when we average over its realizations. Since the dynamical partition function is identically one can average it directly over the dataset? In this way we have
/ Da( (a) exp
d
Zdyn
Σ
() ur) wr)())
a.b
(L.5)
d m
Σ
Riv (á, ()ar(a)ar(()) (i)
E
exp
2 Jab
1'-1
We emphasize anyway that the average over the dataset is not mandatory: the resulting DMFT equations are self-averaging.
75

---

Performing standard manipulation, see [3]] the dynamical partition function, for d + oc, can be written
Zdyn / D(a, Q, exp [ Q,
(L.6)
The dynamical action Sayn is given by
Sayn
d m
Σ
6) b) (( Eku (a))
/ KN (a,
d In det(() In detet(I + E+)
ad
2
11=1
d
ab
(L.7)
where o = n/d and
[+ = ([],,))=+2
hh(1) +
a (a)ar (b)h (Qw m()))
m
1,1
(L.8)
1 Σ a(())((((a) 1=1
Σ ((()))((())))
The kinetic kernels K. and K are implicitly defined in such a way that they reproduce the time derivative part of the dynamical equations (A.10).
In the large d limit, fixing m and ãr, the path integral in Eq. L.5 concentrates on saddle point. The corresponding equations are
0 Σ
(a, ()) al())) (
(I+ (I+
(L.9)
(I + (1, (4000 sw (à,
and
Kr, (0) 0 (a(()))()))))) (1+ (
(L.10)
[I+ + Σ) -
m
7=1
where
Qu 6) = Qu (á, 8) (a)r
E(a, 0) 72 + he(11
((()aa (b)h (Qir (a,
m
(L.11)
1 (()))(((())) Σ ((())(((())))
If Lagrange multipliers are added constrain the norm of the the weights of the first layer, one should provide additional equations for them. Finally the equations for the dynamics of the second layer weights given by
Σ / 2=1
(0,00,0a000 = - / (e,a)
m Σ
a (0) - - (((((a))
m
(L.
Eqs. L.9 12 contain all the information about the dynamics. In order to fully specify the behavior of physical quantities such has the train and test error, is useful to unfold the Grassmann structure of Eqs. (L.9 12
76

---

L.1 Unfolding the Grassmann structure
Causality of the dynamies implies that the following parametrization the most general solution of the saddle point equations
To (a) = Ta (4a)
an (a) (bab
(L.
Qa a. b) = Casltai (go + + (t (I + -) (a. b) Ca(ta, to) (tRA(to: to) 0 ( to),
Plugging this parametrization into the saddle point equations we get that the correlators in Eqs. L.13 satisfy the following DMFT equations
(t, (t) dt
m S Ra(t,s)
m
(s)) ((ra ds
m
(L.14)
m { s)
((ss)
m
dr.(1) =taa(t)ra()) + m
a (t)) [ Ra(t,
dt
(L.15)
(1)
Ma (1, (s)r,
aCog(ta: to)
á aa (ta aaa (raattt) S/² Raltar
(Voltt
Hla
aa (ta)
m
MR (tas s)a, (op tss
(L.
m
7=1
an (ta)
Mc. (1a, s)a, s)ds
aR (b)
vo Rap(las 8n (b)
Hha
(L.17)
an (ta)
Ma (1
Note that we used the notation according to which the prime sign denotes the derivatives of the functions with respect to their argument. The memory kernels M and MC are defined by
R (t, 8) = a MC (1,
+ Ca(l, (C((
m
(L.
ā m
Call, (Co,(a))(C
The kernels in Eq. equations are
18) depend RA and Ca that defined in Eqs.
L. 1.13
The corresponding
[" 18( (e) -(t-
(L.
[ [8(t 8) (t) t')ds 1 6
RA =0
77

---

where
(o(1,8)==22 +h+)
m
===1
1
((t))(()())
Σ
(()))(((()))
(L.20)
m
m
En(t,s) 1 art))r (s) (s))ar(ssh(((ce) s) Ry (t,s).
The Lagrange multipliers vo (1) have to be fixed self-consistently to enforce that Ca.a (1, = 1 given that wa E Ed-11 The corresponding equations are
(rraltt)rraa f/" Ra(ta,
va(ta)= = km
aa (ta)
m
M (ta,
(L.21)
m
aa (ta)
Ma, (las (1as
s)a, Ryalla s)ds
Finally we need to add a set of equation to propagate the diagonal elements of the correlation matrix:
dCa,(ta, ta)
[dCas(ta:t') t') @Czalta: t')
lim dfa t-
(L.22)
Ōta
Ut.
These dynamical equations can be integrated from set of initial conditions that fully specify the initial status the neurons. We will consider a random initial condition for the weights of the first layer so that
To (0)
Va 1,..., m
Caft a (0, 0) 0
Va + = -1...,m m
(L.23)
Caa (0, 0)
Va 1,..., m
(0, 0) 0
Ya, = 1....
Finally, the initial conditions for the weights of the last layer aa (0) are completely arbitrary. The solution of the DMFT equations gives access to the dynamics of the train and test error. The train етror as function of time is defined as
(Ir(
lim Z.(1). d
(L.24)
simple way derive the expression of étr function of the solution of the DMFT equations in the d + oo limit is to consider deformation of Eq. (L.5) which consists replacing
cxp (-n (
(n((aa)
- exp (
(())
(L.25)
For P(a) get back the original expression. The main idea of the derivation is to P(ā) as a source field. In particular we have that
etr (t)
- /de.
f db.
5
In Zágn [P]
(L.26)
8P(A)
P=1
Note that the deformed dynamical partition function Zayn [P] does equal for generic P so that the formula above makes perfectly sense. The deformation of the partition function produces deformation of Sdyn in Eq. (L.7) which consist in replacing
ad Indet + E4) + ad
de + E.)
2
(L.27)
Σ. 0) = ((a))+(ā,6)
78

---

Performing explicitly the derivatives with respect to P one gets
etr(t) / [Ra(t,s)Ec(tss) Calt, Ex((,
(L.28)
The computation of the test error can be done in analogous way
(ts()) = lim E (Wuew) (s) Vnew
1
2
d0000
2
(L.29)
1
,² + (1) + m
Σ WP
(CCu((( - 2 Σ (m(()))
2
The average in Eq. (L.29) performed over the training set and an additional datapoint, not presented in the training set and having the same statistical structure.
In summary, the solution of the DMFT equations gives access to the train and test error dynamics in the large dimensional limit. These equations can be integrated numerically very efficiently. Our goal understand their behavior for infinite number of neurons, m oo fixed sample complexity a. We will be mostly interested in two types of questions: first, given a dataset that is pure noise, what the sample complexities which the network is able interpolate the dataset. Second: given dataset built out of a single index process what is the dynamics of the test and train error.
79


Reviews:
[{"review_id": "ITQOf3sKpC", "reviewer": "Reviewer_jCsC", "summary": "The authors study a dynamical mean field theory (DMFT) characterization of two layer networks, where a few simplifying assumptions are made to allow analysis of large input dimension $d$, number of data points $n$, hidden layer width $m$, and training time $t$. Note this type of joint limits has been very difficult to study for a long time, and any analysis of this kind is highly desirable. \n\nThe authors found several interesting observations, in particular, the fact that overfitting happens on a slower time scale than feature learning, and that training for longer leads to feature unlearning and an increase in generalization error. The paper overall uses a combination of non-rigorous calculations and rigorous proofs, where the non-rigorous aspects include assuming the residuals forms a Gaussian process, several ansatz were used to analyze the DMFT equations, and some perturbative analysis. However, the authors did a good amount of simulations to justify quite a few of these non-rigorous approaches.", "questions": "Given the technical depth of this paper, I have many questions, but they will not affect my recommendation. I may also continue to ask further questions during the discussion period as I plan on reading this paper in more careful detail than it is required for the review. \n\nOn the order of limits \n\nI believe the authors studied the DMFT equations originating in the regime where only $n,d \\to \\infty$, and $m,t$ stayed finite, and sequentially studied the regimes where $m,t$ are large. Do we know if the limits are different if we take $m\\to\\infty$ first, or jointly with $n,d,m$ together? I think at least in the case where $n,t$ are finite, the order of $d$ and $m$ do make a difference in the mean field regime. \n\nOn Figure 10 \n\nI am a bit confused reading Figure 10, as this should provide an empirical justification of the sharp transition, but I don't think I understand how this relaxation time is supporting this claim here. Can you provide a more intuition explanation of this Figure and what it is trying to achieve? \n\nOn Section G.2.1 \n\nThe authors studied the first dynamical regime for the mean field initialization here, where $t = \\Theta(1)$. In particular, there was a very strong simplification here where using a set of ansatz, the authors drastically simplified the DMFT systems (with memories) to a system of $k+1$ ODEs. I would like to understand this set of ansatz better, can you provide a more intuitive explanation of how you arrived at this set of ansatz, and why should we expect a simplification to an ODE system in constant order time?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths \n\n - A fairly comprehensive analysis when the width $m$ and training time $t$ is large, which resulted in a very interesting observation of the slower overfitting time scale, as well as the feature unlearning behaviour. \n - A very good collection of simulations, some of them to justify the non-rigorous approaches, but also to complete the overall picture in understanding the problem setting. \n - In combination with non-rigorous analysis, there are also rigorous results lower bounding the overfitting timescale, which helps to demonstrate where rigorous mathematics are currently at and their limitations. \n\nWeaknesses \n\nWhile it's understandable that the NeurIPS template is not very forgiving for long technical theory papers, I did find the paper somewhat difficult to read. Although I was able to figure some of these things out by asking a series of questions to LLMs, I still think the authors should fix the issues below to improve the reading experience: \n - The main text did not mention the Gaussian approximation done in Appendix A. I don't think it's particularly problematic to make this assumption for the sake of a tractable calculation, I just prefer if the authors can reference this aspect in the main text. \n - For the Gaussian process covariance, I would also prefer if the authors can specific what the expectation is over, in e.g. equation A.4. \n - Also the authors mentioned that the approximation error is vanishing on constant time scales and small on large time scales. It would be nice to reference which Figures that contained these simulations. \n - In Figure 2, the authors defined an $h(z)$ function without specifying how the simulations were done for a real neural network. I understand it's possible to construct an activation function that achieves this correlation function, but it would be more clear to me if the authors specified exactly which network is simulated. \n - In Figure 2, the authors also plotted the value of $a$ as a scalar. This was confusing to me because I thought the weights were a vector. Of course, after I read the argument that since weights are exchangeable, all the $a_i$'s must be the same throughout training, but I think it would be more clear if the authors can point this out in the figure caption. \n - The use of $\\gamma_{GF}$ was confusing to me. I think if the authors say something along the lines of \"there exist a constant $\\gamma_{GF}$ such that a sharp transition happens at...\" that would clear up my confusion, because this variable was not defined but used directly. \n\nAdditionally, I think it might be helpful to move/copy some of the Figures from the appendix to the main text. In particular, I think Figures 30 and 31 are quite clear at demonstrating the feature unlearning effect, hence justifying the third component of the cartoon Figure 1. The fact that a theoretical approach can predict this surprising phenomenon is in my opinion very impressive, and should perhaps be demonstrated early on in the text.", "was_revised": true, "final_justification": "I believe the submission was already in a good state before the rebuttal, minus some clarification points which the authors acknowledged as helpful. I remain supportive of accepting this paper.", "rebuttal": "$\\textbf{Weaknesses.}$\n\n- The Gaussian approximation was mentioned in line 123 of the manuscript. However this a short statement and  we plan to expand on it.\n\n- The expectation in Eq. A.4 is over the data distribution. We will clarify this and related points. \n\n- We will add a reference to the figure showing that approximation error is vanishing on constant time scales  (this can be seen, for instance, in Fig 4 right frame, by noting that the asymptote corresponds to the Bayes error.)\n\n- In the revised version of the manuscript we will specify the activation used in simulations.\n\n- Indeed, because of the symmetric initialization, second layer weights \nconcentrate around a common deterministic function $a(t)$ for all $t$\nthat remain bounded as $n,d\\to\\infty$. This is further discussed in Appendix\nC.4, but we will make it more explicit in the main text.\n\n\n-  $\\gamma_{GF}$ is the threshold defined via Eq.(2.3) and lines 166-167:\nwe will clarify this threshold phenomenon. \n\n- We agree  that it is remarkable that our theory is capable to predict the unlearning phenomenon. \nMoving figure 30 and 31 to the main text is probably hard because of space constraints. Note that \nFigure 5 also illustrate clearly the same phenomenon.\n\n\n$\\textbf{Questions.}$\n\n- $\\textit{Order of limits.}$ As the reviewer writes, in the DMFT analysis we take first $n$ and $d$ to infinity and then we study the large $m$ limit, allowing possibly the time to diverge with $m$.\nHowever, the rigorous results in Section 3 imply that the results obtained from the DMFT analysis have a \nlarger domain of applicability. In particular, no-overfitting take place for $t=o(m^{1/4})$ and long as\n$n/md = \\alpha$ is kept constant (regardless of the order of limits). \nSimilarly, the results in the first regime of DMFT are correct \nfor $t=O((\\log m)^c)$, and a small $c$  (for $n/md = \\alpha$  constant \nregardless of the order of limits). While we do not characterize the full domain of applicability\nof our analysis, these results imply that it is broader than could be assumed from the original derivation.\nIn the revised version  we make the dependence on $m$, $d$ and $n$ more explicit in Sec. 3.\n\n- $\\textit{Figure 10.}$ This figure refers to a setting in which second-layer weights are fixed to $\\gamma\\sqrt{m}$.\nWhen the control parameters (the overparametrization ratio $\\alpha$ or the scale $\\gamma$) are such that \ngradient flow coverges to zero train error, this convergence is exponentially fast in time. \nThe exponential convergence rate can be used to define a relaxation timescale that we estimate as detailed in lines 858-868.\nWhen either $\\gamma$ or $\\alpha$ cross the interpolation threshold, the relaxation time diverges.\nIn the inset of Figure 10 we plot the behavior of the relaxation time as a function of the control parameters.\nThis allows to determine the interpolation threshold $\\alpha_{GF}$.\nIn the main panel we plot the relaxation time as a function of $\\alpha_{GF}-\\alpha$, to verify that the relaxation time diverges as \n$(\\alpha_{GF}-\\alpha)^{-\\nu}$. The data suggest that $\\nu$ is close to $2$. \nIn the revised version of the appendix we will add a plot of the train error in linear-logarithmic scale. This will confirm the exponential relaxation to zero train error and  clarify Figure 10.\n\n- $\\textit{Section G.2.1}.$ The ansatz used on this timescale is presented in \nEqs.~(G.39)-(G.41). Among the scaling forms used in various dynamical regimes, we consider this particularly simple. We simply posit that all quantities have a finite limit as $m\\to \\infty$, without time rescaling. An indication that this should be\nthe case is already contained in the mean field theory [Mei, Montanari, Nguyen, 2018; Chizat, Bach 2019] which obtains a finite non-trivial limit as \n$m\\to\\infty$ and $t=\\Theta(1)$.\nNotice however that  the out-of-diagonal response $R_o$ turns out to be of order $1/m$ in this ansatz. This is intuitive because $R_o(t,s)$ captures the effect of changing the drift for the $j$-th neuron at time $s$ on the $i$-th neuron, $i\\neq j$ at time $t$.\nIt is reasonable to think that this effect is of order $1/m$ because each neuron is multiplied by a factor $1/m$ in the overall function."}, {"review_id": "ujF4xphKLu", "reviewer": "Reviewer_n5Nb", "summary": "This paper analyzes the dynamic behavior of a two-layer neural network under gradient flow training through the dynamic mean field theory, and finds that there is a clear time scale separation in the training process: the early stage is the feature learning stage, and the later stage enters the overfitting and \"feature forgetting\" stage, which further reveals that the generalization ability is closely related to the training dynamics. This is the first paper that theoretically explains the impact of initialization, training time, and network size on overfitting and generalization.", "questions": "1: What challenges arise when extending the analysis to multi-index models (e.g., k=2 and orthogonal), and are there simple examples illustrating the difficulty?\n\n2: Is the Gaussian input assumption crucial, or would the results generalize to more realistic input distributions?\n\n3: Can the theory suggest a practical, observable criterion for determining when to apply early stopping?", "limitations": "No. The paper does not explicitly discuss its limitations or potential societal impacts. It would be helpful if the authors could comment on the constraints of the single-index assumption and Gaussian input setting, as well as the applicability of their analysis to real-world deep learning scenarios.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1: The paper identifies a clear time scale separation between feature learning and overfitting in two-layer neural networks.\n\n2: The paper leverages dynamical mean field theory to provide a precise and quantitative analysis of training dynamics.\n\n3: The paper provides a theoretical explanation for early stopping, initialization effects, and scaling laws.\n\nWeaknesses:\n\n1: The analysis is limited to two-layer networks and gradient flow.\n\n2: The use of a single-index model as a target function limits the generality of the results.\n\n3: The assumption that input data is Gaussian simplifies analysis.\n\n4: Empirical validation is mostly numerical and lacks experiments on real-world datasets.", "was_revised": true, "final_justification": "The authors addressed most of my concerns in the rebuttal, particularly regarding the Gaussian input assumption and generalization of the DMFT framework to multi-index models. While I still find the modeling assumptions restrictive and the lack of real-data validation limiting, I acknowledge the theoretical depth and contribution of this work.", "rebuttal": "The reviewer points out a general weakness of our work.\nAdmittedly, this is a theoretical work that focuses on:\n\n1. A simple model of data (multi-index model with Gaussian features)\n\n2. A simple neural network (2-layer fully connected).\n\nFocusing on a simple example is the price we pay for being able to carry out an analytical study of the gradient descent dynamics.\n\n$\\textit{Earlier theoretical work was limited to significantly simpler cases}$ (eg linear models, networks with 1 neuron) or special regimes (eg neural tangent limit).\n\nDespite the simplicity of the setting, there is a lot that is poorly understood and a lot to learn from this setting. As we explain starting with lines 63 and 64, several of the fundamental question in this\nmodel generalize to more complex ML models and the conceptual picture we develop generalizes as well. \n\nConcerning the empirical validation, we do validate our results against numerical simulations on two layer neural networks, see for example Fig. 2 and 4. \n\nWe limit ourselves to simulations with synthetic data because our theory can only capture the behavior on real data at a qualitative level. In machine learning, this is nearly always the case for a completely analytical theory. The purpose of our simulations is to verify that various approximations made in our analysis are indeed accurate.\n\n$\\textbf{Questions.}$\n\n1. Both the general DMFT theory and the rigorous analysis results of Section 3 apply to general multi-index models.\nAgain, for moderate time-scales and large $m$, the dynamics is captured by the mean field \ntheory of [Mei, Montanari, Nguyen, 2018], [Chizat, Bach, 2018]. If the latter converges to vanishing training error in (rescaled) times of order one, then our analysis extends verbatimly. However, in multi-index models\nconvergence can take time slowly diverging with $d$. We do not expect that this will change the \nqualitative picture we derive, but the analysis becomes technically more complicated, to account for those cases. \n\n2. We assume isotropic Gaussian feature vectors ${\\boldsymbol x}_i$ Universality\nresults can be used to show that the same DMFT captures the dynamics when the ${\\boldsymbol x}_i$'s have independent\nentries (see, e.g. [Celentano, Cheng, Montanari, 2021]). Certain distributions with non-identity covariance can be \nalso treated (e.g. linear transforms of iids), but the analysis becomes more intricate\nand we do not expect qualitatively new insights.\n\n3. A general method that is supported by general theory is to monitor the error on a validation set and stop GF when this is minimal. Interestingly, our theory indicates that ---for large networks--- \nthe validation error will stay close to the minimum for a large interval of time. \nThis suggests that large networks are naturally stable with respect to the choice of stopping time."}, {"review_id": "Fty8xmx5Pr", "reviewer": "Reviewer_hZhE", "summary": "This paper presents an analytical study of the learning dynamics of a two-layer network, performing a supervised regression task, in the limit of large input size ($d$), large training set size ($n$), and (eventually) large number of hidden units ($m$). The scenario is (essentially) that of a teacher-student framework, with random gaussian inputs. The analysis is based on dynamical mean-field theory (DMFT), which reduces the training dynamics to a set of differential equations over a few selected macroscopic quantities. By manipulating the scale of the parameters initialization and analyzing different asymptotic limits, the authors identify several dynamical regimes in different scenarios, characterizing their properties in terms of training and test errors (and relatedly, feature learning and generalization gap). The findings of the DMFT analysis are supported by numerical experiments and some rigorous theorems.", "questions": "Main observations:\n* The second layer weights, which play a crucial role in the analysis, are consdered as being initialized as either $O(1)$ or $O(\\sqrt{m})$. But in a standard setup (with He initialization, for example) they would be initialized as $O(1/\\sqrt{m})$ instead. I think this should be clarified.\n* As I wrote above, I'm also unconvinced about the claim, first encountered in fig. 1 and then used in the discussion, that the $\\ell_1$ norm of the second-layer weights is a proxy for model complexity. This should also be clarified/elaborated upon.\n* As I already mentioned, the use of \"mean field\" or \"lazy\" initialization terminology, which is used ubiquitously, is never properly explained. So the reader finds e.g. fig. 2 and reads \"Here we use mean field initialization\", which was never defined.\n* The details of the simulations are only given sparsely in the main text (e.g. in fig. 4 we get to know the size of $d$, but not in the other figures), and references to the appendix where the details are reported are missing. I don't think I have seen, or possibly I have missed, the place where the complete details of the simulations with the \"actual 2-layer networks\" trained with SGD are reported (fig. 2). What was the size, the activation function, the hyperparameters etc. Right now I don't think that the results are really reproducible. The settings used to produce each figure should be reported, with a clear reference.\n* The choices of the $h(z)$ and $\\hat{\\varphi}(z)$ functions are never explained, and they sometimes change (in fig. 2 it's cubic, in fig. 3 it's quadratic, etc). I think there should be an explanation of how they were chosen, and what the choices entail in terms of the activation functions that give rise to them.\n* In several places in the main text there is a reference to \"SM\", which should be the appendix. Also, in several places in the appendix, we find sentences like \"see the appendices\". But we are already in the appendices. Please review the pointers to the appendices and, where possible, add a reference to the relevant section. This would significantly improve the readability.\n* At line 159 the fixed lazy regime is introduced, with $a(t)=\\gamma \\sqrt{m}$ non evolving with time. However, one could always absorb the entire weight in the activations $\\sigma$ and set $a(t)=1$. Thus, evidently, some assumption about $\\sigma$ and/or its input needs to be made, possibly encoded in the $h$ function. But I don't think I've seen such an assumption mentioned anywhere in the main text.\n* I think regularization (beyond early stopping) should be discussed in the last part of the discussion, about overfitting and feature unlearning.\n\nMinor observations:\n- below line 60: calling alpha the \"overparemeterization\" parameter seems a bit strange, considering that it decreases with the number of parameters. Parameter load, maybe?\n- line 94: one might mention that with the choice $k=1$ this is essentially a noisy \"teacher\" perceptron, as it's called in some related literature\n- line 100: here the large $n, d, m$ limit is presented. But only much later it is clarified that $n,d \\to \\infty$ first at fixed ratio $\\bar{\\alpha}$, and only later is the limit $m \\to \\infty$ taken (at fixed $\\alpha$, implying $\\bar{\\alpha}\\to0$). This seems relevant as it underpins the whole analysis\n- line 101: the rescaled $W_\\mathrm{2nd}$ are analyzed using the norm-1. Why the norm-1 and not the more common norm-2?\n- below line 120, eq. 2.2: I think the $\\mathbf{\\theta}_i$ in the second expression should be $\\mathbf{w}_i$, as in eq. C.1, otherwise it doesn't make much sense. The same issue occurs in eq. B.4 in the appendix. Also, in eq. B.4, the quantity $a_i(t)$ is considered, which I think should be included here too, otherwise we don't get the full system. Also, the first expression is missing a closing $\\rangle$.\n- line 147: the notion of \"good generalization\" is a bit puzzling in the context of learning pure noise. If it's about the generalization gap, this can only be achieved by making the training error as bad as possible right? Which is hardly \"good\".\n- line 165: I don't think that the quantity $\\hat{\\mathcal{R}}_n^g$ was ever mentioned in the main text, since it's part of the gaussian process description.\n- the caption of fig.3 mentions an inset which is not there\n- eq. 2.8 and line 199: suddenly there is no longer an explicit dependence of the quantities on $\\alpha, \\varphi, \\tau$ etc. This is puzzling.\n\nSome typos I found:\n- line 100: \"a $k=1$\" should be \"at $k=1$\"?\n- line 212: the limits for $t \\to \\infty$ are clearly wrong. Probably $m \\to \\infty$ was meant?\n- line 237: \"variables\" -> \"variable\"\n- line 257: missing a left $||$ for $\\sigma^\\prime$\n- line 273: \"scenario\" -> \"scenarios\"\n- line 525: is $\\mathbf{u}_i$ here supposed to be $\\mathbf{v}_i$?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strenghts:\n* Thorough, technically difficult and detailed analysis\n* Non-trivial findings albrit in a simple model, advancing the knowledge of this class of systems and revealing some key features and dynamical regimes\n* The overall analysis, its goals and the conclusions, are fairly well presented\n\nWeaknesses:\n* The details of the analysis are quite hard to follow at times. Mostly, I would say that the paper suffers from trying to cram a lot of content in the page limit, with a large appendix. In several places, it seems that the division between what went in the main text and what went in the appendix was a bit rushed. Also, some jargon is not explained, e.g. the use of \"mean field initialization\" vs \"lazy inizialization\" must be deduced from context. Overall, with the paper as it currently stands, the main text does not stand on its own. Furthermore, the main text does not point to specific places in the appendix where to look up the missing information (such as the details of the simulations, among other things), which is relevant when the appendix spans tens of paes.\n* In terms of significance, I can't help but thinking that regularization would have a dramatic effect on the findings. Also, I'm not entirely convinced by the use of the norm of the second layer as a proxy for the model complexity.", "was_revised": true, "final_justification": "After the discussion phase all my concerns have been addressed satisfactorily. I stand by my initial assessment of recommending acceptance, with minor changes required for clarity and reproducibility.", "rebuttal": "$\\textbf{Weaknesses}$\n\n- $\\textit{Writing.}$ We are going to improve the revised version of the manuscript in two ways:\n\n1. Clarify the notations and terminology.\n\n2. We will connect better the main text to the appendix.\n\n3. Try to make the best selection for the material in main text.\n\n- $\\textit{Regularization}$. We point out that we regularize explicitly the first layer weights by constraining them to have unit norm. \nInstead we do not regularize the second layer weights. \nFrom a technical viewpoint, adding a regularization for second-layer weights \nleads to a simple modification of the same DMFT equations.\nThe final result will be that such a regularization stops the growth of second-layer weights at a value depending on the regularization parameter.\nFor suitable choices of this parameter the unlearning/overfitting dynamical regime will become milder or totally eliminated. Even larger regularization strength will affect the first phase of training and hence induce underfitting.\nWe regard this as a natural (and quite simple) follow-up project.\n\n- $\\textit{Measure of complexity.}$\nThe fact that the $\\ell_1$ norm of second layer weights is a bound on complexity is a classical \nresult by Bartlett, see Ref.[6] (cf lines 145-150 of the manuscript). In principle it could be that the\ncomplexity is significantly smaller because of some special structure in the first layer weights. \nHowever we see no evidence of this in our analysis of the dynamics:\n1. First layer weights remain isotropically distributed (apart from the projection on the direction of the signal). Also, by construction their norm is fixed.\n2. Increase of the generalization error is accurately tracked by the increase in norm of second layer weights.\nThis is shown very clearly in Figure 5 right frame.\n\n$\\textbf{Questions}$\n\n1. The definition of the network in Eq.(1.1) has a factor $1/m$ in front of the sum. Therefore if one defines the second layer weights as $A_i=a_i/m$, then our \n\"lazy initialization\" corresponds to He initialization  $A_i =  \\mathcal{O}(1/\\sqrt m)$,\nwhile our \"mean field initialization\" corresponds to $A_i\\sim \\mathcal{O}(1/m)$.\n(We point out that the distinction between \"lazy\" and  \"mean field\" is standard in the theory literature.)\n\n2. See discussion above for the use of the norm of second layer weights as a measure of complexity.\n\n3. We will add definitions for this terminology: we agree that while common in the theory literature\nit is best to explain it for a broader audience.\n\n4. The connection between activation function and the function  $h(z)$ \nis straightforward the coefficients of polynomial expansion of the latter are the square of the Hermite coefficients of the former. A similar relation holds for $\\widehat\\varphi$. We will clarify this point. \nWe also plan to add the details of the numerical simulations.\n\n5. The choice of the functions $h(z)$ and $\\widehat \\varphi(z)$ in the figures was dictated by desire to select examples such that (i) The qualitative behavior of learning curves is the same as for generic such functions; \n(ii) The relevant phenomena are clearly visible.\nWe underline that the singular perturbation theory analysis applies to general\n$h(z)$ and $\\widehat \\varphi(z)$ \n\n\n6. We apologize for these misprints. We will change \"SM\" to \"the appendix\" and replace the \"see the appendix\" present in the appendix with the corresponding sections.\n\n7. Throughout the paper, it is understood that the activation function does not change with the width $m$.\nWe will clarify this in the text.\n\n\n8. We will add a discussion the effect of adding an explicit  regularization term in the dynamics.\n\n\n$\\textbf{Minor comments.}$\n\nThank you for the detailed comments. We will do our best to address them."}, {"review_id": "7eRDkXugfI", "reviewer": "Reviewer_97D6", "summary": "This paper uses dynamical mean-field theory to study the gradient flow dynamics of training a two-layer neural network on either random noise or a Gaussian multi-index model. The authors study both the lazy and the feature learning regimes, and characterize the timescales at which feature learning and overfitting occur.", "questions": "* It would be nice to state that the assumption $\\mathbb{E}[\\varphi(G)G] \\neq 0$ corresponds to Information Exponent = 1 in Ben Arous et al. 2021, which would give the readers a context of the regime of problems this paper studies compared to the broader literature of high-dimensional SGD dynamics for learning Gaussian single-index models.\n\n* The arguments in the main text seem to suggest usually $a_i(t)$ is constant in $i$ for all $t$, which is understandable in the setting when $a_i$ is close to initialization, but how can one intuitively see this over longer training times?\n\n* What is z at the bottom of page 6? Does it represent rescaled time? It was a bit confusing to suddenly see z.\n\n\nG. Ben Arous et al. \"Online stochastic gradient descent on non-convex losses from high-dimensional inference\"", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The most interesting aspect of the paper to me is the observation that long training time in the feature learning regime can also lead to overfitting, and some notion of early stopping is necessary to achieve optimal test performance. This is in contrast with most recent analyses of training two-layer neural nets to learn multi-index models, which either use $L_2$ regularization or constrain second layer weights, and never notice overfitting as $t/m \\to \\infty$. Relative to the these analyses, the DMFT machinery used here is powerful enough to characterize many different behaviors without significant modifications of the gradient flow.\n\nMy main concern with the paper is its presentation and clarity. The authors mostly describe different dynamical regimes informally throughout the paper, and some formal statements would help better understand each regime. Even having informal theorem statements could help to better keep track of different phenomena that occur during each regime. Also, from reading the main text alone it is not entirely clear which parts of the analysis are rigorous, which parts are non-rigorous calculations and which parts follow from solving DMFT numerically.", "was_revised": true, "final_justification": "I didn't have any major concerns with the paper in my original review, and the authors responses have also addressed my minor comments.", "rebuttal": "$\\textbf{Weaknesses}$\n\nWe will address the problem of readability of the manuscript. \nAs other reviewers requested we plan to:\n\n\n1. Clarify the notations and terminology.\n\n2. State formally a the most important conjectures/results arising from the DMFT analysis.\n\n3. We will connect better the main text to the appendix.\n\n4. Try to make the best selection for the material in main text.\n\n\n\n$\\textbf{Questions}$\n\n1. In the revised version of the manuscript we will clarify the role of\nthe assumption $\\mathbb{E}(\\varphi(G)G)\\neq 0$ and refer to earlier literature concerning this point.\n\n2. The fact that $a_i$ is independent of  $i$ is a consequence of two facts.\nFirst, under the initialization, neurons are exchangeable (in the sense of probability theory),\nand hence they remain exchangeable at all subsequent times. As a consequence of exchangeability \n${\\mathbb E} a_i(t)$ does not depend on $i$. \nSecond, second layer weights $a_i(t)$ concentrate around their expectation for any $t=O(1)$ as\n$n,d\\to\\infty$. Within DMFT, this is discussed in sec. C.4. We could expand on this point from a mathematical perspective.'\n\nThe claim that (under the symmetric initialization that we use) \nthe $a_i$ concentrate around a value independent of $i$ for $t=O(1)$\nas $n,d\\to\\infty$ can in fact be proven rigorously.\n\n3. $z$ is the scaled time. We will make the notation clearer in the revised version."}, {"review_id": "y1TLA5IGKO", "reviewer": "Reviewer_FYDY", "summary": "This paper analyzes training dynamics in overparameterized two-layer networks using Dynamical Mean Field Theory (DMFT). It reveals a timescale separation and describes in details the corresponding dynamical regimes, under both lazy and mean-field initializations.", "questions": "1. The paper identifies a phase where test error increases with training (\"feature unlearning\"), necessitating early stopping. However, benign overfitting (where test error remains low despite interpolation) is common in practice. What specific aspects of the model or data assumptions are responsible for the observed non-benign overfitting?\n\n2. While DMFT is proven to be rigorous for related settings, could the authors clarify which specific predictions in Section 2 are expected to be rigorous and which parts are based on further assumptions or numerics and might have potential limitations? What are the main barriers to full mathematical rigor for these results?\n\n3.  Given DMFT can handle discrete steps, what are the key challenges in extending the core findings to discrete GD or SGD? Would you expect significant qualitative differences?\n\n4. This paper takes n,d to infinity and then m to infinity. Recently there are a lot of papers taking the limit m,n,d to infinity together with m/d, n/md fixed. Do we expect them to be equivalent?\n\n5. L159: what do you mean by fixing a(t)? Does this mean the second-layer weights are fixed while only the first-layer weights are trained? Is every element of the second layer the same?\n\n6. Could you comment on eq.2.6? How far is it from Bayes optimal and what determines the gap?", "limitations": "The authors should comment on whether the results might change when we use GD/SGD, which parts of the results are conjectured to be rigorous, and compare the results with existing literature, e.g. on implicit biases and benign overfitting", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Overall this is a good paper that describes different training regimes in detaild. My main concern is that the presentation in Section 2 lacks clarity on which findings stem directly from the (conjectured to be rigorous) DMFT equations versus those derived from scaling hypotheses or numerical integration (which may have limitations). Moreover, it is worthwhild to compare with existing results on the implicit biases of GD, benign overfitting, etc.  Specifically, why does benign overfitting not occur here?", "was_revised": true, "final_justification": "This is a solid paper and I recommend acceptance. I do not give score 6 because of the existing limitation of DMFT (e.g. Q2 and Q4).", "rebuttal": "$\\textbf{Questions}$\n\n1. In our setting we do observe benign overfitting. At the end of the \"feature unlearning\" phase, the test error is the one achieved by the neural tangent model. This model has benign overfitting [see, e.g. \nGhorbani et al 2016; Montanari, Zhong, 2020].\nNamely, for $n\\to\\infty$ the excess error converges to $0$.\nHowever this happens only for $n$ superpolynomial in $d$. (The phenomenology established in [Montanari, Zhong, 2020]\nfor  neural tangent models matches the general framework for benign overfitting in [Bartlett et al. 2019]\nwhere is known that the excess risk vanishes only very slowly with sample size) \nOn the other  in the mean field/feature learning regime of training the excess error vanishes already when \n$n\\gg d$.\n\n2. There are several possible routes to making the analysis of Section 2.\nWhile a full discussion would be too complex, a possible route would be:   \n(i) Use the DMFT equations for actual 2-layer networks\nthat were proven in Celentano, Cheng, Montanari, arXiv preprint arXiv:2112.07572, 2021\n(instead of the simplified DMFT we use in our paper).    \n(ii) Carry out the singular perturbation analysis of these equations.  \nA first step towards (ii) would be to rigorize the singular perturbation analysis in the simplified\nDMFT that we use in our paper. \nEven in this simplified setting, step (ii) is the most challenging because\nit requires to control the solutions of complex non-linear PDEs in the long time limit. \nThe rigorous results in Section 3 partially address this point.\n\n3. There are no particular complications that make the analysis more complicated for SGD. Indeed, for $m=1$ and single index data, the DMFT analysis of SGD dynamics was performed in:  \nKamali, Urbani, arXiv preprint arXiv:2309.04788, 2023. We do expect a similar behavior.\n\n4. The rigorous results in Section 3 imply that the results obtained from the DMFT analysis have a \nlarger domain of applicability. In particular, no-overfitting take place for $t=o(m^{1/4})$ and long as\n$n/md = \\alpha$ is kept constant (regardless of the order of limits). \nSimilarly, the results in the first regime of DMFT are correct \nfor $t=O((\\log m)^c)$, and a small $c$  (for $n/md = \\alpha$  constant \nregardless of the order of limits). \nIn particular, these claims apply to cases in which $m/d$ is kept constant.  \nWhile we do not characterize the full domain of applicability\nof our analysis, these results imply that it is broader than could be assumed from the original derivation.\n We plan to make this point clearer by making explicit the dependence of the results in section 3 on $n$, $d$.\n\n5. The reviewer is correct. In line 159, we fix the second layer weights at initialization and do not train them. \nSince we use a symmetric initialization, see Sec. C.4, all the second layer weights are the same.\n\n6. The Bayes optimal value of the test error is $\\tau^2/2$ (in the limit $n/d\\to\\infty$ considered here).\nThis is achieved during the feature learning phase of training. In the lazy phase, the test error\nincreases because the model converges to a neural tangent kernel model that only learns the linear component of the target.\n\n$\\textbf{Limitations}$\n\nWe appreciate the referee comments. As discussed above, we will try our best\nto address them and discuss limitations/extensions."}]

Author rebuttal:

Meta-review: This paper provides a precise characterization of the learning dynamics of two-layer networks via dynamical mean field theory (DMFT), under the joint regime of large input dimension $d$, sample size $n$, hidden layer width $m$, and training time $t$. Analyzing these joint limits is highly non-trivial, and the authors develop a refined approach to handle them. The findings are interesting and novel: the learning dynamics can be divided into three distinct stages—(i) mean-field feature learning at $O(1)$ time; (ii) extended feature learning for $1 \ll t \ll m$; and (iii) overfitting and feature unlearning when $t \gg m$. It can help to **mathematically explain several major phenomena in deep learning theory**, including lazy vs. non-lazy training, benign overfitting, double descent, and implicit bias or regularization (early stopping).

The main weakness of the work, as highlighted by reviewers, is clarity of presentation. Since DMFT is unfamiliar to much of the machine learning theory community, the paper would benefit from significant polishing and from making the technical framework more accessible as mentioned by the reviewers. Besides, I also encourage the authors to incorporate the reviewers’ feedback, for instance, the order of the joint limits, the relationship to the information exponent, the calculation between $h$ and activation function, the extension to gradient descent, and the connection between model capacity $||{\bf a}||_1$ (this is actually equivalent to the $\ell_1$-path norm given $||{\bf w}_i||=1$ in the first layer and path-norm has demonstrated to be good model capacity, both empirically and theoretically).

From my own reading, I also suggest consolidating the six motivating questions in the introduction into one or two central guiding questions. Framing the work around “how to precisely characterize learning dynamics for generalization, and identify the phase transition within implicit bias” would make the paper more focused and impactful. I also have some concerns but just name a few:

- Line 89: the authors claim "the **largest** possible gap between linear/kernel learning" (for Gaussian data). It seems too strong without further justification or analysis on "**largest**" if we use other symmetric distributions in recent work for single-index models.

- Line 1496: it seems that $Q_{ll'}$ should be expressed as a sum of the two terms $\tilde{Q}$ and $r_l r_{l'}$, not a difference; otherwise it contradicts Eq. (K.8), which uses $h(\tilde{Q}+r_l r_{l'})$.

---

Overall, this paper develops a promising DMFT-based framework for characterizing the nonlinear learning dynamics of two-layer neural networks. It lays mathematical foundations for understanding **nonlinear dynamics, major phenomena, and scaling laws beyond linear models** and has thus strong potential impact in the ML theory community. All of the reviewers vote for acceptance and I recommend this paper for oral presentation.
Decision: Accept (oral)