Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
Authors: Blake Bordelon
Cengiz Pehlevan
Venue: ICML 2025 poster (2025)
Keywords: deep learning
mean field
learning dynamics
$\mu$P
residual networks
dynamical mean field theory
Primary area: deep_learning->theory
TL;DR: A theory of train and test loss dynamics for randomly initialized deep linear networks with applications to hyperparameter transfer.

Abstract: We theoretically characterize gradient descent dynamics in deep linear networks trained at large width from random initialization and on large quantities of random data. Our theory captures the ``wider is better" effect of mean-field/maximum-update parameterized networks as well as hyperparameter transfer effects, which can be contrasted with the neural-tangent parameterization where optimal learning rates shift with model width. We provide asymptotic descriptions of both non-residual and residual neural networks, the latter of which enables an infinite depth limit when branches are scaled as $1/\sqrt{\text{depth}}$. We also compare training with one-pass stochastic gradient descent to the dynamics when training data are repeated at each iteration. Lastly, we show that this model recovers the accelerated power law training dynamics for power law structured data in the rich regime observed in recent works.

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
Blake Bordelon 123
Cengiz Pehlevan 123
Abstract
Despite the complexity of the dynamics induced by gradient descent from random initial conditions, significant theoret- ical insight into deep network training can be gained by studying tractable limits (large width or large depth net- works) or specialized architectures (such as deep linear networks) or the behavior of training under specific data distributions (such as random isotropic data). While the literature on various special cases of training contains nu- merous works which we review in Section 1.1 below, a theoretical model that can capture simultancously the effect of varying width, depth, richness, data quantity on the learn- ing dynamics randomly initialized networks is missing, Of specific interest is that no existing theory can theoreti- cally characterize the two effects necessary for successful hyperparameter transfer (Yang et al., 2021): (1) monotonic improvement in performance with model width or model depth and (2) approximately consistent optimal learning rates when appropriately parameterizing the model and op- timizer.
We theoretically characterize gradient descent dy- namics in deep linear networks trained at large width from random initialization and on large quantities of random data. Our theory captures the "wider is better" effect of mean-fielddmaxximu- update parameterized networks as well as hy- perparameter transfer effects, which can be con- trasted with the neural-tangent parameterization where optimal learning rates shift with model width. We provide asymptotic descriptions of both non-residual and residual neural networks, the latter of which enables an infinite depth limit when branches are scaled as 1 1/ //eeph.. We also compare training with one-pass stochastic gradi- ent descent to the dynamics when training data are repeated at each iteration. Lastly, we show that this model recovers the accelerated power law training dynamics for power law structured data in the rich regime observed in recent works.
Is there simple theoretical model of deep networks that captures the influence of random initialization, network parameterization, width, and depth in order to characterize the hyperparameter transfer effect?
1. Introduction
Understanding the training dynamics and scaling proper- ties of neural networks is critical to deriving parameteriza- tions and optimizers that achieve stable dynamics across model sizes (Yang et al., 2021; Bordelon et al., 2024c; Yang et al., 2023; Vankadara et al., 2024; Everett et al., 2024), which turn lead significant computational savings in hyperparameter search via optimal hyperparameter transfer from smaller to larger sized models (Achiam et al., 2023). However, characterizing the training dynamics of neural networks is generally complicated due to nonlinear dynam- ics that depend on the parameterization of the network, the weight initialization strategy, optimizer, and data structure.
In this work, we provide such theory, though for a lim- ited model class of deep linear networks, by characterizing the training dynamics from random initialization on ran- dom data which captures both the effect of network width, depth, dataset size, and feature learning strength (richness). Concretely our contributions are
We develop novel dynamical mean field theory (DMFT) which is specific for deep linear neural net- works that enables a prediction of typical case loss and representation trajectories over both initial weights and random data, or stochastic gradient descent (SGD) samples, the disorder. This theory completely char- acterizes the train and test loss dynamics in terms of a closed set of equations for correlation and response functions at each hidden layer of the network.
'John Paulson School of Engineering and Applied Sciences, Harvard University *Center for Brain Sciences "Kempner Institute. Correspondence to: Blake Bordelon <blake baakeleboddloone @ g. hdd,, Cengiz Pehlevan <epehle- van @ seas..aavaaaee hawvard.evusss
We show that our theory provides minimal model that captures the impact of parameterization on train- ing dynamics. In particular, our results capture both the "wider is better" effect of deep networks throughout
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
training as well the hyperparameter transfer effect for networks in j.P Conversely, our theory captures that wider networks can train more slowly and hyperparam- eters can fail to transfer in NTK parameterization.
The limiting N - oc gradient descent dynamics in MLPs for fixed (finite) stream of data in the rich regime have been characterized with partial differential equations in one- hidden layer networks (Mei et al., 2019; Rotskoff & Vanden- Fijnden, 2022) as well as Tensor programs (Yang & Hu, 2021) and dynamical mean field theory (Bordelon & Pehle- van, 2022) techniques for deeper networks. These meth- ods describe the typical case behavior of training nonlinear networks over random realizations of the initial weights. However, the resulting theories are generally intractable as they scale poorly with the size of the training dataset P and the number of steps of training I' outside of the lazy training regime. A key challenge in this deep setting is to also av- erage over the random data to obtain a reduced description where effects of order & are preserved. Various perturbative approximations & have been studied, though these do not capture a full joint limit for data and width and also suffer from intractability at large dataset sizes (Dyer & Gur-Ari, 2020; Roberts et al., 2022; Hanin & Nica, 2019; Bordelon & Pehlevan, 2023).
We extend our theory to provide solution to the train- ing dynamics of deep residual networks, enabling the study of the large depth limit, though in a tractable set- ting. In this setting, we reproduce that hyperparameter transfer across depths by scaling the residual branch by 1/ depth, consistent with prior works.
We also show that our theory can capture the behavior of linear networks trained on structured data, such as power-law covariance eigenvalues. This results in power law scaling of the loss with training iterations, though with improved exponents compared to lazy learning, consistent with recent findings in the shallow case (Bordelon et al., 2024b).
1.1. Related Works
Statics and Bayesian Networks Theory of the statics! of deep Bayesian linear neural networks in the regime where dataset P and width N are proportionally large were first characterized in Li & Sompolinsky (2021), resulting in theory of a renormalized kernel after training. Pacelli et al. (2023) and Baglioni et al. (2024) theoretically and experi- mentally argue that the scale-renormalization of the NNGP kernel continues to capture the effect of feature learning in nonlinear Bayesian networks. Their results were recently extended to convolutional networks (Aiudi et al., 2025; Bas- setti et al., 2024). Learning curves which average over randomly sampled training set were obtained by Zavatone- Veth et al. (2022) with the replica method, allowing contrast between Bayesian inference over all layers with training random feature models, who extended the theory to struc- tured covariates (Zavatone- Veth & Pehlevan, 2023). Cui et al. (2023a) showed that these learning curves also de- scribe learning curves for nonlinear multilayer Perceptrons (MLPs) under Bayes-optimal teacher-student learning setup in the proportional limit. Hanin& & Zlokapa (2024) used similar techniques to analyze weakly nonlinear Bayesian networks, where the nonlincarity is depth dependent, at NN' fixed.
Residual Networks and Large Depth Limits Theory to characterize infinite width and depth training dynamies N + do and L, -> oc have been recently obtained for net- works starting at random initial conditions Bordelon et al. (2024c); Yang et al. (2023). More recently, the work of Bor- delon et al. (2024b) studied depth L = 2 finite width linear networks trained with a projected gradient descent using mean field techniques. In this present work, we characterize exact gradient descent for arbitrary depth L networks. We also study the role of μP scaling to achieve learning rate transfer across width and scaled residual branches (Bordelon et al., 2024a; Yang et al., 2023) for transfer across depths.
Linear Network Gradient Flow ODEs The dynamics of gradient flow training in deep linear networks in the very rich regime (or equivalently for small initialization) were computed by Saxe et al. (2013); Atanasov et al. (2022) as simple set of ordinary differential equations. Kunin et al. (2024) and Dominé et al. (2024) recently derived similar equations for the dynamics of networks at differ- ent laziness/richness scale and unbalanced initializations. However, these methods require number of assumptions including whitened input data, a balance assumption on products of weights in adjacent layers, and lack of bottle- neck or overparameterized layers. Our work goes beyond these assumptions by studying generic (random and non- negligible) initialization of the network weights, which in- troduce interesting dependence of the dynamics on network width and dataset size. In addition, our theory allows for sampling random training data, which will generally not give a whitened empirical covariance".
large width N, depth and dataset size P with LP
Dynamics from Random Initialization In the lazy leamn ing regime, the infinite limit of a randomly initialized net- work is characterized by the Neural Tangent Kernel (NTK) (Jacot et al., 2018; Lee et al., 2019). However, if one adopts scaling that enables feature learning at infinite width (mean field/ / μP scaling) then the dynamics of deep network training from random initialization are more complicated.
This could be interpreted as sampling neural network from an equilibrium distrihution, which can be contrasted with out-of- equilibrium dynamics which are our focus in this work.
EEvn if the data's population distribution is isotropic, the empirical eigenvalues will follow a Marchenko- Pastur law.
2

---

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
Linear Network Models of Neural Scaling Laws. Bor- delon al. (2024a) and Paquette et al. (2024) recently ana- lyzed a linear random feature model trained with gradient descent, finding that training dynamics on data that satis- fies source and capacity conditions recovers Chinchilla-like neural scaling laws (Hoffmann et al., 2022). Bordelon et al. (2024b) recently extended this theory to a case where the random projection matrix is also updated with projected gradient descent, resulting in faster power law conver- gence rate for hard tasks. Their model is mathematically equivalent to one hidden layer linear network trained with a projected version gradient descent. We find this same acceleration behavior in deeper linear networks trained on power law data in this work.
We consider linear (potentially noisy) target function of the form
1 v(Φ) vD w. + OE(x)
(3)
where (e(x)) = 0 and (e(x)²) - 1. All parameters follow gradient descent dynamics on the square loss with the μP scaling of the update from gradient descent which takes the form
w'(++
+
Mo g (())"()) VN
g
(4)
where we introduced the following vectors
vD P
h (1) VN (') h²(()=
Σ
(1)
DMFT Approaches to Machine Learning Theory Dy- namical mean field theory methods, originally developed in the physics of spin glasses, have been used to describe the dynamics of high dimensional optimization (includ- ing stochastic gradient methods) in terms of single vari- able stochastic integro-diifirrentia equations (Agoritsas et al., 2018; Mannelli et al., 2019; Mignacco et al., 2020; Mignacco & Urbani, 2022; Gerbelot et al., 2022: Dandi et al., 2023; Bordelon et al., 2024a;b). These prior works have shown the validity of DMFT to characterizing general linear models or shallow neural networks trained with pa- rameters and data going infinity proportionally. In our work the disorder comes from both the data matrix and the random initial weights of the network and we are interested in the behavior of arbitrary depth networks.
P ==1
1 (') g ()), g² = -0(()=='(()) wt)).
g g'())
(5)
VN
and the instantancous training errors (t) = y( - If faz t). We note that the vectors h (t) defined above dif- fer from the preactivations used in the DMFT of Bordelon & Pehlevan (2022) since ours training set averaged quanti- ties which are computed as a correlation between the errors A₂ and the usual per-sample preactivation vectors. The fact that sums over training points and sums over neurons can be reordered in linear networks enables easy analysis of a proportional limit, where width, data, and input dimension are all comparable, rather than only computing the N - do limit fixed P, D. In addition to capturing better finite width and finite data effects, this theory leads to signifi- cantly simpler and casier-to-come compute description of training dynamics as we illustrate in Appendix Table 1.
2. Deep Linear Network with Random Data
The first model that we study is a vanilla feedforward linear MLP which has L hidden layers of neurons (and L + trainable weights) which maps input x E R2 to outputs (x) defined as
2.1. Dynamics of (S)GD in a Joint Asymptotic Limit In this work we consider two distinct setting that allow us to identify different effects from subsampling data. First, we examine full batch gradient descent with / data points that are reused at each step. In this setting, we study a large system size limit with P, D, N - xo limit where the ratios are constants
vD Noo (wo)
+ VN
w¹) (1/
f(x)
II Emi
x.
Setting (Full Batch GD): P/D =a N/D=V
(6)
The first matrix has dimensions 8 E RND. while all intermediate weight matrices are all of size w E N Lastly, the output w E N maps to a single output. We can provide extensions to unmatched hidden widths, struc- tured data, and multiple (finite) output classes in Appendix F. E. and G, but we begin our analysis with this simplest possible case. We start by characterizing the dynamics for a random training dataset D {a,,s M of size P, and random initial weights
Analyzing fixed points of GD in the above limit is a popular setting for two layer random feature models (Mei & Monta- nari, 2022; Adlam & Pennington, 2020a; Hu & Lu, 2022) and deep Bayesian networks at equilibrium (Li & Sompolin- sky, 2021: Zavatone-Veit et al., 2022; Cui et al., 2023a) but the training dynamics from a random initial condition in such a limit has not been explored In addition to full batch
'The v + oo limit would recover to the commonly studied "infinite width limits" where width is taken to infinity first at fixed P. D that disregards effects of order N/D N/ P/P (Jacot al., 2018; Lee et al., 2019; Bordelon & Pehlevan, 2022).
TP ~ ~N I), Wf, W.O W9 NN(0.)). 1).
3

---

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
10a
v 025
10a
0.25
10a
0.50
V- 0.50
L=4
1.00
2.00
2,00
16
10
5.00
c(0)
DMFT
DMFT
10
10-2
10
10:3
40 80 100
20
80
40
100
(a) Varying Width in NTK-P
(b) Varying Width in MF/pP
(c) Varying MLP Depth
Figure 1. Our theory captures the role of width, depth and dataset size on the test loss [(t) and train loss L(t) dynamics. The behavior of different models across widths depends on how "N scales with N. (a) Test loss of a I = network in NTK parameterization where " e ) which results in kernel method (linear model) in the + limit. (b) In MFP/μPP the parameter "oo = @(1) and the convergence rate improves as y increases. (c) Our theory allows us to track dynamics in deeper networks of arbitrary depth L as well.
gradient descent training, we also look at online stochastic gradient descent (SGD) training, where at each step fresh batch B data points are sampled and used to estimate the gradient with the following proportional limit
In addition to the correlation functions, the theory also requires computing collection of response functions { RRu.. R RA, R,. R..-1 RR. Ru ture the effect of limited data and limited width from nite a and finite effects respectively as well as interac- tions between the forward/backward h g variables (see Appendix A). In the joint limit, each entry of the vectors (v((), (f (1), g'), become statistically independent and identically distributed following a stochastic process known as the single-site process. As an example, the in- termediate layers of the network < < L satisfy the following single site (scalar) processes
Rdu) which cap-
Ar
Setting 2 (Online SGD): B/D = on
N/D E..
(7)
These two settings (full batch GD and online SGD) both converge to the dynamics of gradient descent on the popu- lation loss as a -> co or Q + oc respectively. In either of these settings, large system size limits are taken with the number of iterations fixed, which is appropriate since all learning timescales are (1).
n²(t) = u((t) +
Σ ( + mo, (t,t)) (t))
The key quantities which determine the generalization dy- namics in both of our settings are correlation functions
Pct
g'(t) = (t) + Σ
(t, + m, (t, (t(t))
6'
N (() (t),), C) N (t) g() C(((,,'))=
(11)
1 Ca(t,t)= - A(t) At) Co(t,t)= D v(t) v(t).
where u'(t) NO0 Cj Ch-1) and r'() N(00 G
(8)
colored noise sources. The response functions are computed as derivatives of these single site variables with respect to their noise source
where the error vector v(t) is defined
VD w (t) (t).
vtt) = wt
(9)
(t, = < (rr(())) )
R (t, ( 5g Suu(tt)
(
Noo
We stress that the number of these order parameters does not depend on D,N, but only scales with the number of iterations T' and the number of layers L. which justifies the saddle point derivation in the Appendix A. From the correlation functions, we can directly obtain the test loss (()) and train loss L(t) as
(12)
Intuitively, these functions measure the response of the vari- ables at time / to a "kick" in the noise source at a time l' (due to causality, these only contribute for t' < t), We pro- vide the complete set of DMFT equations in Appendix A.2, which form a closed set of equations for the correlation and response functions. Once these are solved for, the the test
C(t) (t, +o+ 0² 20) Ca((,)) (10)

---

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer emror takes the form
Learning Rate Transfer Experiments from Yang al. (2021) illustrate that the optimal learning rate is approx- imately preserved across model scales in the jPP scaling, while Noci et al. (2024) show experiments where optimal learning rates shift to the right as width increases in NTK parameterization. Our theory can capture both of these ef- fects as we show Figure 2. The key effect is that, at fixed learning rate 17, the rate change of the kernel decays with network width in NTK parameterization, while it is approximately constant in pP.
C(t) - ΣΣ
Re (1, (t))
(t,
SCt
Noise
Bias
(t, 8) (t.
s'et
Limited Width Variance
R Ro (t, R (t, Ca
(13)
a SCE
d'st
10a
Limited Variance
10-²
The last two lines capture the fluctuations in the entries of the error vector v(t) which are due to limited width (finite v)) and limited data (finite a) respectively while the first line would be the error dynamics of an ensembled and bagged model (the error of the average function learned over different initializations and different draws of training data) (Adlam & Pennington, 2020b; Bordelon et al., 2024a). The response function Ro, also depends on both y/ and and captures the effect of finite and a on the bias dynamics, which persist even for an ensembled and bagged predictor.
((3)
10
v- 0.2
0=5
10-4
2.0
5=0
10-4
10-2
10
n
Learning Rate Sweep in NTK-P
Varying Width and Parameterization Our theory cap- tures the effect of varying width through the constant = N/D. Depending on how the model is parameter- ized with respect to model width N, different behaviors are possible as y varies in the proportional limit. In Figure (a)-(b) we compare how models behave across different v in NTK parameterization (where 7o O (v-11/) com- pared to mean field parameterization where "To is constant with respect to V. In NTK parameterization, the network approaches linear model (in parameter space) whereas the dynamics remain non-linear in the v - oo limit. In either case, the key equation which captures the effect of network width the single-site equation for the firs layer 1²2 (1) variables
10-1
10-1
((t)
10-5
D.50
p= 1.00
10-7
5.00
DMFT
+ co,
10-9
(b) Learning Rate Sweep in jP
Figure 2. The theory can capture the failure/success of learning rate transfer in NTK/pP networks. (a) The optimal learning rate increases with model width N/ TD in NTK parameterization. This is due to the fact that the effective feature learning rate "N is decreasing as 1/v0. (b) The optimal learning rate is approxi- mately preserved across v N/D D in μP networks where "yo is constant, enabling learning rate transfer.
D Rd (t, (1)
I V70 Eet
(t) =
Gaussian Process
Limited Width Effects
Tyro
Σ Pet
chit,
(14)
Effect of Limited Data or Batch Size The effect of lim- ited data on the dynamics generates memory term for the training error A(I) response function
Feature Learning Updates
These limited width effects can propagate through the other layers in the forward and backward pass and ultimately bias the dynamies of the network predictions, leading slower training. For mean field parameterization, the feature learning updates scale as 7%o which is constant across model widths, while for NTK parameterization, the feature learning update seales as ~ m// Ve which decrease with model width.
RA (t,t,') 6(t t') 1
Σ R2 (1, (tat") te
Limited Data Bias
(15)
which results a slow buildup in the gap between train and test loss dynamics as we illustrate in Figure 3 (a). When
5

---

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
10a
0.50
a 0.75
50
t= 100
m 2.00
t 250
4.00
0.6
t= 500
2(0)
Test
1000
Trin
Stop
DMFT
10"4
Expt
0.0
20
40
60
80
100
120
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
(a) Test Loss Varying Data 7o
(b) Emergence of Overfitting from Data Repetition
10a
Gg = 0.03
107
== 0.25
dg = 0.30
0.50
dg = 0.50
2.00
c(0)
10
10-1
DMFT
10
10
20
60
80
100
120
20
100
t
t
(c) Online SGD Dynamics
0.25
(d) SGD varying Small og 0.05
Figure 3. Train and test losses of a depth L = 4 linear network versus dataset size P = aD compared to online SGD with batch size ug (a) Networks can fit training data faster for smaller datasets a. (b) The test error can be non-monotonic as function data a if training is performed too long. (c) The dynamics of online SGD with v = 0.5 across batchsizes B = ax D. Smaller batchsizes slow down training through accumulated SGD variance, there not a bottleneck eror as + oc provided the dynamics are stable, (d) Increasing the width eliminates the initial overfitting transient (see Appendix D.2).
data is repeated for many iterations, the loss exhibit overfitting peaks (d' Ascoli al., 2020; Mei & Montanari, 2022; Advani et al., 2020; Zavatone- Veth et al., 2022) as we show in Figure 3. However, for online SGD training the loss monotonically improving with model width and batchsize = D as we illustrate Figure (c)-(d).
scaling 3 8o/vl for constant fo. which recovers the parameterization recovers the depth-uP of (Bordelon et al., 2024c; Yang et al., 2023)*.
We show our theory compared to experiments in Figure where we illustrate the impact of richness Yos branch multiplier 30 and depth on the dynamics when we are adopting the YL scaling that admits a large depth limit, Examples illustrating hyperparameter transfer are provided in Figure 5, where keeping 3 O(1) constant leads to diverging activity at fixed learning rate L + whereas scaling 3 (L-1122 admits well defined limit. Thus hyperparameter transfer in the rich regime is maintained in the B = Bo/ VL. scaling.
3. Residual Networks and Depth Limits
While we studied the large width limit at any fixed finite value of L, to obtain a well defined large depth L + do limit, we next turn to residual networks which improve faithful forward and backward signal propagation
VD Noo
B VN
w*)
( wo)
f(x)
(w
II
I
Infinite Depth Limit When scaling the branch multiplier de.
(16)
as 3
the dynamics admit a L + oc limit. This L limit is simplified version of the result obtained in Bordelon et al. (2024c) where both the response functions and correlation functions survive in the L + oo limit (see
VL
We first provide the DMFT for the proportional N limit for any finite 3 and L. We show that unlike standard feedforward networks, residual networks require characteriz- ing cross-layer correlation and response functions since the residual stream variables are statistically dependent. Once we have derived the proportional limit, we can then consider
AAlernnativel. if one commits B Bo/ Bo/VT. then one take the N, P, - op limit, which gives the same result.
limit first and then take the proportional
6

---

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
10a
109
10a
10
10
E(O)
10-4
1.0
1.0
10
1.5
Y6 4.0
2,0
6.0
2.5
8,0
10
10
50
100
150
200
50
100
150
200
50
100
150
200
(a) Increasing Richness L 16 (b) Increasing Branch Scale L 16
(c) Increasing Depth
Figure 4. The training dynamics of large depth residual networks with random initialization. (a) Increasing leads to acceleration of training. but the initial kerne governing the updates are changed so that [(t)]e-o is preserved. (b) Increasing & alters both the initial kernel and the feature learning dynamics. (c) Increasing depth L with 3u 1.2 eventually converges to limiting dynamics.
dynamics (to be integrated from 0 up to 1), giving single site dynamics
10a
10-1
dh t) = Boddu(T, r)
10
C(0)
+ dr
(T, (o0
P't
10-4
(17)
L=d
10-5
DMFT
where the Brownian motion has covariance structure (du(r,))du(r') (ou(r'(())) 5(r (T)Cx(T (')drrr' which is local in layer time but non-local across training steps (see Appendix C.1 for more detail).
10-6
10-4
10-1
10-2
10-1
(a) No Branch Scaling B =
4. Structured Data & Power Law Scalings
109
While the focus in the paper so far has been on data drawn from an isotropic distribution and on the proportional limit, stress that our methods ics of training structured data. For example, suppose that our covariates æ have the following covariance structure
10
10-2
also characterize the dynam-
c(t)
10--
L=1
10-4
1
(FLZZ) d y(z)
Σ wirk
(18)
10
1=8
10
DMFT
where the target weights w are arbitrary. Of particular interest are trace class covariances with > Ax oo which require developing a dimension-free version of the theory (Cheng Montanari. 2022: Bordelon al., 2024a). We develop approximate mean field theory using the same techniques. The error along the k-th eigenspace of the data (t)(t) = w Noo (t (f) follows the dynamics
10
1
10
n
(b) Depth μP Scaling B = L
Figure Learning rate transfer fails vanilla ResNets but suc- ceeds for scaled resnets, consistent with theory (dashed lines). (a)
constant raw branch scale The optimal learning rate changes by several orders of magnitude.
ResNets of varying depth
(b) ResNets with scaled branch 8 . with 3n = 1.25. The op- timal leamning rate does not shift significantly as depth increases.
(t) = Σ Hx(t,t) [w] - (t =t)())) P'C
(19)
where the Gaussian noise variables (1) and (1) are due to finite width effects and finite dataset (or finite batch) re- spectively. We introduced the transfer function Mr for each eigenmode & that describes the (biased) trajectory of 0s dynamics in the absence of these noise terms (see Appendix
Appendix C.I). In this case, the residual stream can be described as a stochastic process over layer time 7 = E/L driven by a Brownian motion term du(t, 7) and a feature learning correction that depends on correlation and response

---

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
10'
10a
10-
100
10
16
16
10-2
#=32
N=
10-2
W= 64
N- 64
W-
10
4
# 256
256
8
10
512
N- 512
10-1
1024
1024
32
10
N- 2046
N- 2045
64
10
10a
10
10 10a
10a
10
10 10 10"
10
10'
10
(a) SGD Noise N = 1024
(b) Easy Task
16
(c) Hard Task 16
Figure 6. dimension free mean field description of the dynamics captures SGD effects and finite width N effects for power law data covariates. (a) SGD with small batchsize B is approximately captured by the mean field theory for J. = 4. (b) Deep linear networks (L 4) trained on "easy" power law tasks with / > exhibit the same scaling law as in the lazy learning regime. Finite width networks eventually deviate from the infinite width limit with slower convergence rate. (c) Deep linear networks trained on "hard" powerlaw tasks the sense of (Bordelon et al., 2024b) exhibit a speedup their scaling exponent relative to their convergence rate the lazy limit.
E). This transfer function (t) contains finite width N and finite data P effects (but not finite batch effects for SGD). The test loss can be computed as ((t) = Ev Ab (+))) Of particular interest is the behavior of this model under powerlaw covariates
time (Ren et al., 2025; Cohen et al., 2021). While this effect should in principle be availabel from our existing DMFT equations, we leave a more detailed analysis of this effect in this model future work.
LR Transfer For Dimension Free Power Law Features While our learning rate transfer plots exhibited very sharp dependence on learning rate for isotropic features (Figure 2). we can develop more realistic hyperparameter trans- fer curves under our power law feature setting. In this dimension-free setting (D - oo with power law exponent a 1), the theory depends explicitly on N, B > 1. We illustrate that our main findings regarding learning rate trans- fer persist for power law features in Figure 7.
As ~ k- (wi)² k-a8-1
(20)
known as source/capacity conditions (Cui et al., 2023b; Bordelon et al., 2024a) Tasks with larger (smaller) 3 are easier (harder) estimate for the model. We illustrate the behavior of our theory for deep linear networks trained with SGD these power law tasks compared to simulations in Figure 6. Despite this mean field description being only approximate for small N, B, the theory begins to become accurate around N, B ~ 16 for power law eigenspectra. While a linear model would always follow a power law scaling with training iterations ~~ 1-8, we note that large width networks in the feature learning regime can obey the following power law scaling
Discussion
In this work, we developed an asymptotic mean field de- scription of the training dynamics of randomly initialized deep linear networks trained on random data. The theory was able to capture both vanilla MI.P architectures and deep residual neural networks and characterize the impact of parameterization, learning rate, richness parameter, width, depth, dataset size, and batchsize for online SGD. This is the first work to also numerically verify the asymptotic descrip- tion of training dynamics in infinite width residual networks. We demonstrated that our theory can capture the hyperpa- rameter transfer effect for properly parameterized networks across both widths and depths which have been observed empirically prior works. For structured and trace-class co- variates, we modify our theory to approximately capture the impact of finite batchsize B and finite width N.
20 t
1=3
B <1 1 (Hard Tasks)
lim N)t N-000
(21) B21 1 (Easy Tasks)
B
which agrees with the exponents derived by Bordelon al. (2024b) in the one-hidden layer T. - case. However, our theory shows that this improved exponent for the hard task regime due to non-lazy dynamics is in fact the same for deeper networks as well. This indicates that the structure of the task could induce some universal behavior in linear net- works of different architectures. We stress that this scaling law result is derived in the gradient flow limit of our more general DMFT, but edge of stability (EoS) effects could spoil this behavior at long enough training times in discrete
Limitations and Future Directions The current theory, while a signficant improvement in complexity compared to prior works which do not average over the random data (such as Bordelon & Pehlevan (2022); Bordelon et al. (2024c)), becomes costly to compute for very deep networks
$ Not to he confused with the a, 9 of previous sections. We this be consistent with the notation of (Bordelon et al., 2024b).
8

---

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
10a
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
10
N= 16
c(n)
10-2
N 64
N 128
10-3
== 1024
N 512
A Acknowledgements
DMFT
We thank Boris Hanin, Mufan (Bill) Li, Lorenzo Noci, Jacob Zavatone- Veth, Hugo Cui, Francesca Mignacco, Stefano Sarao Manelli, and Clarissa Lauditi for useful and engaging conversations. BB thanks Jason Lee for pointing discrete time effects in power law model would alter the new scaling law behavior on hard tasks after sufficient training time.
10
10
10"1
10
0
(a) NTK Scaling
that
10a
10a1
N=
B.B. is supported by a Google PhD Fellowship. C.P. is supported by NSF grant DMS-2134157, NSF CAREER Award IIS-2239780, and Sloan Research Fellowship. This work has been made possible part by gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence.
k(n)
42
64
10
128
256
10
1024
DMFT
10-1
10
10-1
10a
References
(b) pP Scaling
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint 1XXi:23033088744 2023.
Figure 7. Learning rate transfer experiments for L pP lin- car network trained with SGD (B = 128) on powerlaw features (a, B) (2, 1.75) and "o 0.4. (a) The hyperparameter transfer effect fails for NTK parameterization but (b) succeeds for μP and accurately captured by DMFT (dashed).
Adlam, B. and Pennington, J. The neural tangent kernel in high dimensions: Triple descent and a multi-scale theory of generalization. In International Conference on Machine Learning, pp. 74-84. PMLR, 2020a.
Adlam, B. and Pennington, J. Understanding double de- scent requires fine-grained bias-variance decomposition. Advances in neural information processing systems, 33: 11022-11032. 2020b.
trained for a large number of iterations T and very deep net- works > 1. We summarize the memory and compute requirements to compute our theory Table 1. Because of the L number response functions required to compute our residual network DMFT, the costs are higher than for standard non-residual network (which is linear in T.). Our DMFT solver is more efficient than full batch training in finite width N linear ResNet provided that L < N2P. In addition, the function approximation power of this model is quite limited as we are only considering deep linear net- works the present study. However, despite the failure to capture nonlinear function approximation, many of the dynamical effects from initialization, SGD, and parameteri- zation persist. Some future directions that could be analyzed within this framework include characterizing the impact of learning rate schedules and other optimizers on the dynam- ics, studying the effect of normalization layers, Hessian spectra, and characterizing how nonlincarity changes this theoretical picture.
Advani, M. S., Saxe, A. M., and Sompolinsky, H. High- dimensional dynamics of generalization error in neural networks. Neural Networks, 132:428 044,228-466, 2020.
Agoritsas, E., Biroli, G., Urbani, P., and Zamponi, Out- of of-equilibrium dynamical mean mean-field equations for the perceptron model. Journal Physics A: Mathematical and Theoretical, 51(80085002, 2018.
Aiudi, R., Pacelli, R., Baglioni, P., Vezzani, A., Burioni, R., and Rotondo, P. Local kernel renormalization as a mechanism for feature learning in overparametrized convolutional neural networks. Nature Communications, 16(1):568, 2025.
Atanasov, A., Bordelon, B., and Pehlevan, C. Neural net- works as kernel learners: The silent alignment effect. In

---

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
International Conference on Learning Representations, 2022. URL htttps://openreveeee formmmmmm? id-1NvflqAdoom
Dandi, Y., Krzakala, F., Loureiro, B., Pesce, L.., and Stephan, L. How two-layer neural networks learn, one (giant) step at a time. In NeurIPS 2023 Workshop on Mathematics of Modern Machine Learning, 2023. URL https: openreviee eew net / forum? ? id= BDcaRLhz2.
forum?
Baglioni, P., Pacelli, R., Aiudi, R., Di Renzo, F., Vezzani, A., Burioni, R., and Rotondo, P. Predictive power of bayesian effective action for fully connected one hidden layer neural networks in the proportional limit. Physical Review Letters, (33(2)007331, 2024.
Dominé, C. C., Anguita, N., Proca, A. M., Braun, L.,, Kunin, D., Mediano, P. A., and Saxe, A. M. From lazy to rich: Exact learning dynamics in deep linear networks. arXiv preprint arXiv:2409. /4623, 2024.
Bassetti, F., Gherardi, M., Ingrosso, A., Pastore, M., and Rotondo, P. Feature learning in finite-width bayesian deep linear networks with multiple outputs and convolutional layers. arXiv preprint arXiv:2406.0322 2024.
Dyer, and Gur-Ari, G. Asymptotics of wide networks from feynman diagrams. In International Conference Learning Representations, 2020.
Bordelon, B. and Pehlevan, C. Self-consistett dynamical field theory of kernel cvolution in wide neural networks. arXiv preprint arXiv:2205.00 09653, 2022.
d' Ascoli, S., Refinetti M., Biroli, G., and Krzakala, F. Dou- ble trouble in double descent: Bias and variance (s) in the lazy regime. In International Conference on Machine Learning, pp. 2280-2290. PMLR, 2020.
Bordelon, B. and Pehlevan, C. Dynamics of finite width kernel and prediction fluctuations mean field neural networks. arXiv preprint urXiv:2304.03400 2023.
Everett, K., Xiao, I.., Wortsman, M., Alemi, A., Novak, R., Liu, P. J., Gur, I., Sohl-Dickstein, J., Kaelbling, P., Lee, J., et al. Scaling exponents across parameterrizations and optimizers. arXiv preprint ur arXiv:2407.05877, 2024.
Bordelon, B., Atanasov, A., and Pehlevan, C. A dy- namical model of neural scaling laws. arXiv preprint arXiv:2402.01002,, 2024a.
Gerbelot, C., Troiani, E., Mignacco, F., Krzakala, F,, and Zdeborova, L. Rigorous dynamical mean field theory for stochastic gradient descent methods. arXiv preprint arXiv: 22 0.065911 2022.
Bordelon, B., Atanasov, A., and Pehlevan, C. How feature learning can improve neural scaling laws, 2024b. URL https: //trxia.avvvv///////a//eeeaaaav org/ abs/ 2409.440.. 17858.
Bordelon, B., Noci, L., Li, M. B., Hanin, B., and Pehle- van, Depthwise hyperparameter transfer in residual networks: Dynamics and scaling limit. In The Twelfth International Conference Learning Representations, 2024c. URL https:
Hanin, B. and Nica, M. Finite depth and width cor- rections to the neural tangent kernel. arXiv preprint urXiv:/909.05300 2019.
Hanin, B. and Zlokapa, A. Bayesian inference with deep weakly nonlinear networks. arXiv preprint arXiv:2405. 16630, 2024.
id=K2JehvRKGD.
Cheng, C. and Montanari, A. Dimension free ridge regres- sion. arXiv preprint arXiv:2210.08577, 2022.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint 1rX5v:2003.15556 2022.
Cohen, M., Kaur, S., Li, Y., Kolter, J. Z., and Talwalkar, A. Gradient descent neural networks typically occurs at the edge of stability, arXiv preprint arXiv: 103:::2033.0065. 2021.
Hu, H. and Lu, Y. M. Universality laws for high-dimensional learning with random features. IEEE Transactions on Information Theory, 69(3):1932-1996)) 2022.
Crisanti, A. and Sompolinsky, H. Path integral approach to random neural networks. Physical Review E, 98(6): 062120, 2018.
Jacot, A., Gabriel, F., and Hongler, C. Neural tangent ker- nel: Convergence and generalization in neural networks. Advances neural information processing systems, 31, 2018
Cui, H., Krzakala, F., and Zdeborová, L. Bayes-optimal learning of deep random networks of extensive-width. In International Conference on Machine Learning, pp- 6468-652 6521. PMLR, 2023a.
Kunin, D., Raventós, A., Dominé, C., Chen, F., Klindt, D., Saxe, A., and Ganguli, S. Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning. arXiv preprint arXiv:2406.06155, 2024.
Cui, H., Loureiro, B., Krzakala, F., and Zdeborová, L. Error scaling laws for kernel classification under source and capacity conditions. Machine Learning: Science and Technology, 4(3):035033, 2023b.
10

---

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
Lee, J., Xiao, L., Schoenholz, S., Bahri, Y., Novak, R., Sohl- Dickstein, J., and Pennington, J. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems, 32, 2019.
Roberts, D. A., Yaida, S., and Hanin, B. The principles of deep learning theory. Cambridge University Press Cambridge, MA, USA, 2022.
Rotskoff, G. and Vanden- Fijnen,, E. Trainability and accu- racy of artificial neural networks: An interacting particle system approach. Communications Pure and Applied Mathematics, 75(9):1889-1935, 2022.
Li, Q. and Sompolinsky, H. Statistical mechanics of deep linear neural networks: The backpropagating kernel renor- malization. Physical Review 11(3):03 1059, 2021.
Saxe, A. M., McClelland, J. I.., and Ganguli, S. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprini urXiv:1312.6122 2013.
Mannelli, S. S., Krzakala, F., Urbani, P., and Zdeborova, L. Passed & spurious: Descent algorithms and local minima in spiked matrix-tensor models. In international conference on machine learning, pp. 2019.
PMLR,
Vankadara, L. C., Xu, J., Haas, M., and Cevher, V. On feature learning in structured state space models. In The Thirty-eightt Annual Conference on Neural Information Processing Systems, 2024.
Martin, P. C., Siggia, E., and Rose, H. Statistical dynamics of classical systems. Physical Review A, 8(1):423, 1973.
Yang, G. and Hu, J. Tensor programs iv: Feature learning in infinite-widie neural networks. In International Con- ference on Machine Learning, pp. 11727-11737. PMLR, 2021.
Mei, S. and Montanari, A. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 75(4):667-766, 2022.
Yang, G., Hu, E. J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J. Tuning large neural networks via zero-shot hyper- parameter transfer. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021. URL https: / / openrevieww net/ forum? f 6=Bx6
Mei, S., Misiakiewicz, T., and Montanari, Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit. In Conference Learning Theory, pp. 2388 2464. PMLR, 2019.
Mignacco, F. and Urbani, P. The effective noise of stochas- tic gradient descent. Journal of Statistical Mechanics: Theory and Experiment, 2083(5), 2022.
Yang, G., Yu, D., Zhu, C., and Hayou, S. Tensor programs vi: Feature learning in infinite-depth neural networks. arXiv preprint arXiv:2310.02244 2023.
Mignacco, F,, Krzakala, F., Urbani, P., and Zdeborová, L. Dynamical mean-field theory for stochastic gradient de- scent in gaussian mixture classification. Advances in Neural Information Processing Systems, 33:9540-9550, 2020.
Zavatone-Veth, J. A. and Pehlevan, C. Learning curves for deep structured gaussian feature models, 2023.
Zavatone- Veth, J. A., Tong, W. L., and Pehlevan, C. Con- trasting random and learned features in deep bayesian linear regression. Physical Review E, 105(6)064111, 2022.
Noci, L., Meterez, A., Hofmann, T., and Orvieto, A. Why do learning rates transfer? reconciling optimization and scaling limits for deep learning. arXiv preprint 2rXiv 2024.
Pacelli, R., Ariosto, S., Pastore, M., Ginelli, F., Gherardi, M., and Rotondo, P. A statistical mechanics framework for bayesian deep neural networks beyond the infinite-width limit. Nature Machine Intelligence, 5(12):1497-1500 2023.
Paquette, E., Paquette, C., Xiao, L., and Pennington, J. 4+ phases of compute-optimal neural scaling laws. arXiv preprint arXiv:2405.1500
2024.
Ren, Y., Nichani, E., Wu, D., and Lee, J. D. Emergence and scaling laws in sgd learning of shallow neural networks. arXiv preprint arXiv:2504. 19983, 2025.

---

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
A. Derivation of Mean Field Equations
In this section, we provide mathematical derivation of the DMFT equations that govern the train and test loss dynamics of the deep linear network. We will start by analyzing the non-residual architecture trained with full batch gradient descent and then proceed to study online SGD and the residual network architecture, which involves a larger of cross-layer response functions in the residual stream.
A.I. Non-Residual Deep Linear Network Trained with Full Batch GD
A. 1.1. ISOLATING DEPENDENCE ON THE DISORDER
We start by considering the feedforward depth L network with no residual skip connections defined
VD Noo (w") II E=1
L w*)
(wo x.
f(x)
(22)
In this section, we consider weights are updating these weights with gradient descent on the empirical (training) loss
(, = yn [(xx,)) 1 P Σ (0)" E(() -P0PPEO0
1
VD w By Dep [(x)))
(23)
The update steps are performed using a raw learning rate of Ning following Bordelon & Pehlevan (2022), which gives
wt + 1) 1- Ni w L()), (1++) w Nn E (0, I. 1}-
(24)
Using the chain rule at each step f, we can decompose the weights at time t into their initial values and all of their updates
(1 ( +
770
Σ
VD
((()) = W'(0) 1Vv0
Σ g
VN
P'
w² -- nioo
Σ (t'(t)
(25)
t'
where the vectors (tt), E N that we introduced defined in the following way
VD
A() 1
VD
(1) = P
(()
Xx(()
v(1) = w.
w (1)
VD
No0
(())() VN
('())'),, LE 1.
--
g (1) (t)
1 gf le ( 1- VN
(26)
In the above definitions, we have introduced the data matrix X E PxD and the noise in the target values c E R.,. We can now isolate the dependence of these vectors on the random initial conditions (0) and the updates from gradient descent. Expanding out the weight dynamics, we can close the vt), gf (1))EE dynamics as
(()(() (1) + mo Σ
(f (1')
(1)) (f) 1770
oftt(t
Pst
vD (0) T (1)
v(t) w.
Σ
(C((E))))
Noo
Pa
12

---

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
where we introduced the following correlation functions
1 C1(,1)) =
= 100
(2(,,)) 1 N0 ('()
(27)
and the following fields that depend on the initial conditions (0)
(t) = vD (0) g (t) = VD (o)
Noo
x (1) 1 W (0)h (1). (1
W (0) (1).
(28)
VN
VN
These variables isolate the dependence on all of the random initial conditions for the weights Combined with the defining equations for the variables (t), v(t), (t), we will now aim to characterize the typical case dynamics over draws of the disorder D
== (X, e, (O), (
(29)
Our goal is to characterize the typical case dynamics of the system over random draws of DD
A.1.2. COMPUTING THE DYNAMICAL DISTRIBUTION OVER DRAWS OF THE DISORDER
Moment Generating Function We now aim to characterize the joint distribution of vtt), (1), (t), (h'(t) ( HL) over draws of the random disorder D. To do so, we will start by calculating the moment generating function of these fields with Martin-Sigee Siggia- Rose integral (Martin et al., 1973) over the trajectories
1)); -30)
Zliv.s, (ine fg bee[])]
exp Σ
j.()) o() (0())<())+)) +5a(1) A(I) + ESn()) h2(1)) + +0g (000++0000000 o0))
(30)
This object, once computed, enables computation of arbitrary moments from derivatives with respect to the j variables at j 0. For example, the two-point correlation between the v(l) variables has the form
0²
(vr(O)n(t))) (i (1) Al.. Jastiin Igt hesuyl
(31)
Why No Replicas? At this point, we stress that the dynamics for our variables vtt,, (tt), (t), (t) provide complete and normalized joint distribution unlike in static (replica) mean field caleulations where the distribution of the variables of interest are only known up to normalization constant (eg, where one must perform quenched average of an observable (0) for a Hamiltonian H0, D) which depends on disorder D: [
expl -xp(-88((0 (1)00
where we see the disorder appears in both numerator and denominator). This form of average is usually tackled with the replica method. In our case, this is not necessary since Z directly captures the distribution of our trajectories over draws of disorder D.
d0 exp( 3H)(,D)
Performing Averages We now aim to compute by averaging the dynamical trajectory over the draws of D. To make this concrete, we represent the transition probabilities (conditional on D) with Dirac-delta functions that encode the dynamics. We then represent the Dirac Delta function with Fourier integral for each timestep as
1 = f (1)) A(1)
Xx(1)= -oE)
de (t)dd((t) /
(2T) exp ii(1))
A 1
Xx(1)= - oE)
VD
VD
= f 1- (1)6 h(((1) - VD (PXT(())
VD
= dh (n (2T)D
h²(()) - VD
x A()
exp
(
(32)
P
Performing this for all of the relevant fields, we can begin to take averages over the terms involving each of the random matrices the set of disorder D.
13

---

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
Data Average
The average over the data matrix
X involves terms of the form
VD A (t)u(t)
- P
exp
-1 Trx
(33)
exp
(L) (1))
D 0() =(t')
D
Σ h())) ((())) 1 p
} (())
(34)
2P
Ca(t.
-AA o()) -0) C e (P)
exp
(35)
P
ID RD. (3.6
The resulting underlined terms reveal the following correlation and response functions induced by averaging over this fixed data matrix
Co(l, 1
D v() v(()) Ca(A()= - (1) A( i (t) A(()
- o(())= R (1, = ())
i ((),,, (a((, -
(36)
These correlation and response functions will subset of the order parameters the mean field theory theory (more will arise from other disorder averages). We now introduce delta functions to enforce the definitions these order parameters
(CC(t,))dC.utttt 4nD-1
( (2 1 (G( (00()
1
exp
(37)
/
dds((, 4TP-1
(Fos(e)))as(1)) - (stttt) A(())
exp
(38)
For the response function, the conjugate response function to R is RA and vice versa so it suffices to just enforce the definition for one of them. For concreteness, we can enforce the definition of (1,
/
/
dR (1, (t',)a exp (-DR.. (t, (t)tt))) iv(t) (()))
(39)
2T
First Layer Weight Matrix We now examine the behavior of the first weight matrix (0) IR NxD which is rectangular, For this matrix the two relevant dynamical fields are
1 (o))))) (t VD
VD (t)
x²)
(40)
Noo
Again, we introduce Dirac delta functions for these fields and need to average
(1) + VD
1) ()
exp
-4 -
(41)
Noo
exp
x(() (1) 1 (t)())((())) 10(t))
D
2N
D
Σ (() (()) 1
N gg (t) g((t)
(42)
1 (t'(0)-'' (())) (C))
X exp
(43)
NOo
(t.0"
14

---

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
where we introduced the order parameters
C) = (1)) ((),, C((,)) = g' (t) (1')
1
(44)
= N (1) x (1), Rer (1, 7 (1)))
(45)
As before, we introduce Dirac delta functions to enforce the definitions of these variables.
Hidden Layers Lastly, we examine the disorder averages for the hidden layers (' E R NNN which are square matrices. The random fields that depend on the random matrix (0) are
X = w (0))(()) (t)) = W'(0)T
(46)
Averaging their trajectories over the disorder (0) gives us
i Tr (0) T Σ x
+ +0 ()
x
exp
1 Σ x (1) (() 1 (
- 1 Σ (v) 1 F (e)
exp
2
N
"
CRAMM
C (t.1*)
exp
1
(0) (t) (())
(47)
N
IN A. (v,.)
where the relevant order parameters
()(,t)) = N (t) n(t),, (t,)(,))) 1
N (1) g0)((())))))))))))))))))))) i (())), R (1,1) = N (1) g() xx)x((((())
(1, (1) = - - ((())
(48)
As before, we introduce these order parameters with dual Fourier variables.
Proportional Scaling Regime We notice that many of the terms that arise in the integrals depend on ratios such as f and 5. To deal with these terms we introduce definitions for these ratios
P a D ンー= D'
N
(49)
One can interpret, at fixed D, as a and y representing an increase in the quantity of data or the width of the network respectively. With these constants a, introduced, can now consider how the moment generating function Z behaves as D - xx. However, before this, we must organize and simplify the remaining integrals to enable taking the D - do limit.
Decoupling The Single Site Integrals and their conjugates
Now we aim to express the integrals over the variables A (t), v(t), x (1)
8 dA
(1) (1)
Á (1) (())C,(,))) ΣÅ(!) ( - 17 - -Ru(())))((()
exp
P
50s A() E CC(AOAA) (())))(())))))))))))))))){)))()) A(5)
x exp
2
W
P
ZA IC., Co. Ro مA (t))
(50)
15

---

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
where we introduced the single sife moment generating function for the A(I) fields that has the form of integral over single entry (scalar stochastic process) of the original (t)) vector
da(t)dā(t)
2 Σ ((A +i E ((t) (A(+))(t t) (Rau(t,
Z =
exp
2m
t=0
[Es(A)(()) (CS((())(()))())
x exp
(51)
W
From our disorder average, we also see that there is statistical coupling between utt), (t) and (t) variables. We let the joint moment generating function for this set of variables be 2o (for layer l 0), and express it
/
(t)
20
4
t=
1
(CCA(())))))
27 EEWEOGAU
exp
20
(((1) (5(1 11)0(1)) (a(G(()((1)) + Ein
exp
(t) 5(t (t) Pt (t.ttt)))))) ESn())((t)
cxp
To
II 8 ((()) w² + (1) + (C(Ut)))))
(52)
For the first hidden layer (8 = 1), we have the following joint single site MGF
2, / II dx dx' (Od
4
Σ ((0
1 Σ -eoeeteege)))
exp
2
2
te'
#
Ext() ((1--1)x((() 1 (((((00))))))))
exp
w
Σ (t) -66 -Rg (2
exp
Σ (1) (tgg
exp
IIa 8
(1(() (x(()) 770 E 6 (0((0)
- Ecgeeea'' 1'
177
(53)
P'
16

---

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
For the hidden layers E L2-.. 1}
have the following single site MGFs
(t) (haa
2:
4m
(41) EE )
Σ
exp
2
W
w
Σ
(t,
exp
10
Σ (5(1 (() - ret (1, (1,)))11))
exp
Σ (1) + (1)(
exp
II 8 ((() x() 1 Σg "h 6 (0(0)- 50(1) - 10o Int ((()(())))) p<t
17o
(54)
t
P't
The final layer e = L has the following single site MGF
(t)xx (t))))
2x
Σ (6) E - - (1,))))
exp
II
(+(() x²t)- 1 Σ ch (t:)(()))))))))
Pet
g0((t) (0) 7700 Σ Σn' chi(tt)
c'
(0)0)((()
exp Σ (t)h (t) + (t)+ (t)g
(55)
Now we have reduced all of the field path integrals into single site moment generating functions.
DMFT Action We now group all of the correlation and response functions, as well as their conjugate order parameters into a list q and combine all of the source variables into list j. The moment generating function can be written as
Zlil = dq exp (-D8
(56)
where S is an O(1) DMFT action which has the form
1 8 2/2 Σ (c.t.
11
Rd (t, (t,t') (t', + to
Σ
Ro, (t,tt) R (t', +
R (t, (f)
12'
b
N
In Za [لاد - D
Σ
Zoljio.es
Zz is
(57)
D
D
#=1
k=1
t=1 i=1
17

---

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
Saddle Point Equations We are now in a position take the D - limit. In this limit, the integral over is dominated
by the saddle point of S. The resulting saddle point equations 8q conjugate order parameters for the correlation functions which give
as 0. We start by differentiating with respect to the
as
tt(
1
Σ (o(t)((t)), =0
(t,
2D
0S t') + (1(()))())) =0
(C)(t.t)
2D
0s
1
(a(t,t))
(A())(())) =0
(C'(t,
2D
as
1/
N
(M(O))())) -0
(L,
2
2D
as c(t,tt) +
2D (F(()) =0
(58)
(C)(), 2
where 0r represents an average over the single site distribution defined by the moment generating function مهد
Similarly 0x Q2 denote averages over the distributions defined by the single site moment generating functions these averages all become identical (we will
Zo[ja], {Z E-1- We note that in the limit of zero source amine this zero source limit shortly). Next, can work out the saddle point equations for the response functions
as
(a(t,t))
i Σ
((H)(())) =0
OR (t',t)
as
Rou(t,t))
+ i
(())) =0
(t'
D
k=1
0s
To Rl (t,tt)
(1'0E(()))
R hr
(t',t)
Nvo
as
%o
1 (t (t,t')
+
(1)))))))) =0
ÖRI gu (t',t)
D
as
1 gu
+
9 (t) (()) =0
OR for (t',t)
D
OS
vR (R (t, +
(N(())²())) =0
(59)
i @R (t',)
Lastly, have collection of saddle point equations 0s 0 that define the conjugate order parameters Ò = 0 which must
DC
vanish at the saddle point (Crisanti & Sompolinsky, 2018; Mignacco et al., 2020; Bordelon & Pehlevan, 2022). Thus have the following remaining set of equations
0,
C,(t,t))) (f(t,t) 00 C)(t,t) O.
(60)
Hubbard Transformation Now that we have argued that the correlation and response functions take on deterministic values in the - oo limit, we can represent the quadratic terms in log-density in Å, X. fields as linear averages Gaussian
18

---

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer random variables us "a()), of (1), giving
(0app
Σ (M
- Σ Am)s(()
cxp
cxp
uaNN(o,C.))
((
Σ I
exp
exp
20x
"e-N(0,0-
- Σ Ex
exp
exp
uENA(1O,C) 9)
1
Σ IEORT)CA)
-(oxp/ -i E
<exp
exp
exp
278v
Σ (0 (1.1)
exp Σ ( (-N((,,5HH)
exp
(61)
After introducing these Gaussian random variables, we can now perform integrals over the Fourier variables A, ho Xx, غ2 to obtain defining equations for the random variables of interest. For example, for the A(t) random variable,
dt(t)
A(t) (t)()= - Σ R (t. (t, (000)))) E ((t)
exp
1
2
A(1) ua(l) Enenl {(en())))(())))
8
(a(t)-/(o.c..)
At) = (t)++ + Σ Re ())(),, (a( ((0,
(62)
We can repeat this for all of the remaining fields which give defining stochastic processes for them. The general result is that the random fields are driven by noise component (like usa (t) above) as well as a memory term involving response function (like R (t, t') above).
Simplifying the Response Functions We note that the response functions involve averages over the A, e variables, which we now argue can be viewed as derivatives with respect to the Hubbard variable. For simplicity, we will now examine the zero source limit where all averages are identical.
(t(t,t) = -iittt))=i(((()))) ((()(())()))))
= -i / ((())((()
R (1, (1)(()))) [) 1)
A())(())
exp
i
Σ A(1)
A(I) ua(() - E
2T
us A(t) - -ua(t) - -100-Emm Rd (t,
t
/ II (t)))d&(t)
2T
A(t)
(us(t)
cxp
Σ (t))
AA(L)
Σ (C
(63)
Dua
(t)
MA
which holds via integration by parts and Stein's 's lemma. Repeating this for each of the fields, we find the following response function definitions from the saddle point equations
RA(L,1)= = da(()
Ou(()
(1)
(1,'
Thh (1,
Ous(())
On)(())
0))((()
(t)
dg'(t)
(t,t') =
R (t,
(64)
(r'(t))
Orl (t')
19

---

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
A.2. Final DMFT Equations
Combining the above equations at zero source j 0 after our Hubbard transformations, we arrive at the final single site equations for our fields. The DMFT equations that describe the single site stochastic processes are
<(1) n' - (1) - Σ
- Rig (1,1) + (L)))( (²((), (1) GP 0, G 70
A()) - =a( + Σ
(1, (t, to, ~ ~
(0,)), ua(t)= N(0,1)
Mat
Ra(t,t)oott) (t) GP (. 0, 1 c2)
(t) & (() + Σ
P't
= = (t) + Σ
Ri (t,t)) To (t. g((()) (P(t) C
10(1) d(()) + Σ
[ (t, + C,
(P() +
(t) = (t)) + Σ
[ (t, t') + mac (t,t') (t) (() ~ GP(O, c
(65)
From these equations, all necessary response functions can be computed by differentiating over the noise variables {u, r}.
(L
Ov(()
/an((1)
Ra(t,t')
(t,
(t,''
(us(t')
(v)
(1)(())
(t)
(g'(t)
(t,t)
R (t,t')
(66)
(t'(
Out(tt)
Similarly, the correlation functions can be computed as averages over these noise variables
Ct(t,t)) (o(t)u(t)) (((())))((((((())))) -((t,t)) (1'(()) (',), (g'(()
(67)
We note again that the test and train losses can be obtained from the time-time diagonal of the correlation functions.
c(t) = C.t,t) + L(t = (t,,
(68)
A.3. Centering: Subtracting the Initial Predictor for Lazy Limit Stability
At finite values y our theory will diverge if the lazy limit Yo + 0 taken. This is due to a divergence of the predictor variance (t, 1) which can be eliminated with centering operation. Under centering we define the vector v({) as
vtt = Et) + Σ (tt,t'))))))) Pett
(69)
The key adjustment to the DMFT equations from this operation is
vtt) w. - ( (
1 Σ 70 est
R (1,8)))(())) - Σ CLCEE)))
The correlation functions that need to be tracked are (C., response functions RA, R Rer...., R R.. Ryw..., RL P
hi hi ..., ch- C. C2 gg- ..., C while we need the following
C
20

---

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
B. Comparing Parameterizations
In this section, we briefly comment on how the maximal stable learning rate scales with NTK and in mean field/μP parameterization. NTK parameterization defined as scaling the network that "oo + of this choice is that the change in the correlation functions Cg (t,,t' under fixed number of steps of optimization will decrease with "o at fixed learning rate. As a consequence, higher learning rates can be tolerated at larger widths. However, once the network is wide enough to be sufficiently close to the kernel regime, the maximum stable learning rate stabilizes as the dynamics become asymptotically linear. A comparison showing this trend is provided in 8.
ve However, consequence
y -
NTK
MF
Mmax(u) (V)
10
10"
10
Figure& 8. Maximal stable learning rate function of width v for different parameterizations.
For very rich dynamics " >> 1, the optimal learning rate approximately scales as
10 (1) Mmnx (0 (w((t)
MF (Fixed 7o)
(70)
(w--t) NTK (Fixed /o))),
based on a simple argument about the scale the kernel. On the other hand, in the lazy learning regime To < 1, both MF and NTK parameterizations approach constant optimal learning rate as they approach linear models.
C. Residual Networks
In this section, we consider training residual networks. To streamline the calculation in this section, we will first compute gradient descent on the population risk (incorporating random data can be handled by introducing the A(!) variables as in the previous sections). The key field definitions have the form
h1(() w- VD
(T())
NoN
B
*11(1) = 1²(1)) +
(1)(((())
VN
g0(() = (0) + moo Σ (t)
P'
(0)=-+1(0
(') g (t).
(71)
N
21

---

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
Introducing x (1)
1 VN
(O))((()) and (1)
- VN
g these obey the following DMFT single site
((),
dynamics
(t))= (tt + Σ R (t,)
Pct
(1 = (1) Σ Rgt (1, (t)
(72)
The full gradient descent dynamics can thus be expanded as We can solve simultaneously for correlation and response functions
(t) w² - (t)
Ri (t, + tt
(73)
70
(t()) (t) + Σ
Rd
(t, + mo g1(t). ([t)~gPPp GP())
177
(1) (1(1) + Bu (1) + Σ
(1), (11())9P((0)
P'<
('()) = (1) + mo Σ (t))) ~ ((())~P((0.)
Nec
(t)) = ( (t)
Σ
[ (t, t¹) +
(()) CP(0,CT-1)
(74)
v't
To obtain the response functions Ri, (1, 1'), Rd Rgu(t, (t, 1'), we actually must compute all cross-layer response functions which form a closed system of equations as
Th (t) (t)
0g' 0t"(t")
ōe.1ō(t - +')+
Σ
R (t,"t moCR(t,tt)
Ou'(t))
oh (t) (r)(t))
0g' 0t'(t")
Σ
Re (1,1") + 17 11,0 (3)
k E (1...., L}
())((())
VYO
C'
(t)
(t)
(t, t") mo (1")
+
Σ
e [h,.., (u)(
(1)
Our(t)
(1)
al
(t"+
+ 33) Σ
(t, (() + ("0
E E {1, L}
01)(1))
0r+(11)
(1)(1)
17u Σ (t"
(t)(t)) "a
(ty((t)
(t)(t)
(t")
g - -t') + mo
Σ
(r*(1)
(t(((1))
(1) -1)+1(1)
Th (1") Ou
(t) (t)) + 08t +t')+ Σ
R (t, + 1703C4111
qu
p"
(t)(+') = (g)+(()
(g'(1)
Oh Oh"(t") (t')
+ (8 -t) + B
Σ
[ (t, + m
(t)(+))
After these response functions have been solved for, we must solve for the correlation functions using the single site densities. Concretely, we use the above recursion to obtain the f = & diagonal elements
(h)(1) ( (r)(
)
R (t,t' (30
agg(1) (ul(())
(t.
(75)
22

---

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
Using the above response functions and invoking the linearity of the dynaimes, we have the following decompositions of the hf (1), (t) fields
(t)
(t((t)
(t)))
Σ
(t))
((') (())
(76)
(1)
(rt((T)
ket
(t)
(t) ((())
g (t)
E
Out (t) w (t))
(77)
(r((())
kd
We next compute the correlation and response functions (1,)))) and (1 (1) which have the form
th(t,t)= = E (t) (t' c (s, (')
(t) (t'
Ck+ (s,
(78)
(u) (s)
(rk(s) (r*
kk'ss
kk's
0g' (t') Chi- (s,
(t)(t) 0g
(t,tt))=
Cg (s,
(79)
(s) (s")
(s) Ork(s')
kk'as'
where the base cases are determined by
Ch = Σ H(t,s))((1
1t C2(. und
(80)
where the transfer function H(t, s) satisfies
H(t. 8) 5(t - 8) +
Σ
1
(t,lt, (1, +) H(),
(81)
Yo
P't
The loss can finally be obtained from the f, diagonal entries
C(t) (t,
(82)
We see that the formulas for the response functions depend on the correlation functions and vice versa. These must all be solved for simultaneously.
C.1. The Large L Limit
The infinite depth limit with 3 = Bo/vT. can be interpreted as a collection of ODEs for the response functions and a collection of ODEs for the correlation functions along the hidden layers.
First, we note that under the scaling 9o/ LL, that for any 1, the responses scale as
oh (t) (t)
Bu VL
(t)
Buu
θ
θ
(83)
(t)((t)
VL
and similarly that for all &s L, the response functions for the n.t fields scale
@h((t) θ
(g θ Bo VT. 0,"
( VT.
(
(84)
)r*(1))
To take the large L limit, we introduce notion of "layer time"
L E [0.1]
(85)
and we recognize that the finite L ResNet update equations approximate a SDE with a discretized process over 7 E [0, 1] with step size dr 1/L. We define the limited
A(T,,) () = lim 1(())emtr ((T,1) lim g')eet L
(86)
23

---

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
The functions aim characterize in the limit the limiting correlation functions
(,,,,,00)) = lim (lee)r C)(T,, = lim
(87)
L-000
E-000
and the following (1) limiting response functions
VL
VL R
Bhr 1,1) lim -
lim R/ (1, (1) Rgu(T,
Ri
(1, t') le
(88)
Bo
L000 90
With these functions introduced, the single site equations in the "body" of the ResNet take the form of coupled SDEs
dh(T, t) Bodu(T, t) dg
dg(r,t) - d
(89)
#'c
The noise sources these SDEs have the correlation structure
(du(r, t)du((', ,t)) = 5(7 - t'')d'''' (dr(r, ()d((t))rrrr. , .t) = 5(T - t, d'dddr'
(90)
The h SDE is be integrated forward in time from to and the g SDE is be integrated backwards time from to 0. The necessary response and correlation functions can be computed from this SDE
()(T,()
Ög(T,t) t)
(T,
Rg
Ödu(T,t')
adr(T,t)
tg(
0g(T,t) adr(T,t)
RA, @dul(T
(T,,t)
Cx(T,1,(')= (M(T,)))(T,())) C,(T,1,1')) (o(T, ()g(T,
(91)
However, we note that the single-layer response functions do not actually close, but rather, all possible pairs of layer-times must be considered, ie we must compute the full collection
ah(T, t) ()du(r',tt))
0g(T,t) adr(r',
(T ,,,,",
Rgu (T),
(92)
These four response functions form set of ordinary differential equations
0, Rra (T, 5(t - t')
Bo Σ
[Rrr(T,t,t) mooCh((T,ttt tT,t",
0, (T, (T,
Σ
Rh (,,,t"))) 170 (
p'ct
0 RT -Bu
Σ
R moa BT
1" <t
0 Rgr -ō(t t') Bo
Σ
(93)
eret
Again from these response functions, we can again obtain the final correlation functions,
Cx(T,t,t')= = = E dr'
d' ( + t.
C,(T,t,t)) Σ (/' dr' tt,s (.0 (', tts
(94)
and the test loss dynamics have the form
L(t) = C,(0,
(95)
24

---

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
D. Dynamics of Online SGD
In this section, we characterize discrete time online SGD where at each step 6, a full batch data X()) BR are sampled. This results in the following field definitions
VD B
1
h (t))
X(t) (t) vo x(t))(()+
(96)
where all other variables are defined as before. Averaging over the samples drawn during SGD induces the following
(t) = u (t) + (()) (²(() N (o. - (t, 6(t 11)
(0.
(97)
aB
(t) (t) + oe((), (t ~ (N(,,,)
(98)
The key differences between online SGD and full batch GD training is
1. The noise u is local in training steps (decorrelated across time)
2. There is no nonlocal response function R& (1, (') that governs the buildup of the gap between test loss and train loss. Instead, the variance of (I)()))) = Ct,t, + Ca (t, (f, t) is simply shifted by the SGD noise effect.
The rest of the DMFT equations for (1) (1) (1) are the same as before provide below
D.1. Full DMFT Equations for Online SGD
Below we provide the full set of DMFT equations for the online learning setting where batchsize B, width N and dimension D diverge proportionally
D,N, B B N
(99)
In this limit, we obtain the following stochastic process for (o(1), ((), (b (0)L (1))
(1)) nn - ( Σ [[' R (t + 11 (1()'), (1) -gP (1, c(((t))))
P'c no
A(t) = (t) + ac((), (1( ~ GP (0, ( N(0, - -t))
(1) ((()) + +o((), (1) ~ GP 0. aB
h (t(t) u (t) + Σ
1 Rh (t, 1¹) + +m
1/70
Pet
W((() = ('(t) + Σ
[ (t, + mo ((00)000)))))
est
(()=)(()) + Σ [ (t,t') + m.c (()) Gt (t,
(100)
P'tt
where as before the response functions are defined
Ou(t)
(t)
R (t,t')
(t,t')
(tu
al6(1)
Oig (1)
R (t,t))
/
(1,t))
(101)
Or
(ul((t)
and the correlation functions are defined as usual
C.(t,t') = (o(t)u(t')) = (A(t)A(())))
((1,)))(((()))))) =
(102)
25

---

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
D.2. Analysis of Early Portion of Online SGD
From our theory and simulations, see that for sufficiently large learning rates and sufficiently small width and batch size, the test loss can initially increase with iterations t before subsequently recovering and converging later time. In this section, we analyze this effect approximation of the early loss dynamics. Neglecting the memory terms that buildup over longer training times, we make the following approximations for small
(t,tt m7 t) = - (17
Σ t + 1) (L,)))
~ 1F
(103)
1=1
which can be justified by working out the early dynamics for the response functions at small time. Under this approximation, derive the following recursion for the test loss (t) (u(t)?), which has the form
12 1)) (1 In))² +
L + (1[L+1)2L++))
L 1)(2) + +1) (t()).
(104)
0B
6v
6VGB
We verify that this does accurately predict the first few steps of the loss dynamics (and captures the early blowup) of deep linear networks in Figure 9. Under this early time approximation, the loss will thus increase if
n
(105)
(2L+) (2+1)(2L+1)
6Lv
GLoge
Indeed from simulations, that the early dynamics diverge for sufficiently small ag and
p-0.22
3 30
0.250
210)
0.50
2
1.000
Approdmmae Dynsmics
10a
10
(a) L = 4, 0.25
(b) L = 4, ag = 0.05
Figure Early dynamics online SGD can exhibit an initial blowup followed by recovery to 0. initial blowup approximately C(t) [(1 nT) + r7 (oz" + + + 20 g w= ==')]', which is function of learning rate n7, depth I,, batchsize
and width v. Small width or small batchsize can make this effect more pronounced.
E. Online SGD with Structured (Dimension Free) Data
In this section discuss the effect of structured data the dynamics. In this setting, allow NN(0, with TrA < so in the D + oa limit. Without loss of generality, we let A be diagonal with entries Art As but allow for arbitrary w" In this case, the proper definition of the variables ho and (tt) are
(t) =
B x(t) (t)), (t)) X(t)o(t) + oe(()
(1(() = (0)h v(t) = w'
(t) (()) + (o)
(106)
Noo
Noo
26

---

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
With centering, this gives the following mean field description
( un (t) + oc(()) ub(() N (0,, de caa(t- (.. "A ~ ~N(0,
(t) (t) + (t)x(t)
uy (t) u - + +
Σ Mer
(t Ro (t, nC (t,)) kg(t), rk(t) (((t)~N( N78 C
(107)
70
The definition of the correlation functions for these fields changes to
80 tt,,,t)) = Σ Ae (vx(()) Ch( Σ (R(()))(EE)
(108)
while the response function for the 1.2 and variables change to
(ng(() arl(e)
(vx(1) Ong(t))
(1,
Rom(1,)) =
Ak
(109)
which are both finite due to the trace class condition Σx Ae < cc. To express the test loss, can introduce the k-th mode's transfer function Ha (t, t') which has the following definition
(x(t)) Σ (t, P'c
-
wx
Σ 1: R (t, + (t, u2(t") C"CK"
(110)
Hx(t,t) = 8(t -t'- N
Σ
1o Ro + (t, (a((",t))
(111)
"'
The other hidden layers of the network have the same single site dynamics as before,
n²(1) = =0) w'( Σ
Noo Rhr (t, (') + 7700%(t g/ g)((), x'(0)~((,CC))
Met
18(1) = v((() Σ
[R (1, + mo -,,0 (10000000000000,0000000000000000
PEt
g'(() = (())=>((() + moo Σ (t')
Pet
g(1)= - -00 + Σ [ (t, + (7,0 (4,)))((()) "(00)
E.1. Scaling Laws for Easy and Hard Tasks
We now examine the scaling law behavior with respect to time for the N, B + oo limit. We also study the small learning rate regime, where we can study a continuous time gradient flow dynamics
(ey R - (t)ox(t)
(112)
where K(1)= x (L () + + 2 R) (1, is related to the Neural Tangent Kernel for the model (Jacot et al., 2018). We make the following ansatz for the small 70 dynamics for K(t)
K(t) 1 + 3xx
70 - 0
(113)
where the above is understood to disregard prefactors constants. Following general arguments of Bordelon al. (2024b), must try identifying solutions to the self consistent equation
(t) [1 + - (Ju((t) vx(1) exp (-Ault + 10 wi
(114)
27

---

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
10
=
10-1
0.33
0.50
0.66
0.83
= 1.00
10
(-2011+
100
10a
102
10a
t
Figure 10. Scaling laws for the hard task regime are well predicted by the adjusted exponent.
Following the self-consistency argument, we find that the kernel must scale as ((t) t1-Ax, Combining this with the assumption implies that
2
X
<1
(115)
1+3"
3
This implies the following loss scalings
B> 1
C(t)
28 1+3
(116)
8<1
which we show in Figure 10. The case where < is known as the "hard task" regime while 3 > is the "easy task" regime for linear networks trained on powerlaw features.
F. Different Relative Widths
We can also handle the case where hidden widths {Ne differ across layers
f(x)
DD
w" II
L-1
wx)
(1) w
(117)
NL70
VNe
E=1
In this case, the proportional limit of interest is
N₂ Vt
a
(118)
D
D
We can straightforwardly work out the limit using the same techniques as before.
G. Multiple Output Channels
The case of multiple (but (()) output channels N₂ can also be handled within our formalism. The cost of this analysis is track correlation and response functions across all possible pairs of output channels as well correlations across timesteps. We let the output channel have readout w as before
Sc(w)
II
w:)
(0) x
(119)
Noo
VN
28

---

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
In this case, assume collection of target vectors w. for {1,.., Nej. In this case, the error is governed by the following weight discrepancy
(e(() = w 1
(T
N%
(t)
('t 9e (t), (t) w
(120)
VN
The dynamics for the weights are
(t+ (t)+ + 7700 No
(t)
VD c=1
W'(t 1 ((() 770
9 (D))
VN
/ (t + u. (t) + 77)
(121)
where the variables (t) are defined as
VD P
h ++ (t)
1 w'tt h2(()) N
Σ
(t)
(122)
P
Expanding out the dynamics we find
(((() = xt( + moo
che (')g))
g (1) + 17o0
Catt (1, (n)
(123)
t'c
As hefore we can carry out the DMFT computation but the order parameters that we need to track include cross-output channel correlations
Co (t, = (ve(t)ee((t)) C (t, (A)(t))A
Ch Cheac (t,t'= (H((t))))tt) C "t,eett, (S((t)
(124)
as well as cross channel response functions
Sv.(t)
(t)
ov (t, f)
RA (t,t')
(1))
(U))
(1)
(g)(1)
Re (1,
(r /
(125)
Jut, (t))
These correlations and response functions close in the obvious generalization of the scalar output case.
H. Comparing Computational Costs
In this section compare the cost of training width N and depth L network P data with full batch GD integrating the DMFT equations in different settings.
29

---

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
Finite Network
Linear DMFT
Res. DMFT
Non-Data Averaged Res. DMFT
Memory
N
LTT
1.772
1.72p22
Compute
LºLPT
LT
L
L2
Table Memory and computational cost to describe full batch gradient descent training of width N depth L network trained on dataset of size P for T steps. The non-data averaged residual DMFT refers to the cost of solving the equations (Bordelon et al., 2024c).
30


Reviews:
[{"review_id": "doVLX46Vs8", "reviewer": "Reviewer_pEUF", "summary": "The authors analyze several models of deep linear networks (FCNs, ResNets) trained on Gaussian (iid, and power-law covariance) data with noisy Gradient Descent, focusing on hyperparameter transfer between small and large models. They develop a DMFT formulation of the problem which can accommodate finite width, finite learning effects, and dataset average. This results in a series of non-linear saddle point equations on kernels and gradient kernels which are solved numerically. The numerical solutions agree well with simulations of actual neural networks. At least in muP scaling, they argue that the optimal learning rate transfers properly across widths while the loss changes.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful reading and thoughtful questions. Below we address the key questions and concerns and hope that the reviewer will be satisfied with our answers and consider an increase in their score.\n\n\n**Result Delivered as Complex Nonlinear Equations** \n\nThe theoretical results, while still complicated, are **lower dimensional** than the system that we started with. In general, we went from gradient flow dynamics on $(L-1)N^2 + ND + N$ parameters to a system of equations for $4L+4$ correlation/response functions. Compared to other mean field results which involve *non-Gaussian Monte-carlo averages*, this theory is much simpler since all equations for the order parameters close directly. \n\nThat said, the critique that **mean field descriptions can be pretty complicated** (especially for nonlinear dynamics) is a valid concern. However, we were able to extract some useful insights from the equations including\n\n1. The divergence of response functions with respect to depth $L$ unless the residual branch is scaled as $1/\\sqrt{L}$\n2. Approximate scalings of maximum stable learning rate for mean field and NTK parameterizations (see Figure 7). \n3. The effect of feature learning on power law convergence rates for power law data (see Figure 6).\n4. The DMFT equations generally reveal a **buildup** of finite width (finite $\\nu$) and finite data (finite $\\alpha$) effects over time. Thus the early part of training is much closer to the \"clean limit\" where $\\alpha,\\nu \\to \\infty$ but later in training there are more significant deviations across model sizes or dataset sizes. \n\nWe will mention some of the insights that we can extract from the theory more concretely in the main text and Appendex sections.\n\n\n\n**Shift in Optimal LRs** \n\n\nIt is true that learning rate transfer for SGD is not perfect in our model, especially when going from **very small widths** to **very large widths**, we point out a few things\n\n1. The transfer in $\\mu$P is much better than for NTK scaling, especially at large widths. \n2. The success or failure of transfer is captured by our theory (dashed lines of Figure 2). \n3. In realistic $\\mu$P experiments other architectural details that are not included in our model (like Layernorm or Adam) improve the hyperparameter transfer effect in $\\mu$P. SGD without layernorm in deep nonlinear networks with $\\mu$P looks similar to the \"sharp\" cutoffs we see in our linear network experiments (see Figure 1 (a)-(b) compared to Figure 2b here https://arxiv.org/abs/2309.16620). \n\n\n**Is the Saddle Point Valid? Do all Eigenvalues Collapse to a Dirac Distribution?** \n\n\nThis is a great question and we thank the reviewer for letting us clear this up! TLDR: Yes, the saddle point is valid and no our equations do not indicate a collapse in the density of eigenvalues, but rather capture Marchenko-Pastur like spread in time-constants. \n\nMore detail:\n\n1. We stress that our action is completely independent of $D$ since **none of our vectors of interest carry data indices**. There are only two vectors for each layer $\\mathbf h^\\ell(t), \\mathbf g^\\ell(t)\\in\\mathbb{R}^N$ defined in equation 5. This is the secret sauce of the linear network setting which enables us to take an **exact proportional limit** (note equations 5 and 6 are special for linear networks). Thus the order parameters of interest $C_h^\\ell(t,t') = \\frac{1}{N} \\mathbf h^\\ell(t) \\cdot \\mathbf h^\\ell(t')$, etc, also **do not carry data indices**. Thus the number of order parameters is $\\mathcal{O}_D(1)$ and we are justified to take the saddle point. This is different than the saddle point of prior works (https://arxiv.org/abs/2205.09653, https://arxiv.org/abs/2304.03408) where the action depends on a number of order parameters which grows with $P$. This is why those works were restricted to $N \\to \\infty$ limits with $P$ fixed.  **We do not take a saddle point over full $P\\times P$ matrices but over correlation and response functions.**\n2. To illustrate that we are really capturing the full eigenvalue density etc, we can show that our equations for $R_\\Delta(t,t')$ can recover the **Marchenko-Pastur law** for a Wishart in the lazy training regime.\n\n\nTo illustrate take $\\nu \\to \\infty$, $L=1$ from our DMFT (the flow $\\frac{d}{dt} \\mathbf v(t) = - \\left(\\frac{1}{P} \\mathbf X^\\top \\mathbf X \\right) \\mathbf v(t) + \\mathbf j(t)$), where we have that the response $\\mathcal H(\\omega) \\equiv  \\int d\\tau e^{-i\\omega \\tau}  \\frac{1}{D}  \\text{Tr} \\frac{\\partial \\mathbf v(t+\\tau)}{\\partial \\mathbf j(t)^\\top}$ satisfies the quadratic equation for the **resolvent of a Wishart matrix** \n\\begin{align} \n    \\alpha^{-1} (i\\omega) \\mathcal H(\\omega)^2 + (i\\omega + 1 - \\alpha^{-1}) \\mathcal H(\\omega) - 1 = 0 .\n\\end{align}\nThis will give a **distribution over time-constants (eigenvalues)** instead of a single Dirac delta function.\n\n**Missing Experimental Data**\n\nD=1000, we will add.\n\n**Computational cost with Parallelism**\n\nWe will include this in the analysis.", "claims_and_evidence": "I found several issues with the manuscript. \n\nThe papers claim to give a theoretical result. However, since theoretical results are delivered implicitly via high-dimensional non-linear equations that require a numerical solver, it is not clear what new understanding one has gained. \n\nA second issue is that optimal learning rate does in fact shift in their setting even in muP. The shift in optimal value itself seems somewhat larger than in Ref. https://arxiv.org/pdf/2203.03466 however the sharpness of the loss makes it so that, in figure 2b, choosing the learning rate based on v=0.5 and transferring to v=5, would incur an order of magnitude change to the loss compared to its optimal value. \n\nA third issue is the concentration of the kernel in NTK scaling. While the authors arrive at an action which has D in front of an seemingly $D$ independent action for the kernels, the orders parameters involved have data indices and hence scale with D. In such circumstances, the saddle point is not clearly justified. A simple example is the Wishart Distribution, wherein by normalizing such that the matrix elements average to the identity, one has a similar structure of probability. However, taking a saddle point would yield a delta function distribution of eigenvalues, whereas in fact the width of the eigenvalue distribution is similar to its average. In muP scaling this issue should go away.", "methods_and_evaluation_criteria": "See above.", "theoretical_claims": "I looked at the derivation, but did not rederive it. I raised above some concerns about the correctness of the saddle point equations in the NTK setting. \n\n\nMinor comment. Eq. 60 61 are missing summand indices.", "experimental_designs_or_analyses": "The experiments look sufficient, apart from the previous comment about the sharpness of the loss.", "relation_to_broader_scientific_literature": "The literature review is Ok.", "essential_references_not_discussed": "Nothing essential.", "other_strengths_and_weaknesses": "Strengths: \n\n1. The authors develop a DMFT formalism for deep learning networks which accounts for data-averaging and finite learning rate effects. \n2. Although not entirely clear from the manuscript, which doesn't give away absolute scales of D, they seem to push the numerical envelope of solving DMFT equation further. \n3. The author provides a toy setting which, despite being too complicated to being solved analytically at the moment, may serves as a stepping stone to future analytical research in this important subfield. \n\nWeaknesses: \n\n1. Presentation: The papers claim to give a theoretical result. However, since the theoretical results are delivered implicitly via high-dimensional non-linear equations that require a numerical solver, it is not clear what new understanding one has gained. \n\n2. Deficiencies of the toy setting: A second issue is that the optimal learning rate does in fact shift in their setting even in muP. The shift in optimal value itself seems somewhat larger than in Ref. https://arxiv.org/pdf/2203.03466 however the sharpness of the loss makes it so that, in figure 2b, choosing the learning rate based on v=0.5 and transferring to v=5, would incur an order of magnitude change to the loss compared to its optimal value. \n\n3. Correctness of derivation for NTK scaling. While the authors arrive at an action which has D in front of an seemingly $D$ independent action for the kernels, the orders parameters involved have data indices and hence scale with D. In such circumstances, the saddle point is not clearly justified. A simple example is the Wishart Distribution, wherein by normalizing such that the matrix elements average to the identity, one has a similar structure of probability. However, taking a saddle point would yield a delta function distribution of eigenvalues, whereas in fact the width of the eigenvalue distribution is similar to its average. In muP scaling this issue should go away.\n\nMore minor \n\n4. Some experimental data missing (what is the input dimension in the transfer figures)\n\n5. Does the computational cost for saddle point solvers and networks factor in the ability to parallelize networks? If not, this should be clearly stated.", "other_comments_or_suggestions": "No.", "questions_for_authors": "See previous comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mJmKmecmDo", "reviewer": "Reviewer_4bAU", "summary": "This paper develops a DMFT based theory for deep linear networks (with and without residual connections) in GD and SGD settings. The authors show that the theory captures the effect of initialization, dataset correlations, width and depth. Moreover, they show hyperparameter transfer with width and depth.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful reading and their support. Below we address the weaknesses. \n\n### Weaknesses \n\n**hyperparameter transfer results are known in prior literature in much more complex settings**\n\nWhile there are already several cases where hyperparameter transfer are documented (including in complicated architectures like transformers), we were seeking the simplest possible setting which could be theoretically analyzed. To capture hyperparameter transfer effects one needs a setting where (1) model can exit the kernel regime and (2) wider models perform better. The simplest model we could identify with these properties was **randomly initialized deep linear networks**. \n\n**Appendix Explanations Sparse**\n\nWe thank the reviewer for this comment. We have made the Appendix sections more detailed and provide the closed form set of equations for the correlation and response functions from the single-site equations. The train and test losses can be computed from the correlation functions $C_v$ and $C_\\Delta$. \n\n### Other comments\n\nThank you for finding these typos and missing links. We have fixed these.\n\n*It would be helpful to provide further details for Sections 4 and 5 in the Appendix.*\n\nWe have included additional details to derive the main results of section 4 and 5. \n\n\n### Questions\n\n*Can the authors clarify if they have any additional insights into hyperparameter transfer compared to prior works?*\n\nOur main result for hyperparameter transfer is that in $\\mu$P scaling, the finite width effects accumulate as a combination of effective noise and bias in the dynamics from finite $\\nu  = N/D$ while the feature updates are approximately independent of $\\nu$ (see equation 14). \n\n*Do the authors understand why does the loss increase initially in Figure 3 (c, d)?*\n\nGood question! This initial loss increase is driven by the variance in the predictor from SGD noise (small $\\alpha_B$) and small width (small $\\nu$). This can be seen from an analysis of the early portion of the DMFT equations where, for the first few steps of training the test loss can be approximated as\n\n$$\\mathcal L(t+1) \\approx \\left[(1-\\eta)^2 + \\frac{\\eta^2}{\\alpha_B} + \\frac{\\eta^2}{\\nu} \\right] \\mathcal{L}(t)$$ \n\nThe loss will exponentially increase at early times provided that\n\\begin{align}\n    \\eta > \\frac{2}{1+\\frac{1}{\\alpha_B} + \\frac{1}{\\nu}} .\n\\end{align}\n\nIndeed from the simulations we see that for sufficiently large $\\alpha_B$ and $\\nu$ this initial increase disappears.", "claims_and_evidence": "Yes. The claims made in the paper are supported by evidence. Specifically, the figures validate the theory.", "methods_and_evaluation_criteria": "Yes. The methods and evaluation make sense.", "theoretical_claims": "Yes. I have verified all the theoretical details in the main text and Appendix A, while skimming the remaining Appendices for soundness.", "experimental_designs_or_analyses": "Yes. The experimental designs are sound.", "relation_to_broader_scientific_literature": "This work adds to the ongoing research on understanding the neural networks by focusing on deep linear networks (with residual connections). The work provides insights into the effect of data complexity, width, depth, and random initialization.", "essential_references_not_discussed": "The essential references are discussed.", "other_strengths_and_weaknesses": "Strengths\n\n* The paper is clearly written and is easy to follow\n\n* The insights on the complexity of data, depth and width are insightful\n\nWeaknesses\n\n* To the best of my knowledge, the hyperparameter transfer results are known in prior literature in much more complex settings. \n\n* While the Section 3 results are clearly discussed in the Appendix, I found the details for Section 4 and 5 to be sparse in the Appendix. In Appendix C, the authors mention: \"After these response functions have been solved for, we must solve for the correlation functions using the single site densities. This gives a closed form prediction for the loss dynamics.\" but do not provide details. Similarly, it would be helpful to expand on Appendix E details on structured data.", "other_comments_or_suggestions": "Comments:\n\n* There is a typo in Equation 9, it should be $W^0$ and not $W^1$. \n* Reference for Appendix is missing on line 150.\n\nSuggestions:\n\n* It would be helpful to provide further details for Sections 4 and 5 in the Appendix.", "questions_for_authors": "Questions:\n\n* Can the authors clarify if they have any additional insights into hyperparameter transfer compared to prior works?\n\n* Do the authors understand why does the loss increase initially in Figure 3 (c, d)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k8xGt7Vw44", "reviewer": "Reviewer_HCdG", "summary": "This work theoritically characterizes the gradient descent dynamics in deep linear networks in the asymptotic limit of infinite data and width of the network. They study the limiting behaviour of both deep linear network and residual deep network for both isotropic data and data with power-law covariance and studies the effect of depth, richness, batch-size and hyper-parameter transfer.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful reading and detailed comments and questions. We address the main concerns below.  \n\n### Theoretical Claims\n\n*$\\gamma_0$ is defined nowhere in the text and it is hard to grasp the meaning of this quantity alhtough i see several results depend on it.*\n\nWe will define this hyperparameter more clearly. Mechanically it is present in the definition of the predictor $f = \\frac{1}{\\gamma_0} w^L ... W^0 x$. This scalar basically controls the laziness/richness of the learning dynamics with $\\gamma_0 \\to 0$ giving the lazy (kernel) regime. For $\\gamma_0 \\gg 0$, the dynamics are far from the kernel regime. We have added more explanation of this and also shown that $\\gamma_0 \\to 0$ limit gives the kernel regime. The dependence of $\\gamma_0$ on network width also defines the difference between NTK parameterization and mean-field/$\\mu$P scaling.\n\n*In equation-10, no effect of learning rate or intializations scale is captured. It is well known that rich regime is mostly captured with small initialization and small learning rate, but this result was not seen from the theory result.*\n\nEquation 10 depends implicitly on the hyperparameter $\\gamma_0$ which we are using to control the laziness/richness. We can alternatively manipulate the initail variance of the weight entries which would have a similar effect. \n\n*Linear networks are known to exhibit saddle to saddle behaviour (for rich dynamics)...However, is it true that this dynamics can't be captured by isotropic random data?*\n\nOur dynamics would start at a *single saddle point* for $\\gamma_0 \\gg 1$ limit with the learning rate scaled down. We could induce **multiple saddles** (saddle-to-saddle behaviors) in a multiple output channel setting with isotropic features, especially if we also allow for small initial weights. We have equations for this setting (Appendix G) but have not numerically integrated them yet. Indeed, in the small initialization regime this is exactly the setting of the work of Saxe et al 2014 https://arxiv.org/abs/1312.6120. \n\n*The effect of learning rate seemed to be missing from the result. What about the alignment between singular vector for each layer? Does the traditional result of implicit bias of gradient flow and descent captureed here? It seemed no from the current result. Discussing this in details would be more useful.*\n\nOur theoretical equations depend directly on the learning rate such as in equation 11 and equation 67. \n\nOur theory, since it applies to non-negligible random initialization, does not demand perfect alignment between adjacent weight matrices. However, in the rich regime, the alignment does tend to increase.  Equation 25 reveals that\n\n$$W^\\ell(t) = W^\\ell(0) + \\frac{\\eta\\gamma_0}{\\sqrt N}\\sum_{t'<t} g^{\\ell+1}(t') h^\\ell(t')^\\top$$\n\nThe first term is random and static, while the second term improves alignment of $W^\\ell$'s left singular vectors with the $\\{ g^\\ell(t) \\}$ vectors which are \n\\begin{align}\ng^{\\ell}(t) = \\left[ \\frac{1}{\\sqrt{N}} W^{\\ell}(t) \\right]^\\top ... w^L(t)\n\\end{align}\nIf the random initialization were negligible compared to the second term, then all weights would align and become low rank, consistent with prior works. However, our theory can flexibly handle large random initialization in either lazy or rich regimes (for arbitrary values of $\\gamma_0$).  \n\n### Other Comments or Suggestions\n\n*Please define $\\mu$P*\n\nWe have added a definition of $\\mu$P as \"maximum update parameterization\", a term introduced in https://arxiv.org/abs/2011.14522. We will add this definition. \n\n*In section-2.1, dynamics are studies in the aymptotic limit of P,D,N. However, why is there a further limit on $\\alpha$ and $\\alpha_B$. This limit was unclear.*\n\nWe apologize that our exposition in this section was unclear to the reader. We will try explaining it more clearly in section 2.1. We basically analyze two settings\n\n1. Full batch gradient descent with a fixed random dataset of size $P$ in a proportional scaling regime $P,D,N\\to\\infty$ with fixed ratios $P/D = \\alpha$ and $N/D = \\nu$.\n2. Online stochastic gradient descent where at each step a batch of size $B$ is sampled and used to estimate the gradient. We look at $B,N,D \\to \\infty$ with fixed ratios $B/D = \\alpha_B$ and $N/D = \\nu$. \n\nWe contrast these two settings in Figure 3, where Figure 3 (a,b) is in the first setting and Figure 3  (c,d) is in the second setting.", "claims_and_evidence": "Yes, the claims and evidence are convincing. However, some parameters remains undefined, so it is difficult for the reader to grasp the essence of the results.", "methods_and_evaluation_criteria": "It is mostly linear networks and labels genrated by linear models.", "theoretical_claims": "The proofs and the theory looks mostly correct to me except some minor questions:\n\n1) $\\gamma_{0}$ is defined nowhere in the text and it is hard to grasp the meaning of this quanitity alhtough i see several results depend on it. \n2) In equation-10, no effect of learning rate or intializations scale is captured. It is well known that rich regime is mostly captured with small initialization and small learning rate, but this result was not seen from the theory result. \n3) Linear networks are known to exhibit saddle to saddle behaviour (for rich dynamics)...However, is it true that this dynamics can't be captured by isotropic random data?", "experimental_designs_or_analyses": "Yes, they are valid.", "relation_to_broader_scientific_literature": "Studying the dynamics of deep linear networks is an important first step to understand deep neural networks. This work should be highly relevant.", "essential_references_not_discussed": "Although there are tons of works on deep linear networks, I think the current refernces are sufficient for the paper.", "other_strengths_and_weaknesses": "4) The effect of learning rate seemed to be missing from the result. What about the alignment between singular vector for each layer? Does the traditional result of implicit bias of gradient flow and descent captureed here? It seemed no from the current result. Discussing this in details would be more useful.", "other_comments_or_suggestions": "1) Please define $\\mu P$. \n2) In section-2.1, dynamics are studies in the aymptotic limit of P,D,N. However, why is there a further limit on $\\alpha$ and $\\alpha_{B}$. This imit was unclear.\n\n## update after rebuttal\n\nI have read the rebuttal and decided to stick with my score of weak accept.", "questions_for_authors": "Please answer points 1,2,3,4.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7VCPS7SiM3", "reviewer": "Reviewer_ZpVM", "summary": "This work provides a theoretical analysis of gradient descent dynamics in deep linear networks trained at large widths from random initialisation. Specifically, gradient descent dynamics, hyper-parameter transfer effects and asymptotic descriptions for deep networks were analysed and discussed.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating our theoretical contributions and the novelty of attempting to capture the hyperparameter transfer effect. \n\n### Methods and Evaluation Criteria\n\n*The evaluation is purely theoretical and does not include standard deep learning benchmarks.*\n\nThe purpose of this paper is primarily theoretical and focuses on a linear networks, rather than an experimental paper purporting to perform well on benchmarks. \n\n\n### Theoretical Claims\n\n*Mathematical formulations are elegant but not carefully verified.*\n\nWe provide a derivation of our results in the Appendix using techniques from statistical mechanics. We also verify our equations under the conditions of our theory (randomly initialized deep linear networks trained on random data) using simulations. \n\n\n### Relation To Broader Scientific Literature:\n\n*Insights on width-depth interactions and hyper-parameter transfer are quite novel, but focusing linear networks limits its scope and application in real cases.*\n\nOur focus on linear networks was motivated by finding the **simplest possible model** that exhibits the phenomenon of interest (which in this case is the learning rate transfer effect). \n\n### Other Strengths And Weaknesses\n\n*The paper provides strong theoretical insights, but lacks extensive empirical validation on real-world datasets.*\n\nSince we are focused on the dynamics of **linear networks**, the generalization on most real-world datasets would not be very good. However, we can attempt to include an experiment on a simple task like MNIST, which due to its power law covariance spectrum (see Figure 6a here https://arxiv.org/abs/2409.17858) would likely look similar to our Figure 6c.", "claims_and_evidence": "Claims are backed by rigorous theoretical derivations.", "methods_and_evaluation_criteria": "The evaluation is purely theoretical and does not include standard deep learning benchmarks.", "theoretical_claims": "Mathematical formulations are elegant but not carefully verified.", "experimental_designs_or_analyses": "Experimental experiments are relatively limited.", "relation_to_broader_scientific_literature": "Insights on width-depth interactions and hyper-parameter transfer are quite novel, but focusing linear networks limits its scope and application in real cases.", "essential_references_not_discussed": "Not applicable.", "other_strengths_and_weaknesses": "The paper provides strong theoretical insights, but lacks extensive empirical validation on real-world datasets.", "other_comments_or_suggestions": "None.", "questions_for_authors": "No specific questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviews are mainly positive, and the only negative review has very little concrete criticism. The paper gives a rigorous and complete analysis of the asymptotic behavior of linear DNNs, leading to interesting insights on hyperparameter transfer. I am leaning towards acceptance for this paper.
Decision: Accept (poster)