Dynamical Mean Field Theory of Kernel Evolution in
Wide Neural Networks
Blake Bordelon & Cengiz Pehlevan
John Paulson School of Engineering and Applied Sciences, Center for Brain Science
Harvard University
Cambridge MA, 02138
blake_bordelon@g.harvard.edu, cpehlevan@g.harvard.edu

Abstract
We analyze feature learning in infinite-width neural networks trained with gradient
flow through a self-consistent dynamical field theory. We construct a collectionof deterministic dynamical order parameters which are inner-product kernels for
hidden unit activations and gradients in each layer at pairs of time points, providing
a reduced description of network activity through training. These kernel orderparameters collectively define the hidden layer activation distribution, the evolutionof the neural tangent kernel, and consequently output predictions. We provide asampling procedure to self-consistently solve for the kernel order parameters.

1

Introduction

Deep learning has emerged as a successful paradigm for solving challenging machine learning andcomputational problems across a variety of domains [1, 2]. However, theoretical understandingof the training and generalization of modern deep learning methods lags behind current practice.
Ideally, a theory of deep learning would be analytically tractable, efficiently computable, capableof predicting network performance and internal features that the network learns, and interpretablethrough a reduced description involving desirably initialization-independent quantities.
Several recent theoretical advances have fruitfully considered the idealization of wide neural networks,
where the number of hidden units in each layer is taken to be large. Under certain parameterization,
Bayesian neural networks and gradient descent trained networks converge to gaussian processes(NNGPs) [3–5] and neural tangent kernel (NTK) machines [6–8] in their respective infinite-width
limits. These limits provide both analytic tractability as well as detailed training and generalization
analysis [9–16]. However, in this limit, with these parameterizations, data representations are fixedand do not adapt to data, termed the lazy regime of NN training, to contrast it from the rich regime
where NNs significantly alter their internal features while fitting the data [17, 18]. The fact that the
representation of data is fixed renders these kernel-based theories incapable of explaining featurelearning, an ingredient which is crucial to the success of deep learning in practice [19, 20]. Thus,
alternative theories capable of modeling feature learning dynamics are needed.
Recently developed alternative parameterizations such as the mean field [21] and the µP [22]parameterizations allow feature learning in infinite-width NNs trained with gradient descent. Using
the Tensor Programs framework, Yang & Hu identified a stochastic process that describes theevolution of preactivation features in infinite-width µP NNs [22]. In this work, we study an equivalentparameterization to µP with self-consistent dynamical mean field theory (DMFT) and recover thestochastic process description of infinite NNs using this alternative technique. In the same large
width scaling, we include a scalar parameter γ0 that allows smooth interpolation between lazy andrich behavior [17]. We provide a new computational procedure to sample this stochastic process anddemonstrate its predictive power for wide NNs.
Machine Learning and the Physical Sciences workshop, NeurIPS 2022.

The present work is a short version of our paper appearing in the main meeting [23]. Our contributionsin these works are the following:
1. We develop a path integral formulation of gradient flow dynamics in infinite-width networks in thefeature learning regime. Our parameterization includes a scalar parameter γ0 to allow interpolation
between rich and lazy regimes and comparison to perturbative methods.
2. Using a stationary action argument, we identify a set of saddle point equations that the kernels
satisfy at infinite-width, relating the stochastic processes that define hidden activation evolution to
the kernels and vice versa. We develop a numerical algorithm to solve these equations and show
they are predictive of wide networks feature learning dynamics.
Our theory is inspired by self-consistent dynamical mean field theory (DMFT) from statistical physics[24–30]. This framework has been utilized in the theory of random recurrent networks [31–36],
tensor PCA [37, 38], phase retrieval [39], and high-dimensional linear classifiers [40–43], but has yet
to be developed for deep feature learning. By developing a self-consistent DMFT of deep NNs, wegain insight into how features evolve in the rich regime of network training, while retaining manypleasant analytic properties of the infinite-width limit.

2

Problem Setup and Definitions

Our theory applies to infinite-width networks, both fully-connected and convolutional. For notational
ease we will focus on the MLP formalism. For input xµ ∈ RD , we define the hidden pre-activation
vectors h` ∈ RN for layers ` ∈ {1, ..., L} as
1fµ = √ wL · φ(hL
µ) ,
γ N

1
h`+1
= √ W ` φ(h`µ ) ,µN1h1µ = √ W 0 xµ ,D(1)where θ = Vec{W 0 , ..., wL } are the trainable parameters of the network and φ is a twice differentiable activation function. Inspired by previous works on the mechanisms of lazy gradient based
training, the parameter γ will control the laziness or richness of the training dynamics [17, 18, 22, 44].
Each of the trainable parameters are initialized as Gaussian random variables with unit variance Wij` ∼d
N (0, 1). They evolve under gradient flow dt
θ = −γ 2 ∇θ L. The choice of learning rate γ 2 causes
dof γ. To characterize the evolution of weights, we introduce backpropagadt L|t=0 to be independent√ ∂ftion variables gµ` = γ N ∂hµ` = φ̇(h`µ ) zµ` , where zµ` = √1N W `> gµ`+1 is the pre-gradient signal.
µ

The relevant objects to characterize feature learning are feature and gradient kernels for each hidden
layer ` ∈ {1, ..., L}, defined as Φ`µα (t, s) = N1 φ(h`µ (t)) · φ(h`α (s)) , G`µα (t, s) = N1 gµ` (t) ·NT Kgα` (s). From these kernels {Φ` , G` }L(t, s) =`=1 , we can compute the Neural Tangent Kernel Kµα
PL`+1`∇θ fµ (t) · ∇θ fα (s) = `=0 Gµα (t, s)Φµα (t, s), [6] and the dynamics of the network function fµ
P
X
d
NT K
fµ (t) =Kµα(t, t)∆α (t) ,dt
α=1∆µ (t) = −∂L|fµ (t) ,
∂fµ

(2)

1
0
x
where we define base cases GL+1µα (t, s) = 1, Φµα (t, s) = Kµα = D xµ · xα . The above expressions
demonstrate that knowledge of the temporal trajectory of the NTK on the t = s diagonal gives
the temporal trajectory of the network predictions fµ (t). Following prior works on infinite-width
networks [21, 22, 45, 18], we study the mean field limit N, γ → ∞ , γ0 = √γN = ON (1). The
γ0 = 0 limit recovers the static NTK limit [6], while γ0 > 0 allows feature learning.

3

Self-consistent DMFT

Next, we derive our self-consistent DMFT in a limit where t, P = ON (1). Our goal is to build
a description of training dynamics purely based on representations, and independent of weights.
Studying feature learning at infinite-width enjoys several analytical properties:• The kernel order parameters Φ` , G` concentrate over random initializations but are dynamical,
allowing flexible adaptation of features to the task structure.
2• In each layer `, each neuron’s preactivation h`i and pregradient zi` become i.i.d. draws from a
distribution characterized by a set of order parameters {Φ` , G` , A` , B ` }.
• The kernels are defined as self-consistent averages (denoted by hi) over this distribution of neurons
in each layer Φ`µα (t, s) = φ(h`µ (t))φ(h`α (s)) and G`µα (t, s) = gµ` (t)gα` (s) .
The next section derives these facts from a path-integral formulation of gradient flow dynamics.
3.1

Path Integral Construction

Gradient flow after a random initialization of weights defines a high dimensional stochastic process
over initalizations for variables {h, g}. Therefore, we will utilize DMFT formalism to obtain areduced description of network activity during training. Generally, we separate the contribution oneach forward/backward pass between the initial condition and gradient updates to weight matrix W ` ,√1 W ` (0)φ(h` (t)) ,defining new stochastic variables χ`+1ξµ` (t) = √1N W ` (0)> gµ`+1 (t).
µ (t) =µN
We let Z represent the moment generating functional (MGF) for these stochastic fields
+*
XZ ∞ 
Z[{j ` , v ` }] = exp dt jµ` (t) · χ`µ (t) + vµ` (t) · ξµ` (t) 
, (3)
`,µ

0
{W 0 (0),...wL (0)}Performing integration over possible paths for χ` , ξ ` , we show that the MGF Z can be described by
set of order-parameters {Φ` , Φ̂` , G` , Ĝ` , A` , B ` } [23]Z Y`
Z[{j ` , v ` }] ∝dΦ`µα (t, s)dΦ̂`µα (t, s)dG`µα (t, s)dĜ`µα (t, s)dA`µα (t, s)dBµα(t, s) (4)
`µαts



× exp N S[{Φ, Φ̂, G, Ĝ, A, B, j, v}] ,i
XZ ∞ Z ∞ h`S=dt
ds Φ`µα (t, s)Φ̂`µα (t, s) + G`µα (t, s)Ĝ`µα (t, s) − A`µα (t, s)Bµα(t, s)`µα00+ ln Z[{Φ, Φ̂, G, Ĝ, A, B, j, v}],

(5)

where S is the DMFT action and Z is a single-site MGF, which defines the distribution of fields{χ` , ξ ` } over the neural population in each layer. The order parameters A and B are related to thecorrelations between feedforward and feedback signals in the network.
3.2

Deriving the DMFT Equations from the Path Integral Saddle Point

As N → ∞, the moment-generating function Z is exponentially dominated by the saddle point of S.
The equations that define this saddle point also define our DMFT. We thus identify the kernels that
render S locally stationary (δS = 0). The most important equations are those which define {Φ` , G` }δSδ Φ̂`µα (t, s)δSδ Ĝ`µα (t, s)= Φ`µα (t, s) +1δZ= Φ`µα (t, s) − φ(h`µ (t))φ(h`α (s)) = 0,
Z δ Φ̂`µα (t, s)= G`µα (t, s) +δZ1
= G`µα (t, s) − gµ` (t)gα` (s) = 0,`
Z δ Ĝµα (t, s)(6)where hi denotes an average over the stochastic process induced by Z, which is defined below{u`µ (t)}µ∈[P ],t∈R+ ∼ GP(0, Φ`−1 ) , {rµ` (t)}µ∈[P ],t∈R+ ∼ GP(0, G`+1 ),
Z t X
P
 `−1
 `
`
`
`
hµ (t) = uµ (t) + γ0dsAµα (t, s) + ∆α (s)Φ`−1µα (t, s) zα (s)φ̇(hα (s)),
0zµ` (t) = rµ` (t) + γ0

α=1

Z t
ds
0

P
X
 `

`
Bµα (t, s) + ∆α (s)G`+1µα (t, s) φ(hα (s)),

(7)

α=1

x
0
L
where we define base cases Φ0µα (t, s) = Kµαand GL+1µα (t, s) = 1, A = B = 0. We see thatthe fields {h` , z ` }, which represent the single site preactivations and pre-gradients, are implicit3

functionals of the mean-zero Gaussian processes {u` , r` } which have covariances u`µ (t)u`α (s) =`Φ`−1(t)r` (s) = G`+1(t, s). The other saddle point equations give response functionsµα (t, s) and rµ
 α

 µα

`
`+1
δφ(hµ (t))
δgµ
(t)
−1
`
A`µα (t, s) = γ0−1
which arise due to coupling between
,
B
(t,
s)
=
γ
µα
0
δr ` (s)δu`+1 (s)ααthe feedforward and feedback signals. We note that, in the lazy limit γ0 → 0, the fields approach
Gaussian processes h` → u` , z ` → r` .

Solving the Self-Consistent DMFT
1.0

Initial Density

t

0.6
0.4
0.2
0

NN 0 = 0
NN 0 = 1.0
DMFT
20
40

t

60

80

100

(a) Lazy vs Rich Loss Dynamics
=2

0.4
0.2
0.0

6

4

2

0

h

2

4

6

(b) Initial Preactivation Density

=3

=1

=2

(e) Φ` Dynamics γ0 = 1.0=3

Theory G(g) Final G` kernels γ0 = 1.0

=1

=2

6

4

1.0
0.9
0.8
0.7
0.6
0.5
0.4

0

h

2

4

6

=1
=2
=3
101

N

102

103

(f) Φ` Convergence to DMFT
1.0
0.9
0.8
=1
=2
=3

0.7
0.6

(h) G` Dynamics γ0 = 1.02(c) Final Preactivation Density=3

DMFT G (t, s) Expt G (t, s)=2Expt G

=1

=1
=2
=3

A( DMFT, NN)

Expt
Theory(d) Final Φ` Kernels γ0 = 1

0.6
0.5
0.4
0.3
0.2
0.1
0.0

=3

DMFT (t, s) Expt (t, s)

=1

0.6

A(GDMFT, GNN)

0.0

=1
=2
=3

0.8

0.8

Final Density

4

101

N

102

103

(i) G` Convergence to DMFT

Figure 1: Neural network feature learning dynamics is captured by self-consistent dynamical mean
field theory (DMFT). (a) Training loss curves on a subsample of P = 10 CIFAR-10 training points
in a depth 4 (L = 3, N = 2500) tanh network (φ(h) = tanh(h)) trained with MSE. Increasing γ0
accelerates training. (b)-(c) The distribution of preactivations at the beginning and end of training
matches predictions of the DMFT. (d) The final Φ` (at t = 100) kernel orderPparameters match thefinite width network. (e) The temporal dynamics of the sample-traced kernels µ Φ`µµ (t, s) matches
experiment and reveals rich dynamics across layers. (f) The alignment A(Φ`DM F T , Φ`N N ), definedas cosine similarity, of the kernel Φ`µα (t, s) predicted by theory (DMFT) and width N networks for√different N but fixed γ0 = γ/ N . Errorbars show standard deviation computed over 10 repeats.
Around N ∼ 500 DMFT begins to show near perfect agreement with the NN. (g)-(i) The same plots
but for the gradient kernel G` . Whereas finite width effects for Φ` are larger at later layers ` sincevariance accumulates on the forward pass, fluctuations in G` are large in early layers.
The saddle point equations obtained from the field theory discussed in the previous section must be
solved self-consistently. By this we mean that, given knowledge of the kernels, we can characterize
the distribution of {h` , z ` }, and given the distribution of {h` , z ` }, we can compute the kernels[46, 41]. We use a numerical procedure based on this idea to efficiently solve for the kernels
with an alternating Monte-Carlo strategy. The output of the algorithm are the dynamical kernels`
Φ`µα (t, s), G`µα (t, s), A`µα (t, s), Bµα(t, s), from which any network observable can be computed. We
4provide an example of the solution to the saddle point equations compared to training a finite NN in
Figure 1. We plot Φ` , G` at the end of training and the sample-trace of these kernels through time. We
compare kernels of finite width N network to the DMFT predicted kernels using a cosine-similarity
Tr ΦDM F T ΦN N
alignment metric A(ΦDM F T , ΦN N ) = |ΦDM F T ||ΦN N | , with agreement at large N .
We can also compare the exact DMFT equations to approximation schemes employed in prior
works on wide networks. Specifically, we compare static NTK (NTK), perturbative (Pert.) (leading
corrections to feature evolution of size γ02 ) [47] and gradient independence (Gr. Indep.) (backward
pass weights approximated as independent of forward pass) [48]. In Figure 2, we show that only
DMFT is accurate over a wide range of feature learning strengths γ0 .
H1

0.95

A(H, HNN)

H2

H3

H4

H5

Gr. Ind. Pert. NTK DMFT Expt.

1.00
0.90

DMFT
NTK
Pert.
0.80
Gr. Ind.
0.25 0.50 0.75 1.00 1.25 1.50
0.850(a) Cosine Similarity with NN(b) Predicted Final Kernels

Figure 2: DMFT is accurate for wide range of γ0 , while other approximations including perturbation
theory and gradient independence break down for large γ0 .

5Broader Impacts

We provided a unifying DMFT derivation of feature dynamics in infinite networks trained withgradient based optimization. Our theory interpolates between lazy infinite-width behavior of a static
NTK in γ0 → 0 and rich feature learning. At γ0 = 1, our DMFT construction agrees with thestochastic process derived previously with the Tensor Programs framework [22]. Our saddle point
equations give self-consistency conditions which relate the stochastic fields to the kernels. Theseequations are exactly solveable in deep linear networks and can be efficiently solved with a numericalmethod in the nonlinear case. Comparisons with other approximation schemes show that DMFT canbe accurate at a much wider range of γ0 . We believe our framework could be a useful perspective
for future theoretical analyses of feature learning and generalization in wide networks, including
networks trained with alternative learning rules [49].
Though our DMFT is quite general in regards to the data and architecture, the technique is not entirely
rigorous and relies on heuristic physics techniques. Our theory holds in the T, P = ON (1) andmay break down otherwise; other asymptotic regimes (such as P/N, T / log(N ) = ON (1), etc) may
exhibit phenomena relevant to deep learning practice [50, 51]. The computational requirements ofour method, while smaller than the exponential time complexity for exact solution [22], are still
significant for large P T . In Table 1, we compare the time taken for various theories to compute thefeature kernels throughout T steps of gradient descent.
Requirements
Width-N NN Static NTK Perturbative Full DMFT
Memory for KernelsO(N 2 )O(P 2 )O(P 4 T )O(P 2 T 2 )224Time for Kernels
O(P N T )O(P )O(P T )O(P 3 T 3 )Time for Final Outputs
O(P N 2 T )O(P 3 )O(P 4 )O(P 3 T 3 )Table 1: Computational requirements to compute kernel dynamics and trained network predictions on
P points in a depth N neural network on a grid of T time points trained with P data points for various
theories. DMFT is faster and less memory intensive than a width N network only if N  P T . It is
more computationallyefficient to compute full DMFT kernels than leading order perturbation theory√
when T  P .

5

Acknowledgments and Disclosure of Funding
This work was supported by NSF grant DMS-2134157 and an award from the Harvard Data Science
Initiative Competitive Research Fund. BB acknowledges additional support from the NSF-Simons
Center for Mathematical and Statistical Analysis of Biology at Harvard (award #1764269) and theHarvard Q-Bio Initiative.

References[1] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
[2] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436–444,
2015.
[3] Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science &Business Media, 2012.
[4] Jaehoon Lee, Jascha Sohl-dickstein, Jeffrey Pennington, Roman Novak, Sam Schoenholz, andYasaman Bahri. Deep neural networks as gaussian processes. In International Conference on
Learning Representations, 2018.
[5] Alexander G. de G. Matthews, Jiri Hron, Mark Rowland, Richard E. Turner, and Zoubin
Ghahramani. Gaussian process behaviour in wide deep neural networks. In International
Conference on Learning Representations, 2018.
[6] Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence andgeneralization in neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman,
N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems,
volume 31, pages 8571–8580. Curran Associates, Inc., 2018.
[7] Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha SohlDickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models
under gradient descent. Advances in neural information processing systems, 32, 2019.
[8] Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang.
On exact computation with an infinitely wide neural net. Advances in Neural Information
Processing Systems, 32, 2019.
[9] Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global
minima of deep neural networks. In International conference on machine learning, pages
1675–1685. PMLR, 2019.
[10] B. Bordelon, A. Canatar, and C. Pehlevan. Spectrum dependent learning curves in kernel
regression and wide neural networks. International Conference of Machine Learning, 2020.
[11] Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Spectral bias and task-modelalignment explain generalization in kernel regression and infinitely wide neural networks.
Nature communications, 12(1):1–12, 2021.
[12] Omry Cohen, Or Malka, and Zohar Ringel. Learning curves for overparametrized deep neuralnetworks: A field theory perspective. Physical Review Research, 3(2):023034, 2021.
[13] Arthur Jacot, Berfin Simsek, Francesco Spadaro, Clément Hongler, and Franck Gabriel. Kernelalignment risk estimator: Risk prediction from training data. Advances in Neural Information
Processing Systems, 33:15568–15578, 2020.
[14] Bruno Loureiro, Cedric Gerbelot, Hugo Cui, Sebastian Goldt, Florent Krzakala, Marc Mezard,
and Lenka Zdeborova. Learning curves of generic features maps for realistic datasets with ateacher-student model. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan,
editors, Advances in Neural Information Processing Systems, 2021.
[15] James B Simon, Madeline Dickens, and Michael R DeWeese. Neural tangent kernel eigenvalues
accurately predict generalization. arXiv preprint arXiv:2110.03922, 2021.
6

[16] Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via
over-parameterization. In International Conference on Machine Learning, pages 242–252.
PMLR, 2019.
[17] Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in Neural Information Processing Systems, 32, 2019.
[18] Mario Geiger, Stefano Spigler, Arthur Jacot, and Matthieu Wyart. Disentangling feature andlazy training in deep neural networks. Journal of Statistical Mechanics: Theory and Experiment,
2020(11):113301, 2020.
[19] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal,
Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are
few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image
recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition,
pages 770–778, 2016.
[21] Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of
two-layer neural networks. Proceedings of the National Academy of Sciences, 115(33):E7665–
E7671, 2018.
[22] Greg Yang and Edward J Hu. Tensor programs iv: Feature learning in infinite-width neuralnetworks. In International Conference on Machine Learning, pages 11727–11737. PMLR,
2021.
[23] Blake Bordelon and Cengiz Pehlevan. Self-consistent dynamical field theory of kernel evolution
in wide neural networks. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun
Cho, editors, Advances in Neural Information Processing Systems, 2022.
[24] Paul Cecil Martin, ED Siggia, and HA Rose. Statistical dynamics of classical systems. PhysicalReview A, 8(1):423, 1973.
[25] C De Dominicis. Dynamics as a substitute for replicas in systems with quenched random
impurities. Physical Review B, 18(9):4913, 1978.
[26] Haim Sompolinsky and Annette Zippelius. Dynamic theory of the spin-glass phase. PhysicalReview Letters, 47(5):359, 1981.
[27] Haim Sompolinsky and Annette Zippelius. Relaxational dynamics of the edwards-andersonmodel and the mean-field theory of spin-glasses. Physical Review B, 25(11):6860, 1982.
[28] G Ben Arous and Alice Guionnet. Large deviations for langevin spin glass dynamics. Probability
Theory and Related Fields, 102(4):455–509, 1995.
[29] G Ben Arous and Alice Guionnet. Symmetric langevin spin glass dynamics. The Annals ofProbability, 25(3):1367–1422, 1997.
[30] Gérard Ben Arous, Amir Dembo, and Alice Guionnet. Cugliandolo-kurchan equations for
dynamics of spin-glasses. Probability theory and related fields, 136(4):619–660, 2006.
[31] A Crisanti and H Sompolinsky. Path integral approach to random neural networks. PhysicalReview E, 98(6):062120, 2018.
[32] Haim Sompolinsky, Andrea Crisanti, and Hans-Jurgen Sommers. Chaos in random neuralnetworks. Physical review letters, 61(3):259, 1988.
[33] Moritz Helias and David Dahmen. Statistical Field Theory for Neural Networks. Springer
International Publishing, 2020.
[34] Lutz Molgedey, J Schuchhardt, and Heinz G Schuster. Suppressing chaos in neural networks bynoise. Physical review letters, 69(26):3717, 1992.
7

[35] M Samuelides and Bruno Cessac. Random recurrent neural networks dynamics. The European
Physical Journal Special Topics, 142(1):89–122, 2007.
[36] Kanaka Rajan, LF Abbott, and Haim Sompolinsky. Stimulus-dependent suppression of chaos
in recurrent neural networks. Physical review e, 82(1):011903, 2010.
[37] Stefano Sarao Mannelli, Florent Krzakala, Pierfrancesco Urbani, and Lenka Zdeborova. Passed& spurious: Descent algorithms and local minima in spiked matrix-tensor models. In international conference on machine learning, pages 4333–4342. PMLR, 2019.
[38] Stefano Sarao Mannelli, Giulio Biroli, Chiara Cammarota, Florent Krzakala, Pierfrancesco
Urbani, and Lenka Zdeborová. Marvels and pitfalls of the langevin algorithm in noisy highdimensional inference. Physical Review X, 10(1):011057, 2020.
[39] Francesca Mignacco, Pierfrancesco Urbani, and Lenka Zdeborová. Stochasticity helps tonavigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval
problem. Machine Learning: Science and Technology, 2(3):035029, 2021.
[40] Elisabeth Agoritsas, Giulio Biroli, Pierfrancesco Urbani, and Francesco Zamponi. Out-ofequilibrium dynamical mean-field equations for the perceptron model. Journal of Physics A:
Mathematical and Theoretical, 51(8):085002, 2018.
[41] Francesca Mignacco, Florent Krzakala, Pierfrancesco Urbani, and Lenka Zdeborová. Dynamical
mean-field theory for stochastic gradient descent in gaussian mixture classification. Advancesin Neural Information Processing Systems, 33:9540–9550, 2020.
[42] Michael Celentano, Chen Cheng, and Andrea Montanari. The high-dimensional asymptotics offirst order methods with random data. arXiv preprint arXiv:2112.07572, 2021.
[43] Francesca Mignacco and Pierfrancesco Urbani. The effective noise of stochastic gradientdescent. arXiv preprint arXiv:2112.10852, 2021.
[44] Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Mean-field theory of two-layers
neural networks: dimension-free bounds and kernel limit. In Conference on Learning Theory,
pages 2388–2464. PMLR, 2019.
[45] Lenaic Chizat and Francis Bach. On the global convergence of gradient descent for overparameterized models using optimal transport. Advances in neural information processing
systems, 31, 2018.
[46] Alessandro Manacorda, Grégory Schehr, and Francesco Zamponi. Numerical solution of
the dynamical mean field theory of infinite-dimensional equilibrium liquids. The Journal ofchemical physics, 152(16):164506, 2020.
[47] Daniel A Roberts, Sho Yaida, and Boris Hanin. The principles of deep learning theory. arXiv
preprint arXiv:2106.10165, 2021.
[48] Greg Yang. Tensor programs iii: Neural matrix laws. arXiv preprint arXiv:2009.10685, 2020.
[49] Blake Bordelon and Cengiz Pehlevan. The influence of learning rule on representation dynamics
in wide neural networks, 2022.
[50] Qianyi Li and Haim Sompolinsky. Statistical mechanics of deep linear neural networks: The
backpropagating kernel renormalization. Physical Review X, 11(3):031059, 2021.
[51] Stéphane d’Ascoli, Maria Refinetti, and Giulio Biroli. Optimal learning rate schedules inhigh-dimensional non-convex optimization problems, 2022.

8

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope? [Yes] As described in the abstract and introduction, we provide adynamical field theory of deep networks based on kernel evolution.
(b) Did you describe the limitations of your work? [Yes] We have an explicit limitations as thelast paragraph of the paper in Section 5.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] This work istheoretical and is very unlikely to present negative social impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] We describe that ourtheory holds for NN architectures in the infinite-width N → ∞ limit.
(b) Did you include complete proofs of all theoretical results? [Yes] All claims made in the main
text are supported by derivations in the Appendix.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] Code to reproduce experimental
results is provided in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they werechosen)? [Yes] We provide details of our experiments in 1(c) Did you report error bars (e.g., with respect to the random seed after running experiments
multiple times)? [Yes] Mentioned 10 repeats in Figure 1 caption.
(d) Did you include the total amount of compute and the type of resources used (e.g., type ofGPUs, internal cluster, or cloud provider)? [N/A] This is a theory paper
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A](b) Did you mention the license of the assets? [N/A](c) Did you include any new assets either in the supplemental material or as a URL? [N/A](d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating? [N/A](e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A](b) Did you describe any potential participant risks, with links to Institutional Review Board(IRB) approvals, if applicable? [N/A](c) Did you include the estimated hourly wage paid to participants and the total amount spent on
participant compensation? [N/A]9