SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\frac {\partial^ {2} S}{\partial B _ {\nu \mu} ^ {\ell} (s , t) \partial \Delta_ {\alpha} (t ^ {\prime})} = \gamma \left\langle \frac {\partial}{\partial \Delta_ {\alpha} (t ^ {\prime}) \partial r _ {\nu} ^ {\ell} (s)} \phi (h _ {\mu} ^ {\ell} (t)) \right\rangle \equiv \gamma D _ {\mu \nu \alpha} ^ {A ^ {\ell} \Delta} (t, s, t ^ {\prime})
$$
Now, we consider the second derivatives involving one derivative on a dual variable  $\{\hat{\Phi}^{\ell},\hat{G}^{\ell},A,B\}$  and one of the primal variables  $\{\Phi^{\ell},G^{\ell}\}$ .
$$
\begin{array}{l} \frac {\partial^ {2} S}{\partial \hat {\Phi} _ {\mu \nu} ^ {\ell} (t , s) \partial \Phi_ {\alpha \beta} ^ {\ell^ {\prime}} (t ^ {\prime} , s ^ {\prime})} = \delta_ {\ell , \ell^ {\prime}} \delta_ {\mu \nu} \delta (t - t ^ {\prime}) \delta (s - s ^ {\prime}) \\ - \delta_ {\ell - 1, \ell^ {\prime}} \frac {\partial}{\partial \Phi_ {\alpha \beta} ^ {\ell - 1} \left(t ^ {\prime} , s ^ {\prime}\right)} \left\langle \phi \left(h _ {\mu} ^ {\ell} (t)\right) \phi \left(h _ {\nu} ^ {\ell} (s)\right) \right\rangle \\ \equiv \delta_ {\ell , \ell^ {\prime}} \delta_ {\mu \nu} \delta (t - t ^ {\prime}) \delta (s - s ^ {\prime}) - \delta_ {\ell - 1, \ell^ {\prime}} D _ {\mu \nu \alpha \beta} ^ {\Phi^ {\ell} \Phi^ {\ell - 1}} (t, s, t ^ {\prime}, s ^ {\prime}) \\ \end{array}
$$
$$
\begin{array}{l} \frac {\partial^ {2} S}{\partial \hat {G} _ {\mu \nu} ^ {\ell} (t , s) \partial G _ {\alpha \beta} ^ {\ell^ {\prime}} (t ^ {\prime} , s ^ {\prime})} = \delta_ {\ell , \ell^ {\prime}} \delta_ {\mu \nu} \delta (t - t ^ {\prime}) \delta (s - s ^ {\prime}) - \delta_ {\ell + 1, \ell^ {\prime}} \frac {\partial}{\partial G _ {\alpha \beta} ^ {\ell + 1} (t ^ {\prime} , s ^ {\prime})} \left<   g _ {\mu} ^ {\ell} (t) g _ {\nu} ^ {\ell} (s) \right> \\ \equiv \delta_ {\ell , \ell^ {\prime}} \delta_ {\mu \nu} \delta (t - t ^ {\prime}) \delta (s - s ^ {\prime}) - \delta_ {\ell - 1, \ell^ {\prime}} D _ {\mu \nu \alpha \beta} ^ {G ^ {\ell} G ^ {\ell + 1}} (t, s, t ^ {\prime}, s ^ {\prime}) \\ \end{array}
$$
$$
\frac {\partial^ {2} S}{\partial \hat {\Phi} _ {\mu \nu} ^ {\ell} (t , s) \partial G _ {\alpha \beta} ^ {\ell + 1} (t ^ {\prime} , s ^ {\prime})} = - \frac {\partial}{\partial G _ {\alpha \beta} ^ {\ell + 1} (t ^ {\prime} , s ^ {\prime})} \left<   \phi (h _ {\mu} ^ {\ell} (t)) \phi (h _ {\nu} ^ {\ell} (s)) \right> \equiv - D _ {\mu \nu \alpha \beta} ^ {\Phi^ {\ell}, G ^ {\ell + 1}} (t, s, t ^ {\prime}, s ^ {\prime})
$$
$$
\frac {\partial^ {2} S}{\partial \hat {G} _ {\mu \nu} ^ {\ell} (t , s) \partial \Phi_ {\alpha \beta} ^ {\ell - 1} (t ^ {\prime} , s ^ {\prime})} = - \frac {\partial}{\partial \Phi_ {\alpha \beta} ^ {\ell - 1} (t ^ {\prime} , s ^ {\prime})} \left<   g _ {\mu} ^ {\ell} (t) g _ {\nu} ^ {\ell} (s) \right> \equiv - D _ {\mu \nu \alpha \beta} ^ {G ^ {\ell}, \Phi^ {\ell - 1}} (t, s, t ^ {\prime}, s ^ {\prime})
$$
$$
\frac {\partial^ {2} S}{\partial A _ {\nu \mu} ^ {\ell - 1} (s , t) \partial \Phi_ {\alpha \beta} ^ {\ell - 1} (t ^ {\prime} , s ^ {\prime})} = \gamma \frac {\partial}{\partial \Phi_ {\alpha \beta} ^ {\ell - 1} (t ^ {\prime} , s ^ {\prime})} \left\langle \frac {\partial g _ {\mu} ^ {\ell} (t)}{\partial r _ {\nu} ^ {\ell} (s)} \right\rangle \equiv \gamma D _ {\mu \nu \alpha \beta} ^ {B ^ {\ell - 1}, \Phi^ {\ell - 1}} (t, s, t ^ {\prime}, s ^ {\prime})
$$
$$
\frac {\partial^ {2} S}{\partial B _ {\nu \mu} ^ {\ell} (s , t) \partial \Phi_ {\alpha \beta} ^ {\ell - 1} \left(t ^ {\prime} , s ^ {\prime}\right)} = \gamma \frac {\partial}{\partial \Phi_ {\alpha \beta} ^ {\ell - 1} \left(t ^ {\prime} , s ^ {\prime}\right)} \left\langle \frac {\partial \phi \left(h _ {\mu} ^ {\ell} (t)\right)}{\partial u _ {\nu} ^ {\ell} (s)} \right\rangle \equiv \gamma D _ {\mu \nu \alpha \beta} ^ {A ^ {\ell}, \Phi^ {\ell - 1}} \left(t, s, t ^ {\prime}, s ^ {\prime}\right)
$$
$$
\frac {\partial^ {2} S}{\partial A _ {\nu \mu} ^ {\ell - 1} (s , t) \partial G _ {\alpha \beta} ^ {\ell + 1} (t ^ {\prime} , s ^ {\prime})} = \gamma \frac {\partial}{\partial G _ {\alpha \beta} ^ {\ell + 1} (t ^ {\prime} , s ^ {\prime})} \left\langle \frac {\partial g _ {\mu} ^ {\ell} (t)}{\partial r _ {\nu} ^ {\ell} (s)} \right\rangle \equiv \gamma D _ {\mu \nu \alpha \beta} ^ {B ^ {\ell - 1}, G ^ {\ell + 1}} (t, s, t ^ {\prime}, s ^ {\prime})
$$
$$
\frac {\partial^ {2} S}{\partial B _ {\nu \mu} ^ {\ell} (s , t) \partial G _ {\alpha \beta} ^ {\ell + 1} \left(t ^ {\prime} , s ^ {\prime}\right)} = \gamma \frac {\partial}{\partial G _ {\alpha \beta} ^ {\ell + 1} \left(t ^ {\prime} , s ^ {\prime}\right)} \left\langle \frac {\partial \phi \left(h _ {\mu} ^ {\ell} (t)\right)}{\partial u _ {\nu} ^ {\ell} (s)} \right\rangle \equiv \gamma D _ {\mu \nu \alpha \beta} ^ {A ^ {\ell}, G ^ {\ell + 1}} \left(t, s, t ^ {\prime}, s ^ {\prime}\right) \tag {37}
$$
We note that terms such as  $\frac{\partial}{\partial\Phi_{\alpha\beta}^{\ell - 1}(t',s')}\left\langle \phi (h_\mu^\ell (t))\phi (h_\nu^\ell (s))\right\rangle$  can be further decomposed since the average over the  $\{u_{\mu}^{\ell}(t)\} \sim \mathcal{GP}(0,\Phi^{\ell -1})$  and  $h^\ell$  's explicit dynamics both depend on  $\Phi^{\ell -1}$
$$
\begin{array}{l} \frac {\partial}{\partial \Phi_ {\alpha \beta} ^ {\ell - 1} (t ^ {\prime} , s ^ {\prime})} \left<   \phi (h _ {\mu} ^ {\ell} (t)) \phi (h _ {\nu} ^ {\ell} (s)) \right> = \frac {1}{2} \left<   \frac {\partial^ {2}}{\partial u _ {\alpha} ^ {\ell} (t ^ {\prime}) \partial u _ {\beta} ^ {\ell} (s ^ {\prime})} \phi (h _ {\mu} ^ {\ell} (t)) \phi (h _ {\nu} ^ {\ell} (s)) \right> \\ + \left\langle \frac {\partial}{\partial \Phi_ {\alpha \beta} ^ {\ell - 1} \left(t ^ {\prime} , s ^ {\prime}\right)} \phi \left(h _ {\mu} ^ {\ell} (t)\right) \phi \left(h _ {\nu} ^ {\ell} (s)\right) \right\rangle \tag {38} \\ \end{array}
$$
where the first term comes from differentiating the Gaussian probability density for  $u^{\ell}$  (e.g. Price's theorem) and the second term is an explicit derivative of the preactivation fields with  $u^{\ell}$  treated as constant. Next we consider the nonvanishing terms which involve  $\{\hat{\Delta},\hat{K},\Delta ,K\}$  which give
$$
\begin{array}{l} \frac {\partial^ {2} S}{\partial \hat {\Delta} _ {\mu} (t) \partial \Delta_ {\alpha} (s)} = \delta_ {\mu \alpha} \delta (t - s) + \Theta (t - s) K _ {\mu \alpha} (s) \\ \frac {\partial^ {2}}{\partial \hat {\Delta} _ {\mu} (t) \partial K _ {\alpha \beta} (s)} = \delta_ {\mu \alpha} \Theta (t - s) \Delta_ {\beta} (s) \\ \frac {\partial^ {2} S}{\partial \hat {K} _ {\mu \nu} (t) \partial K _ {\alpha \beta} (t ^ {\prime})} = \delta_ {\mu \alpha} \delta_ {\nu \beta} \delta (t - t ^ {\prime}) \\ \frac {\partial^ {2} S}{\partial \hat {K} _ {\mu \nu} (t) \partial \Phi_ {\alpha \beta} ^ {\ell} (t ^ {\prime} , s ^ {\prime})} = \delta_ {\mu \alpha} \delta_ {\nu \beta} G _ {\alpha \beta} ^ {\ell + 1} (t ^ {\prime}, s ^ {\prime}) \delta (t - t ^ {\prime}) \delta (t - s ^ {\prime}) \\ \frac {\partial^ {2} S}{\partial \hat {K} _ {\mu \nu} (t) \partial G _ {\alpha \beta} ^ {\ell} \left(t ^ {\prime} , s ^ {\prime}\right)} = \delta_ {\mu \alpha} \delta_ {\nu \beta} \Phi_ {\alpha \beta} ^ {\ell - 1} \left(t ^ {\prime}, s ^ {\prime}\right) \delta \left(t - t ^ {\prime}\right) \delta \left(t - s ^ {\prime}\right) \tag {39} \\ \end{array}
$$
This enumerates all possible non-vanishing terms in the Hessian. We can now construct a block matrix of these Hessians by partitioning our order parameters  $\pmb{q} = [q_1, q_2]^\top$  where
$$
\boldsymbol {q} _ {1} = \operatorname {V e c} \left\{\Phi_ {\mu \nu} ^ {\ell} (t, s), G _ {\mu \nu} ^ {\ell} (t, s), K _ {\mu \nu} (t), \Delta_ {\mu} (t), \hat {\Phi} _ {\mu \nu} ^ {\ell} (t, s), \hat {G} _ {\mu \nu} ^ {\ell} (t, s), \hat {K} _ {\mu \nu} (t), \hat {\Delta} _ {\mu} (t) \right\} \tag {40}
$$
$$
\boldsymbol {q} _ {2} = \operatorname {V e c} \left\{A _ {\mu \nu} ^ {\ell} (t, s), B _ {\mu \nu} ^ {\ell} (t, s) \right\}. \tag {41}
$$
This choice will become apparent shortly.
$$
\nabla_ {\boldsymbol {q}} ^ {2} S = \left[ \begin{array}{c c} \nabla_ {\boldsymbol {q} _ {1}} ^ {2} S & \nabla_ {\boldsymbol {q} _ {1}} ^ {2} \boldsymbol {q} _ {2} S \\ \nabla_ {\boldsymbol {q} _ {2}} ^ {2} \boldsymbol {q} _ {1} S & \nabla_ {\boldsymbol {q} _ {2}} ^ {2} S \end{array} \right] \tag {42}
$$
To calculate the full propagator  $\pmb{\Sigma} = -\left[\nabla_{\pmb{q}}^{2}S\right]^{-1}$ , we will assume invertibility of the upper block  $\pmb{\Sigma}^{0} = -\left[\nabla_{\pmb{q}_{1}}^{2}S\right]^{-1}$  and use this in the Schur complement
$$
\boldsymbol {\Sigma} = - \left[ \nabla_ {\boldsymbol {q}} ^ {2} S \right] ^ {- 1} = \left[ \begin{array}{c c} \boldsymbol {\Sigma} _ {1 1} & \boldsymbol {\Sigma} _ {1 2} \\ \boldsymbol {\Sigma} _ {2 1} & \boldsymbol {\Sigma} _ {2 2} \end{array} \right]
$$
$$
\boldsymbol {\Sigma} _ {1 1} = \boldsymbol {\Sigma} ^ {0} - \boldsymbol {\Sigma} ^ {0} \left[ \nabla_ {\boldsymbol {q} _ {1} \boldsymbol {q} _ {2}} ^ {2} S \right] \left(\nabla_ {\boldsymbol {q} _ {2}} ^ {2} S + (\nabla_ {\boldsymbol {q} _ {2} \boldsymbol {q} _ {1}} ^ {2} S) \boldsymbol {\Sigma} ^ {0} (\nabla_ {\boldsymbol {q} _ {1} \boldsymbol {q} _ {2}} ^ {2} S)\right) ^ {- 1} \left[ \nabla_ {\boldsymbol {q} _ {2} \boldsymbol {q} _ {1}} ^ {2} S \right] \boldsymbol {\Sigma} ^ {0}
$$
$$
\pmb {\Sigma} _ {1 2} = \pmb {\Sigma} _ {2 1} ^ {\top} = - \pmb {\Sigma} ^ {0} \left[ \nabla_ {\pmb {q} _ {1} \pmb {q} _ {2}} ^ {2} S \right] \left(\nabla_ {\pmb {q} _ {2}} ^ {2} S + (\nabla_ {\pmb {q} _ {2} \pmb {q} _ {1}} ^ {2} S) \pmb {\Sigma} ^ {0} (\nabla_ {\pmb {q} _ {1} \pmb {q} _ {2}} ^ {2} S)\right) ^ {- 1}
$$
$$
\boldsymbol {\Sigma} _ {2 2} = - \left(\nabla_ {\boldsymbol {q} _ {2}} ^ {2} S + \left(\nabla_ {\boldsymbol {q} _ {2} \boldsymbol {q} _ {1}} ^ {2} S\right) \boldsymbol {\Sigma} ^ {0} \left(\nabla_ {\boldsymbol {q} _ {1} \boldsymbol {q} _ {2}} ^ {2} S\right)\right) ^ {- 1} \tag {43}
$$
We now need to solve for  $\pmb{\Sigma}^{0} = -\left[\nabla_{\pmb{q}_{1}}^{2}S\right]^{-1}$ . To perform this inverse, we again partition  $\pmb{q}_{1}$  into two sets of order parameters  $\pmb{q}_{1} = [\pmb{q}_{1}^{1},\pmb{q}_{1}^{2}]$  where  $\pmb{q}_{1}^{1} = \mathrm{Vec}\{\Phi_{\mu \nu}^{\ell}(t,s),G_{\mu \nu}^{\ell}(t,s),K_{\mu \nu}(t),\Delta_{\mu}(t)\}$  and  $\pmb{q}_{1}^{2} = \mathrm{Vec}\{\hat{\Phi}_{\mu \nu}^{\ell}(t,s),\hat{G}_{\mu \nu}^{\ell}(t,s),\hat{K}_{\mu \nu}(t),\hat{\Delta}_{\mu}(t)\}$
$$
\nabla_ {q _ {1}} ^ {2} S = \left[ \begin{array}{c c} \mathbf {0} & \boldsymbol {U} ^ {\top} \\ \boldsymbol {U} & \boldsymbol {\kappa} \end{array} \right], \kappa \equiv \nabla_ {q _ {1} ^ {2}} ^ {2} S, \boldsymbol {U} \equiv \nabla_ {q _ {1} ^ {2} q _ {1} ^ {1}} ^ {2} S \tag {44}
$$
We seek a physically sensible inverse where the variance of  $q_{1}^{2}$  is vanishing [51, 53]. This leads to the following sub-propagator  $\Sigma^0$
$$
\boldsymbol {\Sigma} ^ {0} = - \left[ \begin{array}{c c} \nabla_ {\boldsymbol {q} _ {1}} ^ {2} S \end{array} \right] ^ {- 1} = \left[ \begin{array}{c c} \boldsymbol {U} ^ {- 1} \boldsymbol {\kappa} \left[ \boldsymbol {U} ^ {- 1} \right] ^ {\top} & - \boldsymbol {U} ^ {- 1} \\ - \left[ \boldsymbol {U} ^ {\top} \right] ^ {- 1} & \boldsymbol {0} \end{array} \right] \tag {45}
$$
Thus given  $\kappa, U$ , we can solve for  $\Sigma^0$  and ultimately for the full propagator  $\Sigma$ . The relevant entries in  $\kappa$  and  $U$  are given by those second derivatives calculated above. We note that each of the field derivatives needed for  $U$  can be computed implicitly from the field dynamics. For example, for the  $\Delta_{\mu}(t)$  derivatives we have
$$
\begin{array}{l} \frac {\partial}{\partial \Delta_ {\nu^ {\prime}} (t ^ {\prime})} h _ {\mu} ^ {\ell} (t) = \gamma \Theta (t - t ^ {\prime}) \Phi_ {\mu \nu^ {\prime}} ^ {\ell - 1} (t, t ^ {\prime}) g _ {\nu} ^ {\ell} (t ^ {\prime}) \\ + \gamma \int_ {0} ^ {t} d s \sum_ {\nu} \left[ A _ {\mu \nu} ^ {\ell - 1} (t, s) + \Phi_ {\mu \nu} ^ {\ell - 1} (t, s) \Delta_ {\nu} (s) \right] \frac {\partial g _ {\nu} ^ {\ell} (s)}{\partial \Delta_ {\nu^ {\prime}} (t ^ {\prime})} \\ \end{array}
$$
$$
\begin{array}{l} \frac {\partial}{\partial \Delta_ {\nu} (t ^ {\prime})} z _ {\mu} ^ {\ell} (t) = \gamma \Theta (t - t ^ {\prime}) G _ {\mu \nu} ^ {\ell + 1} (t, t ^ {\prime}) \phi \left(h _ {\nu} ^ {\ell} \left(t ^ {\prime}\right)\right) \\ + \gamma \int_ {0} ^ {t} d s \sum_ {\nu} \left[ B _ {\mu \nu} ^ {\ell} (t, s) + G _ {\mu \nu} ^ {\ell + 1} (t, s) \Delta_ {\nu} (s) \right] \frac {\partial \phi \left(h _ {\nu} ^ {\ell} (s)\right)}{\partial \Delta_ {\nu^ {\prime}} \left(t ^ {\prime}\right)} \tag {46} \\ \end{array}
$$
These can then be used in the averages such as  $\left\langle \frac{\partial}{\partial \Delta_{\nu'}(t')} \phi(h_\mu^\ell(t)) \phi(h_\nu^\ell(s)) \right\rangle$ . Similarly, we can compute terms such as  $\frac{\partial h_\mu^\ell(t)}{\partial \Phi_{\alpha \beta(t', s')}^\ell}$  through the following closed equations
$$
\begin{array}{l} \frac {\partial h _ {\mu} ^ {\ell} (t)}{\partial \Phi_ {\alpha \beta} ^ {\ell - 1} (t ^ {\prime} , s ^ {\prime})} = \gamma \delta (t - t ^ {\prime}) \delta_ {\mu \alpha} \Theta (t - s ^ {\prime}) \Delta_ {\beta} (s ^ {\prime}) \\ + \gamma \int_ {0} ^ {t} d s \sum_ {\nu} \left[ A _ {\mu \nu} ^ {\ell - 1} (t, s) + \Delta_ {\nu} (s) \Phi_ {\mu \nu} ^ {\ell - 1} (t, s) \right] \frac {\partial g _ {\nu} ^ {\ell} (s)}{\partial \Phi_ {\alpha \beta} ^ {\ell} \left(t ^ {\prime} , s ^ {\prime}\right)} \\ \end{array}
$$
$$
\frac {\partial z _ {\mu} ^ {\ell} (t)}{\partial \Phi_ {\alpha \beta} ^ {\ell - 1} \left(t ^ {\prime} , s ^ {\prime}\right)} = \gamma \int_ {0} ^ {t} d s \sum_ {\nu} \left[ B _ {\mu \nu} ^ {\ell} (t, s) + \Delta_ {\nu} (s) G _ {\mu \nu} ^ {\ell + 1} (t, s) \right] \frac {\partial \phi \left(h _ {\nu} ^ {\ell} (s)\right)}{\partial \Phi_ {\alpha \beta} ^ {\ell - 1} \left(t ^ {\prime} , s ^ {\prime}\right)} \tag {47}
$$
These terms can then be used to compute quantities like  $D^{\Phi^{\ell}}$ .
F Solving for the Propagator
In this section we sketch out the required steps to obtain the propagator  $\pmb{\Sigma}$ .
- Step 1: Solve the infinite width DMFT equations for  $q_{\infty}$  which include the prediction error dynamics  $\Delta_{\mu}(t)$ , the feature kernels  $\Phi_{\mu \nu}^{\ell}(t,s)$ , gradient kernels  $G_{\mu \nu}^{\ell}(t,s)$ . This step corresponds to algorithm in Bordelon & Pehlevan [22] and defines the dynamics one would expect at infinite width [9]. See below for more detail.
- Step 2: Compute the entries of the Hessian of  $S$  evaluated at the  $q_{\infty}$  computed in the first step. Some of these entries look like fourth cumulants of features like  $\kappa = \left\langle \phi(h)^4 \right\rangle - \left\langle \phi(h)^2 \right\rangle^2$  and some of them measure sensitivity of one order parameter to a perturbation in another order parameter  $D^{\Phi^\ell} = \frac{\partial}{\partial \Phi^{\ell-1}} \left\langle \phi(h^\ell)^2 \right\rangle$ . The averages  $\langle \rangle$  used to calculate  $\kappa$  and  $D^{\Phi^\ell}$  should be performed over the infinite width stochastic processes for preactivations  $h^\ell$  which are defined in equation (19).
- Step 3: After populating the entries of the block matrix for the Hessian  $\nabla^2 S$ , we then calculate the propagator  $\Sigma$  with a matrix inversion. Since we discretized time, this is a finite dimensional matrix.
The step 1 above demands a solution to the infinite width DMFT equations (solving for the saddle point  $q_{\infty}$ ). We will now give a detailed set of instructions about how the infinite width limit for  $q_{\infty}$  is solved (step 1 above). This corresponds to the algorithm of Bordelon & Pehlevan 2022 to solve the saddle point equations  $\frac{\partial}{\partial q} S(\pmb{q})|_{q_{\infty}} = 0$  [9].
- Step 1: Start with a guess for the kernels  $\Phi_{\mu \nu}^{\ell}(t,s), G_{\mu \nu}^{\ell}(t,s)$  and for the predictions through time  $f_{\mu}(t)$ . We usually use the lazy limit (e.g.  $\Phi_{\mu \nu}^{\ell}(t,s) = \Phi_{\mu \nu}^{\ell}(0,0)\ldots$ ) as an initial guess.
- Step 2: Sample Gaussian sources  $u_{\mu}^{\ell}(t)$  and  $r_{\mu}^{\ell}(t)$  based on the current covariances  $\Phi^{\ell}$  and  $G^{\ell}$ .
- Step 3: For each sample, solve integral equations for  $h(t)$  and  $z(t)$ .
$$
h _ {\mu} ^ {\ell} (t) = u _ {\mu} ^ {\ell} (t) + \gamma \int_ {0} ^ {t} d s \sum_ {\nu} [ A _ {\mu \nu} ^ {\ell - 1} (t, s) + \Phi_ {\mu \nu} ^ {\ell - 1} (t, s) ] [ \dot {\phi} (h _ {\nu} ^ {\ell} (s)) z _ {\nu} ^ {\ell} (s) ]
$$
$$
z _ {\mu} ^ {\ell} (t) = r _ {\mu} ^ {\ell} (t) + \gamma \int_ {0} ^ {t} d s \sum_ {\nu} \left[ B _ {\mu \nu} ^ {\ell} (t, s) + G _ {\mu \nu} ^ {\ell + 1} (t, s) \right] \phi \left(h _ {\nu} ^ {\ell} (s)\right) \tag {48}
$$
These will be samples from the single site distribution for  $h, z$
- Step 4: Average over the Monte Carlo samples to produce a new estimate of the kernels:  $\Phi^{\ell}(t,s) = \left\langle \phi (h^{\ell}(t))\phi (h^{\ell}(s))\right\rangle$ . A similar procedure is performed for  $G^{\ell}$  and the response functions  $A^\ell, B^\ell$ .
- Step 5: Compute the NTK estimate  $K(t) = \sum_{\ell} G^{\ell + 1}(t, t) \Phi^{\ell}(t, t)$  and then integrate prediction dynamics from the dynamics of the NTK  $\frac{d}{dt} f_{\mu}(t) = \sum_{\nu} K_{\mu \nu}(t) \Delta_{\nu}(t)$ .
- Repeat steps 2-5 until the order parameters converge.
Below we provide a pseudocode algorithm to solve for the propagator elements.
Algorithm 1: Propagator Solver
Data:  $K^x$ ,  $\pmb{y}$ , Initial Guesses  $\{\Phi^\ell, G^\ell\}_{\ell=1}^L$ ,  $\{\pmb{A}^\ell, B^\ell\}_{\ell=1}^{L-1}$ , Sample count  $\mathcal{S}$ , Update Speed  $\beta$   
Result: Propagator Matrix  $\pmb{\Sigma}$
1 Solve DMFT equations with Algorithm 2 for order parameters  $f_{\mu}(t), \Phi_{\mu \alpha}^{\ell}(t,s), \ldots$ ;
2 Draw  $S$  samples  $\{u_{\mu ,n}^{\ell}(t)\}_{n = 1}^{S}\sim \mathcal{GP}(0,\Phi^{\ell -1}),\{r_{\mu ,n}^{\ell}(t)\}_{n = 1}^{S}\sim \mathcal{GP}(0,G^{\ell +1})$
3 Integrate dynamics for each sample to get  $\{h_{\mu ,n}^{\ell}(t),z_{\mu ,n}^{\ell}(t)\}_{n = 1}^{S}$
4 Estimate  $\kappa$  functions with Monte Carlo integration, for instance
5  $\kappa_{\mu \nu \alpha \beta}^{\Phi^{\ell}}(t,s,t^{\prime},s^{\prime}) =$ $\frac{1}{S}\sum_{n\in [S]}\phi (h_{\mu ,n}^{\ell}(t))\phi (h_{\nu ,n}^{\ell}(s))\phi (h_{\alpha ,n}^{\ell}(t^{\prime}))\phi (h_{\beta ,n}^{\ell}(s^{\prime})) - \Phi_{\mu \nu}^{\ell}(t,s)\Phi_{\alpha \beta}^{\ell}(t^{\prime},s^{\prime});$
6 For each sample, compute field sensitivities to error signals, such as  $\frac{\partial h_{\mu,n}^{\ell}(t)}{\partial\Delta_{\nu}(s)}$  and kernels  $\frac{\partial h_{\mu,n}^{\ell}(t)}{\partial\Phi_{\alpha\beta}^{\ell}(t',s')}$  implicitly using equations (46) (47);
7 Use these sensitivities to compute the necessary  $D$  tensors such as
$$
D _ {\mu \nu \alpha} ^ {\Phi^ {\ell} \Delta} = \frac {1}{S} \sum_ {n \in [ S ]} \frac {\partial}{\partial \Delta_ {\alpha} (t ^ {\prime})} \left[ \phi (h _ {\mu , n} ^ {\ell} (t)) \phi (h _ {\nu , n} ^ {\ell} (s)) \right];
$$
8 Invert  $U$  matrix and compute  $\Sigma_0$  in equation (45);
9 Compute the Schur-complement in equation (43) to handle the response functions;
The above propagator solver builds on the solution to the DMFT equations which is provided below.
Algorithm 2: Alternating Monte Carlo Solution to Saddle Point Equations
Data:  $K^x$  y, Initial Guesses  $\{\Phi^\ell ,G^\ell \}_{\ell = 1}^L$ $\{A^\ell ,B^\ell \}_{\ell = 1}^{L - 1}$  , Sample count S, Update Speed  $\beta$  Result: Final Kernels  $\{\Phi^{\ell},G^{\ell}\}_{\ell = 1}^{L}$ $\{A^{\ell},B^{\ell}\}_{\ell = 1}^{L - 1}$  , Network predictions through training  $f_{\mu}(t)$ $\Phi^0 = K^x\otimes 11^\top$ $G^{L + 1} = 11^{\top}$  .   
while Kernels Not Converged do   
3 From  $\{\Phi^\ell ,G^\ell \}$  compute  $K^{NTK}(t,t)$  and solve  $\frac{d}{dt} f_{\mu}(t) = \sum_{\alpha}\Delta_{\alpha}(t)K_{\mu \alpha}^{NTK}(t,t);$    
4  $\ell = 1$  .   
5 while  $\ell < L + 1$  do   
6 Draw  $S$  samples  $\{u_{\mu ,n}^{\ell}(t)\}_{n = 1}^{S}\sim \mathcal{GP}(0,\Phi^{\ell -1}),\{r_{\mu ,n}^{\ell}(t)\}_{n = 1}^{S}\sim \mathcal{GP}(0,G^{\ell +1})$    
7 Integrate dynamics for each sample to get  $\{h_\mu ,n(t),z_\mu ,n(t)\}_{n = 1}^S$    
8 Compute new  $\Phi^{\ell},G^{\ell}$  estimates:   
9  $\tilde{\Phi}_{\mu \alpha}^{\ell}(t,s) = \frac{1}{S}\sum_{n\in [S]}\phi (h_{\mu ,n}^{\ell}(t))\phi (h_{\alpha ,n}^{\ell}(s)),\tilde{G}_{\mu \alpha}(t,s) = \frac{1}{S}\sum_{n\in [S]}g_{\mu ,n}^{\ell}(t)g_{\alpha ,n}^{\ell}(s);$    
10 Solve for Jacobians on each sample  $\frac{\partial\phi(h_n^\ell)}{\partial r_n^{\ell\dagger}},\frac{\partial g_n^\ell}{\partial u_n^{\ell\dagger}};$    
11 Compute new  $A^\ell$ $B^{\ell -1}$  estimates:   
12  $\tilde{A}^{\ell} = \frac{1}{S}\sum_{n\in [S]}\frac{\partial\phi(h_n^\ell)}{\partial r_n^{\ell\dagger}},\tilde{B}^{\ell -1} = \frac{1}{S}\sum_{n\in [S]}\frac{\partial g_n^\ell}{\partial u_n^{\ell\dagger}};$    
13  $\ell \gets \ell +1$    
end   
15  $\ell = 1$    
16 while  $\ell < L + 1$  do   
17 Update feature kernels:  $\Phi^\ell \gets (1 - \beta)\Phi^\ell +\beta \tilde{\Phi}^\ell$ $G^{\ell}\gets (1 - \beta)G^{\ell} + \beta \tilde{G}^{\ell};$    
if  $\ell < L$  then   
19 | Update  $A^\ell \leftarrow (1 - \beta)A^\ell +\beta \tilde{A}^\ell$ $B^{\ell}\gets (1 - \beta)B^{\ell} + \beta \tilde{B}^{\ell}$    
20 end   
21  $\ell \gets \ell +1$    
22 end   
23 end   
24 return  $\{\Phi^\ell ,G^\ell \}_{\ell = 1}^{L},\{A^\ell ,B^\ell \}_{\ell = 1}^{L - 1},\{f_\mu (t)\}_{\mu = 1}^P$
G Leading Correction to the Mean Order Parameters
In this section we use the propagator structure derived in the last section to reason about the leading finite size correction to  $\langle \pmb{q}\rangle$  at width  $N$ . Letting the indices  $i,j,k,n$  enumerate all entries of the order parameters in  $\pmb{q}$  (technically this is a sum over samples and an integral over time for gradient flow), we find the leading Pade Approximant for the mean has the form (App D)
$$
\begin{array}{l} \langle q _ {i} - q _ {i} ^ {\infty} \rangle_ {N} = \frac {N \left\langle (q _ {i} - q _ {i} ^ {\infty}) V \right\rangle_ {\infty} + \frac {N ^ {2}}{2} \left\langle (q _ {i} - q _ {i} ^ {\infty}) V ^ {2} \right\rangle_ {\infty} \dots}{1 + N \left\langle V \right\rangle_ {\infty} + \frac {N ^ {2}}{2} \left\langle V ^ {2} \right\rangle_ {\infty} + \dots} \\ \sim \frac {1}{3 ! N} \sum_ {j k l} \frac {\partial^ {3} S}{\partial q _ {j} \partial q _ {k} \partial q _ {l}} \left\langle \delta_ {i} \delta_ {j} \delta_ {k} \delta_ {l} \right\rangle_ {\infty} + \mathcal {O} (N ^ {- 2}). (49) \\ = \frac {1}{2 N} \sum_ {j k l} \frac {\partial^ {3} S}{\partial q _ {j} \partial q _ {k} \partial q _ {l}} \Sigma_ {i j} \Sigma_ {k l} + \mathcal {O} (N ^ {- 2}) (50) \\ \end{array}
$$
where  $\delta_{j} = \sqrt{N} (q_{j} - q_{j}^{\infty})$  and the derivatives are computed at the saddle point. In the last line, we utilized Wick's theorem and the permutation symmetry of the third derivative  $\frac{\partial^3S}{\partial q_i\partial q_j\partial q_k}$  to evaluate the four point averages in terms of the propagator  $\Sigma_{ij}$ , which was provided in the preceding section E. In practice computing even the full set of second derivatives for the DMFT action to get  $\Sigma$  is quite challenging. Despite the challenge of computing the mean order parameter correction, these corrections are relevant in practice and crucially distinguish the training timescales of deep networks at different widths as we show in Figures 7 and A.4.
G.1 Correction to Mean Predictions and Full MSE Correction
Supposing that we solved for the propagator  $\pmb{\Sigma}$ , using the formalism in the preceding section, we can compute the  $\mathcal{O}(N^{-1})$  correction to the average network prediction error due to finite size. We let  $\langle \Delta (t)\rangle$  represent the average of errors over an ensemble of width  $N$  networks.