SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \frac {d}{d t} \left\langle \Delta_ {\mu} (t) \right\rangle = - \sum_ {\nu} \left\langle K _ {\mu \nu} (t) \Delta_ {\nu} (t) \right\rangle \\ = - \sum_ {\nu} \left\langle K _ {\mu \nu} (t) \right\rangle \left\langle \Delta_ {\nu} (t) \right\rangle - \sum_ {\nu} \operatorname {C o v} \left(K _ {\mu \nu} (t), \Delta_ {\nu} (t)\right) \\ \sim - \sum_ {\nu} \left\langle K _ {\mu \nu} (t) \right\rangle \left\langle \Delta_ {\nu} (t) \right\rangle - \frac {1}{N} \sum_ {\nu} \Sigma_ {\mu \nu \nu} ^ {K \Delta} (t, t) + \mathcal {O} \left(N ^ {- 2}\right) \tag {51} \\ \end{array}
$$
where  $\Sigma_{\mu \nu \nu}^{K\Delta}(t,t)$  is the leading covariance (propagator element) between the kernel  $K_{\mu \nu}(t)$  and prediction error  $\Delta_{\nu}(t)$ . We see that the average kernel  $\langle K_{\mu \nu}(t)\rangle$  (which depends on the finite width  $N$ ) plays an important role in characterizing the timescales of the average prediction dynamics. Once this equation is solved for  $\langle \Delta_{\mu}(t)\rangle$ , the square loss at width  $N$  and time  $t$  has the form
$$
\sum_ {\mu} \left\langle \Delta_ {\mu} (t) ^ {2} \right\rangle \sim \left(1 - \frac {2}{N}\right) \sum_ {\mu} \Delta_ {\mu} ^ {\infty} (t) ^ {2} + \frac {2}{N} \sum_ {\mu} \left\langle \Delta_ {\mu} (t) \right\rangle_ {\infty} \Delta_ {\mu} ^ {\infty} (t) + \frac {1}{N} \sum_ {\mu} \Sigma_ {\mu \mu} ^ {\Delta} (t, t) + \mathcal {O} \left(N ^ {- 2}\right) \tag {52}
$$
We will now comment on the structure of the cross term in this above solution. First, if  $\langle K\rangle \succeq K^{\infty}$  and  $\Sigma^{K\Delta}$  is negligible then the average errors at finite width will decay more rapidly than the infinite width model. However, we suspect that in general,  $\langle K\rangle -K^{\infty}$  contains many negative eigenvalues since signal propagation at finite width tends to reduce the scale of feature kernels [14]. We suspect that this is the cause of the slower dynamics of ensembled predictors for narrower networks in Figure 7 and Figure A.4. Additionally, the term involving  $\Sigma^{K\Delta}$  will generically increase the cross term since the dynamics of  $\Delta$  cause its fluctuations to become anti-correlated with the fluctuations in  $K$ . In general, it is challenging to make strong definitive statements about the relative scale of these competing effects on the cross term. However, we can say more about this solution in the lazy limit, where we find that the cross term will generically be positive, leading to larger MSE (Appendix H.2).
G.2 Perturbation Theory in Rates rather than Predictions
In experiments on deep CNNs trained on CIFAR-10 in 7 and A.4, we find that the loss curves for the ensemble averaged predictors are effectively time rescaled by a function of network width. In this section, we argue that a proper way to account for this is to compute a perturbation expansion in the exponent which defines the rate of decay of the training errors. To illustrate the point, we first consider the case of a single training example before describing larger datasets. In this case, we consider the change of variables  $\Delta(t) = e^{-r(t)}y$ . We now treat  $r$  as an order parameter of the theory with dynamics
$$
\frac {d}{d t} r (t) = K (t) \tag {53}
$$
Note that this equation is now a linear relation between two order parameters  $(r(t), K(t))$ , whereas the relation was previously quadratic. In the lazy limit, if  $K \to K - \epsilon$  then  $r \to r - \epsilon t$ , giving an effective rescaling of training time by  $1 - \frac{\epsilon}{K}$ .
For multiple training examples, we introduce the notion of a transition matrix  $\mathbf{T}(t) \in \mathbb{R}^{P \times P}$  which has dynamics
$$
\frac {d}{d t} \boldsymbol {T} (t) = - \boldsymbol {K} (t) \boldsymbol {T} (t), \boldsymbol {T} (0) = \mathbf {I}. \tag {54}
$$
The solution to the training prediction errors can be obtained at any time  $t$  by multiplying the initial condition  $\Delta(0) = y$  with the transition matrix  $\Delta(t) = T(t)y$ , where  $y$  are the training targets. In this case, the relevant rate matrix, which would be an alternative order parameter is
$$
\boldsymbol {R} (t) = - \log \boldsymbol {T} (t) \tag {55}
$$
where  $\log$  is the matrix logarithm function. Note that in general  $T(t)$  admits a Peano-Baker series solution [62-64]. In the special case where  $\pmb{K}(t)$  commutes with  $\bar{\pmb{K}}(t) = \frac{1}{t}\int_{0}^{t}ds\pmb{K}(s)$ , we obtain the following simplified formula for the rate matrix  $\pmb{R}$
$$
\boldsymbol {R} (t) = \int_ {0} ^ {t} d s \boldsymbol {K} (s) \tag {56}
$$
The benefit of this representation is the elimination of coupled order parameter dynamics which are quadratic in fluctuations (in  $\pmb{\Delta}$  and  $\pmb{K}$ ) into a linear dynamical relation between order parameters  $\pmb{R}$  and  $\pmb{K}$ . An expansion in  $\pmb{R}$  will thus give better predictions at long times  $t$  than a direct expansion in  $\pmb{\Delta}$ . In the lazy  $\gamma \rightarrow 0$  limit, the constancy of  $\pmb{K}(t) = \pmb{K}$  gives the further simplification  $\pmb{R} = \pmb{K}t$ . Working with this representation, we have the following finite width expression for the training loss
$$
\begin{array}{l} \left\langle | \boldsymbol {\Delta} (t) | ^ {2} \right\rangle = \boldsymbol {y} ^ {\top} \left\langle \exp (- 2 \boldsymbol {R} (t)) \right\rangle \boldsymbol {y} \\ \sim \boldsymbol {y} ^ {\top} \exp \left(- 2 \left(\boldsymbol {R} _ {\infty} (t) + \frac {1}{N} \boldsymbol {R} ^ {1} (t)\right)\right) \boldsymbol {y} \\ + \frac {1}{2} \sum_ {\mu \nu \alpha \beta} \Sigma_ {\mu \nu \alpha \beta} ^ {R} (t, t) \frac {\partial^ {2}}{\partial R _ {\mu \nu} \partial R _ {\alpha \beta}} \boldsymbol {y} ^ {\top} \exp (- 2 \boldsymbol {R}) \left. \boldsymbol {y} \right| _ {\boldsymbol {R} = \boldsymbol {R} _ {\infty} (t) + \frac {1}{N} \boldsymbol {R} ^ {1} (t)} + \mathcal {O} (N ^ {- 2}) \tag {57} \\ \end{array}
$$
where  $\langle \pmb{R} \rangle \sim \pmb{R}_{\infty} + \frac{1}{N} \pmb{R}^{1} + \mathcal{O}(N^{-2})$  is the leading correction to the mean  $\pmb{R}$ . In this representation, it is clear that finite width can alter the timescale of the dynamics through a correction to the mean of  $\pmb{R}$ , as well as contribute an additive correction from fluctuations. This justifies the study perturbation analysis of rates  $R_{N}$  as a function of  $1/N$  in Figures 7 and A.4.
H Variance in the Lazy Limit
We can simplify the propagator equations in the lazy  $\gamma \rightarrow 0$  limit. To demonstrate how to use our formalism, we go through the complete process of inverting the Hessian, however, for this case, this procedure is a bit cumbersome. A simplified derivation for the lazy limit can be found below in section H.1 which relies only on linearizing the dynamics around the infinite width solution. In the  $\gamma \rightarrow 0$  limit, all of the  $D$  tensors vanish and the  $\kappa$  tensors are constant in time. Thus, it suffices to analyze the kernels restricted to  $t = 0$  and study the evolution of the prediction variance  $\Delta(t)$ .
$$
\begin{array}{l} S = \int d t \sum_ {\mu} \hat {\Delta} _ {\mu} (t) \left(\Delta_ {\mu} (t) - y _ {\mu} + \int d s \sum_ {\nu} \Theta (t - s) K _ {\mu \nu} \Delta_ {\nu} (s)\right) \\ + \sum_ {\ell} \sum_ {\mu \nu} \left[ \hat {\Phi} _ {\mu \nu} ^ {\ell} \Phi_ {\mu \nu} ^ {\ell} + G _ {\mu \nu} ^ {\ell} \hat {G} _ {\mu \nu} ^ {\ell} \right] + \sum_ {\mu \nu} \hat {K} _ {\mu \nu} \left[ K _ {\mu \nu} - \sum_ {\ell} G _ {\mu \nu} ^ {\ell + 1} \Phi_ {\mu \nu} ^ {\ell} \right] + \sum_ {\ell} \ln \mathcal {Z} _ {\ell} \\ \mathcal {Z} _ {\ell} = \mathbb {E} _ {\left\{u _ {\mu} ^ {\ell} \right\}, \left\{r _ {\mu} ^ {\ell} \right\}} \exp \left(- \sum_ {\mu \nu} \hat {\Phi} _ {\mu \nu} ^ {\ell} \phi \left(u _ {\mu} ^ {\ell}\right) \phi \left(u _ {\nu} ^ {\ell}\right) - \sum_ {\mu \nu} \hat {G} _ {\mu \nu} ^ {\ell} g _ {\mu} ^ {\ell} g _ {\nu} ^ {\ell}\right), g _ {\mu} ^ {\ell} = r _ {\mu} ^ {\ell} \dot {\phi} \left(u _ {\mu} ^ {\ell}\right) \tag {58} \\ \end{array}
$$
where  $\{u_{\mu}^{\ell}\} \sim \mathcal{N}(0,\Phi^{\ell -1}),\{r_{\mu}^{\ell}\} \sim \mathcal{N}(0,G^{\ell +1})$  . Taking two derivatives with respect to  $\{\hat{\Phi}^\ell ,\hat{G}^\ell \}$  give terms of the form
$$
\begin{array}{l} \kappa_ {\mu \nu \alpha \beta} ^ {\Phi^ {\ell}} = \left\langle \phi \left(u _ {\mu} ^ {\ell}\right) \phi \left(u _ {\nu} ^ {\ell}\right) \phi \left(u _ {\alpha} ^ {\ell}\right) \phi \left(u _ {\beta} ^ {\ell}\right) \right\rangle - \Phi_ {\mu \nu} ^ {\ell} \Phi_ {\alpha \beta} ^ {\ell} \\ \kappa_ {\mu \nu \alpha \beta} ^ {G ^ {\ell}} = \left\langle g _ {\mu} ^ {\ell} g _ {\nu} ^ {\ell} g _ {\alpha} ^ {\ell} g _ {\beta} ^ {\ell} \right\rangle - G _ {\mu \nu} ^ {\ell} G _ {\alpha \beta} ^ {\ell} \\ \kappa_ {\mu \nu \alpha \beta} ^ {\Phi^ {\ell}, G ^ {\ell}} = \left\langle \phi \left(u _ {\mu} ^ {\ell}\right) \phi \left(u _ {\nu} ^ {\ell}\right) g _ {\alpha} ^ {\ell} g _ {\beta} ^ {\ell} \right\rangle - \Phi_ {\mu \nu} ^ {\ell} G _ {\alpha \beta} ^ {\ell} \tag {59} \\ \end{array}
$$
Given these we also have the relevant non-vanishing sensitivity tensors
$$
D _ {\mu \nu \alpha \beta} ^ {\Phi^ {\ell + 1} \Phi^ {\ell}} = \frac {\partial^ {2}}{\partial \Phi_ {\alpha \beta} ^ {\ell}} \left\langle \phi (u _ {\mu} ^ {\ell + 1}) \phi (u _ {\nu} ^ {\ell + 1}) \right\rangle , D _ {\mu \nu \alpha \beta} ^ {G ^ {\ell} G ^ {\ell + 1}} = \frac {\partial}{\partial G _ {\alpha \beta} ^ {\ell + 1}} \left\langle g _ {\mu} ^ {\ell} g _ {\nu} ^ {\ell} \right\rangle
$$
$$
D _ {\mu \nu \alpha \beta} ^ {G ^ {\ell} \Phi^ {\ell - 1}} = \frac {\partial}{\partial \Phi_ {\alpha \beta} ^ {\ell - 1}} \left\langle g _ {\mu} ^ {\ell} g _ {\nu} ^ {\ell} \right\rangle \tag {60}
$$
$$
D _ {\mu \nu \alpha \beta} ^ {K \Phi^ {\ell}} = \delta_ {\mu \alpha} \delta_ {\nu \beta} G _ {\mu \nu} ^ {\ell + 1}, D _ {\mu \nu \alpha \beta} ^ {K G ^ {\ell}} = \delta_ {\mu \alpha} \delta_ {\nu \beta} \Phi_ {\mu \nu} ^ {\ell - 1}
$$
$$
D _ {\mu \alpha \beta} ^ {\Delta K} (t) = \int d s \Theta (t - s) \delta_ {\mu \alpha} \Delta_ {\beta} (s) \tag {61}
$$
As before we let  $\mathbf{q}_1 = \mathrm{Vec}\{\Delta_\mu(t), \Phi_{\mu\nu}^\ell, G_{\mu\nu}^\ell, K_{\mu\nu}\}$  and  $\mathbf{q}_2 = \mathrm{Vec}\{\hat{\Delta}_\mu(t), \hat{\Phi}_{\mu\nu}^\ell, \hat{G}_{\mu\nu}^\ell, \hat{K}_{\mu\nu}\}$ . The propagator has the form
$$
\boldsymbol {U} \equiv \nabla_ {\boldsymbol {q} _ {2} \boldsymbol {q} _ {1}} ^ {2} S = \left[ \begin{array}{c c c c} \mathbf {I} + \boldsymbol {\Theta} _ {K} & \mathbf {0} & \mathbf {0} & D ^ {\Delta K} \\ \mathbf {0} & \mathbf {I} - D ^ {\Phi , \Phi} & \mathbf {0} & \mathbf {0} \\ \mathbf {0} & - D ^ {G \Phi} & \mathbf {I} - D ^ {G G} & \mathbf {0} \\ \mathbf {0} & - D ^ {K \Phi} & - D ^ {K G} & \mathbf {I} \end{array} \right], \nabla_ {\boldsymbol {q} _ {2} \boldsymbol {q} _ {2}} ^ {2} S = \left[ \begin{array}{c c c c} \mathbf {0} & \mathbf {0} & \mathbf {0} & \mathbf {0} \\ \mathbf {0} & \kappa^ {\Phi , \Phi} & \kappa^ {\Phi G} & \mathbf {0} \\ \mathbf {0} & \kappa^ {G \Phi} & \kappa^ {G G} & \mathbf {0} \\ \mathbf {0} & \mathbf {0} & \mathbf {0} & \mathbf {0} \end{array} \right] \tag {62}
$$
The propagator of interest is  $\pmb{\Sigma}_{\pmb{q}_1} = \pmb{U}^{-1}\left[\nabla_{\pmb{q}_2\pmb{q}_2}^2 S\right]\pmb{U}^{-1^\top}$ . We can exploit the block structure of  $\pmb{U}$  to find an inverse
$$
\boldsymbol {U} ^ {- 1} = \left[ \begin{array}{c c c c} \boldsymbol {U} _ {\Delta \Delta} ^ {- 1} & \boldsymbol {U} _ {\Delta \Phi} ^ {- 1} & \boldsymbol {U} _ {\Delta G} ^ {- 1} & \boldsymbol {U} _ {\Delta K} ^ {- 1} \\ \boldsymbol {0} & \boldsymbol {U} _ {\Phi \Phi} ^ {- 1} & \boldsymbol {0} & \boldsymbol {0} \\ \boldsymbol {0} & \boldsymbol {U} _ {G \Phi} ^ {- 1} & \boldsymbol {U} _ {G G} ^ {- 1} & \boldsymbol {0} \\ \boldsymbol {0} & \boldsymbol {U} _ {K \Phi} ^ {- 1} & \boldsymbol {U} _ {K G} ^ {- 1} & \mathbf {I} \end{array} \right] \tag {63}
$$
where each sub-block can be computed with the Schur-complement formula. Altogether, we multiply through to get the propagator
$$
\begin{array}{l} \boldsymbol {\Sigma} = \left[ \begin{array}{l l l l} \mathbf {0} & U _ {\Delta \Phi} ^ {- 1} \kappa^ {\Phi \Phi} + U _ {\Delta G} ^ {- 1} \kappa^ {G \Phi} & U _ {\Delta \Phi} ^ {- 1} \kappa^ {\Phi G} + U _ {\Delta G} ^ {- 1} \kappa^ {G G} & \mathbf {0} \\ \mathbf {0} & U _ {\Phi \Phi} ^ {- 1} \kappa^ {\Phi \Phi} & U _ {\Phi \Phi} ^ {- 1} \kappa^ {\Phi G} & \mathbf {0} \\ \mathbf {0} & U _ {G \Phi} ^ {- 1} \kappa^ {\Phi \Phi} + U _ {G G} ^ {- 1} \kappa^ {G \Phi} & U _ {G \Phi} ^ {- 1} \kappa^ {\Phi G} + U _ {G G} ^ {- 1} \kappa^ {G G} & \mathbf {0} \\ \mathbf {0} & U _ {K \Phi} ^ {- 1} \kappa^ {\Phi \Phi} + U _ {K G} ^ {- 1} \kappa^ {G \Phi} & U _ {K \Phi} ^ {- 1} \kappa^ {\Phi G} + U _ {K G} ^ {- 1} \kappa^ {G G} & \mathbf {0} \end{array} \right] \\ \times \left[ \begin{array}{c c c c} \boldsymbol {U} _ {\Delta \Delta} ^ {- 1} & \mathbf {0} & \mathbf {0} & \mathbf {0} \\ \left[ \boldsymbol {U} _ {\Delta \Phi} ^ {- 1} \right] ^ {\top} & \boldsymbol {U} _ {\Phi \Phi} ^ {- 1} & \left[ \boldsymbol {U} _ {G \Phi} ^ {- 1} \right] ^ {\top} & \left[ \boldsymbol {U} _ {K \Phi} ^ {- 1} \right] ^ {\top} \\ \left[ \boldsymbol {U} _ {\Delta G} ^ {- 1} \right] ^ {\top} & \mathbf {0} & \boldsymbol {U} _ {G G} ^ {- 1} & \left[ \boldsymbol {U} _ {K G} ^ {- 1} \right] ^ {\top} \\ \left[ \boldsymbol {U} _ {\Delta K} ^ {- 1} \right] ^ {\top} & \mathbf {0} & \mathbf {0} & \mathbf {I} \end{array} \right] \tag {64} \\ \end{array}
$$
Two of these blocks corresponding to  $K, \Delta$  are especially important for characterizing the fluctuations of network predictions. The covariance structure for  $K$  has the form
$$
\boldsymbol {\Sigma} _ {K} = \boldsymbol {U} _ {K \Phi} ^ {- 1} \boldsymbol {\kappa} ^ {\Phi \Phi} \left[ \boldsymbol {U} _ {K \Phi} ^ {- 1} \right] ^ {\top} + \boldsymbol {U} _ {K G} ^ {- 1} \boldsymbol {\kappa} ^ {G \Phi} \left[ \boldsymbol {U} _ {K \Phi} ^ {- 1} \right] ^ {\top} + \boldsymbol {U} _ {K \Phi} ^ {- 1} \boldsymbol {\kappa} ^ {\Phi G} \left[ \boldsymbol {U} _ {K G} ^ {- 1} \right] ^ {\top} + \boldsymbol {U} _ {K G} ^ {- 1} \boldsymbol {\kappa} ^ {G G} \left[ \boldsymbol {U} _ {K G} ^ {- 1} \right] ^ {\top} \tag {65}
$$
Next we use the fact that  $\mathbf{U}_{\Delta \Phi}^{-1} = \mathbf{U}_{\Delta K}^{-1}\mathbf{U}_{K\Phi}^{-1}$  and that  $\mathbf{U}_{\Delta G}^{-1} = \mathbf{U}_{\Delta K}^{-1}\mathbf{U}_{KG}^{-1}$ , which follows from the block structure of  $\mathbf{U}$ . Consequently we arrive at the identity
$$
\begin{array}{l} \Sigma_ {\Delta} = U _ {\Delta \Phi} ^ {- 1} \kappa^ {\Phi \Phi} [ U _ {\Delta \Phi} ^ {- 1} ] ^ {- 1} + U _ {\Delta G} ^ {- 1} \kappa^ {G \Phi} [ U _ {\Delta \Phi} ^ {- 1} ] ^ {- 1} + U _ {\Delta G} ^ {- 1} \kappa^ {G \Phi} [ U _ {\Delta \Phi} ^ {- 1} ] ^ {- 1} + U _ {\Delta G} ^ {- 1} \kappa^ {G G} [ U _ {\Delta G} ^ {- 1} ] ^ {- 1} \\ = \boldsymbol {U} _ {\Delta K} ^ {- 1} \boldsymbol {\Sigma} _ {K} \left[ \boldsymbol {U} _ {K \Delta} ^ {- 1} \right] ^ {\top}. \tag {66} \\ \end{array}
$$
Lastly, we note that, by the Schur-complement formula that  $U_{\Delta K}^{-1} = -(\mathbf{I} + \Theta_K)^{-1}D^{\Delta K}$ . Thus, writing  $(\mathbf{I} + \Theta_K)\Sigma_\Delta (\mathbf{I} + \Theta_K)^\top = D^{\Delta K}\Sigma_K[D^{\Delta K}]^\top$  as an integral equation, we find
$$
\begin{array}{l} \Sigma_ {\mu \nu} ^ {\Delta} (t, s) + \int_ {0} ^ {t} d t ^ {\prime} \sum_ {\alpha} K _ {\mu \alpha} \Sigma_ {\alpha \nu} ^ {\Delta} (t ^ {\prime}, s) + \int_ {0} ^ {s} d s ^ {\prime} \sum_ {\beta} K _ {\nu \beta} \Sigma_ {\mu \beta} ^ {\Delta} (t, s ^ {\prime}) \\ + \int_ {0} ^ {t} d t ^ {\prime} \int_ {0} ^ {s} d s ^ {\prime} \sum_ {\alpha \beta} K _ {\mu \alpha} K _ {\nu \beta} \Sigma_ {\alpha \beta} ^ {\Delta} \left(t ^ {\prime}, s ^ {\prime}\right) = \sum_ {\alpha \beta} \int_ {0} ^ {t} \Delta_ {\alpha} \left(t ^ {\prime}\right) \int_ {0} ^ {s} d s ^ {\prime} \Delta_ {\beta} \left(s ^ {\prime}\right) \Sigma_ {\mu \alpha , \nu \beta} ^ {K} \tag {67} \\ \end{array}
$$
Differentiation with respect to  $t$  and  $s$  gives a simple differential equation
$$
\begin{array}{l} \frac {\partial^ {2}}{\partial t \partial s} \Sigma_ {\mu \nu} ^ {\Delta} (t, s) + \sum_ {\alpha} K _ {\mu \alpha} \frac {\partial}{\partial s} \Sigma_ {\alpha \nu} ^ {\Delta} (t, s) + \sum_ {\beta} K _ {\nu \beta} \frac {\partial}{\partial t} \Sigma_ {\mu \beta} ^ {\Delta} (t, s) \\ + \sum_ {\alpha \beta} K _ {\mu \alpha} K _ {\nu \beta} \Sigma_ {\alpha \beta} ^ {\Delta} (t, s) = \sum_ {\alpha \beta} \Delta_ {\alpha} (t) \Delta_ {\beta} (s) \Sigma_ {\mu \alpha , \nu \beta} ^ {K} \tag {68} \\ \end{array}
$$
Let  $\{\pmb{\psi}_k\}$  be the eigenvectors of the kernel matrix  $\pmb{K}$ . Projecting these dynamics on the eigenspace  $\Sigma_{k\ell}(t,s) = \pmb{\psi}_k^\top \pmb{\Sigma}(t,s)\pmb{\psi}_\ell$  recovers the equation in the main text
$$
\left(\frac {\partial}{\partial t} + \lambda_ {k}\right) \left(\frac {\partial}{\partial s} + \lambda_ {\ell}\right) \Sigma_ {k \ell} (t, s) = \sum_ {k ^ {\prime} \ell^ {\prime}} \Delta_ {k ^ {\prime}} (t) \Delta_ {\ell^ {\prime}} (s) \Sigma_ {k k ^ {\prime} \ell \ell^ {\prime}} ^ {K} \tag {69}
$$
Replacing  $\Sigma^K = \kappa$  recovers the equation (7) in the main text.
H.1 Perturbed Linear System
In this section, we provide a simpler derivation of the lazy limit training error variance dynamics. In this case, we merely perturb the dynamics around its infinite width value  $\Delta(t) = \Delta_{\infty}(t) + \epsilon^{\Delta}(t)$  and  $K = K_{\infty} + \epsilon^{K}$ , and keep terms only linear in these perturbations. The perturbation  $\epsilon^{K}$  is fixed in time and the dynamics of  $\epsilon^{\Delta}(t)$  are
$$
\frac {d}{d t} \boldsymbol {\epsilon} ^ {\Delta} (t) = - \boldsymbol {K} _ {\infty} \boldsymbol {\epsilon} ^ {\Delta} (t) - \boldsymbol {\epsilon} ^ {K} \boldsymbol {\Delta} _ {\infty} (t) \tag {70}
$$
Projecting this equation on the eigenspace of  $K_{\infty}$  gives
$$
\frac {d}{d t} \epsilon_ {k} ^ {\Delta} (t) = - \lambda_ {k} \epsilon_ {k} (t) - \sum_ {k ^ {\prime}} \epsilon_ {k k ^ {\prime}} ^ {K} \Delta_ {k ^ {\prime}} ^ {\infty} (t) \tag {71}
$$
This immediately recovers the final result of the last section
$$
\begin{array}{l} N \left(\frac {\partial}{\partial t} + \lambda_ {k}\right) \left(\frac {\partial}{\partial t} + \lambda_ {k}\right) \left\langle \epsilon_ {k} ^ {\Delta} (t) \epsilon_ {\ell} ^ {\Delta} (s) \right\rangle = \left(\frac {\partial}{\partial t} + \lambda_ {k}\right) \left(\frac {\partial}{\partial t} + \lambda_ {k}\right) \Sigma_ {k \ell} ^ {\Delta} (t, s) \\ = \sum_ {k ^ {\prime} \ell^ {\prime}} \Sigma_ {k k ^ {\prime} \ell \ell^ {\prime}} ^ {K} \Delta_ {k ^ {\prime}} ^ {\infty} (t) \Delta_ {\ell^ {\prime}} ^ {\infty} (s) \tag {72} \\ \end{array}
$$
Qualitatively, the process of computing this linear correction (in  $\epsilon^K$ ) to the dynamics of  $\Delta$  is identical to the argument utilized in prior work on perturbative feature learning corrections [11]. In that context, the perturbation is caused by small amounts of feature learning, rather than initialization fluctuations.
H.2 Mean Prediction Error Correction in the Lazy Limit
Using a similar heuristic as in the preceding section, we now consider the correction to the mean predictor  $\langle \Delta_{\mu}(t)\rangle$  in the lazy limit. Taylor expanding  $\langle \Delta (t)\rangle$  in powers of  $1 / N$ , we find
$$
\begin{array}{l} \frac {d}{d t} \langle \boldsymbol {\Delta} (t) \rangle = \frac {d}{d t} \boldsymbol {\Delta} ^ {\infty} (t) + \frac {1}{N} \frac {d}{d t} \boldsymbol {\Delta} ^ {1} (t) + \dots \\ = - \left\langle \left(\boldsymbol {K} - \boldsymbol {K} ^ {\infty} + \boldsymbol {K} ^ {\infty}\right) \left(\boldsymbol {\Delta} - \boldsymbol {\Delta} ^ {\infty} + \boldsymbol {\Delta} ^ {\infty}\right) \right\rangle \\ = - \boldsymbol {K} ^ {\infty} \boldsymbol {\Delta} ^ {\infty} - \boldsymbol {K} ^ {\infty} \langle \boldsymbol {\Delta} - \boldsymbol {\Delta} ^ {\infty} \rangle \\ - \left\langle \left(\boldsymbol {K} - \boldsymbol {K} ^ {\infty}\right) \right\rangle \Delta^ {\infty} - \left\langle \left(\boldsymbol {K} - \boldsymbol {K} ^ {\infty}\right) \left(\Delta - \Delta^ {\infty}\right) \right\rangle \\ \sim - \boldsymbol {K} ^ {\infty} \boldsymbol {\Delta} ^ {\infty} - \frac {1}{N} \boldsymbol {K} ^ {\infty} \boldsymbol {\Delta} ^ {1} - \frac {1}{N} \boldsymbol {K} ^ {1} \boldsymbol {\Delta} ^ {\infty} - \frac {1}{N} \left\langle \boldsymbol {\epsilon} ^ {K} \boldsymbol {\epsilon} ^ {\Delta} \right\rangle_ {\infty} + \mathcal {O} (N ^ {- 2}) \tag {73} \\ \end{array}
$$
From the previous section we have that
$$
\frac {d}{d t} \epsilon^ {\Delta} = - K ^ {\infty} \epsilon^ {\Delta} - \epsilon^ {K} \Delta^ {\infty} \Rightarrow \epsilon^ {\Delta} (t) = - \int_ {0} ^ {t} d s \exp (- K ^ {\infty} (t - s)) \epsilon^ {K} \exp (- K ^ {\infty} s) y \tag {74}
$$
Projecting these dynamics onto the eigenspace of the kernel gives
$$
\epsilon_ {k} ^ {\Delta} (t) = - \sum_ {\ell} \epsilon_ {k \ell} ^ {K} \frac {e ^ {- \lambda_ {\ell} t} - e ^ {- \lambda_ {k} t}}{\lambda_ {k} - \lambda_ {\ell}} y _ {\ell} \tag {75}
$$
where  $\ell = k$  should be seen as the limit where  $\lambda_{k}\to \lambda_{\ell}$  of the above. Thus we find that the leading mean correction to the error solves the following differential equation
$$
\begin{array}{l} \left(\frac {d}{d t} + \lambda_ {k}\right) \Delta_ {k} ^ {1} (t) = - \sum_ {\ell} K _ {k \ell} ^ {1} y _ {\ell} e ^ {- \lambda_ {\ell} t} + \sum_ {\ell \ell^ {\prime}} \Sigma_ {k \ell \ell \ell^ {\prime}} ^ {K} \frac {e ^ {- \lambda_ {\ell^ {\prime}} t} - e ^ {- \lambda_ {\ell} t}}{\lambda_ {\ell} - \lambda_ {\ell^ {\prime}}} y _ {\ell^ {\prime}}. \\ = \sum_ {\ell} y _ {\ell} e ^ {- \lambda_ {\ell} t} \left[ - K _ {k \ell} ^ {1} + \Sigma_ {k \ell \ell \ell} ^ {K} t \right] + \sum_ {\ell \neq \ell^ {\prime}} \Sigma_ {k \ell \ell \ell^ {\prime}} ^ {K} \frac {e ^ {- \lambda_ {\ell^ {\prime}} t} - e ^ {- \lambda_ {\ell} t}}{\lambda_ {\ell} - \lambda_ {\ell^ {\prime}}} y _ {\ell^ {\prime}} \tag {76} \\ \end{array}
$$
We see that at late sufficiently large  $t$ , that the terms involving  $\Sigma^K$  will dominate. We can gain more intuition by considering the special case of a single training data point where the mean error correction has the form
$$
\begin{array}{l} \left(\frac {d}{d t} + \lambda\right) \Delta^ {1} (t) = y e ^ {- \lambda t} \left[ - K ^ {1} + t \Sigma^ {K} \right] \Rightarrow \Delta^ {1} (t) = y \left[ - t K ^ {1} + \frac {1}{2} t ^ {2} \Sigma^ {K} \right] e ^ {- \lambda t} \\ \Longrightarrow \left\langle \Delta (t) ^ {2} \right\rangle \sim \Delta^ {\infty} (t) ^ {2} + \frac {1}{N} \left[ 2 y ^ {2} t e ^ {- 2 \lambda t} \left[ - K ^ {1} + \frac {1}{2} t \Sigma^ {K} \right] + \Sigma^ {\Delta} (t, t) \right] + \mathcal {O} (N ^ {- 2}) \\ \sim \Delta^ {\infty} (t) ^ {2} + \frac {2}{N} y ^ {2} t e ^ {- 2 \lambda t} \left[ - K ^ {1} + \Sigma^ {K} t \right] + \mathcal {O} (N ^ {- 2}) \tag {77} \\ \end{array}
$$
While the term involving  $\Sigma^K$  is positive for all  $t$ ,  $K^1$  could be positive or negative for a given architecture. If  $K^1$  is positive, then MSE is initially improved at early times but after  $t > \frac{K^1}{\Sigma^K}$  the MSE is worse than the infinite width. On the other hand, if  $K^1$  is negative (as we suspect is typically the case), then the MSE will strictly decrease with network width for any time  $t$ .
I Two Layer Equations and Time/Time Diagonal
In this section, we analyze two layer networks in greater detail. Unlike the deep network case, two layer networks can be analyzed on the time-time diagonal: ie the dynamics only depend on  $\Phi(t,t)$  and  $G(t,t)$  rather than on all possible off-diagonal pairs of time points. Further, there are no response functions  $A^{\ell}, B^{\ell}$  which complicate the recipe for calculating the propagator (Appendix E).
I.1 A Single Training Point
For a two layer network trained on a single training point with norm constraint  $|\pmb{x}|^2 = D$ , we have the following DMFT action