SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} S [ \{K (t), \hat {K} (t), \Delta (t), \hat {\Delta} (t) \} ] \tag {78} \\ = \int d t \left[ K (t) \hat {K} (t) + \hat {\Delta} (t) \left(\Delta (t) - y + \int d s \Theta (t - s) \Delta (s) K (s)\right) \right] \\ + \ln \mathcal {Z} [ \hat {K}, f ], \mathcal {Z} = \mathbb {E} _ {h, g} \exp \left(- \int d t \hat {K} (t) [ \phi (h (t)) ^ {2} + g (t) ^ {2} ]\right). \\ \end{array}
$$
The saddle point equations are
$$
\begin{array}{l} \frac {\partial S}{\partial \hat {K} (t)} = K (t) - \left\langle [ \phi (h (t)) ^ {2} + g (t) ^ {2} ] \right\rangle = 0 \\ \frac {\partial S}{\partial \hat {\Delta} (t)} = \Delta (t) - y + \int d s \Theta (t - s) \Delta (s) K (s) = 0 \\ \frac {\partial S}{\partial K (s)} = \hat {K} (s) + \Delta (s) \int d t \hat {\Delta} (t) \Theta (t - s) = 0 \\ \frac {\partial S}{\partial \Delta (s)} = \hat {\Delta} (s) + K (s) \int d t \hat {\Delta} (t) \Theta (t - s) = 0 \tag {79} \\ \end{array}
$$
From these equations, we can compute the entries in the Hessian of the DMFT action  $S$ . Letting
$$
\begin{array}{l} \boldsymbol {q} (t) = \left[ \begin{array}{c} \Delta (t) \\ K (t) \end{array} \right] \text {a n d} \hat {\boldsymbol {q}} (t) = \left[ \begin{array}{c} \hat {\Delta} (t) \\ \hat {K} (t) \end{array} \right] \\ \frac {\partial^ {2} S}{\partial \boldsymbol {q} (t) \partial \boldsymbol {q} (s) ^ {\top}} = \mathbf {0} \\ \frac {\partial^ {2} S}{\partial \hat {\boldsymbol {q}} (t) \partial \boldsymbol {q} (s) ^ {\top}} = \left[ \begin{array}{c c} \delta (t - s) + \Theta (t - s) K (s) & \Theta (t - s) \Delta (s) \\ - \left\langle \frac {\partial}{\partial \Delta (s)} (\phi (h (t)) ^ {2} + g (t) ^ {2}) \right\rangle & \delta (t - s) \end{array} \right] \\ \frac {\partial^ {2} S}{\partial \hat {\boldsymbol {q}} (t) \partial \hat {\boldsymbol {q}} (s) ^ {\top}} = \left[ \begin{array}{c c} 0 & 0 \\ 0 & \kappa (t, s) \end{array} \right] \tag {80} \\ \end{array}
$$
where  $\kappa (t,s) = \big{\langle}(\phi (h(t))^2 +g(t)^2)(\phi (h(s))^2 +g(s)^2)\big{\rangle} - K(t)K(s)$  is the NTK's fourth cumulant. We now vectorize our order parameters over time  $\pmb {q} = \mathrm{Vec}\{\pmb {q}(t)\}_{t\in \mathbb{R}_+}$  and  $\hat{\pmb{q}} = \mathrm{Vec}\{\hat{\pmb{q}} (t)\}_{t\in \mathbb{R}_+}$  and express the full Hessian
$$
\nabla^ {2} S = \left[ \begin{array}{c c} \mathbf {0} & \frac {\partial^ {2} S}{\partial \boldsymbol {q} \partial \hat {\boldsymbol {q}} ^ {\top}} \\ \frac {\partial^ {2} S}{\partial \boldsymbol {\hat {q}} \partial \boldsymbol {q} ^ {\top}} & \frac {\partial^ {2} S}{\partial \boldsymbol {\hat {q}} \partial \boldsymbol {\hat {q}} ^ {\top}} \end{array} \right] \Longrightarrow - [ \nabla^ {2} S ] ^ {- 1} = \left[ \begin{array}{c c} (\frac {\partial^ {2} S}{\partial \boldsymbol {\hat {q}} \partial \boldsymbol {q} ^ {\top}}) ^ {- 1} \frac {\partial^ {2} S}{\partial \boldsymbol {\hat {q}} \partial \boldsymbol {\hat {q}}} (\frac {\partial^ {2} S}{\partial \boldsymbol {q} \partial \boldsymbol {\hat {q}} ^ {\top}}) ^ {- 1} & - (\frac {\partial^ {2} S}{\partial \boldsymbol {\hat {q}} \partial \boldsymbol {q} ^ {\top}}) ^ {- 1} \\ - (\frac {\partial^ {2} S}{\partial \boldsymbol {q} \partial \boldsymbol {\hat {q}} ^ {\top}}) ^ {- 1} & \mathbf {0} \end{array} \right] \tag {81}
$$
The covariance matrix of interest (for  $\pmb{q}(t)$ ) is thus
$$
\boldsymbol {\Sigma} _ {q} = \left[ \begin{array}{c c} \mathbf {I} + \boldsymbol {\Theta} _ {K} & \boldsymbol {\Theta} _ {\Delta} \\ - \boldsymbol {D} & \mathbf {I} \end{array} \right] ^ {- 1} \left[ \begin{array}{l l} 0 & 0 \\ 0 & \boldsymbol {\kappa} \end{array} \right] \left[ \begin{array}{c c} \mathbf {I} + \boldsymbol {\Theta} _ {K} & \boldsymbol {\Theta} _ {\Delta} \\ - \boldsymbol {D} & \mathbf {I} \end{array} \right] ^ {- 1 ^ {\top}}. \tag {82}
$$
where  $[\Theta_K](t,s) = \Theta (t - s)K(s)$  and  $[\Theta_{\Delta}](t,s) = \Theta (t - s)\Delta (s)$ . The above equations allow one to use the infinite width DMFT dynamics for  $K(t),\Delta (t)$  to compute the finite size fluctuation dynamics of the kernel  $K$  and the error signal  $\Delta$ .
I.1.1 Computing Field Sensitivities
In this section, we compute  $D(t,s)$  by solving for the sensitivity of order parameters. We start with the DMFT field equations
$$
h (t) = u + \gamma \int_ {0} ^ {t} d s \Delta (s) g (s), z (t) = r + \gamma \int_ {0} ^ {t} d s \Delta (s) \phi (h (t)). \tag {83}
$$
Now, differentiating both sides with respect to  $\Delta(s')$  gives
$$
\frac {\partial h (t)}{\partial \Delta \left(s ^ {\prime}\right)} = \gamma \Theta \left(t - s ^ {\prime}\right) g \left(s ^ {\prime}\right) + \gamma \int_ {0} ^ {t} d s \Delta (s) \frac {\partial g (s)}{\partial \Delta \left(s ^ {\prime}\right)}
$$
$$
\frac {\partial z (t)}{\partial \Delta \left(s ^ {\prime}\right)} = \gamma \Theta \left(t - s ^ {\prime}\right) \phi \left(h \left(s ^ {\prime}\right)\right) + \gamma \int_ {0} ^ {t} d s \Delta (s) \frac {\partial \phi (h (s))}{\partial \Delta \left(s ^ {\prime}\right)}. \tag {84}
$$
We can compute  $D$  Monte carlo by iteratively solving the above equations for each sampled trajectory  $\{h(t), z(t)\}$  [65, 46]. Averaging the necessary fields over the Monte Carlo samples will give us the final expressions for  $D(t, s)$ .
$$
D (t, s) = \left\langle \frac {\partial}{\partial \Delta (s)} \left(\phi (h (t)) ^ {2} + g (t) ^ {2}\right) \right\rangle \tag {85}
$$
Similarly, the uncoupled kernel variance  $\kappa (t,s)$  can be evaluated via Monte Carlo sampling for nonlinear networks.
I.2 Test Point Fluctuation Dynamics
We now are in a position to calculate the test/train kernel and test prediction fluctuations. To do this systematically, we augment  $S$  with the test point prediction  $f_{\star}$  and field  $h_\star$  and introduce the kernel
$K_{\star}(t) = \langle \phi (h(t))\phi (h_{\star}(t)) + g(t)g_{\star}(t)\rangle$  . The test prediction  $f_{\star}$  and field  $h_\star$  have dynamics
$$
h _ {\star} (t) = u _ {\star} + \gamma \int_ {0} ^ {t} d s \Delta (s) \dot {\phi} (h _ {\star} (s)) z (s) K _ {\star} ^ {x}, \langle u _ {\star} u \rangle = K _ {\star} ^ {x}
$$
$$
\frac {\partial}{\partial t} f _ {\star} (t) = K _ {\star} (t) \Delta (t), K _ {\star} (t) = \left\langle \phi (h (t)) \phi \left(h _ {\star} (t)\right) + g (t) g _ {\star} (t) \right\rangle \tag {86}
$$
The augmented action for this DMFT has the form
$$
\begin{array}{l} S = \int d t \hat {f} _ {\star} (t) \left(f _ {\star} (t) - \int d s \Theta (t - s) \Delta (s) K _ {\star} (s)\right) + \int d t \hat {K} _ {\star} (t) K _ {\star} (t) \\ + \int d t \hat {\Delta} (t) \left(\Delta (t) - y + \int d s \Theta (t - s) \Delta (s) K (s)\right) + \int d t \hat {K} (t) K (t) \\ + \ln \mathbb {E} \exp \left(- \int \hat {K} (t) (\phi (h (t)) ^ {2} + g (t) ^ {2}) - \int \hat {K} _ {\star} (t) (\phi (h (t)) \phi (h _ {\star} (t)) + g (t) g _ {\star} (t))\right) \tag {87} \\ \end{array}
$$
We let  $\pmb{q}(t) = [\Delta(t), f_{\star}(t), K(t), K_{\star}(t)]^{\top}$ .
$$
\nabla_ {\hat {\boldsymbol {q}} \hat {\boldsymbol {q}}} ^ {2} S [ \boldsymbol {q}, \hat {\boldsymbol {q}} ] = \left[ \begin{array}{c c c c} 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & \boldsymbol {\kappa} & \boldsymbol {\kappa} _ {\star} ^ {\top} \\ 0 & 0 & \boldsymbol {\kappa} _ {\star} & \boldsymbol {\kappa} _ {\star \star} \end{array} \right], \nabla_ {\hat {\boldsymbol {q}}, \boldsymbol {q}} ^ {2} S [ \boldsymbol {q}, \hat {\boldsymbol {q}} ] = \left[ \begin{array}{c c c c} \mathbf {I} + \boldsymbol {\Theta} _ {K} & 0 & \boldsymbol {\Theta} _ {\Delta} & 0 \\ - \boldsymbol {\Theta} _ {K _ {\star}} & \mathbf {I} & 0 & - \boldsymbol {\Theta} _ {\Delta} \\ - \boldsymbol {D} & 0 & \mathbf {I} & 0 \\ - \boldsymbol {D} _ {\star} & 0 & 0 & \mathbf {I} \end{array} \right]
$$
$$
D (t, s) = \left\langle \frac {\partial}{\partial \Delta (s)} \left(\phi \left(h (t)\right) ^ {2} + g (t) ^ {2}\right) \right\rangle \tag {88}
$$
$$
D _ {\star} (t, s) = \left\langle \frac {\partial}{\partial \Delta (s)} (\phi (h (t)) \phi (h _ {\star} (t)) + g (t) g _ {\star} (t)) \right\rangle \tag {89}
$$
Our total covariance matrix / propagator is thus
$$
\boldsymbol {\Sigma} = \left[ \begin{array}{c c c c} \mathbf {I} + \boldsymbol {\Theta} _ {K} & 0 & \boldsymbol {\Theta} _ {\Delta} & 0 \\ - \boldsymbol {\Theta} _ {K _ {\star}} & \mathbf {I} & 0 & - \boldsymbol {\Theta} _ {\Delta} \\ - D & 0 & \mathbf {I} & 0 \\ - D _ {\star} & 0 & 0 & \mathbf {I} \end{array} \right] ^ {- 1} \left[ \begin{array}{c c c c} 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & \boldsymbol {\kappa} & \boldsymbol {\kappa} _ {\star} ^ {\top} \\ 0 & 0 & \boldsymbol {\kappa} _ {\star} & \boldsymbol {\kappa} _ {\star \star} \end{array} \right] \left[ \begin{array}{c c c c} \mathbf {I} + \boldsymbol {\Theta} _ {K} & 0 & \boldsymbol {\Theta} _ {\Delta} & 0 \\ - \boldsymbol {\Theta} _ {K _ {\star}} & \mathbf {I} & 0 & - \boldsymbol {\Theta} _ {\Delta} \\ - D & 0 & \mathbf {I} & 1 \\ - D _ {\star} & 0 & 0 & \mathbf {I} \end{array} \right] ^ {- 1 ^ {\top}} \tag {90}
$$
This is the equation provided in the main text Equation (8).
I.3 Two Layer Linear Network Closed Form
For a linear network on a single data point, we can compute  $D(t,s)$  and  $\kappa (t,s)$  analytically. We start from the field equations
$$
\frac {d h (t)}{d t} = \gamma \Delta (t) z (t), \frac {d z (t)}{d t} = \gamma \Delta (t) h (t) \tag {91}
$$
We can make a change of variables  $v_{+}(t) = \frac{1}{\sqrt{2}} (h(t) + z(t))$  and  $v_{-}(t) = \frac{1}{\sqrt{2}} (h(t) - z(t))$ . We note that  $v_{+}(0) = \frac{1}{\sqrt{2}} (u + r)$  and  $v_{-}(0) = \frac{1}{\sqrt{2}} (u - r)$  are independent Gaussians. These functions  $v_{+}(t), v_{-}(t)$  satisfy dynamics
$$
\begin{array}{l} \frac {d v _ {+}}{d t} = \gamma \Delta (t) v _ {+} (t), \frac {d v _ {-} (t)}{d t} = - \gamma \Delta (t) v _ {-} (t) \\ \Rightarrow v _ {+} (t) = \exp \left(\gamma \int_ {0} ^ {t} d s \Delta (s)\right) v _ {+} (0) \Rightarrow \frac {\partial v _ {+} (t)}{\partial \Delta (s)} = \gamma v _ {+} (t) \Theta (t - s) \\ \Rightarrow v _ {-} (t) = \exp \left(- \gamma \int_ {0} ^ {t} d s \Delta (s)\right) v _ {-} (0) \Rightarrow \frac {\partial v _ {+} (t)}{\partial \Delta (s)} = - \gamma v _ {-} (t) \Theta (t - s) \tag {92} \\ \end{array}
$$
Now, we use the fact that  $v_{+}(0) = \frac{1}{\sqrt{2}} (u + r)$  and  $v_{-}(0) = \frac{1}{\sqrt{2}} (u - r)$  are independent standard normal random variables to compute  $K(t) = \langle h(t)^2 + z(t)^2 \rangle = \langle v_{+}(t)^2 + v_{-}(t)^2 \rangle$ .
$$
\begin{array}{l} D (t, s) = \frac {\partial}{\partial \Delta (s)} \left\langle h (t) ^ {2} + z (t) ^ {2} \right\rangle = 2 \gamma \left[ \left\langle v _ {+} (t) ^ {2} \right\rangle - \left\langle v _ {-} (t) ^ {2} \right\rangle \right] \Theta (t - s) \\ = 2 \gamma \left[ \exp \left(2 \gamma \int_ {0} ^ {t} d s \Delta (s)\right) - \exp \left(- 2 \gamma \int_ {0} ^ {t} d s \Delta (s)\right) \right] \Theta (t - s) \tag {93} \\ \end{array}
$$
This operator is causal ( $D(t, s) = 0$  for  $s > t$ ) as expected and vanishes as  $t \to 0$ . If we take  $\gamma \to 0$ , we have  $D(t, s) \to 0$  which agrees with our reasoning that fields  $h, z$  only depend on  $\Delta$  in the feature learning regime. Since all fields are Gaussian in the linear network case, we can use Wick's theorem to obtain the exact uncoupled kernel variance in the two layer case.
$$
\begin{array}{l} \kappa (t, s) = \left\langle (h (t) ^ {2} + z (t) ^ {2}) (h (s) ^ {2} + z (s) ^ {2}) \right\rangle - K (t) K (s) \\ = 2 \left\langle h (t) h (s) \right\rangle^ {2} + 2 \left\langle h (t) z (s) \right\rangle^ {2} + 2 \left\langle z (t) h (s) \right\rangle^ {2} + 2 \left\langle z (t) z (s) \right\rangle^ {2} \\ = \left\langle v _ {+} (t) v _ {+} (s) + v _ {-} (t) v _ {-} (s) \right\rangle^ {2} + \left\langle v _ {+} (t) v _ {+} (s) - v _ {-} (t) v _ {-} (s) \right\rangle^ {2} \tag {94} \\ \end{array}
$$
The  $v_{\pm}(t)$  functions are those given above. Using the fact that  $\langle v_{+}(0)^{2} \rangle = \langle v_{-}(0)^{2} \rangle = 1$  allows us to easily compute the single site average above.
J Multiple Samples with Whitened Data
In this section, we analyze the role that sample number plays in dynamics in a simplified model of a two layer linear network trained on whitened data. Concretely, we assume that  $\frac{\pmb{x}_{\mu} \cdot \pmb{x}_{\nu}}{D} = \delta_{\mu \nu}$ . The field equations for preactivations  $h_{\mu}(t)$  and pregradients  $z(t)$  obey
$$
\frac {d}{d t} h _ {\mu} (t) = \gamma \Delta_ {\mu} (t) z (t), \frac {d}{d t} z (t) = \gamma \sum_ {\mu = 1} ^ {P} \Delta_ {\mu} (t) h _ {\mu} (t) \tag {95}
$$
We will assume the targets have unit norm  $|\pmb{y}|^2 = 1$  and we define the projection of  $\Delta$  onto the target as  $\Delta_y(t) = \pmb{y} \cdot \Delta(t)$ . The other  $P - 1$  orthogonal components are denoted  $\Delta_{\perp}(t)$  so that  $\Delta = \Delta_y(t)\pmb{y} + \Delta_{\perp}(t)$  with  $\Delta_{\perp}(t) \cdot \pmb{y} = 0$ . At infinite width,  $\Delta_{\perp} = 0$  and our field equations become
$$
\frac {d}{d t} h _ {y} (t) = \Delta_ {y} (t) z (t), \frac {d}{d t} z (t) = \Delta_ {y} (t) h _ {y} (t), \Delta_ {\perp} (t) = 0, h _ {\perp} \sim \mathcal {N} (0, 1) \tag {96}
$$
However, at finite width  $N$ , the off-target predictions  $\Delta_{\perp}$  fluctuate over random initialization. To model all of the fluctuations simultaneously, we consider the following action
$$
S = \gamma \int d t \sum_ {\mu} \hat {\Delta} _ {\mu} (t) \left(\Delta_ {\mu} (t) - y _ {\mu}\right) + \ln \mathbb {E} \exp \left(\int d t \sum_ {\mu} \hat {\Delta} _ {\mu} (t) z (t) h _ {\mu} (t)\right) \tag {97}
$$
which enforces the constraint that  $\Delta_{\mu}(t) = y_{\mu} - \frac{1}{\gamma}\left\langle z(t)h_{\mu}(t)\right\rangle$  at infinite width. The Hessian over order parameters  $\mathbf{q} = \operatorname{Vec}\{\Delta_{\mu}(t),\hat{\Delta}_{\mu}(t)\}$  has the form
$$
\nabla_ {\boldsymbol {q}} ^ {2} S = \left[ \begin{array}{c c} \boldsymbol {0} & (\gamma \mathbf {I} + \boldsymbol {D}) ^ {\top} \\ \gamma \mathbf {I} + \boldsymbol {D} & \boldsymbol {\kappa} \end{array} \right], D _ {\mu \nu} (t, s) = \left\langle \frac {\partial}{\partial \Delta_ {\nu} (s)} z (t) h _ {\mu} (t) \right\rangle \tag {98}
$$
We thus get the following covariance for predictions  $\pmb{\Sigma}_{\Delta} = (\gamma \mathbf{I} + D)^{-1}\pmb{\kappa}\left[(\gamma \mathbf{I} + D)^{-1}\right]^{\top}$ . We now compute the necessary components of the  $D$  tensor
$$
\frac {\partial h _ {\mu} (t)}{\partial \Delta_ {\nu} (s)} = \gamma \delta_ {\mu \nu} \Theta (t - s) z (s) + \gamma \int_ {0} ^ {t} d t ^ {\prime} \Delta_ {\mu} (t ^ {\prime}) \frac {\partial z (t ^ {\prime})}{\partial \Delta_ {\nu} (s)}
$$
$$
\begin{array}{l} \frac {\partial z (t)}{\partial \Delta_ {\nu} (s)} = \gamma \Theta (t - s) h _ {\nu} (s) + \gamma \int_ {0} ^ {t} d t ^ {\prime} \sum_ {\mu} \Delta_ {\mu} (t ^ {\prime}) \frac {\partial h _ {\mu} (t ^ {\prime})}{\partial \Delta_ {\nu} (s)} \\ = \gamma \Theta (t - s) h _ {\nu} (s) + \gamma \int_ {0} ^ {t} d t ^ {\prime} \Delta_ {y} \left(t ^ {\prime}\right) \frac {\partial h _ {y} \left(t ^ {\prime}\right)}{\partial \Delta_ {\nu} (s)} \tag {99} \\ \end{array}
$$
In the last line, we used the fact that these equations are to be evaluated at the mean field infinite width stochastic process where  $\Delta_{\perp}(t) = 0$ . To compute the sensitivity tensor  $D$ , we find the following equations for our correlators of interest:
$$
\left\langle \frac {\partial h _ {\mu} (t)}{\partial \Delta_ {\nu} (s)} z (t) \right\rangle = \delta_ {\mu \nu} \gamma \Theta (t - s) \left\langle z (s) z (t) \right\rangle , \mu , \nu \neq y
$$
$$
\left\langle \frac {\partial z (t)}{\partial \Delta_ {\nu} (s)} h _ {\mu} (t) \right\rangle = \gamma \Theta (t - s) \delta_ {\mu \nu}, \mu , \nu \neq y \tag {100}
$$
$$
\left\langle \frac {\partial h _ {y} (t)}{\partial \Delta_ {y} (s)} z (t) \right\rangle = \gamma \Theta (t - s) \left\langle z (s) z (t) \right\rangle + \gamma \int_ {0} ^ {t} d t ^ {\prime} \Delta_ {y} (t ^ {\prime}) \left\langle \frac {\partial z (t ^ {\prime})}{\partial \Delta_ {y} (s)} z (t) \right\rangle
$$
$$
\left\langle \frac {\partial z (t)}{\partial \Delta_ {y} (s)} z (t ^ {\prime}) \right\rangle = \gamma \Theta (t - s) \left\langle h _ {y} (s) z (t) \right\rangle + \gamma \int_ {0} ^ {t} d t ^ {\prime \prime} \Delta_ {y} (t ^ {\prime \prime}) \left\langle \frac {\partial h _ {y} (t ^ {\prime \prime})}{\partial \Delta_ {y} (s)} z (t ^ {\prime}) \right\rangle
$$
We therefore see that the components of  $D$  decouple over indices. In the  $y$  direction, we have the following equations
$$
D _ {y} (t, s) = \left\langle \frac {\partial h _ {y} (t)}{\partial \Delta_ {y} (s)} z (t) \right\rangle + \left\langle \frac {\partial z (t)}{\partial \Delta_ {y} (s)} h _ {y} (t) \right\rangle \tag {101}
$$
where the correlators must be solved self-consistently. We will provide this solution in one moment, but first, we will look at the orthogonal directions. For the  $P - 1$  orthogonal directions, we obtain the explicit formula for  $D$  in each of these directions
$$
\begin{array}{l} D _ {\perp} (t, s) = \left\langle \frac {\partial h _ {\perp} (t)}{\partial \Delta_ {\perp} (s)} z (t) \right\rangle + \left\langle \frac {\partial z (t)}{\partial \Delta_ {\perp} (s)} h _ {\perp} (t) \right\rangle \\ = \gamma \Theta (t - s) \langle z (t) z (s) \rangle + \gamma \Theta (t - s) \tag {102} \\ \end{array}
$$
Now, we return to  $D_y$ . To solve these equations we utilize the change of variables employed in the single sample case  $v_{+}(t) = \frac{1}{\sqrt{2}} (h_y(t) + z(t))$ ,  $v_{-}(t) = \frac{1}{\sqrt{2}} (h_y(t) - z(t))$  (see Appendix I.3). This orthogonal transformation decouples the dynamics
$$
\frac {d}{d t} v _ {+} (t) = \gamma \Delta_ {y} (t) v _ {+} (t), \frac {d}{d t} v _ {-} (t) = - \gamma \Delta_ {y} (t) v _ {-} (t) \tag {103}
$$
As a consequence, the field derivatives close
$$
\frac {\partial v _ {+} (t)}{\partial \Delta_ {y} (s)} = \gamma \Theta (t - s) v _ {+} (s) + \int_ {0} ^ {t} d t ^ {\prime} \Delta_ {y} (t ^ {\prime}) \frac {\partial v _ {+} (t ^ {\prime})}{\partial \Delta_ {y} (s)}
$$
$$
\frac {\partial v _ {-} (t)}{\partial \Delta_ {y} (s)} = - \gamma \Theta (t - s) v _ {-} (s) - \int_ {0} ^ {t} d t ^ {\prime} \Delta_ {y} \left(t ^ {\prime}\right) \frac {\partial v _ {-} \left(t ^ {\prime}\right)}{\partial \Delta_ {y} (s)} \tag {104}
$$
The correlator of interest is
$$
\langle h _ {y} (t) z (t) \rangle = \frac {1}{2} \left\langle \left[ v _ {+} (t) + v _ {-} (t) \right] \left[ v _ {+} (t) - v _ {-} (t) \right] \right\rangle = \frac {1}{2} \left\langle v _ {+} (t) ^ {2} - v _ {-} (t) ^ {2} \right\rangle \tag {105}
$$
So we get that
$$
\begin{array}{l} D _ {y} (t, s) = \frac {1}{2} \left\langle \frac {\partial}{\partial \Delta_ {y} (s)} \left(v _ {+} (t) ^ {2} - v _ {-} (t) ^ {2}\right) \right\rangle \\ = \left\langle v _ {+} (t) \frac {\partial v _ {+} (t)}{\partial \Delta_ {y} (s)} \right\rangle - \left\langle v _ {-} (t) \frac {\partial v _ {-} (t)}{\partial \Delta_ {y} (s)} \right\rangle \tag {106} \\ \end{array}
$$
Similarly, we can derive the on-target and off-target uncoupled variances  $\kappa_y(t,s)$  and  $\kappa_{\perp}(t,s)$ , which satisfy
$$
\kappa_ {y} (t, s) = \left\langle v _ {+} (t) v _ {+} (s) + v _ {-} (t) v _ {-} (s) \right\rangle^ {2} + \left\langle v _ {+} (t) v _ {+} (s) - v _ {-} (t) v _ {-} (s) \right\rangle^ {2}
$$
$$
\kappa_ {\perp} (t, s) = \frac {1}{2} \left\langle v _ {+} (t) v _ {+} (s) + v _ {-} (t) v _ {-} (s) \right\rangle \tag {107}
$$
Using these functions, we arrive at the following variance for each of the  $P$  dimensions
$$
\boldsymbol {\Sigma} _ {\Delta_ {y}} = (\gamma \mathbf {I} + \boldsymbol {D} _ {y}) ^ {- 1} \boldsymbol {\kappa} _ {y} (\gamma \mathbf {I} + \boldsymbol {D} _ {y}) ^ {- 1}
$$
$$
\boldsymbol {\Sigma} _ {\Delta_ {\perp}} = \left(\gamma \mathbf {I} + \boldsymbol {D} _ {\perp}\right) ^ {- 1} \boldsymbol {\kappa} _ {\perp} \left(\gamma \mathbf {I} + \boldsymbol {D} _ {\perp}\right) ^ {- 1} \tag {108}
$$
Using the fact that all  $\Delta_{\perp}$  variables are independent and identically distributed under the leading order picture, the expected training loss has the form
$$
\left\langle | \boldsymbol {\Delta} | ^ {2} \right\rangle \approx \Delta_ {y} ^ {\infty} (t) ^ {2} + \frac {2}{N} \Delta_ {y} ^ {1} (t) \Delta_ {y} ^ {\infty} (t) + \frac {1}{N} \Sigma_ {\Delta_ {y}} (t, t) + \frac {(P - 1)}{N} \Sigma_ {\Delta_ {\perp}} (t, t) + \mathcal {O} (N ^ {- 2}). \tag {109}
$$
where  $\langle \Delta_y - \Delta_y^\infty \rangle = \frac{1}{N}\Delta_y^1 (t) + \mathcal{O}(N^{-2})$ . We note that the bias correction if  $\mathcal{O}(N^{-1})$  while the variance is  $\mathcal{O}(P / N)$ . We compare the above leading order theory with and without the bias correction in Appendix Figure A.2.
K Online Learning
Our technology for computing finite size effects can easily be translated to a setting where the neural network is trained in an online fashion, disregarding the effect of SGD noise. At each step, we compute the gradient over the full data distribution  $p(x)$ . Focusing on MSE loss, we study the following equation
$$
\frac {d}{d t} \Delta (\boldsymbol {x}, t) = - \mathbb {E} _ {\boldsymbol {x} ^ {\prime} \sim p \left(\boldsymbol {x} ^ {\prime}\right)} K \left(\boldsymbol {x}, \boldsymbol {x} ^ {\prime}; t\right) \Delta \left(\boldsymbol {x} ^ {\prime}, t\right) \tag {110}
$$
where  $K(\pmb{x},\pmb{x}';t)$  is the dynamic NTK and  $\Delta (\pmb{x},t) = y(\pmb{x}) - f(\pmb{x},t)$  is the prediction error. In general the distribution involves integration over an uncountable set of possible inputs  $\pmb{x}$ . To remedy this, we utilize a countable orthonormal basis of functions for the data distribution  $\{\psi_k(\pmb{x})\}_{k=1}^{\infty}$ . For example, if  $p(\pmb{x})$  were the isotropic Gaussian density for  $\mathcal{N}(0,\mathbf{I})$ , then  $\psi_k$  could be Hermite polynomials. We expand  $\Delta$  and  $K$  in this basis  $\psi_k$ , and arrive at the following differential equation
$$
\frac {d}{d t} \Delta_ {k} (t) = - \sum_ {\ell} K _ {k \ell} (t) \Delta_ {\ell} (t) \tag {111}
$$
By orthonormality, the average turned into a sum over all possible orthonormal functions  $\{\psi_k\}$ . We note that since  $K$  is evolving in time, there is not generally a fixed basis of functions that diagonalize  $K$ , resulting in the couplings across eigenmodes in Equation (111). Since, in online learning, there is no distinction between the training and test distribution, our error of interest is simply  $\mathcal{L}(t) = \sum_{k} \Delta_k(t)^2$ . To obtain the finite size corrections to this quantity, we compute the joint propagator for all variables  $\{K_{k\ell}(t), \Delta_k(t)\}$ . If we wanted to pursue a perturbation theory in rates (Appendix G.2), we could again define a transition matrix  $\pmb{T}$  and rate matrix  $\pmb{R}(t)$  as
$$
\boldsymbol {R} (t) = - \log \boldsymbol {T} (t), \frac {d}{d t} T _ {k \ell} (t) = - \sum_ {k ^ {\prime}} K _ {k k ^ {\prime}} (t) T _ {k ^ {\prime} \ell} (t), T _ {k \ell} (0) = \delta_ {k \ell} \tag {112}
$$
We can then obtain  $\pmb{\Delta} = \exp(-\pmb{R}(t))\pmb{y}$ , where  $y_{k} = \mathbb{E}_{\pmb{x}}\psi_{k}(\pmb{x})y(\pmb{x})$ . Since  $\pmb{R}$  has a finite size mean correction and finite size fluctuations, so too does the error  $\Delta_k(t)$  and the loss  $\mathcal{L}$  (Appendix G.2).
K.1 Two Layer Networks
In the two layer case, instead of tracking kernels, we could instead deal with the distribution over read-in vectors  $\boldsymbol{w} \in \mathbb{R}^{D}$  and readout scalars  $a \in \mathbb{R}$  as in the original works on mean field networks [6, 66]. When training on the population risk equations for  $\boldsymbol{x} \sim \mathcal{N}(0, \mathbf{I})$
$$
\frac {d}{d t} \boldsymbol {w} = a \mathbb {E} _ {\boldsymbol {x}} \Delta (\boldsymbol {x}) \dot {\phi} (\boldsymbol {w} \cdot \boldsymbol {x}) \boldsymbol {x} = \mathbb {E} _ {\boldsymbol {x}} \frac {\partial \Delta (\boldsymbol {x})}{\partial \boldsymbol {x}} \dot {\phi} (\boldsymbol {w} \cdot \boldsymbol {x}) + \mathbb {E} \Delta (\boldsymbol {x}) \ddot {\phi} (\boldsymbol {w} \cdot \boldsymbol {x}) \boldsymbol {w}
$$
$$
\frac {d}{d t} a = \mathbb {E} _ {\boldsymbol {x}} \Delta (\boldsymbol {x}) \phi (\boldsymbol {w} \cdot \boldsymbol {x}) \tag {113}
$$
The action has the form