SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31, pages 8571-8580. Curran Associates, Inc., 2018.
[2] Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances in Neural Information Processing Systems, 32, 2019.
[3] Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems, 32, 2019.
[4] Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in Neural Information Processing Systems, 32, 2019.
[5] Greg Yang and Etai Littwin. Tensor programs iib: Architectural universality of neural tangent kernel training dynamics. In International Conference on Machine Learning, pages 11762-11772. PMLR, 2021.
[6] Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit. In Conference on Learning Theory, pages 2388-2464. PMLR, 2019.
[7] Mario Geiger, Stefano Spigler, Arthur Jacot, and Matthieu Wyart. Disentangling feature and lazy training in deep neural networks. Journal of Statistical Mechanics: Theory and Experiment, 2020(11):113301, 2020.
[8] Greg Yang and Edward J Hu. Tensor programs iv: Feature learning in infinite-width neural networks. In International Conference on Machine Learning, pages 11727-11737. PMLR, 2021.
[9] Blake Bordelon and Cengiz Pehlevan. Self-consistent dynamical field theory of kernel evolution in wide neural networks. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[10] Jiaoyang Huang and Horng-Tzer Yau. Dynamics of deep neural networks and neural tangent hierarchy. In International conference on machine learning, pages 4542-4551. PMLR, 2020.
[11] Ethan Dyer and Guy Gur-Ari. Asymptotics of wide networks from feynman diagrams. In International Conference on Learning Representations, 2020.
[12] Anders Andreassen and Ethan Dyer. Asymptotics of wide convolutional neural networks. arXiv preprint arXiv:2008.08675, 2020.
[13] Daniel A Roberts, Sho Yaida, and Boris Hanin. The principles of deep learning theory. Cambridge University Press Cambridge, MA, USA, 2022.
[14] Boris Hanin. Random fully connected neural networks as perturbatively solvable hierarchies. arXiv preprint arXiv:2204.01058, 2022.
[49] Jean Zinn-Justin. Quantum field theory and critical phenomena, volume 171. Oxford university press, 2021.
[50] Carson C Chow and Michael A Buice. Path integral methods for stochastic differential equations. The Journal of Mathematical Neuroscience (JMN), 5:1-35, 2015.
[51] Moritz Helias and David Dahmen. Statistical Field Theory for Neural Networks. Springer International Publishing, 2020.
[52] A Crisanti and H Sompolinsky. Path integral approach to random neural networks. Physical Review E, 98(6):062120, 2018.
[53] Kai Segadlo, Bastian Epping, Alexander van Meegen, David Dahmen, Michael Kramer, and Moritz Helias. Unified field theoretical approach to deep and recurrent neuronal networks. Journal of Statistical Mechanics: Theory and Experiment, 2022(10):103401, 2022.
[54] Yizhang Lou, Chris E Mingard, and Soufiane Hayou. Feature learning and signal propagation in deep neural networks. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 14248-14282. PMLR, 17-23 Jul 2022.
[55] Carl M Bender and Steven Orszag. Advanced mathematical methods for scientists and engineers I: Asymptotic methods and perturbation theory, volume 1. Springer Science & Business Media, 1999.
[56] Mehran Kardar. Statistical physics of fields. Cambridge University Press, 2007.
[57] Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The large learning rate phase of deep learning: the catapult mechanism. arXiv preprint arXiv:2003.02218, 2020.
[58] Ben Adlam and Jeffrey Pennington. The neural tangent kernel in high dimensions: Triple descent and a multi-scale theory of generalization. In International Conference on Machine Learning, pages 74-84. PMLR, 2020.
[59] Soufiane Hayou. On the infinite-depth limit of finite-width neural networks. Transactions on Machine Learning Research, 2023.
[60] Blake Bordelon, Lorenzo Noci, Mufan Bill Li, Boris Hanin, and Cengiz Pehlevan. Depthwise hyperparameter transfer in residual networks: Dynamics and scaling limit. arXiv preprint arXiv:2309.16620, 2023.
[61] Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc van Zee. Flax: A neural network library and ecosystem for JAX, 2023.
[62] Michael Baake and Ulrike Schlaegel. The peano-baker series. Proceedings of the Steklov Institute of Mathematics, 275(1):155-159, 2011.
[63] Roger W Brockett. Finite dimensional linear systems. SIAM, 2015.
[64] Alexander Atanasov, Blake Bordelon, and Cengiz Pehlevan. Neural networks as kernel learners: The silent alignment effect. In International Conference on Learning Representations, 2022.
[65] B. Bordelon, A. Canatar, and C. Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. International Conference of Machine Learning, 2020.
[66] Phan-Minh Nguyen. Mean field limit of the learning dynamics of multilayer neural networks. arXiv preprint arXiv:1902.02880, 2019.
A Additional Figures
(a) Fourth Moment  $\kappa (t,t)$
(b) Kernel sensitivity  $D(t,100)$
Figure A.1: The  $\kappa$  and  $D$  functions for varying  $\gamma$  in Figure 3. (a) The uncoupled kernel variance  $\kappa(t, t)$  increases monotonically with  $\gamma$ . This reveals that the dynamical filtering of  $\kappa$  is what is responsible for the late time decrease in variance during feature learning. (b) The tensor  $D(t, s)$  measures sensitivity of kernel at time  $t$  to perturbation in  $\Delta$  at time  $s$ . The  $D(t, s)$  entries also increase with  $\gamma$ . This suggests that the reduction in variance of the training error and the kernel are not due to reduction in  $\kappa$ , but rather a dynamical filtering effect due to scale growth in  $K_{\infty}$  and rapid reduction in  $\Delta_{\infty}$ .
Figure A.2: A comparison of the bias and variance corrections in the toy model of Figure 4. At small  $D / N$  (or  $P / N$  for offline training) the leading variance and the leading variance and leading bias both track the experiment. Both the bias and the variance contribute positively towards the total generalization error since the variance correction alone (orange) exceeds the DMFT limiting error (dashed) and the variance and bias correction together (green) exceed variance alone (orange). However, for large  $D / N$  (or  $P / N$ ) the leading order picture fails to describe the finite width experiment, indicating significant variance possibly at higher order scales (like  $D^2 / N^2$ ,  $D^3 / N^3$ , ...).
Figure A.3: The covariance of kernel entries across pairs of time points  $\Sigma_{H^{\ell}}(t,s) = N\operatorname{Cov}(H^{\ell}(t,t),H^{\ell}(s,s))$  for depth 4 linear network trained on whitened data. The variance becomes increasingly localized in time as feature learning  $\gamma$  increases.
(a) Ens. Averaged Train Loss
(b) Raleigh Quotient  $\frac{\Delta^{\top}K\Delta}{|\Delta|^{2}}$
Figure A.4: The ensemble averaged train loss for the same depth 6 CNN trained on a random subsample of  $P = 64$  CIFAR-10 points. Training is full batch gradient descent with  $\gamma = 0.05$ . (a) The ensemble train accuracy for this subset of CIFAR-10 is well modeled as an exponential in time  $\mathcal{L}(t) \propto \exp(-R_N t)$  with a rate  $R_N$  that depends on width. (b) The projection of the errors  $\Delta$  on the average NTK  $\langle K \rangle$  (which is related to the rate of decay of the training loss, see Appendix G) reveals that wider networks are more aligned with their instantaneous error signals. (c) The rates  $R_N$  are indeed a linear function of  $N^{-1}$ , with  $R_N = R_{\infty} + \frac{R_1^1}{N}$ , consistent with the average NTK  $\langle K \rangle$  receiving a  $N^{-1}$  correction. Using ensembling to find a scaling law like that above can thus allow one extrapolate the training rate of infinite width mean field models.
(c) Rates as Function of  $N$
(a) Training MSE
(b) Kernel-Task Alignment
(c) Test Acc.
(d) Single vs Ensembled Ratio
Figure A.5: Width  $N = 64$  depth 6 CNNs trained on the full CIFAR-10 with MSE. An ensemble of size  $E = 10$  randomly initialized networks are trained. (a) Training MSE for varying  $\gamma$ . (b) Final layer kernel-task alignment does strongly depend on  $\gamma$ , despite similar train dynamics. (c) Top-1 classification test accuracy is only slightly different across  $\gamma$ . A small benefit from ensembling is visible late in training. (c) Initialization variance (measured by the ratio of single model to ensembled MSE) for training and test losses. Richer networks have lower variance throughout training. (b) Networks have distinct kernel dynamics when trained with different  $\gamma$  as evidenced by the alignment (cosine similarity) between the final layer feature kernel  $\Phi^L$  and the target test labels  $\mathbf{y}$ .
(e) Single vs Ensembled Ratio
B CIFAR-10 Experimental Details
We trained the following depth 6 CNN architecture in the mean field parameterization using FLAX [61] on a single GPU. The bias parameters were zero in each hidden Conv layer, but were used for the readout weights. The networks were trained with MSE loss on centered 10 dimensional targets  $\pmb{y}_{\mu} \in \mathbb{R}^{10}$  for  $\mu \in [P]$ . Each convolution was followed by an average pooling operation. To obtain mean field behavior, NTK parameterization with a modified final layer is used [7, 9].
from flax import linen as nn   
import jax numpy as jnp   
class CNN(nnModule):   
width: int   
def setup(self): kif  $=$  nn.initializers.normal(stddev  $= 1.0$  # O_N(1) entries self.conv1  $=$  nn.Conv features  $=$  self.width, kernel_init  $=$  kif, use.bias  $=$  False, kernel_size  $= (3,3)$  self.conv2  $=$  nn.Conv features  $=$  self.width, kernel_init  $=$  kif, use.bias  $=$  False, kernel_size  $= (3,3)$  self.conv3  $=$  nn.Conv features  $=$  self.width, kernel_init  $=$  kif, use.bias  $=$  False, kernel_size  $= (3,3)$  self.conv4  $=$  nn.Conv features  $=$  self.width, kernel_init  $=$  kif, use.bias  $=$  False, kernel_size  $= (3,3)$  self.conv5  $=$  nn.Conv features  $=$  self.width, kernel_init  $=$  kif, use.bias  $=$  False, kernel_size  $= (3,3)$  self.readout  $\equiv$  nn.Dense(features  $= 10$  ,use.bias  $\equiv$  True, kernel_init  $\equiv$  kif) return   
def __call__(self,x,train  $\equiv$  True): N  $=$  self.width D  $= 3$  x = self.conv1(x)/ jnp.sqrt(D\*9) x = jnp.sqrt(2.0)*nnrelu(x) x = nn_avg_pool(x, window_shape=(2,2), strides  $\equiv$  (2,2)) # 32 x 32 -> 16 x 16 x = self.conv2(x)/ jnp.sqrt(N\*9)# explicit N{-1/2} x = jnp.sqrt(2.0)*nnrelu(x) x = nn_avg_pool(x, window_shape=(2,2), strides  $\equiv$  (2,2)) # 16 x 16 -> 8 x 8 x = self.conv3(x)/jnp.sqrt(N\*9) x = jnp.sqrt(2.0)*nnrelu(x) x = nn_avg_pool(x, window_shape=(2,2), strides  $\equiv$  (2,2)) # 8 x 8 -> 4 x 4 x = self.conv4(x)/ jnp.sqrt(N\*9) x = jnp.sqrt(2.0)*nnrelu(x) x = nn_avg_pool(x, window_shape=(2,2), strides  $\equiv$  (2,2)) # 4 x 4 -> 2 x 2 x = self.conv5(x)/ jnp.sqrt(N\*9) x = jnp.sqrt(2.0)*nnrelu(x) x = nn_avg_pool(x, window_shape=(2,2), strides  $\equiv$  (2,2)) # 2 x 2 -> 1 x 1 x = x.reshape((x.shape[0], -1)) # flatten x = self.readout(x)/N# for mean field scaling return x
All models were trained with standard SGD with a batch size of 256. Each element in the ensemble of  $E$  networks is trained on identical batches presented in identical order. For the Figure 7 experiments, the raw learning rate is scaled as  $\eta = 10N\sqrt{\gamma}$  with  $\gamma = 0.2$  (note that mean field theory requires scaling the raw learning rate linearly with  $N$  since the raw NTK is  $\mathcal{O}(N^{-1})$  [9]). For Figure A.5, the learning rate is  $\eta = 5N\sqrt{\gamma}$ . We find that choosing  $\eta \propto \sqrt{\gamma}$  gives approximately conserved training
times across  $\gamma$  (though distinct representation dynamics). The Figure A.4 shows the dynamics of fitting  $P = 64$  training points with full batch gradient descent and  $\gamma = 0.1$ .
C Review of DMFT: Deriving the Action
In this section we derive the DMFT action which contains all of the necessary statistical information about randomly initialized finite width  $N$  networks. From the action  $S$  the DMFT saddle point and the propagator can be computed. This derivation follows closely the original derivation by Bordelon & Pehlevan [9]. We start by writing the gradient flow dynamics on weight matrices
$$
\frac {d}{d t} \boldsymbol {W} ^ {\ell} (t) = \frac {\gamma}{\sqrt {N}} \sum_ {\mu = 1} ^ {P} \Delta_ {\mu} (t) \boldsymbol {g} _ {\mu} ^ {\ell + 1} (t) \phi \left(\boldsymbol {h} _ {\mu} ^ {\ell} (t)\right) ^ {\top} \tag {11}
$$
where  $\Delta_{\mu}(t) = -\frac{\partial\mathcal{L}}{\partial f_{\mu}(t)}$  are the error signals and  $g_{\mu}^{\ell}(t) = N\gamma \frac{\partial f_{\mu}(t)}{\partial h_{\mu}^{\ell}(t)}$  are the back-propagation signals. The prediction dynamics satisfy
$$
\frac {d}{d t} f _ {\mu} (t) = \sum_ {\nu} K _ {\mu \nu} (t) \Delta_ {\nu} (t) \tag {12}
$$
where  $K_{\mu \nu}(t)$  is the instantaneous neural tangent kernel (NTK). At finite width  $N$  all of the above quantities depend on the precise initialization of the network. We transform the weight dynamics into an integral equation and use the recurrence for  $h^\ell$  to obtain the following
$$
\begin{array}{l} \boldsymbol {h} ^ {\ell + 1} (t) = \frac {1}{\sqrt {N}} \boldsymbol {W} ^ {\ell} (0) \phi (\boldsymbol {h} _ {\mu} ^ {\ell} (t)) + \gamma \int_ {0} ^ {t} d s \sum_ {\nu} \Phi_ {\mu \nu} ^ {\ell} (t, s) \boldsymbol {g} _ {\nu} ^ {\ell + 1} (s) \\ \boldsymbol {g} _ {\mu} ^ {\ell} (t) = \dot {\phi} (\boldsymbol {h} _ {\mu} ^ {\ell} (t)) \odot \boldsymbol {z} _ {\mu} ^ {\ell} (t) \\ \boldsymbol {z} _ {\mu} ^ {\ell} (t) = \frac {1}{\sqrt {N}} \boldsymbol {W} ^ {\ell} (0) ^ {\top} \boldsymbol {g} _ {\mu} ^ {\ell + 1} (t) + \gamma \int_ {0} ^ {t} d s \sum_ {\nu} G _ {\mu \nu} ^ {\ell + 1} (t, s) \phi \left(\boldsymbol {h} _ {\nu} ^ {\ell} (s)\right) \tag {13} \\ \end{array}
$$
where we introduced the feature and gradient kernels
$$
\Phi_ {\mu \nu} ^ {\ell} (t, s) = \frac {1}{N} \phi \left(\boldsymbol {h} _ {\mu} ^ {\ell} (t)\right) \cdot \phi \left(\boldsymbol {h} _ {\nu} ^ {\ell} (s)\right), G _ {\mu \nu} ^ {\ell} (t, s) = \frac {1}{N} \boldsymbol {g} _ {\mu} ^ {\ell} (t) \cdot \boldsymbol {g} _ {\nu} ^ {\ell} (s). \tag {14}
$$
Written this way, we see that the source of the disorder which depends on the initial random weights  $W^{\ell}(0)$  comes through the fields
$$
\chi_ {\mu} ^ {\ell + 1} (t) = \frac {1}{\sqrt {N}} \boldsymbol {W} ^ {\ell} (0) \phi \left(\boldsymbol {h} _ {\mu} ^ {\ell} (t)\right), \boldsymbol {\xi} _ {\mu} ^ {\ell + 1} (t) = \frac {1}{\sqrt {N}} \boldsymbol {W} ^ {\ell} (0) ^ {\top} \boldsymbol {g} _ {\mu} ^ {\ell} (t). \tag {15}
$$
If we can characterize the distribution of the fields  $\chi_{\mu}^{\ell}(t)$  and  $\xi_{\mu}^{\ell}(t)$ , then we can consequently characterize the distribution of  $h_{\mu}^{\ell}(t), g_{\mu}^{\ell}(t)$ . We therefore choose to study the moment generating functional
$$
Z \left[ \left\{\boldsymbol {j} ^ {\ell}, \boldsymbol {v} ^ {\ell} \right\} \right] = \left\langle \exp \left(\sum_ {\ell \mu} \int d t \left[ \boldsymbol {j} _ {\mu} ^ {\ell} (t) \cdot \boldsymbol {\chi} _ {\mu} ^ {\ell} (t) + \boldsymbol {v} _ {\mu} ^ {\ell} (t) \cdot \boldsymbol {\chi} _ {\mu} ^ {\ell} (t) \right]\right) \right\rangle_ {\boldsymbol {\theta} _ {0}} \tag {16}
$$
Moments of these fields can be computed through differentiation with respect to the sources  $\mathbf{j}$ ,  $\mathbf{v}$  near zero-source  $(\mathbf{j} = \mathbf{v} = 0)$
$$
\begin{array}{l} \left\langle \chi_ {\mu 1} ^ {\ell_ {1}} \left(t _ {1}\right)... \chi_ {\mu n} ^ {\ell_ {n}} \left(t _ {n}\right) \xi_ {\mu 1} ^ {\ell_ {1}} \left(t _ {1}\right)... \xi_ {\mu m} ^ {\ell_ {m}} \left(t _ {m}\right) \right\rangle \\ = \frac {\delta}{\delta j _ {\mu_ {1}} ^ {\ell_ {1}} (t _ {1})} \dots \frac {\delta}{\delta j _ {\mu_ {n}} ^ {\ell_ {n}} (t _ {n})} \frac {\delta}{\delta v _ {\mu_ {1}} ^ {\ell_ {1}} (t _ {1})} \dots \frac {\delta}{\delta v _ {\mu_ {m}} ^ {\ell_ {m}} (t _ {m})} Z [ \left\{\boldsymbol {j} ^ {\ell}, \boldsymbol {v} ^ {\ell} \right\} ] | _ {\boldsymbol {j} = \boldsymbol {v} = 0}. \tag {17} \\ \end{array}
$$
To average over the initial weights, we introduce a Fourier representation of the Dirac-Delta function  $1 = \int dz\delta (z) = \int \frac{dzd\hat{z}}{2\pi}\exp (i\hat{z} z)$ . We perform this transformation for each of the fields to enforce
their definition
$$
\begin{array}{l} \delta \left(\pmb {\chi} _ {\mu} ^ {\ell} (t) - \frac {1}{\sqrt {N}} \pmb {W} ^ {\ell} (0) \phi (\pmb {h} _ {\mu} ^ {\ell} (t))\right) = \int \frac {d \hat {\pmb {\chi}} _ {\mu} ^ {\ell} (t)}{(2 \pi) ^ {N}} \exp \left(\hat {\pmb {\chi}} _ {\mu} ^ {\ell} (t) \cdot \left[ \pmb {\chi} _ {\mu} ^ {\ell} (t) - \frac {1}{\sqrt {N}} \pmb {W} ^ {\ell} (0) \phi (\pmb {h} _ {\mu} ^ {\ell} (t)) \right]\right) \\ \left. \delta \left(\boldsymbol {\xi} _ {\mu} ^ {\ell} (t) - \frac {1}{\sqrt {N}} \boldsymbol {W} ^ {\ell} (0) ^ {\top} \boldsymbol {g} _ {\mu} ^ {\ell} (t)\right) = \int \frac {d \hat {\boldsymbol {\xi}} _ {\mu} ^ {\ell} (t)}{(2 \pi) ^ {N}} \exp \left(\hat {\boldsymbol {\xi}} _ {\mu} ^ {\ell} (t) \cdot \left[ \boldsymbol {\xi} _ {\mu} ^ {\ell} (t) - \frac {1}{\sqrt {N}} \boldsymbol {W} ^ {\ell} (0) ^ {\top} \boldsymbol {g} _ {\mu} ^ {\ell + 1} (t) \right]\right) \right. \tag {18} \\ \end{array}
$$
We insert these Dirac delta functions so that we can directly average over the weights
$$
\begin{array}{l} \ln \mathbb {E} _ {\boldsymbol {W} ^ {\ell} (0)} \exp \left(- \frac {i}{\sqrt {N}} \operatorname {T r} \boldsymbol {W} ^ {\ell} (0) ^ {\top} \int d t \sum_ {\mu} \left[ \hat {\boldsymbol {\chi}} _ {\mu} ^ {\ell + 1} (t) \phi \left(\boldsymbol {h} _ {\mu} ^ {\ell} (t)\right) ^ {\top} + \boldsymbol {g} _ {\mu} ^ {\ell + 1} (t) \hat {\boldsymbol {\xi}} _ {\mu} ^ {\ell} (t) ^ {\top} \right]\right) \\ = - \frac {1}{2} \sum_ {\mu \nu} \int d t d s \left[ \hat {\chi} _ {\mu} ^ {\ell + 1} (t) \cdot \hat {\chi} _ {\nu} ^ {\ell + 1} (s) \Phi_ {\mu \nu} ^ {\ell} (t, s) + \hat {\xi} _ {\mu} ^ {\ell} (t) \cdot \hat {\xi} _ {\nu} ^ {\ell} (s) G _ {\mu \nu} ^ {\ell + 1} (t, s) \right] \\ - \frac {1}{N} \sum_ {\mu \nu} \int d t d s \left(\hat {\chi} _ {\mu} ^ {\ell + 1} (t) \cdot g _ {\nu} ^ {\ell + 1} (s)\right) \left(\phi \left(h _ {\mu} ^ {\ell} (t)\right) \cdot \hat {\xi} _ {\nu} ^ {\ell} (s)\right) \tag {19} \\ \end{array}
$$
where we introduced the kernels  $\Phi^{\ell}, G^{\ell}$ . We next introduce the order parameter
$$
A _ {\mu \nu} ^ {\ell} (t, s) = - \frac {i}{N} \phi \left(\boldsymbol {h} _ {\mu} ^ {\ell} (t)\right) \cdot \hat {\boldsymbol {\xi}} _ {\nu} ^ {\ell} (s) \tag {20}
$$
To enforce the definitions of the new order parameters  $\{\Phi, G, A\}$  we again introduce Dirac-delta functions
$$
\begin{array}{l} \delta \left(N \Phi_ {\mu \nu} ^ {\ell} (t, s) - \phi \left(\boldsymbol {h} _ {\mu} ^ {\ell} (t)\right) \cdot \phi \left(\boldsymbol {h} _ {\nu} ^ {\ell} (s)\right)\right) \\ = \int \frac {d \hat {\Phi} _ {\mu \nu} ^ {\ell} (t , s)}{2 \pi i} \exp \left(\hat {\Phi} _ {\mu \nu} ^ {\ell} (t, s) \left[ N \Phi_ {\mu \nu} ^ {\ell} (t, s) - \phi \left(\boldsymbol {h} _ {\mu} ^ {\ell} (t)\right) \cdot \phi \left(\boldsymbol {h} _ {\nu} ^ {\ell} (s)\right) \right]\right) \tag {21} \\ \end{array}
$$
Analogous constraints for  $G$  and  $A$  are enforced with conjugate variables  $\hat{G}, B$ . After introducing these variables, we find that the moment generating functional has the form
$$
\begin{array}{l} Z = \int \prod_ {\ell \mu \nu t s} \frac {d \hat {\Phi} _ {\mu \nu} ^ {\ell} (t , s) d \Phi_ {\mu \nu} ^ {\ell} (t , s)}{2 \pi i} \frac {d \hat {G} _ {\mu \nu} ^ {\ell} (t , s) d G _ {\mu \nu} ^ {\ell} (t , s)}{2 \pi i} \frac {d \hat {G} _ {\mu \nu} ^ {\ell} (t , s) d G _ {\mu \nu} ^ {\ell} (t , s)}{2 \pi i} \frac {d A _ {\mu \nu} ^ {\ell} (t , s) d B _ {\mu \nu} ^ {\ell} (t , s)}{2 \pi i} \\ \exp \left(N S [ \{\Phi^ {\ell}, \hat {\Phi} ^ {\ell}, G ^ {\ell}, \hat {G} ^ {\ell}, A ^ {\ell}, B ^ {\ell} \} ]\right) \tag {22} \\ \end{array}
$$
where  $S$  is the  $\mathcal{O}(1)$  DMFT action which defines the statistical distribution over the dynamics. The action takes the form
$$
\begin{array}{l} S = \sum_ {\ell \mu \nu} \int d t d s \left[ \hat {\Phi} _ {\mu \nu} ^ {\ell} (t, s) \Phi_ {\mu \nu} ^ {\ell} (t, s) + \hat {G} _ {\mu \nu} ^ {\ell} (t, s) - A _ {\mu \nu} ^ {\ell} (t, s) B _ {\nu \mu} ^ {\ell} (s, t) \right] \\ + \frac {1}{N} \sum_ {\ell = 1} ^ {L} \sum_ {i = 1} ^ {N} \ln \mathcal {Z} _ {\ell} \left[ \left\{j _ {i} ^ {\ell}, v _ {i} ^ {\ell} \right\} \right] \tag {23} \\ \end{array}
$$
where  $\mathcal{Z}_{\ell}$  is the single site stochastic process for layer  $\ell$  which defines the marginal distribution of  $\chi, \xi$ , with the following form
$$
\begin{array}{l} \mathcal {Z} _ {\ell} [ \{j ^ {\ell} (t), v ^ {\ell} (t) \} ] = \int \prod_ {\mu t} \frac {d \hat {\chi} _ {\mu} ^ {\ell} (t) d \chi_ {\mu} ^ {\ell} (t)}{2 \pi} \frac {d \hat {\xi} _ {\mu} ^ {\ell} (t) d \xi_ {\mu} ^ {\ell} (t)}{2 \pi} \exp \left(\int d t \sum_ {\mu} [ j _ {\mu} ^ {\ell} (t) \chi_ {\mu} ^ {\ell} (t) + v _ {\mu} ^ {\ell} (t) \xi_ {\mu} ^ {\ell} (t) ]\right) \\ \exp \left(- \frac {1}{2} \sum_ {\mu \nu} \int d t d s \left[ \hat {\Phi} _ {\mu \nu} ^ {\ell} (t, s) \hat {\chi} _ {\mu} ^ {\ell} (t) \hat {\chi} _ {\nu} ^ {\ell} (s) + \hat {G} _ {\mu \nu} ^ {\ell} (t, s) \hat {\xi} _ {\mu} ^ {\ell} (t) \hat {\xi} _ {\nu} ^ {\ell} (s) \right]\right) \\ \exp \left(- i \sum_ {\mu \nu} \int d t d s \left[ B _ {\mu \nu} ^ {\ell} (t, s) \hat {\xi} _ {\mu} ^ {\ell} (t) \phi \left(h _ {\nu} ^ {\ell} (s)\right) + A _ {\mu \nu} ^ {\ell - 1} (t, s) \hat {\chi} _ {\mu} ^ {\ell} (t) g _ {\nu} ^ {\ell} (s) \right]\right) \\ \exp \left(i \sum_ {\mu} \int d t \left[ \hat {\chi} _ {\mu} ^ {\ell} (t) \chi_ {\mu} ^ {\ell} (t) + \hat {\xi} _ {\mu} ^ {\ell} (t) \xi_ {\mu} ^ {\ell} (t) \right]\right) \tag {24} \\ \end{array}
$$
where in the above, the  $\{h,g\}$  fields should be regarded as functionals of  $\{\chi ,\xi \}$ . At zero source  $\pmb{j}^{\ell},\pmb{v}^{\ell}\rightarrow 0$  this function  $S$  can be regarded as the log density for the complete collection of order parameters  $\pmb {q} = \{\hat{\Phi},\Phi ,\hat{G},G,A,B\}$  which collectively control the dynamics. Concretely, we have that  $p(\pmb {q})\propto \exp (NS(\pmb {q}))$ . In the next section we explore an approximation scheme for averages over this distribution at large  $N$ .
D Cumulant Expansion of Observables
We are interested in a principled power series expansion (in  $1 / N$ ) of any observable average  $\langle O(\pmb{q})\rangle$  that depends on DMFT order parameters  $\pmb{q}$ . At any width  $N$  the observable average takes the form
$$
\langle O (\boldsymbol {q}) \rangle_ {N} = \frac {\int d \boldsymbol {q} \exp (N S (\boldsymbol {q})) O (\boldsymbol {q})}{\int d \boldsymbol {q} \exp (N S (\boldsymbol {q}))} \tag {25}
$$