SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As discussed in the main text, the  $N\to \infty$  limit gives  $\langle O(\pmb {q})\rangle_N\sim O(\pmb{q}_{\infty})$  where  $\frac{\partial S}{\partial q} |_{q_{\infty}} = 0$  by a steepest descent argument [55]. We assume that  $S$  's Hessian is negative semidefinite so that  $\Sigma \equiv -\left[\nabla^{2}S(\pmb {q})|_{\pmb{q}_{\infty}}\right]^{-1}\succeq 0$  and Taylor expand  $S(\pmb {q})$  around the saddle point  $\pmb{q}_{\infty}$  giving  $S(\pmb {q}) = S(\pmb{q}_{\infty}) + \frac{1}{2} (\pmb {q} - \pmb{q}_{\infty})^{\top}\nabla^{2}S(\pmb {q})(\pmb {q} - \pmb{q}_{\infty}) + V(\pmb {q} - \pmb{q}_{\infty})$  . We note that the remainder function  $V$  contains only cubic and higher powers of  $\pmb {q} - \pmb{q}_{\infty}\equiv \delta /\sqrt{N}$  . The variable  $\delta$  will be order  $\mathcal{O}(1)$  This will allow us to verify that additional terms are suppressed in powers of  $1 / N$  . Expanding both the numerator and denominator's integrands in powers of  $V$  , we find
$$
\begin{array}{l} \langle O (\pmb {q}) \rangle_ {N} = \frac {\int d \pmb {q} \exp \left(- \frac {N}{2} (\pmb {q} - \pmb {q} _ {\infty}) ^ {\top} \pmb {\Sigma} ^ {- 1} (\pmb {q} - \pmb {q} _ {\infty}) + N V (\pmb {q} - \pmb {q} _ {\infty})\right) O (\pmb {q})}{\int d \pmb {q} \exp \left(- \frac {N}{2} (\pmb {q} - \pmb {q} _ {\infty}) ^ {\top} \pmb {\Sigma} ^ {- 1} (\pmb {q} - \pmb {q} _ {\infty}) + N V (\pmb {q} - \pmb {q} _ {\infty})\right)} \\ = \frac {\int d \boldsymbol {\delta} \exp \left(- \frac {1}{2} \boldsymbol {\delta} ^ {\top} \boldsymbol {\Sigma} ^ {- 1} \boldsymbol {\delta}\right) \left(1 + N V + \frac {N ^ {2}}{2} V ^ {2} + \dots\right) O (\boldsymbol {q} _ {\infty} + N ^ {- 1 / 2} \boldsymbol {\delta})}{\int d \boldsymbol {\delta} \exp \left(- \frac {1}{2} \boldsymbol {\delta} ^ {\top} \boldsymbol {\Sigma} ^ {- 1} \boldsymbol {\delta}\right) \left(1 + N V + \frac {N ^ {2}}{2} V ^ {2} + \dots\right)} \\ = \frac {\left\langle O \right\rangle_ {\infty} + N \left\langle V O \right\rangle_ {\infty} + \frac {N ^ {2}}{2 !} \left\langle V ^ {2} O \right\rangle_ {\infty} + \frac {N ^ {3}}{3 !} \left\langle V ^ {3} O \right\rangle_ {\infty} + \ldots}{1 + N \left\langle V \right\rangle_ {\infty} + \frac {N ^ {2}}{2 !} \left\langle V ^ {2} \right\rangle_ {\infty} + \frac {N ^ {3}}{3 !} \left\langle V ^ {3} \right\rangle_ {\infty} + \ldots} \\ = \left\langle O \right\rangle_ {\infty} \frac {1 + N \left\langle V O \right\rangle_ {\infty} / \left\langle O \right\rangle_ {\infty} + \frac {N ^ {2}}{2 !} \left\langle V ^ {2} O \right\rangle_ {\infty} / \left\langle O \right\rangle_ {\infty} + \frac {N ^ {3}}{3 !} \left\langle V ^ {3} O \right\rangle_ {\infty} / \left\langle O \right\rangle_ {\infty} + \dots}{1 + N \left\langle V \right\rangle_ {\infty} + \frac {N ^ {2}}{2 !} \left\langle V ^ {2} \right\rangle_ {\infty} + \frac {N ^ {3}}{3 !} \left\langle V ^ {3} \right\rangle_ {\infty} + \dots} \tag {26} \\ \end{array}
$$
where  $\langle \rangle_{\infty}$  represents an average over the Gaussian fluctuation  $\mathcal{N}\left(\pmb{q}_{\infty}, - \frac{1}{N}\left[\nabla_{\pmb{q}}^{2}S(\pmb{q}_{\infty})\right]^{-1}\right)$ . We see that the series in the denominator contains terms of the form  $\frac{N^k}{k!}\left\langle V^k\right\rangle_{\infty}$  while the numerator depends on terms of the form  $\frac{N^k}{k!}\left\langle V^k O\right\rangle_{\infty} / \langle O\rangle_{\infty}$ . In either of these power series, the  $k$ -th term can contribute at most
$$
\frac {N ^ {k} \left\langle V ^ {k} O \right\rangle_ {\infty}}{\langle O \rangle_ {\infty}}, N ^ {k} \left\langle V ^ {k} \right\rangle_ {\infty} \sim \left\{ \begin{array}{l l} \mathcal {O} \left(N ^ {- (k + 1) / 2}\right) & k \text {o d d} \\ \mathcal {O} \left(N ^ {- k / 2}\right) & k \text {e v e n} \end{array} \right. \tag {27}
$$
since  $V$  contributes only cubic and higher terms. Thus each term in the numerator and denominator's series contains increasing powers of  $1 / N$ . Concretely, each of the two series have terms of order  $\{N^0, N^{-1}, N^{-1}, N^{-2}, N^{-2}, \ldots\}$ . Thus any quantity of the form  $\frac{\langle O\rangle}{\langle O\rangle_{\infty}}$  admits a ratio of power series in powers of  $1 / N$ . One could truncate each of the series in the numerator and denominator to a desired order in  $N$ . Alternatively, the denominator could be expanded giving a single series (the cumulant expansion [56]). The first few terms in the cumulant expansion have the form
$$
\begin{array}{l} \langle O \rangle_ {N} = \langle O \rangle_ {\infty} + N [ \langle O V \rangle_ {\infty} - \langle O \rangle_ {\infty} \langle V \rangle_ {\infty} ] \\ + \frac {N ^ {2}}{2} \left[ \left\langle V ^ {2} O \right\rangle_ {\infty} - 2 \left\langle V O \right\rangle_ {\infty} \left\langle V \right\rangle_ {\infty} + 2 \left\langle V \right\rangle_ {\infty} ^ {2} \left\langle O \right\rangle_ {\infty} - \left\langle V ^ {2} \right\rangle_ {\infty} \left\langle O \right\rangle_ {\infty} \right] + \dots \tag {28} \\ \end{array}
$$
In this work, we mainly are interested in the leading order correction to  $\langle O\rangle$  which can always be obtained with the truncation after the terms linear in  $V$  for any observable  $O$ .
D.1 Square Deviation from DMFT
We will now analyze the fluctuation statistics of our order parameters around the saddle point  $\left\langle (\pmb{q} - \pmb{q}_{\infty})(\pmb{q} - \pmb{q}_{\infty})^{\top}\right\rangle_{N}$  which has the form
$$
\begin{array}{l} \left\langle (\boldsymbol {q} - \boldsymbol {q} _ {\infty}) (\boldsymbol {q} - \boldsymbol {q} _ {\infty}) ^ {\top} \right\rangle_ {N} = \frac {\left\langle (\boldsymbol {q} - \boldsymbol {q} _ {\infty}) (\boldsymbol {q} - \boldsymbol {q} _ {\infty}) ^ {\top} \right\rangle_ {\infty} + N \left\langle V (\boldsymbol {q} - \boldsymbol {q} _ {\infty}) (\boldsymbol {q} - \boldsymbol {q} _ {\infty}) ^ {\top} \right\rangle_ {\infty} + \dots}{1 + N \left\langle V \right\rangle_ {\infty} + \dots} \\ = \left[ \frac {\frac {1}{N} \boldsymbol {\Sigma} + \mathcal {O} (N ^ {- 2})}{1 + \mathcal {O} (N ^ {- 1})} \right] \sim \frac {1}{N} \boldsymbol {\Sigma} + \mathcal {O} (N ^ {- 2}), \tag {29} \\ \end{array}
$$
as stated in the main text and verified empirically in Figure 3 (a). The reason that the terms in the numerator involving  $V$  can be no larger than  $\mathcal{O}(N^{-2})$  comes from vanishing of odd moments for  $\pmb{q} - \pmb{q}_{\infty}$  in the unperturbed distribution. Thus the leading expression for  $\langle (\pmb {q} - \pmb {q}_\infty)(\pmb {q} - \pmb {q}_\infty)^\top \rangle$  only depends on  $\boldsymbol{\Sigma}$  and not on  $V$
D.2 Mean Deviation from DMFT
Although the square displacement from DMFT only depended on  $\Sigma$  and not on  $V$ , we note that the average order parameter displacement  $\langle \pmb{q} - \pmb{q}_{\infty}\rangle$  does receive a  $\mathcal{O}(1 / N)$  correction that depends on the perturbed potential  $V$
$$
\begin{array}{l} \langle \pmb {q} - \pmb {q} _ {\infty} \rangle_ {N} = \frac {\langle \pmb {q} - \pmb {q} _ {\infty} \rangle_ {\infty} + N \langle (\pmb {q} - \pmb {q} _ {\infty}) V \rangle_ {\infty} + \frac {N ^ {2}}{2} \left\langle (\pmb {q} - \pmb {q} _ {\infty}) V ^ {2} \right\rangle_ {\infty} + \ldots}{1 + N \langle V \rangle_ {\infty} + \frac {N ^ {2}}{2} \langle V ^ {2} \rangle_ {\infty} + \ldots} \\ \sim \frac {\boldsymbol {\Sigma} \left\langle \frac {\partial V}{\partial \boldsymbol {q}} \right\rangle_ {\infty} + \mathcal {O} (N ^ {- 2})}{1 + \mathcal {O} (N ^ {- 1})} \sim \boldsymbol {\Sigma} \left\langle \frac {\partial V}{\partial \boldsymbol {q}} \right\rangle_ {\infty} + \mathcal {O} (N ^ {- 2}). \tag {30} \\ \end{array}
$$
where in the last line we used Stein's lemma (Gaussian integration by parts) for the Gaussian distribution over  $\mathbf{q}$ . Note that  $\left\langle \frac{\partial V}{\partial \mathbf{q}} \right\rangle_{\infty} \sim \mathcal{O}\left(\frac{1}{N}\right)$  since the derivative of the cubic term in  $V$  gives a quadratic function of  $\mathbf{q} - \mathbf{q}_{\infty}$ , whose average must be  $\mathcal{O}(N^{-1})$ . In this work, we focus primarily on the structure of the propagator, but outline a general recipe for getting the leading mean correction in Appendix G and H.2.
D.3 Covariance of Order Parameters
Lastly, we combine the previous two observations to reason about the scaling of the order parameter covariance over initializations. We note that the leading covariance of the order parameters over random initializations is also given by the propagator:  $\operatorname{Cov}(\pmb{q}) \sim \frac{1}{N}\pmb{\Sigma} + \mathcal{O}(N^{-2})$ , since
$$
\begin{array}{l} \operatorname {C o v} (\boldsymbol {q}) = \left\langle \left(\boldsymbol {q} - \left\langle \boldsymbol {q} \right\rangle_ {N}\right) \left(\boldsymbol {q} - \left\langle \boldsymbol {q} \right\rangle_ {N}\right) ^ {\top} \right\rangle_ {N} \\ = \left\langle \left(\boldsymbol {q} - \boldsymbol {q} _ {\infty}\right) \left(\boldsymbol {q} - \boldsymbol {q} _ {\infty}\right) ^ {\top} \right\rangle_ {N} - \left\langle \left(\boldsymbol {q} _ {\infty} - \langle \boldsymbol {q} \rangle_ {N}\right) \left(\boldsymbol {q} _ {\infty} - \langle \boldsymbol {q} \rangle_ {N}\right) ^ {\top} \right\rangle_ {N} \\ \sim \frac {1}{N} \boldsymbol {\Sigma} + \mathcal {O} \left(N ^ {- 2}\right) \tag {31} \\ \end{array}
$$
due to the arguments above which showed that  $\left\langle (\pmb{q} - \pmb{q}_{\infty})(\pmb{q} - \pmb{q}_{\infty})^{\top}\right\rangle \sim \frac{1}{N}\pmb{\Sigma} + \mathcal{O}(N^{-2})$  and that  $\pmb{q}_{\infty} - \langle \pmb{q}\rangle_{N}\sim \mathcal{O}(N^{-1})$ . Therefore, in the leading order picture, it is safe to associate  $\pmb{\Sigma}$  with the covariance of order parameters over random initializations of the network weights.
E Propagator Structure for the full DMFT Action
In this section, we examine the propagator structure for the full DMFT action. This action is modified from other prior works [9, 46] to include the evolution of the network prediction errors  $\Delta(t)$ . Those prior works noted that  $\Delta$  and the NTK  $K$  are deterministic functions of deterministic order parameters  $\{\Phi^{\ell}, G^{\ell}\}$  in the  $N \to \infty$  limit so those authors did not explicitly include  $\Delta$  or  $K$  in the action. At finite width  $N$ , including  $\Delta$ ,  $K$  in the action is crucial as the fluctuation in prediction errors  $\Delta$  has significant consequences for dynamical fluctuations of kernels through the preactivation and
pre-gradient fields. In this section, we will mainly focus on gradient flow, but we describe large step size in Appendix M.
$$
\begin{array}{l} S = \sum_ {\ell \mu \nu} \int d t d s \left[ \hat {\Phi} _ {\mu \nu} ^ {\ell} (t, s) \Phi_ {\mu \nu} ^ {\ell} (t, s) + \hat {G} _ {\mu \nu} ^ {\ell} (t, s) G _ {\mu \nu} ^ {\ell} (t, s) - \gamma^ {2} A _ {\nu \mu} ^ {\ell} (s, t) B _ {\mu \nu} ^ {\ell} (t, s) \right] \\ + \sum_ {\mu} \int d t \hat {\Delta} _ {\mu} (t) \left[ \Delta_ {\mu} (t) - y _ {\mu} + \sum_ {\nu} \int d s \Theta (t - s) K _ {\mu \nu} (s) \Delta_ {\nu} (s) \right] \\ + \sum_ {\mu \nu} \int d t \hat {K} _ {\mu \nu} (t) \left[ K _ {\mu \nu} (t) - \sum_ {\ell} G _ {\mu \nu} ^ {\ell + 1} (t) \Phi_ {\mu \nu} ^ {\ell} (t) \right] \\ + \sum_ {\ell} \ln \mathcal {Z} _ {\ell} \left[ \boldsymbol {\Delta}, \hat {\boldsymbol {\Phi}} ^ {\ell}, \hat {\boldsymbol {G}} ^ {\ell}, \boldsymbol {\Phi} ^ {\ell - 1}, \boldsymbol {G} ^ {\ell + 1}, \boldsymbol {A} ^ {\ell - 1}, \boldsymbol {B} ^ {\ell} \right] \tag {32} \\ \end{array}
$$
where the single site moment generating functionals  $\mathcal{Z}_{\ell}$  have the form
$$
\begin{array}{l} \mathcal {Z} _ {\ell} = \mathbb {E} _ {\left\{h _ {\mu} ^ {\ell} (t), z _ {\mu} ^ {\ell} (t) \right\}} \exp \left(- \sum_ {\mu \nu} \int d t d s \left[ \phi \left(h _ {\mu} ^ {\ell} (t)\right) \phi \left(h _ {\nu} ^ {\ell} (s)\right) \hat {\Phi} _ {\mu \nu} ^ {\ell} (t, s) + g _ {\mu} ^ {\ell} (t) g _ {\nu} ^ {\ell} (s) \hat {G} _ {\mu \nu} ^ {\ell} (t, s) \right]\right) \\ h _ {\mu} ^ {\ell} (t) = u _ {\mu} ^ {\ell} (t) + \gamma \int_ {0} ^ {t} d s \sum_ {\nu} \left[ \Phi_ {\mu \nu} ^ {\ell - 1} (t, s) \Delta_ {\nu} (s) + A _ {\mu \nu} ^ {\ell - 1} (t, s) \right] g _ {\nu} ^ {\ell} (s), \{u _ {\mu} ^ {\ell} (t) \} \sim \mathcal {G P} (0, \Phi^ {\ell - 1}) \\ z _ {\mu} ^ {\ell} (t) = r _ {\mu} ^ {\ell} (t) + \gamma \int_ {0} ^ {t} d s \sum_ {\nu} \left[ G _ {\mu \nu} ^ {\ell + 1} (t, s) \Delta_ {\nu} (s) + B _ {\mu \nu} ^ {\ell} (t, s) \right] \phi \left(h _ {\nu} ^ {\ell} (s)\right), \left\{r _ {\mu} ^ {\ell} (t) \right\} \sim \mathcal {G P} \left(0, \boldsymbol {G} ^ {\ell + 1}\right) \tag {33} \\ \end{array}
$$
with  $g_{\mu}^{\ell}(t) = \dot{\phi}(h_{\mu}^{\ell}(t))z_{\mu}^{\ell}(t)$ . The saddle point equations give the infinite width evolution of our order parameters.
$$
\begin{array}{l} \frac {\partial S}{\partial \hat {\Phi} _ {\mu \nu} ^ {\ell} (t , s)} = \Phi_ {\mu \nu} ^ {\ell} (t, s) - \left\langle \phi (h _ {\mu} ^ {\ell} (t)) \phi (h _ {\nu} ^ {\ell} (s)) \right\rangle = 0 \\ \frac {\partial S}{\partial \hat {G} _ {\mu \nu} ^ {\ell} (t , s)} = G _ {\mu \nu} ^ {\ell} (t, s) - \left<   g _ {\mu} ^ {\ell} (t) g _ {\nu} ^ {\ell} (s) \right> = 0 \\ \frac {\partial S}{\partial A _ {\nu \mu} ^ {\ell} (s , t)} = - \gamma^ {2} B _ {\mu \nu} ^ {\ell} (t, s) + \gamma \left\langle \frac {\partial \phi \left(h _ {\mu} ^ {\ell} (t)\right)}{\partial r _ {\nu} ^ {\ell} (s)} \right\rangle = 0 \\ \frac {\partial S}{\partial B _ {\nu \mu} ^ {\ell} (s , t)} = - \gamma^ {2} A _ {\mu \nu} ^ {\ell} (t, s) + \gamma \left\langle \frac {\partial g _ {\mu} ^ {\ell} (t)}{\partial u _ {\nu} ^ {\ell} (s)} \right\rangle = 0 \\ \frac {\partial S}{\partial \hat {K} _ {\mu \nu} (t)} = K _ {\mu \nu} (t) - \sum_ {\ell} G _ {\mu \nu} ^ {\ell + 1} (t, t) \Phi_ {\mu \nu} ^ {\ell} (t, t) = 0 \\ \frac {\partial S}{\partial \hat {\Delta} _ {\mu} (t)} = \Delta_ {\mu} (t) - y _ {\mu} + \int_ {0} ^ {t} d s \sum_ {\nu} K _ {\mu \nu} (s) \Delta_ {\nu} (s) = 0 \tag {34} \\ \end{array}
$$
These equations exactly recover the mean field description obtained [9]. Note that  $\langle \rangle$  for field averages is an average defined by  $\mathcal{Z}_{\ell}$  and is distinct from the types averages  $\langle \rangle, \langle \rangle_{\infty}$  we have been considering over the order parameters  $\pmb{q}$ . The complementary set of equations for the primal variables, such as  $\frac{\partial S}{\partial \Phi_{\mu\nu}^{\ell}(t,s)} = 0$ , give that  $\hat{K} = \hat{\Delta} = \hat{\Phi} = \hat{G} = 0$  at the saddle point. We now set out to compute the Hessian  $\nabla_{\dot{q}}^{2}S$ . To simplify the set of expressions, we will only explicitly write out the nonvanishing
blocks. We will start with second derivatives involving only pairs of dual variables  $\{\hat{\Phi},\hat{G},A,B\}$
$$
\begin{array}{l} \frac {\partial^ {2} S}{\partial \hat {\Phi} _ {\mu \nu} ^ {\ell} (t , s) \partial \hat {\Phi} _ {\alpha \beta} ^ {\ell} (t ^ {\prime} , s ^ {\prime})} = \left\langle \phi (h _ {\mu} ^ {\ell} (t)) \phi (h _ {\nu} ^ {\ell} (s)) \phi (h _ {\alpha} ^ {\ell} (t ^ {\prime})) \phi (h _ {\beta} ^ {\ell} (s ^ {\prime})) \right\rangle - \Phi_ {\mu \nu} ^ {\ell} (t, s) \Phi_ {\alpha \beta} ^ {\ell} (t ^ {\prime}, s ^ {\prime}) \\ \equiv \kappa_ {\mu \nu \alpha \beta} ^ {\Phi^ {\ell}} (t, s, t ^ {\prime}, s ^ {\prime}) \\ \end{array}
$$
$$
\begin{array}{l} \frac {\partial^ {2} S}{\partial \hat {G} _ {\mu \nu} ^ {\ell} (t , s) \partial \hat {G} _ {\alpha \beta} ^ {\ell} (t ^ {\prime} , s ^ {\prime})} = \left<   g _ {\mu} ^ {\ell} (t) g _ {\nu} ^ {\ell} (s) g _ {\alpha} ^ {\ell} (t ^ {\prime}) g _ {\beta} ^ {\ell} (s ^ {\prime}) \right> - G _ {\mu \nu} ^ {\ell} (t, s) G _ {\alpha \beta} ^ {\ell} (t ^ {\prime}, s ^ {\prime}) \\ \equiv \kappa_ {\mu \nu \alpha \beta} ^ {G ^ {\ell}} (t, s, t ^ {\prime}, s ^ {\prime}) \\ \end{array}
$$
$$
\begin{array}{l} \frac {\partial^ {2} S}{\partial \hat {\Phi} _ {\mu \nu} ^ {\ell} (t , s) \partial \hat {G} _ {\alpha \beta} ^ {\ell} (t ^ {\prime} , s ^ {\prime})} = \left\langle \phi (h _ {\mu} ^ {\ell} (t)) \phi (h _ {\nu} ^ {\ell} (s)) g _ {\alpha} ^ {\ell} (t ^ {\prime}) g _ {\beta} ^ {\ell} (s ^ {\prime}) \right\rangle - \Phi_ {\mu \nu} ^ {\ell} (t, s) G _ {\alpha \beta} ^ {\ell} (t ^ {\prime}, s ^ {\prime}) \\ \equiv \kappa_ {\mu \nu \alpha \beta} ^ {\Phi^ {\ell} G ^ {\ell}} (t, s, t ^ {\prime}, s ^ {\prime}) \\ \end{array}
$$
$$
\begin{array}{l} \frac {\partial^ {2} S}{\partial \hat {\Phi} _ {\mu \nu} ^ {\ell} (t , s) \partial A _ {\beta \alpha} ^ {\ell - 1} (s ^ {\prime} , t ^ {\prime})} = - \gamma \left\langle \frac {\partial \phi (h _ {\mu} ^ {\ell} (t))}{\partial u _ {\beta} ^ {\ell} (s ^ {\prime})} \phi (h _ {\nu} ^ {\ell} (s)) g _ {\alpha} ^ {\ell} (t ^ {\prime}) \right\rangle \\ - \gamma \left\langle \phi \left(h _ {\mu} ^ {\ell} (t)\right) \frac {\partial \phi \left(h _ {\nu} ^ {\ell} (t)\right)}{\partial u _ {\beta} ^ {\ell} \left(s ^ {\prime}\right)} g _ {\alpha} ^ {\ell} \left(t ^ {\prime}\right) \right\rangle \\ - \gamma \left\langle \phi \left(h _ {\mu} ^ {\ell} (t)\right) \phi \left(h _ {\nu} ^ {\ell} (s)\right) \frac {\partial g _ {\alpha} ^ {\ell} \left(t ^ {\prime}\right)}{\partial u _ {\beta} ^ {\ell} \left(s ^ {\prime}\right)} \right\rangle - \gamma^ {2} \Phi_ {\mu \nu} ^ {\ell} (t, s) B _ {\alpha \beta} ^ {\ell - 1} \left(t ^ {\prime}, s ^ {\prime}\right) \\ \equiv - \gamma \kappa_ {\mu \nu \alpha \beta} ^ {\Phi^ {\ell} B ^ {\ell - 1}} (t, s) \\ \end{array}
$$
$$
\begin{array}{l} \frac {\partial^ {2} S}{\partial \hat {\Phi} _ {\mu \nu} ^ {\ell} (t , s) \partial B _ {\beta \alpha} ^ {\ell} \left(s ^ {\prime} , t ^ {\prime}\right)} = - \gamma \left\langle \frac {\partial \phi \left(h _ {\mu} ^ {\ell} (t)\right)}{\partial r _ {\beta} ^ {\ell} \left(s ^ {\prime}\right)} \phi \left(h _ {\nu} ^ {\ell} (s)\right) \phi \left(h _ {\alpha} ^ {\ell} \left(t ^ {\prime}\right)\right) \right\rangle \\ - \gamma \left\langle \phi \left(h _ {\mu} ^ {\ell} (t)\right) \frac {\partial \phi \left(h _ {\nu} ^ {\ell} (t)\right)}{\partial r _ {\beta} ^ {\ell} \left(s ^ {\prime}\right)} \phi \left(h _ {\alpha} ^ {\ell} \left(t ^ {\prime}\right)\right) \right\rangle \\ - \gamma \left\langle \phi \left(h _ {\mu} ^ {\ell} (t)\right) \phi \left(h _ {\nu} ^ {\ell} (s)\right) \frac {\partial \phi \left(h _ {\alpha} ^ {\ell} \left(t ^ {\prime}\right)\right)}{\partial r _ {\beta} ^ {\ell} \left(s ^ {\prime}\right)} \right\rangle - \gamma^ {2} \Phi_ {\mu \nu} ^ {\ell} (t, s) A _ {\alpha \beta} ^ {\ell} \left(t ^ {\prime}, s ^ {\prime}\right) \\ \equiv - \gamma \kappa_ {\mu \nu \alpha \beta} ^ {\Phi^ {\ell} A ^ {\ell}} (t, s) \\ \end{array}
$$
$$
\begin{array}{l} \frac {\partial^ {2} S}{\partial \hat {G} _ {\mu \nu} ^ {\ell} (t , s) \partial A _ {\beta \alpha} ^ {\ell - 1} (s ^ {\prime} , t ^ {\prime})} = - \gamma \left\langle \frac {\partial g _ {\mu} ^ {\ell} (t)}{\partial u _ {\beta} ^ {\ell} (s ^ {\prime})} g _ {\nu} ^ {\ell} (s) g _ {\alpha} ^ {\ell} (t ^ {\prime}) \right\rangle - \gamma \left\langle g _ {\mu} ^ {\ell} (t) \frac {\partial g _ {\nu} ^ {\ell} (t)}{\partial u _ {\beta} ^ {\ell} (s ^ {\prime})} g _ {\alpha} ^ {\ell} (t ^ {\prime}) \right\rangle \\ - \gamma \left\langle g _ {\mu} ^ {\ell} (t) g _ {\nu} ^ {\ell} (s) \frac {\partial g _ {\alpha} ^ {\ell} \left(t ^ {\prime}\right)}{\partial u _ {\beta} ^ {\ell} \left(s ^ {\prime}\right)} \right\rangle - \gamma^ {2} G _ {\mu \nu} ^ {\ell} (t, s) B _ {\alpha \beta} ^ {\ell - 1} \left(t ^ {\prime}, s ^ {\prime}\right) \\ \equiv - \gamma \kappa_ {\mu \nu \alpha \beta} ^ {G ^ {\ell} B ^ {\ell - 1}} (t, s) \\ \end{array}
$$
$$
\begin{array}{l} \frac {\partial^ {2} S}{\partial \hat {G} _ {\mu \nu} ^ {\ell} (t , s) \partial B _ {\beta \alpha} ^ {\ell} \left(s ^ {\prime} , t ^ {\prime}\right)} = - \gamma \left\langle \frac {\partial g _ {\mu} ^ {\ell} (t)}{\partial r _ {\beta} ^ {\ell} \left(s ^ {\prime}\right)} g _ {\nu} ^ {\ell} (s) \phi \left(h _ {\alpha} ^ {\ell} \left(t ^ {\prime}\right)\right) \right\rangle - \gamma \left\langle g _ {\mu} ^ {\ell} (t) \frac {\partial g _ {\nu} ^ {\ell} (t)}{\partial r _ {\beta} ^ {\ell} \left(s ^ {\prime}\right)} \phi \left(h _ {\alpha} ^ {\ell} \left(t ^ {\prime}\right)\right) \right\rangle \\ = - \gamma \left\langle g _ {\mu} ^ {\ell} (t) g _ {\nu} ^ {\ell} (s) \frac {\partial \phi \left(h _ {\alpha} ^ {\ell} \left(t ^ {\prime}\right)\right)}{\partial r _ {\beta} ^ {\ell} \left(s ^ {\prime}\right)} \right\rangle - \gamma^ {2} G _ {\mu \nu} ^ {\ell} (t, s) A _ {\alpha \beta} ^ {\ell} \left(t ^ {\prime}, s ^ {\prime}\right) \\ \equiv - \gamma \kappa_ {\mu \nu \alpha \beta} ^ {G ^ {\ell} A ^ {\ell}} (t, s) \\ \end{array}
$$
$$
\frac {\partial^ {2} S}{\partial A _ {\mu \nu} ^ {\ell} (t , s) \partial B _ {\beta \alpha} ^ {\ell} (s ^ {\prime} , t ^ {\prime})} = - \gamma^ {2} \delta_ {\mu \alpha} \delta_ {\nu \beta} \delta (t - t ^ {\prime}) \delta (s - s ^ {\prime})
$$
$$
\begin{array}{l} \frac {\partial^ {2} S}{\partial A _ {\nu \mu} ^ {\ell - 1} (s , t) \partial B _ {\beta \alpha} ^ {\ell} (s ^ {\prime} , t ^ {\prime})} = \gamma^ {2} \left\langle \frac {\partial^ {2}}{\partial u _ {\nu} ^ {\ell} (s) \partial r _ {\beta} ^ {\ell} (s ^ {\prime})} \left[ g _ {\mu} ^ {\ell} (t) \phi \left(h _ {\alpha} ^ {\ell} (t ^ {\prime})\right) \right] \right\rangle - \gamma^ {4} B _ {\mu \nu} ^ {\ell - 1} (t, s) A _ {\alpha \beta} ^ {\ell} (t ^ {\prime}, s ^ {\prime}) \\ \equiv \kappa_ {\mu \nu \alpha \beta} ^ {B ^ {\ell - 1} A ^ {\ell}} \left(t, s, t ^ {\prime}, s ^ {\prime}\right) \tag {35} \\ \end{array}
$$
Next, we consider the second derivatives involving only primal variables  $\{\Phi^{\ell}, G^{\ell}, K, \Delta\}$  which all vanish
$$
\frac {\partial^ {2} S}{\partial \Phi_ {\mu \nu} ^ {\ell} (t , s) \partial \Phi_ {\alpha \beta} ^ {\ell^ {\prime}} (t ^ {\prime} , s ^ {\prime})} = 0
$$
$$
\frac {\partial^ {2} S}{\partial G _ {\mu \nu} ^ {\ell} (t , s) \partial G _ {\alpha \beta} ^ {\ell^ {\prime}} (t ^ {\prime} , s ^ {\prime})} = 0
$$
$$
\frac {\partial^ {2} S}{\partial \Phi_ {\mu \nu} ^ {\ell} (t , s) \partial G _ {\alpha \beta} ^ {\ell^ {\prime}} (t ^ {\prime} , s ^ {\prime})} = 0
$$
$$
\frac {\partial^ {2} S}{\partial \Phi_ {\mu \nu} ^ {\ell} (t , s) \partial K _ {\alpha \beta} (s ^ {\prime})} = 0
$$
$$
\frac {\partial^ {2} S}{\partial G _ {\mu \nu} ^ {\ell} (t , s) \partial K _ {\alpha \beta} (s ^ {\prime})} = 0
$$
$$
\frac {\partial^ {2} S}{\partial \Phi_ {\mu \nu} ^ {\ell} (t , s) \partial \Delta_ {\alpha} (s ^ {\prime})} = 0
$$
$$
\frac {\partial^ {2} S}{\partial G _ {\mu \nu} ^ {\ell} (t , s) \partial \Delta_ {\alpha} (s ^ {\prime})} = 0
$$
$$
\frac {\partial^ {2} S}{\partial K _ {\mu \nu} (t) \partial K _ {\alpha \beta} (s)} = 0
$$
$$
\frac {\partial^ {2} S}{\partial K _ {\mu \nu} (t) \partial \Delta_ {\alpha} (s)} = 0
$$
$$
\frac {\partial^ {2} S}{\partial \Delta_ {\mu} (t) \partial \Delta_ {\alpha} (s)} = 0 \tag {36}
$$
Now we consider all derivatives which involve one of the dual variables  $\{\hat{\Phi}^{\ell},\hat{G}^{\ell},A^{\ell},B^{\ell}\}$  and the primal variable  $\Delta$
$$
\frac {\partial^ {2} S}{\partial \hat {\Phi} _ {\mu \nu} ^ {\ell} (t , s) \partial \Delta_ {\alpha} (t ^ {\prime})} = - \left\langle \frac {\partial}{\partial \Delta_ {\alpha} (t ^ {\prime})} [ \phi (h _ {\mu} ^ {\ell} (t)) \phi (h _ {\nu} ^ {\ell} (s)) ] \right\rangle \equiv - D _ {\mu \nu \alpha} ^ {\Phi^ {\ell} \Delta} (t, s, t ^ {\prime})
$$
$$
\frac {\partial^ {2} S}{\partial \hat {G} _ {\mu \nu} ^ {\ell} (t , s) \partial \Delta_ {\alpha} (t ^ {\prime})} = - \left\langle \frac {\partial}{\partial \Delta_ {\alpha} (t ^ {\prime})} [ g _ {\mu} ^ {\ell} (t) g _ {\nu} ^ {\ell} (s) ] \right\rangle \equiv - D _ {\mu \nu \alpha} ^ {G ^ {\ell} \Delta} (t, s, t ^ {\prime})
$$
$$
\frac {\partial^ {2} S}{\partial A _ {\nu \mu} ^ {\ell - 1} (s , t) \partial \Delta_ {\alpha} (t ^ {\prime})} = \gamma \left\langle \frac {\partial}{\partial \Delta_ {\alpha} (t ^ {\prime}) \partial u _ {\nu} ^ {\ell} (s)} g _ {\mu} ^ {\ell} (t) \right\rangle \equiv \gamma D _ {\mu \nu \alpha} ^ {B ^ {\ell - 1}, \Delta} (t, s, t ^ {\prime})
$$