SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The computation of  $\Psi_t, \Psi_w, \Psi_y$  is rather standard and follows the main steps presented in Appendix A. One again assumes the replica symmetric ansatz [21, 22]
$$
q _ {a b} = (r - q) \delta_ {a b} + q, \quad m _ {a} = m, \tag {93}
$$
$$
\hat {q} _ {a b} = \left(- \frac {\hat {r}}{2} + - \hat {q}\right) \delta_ {a b} + \hat {q}, \quad \hat {m} _ {a} = \hat {m}. \tag {94}
$$
Introducing as in Appendix A the variances
$$
V \equiv r - q, \quad \hat {V} \equiv \hat {r} + \hat {q}, \tag {95}
$$
$$
\Psi_ {t} = \frac {\hat {V} q - V \hat {q}}{2} - m \hat {m}, \tag {96}
$$
$$
\Psi_ {w} = - \frac {1}{2} \ln \left(1 + \hat {V} \hat {\sigma} ^ {2}\right) + \frac {\hat {\sigma} ^ {2}}{2} \frac {\hat {q} + \hat {m} ^ {2}}{1 + \hat {V} \hat {\sigma} ^ {2}}, \tag {97}
$$
$$
\Psi_ {y} = \mathbb {E} _ {\eta \sim \mathcal {N} (0, 1)} \ln \left[ \int d \lambda \mathcal {N} (\lambda ; \sqrt {q} \eta , V) \cosh \left(\frac {\sigma \lambda + m}{\sigma^ {2}}\right) \right]. \tag {98}
$$
Therefore the free entropy reads
$$
\begin{array}{l} \Phi = \operatorname * {e x t r} _ {q, m, V, \hat {q}, \hat {m}, \hat {V}} \frac {\hat {V} q - V \hat {q}}{2} - m \hat {m} - \frac {1}{2} \ln \left(1 + \hat {V} \hat {\sigma} ^ {2}\right) + \frac {\hat {\sigma} ^ {2}}{2} \frac {\hat {q} + \hat {m} ^ {2}}{1 + \hat {V} \hat {\sigma} ^ {2}} \\ + \alpha \mathbb {E} _ {\eta \sim \mathcal {N} (0, 1)} \ln \left[ \int d \lambda \mathcal {N} (\lambda ; \sqrt {q} \eta , V) \cosh \left(\frac {\sigma \lambda + m}{\sigma^ {2}}\right) \right]. \tag {99} \\ \end{array}
$$
Figure 12: (Binary isotropic mixture,  $\Sigma_{1,2} = 0.09\mathbb{I}_d$ ,  $\rho_{1,2} = 1/2$ , and  $\mu_1 = -\mu_2$ , see also Fig. 1). (right) MSE (minus the rescaling baseline  $\mathrm{mse}_{\circ}$ ) for the DAE ( $p = 1$ ,  $\sigma = \tanh$ ) (2) (blue), the oracle Tweedie denoiser (76) (green), the Bayes denoiser (101) (red), and the plug-in oracle-PCA denoiser (103) (black). Solid lines correspond to the theoretical formulae 3.3, (81) and (101); simulations correspond to a DAE optimized using the Pytorch implementation of Adam (weight decay  $\lambda = 0.1$ , learning rate  $\eta = 0.05$ , full batch, 2000 epochs, sample complexity  $\alpha = 1$ ), averaged over  $N = 10$  instances, with error bars representing one standard deviation. (right) Cosine similarity (6) for the same algorithms, with identical color code.
The solution of this extremization problem is given by the set of self-consistent equations, imposing that gradients with respect to all parameters be zero:
$$
\left\{ \begin{array}{l} V = \frac {\hat {\sigma} ^ {2}}{1 + \hat {V} \hat {\sigma} ^ {2}} \\ q = \frac {\hat {\sigma} ^ {4} (\hat {q} + \hat {m} ^ {2})}{(1 + \hat {V} \hat {\sigma} ^ {2}) ^ {2}} + \frac {\hat {\sigma} ^ {2}}{1 + \hat {V} \hat {\sigma} ^ {2}} \\ m = \frac {\hat {\sigma} ^ {2} \hat {m}}{1 + \hat {V} \hat {\sigma} ^ {2}} \end{array} \right. \quad \left\{ \begin{array}{l} \hat {V} = - \alpha \frac {1}{\sigma \sqrt {q}} \mathbb {E} _ {\eta \sim \mathcal {N} (0, 1)} \left[ \tanh  \left(\frac {\sigma \sqrt {q} \eta + m}{\sigma^ {2}}\right) \eta \right] \\ \hat {q} = \frac {\alpha V}{\sigma^ {2}} \\ \hat {m} = \frac {\alpha}{\sigma^ {2}} \mathbb {E} _ {\eta \sim \mathcal {N} (0, 1)} \left[ \tanh  \left(\frac {\sigma \sqrt {q} \eta + m}{\sigma^ {2}}\right) \right] \end{array} \right. \tag {100}
$$
This is a simple optimization problem over 6 variables, which can be solved numerically. One can finally use the obtained summary statistics to evaluate the Bayes optimal MSE:
$$
\begin{array}{l} \mathrm {m s e} _ {\mathfrak {b}} = \mathrm {m s e} _ {\circ} - 2 \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} \frac {\Delta}{\sigma^ {2} (1 - \Delta) + \Delta} \left((1 - b \sqrt {1 - \Delta}) m\right) \tanh \left(\frac {\sqrt {1 - \Delta} m + \sqrt {(1 - \Delta) \sigma^ {2} + \Delta} \sqrt {q + V z}}{\sigma^ {2} (1 - \Delta) + \Delta}\right) \\ + \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {u, v \sim \mathcal {N} (0, \Omega)} \left(\frac {\Delta}{\sigma^ {2} (1 - \Delta) + \Delta}\right) ^ {2} q \tanh  \left(\frac {\sqrt {1 - \Delta} m + \sqrt {(1 - \Delta) \sigma^ {2} + \Delta} u}{\sigma^ {2} (1 - \Delta) + \Delta}\right) \tanh  \left(\frac {\sqrt {1 - \Delta} m + \sqrt {(1 - \Delta) \sigma^ {2} + \Delta} v}{\sigma^ {2} (1 - \Delta) + \Delta}\right) \tag {101} \\ \end{array}
$$
$$
\Omega = \left[ \begin{array}{c c} q + V & q \\ q & q + V \end{array} \right] \tag {102}
$$
This completes the derivation of the MSE of the Bayes estimator agnostic of the cluster means.
C.4 A simple plug-in denoiser
Another simple denoiser assuming only perfect knowledge of the variance  $\sigma$ , but imperfect knowledge of  $\mu$ , is simply to plug the PCA estimate  $\hat{\mu}_{\mathrm{PCA}}$  of  $\mu$  into the Tweedie oracle (76), i.e.
$$
t \left(\hat {\boldsymbol {\mu}} _ {\mathrm {P C A}}, \tilde {\boldsymbol {x}}\right) = \frac {\sqrt {1 - \Delta} \sigma^ {2}}{\sigma^ {2} (1 - \Delta) + \Delta} \tilde {\boldsymbol {x}} + \frac {\Delta}{\sigma^ {2} (1 - \Delta) + \Delta} \tanh  \left(\frac {\tilde {\boldsymbol {x}} ^ {\top} \hat {\boldsymbol {\mu}} _ {\mathrm {P C A}} \sqrt {1 - \Delta}}{\sigma^ {2} (1 - \Delta) + \Delta}\right) \times \hat {\boldsymbol {\mu}} _ {\mathrm {P C A}}. \tag {103}
$$
The performance of this plug-in denoiser is plotted in Fig. 12, and contrasted to the one of the Bayes denoiser  $\mathfrak{b}(\cdot)$  (101), the oracle  $t(\cdot)$  (81) and the DAE 3.3. Strikingly, the performance of the PCA plug-in denoiser is sizeably identical to the one of the Bayes-optimal denoiser, both in terms of MSE and cosine similarity. It is important to remind at this point that both the plug-in (103) and the Bayes denoiser (101) still rely on perfect knowledge of the cluster variance  $\sigma$  and the noise level  $\Delta$ , while the DAE (2) is agnostic to these parameters. Yet, these two denoisers make for a fairer comparison than the oracle (81), as they importantly take into account the finite training set. The DAE is relatively close to the Bayes baseline for small noise levels  $\Delta$ , while a gap can be seen to open up for larger  $\Delta$ .
Figure 13: Cosine similarity  $\theta$  (6) (green), weight norm  $\| \hat{\boldsymbol{w}}\|^2 / d$  and skip connection strength  $\hat{b}$  for the MNIST dataset (left), of which for simplicity only 1s and 7s were kept, and FashionMNIST (right), of which only boots and shoes were kept. In solid lines, the theoretical predictions resulting from using Conjecture 3.3 in its generic formulation (58) with the empirically estimated covariances and means. Dots represent numerical simulations of a DAE ( $p = 1$ ,  $\sigma = \tanh$ ) trained with  $n = 784$  training points, using the Pytorch implementation of full-batch Adam, with learning rate  $\eta = 0.05$  over 2000 epochs, weight decay  $\lambda = 0.1$ , averaged over  $N = 5$  instances. Error bars represent one standard deviation. See Appendix D for full details on the preprocessing.
D Details on real data simulations
In this Appendix, we provide further details on the real data experiments presented in Fig. 2 and Fig. 4.
Preprocessing The original MNIST [15] and FashionMNIST [16] data-sets were flattened (vectorized), centered, and rescaled by 400 (MNIST) and 600 (FashionMNIST). For simplicity and ease of discussion, for each data set, only two labels were kept, namely 1s and 7s for MNIST and boots and shoes for FashionMNIST. Note that the visual similarity between the two selected classes should make the denoising problem more challenging.
Means and covariance estimation For each data-set, data from the same class were considered to belong to the same Gaussian mixture cluster. For simplicity, we kept the same number of training points in each cluster, so as to obtain balanced clusters  $(\rho_{1} = \rho_{2} = 1 / 2)$  for definiteness. For each cluster, the corresponding mean  $\mu$  and covariance  $\Sigma$  were numerically evaluated from the empirical mean and covariance over the 6000 boots (shoes) in the FashionMNIST training set, and the 6265 1s (7s) in the MNIST training set. The solid lines in Fig. 2 correspond to using those estimates in the asymptotic formulae of Conjecture 3.3, in their generic form (58) (see Appendix A) for sample complexity  $\alpha = 1$  and  $\ell_2$  regularization  $\lambda = 0.1$ .
Pytorch simulations The red points in Fig. 2 were obtained from numerical simulations, by optimizing a  $p = 1$  DAE (2) with  $\sigma = \tanh$  activation using the Pytorch implementation of the full-batch Adam [43] optimizer, with weight decay  $\lambda = 0.1$ , learning rate  $\eta = 0.05$ , over  $T = 2000$  epochs. For each value of  $\Delta$ ,  $n = 784$  images were drawn without replacement from the data-set training set, corrupted by noise, and fed to the DAE along with the clean image for training. The denoising test MSE, cosine similarity and summary statistics of the trained DAE were estimated after optimization using  $n_{\mathrm{test}} = 1000$  images randomly drawn from the test set (from which also only the relevant classes were kept, and equally represented- i.e. balanced). Each value was averaged over  $N = 10$  instances of the train set, noise realization and test set. The obtained values were furthermore found to be robust with respect to the choice of the optimizer, learning rate and number of epochs.
D.1 Complementary simulations
Fig.2 in the main text addresses the comparison of the theoretical prediction of Conjecture 3.3 with simulations on the real data-set, at the level of the denoising test MSE (5). For completeness, we show here the same comparison for the other learning metrics and summary statistics, namely the cosine similarity  $\theta$  (6), weights norm  $\| \hat{\boldsymbol{w}}\|^2 /d$  and trained skip connection strength  $\hat{b}$ . Theoretical asymptotic characterization are again provided by Conjecture 3.3 (10), (11) and (13).
They are plotted as solid lines in Fig. 13, and contrasted to numerical simulations (dots) on the real data-sets, by training a  $p = 1$  DAE (2) using the Pytorch implementation of the Adam optimizer. All experiment details are the same as Fig. 2 of the main text, and can be found in the previous subsection. As can be observed, there is a gap, for large noise levels  $\Delta$ , between the theory and simulations of the weights norm  $\| \hat{w} \|^2 / d$  (red). On the other hand, the matchings for the cosine similarity  $\theta$  (green) and skip connection strength  $\hat{b}$  (blue) are perfect. Overall, Conjecture (3.3) therefore captures well (with the exception of  $\| \hat{w} \|^2 / d$  at large  $\Delta$ ) the main metrics of the denoising problem on these two real data-sets. A more thorough investigation of the possible Gaussian universality of the problem is left to future work.
E Derivation of Corollary 3.4
In this Appendix, we provide the derivation of Corollary 3.4.
Rescaling component  $\hat{h}$  We first provide a succinct derivation of the sharp asymptotic characterizations for the rescaling component  $\hat{h}$  (3). Because this is only a scalar optimization problem over the single scalar parameter  $c$  of  $h_c$ , the derivation is simple and straightforward. The empirical loss reads
$$
\begin{array}{l} \frac {1}{n d} \hat {\mathcal {R}} (c) = \frac {1}{n d} \sum_ {\mu = 1} ^ {n} \left\| \pmb {x} ^ {\mu} - c \left(\sqrt {1 - \Delta} \pmb {x} ^ {\mu} + \sqrt {\Delta} \pmb {\xi} ^ {\mu}\right) \right\| ^ {2} \\ = (1 - \sqrt {1 - \Delta} c) ^ {2} \frac {1}{n} \sum_ {\mu = 1} ^ {n} \frac {\| \pmb {x} ^ {\mu} \| ^ {2}}{d} + c ^ {2} \Delta \frac {1}{n} \sum_ {\mu = 1} ^ {n} \frac {\| \pmb {\xi} ^ {\mu} \| ^ {2}}{d} + \frac {1}{n} \sum_ {\mu = 1} ^ {n} \frac {\| \pmb {x} ^ {\mu} \| ^ {2}}{d} \frac {(\pmb {x} ^ {\mu}) ^ {\top} \pmb {\xi} ^ {\mu}}{d} \\ = (1 - \sqrt {1 - \Delta} c) ^ {2} \frac {1}{n} \sum_ {\mu = 1} ^ {n} \frac {\| \pmb {x} ^ {\mu} \| ^ {2}}{d} + c ^ {2} \Delta + \mathcal {O} (1 / \sqrt {d}) \\ \stackrel {\mathrm {C L T}} {=} (1 - \sqrt {1 - \Delta} c) ^ {2} \mathbb {E} _ {\mathbb {P}} \left[ \frac {\| \pmb {x} ^ {\mu} \| ^ {2}}{d} \right] + c ^ {2} \Delta \\ = (1 - \sqrt {1 - \Delta} c) ^ {2} \frac {1}{d} \sum_ {k = 1} ^ {K} \rho_ {k} \left(\| \boldsymbol {\mu} _ {k} \| ^ {2} + \operatorname {T r} \boldsymbol {\Sigma} _ {k}\right) + c ^ {2} \Delta \tag {104} \\ \end{array}
$$
Extremizing this last expression with respect to  $c$  leads to
$$
\hat {c} = \frac {\frac {1}{d} \left(\sum_ {k = 1} ^ {K} \rho_ {k} \operatorname {T r} \boldsymbol {\Sigma} _ {k}\right) \sqrt {1 - \Delta}}{\frac {1}{d} \left(\sum_ {k = 1} ^ {K} \rho_ {k} \operatorname {T r} \boldsymbol {\Sigma} _ {k}\right) (1 - \Delta) + \Delta} = \frac {\frac {1}{d} \left(\sum_ {k = 1} ^ {K} \rho_ {k} \int d \nu_ {\gamma} (\gamma) \gamma_ {k}\right) \sqrt {1 - \Delta}}{\frac {1}{d} \left(\sum_ {k = 1} ^ {K} \rho_ {k} \int d \nu_ {\gamma} (\gamma) \gamma_ {k}\right) (1 - \Delta) + \Delta}. \tag {105}
$$
which is exactly (10), as stated in Corollary 3.4. The associated MSE can be then readily computed as
$$
\begin{array}{l} \mathrm {m s e} _ {\hat {r}} = \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {\xi}} \left\| \boldsymbol {x} - \hat {c} \left(\sqrt {1 - \Delta} \boldsymbol {x} ^ {\mu} + \sqrt {\Delta} \boldsymbol {\xi}\right) \right\| ^ {2} \\ = (1 - \sqrt {1 - \Delta} \hat {c}) ^ {2} \frac {1}{d} \sum_ {k = 1} ^ {K} \rho_ {k} \left(\| \mu_ {k} \| ^ {2} + \operatorname {T r} \boldsymbol {\Sigma} _ {k}\right) + \hat {c} ^ {2} \Delta \\ = (1 - \sqrt {1 - \Delta} \hat {c}) ^ {2} \frac {1}{d} \sum_ {k = 1} ^ {K} \rho_ {k} \left(\int d \nu_ {\tau} (\tau) \tau_ {k} ^ {2} + d \int d \mu_ {\gamma} (\gamma) \gamma_ {k}\right) + \hat {c} ^ {2} \Delta \\ = \operatorname {m s e} _ {\circ}. \tag {106} \\ \end{array}
$$
This completes the proof of the asymptotic characterization of the rescaling component  $\hat{r}$ .
Bottleneck network component  $\hat{u}$  For  $\hat{u}$ , one needs to go through the derivation presented in Appendix A by setting  $b = 0$  from the beginning. It is straightforward to realize that the derivation goes through sensibly unaltered, and the only differing step is that the quadratic potential  $\Psi_{\mathrm{quad}}$  is now a constant. One therefore only needs to set  $\hat{b}$  in all the formulae of Conjecture 3.3 to access sharp asymptotics for  $\hat{u}$ . This concludes the derivation of Corollary 3.4
F Derivation of Corollary 3.5
We now turn to the derivation of Corollary 3.5. This corresponds to the limit  $\Delta = 0$ , where the input of the auto-encoder is also the clean data point. In terms of equations, essentially, this means that all the integrals over  $\xi, \lambda_{\xi}$  in the derivation of Conjecture 3.3 as presented in Appendix A should be removed. For the sake of clarity, and because this is an important case of particular interest, we provide here a complete, succinct but self-contained, derivation in the case of RAEs. For technical reasons, we limit ourselves to  $\ell_2$  regularization. Furthermore, note that the model of an RAE with a skip connection is trivial, since such an architecture can achieve perfect reconstruction accuracy simply by setting the skip connection strength to 1 and the weights to 0. Therefore, we consider an RAE without skip connection
$$
f _ {w} (\tilde {x}) = \frac {\boldsymbol {w} ^ {\top}}{\sqrt {d}} \sigma \left(\frac {\boldsymbol {w} \tilde {\boldsymbol {x}}}{\sqrt {d}}\right). \tag {107}
$$
F.1 Derivation
The derivation of asymptotic characterizations for the metrics mse (5),  $\theta$  (6) and summary statistics for the RAE (107) follow the same lines as for the full DAE (2), as presented in Appendix A. As in Appendix A, the first step is to evaluate the replicated partition function
$$
\mathbb {E} _ {\mathcal {D}} Z ^ {s} = \int \prod_ {a = 1} ^ {s} d \boldsymbol {w} _ {a} e ^ {- \beta \sum_ {a = 1} ^ {s} g \left(\boldsymbol {w} ^ {a}\right)} \prod_ {\mu = 1} ^ {n} \sum_ {k = 1} ^ {K} \rho_ {k} \mathbb {E} _ {\boldsymbol {\eta}} \underbrace {e ^ {- \frac {\beta}{2} \sum_ {a = 1} ^ {s} \left\| \boldsymbol {\mu} _ {k} + \boldsymbol {\eta} - \left[ \frac {\boldsymbol {w} _ {a} ^ {\top}}{\sqrt {d}} \sigma \left(\frac {\boldsymbol {w} _ {a} \left(\boldsymbol {\mu} _ {k} + \boldsymbol {\eta}\right)}{\sqrt {d}}\right) \right] \right\| ^ {2}}} _ {(\star)}. \tag {108}
$$
The exponent  $(\star)$  can be expanded as
$$
e ^ {- \frac {\beta}{2} \sum_ {a = 1} ^ {s} \left[ \operatorname {T r} \left[ \frac {\boldsymbol {w} _ {a} \boldsymbol {w} _ {a} ^ {\top}}{d} \sigma \left(\frac {\boldsymbol {w} _ {a} (\boldsymbol {\mu} _ {k} + \eta)}{\sqrt {d}}\right) ^ {\otimes 2} \right] - 2 \sigma \left(\frac {\boldsymbol {w} _ {a} (\boldsymbol {\mu} _ {k} + \eta)}{\sqrt {d}}\right) ^ {\top} \frac {\boldsymbol {w} _ {a} ((\boldsymbol {\mu} _ {k} + \eta))}{\sqrt {d}} + (\| \boldsymbol {\mu} _ {k} \| ^ {2} + \| \eta \| ^ {2} + 2 \boldsymbol {\mu} _ {k} ^ {\top} \eta) \right].} \tag {109}
$$
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {\eta}} (\star) = \sum_ {k = 1} ^ {K} \rho_ {k} e ^ {- \frac {\beta s}{2} \| \boldsymbol {\mu} _ {k} \| ^ {2}} \int \frac {d \boldsymbol {\eta}}{(2 \pi) ^ {\frac {d}{2}} \sqrt {\det  \boldsymbol {\Sigma} _ {k}}} \underbrace {e ^ {- \frac {1}{2} \boldsymbol {\eta} ^ {\top} \left(\boldsymbol {\Sigma} _ {k} ^ {- 1} + \beta s \mathbb {I} _ {d}\right) \boldsymbol {\eta} - \beta s \boldsymbol {\mu} _ {k} ^ {\top} \boldsymbol {\eta}}} _ {P _ {\text {e f f .}} (\boldsymbol {\eta})} \\ \times e ^ {- \frac {\beta}{2} \sum_ {a = 1} ^ {s} \left[ \operatorname {T r} \left[ \frac {\boldsymbol {w} _ {a} \boldsymbol {w} _ {a} ^ {\top}}{d} \sigma \left(\frac {\boldsymbol {w} _ {a} (\boldsymbol {\mu} _ {k} + \boldsymbol {\eta})}{\sqrt {d}}\right) ^ {\otimes 2} \right] - 2 \sigma \left(\frac {\boldsymbol {w} _ {a} (\boldsymbol {\mu} _ {k} + \boldsymbol {\eta}))}{\sqrt {d}}\right) ^ {\top} \frac {\boldsymbol {w} _ {a} (\boldsymbol {\mu} _ {k} + \boldsymbol {\eta})}{\sqrt {d}} \right]}. \tag {110} \\ \end{array}
$$
The effective prior over the  $\eta$  is therefore Gaussian with dressed mean and covariance
$$
P _ {\text {e f f .}} (\boldsymbol {\eta}) = \mathcal {N} (\boldsymbol {\eta}; \boldsymbol {\mu}, \boldsymbol {C}), \tag {111}
$$
$$
\boldsymbol {C} ^ {- 1} = \boldsymbol {\Sigma} _ {k} ^ {- 1} + \beta s \mathbb {I} _ {d}, \quad \boldsymbol {\mu} = - \beta s \boldsymbol {C} \boldsymbol {\mu} _ {k}. \tag {112}
$$
Observe that this mean is  $\mathcal{O}(s)$  and can be safely neglected. Then
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {\eta}} (\star) = \sum_ {k = 1} ^ {K} \rho_ {k} \frac {e ^ {- \frac {\beta s}{2} \| \boldsymbol {\mu} _ {k} \| ^ {2} + \frac {1}{2} \boldsymbol {\mu} ^ {\top} \boldsymbol {C} ^ {- 1} \boldsymbol {\mu}}}{\sqrt {\det  \boldsymbol {\Sigma} _ {k}} \det  (\boldsymbol {C}) ^ {- \frac {1}{2}}} \\ \underbrace {\left\langle \right. e ^ {- \frac {\beta}{2} \sum_ {a = 1} ^ {s} \left[ \right. \operatorname {T r} \left[ \frac {\boldsymbol {w} _ {a} \boldsymbol {w} _ {a} ^ {\top}}{d} \sigma \left(\frac {\boldsymbol {w} _ {a} (\boldsymbol {\mu} _ {k} + \boldsymbol {\eta})}{\sqrt {d}}\right) ^ {\otimes 2} \right] - 2 \sigma \left(\frac {\boldsymbol {w} _ {a} (\boldsymbol {\mu} _ {k} + \boldsymbol {\eta})}{\sqrt {d}}\right) ^ {\top} \frac {\boldsymbol {w} _ {a} (\boldsymbol {\mu} _ {k} + \boldsymbol {\eta})}{\sqrt {d}}} \left. \right\rangle} _ {(a)} \tag {113} \\ \end{array}
$$
Again, like Appendix A, we introduce the local fields:
$$
\lambda_ {a} ^ {\eta} = \frac {\boldsymbol {w} _ {a} (\boldsymbol {\eta} - \boldsymbol {\mu})}{\sqrt {d}}, \quad h _ {a} ^ {k} = \frac {\boldsymbol {w} _ {a} \boldsymbol {\mu} _ {k}}{\sqrt {d}}, \tag {114}
$$
which have correlation
$$
\left\langle \lambda_ {a} ^ {\eta} \lambda_ {b} ^ {\eta} \right\rangle \approx \frac {\boldsymbol {w} _ {a} \boldsymbol {\Sigma} _ {k} \boldsymbol {w} _ {b} ^ {\top}}{d}. \tag {115}
$$
We used the leading order of the covariance  $C$ . One therefore has to introduce the order parameters:
$$
Q _ {a b} = \frac {\boldsymbol {w} _ {a} \boldsymbol {w} _ {b} ^ {\top}}{d} \in \mathbb {R} ^ {p \times p}, \quad S _ {a b} ^ {k} = \frac {\boldsymbol {w} _ {a} \boldsymbol {\Sigma} _ {k} \boldsymbol {w} _ {b} ^ {\top}}{d} \in \mathbb {R} ^ {p \times p}, \quad m _ {a} ^ {k} = \frac {\boldsymbol {w} _ {a} \boldsymbol {\mu} _ {k}}{\sqrt {d}} \in \mathbb {R} ^ {p}. \tag {116}
$$
The distribution of the local fields  $\lambda_{a}^{\eta}$  is then simply:
$$
\left(\lambda_ {a} ^ {\eta}\right) _ {a = 1} ^ {s} \sim \mathcal {N} \left(0, S ^ {k}\right). \tag {117}
$$
$$
\left\langle e ^ {- \frac {\beta}{2} \sum_ {a = 1} ^ {s} \operatorname {T r} \left[ Q _ {a a} \sigma \left(m _ {a} ^ {k} + \lambda_ {a} ^ {\eta}\right) ^ {\otimes 2} \right] - \frac {\beta}{2} \sum_ {a = 1} ^ {s} \left[ - 2 \sigma \left(m _ {a} ^ {k} + \lambda_ {a} ^ {\eta}\right) ^ {\top} \left(m _ {a} ^ {k} \lambda_ {a} ^ {\eta}\right) \right]} \right\rangle_ {\left\{\lambda_ {a} ^ {\eta} \right\} _ {a = 1} ^ {s}}. \tag {118}
$$
Introducing Dirac functions enforcing the definitions of  $Q_{ab}$ ,  $m_a$  brings the replicated function in the following form:
$$
\begin{array}{l} \mathbb {E} Z ^ {s} = \int \prod_ {a = 1} ^ {s} d b _ {a} \prod_ {a, b} d Q _ {a b} d \hat {Q} _ {a b} \prod_ {k = 1} ^ {K} \prod_ {a = 1} ^ {s} d m _ {a} d \hat {m} _ {a} \prod_ {k = 1} ^ {K} \prod_ {a, b} d S _ {a b} ^ {k} d \hat {S} ^ {k} a b e ^ {s} \\ \underbrace {e ^ {- d \sum_ {a \leq b} Q _ {a b} \hat {Q} _ {a b} - d \sum_ {k = 1} ^ {K} \sum_ {a \leq b} S _ {a b} ^ {k} \hat {S} _ {a b} ^ {k} - \sum_ {k = 1} ^ {K} \sum_ {a} d m _ {a} ^ {k} \hat {m} _ {a} ^ {k}}} _ {e ^ {\beta s d \Psi_ {t}}} \\ \underbrace {\int \prod_ {a = 1} ^ {s} d \boldsymbol {w} _ {a} e ^ {- \beta \sum_ {a} g (\boldsymbol {w} ^ {a}) + \sum_ {a \leq b} \hat {Q} _ {a b} \boldsymbol {w} _ {a} \boldsymbol {w} _ {b} ^ {\top} + \sum_ {k = 1} ^ {K} \sum_ {a \leq b} \hat {S} _ {a b} ^ {k} \boldsymbol {w} _ {a} \boldsymbol {\Sigma} _ {k} \boldsymbol {w} _ {b} ^ {\top} + \sum_ {k = 1} ^ {K} \sum_ {a} \hat {m} _ {a} ^ {k} \sqrt {d} \boldsymbol {w} _ {a} \boldsymbol {\mu} _ {k}} _ {e ^ {\beta s d \Psi_ {w}}} \\ e ^ {- \frac {\beta s}{2} \| \boldsymbol {\mu} \| ^ {2}} \underbrace {\left[ \sum_ {k = 1} ^ {K} \rho_ {k} \frac {1}{\sqrt {\det \boldsymbol {\Sigma} _ {k}} \det \left(\boldsymbol {C} ^ {- 1}\right) ^ {\frac {1}{2}}} (a) \right] ^ {\alpha d}} _ {e ^ {\alpha s d ^ {2} \Psi_ {\mathrm {q u a d .}} + \beta s d \Psi_ {y}}}. \tag {119} \\ \end{array}
$$
As in Appendix A, we have introduced the trace, entropic and energetic potentials  $\Psi_t, \Psi_w, \Psi_y$ . Since all the integrands scale exponentially (or faster) with  $d$ , this integral can be computed using a saddle-point method. To proceed further, note that the energetic term encompasses two types of terms, scaling like  $sd^2$  and  $sd$ . More precisely,
$$
\begin{array}{l} \left[ \sum_ {k = 1} ^ {K} \rho_ {k} \frac {1}{\sqrt {\det \boldsymbol {\Sigma} _ {k}} \det \big (\boldsymbol {C} ^ {- 1} \big) ^ {\frac {1}{2}}} (a) \right] ^ {\alpha d} = \left[ 1 - s d \sum_ {k = 1} ^ {K} \rho_ {k} \frac {1}{s d} \ln \left(\sqrt {\det \boldsymbol {\Sigma} _ {k}} \det \big (\boldsymbol {C} ^ {- 1} \big) ^ {\frac {1}{2}}\right) + s \sum_ {k = 1} ^ {K} \rho_ {k} \frac {1}{s} \ln (a) \right] ^ {\alpha d} \\ = e ^ {- s \alpha d ^ {2} \sum_ {k = 1} ^ {K} \rho_ {k} \frac {1}{s d} \ln \left(\sqrt {\det \boldsymbol {\Sigma} _ {k}} \det \left(\boldsymbol {C} ^ {- 1}\right) ^ {\frac {1}{2}}\right) + s \alpha d \sum_ {k = 1} ^ {K} \rho_ {k} \frac {1}{s} \ln (a)} \tag {120} \\ \end{array}
$$
Remark that in contrast to Appendix A, the quadratic term is constant with respect to the network parameters, and therefore one no longer needs to discuss its extremization. Following Appendix A, we assume the RS ansatz
$$
\forall a, b, \qquad S _ {a b} ^ {k} = (r _ {k} - q _ {k}) \delta_ {a b} + q _ {k}, \qquad \hat {S} _ {a b} ^ {k} = - \left(\frac {\hat {r} _ {k}}{2} + \hat {q} _ {k}\right) \delta_ {a b} + \hat {q} _ {k},
$$
$$
\forall a, b, \quad Q _ {a b} = (r - q) \delta_ {a b} + q, \quad \hat {Q} _ {a b} = - \left(\frac {\hat {r}}{2} + \hat {q}\right) \delta_ {a b} + \hat {q}. \tag {121}
$$
In the following, we sequentially simplify the potentials  $\Psi_w, \Psi_y, \Psi_t$  under this ansatz.