SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \Psi_ {w} \\ = \frac {1}{2 d} \operatorname {T r} \left[ \left(\hat {V} \otimes \mathbb {I} _ {d} + \sum_ {k = 1} ^ {K} \hat {V} _ {k} \otimes \boldsymbol {\Sigma} _ {k}\right) ^ {- 1} \odot \left(\hat {q} \otimes \mathbb {I} _ {d} + \sum_ {k = 1} ^ {K} \hat {q} _ {k} \otimes \boldsymbol {\Sigma} _ {k} + d \left(\sum_ {k = 1} ^ {K} \hat {m} _ {k} \boldsymbol {\mu} _ {k} ^ {\top}\right) ^ {\otimes 2}\right) \right] - \frac {1}{d} \mathbb {E} _ {\Xi} \mathcal {M} _ {r} (\Xi), \tag {50} \\ \end{array}
$$
where we introduced the Moreau enveloppe
$$
\begin{array}{l} \mathcal {M} _ {r} (\boldsymbol {\Xi}) \\ = \inf  _ {\boldsymbol {w}} \left\{\frac {1}{2} \left\| \left(\hat {V} \otimes \mathbb {I} _ {d} + \sum_ {k = 1} ^ {K} \hat {V} _ {k} \otimes \boldsymbol {\Sigma} _ {k}\right) \odot \boldsymbol {w} - \left(\hat {V} \otimes \mathbb {I} _ {d} + \sum_ {k = 1} ^ {K} \hat {V} _ {k} \otimes \boldsymbol {\Sigma} _ {k}\right) ^ {- \frac {1}{2}} \odot \left(\left(\hat {q} \otimes \mathbb {I} _ {d}\right) ^ {\frac {1}{2}} \odot \boldsymbol {\Xi} _ {0} + \sum_ {k = 1} ^ {K} \left(\hat {q} _ {k} \otimes \boldsymbol {\Sigma} _ {k}\right) ^ {\frac {1}{2}} \odot \boldsymbol {\Xi} _ {k} + \sqrt {d} \sum_ {k = 1} ^ {K} \hat {m} _ {k} \boldsymbol {\mu} _ {k} ^ {\top}\right) \right\| ^ {2} + g (\boldsymbol {w}) \right\}. \tag {51} \\ \end{array}
$$
Note that while this optimization problem is still cast in a space of dimension  $p \times d$ , for some regularizers  $g(\cdot)$  including the usual  $\ell_{1,2}$  penalties, the Moreau envelope admits a compact analytical expression.
The energetic potential  $\Psi_y$  also simplifies in this limit to
$$
\Psi_ {y} = - \sum_ {k = 1} ^ {K} \rho_ {k} \mathbb {E} _ {\eta , \xi} \mathcal {M} _ {k} (\eta , \xi), \tag {52}
$$
$$
\begin{array}{l} \mathcal {M} _ {k} (\xi , \eta) = \frac {1}{2} \inf _ {x, y} \left\{\mathrm {T r} \left[ V _ {k} ^ {- 1} \left(y - q _ {k} ^ {\frac {1}{2}} \eta - m _ {k}\right) ^ {\otimes 2} \right] + \frac {1}{\Delta} \mathrm {T r} \left[ V ^ {- 1} \left(x - \sqrt {\Delta} q ^ {\frac {1}{2}} \xi\right) ^ {\otimes 2} \right] \right\} \\ \left. \operatorname {T r} \left[ q \sigma \left(\sqrt {1 - \Delta} y + x\right) ^ {\otimes 2} \right] - 2 \sigma \left(\sqrt {1 - \Delta} y + x\right) ^ {\top} \left((1 - \sqrt {1 - \Delta} b) y - b x\right) \right\}. \tag {53} \\ \end{array}
$$
It is immediate to see that the trace potential  $\Psi_t$  can be expressed as
$$
\Psi_ {t} = \frac {\operatorname {T r} \left[ \hat {V} q \right] - \operatorname {T r} [ \hat {q} V ]}{2} + \frac {1}{2} \sum_ {k = 1} ^ {K} \left(\operatorname {T r} \left[ \hat {V} _ {k} q _ {k} \right] - \operatorname {T r} [ \hat {q} _ {k} V _ {k} ]\right) - \sum_ {k = 1} ^ {K} m _ {k} \hat {m} _ {k}. \tag {54}
$$
Putting these results together, the total free entropy reads
$$
\begin{array}{l} \Phi = \operatorname *{extr}_{q,m,V,\hat{q},\hat{m},\hat{V},\{q_{k},m_{k},V_{k},\hat{q}_{k},\hat{m}_{k},\hat{V}_{k}\}_{k = 1}^{K}}\frac{\operatorname{Tr}\left[\hat{V}q\right] - \operatorname{Tr}[\hat{q} V]}{2} +\frac{1}{2}\sum_{k = 1}^{K}(\operatorname {Tr}\left[\hat{V}_{k}q_{k}\right] - \operatorname {Tr}[\hat{q}_{k}V_{k}]) - \sum_{k = 1}^{K}m_{k}\hat{m}_{k} \\ - \alpha \sum_ {k = 1} ^ {K} \rho_ {k} \mathbb {E} _ {\xi , \eta} \mathcal {M} _ {k} (\xi , \eta) - \frac {1}{d} \mathbb {E} _ {\Xi} \mathcal {M} _ {r} (\Xi) \\ + \frac {1}{2 d} \operatorname {T r} \left[ \left(\hat {V} \otimes \mathbb {I} _ {d} + \sum_ {k = 1} ^ {K} \hat {V} _ {k} \otimes \boldsymbol {\Sigma} _ {k}\right) ^ {- 1} \odot \left(\hat {q} \otimes \mathbb {I} _ {d} + \sum_ {k = 1} ^ {K} \hat {q} _ {k} \otimes \boldsymbol {\Sigma} _ {k} + d \left(\sum_ {k = 1} ^ {K} \hat {m} _ {k} \boldsymbol {\mu} _ {k} ^ {\top}\right) ^ {\otimes 2}\right) \right] \tag {55} \\ \end{array}
$$
where we remind that
$$
\begin{array}{l} \mathcal {M} _ {k} (\xi , \eta) = \frac {1}{2} \inf  _ {x, y} \left\{\operatorname {T r} \left[ V _ {k} ^ {- 1} \left(y - q _ {k} ^ {\frac {1}{2}} \eta - m _ {k}\right) ^ {\otimes 2} \right] + \operatorname {T r} \left[ V ^ {- 1} \left(x - q ^ {\frac {1}{2}} \xi\right) ^ {\otimes 2} \right] \right. \\ \left. + \operatorname {T r} \left[ q \sigma \left(\sqrt {1 - \Delta} y + x\right) ^ {\otimes 2} \right] - 2 \sigma \left(\sqrt {1 - \Delta} y + x\right) ^ {\top} \left((1 - \sqrt {1 - \Delta} b) y - b x\right) \right\}. \tag {56} \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {M} _ {r} (\Xi) \\ = \inf  _ {\boldsymbol {w}} \left\{\frac {1}{2} \left\| \left(\hat {V} \otimes \mathbb {I} _ {d} + \sum_ {k = 1} ^ {K} \hat {V} _ {k} \otimes \boldsymbol {\Sigma} _ {k}\right) ^ {\frac {1}{2}} \boldsymbol {w} - \left(\hat {V} \otimes \mathbb {I} _ {d} + \sum_ {k = 1} ^ {K} \hat {V} _ {k} \otimes \boldsymbol {\Sigma} _ {k}\right) ^ {- \frac {1}{2}} \left(\left(\hat {q} \otimes \mathbb {I} _ {d}\right) ^ {\frac {1}{2}} \odot \boldsymbol {\Xi} _ {0} + \sum_ {k = 1} ^ {K} \left(\hat {q} _ {k} \otimes \boldsymbol {\Sigma} _ {k}\right) ^ {\frac {1}{2}} \odot \boldsymbol {\Xi} _ {k} + \sqrt {d} \sum_ {k = 1} ^ {K} \hat {m} _ {k} \boldsymbol {\mu} _ {k} ^ {\top}\right) \right\| ^ {2} + r (\boldsymbol {w}) \right\} \tag {57} \\ \end{array}
$$
A.5 Self-consistent equations
The extremization problem (58) can be recast as a system of self-consistent equations by requiring the gradient with respect to each summary statistic involved in (58) to be zero. This translates into:
$$
\left\{ \begin{array}{l} \hat {q} _ {k} = \alpha \rho_ {k} \mathbb {E} _ {\xi , \eta} V _ {k} ^ {- 1} \left(\operatorname {p r o x} _ {y} ^ {k} - q _ {k} ^ {\frac {1}{2}} \eta - m _ {k}\right) ^ {\otimes 2} V _ {k} ^ {- 1} \\ \hat {V} _ {k} = - \alpha \rho_ {k} q _ {k} ^ {- \frac {1}{2}} \mathbb {E} _ {\xi , \eta} V _ {k} ^ {- 1} \left(\operatorname {p r o x} _ {y} ^ {k} - q _ {k} ^ {\frac {1}{2}} \eta - m _ {k}\right) \eta^ {\top} \\ \hat {m} _ {k} = \alpha \rho_ {k} \mathbb {E} _ {\xi , \eta} V _ {k} ^ {- 1} \left(\operatorname {p r o x} _ {y} ^ {k} - q _ {k} ^ {\frac {1}{2}} \eta - m _ {k}\right) \\ \hat {q} = \alpha \sum_ {k = 1} ^ {K} \rho_ {k} \mathbb {E} _ {\xi , \eta} V ^ {- 1} \left(\operatorname {p r o x} _ {x} ^ {k} - q ^ {\frac {1}{2}} \xi\right) ^ {\otimes 2} V ^ {- 1} \\ \hat {V} = - \alpha \sum_ {k = 1} ^ {K} \rho_ {k} \mathbb {E} _ {\xi , \eta} \left[ q ^ {- \frac {1}{2}} V ^ {- 1} \left(\operatorname {p r o x} _ {x} ^ {k} - q ^ {\frac {1}{2}} \xi\right) \xi^ {\top} - \sigma (\sqrt {1 - \Delta} \operatorname {p r o x} _ {y} + \operatorname {p r o x} _ {x}) ^ {\otimes 2} \right] \end{array} , \right. \tag {58}
$$
$$
\left\{ \begin{array}{l} V _ {k} = \frac {1}{d} \mathbb {E} _ {\boldsymbol {\Xi}} \left[ \left(\operatorname {p r o x} _ {r} \odot (\hat {q} _ {k} \otimes \boldsymbol {\Sigma} _ {k}) ^ {- \frac {1}{2}} \odot (\mathbb {I} _ {p} \otimes \boldsymbol {\Sigma} _ {k})\right) \boldsymbol {\Xi} _ {k} ^ {\top} \right] \\ q _ {k} = \frac {1}{d} \mathbb {E} _ {\boldsymbol {\Xi}} \left[ \operatorname {p r o x} _ {r} \boldsymbol {\Sigma} _ {k} \operatorname {p r o x} _ {r} ^ {\top} \right] \\ m _ {k} = \frac {1}{\sqrt {d}} \mathbb {E} _ {\xi , \eta} [ \operatorname {p r o x} _ {r} \boldsymbol {\mu} _ {k} ] \\ V = \frac {1}{d} \mathbb {E} _ {\boldsymbol {\Xi}} \left[ \left(\operatorname {p r o x} _ {r} \odot (\hat {q} \otimes \mathbb {I} _ {d}) ^ {- \frac {1}{2}}\right) \boldsymbol {\Xi} _ {0} ^ {\top} \right] \\ q = \frac {1}{d} \mathbb {E} _ {\boldsymbol {\Xi}} \left[ \operatorname {p r o x} _ {r} \operatorname {p r o x} _ {r} ^ {\top} \right] \end{array} . \right. \tag {59}
$$
We remind that the skip connection  $b$  is fixed to
$$
b = \frac {\frac {1}{d} \left(\sum_ {k = 1} ^ {K} \rho_ {k} \operatorname {T r} \boldsymbol {\Sigma} _ {k}\right) \sqrt {1 - \Delta}}{\frac {1}{d} \left(\sum_ {k = 1} ^ {K} \rho_ {k} \operatorname {T r} \boldsymbol {\Sigma} _ {k}\right) (1 - \Delta) + \Delta}. \tag {60}
$$
A.6 Sharp asymptotic formulae for the learning metrics
The previous subsections have allowed to obtain sharp asymptotic characterization for a number of summary statistics of the probability (17). These statistics are in turn sufficient to asymptotically characterize the learning metrics discussed in the main text. We successively address the test MSE (5), cosine similarity (6) and training MSE.
MSE The denoising MSE reads
$$
\begin{array}{l} \operatorname {m s e} = \mathbb {E} _ {k, \boldsymbol {\eta}, \boldsymbol {\xi}} \left\| \boldsymbol {\mu} _ {k} + \boldsymbol {\eta} - \left(\hat {b} \sqrt {1 - \Delta} (\boldsymbol {\mu} _ {k} + \boldsymbol {\eta}) + \hat {b} \sqrt {\Delta} \boldsymbol {\xi} + \frac {\hat {\boldsymbol {w}} ^ {\top}}{\sqrt {d}} \sigma \left(\frac {\hat {\boldsymbol {w}} (\sqrt {1 - \Delta} (\boldsymbol {\mu} _ {k} + \boldsymbol {\eta}) + \sqrt {\Delta} \boldsymbol {\xi})}{\sqrt {d}}\right)\right) \right\| ^ {2} \\ = \mathrm {m s e} _ {\circ} + \sum_ {k = 1} ^ {K} \rho_ {k} \mathbb {E} _ {z} ^ {\mathcal {N} (0, 1)} \left[ \mathrm {T r} \left[ q \sigma \left(\sqrt {1 - \Delta m _ {k}} + \sqrt {\Delta q + (1 - \Delta) q _ {k}} z\right) ^ {\otimes 2} \right] \right] \\ - 2 \sum_ {k = 1} ^ {K} \rho_ {k} \mathbb {E} _ {u, v} ^ {\mathcal {N} (0, 1)} \left[ \sigma \left(\sqrt {1 - \Delta} m _ {k} + \sqrt {q _ {k} (1 - \Delta)} u + \sqrt {q} \sqrt {\Delta} v\right) \right] ^ {\top} \left((1 - b \sqrt {1 - \Delta}) m _ {k} + (1 - b \sqrt {1 - \Delta}) \sqrt {q _ {k}} u - b \sqrt {q} \sqrt {\Delta} v\right) \tag {61} \\ \end{array}
$$
$$
\mathrm {m s e} _ {\circ} = d \Delta b ^ {2} + (1 - \sqrt {1 - \Delta} b) ^ {2} \left(\sum_ {k = 1} ^ {K} \rho_ {k} \left(\left\| \boldsymbol {\mu} _ {k} \right\| ^ {2} + \operatorname {T r} \boldsymbol {\Sigma} _ {k}\right)\right) \tag {62}
$$
Cosine similarity By the very definition of the summary statistics  $m_{k},q$
$$
\theta_ {i k} \equiv \frac {\hat {\boldsymbol {w}} _ {i} \boldsymbol {\mu} _ {k}}{\| \hat {\boldsymbol {w}} \| \times \| \boldsymbol {\mu} _ {k} \|} = \frac {(m _ {k}) _ {i}}{\sqrt {q _ {i i}} \| \boldsymbol {\mu} _ {k} \|} \tag {63}
$$
Figure 5: (left) Training MSE for the full DAE (2) ( $p = 1$ ,  $\sigma = \tanh$ ). Solid lines represent the sharp asymptotic formula (68); dots correspond to simulation, training the DAE with the Pytorch implementation of full-batch Adam, over  $T = 2000$  epochs using learning rate  $\eta = 0.05$  and weight decay  $\lambda = 0.1$ . The data was averaged over  $N = 5$  instances; error bars are smaller than the point size. (right) Generalization gap  $\mathrm{mse}_{\hat{f}} - \epsilon_t$ . Solid lines correspond to the asymptotic prediction of Conjecture 3.3 (for the test MSE) and of (68) (for the train MSE), while dots correspond to simulations. Error bars represent one standard deviation. The Gaussian mixture is the isotropic binary mixture, whose parameters are specified in the caption of Fig. 1 in the main text.
Training error The replica formalism also allows to compute a sharp asymptotic characterization for the training MSE
$$
\epsilon_ {t} \equiv \mathbb {E} _ {\mathcal {D}} \left[ \frac {1}{n d} \sum_ {\mu = 1} ^ {n} \left\| \boldsymbol {x} ^ {\mu} - \hat {f} \left(\tilde {\boldsymbol {x}} ^ {\mu}\right) \right\| ^ {2} \right]. \tag {64}
$$
We have skipped the discussion of this learning metric in the main text for the sake of conciseness, but will now detail the derivation. Note that the training error  $\epsilon_{t}$  can be deduced from the risk (4) and the average regularization as
$$
\frac {1}{2} \epsilon_ {t} = \mathbb {E} _ {\mathcal {D}} \left[ \hat {\mathcal {R}} (\hat {\boldsymbol {w}}, \hat {b}) \right] - \mathbb {E} _ {\mathcal {D}} [ r (\hat {\boldsymbol {w}}) ]. \tag {65}
$$
Note that in turn, from the definition of the free entropy, the average risk (training loss) can be computed as
$$
\mathbb {E} _ {\mathcal {D}} \left[ \hat {\mathcal {R}} (\hat {\boldsymbol {w}}, \hat {b}) \right] = - \lim  _ {\beta \rightarrow \infty} \frac {1}{\beta} \frac {\partial \ln Z}{\partial \beta} = - \lim  _ {\beta \rightarrow \infty} \frac {1}{\beta} \frac {\partial \Phi}{\partial \beta}. \tag {66}
$$
Doing so reveals
$$
\begin{array}{l} \epsilon_ {t} = - 2 \lim  _ {\beta \rightarrow \infty} \frac {\partial \left(\Psi_ {y} - d \Psi_ {\mathrm {q u a d}} - \frac {\beta}{2} m (\sqrt {1 - \Delta} b - 1) ^ {2}\right))}{\partial \beta} \\ = \operatorname {m s e} _ {\circ} + \lim  _ {\beta \rightarrow \infty} \sum_ {k = 1} ^ {K} \rho_ {k} \int_ {\mathbb {R} ^ {p}} D \xi D \eta \frac {\int \mathcal {N} \left(\lambda_ {\xi} ; q ^ {\frac {1}{2}} \xi , V _ {k}\right) \mathcal {N} \left(\lambda_ {\eta} ; q _ {k} ^ {\frac {1}{2}} \xi , V\right) (*) e ^ {- \frac {\beta}{2} (*)}}{\int \mathcal {N} \left(\lambda_ {\xi} ; q ^ {\frac {1}{2}} \xi , V _ {k}\right) \mathcal {N} \left(\lambda_ {\eta} ; q _ {k} ^ {\frac {1}{2}} \xi , V\right) e ^ {- \frac {\beta}{2} (*)}}. \tag {67} \\ \end{array}
$$
Taking the  $\beta \to \infty$  limit finally leads to
$$
\begin{array}{l} \epsilon_ {t} = \mathrm {m s e} _ {\circ} + \sum_ {k = 1} ^ {K} \rho_ {k} \mathbb {E} _ {\xi , \eta} \left[ \operatorname {T r} \left[ q \sigma \left(\sqrt {1 - \Delta} \operatorname {p r o x} _ {y} + \operatorname {p r o x} _ {x}\right) ^ {\otimes 2} \right] \right. \\ \left. - 2 \sigma (\sqrt {1 - \Delta} \operatorname {p r o x} _ {y} + \operatorname {p r o x} _ {x}) ^ {\top} ((1 - \sqrt {1 - \Delta} \hat {b}) \operatorname {p r o x} _ {y} - \hat {b} \operatorname {p r o x} _ {x}) \right] \tag {68} \\ \end{array}
$$
The sharp asymptotic formula (68) agrees well with numerical simulations, Fig. 5. As is intuitive, the generalization gap  $\mathrm{mse}_{\hat{f}} - \epsilon_t$  grows with the noise level  $\Delta$ , as the learning problem grows harder.
A.7 Generic asymptotically exact formulae
This last result ends the derivation of the generic version of Conjecture 3.3, i.e. not assuming Assumptions 3.1 and 3.2.
Importantly, note that the characterization (58) is, like the formulae in e.g. [47, 48], asymptotically exact, but not fully asymptotic, as the equations (58) still involve high-dimensional quantities. In practice however, for standard regularizers  $r(\cdot)$  like  $\ell_1$  or  $\ell_2$ , the proximal  $\mathrm{prox}_r$  admits a simple closed-form expression, and the average over  $\Xi$  can be carried out analytically. The only high-dimensional operation left is then taking the trace of linear combinations of  $\Sigma_k$  matrices (or the inverse of such combinations), which is numerically less demanding, and analytically much simpler, than the high dimensional optimization (4) and averages (5) involved in the original problem. We give in the next paragraph an example of how Assumption 3.1 can be relaxed, for a binary mixture with arbitrary (i.e. not necessarily jointly diagonalizable) covariances  $\Sigma_{1,2}$ .
Example: Anisotropic, heteroscedastic binary mixture As an illustration, we provide the equivalent of (58) for a binary mixture, with generically anisotropic and distinct covariances, for  $p = 1$ , thereby breaking free from Assumption 3.1. We index the clusters by  $+, -$  rather than 1, 2 for notational convenience. We remind that
$$
\mathcal {M} _ {\pm} (\xi , \eta) = \frac {1}{2} \inf  _ {x, y} \left\{\frac {1}{\Delta V} (x - \sqrt {q} \sqrt {\Delta} \xi) ^ {2} + \frac {1}{V _ {\pm}} (y - \sqrt {q \pm} \eta \mp m) ^ {2} + q \sigma (\sqrt {1 - \Delta} y + x) ^ {2} - 2 \sigma (\sqrt {1 - \Delta} y + x) ((1 - \sqrt {1 - \Delta} b) y - b x) \right\}. \tag {69}
$$
Then the self-consistent equations (58) simplify to
These equations are, as previously discussed, asymptotically exact as  $d \to \infty$ . While they still involve traces over high-dimensional matrices  $\pmb{\Sigma}_k$ , this is a very simple operation. We have therefore reduced the original high-dimensional optimization problem (4) to the much simpler one of computing traces like (70). Crucially, while these traces cannot be generally simplified without Assumption 3.1, they provide the benefit of a simple and compact expression which bypasses the need of jointly diagonalizable covariances, and which can thus be readily evaluated for any covariance, like real-data covariances. This versatility is leveraged in Appendix D.
A.8 End of the derivation: Conjecture 3.3
We finally provide the last step in the derivation of the fully asymptotic characterization of Conjecture 3.3. Assuming  $r(\cdot) = \lambda /2\| \cdot \| _F^2$  (Assumption 3.2), the Moreau envelope  $\mathcal{M}_r$  assumes a very simple expression, and no longer needs to
be written as a high-dimensional optimization problem. The resulting free energy can be compactly written as:
$$
\begin{array}{l} \Phi = \operatorname * {e x t r} _ {\substack {q, m, V, \hat {q}, \hat {m}, \hat {V}, \{q _ {k}, m _ {k}, V _ {k}, \hat {q} _ {k}, \hat {m} _ {k}, \hat {V} _ {k} \} _ {k = 1} ^ {K}}} \frac {\operatorname {T r} \left[ \hat {V} q \right] - \operatorname {T r} [ \hat {q} V ]}{2} + \frac {1}{2} \sum_ {k = 1} ^ {K} (\operatorname {T r} \left[ \hat {V} _ {k} q _ {k} \right] - \operatorname {T r} [ \hat {q} _ {k} V _ {k} ]) - \sum_ {k = 1} ^ {K} m _ {k} \hat {m} _ {k} \\ - \alpha \sum_ {k = 1} ^ {K} \rho_ {k} \mathbb {E} _ {\xi , \eta} \mathcal {M} _ {k} (\xi , \eta) \\ + \underbrace {\frac {1}{2 d} \operatorname {T r} \left[ \left(\lambda \mathbb {I} _ {p} \odot \mathbb {I} _ {d} + \hat {V} \otimes \mathbb {I} _ {d} + \sum_ {k = 1} ^ {K} \hat {V} _ {k} \otimes \boldsymbol {\Sigma} _ {k}\right) ^ {- 1} \odot \left(\hat {q} \otimes \mathbb {I} _ {d} + \sum_ {k = 1} ^ {K} \hat {q} _ {k} \otimes \boldsymbol {\Sigma} _ {k} + d \left(\sum_ {k = 1} ^ {K} \hat {m} _ {k} \boldsymbol {\mu} _ {k} ^ {\top}\right) ^ {\otimes 2}\right) \right]} _ {(*)} \tag {71} \\ \end{array}
$$
$(*)$  is the only segment which still involves high-dimensional matrices, and is therefore not yet full asymptotic. Assuming Assumption 3.1,  $(*)$  can be massaged into
$$
\begin{array}{l} (*) = \frac {1}{2 d} \sum_ {i = 1} ^ {d} \operatorname {T r} \left[ \left(\lambda + \hat {V} + \lambda_ {i} ^ {k} \hat {V} _ {k}\right) ^ {- 1} \left(\hat {q} + \sum_ {k = 1} ^ {K} \lambda_ {i} ^ {k} \hat {q} _ {k} + \sum_ {1 \leq k, j \leq K} e _ {i} ^ {\top} \mu_ {j} \hat {m} _ {j} \hat {m} _ {k} ^ {\top} \boldsymbol {\mu} _ {k} ^ {\top} e _ {i}\right) \right] \\ \stackrel {d \rightarrow \infty} {=} \frac {1}{2} \int d \nu (\gamma , \tau) \operatorname {T r} \left[\left(\lambda + \hat {V} + \gamma_ {k} \hat {V} _ {k}\right) ^ {- 1} \left(\hat {q} + \sum_ {k = 1} ^ {K} \gamma_ {k} \hat {q} _ {k} + \sum_ {1 \leq k, j \leq K} \tau_ {j} \tau_ {k} \hat {m} _ {j} \hat {m} _ {k} ^ {\top}\right)\right] \tag {72} \\ \end{array}
$$
The fully asymptotic free energy thus becomes
$$
\begin{array}{l} \Phi = \operatorname * {e x t r} _ {q, m, V, \hat {q}, \hat {m}, \hat {V}, \{q _ {k}, m _ {k}, V _ {k}, \hat {q} _ {k}, \hat {m} _ {k}, \hat {V} _ {k} \} _ {k = 1} ^ {K}} \frac {\operatorname {T r} [ \hat {V} q ] - \operatorname {T r} [ \hat {q} V ]}{2} + \frac {1}{2} \sum_ {k = 1} ^ {K} (\operatorname {T r} [ \hat {V} _ {k} q _ {k} ] - \operatorname {T r} [ \hat {q} _ {k} V _ {k} ]) - \sum_ {k = 1} ^ {K} m _ {k} \hat {m} _ {k} \\ - \alpha \sum_ {k = 1} ^ {K} \rho_ {k} \mathbb {E} _ {\xi , \eta} \mathcal {M} _ {k} (\xi , \eta) \\ + \frac {1}{2} \int d \nu (\gamma , \tau) \operatorname {T r} \left[ \left(\lambda + \hat {V} + \gamma_ {k} \hat {V} _ {k}\right) ^ {- 1} \left(\hat {q} + \sum_ {k = 1} ^ {K} \gamma_ {k} \hat {q} _ {k} + \sum_ {1 \leq k, j \leq K} \tau_ {j} \tau_ {k} \hat {m} _ {j} \hat {m} _ {k} ^ {\top}\right) \right] \tag {73} \\ \end{array}
$$
Requiring the free energy to be extremized implies
$$
\left\{ \begin{array}{l} \hat {q} _ {k} = \alpha \rho_ {k} \mathbb {E} _ {\xi , \eta} V _ {k} ^ {- 1} \left(\operatorname {p r o x} _ {y} ^ {k} - q _ {k} ^ {\frac {1}{2}} \eta - m _ {k}\right) ^ {\otimes 2} V _ {k} ^ {- 1} \\ \hat {V} _ {k} = - \alpha \rho_ {k} q _ {k} ^ {- \frac {1}{2}} \mathbb {E} _ {\xi , \eta} V _ {k} ^ {- 1} \left(\operatorname {p r o x} _ {y} ^ {k} - q _ {k} ^ {\frac {1}{2}} \eta - m _ {k}\right) \eta^ {\top} \\ \hat {m} _ {k} = \alpha \rho_ {k} \mathbb {E} _ {\xi , \eta} V _ {k} ^ {- 1} \left(\operatorname {p r o x} _ {y} ^ {k} - q _ {k} ^ {\frac {1}{2}} \eta - m _ {k}\right) \\ \hat {q} = \alpha \sum_ {k = 1} ^ {K} \rho_ {k} \mathbb {E} _ {\xi , \eta} V ^ {- 1} \left(\operatorname {p r o x} _ {x} ^ {k} - q ^ {\frac {1}{2}} \xi\right) ^ {\otimes 2} V ^ {- 1} \\ \hat {V} = - \alpha \sum_ {k = 1} ^ {K} \rho_ {k} \mathbb {E} _ {\xi , \eta} \left[ q ^ {- \frac {1}{2}} V ^ {- 1} \left(\operatorname {p r o x} _ {x} ^ {k} - q ^ {\frac {1}{2}} \xi\right) \xi^ {\top} - \sigma (\sqrt {1 - \Delta} \operatorname {p r o x} _ {y} ^ {k} + \operatorname {p r o x} _ {x} ^ {k}) ^ {\otimes 2} \right] \end{array} \right. \tag {74}
$$
$$
\left\{ \begin{array}{l} q _ {k} = \int d \nu (\gamma , \tau) \gamma_ {k} \left(\lambda \mathbb {I} _ {p} + \hat {V} + \sum_ {j = 1} ^ {K} \gamma_ {j} \hat {V} _ {j}\right) ^ {- 2} \left(\hat {q} + \sum_ {j = 1} ^ {K} \gamma_ {j} \hat {q} _ {j} + \sum_ {1 \leq j, l \leq K} \tau_ {j} \tau_ {l} \hat {m} _ {j} \hat {m} _ {l} ^ {\top}\right) \\ V _ {k} = \int d \nu (\gamma , \tau) \gamma_ {k} \left(\lambda \mathbb {I} _ {p} + \hat {V} + \sum_ {j = 1} ^ {K} \gamma_ {j} \hat {V} _ {j}\right) ^ {- 1} \\ m _ {k} = \int d \nu (\gamma , \tau) \tau_ {k} \left(\lambda \mathbb {I} _ {p} + \hat {V} + \sum_ {j = 1} ^ {K} \gamma_ {j} \hat {V} _ {j}\right) ^ {- 1} \sum_ {j = 1} ^ {K} \tau_ {j} \hat {m} _ {j} \\ q = \int d \nu (\gamma , \tau) \left(\lambda \mathbb {I} _ {p} + \hat {V} + \sum_ {j = 1} ^ {K} \gamma_ {j} \hat {V} _ {j}\right) ^ {- 2} \left(\hat {q} + \sum_ {j = 1} ^ {K} \gamma_ {j} \hat {q} _ {j} + \sum_ {1 \preceq j, l \preceq K} \tau_ {j} \tau_ {l} \hat {m} _ {j} \hat {m} _ {l} ^ {\top}\right) \\ V = \int d \nu (\gamma , \tau) \left(\lambda \mathbb {I} _ {p} + \hat {V} + \sum_ {j = 1} ^ {K} \gamma_ {j} \hat {V} _ {j}\right) ^ {- 1} \end{array} , \right. \tag {75}
$$
which is exactly (13). This closes the derivation of Conjecture 3.3.
As a final heuristic remark, observe that from (74) it is reasonable to expect that all the hatted statistics  $\hat{q}_k,\hat{V}_k,\hat{m}_k,\hat{q},\hat{V}$  should be of order  $\Theta (\alpha)$  as  $\alpha \gg 1$  . As a consequence,  $V_{k} = \Theta (^{1} / \alpha)$  and  $q,m,q_k,m_k$  should approach their  $\alpha \rightarrow \infty$  limit as  $\Theta (^{1} / \alpha)$  . Therefore, the MSE (61) should also approach its  $\alpha \rightarrow \infty$  limit as  $\Theta (^{1} / \alpha)$
A.9 Additional comparisons
We close this appendix by providing further discussion on some points, including the role of the non-linearity  $\sigma(\cdot)$ , the influence of the weight-tying assumption, and of the (an-)isotropy and (homo/hetero)-scedasticity of the clusters.
The activation function The figures in the main text were generated using  $\sigma = \tanh$ . Several studies for RAEs, however, have highlighted that an auto-encoder would optimally seek to place itself in the linear region of its nonlinearity, so as to learn the principal components of the training data, at least in the non-linear untied weights case, see e.g. [3, 8]. In light of these findings, it is legitimate to wonder whether setting a linear activation  $\sigma(x) = x$  would not yield a better MSE. Fig. 6 shows that it is not the case. For the binary isotropic mixture (13), the linear activations yield a worse performance than the tanh activation.