SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Weight-tying We have assumed that the weights of the DAE (2) were tied. While this assumption originates mainly for technical reasons in the derivation, note that it has been also used and discussed in practical setting [3] as a way to prevent the DAE from going to the linear region of its non-linearity, by making the norm of the first layer weights very small, and that of the second layer very large to compensate [8]. A full extension of Conjecture 3.3 to the case of the untied weight is a lengthy theoretical endeavour, which we leave for future work. However, note that Fig. 6 shows, in the binary isotropic case, that untying the weights actually leads to a worse MSE than the DAE with tied weights. This is a very interesting observation, as a DAE with untied weights obviously has the expressive power to implement a DAE with tied weights. Therefore, this effect should be mainly due to the untied landscape presenting local minima trapping the optimization. A full landscape analysis would present a very important future research topic. Finally, remark that this stands in sharp contrast to the observation of [8] for non-linear RAEs, where weight-tying worsened the performance, as it prevents the RAE from implementing a linear principal component analysis.
Figure 6: Same parameters as Fig. 1 in the main text.  $\alpha = 1, K = 2, \rho_{1,2} = 1/2, \Sigma_{1,2} = 0.3 \times \mathbb{I}_d, p = 1$ ; the cluster mean  $\mu_1 = -\mu_2$  was taken as a random Gaussian vector of norm 1. Difference in MSE between the full DAE  $\hat{f}(2)$  and the rescaling network  $\hat{r}(3)$  for  $\sigma(\cdot) = \tanh(\text{blue})$  or  $\sigma(x) = x$  (red). Solid lines correspond to the sharp asymptotic characterization (13), which is a particular case of Conjecture 3.3. Dots represent numerical simulations for  $d = 700$ , training the DAE using the Pytorch implementation of full-batch Adam [43], with learning rate  $\eta = 0.05$  over 2000 epochs, averaged over  $N = 2$  instances. Error bars represent one standard deviation. In dashed green, numerical simulations for the same architecture, but untied weights. The learning parameters were left unchanged.
An important conclusion from these two observations is that, in contrast to RAEs, the DAE (2) does not just implement a linear principal component analysis — weight tying, and the presence of a non-linearity, which are obstacles for RAEs in reaching the PCA MSE, lead for DAEs (2) to a better MSE. The model (2) therefore constitutes a model of a genuinely non-linear algorithm, where the non-linearity is helpful and is not undone during training. Further discussion can be found in Section 4 of the main text.
Heteroscedasticity, anisotropy Fig. 1 and 3 in the main text all represent the isotropic, homoscedastic case. This simple model actually encapsulates all the interesting phenomenology. In this paragraph, we discuss for completeness the generically heteroscedastic, anisotropic case. We consider a binary mixture, with covariances  $\Sigma_1, \Sigma_2$  independently drawn from the Wishart-Laguerre ensemble, with aspect ratios  $5/7$  and  $5/6$ . Therefore, the clusters are anisotropic, and eigendirections associated with the largest eigenvalues are more "stretched" (i.e. induce higher cluster variance). Furthermore, since the set of eigenvectors of  $\Sigma_1, \Sigma_2$  are independent, the two clusters are stretched in different directions. To ease the comparison with Fig. 1 (isotropic, homoscedastic), these Wishart matrices were further divided by 10, so that the trace is approximately 0.09 like in Fig. 1 – i.e., the clusters have the same average extension. Fig. 7 presents the resulting metrics and summary statistics. Again, the agreement between the theory 3.3 and numerical simulation using Pytorch is compelling. Qualitatively, the observations made in Section 4 still hold true in the anisotropic heteroscedastic case:
- The curve of the cosine similarity still displays a non-monotonic behaviour, signalling the preferential activation by the DAE of its skip connection at low  $\Delta$  and of its network component at high  $\Delta$ .
- The skip connection strength  $\hat{b}$  is higher at small  $\Delta$  and decreases, in connection to the previous remark.
- The norm of the weight matrix  $\hat{w}$  overall increases with the noise level  $\Delta$ , signalling an increasing usage by the DAE of its bottleneck network component, again in accordance to the previous remark.
Therefore, the generic case does not introduce qualitative changes compared to the isotropic case.
Strength of the weight decay  $\lambda$  A  $\ell_2$  regularization (weight decay)  $\lambda = 0.1$  was adopted in the main text. In fact, the value of the strength of the weight decay was not found to sensibly influence the curves, and again, the qualitative phenomena discussed in Section 4 are observed for any value. Fig. 8 shows, for an isotropic binary mixture, the MSE difference  $\mathrm{mse}_{\hat{f}} - \mathrm{mse}_{\hat{r}}$  (5) and the cosine similarity  $\theta$  (6) for regularization strength  $\lambda = 0.1$  and  $\lambda = 0.001$ . As can be observed, even reducing the regularization a hundredfold does not change at all the qualitative picture – in particular, the non-monotonicity of the cosine similarity, discussed in Section 4 –, and very little the quantitative values. On the
Figure 7:  $\alpha = 1, K = 2, \rho_{1,2} = 1/2, p = 1, \lambda = 0.1, \sigma = \tanh$ ; the cluster mean  $\mu_1 = -\mu_2$  was taken as a random Gaussian vector of norm 1. The cluster covariances were independently drawn from the Wishart-Laguerre ensemble, with aspect ratios  $5/6$  and  $5/7$ , before being normalized by 10 to match the trace of the isotropic case in Fig. 1. In particular, the eigenvectors are independent, so the mixture is totally anisotropic with different main variance directions and heteroscedastic. (left) In blue, the difference in MSE between the full DAE  $\hat{f}(2)$  and the rescaling network  $\hat{r}(3)$ . Solid lines correspond to the sharp asymptotic characterization of Conjecture 3.3 in its generic formulation (58). Dots represent numerical simulations for  $d = 500$ , training the DAE using the Pytorch implementation of full-batch Adam [43], with learning rate  $\eta = 0.05$  over 2000 epochs, averaged over  $N = 5$  instances. Error bars represent one standard deviation. (right) Cosine similarity  $\theta$  (6) (green), squared weight norm  $\| \hat{w} \|^2 / d$  (red) and skip connection strength  $\hat{b}$  (blue). Solid lines correspond to the formulas (11)(12) and (10) of Conjecture 3.3; dots are numerical simulations. For completeness, the cosine similarity of the first principal component of the train set is plotted in dashed black.
Figure 8: Same parameters as Fig. 1 in the main text.  $\alpha = 1, K = 2, \rho_{1,2} = 1/2, \Sigma_{1,2} = 0.09 \times \mathbb{I}_d, p = 1, \sigma = \tanh$ ; the cluster mean  $\mu_1 = -\mu_2$  was taken as a random Gaussian vector of norm 1. Difference in MSE between the full DAE  $\hat{f}(2)$  and the rescaling network  $\hat{r}(3)$  for  $\lambda = 0.1$  (blue) or  $\lambda = 0.001$  (red). Lines correspond to the sharp asymptotic characterization (13) of Conjecture 3.3. Dots represent numerical simulations for  $d = 700$ , training the DAE using the Pytorch implementation of full-batch Adam [43], with learning rate  $\eta = 0.05$  over 2000 epochs, averaged over  $N = 2$  instances. (right) Cosine similarities, for  $\lambda = 0.1$  (green) and  $\lambda = 0.001$  (orange); lines correspond to the asymptotic formulae of Conjecture 3.3, dots represent the numerical simulations.
Figure 9: Like Fig. 1 of the main text,  $\alpha = 1$ ,  $K = 2$ ,  $\rho_{1,2} = \frac{1}{2}$ ,  $\Sigma_{1,2} = 0.09 \times \mathbb{I}_d$ ,  $p = 1$ ,  $\sigma(\cdot) = \tanh(\cdot)$ ; the cluster mean  $\mu_1 = -\mu_2$  was taken as a random Gaussian vector of norm 1. (left) Solid lines correspond to the difference in MSE between the full DAE  $\hat{f}$  (2) and the rescaling component  $\hat{r}$  (3) as predicted by Conjecture 3.3. Different colors indicate different regularization strength  $\lambda$ . For large noises  $\Delta$  and insufficient regularization  $\lambda$ , the DAE overfits the train set, leading to performances worse than the simple rescaling  $\hat{r}$ , as signalled by positive values of  $\mathrm{mse}_{\hat{f}} - \mathrm{mse}_{\hat{r}}$ . This effect is suppressed for larger regularizations  $\lambda$ .
Figure 10:  $\alpha = 1, K = 1, \rho_{1,2} = \frac{1}{2}, \Sigma_1 = 0.09 \times \mathbb{I}_d, p = 1, \sigma(\cdot) = \tanh(\cdot)$ . (left) Difference between the MSE of the DAE  $\hat{f}$  and the MSE of the rescaling  $\hat{r} \operatorname{mse}_{\hat{f}} - \operatorname{mse}_{\hat{r}}$  in the case of a single Gaussian cluster. Solid lines correspond to the theoretical prediction of Conjecture 3.3 and dots to numerical simulations. In the case of a single cluster, the bottleneck component of the DAE is not needed, and its presence leads to detrimental overfitting, as signalled by the positive value of  $\operatorname{mse}_{\hat{f}} - \operatorname{mse}_{\hat{r}}$ . (right) Difference between the MSE of the DAE  $\hat{f}$  and the MSE of the bottleneck network  $\hat{u}$ , divided by  $d$ . Like the  $K > 1$  case (see Fig. 3(b)), this difference is of order  $\Theta_d(d)$ , and the MSE of PCA (green dots) is sensibly equal.
other hand, observe that at the level of the MSE increasing the weight decay helps palliate the overfitting at large noise levels  $\Delta$ , as we further illustrate in Fig. 9
$K = 1$  cluster We finally discuss the case of unstructured data, where the Gaussian mixture only has a single cluster. Analytically, this corresponds to setting the norm of the cluster mean  $\| \pmb{\mu}\|$  to zero in Conjecture 3.3. The bottleneck component, whose role is to learn the data structure - as given by the cluster mean-and leverage it to improve the denoising performance, is no longer needed when the data is unstructured. It presence actually leads the DAE to overfit, as can be observed in Fig.10 (left). Similarly, the PCA denoiser performs worse in the unstructured case, with an associated MSE remaining of order  $\Theta (d)$ , see Fig.10 (right).
B Tweedie baseline
B.1 Oracle denoiser
Tweedie's formula [42] provides the best estimator of a clean sample  $x$  given a noisy sample  $\tilde{x}$  corrupted by Gaussian noise, as
$$
t (\tilde {\boldsymbol {x}}) = \frac {\tilde {\boldsymbol {x}}}{\sqrt {1 - \Delta}} + \frac {\Delta}{\sqrt {1 - \Delta}} \nabla \mathbb {P} | _ {\tilde {\boldsymbol {x}}}. \tag {76}
$$
Note that this gives the Bayes-optimal estimator for the MSE, assuming perfect knowledge of the clean data distribution  $\mathbb{P}$  from which the so-called score  $\nabla \mathbb{P}$  has to be computed. Of course, this knowledge is inaccessible in general, where the only information on  $\mathbb{P}$  is provided in the form of the train set  $\mathcal{D}$ . Therefore, Tweedie's formula does not give a learning algorithm, but allows to give an oracle lower-bound on the achievable MSEs, as any learning algorithm will yield a higher MSE than this information-theoretic baseline.
For a generic Gaussian mixture (1), Tweedie's formula reduces to the expression
$$
t (\boldsymbol {x}) = \frac {1}{\sqrt {1 - \Delta}} \frac {\sum_ {k = 1} ^ {K} \rho_ {k} \left[ \tilde {\Sigma} _ {k} \left(\tilde {\Sigma} _ {k} + \Delta \mathbb {I} _ {d}\right) ^ {- 1} \boldsymbol {x} + \left(\tilde {\Sigma} _ {k} + \Delta \mathbb {I} _ {d}\right) ^ {- 1} \tilde {\mu} _ {k}\right) \mathcal {N} \left(\boldsymbol {x}; \tilde {\mu} _ {k} , \tilde {\Sigma} _ {k} + \Delta \mathbb {I} _ {d}\right)}{\sum_ {k = 1} ^ {K} \rho_ {k} \mathcal {N} \left(\boldsymbol {x}; \tilde {\mu} _ {k} , \tilde {\Sigma} _ {k} + \Delta \mathbb {I} _ {d}\right)}, \tag {77}
$$
where we noted
$$
\tilde {\boldsymbol {\mu}} _ {k} = \sqrt {1 - \Delta} \boldsymbol {\mu} _ {k}, \quad \tilde {\boldsymbol {\Sigma}} _ {k} = (1 - \Delta) \boldsymbol {\Sigma} _ {k}. \tag {78}
$$
B.2 Oracle test MSE
In general, except in the complete homoscedastic case where all clusters have the same covariance, there is no closed-form asymptotic expression for the MSE achieved by the Tweedie denoiser. In the binary homoscedastic case  $\pmb{\mu}_1 = -\pmb{\mu}_2 \equiv \pmb{\mu}$ ,  $\pmb{\Sigma}_1 = \pmb{\Sigma}_2 = \sigma^2\mathbb{I}_d$  (see Fig. 1), Tweedie's formula (77) reduces to the compact form
$$
t (\boldsymbol {x}) = \frac {\sqrt {1 - \Delta} \sigma^ {2}}{\sigma^ {2} (1 - \Delta) + \Delta} \tilde {\boldsymbol {x}} + \frac {\Delta}{\sigma^ {2} (1 - \Delta) + \Delta} \tanh  \left(\frac {\tilde {\boldsymbol {x}} ^ {\top} \boldsymbol {\mu} \sqrt {1 - \Delta}}{\sigma^ {2} (1 - \Delta) + \Delta}\right) \times \boldsymbol {\mu} \tag {79}
$$
Note that this is of the same form as the DAE architecture (2). The associated MSE reads
$$
\operatorname {m s e} ^ {\star} = \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {\xi}} \left\| \boldsymbol {x} - \left[ b \times (\sqrt {1 - \Delta} \boldsymbol {x} + \boldsymbol {\xi}) + \frac {\Delta \boldsymbol {\mu}}{\sigma^ {2} (1 - \Delta) + \Delta} \sigma \left(\frac {\sqrt {1 - \Delta} \boldsymbol {\mu} ^ {\top} (\sqrt {1 - \Delta} \boldsymbol {x} + \boldsymbol {\xi})}{\sigma^ {2} (1 - \Delta) + \Delta}\right) \right] \right\| ^ {2}. \tag {80}
$$
A sharp asymptotic formula can be found to be
$$
\begin{array}{l} \mathrm {m s e} ^ {\star} = \mathrm {m s e} _ {\circ} + \frac {\sigma_ {e} ^ {4}}{(\sigma^ {2} (1 - \Delta) + \Delta) ^ {2}} \frac {1}{2} \sum_ {s = \pm 1} \mathbb {E} _ {z} ^ {\mathcal {N} (0, 1)} \left[ \sigma \left(\frac {(1 - \Delta) s + \sqrt {1 - \Delta} z \times \sqrt {\Delta + (1 - \Delta) \sigma^ {2}}}{\sigma^ {2} (1 - \Delta) + \Delta}\right) ^ {2} \right] \\ - \frac {2 \Delta}{\sigma^ {2} (1 - \Delta) + \Delta} \frac {1}{2} \sum_ {s = \pm 1} \mathbb {E} _ {z} ^ {\mathcal {N} (0, 1)} \left[ \sigma \left(\frac {(1 - \Delta) s + \sqrt {1 - \Delta} z \times \sqrt {\Delta + (1 - \Delta) \sigma^ {2}}}{\sigma^ {2} (1 - \Delta) + \Delta}\right) \right] \times \left((1 - b \sqrt {1 - \Delta}) s\right). \tag {81} \\ \end{array}
$$
This is the theoretical characterization plotted in Figs. 1 and 3 in the main text.
As discussed in the main text and illustrated in Fig. 3(left), the MSE of the DAE approaches the oracle MSE (81) as the sample complexity  $\alpha$  grows. On the other hand, the DAE MSE does not exactly converge to the oracle MSE as  $\alpha \to \infty$ . Intuitively, this is because of the fact that while the form of the oracle (79) is that of a DAE, it does not have tied weights, like the considered architecture (2). The latter cannot therefore perfectly realize the oracle denoiser, whence a small discrepancy, as more precisely numerically characterized in Fig. 11.
Figure 11: Like Fig. 1 of the main text,  $K = 2$ ,  $\rho_{1,2} = \frac{1}{2}$ ,  $\Sigma_{1,2} = 0.09 \times \mathbb{I}_d$ ,  $p = 1$ ,  $\sigma(\cdot) = \tanh(\cdot)$ ; the cluster mean  $\pmb{\mu}_1 = -\pmb{\mu}_2$  was taken as a random Gaussian vector of norm 1. Solid lines correspond to the difference in MSE between the full DAE  $\hat{f}$  and the rescaling component  $\hat{r}$  as predicted by Conjecture 3.3. In dashed lines, the oracle MSE (81), minus the rescaling MSE  $\mathrm{mse}_{\hat{r}}$ . Different colors correspond to different noise levels  $\Delta$ . As the sample complexity  $\alpha$  becomes large, the denoiser MSE approaches the oracle baseline, but does not become exactly equal to it as  $\alpha \to \infty$ .
C Bayes-optimal MSE
C.1 Assuming imperfect prior knowledge
The Tweedie denoiser discussed in Appendix B is an oracle denoiser, in the sense that perfect knowledge of all the parameters  $\{\pmb{\mu}_k,\pmb{\Sigma}_k\}_{k = 1}^K$  of the Gaussian mixture distribution  $\mathbb{P}$  (1) is assumed. Therefore, though the comparison of the DAE (2) MSE with the oracle Tweedie MSE  $\mathrm{mse}^\star$  does provide useful intuition (see Fig. 3), it importantly does not allow to disentangle which part of the MSE of the DAE is due to the limited expressivity of the architecture, and which is due to the limited availability of training samples - which entails imperfect knowledge of the parameters  $\{\pmb{\mu}_k,\pmb{\Sigma}_k\}_{k = 1}^K$  of the Gaussian mixture distribution  $\mathbb{P}$  (1). In this appendix, we derive the MSE of the Bayes optimal estimator (with respect to the MSE), assuming only knowledge of the distribution from which the parameters  $\{\pmb{\mu}_k,\pmb{\Sigma}_k\}_{k = 1}^K$  are drawn. For simplicity and conciseness, we limit ourselves to the binary isotropic and homoscedastic mixture (see e.g. Fig. 1), for which  $\pmb{\mu}_1 = -\pmb{\mu}_2\equiv \pmb{\mu}^\star$  and  $\pmb {\Sigma}_1 = \pmb {\Sigma}_2 = \sigma \mathbb{I}_d$ . For definiteness, we assume  $\pmb{\mu}\sim \mathbb{P}_{\mu}$ , with  $\mathbb{P}_{\mu} = \mathcal{N}(0,\mathbb{I}_d / d)$ , so that with high probability  $\| \pmb {\mu}\| = 1$ . We shall moreover assume for ease of discussion that  $\sigma$  is perfectly known. Thus, the centroid  $\pmb{\mu}$  is the only unknown parameter.
C.2 Bayes-optimal denoiser without knowledge of cluster means
The corresponding Bayes-optimal denoiser then reads
$$
\mathfrak {b} (\tilde {\boldsymbol {x}}) \equiv \mathbb {E} [ \boldsymbol {x} | \tilde {\boldsymbol {x}}, \sigma , \mathcal {D} ] = \int d \boldsymbol {\mu} \underbrace {\mathbb {E} [ \boldsymbol {x} | \tilde {\boldsymbol {x}} , \boldsymbol {\mu} , \sigma , \mathcal {D} ]} _ {t (\boldsymbol {\mu}, \tilde {\boldsymbol {x}})} \mathbb {P} (\boldsymbol {\mu} | \sigma , \mathcal {D}) \tag {82}
$$
Where we have identified the oracle denoiser (79), and emphasized the dependence on  $\pmb{\mu}$ . Note that this is a slight abuse of notation, as the true oracle  $t(\pmb{\mu}^{\star},\tilde{\pmb{x}})$  involves the ground truth centroid. Further note that  $\mathbb{P}(\pmb {\mu}|\sigma ,\mathcal{D}) = \mathbb{P}(\pmb {\mu}|\mathcal{D})$  since knowledge of  $\sigma$  does not bring information about  $\pmb{\mu}$ . Furthermore, note that as the noisy part of the dataset  $\tilde{\mathcal{D}}\equiv \{\tilde{x}^{\mu}\}_{\mu = 1}^{n}$  brings only redundant information, one further has that
$$
\mathbb {P} (\boldsymbol {\mu} | \mathcal {D}) = \mathbb {P} (\boldsymbol {\mu} | \tilde {\mathcal {D}}), \tag {83}
$$
where we noted  $\tilde{\mathcal{D}} = \{\pmb{x}^{\mu}\}_{\mu = 1}^{n}$  the clean part of the dataset. Thus
$$
\begin{array}{l} \mathfrak {b} (\tilde {\boldsymbol {x}}) = \frac {1}{\mathbb {P} (\breve {\mathcal {D}})} \int d \boldsymbol {\mu} t (\boldsymbol {\mu}, \tilde {\boldsymbol {x}}) \mathbb {P} (\breve {\mathcal {D}} | \boldsymbol {\mu}) \mathbb {P} _ {\boldsymbol {\mu}} (\boldsymbol {\mu}) \\ = \frac {1}{\mathbb {P} (\tilde {\mathcal {D}})} \int d \boldsymbol {\mu} t (\boldsymbol {\mu}, \tilde {\boldsymbol {x}}) \mathbb {P} _ {\boldsymbol {\mu}} (\boldsymbol {\mu}) \prod_ {\mu = 1} ^ {n} \frac {1}{2} \frac {1}{(2 \pi \sigma^ {2}) ^ {d / 2}} \left[ e ^ {- \frac {1}{2 \sigma^ {2}} \| \boldsymbol {x} ^ {\mu} - \boldsymbol {\mu} \| ^ {2}} + e ^ {- \frac {1}{2 \sigma^ {2}} \| \boldsymbol {x} ^ {\mu} + \boldsymbol {\mu} \| ^ {2}} \right] \\ = \frac {1}{\mathbb {P} (\tilde {\mathcal {D}})} \int d \boldsymbol {\mu} t (\boldsymbol {\mu}, \tilde {\boldsymbol {x}}) \frac {1}{(2 \pi / d) ^ {d / 2}} e ^ {- \frac {d}{2} \| \boldsymbol {\mu} \| ^ {2}} e ^ {- \frac {n}{2 \sigma^ {2}} \| \boldsymbol {\mu} \| ^ {2}} \prod_ {\mu = 1} ^ {n} \frac {e ^ {- \frac {1}{2 \sigma^ {2}} \| \boldsymbol {x} ^ {\mu} \| ^ {2}}}{(2 \pi \sigma^ {2}) ^ {d / 2}} \cosh \left(\frac {\boldsymbol {\mu} ^ {\top} \boldsymbol {x} ^ {\mu}}{\sigma^ {2}}\right) \\ = \frac {1}{\sqrt {d} \mathbb {P} (\tilde {\mathcal {D}})} \int d \boldsymbol {\mu} t (\boldsymbol {\mu} / \sqrt {d}, \tilde {\boldsymbol {x}}) \frac {1}{(2 \pi / d) ^ {d / 2}} e ^ {- \frac {1}{2} \| \boldsymbol {\mu} \| ^ {2}} e ^ {- \frac {\alpha}{2 \sigma^ {2}} \| \boldsymbol {\mu} \| ^ {2}} \prod_ {\mu = 1} ^ {n} \frac {e ^ {- \frac {1}{2 \sigma^ {2}} \| \boldsymbol {x} ^ {\mu} \| ^ {2}}}{(2 \pi \sigma^ {2}) ^ {d / 2}} \cosh \left(\frac {\boldsymbol {\mu} ^ {\top} \boldsymbol {x} ^ {\mu}}{\sigma^ {2} \sqrt {d}}\right) \\ = \frac {1}{Z} \int d \boldsymbol {\mu} t (\boldsymbol {\mu} / \sqrt {d}, \tilde {\boldsymbol {x}}) \mathbb {P} _ {b} (\boldsymbol {\mu}), \tag {84} \\ \end{array}
$$
$$
\mathbb {P} _ {b} (\boldsymbol {\mu}) \equiv e ^ {- \frac {1}{2 \bar {\sigma} ^ {2}} \| \boldsymbol {\mu} \| ^ {2}} \prod_ {\mu = 1} ^ {n} \cosh \left(\frac {\boldsymbol {\mu} ^ {\top} \boldsymbol {x} ^ {\mu}}{\sigma^ {2} \sqrt {d}}\right), \tag {85}
$$
with the effective variance
$$
\hat {\sigma} ^ {2} \equiv \frac {\sigma^ {2}}{\sigma^ {2} + \alpha}. \tag {86}
$$
Finally, the partition function  $Z$  is defined as the normalisation of  $\mathbb{P}_b$ , i.e.
$$
Z \equiv \int d \boldsymbol {\mu} \mathbb {P} _ {b} (\boldsymbol {\mu}). \tag {87}
$$
One is now in a position to compute the MSE associated with the Bayes estimator  $\mathsf{b}(\hat{\pmb{x}})$ :
mseb
We have adopted the shortcuts
$$
b = \frac {\sqrt {1 - \Delta} \sigma^ {2}}{\sigma^ {2} (1 - \Delta) + \Delta}, \quad \Omega = \left[ \begin{array}{l l} \frac {\| \boldsymbol {\mu} _ {1} \| ^ {2}}{d} & \frac {\boldsymbol {\mu} _ {1} ^ {\top} \boldsymbol {\mu} _ {2}}{d} \\ \frac {\boldsymbol {\mu} _ {1} ^ {\top} \boldsymbol {\mu} _ {2}}{d} & \frac {\| \boldsymbol {\mu} _ {2} \| ^ {2}}{d}. \end{array} \right]. \tag {89}
$$
(88) shows that the inner averages involved in the Bayes MSE  $\mathrm{mse}_{\mathfrak{h}}$  only depend on the overlaps  $\mu_{1,2}^{\top}\mu_{1,2} / d$  for  $\pmb{\mu}_1,\pmb{\mu}_2$  two independently sampled vectors from  $\mathbb{P}_b$ . Motivated by similar high-dimensional studies, it is reasonable to expect these quantities to concentrate as  $n,d\to \infty$  in the measure  $\mathbb{E}_{\mathcal{D}}\langle \cdot \rangle_{\mathbb{P}_b}$ . A more principle - but much more painstaking way to derive this assumption is to introduce a Fourier representation, and carry out the  $\mathbb{E}_{\mathcal{D}}$  average using the replica method. We refer the interested reader to, for instance, Appendix H of [36], where such a route is taken.
C.3 Derivation
To compute the summary statistics of  $\mathbb{P}_b$ , we again resort to the replica method to compute the moment-generating function (free entropy), see also Appendix A. The replicated partition function reads
$$
\mathbb {E} _ {\mathcal {D}} Z ^ {s} = \int \prod_ {a = 1} ^ {s} d \boldsymbol {\mu} _ {a} e ^ {- \frac {1}{2 \hat {\sigma} ^ {2}} \| \boldsymbol {\mu} _ {a} \| ^ {2}} \prod_ {\mu = 1} ^ {n} \sum_ {s = \pm 1} \frac {1}{2} \mathbb {E} _ {\boldsymbol {\eta} \sim \mathcal {N} (0, \mathbb {I} _ {d})} \left[ \prod_ {a = 1} ^ {s} \cosh \left(\frac {s \frac {\boldsymbol {\mu} _ {a} ^ {\top} \boldsymbol {\mu} ^ {\star}}{d} + \sigma \frac {\boldsymbol {\mu} _ {a} ^ {\top} \boldsymbol {\eta}}{\sqrt {d}}}{\sigma^ {2}}\right) \right]. \tag {90}
$$
Introducing the order parameters
$$
q _ {a b} \equiv \frac {\boldsymbol {\mu} _ {a} ^ {\top} \boldsymbol {\mu} _ {b}}{d}, \quad m _ {a} \equiv \frac {\boldsymbol {\mu} _ {a} ^ {\top} \boldsymbol {\mu} ^ {\star}}{d}, \tag {91}
$$
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {D}} Z ^ {s} = \int \prod_ {a \leq b} d q _ {a b} d \hat {q} _ {a b} \prod_ {a = 1} ^ {s} d m _ {a} d \hat {m} _ {a} e ^ {- \underbrace {d \sum_ {a \leq b} q _ {a b} \hat {q} _ {a b} - d \sum_ {a} m _ {a} \hat {m} _ {a}} _ {e ^ {s d \Psi_ {t}}}} \underbrace {\int \prod_ {a = 1} ^ {s} d \boldsymbol {\mu} _ {a} e ^ {- \frac {1}{2 \hat {\sigma} ^ {2}} \| \boldsymbol {\mu} _ {a} \| ^ {2}} e ^ {\sum_ {a \leq b} \hat {q} _ {a b} \boldsymbol {\mu} _ {a} ^ {\top} \boldsymbol {\mu} _ {b} + \sum_ {a} \boldsymbol {\mu} _ {a} ^ {\top} \boldsymbol {\mu} ^ {*}}} _ {e ^ {s d \Psi_ {w}}}. \\ \underbrace {\left[ n \sum_ {s = \pm 1} \frac {1}{2} \int \mathcal {N} \left(\left\{\lambda_ {a} \right\} _ {a = 1} ^ {s}; 0 , \left(q _ {a b}\right) _ {1 \leq a , b \leq s}\right) \left[ \prod_ {a = 1} ^ {s} \cosh \left(\frac {s m _ {a} + \sigma \lambda_ {a}}{\sigma^ {2}}\right) \right] \right] ^ {n}} _ {e ^ {s \alpha d \Psi_ {y}}}. \tag {92} \\ \end{array}
$$