SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(13) provides a closed-form expression for the summary statistics (12), in terms of the solutions of the low-dimensional optimization (14). The latter can be loosely viewed as an effective loss, subsuming the averaged effect of the finite training set. Further remark that while the regularization  $\lambda$  does not explicitly appear in the expression for the MSE (8), the statistics  $q_{k}, m_{k}$  in (8) depend thereon through (13). In fact, Assumptions 3.1 and 3.2 are not strictly necessary, and can be simultaneously relaxed to address arbitrary convex regularizer  $g(\cdot)$  and generically non-commuting  $\{\pmb{\Sigma}_{k}\}_{k=1}^{K}$  but at the price of more intricate formulae. For this reason, we choose to discuss here Conjecture 3.3, and defer a discussion and detailed derivation of the generic asymptotically exact formulae to Appendix A, see eq. (58). Let us mention that a sharp asymptotic characterization of the train MSE can also be derived; for conciseness, we do not present it here and refer the interested reader to equation (68) in Appendix A. Conjecture 3.3 encompasses as special cases the asymptotic characterization of the components  $\hat{r}, \hat{u}$  (3):
Corollary 3.4. (MSE of components) The test MSE of  $\hat{r}$  (3) is given by  $\mathrm{mse}_{\hat{r}} = \mathrm{mse}_{\circ}$  (9). Furthermore, the learnt value of its single parameter  $\hat{c}$  is given by (10). The test MSE, cosine similarity and summary statistics of the bottleneck network  $\hat{u}$  (3) follow from Conjecture 3.3 by setting  $\hat{b} = 0$ .
The implications of Corollary 3.4 shall be further discussed in Section 4, and a full derivation is provided in Appendix E. Finally, remark that in the noiseless limit  $\Delta = 0$ , the denoising task reduces to a reconstruction task, with the autoencoder being tasked with reproducing the clean data as an output when taking the same clean sample as an input. Therefore Conjecture 3.3 also includes RAEs (by definition, without skip connection) as a special case.
Corollary 3.5. (RAEs) In the  $n, d \to \infty$  limit, the MSE, cosine similarity and summary statistics for an RAE follow from Conjecture 3.3 by setting  $x = 0$  in (14), removing the first term in the brackets in the equation of  $\hat{V}$  (13) and taking the limit  $\Delta, \hat{q}, \hat{b} \to 0$ .
Corollary 3.5 will be the object of further discussion in Section 4. A detailed derivation is presented in Appendix F. Note that Corollary 3.5 provides a characterization of RAEs as a function of the sample complexity  $\alpha$ , where previous studies on non-linear RAEs rely on the assumption of an infinite number of available training samples [13, 8, 9].
Equations (10) and (12) of Conjecture 3.3 thus characterize the statistics of the learnt parameters  $\hat{b},\hat{w}$  of the trained DAE (2). These summary statistics are, in turn, sufficient to fully characterize the
learning metrics (5) and (6) via equations (8) and (11). We thus have reduced the high-dimensional optimization (4) and the high-dimensional average over the train set  $\mathcal{D}$  involved in the definition of the metrics (5) and (6) to a simpler system of equations over  $4 + 6K$  variables (13) which can be solved numerically. It is important to note that all the summary statistics involved in (13) are finite-dimensional as  $d\to \infty$ , and therefore Conjecture 3.3 is a fully asymptotic characterization, in the sense that it does not involve any high-dimensional object. Finally, let us stress once more that the replica method employed in the derivation of these results should be viewed as a strong heuristic, but does not constitute a rigorous proof. While Conjecture 3.3 is stated in full generality, we focus for definiteness in the rest of the manuscript on the simple case  $r = 1,K = 2$ , which is found to already display all the interesting phenomenology discussed in this work, and leave the thorough exploration of  $r > 1,K > 2$  settings to future work. In the next paragraphs, we give two examples of applications of Conjecture 3.3, to a simple binary isotropic mixture, and to real data sets.
Example 1: Isotropic homoscedastic mixture We give as a first example the case of a synthetic binary Gaussian mixture with  $K = 2$ ,  $\pmb{\mu}_1 = -\pmb{\mu}_2$ ,  $\Sigma_{1,2} = 0.09 \times \mathbb{I}_d$ ,  $\rho_{1,2} = 1/2$ , using a DAE with  $\sigma = \tanh$  and  $p = 1$ . Since this simple case exhibits the key phenomenology discussed in the present work, we refer to it in future discussions. The MSE  $\mathrm{mse}_{\hat{f}}$  (8) evaluated from the solutions of the self-consistent equations (13) is plotted as the solid blue line in Fig. 1 (left) and compared to numerical simulations corresponding to training the DAE (2) with the Pytorch implementation of the Adam optimizer [43] (blue dots), for sample complexity  $\alpha = 1$  and  $\ell_2$  regularization (weight decay)  $\lambda = 0.1$ . The agreement between the theory and simulation is compelling. The green solid line and corresponding green dots in Fig. 1 (right) correspond to the replica prediction (11) and simulations for the cosine similarity  $\theta$  (6), and again display very good agreement. Note that for large noise levels, the DAE achieves a worse MSE than the rescaling –as shown by the positive value of  $\mathrm{mse}_{\hat{f}} - \mathrm{mse}_{\hat{r}^{-}}$ , despite the former being a priori expressive enough to realize the latter. This in fact signals that the DAE overfits the training data. That such an overfitting is captured is a strength of our analysis, which allows to cover the effect of a limited sample complexity. Finally, this overfitting can be mitigated by increasing the weight decay  $\lambda$ , see Fig. 9 in Appendix A.
A particularly striking observation is that due to the non-convexity of the loss (4), there is a priori no guarantee that an Adam-optimized DAE should find a global minimum, as described by the Conjecture 3.3, rather than a local minimum. The compelling agreement between theory and simulations in Fig. 1 temptingly suggests that –at least in this case– the loss landscape of DAEs (2) trained with the loss (4) for the data model (1) should in some way be benign. Authors of [12] have shown, for linear RAEs, that there exists a unique global and local minimum for the square loss and no regularizer. Ref. [14] offers further insight for a linear DAE in dimension  $d = 1$ , and shows that, aside from the global minima, the loss landscape only includes an unstable saddle point from which the dynamics easily escapes. Extending these works and intuitions to non-linear DAEs is an exciting research topic for future work.
Example 2: MNIST, FashionMNIST It is reasonable to ask whether Conjecture 3.3 is restricted to Gaussian mixtures (1). The answer is negative – in fact, Conjecture 3.3 also describes well a number of real data distributions. We provide such an example for FashionMNIST [16] (from which, for simplicity, we only kept boots and shoes) and MNIST [15] (1s and 7s), in Fig. 2. For each data set, samples sharing the same label were considered to belong to the same cluster. Note that we purposefully chose closely related classes, for which the clusters are expected to be closer, leading to an a priori harder – and thus more interesting – learning problem. The mean and covariance thereof were estimated numerically, and combined with Conjecture 3.3. The resulting denoising MSE predictions  $\mathrm{mse}_f$  are plotted as solid lines in Fig. 2, and agree very well with numerical simulations of DAEs optimized over the real data sets using the Pytorch implementation of Adam [43]. A full description of this experiment is given in Appendix D.
The observation that the MSEs of real data sets are to such degree of accuracy captured by the equivalent Gaussian mixture strongly hints at the presence of Gaussian universality [44]. This opens a gateway to future research, as Gaussian universality has hitherto been exclusively addressed in classification and regression (rather than denoising) settings, see e.g. [44, 45, 46]. Denoising tasks further constitute a particularly intriguing setting for universality results, as Gaussian universality would signify that only second-order statistics of the data can be reconstructed using a shallow autoencoder.
Figure 2: Difference in MSE between the full DAE (2) and the rescaling component (3) for the MNIST data set (middle), of which for simplicity only 1s and 7s were kept, and FashionMNIST (right), of which only boots and shoes were kept. In blue, the theoretical predictions resulting from using Conjecture 3.3 with the empirically estimated covariances and means, see Appendix D for further details. In red, numerical simulations of a DAE  $(p = 1, \sigma = \tanh)$  trained with  $n = 784$  training points, using the Pytorch implementation of full-batch Adam, with learning rate  $\eta = 0.05$  and weight decay  $\lambda = 0.1$  over 2000 epochs, averaged over  $N = 10$  instances. Error bars represent one standard deviation. (left) illustration of the denoised images: (top left) original image, (top right) noisy image, (bottom left) DAE  $\hat{f}$  (2), (bottom right) rescaling  $\hat{r}$  (3).
4 The role and importance of the skip connection.
Conjecture 3.3 for the full DAE  $\hat{f}$  (2) and Corollary 3.4 for its components  $\hat{r}$ ,  $\hat{u}$  (3) allow to disentangle the contribution of each part, and thus to pinpoint their respective roles in the DAE architecture. We sequentially present a comparison of  $\hat{f}$  with  $\hat{r}$ , and  $\hat{f}$  with  $\hat{u}$ . We remind that  $\hat{f}$ ,  $\hat{r}$ , and  $\hat{u}$  result from independent optimizations over the same train set  $\mathcal{D}$ , and that while  $f_{b,w} = u_w + h_b$ ,  $\hat{f} \neq \hat{u} + \hat{r}$ .
Full DAE and the rescaling component We start this section by observing that for noise levels  $\Delta$  below a certain threshold, the full DAE  $\hat{f}$  yields better MSE than the learnt rescaling  $\hat{r}$ , as can be seen by the negative value of  $\mathrm{mse}_{\hat{f}} - \mathrm{mse}_{\hat{r}}$  in Fig. 1 and Fig. 2. The improvement is more sizeable at intermediate noise levels  $\Delta$ , and is observed for a growing region of  $\Delta$  as the sample complexity  $\alpha$  increases, see Fig. 3 (a). This lower MSE further translates into visible qualitative changes in the result of denoising. As can be seen from Fig. 2 (left), the full DAE  $\hat{f}$  (2) (bottom left) yields denoised images with sensibly higher definition and overall contrast, while a simple rescaling  $\hat{r}$  (bottom right) leads to a still largely blurred image.
We provide one more comparison: for the isotropic binary mixture (see Fig. 1), the DAE test error  $\mathrm{mse}_{\hat{f}}$  in fact approaches the information-theoretic lowest achievable MSE  $\mathrm{mse}^{\star}$  as the sample complexity  $\alpha$  increases. To see this, note that  $\mathrm{mse}^{\star}$  is given by the application of Tweedie's formula [42], that requires perfect knowledge of the cluster means  $\mu_{k}$  and covariances  $\Sigma_{k}$  - it is, therefore, an oracle denoiser. A sharp asymptotic characterization of the oracle denoiser is provided in Appendix B. As can be observed from Fig. 3 (a), the DAE MSE (2) approaches -but does not exactly converge to (see Appendix C)- the oracle test error  $\mathrm{mse}^{\star}$  as the number of available training samples  $n$  grows, and is already sensibly close to the optimal value for  $\alpha = 8$ .
DAEs with(out) skip connection We now turn our attention to comparing the full DAE  $\hat{f}$  (2) to the bottleneck network component  $\hat{u}$  (3). It follows from Conjecture 3.3 and Corollary 3.4 that  $\hat{u}$  (3) leads to a higher MSE than the full DAE  $\hat{f}$  (2), with the gap being  $\Theta(d)$ . More precisely,
$$
\frac {1}{d} \left(\mathrm {m s e} _ {\hat {u}} - \mathrm {m s e} _ {\hat {f}}\right) = \frac {\left(\int d \nu_ {\gamma} (\gamma) \sum_ {k = 1} ^ {K} \rho_ {k} \gamma_ {k}\right) ^ {2} (1 - \Delta)}{\left(\int d \nu_ {\gamma} (\gamma) \sum_ {k = 1} ^ {K} \rho_ {k} \gamma_ {k}\right) (1 - \Delta) + \Delta}. \tag {15}
$$
Figure 3: (left) Solid lines: difference in MSE between the full DAE  $\hat{f}$  (2), with  $\sigma = \tanh$ ,  $p = 1$ , and the rescaling  $\hat{r}$  (3). Dashed: the same curve for the oracle denoiser. Different colours represent different sample complexities  $\alpha$  (solid lines). (right) Difference in MSE between the bottleneck network  $\hat{u}$  (3) and the complete DAE  $\hat{f}$  (2). In blue, the theoretical prediction (15); in red, numerical simulations for the bottleneck network (3) ( $\sigma = \tanh$ ,  $p = 1$ ) trained with the Pytorch implementation of full-batch Adam, with learning rate  $\eta = 0.05$  and weight decay  $\lambda = 0.1$  over 2000 epochs, averaged over  $N = 5$  instances, for  $d = 700$ . In green, the MSE (minus the MSE of the complete DAE (2)) achieved by PCA. Error bars represent one standard deviation. The model and parameters are the same as in Fig. 1.
(a)
(b)
Figure 4: Illustration of the denoised image for the various networks and algorithms. (a) original image (b) noisy image, for  $\sqrt{\Delta} = 0.2$  (c) trained rescaling  $\hat{r}$  (3) (d) full DAE  $\hat{f}$  (2) (e) bottleneck network  $\hat{u}$  (3) (f) PCA. The DAE and training parameters are the same as Fig. 2, see also Appendix D.
The theoretical prediction (15) compares excellently with numerical simulations; see Fig. 3 (right). Strikingly, we find that PCA denoising yields an MSE almost indistinguishable from  $\hat{u}$ , see Fig. 3, strongly suggesting that  $\hat{u}$  essentially learns, also in the denoising setting, to project the noisy data  $\tilde{\pmb{x}}$  along the principal components of the training set. The last two images of Fig. 4 respectively correspond to  $\hat{u}$  and PCA, which can indeed be observed to lead to visually near-identical results. This echoes the findings of [10, 11, 8, 9, 13] in the case of RAEs that bottleneck networks are limited by the PCA reconstruction performance - a conclusion that we also recover from Corollary 3.5, see Appendix F. Crucially however, it also means that compared to the full DAE  $\hat{f}(2)$ , PCA is sizeably suboptimal, since  $\mathrm{mse}_{\mathrm{PCA}} \approx \mathrm{mse}_{\hat{u}} = \mathrm{mse}_{\hat{f}} + \Theta(d)$ .
This last observation has an important consequence: in contrast to previously studied RAEs [10, 12, 11, 9, 13], the full DAE  $\hat{f}$  does not simply learn to perform PCA. In contrast to bottleneck RAE networks [8, 9, 13], the non-linear DAE hence does not reduce to a linear model after training. The non-linearity is important to improve the denoising MSE, see Fig. 1. We stress this finding: trained alone, the bottleneck network  $\hat{u}$  only learns to perform PCA; trained jointly with the rescaling component as part of the full DAE  $f_{b,w}(2)$ , it learns a richer, non-linear representation. The full DAE (2) thus offers a genuinely non-linear learning model and opens exciting research avenues for the theory of autoencoders, beyond linear (or effectively linear) cases. In the next paragraph, we explore further the interaction between the rescaling component and the bottleneck network.
A tradeoff between the rescaling and the bottleneck network Conjecture (3.3), alongside Corollary (3.4) and the discussion in Section 4 provide a firm theoretical basis for the well-known empirical intuition (discussed e.g. in [38]) that skip-connections allow to better propagate information
from the input to the output of the DAE, thereby contributing to preserving intrinsic characteristics of the input. This effect is clearly illustrated in Fig. 4, where the resulting denoised image of an MNIST 7 by  $\hat{r}$ ,  $\hat{f}$ ,  $\hat{u}$  and PCA are presented. While the bottleneck network  $\hat{u}$  perfectly eliminates the background noise and produces an image with a very good resolution, it essentially collapses the image to the cluster mean, and yields, like PCA, the average MNIST 7. As a consequence, the denoised image bears little resemblance with the original image – in particular, the horizontal bar of the 7 is lost in the process. Conversely, the rescaling  $\hat{r}$  preserves the nuances of the original image, but the result is still largely blurred and displays overall poor contrast. Finally, the complete DAE (2) manages to preserve the characteristic features of the original data, while enhancing the image resolution by slightly overlaying the average 7 thereupon.
The optimization of the DAE (4) is therefore described by a tradeoff between two competing effects – namely the preservation of the input nuances by the skip connection, and the enhancement of the resolution/noise removal by the bottleneck network. This allows us to discuss the curious nonmonotonicity of the cosine similarity  $\theta$  as a function of the noise level  $\Delta$ , see Fig. 1 (left). While it may at first seem curious that the DAE seemingly does not manage to learn the data structure better for low  $\Delta$  than for intermediate  $\Delta$  (where the cosine similarity  $\theta$  is observed to be higher), this is actually due to the afore-discussed tradeoff. Indeed, for small  $\Delta$ , the data is still substantially clean, and there is therefore no incentive to enhance the contrast by using the cluster means –which are consequently not learnt. This phase is thus characterized by a large skip connection strength  $\hat{b}$ , and small cosine similarity  $\theta$  and weight norm  $\| \hat{\boldsymbol{w}}\| _F$ . Conversely, at high noise levels  $\Delta$ , the nuances of the data are already lost because of the noise. Hence the DAE does not rely on the skip connection component (whence the small values of  $\hat{b}$ ), and the only way to produce reasonably denoised data is to collapse to the cluster mean using the network component (whence a large  $\| \hat{\boldsymbol{w}}\| _F$ ).
We consider the problem of denoising a high-dimensional Gaussian mixture, by training a DAE via empirical risk minimization, in the limit where the number of training samples and the dimension are proportionally large. We provide a sharp asymptotic characterization of a number of summary statistics of the trained DAE weight, average MSE, and cosine similarity with the cluster means. These results contain as a corollary the case of RAEs. Building on these findings, we isolate the role of the skip connection and the bottleneck network in the DAE architecture and characterize the tradeoff between those two components in terms of preservation of the data nuances and noise removal - thereby providing some theoretical insight into a longstanding practical intuition in machine learning.
We believe the present work also opens exciting research avenues. First, our real data experiments hint at the presence of Gaussian universality. While this topic has gathered considerable attention in recent years, only classification/regression supervised learning tasks have been hitherto addressed. Which aspects of universality carry over to denoising tasks, and how they differ from the current understanding of supervised regression/classification, is an important question. Second, the DAE with skip connection (2) provides an autoencoder model which does not just simply learn the principal components of the training set. It, therefore, affords a genuinely non-linear network model where richer learning settings can be investigated.
We thank Eric Vanden-Eijnden for his wonderful lecture at the les Houches Summer School of July 2022, which inspired parts of this work. We thank Maria Refinetti and Sebastian Goldt for insightful discussions at the very early stages of this project.
[1] Yang Song, Jascha Narain Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. International Conference on Learning Representations, 2021.
[2] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[41] Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Deeply-recursive convolutional network for image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1637–1645, 2016.
[42] Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association, 106:1602 - 1614, 2011.
[43] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.
[44] Sebastian Goldt, Bruno Loureiro, Galen Reeves, Florent Krzakala, Marc Mézard, and Lenka Zdeborová. The gaussian equivalence of generative models for learning with shallow neural networks. In Mathematical and Scientific Machine Learning, 2020.
[45] Hong Hu and Yue M. Lu. Universality laws for high-dimensional learning with random features. IEEE Transactions on Information Theory, 69:1932-1964, 2020.
[46] Andrea Montanari and Basil N Saeed. Universality of empirical risk minimization. In Conference on Learning Theory, pages 4310-4312. PMLR, 2022.
[47] Bruno Loureiro, Cédric Gerbelot, Hugo Cui, Sebastian Goldt, Florent Krzakala, Marc Mézard, and Lenka Zdeborová. Learning curves of generic features maps for realistic datasets with a teacher-student model. Journal of Statistical Mechanics: Theory and Experiment, 2022, 2021.
[48] Bruno Loureiro, Gabriele Sicuro, Cédric Gerbelot, Alessandro Pacco, Florent Krzakala, and Lenka Zdeborova. Learning gaussian mixtures with generalised linear models: Precise asymptotics in high-dimensions. In Neural Information Processing Systems, 2021.
A Derivation of Conjecture 3.3
In this section, we detail the derivation of Conjecture (3.3).
A.1 Derivation technique
In order to access sharp asymptotic characterizations for the metrics and summary statistics mse (5),  $\theta$  (6),  $\hat{b}$  and  $\| \hat{\boldsymbol{w}}\| /d$ , the first step is to observe that any test function  $\varphi (\hat{\boldsymbol{w}},\hat{\boldsymbol{b}})$  of the learnt network parameters  $\hat{\boldsymbol{w}},\hat{\boldsymbol{b}}$  can be written as an average over a limit probability distribution
$$
\mathbb {E} _ {\mathcal {D}} \left[ \varphi (\hat {\boldsymbol {w}}, \hat {b}) \right] = \lim  _ {\beta \rightarrow \infty} \mathbb {E} _ {\mathcal {D}} \frac {1}{Z} \int d w d b \varphi (\boldsymbol {w}, b) e ^ {- \beta \hat {\mathcal {R}} (\boldsymbol {w}, b)} \tag {16}
$$
where  $\hat{\mathcal{R}} (\cdot ,\cdot)$  is the empirical risk (4) and the corresponding probability density
$$
\mathbb {P} _ {\beta} (\boldsymbol {w}, b) = \frac {e ^ {- \beta \hat {\mathcal {R}} (\boldsymbol {w} , b)}}{Z} \tag {17}
$$
is known as the Boltzmann measure in statistical physics, with the parameter  $\beta$  typically referred to as the inverse temperature. The normalization  $Z$  is called the partition function of  $\mathbb{P}_{\beta}$ :
$$
Z = \int d \boldsymbol {w} d b e ^ {- \beta \hat {\mathcal {R}} (\boldsymbol {w}, b)}. \tag {18}
$$
The idea of the replica method [21, 22, 23, 24], building on the expression (16), is to compute the moment generating function (also known as the free entropy) as
$$
\mathbb {E} _ {\mathcal {D}} \ln Z = \lim  _ {s \rightarrow 0} \frac {\ln \left[ \mathbb {E} _ {\mathcal {D}} Z ^ {s} \right]}{s}. \tag {19}
$$
With the moment generating function, it then becomes possible to compute averages like (16). The backbone of the derivation, which we detail in the following subsections, therefore lies in the computation of the replicated partition function  $\mathbb{E}_{\mathcal{D}}Z^{s}$ .
A.2 Replicated partition function
For subsequent clarity, we shall introduce the variable
$$
\boldsymbol {\eta} ^ {\mu} \equiv \boldsymbol {x} ^ {\mu} - \boldsymbol {\mu} _ {k} \tag {20}
$$