SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hugo Cui
Statistical Physics of Computation Lab
hugo.cui@epfl.ch
Lenka Zdeborová
We address the problem of denoising data from a Gaussian mixture using a two-layer non-linear autoencoder with tied weights and a skip connection. We consider the high-dimensional limit where the number of training samples and the input dimension jointly tend to infinity while the number of hidden units remains bounded. We provide closed-form expressions for the denoising mean-squared test error. Building on this result, we quantitatively characterize the advantage of the considered architecture over the autoencoder without the skip connection that relates closely to principal component analysis. We further show that our results accurately capture the learning curves on a range of real data sets.
Machine learning techniques have a long history of success in denoising tasks. The recent breakthrough of diffusion-based generation [1, 2] has further revived the interest in denoising networks, demonstrating how they can also be leveraged, beyond denoising, for generative tasks. However, this rapidly expanding range of applications stands in sharp contrast to the relatively scarce theoretical understanding of denoising neural networks, even for the simplest instance thereof – namely Denoising Auto Encoders (DAEs) [3].
Theoretical studies of autoencoders have hitherto almost exclusively focused on data compression tasks using Reconstruction Auto Encoders (RAEs), where the goal is to learn a concise latent representation of the data. A majority of this body of work addresses linear autoencoders [4, 5, 6, 7]. The authors of [8, 9] analyze the gradient-based training of non-linear autoencoders with online stochastic gradient descent or in population, thus implicitly assuming the availability of an infinite number of training samples. Furthermore, two-layer RAEs were shown to learn to essentially perform Principal Component Analysis (PCA) [10, 11, 12], i.e. to learn a linear model. Ref. [13] shows that this is also true for infinite-width architectures. Learning in DAEs has been the object of theoretical investigations only in the linear case [14], while the case of non-linear DAEs remains theoretically largely unexplored.
Main contributions- The present work considers the problem of denoising data sampled from a Gaussian mixture by learning a two-layer DAE with a skip connection and tied weights via empirical risk minimization. Throughout the manuscript, we consider the high-dimensional limit where the number of training samples  $n$  and the dimension  $d$  are large ( $n, d \to \infty$ ) while remaining comparable, i.e.  $\alpha \equiv \frac{n}{d} = \Theta(1)$ . Our main contributions are:
- Leveraging the replica method, we provide sharp, closed-form formulae for the mean squared denoising test error (MSE) for DAEs, as a function of the sample complexity  $\alpha$  and
the problem parameters. We also provide a sharp characterization for other learning metrics including the weights norms, skip connection strength, and cosine similarity between the weights and the cluster means. These formulae encompass as a corollary the case of RAEs. We show that these formulae also describe quantitatively rather well the denoising MSE for real data sets, including MNIST [15] and FashionMNIST [16].
- We find that PCA denoising (namely denoising by projecting the noisy data along the principal component of the training samples) is widely sub-optimal compared to the DAE, leading to a MSE superior by a difference of  $\Theta(d)$ , thereby establishing that DAEs do not simply learn to perform PCA.
- Building on the formulae, we quantify the role of each component of the DAE architecture (skip connection and the bottleneck network) in its overall performance. We find that the two components have complementary effects in the denoising process –namely preserving the data nuances and removing the noise– and discuss how the training of the DAE results from a tradeoff between these effects.
The code used in the present manuscript can be found in the following repository.
Related works Theory of autoencoders- Various aspects of RAEs have been studied, for example, memorization [17], or latent space alignment [18]. However, the largest body of work has been dedicated to the analysis of gradient-based algorithms when training RAEs. Ref. [5] established that minimizing the training loss leads to learning the principal components of the data. Authors of [11, 12] have analyzed how a linear RAE learns these components during training. These studies were later extended to non-linear networks by [19, 8, 9], at the sacrifice of further assuming an infinite number of training samples to be available -either by considering online stochastic gradient descent, or the population loss. Refs. [20, 13] are able to address a finite sample complexity, but in exchange, have to consider infinite-width architectures, which [13] further shows, also tend to a large extent to learn to perform PCA.
Exact asymptotics from the replica method- The replica method [21, 22, 23, 24] has proven a very valuable gateway to access sharp asymptotic characterizations of learning metrics for high-dimensional machine learning problems. Past works have addressed -among others- single-[25, 26, 27, 28] and multi-index models [29], or kernel methods [30, 31, 32, 33]. While the approach has traditionally addressed convex problems, for which its prediction can be proven e.g. using the convex Gordon minimax theorem [34], the replica method allows to average over all the global minimizers of the loss, and therefore also accommodates non-convex settings. Refs. [35, 36] are two recent examples of its application to non-convex losses. In the present manuscript, we leverage this versatility to study the minimization of the empirical risk of DAEs, whose non-convexity represents a considerable hurdle to many other types of analyses.
Data model We consider the problem of denoising data  $\pmb{x} \in \mathbb{R}^d$  corrupted by Gaussian white noise of variance  $\Delta$ ,
$$
\tilde {\boldsymbol {x}} = \sqrt {1 - \Delta} \boldsymbol {x} + \sqrt {\Delta} \boldsymbol {\xi},
$$
where we denoted  $\tilde{\pmb{x}}$  the noisy data point, and  $\xi \sim \mathcal{N}(0,\mathbb{I}_d)$  the additive noise. The rescaling of the clean data point by a factor  $\sqrt{1 - \Delta}$  is a practical choice that entails no loss of generality, and allows to easily interpolate between the noiseless case ( $\Delta = 0$ ) and the case where the signal-to-noise ratio vanishes ( $\Delta = 1$ ). Furthermore, it allows us to seamlessly connect with works on diffusion-based generative models, where the rescaling naturally follows from the way the data is corrupted by an Ornstein-Uhlenbeck process [1, 2]. In the present work, we assume the clean data  $\pmb{x}$  to be drawn from a Gaussian mixture distribution  $\mathbb{P}$  with  $K$  clusters
$$
\boldsymbol {x} \sim \sum_ {k = 1} ^ {K} \rho_ {k} \mathcal {N} \left(\boldsymbol {\mu} _ {k}, \boldsymbol {\Sigma} _ {k}\right). \tag {1}
$$
The  $k$ -th cluster is thus centered around  $\pmb{\mu}_k \in \mathbb{R}^d$ , has covariance  $\Sigma_k \succeq 0$ , and relative weight  $\rho_k$ .
DAE model An algorithmic way to retrieve the clean data  $\pmb{x}$  from the noisy data  $\tilde{\pmb{x}}$  is to build a neural network taking the latter as an input and yielding the former as an output. A particularly natural choice for such a network is an autoencoder architecture [3]. The intuition is that the narrow hidden layer of an autoencoder forces the network to learn a succinct latent representation of the data, which is robust against noise corruption of the input. In this work, we analyze a two-layer DAE. We further assume that the weights are tied. Additionally, mirroring modern denoising architectures like U-nets [37] or [38, 39, 40, 41], we also allow for a (trainable) skip-connection:
$$
f _ {b, \boldsymbol {w}} (\tilde {\boldsymbol {x}}) = b \times \tilde {\boldsymbol {x}} + \frac {\boldsymbol {w} ^ {\top}}{\sqrt {d}} \sigma \left(\frac {\boldsymbol {w} \tilde {\boldsymbol {x}}}{\sqrt {d}}\right). \tag {2}
$$
The DAE (2) is therefore parametrized by the scalar skip connection strength  $b \in \mathbb{R}$  and the weights  $\boldsymbol{w} \in \mathbb{R}^{p \times d}$ , with  $p$  the width of the DAE hidden layer. The normalization of the weight  $\boldsymbol{w}$  by  $\sqrt{d}$  in (2) is the natural choice which ensures for high dimensional settings  $d \gg 1$  that the argument  $w\tilde{x} / \sqrt{d}$  of the non-linearity  $\sigma(\cdot)$  stays  $\Theta(1)$ . Like [8], we focus on the case with  $p \ll d$ . The assumption of weight-tying affords a more concise theoretical characterization and thus clearer discussions. Note that it is also a strategy with substantial practical history, dating back to [3], as it prevents the DAE from functioning in the linear region of its non-linearity  $\sigma(\cdot)$ . This choice of architecture is also motivated by a particular case of Tweedie's formula [42] (see eq. (79) in Appendix B), which will be the object of further discussion in Section 4.
We also consider two other simple architectures
$$
u _ {\boldsymbol {v}} (\tilde {\boldsymbol {x}}) = \frac {\boldsymbol {v} ^ {\top}}{\sqrt {d}} \sigma \left(\frac {\boldsymbol {v} \tilde {\boldsymbol {x}}}{\sqrt {d}}\right), \quad r _ {c} (\tilde {\boldsymbol {x}}) = c \times \tilde {\boldsymbol {x}}, \tag {3}
$$
which correspond to the building blocks of the complete DAE architecture  $f_{b,\boldsymbol{w}}$  (2) (hereafter referred to as the full DAE). Note that indeed  $f_{b,\boldsymbol{w}} = r_b + u_{\boldsymbol{w}}$ . The part  $u_{\boldsymbol{v}}(\cdot)$  is a DAE without skip connection (hereafter called the bottleneck network component), while  $r_c(\cdot)$  correspond to a simple single-parameter trainable rescaling of the input (hereafter called the rescaling component).
To train the DAE (2), we assume the availability of a training set  $\mathcal{D} = \{\tilde{x}^{\mu}, x^{\mu}\}_{\mu=1}^{n}$ , with  $n$  clean samples  $x^{\mu}$  drawn i.i.d from  $\mathbb{P}(1)$  and the corresponding noisy samples  $\tilde{x}^{\mu} = x^{\mu} + \xi^{\mu}$  (with the noises  $\xi^{\mu}$  assumed mutually independent). The DAE is trained to recover the clean samples  $x^{\mu}$  from the noisy samples  $\tilde{x}^{\mu}$  by minimizing the empirical risk
$$
\hat {\mathcal {R}} (b, \boldsymbol {w}) = \sum_ {\mu = 1} ^ {n} \| \boldsymbol {x} ^ {\mu} - f _ {b, \boldsymbol {w}} (\tilde {\boldsymbol {x}} ^ {\mu}) \| ^ {2} + g (\boldsymbol {w}), \tag {4}
$$
where  $g: \mathbb{R}^{p \times d} \to \mathbb{R}_+$  is an arbitrary convex regularizing function. We denote by  $\hat{b}, \hat{w}$  the minimizers of the empirical risk (4) and by  $\hat{f} \equiv f_{\hat{b},\hat{w}}$  the corresponding trained DAE (2). For future discussion, we also consider training independently the components (3) via empirical risk minimization, by which we mean replacing  $f_{b,w}$  by  $u_{\boldsymbol{v}}$  or  $r_c$  in (4). We similarly denote  $\hat{\boldsymbol{v}}$  (resp.  $\hat{c}$ ) the learnt weight of the bottleneck network (resp. rescaling) component and  $\hat{u} \equiv u_{\hat{v}}$  (resp.  $\hat{r} \equiv r_{\hat{c}}$ ). Note that generically,  $\hat{v} \neq \hat{w}$  and  $\hat{c} \neq \hat{b}$ , and therefore  $\hat{f} \neq \hat{u} + \hat{r}$ , since  $\hat{b}, \hat{w}$  result from their joint optimization as parts of the full DAE  $f_{b,w}$ , while  $\hat{c}$  (or  $\hat{v}$ ) are optimized independently. As we discuss in Section 4, training the sole rescaling  $r_c$  does not afford an expressive enough denoiser, while an independently learnt bottleneck network component  $u_{\boldsymbol{v}}$  essentially only learns to implement PCA. However, when jointly trained as components of the full DAE  $f_{b,w}$  (2), the resulting denoiser  $\hat{f}$  is a genuinely non-linear model which yields a much lower test error than PCA, and learns to leverage flexibly its two components to balance the preservation of the data nuances and the removal of the noise.
Learning metrics The performance of the DAE (2) trained with the loss (4) is quantified by its reconstruction (denoising) test MSE, defined as
$$
\operatorname {m s e} _ {\hat {f}} \equiv \mathbb {E} _ {\mathcal {D}} \mathbb {E} _ {\boldsymbol {x} \sim \mathbb {P}} \mathbb {E} _ {\boldsymbol {\xi} \sim \mathcal {N} (0, \mathbb {I} _ {d})} \left\| \boldsymbol {x} - f _ {\hat {b}, \hat {\boldsymbol {w}}} \left(\sqrt {1 - \Delta} \boldsymbol {x} + \sqrt {\Delta} \boldsymbol {\xi}\right) \right\| ^ {2}. \tag {5}
$$
The expectations run over a fresh test sample  $\pmb{x}$  sampled from the Gaussian mixture  $\mathbb{P}$  (1), and a new additive noise  $\pmb{\xi}$  corrupting it. Note that an expectation over the train set  $\mathcal{D}$  is also included to make  $\mathrm{mse}_{\hat{f}}$  a metric that does not depend on the particular realization of the train set. The denoising
test MSEs  $\mathrm{mse}_{\hat{u}}$ ,  $\mathrm{mse}_{\hat{r}}$  are defined similarly as the denoising test errors of the independently learnt components (3). Aside from the denoising MSE (5), another question of interest is how much the DAE manages to learn the structure of the data distribution, as described by the cluster means  $\pmb{\mu}_k$ . This is measured by the cosine similarity matrix  $\pmb{\theta} \in \mathbb{R}^{p \times K}$ , where for  $i \in [1, p]$  and  $k \in [1, K]$ ,
$$
\theta_ {i k} \equiv \mathbb {E} _ {\mathcal {D}} \left[ \frac {\hat {\boldsymbol {w}} _ {i} ^ {\top} \boldsymbol {\mu} _ {k}}{\| \hat {\boldsymbol {w}} _ {i} \| \| \boldsymbol {\mu} _ {k} \|} \right]. \tag {6}
$$
In other words,  $\theta_{ik}$  measures the alignment of the  $i$ -th row  $\hat{\pmb{w}}_i$  of the trained weight matrix  $\hat{\pmb{w}}$  with the mean of the  $k$ -th cluster  $\pmb{\mu}_k$ .
High-dimensional limit We analyze the optimization problem (4) in the high-dimensional limit where the input dimension  $d$  and number of training samples  $n$  jointly tend to infinity, while their ratio  $\alpha = n / d$  -hitherto referred to as the sample complexity-stays  $\Theta (1)$ . The hidden layer width  $p$ , the noise level  $\Delta$ , the number of clusters  $K$  and the norm of the cluster means  $\| \pmb{\mu}_k\|$  are also assumed to remain  $\Theta (1)$ . This corresponds to a rich limit, where the number of parameters of the DAE is not large compared to the number of samples like in [20, 13], and therefore cannot trivially fit the train set, or simply memorize it [17]. Conversely, the number of samples  $n$  is not infinite like in [8, 9, 19], and therefore importantly allows to study the effect of a finite train set on the representation learnt by the DAE.
3 Asymptotic formulae for DAEs
We now state the main result of the present work, namely the closed-form asymptotic formulae for the learning metrics  $\mathrm{mse}_{\hat{f}}(5)$  and  $\theta$  (6) for a DAE (2) learnt with the empirical loss (4). These characterizations are obtained by first recasting the optimization problem into an analysis of an associated probability measure, and then carrying out this analysis using the heuristic replica method, which we here employ in its replica-symmetric formulation (see Appendix A).
Assumption 3.1. The covariances  $\{\pmb{\Sigma}\}_{k=1}^{K}$  admit a common set of eigenvectors  $\{e_i\}_{i=1}^d$ . We further note  $\{\lambda_i^k\}_{i=1}^d$  the eigenvalues of  $\pmb{\Sigma}_k$ . The eigenvalues  $\{\lambda_i^k\}_{i=1}^d$  and the projection of the cluster means on the eigenvectors  $\{e_i^\top \pmb{\mu}_k\}_{i,k}$  are assumed to admit a well-defined joint distribution  $\nu$  as  $d \to \infty$  - namely, for  $\gamma = (\gamma_1, \dots, \gamma_K) \in \mathbb{R}^K$  and  $\tau = (\tau_1, \dots, \tau_K) \in \mathbb{R}^K$ :
$$
\frac {1}{d} \sum_ {i = 1} ^ {d} \prod_ {k = 1} ^ {K} \delta \left(\lambda_ {i} ^ {k} - \gamma_ {k}\right) \delta \left(\sqrt {d} \boldsymbol {e} _ {i} ^ {\top} \boldsymbol {\mu} _ {k} - \tau_ {k}\right) \xrightarrow {d \rightarrow \infty} \nu (\gamma , \tau). \tag {7}
$$
Moreover, the marginals  $\nu_{\gamma}$  (resp.  $\nu_{\tau}$ ) are assumed to have a well-defined first (resp. second) moment.
Assumption 3.2.  $g(\cdot)$  is a  $\ell_2$  regularizer with strength  $\lambda$ , i.e.  $g(\cdot) = \lambda / 2 \|\cdot\|_F^2$ .
We are now in a position to discuss the main result of this manuscript, which we state under Assumptions 3.1 and 3.2 for definiteness and clarity. These assumptions can actually be relaxed, as we further discuss after the result statement.
Conjecture 3.3. (Closed-form asymptotics for DAEs trained with empirical risk minimization) Under Assumptions 3.1 and 3.2, in the high-dimensional limit  $n, d \to \infty$  with fixed ratio  $\alpha$ , the denoising test MSE  $\mathrm{mse}_{\hat{f}}$  (5) admits the expression
$$
\begin{array}{l} \mathrm {m s e} _ {\hat {f}} - \mathrm {m s e} _ {\circ} = \sum_ {k = 1} ^ {K} \rho_ {k} \mathbb {E} _ {z} \operatorname {T r} \left[ q \sigma \left(\sqrt {1 - \Delta} m _ {k} + \sqrt {\Delta q + (1 - \Delta) q _ {k}} z\right) ^ {\otimes 2} \right] \tag {8} \\ - 2 \sum_ {k = 1} ^ {K} \rho_ {k} \mathbb {E} _ {u, v} \left[ \sigma \Big (\sqrt {1 - \Delta} m _ {k} + \sqrt {q _ {k} (1 - \Delta)} u + \sqrt {\Delta q} v \Big) ^ {\top} \big ((1 - \hat {b} \sqrt {1 - \Delta}) (m _ {k} + \sqrt {q _ {k}} u) - \hat {b} \sqrt {\Delta q} v \big) \right] + o (1), \\ \end{array}
$$
where the averages bear over independent Gaussian variables  $z, u, v \sim \mathcal{N}(0, \mathbb{I}_p)$ . We denoted
$$
\mathrm {m s e} _ {\circ} = d \Delta \hat {b} ^ {2} + \left(1 - \sqrt {1 - \Delta \hat {b}}\right) ^ {2} \left[ \sum_ {k = 1} ^ {K} \rho_ {k} \left(\int d \nu_ {\tau} (\tau) \tau_ {k} ^ {2} + d \int d \nu_ {\gamma} (\gamma) \gamma_ {k}\right) \right]. \tag {9}
$$
The learnt skip connection strength  $\hat{b}$  is
$$
\hat {b} = \frac {\left(\sum_ {k = 1} ^ {K} \rho_ {k} \int d \nu_ {\gamma} (\gamma) \gamma_ {k}\right) \sqrt {1 - \Delta}}{\left(\sum_ {k = 1} ^ {K} \rho_ {k} \int d \nu_ {\gamma} (\gamma) \gamma_ {k}\right) (1 - \Delta) + \Delta} + o (1). \tag {10}
$$
The cosine similarity  $\theta$  (6) admits the compact formula for  $i\in [[1,p]]$  and  $k\in [[1,K]]$
$$
\theta_ {i k} = \frac {\left(m _ {k}\right) _ {i}}{\sqrt {q _ {i i} \int d \nu_ {\tau} (\tau) \tau_ {k} ^ {2}}}, \tag {11}
$$
where we have introduced the summary statistics
$$
q = \lim  _ {d \rightarrow \infty} \mathbb {E} _ {\mathcal {D}} \left[ \frac {\hat {\boldsymbol {w}} \hat {\boldsymbol {w}} ^ {\top}}{d} \right], \quad q _ {k} = \lim  _ {d \rightarrow \infty} \mathbb {E} _ {\mathcal {D}} \left[ \frac {\hat {\boldsymbol {w}} \boldsymbol {\Sigma} _ {k} \hat {\boldsymbol {w}} ^ {\top}}{d} \right], \quad m _ {k} = \lim  _ {d \rightarrow \infty} \mathbb {E} _ {\mathcal {D}} \left[ \frac {\hat {\boldsymbol {w}} \boldsymbol {\mu} _ {k}}{\sqrt {d}} \right]. \tag {12}
$$
Thus  $q, q_k \in \mathbb{R}^{p \times p}$ ,  $m_k \in \mathbb{R}^p$ . The existence of these limits is an assumption of the replica method. The summary statistics  $q, q_k, m_k$  can be determined as solutions of the system of equations
$$
\begin{array}{l} \left\{ \begin{array}{l} \hat {q} _ {k} = \alpha \rho_ {k} \mathbb {E} _ {\xi , \eta} V _ {k} ^ {- 1} \left(\operatorname {p r o x} _ {y} ^ {k} - q _ {k} ^ {\frac {1}{2}} \eta - m _ {k}\right) ^ {\otimes 2} V _ {k} ^ {- 1} \\ \hat {V} _ {k} = - \alpha \rho_ {k} q _ {k} ^ {- \frac {1}{2}} \mathbb {E} _ {\xi , \eta} V _ {k} ^ {- 1} \left(\operatorname {p r o x} _ {y} ^ {k} - q _ {k} ^ {\frac {1}{2}} \eta - m _ {k}\right) \eta^ {\top} \\ \hat {m} _ {k} = \alpha \rho_ {k} \mathbb {E} _ {\xi , \eta} V _ {k} ^ {- 1} \left(\operatorname {p r o x} _ {y} ^ {k} - q _ {k} ^ {\frac {1}{2}} \eta - m _ {k}\right) \\ \hat {q} = \frac {\alpha}{\Delta} \sum_ {k = 1} ^ {K} \rho_ {k} \mathbb {E} _ {\xi , \eta} V ^ {- 1} \left(\operatorname {p r o x} _ {x} ^ {k} - \sqrt {\Delta} q ^ {\frac {1}{2}} \xi\right) ^ {\otimes 2} V ^ {- 1} \\ \hat {V} = - \alpha \sum_ {k = 1} ^ {K} \rho_ {k} \mathbb {E} _ {\xi , \eta} \left[ \frac {1}{\sqrt {\Delta}} q ^ {- \frac {1}{2}} V ^ {- 1} \left(\operatorname {p r o x} _ {x} ^ {k} - \sqrt {\Delta} q ^ {\frac {1}{2}} \xi\right) \xi^ {\top} - \sigma \left(\sqrt {1 - \Delta} \operatorname {p r o x} _ {y} ^ {k} + \operatorname {p r o x} _ {x} ^ {k}\right) ^ {\otimes 2} \right] \end{array} \right. \\ \left\{ \begin{array}{l} q _ {k} = \int d \nu (\gamma , \tau) \gamma_ {k} \left(\lambda \mathbb {I} _ {p} + \hat {V} + \sum_ {j = 1} ^ {K} \gamma_ {j} \hat {V} _ {j}\right) ^ {- 2} \left(\hat {q} + \sum_ {j = 1} ^ {K} \gamma_ {j} \hat {q} _ {j} + \sum_ {1 \leq j, l \leq K} \tau_ {j} \tau_ {l} \hat {m} _ {j} \hat {m} _ {l} ^ {\top}\right) \\ V _ {k} = \int d \nu (\gamma , \tau) \gamma_ {k} \left(\lambda \mathbb {I} _ {p} + \hat {V} + \sum_ {j = 1} ^ {K} \gamma_ {j} \hat {V} _ {j}\right) ^ {- 1} \\ m _ {k} = \int d \nu (\gamma , \tau) \tau_ {k} \left(\lambda \mathbb {I} _ {p} + \hat {V} + \sum_ {j = 1} ^ {K} \gamma_ {j} \hat {V} _ {j}\right) ^ {- 1} \sum_ {j = 1} ^ {K} \tau_ {j} \hat {m} _ {j} \\ q = \int d \nu (\gamma , \tau) \left(\lambda \mathbb {I} _ {p} + \hat {V} + \sum_ {j = 1} ^ {K} \gamma_ {j} \hat {V} _ {j}\right) ^ {- 2} \left(\hat {q} + \sum_ {j = 1} ^ {K} \gamma_ {j} \hat {q} _ {j} + \sum_ {1 \preceq j, l \preceq K} \tau_ {j} \tau_ {l} \hat {m} _ {j} \hat {m} _ {l} ^ {\top}\right) \\ V = \int d \nu (\gamma , \tau) \left(\lambda \mathbb {I} _ {p} + \hat {V} + \sum_ {j = 1} ^ {K} \gamma_ {j} \hat {V} _ {j}\right) ^ {- 1} \end{array} \right. \tag {13} \\ \end{array}
$$
In (13),  $\hat{q}_k, \hat{V}_k, \hat{V}, V \in \mathbb{R}^{p \times p}$  and  $\hat{m}_k \in \mathbb{R}^p$ , and the averages bear over finite-dimensional i.i.d Gaussians  $\xi, \eta \sim \mathcal{N}(0, \mathbb{I}_p)$ . Finally,  $\mathrm{prox}_x^k, \mathrm{prox}_y^k$  are given as the solutions of the finite-dimensional optimization
$$
\begin{array}{l} \operatorname {p r o x} _ {x} ^ {k}, \operatorname {p r o x} _ {y} ^ {k} = \underset {x, y \in \mathbb {R} ^ {p}} {\operatorname {a r g i n f}} \left\{\operatorname {T r} \left[ V _ {k} ^ {- 1} \left(y - q _ {k} ^ {\frac {1}{2}} \eta - m _ {k}\right) ^ {\otimes 2} \right] + \frac {1}{\Delta} \operatorname {T r} \left[ V ^ {- 1} \left(x - \sqrt {\Delta} q ^ {\frac {1}{2}} \xi\right) ^ {\otimes 2} \right] \right. \\ \left. + \operatorname {T r} \left[ q \sigma \left(\sqrt {1 - \Delta} y + x\right) ^ {\otimes 2} \right] - 2 \sigma \left(\sqrt {1 - \Delta} y + x\right) ^ {\top} \left((1 - \sqrt {1 - \Delta} \hat {b}) y - \hat {b} x\right) \right\}. \tag {14} \\ \end{array}
$$
Conjecture 3.3 provides a gateway to probe and characterize the asymptotic properties of the model (2) at the global optimum of the empirical risk (4), whereas a purely experimental study would not have been guaranteed to reach the global solution, and would suffer from finite-size effects. Equation
Figure 1:  $\alpha = 1, K = 2, \rho_{1,2} = 1/2, \Sigma_{1,2} = 0.09 \times \mathbb{I}_d, p = 1, \lambda = 0.1, \sigma(\cdot) = \tanh(\cdot)$ ; the cluster mean  $\pmb{\mu}_1 = -\pmb{\mu}_2$  was taken as a random Gaussian vector of norm 1. (left) In blue, the difference in MSE between the full DAE  $\hat{f}$  (2) and the rescaling component  $\hat{r}$  (3). Solid lines correspond to the sharp asymptotic characterization of Conjecture 3.3. Dots represent numerical simulations for  $d = 700$ , training the DAE using the Pytorch implementation of full-batch Adam, with learning rate  $\eta = 0.05$  over 2000 epochs, averaged over  $N = 10$  instances. Error bars represent one standard deviation. For completeness, the MSE of the oracle denoiser is given as a baseline in green, see Section 4. The performance of a linear DAE  $(\sigma(x) = x)$  is represented in dashed red. (right) Cosine similarity  $\theta$  (6) (green), squared weight norm  $\| \hat{w} \|_F^2 / d$  (red) and skip connection strength  $\hat{b}$  (blue). Solid lines correspond to the formulae (11)(12) and (10) of Conjecture 3.3; dots are numerical simulations. For completeness, the cosine similarity of the first principal component of the clean train data  $\{\pmb{x}^\mu\}_{\mu=1}^n$  is plotted in dashed black.