Title: High-dimensional Asymptotics of Denoising Autoencoders
Venue: NeurIPS 2023
Accepted: yes
Keywords: statistical physics

replica method

autoencoder

exact asymptotics
Abstract: We address the problem of denoising data from a Gaussian mixture using a two-layer non-linear autoencoder with tied weights and a skip connection. We consider the high-dimensional limit where the number of training samples and the input dimension jointly tend to infinity while the number of hidden units remains bounded. We provide closed-form expressions for the denoising mean-squared test error. Building on this result, we quantitatively characterize the advantage of the considered architecture over the autoencoder without the skip connection that relates closely to principal component analysis. We further show that our results capture accurately the learning curves on a range of real datasets.

High-dimensional Asymptotics of Denoising Autoencoders

Abstract

We address the problem of denoising data from a Gaussian mixture using a twolayer non-linear autoencoder with tied weights and a skip connection. We consider the high-dimensional limit where the number of training samples and the input dimension jointly tend to infinity while the number of hidden units remains bounded. We provide closed-form expressions for the denoising mean-squared test error. Building on this result, we quantitatively characterize the advantage of the considered architecture over the autoencoder without the skip connection that relates closely to principal component analysis. We further show that our results accurately capture the learning curves on a range of real data sets.

1 Introduction

Machine learning techniques have a long history of success in denoising tasks. The recent breakthrough of diffusion-based generation [1, 2] has further revived the interest in denoising networks, demonstrating how they can also be leveraged, beyond denoising, for generative tasks. However, this rapidly expanding range of applications stands in sharp contrast to the relatively scarce theoretical understanding of denoising neural networks, even for the simplest instance thereof - namely Denoising Auto Encoders (DAEs) [3].

Theoretical studies of autoencoders have hitherto almost exclusively focused on data compression tasks using Reconstruction Auto Encoders (RAEs), where the goal is to learn a concise latent representation of the data. A majority of this body of work addresses linear autoencoders [4, 5, 6, 7]. The authors of [8, 9] analyze the gradient-based training of non-linear autoencoders with online stochastic gradient descent or in population, thus implicitly assuming the availability of an infinite number of training samples. Furthermore, two-layer RAEs were shown to learn to essentially perform Principal Component Analysis (PCA) [10, 11, 12], i.e. to learn a linear model. Ref. [13] shows that this is also true for infinite-width architectures. Learning in DAEs has been the object of theoretical investigations only in the linear case [14], while the case of non-linear DAEs remains theoretically largely unexplored.

Main contributionsThe present work considers the problem of denoising data sampled from a Gaussian mixture by learning a two-layer DAE with a skip connection and tied weights via empirical risk minimization. Throughout the manuscript, we consider the high-dimensional limit where the number of training samples n and the dimension d are large ( n, d →∞ ) while remaining comparable, i.e. α ≡ n / d = Θ(1) . Our main contributions are:

Leveraging the replica method, we provide sharp, closed-form formulae for the mean squared denoising test error (MSE) for DAEs, as a function of the sample complexity α and

the problem parameters. We also provide a sharp characterization for other learning metrics including the weights norms, skip connection strength, and cosine similarity between the weights and the cluster means. These formulae encompass as a corollary the case of RAEs. We show that these formulae also describe quantitatively rather well the denoising MSE for real data sets, including MNIST [15] and FashionMNIST [16].

We find that PCA denoising (namely denoising by projecting the noisy data along the principal component of the training samples) is widely sub-optimal compared to the DAE, leading to a MSE superior by a difference of Θ( d ) , thereby establishing that DAEs do not simply learn to perform PCA.

Building on the formulae, we quantify the role of each component of the DAE architecture (skip connection and the bottleneck network) in its overall performance. We find that the two components have complementary effects in the denoising process -namely preserving the data nuances and removing the noise- and discuss how the training of the DAE results from a tradeoff between these effects.

The code used in the present manuscript can be found in the following repository.

Related works Theory of autoencoders - Various aspects of RAEs have been studied, for example, memorization [17], or latent space alignment [18]. However, the largest body of work has been dedicated to the analysis of gradient-based algorithms when training RAEs. Ref. [5] established that minimizing the training loss leads to learning the principal components of the data. Authors of [11, 12] have analyzed how a linear RAE learns these components during training. These studies were later extended to non-linear networks by [19, 8, 9], at the sacrifice of further assuming an infinite number of training samples to be available -either by considering online stochastic gradient descent, or the population loss. Refs. [20, 13] are able to address a finite sample complexity, but in exchange, have to consider infinite-width architectures, which [13] further shows, also tend to a large extent to learn to perform PCA.

Exact asymptotics from the replica method - The replica method [21, 22, 23, 24] has proven a very valuable gateway to access sharp asymptotic characterizations of learning metrics for highdimensional machine learning problems. Past works have addressed -among others- single-[25, 26, 27, 28] and multi-index models [29], or kernel methods [30, 31, 32, 33]. While the approach has traditionally addressed convex problems, for which its prediction can be proven e.g. using the convex Gordon minimax theorem [34], the replica method allows to average over all the global minimizers of the loss, and therefore also accommodates non-convex settings. Refs. [35, 36] are two recent examples of its application to non-convex losses. In the present manuscript, we leverage this versatility to study the minimization of the empirical risk of DAEs, whose non-convexity represents a considerable hurdle to many other types of analyses.

2 Setting

Data model We consider the problem of denoising data x ∈ R d corrupted by Gaussian white noise of variance ∆ ,

where we denoted ˜ x the noisy data point, and ξ ∼ N (0 , I d ) the additive noise. The rescaling of the clean data point by a factor √ 1 -∆ is a practical choice that entails no loss of generality, and allows to easily interpolate between the noiseless case ( ∆ = 0 ) and the case where the signal-to-noise ratio vanishes ( ∆ = 1 ). Furthermore, it allows us to seamlessly connect with works on diffusion-based generative models, where the rescaling naturally follows from the way the data is corrupted by an Ornstein-Uhlenbeck process [1, 2]. In the present work, we assume the clean data x to be drawn from a Gaussian mixture distribution P with K clusters

The k -th cluster is thus centered around µ k ∈ R d , has covariance Σ k ⪰ 0 , and relative weight ρ k .

DAE model An algorithmic way to retrieve the clean data x from the noisy data ˜ x is to build a neural network taking the latter as an input and yielding the former as an output. A particularly natural choice for such a network is an autoencoder architecture [3]. The intuition is that the narrow hidden layer of an autoencoder forces the network to learn a succinct latent representation of the data, which is robust against noise corruption of the input. In this work, we analyze a two-layer DAE. We further assume that the weights are tied. Additionally, mirroring modern denoising architectures like U-nets [37] or [38, 39, 40, 41], we also allow for a (trainable) skip-connection:

The DAE (2) is therefore parametrized by the scalar skip connection strength b ∈ R and the weights w ∈ R p × d , with p the width of the DAE hidden layer. The normalization of the weight w by √ d in (2) is the natural choice which ensures for high dimensional settings d ≫ 1 that the argument w ˜ x / √ d of the non-linearity σ ( · ) stays Θ(1) . Like [8], we focus on the case with p ≪ d . The assumption of weight-tying affords a more concise theoretical characterization and thus clearer discussions. Note that it is also a strategy with substantial practical history, dating back to [3], as it prevents the DAE from functioning in the linear region of its non-linearity σ ( · ) . This choice of architecture is also motivated by a particular case of Tweedie's formula [42] (see eq. (79) in Appendix B), which will be the object of further discussion in Section 4.

We also consider two other simple architectures

which correspond to the building blocks of the complete DAE architecture f b, w (2) (hereafter referred to as the full DAE). Note that indeed f b, w = r b + u w . The part u v ( · ) is a DAE without skip connection (hereafter called the bottleneck network component), while r c ( · ) correspond to a simple single-parameter trainable rescaling of the input (hereafter called the rescaling component).

To train the DAE (2), we assume the availability of a training set D = { ˜ x µ , x µ } n µ =1 , with n clean samples x µ drawn i.i.d from P (1) and the corresponding noisy samples ˜ x µ = x µ + ξ µ (with the noises ξ µ assumed mutually independent). The DAE is trained to recover the clean samples x µ from the noisy samples ˜ x µ by minimizing the empirical risk

̸

where g : R p × d → R + is an arbitrary convex regularizing function. We denote by ˆ b, ˆ w the minimizers of the empirical risk (4) and by ˆ f ≡ f ˆ b, ˆ w the corresponding trained DAE (2). For future discussion, we also consider training independently the components (3) via empirical risk minimization, by which we mean replacing f b, w by u v or r c in (4). We similarly denote ˆ v (resp. ˆ c ) the learnt weight of the bottleneck network (resp. rescaling) component and ˆ u ≡ u ˆ v (resp. ˆ r ≡ r ˆ c ). Note that generically, ˆ v = ˆ w and ˆ c = ˆ b , and therefore ˆ f = ˆ u + ˆ r , since ˆ b, ˆ w result from their joint optimization as parts of the full DAE f b, w , while ˆ c (or ˆ v ) are optimized independently. As we discuss in Section 4, training the sole rescaling r c does not afford an expressive enough denoiser, while an independently learnt bottleneck network component u v essentially only learns to implement PCA. However, when jointly trained as components of the full DAE f b, w (2), the resulting denoiser ˆ f is a genuinely non-linear model which yields a much lower test error than PCA, and learns to leverage flexibly its two components to balance the preservation of the data nuances and the removal of the noise.

̸

̸

Learning metrics The performance of the DAE (2) trained with the loss (4) is quantified by its reconstruction (denoising) test MSE, defined as

The expectations run over a fresh test sample x sampled from the Gaussian mixture P (1), and a new additive noise ξ corrupting it. Note that an expectation over the train set D is also included to make mse ˆ f a metric that does not depend on the particular realization of the train set. The denoising

test MSEs mse ˆ u , mse ˆ r are defined similarly as the denoising test errors of the independently learnt components (3). Aside from the denoising MSE (5), another question of interest is how much the DAE manages to learn the structure of the data distribution, as described by the cluster means µ k . This is measured by the cosine similarity matrix θ ∈ R p × K , where for i ∈ J 1 , p K and k ∈ J 1 , K K ,

In other words, θ ik measures the alignment of the i -th row ˆ w i of the trained weight matrix ˆ w with the mean of the k -th cluster µ k .

High-dimensional limit We analyze the optimization problem (4) in the high-dimensional limit where the input dimension d and number of training samples n jointly tend to infinity, while their ratio α = n / d -hitherto referred to as the sample complexity -stays Θ(1) . The hidden layer width p , the noise level ∆ , the number of clusters K and the norm of the cluster means ∥ µ k ∥ are also assumed to remain Θ(1) . This corresponds to a rich limit, where the number of parameters of the DAE is not large compared to the number of samples like in [20, 13], and therefore cannot trivially fit the train set, or simply memorize it [17]. Conversely, the number of samples n is not infinite like in [8, 9, 19], and therefore importantly allows to study the effect of a finite train set on the representation learnt by the DAE.

3 Asymptotic formulae for DAEs

We now state the main result of the present work, namely the closed-form asymptotic formulae for the learning metrics mse ˆ f (5) and θ (6) for a DAE (2) learnt with the empirical loss (4). These characterizations are obtained by first recasting the optimization problem into an analysis of an associated probability measure, and then carrying out this analysis using the heuristic replica method, which we here employ in its replica-symmetric formulation (see Appendix A).

Assumption 3.1. The covariances { Σ } K k =1 admit a common set of eigenvectors { e i } d i =1 . We further note { λ k i } d i =1 the eigenvalues of Σ k . The eigenvalues { λ k i } d i =1 and the projection of the cluster means on the eigenvectors { e ⊤ i µ k } i,k are assumed to admit a well-defined joint distribution ν as d →∞ - namely, for γ = ( γ 1 , ..., γ K ) ∈ R K and τ = ( τ 1 , ..., τ K ) ∈ R K :

Moreover, the marginals ν γ (resp. ν τ ) are assumed to have a well-defined first (resp. second) moment.

Assumption 3.2. g ( · ) is a ℓ 2 regularizer with strength λ , i.e. g ( · ) = λ / 2 ∥·∥ 2 F .

We are now in a position to discuss the main result of this manuscript, which we state under Assumptions 3.1 and 3.2 for definiteness and clarity. These assumptions can actually be relaxed, as we further discuss after the result statement.

Conjecture 3.3. ( Closed-form asymptotics for DAEs trained with empirical risk minimization ) Under Assumptions 3.1 and 3.2, in the high-dimensional limit n, d → ∞ with fixed ratio α , the denoising test MSE mse ˆ f (5) admits the expression

where the averages bear over independent Gaussian variables z, u, v ∼ N (0 , I p ) . We denoted

The learnt skip connection strength ˆ b is

The cosine similarity θ (6) admits the compact formula for i ∈ J 1 , p K and k ∈ J 1 , K K

where we have introduced the summary statistics

Thus q, q k ∈ R p × p , m k ∈ R p . The existence of these limits is an assumption of the replica method. The summary statistics q, q k , m k can be determined as solutions of the system of equations

In (13) , ˆ q k , ˆ V k , ˆ V , V ∈ R p × p and ˆ m k ∈ R p , and the averages bear over finite-dimensional i.i.d Gaussians ξ, η ∼ N (0 , I p ) . Finally, prox k x , prox k y are given as the solutions of the finite-dimensional optimization

Conjecture 3.3 provides a gateway to probe and characterize the asymptotic properties of the model (2) at the global optimum of the empirical risk (4), whereas a purely experimental study would not have been guaranteed to reach the global solution, and would suffer from finite-size effects. Equation

0.0

0.2

0.4

0.6

0.8

1.0

0.4

0.2

0.0

0.2

0.4

mse

f

mse

r

mse

mse

r

mse

f

mse

r

(linear)

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

1.2

||

w

||

2

/

d

b

PCA

(13) provides a closed-form expression for the summary statistics (12), in terms of the solutions of the low-dimensional optimization (14). The latter can be loosely viewed as an effective loss, subsuming the averaged effect of the finite training set. Further remark that while the regularization λ does not explicitly appear in the expression for the MSE (8), the statistics q k , m k in (8) depend thereon through (13). In fact, Assumptions 3.1 and 3.2 are not strictly necessary, and can be simultaneously relaxed to address arbitrary convex regularizer g ( · ) and generically non-commuting { Σ k } K k =1 - but at the price of more intricate formulae. For this reason, we choose to discuss here Conjecture 3.3, and defer a discussion and detailed derivation of the generic asymptotically exact formulae to Appendix A, see eq. (58). Let us mention that a sharp asymptotic characterization of the train MSE can also be derived; for conciseness, we do not present it here and refer the interested reader to equation (68) in Appendix A. Conjecture 3.3 encompasses as special cases the asymptotic characterization of the components ˆ r, ˆ u (3):

Corollary 3.4. ( MSE of components ) The test MSE of ˆ r (3) is given by mse ˆ r = mse ◦ (9) . Furthermore, the learnt value of its single parameter ˆ c is given by (10) . The test MSE, cosine similarity and summary statistics of the bottleneck network ˆ u (3) follow from Conjecture 3.3 by setting ˆ b = 0 .

The implications of Corollary 3.4 shall be further discussed in Section 4, and a full derivation is provided in Appendix E. Finally, remark that in the noiseless limit ∆ = 0 , the denoising task reduces to a reconstruction task, with the autoencoder being tasked with reproducing the clean data as an output when taking the same clean sample as an input. Therefore Conjecture 3.3 also includes RAEs (by definition, without skip connection) as a special case.

Corollary 3.5. ( RAEs ) In the n, d →∞ limit, the MSE, cosine similarity and summary statistics for an RAE follow from Conjecture 3.3 by setting x = 0 in (14) , removing the first term in the brackets in the equation of ˆ V (13) and taking the limit ∆ , ˆ q, ˆ b → 0 .

Corollary 3.5 will be the object of further discussion in Section 4. A detailed derivation is presented in Appendix F. Note that Corollary 3.5 provides a characterization of RAEs as a function of the sample complexity α , where previous studies on non-linear RAEs rely on the assumption of an infinite number of available training samples [13, 8, 9].

Equations (10) and (12) of Conjecture 3.3 thus characterize the statistics of the learnt parameters ˆ b, ˆ w of the trained DAE (2). These summary statistics are, in turn, sufficient to fully characterize the

learning metrics (5) and (6) via equations (8) and (11). We thus have reduced the high-dimensional optimization (4) and the high-dimensional average over the train set D involved in the definition of the metrics (5) and (6) to a simpler system of equations over 4 + 6 K variables (13) which can be solved numerically. It is important to note that all the summary statistics involved in (13) are finite-dimensional as d →∞ , and therefore Conjecture 3.3 is a fully asymptotic characterization, in the sense that it does not involve any high-dimensional object. Finally, let us stress once more that the replica method employed in the derivation of these results should be viewed as a strong heuristic, but does not constitute a rigorous proof. While Conjecture 3.3 is stated in full generality, we focus for definiteness in the rest of the manuscript on the simple case r = 1 , K = 2 , which is found to already display all the interesting phenomenology discussed in this work, and leave the thorough exploration of r > 1 , K > 2 settings to future work. In the next paragraphs, we give two examples of applications of Conjecture 3.3, to a simple binary isotropic mixture, and to real data sets.

Example 1: Isotropic homoscedastic mixture We give as a first example the case of a synthetic binary Gaussian mixture with K = 2 , µ 1 = -µ 2 , Σ 1 , 2 = 0 . 09 × I d , ρ 1 , 2 = 1 / 2 , using a DAE with σ = tanh and p = 1 . Since this simple case exhibits the key phenomenology discussed in the present work, we refer to it in future discussions. The MSE mse ˆ f (8) evaluated from the solutions of the self-consistent equations (13) is plotted as the solid blue line in Fig. 1 (left) and compared to numerical simulations corresponding to training the DAE (2) with the Pytorch implementation of the Adam optimizer [43] (blue dots), for sample complexity α = 1 and ℓ 2 regularization (weight decay) λ = 0 . 1 . The agreement between the theory and simulation is compelling. The green solid line and corresponding green dots in Fig. 1 (right) correspond to the replica prediction (11) and simulations for the cosine similarity θ (6), and again display very good agreement. Note that for large noise levels, the DAE achieves a worse MSE than the rescaling -as shown by the positive value of mse ˆ f -mse ˆ r -, despite the former being a priori expressive enough to realize the latter. This in fact signals that the DAE overfits the training data. That such an overfitting is captured is a strength of our analysis, which allows to cover the effect of a limited sample complexity. Finally, this overfitting can be mitigated by increasing the weight decay λ , see Fig. 9 in Appendix A.

A particularly striking observation is that due to the non-convexity of the loss (4), there is a priori no guarantee that an Adam-optimized DAE should find a global minimum, as described by the Conjecture 3.3, rather than a local minimum. The compelling agreement between theory and simulations in Fig. 1 temptingly suggests that -at least in this case- the loss landscape of DAEs (2) trained with the loss (4) for the data model (1) should in some way be benign. Authors of [12] have shown, for linear RAEs , that there exists a unique global and local minimum for the square loss and no regularizer. Ref. [14] offers further insight for a linear DAE in dimension d = 1 , and shows that, aside from the global minima, the loss landscape only includes an unstable saddle point from which the dynamics easily escapes. Extending these works and intuitions to non-linear DAEs is an exciting research topic for future work.

Example 2: MNIST, FashionMNIST It is reasonable to ask whether Conjecture 3.3 is restricted to Gaussian mixtures (1). The answer is negative - in fact, Conjecture 3.3 also describes well a number of real data distributions. We provide such an example for FashionMNIST [16] (from which, for simplicity, we only kept boots and shoes) and MNIST [15] (1s and 7s), in Fig. 2. For each data set, samples sharing the same label were considered to belong to the same cluster. Note that we purposefully chose closely related classes, for which the clusters are expected to be closer, leading to an a priori harder - and thus more interesting - learning problem. The mean and covariance thereof were estimated numerically, and combined with Conjecture 3.3. The resulting denoising MSE predictions mse ˆ f are plotted as solid lines in Fig. 2, and agree very well with numerical simulations of DAEs optimized over the real data sets using the Pytorch implementation of Adam [43]. A full description of this experiment is given in Appendix D.

The observation that the MSEs of real data sets are to such degree of accuracy captured by the equivalent Gaussian mixture strongly hints at the presence of Gaussian universality [44]. This opens a gateway to future research, as Gaussian universality has hitherto been exclusively addressed in classification and regression (rather than denoising) settings, see e.g. [44, 45, 46]. Denoising tasks further constitute a particularly intriguing setting for universality results, as Gaussian universality would signify that only second-order statistics of the data can be reconstructed using a shallow autoencoder.

0.0

0.2

0.4

0.6

0.8

1.0

3.0

2.5

2.0

1.5

1.0

0.5

0.0

0.5

mse

f

mse

r

theory

MNIST

0.0

0.2

0.4

0.6

0.8

1.0

2.5

2.0

1.5

1.0

0.5

0.0

0.5

mse

f

mse

r

theory

FashionMNIST

4 The role and importance of the skip connection.

Conjecture 3.3 for the full DAE ˆ f (2) and Corollary 3.4 for its components ˆ r, ˆ u (3) allow to disentangle the contribution of each part, and thus to pinpoint their respective roles in the DAE architecture. We sequentially present a comparison of ˆ f with ˆ r , and ˆ f with ˆ u . We remind that ˆ f , ˆ r and ˆ u result from independent optimizations over the same train set D , and that while f b, w = u w + h b , ˆ f = ˆ u +ˆ r .

̸

Full DAE and the rescaling component We start this section by observing that for noise levels ∆ below a certain threshold, the full DAE ˆ f yields better MSE than the learnt rescaling ˆ r , as can be seen by the negative value of mse ˆ f -mse ˆ r in Fig. 1 and Fig. 2. The improvement is more sizeable at intermediate noise levels ∆ , and is observed for a growing region of ∆ as the sample complexity α increases, see Fig. 3 (a). This lower MSE further translates into visible qualitative changes in the result of denoising. As can be seen from Fig. 2 (left), the full DAE ˆ f (2) (bottom left) yields denoised images with sensibly higher definition and overall contrast, while a simple rescaling ˆ r (bottom right) leads to a still largely blurred image.

We provide one more comparison: for the isotropic binary mixture (see Fig. 1), the DAE test error mse ˆ f in fact approaches the information-theoretic lowest achievable MSE mse ⋆ as the sample complexity α increases. To see this, note that mse ⋆ is given by the application of Tweedie's formula [42], that requires perfect knowledge of the cluster means µ k and covariances Σ k - it is, therefore, an oracle denoiser. A sharp asymptotic characterization of the oracle denoiser is provided in Appendix B. As can be observed from Fig. 3 (a), the DAE MSE (2) approaches -but does not exactly converges to (see Appendix C)- the oracle test error mse ⋆ as the number of available training samples n grows, and is already sensibly close to the optimal value for α = 8 .

DAEs with(out) skip connection We now turn our attention to comparing the full DAE ˆ f (2) to the bottleneck network component ˆ u (3). It follows from Conjecture 3.3 and Corollary 3.4 that ˆ u (3) leads to a higher MSE than the full DAE ˆ f (2), with the gap being Θ( d ) . More precisely,

0.0

0.2

0.4

0.6

0.8

0.5

0.4

0.3

0.2

0.1

0.0

0.1

0.2

0.3

mse

f

mse

r

=0.5

=1

=2

=8

mse

mse

r

(a)

0.0

0.2

0.4

0.6

0.8

1.0

0.00

0.02

0.04

0.06

0.08

1

d

(mse

u

mse

f

)

theory

simulations

PCA

(b)

(a)

(b)

(c)

(d)

(e)

(f)

The theoretical prediction (15) compares excellently with numerical simulations; see Fig. 3 (right). Strikingly, we find that PCA denoising yields an MSE almost indistinguishable from ˆ u , see Fig. 3, strongly suggesting that ˆ u essentially learns, also in the denoising setting, to project the noisy data ˜ x along the principal components of the training set. The last two images of Fig. 4 respectively correspond to ˆ u and PCA, which can indeed be observed to lead to visually near-identical results. This echoes the findings of [10, 11, 8, 9, 13] in the case of RAEs that bottleneck networks are limited by the PCA reconstruction performance - a conclusion that we also recover from Corollary 3.5, see Appendix F. Crucially however, it also means that compared to the full DAE ˆ f (2), PCA is sizeably suboptimal , since mse PCA ≈ mse ˆ u = mse ˆ f +Θ( d ) .

This last observation has an important consequence: in contrast to previously studied RAEs [10, 12, 11, 9, 13], the full DAE ˆ f does not simply learn to perform PCA. In contrast to bottleneck RAE networks [8, 9, 13], the non-linear DAE hence does not reduce to a linear model after training. The non-linearity is important to improve the denoising MSE, see Fig. 1. We stress this finding: trained alone, the bottleneck network ˆ u only learns to perform PCA; trained jointly with the rescaling component as part of the full DAE f b, w (2), it learns a richer, non-linear representation. The full DAE (2) thus offers a genuinely non-linear learning model and opens exciting research avenues for the theory of autoencoders, beyond linear (or effectively linear) cases. In the next paragraph, we explore further the interaction between the rescaling component and the bottleneck network.

A tradeoff between the rescaling and the bottleneck network Conjecture (3.3), alongside Corollary (3.4) and the discussion in Section 4 provide a firm theoretical basis for the well-known empirical intuition (discussed e.g. in [38]) that skip-connections allow to better propagate information

from the input to the output of the DAE, thereby contributing to preserving intrinsic characteristics of the input. This effect is clearly illustrated in Fig. 4, where the resulting denoised image of an MNIST 7 by ˆ r , ˆ f , ˆ u and PCA are presented. While the bottleneck network ˆ u perfectly eliminates the background noise and produces an image with a very good resolution, it essentially collapses the image to the cluster mean, and yields, like PCA, the average MNIST 7 . As a consequence, the denoised image bears little resemblance with the original image - in particular, the horizontal bar of the 7 is lost in the process. Conversely, the rescaling ˆ r preserves the nuances of the original image, but the result is still largely blurred and displays overall poor contrast. Finally, the complete DAE (2) manages to preserve the characteristic features of the original data, while enhancing the image resolution by slightly overlaying the average 7 thereupon.

The optimization of the DAE (4) is therefore described by a tradeoff between two competing effects - namely the preservation of the input nuances by the skip connection, and the enhancement of the resolution/noise removal by the bottleneck network. This allows us to discuss the curious nonmonotonicity of the cosine similarity θ as a function of the noise level ∆ , see Fig. 1 (left). While it may at first seem curious that the DAE seemingly does not manage to learn the data structure better for low ∆ than for intermediate ∆ (where the cosine similarity θ is observed to be higher), this is actually due to the afore-dicussed tradeoff. Indeed, for small ∆ , the data is still substantially clean, and there is therefore no incentive to enhance the contrast by using the cluster means -which are consequently not learnt. This phase is thus characterized by a large skip connection strength ˆ b , and small cosine similarity θ and weight norm ∥ ˆ w ∥ F . Conversely, at high noise levels ∆ , the nuances of the data are already lost because of the noise. Hence the DAE does not rely on the skip connection component (whence the small values of ˆ b ), and the only way to produce reasonably denoised data is to collapse to the cluster mean using the network component (whence a large ∥ ˆ w ∥ F ).

5 Conclusion

We consider the problem of denoising a high-dimensional Gaussian mixture, by training a DAE via empirical risk minimization, in the limit where the number of training samples and the dimension are proportionally large. We provide a sharp asymptotic characterization of a number of summary statistics of the trained DAE weight, average MSE, and cosine similarity with the cluster means. These results contain as a corollary the case of RAEs. Building on these findings, we isolate the role of the skip connection and the bottleneck network in the DAE architecture and characterize the tradeoff between those two components in terms of preservation of the data nuances and noise removal thereby providing some theoretical insight into a longstanding practical intuition in machine learning.

We believe the present work also opens exciting research avenues. First, our real data experiments hint at the presence of Gaussian universality. While this topic has gathered considerable attention in recent years, only classification/regression supervised learning tasks have been hitherto addressed. Which aspects of universality carry over to denoising tasks, and how they differ from the current understanding of supervised regression/classification, is an important question. Second, the DAE with skip connection (2) provides an autoencoder model which does not just simply learn the principal components of the training set. It, therefore, affords a genuinely non-linear network model where richer learning settings can be investigated.

References

Yang Song, Jascha Narain Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. International Conference on Learning Representations , 2021.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Pascal Vincent, H. Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. J. Mach. Learn. Res. , 11:3371-3408, 2010.

Reza Oftadeh, Jiayi Shen, Zhangyang Wang, and Dylan A. Shell. Eliminating the invariance on the loss landscape of linear autoencoders. In International Conference on Machine Learning , 2020.

Daniel Kunin, Jonathan M. Bloom, Aleksandrina Goeva, and Cotton Seed. Loss landscapes of regularized linear autoencoders. In International Conference on Machine Learning , 2019.

Xuchan Bao, James Lucas, Sushant Sachdeva, and Roger B Grosse. Regularized linear autoencoders recover the principal components, eventually. Advances in Neural Information Processing Systems , 33:6971-6981, 2020.

Gauthier Gidel, Francis R. Bach, and Simon Lacoste-Julien. Implicit regularization of discrete gradient dynamics in deep linear neural networks. In Neural Information Processing Systems , 2019.

Maria Refinetti and Sebastian Goldt. The dynamics of representation learning in shallow, nonlinear autoencoders. In International Conference on Machine Learning , pages 18499-18519. PMLR, 2022.

A. E. Shevchenko, Kevin Kögler, Hamed Hassani, and Marco Mondelli. Fundamental limits of two-layer autoencoders, and achieving them with gradient methods. ArXiv , abs/2212.13468, 2022.

Carl Eckart and G. Marion Young. The approximation of one matrix by another of lower rank. Psychometrika , 1:211-218, 1936.

Hervé Bourlard and Yves Kamp. Auto-association by multilayer perceptrons and singular value decomposition. Biological Cybernetics , 59:291-294, 1988.

Pierre Baldi and Kurt Hornik. Neural networks and principal component analysis: Learning from examples without local minima. Neural Networks , 2:53-58, 1989.

Phan-Minh Nguyen. Analysis of feature learning in weight-tied autoencoders via the mean field lens. ArXiv , abs/2102.08373, 2021.

Arnu Pretorius, Steve Kroon, and Herman Kamper. Learning dynamics of linear denoising autoencoders. In International Conference on Machine Learning , 2018.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. IEEE , 86:2278-2324, 1998.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. ArXiv , abs/1708.07747, 2017.

Adityanarayanan Radhakrishnan, Mikhail Belkin, and Caroline Uhler. Overparameterized neural networks implement associative memory. Proceedings of the National Academy of Sciences of the United States of America , 117:27162 - 27170, 2019.

Saachi Jain, Adityanarayanan Radhakrishnan, and Caroline Uhler. A mechanism for producing aligned latent spaces with autoencoders. ArXiv , abs/2106.15456, 2021.

Thanh Van Nguyen, Raymond K. W. Wong, and Chinmay Hegde. On the dynamics of gradient descent for autoencoders. In International Conference on Artificial Intelligence and Statistics , 2019.

Thanh Van Nguyen, Raymond K. W. Wong, and Chinmay Hegde. Benefits of jointly training autoencoders: An improved neural tangent kernel analysis. IEEE Transactions on Information Theory , 67:4669-4692, 2019.

Giorgio Parisi. Towards a mean field theory for spin glasses. Phys. Lett , 73(A):203-205, 1979.

Giorgio Parisi. Order parameter for spin glasses. Phys. Rev. Lett , 50:1946-1948, 1983.

Lenka Zdeborová and Florent Krzakala. Statistical physics of inference: thresholds and algorithms. Advances in Physics , 65:453 - 552, 2015.

Marylou Gabrié. Mean-field inference methods for neural networks. Journal of Physics A: Mathematical and Theoretical , 53, 2019.

Elizabeth Gardner and Bernard Derrida. Optimal storage properties of neural network models. Journal of Physics A: Mathematical and general , 21(1):271, 1988.

Manfred Opper and David Haussler. Generalization performance of bayes optimal classification algorithm for learning a perceptron. Physical Review Letters , 66(20):2677, 1991.

Jean Barbier, Florent Krzakala, Nicolas Macris, Léo Miolane, and Lenka Zdeborová. Optimal errors and phase transitions in high-dimensional generalized linear models. Proceedings of the National Academy of Sciences of the United States of America , 116:5451 - 5460, 2017.

Benjamin Aubin, Florent Krzakala, Yue M. Lu, and Lenka Zdeborová. Generalization error in high-dimensional perceptrons: Approaching bayes error with convex optimization. Advances in Neural Information Processing Systems , 33:12199-12210, 2020.

Benjamin Aubin, Antoine Maillard, Jean Barbier, Florent Krzakala, Nicolas Macris, and Lenka Zdeborová. The committee machine: computational to statistical gaps in learning a two-layers neural network. Journal of Statistical Mechanics: Theory and Experiment , 2019, 2018.

Rainer Dietrich, Manfred Opper, and Haim Sompolinsky. Statistical mechanics of support vector networks. Physical review letters , 82(14):2975, 1999.

Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In International Conference on Machine Learning , 2020.

Federica Gerace, Bruno Loureiro, Florent Krzakala, Marc Mézard, and Lenka Zdeborová. Generalisation error in learning with random features and the hidden manifold model. Journal of Statistical Mechanics: Theory and Experiment , 2021, 2020.

Hugo Cui, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Error rates for kernel classification under source and capacity conditions. ArXiv , abs/2201.12655, 2022.

Christos Thrampoulidis, Ehsan Abbasi, and Babak Hassibi. Precise error analysis of regularized m -estimators in high dimensions. IEEE Transactions on Information Theory , 64(8):5592-5628, 2018.

Jacob A. Zavatone-Veth, William Tong, and Cengiz Pehlevan. Contrasting random and learned features in deep bayesian linear regression. Physical review. E , 105 6-1:064118, 2022.

Hugo Cui, Florent Krzakala, and Lenka Zdeborová. Optimal learning of deep random networks of extensive-width. ArXiv , abs/2302.00375, 2023.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 , pages 234-241. Springer, 2015.

Xiao-Jiao Mao, Chunhua Shen, and Yubin Yang. Image restoration using convolutional autoencoders with symmetric skip connections. ArXiv , abs/1606.08921, 2016.

Tong Tong, Gen Li, Xiejie Liu, and Qinquan Gao. Image super-resolution using dense skip connections. In Proceedings of the IEEE international conference on computer vision , pages 4799-4807, 2017.

Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1646-1654, 2016.

Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Deeply-recursive convolutional network for image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1637-1645, 2016.

Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association , 106:1602 - 1614, 2011.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR , abs/1412.6980, 2014.

Sebastian Goldt, Bruno Loureiro, Galen Reeves, Florent Krzakala, Marc Mézard, and Lenka Zdeborová. The gaussian equivalence of generative models for learning with shallow neural networks. In Mathematical and Scientific Machine Learning , 2020.

Hong Hu and Yue M. Lu. Universality laws for high-dimensional learning with random features. IEEE Transactions on Information Theory , 69:1932-1964, 2020.

Andrea Montanari and Basil N Saeed. Universality of empirical risk minimization. In Conference on Learning Theory , pages 4310-4312. PMLR, 2022.

Bruno Loureiro, Cédric Gerbelot, Hugo Cui, Sebastian Goldt, Florent Krzakala, Marc Mézard, and Lenka Zdeborová. Learning curves of generic features maps for realistic datasets with a teacher-student model. Journal of Statistical Mechanics: Theory and Experiment , 2022, 2021.

Bruno Loureiro, Gabriele Sicuro, Cédric Gerbelot, Alessandro Pacco, Florent Krzakala, and Lenka Zdeborová. Learning gaussian mixtures with generalised linear models: Precise asymptotics in high-dimensions. In Neural Information Processing Systems , 2021.

A Derivation of Conjecture 3.3

In this section, we detail the derivation of Conjecture (3.3).

A.1 Derivation technique

In order to access sharp asymptotic characterizations for the metrics and summary statistics mse (5), θ (6), ˆ b and ∥ ˆ w ∥ / d , the first step is to observe that any test function φ ( ˆ w , ˆ b ) of the learnt network parameters ˆ w , ˆ b can be written as an average over a limit probability distribution

where ˆ R ( · , · ) is the empirical risk (4) and the corresponding probability density

is known as the Boltzmann measure in statistical physics, with the parameter β typically referred to as the inverse temperature . The normalization Z is called the partition function of P β :

The idea of the replica method [21, 22, 23, 24], building on the expression (16), is to compute the moment generating function (also known as the free entropy ) as

With the moment generating function, it then becomes possible to compute averages like (16). The backbone of the derivation, which we detail in the following subsections, therefore lies in the computation of the replicated partition function E D Z s .

A.2 Replicated partition function

For subsequent clarity, we shall introduce the variable

if x µ belongs to the k -th cluster. By definition of the Gaussian mixture model, this is just a Gaussian variable: η µ ∼ N (0 , Σ k ) , and the average over the data set { x µ } reduces to averaging over the cluster index k and { η µ } . For notational brevity, we shall name in this appendix ξ µ what is called √ ∆ ξ µ in the main text (i.e., absorbing the variance in the definition of ξ µ ). The replicated partition function then reads

One can expand the exponent ( ⋆ ) as

Therefore

The effective prior over the noises η , ξ is Gaussian with means µ ξ , µ η and covariance C :

Identifying the covariance components leads to

Note that all the sums indexed by a introduce terms of O ( s ) , which can safely be neglected. Identifying the means leads to the following system of equations:

which is solved as

Again, observe that g µ η , g µ ξ = O ( s ) and will be safely neglected later in the derivation. Therefore

We are now in a position to introduce the local fields:

with statistics

(

a

)

We used the leading order of the covariances C ξξ,ηη,ηξ . One therefore has to introduce the summary statistics:

Note that the local fields λ η,ξ a thus follow a Gaussian distribution:

Going back to the computation, ( a ) can be rewritten as

A.3 Reformulating as a saddle-point problem

Introducing Dirac functions enforcing the definitions of Q ab , m a brings the replicated function in the following form:

We introduced the trace, entropic and energetic potentials Ψ t , Ψ w , Ψ y . Since all the integrands scale exponentially (or faster) with d , this integral can be computed using a saddle-point method. To proceed further, note that the energetic term encompasses two types of terms, scaling like sd 2 and sd . More precisely,

Note that the saddle-point method involves an intricate extremization over s × s matrices Q,S k . To make further progress, we assume the extremum is realized at the Replica Symmetric (RS) fixed point

This is a standard assumption known as the RS ansatz [21, 22, 23, 24].

Quadratic potential The quadratic potential Ψ quad . , which correspond to the leading order term in d in the exponent, needs to be extremized first in the framework of our saddle-point analysis. Its expression can be simplified as

which is extremized for

This fixes the skip connection strength b .

Entropic potential We now turn to the entropic potential Ψ w . It is convenient to introduce the variance order parameters

The entropic potential can then be expressed as e βsd Ψ w

Therefore

For a matrix Ξ ∈ R p × d and tensors A , B ∈ R p × d ⊗ R p × d , we denoted ( Ξ ⊙ A ) kl = ∑ ij Ξ ij A ij,kl and ( A ⊙ B ) ij,kl = ∑ rs A ij,rs B rs,kl .

Energetic potential In order to compute the energetic potential Ψ y , one must first compute the inverse of the covariance Ω k , given by

It is straightforward to see that S -1 k , Q -1 share the same block structure as S k , Q , and we shall name for clarity

From the Sherman-Morisson lemma, and noting

one reaches

We will also need the expression of the determinants

One can finally proceed in analyzing the term ( a ) :

where we noted with capital D an integral over a p -dimensional Gaussian distribution N (0 , I p ) . Therefore

A.4 zero-temperature limit

In this subsection, we take the zero temperature β →∞ limit. Rescaling

one has that

where we introduced the Moreau enveloppe

Note that while this optimization problem is still cast in a space of dimension p × d , for some regularizers g ( · ) including the usual ℓ 1 , 2 penalties, the Moreau enveloppe admits a compact analytical expression.

The energetic potential Ψ y also simplifies in this limit to

Where

It is immediate to see that the trace potential Ψ t can be expressed as

Putting these results together, the total free entropy reads

where we remind that

and

A.5 Self-consistent equations

The extremization problem (58) can be recast as a system of self-consistent equations by requiring the gradient with respect to each summary statistic involved in (58) to be zero. This translates into:

We remind that the skip connection b is fixed to

A.6 Sharp asymptotic formulae for the learning metrics

The previous subsections have allowed to obtain sharp asymptotic characterization for a number of summary statistics of the probability (17). These statistics are in turn sufficient to asymptotically characterize the learning metrics discussed in the main text. We successively address the test MSE (5), cosine similarity (6) and training MSE.

MSE The denoising MSE reads

where

Cosine similarity By the very definition of the summary statistics m k , q :

Doing so reveals

Taking the β →∞ limit finally leads to

The sharp asymptotic formula (68) agrees well with numerical simulations, Fig. 5. As is intuitive, the generalization gap mse ˆ f -ϵ t grows with the noise level ∆ , as the learning problem grows harder.

0.0

0.2

0.4

0.6

0.8

1.0

0.00

0.02

0.04

0.06

0.08

t

/

d

theory

simulations

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

mse

f

t

theory

simulations

Training error The replica formalism also allows to compute a sharp asymptotic characterization for the training MSE

We have skipped the discussion of this learning metric in the main text for the sake of conciseness, but will now detail the derivation. Note that the training error ϵ t can be deduced from the risk (4) and the average regularization as

Note that in turn, from the definition of the free entropy, the average risk (training loss) can be computed as

A.7 Generic asymptotically exact formulae

This last result ends the derivation of the generic version of Conjecture 3.3, i.e. not assuming Assumptions 3.1 and 3.2.

Importantly, note that the characterization (58) is, like the formulae in e.g. [47, 48], asymptotically exact , but not fully asymptotic, as the equations (58) still involve high-dimensional quantities. In practice however, for standard regularizers r ( · ) like ℓ 1 or ℓ 2 , the proximal prox r admits a simple closed-form expression, and the average over Ξ can be carried out analytically. The only high-dimensional operation left is then taking the trace of linear combinations of Σ k matrices (or the inverse of such combinations), which is numerically less demanding, and analytically much simpler, than the high dimensional optimization (4) and averages (5) involved in the original problem. We give in the next paragraph an example of how Assumption 3.1 can be relaxed, for a binary mixture with arbitrary (i.e. not necessarily jointly diagonalizable) covariances Σ 1 , 2 .

Example: Anisotropic, heteroscedastic binary mixture As an illustration, we provide the equivalent of (58) for a binary mixture, with generically anisotropic and distinct covariances, for p = 1 , thereby breaking free from Assumption 3.1. We index the clusters by + , -rather than 1 , 2 for notational convenience. We remind that

Then the self-consistent equations (58) simplify to

These equations are, as previously discussed, asymptotically exact as d → ∞ . While they still involve traces over high-dimensional matrices Σ k , this is a very simple operation. We have therefore reduced the original high-dimensional optimization problem (4) to the much simpler one of computing traces like (70). Crucially, while these traces cannot be generally simplified without Assumption 3.1, they provide the benefit of a simple and compact expression which bypasses the need of jointly diagonalizable covariances, and which can thus be readily evaluated for any covariance, like real-data covariances. This versatility is leveraged in Appendix D.

A.8 End of the derivation: Conjecture 3.3

We finally provide the last step in the derivation of the fully asymptotic characterization of Conjecture 3.3. Assuming r ( · ) = λ / 2 ∥·∥ 2 F (Assumption 3.2), the Moreau envelope M r assumes a very simple expression, and no longer needs to

be written as a high-dimensional optimization problem. The resulting free energy can be compactly written as:

( ∗ ) is the only segment which still involves high-dimensional matrices, and is therefore not yet full asymptotic. Assuming Assumption 3.1, ( ∗ ) can be massaged into

The fully asymptotic free energy thus becomes

Requiring the free energy to be extremized implies

which is exactly (13). This closes the derivation of Conjecture 3.3.

As a final heuristic remark, observe that from (74) it is reasonable to expect that all the hatted statistics ˆ q k , ˆ V k , ˆ m k , ˆ q, ˆ V should be of order Θ( α ) as α ≫ 1 . As a consequence, V k = Θ( 1 / α ) and q, m, q k , m k should approach their α →∞ limit as Θ( 1 / α ) . Therefore, the MSE (61) should also approach its α →∞ limit as Θ( 1 / α ) .

A.9 Additional comparisons

We close this appendix by providing further discussion on some points, including the role of the non-linearity σ ( · ) , the influence of the weight-tying assumption, and of the (an-)isotropy and (homo/hetero)-scedasticity of the clusters.

The activation function The figures in the main text were generated using σ = tanh . Several studies for RAEs, however, have highlighted that an auto-encoder would optimally seek to place itself in the linear region of its nonlinearity, so as to learn the principal components of the training data, at least in the non-linear untied weights case, see e.g. [3, 8]. In light of these findings, it is legitimate to wonder whether setting a linear activation σ ( x ) = x would not yield a better MSE. Fig. 6 shows that it is not the case. For the binary isotropic mixture (13), the linear activations yield a worse performance than the tanh activation.

Weight-tying We have assumed that the weights of the DAE (2) were tied. While this assumption originates mainly for technical reasons in the derivation, note that it has been also used and discussed in practical setting [3] as a way to prevent the DAE from going to the linear region of its non-linearity, by making the norm of the first layer weights very small, and that of the second layer very large to compensate [8]. A full extension of Conjecture 3.3 to the case of the untied weight is a lengthy theoretical endeavour, which we leave for future work. However, note that Fig. 6 shows, in the binary isotropic case, that untying the weights actually leads to a worse MSE than the DAE with tied weights. This is a very interesting observation, as a DAE with untied weights obviously has the expressive power to implement a DAE with tied weights. Therefore, this effect should be mainly due to the untied landscape presenting local minima trapping the optimization. A full landscape analysis would present a very important future research topic. Finally, remark that this stands in sharp contrast to the observation of [8] for non-linear RAEs, where weight-tying worsened the performance, as it prevents the RAE from implementing a linear principal component analysis.

0.0

0.2

0.4

0.6

0.8

0.4

0.2

0.0

0.2

0.4

0.6

mse

f

mse

r

tanh, tied

linear, tied

tanh, untied

An important conclusion from these two observations is that, in contrast to RAEs, the DAE (2) does not just implement a linear principal component analysis - weight tying, and the presence of a non-linearity, which are obstacles for RAEs in reaching the PCA MSE, lead for DAEs (2) to a better MSE. The model (2) therefore constitutes a model of a genuinely non-linear algorithm, where the non-linearity is helpful and is not undone during training. Further discussion can be found in Section 4 of the main text.

Heteroscedasticity, anisotropy Fig. 1 and 3 in the main text all represent the isotropic, homoscedastic case. This simple model actually encapsulates all the interesting phenomenology. In this paragraph, we discuss for completeness the generically heteroscedastic, anisotropic case. We consider a binary mixture, with covariances Σ 1 , Σ 2 independently drawn from the Wishart-Laguerre ensemble, with aspect ratios 5 / 7 and 5 / 6 . Therefore, the clusters are anisotropic, and eigendirections associated with the largest eigenvalues are more "stretched" (i.e. induce higher cluster variance). Furthermore, since the set of eigenvectors of Σ 1 , Σ 2 are independent, the two clusters are stretched in different directions. To ease the comparison with Fig. 1 (isotropic, homoscedastic), these Wishart matrices were further divided by 10 , so that the trace is approximately 0 . 09 like in Fig. 1 - i.e., the clusters have the same average extension. Fig. 7 presents the resulting metrics and summary statistics. Again, the agreement between the theory 3.3 and numerical simulation using Pytorch is compelling. Qualitatively, the observations made in Section 4 still hold true in the anisotropic heteroscedastic case:

The curve of the cosine similarity still displays a non-monotonic behaviour, signalling the preferential activation by the DAE of its skip connection at low ∆ and of its network component at high ∆ .

The skip connection strength ˆ b is higher at small ∆ and decreases, in connection to the previous remark.

The norm of the weight matrix ˆ w overall increases with the noise level ∆ , signalling an increasing usage by the DAE of its bottleneck network component, again in accordance to the previous remark.

Therefore, the generic case does not introduce qualitative changes compared to the isotropic case.

Strength of the weight decay λ A ℓ 2 regularization (weight decay) λ = 0 . 1 was adopted in the main text. In fact, the value of the strength of the weight decay was not found to sensibly influence the curves, and again, the qualitative phenomena discussed in Section 4 are observed for any value. Fig. 8 shows, for an isotropic binary mixture, the MSE difference mse ˆ f -mse ˆ r (5) and the cosine similarity θ (6) for regularization strength λ = 0 . 1 and λ = 0 . 001 . As can be observed, even reducing the regularization a hundredfold does not change at all the qualitative picture - in particular, the non-monotonicity of the cosine similarity, discussed in Section 4 - , and very little the quantitative values. On the

0.0

0.2

0.4

0.6

0.8

0.4

0.3

0.2

0.1

0.0

0.1

0.2

0.3

mse

f

mse

r

0.0

0.2

0.4

0.6

0.8

0.0

0.2

0.4

0.6

0.8

1.0

1.2

||

w

||

2

/

d

b

PCA

0.0

0.2

0.4

0.6

0.8

1.0

0.4

0.2

0.0

0.2

0.4

0.6

mse

f

mse

r

=0.1

=0.001

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

=0.1

=0.001

PCA

0.0

0.2

0.4

0.6

0.8

1.0

0.4

0.3

0.2

0.1

0.0

0.1

0.2

0.3

mse

f

mse

r

=0.1

=0.3

=0.5

=0.7

0.0

0.2

0.4

0.6

0.8

1.0

e

0.000

0.002

0.004

0.006

0.008

0.010

0.012

mse

f

mse

r

mse

mse

r

0.0

0.2

0.4

0.6

0.8

1.0

e

0.00

0.02

0.04

0.06

0.08

1

d

(mse

u

mse

f

)

theory

PCA

simulations

other hand, observe that at the level of the MSE increasing the weight decay helps palliate the overfitting at large noise levels ∆ , as we further illustrate in Fig. 9

K = 1 cluster We finally discuss the case of unstructured data, where the Gaussian mixture only has a single cluster. Analytically, this corresponds to setting the norm of the cluster mean ∥ µ ∥ to zero in Conjecture 3.3. The bottleneck component, whose role is to learn the data structure - as given by the cluster mean-and leverage it to improve the denoising performance, is no longer needed when the data is unstructured. It presence actually leads the DAE to overfit, as can be observed in Fig.10 (left). Similarly, the PCA denoiser performs worse in the unstructured case, with an associated MSE remaining of order Θ( d ) , see Fig.10 (right).

B Tweedie baseline

B.1 Oracle denoiser

Tweedie's formula [42] provides the best estimator of a clean sample x given a noisy sample ˜ x corrupted by Gaussian noise, as

Note that this gives the Bayes-optimal estimator for the MSE, assuming perfect knowledge of the clean data distribution P from which the so-called score ∇ P has to be computed. Of course, this knowledge is inaccessible in general, where the only information on P is provided in the form of the train set D . Therefore, Tweedie's formula does not give a learning algorithm, but allows to give an oracle lower-bound on the achievable MSEs, as any learning algorithm will yield a higher MSE than this information-theoretic baseline.

For a generic Gaussian mixture (1), Tweedie's formula reduces to the expression

where we noted

B.2 Oracle test MSE

In general, except in the complete homoscedastic case where all clusters have the same covariance, there is no closed-form asymptotic expression for the MSE achieved by the Tweedie denoiser. In the binary homoscedastic case µ 1 = -µ 2 ≡ µ , Σ 1 = Σ 2 = σ 2 I d (see Fig. 1), Tweedie's formula (77) reduces to the compact form

Note that this is of the same form as the DAE architecture (2). The associated MSE reads

A sharp asymptotic formula can be found to be

This is the theoretical characterization plotted in Figs. 1 and 3 in the main text.

As discussed in the main text and illustrated in Fig. 3(left), the MSE of the DAE approaches the oracle MSE (81) as the sample complexity α grows. On the other hand, the DAE MSE does not exactly converges to the oracle MSE as α →∞ . Intuitively, this is because of the fact that while the form of the oracle (79) is that of a DAE, it does not have tied weights, like the considered architecture (2). The latter cannot therefore perfectly realize the oracle denoiser, whence a small discrepancy, as more precisely numerically characterized in Fig. 11.

10 0

10 1

10 2

10 3

10 4

0.5

0.4

0.3

0.2

0.1

mse

f

mse

r

e

=0.2

e

=0.5

e

=0.8

C Bayes-optimal MSE

C.1 Assuming imperfect prior knowledge

The Tweedie denoiser discussed in Appendix B is an oracle denoiser, in the sense that perfect knowledge of all the parameters { µ k , Σ k } K k =1 of the Gaussian mixture distribution P (1) is assumed. Therefore, though the comparison of the DAE (2) MSE with the oracle Tweedie MSE mse ⋆ does provide useful intuition (see Fig. 3), it importantly does not allow to disentangle which part of the MSE of the DAE is due to the limited expressivity of the architecture, and which is due to the limited availablility of training samples - which entails imperfect knowledge of the parameters { µ k , Σ k } K k =1 of the Gaussian mixture distribution P (1). In this appendix, we derive the MSE of the Bayes optimal estimator (with respect to the MSE), assuming only knowledge of the distribution from which the parameters { µ k , Σ k } K k =1 are drawn. For simplicity and conciseness, we limit ourselves to the binary isotropic and homoscedastic mixture (see e.g. Fig. 1), for which µ 1 = -µ 2 ≡ µ ⋆ and Σ 1 = Σ 2 = σ I d . For definiteness, we assume µ ∼ P µ , with P µ = N (0 , I d / d ) , so that with high probability ∥ µ ∥ = 1 . We shall moreover assume for ease of discussion that σ is perfectly known. Thus, the centroid µ is the only unknown parameter.

C.2 Bayes-optimal denoiser without knowledge of cluster means

The corresponding Bayes-optimal denoiser then reads

Where we have identified the oracle denoiser (79), and emphasized the dependence on µ . Note that this is a slight abuse of notation, as the true oracle t ( µ ⋆ , ˜ x ) involves the ground truth centroid. Further note that P ( µ | σ, D ) = P ( µ |D ) , since knowledge of σ does not bring information about µ . Furthermore, note that as the noisy part of the dataset ˜ D ≡ { ˜ x µ } n µ =1 brings only redundant information, one further has that

where we noted ˇ D = { x µ } n µ =1 the clean part of the dataset. Thus

where we noted with the effective variance

Finally, the partition function Z is defined as the normalisation of P b , i.e.

One is now in a position to compute the MSE associated with the Bayes estimator b (˜ x ) :

mse b

We have adopted the shortcuts

(88) shows that the inner averages involved in the Bayes MSE mse b only depend on the overlaps µ ⊤ 1 , 2 µ 1 , 2 / d for µ 1 , µ 2 two independently sampled vectors from P b . Motivated by similar high-dimensional studies, it is reasonable to expect these quantities to concentrate as n, d →∞ in the measure E D ⟨·⟩ P b . A more principle - but much more painstakingway to derive this assumption is to introduce a Fourier representation, and carry out the E D average using the replica method. We refer the interested reader to, for instance, Appendix H of [36], where such a route is taken.

C.3 Derivation

To compute the summary statistics of P b , we again resort to the replica method to compute the moment-generating function (free entropy), see also Appendix A. The replicated partition function reads

Introducing the order parameters

one reaches

The computation of Ψ t , Ψ w , Ψ y is rather standard and follows the main steps presented in Appendix A. One again assumes the replica symmetric ansatz [21, 22]

Introducing as in Appendix A the variances

Therefore the free entropy reads

one reaches

0.0

0.2

0.4

0.6

0.8

1.0

0.4

0.2

0.0

0.2

mse

mse

DAE

Bayes

oracle

PCA oracle

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Bayes

DAE

PCA

The solution of this extremization problem is given by the set of self-consistent equations, imposing that gradients with respect to all parameters be zero:

This is a simple optimization problem over 6 variables, which can be solved numerically. One can finally use the obtained summary statistics to evaluate the Bayes optimal MSE:

where

This completes the derivation of the MSE of the Bayes estimator agnostic of the cluster means.

C.4 A simple plug-in denoiser

Another simple denoiser assuming only perfect knowledge of the variance σ , but imperfect knowledge of µ , is simply to plug the PCA estimate ˆ µ PCA of µ into the Tweedie oracle (76), i.e.

The performance of this plug-in denoiser is plotted in Fig. 12, and contrasted to the one of the Bayes denoiser b ( · ) (101), the oracle t ( · ) (81) and the DAE 3.3. Strikingly, the performance of the PCA plug-in denoiser is sizeably identical to the one of the Bayes-optimal denoiser, both in terms of MSE and cosine similarity. It is important to remind at this point that both the plug-in (103) and the Bayes denoiser (101) still rely on perfect knowledge of the cluster variance σ and the noise level ∆ , while the DAE (2) is agnostic to these parameters. Yet, these two denoisers make for a fairer comparison than the oracle (81), as they importantly take into account the finite training set. The DAE is relatively close to the Bayes baseline for small noise levels ∆ , while a gap can be seen to open up for larger ∆ .

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

||

w

||

2

/

d

b

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

||

w

||

2

/

d

b

D Details on real data simulations

In this Appendix, we provide further details on the real data experiments presented in Fig. 2 and Fig. 4.

Preprocessing The original MNIST [15] and FashionMNIST [16] data-sets were flattened (vectorized), centered, and rescaled by 400 (MNIST) and 600 (FashionMNIST). For simplicity and ease of discussion, for each data set, only two labels were kept, namely 1 s and 7 s for MNIST and boots and shoes for FashionMNIST. Note that the visual similarity between the two selected classes should make the denoising problem more challenging.

Means and covariance estimation For each data-set, data from the same class were considered to belong to the same Gaussian mixture cluster. For simplicity, we kept the same number of training points in each cluster, so as to obtain balanced clusters ( ρ 1 = ρ 2 = 1 / 2 ) for definiteness. For each cluster, the corresponding mean µ and covariance Σ were numerically evaluated from the empirical mean and covariance over the 6000 boots (shoes) in the FashionMNIST training set, and the 6265 1 s ( 7 s) in the MNIST training set. The solid lines in Fig. 2 correspond to using those estimates in the asymptotic formulae of Conjecture 3.3, in their generic form (58) (see Appendix A) for sample complexity α = 1 and ℓ 2 regularization λ = 0 . 1 .

Pytorch simulations The red points in Fig. 2 were obtained from numerical simulations, by optimizing a p = 1 DAE (2) with σ = tanh activation using the Pytorch implementation of the full-batch Adam [43] optimizer, with weight decay λ = 0 . 1 , learning rate η = 0 . 05 , over T = 2000 epochs. For each value of ∆ , n = 784 images were drawn without replacement from the data-set training set, corrupted by noise, and fed to the DAE along with the clean image for training. The denoising test MSE, cosine similarity and summary statistics of the trained DAE were estimated after optimization using n test = 1000 images randomly drawn from the test set (from which also only the relevant classes were kept, and equally represented- i.e. balanced). Each value was averaged over N = 10 instances of the train set, noise realization and test set. The obtained values were furthermore found to be robust with respect to the choice of the optimizer, learning rate and number of epochs.

D.1 Complementary simulations

Fig.2 in the main text addresses the comparison of the theoretical prediction of Conjecture 3.3 with simulations on the real data-set, at the level of the denoising test MSE (5). For completeness, we show here the same comparison for the other learning metrics and summary statistics, namely the cosine similarity θ (6), weights norm ∥ ˆ w ∥ 2 / d and trained skip connection strength ˆ b . Theoretical asymptotic characterization are again provided by Conjecture 3.3 (10), (11) and (13).

They are plotted as solid lines in Fig. 13, and contrasted to numerical simulations (dots) on the real data-sets, by training a p = 1 DAE (2) using the Pytorch implementation of the Adam optimizer. All experiment details are the same as Fig. 2 of the main text, and can be found in the previous subsection. As can be observed, there is a gap, for large noise levels ∆ , between the theory and simulations of the weights norm ∥ ˆ w ∥ 2 / d (red). On the other hand, the matchings for the cosine similarity θ (green) and skip connection strength ˆ b (blue) are perfect. Overall, Conjecture (3.3) therefore captures well (with the exception of ∥ ˆ w ∥ 2 / d at large ∆ ) the main metrics of the denoising problem on these two real data-sets. A more thorough investigation of the possible Gaussian universality of the problem is left to future work.

E Derivation of Corollary 3.4

In this Appendix, we provide the derivation of Corollary 3.4.

Rescaling component ˆ h We first provide a succinct derivation of the sharp asymptotic characterizations for the rescaling component ˆ h (3). Because this is only a scalar optimization problem over the single scalar parameter c of h c , the derivation is simple and straightforward. The empirical loss reads

Extremizing this last expression with respect to c leads to

which is exactly (10), as stated in Corollary 3.4. The associated MSE can be then readily computed as

This completes the proof of the asymptotic characterization of the rescaling component ˆ r .

Bottleneck network component ˆ u For ˆ u , one needs to go through the derivation presented in Appendix A by setting b = 0 from the beginning. It is straightforward to realize that the derivation goes through sensibly unaltered, and the only differing step is that the quadratic potential Ψ quad . is now a constant. One therefore only needs to set ˆ b in all the formulae of Conjecture 3.3 to access sharp asymptotics for ˆ u . This concludes the derivation of Corollary 3.4

F Derivation of Corollary 3.5

We now turn to the derivation of Corollary 3.5. This corresponds to the limit ∆ = 0 , where the input of the auto-encoder is also the clean data point. In terms of equations, essentially, this means that all the integrals over ξ , λ ξ in the derivation of Conjecture 3.3 as presented in Appendix A should be removed. For the sake of clarity, and because this is an important case of particular interest, we provide here a complete, succinct but self-contained, derivation in the case of RAEs. For technical reasons, we limit ourselves to ℓ 2 regularization. Furthermore, note that the model of an RAE with a skip connection is trivial, since such an architecture can achieve perfect reconstruction accuracy simply by setting the skip connection strength to 1 and the weights to 0 . Therefore, we consider an RAE with out skip connection

F.1 Derivation

The derivation of asymptotic characterizations for the metrics mse (5), θ (6) and summary statistics for the RAE (107) follow the same lines as for the full DAE (2), as presented in Appendix A. As in Appendix A, the first step is to evaluate the replicated partition function

The exponent ( ⋆ ) can be expanded as

Therefore,

The effective prior over the η is therefore Gaussian with dressed mean and covariance

Observe that this mean is O ( s ) and can be safely neglected. Then

Again, like Appendix A, we introduce the local fields:

where

which have correlation

We used the leading order of the covariance C . One therefore has to introduce the order parameters:

The distribution of the local fields λ η a is then simply:

Going back to the computation, ( a ) can be rewritten as

Introducing Dirac functions enforcing the definitions of Q ab , m a brings the replicated function in the following form:

As in Appendix A, we have introduced the trace, entropic and energetic potentials Ψ t , Ψ w , Ψ y . Since all the integrands scale exponentially (or faster) with d , this integral can be computed using a saddle-point method. To proceed further, note that the energetic term encompasses two types of terms, scaling like sd 2 and sd . More precisely,

Remark that in contrast to Appendix A, the quadratic term is constant with respect to the network parameters, and therefore one no longer needs to discuss its extremization. Following Appendix A, we assume the RS ansatz

In the following, we sequentially simplify the potentials Ψ w , Ψ y , Ψ t under this ansatz.

Entropic potential It is convenient to introduce before proceeding the variance order parameters

The entropic potential can then be expressed as e βsd Ψ w

Therefore

Energetic potential The inverse of S k can be computed like in Appendix A, leading to the following expression for ( a ) :

where we noted with capital D an integral over N (0 , I p ) . Therefore

Zero-temperature limit In this subsection, we take the zero temperature β →∞ limit. Rescaling

one has that

where we introduced the Moreau envelope

M r ( Ξ )

For the case of ℓ 2 regularization, the Moreau envelope presents a simple expression, and the entropy potential assumes the simple form

The energetic potential also simplifies in this limit to

where

Trace potential It is immediate to see that the trace potential can be expressed as

Then the total free entropy for RAEs (107) reads

Comparing to the free energy derived in Appendix A, it is immediate to see that the Moreau envelope M k for the RAE just corresponds to removing the second term from the DAE M k , and setting x = 0 . The disappearance of the second term in the Moreau envelope has the effect to kill the V -dependence, resulting in ˆ q = 0 when extremizing with respect to V . These observations, in addition to the already taken limits ∆ , b = 0 , finish the derivation of Corollary 3.5.

0.0

0.5

1.0

1.5

2.0

2.5

3.0

0.2

0.1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

mseRAE

mse

′

theory

simulations

PCA

0.0

0.5

1.0

1.5

2.0

2.5

3.0

0.7

0.8

0.9

1.0

1.1

1.2

1.3

1.4

1.5

||

w

||

2

/

d

PCA

F.2 Reconstruction MSE

Having obtained a sharp asymptotic characterization of the summary statistics for the RAE, we now turn to the learning metrics. It is easy to see that the formula for the cosine similarity θ (6) carries over unchanged from the DAE case, see Appendix A. We therefore focus on the test MSE:

F.3 RAEs are limited by the PCA MSE

We close this Appendix by showing how the RAE Corollary 3.5 recovers the well-known Eckart-Young theorem [10]. The difference mse -mse ′ ◦ for an RAE learning to reconstruct a binary, isotropic homoscedastic mixture (see Fig. 1) is shown in Fig. 14, from a training set with sample complexity α = 1 . Because mse -mse ′ ◦ is essentially a correction to the leading term mse ′ ◦ , estimating this difference is slightly more challenging numerically, whence rather large error bars. Nevertheless, it can be observed that the agreement between the theory (solid blue line) and the simulations (dots) gathered from training an p = 1 RAE using the Pytorch implementation of Adam is still very good. In terms of cosine similarity and learnt weight norm, the agreement is perfect, see Fig. 3.5 (right). For comparison, the performance of PCA reconstruction (crosses) has been overlayed on the RAE curves. Importantly, observe that in terms of MSE, PCA reconstruction always leads to smaller MSE, in agreement with the well-known Eckart-Young theorem [10]. From the cosine similarity, it can be seen that the RAE essentially learns the principal component of the train set, see the discussions in e.g. [12, 11, 8, 9].

We finally stress that, in contrast to all previous figures, the x axis in Fig. 3.5 is the sample complexity α , rather than the noise level ∆ . As a matter of fact, while our result recover the well-known fact that the MSE of RAEs is limited (lower-bounded) by the PCA test error, Corollary 3.5 allows us to investigate how the RAE approaches the PCA

performance as a function of the number of training samples, a characterization which has importantly been missing from previous studies for non-linear RAEs [8, 9].

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the performance of denoising autoencoders (DAEs) in high-dimensional settings. The DAEs are trained on data sampled from a Gaussian mixture with $K$ components perturbed by isotropic Gaussian noise. The DAEs are one-layer networks with arbitrary activation functions, tied weights, and a skip connection. They are trained with an $L_2$-regularized loss function, and the high-dimensional limit is considered, where the ratio of the number of training points to the input dimension converges to a constant α.\n\nThe main results of the paper are formulas for the denoising test mean squared error (MSE) for the full DAE, as well as two simpler architectures: a \"bottleneck network\" in which the skip connection is removed, and a \"scalar linear network\" that simply rescales the input by a scalar. The final formulas involve complicated integrals and optimization problems, but they only depend on low-dimensional quantities, such as the number of neurons in the hidden layer and the number of components in the Gaussian mixture.\n\nThe authors use the derived asymptotic test error to analyze the role and importance of each component in the performance of DAEs. They find that the skip connection is essential for good performance and that DAEs can outperform other denoising methods, such as principal component analysis (PCA). \n\nTaking the noiseless limit, where the variance of the noise converges to zero, they can also derive the test error of reconstruction autoencoders, which are trained with noiseless inputs.\n\nSTRENGTHS:\nThe authors managed to derive fairly complicated asymptotic formulas via the replica method from statistical physics. The empirical experiments seem to confirm the accuracy of their predictions. This provides yet another example of the successful application of the replica method and its surprisingly powerful nature.\nThis paper fills a gap in the literature and complements previous results that mainly focused on RAEs and the infinite data regime (optimization of the population loss).\n\nThe exact formulas derived in this paper are utilized to derive interesting non-trivial results, which can serve as stimuli for future research. Specifically:\n\na) The MSE obtained by training a DAE with gradient descent matches the asymptotic predictions, suggesting that while non-convex, the optimization problem has a favorable landscape.\n\nb) The exact formulas obtained for the Gaussian mixture predictions align with those obtained for a DAE trained on MNIST datasets, indicating the presence of universality phenomena.\n\nc) As α increases, the denoising MSE of the DAE approaches the asymptotic performance of the oracle denoiser (derived in the appendix).\n\nWEAKNESSES:\nThe paper exhibits weaknesses similar to other papers in this field. The exact analytical formulas derived are somewhat obscure, and it seems that several of the observations made in the paper can be deduced directly from the results of the empirical experiments, without relying on the analytical formulas (e.g. the improved performance of the network with skip connections compared to the one without can be easily inferred). The authors should emphasize the results that their analytical formulas allow to obtain.\n\nQUESTIONS:\n- It appears that all the experiments were conducted with a number of neurons $p = 1$. Could you please clarify if the empirical results align with the replica predictions when $p > 1$? Additionally, does the \"universality phenomenon\" manifest itself also in this case?\n\n- Based on Figure 3, it appears that the performance of the DAE approaches that of the oracle denoiser. Can this convergence be rigorously derived from equation (13)?\n\n- What happens when $K = 1$? Do the DAE, PCA, and bottleneck network exhibit similar performance in this scenario?\n\nLIMITATIONS:\nThis theoretical paper does not possess any immediate potential negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
The authors consider a two layer weight-tied denoising autoencoder with a skip connection in the regime of vanishing rate and samples proportional to the dimension. They heuristically derive an exact characterization of the optimal network parameters and the corresponding network performance in the the high-dimensional limit using the replica method. Their experiments show that their results match on synthetic data and are very close to the performance on more practical data.

STRENGTHS:
- Exact characterization of all quantities of interest in the high-dimensional limit. 
- Strong experimental validation of theoretical claims

WEAKNESSES:
- From a theoretical perspective there is a lack of rigor to this method

QUESTIONS:
- Why do the limits in Line 152 exist? 
- In Figure 3 for high noise and low sample complexity the performance of the full DAE is worse than simply rescaling. Since the full network can also act as a rescaling network by setting $w=0$, this implies that during training the global optimum is not found. Can you elaborate on this?

LIMITATIONS:
- It should be stated more clearly that the way the replica method is carried out only serves as a strong heuristic and not a rigorous proof.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents theoretical results of the test error of 2-layer denoising auto-encoder. In a high-dimensional limit regime where data distribution follows from Mixture of Gaussian, closed-form expressions of the error are obtained. The results are further analyzed and supported by numerical experiments on real data sets.\n\nSTRENGTHS:\n- The main result (result 3.3) is highly non-trivial and provides a tight formula for the test error of the 2-layer denoising auto-encoder (DAE). \n- It shows that DAE can perform much better than PCA in terms of MSE error. \n- The role and importance of the skip connection is also highlighted in both theory and in practice, as well as the non-linearity in DAE.\n\nWEAKNESSES:\n- The optimal MSE error mse_o grows with the data dimension d (eq. 9), however the gap between mse_f and mse_o (eq 8) remains a constant, this suggests that the difference between DAE and PCA is not so significant in terms of the relative error of MSE, i.e. | mse_f - mse_o | / mse_o vs. | mse_PCA - mse_o | / mse_o. Thus it is still not very clear whether DAE brings something essentially different to PCA or not (I agree that at least they are different). \n- In the main result 3.3, it is not clear how the regularization term g is used, i.e. under Assumption 3.2, the parameter lambda does not appear in eq. (8), this is a bit strange to me .\n\nQUESTIONS:\n- (line 33) Usually we say the sample complexity is related to n, it is here a bit strange to say it is alpha \n- (line 89) What does this mean sigma(.) stays order 1 ? \n- (eq 13) is there a unique solution in this system of 10 equations of 10 variables ?\n- (line 219) regarding the Gaussian universality, do you need to perform any normalization on the data for this to hold ? The dimension of MNIST seems quite small, what is it in your mind for the universality to hold ? \n- (line 255) to clarify a previous point regarding PCA, does mse_f grows linearly with d ?  Do you have an idea of what is the relative error of MSE of DAE and PCA ?\n\nLIMITATIONS:\n- The main result is very specific, but this is quite normal in the literature .", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors set out to characterize the non-linear behavior of denoising auto-encoders (DAEs), for Gaussian mixtures, in the high dimensional limit with the number of hidden units being fixed. The authors particularly tease out the role of the skip-connection, compared to the reconstruction auto-encoder (RAE) which is known to essentially perform principal component analysis (PCA).\n\nUsing the replica method, the authors obtain closed-form expressions for the mean-squared error (MSE), as well as the cosine-similarity (w.r.t cluster means). The obtained formulae are supported by experiments on both synthetic and real data sets, clearly highlighting the role of the sample complexity and the noise level.\n\nSTRENGTHS:\n- Presents an effective characterization of (non-linear) DAE behavior on Gaussian mixtures, clearly explaining the role of the reconstruction and scaling components supported by compelling empirical evidence (also on real data).\n- Draws a number of important conclusions, pointing to fruitful directions of future work, e.g., L200, L218, L278.\n\nWEAKNESSES:\nOne would wish the long sequence of mathematical expressions could be made less opaque.\n  - This is remedied by a seemingly complete and well-composed supplementary.\n\nQUESTIONS:\n**Technical comments:**\n- L178: could you briefly explain the asymptotic dependence on $\\alpha$?\n- L212: could you elaborate on the choice of closely related classes?\n- I wonder how the method fairs with a mixture of more than 2 classes.\n  - Is it true that $K=2$ also for the MNIST experiments?\n\n**Presentation comments:**\n- L158: perhaps it helps to discuss this rationale before stating the assumptions.\n- Understandably, it is an essential part of the contribution to provide those closed-form expressions, but I wonder which subset of the readers would benefit from having Eqs.13 and 14 in the main paper.\n\nLIMITATIONS:\nDedicating more space to help make the theoretical concepts and techniques employed more accessible, i.e., to a broader set of readers, would have been greatly appreciated.\n\nIt is a limitation of this reviewer that I'm unable to assess the full scale of the theoretical derivations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}