Unveiling Multiple Descents in Unsupervised Autoencoders
Authors: Kobi Rahimi
Yehonathan Refael
Tom Tirer
Ofir Lindenbaum
Venue: Accepted by TMLR ()

Abstract: The phenomenon of double descent has challenged the traditional bias-variance trade-off in supervised learning but remains unexplored in unsupervised learning, with some studies arguing for its absence. In this study, we first demonstrate analytically that double descent does not occur in linear unsupervised autoencoders (AEs). In contrast, we show for the first time that both double and triple descent can be observed with nonlinear AEs across various data models and architectural designs. We examine the effects of partial sample and feature noise and highlight the critical role of bottleneck size in shaping the double descent curve. Through extensive experiments on both synthetic and real datasets, we uncover model-wise, epoch-wise, and sample-wise double descent across several data types and architectures. Our findings indicate that over-parameterized models not only improve reconstruction but also enhance performance in downstream tasks such as anomaly detection and domain adaptation, highlighting their practical value in complex real-world scenarios.

Published in Transactions on Machine Learning Research (09/2025)
Unveiling Multiple Descents in Unsupervised Autoencoders
Kobi Rahimi
kobérahimmi@gmmailcccm
Faculty of Engineering, Ber-lln University, Romat-Gan 5290002, Israel.
Yehonathan Refael
refuelkalim @mail. tau.aa.
Faculty Engineering, Tel Avic University, Tel Aviu 6997801, Israel.
Tom Tirer
abiu.ac.i
Faculty of Engineering, Bar-Bon University, Romat-Gan 5290002, Israel.
Ofir Lindenbaum
lindeninnnn @biu. ac.
Facully of Engineering, Bar-Ilan Universily. Ramat-Gar 5290002, Israe..
Reviewed on OpenReview: https: // openreveee..........mmm.mm. net/ ffr/m/rrmm id= Fd=FqfDssuuna
Abstract
The phenomenon of double descent has challenged the traditional bias-variance trade-off in supervised learning but remains unexplored in unsupervised learning, with some studies arguing for its absence. In this study, we first demonstrate analytically that double descent does not oceur in linear unsupervised autoencoders (AEs). contrast, we show for the first time that both double and triple descent can be observed with nonlinear AEs across various data models and architectural designs. We examine the effects of partial sample and feature noise and highlight the critical role of bottleneck size in shaping the double descent curve. Through extensive experiments on both synthetic and real datasets, we uncover model-wise, epoch-wise, and sample-wise double descent across several data types and architectures. Our findings indicate that over-paree sanz-parmmtterieed models not only improve reconstruction but also enhance performance downstream lasks such as anomaly delection and domain adaptation, highlighting their practical value in complex real-world scenarios.
1
Introduction
In recent years, studies have shown that over-parameterized models outperform smaller models in generalization
Krizhevsky et al. 2012 He et al. 2016), challenging the traditional bias variance tradeoff (Hastie et al. 2009).
This explained by the double descent phenomenon, extensively studied in supervised learning (Belkin
et al. 2019 Nakkiran et al. 2021 Dar al.] (021). However, double descent has yet be demonstrated fully unsupervised sctting, with one study arguing for the absence of the phenomenon in unsupervisod autoencoders (AEs) (Lupidi al. 2023).
In this study,
AEs to explore double descent and its impact on key unsupervised tasks like domain adaptation, anomaly detection, and robustness to noisy data, AEs are widely used in unsupervised tasks such as denoising [Vincent et. al.] 2008 2010), manifold learning (Wang et al. 2014 Duque et al. 2020), clustering
[Song et al.) 2013 Yang et al. 2019), anomaly detection (Sakurada Yairi 2014 Zhou & Paffenroth 2017)
feature selection Han et al. 2018 Gong et al., (2022), domain adaptation (Deng et al. 2014 Yang al.
2021), segmentation (Myronenko) 2019 Baur al.) (2021). and generative modeling (Kingma 2013 Doersch
2016), making them prominent.
when studying double descent in unsupervised learning.

---

Published in Transactions on Machine Learning Research (09/2025)
We start by analyzing linear AEs, whose lack of nonlinear activation functions allows for theoretical exploration. We prove and empirically confirm that double descent. does not occur in linear unsupervised Afs. We then investigate nonlinear AEs and find that multiple descents emerge across different data models and architectures.
We provide extensive evidence showing the phenomenon in unsupervised nonlinear AEs trained on contami- nated data, with "memorization" causing overfitting and performance degradation. However, we find that weer-parametriiee models can extract the true signal, leading to a second descent and improved performance.
Our experiments focus syuthetic and real-world datasets under various contamination scenarios, including noise, domain shifts, and outliers. Additionally, we identify model, epoch, and sample-wise double descent, highlighting the bottleneck size's role in shaping the double descent curve.
These findings provide critical insights into the dynamics unsupervised learning and the interplay between model capacity, noise, and performance. The phenomenon shown Figure [] illustrating the bias-variance tradcoff, the critical regime, and the second descent induced by wyr-parametrization of unsupervised AEs.
Our findings have important implications for key tasks in unsupervised learning. Specifically, show that peeepparaeteriieed models adapt better to target domains when trained source domains despite domain shifts. We also reveal mon-oonoto performance in anomaly detection as model complexity increases, underscoring the practical relevance of our results, particularly when addressing outliers and domain shifts.
Our main contributions are summarized below:
(a) To the best of our knowledge, this is the first demonstration of model and epoch, and sample wise double descent in a fully unsupervised setting. We analyze and contrast the behavior of linear and nonlinear AEs through analytical and empirical evaluations, highlighting their distinct dynamics.
(b) We analyze various factors influencing the presence of double descent, including sample and feature noise, domain shifts, anomalies, and architecture design (bottleneck size).
(c) We show that double descent reconstruction loss of AE causes non-monolonic performance real-world, downstream tasks like domain adaptation and anomaly detection.
2
Related Work
Most research on double descent has focused on supervised settings. Model-wise double descent was demonstrated in (Spigler et al.] 2018), while (Li et al. 2020 Bartlett et al. 2020 Nakkiran et al. 2021 Gamba al. 2022; Hastie et al. 2022) explore the impact of feature, label noise, and Signal-to-Noise ratio
(SNR) on the double descent curve. Nakkiran et al. 2021 Dubova 2022 Gamba et al.
2022 Kausik et al.
2023 Sonthalia & Nadakuditi
2023 demonstrated epoch- epoch-wise and sample-wise double descent, and multiple
descents were discussed in (Adlam & Pennington 2020 Liang et al. 2020 Chen et al. 2021
While double descent. is well-studied in supervised set Lings, its presence in unsupervised tasks is less understood.
For example, (Lupidi et al.) 2023) argued that model-wise double descent does not occur in unsupervised AEs, and (Gedon et al. 2022) found it absent in principal component analysis (PCA) (Shlens) 2014b). In
contrast, supervised PCA tasks like principal component regression (PCR) (Massy) 1965) show evidence
Variance
Regime
1.0
Critical Regime
Demonstration of double descent phe- nomenon with unsupervised AEs for the "sample noise" sccnario (Subscction 3.1). We present the test loss for varying epochs and hidden layer sizes.
Figure 1:
Test Loss
0.8
150
Epochs
0.6
Intoroclaiion Threshold
100
0.4
50
0.2
100
200
300
400
500
Hidden Layer Size
2

---

Published in Transactions on Machine Learning Research (09/2025)
of the phenomenon (Xu & Hsu 2019 Teresa et al. 2022). highlighting key differences and challenges in observing double descent in unsupervised settings.
In this work, we study AEs' unsupervised objective of minimizing reconstruction error, ([) - x æ where æ is the imput and ((x) the AE's output. This learning framework, which has not previously shown double descent, falls outside the settings explored prior studies. We find that linear AEs lack double descent, while nonlinear AEs show double and triple descent, underscoring the role of nonlinearity. Our analysis spans model, epoch, and sample levels, exploring the impact of bottleneck size, hidden layers, noise magnitude, and sample contamination. We also demonstrate double descent in real-world unsupervised scenarios like domain shifts, anomalics, and additive noise, with implications beyond reconstruction tasks.
3
Data Model
This section details the training data, testing data, and contamination models used to study double descent.
3.1 Subspace Data Model
We revisited the model used @uppdd et al. 2023 which argued that "double descent does not occur ((O,IJ) [a)), representing latent features and embedded them to higher-dimensional space using H of size D x d, (D d, H ~NO, Our dataset differs from (Lupidi et al. 2023 and explore four scenarios in our study:
self-supervised settings." We sampled i..dd Gaussian vectors of size d (z, ~ (0,
Sample Noise. We study how the proportion of noisy training samples (p), affects the test loss curve. Unlike Lupidi et al. 2023), which adds noise to all samples, we vary and use different SNR values (Appendix A Table This gives the following equation for the training samples:
[HH + Ei, with probability P, with probability 1-pp
a
Dx1
(1)
[HH,
where 6; (0, represents noise added to samples with probability p and controls the SNR.
Feature Noise. We study the impact of noisy training features the test loss curve by selecting the same [D.p] pl features to be noisy across all samples. This simulates D p] unreliable or noisy measuring tools.
When noise added to the data, we quantify the ratio between signal and noise power using the SNR. A high SNR indicates high-quality (clean) data, whereas a low SNR corresponds to lower-quality (noisy) data, where the noise dominates. We report SNR values in decibels, defined SNR [dB] 20 log [[[))HH22|la) To ensure the training data is sufficiently noisy, consider cases where SNR < 0, meaning the noise power
A|||e|a
Loss
Loss
sidden
Hidder Layer
Output
0.8
2.
Liye!
Hidden Layer size
70"
0.8
Hiodon eyer
B
1.0
10"
6
0.2
10
15
75
30
35
40
45
10
15
20
25
30
35
40
45
Encoder
Decoder
Embedding Layersinn size
Embedding Layer size
(a) The FCN AE model structure.
(b) Linear AE models.
(c) Nonlinear AE models.
Figure 2: (a): Illustration of our AE model. The embedding layer serves as a bottleneck when the hidden layers are larger. (b), (c): Test losses of lincar and nonlinear AEs for different hidden and embedding sizes. Linear AEs do not exhibit double descent, whereas in nonlinear AEs, we clearly see the phenomenon when varying both the hidden layer and embedding layer size. Train losses Iilustrated in Appendix D Figure T8 AEs were trained on the subspace data model (Subsection 3.1 with 90% sample noise and SNR -15 dB.
3

---

Published in Transactions on Machine Learning Research (09/2025)
exceeds that of the signal. The SNR calculations for both sample and feature noise scenarios presented in Appendix B and visualization of the data generation is illustrated in Appendix A F Figure
14
Domain Shift. We analyze test loss behavior under domain shifts by projecting the test and train latent features using H and H" respectively, with shift modeled as H = + s. H where Ha (N(00,1 adds perturbations, and 8 controls the shift,
Hz,
if train,
2; = H 24 if test.
This data model is illustrated in Appendix A Figure
15
Anomalics. We examine how train-set anomalics affect the test loss curve. Normal samples are {8Hz, N. 1 1=1
while anomalies follow Ip). The Aogaal-ooAnomalle Ratio (SAR), controlled by 3, sets their magnitude ratio. We replace p- 100% of normal samples with anomalies. Illustration shown Appendix A Figure 16
We additionally provide results for nonlinear subspace data model across all the aforementioned contamina- tion scenarios (i.e., sample and feature noise, domain shift, and anomalies) in Appendix F.4
In the following subsections (i..., 3.2 3.3
we present real-world, high-dimensional datasets from diverse domains, trained on different architectures This setup allows us to demonstrate the generalization of the double descent phenomenon across both datasets and model types. Due to their high dimensionalityy these datascts are well-suited for dimensionality reduction via AEs, as also explored in prior work (Eraslan al., Yan al. 2016), (Wang et al., 2016), (Meng al. 2017).
2019),
3.2 Single-Cell RNA Data
We used single-cell RNA data from (Tran et al. 2020 demonstrate our findings in challenging, high- dimensional real-world setting. The dataset was chosen due to its inherent domain shifts, which include 5 distinet domains (biological batches) stemming from differences in laboratory conditions and mcasurement technologies. This makes it particularly suitable for testing our claims about double descent under real-world distribution shifts, which are common in biological and medical data (Haghverdi et al.] 2018 Shaham et al., 2017). Each batch represents 15 different cell types, where each cell (sample) in this dataset contains over 15,000 genes (features), making it a high-dinensional dataset. Due to the real-world nature, we cannot control the shifts between the training (source) and test (larget) datasets. This dataset used for sample noise, feature noise, and domain shift experiments.
3.3 CelebA Data
The CelebA dataset [Lu et al. 2015 was selected evaluate the impact of model complexity on unsupervised anomaly detection performance (Han et al.] (022).. CelebA provides a rich, real-world setting with data attributes, making it particularly suitable for testing whether model complexity translates into non-monotonic trends in downstream tasks such as anomaly detection.
3.4 MNIST and CIFAR-10
We employed the MNIST (LCunn al. 1998) and CIFAR-10 (Krizhevsky al. 2009) datasets to evaluate our findings on standard image benchmarks, demonstrating double descent under sample noise, feature noise, and domain shift scenarios. These datasets also support reproducibility across different model archilectures and data domains.
For more information about the noulinear data model, single-cell RNA, celebA, MNIST, and CIFAR-10 datasets, refer to Appendix A
4 Model Architecture
This section details the model architectures used to study double descent.
4

---

Published in Transactions on Machine Learning Research (09/2025)
4.1
Fully-connected Neural Network (FCN)
We used FCNs (see architecture in Figure 2(a) to study double and triple descent across all mentioned data models. FCNs are widely used with tabular data and have recently demonstrated state-of-the-art results in
various tasks (Gorishniy et al. 2024a b Shenkar & Wolf 2022 Rauf et al. 2024 Svirsky & Lindenbaum)
4.2 Convolutional Neural Network (CNN)
To illustrate the generality of our findings, we additionally present double descent phenomena using CNNs (architecture provided in Appendix [ Figure 17) trained image datasets. A subset of the results is included in Subsection 6.1 while the majority, along with further experimental details, can be found in Appendix F.3 The results in the main paper for this architecture also include the training loss, whereas all other figures in the main paper, based on FCNs, show only test losses, with corresponding training losses provided in the Appendix. Across all experiments involving CNNs, the training loss consistently shows a monotonic decrease, similar to the other settings with FCNs.
5 No Double Descent in Linear AEs
This section presents theoretical analysis and empirical evidence showing that linear AEs do not display the double descent phenomenon. These results, along with our subsequent empirical findings regarding multiple descents in nonlinear AEs, highlight the crucial role of nonlinearily in shaping the test loss curve.
Notation and setup. Consider a general linear AF,, $):;0): TR. R.D,, which consists of I. 2 2 layers and is parameterized by 0 { (wrrx .....y :...,."._, 1xdz 2, w. w. "]. Here, W, represents the weights parameters associated with the i-th layer, for i [L]. For simplicity, we denote D {d: I [L]} (including d,. dg D). Accordingly, let Formally, the model 4 is given by, ((x; 0) = WWW--1" W.W...
min d: to be the bottleneck of the AE, namely, Vi E [4]. di, and S D.
Training the AE is based on empirical risk minimization:
N min 2(0: = min
N [xx;; 0) =x;l?3
According to the formulation of noisy training samples in [ let Pa denote the distribution from which the 2;'s are drawn, i.e., H; ~ Pa. Similarly, let Pa represent the distribution from which the non-noisy evaluation samples 2t's are drawn (see the bottom part of (1). Then, for the trained (æ;; fore), where gopt argminng N Ei= a(), we formalize the generalization error (9) on the distribution of the clean data as:
C 0or Pa) = Ep. (;; 00rt) - =
Note that the error depends on the AE architecture, which is characterized by the set of dimensions D. Theoretical results. The following theorem shows that double descent does not occur in linear ALs, We use the above notations, where a subscript (a or b) distinguishes between different AEs.
.NN) ) E RDNN be the training set, i.idd sampled that the number of samples N is large enough such that rank(X) = D almost surely.
Assumption 5.1. Let X = (æ1,
.2, 02.
Assume
Theorem 5.2. Let (a(:)), (p(:;0) RDD + RDD be two linear AEs, each with L > layers and orchitectures defined by DD and Do, with bottleneck sizes ma and my, where Ma < mb I D. Let 0 and g be the optimal paramelers under assumplion 5.1 Let denole the dimension of the support of Pa (the dimension of the span of the samples ~ Pa). Then, the following holds,
g (DDn: .Ps) (D( Pa).
with equality if: (1) m. ma, even if D., + Do, or (2) Maz my D.
5

---

Published in Transactions on Machine Learning Research (09/2023)
Lott
0.
0.
45%
10% vendercte
50%
79%
52%
1.00
122
90% earple rolu
50% sample robu
70% sample roduc
nample
samplencie
0.00
02
sayple pappeesi
06
1056
1000
0.00
20
Layar
(a) Subspace model. SNR = -55 dB.
(b) Single-cell RNA. SNR = -17 dB.
(c) MNIST SNR = -17 dB.
Figure 3: Model-wise double and triple descents for varying sample noise. Train losses are depicted in
Appendix D Figure 19
Recall that the models are trained using noisy data sampled from Pa and tested clean data drawn from Pa to see if there is noise memorization (high lest loss) or signal learning (low test loss).
The proof of Theorem 5.2 can be found in Appendix This result shows that the G (test loss) decreases as the bottleneck size m increases. Interestingly, it implics that the 9 remains unchanged if the bottleneck dimension preserved, even if the size of other layers increases. Since the proof does not depend on the similarity between Px and Pæ: the theorem holds for scenarios where P # Pa, including the cases of domain shifts and anomalies discussed in Subsection 3.1
Empirical results. To validate our theory, used the data model in Subsection 3.1 and an FCN AE without nonlinear activations (Figure [[a]] Training linear AEs of varying sizes (Figure 2(b) shows that test loss decreases as bottleneck size increases and is unaffected by non-bottleneck layer size. These results align with our theorem, explaining the absence of double descent in linear AEs.
6 Double Descent in Nonlinear AEs
We demonstrate the double descent phenomenon using FCN undercomplete AEs (Figure ((a) minimizing mean squared error (MSE) between inputs and outputs. Results are based on 5 to 15 random sceds, with implementation details in Appendix A We trained models on contaminated datasets, drawn from Pa and tested on clean datasampled from Pa to isolate the effect on the learned model, showing noise memorization (high test loss) versus signal learning (low test loss) and observed model, epoch, and sample-wise multiple descents. Unlike standard analyses of PCA (Josse & Husson 2012 Minka 2000), where both training and test data are typically noisy and the goal is to recover the underlying signal by attenuating noise-dominated components, our setup deliberately uses clean test data to isolate the effect of noise memorization in the learned model. Train loss results of the FCNs are detailed in Appendix D with additional findings on nonlinear synthetic datasets in Appendix F.4 We also present double descent results for CNN AE models
(Appendix A. Figure 17 in Subscction 6.1 Figures
3(c)
4(c)
5(b] and in Appendix F.3
1.50
158
S4R D/33
BVR- db
Tost Loog
Tese Loen
1.00
SYR 21 dB
6/00
5.75
025
1900
Hidden
Size
Hidden S.50
Crannelo
(a) Subspace model. Sample noise 80%.
(b) Single-cell RNA. Sample noise 40%
(c) MNIST. Sample noise 50%.
Figure 4: The effect of SNR (sample noise case) on the test loss curve. Train losses are illustrated in
Appendix D Figure 20
6

---

Published in Transactions on Machine Learning Research (09/2025)
3.0
target (shift 0.1)
target = MNIST-M
2.5
target (shift 0.5)
2.0
target (shift 1)
Test Loss
target (shilt 2)
Test
1.5
target (shilt 3)
target (shift 4)
20
40
60
100
10
20
30
50
Hidden Layer Size
Channels
Subspace data model.
(b) MNIST.
Figure 5: Model-wise non-monotonic behavior and double descent for varying domain shifts. Appendix
Figure 21 shows the train loss behavior.
6.1 Model-Wise Double Descent
This scction analyzes test loss with increasing model sizes, breaking down the double descent phenomenon into hidden-wise and bollleneck-wiiee variations and showing their impact on test loss. We find that various contaminations (Section 3) manipulate the interpolation threshold location and value. Double descent is more common with severe contaminations, as models in the critical regime interpolate noise instead of learning the signal, leading to higher test loss.
Figure [(c) illustrates double descent in nonlinear AEs, concluding that nonlinear activations cause this phenomenon in unsupervised AEs. We define bottleneck-urise and hidden-wise double descent distinguish between various model sizes and highlight the significance of our architectural choices. FCN undercomplete AFs (m < D 50) have a bottleneck set by the smallest layer, with the critical regime and second descent. (improving reconstruction) occurring when hidden layers exceed this size (above the white dashed line). Increase in bottleneck shifts the critical regime. Hidden layers above the dashed line set the embedding layer as the bottleneck, while smaller hidden layers become the bottleneck (Figure 2(a) FCN overcomplete AEs (m 2 D) trivially learn the identity function and excluded from this study (see Appendix F.2 Figure 34
Sample and feature noise. Figure ((a) (FCN) shows that increasing sample noise raises test loss, as higher noise levels dominate training and degrade performance. Higher noise levels also shape the double descent curve by shifting the critical regime to larger models, which are needed to memorize more noise. At low noise levels such as 0 20% sample noise, double descent is absent due to insufficient noise in the training data, making models in the critical regime learn the signal and preventing test loss increase. This underscores noise as a key factor in the phenomenon. Figure 3(b) (FCN) demonstrates triple descent in single-cell RNA data, with similar test loss behavior across both interpolation phasos. Appendices F.3 provide additional evidence double descent with feature noise. Figure 3(c) presents double descent results for CNNs trained on the MNIST dataset. Additional results of CNNs trained on MNIST and CIFAR-10 datasets and further details regarding these experiments
presented in Appendix F.3
We also observed sinal ascent, where test loss increases after double descent, mirroring patterns seen in supervised learning (Xue et al. 2022). This is demonstrated for single-cell RNA data with 0- 20% sample noise (Appendix F.5 Figure 56(b).. To demonstrate the generality of the phenomenon, we also report double and triple descent under Laplacian noise and sparse AEs (Appendix F.6).
SNR. Figure shows that SNR plays a key role in shaping test loss and revealing double descent. Negative SNR lets noise dominate the training set, causing memorization in the critical regime and exposing double descent. As SNR. increases, noise influence diminishes, allowing better signal learning and lowering test loss. At SNR 0 dB (Figures 4(a) (c)).. double descent is absent or barely recognizable because noise is not dominant enough to trigger memorization.
Domain shift. Our experiments reveal non-monotonic behavior in test loss under domain shifts for the subspace data model using FCNs (Figure 5(a)). As the shift increases, test loss rises, but low shifts (0.1, 0.5) show no non- non-monotyyiccty due strong cource-target alignment, allowing interpolating models perform well. Over-parameteriize models further improve target. reconstruction by reducing test. loss. We also present.
7

---

Published in Transactions on Machine Learning Research (09/202))
Critical Regime
aramily samples 70%
anomaly samples 70%
Clean Tost L203
samplea
Aetomat Teen Love
anomaly comples 80%
anomayy camples
ancmady sumples
BOG
ancmaal
ancomaly sumples
002
anomaay comples
0.7
30g
220 Hiddwn Dimension
100
Hiddon Dimersion
Hidden Dimeesionn Size
(a) Syuthetic anomaly data with SAR = -15 dB.
anoraaly sumples = 70%
anomall samples = 70%
1.30
Olaan Tost Lots
80%
0.
28
sampla
6.0
Acomaty
24
ROC
sommly samphes 70%
033
anotslly
Hidden Dimension Sizo
Hiddon Dimension Stoe
Hiddon Dimrension Sizo
(b) Synthetic anomaly data with SAR dB.
Figure 6: Left column: test loss of the clean (normal) samples. A double descent pattern emerges for low SARs and high anomaly presence in the training data. Middle column: test. loss computed only over the anomalous samples. Right column: Non-monotonic behavior of the ROC-AUC, The level of SAR, illustrated in (a) and (b), influences the occurrence of double descent in the test curve.
double descent behavior when CNNs are trained on MNIST and tested on the MNIST-M dataset in Figure Subsection 7.1 F Figure 12 demonstrates double and triple descent in single-cell RNA data under real domain shift, reinforcing the phenomenon's 's presence with real-world evidence.
5(b)
Anomalies. For the first time, we show that anomalies can cause double descent in test loss and non- monotonicity in anomaly detection (Cheng et al., 2021). Using data from 3.1 we analyze test loss and detection quality via receiver opcrating characteristic area under the curve (ROC-AUC) (Hanley & McNeil Fawcell 2006), which distinguishes clean (low reconstruction error) from anomalous samples (high MMalhotra et al. 2016 Borghesi et al. 2019 Lindenbaum et al. 2024h. In the critical regime (Figure anomaly memorization increases test loss for clean samples and reduces ROC-AUC, Larger models exhibit secondary descent, especially with low SAR and many anomalies, improving anomaly detection to match under-paaaameterizee models while lowering test. loss (Lerman & Maunu 2018b a Han et al. 2022 Lindenbaum et al. 2024). Figure 6(b] shows no double descent at high SAR, similar to previous results where contamination is not significant, yet critical regime models still perform the worst. Additional real-world insights are in Subsection 7.2 In this case keep Pz and Pa the same to evaluate model performance both clean and anomalous data to generate the ROC curve and compute the AUC, providing mcaningful metric for assessing anomaly detection and demonstrating the relevance of our findings to downstream tasks.
1982
error
6(a)
In conclusion, factors such as model nonlinearity, bottleneck size, and severe contaminations, like high noise levels, large domain shifts, many anomalics, and low SNR, affect double descent and incrcase test loss, occasionally shifting the critical regime.
6.2 Epoch-Wise Double Descent
Inspired by the surprising results in Nakkiran et al. 2021 which demonstrate epoch-wise double descent in a supervised setting, we study the existence of this phenomenon in our unsupervised framework. Figures 7 and 20 (Appendix E show how noisy samples (and fcatures) affect the test loss, respectively. Higher noise or lower SNR increases test loss, as shown in Figure and Appendix E. Figure 30 The phenomenon also occurs with domain shifts (Figure 9] where stronger shifts lead to higher test loss, shown in Figure [(a)
8

---

Published in Transactions on Machine Learning Research (09/2025)
6.3 Sample- Wise Double Descent
This section examines how the number of training samples affects the test loss. Model complexity and sample size determine whether a model is over- or under-parameterived, shifting the interpolation threshold, as shown in Figure 10 This shift can sometimes make larger training sets lead to worse performance than smaller ones, a phenomenon also observed in supervised sellings by (Nakkiran et al. 2021
We analyze the effect of increasing training samples while keeping model size fixed and observe non non-monotonic trends in the test loss curve (Figures 11(b) [[(c)] and Appendix Figure 31). In some cases, this leads double descent, additional results using more complex data models found in Appendices F.3 and F.4 These results confirm
shown in Figure [a))) Appendix [ and Figure 26 present the training losses, along with
the effects of noise, SNR, and domain shift, which are consistent with the findings in Subsections 6.1 and 6.2
7 Real World Applications
This section highlights the application of our findings to critical tasks in unsupervised learning, such as domain adaptation and anomaly detection, focusing on the importance of model size selection in AEs rather than competing with state-of-the-art methods for these tasks.
7.1 Domain Adaptation
Domain shifts common in machine learning, where differences between training and testing distributions can degrade performance unseen data. Various domain adaptation methods (Peng et al. 2019 Chang
et a..
2019 Zhou et al. 2022 Rozner et al. 2023 Yampolsky et al. 2023 aim to minimize this shift. In biology, this challenge, known as the "batch effect", arises when integrating datasets collected under different. conditions (Tran et al. 2020
This section explores the relationship between model size and its ability to address distribution shifts in real-world single-cell RNA data (Subsection 3.2) We demonstrate the benefits of wee--paamettriize models in unsupervised tasks under real domain shifts, revealing multiple descents. Related findings in supervised learning were shown by Tripuraneni et al. 2021 l and (Kausik et al. 2023) under different assumptions.
Appendix F. Figure b2(b) visualizes source and target datasets using UMAP embeddings (MeInnes et al, 2018). Figures 12(a)
12 show test and train losses for models trained on source and tested on target datasets, where testing on the 'Wang' dataset revealed triple descent.
Since we are working with a real-world dataset, we do not have control over Pa and Pa. In particular, we cannot control the distributional shift between the training and test sets. We used KL-divergence (KLD) [Shlens] 2014a) to quantily the distribution shift between the source and larget dalasets, as shown in the test Joss legend in Figure [2(a) Higher KLD values correspond to larger shifts and increased test loss, consistent
with the simulated results in Figure 5(a)
20% sample noise
1.10
70% sample naise
30% sample naise
80% sample noise
40% sample noise
Test Loes
1.06
90% sample noise
Test
0.04
0.03
0.02
0
25
50
100
125
150
175
200
200
400
600
800
1000
Epochs
Epochs
(a) Subspace data model. SNR -2 dB.
(b) Single-cell RNA data. SNR -17 dB.
Figure 7: Epoch-wise double descent for different number of noisy samples. Train losses are depicted in
Appendix D Figure 22

---

Published in Transactions on Machine Learning Research (09/2025)
0.20
snr 20 dB
snr -18 dB
1.20
Teat Loss
snr -17 dB
0.15
Test
1.10
0.05
1.00
150
175
Epochs
Epochs
(a) Subspace data model. Sample noise = 40%.
(b) Single-cell RNA data. Sample noise = 90%.
Figure 8: Epoch-wise double descent, (sample noise) influenced by the SNR. Train losses are exhibited in Appendix Figure
To assess domain adaptation, we analyzed the AEs embeddings learned the bottlencck layer using a k 10 nearest neighbors domain adaptation lest. (KNN-DAT) (Schilling) 1986), Section 3, measuring domain mixing. KNN-DAT of indicates complete separation and lower values signify better mixing. Improved mixing reflects enbeddings where target samples are more aligned with source samples.
12(() show UMAP representations based on AE embeddings. Under-parameterrieel models achieve better KNN-DAT than critical-regime models but at the cost of poor source data learning (high train loss). Critical-regime models focus on domain-specifie features, leading to higher KNN-DAT. parameterizze models perform best, achieving a KNN-DAT of 0.75, reduced test loss, and improved target data reconstruction. This highlights their effectiveness, ouer-parameterized models facilitate the transition between source and target datasets, serving viable domain adaptation strategy. Additional results for the subspace data model are in Appendix F.1
Figures c
12(d)
7.2 Anomaly Detection
Unsupervised anomaly detection is vital in machine learning, with applications across various fields. AEs are widely used for this task (Chandola et al. 2009: Chen et al. 2018 Rozner et al. 2024 Lindenbaum et al., 2024h. We trained AEs on both normal and anomaly data, detecting anomalies via reconstruction loss as detailed in Subsection ion 6.1 As noled in Subsection 6.1 (the 'Anomalies' section), we align Pa and Pa to assess the model's anomaly detection capabilities in the context of downstream tasks.
We examined how model size impacts anomaly detection using the CelebA dataset (Subsection 3.3 As shown in (Han et al. 2022), small models outperform larger ones, achieving the highest ROC-AUC (Figure With uncontrolled positive SAR, we observed double descent in test loss but identified non-moneeee non-monotonic
[3)
1.6
target (shift 1)
target (shift 2)
2.0
target (shift 3)
Test Loss
target (shift 4)
1.5
1.2
1.0
target Mutaro b2
1.0
target = Segerstolpe
targel XSn_b5
0.0
50
75
100
125
150
175
200
200
430
500
800
1000
Epochs
Epochs
(a) Subspace data model with 5% noisy samples and
(b) Single-cell RNA data. The 'Wang' batch excluded due noisy results.
SNR 20 dB.
Figure 9: Epoch-wise double descent influenced by the domain shift.. In (a) we introduce noise to emphasize the double descent curve. Train losses shown in Appendix D Figure 24
10

---

Published in Transactions on Machine Learning Research (09/2025)
1.
200 samples
5000 samples
Figure 10: In the yellow interval, more data hurts performance as using 10000 samples worsens test loss compared to 5000. We used the subspace data model with sample noise of 70% and SNR of -15 dB, Train loss results are detailed in Figure 25
10000 samples
Loss
1est
Appendix D
100
300 Hidden Layer Size
200
500
ROC-AUC behavior. Thus, intermediate models should be avoided, as their anomaly detection performance lags behind under and over-parameterized models.
8 Discussion
In this section, we identify and highlight several fundamental distinctions between double descent behavior in supervised learning and the unsupervised setting explored in this work.
Noise structure and data model. In supervised learning, the input æ is typically clean, while noise is introduced in the labels y. contrast, our AEs operate under an unsupervised reconstruction objective where the input serves as both source and target. Consoquently, any corruption affects both sides of the learning signal (see last. part of Section 2 and Section 3 inducing distinct. generalization dynamics not. observed in supervised settings.
Presence of double descent in linear models. Double deseent has been extensively observed in supervised linear models such as linear regression (Belkin et al., 2019 Bartlett et al. 2020). However, our theoretical and empirical results demonstrate that linear unsupervised AÉs do not exhibit double descent. This highlights a structural divergence between supervised and unsupervised learning, where nonlinearity becomes essential for the emergence of multiple descent phases.
Training objective learning dynamics. While both supervised models and AEs often minimize the MSE, the training dynamics differ significantly. In supervised learning, the test distribution generally matches the training distribution. In contrast, our approach involves training noisy imputs and evaluating on clean data, creating a distribution shift and affecting how generalization and overfitting emerge during training.
Interpretation over-parameterriztion. In supervised lasks, interpolation typically lied to the total number of parameters. Over- preemearamttiiiiiioonn can be achieved by redistributing capacity across layers without necessarily increasing model size. However, in our setup, achieving interpolation requires joint scaling of the hidden and bottleneck layers (see Figure 2(c) and Subsection 6.1
9 Conclusions
Our study comprehensively studied double descent in unsupervised learning using AFs. Analytically and empirically, we showed that linear AEs lack double descent. In contrast, nonlinear ABs exhibit multiple
(8) -1 (A)
scriple noloo sample rolee 20 % suriple rase
8.0
815 (9l
27
1.
Number of Samples
Number of Samples
Namber of Bamples
(a) sample noise 90%.
(b) SNR -15
(c) Varying domain shifts.
Figure 11: Sample-wise non-monotonicity and double descent for the subspace data model. Feature noise
scenario results are presented in Appendix E
Figure 31 and the training losses in Appendix D Figure 26
11

---

Published in Transactions on Machine Learning Research (09/2025)
target= = Wang. KLD = 0.246
0.7
Baron
target= Mutaro,
0.149
0.8
Test Loss
target= = Xin, KLD = 0.14
Train Loss
target= Segerstolpe, KLD 00133
0.5
6
0.2
2000
450
2000
Hidden Layer Size
Hidden Layer Size
(a) Test losscs of different targets.
Train loss of data.
source (Baron)
target (Mutaro)
target (Segerstolpe)
target (Wang)
target (Xin)
source (Baron)
(Baren)
largel (Mutare)
target (Mutaro)
target (Segerstolpe)
target (Segerstolpe)
target (Wang)
target (Wang)
target (Xin)
target (Xin)
KNN-DAT 0.9
KNN-DAT 0.93
KNN-DAT 0.75 (best results)
Hidden layer size 50.
(d) Hidden layer size 450.
(c) Hidden layer size 2000.
Double descent in real-world domaín shift. (a); (b): Test and train losses utilizing the single-cell RNA dataset. Models are trained source data and evaluated on target data. (c), (d), (e): UMAP of vectors extracted from the encoder's output and KNN-DAT results for different model sizes. As can be seen, Over-parameterized models case adaptation between source and target datasets.
Figure 12:
descents and non-monotonic behaviors across model-wise, epoch-wise, and sample-wise levels for various data models and architectural designs. We analyzed the effects sample and feature noise and emphasized the importance bottlemeck size shaping double descent. Our findings also show that over-parameeterized models enhance both reconstruction and downstream tasks like anomaly detection and domain adaptation, underscoring their real-world relevance. Although size due to computational constraints, we believe the scope and depth of the evaluation offer strong and conclusive foundation. This work raises broader implications for the use of model capacity in unsupervised settings. It suggests that double descent is not unique to supervised learning and must be considered when designing AEs for real-world applications. Several open rescarch questions emerge from our study, including the need for a deeper Aheoretical understanding double descent. unsupervised models, how different.
experiments
conducted
datasets of moderate
ancrely EVTple 10%
anoraly ardes 104
ancrey cample: 10%
TentLose
Snunedy somphs 20%
anorely and 28%
acorndy sorolos 20%
aromedy Somple 30%
anonalyy sendle
snecady comols 90%
wwwmaay andde
anerody comple
wwmmly oordee
ancedsy candees
antrady
wwomaly onrdes
ancresy campee
unorally ando
arerndy surplcy
enonalyy wargee
arersdy
anomaly
ancrady 106 300
100
10
200
250
200
250
40
8
50
150
150
253
23
20
23
40
300
40
Figure 13: Left, middle: test loss of clean and anomaly data. Right: non-monotonie behavior of the ROC-AUC for the celebA dataset.
12

---

Published in Transactions on Machine Learning Research (09/2025)
levels of nonlinearity influence the phenomenon, and the role of architecture-specifec factors in shaping generalization behavior. We hope these findings contribute to the growing understanding of learning dynamics in unsupervised models and inspire future research into both theoretical and practical aspects of model generalization in unsupervised learning framoworks.
Acknowledgment
TT was supported by ISF grant No. 1940/23 and MOST grant No. 0007091. OL was supported by MOST grant No. 0007341.
References
Ben Adlam and Jeffrey Pennington. The neural tangent kernel in high dimensions: Triple descent and a multi-scale theory of generalization. In International Conference on Machine Learning, pp. 74-84. PMLR, 2020.
Peter I Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117(48))30063 30070, 2020.
Christoph Baur, Stefan Denner, Benedikt Wiestler, Nassir Navab, and Shadi Albarqouni. Autoencoders for unsupervised anomaly segmentation in brain mr images: comparative study. Medical Image Analysis, 69: 101952, 2021.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, 116(32): 15849 15854, 2019.
Andrea Borghesi, Andrea Bartolini, Michele Lombardi, Michela Milano, and Luca Benini, Anomaly detection using autoencoders high performance computing systems. In Proceedings of the AAAI Conference on artificial intelligence, volume 33, pp. 9428 9433, 2019.
Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly detection: A survey. ACM computing surveys (CSUR), (3):11558 2009.
Woong-Gi Chang, Tackgeun You, Seonguk Seo, Suha Kwak, and Bohyung Han. Domain-specific batch normalization for unsupervised domain adaptation. In Proceedings of the IEEE/CVF conference on Computer Vision and Pallern Recognition, pp. 7354 7362, 2019.
Lin Chen, Yifei Min, Mikhail Belkin, and Amin Karbasi. Multiple descent: Design your own generalization curve. Advances in Neural Information Processing Systems, 34:8898 8812, 2021.
Zhaomin Chen, Chai Kiat Yeo, Bu Sung Lee, and Chiew Tong Lau. Autoencoder-based network anomaly detection. In 2018 Wireless telecommunications symposium (WTS), pp. 1 5. IEEE, 2018
Zhen Cheng, Siwei Wang, Pei Zhang, Siqi Wang, Xinwang Liu, and En Zhu. Improved autoencoder for unsupervised anomaly detection. International Journal of Intelligent Systems, 36(12):7103- 2021.
Yehuda Dar, Vidya Muthukumar, and Richard G Baraniuk. A farewell to the bias-variance tradeofI? an overview of the theory of overparameterized machine learning. arXiv preprint arXiv:2109.02355 2021.
Jun Deng, Zixing Zhang, Florian Eyben, and Björn Schuller. Autocncoderrbasse unsupervised domain adaptation for speech emotion recognition. IEEE Signal Processing Letters, 21 (1( 1072, 2014.
Carl Doersch. Tutorial on variational autoencoders. arXiv prepriat arXiv:1606.00 050.00006050.0. 2016.
Marina Dubova. Generalizing with overly complex representations. In NeurIPS 2022 Workshop on Information-
Theoretic Principles in Cognétive Systems, 2022.
13

---

Published in Transactions on Machine Learning Research (09/2025)
Andrés F Duque, Sacha Morin, Guy Wolf, and Kevin Moon. Extendable and invertible manifold learning with geometry regularized autoencoders. In 2020 TEEE International Conference on Big Dala (Big Dala), pp. 5027 5036. LEEE, 2020.
Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychometrika, (3):211-218, 1936.
Gökcen Eraslan, Lukas M Simon, Maria Mircea, Nikola S Mueller, and Fabian J Theis. Single-cell rna-seq denoising using deep count autoencoder. Nalure communications, (0(1):390, 2019.
Tom Fawcett. An introduction to roc analysis. Pallern recognition letters, 27( 0774844 2006.
Matteo Gamba, Erik Englesson, Márten Björkman, and Hossein Azizpour. Deep double descent via smooth interpolation. arXie preprint arXiv:2209.1000 2022.
Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, llya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.0400 2024.
Daniel Gedon, Antônio H Ribeiro, and Thomas B Schön. No double descent in pea: Training and pre-trainnnt pre-training high dimensions. 2022.
Xiaoling Gong, Ling Yu, Jian Wang, Kai Zhang, Xiao Bai, and Nikhil R Pal. Unsupervised feature selection via adaptive autoencoder with redundancy control. Newral Networks, 150:87- 101, 2022.
Yury Gorishniy, Akim Kotelnikov, and Artem Babenko. Tabm: Advancing tabular deep learning with parameter-cfficient ensembling. arXiv preprint orXiv:4410224110, 2024a.
Yury Gorishniy, Ivan Rubachev, Nikolay Kartashev, Daniil Shlenskii, Akim Kotelnikov, and Artem Babenko. Tabr: Tabular deep learning meets nearest neighbors. In The Twelfth International Conference on Learning Representations, 2024b.
Lalch Haghverdi, Aaron TL Lun, Michael D Morgan, and John C Marioni. Batch effects in single-cell rna-sequencing data are corrected by matching mutual nearest neighbors. Nature biotechnology, 36(5): 421-427, 2018
Kai Han, Yunhe Wang, Chao Zhang, Chao Li, and Chao Xu. Autoencoder inspired unsupervised feature selection. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp- 2941 2945. IEEE, 2018.
Songqiao Han, Xiyang Hu, Hailiang Huang, Minqi Jiang, and Yue Zhao. Adbench: Anomaly detection benchmark. Advances in Newral Information Processing Systems, 35:32142 15:32142-32159, 2022.
James A Hanley and Barbara J McNeil. The meaning and use of the area under a receiver operating characteristic (roc) curve. Radiology, (43(1): 36, 1982.
Trevor Hastic, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistioal learning: data mining, inference, and prediction, volume 2. Springer, 2009.
Trevor Hastie, Andrea Montanari, Saharou Rosset, and Ryan Tibshirani. Surprises in high-dimensional ridgeless least squares interpolation. Annals of statistios, 50(2):949, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Tn Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770 778, 2016.
Julie Josse and François Husson. Selecting the number of components in principal component analysis using cross-validation approximations. Computational Statistics & Data Analysis, 56(6):1869-1177 2012.
Chinmaya Kausik, Kashvi Srivastava, and Rishi Sonthalia. Double descent and overfitting under noisy inputs and distribution shift for linear denoisers. arXiv preprint ar Xiv:2330.. 17297, 2023.
14

---

Published in Transactions on Machine Learning Research (09/2025)
Kenji Kawaguchi. Deep learning without poor local minima. Advances in neural information processing systems, 29, 2016.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 1rXi::1112.688 2014.
arXiv preprint
DP Kingma. Auto-encoding variational bayes. arXi preprint orXiv:1818.6114, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, lyaa Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document. recognition. Proceedings of the IEFE, 86(11):2278 2224,, 1998.
Gilad Lerman and Tyler Maunu. Fast, robust and non-convex subspace recovery. Information and Inference: A Journal of the IMA, 7(2):277-38) -3:66,,,3,, 2018a.
Gilad Lerman and Tyler Maunu. An overview of robust subspace recovery. Proceedings of the IEEE, 106(8): 1380-1410, 2018b.
Zhu Li, Weijie Su, and Dino Sejdinovic. Benign overfitting and noisy features. arXiv preprint 20x::2008000000 2020.
Tengyuan Liang, Alexander Rakhlin, and Xiyu Zhai. On the multiple descent of minimum-norm interpolants and restricted lower isomctry of kernels. In Conference on Learning Theory, pp. 2683-2711. PMLR, 2020.
Ofir Lindenbaum, Yariv Aizembud, and Yuval Kluger. Probabilistic robust autoencoders for outlier detection. The Conference on Uncertainty in Artificial Intelligence (UAI), 2024.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Alisia Lupidi, Yonatan Gideoni, and Dulhan Jayalath. Does double descent occur in self-supervised learning? arXiv preprint arXt::2307.0787. 2023.
Pankaj Malhotra, Anusha Ramakrishman, Gaurangi Auand, Lovekesh Vig, Puneet Agarwal, and Gautam Shroff. Lstm-based encoder-decoder for multi-sensor anomaly detection. arXi preprint Xiv::6607 160.00017,0,, 2016.
William F Massy. Principal components regression in exploratory statistical research. Journal of the American Statistical Association, 60(30)):2342555 1965.
Leland Mclnnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint wxXi::1822.03220 2018
Qinxue Meng, Daniel Catchpoole, David Skillicom, and Paul J Kennedy. Relational autoencoder for feature extraction. In 2017 International joint conference on neural networks (IJCNN), pp. 364-371. IEEE, 2017.
Thomas Minka. Automatic choice of dimensionality for pea. Advances in neural information processing systems, 13, 2000.
Audriy Myronenko. 3d mri brain turnor segmentation using autoencoder regularization. In Bruinlesion: Glioma, Multiple Selerosis, Stroke and Traumatic Brain Injuries: 4th International Workshop, BrainLes 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Revised Selected Papers, Part II 4, pp. 311-320. Springer, 2019.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Nya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021 (12))(1)112000) 2021.
15

---

Published in Transactions on Machine Learning Research (09/2025)
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Sacnko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 1406 1415, 2019.
Hafiz Tayyab Rauf, Andre Freitas, and Norman W Paton. Tablede: Deep clustering for tabular data. arviv preprint Xrx:224005. 17728, 2024.
Amit Rozner, Barak Battash, Lior Wolf, and Ofrr Lindenbaum. Domain genaii-ggneralizable multiple-domain clustering. Transactions on Machine Learning Reseorch, 2023.
Amit Rozner, Barak Battash, Henry Li, Lior Wolf, and Ofir Lindenbaum. Anomaly detection with variance stabilized density estimation. The Conference Uncertainty in Artificial Intelligence (UAI), 2024.
Mayu Sakurada and Takehisa Yairi. Anomaly detection using autoencoders with nonlinear dimensionality reduction. In Proceedings of the MLSDA 2014 2nd workshop on machine learning for sensory data analysis, pp. 11, 2014.
Mark F Schilling. Multivariate two-samplc tests based nearest neighbors. Journal of the Americon Statistical Association, 81(395):799-80) 806, 1986.
Uri Shaham, Kelly P Stanton, Jun Zhao, Huamin Li, Khadir Raddassi, Ruth Montgomery, and Yuval Kluger. Removal of batch effects using distribution-matchiee residual networks. Bioinformatics, 33(16):2539-25 -546, 2017.
Tom Shenkar and Lior Wolf. Anomaly delection for tabular data with internal contrastive learning. In International conference on learning representations, 2022.
Jonathon Shlens. Notes on kullback-leibler divergence and likelihood. arXiv preprint arXio:1404.2000, 2014a.
Jonathon Shlens. tutorial on principal component analysis. arXiv preprint arXiv: 1 1100, 2014b.
Chunfeng Song, Feng Liu, Yongzhen Huang, Liang Wang, and Tieniu Tan. Auto-encoder based data clustering. In Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications: 18th
Bbrraamericun Congress, CIARP 2013, Havana, Cuba, November 20-23, 2013, Proceedings, Part 18, pp- 117 124. Springer, 2013.
Rishi Sonthalia and Raj Rao Nadakuditi. Training data size induced double descent for denoising feedforward neural networks and the role of training noise. Transactions on Machine Learning Research, 2023.
Stefano Spigler, Mario Geiger, Stéphane d'Ascoli, Levent Sagun, Giulio Biroli, and Matthieu Wyart. A jamming transition from under-to over-parametrization affects loss landscape and generalization. arXiv preprint vrXie:1810.0966: 2018.
Jonathan Svirsky and Ofir Lindenbaum. Interpretable deep clustering for tabular data. In forty-forst International Conference on Machine Learning.
Ningynan Teresa, David W Hogg, and Soledad Villar. Dimensionality reduction, regularization, and general- ization in overparameterized regressions. SIAM Journal on Mathemalics of Data Science, 4(1):126 152, 2022.
Hoa. Thi Nhu Tran, Kok Siong Ang, Marion Chevrier, Xiaomeng Zhang, Nicole Yee Shin Lee, Michelle Goh, and Jinmiao Chen. A benchmark of batch-effect correction methods for single-cell rma sequencing data. Genome biology, 21:1 32, 2020.
Nilesh Tripuraneni, Ben Adlam, and Jeffrey Pennington. Overparameterization improves robustness to covariate shift in high dimensions. Aduances in Neural Information Processing Systems, 34:13883-1356 2021.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Picrre-Antoint Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1103, 2008.
16

---

Published in Transactions on Machine Learning Research (09/2025)
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in a deep net with local denoising criterion. Journal of machine learning research, 11(12), 2010.
Wei Wang, Yan Huang, Yizhou Wang, and Liang Wang. Generalized autoencoder: A neural network framework for dimensionality reduction. In Proceedings of the IEEE conference computer vision and pollern recognilion workshops, pp. 490 497, 2014.
Yasi Wang, Hongxun Yao, and Sicheng Zhao. Auto-encoder based dimensionality reduction. Neurocomputing, 184:232 242, 2016.
Ji Xu and Daniel J Hsu. On the number of variables to use in principal component regression. Advances in neural information processing systems, 32, 2019.
Yihao Xue, Kyle Whitecross, and Baharan Mirzasoleiman. Investigating the impact of model width and density on generalization in presence of label noise. arXiv preprint arXiv:2208. 0000:2208.00003,, 2022.
Tamir Baruch Yampolsky, Ronen Talmon, and Ofir Lindenbaum. Domain and modality adaptation using multi-kernel matching. In 2023 31st European Signal Processing Conference (EUSIPCO), pp. 1285-1289. IEEE, 2023.
Xinchen Yan, Jimei Yang, Kibyuk Sohn, and Honglak Lee. Attribule2inage: Conditional image generation from visual attributes. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11 11-14, 2016, Proceedings, Part IV 14, pp. 776-791. Springer, 2016.
Shuai Yang, Kui Yu, Fuyuan Cao, Hao Wang, and Xindong Wu. Dual-representation based autoencoder for domain adaptation. IEEE Transactions on Cybernetics, 52(8):7464 7477, 2021.
Xu Yang, Cheng Deng, Feng Zheng, Junchi Yan, and Wei Liu. Deep spectral clustering using dual autoencoder network. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 4066-4075 2019.
Chong Zhou and Randy C Paffennoth. Anomaly detection with robust deep autoencoders. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and dala mining, pp. (65-674, 2017.
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. IEFE Transactions Pattern Analysis and Machine Intelligence, 44444 2022.
17

---

Published in Transactions on Machine Learning Research (09/2025)
A Implementation Details
This section provides complete implementation details for all experiments conducted in the paper. Illustrations of the subspace data model generation introduced in Subsection 3. for the scenarios of sample and feature noise, domain shift, and anomalies are displayed in Figures 14
15 and 16 respectively.
24z
Sample roive
Feature
98
Number
features
Number samples
Figure 14: Train samples generation for the scenarios of sample and feature noise with p = 0.5. The first (leftmost) matrix depicts the latent vectors Z. The second matrix illustrates the latent vectors being projected into higher dimensional space, and the rightmost matrices contain clean (blue) and noisy (red) samples features respectively. The test samples remain clean.
HZ Hraia
Z train
Number
Zust
H" l
4eat
Number of samples
Figure 15: Train and test samples generation for the scenario of domain shift. The matrix on the left depicts the latent vectors Z and the two middle matrices represent the separation to source (Ztraia) and target (Zrest). The two rightmost matrices illustrate the latent vectors of the train and test data being projected into a higher dimensional space with different matrices (H, H,),,, resulting in domain shift.
PHZ
Anomaly N(0.1p)
Clean and anomaly samples
Number of features
Mimoer
Camples
Replace 1-00% clea samples with anomalies
Figure 16: Train samples generation for the case of anomalies and p = 0.5. The matrix on the left depicts the clean data, the middle matrix represents the anomaly samples, and the rightmost matrix contains both clean (blue) and outlier (red) samples. Test samples remain clean.
and other variables for the training process with the subspace data model, nonlinear subspace model, (Appendix F.4), single-cell RNA, CelobA, MNIST, and CIFAR-10 (Appendix F.3h datasets. The training optimizer utilized was Adam [Kingma & Ba 2014). and the loss function for reconstruction is the mean squared error, which mentioned in this Section.
Parameters. Table [ details the
18

---

Published in Transactions on Machine Learning Research (09/2025)
Table 1: Parameters and hyper-parameters
PARAMETERS
LINEAR/ NONLINEAR SUBSPACE
RNA
CELEBA
MNIST/ CIFAR-10
MODEL
FCN
FCN
FCN
CNN
LEARNING RATE
0.001
0.001
0.001
0.001
OPTIMIZER
ADAM
ADAM
ADAM
ADAM
EPOCHS
200
1000
200
1000
BATCH SIZE:
10
128
10
128
DATA'S LATENT FEATURES SIZE (d)
20
NUMDER OP FEATURES (D)
1000
40
781/ 3072
TRAIN DATASRT SIXR
5000
5000
3000
5000
SNR/ SAR [dB]
-20 -8, -17, -15, -10, -7, -5, -2, 0,
CONTAMIN ATTON PERCENTAGE (p)
0. 0.1, 0.2.....
DOMAIN SMIFT SCALE (8)
4
EMBEDDING LAYER SIXR
25, 30, 45
20, 100, 300
25
10, 30, 50, 500
HIDDEN LAYER SIZE
- 500
10 3000
4-400
CHANNELS
1-64
Data. For the subspace data model, nonlinear subspace data model, MNIST, and CIFAR-10 datascts, we generate 5000 samples for training and 10000 for testing across all scenarios (sample noise, feature noise, domain shift, and anomaly detection). Regarding the single-cell RNA data, we have focused on dataset number& 4 from (Tran et al.) 2020). which includes 5 distinct domains (biological batches) named 'Baron,, 'Mutaro,' "Segerstolpe," 'Wang," and 'Xin', each representing 15 different cell types. Each cell (sample) in this dataset contains over 15000 genes (features). To facilitate the training of deep models while preserving the domain shift, we have retained the top 1000 prominent features. We utilize the 'Baron' biological batch as our source data for the scenario of domain shift, comprising 5000 training samples, while the target batches are 'Mutaro' (2122 samples), 'Segerstople' (2127 samples), 'Wang' (457 samples), and 'Xin' (1492 samples). As for the sample and feature noise scenarios, we use the 'Baron' domain for both sample and feature noise scenarios due to its largest sample size (8569). We allocate 5000 samples for training and introduce noise to specific samples and features, as described in subsection 3.1 The calculations of the SNR for both sample and feature noise cases are provided in Section
E The reserved 3569 samples are for testing. Please be aware that all the domains in this dataset are inherently noisy, reflecting their real-world nature. Therefore, even when no additional noise is applied (p 0), the data remains noisy. This may account for why the test loss does not decrease monotonically as the model size increases for cases with low noise levels, as shown in Appendix F.5 Figure 56(b)
For the cclebA datasct, including over 200K samples and 4547 anomalics, each characterized by 40 binary altributes, we sub-sample 3000 clean samples and replace [3000 -p] of them with anomalies. This ensures that ~ p- 100% of the data is contaminated with anomalies. Due to the limited availability of anomaly data (4547 samples), the test set includes [(1 p1- 4547] anomalics along with an equal number of clean samples.
Models. All experiments, including the subspace data model, nonlinear subspace data model, single-cell RNA, and celebA datasets are conducted using the same FCN AE architecture. To facilitate the exploration of double descent in both embedding layer size and hidden layer size, we employ simplified model mentioned in (Lupidi et al.) 2023) consisting of a single hidden layer for both the encoder and docoder, as depicted in Figure [aa) We also utilize a CNN AE architecture consisting of three convolution layers in the encoder part, followed by a bottleneck layer, and then a decoder part consisting of three deconvolution layers trained on the MNIST and CIFAR-10 datasets as illustrated in Figure 17 (results for the CNN AE are reported in
Appendix F.3
We work with undercomplete AEs to encourage the acquistiinn of a meaningful embedding in the latent space and prevent the model from learning the identity function. The size of these models determined by the sizes of the hidden layers (for FCN), the number of channels (for CNN), and the bottleneck layer, while the width of these models remains constant.
19

---

Published in Transactions on Machine Learning Research (09/2025)
CXEXE
S chareek
Filan BeSneos Malen
20
Figure 17: Demonstration of CNN AE model structure. represent the width, height and number of channels of each dataset. For MNIST, n 28, == 1 and for CIFAR-10, n =2, == 33 Channels represent the number of channols in cach layer of the CNN AE.
Loss function. All AEs are trained with the mean squared error (MSE) loss function:
MSE
N Σ y 91112
Where N is the number of training data samples, yi is the true vector, and Di the predicted vector. Due to contamination in the training dataset, the norm of train samples tends to be higher than that of the clean test samples. As the MSE loss is not scale-invariant, we opt to normalize both train and test losses only after the training process is complete, using E |yy: -vll. Where y the mcan of (y:) This strategy enables us to continue utilizing the MSE loss function while facilitating a fair and meaningful comparison between train and test losses.
Results. Ensuring the robustness the findings across various model initializations and enhancing their reliability, all figures combine several results of different random seeds. The bolded curves in each figure represent the average across these results, and the transparent curve around each bolded curve represents the +l standard error from the mean.
Environments and Computational Time. All experiments were conducted NVIDIA RTX 6000 Ada Generation with 47988 MiB, NVIDIA GeForce RTX 3080 with 10000 MiB, Tesla V100-SXM2-32GB with 34400 MiB, and NVIDIA GeForce GTX 1080 Ti with 11000 MiB.
Each result Figure represents an average over 10 seeds. The hidden layer sizes for the subspace data model range from to 500 with a step size of 4, and for the single-cell RNA data, they range from 10 to 500 with step size of 10, and from 500 3000 with a step size of 50. This results in 125 and 110 models trained for each dataset, respectively. Figure ((a) illustrates 10 different sample noise levels, requiring the training of 125 10 10 12, 500 models. Similarly, Figure 3(b) depicts 4 different sample noise levels, corresponding to 110 x 10x x 4 = 400 trained models. In total, 16,900 models, each with to 8 million parameters trained on 5,000 data points were needed to obtain the results. In Appendix F.3 Figure 35 we present results for CNNs including between to 64 channels trained on 5,000 images from the MNIST dataset including 5 levels of sample noise and 6 levels of feature noise for 10 different seeds. These experiments require training 64 x 11 10 = 7, 040 models with up 13 million parameters. Each evaluation of specific experiment takes several days if trained the NVIDIA RTX 6000 and weeks if trained the other mentioned GPUs to obtain the results.
B SNR Calculations
In this section, we will outline our approach for calculating the signal-to-noise ratio (SNR) for all experiments involving the addition of noise. Initially, we convert the SNR from decibels to linear SNR using the formula:
[NR(JB]
SNR 10
(2)
We have closed-form equation for the subspace data model determine the scalar 3 required to multiply the train samples and achieve the desired linear SNR value. We use the fact that both train and noise
20

---

Published in Transactions on Machine Learning Research (09/2025)
sampled from ii.d normal distribution and calculate 8 for the sample noise, feature noise, domain shift, and anomalies.
Notations:
z - d x 1 vector. Represents a vector in a latent space of size d. H D d matrix. Represents random matrix to project from dimensional space into a higher-dimensional space (D > d).
D x 1 vector. Represents the noise added to vector with D dimensions.
For the scenario of sample noise, where a particular sample is affected by noise across all its features:
211
E|3²2 Hz)
E
Hz
SNR² 2
(3)
El
Ele
E
1-=1
B²Ez12 Em [
H|
3 Ex 1 TD. Lixd=]
3². D. Ex
2
B d.
D
D
D
Isolating 9, we get that B
SNR vd
(1) Given vector (0, of D iid. samples, E Ei-
Elag] = D.
(2) Given a matrix M ~ (00 In) of size D x D where all entries are i.i.d., then MI + +... MD.1
M + +... +MM111
M[ =
M1 +... + MD.D
M + ..... + MDD
D
D IDXD
0
D
For the scenario of feature noise, each train sample has only D D.p noisy features, meaning the noise vector contains values for only D-p entries. Consequently, determined by V SNR. For practitioners who want to explore the scenario involving domain shift, where the source and target noisy, note that the matrix responsible for projecting &test into a higher-dinennsonal space is denoted H = H + H where H is sampled from a standard normal distribution (0, and both H and H are ii.d. Consequently, H W(0, 1 2²). Substituting H with H" in equation equation 3 we find that [ (H' " D- (1+82). Ta, leading to SNR2 (1 + s²) 88 d, therefore 8
SNR In other words, since the covariance matrix of +ss
H" is (1 I we need to make sure we first normalize the matrix by vI+ to maintain the identity covariance matrix.
For other datasets, such as the single-cell RNA dataset, we normalize each sample æ by its norm ||=|| and similarly normalize each noise vector 6, yielding: =
a ? 1. By employing equation equation attain the intended linear SNR factor 8, and then scale down è by This guarantees that the linear SNR Eccated B.
|xl|| and è |e]| This ensures that the ratio equuls
8, yielding Esaatd
3
C Proof of Theorem 5.2
Assumption C.1. Let DxN (æ1, x2,...,,NN" represent the training set, with each column æ, corre- sponding to a data point according the formulation equation (1. Assume that the number of samples N is large enough such that the rank of the data matrix X is almost surely equal to the number of features D. This assumption is reasonable due to the noise added to many samples.
21

---

Published in Transactions on Machine Learning Research (09/2025)
Theorem 4.1 [o double descent in linear AEs] Let (a(:;0), Dol:;0): :R - -RD be two linear AEs, each with 1 2 layers and architectures defined by Da and Dr (refer to section for definition), with bottleneck sizes ma and ma, where ma < my S. D. Let 0apt and 0gee be the optimal parameters under assumption C.1 Let d denote the dimension of the support of Pa (the dimension of the span of the samples 2 ~ Pa). Then, the following holds,
g (Dr or PPs) (Da; 00p .Pa)
with the inequality replaced with equalily if: (1) my, even is Dr + Do. or (2) May 2 D.
Proof. Recall that the models trained using noisy data sampled from Pn and tested clean data drawn from Pæ to see if there is noise memorization (high test loss) signal learning (low test loss). For simplicity of notation, we will omit the attributions a and b unless explicitly needed. Without the loss of generality, we denote the training set by X DxN = (æ,, 22,..., xN) where the matrix column x. Dx1 are w.r.t formulation O Recall that rank(X X) = D, with probability 1 under assumption C.1 Let Σ E RDDD represent the covariance matrix, defined as Σ XXT. Let its singular value decomposition (SVD) be expressed as Σ U UAU and similarly DD ÜNÜT. Note that the cigenvectors of Σ, which are the columns of U, are arranged in descending order corresponding to their associated eigenvalues in A. By Lemma C.2 since rank then for the optimal solution, 0opt argming X), have that rank (W W220 m, means that the rank of the inner layers multiplication is exactly the size of the bottlencck. By Theorem 3.2, proof Case II in (Kawaguchi 2016), all local minima of the considered optimization problem are global minima, and the expression of a global minimum of the training optimization, namely
D m,
1 N
gopt argming argming
(xx:0) - XI
is given by
0ept = U X (XXT)"
where UDxm the matrix containing the first m eigenvectors of the full eigenbasis UDxD of XXT, namely Un [ui,.... uml (those corresponding to the largest eigenvalues), and the operator (.) stands for the pesudo inverse. Since Dx XX is invertible (rank(X) D), we have
0 = U,
and consequently,
4(x U
Since both, DDD and DDD are orthonormal and span R (by the SVD property), there exists orthogonal matrix RDxD such that U RÚ. This matrix R= UÜT is unitary by construction, satisfying R R = IDXD, and represents rotation or reflection aligning the test and training eigenspaces. Thus U and Ú are related by this R, such that
U
Although the truncated matrices Um and U. generally span different m-dimeenioonll subspaces, both reside within the same ambient D-dimensional space, thus enabling such an orthonormal transformation R.
Calculating the generalization error,
G(D; 0ppt Pa) = E [(:; 0cpt) 2 2
E [|| U - 2|12
E [I] U, U )
E [ (II Un U U Um)
E [Tr ((I (U U
Un
Tr [[I U, U J [ []]
=
Tr [(T -U,, U [
22

---

Published in Transactions on Machine Learning Research (09/2025)
Ir [ Tr U m U Σg
Σ> TF RÚ U mg]
Tr(2a)
U R x 21
(4)
where EDxD= = R i Now, since there exists at least d different indexes j E [D] for which u. R of 0 (otherwise the latent dimension of the test-set would be strictly lower than d in contradiction), then for
ma + 1 < d and ms 1 < E
| R =11l]] I|l R w 0. Hance, since the larger is m in (4)
the lower is 9, we have
g DD:: 000 .Pa) (D, 0ar Pa).
and obviously, if ma, my 2 D, then
(D: 0gr0 Pa) (Da; 02r Pa) 0.
Lemma C.2. Consider the pre-trained linear AE, o(:; ((P))) : RD + RR, of L 2 loyers, where 0 (wiixD - xd WL WW D J for which its bottleneck size holds m(s D), and training set X.
of any dimension satisfies rank (X) 2 m, then rank (WL-1.. W2) m.
Proof. Let X E RDN be the training data, with (xn 2 m. The reconstruction is given by
x WWWW N.WWii..WWWxx.. W.W,X.
Accordingly, the reconstruction loss is
L C= - X WTWL-1 W
If rank WL-1 W2)= m, then the rank of X is most d. This means that WL-1 W2 maps X into subspace of dimension d < However, the optimal low-rank approximation of X that minimizes the reconstruction loss is achieved by projecting X onto the subspace spanned by its top singular vectors (the rank-m approximation). By the Bokarl-Young-NBe Mirsky theorem (Eckart & Young 1936),
Coptimal
|X x
where Xm is the rank-m approximation of X. II rank L. ..2)) < m, the projection would not span the top singular vectors, lcading to
Coptimals
contradicting the assumption that weights optimal.
Remark C.3 (Generalization for noisy test data). Throughout the paper, we trained non-linear models contaminated datasets, drawn from Pæ and tested on clean data sampled from Pa to isolate the effect the learned model, showing noise memorization (high test loss) versus signal learning (low test loss) and observed model, epoch, and sample-wise multiple descents. We followed similar approach as in supervised learning, where models are trained with noisy labels and tested on clean ones (in our unsupervised scenario, we trained on noisy data and tested on clean data, similar (Lupidi et al. 2023 ). We also showed that for the case of linear models, the test loss decreases as model size increases. If we define the generalization loss differently
than the definition in the paper, to account for noise in the test data, then the test loss does not have to
23

---

Published in Transactions on Machine Learning Research (09/2025)
monotonically decrease when increases.
gD Pa) = E [ + n. 0 =
E
U U ( n2) x [
E
[ U I) + Um U
E
[|(T
Um m U | 1
U m
2 + 2((T
Un U Jr U U
E
Um U ):6|
+
m
Tr
U U
U on
-
Tr
m
E
U
clean error
projected noise contribation
We used the assumption that the expectation of the noise is zero. When increases, the first term decreases while the second termn increases, implying that increasing does not have to monotonically decrease this definition of the generalization error.
D
Train Loss Results
In this section, we provide the train loss figures corresponding to each of the test losses mentioned in the main paper.
Train Loss
Train Loss
1.0
1.0
08
30"
08
Hidden Layer
Hidden Layer
08
04
04
02
6
15
20
25
35
40
10
15
20
30
35
Embedding Layer size
Embedding
Layer size
(a) Linear FON AEs.
(b) Nonlinear FCN AEs.
Figure 18: Heatmap of train losses for both linear and nonlinear AEs.
1:
1.0
10% ronile
42% romnle
Tidt Look
0.5
Trai Logg
52% somnle
72% samci
0.
60
82% sanci
Train
0.2
05
123
60
60
Hilden Jayar St%
Midian syar Siza
Charnels
(a) Subspace model. SNR -15 dB.
(b) Single-cell RNA. SNR -17 dB.
(c) MNIST: SNR -17 dB.
Figure 19: Model-wise train losses for the of varying sample noise.
24

---

Published in Transactions on Machine Learning Research (09/2025)
10
0.3
SMR -0
00
Toun Lowe
SN6 -0
08
Train
02
60
1900
04
20
50
60
Hiddan Layer Siza
Hidlon Layar Size
Channele
(a) Subspace model. Sample noise
(b) Single-cell RNA. Sample noise 40%.
MNIST. Sample noise 50%.
80%.
Figure 20: Model-wise train losses for the case of noisy samples and varying SNR.
source (MNIST)
Train Loss
Train Loss
0.06
0.2
0.0
0.00
Layer
Channels
(8) Subspace data model.
(b) MNIST.
Figure 21: Train losses of source data.
0.4
100
Train Loss
0.3
Thain Loss
0.9
0.2
0.8
0.1
0.7
100
1000
Epochs
(a) Subspace data model with SNR -2 dB.
(b) Single-cell RNA data with SNR = -17 dB.
Figure 22: Epoch-wise train losses for varying number of noisy samples.
0.7
1.00
15.0
0.6
0.95
Train Loss
0.5
0.90
Train
0.4
0.85
0.3
0.80
0.75
100
Epochs
Epochs
(a) Subspace data model. Sample noise 40%.
(b) Single-cell RNA data with SNR = -17 dB.
Figure 23: Epoch-werr wise train losses for the case of sample noise influenced by the SNR.
25

---

Published in Transactions on Machine Learning Research (09/2025)
1.1
1.0
Train Loss
Train
0.8
0.4
50
100
125
150
175
200
200
430
600
850
5000
Epochs
Epochs
(a) Subspace data model with 3% noisy samples and SNR = 20 dB.
(b) Single-cell RNA data trained on the 'Baron' batch.
Figure 24: Epoch-wist wise train loss influenced by the amount of domain shift.
1.0
20 servie comple
Train Los8
Figure 25: Train loss the subspace data model with sample noise 70% and SNR = -15 dB for differ- ent number of training samples.
0.4
0.
0.0
100
300 Hidden Layer Size
200
100
Figure 26: Train loss rises the number of samples increases for the scenario of sample noise. Similar behavior exists for feature noise and domain shift sce- narios. Train loss rises because a larger training set makes the model relatively undor-parametorized re- ducing training effectiveness and resulting in higher train loss. However, despite the increase training loss with more samples, the test loss decreases, suggesting improved reconstruction of the test samples.
70 sample nolee
80 sample noise
Train Loss
0.100
sample noise
0.025
Samples
26

---

Published in Transactions on Machine Learning Research (09/2025)
E Results for Feature Noise
In this section, we present the results for the feature noise scenario. Feature noise adds complexity since each sample contains noise in some of its features. As a result, the model never encounters samples with entirely clean features, making it unable to isolate and focus on clean data. Consequently, the model experiences difficulty in learning the correct data structure. Surprisingly, increasing feature noise actually leads to a decrease in the test loss for the single-cell RNA dataset (Figure 27(b]] This can also be observed in Appendix Figure 46(b) Moreover, the peak shifts left the number of noisy
F.3. Figure 35(b) and Appendix F.4
features rises in Figure 27(a)
Natur
Test Loss
20
Test
10
100
300 Hidden Layer Size
500
500
750
1000
1250
1600
1750
2000
Hidden Layer Size
1.0
130%
Train Loss
Train
0.4
0.2
100
200
900
400
500
250
500
750
1000
1250
1500
1750
2000
Hidden Layer Size
Hidden Layer Size
(a) Final ascent phenomenon Xue et al. the Subspace data model trained with SNR- -13 dB.
2022 for
(b) Non-monotonic behavior for the Single-cell
RNA data trained with SNR = -12 DB.
Figure 27: Test loss exhibits misel-wise double descent and non-monotonic behaviors for the case of varying feature noise.
JUN 24h
Test Loss
Test Loss
30
20
50
100
150
300
1000
1500
2000
2500
Hidden Layer Size
Hidden Layer Size
1.0
10.0
0.8
Train Loss
0.8
0.8
Train
0.4
0.2
100
150
200
250
300
500
1000
1500
2000
2600
Hidden Layer Size
Hidden Layer Size
(a) Subspace data model with 40% noisy features. Reyond hidden layer of size 300, the test loss rises.
(b) Single-cell RNA data with 10% noiay features. Beyond hidden layer size of 2600, the test loss continues to decrease, and the train loss increases.
Figure 28: The effect of SNR for the case noisy features on the test loss curve.
27

---

Published in Transactions on Machine Learning Research (09/2025)
DK
Test Loss
Test
100 Epochs
175
Epochs
0.5
1.0
Train Lees
Trein Loss
0.2
0.0
100
Epochs
Epochs
(a) Subspace data model with SNR = -12 dB.
(b) Single-cell RNA data with SNR -12 dB.
Figure 29: Epoch-wise double descent influenced by the number of noisy features.
0.5
0.4
Test Loss
Test
0.1
100 Epoche
150
175
200
Epochs
0.5
Train Lass
Trein Loss
0.7
Epochs
Epochs
(a) Subspace data model.
Feature noise 80%.
(b) Single-cell RNA data. Feature noise 50%.
Figure 30: Epoch-wise double descent for the case of feature noise influenced by the SNR.
Bore 19/30
Test Loss
Test
1.6
1.0
10000
20000
30000
40000
50000
10000
20000
30000 40000
50000
Number of Semples
Number of Samples
(a) Sample-wise non-monotonicity for varying
(b) Sample-wise non-monotonicity for varying num- ber of noisy features. SNR = -13 dB.
SNRs in the scenario of feature noise = 90 %.
Figure 31: Sample-wise non-monotonicity pattern for the subspace data model for the scenario of feature noise.
28

---

Published in Transactions on Machine Learning Research (09/2025)
F Additional Experiments
F.1 More Results for Domain Adaptation
This section presents the UMAP visualizations of the different domains for both the subspace model and single-cell RNA data in Figure 32 Results for different model sizes trained on the subspace model dataset are also reported in Figure
Source Target
(Banos_ trrpot tarpet (eogesttpeps target (Wang (o) target (Kgy
(a) The UMAP representation shows a clear domain
(b) Clusters represent different cell types. Different domains are represented by different colors.
shift between the source and target datasets.
Figure 32: UMAP representations of source and target datasets for the subspace data model (a) and single-cell RNA dataset (b).
Figure 33 illustrates the results based a similar experiment conducted in Section 7.1 for the subspace data model. As expected, the interpolating models exhibit the poorest. KNN-DAT outcomes. Over-parammeterized models introduce decrease in the test loss indicating an improved reconstruction of the target data. In this scenario, we noticed that smaller models perform botter than pyerppaaanttrriedd models based on KNN-DAT results. We think that the small size of the hidden layer (4) and the high dimensionality of the dataset (50 features) result significant information loss in these layers. This could lead to closely clustered vectors the embedding space, ultimately causing low KNN-DAT results. However, a hidden layer of size 4 indicates insufficient capacity to represent the signal, shown by the high values of test and train losses in Figure 33
F.2 Overcomplete AEs
In this section, show that overcomplete AEs learn the identity function, rendering them irrelevant for our study. Figure 34 is extension of Figure 2 to overcomplete AFs, where models are trained on the subspace data model (Subsection 3.1) with D = 50 features. Models with embeddings and hidden layers exceeding D (above the white dashed lines), making them overcomplete, achieve zero train and test reconstruction loss, confirming they learn the identity function, resulting the absence of double descent.
F.3 Double Descent Results For CNNs Trained on MNIST and CIFAR-10
In this section, we demonstrate that the double descent phenomenon can be reproduced other unsupervisod AE architectures. We employed the MNIST (LeCun et al., 1998) and CIFAR-10 (Krizhevsky al., 2009) datasets and trained undercomplete CNNs detailed in Figure 17 For the case of sample noise, the noise is added to p 100% of the images, and present results for both MNIST and CIFAR-10. while all the other cases are presented using the MNIST datasct. For the feature noise scenario, noise is introduced to p 100% of the pixels of each image. To demonstrate the phenomenon with the presence of domain shift, the models are trained on the MNIST-M and MNIST datasets and tested on MNIST and MNIST-M, respectively. Results for model-wise double descent for varying amounts of sample and feature noise cases for the MNIST datasct. respectively. Varying levels of sample noise and SNRs for the CIFAR-10
are presented in Figures 3(c)
dataset presented in Figure 36
29

---

Published in Transactions on Machine Learning Research (09/2025)
target (shift 3)
source
1.0
Train Loss
2.0
Test
1.5
1.0
100
Hidden Leyer Size
Hidden Layer Size
UMAP for hidden byyer $20
LIMPP for hiddon layor size 12
UMAP for hiddon layer size = 100
Source
Source Target
Source Target
Target
KNN-DAT - 0.68
KNN-DAT 0.88
KNN-DAT 0.81
Figure 33: UMAP of the embedding vectors with a size of 45 and KNN-DAT results for different model sizes trained on the subspace data model for a shift of 3.
Test Loss
Test Loss
20
x D- 30)
Hidden Layer size
Oe D 501
Hidden Layer
0.8
1.5
0.4
10
10'
10"
0.5
10
10a
Embedding Layer size
Embedding Layer size
Train Loss
Train Loss
1.0
14 18
0.8
Hidden Layer size
1
10*
0.8
Hiöden Layer
0.8
0.6
0.4
0.4
10'
10"
0.2
0.2
10 Embedding Layer size
Embedding Layer size
(a) Linear AEs models.
(b) Nonlinear AEs models.
Figure 34: An extension of the heatmaps presented in Figure to overcomplete AEs (models above the white dashed lines) resulting in zero train and test reconstruction loss for both lincar and nonlinear AEs.
In Figure 37 we show the test. and train loss results (lop two sub-figures) for three different models (3, 5, and 60 channels) trained on MNIST with 50% sample noise and an SNR -15 dB and find out that over-parameteriied models can reduce the noise levels in an image. The smallest model, with channels, is under-pant paaderr-rrammtrizeddd The second model, within the critical regime, with channels, performs poorly, while the third is wer-paraameterized, containing 60 channels. Interestingly, We noticed that though
30

---

Published in Transactions on Machine Learning Research (09/2025)
1.50
125
1.25
hale
Test Loss
Train Loss
1.00
0.75
0.50
0.25
0
0.00
Channels
Channels
Figure 35: Model-wise double descent for CNNs trained on MNIST with varying levels of feature noise and
SNR -20 dB.
35
3.0
3.0
2.5
2.5
Test Loes
2.0
1.
Tost
1.5
1.0
1.0
0.0
0.0
Channels
Channels
1.0
0.8
Train Loes
Thain Loss
0.6
0.4
0.2
30
50
60
10
36
40
60
Channels
Channels
(a) SNR -20 dB.
(b) Sample noise 90%
Figure 36: Model-wise double descent for CNNs trained on CIFAR-10 with varying levels sample noise (a) and SNRs (b).
AE was not trained to remove noise (as in denoising AEs (inneent et al. 2008 2010 poocpparamttrrieed models were able to reduce noise to some exlent. In contrast, models within the critical regime performed significantly worse.
After training, we evaluatod each model by feeding it images with varying SNR values and examining the recoestructed outputs (bottom sub-figure in Figure 37 The over-parameterrize model produced the best-quality reconstructed images. Following that, the mndee--arramteerized model performed moderately well, and the model in the critical regime generated the noisiest images. This is because the critical model focused memorizing the noise during training instead of learning the underlying signal, resulting in consistently noisy outputs. In contrast, the over-parameterized model had enough capacity to memorize the noise and learn the signal. While the muder-paraaeterrin model cleans the images better than the critical model, it still distorts some details compared to the parameterized model due to its limited capacity.
To quantify noise reduction, used the Peak Signal-to-Noise Ratio (PSNR), metric that assesses signal quality by comparing the original image to its noisy version. PSNR measures the ratio between the maximum possible value of a signal (R²) and the power of the noise (MSE). Higher PSNR values indicate better quality, meaning less noise. The formula for PSNR is:
R² PSNR 10 log MSE( +
where æ + e represents the noisy image (c is the noise), and x the clean version. This metric, expressed in decibels, allows us to evaluate how well each model cleans the images. As shown, the par-pparameterized model
31

---

Published in Transactions on Machine Learning Research (09/2025)
1.25
Test Loss
1.00
Train Loss
0.2
0.25
0.0
0.00
Channels
Channels
Input image
chennela
60
g
2
g
PSNRR 18.76
PSNR 13.65
PSAR 931.99
PSNR - 13.70
PSNR 12.02
PSNR
9
9
7
PSNR 18.07
F8NA 12.93
PSNR - 24.96
PSNR 13.97
PSNR 11.43
PSNR 14.67
2
2
PSNR 17.21
PSNR 12,22
PSNR - 23.04
PSNR 13.61
12.07
PSNR 83
SNR [dB]
Reconstruction
SNR SNR=0 [cBB
Reconstruction
Input image
chunnels
channels
channels
Input image
channels
channets
ED
PSNR 12.54
PSNR 10.44
PSNR 13.36
PSNR 11.05
PSNR 10.35
PSNR 12
PSNR 13.03
PSNR 12.23
PSMR 1SMR-10:1 10.11
PSNR - 12 50
PSNR 12.24
PSNR- 10.24
PSNR - 12.61
SNR [[]B]]
Reconstrction
SNR -4 [JB]
Reconstruction
Figure 37: Models trained on 50% noisy MNIST images with SNR = -15 [dB] and tested on MNIST images with different values of SNR.
consistently achieves the highest PSNR values (highlighted in bold green), while the poorly interpolating model, which primarily memorized noise, produces the lowest PSNR values (in red). In conclusion, over- parameterized models are copable of reducing noise when trained on noisy data, even without being explicitly lasked to do 80.
We proceed by illustrating the impact of SNR on the test loss curve for both sample and feature noise sccnarios in Figures [(c) 38 respectively. As expected, low SNR values unveil double descent and increase test. loss. We then investigate the effect of domain shifts between the training and testing datasets in two cases. First, models trained on the MNIST dataset and tested the MNIST-M dataset, shown in Figure 5(b) Second, models are trained on MNIST-M and tested on MNIST, as seen in Figure 39(b) In both cases, the model-wise double descent curve is observed.
We further illustrate this phenomenon along the epochs axis, displaying non mo-ooonooo behavior and double descent under different levels of sample and feature noise (Figure 40) and showing the impact of SNR variation (Figure 41) for models trained on contaminated MNIST. Additionally, we provide similar results under domain shift conditions between the train and test datasets (Figure 42h. Sample-wise double descent. and
32

---

Published in Transactions on Machine Learning Research (09/2025)
00
Test LOGS
Train
10
20
30
40
00
20
20
40
50
02
Channels
Channels
Figure 38: Model-wise double descent for CNN trained MNIST with varying levels of SNRs and feature noise 10%.
Test Loss
0.25
0.20
0:15
Trein
0.10
0:15
0.05
0.00
Channels
Channels
Figure 39: Wisel-wise double descent for CNNs trained on MNIST-M and tested on MNIST.
non-monotonic behavior observed as well in all contamination setups. The cases of varying levels of sample noise and feature noise are displayed in Figure 43 and for varying SNRs for both scenarios in Figure 44 Sample-wise double descent is also illustrated in Figure 45 for when domain shift is present between the training data (MNIST) and the testing data (MNIST-M).
54
Teat Lass
That Lose
66
950
960
Epochs
Epochs
Train Loos
Train Loes
0.4
650
(a) SNR = 0 dB.
(b) SNR = -5 dB.
Figure 40: wisch-wise non-monotonic behavior for CNNs trained on noisy version of MNIST with varying levels of sample noise (a) and feature noise (b).
F.4 Double Descent Results for the Nonlinear Subspace Data Model
have developed new dataset with nonlinear characteristics to investigate the double descent phenomenon in more complex scenarios. Although the single-cell RNA A dataset is already nonlinear, we have created this dataset to demonstrate the reproducibility of the double descent phenomenon across various datasets.
Building the subspace data model discussed in Subsection 3.1
As the subspace data model discussed Subsoction 3.1
we sample N latent vectors {2 from normal N.
distribution and project them a higher dimension using a random matrix H, The key difference is the inclusion of noulinear components 2f and 23, each projected higher dimensional space with different.
33

---

Published in Transactions on Machine Learning Research (09/2025)
0.0
Tost L000
Test
36
650
Epochs
Epocht
Then Lode
0.6
Train
0.
0.4
950
(a) Sample noise = 100%.
(b) Feature noise = 40%.
Figure 41: Epoch-wise double descent and non-monotonic behavior for CNNs trained on MNIST with varying
SNRs. (a): sample noise, (b): feature noise.
1.0
1.50
0.
Teen Lose
Lo00
Toat
0.4
0.50
0.5
6
860
0.00
1000
0.0
1.0
0.8
Tran L006
0.0
10
15
25
950
1000
10
20
30
975
1000
Epochs
Epochs
(a) Source MNIST, target = MNIST-M.
(b) Source MNIST-M, target MNIST
Figure 42: Epoch-wise double descent and non-monotonic behavior for domain shift.
random matrices H2 and H3. To create contaminated setups of sample and feature noise, noise is added to p 100% of the data where 3 controls the SNR:
B(H H2²? + H32 + with probability p, 3(Hiz; H22 H32e), with probability 1
For the domain shift. scenario, we divide the latent vectors into training and testing sets and use the same parameter 's' described in Subsection to control the shift between the train and test in the following manner: H H, + H' for 1 s and get:
H Zirinn + H2 (2 + Hy (z(iaia))) train,
2; H zlsst + H (z)as))² 2 + H (zlest)3. )3 if test.
For anomaly detection, clean samples are represented by 3(H, + H₂2, + Hzz;), with 100% of them replaced by anomalies sampled from a normal distribution, as detailed Subsection 3.1
The model used in this section the same FCN model deseribed Figure ( We start by presenting
results for the sample and feature noise scenarios depicted in Figure 46 As shown, the test. loss results for
34

---

Published in Transactions on Machine Learning Research (09/2025)
90
Test LOGS
1.5
Teat
0.5
20006
Samples
Number Semples
0.4
0.0
Train Loos
0.5
Train Loos
0.1
10300
20000 Number of Samples
30000
40000
10200
20000 Number Samples
33000
50000
(a) SNR 17 dB
(b) SNR -20 dB
Figure 43: Sample-wise double descent and non-monotonic behavior for CNNs trained on contaminated
MNIST with varying levels of sample noise (a) and feature noise (b).
1.85
Tost L000
Thal Los0
0.72
12000
200ED
30000
4000
10000
20000
30000
43000
80000
Number of Semples
Numbar of Samples
0.5
Tren Lose
0.5
Trein
0.1
10000
20003
30000
40000
50000
10000
20000 Number Samples
30000
40000
50000
Number Semples
(a) Sample noise 50%.
(b) SNR -20 dB.
Figure 44: Sample-wise double descent and non-monotonic behavior for CNNs trained contaminated MNIST with varying levels of SNR. (a): sample noise, (b): feature noise.
the case of sample noise (Figure 46(a) resemble those of the subspace data model presented in Figure ((a) Figure 46(b) demonstrates the model-wise final ascent phenomenon for the case of feature noise as elaborated in Appendix F.5 Figure 47 shows how the SNR affects the test loss curve for both sample and feature noise cases. As observed, the test loss increases with decreasing SNR. Additionally, the final ascent in the test loss is depicted in 47(b] for the fcature noise scenario, where the slope becomes stceper as the SNR decreases. We also demonstrate the double descent and final ascent results regarding the domain shift scenario in Figure 48 and the anomaly detection capabilities in Figure 49
We also observed wisch-wise double descent and non-monotonic behavior for this dataset, as shown in Figure 50 for different percentages of sample and fealure noise and in Figure 5 for varying SNRs under the same noise conditions. Additionally, epoch-wise double descent is also observed when a domain shift is present between train and test sets, as depicted in Figure 52 Instances of sample-wist double descent and
non-monotonic curves are also reported and displayed in Figure 53 for varying levels of sample and feature noise, Figure 54 for varying levels of SNR, and in Figure 55 for domain shift.
35

---

Published in Transactions on Machine Learning Research (09/2025)
325
0.33
Tuain Lo00
0.30
0.75
0025
0020
225
30000 Number of Samples
Number Samples
Figure 45: Sample-wise wise double descent for models trained on the MNIST dataset and tested on the MNIST-M dataset.
1.25
05 sample nole
1.00
Loas
30%
Tost Losa
405 1dea
00% pla Idda
200
300
400
200
400
Hidden Layer Size
Hidden Layer Size
1.0
2% siee
Tran Losa
0.5
40% sample
50% sample
30% sample robs
sample
Train
70% sample
102
3001
300
400
Hidden Layer Size
Hilden Layer Size
(a) Sample naise scenario with SNR -13 dB.
(b) Feature noise scenario exhibits final
(Appendix) F.5
with SNR = -20
Figure 46: Model-wise double descent for the nonlinear subspace data model with varying levels of sample noise (a) and final ascent with varying levels of feature noise (b).
F.5 Final Ascent Phenomenon
While training various models different datasets contaminated with sample and feature noise at different SNR levels and domain shifts between train and test sets, we observed final ascent phenomenon characterized by a pattern of
test loss. The phenomenon was first observed in (Xue et al. 2022) in supervised learning with label noise. We suspect a potential connection to this phenomenon in unsupervised learning, which we have yet fully analyze. We refer to Figure 56(a) which illustrates the final ascent results for the subspace data model under extreme conditions of 100% sample noise, continuation of Figure (aa We also present the final ascent results for the single-cell RNA dataset in Figure 56(b) b) Another instance of final ascent with the presence of varying feature noise is illustrated in
Figure (7 for the subspace data model and in Figures 46(b) 47(b) for the nonlincar subspace data model.
Results are also replicated using the nonlinear subspace data model under various domain shifts, as observed
in Figure 48
F.6 Multiple Descents Under Different Noise Types and Sparse AEs
This section explores the emergence of double and triple descent for noise distributions beyond Gaussian noise and sparse AEs. Figure 57 illustrates the phenomenon for the subspace data model and single-cell RNA datasets when subjected to Laplacian noise. The experimental setup mirrors that of Figure 3(a) As shown, both datasets exhibit similar results under these conditions.
We extend our rescarch to recent applications of AEs, including sparse AEs, which are increasingly utilized explainable AI (XAI) (Gao et al.) 2024) and have been adopted by Google in their Gemini project. Using sparse CNN AEs, trained models the MNIST dataset containing 80% noisy samples and observed
36

---

Published in Transactions on Machine Learning Research (09/2025)
1.0
Thal Look
0.
Hidden Layer
Hidden Layer
Train Lo06.
Train Loes
0.6
Hidden
Hiddon
(a) Double descent with sample noise 80%.
(b) Double descent and final ascent with fea- ture noise = 100%.
Figure 47: Effect of SNR on the Lest loss curve as a function model size. (a): sample noise scenario. (b): feature noise scenario.
14
0.
Teet Loos
0.6
0.8
Hidden Layer Size
Hidden Size
Figure 48: Model-wise double descent and final ascent for the scenario of domain shift.
the emergence of double descent. The models were configured with an embedding layer of size 550, and the parameter k, determining the top highest embedding values to retain, was set to 500. The results are illustrated Figure 58
37

---

Published in Transactions on Machine Learning Research (09/202))
Critical Regime
77%
Clean Toot Loos
unuenalyy rako 90%
Air 97%
Aaconal Text Loge
AUC
Hidden Dimamaion Siza
Hillen Dimeriio Sieu
Hiddan Demeraaion Bize
Figure 49: Nonlinear anomaly data with SAR = -15 [dB]. Left: test loss of the clean samples. A double descent pattern emerges for low SARs and high anomaly presence in the training data. Middle: test loss of the anomaly data. Right: Non-monotonic behavior of the ROC-AUC.
25
Tost Loos
Toxt Locs
20
2.00
75
100 Epochs
125
150
175
200
75
100 Epochs
125
150
175
200
0.4
0.15
Train Loo6
0.2
Tran
0.1
Epochs
Epochs
(a) SNR = 0 dB.
(b) SNR -15 dB.
Figure 50: Epoch-wise double descent and non-monotonic behavior for varying levels of sample noise (a) and
feature noise (b). For the scenario of feature noise, we mostly noticed the non-monotonic curve at 10%.
0.10
1066
0.06
Tool
50
75
100
125
150
175
200
100
125
150
200
Epocha
Egocha
Train 1006
0.15
0.3
Tran
0.1
0.06
Epochs
Epochs
(a) Sample noise 30%.
(b) Feature noise 10%.
Figure 51: Epoch-wise double descent and non-monotonic behavior for varying levels of SNR. (a): sample
noise, (b): feature noise.
38

---

Published in Transactions on Machine Learning Research (09/2025)
Tost
0.a
Train 92
83
Epochs
Epochs
Figure 52: Epoch-wise double descent for when domain shift is present between train and test
Toot Loos
Text Loos
0.2
10000
20000
20000
12000
50000
10000
30000 Number Semples
2000
40000
30023
Number Samples
0.150
0.105
Trein Lose
3.100
Teein Loes
0.6
0.015
0.5
0.505
10000
20000
30303
<0000
50000
10000
30000 Number Samples
20003
40003
30000
Number of Samples
(a) SNN -15 dB.
(b) SNR -20 dB.
Figure 53: Sample-wisc double descent and non-monotonic behavior for varying levels of sample noise (a) and feature noise (b).
16
Tost Loos
Teat Loos
10000
20000
30000
42000
50002
10000
20003
30000 Semples
40000
50003
Number Samples
Number
0.150
0.125
Trein Loss
Trann Lose
0.40
0.075
0.36
0.000
0.30
0.26
18000
20000
30000
40000
50000
Number of Samples
Number of Samples
(a) Sample noise 90%.
(b) SNR -20 dB.
Figure 54: Sample-wise double descent and non-monotonic behavior for varying levels of SNR. (a): sample
noise, (b): feature noise.
0.30
Test
Taah
0.28
0.28
20000
Samples
Number
Figure 55: Sample-wise non-monotonic behavior for the domain shift scenario.
39

---

Published in Transactions on Machine Learning Research (09/2025)
100% sample robie
1.0
Text Loos
Toal Look
100
200
300
400
800
600
1030
1000
2000
2000
3303
Hidden Layer Size
Hidden Layer Size
130%
Trein Lo8k
0.0
0.0
0.6
Tean
0.4
0.
100
500
400
600
500
1000
1500
2000
2500
5000
Layer
Layer
(a) Subspace data model and SNR -15 dB.
(b)
Single-cell RNA data and SNR -00
dB.
Figure 56: Test loss exhibits model-wise double descent followed by a final ascont for the scenario varying sample noise.
00
0.6
Text Lote
Toat Look
0.4
0.8
100
200
200
400
600
500
1030
1000
2000
2000
3000
Hiddon Layer Size
Hidden Layer Size
0.0
0.6
Train
Tren
0.
0.0
100
200
500
400
500
500
1000
1500
2000
2500
3000
Hidden Layer
Layer
(a) Subspace data model and SNR -15 dB.
(b) Single-cell RNA data and SNR -17
dB.
Figure 57: Test loss exhibits model-wise double descent for the case Laplace noise.
Tosn Loss
1.00
Loes
0.4
0.2
0.25
0.00
10
20
60
20
60
Channels
Figure 58: Test loss exhibits modcl-wise double descont for sparse CNN AEs trained on MNIST with embedding layer size 550 and = 500.
40


Reviews:
[{"review_id": "XSKugw2riy", "reviewer": "Reviewer_p8vq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nDouble descent in unsupervised settings remains a wide open question, and it is refreshingly interesting to see a paper that tackes this topic. \n\nThe authors do a very good job at motivating their study, and the related works section is quite well-written. I am not an expert on double descent so I may have missed some papers, but to the best of my knowledge the authors are right to claim that  \"this is the first demonstration of model and epoch, and sample-wise double descent in a fully unsupervised setting\". This is an important contribution!\n\nI thought the experiments were nicely designed, and I enjoyed the focus on downstream tasks.\n\n**Weaknesses**\n\nI have one major concern about this paper: I do not follow the reasoning of the proof of the main theorem. The main theorem claims that linear AEs with a larger latent space will *always* generalise better (the only regularity condition being that the covariance matrix of the data is full rank almost everywhere). In other words, linear AEs do not overfit with respect to the latent space dimension.\n\nFirst, I find the theorem quite hard to believe from practical grounds: unless I am missing something, if it were true, when doing dimensionality selection in PCA using a validation set, the model with the largest dimensionality would always be chosen. Choosing the number of latent dimensions in PCA using a validation set is very commonly done (see e.g. Minka, 2000, Josse & Husson 2012), and generally works very well. However, this theorem seems to imply that this is impossible.\n\nRegarding the proof (available in Appendix C), I do not follow a particular point. The authors denote the empirical covariance of the data set by $\\Sigma = X X^T$, whose $m$ largest eigenvectors are stored in a matrix $U_m$, and the theoretical covariance of the test distribution by $\\tilde{\\Sigma} = \\mathbb{E} [ x x^T]$, whose $m$ largest eigenvectors are stored in a matrix $\\tilde{U}_m$. They then claim that there exists an unitary matrix $R$ such that $U_m = R \\tilde{U}_m$. I do not see how this can be the case, since $U_m$ and $\\tilde{U}_m$ have no reason to span the same subspace. I also do not understand what is the dimension of this matrix $R$. In some lines it looks like it should be $D$ by $D$, and is others $m$ by $m$.\n\n\n\n**Minor things**\n\n- In many cases in the experiments, the distribution of the test set is not the same as the distribution of the training set. It would be nice to summarise somewhere which experiments are following the standard ML dogma of using the same distribution, and which experiments are not.\n\n- This is very minor, but the common practice in ML is generally to write data matrices in a way such that each row is an observation, and each column a feature. Here the authors do the opposite.\n\n**Additional references**\n\n- Josse & Husson, Selecting the number of components in principal component analysis using cross-validation approximations, CSDA 2012\n- Minka, Automatic choice of dimensionality for PCA, NeurIPS 2000", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors study the double descent phenomenon in what is perhaps the most basic unsupervised model that can be overparametrised: autoencoders (AEs).\n\nThey start from the simplest kind of AE: linear ones. In this context, they show (empirically and theoretically) that double descent does not occur. As I detail below, I have some issues with this claim.\n\nThen, they move on to nonlinear AEs, and show empirically that double descent *does* occur in this context, both in terms of the dimension of the latent space and in terms of the size of the involved networks.\n\nAll experiments are performed over a wide variety of datasets, architectures, and corruption schemes (double descent is often easier to spot in setting with corrupted data). They also look at the impact of double descent on downstream tasks (domain adaptation and anomaly detection).", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Clarifying whether or not the main theorem is true, and if it implies that PCA does not overfit with respect to the latent space dimension.", "broader_impact_concerns": "I have no concerns here.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SUH8btMPbM", "reviewer": "Reviewer_RR8M", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\nThe paper investigates an open research topic and provides novel theoretical and empirical insights that should be of interest to the community.\n\n# Weaknesses\nReadability can be improved, outlining the consequences of this work and highlighting next steps for future research.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates multiple descents in unsupervised autoencoders. It provides theoretical insights for linear autoencoders and empirical insights for non-linear autoencoders.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Below are a few points on how to improve the readability of the paper and make its contribution more appealing to the community.\n\nThe acronym \"SNR\" is not explained but used quite frequently -- it is clarified in Appendix B, though. However, to be more self-contained, I recommend explaining it when used for the first time.\n\nIs there any reason behind using the single-cell RNA data set and the CelebA data set? I think, in particular, for the real-world data sets, it is important to justify why they have been selected.\nFurthermore, I am missing at least a brief description of the data sets. It is okay to put the details in the appendix, but in order to be more self-contained, the paper should provide a minimal level of information, such as the domain and dimensionality of the data.\nI think it might be a good idea to have a section briefly describing all the data sets used in this paper. For instance, after describing those initial data sets, on page 6 (Figure 5), you suddenly present an experiment involving MNIST which was not mentioned before. Later on, you also have experiments on CIFAR-10 -- please also provide a reference for those data sets.\n\nSections 3.2 - 3.5 are breaking the flow. Can you try to restructure them such that there is a better reading flow?\n\nSection 5: What is \"undercomplete FCN\"?\n\nConclusion: I miss a discussion of the consequences or impact of this work. What are the lessons learned? Are there any research questions that are still open or are arising after your study?\n\n\nMinor:\n- page 5 \"following sentence\" -> \"following theorem\"\n- The generalization error (GE) is denoted as G. However, in the text, GE is used instead of G. This might be a bit confusing. Maybe you can make it more consistent.\n- Whenever possible, use vector graphics", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Uy8vE4JFf0", "reviewer": "Reviewer_CF33", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper presents results that challenge the popular belief that double descent is specific to supervised learning. Studying double (and multiple) descent in unsupervised learning is interesting, and the utility studies to connect the observations to real world applications is a plus.\n2. The breadth and coverage of the empirical analyses are good. The authors investigated the effect of sample noise, feature noise, domain shift and anomalies, and experimented with two popular types of data with significant domain gap: single-cell RNA sequencing data and CelebA human face images.\n3. The observation that the presence of nonlinearities determines the existence of double descent in unsupervised autoencoders is interesting and well supported by results.\n4. I like Figure 2(b) and 2(c). These plots provide highly intuitive visualization of double descent along both the hidden layer size axis and the embedding layer size axis in nonlinear autoencoders and the lack thereof in linear autoencoders.\n\nWeaknesses:\n1. I would suggest some minor figure-making improvements. \n\n    a. The legends for most double descent figures need to be enlarged, and some legends need to be better placed (for example, Figure 12(b) legend can be put in lower right instead).\n\n    b. The panel placements for most figures are fine, but for Figure 15, enlarging panels (a) through (e) to reduce the whitespace within the same row might be helpful.\n\n    c. [Optional] I personally feel there are too many double descent curves under different conditions but they are not made super obvious --- I understand they are written in captions, but I wonder if centralizing them and making clear distinctions what conditions they respectively correspond to might be helpful. I would recommend the authors consider merging some of them into panels within the same figure, putting the train losses to appendix (like what the authors have already done for some experiments), and clearly labeling the respective conditions. This might make things clearer but it might make things messier, so it’s up to the authors to try and decide the best way of presenting these results.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In machine learning, as the model size grows compared to the data size, the test error is expected to decrease and increase as the model becomes overparameterized. Double descent is a surprising phenomenon where further overparameterization leads to a second decrease in test error. Double descent has long been observed in supervised learning, but is relatively understudied for unsupervised learning. The authors attempt to investigate the mystery of double descent in unsupervised learning.\n\nThe authors first empirically show that double descent does not happen in linear unsupervised autoencoders, but double (and multiple descent) can consistently present in nonlinear unsupervised autoencoders across model and architectural design. Then, they identify that bottleneck size has a big influence of the double descent curve. They further demonstrate the practical value of the observation by showing that overparameterizing models to the double descent regime can improve downstream tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Refer to weakness 1.", "broader_impact_concerns": "Nothing I am aware of.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)