Unraveling the Enigma of Double Descent: An In-depth Analysis through the Lens of Learned Feature Space
Authors: Yufei Gu
Xiaoqing Zheng
Tomaso Aste
Venue: ICLR 2024 poster (2024)
Keywords: neural network
double descent
classification
interpretability
Primary area: learning theory

Abstract: Double descent presents a counter-intuitive aspect within the machine learning domain, and researchers have observed its manifestation in various models and tasks. While some theoretical explanations have been proposed for this phenomenon in specific contexts, an accepted theory for its occurring mechanism in deep learning remains yet to be established. In this study, we revisit the phenomenon of double descent and demonstrate that the presence of noisy data strongly influences its occurrence. By comprehensively analysing the feature space of learned representations, we unveil that double descent arises in imperfect models trained with noisy data. We argue that while small and intermediate models before the interpolation threshold follow the traditional bias-variance trade-off, over-parameterized models interpolate noisy samples among robust data thus acquiring the capability to separate the information from the noise. The source code is available at \url{https://github.com/Yufei-Gu-451/double_descent_inference.git}.

s "Department of Computer Science, University College London {yufei.gu gu. t.aste)@ucl.cc ac. uk, zhengxq@ fudan. edu cn Published as a conference paper at ICLR 2024
UNRAVELING THE ENIGMA OF DOUBLE DESCENT: AN IN-DEPTH ANALYSIS THROUGH THE LENS OF LEARNED FEATURE SPACE
Yufei Gu Xiaoqing Zheng*.,, and Tomaso Aste
School of Computer Science, Fudan University
Shanghai Key Laboratory of Intelligent Information Processing
ABSTRACT
Double descent presents a counter-intuitive aspect within the machine learning domain, and researchers have observed its manifestation in various models and tasks. While some theoretical explanations have been proposed for this phe- nomenon specific contexts, an accepted theory for its occurring mechanism in deep learning remains yet to be established. In this study, we revisit the phenomenon of double descent and demonstrate that the presence of noisy data strongly influences its occurrence. By comprehensively analysing the feature space of learned representations, we unveil that double descent arises in imper- feet models trained with noisy data. We argue that while small and intermedi- ate models before the interpolation threshold follow the traditional bias- variance trade-off, perepaaameterized models interpolate noisy samples among robust data thus acquiring the capability to separate the information from the noise. The source code is available https::////// httpp::///gttubbb/p com www/ooffeiie..ooooooo///ooeeeoee httr//uuffii-uu555
INTRODUCTION
As well-known property of machine learning, the bias-variance trade-off suggested that the vari- ance of the parameter estimated across samples can be reduced by increasing the bias in the es- timated parameters (Geman al. 1992). When model ander-parameterized, there exists combination of high bias and low variance. Under this circumstance, the model becomes trapped under-fitting, signifying its inability grasp the fundamental underlying structures present the data. When model is peeetttraettr,,,,, there exists a combination of low bias and high variance. This is usually due to the model over-fitting noisy and unrepresentative data in the training set. Tra- ditional statistical intuition suggests difficulty in reducing bias and variance simultancously and advises finding a 'sweet spot' between underfitting and overfitting. This conjugate property pre- vents machine learning estimators from generalizing well beyond their dataset (Hastic et al. [2009). However, modern machine learning practices have achieved remarkable success with the traming of highly parameterized sophisticated models that can precisely fit the training data ('interpolate" the training set) while attaining impressive performance on unseen test data.
To reconeile this contradiction between the classical bias-variance trade-off theory and the modern practice, Belkin et al.] (019)) presented a unified generalization performance curve called "Double Descent'. The novel theory posits that, with an increase in model width, the error initially follows the conventional 'U-shaped curve', reaching an interpolation threshold where the model precisely fits the training data; Beyond this point, the error begins to decrease, challenging the tra- ditional understanding and presenting new perspective on model behavior. Similar behaviour was previously observed Vallet al. (1989); Opper (1995) 2001), but it had been largely overlooked until the double descent concept was formally raised.
"This work done during the author's internship Fudan University.

---

Published as a conference paper at ICLR 2024
The double descent phenomenon is prevalent across various machine learning models. In the con- text of linear models, a series of empirical and theoretical studies have emerged to analyze this phe- nomenon (Bartett et al. 2020; Loog et al. 2020 Belkin et al.) 2020; Deng et al. [0022] Hastie al.) 2022). In deep learning, many researchers have offered additional evidence supporting the presence and prevalence of the double descent phenomenon across broader range of deep learning models, including CNNs, ResNets, and Transformers. in diverse experimental setups (Nakkiran et al. [002]] Chang et al. 2021; Liu et al., 2022; Gamba et al. 2022). Nakkiran et al. (2221) specifically con- cluded that this phenomenon is most pronounced when dealing with label noise in the training set, while can manifest even without label noise for the first time. Despite the widely accepted empirical
observation of the double descent phenomenon within the domain of deep learning, comprehending the underlying mechanism of this occurrence in deep neural networks continues to pose significant question.
is well-know that deep neural networks accomplish end-to-end learning frameworks through integrating feature learning with classifier training (Zhoul 202I). A neural network tailored for classification tasks can be divided into two components. The initial segment is dedicated to a fea- ture space transformation process by selecting information pertinent to the output while discarding information associated with the input. Specifically, the conversion of the original feature space rep- resented by the input layer to the final feature space represented by the final representation layer is referred to as the learned feature space, which is shaped by the acquired representations within fully-trained neural networks. The subsequent stage typically entails constructing a classifier based on this learned feature space, intending to complete machine-lcarning tasks.
In this paper, highlight the crucial role that noisy samples play in shaping the behaviour of deep learning models through a comprehensive exploration of the interplay among model width, noisy training data, and interpretation of the learned feature space. We demonstrated that while small and intermediate models follow the traditional bias-variance trade-off, over-parametern models which passed the interpolation threshold, interpolate more correct training data around noisy ones of matching classes. This phenomenon demonstrates a correlation with the double descent phenomenon, and we propose that this is attributed to models acquiring the ability to effectively 'isolate' noise from the information in the training dataset. We posit that this phenomenon has not been previously documented and could serve to elucidate the internal mechanism of double descent.
The main contributions of this study summarized follows. We proposed that there exists robust correlation between the neural network's interpolation strategy of noisy data and the dou- ble descent phenomenon. Utilizing the k-nearest neighbour algorithm, we indicate that this phe- nomenon results from broader architectures genuinely interpolating noisy samples among correct samples of the same original class, showcasing enhanced capability isolate noise information. Through conducting experiments on various neural architectures trained on MNIST and CIFAR- -O, we demonstrated that the label prediction accuracy of noisy samples strongly aligns with the test per- formance on unseen data. We believe our investigation can shed light on the mechanism by which over wverpprraeeterization enhances generalization.
2 RELATED WORK
The initial proposal of the double descent phenomenon as general concept was made by Belkin ett al. (2019) for increasing model size (model-wise). However, Nakkiran et al. (0221) have also shown similar trend w.r.t. size of the training dataset (sample-wise) and training time (epoch- wise). The research further reports observation of all forms of double descent most strongly in settings with label noise in the train set. While the double descent phenomenon widely discussed in these alternative perspectives, we focus our attention on the original model-wise phenomenon in this work, briefed as 'double descent" in the subsequent discussion.
Double Descent has been shown as a robust phenomenon that occurs over diverse tasks, archi- tectures, and optimization techniques. Subsequently, a substantial volume of research has been conducted to investigate the double descent phenomenon regarding its mechanism. A stream of theoretical literature has emerged, delving into the establishment of generalization bounds and con- ducting asymptotic analyses. particularly focused on linear functions (Advani al. 2020 Belkin et al. 2020; [Bartlett et al. 20200 Muthukkmar et al. 202T Mei & Montanari] 2022. Hastie et al.
2022
Hastie et al. (2022) highlight that when the linear regression model is misspecified, the best
2

---

Published as a conference paper at ICLR 2024
generalization performance can occur in the perrpparamtteriee regime. Mei & Montanari (2022)
further concluded that the occurrence of the double descent phenomenon can be attributed to model misspecification, which arises when there is mismatch between the model structure and the model family. While people believe the same intuition extends to deep learning as well, nevertheless, the underlying mechanism still eludes comprehensive understanding of "deep' double descent.
The prevailing literature that explores the double descent phenomenon in deep neural networks learning provides range of explanations: bias-variance decomposition (Yang al. 2020); samples to parameters ratio (Belkin et al.) 2020 Nakkiran et al. 202T]; decision boundaries (Somepalli
al. 2022); and the sharpless of interpolation (Gamba et al. (022). While offering an extensive
array of experimental findings that replicate the double descent phenomenon, Nakkiran et al. (202I) concluded that the emergence of double descent exhibits a robust correlation with the inclusion of
noisy labels in the training dataset. [Somepalli et al.] (2022) further studied the decision boundaries of random data points and established relationship between the fragmentation of class regions and the double descent phenomenon. Recently, Chaudhary et al. (2023) revisited the phenomenon and proposed that double descent may not inherently characterize deep learning.
Beyond the thorough examinations of double descent, emerging theory of wver-parameterized machine learning (TOPML) is growing, seeking to elucidate the second 'descent' in the double descent phenomenon (Dar et al. 2021). The inquiry posed by Zhou (202I) revolves around the puzzling aspect of why werrppaamettrrzzed models do not succumb to overfitting and focus on the feature space transformation aspect. Chang et al.] (2021) analytically identify regimes when training over-parameterized networks and then pruning is preferable to training small models di- rectly. Teague (2022) intriguingly explores the concept of implicit regularization arising from over- parameterization and draws a connection between the generalization capacity and the geometric properties of volumes within hyperspaces. The work by Liu al. (2022) introduces a robust train- ing method for over- weer-paramterized deep networks in classification tasks, specifically when training labels are corrupted. Although our research shares a thematic connection, our study concentrates on the implicit mechanisms through which over- parammttttteeed models acquire the ability to discern and handle label noise. Consequently, our focus differs from their research.
In contrast to existing studies, our approach distinguishes itself by exploring the phenomenon through the lens of the learned feature space generated by trained models. The most related works to ours are the concurrent ones by Gamba et al. (2022), , which measure sharpness clean and noisily- labeled data points and empirically showed that smooth interpolation emerging both for large over- parameterized networks and trained large models confidently predicts the noisy training targets over large volumes around each training point. While their study also entailed an analysis involving sep- arate evaluations of clean and noisily labelled data points, employed the k-nearest neighbour algorithm to infer the inter-relationship between these two categories of training data in the learned feature space and demonstrated that for over-parameterized models, noisy labelled data tends to be classified alongside its genuine counterparts. Our approach suggests that the double descent phe- nomenon consequence of deep learning models learning how to separate the noise from the information in the training set.
METHODOLOGY
We first assess the contribution of data noise to the double descent phenomenon. Bartlett et al. (2020) describes an intuitive way of understanding properties of significant over-parameteriizatonn in linear regression: fitting the training data exactly but with near-optimal prediction accuracy occurs if and only if there are many low-variance (and hence, unimportant) directions in parameter space where the label noise can be hidden. While non-linear models are different from linear models in terms of parameter space, we propose that neural networks approach optimal prediction through a similar mechanism of concealing label noise. We hypothesize that the emergence of the double descent phenomenon can be attributed to the progressively over- wer--arameterized models effectively isolating noisy data within the training set, thus diminishing the influence of interpolating these noisy data points.
Since this proposition is inspired by Nakkiran et al. (202)) 's empirical conclusion, initiate experimentation by replicating the double descent phenomenon as per the experimental framework outlined by this paper. This approach offers a comprehensively manageable setup and establishes
3

---

Published as a conference paper at ICLR 2024
comparative analysis of the phenomenon across diverse tasks and optimization techniques. With an increasing ratio of label noise introduced to the training dataset, the test error peak at the inter- polation threshold becomes evident, signifying definite correlation with the pronounced double descent phenomenon.
3.1 EXPERIMENT SETUP
We replicate the phenomenon of deep double descent across three frequently employed neural net- work architectures: Fully Connected NNs, Standard CNNs, and ResNet1 following the experiment setup of Nakkiran al. (202T).
Fully Connected Neural Networks (FCNN): We adopt a simple two-layer fully connected neural network with varying width & on the first hidden layer, for k in the range of [1, 1000].
The second fully connected layer adopted as the classifier.
Standard Convolutional Networks (CNN): We consider a family of standard convolutional
neural networks formed by 4 convolutional stages of controlled base width [k. 2k, 4k, 8k], for A in the range of [1. 64], along with fully connected layer as the classifier. The MaxPool 2, 2, 2, 4]. For all the convolution layers, the kernel size 3, stride 1, and padding 1. This architecture implementation is adopted from Nakkiran et al. (202T).
Residual Neural Networks (ResNet18); We parameterize a family of ResNet18s from He et al.] (2016) using 4 ResNet blocks, each consisting of two
layers. We scale the layer width (number of filters) of convolutional layers as [k, 2k, 4k, 8k] for varied range of k within [1, 64] and the strides are [1, 2, 2, 2]. The implementation is adopted from https: / whb............... com/ wwt.k/innnn/gtttaa//atae war....iia.................. git
We describe the experiment setup along with syper-parameten: we used in the training of all neural architectures below. All networks are trained using the Stochastic Gradient Descent (SGD) opti- mizer with zero momentum and Cross-entropy Loss for evaluation. A separate learning rate scheme employed for different models: Ir / [cppckcch500
0.05
for FCNNs and Ir (epoock)
0.05 for CNNs
and ResNet18s. For every model, the learning rate starts with an initial value of 0.05 and updates every 50 epochs. A batch size of 128 is employed, and explicit data augmentation or regular- ization techniques (including dropouts or pruning) are utilized in our experimental setup. Utilizing sequence of models trained under uniform training scheme showcases the double descent phe- nomenon across different models.
3.2 INTRODUCTION OF LABEL NOISE
While the presence of label errors has been established in the majority of commonly used bench- marking datasets including MNIST and CIFAR-IO (Zhang] 2017 Northcutt et al., 2021b), to fa- cilitate comparative analysis of our findings, we intentionally introduce explicit label noise. In experimental setup. label noise with probability p signifies that during training, samples assigned a uniformly random label with probability p. while possessing the correct label with prob- ability (1 p) otherwise. We interpret neural network with input variable x E Rd and learnable parameter 0, incorporating all weights and biases. The training dataset is characterized as a collec- tion of training points (Nin yn), where ranges from to N. A training dataset with N data points comprises m p x N noise labelled data points and m = (1 p) x N clean data points.
important note that the label noise is sampled only once and not per epoch, and this origi- nal and random label information is stored before each experiment. Each experiment is replicated multiple times, and the final result obtained by averaging the outcomes. Owing to the presence of inherent label errors. the probability is solely employed to signify the extent of explicit label noise and does not reflect the overall noise level present in the dataset. Our primary experimental conclusion is drawn from comparing results obtained under dataset with a relatively noisy level.
3.3 LEARNED FEATURE SPACE INTERPRETATION STRATEGY
Our primary research inquiry aims to gain insights into the diverse strategies employed by deep neu- ral networks of various widths in interpolating noisy labelled data in classification tasks. Based on
4

---

Published as a conference paper at ICLR 2024
the existence of benign aver-parameterizzaton,, assume that test images, akin to training images mislabeled, are more likely to be correctly classified by parameterizee models. For instance, anticipate that optimal classifier should yield accurate predictions for unseen images. it was trained on similar images with an adversarial label. Thus, we further hypothesize that the closest neighbours of mislabeled training images correctly classified for a model with better generalization performance. To validate our hypothesis, we adopted the k-nearest neighbour algo- rithm with cosine similarity to interpret the relative locations of clean and noisy labelled data in the learned feature space. This methodology allows us to delineate the prediction strategy through an in- direct approach and subsequently compare our findings with the overall generalization performance of these pre-trained models.
We consider I--layer neural networks, while representations on each layer are represented by fi(X), for 1 ...... T.J. Our investigation centred on the penultimate layer representations before clas- sification denoted as (X)) Acknowledging the difficulties of interpreting high-dimensional feature spaces, our approach mainly entails characterizing the relative positioning of training data points. This involves establishing a connection between the hidden features of noisy data points in the learned feature space and their k-nearest neighbours within the complementary subset of clean data. So we calculate the prediction accuracy P of mislabeled training data and the majority of its nearest neighbours in feature space in the same class:
Σ - by ( St. (xi,a))
P
m
In this context, (11, 91 ..... (Im, ymm denotes the subset of noisy training data with original labels, and (24,1, 1t,1), .... ( (i))) denotes the subset of clean training data neighbours I; for all = [I..... m]. Given the cosine similarity Sc B) == || between two representations A and B, let fL-1 (.F.,))....---11
A-B
be reordering of the representations of the clean training -... 5..i)))...... Sc(ff- fL- (.). The closest neigh-
data such that bours are selected, and a Majority function M ( ft- f describes the procedure for making predictions based on the majority label among the k- k-nearest neighbours. The predictions are subsequently compared with the original labels of noisy data, and P denotes the proportion of accurate predictions among all the noisy data.
In line with our hypothesis, the prediction accuracy P is indicative of the neural architecture's in- terpolation approach toward these noisy labelled data points. Due to the uncertain distribution of data and their representations, the k-NN method functions solely as estimation technique, and our comparison focuses on variations P.
In the next section, we present our empirical study and results of the learned feature space interpre- tation for trained neural networks of various widths in relationship to double descent.
EXPERIMENTS
In this section, we present our experiment results with variety of neural architectures of increasing size trained for image classification tasks on either MNIST or CIFAR-10 datasets.
4.1 MNIST
We commence our exploration by revisiting one of the earliest experiments that revealed the pres- ence of the double descent phenomenon, as presented in the work of Belkin et al. (201)). This experiment involved the training of basic two-layer FCNN on a dataset comprising 4000 samples from the MNIST dataset (N = 4000). When zero label noise (p 0%) is introduced, as depicted in Figure Ta), both the train and the test error curve decrease monotonically with increasing model width. The training error exhibits faster decline and approaches zero at approximately A 15, indicating the occurrence of an interpolation threshold under these conditions. The test error still decreases after this interpolation threshold, demonstrating benign prramttrettii...i.nnn... This no- ticeable alteration in the shape of the test error curve at the interpolation threshold may be attributed to the intrinsic label errors of MNIST (Northcutt et al. 2021a)
With label noise (p = 10% (2%% introduced in the training dataset, a more distinguishable test error peak starts to form around the interpolation threshold shown in Figure bb, e). Starting with small
5

---

Published as a conference paper at ICLR 2024
Doperneer bata to MTST 2t 4204
and Koull 1955 Owe30 p-1tes
Thel AGaro
28
1000
(a) FCNN on MNIST (p = 0%)
(b) FCNN on MNIST (p- = 10%)
(c) FCNN on MNIST (p- = 20%)
The phenomenon of double descent on two-layer FCNNs trained on MNIST (N = 4000), under varying explicit label noise ratios of p = [0%, 10,, 20%] and the prediction accuracy noisy labelled data denoted as P when p > O. Optimized with SGD for 4000 epochs and decreasing learning rate. The test error curve of p = [10%, 20%] performs the double descent phenomenon and the prediction accuracy P of in-context noisy data learning showed a correlation generalization performance beyond context.
Figure 1:
models, the test error curve exhibits U-shaped pattern in the first half and the bias-variance trade- off is reached at the bottom of the U-shaped curve. The decrease of both train and test error before this bias variina trade-off indicates the model undergoes under-fitting stage. With the expansion of the model size, the model transitions into an overfitting regime, leading to peak in the test error around the interpolation threshold, while the train error converges to O. Finally, after surpassing this test error peak, the testing loss experiences a decline once more showcasing how highly over- parameterized models can ultimately yield improved generalization performance. This intriguing behaviour is referred to as double descent. As the test loss undergoes a pattern of first decreasing, then increasing, and finally decreasing again, the test accuracy exhibits the opposite trend, i.e., first increasing, then decreasing, and then increasing once more. As Nakkiran et al. (202T) reported, the test loss peak is positively correlated with the label noise ratio p. In other words, as the label noise ratio increases, the peak in the test loss also tends to increase accordingly.
To comprehend the influence of noisy labelled data on the variation of the learned model during the double descent phenomenon, we examine the percentage P of noisy data, and the majority of its nearest neighbours in feature space are in the same class. In this experiment setup, as depicted in Figure c), the trajectory of the P curve follows the pattern observed in the test accuracy curve, indicating a consistent alignment between the two. When test loss first decreases as the model ex- pands, test accuracy and P both rise. With the introduction of label noise, test accuracy decreases while P fluctuates around 70%. When passing the interpolation threshold, both the test accuracy and P significantly rise 85% and nearly 100%, substantially affirming the validity of our hypothesis. In the meantime, the trend of test loss is opposite to both test accuracy and P. while changing cor- respondingly before and after the interpolation threshold. The computation of percentage P relies solely on the training dataset and does not incorporate unseen data. Consequently, this percent- age may be regarded as a weak predictor of generalization performance, though it necessitates knowledge of the distribution between clean and noisy data.
Recall that the noisy data points will be predicted based on their randomly assigned labels during this stage as the model fully interpolates the training set, we interpret the ascending P curve in the parr-parameterized region as valid 'isolation' of the noisy labelled data in the learned feature space. When the neural network positions the learned representation of noisy data points closer to the class associated with their 'incorrect' labels, it results in lower P prediction accuracy. Under these circumstances, similar test samples also tend to be predicted incorrectly due the influence of these noisy labels. If the neural network successfully identifies the underlying data patterns and interpolates the correct samples around noisy data points, the prediction accuracy P will be high by
6

---

Published as a conference paper at ICLR 2024
nearby class members. Consequently, test samples resembling the noisy training data have higher likelihood of being accurately identified due to their proximity to the interpolated surrounding clean data points and becoming less susceptible to the influence of the 'isolated' noisy labelled data. This explanation accounts for the improved generalization performance for over-parameterized models.
4.2 CIFAR-10
Eapet CEAGLE 55000
Expartmert: Aacuta Q3A130 53008
Expettrand Fazz CIARLE (9- 10-50000
14
wih mo
L3W00 cee Lser nittoo
(a) CNN on CIFAR- (p 0%)
(b) CNN on CIFAR-IO (p = 10%)
(c) CNN on CIFAR-I0 (p- = 20%)
The phenomenon of double descent on five-layer CNNs trained on CIFAR-10 (N 50000), under varying label noise ratios of p = 10%, 10%, 20%] and the prediction accuracy of noisy labelled data denoted as P when p > 0. Optimized with SGD for 200 epochs and decreasing learning rate. The test error curve of p = [0%, 10% 20%] performs the double descent phenomenon and the prediction accuracy P of in-context noisy data learning showed correlation generalization performance beyond context.
Figure 2:
After a baseline case of deep learning with FCNNs on MNIST, we extended the experiment results to image classification on CIFAR-10, using two famous neural network architectures: CNN and ResNet. We first visit the experiment when the complete CIFAR- 10 dataset (N 50000) with zero explicit label noise (p- 0%) is trained with five-layer CNNs: we observe that in Figure 4.2 a), proposed double descent peak arises around the interpolation threshold. We posit that the variance in observations could stem from higher ratio of label errors in CIFAR-10 compre10comparee to MNIST, as sug- gested by research estimates (Zhang 2017 Northcutt et al. 2021b). Nevertheless, when additional label noise (p- = 20%) is introduced, the test error peak still increases accordingly, proving the positive correlation between noise and double descent according to Figure (.2bb)),2((b.). While label noise inereases the difficulties of models fitting the train set and shifts the interpolation threshold rightwards, the test error peak also shifts correspondingly.
When we review the noisy labelled data prediction test in the learned feature space, we can also observe an aligned correlation between the trend of generalization performance and prediction accu- racy P. When test accuracy first increases and then decreases as the model expands, P rises about 60% and then decreases before the threshold when the peak of test error arises. Then, as the test error reduces and test accuracy rises, the P curve rises again about 60% for paertparaeterrized models. The observed correlation between the trend of test loss and P remains consistent. The same conclusion can be derived from the experiment context Figure of ResNet18s trained on the sample task. An important observation is that the prediction accuracy P demonstrated in the learn- ing experiments of CIFAR-10 is significantly lower compared to those on MNIST. We attribute this variance to the intricate feature space representations, which are more challenging capture us- ing cosine similarities. Our analysis focused more on how the interpolation strategy varied across the same neural architecture of different widths, and the k-NN methodology can only serve as an interpretation technique.
In conclusion, by constructing P represent the distribution of noisy labelled training data in the learned feature space, we propose that this k-NN prediction ability of the model is connected to the size of the model in the same way with double descent. Furthermore, We suggest that this correlation
7

---

Published as a conference paper at ICLR 2024
can be attributed to the model's interpolation of correct samples around the noise labels, effectively isolating the misleading effect of the latter.
Coperment Desutt CFC PA5-20 -50000 p-1h1
supartiinnt auc M4LSON pailin
Expetmard Pare on PAB (1- 993900
13
32 16
(a) ResNet18 on CIFAR-10 (p = 0%)
(b) ResNet18 on CIFAR-IO (v 10%)
(c) ResNeetl88 on CIFAR-10 (p 20%)
Figure 3:
The phenomenon of double descent on ResNet18s trained on CIFAR-10 (N = 50000),
under varying label noise of p [0%, 10%, 20%] and the prediction accuracy P when p > 0. Optimized with SGD for 200 epochs and decreasing learning rate. The test error curve under p = (0%, 10%, 20%] performs the double descent phenomenon and the prediction accuracy P of in- context noisy data learning showed a correlation to the generalization performance beyond context.
CONCLUSION
In this paper, we have studied the feature space of learned representations projected by various neu- ral networks. By analyzing the behaviour of the model when achieving perfect fit on the training set, we observe that: as the model's 's complexity and capacity increase, there is clear and distinctive isolation of noisy data points within their original 'class-mates' in the learned feature space. This intriguing observation indicates that broader neural networks interpolate correct samples through- out the vicinity of noisy samples the learned feature space, effectively 'isolating' the impact of introduced label noise without explicit regularization techniques. We believe this implicit property plays a significant role in understanding how parametrrizaation enhances generalization. We posit that the mechanism behind this phenomenon can be attributed to two key factors. First, the gradient descent optimization algorithm endeavours to strike a perfect balance among training data points to minimize the loss function, which can lead to the isolation of noisy data points. Second, the characteristics of high-dimensional space may facilitate a beneficial separation of noisy data points from clean data in the learned feature space.
From the perspective of this paper, the observed double decent phenomenon is associated with im- perfect models, first behaving as predicted by the classical learning theory and increasing their error on the test set until they reach full interpolation on the train set. Then, they make use of the extra parameters above the interpolation point to better separate noise from information. We intentionally referred to 'imperfect models' because we claim that double descent should not be observable in perfect learners where perfect regularization should eliminate the effect. From our perspective, the double descent phenomenon is strictly related to imperfect models learning from noisy data.
Let's expand on this by restricting our considerations to supervised learning for classification prob- lems: For an ideal training set with full information (no noise and no missing cases), a learner will learn the exact model at the interpolation point where the train set is exactly classified and the error becomes zero. In this case, at interpolation, also the test set will be exactly classified because the learner has learned the exact map. Before that interpolation, the error decreases with every new example. Clearly, this system does not show any double descent. At the interpolation point, the number of model parameters is approximately equal to the number of examples in the complete train set (i.e. the model is the matching table).
8

---

Published as a conference paper at ICLR 2024
No double descent peak
Double descent peak
Intorydation:
Nose solation:
Inturpelation ralay data ere fitee
Noss belation:
mosydala are Sitel
nosy are denacted and lnclited
tosy data are
derected inclited
Error
Error
Imporfert arreers
regskarloed imporfcc
Perfoct fournol
if nodel porometers is deompees
#mmssal parumetar cearr
Emodelpanmetres D. eoappeer
#imddal peramatur aut cxampres
Model complexity
Model complexity
Figure 4: Exemplification of the emergence of the double descent effect in the perspective of the present paper. Starting from the bottom of the left panel: 1. the perfect learner, in the absence of noise, will learn the perfect model containing several parameters approximately equal to the number of significant examples; in the presence of noise, the perfect learner needs to learn also to distinguish information from noise and it will learn perfect model with several parameters which higher the number of noisy examples; 3. the imperfect learner with optimal regularization will learn a well well-perroriing model which distinguishes information from noise using several parameters which higher the number of examples but should not show double descent peak. On the right panel: 4. the imperfect learner with sub-optimal regularization will learn a well-performing model first interpolating noise and signal and then learning to distinguish information from noise using several parameters which is higher the number of examples showing the double descent peak; 5. the larger is the noise, the higher the double descent peak is, and the larger is the number of parameters needed to distinguish information from noise.
In the presence of noise, the learner cannot Iearn the full map by interpolation. Indeed, the bias- variance trade-off predicts a U-shaped performance curve with first the learner increasing out-of- sample performances from digesting examples and then losing performances by trying to fit the in-sample noisy data. However, learner who makes good use of regularization can do better and learn how to account for uncertainty. Likely, the number of model parameters needed to learn both the underlying map and to handle the noise is larger than the number of parameters needed to learn only the map the ideal case without noise (the interpolation point). If the model learner with perfect regularization, we do not expect the manifestation of any double descent, but rather gradual improvement in out-of-sample performances with the number of parameters until a saturation point is reached where performances cannot be improved without further examples. We expect that this saturation point should occur at a number of parameters larger than the interpolation point. Double descent instead observed for imperfect learners who first try interpolate the noise and then start an intrinsic regularization via vver-parameterization.. Our particular model of the emergence of the double descent phenomenon is illustrated In Fig.
perfect
While the present study provides valuable insights into the mechanisms leading to the occurrence of the double descent phenomenon, a comprehensive theoretical framework is yet be developed. Further investigations are needed to unravel the intricate details and offer a deeper understanding of this intriguing phenomenon. Another aspect that warrants attention is the impact of the architectures of deep neural networks. Studies have demonstrated that neural networks exhibit diverse behaviours based on their depth and width, even when the total number of parameters is identical. It piques our curiosity to investigate whether this variation in network shape could influence the underlying mechanisms of the double descent phenomenon. Another potential avenue for future exploration could involve implicit sparsity. We have observed that feature representations tend to exhibit sparsity in over paraamettttri neural networks, particularly when dealing with high-dimensional feature spaces.
Reproducibility Statement To facilitate reproducibility, we provide a detailed description of the neural architectures and full training details including exact settings of hyper-parameters in Sec- tion 3 Our code is provided publicly available at https: /ttp///ttttbb/t/ com/ double _elle__eecent__.e wnn.inffeenc..

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
The authors acknowledge the discussion with Siddharth Chaudhary, Joe Down, Nathan D Souza, and Huijie Yan, all UCL students who took part the academic year 2022-23 project COMP003 I. With them, we had invaluable discussions and great inspiration. This paper originated from that project and would not have been conceived without Please see CCauddhary et al. (2023) for the reference to that work. The author, T.A., acknowledges partial financial support from ESRC (ES/K002309/1), EPSRC (33//031730/1), and EC (H2220-ICT-2018- 825215). X.Z. knowledges partial support from the National Natural Science Foundation of China (No. 62076068).
REFERENCES
Madhu S Advani, Andrew M Saxe, and Haim Sompolinsky. High-dimensional dynamics of gener- alization error in neural networks. Neural Networks, 132:428 446, 2020.
Peter I. Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117(48):30033-30077 2020.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, (16(3)):5849--5588
2019.
Mikhail Belkin, Daniel Hsu, and Ji Xu. Two models of double descent for weak features. SIAM Journal on Mathematics of Data Science, 1677:116, 2020.
Xiangyu Chang, Yingeong Li, Samet Oymak, and Christos Thrampoulidis. Provable benefits of overparameterization in model compression: From double descent to pruning neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 6974-6983, 2021.
Siddharth Chaudhary, Joe Down, Nathan D'Souza Yufei Gu, and Huijie Yan. Parameter-wise dou- ble descent-a unified model or not? Available at SSRN 4469495, 2023.
Yehuda Dar, Vidya Muthukumar, and Richard G Baraniuk. A farewell to the bias-variance tradeoff? an overview of the theory of overparameterized machine learning. arXiv preprint arXiv:2109.02355 02..0090.025.... 2021.
Zeyu Deng, Abla Kammoun, and Christos Thrampoulidis. A model of double descent for high- dimensional binary linear classification. Information and Inference: Journal of the IMA, 11(2): 435-495, 2022.
Matteo Gamba, Erik Englesson, Mårten Björkman, and Hossein Azizpour. Deep double descent via smooth interpolation. arXiv preprint wiXiv:2209.1008/ 2022.
Stuart Geman, Elie Bienenstock, and René Doursat. Neural networks and the bias/variance dilemma. Neural computation, 4(1):1-58 1992.
Trevor Hastie, Robert Tibshirani, Jerome Friedman, and Jerome Friedman. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high- dimensional ridgeless least squares interpolation. Annals of statistics, 50(2):949, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Sheng Liu, Zhihui Zhu, Qing Qu, and Chong You. Robust training under label noise by over- parameterization. In International Conference on Machine Learning, pp. 14153-14172. PMLR, 2022.
Marco Loog. Tom Viering, Alexander Mey, Jesse H Krijthe, and David MJ Tax. A brief prehistory of double descent. Proceedings of the National Academy of Sciences, 17(20): 10220006222200 2020.
10

---

Published as a conference paper at ICLR 2024
Song Mei and Andrea Montanari. The generalization enror of random features regression: Precise asymptotics and the double descent curve. Communicatioon on Pure and Applied Mathematics, 75(4):667-766, 2022.
Vidya Muthukumar, Adhyyan Narang, Vignesh Subramanian, Mikhail Belkin, Daniel Hsu, and Anant Sahai. Classification vs regression in overparameterized regimes: Does the loss function matter? The Journal of Machine Learning Research, 22(1):10104-100 10172, 2021.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep
double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, (021(12):12400)) 2021.
Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70: 1311-1441, 2021a.
Curtis Northeutt, Anish Athalye, and Jonas Mueller. Pervasive label errors test sets destabilize machine learning benchmarks. arXiv preprint arX1v:2/03.14749, 2021b.
Manfred Opper. Statistical mechanics of learning: Generalization. The handbook of brain theory and neural networks, pp. 922-925, 1995.
Manfred Opper. Learning to generalize. Frontiers of Life, 3(part 2):763-775, 2001.
Gowthami Somepalli, Liam Fowl, Arpit Bansal, Ping Feh-Chiang. Yehuda Dar, Richard Baraniuk, Micah Goldblum, and Tom Goldstein. Can neural nets learn the same model twice? investigating reproducibility and double descent from the decision boundary perspective. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 13699-13708, 2022.
Nicholas Teague. Geometric regularization from overparameterization.
arXiv preprint
arXiv:2202.0922 2022.
Vallet, J-G Cailton, and Ph Refregier. Linear and nonlinear extension of the pseudo-inverse solu- tion for learning boolean functions. EPL (Europhysics Letters), 9(4):315, 1989.
Zitong Yang, Yaodong Yu, Chong You, Jacob Steinhardt, and Yi Ma. Rethinking off for generalization of neural networks. In International Conference on Muchine Learning, pp- 10767-10777. PMLR, 2020.
trade-
Xinbin Zhang. A method of data label checking and the wrong labels in mnist and cifar10. Available at SSRN 3072167, 2017.
Zhi- Hua Zhou. Why over-parameterization of deep neural networks does not overfit? Science China Information Sciences, 64:1-3, 2021.
A EXPERIMENT DETAILS
A. SCALING OF MODEL PARAMETERIZATION
The scaling of model parameterization to the universal layer width unit K shown in Figure In our experiment results, we labelled the x-axis scale as the model width parameter k instead of the scaling of parameterization. However, we believe this labelling does not impact our conclusion because of the monotonic relationship between these two factors.
A.2 MODEL PERFORMANCE IN THE PRESENCE HIGH LABEL NOISE
Here, we provided two additional experiment results when high label noise ratio 40% is introduced to the training dataset of MNIST and CIFAR-10 that, due to page limitations, could not be included in the main context. Noise has substantial impact the model's classification accuracy, leading to a reduction in test accuracy. It is evident that the prediction accuracy curve P exhibits significant instability when subjected to high percentage of noisy samples, with correct samples becoming sparser in the learned feature space. However, the correlation between and generalization performance remains consistent.

---

Published as a conference paper at ICLR 2024
144
100
107
100
000
Layer Walth 31
Loyer Wlath
Laypt 09.00
(a)) FCNN
(b) Five-layer CNNs
(c) ResNet18s
Scaling of the number of parameters as model size with layer width unit of the three neural architectures used in experiments in the Methodology section ??. We apply logarithmic scale to the parameter counts P of all neural architectures.
Figure 5:
Experiment Results MNIST (N-4000, pu40091
1.0
2.0
KM Labe CoS. 12077 (2077A)
Ath Lib 2000 00 1120000
A00/e0 (32044)
0.4
0.2
Accurnty
indiztionn
indiztoonn accuracy
0.0
0.2
Nf Nouron (0
Convokutionalagg
50
Lioo
Luoee
Cross ESINDy
1.5
.00
250
300
1500
Mamber 51idin Naturomn
(a) FCNN on MNIST (p = 40%)
(b) CNN on CIFAR-10 (p= 1p=440 40%)
Figure 6: The phenomenon of double descent and the prediction accuracy P when high label noise of p = 40% is introduced on the training dataset.
12


Reviews:
[{"review_id": "ag9Zw8L0hd", "reviewer": "Reviewer_P23K", "summary": "The paper performs a series of empirical studies into the double descent phenomena. The main take-away is that double descent appears when the model is faced with label noise.", "questions": "* What is the 'wedge product' in Eq. 1?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(I stress that I have low confidence in my review. It is highly likely that I missed something important)\n\n* The double descent is, currently, highly counter-intuitive to many, so any insights into the phenomena are valuable.\n* The stated hypothesis regarding label noise is clear and seems intuitive.", "weaknesses": "(I stress that I have low confidence in my review. It is highly likely that I missed something important)\n\n* As a non-expert, I interpret the experiments as showing that double descent phenomena appear with label noise in large models. I acknowledge that this is an interesting observation, but it does not seem sufficient to justify conclusions such as double descent not appearing in well-regularized models. I can see the intuition, but, in my non-expert view, the experimental data seems insufficient.\n* The paper considers only a small selection of data sets and models. It is not clear to me if such is sufficient to draw conclusions.\n* The paper does not provide a theoretical explanation (which is fine, but then I had wished for a wider selection of experiments).\n* I found it very difficult to read the t-SNE plots.\n* [minor] It appears that the ICLR formatting instructions were disregarded when changing the font size of the paper title.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zzWumKwgFB", "reviewer": "Reviewer_9d7L", "summary": "This article focuses on \"Double Descent\" phenomenon, suggesting that the ability of a model to adapt to and recognize noisy data has a significant impact on its overall generalization performance and contributes to the phenomenon of double descent, and proposing the use of a k-nearest neighbor algorithm to infer the relative positions of clean and noisily labeled data in the learned feature space.", "questions": "See weakness.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1.This paper is feasible to understand the generalization ability and double descent phenomenon through the perspective of noise.", "weaknesses": "1.The methodology section of this paper (only 14 lines of text and only 6 lines of description of the actual methodology) is poorly described and lacks math-related descriptions, making it difficult to accurately understand the author's intention.\n\n2.Is it optimal to use K-nearest neighbors here? the feature space is probably in a high-dimensional Manifolds, have the authors considered more complex cases?\n\n3. I don't think this article convinces me enough about the mechanism by which it reveals that over-parameterization enhances generalization. The mechanism between over-parameterization and generalization cannot be revealed in depth only by the loss curves and T-sne images under a small number of experiments.\n\n4. It is well recognized that over-parameterization is beneficial, especially since recent large models have similarly benefited from over-parameterization, and the authors should discuss the advantages of this paper in the context of recent over-parameterization research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V9xQGxCmMR", "reviewer": "Reviewer_d6tR", "summary": "The research aims to show that the phenomenon of double descent with respect to model size appears in neural networks due to the addition of noisy labels, i.e., smaller models learn this noise, while larger size models are able to avoid it and recognize wrong labels while learning useful information. The proposed approach to this aim is to measure the nearest neighbor prediction for mislabeled examples based on learned representations (on the penultimate layer) and then demonstrate that this prediction corresponds to the original (true) label, no matter that network \"correctly\" predicts wrong label. t-SNE visualizations of representations are used as an additional evidence to this point.", "questions": "1 - What is the main goal of the research done?\n\n2 - What is the observation made with respect to the selected metric (nearest neighbors) from the double descent plots?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Investigation of the ability of large neural networks to interpolate noisy labels, but nevertheless have good generalization is an important contribution to the current research. The mechanisms behind this phenomenon shed light on the generalization abilities of such models.", "weaknesses": "The abstract of the paper states that the main contribution is to demonstrate that model-size double descent is happening only with label noise present in the dataset. Nevertheless, the experiment with CIFAR100in the paper directly disproves this claim already. Further, it is claimed that double descent will not happen in the correctly regularized networks, but no regularization is ever discussed in the paper itself, except for an unusual claim that overparameterization is a form of regularization (which is also never discussed in the paper). Together, this leaves only one possible contribution - explaining how interpolating networks deal with mislabeled examples. This is aimed to be explained through nearest neighbors classification on the representations for mislabeled examples - showing that this accuracy is very high for interpolating models. Nevertheless, only the MNIST experiment indeed demonstrates high accuracy of these predictions together with the high accuracy on noisy task, in other experiments this accuracy is considerably low. t-SNE visualizations can be simply explained by the ability of larger models to get a more distinguished representations for different classes, but the positioning of the noisy labeled ones does not change much (see class 0 in fig.2 for example). Thus, the last contribution is not significantly supported by the results.\n\nMinor:\n\n- please use \\citep and \\citet for the citations outside of the sentence and in the sentence\n\n- please check the grammar and typos in the paper \n\n- please improve the explanation of the formula (1), it is currently extremely convoluted\n\n- the font of the title does not correspond to ICLR style", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P8xhLddXZx", "reviewer": "Reviewer_Ea52", "summary": "This paper presents an empirical study of the relation between model capacity and generalization, while considering the learned feature space. In particular, it analyzes the phenomena of double descent, in which test performance improves after a certain interpolation threshold, contrary to the traditional U-shaped curve.  This work argues that models with small capacity first overfit to the noise present in the data and that overparameterized models then learn to “recognize” noisy labels, separating signal from noise.", "questions": "-", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "In contrast to most previous empirical works on double descent, which only analyze cross entropy loss, double descent in terms of accuracy is also studied. As shown in their plots (see Figures 1, 3 and 5) the double descent peak is considerably smaller (sometimes even absent) when looking at accuracy. tSNE visualizations give more intuition about the phenomenon, and experiments are performed for 3 models and 3 noise levels and 2 datasets.", "weaknesses": "- One main weakness is the fact that a big part of the paper is devoted to reproducing experiments from [1]. In these experiments, the only significant difference is the fact that they report both accuracy and cross entropy loss.\n\n- I fully agree with the author’s comment on section 3 regarding tSNE maps: “While this visualization may not accurately represent the intricate inherent structure of high-dimensional features, it aims to gain insights into the clustering and distribution of data points, thereby enhancing our understanding of the model’s internal representations.” While tSNE maps might give intuition about a phenomenon, they can also fabricate patterns and might not be sufficient evidence for a proposition.\n\n- The writing is sometimes confusing. For instance, in section 3.3: “we calculate the prediction accuracy Kp of noisy labelled data of the original clean labels for each noisy labeled data point matching the label prediction of its k-nearest neighbors in the learned feature space”. \n\n Thus, I do not see substantial empirical nor theoretical contributions in this work. \n\n[1] DEEP DOUBLE DESCENT: WHERE BIGGER MODELS AND MORE DATA HURT Nakkiran et al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)