The Final Ascent: When Bigger Models Generalize Worse on Noisy-Labeled Data
Authors: Yihao Xue
Kyle Whitecross
Baharan Mirzasoleiman
Venue: Submitted to ICLR 2023 (2023)
Keywords: supervised learning
generalization
overfitting
memorization

Abstract: Increasing the size of overparameterized neural networks has been shown to improve their generalization performance. However, real-world datasets often contain a significant fraction of noisy labels, which can drastically harm the performance of the models trained on them. In this work, we study how neural networks' test loss changes with model size when the training set contains noisy labels. We show that under a sufficiently large noise-to-sample size ratio, generalization error eventually increases with model size. First, we provide a theoretical analysis on random feature regression and show that this phenomenon occurs as the variance of the generalization loss experiences a second ascent under large noise-to-sample size ratio. Then, we present extensive empirical evidence confirming that our theoretical results hold for neural networks. Furthermore, we empirically observe that the adverse effect of network size is more pronounced when robust training methods are employed to learn from noisy-labeled data. Our results have important practical implications: First, larger models should be employed with extra care, particularly when trained on smaller dataset or using robust learning methods. Second, a large sample size can alleviate the effect of noisy labels and allow larger models to achieve a superior performance even under noise. 

Under review as a conference paper at ICLR 2023
THE FINAL ASCENT: WHEN BIGGER MODELS GENER- ALIZE WORSE ON NOISY-LABELED DATA
Anonymous authors Paper under double-blind review
ABSTRACT
Increasing the size of overparameterized neural networks has been shown to im- prove their generalization performance. However, real-world datasets often contain a significant fraction of noisy labels, which can drastically harm the performance of the models trained on them. In this work, study how neural networks' test loss changes with model size when the training set contains noisy labels. We show that under a sufficiently large noise-to- noise-to-sample size ratio, generalization error eventually increases with model size. First, we provide a theoretical analysis on random feature regression and show that this phenomenon occurs as the variance of the generalization loss experiences second ascent under large noise-to-sane samsle size ratio. Then, we present extensive empirical evidence confirming that our theoretical results hold for neural networks. Furthermore, we empirically observe that the adverse effect of network size is more pronounced when robust training methods employed to learn from noisy-labeled data. Our results have important practical implications: First, larger models should be employed with extra care, particularly when trained on smaller datasets or using robust learning methods. Second, a large sample size can alleviate the effect of noisy labels and allow larger models achieve a superior performance even under noise.
INTRODUCTION
Modern neural networks of ever-increasing size, with billions of parameters, have achieved an unprecedented success in various tasks. However, real- world training datasets are usually unlabeled and the commonly used crowd-sourcing and automatic-labeling techniques can introduce a lot of noisy labels (Krishna al. 2016). wver-parameterized models can easily overfit the noisy labels and suffer drastic drop in their generalization performance (Zhang et al., 2016). This phenomenon has
inspired recent body of work on dealing with high levels of noisy labels (Han et al. 2018 Zhang
& Sabuncu| 22018: Jiang et al. 2018 Mirzasoleiman et al. 2020 Liu et al. [0200] Li et al. 2020).
However, the effect of model size on neural networks' generalization performance on noisy data has been overlooked, and the following important question has remained unanswered: can large models
trained on nabse-l data be trusted and safely used?
Contradictory to the classical view of bias-variance trade-off, increasing the size of overparameterized neural networks only improves generalization Neyshabur et al. (2014). To explain this, (Belkin et al. 2019) proposed the the double descen! phenomenon, suggesting that the test error follows a U-shaped curve until the training set can be fit exactly, but then it begins to descend again and reaches its minimum in the overparameterized regime. This has further been investigated by body of recent work (see ( focusing on confirming or reproducing the double descent curve. Here, instead show that label noise change the above picture and introduce a final ascent to the monotonically decreasing loss curve in the overparameterized regime. Specifically, we analyze the generalization performance of models of increasing size, in terms of both network width and density, under varying sample size and levels of noisy labels. We find that,
when noise-to- sumpple--su ratio is sufficiently large, increusing the width or density of the model beyond a certain point only hurts the generalization performance.
We first provide theoretical evidence from random feature regression studied in prior work (Yang et al. 2020) and show that the test generalization loss can be decomposed into decreasing bias,

---

Under review as a conference paper at ICLR 2023
unimodal iniee-indeeenddnt variance and an increasing noise-dependent variance. The noise- dependent variance is more pronounced when the voise-to-sample size ratio is large, which leads to a second ascent in the total variance (c.f. Figures [I] and 2 and an ascent in the test loss. Interestingly, our analysis also demonstrates that under a large by keeping randomly selected fraction of weights can improve the generalization. Our analysis complements the double-descent phenomena, by providing a complete picture of the generalization curve vs. model size under various levels of noisy labels.
size ratio, reducing model density
Through extensive experiments, we corroborate our theoretical results and show their validity for neural networks by showing that (1) sufficiently large label noise can lead to the final ascent in test loss and (2) sufficiently large sample size can eliminate the final ascent. In addition, we study the effect of model size on the performance of state-of-the-art methods for robust learning against noisy labels. We show that, perhaps surprisingly, the adverse effect of larger models can be observed even under a smaller noise-to-see voiselto-sample size ratio, when robust methods are employed. Finally, we take closer look into the smoothness of the learned networks trained with noisy labels. We show that noisy labels can turn the previously suggested negative correlation between network size and model complexity into positive one.
Our work is a step toward understanding the complicated overfitting and generalization behavior of modern machine learning models trained on large real-world data. In practice, our results also have several important implications: First, reducing width or dropping a fraction of (even randomly selected) weights can alleviate the effect of noisy labels. Second, when training large models, larger sample size can effectively counter the effect of noisy labels and even remove the final ascent. This explains why the harm of larger models on noisy-labeled data have not been observed by prior work (Arpit et al. 2017). Finally, large models should be used with extra care even on large datasets when training with robust methods, as larger width or density can hurt the performance of the robust learning algorithms.
2 RELATION TO PRIOR WORK
A bulk of recent work has studied the double descent phenomena by analyzing the error of linear regression (Hastie et al. 2019 Belkin et al. 2020 Derezinski et al. 2020). random feature regression (Mei & Montanari] 2019 Adlam & Pennington, 2020a Yang et al. 2020 d'Ascoli et al. (020) or even NTK regression/ Adlam & Pennington| (2020b). showing these models can capture some important features of double descent. The primary focus of these works has been on the behavior of the total test error. few of them decompose the test error into bias and variance (Mei & Montanari, 2019 Yang et al. 2020). Most relevant to our analysis are (Adlam & Pennington 2020a d Ascoli et al. 2020 which decompose the variance into several sources. However, we take different asymptotic limit than theirs. Given training examples and d input dimensions, we let ă tend to infinity instead of constant. The benefit of this limit is that we can derive closed-form expression of different terms in the test error and therefore obtain more interpretable result. Our setting is particularly comparable to that of (Yang et al. 2020) which analyzed the noiseless case, in the same asymptotic limit. Our result shows that label noise contributes monotonically increasing term in the variance, in addition the bias and noise-independent variance term derived in (Yang et al. 2020). Furthermore, we include model den- sity into the analysis by adding another layer and show that density plays different role than width. In noiseless setting, our Theorem Saanrrem@rrppee reproduces the empirical observation made by Golubeva et al. (2020) that increasing width while keeping the number of parameters fixed by reducing density improves gen- eralization (see Appendix A.7), which separates the benefit of width from the effecct of model capacity.
The role of label noise in double descent has been discussed before in either empirical Nakkiran et al.) 2202)) or theoretical Adlam & Pennington (2020a) studies. However, these works focus on the settings where double descent still holds and only highlight that label noise exacerbates the peak of test loss at the interpolation threshold. In contrast, we find that label noise essentially change the monotonicity of the loss curve by adding a final ascent. Furthermore, we show that this ascent can be removed by using sufficiently large sample size.
2

---

Under review as a conference paper at ICLR 2023
K = 0.00
= 0.60
Variance (theory)
0.20
0.3
0.4
0.15
0.3
0.10
0.2
Vontaice VanantRaae
0.3
0.6
0.05
0.1
Variance
0.8
0.00
0.0
0.2
10
10
10a
10
100
10*
p/d
p/d
0.1
Variance (experiment)
noise ratio
0.0
noise ratio 0.95
0.15
0.20
0.0
101 10
0.10
0.15
Width
Figure Variance ResNet 34 on CIFAR-10 under different levels of la- bel noise. For noise levels below 0.5, the variance curve is unimodal. For noise levels 0.6 and 0.8, the variance curve is unimodal until width 128 and after this point begins to ascend again.
0.05
0..00
10
102
10
10"
10
102
10a
10
Width
Width
Figure 1: Top: the variance curve plotted based on the expressions Theorem and Appendix A.2 Bottom:
variance of ReLu networks trained on MNIST.
THEORETICAL ANALYSIS OF RANDOM FEATURE REGRESSION
We first provide theoretical evidence for our main finding by analyzing the risk of a two layer neural network with random first layer. Our result builds on (Yang et al. 2020), which studied the case where the model is trained without label noise. In particular, we extend their result to the noisy setting and show that the last ascent can be understood through the lens of bias-variance decomposition. Moreover, in Section 3.2 we show that with simple modification to the setting, model density into the analysis. It turns out that reducing the density of wide models is equivalent to using stronger regularization in the limit, which leads to better generalization in noisy settings.
include
3.1 THE SECOND ASCENT IN VARIANCE CAUSED BY LABEL NOISE
Suppose we have a training dataset (X, yy where X = [a,, .2,.... .a......a Fn, and y= [wn, V2..... yn with x; IRd and y/ I are feature vector and label point data point We assume each æ; is independently drawn from Gaussian distribution (O, Ia/d) and labels are generated by yi = a. 0 c; where 8 Rd with entries independently drawn from N 1) and c; is the label noise drawn from ((0, a²) for each I;. The learned function is given by random feature ridge regression. Specifically, we learn two-layer linear network where the first layer W E Rexd has each entry randomly drawn from (0, 1/d) and the second layer is given by
3 = arg min ([ X) x) 3 yl ,ERR
|lllaa = (WXX W + " 0+
where = [EL, E2. 62,, c,. eT
Given clean test example (x, y) where a ~ N'O, Le/d) and x 0. the prediction of the learned model is given by f(x) (Wa)) T B. The expected risk then be written as
Risk
EgE x W. (((()()) - y)² Eo (Ex ((x.).e((() w)²
EpE V x (.xl(x))
(1)
Blas
Variance
B ā B...BB. B 112 Ex.w Ex.
Ex, E|
2
Varianceaisn
Varianceaiie
where A W (WXXTT) T - WX and B = AX See the derivation in Appendix
A.I Here we first decompose the risk into bias and variance and then further decompose the variance into terms where the second term Variance captures the role of label noise.
3

---

Under review as a conference paper at ICLR 2023
Our analysis is performed in the nigh-dimensional asymptotic limit where n, n,d.p p oc, with à v.
5 = 7, [n/d] K held constant. To further simplify the analysis we let d = co. Note that here the noise-to-san sample size ratio is still finite. Yang et al. (2000) analyzed bias-variance decomposition in the same setting without considering label noise. Therefore we can simply use their expressions for Bias and Variance,iean in Appendix A.2 Bias is monotonically decreasing and Varianceciemn is unimodal. The explicit expression for Variance is given below.
Theorem 1. For a 2-layer linear network with hidden neurons and a random first layer, consider learning the second layer by ridge regression with parameter X on training examples with feature for some fixed X and k. The asymptotic expression (where n, d. + 00 with z = 30 and 2/2 = Q of Variance noise  is given by
dimension d and label noise with variance a. let À = à Xo and o²
K 7 2 + 1 x² (3A0 - 2) + 2 + 3Ao +
Variancenoiss
2
(2Xg 2)7 + X3 + 2Ao
We plot the above expression as well as the expression
10
of Variance.kaan in the top row of Figure ! We set An 0.075. Variance monotonically increases and Variance is unimodal. Since Variance scales with K, a second ascent in the variace occurs when & is large enough. As result, the risk, which is the sum of Variance and monotonically decreasing Bias² (Figure 3). finds its minimum at intermediate width.We show the derivation of Variance in Appendix [ Alternatively, one can also obtain this
0.8
Rick, 0.00
Varianco 8- 0.00
Rick 0.50
Variance, 0.60
0.4
02
0.0
10
prd
Figure 3: Risk, bias and variance given by expressions in section A.2 and Theorem result by applying Adlam & Pennington (2020a)'s 's Lemma and Theorem and taking the same limit as ours, since our Variancenoise here is the sum of their Ve. Vre. Vxe and Vpxe- We note that the final ascent phenomenon has not been revealed by AAdlam Pennington (2020a) since they only studied the shape of the loss curve for the ridgeless setting where the loss only decreases in the overparameterizatton regime regardless of the noise.
3.2 INCORPORATING MODEL DENSITY INTO THE ANALYSIS
When the width of the model is beyond optimal, instead of directly reducing the width, one can mask a fixed set of the parameters throughout training to have fewer trainable parameters. We call this reducing the model density, with density defined as the fraction of remaining parameters. We study the effect of density for two reasons; First, changing density gives us more fine-grained capacity control. In practice, by changing width one can only get (width x m) parameters where m is the number of parameters at width 1. However, changing density can yield arbitrary number of parameters S (width m) and therefore achieves more fine-grained and architecture- -nndteeeure-ddeeeedddet capacity control. This is even more important for heavily engineered architectures whose width is particularly hard to reduce, e.g. InceptionRes.eet
Szegedy et al. (0117), where we experiment with in Section 4.3 Second, As will show below (Figure 4d)) under smaller density we can always find lower test loss than the test loss achieved by the optimal density at a larger width.
In our analysis below, we consider the case where the set of trainable weights are selected randomly instead of following any weight selection strategies. This to make sure the selected subset of weights are independent of other factors, e.g., initialization and the data, since otherwise the observed change in generalization can not be merely attributed to the number of parameters.
The setting of Theorem where the first layer of the network is random and the second layer is trained can not distinguish reducing density from reducing width because the parameter of the second layer is vector. Therefore we instead consider three-layer linear network where the first layer is the same as in Theorem I whose output yields random features, the second layer is randomly masked and trained, and the last layer is also random. Formally, the function represented by the network
is f(x) = (V @ M))x) μ. The first layer parameter W and input data X are the same as in Theorem I V E R is the second-layer parameters and M {0, 1779 is the mask applied to V. whose entries are independently drawn from Bernoulli(a) where a e [0. 1] is the density.
4 E R29 is the last-layer parameter whose entries are independently drawn from (0, 1/q) (we set
4

---

Under review as a conference paper at ICLR 2023
15.2
21,57
43
16.23
16.23
0.9
Y 2.26
2.34
4.38
98
Y 2.26
0.0.32
0-0.16
1.5
04
1:55
5.00
1,65
32
5:23
5.33
0.5
2:11
0:31
2.25
2:55
505
p/d
(a) Risk (x
(b) Risk 0.64)
(c) Variance 0.64) (d) 0.64, varying
Figure 4: (a) Risk decreases along both axes when = O. When = 0.64, (b) optimal risk is achieved by intermediate width and density because (c) variance increases along both axes when width is sufficiently large. (d) For 01 < there exist models at density on whose risk is smaller than the risk achieved by optimal width at density a2.
the variance to 1/g, so that & does not appear in the final expressions and we can plot the risk against only a and in 2-D plane. We study other variants, e.g, set it to 1/d and let in Appendix A.6). We show that, the above setting, in terms of the risk and its decomposition, equivalent to replacing the Åo in Theorem with λa/a. See the proof in Appendix A..4
Theorem 2. For a 3-layer linear network with p first-laye hidden neurons and both the first and last layers random, consider learning the second layer by ridge regression with random masks drawn from Bernoulli(a). Let X, n and a be the ridge regression parameter, number of training examples and noise level, respectively. Let X 2Ao and o² 222 for some fixed Xo and k. The asymptotic expressions (where n, d.p-p oo with 3 oc and y = QY) of Risk, Bias?, Variance clean and Varianceniaa are given by their counterparts in Theorem with No substituted with Xo/ax
We plot the risk and variance given by Theorem 2 in Figure We set Ao to 0.05. Plots of variance under = 0 (Variancedlemn) bias, decomposition of variance are in Appendix A.S Under zero noise, the risk monotonically decreases along both axes. In contrast, when K large, both the risk and variance increase along both axes when width and density are large. In Figure 4d we further show that tuning both width and density can achieve better generalization than only tuming width. Therefore, when looking for an optimal model capacity, it is necessary to consider the effect of density.
We note that our theorem also subsumes the phenomenon empirically observed by Golubeva et al. (2020) that increasing width while keeping the number of parameters fixed improves generalization, although this is not the main focus of this paper. And sufficiently large noise can change this picture (see risk curve with fixed any in Appendix A.7).
4 EXPERIMENTS ON NEURAL NETWORKS WHEN LARGER MODELS GENERALIZE WORSE ON NOISY-LABELED DATA
The main purpose of this section is provide empirical evidence for our theoretical analysis. We show that the final ascent in test loss can be observed across datasets, model architectures and noise types. In Section 4.2.1 we also demonstrate that the second ascent variance (analyzed in section 3.1 for linear models) occurs for neural networks, providing insights to understand the observed generalization behavior through bias-variance trade-off.
4.1 EXPERIMENTAL SETUP
Noise Type. Our theoretical example (Section 3 is a regression problem where label noise follows Gaussian distribution. The experiments below are for classification tasks since the most commonly considered task for neural networks and also empirically studied for confirming the double descent phenomenon Belkin al. 2019): Yang et al. (2000); Nakkiran et al. (002I). We consider three different noise types. On MNIST, CIFAR-10 and CIFAR- 100, if not stated otherwise, the noise type is random label flipping (also called symmetric noise). In addition, we consider asymmetric noise CIFAR-10 and CIFAR- 100 where wrong labels generated in class- dass-dependent way to mimic real-world noise. Images in Red Stanford Car are crawled from the web and therefore have real-world web noise.
5

---

Under review as a conference paper at ICLR 2023
CIFARLO (50000)
0.6
0.8
0.0
0.8
0.6
1.22
3.33
manatooic
dbuble descent
0.6
Test Loss
0.4
1.10
ORAR100 (10000
3.15
T0.tt Los5
0.05
0.4
Rinal ascen
final ascent
0.95
0.8
0.95
Width
16
16
10'
Width
10a
20"
-2P
(a) MNIST, 3K data, Sym. noise
(b) CIFAR-10, 10K, Sym.
(c) CIFAR-10/100, Asym.
Figure 5: Test loss on (a) MNIST with 3K training examples and symmetric noise, (b) CIFAR-10 with 10K training examples and symmetric noise, (c) CIFAR-10 with 50K (all) training examples and CIFAR- 1100 with 10K training examples, both with asymmetric noise.
MNIST and CIFAR-10 with symmetric noise. We train two-layer ReLU networks on MNIST
LeCun 1998) and ResNet34 (He et al., (016) on CIFAR-10 10 (Krizhevsky et al. 2009). The width of the two-layer network is controlled by the number of hidden neurons, And the width of ResNet is controlled by the number of convolutional layer filters: for width w. there are w, 2w, 4w, 8w filters in each layer of the four Residual Blocks, respectively. Measuring bias and variance: To empirically verify Theorem I which says label noise can lead to second ascent in the variance, measure the variance using the unbiased estimator proposed in Yang et al. (2020); we randomly split the training set for each model into N subsets (N 20 for MNIST and N 5 for CIFAR- 10), train a copy of the model for each subset and compute the variance of the network output across the subsets. In both settings we MSE loss because the bias-variance decomposition is only well-defined for MSE (see Eq. T (Yang et al., 2020) also proposed an estimator for Cross Entropy loss, but it is biased and therefore may skew the results. Details of training hyperparameters are in Appendix B
CIFAR10/100 with asymmetic noise. We also perform experiments on CIFAR-10 and CIFAR-100 with 40% asymmetric noise. Details of noise generalization can be found in Appendix B.1 Cross Entropy loss.
We use
Red Stanford Car. We train InceptionnesNNet-22 (Szegedy al. 2017) on Red Stanford Cars (Jiang iet al. (2020) that contains controlled real-world web noise. We consider noise levels 30% and 70%. The details of the dataset are in Appendix [B.] and the training setup is in Appendix [.2] Due the intricate nature of the model architecture, we only vary the density of this model. For each density, we repeat the experiment times and take the average of the performance.
4.2 LARGER WIDTH HURTS GENERALIZATION
We first train models of increasing width under various levels of label noise. Results are presented in Figure Under small label noise, the test loss is either monotonically decreasing (0% noise in 5a) or double descent (60% noise in [al 20% and 40% noise in 5b). When the noise level is sufficiently large, there is eventually a width beyond which the test loss increases. For MNIST with 80% and 95% label noise (Figure 5a). and CIFAR-10 with 60% and 80% noise (Figure 5b). the final ascent occurs and the loss curve is a double U-shape, with the globally optimal width in the under parameterized regime, i.e., the bottom of the first U-shape. Figure Higure55//shov shows the final ascent under asymmetric noise. We note that the noise threshold where the final ascent happens can depend on both dataset and noise type. Plots of the test accuracy can be found in appendix C.I
4.2. THE SECOND ASCENT IN VARIANCE
Our next experiment shows that the behavior shown in Figure can be explained through bias- variance decomposition, as is in our theoretical example. In Section 3. we attributed the final ascent in test loss to the second ascent in total variance for random feature regression. Here, the same applies to neural networks trained on MNIST (Figures bottoo row) and CIFAR- 10 (Figure 2). The shape of the variance curve is exactly as predicted by our theorem (Figure | top row). Plots of the bias curve can be found in appendix C.1

---

Under review as a conference paper at ICLR 2023
1.0
63
225
2
Test
0..6
2M
130
0.2
10
922
0.50
0.50
0.6
05
04
26 Density
03
Density
Density
(a) MNIST
(b) CIFAR-I0
(c) Car (30%)
(d) Car (70%)
Figure 6: Test performance models with varied densities. Increasing noise level changes the test loss curve from decreasing more like U-shaped. Therefore the optimal generalization achieved by models at intermediate density.
256
4596
2048
0.85
0.20
128
1.1
0.3
256
0 00
0.840
128
0.10
1.0
0.2
16:
0.05
0.9
6/ 600600
Density
Density
Density
(a) Test Loss (MNIST)
(b) Variance (MNIST)
(c) Test Loss (CIFAR-10) (d) Variance (CIFAR-10)
Figure 7: Test loss (c) and variance (d) of models at varied widths and densities. For MNIST the lowest loss is achieved by width 8192 and density 0.1. For CIFAR- 10 the lowest loss is achieved by width 11 with density 0.1. In both settings, the variance increases with both density and width after certain point. The trend of both loss and variance predicted by Theorem (Figure 4
4.3 SMALLER DENSITY HELPS GENERALIZATION
Next we vary the model density. At initialization, we randomly select a certain fraction of weights and keep them zero throughout the training. F how the performance changes with density under different noise levels. Under small label noise, test loss decreases when density increases. Under larger label noise, the test loss is U-shaped. Notably, for InceptionResNet-v22 (Figure 6d)) there is no straightforward way to reduce the width, however reducing density allows us to further control the capacity and achieve better generalization.
Smaller density vs. stronger regularization. One may notice that Theorem implies reducing density is equivalent increasing the strength of regularization by the inverse factor. Such equivalence may not hold for neural networks considering the non-linearity and the non-convexity. Interestingly, our experiment shows that reducing density can achieve better generalization than increasing la regularization. We defer the plots and discussion Appendix C.3
4.3.1 JOINT EFFECT OF DENSITY AND WIDTH
We conduct further experiments on MNIST and CIFAR-10 10 with 80% label noise and vary both width and density, Comparing Figure with Figures 4b and 4c confirms that empirical results align well with our theory. Figures 77b and 7d show the variance of the test loss. In both figures there is an increasing trend towards the top-right corner, though for CIFAR-10 the behavior non-monotonic and more complicated. Figures and ( demonstrate the test loss with the annotated number in red showing which model achieves the lowest test loss. As we can see, there an increasing trend the loss towards the top-right corner. Notably, in both settings the lowest loss is achieved by models with small densities. This happens for two reasons: (1) reducing density a more fine reggrriined capacity control; (2) suggested by theoretical example (Figure (4)), models of optimal width under smaller density has better generalization. Plots for the test accuracy and monotonically decreasing (along both axes) bias Figure 17 Appendix C.2

---

Under review as a conference paper at ICLR 2023
220
Test Loss
0.85
Test Loss
Test Loss
0.80
2.15
4.50
0.75
3000
210
0.70
6000
10000
2.55
25300
4.40
10000
0.65
2.00
50300
10
10a
10a
10
10
107
101
10"
Width
Width
Width
(a) MNIST
(b) CIFAR-IF 10
(c) CIFAR-
Figure 8: Legend shows number of training examples. As dataset size grows, test loss decreases and the final ascent becomes less pronounced.
0.25 Jxym
5.4 sgmm
0.3
0.4
0.6
24
Test Accuracy
Test Loss
Test
1.8
Selt 1:
(9
1.7-
66
1.8
1.5
16
Width
Width
(a) CIFAR-I0
(b) CIFAR-
(c) DivideMix, CIFARIOO, 60%
Figure 9: Test performance of models trained with ELR and DivideMix. Solid line shows test loss and dashed line shows test accuracy. The ascent the loss can still be observed here. By comparing that ELR even makes the phenomenon more pronounced.
(a) with Figures 8bland and &c
5 EFFECT OF SAMPLE SIZE ON GENERALIZATION
In Theorem the scale of Variance is controlled by the noise-te tes-tt--samm sample-siisee ratio instead of noise itself. This implies that increasing sample size alleviates the effect of noise. In this section, explore whether the same applies neural networks. We fix the noise ratio and vary the number of training examples. The experiment on MNIST is in the same setting as in Section 3 and we set noise level to 80%. For CIFAR- 10 and CIFAR- --0 we use Cross Entropy loss and the experimental details can be found in B.3 We consider 60% symmetric noise. We plot the test loss curve in Figure 8 On MNIST and CIFAR-10, the final ascent is less pronounced and eventually removed as we increase the sample size. However, on CIFAR- 100 the final ascent still occurs even the full dataset (although the test accuracy shows a different pattern (see Appendix (.4). We suspect is because CIFAR- 100 has fewer examples (and thus larger fraction of wrong labels) per class, which leads the shape of the curve to be affected more drastically by noise.
Discussion. Nakkiran et al. (202T) empirically observed that increasing sample size shrinks the area under the double descent curve and shifts the curve to the right. This suggests the possibility that in case the final ascent is only shifted, i.e., still occurs but at size larger than the largest width in Figure X rather than being eliminated. Although is impossible to exclude such possibility through experiments, note that, this possibility is not supported by theorem that implies sufficiently large sample size should completely remove the final ascent.
WHEN ROBUST ALGORITHMS ARE APPLIED
The above experiments are all conducted with standard training procedures. In practice, robust learning methods are usually applied achieve satisfactory test performance when the noise level high. One may expect the final ascent to be mitigated in this case, since robust methods are specially designed to encounter the effect label noise. However, perhaps surprisingly, we find that robust methods only make the final ascent more pronounced. We experiment with two SOTA algorithms, ELR Liu et al. (020) and DivideMix Li et al. (2020). We train ResNet34 on full CIFAR- 10 and CIFAR- 100 with the same training setup reported in the original papers (Liu et al. 2020 Li al.) 2020) (see details in Appendix B.4p and plot the test performance in Figure 9 When the models tained on CIFAR- 10/100 with ELR, the final ascent occurs under only 40% noise. In Figure
8

---

Under review as a conference paper at ICLR 2023
9a we also see that asymmetric noise exacerbates the phenomenon. When models are trained with DivideMix, the final ascent occurs under 60% symmetric noise on CIFAR-100 100 (Figure 99) but does not occur on CIFAR-10 10 is also the case with standard training procedure. Experiments for more noise levels are shown in Appendix C.5 The density experiments are also in Appendix C.5 which
includes training InceptionResNet->> with ELR Red Stanford Car.
7 A CLOSER LOOK INTO THE COMPLEXITY OF LEARNED FUNCTIONS
A hypothesis for double descent made by Belkin et al. (0199) is: the learning algorithm happens to have the right inductive bias towards the 'low complexity' function which generalizes better while fitting the training set. A larger model size, i.e., a richer function class, provides more choices for the algorithm thus letting it find a function with lower complexity. Given our finding that large noise flips the correlation between model size and generalization, it is natural to ask 'does large noise also flip the correlation between model size and
0% noise
40% noise
140
Trajectory length
128
120
130-
1000
the complexity of learned functions?'. In this section, we provide
empirical evidence suggesting that the answer is yes. Recently, Kalimeris et al. (2019) qualitatively measured complexity of neural networks based on how much their prediction can be explained by a smaller model. This can be problematic in our case since it already takes 'smaller models have lower complexity' as premise. We instead make use of the result in (Loukas et al. (02T), which shows bias parameter of the first layer of a neural network travels longer during training when it is fitting a more complex function (a function with larger Lipschitz constant on the data). We use the same setting as in Section Louks al. (202T) to get tight estimates for complexity: we train CNN with one identify layer plus 2 convolutional layers on CIFAR- 10, DOG vs AIRPLANE classes. The per epoch bias trajectory length of models at different widths is plotted in Figure TO The result shows that wider models have shorter trajectory length clean dataset. Thus, a larger size implies lower complexity of the learned function. In contrast, with 40% label noise, larger models have longer trajectory thus learning functions of higher complexity. We refer the readers to Appendix for more experimental results including: measuring variance of the first layer bias (which is also correlated with complexity complexityLLouka Loukas al. (202T)) and spectral norm (known as an upper bound for the Lipschitz constant (Szegedy et al. 2013) and the above measures at different densities.
60
Epoch
100
Epoch
60
100
Figure 10: Trajectory Length
CONCLUSION
Our work provides new insights into the overfitting and generalization of neural networks by showing a counter example for the recently observed generalization benefits of increasing model size. We considered two ways of varying model size: changing width and randomly dropping fraction of trainable parameters. We made two key observations: (1) when noise is sufficiently large, generalization gets worse if we increase the model size beyond some point, leading to the final ascent; (2) sufficiently large sample size can alleviate or even eliminate the final ascent. We provided theoretical evidence from random feature regression and explain such phenomenon through the lens of bias-variance decomposition. We also performed extensive experiments verifying that the above holds for neural networks. Additionally, training models with SOTA robust methods, namely ELR and DivideMix, does not mitigate the final ascent but even exacerbates it in some settings. We also provided empirical evidence which suggests the originally positive correlation between size and com- plexity of learned functions can be flipped by label noise. While our analysis in this paper is limited to label noise, would be valuable to investigate whether same conclusion holds for input feature noise.
REFERENCES
Ben Adlam and Jeffrey Pennington. Understanding double descent requires a fine-graine grained bias- variance decomposition. Advances in neural information processing systems, 33:1 33:11022-1100 2020a.

---

Under review as a conference paper at ICLR 2023
Ben Adlam and Jeffrey Pennington. The neural tangent kernel in high dimensions: Triple descent and a multi-scale theory of generalization. volume 119. PMLR, 2020b.
Devansh Arpit, Stanistaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning, pp. 233-242.
PMLR, 2017.
Zhidong Bai and Jack W Silverstein. Spectral analysis of large dimensional random matrices, volume 20. Springer, 2010.
Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 1177449):30033330077, 2020.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learnin practice and the classical varinnae trade-off. Proceedings of the National Academy of Sciences, 116(32):18849-15854 2019.
Mikhail Belkin, Daniel Hsu, and Ji Xu. Two models of double descent for weak features. SIAM Journal on Mathematics of Data Science, 2(4):1167-1188) 2020.
Yuan Cao, Zixiang Chen, Mikhail Belkin, and Quanquan Gu. Benign overfitting in two-layer convolutional neural networks. arXiv preprint arXiv:2202.06556, 2022.
Niladri S Chatterji, Philip M Long, and Peter L Bartlett. The interplay between implicit bias and benign overfitting in two-layer linear networks. arXiv preprint arXiv:2108. 10.8///48) 1489, 2021.
Michal Derezinski, Feynman T Liang, and Michael W Mahoney. Exact expressions for double descent and implicit regularization via surrogate random design. Advances in neural information processing systems, 33:5152-5164, 2020.
Stéphane d' Ascoli, Maria Refinetti, Giulio Biroli, and Florent Krzakala. Double trouble in double descent: Bias and variance (s) in the lazy regime. In International Conference on Machine Learning, pp. 2280-2290. PMLR, 2020.
Spencer Frei, Niladri S Chatterji, and Peter Bartlett. Benign overfitting without linearity: Neural network classifiers trained by gradient descent for noisy linear data. In Conference on Learning Theory, pp. 2668-2703 PMLR, 2022.
Anna Golubeva, Behnam Neyshabur, and Guy Gur-Ari. Are wider nets better given the same number of parameters? arXiv preprint arXiv:2010. 14495, 2020.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in neural information processing systems, pp. 3527-85377 2018.
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high- dimensional ridgeless least squares interpolation. arXiv preprint 1903:v:1933.08600 2019.
Kaiming Hc. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks on corrupted labels, In International Conference on Machine Learning, pp. 2309-2318, 2018
Lu Jiang, Di Huang, Mason Liu, and Weilong Yang. Beyond synthetic noise: Deep learning on controlled noisy labels. International Conference on Machine Learning, pp. 4804 4815. PMI.R, 2020.
Dimitris Kalimeris, Gal Kaplun, Preetum Nakkiran, Benjamin Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang. Sgd on neural networks learns functions of increasing complexity. Advances in neural information processing systems, 32, 2019.
10

---

Under review as a conference paper at ICLR 2023
Ranjay A Krishna, Kenji Hata, Stephanie Chen, Joshua Kravitz, David A Shamma, Li Fei-Fei, and Michael S Bernstein. Embracing error to enable rapid crowdsoureing. In Proceedings of the 2016 CHI conference human factors in computing systems, pp. 3167-3179, 2016.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/nmist// 1998.
Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi-
supervised learning, arXiv preprint arXivv2002.07399, 2020.
Sheng Liu, Jonathan Niles Weed, Narges Razavian, and Carlos Srmaadez--Gradda Early-learning regularization prevents memorization of noisy labels. Advances in neural information processing systems, 33:20331-20342, 2020.
Andreas Loukas, Marinos Poiitis, and Stefanie Jegelka. What training reveals about neural network complexity. Advances in Neural Information Processing Systems, 34, 2021.
Neil Mallinar, James B Simon, Amirhesam Abedsoltan, Parthe Pandit, Mikhail Belkin, and Pree- tum Nakkiran. Benign, tempered, or catastrophic: A taxonomy of overfitting. arXiv preprint 2rXhv:2207.06569 2022.
Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 2019.
Baharan Mirzasoleiman, Kaidi Cao, and Jure Leskovee. Coresets for robust training of deep neural networks against noisy labels. Advances in Neural Information Processing Systems, 33, 2020.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang. Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory 2021.
and Experiment,
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. arXiv preprint arXiv: 1et.111126664444 2014.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1944-1952, 2017.
Hanie Sedghi, Vineet Gupta, and Philip M Long. The singular values of convolutional layers. arXiv preprint arX:v: 1805. 2018.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander Alemi. Inception-vé inception- resnet and the impact of residual connections on learning. In Thirty-first AAAI conference on artificial intelligence, 2017.
Alexander Tsigler and Peter Bartlett. Benign overfitting in ridge regression. arXiv preprint arXiv:2009. /4286, 2020.
Zitong Yang, Yaodong Yu, Chong You, Jacob Steinhardt, and Yi Ma. Rethinking bias-variance trade-off for generalization of neural networks. In International Conference on Machine Learning, pp. 10767-10777. PMLR, 2020.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint 1xXiv:6611.0355 2016.
Zhilu Zhang and Mert Sabuneu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in neural information processing systems, pp. 5778-8788, 2018.
11

---

Under review as a conference paper at ICLR 2023
A THEORETICAL RESULTS
Notations: We use bold-faced letter for matrix and vectors. The training dataset is denoted by (X, y) where X [21 ........ Zn and y = [/ 92...... yn]] ..]] with E IR and yi R are feature vector and label point data point We assume each æ; is independently drawn from a Gaussian distribution (0, Ia/d) and labels are generated by Y/- x; 0 E; where 0 E IR with entries independently drawn from NO0 1) and G E R is the label noise drawn from (0, o²) for each I;. And we assume each test example (xr y) is clean, Le., y x 0.
A.I BIAS- DECOMPOSITION OF THE MSE LOSS IN SECTION
Here we show the derivation of Equation
Bias =EE En [x W ..f f y)²
= EE (E e (( x) (3) y)²
= -EgEE xx (Ex, W. .6 (BO + AE) 0))
axx" (Ex,w, (BO) -0] -E Tr[(EX, W. new....BBBB (B0) Txxx'
=1E Tr[(Ex, (B0) - 0) EE (ax T) (Bxx,, (B0) (BO)) 0))
Eo |Ex, .e (B0) - 0
Eo Tr B 1)00' (Ex, B 1)
TE x w B 1)E0 B.Ex.B--1. I)]
d
[x W 1
Variance = EgE V x, w fz)
-EgEE w x
(BO + Ac) Ex,w, (BO Ac)]
E,Ex x w BB + Ac) Ex w B0
EoEx.w.cll .lw. Ex B)0 + Acl
d
||
Fx.w B)0 Ex,
Wx, w B Ex
Ex
Variance,jaa
Variance.sss
A.2 EXPRESSIONS OF BIAS AND VARIANCE_CIFAN
Yang et al. (2000) analyzed the biri-variance decomposition without considering label noise. Hence the variance in their analysis corresponds to Variance, in ours. We show their expressions below, based which we plot the dashed line in Figure (Variance:tan)) and dashed gray in Figure
(Bias?).
1 Bias²
(do 7)
1
(Ani))
(1-7)(1-21) 2y
( 751 1,
Variance
(0077
(An.1/1
2-1 -
- t
1003(00,7) 7)3,
<> 1,
20 1/7]
where
p (1()07)) =>o(7 1) (r - 1)2,
12 7) = =Vooo + +1] +17+2(M) 2(xp 1)y + +3 p2()o:7) =02 7) - - 7 1.
12

---

Under review as a conference paper at ICLR 2023
A.3 PROOF OF THEOREM! 1
Lemma 3. Define B = W (WWT + W |BBT
AAA 1l2 - 0 almost surely, i.e.
PB T -AAA 1| S e) 2 1 å where and both tend to under the asymptotics oo,
d 00 and n/d > 0c.
Proof. Define
d Xx L1, A n1
V WW T + AoT)
WXX W + -
Ω
V.
Then have
BB - n
AA SwAw a + MWAW & + &WWW T & WWAW Ω
www Ω + www WWW Ω
By triangle inequality and the sub- sub-multiplicative property of spectral norm we have
n1 J AA < ||
BB
2 + 22
W
2 + 2 ||
|
2 |S1||
remains to show that with the asymptotic assumption n/d > oc, |2 = and
A ll almost
surely, and || ||[la ll2 and || | can be bounded from above Yang et (2020):
d
P(|||
2 <
2
-d/2 d/2
n
| 2 S V
W
A
A(1)
W 2
1
+ vπ <0
1
| 2 <
oc.
Ao
Therefore we have |BB 3/2 AA lll = 0 almost surely.
Corollary 4. - A 3 / l& almost surely.
Proof. By lemma 3 we have
Tr
=
BB
AA
1
)
TTiB"
n
AAT)
d
d
<
LBB
AA 12
d
0.
which yields A
o² B|B and thus d
|A| 2 B
Now it only remains to compute 1 B B By Sherman-Morrison formula,
B (I+ 1)
=Q)
where Ao1 Q (d/p) W and n d/p 1/7. Let FQ be the empirical spectral distribution of i.e.,
1
r (x)x) - #b < d: X₂ 5z)
13

---

Under review as a conference paper at ICLR 2023
where #S denotes the cardinality of the set S and A, denotes the j-th eigenvalue of Q. Then
B
x) x)
d
(x))
d
By Marchenko- Pastur Law Bai Silverstein (2010).
B
2
1
v(++ x)(x 17 (1
dx
2x
ner + 0)
1
022/1² - (3a - 2022/11 0² -3a 2
/1/ (2a 20 2/a2)/+ +2 + + aan+a++2
(2)
2a
Combining equation and corollary and substituting a = \o² , = 7 into the result completes the proof.
A.4
PROOF OF THEOREM 2
Define the following shorthand
WX
D. diag M.,,, M.2,..... Mip)
i-th entry of
V i-th row of V
E Σ
D.D
==1
FF EF EF AI)-
The parameter of the second layer is given by ridge regression:
V
arg min ((V M)W p --y VI
VERGXP
arg min VEReXP
V.V. D.F- I
Solving the above yields
V = - (y FF (.)
Given clean test example (x, v). the expression of the risk is
Risk =E
4s V. Wx - - x
-EI (y' H)F EWE> x TT
T Txxll
=E - =F EW - 0 xx12
Observe that the diagonal matrix Σ. whose diagonal entries En_ n¹ M.j = -. (T M converge in probability to as q co, captures all the effects of μ and M on the risk. Thus can replace Σ with aI,
Risk =E a N FW a² N
a
It remains to show that the expression of the risk is exactly the same as the risk in the setting of Section 3.1 with replaced by a/a. Write down the risk in Section 3.1 (for convenience denote by Riskg):
Risk, = =Elly F (FFF + AI)
Wa O x
1 F
=E FW x AT RS
F FTF Wx
4
Equation is obtained by applying Woodbury matrix identity to (FFT + AI) is easy to check that replacing À in RHS of equation with A/a yields the same RHS of equation 3
14

---

Under review as a conference paper at ICLR 2023
23.67
16.2
1,24
1.05
4.79
54
36
21.07
10.33
2.26
4.78
0.00
023
22
631
8.05
0.25
(a) Bias
(b) Variance
(c) Variance K
Figure 11: Expressions of Bias", Varianceolemn and Variancenoise! K in Theorem
Y
(a) Risk (K
(b) Risk (K 0.64)
(c) Variance (x
(d) Variance 0.64)
Figure 12: Expressions of the risk and variance under different noise levels with 7. We let
Ao = 0.05.
A.5 BIAS² AND VARIANCENOSSE/ IN SECTION 3.2
In Figure we plot the expressions of Bias and Variance based on Theorem Bias monotonically decreases along both axes and Variance monotonically increases along both axes. Variancealean variance unimodal along y-axis (width), manifests more complicated behavior along x-axis (density), and decreases along both axes once width is sufficiently large, and thus it differs from the variance in classical Varrinai tradcoff.
A.6 VARIANTS OF THEOREM 2
Theorem we fa's entries be drawn from (0 1/g) so that g does not appear in the expression of the risk. Alternatively can let μ's entries be drawn from (0, 1/d) and assume q/d the risk and its decomposition are dependent on y, T, a. Similar to the proof in A.4 we can show that in this case just need to replace expressions risk. We further let r = y and plot the result in Figures] and 13
Then
in the setting of Theorem [ with Ao/ (r/(ra) to get the
A.7 CONNECTION TO THE OBSERVATION MADE BY GOLUBEVA ET AL. (2020)
Our Theorem confirms the observation in Golubeva et al. (200)) that increasing width while keeping the number of parameters fixed (via reducing density) benefits generalization. This disentangles the benefit of increasing width from the effect of enlarging model capacity. In our setting, the expected number of parameters is in proportion
ya. We fix yo and plot the risk curve with varied ? (s.t.a < 1) in Figure [4] When = 0 the test loss decreases with width. However, large noise here can still change the shape of the curve. When 0.64, the test loss is either U-shaped or increasing.
B DETAILS OF EXPERIMENTAL SETTING
All experiments implemented using PyTorch. We eight Nvidia A40 to the experiments.
15

---

Under review as a conference paper at ICLR 2023
DA
Y
(a) Bias
Variance /
Figure 13: Expressions of Bias and Variance / with T = 7--
0.5
0.4
ye 0.50
y2
0.0335
p01 2.00
3.2
ya 10.00
0.05
0.3
0.025
3.0
0.2
0.04
0.020
0.035
2.6
0.00
10
10
p/d
p/d
p/d
0.65
0.50
0.40
1.00
0.39
0.50
0.38
0.384
0.45
0.37
0.312
0.380
0.36
10 -
2.00
155
101
107
107
10
107
16
3.4051
p/d
p/d
p/d
Figure 14: Risk curve with fixed ya. Top: x
O, the test loss decreases with width. Bottom: 0.64, the test is either U-shaped increasing.
B.1 NOISE TYPES
Symmetric noise of examples.
Symmetric noise is generated by randomly assigning labels certain fraction
Asymmetric noise Asymmetric noise class-dependent. We follow the scheme proposed in Patrini ct al. (2017) which is also widely used in robust method papers (c.g., ELR Liu et al. (020)) and DivideMix Li et al. (2020)). For CIFAR-10, 10, labels are randomly flipped according to the following TORD-AAIPPLAN
DEER -HRRSE CAT- DOG- For CIFAR- 100, since the 100 classes are grouped into 20 superclasses, e.g. AQUATIC MAMMALS contains BEAVER, DOLPHIN, OTTER, SEAL and WHALE, flip labels of each class into the next one circularly within super-classes. For both datasets, the fraction of mislabeled examples in the training set the noise level.
map: TRUCK->A PAATOMOOBBLL
Web Noise Red Stanford Car Jiang et al. 2020) contains images crawled from web. There are 10 different noise levels (0%, 5% 10%. 15%. 20% 30%, 40%, 60%, 80%) for this dataset and we choose 40% and 80%. For each noise level, the mislabeled web images can only be downloaded from the provided URLs. The training splits and labels are in provided files [1] The original dataset size is 8144. For 80% noise, there are 6469 web images. However, 590 of the URL links are not functional and among the downloaded JPG files 1871 are corrppte//uoopeabble hence we end up with 1675 clean examples and 4008 noisy examples, i.e., the actual noise level is 70.53%. For 40% noise, there are 3241 web images with 313 nonddownlo and 963 not unopenable. Therefore actual noise level is 29.03%.
details cont rol 1ed- toi sy-swbbbbb-b web Tabel s/cown oad. html
See their
webpage for
htcps://google.github is/
16

---

Under review as a conference paper at ICLR 2023
B.2 BASIC EXPERIMENTS
MNIST and CIFAR-10 with symmetric noise On MNIST, we train two-layer ReLU network for 200 epochs using SGD with batch size 64, momentum 0.9, initial learning rate 0. 1, a weight decay of 5 x 10-2, a learning rate decay of 0.1 every 50 epochs. On CIFAR-10 We train ResNet34 for 1000 epochs using SGD with batch size 128, momentum 0.9, initial learning rate 0.1, weight decay of 5 x 10 learning rate decay of 0.1 every 400 epochs. For width w. there w, 2w, 4w, Sw filters in each layer of the four Residual Blocks, respectively. We use MSE loss in both settings.
CIFAR- 10/100 with asymmetric noise On CIFAR-1O, we train ResNet34 for 500 epochs using SGD with batch size 128, momentum 0.9, initial learning rate 0.1, weight decay of 5 x 10 learning rate decay of 0.1 every 100 epochs. On CIFAR- 100, we train ResNet34 for 1000 epochs using SGD with batch size 128, momentum 0.9, initial learning rate 0.1, weight decay of x 10-1, learning rate decay of 0.1 every 200 epochs. We use Cross Entropy loss in both settings.
Red Stanford Car On Red Stanford CaDiang et al. (0200) with 80% noise. We train InceptionResNet. V2Szegedy et al. (2017) for 160 epochs with an initial learning rate of 0.1, and weight decay of 1 x 10 using SGD and momentum of 0.9 with a batch size of 32. We anneal the learning rate by factor of 10 epochs 80 and 120, respectively. We Cross Entropy loss.
B.3 VARYING SAMPLE SIZE
MNIST The setup is the same as in B.2
CIFAR-10/100 On both datasets the noise level is 60%. On CIFAR-10, train ResNet34 for 1000 epochs using SGD with momentum 0.9, initial learning rate 0.1, weight decay of 5 x 10-4 learning rate decay of 0.1 every 100 epochs. The batch size is 128 for sample size 50000, 64 for sample size 25000, 10000, 5000, and 32 for sample size 2500. We set batch size this way to make sure the model can converge and the largest few models can fit each of the training set (e.g.. when trained with batch size 128 on 2500 examples even the largest model does not fit the training set; when trained with batch size 64 on 50000 examples the model converges too slowly). On CIFAR- 100, we train ResNet34 for 500 epochs using SGD with momentum 0.9, initial learning rate 0.1, weight decay of x 10-4, learning rate decay of 0.1 every 100 epochs.
B.4 EXPERIMENTS WITH ROBUST METHODS
ELR leverages the early learning phenomenon where the network fits clean examples first and then mislabeled examples. It hinders learning wrong labels by regularizing the loss with term that encourages the alignment between the model's prediction and the running average of the predictions in previous rounds. DivideMix dynamically discards labels that are highly likely be noisy and trains the model in a semi- supervised manner. For both ELR and DivideMix we use the same setup the original papers. [iu et al. (2020); et al. (2020).
ELR We train ResNet-34 using SGD with momentum 0.9, weight decay of 0.001, and a batch size of 128. The network trained for 120 epochs on CIFAR-10 and 150 epochs on CIFAR-100, The learning rate is 0.02 at initialization, and decayed by factor of 100 at epochs 40 and 80 for CIFAR-10 and at epochs 80 and 120 for CIFAR-100. We use 0.7 for the temporal ensembling parameter, and for the ELR regularization coefficient.
DivideMix We train ResNet-34 for 200 epochs using SGD with batch size 64, momentum 0.9, initial learning rate 0.02, weight decay of 5 x 10 and learning rate decay of 0.1 at epoch 150. We use 150 for the unsupervised loss weight.
C ADDITIONAL. PLOOTS FOR SECTIONS TO 6
C. BASIC EXPERIMENTS: VARYING WIDTHI
Bias² and test accuracy on MNIST and CIFAR-10 shown in Figure 15
Test accuracy CIFAR-
10/100 with asymmetric noise is shown in Figures [6a and [6b] We also plot test loss and test
accuracy of models trained CI with 30% asymmetric noise in Figures T6c 16d Here
17

---

Under review as a conference paper at ICLR 2023
E 22
Tox PA PRAC
Withh
(a) Bias MNIST
(b) Test Acc. MNIST
(c) Bias
CIFAR-10
(d) Test Acc. CIFAR-10
Figure 15: Bias" and test accuracy MNIST and CIFAR-10.
1.00
1.05
600
3.04
1.02
20.3
1.00
01
101
10
Wilth
Witth
Width
Width
(a) CIFAR 10, 40%
(b) CIFAR 100, 40% (c) CIFAR 10, 30%
(d) CIFAR 10, 30%
Figure 16: We plot accuracy on CIFAR-10 (a) and CIFAR- 100 (b) under 40% asymmetric noise. We also plot test loss (c) and test accuracy (d) on CIFAR-I0 under 30% asymmetric noise.
4095
81.9.4.3
70
128
256
30
2102
3086
256
128
1024
112
1024
0.85
250
0.5
64
40
32
12
985
075
000 0000 000
6.00
0000
Denstty
Density
Dansity
(a) Test Acc. (MNIST)
(b) Test Acc. (CIFAR-10) 10)
Bias (MNIST)
(d) Bias (CIFAR-10)
Figure 17: Test accuracy and bias plotted with varied width and density.
we can still observe slight ascent from width 128 256. As expected, the final ascent in this setting is less pronounced than under 40% noise (Figure 5c
C.2 BASIC EXPERIMENTS: JOINT EFFECT OF WIDTH AND DENSITY
We plot test accuracy and bias in Figure 17
The bias decreases when density and width increase. In both settings, the optimal accuracy is achieved by models with smallest density.
C.3 SMALLER DENSITY VS. STRONGER l REGULARIZATION
In Theorem we show that reducing density is equivalent to increasing the strength of l2 regular- ization. To see whether this applies to more complex settings, we train ResNet34 at width 16 on CIFAR-10 with 50% symmetric noise. We vary weight decay and network density. Results are shown in Figure 18 Both loss curves are U-shaped, but reducing density can achieve lower loss/higher accuracy than increasing l2 regularization. The con-viuivveence itself is not unexpected given the non-linearity in neural networks and how dropping large amount of connection can significantly change the training dynamics of SGD. However, would be interesting to investigate why reducing density is better and whether it is the same case in other settings.
C.4 VARYING SAMPLE SIZE
We plot test accuracy with varied sample size in Figure 19
18

---

Under review as a conference paper at ICLR 2023
60
Test Accuracy (%)
0.25
densily
dersity
50
Loss
0.80
0.75
0.70
0.65
0.60
inverse of wdl scale or density
inverse wd scale or density
Figure 18: When varying weight decay, the density is fixed When varying density, the weight decay is fixed to 0.0005. The x-axis represents the inverse of the scale-up of weight decay, i.e., the current weight decay divided by 0.0005, for the weight decay curve (purple), and model density for the density curve (red).
Test Accuracy
70
10000
6000
3000
Test Accura
55
50
10000
23200
30200
Test Accuracy
50000
10000
40
25
35
20
30
30
20
25
15
10a
10a Width
10a
104
10'
10a
107
10a
Width
Width
(a) MNIST
(b) CIFAR 10
(c) CIFAR 100
Figure 19: Test accuracy MNIST and CIFAR-10/100 with varied sample size.
0% Accciacyy Test
J5s
acc best
(%) Tauaaaeaa Aoennaa
40
307
33
10
Width
Width
(a) CIFAR 10, 80% sym
(b) CIFAR 100, 80% sym
Figure 20: Test performance models at increasing width trained with ELR on CIFAR-10/100.
C.5 MORE RESULTS FOR MODELS TRAINED WITH ROBUST METHODS
In addition to the experiments presented the main paper, we vary width of models trained with ELR under 80% noise and vary density
show the results the following experiments on CIFAR-10/100: width experiments with ELR under 80% noise 20 density experiments with ELR 21 density experiments with DivideMix under 60% noise 22 We also show the result density experiments on Red Stanford Car (70% noise) where train InceptionResNet-22 with ELR 23 In Figures 20 and 21 "best_acc' refers to the highest test accuracy observed during training, which is usually reported in robust method papers.
D MEASURING COMPLEXITY OF LEARNED FUNCTIONS
We consider three measures:
[b 1++ lla where T is set of iteration a15
Trajectory length of the first layers bias. Σter
indices, ((() is the parameter of the first layer bias iteration f, and eyt) is the gradient
19

---

Under review as a conference paper at ICLR 2023
515
.510
0.64
114
810
Test Loss
-565
566
81
28
26.5
0.4
Test
25
638
0.9
06.0
Density
Density
Density
(a) C10, 30% asym, width 128
(b) CIO, 60% sym, width 128
(c) C10, 60% sym, width 256
(oloeennod 1581
Accrree 1581
5.0
1es.
Less
2.a
54
30
2.48
1.43
4.-
2.4
Density
Density
Density
(d) C10, 80% sym, width 128
(c) C100, 60% sym, width 128
(f) C100, 80% sym, width 64
Figure 21: Test performance of models at increasing density trained with ELR CIFAR-10/100.
1.55
71.
Cccnon 35R
73.4
Test Loss
1.60
732
330
72
72.
10.2
22.4
Density
Figure 22: Test performance of models at increasing density trained with DivideMix CIFAR-100. We set the width to 64.
38.0
7.0
Loss
Accuracy
4.0
0.4
Density
Figure 23: Test performance of models at increasing density trained with ELR on Red Stanford Car.
of the loss w.r.t. the network's 's output epoch t. Loukas al. (2021) shows that, under certain conditions, the above can be both upper and lower bounded in terms of the Lipschitz constants of functions represented by the network iterations in T (see their Theorem 1). This implies that the first layer bias travels longer during training when it is fitting a more complex function.
Variance of the first layer bias avgier|(b a 113 Loukas al. (021) also provides a lower bound for the above in terms of the Lipschitz constant and Eycy) (their Corollary 2). Thus larger Lipschitz constant leads higher lower bound, meaning that when fitting lower complexity function, the network's 's bias will update more frequently during training.
(4)
Product of spectral norms of the layer parameters. This known as an upper bound for the network's 's Lipschitz constant SSegedy al. (2013). For convolutional layers, the spectral norm is computed using the Fi-T-based algorithm Sedghi et al. (018)).
Our setting the same as in Loukas et al. (202 (Task Section 6). We train a CNN on CIFAR-10 DOG AIRPLANE The CNN has identify layer, convolutional layers with kernel size 5,
20

---

Under review as a conference paper at ICLR 2023
0%
40%
0%
40%
40%
Trajectory length
Trajectory length
Variance
Variaace
aox
16
38"
30
Epoch
Epoch
Epach
Epoch
(a) Trajectory, width
(b) Trajectory, density
(c) Variance, width
(d) Variance, density
Figure 24: Trajectory length and variance of the first layer bias during each epoch. We vary both width and density. When varying the density we fix the width 128.
1000
0.0
0.4
Spactral Norm
Spectral Norm
1000
1000
1090
125 26 25
0F Density
90
Width
Wldth
Density
(a) 0% noise, width
(b) 40% noise, width
(c) 0% noise, density
(d) 40% noise, density
Figure 25: When width or density increases, spectral norm decreases under 0% noise but increases under 40% noise.
fully connected ReLu layer with size 384 and a linear layer. The width is controlled by the number of convolutional channels. For width w there are 16w channels in each convolutional layer. We use BCE loss and train the models for 200 epochs using vanilla SGD with batch size 1. We use exponential learning rate decay with a factor 10-50, The trajectory length and variance are computed every epoch, i.e., we let T be the iterations of each SGD epoch. Thus they capture the complexity of the function represented by the CNN at every epoch. The results are shown in Figure 24 We see that noise does change the relative complexity of functions learned by models at increasing width/density, The pattern is more clear for width (Figures 24a and (4c) where both quantities decrease under 0% noise, and increase under 40% noise, when width increases. In Figure 25 we plot the product of layer-wise spectral norms of the neural network at the last epoch and shows a similar pattern. These results suggest that the originally negative correlation between size and complexity/smoothness can be flipped by label noise.
E POTENTIAL CONNECTION TO BENIGN OVERFITTING
Benign overfitting refers to the phenomenon where models trained to overfit the training set still
achieve nearly optimal generalization performance Bartlett et al. (2020) Tsigler & Bartlett (2020) Chatterji et al.] (202)) Cao et al. (2022); ; Frei et al. (2022): Mallinar et al. (2022). Recently Cao al. (022)) shows that sufficiently large models overfit benignly if the product between sample size and signal-to-noise ratio (SNR) is large, and catastrophically otherwise. Interestingly, this is the same condition for the final ascent to not occur in our theoretical setting (Section 3 the SNR in our setting is 1/a²). Therefore one can explain our result from the perspective of bemign overfitting: when the condition is satisfied, a sufficiently large model overfits benignly, and therefore increasing model size toward sufficiently large can possibly help generalization. That said, our result has broader implications because it (I) compares models at arbitrary size that may not be sufficiently large and (2) does not restrict to the benign overfitting regime. Indeed, neural networks along with many other real interpolating methods are in the tempered overfitting regime Mallinar et al. (2022). Our result suggests that the condition for benign overfitting identified in Cao et al. (2022) can be possibly extended to judge the relative benignity of tempered overfitting at different model sizes. We see theoretically establishing the above as important direction of future work.
21

---

Under review as a conference paper at ICLR 2023
64
Risk
Risk
Risk
04.1
20
69 Width
46
166
106
Width
Width
Width
Figure 26: We plot the risk as a function of the width for the 'prescient' selection model using
Theorem 2.1 in Belkin et al. (2020). We see that as noise increases, the risk in the overparameterized regime turns from a U-shaped into a descent.
F NON-ASYMPTOTIC RESULTS FROM PREVIOUS WORK
We clarify here that Theorem cannot be obtained from taking the limit of previous work. The setting our paper differs from previous ones that aim at deriving non-asymptotic results. We chose this setting as it allows incorporating the three important sources of randomness, namely training data, label noise, and random features that mimics the randomness in initialization and optimization (as opposed e.g. models with fixed design without random feature), and allows deriving interpretable closed form results (as opposed to non-linear feature or n/d=const limit). We not aware of any non-asymptotic result that gives explicit expression of the bias-variance tradeoff, as we derived in this setting. We give a few examples that the final ascent has not been reflected in previous theoretical models on double descent: [elkin et al. (020):: Corollary 2.2 (Gaussian model with random selected features) doesn't It capture the final ascent since the noise dependent term always decreases with the width. The 'prescient' selection model (their Fig 2) has an ascent in the end, however, as we plotted in Figure 26 (1) the ascent exists even without noise, thus the result doesn't show the double descent phenomenon in the first place, and (2) the ascent diminishes as we increase noise, and therefore shows an effect opposite to our empirical observation. The Fourier series model in their Theorem 3.1 used to show double descent for noise free setting. Hastic et al. (2019): Among the models they considered, only the minimum-norm least squares with isotropic features can manifest a U-shaped curve in the overparameterization regime (their Fig 1). However, the effect of noise here is exactly the opposite of what observed. As described in their Sec 3.2, in the overparameterization regime, the loss monotonically decreases when SNR<1 (noise is large) and has local minimum when SNR>=1 (noise is small).
22


Reviews:
[{"review_id": "zo2G0ZTivF", "reviewer": "Reviewer_romc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the role of overparameterization for generalization when training with noisy labels. Specifically, the authors focus on the noisy label training data setting and investigate how test loss of model changes with respect to different levels of overparameterization. This paper provides two interesting empirical findings: (1). 'the variance of the generalization loss experiences a second ascent when the noise-to-sample size ratio is large'; (2). when applying robust training methods (robust towards label noise), the robust training methods make the last ascent more pronounced. The paper also provide theoretical analysis for the first finding in a simplified setting. ", "summary_of_the_review": "This paper provide interesting theoretical and empirical findings on overparameterization under label noise, I would recommend acceptance.\n\n", "strength_and_weaknesses": "**Strength**:\n\n1. This paper identifies two interesting findings on the effect of “overparameterization under label noise: \n \n>(1). the variance of the generalization loss has a second ascent when the noise is large; and \n\n>(2). the robust training methods (robust towards label noise) make the last ascent more pronounced. \n\nThese two findings could be practically useful for handling label noise, especially when we apply robust training methods for training highly overparameterized models.\n\n\n**Weaknesses**:\n\n1. [minor] There seems to be only one point for demonstrating the 'last descent' phenomenon on CIFAR10 (as shown in Figure 2 and Figure 5(b)). It would be interesting to study an even wider network to further examine the trend (if computation allows).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ulgIINEjTY", "reviewer": "Reviewer_fFmQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the effect of label noise on the test error of overparameterized models. In particular, it shows that when the labels are sufficiently noisy, the test error increases again with model size (after the double descent peak). The authors show this theoretically in a simple linear random features model, and provide empirical evidence for it in real neural networks.", "summary_of_the_review": "The main findings of the paper lack novelty, which could hurt the paper's interest to the ICLR community. In addition, the relationship of the theoretical results to previous work is not fully explained or explored.", "strength_and_weaknesses": "**Strengths**\n- The paper provides closed-form solutions for the bias and variance. \n- There are experiments on real neural network architectures.\n\n**Weaknesses**\n- There is a lack of novelty. The main theoretical results are a minor extension of Yang et al., and it's unclear whether they could just be obtained by taking a limit of previously published results.\n- The analysis is restricted to linear random features. Why is this restriction necessary?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_YPRDcAtjuI", "reviewer": "Reviewer_gtPU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper demonstrates the existence of a phenomenon called “final ascent” where noisy labels can hurt the generalization models with larger capacity, which contrasts with “double descent” where increasing the model capacity first hurts the model performance and but improves the performance again. First, the paper theoretically proves the existence of the phenomenon in the recently popular random feature regression framework. Roughly speaking, the result shows that in the presence of label noise, the risk of the analytical solution has an extra term governed by the noise-to-sample size ratio. With similar technique, the paper also proves a theorem that shows sparsity can alleviate final ascent. In the second part, the paper conducts a series of experiments on MNIST, CIFAR10/100 and MLP, LeNet and ResNet34 and show that realistic models and data also exhibit final ascent.", "summary_of_the_review": "The paper makes a solid theoretical contribution to the line of work on random feature models and conducts thorough experiments. I do not have major complaints about the paper and thus recommend accept.", "strength_and_weaknesses": "### Strength\n1. The proposed phenomenon is timely for the discussion of scaling and the message that larger model can hurt potentially generalization performance is very important\n2. The theoretical component seems technically solid\n3. The experimental results on deep neural network are thorough and match the prediction of the proposed random feature regression model\n4. The observation that robust methods exacerbate final ascent is extremely interesting and could lead to fruitful future research\n### Weakness\n1. The phenomenon only becomes relevant with extremely large label noise (> 60%) which makes it less interesting for most practical cases where we can expect a reasonable signal-to-noise ratio.\n2. The plots for 5c seem to contain fewer settings than the other ones. Why? I personally would be interested to see more\n3. There is a gap between the regression setting of the analytical model and classification setting of the experiments.\n4. It also seems like architectures can have a non-trivial effect on the phenomenon.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Jo3SDS20Mr", "reviewer": "Reviewer_RtYb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the effect of label noise on generalization when training\noverparameterized models.  Prior work has given theoretical and empirical\nevidence that overparameterization improves generalization.  This work shows\nthat, when the labels of the training data are very noisy, overparameterization\nmay actually harm generalization and provides a theoretical analysis of the\nphenomenon for 2-layer linear networks.  They provide empirical evidence for\nthis effect using both synthetic and real noise for classification tasks.  They\nalso consider using recent robust training methods (DivideMix and ELR) and find\nthat in some cases using these methods may actually exacerbate the phenomenon\nin the sense that it appears for lower noise rates.  They conclude that using\nlarger models may not always be the right choice in the presence of label noise\nand using robust training methods does not seem to make things better.\n", "summary_of_the_review": "\nThis paper studies an interesting and well-motivated setting and presents some\ntheoretical and empirical results.  I believe that there is some value in the\nextensions of the theoretical results of [1] in the noisy settings and\nempirical observations on using robust training methods are interesting.\nHowever, the theoretical results presented in this work seem to heavily rely on\nprior work and do not seem particularly surprising and novel.  I think it would\nbe be more interesting if the noise models in the theoretical framework were\nmore aligned with the experimental setting (random classification noise).\nMoreover, I am not sure on the practical significance of the empirical results\nas considering that the majority of the training data are corrupted seems to be\na bit unrealistic.  Therefore, overall, I am not sure whether this work is\nabove the threshold for ICLR.\n", "strength_and_weaknesses": "Strengths\n\n1. Studying the effect of overparameterization in noisy settings is\na well-motivated and natural question. This paper provides some theoretical and\nempirical results and also some observations about overparameterization in\nconjunction with robust training techniques in noisy settings that fit in this\nline of research.\n\nWeaknesses\n\n\n1. The bias-variance decomposition closely follows the proofs of the prior work\nof Yang et al. [1].  The theoretical results on the 2-layer linear network\n(Theorem 1) seems to follow by using the same techniques as in [1]. Overall, I\nfound the theoretical results presented in this work not particularly\ntechnically challenging or novel as they follow closely the model and proofs of\n[1].\n\n2. There is a discrepancy between the noise models in the theoretical and\nexperimental results.  The theoretical results consider a regression setting\nwith additive gaussian noise and the experimental results consider\nclassification settings with symmetric or asymmetric random classification\nnoise.  I agree with the authors that studying label noise in classification\nsettings is more relevant so I think it would be better if the theoretical\nresults also considered a similar setting.\n\n3. In the experimental results the noise rates required so that the ascent in\ntest loss happens seem to be very large (in most cases above 50%).  For\nexample, in MNIST the noise rates are 80% and 95%.  For such large noise rates\nin the training data, it does not seem very surprising that the test loss is\neventually increasing as the noisy examples are the majority.  I think it would\nbe interesting to also consider adding random noise to the test data and see\nwhether the \"last ascent\" would then turn into a \"descent\".\n\n[1] Zitong Yang, Yaodong Yu, Chong You, Jacob Steinhardt, and Yi Ma. Rethinking bias-variance trade-off for generalization of neural networks. In International Conference on Machine Learning, pp. 10767–10777. PMLR, 2020.\n\n\nTypos and Minor Issues\n\nAbstract, \"particularly when trained on smaller dataset\" -> datasets\n\nTheorem 1, \"$n/d = \\infty$ should be $n/d \\to \\infty$ (The same for Theorem 2).\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject