Deep Double Descent: Where Bigger Models and More Data Hurt
Authors: Preetum Nakkiran
Gal Kaplun
Yamini Bansal
Tristan Yang
Boaz Barak
Ilya Sutskever
Venue:  (2020)
Keywords: deep learning
double descent
optimization
SGD
complexity

Abstract: We show that a variety of modern deep learning tasks exhibit a "double-descent" phenomenon where, as we increase model size, performance first gets worse and then gets  better.  Moreover, we show that double descent occurs not just as a function of model size, but also as a function of the number of training epochs. We unify the above phenomena by defining a new complexity measure we call the effective model complexity, and conjecture a generalized double descent with respect to this measure. Furthermore, our notion of model complexity allows us to identify certain regimes where increasing (even quadrupling) the number of train samples actually hurts test performance.

Published as a conference paper at ICLR 2020
DEEP DOUBLE DESCENT: WHERE BIGGER MODELS AND MORE DATA HURT
Preetum Nakkiran"
Gal Kaplun
Yamini Bansall
Tristan Yang
Harvard University
Harvard University
Harvard University
Harvard University
Boaz Barak Harvard University
Dya Sutskever
OpenAl
ABSTRACT
We show that a variety of modern deep learning tasks exhibit a "double-descent" phenomenon where, we increase model size, performance first gets worse and then gets better. Moreover, we show that double descent occurs not just as a function of model size, but also as function of the number of training epochs. We unify the above phenomena by defining new complexity measure we call the effective model complexity and conjecture a generalized double descent with respect to this measure. Furthermore, notion of model complexity allows identify certain regimes where increasing (even quadrupling) the number of train samples actually hurts test performance.
INTRODUCTION
Classical Regime:
Modem Regime: Larger Model Better
Bas-Varanne Tradeoff
0.5
Test
0.7
Optimal Early Stopping
0.4-
Test Error
0.6
10
TTaiin
Epochs
0.5
Interpllttion Threshold
0.2
0.4
0.1
0.3
1000
40
ResNet18 width parameter
ResNet18 Width Parameter
Figure 1: Left: Train and test error as a function of model size, for ResNet18s of varying width on CIFAR-10 with 15% label noise. Right: Test error, shown for varying train epochs. All models trained using Adam for 4K epochs. The largest model (width 64) corresponds to standard ResNet18.
The bias-variance trade-off is a fundamental concept in classical statistical learning theory (c.g., Hastie et al. (005)). The idea is that models of higher complexity have lower bias but higher vari- ance. According to this theory, once model complexity passes certain threshold, models "overfit" with the variance term dominating the test error, and hence from this point onward, increasing model complexity will only decrease performance (i.e., increase test error). Hence conventional wisdom in classical statistics is that, once pass certain threshold, "larger models are worse."
However, modern neural networks exhibit no such phenomenon. Such networks have millions of parameters, more than enough fit even random labels (Zhang et al. (2016)), and yet they perform much better on many tasks than smaller models. Indeed, conventional wisdom among practitioners is that "larger models better" (Krizhevsky et al. (2012), Huang al. (2018), Szegedy et al.
Work performed in part while Preetum Nakkiran interning OpenAI, with Ilya Sutskever. We espe- cially thank Mikhail Belkin Christopher Olah for helpful discussions throughout this work. Correspondence Email: preet une cs. harvard, evard.edu I Equal contribution

---

Published as a conference paper at ICLR 2020
Test Error
Train Error
0.8
0.80
Interpoletion Threshord
Modelwize Descont
1k
0.6
0.60
Double
Epochs
0.4
100
100
0.3
0.2
60 1
0.01
30 45
ResNet18 Width Parameter
ResNetla Width Parameter
Test error function of model size and train epochs. The horizontal line corre- sponds to mosel-wise double descent-varying model size while training for as long as possible. The vertical line corresponds to epoch-wise double descent, with test error undergoing double-descent as train time increases. Right Train error of the corresponding models. All models are Resnet trained on CIFAR-10 with 15% label noise, data-augmentation, and Adam for up to 4K epochs.
Figure 2: Left:
2015). Radford et al. (2019)). The effect of training time on test performance is also up for debate. In some settings, "early stopping" improves test performance, while in other settings training neu- ral networks to zero training eπror only improves performance. Finally, if there is thing both classical statisticians and deep learning practitioners agree on is "more data is always better".
In this paper, present empirical evidence that both reconcile and challenge some of the above "conventional wisdoms." We show that many deep learning settings have two different regimes. In the madeer-prrmetterized regime, where the model complexity is small compared the number of samples, the test error as a function of model complexity follows the U-like behavior predicted by the classical bias/variance tradeoff. However, once model complexity is sufficiently large to interpolate i.e., achieve (close to) zero training error, then increasing complexity only decreases test error, following the modern intuition of "bigger models are better". Similar behavior was previously observed in Opper (1995) 2001). Advani & Saxe (2017). Spigler et al. (2018). and Geiger et al. (2019)). This phenomenon was first postulated it "double descent", and demonstrated it for decision trees, random features, and 2-layer neural networks with & loss, on variety of learning tasks including MNIST and CIFAR-10.
generality by Bellin et al. 2018) who named
Main contributions. We show that double descent a robust phenomenon that occurs a variety of tasks, architectures, and optimization methods (see Figure [ and Section 5 our experiments summarized in Table A). Moreover, we propose a much more general notion of "double descent" that goes beyond varying the number of parameters. We define the effective model complexity (EMC) of a training procedure as the maximum number of samples on which it can achieve close to zero training error. The EMC depends not just on the data distribution and the architecture of the classifier but also on the training procedure-ane and in particular increasing training time will increase the EMC.
We hypothesize that for many natural models and learning algorithms, double descent occurs function of the EMC. Indeed we observe "epoch-wise double descent" when we keep the model fixed and increase the training time, with performance following classical U-like curve in the underfitting stage (when the EMC is smaller than the number of samples) and then improving with training time once the EMC is sufficiently larger than the number of samples (see Figure As corollary, early stopping only helps in the relatively narrow parameter regime of critically parameterized models.
Sample non-monotonicity. Finally, our results shed light on test performance as a function of the number of train samples. Since the test error peaks around the point where EMC matches the number of samples (the transition from the under- to over- premettraattti)ia)ooon)), increasing the number of samples has the effect of shifting this peak to the right. While in most settings increasing the number of samples decreases error, this shifting effect can sometimes result in setting where more data is worse! For example, Rngure@ddemonstratte cases in which increasing the number of samples by factor of 4.5 results in worse test performance.
2

---

Published as a conference paper at ICLR 2020
Test loss (per-token perplexity) as function of Transformer model size (embed- ding dimension dmoode) on language trans- lation (IWSLT' German-to-Englisn curve for 18k samples generally lower than the one for 4k samples, but also shifted to the right, since fitting 18k samples requires larger model. Thus, for some models, the performance for 18k samples is worse than for 4k samples.
Figure 3:
Somples Samples
Loss
The
feroddly rilinge
Transformer Embedoing Dimeesion (drose)
2
OUR RESULTS
To state our hypothesis more precisely, we define the notion of effective model complexity. We define training procedure T to be any procedure that takes as input set S {(x1, 21), (In, yh of labeled training samples and outputs classifier T(S) mapping data to labels. We define the effective model complexity of T (w.r.t. distribution D) to be the maximum number of samples which T achieves on average = 0 training error.
Definition (Effectivve Model Complexity) The Effective Model Complexity (EMC) of a training is defined as:
procedure T, with respect to distribution 'D and parameter
FMCD,, = max (n ES-DD (Errors(T(S)) S
where Errors(M) is the mean error of model M on train samples S.
Our main hypothesis can be informally stated as follows:
Hypothesis 1 (Generalized Double Descent hypothesis, informal) For any natural data distribu- tion D. naarallnetwoor-bbseeee training procedure T, and small >> 0, if we consider the task of predicting labels based samples from 'D then:
Lnnderppremettrized regime. IfEMCD,. (T) is sufficiently smaller than n, any perturbation Tf that increases its effective complexity will decrease the test error.
Over Over-parameterized regime. If EMCD, (T) is sufficiently larger than n, any perturbation of T that increases its effective complexity will decrease the test error.
Critically parameterized regime. EfC,,,,(TT) n, then perturbation T that increases its effective complexity might decrease or increase the test error.
Hypothesis Dis informal in several ways. We do not have a principled way to choose the parameter (and currently heuristically use 0.1). We also are yet to have formal specification for "sufficiently smaller" and "sufficiently larger". Our experiments suggest that there is a critical interval around the interpolation threshold when EMCD.. increasing complexity helps performance, while within this interval it may hurt performance. The width of the critical interval depends on both the distribution and the training procedure in ways we do not yet completely understand.
below and above this interval
We believe Hypothesis I and test performance and helps reconcile some of the competing intuitions about them. The main result of this paper an experimental validation of Hypothesis under a variety of settings, where considered several natural choices of datasets, architectures, and optimization algorithms, and we changed the "interpolation threshold" by varying the number model parameters, the length of training, the amount of label noise in the distribution, and the number of train samples.
Isheds light on the interaction between optimization algorithms, model size,
Model-wise Double Descent. In Section 5 we study the test error of models of increasing size, for fixed large number of optimization steps. We show that "model-wise double-descem"" occurs for various modern datasets 10.AR-10, CIFAR-100, IWSLT 14 de-en, with varying amounts of label noise), model architectures (CNNs, ResNets, Transformers), optimizers (SGD, Adam), number
3

---

Published as a conference paper at ICLR 2020
of train samples, and training procedures datt--augmentttion and regularization). Moreover, the peak in test error systematically occurs at the interpolation threshold. In particular, we demonstrate realistic settings in which bigger models are worse.
Epoch-wise Double Descent. In Section O we study the test error of a fixed, large architecture over the course of training. We demonstrate, in similar settings above, a corresponding peak in test performance when models are trained just long enough to reach x 0 train error. The test error of a large model first decreases (at the beginning of training), then increases (around the critical regime), then decreases once more (at the end of training) that is, training longer can correct overfitling.
Sample-wise Non-monotonicity. In Section 7 we study the test error of fixed model and training procedure, for varying number of train samples. Consistent with our generalized double-descent hypothesis, we observe distinct test behavior in the "critical regime". when the number of samples is near the maximum that the model can fit. This often manifests as a long plateau region, in which taking significantly more data might not help when training to completion (as is the case for CNNs on CIFAR-10) 100.. Moreover, we show settings (Transformers on IWSLT' 14 en-de), where this manifests as a peak peak-and for a fixed architecture and training procedure, more data actually hurts.
Remarks on Label Noise. We observe all forms of double descent most strongly in settings with label noise in the train set (as is often the case when collecting train data the real-world), How- ever, we also show several realistic settings with test-error peak even without label noise: ResNets (Figure 4a) and CNNs (Figure 20) on CIFAR- 100; Transformers on IWSLT (Figure S More- over, all our experiments demonstrate distinctly different test behavior in the critical regime- often manifesting as a "plateau" in the test error in the noiseless case which develops into a peak with added label noise. See Section & for further discussion.
RELATED WORK
Model-wise double descent was first proposed as a general phenomenon by Belkin et al. (2018)
Similar behavior had been observed in Opper (195;; 2001), Advani & Saxe (2017). Spigler pergler& et al.
(2018), and Geiger et al. 2019b).. Subsequently. there has been a large body of work studying the
double descent phenomenon. A growing list of papers that theoretically analyze it in the tractable
setting of linear least squares regression includes [Blkin et al. (2019); Hastie et al. (2019); Bartlett
ct al. 2019); Muthukumar al. (2019); Bibas al. (2T2221))))) M0:::)]MMin Mitral 221 (211)) & Montanari (2019).
Moreover, Geiger et al. (2019) 9a) provide preliminary results for model-wise double descent in con- volutional networks trained on CIFAR-10. Our work differs from the above papers two crucial First, we extend the idea of double-descent beyond the number of parameters to incorporate the training procedure under unified notion of "Effective Model Complexity", leading to novel in- sights like epoch-wis whse double descent and sample non- non-monotonicity The notion that increasing train time corresponds to increasing complexity was also presented in Nakkiran et al. (2019). Second, we provide extensive and rigorous demonstration of double-descent in modern deep learning, span- ning a variety of architectures, datasets, and optimization procedures. An extended discussion of the related work is provided in Appendix C
aspects:
EXPERIMENTAL SETUP
We briefly describe the experimental setup here; full details are in Appendix B We consider three families of architectures: ResNets, standard CNNs, and Transformers. ResNets: We parameterize a family of ResNet18s (He et al. (2016) by scaling the width (number of filters) of convolutional layers. Specifically, corresponds to & = 64. Standard CNNs: We consider a simple family of 5-layer CNNs, with 4 convolutional layers of widths [k; 2k: 4k, 8k] for varying k. and a fully-connected layer. For context, the CNN with width h = 64, can reach over 90% test accuracy on CIFAR-10 10 with data- augmentation. Transformers: We consider the 6 layer encoder-decoder from Vaswani et al. (2017). as implemented by Ott et al. (2019). We scale the size of the network by modifying the embedding dimension d'oslel, and setting the width of the fully-connected layers proportionally (dg = 4 dmmote))-
use layer widths [ks 2k, 4k, 8k] for varying k. The standard ResNet18
'The raw data from
experiments
available at:
httpa://gi com.
ha harvard-man wacarr--machinee ne- www.neaannnnngn www.neeeenennnnrririibbba.a.eeea wtw.ab/eeeddeceeettteeeeeeeeeeee
4

---

Published as a conference paper at ICLR 2020
For ResNets and CNNs, train with cross-entropy loss, and the following optimizers: (1) Adam with learning-rate rate 0.0001 for 4K epochs; (2) SGD with learning rate x vT for 500K gradient steps. We train Transformers for 80K gradient steps, with 10% label smoothing and no drop-out.
Label Noise. In our experiments, label noise of probability p refers to training on a samples which have the correct label with probability (1 p). and uniformly random incorrect label otherwise (label noise is sampled only once and not per epoch). Figure Figure[ploot test error on the noisy distribu- tion, while the remaining figures plot test error with respect to the clean distribution (the two curves are just linear rescaling of one another).
5
MODEL-WIL WISE DOUBLE DESCENT
0.8
6%
2% lstel
10% lubel rodue
0.4
3% label rodue
20% intel noten
12% iabel nolee
15%, lobel mose
0.6
0.3
25% notee
0.5
Test
0.2
0.1
0.3
20
30
40
50
60
64
10
20
30
40
60 64
ResNett8
RecNet18 Width Farameter
0.8
0.5
0.3
0.4
0.2
0.0
20
30
40
50
0.0
10
20
30
60
ResNett8 Width Parometer
ResNitt18 Width Parameter
(a) CIFAR-100. There is peak test error even with no lahel noise.
(b) CIFAR-10. There "plateau" test eroor around the interpolation point with no lahel noise, which develops into peak for added label noise.
Figure 4: Midel-wise double descent for ResNet18s. Trained on CIFAR- 100 and CIFAR-10, with varying label noise. Optimized using Adam with LR 0.0001 for 4K epochs, and data augganttmmttnionn.
In this section, we study the test error of models of increasing size, when training to completion (for fixed large number of optimization steps). We demonstrate model-wisc double descent across different architectures, datasets, optimizers, and training procedures. The critical region exhibits distinctly different test behavior around the interpolation point and there is often peak in test error that becomes more prominent in settings with label noise.
For the experiments in this section (Figures 4 6 7 8 notice that all modifications which increase the interpolation threshold (such as adding label noise, using data augmentation, and increasing the number of train samples) also correspondingly shift the peak in test Additional plots showing the early-stopping behavior of these models, and additional experiments showing double descent in settings with no label noise (c.g. Figure 19) are in Appendix E.2 We also observed model-wise double descent for adversarial training, with prominent robust test error peak even in settings without label noise. See Figure Appendix
towards larger models.
Discussion. Fully understanding the mechanisms behind model-wise double descent in deep neu- ral networks remains an important open question. However, an analog model-wise double descent occurs for linear models. A recent stream of theoretical works analyzes this setting (Bartlett et al. (2019); Muthukumar et al. (209)); Belkin et al. (2019); Mei & Montanari (2019); Hastie et al. 2019 We believe similar mechanisms may be at work deep neural networks.
Informally, our intuition that for model-sizes at the interpolation threshold, there effectively only model that fits the train data and this interpolating model is very sensitive to noise in the
5

---

Published as a conference paper at ICLR 2020
0.6
0.6
18241
0.4
0.3
0.3
10
20
30
40
50
60
64
20
30
50
60
64
CNN Width Parameter
CNN Width Parameter
0.6
0.6
0.5
0.5
0.4
CNN Width Parameter
CNN Width Parameter
(a) Without data augmentation.
(h) With data augmentation.
Figure 5: Effect of Data Augmentation. 5-layer CNNs on CIFARIO, with and without data- augmentation. Pata-auumentation shifts the interpolation threshold to the right, shifting the test error peak accordingly. Optimized using SGD for 500K steps. See Figure 27 for larger models.
0.5
0.9
0.7
0.3
0.6
0.5
10
20
30
40
50
60 64
CNN Wiath Parameter
CNN Width Parameter
0.6
0.5
0.4
0.1
0.0
0.0
Farameter
CNN Width Parameter
Figure 6: SGD vs. Adam. 5-Layer CNNs on CIFAR-10 10 with label noise, and no data augmentation. Optimized using SGD for 500K gradient steps, and Adam for 4K epochs.
Figure 7: Noiseless settings. 5-layer CNNs CIFAR-100 with no label noise; note the peak test error. Trained with SGD and no data augmentation. See Fig- ure 20 for the early-stopping behavior of these models.
train set and/or model mis-specification. That is, since the model is just barely able to fit the train data, forcing fit even slightly-noisy or mis-specified labels will destroy its global structure, and result in high test error. (See Figure 28 in the Appendix for experiment demonstrating this noise sensitivity, by showing that ensembling helps significantly in the critically-pan parameterized regime). However for over-parameterized models, there many interpolating models that fit the train set, and SGD able to find one that "memorizes" (or "absorbs") the noise while still performing well on the distribution.
The above intuition is theoretically justified for linear models. In general, this situation manifests without label noise for linear models (Mei & Montanari (2019), and occurs whenever there

---

Published as a conference paper at ICLR 2020
Figure 8: Transformers on language trans- lation tasks: Multi-head-attention encoder- decoder Transformer model trained for 80k gradient steps with labeled smoothed cross-entropy loss on 14 German- to-English (160K sentences) and WMT Fingish-to-French to-h--hhh- (subsampled to 200K sen- tences) dataset. Test loss measured as per- token perplexity.
500
Transformet Model (Embediing D mension)
100
200
300
460
500
(embedding mamssn)))
is model mis- spiisppeciiiattio between the structure of the true distribution and the model family. We believe this intuition extends to deep learning as well, and consistent with our experiments.
EPOCH-WISE DOUBLE DESCENT
In this section, we demonstrate a novel form of double-descent with respect to training epochs, which is consistent with our unified view of effective model complexity (EMC) and the generalized double descent hypothesis. Increasing the train time increases the EMC and thus sufficiently large model transitions from under- to over-parameterized over the course of training.
Small
Intermediate
Large
Models
Models
Models
0.6
Small Model
parameter
Intermediatte Model
Test Error
0.5
0.4
Largo Model
Epochs
0.3
3.0
0.1
0.2
Epochs
ResNet18 Width Parameter
60
Figure 9: Left: Training dynamics for models in three regimes. Models are ResNet18. 8& CIFAR10 with 20% label noise, trained using Adam with learning rate 0.0001, and data augmentation. Right: Test error (Model size Epochs). Three slices of this plot are shown the left.
As illustrated in Figure sufficiently large models can undergo a "double descent" behavior where test error first decreases then increases near the interpolation threshold, and then decreases again. In contrast, for "medium sized" models, for which training to completion will only barely reach Rs error, the test eπror as a function of training time will follow a classical U-like curve where it is better to stop early. Models that are too small to reach the approximation threshold will remain in the "under parameterized" regime where increasing train time monotonically decreases error. Our experiments (Figure T0) show that many settings of dataset and architecture exhibit spoch-wiss double descent, in the presence of label noise. Further, this phenomenon robust across optimizer variations and learning rate schedules (see additional experiments in Appendix E.I). As in model- wise double descent, the test error peak is accentuated with label noise.
Conventional wisdom suggests that training is split into two phases: (1) In the first phase, the net- work learns a function with a small generalization gap (2) In the second phase, the network starts to over-fit fit the data leading to increase in test error. Our experiments suggest that this is not the complete picture-in in some regimes, the test error decreases again and may achieve a lower value at the end of training compared to the first minimum (see Fig 10 for 10% label noise).

---

Published as a conference paper at ICLR 2020
0.8
0.5
0.4
lcbel robs
Test Error
libel robd
Test Ernor
0.4
22% robs
0.3
0.6
0.
0.2
0.8
0.6
0.4
Trai ere
0.6
Train Error
04
Train
0.2
0.0
Epochs
Epochs
Epachs
(a) ResN on CIFARIO.
(b) ResNet18 on CIFARIOO.
(c) 5-layer CNN on CIFAR 10.
Figure 10: Epoch-wise double descent for ResNet18 and CNN (width=128). ResNets trained using learning rate.
Adam with learning rate 0.0001, and CNNs trained with SGD with
7 SAMPLE-WISH NON-MONOTONICITY
In this section, we investigate the effect of varying the number of train samples, for fixed model and training procedure. Previously, in model-wise and epoch-wise double descent, we explored behavior in the critical regime, where EMCD,.(T) = n, by varying the EMC. Here, we explore the critical regime by varying the number of train samples By increasing the same training procedure T can switch from being effectively over-parameterized to effectively under-parammetrrized.
We show that increasing the number of samples has two different effects on the test error vs. model complexity graph. On the one hand, (as expected) inereasing the number of samples shrinks the area under the curve. On the other hand, increasing the number of samples also has the effect of "shifting the curve to the right" and increasing the model complexity at which test error peaks.
Effect of More Samples (10% label noise)
Smaller (drass (A)
25000
56290 Jumoiie
Srot
6.5
0.0
0.3
Cross Entropy Test
1000
120
Width Parameter
20
Effect of More Samples (20% label noise)
0.6
12592
0.5
50200 serviee
More samples Increase loss
0.4
0.2
120
Width
Number Samples
(a) Model-wise double descent for 5-layer CNNs on CIFAR-I0, varying dataset sizes. Top:
(b) Sample-wise non-monotonicity Test loss (per-word perplexity) function of number of train samples, for two transformer models trained to completion on IWSLT' 14. For both model sizes, there is regime where more samples hurt performance. Compare Figure of model-wise double-descent the identical setting.
There is a range of model sizes (shaded green) where training on 2x more samples does not im- prove test error. Bottom: There is range of model sizes (shaded red) where training on 4x more samples does not improve test error.
Figure 11: Sample-wise non-monotonicity.
8

---

Published as a conference paper at ICLR 2020
These twin effects shown in Figure Ila Note that there is range of model sizes where the effects "cancel out"-and having 4x more train samples does not help test performance when training completion. Outside the critically- paraaletttaam regime, for sufficiently under- over- parameterized models, having more samples helps. This phenomenon corroborated in Figure 12 which shows test error as function of both model and sample size, in the same setting as Figure la
Text Frrze
0:55
0000
Intammadiins
33855
0.90
Large Model 1261
25000
8839
0.35
0250
0.0
0.29
49
66
0.20
0.20
12k
20k
30%
40k
50k
Mudil Suor (Wdt) Plsanstee
Semples
Errr
0.35
29091
0.25
17078
2200
02 0.10
3125
0.00
Medel Suz (ChN (Wd2 Plramste))
Num. Train Sumplis
Figure 12: Left: Test Error as function of model size and number of train samples, for 5-layer CNNs on CIFAR-10 20% noise. Note the ridge of high test error again lies along the interpolation threshold. Right: Three slices of the left plot, showing the effect of more data for models of different sizes. Note that, when training to completion, more data helps for small and large models, but does not help for prittcallly-paramettriedd models (green).
In some settings, these two effects combine to yield a regime of model sizes where more data actually hurts test performance as Figure 3 (see also (Tb). Note that this phenomenon is not unique to DNNs: more data can hurt even for linear models (see Appendix D).
8 CONCLUSION AND DISCUSSION
We introduce a generalized double descent hypothesis: models and training procedures exhibit atyp- ical behavior when their Effective Model Complexity is comparable to the number train samples. We provide extensive evidence for our hypothesis in modern deep learning settings, and show that is robust to choices dataset, architecture, and training procedures. In particular, we demon- strate "model-wise double descent" for modern deep networks and characterize the regime where bigger models can perform worse. We also demonstrate "epoch-wise double descent," which, to the best of our knowledge, has not been previously proposed. Finally, show that the double descent phenomenon can lead to a regime where training on more data leads to worse test performance. Preliminary results suggest that double descent also holds as we vary the amount of regularization for a fixed model (see Figure 22
We also believe our characterization of the critical regime provides useful way of thinking for practitioners- if model and training procedure are just barely able to fit the train set, then small changes to the model or training procedure may yield unexpected behavior (e.g. making the model slightly larger smaller, changing regularization, etc. may hurt test performance).
Early stopping. We note that many of the phenomena that we highlight often do not occur with optimal ctrly-stopping. However, this is consistent with our generalized double descent hypothesis: early stopping prevents models from reaching 0 train error then we would not expect to see double- descent, since the EMC does not reach the number of train samples. Further, show at least one

---

Published as a conference paper at ICLR 2020
setting where model-wiss double descent can still occur even with optimal early stopping (ResNets on CIFAR-100 100 with no label noise, see Figure [9). We have not observed settings where more data hurts when optimal carly-stopping is used. However, we not aware of reasons which preclude this from occurring. We leave fully understanding the optimal early stopping behavior of double descent an important open question for future work.
Label Noise. In our experiments. we observe double descent most strongly in settings with label noise. However, we believe this effect is not fundamentally about label noise, but rather about model missspecification. For example, consider setting where the label noise is not truly random, but rather pseudorandom (with respect to the family of classifiers being trained). In this setting, the performance of the Bayes optimal classifier would not change (since the pseudorandom noise is deterministic, and invertible), but we would observe an identical double descent as with truly random label noise. Thus, we view adding label noise as merely a proxy for making distributions "harder" i.. increasing the amount of model mis speiffccciinntion..
Other Notions of Model Complexity. Our notion of Effective Model Complexity is related to classical complexity notions such as Rademacher complexity, but differs in several crucial ways: (1) EMC depends on the true labels of the data distribution, and (2) EMC depends on the training procedure, not just the model architecture.
Other notions of model complexity which do not incorporate features (1) and (2) would not suffice to characterize the location of the couble-descent peak. Rademacher complexity, for example, is determined by the ability of a model architecture to randomly- labeled train set. But Rademacher complexity and VC dimension are both insufficient to determine the model-wiss double descent peak location, since they do not depend on the distribution of labels- and our experiments show that adding label noise shifts the location of the peak.
Moreover, both Rademacher complexity and VC dimension depend only on the model family and data distribution, and not on the training procedure used to find models. Thus, they are not capable of capturing train-time double-descent effects, such as "epoch-wise" double descent, and the effect of data autaaauggentttionn the peak location.
ACKNOWLEDGMENTS
We thank Mikhail Belkin for extremely useful discussions in the early stages of this work. We thank Christopher Olah for suggesting the Model Size x Epoch visualization, which led to the investigation of epoch-wise double descent, as well as for useful discussion and feedback. We also thank Alec Radford, Jacob Steinhardt, and Vaishaal Shankar for helpful discussion and suggestions. P.N. thanks OpenAI, the Simons Institute, and the Harvard Theory Group for research environment that enabled this kind of work.
We thank Dimitris Kalimeris, Benjamin L.. Edelman, and Sharon Qian, and Aditya Ramesh for comments on an early draft of this work.
This work supported in part by NSF grant CAREER CCF 1452961, BSF grant 2014389, NSF US- ICCS proposal 1540428, Google Research award, Facebook research award, a Simons Investiga- tor Award, a Simons Investigator Fellowship, and NSF Awards CCF 1715187, CCF 1565264, CCF 1301976, IIS 1409097, and CNS 1618026. Y.B. would like to thank the MIT-IBM Watson AI Lab for contributing computational resources for experiments.
REFERENCES
Madhu S Advani and Andrew M Saxe. High-dimensional dynamics of generalization error in neural networks. arXiv preprint arXi::1710.036677 2017.
Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. arXiv preprint arXiv: 1300, 2019.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine learning and the Mas-vaaiance trade-off. arXiv preprint 1rxivv1812.... 11 2018.
10

---

Published as a conference paper at ICLR 2020
Mikhail Belkin, Daniel Hsu, and Ji Xu. Two models of double descent for weak features. arXiv 2019.
preprint
Koby Bibas, Yaniv Fogel, and Meir Feder. A new look at an old problem: A universal learning approach linear regression. arXiv preprint arXiv:1905.047000 2019.
Mauro Cettolo, Christian Girardi, and Marcello Federico. Wit Web inventory of transcribed and translated talks. In Proceedings of the 16 Conference of the European Association for Machine Translation (EAMT), pp. 261-268, Trento, Italy, May 2012.
Robert PW Duin. Small sample size generalization. In Proceedings of the Scandinavian Confer- on Image Analysis, volume 2, pp. 957-964. PROCEEDINGS PUBLISHED BY VARIOUS PUBLISHERS, 1995.
Robert PW Duin. Classifiers in almost empty spaces. In Proceedings 15th International Conference on Pottern Recognition. (CPR-2000, volume 2, pp. 1-7. IEEE, 2000.
Mario Geiger, Arthur Jacot, Stefano Spigler, Franck Gabriel, Levent Sagun, Stéphane d'Ascoli, Giulio Biroli, Clément Hongler, and Matthieu Wyart. Scaling description of generalization with number of parameters in deep learning. arXiv preprint arXiv: 1901.01608, 2019a.
Mario Geiger, Stefano Spigler, Stéphane d' Ascoli, Levent Sagun, Marco Baity-Jesi, Giulio Biroli, and Matthieu Wyart. Jamming transition as paradigm understand the loss landscape of deep neural networks. Physical Review E, 100(1):012115, 2019b.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6577 2014.
Trevor Hastie, Robert Tibshirani, Jerome Friedman, and James Franklin. The elements of statistical learning: data mining, inference and prediction. The Mathematical Intelligencer, 27(2):33-85, 2005.
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high- dimensional ridgeless least squares interpolation. arXiv preprint arXiv:1903.0Ss 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference computer vision, pp. 630-645. Springer, 2016.
Yanping Huang, Yonglong Cheng, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoe V. Le, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. CoRR, abs/1811.06965, 2018. URL http::////axiii wtt/:iivrrr//ba/. hbs/aaass/111... 06965
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Marco Loog and Robert PW Duin. The dipping phenomenon. In Joint IAPR International Work- shops on Statistical Techniques Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR), pp. 310-317. Springer, 2012.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.0608; 2017.
Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and double descent curve. arXiv preprint urXiv:1908.05350 2019.
Partha P. Mitra. Understanding overfitting peaks in generalization emoor: Analytical risk curves for 12 and penalized interpolation. ArXiv, abs/1906.03667, 2019.
Vidya Muthukumar, Kailas Vodrahalli, and Anant Sahai. Harmless interpolation of noisy data in regression. arXiv preprint arXiv: 1903.09.99 2019.

---

Published as a conference paper at ICLR 2020
Preetum Nakkiran, Gal Kaplun, Dimitris Kalimeris, Tristan Yang, Benjamin L Edelman, Fred Zhang, and Boaz Barak. Sgd on neural networks learns functions of increasing complexity. arXiv preprint arXiv:1905.11660 2019.
Brady Neal, Sarthak Mittal, Aristide Baratin, Vinayak Tantia, Matthew Scicluna, Simon Lacoste- Julien, and Ioannis Mitliagkas. A modern take on the bias- tradcoff in neural networks. arXiv preprint arXiv:1810.0859,, 2018.
Manfred Opper. Statistical mechanics of learning: Generalization. The Handbook of Brain Theory and Neural Networks, 922-925., 1995.
Manfred Opper. Learning to generalize. Frontiers of Life, 3(part 2), pp.763-775., 2001.
Myle Ott, Sergey Edunov, Alexci Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of Demonstrations, 2019.
NAACL-HLT 2019:
David
Page.
How to
train your resnet.
https://myrtle..
2018
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NeurIPS Autodiff Workshop, 2017.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Advances in neural information processing systems, pp. 1177-1184, 2008.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units, ArXiv, abs/ 508 .7909,99999. 2015.
Marina Skurichina and Robert PW Duin. Bagging, boosting and the random subspace method for linear classifiers. Pattern Analysis Applications, 122):12 2002.
Stefano Spigler, Mario Geiger, Stéphane d' A'scolii Levent Sagun, Giulio Biroli, and Matthieu Wyart. A jamming transition from under-to over- over--arannnn parametrization affects loss landscape and generaliza- tion. arXiv preprini (XXX::811000966: 2018.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Computer Vision and Puttern Recognition (CVPR), 2015. URI.
1409. 4842.
Gerard Trunk. A problem of dimensionality: simple example. IEEE Transactions on pattern analysis and machine intelligence, (3):306-307, 1979.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, abs/1706.03762, 2017.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. ICLR, 1.005330, 2016.
12

---

Published as a conference paper at ICLR 2020
A
SUMMARY TABLE OF EXPERIMENTAL RESULTS
Mouble-Descoos
Dataset
Architecture
Opt.
Aug.
% Noise
Model
Epoch
Figurc(s)
CIFAR 10
CNN
SGD
0
X
X
27
10
0
X
10
20
SGD w.d.
Adam
ResNet
Adam
0
10
5
10
10
Various
20
(subsampled)
CNN
SGD
10
SGD
20
(adversarial)
ResNet
SGD
Robust eIT.
26
CIFAR 100
ResNet
Adam
0
19
10
10
CNN
SGD
0
IWSLT '14 (subsampled) WMT 14 en-fr
Transformer
Adam
0
xx
24
Transformer
Adam
0
Transformer
Adam
x
B APPENDIX: EXPERIMENTAL DETAILS
B.1 MODELS
We use the following families of architectures. The PyTorch Paszke et al. (2017)
specification of ResNets and CNNs are available htttps://gitlse com/
ResNets. We define family of ResNet18s of increasing size as follows. We follow the Preac- tivation ResNet18 architecture He al. (2016), using ResNet blocks, each consisting of two BatchNorm-Rell
layers. The layer widths for the 4 blocks are ik 2k. 4k, 8k] for varying & E N and the strides are [1, 2, 2]. The standard ResNet18 corresponds to t 64 con- volutional channels in the first layer. The scaling of model size with & is shown in Figure 13b Our implementation adapted from https::///// //g /ttp//ggtttutte thub. com/ kua kuangl huupytttppotttei
far
Standard CNNs. We consider a simple family of 5-layer CNNs, with four Nony-BatchNorm- ReI.U-MaXPoo layers and fully-connected output layer. We scale the four convolutional layer widths as [k: 2k, 4k, 8k.. The MaxPool is [1, 2, 2, 8]. For all the convolution layers, the kernel size 3, stride 1 and padding=1. This architecture is based on the "backbone" architecture from Page 2018). For & = 64, this CNN has 1558026 parameters and can reach > 90% test accuracy on CIFAR- 10 Krizhevsky (2099) with augeentugmnnniion. The scaling of model size with is shown
in Figure 13a
Transformers. We consider the encoder-decoder Transformer model from Vaswani al. (2017) with 6 layers and attention heads per layer, as implemented by fairseq Ott et al. (2019). We seale the size of the network by modifying the embedding dimension (dmoda)) and scale the width of the fulle-connected layers proportionally (dæ = 4dmatel). We train with 10% label smoothing and no drop-out, for 80 gradient steps.
13

---

Published as a conference paper at ICLR 2020
Salayar Paratndtars Vlahh
Rechae1B: Peramraters Whtth
Tranformere Paremettrra Emberding Dimenaioon
37"
(a) 5-layer CNNs
(b) ResNet18s
(c) Transformers
Figure 13: Scaling of model size with parameterization of width & embedding dimension.
B.2 IMAGE CLASSIFICATION EXPERIMENTAL. SETUP
We describe the details of training for CNNs and ResNets below.
Loss function: Unless stated otherwise, we use the cross-entropy loss for all the experiments.
Data-augmentttion: In experiments where data-augmentation was used, apply RandomCrop (32, padding=4) and RandomHori wwt.rooooiiotttaiiol. In experiments with added label noise, the label for all augmentations of given training sample given the same label.
Regularization: No explicit regularization like weight decay dropout was applied unless explic- itly stated.
Initialization: We use the default initialization provided by PyTorch for all the layers.
Optimization:
Adam: Unless specified otherwise, learning rate was set at constant to 1e-4 and all other parameters were set their default Py PyToch values.
Unless specified otherwise, learning rate schedule unverse-qquare root (defined be- low) was used with initial learning rate "o 0.1 and updates every L 512 gradient steps. No momentum was used.
SGD:
We found our results are robust to various other natural choices of optimizers and learning rate schedule. We used the above settings because (1) they optimize well, and (2) they do not require experiment-specific hyperparameter tuning, and allow us to use the same optimization across many experiments.
Batch size: All experiments use batchsize of 128.
Learning rate schedule descriptions:
Inverse-square root (1o, L): At gradient step t, the learning rate is set (t) We set learning-rate with respect to number of gradient steps, and not epochs,
70
/ /+11//122 in order to allow comparison between experiments with varying train-set sizes.
Dynamic drop (yo, drop, patience): Starts with an initial learning rate of % and drops by a factor of 'drop' if the training loss has remained constant or become worse for 'patience' number of gradient steps.
B.3 NEURAL MACHINE TRANSLATION: EXPERIMENTAL SETUP
Here we describe the experimental setup for the neural machine translation experiments.
Training procedure.
14

---

Published as a conference paper at ICLR 2020
In this setting, the distribution D consists of triples
(z, y. E Virer E Vigt i {0,..... |e1
where Ver and Vite are the source and target vocabularies, the string x is a sentence in the source language, is its translation in the target language, and is the index of the token be predicted by the model. We assume that xr, y is distributed uniformly on (0...., (....,,yl>> [v])-
A standard probabilistic model defines autoregressive factorization of the likelihood:
PM (M) =
PM (B4 2)).
Given a set of training samples S, we define
Errors(M)
- log par (w:les..))
[S]
In practice, S is not constructed from independent samples from D, but rather by first sampling (I,y) y) and then including all (r, y. 0), ................................ (z, y. (yl) S.
For training transformers, we replicate the optimization procedure specified in Vaswani et al. (2017) section 5.3, where the learning rate schedule consists of a "warmup" phase with linearly increasing learning rate followed by phase with inverse square-root decay. We preprocess the data using byte
pair encoding (BPE) as described in Sennrich et al. (2015). We use the implementation provided by fairseq /ettps:/ggitee
com. wwwpppttttooocctt pwtorcc/ftirreeeee./eepppp fairseq..
Datasets.
The IWSLT 14 German to English dataset contains TED Talks as described in Cettolo al. (0122. The WMT '14 English French dataset is taken from http:/ www. statmt.
org wmt latin-tasst tta...tmskllll
B.4 PER-SECTION EXPERIMENTAL DETAILS
Here we provide full details for experiments in the body, when not otherwise provided.
Introduction: Experimental Details Figure 1 All models were trained using Adam with learning- rate 0.0001 for 4K epochs. Plotting means and standard deviations for trials, with random network initialization.
Model-wise Double Descent: Experimental Details Figure 7 Plotting means and standard devia- tions for 5 trials, with random network initialization.
Sample-wise Nonmonotonicity: Experimental Details Figure Tla All models are trained with SGD for 500K epochs, and data-augmentation. Bottom: Mcans and standard deviations from trials with random initialization, and random subsampling of the train set.
15

---

Published as a conference paper at ICLR 2020
C EXTENDED DISCUSSION OF RELATED WORK
Belkin et al. (2018): This paper proposed, very general terms, that the apparent contradiction between traditional notions of the bias-variance trade-off and empirically successful practices in deep learning can be reconciled under a double-descent curve-as model complexity increases, the test error follows the traditional "U-shaped curve", but beyond the point of interpolation. the error starts to decrease. This work provides empirical evidence for the double-descent curve with fully connected networks trained on subsets of MNIST, CIFARIO, SVHN and TIMIT datasets. They use the l2 loss for their experiments. They demonstrate that neural networks are not an aberration in this regard- double-descent general phenomenon observed also linear regression with random features and random forests.
Theoretical works on linear least squares regression: A variety of papers have attempted to the- oretically analyze this behavior in restricted settings, particularly the case of least squares regression under various assumptions on the training data, feature spaces and regularization method.
1. Advani & Saxe (2017): Hastic et al. (2019) both consider the linear regression problem stated above and analyze the generalization behavior in the asymptotic limit N, D - oo using random matrix theory. |Hastie et al. (2019) highlight that when the model is mis- specified, the minimum of training error can occur for over-parameterized models
2. Belkin et (0199) Linear least squares regression for two data models, where the input data is sampled from Gaussian and Fourier series model for functions on cirele. They provide finite-sample analysis for these two cases
3. Bartlett al. (2019) provides generalization bounds for the minimum 12-norm interpolant for Gaussian features
4. Muthukumar et al. (2019) characterize the fundamental limit of of any interpolating solu- tion in the presence of noise and provide some interesting Fourier-theoretic theoretic interpretations.
5. Mei & Montanari (2019): This work provides asymptotic analysis for ridge regression over random features
Similar double descent behavior, restricted settings, was investigated in Trunk (197 Opper (1995 2001): Skurichina & Duin 2002)..
Neal ct al. (2018) conducts a study of bias and variance in modern neural networks, observing that both bias and variance decrease with increasing model size, contrary to conventional wisdom.
Geiger et al. (2019b) showed that deep fully connected networks trained on the MNIST dataset with hinge loss exhibit a "jamming transition" when the number of parameters exceeds a threshold that allows training to near-zero train loss. l Geiger et al. (2019a) provide further experiments on CIFAR- 10 with convolutional network. They also highlight interesting behavior with ensembling around the critical regime, which is consistent with our informal intuitions in Section 5 and our experiments
in Figures 28 29
Advani & Saxe (2017): : Geiger et al. 2019b also point out that double-descent is not observed when optimal early parly-stpppig used.
The study of sample non-monotonicity in learning algorithms had also existed prior to double de- scent, including in Duin [1995] 2000): Opper (200T): : Loog Duin 2012).
16

---

Published as a conference paper at ICLR 2020
D RANDOM FEATURES: A CASE STUDY
Samples Features Heatmap
RFF
2000
2000
1750
0335
1750
1500
0.30
07
05
500
500
250
0.00
250
03
0.00
Figure 14: Random Fourier Features on the Fashion MNIST dataset. The setting is equivalent to two-layer neural network with activation, with randomly -inttlyy-iitializeed first layer that is fixed throughout training. The second layer is trained using gradient flow.
In this section, for completeness sake, show that both the model- and sample-win wisp double de- scent phenomena are not unique to deep neural networks-thor exist even in the setting of Random Fourier Features Rahimi & Recht 2008). This setting is equivalent with e -iz activation. The first layer is initialized with a (O, 4) Gaussian distribution and then fixed throughout training. The width (or embedding dimension) of the first layer parameterizes the model size. The second layer is initialized with Os and trained with MSE loss.
two-layer neural network
Figure shows the grid of Test Error function of both number of samples n and model size d. Note that in this setting EMC = (the embedding dimension), As result, as demonstrated in the figure, the peak follows the path of d. Both model-wise and sample-wise (see figure [5) double descent phenomena are captured, by horizontally and vertically crossing the grid, respectively.
Test
0.7
Train
0.6
0.5
Error
0.4
0.3
0.2
0.1
0.0
500
1000 Number of Samples
1500
2000
Figure 15: Sample-wise double-descent slice for Random Fourier Features on the Fashion MNIST dataset. this figure the embedding dimension (number random features) 1000.
17

---

Published as a conference paper at ICLR 2020
E APPENDIX: ADDITIONAL EXPERIMENTS
E.1 EPOCH-WIL WISH DOUBLE DESCENT: ADDITIONAL RESULTS
Here, we provide a rigorous evaluation of epoch- wise double descent for a variety optimizers and learning rate schedules. We train ResNet18 CIFAR-10 with data-augmentation and 20% label noise with three different aptimizers-Adam SGD, SGD + Momentum (momentum set to 0.9) and three different learning rate values of initial learning rate. We observe that double-descent occurs reliably for all optimizers and learning rate schedules and the peak of the double descent curve shifts with the interpolation point.
inverse-square root, dynamic drop for differnet
2:C00L
D.C00
I D-C0OL
0.8
5:001
0.001
DICOL
Test Error
0.6
0.6
0.0
Test
0.4
Test
0.4
0.2
0.8
0.0
0.6
Train Erer
0.6
0.4
Train
0.4
0.4
0.0
107
10
10
18P
10
10
13
10a
10P
10
10
12
10
13P
10*
Iterations
(a) Constant learning rate
(b) Inverse-square root learning
(c) Dynamic learning rate
rate
Figure 16: Epoch-wise double descent for ResNet18 trained with Adam and multiple learning rate schedules
practical recommendation resulting from epoch-wise double descent is that stopping the training when the test error starts to increase may not always be the best strategy. In some cases, the test error may decrease again after reaching a maximum, and the final value may be lower than the minimum earlier in training.
5.000L
0.001
=
0.001
0.8
0:01
Test Error
0.6
0.6
9.0
Test
0.4
9.4
0.2
0.8
0.8
0.8
0.6
0.
Train Emor
0.6
02
0.0
0.0
0.0
107
10+
10"
10P
10*
107
13r
10"
10P
10
107
13r
16'
10P
10*
lterations
llerations
Blerations
(a) Constant learning rate
(b) Inverse-square root learning
(c) Dynamic learning rate
rate
Figure 17: Epsch-wise double descent for ResNet18 trained with SGD and multiple learning rate schedules
18

---

Published as a conference paper at ICLR 2020
bicto1 0:001
biteot V- 0001
D.000:
0.8
0600
2:03
1- D.2
Test Error
1 D.C3
0.6
0.6
Test
0.2
02
022
0.8
0.6
Engr
0.6
0.4
04
04
10a
10
18
10a
10
10
10
10a Iterations
15F
10
Constant learning (b) Inverse-square root learning
(c) Dynamic learning rate
rate
Figure 18: wisch-wise double descent for ResNet18 8 trained with Mom+Momn and multiple learning rate schedules
E.2 MODEL--WISH DOUBLE DESCENT: ADDITIONAL. RESULTS
E.2.1
CLEAN SETTINGS WITH MODEL-WISE DOUBLE DESCENT
CIFARIO0, ResNet18
Test Error
Train Error
0.80
0.9
0.8
1k
1k
0.60
0.7
0.6
Epochs
0.40
0.5
0.4
0.20
10
10
0.01
15
30
45
60
15
30
45
60
ResNet18
Parameter
ResNet18 Width Parameter
Test Time Dynamics for ResNet18 on Cifar100 With No Noise
Optimal Stopping Final train Loss
jstt Error
0.6
Epochs
0.4
100
1000
10
20
40
60
ResNet18 Width Parameter
Figure 19: Top: Train and test performance as function of both model size and train epochs. Bottom: Test error dynamics of the same model (ResNet18, CIFAR- with no label noise, auta-ntnmentttion and Adam optimizer trained for 4k epochs with learning rate 0.0001). Note that with optimal early stopping this setting exhibits double descent.
19

---

Published as a conference paper at ICLR 2020
CIFAR100, Standard CNN
Test Error
Train Error
0.80
1K
0.9
1k
0.8
0.60
Epochs
0.40
0.6
0.20
0.5
10
10
15
30
45
60
15
30
45
60
Width Parameter
Width Parameter
Test Time Dynamics for CNNs Cifar100 With No Noise
Optimal Early Stopping
0.8
Test Error
Epochs
0.7
100
0.6
0.5
1000
0.4
10
10
60
Figure 20: Top: Train and test performance as function of both model size and train epochs. Bottom: Test error dynamics of the same models. 5-Layer CNNs, CIFAR-100 with no label noise, no augmeetggmnttoionn Trained with SGD for le6 steps. Same experiment as Figure
20

---

Published as a conference paper at ICLR 2020
E.2.2
WEIGHT DECAY
No Regularization
Optimal Earty
Tea Emor
0.5
0.4
80
0.3
100
0.2
0.5
1000
Tean Erer
20
40
60
80
100
120
0.5
5e-4 Weight Decay
0.
Optimal Earty Stopping
0.6
0.0
0.4
2.0
100
Test
0.2
1000
20
60
100
120
ResNet18 width parameter
120
Figure 21: Left: Test error dynamics with weight decay of Se-4 (bottom left) and without weight decay (top left). Right: Test and train error and test loss for models with varying amounts of weight decay. All models are 5-Layer CNNs on CIFAR-10 with 10% label noise, trained with data- data-augmentttion and SGD for 500K steps.
Here, we now study the effect of varying the level of regularization on test error. We train CIFARIO with data-augmentation and 20% label noise on ResNet18 for weight decay co-efficients rang- ing from 0 to 0.1. We train the networks using SGD inverse-square root learning rate. Figure below shows a picture qualitatively very similar to that observed for model-wise double descent wherein "model complexity" is now controlled by the regularization parameter. This confirms generalized double descent hypothesis along yet another axis of Effective Model Complexity.
0.8
0.80
1k
0.60
Epochs
0.4
Epochs
100
100
0.40
0.3
0.20
0.2
10
10
0.01
1
1
85
6.2e3
4.5e4
Inf
11
85
6.2e3
4.5e4
inf
1/A
1/A
(a) Test Error
(b) Train Loss
Figure 22: Generalized double descent for weight decay. We found that using the same initial learning rate for all weight decay values led training instabilities. This resulted some noise in the Test Error (Weight Decay Epochs) plot shown above.
21

---

Published as a conference paper at ICLR 2020
E.2.3
EARLY STOPPING DOES NOT EXHIBIT DOUBLE DESCENT
Language models
18
Enn-ooy Test
16
14
Epochs
12
10
100
50
100
150
200
100
150
200
Transtormer Embedding Dim (dmode))
Transtormer Embedding Dim (dmode)
Figure 23: Model-wise dynamics for subsampled IWSLT'14 14 dataset. Left: 4k samples,
Right: 18k samples. Note that with optimal early-stopping, more samples is always better.
De-En
En-Fr
10
Epechs
Cross- Entropy
10
200
200 Transformer Embedding Dim (dmostr)
Transformer Embedding Dim (dmcola)
Figure 24: Model-wise test error dynamics for IWSLT de-en and subsampled 14 en-fr datasets. Left: IWSLT114, Right: subsampled (200k samples) WMT'14. Note that with optimal early-stopping, the test error is much lower for this task.
CIFARIO, 10% % noise, SGD
Train
Error
tooets
100
120
0.
03
15
30
45
60
75
15
30
45
75
Time Dynamics for CIFAR-10 with Nolse
3.7
Tel Errur
Epacc hs
3.5
100
3.3
Width Parameter
Figure 25: Top: Train and test performance function of both model size and train epochs. Bottom: Test error dynamics of the same model (CNN, on CIFAR-I0 with 10% label noise, data- augmentation and SGD optimizer with learning rate o 1/VT).
22

---

Published as a conference paper at ICLR 2020
E.2.4 TRAINING PROCEDURE
0.9
6p5-
Robust Texz Error
0.7
0.5
0.4
ResNec18 Width Parameter
Standard Tozz Enro
0.8
0.6
Restiet18 Width Parameter
0.8
Train Error
0.6
0.0
1
20
40
60
00
100
128
ResNet188 Width Parameter
Figure 26: Model-wise double descent for adversarial training ResNet18s on CIFAR-10 (sub- sampled 25k train samples) with no label noise. We train for L2 robustness of radius 0.5 and 1.0, using 10-step PGD (Goodfellow et al. (2014); Madry et al. (2017)). Trained using SGD (batch size 128) with learning rate 0.1 for 400 epochs, then 0.01 for 400 epochs.
0.6
0% label noise
10% label noise
0.4
Test
0.3
0.2
100
120
CNN Width Parameter
0.6
0.5
Train Error
0.4
0.3
0.2
0.1
0.0
100
120
CNN Width Parameter
Figure 27
23

---

Published as a conference paper at ICLR 2020
E.3
ENSEMBLING
Standard Ensemble
0.4
Test Error
0.3
0.2
0.1
0.0
40
ResNet18 width parameter
Figure 28: Effect of Ensembling (ResNets, 15% label noise). Test enror of an ensemble models, compared to the base models. The ensembled classifier is determined by plurality vote the 5 base models. Note that emsembling helps most around the critical regime. All models are ResNet1 8s trained on CIFAR-10 with 15% label noise, using Adam for 4K epochs (same setting as Figure D Test error is measured against the original (not noisy) test set, and each model in the ensemble trained using train with independently-smpee 15% label noise.
Standard
0.8
Ensemble
Test Error
0.6
0.2
0.0
CNN width parameter
Figure 29: Effect of Ensembling (CNNs, no label noise). Test error of an ensemble of models, compared the base models. All models 5-layer CNNs trained CIFAR-10 with label noise, using SGD and no data augmentation. (same setting as Figure 7).
24


Reviews:
[{"review_id": "rJxvG0y6KH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I do not have much to say about the paper except that I like the sumilation, and found rather interesting. It shows a rather extensive set of simulations, that enrich the observations of the so-called double descent phenomena, and shows empirically its apparent generality. Also, the Effective Model Complexity seems to be a good description of what is empirically observe. All in all, i definitely support publication.\n\nI have, however, two comments: first, a minor one: there is a part of the story on double descent that is completely forgotten: the fact that there is a pic in the data for some classificators at #example close to #parameters was known at least in the 90! It is discussed in \"Statistical Mechanics of Learning\" A. Engel & C. Van den Broeck, 2001, page 61, with a plot that  comes rather from an old work of Pr. Manfred Opper (1995), can be seen here: http://www.ki.tu-berlin.de/fileadmin/fg135/publikationen/opper/Op01.pdf FIG 10 \nFrom this work, it is also rather clear that increasing the number of training examples can hurt performances (this is exactly what the plot says), at least without regularisation. Since the authors are quite thorough in their review of history -especially in the appendix- I thought I would point this reference.\n\nThe second comment is that the paper should acknowledge the work of the  Geiger et al more explicitly, and cite as well the related work of  Spiegler et al (https://arxiv.org/abs/1810.09665) rather explicitly! Virtually all they show was already shown in this work (which by the way PREDATES Belkin et al), albeit only for a simpler set of data.  I agree that the authors extend a lot on this papers, especially in terms of dataset and completeness of experiments, but they are definitely closely related, and the fact that it is not cited is a serious flaw to the current version.\n\nAlso, in the conclusion  \"We note that many of the phenomena that we highlight often do not occur with optimal early- stopping. However, this is consistent with our generalized double descent hypothesis:…\". Again this was shown (empirically) as well in the above mentioned papers and also explained in \"physics terms\" where the peak is like a phase transition (jamming), if one stop the dynamics.\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1xTYre6tB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides a valuable and detailed empirical study of the double descent behaviour in neural networks. It investigates presence of this behaviour in a range of neural network architectures and apart of identifying it as a function of the model size it also identifies it as a function of training time which I believe is novel. Overall I think the paper presents results valuable to the community. At the same time it has several issues that need to be addressed. If the issues are addressed in a satisfactory manner I will recommend acceptance of the paper. \n\nIssues and questions: \n\n** \"Such a phenomenon was first postulated by Belkin et al. (2018) who named it “double descent” and demonstrated it on MNIST with decision trees, random features, and 2-layer neural networks with `2 loss.\" This is not a correct statement. The authors cite works (Advani & Saxe 2017) and there is also \"Stefano Spigler, Mario Geiger, Stephane d’Ascoli, Levent Sagun, Giulio Biroli, and Matthieu Wyart. A jamming transition from under-to over-parametrization affects loss landscape and generalization. arXiv preprint arXiv:1810.09665, 2018.\" that both identified this behaviour prior to the Belkin et al. (2018) work. There is even a much older line of work identifying the peak in the generalization error by Opper's group: \nhttp://www.ki.tu-berlin.de/fileadmin/fg135/publikationen/opper/Op03b.pdf (1995)\nSiegfried B¨os and Manfred Opper. Dynamics of training. In Advances in Neural Information\nProcessing Systems, pages 141–147, 1997.\nhttp://www.ki.tu-berlin.de/fileadmin/fg135/publikationen/opper/Op01.pdf\n\nMoreover, as is only said in the supplementary material, Geiger et al. also tested on the CIFAR dataset, while the introduction of the present article only mentions previous experiments on MNIST. The fact that previous work also observed this in CIFAR should be moved in the introduction. \n\n** The authors claim to define the \"effective model complexity (EMC)\" and claim this as one of the main results of the paper.\n\nPresenting this as a new measure is strange, this exactly is what Geiger et all call jamming transition/threshold in Stefano Spigler, Mario Geiger, St´ephane d’Ascoli, Levent Sagun, Giulio Biroli, and Matthieu Wyart. A jamming transition from under-to over-parametrization affects loss landscape and generalization. arXiv preprint arXiv:1810.09665, 2018. And very closely related to what Belkin calls \"interpolation threshold\" (to define interpolation threshold we could fix the model and vary the number of samples, Belkin et all fix number of samples and vary size of the model, but one is just the dual of the other). Giving it a yet completely new name seems to create more noise than value.\n\nThe relation between EMC and the jamming transition is discussed in the supplement. However, not in a accurate way. The authors say \"a ”jamming transition” when the number of parameters exceeds a threshold that allows training to near-zero train loss.\" but jamming is inspired by the physical phenomena when spheres are added into a finite volume box and the more spheres we have the harder it gets to fit them all in until it is not possible anymore. The analogy here is that fitting sphere in = reaching training error zero. Then number of spheres = number of samples. Thus the more samples the harder it is to get the training error to zero, leading to the jamming transition. Both in training and jamming the number of samples at which this happened depends on the details of the protocol/training, thus it does depend on things such as regularization.\n\nThe only aspect that I have not seen covered in the jamming analogy is the epoch-wise double descent.\n\nIn any case the discussion in the paper needs to be adjusted and these relevant relations to previous work corrected and moved from the supplement to the introduction.  \n\n** \"Informally, our intuition is that for model-sizes at the interpolation threshold, there is effectively only one model that fits the train data and this interpolating model is very sensitive to noise in the train set and/or model mis-specification.\" This intuition is correct I believe. However, it should not be called \"our intuition\" as it already appeared in the line of work by Opper cited above. \n\n** The authors present as another main result the fact that under comparable training conditions training with more data samples provides worse generalization, examples of this is also included already in the papers by Opper et al. cited above. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1lrZBqMoH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper defines the effective model complexity(EMC) that defines the complexity of the model. EMC depends on several factors such as data distribution and architecture of the classifier. \n \nThe paper empirically shows that the double descent phenomenon occurs as a function of EMC. \n\nThe paper provides interesting perspectives of their experiments and gives the intuitions for these observations. However, these are mainly hypotheses.\n\nI like the paper for its rigorous experiments and providing intutions from their observations.\n\nHowever, I am very new to this area of research and will only provide my review on my understanding.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper experimentally analyzes the double descent phenomenon for deep models. While, as the reviewers have mentioned, this phenomenon has been observed for some time, some of its specificities still elude us. As a consequence, I am happy to see this paper presented to ICLR.

That being said, given the original lack of proper references as well as the recent public announcements about this paper giving it visibility, I want to make it absolutely clear that this paper is accepted with the assumption that proper credit will be given to past work and that efforts will be made to draw connections between all these works.
Decision: Accept (Poster)