Mitigating Deep Double Descent by Concatenating Inputs
Authors: John Chen
Qihan Wang
Anastasios Kyrillidis
Venue:  (2021)
Keywords: deep double descent
feedforward neural network
image classificaiton

Abstract: The double descent curve is one of the most intriguing properties of deep neural networks. It contrasts the classical bias-variance curve with the behavior of modern neural networks, occurring where the number of samples nears the number of parameters. In this work, we explore the connection between the double descent phenomena and the number of samples in the deep neural network setting. In particular, we propose a construction which augments the existing dataset by artificially increasing the number of samples. This construction empirically mitigates the double descent curve in this setting. We reproduce existing work on deep double descent, and observe a smooth descent into the overparameterized region for our construction. This occurs both with respect to the model size, and with respect to the number epochs.

Under review as a conference paper at ICLR 2021
MITIGATING DEEP DOUBLE DESCENT BY CONCATE- NATING INPUTS
Anonymous authors Paper under double-bling review
ABSTRACT
The double descent curve is one of the most intriguing properties of deep neural networks. It contrasts the classical bias-variance curve with the behavior of mod- ern neural networks, occurring where the number of samples nears the number of parameters. In this work. we explore the connection between the double de- scent phenomena and the number of samples in the deep neural network setting. In particular, we propose construction which augments the existing dataset by artificially increasing the number of samples. This construction empirically miti- gates the double descent curve in this setting. We reproduce existing work on deep double descent, and observe smooth descent into the overparameterized region for our construction. This occurs both with respect to the model size, and with respect to the number epochs.
INTRODUCTION
Underparameeterzation and overparameterization are at the heart of understanding modern neural networks. The traditional notion of underparametterization and overparameterization led the clas- sic U-shaped generalization error curve (Tevor Hastie & Friedman] 2001 Stuart Geman & Doursat 1992]. where generalization would worsen when the model had either too few (underparameterizzed or too many parameters (overparameterized), Correspondingly. it was expected that an underparam- eterized model would underfit and fail to identify more complex and informative patterns, and an overparameterized model would overfit and identify non-informative patterns. This view no longer holds for modern neural networks.
It is widely accepted that neural networks are vastly overparameterized, yet generalize well. There is strong evidence that increasing the number of parameters leads to better generalization (Zagoruyko & Komodakis 2016 Huang al. 2017: Larsson et al., 2016), and models often trained to achieve zero training loss (Salakhutdinov (207)). while still improving in generalization error, whereas the traditional view would suggest overfitting.
To bridge the gap, Belkin et (2018a) proposed the double descent curve, where the underparame- terized region follows the U-shaped curve, and the overparameterized region smoothly decreases in generalization error, as the number of parameters increases further. This results in peak in general- ization error, where fewer number of samples would counter-intuitively decrease the error. There has been extensive experimental evidence of the double descent curve in deep learning (Nakkiran al. 2019 Yang et al.) (020), well as in models such as random forests, and one layer neural networks (Belkin et al. 2018a Ba et al. 2020).
One recurring theme in the definition of overparameterizzttion and underparameterrzzaion lies in the number of neural network parameters relative to the number of samples (Belkin et | 20188a
Nakki- ran al.) 2019 Ba al., 2020 Bibba al.) 2019 Muthukumar et al., 2019 Hastie et al., (019).
high level, greater number of parameters than samples is generally considered overparameteri- zation, and fewer considered underparamettrizzaion..
However, this leads the question "What is sample?" In this paper, revisit the fundamen- tal underpinnings of overparameterization and underparameterizztion,, and stress test when means to be overparameterized underparameterizee, through extensive experiments of cleverly structed input. We artificially augment existing datasets by simply stacking every combination of inputs, and show the mitigation of the double descent curve in the deep neural network setting. We

---

Under review as a conference paper at ICLR 2021
humbly hypothesize that in deep neural networks can, perhaps, artificially increase the number of samples without increasing the information contained in the dataset, and by implicitly changing the classification pipeline mitigate the double descent curve. In particular, the narrative of our paper obeys the following:
We propose a simple construction to artificially augment existing datasets of size (()) by stacking inputs produce dataset of size (n²).
We demonstrate that the construction has no impact on the double descent curve the linear regression case.
We show experimentally that those results on double descent curve do not extend to the case of neural networks.
Concretely, we reproduce results from recent landmark papers, and present the difference in behavior with respect to the double descent curve.
2 RELATED WORKS
The double descent curve was proposed recently in (Belkin et al. 2018a), where the authors define overparameterization and underparameterization as the proportion of parameters to samples. The authors explain the phenomenon through the model capacity class. With more parameters in the overparameterized region, there is larger "capacity" (i.e., the model class contains more candidates), and thus may contain better, simpler models by Occam's 's Razor rule. The interpolation region is sug- gested to exist when the model capacity is capable of fitting the data nearly perfectly by overfitting on non-informative features, resulting in higher test error. Experiments included a one layer neural network, random forests, and others.
The double descent curve is also observed in deep neural networks (Nakkiran et 2019). with the additional observation of epoch-wise double descent. Experimentation is amplified by label noise. With the observation of unimodel variance (Neal et 2018 Yang et al. (2020) also decomposes the risk into bias and variance, and posits that the double descent curve arises due to the bell-shaped variance curve rising faster than the bias decreases.
There is substantial theoretical work on double descent, particularly in the least squares regression setting. Advani & Saxe (2017) analyses this linear setting and proves the existence of the interpola- tion region, where the number of parameters equals the number of samples in the asymptotic limit where samples and parameters tend to infinity. Hastic et al. (2019) follows a similar line of work, and proves that regularization reduces the peak in the interpolation region. [Belkin et al. (201bb) requires only finite samples, where the features and target be jointly Gaussian. Other papers with similar setup include (Bartlett al. 2 Muthukumar et al. 2019 Bibas et al. 2019 Mitra 2019 Mei & Montanari 2019).
Ba al. (020) analyses the least squares regression setting for two layer linear neural networks in the asymptotic setting, where the double descent curve is present when only the second layer is optimized. There is also work proving that optimally tuned f2-norm regularization mitigates the double descent curve for certain linear regression models with isotropic data distribution (Nakkiran)
2019 This setting has also been studied with respect the variance in the parameter space (Bartlet et al. 2019). Multiple descent has also been studied, and in particular there is work to show in the linear regression setting that multiple descent curves can be directly designed by the user (Chen al.) 2020). Additionally, there is supporting evidence of double descent in the sample-wise perspective Nakkiran al., 2020).
There is other work in this area, including studying the double descent curve for least squares in random feature models (Belkin et al. 2219a; d' Ascoli et al. 20200] Ghorbani et al.) 2019), 9,, leveraging the Neural Tangent Kernel to argue for certain number of parameters the output of the neural network
diverges (Giger et al. 2020). characterizing the double descent in non-linear settings (Caron & Chretien 2020), kernel leannig (Belkin et al., 2018b; Liang al. 2019), and connecting other fields (Geiger et al. 2019). Lastly, we note here that, in the deep neural network setting, models can be trained to zero training loss even with random labels (Zhang et al. 2016).
2

---

Under review as a conference paper at ICLR 2021
0.20
0.15
Test MSE
0.3
0.10
Test
0.2
0.05
0.1
0.00
0.0
20
40
60
20
40
60
Number samples
Number of samples
Right: The concatenated inputs construction. Plots of the Test MSE verses number of samples (pre-concatenation) for min-norm ridgeless regression. where d = 30. Following Nakkiran (2019). inputs are drawn ~ NOO, Ta), target y Ox + (0,
Figure 1: Left: The standard case.
where 0 are the parameters, ||0||2 1, 0.1. θ Xty.
THE CONCATENATED INPUTS CONSTRUCTION
We introduce the concatenated inputs construction, which our main hypothesis is based The concatenated inputs construction refers to the general idea concatenating pairs of inputs and element-wise adding and averaging pairs of outputs produce new inputs and targets. This way the size of dataset can be artificially -but non-trivially- increased.
This construction be applied both the regression setting and the classification setting. In the setting of linear regression, for given input pairs, (41.81). (x2, B2), an augmented dataset can be constructed as:
(|ا1, 21 t1
([x1,
21 (1x2, v1), 2art (102, 2
where [a, 3) represents concatenation the input a, B. In the setting of classification, the process identical, where the targets produced by dlement-wise addition and then averaged to sum to 1. The averaging is not strictly necessary even in the deep neural network classification case, where the binary cross entropy loss be used instead of cross entropy. For test data, concatenate the same input with itself, and the target is the original target. This way a dataset of size O(n) is artificially augmented size (
Concretely, our reasons for the concatenated inputs construction are as follows: i) there is limited injection of information semantic meaning: ii) the number of samples significantly increased. For the purposes of understanding underparameterization overparameterizatio and the double de- scent curve, such construction tries isolate the definition of a sample. We revisit and assess these implications in the context of extensive experiments in the following sections.
4 RESULTS
In this section, we reproduce settings from benchmark double descent papers, add the concatenated inputs construction and analyze the findings. particular, begin with the linear regression, move to one hidden layer feedforward neural networks, and then deep neural networks, in both the model parameter-wis double descent and epoch-wise double descent. Finally, we analyze the performance of deep neural networks for the concatenated inputs construction, and the behavior of the double descent curve in the classification setting, when we transfer from the cross entropy the binary cross entropy loss.
4.1
LINEAR REGRESSION
The linear regression setting has been a fruitful testbed for empirical work in double descent, as well yielding substantial theoretical understanding. The concatenated inputs construction is applied

---

Under review as a conference paper at ICLR 2021
15
15-
1,0
10-
0.5
10a
10"
10a
10
10'
10'
number paremetioss
number of pa paremeters
number of paremeterss
0.6
0.4
0.4
0.2
10
10a
10
10
10'
10
number
number
Figure 2: Top row: Loss against number of parameters. Bottom row: Error against number of parameters. In order from left to right: Input 32 x 32 image, label is one-hot vector and loss is Cross Entropy, 2. Input is 64 x 32 image (two of the same image stacked on top of each other), label one-hot vector and loss is Cross Entropy, 3. Input is 64 x 32 image (two different 32 x 32 images stacked top of each other), label is two-hot vector of values 0.5 and 0.5 and loss is Cross Entropy. All models trained with batch size 100 for 1000 epochs on a subset (n = 4000) of the 0.999. Models
MNIST dataset. Adam optimizer is used with learning rate 0.001, 31 0.9, are feedforward neural networks with a single layer of hidden units with ReLU activation.
similarly here, however with different motivation. Namely, we wish motivate that the concate- nated inputs construction is not expected to add any information and is therefore not expected to impact the double descent curve.
We reproduce the linear regression setting from Nakkiran (2019) given in Figure | For the con- catenated inputs construction, we first draw the number samples before concatenation and con- struction of the augmented dataset. We observe that, by construction, the concatenated inputs con- struction does not affect the double descent curve, and the peak occurs in the exact same location. We also make the remark here that it is not surprising that this is the case, and it is not complicated to understand why from a theoretical perspective.
4.2 ONE HIDDEN LAYER FEEDFORWARD NEURAL NETWORK
Following linear regression, we move to neural networks. We train feedforward neural network with one hidden layer and ReLU activations on subset of the MNIST dataset, reproducing the experimental setup from Belkin et al. (2018a) Sa). We vary the number of parameters in the neural network by changing the size of the hidden layer. We use the Cross Entropy loss instead of the original MSE loss due to the prevalence of those losses for image classification tasks. This is shown Figure 2
We observe the double descent in the loss versus number of parameters, but were unable to produce the double descent in the error. the rightmost plot, the double descent curve completely removed in the concatenated inputs construction relative to the other two settings. Namely, smooth decrease in loss observed, where there is clear double descent in the other cases. Furthermore, we provide the extra setting of concatenating each input only by itself, and the double descent curve is present almost exactly in this scenario. This provides further evidence that the disappearance of the double descent is not due the extra parameters which originate from the larger sized inputs. In this setting, appears that the behavior of underparameterizztion and overparameterization can be altered by simply artificially increasing the number of samples through concatenating images.

---

Under review as a conference paper at ICLR 2021
1.75
15
150
1.25
loss
1.0
0.75
0.5
0.50
0.25
0.0
0
10
20
30
40
50
60
20
40
50
Width ResNet-18
Width of ResNet-18
0.6
0.5
0.5
0.4
0.4
erro
0.3
error
0.3
0.2
0.2
0.1
0.1
0.0
10
20
30
40
60
10
20
30
40
50
60
Width ResNet-18
Width of ResNet-18
Figure 3: Left: Standard one-hot vector setup, Right: Concatenated inputs construction. Top: Loss. Bottom:
Error. All models trained with batch size of 128 for 400 epochs with 15% label noise the CIFARIO dataset. Label noise defined by changing p% and prior to the concatenated inputs construction. Adam optimizer used with learning rate 0.0001, & 0.9. B2 0.999. Models are ResNer18 architecture where the width & is varied. & 64 is the standard ResNet18. The plot is visually similar when plotted against parameters, since concatenated inputs construction typically adds less than 5% of the total parameters.
samples to random wrong labels and is applied prior to training.
In addition, the model trained on MNIST and one-hot vectors can be concatenated with itself, with all other parameters being zero, to produce a model with two times the number of hidden units which can be applied to the concatenated inputs construction. We consider this setting in the context of a possible explanation of the interpolation region, where the number of parameters nears that of samples. Concretely, it is possible for neural network with double the hidden units in the concate- nated inputs construction to recover the double descent curve by learning two smaller, disconnected networks, where each of the smaller networks are the ones learned in the double descent peak of the standard, one-hot case. However, in practice while the network can do so, does not appear to, which leads to the smooth descent in the rightmost plot in Figure
4.3 DEEP NEURAL NETWORKS
In this section, we investigate deep double descent in several settings, including model size double descent, and epoch-wise double descent. Finally, we decompose the loss into bias and variance to further investigate the effect of the concatenated inputs construction.
4.3.1 MODEL SIZE DOUBLE DESCENT
ResNet18 CIFARIO. We train ResNet18 architecture the CIFAR10 dataset in Figure 3 reproducing the experimental setup from Nakkiran et al. (2019). We vary the number of parameters deep neural
in the neural network by changing the width k. In this experimental setup, networks and reproduce the double descent with respect to the model size. Immediately, we see that the double descent curve is relatively mitigated in the case of the concatenated inputs construction
5

---

Under review as a conference paper at ICLR 2021
0.8
0.8
0.6
error
error
0.4
0.4
0.2
0.2
0.0
0
20
30
40
50
60
0
20
40
50
60
Width ResNet-18
Width of ResNet-18
Figure 4: Left: Standard one-hot vector setup, Right: Concatenated inputs construction. All models trained with batch size of 128 for 400 epochs with 15% label noise the CIFAR 100 dataset. Label noise is defined samples to random wrong labels and is applied prior training, and prior to stacking
by changing p% for two-hot vectors. Adam optimizer is used with learning rate 0.0001, 3r = 0.9, B2 0.999. Models are ResNet18 architecture where the width k is varied.
where from k = 5 to 20. Here, the curve is much smoother, although not entirely smooth. Notably, the concatenated inputs construction retains the test error across k, except the interpolation region where it achieves significantly lower test error (> 5%). We also note that the conclusion is identical when error is plotted against parameters, since the concatenated inputs construction typically adds very small number of parameters (< 5%).
We also present the plot for in the one-hot case, where the model follows the classical U shaped curve prior to the interpolation region, and then steadily decreases in loss after. The concatenated inputs construction exhibits more interesting behavior where the U curve is significantly muted and mitigated, and there is clear peak of the curve.
Figure 3 (Top row). Here, we observe a clear double descent
ResNet18 CIFAR100. We train ResNet18 architecture the CIFAR100 dataset in Figure 4. reproducing the experimental setup from Nakkiran et al. (2019) and almost identical setup in ResNet18 8 CIFAR10. The results similar and slightly clearer here, where the double de- scent curve in error entirely disappears with the concatenated inputs construction, and test error is decreased in the interpolation region. Again, the test error difference significant (> 5%))
4.3.2
EPOCH-WISE WISE DOUBLE DESCENT
0.8
0.8
2
0.7
0.6
64
64
0.5
.TTPr
0.4
0.4
0.3
0.2
0.2
10a
10a
102
10a
10a
10a
102
10a
epochs
epochs
Figure 5: Left: Standard hot vector setup, Right: Concatenated inputs construction. Models are ResNet18 architecture CIFAR-I0 where the emorr during training is plotted above.
ResNet18 CIFARIO. We use the same setting as in the previous section, except train models for am additional 600 epochs, for a total of 1000 epochs. Figure [ we plot test error against epochs. In the hot setting, expected observe U shape for medium sized models and

---

Under review as a conference paper at ICLR 2021
1.0
0.9
0.8
0.8
erro
0.7
0.7
0.6
0.6
0.5
13
13
0.5
0.4
64
10
10
10a
10a
100
10a
102
10a
epochs
epochs
Figure 6: Left: Standard one-hol vector setup, Right: Concatenated inputs construction. Models are ResNet18 architecture CIFAR-I00 100 where the error during training is plotted above.
double descent for larger models [Nkkirrnn et al. 2019). For the concatenated inputs construction, flat U shape and double descent is indeed observed, although they are significantly mitigated. The mitigation allows a 5% improvement at the end of training for medium sized models and 10% improvement the middle of training for large sized models.
ResNet18 CIFAR100. Similarly to above, we use the same setting as in the previous section, except we train models for a total 1000 epochs. Results are given Figure 66 The one hot results mirror ResNet18 CIFARIO 10, as expected (Nakkiran al. 2019). For the concatenated input construction, we observe complete mitigation of the double descent curve for medium and large sized models. For large models, this is more than 10% gap.
4.3.3 BIAS VARIANCE DECOMPOSITION
In this section, we follow Yang et al. (0200) and decompose the loss into bias and variance. Namely, let denote the cross entropy loss, T random variable representing the training set, the true one-hot label, स is the average log- rog-robbability after normalization, and á is the output of the neural network. Then,
Ey [CE #)] = Dxz(=|(#) + Er (DxI
where the first component is the bias and the second component is the variance. On high level, the variance can then be estimated by training separate same capacity models on different splits of the dataset, and then measuring the difference in outputs on the same set. The bias then computed by subtracting the empirical variance from the empirical risk. For finer details, see Yang et al. (020
For training, follow Yang et al. (2020) and train ResNet-34 (He al. 2016) the CIFARIO dataset, with stochastic gradient descent (learning rate = 0.1, momentum 0.9). The learning rate decayed factor of 0.1 every 200 epochs, and a weight decay of 5e-4 used. The width is of the
network is varied suitably between and 64. We also make 5 splits of 10,000 training samples for the calculation of bias and variance.
We present results in Figure 7 The concatenated inputs construction significantly delays and smoothens the increase in variance relative the standard case, where the unimodal variance is significantly sharper. This impacts the shape of the test error, where in this setting see a shifted bump in test error for the concatenated inputs construction. One possible explanation in the case of deep neural networks the concatenated inputs construction is form of implicit regularization for small models, which controls overfitting and leads to a smoother variance curve.

---

Under review as a conference paper at ICLR 2021
64G
144
6:18
crasated undd ingia constrction mite-115
011
Figure 7: Left: Bias and Variance against width k of ResNet-34 in the standard case. Middle; Bias and Variance against width of ResNet-34 34 in the concatenated inputs construction. Right: Test error against width k of ResNet-34 34 in the standard case and concatenated inputs construction.
5 DISCUSSION
We revisit the topic of overparameetrrization, underparameetrriaatoon and the double descent curve. The understanding of overparameterization, underparamettrizationn and the double descent curve strongly tied to the number of samples. appears that given a fixed number of unique samples it is possible to manipulate overparameteriizaioo and underparameterizztion by artificially boosting the number of samples by the concatenated inputs construction. This is based on the observation of removing or shifting the double descent curve through the concatenated inputs construction.
A similar topic on double descent is the possible explanation that the model is being forced to fit the training data perfectly as possible, and at some model capacity it is possible to fit the training data perfectly by overfitting on non-existent, or weakly present, features. This results in overfitting and the double descent curve. Interestingly, the concatenated inputs construction generally removes the double descent curve, even though it is possible to build models for the concatenated inputs construction from models for the standard setting. This suggests a possible route improve on the understanding of the relationship to the model capacity.
Furthermore, we consider the topic of underfitting. is well know that overparameterized neural networks don't ton't exhibit strong overfitting in practice, even though they can memorize the dataset (Zhang et al., 2016). The experimental results in this work regarding underparameterrizaton. over- parameterization and the double descent curve support that the hehavior of the neural network can change with respect to the number of samples, even if the majority of samples add limited infor- mation, via the concatenated inputs construction. In this view, the concatenated inputs construction creates possibly a huge dataset, for example 50, 0002 samples for the originally 50, 000 samples CIFAR10 dataset where 50, 000 2, 500, 000, 000 is far larger than any neural network for the CIFARIO dataset. Yet, there no noticeable underfitting. Namely, the concatenated inputs con- struction very quickly achieves comparable performance compared the standard one-hot vector deep learning setting. This suggests we may need to rethink the relationship between underfitting and the number of parameters, samples, and model capacity.
CONCLUSION
In this paper, we examine the double descent phenomena through the concatenated inputs construc- tion. The concatenated inputs construction is designed to add limited information whilst artificially augmenting the size of the dataset. As constructed, in the linear regression setting there is no impact on the location of the peak. Yet, in the neural network setting, the concatenated inputs construc- tion generally removes the double descent curve. Finally, we explore this phenomena through bias variance decomposition, and draw connections with samples, model capacity, and underfitting in discussion.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Madhu S Advani and Andrew M Saxe. High-dimensional dynamics of generalization error in neural networks. arXiv preprint arXiv: 2017.
Jimmy Ba, Murat Erdogdu. Taiji Suzuki, Denny Wu, and Tianzong Zhang. Generalization of two- layer neural networks: An asymptotic viewpoint. In ICLR, 2020.
Peter L Bartlett, Philip M Long, Gabor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. arXiv preprint arXiv: 300, 2019.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine learning practice and the bias-variance trade-off. arXiv preprint arXiv:1812.11112 2018a.
Mikhail Belkin, Siyuan Ma, and Soumik Mandal. To understand deep learning we need to under- stand kernel learning, 2018b.
Mikhail Belkin, Daniel Hsu, and Ji Xu. Two models of double descent for weak features, 2019a.
Mikhail Belkin, Daniel Hsu, and Ji Xu. Two models of double descent for weak features. arXiv preprint arXiv:1903.075:: 2019b.
Koby Bibas, Yaniv Fogel, and Meir Feder. A new look at old problem: A universal learning approach linear regression. arXiv preprint (eKiv:/9955044779 2019.
Emmanuel Caron and Stephane Chretien. A finite sample analysis of the double descent phe- nomenon for ridge function estimation, 2020.
Lin Chen, Yifei Min, Mikhail Belkin, and Amin Karbasi. Multiple descent: Design your own generalization curve, 2020.
Stéphane d' Ascoli, Levent Sagun, and Giulio Biroli. Triple descent and the two kinds of overfitting: Where why do they appear?, 2020.
Mario Geiger, Stefano Spigler, Stéphane d' Ascoli, Levent Sagun, Marco Baity-Jesi, Giulio Biroli, and Matthieu Wyart. Jamming transition paradigm understand the loss landscape of deep neural networks. Physical Review E, 100(1), Jul 2019. ISSN 2470-0053, doi: 10. 10.1100 1033pphy/sseee. 100.012115. URLpttp:://
dei org/ 10. 1103 100. 012115
Mario Geiger, Arthur Jacot, Stefano Spigler. Franck Gabriel, Levent Sagun, Stéphane d' Ascoli, Giulio Biroli, Clément Hongler, and Matthieu Wyart. Scaling description of generalization with number of parameters in deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2020(2)003301,, Feb 2020. ISSN 1742-5468.
doi: 10.
URL http:
7/dx org 10. 0. 088 42 - 5468/446688 ab633c
Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Linearized two-layers neural networks in high dimension, 2019.
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high dimen- sional ridgeless least squares interpolation. arXiv preprint
2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In CVPR, 2017.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Ultra-deep neural networks without residuals. arXiv preprint arXiv: 1605.0760057744 2016.
Tengyuan Liang, Alexander Rakhlin, and Xiyu Zhai. On the multiple descent of minimum-now norm interpolants and restricted lower isometry of kernels, 2019.

---

Under review as a conference paper at ICLR 2021
Song Mei and Andrea Montanari. The generalization enror of random features regression: Precise asymptotics and double descent curve. arXiv preprint 1rXiv:1988.053355 2019.
Partha P. Mitra. Understanding overfitting peaks in generalization error: Analytical risk curves for 12 and penalized interpolation. arXiv preprint arXiv: 03067, 2019.
Vidya Muthukumar, Kailas Vodrahalli, and Anant Sahai. Harmless interpolation of noisy data in regression. arXiv preprint arXiv:/903.09: 1903.09999, 2019.
Preetum Nakkiran. More data can hurt for linear regression: Sample-wise double descent, 2019.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. arXiv preprint arXiv: 19Xi::991,.0229, 2019.
Preetum Nakkiran, Prayaag Venkat, Sham Kakade, and Tengyu Ma. Optimal regularization can mitigate double descent, 2020.
Brady Neal, Sarthak Mittal, Aristide Baratin, Vinayak Tantia, Matthew Scicluna, Simon Lacoste- Julien, and Ioannis Mitliagkas. modern take on the bias-variance tradcoff neural networks, 2018.
Ruslan Salakhutdinov. https::/simoonsberkee
Deep learning tutorial at the simons institute, berkeley. 2017.
Elic Bienenstock Stuart Geman and Ren Doursat. Neural networks and the bias/variance dilemma. Neural Computation, 1992.
Robert Tibshirani Trevor Hastie and Jerome Friedman. The elements of statistical learning, volume 1. Springer, 2001.
Zitong Yang, Yaodong Yu, Chong You, Jacob Steinhardt, and Yi Ma. Rethinking bias-variance trade-off for generalization of neural networks. arXiv preprint arXiv:2002.11328, 2020.
Sergey Zagoruyko and Nikos Komodakis.
Wide residual networks.
arXiv preprint
17X.v:1605.07146. 2016.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization, 2016.
10


Reviews:
[{"review_id": "npnC80YwuTh", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "There has been a recent surge of research interest in explaining the double descent phenomenon, which contradicts the classical bias-variance tradeoff and may contribute to the success of DNNs and other overparametrized models. This paper proposes to mitigate double descent by artificially augmenting the dataset with concatenated inputs, and presents some empirical results for linear regression and neural networks.\n\nAlthough the paper is on a timely and interesting topic, the idea of using data augmentation to mitigate overfitting is certainly not new and has been understood in many areas of machine learning. In fact, augmenting the dataset with duplicates or artificially constructed data is equivalent to incorporating certain prior structure or imposing some form of regularization, which in turn should have an impact on the double descent curve. On the theoretical side, the paper does not provide any rigorous theoretical analysis; on the methodological side, the data augmentation strategy is defined in a restrictive manner and seems to be of little practical value.\n\nMore comments:\n1) Figures 2-4: It is difficult to decipher these figures without legends. I can’t find anywhere in the text or captions mentioning what the orange and blue curves mean. Also, how are the loss and error defined?\n\n2) The paper argues that the sample size, and hence overparametrization and underparametrization, in the double descent phenomenon is ambiguous and proposes to increase the sample size by artificially augmenting the dataset. In fact, the notion of sample size usually comes with the i.i.d. assumption, i.e., the samples are drawn independently from a common distribution. In this sense, augmenting the dataset with concatenated samples does not directly increase the sample size. Different ways of constructing the concatenated samples lead to different effective sample sizes, which should be properly defined in order to study their impact on the double descent curve.\n\n3) The paper aims to mitigate the double descent curve, which, however, is not necessarily harmful. A more critical question is how the lowest test error in the overparametrized regime compares to the optimal test error in the traditional underparametrized regime. If double descent helps to generalize well, why should we mitigate it?", "extra_scores": {}, "extra_text": {}}, {"review_id": "lzsT0KJj67", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper empirically studies the Double Descent phenomenon by using a cool construction that squares the dataset size by concatenation of every pair of inputs and linear interpolation of the labels. The main empirical finding is that this construction mitigates double descent. \n\n++++++\n\nMain Comments:\nThe concatenated input construction is a really cool experiment and I enjoyed reading about it. Nevertheless, the empirical findings are not strong enough to warrant publication in my opinion. While I believe that input/label concatenation/interpolation mitigates double descent, this statement by itself is not very interesting to the community. I believe that as the authors note, the concatenated inputs provide some sort of regularization (my intuition that something close to label smoothing/mixmatch is happening) and we already know that regularization mitigates double descent. Moreover, if we look at the paper from the perspective of EMC (Nakkiran 19') the results are not surprising as we not only change the number of samples but also the difficulty of the distribution (and add some parameters to the model). \n\nPros: \nThe concatenated input construction is an interesting and original experiment and using it to mitigate double descent is somewhat surprising. \n\nCons: \nWhile the experiment is original, as of now it is unclear what is causing the mitigation of double descent and what is the broad point the authors try to convey.  \n\nPoints for improvements:\nWhile mitigating double descent by the concatenated input construction is not a fundamental question by itself, the question that the authors sought to study 'what is a sample' is a very interesting one. I believe that going further down that road can make an interesting and fundamental investigation.\n\nArgument for my assessment: \nIn my opinion the paper with a better story (and further experimental/theoretical evidence that support it) could be publishable but it falls short at its current state. \n\nMinor Comments:\nI would put the MNIST experiments in the appendix. Double Descent is not about test loss which often behaves very differently than test error (cf. figure 2 in https://arxiv.org/abs/1710.10345) and having a convincing experiment on CIFAR is enough.  \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "f_luMyGHZHE", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**: In this article, the authors proposed a data augmentation procedure by concatenating the input data to produce an augmented dataset of size $O(n^2)$ from an original dataset of size $O(n)$, so as to mitigate the double descent curve. The authors showed experimentally that such construction does not impact the double descent curves in linear regression but can largely alleviate those in the case of (deep) neural nets. \n\n**Strong points**: The idea of (artificial) data augmentation looks interesting. Experiments are conducted on linear regression, single-hidden-layer ReLU network, as well as on ResNet18. \n\n**Weak points**: The organization and presentation of the article can be significantly improved, there are many places in the paper where things are not clearly explained and properly compared to the existing literature. Both the theoretical and empirical contributions of this work are somewhat limited. See **detailed comments** below.\n\n**Recommendation**: The idea in this paper looks interesting. However, there is visually no theoretical contribution and the empirical contribution is also somewhat limited. In general, this paper does not meet the standards to be published at ICLR.\n\n**Detailed comments**: \n\n* P3 Sec 3: just some comments on the proposed \"concatenated inputs construction\": (i) at the same time of augmenting the size of the dataset, this procedure also increases the **dimensionality** of the data (by a factor of 2 I think); and (ii) this procedure implicitly introduces, at least in the case of regression, some sort of invariance or implicit bias in the dataset. For instance, here the input $[x_1, x_2]$ and $[x_2, x_1]$ correspond to **exactly** the same target $(y_1 + y_2)/2$. And I think this imposes some structural constraints in the trained network which may help \"mitigate\" the double descent test curve?\n* Sec 4.1 and Figure 1: \"We also make the remark here that it is not surprising that this is the case, and it is not complicated to understand why from a theoretical perspective.\": it would be helpful to compare the empirical results here (in the left plot of Figure 1) to the theoretical prediction in for example (Bartlett et al. 2019) or (Hastie et al. 2019), and perhaps with different values of $\\sigma^2$: it is somehow less interesting to see an almost straight test MSE line and we clearly do not see the classical U-shaped curve here. Also, I am not sure that the theoretical understanding of the right plot of Figure 1 easily follows from existing results since, the concatenated data are no longer standard Gaussian, and I think they are even not linear transformation or simple Kronecker product of the original Gaussian data.\n* Is Figure 2 trying to reproduce Figure 4 in (Belkin et al. 2018a)? Note that the way of counting the number of parameters is slightly different there and I think this explains that under the same setting (training set size $n = 4 \\cdot 10^3$) the test loss peaks are observed at different numbers of parameters (around $10^4$ here and around **same** $4 \\cdot 10^3$ in (Belkin et al. 2018a)). And I believe the curves could be made more smooth and with confidence intervals.\n* Figure 2 top right plot \"the double descent curve is completely removed\": it would of interest to check whether there is a double-descent-type peak around $O(n^2)$, to help better understand the proposed construction.\n* Sec 4.2: it seems to me that the remark \"double descent in the loss versus number of parameters, but were unable to produce the double descent in the error\" was already mentioned in (Nakkiran et al. 2019).\n* Figure 3: even larger width $k$ is expected here for concatenated inputs since the training loss is still at a relatively high level. The same applies to Figure 4.\n* The legend, x, and y-axes are hardly visible in Figure 7.", "extra_scores": {}, "extra_text": {}}, {"review_id": "a0eGWMj8EOZ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper investigates the double descent phenomenon. It proposes the augmentation of the dataset via concatenating the covariate x and interpolating the label y, which increases the data size from n to n^2. The paper shows that the phenomenon of double descent can be mitigated via augmenting the input. The idea of investigating double descent from manipulating samples is novel and interesting.\n\nHowever, the paper is quite poorly-written. For example, I can't tell what the yellow and blue lines in Figure 2 and later figures represent, what is the difference between loss and error (which is never defined in the paper), and how the figures are generated from the concatenated data. The whole experimental setup is unclear to me. The authors are suggested to provide a concrete introduction and experimental details for it.\n\nFurthermore, the paper does not provide enough reasoning how simply concatenating the x and averaging y can mitigate the double descent phenomenon. First of all, are we supposed to train on the augmented data instead of the original data? Assume that we are running on completely synthetic data where y has a non-linear dependence with x (e.g. in the synthetic data y is the output of 3 layer neural network), then it seems that the pair ([x1,x2], (y1+y2)/2) will only hurt the performance since it adds extra complexity (one more linear transformation on the data) to really fit the model. It is also surprising why before and after concatenation, the error can be that much different with the same network width in Figure 4. The author mentioned at the end of Section 4 that one reason could be that the variance is smoothed due to implicit regularization from concatenation. Is it the case for all the experiments conducted above?\n\nThe authors are suggested to analyze the effect of concatenation on double descent thoroughly (e.g. analyze the bias and variance separately in the linear regression case to show why the double descent phenomenon is mitigated), provide convincing reasoning and more experiment details.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed an augmentation construction to mitigate the double descent. For any pairs of data points, the constructed input is simply concatenation of two inputs and the constructed label is the average of their corresponding labels. The authors further empirically show that this would mitigate double descent.

Reviewers unanimously like the main idea of the paper but they have other major concerns about this work. The main concern is that we already know double descent is not a practical issue since it can be mitigated by early stopping or proper regularization (Nakkiran 20'). Therefore, the main benefit from this paper could come from a better understanding of double descent using the observations from this construction. However, the paper does not provide us with insightful theoretical or empirical findings beyond the main observation. There are a couple of other concerns as well about discussions around #samples and the fact that the proposed construction is not i.i.d. and also lack of proper discussion about the relationship between the proposed construction and regularization techniques.

Unfortunately, authors did not responded to reviewers concerns. Nonetheless, I encourage authors to read reviewers' specific feedbacks, incorporate them and resubmit their work.

Given the above concerns, I recommend rejecting the paper.
Decision: Reject