Mitigating Deep Double Descent by Concatenating Inputs

1 INTRODUCTION . Underparameterization and overparameterization are at the heart of understanding modern neural networks . The traditional notion of underparameterization and overparameterization led to the classic U-shaped generalization error curve ( Trevor Hastie & Friedman , 2001 ; Stuart Geman & Doursat , 1992 ) , where generalization would worsen when the model had either too few ( underparameterized ) or too many parameters ( overparameterized ) . Correspondingly , it was expected that an underparameterized model would underfit and fail to identify more complex and informative patterns , and an overparameterized model would overfit and identify non-informative patterns . This view no longer holds for modern neural networks . It is widely accepted that neural networks are vastly overparameterized , yet generalize well . There is strong evidence that increasing the number of parameters leads to better generalization ( Zagoruyko & Komodakis , 2016 ; Huang et al. , 2017 ; Larsson et al. , 2016 ) , and models are often trained to achieve zero training loss ( Salakhutdinov , 2017 ) , while still improving in generalization error , whereas the traditional view would suggest overfitting . To bridge the gap , Belkin et al . ( 2018a ) proposed the double descent curve , where the underparameterized region follows the U-shaped curve , and the overparameterized region smoothly decreases in generalization error , as the number of parameters increases further . This results in a peak in generalization error , where a fewer number of samples would counter-intuitively decrease the error . There has been extensive experimental evidence of the double descent curve in deep learning ( Nakkiran et al. , 2019 ; Yang et al. , 2020 ) , as well as in models such as random forests , and one layer neural networks ( Belkin et al. , 2018a ; Ba et al. , 2020 ) . One recurring theme in the definition of overparameterization and underparameterization lies in the number of neural network parameters relative to the number of samples ( Belkin et al. , 2018a ; Nakkiran et al. , 2019 ; Ba et al. , 2020 ; Bibas et al. , 2019 ; Muthukumar et al. , 2019 ; Hastie et al. , 2019 ) . On a high level , a greater number of parameters than samples is generally considered overparameterization , and fewer is considered underparameterization . However , this leads to the question “ What is a sample ? ” In this paper , we revisit the fundamental underpinnings of overparameterization and underparameterization , and stress test when it means to be overparameterized or underparameterized , through extensive experiments of a cleverly constructed input . We artificially augment existing datasets by simply stacking every combination of inputs , and show the mitigation of the double descent curve in the deep neural network setting . We humbly hypothesize that in deep neural networks we can , perhaps , artificially increase the number of samples without increasing the information contained in the dataset , and by implicitly changing the classification pipeline mitigate the double descent curve . In particular , the narrative of our paper obeys the following : • We propose a simple construction to artificially augment existing datasets of sizeO ( n ) by stacking inputs to produce a dataset of size O ( n2 ) . • We demonstrate that the construction has no impact on the double descent curve in the linear regression case . • We show experimentally that those results on double descent curve do not extend to the case of neural networks . Concretely , we reproduce results from recent landmark papers , and present the difference in behavior with respect to the double descent curve . 2 RELATED WORKS . The double descent curve was proposed recently in ( Belkin et al. , 2018a ) , where the authors define overparameterization and underparameterization as the proportion of parameters to samples . The authors explain the phenomenon through the model capacity class . With more parameters in the overparameterized region , there is larger “ capacity ” ( i.e. , the model class contains more candidates ) , and thus may contain better , simpler models by Occam ’ s Razor rule . The interpolation region is suggested to exist when the model capacity is capable of fitting the data nearly perfectly by overfitting on non-informative features , resulting in higher test error . Experiments included a one layer neural network , random forests , and others . The double descent curve is also observed in deep neural networks ( Nakkiran et al. , 2019 ) , with the additional observation of epoch-wise double descent . Experimentation is amplified by label noise . With the observation of unimodel variance ( Neal et al. , 2018 ) , Yang et al . ( 2020 ) also decomposes the risk into bias and variance , and posits that the double descent curve arises due to the bell-shaped variance curve rising faster than the bias decreases . There is substantial theoretical work on double descent , particularly in the least squares regression setting . Advani & Saxe ( 2017 ) analyses this linear setting and proves the existence of the interpolation region , where the number of parameters equals the number of samples in the asymptotic limit where samples and parameters tend to infinity . Hastie et al . ( 2019 ) follows a similar line of work , and proves that regularization reduces the peak in the interpolation region . Belkin et al . ( 2019b ) requires only finite samples , where the features and target be jointly Gaussian . Other papers with similar setup include ( Bartlett et al. , 2019 ; Muthukumar et al. , 2019 ; Bibas et al. , 2019 ; Mitra , 2019 ; Mei & Montanari , 2019 ) . Ba et al . ( 2020 ) analyses the least squares regression setting for two layer linear neural networks in the asymptotic setting , where the double descent curve is present when only the second layer is optimized . There is also work in proving that optimally tuned ` 2-norm regularization mitigates the double descent curve for certain linear regression models with isotropic data distribution ( Nakkiran , 2019 ) . This setting has also been studied with respect to the variance in the parameter space ( Bartlett et al. , 2019 ) . Multiple descent has also been studied , and in particular there is work to show in the linear regression setting that multiple descent curves can be directly designed by the user ( Chen et al. , 2020 ) . Additionally , there is supporting evidence of double descent in the sample-wise perspective ( Nakkiran et al. , 2020 ) . There is other work in this area , including studying the double descent curve for least squares in random feature models ( Belkin et al. , 2019a ; d ’ Ascoli et al. , 2020 ; Ghorbani et al. , 2019 ) , leveraging the Neural Tangent Kernel to argue for certain number of parameters the output of the neural network diverges ( Geiger et al. , 2020 ) , characterizing the double descent in non-linear settings ( Caron & Chretien , 2020 ) , kernel learning ( Belkin et al. , 2018b ; Liang et al. , 2019 ) , and connecting to other fields ( Geiger et al. , 2019 ) . Lastly , we note here that , in the deep neural network setting , models can be trained to zero training loss even with random labels ( Zhang et al. , 2016 ) . 3 THE CONCATENATED INPUTS CONSTRUCTION . We introduce the concatenated inputs construction , on which our main hypothesis is based on . The concatenated inputs construction refers to the general idea of concatenating pairs of inputs and element-wise adding and averaging pairs of outputs to produce new inputs and targets . This way the size of a dataset can be artificially –but non-trivially– increased . This construction can be applied both to the regression setting and the classification setting . In the setting of linear regression , for given input pairs , ( x1 , y1 ) , ( x2 , y2 ) , an augmented dataset can be constructed as : { ( [ x1 , x1 ] , y1+y1 2 ) , ( [ x1 , x2 ] , y1+y2 2 ) , ( [ x2 , x1 ] , y2+y1 2 ) , ( [ x2 , x2 ] , y2+y2 2 ) } , where [ α , β ] represents concatenation of the input α , β . In the setting of classification , the process is identical , where the targets are produced by element-wise addition and then averaged to sum to 1 . The averaging is not strictly necessary even in the deep neural network classification case , where the binary cross entropy loss can be used instead of cross entropy . For test data , we concatenate the same input with itself , and the target is the original target . This way a dataset of size O ( n ) is artificially augmented to size O ( n2 ) . Concretely , our reasons for the concatenated inputs construction are as follows : i ) there is limited injection of information or semantic meaning ; ii ) the number of samples is significantly increased . For the purposes of understanding underparameterization , overparameterization and the double descent curve , such a construction tries to isolate the definition of a sample . We revisit and assess these implications in the context of extensive experiments in the following sections . 4 RESULTS . In this section , we reproduce settings from benchmark double descent papers , add the concatenated inputs construction and analyze the findings . In particular , we begin with the linear regression , move to one hidden layer feedforward neural networks , and then deep neural networks , in both the model parameter-wise double descent and epoch-wise double descent . Finally , we analyze the performance of deep neural networks for the concatenated inputs construction , and the behavior of the double descent curve in the classification setting , when we transfer from the cross entropy to the binary cross entropy loss . 4.1 LINEAR REGRESSION . The linear regression setting has been a fruitful testbed for empirical work in double descent , as well as yielding substantial theoretical understanding . The concatenated inputs construction is applied similarly here , however with different motivation . Namely , we wish to motivate that the concatenated inputs construction is not expected to add any information and is therefore not expected to impact the double descent curve . We reproduce the linear regression setting from Nakkiran ( 2019 ) , given in Figure 1 . For the concatenated inputs construction , we first draw the number of samples before concatenation and construction of the augmented dataset . We observe that , by construction , the concatenated inputs construction does not affect the double descent curve , and the peak occurs in the exact same location . We also make the remark here that it is not surprising that this is the case , and it is not complicated to understand why from a theoretical perspective . 4.2 ONE HIDDEN LAYER FEEDFORWARD NEURAL NETWORK . Following linear regression , we move to neural networks . We train a feedforward neural network with one hidden layer and ReLU activations on a subset of the MNIST dataset , reproducing the experimental setup from Belkin et al . ( 2018a ) . We vary the number of parameters in the neural network by changing the size of the hidden layer . We use the Cross Entropy loss instead of the original MSE loss due to the prevalence of those losses for image classification tasks . This is shown in Figure 2 . We observe the double descent in the loss versus number of parameters , but were unable to produce the double descent in the error . In the rightmost plot , the double descent curve is completely removed in the concatenated inputs construction relative to the other two settings . Namely , a smooth decrease in loss is observed , where there is a clear double descent in the other cases . Furthermore , we provide the extra setting of concatenating each input only by itself , and the double descent curve is present almost exactly in this scenario . This provides further evidence that the disappearance of the double descent is not due to the extra parameters which originate from the larger sized inputs . In this setting , it appears that the behavior of underparameterization and overparameterization can be altered by simply artificially increasing the number of samples through concatenating images . In addition , the model trained on MNIST and one-hot vectors can be concatenated with itself , with all other parameters being zero , to produce a model with two times the number of hidden units which can be applied to the concatenated inputs construction . We consider this setting in the context of a possible explanation of the interpolation region , where the number of parameters nears that of samples . Concretely , it is possible for a neural network with double the hidden units in the concatenated inputs construction to recover the double descent curve by learning two smaller , disconnected networks , where each of the smaller networks are the ones learned in the double descent peak of the standard , one-hot case . However , in practice while the network can do so , it does not appear to , which leads to the smooth descent in the rightmost plot in Figure 2 .

Summary: The paper investigates the double descent phenomenon. It proposes the augmentation of the dataset via concatenating the covariate x and interpolating the label y, which increases the data size from n to n^2. The paper shows that the phenomenon of double descent can be mitigated via augmenting the input. The idea of investigating double descent from manipulating samples is novel and interesting.