Training Data Size Induced Double Descent For Denoising Neural Networks and the Role of Training Noise Level

1 INTRODUCTION . Denoising noisy training data is a widely used technique for pretraining networks to learn good representations of the data . Two extremely common examples of pretraining via denoising are Masked Language Modelling ( MLM ) ( Devlin et al. , 2019 ) and Stacked Denoising Autoencoders ( SDAE ) ( Vincent et al. , 2010 ) . For many modern problem , we work at large scales in terms of the number of parameters and the number of training samples . Recently there has been significant work in understanding the effect of scaling the number of parameters in the neural network . This resulted in the discovery of the much celebrated double descent phenomena ( Belkin et al. , 2019 ) . However , we do not have as good of an understanding of the effect of scaling the number of data points . Works such as Nakkiran et al . ( 2020 ) ; Nakkiran ( 2020 ) ; d ’ Ascoli et al . ( 2020 ) ; Adlam & Pennington ( 2020 ) show either empirically or via theoretical analysis that sample wise double descent exists . However , these were in the regime of supervised learning . On the other hand , our motivation comes from understanding denoising autoencoders . For MLM and SDAEs the denoising is a pretraining procedure , in which case the generalization error would depend on the downstream task . We shall instead look at the generalization error with respect to denoising test data . The difference between prior supervised learning set up and our denoising set up can be seen in Figure 1 . In an attempt to theoretically understand the denoising setting , we look at the simplest setting . Specifically we look at the case , when our network is a linear ( with respect to the inputs ) network and we are denoising data that lies on a line embedded in high dimensional space . In this setting , we derive the exact asymptotics for the generalization error . We see that in this case , the generalization error spikes at the interpolation threshold ( Figure 5a ) and the amount of noise that we want to add also spikes at the interpolation threshold ( Figure 5b ) . From the theoretical analysis , we see that the spike occurs due to the variance of the model increasing . Contributions . The main contributions of our works are as follows . 1 . We empirically show that when denoising data using a feedforward network , the curve for the generalization error versus the number of training data points as well the curve for the ratio of the test data SNR to the optimal training data SNR has double descent . Further changing the training data SNR can mitigate the double descent in the generalization error curve . 2 . Assuming we have mean 0 , rotational invariant noise , we derive an analytical formula for the expected mean squared generalization error for denoising rank 1 data by a linear network . Further , we use the same method to derive an approximation for higher rank data and experimentally verify the accuracy of the formula for general low rank data . 3 . Using our formula , we show that even in this simple model , we see that the double descent exists for the generalization error and for the amount of noise that should be added versus the number of training data points . Related work . Understanding deep neural networks is current active area of research with many exciting theoretical results . The discovery that fixed depth infinite width neural networks can be thought of as kernel regression ( Jacot et al. , 2018 ) and the discovery of double descent for neural networks ( Belkin et al. , 2019 ) has sparked significant research into understanding the generalization in the linear regime ( in parameters not inputs ) . The exact asymptotic for generalization loss were first understood for ridge regression ( Bartlett et al. , 2020 ; Hastie et al. , 2019 ; Belkin et al. , 2020 ; Advani & Saxe , 2020 ) . This was further generalized to understand the situation for the Random Features model and the Neural Tangent Kernel ( NTK ) model ( Mei & Montanari , 2019 ; Ghorbani et al. , 2019 ; Adlam & Pennington , 2020 ) . A partial list of recent work for supervised learning includes Jacot et al . ( 2020 ) ; Mel & Ganguli ( 2021 ) ; Derezinski et al . ( 2020 ) ; d ’ Ascoli et al . ( 2020 ) ; Dobriban & Wager ( 2015 ) ; Geiger et al . ( 2019 ) ; Lampinen & Ganguli ( 2019 ) ; Liang et al . ( 2020 ) ; Muthukumar et al . ( 2019 ) ; Loureiro et al . ( 2021 ) . However , there has been no work , to our knowledge , that looks at the problem either empirically or theoretically for the denoising set up . The idea of adding noise to improve generalization has been seen before . One popular strategy is to use Dropout ( Hinton et al. , 2012 ; Wan et al. , 2013 ; Srivastava et al. , 2014 ) , where we randomly zero out either neurons or connections . Another idea that is commonly used is data augmentation . In a revolutionary paper , Krizhevsky et al . ( 2012 ) showed that augmenting the dataset with noisy versions of the images , greatly improved the accuracy . Another area where noise is useful is adversarial learning . Dong et al . ( 2021 ) shows epoch wise double descent for adversarial training . In terms of recent theoretical work related to SDAEs , Pretorius et al . ( 2018 ) derived the learning dynamics of a linear autoencoder in the presence of noise . They also establish some relationships between the noise added and weight decay . However , they do not look at the generalization error or quantify the optimal amount of noise that should be added . Gnansambandam & Chan ( 2020 ) looked at the problem of what is the optimal amount of noise that should be added . However , they studied this from the perspective of minimizing the variance of the performance . 2 SET-UP . Let U ∈ RM×r be our feature matrix . For ease of notation , we assume that the columns of U have unit norm and are pairwise orthogonal . Then to generateN data points , we sample our latent variables V ∈ Rr×N and Σ ∈ Rr×r+ such that V has columns that have unit norm and are pairwise orthogonal and Σ is a diagonal matrix such that ‖Σ‖F = 1 . Then a data matrix X is given by X = UΣV T . For us , we have two matrices Xtrn and Xtst that correspond to the train and test data sets . Hence we have corresponding V Ttrn ∈ Rr×Ntrn , V Ttst ∈ Rr×Ntst , and Σtrn , Σtst . We make no other assumptions on U , Vtrn , ΣtrnVtst , Σtst except that they are given and fixed . Finally , let θtst , θtrn ∈ R+ be scalars that will scale the singular values of Xtrn , Xtst so that we can control the SNR . We also assume that θtst is fixed and that we have control over θtrn . Let c = M/Ntrn and let Atrn , Atst be noise matrices that are added to the training and the test data . Let W be the linear autoencoder that is the solution to the following problem minimizeŴ ‖θtrnXtrn − Ŵ ( θtrnXtrn +Atrn︸ ︷︷ ︸ Ytrn ) ‖2F . ( 1 ) Then , the expected mean squared error , is given by R ( θtrn , θtst , c , Σtrn , Σtst ) : = E [ ‖θtstXtst −W ( θtstXtst +Atst ) ‖2F Ntst ] . ( 2 ) 2.1 ASSUMPTIONS ABOUT THE NOISE . We assume that each entry of the noise matrix A has mean 0 , variance 1/M and that the entries of A are pairwise uncorrelated . Additionally , we shall assume that A is rotationally bi-invariant . That is , if Q is an M by M ( N by N ) orthogonal matrix , then QA ( AQ ) has the same distribution as A . Another way to phrase this is if A = UAΣAV TA is the SVD , then UA and VA are uniformly random orthogonal matrices and are independent from ΣA and each other . Finally , we shall assume that A has full rank with probability 1 and that the limiting distribution of the eigenvalues of ATA converge to the Marchenko-Pastur distribution . While such assumptions on the noise may seem restrictive . This encompasses a large family of noise distributions . Proposition 1 . If B is a random matrix that has full rank with probability 1 and its entries are independent , have mean 0 , have variance 1/M , and bounded fourth moment , and P , Q are uniformly random orthogonal matrices . Then A = PBQ satisfies all of our noise assumptions . 2.2 SIGNAL TO NOISE RATIO ( SNR ) . A quantity of interest to us will be the SNR , given by ‖X‖F /‖A‖F . Hence , we need to normalize everything by ‖A‖F . In this case , due to our assumptions , we have that E [ ‖A‖2F ] = N . Hence , for any variables and constants , if it has a hat , then that refers to that variable or constant normalized by√ N . For example , given θtrn , Xtrn , and Atrn , then we have that ‖θtrnXtrn‖F /‖Atrn‖F = θtrn/‖Atrn‖F ≈ θtrn/ √ Ntrn = : θ̂trn . 3 EMPIRICAL DOUBLE DESCENT . We first show that sample wise double descent occurs for denoising neural networks empirically . Figure 2 shows that if we train a feedforward network to denoise data such that the training data signal to noise ratio ( SNR ) θ̂trn is the same SNR as that of the test data set ( θ̂tst ) , then the curve for the denoising generalization error vs the number of training samples has the shape of a double descent curve . Thus , together with prior work , this suggests that the double descent with respect to the number of data points is a universal phenomena . However , unlike other hyperparameters , such as number of features and number of training epochs , we can not arbitrarily change the number of data points as we are limited by the data set that we have . Hence it could be the case , that the maximum number of data points that we have corresponds to the peak of the generalization error curve . However , we can look at the amount of noise that we add to the training data . To see the effect of the noise , for a variety of different θ̂trn/θ̂tst , we compute the denoising generalization error versus the number of data points curve . We do this for MNIST and CIFAR dataset . We create test data sets by taking the test data for each and then adding Gaussian noise . We fix the test SNR to be 1 for both datasets . Hence we know the test data SNR . We then take various different fractions of the training data and train a 3 layer ReLU neural network ( without bias ) for various levels of training noise . For each of pair of parameters ( number of training data points and the level of training noise ) , we compute the generalization error averaged over 20 trials for MNIST and 5 trials for CIFAR . Here the test noise and training noise is resampled for each trial . The plots for the generalization error can be seen in Figures 3a ( MNIST ) and 3b ( CIFAR10 ) . The first thing we notice is that for most ratios for the test SNR to training SNR we see sample wise double descent . Further , we see that the optimal denoising error does not occur when the train SNR is equal to the test SNR . This is very surprising as it contradicts standard thought that training data distribution should be the same as the test data distribution . Interestingly , as seen in Figures 4 and 4b , we see that the optimal ratio depends on the number of data points and the shape of the curve for the values θ̂trn/θ̂tst that results in the best generalization error versus the number data points also has the shape of a double descent curve .

Summary: In this paper, the authors study overparameterization of autoencoders in terms of varying sample size. They derive an analytic expression for the bias-variance decomposition of the loss for a bottleneck of size 1. Furthermore, they approximate the bias-variance decomposition for larger bottleneck sizes. This is done with all linear networks. They experimentally show double descent in nonlinear networks as well.