Provable More Data Hurt in High Dimensional Least Squares Estimator

1 INTRODUCTION . More data hurt refers to the phenomenon that training on more data can hurt the prediction performance of the learned model , especially for some deep learning tasks . Loog et al . ( 2019 ) shows that various standard learners can lead to sample-wise non-monotonicity . Nakkiran et al . ( 2019 ) experimentally confirms the sample-wise non-monotonicity of the test accuracy on deep neural networks . This challenges the conventional understanding in large sample properties : if an estimator is consistent , more data makes the estimator more stable and improves its finite-sample performance . Nakkiran ( 2019 ) considers adding one single data point to a linear regression task and analyzes its marginal effect to the test risk . Dereziński et al . ( 2019 ) gives an exact non-asymptotic risk of the high-dimensional least squares estimator and observes the sample-wise non-monotonicity on mean square error . For adversarially robust models , Min et al . ( 2020 ) proves that more data may increase the gap between the generalization error of adversarially-trained models and standard models . Chen et al . ( 2020 ) shows that more training data causes the generalization error to increase in the strong adversary regime . In this work , we derive the finite-sample distribution of the prediction risk under linear models and prove the “ more data hurt ” phenomenon from an asymptotic point of view . Intuitively , the “ more data hurt ” stems from the “ double descent ” risk curve : as the model complexity increases , the prediction risk of the learned model first decreases and then increases , and then decreases again . The double descent phenomenon can be precisely quantified for certain simple models ( Hastie et al . ( 2019 ) ; Mei & Montanari ( 2019 ) ; Ba et al . ( 2019 ) ; Belkin et al . ( 2019 ) ; Bartlett et al . ( 2020 ) ; Xing et al . ( 2019 ) ) . Among these works , Hastie et al . ( 2019 ) and Mei & Montanari ( 2019 ) use the tools from random matrix theory and explicitly prove the double descent curve of the asymptotic risk of linear regression and random features regression in high dimensional setup . Ba et al . ( 2019 ) gives the asymptotic risk of two-layer neural networks when either the first or the second layer is trained using a gradient flow . The second decline of the prediction risk in the double descent curve is highly related to the more data hurt phenomenon . In the over-parameterized regime when the model complexity is fixed while the sample size increases , the degree of over-parameterization decreases and becomes close to the interpolation boundary ( for example p/n = 1 in Hastie et al . ( 2019 ) ) , in which a high prediction risk is achieved . However , the existing asymptotic results , which focus on the first-order limit of the prediction risk , can not fully describe the more data hurt phenomenon . In fact , the “ double descent ” curve is a function of the limiting ratio lim p/n , which may not be able to characterize the empirical prediction risk in finite sample situations . There will be a non-negligible discrepancy between the empirical prediction risk and its limit , especially when the sample size or dimension is small . Finegrained second-order results are thus needed to fully characterize such discrepancy and further , a confidence band for the prediction risk can be constructed to evaluate its finite sample performance . We take Figure 1 as an example to illustrate this . According to the first-order limit , given a fixed dimension p = 100 , the prediction risks at sample size n = 90 and n = 98 are about 10.20 and 49.02 . More data hurt seems true . However , the 95 % confidence interval of the prediction risks with sample size 98 is [ 4.91 , 142.12 ] , which contains the risk for n = 90 . Then more data hurt is not statistically significant . Hence , in this work , we characterize the second-order fluctuations of the prediction risk and make attempts to fill this gap . We employ the linear regression task in Hastie et al . ( 2019 ) and Nakkiran ( 2019 ) , and introduce new tools from the random matrix theory , e.g . the central limit theorems for linear spectral statistics in Bai & Silverstein ( 2004 ) ; Bai et al . ( 2007 ) , to derive the central limit theorem of the prediction risk . Consider a linear regression task with n data points and p features , the setup of the more data hurt is similar to that in the classical asymptotic analysis in Van der Vaart ( 2000 ) . According to the classical asymptotic analysis with p fixed and n → ∞ , the least square estimator is unbiased and√ n-consistent to the ground truth . This implies that the more data will not hurt and even improve the prediction performance . However , the story is very different in the over-parameterized regime . The prediction risk doesn ’ t decrease monotonously with n when p > n. More data does hurt in the over-parameterized case . In the following , we will justify this phenomenon by developing the second-order asymptotic results as both n and p tend to infinity . We assume p/n → c , and denote 0 < n1 < n2 < +∞ , c1 = p/n1 and c2 = p/n2 . Then the direct comparison of the prediction risk between sample sizes n1 and n2 can be decomposed into three parts : ( i ) the gap between the finitesample risk under n = n1 and the asymptotic risk with c = c1 ; ( ii ) the gap between the finite-sample risk under n = n2 and the asymptotic risk with c = c2 ; ( iii ) the comparison between two asymptotic risks under c = c1 and c = c2 . Theorem 1 and 2 of Hastie et al . ( 2019 ) give answers to the task ( iii ) . For ( i ) and ( ii ) , we develop in this paper the convergence rate and the limiting distribution of the prediction risk as n , p → +∞ , p/n → c. Furthermore , the confidence interval of the finite-sample risk can be obtained as well . The main goal of this paper is to study the second order asymptotic behavior of two different types of conditional prediction risk in the linear regression model . One is RX , β ( β̂ , β ) given both the training data and regression coefficient while the other is RX ( β̂ , β ) given the training data only . We summarize our main results as follows : ( 1 ) The regression coefficient is set to be either random or nonrandom to cover more cases . Different convergence rates and limiting distributions of both prediction risk are derived under various scenarios . ( 2 ) In particular , the finite-sample distribution of the conditional prediction risk given both the training data and regression coefficient is derived and the sample-wise double descent is characterized in Theorem 4.2 and Theorem 4.5 ( see Figure 1 ) . Under certain assumptions , the more data hurt phenomenon can be confirmed by comparing the confidence intervals built via the central limit theorems . ( 3 ) Our results incorporate non-Gaussian observations . For Gaussian data , the limiting mean and variance in the central limit theorems have simpler forms , see Section 4.2 and 4.3 for more details . The rest of this paper is organized as follows . Section 3 introduces the model settings and two different prediction risk . Section 4 presents the main results on CLTs for the two types of risk with discussion . Section 5 conducts simulation experiments to verify the main results . All the technical proofs and lemmas are relegated to the appendix in the supplementary file . 2 RELATED WORK . Double Descent The double descent curve describes how generalization ability changes as model capacity increases . It subsumes the classical bias-variance trade-off , a U-shape curve , and further show that the test error exhibits a second drop when the model capacity exceeds the interpolation threshold ( Belkin et al . ( 2018 ) ; Geiger et al . ( 2019 ) ; Spigler et al . ( 2019 ) ; Advani & Saxe ( 2017 ) ) . The double descent phenomenon has been quantified for certain models , including two-layer neural networks via non-asymptotic bounds or asymptotic risk ( Belkin et al . ( 2019 ) ; Muthukumar et al . ( 2020 ) ; Hastie et al . ( 2019 ) ; Mei & Montanari ( 2019 ) ; Ba et al . ( 2019 ) ) . As our results are based on the linear regression , we mainly focus on the literature of linear models . Muthukumar et al . ( 2020 ) and Bartlett et al . ( 2020 ) derive the generalization bounds for over-parametrized linear models and show the benefits of the interpolation . Hastie et al . ( 2019 ) gives the first-order limit of the generalization error for linear regressions as n , p → +∞ . Dereziński et al . ( 2019 ) provides an exact non-asymptotic expression for double descent of the high-dimensional least square estimator . Wu & Xu ( 2020 ) extends the first-order limit of the prediction error of the generalized weighted ridge estimator to more general case with anisotropic features and signals . Montanari et al . ( 2019 ) , Deng et al . ( 2019 ) and Kini & Thrampoulidis ( 2020 ) investigate the sharp asymptotic of binary classification tasks with the max-margin solution and the maximum likelihood solution . Emami et al . ( 2020 ) and Gerbelot et al . ( 2020a ) consider the double descent in generalized linear models . Furthermore , the double descent phenomenon is also observed on linear tasks with various problems and assumptions , e.g . LeJeune et al . ( 2020 ) ; Gerbelot et al . ( 2020b ) ; Javanmard et al . ( 2020 ) ; Dar & Baraniuk ( 2020 ) ; Xu & Hsu ( 2019 ) ; Dar et al . ( 2020 ) . Xing et al . ( 2019 ) sharply quantifies the benefit of interpolation in the nearest neighbors algorithm . Mei & Montanari ( 2019 ) derives the limiting risk on the random features model and shows that minimum generalization error is achieved by highly overparametrized interpolators . Ba et al . ( 2019 ) gives the limiting risk of the regression problem under two-layer neural networks . However , the existing asymptotic results focus on the first-order limit of the prediction risk and do not indicate the convergence rate . There are very few second-order results in the literature , Shen & Bellec ( 2020 ) establishes the asymptotic normality for the derivatives of 2-layers neural network , but not the exact limiting distribution of the risk . In this work , we are the first to develop results on second-order fluctuations of the prediction risk in linear regressions and provide its corresponding confidence intervals . The more data hurt phenomenon is further justified from the asymptotic point of view . Random Matrix Theory The primary tool for analyzing the second-order fluctuations of prediction risk comes from random matrix theory . In particular , Bai & Silverstein ( 2004 ) refines the central limit theorem for linear spectral statistics of large dimensional sample covariance matrix with general population and the population is not necessary to be Gaussian . Similar central limit theorems are also developed for other random matrix ensembles , see Sinai & Soshnikov ( 1998 ) ; Bai & Yao ( 2005 ) ; Zheng ( 2012 ) . Other than the central limit theorem for linear spectral statistics , Bai et al . ( 2007 ) and Pan & Zhou ( 2008 ) study the asymptotic fluctuation of eigenvectors of sample covariance matrices . Bai & Yao ( 2008 ) considers the fluctuation of quadratic forms . All these technical tools and results are adopted and fully utilized in this paper , especially those related to Stieltjes transform , which are closely connected to the prediction risk studied in this paper .

Summary: This paper investigates the phenomenon of double descent, also referred to as "more data hurts", in high dimensional linear regression using the least square estimator.  In the same setup, previous sharp results were already established in the asymptotic regime. Non-asymptotic results are also known but are less precise. The authors of this paper try to provide a new type of results that fill the gap between the two regimes (asymptotic vs non-asymptotic). To do so they have managed to derive second order (CLT type) asymptotic results for different risks based on more refined random matrix theory results.