Provable More Data Hurt in High Dimensional Least Squares Estimator

1 INTRODUCTION . More data hurt refers to the phenomenon that training on more data can hurt the prediction performance of the learned model , especially for some deep learning tasks . Loog et al . ( 2019 ) shows that various standard learners can lead to sample-wise non-monotonicity . Nakkiran et al . ( 2019 ) experimentally confirms the sample-wise non-monotonicity of the test accuracy on deep neural networks . This challenges the conventional understanding in large sample properties : if an estimator is consistent , more data makes the estimator more stable and improves its finite-sample performance . Nakkiran ( 2019 ) considers adding one single data point to a linear regression task and analyzes its marginal effect to the test risk . Dereziński et al . ( 2019 ) gives an exact non-asymptotic risk of the high-dimensional least squares estimator and observes the sample-wise non-monotonicity on mean square error . For adversarially robust models , Min et al . ( 2020 ) proves that more data may increase the gap between the generalization error of adversarially-trained models and standard models . Chen et al . ( 2020 ) shows that more training data causes the generalization error to increase in the strong adversary regime . In this work , we derive the finite-sample distribution of the prediction risk under linear models and prove the “ more data hurt ” phenomenon from an asymptotic point of view . Intuitively , the “ more data hurt ” stems from the “ double descent ” risk curve : as the model complexity increases , the prediction risk of the learned model first decreases and then increases , and then decreases again . The double descent phenomenon can be precisely quantified for certain simple models ( Hastie et al . ( 2019 ) ; Mei & Montanari ( 2019 ) ; Ba et al . ( 2019 ) ; Belkin et al . ( 2019 ) ; Bartlett et al . ( 2020 ) ; Xing et al . ( 2019 ) ) . Among these works , Hastie et al . ( 2019 ) and Mei & Montanari ( 2019 ) use the tools from random matrix theory and explicitly prove the double descent curve of the asymptotic risk of linear regression and random features regression in high dimensional setup . Ba et al . ( 2019 ) gives the asymptotic risk of two-layer neural networks when either the first or the second layer is trained using a gradient flow . The second decline of the prediction risk in the double descent curve is highly related to the more data hurt phenomenon . In the over-parameterized regime when the model complexity is fixed while the sample size increases , the degree of over-parameterization decreases and becomes close to the interpolation boundary ( for example p/n = 1 in Hastie et al . ( 2019 ) ) , in which a high prediction risk is achieved . However , the existing asymptotic results , which focus on the first-order limit of the prediction risk , can not fully describe the more data hurt phenomenon . In fact , the “ double descent ” curve is a function of the limiting ratio lim p/n , which may not be able to characterize the empirical prediction risk in finite sample situations . There will be a non-negligible discrepancy between the empirical prediction risk and its limit , especially when the sample size or dimension is small . Finegrained second-order results are thus needed to fully characterize such discrepancy and further , a confidence band for the prediction risk can be constructed to evaluate its finite sample performance . We take Figure 1 as an example to illustrate this . According to the first-order limit , given a fixed dimension p = 100 , the prediction risks at sample size n = 90 and n = 98 are about 10.20 and 49.02 . More data hurt seems true . However , the 95 % confidence interval of the prediction risks with sample size 98 is [ 4.91 , 142.12 ] , which contains the risk for n = 90 . Then more data hurt is not statistically significant . Hence , in this work , we characterize the second-order fluctuations of the prediction risk and make attempts to fill this gap . We employ the linear regression task in Hastie et al . ( 2019 ) and Nakkiran ( 2019 ) , and introduce new tools from the random matrix theory , e.g . the central limit theorems for linear spectral statistics in Bai & Silverstein ( 2004 ) ; Bai et al . ( 2007 ) , to derive the central limit theorem of the prediction risk . Consider a linear regression task with n data points and p features , the setup of the more data hurt is similar to that in the classical asymptotic analysis in Van der Vaart ( 2000 ) . According to the classical asymptotic analysis with p fixed and n → ∞ , the least square estimator is unbiased and√ n-consistent to the ground truth . This implies that the more data will not hurt and even improve the prediction performance . However , the story is very different in the over-parameterized regime . The prediction risk doesn ’ t decrease monotonously with n when p > n. More data does hurt in the over-parameterized case . In the following , we will justify this phenomenon by developing the second-order asymptotic results as both n and p tend to infinity . We assume p/n → c , and denote 0 < n1 < n2 < +∞ , c1 = p/n1 and c2 = p/n2 . Then the direct comparison of the prediction risk between sample sizes n1 and n2 can be decomposed into three parts : ( i ) the gap between the finitesample risk under n = n1 and the asymptotic risk with c = c1 ; ( ii ) the gap between the finite-sample risk under n = n2 and the asymptotic risk with c = c2 ; ( iii ) the comparison between two asymptotic risks under c = c1 and c = c2 . Theorem 1 and 2 of Hastie et al . ( 2019 ) give answers to the task ( iii ) . For ( i ) and ( ii ) , we develop in this paper the convergence rate and the limiting distribution of the prediction risk as n , p → +∞ , p/n → c. Furthermore , the confidence interval of the finite-sample risk can be obtained as well . The main goal of this paper is to study the second order asymptotic behavior of two different types of conditional prediction risk in the linear regression model . One is RX , β ( β̂ , β ) given both the training data and regression coefficient while the other is RX ( β̂ , β ) given the training data only . We summarize our main results as follows : ( 1 ) The regression coefficient is set to be either random or nonrandom to cover more cases . Different convergence rates and limiting distributions of both prediction risk are derived under various scenarios . ( 2 ) In particular , the finite-sample distribution of the conditional prediction risk given both the training data and regression coefficient is derived and the sample-wise double descent is characterized in Theorem 4.2 and Theorem 4.5 ( see Figure 1 ) . Under certain assumptions , the more data hurt phenomenon can be confirmed by comparing the confidence intervals built via the central limit theorems . ( 3 ) Our results incorporate non-Gaussian observations . For Gaussian data , the limiting mean and variance in the central limit theorems have simpler forms , see Section 4.2 and 4.3 for more details . The rest of this paper is organized as follows . Section 3 introduces the model settings and two different prediction risk . Section 4 presents the main results on CLTs for the two types of risk with discussion . Section 5 conducts simulation experiments to verify the main results . All the technical proofs and lemmas are relegated to the appendix in the supplementary file . 2 RELATED WORK . Double Descent The double descent curve describes how generalization ability changes as model capacity increases . It subsumes the classical bias-variance trade-off , a U-shape curve , and further show that the test error exhibits a second drop when the model capacity exceeds the interpolation threshold ( Belkin et al . ( 2018 ) ; Geiger et al . ( 2019 ) ; Spigler et al . ( 2019 ) ; Advani & Saxe ( 2017 ) ) . The double descent phenomenon has been quantified for certain models , including two-layer neural networks via non-asymptotic bounds or asymptotic risk ( Belkin et al . ( 2019 ) ; Muthukumar et al . ( 2020 ) ; Hastie et al . ( 2019 ) ; Mei & Montanari ( 2019 ) ; Ba et al . ( 2019 ) ) . As our results are based on the linear regression , we mainly focus on the literature of linear models . Muthukumar et al . ( 2020 ) and Bartlett et al . ( 2020 ) derive the generalization bounds for over-parametrized linear models and show the benefits of the interpolation . Hastie et al . ( 2019 ) gives the first-order limit of the generalization error for linear regressions as n , p → +∞ . Dereziński et al . ( 2019 ) provides an exact non-asymptotic expression for double descent of the high-dimensional least square estimator . Wu & Xu ( 2020 ) extends the first-order limit of the prediction error of the generalized weighted ridge estimator to more general case with anisotropic features and signals . Montanari et al . ( 2019 ) , Deng et al . ( 2019 ) and Kini & Thrampoulidis ( 2020 ) investigate the sharp asymptotic of binary classification tasks with the max-margin solution and the maximum likelihood solution . Emami et al . ( 2020 ) and Gerbelot et al . ( 2020a ) consider the double descent in generalized linear models . Furthermore , the double descent phenomenon is also observed on linear tasks with various problems and assumptions , e.g . LeJeune et al . ( 2020 ) ; Gerbelot et al . ( 2020b ) ; Javanmard et al . ( 2020 ) ; Dar & Baraniuk ( 2020 ) ; Xu & Hsu ( 2019 ) ; Dar et al . ( 2020 ) . Xing et al . ( 2019 ) sharply quantifies the benefit of interpolation in the nearest neighbors algorithm . Mei & Montanari ( 2019 ) derives the limiting risk on the random features model and shows that minimum generalization error is achieved by highly overparametrized interpolators . Ba et al . ( 2019 ) gives the limiting risk of the regression problem under two-layer neural networks . However , the existing asymptotic results focus on the first-order limit of the prediction risk and do not indicate the convergence rate . There are very few second-order results in the literature , Shen & Bellec ( 2020 ) establishes the asymptotic normality for the derivatives of 2-layers neural network , but not the exact limiting distribution of the risk . In this work , we are the first to develop results on second-order fluctuations of the prediction risk in linear regressions and provide its corresponding confidence intervals . The more data hurt phenomenon is further justified from the asymptotic point of view . Random Matrix Theory The primary tool for analyzing the second-order fluctuations of prediction risk comes from random matrix theory . In particular , Bai & Silverstein ( 2004 ) refines the central limit theorem for linear spectral statistics of large dimensional sample covariance matrix with general population and the population is not necessary to be Gaussian . Similar central limit theorems are also developed for other random matrix ensembles , see Sinai & Soshnikov ( 1998 ) ; Bai & Yao ( 2005 ) ; Zheng ( 2012 ) . Other than the central limit theorem for linear spectral statistics , Bai et al . ( 2007 ) and Pan & Zhou ( 2008 ) study the asymptotic fluctuation of eigenvectors of sample covariance matrices . Bai & Yao ( 2008 ) considers the fluctuation of quadratic forms . All these technical tools and results are adopted and fully utilized in this paper , especially those related to Stieltjes transform , which are closely connected to the prediction risk studied in this paper .

Summary: In this article, the authors characterized the second-order fluctuation of the prediction risk of the (min-norm) least square estimator, by assuming an underlying noisy teacher model $y_i = \beta^T x_i + \epsilon_i$, in the regime where the data dimension $p$ and the number of training samples $n$ grow large at the same pace. Results in both the under-parameterized (Theorem 4.1 and 4.2) and over-parameterized regimes (Theorem 4.3-4.5) were provided, under the statistical model where the data $x_i$ are zero-mean random vectors with **generic** i.i.d. entries and then "rotated" to have some possible covariance structures. Numerical experiments for relatively small values of $n,p$ were conducted to support the theoretical assessment.