Consistent Estimation for PCA and Sparse Regression with Oblivious Outliers

We develop machinery to design efficiently computable and consistent estimators , achieving estimation error approaching zero as the number of observations grows , when facing an oblivious adversary that may corrupt responses in all but an α fraction of the samples . As concrete examples , we investigate two problems : sparse regression and principal component analysis ( PCA ) . For sparse regression , we achieve consistency for optimal sample size n & ( k log d ) /α2 and optimal error rate O ( √ ( k log d ) / ( n · α2 ) ) where n is the number of observations , d is the number of dimensions and k is the sparsity of the parameter vector , allowing the fraction of inliers to be inverse-polynomial in the number of samples . Prior to this work , no estimator was known to be consistent when the fraction of inliers α is o ( 1/log log n ) , even for ( non-spherical ) Gaussian design matrices . Results holding under weak design assumptions and in the presence of such general noise have only been shown in dense setting ( i.e. , general linear regression ) very recently by d ’ Orsi et al . ( dNS21 ) . In the context of PCA , we attain optimal error guarantees under broad spikiness assumptions on the parameter matrix ( usually used in matrix completion ) . Previous works could obtain non-trivial guarantees only under the assumptions that the measurement noise corresponding to the inliers is polynomially small in n ( e.g. , Gaussian with variance 1/n2 ) . To devise our estimators , we equip the Huber loss with non-smooth regularizers such as the ` 1 norm or the nuclear norm , and extend d ’ Orsi et al. ’ s approach ( dNS21 ) in a novel way to analyze the loss function . Our machinery appears to be easily applicable to a wide range of estimation problems . We complement these algorithmic results with statistical lower bounds showing that the fraction of inliers that our PCA estimator can deal with is optimal up to a constant factor . 1 Introduction . Estimating information from structured data is a central theme in statistics that by now has found applications in a wide array of disciplines . On a high level , a typical assumption in an estimation problem is the existence of a –known a priori– family of probability distributions P : { β β ∈ Ω } over some spaceZ that are each indexed by some parameter β ∈ Ω . We then observe a collection 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . of n independent observations Z ( Z1 , . . . , Zn ) drawn from an unknown probability distribution β∗ ∈ P. The goal is to ( approximately ) recover the hidden parameter β∗.1 Oftentimes , real-world data may contain skewed , imprecise or corrupted measurements . Hence , a desirable property for an estimator is to be robust to significant , possibly malicious , noise perturbations on the given observations . Indeed , in the last two decades , a large body of work has been developed on designing robust algorithms ( e.g see ( BGN09 ; BBC11 ; MMYSB19 ) ) . However , proving strong guarantees often either demands strong assumptions on the noise model or requires that the fraction of perturbed observations is small . More concretely , when we allow the noise to be chosen adaptively , i.e. , chosen dependently on the observations and hidden parameters , a common theme is that consistent estimators – estimators whose error tends to zero as the number of observations grows – can be attained only when the fraction of outliers is small . In order to make vanishing error possible in the presence of large fractions of outliers , it is necessary to consider weaker adversary models that are oblivious to the underlying structured data . In recent years , a flurry of works have investigated oblivious noise models ( CLMW11 ; ZLW+10 ; TJSO14 ; BJKK17 ; SZF18 ; SBRJ19 ; PJL21 ; dNS21 ) . These results however are tailor-made to the specific models and problems . To overcome this limitation , in this paper we aim to provide a simple blueprint to design provably robust estimators under minimalistic noise assumptions for a large class of estimation problems . As a testbed for our blueprint , we investigate two well-studied problems : Principal component analysis ( PCA ) : Given a matrix Y : L∗ + N where L∗ ∈ n×n is an unknown parameter matrix and N is an n-by-n random noise matrix , the goal is to find an estimator L̂ for L∗ which is as close as possible to L∗ in Frobenius norm . Sparse regression : Given observations ( X1 , y1 ) , . . . , ( Xn , yn ) following the linear model yi 〈Xi , β∗〉 + ηi where Xi ∈ d , β∗ ∈ d is the k-sparse parameter vector of interest ( by k-sparse we mean that it has at most k nonzero entries ) and η1 , . . . ηn is noise , the goal is to find an estimator β̂ for β∗ achieving small squared prediction error 1n ‖X ( β̂ − β∗ ) ‖2 , where X is the matrix whose rows are X1 , . . . , Xn .2 Principal component analysis . A natural way to describe principal component analysis under oblivious perturbations is that of assuming the noise matrix N to be an n-by-n matrix with a uniformly random set of α · n2 entries bounded by some small ζ > 0 in absolute value . In these settings , we may think of the α fraction of entries with small noise as the set of uncorrupted observations . Moreover , for ζ > 0 , the fact that even for uncorrupted observations the noise is non-zero allows us to capture both gross sparse errors and small entry-wise noise in the measurements at the same time ( for example if ζ 1 then the model captures settings with additional standard Gaussian noise ) . Remarkably , for ζ 0 , Candès et al. ’ s seminal work ( CLMW11 ) provided an algorithm that exactly recovers L∗ even for a vanishing fraction of inliers , under incoherence conditions on the signal L∗ . The result was slightly extended in ( ZLW+10 ) where the authors provided an algorithm recovering L∗ up to squared error O ( ζ2 · n4 ) thus allowing polynomially small measurement noise ζ , but still failing to capture settings where standard Gaussian measurement noise is added to the sparse noise . Even for simple signal matrices L∗ , prior to this work , it remained an open question whether consistent estimators could be designed in presence of both oblivious noise and more reasonable measurement noise ( e.g. , standard Gaussian ) . Linear regression . Similarly to the context of principal component analysis , a convenient model for oblivious adversarial corruptions is that of assuming the noise vector η to have a random set of α · n entries bounded by ζ in absolute value ( here all results of interest can be extended to any ζ > 0 just by scaling , so we consider only the case ζ 1 ) . This model trivially captures the classical settings with Gaussian noise ( a Gaussian vector with variance σ2 will have an α Θ ( 1σ ) fraction of inliers ) and , again , allows us to think of the α-fraction of entries with small noise as the set of uncorrupted observations . Early works on consistent regression focused on the regime with Gaussian design X1 , . . . , Xn ∼ N ( 0 , Σ ) and deterministic noise η.3 ( BJKK17 ) presented an estimator achieving error 1In this paper , we assume Ω ⊆ d andZ ⊆ D for some d , D , denote random variables in bold face , and hide absolute constant factors with the notation O ( · ) , Ω ( · ) , & , . and logarithmic factors with Õ ( · ) , Ω̃ ( · ) . 2Our analysis also works for the parameter error ‖β∗ − β̂‖ . 3For Gaussian design X one may also consider a deterministic noise model . This noise model is subsumed by the random noise model discussed here ( if X is Gaussian ) . As shown in ( dNS21 ) , roughly speaking the Õ ( d/ ( α2 · n ) ) for any α larger than a fixed constant . ( SBRJ19 ) extended this result by achieving comparable error rates even for a vanishing fraction of inliers α & 1/log log n. Assuming n & d2/α2 ( TJSO14 ) proved that the Huber-loss estimator ( Hub64 ) achieves optimal error rate O ( d/α2 · n ) even for polynomially small fraction of inliers α & √ d/n . This line of work culminated in ( dNS21 ) which extended the result of ( TJSO14 ) achieving the same guarantees with optimal sample complexity n & d/α2 . For Gaussian design , similar result as ( dNS21 ) can be extracted from independent work ( PF20 ) . Furthermore , the authors of ( dNS21 ) extended these guarantees to deterministic design matrices satisfying only a spreadness condition ( trivially satisfied by sub-Gaussian design matrices ) . Huber loss was also analyzed in context of linear regression in ( SZF18 ; DT19 ; SF20 ; PJL21 ) . These works studied models that are different from our model , and these results do not work in the case α o ( 1 ) . ( SZF18 ) used assumptions on the moments of the noise , ( DT19 ) and ( SF20 ) studied the model with non-oblivious adversary , and the model from ( PJL21 ) allows corruptions in covariates . In the context of sparse regression , less is known . When the fraction of observations is constant α > Ω ( 1 ) , ( NTN11 ) provided a first consistent estimator . Later ( DT19 ) and ( SF20 ) improved that result ( assuming α > Ω ( 1 ) , but these results also work with non-oblivious adversary ) . In the case α o ( 1 ) , the algorithm of ( SBRJ19 ) ( for Gaussian designs ) achieves nearly optimal error convergence Õ ( ( k log2 d ) /α2n ) , but requires α & 1/log log n. More recently , ( dNS21 ) presented an algorithm for standard Gaussian design X ∼ N ( 0 , 1 ) n×d , achieving the nearly optimal error convergence Õ ( k/ ( α2 · n ) ) for nearly optimal inliers fraction α & √ ( k · log2 d ) /n . Both ( SBRJ19 ) and ( dNS21 ) use an iterative process , and require a bound ‖β∗‖ 6 dO ( 1 ) ( for larger ‖β∗‖ these algorithms also work , but the fraction of inliers or the error convergence is worse ) . The algorithm from ( dNS21 ) , however , heavily relies on the assumption that X ∼ N ( 0 , 1 ) n×d and appears unlikely to be generalizable to more general families of matrices ( including non-spherical Gaussian designs ) . Our contribution . We propose new machinery to design efficiently computable consistent estimators achieving optimal error rates and sample complexity against oblivious outliers . In particular , we extend the approach of ( dNS21 ) to structured estimation problems , by finding a way to exploit adequately the structure therein . While consistent estimators have already been designed under more benign noise assumptions ( e.g . the LASSO estimator for sparse linear regression under Gaussian noise ) , it was previously unclear how to exploit this structure in the setting of oblivious noise . One key consequence of our work is hence to demonstrate what minimal assumptions on the noise are sufficient to make effective recovery ( in the sense above ) possible . Concretely , we show Oblivious PCA : Under mild assumptions on the noise matrix N and common assumption on the parameter matrix L∗ –traditionally applied in the context of matrix completion ( NW12 ) – we provide an algorithm that achieves optimal error guarantees . Sparse regression : Under mild assumptions on the design matrix and the noise vector –similar to the ones used in ( dNS21 ) for dense parameter vectors β∗ – we provide an algorithm that achieves optimal error guarantees and sample complexity . For both problems , our analysis improves over the state-of-the-art and recovers the classical optimal guarantees , not only for Gaussian noise , but also under much less restrictive noise assumptions . At a high-level , we achieve the above results by equipping the Huber loss estimator with appropriate regularizers . Our techniques closely follow standard analyses for M-estimators , but crucially depart from them when dealing with the observations with large perturbations . Furthermore , our analysis appears to be mechanical and thus easily applicable to many different estimation problems .

Summary: This paper studies sparse regression and principal component analysis under  oblivious perturbations.   Main contributions: 1. The authors propose estimators that achieve the optimal error guarantees for PCA and sparse regression under oblivious perturbations separately, by minimizing the Huber loss function with some regularization. For sparse regression the estimator also achieves the optimal sample complexity.  2. This machinery has the potential to be applied to other estimation problems.  Post rebuttal: I acknowledge that I have read the authors responses and other reviewers' comments.