Consistent Estimation for PCA and Sparse Regression with Oblivious Outliers

We develop machinery to design efficiently computable and consistent estimators , achieving estimation error approaching zero as the number of observations grows , when facing an oblivious adversary that may corrupt responses in all but an α fraction of the samples . As concrete examples , we investigate two problems : sparse regression and principal component analysis ( PCA ) . For sparse regression , we achieve consistency for optimal sample size n & ( k log d ) /α2 and optimal error rate O ( √ ( k log d ) / ( n · α2 ) ) where n is the number of observations , d is the number of dimensions and k is the sparsity of the parameter vector , allowing the fraction of inliers to be inverse-polynomial in the number of samples . Prior to this work , no estimator was known to be consistent when the fraction of inliers α is o ( 1/log log n ) , even for ( non-spherical ) Gaussian design matrices . Results holding under weak design assumptions and in the presence of such general noise have only been shown in dense setting ( i.e. , general linear regression ) very recently by d ’ Orsi et al . ( dNS21 ) . In the context of PCA , we attain optimal error guarantees under broad spikiness assumptions on the parameter matrix ( usually used in matrix completion ) . Previous works could obtain non-trivial guarantees only under the assumptions that the measurement noise corresponding to the inliers is polynomially small in n ( e.g. , Gaussian with variance 1/n2 ) . To devise our estimators , we equip the Huber loss with non-smooth regularizers such as the ` 1 norm or the nuclear norm , and extend d ’ Orsi et al. ’ s approach ( dNS21 ) in a novel way to analyze the loss function . Our machinery appears to be easily applicable to a wide range of estimation problems . We complement these algorithmic results with statistical lower bounds showing that the fraction of inliers that our PCA estimator can deal with is optimal up to a constant factor . 1 Introduction . Estimating information from structured data is a central theme in statistics that by now has found applications in a wide array of disciplines . On a high level , a typical assumption in an estimation problem is the existence of a –known a priori– family of probability distributions P : { β β ∈ Ω } over some spaceZ that are each indexed by some parameter β ∈ Ω . We then observe a collection 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . of n independent observations Z ( Z1 , . . . , Zn ) drawn from an unknown probability distribution β∗ ∈ P. The goal is to ( approximately ) recover the hidden parameter β∗.1 Oftentimes , real-world data may contain skewed , imprecise or corrupted measurements . Hence , a desirable property for an estimator is to be robust to significant , possibly malicious , noise perturbations on the given observations . Indeed , in the last two decades , a large body of work has been developed on designing robust algorithms ( e.g see ( BGN09 ; BBC11 ; MMYSB19 ) ) . However , proving strong guarantees often either demands strong assumptions on the noise model or requires that the fraction of perturbed observations is small . More concretely , when we allow the noise to be chosen adaptively , i.e. , chosen dependently on the observations and hidden parameters , a common theme is that consistent estimators – estimators whose error tends to zero as the number of observations grows – can be attained only when the fraction of outliers is small . In order to make vanishing error possible in the presence of large fractions of outliers , it is necessary to consider weaker adversary models that are oblivious to the underlying structured data . In recent years , a flurry of works have investigated oblivious noise models ( CLMW11 ; ZLW+10 ; TJSO14 ; BJKK17 ; SZF18 ; SBRJ19 ; PJL21 ; dNS21 ) . These results however are tailor-made to the specific models and problems . To overcome this limitation , in this paper we aim to provide a simple blueprint to design provably robust estimators under minimalistic noise assumptions for a large class of estimation problems . As a testbed for our blueprint , we investigate two well-studied problems : Principal component analysis ( PCA ) : Given a matrix Y : L∗ + N where L∗ ∈ n×n is an unknown parameter matrix and N is an n-by-n random noise matrix , the goal is to find an estimator L̂ for L∗ which is as close as possible to L∗ in Frobenius norm . Sparse regression : Given observations ( X1 , y1 ) , . . . , ( Xn , yn ) following the linear model yi 〈Xi , β∗〉 + ηi where Xi ∈ d , β∗ ∈ d is the k-sparse parameter vector of interest ( by k-sparse we mean that it has at most k nonzero entries ) and η1 , . . . ηn is noise , the goal is to find an estimator β̂ for β∗ achieving small squared prediction error 1n ‖X ( β̂ − β∗ ) ‖2 , where X is the matrix whose rows are X1 , . . . , Xn .2 Principal component analysis . A natural way to describe principal component analysis under oblivious perturbations is that of assuming the noise matrix N to be an n-by-n matrix with a uniformly random set of α · n2 entries bounded by some small ζ > 0 in absolute value . In these settings , we may think of the α fraction of entries with small noise as the set of uncorrupted observations . Moreover , for ζ > 0 , the fact that even for uncorrupted observations the noise is non-zero allows us to capture both gross sparse errors and small entry-wise noise in the measurements at the same time ( for example if ζ 1 then the model captures settings with additional standard Gaussian noise ) . Remarkably , for ζ 0 , Candès et al. ’ s seminal work ( CLMW11 ) provided an algorithm that exactly recovers L∗ even for a vanishing fraction of inliers , under incoherence conditions on the signal L∗ . The result was slightly extended in ( ZLW+10 ) where the authors provided an algorithm recovering L∗ up to squared error O ( ζ2 · n4 ) thus allowing polynomially small measurement noise ζ , but still failing to capture settings where standard Gaussian measurement noise is added to the sparse noise . Even for simple signal matrices L∗ , prior to this work , it remained an open question whether consistent estimators could be designed in presence of both oblivious noise and more reasonable measurement noise ( e.g. , standard Gaussian ) . Linear regression . Similarly to the context of principal component analysis , a convenient model for oblivious adversarial corruptions is that of assuming the noise vector η to have a random set of α · n entries bounded by ζ in absolute value ( here all results of interest can be extended to any ζ > 0 just by scaling , so we consider only the case ζ 1 ) . This model trivially captures the classical settings with Gaussian noise ( a Gaussian vector with variance σ2 will have an α Θ ( 1σ ) fraction of inliers ) and , again , allows us to think of the α-fraction of entries with small noise as the set of uncorrupted observations . Early works on consistent regression focused on the regime with Gaussian design X1 , . . . , Xn ∼ N ( 0 , Σ ) and deterministic noise η.3 ( BJKK17 ) presented an estimator achieving error 1In this paper , we assume Ω ⊆ d andZ ⊆ D for some d , D , denote random variables in bold face , and hide absolute constant factors with the notation O ( · ) , Ω ( · ) , & , . and logarithmic factors with Õ ( · ) , Ω̃ ( · ) . 2Our analysis also works for the parameter error ‖β∗ − β̂‖ . 3For Gaussian design X one may also consider a deterministic noise model . This noise model is subsumed by the random noise model discussed here ( if X is Gaussian ) . As shown in ( dNS21 ) , roughly speaking the Õ ( d/ ( α2 · n ) ) for any α larger than a fixed constant . ( SBRJ19 ) extended this result by achieving comparable error rates even for a vanishing fraction of inliers α & 1/log log n. Assuming n & d2/α2 ( TJSO14 ) proved that the Huber-loss estimator ( Hub64 ) achieves optimal error rate O ( d/α2 · n ) even for polynomially small fraction of inliers α & √ d/n . This line of work culminated in ( dNS21 ) which extended the result of ( TJSO14 ) achieving the same guarantees with optimal sample complexity n & d/α2 . For Gaussian design , similar result as ( dNS21 ) can be extracted from independent work ( PF20 ) . Furthermore , the authors of ( dNS21 ) extended these guarantees to deterministic design matrices satisfying only a spreadness condition ( trivially satisfied by sub-Gaussian design matrices ) . Huber loss was also analyzed in context of linear regression in ( SZF18 ; DT19 ; SF20 ; PJL21 ) . These works studied models that are different from our model , and these results do not work in the case α o ( 1 ) . ( SZF18 ) used assumptions on the moments of the noise , ( DT19 ) and ( SF20 ) studied the model with non-oblivious adversary , and the model from ( PJL21 ) allows corruptions in covariates . In the context of sparse regression , less is known . When the fraction of observations is constant α > Ω ( 1 ) , ( NTN11 ) provided a first consistent estimator . Later ( DT19 ) and ( SF20 ) improved that result ( assuming α > Ω ( 1 ) , but these results also work with non-oblivious adversary ) . In the case α o ( 1 ) , the algorithm of ( SBRJ19 ) ( for Gaussian designs ) achieves nearly optimal error convergence Õ ( ( k log2 d ) /α2n ) , but requires α & 1/log log n. More recently , ( dNS21 ) presented an algorithm for standard Gaussian design X ∼ N ( 0 , 1 ) n×d , achieving the nearly optimal error convergence Õ ( k/ ( α2 · n ) ) for nearly optimal inliers fraction α & √ ( k · log2 d ) /n . Both ( SBRJ19 ) and ( dNS21 ) use an iterative process , and require a bound ‖β∗‖ 6 dO ( 1 ) ( for larger ‖β∗‖ these algorithms also work , but the fraction of inliers or the error convergence is worse ) . The algorithm from ( dNS21 ) , however , heavily relies on the assumption that X ∼ N ( 0 , 1 ) n×d and appears unlikely to be generalizable to more general families of matrices ( including non-spherical Gaussian designs ) . Our contribution . We propose new machinery to design efficiently computable consistent estimators achieving optimal error rates and sample complexity against oblivious outliers . In particular , we extend the approach of ( dNS21 ) to structured estimation problems , by finding a way to exploit adequately the structure therein . While consistent estimators have already been designed under more benign noise assumptions ( e.g . the LASSO estimator for sparse linear regression under Gaussian noise ) , it was previously unclear how to exploit this structure in the setting of oblivious noise . One key consequence of our work is hence to demonstrate what minimal assumptions on the noise are sufficient to make effective recovery ( in the sense above ) possible . Concretely , we show Oblivious PCA : Under mild assumptions on the noise matrix N and common assumption on the parameter matrix L∗ –traditionally applied in the context of matrix completion ( NW12 ) – we provide an algorithm that achieves optimal error guarantees . Sparse regression : Under mild assumptions on the design matrix and the noise vector –similar to the ones used in ( dNS21 ) for dense parameter vectors β∗ – we provide an algorithm that achieves optimal error guarantees and sample complexity . For both problems , our analysis improves over the state-of-the-art and recovers the classical optimal guarantees , not only for Gaussian noise , but also under much less restrictive noise assumptions . At a high-level , we achieve the above results by equipping the Huber loss estimator with appropriate regularizers . Our techniques closely follow standard analyses for M-estimators , but crucially depart from them when dealing with the observations with large perturbations . Furthermore , our analysis appears to be mechanical and thus easily applicable to many different estimation problems .

Summary: The paper studies algorithms for sparse regression and principle component analysis using Huber loss in the presence of overwhelming oblivious noise. In particular, the fraction of inliers, $\alpha$, is going to $0$ but it is still possible to achieve consistency as the noise model is benign (oblivious, symmetric noise). The paper obtains optimal error rates and dependence on $\alpha$ for a wide range of distribution assumptions. These rates are achieved by using regularized Huber loss (nuclear norm for the matrix case and $\ell_1$-norm for regression case).