List-Decodable Mean Estimation in Nearly-PCA Time

√ k log k ) from the truth . Our algorithm runs in time Õ ( ndk ) 2 , where n is the dataset size . This runtime nearly matches the cost of performing k-PCA on the data , a natural bottleneck of known algorithms for ( very ) special cases of our problem , such as clustering well-separated mixtures . Prior to our work , the fastest runtimes were Õ ( n2dk2 ) [ DKK20 ] , and Õ ( ndkC ) [ CMY20 ] for an unspecified constant C ≥ 6 . Our approach builds on a novel soft downweighting method we term SIFT , arguably the simplest known polynomial-time mean estimator in the list-decodable setting . To develop our fast algorithms , we boost the computational cost of SIFT via a careful “ win-win-win ” analysis of an approximate Ky Fan matrix multiplicative weights procedure we develop , which may be of independent interest . ∗Part of this work was done as an intern at Microsoft Research . 2Throughout this work , the Õ notation hides logarithmic factors in n and the failure probability . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 1 Introduction . Mean estimation has emerged as one of the cornerstone tasks in robust statistics , as the most basic in a hierarchy of increasingly complex estimation problems . The problem is straightforward to state : given samples from a “ nice ” ground-truth distribution D , where an adversary has ( arbitrarily ) corrupted a fraction of the data , recover the mean of D as accurately as possible . Due to its fundamental nature , robust mean estimation has received extensive study in the statistics , theoretical computer science , and machine learning communities , starting from the 1960s [ Ans60 , Tuk60 , Hub64 , Tuk75 ] . Despite the apparent simplicity of the problem , efficient algorithms achieving near-optimal error rates were not known in high-dimensional settings until recently [ LRV16 , DKK+19 ] . These works studied mean recovery in the traditional setting where a majority of the data is “ trusted , ” i.e . the fraction of corruptions is strictly less than 12 . For the standard formulation of robust mean estimation , this assumption is necessary . Indeed , if only an α ≤ 12 fraction of points can be trusted , the dataset could consist of O ( 1α ) well-separated clusters of “ good ” points . The mean of each individual cluster is an equally valid solution to the estimation problem , so asking for a single solution is ill-posed . In many settings of theoretical and practical interest , having a majority of inlier points is too strong of an assumption . To circumvent the issue of well-posedness in the α ≤ 12 regime , [ CSV17 ] proposed a relaxed learning notion termed list-decodable learning . Rather than restricting to a single hypothesis , the algorithm may output a list of O ( 1α ) hypotheses , guaranteeing at least one of them is close to the truth . In the context of robust mean estimation , this amounts to outputting O ( 1α ) candidate means . A natural problem in its own right , list-decodable mean estimation also generalizes a number of other problems . A prototypical example is learning well-separated mixture models , which has received extensive treatment in the literature [ Das99 , VW04 , AM05 , DS07 , AK05 , RV17 , HL18 , DKS18 , KSS18 ] . In this problem , data is drawn from a uniform mixture3 of k “ nice ” distributionsD1 , . . . , Dk with far apart means , and the goal is to recover clusters coming from each component . By running a list-decodable estimation procedure with α = 1k , each cluster is an equally valid “ ground truth , ” so the output must contain candidate means close to each true mean . If the candidates are sufficiently close , standard techniques recover the true clustering . List-decodable mean estimation robustly extends this problem to tolerate adversarial noise or non-uniformity , up to constants in the output size . Moreover , list-decodable mean estimation can be used to model important data science applications such as crowdsourcing ( where a majority of respondents could be unreliable or malicious ) [ SVC16 , MV18 ] , or semi-random community detection in stochastic block models [ CSV17 ] . This primitive is particularly useful in the context of semi-verified learning [ CSV17 , MV18 ] , where a learner can audit a small amount of trusted data . Even if the trusted dataset is too small to directly learn from , in conjunction with a list-decodable learning procedure it can pinpoint a candidate hypothesis consistent with the verified data ( indeed , only roughly log 1α vetted points are required ) . The first tractable algorithm for high-dimensional list-decodable mean estimation was given by [ CSV17 ] . Their work considered the setting where D has ( unknown ) covariance Σ σ2I for some known σ ( i.e . a second moment bound ) . In this setting , [ CSV17 ] gave a sample-optimal , polynomial-time algorithm outputtingO ( 1α ) candidate means , so some candidate is within ` 2 distance O ( σ √ α−1 logα−1 ) from the mean of D. As later demonstrated in [ DKS18 ] , this error is optimal up to logarithmic factors . However , the [ CSV17 ] algorithm heavily relies on black-box semidefinite programming solvers , and thus the runtime is prohibitively large in high-dimensional instances . The goal of our work is to develop much faster , truly scalable algorithms for list-decodable mean estimation which achieve optimal statistical guarantees . This goal fits broadly into a larger line of work focused on understanding the computational cost of robustness for basic statistical tasks . In some settings , this line has demonstrated strong evidence that robustness comes at an inherent computational cost [ DKS17 , HL19 ] . In contrast , recent algorithms have been developed that achieve robustness essentially “ for free ” in many other settings [ CDG19 , DHL19 , CDGW19 , LY20 , JLT20 ] . While list-decodable mean estimation has received a fair amount of attention , there have only been a few results achieving improved runtimes . One line of work proposed an algorithm framework termed multi-filtering [ DKS18 , DKK20 ] , based on learning multiple candidate “ weight functions. ” In particular , [ DKK20 ] uses this approach to design an algorithm achieving near-optimal error in time Õ ( n2dα−2 ) , where n is the overall dataset size . While this dramatically improves over the runtime 3Some algorithms extend beyond the uniform setting ; we present it this way for simplicity of exposition . in [ CSV17 ] , the quadratic dependence on n is not ideal in high-dimensional settings . Concurrently , the work [ CMY20 ] proposes a different , descent-based algorithm based on ( approximate ) positive semidefinite programming , achieving optimal error ( up to constant factors ) in time Õ ( ndα−C ) for some constant C ≥ 6 . When α = Θ ( 1 ) , the [ CMY20 ] runtime is nearly-linear in the problem input size . However , if α−1 scales polynomially with d , e.g . a mixture model with many components in moderate dimension , then this large dependence on α−1 may also be prohibitively slow . In contrast to this somewhat murky runtime landscape , the state of affairs for clustering separated mixture models is relatively clear . The fastest known algorithm for clustering a mixture of k wellseparated components is almost twenty years old [ VW04 ] , and runs in time Õ ( ndk ) , as a relatively simple and elegant application of ( approximate ) k-principal components analysis ( k-PCA ) . Since list-decodable mean estimation can be thought of as the natural robust analog to clustering mixture models , it is natural to ask : Can we perform list-decodable mean estimation as efficiently as learning mixture models ? Concretely , since clustering mixture models is an instance of list-decodable learning with α = k−1 , the question becomes : can we solve list-decodable mean estimation in time Õ ( ndα ) ? This presents itself as a natural barrier , since further improvement would imply faster learning of mixture models . 2 Our results . Our main contribution is to answer this question affirmatively with near-optimal statistical guarantees . We start by formally defining the problem we study , which we call “ list-decodable mean estimation. ” Definition 1 . Given T ⊂ Rd and 0 < α ≤ 12 such that an α-fraction of T are independent draws from a distribution D with unknown target mean µ∗ and unknown covariance Σ σ2I , output a small list of vectors { µ̂j } j∈L such that minj∈L ‖µ̂j − µ∗‖2 is minimized . We emphasize that no assumptions are made on the remaining 1− α fraction of T , i.e . they may be arbitrary and chosen by a computationally unbounded adversary allowed to inspect the inliers . In the setting of Definition 1 , our main result states that we can solve list-decodable mean estimation nearlyoptimally in the time it takes to perform k-PCA poly-logarithmically many times , for k = Θ ( 1α ) . Theorem 1 ( See Corollary 3 of the supplementary material ) . Given |T | = n = Ω ( dα ) samples , there is an algorithm with runtime Õ ( nd α ) that with high probability outputs a list of m vectors { µj } j∈ [ m ] , for m = O ( 1α ) , such that minj∈ [ m ] ‖µ ∗ − µj‖2 = O ( σα− 1 2 √ logα−1 ) . A few remarks are in order . First , a list size of Ω ( 1α ) , sample complexity of Ω ( d α ) , and error of Ω ( σα− 1 2 ) are information-theoretically necessary for this problem [ DKS18 ] . We also provide an algorithm in the supplementary material returning m = O ( 1α ) hypotheses { µj } j∈ [ m ] attaining optimal error minj∈ [ m ] ‖µ∗ − µj‖2 = O ( σα− 1 2 ) , with an additive runtime overhead of Õ ( α−6 ) . Without loss of generality α−1 = o ( d ) , as otherwise there is a trivial algorithm ( cf . Appendix A of the supplementary material ) , so the α−6 additive term is only dominant in the range α−1 ∈ [ ω ( √ d ) , o ( d ) ] assuming n = Θ ( dα ) . Even with this overhead , our runtime is the best-known in all parameter regimes . Our approach is inspired by the way in which fast algorithms for robust mean estimation in the α→ 1 regime were built . First , a “ simple ” polynomial ( but not nearly-linear ) time algorithm — namely , the filter — was developed [ DKK+19 , DKK+17 , Ste18 ] . Then , building on the framework of this simple algorithm , near-linear time algorithms were developed [ CDG19 , DHL19 ] using tools from the continuous optimization community — specifically , matrix multiplicative weights regret analyses . In this paper , we accomplish both these steps for the α 12 regime . First , we design a simple “ basic ” algorithm for the list-decodable mean estimation problem , and then speed it up using matrix regret minimization tools . As explained below , both steps require substantially new ideas . SIFT : a new , simple algorithm for list-decodable learning . Our first main contribution is a novel algorithm for list-decodable mean estimation , which we call SIFT ( Subspace Isotropic FilTering ) ,4 4After the initial dissemination of this work , we were made aware there is another ( unrelated ) algorithm named SIFT developed in the computer vision community [ Low99 ] . achieving optimal statistical guarantees in time Õ ( n 2d α ) . While by itself , SIFT does not achieve a nearly-linear runtime , its framework is vital in designing our more sophisticated algorithms . Crucially , SIFT is conceptually different from all prior approaches for our problem , with a complete decoupling of a “ filtering ” step and a “ clustering ” step ; it is these differences that allow for our later speedups . The main advantage of SIFT is its simplicity . All prior algorithms for list-decodable mean estimation were quite complicated , with involved and lengthy analyses , whereas an analysis of SIFT fits in just over two pages . We believe SIFT is of independent interest ( both theoretically and practically ) , and its framework may find applications in other list-decodable learning settings . FastSIFT : speeding up SIFT via Ky Fan regret minimization . While each SIFT iteration can be performed in time Õ ( ndk ) , the algorithm requires Θ ( n ) iterations in the worst case , as there are instances where each iteration removes only one data point . The main challenge is to combine the SIFT analysis with a downweighting procedure guaranteeing termination in polylogarithmically many iterations . To achieve this , we use tools from semidefinite programming ( SDP ) to design iterative schemes with stronger termination guarantees . This mirrors , and is inspired by , the α→ 1 regime , where tools such as packing SDP solvers [ CDG19 ] and matrix multiplicative weights [ DHL19 ] were used to speed up the basic filter . We note that [ CMY20 ] also uses SDP tools to obtain their runtime , but in a substantially different way : they design a black-box Ky Fan norm packing SDP subroutine , whereas we directly use a ( stronger ) regret guarantee to guide weight removal . Finally , while there have been substantial recent runtime advances for solving SDPs ( see e.g . [ JKL+20 ] ) , these guarantees are too weak for our purposes , requiring the development of new approximate SDP tools .

Summary: This article is abound list-decodable learning in which we consider the mean estimation problem when more than half of the data are corrupted data. In this framework, the mean cannot be recovered exactly but one can obtain a list of $k$ potential mean and say that the mean of the inliers is among these $k$ values. This article exhibit an algorithm that attain optimal statistical bounds (up to log factors) in a linear running time (linear in the sample size but also the dimension and in $k$).