Title: List-Decodable Mean Estimation in Nearly-PCA Time
Authors: Ilias Diakonikolas
Daniel Kane
Daniel Kongsgaard
Jerry Li
Kevin Tian
Venue: NeurIPS 2021
Keywords: robust statistics
learning theory
mixture models
semidefinite programming
list-decodable learning

Abstract: Robust statistics has traditionally focused on designing estimators tolerant to a minority of contaminated data. {\em List-decodable learning}~\cite{CharikarSV17} studies the more challenging regime where only a minority $\tfrac 1 k$ fraction of the dataset, $k \geq 2$, is drawn from the distribution of interest, and no assumptions are made on the remaining data. We study the fundamental task of list-decodable mean estimation in high dimensions. Our main result is a new algorithm for bounded covariance distributions 
with optimal sample complexity and near-optimal error guarantee, running in {\em nearly-PCA time}. Assuming the ground truth distribution on $\mathbb{R}^d$ has identity-bounded covariance, our algorithm outputs $O(k)$ candidate means, one of which is within distance $O(\sqrt{k\log k})$ from the truth. 
  
 Our algorithm runs in time $\widetilde{O}(ndk)$, where $n$ is the dataset size. This runtime nearly matches the cost of performing $k$-PCA on the data, a natural bottleneck of known algorithms for (very) special cases of our problem, such as clustering well-separated mixtures. Prior to our work, the fastest runtimes were $\widetilde{O}(n^2 d k^2)$~\cite{DiakonikolasKK20}, and $\widetilde{O}(nd k^C)$ \cite{CherapanamjeriMY20} for an unspecified constant $C \geq 6$. Our approach builds on a novel soft downweighting method we term SIFT, arguably the simplest known polynomial-time mean estimator in the list-decodable setting. To develop our fast algorithms, we boost the computational cost of SIFT via a careful ``win-win-win'' analysis of an approximate Ky Fan matrix multiplicative weights procedure we develop, which may be of independent interest.

TL;DR: We give a state-of-the-art algorithm for list-decodable mean estimation, the robust generalization of learning mixture models, attaining optimal error in polylogarithmic calls to approximate PCA.

Review discussion:
The article is quite significant because it exhibit a considerable faster algorithm than previous works for the same optimal error guarantees.  The article is clear and the quality of the redaction is ok.
Remark : I would ask the authors to be more mindful of the abbreviations they use. Their are a lot of abbreviations that are not transparent, for instance the notation PCA is never made clear in  the article : what algorithm does it represent exactly ?
Also SIFT is not transparent and in fact there already exist a SIFT algorithm in image processing so this might not be the best choice of name for the algorithm.
Similarly the reader also have to remember what is MMW...
No
Thank you for your careful reading, and your feedback regarding the exposition. We will make sure to be more explicit in a revision in terms of the abbreviations we use, especially those which are less well-known such as MMW. We will also add a pointer to the existence of the prior algorithm named SIFT in the image processing literature; thank you for this observation. To address your question regarding PCA, every time we use this abbreviation in our article we mean principal component analysis, a technique for identifying maximum variance directions.
We addressed your concerns regarding practicality in a meta-comment. We hope you found our discussion clarifying, and that it elevates your view of our paper. Thanks once again for your reviewing efforts.
Thank you for the answers.
Ok regarding the names of the algorithms..
I will not increase my score because given the complexity of the proposed algorithm I am really curious whether it works in practice or not. Even in toy datasets, complexity bounds are given but no practical running time on laptop for instance. This is my main concern about this article and even though this is a theoretical article, the main advantage of this article is the low complexity of the proposed algorithm hence for the article to rate higher, the reader must be convinced that the algorithm is really practically usable.
Strength: This paper improved all the previous results (e.g. [Charikar, Steinhardt, Valiant, STOC’17], [Diakonikolas, Kane, Kongsgaard NeurIPS’20], [Cheapanamjeri, Mohanty, Yau
FOCS’20]) running time. This paper is a truly breakthrough result.
More impressively, this paper proposed a super fancy technique called Subspace Isotropic FilTering
(SIFT) . To the best of my knowledge, I have never seen such a thing before, this is completely new.
Clearly, this paper should be at least an oral presentation at NeurIPS.
Minor:
In Reference, [AK+05], should be [AK05], the et al should be removed. This might be a bug of google scholar citation. Clearly, this paper only has two authors.
In Reference, the [DKK20] conference information is missing. It is a NeurIPS 2020 paper.
In Reference, the [LY20] conference information is missing. It is a NeurIPS 2020 paper.
In Reference, the [JLT20] conference information is missing, it is a NeurIPS 2020 paper.
This paper mentioned SDP in a number of places. Several recent, state-of-art SDP solvers should be cited ``A Faster Interior Point Method for Semidefinite Programming’’ and “Solving Tall Dense SDPs in the Current Matrix Multiplication Time”.
No
Thank you very much for your encouraging feedback; we are happy to hear that you found our result a breakthrough, and that you found our techniques interesting.
We will be sure to incorporate your feedback regarding changes in our citations, as well as the suggested pointers to the SDP literature; we agree these are relevant to our method.
This paper studies the list-decodable mean estimation problem: that is, if only $\frac{1}{\alpha}$-fraction of the data is the good point, we want to output a list of $O(\frac{1}{\alpha})$ solutions, one of which is guaranteed to be close to the true mean of the distribution. This problem can be seen as an extended to the outlier robust mean estimation problem and has received significant attention in recent years.
In this work, the authors give a new algorithm for the bounded covariance distribution that runs in $\tilde{O}(\frac{nd}{\alpha})$ time, which nearly matches the time to perform $k$-PCA on the data, which can be seen a natural bottleneck of this problem. This result is better than the previous one in [CMY20](FOCS 2020)(runs in time $\tilde{O}(\frac{nd}{\alpha^C})$ where $C \ge 6$) and only increase the error by logarithmic factor.
This paper first gives a simple version of the algorithm that runs in $\tilde{O}(\frac{n^2 d}{\alpha})$ but is conceptually new.
Then, following the result in [DHL19], the author uses the matrix multiplicative weights regret minimization framework to reduce the runtime to $\tilde{O}(\frac{nd}{\alpha})$. One technical contribution here is to give a Ky Fan k-norm generalization of MMW.
This paper is technically very strong and the writing is well. The authors provide a lot of intuition for the proofs. This paper is a pure theory paper.
However, a set of accompanying experiment results would be also nice.
Some minor questions:
- For the second "=" in the equation which starts in line 211, is it should be "$\le$" instead of "="?
No
Thank you for your thoughtful comments; we are glad you enjoyed the writing of the paper, and found the intuition we provided for our proofs useful. We also would like to point out (in reference to the “Main Review” feedback) that in addition to our result which improves [CMY20] by a factor of at least $\alpha^{-5}$ and only slightly loses in the estimation error, we provide an alternative algorithm which improves [CMY20] by a factor of at least $\min(\alpha^{-5}, nd)$ which matches the estimation error up to at most a constant (see Theorem 4 in the supplement).
Additionally, thank you for your clarifying question regarding Line 211. This equality is indeed intended to be an equality, see Fact 2 in the supplement for the derivation.
Thanks very much to the reviewers for the helpful feedback. We would like to address the feedback regarding the inclusion of experiments here, which was mentioned by all reviewers.
Our main contribution is a theoretical proof-of-concept of the practical tractability of the fundamental problem we study, by giving both improved runtimes and new, arguably simpler, techniques decoupling filtering and clustering. Our conceptual approach is new, and our runtimes at minimum remove overheads of roughly $n/\alpha$ or $\alpha^{-5}$ from prior bounds. We aim for these results to serve as a first step towards practical solutions to list-decodable mean estimation. Our new baseline algorithm, SIFT (Theorem 2 in the submission) is indeed only a few lines of code, and is the simplest algorithm we are aware of for solving our problem. Our algorithm with the strongest theoretical guarantees (Theorem 1 in the submission), shows how to use semidefinite programming to improve the theoretical performance of SIFT, an idea which has already experimentally yielded benefits in the minority-outlier regime (see [DHL19]).
We strongly agree that developing practical list-decodable mean estimators is an important undertaking, and plan to do so in follow-up work.
However, this is likely an endeavor requiring substantial care in its own right, and thus we believe it is outside the focus of our submission.

Meta-review:
role: PC
data: comment: This paper gives a nearly "PCA" time algorithm for the problem of list-decodable mean estimation. There are two relevant parameters in the problem: the underlying dimension d, the number of samples n, and the fraction of inliers \alpha. There's a significant history of works on this problem -- starting with polynomial (but large exponent) running time in both d and 1/\alpha. A significant leap was made in a recent work that gave linear time algorithm in nd but a large polynomial (1/\alpha^5) dependence on \alpha. This paper resolves the issue completely and gives an algorithm with a quasi-linear dependence in nd/\alpha. 

List-decodable mean estimation is a fundamental question in algorithmic robust statistics (it generalizes mean estimation and spherical mixture clustering problems) and finding efficient algorithms for the problem is an important research direction. This paper makes a foundational contribution to this area by giving a nearly optimal algorithm for the problem. While there were concerns of whether the nearly optimal running time in this paper translates into practical gains, the authors' response convincingly clarified that a provably linear time algorithm paves the way for a practically fast one. 

We recommend acceptance. 



Decision: Accept (Spotlight)