Title: value: A Spectral Algorithm for List-Decodable Covariance Estimation in Relative Frobenius Norm
Authors: value: Ilias Diakonikolas
Daniel Kane
Jasper C.H. Lee
Ankit Pensia
Thanasis Pittas
Venue: NeurIPS 2023
Keywords: value: robust statistics
covariance estimation
list-decodable learning

Abstract: value: We study the problem of list-decodable Gaussian covariance estimation. Given a multiset $T$ of $n$ points in $\mathbb{R}^d$ such that an unknown $\alpha<1/2$ fraction of points in $T$ are i.i.d. samples from an unknown Gaussian $\mathcal{N}(\mu, \Sigma)$, the goal is to output a list of $O(1/\alpha)$ hypotheses at least one of which is close to $\Sigma$ in relative Frobenius norm. Our main result is a $\mathrm{poly}(d,1/\alpha)$ sample and time algorithm for this task that guarantees relative Frobenius norm error of $\mathrm{poly}(1/\alpha)$. Importantly, our algorithm relies purely on spectral techniques. As a corollary, we obtain an efficient spectral algorithm for robust partial clustering of Gaussian mixture models (GMMs) --- a key ingredient in the recent work of [BakDJKKV22] on robustly learning arbitrary GMMs. Combined with the other components of [BakDJKKV22], our new method yields the first Sum-of-Squares-free algorithm for robustly learning GMMs, resolving an open problem proposed by Vempala and Kothari. At the technical level, we develop a novel multi-filtering method for list-decodable covariance estimation that may be useful in other settings.


Review discussion:
List-decodable covariance estimation is a very natural and important problem. The relative Frobenius norm metric the authors consider is the natural choice here since it captures the "intrinsic" scale of the problem induced by the true covariance matrix -- even in the regime where it is not related to the total variation distance.
The authors show how to extend the techniques for list-decodable mean estimation to this setting. This requires non-trivial modifications since covariance estimation in general is a more difficult and much less well-understood task than mean estimation.
The paper is fairly well-written and the authors give a nice proof overview in Section 1.2.
The relation to IK22 was not clear to me in the beginning, and I believe it would be important to clarify already in the beginning. My main confusion comes from the fact that in Line 49 you say that small relative Frobenius norm is equivalent to small TVD.
Then you go on to say that IK22 achieve error in TVD and you in relative Frobenius norm. At this point it seems that the two error guarantees are the same, and it also seems like your algorithm is breaking the SQ lower bound.
Only later you clarify that for the equivalence to hold the relative frobenius norm needs to be smaller than 1. (You do say in Line 49 that it only holds if the relative Frobenius norm is smaller than a small enough constant but do not specify what this is and how it relates to $\mathrm{poly(1/\alpha)}$, your error guarantee.
You say that your list-decoding algorithm can be used to obtain a "sum-of-squares-free" algorithm for robustly clustering non-spherical Gaussians. Does this come with any (theoretical) speed-ups? If yes, I think it would be good to mention what these are.
Maybe I missed this, but I didn't see any statements about limitations in their work.
No ethics review needed.
We thank the reviewer for their effort and feedback, and address their points below.
**(Relation to [IK22])**
Thank you for pointing out the issue with phrasing. We will emphasize earlier on in the paper that only relative Frobenius error <= 1 would imply TV distance bounds.
**(Speed-ups)**
As mentioned in the statement of Theorem 1.2 and Remark F.2 in the Appendix, the runtime of our algorithm is $\tilde{O}(m^2d^2)$, where $m$ is the sample size and $d$ the ambient dimensionality. SDP optimization and SoS algorithms are generally much slower (note that in [BDJKKV22] the runtime is polynomial in these parameters with the power being some large constant). We thus expect a sizable speed-up in the SoS-free version that we provide. If the paper is accepted, we will move the runtime discussion into the main body given the extra page.
Thank you for your response and answering my questions.
This is a very nice paper that tackles an important problem in robust statistics, and the result has direct applications in robustly learning GMMs. The contributions of the paper are novel algorithms that significantly improve the state-of-art algorithm in terms of sample complexity and answer size. The paper is very well written and it is a pleasure to read. In our opinion, this is an important contribution to the literature.
The organization of this paper can be improved.
In particular, I think the stability section deserves some space in the main paper and
the paper could benefit from a discussion section at the end of the paper.
N/A
No ethics review needed.
We thank the reviewer for their effort and comments. If the paper is accepted, we will move content into the main body as suggested, using the extra page allowed in the camera ready version.
Strengths:
+ A fairly simple to describe algorithm based on multi-filtering
+ avoids solve large convex programs
+ achieves a reasonable $poly(d/\alpha)$ sample complexity and running time guarantee, much better than most previous algorithms
The main theorems in the paper only talk of recovering the covariance matrix, but not a pair $(\tilde{\mu}, \tilde{\Sigma})$ where both the mean and covariance are close. Are these guarantees also achievable? If yes, I strongly suggest the authors include such a complete result
Your algorithm removes points one by one in line 16, what is the obstacle in removing a large fraction of the points at each step?
Minor comments:
1. In Def 1.1, since the adversary choses all the outliers after looking at the distribution, is it not equivalent to the setting of having exactly $\alpha m$ inliers?
2. Theorem 1.2, first bullet point -- it's awkward to recall within a theorem statement, I suggest removing the sentence "Recall ... $\ell \le \epsilon n$"
Theoretical work.
See weakness pointed above.
No ethics review needed.
We thank the reviewer for their time and effort. We address their questions below:
**(Guarantees on pairs of covariance-mean candidates)**
Yes, it is possible to output $(\hat{\mu},\hat{\Sigma})$ pairs such that, $|| \hat{\Sigma}^{-1/2}\Sigma \hat{\Sigma}^{-1/2} - I ||_{F} \leq \mathrm{poly}(1/\alpha)$ (as in the paper) and furthermore that $\hat{\mu}$ is a bounded estimate of the true $\mu$ in Mahalanobis norm with respect to $\hat{\Sigma}$: $(\hat{\mu} - \mu) \hat{\Sigma} (\hat{\mu} - \mu) \leq \mathrm{poly}(1/\alpha)$.
The idea is simple: after running our list-decoding algorithm for $\hat{\Sigma}$, for each candidate $\hat{\Sigma}$, whiten the data by $\hat{\Sigma}$ and run a list-decoding algorithm for candidate means $\hat{\mu}$ (see for example [CSV17, DKK20]).
To show this works, we need to check the main assumption for the list-decoding algorithm for the mean, that the data distribution has bounded covariance. After whitening, the data is distributed with covariance $\hat{\Sigma}^{-1/2}\Sigma \hat{\Sigma}^{-1/2}$, which has operator norm $\le \mathrm{poly}(1/\alpha)$ by our list-decoding guarantees.
**(Removing single point vs multiple points per filtering iteration)**
Some other filtering-based algorithms in robust statistics filters by flipping an independent coin for each sample (with bias proportional to the score), thus deleting multiple samples for each filter. In these works, this results in slightly better runtime analysis, replacing a factor of $m$ by $d$.
However, most of these other works assume a known covariance (or at least, sphericality), which does not apply immediately in our setting. Given that our runtime is $\tilde{O}(m^2d^2)$, and the sample complexity is $d^2$ resulting in a $d^6$ runtime, the improvement would only bring the runtime down to $d^5$ anyway, so we did not pursue this potential improvement. We emphasize that the focus of this work is getting the first SoS-free spectral algorithm, and not getting the fastest runtime. We hope that the tools and techniques in our paper can inspire future work on optimizing the runtime of the algorithm.
**(Contamination model)**
The contamination model that we define is slightly stronger in the sense that the adversary can also remove a small fraction of inliers too. Subtractive corruption can introduce biases into the data, and is non-trivial to handle.
Thank you for your detailed responses.
Consider including these comments to your paper, if you think they might be helpful to the reader.
The paper proposes a simple and efficient algorithm for a fundamental problem in robust statistics. The proposed algorithm also has an application in learning Mixtures of Gaussian distributions, resolving an open problem in the literature.
Although the main idea of the algorithm is simple, the analysis is quite delicate and requires a considerable amount of technical work, as well as careful choice of the overall objective (decoding only up to a list and guarantees in terms of the relative Frobenius norm).
Moreover, the presentation of the setting, the results, the techniques and the related work are complete and organized and the proofs are split in several lemmas that might be of independent interest.
One minor presentation issue is that the paper becomes overly technical in some places.
For example, in section 1.2 (and in particular in the beginning of the paragraph), it might be preferable to provide some abstract description of the main approach, instead of (or in addition to) defining the particular quantities that the algorithm uses (like the covariance matrix whose eigenvalues determine whether the algorithm stops). In other parts of the paper, however, providing exact (and therefore technical) claims is indeed important (for example in the theorem statements and the proofs).
My main question concerns the choice of the relative Frobenius norm. I understand the existence of lower bounds for different metrics, the intuition provided in the paper regarding the relative Frobenius norm, as well as the fact that minimizing the relative Frobenius norm has applications in learning GMMs, but what are other metrics that one might hope to get results for (if any)?
Yes, limitations are adequately addressed.
No ethics review needed.
Thank you for your appreciation of our work and for your question. There are some variants for the relative Frobenius norm one could consider: 1) currently the relative Frobenius norm is measured with respect to the basis of the output $H$, one could ask for measuring with respect to the true $\Sigma$ instead, and 2) instead of Frobenius norm (which is already stronger than the spectral norm), one could perhaps ask for even stronger guarantees such as the nuclear norm.
While we believe 1) should not be possible in general, it is an interesting direction to investigate other error norms for future work.
For 1), this should not be possible based on evidence in known SQ lower bounds. Suppose the true $\Sigma$ is very thin in an unknown (and computationally difficult to detect) direction $v$ and unit variance in all other directions. To ensure $\Sigma^{-1/2} H \Sigma^{-1/2} - I$ has a small Frobenius norm, the output $H$ would also have to be (i) small in the direction $v$ and (ii) have roughly unit variance in all other directions. Such an $H$ would contradict known SQ lower bounds. See for example [DKS17,DKPPS21].
For 2), the naïve bound on the relative nuclear norm blows up our guarantees by a factor of $\sqrt{d}$. We don’t know if it is possible to remove this $d$-dependence. The algorithm we propose is intrinsically $\ell_2$ in nature: we consider flattening second moment matrices of samples, and consider the Euclidean geometry of these flattened matrices. The $\ell_2$ geometry of flattened matrices naturally correspond to the Frobenius norm, and so it is unclear if the current algorithm would also offer any guarantees with respect to the relative nuclear norm (which is $\ell_1$ in terms of the eigenvalues of the matrices, instead of $\ell_2$).
Thank you for your detailed response to my question.
We thank the reviewers for the in-depth reviews, and for the positive assessment of our paper. We are particularly encouraged by reviewers recognizing our paper for 1) tackling a fundamental/important problem in robust statistics (Reviewers ymEk, Wync, WZLc), and 2) proposing a simple and efficient algorithm based on multi-filtering (Reviewers ymEk, 64Ly, Wync, WZLc). We are also pleased that the reviewers found our paper well-written (Reviewers ymEk, Wync, WZLc), and we will take into account the further suggestions to improve certain parts of the presentation, particularly using the extra page available in the camera-ready version, if the paper is accepted.

Meta-review:
role: PC
data: comment: value: The paper addresses the problem of estimating the covariance matrix of a high-dimensional Gaussian distribution from corrupted samples in robust statistics. It introduces a simple and efficient algorithm that outputs a concise list of matrices, containing a matrix close to the true covariance in relative Frobenius norm. The algorithm computes empirical covariances for each sample, filters or splits the sample sets based on variance, and provides accurate estimates when projections on the direction of maximum variance show multiple modes in empirical covariances. The approach extends to Gaussian Mixture Models and provides a new algorithm for robustly learning GMMs without using semi-definite programming. While some sections are highly technical, the paper's presentation is organized and comprehensive, and it tackles an open problem in the field with significant contributions. One referee wondered about the choice of relative Frobenius norm over other metrics and potential metrics for obtaining results.

Decision: Accept (spotlight)