SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ilias Diakonikolas
University of Wisconsin-Madison ilias@cs.wisc.edu
Daniel M. Kane
University of California, San Diego  
dakane@cs.ucsd.edu
Jasper C.H. Lee
University of Wisconsin-Madison jasper.lee@wisc.edu
Ankit Pensia
IBM Research  
ankitp@ibm.com
Thanasis Pittas
University of Wisconsin-Madison pittas@wisc.edu
We study the problem of list-decodable Gaussian covariance estimation. Given a multiset  $T$  of  $n$  points in  $\mathbb{R}^d$  such that an unknown  $\alpha < 1/2$  fraction of points in  $T$  are i.i.d. samples from an unknown Gaussian  $\mathcal{N}(\mu, \Sigma)$ , the goal is to output a list of  $O(1/\alpha)$  hypotheses at least one of which is close to  $\Sigma$  in relative Frobenius norm. Our main result is a poly  $(d, 1/\alpha)$  sample and time algorithm for this task that guarantees relative Frobenius norm error of  $\mathrm{poly}(1/\alpha)$ . Importantly, our algorithm relies purely on spectral techniques. As a corollary, we obtain an efficient spectral algorithm for robust partial clustering of Gaussian mixture models (GMMs) — a key ingredient in the recent work of [BDJKKV22] on robustly learning arbitrary GMMs. Combined with the other components of [BDJKKV22], our new method yields the first Sum-of-Squares-free algorithm for robustly learning GMMs, resolving an open problem proposed by Vempala [Vem22] and Kothari [Kot21]. At the technical level, we develop a novel multi-filtering method for list-decodable covariance estimation that may be useful in other settings.
Robust statistics studies the efficient (parameter) learnability of an underlying distribution given samples some fraction of which might be corrupted, perhaps arbitrarily. While the statistical theory of these problems has been well-established for some time [Hub64; HR09], only recently has the algorithmic theory of such estimation problems begun to be understood [DKKLMS16; LRV16; DK23].
A classical problem in robust estimation is that of multivariate robust mean estimation — that is, estimating the mean of an unknown distribution in the presence of a small constant fraction of outliers. One of the original results in the field is that given samples from a Gaussian  $\mathcal{N}(\mu, I)$  with an  $\epsilon$ -fraction of outliers (for some  $\epsilon < 1/2$ ), the unknown mean  $\mu$  can be efficiently estimated to an error of  $O(\epsilon \sqrt{\log 1/\epsilon})$  in the  $\ell_2$ -norm [DKKLMS16]. Note that the use of the  $\ell_2$ -norm here is quite natural, as the total variation distance between two identity-covariance Gaussians is roughly proportional to the  $\ell_2$ -distance between their means; thus, this estimator learns the underlying distribution to total variation distance error  $O(\epsilon \sqrt{\log 1/\epsilon})$ . This bound cannot be substantially improved, as learning to error  $o(\epsilon)$  in total variation distance is information-theoretically impossible.
While the above algorithmic result works when  $\epsilon$  is small (less than  $1/2$ ), if more than half of the samples are corrupted, it becomes impossible to learn with only a single returned hypothesis—the corruption might simply simulate other Gaussians, and no algorithm can identify which Gaussian is the original one. This issue can be circumvented using the list-decodable mean estimation
paradigm [CSV17], where the algorithm is allowed to output a small list of hypotheses with the guarantee that at least one of them is relatively close to the target. List-decodable learning is closely related to semi-verified learning [CSV17; MV18], where a learner can choose to audit a small amount of data. The framework has been shown to capture a multitude of applications, including crowdsourcing [SVC16; MV18], semi-random community detection in stochastic block models [CSV17] and clustering (as we also show in this work). For Gaussian mean estimation in particular, if an  $\alpha$ -fraction of the samples are clean (i.e., uncorrupted), for some  $\alpha < 1/2$ , there exist polynomial-time algorithms that return a list of  $O(1/\alpha)$  hypotheses such that (with high probability) at least one of them is within  $\tilde{O}(\sqrt{1/\alpha})$  of the mean in  $\ell_2$ -distance [CSV17]. Note that while this  $\ell_2$ -norm distance bound does not imply a good bound on the total variation distance between the true distribution and the learned hypothesis, it does bound their distance away from one, ensuring some non-trivial overlap.
Another important, and arguably more complex, problem in robust statistics (the focus of this work) is that of robustly estimating the covariance of a multivariate Gaussian. It was shown in [DKKLMS16] that given  $\epsilon$ -corrupted samples from  $\mathcal{N}(0,\Sigma)$  (for  $\epsilon < 1/2$ ) there is a polynomial-time algorithm for estimating  $\Sigma$  to error  $O(\epsilon \log 1/\epsilon)$  in the relative Frobenius norm, i.e., outputting a hypothesis covariance  $\widetilde{\Sigma}$  satisfying  $\| \widetilde{\Sigma}^{-1/2}\Sigma \widetilde{\Sigma}^{-1/2} - I \|_{\mathrm{F}} \leq \epsilon$ . This is again the relevant metric, since, if  $\| \widetilde{\Sigma}^{-1/2}\Sigma \widetilde{\Sigma}^{-1/2} - I \|_{\mathrm{F}} \leq 2/3$ , then the total variation distance between  $\mathcal{N}(0,\Sigma)$  and  $\mathcal{N}(0,\widetilde{\Sigma})$  is proportional to the relative Frobenius norm  $\| \widetilde{\Sigma}^{-1/2}\Sigma \widetilde{\Sigma}^{-1/2} - I \|_{\mathrm{F}}$  [DMR18].
A natural goal, with a number of applications, is to extend the above algorithmic result to the list-decodable setting. That is, one would like a polynomial-time algorithm that given corrupted samples from  $\mathcal{N}(0,\Sigma)$  (with an  $\alpha$ -fraction of clean samples, for some  $\alpha < 1/2$ ), returns a list of  $O(1/\alpha)$  many hypotheses with the guarantee that at least one of them has some non-trivial overlap with the true distribution in total variation distance. We start by noting that the sample complexity of list-decodable covariance estimation is  $\mathrm{poly}(d/\alpha)$ , albeit via an exponential time algorithm. The only known algorithm for list-decodable covariance estimation (with total variation error guarantees) is due to [IK22]. This algorithm essentially relies on the Sum-of-Squares method and has (sample and computational) complexity  $d^{\mathrm{poly}(1/\alpha)}$ . Intriguingly, there is compelling evidence that this complexity bound cannot be improved. Specifically, [DKS17; DKPPS21] showed that any Statistical Query (SQ) algorithm for this task requires complexity  $d^{\mathrm{poly}(1/\alpha)}$ . Combined with the reduction of [BBHLT20], this implies a similar lower bound for low-degree polynomial tests. These lower bounds suggest an intrinsic information-computation gap for the problem.
The aforementioned lower bound results [DKS17; DKPPS21] establish that it is SQ (and low-degree) hard to distinguish between a standard multivariate Gaussian,  $\mathcal{N}(0,I)$ , and a distribution  $P$  that behaves like  $\mathcal{N}(0,I)$  (in terms of low-degree moments) but  $P$  contains an  $\alpha$ -fraction of samples from a Gaussian  $\mathcal{N}(0,\Sigma)$ , where  $\Sigma$  is very thin in some hidden direction  $v$ . Since the distribution  $\mathcal{N}(0,\Sigma)$  is very thin along  $v$ , i.e., has very small variance, the obvious choice of  $\mathcal{N}(0,I)$  essentially has no overlap with  $\mathcal{N}(0,\Sigma)$  — making this kind of strong list-decoding guarantee (closeness in total variation distance) likely computationally intractable.
Interestingly, there are two possible ways that a pair of mean zero Gaussians can be separated [DHKK20; BDHKKK20]: (1) one could be much thinner than the other in some direction, or (2) they could have many orthogonal directions in which their variances differ, adding up to something more substantial. While the lower bounds of [DKS17; DKPPS21] seem to rule out being able to detect deviations of the former type in fully polynomial time (i.e.,  $\mathrm{poly}(d / \alpha)$ ), it does not rule out efficiently detecting deviations of the latter. In particular, we could hope to find (in  $\mathrm{poly}(d / \alpha)$  time) a good hypothesis  $\widetilde{\Sigma}$  such that  $\| \widetilde{\Sigma}^{-1 / 2}\Sigma \widetilde{\Sigma}^{-1 / 2} - I\|_{\mathrm{F}}$  is not too big. While this does not exclude the possibility that  $\Sigma$  is much thinner than  $\widetilde{\Sigma}$  in a small number of independent directions, it does rule out the second kind of difference between the two. The main goal of this paper is to provide an elementary (relying only on spectral techniques) list-decoding algorithm of this form. Given corrupted samples from a Gaussian  $\mathcal{N}(0,\Sigma)$ , we give a poly  $(d / \alpha)$ -time algorithm that returns a small list of hypotheses  $\widetilde{\Sigma}$  such that for at least one of them we have that  $\| \widetilde{\Sigma}^{-1 / 2}\Sigma \widetilde{\Sigma}^{-1 / 2} - I\|_{\mathrm{F}} < \mathrm{poly}(1 / \alpha)$ .
In addition to providing the best qualitative guarantee we could hope to achieve in fully polynomial time, the above kind of "weak" list-decoding algorithm has interesting implications for the well-studied problem of robustly learning Gaussian mixture models (GMMs). [BDJKKV22] gave a polynomial-time algorithm to robustly learn arbitrary mixtures of Gaussians (with a constant number
of components). One of the two main ingredients of their approach is a subroutine that could perform partial clustering of points into components that satisfy exactly this kind of weak closeness guarantee. [BDJKKV22] developed such a subroutine by making essential use of a Sum-of-Squares (SoS) relaxation, for the setting that the samples come from a mildly corrupted mixture of Gaussians (with a small constant fraction of outliers). As a corollary of our techniques, we obtain an elementary spectral algorithm for this partial clustering task (and, as already stated, our results work in the more general list-decoding setting). This yields the first SoS-free algorithm for robustly learning GMMs, answering an open problem in the literature [Vem22; Kot21].
Our main result is a polynomial-time algorithm for list-decoding the covariance of a Gaussian in relative Frobenius norm, under adversarial corruption where more than half the samples could be outliers. Definition 1.1 makes precise the corruption model, and Theorem 1.2 states the guarantees of our main algorithm (Algorithm 1).
Definition 1.1 (Corruption model for list-decoding). Let the parameters  $\epsilon, \alpha \in (0,1/2)$  and a distribution family  $\mathcal{D}$ . The statistician specifies the number of samples  $m$ . Then a set of  $n \geq \alpha m$  i.i.d. points are sampled from an unknown  $D \in \mathcal{D}$ . We call these  $n$  samples the inliers. Upon inspecting the  $n$  inliers, a (malicious and computationally unbounded) adversary can replace an arbitrary  $\ell \leq \epsilon n$  of the inliers with arbitrary points, and further add  $m - n$  arbitrary points to the dataset, before returning the entire set of  $m$  points to the statistician. The parameter  $\alpha$  is also known  $a$ -priori to the statistician, but the number  $n$  chosen by the adversary is unknown. We refer to this set of  $m$  points as an  $(\alpha, \epsilon)$ -corrupted set of samples from  $D$ .
In our context, the notion of list-decoding is as follows: our algorithm will return a polynomially-sized list of matrices  $H_{i}$ , such that at least one  $H_{i}$  is an "approximate square root" of the true covariance  $\Sigma$  having bounded dimension-independent error  $\| H_i^{-1 / 2}\Sigma H_i^{-1 / 2} - I\|_{\mathrm{F}}$ . As discussed earlier, the bound we guarantee is poly  $(1 / \alpha)$ , which is in general larger than 1 and thus does not lead to non-trivial total variation bounds, thus circumventing related SQ lower bounds. Our main theorem is:
Theorem 1.2 (List-Decodable Covariance Estimation in Relative Frobenius Norm). Let  $C' > 0$  be a sufficiently large constant and  $\epsilon_0 > 0$  be a sufficiently small constant. Let the parameters  $\alpha \in (0,1/2)$ ,  $\epsilon \in (0,\epsilon_0)$ , and failure probability  $\delta \in (0,1/2)$  be known to the algorithm. Let  $D$  be the Gaussian distribution  $\mathcal{N}(\mu,\Sigma)$  with mean  $\mu \in \mathbb{R}^d$  and full-rank covariance  $\Sigma \in \mathbb{R}^{d\times d}$ . There is an  $\tilde{O}(m^2d^2)$ -time algorithm (Algorithm 1) such that, on input  $\alpha, \delta$  and an  $(\alpha,\epsilon)$ -corrupted set of  $m$  points from  $D$  (Definition 1.1) for any  $m > C' \frac{d^2\log^5(d / \alpha\delta)}{\alpha^6}$ , with probability at least  $1 - \delta$ , the algorithm returns a list of at most  $O(1 / \alpha)$  many sets  $T_i$  which are disjoint subsets of samples, each of size at least  $0.5\alpha m$ , and there exists a  $T_i$  in the output list such that:
- Recall the notation in the corruption model (Definition 1.1) where  $n$  is the size of the original inlier set  $S$  and  $\ell$  is the number of points in  $S$  that the adversary replaced— $n$  and  $\ell$  are unknown to the algorithm except that  $n \geq \alpha m$  and  $\ell \leq \epsilon n$ . The set  $T_i$  in the returned list satisfies that  $|T_i \cap S| \geq (1 - 0.01\alpha)(n - \ell)$ .
- Denote  $H_{i} \coloneqq \mathbf{E}_{X \sim T_{i}}[XX^{\top}]$ . The matrix  $H_{i}$  satisfies  $\| H_{i}^{-1/2}\Sigma H_{i}^{-1/2} - I\|_{\mathrm{F}} \lesssim \frac{1}{\alpha^{4}}\log \left(\frac{1}{\alpha}\right)$ .
Algorithm 1 is an iterative spectral algorithm, as opposed to involving large convex programs based on the sum-of-squares hierarchy. We state Algorithm 1 and give a high-level proof sketch of why it satisfies Theorem 1.2 in Section 3. The formal proof of Theorem 1.2 appears in Appendix F (where we allow  $\Sigma$  to be rank-deficient as well).
We also briefly remark that, if we wish to list-decode pairs of (mean, covariance), then we can run the following straightforward augmentation to Algorithm 1: after running Algorithm 1, for each  $\hat{\Sigma}$  in the output list, whiten the data using  $\hat{\Sigma}$  and run a standard list-decoding algorithm for the mean [CSV17; DKK20b]. These algorithms work if the data is distributed with a bounded covariance: by the guarantees of Theorem 1.2, after whitening, the data has covariance bounded by  $\mathrm{poly}(1 / \alpha)\cdot I$ .
As a corollary of our main result, the same algorithm (but with a slightly higher sample complexity) also achieves outlier-robust list-decoding of the covariances for the components of a Gaussian mixture model, in relative Frobenius norm. Definition 1.3 and Theorem 1.4 state the corresponding corruption model and theoretical guarantees on Algorithm 1.
Definition 1.3 (Corruption model for samples from Gaussian Mixtures). Let  $\epsilon \in (0,1/2)$ . Consider a Gaussian mixture model  $\sum_{p} \alpha_{p} \mathcal{N}(\mu_{p}, \Sigma_{p})$ , where the parameters  $\alpha_{p}, \mu_{p}$  and  $\Sigma_{p}$  are unknown and satisfy  $\alpha_{p} \geq \alpha$  for some known parameter  $\alpha$ . The statistician specifies the number of samples  $m$ , and  $m$  i.i.d. samples are drawn from the Gaussian mixture, which are called the inliers. The malicious and computationally unbounded adversary then inspects the  $m$  inliers and is allowed to replace an arbitrary subset of  $\epsilon \alpha m$  many inlier points with arbitrary outlier points, before giving the modified dataset to the statistician. We call this the  $\epsilon$ -corrupted set of samples.
Theorem 1.4 (Outlier-Robust Clustering and Estimation of Covariances for GMM). Let  $C' > 0$  be a sufficiently large constant and  $\epsilon_0 > 0$  be a sufficiently small constant. Let the parameters  $\alpha \in (0,1/2)$ ,  $\epsilon \in (0,\epsilon_0)$ , and failure probability  $\delta \in (0,1/2)$  be known. There is an  $\tilde{O}(m^2 d^2)$ -time algorithm such that, on input  $\alpha, \delta$ , and  $m > C' \frac{d^2 \log^5(d / \alpha \delta)}{\alpha^6}$  many  $\epsilon$ -corrupted samples from an unknown  $k$ -component Gaussian mixture  $\sum_{p=1}^{k} \alpha_p \mathcal{N}(\mu_p, \Sigma_p)$  over  $\mathbb{R}^d$  as in Definition 1.3, where all  $\Sigma_p$ 's are full-rank and all  $\alpha_p$  satisfies  $\alpha_p \geq \alpha$  and  $k \leq \frac{1}{\alpha}$  is unknown to the algorithm, with probability at least  $1 - \delta$  over the corrupted samples and the randomness of the algorithm, the algorithm returns a list of at most  $k$  many disjoint subsets of samples  $\{T_i\}$  such that:
- For the  $p^{\text{th}}$  Gaussian component, denote the set  $S_{p}$  as the samples in the inlier set  $S$  that were drawn from component  $p$ . Let  $n_{p}$  be the size of  $S_{p}$ , and let  $\ell_{p}$  be the number of points in  $S_{p}$  that the adversary replaced— $n_{p}$  and  $\ell_{p}$  are both unknown to the algorithm except that  $\mathbf{E}[n_{p}] = \alpha_{p} m \geq \alpha m$  for each  $p$  and  $\sum_{p} \ell_{p} \leq \epsilon \alpha m$ . Then, for every Gaussian component  $p$  in the mixture, there exists a set  $T_{i_{p}}$  in the returned list such that  $|T_{i_{p}} \cap S_{p}| \geq (1 - 0.01\alpha)(n_{p} - \ell_{p})$ .
- For every component  $p$ , there is a set of samples  $T_{i_p}$  in the returned list such that, defining  $H_{i_p} = \mathbf{E}_{X \sim T_{i_p}}[XX^\top]$ , we have  $H_{i_p}$  satisfying  $\| H_{i_p}^{-1/2}\Sigma_pH_{i_p}^{-1/2} - I\|_{\mathrm{F}} \lesssim (1/\alpha^4)\log(1/\alpha)$ .
- Let  $\Sigma$  be the (population-level) covariance matrix of the Gaussian mixture. For any two components  $p \neq p'$  with  $\|\Sigma^{-1/2}(\Sigma_p - \Sigma_{p'})\Sigma^{-1/2}\|_{\mathrm{F}} > C(1/\alpha)^5\log(1/\alpha)$  for a sufficiently large constant  $C$ , the sets  $T_{i_p}$  and  $T_{i_{p'}}$  from the previous bullet are guaranteed to be different.
The theorem states that, not only does Algorithm 1 achieve list-decoding of the Gaussian component covariances, but it also clusters samples according to separation of covariances in relative Frobenius norm. The recent result of [BDJKKV22] on robustly learning GMMs also involves an algorithmic component for clustering. Their approach is based on the sum-of-squares hierarchy (and thus requires solving large convex programs) while, Algorithm 1 is a purely spectral algorithm.
We also emphasize that the list size returned by Algorithm 1, in the case of a  $k$ -component Gaussian mixture model, is at most  $k$  — instead of a weaker result such as  $O(1 / \alpha)$  or polynomial/exponentially large in  $1 / \alpha$ . This is possible because Algorithm 1 keeps careful track of samples and makes sure that no more than a  $0.01\alpha$ -fraction of samples is removed from each component or mis-clustered into another component. We prove Theorem 1.4 in Appendix G.
1.2 Overview of Techniques
At a high level, our approach involves integrating robust covariance estimation techniques from [DKKLMS16] with the multifilter for list-decoding of [DKS18]. For a set  $S$ , we use  $SS^{\top}$  to denote the set  $\{xx^{\top}:x\in S\}$ . [DKKLMS16] states that in order to estimate the true covariance  $\Sigma$ , it suffices to find a large subset  $S$  of points with large overlap with the original set of good points, so that if  $\Sigma' = \mathbf{Cov}(S)$  then  $\mathbf{Cov}((\Sigma')^{-1/2}SS^{\top}(\Sigma')^{-1/2})$  has no large eigenvalues. This will imply that  $\| (\Sigma')^{-1/2}(\Sigma - \Sigma')(S')^{-1/2} \|_{\mathrm{F}}$  is not too large (Lemma E.2).
As in [DKKLMS16], our basic approach for finding such a set  $S$  is by the iteratively repeating the following procedure: We begin by taking  $S$  to be the set of all samples and repeatedly check whether or not  $\mathbf{Cov}((\Sigma')^{-1/2} SS^{\top} (\Sigma')^{-1/2})$  has any large eigenvalues. If it does not, we are done. If it has a large eigenvalue corresponding to a matrix  $A$  (normalized to have unit Frobenius norm), we consider the values of  $f(x) = \langle A, (\Sigma')^{-1/2} xx^{\top} (\Sigma')^{-1/2} \rangle$ , for  $x$  in  $S$ , and attempt to use them to find outliers. Unfortunately, as the inliers might comprise a minority of the sample, the values we get out of this formula might end-up in several reasonably large clusters, any one of which could plausibly contain the true samples; thus, not allowing us to declare any particular points to be outliers
with any degree of certainty. We resolve this issue by using the multifilter approach of [DKS18]—we either (i) iteratively remove outliers, or (ii) partition the data into clusters and recurse on each cluster. In particular, we note that if there is large variance in the  $A$ -direction, one of two things must happen, either: (i) The substantial majority of the values of  $f(x)$  lie in a single cluster, with some extreme outliers. In this case, we can be confident that the extreme outliers are actual errors and remove them (Section 3.2). (ii) There are at least two clusters of values of  $f(x)$  that are far apart from each other. In this case, instead of simply removing obvious outliers, we replace  $S$  by two subsets  $S_{1}$  and  $S_{2}$  with the guarantee that at least one of the  $S_{i}$  contains almost all of the inliers. Naively, this can be done by finding a value  $y$  between the two clusters so that very few samples have  $f(x)$  close to  $y$ , and letting  $S_{1}$  be the set of points with  $f(x) < y$  and  $S_{2}$  the set of points with  $f(x) > y$  (Section 3.3). In either case, we will have cleaned up our set of samples and can recurse on each of the returned subsets of  $S$ . Iterating this technique recursively on all of the smaller subsets returned ensures that there is always at least one subset containing the majority of the inliers, and that eventually once it stops having too large of a covariance, we will return an appropriate approximation to  $\Sigma$ .
We want to highlight the main point of difference where our techniques differ notably from [DKS18]. In order to implement the algorithm outlined above, one needs to have good a priori bounds for what the variance of  $f(x)$  over the inliers ought to be. Since  $f(\cdot)$  is a quadratic polynomial, the variance of  $f$  over the inliers, itself depends on the covariance  $\Sigma$ , which is exactly what we are trying to estimate. This challenge of circular dependence does not appear in [DKS18]: their goal was to estimate the unknown mean of an identity-covariance Gaussian, and thus it sufficed to use a linear polynomial  $f$  (instead of a quadratic polynomial). Importantly, the covariance of a linear polynomial does not depend on the (unknown) mean (it depends only on the covariance, which was known in their setting). In order to overcome this challenge, we observe that if  $S$  contains most of the inliers, then the covariance of  $S$  cannot be too much smaller than the true covariance  $\Sigma$ . This allows us to find an upper bound on  $\Sigma$ , which in turn lets us upper bound the variance of  $f(x)$  over the good samples (Lemma 3.2).
Related Work We refer the reader to [DK23] for an overview of algorithmic robust statistics. We mention the most relevant related work here and discuss additional related work in Appendix A. Algorithms for list-decodable covariance estimation were developed in the special cases of subspace estimation and linear regression in [KKK19; BK21; RY20b; RY20a; DJKS22]. On the other hand, [DKS17; DKPPS21] present SQ lower bounds for learning Gaussian mixture models and list-decodable linear regression (and thus list-decodable covariance estimation), respectively.