Title: A Spectral Algorithm for List-Decodable Covariance Estimation in Relative Frobenius Norm
Venue: NeurIPS 2023
Accepted: yes
Keywords: robust statistics

covariance estimation

list-decodable learning
Abstract: We study the problem of list-decodable Gaussian covariance estimation. Given a multiset $T$ of $n$ points in $\mathbb{R}^d$ such that an unknown $\alpha<1/2$ fraction of points in $T$ are i.i.d. samples from an unknown Gaussian $\mathcal{N}(\mu, \Sigma)$, the goal is to output a list of $O(1/\alpha)$ hypotheses at least one of which is close to $\Sigma$ in relative Frobenius norm. Our main result is a $\mathrm{poly}(d,1/\alpha)$ sample and time algorithm for this task that guarantees relative Frobenius norm error of $\mathrm{poly}(1/\alpha)$. Importantly, our algorithm relies purely on spectral techniques. As a corollary, we obtain an efficient spectral algorithm for robust partial clustering of Gaussian mixture models (GMMs) --- a key ingredient in the recent work of [BakDJKKV22] on robustly learning arbitrary GMMs. Combined with the other components of [BakDJKKV22], our new method yields the first Sum-of-Squares-free algorithm for robustly learning GMMs, resolving an open problem proposed by Vempala and Kothari. At the technical level, we develop a novel multi-filtering method for list-decodable covariance estimation that may be useful in other settings.

A Spectral Algorithm for List-Decodable Covariance Estimation in Relative Frobenius Norm

Abstract

We study the problem of list-decodable Gaussian covariance estimation. Given a multiset T of n points in R d such that an unknown α < 1 / 2 fraction of points in T are i.i.d. samples from an unknown Gaussian N ( µ, Σ) , the goal is to output a list of O (1 /α ) hypotheses at least one of which is close to Σ in relative Frobenius norm. Our main result is a poly( d, 1 /α ) sample and time algorithm for this task that guarantees relative Frobenius norm error of poly(1 /α ) . Importantly, our algorithm relies purely on spectral techniques. As a corollary, we obtain an efficient spectral algorithm for robust partial clustering of Gaussian mixture models (GMMs) - a key ingredient in the recent work of [BDJKKV22] on robustly learning arbitrary GMMs. Combined with the other components of [BDJKKV22], our new method yields the first Sum-of-Squares-free algorithm for robustly learning GMMs, resolving an open problem proposed by Vempala [Vem22] and Kothari [Kot21]. At the technical level, we develop a novel multi-filtering method for list-decodable covariance estimation that may be useful in other settings.

1 Introduction

Robust statistics studies the efficient (parameter) learnability of an underlying distribution given samples some fraction of which might be corrupted, perhaps arbitrarily. While the statistical theory of these problems has been well-established for some time [Hub64; HR09], only recently has the algorithmic theory of such estimation problems begun to be understood [DKKLMS16; LRV16; DK23].

A classical problem in robust estimation is that of multivariate robust mean estimation - that is, estimating the mean of an unknown distribution in the presence of a small constant fraction of outliers. One of the original results in the field is that given samples from a Gaussian N ( µ, I ) with an ϵ -fraction of outliers (for some ϵ < 1 / 2 ), the unknown mean µ can be efficiently estimated to an error of O ( ϵ √ log 1 /ϵ ) in the ℓ 2 -norm [DKKLMS16]. Note that the use of the ℓ 2 -norm here is quite natural, as the total variation distance between two identity-covariance Gaussians is roughly proportional to the ℓ 2 -distance between their means; thus, this estimator learns the underlying distribution to total variation distance error O ( ϵ √ log 1 /ϵ ) . This bound cannot be substantially improved, as learning to error o ( ϵ ) in total variation distance is information-theoretically impossible.

While the above algorithmic result works when ϵ is small (less than 1 / 2 ), if more than half of the samples are corrupted, it becomes impossible to learn with only a single returned hypothesis-the corruption might simply simulate other Gaussians, and no algorithm can identify which Gaussian is the original one. This issue can be circumvented using the list-decodable mean estimation

Thanasis Pittas

paradigm [CSV17], where the algorithm is allowed to output a small list of hypotheses with the guarantee that at least one of them is relatively close to the target. List-decodable learning is closely related to semi-verified learning [CSV17; MV18], where a learner can choose to audit a small amount of data. The framework has been shown to capture a multitude of applications, including crowdsourcing [SVC16; MV18], semi-random community detection in stochastic block models [CSV17] and clustering (as we also show in this work). For Gaussian mean estimation in particular, if an α -fraction of the samples are clean (i.e., uncorrupted), for some α < 1 / 2 , there exist polynomial-time algorithms that return a list of O (1 /α ) hypotheses such that (with high probability) at least one of them is within ˜ O ( √ 1 /α ) of the mean in ℓ 2 -distance [CSV17]. Note that while this ℓ 2 -norm distance bound does not imply a good bound on the total variation distance between the true distribution and the learned hypothesis, it does bound their distance away from one, ensuring some non-trivial overlap.

Another important, and arguably more complex, problem in robust statistics (the focus of this work) is that of robustly estimating the covariance of a multivariate Gaussian. It was shown in [DKKLMS16] that given ϵ -corrupted samples from N (0 , Σ) (for ϵ < 1 / 2 ) there is a polynomial-time algorithm for estimating Σ to error O ( ϵ log 1 /ϵ ) in the relative Frobenius norm, i.e., outputting a hypothesis covariance ˜ Σ satisfying ∥ ˜ Σ -1 / 2 Σ ˜ Σ -1 / 2 -I ∥ F ≤ ϵ . This is again the relevant metric, since, if ∥ ˜ Σ -1 / 2 Σ ˜ Σ -1 / 2 -I ∥ F ≤ 2 / 3 , then the total variation distance between N (0 , Σ) and N (0 , ˜ Σ) is proportional to the relative Frobenius norm ∥ ˜ Σ -1 / 2 Σ ˜ Σ -1 / 2 -I ∥ F [DMR18].

A natural goal, with a number of applications, is to extend the above algorithmic result to the listdecodable setting. That is, one would like a polynomial-time algorithm that given corrupted samples from N (0 , Σ) (with an α -fraction of clean samples, for some α < 1 / 2 ), returns a list of O (1 /α ) many hypotheses with the guarantee that at least one of them has some non-trivial overlap with the true distribution in total variation distance. We start by noting that the sample complexity of list-decodable covariance estimation is poly( d/α ) , albeit via an exponential time algorithm. The only known algorithm for list-decodable covariance estimation (with total variation error guarantees) is due to [IK22]. This algorithm essentially relies on the Sum-of-Squares method and has (sample and computational) complexity d poly(1 /α ) . Intriguingly, there is compelling evidence that this complexity bound cannot be improved. Specifically, [DKS17; DKPPS21] showed that any Statistical Query (SQ) algorithm for this task requires complexity d poly(1 /α ) . Combined with the reduction of [BBHLT20], this implies a similar lower bound for low-degree polynomial tests. These lower bounds suggest an intrinsic information-computation gap for the problem.

The aforementioned lower bound results [DKS17; DKPPS21] establish that it is SQ (and low-degree) hard to distinguish between a standard multivariate Gaussian, N (0 , I ) , and a distribution P that behaves like N (0 , I ) (in terms of low-degree moments) but P contains an α -fraction of samples from a Gaussian N (0 , Σ) , where Σ is very thin in some hidden direction v . Since the distribution N (0 , Σ) is very thin along v , i.e., has very small variance, the obvious choice of N (0 , I ) essentially has no overlap with N (0 , Σ) -making this kind of strong list-decoding guarantee (closeness in total variation distance) likely computationally intractable.

Interestingly, there are two possible ways that a pair of mean zero Gaussians can be separated [DHKK20; BDHKKK20]: (1) one could be much thinner than the other in some direction, or (2) they could have many orthogonal directions in which their variances differ, adding up to something more substantial. While the lower bounds of [DKS17; DKPPS21] seem to rule out being able to detect deviations of the former type in fully polynomial time (i.e., poly( d/α ) ), it does not rule out efficiently detecting deviations of the latter. In particular, we could hope to find (in poly( d/α ) time) a good hypothesis ˜ Σ such that ∥ ˜ Σ -1 / 2 Σ ˜ Σ -1 / 2 -I ∥ F is not too big. While this does not exclude the possibility that Σ is much thinner than ˜ Σ in a small number of independent directions, it does rule out the second kind of difference between the two. The main goal of this paper is to provide an elementary (relying only on spectral techniques) list-decoding algorithm of this form. Given corrupted samples from a Gaussian N (0 , Σ) , we give a poly( d/α ) -time algorithm that returns a small list of hypotheses ˜ Σ such that for at least one of them we have that ∥ ˜ Σ -1 / 2 Σ ˜ Σ -1 / 2 -I ∥ F < poly(1 /α ) .

In addition to providing the best qualitative guarantee we could hope to achieve in fully polynomial time, the above kind of 'weak' list-decoding algorithm has interesting implications for the wellstudied problem of robustly learning Gaussian mixture models (GMMs). [BDJKKV22] gave a polynomial-time algorithm to robustly learn arbitrary mixtures of Gaussians (with a constant number

of components). One of the two main ingredients of their approach is a subroutine that could perform partial clustering of points into components that satisfy exactly this kind of weak closeness guarantee. [BDJKKV22] developed such a subroutine by making essential use of a Sum-of-Squares (SoS) relaxation, for the setting that the samples come from a mildly corrupted mixture of Gaussians (with a small constant fraction of outliers). As a corollary of our techniques, we obtain an elementary spectral algorithm for this partial clustering task (and, as already stated, our results work in the more general list-decoding setting). This yields the first SoS-free algorithm for robustly learning GMMs, answering an open problem in the literature [Vem22; Kot21].

1.1 Our Results

Our main result is a polynomial-time algorithm for list-decoding the covariance of a Gaussian in relative Frobenius norm, under adversarial corruption where more than half the samples could be outliers. Definition 1.1 makes precise the corruption model, and Theorem 1.2 states the guarantees of our main algorithm (Algorithm 1).

Definition 1.1 (Corruption model for list-decoding) . Let the parameters ϵ, α ∈ (0 , 1 / 2) and a distribution family D . The statistician specifies the number of samples m . Then a set of n ≥ αm i.i.d. points are sampled from an unknown D ∈ D . We call these n samples the inliers . Upon inspecting the n inliers, a (malicious and computationally unbounded) adversary can replace an arbitrary ℓ ≤ ϵn of the inliers with arbitrary points, and further add m -n arbitrary points to the dataset, before returning the entire set of m points to the statistician. The parameter α is also known a-priori to the statistician, but the number n chosen by the adversary is unknown. We refer to this set of m points as an ( α, ϵ ) -corrupted set of samples from D .

In our context, the notion of list-decoding is as follows: our algorithm will return a polynomially-sized list of matrices H i , such that at least one H i is an 'approximate square root' of the true covariance Σ having bounded dimension-independent error ∥ H -1 / 2 i Σ H -1 / 2 i -I ∥ F. As discussed earlier, the bound we guarantee is poly(1 /α ) , which is in general larger than 1 and thus does not lead to non-trivial total variation bounds, thus circumventing related SQ lower bounds. Our main theorem is:

Theorem 1.2 (List-Decodable Covariance Estimation in Relative Frobenius Norm) . Let C ′ > 0 be a sufficiently large constant and ϵ 0 > 0 be a sufficiently small constant. Let the parameters α ∈ (0 , 1 / 2) , ϵ ∈ (0 , ϵ 0 ) , and failure probability δ ∈ (0 , 1 / 2) be known to the algorithm. Let D be the Gaussian distribution N ( µ, Σ) with mean µ ∈ R d and full-rank covariance Σ ∈ R d × d . There is an ˜ O ( m 2 d 2 ) -time algorithm (Algorithm 1) such that, on input α, δ and an ( α, ϵ ) -corrupted set of m points from D (Definition 1.1) for any m > C ′ d 2 log 5 ( d/αδ ) α 6 , with probability at least 1 -δ , the algorithm returns a list of at most O (1 /α ) many sets T i which are disjoint subsets of samples, each of size at least 0 . 5 αm , and there exists a T i in the output list such that:

Recall the notation in the corruption model (Definition 1.1) where n is the size of the original inlier set S and ℓ is the number of points in S that the adversary replacedn and ℓ are unknown to the algorithm except that n ≥ αm and ℓ ≤ ϵn . The set T i in the returned list satisfies that | T i ∩ S | ≥ (1 -0 . 01 α )( n -ℓ ) .

Denote H i := E X ∼ T i [ XX ⊤ ] . The matrix H i satisfies ∥ H -1 / 2 i Σ H -1 / 2 i -I ∥ F ≲ 1 α 4 log( 1 α ) .

Algorithm 1 is an iterative spectral algorithm, as opposed to involving large convex programs based on the sum-of-squares hierarchy. We state Algorithm 1 and give a high-level proof sketch of why it satisfies Theorem 1.2 in Section 3. The formal proof of Theorem 1.2 appears in Appendix F (where we allow Σ to be rank-deficient as well).

We also briefly remark that, if we wish to list-decode pairs of (mean, covariance), then we can run the following straightforward augmentation to Algorithm 1: after running Algorithm 1, for each ˆ Σ in the output list, whiten the data using ˆ Σ and run a standard list-decoding algorithm for the mean [CSV17; DKK20b]. These algorithms work if the data is distributed with a bounded covariance: by the guarantees of Theorem 1.2, after whitening, the data has covariance bounded by poly(1 /α ) · I .

As a corollary of our main result, the same algorithm (but with a slightly higher sample complexity) also achieves outlier-robust list-decoding of the covariances for the components of a Gaussian mixture model, in relative Frobenius norm. Definition 1.3 and Theorem 1.4 state the corresponding corruption model and theoretical guarantees on Algorithm 1.

Definition 1.3 (Corruption model for samples from Gaussian Mixtures) . Let ϵ ∈ (0 , 1 / 2) . Consider a Gaussian mixture model ∑ p α p N ( µ p , Σ p ) , where the parameters α p , µ p and Σ p are unknown and satisfy α p ≥ α for some known parameter α . The statistician specifies the number of samples m , and m i.i.d. samples are drawn from the Gaussian mixture, which are called the inliers . The malicious and computationally unbounded adversary then inspects the m inliers and is allowed to replace an arbitrary subset of ϵαm many inlier points with arbitrary outlier points, before giving the modified dataset to the statistician. We call this the ϵ -corrupted set of samples.

Theorem 1.4 (Outlier-Robust Clustering and Estimation of Covariances for GMM) . Let C ′ > 0 be a sufficiently large constant and ϵ 0 > 0 be a sufficiently small constant. Let the parameters α ∈ (0 , 1 / 2) , ϵ ∈ (0 , ϵ 0 ) , and failure probability δ ∈ (0 , 1 / 2) be known. There is an ˜ O ( m 2 d 2 ) -time algorithm such that, on input α, δ , and m > C ′ d 2 log 5 ( d/αδ ) α 6 many ϵ -corrupted samples from an unknown k -component Gaussian mixture ∑ k p =1 α p N ( µ p , Σ p ) over R d as in Definition 1.3, where all Σ p 's are full-rank and all α p satisfies α p ≥ α and k ≤ 1 α is unknown to the algorithm, with probability at least 1 -δ over the corrupted samples and the randomness of the algorithm, the algorithm returns a list of at most k many disjoint subsets of samples { T i } such that:

For the p th Gaussian component, denote the set S p as the samples in the inlier set S that were drawn from component p . Let n p be the size of S p , and let ℓ p be the number of points in S p that the adversary replacedn p and ℓ p are both unknown to the algorithm except that E [ n p ] = α p m ≥ αm for each p and ∑ p ℓ p ≤ ϵαm . Then, for every Gaussian component p in the mixture, there exists a set T i p in the returned list such that | T i p ∩ S p | ≥ (1 -0 . 01 α )( n p -ℓ p ) .

For every component p , there is a set of samples T i p in the returned list such that, defining H i p = E X ∼ T i p [ XX ⊤ ] , we have H i p satisfying ∥ H -1 / 2 i p Σ p H -1 / 2 i p -I ∥ F ≲ (1 /α 4 ) log(1 /α ) .

̸

Let Σ be the (population-level) covariance matrix of the Gaussian mixture. For any two components p = p ′ with ∥ Σ -1 / 2 (Σ p -Σ p ′ )Σ -1 / 2 ∥ F > C (1 /α ) 5 log(1 /α ) for a sufficiently large constant C , the sets T i p and T i p ′ from the previous bullet are guaranteed to be different.

The theorem states that, not only does Algorithm 1 achieve list-decoding of the Gaussian component covariances, but it also clusters samples according to separation of covariances in relative Frobenius norm. The recent result of [BDJKKV22] on robustly learning GMMs also involves an algorithmic component for clustering. Their approach is based on the sum-of-squares hierarchy (and thus requires solving large convex programs) while, Algorithm 1 is a purely spectral algorithm.

We also emphasize that the list size returned by Algorithm 1, in the case of a k -component Gaussian mixture model, is at most k -instead of a weaker result such as O (1 /α ) or polynomial/exponentially large in 1 /α . This is possible because Algorithm 1 keeps careful track of samples and makes sure that no more than a 0 . 01 α -fraction of samples is removed from each component or mis-clustered into another component. We prove Theorem 1.4 in Appendix G.

1.2 Overview of Techniques

At a high level, our approach involves integrating robust covariance estimation techniques from [DKKLMS16] with the multifilter for list-decoding of [DKS18]. For a set S , we use SS ⊤ to denote the set { xx ⊤ : x ∈ S } . [DKKLMS16] states that in order to estimate the true covariance Σ , it suffices to find a large subset S of points with large overlap with the original set of good points, so that if Σ ′ = Cov ( S ) then Cov ((Σ ′ ) -1 / 2 SS ⊤ (Σ ′ ) -1 / 2 ) has no large eigenvalues. This will imply that ∥ (Σ ′ ) -1 / 2 (Σ -Σ ′ )(Σ ′ ) -1 / 2 ∥ F is not too large (Lemma E.2).

As in [DKKLMS16], our basic approach for finding such a set S is by the iteratively repeating the following procedure: We begin by taking S to be the set of all samples and repeatedly check whether or not Cov ((Σ ′ ) -1 / 2 SS ⊤ (Σ ′ ) -1 / 2 ) has any large eigenvalues. If it does not, we are done. If it has a large eigenvalue corresponding to a matrix A (normalized to have unit Frobenius norm), we consider the values of f ( x ) = ⟨ A, (Σ ′ ) -1 / 2 xx ⊤ (Σ ′ ) -1 / 2 ⟩ , for x in S , and attempt to use them to find outliers. Unfortunately, as the inliers might comprise a minority of the sample, the values we get out of this formula might end-up in several reasonably large clusters, any one of which could plausibly contain the true samples; thus, not allowing us to declare any particular points to be outliers

with any degree of certainty. We resolve this issue by using the multifilter approach of [DKS18]-we either (i) iteratively remove outliers, or (ii) partition the data into clusters and recurse on each cluster. In particular, we note that if there is large variance in the A -direction, one of two things must happen, either: (i) The substantial majority of the values of f ( x ) lie in a single cluster, with some extreme outliers. In this case, we can be confident that the extreme outliers are actual errors and remove them (Section 3.2). (ii) There are at least two clusters of values of f ( x ) that are far apart from each other. In this case, instead of simply removing obvious outliers, we replace S by two subsets S 1 and S 2 with the guarantee that at least one of the S i contains almost all of the inliers. Naïvely, this can be done by finding a value y between the two clusters so that very few samples have f ( x ) close to y , and letting S 1 be the set of points with f ( x ) < y and S 2 the set of points with f ( x ) > y (Section 3.3). In either case, we will have cleaned up our set of samples and can recurse on each of the returned subsets of S . Iterating this technique recursively on all of the smaller subsets returned ensures that there is always at least one subset containing the majority of the inliers, and that eventually once it stops having too large of a covariance, we will return an appropriate approximation to Σ .

We want to highlight the main point of difference where our techniques differ notably from [DKS18]. In order to implement the algorithm outlined above, one needs to have good a priori bounds for what the variance of f ( x ) over the inliers ought to be. Since f ( · ) is a quadratic polynomial, the variance of f over the inliers, itself depends on the covariance Σ , which is exactly what we are trying to estimate. This challenge of circular dependence does not appear in [DKS18]: their goal was to estimate the unknown mean of an identity-covariance Gaussian, and thus it sufficed to use a linear polynomial f (instead of a quadratic polynomial). Importantly, the covariance of a linear polynomial does not depend on the (unknown) mean (it depends only on the covariance, which was known in their setting). In order to overcome this challenge, we observe that if S contains most of the inliers, then the covariance of S cannot be too much smaller than the true covariance Σ . This allows us to find an upper bound on Σ , which in turn lets us upper bound the variance of f ( x ) over the good samples (Lemma 3.2).

Related Work We refer the reader to [DK23] for an overview of algorithmic robust statistics. We mention the most relevant related work here and discuss additional related work in Appendix A. Algorithms for list-decodable covariance estimation were developed in the special cases of subspace estimation and linear regression in [KKK19; BK21; RY20b; RY20a; DJKS22]. On the other hand, [DKS17; DKPPS21] present SQ lower bounds for learning Gaussian mixture models and list-decodable linear regression (and thus list-decodable covariance estimation), respectively.

[IK22] gave the first algorithm for general list-decodable covariance estimation that achieves nontrivial bounds in total variation distance using the powerful sum-of-squares hierarchy. Their algorithm outputs an exp(poly(1 /α )) -sized list of matrices containing an H i that is close to the true Σ in two metrics (i) relative Frobenius norm: ∥ H -1 / 2 i Σ H -1 / 2 i -I ∥ F = poly(1 /α ) and (ii) multiplicative spectral approximation: poly( α )Σ ⪯ H i ⪯ poly(1 /α )Σ . Their algorithm uses d poly(1 /α ) samples, which seems to be necessary for efficient (statistical query) algorithms achieving multiplicative spectral approximation [DKS17; DKPPS21]. In comparison, Theorem 1.2 uses only poly( d/α ) samples, returns a list of O (1 /α ) matrices, but approximates only in the relative Frobenius norm: ∥ H -1 / 2 i Σ H -1 / 2 i -I ∥ F = poly(1 /α ) .

2 Preliminaries

For a vector v , we let ∥ v ∥ 2 denote its ℓ 2 -norm. We use I d to denote the d × d identity matrix; We will drop the subscript when it is clear from the context. For a matrix A , we use ∥ A ∥ F and ∥ A ∥ op to denote the Frobenius and spectral (or operator) norms, respectively. We denote by ⟨ v, u ⟩ , the standard inner product between the vectors u, v . For matrices U, V ∈ R d × d , we use ⟨ U, V ⟩ to denote the trace inner product ∑ ij U ij V ij . For a matrix A ∈ R d × d , we use A ♭ to denote the flattened vector in R d 2 , and for a v ∈ R d 2 , we use v ♯ to denote the unique matrix A such that A ♭ = v ♯ . For a matrix A , we let A † denote its pseudo-inverse. We use ⊗ to denote the Kronecker product. For a matrix A , we use ker( A ) for the null space of A . We use X ∼ D to denote that a random variable X is distributed according to the distribution D . We use N ( µ, Σ) for the Gaussian distribution with mean µ and covariance matrix Σ . For a set S , we use X ∼ S to denote that X is distributed uniformly at random from S . We use a ≲ b to denote that there exists an absolute universal constant C > 0 (independent of the variables or parameters on which a and b depend) such that a ≤ Cb .

2.1 Deterministic Conditions

Our algorithm will rely on the uncorrupted inliers satisfying a set of properties, similar to the 'stability conditions" from [DKKLMS16]. Intuitively, these are are concentration properties for sets of samples, but with the added requirement that every large subset of the samples also satisfies these properties.

Definition 2.1 ( ( η, ϵ ) -Stable Set) . Let D be a distribution with mean µ and covariance Σ . We say a set of points A ⊂ R d is ( η, ϵ ) -stable with respect to D , if for any subset A ′ ⊆ A with | A ′ | ≥ (1 -ϵ ) | A | , the following hold: for every v ∈ R d , symmetric U ∈ R d × d , and every even degree2 polynomial p :

(L.4) Var X ∼ A ′ [ p ( X )] ≤ 4 Var X ∼ D [ p ( X )] .

(L.5) The null space of second moment matrix of A ′ is contained in the null space of Σ , i.e., ker (∑ x ∈ A ′ xx ⊤ ) ⊆ ker(Σ) .

A Gaussian dataset is 'stable' with high probability [DKKLMS16]; formally, we have Lemma 2.2, proved in Appendix D.1. Moreover, Lemma 2.2 can be extended to a variety of distributions (cf. Remark D.1).

Lemma 2.2 (Deterministic Conditions Hold with High Probability) . For a sufficiently small positive constant ϵ 0 and a sufficiently large absolute constant C , a set of m>Cd 2 log 5 ( d/ ( ηδ )) /η 2 samples from N ( µ, Σ) , with probability 1 -δ , is ( η, ϵ ) -stable set with respect to µ, Σ for all ϵ ≤ ϵ 0 .

3 Analysis of a Single Recursive Call of the Algorithm

Our algorithm, Algorithm 1, filters and splits samples into multiple sets recursively, until we can certify that the empirical second moment matrix of the 'current data set" is suitable to be included in the returned list of covariances. As a reference point, we define the notations and assumptions necessary to analyze each recursive call below. However, before moving to the formal analysis we will first give an informal overview of the algorithm's steps and the high-level ideas behind them.

Assumption 3.1 (Assumptions and notations for a single recursive call of the algorithm) .

S = { x i } n i =1 is a set of n uncontaminated samples, which is assumed to be ( η, 2 ϵ 0 ) -stable with respect to the inlier distribution D having mean and covariance µ, Σ (c.f. Definition 2.1). We assume η ≤ 0 . 001 , ϵ 0 = 0 . 01 , and Var X ∼ D [ X ⊤ AX ] ≤ C 1 ( ∥ Σ 1 / 2 A Σ 1 / 2 ∥ 2 F + ∥ Σ 1 / 2 Aµ ∥ 2 2 ) for all symmetric d × d matrices A and a constant C 1 .

T is the input set to the current recursive call of the algorithm (after the adversarial corruptions), which satisfies | S ∩ T | ≥ (1 -2 ϵ 0 ) | S | and | T | ≤ (1 /α ) | S | .

We denote H = E X ∼ T [ XX ⊤ ] .

We denote by ˜ S, ˜ T the versions of S and T normalized by H † / 2 : ˜ S = { H † / 2 x : x ∈ S } and ˜ T = { H † / 2 x : x ∈ T } . We use the notation ˜ x for elements in ˜ S and ˜ T , and x for elements in S and T . Similarly, we use the notation ˜ X for random variables with support in ˜ S or ˜ T .

The mean and covariance of the inlier distribution D after transformation with H † / 2 are denoted by ˜ µ := H † / 2 µ , ˜ Σ := H † / 2 Σ H † / 2 . We denote the empirical mean and covariance of the transformed inliers in T by ˆ µ := E ˜ X ∼ ˜ S ∩ ˜ T [ ˜ X ] and ˆ Σ := Cov ˜ X ∼ ˜ S ∩ ˜ T [ ˜ X ] .

Much of the algorithm uses the fact that, for a Gaussian, even quadratic polynomials have a small variance. We will leverage this for filtering and clustering samples. See Appendix E for the proof.

Lemma 3.2. Make Assumption 3.1 and recall that H = E X ∼ T [ XX ⊤ ] , where T is the corrupted version of a stable inlier set S . For every symmetric matrix A with ∥ A ∥ F = 1 , we have that Var X ∼ D [( H † / 2 X ) ⊤ A ( H † / 2 X )] ≤ 18 C 1 /α 2 .

Armed with Lemma 3.2, we can now give a high-level overview of a recursive call of Algorithm 1:

In our notation, we call the current data set T . Denoting H = E X ∼ T [ XX ⊤ ] for its the empirical second moment matrix, we construct the normalized data set ˜ T = { H -1 / 2 x : x ∈ T } . The normalization allows us to bound the covariance Σ in terms of H .

Since we are trying to estimate a covariance, consider the vectors ˜ s = { (˜ x ˜ x ⊤ ) ♭ : ˜ x ∈ ˜ T } , which are the second moment matrices of each data point flattened into vectors.

The first step is standard in filtering-based outlier-robust estimation: we test whether the covariance of the ˜ s vectors is small. If so, we are able to prove that the current H is a good approximate square root of Σ (c.f. Section 3.1) hence we just return H .

If the first test fails, that would imply that the empirical covariance of the ˜ s vectors is large in some direction. We want to leverage this direction to make progress, either by removing outliers through filtering or by bi-partitioning our samples into two clear clusters.

To decide between the 2 options, consider projecting the ˜ s vectors onto their largest variance direction. Specifically, let A be the matrix lifted from the largest eigenvector of the covariance of the ˜ s vectors. Define the vectors ˜ y = ˜ x ⊤ A ˜ x = ⟨ ˜ x ˜ x ⊤ , A ⟩ for ˜ x ∈ ˜ T , corresponding to the 1-dimensional projection of ˜ s onto the A ♭ direction. Since we have failed the first test, these ˜ y elements must have a large variance. We will decide to filter or divide our samples, based on whether the αm -smallest and αm -largest elements of the ˜ y s are close to each other.

If they are close, yet we have large variance, we will use this information to design a score function and perform filtering that removes a random sample with probability proportional to its score. We will then go back to Step 1. This would work because by Lemma 3.2 and by stability (Definition 2.1), the (unfiltered) inliers have a small empirical variance within themselves, meaning that the large total empirical variance is mostly due to the outlier.

Ideally, the score of a sample would be (proportional to) the squared distance between the sample and the mean of the inliers-the total inlier score would then be equal to the inlier variance. However, since we do not know which points are the inliers, we instead use the median of all the projected samples as a proxy for the unknown inlier mean. We show that the distance between the αm -smallest and largest ˜ y s bounds the difference between the ideal and proxy scores.

Otherwise, αm -smallest and αm -largest elements of the ˜ y s are far apart. By the stability condition Definition 2.1 (specifically, Condition (L.3)), most of the inliers must be close to each other under this 1-dimensional projection. Therefore, the large quantile range necessarily means there is a threshold under this projection to divide the samples into two sets, such that each set has at least αm points and most of the inliers are kept within a single set.

The score function mentioned in Step 6 upper bounds the maximum variance we check in Step 3. For simplicity, in the actual algorithm (Algorithm 1) we use the score directly for the termination check instead of checking the covariance, but it does not matter technically which quantity we use.

Remark 3.3 (Runtime of Algorithm 1) . We claim that each 'loop' in Algorithm 1 takes ˜ O ( md 2 ) time to compute. The number of times we run the 'loop' is at most O ( m ) , since each loop either ends in termination, removes 1 element from the dataset, or splits the dataset, all of which can happen at most O ( m ) times. From this, we can conclude a runtime of ˜ O ( m 2 d 2 ) . The sample complexity of our algorithm is also explicitly calculable to be ˜ O ( d 2 /α 6 ) , which follows from Lemma 2.2 and the choice of parameter η = Θ( α 3 ) from Theorem F.1 (the formal version of Theorem 1.2).

To see the runtime of a single loop: the most expensive operations in each loop are to compute H t , its pseudo-inverse, and to compute the symmetric matrix A in Line 7 that is the top eigenvector of a d 2 × d 2 matrix. Computing H t trivially takes O ( md 2 ) time, resulting in a d × d matrix. Its pseudoinverse can be computed in O ( d ω ) time, which is dominated by O ( md 2 ) since m ≳ d . Lastly, we observe that, instead of actually computing the top eigenvector in Line 7 to yield the matrix A , it suffices in our analysis to compute a matrix B whose Rayleigh quotient ( B ♭ ) ⊤ ( Cov ˜ X ∼ ˜ T t [ ˜ X ⊗ 2 ] ) B ♭ / (( B ♭ ) ⊤ B ♭ )

Algorithm 1 List-Decodable Covariance Estimation in Relative Frobenius Norm

power iterations. Since

we can compute each matrix-vector product in O ( | T t | d 2 ) ≤ O ( md 2 ) time, thus yielding an ˜ O ( m 2 d 2 ) runtime for the power iteration.

3.1 Certificate Lemma: Bounded Fourth Moment Implies Closeness

The first component of the analysis is our certificate lemma, which states that, if the empirical covariance of the (flattened) second moment matrices of current data set (after normalization) is bounded, then the empirical second moment matrix H of the current data set is a good approximation to the covariance of the Gaussian component we want to estimate.

Lemma 3.4 (Case when we stop and return) . Make Assumption 3.1. Let w ∈ R d 2 be the leading eigenvector of the Cov ˜ X ∼ ˜ T [ ˜ X ⊗ 2 ] with ∥ w ∥ 2 = 1 , and let A ∈ R d × d be w ♯ . Note that ∥ w ∥ 2 = 1 implies ∥ A ∥ F = 1 . Then, we have ∥ ∥ H † / 2 Σ H † / 2 -I ∥ ∥ 2 F ≲ (1 /α ) Var ˜ X ∼ ˜ T [ ˜ X ⊤ A ˜ X ] + 1 /α 2 .

See Appendix E.2 for the proof. Our termination check of Line 12 uses the score f ( ˜ x ) = ( ˜ x ⊤ A ˜ x -˜ y median ) 2 where ˜ y median is the median of { ˜ x ⊤ A ˜ x : ˜ x ∈ ˜ T } . Since Var ˜ X ∼ ˜ T [ ˜ X ⊤ A ˜ X ] ≤ E ˜ X ∼ ˜ T [ f ( ˜ X )] our check ensures that Var ˜ X ∼ ˜ T [ ˜ X ⊤ A ˜ X ] ≤ poly(1 /α ) before returning.

3.2 Filtering: Removing Extreme Outliers

As discussed in the algorithm outline, if the termination check fails, namely the expected score over the entire set of ˜ T is large, then we proceed to either filter or bi-partition our samples. This subsection

states the guarantees of the filtering procedure, which assumes that the αm -smallest and largest elements in the set { ˜ x ⊤ A ˜ x : ˜ x ∈ ˜ T } have distance at most R for some R = poly(1 /α ) .

Recall from Lemma 3.2 that Var ˜ X ∼ H † / 2 D [ ˜ X ⊤ A ˜ X ] = E ˜ X ∼ H † / 2 D [( ˜ X ⊤ A ˜ X -E ˜ X ∼ H † / 2 D [ ˜ X ⊤ A ˜ X ]) 2 ] is bounded by O (1 /α 2 ) . By stability, the same is true for E ˜ X ∼ ˜ S ∩ ˜ T [( ˜ X ⊤ A ˜ X -E ˜ X ∼ ˜ S ∩ ˜ T [ ˜ X ⊤ A ˜ X ]) 2 ] . Notice that this looks almost like our score function f ( ˜ X ) , except that in f ( ˜ X ) we use ˜ y median for centering instead of E ˜ X ∼ ˜ S ∩ ˜ T [ ˜ X ⊤ A ˜ X ] , since the latter quantity is by definition unknown to the algorithm. In Lemma E.3, we show that the two quantities have distance upper bounded by O ( R ) , where R is the quantile distance in our assumption, which in turn implies that the inliers in ˜ S ∩ ˜ T contribute very little to E ˜ X ∼ ˜ T [ f ( ˜ X )] . Given that E ˜ X ∼ ˜ T [ f ( ˜ X )] is large, by virtue of having failed the termination check, we can then conclude that most of the score contribution comes from the outliers. Thus, we can safely use the score to randomly pick an element in the dataset for removal, with probability proportional to its score, and the element will be overwhelmingly more likely to be an outlier rather than an inlier. Lemma 3.5 below states the precise guarantees on the ratio of the total score of the inliers versus the total score over the entire dataset.

Lemma3.5 (Filtering) . Make Assumption 3.1. Let A be an arbitrary symmetric matrix with ∥ A ∥ F = 1 . Let R = C (1 /α ) log(1 /η ) for C ≥ 100 √ C 1 . Define ˜ y median = Median( { ˜ x ⊤ A ˜ x | ˜ x ∈ ˜ T } ) . Define the function f ( ˜ x ) := ( ˜ x ⊤ A ˜ x -˜ y median ) 2 . Let m 1 be a number less than | S | / 3 . Denote by q i the i -th smallest point of { ˜ x ⊤ A ˜ x | ˜ x ∈ ˜ T } . If q | T |-m 1 -q m 1 ≤ R and E X ∼ T [ f ( x )] > C ′ R 2 /α 3 for C ′ ≥ 720 /ϵ 0 , that is, in the case where the check in Line 12 fails, then, the function f ( · ) satisfies ∑ ˜ x ∈ ˜ T f ( ˜ x ) > 40 ϵ 0 1 α 3 ∑ ˜ x ∈ ˜ S ∩ ˜ T f ( ˜ x ) .

The score ratio determines (in expectation) the ratio between the number of outliers and inliers removed. In Lemma 3.5, the ratio is in the order of 1 /α 3 -this will allow us to guarantee that at the end of the entire recursive execution of Algorithm 1, we would have removed at most a 0 . 01 α fraction of the inliers. See Remark F.5 in Appendix F for more details.

3.3 Divider: Identifying Multiple Clusters and Recursing

The previous subsections covered the cases where (i) the expected score is small, or (ii) the expected score over ˜ T is large and the α and 1 -α quantiles of { ˜ X ⊤ A ˜ X : ˜ x ∈ ˜ T } are close to each other. What remains is the case when both the expected score is large yet the quantiles are far apart. In this instance, we will not be able to make progress via filtering using the above argument. This is actually an intuitively reasonable scenario, since the outliers may in fact have another ≈ αm samples that are distributed as a different Gaussian with a very different covariance-the algorithm would not be able to tell which Gaussian is supposed to be the inliers. We will argue that, when both the expected score is large and the quantiles are far apart, the samples are in fact easy to bipartition into 2 clusters, such that the most of the inliers fall within 1 side of the bipartition. This allows us to make progress outside of filtering, and this clustering mechanism also allows us to handle Gaussian mixture models and make sure we (roughly) handle components separately.

The key intuition is that, by the stability Conditions (L.3) and (L.4), we know that the inliers under the 1-dimensional projection { ˜ X ⊤ A ˜ X : ˜ X ∈ ˜ S ∩ ˜ T } must be well-concentrated, in fact lying in an interval of length ˜ O (1 /α ) . The fact that the quantile range is wide implies that there must be some point within the range that is close to very few samples ˜ X ⊤ A ˜ X , by an averaging argument. We can then use the point as a threshold to bipartition our samples. See below for the precise statement.

Algorithm 2 Divider for list decoding

1: function FINDDIVIDER( T , n ′ , m 1 )

2: Let the m 1 -th smallest point be q m 1 and m 1 -th largest point be q | T |-m 1 .

4: Find a subinterval I ′ with at most n ′ / 2 points and return its midpoint.

3: Divide the interval [ q m 1 , q | T |-m 1 ] into 2 m ′ /n ′ equally-sized subintervals.

the i -th smallest point of the set T be q i . Suppose q | T |-m 1 -q m 1 ≥ R such that R ≥ 10( m ′ /n ′ ) r . Then, given T and n ′ , m 1 , Algorithm 2 returns a point t such that if we define T 1 = { x ∈ T : x ≤ t } and T 2 = T \ T 1 then: (i) min( | T 1 | , | T 2 | ) ≥ m 1 and (ii) S proj ⊆ T 1 or S proj ⊆ T 2 .

Proof. The last step of the algorithm must succeed by an averaging argument. Consider the midpoint t of the returned subinterval I ′ , which is at least q m 1 and at most q | T |-m 1 . Since T 1 contains all points at most q m 1 , and T 2 contains all points at most q | T |-m 1 , we must have min( | T 1 | , | T 2 | ) ≥ m 1 . Lastly, we verify the second desideratum, which holds if t / ∈ I . For the sake of contradiction, if t ∈ I , then since I has length r and I ′ has length at least R/ (2 m/n ′ ) ≥ 5 r , then I ⊆ I ′ . However, since | I ′ ∩ T | ≤ n ′ / 2 , we know that I cannot be strictly contained in I ′ , reaching the desired contradiction.

4 High-Level Proof Sketch of Theorem 1.2

We discuss the proof strategy of Theorem 1.2 at a high level. See Appendix F for the complete proof.

Proof Sketch Recall that Algorithm 1 is a recursive algorithm: each call repeatedly filters out samples before either terminating or splitting the dataset into two and recursively calling itself. The execution of Algorithm 1 can thus be viewed as a binary tree, with each node being a recursive call.

The high-level idea for proving Theorem 1.2 is straightforward, though involving technical calculations to implement. Consider the subtree grown from the root recursive call, up to and including a certain level j . We proceed by induction on the height j of such subtree, and claim there must exists a leaf node in this subtree such that most of the inliers remain in the input dataset of the leaf node.

Concretely, let T j be the subtree of height j grown from the root node. We claim that there must be a leaf in this subtree, whose input set T satisfies

recalling that α is the proportion of inliers, ϵ 0 is the maximum fraction of inliers removed by the adversary, ℓ is the actual (unknown) number of inliers removed and m is the size of the original dataset returned by the adversary. The left hand side keeps track of both a) how many inliers we have accidentally removed, through the | S \ T | term, and b) the relative proportions of the outliers we have removed versus the inliers we have removed, by comparing both terms on the left hand side.

For the induction step, we need to analyze the execution of a single recursive call. We show that (1) implies Assumption 3.1, and so the inductive step can follow the case analysis outlined in Section 3either we terminate, or we decide to either filter or bipartition the sample set. To convert this into a high-probability statement, we use a standard (sub-)martingale argument in Appendix F.1.

References

[AK05]

S. Arora and R. Kannan. 'Learning mixtures of separated nonspherical Gaussians'. In: The Annals of Applied Probability 15.1A (2005), pp. 69-92.

[BBHLT20]

M. Brennan, G. Bresler, S. B. Hopkins, J. Li, and T.Schramm. 'Statistical Query Al- gorithms and Low-Degree Tests Are Almost Equivalent'. In: CoRR abs/2009.06107 (2020).

A Additional Related Work

Gaussian Mixture Models Gaussian mixture models (GMM) have a long history in statistics and theoretical computer science. In theoretical computer science, a system investigation of efficient algorithms for GMM was initiated by [Das99], and then improved upon in a series of works, see, for example, [VW04; AK05; DS07; KK10]. However, these algorithms imposed additional assumptions on the parameters of GMM, for example, separation between the components. Later, [MV10; BS10] gave algorithms without any assumptions on the components but these algorithms were sensitive to outliers. The field of algorithmic robust statistics has developed in the last decade to develop outlier-robust algorithms. A recent line of work [BDHKKK20; LM21; LM22; BDJKKV22] has developed efficient algorithms for learning GMM robustly.

List-decodable Learning The model of list-decodable learning was first introduced in [BBV08] and was first used in the context of high-dimensional estimation in [CSV17]. Regarding mean estimation in particular, a body of work presented algorithms with increasingly better guarantees [DKS18; KS17; RY20a; CMY20; DKK20a; DKKLT21; DKKLT22; DKKPP22; ZS22].

B Omitted Preliminaries

B.1 Notation

We use Z + to denote positive integers. For n ∈ Z + , we denote [ n ] def = { 1 , . . . , n } and use S d -1 for the d -dimensional unit sphere. For a vector v , we let ∥ v ∥ 2 denote its ℓ 2 -norm. We use I d to denote the d × d identity matrix. We will drop the subscript when it is clear from the context. For a matrix A , we use ∥ A ∥ F and ∥ A ∥ op to denote the Frobenius and spectral (or operator) norms respectively. We say that a symmetric d × d matrix A is PSD (positive semidefinite) and write A ⪰ 0 if for all vectors x ∈ R d we have that x ⊤ Ax ≥ 0 . We denote λ max ( A ) := max x ∈S d -1 x ⊤ Ax . We write A ⪯ B when B -A is PSD. For a matrix A ∈ R d × d , tr( A ) denotes the trace of the matrix A. We denote by ⟨ v, u ⟩ , the standard inner product between the vectors u, v . For matrices U, V ∈ R d × d , we generalize the inner product as follows: ⟨ U, V ⟩ = ∑ ij U ij V ij . For a matrix A ∈ R d × d , we use A ♭ to denote the flattened vector in R d 2 , and for a v ∈ R d 2 , we use v ♯ to denote the unique matrix A such that A ♭ = v ♯ . For a matrix A , we let A † denote its pseudo-inverse. We use ⊗ to denote the Kronecker product. For a matrix A , we use ker( A ) for the null space of A and use P A to denote the projection onto the range of A , i.e., P A = AA † [Ber18, Proposition 8.1.7 (xii)].

We use the notation X ∼ D to denote that a random variable X is distributed according to the distribution D . For a random variable X , we use E [ X ] for its expectation. We use N ( µ, Σ) to denote the Gaussian distribution with mean µ and covariance matrix Σ . For a set S , we use U ( S ) to denote the uniform distribution on S and use X ∼ S as a shortcut for X ∼ U ( S ) .

We use a ≲ b to denote that there exists an absolute universal constant C > 0 (independent of the variables or parameters on which a and b depend) such that a ≤ Cb .

B.2 Miscellaneous Facts

In this section, we state standard facts that will be used in our analysis.

where the first inequality line uses that Var [ A + B ] = Var [ A ] + Var [ B ] + 2 Cov [ A,B ] ≤ Var [ A ] + Var [ B ] +2 √ Var [ A ] Var [ B ] = ( √ Var [ A ] + √ Var [ B ]) 2 ≤ 2( Var [ A ] + Var [ B ]) , and the last inequality uses the following: The first term uses the fact that for a symmetric matrix B and a vector v , Var Z ∼N (0 ,I ) [ Z ⊤ BZ ] = 2 ∥ B ∥ 2 F and Var Z ∼N (0 ,I ) [ v ⊤ Z ] = ∥ v ∥ 2 2 , with B := Σ 1 / 2 A Σ 1 / 2 and v := Σ 1 / 2 Aµ .

Fact B.2 (Azuma-Hoeffding Inequality) . Let { X n } n ∈ N be a submartingale or supermatingale with respect to a sequence { Y n } n ∈ N . If for all n = 1 , 2 , . . . it holds | X n -X n -1 | ≤ c n almost surely, then for any ϵ > 0 and n ∈ N

Fact B.3 (See, e.g., [Ber18, Proposition 11.10.34]) . Let R and M be two square matrices, then ∥ RM ∥ F ≤ min(( ∥ R ∥ F ∥ M ∥ op , ∥ R ∥ op ∥ M ∥ F ) .

Fact B.4 (Frobenius norm and projection matrices) . Let A be a symmetric matrix. If L 1 and L 2 are two PSD projection matrices satisfying L 1 ⪯ L 2 , then ∥ L 1 AL ⊤ 1 ∥ F ≤ ∥ L 2 AL ⊤ 2 ∥ F .

Fact B.5 (Pseudo-inverse of a matrix) . Let G be a square matrix and define R = G † . Let the SVD of G be U Λ V ⊤ . Then R = V Λ † U ⊤ , P G = U ΛΛ † U ⊤ , and R = R P G . Moreover, Let ∆ be any arbitrary diagonal matrix such that if Λ i,i = 0 , then ∆ i,i = 0 , and define L = V ∆ V ⊤ . Then L = LRG .

Proof. The first follows from [Ber18, Chapter 8], the second follows from [Ber18, Proposition 8.1.7 (xii)], and the third follows from the first two and the fact that Λ † ΛΛ † = Λ . The final claim follows from the following series of equalities:

̸

where we use that ∆ is diagonal and for each i such that ∆ i,i = 0 , Λ † i,i Λ i,i = 1 .

Fact B.6 ([Ber18, Fact 8.4.16]) . Let A and B be two square matrices and suppose that A is full rank. Then P BA = P B .

C Linear Algebraic Results

In this section, we prove the proofs of certain linear algebraic results that we need. In particular, we provide the proofs of Lemmata C.1 and C.3 that will be used later.

C.1 Relative Frobenius Norm to Frobenius Norm: Proof of Lemma C.1

In this section, we provide the proof of Lemma C.1.

Lemma C.1. Consider two arbitrary full rank, positive definite matrices Σ 1 and Σ 2 , and suppose there exists a full rank, positive definite matrix H such that (i) ∥ I -H † / 2 Σ 1 H † / 2 ∥ F ≤ ρ and (ii) ∥ I -H † / 2 Σ 2 H † / 2 ∥ F ≤ ρ . Then, for an arbitrary full rank, positive definite matrix Σ , we have

Proof of Lemma C.1. Let G = H -1 / 2 Σ 1 / 2 . Let ˜ Σ 1 = Σ -1 / 2 Σ 1 Σ -1 / 2 and ˜ Σ 2 = Σ -1 / 2 Σ 2 Σ -1 / 2 , both of which are positive definite matrices. Since Σ is full rank, we obtain that

This directly gives us that G ˜ Σ 1 G ⊤ = I -H Σ 1 H and G ˜ Σ 2 G ⊤ = I -H Σ 2 H . Overall, we have obtained the following:

We first state a result, proved in the end of this section, that transfers closeness in relative Frobenius norm to Frobenius norm of the difference (along most of the directions).

Lemma C.2. Let G be a d × d matrix and let B be an arbitrary symmetric matrix of the same dimension. Recall that P G denotes the projection matrix onto range of G . Suppose that

for some ρ ≥ 1 . Let the SVD of G be G = U Λ V ⊤ for some orthonormal matrices U and V and a diagonal matrix Λ .

Then there exists a d × d matrix L such that

( L captures directions of G ) L = V ∆ B V ⊤ for some binary diagonal matrix ∆ B ⪯ ΛΛ † . That is, the diagonal entries of ∆ B are all in { 0 , 1 } , and furthermore, all the 1 diagonal entries of ∆ B are also non-zero diagonal entries in Λ .

(rank of L ) The rank of L satisfies ∥ Λ -∆ B ∥ F ≤ 2 ρ and

(closeness of G † and B along L ) ∥ ∥ L ( RR ⊤ -B ) L ⊤ ∥ ∥ F ≤ 2 ρ ∥ B ∥ op , where R = G † .

We will now apply Lemma C.2 twice following the same notation as in its statement. In particular, the SVD decomposition of G = U ∆ V ⊤ and R = G † . Lemma C.2 implies that there exist two binary diagonal matrices ∆ 1 and ∆ 2 , where ∆ 1 is for ˜ Σ 1 and ∆ 2 is for ˜ Σ 2 such that (i) both ∥ I -∆ 1 ∥ F and ∥ I -∆ 1 ∥ F are at 2 ρ , and (ii) V ∆ 1 V ⊤ ( RR ⊤ -˜ Σ 1 ) V ∆ 1 V ⊤ and V ∆ 2 V ⊤ ( RR ⊤ -˜ Σ 2 ) V ∆ 2 V ⊤ have Frobenius norms at most 2 ρ ∥ ˜ Σ 1 ∥ op and 2 ρ ∥ ˜ Σ 2 ∥ op, respectively. Let ∆ = ∆ 1 × ∆ 2 and define L = V ∆ V ⊤ . By (i), we have that ∥ I -L ∥ F = ∥ I -∆ ∥ F ≤ 4 ρ . Since L ⪯ V ∆ 1 V ⊤ and L is a projection matrix, Fact B.4 along with (ii) above implies that

Similarly for ˜ Σ 2 . By triangle inequality, we obtain that

Since L is a projection matrix, triangle inequality implies that

This completes the proof.

We now provide the proof of Lemma C.2.

Lemma C.2. Let G be a d × d matrix and let B be an arbitrary symmetric matrix of the same dimension. Recall that P G denotes the projection matrix onto range of G . Suppose that

for some ρ ≥ 1 . Let the SVD of G be G = U Λ V ⊤ for some orthonormal matrices U and V and a diagonal matrix Λ .

Then there exists a d × d matrix L such that

( L captures directions of G ) L = V ∆ B V ⊤ for some binary diagonal matrix ∆ B ⪯ ΛΛ † . That is, the diagonal entries of ∆ B are all in { 0 , 1 } , and furthermore, all the 1 diagonal entries of ∆ B are also non-zero diagonal entries in Λ .

(rank of L ) The rank of L satisfies ∥ Λ -∆ B ∥ F ≤ 2 ρ and

(closeness of G † and B along L ) ∥ ∥ L ( RR ⊤ -B ) L ⊤ ∥ ∥ F ≤ 2 ρ ∥ B ∥ op , where R = G † .

Proof. Let u 1 , u 2 , . . . , u d be the columns of U and let v 1 , v 2 , . . . , v d be the columns of V . Let the entries of Λ be λ 1 , λ 2 , . . . , λ d , which are non-negative by SVD property. Let J be the set of i 's in [ d ] such that λ i > 0 .

Define the matrices F := U ⊤ GBG ⊤ U and B ′ := V ⊤ BV . Then since G ⊤ = V Λ U ⊤ , we have F = U ⊤ GBG ⊤ U = U ⊤ U Λ V ⊤ BV Λ U ⊤ U = Λ V ⊤ BV Λ = Λ B ′ Λ .

Thus the ( i, j ) -th entry of F is B ′ i,j λ i λ j .

Now define the matrix M := U ⊤ P G U . From Fact B.5, we have that P G = U ΛΛ † U ⊤ . And thus, M = ΛΛ † , which is the diagonal matrix with entries M ii = 1 for each i ∈ J , and M ii = 0 otherwise.

Using the invariance of Frobenius norm under orthogonal transform, we obtain the following series of equalities:

Let I ⊆ J be the set of i 's in J such that λ 2 i B ′ i,i < (1 / 2) . Then ∑ i ∈I ( 1 -B ′ i,i · λ 2 i ) 2 > |I| / 4 . Thus if |I| is larger than 4 ρ 2 , then the Frobenius norm squared is larger than r 2 , which is a contradiction.

If i ∈ J \ I , then λ 2 i ≥ 1 / (2 B ′ i,i ) and not just λ i > 0 .

Define the matrix L to be L := ∑ i : i ∈J\I v i v ⊤ i , which is equal to V ∆ B V ⊤ where ∆ B is a diagonal matrix with the i -th diagonal entry equal to 1 if i ∈ J \ I and 0 otherwise. By definition, L has rank |J | - |I| and furthermore, the 1 diagonal entries of ∆ B are always non-zero diagonal entries in Λ .

Let R = G † , and by Fact B.5, we have that R = V Λ † U ⊤ . Thus LR = V ∆ B V ⊤ V Λ † U ⊤ = V ∆ B Λ † U ⊤ , and thus the singular values of LR satisfy that they are at most

We first need two more observations: L = LRG and LRR ⊤ L ⊤ = LR P G R ⊤ L ⊤ , both of which hold by Fact B.5. Finally, we show that RR ⊤ and B are close along the directions in L .

This completes the proof.

C.2 Proof of Lemma C.3

We now restate and prove Lemma C.3.

Lemma C.3. Let A and B be two PSD matrices with P B ⪯ P A or equivalently ker( A ) ⊆ ker( B ) . Then ∥ ABA -P B ∥ F ≤ 2 ∥ ABA -P A ∥ F .

Proof. Applying triangle inequality, we have that

It thus suffices to show that the second term, ∥ P B -P A ∥ F, is less than ∥ ABA -P A ∥ F. Let d A be the rank of A and d B be the rank of B . We will show the following two results, which imply the desired bound:

In the rest of the proof, we will establish the above two inequalities.

We begin with the second inequality in (5). By the lemma assumptions, we have that the null space of A being a subspace of the null space of B . This is equivalent to saying that the column space of B being a subspace of the column space of A . In particular, this also implies that d B ≤ d A . It is easy to see that rank( ABA ) ≤ rank( B ) ≤ rank( A ) = rank( P A ) , and the eigenvalues of P A are either 0 or 1 . We now state a simple result lower bounding the Frobenius norm by the difference of the ranks for such matrices:

Claim C.4 (Bound on rank difference) . Let A ′ be a symmetric matrix such that the eigenvalues of A ′ belong to { 0 , 1 , -1 } . Let B ′ be an arbitrary matrix. Then ∥ B ′ -A ′ ∥ 2 F ≥ rank( A ′ ) -rank( B ′ ) .

Proof. Let u 1 , . . . u d be the orthonormal eigenvectors of A ′ with eigenvalues λ 1 , λ 2 , . . . , λ d such that | λ i | = 1 for i ≤ rank( A ′ ) and λ i = 0 otherwise. Without loss of generality, we assume rank( B ′ ) ≤ rank( A ′ ) , otherwise the result is trivial. Let I be the set of i 's such that i ≤ rank( A ′ ) and B ′ u i = 0 .

Since the rank of B ′ is less than the rank of A ′ , it follows that B ′ u i is nonzero for at most rank( B ′ ) many u i 's in { u i : i ≤ rank( A ′ ) } . Thus |I| is at least rank( A ′ ) -rank( B ′ ) .

By Claim C.4, we have that rank( P B ) -rank( ABA ) ≤ ∥ P B -ABA ∥ 2 F . Thus, it implies the desired inequality, | d B -d A | ≤ ∥ ABA -P A ∥ 2 F .

We will now establish the first inequality in (5). Observe that both P A and P B are PSD matrices, whose eigenvalues are either 1 or 0 . The following result upper bounds the Frobenius norm of P A -P B in terms of the difference of their ranks.

Claim C.5. If A and B are two PSD matrices such that P B ⪯ P A , then ∥ P A -P B ∥ 2 F = rank( A ) -rank( B ) .

Proof. Let d A be the rank of A and d B be the rank of B . We have that d B = rank( P B ) ≤ rank( P A ) = d A . Consider any orthonormal basis v 1 , . . . , v d in R d , such that v 1 , . . . , v d A spans the column space of P A , and v 1 , . . . , v d B spans the column space of B . Since the column space of P B is a subspace of that of P B , such an orthonormal basis always exists.

Moreover, since the eigenvalues of P A are either 0 or 1 , we have that P A v i = v i for i ≤ d A and 0 otherwise. Similarly, P B v i = v i for i ≤ d B and 0 otherwise.

We will also use the following basic fact: for any square matrix J and any set of orthonormal basis vectors u 1 , . . . , u d , we have that ∥ J ∥ 2 F = ∑ d i =1 ∥ Ju i ∥ 2 2 . Applying this fact to P A -P B and the orthonormal basis v 1 , . . . , v d above, we get the following:

where the second equality uses that P A v i = P B v i = 0 for i ≥ d A , and the third equality uses that P A v i = P B v i = v i for i ≤ d B and P B v i = 0 and P A v i = v i for i ∈ { d B +1 , . . . , d A } .

2

D Stability: Sample Complexity and Its Consequences

In this section, we prove results related to stability. Appendix D.1 proves the sample complexity for the stability condition to hold for Gaussians and Appendix D.2 focuses on some consequences of stability.

Remark D.1. We note that the deterministic conditions of Definition 2.1 are not specific to the Gaussian distribution but hold with polynomial sample complexity poly( d/η ) for broader classes of distributions, roughly speaking, distributions with light tails for degree-4 polynomials.

Condition (L.1) is satisfied with polynomial sample complexity by distributions that have bounded second moment.

Condition (L.3) at the population level is a direct implication of hypercontractivity of degree-2 polynomials. The log(1 /η ) factor in the condition as stated currently is tailored to the Gaussian distribution but it is not crucial for the algorithm and could be relaxed to some polynomial of 1 /η ; this would translate to incurring some poly(1 /η ) to the final error guarantee of the main theorem. After establishing the condition at the population level, it can be transferred to sample level with polynomial sample complexity.

Regarding Condition (L.2), every entry of the Frobenius norm is a degree 2 polynomial, thus the property holds for distributions with light tails for these polynomials, e.g., hypercontractive degree-2 distributions.

Condition (L.4) can also be derived by hypercontractivity of degree 4 polynomials similarly.

Condition (L.5) holds almost surely for continuous distributions.

D.1 Sample Complexity of Stability Conditions

Lemma 2.2 (Deterministic Conditions Hold with High Probability) . For a sufficiently small positive constant ϵ 0 and a sufficiently large absolute constant C , a set of m>Cd 2 log 5 ( d/ ( ηδ )) /η 2 samples from N ( µ, Σ) , with probability 1 -δ , is ( η, ϵ ) -stable set with respect to µ, Σ for all ϵ ≤ ϵ 0 .

Proof. Let A be a set of i.i.d. samples from N ( µ, Σ) . Wecheck the four conditions from Definition 2.1 separately. Since A is a set of i.i.d. samples from a Gaussian, it satisfies Condition (L.1) with high probability (see, e.g., [Li18, Corollary 2.1.13]).

We start with Condition (L.2). Let X ∼ N ( µ, Σ) , then X -µ ∼ N (0 , Σ) . Equivalently, X -µ = Σ 1 / 2 Z with Z ∼ N (0 , I ) . Let A be a set of m i.i.d. samples from N ( µ, Σ) and like before, for every x ∈ A , write x -µ = Σ 1 / 2 z , where z are corresponding i.i.d. samples from N (0 , I ) . Let a subset A ′ ⊂ A with | A ′ | ≥ | A | (1 -ϵ ) . Then, we have the following

where the first line uses the rewriting that we introduced before, the second line uses the cyclic property of the trace operator, and the last line uses [Li18, Corollary 2.1.13]. As stated in that corollary, the inequality holds with probability 1 -δ/ 4 as long as the sample complexity is a sufficiently large multiple of ( d 2 /ϵ 2 ) log(4 /δ ) . Fixing ϵ = ϵ 0 a sufficiently small positive absolute constant can make the right hand side of the above inequality 0 . 1 ∥ Σ 1 / 2 U Σ 1 / 2 ∥ F.

Condition Condition (L.1) can be shown identically using [Li18, Corollary 2.1.9].

We now turn our attention to Condition (L.3). [DKKLMS16, Lemma 5.17] implies that with the stated number of samples, with high probability, it holds that

Since | A ′ | ≥ | A | (1 -ϵ 0 ) ≥ | A | / 2 , Condition (L.3) holds. Finally, Condition (L.4) follows by noting that Var X ∼ A ′ [ p ( X )] ≤ 2 Var X ∼ A [ p ( X )] and using [DKKLMS16, Lemma 5.17] again with ϵ = ϵ 0 . In order for the conclusion of that lemma to hold with probability 1 -δ/ 4 , the number of samples of the Gaussian component is a sufficiently large multiple of d 2 log 5 (4 d/ ( ηδ )) /η 2 . A union bound over the events corresponding to each of the four conditions concludes the proof.

Lastly, we show Condition (L.5). We will in fact show a stronger statement: suppose the rank of Σ is d Σ . Then, if we take m ≥ d Σ +1 i.i.d. samples { µ + z 1 , µ + z 2 , . . . , µ + z m } from N ( µ, Σ) , it must be the case that with probability 1, for all A ′ ⊆ A with | A ′ | ≥ d Σ +1 , we have ∑ x ∈ A ′ ( v ⊤ x ) 2 = 0 (or equivalently, v being orthogonal to all vectors x ∈ A ′ ) implying v ⊤ Σ v = 0 for all vectors v ∈ R d .

We will need the following straightforward claim: if A ′ = { µ + z ′ 0 , . . . , µ + z ′ d Σ } (of size d Σ +1 ) is such that 1) { z ′ 1 -z ′ 0 , . . . , z ′ d Σ -z ′ 0 } are linearly independent and 2) z ′ i all lie within the column space of Σ , then for any vector v ∈ R d , v being orthogonal to all of A ′ implies that v ⊤ Σ v = 0 . To see why this is true, observe that { z ′ 1 -z ′ 0 , . . . , z ′ d Σ -z ′ 0 } , which are linearly independent vectors that all lie in the column space of Σ , must span the entirety of the column space of Σ by dimension counting. Thus, if v is orthogonal to all the points in A ′ , then v is orthogonal to all of { z ′ 1 -z ′ 0 , . . . , z ′ d Σ -z ′ 0 } , which implies v ⊤ Σ v = 0 .

We can now show the following inductive claim on the size of A , which combined with the above claim implies the strengthened version of Condition (L.5). Specifically, we show inductively that, with probability 1 over the sampling of A , for any subset A ′ ⊂ A of size at most d Σ +1 , the claim conditions hold for A ′ . Namely, if A ′ = { µ + z ′ 0 , . . . , µ + z ′ j } for some j ≤ d Σ , for some arbitrary ordering of the elements in A ′ , then 1) the set { z ′ 1 -z ′ 0 , . . . , z ′ j -z ′ 0 } is linearly independent and 2) z ′ i all lie in the column space of Σ .

The base case of | A | = 1 is trivial. Now suppose the above statement is true for | A ℓ | = ℓ , and consider sampling a new point µ + z ℓ and adding it to A ℓ to form A ℓ +1 . Take any subset A ′ of A ℓ +1 of size at most d Σ +1 . If A ′ ⊂ A ℓ we are already done by the induction hypothesis. Otherwise, A ′ = { µ + z ℓ }∪ A ′ ℓ , where A ′ ℓ = { µ + z ′ 0 , . . . , µ + z ′ j } has size at most d Σ and satisfies the conditions in the induction by the inductive hypothesis. The space of µ + z such that z -z ′ 0 lies in the span of { z ′ 1 -z ′ 0 , . . . , z ′ j -z ′ 0 } has measure 0 under N ( µ, Σ) , given there are strictly fewer than d Σ many linearly independent vectors in { z ′ 1 -z ′ 0 , . . . , z ′ j -z ′ 0 } , all of which lie in the column space of Σ . Furthermore, there is only a finite number of such A ′ ℓ . Thus, with probability 1, the new point µ + z ℓ will be sampled such that for all such A ′ ℓ , { µ + z ℓ } ∪ A ′ ℓ remains linearly independent. Additionally, also with probability 1, z ℓ will lie in the column space of Σ . This completes the proof of the inductive claim, which as we have shown implies Condition (L.5).

D.2 Facts Regarding Stability

The following corollaries will be useful later on. In particular, since our algorithm will perform transformations of the dataset, we will need Corollary D.2 to argue about the stability properties of these sets. Moreover, the next lemma (Lemma D.3) will be used to extract the final Frobenius norm guarantee of our theorem given the first bullet of that theorem. The proofs are deferred to Appendix D.4.

Corollary D.2 (Stability of Transformed Sets) . Let D be a distribution over R d with mean µ and covariance Σ . Let A be a set of points in R d that satisfies Conditions (L.1) and (L.2) of Definition 2.1 of ( η, ϵ ) -stability with respect to D . Then we have that for any symmetric transformation P ∈ R d × d and every A ′ ⊂ A with | A ′ | ≥ (1 -ϵ ) | A | , the following holds:

The following lemma (proved in Appendix D.4) will be needed to prove our certificate lemma to transfer the final guarantees of our algorithm from the empirical covariance of the sample set to the actual covariance of the Gaussians.

Lemma D.3. Let A be a set of points in R d satisfying Conditions (L.1) and (L.2) of ( η, ϵ ) -stability (Definition 2.1) with respect to µ, Σ . Let A ′ ⊆ A such that | A ′ | ≥ (1 -ϵ ) | A | . If P, L ∈ R d × d are symmetric matrices with the property that ∥ Cov X ∼ A ′ [ PX ] -L ∥ F ≤ r , then

D.3 Certificate Lemma

The following result shows that if the covariance matrix of X ⊗ 2 over the corrupted set of points is bounded in operator norm, then the empirical second moment matrix of T is a good estimate of covariance of inliers in Frobenius norm. We prove a more general version below that allows for arbitrary linear transformation matrix P and since we are interested in guarantees for relative Frobenius norm, our algorithm will chose P so that E X ∼ T [ PXX ⊤ P ] is equal to identity.

Lemma D.4 (Bounded fourth-moment and second moment imply closeness of covariance) . Let T be a set of n points in R d . Let P be an arbitrary symmetric matrix. Let w ∈ R d 2 be the leading eigenvector of the Cov X ∼ T [( PX ) ⊗ 2 ] with ∥ w ∥ 2 = 1 , and let A ∈ R d × d be w ♯ . Note that ∥ w ∥ 2 = 1 implies ∥ A ∥ F = 1 . For every subset T ′ ⊆ T with | T ′ | ≥ ( α/ 2) | T | , we have that

In particular, suppose that there exists a set S that satisfies Conditions (L.1) and (L.2) of ( η, ϵ )-stability (Definition 2.1) with respect to µ and Σ for some η and ϵ , and suppose that | S ∩ T ′ | ≥ min( α | T | / 2 , (1 -ϵ ) | S | ) . Then

Proof. Let ˜ T define the set { PX : X ∈ T } . In this notation, w is the leading eigenvector of Cov ˜ x ∼ ˜ T [ ˜ X ⊗ 2 ] with ∥ w ∥ 2 = 1 , and A = w ♯ . Then, for any ˜ T ′ ⊆ ˜ T with | ˜ T ′ | ≥ α | ˜ T ′ | / 2 , we have the following

where the penultimate step above holds by lifting the flattened matrix inner products in the previous line to the matrix inner product, and the last line uses the variation characterization of the Frobenius norm. This implies the first inequality since E ˜ X ∼ ˜ T [ ˜ X ˜ X ⊤ ] = E X ∼ T [ PXX ⊤ P ] .

We will now establish the second inequality. Using the relation between the second moment matrix and the covariance, we have the following:

where the last step uses that E ˜ X ∼ ˜ T ′ [ ˜ X ] E ˜ X ∼ ˜ T ′ [ ˜ X ⊤ ] has rank one and thus its Frobenius and operator norm match. Thus to establish the desired inequality, it suffices to prove that E ˜ X ∼ ˜ T ′ [ ˜ X ] E ˜ X ∼ ˜ T ′ [ ˜ X ⊤ ] ⪯ (2 /α ) · E ˜ X ∼ ˜ T [ ˜ X ˜ X ⊤ ] . Indeed, we have that

where the last expression is in fact equal to (2 /α ) · E ˜ X ∼ ˜ T [ ˜ X ˜ X ⊤ ] . This completes the proof for the second statement.

We now consider the case when T contains a large stable subset. Let T ′ = T ∩ S , which satisfies that | T ′ | ≥ ( α/ 2) | T | and | T ′ | ≥ (1 -ϵ ) | S | . By applying Lemma D.3 with L = E X ∼ T [ PXX ⊤ P ] to the second inequality in the statement with T ′ , we obtain the desired result.

D.4 Proofs of Corollary D.2 and lemma D.3

Corollary D.2 (Stability of Transformed Sets) . Let D be a distribution over R d with mean µ and covariance Σ . Let A be a set of points in R d that satisfies Conditions (L.1) and (L.2) of Definition 2.1 of ( η, ϵ ) -stability with respect to D . Then we have that for any symmetric transformation P ∈ R d × d and every A ′ ⊂ A with | A ′ | ≥ (1 -ϵ ) | A | , the following holds:

Proof. We explain each condition separately. We begin with the first condition as follows: For any unit vector u in R d , we have that

where the first inequality uses Condition (L.1) of Definition 2.1 for v = Pu .

We now focus our attention to the second condition. Denote ˆ Σ := 1 | A ′ | ∑ x ∈ A ′ ( x -µ )( x -µ ) ⊤ for saving space. Let Q be any symmetric matrix Q ∈ R d × d with ∥ Q ∥ F ≤ 1 . Using the cyclical property of trace, we obtain

where the last step applies Condition (L.2) of Definition 2.1 with V = PQP . Finally, using the cyclic properties of the trace operator,

where the first inequality uses Fact B.3 and the second inequality uses ∥ Q ∥ F ≤ 1 . Thus we have the following:

where the second equality is due to the fact that P ( ˆ Σ -Σ ) P is itself symmetric.

Lemma D.3. Let A be a set of points in R d satisfying Conditions (L.1) and (L.2) of ( η, ϵ ) -stability (Definition 2.1) with respect to µ, Σ . Let A ′ ⊆ A such that | A ′ | ≥ (1 -ϵ ) | A | . If P, L ∈ R d × d are symmetric matrices with the property that ∥ Cov X ∼ A ′ [ PX ] -L ∥ F ≤ r , then ∥ P Σ P -L ∥ = O ( r + ∥ L ∥ op ) .

Proof. To simplify notation, let ̂ Σ := Cov X ∼ A ′ [ X ] and Σ := E X ∼ A ′ [( X -µ )( X -µ ) ⊤ ] . Observe that Σ = ̂ Σ+( µ -ˆ µ )( µ -ˆ µ ) ⊤ . We apply triangle inequality and Corollary D.2 with to obtain the following:

where the first inequality follows from the triangle inequality and the second inequality uses Corollary D.2.

We now analyze the desired expression by using the triangle inequality and the lemma assumption as follows:

where the first term was bounded by r by assumption and the second term uses the bound (6). Thus it suffices to show that ∥ P Σ P ∥ op = O ( r + ∥ L ∥ op ) . To this end, we again use triangle inequality as follows:

where the second inequality uses the bound (6). Rearranging the above display inequality, we obtain ∥ P Σ P ∥ op = O ( r + ∥ L ∥ op ) .

E Analysis of a Single Recursive Call of the Algorithm: Proofs from Section 3

In this section, we present the details that were omitted from Section 3. We recall Assumption 3.1 from the main body.

Assumption 3.1 (Assumptions and notations for a single recursive call of the algorithm) .

S = { x i } n i =1 is a set of n uncontaminated samples, which is assumed to be ( η, 2 ϵ 0 ) -stable with respect to the inlier distribution D having mean and covariance µ, Σ (c.f. Definition 2.1). We assume η ≤ 0 . 001 , ϵ 0 = 0 . 01 , and Var X ∼ D [ X ⊤ AX ] ≤ C 1 ( ∥ Σ 1 / 2 A Σ 1 / 2 ∥ 2 F + ∥ Σ 1 / 2 Aµ ∥ 2 2 ) for all symmetric d × d matrices A and a constant C 1 .

T is the input set to the current recursive call of the algorithm (after the adversarial corruptions), which satisfies | S ∩ T | ≥ (1 -2 ϵ 0 ) | S | and | T | ≤ (1 /α ) | S | .

We denote H = E X ∼ T [ XX ⊤ ] .

We denote by ˜ S, ˜ T the versions of S and T normalized by H † / 2 : ˜ S = { H † / 2 x : x ∈ S } and ˜ T = { H † / 2 x : x ∈ T } . We use the notation ˜ x for elements in ˜ S and ˜ T , and x for elements in S and T . Similarly, we use the notation ˜ X for random variables with support in ˜ S or ˜ T .

The mean and covariance of the inlier distribution D after transformation with H † / 2 are denoted by ˜ µ := H † / 2 µ , ˜ Σ := H † / 2 Σ H † / 2 . We denote the empirical mean and covariance of the transformed inliers in T by ˆ µ := E ˜ X ∼ ˜ S ∩ ˜ T [ ˜ X ] and ˆ Σ := Cov ˜ X ∼ ˜ S ∩ ˜ T [ ˜ X ] .

E.1 Normalization and Proof of Lemma 3.2

We start with a simple result stating that after normalization with H † / 2 , the mean and covariance of the inliers, both empirical and population-level, are bounded. This is intuitive since the inliers constitute Ω( α ) -fraction of the overall samples and the second moment of the complete set after normalization is bounded by identity.

Lemma E.1 (Normalization) . Make Assumption 3.1 and recall the notations ˜ µ, ˜ Σ , ˆ µ, ˆ Σ that were defined in Assumption 3.1. We have that:

∥ ˆ Σ ∥ op ≤ 2 /α .

∥ ˆ µ ∥ 2 ≤ √ 2 /α .

∥ ˜ µ ∥ 2 ≤ √ 3 /α

For every matrix A with ∥ A ∥ F ≤ 1 , | E ˜ X ∼ ˜ S [ ˜ X ⊤ A ˜ X ] -E X ∼ D [( H † / 2 X ) ⊤ A ( H † / 2 X )] | < 1 /α .

Proof. We prove each part below:

Establishing Item 1 The transformation H is such that E ˜ X ∼ ˜ T [ ˜ X ˜ X ⊤ ] = E X ∼ T [ H † / 2 XX ⊤ H † / 2 ] = H † / 2 HH † / 2 = P H , which has operator norm at most 1 . By assumption, we have that ∣ ∣ ∣ ˜ S ∩ ˜ T ∣ ∣ ∣ ≥ (1 -2 ϵ 0 ) α | T | ≥ 0 . 5 α | T | . Thus, we obtain the following inequality:

By applying (8), we obtain

implying that ∥ ˆ Σ ∥ op ≤ 2 /α , since P H is just a projection matrix with 0 / 1 eigenvalues.

Establishing Item 2 Since, for any random vector X , we have that E [ X ] E [ X ] ⊤ ⪯ E [ XX ⊤ ] (which is equivalent to Cov [ X ] ⪰ 0 ), we obtain

where the last inequality uses (8). This implies that ∥ ˆ µ ∥ 2 2 ≤ 2 /α .

Establishing Item 3 The goal is to bound the population-level covariance ∥ ˜ Σ ∥ op. To this end, we will use the bounds from Items 1 and 2 which bounds their empirical versions and relate the empirical versions to the population ones via the deterministic conditions.

Consider an arbitrary subset ˜ S 1 of ˜ S satisfying ∣ ∣ ∣ ˜ S 1 ∣ ∣ ∣ ≥ ∣ ∣ ∣ ˜ S ∣ ∣ ∣ (1 -2 ϵ 0 ) . We first note that by Corollary D.2, we have that

Now define ˜ Σ 1 := E ˜ X ∼ ˜ S ∩ ˜ T [( ˜ X -˜ µ )( ˜ X -˜ µ ) ⊤ ] , the centered second moment matrix of ˜ S ∩ ˜ T , which satisfies ˜ Σ 1 = ˆ Σ+( ˜ µ -ˆ µ )( ˜ µ -ˆ µ ) ⊤ . We have that

where in the last line we use (9) for ˜ S 1 = ˜ S ∩ ˜ T and the fact that ∣ ∣ ∣ ˜ S ∩ ˜ T ∣ ∣ ∣ ≥ (1 -2 ϵ 0 ) | ˜ S | . Rearranging, we have that ∥ ˜ Σ ∥ op < 1 . 25 ∥ ˆ Σ ∥ op for which we can use Item 1 to further upper bound it by 3 /α .

Establishing Item 4 We use a similar argument as in Item 3:

where the first step uses the triangle inequality, the second inequality uses Equation (9) for ˜ S 1 = ˜ S ∩ ˜ T , and the last inequality uses Items 2 and 3.

Establishing Item 5 For the covariance condition, we have the following:

where the last step upper bounds the first term using Equation (9) for ˜ S 1 = ˜ S , which trivially satisfies | ˜ S | ≥ | ˜ S | (1 -2 ϵ 0 ) . The overall expression is upper bounded by (1 . 1 × 3) /α by Item 3. The mean condition has a similar proof:

where we use Equations (9) and (10).

Establishing Item 6 Consider an arbitrary square matrix A with ∥ A ∥ F ≤ 1 . We have that

where the inequality (a) uses the fact that ∥ uv ⊤ -wz ⊤ ∥ 2 F ≤ ∥ u -w ∥ 2 2 + ∥ v -z ∥ 2 2

for the last term.

The previous result implies that the mean and covariance of the inlier distribution D after transformation with H † / 2 is bounded. Since the variance of degree-two polynomials under D is bounded by assumption, we obtain the following bound on the variance:

Lemma 3.2. Make Assumption 3.1 and recall that H = E X ∼ T [ XX ⊤ ] , where T is the corrupted version of a stable inlier set S . For every symmetric matrix A with ∥ A ∥ F = 1 , we have that Var X ∼ D [( H † / 2 X ) ⊤ A ( H † / 2 X )] ≤ 18 C 1 /α 2 .

Proof. By Assumption 3.1, we have that Var X ∼ D [ X ⊤ BX ] ≤ C 1 ( ∥ Σ 1 / 2 B Σ 1 / 2 ∥ 2 F + ∥ Σ 1 / 2 Bµ ∥ 2 2 ) for all B ∈ R d × d . Applying this to B = H † / 2 AH † / 2 , we have that

where we use that ∥ ˜ Σ 1 / 2 A ˜ Σ 1 / 2 ∥ 2 F = ∥ Σ 1 / 2 H † / 2 AH † / 2 Σ 1 / 2 ∥ 2 F and ∥ Σ 1 / 2 H † / 2 AH † / 2 µ ∥ 2 2 = ∥ ˜ Σ 1 / 2 A ˜ µ ∥ 2 2 since ˜ Σ = H † / 2 Σ H † / 2 and ˜ µ = H † / 2 µ . By Lemma E.1, ∥ ˜ Σ ∥ op ≤ 3 /α . Since ∥ A ∥ F = 1 , we have that the first term in (11) is bounded as follows: ∥ ˜ Σ 1 / 2 A ˜ Σ 1 / 2 ∥ 2 F ≤ ∥ ˜ Σ ∥ 2 op ∥ A ∥ 2 F ≤ 9 /α 2 , using the fact that ∥ AB ∥ F ≤ ∥ A ∥ op ∥ B ∥ F (Fact B.3). We also have that ∥ ˜ Σ 1 / 2 A ˜ µ ∥ 2 2 ≤ ∥ ˜ Σ ∥ op ∥ A ∥ 2 op ∥ ˜ µ ∥ 2 2 ≤ 9 /α 2 , where we the bounds from Lemma E.1. The desired conclusion then follows by applying these upper bounds in (11).

E.2 Certificate Lemma: Proof of Lemma 3.4

Lemma E.2 (Case when we stop and return; Formal version of Lemma 3.4) . Make Assumption 3.1. Let w ∈ R d 2 be the leading eigenvector of the Cov ˜ X ∼ ˜ T [ ˜ X ⊗ 2 ] with ∥ w ∥ 2 = 1 , and let A ∈ R d × d be w ♯ . Note that ∥ w ∥ 2 = 1 implies ∥ A ∥ F = 1 . For every subset ˜ T ′ ⊆ ˜ T with | ˜ T ′ | ≥ ( α/ 2) | ˜ T | , we have that

In particular, we have that ∥ ∥ H † / 2 Σ H † / 2 -P H ∥ ∥ 2 F ≲ 1 α Var ˜ X ∼ ˜ T [ ˜ X ⊤ A ˜ X ] + 1 α 2 and ∥ ∥ H † / 2 Σ H † / 2 -P Σ ∥ ∥ 2 F ≲ 1 α Var ˜ X ∼ ˜ T [ ˜ X ⊤ A ˜ X ] + 1 α 2 .

Proof. We will apply the main certificate lemma, Lemma D.4, with P = H † / 2 . The desired result then follows by noting that E X ∼ T [ PXX ⊤ P ] = H † / 2 HH † / 2 = P H and the operator norm of P H is at most 1 . We will use the following result, proved in Appendix C, to upper bound the frobenius norm of H † / 2 Σ H † / 2 -P Σ :

Lemma C.3. Let A and B be two PSD matrices with P B ⪯ P A or equivalently ker( A ) ⊆ ker( B ) . Then ∥ ABA -P B ∥ F ≤ 2 ∥ ABA -P A ∥ F .

Before providing the proof sketch of Lemma C.3 below, we show that it suffices to prove the desired result. Since H is the second moment matrix of T , we have that ker( H ) ⊆ ker( ∑ x ∈ S ∩ T xx ⊤ ) , which is further contained in ker(Σ) by Condition (L.5) in Definition 2.1. We apply Lemma C.3 with B = Σ and A = H , which gives the desired result.

We now give a brief proof sketch of Lemma C.3. By triangle inequality, it suffices to show that ∥ P A -P B ∥ F ≤ ∥ ABA -P A ∥ F. Since ker( A ) ⊂ ker( B ) , P A -P B is again a projection matrix of rank equal to rank( A ) -rank( B ) and thus its frobenius norm is equal to square root of rank( A ) -rank( B ) . On the other hand, the matrices ABA and P A have a rank difference of at least rank( A ) -rank( B ) . Combining this observation with the fact that P A has binary eigenvalues, we can lower bound ∥ ABA -P A ∥ F by square root of rank( A ) -rank( B ) .

E.3 Filtering: Proof of Lemma 3.5

We first show that if we take the median of a quadratic polynomial p over the corrupted set, then the sample mean of p over the inliers is not too far from the median if the left and right quantiles are close.

Lemma E.3 (Quantiles of quadratics after normalization) . Make Assumption 3.1. Let A be an arbitrary symmetric matrix with ∥ A ∥ F = 1 . Define ˜ y median = Median ({ ˜ x ⊤ A ˜ x | ˜ x ∈ ˜ T }) . Let m 1 be any number less than | S | / 3 . Denote by q i the i -th smallest point of { ˜ x ⊤ A ˜ x | ˜ x ∈ ˜ T } .

Suppose that q | T |-m 1 -q m 1 ≤ R for R := C (1 /α ) log(1 /η ) with C > 100 √ C 1 (recall that the absolute constant C 1 is defined in Assumption 3.1). Then, ∣ ∣ ∣ E ˜ X ∼ ˜ S [ ˜ X ⊤ A ˜ X ] -˜ y median ∣ ∣ ∣ ≤ 2 R .

Proof. Let µ ′ = E ˜ X ∼ ˜ S [ ˜ X ⊤ A ˜ X ] . (Not to be confused with ˆ µ and ˜ µ in Assumption 3.1, which are expectations of ˜ X in the samples and at the population level, instead of the quadratic form in the definition of µ ′ .)

Given that ˜ y median (by definition) lies within the interval [ q m 1 , q | T |-m 1 ] which has length at most R , it suffices to argue that µ ′ also lies within distance R of that interval. Namely, µ ′ ≥ q m 1 -R and µ ′ ≤ q | T |-m 1 + R .

Let σ 2 := Var X ∼ D [( H † / 2 X ) ⊤ A ( H † / 2 X )] . The ( η, 2 ϵ 0 ) -stability of S in Assumption 3.1 (see Condition (L.3)) implies that all but an η -fraction of { ˜ x ⊤ A ˜ x : ˜ x ∈ ˜ S ∩ ˜ T } lies in the interval

E X ∼ D [( H † / 2 X ) ⊤ A ( H † / 2 X )] ± 10 σ log(1 /η ) since the size of the set ˜ S ∩ ˜ T } is at least | S | (1 -2 ϵ 0 ) (by Assumption 3.1).

Thus, for at least (1 -η ) | S ∩ T | ≥ | S | (1 -2 ϵ 0 -η ) | S | points in T , the value of ˜ x ⊤ A ˜ x lies in the interval E X ∼ D [( H † / 2 X ) ⊤ A ( H † / 2 X )] ± 10 σ log(1 /η ) . Since η ≤ 0 . 01 and ϵ 0 < 1 / 4 , the number of such points is at least | S | / 3 .

Rephrasing, there is an interval [ y left , y right ] that contains at least | S | / 3 points in ˜ T , where

Therefore, there are at most | T | - | S | / 3 points in ˜ T that are less than y left , implying that q | T |-| S | / 3 ≥ y left . Furthermore, since, by assumption we have m 1 ≤ | S | / 3 , this implies that q | T |-m 1 ≥ y left . By a symmetric argument, we also have that q m 1 ≤ y right .

Next, recall that by Item 6 of Lemma E.1, we have that | E X ∼ D [( H † / 2 X ) ⊤ A ( H † / 2 X )] -µ ′ | = | E X ∼ D [( H † / 2 X ) ⊤ A ( H † / 2 X )] -E ˜ X ∼ ˜ S [ ˜ X ⊤ A ˜ X ] | < 1 /α . Thus, we have q | T |-m 1 ≥ y left ≥ µ ′ -10 σ log(1 /η ) -1 /α . Rearranging, we get µ ′ ≤ q | T |-m 1 +10 σ log(1 /η ) + 1 /α . Symmetrically, we get that µ ′ ≥ q m 1 -10 σ log(1 /η ) -1 /α .

Finally, we argue that 10 σ log(1 /η ) + 1 /α ≤ R , showing that µ ′ ≤ q | T |-m 1 + R and µ ′ ≥ q m 1 -R , which as we argued at the beginning of the proof is sufficient to show the lemma statement. To argue this, by Lemma 3.2, we have that σ ≤ 5 √ C 1 /α . Thus, it suffices to choose, as in the lemma statement, that R = ( C/α ) log(1 /η ) for some sufficiently large C > 0 .

The above result implies that if we calculate the 'squared deviation' of the projections of inliers centered around the empirical median of the corrupted set , then it will be of the order of the empirical variance of inliers up to a factor of R 2 (by triangle inequality). Moreover, the empirical variance of the inliers is of the order O (1 /α 2 ) by Lemma 3.2. Thus if the variance of the corrupted set is much larger than the upper bound, then we can assign score to each point, the function f below, such that the contribution from outliers is much larger than the inliers.

Lemma E.4 (Filtering; Formal version of Lemma 3.5) . Make Assumption 3.1. Let A be an arbitrary symmetric matrix with ∥ A ∥ F = 1 . Let R = C (1 /α ) log(1 /η ) for C ≥ 100 √ C 1 . Define ˜ y median = Median( { ˜ x ⊤ A ˜ x | ˜ x ∈ ˜ T } ) . Define the function f ( ˜ x ) := ( ˜ x ⊤ A ˜ x -˜ y median ) 2 . Let m 1 be a number less than | S | / 3 . Denote by q i the i -th smallest point of { ˜ x ⊤ A ˜ x | ˜ x ∈ ˜ T } .

If q | T |-m 1 -q m 1 ≤ R and E X ∼ T [ f ( x )] > C ′ R 2 /α 3 for C ′ ≥ 720 /ϵ 0 , that is, in the case where the check in Line 12 fails, then, the function f ( · ) satisfies

Proof. The core argument for the proof is that ∑ ˜ x ∈ ˜ S ∩ ˜ T f ( ˜ x ) can be upper bounded under the assumption that q | T |-m 1 -q m 1 ≤ R , as shown in the following claim.

Claim E.5. Assuming the conditions in Lemma E.4, we have ∑ x ∈ ˜ S ∩ ˜ T f ( ˜ x ) ≤ 9 | S | R 2 .

˜

Proof. Let µ ′ = E ˜ X ∼ ˜ S [ ˜ X ⊤ A ˜ X ] . We have the following series of inequalities:

where the last line uses that R ≥ 12 √ C 1 /α , which is satisfied for C ≥ 12 √ C 1 . This completes the proof of Claim E.5.

On the other hand, we have that

This completes the proof of Lemma E.4.

F Proof of Theorem 1.2

In this section, we present the main algorithmic result of the paper. As previously stated, the theorem holds for distributions beyond Gaussians, as long as the input set of points satisfies the deterministic conditions from Section 2.1 and the distribution meets a mild requirement. We now restate and prove the theorem in this more general form.

Theorem F.1 (Weak List Decodable Covariance Estimation) . Let the ambient dimension be d ∈ Z + , let C be a sufficiently large absolute constant, and let the parameters α ∈ (0 , 1 / 2) , ϵ ∈ (0 , ϵ 0 ) for a sufficiently small positive constant ϵ 0 , and failure probability δ ∈ (0 , 1 / 2) be known to the algorithm. Let D be a distribution with mean µ ∈ R d , covariance Σ ∈ R d × d , and assume that Var X ∼ D [ X ⊤ AX ] = O ( ∥ Σ 1 / 2 A Σ 1 / 2 ∥ 2 F + ∥ Σ 1 / 2 Aµ ∥ 2 2 ) . There is a polynomial-time algorithm such that, on input α, δ as well as an ( α, ϵ ) -corruption of a set S (Definition 1.1) with | S | > C/ ( α 6 ϵ 2 0 ) log(1 / ( αδ )) which is (( ϵ 0 / 40) α 3 , 2 ϵ 0 )) -stable with respect to D (Definition 2.1), with probability at least 1 -δ over the randomness of the algorithm, the algorithm returns a list of at most O (1 /α ) many sets T i which are disjoint subsets of samples in the input, each T i has size at least 0 . 5 αm , and there exists a T i in the output list such that:

Recall the notation in the corruption model (Definition 1.1) where n is the size of the original inlier set S and ℓ is the number of points in S that the adversary replacedn and ℓ are unknown to the algorithm except that n ≥ αm and ℓ ≤ ϵn . The set T i satisfies that | T i ∩ S | ≥ (1 -0 . 01 α )( n -ℓ ) .

The Gaussian distribution N ( µ, Σ) satisfies the distributional assumption of Theorem F.1 (c.f. Fact B.1), and a set of size m = C ′ d 2 log 5 ( d/αδ ) α 6 from N ( µ, Σ) is (( ϵ 0 / 40) α 3 , 2 ϵ 0 )) -stable with probability 1 -δ for a constant ϵ 0 (c.f. Lemma 2.2). Thus, Theorem F.1 theorem covers the Gaussian

case D = N ( µ, Σ) with polynomial sample complexity, and it directly implies Theorem 1.2, the main result of this work.

We achieve the guarantee of Theorem F.1 by combining the procedure of the previous section in a recursive algorithm (Algorithm 1). We maintain the notation of Assumption 3.1 with the addition of subscripts that indicate the number of filtering steps performed.

We restate the algorithm below, before proving Theorem F.1:

Algorithm 3 List-Decodable Covariance Estimation in Relative Frobenius Norm

Proof of Theorem F.1. Since the final error guarantees do not depend on ϵ , without loss of generality we use the maximum level of corruptions ϵ = ϵ 0 = 0 . 01 in the following proof. We will also use the letter η to denote the expression ( ϵ 0 / 40) α 3 as in the stability assumption in the theorem statement on the inlier set. We will proceed to prove Theorem F.1 by induction, and crucially use the following fact that 'union bounds also work under conditioning'.

Fact F.2. If event A happens with probability 1 -τ 1 and event B happens with probability 1 -τ 2 conditioned on event A , then the probability of both A and B happening is at least 1 -τ 1 -τ 2 .

The technical bulk for proving Theorem F.1 is Lemma F.4, an inductive claim over the levels of recursion, that (in addition to some other basic properties) with high probability over any level, either (i) there was a recursive call at a prior level whose input contains most of the inliers, and by definition that call has terminated, or (ii) there exists some recursive call at this current level whose input contains most of the inliers. We will use the following notation (Definition F.3) to denote various objects in the recursive execution of our algorithm.

Definition F.3 (Notation for Recursion Tree) . We define a rooted binary tree T that corresponds to the execution of our recursive algorithm (Algorithm 1). The root node at level 0 corresponds to the top-level call of COVLISTDECODING (Algorithm 1) and for every non-leaf node, its left child corresponds to a call of COVLISTDECODING from Line 21 and its right child corresponds to a call

from Line 21. Thus, every node is uniquely specified by the level in the tree and its position within that level. We denote by T i,j the i th node of level j , where the node numbers are also 0-indexed (e.g., the root node is T 0 , 0 ). For the node T i,j , we denote by T ( i,j ) (0) the input data set to the corresponding recursive call. In order to refer to the working data set at the t th iteration of the main loop in the execution of node T i,j , we use the notation T ( i,j ) ( t ) and ˜ T ( i,j ) ( t ) exactly in the same way as T ( t ) and ˜ T ( t ) are used in Algorithm 1. Finally, T j (i.e., using a single subscript) refers to the subtree growing from the root and including all the nodes up to and including the j th level.

Lemma F.4. In the context of Theorem F.1, consider the recursion tree of COVLISTDECODING (Definition F .3). Then, for any j ∈ { 0 , . . . , 9 /α } , we have that with probability at least 1 -0 . 01 αjδ the following holds:

(Size decrease under recursion) The input of every node T i,j of level j satisfies | T ( i,j ) (0) | ≤ m -jαm/ 9

(Disjoint inputs) Consider the subtree T j growing from the root and truncated at (but including) level j . All the leaves in T j have disjoint input sets. Note that the leaves may not all be at level j , and some might be at earlier levels.

(Bound on inliers removed) Consider the subtree T j growing from the root and truncated at (but including) level j . There exists a leaf T i ′ ,j ′ in T j (but the level of the leaf j ′ is not necessarily equal to j ) such that its input T ( i ′ ,j ′ ) (0) satisfies

where as in Definition 1.1, m is the size of the original input set of the root node T 0 , 0 and ℓ ≤ ϵ 0 n is the number of samples that the adversary replaced in S . In particular, the number of inliers removed by the algorithm until T ( i ′ ,j ′ ) (0) is at most ( j +1) α 3 ( ϵ 0 / 20) m .

Remark F.5. We remark on why (12) uses a cubic power of α , which is the same cubic power appearing in Lemma E.4 that states that the total score of all the outliers is a 1 /α 3 factor more than the total score of the inliers. The three powers of α are due to the following reasons: 1) the inliers can be as few as an α -fraction of all the samples, 2) the depth of the recursion tree can be as much as O (1 /α ) , meaning that a set can be filtered O (1 /α ) times, and 3) Theorem F.1 guarantees that we remove at most an O ( α ) fraction of inliers. We also remark that, by tweaking parameters in the algorithm and increasing the power of poly(1 /α ) in our covariance estimation guarantees, we can make the score function ratio (and hence the poly( α ) factor in (12)) as large a power of α as we desire. This would guarantee that we remove at most a smaller poly( α ) fraction of inliers.

Proof. We prove Lemma F.4 by induction on the level number j . For the base case of j = 0 , Conditions 1 and 2 are trivial. As for Condition 3, at j = 0 we have | T (0 , 0) (0) | = m and | S \ T (0 , 0) (0) | ≤ ℓ by the definition of the contamination model (Definition 1.1), which directly implies α 3 ( ϵ 0 / 40) | T (0 , 0) (0) | + | S \ T (0 , 0) (0) | ≤ α 3 ( ϵ 0 / 40) m + ℓ ≤ α 3 ( ϵ 0 / 20) m + ℓ as desired.

To show the inductive case, we assume the lemma statement is true for some j and we will show the statement for the case j +1 .

Conditions 1 and 2 These are trivially true, by virtue of the fact that the recursive algorithm, after it is done with removing points via filtering, it partitions the input set using FINDDIVIDER which guarantees that both output sets have size at least m 1 = αm/ 9 (Lemma 3.6).

Condition 3 Recall the notation T ( i,j ) ( t ) from Definition F.3 that is used to denote the set of points in the variable T t after the t th iteration in the i th recursive call in level j .

By Fact F.2, we will condition on Condition 3 being true for T j , the subtree truncated at (but including) level j and show that Condition 3 holds also for T j +1 , the subtree truncated at level j +1 , except with probability 0 . 01 αδ . Concretely, the conditioning implies that there exists a leaf T i ′ ,j ′ of the subtree

T j whose input set T ( i ′ ,j ′ ) (0) satisfies the inequality in Condition 3 for j ′ ≤ j . If j ′ < j , then we are done (since it continues to be a leaf and the desired bound on the right hand size in Equation (12) is only larger). Otherwise, in the case of j ′ = j , we have to analyze the execution of the recursive call associated with this leaf node in the recursion tree T j .

To begin with the analysis, we verify that the input set T ( i ′ ,j ) (0) satisfies Assumption 3.1. We first check that | S ∩ T ( i ′ ,j ) (0) | ≥ n (1 -2 ϵ 0 ) :

where the last line uses that α ≤ 1 / 2 . Then it remains to check that | T ( i ′ ,j ) (0) | ≤ (1 /α ) | S | , but this is trivially satisfied since it holds for the input set T (0 , 0) (0) that we start with: | T ( i ′ ,j ) (0) | ≤ | T (0 , 0) (0) | = m ≤ (1 /α ) n = (1 /α ) | S | .

In a recursive call, the algorithm iteratively removes points (through Line 16) before either terminating (Line 12) or calling FINDDIVIDER and recursing (Line 21). Denote the iteration just before either terminating or running FINDDIVIDER by t ∗ i ′ . We need to argue that the iterative filtering of points prior to iteration t ∗ i ′ still roughly preserves Condition 3. Lemma F.6, that is stated below and proved in Lemma F.6, captures the standard martingale argument for filtering-based robust algorithms, and it informally states that if the input set T ( i ′ ,j ) (0) satisfies Assumption 3.1, then Condition 3 is roughly preserved.

Lemma F.6. Recall the notations and assumptions from Assumption 3.1. Consider an execution of COVLISTDECODING ( T 0 ) with T 0 = T (with T satisfying the assumptions from Assumption 3.1), and suppose that | T 0 | ≥ C ′ / ( α 6 ϵ 2 0 ) log (1 / ( αδ )) . Further assume that | S \ T 0 | ≤ 1 . 5 ϵ 0 | S | (which is a strengthening over Assumption 3.1). Moreover, denote by T t the dataset through the loop iterations of Algorithm 1. Then, with probability at least 1 -0 . 01 αδ , it holds that

simultaneously for all iterations t until the execution of COVLISTDECODING enters either Line 13 or Line 19.

Concretely, applying Lemma F.6 for input set T ( i ′ ,j ) (0) and with failure probability 0 . 01 αδ in place of δ . The lemma is applicable since its first requirement | S \ T ( i ′ ,j ) (0) | ≤ 1 . 5 ϵ 0 | S | has been already checked in (13), and for its second requirement we have that | T ( i ′ ,j ) (0) | ≥ | T ( i ′ ,j ) (0) ∩ S | ≥ (1 -1 . 5 ϵ 0 ) | S | ≥ C ′ / ( α 6 ϵ 2 0 ) log(1 / ( αδ )) , where the last one uses (13) and the next step uses our assumption | S | > C/ ( α 6 ϵ 2 0 ) log(1 / ( αδ )) . The lemma then yields that, with probability at least 1 -0 . 01 αδ , T ( i ′ ,j ) ( t ∗ i ′ ) is such that

Now, either the recursive call terminates at iteration t ∗ i ′ or it goes into Line 19. In the former case, we are done (since it is now a leaf node). Otherwise, the recursive call uses FINDDIVIDER to partition T ( i ′ ,j ) ( t ∗ i ′ ) into T ′ and T ′′ . We need to show that at least one of T ′ and T ′′ satisfies Condition 3 (for case j +1 ), using Lemma 3.6. Let us now derive what parameters and set to invoke Lemma 3.6 with.

Using (i) Condition (L.3) of the stability property of the original inlier set S , and (ii) the fact that ( H † / 2 X ) ⊤ A ( H † / 2 X ) -E X ∼N ( µ, Σ) [( H † / 2 X ) ⊤ A ( H † / 2 X )] is an even quadratic in X for any H

and A , it must be the case that a 1 -η fraction (recall η is a shorthand for ( ϵ 0 / 40) α 3 ) of the points x ∈ S satisfies

Denote the set of points x ∈ S with the above property by S H,A ⊆ S , and let S ( i ′ ,j ) core = T ( i ′ ,j ) ( t ∗ i ′ ) ∩ S H,A . We will therefore apply Lemma 3.6 with S proj = { ( H † / 2 x ) ⊤ A ( H † / 2 x ) : x ∈ S ( i ′ ,j ) core } , T = { ( H † / 2 x ) ⊤ A ( H † / 2 x ) : x ∈ T ( i ′ ,j ) ( t ∗ i ′ ) } and diameter r = 100log(2 /η ) √ C 1 /α . It remains to check that for q m 1 = q left and q | T ( i ′ ,j ) ( t ∗ i ′ ) |-m 1 = q right of T ( i ′ ,j ) ( t ∗ i ′ ) , we have q | T ( i ′ ,j ) ( t ∗ i ′ ) |-m 1 -q m 1 ≥ 10( | T ( i ′ ,j ) ( t ∗ i ′ ) | / | S proj | ) r , to make sure that the application of Lemma 3.6 is valid.

To show this, recall that Line 14 fails whenever FINDDIVIDER is called, meaning that q | T ( i ′ ,j ) ( t ∗ i ′ ) |-m 1 -q m 1 ≥ R . Thus, we need to verify that the definition of R in Algorithm 1 satisfies R ≥ 10( | T ( i ′ ,j ) ( t ∗ i ′ ) | / | S proj | ) r . The main step is to lower bound | S proj | = | S ( i ′ ,j ) core | . Since S ( i ′ ,j ) core ⊂ S , we can lower bound the above by

Combining with the fact that | T ( i ′ ,j ) ( t ∗ i ′ ) | ≤ m , we have that 10( | T ( i ′ ,j ) ( t ∗ i ′ ) | / | S proj | ) r ≤ 20( m/n ) r ≤ (20 /α ) r . Recalling that r = 100log(2 /η ) √ C 1 /α , η = ( ϵ 0 / 40) α 3 and ϵ 0 = 0 . 001 , the definition of R in Algorithm 1 satisfies R ≥ 6000 √ C 1 (1 /α 2 ) log(1 / ( ϵ 0 α )) ≥ (20 /α ) r ≥ 10( | T ( i ′ ,j ) ( t ∗ i ′ ) | / | S proj | ) r .

Knowing that the application of Lemma 3.6 is valid, the lemma then guarantees that either T ′ or T ′′ contains all of S ( i ′ ,j ) core . Without loss of generality, we assume this happens for T ′ . We now check Condition 3 for case j +1 on T ′ :

Thus Condition 3 is satisfied by T ′ for case j +1 , completing the inductive proof of this lemma.

We now use the lemma to conclude the proof of Theorem F.1.

By Lemma F.4, we know that with probability at least 1 -0 . 09 δ , the recursion tree generated up to level 9 /α , namely T 9 /α , satisfies the lemma guarantees. In particular, since (i) by Line 19 in Algorithm 1, the input to any recursive call must have size at least m 1 = αm/ 9 , and (ii) Condition 1 of Lemma F.4 applied to j = m/m 1 = 9 /α yields an input size upper bound of 0, we can conclude that the execution of the algorithm must have completely terminated by level 9 /α . We know by Lemma F.4 that there exists a leaf node i in the recursion tree whose input satisfies Condition 3 of the third part of the lemma statement. That is, letting j denote the level of that leaf, we have that

The recursion node T i,j starts with that input set T ( i,j ) (0) and, before terminating, it may perform some filtering steps. As in the proof of Lemma F.4, we will use Lemma F.6 to analyze the 'working set' right before the recursive call terminates. If we denote by t ∗ the number of filtering steps before termination, Lemma F.6 yields that except with probability 0 . 01 αδ , we have

Note that by Fact F.2, the total failure probability is upper bounded by 0 . 09 δ +0 . 01 αδ which is less than δ .

We are now ready to prove the first bullet of Theorem F.1. The above inequality implies that

(19) further implies that | T ( i,j ) ( t ∗ ) ∩ S | ≥ ( n -ℓ )(1 -ϵ 0 α ) ≥ n (1 -ϵ 0 )(1 -ϵ 0 α ) ≥ ( α/ 2) m (since n ≥ αm and ϵ 0 and α are small), meaning that the set T ( i,j ) ( t ∗ ) will indeed be returned. This completes the proof of the first bullet of Theorem F.1.

We now move to the second bullet. Defining H = E X ∼ T ( i,j ) ( t ∗ ) [ XX ⊤ ] , we need to show that ∥ H † / 2 Σ H † / 2 -HH † ∥ F ≲ (1 /α 4 ) log 2 (1 /α ) . We will apply Lemma E.2 to show the bounds in the second bullet, which requires checking that Assumption 3.1 is satisfied by T ( i,j ) ( t ∗ ) . Earlier, right below (19), we have already checked that | T ( i,j ) ( t ∗ ) ∩ S | ≥ ( α/ 2) | T ( i,j ) ( t ∗ ) | . The only remaining condition to check is | T ( i,j ) ( t ∗ ) | ≤ | S | /α , but this is trivially true since T ( i,j ) ( t ∗ ) ⊂ T (0 , 0) (0) (the original input set to the recursive algorithm) and | T (0 , 0) (0) | ≤ | S | /α by Definition 1.1. Thus, Lemma E.2 yields that

where the penultimate inequality uses the fact that, if the algorithm terminated, it must be the case that E ˜ X ∼ T ( i,j ) ( t ∗ ) [ f ( ˜ X )] ≤ C ′ R 2 /α 3 by Line 12 in Algorithm 1. Taking a square root on both sides implies that

The same guarantee holds for ∥ ∥ H † / 2 Σ H † / 2 -P Σ ∥ ∥ F , also following from Lemma E.2.

Lastly, we check that we only return O (1 /α ) sets in the output list. This is true by construction: (i) Line 13 only allows sets to be output if they have size at least Ω( αm ) , (ii) there were only m points to begin with and (iii) all the leaves have disjoint input sets by Lemma F.4.

F.1 Proof of Lemma F.6

We now prove that filtering does not remove too many inliers using a standard martingale argument.

Lemma F.6. Recall the notations and assumptions from Assumption 3.1. Consider an execution of COVLISTDECODING ( T 0 ) with T 0 = T (with T satisfying the assumptions from Assumption 3.1), and suppose that | T 0 | ≥ C ′ / ( α 6 ϵ 2 0 ) log (1 / ( αδ )) . Further assume that | S \ T 0 | ≤ 1 . 5 ϵ 0 | S | (which is a strengthening over Assumption 3.1). Moreover, denote by T t the dataset through the loop iterations of Algorithm 1. Then, with probability at least 1 -0 . 01 αδ , it holds that

simultaneously for all iterations t until the execution of COVLISTDECODING enters either Line 13 or Line 19.

Proof. We will use the notation of Assumption 3.1 for a single call of Algorithm 1. Denote by t the iteration count. Also define the stopping time t end to be the first iteration when α 3 ( ϵ 0 / 40) | T t end | + | S \ T t end | > 2 ϵ 0 | S | or when the iteration goes into Line 12 or Line 19 instead of Line 16 (that is, when E X ∼ T t end [ f ( H † / 2 t X )] ≤ C ′ R 2 /α 3 or when q right -q left > R for the iteration t end ). Now, define ∆ t = α 3 ( ϵ 0 / 40) | T min { t,t end } | + | S \ T min { t,t end } | .

In order to prove the lemma, we will show that at the first t ∗ (if one exists) such that α 3 ( ϵ 0 / 40) | T t ∗ | + | S \ T t ∗ | > α 3 ( ϵ 0 / 40) | T 0 | + | S \ T 0 | + α 3 ( ϵ 0 / 40) | T 0 | , then ∆ t ∗ > α 3 ( ϵ 0 / 40) | T 0 | + | S \ T 0 | + α 3 ( ϵ 0 / 40) | T 0 | as well. Afterwards, we will show that ∆ t is a sub-martingale and use sub-martingale tail bounds to show that the sequence ∆ t remains small over an entire trajectory of | T 0 | steps with high probability.

The first step is easy to show, since the threshold of α 3 ( ϵ 0 / 40) | T 0 | + | S \ T 0 | + α 3 ( ϵ 0 / 40) | T 0 | < α 3 ( ϵ 0 / 20) | T 0 | +1 . 5 ϵ 0 | S | ≤ α 2 ( ϵ 0 / 20) | S | +1 . 5 ϵ 0 | S | ≤ 2 ϵ 0 | S | , where the first inequality is by the lemma assumption. Therefore, t ∗ ≤ t end if it exists, meaning that ∆ t ∗ = α 3 ( ϵ 0 / 40) | T t ∗ | + | S \ T t ∗ | .

Now we need to show that ∆ t is a sub-martingale with respect to the sequence T min { t,t end } . If t ≥ t end , then ∆ t +1 is by definition equal to ∆ t and thus E [∆ t +1 | T min { t,t end } ] = ∆ t . Otherwise, we have that E [∆ t +1 | T t ] is equal to ∆ t plus the expected number of inliers removed minus α 3 ( ϵ 0 / 40) times the expected number of all points removed by our filter. Since the stopping condition is not satisfied, we have | S \ T t | ≤ 2 ϵ 0 | S | , meaning that Assumption 3.1 continues to hold for T t , as well as that the other conditions for Lemma E.4 hold. We can therefore apply Lemma E.4 to obtain

Summarizing, the above case analysis shows that { ∆ t } t ∈ N is a sub-martingale with respect to T min { t,t end } .

We note also that for every t , | ∆ t -∆ t +1 | ≤ 1 with probability 1. Thus, using the standard Azuma-Hoeffding inequality for sub-martingales (Fact B.2), we have that for every t ≤ | T 0 | ,

By a union bound over all t ∈ [ | T 0 | ] , we have

where the last inequality uses that | T 0 | > C ′ α 6 ϵ 2 0 log ( 1 αδ ) for a sufficiently large absolute constant C ′ .

G Applications to Learning Gaussian Mixture Models

In this section, we prove the applications of our result to robustly learning Gaussian mixture models.

Theorem 1.4 (Outlier-Robust Clustering and Estimation of Covariances for GMM) . Let C ′ > 0 be a sufficiently large constant and ϵ 0 > 0 be a sufficiently small constant. Let the parameters α ∈ (0 , 1 / 2) , ϵ ∈ (0 , ϵ 0 ) , and failure probability δ ∈ (0 , 1 / 2) be known. There is an ˜ O ( m 2 d 2 ) -time algorithm such that, on input α, δ , and m > C ′ d 2 log 5 ( d/αδ ) α 6 many ϵ -corrupted samples from an unknown k -component Gaussian mixture ∑ k p =1 α p N ( µ p , Σ p ) over R d as in Definition 1.3, where all Σ p 's are full-rank and all α p satisfies α p ≥ α and k ≤ 1 α is unknown to the algorithm, with probability at least 1 -δ over the corrupted samples and the randomness of the algorithm, the algorithm returns a list of at most k many disjoint subsets of samples { T i } such that:

For the p th Gaussian component, denote the set S p as the samples in the inlier set S that were drawn from component p . Let n p be the size of S p , and let ℓ p be the number of points in S p that the adversary replacedn p and ℓ p are both unknown to the algorithm except that E [ n p ] = α p m ≥ αm for each p and ∑ p ℓ p ≤ ϵαm . Then, for every Gaussian component p in the mixture, there exists a set T i p in the returned list such that | T i p ∩ S p | ≥ (1 -0 . 01 α )( n p -ℓ p ) .

For every component p , there is a set of samples T i p in the returned list such that, defining H i p = E X ∼ T i p [ XX ⊤ ] , we have H i p satisfying ∥ H -1 / 2 i p Σ p H -1 / 2 i p -I ∥ F ≲ (1 /α 4 ) log(1 /α ) .

̸

Let Σ be the (population-level) covariance matrix of the Gaussian mixture. For any two components p = p ′ with ∥ Σ -1 / 2 (Σ p -Σ p ′ )Σ -1 / 2 ∥ F > C (1 /α ) 5 log(1 /α ) for a sufficiently large constant C , the sets T i p and T i p ′ from the previous bullet are guaranteed to be different.

We show the first two bullets as a direct corollary of Theorem 1.2 applied to each Gaussian component D = N ( µ p , Σ p ) as the inlier distribution. The proof of the final bullet involves some more involved linear algebraic lemmas. Here we give only the high-level proof of the final bullet, and defer the proof of these individual lemmas to Appendix C.

Proof. First observe that, with a sufficiently large poly( d, 1 α , log 1 δ ) samples, with probability at least 1 -δ/ 4 , the number of inliers n p from component p is at least 0 . 99 αm for all components p . We will condition on this event.

Furthermore, noting that in the corruption model defined in Definition 1.3, at most ϵαm points can be corrupted total, this implies that ℓ p ≤ ϵαm ≤ ( ϵ/ 0 . 99) n p (since n p ≥ 0 . 99 αm ).

Therefore, we apply Theorem 1.2 to all the k Gaussian components, using parameter 0 . 99 α in place of α , ϵ/ 0 . 99 in place of ϵ , and failure probability δ/ (4 k ) . This implies that, with a union bound, using m = poly( d, 1 /α, log k/δ ) = poly( d, 1 /α, log 1 /δ ) samples (since k ≲ 1 /α , for otherwise the Gaussian mixture will have weight greater than 1), with probability at least 1 -δ/ 4 , the two bullet points in the theorem holds for all the Gaussian components, directly guaranteed by Theorem 1.2.

It remains to show that the algorithm will return at most k subsets in its output list. Noting that, across the subsets output by the algorithm for each of the components, there is already a total of (1 -0 . 01 α ) ∑ k p =1 ( n p -ℓ p ) points being output. Recall that ∑ p n p = m and ∑ p ℓ p ≤ ϵαm . Thus, in

the output subsets corresponding to the components, there are a total of at least m -( ϵ +0 . 01) αm ≥ m -0 . 4 αm points (where we used that ϵ is bounded by sufficiently small constant. Since (i) there were only m points in the input and (ii) all the output subsets are disjoint subsets of the input, this means that there are at most 0 . 4 αm points that can be output in any additional subsets not corresponding to any Gaussian component. However, by Theorem 1.2, any output subset must have size at least 0 . 5(0 . 99 α ) m , and so there cannot be any extra subset output in the list that does not correspond to any Gaussian component. This shows that there can be at most k subsets in the output list.

We turn to the final bullet of the theorem. We will show that any two components p and p ′ that belong to the same set T i in the output of the algorithm satisfy that ∥ Σ -1 / 2 Σ p Σ -1 / 2 -Σ -1 / 2 Σ p ′ Σ -1 / 2 ∥ F ≲ 1 α 5 log(1 /α ) . Taking the contrapositive yields the final theorem bullet.

Let H be the second moment matrix of this set, that is, H = E X ∼ T i [ XX ⊤ ] . Then H satisfies that ∥ I -H † / 2 Σ p H † / 2 ∥ F ≤ r and ∥ I -H † / 2 Σ p ′ H † / 2 ∥ F ≤ r for r ≲ (1 /α 4 ) log(1 /α ) , by the second bullet point of the theorem. Furthermore, H is full rank because H is the second moment matrix of a large subset of empirical points of component p and thus Condition (L.5) implies that ker( H ) is contained in ker(Σ p ) , which is empty since Σ p is full rank by the theorem assumption.

We will now apply the following result (Lemma C.1) to infer that Σ p and Σ p ′ must be close, in the sense of the final theorem bullet point. We defer the proof of this lemma to Appendix C.

We will now apply Lemma C.1 to Σ 1 = Σ p and Σ 2 = Σ p ′ , and upper bound the right hand side of the lemma to give the final theorem bullet point.

Weclaim that the operator norm of Σ -1 / 2 Σ p Σ -1 / 2 is bounded by 1 /α , due to the fact that component p has mass at least α . Indeed, we have that

This in particular implies that Σ -1 / 2 Σ p Σ -1 / 2 ⪯ 1 α p I ⪯ 1 α I , and hence ∥ Σ -1 / 2 Σ p Σ -1 / 2 ∥ op ≤ 1 α . The same conclusion holds for the component p ′ . Recalling that r ≲ (1 /α 4 ) log(1 /α ) , the right hand side of Lemma C.1 can be upper bounded by O ((1 /α 5 ) log(1 /α )) , thus yielding the final bullet in the theorem, as discussed above.

We end this section with a brief discussion on the proof of Lemma C.1. The lemma states that, if there exists a single matrix H that is an approximate square root to both Σ 1 and Σ 2 , in the sense that ∥ I -H † / 2 Σ 1 H † / 2 ∥ F and ∥ I -H † / 2 Σ 1 H † / 2 ∥ F are both upper bounded, then Σ 1 and Σ 2 cannot be too far from each other, in that Σ 1 -Σ 2 must also have bounded Frobenius norm. In fact, this can be generalized to the normalized version Σ -1 / 2 (Σ 1 -Σ 2 )Σ -1 / 2 for any positive definite matrix Σ .

For simplicity of the discussion, let us assume for now that Σ = I . To show Lemma C.1, a natural first idea is to show that, if ∥ I -H † / 2 Σ 1 H † / 2 ∥ F is bounded, say by some quantity ρ , then so must be ∥ HH ⊤ -Σ 1 ∥ F by some quantity related to O ( ρ ) , times the operator norm of Σ 1 for appropriate scaling. Unfortunately, that is not true: Even for Σ 1 = I , we can have H being infinity in the first diagonal element and 1s in the other diagonal elements, and ∥ I -H † / 2 Σ 1 H † / 2 ∥ F would only be 1 yet HH ⊤ -Σ 1 is infinite in the first dimension. The crucial observation, then, is that a slightly weaker statement is true: On an orthonormal basis that depends only on H and not on Σ 1 , HH ⊤ approximates Σ 1 in most dimensions, namely d -O ( ρ 2 ) many dimensions. This statement is formally captured by Lemma C.2. Thus, in these dimensions, we can bound the difference between Σ 1 and Σ 2 (in Frobenius norm). In the rest of the dimensions where HH ⊤ does not approximate either Σ 1 or Σ 2 well, we can bound these dimensions' contribution to the Frobenius norm of Σ 1 -Σ 2 simply by max( ∥ Σ 1 ∥ op , ∥ Σ 2 ∥ op ) times the number of such dimensions, which is small. Combining these bounds yields Lemma C.1 as desired.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the problem of list-decodable covariance estimation. In this problem, we are given a number of samples, an $\\alpha$-fraction of which were i.i.d. drawn from a Gaussian distribution $N(\\mu,\\Sigma)$. The authors design an algorithm that runs in time $\\mathrm{poly}(d/\\alpha)$ and produces a list of size $O(1/\\alpha)$ such that one element in the list is close to $\\Sigma$ in relative Frobenius norm (up to error $\\mathrm{poly(1/\\alpha)}$).\n\nPrevious work has given an $d^{\\mathrm{poly(1/\\alpha)}}$-time algorithm that achieves the stronger guarantee that one element in the list is close in total variation distance to $N(0,\\Sigma)$ (in this case the list contains tuples of estimates, one for the mean and one for the covariance). While the TVD is related to the error metric the authors consider, the recovery guarantees of the authors do not imply anything about TVD-closeness. This is unsurprising, there are known SQ lower bounds that suggest that for TVD guarantees $d^{\\mathrm{poly(1/\\alpha)}}$ samples are required. The authors show that if one settles for the weaker guarantee of \"only\" relative Frobenius norm error, efficient algorithms, i.e., that run in time polynomial in all problem parameters, are possible.\n\nTheir techniques are similar to the techniques for list-decodable mean estimation with additional technical tweaks to handle the more challenging covariance estimation setting. They also provide an application to robustly learning mixtures of non-spherical Gaussians.\n\nSTRENGTHS:\nList-decodable covariance estimation is a very natural and important problem. The relative Frobenius norm metric the authors consider is the natural choice here since it captures the \"intrinsic\" scale of the problem induced by the true covariance matrix -- even in the regime where it is not related to the total variation distance.\n\nThe authors show how to extend the techniques for list-decodable mean estimation to this setting. This requires non-trivial modifications since covariance estimation in general is a more difficult and much less well-understood task than mean estimation.\n\nThe paper is fairly well-written and the authors give a nice proof overview in Section 1.2.\n\nWEAKNESSES:\nThe relation to IK22 was not clear to me in the beginning, and I believe it would be important to clarify already in the beginning. My main confusion comes from the fact that in Line 49 you say that small relative Frobenius norm is equivalent to small TVD. Then you go on to say that IK22 achieve error in TVD and you in relative Frobenius norm. At this point it seems that the two error guarantees are the same, and it also seems like your algorithm is breaking the SQ lower bound.\n\nOnly later you clarify that for the equivalence to hold the relative frobenius norm needs to be smaller than 1. (You do say in Line 49 that it only holds if the relative Frobenius norm is smaller than a small enough constant but do not specify what this is and how it relates to $\\mathrm{poly(1/\\alpha)}$, your error guarantee.\n\nQUESTIONS:\nYou say that your list-decoding algorithm can be used to obtain a \"sum-of-squares-free\" algorithm for robustly clustering non-spherical Gaussians. Does this come with any (theoretical) speed-ups? If yes, I think it would be good to mention what these are.\n\nLIMITATIONS:\nMaybe I missed this, but I didn't see any statements about limitations in their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper studies the problem of robustly estimating the covariance of a multivariate Gaussian distribution in the presence of outliers. Specifically, the authors focus on a list-decodable setting, where the algorithm is allowed to output a small list of hypotheses with the guarantee that at least one of them is close to the target. The authors propose a novel polynomial-time algorithm that efficiently list-decodes the covariance of a Gaussian distribution in relative Frobenius norm under adversarial corruption.

STRENGTHS:
This is a very nice paper that tackles an important problem in robust statistics, and the result has direct applications in robustly learning GMMs. The contributions of the paper are novel algorithms that significantly improve the state-of-art algorithm in terms of sample complexity and answer size. The paper is very well written and it is a pleasure to read. In our opinion, this is an important contribution to the literature.

WEAKNESSES:
The organization of this paper can be improved. In particular, I think the stability section deserves some space in the main paper and 
the paper could benefit from a discussion section at the end of the paper.

QUESTIONS:
*

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSummary: This paper studies the following problem: given a collection of datapoints in $R^d$, of which a small $\\alpha$ fraction (inliers) are guaranteed to be drawn from an unknown Gaussian distribution, and the rest are drawn adversarially (outliers), the goal is approximately identify the distribution. Note that for $\\alpha < 1/2,$ it is not possible to uniquely identify the distribution, but one could hope to output a list of $\\alpha^{-1}$ hypothesis distributions, one of which is correct (this is called list decoding).\n\nThis problem has received significant attention recently from a computational perspective. By now, several algorithms are known for list-decoding in these settings. However, most of them are based on solving large, expensive convex programs (sums-of-squares SDPs) and require $d^{poly(1/\\alpha)}$ running time and sample complexity. While the strongest of these algorithms can recover a list of hypotheses such that at least one of them has a bounded total variation distance from the unknown Gaussian, there are lower bounds showing that such a recovery requires  $d^{poly(1/\\alpha)}$ samples for a wide class of algorithms. \n\nThis paper presents a significantly simpler and faster algorithm ($poly(d/\\alpha)$ time and samples) that is able to generate a list of $O(1/\\alpha)$ hypotheses such that at least one of the covariance matrices is close to the original covariance matrix in relative Frobenius norm. This is necessarily a weaker guarantee than total variation distance, since it bypasses the known lower bounds.\n\nSTRENGTHS:\nStrengths:\n+ A fairly simple to describe algorithm based on multi-filtering\n+ avoids solve large convex programs\n+ achieves a reasonable $poly(d/\\alpha)$ sample complexity and running time guarantee, much better than most previous algorithms\n\nWEAKNESSES:\nThe main theorems in the paper only talk of recovering the covariance matrix, but not a pair $(\\tilde{\\mu}, \\tilde{\\Sigma})$ where both the mean and covariance are close. Are these guarantees also achievable? If yes, I strongly suggest the authors include such a complete result\n\nQUESTIONS:\nYour algorithm removes points one by one in line 16, what is the obstacle in removing a large fraction of the points at each step?\n\nMinor comments:\n1. In Def 1.1, since the adversary choses all the outliers after looking at the distribution, is it not equivalent to the setting of having exactly $\\alpha m$ inliers?\n2. Theorem 1.2, first bullet point -- it's awkward to recall within a theorem statement, I suggest removing the sentence \"Recall ... $\\ell \\le \\epsilon n$\"\n\nLIMITATIONS:\nTheoretical work. \nSee weakness pointed above.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on a fundamental problem in the field of robust statistics, the problem of estimating the covariance matrix of a high-dimensional Gaussian distribution, given access to corrupted samples. The amount of corruption allowed in the setting considered is such that there is not enough information for retrieving an estimate of the unknown covariance matrix. Moreover, prior work has provided strong evidence that even if the algorithm is allowed to output a list of candidate matrices, it is computationally intractable to form a list that contains a multiplicative spectral approximation of the ground truth covariance matrix. This work overcomes the existing lower bounds (by relaxing the measure of closeness between matrices) and proposes a simple and efficient algorithm that outputs a short list of matrices that contains (with high probability) a matrix that is close to the ground truth in relative Frobenius norm. En route, it is shown that the same algorithm, when the samples are drawn from a Gaussian Mixture Model (GMM) corrupted with a small fraction of outliers, clusters the samples according to the covariance matrix of the corresponding generating Gaussian, up to error in relative Frobenius norm. The latter result, combined with existing results from [BDJKKV22], provides a new algorithm for robustly learning GMMs that does not use semi-definite programming.\n\nThe main algorithm recursively computes the empirical covariances corresponding to each sample (in the current set of samples) and measures the variance of these quantities across the samples (variance of the point-wise empirical covariances). If the variance is small, then the point-wise empirical covariances agree and therefore the empirical covariance of the current set is provided to the output. If the variance is large, the current sample set is either filtered (removing outliers) or split in two sets (each of which will be recursively processed in the next steps) according to whether the empirical covariances seem to have multiple modes when projected on the direction of maximum variance.\n\nSTRENGTHS:\nThe paper proposes a simple and efficient algorithm for a fundamental problem in robust statistics. The proposed algorithm also has an application in learning Mixtures of Gaussian distributions, resolving an open problem in the literature. Although the main idea of the algorithm is simple, the analysis is quite delicate and requires a considerable amount of technical work, as well as careful choice of the overall objective (decoding only up to a list and guarantees in terms of the relative Frobenius norm). Moreover, the presentation of the setting, the results, the techniques and the related work are complete and organized and the proofs are split in several lemmas that might be of independent interest.\n\nWEAKNESSES:\nOne minor presentation issue is that the paper becomes overly technical in some places. For example, in section 1.2 (and in particular in the beginning of the paragraph), it might be preferable to provide some abstract description of the main approach, instead of (or in addition to) defining the particular quantities that the algorithm uses (like the covariance matrix whose eigenvalues determine whether the algorithm stops). In other parts of the paper, however, providing exact (and therefore technical) claims is indeed important (for example in the theorem statements and the proofs).\n\nQUESTIONS:\nMy main question concerns the choice of the relative Frobenius norm. I understand the existence of lower bounds for different metrics, the intuition provided in the paper regarding the relative Frobenius norm, as well as the fact that minimizing the relative Frobenius norm has applications in learning GMMs, but what are other metrics that one might hope to get results for (if any)?\n\nLIMITATIONS:\nYes, limitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}