Title: Robust Mixture Learning when Outliers Overwhelm Small Groups
Venue: NeurIPS 2024
Accepted: yes
Keywords: robust statistics

mixture learning

list-decodable learning

small group

outliers

mean estimation

efficient algorithms
Abstract: We study the problem of estimating the means of well-separated mixtures when an adversary may add arbitrary outliers. While strong guarantees are available when the outlier fraction is significantly smaller than the minimum mixing weight, much less is known when outliers may crowd out low-weight clusters – a setting we refer to as list-decodable mixture learning (LD-ML). In this case, adversarial outliers can simulate additional spurious mixture components. Hence, if all means of the mixture must be recovered up to a small error in the output list, the list size needs to be larger than the number of (true) components. We propose an algorithm that obtains order-optimal error guarantees for each mixture mean with a minimal list-size overhead, significantly improving upon list-decodable mean estimation, the only existing method that is applicable for LD-ML. Although improvements are observed even when the mixture is non-separated, our algorithm achieves particularly strong guarantees when the mixture is separated: it can leverage the mixture structure to partially cluster the samples before carefully iterating a base learner for list-decodable mean estimation at different scales.

Robust Mixture Learning when Outliers Overwhelm Small Groups

Abstract

We study the problem of estimating the means of well-separated mixtures when an adversary may add arbitrary outliers. While strong guarantees are available when the outlier fraction is significantly smaller than the minimum mixing weight, much less is known when outliers may crowd out low-weight clusters - a setting we refer to as list-decodable mixture learning (LD-ML). In this case, adversarial outliers can simulate additional spurious mixture components. Hence, if all means of the mixture must be recovered up to a small error in the output list, the list size needs to be larger than the number of (true) components. We propose an algorithm that obtains order-optimal error guarantees for each mixture mean with a minimal list-size overhead, significantly improving upon list-decodable mean estimation, the only existing method that is applicable for LD-ML. Although improvements are observed even when the mixture is non-separated, our algorithm achieves particularly strong guarantees when the mixture is separated: it can leverage the mixture structure to partially cluster the samples before carefully iterating a base learner for list-decodable mean estimation at different scales.

1 Introduction

Estimating the mean of a distribution from empirical data is one of the most fundamental problems in statistics. The mean often serves as the primary summary statistic of the dataset or is the ultimate quantity of interest that is often not precisely measurable. In practical applications, data frequently originates from a mixture of multiple groups (also called subpopulations) and a natural goal is to estimate the distinct means of each group separately. For example, we might like to use representative individuals to study how a complex decision or procedure would impact different subpopulations. In other applications, such as genetics [1] or astronomy [2] research, finding the means themselves can be a crucial first step towards scientific discovery. In both scenarios, the algorithm should output a list of estimates that are close to the unobservable true means.

However, in practice, the data may also contain outliers, for example due to measurement errors or abnormal events. We would like to find good mean estimates for all inlier groups even when the proportion of such additive adversarial contaminations is larger than some smaller groups that we want to properly represent. The central open question that motivates our work is thus:

What is the cost of efficiently recovering small groups that may be outnumbered by outliers?

More specifically, consider a scenario where the practitioner would like to recover the means of small but significant enough inlier groups which constitute at least w low ∈ (0 , 1) proportion of the

(corrupted) data. If k is the number of such inlier groups, for all i ∈ [ k ] , we then denote by w i ⩾ w low the unknown weight of the i -th group with mean µ i . Further, we use ε to refer to the proportion of additive contamination - the data that comes from an unknown adversarial distribution. The goal is to estimate the unknown means µ i for all i ∈ [ k ] .

Existing works on robust mixture learning such as [3, 4] consider the problem when the fraction of additive adversarial outliers is smaller than the weight of the smallest subgroup, i.e. ε < w low . However, for large outlier proportions where ε ⩾ w low , these algorithms are not guaranteed to recover small clusters with w i ⩽ ε . In this case, outliers can form additional spurious clusters that are indistinguishable from small inlier groups. As a consequence, generating a list of size equal to the number of components would possibly lead to neglecting the means of small groups. In order to ensure that the output contains a precise estimate for each of the small group means, it is thus necessary the estimation algorithm to provide a list whose size is strictly larger than the number of components. We call this paradigm list-decodable mixture learning (LD-ML), following the footsteps of a long line of work on list-decodable learning (see Sections 2 and 5).

Specifically, the main challenge in LD-ML is to provide a short list that contains good mean estimates for all inlier groups. We first note that there is a minimum list size the algorithm necessarily has to output to guarantee that all groups are recovered. For example, consider an outlier distribution that includes several copies of the smallest inlier group distribution with means spread out throughout the domain. Since inlier groups are indisntinguishable from spurious outlier ones, the shortest list that includes means of all inlier groups must be of size at least | L | ⩾ k + ε min i w i . Here, ε min i w i can be interpreted as the minimal list-size overhead that is necessary due to "caring" about groups with weight smaller than ε . The key question is hence how good the error guarantees of an LD-ML algorithm can be when the list size overhead stays close to ε min w i , while being agnostic to w i aside from the knowledge of w low . Furthermore, we are interested in computationally efficient algorithms for LD-ML, especially when dealing with high-dimensional data.

To the best of our knowledge, the only existing efficient algorithms that are guaranteed to recover inlier groups with weights w i ⩽ ε are list-decodable mean estimation (LD-ME) algorithms. LD-ME algorithms model the data as a mixture of one inlier and outlier distribution with weights α ⩽ 1 / 2 and 1 -α respectively. Provided with the weight parameter α , they output a list that contains an estimate close to the inlier mean with high probability. However, for the LD-ML setting, the inlier weights w i are not known and we would have to use LD-ME algorithms with w low as weight estimates for each group. This leads to suboptimal error in particular for large groups, that hence (somewhat counter intuitively) would have to "pay" for the explicit constraint to recover small groups. Furthermore, even if LD-ME were provided with w i , by design it would treat inlier points from other components also as outliers, unnecessarily inflating the fraction of outliers to 1 -w i instead of ε .

Contributions In this paper, we propose an algorithm that (i) correctly estimates the weight of each component only given a lower bound and (ii) does not overestimate proportion of outliers when components are well-separated. In particular, we construct a meta-algorithm that uses mean estimation algorithms as base learners that are designed to deal with adversarial corruptions. This meta-algorithm inherits guarantees from the base learner and any improvement of the latter translates to better results for LD-ML. For example, if the base learner runs in polynomial time, so does our meta-algorithm. Our approach of using the output of weak base learners to achieve better performance is reminiscent of the boosting paradigm that is common in machine learning practice.

Our algorithm achieves significant improvements in error and list-size guarantees for multiple settings. For ease of comparison, we summarize error improvements for inlier Gaussian mixtures in Table 1 (see Theorem 3.3 for the general result regarding distributions with bounded moments). The main focus of our contributions is represented in the second row; that is the setting where outliers outnumber some inlier groups with weight w j ⩽ ε and the inlier components are well-separated , i.e., ∥ µ i -µ j ∥ ≳ 1 √ log 1 w low , where µ i 's are the inlier component means. As we mentioned before, robust mixture learning algorithms, such as [4, 7], are not applicable here and the best error guarantees in prior work is achieved by an LD-ME algorithm, e.g. from [3]. While its error bounds are of order O ( √ log 1 w low ) for a list size of O ( 1 w low ) , our approach guarantees error O ( √ log ε w i ) for a list size

Table 1: For a mixture of Gaussian components N ( µ i , I d ) , we show upper and lower bounds for the error of the i -component given a output list L (of the respective algorithm) min ˆ µ ∈ L ∥ ˆ µ -µ i ∥ . When the error doesn't depend on i , all means have the same error guarantee irrespective of their weight. Note that depending on the type of inlier mixture, different methods in [3] are used as the 'best prior work': robust mixture learning for the first row and list-decodable mean estimation for the rest.

of k + O ( ε w low ) . Remarkably, we obtain the same error guarantees as if an oracle would run LD-ME on each inlier group with the correct weight w i separately (with outliers). Hence, the only cost for recovering small groups is the increased list-size overhead of order O ( ε w low ) . Further, a sub-routine in our meta-algorithm also obtains novel guarantees under no separation assumption, as shown in the third row of Table 1. This algorithm achieves the same error guarantees for similar list size as a base learner that knows the correct weights of the inlier components.

Based on a reduction argument from LD-ME to LD-ML, we also provide information-theoretic (IT) lower bounds for LD-ML. If the LD-ME base learners achieve the IT lower bound (possible for inlier Gaussian mixtures), so does our LD-ML algorithm. In synthetic experiments, we implement our meta-algorithm with the LD-ME base learner from [8] and show clear improvements compared to the only prior method with guarantees, while being comparable or better than popular clustering methods such as k-means and DBSCAN for various attack models.

2 Settings

We now introduce the learning settings that appear in the paper. Let d ∈ N + be the ambient dimension of the data and k ∈ N + be the number of mixture components (inlier groups/clusters).

2.1 List-decodable mixture learning under adversarial corruptions

We focus on mixtures that consist of distributions that are sufficiently bounded in the following sense.

Definition 2.1. Let t ∈ N + be even and let D ( µ ) be a distribution on R d with mean µ . We say that D ( µ ) has sub-Gaussian t -th central moments if for all even s ⩽ t and for every v ∈ R d with ∥ v ∥ = 1 , E x ∼ D ⟨ x -µ, v ⟩ s ⩽ ( s -1)!! .

This class of distributions is closely related to commonly studied distributions in the literature (see, e.g., [5]) with bounded t -th moment. Our requirement for the boundedness of all moments s ⩽ t stems from the fact that our algorithm should adapt to unknown and possibly non-uniform mixture weights.

We assume that we are given samples from a corrupted d -dimensional mixture of k inlier distributions D i ( µ i ) satisfying Definition 2.1, where the mixture is defined as

and ∑ k i =1 w i + ε = 1 , where for all i = 1 , . . . , k , it holds that w i ⩾ w low . Further, an ε > 0 proportion of the data comes from an outlier distribution Q chosen by the adversary with full knowledge of our algorithm and inlier mixture. Samples drawn from D i ( µ i ) constitute the i th inlier cluster . The goal in mixture learning under corruptions as in Eq. (2.1), is to design an algorithm that takes in i.i.d. samples from X and outputs a list L , such that for each i ∈ [ k ] , there exists ˆ µ ∈ L with small estimation error ∥ µ i -ˆ µ ∥ .

To the best of our knowledge, we are the first to study the list-decodable mixture learning problem (LD-ML) that considers the case of large fractions of outliers ε ⩾ min i w i and the goal is to achieve

small estimation errors while the list size | L | remains small. While in robust estimation problems, the fractions of inliers and outliers are usually provided to the algorithm, in mixture learning, the mixture proportions are explicit quantities of interest. Throughout the paper, we hence assume that both the true weights w i of the mixture and the fraction of outliers ε are unknown . Instead, by definition in Eq. (2.1), we assume knowledge of a valid lower bound w low ⩽ min i w i .

Note that when ε ≲ min i w i , the problem is known as robust mixture learning and can be solved with list size | L | = k as discussed in [3, 4, 7]. However, algorithms for robust mixture learning fail when the fraction of outliers becomes comparable to the inlier group size. In the presence of 'spurious' adversarial clusters, it is information-theoretically impossible to output a list L , such that (i) | L | = k and (ii) L contains precise estimate for each true mean.

2.2 Mean estimation under adversarial corruptions

In order to solve LD-ML, we use mean estimation procedures that have provable guarantees under adversarial contamination. Mean estimation can be viewed as a particular case of the mixture learning problem in Eq. (2.1) with k = 1 , the fraction of inliers α = w 1 and the fraction of outliers ε = 1 -α . The mean estimation algorithms we use to solve LD-ML with w low need to exhibit guarantees under a stronger adversarial model, where the adversary can also replace a small fraction (depending on w low ) of the inlier points; see details in Definition B.1. This is a special case of the general contamination model as opposed to the slightly more benign additive contamination model in Eq. (2.1). For different regimes of α we use black-box learners that solve corresponding regime when provided with α .

Robust mean estimation When the majority of points are inliers, we are in the RME setting. Robust statistics has studied this setting with different corruption models and efficient algorithms are known to achieve information-theoretically optimal error guarantees (see Section 5).

List-decodable mean estimation When inliers form a minority, we are in the list-decodable setting and are required to return a list instead of a single estimate. We refer to this setting as cor-kLD ( corrupted known list-decoding ). For mixture learning, α is usually unknown and we need to solve the cor-aLD ( corrupted agnostic list-decoding ) problem (i.e., α is not provided , but instead a lower bound α low ∈ [ w low , α ] is given to the algorithm). Finally, when only additive adversarial contamination is present, as in Eq. (2.1), we recover the standard list-decoding setting studied in prior works (see Section 5) that we call sLD ( simple list-decoding ). In Appendix G we show that two algorithms designed for sLD also exhibit guarantees for cor-kLD for any w low .

3 Main results

We now present our main results for list-decodable and robust mixture learning defined in Section 2. In Section 3.1, we provide algorithmic upper bounds and information-theoretic lower bounds. For the special case of spherical Gaussian mixtures, we show in Section 3.1 that we achieve optimality. Our results are constructive as we provide a meta-algorithm for which these bounds hold.

As depicted in Figure 1, our meta-algorithm (Algorithm 2) is a two-stage process. The outer stage (Algorithm 6) reduces the problem to mean estimation by leveraging the mixture structure and splitting the data into a small collection T of sets T . Each set T ∈ T should (i) contain at most one inlier cluster (and few samples from other clusters) and (ii) the total number of outliers across all sets should be at most O ( εn ) . We then run the inner stage (Algorithm 3) on sets T , which outputs a mean estimate for the inlier cluster in T . First, a cor-aLD algorithm identifies the weight of the inlier cluster and returns the result of a cor-kLD base learner with this weight. Then, if the weight is large, we improve the error via an RME base learner. A careful filtering procedure in both stages achieves the significantly reduced list size and better error guarantees. We require the base learners to satisfy the following set of assumptions.

Assumption 3.1 (Mean-estimation base learners for mixture learning) . Let t be an even integer and consider the corruption setting defined in Definition B.1. Further, let the inlier distribution D ( µ ∗ ) ∈ D where D is the the family of distributions satisfying Definition 2.1 for t . We assume that

Outer stage (Algorithm 6): creates collection T of sets T

Inner stage (Algorithm 3): for each T in T :

Run algorithm for cor-aLD

Improve errors if possible

Base

cor-kLD

Algorithm

Base

RME

Algorithm

(a) for α ∈ [ w low , 1 / 3] in the cor-kLD regime, there exists an algorithm A kLD that uses N LD ( α ) samples and T LD ( α ) time to output a list of size bounded by 1 /α O (1) that with probability at least 1 / 2 contains some ˆ µ with ∥ ˆ µ -µ ∗ ∥ ⩽ f ( α ) , where f is non-increasing.

(b) for α ∈ [1 -ε RME , 1] , with 0 ⩽ ε RME ⩽ 1 / 2 -2 w 2 low in the RME regime, there exists an RME algorithm A R that uses N R ( α ) samples and T R ( α ) time to output with probability at least 1 / 2 some ˆ µ with ∥ ˆ µ -µ ∗ ∥ ⩽ g ( α ) , where g is non-increasing.

Note that the sample and time-complexity functions such as N LD and T LD , might depend on t , for example growing as d t . We emphasize that (i) the guarantees of our meta-algorithm depend on the guarantees of the base learners and (ii) we only require the base learners to work in the wellstudied setting with known fraction of inliers. Corollary 3.4 uses known base learners for Gaussian distributions achieving information-theoretically optimal error bounds. There also exists base learners for distributions beyond Gaussians, such as bounded covariance or log-concave distributions, see, e.g. [9].

3.1 Upper bounds for list-decodable mixture learning

Key quantities that appear in our error bounds are the relative proportion of inliers ˜ w i and outliers ˜ ε i :

These quantities reflect that each set T in the inner stage contains at most one inlier cluster and a small ( ≲ w 2 low ) fraction of points from other inlier clusters. We now present a simplified version of our main result in Theorem 3.3 (see Theorem C.1 for the detailed result) that allows for a more streamlined presentation of the results using the following 'well-behavedness' of f and g .

Assumption 3.2. Let f , g be as defined in Assumption 3.1. For some C > 0 , we assume (i) ε RME ⩾ 0 . 01 , (ii) ∀ x ∈ (0 , 1 / 3] , f ( x/ 2) ⩽ Cf ( x ) , and (iii) ∀ x ∈ [0 . 99 , 1] , g ( x -(1 -x ) 2 ) ⩽ Cg ( x ) .

We are now ready to state the main result of the paper.

Theorem 3.3. Let d, k ∈ N + , w low ∈ (0 , 1 / 2] , and t be an even integer. Let X be a d -dimensional mixture distribution following Eq. (2.1) . Let A kLD and A R satisfy Assumptions 3.1 and 3.2 for some even t . Further, suppose that ∥ µ i -µ j ∥ ≳ √ t (1 /w low ) 4 /t + f ( w low ) for all i = j ∈ [ k ] .

̸

Then there exists an algorithm that, given poly( d, 1 /w low ) · ( N LD ( w low ) + N R ( w low )) i.i.d. samples from X as well as d , k , w low , and t , runs in time poly( d, 1 /w low ) · ( T LD ( w low ) + T R ( w low )) and with probability at least 1 -w O (1) low outputs a list L of size | L | ⩽ k + O ( ε/w low ) where, for each i ∈ [ k ] , there exists ˆ µ ∈ L such that

If the relative weight of the i -th cluster is large, i.e., ˜ ε i ⩽ 0 . 001 , then the error is further bounded by

The proof together with a more general statement, Theorem C.1, can be found in Appendix C.

Note that for a mixture setting with k ⩾ 2 , the assumption w low ⩽ 1 /k ⩽ 1 / 2 is automatically fulfilled. Also, for large weights ˜ w i such that log(1 / ˜ w i ) ≪ t , the t ′ that minimizes √ t ′ (1 / ˜ w i ) 1 /t ′ is smaller than t , and for small weights the minimizer is t ′ = t .

Gaussian case For Gaussian inlier distributions, LD-ME and RME base learners with guarantees for Assumption 3.1 have already been developed in prior work. We can thus readily use them in the meta-algorithm to arrive at the following statement with the relative proportions defined in Eq. (3.1).

̸

Corollary 3.4 (Gaussian case) . Let d, k, w low and t be as in Theorem 3.3. Let X be as in Eq. (2.1) with D i ( µ i ) = N ( µ i , I d ) with µ i 's satisfying ∥ µ i -µ j ∥ ≳ √ log 1 /w low for all i = j ∈ [ k ] . There exists an algorithm that for t = O (log 1 /w low ) , given N = poly( d t , (1 /w low ) t ) i.i.d. samples from X and w low , runs in poly( N ) time and outputs a list L such that with high probability | L | = k + O ( ε/w low ) and, for all i ∈ [ k ] , there exists ˆ µ ∈ L such that

If the relative weight of the i -th cluster is large, i.e. ˜ ε i ⩽ 0 . 001 , then the error is further bounded by

Proof. Theorem 6.12 from [5] provides an LD-ME algorithm A kLD achieving error f ( α ) ⩽ O ( √ t ′ (1 /α ) 1 /t ′ ) for all t ′ ⩽ t . The sample and time complexity scale as poly( d t , (1 /α ) t ) . Also, Theorem 5.1 from [10] provides a robust mean estimation algorithm A R such that for a small enough constant fraction of outliers ε = 1 -α it achieves error g ( α ) = O ((1 -α ) √ log 1 / (1 -α )) with sample complexity ˜ Ω( d/ε 2 ) . Using these A kLD and A R , we recover the desired bounds.

Comparison with prior work We now compare our result with the only previous method that can achieve guarantees in the LD-ML setting with unknown w i . As discussed in [3], algorithms for the simple list-decoding model with α = w low can be used for LD-ML by viewing a single mixture component as the 'ground truth' distribution and effectively treating all other inlier components and original outliers as outliers. Besides requiring a much larger list size of O (1 /w low ) ≫ k + O ( ε/w low ) and error O ( √ log 1 /w low ) , this approach has two drawbacks that manifest in the suboptimal guarantees: 1) For larger clusters i with w i ≫ w low , LD-ME only achieves an error O ( √ log 1 /w low ) . Our result, even without separation assumption, achieves a sharper error bound O ( √ log 1 /w i ) . 2) When the mixture is separated, LD-ME cannot exploit the structure since it still models the data as w low N ( µ i , I d ) + (1 -w low ) Q for each i , so that the algorithm inevitably treats all other true components as outliers. This results in the error O ( √ log 1 /w low ) ≫ O ( √ log 1 / ˜ w i ) = O (1) (when ε ∼ w i ≪ 1 ). We refer to Appendix A for further illustrative examples. As a simple example, consider the uniform inlier mixture with ε = w i = 1 / ( k +1) , where k is large. In this case, previous results have error guarantees O ( √ log k ) , while we obtain error O (1) .

Separation assumption For the problem of learning mixture models, a separation assumption is common in the literature [3, 9, 11, 12]. We require separation ∥ µ i -µ j ∥ ≳ √ t (1 /w low ) 4 /t , which we believe to be sub-optimal for the case of finite t . In cases when ε is small (namely ε ≲ w low ), there exist prior works on clustering allowing smaller separation. Specifically, when t = 2 , a recent work [13] only requires ∥ µ i -µ j ∥ ≳ 1 / √ w low . For a general t ⩾ 4 , [9] succeeds under separation (1 /w low ) 2 /t . We leave the possible relaxation of the separation requirement in the case of general t and large ε for the future work.

In the Gaussian case we require separation ∥ µ i -µ j ∥ ≳ √ log 1 /w low , which is optimal in the uniform ( w i = 1 /k ) case. Indeed, without the separation assumption, even in the noiseless uniform Gaussian case, [6] shows that no efficient algorithm can obtain error asymptotically better than Ω( √ log 1 /w i ) . In Corollary B.5, we prove that the inner stage (Algorithm 3) of our algorithm, without knowledge of w i and separation assumption, achieves with high probability matching error guarantees O ( √ log 1 /w i ) with list size bounded by O (1 /w low ) .

3.2 Information-theoretical lower bounds and optimality

Next, we present information-theoretical lower bounds for list-decodable mixture learning on wellseparated distributions X as defined in Eq. (2.1). We show that our error is optimal as long as the list size is required to be small. Our proof uses a simple reduction technique and leverages established lower bounds in [3] for the list-decodable mean estimation model ( sLD in Section 2).

Algorithm 1 Outer stage, informal (see Algorithm 6)

̸

Proposition 3.5 (Information-theoretic lower bounds) . Let A be an algorithm that, given access to X , outputs a list L that, with probability ⩾ 1 / 2 , for each i ∈ [ k ] contains ˆ µ ∈ L with ∥ ˆ µ -µ i ∥ ⩽ β i .

̸

(a) Consider the case with ∥ µ i -µ j ∥ ≳ (1 /w low ) 4 /t for i = j ∈ [ k ] , D i ( µ i ) having t -th bounded sub-Gaussian central moments and β i ⩽ C (1 /w low ) 1 /t for each i ∈ [ k ] . If for some s ∈ [ k ] it holds that w s ⩽ ε , then algorithm A must either have error bound β s = Ω((1 / ˜ w i ) 1 /t ) or | L | ⩾ k + d -1 .

̸

(b) Consider the case with ∥ µ i -µ j ∥ ≳ √ log 1 /w low for i = j ∈ [ k ] , D i ( µ i ) = N ( µ i , I d ) and β i ⩽ C √ log 1 /w low for each i ∈ [ k ] . If for some s ∈ [ k ] it holds that w s ⩽ ε , then algorithm A must either have error bound β s = Ω( √ log 1 / ˜ w i ) or | L | ⩾ k +min { 2 Ω( d ) , (1 / ˜ w i ) ω (1) } .

In the Gaussian inlier case, Corollary 3.4 together with Proposition 3.5 imply optimality of our meta-algorithm. Indeed, if one plugs in optimal base learners (as in the proof of Corollary 3.4), we obtain error guarantee that matches lower bound. In particular, 'exponentially' larger list size is necessary for asymptotically smaller error. For inlier components with bounded sub-Gaussian moments, [3] obtains information-theoretically (nearly-)optimal LD-ME base learners.

Furthermore, in [3], formal evidence of computational hardness was obtained (see their Theorem 5.7, which gives a lower bound in the statistical query model introduced by [14]) that suggests obtaining error Ω t ((1 / ˜ w s ) 1 /t ) requires running time at least d Ω( t ) . This was proved for Gaussian inliers and the running time matches ours up to a constant in the exponent.

4 Algorithm sketch

We now sketch our meta-algorithm specialized to the case of separated Gaussian components N ( µ i , I d ) and provide intuition for how it achieves the guarantees in Corollary 3.4. In this section, we only discuss how to obtain an error of O ( √ log 1 / ˜ w i ) for each mean when ε ≳ min i w i . We refer to Appendix D for how to achieve the refined error guarantee of O (˜ ε i √ log 1 / ˜ ε i ) when ˜ ε i is small.

As discussed in Section 3.1, running an out-of-the-box LD-ME algorithm for the sLD problem on our input with parameter α = w low would give sub-optimal guarantees. In contrast, our twostage Algorithm 2, equipped with the appropriate cor-kLD and RME base learners as depicted in Figure 1, obtains for each component an error guarantee that is as good as if we had access to the samples only from this component and from the outliers. We now give more details about the outer stage, Algorithm 1, and inner stage, Algorithm 3, and describe on a high-level how they contribute to a short output list with optimal error bound in Corollary 3.4 for large outlier fractions.

4.1 Inner stage: list-decodable mean estimation with unknown inlier fraction

We now describe how to use a black-box cor-kLD algorithm to obtain a list-decoding algorithm A aLD for the cor-aLD mean-estimation setting with access only to α low ⩽ α . A aLD is used

in the proof of Corollary B.5 and plays a crucial role (see Figure 1) in our meta-algorithm. In particular, it deals with the unknown weight of the inlier distribution in each set returned by the outer stage. Note that estimating α from the input samples is impossible by nature. Indeed, we cannot distinguish between potential outlier clusters of arbitrary proportion ⩽ 1 -α and the inlier component. Underestimating the size of a large component would inevitably lead to a suboptimal error guarantee. We now show how to overcome this challenge and achieve an error guarantee O ( √ log 1 /α ) for a list size 1 + O ((1 -α ) /α low ) for the cor-aLD setting. Here we only outline our algorithm and refer to Appendix D for the details.

Algorithm 3 first produces a large list of estimates corresponding to many potential values of α and then prunes it while maintaining a good estimate in the list. In particular, for each ˆ α ∈ A := { α low , 2 α low , . . . , ⌊ 1 / (3 α low ) ⌋ α low } , we run A kLD with parameter ˆ α to obtain a list of means. We append ˆ α to each mean in the list and obtain a list of pairs (ˆ µ, ˆ α ) . We concatenate these lists of pairs for all ˆ α and obtain a list L of size O (1 /α 2 low ) . By design, one element of A is close to the true α , so the list L contains at least one ˆ µ that is O ( √ log 1 /α ) -close - the error guarantee that we aim for and there is indeed at least an α -fraction of samples near ˆ µ . We call such a hypothesis 'nearby".

Finally, we prune this concatenated list by verifying for each ˆ µ whether there is indeed an ˆ α -fraction of samples 'not too far" from it. This is similar to pruning procedures with known α proposed in prior work (see Proposition B.1 in [3]). Our procedure (i) never discards a 'nearby" hypothesis, and outputs a list where (ii) every hypothesis contains a sufficient number of points close to it and (iii) all hypotheses are separated. Property (i) implies that the final error is O ( √ log 1 /α ) and properties (ii) and (iii) imply list size bound 1 + O ((1 -α ) /α low ) . Note that when α < α low , the list size can be simply upper bounded by O (1 /α low ) , see Remark B.4.

4.2 Two-stage meta-algorithm

Even though we could run A aLD on the entire dataset with α low = w low , we would only achieve an error for the i th inlier cluster mean of O ( √ log 1 /w i ) - which can be much larger than O ( √ log 1 / ˜ w i ) - for a list of size O (1 /w low ) . While A aLD takes into account the unknown weight of the clusters, it still treats other inlier clusters as outliers. We now show that if the outer stage Algorithm 1 of our meta-algorithm Algorithm 2 separates the samples into a not-too-large collection T of sets with certain properties, running A aLD separately on each of the sets can lead to the desired guarantees. In particular, let us assume that T consists of potentially overlapping sets such that:

For each inlier cluster C ∗ , there exists one set T ∈ T such that T contains (almost) all points from C ∗ and at most O ( εn ) other points,

It holds that ∑ T ∈ T | T | ⩽ n + O ( εn ) .

By (1), for every inlier cluster C ∗ with a corresponding true weight w ∗ , there exists a set T such that the points from C ∗ constitute at least an ˜ w -fraction of T with ˜ w := Ω( w ∗ / ( w ∗ + ε )) . By Section 4.1, applying A aLD with α low = w low · n/ | T | on such a T then yields a list of size 1+ O ((1 -˜ w ) /w low ) with an estimation error at most O ( √ log 1 / ˜ w ) . If T contains (almost) no inliers, that is, there is no inlier component that should recovered, then A kLD returns a list of size O ( | T | / ( w low n )) .

Now, by the two properties, (almost) all inlier points lie in at most k sets of T , and all other sets of T contain in total at most O ( εn ) points. Hence, concatenating all lists outputted by A aLD applied to all T ∈ T leads to a final list size bounded by k + O ( ε/w low ) .

4.3 Outer stage: separating inlier clusters

We now informally describe the outer stage that produces the collection of sets T with the desiderata described in Section 4.2, leaving the details to Appendix E. The main steps are outlined in pseudocode in Algorithm 1.

Given a set X of N = poly( d t , 1 /w low ) i.i.d. input samples from the distribution Eq. (2.1) with Gaussian inlier components, the first step of the meta-algorithm is to run Algorithm 1 on X and w low with ∆ = O ( √ log 1 /w low ) . Algorithm 1 runs an sLD algorithm on the samples and produces a (large) list of estimates L such that, for each mean, at least one estimate is O ( √ log 1 /w low ) -close to it. It then add sets to T that correspond to these estimates via a dynamic 'two-scale" process.

Specifically, for each ˆ µ ∈ L , we construct two sets S (1) ˆ µ ⊆ S (2) ˆ µ consisting of samples close to ˆ µ . By construction, we guarantee that if S (1) ˆ µ contains a non-negligible fraction of samples from any inlier cluster C ∗ , then S (2) ˆ µ contains (almost) all samples from C ∗ (see Theorem B.7 (ii)).

Now we very briefly illustrate how this process could be helpful in proving properties (1) and (2). Observe that, as long as there exists some ˆ µ with | S (2) ˆ µ | ⩽ 2 | S (1) ˆ µ | , we add S (2) ˆ µ to T and remove the samples from S (1) ˆ µ . Consider one such ˆ µ . For property (1), we merely note that if S (1) ˆ µ contains a part of an inlier cluster C ∗ , then S (2) ˆ µ contains (almost) all of C ∗ , so we add to T a set that contains (almost) all of C ∗ ; otherwise, when we remove S (1) ˆ µ we remove (almost) no points from C ∗ , so (almost) all the points from C ∗ remain in play. For property (2), we merely note that whenever we add S (2) ˆ µ to T , increasing the number of points in it by | S (2) ˆ µ | , we also remove the samples from S (1) ˆ µ , reducing the number of samples by | S (1) ˆ µ | ⩾ | S (2) ˆ µ | / 2 . The proof of the properties uses some additional arguments of a similar flavor, and we defer it to Appendix E.

5 Related work

List-decodable mean estimation Inspired by the list-decoding paradigm that was first introduced for error-correcting codes for large error rates [15], list-decodable mean estimation has become a popular approach for robustly learning the mean of a distribution when the majority of the samples are outliers. A long line of work has proposed efficient algorithms with theoretical guarantees. These algorithms are either based on convex optimization [9, 16], a filtering approach [3, 17], or low-dimensional projections [18]. Near-linear time algorithms were obtained in [19] and [8]. The list-decoding paradigm is not only used for mean estimation but also other statistical inference problems. Examples include sparse mean estimation [20, 21], linear regression [22-24], subspace recovery [25, 26], clustering [27], stochastic block models and crowd sourcing [16, 28].

Robust mean estimation and mixture learning When the outliers constitute a minority, algorithms typically achieve significantly better error guarantees than in the list-decodable setting. Robust mean estimation algorithms output a single vector close to the mean of the inliers. In a variety of corruption models, efficient algorithms are known to achieve (nearly) optimal error

Robust mixture learning tackles the model in Eq. (2.1) with ε ≪ min i w i and aims to output exactly k vectors with an accurate estimate for the population mean of each component [3, 4, 7, 9, 11, 13, 29, 30]. These algorithms do not enjoy error guarantees for clusters with weights w i < ε . To the best of our knowledge, our algorithm is the first to achieve non-trivial guarantees in this larger noise regime.

Robust clustering Robust clustering [31] also addresses the presence of small fractions of outliers in a similar spirit to robust mixture learning, conceptually implemented in the celebrated DBScan algorithm [32]. Assuming the output list size is large enough to capture possible outlier clusters, these methods may also be used to tackle list-decodable mixture learning - however, they do not come with an inherent procedure to determine the right choice of hyperparameters that ultimately output a list size that adapts to the problem.

6 Discussion and future work

In this work, we prove that even when small groups are outnumbered by adversarial data points, efficient list-decodable algorithms can provide an accurate estimation of all means with minimal list size. The proof for the upper bound is constructive and analyzes a plug-and-play meta-algorithm (cf. Figure 1) that inherits guarantees of the black-box cor-kLD algorithm A kLD and RME algorithm A R , which it uses as base learners. Notably, when the inlier mixture is a mixture of Gaussians with identity covariance, we achieve optimality. Furthermore, any new development for the base learners automatically translates to improvements in our bounds.

We would like to end by discussing the possible practical impact of this result. Since an extensive empirical study is out of the scope of this paper, besides the fact that ground-truth means for unsupervised real-world data are hard to come by, we provide preliminary experiments on synthetic data. Specifically, we generate data from a separated k -Gaussian mixture with additive contaminations as in Eq. (2.1) and different types of adversarial distributions (see detailed description in Appendix I). We focus on the regime ε ∼ w i where our algorithms shows the largest theoretical improvements.

Kmeans

Robust Kmeans

DBScan

LD-ME

Ours

Attack 1

Attack 2

0.0

2.5

5.0

7.5

10.0

12.5

15.0

Worst Estimation Error

Attack 1

Attack 2

0

5

10

15

20

25

30

List Size

n/a

0.05

0.10

0.15

0.20

w

low

0

10

20

30

40

50

List Size

Ours

LD-ME

0.05

0.10

0.15

0.20

w

low

0.2

0.3

0.4

0.5

0.6

0.7

Worst Estimation Error

Ours

LD-ME

We then compare the output of our algorithm with the vanilla LD-ME algorithm from [8] with w low = 0 . 02 and (suboptimal) LD-ML guarantees as well as well-known (robust) clustering heuristics without LD-ML guarantees, such as the k -means [33], Robust k -means [34], and DBSCAN [32]. Even though none of these heuristics have LD-ML guarantees, they are commonly used and known to also perform well in practice in noisy settings. In Figure 2 (left), we fix the list size to 10 and plot the errors for the worst inlier cluster, typically the smallest. We compare the performance of the algorithms by plotting the worst-case estimation errors for a given list size and list sizes that algorithms require to achieve a given worst-case estimation error. In Figure 2 (right), we fix the error and plot the minimal list size at which competing algorithms reach the same or smaller worst estimation error. Further details on the experiments are provided in Appendix I. In a different experiment (see Figure 3 and Appendix I.1 for details), we observe that our approach outperforms LD-ME when w low varies, both in achieving smaller list size and smaller estimation error.

Overall, in line with our theory, our method significantly outperforms the LD-ME algorithm, and performs better or on par with the heuristic approaches. Additional experimental comparison and implementation details can be found in Appendix I. Even though these experiments do not allow conclusive statements about the improvement of our algorithm for mixture learning for real-world data, they do provide encouraging evidence that effort could be well-spent on follow-up empirical and theoretical work building on our results. For example, it would be interesting to conduct a more extensive empirical study comparing our algorithm with a variety of robust clustering algorithms. Additionally, practical data often contains components with varying scales. An interesting direction for future work could be to extend our algorithm to handle differently scaled covariances in an agnostic manner.

References

David R Bickel. Robust cluster analysis of microarray gene expression data with the number of clusters determined biologically. Bioinformatics , 19(7):818-824, 2003.

Eric D Feigelson and G Jogesh Babu. Statistical methods for astronomy. arXiv preprint arXiv:1205.2064 , 2012.

Ilias Diakonikolas, Daniel M Kane, and Alistair Stewart. List-decodable robust mean estimation and learning mixtures of spherical Gaussians. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing , pages 1047-1060, 2018.

Ainesh Bakshi, Ilias Diakonikolas, He Jia, Daniel M Kane, Pravesh K Kothari, and Santosh S Vempala. Robustly learning mixtures of k arbitrary Gaussians. In Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing , pages 1234-1247, 2022.

Ilias Diakonikolas and Daniel M Kane. Algorithmic high-dimensional robust statistics . Cambridge University Press, 2023.

Oded Regev and Aravindan Vijayaraghavan. On learning mixtures of well-separated Gaussians. In 2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS) , pages 85-96. IEEE, 2017.

Misha Ivkov and Pravesh K Kothari. List-decodable covariance estimation. In Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing , pages 1276-1283, 2022.

Ilias Diakonikolas, Daniel M Kane, Daniel Kongsgaard, Jerry Li, and Kevin Tian. Clustering mixture models in almost-linear time via list-decodable mean estimation. In Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing , pages 1262-1275, 2022.

Pravesh K Kothari, Jacob Steinhardt, and David Steurer. Robust moment estimation and improved clustering via sum of squares. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing , pages 1035-1046, 2018.

Ilias Diakonikolas, Gautam Kamath, Daniel Kane, Jerry Li, Ankur Moitra, and Alistair Stewart. Robust estimators in high-dimensions without the computational intractability. SIAM Journal on Computing , 48(2):742-864, 2019.

Samuel B Hopkins and Jerry Li. Mixture models, robustness, and sum of squares proofs. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing , pages 1021-1034, 2018.

Allen Liu and Jerry Li. Clustering mixtures with almost optimal separation in polynomial time. In Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing , pages 1248-1261, 2022.

Ilias Diakonikolas, Daniel M Kane, Jasper CH Lee, and Thanasis Pittas. Clustering mixtures of bounded covariance distributions under optimal separation. arXiv preprint arXiv:2312.11769 , 2023.

Michael Kearns. Efficient noise-tolerant learning from statistical queries. Journal of the ACM (JACM) , 45(6):983-1006, 1998.

Peter Elias. List decoding for noisy channels. Technical report, Research Laboratory of Electronics, Massachusetts Institute of Technology, 1957.

Moses Charikar, Jacob Steinhardt, and Gregory Valiant. Learning from untrusted data. In Proceedings of the 49th Annual ACM SIGACT Symposium on Theory of Computing , pages 47-60, 2017.

Ilias Diakonikolas, Daniel Kane, and Daniel Kongsgaard. List-decodable mean estimation via iterative multi-filtering. Advances in Neural Information Processing Systems , 33:9312-9323, 2020.

Ilias Diakonikolas, Daniel Kane, Daniel Kongsgaard, Jerry Li, and Kevin Tian. List-decodable mean estimation in nearly-PCA time. Advances in Neural Information Processing Systems , 34:10195-10208, 2021.

Yeshwanth Cherapanamjeri, Sidhanth Mohanty, and Morris Yau. List decodable mean estimation in nearly linear time. In 2020 IEEE 61st Annual Symposium on Foundations of Computer Science (FOCS) , pages 141-148. IEEE, 2020.

Ilias Diakonikolas, Daniel Kane, Sushrut Karmalkar, Ankit Pensia, and Thanasis Pittas. Listdecodable sparse mean estimation via difference-of-pairs filtering. Advances in Neural Information Processing Systems , 35:13947-13960, 2022.

Shiwei Zeng and Jie Shen. List-decodable sparse mean estimation. Advances in Neural Information Processing Systems , 35:24031-24045, 2022.

Sushrut Karmalkar, Adam Klivans, and Pravesh Kothari. List-decodable linear regression. Advances in Neural Information Processing Systems , 32, 2019.

Prasad Raghavendra and Morris Yau. List decodable learning via sum of squares. In Proceedings of the Fourteenth Annual ACM-SIAM Symposium on Discrete Algorithms , pages 161-180. SIAM, 2020.

Ilias Diakonikolas, Daniel Kane, Ankit Pensia, Thanasis Pittas, and Alistair Stewart. Statistical query lower bounds for list-decodable linear regression. Advances in Neural Information Processing Systems , 34:3191-3204, 2021.

Ainesh Bakshi and Pravesh K Kothari. List-decodable subspace recovery: Dimension independent error in polynomial time. In Proceedings of the 2021 ACM-SIAM Symposium on Discrete Algorithms (SODA) , pages 1279-1297. SIAM, 2021.

Prasad Raghavendra and Morris Yau. List decodable subspace recovery. In Conference on Learning Theory , pages 3206-3226. PMLR, 2020.

Maria-Florina Balcan, Avrim Blum, and Santosh Vempala. A discriminative framework for clustering via similarity functions. In Proceedings of the fortieth annual ACM symposium on Theory of computing , pages 671-680, 2008.

Michela Meister and Gregory Valiant. A data prism: Semi-verified learning in the small-alpha regime. In Conference On Learning Theory , pages 1530-1546. PMLR, 2018.

Ainesh Bakshi, Ilias Diakonikolas, Samuel B Hopkins, Daniel Kane, Sushrut Karmalkar, and Pravesh K Kothari. Outlier-robust clustering of Gaussians and other non-spherical mixtures. In 2020 IEEE 61st Annual Symposium on Foundations of Computer Science (FOCS) , pages 149-159. IEEE, 2020.

Allen Liu and Ankur Moitra. Settling the robust learnability of mixtures of Gaussians. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing , pages 518-531, 2021.

Luis Angel García-Escudero, Alfonso Gordaliza, Carlos Matrán, and Agustín Mayo-Iscar. A review of robust clustering methods. Advances in Data Analysis and Classification , 4:89-109, 2010.

Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In kdd , pages 226-231, 1996.

Stuart Lloyd. Least squares quantization in pcm. IEEE transactions on information theory , pages 129-137, 1982.

Christian Brownlees, Emilien Joly, and Gábor Lugosi. Empirical risk minimization for heavytailed losses. The Annals of Statistics , 43(6):2507-2536, 2015.

Ilias Diakonikolas, Gautam Kamath, Daniel M. Kane, Jerry Li, Ankur Moitra, and Alistair Stewart. Robustly learning a Gaussian: Getting optimal error, efficiently. In Artur Czumaj, editor, Proceedings of the Twenty-Ninth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2018, New Orleans, LA, USA, January 7-10, 2018 , pages 2683-2702. SIAM, 2018.

A Examples

k inlier cluster, c outlier clusters. One tricky adversarial distribution is the Gaussian mixture model itself. In particular, we consider

where the first k Gaussian components are inliers and Q is a GMM with c components, which we call fake clusters. Since all inlier cluster weights are identical, we denote w := w i = 1 / ( k + c ) . Assume that 1 ≪ c ≪ k , which corresponds to ε ≫ w . Then, relative weights are ˜ w = 1 / ( c +1) ≈ 1 /c . Due to large adversary, previous results on learning GMMs cannot be applied, leaving vanilla listdecodable learning. However, the latter also cannot guarantee anything better that Ω( √ tk 1 /t ) even with the knowledge of k , as long as list size is O ( k + c ) , which can be much worse than our guarantees of O ( √ tc 1 /t ) for the same list size.

Their drawback is that they do not utilize separation between true clusters, i.e., for each i , they model the data as

where Q can be 'arbitratily adversarial' for recovering µ i .

Big + small inlier clusters Consider the mixture

where ∥ µ 1 -µ 2 ∥ = Ω( √ log 1 /w ) , w ≪ ε ≪ 1 , and Q is chosen adversarially. In this example we have two inlier clusters, one with large weight ≈ 1 and another with small weight w . Adversarial distribution Q has large weight relative to the small cluster, but still negligible weight compared to the large one.

Previous methods would either (i) recover large cluster with optimal error O ( ε ) (see, e.g., [35]) but miss out small cluster or (ii) recover both clusters using list-decodable mean estimation with known α = w , but with suboptimal errors O ( √ log 1 /w ) and list size O (1 /w ) . In contrast, Corollary 3.4 guarantees list size at most 1 + O ( ε/w ) , error O ( √ log ε/w ) for the small cluster, and error O ( ε √ log 1 /ε ) for the larger. In general, we achieve (i) optimal errors for both clusters and (ii) optimal (up to constants) list size.

B Inner and outer stage algorithms and guarantees

Our meta-algorithm Algorithm 2 assumes black-box access to a list-decodable mean estimation algorithm and a robust mean estimation algorithm for sub-Gaussian (up to the t th moment) distributions. From these we obtain stronger mean estimation algorithms when the fraction of outliers is unknown, and finally stronger algorithms for learning separated mixtures when the fraction of outliers can be arbitrarily large. Our algorithm achieves guarantees with polynomial runtime and sample complexity if the black-box learners achieve the guarantees for their corresponding mean estimation setting. In this section we discuss the corruption model and inner and out stage of the meta-algorithm in detail and prove properties needed for the proof of the main Theorem 3.3.

B.1 Detailed setting

In order to achieve these guarantees, our black-box algorithms need to work under a model in which an adversary is allowed to remove a small fraction of the inliers and to add arbitrarily many outliers. In our proofs, for simplicity of exposition, we require the algorithms to have mean estimation guarantees for a small adversarially removed fraction of w 2 low . Formally, the corruption model as defined as follows.

Definition B.1 (Corruption model) . Let d ∈ N + , and α ∈ [ w low , 1] . Let D be a d -dimensional distribution. An input of size n according to our corruption model is generated as follows:

Algorithm 2 FullAlgorithm

Input: Samples S = { x 1 , . . . , x n } , w low , algorithms A kLD , and A R . Output: List L .

1: Run OuterStage (Algorithm 6) on S and let T be the returned list.

2: L ←∅ .

3: for T ∈ T do

4: Run InnerStage (Algorithm 3) on T with α low = w low · n | T | .

5: Add the elements of the returned list to L .

6: return L .

Algorithm 3 InnerStage

Input: Samples S = { x 1 , . . . , x n } , α low ∈ [ w low , 1] , A kLD , and A R .

Output:

List L .

1: α low ← min(1 / 100 , α low )

2: M ←∅

3: for ˆ α ∈ { α low , 2 α low , . . . , ⌊ 1 / (3 α low ) ⌋ α low } do

4: run A kLD on S with fraction of inliers set to ˆ α

5: add the pair (ˆ µ, ˆ α ) to M for each output ˆ µ

6: Let L be the output of ListFilter (Algorithm 4) run on S , α low , and M

7: for (ˆ µ, ˆ α ) ∈ L do

8: replace ˆ µ by the output of ImproveWithRME (Algorithm 5) run on S , ˆ µ , τ = 40 ψ t (ˆ α ) + 4 f (ˆ α ) , and A R

9: return L

Draw a set C ∗ of n 1 = ⌈ αn ⌉ i.i.d. samples from the distribution D .

An adversary is allowed to arbitrarily remove ⌊ w 2 low n 1 ⌋ samples from C ∗ . We refer to the resulting set as S ∗ with size n 2 = | S ∗ | .

An adversary is allowed to add n -n 2 arbitrary points to S ∗ . We refer to the resulting set as S adv with size n 3 = | S adv | .

If n 3 < n , pad S adv with n -n 3 arbitrary points and call the resulting set S .

Return S .

We call cor-kLD the model when w low and α are given to the algorithm and cor-aLD the model when w low and lower bound α low ⩾ w low are given to the algorithm, such that α ⩾ α low . Note that α is not provided in cor-aLD model.

Note that in Definition B.1 | S | = n and S ∗ constitutes at least an α (1 -w 2 low ) -fraction of S .

B.2 Inner stage algorithm and guarantees

The algorithm consists of three steps: (1) Constructing a list of hypotheses, (2) Filtering the hypotheses, and (3) Improving the hypotheses if α ⩾ 1 -ε RME . For convenience, we restate the InnerStage algorithm introduced in the main text.

Theorem B.2 (Inner stage guarantees) . Let d ∈ N + , w low ∈ (0 , 10 -4 ] , w low ⩽ α low ⩽ α ⩽ 1 , and t be an even integer. Let D ( µ ∗ ) be a d -dimensional distribution with mean µ ∗ ∈ R d and sub-Gaussian t -th central moments.

Consider the cor-aLD corruption model in Definition B.1 with parameters d , w low , α and distribution D = D ( µ ∗ ) . Let A kLD and A R satisfy Assumption 3.1 with high success probability (see Remark B.3).

Then InnerStage (Algorithm 3), given an input of poly( d, 1 /w low ) · ( N LD ( w low ) + N R ( w low )) samples from the cor-aLD corruption model, and access to the parameters d , w low , α low ,

̸

Algorithm 4 ListFilter Input: Samples S = { x 1 , . . . , x n } , α low ∈ [ w low , 1 / 100] , and M = { (ˆ µ 1 , ˆ α 1 ) , . . . , (ˆ µ m , ˆ α m ) } Output: List L 1: define β ( α ) = 10 ψ t ( α ) + f ( α ) 2: let v ij be a unit vector in the direction of ˆ µ i -ˆ µ j for ˆ µ i = ˆ µ j ∈ { ˆ µ, for (ˆ µ, ˆ α ) ∈ M } 3: J ←∅ 4: for (ˆ µ i , ˆ α i ) ∈ M in decreasing order of ˆ α i do 5: if exists j ∈ J , such that ∥ ˆ µ i -ˆ µ j ∥ ⩽ 4 β (ˆ α i ) then continue 6: T i ← ⋂ j ∈ J { x ∈ S, s.t. | v ⊤ ij ( x -ˆ µ i ) | ⩽ β (ˆ α i ) } . 7: if | T i | < 0 . 9ˆ α i n then remove (ˆ µ i , ˆ α i ) from M and continue 8: add i to J 9: for j ∈ J \ { i } do 10: T j ← T j ⋂{ x ∈ S, s.t. | v ⊤ ij ( x -ˆ µ j ) | ⩽ β (ˆ α i ) } 11: if | T j | < 0 . 9ˆ α j n then : 12: remove (ˆ µ j , ˆ α j ) from M 13: rerun ListFilter (Algorithm 4) with the new M 14: return { (ˆ µ i , ˆ α i ) , for i ∈ J }

Algorithm 5 ImproveWithRME

Input: Samples S = { x , . . . , x } , vector ˆ µ , threshold τ , and A

1 n R Output: A vector ˜ µ ∈ R d

1: ˜ β ← τ

2: let ˜ α be the smallest value in [1 -ε RME , 1] that satisfies g (˜ α ) ⩽ ˜ β/ 2 . If none exists, return ˆ µ

3: ˜ µ ← ˆ µ and let µ RME be the output of A R run on S with inlier fraction set to ˜ α .

4: while ∥ ˜ µ -µ RME ∥ ⩽ 3 ˜ β/ 2 do

5: ˜ µ ← µ RME

6: ˜ β ← g (˜ α )

7: let ˜ α ′ be the smallest in [ ˜ α + w 2 low , 1] such that g (˜ α ′ ) ⩽ ˜ β/ 2 . If none exists, break

8: ˜ α ← ˜ α ′

9: let µ RME be the output of A R on S with inlier fraction set to ˜ α

10: return ˜ µ

and t , runs in time poly( d, 1 /w low ) · ( T LD ( w low ) + T R ( w low )) and outputs a list L of size | L | ⩽ 1 + O ((1 -α ) /α low ) such that, with probability 1 -w O (1) low ,

There exists ˆ µ ∈ L such that

If α ⩾ 1 -ε RME , then there exists ˆ µ ∈ L such that

Proof of Theorem B.2 can be found in Appendix D.

Remark B.3. For any r ∈ N , we can increase probabilities of success of A kLD and A R from 1 / 2 to 1 -2 -r in the following way: we increase number of samples by a factor of r , randomly split S into r subsets of equal size, apply A kLD and A R to these subsets and concatenate their outputs. In the proofs we assume that the success probabilities are 1 -w C low for large enough constant C . This increases the size of the list returned by A kLD , the number of samples, and the running time by a factor O (log(1 /w low )) . In particular, we assume that the size of the list returned by A kLD is much smaller than the inverse failure probability.

Remark B.4. In the execution of the meta-algorithm, it may happen that Algorithm 3 is run on set T with almost no inliers, i.e., α < α low . We note that from the analysis (see Appendix D, or [3], Proposition B.1), we always have upper bound | L | = O (1 /α low ) .

An immediate consequence of Theorem B.2 are the following guarantees of directly applying Algorithm 3 to the mixture learning case with no separation. Here we present upper bounds for Algorithm 3, when no separation assumptions are imposed.

Corollary B.5. Let d, k ∈ N + , w low ∈ (0 , 10 -4 ] , and t be an even integer. For all i = 1 , . . . , k , let D i ( µ i ) be a d -dimensional distribution with mean µ i ∈ R d and sub-Gaussian t -th central moments. Let ε > 0 and, for all i = 1 , . . . , k , let w i ∈ [ w low , 1] , such that ∑ k i =1 w i + ε = 1 . Let X be the d -dimensional mixture distribution

where Q is an unknown adversarial distribution that can depend on all the other parameters. Let A kLD and A R satisfy Assumption 3.1.

Then there exists an algorithm that, given poly( d, 1 /w low ) · ( N LD ( w low ) + N R ( w low )) i.i.d. samples from X , and given also d , k , w low , and t , runs in time poly( d, 1 /w low ) · ( T LD ( w low ) + T R ( w low )) and outputs a list L of size | L | = O (1 /w low ) , such that, with probability at least 1 -w O (1) low :

For each i ∈ [ k ] , there exists ˆ µ ∈ L such that

For each i ∈ [ k ] , if w i ⩾ 1 -ε RME , then there exists ˆ µ ∈ L such that

Proof. Proof follows by applying Theorem B.2 to X with α low = w low and treating each component as a corresponding inlier distribution with α = w i . This gives error upper bound for all inlier components, furthermore, since α low = w low , list size can be bounded as | L | ⩽ 1 + O ((1 -α ) /α low ) = O (1 /w low ) .

B.3 Outer stage algorithm and guarantees

In the outer stage, presented in Algorithm 6, we make use of the list-decodable mean estimation algorithm in Theorem B.2 in order to solve list-decodable mixture estimation with separated means. We now present results on the outer stage algorithm. For ease of notation, when it's clear from the context, we drop the indices and refer to elements µ j ∈ M for some j ∈ [ | M | ] as µ and their corresponding sets S (1) j , S (2) j , as defined in lines 6-7 in Algorithm 6, as S (1) , S (2) . Further, for i ∈ [ k ] , let C ∗ i denote the set of points corresponding to the i -th inlier component, also called the i -th inlier cluster.

Theorem B.6 (Outer stage guarantees, beginning of execution) . Let S consist of n i.i.d. samples from X as in the statement of Theorem C.1. Run OuterStage (Algorithm 6) on S and consider the first iteration of the while-loop and for each µ ∈ M , denote the corresponding sets as S (1) , S (2) . Then, with probability at least 1 -w O (1) low , we have that

(i) the list M that A sLD outputs has size | M | ⩽ 2 /w low ,

(ii) for each i ∈ [ k ] , there exists m i ∈ [ | M | ] such that ∣ ∣ ∣ S (1) m i ∩ C ∗ i ∣ ∣ ∣ ⩾ (1 -w 2 low 2 ) | C ∗ i | ,

(iii) for each i ∈ [ k ] and µ ∈ M , we have ∣ ∣ S (1) ∩ C ∗ i ∣ ∣ < w 4 low | C ∗ i | or ∣ ∣ S (2) ∩ C ∗ i ∣ ∣ ⩾ (1 -w 2 low 2 ) | C ∗ i | ,

(iv) for each i ∈ [ k ] and µ ∈ M such that ∣ ∣ S (2) ∩ C ∗ i ∣ ∣ ⩾ w 4 low | C ∗ i | , we have ∑ i ′ ∈ [ k ] \{ i } ∣ ∣ S (2) ∩ C ∗ i ′ ∣ ∣ ⩽ w 4 low n ,

̸

(v) for i = i ′ ∈ [ k ] and for j, j ′ ∈ [ | M | ] , if | S (2) j ∩ C ∗ i | ⩾ w 4 low | C ∗ i | and | S (2) j ′ ∩ C ∗ i ′ | ⩾ w 4 low | C ∗ i ′ | , then S (2) j ∩ S (2) j ′ = ∅ .

Algorithm 6 OuterStage

̸

̸

In words, Theorem B.6 (ii) states that at initialization , OuterStage represents each inlier cluster well, i.e., for each i , the i -th cluster is almost entirely contained in some set S (1) j for some j ∈ [ | M | ] . Next, (iii) states that either S (1) j intersects negligibly some true component, or S (2) j contains almost entirely the same component. Further, (iv) and (v) state that sets that sufficiently intersect with some true component must be separated from other components and each other.

We now introduce some notation to present the next theorem that establishes further guarantees for the algorithm output during execution. For T , a collection of sets that is the output of Algorithm 6, we define

In words, it is the set of inlier components for which a corresponding set with "sufficiently many" points from the i -th component was added to T . It may happen that for a given index i ∈ G , several j ∈ [ | M | ] satisfy S (2) j ∈ T and ∣ ∣ ∣ S (1) j ∩ C ∗ i ∣ ∣ ∣ ⩾ w 4 low | C ∗ i | . We define g i ∈ [ | M | ] to denote the index of the first such set S (2) g i added to T .

Further, we define U i := ( C ∗ i ∩ S (2) g i ) \ S (1) g i to be the set of inlier points from the i -th component, which were not removed from S at the iteration corresponding to g i . Let U := ∪ i ∈ G U i denote the union of such 'left-over' inlier points.

Theorem B.7 (Outer stage guarantees, during execution) . Let S consist of n i.i.d. samples from X as in the statement of Theorem C.1. Run OuterStage (Algorithm 6) on S and consider the moment when the sets S (2) i are added to T . We have that, with probability at least 1 -w O (1) low , all of the following are true:

̸

̸

(iv) if when the else statement is triggered, | S | ⩾ 0 . 1 w low n , then at least a 0 . 4 -fraction of the samples in S are adversarial, or equivalently, | S | ⩽ 2 . 5 εn .

Note that the else statement of OuterStage can only be triggered once, at the end of the execution. In words, Theorem B.7 (i) states that, for i ∈ G , samples from i -th cluster that remained in S after S (1) g i was removed, constitute a small (comparable with ε ) fraction. Further, (ii) states that the sets added to T , corresponding to i ∈ G , almost entirely contain C ∗ i . Finally, (iii) describes the sets that do not correspond to any g i , i ∈ G . These sets must either be small, or contain a significant amount of outlier points in the neighborhood. The proofs of Theorems B.6 and B.7 can be found in Appendix E.

C Proof of Theorem 3.3

In this section, we state and prove a refined version of our main result, Theorem C.1, from which the statement of Theorem 3.3 directly follows.

C.1 General theorem statement

We define

̸

which captures a tail decay of a distribution with sub-Gaussian t -th central moments: P x ∼ D ( ⟨ x -µ, v ⟩ t ⩾ ψ t ( α ) ) ≲ α .

We now state our main result for list-decodable mixture learning. Recall that ε RME is defined in Assumption 3.1.

Theorem C.1 (Main mixture model result) . Let d, k ∈ N + , w low ∈ (0 , 10 -4 ] , and t be an even integer. For all i = 1 , . . . , k , let D i ( µ i ) be a d -dimensional distribution with mean µ i ∈ R d and sub-Gaussian t -th central moments. Let ε > 0 and, for all i = 1 , . . . , k , let w i ∈ [ w low , 1] , such that ∑ k i =1 w i + ε = 1 . Let X be the d -dimensional mixture distribution

where Q is an unknown adversarial distribution that can depend on all the other parameters. Let A kLD and A R satisfy Assumption 3.1. Further, suppose that ∥ µ i -µ j ∥ ⩾ 200 ψ t ( w 4 low )+200 f ( w low ) for all i = j ∈ [ k ] .

Then there exists an algorithm (Algorithm 2) that, given poly( d, 1 /w low ) · ( N LD ( w low ) + N R ( w low )) i.i.d. samples from X , and given also d , k , w low , and t , runs in time poly( d, 1 /w low ) · ( T LD ( w low ) + T R ( w low )) and with probability at least 1 -w O (1) low outputs a list L of size | L | ⩽ k + O ( ε/w low ) such that, for each i ∈ [ k ] , there exists ˆ µ ∈ L such that:

If the relative weight of the i -th inlier cluster is large, i.e., ˜ w i ⩾ 1 -ε RME +2 w 2 low , then there exists ˆ µ ∈ L such that

Further, we assume w low ∈ (0 , 1 / 10000] , since this simplifies some of the proofs. We note that in a mixture with k components we necessarily have w low ⩽ 1 /k . Furthermore, when w low ∈ (1 / 10000 , 1 / 2] , then we obtain the same result by replacing w low with w low / 5000 throughout the statements and the proof. This would only affect both list size and error guarantees by at most a multiplicative constant, which is absorbed in the Big-O notation.

Proof of Theorem 3.3. Proof follows directly from Theorem C.1, by noticing that Assumption 3.2 allows to replace f ( ˜ w i / 10) by Cf ( ˜ w i ) and g ( ˜ w i -3 w 2 low ) by Cg ( ˜ w i ) for some constant C > 0 large enough.

C.2 Proof of Theorem C.1

We now show how to use the results on the inner and outer stage, Theorem B.2 and Theorem B.7 respectively, to arrive at the guarantees for the full algorithm Algorithm 2 in Theorem C.1. For simplicity of the exposition, we split the proof of Theorem C.1 into two separate parts, proving that (i) the output list contains an estimate with small error and that (ii) the size of the output list is small. In what follows we condition on the event E ′ from the proof of Theorem B.7.

(i) Proof of error statement We now prove that, conditioned on the event E , the list L output by Algorithm 2 for each i ∈ [ k ] contains an estimate ˆ µ ∈ L , such that,

∥ ˆ µ -µ i ∥ ⩽ O ( ψ t ( ˜ w i / 10) + f ( ˜ w i / 10)) ,

We start by showing that list-decoding error guarantees as in (1) are achievable for all inlier clusters and proceed by improving the error to (2) with RME base learner. Recall that G is as defined in Eq. (B.1).

Proof of (1) We now show how the output of the base learner and filtering procedure lead to the error in (1). Fix i ∈ [ k ] . Recall that C i denotes the set of w i n points from i -th inlier component with mean µ i .

̸

∑ j = i ∣ ∣ ∣ S (2) g i ∩ C ∗ j ∣ ∣ ∣ ⩽ w 4 low n by Theorem B.6 (iv), and that the total number of adversarial points is at most ( ε + w 4 low ) n .

Therefore, the fraction of points from C ∗ i in S (2) g i is at least (1 -w 2 low ) w i w i + ε + w 3 low , which implies α ⩾ ˜ w i as in Definition B.1. Then, by Theorem B.2, the InnerStage algorithm applied to T leads to error ∥ ˆ µ -µ i ∥ ⩽ O ( ψ t ( ˜ w i / 4) + f ( ˜ w i / 4)) . Otherwise, if i ̸∈ G , when the OuterStage algorithm reaches the else statement, S contains at least (1 -O ( w 3 low )) | C ∗ i | samples from C ∗ i . Indeed, since i / ∈ G , each time we remove points from S , we remove at most w 4 low n points from C ∗ i . By Theorem B.6 (i), we do at most O (1 /w low ) removals, so when the OuterStage algorithm reaches the else statement, S contains at least (1 -O ( w 3 low )) | C ∗ i | samples from C ∗ i .

We showed that samples from C ∗ i make up at least a (1 -w 2 low ) w i n/ | S | fraction of S . Based on this fact we can then use Theorem B.7 (iv) and the assumption on the range of w low to conclude that | S | ⩽ 2 . 5 εn and that the fraction of inliers is at least (1 -w 2 low ) w i / (2 . 5 ε ) . Therefore, S can be seen as containing samples from the corruption model cor-aLD with α at least w i / (2 . 5 ε ) ⩾ w i / (2 . 5( w i + ε )) . Since S is added to T in the else statement, applying InnerStage yields the error bound as in (1).

Proof of (2): Next, we prove that for all inlier components i with large weight, i.e., such that w i / ( w i + ε ) ⩾ 1 -ε RME , there exists a set T ∈ T that consists of samples from the corruption model cor-aLD with α ⩾ w i / ( w i + ε ) -2 w 2 low . Then, running InnerStage , in particular the RME base learner, results in the error bound as in (2) by Theorem B.2 (ii). If i ∈ G , in the previous paragraph we showed that there exists T ∈ T , such that the corresponding α ⩾ w i w i + ε + w 3 low ⩾ w i w i + ε -2 w 2 low . We now prove by contradiction that the case i / ∈ G does not occur. Now assume i / ∈ G so that as we argued before when the else statement is triggered, S contains at least (1 -O ( w 3 low )) | C ∗ i | samples from C ∗ i . By Theorem B.6 (ii), for some m i ∈ [ | M | ] , we have that | S (1) m i ∩ C ∗ i | ⩾ (1 -w 2 low / 2 -O ( w 3 low )) | C ∗ i | and by Theorem B.6 (iv), S (2) m i contains at most w 4 low n samples from other true clusters. Then, since | S (2) m i | > 2 | S (1) m i | , we have that | S (2) m i | contains at least

adversarial samples. Therefore, ε ⩾ (1 -1 . 5 w 2 low ) | C ∗ i | /n , and using that | C ∗ i | ⩾ w i n -w 10 low n , we have ε ⩾ (1 -1 . 5 w 2 low ) w i -w 10 low . However, this contradicts w i / ( w i + ε ) ⩾ 1 -ε RME unless ε RME ⩾ 1 / 2 -2 w 2 low , which is prohibited by the assumptions in Theorem C.1. Therefore whenever w i / ( w i + ε ) ⩾ 1 -ε RME we are in the case i ∈ G .

(ii) Proof of small list size We now prove that on the set E , we have that | L | ⩽ k + O ( ε/w low ) . Here, we need to carefully analyze iterations in the while loop where an inlier component is "selected" for the first time in order to obtain a tight list size bound. Recall that g i corresponds to the index in R that is first selected for the i -th inlier cluster.

First selection of a component: For any i ∈ [ k ] , if i ∈ G , then Theorem B.7 (ii) implies that running InnerStage on S (2) g i produces a list of size at most 1+ O (( | S (2) g i \ C ∗ i | ) / ( w low n )) . Then, over all i ∈ G , these sets S (2) g i contribute to the list size | L | at most k + O ( ∑ k i =1 | S (2) g i \ C ∗ i | / ( w low n ) ) .

Furthermore, by Theorem B.6 (v), all these sets S (2) g i are disjoint and each of them contains at most w 4 low n samples from other true clusters. Therefore ∑ k i =1 | S (2) g i \ C ∗ i | ⩽ εn + O ( w 3 low ) n . Then the contribution to | L | of all these S i 's corresponding to true clusters is at most k + O ( ( ε + w 3 low ) /w low ) . Note that if ε ⩽ w 3 low and w low is small enough, Algorithm 3 actually produces a list of size 1 in each run considered above, so the contribution is exactly k ; otherwise we can bound the contribution by k + O ( ε/w low ) .

Samples left over from a component: Next, all inlier samples that were not removed, i.e., constituting U , can be considered outlier points for the future iterations, which, by Theorem B.7 (i), only increases the outlier fraction to ˜ ε = 3 ε + O ( w 2 low ) . For the same reason as above, without loss of generality, we can consider ε > w 2 low since otherwise, the corresponding list size overhead (for small enough w 2 low ) would again amount to zero.

Clusters of adversarial samples: For iterations where a set S (2) j was added to the final list, which does not correspond to some g i , i ∈ G , Theorem B.7 (iii), states that either (i) at least half of the samples in S (2) j were adversarial, or (ii) the cardinality of the set on which Algorithm 3 was executed is small. In both cases the set S (2) j contributes at most O ( ε/w low ) to the final list size | L | .

List size in the else statement: Finally, when the algorithm reaches the else statement, as argued in the first part, by Theorem B.7 (iv), at that iteration | S | ⩽ O ( ε ) n . Since Algorithm 3 always produces a list of size bounded by O ( | S | / ( w low n )) (see Remark B.4), the contribution to | L | at this iteration is bounded by O ( ε/w low ) .

Overall, we obtain the desired bound on | L | of k + O ( ε/w low ) .

D Proof of Theorem B.2

(i) Proof of error statement We now prove that, with probability 1 -w O (1) low , for the output list L of Algorithm 6,

there exists ˆ µ ∈ L such that

if α ⩾ 1 -ε RME , then there exists ˆ µ ∈ L such that

By Lemma D.1 we have | M | ⩽ 1 /w O (1) low and, with probability at least 1 -w O (1) low , there exists (ˆ µ, ˆ α ) ∈ M such that ˆ α ⩾ α/ 4 and ∥ ˆ µ -µ ∗ ∥ ⩽ f (ˆ α ) . Then Lemma D.2 implies that, with probability at least 1 -| M | 2 w O (1) low , (ˆ µ, ˆ α ) will not be removed from M . Therefore, either (ˆ µ, ˆ α ) ∈ L , or there exists (˜ µ, ˜ α ) ∈ L such that (i) ˜ α ⩾ ˆ α and (ii) ∥ ˜ µ -ˆ µ ∥ ⩽ 4 β (ˆ α ) . The latter case implies that ∥ ˜ µ -µ ∗ ∥ ⩽ 40 ψ t ( α/ 4) + 4 f ( α/ 4) .

For the second part, set first ˜ µ = ˆ µ . Then, in the i th iteration, ˜ µ moves away by at most (3 τ/ 2) / 2 i -1 . Since ∑ ∞ i =1 1 / 2 i ⩽ 1 , the distance between ˜ µ and ˆ µ is always bounded by 3 τ . Now, assume that indeed α ⩾ 1 -ε RME and ∥ ˆ µ -µ ∗ ∥ ⩽ τ . Whenever ˜ α ⩽ α , with high probability A R produces some µ RME such that ∥ µ RME -µ ∗ ∥ ⩽ g (˜ α ) ⩽ ˜ β/ 2 . Furthermore, as long as ˜ α ⩽ α , at the moment

of the while statement check we have ∥ ˜ µ -µ ∗ ∥ ⩽ ˜ β : in the first iteration this is by assumption, and in later iterations it follows because ˜ µ is the former µ RME . Therefore the while statement check passes as long as ˜ α ⩽ α .

There exists the possibility that the algorithm returns or breaks even though ˜ α ⩽ α . If the algorithm returns early, then the error τ achieved by ˆ µ is already within a factor of two of the optimal. If the algorithm breaks, either ˜ α + w 2 low > 1 , case in which ˜ µ already satisfies ∥ ˜ µ -µ ∗ ∥ ⩽ g (1 -w 2 low ) , or else ∥ ˜ µ -µ ∗ ∥ is already within a factor of two of the optimal. Therefore these cases do not affect the error negatively.

Finally, let us consider what happens when ˜ α > α and the while statement check continues to pass. The first time we reach some ˜ α > α , we must have ∥ ˜ µ -µ ∗ ∥ ⩽ ˜ β ⩽ 2 g ( α -w 2 low ) . Then, in later iterations, ˜ µ can move from this estimate by a distance of most 3 ˜ β ⩽ 6 g ( α -w 2 low ) , by the same argument as the argument that ∥ ˜ µ -ˆ µ ∥ ⩽ 3 τ . Overall, at the end we have

The number of runs is at most 1 /w 2 low , so with probability 1 -w O (1) low all runs of A R succeed.

We showed that there exists (ˆ µ, ˆ α ) ∈ L such that ˆ α ⩾ α/ 4 and ∥ ˆ µ -µ ∗ ∥ ⩽ 40 ψ t (ˆ α ) + 4 f (ˆ α ) . This error can increase by running ImproveWithRME with τ = 40 ψ t (ˆ α ) + 4 f (ˆ α ) to at most

Furthermore, if α ⩾ 1 -τ min , this (ˆ µ, ˆ α ) ∈ L satisfies the conditions of ImproveWithRME , so with high probability the error is reduced by running ImproveWithRME with τ = 40 ψ t (ˆ α ) +4 f (ˆ α ) to ∥ ˆ µ -µ ∗ ∥ ⩽ 8 g ( α -w 2 low ) .

(ii) Proof of list size We now prove that | L | ⩽ 1 + O ((1 -α ) /α low ) .

First, assume that α ⩽ 9 / 10 . Since all ˆ α s ⩾ α low , we have that | L | ⩽ 10 / (9 α low ) ⩽ 12(1 -α ) /α low .

For the rest of the proof we assume that α > 9 / 10 . We analyze sets J and T i for i ∈ J at the end of execution of ListFilter . In particular, recall that L = { (ˆ µ i , ˆ α i ) , i ∈ J } . Also, at the end of Algorithm 4 we have the following expression for T i :

̸

where v ij are unit vectors in direction ˆ µ i -ˆ µ j . Select the s ∈ J for which ˆ α s is maximized. By (i), with probability at least 1 -w 60 low , there exists a hypothesis in J with ˆ α ⩾ α/ 4 ⩾ 0 . 2 . Then we have that ˆ α s ⩾ 0 . 2 . In addition, for all hypotheses, ˆ α s ⩽ 1 / 3 . Let j ∈ J be such that j = s . We will show that at least half of the points in T j are adversarial, i.e., | T j | ⩾ 2 | T j ∩ C ∗ | . If this is indeed the case, we can treat all inlier points in all T j as outliers, as it would at most double total number of outlier points in S .

̸

Now, assume that for some j = s , | T j | < 2 | T j ∩ C ∗ | . Note that, because | T s | ⩾ 0 . 9 · 0 . 2 n = 0 . 18 n and | C ∗ | ⩾ 0 . 9 n , | T s ∩ C ∗ | ⩾ 0 . 18 n -0 . 1 n ⩾ 0 . 08 n . Therefore

Also note that Lemma H.2 for radius 10 ψ t (1 / 2) ⩽ 10 ψ t (ˆ α s ) implies that, with exponentially small failure probability,

Since these two sets necessarily intersect, we can bound | v ⊤ sj (ˆ µ s -µ ∗ ) | ⩽ 2 β (ˆ α s ) , implying that | v ⊤ sj (ˆ µ j -µ ∗ ) | ⩾ 2 β (ˆ α j ) , since ∥ ˆ µ s -ˆ µ j ∥ ⩾ 4 β (ˆ α j ) . Thus, if | v ⊤ sj ( x -ˆ µ j ) | ⩽ β (ˆ α j ) , then | v ⊤ sj ( x -µ ∗ ) | > β (ˆ α j ) , implying that

However, Lemma H.2 tells us that with high probability only a small fraction of points in C ∗ satisfies | v ⊤ sj ( x -µ ∗ ) | > β (ˆ α j ) . In particular, applying the lemma with radius 10 ψ t (ˆ α j ) , we get that with exponentially small failure probability,

From eqs. (D.1) and (D.2) it follows that | T j ∩ C ∗ | ⩽ ˆ α j | C ∗ | / 50 . Using that | T j | ⩾ 9ˆ α j n/ 10 ⩾ 9ˆ α j | C ∗ | / 10 by the properties of ListFilter , we obtain

which is a contradiction.

Therefore, for all j ∈ J such that j = s , we have that | T j | ⩾ 2 | T j ∩ C ∗ | . As we said in the beginning, by treating all inlier points in those T j as outliers we at most double total number of outlier points. Since there are at most (1 -α + αw 2 low ) n outlier points and the sets T j are non-intersecting, we get ∑ j ∈ J \{ s } | T j | ⩽ 2(1 -α + αw 2 low ) n . The lower bound on the size | T j | ⩾ 9 α low n/ 10 implies

̸

Note that in InnerStage we set α low = min( α low , 1 / 100) . Therefore, for the original α low , the list size is bounded by 1 + 240(1 -α ) /α low .

Conclusion Combining the probabilities of success of all steps, we get that the algorithm succeeds with probability at least 1 -w O (1) low for some large constant in the exponent. Our algorithm, ignoring the calls to A kLD and A R , has sample complexity and time complexity bounded by poly( d, 1 /w low ) , which gives the desired sample and time complexity when taking A kLD and A R into consideration. This completes the proof of the theorem.

D.1 Auxiliary lemmas and proofs

Lemma D.1 (List initialization) . Let S , α low and α be as in cor-aLD model. If InnerStage (Algorithm 3) is run with S and α low , the size of M is at most 1 /w O (1) low , all (ˆ µ, ˆ α ) ∈ M satisfy ˆ α ⩽ 1 / 3 , and with probability at least 1 -w O (1) low there exists (ˆ µ, ˆ α ) ∈ M such that α/ 4 ⩽ ˆ α ⩽ min( α, 1 / 3) and ∥ ˆ µ -µ ∗ ∥ ⩽ f (ˆ α ) .

Proof. There are at most 1 /α low choices for ˆ α , and for each of them the output of A kLD has size at most 1 /w O (1) low , so | M | ⩽ 1 /w O (1) low . With probability 1 -w O (1) low , A kLD succeeds in all up to 1 /α low runs. Then we are guaranteed to produce one ˆ α with α/ 4 ⩽ ˆ α ⩽ min( α, 1 / 3) , and then A kLD is guaranteed to produce one corresponding ˆ µ with ∥ ˆ µ -µ ∗ ∥ ⩽ f (ˆ α ) .

Lemma D.2 (Good hypotheses are not removed) . Let S , α low and α be as in cor-aLD model. Run ListFilter (Algorithm 4) on S and α low with M obtained from InnerStage (Algorithm 3) and call a hypothesis (ˆ µ, ˆ α ) ∈ M good if ˆ α ⩾ α/ 4 and ∥ ˆ µ -µ ∗ ∥ ⩽ f (ˆ α ) . Then, with probability at least | M | 2 w O (1) low , no good hypothesis is removed from M (including in any of the reruns triggered by the algorithm).

Proof. Let ℓ be an arbitrary iteration of the outer for loop. Then, at the beginning of the ℓ th iteration,

T j ∩ T s = ∅ for any j < s ∈ J ,

| T j | ⩾ 0 . 9ˆ α j n for any j ∈ J ,

| J | ⩽ 10 / (9ˆ α ℓ ) .

Indeed, the second property follows directly from the algorithm.

For the first property, assume that for j < s ∈ J , there exists x ∈ S , such that x ∈ T j ∩ T s . This would imply that | v ⊤ js (ˆ µ j -ˆ µ s ) | ⩽ 2 β (ˆ α s ) , so ∥ ˆ µ j -ˆ µ s ∥ ⩽ 2 β (ˆ α s ) . However, in this case the first 'if' condition would pass and we would not add s to J . Thus, T j ∩ T s = ∅ .

For the third property, note that

which implies | J | ⩽ 10 / (9ˆ α ℓ ) .

Now, let s be the index of a hypothesis with ˆ α s ⩾ α/ 4 and ∥ ˆ µ s -µ ∗ ∥ ⩽ f (ˆ α s ) . If s was skipped in the s th iteration (i.e., there exists j ∈ J with ˆ µ j close to ˆ µ s ), then (ˆ µ s , ˆ α s ) is trivially not removed from M . For the rest of the proof we assume that s is not skipped.

For the sake of the analysis, we introduce the analogue of the sets T s , which we call ˜ T s , defined for points in the set C ∗ (i.e., all inlier points before the adversarial removal), and show that (i) ∣ ∣ ∣ ˜ T s ∣ ∣ ∣ is large and (ii) ∣ ∣ ∣ ˜ T s \ T s ∣ ∣ ∣ is small. In particular, let

where we recall β (ˆ α s ) = 10 ψ t (ˆ α s ) + f (ˆ α s ) . Note that | T s | ⩾ ∣ ∣ ∣ ˜ T s ∣ ∣ ∣ -| C ∗ \ S ∗ | ⩾ ∣ ∣ ∣ ˜ T s ∣ ∣ ∣ -w 2 low | C ∗ | . Also, for any α ′ ⩽ ˆ α s , applying Lemma H.2 with radius 10 ψ t ( α ′ ) , using that ∥ ˆ µ s -µ ∗ ∥ ⩽ f (ˆ α s ) ⩽ f ( α ′ ) and t ⩾ 2 , we get that with exponentially small failure probability,

Consider the s th iteration. Using a union bound over | J | ⩽ 2 /α low directions, and since all ˆ α s ⩾ α low , we get that with exponentially small failure probability

where we used that and | J | ⩽ 10 / (9ˆ α s ) . This implies that

i.e., (ˆ µ s , ˆ α s ) is not removed from M during s th iteration.

The pair (ˆ µ s , ˆ α s ) could also be removed during later iterations, when we recalculate T s by removing points along new directions. However, following a similar argument, we show that still, with high probability, | T s | ⩾ 0 . 9ˆ α s n . Assume that we are now in the k th iteration of the outer cycle, where k > s . We define again ˜ T s and sets A,B :

so that ˜ T s = A ∩ B and again | T s | ⩾ ∣ ∣ ∣ ˜ T s ∣ ∣ ∣ -w 2 low | C ∗ | . It is crucial that we have different right hand sides in the definitions of A i and B i (we wrote them in boldface to emphasize this).

Using a union bound again, we write

Using eq. (D.3), with exponentially small failure probability, for all i ∈ J ,

̸

Next, note that before the last element was added, we had that (i) T i ⋂ T j = ∅ for any i = j ∈ J and (ii) | T i | ⩾ 0 . 9ˆ α i n for any i ∈ J . This implies that ∑ i ∈ J ˆ α i < 10 / 9 + ˆ α last < 19 / 9 , where ˆ α last corresponds to the element which was added last (it might happen that after addition of the last element, we have | T i | < 0 . 9ˆ α i n for several i ∈ J ). Therefore, as before, we obtain that

therefore | T s | ⩾ ∣ ∣ ∣ ˜ T s ∣ ∣ ∣ -w 2 low | C ∗ | ⩾ 0 . 92 | C ∗ | ⩾ 0 . 9ˆ α s n and (ˆ µ s , ˆ α s ) will not be removed from M .

We established that in a single run of the algorithm a good hypothesis is removed with exponentially small probability. The number of good hypotheses is bounded by | M | . Furthermore, the number of runs of the algorithm is also bounded by | M | , since whenever the algorithm is rerun a hypothesis is removed from M . Then, by a union bound, we can bound the probability that any good hypothesis is removed in any run of the algorithm by | M | 2 w O (1) low .

E Proof of outer stage algorithm guarantees in Appendix B.3

Recall that γ = 4 ψ t ( w 4 low ) and γ ′ = 160 ψ t ( w low / 4) + 16 f ( w low / 4) .

E.1 Proof of Theorem B.6

In what follows we condition on the event E that the events under which the conclusions in Lemmas H.2 and H.3 hold and that A sLD succeeds. This event holds with probability 1 -w O (1) low by Assumption 3.1, Remark B.3 and union bound (also see Appendix G).

Proof of Theorem B.6 (i) The list size bound follows from the standard results on A sLD (see [3], Proposition B.1).

̸

Proof of Theorem B.6 (ii) Guarantees of A sLD imply that there exists µ i ∈ M such that ∥ µ i -µ ∗ ∥ ⩽ γ ′ . By Lemma H.3, a (1 -w 2 low / 2) -fraction of the samples in C ∗ are γ -close to µ ∗ along each direction v ij with i = j ∈ [ | M | ] . Then, the same (1 -w 2 low / 2) -fraction of samples are ( γ + γ ′ ) -close to µ i along each direction v ij , so they are included in S (1) i .

Proof of Theorem B.6 (iii) Suppose | S (1) i ∩ C ∗ | ⩾ w 4 low | C ∗ | . Previous point implies that there exists µ j ∈ M be such that ∥ µ j -µ ∗ ∥ ⩽ γ ′ . Then at least an w 4 low -fraction of the samples in C ∗ are ( γ + γ ′ ) -close to µ i in direction µ i -µ j . By Lemma H.2, µ ∗ is also γ -close in direction µ i -µ j to more than a (1 -w 4 low ) -fraction of the samples in C ∗ , so it is γ -close to at least one sample in any w 4 low -fraction of samples in C ∗ . Therefore µ ∗ is also (2 γ + γ ′ ) -close to µ i in direction µ i -µ j . Then ∥ µ i -µ j ∥ ⩽ 2 γ +2 γ ′ and ∥ µ i -µ ∗ ∥ ⩽ 2 γ +3 γ ′ . Again, using Lemma H.3 we obtain that there exists a (1 -w 2 low / 2) -fraction of the samples in C ∗ , which is included in S (2) i .

Proof of Theorem B.6 (iv) Similarly, if | S (2) i ∩ C ∗ | ⩾ w 4 low | C ∗ | , then there exists µ j ∈ M , such that at least an w 4 low -fraction of the samples in C ∗ are (3 γ + 3 γ ′ ) -close to µ i in direction µ i -µ j . By the same arguments as in previous paragraph, we obtain that ∥ µ i -µ j ∥ ⩽ 4 γ +4 γ ′ and ∥ µ i -µ ∗ ∥ ⩽ 4 γ +5 γ ′ .

Then any other true cluster with mean ( µ ∗ ) ′ and set of samples ( C ∗ ) ′ satisfies ∥ µ ∗ -( µ ∗ ) ′ ∥ ⩾ 16 γ +16 γ ′ , so ∥ µ i -( µ ∗ ) ′ ∥ ⩾ 12 γ +11 γ ′ . From guarantees of A sLD , there exists µ ′ j ∈ M such that ∥ µ ′ j -( µ ∗ ) ′ ∥ ⩽ γ ′ . Then ∥ µ i -µ ′ j ∥ ⩾ 12 γ +10 γ ′ . By Lemma H.2, more than an w 4 low -fraction of the samples from ( C ∗ ) ′ are γ -close to ( µ ∗ ) ′ in direction µ i -µ ′ j , so also ( γ + γ ′ ) -close to µ ′ j in direction µ i -µ ′ j , so also (11 γ +9 γ ′ ) -far from µ i in direction µ i -µ ′ j . Then S (2) i selects at most a w 4 low -fraction of the samples from ( C ∗ ) ′ . Overall, S (2) i selects from all other true clusters at most w 4 low n samples.

Proof of Theorem B.6 (v) Note that by the same argument, ∥ µ i -µ ∗ ∥ ⩽ 4 γ +5 γ ′ and ∥ µ i ′ -( µ ∗ ) ′ ∥ ⩽ 4 γ +5 γ ′ . However, ∥ µ ∗ -( µ ∗ ) ′ ∥ ⩾ 16 γ +16 γ ′ , so also ∥ µ i -µ i ′ ∥ ⩾ 8 γ +6 γ ′ , so S (2) i and S (2) i ′ are disjoint by the condition that each selects only samples that are (3 γ +3 γ ′ ) -close along direction µ i -µ i ′ to the respective means µ i and µ i ′ .

E.2 Proof of Theorem B.7

In the sequel, for any i ∈ G , let m i be the index in R after initialization that satisfies Theorem B.6 (ii). We condition on the event E ′ that event E from the proof of Theorem B.6 holds and that both || C ∗ i |-w i n | ⩽ w 10 low n for all i ∈ [ k ] and the number of adversarial points lies in the range εn ± w 10 low n . By Hoeffding's inequality and the union bound, the probability of E ′ is at least 1 -w O (1) low .

Proof of Theorem B.7 (i) Let i ∈ G , and consider the beginning of the iteration when µ g i is selected. Then, using that all previous iterations could have removed at most O ( w 3 low ) | C ∗ i | samples from C ∗ i , we have that

Therefore at the iteration in which µ g i is selected, we still have m i ∈ R . We now discuss two cases: First, consider the case that | S (2) m i | ⩽ 2 | S (1) m i | . Then, because we selected µ g i ∈ M and not µ m i ∈ M it means that | S (1) g i | ⩾ | S (1) m i | ⩾ (1 -w 2 low / 2 -O ( w 3 low )) | C ∗ i | . Note also by Theorem B.6 (iv), the number of samples from other true clusters in S (2) g i is at most w 4 low n . Then the number of adversarial samples in S (2) g i is at least

Then, either ∣ ∣ ∣ S (2) g i \ S (1) g i ∣ ∣ ∣ = O ( w 2 low ) | C ∗ i | and | U i | ⩽ ∣ ∣ ∣ S (2) g i \ S (1) g i ∣ ∣ ∣ = O ( w 2 low ) | C ∗ i | , or ∣ ∣ ∣ S (2) g i \ S (1) g i ∣ ∣ ∣ ≫ w 2 low | C ∗ i | . In the latter case, even if S (2) g i \ S (1) g i consists of adversarial examples only, then, since | S (2) g i | ⩽ 2 | S (1) g i | , U i contains at most double the number of adversarial examples in S (1) g i , i.e. | U i | ⩽ 2 V i where V i denotes the number of adversarial examples in S (1) g i .

Now consider the case that | S (2) m i | > 2 | S (1) m i | . By Theorem B.6 (iv), the number of samples from true clusters in S (2) m i is at most | C ∗ i | + w 4 low n ⩽ 1 . 02 | S (1) m i | , so the number W i of adversarial samples in S (2) m i is at least W i ⩾ | S (2) m i | -1 . 02 | S (1) m i | ⩾ 0 . 98 | S (1) m i | ⩾ 0 . 96 | C ∗ i | . Then, | U i | = ∣ ∣ ∣ ( C ∗ i ∩ S (2) g i ) \ S (1) g i ∣ ∣ ∣ ⩽ | C ∗ i | ⩽ 2 W i .

Finally note that by Theorem B.6 (v), the sets S (2) g i and S (2) m i are disjoint from any other sets S (2) g j and S (2) m j that correspond to another component C ∗ j . Therefore, the number of adversarial examples in the S (2) m i in the second case and S (2) g i in the first case is smaller than the total number of adversarial examples, i.e.

Therefore, we directly obtain

Proof of Theorem B.7 (ii) Each iteration before g i was selected, removed at most w 4 low | C ∗ i | samples from C ∗ i , so all previous iterations removed at most O ( w 3 low ) | C ∗ i | samples from C ∗ i . Then, by Lemma B.6 (iii), S (2) g i contains at least (1 -w 2 low / 2 -O ( w 3 low )) | C ∗ i | samples from C ∗ i . The statement follows then since on the event E ′ , we have w ∗ n -w 10 low n ⩽ | C ∗ | ⩽ w ∗ n + w 10 low n .

Proof of Theorem B.7 (iii) Here, either for all i ∈ [ k ] , | S (1) j ∩ C ∗ i | < w 4 low | C ∗ i | or i ∈ G and the algorithm had already selected in a previous iteration µ g i ∈ M with | S (1) g i ∩ C ∗ i | ⩾ w 4 low | C ∗ i | . Consider a first case, in which | S (1) j ∩ C ∗ i | < w 4 low | C ∗ i | for all i ∈ [ k ] . Then the total number of samples from true clusters in S (1) j is at most w 4 low n . Using that | S (1) j | > 100 w 4 low n , it follows that more than half of the samples in S (1) j are adversarial.

The second case is that | S (1) j ∩ C ∗ i | ⩾ w 4 low | C ∗ i | for some i ∈ G for which in a previous iteration g i we had that | S (1) g i ∩ C ∗ i | ⩾ w 4 low | C ∗ i | . Note that at most w 2 low | C ∗ i | / 2 of the samples in S ∩ C ∗ i are not considered adversarial at this point (the ones that were outside S (2) g i ). Also, by Theorem B.6 (iv), S (1) j contains at most w 4 low n samples from other true clusters. Therefore either more than half of the samples in S (1) j are considered adversarial or

Proof of Theorem B.7 (iv) Suppose that when the algorithm reaches the else statement we have for some i ∈ [ k ] that i ∈ R and | S (1) m i ∩ C ∗ i | ⩾ 20 w 2 low | C ∗ i | . We have that | S (2) m i ∩ C ∗ i | is at most | S (1) m i ∩ C ∗ i | + w 2 low | C ∗ i | / 2 , where we use that by Theorem B.6 (ii), at most w 2 low | C ∗ i | / 2 samples can fail to be captured by S (1) m i . By Theorem B.6 (iv), furthermore, the number of samples from other true clusters in S (2) m i is at most w 4 low n . Therefore, using that | S (2) m i | > 2 | S (1) m i | , the number of adversarial samples in S (2) m i is at least

where in the last inequality we used that | S (2) m i | > 100 w 4 low n . Let V be the union, over all i ∈ [ k ] , of all sets S (2) m i such that i ∈ R and | S (1) m i ∩ C ∗ i | ⩾ 20 w 2 low | C ∗ i | . Theorem B.6 (v) gives that all such sets S (2) m i are disjoint. Therefore at least a 0 . 44 -fraction of the samples in V are adversarial.

Consider now for some i ∈ [ k ] how many samples from S ∩ C ∗ i can be outside V when the algorithm reaches the else statement. By Theorem B.6 (ii), S (1) m i can fail to capture at most w 2 low | C ∗ i | / 2 samples from C ∗ i , and we have no guarantee that these samples are in V . Consider now the samples in S (1) m i ∩ C ∗ i . If i ∈ R , we may miss up to 20 w 2 low | C ∗ i | of these samples if | S (1) m i ∩ C ∗ i | < 20 w 2 low | C ∗ i | , because in this case we do not include S (2) m i in V . On the other hand, if i ̸∈ R , there are at most 100 w 4 low n samples in S (1) m i ∩ C ∗ i . Then the total number of samples from S ∩ C ∗ i outside V is at most w 2 low | C ∗ i | / 2 + 20 w 2 low | C ∗ i | +100 w 4 low n . Summed across all i ∈ [ k ] , this makes up at most 21 w 2 low n samples.

Overall, the number of adversarial samples in S when the algorithm reaches the else statement is at least

0 . 44 | V | +( | S | - | V | -21 w 2 low n ) = | S | -0 . 56 | V | -21 w 2 low n ⩾ 0 . 44 | S | -21 w 2 low n ⩾ 0 . 4 | S | where in the last inequality we also used that | S | ⩾ 0 . 1 w low n .

F Proof of Proposition 3.5

We now prove lower bounds for the case of Gaussian distributions and distributions with t -th subGaussian moments.

F.1 Case b): For the Gaussian inliers

We first focus on the case when D i ( µ i ) = N ( µ i , I ) . The proof goes through an efficient reduction from the problem considered by Proposition F.1 to the problem solved by algorithm A .

Proposition F.1 ([5], Proposition 5.11) . Let D be the class of identity covariance Gaussians on R d and let 0 < α ⩽ 1 / 2 . Then any list-decoding algorithm that learns the mean of an element of D with failure probability at most 1 / 2 , given access to (1 -α ) -additively corrupted samples, must either have error bound β = Ω( √ log 1 /α ) or return min(2 Ω( d ) , (1 /α ) w (1) ) many hypotheses.

First, we describe the means of the components in the input distribution to algorithm A . Let ¯ µ 1 , . . . , ¯ µ k -1 ∈ R d be any set of k -1 points with pairwise separation larger than 2 C √ log 1 /w low . Then let µ k = (¯ µ, 0) ∈ R d +1 and µ i = (¯ µ i , 2 C √ log 1 /w low +1) ∈ R d +1 for all i ∈ [ k -1] . Then µ 1 , . . . , µ k also have pairwise separation larger than 2 C √ log 1 /w low .

Then, given n points y 1 , . . . , y n ∈ R d as in the input to the problem in Proposition F.1 (i.e. (1 -α ) -additively corrupted samples), we generate n points that we give as input to algorithm A as follows: let S = { 1 , . . . , n } , and then for each of the n points, draw i ∼ Unif { 1 , . . . , k } and generate the point as follows:

if i ∈ [ k -1] , sample the point from N ( µ i , I d +1 ) ,

if i = k , sample j ∼ S uniformly at random, remove j from S , sample g ∼ N (0 , 1) , and let the point be ( y j , g ) ∈ R d +1 .

We note that this construction simulates an input sampled i.i.d. according to the mixture 1 k N ( µ 1 , I d +1 ) + . . . + 1 k N ( µ k -1 , I d +1 ) + α k N ( µ k , I d +1 ) + 1 -α k Q ′ for some Q ′ . Then with success probability at least 1 / 2 running A on this input with w low = α k returns a list L such that there exists ˆ µ ∈ L with ∥ ˆ µ -µ k ∥ ⩽ β k . Note that this implies that ∥ (ˆ µ ) 1: d -¯ µ ∥ ⩽ β k . Finally, we create a pruned list L ′ as follows: initialize L ′ = L and then for each i ∈ [ k -1] remove all ˆ µ ∈ L ′ such that ∥ ˆ µ -µ i ∥ ⩽ C √ log 1 /w low . Then we return L ′ as the output for the original problem in Proposition F.1.

̸

Let us analyze now this output. The separation between the means ensures that any hypothesis ˆ µ ∈ L that is C √ log 1 /w low -close to µ k is not removed in the pruning. Therefore L ′ continues to contain a hypothesis ˆ µ such that ∥ (ˆ µ ) 1: d -¯ µ ∥ ⩽ β k . Then, if β k = Ω( √ log 1 /α ) and | L ′ | < min { 2 Ω( d ) , (( w k + ε ) /w k ) ω (1) } , this reduction violates the lower bound of Proposition F.1. Therefore we must have either β k = Ω( √ log 1 /α ) or | L ′ | ⩾ min { 2 Ω( d ) , (1 / ˜ w k ) ω (1) } .

Finally, we show that these lower bounds on β k and | L ′ | imply the desired lower bound for A . Consider first the case: β k = Ω( √ log 1 /α ) . Note that in the input to algorithm A we have ˜ w k = α . Therefore β k = Ω( √ log 1 /α ) corresponds to the desired lower bound in the lemma statement. Consider second the case: | L ′ | ⩾ min { 2 Ω( d ) , (1 / ˜ w k ) ω (1) } . We note that, for each i ∈ [ k -1] , the original list L must contain some ˆ µ ∈ L such that ∥ ˆ µ -µ i ∥ ⩽ C √ log 1 /w low . Furthermore, because the means µ i have pairwise separation larger than 2 C √ log 1 /w low , the original list L must contain at least k -1 means of this kind. However, all of these means are removed in the pruning procedure, so | L | ⩾ k -1 + | L ′ | , so | L | ⩾ k -1 + min { 2 Ω( d ) , (1 / ˜ w k ) ω (1) } . This matches the desired lower bound in the lemma statement. (The choice to make the hidden mean the k -th mean was without loss of generality, as the distribution is invariant to permutations of the components.)

F.2 Case a): For distributions with t -th sub-Gaussian moments

The proof for the case when D i ( µ i ) has sub-Gaussian t -th central moments employs the same reduction scheme, but reduces from Proposition F.2.

Proposition F.2 ([5], Proposition 5.12) . Let D be the class of distributions on R d with bounded t -th central moments for some positive even integer t , and let 0 < α < 2 -t -1 . Then any list-decoding algorithm that learns the mean of an element of D with failure probability at most 1 / 2 , given access to (1 -α ) -additively corrupted samples, must either have error bound β = Ω( α -1 /t ) or return a list of at least d hypotheses.

Furthermore, in [3], formal evidence of computational hardness was obtained (see their Theorem 5.7, which gives a lower bound in the statistical query model introduced by [14]) that suggests obtaining error Ω t ((1 / ˜ w s ) 1 /t ) requires running time at least d Ω( t ) . This was proved for Gaussian inliers and the running time matches ours up to a constant in the exponent.

G Stability of list-decoding algorithms

In this section we discuss two of the existing list-decodable mean estimation algorithms for identitycovariance Gaussian distributions and show that they also work when a w 2 low -fraction of the inliers is adversarially removed.

First, we consider the algorithm in Theorem 3.1 in [3]. A central object in their analysis is an ' α -good multiset", which is a multiset of samples such that all are within distance O ( √ d ) of each other and at least an α -fraction of them come from a (1 -Ω( α )) -fraction of an i.i.d. set of samples from a Gaussian distribution N ( µ, I d ) . Then their algorithm essentially works as long as the input contains an α -good multiset. For our case, after the removal of a w 2 low -fraction of inliers, the input essentially continues to contain a (1 -w 2 low ) α -good multiset, so the algorithm continues to work in our corruption model.

Second, we consider the algorithm in Theorem 6.12 in [5]. The main distributional requirement of their algorithm is that E x,y ∼ S ∗ [ p 2 ( x -y )] ⩽ 2 E g,h ∼ N (0 ,I d ) [ p 2 ( g -h )] for all degree-( t/ 2) polynomials p , where S ∗ is the set of inliers. Concentration arguments give with high probability that E x,y ∼ C ∗ [ p 2 ( x -y )] ⩽ 1 . 5 E g,h ∼ N (0 ,I d ) [ p 2 ( g -h )] . Furthermore, the distribution over x, y ∼ S ∗ can be seen as a (1 -w 2 low ) 2 -fraction of the distribution over x, y ∼ C ∗ . Then Fact G.1, which follows by standard probability calculations, also gives that any event under the former distribution can be bounded in terms of the second distribution:

Fact G.1. For any event A ,

Proof. We have that

where we used that ( t -1)!! ⩽ t t/ 2 = √ t t .

Lemma H.2. Let D be a d -dimensional distribution with mean µ ∗ ∈ R d and sub-Gaussian t -th central moments with parameter 1 . Let C ∗ be a set of i.i.d. samples drawn from D . Fix a unit vector v ∈ R d . Then with probability at least 1 -exp ( -2 | C ∗ | ( √ t R ) 2 t ) ,

Proof. The result follows by Lemma H.1 and a Binomial tail bound.

where probabilities are taken over a uniform sample from S ∗ and C ∗ respectively.

Overall we obtain

so for w low small enough we have E x,y ∼ S ∗ [ p 2 ( x -y )] ⩽ 2 E g,h ∼ N (0 ,I d ) [ p 2 ( g -h )] and their algorithm continues to work in our corruption model.

H Concentration bounds

In this section we prove some concentration bounds essential to our analysis.

Lemma H.1. Let D be a d -dimensional distribution with mean µ ∗ ∈ R d and sub-Gaussian t -th central moments with parameter 1 . Fix a unit vector v ∈ R d . Then

Lemma H.3. Let D be a d -dimensional distribution with mean µ ∗ ∈ R d and sub-Gaussian t -th central moments with parameter 1 . Let C ∗ be a set of i.i.d. samples drawn from D . Fix m unit

Proof. By Lemma H.1 and a union bound over the m directions, we get

Then the result follows by a Binomial tail bound.

I Experimental details

Adversarial line and adversarial clusters The following figure illustrates the adversarial distributions used in Figure 2 and further in this section.

True Cluster

Fake Cluster

Data Distribution We consider a mixture of k = 7 well-separated ( ∥ µ i -µ j ∥ ⩾ 40 ) d = 100 dimensional inlier clusters whose subgroup sizes range from 0 . 3 to 0 . 02 . The experiments are conducted once using a Gaussian distribution and once using a heavy-tailed t-distribution with five degrees of freedom for both inlier and adversarial clusters. In Figure 6 the latter suggests that our algorithm works comparatively well even for mixture distributions which do not fulfill our assumptions. We set w low = 0 . 02 and ε = 0 . 12 so that it is larger than the smallest clusters but smaller than the largest ones and set the total number of data points to 10000 . The Gaussian noise model simply computes the empirical mean and covariance matrix of the clean data and samples 1200 noisy samples from a Gaussian distribution with this mean and covariance. The adversarial cluster model and the adversarial model are as depicted in Figure 4.

Attack distributions We consider three distinct adversarial models (see Figure 4 for reference).

Adversarial clusters : After sampling the inlier cluster means, we choose the cluster with the smallest weight. Let µ s denote its mean. Then, we sample a random direction v c with ∥ v c ∥ = 10 . After that, we sample three directions v 1 , v 2 and v 3 with ∥ v i ∥ = 10 . Then we put three additional (outlier) clusters with means at µ s + v c + v i . This roughly corresponds to the right picture in Figure 4. The samples for each adversarial cluster are drawn from a distribution that matches the covariance of the inlier clusters, with the sample size being twice as large as of the affected inlier cluster.

Adversarial line : After sampling the inlier cluster means, we again choose the cluster with the smallest weight. Let µ s denote its mean. Then, we sample a random direction v c with ∥ v c ∥ = 10 . We put three additional (outlier) clusters with means at µ s + v c , µ s +2 v c and µ s +3 v c , which form a line as shown in Figure 4. The samples are drawn similarly to the adversarial clusters, with the difference that the covariance is scaled by a factor of 5 in the direction of the line.

.

Kmeans

Robust Kmeans

DBScan

LD-ME

Ours

Attack 1

Attack 2

Attack 3

0

10

20

30

40

50

60

Worst Estimation Error

Attack 1

Attack 2

Attack 3

0

5

10

15

20

25

30

List Size

n/a

n/a

Kmeans

Robust Kmeans

DBScan

LD-ME

Ours

Attack 1

Attack 2

Attack 3

0

10

20

30

40

50

60

Worst Estimation Error

Attack 1

Attack 2

Attack 3

0

5

10

15

20

25

30

List Size

n/a n/a

Gaussian adversary : Here we simply introduce noise matching the empirical mean and covariance of all inlier data (i.e., as if all inlier clusters are generated from the same Gaussian distribution).

Note that in the first and second attack, the adversary creates clusters that do not respect the separation assumption of the true inlier clusters: either adversarial clusters are placed around the smallest inlier cluster (Adversarial Cluster), or the adversarial clusters form a line, pointing out in some fixed direction (Adversarial Line).

Implementation details We implement the list-decodable mean estimation base learner in our InnerStage algorithm (Algorithm 3) based on [8]. It leverages an iterative multi-filtering strategy and one-dimensional projections. In particular, we use the simplified gaussian version of the algorithm. It is designed for distributions sampled from a Gaussian but also shows promising results for the experiments involving a heavy-tailed t-distribution as depicted in Figure 6. The robust mean estimator used to improve the mean hypotheses for large clusters is omitted in our implementation.

Hyper-parameter search and experiment execution The hyper-parameters of our algorithm are tuned beforehand based on the experimental setup. For the comparative algorithms, hyper-parameter searches are conducted within each experiment after initial tuning. For our algorithm, key parameters include the pruning radius γ used in the OuterStage routine (Algorithm 6) and β used in the InnerStage (Algorithm 4). In addition, parameters for the LD-ME base learner, such as the cluster concentration threshold, also require careful selection, resulting in a total of 7 parameters. The tuning for these was performed using a grid search comparing about 250 different configurations. Similarly, we independently tune the vanilla LD-ME algorithm, which we run with w low as weight parameter. For

10 1

List Size

10 0

10 1

10 2

Worst Estimation Error

Attack 1

Kmeans

Robust Kmeans

DBScan

Ours

LD-ME

10 1

List Size

10 0

10 1

10 2

Worst Estimation Error

Attack 2

Kmeans

Robust Kmeans

DBScan

Ours

LD-ME

10 1

6×10 0

2×10 1

3×10 1

4×10 1

List Size

10 0

10 1

10 2

Worst Estimation Error

Attack 3

Kmeans

Robust Kmeans

DBScan

Ours

LD-ME

DBSCAN, we optimize the list size and error metrics by searching over a range of 100 values for ε , which controls the maximum distance between samples considered in the same neighbourhood. The minimum samples threshold, which validates the density based clusters, is pretuned beforehand and adjusted based on w low . For k -means and its robust version, utilizing a median-of-means weighting scheme, we explore 21 values for k , including the true number of clusters. Each parameter setting is executed 100 times to account for stochastic variations in the algorithmic procedures, such as k -means initialization. The list size and worst estimation error for each list of clusters obtained is visualized exemplarily for one iteration of the experiment in Figure 7. The plot provides insight into how the different algorithms perform and vary with different list sizes.

Evaluation details Note that we have two sources of randomness: the data is random and also the algorithms themselves are random (except DBSCAN). For a clear comparison, we sample and fix one dataset for each attack model. we plot the performance of 100 runs of each algorithm for each parameter setting, each time recording the returned list size together with the worst estimation error max i ∈ [ k ] min ˆ µ ∈ L ∥ µ i -ˆ µ ∥ . Then we either (i) report the worst estimation error for all runs with constrained list size (we pick the list size most frequently returned by our algorithm, specifically 7 or 10 in our experiments) (see Figure 5, left), or (ii) report the smallest list size required to achieve the same or smaller worst estimation error (we pick the 75 th quantile of errors of our algorithm for a threshold) (see Figure 5, right). Under size constraint (i), the bar plots correspond to the median over the runs, with error bars indicating the 25 th and 75 th quantiles. Under error constraint (ii), the bar plots represent the minimum list size for which the median over the runs falls below the threshold, while the error bars show the minimum list size for which the 25 th and 75 th quantiles meet the constraint. Note that 'n/a' indicates that, within the scope of our parameter search, no list size achieves an error below the specified constraint.

In Figure 6 we study the numerical stability of our approach. In particular, whether the performance degrades when inlier distribution does not satisfy required assumptions. We observe that if one uses our meta-algorithm with base learner designed for Gaussian inliers, we still obtain stable results even in the case of heavy-tailed inlier distribution.

I.1 Variation of w low

To study the effect of varying w low input on the performance of our approach and LD-ME, we introduce a new noise model. As illustrated in Figure 8, we consider a mixture of k = 3 wellseparated clusters: one small cluster with a weight of 0 . 045 and two large clusters, each with a weight of 0 . 2 . We place two adversarial clusters (see paragraph on attack distributions for details): one near the small cluster and another near one of the large clusters. Furthermore, uniform noise is introduced,

True Cluster

Fake Cluster

Uniform noise

0.05

0.10

0.15

0.20

w

low

0

20

40

60

80

Worst Estimation Error

Ours

LD-ME

0.05

0.10

0.15

0.20

w

low

0.2

0.3

0.4

0.5

0.6

0.7

Worst Estimation Error

Ours

LD-ME

0.05

0.10

0.15

0.20

w

low

0

10

20

30

40

50

List Size

Ours

LD-ME

spanning the range of the data generated by the inlier and its nearby outlier cluster and accounting for 10% of the data in this region. Overall, ε = 0 . 56 and we draw 22650 samples from this mixture distribution.

For both algorithms we run 100 seeds for each w low ranging from 0.02 to 0.2, which corresponds to the weight of the largest inlier cluster. In Figure 9, we plot the median estimation error with error bars showing the 25 th and 75 th quantiles for the small cluster (top left) and the large cluster near the outlier cluster (top right). As expected from our theoretical results, we observe that our algorithm performs roughly constant in estimating the mean of the large cluster, regardless of the initial w low . Meanwhile, the estimation error of LD-ME increases as w low decreases further below the true cluster weight. Furthermore, the plots show that our approach does consistently outperform LD-ME in terms of both worst estimation error and list size. Figure 10 also compares the performance of the clustering algorithms in this experimental setup with results similar to the ones obtained in the previous experimental settings.

I.2 Computational resources

Our implementation of the algorithm and experiments leverages multi-threading. It utilizes CPU resources of an internal cluster with 128 cores, which results in a execution time of about 5 minutes for a single run of the experiment for one noise model with 10000 samples. We remark that classic

0

10 1

List Size

10 0

10 1

Worst Estimation Error

Kmeans

Robust Kmeans

DBScan

Ours

LD-ME

Kmeans

Robust Kmeans

DBScan

LD-ME

Ours

2

4

6

8

10

Worst Estimation Error

2

4

6

8

10

List Size

n/a

n/a

n/a

0

approaches like k -means and DBSCAN perform fast and the most time-consuming part is the execution of the LD-ME base learner. Given our experimental setup with three noise models, it takes about 15 minutes to reproduce all our results for one data distribution.

Attack 1

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We summarized our contributions in Table 1 in the introduction, and all the claims made there (including more general results) appear in Section 3 and Appendix C.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of our theoretical results in Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Our assumptions are stated in Sections 2 and 3, in particular Assumption 3.1. The complete proof is given in Appendix C.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide experimental details, including explanation of the adversarial noise distribution in Appendix I.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We uploaded zip archive containing the code together with instructions on how to reproduce the experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide short version of experimental details in Section 6 and more detailed one in Appendix I.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: As described in Appendix I, we plot the median of the metric (error or list size) accompanied by the 25% and 75% percentiles, acting as error bars. The statistics was collected over 100 reruns of all algorithms under comparison.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide experimental details, including computational resources, in Appendix I.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We reviewed the NeurIPS Code of Ethics and confirm that our paper respects it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Main focus of our result is to deepen our theoretical understanding of fundamental problems in statistics, such as mixture learning.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: To the best of our judgement, our paper does not pose misuse risks, as it has theoretical focus.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: Our work does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our work does not use release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our work does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our work does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors introduce the list-decodable mixture learning problem, which can be considered as the extension of the list-decodable mean estimation problem. In such case, data is drawn from a weighted mixture of $k$ inlier distributions and an adversarial outlier distribution. A notable aspect of this paper’s problem setting is that the fraction of outlier $\\varepsilon$ can exceed the  fraction of certain inlier clusters $w_i$s. As a result, the required list size is at least $k+\\varepsilon/\\min w_i$. In this paper, The primary objective of this study is to compute a compact list $L$ of means such that for any $\\mu_i$ of an inlier group, there exists $\\hat{\\mu}\\in L$ with a sufficiently small $\\| \\hat{\\mu}-\\mu_i \\|$. The only known knowledge is $w_{low}$, a lower bound of all $w_i$s.\n\nThe algorithm proposed by the author is meta in the sense that it uses other algorithms (Robust Mean Estimation and List-Decodable Mean Estimation) as base components. It comprises two stages outlined as follows:\n\n1. outer stage: Iteratively separates the inlier clusters from each other and from the outliers. This stage produces the collection of sets $\\mathcal{T}$ satisfying certain properties. Then for every $T\\in \\mathcal{T}$, run the inner stage.\n2. Inner stage: uses the cor-kLD algorithm to derive the cor-aLD algorithm that restricts access to the $\\alpha_{low}$ s.t. $\\alpha_{low}\\leq \\alpha$. In this stage, for every $T\\in \\mathcal{T}$, a refined list of mean estimates is produced.\n\nSTRENGTHS:\n- This study is interesting. It appears to be the first one to address the challenge of mixture learning in cases where outliers overwhelm small inlier groups\n- The problem setting of this paper is practical. The algorithm is designed to work with limited information, only having access to the lower bound of inlier group fractions.\n- The paper presents theoretical lower bounds for the error. Remarkably, the proposed meta-algorithm achieves a matching lower bound.\n\nWEAKNESSES:\n- The two stage meta-algorithm can be time-consuming.\n\nQUESTIONS:\n- The paper lacks a presentation of the time complexity, and the experimental part also fails to provide results from this aspect. It would be beneficial to include information regarding the time complexity of the proposed algorithm to provide a comprehensive understanding of its computational efficiency.\n\nLIMITATIONS:\nnot applicable", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of estimating the means of well-separated mixtures in the presence of adversarial outliers, a scenario where traditional algorithms may fail. The authors introduce the concept of list-decodable mixture learning (LD-ML), which is particularly relevant when outliers can outnumber smaller inlier groups.\n\nSTRENGTHS:\nThe paper introduces a new problem formulation (LD-ML) and a creative combination of existing ideas (base learners for adversarial corruptions) to solve it. This originality addresses significant limitations in prior work, enhancing the robustness and applicability of mixture learning algorithms. The research quality is high, with rigorous theoretical analysis and well-designed experiments. The thoroughness of the proofs and the clarity of the theoretical contributions reflect a strong understanding of the subject matter.\n\nWEAKNESSES:\nThe paper did not include real-world datasets and additional robust learning methods for comparison. It would be great if the impact of the assumptions can be clarified.\n\nQUESTIONS:\n1. Have you considered validating your algorithm on real-world datasets? If so, could you provide details on these experiments? If not, what were the main obstacles?It would be beneficial to include results from real-world datasets to demonstrate the practical utility and robustness of your method.\n\n2. How does your algorithm perform when the assumption of well-separated mixture components does not hold? Have you tested its robustness in such scenarios?  \n\n3. Can you provide a detailed analysis of the computational complexity of your algorithm? How does it scale with the number of dimensions and mixture components? \n\n4. How does your algorithm compare to other state-of-the-art robust learning methods in terms of practical performance metrics like runtime and ease of implementation?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-20"}

comments: SUMMARY:
The authors investigate the problem of mean estimation for a well-separated mixture in the presence of arbitrary outliers introduced by an adversary. They propose a meta-algorithm that leverages robust mean estimation algorithms as base learners, each with a set of prescribed properties. The authors provide an error guarantee of $\mathcal{O}(\sqrt{\log \frac{\epsilon}{w_i}})$ for a list size of $k + \mathcal{O}(\frac{\epsilon}{w_{low}})$, where $k$ is the total number of inlier groups, $\epsilon$ is the proportion of outliers, and $w_i$ is the proportion of inlier group $i$. Additionally, their approach is capable of handling cases where some $w_i \leq \epsilon$.

STRENGTHS:
The main contribution of the paper lies in developing the meta-algorithm (Algorithm B.1) which first creates clever partitions of the original set (Algorithm D.1) and then uses the appropriate base learners on the partitioned sets (Algorithm C.1). While I have not gone through the mathematical details very carefully, the results seem very interesting and match the error guarantees of base learners run with the full information on the weight proportion of each inlier group separately. The paper is written well and the ideas and corresponding arguments are presented clearly in the main paper with technical details deferred to the appendix.

WEAKNESSES:
The paper makes significant contributions, and I did not identify any particular weaknesses. While I possess general knowledge of the research area, I am not an expert, and my review should be considered within that context.

QUESTIONS:
Please see above.

LIMITATIONS:
The limitations are adequately addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-22

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission considers the problem of list-decoding the means of a mixture of $k$ sub-Gaussian components, under adversarial *additive* contamination which can have size $\\epsilon n$ larger than the smallest-weight component. Instead of only yielding guarantees scaling with the known component weight lower bound $\\alpha_{low}$, the (meta-)algorithm in this paper achieves error that depends directly on $w_i$, the weight of that particular component, and decays also with $\\epsilon$. Furthermore, the size of the output list is guaranteed $k + O(\\epsilon/\\alpha_{low})$ instead of the more common and weaker $O(1/\\alpha_{low})$ guarantee.\n\nNote: this is an emergency review, so I did not read the paper as closely and carefully as I'd like.\n\nSTRENGTHS:\nTo me, the main strength of the paper lies in identifying the regimes where much stronger guarantees (e.g. mean estimation error dependence on actual component weight and not just $\\alpha$) can be made compared to prior works, which are worst-case optimal in some sense of the phrase. The (meta-)algorithmic ideas are also simple enough to be implemented (which I consider a plus and not a minus).\n\nWEAKNESSES:\n- The introduction claims results even when there is no separation between mixture components, but Theorem 3.3 makes a separation assumption? Did I miss something?\n\n====\n\nBelow are hopefully some constructive feedback that the authors can use to improve the paper:\n\n- These guarantees seem possible only because of the distance separation vs concentration assumption (both depend on \"$t$\"), and I think there needs to be more discussion of the intuition on the quantitative tradeoff in the main body.\n\n- The contexualization in prior work seems incomplete to me. I have two related complaints, both (roughly) related to finite covariance assumption setting.\n\n1. The $\\epsilon \\gg \\min_i w_i$ regime seems to me *almost* covered by the standard setting of learning the means of finite covariance mixtures: the contamination distribution $Q$ can always be trimmed a little bit to get finite covariance, and just be regarded as part of the mixture. Upon googling, it seems that there is more recent work by Diakonikolas, Kane, Lee and Pittas (Clustering Mixtures of Bounded Covariance Distributions Under Optimal Separation) which does handle such mixtures, and in fact components with different covariance sizes. I guess the caveat is that, under the above modelling, the mean of $Q$ might not be sufficiently bounded away from the genuine components. Is my understanding correct? I think a comparison to finite-covariance mixture learning literature, from the framework perspective, would be helpful.\n\n2. The intro of this submission really focuses on the sub-Gaussian mixture case ($t \\approx \\log 1/w_{low}$), but the general result applies also to $t = 2$. So I think a direct comparison to the technical results of the above paper as well as DKKLT22 is needed. For example, the separation assumption in this submission is much larger ($1/w_{low}^2$) than these prior works ($1/\\sqrt{w_{low}}$). In general, I think it'd be very useful to clarify the results in this paper with respect to these DKKLT22 and DKLP papers (and other related works).\n\n- Writing: Section 3.1 was quite dense to read (though possibly because I have to read it quickly for the emergency review). It might be better to present the Gaussian case and the relation to prior work, before presenting the full result in generality and explaining precisely the generalizations.\n\n- Minor thing in line 114: I'm not sure it's strictly true that \"in most robust estimation problems, the fractions of inliers and outliers are usually provided to the algorithm\". Doesn't filtering work even if the number of outliers is unknown, as long as the inlier variance is known? One could keep filtering until the remaining data set has small-enough covariance?\n\nQUESTIONS:\n- Any intuition for why $w_{low}^2$ is the right quantity to compare with $w_i$ and $\\epsilon$? Is it a necessity of the problem setting or is it just an artifact of the algorithmic construction?\n\n- Definition 2.1 is essentially assuming boundedness compared to the identity covariance Gaussian. For mixtures where the components have, say, covariance bounded by $c$ times identity, does the algorithm need to know $c$, or can $c$ be estimated easily in this context?\n\n- How hard is it to go beyond the corruption model of (2.1), to get *adaptive* additive corruption? That is, the corruption isn't just drawn from a fixed $Q$, but that the corrupted points can depend directly on the drawn inlier samples?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of estimating the means of a mixture model in the presence of outliers. More specifically, each component has unknown weight $w_i$ that is lower bounded by a known quantity $w_{\\mathrm{low}}$, each component is assumed to be (sub)-Gaussian, and the mixture also includes an adversarially selected distribution (outliers) with weight $\\epsilon$, which can in general be larger than $w_{\\mathrm{low}}$, meaning that the outliers can form entire clusters on their own. Correctly estimating the means of all true components is impossible and thus the goal here is to instead output a small list of candidate means. Since the $w_i$’s are unknown, naive applications of existing list-decoding algorithms would result in estimation errors and list size guarantees that are a function of $w_{\\mathrm{low}}$ only. The contribution of this paper is to show that it is indeed possible to obtain more fine-grained guarantees where the error for the $i$-th component scales with $w_i$ instead of $w_{\\mathrm{low}}$ and the list size is equal to the true number of components plus a small overhead $O(\\epsilon/w_{\\mathrm{low}})$. Regarding the list-size, $\\epsilon/w_{\\mathrm{low}}$ is the number of extra components that the outliers can form, thus this term is unavoidable in the list size. The paper provides information-theoretic error lower-bounds justifying that the precise error rates achieved are qualitatively tight. The regimes where the final algorithm has qualitatively best-possible performance includes the cases where the means are pairwise separated as well as without that assumption, and also the case where $\\epsilon \\ll w_0$ for separated mixtures (where it matches existing qualitatively optimal algorithms).\n\nThe algorithm (for the separated components case) in based on the following parts shown in the paper: First, there is a way to obtain list-decodable mean estimation algorithms that do not need knowledge of the fraction of inliers. This works by collecting the answers of a list-decodable mean estimator for multiple candidate values for the fraction of inliers and carefully pruning the results to keep only hypothesis with a large number of points close to them. Second, the paper proposes a procedure that splits the dataset into (not too many) parts where each part includes at most one (almost entire) cluster of inliers, so that calling the agnostic list-decodable mean estimation of the previous sentence can produce the improved error guarantees. Finally, for the case where $\\epsilon \\ll w_{\\mathrm{low}}$, existing robust mean estimation algorithms can be employed to further improve the error.\n\nOverall, the paper closes a gap in the literature. The result is non-trivial and a useful addition to the literature, thus I am recommending acceptance.\n\nSTRENGTHS:\n* A positive point of the algorithm is that it is a meta-algorithm, in the sense that only performs calls to existing algorithms (for list-decoding and robust mean estimation), and performs computationally simple processing of the dataset based on the outputs of these algorithms. The algorithm is thus efficient and takes advantage of computational efficiency of the base learners.\n\n* The paper contains some experiments to demonstrate practical performance advantages.\n\nWEAKNESSES:\n* The contamination model of eq (2.1) requires that corruptions come i.i.d. from some distribution. Since the base learners can also handle some adversarial corruptions why can’t the final algorithm work for fully adversarial contamination?\n\n* On presentation: The result about non-separated mixtures is not discussed in the main body. It would be good to at least provide the main ideas regarding why similar error can be achieved for the non-separated case, since the algorithm discussed in the main body seems to use crucially the separation of components. Another slightly confusing point is that the introduction talks about Gaussianity a lot while Theorem 3.3 works even for bounded second moment distributions. Existing black box learners work for these settings too but for some reason the existence of black box learners is emphasized only for the Gaussian case (line 193) in Section 3.1.\n\nQUESTIONS:\n* The algorithm that gets errors all the way down to $O(\\sqrt{\\log(1/w_i})$ has super-polynomial complexity. Does the existing SQ lower bound from list-decodable mean estimation in [3] justify this via some easy reduction? If so, it would be good to include.\n\n* Is the optimal separation for the bounded moments case indeed $O(1/w_{\\mathrm{low}}^{4/t})$? I just want to make sure that there is no typographical error in the exponent, since for bounded second moments usually things scale with square root of $w_\\mathrm{low}$.\n\n* Also see first point in \"weaknesses\".\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-27"}