Title: Sample-Efficient Private Learning of Mixtures of Gaussians
Venue: NeurIPS 2024
Accepted: yes
Keywords: Differential Privacy

Density Estimation

Mixtures of Gaussians

Sample Complexity
Abstract: We study the problem of learning mixtures of Gaussians with approximate differential privacy. We prove that roughly $kd^2 + k^{1.5} d^{1.75} + k^2 d$ samples suffice to learn a mixture of $k$ arbitrary $d$-dimensional Gaussians up to low total variation distance, with differential privacy. Our work improves over the previous best result (which required roughly $k^2 d^4$ samples) and is provably optimal when $d$ is much larger than $k^2$. Moreover, we give the first optimal bound for privately learning mixtures of $k$ univariate (i.e., $1$-dimensional) Gaussians. Importantly, we show that the sample complexity for learning mixtures of univariate Gaussians is linear in the number of components $k$, whereas the previous best sample complexity was quadratic in $k$. Our algorithms utilize various techniques, including the inverse sensitivity mechanism, sample compression for distributions, and methods for bounding volumes of sumsets.

Sample-Efficient Private Learning of Mixtures of Gaussians

Abstract

We study the problem of learning mixtures of Gaussians with approximate differential privacy. We prove that roughly kd 2 + k 1 . 5 d 1 . 75 + k 2 d samples suffice to learn a mixture of k arbitrary d -dimensional Gaussians up to low total variation distance, with differential privacy. Our work improves over the previous best result [AAL24b] (which required roughly k 2 d 4 samples) and is provably optimal when d is much larger than k 2 . Moreover, we give the first optimal bound for privately learning mixtures of k univariate (i.e., 1 -dimensional) Gaussians. Importantly, we show that the sample complexity for learning mixtures of univariate Gaussians is linear in the number of components k , whereas the previous best sample complexity [AAL21] was quadratic in k . Our algorithms utilize various techniques, including the inverse sensitivity mechanism [AD20b, AD20a, HKMN23], sample compression for distributions [ABDH + 20], and methods for bounding volumes of sumsets.

1 Introduction

Learning Gaussian Mixture Models (GMMs) is one of the most fundamental problems in algorithmic statistics. Gaussianity is a common data assumption, and the setting of Gaussian mixture models is motivated by heterogeneous data that can be split into numerous clusters, where each cluster follows a Gaussian distribution. Learning mixture models is among the most important problems in machine learning [Bis06], and is at the heart of several unsupervised and semi-supervised machine learning models. The study of Gaussian mixture models has had numerous scientific applications dating back to the 1890s [Pea94], and is a crucial tool in modern data analysis techniques in a variety of fields, including bioinformatics [LKWB22], anomaly detection [ZSM + 18], and handwriting analysis [Bis06].

In this work, we study the problem of learning a GMM from samples. We focus on the density estimation setting, where the goal is to learn the overall mixture distribution up to low total variation distance. Unlike the parameter estimation setting for GMMs, density estimation can be done even without any boundedness or separation assumptions on the parameters of the components. In fact, it is known that mixtures of k Gaussians in d -dimensions can be learned up to total variation distance α using ˜ O ( kd 2 /α 2 ) samples [ABH + 18].

Ensuring data privacy has emerged as an increasingly important challenge in modern data analysis and statistics. Differential privacy (DP) [DMNS06] is a rigorous way of defining privacy, and is considered to be the gold standard both in theory and practice, with deployments by Apple [Tea17], Google [EPK14], Microsoft [DKY17], and the US Census Bureau [DLS + 17]. As is the case for many data analysis tasks, standard algorithms for learning GMMs leak potentially sensitive information about the individuals who contributed data. This raises the question of whether we can do density estimation for GMMs under the constraint of differential privacy.

Shyam Narayanan

Private density estimation for GMMs with unrestricted Gaussian components is a challenging task. In fact, privately learning a single unrestricted Gaussian has been the subject of multiple recent studies [AAK21, KMS + 22b, AL22, KMV22, AKT + 23, HKMN23]. Private learning of GMMs is significantly more challenging, because even without privacy constraints, parameter estimation for GMMs requires exponentially many samples in terms of the number of components [MV10]. Therefore, it is not clear how to use the typical recipe of 'adding noise' to the estimated parameters or 'privately choosing' from the finite-dimensional space of parameters. Consequently, the only known sample complexity bounds for privately learning unrestricted GMMs are loose [AAL24b, AAL21].

Let us first formally define the problem of learning GMMs. We represent a GMM D = ∑ k i =1 w i N ( µ i , Σ i ) by its parameters, namely { ( w i , µ i , Σ i ) } k i =1 , where w i ≥ 0 , ∑ i w i = 1 , µ i ∈ R d , and Σ i is a positive definite matrix. In the following, a GMM learning algorithm A receives a set of data points in R d and outputs a (representation of) a GMM. The total variation distance between two distributions is d TV ( ˜ D , D ) = 1 2 ∫ R d |D ( x ) -˜ D ( x ) | dx 2 .

Definition 1.1 (Learning GMMs) . For α, β ∈ (0 , 1) , we say A learns GMMs with n samples up to accuracy α and failure probability β if for every GMM D , given samples X 1 , . . . , X n i.i.d. ∼ D , it outputs (a representation of) a GMM ˜ D such that d TV ( ˜ D , D ) ≤ α with probability at least 1 -β .

α and β are called the accuracy and failure probability, respectively. For clarity of presentation, we will typically fix the value of β (e.g., β = 1 / 3 ). The above definition does not enforce the constraint of differential privacy. The following definitions formalizes (approximate) differential privacy.

Definition 1.2 (Differential Privacy (DP) [DMNS06, DKM + 06]) . Let ε, δ ≥ 0 . A randomized algorithm A : X n →O is said to be ( ε, δ ) -differentially private ( ( ε, δ ) -DP) if for any two neighboring datasets X , X ′ ∈ X n and any measurable subset O ⊂ O ,

If the GMM learner A of Definition 1.1 is ( ε, δ ) -DP, we say that A privately learns GMMs. Formally, we have the following definition.

Definition 1.3 (Privately learning GMMs) . Fix the number of samples n , dimension d , and number of mixture components k . For α, β ∈ (0 , 1) and ε, δ ≥ 0 , a randomized algorithm A , that takes as input X 1 , . . . , X n ∈ R d , ( ε, δ ) -privately learns GMMs up to accuracy α and failure probability β , if:

For any GMM D that is a mixture of up to k Gaussians in d dimensions, if X = { X 1 , . . . , X n } i.i.d. ∼ D , A ( X 1 , . . . , X n ) outputs a GMM ˜ D such that d TV ( ˜ D , D ) ≤ α with probability at least 1 -β (over the randomness of the data X and the algorithm A ).

For any neighboring datasets X , X ′ ∈ ( R d ) n (not necessarily drawn from any GMM) and any measurable subset O ⊂ O , P [ A ( X ′ ) ∈ O ] ≤ e ε · P [ A ( X ) ∈ O ] + δ.

Finally, we assume a default value for β of 1 / 3 , meaning that if not stated, the failure probability β is assumed to equal 1 / 3 .

Our main goal in this paper is to understand the number of samples (as a function of the dimension d , the number of mixture components k , the accuracy α , and the privacy parameters ε, δ ) that are needed to privately and accurately learn the GMM up to low total variation distance.

1.1 Results

In this work, we provide improved sample complexity bounds for privately learning mixtures of arbitrary Gaussians, improving over previous work of [AAL21, AAL24b]. Moreover, our sample complexity bounds are optimal in certain regimes, when the dimension is either 1 or a sufficiently large polynomial in k and log 1 δ . For general dimension d , we prove the following theorem.

Theorem 1.4. For any α, ε, δ ∈ (0 , 1) , k, d ∈ N , there exists an inefficient ( ε, δ ) -DP algorithm that can learn a mixture of k arbitrary full-dimensional Gaussians in d dimensions up to accuracy α , using the following number of samples:

Notably, the mixing weights and the means can be arbitrary and the covariances of the Gaussians can be arbitrarily poorly conditioned, as long as the covariances are non-singular 3 .

We remark that we omit the dependence on β (and assume by default a failure probability of 1 / 3 ). However, it is well-known that one can obtain failure probability β with only a multiplicative O (log 1 /β ) blowup in sample complexity, in a black-box fashion 4 . In fact, our analysis can yield even better dependencies on β in some regimes, though to avoid too much complication, we do not analyze this.

For reasonably large dimension, i.e., d ≥ k 2 log 2 (1 /δ ) , this can be simplified to ˜ O ( kd 2 α 2 + kd 2 αε ) , which is in fact optimal (see Theorem 1.6). Hence, we obtain the optimal sample complexity for sufficiently large dimension. Theorem 1.4 also improves over the previous best sample complexity upper bound of [AAL24b], which uses

samples. Our results provide a polynomial improvement in all parameters, but to simplify the comparison, if we ignore dependencies in the error parameter α and privacy parameters ε, δ , we improve the sample complexity from k 2 d 4 to kd 2 + k 2 d + k 1 . 5 d 1 . 75 : note that our result is quadratic in the dimension whereas [AAL24b] is quartic.

When the dimension is d = 1 , we can provide an improved result, which is optimal for learning mixtures of univariate Gaussians (see Theorem 1.6 for a matching lower bound).

Theorem 1.5. For any α, ε, δ ∈ (0 , 1) , k ∈ N , there exists an inefficient ( ε, δ ) -DP algorithm that can learn a mixture of k arbitrary univariate Gaussians (of nonzero variance) up to accuracy α , using the following number of samples:

For privately learning mixtures of univariate Gaussians, the previous best-known result for arbitrary Gaussians required ˜ O ( k 2 log 3 / 2 (1 /δ ) α 2 ε ) samples [AAL21]. Importantly, we are the first paper to show that the sample complexity can be linear in the number of components.

Our work purely focuses on sample complexity, and as noted in Theorems 1.4 and 1.5, they do not have polynomial time algorithms. We note that the previous works of [AAL21, AAL24b] also do not run in polynomial time. Indeed, there is reason to believe that even non-privately, it is impossible to learn GMMs in polynomial time (in terms of the optimal sample complexity) [DKS17, BRST21, GVV22].

Finally, we prove the following lower bound for learning GMMs in any fixed dimension d .

Theorem 1.6. Fix any dimension d ≥ 1 number of components k ≥ 2 , any α, ε at most a sufficiently small constant c ∗ , and δ ≤ ( αε/d ) O (1) . Then, any ( ε, δ ) -DP algorithm that can learn a mixture of k arbitrary full-dimensional Gaussians in d dimensions up to total variation distance α , with probability at least 2 / 3 , requires at least the following number of samples:

Note that for d = 1 , this matches the upper bound of Theorem 1.5, thus showing that our univariate result is near-optimal in all parameters α, ε, δ . Moreover, our lower bound refutes the conjecture of [AAL21], which conjectures that only Θ ( k α 2 + k αε + log(1 /δ ) ε ) samples are needed in the univari- ate case and Θ ( kd 2 α 2 + kd 2 αε + log(1 /δ ) ε ) samples are needed in the d -dimensional case. However, we note that our lower bound asymptotically differs from the conjectured bound in [AAL21] only when δ is extremely small.

1.2 Related work

In the non-private setting, the sample complexity of learning unrestricted GMMs with respect to total variation distance (a.k.a. density estimation) is known to be ˜ Θ( kd 2 /α 2 ) [ABM18, ABH + 18], where the upper bound is obtained by the so-called distributional compression schemes.

In the private setting, the only known sample complexity upper bound for unrestricted GMMs [AAL24b] is roughly k 2 d 4 log(1 /δ ) / ( α 4 ε ) , which exhibits sub-optimal dependence on various parameters 5 . This bound is achieved by running multiple non-private list-decoders and then privately aggregating the results. For the special case of axis-aligned GMMs, an upper bound of k 2 d log(1 /δ ) 3 / 2 / ( α 2 ε ) is known [AAL21]. These are the only known results even for privately learning (unbounded) univariate GMMs. In other words, the best known upper bound for sample complexity of privately learning univariate GMMs has quadratic dependence on k .

In the related public-private setting [BKS22, BBC + 23], it is assumed that the learner has access to some public data. In this setting, [BBC + 23] show that unrestricted GMMs can be learned with a moderate amount of public and private data.

Assuming the parameters of the Gaussian components (and the condition numbers of the covariance matrices) are bounded, one can create a cover for GMMs and use private hypothesis selection [BSKW19] or the private minimum distance estimator [AAK21] to learn the GMM. On the flip side, [ASZ21] prove a lower bound on the sample complexity of learning GMMs, though their lower bound is weaker than ours and is only against pure-DP algorithms.

The focus of our work is on density estimation. A related problem is learning the parameters a GMM, which has received extensive attention in the (non-private) literature (e.g., [Das99, MV10, BS10, LM21, BDJ + 22, LL22] among many other papers). To avoid identifiability issues, one has to assume that the Gaussian components are sufficiently separated and have large-enough weights. In the private setting, the early work of [NRS07] demonstrated a privatized version of [VW04] for learning GMMs with fixed (known) covariance matrices. The strong separation assumption (of Ω( k 1 / 4 ) ) between the Gaussian components in [NRS07] was later relaxed to a weaker separation assumption [CCAd + 23]. A substantially more general result for privately learning GMMs with unknown covariance matrices was established in [KSSU19], based on a privatized version of [AM05]. Yet, this approach also requires a polynomial separation (in terms of k ) between the components, as well as a bound on the spectrum of the covariance matrices. [CKM + 21] weakened the separation assumption of [KSSU19] and improved over their sample complexity. This result is based on a generic method that learns a GMM using a private learner for Gaussians and a non-private clustering method for GMMs. Finally, [AAL23] designed an efficient reduction from private learning of GMMs to its non-private counterpart, removing the boundedness assumptions on the parameters and achieving minimal separation (e.g., by reducing to [MV10]). Nevertheless, unlike density estimation, parameter estimation for unrestricted GMMs requires exponentially many samples in terms of k [MV10].

A final important question is that of efficient algorithms for learning GMMs. Much of the work on learning GMM parameters focuses on computational efficiency (e.g,. [MV10, BS10, LM21, BDJ + 22, LL22]), as does some work on density estimation (e.g., [CDSS14, ADLS17]). However, under some standard hardness assumptions, it is known that even non-privately learning mixtures of k d -dimensional Gaussians with respect to total variation distance cannot be done in polynomial time as a function of k and d [DKS17, BRST21, GVV22].

Addendum. In a concurrent submission, [AAL24a] extended the result of [AAL24b] for learning unrestricted GMMs to the agnostic (i.e., robust) setting. In contrast, our algorithm works only in the realizable (non-robust) setting. Moreover, [AAL24a] slightly improved the sample complexity result of [AAL24b] from ˜ O (log(1 /δ ) k 2 d 4 / ( εα 4 )) to ˜ O (log(1 /δ ) k 2 d 4 / ( εα 2 )) . The sample complexity of our approach is still significantly better than [AAL24a] in terms of all parameters-similar to the way it improved over [AAL24b].

2 Technical overview and roadmap

We highlight some of our conceptual and technical contributions. We mainly focus on the highdimensional upper bound, and discuss the univariate upper bound at the end.

2.1 Reducing to crude approximation

Suppose we are promised a bound on the means and covariances of the Gaussian components, i.e., 1 R · I ≼ Σ i ≼ R · I and ∥ µ i ∥ 2 ≤ R for all i ∈ [ k ] . In this case, there is in fact a known algorithm, using private hypothesis selection [BSKW19, AAK21], that can privately learn the distribution using only O ( kd 2 log R α 2 + kd 2 log R αε ) samples. Moreover, with a more careful application of the hypothesis selection results (see Appendix D), we can prove that result holds even if ( µ i , Σ i ) are possibly unbounded, but we have some very crude approximation. By this, we mean that if for each i ∈ [ k ] we know some ˆ Σ i such that 1 R · Σ i ≼ ˆ Σ i ≼ R · Σ i , then it suffices to have n = O ( kd 2 log R α 2 + kd 2 log R αε ) samples to learn the full GMM in total variation distance.

Our main goal will be to learn every covariance Σ i with such an approximation, for R = poly ( k, d, 1 α , 1 ε ) , so that log R can be hidden in the ˜ O factor. To explain why this goal is sufficient, suppose we can crudely learn every covariance Σ i with approximation ratio R , using n ′ samples. We then need O ( kd 2 log R α 2 + kd 2 log R αε ) = ˜ O ( kd 2 α 2 + kd 2 αε ) additional samples to learn the full distri- bution using hypothesis selection, so the total sample complexity is ˜ O ( n ′ + kd 2 log R α 2 + kd 2 log R αε ) . Hence, we will aim for this easier goal of crudely learning each covariance, for both Theorem 1.4 and Theorem 1.5, using as few samples n ′ as possible. We will also need to approximate each mean µ i , though for simplicity we will just focus on covariances in this section.

2.2 Overview of Theorem 1.5 for univariate GMMs

The main goal will be to simply provide a rough approximation to the set of standard deviations σ i = √ Σ i , as we can finish the procedure with hypothesis selection, as discussed above.

Bird's eye view: Say we are given a dataset X = { X 1 , . . . , X n } : note that every X j ∈ R since we are dealing with univariate Gaussians. The main insight is to sort the data in increasing order (i.e., reorder so that X 1 ≤ X 2 ≤ · · · ≤ X n ) and consider the unordered multiset of successive differences { X 2 -X 1 , X 3 -X 2 , . . . , X n -X n -1 } . One can show that if a single datapoint X j is changed, then the set of consecutive differences (up to permutation) does not change in more than 3 locations (see Lemma F.5 for a formal proof).

We then apply a standard private histogram approach. Namely, for each integer a ∈ Z , we create a corresponding bucket B a , and map each X j +1 -X j into B a if 2 a ≤ X j +1 -X j < 2 a +1 . If some mixture component i had variance Σ i = σ 2 i , we should expect a significant number of X j +1 -X j to at least be crudely close to σ i , such as for the X j drawn from the i th mixture component. So, some corresponding bucket should be reasonably large. Finally, by adding noise to the count of each bucket and taking the largest noisy counts, we will successfully find an approximation to all variances.

In more detail: For each (possibly negative) integer a let c a be the number of indices i such that 2 a ≤ X j +1 -X j < 2 a +1 . We will prove that, if the weight of the i th component in the mixture is w i and there are n points, then we should expect at least Ω( w i · n ) indices j to be in a bucket a with 2 a within a poly( n ) multiplicative factor of the standard deviation σ i (see Lemma F.3). The point of this observation is that there are at most O (log n ) buckets B a with 2 a between σ i poly( n ) and O ( σ i ) , but there are at least Ω( w i · n ) indices mapping to one of these buckets. So by the Pigeonhole principle, one of these buckets has at least Ω ( w i · n log n ) indices, i.e., c a ≥ Ω ( w i · n log n ) for some a with σ i poly( n ) ≤ 2 a ≤ O ( σ i ) .

Moreover, we know that if we change a single data point X j , the set of consecutive differences { X j +1 -X j } after sorting changes by at most 3 . So, if we change a single X j , at most 3 of the counts c a can change, each by at most 3 .

Now, for every integer a , draw a noise value from the Truncated Laplace distribution (see Definition A.2 and Lemma A.3), and add it to c a to get a noisy count ˜ c a . The details of the noise distribution are not important, but the idea is that this distribution is always bounded by O ( 1 ε log 1 δ ) . Moreover, the Truncated Laplace distribution preserves ( ε, δ ) -DP. This means that the counts { ˜ c a } a ∈ Z will have ( O ( ε ) , O ( δ )) -DP, because the true counts c a only change minimally across adjacent datasets.

a

Our crude approximation to the set of standard deviations will be the set of 2 such that ˜ c a exceeds some threshold which is a large multiple of 1 ε log 1 δ . If n ≥ ˜ O ( k log(1 /δ ) αε ) and the weight w i ≥ α/k , it is not hard to verify that w i · n log n exceeds a large multiple of 1 ε log 1 δ . So, for each i ≤ k with weight at least α/k , some corresponding a with σ i poly( n ) ≤ 2 a ≤ O ( σ i ) will have count c a significantly exceeding the threshold, and thus noisy count ˜ c a exceeding the threshold. This will be enough to crudely approximate the values σ i coming from large enough weight. We can ignore any component with weight less than α/k , as even if all but one of components have such small weight, together they only contribute α weight. So, we can ignore them and it will only cost us α in total variation distance, which we can afford.

Putting everything together: In summary, we needed ˜ O ( k log(1 /δ ) αε ) samples to approximate each covariance (of sufficient weight) up to a poly( n ) multiplicative factor. By setting R = poly( n ) and using the reduction described in Section 2.1, we then need an additional O ( k log n α 2 + k log n αε ) samples. If we set n = ˜ O ( k α 2 + k log(1 /δ ) αε ) , we will obtain that n ≥ ˜ O ( k log(1 /δ ) αε ) + O ( k log n α 2 + k log n αε ) , which is sufficient to solve our desired problem in the univariate setting.

Note that this proof relies heavily on the use of private histograms and the order of the data points in the real line. Therefore, it cannot be extended to the high-dimensional setting. We will use a completely different approach to prove Theorem 1.4.

2.3 Overview of Theorem 1.4 for general GMMs

As in the univariate case, we only need rough approximations of the covariances. We will learn the covariances one at a time: in each iteration, we privately identify a single covariance ˆ Σ i which crudely approximates some covariance Σ i in the mixture, with ( ε/ √ k log(1 /δ ) , δ/k ) -DP. Using the well-known advanced composition theorem (see Theorem A.1), we will get an overall privacy guarantee of ( ε, δ ) -DP. However, to keep things simple, we will usually aim for ( ε, δ ) -DP when learning each covariance, and we can replace ε with ε/ √ k log(1 /δ ) and δ with δ/k at the end.

A natural approach for parameter estimation, rather than learn Σ i one at a time, is to learn all of the covariances together. However, we believe that this approach would cause the sample complexity to multiply by a factor of k , compared to learning a single covariance. The advantage of learning the covariances one at a time is that we can apply advanced composition: this will cause the sample complexity to multiply by roughly √ k log(1 /δ ) instead.

In the rest of this subsection, our main focus is to identify a single crude approximation ˆ Σ i .

Applying robustness-to-privacy: The first main insight is to use the robustness-to-privacy conversion of Hopkins et al. [HKMN23] (see also [AUZ23]). Hopkins et al. prove a black-box (but not computationally efficient) approach that can convert robust algorithms into differentially private algorithms, using the exponential mechanism and a well-designed score function. This reduction only works for finite dimensional parameter estimation and therefore cannot be applied directly to density estimation. On the other hand, parameter estimation for arbitrary GMMs requires exponentially many samples in the number of components [MV10]. However, we will demonstrate that this lower bound does not hold when we only need a very crude estimation of the parameters.

The idea is the following. For a dataset X = { X 1 , . . . , X n } , let S = S ( ˜ Σ , X ) be a score function, which takes in a dataset X of size n and some candidate covariance ˜ Σ , and outputs some non-negative integer. At a high level, the score function S ( ˜ Σ , X ) will be the smallest number of data points t that we should change in X to get to some new data set X ′ with a specific desired property: X ′ should 'look like' a sample generated from a mixture distribution with ˜ Σ being the covariance of

one of its components - namely, a component with a significant mixing weight. One can define 'looks like' in different ways, and we will adjust the precise definition later. We remark that this notion of score roughly characterizes robustness, because if the samples in X were truly drawn from a Gaussian mixture model with covariances { Σ i } k i =1 , we should expect S (Σ i , X ) to be 0 (since X should already satisfy the desired property), but if we altered k data points, the score should be at most k . The high-level choice of score function is somewhat inspired by a version of the exponential mechanism called the inverse sensitivity mechanism [AD20b, AD20a], though the precise way of defining the score function requires significant care and is an important contribution of this paper.

The robustness-to-privacy framework of [HKMN23], tailored to learning a single covariance, implies the following general result, which holds for any score function S following the blueprint above. For now, we state an informal (and slightly incorrect) version.

Theorem 2.1 (informal - see Theorem C.3 for the formal statement) . For any η ∈ [0 , 1) , and any dataset X of size n , define the value V η ( X ) to be the volume (i.e., Lebesgue measure) of the set of covariance matrices ˜ Σ (where the covariance can be viewed as a vector by flattening), such that S ( ˜ Σ , X ) ≤ η · n .

Fix a parameter η < 0 . 1 , and suppose that for any dataset X of size n such that V η/ 2 ( X ) is strictly positive, the ratio of volumes V η ( X ) /V η/ 2 ( X ) is at most some K (which doesn't depend on X ). Then, if n ≥ log K ε · η , there is a differentially private algorithm that can find a covariance ˜ Σ of low score (i.e., where S ( ˜ Σ , X ) ≤ η · n ) using n samples.

Note that Theorem 2.1 does not seem to say anything about whether X comes from a mixture of Gaussians. However, we aim to instantiate this theorem with a score function that is carefully designed for GMMs. Recall that we want S ( ˜ Σ , X ) to capture the number of points in X that need to be altered to make it look like a data set that was generated from a mixture, with ˜ Σ being the covariance of one of the Gaussian components. In other words, S ( ˜ Σ , X ) should be small if (and hopefully only if) a 'mildly corrupted' version of X includes a subset of points that are generated from a Gaussian with covariance ˜ Σ . At the heart of designing such a score function, one needs to come up with a form of 'robust Gaussianity tester' that tells whether a given set of data points are generated from a Gaussian distribution. Aside from this challenge, the volume ratio associated with the chosen score function needs to be small for every dataset X otherwise the above theorem would require a large n (i.e., number of samples). These two challenges are, however, related. If the robust Gaussianity tester has high specificity-i.e., rejects most of the sets that are not generated from a (corrupted) Gaussian-then the volume ratio is likely to be small (i.e., a smaller number of candidates ˜ Σ would receive a good/low score).

First Attempt: We first try an approach which resembles that of [HKMN23] for privately learning a single Gaussian. We 'define' S ( ˜ Σ , X ) as the smallest integer t satisfying the following property: there exists a subset Y ⊂ X of size n/k , such that we can change t data points from Y to get to Y ′ , where Y ′ 'looks like' i.i.d. samples from a Gaussian with some covariance Σ that is 'similar to' ˜ Σ . The choice of Y having size n/k is motivated by the fact that each mixture component, on average, has n/k data points in X .

The notions of 'looks like' and 'similar to' of course need to be formally defined. We say Σ is 'similar to' ˜ Σ (or Σ ≈ ˜ Σ ) if they are spectrally close, i.e., 0 . 5Σ ≼ ˜ Σ ≼ 2Σ . We say that Y ′ 'looks like' samples from a Gaussian with covariance Σ if some covariance estimation algorithm predicts that Y ′ came from a Gaussian with covariance Σ . The choice of covariance estimation algorithm will be quite nontrivial and ends up being a key ingredient in proving Theorem 1.4.

To apply Theorem 2.1, we first set η = c/k for some small constant c . We cannot set a larger value η , because if we change t ≈ n/k data points, we could in fact create a totally arbitrary new Gaussian component with large weight. Since there is no bound on the eigenvalues of the covariance matrix, this could cause the volume V η ( X ) to be infinite. The main question we must answer is how to bound the volume ratio V η ( X ) /V η/ 2 ( X ) . To answer this question, we first need to understand what it means for S ( ˜ Σ , X ) ≤ η · n . If S ( ˜ Σ , X ) ≤ η · n , then there exists a corresponding set Y ⊂ X of size n/k , and we can change η · n = c · | Y | points from Y to get to some Y ′ which looks like samples from a Gaussian with covariance Σ ≈ ˜ Σ . Thus, Y looks like c -corrupted samples from such a Gaussian (i.e.,

a c fraction of the data is corrupted). This motivates using a robust covariance estimation algorithm: indeed, robust algorithms can still approximately learn Σ even if a small constant fraction of data is corrupted, so for any Y ⊂ X , we expect that no matter how we change a c fraction of Y to obtain Y ′ , the robust algorithm's covariance estimate should not change much. So, for any fixed Y , the set of possible Σ , and thus the set of possible ˜ Σ , should not be that large.

In summary, to bound V η ( X ) versus V η/ 2 ( X ) , there are at most ( n n/k ) choices for Y ⊂ X in the former case, and at least 1 choice in the latter case (since we assume V η/ 2 ( X ) > 0 in Theorem 2.1). Moreover, for any such Y , the volume of corresponding ˜ Σ should be exponential in d 2 (either for V η ( X ) or V η/ 2 ( X ) ), since the dimensionality of the covariance is roughly d 2 . So, this suggests that the overall volume ratio is at most ( n n/k ) · e O ( d 2 ) . Since log ( n n/k ) ≈ ( n/k ) · log k , if we plug into Theorem 2.1 it suffices to have n ≥ ( n/k ) log k + d 2 . Unfortunately this is impossible unless ε ≥ log k .

ε · ( c/k )

These ideas will serve as a good starting point, though we need to improve the volume ratio analysis. To do so, we also modify the robust algorithm, by strengthening the assumptions on what it means for samples to 'look like' they came from a Gaussian.

Improvement via Sample Compression: To improve the volume ratio, we draw inspiration from a technique called sample compression , which has been used in previous work on non-private density estimation for mixtures of Gaussians [ABH + 18, ABDH + 20]. The idea behind sample compression is that one does not need the full set Y to do robust covariance estimation; instead, we look at a smaller set of samples. For instance, if Y ⊂ X looks like c -corrupted samples from a Gaussian of covariance Σ ≈ ˜ Σ , we expect that a random subset Z of Y also looks like c -corrupted samples from such a Gaussian. Moreover, as long as one uses m ≥ O ( d ) corrupted samples from a Gaussian, we can still (inefficiently) approximate the covariance. This motivates us to modify the robust algorithm as follows: rather than just checking whether Y looks like c -corrupted samples from a Gaussian of covariance roughly ˜ Σ , we also test whether an average subset Z ⊂ Y of size m does as well. Therefore, if ˜ Σ has low score, there exists a corresponding set Z ⊂ X of size m , and there are only ( n m ) ≤ e m log n choices for Z . So, now it suffices to have n ≥ m log n + d 2 ε · ( c/k ) , which will give us a bound of ˜ O ( d 2 k/ε ) , as long as m ≤ O ( d 2 ) . Importantly, we still check the robust algorithm on Y of size roughly n/k , which allows us to keep the robustness threshold η at roughly c/k .

There is one important caveat that for each subset Z , there is a distinct corresponding covariance Σ , and the volume of ˜ Σ ≈ Σ can change drastically as Σ changes. (For instance, the volume of ˜ Σ ≈ T · Σ is T Θ( d 2 ) times as large as the volume of ˜ Σ ≈ Σ . Since we have no bounds on the possible covariances, T could be unbounded.) For our volume ratio to actually be bounded by about ( n m ) · e O ( d 2 ) , we want the volume of ˜ Σ ≈ Σ to stay invariant with respect to Σ . This can be done by defining a 'normalized volume' where the normalization is inversely proportional to the determinant (see Appendix C.1 for more details). The robustness-to-privacy conversion (Theorem 2.1) will still hold.

While the bound of ˜ O ( d 2 k/ε ) seems good, we recall that this bound is merely the sample complexity for ( ε, δ ) -DP crude approximation of a single Gaussian component. As discussed at the beginning of this subsection, to learn all k Gaussians, we actually need ( ε/ √ k log(1 /δ ) , δ/k ) -DP, rather than ( ε, δ ) -DP, when crudely approximating a single component. This will still end up leading to a significant improvement over previous work [AAL24b], but we can improve the volume ratio even further and thus obtain even better sample complexity.

Improving Dimension Dependence: Previously, we used the fact that the volume of candidate ˜ Σ (corresponding to a fixed Z ) was roughly exponential in d 2 for either V η/ 2 ( X ) or V η ( X ) , so the ratio should also be roughly exponential in d 2 . Here, we improve this ratio, which will improve the overall volume ratio.

First, we return to understanding the guarantees of the robust algorithm. It is known that, given m ≥ O ( d ) samples from a Gaussian of covariance Σ , we can provide an estimate ˆ Σ such that (1 -O ( √ d/m ))Σ ≼ ˆ Σ ≼ (1 -O ( √ d/m ))Σ . As above, we need to solve this even if a c fraction of

samples are corrupted. While this can cause the relative spectral error to increase from 1 ± O ( √ d/m ) to 1 ± O ( c + √ d/m ) , for now let us ignore the additional c factor.

If V η/ 2 ( X ) > 0 , then there is some covariance Σ and some set Y of size n/k , where the robust algorithm thinks Y looks like (possibly corrupted) Gaussian samples of covariance Σ . So, every ˜ Σ such that 0 . 5Σ ≼ ˜ Σ ≼ 2Σ has score of at most η/ 2 · n . This gives us a lower bound on V η/ 2 ( X ) . We now want to upper bound V η ( X ) . If S ( ˜ Σ , X ) < ηn , we still have that the robust algorithm thinks some Y looks like Gaussian samples of covariance Σ , where 0 . 5Σ ≼ ˜ Σ ≼ 2Σ . But now, we use the additional fact that for some Z ⊂ Y of size m , the robust algorithm on Z finds a covariance ˆ Σ . By the accuracy of the robust algorithm, Σ and ˆ Σ should be similar, i.e., (1 -O ( √ d/m ))Σ ≼ ˆ Σ ≼ (1 -O ( √ d/m ))Σ (where we ignored the c factor). Thus, there exists some Z ⊂ X of size m and a ˆ Σ corresponding to Z , such that 0 . 5(1 -O ( √ d/m )) ˆ Σ ≼ ˜ Σ ≼ 2(1 + O ( √ d/m )) ˆ Σ .

Therefore, from η/ 2 to η , we have dilated the candidate set of ˜ Σ by a factor of 1 + O ( √ d/m ) in the worst case, and we have at least 1 choice in the η/ 2 case but at most ( n m ) choices in the η case. Thus, the overall volume ratio is at most ( n m ) · (1 + O ( √ d/m )) d 2 = e O ( m log n + d 2 · √ d/m ) , since the dimension of ˜ Σ is roughly d 2 . Consequently, it now suffices to have n ≥ m log n + d 2 · √ d/m ε · ( c/k ) : setting m = d 5 / 3 gives us an improved bound of ˜ O ( d 5 / 3 k/ε ) for learning a single Σ i .

There are some issues with this approach: most notably, we ignored the fact that the spectral error is really c + √ d/m rather than √ d/m . However, the robust algorithm can do better than just estimating up to spectral error c + √ d/m : it can also get an improved Frobenius error. While we will not formally state the guarantees on the robust algorithm here (see Theorem B.3 for the formal statement), the main high-level observation is that if the robust estimator ˆ Σ can be 1 ± c times as large as the true covariance Σ in only O (1) directions then for an average direction the ratio of ˆ Σ to Σ will be 1 ± O ( √ d/m ) . We can utilize this observation to bound the volume ratio, using some careful ε -net arguments (this is executed in Appendix C.3). Our dimension dependence of d 5 / 3 will increase to d 7 / 4 , though this still improves over the previous d 2 bound.

We will formally define the score function S ( ˜ Σ , X ) in Appendix E.1 and fully analyze the application of the robustness-to-privacy conversion, as outlined here, in Appendix E.2.

Accuracy: One important final step that we have so far neglected is ensuring that any ˜ Σ of low score must be crudely close to some Σ i , if X is actually drawn from a GMM. We will just focus on the case where S ( ˜ Σ , X ) = 0 , so some Y ⊂ X of size n/k looks like a set of samples from a Gaussian with covariance ˜ Σ . If the samples Y all came from the i -th component of the GMM, then it will not be difficult to show ˜ Σ is similar to Σ i . The more difficult case is when data point in Y are generated from several components.

However, if n ≥ 20 k 2 d , then | Y | ≥ 20 kd , which means that by the Pigeonhole Principle, at least 20 d points in Y come from the same mixture component ( µ i , Σ i ) . We are able to prove that, with high probability over samples drawn from a single Gaussian component N ( µ i , Σ i ) , that the empirical covariance of any subset of size at least 20 d is crudely close to Σ i (see Corollary B.5). As a result, when verifying that a subset Y 'looks like' i.i.d. Gaussian samples with covariance Σ , we can ensure that the empirical covariance of every subset Z ⊂ Y of size 20 d is crudely close to Σ . Thus, if the score S ( ˜ Σ , X ) = 0 , ˜ Σ is close to Σ , which is crudely close to some Σ i .

We also formally analyze the accuracy in Appendix E.2.

Putting everything together: To crudely learn a single Gaussian component with ( ε, δ ) -DP, we will need n ≥ ˜ O ( d 7 / 4 k/ε ) samples to find some covariance ˜ Σ with low score, and we also need n ≥ O ( k 2 d ) so that a covariance ˜ Σ of low score is actually a crude approximation to one of the real mixture components. To crudely approximate all components, we learn each Gaussian component with ( ε/ √ k log(1 /δ ) , δ/k ) -DP. The advanced composition theorem will imply that repeating this procedure k times on the same data (to learn all k components) will be ( ε, δ ) -DP. Hence, by replacing

ε with ε/ √ k log(1 /δ ) , we get that it suffices for n ≥ ˜ O ( d 7 / 4 k 3 / 2 √ log(1 /δ ) ε + k 2 d ) , if we need to crudely learn all of the covariances. Finally, we can apply the private hypothesis selection technique (recall Section 2.1), which requires an additional ˜ O ( d 2 k α 2 + d 2 k αε ) . Combining these terms will give the final sample complexity.

We remark that the sample complexity obtained above is actually better than the complexity in Theorem 1.4. There are two reasons for this. The first is that we have been assuming each component has weight 1 /k , meaning it contributes about n/k data points. In reality, the weights may be arbitrary and thus some components may have much fewer data points. However, it turns out that one can actually ignore any component with less than α/k weight, if we want to solve density estimation up to total variation distance α . This will multiply the sample complexity terms ˜ O ( d 7 / 4 k 3 / 2 √ log(1 /δ ) ε + k 2 d ) , needed for crude approximation, by a factor of 1 /α . Finally, the informal Theorem 2.1 is slightly inaccurate, and the accurate version of the theorem will end up adding the additional term of k 3 / 2 log 3 / 2 (1 /δ ) αε . Along with the final term ˜ O ( d 2 k α 2 + d 2 k αε ) from the private hypothesis selection, these terms will exactly match Theorem 1.4.

2.4 Roadmap

In Appendix A, we note some general preliminary results. In Appendix B, we note some additional preliminaries on robust learning of a single Gaussian. In Appendix C, we discuss the robustnessto-privacy conversion and prove some volume arguments needed for Theorem 1.4. In Appendix D, we explain how to reduce to the crude approximation setting, using private hypothesis selection. In Appendix E, we design and analyze the algorithm for multivariate Gaussians, and prove Theorem 1.4. In Appendix F, we design and analyze the algorithm for univariate Gaussians, and prove Theorem 1.5. In Appendix G, we prove Theorem 1.6. Finally, Appendix H proves some auxiliary results that we state in Appendices B and C.

Limitations

Our results are on theoretical guarantees on the sample complexity of privately learning Mixtures of Gaussians. We do not provide any efficient or practical algorithms, and focus on statistical guarantees. We also do not discuss how to set the parameters and accuracy guarantees for practical applications, this is a question best left to practitioners. Finally, we assume each sample is i.i.d. drawn from a Gaussian Mixture Model distribution, though we remark that we use a 'robustness-to-privacy' framework that will automatically make our algorithm robust to a roughly α/k fraction of corruptions.

Acknowledgements. Hassan Ashtiani was supported by an NSERC Discovery grant. Shyam Narayanan was supported by an NSF Graduate Fellowship and a Google Fellowship.

References

[AAK21] Ishaq Aden-Ali, Hassan Ashtiani, and Gautam Kamath. On the sample complexity of privately learning unbounded high-dimensional gaussians. In Algorithmic Learning Theory , pages 185-216. PMLR, 2021.

[AAL21] Ishaq Aden-Ali, Hassan Ashtiani, and Christopher Liaw. Privately learning mixtures of axis-aligned gaussians. Advances in Neural Information Processing Systems , 34:39253938, 2021.

[AAL23] Jamil Arbas, Hassan Ashtiani, and Christopher Liaw. Polynomial time and private learning of unbounded gaussian mixture models. In International Conference on Machine Learning , pages 1018-1040. PMLR, 2023.

[AAL24a] Mohammad Afzali, Hassan Ashtiani, and Christopher Liaw. Agnostic private density estimation for gmms via list global stability. arXiv e-prints , pages arXiv-2407, 2024.

[AAL24b] Mohammad Afzali, Hassan Ashtiani, and Christopher Liaw. Mixtures of gaussians are privately learnable with a polynomial number of samples. In Algorithmic Learning Theory , pages 1-27. PMLR, 2024.

[ABDH + 20] Hassan Ashtiani, Shai Ben-David, Nicholas JA Harvey, Christopher Liaw, Abbas Mehrabian, and Yaniv Plan. Near-optimal sample complexity bounds for robust learning of gaussian mixtures via compression schemes. Journal of the ACM (JACM) , 67(6):1-42, 2020.

[ABH + 18] Hassan Ashtiani, Shai Ben-David, Nicholas Harvey, Christopher Liaw, Abbas Mehrabian, and Yaniv Plan. Nearly tight sample complexity bounds for learning mixtures of gaussians via sample compression schemes. Advances in Neural Information Processing Systems , 31, 2018.

[ABM18] Hassan Ashtiani, Shai Ben-David, and Abbas Mehrabian. Sample-efficient learning of mixtures. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

[AD20a] Hilal Asi and John C Duchi. Instance-optimality in differential privacy via approximate inverse sensitivity mechanisms. Advances in neural information processing systems , 33:14106-14117, 2020.

[AD20b] Hilal Asi and John C Duchi. Near instance-optimality in differential privacy. CoRR , abs/2005.10630, 2020.

[ADLS17] Jayadev Acharya, Ilias Diakonikolas, Jerry Li, and Ludwig Schmidt. Sample-optimal density estimation in nearly-linear time. In Proceedings of the Twenty-Eighth Annual ACM-SIAM Symposium on Discrete Algorithms , pages 1278-1289. SIAM, 2017.

[AKT + 23] Daniel Alabi, Pravesh K Kothari, Pranay Tankala, Prayaag Venkat, and Fred Zhang. Privately estimating a gaussian: Efficient, robust, and optimal. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing , pages 483-496, 2023.

[AL22] Hassan Ashtiani and Christopher Liaw. Private and polynomial time algorithms for learning gaussians and beyond. In Conference on Learning Theory , pages 1075-1076. PMLR, 2022.

[AM05] Dimitris Achlioptas and Frank McSherry. On spectral learning of mixtures of distributions. In International Conference on Computational Learning Theory , pages 458-469. Springer, 2005.

[ASZ21] Jayadev Acharya, Ziteng Sun, and Huanyu Zhang. Differentially private assouad, fano, and le cam. In Algorithmic Learning Theory , pages 48-78. PMLR, 2021.

[AUZ23] Hilal Asi, Jonathan Ullman, and Lydia Zakynthinou. From robustness to privacy and back. In International Conference on Machine Learning , pages 1121-1146. PMLR, 2023.

[BBC + 23] Shai Ben-David, Alex Bie, Clément L. Canonne, Gautam Kamath, and Vikrant Singhal. Private distribution learning with public data: The view from sample compression. In Advances in Neural Information Processing Systems (NeurIPS) , 2023.

[BDJ + 22] Ainesh Bakshi, Ilias Diakonikolas, He Jia, Daniel M Kane, Pravesh K Kothari, and Santosh S Vempala. Robustly learning mixtures of k arbitrary gaussians. In Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing , pages 12341247, 2022.

[Bis06] Christopher M. Bishop. Pattern Recognition and Machine Learning . Springer, 2006.

[BKS22] Alex Bie, Gautam Kamath, and Vikrant Singhal. Private estimation with public data. Advances in Neural Information Processing Systems , 2022.

[BRST21] Joan Bruna, Oded Regev, Min Jae Song, and Yi Tang. Continuous lwe. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing , pages 694-707, 2021.

[BS10] Mikhail Belkin and Kaushik Sinha. Polynomial learning of distribution families. In 2010 IEEE 51st Annual Symposium on Foundations of Computer Science , pages 103-112. IEEE, 2010.

[BSKW19] Mark Bun, Thomas Steinke, Gautam Kamath, and Zhiwei Steven Wu. Private hypothesis selection. Advances in Neural Information Processing Systems , 32, 2019.

[BUV14] Mark Bun, Jonathan Ullman, and Salil Vadhan. Fingerprinting codes and the price of approximate differential privacy. In Proceedings of the forty-sixth annual ACM symposium on Theory of computing , pages 1-10, 2014.

[CCAd + 23] Hongjie Chen, Vincent Cohen-Addad, Tommaso d'Orsi, Alessandro Epasto, Jacob Imola, David Steurer, and Stefan Tiegel. Private estimation algorithms for stochastic block models and mixture models. arXiv preprint arXiv:2301.04822 , 2023.

[CDSS14] Siu-On Chan, Ilias Diakonikolas, Rocco A Servedio, and Xiaorui Sun. Efficient density estimation via piecewise polynomial approximation. In Proceedings of the forty-sixth annual ACM symposium on Theory of computing , pages 604-613, 2014.

[CKM + 21] Edith Cohen, Haim Kaplan, Yishay Mansour, Uri Stemmer, and Eliad Tsfadia. Differentially-private clustering of easy instances. In International Conference on Machine Learning , pages 2049-2059. PMLR, 2021.

[Das99] Sanjoy Dasgupta. Learning mixtures of gaussians. In 40th Annual Symposium on Foundations of Computer Science (Cat. No. 99CB37039) , pages 634-644. IEEE, 1999.

[DK22] Ilias Diakonikolas and Daniel M. Kane. Algorithmic High-Dimensional Robust Statistics . Cambridge University Press, 2022.

[DKM + 06] Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. In Advances in Cryptology-EUROCRYPT 2006: 24th Annual International Conference on the Theory and Applications of Cryptographic Techniques, St. Petersburg, Russia, May 28-June 1, 2006. Proceedings 25 , pages 486-503. Springer, 2006.

[DKS17] Ilias Diakonikolas, Daniel Kane, and Alistair Stewart. Statistical query lower bounds for robust estimation of high-dimensional gaussians and gaussian mixtures. In 58th IEEE Annual Symposium on Foundations of Computer Science , FOCS '17, pages 73-84, 2017.

[DKY17] Bolin Ding, Janardhan Kulkarni, and Sergey Yekhanin. Collecting telemetry data privately. In Advances in Neural Information Processing Systems (NeurIPS) , pages 3571-3580, 2017.

[DLS + 17] Aref N. Dajani, Amy D. Lauger, Phyllis E. Singer, Daniel Kifer, Jerome P. Reiter, Ashwin Machanavajjhala, Simson L. Garfinkel, Scot A. Dahl, Matthew Graham, Vishesh Karwa, Hang Kim, Philip Lelerc, Ian M. Schmutte, William N. Sexton, Lars Vilhuber, and John M. Abowd. The modernization of statistical disclosure limitation at the u.s. Census Bureau, 2017. In the September 2017 meeting of the Census Scientific Advisory Committee , 2017.

[DMNS06] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3 , pages 265-284. Springer, 2006.

[DR14] Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science , 9(3-4):211-407, 2014.

[EPK14] Úlfar Erlingsson, Vasyl Pihur, and Aleksandra Korolova. RAPPOR: randomized aggregatable privacy-preserving ordinal response. In Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security (CCS) , pages 10541067, 2014.

[GDGK20] Quan Geng, Wei Ding, Ruiqi Guo, and Sanjiv Kumar. Tight analysis of privacy and utility tradeoff in approximate differential privacy. In International Conference on Artificial Intelligence and Statistics (AISTATS) , volume 108 of Proceedings of Machine Learning Research , pages 89-99. PMLR, 2020.

[GVV22] Aparna Gupte, Neekon Vafa, and Vinod Vaikuntanathan. Continuous LWE is as hard as LWE & applications to learning gaussian mixtures. In 63rd IEEE Annual Symposium on Foundations of Computer Science, FOCS 2022, Denver, CO, USA, October 31 November 3, 2022 , pages 1162-1173. IEEE, 2022.

[HKMN23] Samuel B Hopkins, Gautam Kamath, Mahbod Majid, and Shyam Narayanan. Robustness implies privacy in statistical estimation. In Symposium on Theory of Computing , pages 497-506, 2023.

[KMS22a] Gautam Kamath, Argyris Mouzakis, and Vikrant Singhal. New lower bounds for private estimation and a generalized fingerprinting lemma. In Advances in Neural Information Processing Systems 35 , NeurIPS '22, 2022.

[KMS + 22b] Gautam Kamath, Argyris Mouzakis, Vikrant Singhal, Thomas Steinke, and Jonathan Ullman. A private and computationally-efficient estimator for unbounded gaussians. In Conference on Learning Theory , pages 544-572. PMLR, 2022.

[KMV22] Pravesh Kothari, Pasin Manurangsi, and Ameya Velingker. Private robust estimation by stabilizing convex relaxations. In Conference on Learning Theory , pages 723-777. PMLR, 2022.

[KSSU19] Gautam Kamath, Or Sheffet, Vikrant Singhal, and Jonathan Ullman. Differentially private algorithms for learning mixtures of separated gaussians. Advances in Neural Information Processing Systems , 32, 2019.

[KV18] Vishesh Karwa and Salil Vadhan. Finite sample differentially private confidence intervals. In Proceedings of the 9th Conference on Innovations in Theoretical Computer Science , ITCS '18, pages 44:1-44:9, Dagstuhl, Germany, 2018. Schloss DagstuhlLeibniz-Zentrum fuer Informatik.

[LKWB22] Ta-Chun Liu, Peter N Kalugin, Jennifer L Wilding, and Walter F Bodmer. Gmmchi: gene expression clustering using gaussian mixture modeling. BMC bioinformatics , 23(1):457, 2022.

[LL22] Allen Liu and Jerry Li. Clustering mixtures with almost optimal separation in polynomial time. In Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing , pages 1248-1261, 2022.

[LM21] Allen Liu and Ankur Moitra. Settling the robust learnability of mixtures of gaussians. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing , pages 518-531, 2021.

[MH08] Arakaparampil M Mathai and Hans J Haubold. Special functions for applied scientists , volume 4. Springer, 2008.

[MV10] Ankur Moitra and Gregory Valiant. Settling the polynomial learnability of mixtures of gaussians. In 2010 IEEE 51st Annual Symposium on Foundations of Computer Science , pages 93-102. IEEE, 2010.

[Nar23] Shyam Narayanan. Better and simpler lower bounds for differentially private statistical estimation. arXiv preprint arXiv:2310.06289 , 2023.

[NRS07] Kobbi Nissim, Sofya Raskhodnikova, and Adam Smith. Smooth sensitivity and sampling in private data analysis. In Proceedings of the thirty-ninth annual ACM symposium on Theory of computing , pages 75-84, 2007.

[Pea94] Karl Pearson. Contributions to the mathematical theory of evolution. Philosophical Transactions of the Royal Society of London , 1894.

[PH24] Victor S Portella and Nick Harvey. Lower bounds for private estimation of gaussian covariance matrices under all reasonable parameter regimes. arXiv preprint arXiv:2404.17714 , 2024.

[Tea17] Apple Differential Privacy Team. Learning with privacy at scale. Apple Machine Learning Journal , 1(8), 2017.

[VW04] Santosh Vempala and Grant Wang. A spectral algorithm for learning mixture models. Journal of Computer and System Sciences , 68(4):841-860, 2004.

[ZSM + 18] Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In International conference on learning representations , 2018.

A Preliminaries

A.1 Differential Privacy

We state the advanced composition theorem of differential privacy.

Theorem A.1 (Advanced Composition Theorem [DR14, Theorem 3.20]) . Let ε, δ, δ ′ ≥ 0 be arbitrary parameters. Let A 1 , . . . , A k be algorithms on a dataset X , where each A i is ( ε, δ ) -DP. Then, the concatenation A ( X ) = ( A 1 ( X ) , . . . , A k ( X )) is (√ 2 k log 1 δ ′ · ε + kε · ( e ε -1) , k · δ + δ ′ ) -DP.

Moreover, this holds even if the algorithms A i are adaptive. By this, we mean that for all i ≥ 2 the algorithm A i is allowed to depend on A 1 ( X ) , . . . , A i -1 ( X ) . However, privacy must still hold for A i , conditioned on the previous outputs A 1 ( X ) , . . . , A i -1 ( X ) .

Next, we note the properties of the Truncated Laplace distribution and mechanism.

Definition A.2 (Truncated Laplace Distribution) . For ∆ , ε, δ > 0 , the Truncated Laplace Distribution TLap(∆ , ε, δ ) is the distribution with PDF proportional to e -| x |· ε/ ∆ on the region [ -A,A ] , where A = ∆ ε · log ( 1 + e ε -1 2 δ ) , and PDF 0 outside the region [ -A,A ] .

Lemma A.3 (Truncated Laplace Mechanism [GDGK20, Theorem 1]) . Let f : X n → R be a realvalued function, and let ∆ > 0 , such that for any neighboring datasets X , X ′ , | f ( X ) -f ( X ′ ) | ≤ ∆ . Then, the mechanism A that outputs f ( X ) + TLap(∆ , ε, δ ) is ( ε, δ ) -DP.

A.2 Matrix and Concentration Bounds

In this section, we note some standard but useful lemmas. We first note the Courant-Fischer theorem.

Theorem A.4 (Courant-Fischer) . Let A ∈ R d × d be a real symmetric matrix, with eigenvalues λ 1 ≥ λ 2 ≥ · · · ≥ λ d . Then, for each k ,

where V refers to a linear subspace of R d .

By setting A = J ⊤ J , we have the following corollary.

Corollary A.5. Let J ∈ R d × d be a real (possibly asymmetric) matrix with singular values σ 1 ≥ σ 2 ≥ · · · ≥ σ d . Then, for each k ,

Next, we note a basic proposition.

Proposition A.6. [HKMN23, Lemma 6.8] Let M ∈ R d × d be a real symmetric matrix, and J ∈ R d × d be any real-valued (but not necessarily symmetric) matrix such that ∥ JJ ⊤ -I ∥ op ≤ ϕ ≤ 1 , for some parameter ϕ . Then, ∥ J ⊤ MJ ∥ 2 F ≤ (1 + 3 ϕ ) · ∥ M ∥ 2 F .

We also note the Hanson-Wright inequality.

Lemma A.7 (Hanson-Wright) . Given a d × d matrix M ∈ R d × d and a d -dimensional Gaussian vector X ∼ N (0 , I ) , for any t ≥ 0 ,

for some universal constant c > 0 .

Finally, we note a folklore simple characterization of total variation distance (see also [AAL23, Theorem 1.8]).

Lemma A.8. For any µ 1 , µ 2 ∈ R d and positive definite Σ 1 , Σ 2 ∈ R d × d ,

B Robust Estimation of a Single Gaussian

In this section, we establish what a robust algorithm can do for learning a single high-dimensional Gaussian. We will state the accuracy guarantees in terms of the following definition, which combines Mahalanobis distance, spectral distance, and Frobenius distance.

Definition B.1. Given mean-covariance pairs ( µ, Σ) and (ˆ µ, ˆ Σ) , and given parameters γ, ρ, τ > 0 , we say that (ˆ µ, ˆ Σ) ≈ γ,ρ,τ ( µ, Σ) if

∥ Σ -1 / 2 ˆ ΣΣ -1 / 2 -I ∥ op ≤ γ . (Spectral distance)

∥ Σ -1 / 2 ˆ ΣΣ -1 / 2 -I ∥ F ≤ ρ . (Frobenius distance)

∥ Σ -1 / 2 (ˆ µ -µ ) ∥ 2 ≤ τ . (Mahalanobis distance)

We will also use ≈ γ,τ as a shorthand for ≈ γ, √ d · γ,τ , i.e., there is no additional Frobenius norm condition beyond what is already imposed by the operator norm condition.

Although ≈ γ,ρ,τ is neither symmetric nor transitive, symmetry and transitivity happen in an approximate sense. Namely, the following result holds: we defer the proof to Appendix H. We remark that similar results are known (e.g., [AL22, Lemma 3.2], [AAL23, Lemma 4.1]).

Proposition B.2 (Approximate Symmetry and Transitivity) . Fix any γ, ρ, τ > 0 such that γ ≤ 0 . 1 . Then, for any ( µ 1 , Σ 1 ) ≈ γ,ρ,τ ( µ 2 , Σ 2 ) , we have that ( µ 2 , Σ 2 ) ≈ 2 γ, 2 ρ, 2 τ ( µ 1 , Σ 1 ) , and for any ( µ 1 , Σ 1 ) ≈ γ,ρ,τ ( µ 2 , Σ 2 ) and ( µ 2 , Σ 2 ) ≈ γ,ρ,τ ( µ 3 , Σ 3 ) , we have that ( µ 1 , Σ 1 ) ≈ 4 γ, 4 ρ, 4 τ ( µ 3 , Σ 3 ) .

We note the following theorem about the accuracy of robustly learning a single Gaussian. While this result will roughly follow from known results and techniques in the literature, we were unable to find a formal statement of the theorem, so we prove it in Appendix H.

Theorem B.3. For some universal constants c 0 ∈ (0 , 0 . 01) and C 0 > 1 , the following holds. Fix any η ≤ γ ≤ c 0 , β < 1 , and ρ such that ˜ O ( η ) ≤ ρ ≤ c 0 √ d . Let n ≥ ˜ O ( d +log(1 /β ) γ 2 + ( d +log(1 /β )) 2 ρ 2 ) . Then, there exists an (inefficient, deterministic) algorithm A 0 with the following property. For any µ, Σ , with probability at least 1 -β over X = { X 1 , . . . , X n } ∼ N ( µ, Σ) , and for all datasets X ′ with d H ( X , X ′ ) ≤ η · n , we have

We remark that A 0 may have knowledge of η, γ, ρ, β , but does not have knowledge of µ, Σ , or the uncorrupted data X .

B.1 Directional bounds

In this section, we note that, when given i.i.d. samples from a Gaussian, with high probability one cannot choose a reasonably large subset with an extremely different empirical mean or covariance.

First, we note the following proposition, for which the proof follows by an ε -net type argument.

Proposition B.4. Let n ≥ n ′ ≥ 20 d , and L ≥ C 1 n 2 for a sufficiently large constant C 1 . Suppose that some data points X = { X 1 , . . . , X n } are i.i.d. sampled from N ( 0 , I ) . Then, with probability at least 1 -n -Ω( n ′ ) , the following both hold.

For all X i ∈ X , ∥ X i ∥ 2 ≤ L .

For all subsets Y ⊂ X of size n ′ , there does not exist any real number r and any unit vector v such that |⟨ X i , v ⟩ -r | ≤ 1 L for all X i ∈ Y .

Proof. If ∥ X i ∥ 2 ≥ L , then X ⊤ MX ≥ L 2 for M the d × d identity matrix I . However, E [ X ⊤ MX ] = d . So, by Lemma A.7, the probability of this event is at most 2 e -c · min( n 4 /d,n 2 ) ≤ 2 e -c · n 2 ≤ n -Ω( n ′ ) .

Next, we bound the probability that the first item holds but the second item doesn't hold. First, we can create a 1 L 2 -net of unit vectors v ′ , of size (3 L 2 ) d ≤ L 3 d , and a 1 L -net of real numbers r ′

from [ -L, L ] , of size 2 L 2 . Now, suppose that the first item holds, but there is some unit vector v and real number r with |⟨ X i , v ⟩ -r | ≤ 1 L for all X i ∈ Y , for some Y ⊂ X of size n ′ . In this case, ⟨ X i , v ⟩ ≤ ∥ X i ∥ 2 ≤ L for all X i ∈ X , so we can assume that | r | ≤ L . Thus, if v ′ is the closest unit vector to v in the net and r ′ is the closest real number to r in the net, then |⟨ X i , v ′ ⟩ -r ′ | ≤ |⟨ X i , v ⟩ -r | + |⟨ X i , v ′ -v ⟩| + | r -r ′ | ≤ 1 L + ∥ X i ∥ 2 · 1 L 2 + 1 L ≤ 3 L .

In other words, if the first event holds and the second does not, there exists v ′ , r ′ in the net and Y ⊂ X of size n ′ , such that |⟨ X i , v ′ ⟩ -r ′ | ≤ 3 L for all X i ∈ Y . We can now bound this via a union bound. To perform the union bound, we first bound the probability of this event holding for some fixed v ′ , r ′ , Y .

For any fixed v ′ , r ′ , Y , note that ⟨ X i , v ′ ⟩ X i ∈ Y is just n ′ i.i.d. copies of N (0 , 1) . Let's call these values z 1 , . . . , z n ′ . If there is some r ′ such that | z i -r ′ | ≤ 3 L for all i , then | z i -z 1 | ≤ 6 L for all i . Since the PDF of a N (0 , 1) is uniformly bounded by at most 1 2 , for any fixed z 1 , the probablity that any other z i is within 6 L of z 1 is at most 6 L , so the overall probability is at most (6 /L ) n ′ -1 .

Now, the union bound is done over ( n n ′ ) choices of Y , at most L 3 d choices of v ′ , and at most 2 L 2 choices of Y . Thus, the overall probability is at most (6 /L ) n ′ -1 · n n ′ · L 3 d · 2 L 2 ≤ (6 n ) n ′ /L n ′ -3 -3 d ≥ (6 n/L 0 . 7 ) n ′ . Thus, as long as L ≥ 100 n 2 , this is much smaller than n -Ω( n ′ ) .

By shifting and scaling appropriately, we have the following corollary.

Corollary B.5. Let n ≥ n ′ ≥ 20 d , and L ≥ C 1 n 2 , for C 1 the same constant as in Proposition B.4. Suppose that some data points X = { X 1 , . . . , X n } are drawn from N ( µ, Σ) . Then, with probability at least 1 -n -Ω( n ′ ) , the following both hold.

For all X i ∈ X , ∥ Σ -1 / 2 ( X i -µ ) ∥ 2 ≤ L .

For all subsets Y ⊂ X of size n ′ , there does not exist any real number r and any nonzero vector v such that |⟨ X i , v ⟩ -r | ≤ 1 L ∥ Σ 1 / 2 v ∥ 2 for all X i ∈ Y .

B.2 Modified robust algorithm

We can modify the robust algorithm of Theorem B.3 to have the following guarantees.

Lemma B.6. Let c 0 , C 0 be as in Theorem B.3, and C 1 be as in Proposition B.4. Let L := C 1 · n 2 . Also, fix any η ≤ γ ≤ c 0 , any e -d ≤ β < 1 , and and any ρ such that ˜ O ( η ) ≤ ρ ≤ c 0 √ d . Finally, suppose n ≥ m ≥ ˜ O ( d γ 2 + d 2 ρ 2 ) .

Then, there exists an (inefficient, deterministic) algorithm A that, on a dataset Y of size m , outputs either some (ˆ µ, ˆ Σ) or ⊥ , with the following properties.

̸

For any datasets Y , Y ′ with d H ( Y , Y ′ ) ≤ η · m , if A ( Y ) = (ˆ µ, ˆ Σ) = ⊥ and A ( Y ′ ) = (ˆ µ ′ , ˆ Σ ′ ) = ⊥ , then

For any fixed µ, Σ , with probability at least 1 -O ( β ) over Y 1 , . . . , Y m ∼ N ( µ, Σ) , A ( Y ) = ⊥ and

Fix an integer k ≥ 1 , and additionally suppose that m ≥ 40 d · k . Suppose that X = { X 1 , . . . , X n } are drawn i.i.d. from some mixture of k Gaussians ( µ i , Σ i ) . Then, with probability at least 1 -O ( β ) over X , for every Y ′ ⊂ X of size m and every Y with d H ( Y , Y ′ ) ≤ m/ 2 , either A ( Y ) = ⊥ , or A ( Y ) = (ˆ µ, ˆ Σ) , where there is some i ∈ [ k ] such that 1 9 L 4 · Σ i ≼ ˆ Σ ≼ 9 L 4 · Σ i and ∥ Σ -1 / 2 i (ˆ µ -µ i ) ∥ 2 ≤ 10 L 3 .

As in Theorem B.3, A has knowledge of η, γ, ρ, β , but not µ or Σ .

̸

̸

Proof. The algorithm A works as follows. Start off by computing A 0 ( Y ) = (ˆ µ, ˆ Σ) , where A 0 is the algorithm of Theorem B.3. We can run the algorithm on any arbitrary dataset, even if it didn't come from Gaussian samples, and we can assume that the algorithm A 0 always outputs some mean-covariance pair (for instance, by having it output ( 0 , I ) by default instead of ⊥ ). Next, A , on a dataset Y , checks if any of the following three conditions hold.

(a) There exists Y ′ such that d H ( Y , Y ′ ) ≤ η · m , and A 0 ( Y ′ ) ̸≈ 8 C 0 γ, 8 C 0 ρ, 8 C 0 γ A 0 ( Y ) .

(b) There exists Y i ∈ Y such that ∥ ˆ Σ -1 / 2 ( Y i -ˆ µ ) ∥ 2 > 3 L .

(c) There exists a subset Z ⊂ Y of size 20 d , and a unit vector v and real number r , such that for all Y i ∈ Z , |⟨ Y i , v ⟩ -r | < 1 2 L · ∥ ˆ Σ 1 / 2 v ∥ 2 .

If any of these conditions hold, the output is A ( Y ) = ⊥ . Otherwise, the output is A ( Y ) = A 0 ( Y ) .

̸

We now verify that the algorithm satisfies the required properties. Property 1 clearly holds, because if there exist datasets Y , Y ′ with d H ( Y , Y ′ ) ≤ η · m , A ( Y ) = ⊥ and A ( Y ′ ) = ⊥ , and A ( Y ) ̸≈ 8 C 0 γ, 8 C 0 ρ, 8 C 0 γ A ( Y ′ ) , then we would have in fact set A ( Y ) to ⊥ .

̸

For Property 2, note that with probability 1 -O ( β ) , A 0 ( Y ) satisfies the desired property by Theorem B.3, since β ≥ e -d so d +log1 /β = O ( d ) . So, we just need to make sure that we don't set A ( Y ) to be ⊥ . Since Y 1 , . . . , Y m ∼ N ( µ, Σ) , then with probability 1 -O ( β ) , both Y and any Y ′ with d H ( Y , Y ′ ) ≤ η · n satisfy the conditions of Theorem B.3. In other words, A 0 ( Y ) ≈ C 0 γ,C 0 ρ,C 0 γ and A 0 ( Y ′ ) ≈ C 0 γ,C 0 ρ,C 0 γ . So, by Proposition B.2, A 0 ( Y ′ ) ≈ 8 C 0 γ, 8 C 0 ρ, 8 C 0 γ A 0 ( Y ) for any Y ′ with d H ( Y , Y ′ ) ≤ η · n . Thus, condition a) is not met.

Moreover, by Corollary B.5, with failure probability at most m -Ω(20 d ) ≤ β , ∥ Σ -1 / 2 ( Y i -µ ) ∥ 2 ≤ L for all Y i ∈ Y , and for all Z ⊂ Y of size 20 d , there does not exist a real r and a nonzero vector v such that |⟨ Y i , v ⟩ -r | ≤ 1 L · ∥ Σ 1 / 2 v ∥ 2 for all Y i ∈ Y . Now, we know that by Theorem B.3, (ˆ µ, ˆ Σ) ≈ 0 . 5 , 0 . 5 ( µ, Σ) . So,

and if |⟨ Y i , v ⟩ -r | ≤ 1 2 L · ∥ ˆ Σ 1 / 2 v ∥ 2 , then |⟨ Y i , v ⟩ -r | ≤ 1 L · ∥ Σ 1 / 2 v ∥ 2 , for all Y i ∈ Y . Thus, conditions b) and c) are also not met, so Property 2 holds.

For Property 3 , note that if m ≥ 40 d · k , then | Y ∩ X | ≥ 20 d · k , i.e., Y contains at least 20 dk uncorrupted points. So, by the Pigeonhole Principle, for every possible Y , there is some index i ∈ [ k ] such that at least 20 d points in Y ∩ X come from the i th Gaussian component.

Now, let us condition on the event that Corollary B.5 holds for X , where n ′ = 20 d , which happens with at least 1 -n -Ω( n ′ ) ≥ 1 -β probability. We claim that for every possible Y , and for an i ∈ [ k ] such that 20 d points Z ⊂ Y ∩ X came from the i th Gaussian component, then either A ( Y ) = ⊥ or if A ( Y ) = (ˆ µ, ˆ Σ) then 1 9 L 4 · Σ i ≼ ˆ Σ ≼ 9 L 4 · Σ i and ∥ Σ -1 / 2 i (ˆ µ -µ i ) ∥ 2 ≤ 10 L 3 .

First, we verify that 1 9 L 4 · Σ i ≼ ˆ Σ ≼ 9 L 4 · Σ i . Otherwise, there exists a unit vector v such that either v ⊤ ˆ Σ v ≥ 9 L 4 · v ⊤ Σ i v or v ⊤ ˆ Σ v ≤ 1 9 L 4 · v ⊤ Σ i v . In the former case, by Part 1 of Corollary B.5, for all Y i ∈ Z , ∥ Σ -1 / 2 i ( Y i -µ i ) ∥ 2 ≤ L , so

Thus, if we set r = ⟨ v, µ i ⟩ , then |⟨ Y i , v ⟩ -r | < 1 2 L · ∥ ˆ Σ 1 / 2 v ∥ 2 for all Y i ∈ Z , so the algorithm A would have output ⊥ due to condition c). Alternatively, if there exists a unit vector v such that v ⊤ ˆ Σ v ≤ 1 9 L 4 · v ⊤ Σ i v , then by condition b), ∥ ˆ Σ -1 / 2 ( Y i -ˆ µ ) ∥ 2 ≤ 3 L for all Y i ∈ Z , so

So, there exists r = ⟨ v, ˆ µ ⟩ such that |⟨ v, Y i ⟩ -r | ≤ 1 L · √ v ⊤ Σ i v which is impossible by Part 2 of Corollary B.5

Next, we verify that ∥ Σ -1 / 2 i (ˆ µ -µ i ) ∥ 2 ≤ 10 L 3 . By Triangle inequality, for every Y i ∈ Z , ∥ Σ -1 / 2 i ( Y i -ˆ µ ) ∥ 2 + ∥ Σ -1 / 2 i ( Y i -µ i ) ∥ 2 ≥ ∥ Σ -1 / 2 i ( µ i -ˆ µ ) ∥ 2 . But, ∥ Σ -1 / 2 i ( Y i -µ i ) ∥ 2 ≤ L by Part 1 of Corollary B.5, and ∥ Σ -1 / 2 i ( Y i -ˆ µ ) ∥ 2 ≤ 3 L 2 · ∥ ˆ Σ -1 / 2 ( Y i -ˆ µ ) ∥ 2 ≤ 3 L 2 · 3 L ≤ 9 L 3 , because we just proved that 9 L 4 Σ i ≽ ˆ Σ and assuming we do not reject due to condition b). Thus, ∥ Σ -1 / 2 i ( µ i -ˆ µ ) ∥ 2 ≤ 9 L 3 + L ≤ 10 L 3 .

C Volume and the Robustness-to-Privacy Conversion

In this section, we explain the robustness-to-privacy conversion [HKMN23] that we will utilize in proving Theorem 1.4. We will also need some relevant results about computing volume , which will be important in the robustness-to-privacy conversion.

C.1 Normalized Volume

Given a pair ( µ, Σ) , where µ ∈ R d and Σ ∈ R d × d is positive definite, we will define Proj( µ, Σ) ∈ R d · ( d +3) / 2 to represent the coordinates of µ along with the upper-diagonal coordinates of Σ . For any set Ω of mean-covariance pairs ( µ, Σ) , define Proj(Ω) := { Proj( µ, Σ) : ( µ, Σ) ∈ Ω } . Because Σ is symmetric, Proj( µ, Σ) fully encodes the information about µ and Σ . We also define vol(Ω) to be the Lebesgue measure of Proj(Ω) , i.e., the d ( d +3) 2 -dimensional measure of all points Proj( µ, Σ) for ( µ, Σ) ∈ Ω . Next, we will define the normalized volume

where θ = Proj( µ, Σ) , and we take a Lebesgue integral over θ ∈ Proj(Ω) .

To motivate this choice of normalized volume, we will see that the volume is invariant under some basic transformations.

Given a function f : R D → R D , for some integer D ≥ 1 we recall that the Jacobian J at some x is the matrix with J ij = ∂f i ∂x j ( x ) . For a function that takes a symmetric matrix Σ and outputs the symmetric matrix A Σ A ⊤ , we view D = d ( d +1) 2 and the function Σ ↦→ A Σ A ⊤ as a function f from R D → R D by taking the upper triangular part of both Σ and A Σ A ⊤ . Note that f is a linear function (thus f ( x ) = J · x where J ∈ R D × D , and moreover, the following fact is well-known.

Lemma C.1. [MH08, Theorem 11.1.5] For J as defined above, det( J ) = det( A ) d +1 .

Now, for any fixed µ ∈ R d and positive definite Σ ∈ R d × d , consider the transformation (ˆ µ, ˆ Σ) ↦→ (Σ 1 / 2 ˆ ΣΣ 1 / 2 , Σ 1 / 2 ˆ µ + µ ) , viewed as a linear map g from Proj(ˆ µ, ˆ Σ) → Proj(Σ 1 / 2 ˆ µ + µ, Σ 1 / 2 ˆ ΣΣ 1 / 2 ) . By setting A := Σ 1 / 2 , note that the map g behaves like f on the last d ( d +1) 2 coordinates, and on the first d coordinates, it is simply an affine map ˆ µ ↦→ Σ 1 / 2 ˆ µ + µ . Therefore, the overall linear map g has determinant det(Σ 1 / 2 ) · det(Σ 1 / 2 ) d +1 = (det Σ) ( d +2) / 2 .

From this, we can infer the following.

Lemma C.2. Fix any µ ∈ R d and positive definite Σ ∈ R d × d . Let h be the map (ˆ µ, ˆ Σ) ↦→ (Σ 1 / 2 ˆ µ + µ, Σ 1 / 2 ˆ ΣΣ 1 / 2 ) . Then, for any set S of mean-covariance pairs, the normalized volume of S equals the normalized volume of h ( S ) .

Proof. Let g be the corresponding map from Proj(ˆ µ, ˆ Σ) to Proj(Σ 1 / 2 ˆ µ + µ, Σ 1 / 2 ˆ ΣΣ 1 / 2 ) . Using a simple integration by substitution, we have

C.2 Robustness to Privacy Conversion

We note the following restatement of (the inefficient, approx-DP version of) the main robustness-toprivacy conversion of [HKMN23].

In the following theorem, we will think of the parameter space as lying in R D , with some normalized volume vol n (Ω) = ∫ θ ∈ Ω p ( θ ) dθ, where p ≥ 0 is some nonnegative Lebesgue-measurable function and the integration is Lebesgue integration. In our application, we will think of θ = Proj( µ, Σ) , and p ( θ ) = (det Σ) -( d +2) / 2 , to match with (1).

Theorem C.3. Restatement of [HKMN23, Lemma 4.2] Let 0 < η < 0 . 1 and 10 η ≤ η ∗ ≤ 1 be fixed parameters. Also, fix privacy parameters ε 0 , δ 0 < 1 and confidence parameter β 0 < 1 . Let S = S ( θ, X ) ∈ R ≥ 0 be a score function that takes as input a dataset X = { X 1 , . . . , X n } and a parameter θ ∈ Θ ⊂ R D . For any dataset X and any 0 ≤ η ′ ≤ 1 , let V η ′ ( X ) be the D -dimensional normalized volume of points θ ∈ Θ with score at most η ′ · n , i.e., V η ′ ( X ) = vol n ( { θ ∈ Θ : S ( θ, X ) ≤ η ′ · n } ) .

Suppose the following properties hold:

(Bounded Sensitivity) For any two adjacent datasets X , X ′ and any θ ∈ Θ , |S ( θ, X ) -S ( θ, X ′ ) | ≤ 1 .

(Volume) For some universal constant C , and for any X of size n such that there exists θ with S ( θ, X ) ≤ 0 . 7 η ∗ n , n ≥ C · log( V η ∗ ( X ) /V 0 . 8 η ∗ ( X ))+log(1 /δ 0 ) ε 0 · η ∗ .

Then, there exists an ( ε 0 , δ 0 ) -DP algorithm M , that takes as input X and outputs either some θ ∈ Θ or ⊥ , such that for any dataset X , if n ≥ C · max η ′ : η ≤ η ′ ≤ η ∗ log( V η ′ ( X ) /V η ( X ))+log(1 / ( β 0 · η )) ε 0 · η ′ , then M ( X ) outputs some θ ∈ Θ of score at most 2 ηn with probability 1 -β 0 .

We remark that the algorithm M is allowed prior knowledge of n, D, η, η ∗ , ε 0 , δ 0 , β 0 , as well as the domain Θ , function p ( θ ) that dictates the normalized volume, and score function S .

We remark that the original result in [HKMN23] assumes the volumes are unnormalized, but the result immediately generalizes to normalized volumes. To see why, consider a modified domain Θ ′ ∈ R D +1 , where θ ′ = ( θ, z ) ∈ Θ ′ if and only if θ ∈ Θ and 0 ≤ z ≤ p ( θ ) . Also, consider a modified score function S ′ acting on Θ ′ , where S ′ (( θ, z ) , X ) := S ( θ, X ) for for any ( θ, z ) ∈ Θ ′ .

Then, note that the unnormalized volume of Θ ′ , by Fubini's theorem, is precisely ∫∫ ( θ,z ) ∈ Θ ′ 1 dzdθ = ∫ θ ∈ Θ p ( θ ) dθ , which is precisely the normalized volume of corresponding to the new Θ ′ precisely match the normalized volumes corresponding to the old Θ . A similar calculation will give us that the unnormalized volume of points ( θ, z ) in Θ ′ with S ′ (( θ, z ) , X ) ≤ t equals the normalized volume of points θ ∈ Θ with S ( θ, X ) ≤ t , for any t .

Thus, to privately find a parameter θ ∈ Θ of low score with respect to S (assuming the normalized volume constraints hold), can create Θ ′ and S ′ , and apply the unnormalized version of Theorem C.3 to obtain some ( θ, z ) . Also, if S ′ (( θ, z ) , X ) ≤ 2 ηn , then S ( θ ) = S ′ (( θ, z ) , X ) ≤ 2 ηn , and if outputting ( θ, z ) is ( ε, δ ) -DP, then so is simply outputting θ .

C.3 Computing Normalized Volume Ratios

Lemma C.4. Let M ∈ R d × d be symmetric with ∥ M ∥ op ≤ 0 . 1 , and let 1 ≤ ν ≤ 2 be a scaling parameter. Then,

Proof. If the eigenvalues of M are λ 1 , . . . , λ d ∈ [ -0 . 1 , 0 . 1] , then

Note that for x ∈ [ -0 . 1 , 0 . 1] and ν ≥ 1 , 1+ ν · x 1+ x is an increasing function. Therefore,

where we used the fact that 1 ≤ ν ≤ 2 .

We note an important lemmas about the normalized volumes of certain sets.

Lemma C.5. Fix some γ, τ ≤ 0 . 1 and some scaling parameters 1 ≤ ν 1 ≤ 2 and 1 ≤ ν 2 . Let R 1 be the set of ( µ, Σ) ≈ γ,τ ( 0 , I ) and R 2 be the set of ( µ, Σ) ≈ ν 1 · γ,ν 2 · τ ( 0 , I ) . Then, the ratio of normalized volumes

Proof. By definition of normalized volume, we have

where we are slightly abusing notation because we are truly integrating along the upper-triangular part of M . (Overall, the integral is d ( d +3) 2 -dimensional.) We have a similar expression for vol n ( R 1 ) .

Let us consider the map sending ( µ, I + M ) ↦→ ( ν 2 · µ, I + ν 1 · M ) . This is a linear map, and for ∥ µ ∥ 2 ≤ γ and symmetric ∥ M ∥ op ≤ τ , this is a bijective map from R 1 to R 2 . Therefore, by an integration by substitution, we have

Above, the first line is integration by substitution, and the second line uses Lemma C.4.

Next, we note the following bound on the size of a net of matrices with small Frobenius norm.

Lemma C.6. Fix some γ ≤ 0 . 1 and 1 d ≤ ρ ≤ 0 . 1 d . Let R 3 be the set of symmetric matrices M ∈ R d × d with ∥ M ∥ op ≤ γ and ∥ M ∥ F ≤ ρ . Then, for any 1 ≤ ℓ ≤ d , there exists a net B of size at most e O ( ℓ · d log d ) such that for any M ∈ R 3 , there exists B ∈ B such that ∥ M -B ∥ op ≤ 2 ρ √ ℓ .

√

and

Proof. First, note that the unit sphere in R d has a 1 d 10 -net (in Euclidean distance) of size e O ( d log d ) . Let B 0 be this net. The net B will be the set of matrices ∑ ℓ i =1 κ i w i w ⊤ i , where every w i ∈ B 0 , every κ i is an integral multiple of 1 d 10 , and every | κ i | ≤ γ . The cardinality of B is at most |B 0 | ℓ · ( d 10 ) ℓ ≤ e O ( ℓ · d log d ) .

Now, we show that B is actually a net. For any matrix M ∈ R 3 , we can write M = ∑ d i =1 λ i v i v ⊤ i , where λ i , v i are the eigenvalues and eigenvectors, respectively, of M . Assume the eigenvalues are sorted so that | λ i | are in decreasing order. Now, since ∑ d i =1 λ 2 i ≤ ρ 2 , which means that | λ i | ≤ ρ √ ℓ for all i > ℓ.

Now, let B 1 = ∑ i>ℓ λ i v i v ⊤ i : since the v i 's are orthogonal and | λ i | ≤ ρ √ ℓ , this means that ∥ B 1 ∥ op ≤ α √ ℓ . Also, note that M = B 1 + ∑ ℓ i =1 λ i v i v ⊤ i . Suppose we replace each λ i with κ i by rounding to the nearest multiple of 1 /d 10 , and replace each v i with w i ∈ B 0 such that ∥ v i -w i ∥ 2 ≤ 1 /d 10 . Then, ∑ ℓ i =1 κ i w i w ⊤ i is in B , and by Triangle inequality, we have

We also note the following lemma. We defer the proof to Appendix H.

Lemma C.7. Fix any µ and positive definite Σ . Then, for any µ 1 , µ 2 , Σ 1 , Σ 2 , we have that ( µ 1 , Σ 1 ) ≈ γ,ρ,τ ( µ 2 , Σ 2 ) if and only if (Σ 1 / 2 µ 1 + µ, Σ 1 / 2 Σ 1 Σ 1 / 2 ) ≈ γ,ρ,τ (Σ 1 / 2 µ 2 + µ, Σ 1 / 2 Σ 2 Σ 1 / 2 ) .

Our main lemma in this subsection is the following, which roughly bounds the normalized volume of a Frobenius norm ball 'fattened' by an operator norm ball.

Lemma C.8. Let c 1 ∈ (0 , 0 . 01) be a sufficiently small universal constant. Fix any µ and positive definite Σ . Fix some parameters γ 1 , γ 2 ∈ ( 1 d , c 1 ) and ρ 2 ∈ ( 1 d , γ 3 1 1000 · √ d ) . Let T 1 ( µ, Σ) represent the set of { ( µ 1 , Σ 1 ) } such that ( µ 1 , Σ 1 ) ≈ γ 1 ,γ 1 ( µ, Σ) . Let T 2 ( µ, Σ) represent the set of { ( µ 2 , Σ 2 ) } such that ( µ 2 , Σ 2 ) ≈ γ 1 ,γ 1 ( µ 1 , Σ 1 ) for some ( µ 1 , Σ 1 ) ≈ γ 2 ,ρ 2 ,γ 2 ( µ, Σ) . Then,

Furthermore, both vol n ( T 1 ( µ, Σ)) and vol n ( T 2 ( µ, Σ)) do not change even if we change µ, Σ .

Proof. For now, we also assume that µ = 0 and Σ = I .

Let 1 ≤ ℓ ≤ d be decided later, and let B be the net from Lemma C.6, where we set γ = γ 2 and ρ = ρ 2 . Let B 1 be a 1 d 10 -net (in Euclidean distance) over the d -dimensional unit ball, i.e., over µ with ∥ µ ∥ 2 ≤ 1 . Note that |B 1 | ≤ e O ( d log d ) , and recall that |B| ≤ e O ( ℓ · d log d ) . Now, for any ( µ 2 , Σ 2 ) ∈ T 2 ( 0 , I ) , we can write ( µ 2 , Σ 2 ) ≈ γ 1 ,γ 1 ( µ 1 , Σ 1 ) , where ∥ Σ 1 -I ∥ op ≤ γ 2 , ∥ Σ 1 -I ∥ F ≤ ρ 2 , and ∥ µ 1 ∥ 2 ≤ γ 2 . Now, we can choose some µ ′ ∈ B 1 with ∥ µ 1 -µ ′ ∥ 2 ≤ d -10 and B ∈ B such that ∥ (Σ 1 -I ) -B ∥ op ≤ 2 ρ 2 √ ℓ .

What can we say about the relationship between ( µ 2 , Σ 2 ) and ( µ ′ , I + B ) ? First, note that because ( µ 2 , Σ 2 ) ≈ γ 1 ,γ 1 ( µ 1 , Σ 1 ) , we have (1 -γ 1 ) · Σ 1 ≼ Σ 2 ≼ (1+ γ 1 ) · Σ 1 . Next, since ∥ Σ 1 -( I + B ) ∥ op = ∥ (Σ 1 -I ) -B ∥ op ≤ 2 ρ 2 √ ℓ , and since Σ 1 has all eigenvalues between 1 -γ 2 ≥ 1 2 and 1 + γ 2 ≤ 2 , this means ( 1 -4 ρ 2 √ ℓ ) · Σ 1 ≼ ( I + B ) ≼ ( 1 + 4 ρ 2 √ ℓ ) · Σ 1 . Thus, if 4 ρ < √ ℓ , we have that

Next, because ( µ 2 , Σ 2 ) ≈ γ 1 ,γ 1 ( µ 1 , Σ 1 ) , we have ∥ Σ -1 / 2 1 ( µ 2 -µ 1 ) ∥ 2 ≤ γ 1 , which means ∥ µ 2 -µ 1 ∥ 2 ≤ 2 γ 1 . Thus, because ∥ µ 1 -µ ′ ∥ 2 ≤ d -10 ≤ γ 1 , this means that ∥ µ 2 -µ ′ ∥ 2 ≤ 3 γ 1 . Moreover,

assuming 10 ρ ≤ √ ℓ , the eigenvalues of I + B are at least (1 -γ 1 ) · ( 1 -4 ρ 2 √ ℓ ) ≥ 1 4 , which means that ∥ ( I + B ) -1 / 2 ( µ 2 -µ ′ ) ∥ 2 ≤ 6 γ 1 .

In summary, if 10 ρ ≤ √ ℓ , then

Note that if γ 1 ≤ 0 . 1 and 10 ρ ≤ √ ℓ , then by a simple calculation, 1+ γ 1 1 -(4 ρ 2 / √ ℓ ) ≤ 1 + γ 1 + 10 ρ 2 √ ℓ and 1 -γ 1 1+(4 ρ 2 / √ ℓ ) ≥ 1 -γ 1 -4 ρ 2 √ ℓ . This implies that ( µ 2 , Σ 2 ) ≈ γ 1 +10 ρ 2 / √ ℓ, 6 γ 1 ( µ ′ , I + B ) , for some µ ′ ∈ B 1 and B ∈ B . Note that this holds for any ( µ 2 , Σ 2 ) ∈ T 2 ( 0 , I ) .

Therefore, recalling that |B 1 | ≤ e O ( d log d ) , and |B| ≤ e O ( ℓ · d log d ) we can cover T 2 ( 0 , I ) with at most e O ( ℓ · d log d ) regions, each of which is the set of ( µ , Σ ) ≈ √ ( µ ′ , Σ ′ ) .

2 2 γ 1 +10 ρ 2 / ℓ, 6 γ 1 Now, by Lemma C.7, ( µ 2 , Σ 2 ) ≈ γ 1 +10 ρ 2 / √ ℓ, 6 γ 1 ( µ ′ , Σ ′ ) if and only if µ 2 = Σ ′ 1 / 2 µ 0 + µ ′ and Σ 2 = Σ ′ 1 / 2 Σ 0 Σ ′ 1 / 2 , where ( µ 0 , Σ 0 ) ≈ γ 1 +10 ρ 2 / √ ℓ, 6 γ 1 ( 0 , I ) . By Lemma C.2, this means that the volume of { ( µ 2 , Σ 2 ) : ( µ 2 , Σ 2 ) ≈ γ 1 +10 ρ 2 / √ ℓ, 6 γ 1 ( µ ′ , Σ ′ ) } equals the volume of { ( µ 0 , Σ 0 ) : ( µ 0 , Σ 0 ) ≈ γ 1 +10 ρ 2 / √ ℓ, 6 γ 1 ( 0 , I ) } .

Thus, if 10 ρ ≤ √ ℓ , then T 2 ( 0 , I ) can be covered by e O ( ℓ · d log d ) regions, each of which has the same normalized volume as the volume of { ( µ 0 , Σ 0 ) : ( µ 0 , Σ 0 ) ≈ γ +10 ρ / √ ℓ, 6 γ ( 0 , I ) } . Moreover, if we

1 2 1 further have 10 ρ 2 √ ℓ ≤ γ 1 , then by Lemma C.8, this is at most ( 1 + 10 ρ 2 √ ℓ · γ 1 ) 2 d 2 · 6 d · vol n ( T 1 ( 0 , I )) .

Finally, we show how to remove the assumption that µ = 0 and Σ = I . First, note that for any general µ, Σ , ( µ 1 , Σ 1 ) ∈ T 1 ( 0 , I ) if and only if (Σ 1 / 2 µ 1 + µ, Σ 1 / 2 Σ 1 Σ 1 / 2 ) ∈ T 1 ( µ, Σ) , by Lemma C.7. So, by Lemma C.2, vol n ( T 1 ( µ, Σ)) = vol n ( T 1 ( 0 , I )) . Next, ( µ 2 , Σ 2 ) ≈ γ 1 ,γ 1 ( µ 1 , Σ 1 ) and ( µ 1 , Σ 1 ) ≈ γ 2 ,ρ 2 ,γ 2 ( 0 , I ) if and only if (Σ 1 / 2 µ 2 + µ, Σ 1 / 2 Σ 2 Σ 1 / 2 ) ≈ γ 1 ,γ 1 (Σ 1 / 2 µ 1 + µ, Σ 1 / 2 Σ 1 Σ 1 / 2 ) and (Σ 1 / 2 µ 1 + µ, Σ 1 / 2 Σ 1 Σ 1 / 2 ) ≈ γ 2 ,ρ 2 ,γ 2 ( µ, Σ) , by two applications of Lemma C.7. So, by Lemma C.2, vol n ( T 2 ( µ, Σ)) = vol n ( T 2 ( 0 , I )) . Thus, the volume ratios stay the same as well.

D Fine Approximation via Hypothesis Selection

In this section, we prove that if we know a very crude approximation to all of the Gaussian mixture components with sufficiently large weight, we can privately learn the full density of the Gaussian mixture model up to low total variation distance. This will be useful for proving both Theorem 1.4 and Theorem 1.5.

We start with the following auxiliary lemma.

Lemma D.1. Let G ≥ 1 and ζ ≤ 1 be some parameters. For some d ≥ 1 , let ˆ µ ∈ R d , and let ˆ Σ ∈ R d × d be positive definite. Let U ˆ µ, ˆ Σ be the set of ( µ, Σ) such that 1 G · ˆ Σ ≼ Σ ≼ G · ˆ Σ and ∥ ˆ Σ -1 / 2 ( µ -ˆ µ ) ∥ 2 ≤ G . Then, there exists a net B ˆ µ, ˆ Σ of size O ( G √ d/ζ ) d ( d +3) such that for every ( µ, Σ) ∈ U ˆ µ, ˆ Σ , there exists ( µ ′ , Σ ′ ) ∈ B ˆ µ, ˆ Σ such that d TV ( N ( µ, Σ) , N ( µ ′ , Σ ′ )) ≤ ζ .

Proof. First, assume that ˆ Σ = I and ˆ µ = 0 . Now, let us consider the set U := U 0 ,I = { ( µ, Σ) : ∥ µ ∥ 2 ≤ G, 1 G · I ≼ Σ ≼ G · I } . Let G ′ ≥ G be a parameter that we will set later. Now, we can look

at the un-normalized volume of U viewed in R d ( d +3) / 2 (i.e., we are projecting to the upper-triangular part of Σ ). First, we claim (using a standard volume argument) that there is a cover B ⊂ U of size (2 G ′ ) d ( d +3) , such that for every ( µ, Σ) ∈ U , there is ( µ ′ , Σ ′ ) ∈ B with ∥ µ -µ ′ ∥ 2 ≤ 1 G ′ and ∥ Σ -Σ ′ ∥ op ≤ 1 G ′ .

̸

To see why, consider a maximal packing of ( µ ′ i , Σ ′ i ) ∈ U such that for every i = j , either ∥ µ ′ i -µ ′ j ∥ 2 > 1 G ′ or ∥ Σ ′ i -Σ ′ j ∥ op > 1 G ′ . Then, this set is clearly a cover of U (or else we could have increased the size of the packing, breaking maximality). Then, the sets S i := { ( µ, Σ) : ∥ µ -µ ′ i ∥ ≤ 1 2 G ′ , ∥ Σ -Σ ′ i ∥ op ≤ 1 2 G ′ } are disjoint, and are all contained in the set S := { ( µ, Σ) : ∥ µ ∥ 2 ≤ 2 G ′ , ∥ Σ ∥ op ≤ 2 G ′ } . S is just a shifting and scaling of S i by a factor of 4( G ′ ) 2 , and thus vol( S ) = (4( G ′ ) 2 ) d ( d +3) / 2 · vol( S i ) = (2 G ′ ) d ( d +3) · vol( S i ) . Because every S i is disjoint and contained in S , the number of such indices i is at most (2 G ′ ) d ( d +3) .

Next, consider any ( µ, Σ) ∈ U and ( µ ′ , Σ ′ ) ∈ B with ∥ µ -µ ′ ∥ 2 ≤ 1 G ′ , ∥ Σ -Σ ′ ∥ op ≤ 1 G ′ . Then, ∥ Σ -1 / 2 Σ ′ Σ -1 / 2 -I ∥ op = ∥ Σ -1 / 2 (Σ ′ -Σ)Σ -1 / 2 ∥ op ≤ ∥ Σ -1 ∥ op · ∥ Σ ′ -Σ ∥ op ≤ G G ′ . Also, ∥ Σ -1 / 2 ( µ -µ ′ ) ∥ 2 ≤ ∥ Σ -1 / 2 ∥ op · ∥ µ -µ ′ ∥ 2 ≤ √ G G ′ . In other words, we have found a net B of size (2 G ′ ) d ( d +3) such that for every ( µ, Σ) ∈ U , there exists ( µ ′ , Σ ′ ) ∈ B with ( µ ′ , Σ ′ ) ≈ G/G ′ ,G/G ′ ( µ, Σ) .

Next, consider general ˆ µ, ˆ Σ . Note that ( µ, Σ) ∈ U ˆ µ, ˆ Σ is equivalent to µ = ˆ Σ 1 / 2 µ 0 + ˆ µ , where ∥ µ 0 ∥ 2 ≤ G , and Σ = ˆ Σ 1 / 2 Σ 0 ˆ Σ 1 / 2 , where 1 G · I ≼ Σ 0 ≼ G · I . So, if we consider the map f : ( µ 0 , Σ 0 ) ↦→ ( ˆ Σ 1 / 2 µ 0 + ˆ µ, ˆ Σ 1 / 2 Σ 0 ˆ Σ 1 / 2 ) , this bijectively maps U to U ˆ µ, ˆ Σ . We can also consider the cover B ˆ µ, ˆ Σ = f ( B ) , where B is the cover constructed for U . Then, by Lemma C.7, ( µ ′ 0 , Σ ′ 0 ) ≈ G/G ′ ,G/G ′ ( µ 0 , Σ 0 ) if and only if f ( µ ′ 0 , Σ ′ 0 ) ≈ G/G ′ ,G/G ′ f ( µ 0 , Σ 0 ) .

Hence, regardless of the choice of ˆ µ, ˆ Σ , for every ( µ, Σ) ∈ U ˆ µ, ˆ Σ , there exists ( µ ′ , Σ ′ ) ∈ B ˆ µ, ˆ Σ with ( µ ′ , Σ ′ ) ≈ G/G ′ ,G/G ′ ( µ, Σ) . Moreover, |B ˆ µ, ˆ Σ | = |B| ≤ (2 G ′ ) d ( d +3) . Moreover, if ( µ ′ , Σ ′ ) ≈ G/G ′ ,G/G ′ ( µ, Σ) , then ∥ Σ -1 / 2 ( µ ′ -µ ) ∥ 2 ≤ G G ′ and ∥ Σ -1 / 2 ΣΣ -1 / 2 -I ∥ F ≤ √ d · ∥ Σ -1 / 2 ΣΣ -1 / 2 -I ∥ op ≤ √ d · G G ′ . So, by Lemma A.8, d TV ( N ( µ, Σ) , N ( µ ′ , Σ ′ )) ≤ √ d · G G ′ . Hence, if we set G ′ = G √ d ζ , then the size of the net B ˆ µ, ˆ Σ is O ( G √ d/ζ ) d ( d +3) and for every ( µ, Σ) ∈ U ˆ µ, ˆ Σ , there exists ( µ ′ , Σ ′ ) ∈ B ˆ µ, ˆ Σ such that d TV ( N ( µ, Σ) , N ( µ ′ , Σ ′ )) ≤ ζ .

Next, we note the following result about differentially private hypothesis selection.

Theorem D.2. [BSKW19] Let H = { H 1 , . . . , H M } be a set of probability distributions over some domain D . There exists an ε -differentially private algorithm (with respect to a dataset X = { X 1 , . . . , X n } ) which has following guarantees.

Let D be an unknown probability distribution over D , and suppose there exists a distribution H ∗ ∈ H such that d TV ( D , H ∗ ) ≤ α . If n ≥ O ( log M α 2 + log M αε ) , and if X 1 , . . . , X n are samples drawn independently from D , then the algorithm will output a distribution ˆ H ∈ H such that d TV ( D , ˆ H ) ≤ 4 α with probability at least 9/10.

Given Lemma D.1 and Theorem D.2, we are in a position to convert any crude approximation into a fine approximation. Namely, we prove the following result.

Lemma D.3. Let D represent an unknown GMM with representation { ( w i , µ i , Σ i ) } k i =1 . Let G ≥ 1 be some fixed parameter. Then there exists an ε -differentially algorithm, on n ≥ O ( d 2 k · log( G · k · k ′ · √ d/α ) α 2 + d 2 k · log( G · k · k ′ · √ d/α ) αε ) samples, with the following property.

Suppose the algorithm is given as input a set { (ˆ µ j , ˆ Σ j ) } k ′ j =1 for some k ′ ≥ 1 , such that for every ( µ i , Σ i ) with weight w i ≥ α k , there exists j ≤ k ′ such that 1 G · ˆ Σ j ≼ Σ i ≼ G · ˆ Σ j and ∥ ˆ Σ -1 / 2 j ( µ i -ˆ µ j ) ∥ 2 ≤ G . Then, if the samples are X 1 , . . . , X n i.i.d. ∼ D , then with probability at least 9 / 10 , the algorithm outputs a mixture of at most k Gaussians H , with d TV ( D , H ) ≤ O ( α ) .

Proof. Let ζ = α k . For each (ˆ µ j , ˆ Σ j ) , let B ˆ µ j , ˆ Σ j be as in Lemma D.1. We define ˆ B = ⋃ j ≤ k ′ B ˆ µ j , ˆ Σ j .

Let H be the set of hypotheses consisting of mixtures of up to k Gaussians N ( µ ′ i , Σ ′ i ) with weights w ′ i , with every ( µ ′ i , Σ ′ i ) ∈ ˆ B and with every w ′ i an integral multiple of ζ . Note that the number of hypothesis M = |H| is at most | ˆ B| O ( k ) · (1 /ζ ) O ( k ) = ( k ′ · G · √ d/ζ ) O ( d 2 · k ) . Since we set ζ = α/k , M ≤ ( G · kk ′ √ d α ) O ( d 2 · k ) .

Next, let us consider the true GMM D , with representation { ( w i , µ i , Σ i ) } k i =1 . Because every ( µ i , Σ i ) with w i ≥ α k satisfies 1 G · ˆ Σ j ≼ Σ i ≼ G · ˆ Σ j and ∥ ˆ Σ -1 / 2 j ( µ i -ˆ µ j ) ∥ 2 ≤ G, by Lemma D.1, there exists ( µ ′ i , Σ ′ i ) ∈ ˆ B such that d TV ( N ( µ i , Σ i ) , ( µ ′ i , Σ ′ i )) ≤ ζ . Moreover, we can round each w i ≥ α k to some w ′ i which is an integral multiple of ζ , such that | w i -w ′ i | ≤ ζ . Finally, for each w i < α k , we can choose an arbitrary Gaussian in ˆ B and round w i to some w ′ i . Then, the total variation distance between D and the GMM with representation { ( w ′ i , µ ′ i , Σ ′ i ) } k i =1 is at most

Hence, if we set ζ = α k , there exists a distribution H ∗ ∈ H with d TV ( D , H ∗ ) ≤ O ( α ) .

Therefore, the algorithm of Theorem D.2, using n ≥ O ( d 2 k · log( G · k · k ′ · √ d/α ) α 2 + d 2 k · log( G · k · k ′ · √ d/α ) αε ) samples, will find H ∈ H such that d TV ( D , H ) ≤ O ( α ) , and is ε -differentially private.

Pseudocode: We give a simple pseudocode for the algorithm of Lemma D.3, in Algorithm 1.

Input: Samples X 1 , . . . , X n , and crude predictions (ˆ µ j , ˆ Σ j ) , where 1 ≤ j ≤ k ′ for some k ′ . Output: An ( ε, 0) -DP prediction H , which is a mixture over at most k Gaussians.

1 Set ζ = α/k .

2 for j = 1 to k ′ do

3 Define the sets B ˆ µ j , ˆ Σ j as in Lemma D.1, for parameters G,ζ .

4 end

5 ˆ B ← ⋃ j ≤ k ′ B ˆ µ j , ˆ Σ j .

6 Let H be the set of mixtures { ( w ′ i , µ ′ i , Σ ′ i ) } of k or fewer components, where every ( µ ′ i , Σ ′ i ) ∈ ˆ B and every w ′ i is an integral multiple of ζ.

7 Run the ε -DP algorithm of Theorem D.2 on X 1 , . . . , X n with respect to H .

E The High Dimensional Setting

In this section, we provide the algorithm and analysis for Theorem 1.4. We first describe and provide pseudocode for the algorithm, and then we prove that it can privately learn mixtures of d -dimensional Gaussians with low sample complexity.

E.1 Algorithm

High-Level Approach: Suppose that the unknown distribution is a GMM with representation { ( w i , µ i , Σ i ) } k i =1 .

Define Θ to be the set of all feasible mean-covariance pairs ( µ, Σ) , i.e., where µ ∈ R d and Σ ∈ R d × d is positive definite. We also start off with a set Ω = Θ , which will roughly characterize the region of 'remaining' mean-covariance pairs.

At a high level, our algorithm proceeds as follows. We will first learn a very crude approximation of each ( µ i , Σ i ) , one at a time. Namely, using roughly ( ε √ k log(1 /δ ) , δ k ) -DP, we will learn some

(ˆ µ 1 , ˆ Σ 1 ) ∈ Ω which is 'vaguely close' to some ( µ i , Σ i ) . We will then remove every ( µ, Σ) that is vaguely close to this (ˆ µ, ˆ Σ) from our Ω , and then attempt to repeat this process. We will repeat it up to k times, in hopes that every ( µ i , Σ i ) is vaguely close to some (ˆ µ j , ˆ Σ j ) . By advanced composition, the full set of { (ˆ µ j , ˆ Σ j ) } is still ( ε, δ ) -DP.

At this point, the remainder of the algorithm is quite simple. Namely, we have some crude estimate for every ( µ i , Σ i ) (it is 'vaguely close' to some (ˆ µ j , ˆ Σ j ) ). Moreover, one can create a fine net of roughly e ˜ O ( d 2 ) ( µ, Σ) -pairs that cover all mean-covariance pairs vaguely close to each (ˆ µ j , ˆ Σ j ) , since the dimension of ( µ, Σ) is O ( d 2 ) . Moreover, the weight vector ( w 1 , . . . , w k ) has a fine net of size roughly e ˜ O ( k ) . As a result, we can reduce the problem to a small set of hypotheses: there are roughly e ˜ O ( d 2 ) choices for each ( µ i , Σ i ) , and e ˜ O ( k ) choices for w , for a total of e ˜ O ( k · d 2 ) choices for the mixture of Gaussians. We can then apply known results on private hypothesis selection [BSKW19], which will suffice.

The main difficulty in the algorithm is in privately learning some (ˆ µ, ˆ Σ) which is 'vaguely close' to some ( µ i , Σ i ) . We accomplish this task by applying the robustness-to-privacy conversion of [HKMN23], along with a carefully constructed score function, which we will describe later in this section.

Algorithm Description: Let c 0 ≤ 0 . 01 and C 0 > 1 be the constants of Lemma B.6, and c 1 ≤ 0 . 01 be the constant of Lemma C.8. Let c 2 ≤ 0 . 1 · min( c 0 , c 1 ) be a sufficiently small constant. We set parameters η ∗ = c 2 · α 8 k , η = η ∗ 10 , ε 0 = ε √ 4 k log(1 /δ ) , δ 0 = δ 2 k , β 0 = e -d . We also set m = √

parameters γ = τ = c 2 and ρ = c 2 · d 1 / 8 . (See Lines 1-4 of Algorithm 2 for a more precise description of some of the parameters.)

Wenow define the main score function. To do so, we first set some auxiliary parameter η ′ = γ ′ = c 2 8 C 0 and ρ ′ = ρ 8 C 0 . We will consider the (deterministic) algorithm A of Lemma B.6, where A is given parameters η ′ , γ ′ , ρ ′ , β 0 , that acts on a dataset Z and outputs either ⊥ or some (ˆ µ, ˆ Σ) . Next, for some domain Ω ⊂ Θ of 'feasible' mean-covariance pairs ( µ, Σ) , we will define a function f Ω , which takes as input a dataset Y of size N and a mean-covariance pair ( µ, Σ) ∈ Θ , and outputs

Finally, given a dataset X of size n and ( ˜ µ, ˜ Σ) ∈ Θ , we define the score function

Note that ( ˜ µ, ˜ Σ) does not need to be in Ω , but it must satisfy ( ˜ µ, ˜ Σ) ≈ γ,τ ( µ, Σ) for some ( µ, Σ) ∈ Ω . Also, note that it is possible that no matter how one changes the data points, the conditions are never met (for instance if ( ˜ µ, ˜ Σ) ̸≈ γ,τ ( µ, Σ) for any ( µ, Σ) ∈ Ω ). This is not a problem: we will simply set the score to be + ∞ if this occurs.

With this definition of score function S Ω , we can let M be an ( ε 0 , δ 0 ) -DP algorithm based on Theorem C.3, with the settings of n, η, η ∗ , ε 0 , δ 0 , β 0 as above, and with D = n ( n +3) 2 . Here, we recall that (˜ µ, ˜ Σ) is viewed as n ( n +3) 2 -dimensional by only considering the upper-triangular part of ˜ Σ . We also assume the domain is Θ and the normalized volume is as in (1).

Given this algorithm M (which implicitly depends on Ω ), the algorithm works as follows. We will actually draw a total of n + n ′ samples (for n as above, and n ′ to be defined later), though we start by just looking at the first n samples X = { X 1 , . . . , X n } . We initially set Ω = Θ , and run the following procedure up to k times, or until the algorithm outputs ⊥ . For the j th iteration, we use M ( X ) to

compute some prediction (ˆ µ j , ˆ Σ j ) (or possibly we output ⊥ ). Assuming we do not output ⊥ , we remove from Ω every ( µ, Σ) such that n -12 · ˆ Σ j ≼ Σ ≼ n 12 · ˆ Σ j and ∥ ˆ Σ -1 / 2 j ( µ -ˆ µ j ) ∥ 2 ≤ n 12 .

At the end, we have computed some { (ˆ µ j , ˆ Σ j ) } k ′ j =1 where 0 ≤ k ′ ≤ k . If k ′ = 0 we simply output ⊥ . Otherwise, we will draw n ′ = ˜ O ( d 2 k α 2 + d 2 k αε ) fresh samples. We run the ε -DP hypothesis selection based algorithm, Algorithm 1 on the fresh samples, using parameters G = n 12 and { (ˆ µ j , ˆ Σ j ) } k ′ j =1 .

Pseudocode: We give pseudocode for the algorithm in Algorithm 2.

̸

E.2 Analysis of Crude approximation

In this section, we analyze Lines 7-17 of Algorithm 2. The main goal is to show that the algorithm is private, and that for samples drawn from a mixture of Gaussians, every component ( µ i , Σ i ) with large enough weight w i is 'vaguely close' to some (ˆ µ j , ˆ Σ j ) computed by the algorithm.

First, we show that when the samples are actually drawn as a Gaussian mixture, then under some reasonable conditions, any ( µ, Σ) close to a true mean-covariance pair has low score.

Proposition E.1. Suppose X = { X 1 , . . . , X n } is drawn from a Gaussian mixture model, with representation { ( w i , µ i , Σ i ) } k i =1 . Then, with 1 -O ( k · β 0 ) probability over X , for any set Ω

of mean-covariance pairs, for all i ∈ [ k ] such that ( µ i , Σ i ) ∈ Ω and w i ≥ α/k , and for all ( ˜ µ, ˜ Σ) ≈ γ,τ ( µ i , Σ i ) , S Ω (( ˜ µ, ˜ Σ) , X ) = 0 .

Proof. Fix any i ∈ [ k ] with w i ≥ α/k . Set µ = µ i and Σ = Σ i . Also, let X ′ = X , and Y be a random subset of size N among the samples in X actually drawn from N ( µ i , Σ i ) . Note that the number of such samples has distribution Bin( n, w i ) , which for w i ≥ α k and n = 2 k α · N , is at least N with e -Ω( N ) ≤ β 0 failure probability.

Then, Y is just N i.i.d. samples from N ( µ i , Σ i ) . So, if we draw a random subset Z of size m of Y , it has the same distribution as m i.i.d. samples from N ( µ i , Σ i ) . We apply Part 2 of Lemma B.6 (where we use parameters η ′ , γ ′ , ρ ′ in the application). Note that m ≥ ˜ O ( d γ ′ 2 + d 2 ρ ′ 2 ) , so with probability at least 1 -O ( β 0 ) over Z , A ( Z ) ≈ 8 C 0 γ ′ , 8 C 0 ρ ′ , 8 C 0 γ ′ ( µ i , Σ i ) . By our setting of γ ′ , ρ ′ , this means that A ( Z ) ≈ γ,ρ,γ ( µ i , Σ i ) .

In other words, for each index i ∈ ( N m ) and corresponding subset Z of Y , if we let W i be the indicator that A ( Z ) ̸≈ γ,ρ,γ ( µ i , Σ i ) , then P ( W i = 1) ≤ O ( β 0 ) . While the values of W i are not necessarily independent, by linearity of expectation we have that E [ ∑ W i ] ≤ ( N m ) · O ( β 0 ) , so the probability that E [ ∑ W i ] ≥ 1 3 · ( N m ) is at most O ( β 0 ) by Markov's inequality. Moreover, because N ≥ m ≥ ˜ O ( d γ ′ 2 + d 2 ρ ′ 2 ) , we can apply A on Y and we again obtain that with probability 1 -O ( β 0 ) , A ( Y ) ≈ γ,ρ,γ ( µ i , Σ i ) .

In summary, for any fixed i ∈ [ k ] with w i ≥ α k , with probability at least 1 -O ( β 0 ) over X , there exists Y ⊂ X of size N , such that A ( Y ) ≈ γ,ρ,γ ( µ i , Σ i ) and at least 2 / 3 of the subsets Z ⊂ Y of size m have A ( Z ) ≈ γ,ρ,γ ( µ i , Σ i ) . Thus, for any set Ω , if ( µ i , Σ i ) ∈ Ω then S Ω (( ˜ µ, ˜ Σ) , X ) = 0 for all ( ˜ µ, ˜ Σ) ≈ γ,τ ( µ i , Σ i ) . The proof follows by a union bound over all i ∈ [ k ] .

Next, we show that for any dataset X , if there is even a single ( µ ∗ , Σ ∗ ) with low score, there must be a region of ( ˜ µ, ˜ Σ) which all has low score.

Proposition E.2. Suppose that t = min µ ∗ , Σ ∗ S Ω (( µ ∗ , Σ ∗ ) , X ) . Then, there exists some µ, Σ such that for all ( ˜ µ, ˜ Σ) ≈ γ,τ ( µ, Σ) , we have that S Ω (( ˜ µ, ˜ Σ) , X ) = t.

Proof. Fix µ ∗ , Σ ∗ so that S Ω (( µ ∗ , Σ ∗ ) , X ) = t . Then, there is some ( µ, Σ) and some X ′ , Y such that d H ( X , X ′ ) = t , Y ⊂ X ′ , | Y | = N , and f Ω ( Y , µ, Σ) = 1 . Thus, for any ( ˜ µ, ˜ Σ) ≈ γ,τ ( µ, Σ) , by definition, we have that S Ω (( ˜ µ, ˜ Σ) , X ) ≤ t . But since t is the minimum possible score, we in fact have equality: S Ω (( ˜ µ, ˜ Σ) , X ) = t for all such ( ˜ µ, ˜ Σ) .

Next, we want to show that regardless of the dataset (even if not drawn from any Gaussian mixture distribution), the set of points of low score can't have that much volume.

Proposition E.3. Fix a dataset X of size n . Then, the set of ˜ µ, ˜ Σ such that S Ω (( ˜ µ, ˜ Σ) , X ) ≤ η ∗ · n can be partitioned into at most ( n m ) regions S i , which is indexed by some ( µ ′ i , Σ ′ i ) . Moreover, for all ( ˜ µ, ˜ Σ) ∈ S i , there exists ( µ, Σ) such that ( ˜ µ, ˜ Σ) ≈ γ,τ ( µ, Σ) and ( µ, Σ) ≈ 8 γ, 8 ρ, 8 τ ( µ ′ i , Σ ′ i ) .

Proof. Pick any ( ˜ µ, ˜ Σ) with score at most η ∗ · n . Let X ′ , Y ′ , µ, Σ be such that d H ( X , X ′ ) ≤ η ∗ · n , Y ′ ⊂ X ′ , | Y ′ | = N , ( ˜ µ, ˜ Σ) ≈ γ,τ ( µ, Σ) , and f Ω ( Y ′ , µ, Σ) = 1 . If we define Y ⊂ X of size N to be the corresponding subset as Y ′ is to X ′ , then d H ( Y , Y ′ ) ≤ η ∗ · n = c 2 4 · N .

Now, if we take a random subset Z ′ of size m in Y ′ , and look at the corresponding subset Z of size m in Y , by a Chernoff bound, with at least 0 . 99 probability d H ( Z , Z ′ ) ≤ c 2 2 · m . Moreover, with at least 2 / 3 probability over the random subset Z ′ , A ( Z ′ ) ≈ γ,ρ,τ ( µ, Σ) . Therefore, there always exists a subset Z ⊂ X of size m and a set Z ′ of size m such that d H ( Z , Z ′ ) ≤ c 2 2 · m and A ( Z ′ ) ≈ γ,ρ,τ ( µ, Σ) .

̸

Now, for any fixed Z ⊂ X of size m , if we look at any sets Z ′ , Z ′′ of size m and Hamming distance at most c 2 2 · m from Z , then d H ( Z ′ , Z ′′ ) ≤ c 2 · m. So, by Property 1 of Lemma B.6, for every such Z ′ and Z ′′ with A ( Z ′ ) , A ( Z ′′ ) = ⊥ , we must have A ( Z ′′ ) ≈ γ,ρ,τ A ( Z ′ ) .

̸

̸

To complete the proof, we order the subsets Z 1 , Z 2 , . . . , Z ( n m ) of size m in X . For each i ≤ ( n m ) , if there exists some Z ′ such that d H ( Z i , Z ′ ) ≤ c 2 2 · m and A ( Z ′ ) = ⊥ , choose an arbitrary such Z ′ and let ( µ ′ i , Σ ′ i ) := A ( Z ′ ) . Otherwise, we do not define ( µ ′ i , Σ ′ i ) . Then, for any ( ˜ µ, ˜ Σ) of score at most η ∗ · n , there exists ( µ, Σ) , a subset Z i ⊂ X of size m , and a set Z ′ i of size m such that ( ˜ µ, ˜ Σ) ≈ γ,τ ( µ, Σ) , d H ( Z i , Z ′ i ) ≤ c 2 2 · m , and A ( Z ′ i ) ≈ γ,ρ,τ ( µ, Σ) . Because A ( Z ′ i ) = ⊥ and Z ′ i has Hamming distance at most c 2 2 · m from Z i , this also implies that A ( Z ′ i ) ≈ γ,ρ,τ ( µ ′ i , Σ ′ i ) . By Proposition B.2, this means that ( µ, Σ) ≈ 8 γ, 8 ρ, 8 τ ( µ ′ i , Σ ′ i ) , which completes the proof.

Proposition E.4. Suppose X = { X 1 , . . . , X n } is drawn from a Gaussian mixture model, with representation { ( w i , µ i , Σ i ) } k i =1 . Then, with probability at least 1 -O ( β 0 ) over the randomness of X , for any set Ω of mean-covariance pairs, and for every ( ˜ µ, ˜ Σ) with S Ω (( ˜ µ, ˜ Σ) , X ) ≤ N 2 , there exists an index i ∈ [ k ] such that 1 O ( n 8 ) · Σ i ≼ ˜ Σ ≼ O ( n 8 ) · Σ i and ∥ Σ -1 / 2 i ( ˜ µ -µ i ) ∥ 2 ≤ O ( n 6 ) .

Proof. Weapply Part 3 of Lemma B.6, though we use N to replace the value m in Lemma B.6. We are assuming N ≥ 40 d · k . So, by definition of score, ( ˜ µ, ˜ Σ) ≈ γ,τ ( µ, Σ) where f Ω ( Y ′ , µ, Σ) = 1 , for some Y ′ which can be generated by taking a subset of X of size at most N and altering at most N/ 2 elements. Since f Ω ( Y ′ , µ, Σ) = 1 , this means that if (ˆ µ, ˆ Σ) = A ( Y ′ ) then (ˆ µ, ˆ Σ) ≈ γ,ρ,τ ( µ, Σ) . So, by Proposition B.2, ( ˜ µ, ˜ Σ) ≈ 8 γ, 8 τ (ˆ µ, ˆ Σ) . Assuming γ = τ = c 2 is sufficiently small, this means that ( ˜ µ, ˜ Σ) ≈ 1 / 2 , 1 / 2 (ˆ µ, ˆ Σ) .

By Part 3 of Lemma B.6, with probability at least 1 -O ( β 0 ) , there exists i ∈ [ k ] such that 1 O ( n 8 ) · Σ i ≼ ˆ Σ ≼ O ( n 8 ) · Σ i and ∥ Σ -1 / 2 i (ˆ µ -µ i ) ∥ 2 ≤ O ( n 6 ) . However, we know that 1 2 ˆ Σ ≼ ˜ Σ ≼ 2 ˆ Σ , which means that 1 O ( n 8 ) · Σ i ≼ ˜ Σ ≼ O ( n 8 ) · Σ i . Moreover, ∥ ˆ Σ -1 / 2 (ˆ µ -˜ µ ) ∥ 2 ≤ 0 . 5 , so ∥ Σ -1 / 2 i (ˆ µ -˜ µ ) ∥ 2 ≤ O ( n 4 ) . Thus, by triangle inequality, we have that ∥ Σ -1 / 2 i ( ˜ µ -µ i ) ∥ 2 ≤ O ( n 6 ) .

Now, given a set X = { X 1 , . . . , X n } drawn from a GMM with representation { ( w i , µ i , Σ i ) } k i =1 , we say X is regular if it satisfies Propositions E.1 and E.4. In other words:

for any set Ω of mean-covariance pairs, for all i ∈ [ k ] such that ( µ i , Σ i ) ∈ Ω and w i ≥ α/k , and for all ( ˜ µ, ˜ Σ) ≈ γ,τ ( µ i , Σ i ) , S Ω (( ˜ µ, ˜ Σ) , X ) = 0 .

for any set Ω of mean-covariance pairs, and for every ( ˜ µ, ˜ Σ) with S Ω (( ˜ µ, ˜ Σ) , X ) ≤ N 2 , there exists an index i ∈ [ k ] such that 1 O ( n 8 ) · Σ i ≼ ˜ Σ ≼ O ( n 8 ) · Σ i and ∥ Σ -1 / 2 i ( ˜ µ -µ i ) ∥ 2 ≤ O ( n 6 ) .

When we say X is regular, the components ( µ i , Σ i ) and weights w i are implicit.

We now show that every step of the crude approximation (Lines 8-16 in Algorithm 2) will, with 1 -O ( β 0 ) probability, find some crude approximation (ˆ µ j , ˆ Σ j ) to some ( µ i , Σ i ) , as long as ( µ i , Σ i ) ∈ Ω .

Lemma E.5. For any Ω ⊂ Θ , the corresponding algorithm M (which depends on Ω ) is ( ε 0 , δ 0 ) -DP on datasets X .

Suppose additionally that X is regular, and that there exists i ∈ [ k ] such that w i ≥ α k and ( µ i , Σ i ) ∈ Ω . Then, with 1 -O ( β 0 ) probability (just over the mechanism M ), M ( X ) = ( ˜ µ, ˜ Σ) and

There exists i ∈ [ k ] such that 1 O ( n 8 ) · Σ i ≼ ˜ Σ ≼ O ( n 8 ) · Σ i and ∥ Σ -1 / 2 i ( ˜ µ -µ i ) ∥ 2 ≤ O ( n 6 ) .

There exists ( µ, Σ) ∈ Ω such that ( ˜ µ, ˜ Σ) ≈ γ,τ ( µ, Σ) .

Proof. We will apply Theorem C.3. We first need to check the necessary conditions. It follows almost immediately from the definition of S Ω that S Ω (( ˜ µ, ˜ Σ) , · ) has the bounded sensitivity property with respect to neighboring datasets, for any fixed Ω , ˜ µ, ˜ Σ .

To check the volume condition, note that for any dataset X of size n , if min µ, Σ S Ω (( µ, Σ) , X ) ≤ 0 . 7 η ∗ n , then by Proposition E.2, there exists µ, Σ such that S Ω (( ˜ µ, ˜ Σ) , X ) ≤ 0 . 8 η ∗ n for all ( ˜ µ, ˜ Σ) ≈ γ,τ ( µ, Σ) . Conversely, for any X , by Proposition E.3, the set of ( ˜ µ, ˜ Σ) of score at most η ∗ n can be partitioned into ( n m ) regions S i , indexed by ( µ ′ i , Σ ′ i ) . Moreover, each S i is contained in the set of ( ˜ µ, ˜ Σ) such that there exists ( µ, Σ) such that ( ˜ µ, ˜ Σ) ≈ γ,τ ( µ, Σ) ≈ 8 γ, 8 ρ, 8 τ ( µ ′ i , Σ ′ i ) .

Let us use the notation of Lemma C.8, where γ 1 = γ , γ 2 = 8 γ , and ρ 2 = 8 ρ . Then, we have that the set of points with score at most 0 . 8 η ∗ n contains T 1 ( µ, Σ) for some ( µ, Σ) , but the set of points with score at most η ∗ n is contained in the union of T 2 ( µ ′ i , Σ ′ i ) for at most ( n m ) choices of ( µ ′ i , Σ ′ i ) . So, as long as min µ, Σ S Ω (( µ, Σ) , X ) ≤ 0 . 7 η ∗ n , we can apply Lemma C.8 to obtain

since ρ 2 = 8 ρ = 8 c 2 · d 1 / 8 and γ 1 = γ = c 2 , and since c 2 is a constant. Thus, if

we have that M is ( ε 0 , δ 0 ) -DP, by Theorem C.3. This holds by our parameter settings, assuming K is sufficiently large.

Next, we prove accuracy. Assume that X is regular. We again adopt the notation of Lemma C.8, (with γ 1 = γ , γ 2 = 8 γ , and ρ 2 = 8 ρ ). By the first condition of regularity, for all i ∈ [ k ] such that ( µ i , Σ i ) ∈ Ω and w i ≥ α/k , every ( ˜ µ, ˜ Σ) ∈ T 1 ( µ i , Σ i ) satisfies S Ω (( ˜ µ, ˜ Σ) , X ) = 0 . We still have that the set of points of score at most η ∗ n is contained in the union of at most ( n m ) sets T 2 ( µ ′ i , Σ ′ i ) . Thus,

where we recall that we set η = η ∗ 10 . So, by Theorem C.3, as long as

which indeed holds, M ( X ) outputs some ( ˜ µ, ˜ Σ) of score at most 2 ηn , with 1 -β 0 probability. By the second condition of regularity' there exists an index i ∈ [ k ] such that 1 O ( n 8 ) · Σ i ≼ ˜ Σ ≼ O ( n 8 ) · Σ i and ∥ Σ -1 / 2 i ( ˜ µ -µ i ) ∥ 2 ≤ O ( n 6 ) . Moreover, because ( ˜ µ, ˜ Σ) has finite score, ( ˜ µ, ˜ Σ) ≈ γ,τ ( µ, Σ) for some ( µ, Σ) ∈ Ω .

Lemma E.6. Lines 7-17 of Algorithm 2 are ( ε, δ ) -DP. Moreover, for every regular set X , with probability at least 1 -O ( k · β 0 ) over the randomness of Algorithm 2, for every i ∈ [ k ] such that w i ≥ α k , there exists j such that n -12 · ˆ Σ j ≼ Σ i ≼ n 12 · ˆ Σ j and ∥ ˆ Σ -1 / 2 j ( µ -ˆ µ j ) ∥ 2 ≤ n 12 .

Proof. The privacy guarantee is immediate by (adaptive) advanced composition. Indeed, the algorithm M at each step is ( ε 0 , δ 0 ) -DP, and only depends on X and Ω , which is determined only by the output of all previous runs of M .

Now, let's say that X = { X 1 , . . . , X n } is regular, and we run Algorithm 2 on X . Now, after j ≥ 0 steps of the loop, define P j to be the set of indices i ∈ [ k ] such that w i ≥ α k and ( µ i , Σ i ) ∈ Ω j (where Ω j refers to the set Ω after j steps of the loop are completed). Likewise, define Q j to be the set of indices i such that there exists (ˆ µ, ˆ Σ) ∈ Ω j with 1 n 10 · Σ i ≼ ˆ Σ ≼ n 10 · Σ i and ∥ Σ -1 / 2 i (ˆ µ -µ i ) ∥ 2 ≤ n 10 . Note that P 0 ⊂ Q 0 = [ k ] , and that P j ⊂ Q j always. Moreover, because Ω only shrinks, P j +1 ⊂ P j and Q j +1 ⊂ Q j always.

̸

Now, after some step j < k , we claim that if P j = ∅ , then | Q j +1 | ≤ | Q j | -1 with failure probability at most O ( β 0 ) , i.e., Q decreases in size for the next step if P isn't currently empty. The failure probability is only over the randomness of M at each step, and holds for any regular dataset X . A union bound says the total failure probability is O ( k · β 0 ) .

To see why this holds, note that if some index i ∈ P j , then ( µ i , Σ i ) ∈ Ω j . So, we can apply Lemma E.5. At step j +1 , we find some (ˆ µ j +1 , ˆ Σ j +1 ) , with the following properties. First, there exists an index i ′ ∈ [ k ] with 1 O ( n 8 ) · Σ i ′ ≼ ˆ Σ j +1 ≼ O ( n 8 ) · Σ i ′ and ∥ Σ -1 / 2 i ′ (ˆ µ j +1 -µ i ′ ) ∥ 2 ≤ O ( n 6 ) . Second, (ˆ µ j +1 , ˆ Σ j +1 ) ≈ γ,τ ( µ, Σ) , for some ( µ, Σ) ∈ Ω j .

We claim this means i ′ ∈ Q j . To see why, it suffices to show that 1 n 10 · Σ i ′ ≼ Σ ≼ n 10 · Σ i ′ and ∥ Σ -1 / 2 i ′ ( µ -µ i ′ ) ∥ 2 ≤ n 10 , by definition of Q j and because ( µ, Σ) ∈ Ω j . However, we know that 1 O ( n 8 ) · Σ i ′ ≼ ˆ Σ j +1 ≼ O ( n 8 ) · Σ i ′ and 1 2 ˆ Σ j +1 ≼ Σ ≼ 2 ˆ Σ j +1 , which implies 1 n 10 · Σ i ′ ≼ Σ ≼ n 10 · Σ i ′ . Also, we know that ∥ Σ -1 / 2 i ′ (ˆ µ j +1 -µ i ′ ) ∥ 2 ≤ O ( n 6 ) and ∥ Σ -1 / 2 ( µ -ˆ µ j +1 ) ∥ 2 ≤ τ ≤ 1 The latter inequality along with the fact that Σ ≼ n 10 · Σ i ′ implies that ∥ Σ -1 / 2 i ′ ( µ -ˆ µ j +1 ) ∥ 2 ≤ n 5 . So, by triangle inequality, ∥ Σ -1 / 2 ′ ( µ -µ i ′ ) ∥ 2 ≤ O ( n 6 ) ≤ n 10 .

i

Next, we claim that i ′ ̸∈ P j +1 , so i ′ ̸∈ Q j +1 . Indeed, we have that 1 O ( n 8 ) · Σ i ′ ≼ ˆ Σ j +1 ≼ O ( n 8 ) · Σ i ′ , which immediately means 1 n 10 · ˆ Σ j +1 ≼ 1 O ( n 8 ) · ˆ Σ j +1 ≼ Σ i ′ ≼ O ( n 8 ) · ˆ Σ j +1 ≼ n 10 · ˆ Σ j +1 . Also, ∥ Σ -1 / 2 i ′ (ˆ µ j +1 -µ i ′ ) ∥ 2 ≤ O ( n 6 ) , and since Σ i ′ ≼ n 10 · ˆ Σ j +1 , this means ∥ ˆ Σ -1 / 2 j +1 ( µ i ′ -ˆ µ j +1 ) ∥ 2 ≤ O ( n 11 ) ≤ n 12 . Thus, the algorithm will remove ( µ i ′ , Σ i ′ ) from Ω at step j +1 , so i ′ ̸∈ Q j +1 .

Thus, either P j is empty, or Q j decreases in size to Q j +1 for each 0 ≤ j ≤ k +1 . This implies that P k is empty, i.e., for every i ∈ [ k ] such that w i ≥ α k , ( µ i , Σ i ) ̸∈ Ω j . This can only happen if each such ( µ i , Σ i ) was removed at some point. So, for every such i , there was some index j such that n -12 · ˆ Σ j ≼ Σ i ≼ n 12 · ˆ Σ j and ∥ ˆ Σ -1 / 2 j ( µ -ˆ µ j ) ∥ 2 ≤ n 12 .

E.3 Summary and Completion of Analysis

We first quickly summarize how to put everything together to prove Theorem 1.4. To prove our main result, need to ensure that our algorithm is private, uses few enough samples, and accurately learns the mixture. Privacy will be simple, as we have shown in Lemma E.6 that the crude approximation is private, and the hypothesis selection is private as shown in Appendix D. The sample complexity will come from the settings of n and n ′ in lines 3 and 18 of the algorithm, and from our setting of parameters. Finally, we showed in Lemma E.6 that we have found a set of (ˆ µ j , ˆ Σ j ) , of at most k meancovariance pairs, such that every true ( µ i , Σ i ) of sufficiently large weight is crudely approximated by some (ˆ µ j , ˆ Σ j ) . Indeed, this is exactly the situation for which we can apply the result on private hypothesis selection (Lemma D.3, based on [BSKW19]).

We are now ready to complete the analysis and prove Theorem 1.4.

Proof of Theorem 1.4. By Lemma E.6, note that the algorithm up to Line 17 is ( ε, δ ) -DP with respect to X 1 , . . . , X n , and does not depend on X n +1 , . . . , X n + n ′ . Assuming { (ˆ µ j , ˆ Σ j ) } from these lines is fixed, Lines 18-20 are ( ε, δ ) -DP with respect to X n +1 , . . . , X n + n ′ , by Lemma D.3, and do not depend on X 1 , . . . , X n . So, the overall algorithm is ( ε, δ ) -DP.

Next, we verify accuracy. Note that by Lemma E.6, and for G = n 12 , the sets (ˆ µ j , ˆ Σ j ) that we find satisfy the conditions for Lemma D.3, with failure probability at most O ( k · β 0 ) . This probability is at most 0 . 1 , assuming e d is significantly larger than k . So, it suffices for n ′ , the number of samples used in Line 18 of Algorithm 2, to satisfy n ′ ≥ O ( d 2 k · log( G · k √ d/α ) α 2 + d 2 k · log( G · k √ d/α ) αε ) =

( 2 2 )

˜ O d k α 2 + d k αε , where the last part of the bound holds by our assumptions on G and n . Thus, the total sample complexity is

By Lemma D.3, with failure probability at most 0 . 1 , Lines 18-20 of the algorithm will successfully output a hypothesis which is a mixture of k Gaussians, with total variation distance at most O ( α ) from the right answer. So, the overall success probability is at least 0 . 8 .

Finally, we note that the assumption that e d is much larger than k can be made WLOG, by padding d to be a sufficiently large multiple of log k . Namely, if given samples X i = ( X i, 1 , . . . , X i,d ) , we can add coordinates X i,d +1 , . . . , X O (log k ) ∼ N (0 , 1) . Then, if the original samples were a mixture of k

Gaussians N ( µ i , Σ i ) , the new distribution is a mixture of k Gaussians N (( µ i 0 ) , ( Σ i 0 0 I )) , with the same mixing weights. So, we can learn the mixture distribution up to total variation distance α in the larger space, and then remove all except the first d coordinates, to output our final answer. This will not affect the sample complexity by more than a poly(log k ) factor.

F The Univariate Setting

In this section, we provide the algorithm and analysis for Theorem 1.5. We first describe and provide pseudocode for the algorithm, and then we prove that it can privately learn mixtures of Gaussians with low sample complexity.

We will use σ i := √ Σ i to denote the standard deviation of a univariate Gaussian N ( µ i , Σ i ) .

F.1 Algorithm

As in the algorithm in Appendix E, we will actually draw a total of n + n ′ samples. We start off by only considering the first n data points X = { X 1 , . . . , X n } , where each X j ∈ R . Now, let Y = { Y 1 , . . . , Y n } be the sorted version of X , i.e., Y 1 ≤ Y 2 ≤ · · · ≤ Y n , and ( X 1 , . . . , X n ) and ( Y 1 , . . . , Y n ) are the same up to permutation. Finally, for each j ≤ n -1 , define Z j to be the ordered pair ( Y j , Y j +1 -Y j ) , and define Z = Z ( X ) to be the unordered multiset of Z j 's. (Note that Z depends deterministically on X .)

The algorithm now works as follows. Suppose we have data X = { X 1 , . . . , X n } . After sorting to obtain Y 1 , . . . , Y n , let r j = Y j , s j = Y j +1 -Y j , and Z j = ( r j , s j ) . So, Z ( X ) = { ( r j , s j ) } 1 ≤ j ≤ n -1 , viewed as an unordered set. Note that every s j ≥ 0 , and the r j 's are in nondecreasing order. We will create a set of buckets that can be bijected onto Z 2 . For each Z j = ( r j , s j ) , if s j > 0 we assign Z j to the bucket labeled ( a, b ) ∈ Z 2 if 2 a ≤ s j < 2 a +1 and b · n 5 · 2 a ≤ r j < ( b +1) · n 5 · 2 a . If s j = 0 , we do not assign Z j to any bucket.

For each element e ∈ Z 2 , we keep track of the number of indices j ∈ [ n -1] with Z j sent to e . In other words, for e = ( a, b ) , we define the count c e = # { j : 2 a ≤ s j < 2 a +1 , b · n 5 · 2 a ≤ r j < ( b +1) · n 5 · 2 a } . For each c e , we sample an independent draw g e ∼ TLap(1 , ε/ 10 , δ/ 10) , and define ˜ c e = c e + g e . Finally, we let S = { (ˆ µ i , ˆ Σ i ) } be the the set of pairs ( b · n 5 · 2 a , 2 2 a ) where e = ( a, b ) satisfies ˜ c e > 100 ε log 1 δ .

Hence, we have computed some { (ˆ µ i , ˆ Σ i ) } k ′ i =1 , for some k ′ ≥ 0 . (We will show in the analysis that k ′ ≤ n -1 always, so k ′ is finite.) If k ′ = 0 we simply output ⊥ . Otherwise, we will draw ˜ O ( k α 2 + k αε ) fresh samples. We run the ε -DP hypothesis selection based algorithm, Algorithm 1 on the fresh samples, using parameters G = n 10 and { (ˆ µ i , ˆ Σ i ) } k ′ i =1 .

Pseudocode. We give pseudocode for the algorithm in Algorithm 3.

F.2 Analysis

We start by analyzing Lines 1-19, which generates the set S = { (ˆ µ i , ˆ Σ i ) } of candidate meancovariance pairs.

First, we note a very basic proposition.

Proposition F.1. Let ε, δ ≤ 1 . Then, with probability 1 , TLap(1 , ε/ 10 , δ/ 10) is at most 100 ε · log 1 δ in absolute value.

Proof. By definition of Truncated Laplace, | TLap(1 , ε/ 10 , δ/ 10) | ≤ 1 ( ε/ 10) · log ( 1 + e ε/ 10 -1 2 δ/ 10 ) with probability 1 . For ε, δ ≤ 1 , this is less than 100 ε · log 1 δ .

Next, we show that S is not too large.

Lemma F.2. The number of e = ( a, b ) ∈ Z 2 such that ˜ c e > 100 ε · log 1 δ is at most n -1 . Thus, | S | ≤ n -1 .

Proof. Note that every Z j only increases the count of a single c e , so it suffices to prove that ˜ c e > 100 ε · log 1 δ only if c e ≥ 1 .

To prove this, suppose that c e = 0 . Then, ˜ c e ∼ TLap(1 , ε/ 10 , δ/ 10) ≤ 100 ε · log 1 δ , by Proposition F.1. Thus, ˜ c e > 100 ε · log 1 δ can only happen if c e > 0 , meaning c e ≥ 1 .

Next, we show that for every Gaussian component ( µ i , σ i ) of sufficiently large weight w i , there are several pairs Z j = ( Y j , Y j +1 -Y j ) that crudely approximate ( µ i , σ i ) .

Lemma F.3. Let n ≥ K · k log(1 /δ ) αε , where K = poly log( k, 1 α , 1 ε , log 1 δ ) is sufficiently large. Let X be n i.i.d. samples from a GMM with representation { ( w i , µ i , Σ i ) } k i =1 . Then, with failure probability at least 0 . 99 , for every i ∈ [ k ] with w i ≥ α k , there are at least α 8 k · n indices j such that Y j ∈ [ µ i -σ i , µ i + σ i ] and σ i 10 4 · n 4 ≤ Y j +1 -Y j ≤ 2 σ i .

Proof. Fix some i ∈ [ k ] such that w i ≥ α k . By a Chernoff bound, since w i · n ≥ α k · n is a sufficiently large multiple of log k , with failure probability at most 1 1000 k , the number of data points X j from component i is at least α 2 k · n . Let us condition on T i ⊂ [ n ] , the set of indices coming from the i th mixture component, and condition on | T i | ≥ α 2 k · n . Then, by a Chernoff bound, with failure

̸

probability at most 1 1000 k , at least α 4 k · n points X r : r ∈ T i are in the interval [ µ i -σ i , µ i + σ i ] . Next, note that for any X r , X r ′ ∼ N ( µ i , σ 2 i ) , X r -X r ′ ∼ N (0 , 2 σ 2 i ) , and thus has magnitude at least σ i 10 4 n 3 with at most 1 1000 n 3 failure probability. So, by a union bound over all r = r ′ ∈ T i , with at most 1 1000 n ≤ 1 1000 k failure probability, all data points X r , X r ′ for r = r ′ ∈ T i are separated by at least σ i 10 4 n 3 .

So, with at least 3 1000 k failure probability, there is a subset U i ⊂ T i of size at least α 4 k · n, such that for every r, r ′ ∈ U i , | X r -X r ′ | , and for every r ∈ U i , X r ∈ [ µ i -σ i , µ i + σ i ] . By a union bound, this holds simultaneously for all i ∈ [ k ] with w j ≥ α k , with probability at least 0 . 997 .

Conditioned on this event, if we consider the elements X in sorted order (i.e., Y ), between every consecutive pair X r , X r ′ : r, r ′ ∈ U i (after sorting), we know X r ′ -X r ≥ σ i 10 4 · n 3 . So, there exist some X r ≤ Y j , Y j +1 ≤ X r ′ such that Y j +1 -Y j ≥ σ i 10 4 · n 4 . Hence, for every i ∈ [ k ] with w i ≥ α k , there exist at least | U i | -1 ≥ α 4 k -1 ≥ α 8 k indices j such that Y j +1 -Y j ≥ σ i 10 4 · n 4 . Moreover, note that Y j , Y j +1 ∈ [ µ i -σ i , µ i + σ i ] , so Y j +1 -Y j ≤ 2 σ i . Hence, conditioned on the event from the previous paragraph, the lemma holds.

Given the above lemma, we can prove that for every ( µ i , Σ i ) in the Gaussian mixture with at least α k weight, there is some corresponding crude approximation (ˆ µ, ˆ Σ) ∈ S .

Lemma F.4. Assume that the conditions and result of Lemma F .3 holds. Then, there exists (ˆ µ, ˆ Σ) ∈ S such that n -10 · ˆ Σ ≼ Σ i ≼ n 10 · ˆ Σ and ∥ ˆ Σ -1 / 2 ( µ i -ˆ µ ) ∥ 2 ≤ n 6 .

Proof. Fix any i with w i ≥ α k . Then, for any j with Y j , Y j +1 ∈ [ µ i -σ i , µ i + σ i ] and σ i 10 4 · n 4 ≤ Y j +1 -Y j ≤ 2 σ i , the index i contributes to the count of some bucket e = ( a, b ) , where ⌊ log 2 σ i 10 5 · n 4 ⌋ ≤ a ≤ ⌊ log 2 (2 σ i ) ⌋ . Moreover, Y j ∈ [ b · n 5 · 2 a , ( b + 1) · n 5 · 2 a ) . Therefore, if we consider the set V i of e = ( a, b ) such that ⌊ log 2 σ i 10 5 · n 4 ⌋ ≤ a ≤ ⌊ log 2 (2 σ i ) ⌋ and [ b · n 5 · 2 a , ( b + 1) · n 5 · 2 a ) ∩ [ µ i -σ i , µ i + σ i ] is nonempty, the sum of the counts c e across such e ∈ V i is at least α 8 k · n . But there are at most O (log n ) choices of a , and since n 5 · 2 a > 2 σ , there are at most two choices for b for any fixed a . So, | V i | ≤ O (log n ) , and ∑ e ∈ V i c e ≥ α 8 k · n . Assuming that α 8 k · n ≥ O (log n · 1 ε log 1 δ ) , i.e., n ≥ K · k log(1 /δ ) αε for a sufficiently large polylogarithmic factor K = poly log ( k, 1 α , 1 ε , log 1 δ ) , one of these buckets e = ( a, b ) ∈ V j must have c e > 200 ε log 1 δ . In this case, ˜ c e = c e +TLap(1 , ε/ 10 , δ/ 10) > 100 ε log 1 δ by Proposition F.1, so we include the pair (ˆ µ, ˆ Σ) := ( b · n 5 · 2 a , 2 2 a ) in S .

Hence, there exists ( a, b ) ∈ V i such that (ˆ µ, ˆ Σ) := ( b · n 5 · 2 a , 2 2 a ) in S . By definition of V i , σ i n 5 ≤ 2 a ≤ 2 σ i , so 1 n 10 · Σ i ≤ 2 2 a ≤ 4Σ i ≤ n 10 · Σ i . Moreover, | ˆ Σ -1 / 2 ( µ i -ˆ µ ) | = 2 -a · | µ i -ˆ µ | . But the definition of V i means ˆ µ = b · n 5 · 2 a satisfies ˆ µ ≤ µ i + σ i and ˆ µ ≥ µ i -σ i -n 5 · 2 a ≥ µ i -(2 n 5 +1) · σ i . So, | ˆ µ -µ i | ≤ (2 n 5 +1) · σ i ≤ n 6 · 2 a . Thus, | ˆ Σ -1 / 2 (ˆ µ -µ i ) | ≤ n 6 .

We now prove that the mechanism (until Line 19) is differentially private. First, we note the following auxiliary claim.

̸

Lemma F.5. Let X , X ′ be adjacent datasets of size n (i.e., only differing on a single element). Then, the corresponding sets Z = Z ( X ) and Z ′ = Z ( X ′ ) differ in distance at most 3 , where by distance we mean that there exists a permutation of the elements in Z and Z ′ , respectively, such that at most three indices i ≤ n -1 satisfy Z i = Z ′ i .

Proof. Note that for the sorted versions Y , Y ′ of X , X ′ , respectively, we can convert from Y to Y ′ by removing one data point and adding one more data point, without affecting the order of any other data points.

Suppose we remove some Y j from Y . If j = 1 , then this just removes Z 1 , and if j = n , then this just removes Z n -1 . If j ≥ 2 , this modifies Z j -1 and removes Z j . Likewise, if we add some new Y j ′ , this will either add one new ordered pair to Z (if Y j ′ is either the smallest or largest element), or replace one ordered pair in Z with two new pairs. Therefore, if we remove a Y j and then add a Y j ′ , this will change at most 3 of the ordered pairs in Z .

We now prove privacy.

̸

Lemma F.6. The set S = { (ˆ µ j , ˆ Σ j ) } of candidate mean-covariance pairs is ( ε, δ ) -DP with respect to X = { X 1 , . . . , X n } .

Proof. Let X , X ′ be adjacent datasets of size n . Then, the corresponding sets Z , Z ′ (after a possible permutation) differ in 3 elements. Therefore, if we let { ˜ c e } e ∈ Z 2 be the counts for Z and { ˜ c ′ e } e ∈ Z 2 be the counts for Z ′ , we have that ∥ ˜ c e -˜ c ′ e ∥ 1 ≤ 6 . Because changing a single count c e by 1 leads to ( ε/ 10 , δ/ 10) -DP, overall, the counts { ˜ c e } will satisfy ( ε, δ ) -DP. Finally, S is a deterministic function of the noisy counts ˜ c e , and therefore must also be ( ε, δ ) -DP.

Finally, we can incorporate the fine approximation (i.e., Lines 20-22 of Algorithm 3) and prove Theorem 1.5.

Proof of Theorem 1.5. By Lemma F.6, the algorithm up to Line 19 is ( ε, δ ) -DP with respect to X 1 , . . . , X n , and does not depend on X n +1 , . . . , X n + n ′ . Assuming S = { (ˆ µ i , ˆ Σ i ) } from these lines is fixed, Lines 20-22 are ( ε, δ ) -DP with respect to X n +1 , . . . , X n + n ′ , by Lemma D.3, and do not depend on X 1 , . . . , X n . So, the overall algorithm is ( ε, δ ) -DP.

Next, we verify accuracy. Note that by Lemma F.4, and for G = n 10 , the sets (ˆ µ i , ˆ Σ i ) that we find satisfy the conditions for Lemma D.3, with failure probability at most 0 . 01 . Moreover, the size of S = { (ˆ µ i , ˆ Σ i ) } is at most n , by Lemma F.2. So, because d = 1 , it suffices for n ′ , the number of samples used in Line 20 of Algorithm 3, to satisfy n ′ ≥ O ( k · log( G · kn/α ) α 2 + k · log( G · kn/α ) αε ) =

˜ O ( k α 2 + k αε ) , where the last part of the bound holds by our assumptions on G and n . Thus, the total sample complexity is

By Lemma D.3, with failure probability at most 0 . 1 , Lines 20-22 of the algorithm will successfully output a hypothesis which is a mixture of k Gaussians, with total variation distance at most O ( α ) from the right answer. So, the overall success probability is at least 0 . 8 .

G Lower Bound

In this section, we prove Theorem 1.6. The proof of the lower bound will, at a high level, follow from known lower bounds for privately learning a single Gaussian [KV18, KMS22a], which we now state.

Theorem G.1. [KV18] For some sufficiently small constant c ∗ , ( ε, δ ) -privately learning an arbitrary univariate Gaussian N ( µ, σ 2 ) up to total variation distance c ∗ requires Ω ( log(1 /δ ) ε ) samples.

Moreover, this lower bound holds even if we are promised that | µ | ≤ (1 /δ ) C for a sufficiently large constant C , and σ = 1 .

Note that this lower bound immediately implies the same lower bound for general dimension d .

Theorem G.2. [KMS22a] Let α be at most a sufficiently small constant, and let δ ≤ ( αε d ) C for a sufficiently large constant C . Then, ( ε, δ ) -privately learning a d -dimensional Gaussian N ( µ, Σ) up to total variation distance α requires ˜ Ω ( d 2 αε ) samples.

Moreover, this lower bound holds even if we are promised that µ = 0 , and I ≼ Σ ≼ 2 I .

Note that a tighter version of the above theorem has been proved in [Nar23, PH24]; we also refer the reader to [BUV14].

Proof Sketch of Theorem 1.6. First, the lower bound of kd 2 α 2 is already known - see [ABH + 18].

The lower bound of kd 2 αε will follow from Theorem G.2. To explain how, we consider k distinct Gaussians N ( µ i , Σ i ) , where the means µ i are known and very far away from each other, and I ≼ Σ i ≼ 2 I are unknown. The overall mixture that we will try to learn is the uniform mixture over N ( µ i , Σ i ) , i.e., every weight w i = 1 /k . By making them very far away from each other, we are

making learning the full mixture equivalent to learning each component (on average). Namely, even if we are given the information of which Gaussian each sample comes from, we will need to learn at least 2 / 3 of the Gaussians up to total variation distance O ( α ) , to learn the full mixture up to total variation distance α . Hence, we will need at least k times as many samples as for learning a single Gaussian, which means we need ˜ Ω ( kd 2 ) total samples.

αε

The lower bound of k log(1 /δ ) αε will follow from Theorem G.1. Note that it suffices to prove the lower bound in the univariate case. We plant k distinct Gaussians N ( µ i , 1) , where the µ i are very far away from each other, i.e., pairwise | µ i -µ j |≫ (1 /δ ) 10 C . We also assume that µ 1 = 0 is known, and the remaining µ i are unknown but we are promised the value of each µ i up to error (1 /δ ) C . The overall mixture will have the first Gaussian N (0 , 1) of weight w 1 = 1 -α/c ∗ , and the remaining Gaussians N ( µ i , 1) each have weight w i = α/ ( c ∗ · ( k -1)) . Even if we are given the information of which Gaussian component each sample comes from, to learn the overall mixture up to error α , we need to learn at least 2 / 3 of the small-weight components, each up to total variation distance O ( c ∗ ) . Hence, we will need log(1 /δ ) ε samples from most of the small-weight components. Since the small weight components have weight Θ( α/k ) , we need Ω ( k log(1 /δ ) αε ) total samples.

We now give a formal proof of Theorem 1.6.

G.1 Formal Proof of Theorem 1.6

First, we note two lemmas that will be helpful in proving the Theorem.

Lemma G.3. Let α ∈ [0 , 1) . Let f be a probability density function over R d . Assume g : R d → R ≥ 0 exists such that

Then there exists h such that h is a probability density function over R d , and

Proof. We know that

Therefore, G := ∫ R d g ( x ) d x = 1 ± α.

Now two cases are possible either G < 1 , or G > 1 , otherwise we are done. If G < 1 , let h ( x ) be the density function corresponding to the following distribution: with probability G take a sample from g ( x ) /G , and with probability 1 -G select 0 . Then

as desired. If G > 1 , let h ( x ) be a density function as follows: ∀ x : 0 ≤ h ( x ) ≤ g ( x ) , and ∫ R d h ( x ) d x = 1 . It is easy to see such an h exists by greedily picking h . Then we have ∫ R d | g ( x ) -h ( x ) | d x = ∫ R d g ( x ) -h ( x ) d x ≤ G -1 ≤ α . Therefore, we can write

as desired.

Lemma G.4. Suppose w ∈ (0 , 1) is fixed. Suppose an ( ε, δ ) differentially private algorithm exists that takes n samples from the mixture D = wD 1 +(1 -w ) D 2 , and learns D 1 in total variation distance up to error α , with success probability 1 -β . Moreover, assume while sampling it is known which component the sample is sampled from. Then there exists an ( ε, δ ) differentially private algorithm A that takes nw/γ samples from D 1 and outputs an estimate of D 1 up to total variation distance α , with success probability 1 -β -γ .

Proof. We can view the sampling procedure of the mixture as sampling a random variable t ∼ Bin( n, w ) , and taking t samples from D 1 and n -t samples from D 2 . In order to make an algorithm using nw/γ samples we can take that many samples from D 1 , and then sample t from Bin( n, w ) , and run A on n data points constructed as follows: If t is smaller than nw/γ , use t of the samples taken from D 1 and set the rest to 0 , If t is larger than nw/γ , just run A on all zeroes. Finally output the output of A on this input. Clearly, this would be ( ε, δ ) differentially private. We know the Algorithm succeeds with probability 1 -β , over the random coins of the algorithm and the randomness of sampling, if the input is sampled from wD 1 + (1 -w ) D 2 . From Markov's inequality, we know P [ t ≤ nw/γ ] ≥ 1 -γ . Therefore, our constructed sample is drawn i.i.d from wD 1 +(1 -w ) D 2 with probability at least 1 -γ , where D 2 is the fixed 0 distribution. Therefore, there exists an ( ε, δ ) differentially private algorithm that takes nw/γ many samples from D 1 and outputs an estimate of D 1 up to total variation distance α , with success probability 1 -β -γ , as desired.

We now prove Theorem 1.6.

Proof. We prove the lower bound terms one by one. The first term kd 2 α 2 (the non private term) is known by previous work [ABH + 18]. We prove the lower bound for the second term and the last term here using Theorems G.1 and G.2.

Let's prove the second term kd 2 αε . We apply Theorem G.2, this theorem implies that for any α smaller than a sufficiently small constant, and δ ≤ ( αε d ) C for a sufficiently large C , any ( ε, δ ) differentially private algorithm A taking n samples in R d , satisfying

must also satisfy n = ˜ Ω( d 2 αε ) .

Let µ ′ i s be k distinct vectors in R d each having ℓ 2 distance M from each other, for M to be set later. To see why such a set exists, we can take µ i = Mie 1 , where e 1 is the first unit vector. Now consider the following set of Gaussians: D i = N ( µ i , Σ i ) , where µ i 's are known and constructed as above and Σ i unknown. Consider the uniform mixture D over these Gaussians, with weights w i = 1 /k . We also assume that when sampling from this distribution we know that which component the samples came from. Consider an ( ε, δ ) differentially private algorithm A that takes n samples from D and outputs a distribution ˆ D such that d TV ( D, ˆ D ) ≤ α with probability 2 / 3 .

Now consider a sample from D i , from standard Gaussian tail bounds we know that at least 1 -exp( -M 2 / 800) fraction of the mass of D i is contained within a ball of radius M/ 10 , around µ i . Let B i denote this ball, and note that B i 's are disjoint.

Let f, f i , ˆ f be the probability density functions corresponding to D,D i , ˆ D respectively. Assuming, we are in the success regime, we can write

Now let B ⊆ [ k ] be the set of indices i such that ∫ B i ∣ ∣ ∣ f ( x ) -ˆ f ( x ) ∣ ∣ ∣ d x ≥ 200 α/k , and G be its complement. Then we have that |B| ≤ k/ 100 . Therefore, there exists G such that |G| ≥ 0 . 99 k , and ∀ i ∈ G : ∫ B i ∣ ∣ ∣ f ( x ) -ˆ f ( x ) ∣ ∣ ∣ d x ≤ 200 α/k . Assume i ∈ G is one such index. Note that f ( x ) = ∑ k j =1 f j ( x ) /k . Therefore, we can write

̸

where the last inequality comes from taking M ≥ 30 √ log( k/ 100 α ) . We show that using the distribution ˆ D , we can construct an answer to the problem of learning the Gaussian D i . To do so first take g i to be equal to k ˆ f ( x ) over B i and 0 everywhere else. We have

Now we may apply Lemma G.3, and deduce that given ˆ D we can construct probability density functions and distributions ˆ D i 's such that d TV ( ˆ D i , D i ) ≤ 800 α , for all i ∈ G . To recap, so far we have shown that given an ( ε, δ ) differentially private algorithm that takes as inputs samples from our constructed mixture of Gaussians and outputs a density ˆ D that has total variation distance at most α , from the ground truth distribution with success probability 2 / 3 , we can use ˆ D to construct densities ˆ D i such that d TV ( ˆ D i , D i ) ≤ 800 α , for 0 . 99 k of the indices i . This implies that there exists a fixed index i for which the component D i is learned up to error 800 α with success probability 2 / 3 -0 . 01 ≥ 0 . 65 . Applying Lemma G.4, implies that there must exist an ( ε, δ ) differentially private algorithm that takes 100 n/k samples from D i and estimates its density up to total variation distance 800 α , with success probability at least 0 . 6 . Therefore, applying Theorem G.2, we conclude that n = ˜ Ω( kd 2 αε ) .

Now let's prove the last term k log(1 /δ ) αε . We aim to apply Theorem G.1. Let µ i 's be k distinct values in R , each having distance M from each other, for M ≫ (1 /δ ) 10 C to be set later, where µ 1 = 0 . It is easy to see such a set exists. Now consider the following set of Gaussians: D i = N ( µ i , 1) , where µ i 's are known up to log(1 /δ ) C , and µ 1 = 0 is also known. Consider the mixture D over these Gaussians, with weights w 1 = 1 -α/c ∗ , and w i = α/ ( c ∗ ( k -1)) . We also assume that when sampling from the mixture we know which component each sample comes from. Consider an ( ε, δ ) differentially private algorithm A that takes n samples from D and outputs a distribution ˆ D such that d TV ( D, ˆ D ) ≤ α with probability 2 / 3 .

Now consider a sample from D i , from standard Gaussian tail bounds we know that at least 1 -exp( -M 2 / 200) fraction of the mass of D i is contained within a ball of radius M/ 10 , around µ i . Let B i denote this ball, and note that B i 's are disjoint.

̸

Let f, f i , ˆ f be the probability density functions corresponding to D,D i , ˆ D respectively. Assuming, we are in the success regime, similar to the proof of the previous term, we can show that there exists a set G of indices such that |G| ≥ 0 . 99 k , and ∀ i ∈ G : ∫ B i ∣ ∣ ∣ f ( x ) -ˆ f ( x ) ∣ ∣ ∣ d x ≤ 200 α/k . Moreover, with a similar argument as the previous term for i ∈ G we can say as long as M ≥ 20 √ log( k/ 100 α ) , ∫ B i ∣ ∣ ∣ w i f i ( x ) -ˆ f ( x ) ∣ ∣ ∣ d x ≤ 300 α/k . We show that using the distribution ˆ D , we can construct an answer to the problem of learning the Gaussian D i , for i = 1 . To do so take g i to be equal to ˆ f ( x ) /w i , over B i and 0 everywhere else. We have

̸

Now we may apply Lemma G.3, and deduce that given ˆ D , we can construct probability density functions and distributions ˆ D i 's such that d TV ( ˆ D i , D i ) ≤ 800 c ∗ , for all i ∈ G . To recap, so far we have shown that given an ( ε, δ ) differentially private algorithm that takes as inputs samples from our constructed mixture of Gaussians and outputs density ˆ D that has total variation distance at most α from the ground truth distribution with success probability 2 / 3 , we can use ˆ D to construct densities ˆ D i such that d TV ( ˆ D i , D i ) ≤ 800 c ∗ , for 0 . 99 k of the indices i . This implies that there exists a fixed index i = 1 , for which the component D i is learned up to error 800 c ∗ , with success probability 2 / 3 -0 . 01 ≥ 0 . 65 . Applying Lemma G.4, implies that there must exist an ( ε, δ ) differentially private algorithm that takes 100 nw i samples from D i and estimates its density up to total variation distance 800 c ∗ , with success probability 0 . 6 . Therefore, applying Theorem G.1, and noting that w i = α/ ( c ∗ ( k -1)) we conclude that n = Ω( k log(1 /δ ) αε ) .

H Omitted Proofs

In this section, we prove Proposition B.2, Theorem B.3, and Lemma C.7.

H.1 Proof of Proposition B.2

First, we note a basic fact.

We now prove Proposition B.2.

First, assume ( µ 1 , Σ 1 ) ≈ γ,ρ,τ ( µ 2 , Σ 2 ) . This means ∥ J 1 J ⊤ 1 -I ∥ op ≤ γ and ∥ J 1 J ⊤ 1 -I ∥ F ≤ ρ . We then have that Σ -1 / 2 1 Σ 2 Σ -1 / 2 1 = J -1 1 ( J -1 1 ) ⊤ = ( J ⊤ 1 J 1 ) -1 . Now, note that J 1 J ⊤ 1 and J ⊤ 1 J 1 are both symmetric and have the same eigenvalues. If we call these eigenvalues λ 1 , . . . , λ d , then the eigenvalues of Σ -1 / 2 1 Σ 2 Σ -1 / 2 1 are λ -1 1 , . . . , λ -1 d . Now, our assumption ( µ 1 , Σ 1 ) ≈ γ,ρ,τ ( µ 2 , Σ 2 ) implies that 1 -γ ≤ λ i ≤ 1+ γ and ∑ (1 -λ i ) 2 ≤ ρ 2 . This means that, assuming γ ≤ 0 . 1 , 1 -2 γ ≤ 1 1+ γ ≤ λ -1 i ≤ 1 1 -γ ≤ 1 + 2 γ, and ∑( 1 -λ -1 i ) 2 ≤ ∑ (1 -λ i ) 2 · λ -2 i ≤ 2 · ∑ (1 -λ i ) 2 = 2 ρ 2 . This means that ∥ Σ -1 / 2 1 Σ 2 Σ -1 / 2 1 ∥ op ≤ 2 γ and ∥ Σ -1 / 2 1 Σ 2 Σ -1 / 2 1 ∥ F ≤ 2 ρ .

Finally, Σ -1 / 2 1 ( µ 1 -µ 2 ) = J -1 1 Σ -1 / 2 2 ( µ 1 -µ 2 ) . Because Σ -1 / 2 2 ( µ 1 -µ 2 ) has magnitude at most τ by our assumption, J -1 1 Σ -1 / 2 2 ( µ 1 -µ 2 ) has magnitude at most the maximum singular value of J -1 1 times τ . But every singular value of J -1 1 is some λ -1 / 2 i which is at most 2 , so ∥ Σ -1 / 2 1 ( µ 2 -µ 1 ) ∥ 2 = ∥ J -1 1 Σ -1 / 2 2 ( µ 1 -µ 2 ) ∥ 2 ≤ 2 τ .

Next, assume ( µ 1 , Σ 1 ) ≈ γ,ρ,τ ( µ 2 , Σ 2 ) and ( µ 2 , Σ 2 ) ≈ γ,ρ,τ ( µ 3 , Σ 3 ) . First, note that Σ -1 / 2 3 Σ 1 Σ -1 / 2 3 = J 2 J 1 J ⊤ 1 J ⊤ 2 = ( J 2 J 1 )( J 2 J 1 ) ⊤ . If the eigenvalues of J 1 J ⊤ 1 are { λ i } and the eigenvalues of J 2 J ⊤ 2 are { λ ′ i } , then the singular values of J 1 and J 2 are { √ λ i } and { √ λ ′ i } , respectively. By our assumption, 1 -γ ≤ λ i , λ ′ i ≤ 1+ γ , which means that √ 1 -γ ≤ √ λ i , √ λ ′ i ≤ √ 1 + γ . Thus, the singular values of J 2 J 1 are between 1 -γ and 1 + γ , which means that the eigenvalues of ( J 2 J 1 )( J 2 J 1 ) ⊤ are between (1 -γ ) 2 and (1 + γ ) 2 . Hence, for γ ≤ 0 . 1 , ∥ J 2 J 1 J ⊤ 1 J ⊤ 2 -I ∥ op ≤ 4 γ .

Assume that λ i , λ ′ i are in decreasing order. We now consider the k th largest singular value of J 2 J 1 . If σ k := √ λ k is the k th largest singular value of J 1 and σ ′ k := √ λ ′ k is the k th largest singular value of J 2 , by Corollary A.5 there exist subspaces V k , V ′ k of dimension d -k +1 such that ∥ J 1 v ∥ 2 ≤ σ k ∥ v ∥ 2 for all v ∈ V k and ∥ J 2 v ∥ 2 ≤ σ ′ k ∥ v ∥ 2 for all v ∈ V ′ k . Therefore, for every v ∈ V k ∩ J -1 1 V ′ k (note that J 1 is invertible since J 1 J ⊤ 1 has all eigenvalues between 1 -γ and 1 + γ ) we have that ∥ J 2 J 1 v ∥ 2 ≤ σ ′ k ∥ J 1 v ∥ 2 ≤ σ k σ ′ k ∥ v ∥ 2 . Because V k and J -1 1 V ′ k both have dimension d -k +1 , their intersection has dimension at least d -2 k +2 . So, there is a subspace of dimension at least d -2 k +2 such that every v in the subspace has ∥ J 2 J 1 v ∥ 2 ≤ σ k σ ′ k · ∥ v ∥ 2 .

Thus, the (2 k -1) th largest singular value of J 2 J 1 is at most σ k σ ′ k , so the (2 k -1) th largest eigenvalue of J 2 J 1 J ⊤ 1 J ⊤ 2 is at most λ k λ ′ k . The same argument, looking at the smallest singular values, tells us that the (2 k -1) th smallest eigenvalue of J 2 J 1 J ⊤ 1 J ⊤ 2 is at least λ d -k +1 λ ′ d -k +1 . Thus, for any t , the t th largest eigenvalue of J 2 J 1 J ⊤ 1 J ⊤ 2 is at most λ ⌊ ( t +1) / 2 ⌋ λ ′ ⌊ ( t +1) / 2 ⌋ and at least λ ⌈ ( d + t ) / 2 ⌉ λ ′ ⌈ ( d + t ) / 2 ⌉ .

Overall, this means that

where the fourth line uses Fact H.1. Thus, ∥ Σ -1 / 2 3 Σ 1 Σ -1 / 2 3 -I ∥ F ≤ 4 ρ .

Finally, note that ∥ Σ -1 / 2 3 ( µ 1 -µ 3 ) ∥ 2 ≤ ∥ Σ -1 / 2 3 ( µ 1 -µ 2 ) ∥ 2 + ∥ Σ -1 / 2 3 ( µ 2 -µ 3 ) ∥ 2 = ∥ J 2 Σ -1 / 2 2 ( µ 1 -µ 2 ) ∥ 2 + ∥ Σ -1 / 2 3 ( µ 2 -µ 3 ) ∥ 2 . By our assumptions, both ∥ Σ -1 / 2 2 ( µ 1 -µ 2 ) ∥ 2 and ∥ Σ -1 / 2 3 ( µ 2 -µ 3 ) ∥ 2 are at most τ , and J 2 has operator norm at most 1 . 1 ≤ 2 , which means that ∥ Σ -1 / 2 3 ( µ 1 -µ 3 ) ∥ 2 ≤ 3 τ .

H.2 Proof of Theorem B.3

First, we note a series of known results that will be key to proving the theorem. We start with the bound for robust covariance estimation in spectral error.

Lemma H.2 (e.g., [DK22, Exercise 4.3]) . Fix any η ∈ (0 , η 0 ) , where η 0 < 0 . 01 is a small universal constant, and fix any β ∈ (0 , 1) . There is a (deterministic, inefficient) algorithm A 1 with the following property. Let Σ ∈ R d × d be any covariance matrix, and let X = { X 1 , . . . , X n } ∼ N (0 , Σ) , where n ≥ O (( d +log(1 /β )) /η 2 ) . Then, with probability at least 1 -β over the randomness of X , for any η -corruption X ′ = { X ′ 1 , . . . , X ′ n } of X , A 1 ( X ′ ) outputs ˆ Σ 1 such that ∥ Σ -1 / 2 ˆ Σ 1 Σ -1 / 2 -I ∥ op ≤ O ( η ) . Importantly, A 1 may have knowledge of η and β , but does not have knowledge of X or Σ .

Next, we prove how to robustly estimate the covariance up to Frobenius error. We start with the following structural lemma.

Lemma H.3. There exists a universal constant c ∈ (0 , 0 . 1) with the following property. Fix any 1 ≤ k ≤ d , and let n ≥ ˜ O ( d · k ) be a sufficiently large (i.e., n ≥ dk · ( C 3 log( dk )) C 4 for some absolute constants C 3 , C 4 ). If we sample i.i.d. X = { X 1 , . . . , X n } ∼ N (0 , I ) , then with probability at least 1 -e -cn over X , for all symmetric matrices P ⊂ R d × d of rank at most k and Frobenius norm 1 , and for all subsets S ⊂ [ n ] of size at least (1 -c ) · n , ∣ ∣ 1 n ∑ i ∈ S ⟨ X i X ⊤ i -I, P ⟩ ∣ ∣ ≤ 0 . 1 .

Proof. Consider any fixed P ⊂ R d × d of rank k and Frobenius norm 1 , and fix any integer m . For any data points X 1 , . . . , X m i.i.d. ∼ N (0 , I ) , let X = ( X 1 , . . . , X m ) ∈ R m · d be the concatenation of X 1 , . . . , X m , and let Q ∈ R ( md ) × ( md ) be the block matrix

Then, 1 m ∑ m i =1 ⟨ X i X ⊤ i -I, P ⟩ = 1 m ∑ m i =1 ( X ⊤ i PX i -Tr( P ) ) = 1 m ( X ⊤ QX -Tr( Q ) ) . By the Hanson-Wright inequality, we have that for any fixed ∥ P ∥ F ≤ 1 , and for any t ≤ 1 ,

By setting t = 0 . 01 , we have that for some universal constant c 1 ,

Now, we draw X 1 , . . . , X n ∼ N (0 , I ) , and take a union bound over all subsets S ⊂ [ n ] of size at least (1 -c ) n (with m = | S | ) and a union bound over a net of possible matrices P . The number of options for S is at most ∑ i ≤ cn ( n i ) ≤ ( e/c ) cn . For P , we can choose a 1 /n 10 -sized net over the Frobenius norm metric (i.e., the distance between two matrices P 1 , P 2 is ∥ P 1 -P 2 ∥ F ) for each of the k nonzero eigenvalues and eigenvectors in the unit d -dimensional sphere, which has size at most n 100 d · k . Therefore, by a union bound, the probability that ∣ ∣ 1 n ∑ i ∈ S ⟨ X i X ⊤ i -I, P ′ ⟩ ∣ ∣ ≤ 0 . 01 for every | S | ≥ (1 -c ) n and every P ′ in the net is at least 1 -(2 e/c ) cn · n 100 d · k · e -c 1 n/ 2 .

Finally, we consider P outside of the net. For any symmetric P of rank k and Frobenius norm 1 , it has Frobenius distance at most 1 /n 8 from some P ′ in the net. Let us consider the event that every ∥ X i ∥ 2 2 ≤ 10 n , which for n ≥ d occurs with failure probability at most 2 n · e -c 1 n by Hanson-Wright. Under this event, ⟨ X i X ⊤ i -I, P -P ′ ⟩ ≤ ∥ X i X ⊤ i -I ∥ F · ∥ P -P ′ ∥ F ≤ (10 n + √ d ) /n 8 ≤ 0 . 01 .

As a result, with failure probability at most (2 e/c ) cn · n 100 d · k · e -c 1 n/ 2 +2 n · e -c 1 n ≤ e -cn (assuming c is sufficiently small), we have both properties. Namely, ∣ ∣ 1 n ∑ i ∈ S ⟨ X i X ⊤ i -I, P ′ ⟩ ∣ ∣ ≤ 0 . 01 for every | S | ≥ (1 -c ) n and every P ′ in the net, and for any P , ⟨ X i X ⊤ i -I, P -P ′ ⟩ 0 . 01 for the closest P ′ in the net to P . Overall, this means that for all such P and S , ∣ ∣ 1 n ∑ i ∈ S ⟨ X i X ⊤ i -I, P ⟩ ∣ ∣ ≤ 0 . 1 .

We prove another lemma which contrasts with Lemma H.3. √

Lemma H.4. Fix any 2 ≤ ρ ≤ d . Let k = 4 d/ρ 2 , and let n ≥ ˜ O ( d · k ) and c ∈ (0 , 0 . 1) be as in Lemma H.3. Fix any covariance matrix Σ and let X = { X 1 . . . , X n } ∼ N (0 , Σ) . Then, with probability at least 1 -e -cn , for every ˜ Σ such that 0 . 95 · Σ ≼ ˜ Σ ≼ 1 . 05 · Σ and ∥ Σ -1 / 2 ˜ ΣΣ -1 / 2 -I ∥ F ≥ ρ, for every symmetric matrix P ∈ R d × d of rank at most k and Frobenius norm 1 , and for every S ⊂ [ n ] of size at least (1 -c ) n ,

Proof. Let J = ˜ Σ -1 / 2 Σ 1 / 2 , and let Y i = Σ -1 / 2 X i . Then, ˜ Σ -1 / 2 X i = JY i , which means that ˜ Σ -1 / 2 X i X ⊤ i ˜ Σ -1 / 2 -I = JY i Y ⊤ i J ⊤ -I = J ( Y i Y ⊤ i -I ) J ⊤ +( JJ ⊤ -I ) .

From now on, we assume that for all S ⊂ [ n ] of size at least (1 -c ) · n , and for all symmetric matrices P of rank k and Frobenius norm 1 , ∣ ∣ 1 n ∑ i ∈ S ⟨ Y i Y ⊤ i -I, P ⟩ ∣ ∣ ≤ 0 . 1 . This happens with at least e -cn probability, by Lemma H.3.

Now, for any subset S ⊂ [ n ] ,

Now, note that by our assumptions, ∥ Σ -1 / 2 ˜ ΣΣ -1 / 2 -I ∥ op ≤ 0 . 05 and ∥ Σ -1 / 2 ˜ ΣΣ -1 / 2 -I ∥ F ≥ ρ . Thus, by Proposition B.2, ∥ ˜ Σ -1 / 2 Σ ˜ Σ -1 / 2 -I ∥ op ≤ 0 . 1 , and by Proposition B.2 again, applied the reverse direction this time, ∥ ˜ Σ -1 / 2 Σ ˜ Σ -1 / 2 -I ∥ F ≥ ρ/ 2 . We just showed ∥ JJ ⊤ -I ∥ op ≤ 0 . 1 , so by Proposition A.6, ∥ J ⊤ PJ ∥ F ≤ 2 ∥ P ∥ F ≤ 2 . Therefore, ∣ ∣ 1 n ∑ i ∈ S ⟨ Y i Y ⊤ i -I, J ⊤ PJ ⟩ ∣ ∣ ≤ 0 . 2 .

Conversely, we just showed ∥ JJ ⊤ -I ∥ F ≥ ρ/ 2 . So, if we order the eigenvalues of JJ ⊤ as λ 1 , λ 2 , . . . , λ d (and the corresponding unit eigenvectors v 1 , . . . , v d ) such that | λ i -1 | are in decreasing order, then ∑ d i =1 ( λ i -1) 2 ≥ ρ 2 / 4 , which means that ∑ 4 d/ρ 2 i =1 ( λ i -1) 2 ≥ 1 . So, if we choose P to be ( ∑ 4 d/ρ 2 i =1 ( λ i -1) 2 ) -1 / 2 ∑ 4 d/ρ 2 i =1 ( λ i -1) v i v ⊤ i , we have that ∥ P ∥ F = 1 and

Therefore,

Now, we can show how to learn the covariance of Σ up to Frobenius error.

Lemma H.5. Fix any η ∈ (0 , η 0 ) , where η 0 < 0 . 01 is a small universal constant, any β ∈ (0 , 1) , and any ˜ O ( η ) ≤ ρ ≤ √ d . There is a (deterministic, possibly inefficient) algorithm A 2 with the following property. Let Σ ∈ R d × d be any covariance matrix, and let X = { X 1 , . . . , X n } ∼ N (0 , Σ) , where n ≥ O ( d 2 +log 2 (1 /β ) ρ 2 +log( 1 β ) ) . Then, with probability at least 1 -β over the randomness of X , for any η -corruption X ′ = { X ′ 1 , . . . , X ′ n } of X , A 2 ( X ′ ) outputs ˆ Σ 2 such that ∥ Σ -1 / 2 ˆ Σ 2 Σ -1 / 2 -I ∥ F ≤ O ( ρ ) . Importantly, A 2 may have knowledge of η , ρ , and β , but does not have knowledge of X or Σ .

Proof. In the case that ρ ≤ 2 , the claim follows immediately from known results (for instance, it is implicit from [HKMN23]).

Alternatively, assume that ρ ≥ 2 . In this case, the algorithm works as follows. Assume η ≤ η 0 ≤ c/ 2 , where c is the constant in Lemma H.3. First, compute ˆ Σ 1 based on Lemma H.2. Note that (1 -O ( η )) · Σ ≼ ˆ Σ 1 ≼ (1 + O ( η )) · Σ with 1 -β probability, since the number of samples is sufficiently large. Now, find any positive definite ˜ Σ and a set T ⊂ [ n ] of size at least (1 -η ) n , such that:

First, we note that ˜ Σ = Σ is a feasible choice of ˜ Σ . Indeed, the first condition trivially holds. For the second condition, let T be the subset of uncorrupted data points (i.e., X ′ i = X i ). Then, for any S ⊂ T , the data points X ′ i for i ∈ S are the same as X i , so by Lemma H.3, with 1 -β probability, for every such S , ∣ ∣ 1 n ∑ i ∈ S ⟨ Σ -1 / 2 X i X ⊤ i Σ -1 / 2 -I, P ⟩ ∣ ∣ ≤ 0 . 1 .

Next, we show that every ˜ Σ with ∥ Σ -1 / 2 ˜ ΣΣ -1 / 2 -I ∥ F ≥ ρ is infeasible. First, we may assume that 0 . 95Σ ≼ ˜ Σ ≼ 1 . 05Σ , as otherwise, we cannot simultaneously satisfy (1 -O ( η )) · ˆ Σ 1 ≼ ˜ Σ ≼ (1 + O ( η )) · ˆ Σ 1 and (1 -O ( η )) · Σ ≼ ˆ Σ 1 ≼ (1 + O ( η )) · Σ , assuming η ≤ c/ 2 is sufficiently small.

Hence, we just have to verify the infeasibility for every ˜ Σ such that ∥ Σ -1 / 2 ˜ ΣΣ -1 / 2 -I ∥ F ≥ ρ and 0 . 95Σ ≼ ˜ Σ ≼ 1 . 05Σ . Indeed, for any subset T of size at least (1 -η ) n , let S be the uncorrupted points in T . Because there are at most ηn uncorrupted points, | S | ≥ (1 -2 η ) n . So by Lemma H.4, with 1 -β probability, for every such S , ∣ ∣ ∣ 1 n ∑ i ∈ S ⟨ ˜ Σ -1 / 2 X i X ⊤ i ˜ Σ -1 / 2 -I, P ⟩ ∣ ∣ ∣ ≥ 0 . 8 .

Therefore, with at most O ( β ) failure probability, some ˆ Σ 2 := ˜ Σ is returned, and it satisfies ∥ Σ -1 / 2 ˜ ΣΣ -1 / 2 -I ∥ F ≤ ρ .

Next, we note some results on robust mean estimation. We first note the definition of stability, and some properties.

Lemma H.6 ([DK22, Proposition 3.3]) . Let n ≥ O (( d + log(1 /β )) /α 2 ) , for some α ≥ O ( η √ log 1 /η ) . Let X = { X i } n i =1 i.i.d. ∼ D , where D is a subgaussian random variable with mean µ ∈ R d and covariance I . Then, with probability 1 -β , for all vectors b ∈ [0 , 1] n such that E i b i ≥ 1 -η and all unit vectors v ∈ R d , we have:

| E i b i ⟨ v, X i -µ ⟩| ≤ α.

∣ ∣ E i b i ⟨ v, X i -µ ⟩ 2 -1 ∣ ∣ ≤ α.

Given a dataset X with these properties, call it ( η, α ) -stable with respect to µ .

Lemma H.7 (implicit from [DK22, Section 2]) . Fix η sufficiently small and α = ˜ O ( η ) . There is a deterministic algorithm A 3 that, on a dataset X ′ , outputs ˆ µ such that ∥ ˆ µ -µ ∥ 2 ≤ O ( α ) , for any η -corruption X ′ of any X that is ( η, α ) -mean stable with respect to any µ ∈ R d . Importantly, A 3 does not require knowledge of X or µ .

We now prove Theorem B.3.

Proof. We first show how to estimate the covariance Σ . First, we apply a 'sample pairing' trick (e.g., see [DK22, Section 4.4]). Namely, assume WLOG that n is even, and define ˜ X to be the set { ( X 2 i -1 -X 2 i ) / √ 2 } n/ 2 i =1 , and ˜ X ′ = { ( X ′ 2 i -1 -X ′ 2 i ) / √ 2 } n/ 2 i =1 . Note that ˜ X are i.i.d. samples from N (0 , Σ) , and ˜ X ′ is at most 2 η -corrupted.

Now, because η ≤ γ , ˜ X ′ is at most 2 γ corrupted, so Lemma H.2 on ˜ X ′ (replacing η with 2 γ ) gives us some ˆ Σ 1 such that ∥ Σ -1 / 2 ˆ Σ 1 Σ -1 / 2 -I ∥ op ≤ O ( γ ) , by our assumed bound on the number of samples. Next, Lemma H.5 on ˜ X ′ gives us some ˆ Σ 2 such that ∥ Σ -1 / 2 ˆ Σ 2 Σ -1 / 2 -I ∥ F ≤ O ( ρ ) , by our assumed bound on the number of samples. So, we can set ˆ Σ to be any covariance such that ∥ ˆ Σ -1 / 2 ˆ Σ 1 ˆ Σ -1 / 2 -I ∥ op ≤ O ( γ ) and ∥ ˆ Σ -1 / 2 ˆ Σ 2 ˆ Σ -1 / 2 -I ∥ F ≤ O ( ρ ) . Note that Σ satisfies these properties, and any ˆ Σ that satisfies these properties must satisfy ∥ Σ -1 / 2 ˆ ΣΣ -1 / 2 -I ∥ F ≤ O ( γ ) and ∥ Σ -1 / 2 ˆ ΣΣ -1 / 2 -I ∥ F ≤ O ( ρ ) , by the approximate symmetry and transitivity properties (Proposition B.2).

Now, we estimate the mean µ . Taking the original data X ′ , we compute { ˆ Σ -1 / 2 X ′ i } . By stability (Lemma H.7), we know that with 1 -β probability, { Σ -1 / 2 X i } is ( η, γ ) -stable with respect to µ (where we are using the uncorrupted data and the true covariance Σ ). Letting J = ˆ Σ -1 / 2 Σ 1 / 2 , we know that J has all singular values between 1 -O ( γ ) and 1 + O ( γ ) , and that { J -1 · ˆ Σ -1 / 2 X i } is ( η, γ ) -stable with respect to µ . Moreover, note that we can write ⟨ v, ˆ Σ -1 / 2 ( X i -µ ) ⟩ = ⟨ Jv, J -1 · ˆ Σ -1 / 2 ( X i -µ ) ⟩ , and that 1 -O ( γ ) ≤ ∥ Jv ∥ 2 ≤ 1+ O ( γ ) . Therefore, { ˆ Σ -1 / 2 X i } is ( η, O ( γ )) -stable with respect to ˆ Σ -1 / 2 µ , which means that by Lemma H.7, A 3 on { ˆ Σ -1 / 2 X ′ i } outputs some value ˆ ν such that ∥ ˆ ν -ˆ Σ -1 / 2 µ ∥ 2 ≤ O ( γ ) . Thus, by setting ˆ µ := ˆ Σ 1 / 2 · ˆ ν, we have that ∥ ˆ Σ -1 / 2 (ˆ µ -ν ) ∥ 2 ≤ O ( γ ) , which means that ∥ Σ -1 / 2 (ˆ µ -µ ) ∥ 2 = ∥ J -1 ˆ Σ -1 / 2 (ˆ µ -µ ) ∥ 2 ≤ (1 + O ( γ )) · ∥ ˆ Σ -1 / 2 (ˆ µ -µ ) ∥ 2 ≤ O ( γ ) .

H.3 Proof of Lemma C.7

In this subsection, we prove Lemma C.7.

Proof. First, note that for any positive definite matrices Σ 1 , Σ 2 , ∥ Σ -1 / 2 2 Σ 1 Σ -1 / 2 2 -I ∥ op ≤ γ is equivalent to (1 -γ ) I ≼ Σ -1 / 2 2 Σ 1 Σ -1 / 2 2 ≼ (1 + γ ) I . Since M being PSD implies AMA ⊤ is PSD

(and vice versa if A is invertible), the previous statement is thus equivalent to (1 -γ ) · Σ 2 ≼ Σ 1 ≼ (1 + γ )Σ 2 . Next, note that

where the first two lines are a straightforward expansion, and the final line simply uses the fact that Tr( AB ) = Tr( BA ) for any matrices A,B . Finally, note that

Now, consider replacing Σ 1 with Σ 3 := Σ 1 / 2 Σ 1 Σ 1 / 2 , Σ 2 with Σ 4 := Σ 1 / 2 Σ 2 Σ 1 / 2 , µ 1 with µ 3 := Σ 1 / 2 µ 1 + µ , and µ 2 with µ 4 := Σ 1 / 2 µ 2 + µ . Again, since M being PSD implies AMA ⊤ is PSD (and vice versa), we have that (1 -γ ) · Σ 2 ≼ Σ 1 ≼ (1+ γ ) · Σ 2 if and only if (1 -γ ) · Σ 4 ≼ Σ 3 ≼ (1+ γ ) · Σ 4 . Moreover, note that

and

which means that (4) would be the same if we replaced Σ 1 with Σ 3 and Σ 2 with Σ 4 . Finally,

which means that (4) would be the same if we replaced µ 1 with µ 3 , µ 2 with µ 4 , Σ 1 with Σ 3 , and Σ 2 with Σ 4 .

Overall, by the definition of ≈ γ,ρ,τ , we have that ( µ 1 , Σ 1 ) ≈ γ,ρ,τ ( µ 2 , Σ 2 ) if and only if ( µ 3 , Σ 3 ) ≈ γ,ρ,τ ( µ 4 , Σ 4 ) .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We give complete proofs for all theorem statements.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss limitations about the data assumptions and practical algorithms, at the end of the main body.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We give complete proofs for all theorems.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: No experiments.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: No experiments.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: No experiments.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: No experiments.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: No experiments.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our work is entirely theoretical, and we do not see any potential harms coming from our research.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work is entirely theoretical, and we do not see any potential negative societal impacts coming from our research.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our work is entirely theoretical, and poses no such risks.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: Our paper does not use any existing assets, such as code/data/models/etc.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our paper does not release any new assets

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing or research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing or research with human subjects.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the important problem of private learning of the Gaussian mixture model to estimate the underlying distribution within a desired total variation distance. By combining different techniques, the authors succeed in deriving bounds that are of quadratic dimension, thus significantly improving the existing results.\n\nSTRENGTHS:\nThe paper contains several new results that significantly improve the state of the art. These include i) Theorem 1.3, which establishes a bound with quadratic complexity for any dimension, ii) Theorem 1.4, which proposes an improved bound for $d=1$, showing that the sample complexity can be linear in the number of components, iii) Theorem 1.5, which proposes a lower bound on the sample complexity. The latter, combined with Theorem 1.4, shows the optimality of the established bounds for the univariate Gaussian distributions.  In addition, the paper is generally well and smoothly written.\n\nWEAKNESSES:\n- The paper could benefit from some numerical verifications of the results/algorithms used.\n\n- The appendix section is poorly organized. Without an outline and proof, it is difficult to follow such a long appendix.\n\nQUESTIONS:\n1. It would be better to explain more the difference between density estimation and parameter estimation.\n\n2. How do the bounds of Theorems 1.3, 1.4., and 1.5 behave with respect to the failure probability $\\beta$ order-wise?\n\n3. Can the authors discuss the assumption $R=n^{100}$ (line 160) a bit more? Why does it depend on $n$ (and not $k$ or $d$?)? Also, is the polynomial dependence restrictive?\n\n4. The first sentence of the informal statement of Theorem 2.1. does not seem rigorous enough. Is such a $\\tilde{\\Sigma}$ unique? I don't think so. Then, if not, which choice of $\\tilde{\\Sigma}$ is taken into account when computing $V_{\\eta}(\\mathbf{X})$? Probably maximum volume?\n\n5. The appendix sections are very difficult to follow. There should be a detailed outline at the beginning to guide the reader as to what each appendix deals with. Also, there should be sufficient references in the main text. For example, after the informal statement of Theorem 2.1., it should be clearly stated where the full statement and proof can be found. Similarly, after each paragraph (e.g., Section 2.1) or after each mentioned previously established result (e.g., advanced composition theorem), there should be an exact pointer to where the complete version can be found in the appendices.\n\n6. Since the submission was allowed for 9 pages, the authors had 1.5 more pages. I think some material from the appendices, for example the algorithms used to privately learn GMM, could be moved to the main text.\n\n7. Finally, what is missing the most is the experimental section. If I'm not mistaken, this should be easy to do. Although not all theoretical work requires experimental verification, I believe that if the experiments verify the theoretical finding, this would greatly increase the importance of the results.\n\nMinor comments:\n\n- Line 52: it is better to add \"The parameters $\\alpha$ and $\\beta$...\".\n\n- The constant $c^*$ in Theorem 1.5. does not appear in the bound. \n\n- In the first paragraph of Section 2.1, it is mentioned that \"... as we can finish the procedure with hypothesis selection, as discussed above\". However, hypothesis selection has not been sufficiently discussed.\n\n- Is it true that the intuition given in Section 2.1. holds if $ n \\gg k^2$?\n\n- Typo line 178 (a a)\n\n- Is $k$ in lines 203-204 the number of components in the Gaussian mixture? Why does this statement (\"if we altered $k$ data points...\") only hold for $k$ changes?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

comments: SUMMARY:
This paper investigates the sample complexity of privately learning mixtures of Gaussians. The authors achieve a sample complexity of approximately $O(kd^2+k^{1.5}d^{1.75}+k^2d)\log R$ where $R$ is an upper bound on the condition number of the covariance matrix and the norm of the mean. This result improves upon the previous best bound of $O(k^2d^4)$. Additionally, they constructed a lower bound of $\Omega(kd^2+k\log(1/\delta)/\epsilon)$, which refutes a conjecture from prior work, and they achieve optimal sample complexity when $d=1$.

STRENGTHS:
The improvement in sample complexity is significant.
The paper provides a thorough technical overview of the upper bound, detailing the incorporation of sample compression and the methods used to enhance dimension independence within the robustness-to-privacy conversion technique.

WEAKNESSES:
The paper is technically dense, making it challenging to understand and verify all the details.
The authors do not fully utilize the page limit, which could have been used to explain the techniques more clearly.
The technique for establishing the lower bound is not discussed in the main body of the paper.
Including formal definitions and crucial lemmas in the main text could help readers better understand the key ideas. For instance, the definition of volume was unclear until I consulted the appendix.

QUESTIONS:
1. Where does the $k^2$ term come from in the upper bound?
2. Are there alternative methods to address this problem that do not rely on the robustness-to-privacy conversion technique?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis works focuses on the task of density estimation of a mixture of Gaussians under the restriction of differential privacy. Unlike parameter estimation, density estimation does not require accurately estimating the mixture's parameters, but instead bounds the total variation distance between estimated and true distributions. This task can be achieved even without any boundedness or separation assumptions on the parameters of the components.\n\nThis task is known to require sample size of $O(k d^{2}/\\alpha^{2})$ even in the non-private setting, where $k$ is the number of components, $d$ is the dimension, $\\alpha$ is the bound on the TV distance, and the $O$ notation ignores logarithmic factors. In the private setting, previous work [1] achieved $\\alpha$ accuracy guarantee under $(\\epsilon, \\delta)$-DP with $O(k^{2} d^{4}/\\alpha^{2} \\epsilon)$ (the exact bound includes several other terms that depend on $\\log(1/\\delta)$ as well).\n\nThis work provides an improved bound, reducing the dependence on the parameters to $O \\left(\\frac{k d^{2}}{\\alpha^{2}} + \\frac{k d^{2}}{\\alpha \\epsilon} \\right)$ in the high dimensional regime, $O \\left(\\frac{k}{\\alpha^{2}} + \\frac{k \\cdot \\log(1/\\delta)}{\\alpha \\epsilon} \\right)$ in the univariate setting, and lower bounds which asymptotically match the upper bound in the univariate case and nearly match it in the multivariate one.\n\nThe proposed algorithm relays on first achieving crude estimation of the parameters and then using hypothesis selection to improve the estimation. The crude estimation is achieved using robust GMM estimation techniques and robusteness-to-privacy conversion, based on an inverse sensitivity-like instantiation of the exponential mechanism. This method is computationally inefficient.\n\n[1] Mohammad Afzali, Hassan Ashtiani, and Christopher Liaw. Mixtures of gaussians are privately learnable with a polynomial number of samples. In Algorithmic Learning Theory, pages 1–27. PMLR, 2024.\n\nSTRENGTHS:\nThe results presented in this work provide a significant improvement over the previously known ones. Though the proof technique is highly involved, the authors presented the proof outline in a relatively intuitive way, and provided motivation for the various steps it includes.\n\nWEAKNESSES:\nDespite the great work done by the authors and my best efforts, I was not able to follow up all the proof structure. In particular, I could not find the justification for some of terms that appeared in the bound presented on Theorem 1.3. It seems to me like a section that \"puts everything together\" will be of great benefit. I will try to describe my current understanding and existing gaps.\n\nTo the best of my understanding:\n* The $\\frac{k d^{2}}{\\alpha^{2}}$ and $\\frac{k d^{2}}{\\alpha \\epsilon}$ were explained at the \"Reducing to “crude approximation”\" section, and they represent the sample size required to accurately and privately learn the GMM given some crude estimation of its components, using hypothesis selection method.\n* The $\\frac{d^{1.75} k^{1.5} \\sqrt{\\log(1/\\delta)}}{\\alpha \\epsilon}$ was explained at the \"Improving Dimension Dependence\" section, where $\\frac{d^{1.75} k}{\\alpha \\epsilon}$ results from the fact $O\\left(\\frac{d^{1.75} k}{c \\epsilon} \\right)$ points are required to get the crude estimation of the parameters of a single component under $c$-robustness, which then can be translated to a DP estimation with $\\alpha$ accuracy using rubousteness-to-privacy conversion techniques (Theorem 2.1), and the additional $\\sqrt{k \\cdot \\log(1/\\delta)}$ term results from advanced composition over $k$ components.\n* I failed to understand where the additional two terms ($\\frac{(k \\cdot \\log(1/\\delta))^{1.5}}{\\alpha \\epsilon}$ and $\\frac{k^{2} d}{\\alpha}$) come from, and I suspect they were accumulated at some point during the rubousteness-to-privacy transformation.\n\nQUESTIONS:\nAs I mentioned before, I will find an additional section that brings all the components together to outline the final proof method, focusing on the final achieved bound, to be very useful.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of learning a mixture of $k$ $d$-dimensional Gaussians using a differentially private mechanism with respect to the samples. It provides an improved sample complexity which has asymptotically optimal dependence on the dimension $d$ for small $k$, the lower bound is also given by the paper. Additionally, for $d=1$ the paper provides optimal sample complexity.  \nThe paper follows a high level approach of obtaining crude approximations of the mean and covariances of the component Gaussians, which can be used to reduce – using a net based argument – the problem to private hypothesis selection for which existing algorithm (needing only log the size of the net number of samples) can be used. To obtain the crude approximations respecting the DP guarantee, the paper uses a variant of the exponential mechanism with a carefully constructed scoring function measuring the distance between the a candidate Gaussian and any “heavy” component of Gaussian mixture. This uses a number of samples depending on the size of the sample set $n$ and the dimensionality the hypotheses $d^2$. A naive approach fails due to the blowup incurred, and the paper utilizes sample compression to reduce the dependence to $O(d\\\\log n)$ and improves the dimensionality dependence to $O(d)$ via an estimation argument.  \nThe above is outlined in more detail in the main paper and proved formally in the appendices. While all the details have not been verified by the reviewer, the approach taken by the paper seems correct.\n\nSTRENGTHS:\n1. The paper proves improved (and in some cases optimal) sample complexity for privately learning Gaussian mixtures.\n2. The paper uses a novel crude approximation based approach paired with existing algorithm for private hypothesis selection. \n3. The paper leverages inverse sensitivity mechanisms for  decoding the crude approximations, techniques for sample compression, and combines them with a way to improve the dimensionality dependence.\n4. The main result as well the one for univariate case and the lower bound together constitute notable progress on a well studied problem.\n5. The paper is well written and the provided roadmap greatly aids the understanding.\n\nWEAKNESSES:\n1. The sample complexity does not match the lower bound in the dependence on $k$.\n2. The degradation on the dependence on $k$ from the univariate to the multi-variate case is not explained in the main paper.\n3. The details of the various technical parts are tedious and could be alleviated by better presentation, e.g. by listing all the parameters and their dependencies in a table.\n\nQUESTIONS:\nIt will be useful to combine definitions 1.1 and 1.2 into a precise definition of privately learning GMMs.\n\nLIMITATIONS:\nYes, addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-17"}