Title: Private estimation algorithms for stochastic block models and mixture models
Venue: NeurIPS 2023
Accepted: yes
Keywords: differential privacy

stochastic block model

Gaussian mixture model

sum of squares
Abstract: We introduce general tools for designing efficient private estimation algorithms, in the high-dimensional settings, whose statistical guarantees almost match those of the best known non-private algorithms.
To illustrate our techniques, we consider two problems: recovery of stochastic block models and learning mixtures of spherical Gaussians.

For the former, we present the first efficient $(\epsilon, \delta)$-differentially private algorithm for both weak recovery and exact recovery. Previously known algorithms achieving comparable guarantees required quasi-polynomial time. 

For the latter, we design an  $(\epsilon, \delta)$-differentially private algorithm that recovers the centers of the $k$-mixture when the minimum separation is at least $	O(k^{1/t}\sqrt{t})$. For all choices of $t$, this algorithm requires sample complexity $n\geq k^{O(1)}d^{O(t)}$ and time complexity $(nd)^{O(t)}$. Prior work required either an additional additive $\Omega(\sqrt{\log n})$ term in the minimum separation or an explicit upper bound on the Euclidean norm of the centers.

Private estimation algorithms for stochastic block models and mixture models

Abstract

We introduce general tools for designing efficient private estimation algorithms, in the high-dimensional settings, whose statistical guarantees almost match those of the best known non-private algorithms. To illustrate our techniques, we consider two problems: recovery of stochastic block models and learning mixtures of spherical Gaussians.

For the former, we present the first efficient ( ε, δ ) -differentially private algorithms for both weak recovery and exact recovery. Previously known algorithms achieving comparable guarantees required quasi-polynomial time. We complement these results with an information-theoretic lower bound that highlights how the guarantees of our algorithms are almost tight.

For the latter, we design an ( ε, δ ) -differentially private algorithm that recovers the centers of the k -mixture when the minimum separation is at least O ( k 1 /t √ t ) . For all choices of t , this algorithm requires sample complexity n ⩾ k O (1) d O ( t ) and time complexity ( nd ) O ( t ) . Prior work required either an additional additive Ω( √ log n ) term in the minimum separation or an explicit upper bound on the Euclidean norm of the centers.

1 Introduction

Computing a model that best matches a dataset is a fundamental question in machine learning and statistics. Given a set of n samples from a model, how to find the most likely parameters of the model that could have generated this data? This basic question has been widely studied for several decades, and recently revisited in the context where the input data has been partially corrupted (i.e., where few samples of the data have been adversarially generated-see for instance [37, 18, 22, 20]). This has led to several recent works shedding new lights on classic model estimation problems, such as the Stochastic Block Model (SBM) [28, 46, 44, 25, 21, 40] and the Gaussian Mixture Model (GMM) [30, 36, 9, 10] (see Definitions 1.1 and 1.2).

Privacy in machine learning and statistical tasks has recently become of critical importance. New regulations, renewed consumer interest as well as privacy leaks, have led the major actors to adopt privacy-preserving solutions for the machine learning [1, 2, 3]. This new push has resulted in a flurry of activity in algorithm design for private machine learning, including very recently for SBMs and GMMs [55, 32, 16, 60]. Despite this activity, it has remain an open challenge to fully understand how privacy requirements impact model estimation problems and in particular their recovery thresholds and the computational complexity. This is the problem we tackle in this paper.

While other notions of privacy exist (e.g. k -anonymity), the de facto privacy standard is the differential privacy (DP) framework of Dwork, McSherry, Nissim, and Smith [24]. In this framework, the privacy quality is governed by two parameters, ε and δ , which in essence tell us how the probability of seeing a given output changes (both multiplicatively and additively) between two datasets that differ by any individual data element. This notion, in essence, quantifies the amount of information leaked by a given algorithm on individual data elements. The goal of the algorithm designer is to come up with differentially private algorithms for ε being a small constant and δ being of order 1 /n Θ(1) .

Differentially private analysis of graphs usually considers two notions of neighboring graphs. The weaker notion of edge-DP defines two graphs to be neighboring if they differ in one edge. Under the stronger notion of node-DP , two neighboring graphs can differ arbitrarily in the set of edges connected to a single vertex. Recently, there is a line of work on node-DP parameter estimation in random graph models, e.g. Erdös-Rényi models [61] and Graphons [12, 13]. However, for the more challenging task of graph clustering, node-DP is sometimes impossible to achieve. 2 Thus it is a natural first step to study edge-DP graph clustering.

Very recently, Seif, Nguyen, Vullikanti, and Tandon [55] were the first to propose differentially private algorithms for the Stochastic Block Model, with edge-DP. Concretely, they propose algorithms achieving exact recovery (exact identification of the planted clustering) while preserving privacy of individual edges of the graph. The proposed approach either takes n Θ(log n ) time when ε is constant, or runs in polynomial time when ε is Ω(log n ) .

Gaussian Mixture Models have also been studied in the context of differential privacy by [32, 16, 60] using the subsample-and-aggregate framework first introduced in [52] (see also recent work for robust moment estimation in the differential privacy setting [35, 8, 29]). The works of [32, 16] require an explicit bound on the euclidean norm of the centers as the sample complexity of these algorithms depends on this bound. For a mixture of k Gaussians, if there is a non-private algorithm that requires the minimum distance between the centers to be at least ∆ , then [16, 60] can transform this non-private algorithm into a private one that needs the minimum distance between the centers to be at least ∆+ √ log n , where n is the number of samples.

In this paper, we tackle both clustering problems (graph clustering with the SBM and metric clustering with the GMM) through a new general privacy-preserving framework that brings us significantly closer to the state-of-the-art of non-private algorithms. As we will see, our new perspective on the problems appear to be easily extendable to many other estimation algorithms.

From robustness to privacy In recent years a large body of work (see [19, 22, 40, 18, 37, 14, 36, 30] and references therein) has advanced our understanding of parameter estimation in the presence of adversarial perturbations. In these settings, an adversary looks at the input instance and modifies it arbitrarily, under some constraints (these constraints are usually meant to ensure that it is still information theoretically possible to recover the underlying structure). As observed in the past [23, 35, 41], the two goals of designing privacy-preserving machine learning models and robust model estimation are tightly related. The common objective is to design algorithms that extract global information without over-relaying on individual data samples.

Concretely, robust parameter estimation tends to morally follow a two-steps process: (i) argue that typical inputs are well-behaved, in the sense that they satisfy some property which can be used to accurately infer the desired global information, (ii) show that adversarial perturbations cannot significantly alter the quality of well-behaved inputs, so that it is still possible to obtain an accurate estimate. Conceptually, the analysis of private estimation algorithms can also be divided in two parts: utility , which is concerned with the accuracy of the output, and privacy , which ensures there is no leak of sensitive information. In particular, the canonical differential privacy definition can be interpreted as the requirement that, for any distinct inputs Y, Y ′ , the change in the output is proportional to the distance 3 between Y and Y ′ .

It is easy to see this as a generalization of robustness: while robust algorithm needs the output to be stable for typical inputs, private algorithms requires this stability for any possible input. Then,

stability of the output immediately implies adding a small amount of noise to the output yields privacy. If the added noise is small enough, then utility is also preserved.

Our work further tightens this connection between robustness and privacy through a simple yet crucial insight: if two strongly convex functions over constrainted sets -where both the function and the set may depend on the input- are point-wise close (say in a ℓ 2 -sense), their minimizers are also close (in the same sense). The alternative perspective is that projections of points that are close to each other, onto convex sets that are point-wise close, must also be close. This observation subsumes previously known sensitivity bounds in the empirical risk minimization literature (in particular in the output-perturbation approach to ERM, see Section 2 for a comparison).

The result is a clean, user-friendly, framework to turn robust estimation algorithms into private algorithms , while keeping virtually the same guarantees. We apply this paradigm to stochastic block models and Gaussian mixture models, which we introduce next.

Stochastic block model The stochastic block model is an extensively studied statistical model for community detection in graphs (see [4] for a survey).

Model 1.1 (Stochastic block model) . In its most basic form, the stochastic block model describes the distribution 4 of an n -vertex graph G ∼ SBM n ( d, γ, x ) , where x is a vector of n binary 5 labels, d ∈ N , γ > 0 , and for every pair of distinct vertices i, j ∈ [ n ] the edge { i, j } is independently added to the graph G with probability (1 + γ · x i · x j ) d n .

̸

For balanced label vector x , i.e., with roughly the same number of +1 's and -1 's, parameter d roughly corresponds to the average degree of the graph. Parameter γ corresponds to the bias introduced by the community structure. Note that for distinct vertices i, j ∈ [ n ] , the edge { i, j } is present in G with probability (1 + γ ) d n if the vertices have the same label x i = x j and with probability (1 -γ ) d n if the vertices have different labels x i = x j . 6

Given a graph G sampled according to this model, the goal is to recover the (unknown) underlying vector of labels as well as possible. In particular, for a chosen algorithm returning a partition ˆ x ( G ) ∈ {± 1 } n , there are two main objective of interest: weak recovery and exact recovery . The former amounts to finding a partition ˆ x ( G ) correlated with the true partition. The latter instead corresponds to actually recovering the true partition with high probability. As shown in the following table, by now the statistical and computational landscape of these problems is well understood [17, 42, 48, 49, 28]:

Learning mixtures of spherical Gaussians The Gaussian Mixture Model we consider is the following.

̸

Model 1.2 (Mixtures of spherical Gaussians) . Let D 1 , . . . , D k be Gaussian distributions on R d with covariance Id and means µ 1 , . . . , µ k satisfying ∥ µ i -µ j ∥ ⩾ ∆ for any i = j . Given a set Y = { y 1 , . . . , y n } of n samples from the uniform mixture over D 1 , . . . , D k , estimate µ 1 , . . . , µ k .

√

It is known that when the minimum separation is ∆ = o ( log k ) , superpolynomially many samples are required to estimate the means up to small constant error [54]. Just above this threshold, at separation k O (1 /γ ) for any constant γ , there exist efficient algorithms based on the sum-ofsquares hierarchy recovering the means up to accuracy 1 / poly( k ) [30, 36, 59]. In the regime

where ∆ = O ( √ log k ) these algorithms yield the same guarantees but require quasipolynomial time. Recently, [39] showed how to efficiently recover the means as long as ∆ = O (log( k ) 1 / 2+ c ) for any constant c > 0 .

1.1 Results

Stochastic block model We present here the first ( ε, δ ) -differentially private efficient algorithms for exact recovery. In all our results on stochastic block models, we consider the edge privacy model, in which two input graphs are adjacent if they differ on a single edge (cf. Definition C.1).

Theorem 1.3 (Private exact recovery of SBM) . Let x ∈ {± 1 } n be balanced 7 . For any γ, d, ε, δ > 0 satisfying

there exists an ( ε, δ ) -differentially edge private algorithm that, on input G ∼ SBM n ( d, γ, x ) , returns ˆ x ( G ) ∈ { x, -x } with probability 1 -o (1) . Moreover, the algorithm runs in polynomial time.

For any constant ε > 0 , Theorem 1.3 states that ( ε, δ ) -differentially private exact recovery is possible, in polynomial time, already a constant factor close to the non-private threshold. Previous results [55] could only achieve comparable guarantees in time O ( n O (log n ) ) . It is also important to observe that the theorem provides a trade-off between signal-to-noise ratio of the instance (captured by the expression on the left-hand side with γ, d ) and the privacy parameter ε . In particular, we highlight two regimes: for d ⩾ Ω(log n ) one can achieve exact recovery with high probability and privacy parameters δ = n -Ω(1) , ε = O (1 /γ +1 /γ 2 ) . For d ⩾ ω (log n ) one can achieve exact recovery with high probability and privacy parameters ε = o (1) , δ = n -ω (1) . Theorem 1.3 follows by a result for private weak recovery and a boosting argument (cf. Theorem C.3 and Appendix C.2).

Further, we present a second, exponential-time, algorithm based on the exponential mechanism [43] which improves over the above in two regards. First, it gives pure differential privacy. Second, it provides utility guarantees for a larger range of graph parameters. In fact, we will also prove a lower bound which shows that its privacy guarantees are information theoretically optimal. 8 All hidden constants are absolute and do not depend on any graph or privacy parameters unless stated otherwise. In what follows we denote by err (ˆ x, x ) the minimum of the hamming distance of ˆ x and x , and the one of -ˆ x and x , divided by n .

Theorem 1.4 (Slightly informal, see Theorem C.18 in the supplements for full version) . Let γ √ d ⩾ Ω(1) , x ∈ {± 1 } n be balanced, and ζ ⩾ exp ( -Ω ( γ 2 d )) . For any ε ⩾ Ω ( log(1 /ζ ) γd ) , there exists an algorithm which on input G ∼ SBM n ( γ, d, x ) outputs an estimate ˆ x ( G ) ∈ {± 1 } n satisfying err (ˆ x ( G ) , x ) ⩽ ζ with probability at least 1 -ζ . In addition, the algorithm is ε -differentially edge private. Further, we can achieve error Θ ( 1 / √ log(1 /ζ ) ) with probability 1 -e -n .

Acouple of remarks are in order. First, our algorithm works across all degree-regimes in the literature and matches known non-private thresholds and rates up to constants. 9 In particular, for γ 2 d = Θ(1) , we achieve weak/partial recovery with either constant or exponentially high success probability. Recall that the optimal non-private threshold is γ 2 d > 1 . For the regime, where γ 2 d = ω (1) , it is known that the optimal error rate is exp( -(1 -o (1)) γ 2 d ) [64] even non-privately which we match up to constants - here o (1) denotes a function that tends to zero as γ 2 d tends to infinity. Moreover, our algorithm achieves exact recovery as soon as γ 2 d = Ω(log n ) since then ζ < 1 n . This also matches known non-private threshholds up to constants [5, 47]. We remark that [55] gave an ε -DP exponential time algorithm which achieved exact recovery and has inverse polynomial success probability in the utility case as long as ε ⩾ Ω( log n γd ) . We recover this result as a special case (with slightly worse constants). In fact, their algorithm is also based on the exponential mechanism, but their analysis only applies to the setting of exact recovery, while our result holds much more generally. Another

crucial difference is that we show how to privatize a known boosting technique frequently used in the non-private setting, allowing us to achieve error guarantees which are optimal up to constant factors.

It is natural to ask whether, for a given set of parameters γ, d, ζ one can obtain better privacy guarantees than Theorem 1.4. Our next result implies that our algorithmic guarantees are almost tight.

Theorem 1.5 (Informal, see Theorem C.22 in the supplements for full version) . Suppose there exists an ε -differentially edge private algorithm such that for any balanced x ∈ {± 1 } n , on input G ∼ SBM n ( d, γ, x ) , outputs ˆ x ( G ) ∈ {± 1 } n satisfying

This lower bound is tight for ε -DP exact recovery. By setting ζ = 1 /n and η = 1 / poly( n ) , Theorem 1.5 implies no ε -DP exact recovery algorithm exists for ε ⩽ O ( log n γd ) . There exist ε -DP algorithms (Algorithm C.19 in the supplements and the algorithm in [55]) exactly recover the community for any ε ⩾ Ω( log n γd ) .

Notice Theorem 1.5 is a lower bound for a large range of error rates (partial to exact recovery). For failure probability η = ζ , the lower bound simplifies to ε ⩾ Ω( log(1 /ζ ) γd ) and hence matches Theorem 1.4 up to constants. For exponentially small failure probability, η = e -n , it becomes ε ⩾ Ω( 1 ζγd ) . To compare, Theorem 1.4 requires ε ⩾ Ω( 1 ζ 2 γd ) in this regime, using the substitution √ log(1 /ζ ) → ζ .

Further, while formally incomparable, this ε -DP lower bound also suggests that the guarantees obtained by our efficient ( ε, δ ) -DP algorithm in Theorem 1.3 might be close to optimal. Note that setting ζ = 1 n in Theorem 1.5 requires the algorithm to exactly recover the partitioning. In this setting, Theorem 1.3 implies that there is an efficient ( ε, n -Θ(1) ) -DP exact recovery algorithm for ε ⩽ O ( √ log n γd ) . Theorem 1.5 states any ε -DP exact recovery algorithm requires ε ⩾ Ω( log n γd ) . Further, for standard privacy parameters that are required for real-world applications, such as ε ≈ 1 and δ = n -10 , Theorem 1.3 requires that γd ⩾ Ω(log n ) . Theorem 1.5 shows that for pure-DP algorithms with the same setting of ε this is also necessary. We leave it as fascinating open questions to bridge the gap between upper and lower bounds in the context of ( ε, δ ) -DP.

Learning mixtures of spherical Gaussians Our algorithm for privately learning mixtures of k spherical Gaussians provides statistical guarantees matching those of the best known non-private algorithms.

Theorem 1.6 (Privately learning mixtures of spherical Gaussians) . Consider an instance of Model 1.2. Let t > 0 be such that ∆ ⩾ O ( √ tk 1 /t ) . For n ⩾ Ω ( k O (1) · d O ( t ) ) , k ⩾ (log n ) 1 / 5 , there exists an algorithm, running in time ( nd ) O ( t ) , that outputs vectors ˆ µ 1 , . . . , ˆ µ k satisfying

with high probability, for some permutation π : [ k ] → [ k ] . Moreover, for ε ⩾ k -10 , δ ⩾ n -10 , the algorithm is ( ε, δ ) -differentially private 10 for any input Y .

The conditions ε ⩾ k -10 , δ ⩾ n -10 in Theorem 1.6 are not restrictive and should be considered a formality. Moreover, setting ε = 0 . 01 and δ = n -10 already provides meaningful privacy guarantees in practice. The condition that k ⩾ (log n ) 1 / 5 is a technical requirement by our proofs.

Prior to this work, known differentially private algorithms could learn a mixture of k -spherical Gaussian either if: (1) they were given a ball of radius R containing all centers [32, 16]; 11 or (2) the minimum separation between centers needs an additional additive Ω( √ log n ) term [16, 60] 12 .

Then,

To the best of our knowledge, Theorem 1.6 is the first to get the best of both worlds. That is, our algorithm requires no explicit upper bounds on the means (this also means the sample complexity does not depend on R ) and only minimal separation assumption O ( √ log k ) . Furthermore, we remark that while previous results only focused on mixtures of Gaussians, our algorithm also works for the significantly more general class of mixtures of Poincaré distributions. Concretely, in the regime k ⩾ √ log d , our algorithm recovers the state-of-the-art guarantees provided by non-private algorithms which are based on the sum-of-squares hierarchy [36, 30, 59]: 13

If ∆ ⩾ k 1 /t ∗ for some constant t ∗ , then by choosing t ⩾ Ω( t ∗ ) the algorithm recovers the centers, up to a 1 / poly( k ) error, in time poly( k, d ) and using only poly( k, d ) samples. √

If ∆ ⩾ Ω( log k ) then choosing t = O (log k ) the algorithm recovers the centers, up to a 1 / poly( k ) error, in quasi-polynomial time poly( k O ( t ) , d O ( t 2 ) ) and using a quasipolynomial number of samples poly( k, d O ( t ) ) .

For simplicity of exposition we will limit the presentation to mixtures of spherical Gaussians. We reiterate that separation Ω( √ log k ) is information-theoretically necessary for algorithms with polynomial sample complexity [54].

Subsequently and independently of our work, the work of [6] gives an algorithm that turns any non-private GMM learner into a private one based on the subsample and aggregate framework. They apply this reduction to the classical result of [45] to give the first finite-sample ( ε, δ ) -DP algorithm that learns mixtures of unbounded Gaussians, in particular, the covariance matrices of their mixture components can be arbitrary.

2 Techniques

We present here our general tools for designing efficient private estimation algorithms in the highdimensional setting whose statistical guarantees almost match those of the best know non-private algorithms. The algorithms we design have the following structure in common: First, we solve a convex optimization problem with constraints and objective function depending on our input Y . Second, we round the optimal solution computed in the first step to a solution X for the statistical estimation problem at hand.

We organize our privacy analyses according to this structure. In order to analyze the first step, we prove a simple sensitivity bound for strongly convex optimization problems, which bounds the ℓ 2 -sensitivity of the optimal solution in terms of a uniform sensitivity bound for the objective function and the feasible region of the optimization problem.

For bounded problems -such as recovery of stochastic block models- we use this sensitivity bound, in the second step, to show that introducing small additive noise to standard rounding algorithms is enough to achieve privacy.

For unbounded problems -such as learning GMMs- we use this sensitivity bound to show that on adjacent inputs, either most entries of X only change slightly, as in the bounded case, or few entries vary significantly. We then combine different privacy techniques to hide both type of changes.

Privacy from sensitivity of strongly convex optimization problems Before illustrating our techniques with some examples, it is instructive to explicit our framework. Here we have a set of inputs Y and a family of strongly convex functions F ( Y ) and convex sets K ( Y ) parametrized by these inputs. The generic non-private algorithm based on convex optimization we consider works as follows:

Compute ˆ X := argmin X ∈K ( Y ) f Y ( X ) ;

Round ˆ X into an integral solution.

For an estimation problem, a distributional assumption on Y is made. Then one shows how, for typical inputs Y sampled according to that distribution, the above scheme recovers the desired structured information.

We can provide a privatized version of this scheme by arguing that, under reasonable assumptions on F ( Y ) and K ( Y ) , the output of the function argmin X ∈K ( Y ) f Y ( X ) has low ℓ 2 -sensitivity. The consequence of this crucial observation is that one can combine the rounding step 2 with some standard privacy mechanism and achieve differential privacy. That is, the second step becomes:

2. Add random noise N and round ˆ X + N into an integral solution.

Our sensitivity bound is simple, yet it generalizes previously known bounds for strongly convex optimization problems (we provide a detailed comparison later in the section). For adjacent Y, Y ′ ∈ Y , it requires the following properties:

(i) For each X ∈ K ( Y ) ∩ K ( Y ′ ) it holds | f Y ( X ) -f Y ′ ( X ) | ⩽ α ;

(ii) For each X ∈ K ( Y ) its projection Z onto K ( Y ) ∩K ( Y ′ ) satisfies | f Y ( X ) -f Y ′ ( Z ) | ⩽ α.

Here we think of α as some small quantity (relatively to the problem parameters). Notice, we may think of (i) as Lipschitz-continuity of the function g ( Y, X ) = f Y ( X ) with respect to Y and of (ii) as a bound on the change of the constrained set on adjacent inputs. In fact, these assumptions are enough to conclude low ℓ 2 sensitivity. Let ˆ X and ˆ X ′ be the outputs of the first step on inputs Y, Y ′ . Then using (i) and (ii) above and the fact that ˆ X is an optimizer, we can show that there exists Z ∈ K ( Y ) ∩ K ( Y ′ ) such that

By κ -strong convexity of f Y , f Y ′ this implies

which ultimately means ∥ ˆ X -ˆ X ′ ∥ 2 2 ⩽ O ( α/κ ) (see Lemma B.1). Thus, starting from our assumptions on the point-wise distance of f Y , f Y ′ we were able to conclude low ℓ 2 -sensitivity of our output!

A simple application: weak recovery of stochastic block models The ideas introduced above, combined with existing algorithms for weak recovery of stochastic block models, immediately imply a private algorithm for the problem. To illustrate this, consider Model 1.1 with parameters γ 2 d ⩾ C , for some large enough constant C > 1 . Let x ∈ {± 1 } n be balanced. Here Y is an n -byn matrix corresponding to the rescaled centered adjacency matrix of the input graph:

The basic semidefinite program [28, 46] can be recast 14 as the strongly convex constrained optimization question of finding the orthogonal projection of the matrix Y onto the set K := { X ∈ R n × n ∣ ∣ X ⪰ 0 , X ii = 1 n ∀ i } . That is

Let f Y ( X ) := ∥ X ∥ 2 F -2 ⟨ X,Y ⟩ and notice that ˆ X = argmin X ∈K f Y ( X ) . It is a standard fact that, if our input was G ∼ SBM n ( d, γ, x ) , then with high probability X ( G ) = argmin X ∈K f Y ( G ) ( X ) would have leading eigenvalue-eigenvector pair satisfying

This problem fits perfectly the description of the previous paragraph. Note that since the constraint set does not depend on Y , Property (ii) reduces to Property (i). Thus, it stands to reason that the projections ˆ X, ˆ X ′ of Y, Y ′ are close whenever the input graphs generating Y and Y ′ are adjacent. By Hölder's Inequality with the entry-wise infinity and ℓ 1 -norm, we obtain | f Y ( X ) -f Y ′ ( X ) | ⩽ 2 ∥ X ∥ ∞ ∥ Y -Y ′ ∥ 1 . By standard facts about positive semidefinite matrices, we have ∥ X ∥ ∞ ⩽ 1 n

for all X ∈ K . Also, Y and Y ′ can differ on at most 2 entries and hence ∥ Y -Y ′ ∥ 1 ⩽ O ( 1 γd ) . Thus, ∥ ˆ X -ˆ X ′ ∥ 2 F ⩽ O ( 1 nγd ) .

The rounding step is now straightforward. Using the Gaussian mechanism we return the leading eigenvector of ˆ X + N where N ∼ N (0 , 1 nγd · log(1 /δ ) ε 2 ) n × n . This matrix has Frobeinus norm significantly larger than ˆ X but its spectral norm is only

Thus by standard linear algebra, for typical instances G ∼ SBM n ( d, γ, x ) , the leading eigenvector of ˆ X ( G ) + N will be highly correlated with the true community vector x whenever the average degree d is large enough. In conclusion, a simple randomized rounding step is enough!

Remark 2.1 (From weak recovery to exact recovery) . In the non-private setting, given a weak recovery algorithm for the stochastic block model, one can use this as an initial estimate for a boosting procedure based on majority voting to achieve exact recovery. We show that this can be done privately. See Appendix C.2 in the supplements.

An advanced application: learning mixtures of Gaussians In the context of SBMs our argument greatly benefited from two key properties: first, on adjacent inputs Y -Y ′ was bounded in an appropriate norm; and second, the convex set K was fixed. In the context of learning mixtures of spherical Gaussians as in Model 1.2, both this properties are not satisfied (notice how one of this second properties would be satisfied assuming bounded centers!). So additional ideas are required.

The first observation, useful to overcome the first obstacle, is that before finding the centers, one can first find the n -byn membership matrix W ( Y ) where W ( Y ) ij = 1 if i, j where sampled from the same mixture component and 0 otherwise. The advantage here is that, on adjacent inputs, ∥ W ( Y ) -W ( Y ′ ) ∥ 2 F ⩽ 2 n/k and thus one recovers the first property. 15 Here early sum-of-squares algorithms for the problem [30, 36] turns out to be convenient as they rely on minimizing the function ∥ W ∥ 2 F subject to the following system of polynomial inequalities in variables z 11 , . . . , z 1 k , . . . , z nk , with W ij = ∑ ℓ z iℓ z jℓ for all i, j ∈ [ n ] and a parameter t > 0 .

For the scope of this discussion, 17 we may disregard computational issues and assume we have access to an algorithm returning a point from the convex hull K ( Y ) of all solutions to our system of inequalities. 18 Each indicator variable z iℓ ∈ { 0 , 1 } is meant to indicate whether sample y i is believed

to be in cluster C ℓ . In the non-private setting, the idea behind the program is that -for typical Y sampled according to Model 1.2 with minimum separation ∆ ⩾ k 1 /t √ t - any solution W ( Y ) ∈ K ( Y ) is close to the ground truth matrix W ∗ ( Y ) in Frobenius norm: ∥ W ( Y ) -W ∗ ( Y ) ∥ 2 F ⩽ 1 / poly( k ) . Each row W ( Y ) i may be seen as inducing a uniform distribution over a subset of Y . 19 Combining the above bound with the fact that subgaussian distributions at small total variation distance have means that are close, we conclude the algorithm recovers the centers of the mixture.

While this program suggests a path to recover the first property, it also possesses a fatal flaw: the projection W ′ of W ∈ K ( Y ) onto K ( Y ) ∩ K ( Y ′ ) may be far in the sense that |∥ W ∥ 2 F -∥ W ′ ∥ 2 F | ⩾ Ω( ∥ W ∥ 2 F + ∥ W ′ ∥ 2 F ) ⩾ Ω( n 2 /k ) . The reason behind this phenomenon can be found in the constraint ∑ i z iℓ = n/k . The set indicated by the vector ( z 1 ℓ . . . , z nℓ ) may be subgaussian in the sense of P ( Y ) for input Y but, upon changing a single sample, this may no longer be true. We work around this obstacle in two steps:

We replace the above constraint with ∑ i z iℓ ⩽ n/k .

We compute ˆ W := argmin W solving P ( Y ) ∥ J -W ∥ 2 F , where J is the all-one matrix. 20

The catch now is that the program is satisfiable for any input Y since we can set z il = 0 whenever necessary. Moreover, we can guarantee property (ii) (required by our sensitivity argument) for α ⩽ O ( n/k ) , since we can obtain W ′ ∈ K ( Y ) ∩ K ( Y ′ ) simply zeroing out the row/column in W corresponding to the sample differing in Y and Y ′ . Then for typical inputs Y , the correlation with the true solution is guaranteed by the new strongly convex objective function.

We offer some more intuition on the choice of our objective function: Recall that W ij indicates our guess whether the i -th and j -th datapoints are sampled from the same Gaussian component. A necessary condition for W to be close to its ground-truth counterpart W ∗ , is that they roughly have the same number of entries that are (close to) 1. One way to achieve this would be to add the lower bound constraing ∑ ℓ z iℓ ≳ n k . However, such a constraint could cause privacy issues: There would be two neighboring datasets, such that the constraint set induced by one dataset is satisfiable, but the constraint set induced by the other dataset is not satisfiable. We avoid this issue by noticing that the appropriate number of entries close to 1 can also be induced by minimizing the distance of W to the all-one matrix. This step is also a key difference from [35], explained in more detail below.

From low sensitivity of the indicators to low sensitivity of the estimates For adjacent inputs Y, Y ′ let ˆ W, ˆ W ′ be respectively the matrices computed by the above strongly convex programs. Our discussion implies that, applying our sensitivity bound, we can show ∥ ˆ W -ˆ W ′ ∥ 2 F ⩽ O ( n/k ) . The problem is that simply applying a randomized rounding approach here cannot work. The reason is that even tough the vector ˆ W i induces a subgaussian distribution, the vector ˆ W i + v for v ∈ R n , might not . Without the subgaussian constraint we cannot provide any meaningful utility bound. In other words, the root of our problem is that there exists heavy-tailed distributions that are arbitrarily close in total variation distance to any given subgaussian distribution.

On the other hand, our sensitivity bound implies ∥ ˆ W -ˆ W ′ ∥ 2 1 ⩽ o ( ∥ ˆ W ∥ 1 ) and thus, all but a vanishing fraction of rows i ∈ [ n ] must satisfy ∥ ˆ W i -ˆ W ′ i ∥ 1 ⩽ o ( ∥ ˆ W i ∥ 1 ) . For each row i , let µ i , µ ′ i be the means of the distributions induced respectively by ˆ W i , ˆ W ′ i . We are thus in the following setting:

For a set of (1 -o (1)) · n good rows ∥ µ i -µ ′ i ∥ 2 ⩽ o (1) ,

For the set B of remaining bad rows, the distance ∥ µ i -µ ′ i ∥ 2 may be unbounded.

We hide differences of the first type as follows: pick a random subsample S of [ n ] of size n c , for some small c > 0 , and for each picked row use the Gaussian mechanism. The subsampling step is useful as it allows us to decrease the standard deviation of the entry-wise random noise by a factor n 1 -c . We hide differences of the second type as follows: Note that most of the rows are clustered together in space. Hence, we aim to privately identify the regions which contain many of the rows.

Formally, we use a classic high dimensional ( ε, δ ) -differentially private histogram learner on S and for the k largest bins of highest count privately return their average (cf. Lemma A.13). The crux of the argument here is that the cardinality of B ∩ S is sufficiently small that the privacy guarantees of the histogram learner can be extended even for inputs that differ in |B ∩ S | many samples. Finally, standard composition arguments will guarantee privacy of the whole algorithm.

Comparison with the framework of Kothari-Manurangsi-Velingker Both Kothari-ManurangsiVelingker [35] and our work obtained private algorithms for high-dimensional statistical estimation problems by privatizing strongly convex programs, more specifically, sum-of-squares (SoS) programs. The main difference between KMV and our work lies in how we choose the SoS program. For the problem of robust moment estimation, KMV considered the canonical SoS program from [30, 36] which contains a minimum cardinality constraint (e.g., ∑ l z il ≳ n k in the case of GMMs). Such a constraint is used to ensure good utility. However, as alluded to earlier, this is problematic for privacy: there will always exist two adjacent input datasets such that the constraints are satisfiable for one but not for the other. KMV and us resolve this privacy issue in different ways.

KMVuses an exponential mechanism to pick the lower bound of the minimum cardinality constraint. This step also ensures that solutions to the resulting SoS program will have low sensitivity. In contrast, we simply drop the minimum cardinality constraint. Then the resulting SoS program is always feasible for any input dataset! To still ensure good utility, we additionally pick an appropriate objective function. For example, in Gaussian mixture models, we chose the objective ∥ W -J ∥ 2 F . Our approach has the following advantages: First, the exponential mechanism in KMV requires computing O ( n ) scores. Computing each score requires solving a large semidefinite program, which can significantly increase the running time. Second, proving that the exponential mechanism in KMV works requires several steps: 1) defining a (clever) score function, 2) bounding the sensitivity of this score function and, 3) showing existence of a large range of parameters with high score. Our approach bypasses both of these issues.

Further, as we show, our general recipe can be easily extended to other high dimensional problems of interest: construct a strongly convex optimization program and add noise to its solution. This can provide significant computational improvements. For example, in the context of SBMs, the framework of [35] would require one to sample from an exponential distribution over matrices. Constructing and sampling from such distributions is an expensive operation. However, it is well-understood that an optimal fractional solution to the basic SDP relaxation we consider can be found in near quadratic time using the standard matrix multiplicative weight method [7, 58], making the whole algorithm run in near-quadratic time. Whether our algorithm can be sped up to near-linear time, as in [7, 58], remains a fascinating open question.

Comparison with previous works on empirical risk minimization Results along the lines of the sensitivity bound described at the beginning of the section (see Lemma B.1 for a formal statement) have been extensively used in the context of empirical risk minimization [15, 34, 57, 11, 63, 50]. Most results focus on the special case of unconstrained optimization of strongly convex functions. In contrast, our sensitivity bound applies to the significantly more general settings where both the objective functions and the constrained set may depend on the input. 21 Most notably for our settings of interest, [15] studied unconstrained optimization of (smooth) strongly convex functions depending on the input, with bounded gradient. We recover such a result for X ′ = X in (ii) . In [50], the authors considered constraint optimization of objective functions where the domain (but not the function) may depend on the input data. They showed how one can achieve differential privacy while optimize the desired objective function by randomly perturbing the constraints. It is important to remark that, in [50], the notion of utility is based on the optimization problem (and their guarantees are tight only up to logarithmic factors). In the settings we consider, even in the special case where f does not depend on the input, this notion of utility may not correspond to the notion of utility required by the estimation problem, and thus, the corresponding guarantees can turn out to be too loose to ensure the desired error bounds.

Acknowledgments and Disclosure of Funding

This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No 815464).

References

Tackling urban mobility with technology. https://europe.googleblog.com/2015/11/ tackling-urban-mobility-with-technology.html , 2015. Accessed: 2022-11-06.

Learning with privacy at scale. https://docs-assets.developer.apple.com/ ml-research/papers/learning-with-privacy-at-scale.pdf , 2017. Accessed: 202211-06.

Disclosure avoidance for the 2020 census: An introduction. https: //www2.census.gov/library/publications/decennial/2020/ 2020-census-disclosure-avoidance-handbook.pdf , 2021. Accessed: 2022-1106.

Emmanuel Abbe. Community detection and stochastic block models: recent developments. The Journal of Machine Learning Research , 18(1):6446-6531, 2017.

Emmanuel Abbe, Afonso S Bandeira, and Georgina Hall. Exact recovery in the stochastic block model. IEEE Transactions on information theory , 62(1):471-487, 2015.

Jamil Arbas, Hassan Ashtiani, and Christopher Liaw. Polynomial time and private learning of unbounded gaussian mixture models. arXiv preprint arXiv:2303.04288 , 2023.

Sanjeev Arora and Satyen Kale. A combinatorial, primal-dual approach to semidefinite programs. In Proceedings of the thirty-ninth annual ACM symposium on Theory of computing , pages 227-236, 2007.

Hassan Ashtiani and Christopher Liaw. Private and polynomial time algorithms for learning gaussians and beyond. In Po-Ling Loh and Maxim Raginsky, editors, Proceedings of Thirty Fifth Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 1075-1076. PMLR, 02-05 Jul 2022.

Ainesh Bakshi, Ilias Diakonikolas, Samuel B. Hopkins, Daniel Kane, Sushrut Karmalkar, and Pravesh K. Kothari. Outlier-robust clustering of gaussians and other non-spherical mixtures. In Sandy Irani, editor, 61st IEEE Annual Symposium on Foundations of Computer Science, FOCS 2020, Durham, NC, USA, November 16-19, 2020 , pages 149-159. IEEE, 2020.

Ainesh Bakshi, Ilias Diakonikolas, He Jia, Daniel M. Kane, Pravesh K. Kothari, and Santosh S. Vempala. Robustly learning mixtures of k arbitrary gaussians. In Stefano Leonardi and Anupam Gupta, editors, STOC '22: 54th Annual ACM SIGACT Symposium on Theory of Computing, Rome, Italy, June 20 - 24, 2022 , pages 1234-1247. ACM, 2022.

Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 2014 IEEE 55th annual symposium on foundations of computer science , pages 464-473. IEEE, 2014.

Christian Borgs, Jennifer Chayes, and Adam Smith. Private graphon estimation for sparse graphs. Advances in Neural Information Processing Systems , 28, 2015.

Christian Borgs, Jennifer Chayes, Adam Smith, and Ilias Zadik. Revealing network structure, confidentially: Improved rates for node-private graphon estimation. In 2018 IEEE 59th Annual Symposium on Foundations of Computer Science (FOCS) , pages 533-543. IEEE, 2018.

Moses Charikar, Jacob Steinhardt, and Gregory Valiant. Learning from untrusted data. In Proceedings of the 49th Annual ACM SIGACT Symposium on Theory of Computing , pages 47-60, 2017.

Kamalika Chaudhuri, Claire Monteleoni, and Anand D Sarwate. Differentially private empirical risk minimization. Journal of Machine Learning Research , 12(3), 2011.

Edith Cohen, Haim Kaplan, Yishay Mansour, Uri Stemmer, and Eliad Tsfadia. Differentiallyprivate clustering of easy instances. In International Conference on Machine Learning , pages 2049-2059. PMLR, 2021.

Aurelien Decelle, Florent Krzakala, Cristopher Moore, and Lenka Zdeborová. Asymptotic analysis of the stochastic block model for modular networks and its algorithmic applications. Physical Review E , 84(6):066106, 2011.

Ilias Diakonikolas, Gautam Kamath, Daniel Kane, Jerry Li, Ankur Moitra, and Alistair Stewart. Robust estimators in high-dimensions without the computational intractability. SIAM Journal on Computing , 2019.

Ilias Diakonikolas and Daniel M Kane. Recent advances in algorithmic high-dimensional robust statistics. arXiv preprint arXiv:1911.05911 , 2019.

Ilias Diakonikolas, Daniel M. Kane, Sushrut Karmalkar, Ankit Pensia, and Thanasis Pittas. Robust sparse mean estimation via sum of squares. In Po-Ling Loh and Maxim Raginsky, editors, Conference on Learning Theory, 2-5 July 2022, London, UK , volume 178 of Proceedings of Machine Learning Research , pages 4703-4763. PMLR, 2022.

Jingqiu Ding, Tommaso d'Orsi, Rajai Nasser, and David Steurer. Robust recovery for stochastic block models. In 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS) , pages 387-394. IEEE, 2022.

Tommaso d'Orsi, Pravesh K. Kothari, Gleb Novikov, and David Steurer. Sparse PCA: algorithms, adversarial perturbations and certificates. In Sandy Irani, editor, 61st IEEE Annual Symposium on Foundations of Computer Science, FOCS 2020, Durham, NC, USA, November 16-19, 2020 , pages 553-564. IEEE, 2020.

Cynthia Dwork and Jing Lei. Differential privacy and robust statistics. In STOC'09Proceedings of the 2009 ACM International Symposium on Theory of Computing , pages 371-380. ACM, New York, 2009.

Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam D. Smith. Calibrating noise to sensitivity in private data analysis. In Shai Halevi and Tal Rabin, editors, Theory of Cryptography, Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006, Proceedings , volume 3876 of Lecture Notes in Computer Science , pages 265-284. Springer, 2006.

Yingjie Fei and Yudong Chen. Achieving the Bayes error rate in synchronization and block models by SDP, robustly. IEEE Trans. Inform. Theory , 66(6):3929-3953, 2020.

Noah Fleming, Pravesh Kothari, Toniann Pitassi, et al. Semialgebraic proofs and efficient algorithm design. Foundations and Trends® in Theoretical Computer Science , 14(1-2):1-221, 2019.

M. Grötschel, L. Lovász, and A. Schrijver. The ellipsoid method and its consequences in combinatorial optimization. Combinatorica , 1(2):169-197, 1981.

Olivier Guédon and Roman Vershynin. Community detection in sparse networks via Grothendieck's inequality. Probab. Theory Related Fields , 165(3-4):1025-1049, 2016.

Samuel B Hopkins, Gautam Kamath, and Mahbod Majid. Efficient mean estimation with pure differential privacy via a sum-of-squares exponential mechanism. In Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing , pages 1406-1417, 2022.

Samuel B. Hopkins and Jerry Li. Mixture models, robustness, and sum of squares proofs. In Ilias Diakonikolas, David Kempe, and Monika Henzinger, editors, Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, STOC 2018, Los Angeles, CA, USA, June 25-29, 2018 , pages 1021-1034. ACM, 2018.

William B Johnson. Extensions of lipschitz mappings into a hilbert space. Contemp. Math. , 26:189-206, 1984.

Gautam Kamath, Or Sheffet, Vikrant Singhal, and Jonathan Ullman. Differentially private algorithms for learning mixtures of separated gaussians. Advances in Neural Information Processing Systems , 32, 2019.

Vishesh Karwa and Salil P. Vadhan. Finite sample differentially private confidence intervals. In Anna R. Karlin, editor, 9th Innovations in Theoretical Computer Science Conference, ITCS 2018, January 11-14, 2018, Cambridge, MA, USA , volume 94 of LIPIcs , pages 44:1-44:9. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2018.

Daniel Kifer, Adam Smith, and Abhradeep Thakurta. Private convex empirical risk minimization and high-dimensional regression. In Conference on Learning Theory , pages 25-1. JMLR Workshop and Conference Proceedings, 2012.

Pravesh Kothari, Pasin Manurangsi, and Ameya Velingker. Private robust estimation by stabilizing convex relaxations. In Po-Ling Loh and Maxim Raginsky, editors, Conference on Learning Theory, 2-5 July 2022, London, UK , volume 178 of Proceedings of Machine Learning Research , pages 723-777. PMLR, 2022.

Pravesh K. Kothari, Jacob Steinhardt, and David Steurer. Robust moment estimation and improved clustering via sum of squares. In Ilias Diakonikolas, David Kempe, and Monika Henzinger, editors, Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, STOC 2018, Los Angeles, CA, USA, June 25-29, 2018 , pages 1035-1046. ACM, 2018.

Kevin A. Lai, Anup B. Rao, and Santosh S. Vempala. Agnostic estimation of mean and covariance. In Irit Dinur, editor, IEEE 57th Annual Symposium on Foundations of Computer Science, FOCS 2016, 9-11 October 2016, Hyatt Regency, New Brunswick, New Jersey, USA , pages 665-674. IEEE Computer Society, 2016.

Jean B. Lasserre. New positive semidefinite relaxations for nonconvex quadratic programs. In Advances in convex analysis and global optimization (Pythagorion, 2000) , volume 54 of Nonconvex Optim. Appl. , pages 319-331. Kluwer Acad. Publ., Dordrecht, 2001.

Allen Liu and Jerry Li. Clustering mixtures with almost optimal separation in polynomial time. In Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing , pages 1248-1261, 2022.

Allen Liu and Ankur Moitra. Minimax rates for robust community detection. CoRR , abs/2207.11903, 2022.

Xiyang Liu, Weihao Kong, and Sewoong Oh. Differential privacy and robust statistics in high dimensions. In Conference on Learning Theory , pages 1167-1246. PMLR, 2022.

Laurent Massoulié. Community detection thresholds and the weak ramanujan property. In Proceedings of the forty-sixth annual ACM symposium on Theory of computing , pages 694-703, 2014.

Frank McSherry and Kunal Talwar. Mechanism design via differential privacy. In 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS'07) , pages 94-103. IEEE, 2007.

Ankur Moitra, William Perry, and Alexander S Wein. How robust are reconstruction thresholds for community detection? In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing , pages 828-841, 2016.

Ankur Moitra and Gregory Valiant. Settling the polynomial learnability of mixtures of gaussians. In 2010 IEEE 51st Annual Symposium on Foundations of Computer Science , pages 93-102. IEEE, 2010.

Andrea Montanari and Subhabrata Sen. Semidefinite programs on sparse random graphs and their application to community detection. In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing , pages 814-827, 2016.

Elchanan Mossel, Joe Neeman, and Allan Sly. Consistency thresholds for the planted bisection model. In Proceedings of the forty-seventh annual ACM symposium on Theory of computing , pages 69-75, 2015.

Elchanan Mossel, Joe Neeman, and Allan Sly. Reconstruction and estimation in the planted partition model. Probability Theory and Related Fields , 162(3):431-461, 2015.

Elchanan Mossel, Joe Neeman, and Allan Sly. A proof of the block model threshold conjecture. Combinatorica , 38(3):665-708, 2018.

Andres Munoz, Umar Syed, Sergei Vassilvtiskii, and Ellen Vitercik. Private optimization without constraint violations. In International Conference on Artificial Intelligence and Statistics , pages 2557-2565. PMLR, 2021.

Yurii Nesterov. Squared functional systems and optimization problems. In High performance optimization , volume 33 of Appl. Optim. , pages 405-440. Kluwer Acad. Publ., Dordrecht, 2000.

Kobbi Nissim, Sofya Raskhodnikova, and Adam Smith. Smooth sensitivity and sampling in private data analysis. In STOC'07-Proceedings of the 39th Annual ACM Symposium on Theory of Computing , pages 75-84. ACM, New York, 2007.

Pablo A Parrilo. Structured semidefinite programs and semialgebraic geometry methods in robustness and optimization . PhD thesis, California Institute of Technology, 2000.

Oded Regev and Aravindan Vijayaraghavan. On learning mixtures of well-separated gaussians. In 2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS) , pages 85-96. IEEE, 2017.

Mohamed M. Seif, Dung Nguyen, Anil Vullikanti, and Ravi Tandon. Differentially private community detection for stochastic block models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 15858-15894. PMLR, 2022.

N. Z. Shor. Quadratic optimization problems. Izv. Akad. Nauk SSSR Tekhn. Kibernet. , (1):128139, 222, 1987.

Shuang Song, Kamalika Chaudhuri, and Anand D Sarwate. Stochastic gradient descent with differentially private updates. In 2013 IEEE global conference on signal and information processing , pages 245-248. IEEE, 2013.

David Steurer. Fast sdp algorithms for constraint satisfaction problems. In Proceedings of the twenty-first annual ACM-SIAM symposium on Discrete Algorithms , pages 684-697. SIAM, 2010.

David Steurer and Stefan Tiegel. Sos degree reduction with applications to clustering and robust moment estimation. In Proceedings of the 2021 ACM-SIAM Symposium on Discrete Algorithms (SODA) , pages 374-393. SIAM, 2021.

Eliad Tsfadia, Edith Cohen, Haim Kaplan, Yishay Mansour, and Uri Stemmer. Friendlycore: Practical differentially private aggregation. In International Conference on Machine Learning , pages 21828-21863. PMLR, 2022.

Jonathan Ullman and Adam Sealfon. Efficiently estimating erdos-renyi graphs with node differential privacy. Advances in Neural Information Processing Systems , 32, 2019.

Martin J. Wainwright. High-Dimensional Statistics: A Non-Asymptotic Viewpoint . Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2019.

Di Wang, Minwei Ye, and Jinhui Xu. Differentially private empirical risk minimization revisited: Faster and more general. Advances in Neural Information Processing Systems , 30, 2017.

Anderson Y Zhang and Harrison H Zhou. Minimax rates of community detection in stochastic block models. The Annals of Statistics , 44(5):2252-2280, 2016.

A Preliminaries

We use boldface characters for random variables. We hide multiplicative factors logarithmic in n using the notation ˜ O ( · ) , ˜ Ω( · ) . Similarly, we hide absolute constant multiplicative factors using the standard notation O ( · ) , Ω( · ) , Θ( · ) . Often times we use the letter C do denote universal constants independent of the parameters at play. We write o (1) , ω (1) for functions tending to zero (resp. infinity) as n grows. We say that an event happens with high probability if this probability is at least 1 -o (1) . Throughout the paper, when we say "an algorithm runs in time O ( q ) " we mean that the number of basic arithmetic operations involved is O ( q ) . That is, we ignore bit complexity issues.

Vectors, matrices, tensors We use Id n to denote the n -byn dimensional identity matrix, J n ∈ R n × n the all-ones matrix and 0 n , 1 n ∈ R n to denote respectively the zero and the all-ones vectors. When the context is clear we drop the subscript. For matrices A,B ∈ R n × n we write A ⪰ B if A -B is positive semidefinite. For a matrix M , we denote its eigenvalues by λ 1 ( M ) , . . . , λ n ( M ) , we simply write λ i when the context is clear. We denote by ∥ M ∥ the spectral norm of M . We denote by R d ⊗ t the set of real-valued ordert tensors. for a d × d matrix M , we denote by M ⊗ t the t -fold Kronecker product M ⊗ M ⊗··· ⊗ M ︸ ︷︷ ︸ t times . We define the flattening , or vectorization , of M to

̸

be the d t -dimensional vector, whose entries are the entries of M appearing in lexicographic order. With a slight abuse of notation we refer to this flattening with M , ambiguities will be clarified form context. We denote by N ( 0 , σ 2 ) d ⊗ t the distribution over Gaussian tensors with d t entries with standard deviation σ . Given u, v ∈ {± 1 } n , we use Ham( u, v ) := ∑ n i =1 ✶ [ u i = v i ] to denote their Hamming distance. Given a vector u ∈ R n , we let sign( u ) ∈ {± 1 } n denote its sign vector. A vector u ∈ {± 1 } n is said to be balanced if ∑ n i =1 u i = 0 .

Graphs We consider graphs on n vertices and let G n be the set of all graphs on n vertices. For a graph G on n vertices we denote by A ( G ) ∈ R n × n its adjacency matrix. When the context is clear we simply write A. Let V ( G ) (resp. E ( G ) ) denote the vertex (resp. edge) set of graph G . Given two graphs G,H on the same vertex set V , let G \ H := ( V, E ( G ) \ H ( G )) . Given a graph H , H ′ ⊆ H means H ′ is a subgraph of H such that V ( H ′ ) = V ( H ) and E ( H ) ⊆ E ( H ) . The Hamming distance between two graphs G,H is defined to be the size of the symmetric difference between their edge sets, i.e. Ham( G,H ) := | E ( G ) △ E ( H ) | .

A.1 Differential privacy

In this section we introduce standard notions of differential privacy [24].

Definition A.1 (Differential privacy) . An algorithm M : Y → O is said to be ( ε, δ ) -differentially private for ε, δ > 0 if and only if, for every S ⊆ O and every neighboring datasets Y, Y ′ ∈ Y we have

To avoid confusion, for each problem we will exactly state the relevant notion of neighboring datasets. Differential privacy is closed under post-processing and composition.

Lemma A.2 (Post-processing) . If M : Y → O is an ( ε, δ ) -differentially private algorithm and M ′ : Y → Z is any randomized function. Then the algorithm M ′ ( M ( Y )) is ( ε, δ ) -differentially private.

In order to talk about composition it is convenient to also consider DP algorithms whose privacy guarantee holds only against subsets of inputs.

Definition A.3 (Differential Privacy Under Condition) . An algorithm M : Y → O is said to be ( ε, δ ) -differentially private under condition Ψ (or ( ε, δ ) -DP under condition Ψ ) for ε, δ > 0 if and only if, for every S ⊆ O and every neighboring datasets Y, Y ′ ∈ Y both satisfying Ψ we have

It is not hard to see that the following composition theorem holds for privacy under condition.

Lemma A.4 (Composition for Algorithm with Halting, [35]) . Let M 1 : Y → O 1 ∪ {⊥} , M 2 : O 1 × Y → O 2 ∪ {⊥} , . . . , M t : O t -1 × Y → O t ∪ {⊥} be algorithms. Furthermore, let M denote the algorithm that proceeds as follows (with O 0 being empty): For i = 1 . . . , t compute o i = M i ( o i -1 , Y ) and, if o i = ⊥ , halt and output ⊥ . Finally, if the algorithm has not halted, then output o t . Suppose that:

For any 1 ⩽ i ⩽ t , we say that Y satisfies the condition Ψ i if running the algorithm on Y does not result in halting after applying M 1 , . . . , M i .

M 1 is ( ε 1 , δ 1 ) -DP.

M i is ( ε i , δ i ) -DP (with respect to neighboring datasets in the second argument) under condition Ψ i -1 for all i = { 2 , . . . , t } .

Then M is ( ∑ i ε i , ∑ i δ i ) -DP.

A.1.1 Basic differential privacy mechanisms

The Gaussian and the Laplace mechanism are among the most widely used mechanisms in differential privacy. They work by adding a noise drawn from the Gaussian (respectively Laplace) distribution to the output of the function one wants to privatize. The magnitude of the noise depends on the sensitivity of the function.

Definition A.5 (Sensitivity of function) . Let f : Y → R d be a function, its ℓ 1 -sensitivity and ℓ 2 -sensitivity are respectively

For function with bounded ℓ 1 -sensitivity the Laplace mechanism is often the tool of choice to achieve privacy.

Definition A.6 (Laplace distribution) . The Laplace distribution with mean µ and parameter b > 0 , denoted by Lap ( µ, b ) , has PDF 1 2 b e -| x -µ | /b . Let Lap( b ) denote Lap(0 , b ) .

A standard tail bound concerning the Laplace distribution will be useful throughout the paper.

Fact A.7 (Laplace tail bound) . Let x ∼ Lap ( µ, b ) . Then,

The Laplace distribution is useful for the following mechanism

Lemma A.8 (Laplace mechanism) . Let f : Y → R d be any function with ℓ 1 -sensitivity at most ∆ f, 1 .

Then the algorithm that adds Lap ( ∆ f, 1 ε ) ⊗ d to f is ( ε, 0) -DP.

It is also useful to consider the "truncated" version of the Laplace distribution where the noise distribution is shifted and truncated to be non-positive.

Definition A.9 (Truncated Laplace distribution) . The (negatively) truncated Laplace distribution w with mean µ and parameter b on R , denoted by tLap ( µ, b ) , is defined as Lap ( µ, b ) conditioned on the value being non-positive.

Lemma A.10 (Truncated Laplace mechanism) . Let f : Y → R be any function with ℓ 1 -sensitivity at most ∆ f, 1 . Then the algorithm that adds tLap ( -∆ f, 1 ( 1 + log(1 /δ ) ε ) , ∆ f, 1 /ε ) to f is ( ε, δ ) -DP.

The following tail bound is useful when reasoning about truncated Laplace random variables.

Lemma A.11 (Tail bound truncated Laplace) . Suppose µ < 0 and b > 0 . Let x ∼ tLap ( µ, b ) . Then,for y < µ we have that

In constrast, when the function has bounded ℓ 2 -sensitivity, the Gaussian mechanism provides privacy. Lemma A.12 (Gaussian mechanism) . Let f : Y → R d be any function with ℓ 2 -sensitivity at most ∆ f, 2 . Let 0 < ε, δ ⩽ 1 . Then the algorithm that adds N ( 0 , ∆ 2 f, 2 · 2 log(2 /δ ) ε 2 · Id ) to f is ( ε, δ ) -DP.

A.1.2 Private histograms

Here we present a classical private mechanism to learn a high dimensional histogram.

Lemma A.13 (High-dimensional private histogram learner, see [33]) . Let q, b , ε > 0 and 0 < δ < 1 /n . Let { I i } ∞ i = -∞ be a partition of R into intervals of length b , where I i := { x ∈ R | q +( i -1) · b ⩽ x < q + i · b } . Consider the partition of R d into sets { B i 1 ,...,i d } ∞ i 1 ,...,i d =1 where

Let Y = { y 1 , . . . , y n } ⊆ R d be a dataset of n points. For each B i 1 ,...,i d , let p i 1 ,...,i d = 1 n |{ j ∈ [ n ] | y j ∈ B i 1 ,...,i d }| . For n ⩾ 8 εα · log 2 δβ , there exists an efficient ( ε, δ ) -differentially private algorithm that returns ˆ p 1 ,..., 1 , . . . , ˆ p i 1 ,...,i d , . . . satisfying

Proof. We consider the following algorithm, applied to each i 1 , . . . , i d ∈ N on input Y :

First we argue utility. By construction we get ˆ p i 1 ,...,i d = 0 whenever p i 1 ,...,i d = 0 , thus we may focus on non-zero p i 1 ,...,i d . There are at most n non zero p i 1 ,...,i d . By choice of n, δ and by Fact A.7 the maximum over n independent trials τ ∼ Lap ( 0 , 2 nε ) is bounded by α in absolute value with probability at least β .

It remains to argue privacy. Let Y = { y 1 , . . . , y n } , Y ′ = { y ′ 1 , . . . , y ′ n } be adjacent datasets. For i 1 , . . . , i d ∈ N , let

Since Y, Y ′ are adjacent there exists only two set of indices I := { i 1 , . . . , i d } and J := { j 1 , . . . , j d } such that p I = p ′ I and p J = p ′ J . Assume without loss of generality p I > p ′ I . Then it must be p I = p ′ I +1 /n and p J = p ′ J -1 /n. Thus by the standard tail bound on the Laplace distribution in Fact A.7 and by Lemma A.8, we immediately get that the algorithm is ( ε, δ ) -differentially private.

̸

̸

A.2 Sum-of-squares and pseudo-distributions

We introduce here the sum-of-squares notion necessary for our private algorithm learning mixtures of Gaussians. We remark that these notions are not needed for Appendix C.

Let w = ( w 1 , w 2 , . . . , w n ) be a tuple of n indeterminates and let R [ w ] be the set of polynomials with real coefficients and indeterminates w,. . . , w n . We say that a polynomial p ∈ R [ w ] is a sum-of-squares (sos) if there are polynomials q 1 , . . . , q r such that p = q 2 1 + · · · + q 2 r .

A.2.1 Pseudo-distributions

Pseudo-distributions are generalizations of probability distributions. We can represent a discrete (i.e., finitely supported) probability distribution over R n by its probability mass function D : R n → R such that D ⩾ 0 and ∑ w ∈ supp( D ) D ( w ) = 1 . Similarly, we can describe a pseudo-distribution by its mass function. Here, we relax the constraint D ⩾ 0 and only require that D passes certain low-degree non-negativity tests.

Concretely, a levelℓ pseudo-distribution is a finitely-supported function D : R n → R such that ∑ w D ( w ) = 1 and ∑ w D ( w ) f ( w ) 2 ⩾ 0 for every polynomial f of degree at most ℓ/ 2 . (Here, the summations are over the support of D .) A straightforward polynomial-interpolation argument shows

that every level-∞ -pseudo distribution satisfies D ⩾ 0 and is thus an actual probability distribution. We define the pseudo-expectation of a function f on R d with respect to a pseudo-distribution D , denoted ˜ E D ( w ) f ( w ) , as

The degreeℓ moment tensor of a pseudo-distribution D is the tensor E D ( w ) (1 , w 1 , w 2 , . . . , w n ) ⊗ ℓ . In particular, the moment tensor has an entry corresponding to the pseudo-expectation of all monomials of degree at most ℓ in w . The set of all degreeℓ moment tensors of probability distribution is a convex set. Similarly, the set of all degreeℓ moment tensors of degree d pseudo-distributions is also convex. Key to the algorithmic utility of pseudo-distributions is the fact that while there can be no efficient separation oracle for the convex set of all degreeℓ moment tensors of an actual probability distribution, there's a separation oracle running in time n O ( ℓ ) for the convex set of the degreeℓ moment tensors of all levelℓ pseudodistributions.

Fact A.14 ([56, 53, 51, 38]) . For any n, ℓ ∈ N , the following set has a n O ( ℓ ) -time weak separation oracle (in the sense of [27]):

This fact, together with the equivalence of weak separation and optimization [27] allows us to efficiently optimize over pseudo-distributions (approximately)-this algorithm is referred to as the sum-of-squares algorithm.

The levelℓ sum-of-squares algorithm optimizes over the space of all levelℓ pseudo-distributions that satisfy a given set of polynomial constraints-we formally define this next.

Definition A.15 (Constrained pseudo-distributions) . Let D be a levelℓ pseudo-distribution over R n . Let A = { f 1 ⩾ 0 , f 2 ⩾ 0 , . . . , f m ⩾ 0 } be a system of m polynomial inequality constraints. We say that D satisfies the system of constraints A at degree r , denoted D r A , if for every S ⊆ [ m ] and every sum-of-squares polynomial h with deg h + ∑ i ∈ S max { deg f i , r } ⩽ ℓ ,

We write D A (without specifying the degree) if D 0 A holds. Furthermore, we say that D r A holds approximately if the above inequalities are satisfied up to an error of 2 -n ℓ · ∥ h ∥ · ∏ i ∈ S ∥ f i ∥ , where ∥·∥ denotes the Euclidean norm 22 of the coefficients of a polynomial in the monomial basis.

We remark that if D is an actual (discrete) probability distribution, then we have D A if and only if D is supported on solutions to the constraints A .

We say that a system A of polynomial constraints is explicitly bounded if it contains a constraint of the form {∥ w ∥ 2 ⩽ M } . The following fact is a consequence of Fact A.14 and [27],

Fact A.16 (Efficient Optimization over Pseudo-distributions) . There exists an ( n + m ) O ( ℓ ) -time algorithm that, given any explicitly bounded and satisfiable system 23 A of m polynomial constraints in n variables, outputs a levelℓ pseudo-distribution that satisfies A approximately.

A.2.2 Sum-of-squares proof

Let f 1 , f 2 , . . . , f r and g be multivariate polynomials in w . A sum-of-squares proof that the constraints { f 1 ⩾ 0 , . . . , f m ⩾ 0 } imply the constraint { g ⩾ 0 } consists of sum-of-squares polynomials ( p S ) S ⊆ [ m ] such that

We say that this proof has degree ℓ if for every set S ⊆ [ m ] , the polynomial p S Π i ∈ S f i has degree at most ℓ . If there is a degree ℓ SoS proof that { f i ⩾ 0 | i ⩽ r } implies { g ⩾ 0 } , we write:

Sum-of-squares proofs satisfy the following inference rules. For all polynomials f, g : R n → R and for all functions F : R n → R m , G : R n → R k , H : R p → R n such that each of the coordinates of the outputs are polynomials of the inputs, we have:

Low-degree sum-of-squares proofs are sound and complete if we take low-level pseudo-distributions as models.

Concretely, sum-of-squares proofs allow us to deduce properties of pseudo-distributions that satisfy some constraints.

Fact A.17 (Soundness) . If D r A for a levelℓ pseudo-distribution D and there exists a sum-ofsquares proof A r ′ B , then D r · r ′ + r ′ B .

If the pseudo-distribution D satisfies A only approximately, soundness continues to hold if we require an upper bound on the bit-complexity of the sum-of-squares A r ′ B (number of bits required to write down the proof).

In our applications, the bit complexity of all sum of squares proofs will be n O ( ℓ ) (assuming that all numbers in the input have bit complexity n O (1) ). This bound suffices in order to argue about pseudo-distributions that satisfy polynomial constraints approximately.

The following fact shows that every property of low-level pseudo-distributions can be derived by low-degree sum-of-squares proofs.

Fact A.18 (Completeness) . Suppose d ⩾ r ′ ⩾ r and A is a collection of polynomial constraints with degree at most r , and A ⊢ { ∑ n i =1 w 2 i ⩽ B } for some finite B .

Let { g ⩾ 0 } be a polynomial constraint. If every degreed pseudo-distribution that satisfies D r A also satisfies D r ′ { g ⩾ 0 } , then for every ε > 0 , there is a sum-of-squares proof A d { g ⩾ -ε } .

A.2.3 Explictly bounded distributions

We will consider a subset of subgaussian distributions denoted as certifiably subgaussians. Many subgaussians distributions are known to be certifiably subgaussian (see [36]).

Definition A.19 (Explicitly bounded distribution) . Let t ∈ N . A distribution D over R d with mean µ is called 2 t -explicitly σ -bounded if for each even integer s such that 1 ⩽ s ⩽ t the following equation has a degree s sum-of-squares proof in the vector variable u

Furthermore, we say that D is explicitly bounded if it is 2 t -explicitly σ -bounded for every t ∈ N . A finite set X ⊆ R d is said to be 2 t -explicitly σ -bounded if the uniform distribution on X is 2 t -explicitly σ -bounded.

Sets that are 2 t -explicitly σ -bounded with large intersection satisfy certain key properties. Before introducing them we conveniently present the following definition.

Definition A.20 (Weight vector inducing distribution) . Let Y be a set of size n and let p ∈ [0 , 1] n be a vector satisfying ∥ p ∥ 1 = 1 . We say that p induces the distribution D with support Y if

Theorem A.21 ([36, 30]) . Let Y ⊆ R d be a set of cardinality n . Let p, p ′ ∈ [0 , 1] n be weight vectors satisfying ∥ p ∥ 1 = ∥ p ′ ∥ 1 = 1 and ∥ p -p ′ ∥ 1 ⩽ β . Suppose that p (respectively p ′ ) induces a 2 t -explicitly σ 1 -bounded (resp. σ 2 ) distribution over Y with mean µ ( p ) (resp. µ ( p ′ ) ). There exists an absolute constant β ∗ such that, if β ⩽ β ∗ , then for σ = σ 1 + σ 2 :

In the context of learning Gaussian mixtures, we will make heavy use of the statement below.

Theorem A.22 ([36, 30]) . Let Y be a 2 t -explicitly σ -bounded set of size n . Let p ∈ R n be the weight vector inducing the uniform distribution over Y . Let p ′ ∈ R n be a unit vector satisfying ∥ p -p ′ ∥ 1 ⩽ β for some β ⩽ β ∗ where β ∗ is a small constant. Then p ′ induces a 2 t -explicitly ( σ + O ( β 1 -1 / 2 t )) -bounded distribution over Y .

B Stability of strongly-convex optimization

In this section, we prove ℓ 2 sensitivity bounds for the minimizers of a general class of (strongly) convex optimization problems. In particular, we show how to translate a uniform point-wise sensitivity bound for the objective functions into a ℓ 2 sensitivity bound for the minimizers.

Lemma B.1 (Stability of strongly-convex optimization) . Let Y be a set of datasets. Let K ( Y ) be a family of closed convex subsets of R m parametrized by Y ∈ Y and let F ( Y ) be a family of functions f Y : K ( Y ) → R , parametrized by Y ∈ Y , such that:

(i) for adjacent datasets Y, Y ′ ∈ Y and X ∈ K ( Y ) there exist Z ∈ K ( Y ) ∩ K ( Y ′ ) satisfying | f Y ( X ) -f Y ′ ( Z ) | ⩽ α and | f Y ( Z ) -f Y ′ ( Z ) | ⩽ α.

(ii) f Y is κ -strongly convex in X ∈ K ( Y ) .

Then for Y, Y ′ ∈ Y , ˆ X := arg min X ∈K ( Y ) f Y ( X ) and ˆ X ′ := arg min X ′ ∈K ( Y ′ ) f Y ′ ( X ′ ) , it holds

Proof. Let Z ∈ K ( Y ) ∩ K ( Y ′ ) be a point such that ∣ ∣ ∣ f Y ( ˆ X ) -f Y ′ ( Z ) ∣ ∣ ∣ ⩽ α and | f Y ( Z ) -f Y ′ ( Z ) | ⩽ α . By κ -strong convexity of f Y and f Y ′ (Proposition G.2) it holds

Suppose w.l.o.g. f Y ( ˆ X ) ⩽ f Y ′ ( ˆ X ′ ) , for a symmetric argument works in the other case. Then

and

It follows as desired

C Private recovery for stochastic block models

In this section, we present how to achieve exact recovery in stochastic block models privately and thus prove Theorem 1.3. To this end, we first use the stability of strongly convex optimization (Lemma B.1) to obtain a private weak recovery algorithm in Appendix C.1. Then we show how to privately boost the weak recovery algorithm to achieve exact recovery in Appendix C.2. In Appendix C.4, we complement our algorithmic results by providing an almost tight lower bound on the privacy parameters. We start by defining the relevant notion of adjacent datasets.

Definition C.1 (Adjacent graphs) . Let G,G ′ be graphs with vertex set [ n ] . We say that G,G ′ are adjacent if | E ( G ) △ E ( G ′ ) | = 1 .

Remark C.2 (Parameters as public information) . We remark that we assume the parameters n, γ, d to be public information given in input to the algorithm.

C.1 Private weak recovery for stochastic block models

In this section, we show how to achieve weak recovery privately via stability of strongly convex optimization (Lemma B.1). We first introduce one convenient notation. The error rate of an estimate ˆ x ∈ {± 1 } n of the true partition x ∈ {± 1 } n is defined as err(ˆ x, x ) := 1 n · min { Ham(ˆ x, x ) , Ham(ˆ x, -x ) } . 24 Our main result is the following theorem.

Theorem C.3. Suppose γ √ d ⩾ 12800 , ε, δ ⩾ 0 . There exists an (Algorithm C.4) such that, for any x ∈ {± 1 } n , on input G ∼ SBM n ( γ, d, x ) , outputs ˆ x ( G ) ∈ {± 1 } n satisfying

with probability 1 -exp( -Ω( n )) . Moreover, the algorithm is ( ε, δ ) -differentially private for any input graph and runs in polynomial time.

Before presenting the algorithm we introduce some notation. Given a graph G , let Y ( G ) := 1 γd ( A ( G ) -d n J ) where A ( G ) is the adjacency matrix of G and J denotes all-one matrices. Define K := { X ∈ R n × n ∣ ∣ X ⪰ 0 , X ii = 1 n ∀ i } . The algorithm starts with projecting matrix Y ( G ) to set K . To ensure privacy, then it adds Gaussian noise to the projection X 1 and obtains a private matrix X 2 . The last step applies a standard rounding method.

Algorithm C.4 (Private weak recovery for SBM) .

.

Input: Graph G

Operations:

Rounding: Compute the leading eigenvector v of X 2 and return sign( v ) .

In the rest of this section, we will show Algorithm C.4 is private in Lemma C.7 and its utility guarantee in Lemma C.8. Then Theorem C.3 follows directly from Lemma C.7 and Lemma C.8.

Privacy analysis Let Y be the set of all matrices Y ( G ) = 1 γd ( A ( G ) -d n J ) where G is a graph on n vertices. We further define q : Y → K to be the function

We first use Lemma B.1 to prove that function q is stable.

Lemma C.5 (Stability) . The function q as defined in Eq. (C.1) has ℓ 2 -sensitivity ∆ q, 2 ⩽ √ 24 nγd .

Proof. Let g : Y × K → R be the function g ( Y, X ) := ∥ X ∥ 2 F -2 ⟨ Y, X ⟩ . Applying Lemma B.1 with f Y ( · ) = g ( Y, · ) , it suffices to prove that g has ℓ 1 -sensitivity 4 nγd with respect to Y and that it is 2 -strongly convex with respect to X . The ℓ 1 -sensitivity bound follows by observing that adjacent Y, Y ′ satisfy ∥ Y -Y ′ ∥ 1 ⩽ 2 γd and that any X ∈ K satisfies ∥ X ∥ ∞ ⩽ 1 n . Thus it remains to prove strong convexity with respect to X ∈ K . Let X,X ′ ∈ K then

That is g ( Y, X ) is 2 -strongly convex with respect to X . Note any X ∈ K is symmetric. Then the result follows by Lemma B.1.

Remark C.6. In the special case where the contraint set K does not depend on input dataset Y (e.g. stochastic block models), the proof can be cleaner as follows. Let f Y ( X ) := ∥ X ∥ 2 F -2 ⟨ X,Y ⟩ . Let ˆ X := argmin X ∈K f Y ( X ) and ˆ X ′ := argmin X ∈K f Y ′ ( X ) . Suppose without loss of generality f Y ( ˆ X ) ⩽ f Y ′ ( ˆ X ′ ) , for a symmetric argument works in the other case. Then

Then it is easy to show the algorithm is private.

Lemma C.7 (Privacy) . The weak recovery algorithm (Algorithm C.4) is ( ε, δ ) -DP.

Proof. Since any X ∈ K is symmetric, we only need to add a symmetric noise matrix to obtain privacy. Combining Lemma C.5 with Lemma A.12, we immediately get that the algorithm is ( ε, δ ) -private.

Utility analysis Now we show the utility guarantee of our priavte weak recovery algorithm.

Lemma C.8 (Utility) . For any x ∈ {± 1 } n , on input G ∼ SBM n ( γ, d, x ) , Algorithm C.4 efficiently outputs ˆ x ( G ) ∈ {± 1 } n satisfying

with probability 1 -exp( -Ω( n )) .

To prove Lemma C.8, we need the following lemma which is an adaption of a well-known result in SBM [28, Theorem 1.1]. Its proof is deferred to Appendix H.

Lemma C.9. Consider the settings of Lemma C.8. With probability 1 -exp( -Ω( n )) ,

Proof of Lemma C.8. By Lemma C.9, we have

with probability 1 -exp( -Ω( n )) . We condition our following analysis on this event happening.

Let u be the leading eigenvector of X 1 ( G ) . Let λ 1 and λ 2 be the largest and second largest eigenvalues of X 1 ( G ) . By Weyl's inequality (Lemma F.1) and the assumption γ √ d ⩾ 12800 , we have

Let v be the leading eigenvector of X 1 ( G ) + W . By Davis-Kahan's theorem (Lemma F.2), we have

Putting things together and using Fact E.1, we have

with probability 1 -exp( -Ω( n )) .

Observe Ham(sign( y ) , x ) ⩽ ∥ y -x ∥ 2 for any y ∈ R n and any x ∈ {± 1 } n . Then with probability 1 -exp( -Ω( n )) ,

Proof of Theorem C.3. By Lemma C.7 and Lemma C.8.

C.2 Private exact recovery for stochastic block models

In this section, we prove Theorem 1.3. We show how to achieve exact recovery in stochastic block models privately by combining the private weak recovery algorithm we obtained in the previous section and a private majority voting scheme.

Since exact recovery is only possible with logarithmic average degree (just to avoid isolated vertices), it is more convenient to work with the following standard parameterization of stochastic block models. Let α > β > 0 be fixed constants. The intra-community edge probability is α · log n n , and the intercommunity edge probability is β · log n n . In the language of Model 1.1, it is SBM n ( α + β 2 · log n, α -β α + β , x ) . Our main result is the following theorem.

Theorem C.10 (Private exact recovery of SBM, restatement of Theorem 1.3) . Let ε, δ ⩾ 0 . Suppose α, β are fixed constants satisfying 25

Then there exists an algorithm (Algorithm C.12) such that, for any balanced 26 x ∈ {± 1 } n , on input G ∼ SBM n ( α + β 2 · log n, α -β α + β , x ) , outputs ˆ x ( G ) ∈ { x, -x } with probability 1 -o (1) . Moreover, the algorithm is ( ε, δ ) -differentially private for any input graph and runs in polynomial time.

Remark C.11. In a standard regime of privacy parameters where ε ⩽ O (1) and δ = 1 / poly( n ) , the private exact recovery threshold Eq. (C.2) reads

Recall the non-private exact recovery threshold is √ α - √ β > √ 2 . Thus the non-private part in Eq. (C.2) , i.e. 16 , is close to optimal.

Algorithm C.12 starts with randomly splitting the input graph G into two subgraphs G 1 and G 2 . Setting the graph-splitting probability to 1 / 2 , each subgraph will contain about half of the edges of G . Then we run an ( ε, δ ) -DP weak recovery algorithm (Algorithm C.4) on G 1 to get a rough estimate ˜ x ( G 1 ) of accuracy around 90% . Finally, we boost the accuracy to 100% by doing majority voting (Algorithm C.13) on G 2 based on the rough estimate ˜ x ( G 1 ) . That is, if a vertex has more neighbors from the opposite community (according to ˜ x ( G 1 ) ) in G 2 , then we assign this vertex to the opposite community. To make the majority voting step private, we add some noise to the vote.

Algorithm C.12 (Private exact recovery for SBM) .

Input:

Graph G

Operations:

Graph-splitting: Initialize G 1 to be an empty graph on vertex set V ( G ) . Independently put each edge of G in G 1 with probability 1 / 2 . Let G 2 = G \ G 1 .

Rough estimation on G 1 : Run the ( ε, δ ) -DP partial recovery algorithm (Algorithm C.4) on G 1 to get a rough estimate ˜ x ( G 1 ) .

Majority voting on G 2 : Run the ( ε, 0) -DP majority voting algorithm (Algorithm C.13) with input ( G 2 , ˜ x ( G 1 )) and get output ˆ x .

Return ˆ x .

Input: Graph G , rough estimate ˜ x ∈ {± 1 } n

Operations:

̸

Return ˆ x .

In the rest of this section, we will show Algorithm C.12 is private in Lemma C.15 and it recovers the hidden communities exactly with high probability in Lemma C.17. Then Theorem C.10 follows directly from Lemma C.15 and Lemma C.17.

Privacy analysis. We first show the differential privcay of the majority voting algorithm (Algorithm C.13) with respect to input graph G (i.e. assuming fixed the input rough estimate).

Lemma C.14. Algorithm C.13 is ( ε, 0) -DP with respect to input G .

Proof. Observing the ℓ 1 -sensitivity of the degree count function Z in step is 2 , the ( ε, 0) -DP follows directly from Laplace mechanism (Lemma A.12) and post-processing (Lemma A.2).

Then the privacy of the private exact recovery algorithm (Algorithm C.12) is a consequence of composition.

Lemma C.15 (Privacy) . Algorithm C.12 is ( ε, δ ) -DP.

Proof. Let A 1 : G n → {± 1 } n denote the ( ε, δ ) -DP recovery algorithm in step 2. Let A 2 : G n × {± 1 } n →{± 1 } n denote the ( ε, 0) -DP majority voting algorithm in step 3. Let A be the composition of A 1 and A 2 .

We first make several notations. Given a graph H and an edge e , H e is a graph obtained b adding e to H . Given a graph H , G 1 ( H ) is a random subgraph of H by keeping each edge of H with probability 1 / 2 independently.

Now, fix two adjacent graphs G and G e where edge e appears in G e but not in G . Also, fix two arbitrary possible outputs x 1 , x 2 ∈ {± 1 } n of algorithm A . 27 It is direct to see,

Since both A 1 and A 2 are ( ε, δ ) -DP, we have for each H ⊆ G ,

Plugging Eq. (C.5) and Eq. (C.6) into Eq. (C.4), we obtain

Similarly, we can show

Utility analysis. We first show the utility guarantee of the priavte majority voting algorithm.

Lemma C.16. Suppose G is generated by first sampling G ∼ SBM n ( α + β 2 · log n, α -β α + β , x ) for some balanced x and then for each vertex removing at most ∆ ⩽ O (log 2 n ) adjacent edges arbitrarily. Then on input G and a balanced rough estimate ˜ x satisfying Ham(˜ x, x ) ⩽ n/ 16 , Algorithm C.13 efficiently outputs ˆ x ( G ) such that for each vertex v ,

̸

̸

Proof. Let us fix an arbitrary vertex v and analyze the probability P (ˆ x ( G ) v = x v ) . Let r := Ham(˜ x, x ) /n . Then it is not hard to see

̸

A ∼ Binomial((1 / 2 -r ) n -∆ , α log n n ) , corresponding to the number of neighbors that are from the same community and correctly labeled by ˜ x ,

B ′ ∼ Binomial( rn -∆ , β log n n ) , corresponding to the number of neighbors that are from the different community but incorrectly labeled by ˜ x ,

B ∼ Binomial((1 / 2 -r ) n, β log n n ) , corresponding to the number of neighbors that are from the different community and correctly labeled by ˜ x ,

A ′ ∼ Binomial( rn, α log n n ) , corresponding to the number of neighbors that are from the same community but incorrectly labeled by ˜ x ,

W ∼ Lap(0 , 2 /ε ) , independently.

The ∆ term appearing in both A and B ′ corresponds to the worst case where ∆ 'favorable' edges are removed. If r ⩾ Ω(1) , then ∆ = O (log 2 n ) is negligible to rn = Θ( n ) and we can safely ignore the effect of removing ∆ edges. If r = o (1) , then we can safely assume ˜ x is correct on all vertices and ignore the effect of removing ∆ edges as well. Thus, we will assume ∆ = 0 in the following analysis.

For any t, t ′ , we have

We choose t, t ′ by first picking two constants a, b > 0 satisfying a + b < 1 and then solving

where

By Fact A.7,

By Fact E.4 and the assumption r ⩽ 1 / 16 , we have

Setting b = 1 / 2 , by Fact E.4 and the assumption r ⩽ 1 / 16 , we have

Further setting a = 1 / 3 , we have

̸

Finally, plugging the assumption r ⩽ 1 / 16 to conclude.

Then it is not difficult to show the utility guarantee of our priavte exact recovery algorithm.

Lemma C.17 (Utility) . Suppose α, β are fixed constants satisfying

Then for any balanced x ∈ {± 1 } n , on input G ∼ SBM n ( α + β 2 · log n, α -β α + β , x ) , Algorithm C.12 efficiently outputs ˆ x ( G ) satisfying ˆ x ( G ) ∈ { x, -x } with probability 1 -o (1) .

Proof. We will show the probability of a fixed vertex being misclassified is at most o (1 /n ) . Then by union bound, exact recovery can be achieved with probability 1 -o (1) .

As the graph-splitting probability is 1 / 2 , G 1 follows SBM n ( α 2 · log n n , β 2 · log n n , x ) . By Theorem C.3, the rough estimate ˜ x ( G 1 ) satisfies 28

with probability at least 1 -exp( -Ω( n )) . Without loss of generality, we can assume Ham(˜ x ( G 1 ) , x ) ⩽ rn , since we consider -x otherwise. By Fact E.2, the maximum degree of G 1 is at most ∆ := 2 log 2 n with probability at least 1 -n exp( -(log n ) 2 / 3) . In the following, we condition our analysis on the above two events regarding ˜ x ( G 1 ) and G 1 .

Now, let us fix a vertex and analyze the probability p e that it is misclassified after majority voting. With G 1 being fixed, G 2 can be thought of as being generated by first sampling G and then removing G 1 from G . To make r ⩽ 1 / 16 , it suffices to ensure α -β > 500 2 ε 2 · log(2 /δ ) log n by Eq. (C.9).Then by Lemma C.16, we have

To make p e at most o (1 /n ) , it suffices to ensure

Note ( α -β ) 2 / ( α + β ) > ( √ α - √ β ) 2 for α > β . Therefore, as long as

Algorithm C.12 recovers the hidden communities exactly with probability 1 -o (1) .

Proof of Theorem C.10. By Lemma C.15 and Lemma C.17.

C.3 Inefficient recovery using the exponential mechanism

In this section, we will present an inefficient algorithm satisfying pure privacy which succeeds for all ranges of parameters - ranging from weak to exact recovery. The algorithm is based on the exponential mechanism [43] combined with the majority voting scheme introduced in section Appendix C.2. In particular, we will show

Theorem C.18 (Full version of Theorem 1.4) . Let γ √ d ⩾ 12800 and x ∈ {± 1 } n be balanced. Let ζ ⩾ 2 exp ( -γ 2 d 512 ) . For any ε ⩾ 64 log(2 /ζ ) γd , there exists an algorithm, Algorithm C.19, which on input G ∼ SBM n ( γ, d, x ∗ ) outputs an estimate ˆ x ( G ) ∈ {± 1 } n satisfying

with probability at least 1 -ζ . In addition, the algorithm is ε -private. Further, by slightly modifying the algorithm, we can achieve error 20 / √ log(1 /ζ ) with probability 1 -e -n . 29

Acouple of remarks are in order. First, our algorithm works across all degree-regimes in the literature and matches known non-private thresholds and rates up to constants. We remark that for ease of exposition we did not try to optimize these constants. In particular, for γ 2 d a constant we achieve weak recovery. We reiterate, that γ 2 d > 1 is the optimal non-private threshold. For the regime, where γ 2 d = ω (1) , it is known that the optimal error rate is exp ( -(1 -o (1)) γ 2 d ) even non-privately [64], where o (1) goes to zero as γ 2 d tends to infinity. We match this up to constants. Moreover, our algorithm achieves exact recovery as soon as γ 2 d ⩾ 512 log n since then ζ < 1 n . This also matches known non-private threshholds up to constants [5, 47]. Also, our dependence on the privacy parameter ε is also optimal as shown by the information-theoretic lower bounds in Appendix C.4.

We also emphasize, that if we only aim to achieve error on the order of

we can achieve exponentially small failure probability in n , while keeping the privacy parameter ε the same. This can be achieved, by ommitting the boosting step in our algorithm and will be clear from the proof of Theorem C.18. We remark that in this case, we can also handle non-balanced communities.

Again, for an input graph G , consider the matrix Y ( G ) = 1 γd ( A ( G ) -d n J ) . For x ∈ {± 1 } n we define the score function

Since the entries of A ( G ) are in [0 , 1] and adjacent graphs differ in at most one edge, it follows immediately, that this score function has sensitivity at most

Algorithm C.19 (Inefficient algorithm for SBM) .

Input:

Graph G , privacy parameter ε > 0

Operations:

Graph-splitting: Initialize G 1 to be an empty graph on vertex set V ( G ) . Independently assign each edge of G to G 1 with probability 1 / 2 . Let G 2 = G \ G 1 .

Rough estimation on G 1 : Sample ˜ x from the distribution with density

Majority voting on G 2 : Run the ε -DP majority voting algorithm (Algorithm C.13) with input ( G 2 , ˜ x ( G 1 )) . Denote its output by ˆ x .

Return ˆ x .

We first analyze the privacy guarantees of the above algorithm.

Proof. For simplicity and clarity of notation, we will show that the algorithm satisfies 2 ε -DP. Clearly, the graph splitting step is 0-DP. Step 2 corresponds to the exponential mechanism. Since the sensitivity of the score function is at most ∆ = 2 γd it follows by the standard analysis of the mechanism that this step is ε -DP [43]. By Lemma C.14, the majority voting step is also ε -DP. Hence, the result follows by composition (cf. Lemma A.4).

Next, we will analyze its utility.

with probability at least 1 -ζ .

Proof. We will first show that the rough estimate ˜ x obtained in step 2 achieves

with probability e -n . This will prove the second part of the theorem - for this we don't need that ζ ⩽ exp( -640) . In fact, arbitrary ζ works. The final error guarantee will then follow by Lemma C.16. First, notice that similar to the proof of [28, Lemma 4.1], using Bernstein's inequality and a union bound, we can show that (cf. Fact H.2 for a full proof)

with probability at least 1 -exp -10 n . Recall that s G ( x ) = ⟨ x, Y ( G ) x ⟩ . Let α = 5 √ log(1 /ζ ) . We call x ∈ {± 1 } n good if s G ( x ) ⩾ (1 -3 α ) n . It follows that for good x it holds that

Which implies that

Lemma C.21. Let γ √ d ⩾ 12800 and x ∈ {± 1 } n be balanced. Let exp( -640) ⩾ ζ ⩾ 2 exp ( -γ 2 d 512 ) , ε ⩾ 64 log(2 /ζ ) γd , and G ∼ SBM n ( γ, d, x ∗ ) , the output ˆ x ( G ) ∈ {± 1 } n of Algorithm C.19 satisfies

where we used that α ⩽ 1 / 4 and that √ 1 -4 x ⩽ 1 -2 x for x ⩾ 0 . Hence, we have for good x that

Since s G ( x ∗ ) ⩾ (1 -α ) n ,there is at least one good candidate. Hence, we can bound the probability that we do not output a good x as

where we used that

We will use Lemma C.16 to proof the final conclusion of the theorem. In what follows, assume without loss of generality that Ham( x, x ∗ ) < Ham( x, -x ∗ ) . The above discussion implies that

where the last inequality uses ζ ⩽ e -640 . Further, by Fact E.2 it also follows that the maximum degree of G 2 is at most O ( log 2 n ) (by some margin). Recall that G 2 ∼ SBM ( d, γ, x ∗ ) . In the parametrization of Lemma C.16 this means that

Thus, it follows that the output ˆ x of the majority voting step satisfies for every vertex v

̸

By Markov's Inequality it now follows that

C.4 Lower bound on the parameters for private recovery

In this section, we prove a tight lower bound for private recovery for stochastic block models. Recall the definition of error rate, err( u, v ) := 1 n · min { Ham( u, v ) , Ham( u, -v ) } for u, v ∈ {± 1 } n . Our main result is the following theorem.

Theorem C.22 (Full version of Theorem 1.5) . Suppose there exists an ε -differentially private algorithm such that for any balanced x ∈ {± 1 } n , on input G ∼ SBM n ( d, γ, x ) , outputs ˆ x ( G ) ∈ {± 1 } n satisfying

where 30 1 /n ⩽ ζ ⩽ 0 . 04 and the randomness is over both the algorithm and stochastic block models. Then,

Remark C.23. Both terms in lower bound Eq. (C.10) are tight up to constants by the following argument. Considering typical privacy parameters ε ⩽ 1 , then e 2 ε -1 ≈ 2 ε . For exponentially small failure probability, i.e. η = 2 -Ω( n ) , the lower bound reads ε ⩾ Ω( 1 γd · 1 ζ ) , which is achieved by Algorithm C.19 without the boosting step - see the discussion after Theorem C.18. For polynomially small failure probability, i.e. η = 1 / poly( n ) , the lower bound Eq. (C.10) reads ε ⩾ Ω( 1 γd · log 1 ζ ) , which is achieved by Theorem C.18.

By setting ζ = 1 /n in Theorem C.22, we directly obtain a tight lower bound for private exact recovery as a corollary.

Corollary C.24. Suppose there exists an ε -differentially private algorithm such that for any balanced x ∈ {± 1 } n , on input G ∼ SBM n ( d, γ, x ) , outputs ˆ x ( G ) ∈ {± 1 } n satisfying

where the randomness is over both the algorithm and stochastic block models. Then,

Remark C.25. The lower bound Eq. (C.11) for priavte exact recovery is tight up to constants, since there exists an (inefficient) ε -differentially priavte exact recovery algorithm with ε ⩽ O ( log n γd ) and η = 1 / poly( n ) by Theorem C.18 and [55, Theorem 3.7].

In rest of this section, we will prove Theorem C.22. The proof applies the packing lower bound argument similar to [29, Theorem 7.1]. To this end, we first show err( · , · ) is a semimetric over {± 1 } n .

Lemma C.26. err( · , · ) is a semimetric over {± 1 } n .

Proof. Symmetry and non-negativity are obvious from the definition. We will show err( · , · ) satisfies triangle inequality via case analysis. Let u, v, w ∈ {± 1 } n be three arbitrary sign vectors. By symmetry, we only need to consider the following four cases.

Case 1: Ham( u, v ) , Ham( u, w ) , Ham( v, w ) ⩽ n/ 2 . This case is reduced to showing Hamming distance satisfies triangle inequality, which is obvious.

Case 2: Ham( u, v ) , Ham( u, w ) ⩽ n/ 2 and Ham( v, w ) ⩾ n/ 2 . We need to check two subcases. First,

Second,

Case 3: Ham( u, v ) ⩽ n/ 2 and Ham( u, w ) , Ham( v, w ) ⩾ n/ 2 . This case can be reduced to case 1 by considering u, v, -w .

Case 4: Ham( u, v ) , Ham( u, w ) , Ham( v, w ) ⩾ n/ 2 . This case can be reduced to case 2 by considering -u, v, w .

Proof of Theorem C.22. Suppose there exists an ε -differentially private algorithm satisfying the theorem's assumption.

We first make the following notation. Given a semimetric ρ over {± 1 } n , a center v ∈ {± 1 } n , and a radius r ⩾ 0 , define B ρ ( v, r ) := { w ∈ {± 1 } n : 1 ⊤ w = 0 , ρ ( w,v ) ⩽ r } .

Pick an arbitrary balanced x ∈ {± 1 } n . Let M = { x 1 , x 2 , . . . , x m } be a maximal 2 ζ -packing of B err ( x, 4 ζ ) in semimetric err( · , · ) . By maximality of M , we have B err ( x, 4 ζ ) ⊆ ∪ i m =1 B err ( x i , 2 ζ ) , which implies

For each i ∈ [ m ] , define Y i := { w ∈ {± 1 } n : err( w,x i ) ⩽ ζ } . Then Y i 's are pairwise disjoint. For each i ∈ [ m ] , let P i be the distribution over n -vertex graphs generated by SBM n ( d, γ, x i ) . By our assumption on the algorithm, we have for any i ∈ [ m ] that

Combining the fact that Y i 's are pairwise disjoint, we have

In the following, we will lower bound P G ∼ P 1 (ˆ x ( G ) ∈ Y i ) for each i ∈ [ m ] \ { 1 } using group privacy.

Note each P i is a product of ( n 2 ) independent Bernoulli distributions. Thus for any i, j ∈ [ m ] , there exists a coupling ω ij of P i and P j such that, if ( G , H ) ∼ ω , then

where p = 2 γd/n and N ij = Ham( x i , x j ) · ( n -Ham( x i , x j )) . Applying group privacy, we have for any two graphs G,H and for any S ⊆ {± 1 } n that 31

For each i ∈ [ m ] , taking expectations on both sides of Eq. (C.14) with respect to coupling ω i 1 and setting S = Y i , we have

The left side of Eq. (C.15) is equal to

Upper bounding the right side of Eq. (C.15) by Cauchy-Schwartz inequality, we have

Using the formula for the moment generating function of binomial distributions, we have

and it is easy to see

Putting things together, Eq. (C.15) implies for each i ∈ [ m ] that

Since x i ∈ B err ( x, 4 ζ ) for i ∈ [ m ] , by assuming ζ ⩽ 1 / 16 , we have

Recalling p = 2 γd/n and combining Eq. (C.12), Eq. (C.13), Eq. (C.16) and Eq. (C.17), we have

By taking logarithm on both sides, using t ⩾ log(1 + t ) for any t > -1 , and assuming ζ ⩽ 1 / (8 e ) , we have

D Private algorithms for learning mixtures of spherical Gaussians

In this section we present a private algorithm for recovering the centers of a mixtures of k Gaussians (cf. Model 1.2). Let Y ⊆ ( R d ) ⊗ n be the collection of sets of n points in R d . We consider the following notion of adjacency.

Definition D.1 (Adjacent datasets) . Two datasets Y, Y ′ ∈ Y are said to be adjacent if | Y ∩ Y ′ | ⩾ n -1 .

Remark D.2 (Problem parameters as public information) . We consider the parameters n, k, ∆ to be public information given as input to the algorithm.

Next we present the main theorem of the section.

Theorem D.3 (Privately learning spherical mixtures of Gaussians) . Consider an instance of Model 1.2. Let t ∈ N be such that ∆ ⩾ O ( √ tk 1 /t ) . For n ⩾ Ω ( k O (1) · d O ( t ) ) , k ⩾ (log n ) 1 / 5 , there exists an algorithm, running in time ( nd ) O ( t ) , that outputs vectors ˆ µ 1 , . . . , ˆ µ ℓ satisfying

with high probability, for some permutation π : [ k ] → [ k ] . 32 Moreover, for ε ⩾ k -10 , δ ⩾ n -10 , the algorithm is ( ε, δ ) -differentially private for any input Y .

We remark that our algorithm not only works for mixtures of Gaussians but for all mixtures of 2 t -explicitly bounded distributions (cf. Definition A.19).

Our algorithm is based on the sum-of-squares hierarchy and at the heart lies the following sumof-squares program. The indeterminates z 11 , . . . , z 1 k , . . . , z nk and vector-valued indeterminates µ ′ 1 , . . . , µ ′ k , will be central to the proof of Theorem D.3. Let n, k, t be fixed parameters.

We remark that the moment constraint encodes the 2 t -explicit 2 -boundedness constraint introduced in Definition A.19. Note that in the form stated above there are infinitely many constraints, one for each vector v . This is just for notational convenience. This constraint postulates equality of two polynomials in v . Formally, this can also be encoded by requiring there coefficients to agree and hence eliminating the variable v . It is not hard to see that this can be done adding only polynomially many constraints. Further, the matrix variable Q represents the SOS proof of the 2 t -explicit 2 -boundedness constraint and we can hence deduce that for all 0 ⩽ s ⩽ t

Before presenting the algorithm we will introduce some additional notation which will be convenient. We assume t, n, k to be fixed throughout the section and drop the corresponding subscripts. For Y ∈ Y , let Z ( Y ) be the set of degree10 t pseudo-distributions satisfying P ( Y ) . For each ζ ∈ Z ( Y ) define W ( ζ ) as the n -byn matrix satisfying

We let W ( Y ) := { W ( ζ ) | ζ ∈ Z ( Y ) } .

Recall that J denotes the all-ones matrix. We define the function g : R n × n → R as

We also consider the following function

Definition D.4 (Soft thresholding function) . We denote by ϕ : [0 , 1] → [0 , 1] the function

Notice that ϕ ( · ) is 1 0 . 9 -0 . 8 = 10 Lipschitz. Next we introduce our algorithm. Notice the algorithm relies on certain private subroutines. We describe them later in the section to improve the presentation.

and let

Algorithm D.5 (Private algorithm for learning mixtures of Gaussians) . Input: Set of n points Y ⊆ R d , ε , δ > 0 , k, t ∈ N , d ∗ = 100log n,b = k -15 .

Compute W = W ( ˆ ζ ( Y )) .

Pick τ ∼ tLap ( -n 1 . 6 ( 1 + log(1 /δ ) ε ) , n 1 . 6 ε ) .

If | τ | ⩾ n 1 . 7 or ∥ ϕ ( W ) ∥ 1 ⩽ n 2 k · ( 1 -1 n 0 . 1 -1 k 100 ) + τ reject.

For all i ∈ [ n ] , compute the n -dimensional vector

Pick a set S of n 0 . 01 indices i ∈ [ n ] uniformly at random.

For each i ∈ S let ¯ ν ( i ) = ν ( i ) + w where w ∼ N ( 0 , n -0 . 18 · log(2 /δ ) ε 2 · Id ) .

Pick Φ ∼ N ( 0 , 1 d ∗ ) d ∗ × d , q u.a.r. ∼ [0 , b ] and run the histogram learner of Lemma A.13 with input Φ ¯ ν (1) , . . . , Φ ¯ ν ( n 0 . 01 ) and parameters

Let B 1 , . . . , B k be the resulting d ∗ -dimensional bins with highest counts. Break ties randomly.

Reject if min i ∈ [ k ] ∣ ∣ { j ∣ ∣ Φ ¯ ν ( j ) ∈ B i }∣ ∣ < n 0 . 01 2 k .

For each l ∈ [ k ] output

For convenience, we introduce some preliminary facts.

Definition D.6 (Good Y ) . Let Y be sampled according to Model 1.2. We say that Y is good if:

for each ℓ ∈ [ k ] , there are at least n k -n 0 . 6 and most n k + n 0 . 6 points sampled from D ℓ in Y . Let Y ℓ ⊆ Y be such set of points.

Each Y ℓ is 2 t -explicitly 2 -bounded.

It turns out that typical instances Y are indeed good.

Lemma D.7 ([30, 36]) . Consider the settings of Theorem D.3. Then Y is good with high probability. Further, in this case the sets Z ( Y ) and W ( Y ) are non-empty.

D.1 Privacy analysis

In this section we show that our clustering algorithm is private.

Lemma D.8 (Differential privacy of the algorithm) . Consider the settings of Theorem D.3. Then Algorithm D.5 is ( ε, δ ) -differentially private.

We split our analysis in multiple steps and combine them at the end. On a high level, we will argue that on adjacent inputs Y, Y ′ many of the vectors ν ( i ) by the algorithm are close to each other and a small part can be very far. We can then show that we can mask this small difference using the

Gaussian mechanism and afterwards treat this subset of the vectors as privatized (cf. Lemma I.4). Then we can combine this with known histogram learners to deal with the small set of ν ( i ) 's that is far from each other on adjacent inputs.

D.1.1 Sensitivity of the matrix W

Here we use Lemma B.1 to reason about the sensitivity of ϕ ( W ( ˆ ζ ( Y ))) . For adjacent datasets Y, Y ′ ∈ Y we let ˆ ζ , ˆ ζ ′ be the pseudo-distribution corresponding to W ( ˆ ζ ( Y )) and W ( ˆ ζ ( Y ′ )) computed in step 1 of the algorithm, respectively. We prove the following result.

Lemma D.9 ( ℓ 1 -sensitivity of ϕ ( W ) ) . Consider the settings of Theorem D.3. Let W,W ′ be respectively be the matrices computed in step 1 by Algorithm D.5 on adjacent inputs Y, Y ′ ∈ Y . Then

For all but n 0 . 8 rows i of ϕ ( W ) , ϕ ( W ′ ) , it holds

Proof. The second inequality is an immediate consequence of the first via Markov's inequality. Thus it suffices to prove the first. Since ϕ ( · ) is 10-Lipschitz, we immediately obtain the result if

Thus we focus on this inequality. To prove it, we verify the two conditions of Lemma B.1. First notice that g is 2 -strongly convex with respect to its input W . Indeed for W,W ′ ∈ W ( Y ) , since ∀ i, j ∈ [ n ] , W ij ⩾ 0 it holds that

where we used that ∇ g ( W ) = 2 W -(10) 10 k 300 J . Thus it remain to prove (i) of Lemma B.1.

Let ˆ ζ ∈ Z ( Y ) , ˆ ζ ′ ∈ Z ( Y ′ ) be the pseudo-distributions such that W Y ( ˆ ζ ) = W and W Y ( ˆ ζ ′ ) = W ′ . We claim that there always exists ζ adj ∈ Z ( Y ) ∩ Z ( Y ′ ) such that

Note that in this case the second point is always true since g doesn't depend on Y . Together with Lemma B.1 these two inequalities will imply that

By assumption on n , an application of Cauchy-Schwarz will give us the desired result.

So, let i be the index at which Y, Y ′ differ. We construct ζ adj as follows: for all polynomials p of degree at most 10 t we let

By construction ζ adj ∈ Z ( Y ) ∩ Z ( Y ′ ) . Moreover, W ( ζ ) , W ( ζ adj ) differ in at most 2 n/k entries. Since all entries of the two matrices are in [0 , 1] , the first inequality follows by definition of the objective function.

D.1.2 Sensitivity of the resulting vectors

In this section we argue that if the algorithm does not reject in step 3 then the vectors ν ( i ) are stable on adjacent inputs. Concretely our statement goes as follows:

Lemma D.10 (Stability of the ν ( i ) 's) . Consider the settings of Theorem D.3. Suppose Algorithm D.5 does not reject in step 3, on adjacent inputs Y , Y ′ ∈ Y . Then for all but 6 n k 50 indices i ∈ [ n ] , it holds:

The proof of Lemma D.10 crucially relies on the next statement.

Lemma D.11 (Covariance bound) . Consider the settings of Theorem D.3. Let W be the matrix computed by Algorithm D.5 on input Y ∈ Y . For i ∈ [ n ] , if ∥ ϕ ( W i ) ∥ 1 ⩾ n k · ( 1 -10 k 50 ) then ν ( i ) induces a 2 -explicitly 40 -bounded distribution over Y .

Proof. First, by assumption notice that there must be at least n k · ( 1 -10 k 50 ) entries of ϕ ( W i ) larger than 0 . 8 . We denote the set of j ∈ [ n ] such that W ij ⩾ 0 . 8 by G . Let ζ ∈ Z ( Y ) be the degree 10 t pseudo-distribution so that W = W ( ζ ( Y )) . Since ζ satisfies P ( Y ) , for ℓ ∈ [ k ] it follows from the moment bound constraint for s = 1 that for all unit vectors u it holds that

Using the SOS triangle inequality (cf. Fact I.2) 2 a,b ( a + b ) 2 ⩽ 2( a 2 + b 2 ) it now follows that

and thus

̸

Furthermore using P ( Y ) 2 { z iℓ z iℓ ′ = 0 } for ℓ = ℓ ′ we have

Now, for fixed j , j ′ ∈ [ n ] , using

with a = ∑ ℓ ∈ [ k ] z iℓ z jℓ and b = ∑ ℓ ′ ∈ [ k ] z iℓ ′ z j ′ ℓ ′ we get ˜ E ζ     ∑ ℓ ∈ [ k ] z iℓ z jℓ     ∑ ℓ ′ ∈ [ k ] z iℓ ′ z j ′ ℓ ′     ⩾ ˜ E ζ   ∑ ℓ ∈ [ k ] z iℓ z jℓ + ∑ ℓ ′ ∈ [ k ] z iℓ ′ z j ′ ℓ ′   -1 = W ij + W ij ′ -1 .

Now if j, j ′ ∈ G we must have

Since ϕ ( W ij ) ⩽ 1 by definition and ∥ ϕ ( W i ) ∥ 1 ⩾ n k · ( 1 -10 k 50 ) , we conclude

as desired.

We can now prove Lemma D.10.

Proof of Lemma D.10. Let W,W ′ be the matrices computed by Algorithm D.5 in step 1 on input Y, Y ′ , respectively. Let G ⊆ [ n ] be the set of indices i such that

Notice that |G| ⩾ n -n 0 . 8 by Lemma D.9. Since on input Y the algorithm did not reject in step 3 we must have

Let g W be the number of indices i ∈ G such that ∥ ϕ ( W ) i ∥ 1 ⩾ n k · ( 1 -1 k 50 ) . It holds that

Rearring now yields

Similarly, let g W ′ be the number of indices i ∈ G such that ∥ ϕ ( W ′ ) i ∥ 1 ⩾ n k · ( 1 -1 k 50 ) . By an analogous argument it follows that g W ′ ⩾ n · ( 1 -3 k 50 ) . Thus, by the pigeonhole principle there are at least g W ⩾ n · ( 1 -6 k 50 ) indices i such that

Combining these with Lemma D.11 we may also add

the distribution induced by ∥ ϕ ( W i ) ∥ -1 1 ϕ ( W i ) is 2 -explicitly 40 -bounded,

the distribution induced by ∥ ϕ ( W ′ i ) ∥ -1 1 ϕ ( W ′ i ) is 2 -explicitly 40 -bounded.

Using that for non-zero vectors x, y it holds that ∥ ∥ ∥ x ∥ x ∥ -y ∥ y ∥ ∥ ∥ ∥ ⩽ 2 ∥ x ∥ ∥ x -y ∥ points 1 to 3 above imply that

Hence, applying Theorem A.21 with t = 1 it follows that

D.1.3 From low sensitivity to privacy

In this section we argue privacy of the whole algorithm, proving Lemma D.8. Before doing that we observe that low-sensitivity is preserved with high probability under subsampling.

Fact D.12 (Stability of S ) . Consider the settings of Theorem D.3. Suppose Algorithm D.5 does not reject in step 3, on adjacent inputs Y , Y ′ ∈ Y . With probability at least 1 -e -n Ω(1) over the random choices of S , for all but 10 n 0 . 01 k 50 indices i ∈ S , it holds:

Proof. There are at most 6 n k 50 such indices in [ n ] by Lemma D.10. By Chernoff's bound, cf. Fact E.4, the claim follows.

Finally, we prove our main privacy lemma.

Proof of Lemma D.8. For simplicity, we will prove that the algorithm is (5 ε, 5 δ ) -private. Let Y, Y ′ ∈ Y be adjacent inputs. By Lemma A.10 and Lemma D.9 the test in step 3 of Algorithm D.5 is ( ε, δ ) -private.

Thus suppose now the algorithm did not reject in step 3 on inputs Y, Y ′ . By composition (cf. Lemma A.4) it is enough to show that the rest of the algorithm is ( ε, δ ) -private with respect to Y, Y ′ under this condition. Next, let ν (1) Y , . . . , ν ( n ) Y and ν (1) Y ′ , . . . , ν ( n ) Y ′ be the vectors computed in step 4 of the algorithm and S be the random set of indices computed in step 5. 33 By Lemma D.10 and Fact D.12 with probability 1 -e -n Ω(1) over the random choices of S we get that for all but 10 n 0 . 01 k 50 indices i ∈ S , it holds that

Denote this set of indices by G . Note, that we may incorporate the failure probability e -n Ω(1) ⩽ min { ε/ 2 , δ/ 2 } into the final privacy parameters using Fact I.3.

Denote by V , V ′ the | S | -byd matrices respectively with rows ν ( i 1 ) Y , . . . , ν ( i | S | ) Y and ν ( i 1 ) Y ′ , . . . , ν ( i | S | ) Y ′ , where i 1 , . . . , i | S | are the indices in S . Recall, that |G| rows of V and V ′ differ by at most O ( n -0 . 1 ) in ℓ 2 -norm. Thus, by the Gaussian mechanism used in step 6 (cf. Lemma A.12) and Lemma I.4 it is enough to show that step 7 to step 9 of the algorithm are private with respect to pairs of inputs V and V ′ differing in at most 1 row. 34 In particular, suppose these steps are ( ε 1 , δ 1 ) -private. Then, for m = n 0 . 01 -|G| ⩽ 10 n 0 . 01 k 50 , by Lemma I.4 it follows that step 6 to step 9 are ( ε ′ , δ ′ ) -differentially private with

Consider steps 7 and 8. Recall, that in step 7 we invoke the histogram learner with parameters

Hence, by Lemma A.13 this step is ( ε ∗ , δ ∗ ) -private since

for ε ⩾ k -10 . Step 8 is private by post-processing.

Next, we argue that step 9 is private by showing that the average over the bins has small ℓ 2 -sensitivity. By Lemma A.4 we can consider the bins B 1 , . . . , B k computed in the previous step as fixed. Further, we can assume that the algorithm did not reject in step 8, i.e., that each bin contains at least n 0 . 01 2 k points of V and V ′ respectively. As a consequence, every bin contains at least two (projections of) points of the input V or V ′ respectively. In particular, it contains at least one (projection of a) point which is present in both V and V ′ . Fix a bin B l and let ¯ ν ∗ be such that it is both in V and V ′ and Φ ¯ ν ∗ ∈ B l . Also, define

Assume V and V ′ differ on index j . We consider two cases. First, assume that Φ ¯ ν ( j ) Y and Φ ¯ ν ( j ) Y ′ both lie in B l . In this case, S l = S ′ l and using Lemma E.5 it follows that with probability n -100 ⩽ min { ε/ 2 , δ/ 2 } it holds that

And hence we can bound

Next, assume that Φ ¯ ν ( j ) Y ̸∈ B l and Φ ¯ ν ( j ) Y ′ ∈ B l (the other case works symetrically). It follows that S l = S ′ l -1 and we can bound

Hence, the ℓ 2 -sensitivity is at most ∆ := 400 · k -11 n 0 . 01 . Since

and w ′ ∼ N ( 0 , 32 · k -120 · log(2 kn/δ ) ε 2 · Id ) it follows that outputing ˆ µ l is ( ε ∗ /k, δ ∗ /k ) -DP by the Gaussian Mechanism that. By Lemma A.4 it follows step 9 is ( ε ∗ , δ ∗ ) -private.

Hence, by Lemma A.4 it follows that step 7 to step 9 are (2 ε ∗ , 2 δ ∗ ) -differentially private. Using m ⩽ 10 n 0 . 01 k 10 it now follows by Lemma I.4 that step 6 to step 9 are ( ε ′ , δ ′ ) -private for

Thus, combined with the private check and Fact I.3 in step 3 the whole algorithm is (5 ε, 5 δ ) -private.

D.2 Utility analysis

In this section we reason about the utility of Algorithm D.5 and prove Theorem D.3. We first introduce some notation.

Definition D.13 (True solution) . Let Y be an input sampled from Model 1.2. Denote by W ∗ ( Y ) ∈ W ( Y ) the matrix induced by the true solution (or ground truth). I.e., let

W ∗ ( Y ) ij = { 1 if i , j were both sampled from the same component of the mixture, 0 otherwise.

Whenever the context is clear, we simply write W ∗ to ease the notation.

First, we show that in the utility case step 3 of Algorithm D.5 rejects only with low probability. Lemma D.14 (Algorithm does not reject on good inputs) . Consider the settings of Theorem D.3. Suppose Y is a good set as per Definition D.6. Then ∥ ∥ ∥ W ( ˆ ζ ( Y )) ∥ ∥ ∥ 1 ⩾ n 2 k · ( 1 -n -0 . 4 -1 (10) 10 k 300 ) and Algorithm D.5 rejects with probability at most exp ( -Ω ( n 1 . 7 )) .

Proof. Since Y is good, there exists W ∗ ∈ W ( Y ) , corresponding to the indicator matrix of the true solution, such that

Since g ( W ( ˆ ζ ( Y ))) ⩽ g ( W ∗ ) it follows that

Since, ∥ ∥ ∥ W ( ˆ ζ ( Y )) ∥ ∥ ∥ 1 ⩾ ⟨ J, W ( ˆ ζ ( Y )) ⟩ the first claim follows rearranging the terms. This means that the algorithm rejects only if | τ | ⩾ n 1 . 7 . Recall that τ ∼ tLap ( -n 1 . 6 ( 1 + log(1 /δ ) ε ) , n 1 . 6 ε ) . Hence,by Lemma A.11 it follows that

The next step shows that on a good input Y the matrix ϕ ( W ( ˆ ζ ( Y ))) is close to the true solution.

Lemma D.15 (Closeness to true solution on good inputs) . Consider the settings of Theorem D.3. Suppose Y is a good set as per Definition D.6. Let W ( Y ) ∈ W ( Y ) be the matrix computed by Algorithm D.5. Suppose the algorithm does not reject. Then

The proof is similar to the classical utility analysis of the sum-of-squares program found, e.g., in [30, 26]. We defer it to Appendix I.

Together, the above results imply that the vectors ν ( i ) computed by the algorithm are close to the true centers of the mixture.

Lemma D.16 (Closeness to true centers) . Consider the settings of Theorem D.3. Suppose Y is a good set as per Definition D.6. Let W ∈ W ( Y ) be the matrix computed by Algorithm D.5. Suppose the algorithm does not reject in step 3. Then for each ℓ ∈ [ k ] , there exists n k · ( 1 -2 k 47 ) indices i ∈ [ n ] , such that

Proof. We aim to show that for most indices i ∈ [ n ] the vectors ∥ ϕ ( W i ) ∥ -1 1 ϕ ( W i ) and ∥ W ∗ i ∥ -1 1 W ∗ i induce a 2 -explicitly 40 -bounded distribution over Y . If additionally the two vectors are close in ℓ 1 -norm, the result will follow by Theorem A.21.

Note that ∥ W ∗ i ∥ -1 1 W ∗ i induces a 2 -explicitly 40 -bounded distribution by Lemma D.7. By Markov's inequality and Lemma D.15 there can be at most n/k 48 indices j ∈ [ n ] such that

Consider all remaining indices i . It follows that

Hence, by Lemma D.11 the distribution induced by ∥ ϕ ( W i ) ∥ -1 1 ϕ ( W i ) is 2 -explicitly 40 -bounded distribution. Further, using ∥ W ∗ i ∥ 1 ⩾ n k ( 1 -k n 0 . 4 ) we can bound

Hence, by Theorem A.21 for each l ∈ [ k ] there are at least n k -n 0 . 6 -n k 48 ⩾ n k · ( 1 -2 k 47 ) indices i such that

The result now follows by standard concentration bounds applied to the distribution induced by ∥ W ∗ i ∥ -1 1 W ∗ i .

An immediate consequence of Lemma D.16 is that the vectors ¯ ν ( i ) inherits the good properties of the vectors ν ( i ) with high probability.

Corollary D.17 (Closeness to true centers after sub-sampling) . Consider the settings of Theorem D.3. Suppose Y is a good set as per Definition D.6. Let W ∈ W ( Y ) be the matrix computed by Algorithm D.5. Suppose the algorithm does not reject. Then with high probability for each ℓ ∈ [ k ] , there exists n 0 . 01 k · ( 1 -150 k 47 ) indices i ∈ S , such that

Proof. For each ℓ ∈ [ k ] , denote by T ℓ the set of indices in [ n ] satisfying

By Lemma D.16 we know that T ℓ has size at least n k · ( 1 -2 k 47 ) . Further, let S be the set of indices selected by the algorithm. By Chernoff's bound Fact E.4 with probability 1 -e -n Ω(1) , we have | S ∩ T ℓ | ⩾ n 0 . 01 k · ( 1 -150 k 47 ) . Taking a union bound over all ℓ ∈ [ k ] we get that with probability 1 -e -n Ω(1) , for each ℓ ∈ [ k ] , there exists n 0 . 01 k · ( 1 -150 k 47 ) indices i ∈ S such that

Now, we obtain the corollary observing (cf. Fact E.1 with m = 1 ) that with probability at least 1 -e -n Ω(1) , for all i ∈ S

For each ℓ , denote by G ℓ ⊆ S the set of indices i ∈ S satisfying

Let G := ⋃ ℓ ∈ [ k ] G ℓ . We now have all the tools to prove utility of Algorithm D.5. We achieve this by showing thst with high probability, each bin returned by the algorithm at step 7 satisfies G ℓ ′ ⊆ B ℓ for some ℓ , ℓ ′ ∈ [ k ] . Choosing the bins small enough will yield the desired result.

Lemma D.18 (Closeness of estimates) . Consider the settings of Theorem D.3. Suppose Y is a good set as per Definition D.6. Let W ∈ W ( Y ) be the matrix computed by Algorithm D.5. Suppose the algorithm does not reject. Then with high probability, there exists a permutation π : [ k ] → [ k ] such that

Proof. Consider distinct ℓ, ℓ ′ ∈ [ k ] . By Corollary D.17 for each ¯ ν ( i ) , ¯ ν ( j ) ∈ G ℓ it holds that

for some universal constant C > 0 . Moreover, by assumption on µ ℓ , µ ℓ ′ for each ¯ ν ( i ) ∈ G ℓ and ¯ ν ( j ) ∈ G ℓ ′

̸

Thus, by Lemma E.5 with probability at least 1 -e Ω( d ∗ ) ⩾ 1 -n -100 it holds that or each ¯ ν ( i ) , ¯ ν ( j ) ∈ G ℓ and ¯ ν r ∈ G ℓ ′ with ℓ ′ = ℓ ,

for some other universal constant C ∗ > C . Let Q Φ ( G ℓ ) ⊆ R d ∗ be a ball of radius C ∗ · ( k -25 ) such that ∀ i ∈ G ℓ it holds Φ ¯ ν ( i ) ∈ Q Φ ( G ℓ ) . That is, Q Φ ( G ℓ ) contains the projection of all points in G ℓ .

Recall that d ∗ = 100log( n ) ⩽ 100 k 5 and b = k -15 . Let B = { B i } ∞ i =1 be the sequence of bins computed by the histogram learner of Lemma A.13 for R d ∗ at step 7 of the algorithm. By choice of b , and since q is chosen uniformly at random in [0 , b ] , the probability that there exists a bin B ∈ B containing Q Φ ( G ℓ ) is at least

where we used that d ∗ = 100log n ⩽ 100 k 5 . A simple union bound over ℓ ∈ [ k ] yields that with high probability for all ℓ ∈ [ k ] , there exists B ∈ B such that Q Φ ( G ℓ ) ⊆ B . For simplicity, denote such bin by B ℓ .

We continue our analysis conditioning on the above events, happening with high probability. First, notice that for all l ∈ [ k ]

and thus there cannot be ℓ, ℓ ′ ∈ [ k ] such that Q Φ ( G ℓ ) ⊆ B ℓ and Q Φ ( G ′ ℓ ) ⊆ B ℓ . Moreover, by Corollary D.17 and

and hence it must be that step 7 returned bins B 1 , . . . , B k . This also implies that the algorithm does not reject. Further, by Lemma E.5 for all ¯ ν ( i ) , ¯ ν ( j ) such that Φ ¯ ν ( i ) , Φ ¯ ν ( j ) ∈ B l it holds that

And hence, by triangle inequality, we get

Finally, recall that for each ℓ ∈ [ k ] ,

where w ′ ∼ N ( 0 , N ( 0 , 32 · k -120 · log(2 kn/δ ) ε 2 · Id )) . Since by choice of n, k, ε it holds that

we get with probability at least 1 -e -k Ω(1) for each ℓ ∈ [ k ] , by Fact E.1, with m = 1 , and a union bound that

Since all ¯ ν ( i ) such that Φ ¯ ν ( i ) ∈ B l are at most O ( k -12 ) -far from µ l , also their average is. We conclude that

This completes the proof.

Now Theorem D.3 is a trivial consequence.

Proof of Theorem D.3. The error guarantees and privacy guarantees immediately follows combining Lemma D.8, Lemma D.15, Lemma D.14 and Lemma D.18. The running time follows by Fact A.16.

E Concentration inequalities

We introduce here several useful and standard concentration inequalities.

Fact E.1 (Concentration of spectral norm of Gaussian matrices) . Let W ∼ N (0 , 1) m × n . Then for any t , we have

where σ min ( · ) and σ max ( · ) denote the minimum and the maximum singular values of a matrix, respectively.

Let W ′ be an n -byn symmetric matrix with independent entries sampled from N (0 , σ 2 ) . Then ∥ W ′ ∥ ⩽ 3 σ √ n with probability at least 1 -exp( -Ω( n )) .

Fact E.2 (Maximum degree of Erd˝ os-Rényi graphs) . Let G be an Erd˝ os-Rényi graph on n vertices with edge probability p . Then with probability at least 1 -n exp( -np/ 3) , any vertex in G has degree at most 2 np .

Fact E.3 (Gaussian concentration bounds) . Let X ∼ N (0 , σ 2 ) . Then for any t ⩾ 0 ,

Fact E.4 (Chernoff bound) . Let X 1 , . . . , X n be independent random variables taking values in { 0 , 1 } . Let X := ∑ n i =1 X i and let µ := E X . Then for any δ > 0 ,

Lemma E.5 ([31]) . Let Φ be a d -byn Gaussian matrix, with each entry independently chosen from N (0 , 1 /d ) . Then, for every vector u ∈ R n and every α ∈ (0 , 1)

F Linear algebra

Lemma F.1 (Weyl's inequality) . Let A and B be symmetric matrices. Let R = A -B . Let α 1 ⩾ · · · ⩾ α n be the eigenvalues of A . Let β 1 ⩾ · · · ⩾ β n be the eigenvalues of B . Then for each i ∈ [ n ] ,

Lemma F.2 (Davis-Kahan's theorem) . Let A and B be symmetric matrices. Let R = A -B . Let α 1 ⩾ · · · ⩾ α n be the eigenvalues of A with corresponding eigenvectors v 1 , . . . , v n . Let β 1 ⩾ · · · ⩾ β n be the eigenvalues of B with corresponding eigenvectors u 1 , . . . , u n . Let θ i be the angle between ± v i and ± u i . Then for each i ∈ [ n ] ,

̸

G Convex optimization

Proposition G.1. Let f : R m → R be a convex function. Let K ⊆ R m be a convex set. Then y ∗ ∈ K is a minimizer of f over K if and only if there exists a subgradient g ∈ ∂f ( y ∗ ) such that

Proof. Define indicator function

Then for y ∈ K , one has

Note y ∗ is a minimizer of f over K , if and only if y ∗ is a minimizer of f + I K over R m , if and only if 0 m ∈ ∂ ( f + I K )( y ∗ ) = ∂f ( y ∗ ) + ∂I K ( y ∗ ) , if and only if there exists g ∈ ∂f ( y ∗ ) such that ⟨ g, y -y ∗ ⟩ ⩾ 0 for any y ∈ K .

Proposition G.2 (Pythagorean theorem from strong convexity) . Let f : R m → R be a convex function. Let K ⊆ R m be a convex set. Suppose f is κ -strongly convex over K . Let x ∗ ∈ K be a minimizer of f over K . Then for any x ∈ K , one has

Proof. By strong convexity, for any subgradient g ∈ ∂f ( x ∗ ) one has

By Proposition G.1, ⟨ x -x ∗ , g ⟩ ⩾ 0 for some g ∈ ∂f ( x ∗ ) . Then the result follows.

H Deferred proofs SBM

We prove Lemma C.9 restated below.

Lemma H.1 (Restatement of Lemma C.9) . Consider the settings of Lemma C.8. With probability 1 -exp( -Ω( n )) over G ∼ SBM n ( γ, d, x ) ,

Proof. Recall K = { X ∈ R n × n : X ⪰ 0 , X ii = 1 /n ∀ i } . Let X ∗ := 1 n xx ⊤ . Since ˆ X = ˆ X ( Y ( G )) is a minimizer of min X ∈K ∥ Y ( G ) -X ∥ 2 F and X ∗ ∈ K , we have

The infinity-to-one norm of a matrix M ∈ R m × n is defined as

By [28, Fact 3.2], every Z ∈ K satisfies

where K G ⩽ 1 . 783 is Grothendieck's constant. Similar to the proof of [28, Lemma 4.1], using Bernstein's inequality and union bound, we can show (cf. Fact H.2)

with probability 1 -exp( -Ω( n )) . Putting things together, we have

with probability 1 -exp( -Ω( n )) .

Fact H.2. Let γ > 0 , d ∈ N , x ∗ ∈ {± 1 } n , and G ∼ SBM ( γ, d, x ∗ ) . Let Y ( G ) = 1 γd ( A ( G ) -d n J ) , where A (( G )) is the adjacency matrix of ( G ) with entries d/n on the diagonal. Then

with probability at least 1 -e -10 n .

Proof. The result will follow using Bernstein's Inequality and a union bound. Define E := Y ( G ) -1 n x ∗ ( x ∗ ) ⊤ . Fix x ∈ {± 1 } n and for 1 ⩽ i < j ⩽ n , let Z i,j := E i,j x i x j . Then x ⊤ E x = 2 ∑ 1 ⩽ i<j ⩽ n Z i,j . Note that

By Bernstein's Inequality (cf. [62, Proposition 2.14]) it follows that

Hence, by a union bound over all x ∈ {± 1 } n it follows that

with probability at least 1 -e -10 n .

I Deferred proofs for clustering

In this section, we will prove Lemma D.15 restated below.

Lemma (Restatement of Lemma D.15) . Consider the settings of Theorem D.3. Suppose Y is a good set as per Definition D.6. Let W ( Y ) ∈ W ( Y ) be the matrix computed by Algorithm D.5. Suppose the algorithm does not reject. Then

We will need the following fact about our clustering program. Similar facts where used, e.g., in [30, 26]. One difference for us is that we don't have a constraint on the lower bound on the cluster size indicated by our SOS variables. However, since we maximize a variant of the ℓ 1 norm of the second moment matrix of the pseudo-distribution this will make up for this.

Fact I.1. Consider the same setting as in Lemma D.15. Let 0 < δ ⩽ 1 1 . 5 · 10 10 · 1 k 201 and denote by C 1 , . . . , C k ⊆ [ n ] the indices belonging to each true cluster. Then W ( Y ) satisfies the following three properties:

For all i, j ∈ [ n ] it holds that 0 ⩽ W i,j ⩽ 1 ,

for all i ∈ [ n ] it holds that ∑ n j =1 W i,j ⩽ n k and for at least (1 -1 1000 k 100 ) n indices i ∈ [ n ] it holds that ∑ n j =1 W i,j ⩾ (1 -1 (10) 6 k 200 ) · n k ,

for all r ∈ [ k ] it holds that ∑ i ∈ C r ,j ̸∈ C r W i,j ⩽ δ · n 2 k .

We will prove Fact I.1 at the end of this section. With this in hand, we can proof Lemma D.15.

Proof of Lemma D.15. For brevity, we write W = W ( Y ) . Since ϕ ( W ∗ ) = W ∗ and ϕ is 10Lipschitz we can also bound

Let δ ⩽ 1 1 . 5 · 10 10 · 1 k 201 and again let C 1 , . . . , C k ⊆ [ n ] denote the indices belonging to each true cluster. Note that by assumption that Y is a good sample it holds for each r ∈ [ k ] that n k -n 0 . 6 ⩽ | C r | ⩽ n k + n 0 . 6 .

Let r, r ′ ∈ [ k ] . We can write

Note that we can bound the second sum by k · δ n 2 k using Item 3. Further, in what follows consider only indices i such that ∑ n j =1 W i,j ⩾ (1 -1 (10) 6 k 200 ) · n k . By Item 2 we can bound the contribution of the other indices by

Focusing only on such indices, for the first sum in Eq. (I.1), fix r ∈ [ k ] . We will aim to show that most entries of W are large if and only if the corresponding entry of W ∗ is 1. By Item 3 and Markov's Inequality, it follows that for at least a (1 -1 1000 k 100 ) -fraction of the indices i ∈ C r it holds that

where we used that | C r | ⩾ n k -n 0 . 6 . Call such indices good . Notice that for good indices it follows using Item 2 that

Denote by G the number of j ∈ C r such that W i,j ⩾ 1 -1 1000 k 100 . Using the previous display and that W i,j ⩽ 1 we obtain

where we also used | C r | ⩽ n k + n 0 . 6 . Rearranging now yields

We can now bound

Putting everything together, it follows that

It remains to verify Fact I.1.

Proof of Fact I.1. Let P = P n,k,t ( Y ) be the system of Eq. ( P n,k,t ( Y ) ). Recall that W i,j = ˜ E ∑ l ∈ [ k ] z i,l z j,l . Since

it follows that 0 ⩽ W i,j ⩽ 1 . Further, for each i ∈ [ n ] it holds that

implying that ∑ j ∈ [ n ] W i,j ⩽ n k . Further, by Lemma D.14

Denote by W i the i -th row of W and by L the number of rows which have ℓ 1 norm at least (1 -1 (10) 6 k 200 ) · n k . Since for all i it holds that ∥ W i ∥ 1 ⩽ n k it follows that

Rearranging then yields L ⩾ (1 -1 1000 k 100 ) · n which proofs Item 2.

It remains to verify Item 3. Fix r, l ∈ [ k ] and define z l ( C r ) = k n ∑ i ∈ C r z i,l . Let t > 0 be an integer. We aim to show that for all unit vectors v it holds that

̸

̸

where ∆ is the minimal separation between the true means. Before proving this, let us examine how we can use this fact to prove Item 3. Note, that for all r = r ′ it holds that

Hence, if the above SOS proof indeed exists, we obtain

̸

̸

In the remainder of this proof we will prove Eq. (I.2). We will use the following SOS version of the triangle Inequality (cf. Fact I.2)

Recall that µ ′ l = k n ∑ n i =1 z i,l y i and denote by µ π ( i ) the true mean corresponding to the i -th sample. Let v be an arbitrary unit vector, it follows that

̸

̸

where we used that P 1 ∑ k r =1 z l ( C r ) ⩽ 1 . Using the SOS triangle inequality again and that P 2 z i,l ⩽ 1 we obtain

̸

We start by bounding the first sum. Recall that by assumption the uniform distribution over each true cluster is 2 t -explicitly 2-bounded. It follows that

where we used that | C r | ⩽ n k + n 0 . 6 . To bound the second sum, we will use the moment bound constraints. In particular, we know that

Combining Eq. (I.4) and Eq. (I.5) now yields

̸

Note that by assumption ∆ ⩾ O ( √ tk 1 /t ) . Overloading notation, we can choose the t parameter in the SOS proof to be 202 times the t parameter in the lower bound in the separation to obtain 35

I.1 Small Lemmas

Fact I.2 (Lemma A.2 in [36]) . For all integers t > 0 it holds that

Fact I.3. Let ε, δ > 0 . Let M : Y → O be a randomized algorithm that, for every pair of adjacent inputs, with probability at least 1 -γ ⩾ 1 / 2 over the internal randomness of Y 36 satisfies ( ε, δ ) -privacy. Then M is ( ε +2 γ, δ + γ ) -private.

Proof. Let X,X ′ be adjacent input and let B be the event under which M is ( ε, δ ) -private. By assumption, we know that P ( B ) ⩾ 1 -γ . Let S ∈ O , it follows that

where we used that log(1 -γ ) ⩾ -2 γ for γ ∈ [0 , 1 / 2] .

I.2 Privatizing input using the Gaussian Mechanism

In this section, we will proof the following helpful lemma used in the privacy analysis of our clustering algorithm (Algorithm D.5). In summary, it says that when restricted to some set our input has small ℓ 2 sensitivity, we can first add Gaussian noise proportional to this sensitivity and afterwards treat this part of the input as "privatized". In particular, for the remainder of the privacy analysis we can treat this part as the same on adjacent inputs. Note that we phrase the lemma in terms of matrix inputs since this is what we use in our application. Of course, it also holds for more general inputs.

Lemma I.4. Let V, V ′ ∈ R n × d , m ∈ [ n ] and ∆ > 0 be such that there exists a set S of size at least n -m satisfying

where V i , V ′ i denote the rows of V, V ′ , respectively. Let A 2 : R n × d → O be an algorithm that is ( ε 2 , δ 2 ) -differentially private in the standard sense, i.e,., for all sets S ⊆ O and datasets X,X ′ ∈ : R n × d differing only in a single row it holds that

Further, let A 1 : R n × d → R n × d be the Gaussian Mechanism with parameters ∆ , ε 1 , δ 1 . I.e., on input M it samples W ∼ N ( 0 , 2∆ 2 · log(2 /δ 1 ) ε 2 1 ) n × d and outputs M + W .

Then for

A 2 ◦A 1 is ( ε , δ ) -differentially private with respect to V and V , i.e., for all sets S ⊆ O it holds that

Proof. Without loss of generality, assume that S = { 1 , . . . , m } . Denote by V 1 , V 2 the first m and last n -m rows of V respectively. Analogously for V ′ 1 , V ′ 2 . We will later partitin the noise W of the Gaussian mechanism in the same way. Further, for a subset A of R n × n and Y ∈ R m × n define

Note that ( X Y ) ∈ A if and only if X ∈ T A,Y .

Let S ⊆ O . It now follows that

where the inequality follows by the guarantees of the Gaussian Mechanism. Further, we can bound

where the inequality follows by the privacy guarantees of A 2 combined with standard group privacy arguments.

Putting the above two displays together and plugging in the definition of ε ′ , δ ′ we finally obtain

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors investigate graph clustering with the Stochastic Block Model (SBM) and metric clustering with the Gaussian Mixture Model (GMM) within a novel privacy-preserving framework. They propose differentially private (DP) algorithms for both problems, which offer significantly improved performance guarantees compared to state-of-the-art DP algorithms.\n\nSpecifically, the authors present efficient  $(\\epsilon,\\delta)$-differentially private SBM algorithms for both weak recovery and exact recovery. These algorithms operate in polynomial time, outperforming the previous quasi-polynomial time DP algorithms. Additionally, they provide a lower bound that demonstrates the optimality of the privacy guarantees from an information-theoretic perspective.\n\nFurthermore, the authors develop private algorithms for learning the mixture of $k$ spherical Gaussians under the condition that the minimum separation is at least $O(k^{1/t}\\sqrt{t})$. Notably, these algorithms do not require an additional additive term of $\\Omega(\\sqrt{\\log n})$ in the minimum separation or an explicit upper bound on the Euclidean norm of the centers, which were assumed in prior works.\n\nThe proposed algorithms are built upon a simple yet powerful connection between robustness and privacy. The authors formalize this connection as a framework for transforming robust estimation algorithms into private algorithms. They show that if two strongly convex functions over constrained sets, where both the function and the set may depend on the input, are point-wise close, their minimizers are also close. Conversely, they establish that projections of points that are close to each other onto convex sets that are point-wise close must also be close.\n\nSTRENGTHS:\nThe paper is well-written and addresses fundamental clustering problems, including SBM and GMM, providing novel differentially private algorithms that make remarkable advancements in the field. The lower bound also demonstrates the optimality of the privacy guarantees of the SBM algorithm.\n\nMoreover, their techniques of translating robust estimation algorithms to DP algorithms seem readily extended to some other estimation algorithms. Though the ideas are intuitively simple, the formal connections and applications to clustering algorithms are highly nontrivial.\n\nWEAKNESSES:\nThere is no experiment in the paper (but it is a strong theory paper).\n\nQUESTIONS:\nNo.\n\nLIMITATIONS:\nIt might be useful to briefly discuss why the current framework does not achieve $(\\log k)^{c+1/2}$ separation in polynomial time.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a general framework for designing efficient private estimation algorithms from robust non-private algorithms. Their approach, based on a certain stability analysis for convex optimization, is applied to weak and exact recovery in the stochastic block model (SBM) and for learning mixtures of spherical Gaussians. They obtain the first efficient $(\\varepsilon,\\delta)$-private algorithms for estimation in the first setting (with a new lower bound suggesting that there is little room for improvement), as well as one for the second setting which allows for far lower separation between centers than before.\n\nSTRENGTHS:\n- The approach introduced in this paper appears very useful and flexible, and I anticipate future work building off this approach to address a variety of high-dimensional private estimation tasks\n- They make significant progress on two important problems in private estimation\n- Presentation is strong, prior work and guarantees are clearly delineated, their results are clearly stated, and, while there is a bit of technicality in the second application, the authors build up to it nicely with the first SBM results\n- The Gaussian mixture application requires some clever sum-of-squares analysis. More broadly, there is certainly technical depth to this work beyond the general framework\n\nWEAKNESSES:\n- I sometimes lost track of the role of different parameters. I think some may be obvious to those more familiar with this subfield of privacy, and the many special cases mentioned in the discussion were helpful. In any extended version of this work, I would suggest adding more discussion of terms appearing in bounds from the classic (non-private) settings.\n- The choice of objective function in the Gaussian mixture application wasn't clear to me - where does the all ones matrix come from?\n- A fair bit of technical detail was pushed to the supplement, which was a bit tricky to parse. In particular, while the high level approach was clear to me, I was unable to verify correctness for the Gaussian mixture proofs (though I mostly blame this on my unfamiliarity with the SoS program they consider)\n\nMinor nits:\n- On lines 207-208, it was unclear to me why $k \\geq \\sqrt{\\log d}$ should be considered a high-dimensional regime\n- On line 272, the objective function $f$ is not defined (though clear enough from context)\n- \"the basic\" repeated on line 360\n\nQUESTIONS:\nIn addition to those above:\n- Are the authors aware of their sensitivity analysis appearing in any previous convex programming literature? It is simple enough that I am a bit surprised it was not previously known. For instance, if their Condition (ii) was replaced with a Hausdorff distance bound between K(Y) and K(Y'), I think this is essentially folklore (though of course Condition (ii) is more sophisticated than this).\n\nLIMITATIONS:\nThe authors were clear about all assumptions and I don't anticipate any negative impacts.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies private parameter estimation in certain high-dimensional statistical inference tasks. In particular, community detection in stochastic block models (where the privacy is with respect to edge changes) and estimation of unknown means in isotropic Gaussian mixture models.\nAlthough these problems have been studied in the literature, the paper provides new algorithms with improved sample and computational complexities, which closely match that of non-private estimation.\nAlthough seemingly disjoint, the paper's results follow a common methodology of privatizing the existing non-private algorithms that rely on (strongly) convex optimization.\n\nSTRENGTHS:\nThe paper proposes a generic privatization procedure relying on strongly convex optimization, and then the paper applies it to two problems of interest, leading to improved sample and computational complexities. The paper's results are novel and interesting.\n\nWEAKNESSES:\nI have the following comments:\n\n1. The paper's writing needs to be improved and it seems that the submitted version is missing many details. For example, the notation for the indicator function is not displayed due to LaTeX error and many algorithmic environments are referenced as theorems by cleveref.\n\n\n2. The comparison with work of Kothari-Manurangsi-Velingker is still not clear. Perhaps the paper should begin by explaining the similarities and then the differences.\n\nPlease also see the comments below.\n\nQUESTIONS:\n# Major Comments\n\n1. I suggest adding the notion of privacy (node/sample differential privacy) in the theorem statements.\n\n2. Page, Line 45: \"However, for the more challenging task of graph clustering, node-DP is sometimes impossible to achieve.\"\n    I do not follow. For the SBM, I believe that node-DP is possible to achieve since the information is not concentrated on a single node.\n\n3. Footnote 11: I suggest explicitly adding the notion of privacy in the theorem statement (see the comment above as well).\n\n\n4. Line 259: Where is the first step (i) from Line 253 used in the result here? Does it not follow from (ii) alone? Moreover, what if $K(Y)$ is empty?\n\n5. Line 270, The convex program for SBM. The program from the citation [24] is different from the one displayed here. Especially, it is linear in X and not quadratic. Please clarify.\n\n# Other Comments\n\n1. Page 2, Line 49: What is meant by \"non-robust setting\"\n\n\n\n3. Page 2, Line 64: I believe there are more representative papers for outlier robust estimation.\n\n4. Page 3, Model 1.1: Please give intuition about the parameters in the model, for example, d is the average degree.\n\n5. Theorem 1.3: Please add the weak recovery guarantee in the theorem statement.\n\n6. Line 266: Describe explicitly the relation between the input and Y\n\n3. Lines 182-188: I found the whole paragraph confusing. I suggest rephrasing.\n\n4. Theorem 1.6: Why does $k$ need to be large?\n\n6. Line 210: Put $t^*$ in the runtime or say that it is constant.\n\n7. Line 250-251 Please add citations here \"Our sensitivity bound is simple, yet it generalizes previously known bounds for strongly convex optimization problems\"\n\n8. Line 277: This was confusing and what is the norm $\\|\\cdot\\|_1?$. Please rephrase and explain that the function $f$ does not contain the quadratic in $Y$. Although it does not affect the optimization problem, but it is important for the sensitivity analysis.\n\n9. Line 312: \"While this program suggests a path to recover the first property,...\" Is it even feasible?\n\n10. Line 321: Briefly explain that it is so by setting $z_i$ to zero.\n\n11. Line 341-347: This was confusing for me. What histograms are being referred to here? I understand the space might have been an issue originally but the accepted version can be longer.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problems of weak and strong recovery for stochastic block models, as well as the problem of estimating the means of Gaussian Mixture Models with sphrerical Gaussians, under the constraint of differential privacy. Their contributions include:\n\n* A computationally efficient algorithm that performs strong recovery for the stochastic block model under $(\\varepsilon, \\delta)$-edge privacy.\n* A computationally inefficient algorithm that performs both weak and strong recovery for the stochastic block model under $\\varepsilon$-edge privacy, as well as a matching information-theoretic lower bound.\n* A computationally efficient algorithm for estimating the means of mixtures of spherical Gaussians under $(\\varepsilon, \\delta)$-DP.\n\nThe algorithms are all derived by applying a general framework that is developed as part of the work which involves privatizing the solutions of strongly convex optimization programs. Additionally, the results given in the paper improve upon the prior work of Seif et. al. and Kamath et. al. on SBMs and GMMs, respectively.\n\nSTRENGTHS:\nAs mentioned in my summary, the paper provides a general framework for privatizing the solutions of convex programs, which they apply in a multitude of settings. I consider this to be a significant contribution in the private statistics literature. In addition to that, the paper is very clearly written and the description of the techniques if quite accessible.\n\nWEAKNESSES:\nNo concrete weaknesses.\n\nQUESTIONS:\nThere has been recent work by Arbas et. al. on private estimation of GMMs (see https://arxiv.org/abs/2303.04288). I read both works and understand that they go in different directions (meaning that the work of Arbas et. al. doesn't supersede the present paper). However, unless I missed something the paper is not mentioned at all in the related work section (or anywhere else in the submitted pdfs), which feels like an important ommission. Thus, it'd be great if the authors could provide a comparison of their work with that paper (and also include it in their discussion of related work).\n\nLIMITATIONS:\nThis is a purely theoretical work with no obvious societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-08-01"}