Private estimation algorithms for stochastic block models and mixture models
Authors: Hongjie Chen
Vincent Cohen-Addad
Tommaso d'Orsi
Alessandro Epasto
Jacob Imola
David Steurer
Stefan Tiegel
Venue: NeurIPS 2023 spotlight (2023)
Keywords: differential privacy
stochastic block model
Gaussian mixture model
sum of squares

Abstract: We introduce general tools for designing efficient private estimation algorithms, in the high-dimensional settings, whose statistical guarantees almost match those of the best known non-private algorithms.
To illustrate our techniques, we consider two problems: recovery of stochastic block models and learning mixtures of spherical Gaussians.

For the former, we present the first efficient $(\epsilon, \delta)$-differentially private algorithm for both weak recovery and exact recovery. Previously known algorithms achieving comparable guarantees required quasi-polynomial time. 

For the latter, we design an  $(\epsilon, \delta)$-differentially private algorithm that recovers the centers of the $k$-mixture when the minimum separation is at least $	O(k^{1/t}\sqrt{t})$. For all choices of $t$, this algorithm requires sample complexity $n\geq k^{O(1)}d^{O(t)}$ and time complexity $(nd)^{O(t)}$. Prior work required either an additional additive $\Omega(\sqrt{\log n})$ term in the minimum separation or an explicit upper bound on the Euclidean norm of the centers.

Private estimation algorithms for stochastic block models and mixture models
Hongjie Chen ETH Zürich
Vincent Cohen-Addad Google Research
Tommaso d'Orsi Bocconi
Alessandro Epasto Google Research
Jacob Imola UC San Diego
David Steurer
Stefan Tiegel ETH Zürich
ETH Zürich
Abstract
We introduce general tools for designing efficient private estimation algorithms, the high-dimensional settings, whose statistical guarantees almost match those of the best known non-private algorithms. To illustrate our techniques, we consider problems: recovery of stochastic block models and learning mixtures of spherical Gaussians.
For the former, we present the first efficient (s, 8) diifferrrntill private algorithms for both weak recovery and exact recovery. Previously known algorithms achieving comparable guarantees required mousi-polynomial time. We complement these sults with an information-theoretiee lower bound that highlights how the guarantees of our algorithms are almost tight.
For the latter, we design an (c, 6) diffeeerrentlllly private algorithm that recovers the centers of the k-mixture when the minimum separation least Ot Vt). For all choices of I, this algorithm requires sample complexity n > 10(1) and time complexity (nt) Prior work required either an additional additive 9 Vllloo n) term in the minimum separation or an explicit upper bound on the Euclidean norm of the centers.
Introduction
Computing model that best matches dataset fundamental question in machine learning and statistics. Given set of n samples from model, how to find the most likely parameters of the model that could have generated this data? This basic question has been widely studied for several decades, and recently revisited in the context where the input data has been partially corrupted (i.e., where few samples of the data have been adversarially generated- see for instance [37
18 22 20)
This has led to several recent works shedding new lights on classic model estimation problems, such as the Stochastic Block Model (SBM) 228
46 44
25 21
and the Gaussian Mixture Model
(GMM) 30 36 9
[0]
(see Definitions I.I and 1.2
Privacy in machine learning and statistical tasks has recently become of critical importance. New regulations, renewed consumer interest well privacy leaks, have led the major actors to adopt rawacc-preservvng solutions for the machine learning [T] 2 (3. This new push has resulted in a flurry of activity in algorithm design for private machine learning. including very recently for SBMs and GMMs [55 32 16 60). Despite this activity, it has remain an open challenge to fully understand how
privacy requirements impact model estimation problems and in particular their recovery thresholds and the computational complexity. This the problem we tackle in this paper.
Much of this work was done while the author was at ETH Zürich.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

While other notions of privacy exist (e.g. k-anonymity), the de facto privacy standard is the differential privacy (DP) framework of Dwork, McSherry, Nissim, and Smith [24]. In this framework, the privacy quality is governed by two parameters, e and 6, which in essence tell us how the probability of seeing a given output changes (both multiplicatively and additively) between two datasets that differ by any individual data element. This notion, in essence, quantifies the amount of information leaked by given algorithm on individual data elements. The goal of the algorithm designer is to come up with differentially private algorithms for being a small constant and ó being of order 1/n
Differentially private analysis of graphs usually considers two notions of neighboring graphs. The weaker notion of edge-DD defines two graphs to be neighboring if they differ in one edge. Under the stronger notion of node-DP, two neighboring graphs can differ arbitrarily in the set of edges connected to a single vertex. Recently, there is line of work node-DP parameter estimation in random graph models, c.g. Penös--Réniiie models [61] and Graphons [12. 13]. However, for the more challenging task of graph clustering, node-DP is sometimes impossible to achieve Thus is natural first step to study edge-DP graph clustering.
Very recently, Seif, Nguyen, Vullikanti, and Tandon private algorithms for the Stochastic Block Model, with edge-DP. Concretely, they propose algorithms achieving exact recovery (exact identification of the planted clustering) while preserving privacy of individual edges of the graph. The proposed approach either takes n (logg)) time when is constant, or runs in polynomial time when E is 22 n))
551 were the first to propose differentially
Gaussian Mixture Models have also been studied in the context of differential privacy by 32 16
60
using the sasaammpl-anndaggeeeee robust moment estimation in the differential privacy setting [35 g 29) The works of 332 [6]
framework first introduced in [52 (see also recent work for
require an explicit bound on the euclidean norm of the centers as the sample complexity of these algorithms depends on this bound. For a mixture of & Gaussians, if there is a non-private algorithm that requires the minimum distance between the centers to be at least A, then [16 (60) can transform this non-private algorithm into a private one that needs the minimum distance between the centers to be at least + vlog n, where n is the number of samples.
In this paper, we tackle both clustering problems (graph clustering with the SBM and metric clustering with the GMM) through a new general closer to the state-of-the- of non-private algorithms. As we will see, our new perspective on the problems appear to easily extendable to many other estimation algorithms.
framework that brings us significantly
From robustness to privacy In recent years large body of work (see 19 22
37
14]
36
30
and references therein) has advanced our understanding of parameter estimation in the presence of adversarial perturbations. In these settings, an adversary looks at the input instance and modifies it arbitrarily, under some constraints (these constraints are usually meant to ensure that it is still information theoretically possible to recover the underlying structure). As observed in the past machine learning models and robust model estimation are tightly related. The common objective is to design algorithms that extract global information without over-relaying on individual data samples.
23 35 4I), the two goals of designing
Concretely, robust parameter estimation tends to morally follow a two-steps process: (i) argue that typical inputs are well-behaved, in the sense that they satisfy some property which can be used to accurately infer the desired global information, (ii) show that adversarial perturbations cannot significantly alter the quality of well-behaved inputs, so that it still possible to obtain an accurate estimate. Conceptually, the analysis of private estimation algorithms can also be divided in two parts: utility, which is concerned with the accuracy of the output, and privacy, which ensures there no leak of sensitive information. In particular, the canonical differential privacy definition can be interpreted as the reguirement that, for any distinct inputs Y, Y", the change in the output is proportional to the distance between Y and Y".
It is easy see this as generalization of robustness: while robust algorithm needs the output to be stable for typical inputs, private algorithms requires this stability for any possible input. Then,
In particular. we cannot hope to achieve exact recovery. We could isolate vertex by removing all of its adjacent edges. Then is impossible to cluster this vertex correctly. See below for formal definitions. The notion of distance is inherently application dependent. For example, it could be Hamming distance.
2

---

stability of the output immediately implies adding small amount of noise to the output yields privacy. If the added noise is small enough, then utility is also preserved.
Our work further tightens this connection between robustmess and privacy through a simple yet crucial insight: if two strongly convex functions over constrainted sets -where both the function and the set may depend on the input- are point-wise close (say in a (g-sense), their minimizers are also close (in the same sense). The alternative perspective is that projections of points that are close to each other, onto convex sets that are point-wise close, must also be close. This observation subsumes previously known sensitivity bounds in the empirical risk minimization literature (in particular in the sutput-perturbation approach to ERM, see Section 2 for a comparison).
The result is a clean, user-friendly. framework to turn robust estimation algorithmss into private algorithms, while keeping virtually the same guarantees. We apply this paradigm to stochastic block models and Gaussian mixture models, which we introduce next.
Stochastic block model The stochastic block model is an extensively studied statistical model for community detection in graphs (see [4] for a survey).
Model 1.1 (Stochastic block model). In its most basic form, the stochastic block model describes the distribution of an n-vertex graph G ~ SBM,, (d. 7, x), where a vector of n binary labels, d E N. y > 0, and for every pair of distinct vertices i, E [n] the edge {i,3} is independently added to the graph G with probability (1 +1+7- I4 x4-Tg)) xx
For balanced label vector æ; i.e., with roughly the same number of +1's and 1's, parameter d roughly corresponds to the average degree of the graph. Parameter 7y corresponds to the bias introduced by the community structure. Note that for distinct vertices i [n]. the edge { is present in G with probability (1 + + d if the vertices have the same label x;; 2y and with probability (1 7) d if the vertices have different labels x; * z
a
Given a graph G sampled according to this model, the goal is to recover the (unknown) underlying vector of labels as well possible. In particular, for chosen algorithm returning partition (GG) E {+1}", there are two main objective of interest: weak recovery and exact recovery. The former amounts to finding partition ă(G) correlated with the true partition. The latter instead corresponds to actually recovering the true partition with high probability. As shown in the following table, by now the statistical and computational landscape of these problems is well understood 28
can be achieved (and effi- ciently so) iff
Objective
weak recovery
((x,,, (1, > 10-0(1 o(1)
72 > 1
exact recovery
PG--sM,
(GG E {ar -x}
> 1 - o(1)
d log
1
72
> 1
Learning mixtures of spherical Gaussians The Gaussian Mixture Model we consider is the following.
Model 1.2 (Mixtures of spherical Gaussians). Let DI...., Dy be Gaussian distributions on Rd with covariance Td and means (*,,..-.10 lik satisfying [|1s; - ptll > A for any i i j. Given a set Y (yI,...,, }n of n samples from the uniform mixture over Dı,..., DE, estimate pı,.... Pk-
It is known that when the minimum separation is A o(Vlog superpolynomially many samples are required to estimate the means up to small constant error [54]. Just above this threshold, at separation (0(1/)) for any constant 7, there exist efficient algorithms based on the sum-of- squares hierarchy recovering the means up to accuracy 1/ poly (3 36 59) In the regime
*We use bold characters to denote random variables.
"More general versions of the stochastic block model allow for more than two labels and general edge probabilities depending on the label assignment. However, many of the algorithmic phenomena of the general version can their essence already observed for the basic version that we consider in this work. 't times we may write d,. to emphasize that these may be functions of n. We write o(1), w1)) for functions tending zero (resp. infinity) grows.
3

---

where A - ((vlog k) these algorithms yield the same guarantees but require quasipolynomial time. Recently, 39] showed how to efficiently recover the means as long as A = (Olog(k) (o(log())/2+t)) for any constant c 0.
1.1 Results
Stochastic block model We present here the first (e, didfferrrettillly private efficient algorithms for exact recovery. In all our results on stochastic block models, we consider the edge privacy model, in which two input graphs are adjacent if they differ on a single edge (ef. [Definition C.I..
Theorem 1.3 (Private exact recovery of SBM), Let x E {+1}" be balanced) For any 7, d, e, 8 > 0 satisfying
d
(1
1--1--9
((1) and
yd log n
1
1og(1/5) log n 8 )
1
log n
there exists an (e, -idffferrentt edge private algorithm that, on input G ~ SBM, (d, 7, x), returns ((G) E (a,- -x) with probability 1 o(1). Moreover, the algorithm runs polynomial time.
For any constant 0. Theorem 1.3 states that (z, diifferrentllll private exact recovery is possible, in polynomial time, already a constant factor close to the non-private threshold. Previous results [55] could only achieve comparable guarantees in time O(n ((og)))) ) It is also important to observe that the theorem provides a trade-off between signal-to-noise ratio of the instance (captured by the expression on the left-hand side with y, d) and the privacy parameter E In particular, we highlight two regimes: for d > Milog n) one can achieve exact recovery with high probability and privacy parameters 8 R(1) E 0(1/7 1//2). For d b wlog n) o(1), 8 (1) Theorem follows by result for
achieve exact recovery with
high probability and privacy parameters
private weak recovery and a boosting argument (cf. Theorem C.3
and Appendix C.2).
Further, we present second, exponential-times algorithm based on the exponential mechanism 43 which improves over the above in two regards. First, it gives pure differential privacy. Second, provides utility guarantees for a larger range of graph parameters. In fact, we will also prove a lower bound which shows that its privacy guarantees are information theoretically optimal 8 All hidden constants are absolute and do not depend on any graph or privacy parameters unless stated otherwise. In what follows we denote by err .) the minimum of the hamming distance of é and x, and the one of - and x, divided by n.
Theorem 1.4 (Slightly informal, see Theorem C.18 in the supplements for full version). Let yv > (1). x E {=1}" be balanced, and Ç > exp (-St (7²d)). For any & > Ω an algorithm which on input G SBM (7; d, x) outputs estimate ((G) {=1)" satisfying err (f(G), x) & Ç with probability at least 1 - Ç. In addition, the algorithm z-differentially edge
log(1/C)
there exists
private. Further, we can achieve error θ (1/ (///)))))) with probability 1
couple of remarks are in order. First, our algorithm works across all degree-regimes in the literature and matches known non-prin non-private thresholds and rates up to constants @ In particular, for 72d = (L). we achieve weak/partial recovery with either constant or exponentially high success probability. Recall that the optimal non-private threshold is 2d > 1. For the regime, where 72d = (11, it is
known that the optimal error rate is cxp(-(1 2 64 even non-privately which we match up to constants here o(1) denotes function that tends to zero as 7 d tends to infinity. Moreover, our algorithm achieves exact recovery as soon as 22d 22og n) since then < << 4. This also matches
known non- con-private threshholds up to constants 5 47 We remark that (55] gave an e-DP exponential
time algorithm which achieved exact recovery and has inverse polynomial success probability the
utility case as long as E > 22 )) We recover this result as a special case (with slightly worse constants). In fact, their algorithm also based on the exponential mechanism, but their analysis only applies to the setting of exact recovery, while our result holds much more generally. Another
A vector v. E [+1 is said to be balanced if Σ._ xi =0 0.
IIt is optimal the "small error" regime, otherwise is almost optimal. See Theorem C.22 for more detail. For ease of exposition we did not try to optimize these constants.

---

crucial difference is that we show how to privatize known boosting technique frequently used in the non-private setting, allowing us to achieve error guarantees which are optimal up to constant factors.
is natural to ask whether, for a given set of parameters ?: d, C one can obtain better privacy guarantees than Theorem 1.4 Our next result implies that our algorithmic guarantees almost tight.
Theorem 1.5 (Informal, see Theorem C.22 in the supplements for full version). Suppose there exists an e-differentially edge private algorithm such that for any balanced x E [+1])" on input G SBM, (d, outputs (G) E {=1)" satisfying
P ((er(E(G)) x) <5) 21-11-
<
Then,
log(1/() 7d
log(1/n) Çncyd
& 2 Ω
(1.1)
This lower bound tight for e-DP exact recovery. By setting Ç 1/n and rj 1/ poly(n).
Theorem 1.5 implies no s-DP exact recovery algorithm exists for E < O log
There exist E-
d
DP algorthhms (Algorithm C.19 in the supplements and the algorithm in (55)) exactly recover the community for any > S2( "%)).
Notice Theorem 1.5] lower bound for large range of error rates (partial to exact recovery). For failure probability = 6. the lower bound simplifies to 22 ( and hence matches Theorem 1.4 up to constants. For exponentially small failure probability, 7 e'n, it becomes E 2 52
E-d To compare, Theorem 1.4 requires & > 2 <2nd) in this regime, using the substitution
/log(I/()+<5
Further, while formally incomparable, this E-DP lower bound also suggests that the guarantees obtained by our efficient (e. 5) algorithm in Theorem 1.3 might be close to optimal. Note that
setting < = 1 in Theorem 1.5 requires the algorithm to exactly recover the partitioning. In this
setting, Theorem 1.3 implies that there is efficient (c, ) exact recovery algorithm for O(y log
Theorem 1.5
states any E-DP exact recovery algorithm requires e 52( log 7d
7d
Further, for standard privacy parameters that are required for real-world applications, such as e Rs and 8 n --00 Theorem 1.3 requires that yd > ()logn.. Theorem 1.5 shows that for pure-DP algorithms with the same setting of & this is also necessary. We leave it as fascinating open questions bridge the gap between upper and lower bounds in the context of (e, 8DDP.
Learning mixtures of spherical Gaussians Our algorithm for privately learning mixtures of spherical Gaussians provides statistical guarantees matching those of the best known non-private algorithms.
Theorem 1.6 (Privately learning mixtures of spherical Gaussians), Consider an instance Model 1.2 Let t > O be such that A > O (Vik//). For > ft (ko(1) . .d(1)) ,k > (log n) there exists an algorithm, running in time (nd)0() (hat outputs vectors j.:...., fas satisfying
max i - & O(k-12),
KE(K]
with high probability, for some permutation π [k]] [k] Moreover,for > k-10 ,8 1 the algorithm is (e, (e,6)-differentiall)) private for any input
The conditions > 1-10 , n 11 in Theorem 1.6 are not restrictive and should be considered formality. Moreover, setting 0.01 and
already provides meaningful privacy guarantees in practice. The condition that & > (log n) 1/5 is a technical requirement by our proofs.
Prior to this work, known differentially private algorithms could learn a mixture of k-spherical Gaussian either if: (1) they were given ball of radius R containing all centers 32 [6]
(2)
the minimum separation between centers needs an additional additive Q 2(VTog n.) term 16 60
1oTw input datasets are adjacent if they differ single sample. See Definition D. in the supplements.
"In B2 [6] sample complexity of algorithm depends on this radius R. Foror & & o(1) algorithm provides significant improvement vlog k ol o(vlogn).
5

---

To the best of our knowledge, Theorem 1.6 is the first to get the best of both worlds. That is, our algorithm requires no explicit upper bounds on the means (this also means the sample complexity does not depend on R) and only minimal separation assumption OVvlog k). Furthermore, we remark that while previous results only focused on mixtures of Gaussians, our algorithm also works for the significantly more general class of mixtures of Poincaré distributions. Concretely, in the regime 2 vlog d, our algorithm recovers the state-of-the-an guarantees provided by non-prin non-private algorithms which are based the sum-of-squares hierarchy [36 30 59
If A > g2/t" for some constant t", then by choosing 3 lit*) ) the algorithm recovers the centers, up to 1/ poly(k) error, in time poly (k; d) and using only poly(k, d) samples. If > Q Vog k) then choosing t = Oloog k) the algorithm recovers the centers, up to 1/ poly(k) error, in quasi-polynomial time poly(k00)) do(()) and using a quasi- polynomial number of samples (k, do(e)).
For simplicity of exposition will limit the presentation mixtures of spherical Gaussians. We reiterate that separation D2( ([ooo K) is informaiion-theorrtiallly necessary for algorithms with polynomial sample complexity [54).
Subsequently and independently of our work, the work of 6 gives an algorithm that turns any non-private GMM learner into private one based on the subsample and aggregate framework. They apply this reduction to the classical result [45] to give the first finite-sample (z, 6)-DP algorithm that learns mixtures of unbounded Gaussians, in particular, the covariance matrices of their mixture components can be arbitrary.
Techniques
We present here our general tools for designing efficient private estimation algorithms in the high- dimensional setting whose statistical guarantees almost match those of the best know non-prin non-private algorithms. The algorithms we design have the following structure in common: First, we solve a convex optimization problem with constraints and objective function depending on our input Y. Second, we round the optimal solution computed in the first step to solution X for the statistical estimation problem hand.
We organize our privacy analyses according to this structure. In order to analyze the first step, we prove a simple sensitivity bound for strongly convex optimization problems, which bounds the fz-sensitivity of the optimal solution in terms of a uniform sensitivity bound for the objective function and the feasible region of the optimization problem.
For bounded problems -uch as recovery stochastic block models- we use this sensitivity bound, in the second step, show that introducing small additive noise to standard rounding algorithms is enough to achieve privacy.
For unbounded problems -such as learning GMMs- we use this sensitivity bound to show that on adjacent inputs, either most entries of X only change slightly, as in the bounded case, or few entries vary significantly. We then combine different privacy techniques hide both type of changes.
Privacy from sensitivity of strongly convex optimization problems Before illustrating our techniques with some examples, it is instructive to explicit our framework. Here we have set of inputs y and a family of strongly convex functions F())) and convex sets KC())) parametrized by these inputs. The generic non-private algorithm based on convex optimization we consider works as follows:
1. Compute X == argminn xex(y) fr (X);
2. Round X into an integral solution.
For an estimation problem, a distributional assumption on ) is made. Then one shows how, for typical inputs Y sampled according to that distribution, the above scheme recovers the desired structured information.
We remark that [39 give polynomial time algorithm for separation ([((og())/2+t) for constant > 0 in the non- non-privan private setting but for less general class of mixture distributions.

---

We can provide privatized version of this scheme by arguing that, under reasonable assumptions on F()) and XC))), the output of the function argmin x EK(Y) (x( has low l2 fe-sensitivity, The consequence of this crucial observation is that one can combine the rounding step with some standard privacy mechanism and achieve differential privacy. That is, the second step becomes:
2. Add random noise N and round X + N into an integral solution.
Our sensitivity bound is simple, yet it generalizes previously known bounds for strongly convex optimization problems (we provide a detailed comparison later in the section). For adjacent Y, Y' e Y, it requires the following properties:
(i) For each X E K((Y) n K(Y') it holds Ify
(x)
(ii) For each X K(Y) its projection Z onto (CC")) satisfies (fr fr- & a.
Here we think of a as some small quantity (relatively to the problem parameters). Notice, we may think of (i) as Lipschitz-coniiuity of the function x)) = (Xx with respect to Y and of (ii) as a bound on the change of the constrained set on adjacent inputs. In fact, these assumptions are enough to conclude low lz sensitivity. Let X and X' be the outputs of the first step on inputs Y,Y". Then using (i) and (ii) above and the fact that X is an optimizer, we can show that there exists Z E KCY) n K(Y") such that
[Xy - (z +
O(a).
By k-strong convexity of fv. fY this implies
X
22 < O(a/k)
which ultimately means || -X--X -x113 & (()/k)) (see Lemma B. Thus, starting from our assumptions on the point-wise distance of fy fy we were able to conclude low fz-sensitivity of our output!
A simple application: weak recovery of stochastic block models The ideas introduced above, combined with existing algorithms for weak recovery stochastic block models, immediately imply private algorithm for the problem. To illustrate this, consider Model 1.1 with parameters 3d > C, for some large enough constant C > 1. Let æ E {=1}" be balanced. Here is an n-by-n matrix corresponding to the rescaled centered adjacency matrix of the input graph:
4)
if E E(G)
Yij
otherwise.
The basic semidefinite program timization question of finding the orthogonal projection of the matrix Y onto the set IC =
28 46 can be recas the strongly convex constrained op-
{X E gaxx X 0, vi] That
X argminxex || - -xIE
Let |X - 2(X, Y) and notice that X = argmin ex exc fy((X. is standard fact that, if our input was G SBM,, (d, 7, x). then with high probability X X(G) argminxex (X) would have leading eigenvalue
pair satisfying
GG) > 1 - 0(1772) and (G1(G), x/ |x]]] - 0 (1/)2d)
This problem fits perfectly the description of the previous paragraph. Note that since the constraint set does not depend on Y, Property (ii) reduces to Property (i). Thus, it stands to reason that the projections X, X' of Y, Y' are close whenever the input graphs generating Y and Y" are adjacent. By Hölder's s Inequality with the entry-wise infinity and -norrm, we obtain (X) - fy (X) <
2 | oz || - Y By standard facts about positive semidefinite matrices, we have [ bo
is linear in X instead of quadratic. However, both programs have similar utility guarantees and the utility proof of our program is an adaption of that in (28) (see Lemma C.8) We use the quadratic objective function to achieve privacy via strong convexity.
"The objective function in 228 46
7

---

for all X E K. Also, Y and Y' can differ on at most 2 entries and hence ||Y - 1 < ((())) Thus, || - X O wyd
The rounding step now straightforward. Using the Gaussian mechanism we return the leading eigenvector of X + N where N ~ [0, 1 log(1/0) yuxr. This matrix has Frobeinus norm
nyd
significantly larger than X but its spectral norm is only
[N & Volog(1/0)
1og(1/5) yd
nyd
Thus by standard linear algebra, for typical instances G ~ SBM,, 7, 2). the leading eigenvector of X (G) + N will be highly correlated with the true community vector x whenever the average degree is large enough. In conclusion, a simple randomized rounding step enough!
Remark 2.1 (From weak recovery to exact recovery). In the non-private setting, given a weak recovery algorithm for the stochastic block model, one can use this as an initial estimate for boosting procedure based on majority voting to achieve exact recovery. We show that this can be done privately. See See Appendix C.2 in the supplements.
An advanced application: learning mixtures of Gaussians In the context of SBMs our argument greatly benefited from two key properties: first, on adjacent inputs Y Y' was bounded in an appropriate norm; and second, the convex set K. was fixed. In the context of learning mixtures of spherical Gaussians as in Model 1.2, both this properties are not satisfied (notice how one of this second properties would be satisfied assuming bounded centers!). So additional ideas are required.
The first observation, useful to overcome the first obstacle, is that before finding the centers, one can first find the n-by-n membership matrix WYY) where WY)),; from the same mixture component and otherwise. The advantage here is that, on adjacent inputs, [] - W('')ll < 2n/k and thus one recovers the first property 15 Here early sum-of-saue -f-urrrr--eess algorithms for the problem 330 36 turns out to be convenient as they rely on minimizing the function
1 if i, where sampled
||W subject to the following system of polynomial inequalities in variables ž11 ........ , 21k Ank-
with Wij ES Ziz for all i, E [n]] and parameter / >>> 0.
2/e Zit
Vi E [n] l E [k]
(indicators)
Σ Zie & 1
Vi E [m]]
(cluster membership)
(E(M]
Zie Zie" = 0
iA E [m] l E [k]
(unique membership)
Σ Zie = n/k
3A E [k]
(size of clusters
k Σ Zit V/i
1/6 n
34 [k]
(means of clusters)
b (yi u) < (26)²
||ull
Vu E Rd le [K]
(subgaussianity of t-moment)
(P(Y))
For the scope of this discussion we may disregard computational issues and assume have access to an algorithm returning a point from the convex hull KC(Y) ) of all solutions to our system of {0, 1} meant to indicate whether sample yi is believed
inequalities [8 Each indicator variable zit E
15 Nottceee for typical inputs Y from Model 1.2 one expect [[ |f Re n²/k:
"Formally, would replace the constraint on the size of the clusters by one which requires them be of size (1 i ax) -. for some small a.
"While this is far from being true, it turns out that having access to pseudo- pseudo-distribute distribution satisfying P enough for subsequent argument work, albeit with some additional technical work required.
'WW remark that priori it is also not clear how to encode the subgaussian constraint a way that we could recover degree-t beeudo-distrbuur satisfying P(Y)) in polynomial time. By now this is well understood, we discuss this in Appendix Alin the supplements.
8

---

to be in cluster Ce. In the non con-private setting, the idea behind the program is that for typical Y
sampled according to Model 1.2 with minimum separation A > kl/² any solution W WYY) E
K(Y) close to the ground truth matrix W-(Y) Frobenius norm: I|W(Y) W (Y)
1/ poly(k)). Each row WYY); may be seen as inducing a uniform distribution over subset of Y Combining the above bound with the fact that subgaussian distributions at small total variation
distance have means that are close, we conclude the algorithm recovers the centers of the mixture.
While this program suggests a path to recover the first property, it also possesses a fatal flaw: the projection W of W E XCY onto K(YY) n K(Y")) may be far in the sense that |||W|| - |W]|
2 2 F + [[ > (kn2//k) The reason behind this phenomenon can be found in the constraint tee n/ k. The set indicated by the vector (21e ... zne) may be subgaussian in the sense of P for input Y but, upon changing single sample, this may no longer be true. We work around
this obstacle in two steps:
1. We replace the above constraint with E; 2ie & n/k:
2. We compute W := argminw salving P(Y) || WIf where J is the all-one matrix
20
The catch now is that the program is satisfiable for any imput Y since we can set 0 whenever necessary. Moreover, we can guarantee property (ii) (required by our sensitivity argument) for & (n/kk), since we can obtain W' E K(Y) n K(YY") simply zeroing out the row/column in W corresponding to the sample differing in Y and Y". Then for typical inputs Y, the correlation with the true solution is guaranteed by the new strongly convex objective function.
We offer some more intuition on the choice of our objective function: our guess whether the i-th and j-th datapoints are sampled from the same Gaussian component. A necessary condition for W to be close its ground-truth counterpart W is that they roughly have the same number of entries that are (close to) 1. One way to achieve this would be to add the lower bound constraing Σ, 244 Z n. be two neighboring datasets, such that the constraint set induced by one dataset is satisfiable, but the constraint set induced by the other dataset not satisfiable We avoid this issue by noticing that the appropriate number of entries close to can also be induced by minimizing the distance of W to the all-one matrix. This step is also a key difference from (35), explained in more detail below.
Recall that Wi indicates
k. However, such a constraint could cause privacy issues: There would
From low sensitivity of the indicators to low sensitivity of the estimates For adjacent inputs Y, Y let W, W' be respectively the matrices computed by the above strongly convex programs. Our discussion implies that, applying our sensitivity bound, we can show || - I & O(n/k) The problem is that simply applying a randomized rounding approach here cannot work. The reason is that even tough the vector W; induces a subgaussian distribution, the vector W, + v for E R", might not. Without the subgaussian constraint we cannot provide any meaningful utility bound. In other words, the root of our problem is that there exists heavy-tailed distributions that are arbitrarily close in total variation distance to any given subgaussian distribution.
On the other hand, our sensitivity bound implies || W - WiIR < o(I[W]]))) fraction of rows i E [n] must satisfy ||W,- W/ < o oll For each row é, let μ4 pá be the means of the distributions induced respectively by W., W.. We are thus in the following setting:
[|1) and thus, all but vanishing
1. For a set of o(1) good rows |1 Aill2 & o(1),
2. For the set B of remaining bad rows, the distance ||ps 11112 may be unbounded.
We hide differences of the first type as follows: pick a random subsample S of [n] of size nf, for some small c >>> 0, and for each picked row the Gaussian mechanism. The subsampling step is useful as it allows us to decrease the standard deviation of the entry-wise random noise by a factor nl-e We hide differences of the second type as follows: Note that most of the rows are clustered together in space. Hence, we aim to privately identify the regions which contain many of the rows.
More generally, may think of vector E R² the vector inducing the distribution given by e/|le|h onto the set of n elements.
2eW remark that for technical reasons our function in Appendix D.I in the supplements will be slightly different. We do not discuss here avoid obfuscating our main message.

---

Formally, we use a classic high dimensional (s, difffrreeti private histogram learner on S and for the & largest bins of highest count privately return their average (cf. [eemma A.13). The crux of the argument here is that the cardinality of B n S is sufficiently small that the privacy guarantees of the histogram learner can be extended even for inputs that differ in [B n S] many samples. Finally, standard composition arguments will guarantee privacy of the whole algorithm.
Comparison with the framework of Kothari-Manuranggs- Vellngker Both Kothari-Manurangss Velingker (35) and our work obtained private algorithms for high-dimensional statistical estimation problems by privatizing strongly convex programs, more specifically, sum-of-squares (SoS) programs. The main difference between KMV and our work lies how we choose the SoS program. For the problem of robust moment estimation, KMV considered the canonical SoS program from 30 [6] which contains a minimum cardinality constraint (e.g., Σ 24 2 a in the case of GMMs). Such a constraint is used to ensure good utility. However, alluded to earlier, this is problematic for privacy: there will always exist two adjacent input datasets such that the constraints are satisfiable for one but not for the other. KMV and us resolve this privacy issue in different ways.
k
KMV uses an exponential mechanism to pick the lower bound of the minimum cardinality constraint. This step also ensures that solutions to the resulting SoS program will have low sensitivity. In contrast, we simply drop the minimum cardinality constraint. Then the resulting SoS program is always feasible for any input dataset! To still ensure good utility, we additionally pick an appropriate objective function. For example, in Gaussian mixture models, chose the objective ||W-Jl': Our approach has the following advantages: First, the exponential mechanism in KMV requires computing (n) scores. Computing each score requires solving large semidefinite program, which can significantly increase the running time. Second, proving that the exponential mechanism in KMV works requires several steps: 1) defining a (clever) score function, 2) bounding the sensitivity of this score function and, 3) showing existence of a large range of parameters with high score. Our approach bypasses both of these issues.
Further, as we show, our general recipe can be easily extended to other high dimensional problems of interest: construct strongly convex optimization program and add noise its solution. This can provide significant computational improvements. For example, in the context of SBMs, the framework of [35] would require one to sample from an exponential distribution over matrices. Constructing and sampling from such distributions is an expensive operation. However, it is well-understood that an optimal fractional solution to the basic SDP relaxation we consider
be found in near quadratic time using the standard matrix multiplicative weight method 7 [58). making the whole algorithm run in near-quadratic time. Whether our algorithm can be sped up to near-linear time, as in (88 remains a fascinating open question.
Comparison with previous works on empirical risk minimization Results along the lines of the sensitivity bound described at the beginning of the section (see LeeeLemma B.I for a formal statement)
have been extensively used in the context of empirical risk minimization [5] 34, 57 63 50). Most results focus on the special case of unconstrained optimization of strongly convex functions. In contrast, our sensitivity bound applies to the significantly more general settings where both the objective functions and the constrained set may depend on the input 21 Most notably for our settings of interest, [15] studied unconstrained optimization of (smooth) strongly convex functions depending on the input, with bounded gradient. We recover such a result for X' X in (ii). [50], the authors considered constraint optimization of objective functions where the domain (but not the function) may depend on the input data. They showed how one can achieve differential privacy while optimize the desired objective function by randomly perturbing the constraints. It is important to remark that, in [50]. the notion of utility is based on the optimization problem (and their guarantees are tight only up to logarithmic factors). In the settings we consider, even in the special case where does not depend on the input, this notion of utility may not correspond to the notion of utility required by the estimation problem, and thus, the corresponding guarantees can turn out to be too loose to ensure the desired error bounds.
TTe attentive reader may argue that one could cast convex optimization over a constrained domain as unconstrained optimization of a new convex function with the appropriate penalty terms. In practice however. this turns out be hard to do for constraints such as Definition A.
10

---

Acknowledgments and Disclosure of Funding
This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No 815464).
References
[1] Tackling urban mobility with technology.
http::////europe.oooooooooooo httpl/log..c//211////11 2015. Accessed: 2022-11-06.
[2] Learning with privacy scale. https: http:/oooo--ssttt.deeeellpllpppo deweloper........o apple com/
m1 www.reearccppppere/eeeeeeee
pdf 2017. Accessed: 2022-
11-06.
[3] Disclosure avoidance for the 2020 census: An introduction.
https
7/www2. census http:///rarr/pbbaaaaaoooooe/eee/
2020 census disclosure htw.:eeaaaaaacace--aaaaaoooooooo handbook, pdf, 2021.
Accessed: 2022-11
06.
[4] Emmanuel Abbe. Community detection and stochastic block models: recent developments, The Journal of Machine Learning Research, 18(1):6446-653)) 65311, 2017.
[5] Emmanuel Abbe, Afonso S Bandeira, and Georgina Hall. Exact recovery in the stochastic block model. IEEE Transactions on information theory, 62(1))471-455 487, 2015.
[6] Jamil Arbas, Hassan Ashtiani, and Christopher Liaw. Polynomial time and private learning of unbounded gaussian mixture models. arXiv preprint arXiv:2303.04288, 2023.
[7] Sanjeev Arora and Satyen Kale. A combinatorial, primal-dual approach to semidefinite programs. In Proceedings of the thirty-ninth annual ACM symposium on Theory of computing, pages 227-236, 2007.
[8] Hassan Ashtiani and Christopher Liaw. Private and polynomial time algorithms for learning gaussians and beyond. In Po-Ling Loh and Maxim Raginsky, editors, Proceedings of Thirty Fifth Conference on Learning Theory, volume 178 Proceedings of Machine Learning Research, pages 1075-1076. PMLR, 02-05 Jul 2022.
[9] Ainesh Bakshi, Hias Diakonikolas, Samuel B. Hopkins, Daniel Kane, Sushrut Karmalkar, and Pravesh K. Kothari. Outlier-robust clustering of gaussians and other non-spherical mixtures. In Sandy Irani, editor, 61 IEEE Annual Symposium on Foundations of Computer Science, FOCS 2020, Durham, NC, USA, November 16-19, 2020, pages 149-159. IEEE, 2020.
[10] Ainesh Bakshi, Ilias Diakonikolas, He Jia, Daniel M. Kane, Pravesh K. Kothari, and Santosh S. Vempala. Robustly learning mixtures of k arbitrary gaussians. Stefano Leonardi and Anupam Gupta, editors, STOC '22: 54th Annual ACM SIGACT Symposium on Theory of Computing, Rome, Italy, June 20 24, 2022, pages 1234-122 1244. ACM, 2022.
[11] Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 2014 JEEE 55th annual symposium on foundations of computer science, pages 464 473. IEEE, 2014.
[12] Christian Borgs, Jennifer Chayes, and Adam Smith. Private graphon estimation for sparse graphs. Advances in Neural Information Processing Systems, 28, 2015.
[13] Christian Borgs, Jennifer Chayes, Adam Smith, and Ilias Zadik. Revealing network structure. confidentially: Improved rates for node-private graphon estimation. In 2018 IEEE 59th Annual Symposium on Foundations of Computer Science (FOCS), pages 533-543. IEEE, 2018.
[14] Moses Charikar, Jacob Steinhardr, and Gregory Valiant. Learning from untrusted data. In Proceedings of the 49th Annual ACM SIGACT Symposium on Theory of Computing, pages 47-60, 2017.
11

---

[15] Kamalika Chaudhuri. Claire Monteleoni, and Anand D Sarwate. Differentially private empirical risk minimization. Journal of Machine Learning Research, 12(3). 2011.
[16] Edith Cohen, Haim Kaplan, Yishay Mansour, Uri Stemmer, and Eliad Tsfadia. Differentially- private clustering of easy instances. In International Conference on Machine Learning, pages 2049-2059. PMLR, 2021.
[17] Aurelien Decelle, Florent Krzakala, Cristopher Moore, and Lenka Zdeborová. Asymptotic analysis of the stochastic block model for modular networks and its algorithmic applications. Physical Review E, 84(6):066106 06, 2011.
[18] Ilias Diakonikolas, Gautam Kamath, Daniel Kane, Jerry Li, Ankur Moitra, and Alistair Stewart. Robust estimators in high-dimensions without the computational intractability. SIAM Journal on Computing, 2019.
[19] Ilias Diakonikolas and Daniel M Kane, Recent advances in algorithmic high-dimensional robust statistics. arXiv preprint
2019.
[20] Ilaas Diakonikolas, Daniel M. Kane, Sushrut Karmalkar, Ankit Pensia, and Thanasis Pittas. Robust sparse mean estimation via sum of squares. In Po-Ling Loh and Maxim Raginsky, editors. Conference on Learning Theory, 2-5 July 2022, London, UK, volume 178 of Proceedings of Machine Learning Research, pages 4703-4763. PMLR, 2022.
[21] Jingqiu Ding, Tommaso d' O'Orsi, Rajai Nasser, and David Steurer. Robust recovery for stochastic block models. In 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS), pages 387-394. IEEE, 2022.
[22] Tommaso d' O'Ori,, Pravesh K. Kothari, Gleb Novikov, and David Steurer. Sparse PCA: algorithms, adversarial perturbations and certificates. In Sandy Irani, editor, 61 IEEE Annual Symposium on Foundations of Computer Science, FOCS 2020, Durham, NC, USA, November 16-19, 2020, pages 553-564 IEEE, 2020.
[23] Cynthia Dwork and Jing Lei. Differential privacy and robust statistics. In STOC'09- Proceedings of the 2009 ACM International Symposium on Theory of Computing, pages 371-380. ACM, New York, 2009.
[24] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam D. Smith. Calibrating noise to sensitivity in private data analysis. In Shai Halevi and Tal Rabin, editors, Theory of Cryptography, Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006, Proceedings, volume 3876 of Lecture Notes in Computer Science, pages 284..28..5 Springer, 2006.
[25] Yingjie Fei and Yudong Chen. Achieving the Bayes error rate in synchronization and block models by SDP, robustly. IEEE Trans. Inform. Theory, 66(6):3929-3955 2020.
[26] Noah Fleming, Pravesh Kothari, Toniann Pitassi, et al. Semialgebraic proofs and efficient algorithm design. Foundations and Trends@ in Theoretical Computer Science, 14(1-2):1-22) 2019.
[27] M. Grötschel, L. Lovász, and A. Schrijver. The ellipsoid method and its consequences in combinatorial optimization. Combinatorica, 162):199-197, 1981.
[28] Olivier Guédon and Roman Vershynin. Community detection in sparse networks via Grothendieck's 's inequality. Probab. Theory Related Fields, 16555-4))1025-10499 2016.
[29] Samuel B Hopkins, Gautam Kamath, and Mahbod Majid. Efficient mean estimation with pure differential privacy via a sum-of-squares exponential mechanism. In Proceedings of the 54th Annual ACM SIGACT Symposium Theory of Computing, pages 1406-1417, 2022.
[30] Samuel B. Hopkins and Jerry Li. Mixture models, robustness, and sum of squares proofs. In Iliss Diakonikolas, David Kempe, and Monika Henzinger, editors, Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, STOC 2018, Los Angeles, CA, USA, June 25-29, 2018, pages 1021-1034, ACM, 2018
12

---

[31] William B Johnson. Extensions of lipschitz mappings into a hilbert space. 26:189-206, 1984.
Contemp. Math.,
[32] Gautam Kamath, Or Sheffet, Vikrant Singhal, and Jonathan Ullman. Differentially private algorithms for learning mixtures of separated gaussians. Advances in Neural Information Processing Systems, 32, 2019.
[33] Vishesh Karwa and Salil P. Vadhan. Finite sample differentially private confidence intervals. In Anna R. Karlin, editor, 9th Innovations in Theoretical Computer Science Conference, ITCS 2018, January 11-14, 2018, Cambridge, MA, USA, volume 94 of LIPIcs, pages 44:1-44:9. Schloss Dagstuhl Leibniz-Zentruu für Informatik, 2018.
[34] Daniel Kifer, Adam Smith, and Abhradeep Thakurta. Private convex empirical risk minimization and high-dimensional regression. In Conference on Learning Theory, pages 25-1. JMLR Workshop and Conference Proceedings, 2012. 2012
[35] Pravesh Kothari, Pasin Manurangsi, and Ameya Velingker. Private robust estimation by stabiliz- ing convex relaxations. In Po-Ling Loh and Maxim Raginsky, editors, Conference on Learning Theory, 2-5 July 2022, London, UK, volume 178 of Proceedings of Machine Learning Research, pages 723-777. PMLR, 2022.
[36] Pravesh K. Kothari, Jacob Steinhardt, and David Steurer. Robust moment estimation and improved clustering via of squares. In Ilaas Diakonikolas, David Kempe, and Monika Henzinger, editors, Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, STOC 2018, Los Angeles, CA, USA, June 25-29, 2018, pages 1035-1046. ACM, 2018.
[37] Kevin A. Lai, Anup B. Rao, and Santosh S. Vempala. Agnostic estimation of mean and covariance. In Irit Dinur, editor, IEEE 57th Annual Symposium Foundations of Computer Science, FOCS 2016, 9-11 October 2016, Hyatt Regency, New Brunswick, New Jersey, USA, pages 665-674. IEEE Computer Society, 2016.
[38] Jean B. Lasserre. New positive semidefinite relaxations for nonconvex quadratic programs. In Advances in convex analysis and global optimization (Pythagorion, 2000), volume 54 of Nonconvex Optim. Appl., pages 319-331. Kluwer Acad. Publ., Dordrecht, 2001.
[39] Allen Liu and Jerry Li. Clustering mixtures with almost optimal separation in polynomial time. In Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing, pages 1248 1248-1261, 2022.
[40] Allen Liu and Ankur Moitra. abs/2207.11900 19033, 2022.
Minimax rates for robust community detection.
CoRR,
[41] Xiyang Liu, Weihao Kong, and Sewoong Oh. Differential privacy and robust statistics in high dimensions. In Conference on Learning Theory, pages 1167-1246. PMLR, 2022.
[42] Laurent Massoulié. Community detection thresholds and the weak ramanujan property. In Proceedings of the forty-sixth annual ACM symposium on Theory of computing, pages 694-703, 2014.
[43] Frank McSherry and Kunal Talwar. Mechanism design via differential privacy. 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS 07), pages 94-103 IEEE, 2007.
[44] Ankur Moitra, William Perry, and Alexander S Wein. How robust are reconstruction thresholds for community detection? In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing, pages 828-6 033-841,22 2016.
[45] Ankur Moitra and Gregory Valiant. Settling the polynomial learnability of mixtures of gaussians. In 2010 IEEE 5/ Annual Symposium on Foundations of Computer Science, pages 93-102. IEEE, 2010.
[46] Andrea Montanari and Subhabrata Sen. Semidefinite programs on sparse random graphs and their application to community detection. In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing, pages 814-827, 2016. 2016.
13

---

[47] Elchanan Mossel, Joe Neeman, and Allan Sly. Consistency thresholds for the planted bisection model. In Proceedings of the forty-seventh anmual ACM symposium on Theory of computing. pages 69-75, 2015.
[48] Elchanan Mossel, Joe Neeman, and Allan Sly. Reconstruction and estimation in the planted partition model. Probability Theory and Related Fields, 162(3):431-46) 2015.
[49] Elchanan Mossel, Joe Neeman, and Allan Sly. A proof of the block model threshold conjecture. Combinatorica, 38(3):665-7086 2018.
[50] Andres Munoz, Umar Syed, Sergei Vassilvtiskii, and Ellen Vitercik. Private optimization without constraint violations. In International Conference on Artificial Intelligence and Statistics, pages 2557-2565. PMLR, 2021.
[51] Yurii Nesterov. Squared functional systems and optimization problems. High performance optimization, volume 33 of Appl. Optim., pages 405 4400. Kluwer Acad. Publ., Dordrecht, 2000.
[52] Kobbi Nissim, Sofya Raskhodnikova, and Adam Smith. Smooth sensitivity and sampling in private data analysis. In STOC 370ceoi--roccedding Proceedings of the 39th Anmual ACM Symposium on Theory of Computing, pages 75-84. ACM, New York, 2007.
[53] Pablo A Parrilo. Structured semidefinite programs and semialgebraic geometry methods in robustness and optimization. PhD thesis, California Institute of Technology, 2000.
[54] Oded Regev and Aravindan Vijayaraghavan. On learning mixtures of well-separated gaussians. In 2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS), pages 85-96. IEEE, 2017.
[55] Mohamed M. Seif, Dung Nguyen, Anil Vullikanti, and Ravi Tandon. Differentially private community detection for stochastic block models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages
PMLR, 2022.
[56]
N. Z. Shor. Quadratic optimization problems. Izv. Akad. Nauk SSSR Tekhn. Kibernet., (1):128- 139, 222, 1987.
[57] Shuang Song, Kamalika Chaudhuri, and Anand D Sarwate. Stochastic gradient descent with differentially private updates. In 2013 IEEE global conference on signal and information processing, pages 245-248. IEEE, 2013.
[58] David Steurer. Fast sdp algorithms for constraint satisfaction problems. In Proceedings of the twenty-frrs annual ACM-SIAM symposium on Discrete Algorithms, pages 684-697. SIAM, 2010.
[59] David Steurer and Stefan Tiegel. Sos degree reduction with applications to clustering and robust moment estimation. In Proceedings of the 2021 ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 374-393. SIAM, 2021.
[60] Eliad Tsfadia, Edith Cohen, Haim Kaplan, Yishay Mansour, and Uri Stemmer. Friendlycore: Practical differentially private aggregation. In International Conference on Machine Learning, pages 21828-21 2186-.. PMLR, 2022.
[61] Jonathan Ullman and Adam Sealfon. Efficiently estimating erdos-renyi graphs with node differential privacy. Advances in Neural Information Processing Systems, 32, 2019.
[62] Martin J. Wainwright. Digh--Diensioonal Statistics: A Non-Asymptotic Viewpoint. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2019.
[63] Di Wang, Minwei Ye, and Jinhui Xu. Differentially private empirical risk minimization revisited: Faster and more general. Advances in Neural Information Processing Systems, 30, 2017.
[64] Anderson Y Zhang and Harrison H Zhou. Minimax rates of community detection in stochastic block models. The Annals of Statistics, 44(5):2252-288,, 2016.
14

---

A Preliminaries
We use boldface characters for random variables. We hide multiplicative factors logarithmic in n. using the notation 0 Similarly, we hide absolute constant multiplicative factors using the standard notation ( (.). Often times we use the letter C' do denote universal constants independent of the parameters at play. We write o()), for functions tending to zero (resp. infinity) as n grows. We say that an event happens with high probability if this probability is at least 1 1- (1). Throughout the paper, when we say "an algorithm runs in time (gq)" we mean that the number of basic arithmetic operations involved is Ogg). That is, we ignore bit complexity issues.
Vectors, matrices, tensors We use Id,, to denote the n-by-n dimensional identity matrix, In E RTxx the all-ones matrix and On , 1, R" to denote respectively the zero and the all-ones vectors. When the context is clear we drop the subscript. For matrices A, B E R we write A 2 B A B is positive semidefinite. For matrix M, denote its eigenvalues by (M , y....y (M), we simply write X; when the context is clear. We denote by |M]] the spectral norm of M. We denote by Rd the set of real-valued order-t tensors. for a d x d matrix M. we denote by M the t-fold Kronecker product M X M @ @ M. We define the flattening, or vectorization, of M to
times
be the d' o'-dimensional vector, whose entries the entries of M appearing in lexicographic order. With a slight abuse of notation we refer to this flattening with M, ambiguities will be clarified form context. We denote by N (0, o²) d the distribution over Gaussian tensors with d' entries with standard deviation a. Given u, v E {+1}", we use Ham(u, (u, v) == Ei=1 1 [ua to denote their Hamming distance. Given a vector u E TR",, we let sign(u) E {+1) denote its sign vector. vector (+1)" is said to be balanced if Σ 7 ==1 U4 0.
Graphs We consider graphs on n vertices and let Gn be the set of all graphs on n vertices. For a graph G on n vertices we denote by A(G) E R=x its adjacency matrix. When the context is clear we simply write A. Let V(G) (resp. E(G)) denote the vertex (resp. edge) set of graph G. Given two graphs G, H on the same vertex set V. let G Hl == E(G) H(G)). Given graph H, H' C H means H' is a subgraph of H such that V(H') = ((H) and E(H) E(H). The Hamming distance between two graphs G, is defined be the size of the symmetric difference between their edge sets, i.e. Ham H) == ((G))
A.1 Differential privacy
In this section we introduce standard notions of differential privacy 24
Definition A.1 (Differential privacy). An algorithm M y + O is said be (E, 8) difffeerenti private for s, 8 > 0 if and only if. for every S S O and every neighboring datasets Y, y we have
PIM(Y) E S) &
(...)..PPMM.... PP(((Y') 5
To avoid confusion, for each problem we will exactly state the relevant notion of neighboring datasets. Differential privacy closed under post-processing and composition.
Lemma A.2 (Post-processing). M : y O is an (E, -iifferrrrelltilllly private algorithm and M' y + Z is any randomized function. functiion Then the algorithm M' (M(Y)) is (e, -)ifferreenl private.
In order to talk about composition it is convenient to also consider DP algorithms whose privacy guarantee holds only against subsets inputs.
Definition A.3 (Differential Privacy Under Condition). An algorithm M : y - O is said to be (e, -idfifferentiall private under condition A (or (e, 8DP under condition 4) 5 > 0 if and only if. for every C O and every neighboring datasets Y, Y' E Y both satisfying V we have
PIM(Y) E < P(M(Y') +
It is not hard to see that the following composition theorem holds for privacy under condition.
15

---

Lemma A.4 (Composition for Algorithm with Halting, (351). Let Let + O2 ff M 0 y + O2 {1) ....,M, O,-1 y + O {r} be algorithms. Furthermore, let M denote the algorithm that proceeds follows (with Oo being empry): For = o; = MM(o0s1, and, if 1, halt and output Finally, if the algorithm has not halted, then oulput or- Suppose that:
compute
For any & i & t. say that Y satisfies the condition V, if running the algorithm Y does not result in halting after applying MI....,M,
.M1 is (51,51)-Dr)
MM is (ei, 6DP (with respect to neighboring datasets in the second argument) under condition V; for all i (2....,I}
Then M is (E Eig E, 6.)-
A.1.1 Basic differential privacy mechanisms
The Gaussian and the Laplace mechanism are among the most widely used mechanisms in differential privacy. They work by adding a noise drawn from the Gaussian (respectively Laplace) distribution to the output of the function sensitivity of the function.
wants to privatize. The magnitude of the noise depends the
Definition A.5 (Sensitivity of function). Let f y - Rdd be function, its fi-sensitivity and l2 Se-sensitvv are respectively
f = max Y YYYEY
(Is(Y)- - f(Y'
A f2 =
max YYE'Y
- f(Y)l2
Y adjacens
Y Y' adjacent
For function with bounded & fj-sensitivity the Laplace mechanism is often the tool of choice to achieve privacy.
Definition A.6 (Laplace distribution). The Laplace distribution with mean μ and parameter b > 0, denoted by Lap(y, b), has PDF 26 ---le-u1/0 Let Lap(b) denote Lap(O, b).
A standard tail bound concerning the Laplace distribution will be useful throughout the paper.
Fact A.7 (Laplace tail bound). Let Lap(p, b). Then,
P p1) t/bb
The Laplace distribution is useful for the following mechanism Lemma A.8 (Laplace mechanism). Let f y + IRd be any function with fi-sensitivity at most f.1-
Then the algorithm that adds Lap A @d
(e. 0)-
is also useful to consider the "truncated" version of the Laplace distribution where the noise distribution is shifted and truncated to be non-positive.
Definition A.9 (Truncated Laplace distribution). The (negatively) truncated Laplace distribation w with mean i and parameter b on R, denoted by (Tap(,, b), is defined as Lap(p, b) conditioned on the value being non-positive.
Lemma A.10 (Truncated Laplace mechanism). Let f Y + R be any function with E Ei-sensitivity at most f Then the algorithm that adds tLap (-A f, [1+ log(1/6) A to f is (e, ().P).
The following tail bound is useful when reasoning about truncated Laplace random variables. Lemma A.11 (Tail bound truncated Laplace). Suppose < 0 and b > O. Let (Lp((,,)) Thenfor y < μ. we have that
(V/D//) m/b)
P F(x < y) < enab
In constrast, when the function has bounded e ssseiiiity the Gaussian mechanism provides privacy. Lemma A.12 (Gaussian mechanism). Let f y - ddd be any function with fg-sensitivity at most A 1.2 Let 0 < s, & Then the algorithm that udds (0. 2 log(2/5)) Id) to ʃ is (e, 6)-DP.
16

---

A.1.2 Private histograms
Here we present a classical private mechanism to learn high dimensional histogram.
A.13 (High-dimensional private histogram learner, see (33]). Let q,b,e > 0 and 1/n. Let - be partition of I. into intervals of length where {o E g bb b}. Consider the partition TR. into seis (Big da
Lemma
O
100
where
B == fDER"
Vj 2g 11,3
Let Y = Iyl......ub Bnt S Rd be dataset of n points. For each B......ka let Pr Kij [m] Bj Bis...dall For 2 8 log 2 there exists an efficient (E, 8) d)-ffferrettially 53
private algorithm that returns Pl Pi Pai satisfying
I
Ps a EB.
Proof. We consider the following algorithm, applied to each , is E N on input Y:
If Ps Lap (0, ne
set Pii....aa 0, otherwise let Pit....e Pii....ds + where t
2. If Pa
3 (log(2/)) set Pii,...fa ia = 0.
First argue utility. By construction get Pi.....4 0 whenever Pi.....4 O, thus may focus on non-zero Pii....is. There are at most n. non zero Pi......a. By choice of n, and by Fact A.7 the maximum over n independent trials T ~ Lap (0, ne is bounded by a absolute value with probability least B.
remains to argue privacy, Let = fy....., An Y (uh....,y'i v'3 be adjacent datasets. For i,, i,..... id E N, let
Pizmadd =
KÍ E [n] E Biuniiaal
Pri....is {j E [n] V/ Bi,...ie)l
Since Y, Y' are adjacent there exists only
of indices I= (i1,..., (d) and J (i ( ja)
such that pr # ply and PJ * P'y Assume without loss of generality pr > pzz- Then it must be Dr = p/r 1/n and Pz = p' 1/n. Thus by the standard tail bound on the Laplace distribution in Fact A.7 and Lemma A.8 we immediately get that the algorithm is (e, diffferrrett private. O
A.2 Sum-of-squares and pseudo- -iaudd-iistributioss
We introduce here the sum-of-squares notion necessary for our private algorithm learning mixtures of Gaussians. We remark that these notions not needed for (ArpAdded C
Let w (W1, ....... w.) be tuple of indeterminates and let R[w]] be the set of polynomials with real coefficients and indeterminates W,..., wn. We that polynomial E R[w] is sum-offsquares (sos) if there are polynomials qI...., Gr such that p = of + ...... o2
A.2.1 Pseudo-distributions
generalizations of probability distributions. We represent discrete (i.e.. finitely supported) probability distribution over IR" by its probability mass function D: IR" + IR such that D 0 and E Ee D(w) 1. Similarly, we can describe pscudo-distribttion by its mass function. Here, relax the constraint D and only require that D passes certain low-degree non-negativity tests.
Concretely, level-f pseuuoodistrriution is finitely-supported function D R - R such that Σ.. D(w) = and En, D(w)f(w) > for every polynomial f of degree at most f/2. (Here, the argument shows
summations over the support of D.) A straightforward
17

---

that every level-oo- distribution satisfies D > 0 and is thus an actual probability distribution. We define the eseudo-expectation of function f on I2 with respect to a pseudo-distribution D, denoted [ as
ED( = Σ ((w)f(w).
(A.1)
The degree-f moment tensor of a aseudo-distribunn D is the tensor Ep(w)) (1, W1, W2, ..2....... .... w.) ) In particular, the moment tensor has an entry corresponding to the of degree at most l in The set of all degree-l moment tensors of probability distribution is convex set. Similarly, the set of all degree- moment tensors of degree d pseudo-distributions is also convex. Key to the algorithmic utility of pseudo-distributions is the fact that while there can be efficient separation oracle for the convex set of all degree-l moment tensors of an actual probability distribution, there's a separation oracle running in time n o()) for the convex set of the degree- moment tensors of all level-f pseudodistributions..
of all monomials
Fact A.14 (156 53 51 38
For any n, l N, the following set has a (()) weak separation
oracle (in the sense of (27)):
(ED(w) (1, w2, ....... w degree-d pseudo-distribution D over R.
(A.2)
This fact, together with the equivalence of weak separation and optimization [27 allows us to efficiently optimize over pseudo-distributions (approximately)-thi algorithm referred to as the uum-of-squares algorithm.
The level-ê sum-of-squaree algoritlum optimizes the space of all level-f pseudo-distribution that satisfy a given set polynomial constraints we formally define this next.
Definition A.15 (Constrained ppeudo-distributionn). Let be a level-l pseudo-distribution over IR.. Let A = (fi > 0, f2 > ..... fm > 0} be a system of m polynomial inequality constraints. We say that D satisfies the system of constraints A at degree r, denoted D J A. everr every sum-of-squares polynomial l with deg h + EieS max/deg ri & t.
every S S [m]] and
Enhh II 1.200 iES
We write D D A (without specifying the degree) if D holds. Furthermore, we say that D t= A holds approximately if the above inequalities are satisfied up to an error of2 |[h]]| || Iles [s
where ||.1l denotes the Euclidean norn" of the coefficients of a polynomial in the monomial basis.
We remark that if D is actual (discrete) probability distribution, then have D } A if and only D is supported on solutions to the constraints A.
We say that system A of polynomial constraints is explicitly bounded it contains constraint of the form [ & M). The following fact is a consequence of Fact A. 14 (77)
Fact A.16 (Efficient Optimization over Pseudo distridditiibbtinns).. There exists an (n + (m))())ttmm
algorithm that, given any explicitly bounded and sutisfiable system in variables, outputs a level-l pseudo-distributi that satisfies A approximately.
E3 A of m polynomial constraints
A.2.2 Sum-of-squares proof
Let J2,...... fr and be multivariate polynomials in w. A sum-of-xquares proof that the constraints {f1 3 0,..., sm 0} imply the constraint {g > 0} consists of sum-of-squares polynomials (ps) such that
g
Σ ps Iessf SCim]
(A.3)
"hhe choice of norm is not important here because the factor 2 swamps the effects of choosing another norm.
Here, assume that the bit complexity of the constraints in is (n + 01))
18

---

We say that this proof has degree É if for every set S S [m], the polynomial pellkssfi has degree at most If there is degree e SoS proof that (f+ >
< r} implies {g > 0}, we write:
U.. 0 i r] tz {g 0}.
(A.4)
Sum-of- -quuresss proofs satisfy the following inference rules. For all polynomials f. I² + R and for all functions F: R" + Rm". G: R. + R H: R - IR" such that each of the coordinates of the outputs polynomials of the inputs, have:
1f
(f 1 1000)
(addition and multiplication)
Al (f+
Alee 9/
AlE B, T,B.BBC
(transitivity)
A 1 C
(F2 0) (G2 0)
(substitution)
(F(H) > 0} Hedoo((H) {G(H) 0}
Low-degree sum-of-squares proofs are sound and complete if we take low-level pseudo-distributions as models.
Concretely, sum-of-squares proofs allow us to deduce properties of pseudo-distribution that satisfy some constraints.
Fact A.17 (Soundness). If D F A a level-E squares proof A B, then D B.
and there exists a sum-of-
the pseudo-distribution satisfies only approximately, soundness continues to hold we require an upper bound on the bit-complexity of the sum-of-squares A 1,- B (number of bits required to write down the proof).
In our applications, the bit complexity of all sum of squares proofs will be n (()) (assuming that all numbers in the input have bit complexity n()) This bound suffices in order to argue about pseudo-distributions that satisfy polynomial constraints approximately.
The following fact shows that every property of low-level pseudo-distributions can be derived by low-degree sum-of-squares proofs.
Fact A.18 (Completeness). Suppose d 2 r' >> r and A is collection of polynomial constraints with degree at most r, and A (Ei w² < B)for some finite B.
Let {g > 0} be polynomial constraint. If every degree-d ageudo-distribun that satisfies D I-- A also satisfies D + {g 2 0}, then for every > there is a sum-of-squares proof A la {g > s}..
A.2.3 Explictly bounded distributions
We will consider a subset subgaussian distributions denoted as certifiably subgaussians. Many subgaussians distributions are known to be certifiably subgaussian (see (36]).
Definition A.19 (Explicitly bounded distribution). Let t E N. distribution over gd with mean μ is called 2t-explicitly o o-bounded for each even integer such that 1 < & t the following equation has a degree s sum-of cam-oofsques of-uqueres proof in the vector variable u
2s
X (x u, < (as)" (oas))-|lul]))
E
1lu1
Furthermore, we say that D is explicitly bounded if 21-explicitly a-bounded for every i E N. finite set X C Rdd is said to be 21-explicitly oboundded if the uniform distribution on X is 21-explicitly a-bounded.
Sets that are 21-explicitly g-bounded with large intersection satisfy certain key properties. Before introducing them we conveniently present the following definition.
19

---

Definition A.20 (Weight vector inducing distribution), Let Y be a set of size and let [0, 1|" be a vector satisfying |p|| 1. We say that p induces the distribution D with support Y if
Py-D y - yi Pt
Theorem A.21
336 30
Let Y C Rd be a set of cardinality n. Let p,p' E 10. be weight lpp'
and |p p'11, & 3. Suppose that p (respectively p' induces 2t-explicitly 0i bo-uundded (resp. 02) distribution over with mean p(g) (resp. P(pry). There exists an absolute constant 8 such that, if 3 & 3r, then for o 01 02
vectors satisfying 1pl,
[M)) Ma)il //--1/21 - (vot).
In the context of learning Gaussian mixtures, we will make heavy use of the statement below.
Theorem A.22 (36) (0)). Let Y be 2t-explicitly o-bounded set of size Let R be the weight vector inducing the uniform distribution over Y. Let p' E Rn be a unit vector satisfying for some & 3 where 3 is small constant. Then yf induces 21-explicitly (o + (0(B)-1/2)))bbunni distribution over Y.
lpp --111 &
B Stability of strongly-convex optimization
In this section, we prove & sensitivity bounds for the minimizers of a general class of (strongly) convex optimization problems. In particular, show how translate uniform point-wise sensitivity bound for the objective functions into & sensitivity bound for the minimizers.
Lemma B.1 (Stability of strongly- convexx optimization). Let y be a set of datasets. Let K(Y) be family of closed convex subsets of Thm parametrized by Y y and let F(Y) be a family of functions fy K(YY) > IR, parametrized by Y E E., such that:
(i) for adjacent datasets Y,Y' E y and E X(Y) there exist K(Y) NI((Y)) satisfying frx)))r((Z) & a and [fr fr-(2) aa.
(ii) fy is k-strongly convex in X E K(Y).
Then for Y,Y E V, X arg fy (X) and X' := arg fx' it holds
|
X
2
12a
Proof. Let Z K((Y) n K(Y') be a point such that - < a and (fr(2) - fr.(2) < a. By s-strong convexity of fy and fr Proposition G.2 holds
| X
8 2 + 7 x* 1
4 k
(rr(Z)- ---(0~r(+++))) +fr -v.)))xr(8'))
Suppose w.l.o.g. fr(8)= & fr.(8',, for a symmetric argument works the other case, Then and
[r(Z) < (r. +a ta fx 2a
fy(8) & & < fx ta.
follows as desired
- fv 1x) & 3a.
20

---

C Private recovery for stochastic block models
In this section, we present how to achieve exact recovery in stochastic block models privately and thus prove Theorem 1.3 To this end, we first use the stability of strongly convex optimization (Lemma B.1 to obtain private weak recovery algorithm in Appendixx C.T Then we show how to privately boost the weak recovery algorithm to achieve exact recovery Appendix C.2) In Appendix C.4, we complement our algorithmic results by providing an almost tight lower bound on the privacy parameters. We start by defining the relevant notion of adjacent datasets.
Definition C.1 (Adjacent graphs). Let G, G'' be graphs with vertex set [n]. We say that G, G' are adjacent if (EG)) 1.
Remark C.2 (Parameters as public information). We remark that we assume the parameters n, "y Y, be public information given in input to the algorithm.
C.1 Private weak recovery for stochastic block models
In this section, we show how to achieve weak recovery privately via stability of strongly con- vex optimization (Lemma B.I. We first introduce one convenient notation. The error rate of an estimate & E {=1}" of the true partition æ E {+1}" is defined as err(ă, == min(Ham (), Ham(2, ] Our main result is the following theorem.
Theorem C.3. Suppose v d > 12800, .E, 3 There exists an Algorithmm C.4 such that, for any {=1)", inpur G SBM, (7. d), outputs 2 ()) E {=1}" satisfying
err (#(G), o (1-a+ - 7d
log(2/0)
with probability 1 exp(-S(n)), Moreover, the algorithm is (e,. private for any input graph and runs polynomial time.
introduce some notation. Given graph G, let Y(G) yd (A(G) J) where A(G) is the adjacency matrix of G and J denotes all-one matrices. Define K= {x [ X = 0, X;; = Vi]. The algorithm starts with projecting matrix Y(G) to set K. To ensure privacy, then adds Gaussian noise the projection X₁ and obtains private matrix
Before presenting the algorithm
X2. The last step applies a standard rounding method.
Algorithm C.4 (Private weak recovery for SBM). Input: Graph G.
Operations:
Projection: X1 argminn argmin - X
Noise addition: X2 t- X1 W where W N (0, neyd
1og(2/6) xxn
Rounding: Compute the leading eigenvector v of X and return sign(v).
In the rest of this section, we will show Algorithm C.4 is private in Lemma C.7 and its utility guarantee in Lemma C.8 Then Theorem C.3 follows directly from Lemma C.7 and Lemma C.8
Privacy analysis Let y be the set of all matrices Y(G) 1 vertices. We further define q - IC be the function 7d (A(G) d JJ where G is graph on
(YY) == argmin X1I3
(C.1)
We first use Lemma B.I prove that function q is stable.
21 Lemma C.5 (Stability). The function as defined in Eq. (C.1) has le-seisitivity 4 & \ V221
nyd
Note (,,x)) (1 2err((, x)) for any à, x E (+1)".
21

---

Proof. Let 9: : x X + R be the function g(Y, == | - X(Y,X). Applying Lemma B. with fr()) = =Y(Y, it suffices to prove that 9 has ei-seniitvvity and with respect to Y and that it is 2-strongly convex with respect to X. The L f--sensitivity bound follows by observing that adjacent Y, Y" satisfy ||Y- -Y < 7 and that any X K. satisfies || < + Thus it remains to prove strong convexity with respect to X E K. Let X' E K then
| p | | + 2(X" X, X) + | - X
= | F + 2(X X, + - Y) + || -X'I
(X' X, Vo(X, + 2(X', Y) |
That is xX) is 2-strongly convex with respect to X. Note any X IC is symmetric. Then the result follows by Lemma B.1
Remark C.6. In the special case where the contraint set K. does not depend on input dataset Y
(e.g. stochastic block models), the proof can be cleaner as follows. Let fx) := X 2(X, Y). Let X = argminxes ex fy (X) and X' == argmin xex (X). Suppose without loss of generality (x & frr(X') for a symmetric argument works in the other case. Then
(x) & fx((X') < (x) < fx())++ a
Then it is easy to show the algorithm is private.
Lemma C.7 (Privacy). The weak recovery algorithm Algorithm C.4 is (5,5)-DP.
Proof. Since any K is symmetric. we only need to add symmetric noise matrix to obtain privacy. Combining Lemma C.5 with Lemma A.12 we immediately get that the algorithm is (c, -privaee
Utility analysis Now we show the utility guarantee of our priavte weak recovery algorithm. Lemma C.8 (Utility). For any x E {+1}", on input G ~ SBM, (7, d, x). Algorithm C.4 C.4 efficientll outputs (G) E [+1}" satisfying
6400 yV
7000
log(2/8) e²
err ( <
y
with probability exp(-I(n))
To prove Lemma C.8 need the following lemma which is adaption of a well-known result in SBM (28] Theorem I.1]. Its proof is deferred to copppeddix H
Lemma C.9. Consider the settings of Lemma C.8
With probability 1-xxp(-((r))
1
800
(G)
xx
n
7 y
Proof of Lemma C.& By Lemma C.9
we have
X1(G)
1
Tx T
< (x(()) xx
<
800
=: r(7,d)
Tva
with probability - ex))- We condition our following analysis on this event happening.
Let u be the leading eigenvector of (G)). Let A and xg be the largest and second largest eigenvalues of (G). By Weyl's inequality Lemma F.1 and the assumption y 12800, have
A1 2 > 1- 2r(7, 1 2
Let be the leading eigenvector of (G) + W. By Davis-Kahan' theorem Lemma F.2p. we have
2 W|
-u-vl < x X
< [|].
22

---

xt-\/Vn| <2 X1(G)
_x=
< 2r(().))
n
Putting things together and using Fact E. we have
24v6
v vlog(2/0)
||v x/ vnl
<
|
v/l ||u x/ x/Val
+ 2r(7:d)
Vnd
with probability exp(-9(mn))
Observe Ham(sign(v),zz) & | - x for any E R and any x E {+1}". Then with probability
1 exp(-S(n)).
6400 vvd
7000
1og(2/6)
(lammsign(v),.. < || 2/Vali
"
7
8²
Proof of Theorem C.3
By Lemma C.7 and Lemma C.8
C.2 Private exact recovery for stochastic block models
In this section, we prove Theorem 1.3 We show how to achieve exact recovery in stochastic block models privately by combining the private weak recovery algorithm obtained in the previous section and private majority voting scheme.
Since exact recovery is only possible with logarithmic average degree (just to avoid isolated vertices), is more convenient work with the following standard parameterization of stochastic block models. Let > 3 > 0 be fixed constants. The intra-community edge probability a logn and the inter- community edge probability is B. log In the language of Modee 1.1 is SBM,, a+ llog a+3: z)) Our main result is the following theorem.
n
a-8
Theorem C.10 (Private exact recovery of SBM, restatement of Theorem 1.3 a, 3 are fixed constants satisfying
Let > 0. Suppose
1
log(2/6) log
1
Va
3
> 16 and a - B >
Ω
(C.2)
26 Then there exists an algorithm Algorithm C.12 such that,for any balanced x. E {+1}" on input G ~ SBM,, ( loo a-9 , the algorithm is (e, 8) diifferrentially private for any inpul graph and runs in polynomial time.
F), outputs & G) {x,-x} with probability 1 - o(1). Moreover,
Remark C.11. In a standard regime of privacy parameters where E (1) and 1/ poly(n), the private exact recovery threshold Eg. (C.2) reads
Va /
16 and a 3- (2-2 +
Recall the non-private exact recovery threshold is Jo vB 73 V2. Thus the non-private part in Eq.. (C.2) iee. 16, close to optimal.
Algorithm C.12 starts with randomly splitting the input graph G into two subgraphs G₁ and Go- Setting the graph-splitting probability to 1/2, each subgraph will contain about half of the edges of G. Then we run an (e, 8DD weak recovery algorithm (Algorithm C.4) on G1 to get rough estimate i(G,) of accuracy around 90%. Finally, we boost the accuracy to 100% by doing majority voting (Algorithm C.13) on G2 based on the rough estimate ā ā(G That if vertex has more neighbors from the opposite community (according to äi (G1)) in G2, then we assign this vertex to the opposite community. To make the majority voting step private, we add some noise to the vote.
"I the language of Model 1.1 for any have vae /B 2 and only if d / (177) T.
Recall vector (11)" is said to be balanced if En 34
23

---

Algorithm C.12 (Private exact recovery for SBM). Input: Graph
Operations:
Graph-splitting Initialize 11 to be an empty graph on vertex ser V(G). Independentl put each edge of G in G1 with probability 1/2. Let G2 G G..
2. Rough estimation on G1: Run the (e, 8)-DP partial recovery algorithm Algorithm C.4 on G1 to get rough estimate ø6 (1)
Majority voting on G2: Run the (e, D majority voting algorithm Algorithm C.13 with imput (G2, (G))) and get output &.
Return &.
Algorithm C.13 (Private majority voting).
Input: Graph G, rough estimate à {=1}" Operations:
1. For each verlex VV(G), ler Zn = S. D, where
D. = E(,,))BBG)) (a.
S. = (M,))ER(()) [ia==
Set X = Sig + W.) i(Gi)o where W. ~ Lap(2/s).
Return &.
In the rest of this section, we will show Algorithm C.12 is private in Lemma C.15 and it recovers the hidden communities exactly with high probability in Lemma C.17 Then Theorem C.10 follows
directly from Lemma C.I5 and Lemma C.17
Privacy analysis. We first show the differential privcay of the majority voting algorithm (Algo- rithm C.13) with respect to input graph G (i.e. assuming fixed the input rough estimate). Lemma C.14. Algorithm C.13 (s, O)-DP with respect to input G.
Proof. Observing the & fi-sensitivity of the degree count function Z in step is 2, the (c, OD-DP follows directly from Laplace mechanism (Lemma A.12) and post- proccssocssigg (Lemma A.2)
Then the privacy of the private exact recovery algorithm (Algorithm C.12 is a consequence of composition. Lemma C.15 (Privacy). Algorithm C.12 is (e,8)-DP.
Proof. Let A : Gy + 1 denote the (e, DDD recovery algorithm in step 2. Let A₂ Gn x (+1)" + {+1}" denote the (e, ODP majority voting algorithm in step 3. Let be the composition of A1 and AAg
We first make several notations. Given graph H and edge He is graph obtained b adding to H. Given a graph H, G₁ (H) is a random subgraph of H by keeping each edge of H with probability 1/2 independently.
Now, fix two adjacent graphs and Ge where edge appears in Ge but not in G. Also, fix two arbitrary possible outputs æ1, 22 E [=1]" of algorithm 2 It is direct see,
IP (AAG)) (21,
Σ (H) 2 (A2 H. x2) P (G) = H). (C.3) HCG
We can imagine that algorithm A first outputs (01, and then outputs #2 post- post-processing step.

---

Since P(G = - H.) for any H C G, we have
((A(G.) = (21, = Σ =
x1 = x2) (G. =
HCG
+ (A.(H) = PAA(GGe = PG He)
(C.4)
Since both Ai and A2 are (c, 6)-P,, we have for each H C
IP (Ai (H)) 21) < et P (Ai H))
21) +
(C.5)
P ((A2GGn \ H, 22 < e (A2(G) H, = x2) +
(C.6)
Plugging Eq. (C.5) and Eq. (C.6) into Eq. (C.4) we obtain
P P(A(G.) (111 x2)) <
le (Ai =1) P (A2(G' H, = 22) + (G) H)
HCG
e (A(G) (21, +
Similarly, show
P(A(G) = x2) & ((P(G) = (x1, + 8.
(C.7)
Utility analysis. We first show the utility guarantee of the priavte majority voting algorithm. a-8 Lemma C.16. Suppose G is generated by first sampling G ~ SBM,, ( llogn, a+8 2) for some balanced x and then for each vertex removing at most A < O(log" n) adjacent edges arbitrarily. Then input G and balanced rough estimate ă satisfying Ham x)) n/16, Algorithm C.13 efficiently oulputs 3:( (G) such that for each vertex v,
1 E(a - llo + 2- exp P (o (T(G) # 2x) < exp (- (-64
(-102
(a (0-B3) 3)2
-logn
16
a +
Proof. Let us an arbitrary vertex and analyze the probability P ( # sz). Let r = Ham x), x)/n. Then is not hard to see
P ( x.) & B + A' A
-
(C.8)
where
~ Binomial((1/2 - r)nn A a log corresponding to the number neighbors that are from the same community and correctly labeled by
B' Binooiia(rr - A, 3 logg)) corresponding to the number of neighbors that are from the different community but incorrectly labeled by ã,
B ~ Binomial r)n, 3 log"), corresponding to the number of neighbors that are from the different community and correctly labeled by ã,
A' Binomial a logn ) corresponding to the number of neighbors that from the same community but incorrectly labeled by ă.
W Lap(0. 2/c), independently.
The A term appearing in both A and B' corresponds to the worst case where A "favorable" edges are removed. r > 21)), then A = O(log" n) is negligible to rn = (n) and we can safely ignore the effect of removing edges. If o(1). then we can safely assume is correct on all vertices and ignore the effect of removing A edges as well. Thus, we will assume A = 0 in the following analysis.
For any t, t', we have
P ( + - 0) < (A' + B W W> (A + B' <t)
P (A' + t- f') + (W t'+ + + <t).
We choose t, t' by first picking two constants b > 0 satisfying + b < 1 and then solving
25

---

E[A' + B - == a. ( + B'] E|A' - B]) and
=1= b) (E A + B E A + BJ).
By Fact A.7
I'e 2
(1/4- r)
P(W > t') & exp
< exp
=E(a - 3) loo
2
By Fact E.4 and the assumption
< 1/16, we have
(R(A B - 1)2
(a -82
(A B B'
< t) exp
exp -11/4-r)
-log
2E B']
a B
Setting b = 1/2, by
Fact E.4 and the assumption & 11/16, we have
(t- E[A' + BI)²
2(1/4 r)²
(a }3
((A' + B 3 1 < exp
< exp
log
t ビ- E +B]
7
a + B
Further setting a 1/3, have
1/4 -r
(1/4 - r)
(a: 3)
(f(()), # x.) & exp
=(a B) logg
+2-exp
log
12
9
B
Finally, plugging the assumption r < 1/16 to conclude.
Then it is difficult show the utility guarantee of our priavte exact recovery algorithm. Lemma C.17 (Utility). Suppose a 3 are fixed constants satisfying
1og(2/8) 1
Va - /B> 16 and a- a3330
log n
Then for any balanced x E {=1}", on input G ~ SBM,, f logn, 2-8 2)) efficiently outputs (G) satisfying ((G) {x, [x,-x} with probability o(11.)
x). Algorithm C.12
Proof. We will show the probability of a fixed vertex being misclassified is at most o(/nn). Then by union bound, exact recovery can be achieved with probability o(1).
As the graph-splitting probability is 1 1/2, G1 follows SBM,, log 8 the rough estimate ((G,) satisfie
log .r.. By Theorem C.3
14000 log(2/8)
arr(t(G)).z <r== o(1) + (a B)&² log n
(C.9)
with probability at least exp(- (a))) < rn, since we consider -x otherwise. By Fact E.2 the maximum degree of G is at most A = 2 log" n with probability at least exp( (log n)² 9/3.. In the following, we condition our analysis on the above two events regarding ã (G,) and G.
Without loss of generality, we can assume
Now, let us fix a vertex and analyze the probability that it is misclassified after majority voting. With G2 being fixed, G2 can be thought of being generated by first sampling G and then removing
G from G. To make r < 1/16, it suffices to ensure a 3 5002
1ox(2/8)
by Eq. (C.9) Then by
log
Lemma C.16. we have
Pe < exp (-61 =(a B3 logn
1
(a - -8)2
+ exp
log
162
a
To make at most (1/n., it suffices to ensure
1
(a - 8)
c(a - 3) > and
> 1.
64
162
B
28 is easy to make the output of Algorithm C.4 balanced at the cost of increasing the enror rate by factor of most 2.
26

---

Note (a- (a- > (vā v3)² for > B. Therefore, as long as
vā v 3 > 16 and a 3 5002 log(2/6) 64
+
log
E
Algorithm C.12 recovers the hidden communities exactly with probability 1-o(1).
Proof of Theorem C.10 By Lemma C.15 and Lemma C.17
C.3 Inefficient recovery using the exponential mechanism
In this section, we will present an inefficient algorithm satisfying pure privacy which succeeds for all ranges of parameters ranging from weak exact recovery. The algorithm is based on the exponential mechanism [43] combined with the majority voting scheme introduced in section Appendix C.2 In
particular, we will show
Theorem C.18 (Full version of Theorem 1.4). Let rV > 12800 and x E {+1)" be balanced. Let < exp
there exists an algorithm, Algorithm C.19 which on impur G SBM, (%, x²) outputs an estimate (GG) E {=13" satisfying
>²d')
For any
64 (2/(()
512
err (
with probability least Ç. In addition, the algorithum e-private. Further, by slightly modifying the algorithm, we can achieve error 20/ (1g()/() with probability 1 --
29
couple of remarks in order. First, algorithm works across all degree-regimes in the literature and matches known non-private thresholds and rates up to constants. We remark that for ease of exposition did not try optimize these constants. In particular, for for constant weak recovery. We reiterate, that > > is the optimal non-private threshold. For the regime, where dd (1), known that the optimal error rate is exp (-(1 -1-1-o00 o(1))=²d) (-(1-o(1)>>')) even non-privately 64 where (11) goes zero as 22d tends to infinity. We match this up constants. Moreover, our algorithm achieves exact recovery soon as 72d dd > 512 log n since then < 3. This also matches known non-private threshholds up to constants
achieve
Also, our dependence on the privacy parameter is also optimal as shown by the information -itioon-tteretic lower bounds in Appendix C.4
We also emphasize, that if only aim to achieve
the order of
VV = + (Vioe(/aa)
can achieve exponentially small failure probability in while keeping the privacy parameter the same. This can be achieved, by ommitting the boosting step in our algorithm and will be clear from the proof of Theorem C.18 We remark that in this case, we can also handle non-balanced communities.
Again, for input graph G, consider the matrix Y(G) (A(G) _J) For {+1}" define the score function
sc(z) = (z, (G)a).
Since the entries of A(G) are in [0, 1] and adjacent graphs differ in at most one edge, it follows immediately, that this score function has sensitivity at most
2 max G-G
2
A max G~G"
[sG(x) so (x)) (E11)
(A(G))
y
7h
=E(+1)"
The first, smaller, error guarantee additionally needs the requirement that exp(-640). The second one does not.
27

---

Algorithm C.19 (Inefficient algorithm for SBM). Input: Graph G, privacy parameter & > Operations:
Graph-splittin: Initialize G1 to be an empty graph on vertex (G).. Independentl assign each edge of G to G1 with probability 1/2. Let G2 = G G..
2. Rough estimation on G2: Sample it from the distribution with density
p(x) x exp
2A (x, ((G)))))
where A
yd
Majority voting on Gz: Run the E-DP majority voting algorithm (Algorithm C.I3) with input (G2, (GG)). Denote its output by x.
4. Return &.
We first analyze the privacy guarantees of the above algorithm.
Lemma C.20.
Algorithm C.19 is E-DP.
Proof. For simplicity and clarity of notation, we will show that the algorithm satisfies 2e-DP. Clearly, the graph splitting step is O-DP. Step coorresponds to the exponential mechanism. Since the sensitivity of the score function is at most step c-DP [43]. Lemma C.14 by composition (ef. Lemma A.4)
7 it follows by the standard analysis of the mechanism that this
the majority voting step is also e-DP. Hence, the result follows
Next, we will analyze its utility.
Lemma C.21.
Let 7vd > 12800 and x E {=1}" be balanced. Let exp(-640) > < 64log(2/S) and ~ SB, x, the output à (G) E {=1)" of Algo dd
2d E
2 eexp
>
512
rithm C.19 satisfies
err (#(G),z") ES
with probability at least 1 C.
Proof. We will first show that the rough estimate 7 obtained in step 2 achieves
20
err (i,
Ilg(L1/C)
with probability em This will prove the second part of the theorem for this we don't need that < & exp(-640). In fact, arbitrary < works. The final error guarantee will then follow by[eema C.16 First, notice that similar the proof of (28] Lemma 4. IJ, using Bernstein's inequality and union bound, we can show that (cf. Fact H.2 for full proof)
max (+11)
(x,
(()
1 x="(x())
σ))
&
100n
5
T
Vlg (1/C)
with probability at least exp
-10n, 110m Recall that sG (z) (z, (G)x)) Let VloRti/c)' We
call x E (+1}" good if sG (x) > (1 - 3a)n. follows that for good x it holds that
(z. (2, (x, (z) | x. Y(()) - [x))) x x²)
> (1 - 144)m.
Which implies that
1 4a
4a
2 2err(r, (0) 11 v1 4a =
E Aa,
vI 4a
- 2a
2x
28

---

where we used that & 1/4 and that v1 - 4x < 1 - 2x for > 0. Hence, we have for good x that
20
err xx,
llg (1/c)
Since sG (xx) > (1 (e),,,thre is at least one good candidate. Hence, we can bound the probability that we do not output good x as
exp f (1 30)m) .e"
[1-200 25a (11-20)
exp
A EEn
23 ar))
exp
where we used that
2ca A
64 log (2/5).
5yd
> 320 (1/C) 2.
y
Vlgg (1/c)
We will use Lemma C.16 to proof the final conclusion of the theorem. In what follows, assume without loss of generality that Ham (x, x²) < Ham (I, -x*). The above discussion implies that
40n
Ham (x,x') x*) & 8on Vlog (1/C)
<
16
where the last inequality uses < e e-640 Further, by Fact E.2 it also follows that the maximum
degree of G2 is at most O (log* n) (by some margin). Recall that G2 SBM (d, 7,x"). In the
parametrization Lemma C. this means that
(1 + 7)
(1 - 7)
log
log
2yd
2d
a- B
+ 3 log
log n
Thus, it follows that the output of the majority voting step satisties for every vertex
(-10=
(a 8)2
P (a( # 2.) & exp
=(a - -B) llgg
+ exp
a B log
64
162
1
exp
eexp
7Pd)
162
<
8
By Markov's Inequality it now follows that
IP (err (f(G), > E
C.4 Lower bound on the parameters for private recovery
In this section, we prove a tight lower bound for private recovery for stochastic block models. Recall the definition of error rate, crr(u, v) = min(Ham(uu v), Ham(u, -u)} for u, [=1]". Our main result is the following theorem.
Theorem C.22 (Full version of Theorem 1.5). Suppose there exists an e-differentially private algo-
rithm such that for any balanced (=1)", impul G SBM,, 2), outputs & ((G) {+1) satisfying
((err()(G),aa) x) >1 1--1
30 where Then,
1/n A & 0.04 and the randomness is over both the algorithm and stochastic block models.
log(1/C)
log(1/1)
22 1 > s
+
(C.
7
Çneyd
EErrrr rate less than 1/n already means exact recovery. Thus it does not make sense to set to any value strictly smaller than 1/n. The upper bound ( 0.04 just technical condition proof needs for Eq. (C.12)

---

Remark C.23. Both terms in lower bound Eg. (C.10) are tight up to constants by the following
argument. Considering rypical privacy parameters & & I. then e2" -1 22. For exponentially small
failure probability, ie.. 7 2-9(m), the lower bound reads b 9 La Z). which is achieved by
Algorithm C.19 without the boosting step see the discussion after Theorem C.18) For polynomially small faiture probability, i.e. = 1/ poly(n), the lower bound Eq. (C.10) reads 2 92 - log t)
7d
which is achieved by Theorem C.18
By setting <= 1/n in Theorem C.22 we directly obtain a tight lower bound for private exact recovery as corollary.
Corollary C.24. Suppose there exists an E-differentially private algorithm such that for any balanced E {=1 input G SBM,, (d, 2), outputs 3 (GG) E {=1}" satisfying
P (& E (x,-x) > - 7)
where the randomness is over both the algorithm and stochastic block models. Then,
llg(n) + log 4
25 - 1 52
(C.11)
7d
Remark C.25. The lower bound Eg. (frr priavte exact recovery is tight up to constants, since there exists (inefficient) E c-differentially priavte exact recovery algorithm with & O log
and
7P
? 1/ poly(n) by Theorem C.18 and [55 Theorem 3.7).
The proof applies the packing lower bound argument similar to 22 Theorem 7.1]. To this end, we first show crr(.,-) is semimetric over
In rest of this section, we will prove Theorem C.22
{=1)".
Lemma C.26. err(-.) is a semimetric over {+1}".
Proof. Symmetry and non-negativity obvious from the definition. We will show err(:.)) satisfies triangle inequality via case analysis. Let u, w E {+1}" be three arbitrary sign vectors. By symmetry, we only need to consider the following four cases.
Case Ham(u, v), Ham (u, w²), Ham(u, w) < n/2. This case is reduced to showing Hamming distance satisfies triangle inequality, which is obvious.
Case 2: Ham(u, v), Ham w) & n/2 and Ham(u, w) > n/2. We need to check two subcases. First,
err(u, v) & err(u, w) + err(v, w) + Ham u, v) + Ham (U, w) < Ham(u, w) + n Ham u, v) + H(u, v) + H (u, w) < Ham(u, w) + n
Ham,u, & n/2.
Second,
err(v, w) < err(u, v) + err(u, w) + n < Ham (u, w) + Ham (u, v) + Ham w) - n < 2 Ham(v, w).
Case 3: Ham(u, v) < n/2 and Ham(u, w), Ham (v, w) n/2. This case can be reduced to case 1 by considering u, v, w.
Case 4: Ham(u, v), Ham(u, w), Ham w) > n/2. This case can be reduced to case 2 by consider- ing u, v, W.
Proof of Theorem C.22) Suppose there exists an e-differentially private algorithm satisfying the theorem's assumption.
We first make the following notation. Given a semimetric over (+1}", "e, a center v E {+1}², and a radius> r > O, define (v,r)) = (w E (+1]²" 1 = =0, v) $ r).
Pick arbitrary balanced x: (+1)". Let M (xr ,,,,..,, ....,, " be maximal 2 2ç-packing of
Bere (I, 4() in semimetric err(.,)) )). By maximality of M, we have Bar (z, 4C) C U Ban (xf 2(), which implies
|err (z, 43) <
Borr (x* 2()
30

---

BHam (x,, 4C)1 <
Σ
2
BHam (x', (2)] =2m |BHam (x. 2()
==1
4(m
BHam (x, 4(ra)
(2(m)
1
2Çn
2m >
(C.12)
Baam (r, 2(nil
(m/2)²
25
Sec
(n
2
For each i [m], define Y, = {w E [=1}" crr(w, x²) < (). Then Y''s are pairwise disjoint. For
each i [m]]. let P, be the distribution over n-vertex graphs generated by SBM,, 7; x*). By our assumption the algorithm, we have for any E [m]] that
P
( G) E Y)) 1/-
G~P,
Combining the fact that Yj's are pairwise disjoint, we have
T78
m
(#(G) e Y,)
P
E
E
Gam G-p
(f(G) E Y,) n. (C.13)
Gi
G
In the following, we will lower bound PG (+(G) E Y, for each i e [m] \ {1} using group privacy.
Note each P. is a product of (2) independent Bernoulli distributions. Thus for any i, E [m]. there exists a coupling Wij of P: and P3 such that, if (G, H) ~ w, then
Ham G, H) Binomial(Na; (Naj.
where p = 2yd/n and Ni- = Ham(x _ ) (n Ham (x+, x²) x²)). Applying group privacy, we have for any two graphs G, H and for any S S {+1}" that" 31
P (a(() E S) & exp(= Ham (aamG,Aa (G, - (+ S).
(C.14)
For each i E [m]. taking expectations both sides of Eq. (C.14) with respect coupling wa and setting S = Yi, we have
E
P ((t(G) EY)
&
E (G.H)own
exple Ham(G, P Yy).
(C.15)
The left side
Eq. (C.15) is equal to
E
P P(((G)
= P (a(G) E Y,) G-P:
> 1-17.
(G.H)
Upper bounding the right side of Eq. (C.15) by Cauchy-Schward inequality, have
E
exp(e Ham (G, H)) (&(HH E Y;)
(G.H)-ws
1/2
1/2
E (G.H)
exp(2e Ham(G, H))
E (G.H)owa
(& x2)²)
1/2
(c x3)²) 1/2
E B.-Boomiall(N...
exp(2e x)
E H
Using the formula for the moment generating function of binomial distributions, we have
E X-BBiomial(NNa.p)
exp(2= X) (1 p ppp.ppp... N
and it easy to see
(((H) E Y.)² = E (E (BL)E))))))
P. (&(H) E Y!). H
H~R
H-P;
Putting things together,
Eq. (C.15)
implies for each i E [m] that
(1 (1-m) 7)
P H-P.
(+(H) E Y)) (1- c Na
(C.16)
In Eq. (C.14) the randomness only comes from the algorithm.
31

---

Since æ² E (C) for E [m]] by assuming < 1/16, we have
N11 = Ham(æ*, x²) (n Ham (x' ,²))) & (Kn((=8(n)) 8(n).
(C.17)
Recalling 2yd/ n and combining Eq. (C.12) Eq. (C.13)
Eq. (C.16) and Eq. (C.17) we have
(1 - 1))
(m - 1) (1 p + p 22 j8cn(n (on((n-Caa 8(m)
< 17.
By taking logarithm on both sides, using > log(1 + 1) for any 1> 11, and assuming Ç < 1/(ee),
we have
log
log
2² - 12
+
y
(noyd'
D Private algorithms for learning mixtures of spherical Gaussians
In this section we present private algorithm for recovering the centers of a mixtures of lo Gaussians (ef. Model 1.2). Let y S (R.) Sn be the collection of sets of points in Rd. We consider the following notron of adjacency.
Definition D.1 (Adjacent datasets). Two datasets Y,Y' E y are said to be adjacent Y nY 1.
Remark D.2 (Problem parameters as public information). We consider the parameters n, k. A to be public information given input to the algorithm.
Next we present the main theorem of the section.
Theorem D.3 (Privately learning spherical mixtures of Gaussians), Consider an instance of Model 1.2 Let & E N be such that A > O (Vtk>/t). For 2 Ω (kOO) d0(0) ,k 2 (log n) there exists an algorithm, running in time (nd)o(t), that outputs vectors P1..... .......Ae fig satisfying
jike -Mxcelll & O(k-12),
max
with high probability, for some permutation [k] + [k] the algorithm is (e, -idifefrrettill private for any input Y.
32
Moreover, for k- 10
We remark that algorithm not only works for mixtures of Gaussians but for all mixtures of 2t-explicitly bounded distributions (cf. Definition A.I9).
Our algorithm is based on the oum-of-square hierarchy and at the heart lies the following sum- of-squares program. The indeterminates 211...., 21k.... ... Ank and vector-valued indeterminates u/....,14 .1.. will be central to the proof of Theorem D.3 Let n, k, t be fixed parameters.
2it
248
Vi [n] L [k] (indicators)
Zit & 1
Vi E [n]]
(cluster mem.)
RE|K]
Zit : Zi8' =
E [n] , E [k]
(uniq, mem.)
Σ Zit & u/k
3A [k]
(size of clusters)
k Σ
1/6
Zi8 1k
3A E [k]
(means of clusters)
k
VU E Id:
Zee (y)
1b, 2)2 1
=1 = (2s)* ||u112
Vs < t, E [k]
(t moment)
(Pn),,(Y))
We remark that chose constants to optimize readibility and not the smallest possible ones.
32

---

We remark that the moment constraint encodes the 2t-explicit 2- b-bundeedness constraint introduced in(Deiinition A.19 Note that in the form stated above there are infinitely many constraints, one for each vector v. This is just for notational convenience. This constraint postulates equality of two polynomials in v. Formally, this can also encoded by requiring there coefficients agree and hence eliminating the variable v. It is not hard to see that this can be done adding only polynomially many constraints. Further, the matrix variable Q represents the SOS proof the 21-explicit 2-boundedness constraint and we can hence deduce that for all 0 &
ly:
2))2 <2s)
Before presenting the algorithm we will introduce some additional notation which will be convenient.
We assume t. n. k to be fixed throughout the section and drop the corresponding subscripts. For Y E V, let ZY) be the set of degree- 10t pseudo-distributions satisfying P(Y). For each Ç E (YY define (C) as the n-by-n matrix satisfying
W()) E.
Σ EE(M]
Z5e Zze
We let W(Y)= == (W() 15 z(Y)).
Recall that J/ denotes the all-ones matrix. We define the function 9 Roxn - R as
9(W) = ||W|| (10) 300
300 (J. W
and let
W(C(Y) argmin sW).
We also consider the following function
Definition D.4 (Soft thresholding function). We denote by ò (0, 1] + [0, 1] the function
0
ix < 0.8.
(xx)
ix > 0.9, otherwise.
0.8.8 0.9-0.8
Notice that ((-) is 0.9-0.8 Lipschitz. Next we introduce our algorithm. Notice the algorithm relies on certain private subroutines. We describe them later the section to improve the presentation.
33

---

Algorithm D.5 (Private algorithm for learning mixtures of Gaussians).
Input: Set of a points Y S TR ,,, > 0, O, L N, d 100 log n, _ k:
Compute (((()))
(-n2.6 2.6 (1+ (oR(1/6)
LG
2. Pick T ~ 1Lap
3. If [ 22 n. or (1)( < "h
(1- non
1
reject.
4. For all [n], compute the n-dimensional vector
0
if |lo(W))ll
(0)
[lo(Wall'' E, ((We)) ys
otherwise.
5. Pick set 8 of n001 indices i [n] uniformly random.
For each 8 let D( vl0 + where N (0. log(2/8) Id
7. Pick 1 ~ N (0, ) ddd xd u.a.r 10, b] and run the histogram learner of Lemma A.13 D and parameters
with input (
8
10k
q a 3 =-,====--
n2.01
n
Let B,...., randomly.
Bg be the resulting d' dimeensiion bins with highest counts. Break ties
8. Reject if min Kj ΦD!i) E B. <
x 0.01
2k
9. For each l [k]] outpur
Σ
zli) =
+w',
{j FD( eB.]
#pb)eB
where w' N (o. (0,32. -120 lom (2km /6)
120
Id
For convenience, we introduce some preliminary facts.
Definition D.6 (Good Y). Let Y be sampled according Model 1.2 We say that Y is good if:
1. for each E [k],, there are at least To - 0.6 and most z + n 0.6 points sampled from D in
Y. Let Yt Y be such set of points.
2. Each Y is 21-explicitly 2-bounded.
turns out that typical instances Y are indeed good. Lemma D.7 30 36). Consider the settings Theorem D.3 Further, in this case the sets (YY and W(Y) are non-empty.
Then Y is good with high probability.
D.I Privacy analysis
In this section we show that our clustering algorithm is private. Lemma D.8 (Differential privacy of the algorithm). Consider the settings of Theorem D.3 Then Algorithm D.5 is (s, 5) diffferentiall private.
We split our analysis in multiple steps and combine them at the end. On a high level, we will argue that on adjacent inputs Y, Y' many of the vectors vن by the algorithm are close to each other and small part can be very far. We can then show that we can mask this small difference using the
34

---

Gaussian mechanism and afterwards treat this subset of the vectors as privatized (cf. Lemma 1.4). Then we can combine this with known histogram learners to deal with the small set of p that is far from each other on adjacent inputs.
D.1.1 Sensitivity of the matrix W
Here we Lsema B.1 to reason about the sensitivity of ob (W((C)))) For adjacent datasets Y, Y' Y' we let Č ç' be the pseudo distriiiittiittionn corresponding to W W(C(Y) and W({(Y') computed in step oof the algorithm, respectively. We prove the following result.
Lemma D.9 (f1- (e-ssenitiivtyy of ((W)), Consider the settings of Theorem D.3 Let W, W be re- spectively be the matrices computed in step 1 by Algorithm D.3 on adjacent inputs Y, Y' E y. Then
[(((W) *(W)) & <n
For all but n.8 rows of ((W), (W"), holds
lo(W); <(W))ill nn.
Proof. The second inequality is an immediate consequence of the first via Markov's inequality. Thus suffices to prove the first. Since æ(.) 10-Lipschitz, we immediately obtain the result
((C()) - W(((")) & n 1.55
Thus we focus on this inequality. To prove we verify the two conditions of Lemma B.1 First notice that g is 2-strongly convex with respect to its input W. Indeed for W, W E W(Y), since Vi,j [n] We > it holds that
[w' ||WI| + I|W - w'] + 2(W" - W,
|W + I| w'll + 2(W - WW + (W' W,
J)}
=(W) + |[ - w'll" + (W' W, V.))) + (W", (10))0,300) J)
where we used that Vg(W) 2W (3000. Thus it remain prove (i) of Lemma B.1 Let Ć 2()), ZYY') be the pseudo-distributions such that Wy((C) W and Wy Wy(C") W We claim that there always exists Suyj E 2 (Y) ZY") such that
1. (s(W(()) (Gat))) < 2n (110)10,,00 + +1 < 3 (10) 3000
2. 1gr (W (Gatt)) 0.
Note that in this case the second point always true since g doesn't 't depend on Y. Together with Lemma B.I these two inequalities will imply that
W((C)) - (((t))))((((())))))) < 18 (10) 300
By assumption on n, an application of Cauchy- -ehhhar will give us the desired result.
So, let be the index at which Y, Y differ. We construct Suij as follows: for all polynomials p of degree at most 10t we let
[ [p] if does not contain variables ziz for any E [k] otherwise.
Ec,, [p]
By construction Suij E 2(Y) n (YY). Moreover, W(C), (Gaa)) differ in at most 2 entries.
Since all entries of the two matrices are in [0, 1), the first inequality follows by definition of the objective function.
D
35

---

D.1.2 Sensitivity of the resulting vectors
In this section we argue that if the algorithm does not reject in step 3 then the vectors u(0 are stable on adjacent inputs. Concretely our statement goes as follows:
Lemma D.10 (Stability the ")) Consider the settings of Theorem D.3 Suppose Algorithm D.5
does not reject in step 3 on adjacent inputs Y. Y' EV. Then for all but k5o indices E [m], it holds:
or
| - -49-1 <0(n
The proof of Lemma D.10 crucially relies on the next statement. Lemma D.11 (Covariance bound). Consider the settings of Theorem 12.3 computed by Algorithm D.5 on imput Y E y. For i E [n], if [o(())ll > induces a 2-xxpiicuty 4U-bounded distribution over Y.
Let W be the matrix
10
then v
g00
Proof. First, by assumption notice that there must be at least E (1 -
1 entries of ((W)) larger
10
than 0.8. We denote the set of j E [n] such that Wij 2 0.8 by 9. Let Ç E ((Y) ) be the degree 10t -eerrdddistrbuutio that W (((Y)). Since satisfies P(Y), for l [R] it follows from the moment bound constraint for = that for all unit vectors u holds that
4
0
Ege Ejely;
14, u)² <
71
a.b
Using the SOS triangle inequality (ef.
Fact 1.2)
(a + b)2 < 2(a² + +12 it now follows that 2
k2
E
ZzeZf' (yj - - 82
3 8ld
n
j'E[[]
and thus
2
3 Es 0
Σ
2st 2122z8220 (y) )82
8ld.
n1
LE(k] j J'E[m]
Furthermore using P(Y) t teeeze
0} for o l' have
Ec 242z = E5
2seZze
Σ
kE(k)j''ER
(E(R) JE[n
(EE()
Now, for fixed j.j' [n], using
{a² a, b² = bb 0(1) {1 ab -- b- ab (a - b)2 > 0}
with EiE(H) Zie2ze and b EreR 240 zye we get
E5
ZieZze
Σ ('E(A]
z4e Z
ES
Σ
Zee3EE +
Σ
2str Zyre
CE[K]
(E(K]
Wy + Wey 1.
Now if j,J 9 we must have
Σ
Eg 2e2 = E5
Σ
25e
Σ zie 2
0.6.
KE[K]
BE|M
(EE(K]
10 Since ((W)) & 1 by definition and [[o(W))ll > E (1-
). we conclude
R
1o(W))li= -2
Σ
(W 02
36

---

35. k2
Ee [2le (1) yy 802
n²
j''e(m))
3 40ld.
as desired.
We now prove Lemma D.I0
Proof of Lemma D.10 Let W, W be the matrices computed by Algorithm D.5 in step on input Y, Y,Y', respectively, Let G [n] be the of indices i such that
(<W)); ;
(W"):ll < n...8
Notice that [g] > n must have
0.8 by Lemma D.9 Since on input Y the algorithm did not reject in step 3
n² 1 1
(1
n²²
(1
2
n 1.7
n.
1:100
k
1:100
Let gw be the number of indices i such that [[o(W)
)
It holds that
Re0
n
(1
2
1
1:100
gw
(n
[9])
+
(G)
gw) k
1
5050
2
< gw
+
1,50
k-50
n
n
< gw
+
1
+
k
k50
k
1000
k550
Rearring yields
gw 2 n
A-50
Similarly, let gw be the number of indices such that llo(W") > (1- By
analogous argument follows that gw n (1- ). Thus, by the pigeonhole principle there are at least gw e n (1 ) indices such that
|lll): > t x
2. [0(W"):ll > _ (1 К
3. [o(WW, o (W'),ll < 0...
Combining these with Lemma D.11 we may also add
4. the distribution induced by [[o(W))]] (WW)) 2-explicitly 40-bounded,
5. the distribution induced by [[o(W) ((W)) is 2-explicitly 40-bounded.
Using that for non-zero vectors ær, it holds that imply that
2
|
wll points 1 to 3 above
0,8
||o(Wa)] ((W))
((W,) ( (WWII (W)
(n-20.)
Hence, applying Theorem A.2I with /. follows that
( u60 ||_
$ 00
37

---

D.1.3 From low sensitivity to privacy
In this section we argue privacy of the whole algorithm, provingLLemma observe that low-sensitivity is preserved with high probability under subsampling.
Lemma D.8 Before doing that we
Fact D.12 (Stability of S). Consider the settings of Theorem D.3 Suppose Algorithm D.5 does not reject in step 3 on adjacent imputs Y, Y' E V. With probability at least e
over the random
choices of S, for all but 10n ksa indices E S, it holds:
10 <0(n-.1)
Proof. There are at most 6n the claim follows.
such indices in [n] by Lemma D.10 By Chernoff's bound, cf.
Fact E.4
Finally, we prove our main privacy lemma.
Proof of Lemma D.8 For simplicity, we will prove that the algorithm is (5),50) -private. Let Y,
Y be adjacent inputs. By Lemma A.10 and Lemma D.9 the test in step of Algorithm D.5 is (eivv-rrivaee
Thus suppose now the algorithm did not reject in step 3 on inputs Y,Y' By composition (cf. Lemma A.4) it is enough show that the rest of the algorithm is (e, 6)-riiatee with respect to Y,
under this condition. Next, let vy ..... ( and v/")
u . ........, be the vectors computed in step
of the algorithm and S be the random set of indices computed step 5 F3 By Lemma D.10 and
Fact D.12 with probability e- n over the random choices of S we get that for all but 10m 0.02
100
indices x S, holds that
|0 - - & (n
Denote this of indices by G. Note, that we may incorporate the failure probability min {e/2, 6/2} into the final privacy parameters using Fact I.3
R()
Denote by V. V the SS--yy-d matrices respectively with rows vy where . are the indices in S Recall, that []] rows of V and V differ by at most O (n 0.1)
(61) By ((i)) and vy
(1)1)
in f2-norm. Thus, by the Gaussian mechanism used in step 6 (cf. Lemma 1. and LnmLmmma 1.4 it is
enough to show that step to step of the algorithm are private with respect to pairs of inputs V and V differing in at most row In particular, suppose these steps are (51, 81 --private. Then, for
m 0.01 - 119
9.01
by Lemma 1.4 it follows that step to step 9 are (s', (=',8")-diffeee dedd8ddiiffeeeniially
100
private with
e' == 8 + mE1,
8' = e e (m- 81 5.
Consider steps 7 and g Recall, that in step 7 we invoke the histogram learner with parameters
8 _ee =
10
b k u.a.r. [0, a 10 3 =n
10 5t
00001
Hence, by Lemma A.13 this step is (E*,8*) ( private since
200 20 0.01
20 n 0.01
log
log n
llog n < n,
Exa
5
10 1000
h00
for 8 >> k 10
Step
is private by post-processing.
Note that since this does not depend Y or Y', respectively, cases. Formally, this can be shown, e.g., direct calculation or using Lemma A.4
this be the same in both
Note that for the remainder of the analysis, these do not correspond to V and V since those differ in m Lemma 1.4 handles this difference.
38

---

Next, we argue that step i private by showing that the average over the bins has small l fz-sensitivity. Bememmaa A.4 we can consider the bins ....... Bs computed in the previous step as fixed. Further, we can assume that the algorithm did not reject in step 8 i.e., that each bin contains at least n no. points of V and V' respectively. As a consequence, every bin contains at least two (projections of) points of the imput V or V' respectively. In particular, it contains at least one (projection of a) point which is present in both V and V Fix a bin B, and let 1 be such that is both in V and V and 2D E Bj. Also, define
24
S =
{
( D Y
E
B B
Si = j
1 1
Assume V and V differ on index j. We consider two cases. First, assume that Do and $ both lie in Br. In this case, St = St and using Lemma E.5 follows that with probability n 100 min {e/2, 6/2] it holds that
10 (1 (\07)) 10
27 11)
1
+
20 20 dd b & 200 k- k
And hence we can bound
Dy
400 k-11
Σ
Si
S
S
n.01
EB
2 eB.
Next, assume that Doll
B, and Boll
E B, (the other case works symetrically). It follows that
S = S and we can bound
1
Σ
Σ
S Σ
(S)
-
1)
Σ
v(
5
5
S S
EB
EB
40 EB.
EB,
2
St D +
Σ
S S
40 EB,
D
Σ
D
S
S
ΦD EB:
d b
20 --11
St
0.01
Hence, the Éz-sensitivity is at most A
100 11
Since
00
242 1og(2/(5")
/b:) 32 k
log(2km/5)
120
(6*
(²*/k)2
62
and w' N (0, 32 k 1og(2kn/0) 2
ld it follows that outputing fú is (E*/k, 8* /k))DD by the
Gaussian Mechanism that. By Lemma A.4 it follows step 9 is """,,""--iivate
Hence, by Lemma A.4 it follows that step 7 to step are (2e", 25*) -diferrentially private. Using
m < 10n 6.07
now follows by Lemma I.4 that step 6 to step are (=', ('',8')-privatt private for
210
= + 2ms" < 3E,
8 + < 38. = 2 me (m-1)25" (m-1mm))) 8* + 8 < 2me
Thus, combined with the private check and Fact 1.3 in step 3 the whole algorithm is (5e, -privviee
39

---

D.2
Utility analysis
In this section we reason about the utility of Algorithm D.5 and prove Theorem D.3 We first introduce some notation.
Definition D.13 (True solution). Let Y be an input sampled from Model 1.2 Denote by WY(Y) W(Y) the matrix induced by the true solution (or ground truth). Le., let
{
ifi,j were both sampled from the same component of the mixture, otherwise.
W" (Y)
Whenever the context is clear, we simply write W to ease the notation.
First, we show that in the utility case step Algorithm D.5 rejects only with low probability.
Lemma D.14 (Algorithm does not reject good inputs). Consider the settings of Th D.3
Suppose Y is a good set as per Definition D.6 Then | 3 x²² K and Algorithm D.5 rejects with probability most exp (-52 (n ()).
(1 (1-18 n 0..4
(10) 10000
Proof. Since Y is good, there exists W E W(Y), corresponding the indicator matrix of the true solution, such that
n² + 1.6 (10) 20
n n 1.6
g( (')
||
10
2000
(J. (J, W*)
k
n²
k (10)
))
k
1 +
1
0.4
0.4
Since ((W(((())) < g( (")) it follows that
(10) 1000 (J, W (((())) > (((W((())) n (110)102300 (1
k
k
n2. 0.4
Since, | > (J, (((())) the first claim follows rearranging the terms. This means
that the algorithm rejects only if ||| > 1.77 Recall that t ~ tLap (-n (1 log(1/0) x 1.0
Hence,by Lemma A. follows that
exp (-n! + log(1/8)) exp (-12 (22((1))))
( > n 17)
exp (-e - log((1/))
The next step shows that good input Y the matrix (((() is close the solution.
Lemma D.15 (Closeness to true solution on good inputs). Consider the settings Theorem D.3 Suppose Y is a good set as per Definition D.6, Let W(Y) E W(Y) be the matrix computed by Algorithm D.5 Suppose the algorithm does not reject. Then
n
()(W(Y)) w <
k 1.9
The proof is similar to the classical utility analysis of the sum-of-squares program found, e.g., in 26). We defer it to Appepdixx II
Together, the above results imply that the vectors ()) computed by the algorithm close the true centers of the mixture.
Lemma D.16 (Closeness to true centers). Consider the settings Theorem D.3 Suppose Y is good set per Definition D.6 Let W E W(Y) be the matrix computed by Algorithm 1).5) Suppose
the algorithm does not reject in step Then for each & E [Kl], there exists 447 indices
i E [m), such that
2000 (W)
He/ < <0 (k-)
40

---

Proof. We aim to show that for most indices E [n] the vectors (o(W))) ((W,) and ||i W; induce 2-explicitly 40-bounded distribution over Y. If additionally the two vectors are close in (-norm, the result will follow by Theorem A.21
Note that [l Wi induces a 2-explicitly 40-bounded distribution by Lemma D.7 By Markov's inequality and Lemma D.15 there can be at most n/k46 indices j E [x] such that
[({(W) will,
n h
A.50
Consider all remaining indices i. It follows that
k
3
10
[1o(W))l > ||ilh (lo(W); will K
0.4
550
k
A-50
Hence, by Lemma D.11 the distribution induced by ((WW)lll" (WW) is 2-explicitly 40-bounded distribution. Further, using [|ll| > (1 _ we can bound
|o( <(W)) II W | W IIT |W
|I W <(W,) (lo( Will, (W.)l, I|1 - || 1 wll,, + ( (WWl) 16 (.) - wil))
< [[o(W.) |W (IIl((W,)
|W - (l;) will 50
A.50
Hence, by such that
Theorem A.21 for each / [k] there are least B
0.6
(1- ( indices
( (w) - WIT Σ Wiyy,
0 ( 2k-
The result now follows by standard concentration bounds applied to the distribution induced by ||W11] Wy
An immediate consequence Lemma D.16 is that the vectors 7 inherits the good properties the vectors u(i) with high probability.
Corollary D.17 (Closeness to true centers after sub- sampling)) Consider the settings of Theorem D.3 Suppose Y is a good set as per Definition D.6 Let W E W(Y) be the matrix computed by Algorithm D.5 Suppose the algorithm does not reject. Then with high probability for each & [k].
there exists
1.01
150
indices i S, such that
k
[ | uell < (k
Proof. For each é [k), denote by Te the of indices in [n] satisfying
|
(W) nell < O (k= (k-26)
By Lemma D.16 we know that Te has size at least (1
k Further, S be the set of indices
selected by the algorithm. By Chernoff's bound Fact E.4 with probability
e 90)) we have
k 1.0). Taking a union bound over all l E [k)] we get that with probability 1 R() for each l [k], there exists a k (1
|S n Tel > 0.001
0.01
150 indices S such that k
1000 (W) uell & O(k*** (6-25)
Now, we obtain the corollary observing (cf. Fact E. with m = 1) that with probability at least
1 - 200 for all i E S
100 10 (w))l = /|wl/ < n -005 Vlg(2/8)
Vd < n -004 & $0 0(k (1:-
41

---

For each L, denote by Ge C S the set of indices i S satisfying
1 - -Mel1 < < (k
Let G U Ge. We now have all the tools to prove utility of Algorithm D.5 We achieve this by ee(k]
showing thst with high probability, each bin returned by the algorithm at step 7 satisfies Gr S B
for some e, E [k]. Choosing the bins small enough will yield the desired result.
Lemma D.18 (Closeness of estimates). Consider the settings of Theorem D.3 Suppose Y is a good set per Definition D.6 Let W E W(Y) be the matrix computed by by|Algorithm D.S Suppose the
algorithm does not reject. Then with high probability, there exists a permutation क [k] + [k] such
that
| in & (k(k-2))
max [E(K)
Proof. Consider distinct E E [k:]. By Corollary D.17 for each Di() xt) E Gr it holds that
| b60ll C k 25
for some universal constant C > Moreover, by assumption on pe, pe for each n E Ge and p E ge
3 A - 0
Thus, by Lemma E.5 with probability at least z(i), 100) E Ge and D E Ge with f' +l,
> 1 - n 100 it holds that or each cl(d')
C k k-
and
|| |DD -bD(")
> C -25
for some other universal constant C* C. Let Q+ C Rdd be ball of radius C* (k-25) such that E 93 it holds 4p(()) Q+ That is, Q9())) contains projection all points in 9:. Recall that d² 100 log(n) < 100 and b = k-15, Let B
(B. be the sequence of bins computed by the histogram learner of Lemma A.13 for Rd" at step 7 of the algorithm. By choice of and since q is chosen uniformly at random in [0, 6]. the probability that there exists a bin E B containing Q+ (Gr) is at least
C
100C k-20 >1 Ok-3)
d'
26
b
where used that d= 100 log 100k A simple union bound over & [k] yields that with high probability for all E [R]. there exists B E B such that (Gr)) C B. For simplicity, denote such bin by B
continue our analysis conditioning on the above events, happening with high probability, First, notice that for all ( E [k]
A
max ||u v' < d= 13 100% 25 u.u EB,
k1
and thus there cannot be l, [k] such that Q+ (9)) C B, and Q4(9') C B₁. Moreover, by Corollary D.17 and
0.01
150
min
Ge
KE
k
f:47
and hence
150
0.01
150
S 9
< n
0.01
A-47
k
f-16
it must be that step returned bins B B B This also implies that the algorithm does not reject.
Further, by Lemma E.5 for all [((), (ااا such that tz((), pD7()) E B, it holds that
[000 n0) < c² ( Φこ61
< C Vd . (k-12)
42

---

And hence, by triangle inequality, we get
| 11 {
Finally, recall that for each E [k].
p
Σ x
+ tw
{il 4D() E BB,}
BODIE,
where w' N (0. N (0.32. 1-120 log(2kn 8²
Id
Since by choice of n, k, E it holds that
32 ko 120 log log(2kn/0 $ -0-
we get with probability at least bound that
e ( for each & E [k), by Fact E. with m 1, and union
|w < O (k (k-
Since all (6)) such that (1 conclude that
E B₁ are at most O 11-1 -far far from (4,, also their average is. We
pe -1112 & 1(k
|w/l < O(k-12).
This completes the proof.
Now Theorem D.3 is trivial consequence.
Proof of Theurem D.3 The error guarantees and privacy guarantees immediately follows combining
Lemma D.8 Lemma D.15
Lemma D.14 and Lemma D.I8 The running time follows by Fact A.16
E
Concentration inequalities
We introduce here several useful and standard concentration inequalities.
Fact F.1 (Concentration of spectral norm of Gaussian matrices). Let W ~ (0, Then for any I, we have
P P(ym Vn- t & minu (W) & maxx (W) & Vm +t+ > 1 2 exp
where mmin() (.) and m.nax()) denote the minimum and the maximum singular values of matrix, respectively.
Let W' be an n-by-n symmetric matrix with independent entries sampled from N 02). Then
||W|| < /n with probability at least 1 - exp(-S(rt)).
Fact F.2 (Maximum degree of Erdös-Rény graphs). Let G be Erdös-Rényi graph vertices with edge probability p. Then with probability at least 1 - n exp(-np) (nn/3)) any vertex in G has degree most 2np.
Fact E.3 (Gaussian concentration bounds). Let NN o²). Then for any 2 0.
12
max {IP (X (X 1)} & exp
2
Fact E.4 (Chernoff bound). Let XI,..., Xn be independent random variables taking values in
(0, 1). Lei X En Xi and let = E X. Then for any ò 0,
82
P < (1 (1-88)6 8)P4) exp
52
P (1+ exp
2 2+
Lemma E.5 (311). Let & be d-by-n Gaussian matrix, with each entry independently chosen from N NO, 1/d). Then, for every vector u E Thn and every a E (0, 1)
P ([ (I + a) [ull) > 1 2(d?d)
43

---

Linear algebra
Lemma F.1 (Weyl's inequality). Let A and be symmetric matrices. Let R A B. Let
01 > .... 3 On be the eigenvalues of A. Let B1 2 ... 3 Sn be the eigenvalues of B. Then for each i E [n].
la: 3:1 < ||||
Lemma F.2 (Davis-Kahan's theorem). Let and B be symmetric matrices. Let R A B. Let 01 2 ... 3 be the eigenvalues of A with corresponding eigenvectors U1...., Un- Let 31 3 ... 2 Bn be the eigenvalues of with corresponding eigenvectors ui...., Uy. Let 0 be the
angle between iı; and Lu; Then for each E [ml]
R|
sin()0,) < minj aa ajl
G Convex optimization
Proposition G.1. Let f : R + R be a convex function. Let K. C Rd be convex set. Then y E K. is minimizer of f over K. and only if there exists subgradient 9 Oy such that
w y .g)
by E K.
Proof. Define indicator function
00 J yEK, xc,
1x(y)
& K.
Then for y E K. one has
0K((y) = [g ERT (0.0- v/) > 4AA EK).
Note y" is a minimizer of f over K. if and only if y is minimizer of over IR" if and only if Om E Of + (z)(())) = 0f(r") + (yy)"" ), if and only if there exists E 0f("') such that yg.y y²) > 0 for any y K.
Proposition G.2 (Pythagorean theorem from strong convexity). Let f R + R be convex function. Let KC S "n be a convex set. Suppose f is k-strongly convex over K. Let x- E IC be minimizer over K. Then for any x E K., has
- x 2
2
Proof. By strong convexity, for any subgradient g =f(x*)) one has
f(x) > f(x*) + (x x=
+ K
|| - 111
2
By Proposition G.1 (a x9 > for some g E 0f(x') Then the result follows.
H Deferred proofs SBM
We prove Lemma C.9 restated below.
Lemma H.1 (Restatement Lemma C.9). Consider the settings of Lemma C.8
With probability
1 exp( (n)))) over G ~ S,,,
X(Y(G))
800
ET
n
F
Proof. Recall == (X E Raxn; X 2 0, X, 1/n Vij. Let X = T.x Since X= X(Y(()) is a minimizer of minxex| (G) XII and X K. we have
8 | || - =
x x x
2 < - -X))
44

---

The infinity-to-one norm of a matrix M mxxx is defined as
| xx--11 = max ((u, Me) a E {=1]", {+1)").
By 28 Fact 3.2], every Z E K. satisfies
( KG (|) X" l
n
where KG & 1.783 is Grothendieck's 's constant. Similar to the proof of 28 Lemma 4.1], using Bernstein's 's inequality and union bound, we can show (cf. Fact
100n
( - 1
with probability 1 -- exp(-x((n)) Putting things together, we have
X(Y)))) -xx
400 KG
n F yVd
with probability exp(-S((n)).
Fact H.2. Let γ > 0,d E N E {=1}", and G ~ SBM (r, "r,d,x"). Let Y(G) =
yd (A(G) - JJ. where A((G)) is the adjacency matrix of (G) with entries d/n on the diago- nal. Then
100n
(Y(()) (x²( 7
max 2E(#1)
with probability at least 1 e-10
Proof. The result will follow using Bernstein's Inequality and a union bound. Define E = Y(G) L (x)).. Fix E [=1]" and for 1 & i < < n, let Zis = Eigtt Then x² Ex 2 E DiskjEn Zz.j. Note that
E Zi,j = 0,
1 (
d 11) ydn
1
y
d
1
(" -1)
1
1
E 2 Var (G)a) < EY ii < (1 +
nnn
22
3
(1+
d
n2n
2n
By Bernstein's Inequality (cf.
62
Proposition 2.14]) it follows that
10
50n vVd
2
100n
P
Σ Zij
>
< P
Σ
Zi,
< 2 exp
vd
3
100
i<j
{<)
7dn
3-
10tn
2exp
< exp (-50m)
3
100 vd
Hence, by a union bound over x E {+1)" follows that
100m
x - -x
max =E(#1)"
with probability at least 1 10nn
45

---

I Deferred proofs for clustering
In this section, we will prove Lemma D.15 D.15rrestatdd below.
Lemma (Restatement of Lemma D. Consider the settings of Theorem D.3 Suppose Y is good set as per Definition D.6 Let (Y) E W(Y) be the matrix computed by Algorithm D.5 Suppose
the algorithm does not reject. Then
n2 k A95
lo(W(Y)) - W/
11
<
We will need the following fact about our clustering program. Similar facts where used, c.g.. in [3 (6).. One difference for us is that we don't 't have constraint on the lower bound on the cluster size indicated by our SOS variables. However, since we maximize variant of the norm of the second moment matrix of the pseudo-distribution this will make for this.
Fact I.1. Consider the same setting as in Lemma D.15 Let O < & Thino K and denote by C1...., Ck [n] the indices belonging to each true cluster. Then W(Y) satisfies the following three properties:
1. For all i,j [n] it holds that 0 & Wij & 1,
2. for all i E [n] it holds that Σ}=1 W < a and for at least (1 1000k000 ) indices E [n]]
it holds that E Wi > (1 (10)58200
3. for all E [k] holds that Eiec..jc. W. E
2 K
We will prove Fact 1.1 at the end of this section. With this in hand, we can
D.15)
Proof of Lemma D.15 For brevity, we write W W(Y). Since s(W") W and & is 10-
Lipschitz we can also bound
[(((W) will, I1, & 10 | w
Let 8 & 1.5-10 paw and again let C₁, C₁ C [n]] denote the indices belonging to each
true cluster. Note that by assumption that good sample it holds for each r E [k] that Rt n < Crl
zt + n.
Let r, E [kt]. We can write
|W- w = Σ Σ
W 1| Σ Σ
WLs - 0
(1.1)
r=1 JJEC.
JEC.
Note that we can bound the second sum by ht 5 n
T using Item 3
Further, in what follows consider
only indices such that Σド= W., jij (1 (10) of the other indices
By Item 2 we can bound the contribution
C" + ne n.
2
n
1000
10000-100
k
Focusing only on such indices, for the first sum in Eq. (L.1) fix r E [k). We will aim to show that most entries W are large and only the corresponding entry of W is By Item and Markov's Inequality, it follows that for at least (1
1000k100 -fraction of the indices i E C, it holds that
Σ Was & 1000 " IEC.
< 10000-100 Tー & 20002 101 N
K.J.
where used that |C th 0.6. Call such indices good. Notice that for good indices it follows using [tem 2 that
Wis # Σ JEC.
1
2000 101 8).
(10)6
46

---

Denote by G the number of j E C, such that W >
Using the previous display and
1000
that Wis i. < 1 obtain
(1 -1 (10)6 20004 1014
<
Σ
G). (1 1000 TOT
JEC.
< G 1000 TOC
+ kn (1
1000g100
$ G 1000 TOU
E (1 kn
where we also used
|r|l
<
n. Rearranging now yields
G
71 k
1
1
2 1.10% 8
>
n
(1
10004 - 2 101
1000 100
n0.4
k
We can now bound
Σ Wis 1
Σ
W 11+
Σ
Waj 11
IJECC
C.ECC. is good
i.jEC., good
|C,|- ((|C,|- G) |Crl
1000 +
10000100
|C,|
| (1+ 500l)
G. Crr
(1
300
2
1000
2 10
n
(30 10 101 +
11
n²
(30 - 100h 100 8 +
11
500k
500
W
125
Putting everything together, it follows that
2
[(o(W) w < ([((W) w & 10 4
8k
+
1000k-100
125
3
n"
K
It remains to verify
Fact 1.1
Proof of Fact I.1 Let P
Pn, the system
Eq. (P (Pn,,((Y))
Recall that W.
Eie( 21,15, Since
P 4
0
2,12 & Σ 24 <
[E(k]
[c(K)
it follows that 0 & Wid & 1. Further, for each [n]] it holds that
P 4
Σ
21,12.1
24,1 &
JE(M
(E(R]
implying that Σje(n) W i
$ x. Further, by Lemma D.14
n² n 0.4
2 k
(1-
W 11
>
k
(10) 300
(10)
Denote by W, the i-th row of W and by L the number of rows which have E norm at least [10] F. Since for all i it holds that |Well1 < E it follows that
n
k (1
9-300
Will, < L.
K
+ (n - L) (1
(10J
n
k
n n²
-1
L
[10)9220
k k
(10)og200
47

---

Rearranging then yields L > (1 10000 which proof' Item 2
It remains to verify Item 3 Fix r, [k]] and define (C,) = A Eiec, Zi,. Let > 0 be an integer.
We aim to show that for all unit vectors it holds that
8 k
100
(C,)
(C,)) - ue, w)2t M'rr
Σ
(1.2)
2
where A is the minimal separation between the true means. Before proving this, let us examine how we can use this fact to prove Item 3 Note, that for all r o r' it holds that
(ur Me's P. Pall
A 2t
[
Hence, if the above SOS proof indeed exists, we obtain
n²
Σ
W
E
Σ
21,1211 =
E2 (C,) Σ (C,)
122
iEC,.J&CC
1=1
VEC..
r'zr
(c,(C,): Σ ((Cr) |r -oy;;
21
222
like -
r'sr
8
1
n2a
k2
8
k
k
In the remainder of this proof we will prove Eq. (1.2) triangle Inequality (ef. Fact I.2)
We will use the following SOS version of the
R.y 28
c + y)
< 2 (a. + y²t).
Zi= 2.194 and denote by Mat()) the true mean corresponding to the i-th sample. Let be an arbitrary unit vector, it follows that
Recall that 14 E
P 100
(z,(
Σ (C,)) -ky,, u) 0)
2
2
< (C,))
(C,))) - (1 1)2 + (pp - ph, 2)
r'gr
2
221-1 2
k n 2.1(M7(1) - H/, v),21,.
(C+) 14, 0)
A
n
where used that P
1
Σ z1(C)) & 1. Using the SOS triangle inequality again and that
P 2 2d,1 < 1 we obtain
P 10c
{
(C,)
Σ 2 (CCr) - Myr, 2)
2
r'
21-11
k
1
yy, MT(1), e)2t
+ h: 14, - 1)20 0)00
Σ an
2
n
We start by bounding the first sum. Recall that by assumption the uniform distribution over each true cluster is 2f-explicitly 2-bounded. It follows that
1 k Σ ly. Mr 2)22 <
Σ k |c,l (2t)* ||01 (I.3)
y Ma(1),
Σ
21
k
iec.
k
1 1+ u.
(2t)² < 2((t)'),
(I.4)
48

---

where we used that |C,| < constraints. particular, we know that
+ n. To bound the second sum, we will use the moment bound
maly; -4, (20))
P 10z
(I.5)
Combining
Eq. (1.4) and Eq. (I.5) now yields
22t+1 (26) <
8t A
(C.)))
Σ r'
(pr -)),
21
2
Note that by assumption A > ((tt)/) Overloading notation, we can choose the parameter in the SOS proof to be 202 times the parameter in the lower bound in the separation to obtain 33
Wis < n
Σ
h
IEC.
L1
Small Lemmas
Fact 1.2 (Lemma A.2 in (36 For all integers > 0 holds that
E.y 28
(x
y) < 2x + )
Fact 1.3. Let e, > O. Let M: + be a randomized algorithm that, for every pair of adjacent inputs, with probability least privacy. Then M is (E + 27, 8 + 7) pprivate.
1/2 over the internal randomness of > satisfies (s, 6)-
Proof. Let X, X' be adjacent input and let B be the event under which M is (e, 0)-rivvaee. By assumption, know that TP (B)
y. Let S E O, it follows that
P ((M(X) P(B) ( (MM()) E $ B) + PPB") (M(() E B")
< ((M(X)E +Y <MP(((()) E (S|B)+6+7
e P(M(X) E5)+5+7
P(B
etlog 1 PMM(X) E ES) 76
< t27 (MM(X) S) + (8+7),
where we used that log(1 - 7) > 22 for [0. 1/2].
I.2 Privatizing input using the Gaussian Mechanism
In this section, we will proof the following helpful lemma used in the privacy analysis of our clustering algorithm Algorithm D.5). In summary, it says that when restricted to some set our input has small l sensitivity. we can first add Gaussian noise proportional to this sensitivity and afterwards treat this treat
part of the input "privatized". In particular, for the remainder of the privacy analysis this part as the same on adjacent inputs. Note that we phrase the lemma in terms of matrix inputs since this is what use in application. Of course, also holds for more general inputs.
Note that this influences the exponent in running time sample complexity only by a constant factor hence doesn't violate the assumptions of Theorem D).3 In particular, this randomness is independent of the input
49

---

Lemma 14. Let V, V' E Ra m E [n] and A > 0 be such that there exists a set S of size at least n m. satisfying
E. IV; - V2122 < A2,
where V1, V denote the rows of V, V', respectively. Let A2: RRu - O be an algorithm that is (E2, diffeerreet private in the standard sense, ie,,, for all sets S C O and datasets X, X' E: Baxd differing only in single row it holds that
PP (A2 (X) S) & P (X') S) + 62.
Further, let A1: Raxd + Rrxd be the Gaussian Mechanism with parameters E,,,, 01, Le,, on input axd and outputs M W.
M it samples W N (0.242.
o9
Then for
e' == E1 + m
8' = e me! (m-1m))) + 81
A2 Ai is (s', ( ddifffereeti private with respect and V", ie., for all sets S C O it holds that PP Al)
An) (V) E E) & e ((A2 A1) (V") E ES) 8
Proof. Without loss of generality, assume that S = fl.................. m). Denote by V1, V/2 the first m and last m rows of V respectively. Analogously for V1, Vig We will later partitin the noise W of the Gaussian mechanism in the same way. Further, for a subset A of Raxx and Y E Rmxn define
TAY { x (
X y
E EA
C (((nm))xn
X
Note that
E Af and only if X E TAY.
Let S C O. It now follows that
F FAa.W [(Ab o Ai) E
E A..W
|K
(V
W A5 (5)
V1 + Wi
EA7' [s])] W3
E
E
K
1/2 + W₂)
A2. A2
W2
E
E
V1
+ W1 Tx ((s).ssts]] W2
Ag. W2
[
Vi + W1 T A (51()) }1
E
E
W2 + 61
Aa.
W
W.
{
V + 1/2
W W2
A 45' (5) }} + +
E
K
where the inequality follows by the guarantees of the Gaussian Mechanism. Further, we can bound
E |r A
[
(1%
Vi + Wi
1/2 + W2
E (8)
=
W
A2
V
A2
V + W1 V
W2
ES
w
< mea
mE2
K
A2
V'
+
W1
ES!
W + me (m - 1)e 82
Aa
V/2
W
K
V
+ W1 W2
E A (5)
[s)]]
+ mm (m -nn 1)ez 1)2282,
W
V/2
where the inequality follows by the privacy guarantees of A₂ combined with standard group privacy arguments.
Putting the above two displays together and plugging in the definition of E', 8 we finally obtain
PAL.W [(A2 o A1) (V) E S] < o² P PA (A4 Ar) (V') E $] 5'.
50


Reviews:
[{"review_id": "l89O9znUTj", "reviewer": "Reviewer_GVoS", "summary": "The paper considers the problems of weak and strong recovery for stochastic block models, as well as the problem of estimating the means of Gaussian Mixture Models with sphrerical Gaussians, under the constraint of differential privacy. Their contributions include:\n\n* A computationally efficient algorithm that performs strong recovery for the stochastic block model under $(\\varepsilon, \\delta)$-edge privacy.\n* A computationally inefficient algorithm that performs both weak and strong recovery for the stochastic block model under $\\varepsilon$-edge privacy, as well as a matching information-theoretic lower bound.\n* A computationally efficient algorithm for estimating the means of mixtures of spherical Gaussians under $(\\varepsilon, \\delta)$-DP.\n\nThe algorithms are all derived by applying a general framework that is developed as part of the work which involves privatizing the solutions of strongly convex optimization programs. Additionally, the results given in the paper improve upon the prior work of Seif et. al. and Kamath et. al. on SBMs and GMMs, respectively.", "questions": "There has been recent work by Arbas et. al. on private estimation of GMMs (see https://arxiv.org/abs/2303.04288). I read both works and understand that they go in different directions (meaning that the work of Arbas et. al. doesn't supersede the present paper). However, unless I missed something the paper is not mentioned at all in the related work section (or anywhere else in the submitted pdfs), which feels like an important ommission. Thus, it'd be great if the authors could provide a comparison of their work with that paper (and also include it in their discussion of related work).", "limitations": "This is a purely theoretical work with no obvious societal impact.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "As mentioned in my summary, the paper provides a general framework for privatizing the solutions of convex programs, which they apply in a multitude of settings. I consider this to be a significant contribution in the private statistics literature. In addition to that, the paper is very clearly written and the description of the techniques if quite accessible.", "weaknesses": "No concrete weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback and suggestions.\n\n## Comparison with Arbas-Ashtiani-Liaw (AAL)\nAAL [1] is a subsequent and independent work to ours. Their technique (as well as many previous papers, e.g. [2,3]) is based on the Subsample-And-Aggregate framework, while our approach is based on a different framework which directly privatizes convex optimization programs. We will provide a comparison with AAL [1] in an updated version of our paper.\n\n[1] Arbas, Jamil, Hassan Ashtiani, and Christopher Liaw. \"Polynomial time and private learning of unbounded Gaussian Mixture Models.\" _arXiv preprint arXiv:2303.04288_ (2023).\n\n[2] Cohen, Edith, et al. \"Differentially-private clustering of easy instances.\" _International Conference on Machine Learning_. PMLR, 2021.\n\n[3] Tsfadia, Eliad, et al. \"Friendlycore: Practical differentially private aggregation.\" _International Conference on Machine Learning_. PMLR, 2022."}, {"review_id": "zAyykzdkcR", "reviewer": "Reviewer_NQm5", "summary": "The paper studies private parameter estimation in certain high-dimensional statistical inference tasks. In particular, community detection in stochastic block models (where the privacy is with respect to edge changes) and estimation of unknown means in isotropic Gaussian mixture models.\nAlthough these problems have been studied in the literature, the paper provides new algorithms with improved sample and computational complexities, which closely match that of non-private estimation.\nAlthough seemingly disjoint, the paper's results follow a common methodology of privatizing the existing non-private algorithms that rely on (strongly) convex optimization. \n", "questions": "# Major Comments\n\n1. I suggest adding the notion of privacy (node/sample differential privacy) in the theorem statements.\n\n2. Page, Line 45: \"However, for the more challenging task of graph clustering, node-DP is sometimes impossible to achieve.\"\n    I do not follow. For the SBM, I believe that node-DP is possible to achieve since the information is not concentrated on a single node.\n\n3. Footnote 11: I suggest explicitly adding the notion of privacy in the theorem statement (see the comment above as well).\n\n\n4. Line 259: Where is the first step (i) from Line 253 used in the result here? Does it not follow from (ii) alone? Moreover, what if $K(Y)$ is empty?\n\n5. Line 270, The convex program for SBM. The program from the citation [24] is different from the one displayed here. Especially, it is linear in X and not quadratic. Please clarify.\n\n# Other Comments\n\n1. Page 2, Line 49: What is meant by \"non-robust setting\"\n\n\n\n3. Page 2, Line 64: I believe there are more representative papers for outlier robust estimation.\n\n4. Page 3, Model 1.1: Please give intuition about the parameters in the model, for example, d is the average degree.\n\n5. Theorem 1.3: Please add the weak recovery guarantee in the theorem statement.\n\n6. Line 266: Describe explicitly the relation between the input and Y\n\n3. Lines 182-188: I found the whole paragraph confusing. I suggest rephrasing.\n\n4. Theorem 1.6: Why does $k$ need to be large?\n\n6. Line 210: Put $t^*$ in the runtime or say that it is constant.\n\n7. Line 250-251 Please add citations here \"Our sensitivity bound is simple, yet it generalizes previously known bounds for strongly convex optimization problems\"\n\n8. Line 277: This was confusing and what is the norm $\\|\\cdot\\|_1?$. Please rephrase and explain that the function $f$ does not contain the quadratic in $Y$. Although it does not affect the optimization problem, but it is important for the sensitivity analysis.\n\n9. Line 312: \"While this program suggests a path to recover the first property,...\" Is it even feasible?\n\n10. Line 321: Briefly explain that it is so by setting $z_i$ to zero.\n\n11. Line 341-347: This was confusing for me. What histograms are being referred to here? I understand the space might have been an issue originally but the accepted version can be longer.\n\n\n", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "The paper proposes a generic privatization procedure relying on strongly convex optimization, and then the paper applies it to two problems of interest, leading to improved sample and computational complexities. The paper's results are novel and interesting.\n", "weaknesses": "I have the following comments:\n\n1. The paper's writing needs to be improved and it seems that the submitted version is missing many details. For example, the notation for the indicator function is not displayed due to LaTeX error and many algorithmic environments are referenced as theorems by cleveref.\n\n\n2. The comparison with work of Kothari-Manurangsi-Velingker is still not clear. Perhaps the paper should begin by explaining the similarities and then the differences.\n\nPlease also see the comments below.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback and suggestions.\n\n**comment:**\n> Line 45: \"However, for the more challenging task of graph clustering, node-DP is sometimes impossible to achieve.\" I do not follow. For the SBM, I believe that node-DP is possible to achieve since the information is not concentrated on a single node.\n\n**response:** What we wanted to convey is that *node-DP exact recovery is impossible to achieve*. The proof is as follows. For any graph $G$, we could obtain a neighboring graph $G'$ by picking an arbitrary vertex $v$ and removing all of its adjacent edges. Since vertex $v$ is isolated in $G'$, it is impossible to cluster this vertex correctly. By privacy, it is also impossible to cluster this vertex correctly in the original graph $G$. We will state this claim more accurately in an updated version of our paper.\n\n**comment:**\n> Line 259: Where is the first step (i) from Line 253 used in the result here? Does it not follow from (ii) alone? Moreover, what if $K(Y)$ is empty?\n\n**response:** Both step (i) and (ii) are used to show the inequality on Line 259. There is actually a typo on Line 254: $|f_Y(X)-f_{Y}(Z)|\\leq\\alpha$ should be $|f_Y(X)-f_{Y'}(Z)|\\leq\\alpha$. We will fix it in an updated version of our paper.\nThe proof is in the proof of Lemma B.1 in the supplementary material of our submission. We reproduce it here. Let $Z\\in K(Y)\\cap K(Y')$ be a point such that $|f_Y(Z)-f_{Y'}(Z)|\\leq \\alpha$ by (i) and $|f_{Y}(\\hat{X}) - f_{Y'}(Z)|\\leq \\alpha$ by (ii). Then $f_{Y}(Z) \\leq f_{Y'}(Z)+\\alpha \\leq f_{Y}(\\hat{X})+2\\alpha$, where the first inequality follows from (i) and the second inequality follows from (ii). It is easy to see $f_{Y}(\\hat{X})-f_{Y}(Z) \\leq 0$ by optimality of $\\hat{X}$. Thus $|f_{Y}(\\hat{X})-f_{Y}(Z)| \\leq 2\\alpha$.\nSuppose without loss of generality $f_Y(\\hat{X}) \\leq f_{Y'}(\\hat{X}')$, for a symmetric argument works in the other case. Then\n$f_{Y}(\\hat{X}) \\leq f_{Y'}(\\hat{X}') \\leq f_{Y'}(Z) \\leq f_{Y}(\\hat{X})+\\alpha$,\nwhere the second inequality follows from the optimality of $\\hat{X}'$ and the third inequality follows from (ii). Since $f_{Y'}(\\hat{X}')$ and $f_{Y'}(Z)$ are sandwiched between $f_{Y}(\\hat{X})$ and $f_{Y}(\\hat{X})+\\alpha$, then $|f_{Y'}(\\hat{X}') - f_{Y'}(Z)| \\leq \\alpha$.\nTherefore, $|f_{Y}(\\hat{X})-f_{Y}(Z)| + |f_{Y'}(\\hat{X}') - f_{Y'}(Z)| \\leq 3\\alpha$.\n\n**comment:**\n> Line 270, The convex program for SBM. The program from the citation [24] is different from the one displayed here. Especially, it is linear in $X$ and not quadratic. Please clarify.\n\n**response:** The objective function in the citation [24] (i.e. GV [1]) is indeed linear in $X$, while the objective function of our program is quadratic in $X$. However, both programs have similar utility guarantees and the utility proof of our program is an adaption of that in GV. (See Lemma H.1 in the supplementary material of our submission.) \nThe reason why we used the quadratic objective function (instead of the linear one in GV) is to achieve privacy via strong convexity.\n\n[1] Olivier Guédon and Roman Vershynin. Community detection in sparse networks via Grothendieck’s inequality. Probab. Theory Related Fields, 165(3-4):1025–1049, 2016.\n\n\n**comment:**\n> Line 312: \"While this program suggests a path to recover the first property,...\" Is it even feasible?\n\n**response:** More precisely, the 4-th constraint in $P(Y)$ on line 301 should be $(1-\\alpha)n/k \\leq \\sum_{i}z_{i\\ell} \\leq (1+\\alpha)n/k$ (for some appropriately chosen $\\alpha$) instead of $\\sum_{i}z_{i\\ell}=n/k$. Here we were being a bit informal to make the discussion cleaner and more accessible. We will clarify it in an updated version of our paper.\n\n\n**comment:**\n> Line 341-347: What histograms are being referred to here?\n\n**response:** We consider the histogram in $\\mathbb{R}^d$ where each bin is a $d$-dimensional cube.\nFormally, let $q\\in\\mathbb{R}$ and $\\set{I_i}\\_{i\\in\\mathbb{Z}}$ be a partition of $\\mathbb{R}$ into intervals of length $b$, where $I_i:=\\set{x\\in\\mathbb{R}: q+ (i-1) \\cdot b\\leq x< q + i\\cdot b}$. \nPartition $\\mathbb{R}^d$ into sets $\\set{B_{i_1,\\dots,i_d}}\\_{i_1,\\dots, i_d\\in\\mathbb{Z}}$ where $B_{i_1,\\dots,i_d} := \\set{x\\in\\mathbb{R}^d : \\forall j\\in [d], x_j \\in I_{i_j}}$.\nWe will clarify it in an updated version of our paper.\n\n\n**We will fix all the remaining issues pointed out by the reviewer in an updated version of our paper.**"}, {"review_id": "EjdK128ACw", "reviewer": "Reviewer_SDAx", "summary": "This paper introduces a general framework for designing efficient private estimation algorithms from robust non-private algorithms. Their approach, based on a certain stability analysis for convex optimization, is applied to weak and exact recovery in the stochastic block model (SBM) and for learning mixtures of spherical Gaussians. They obtain the first efficient $(\\varepsilon,\\delta)$-private algorithms for estimation in the first setting (with a new lower bound suggesting that there is little room for improvement), as well as one for the second setting which allows for far lower separation between centers than before.", "questions": "In addition to those above:\n- Are the authors aware of their sensitivity analysis appearing in any previous convex programming literature? It is simple enough that I am a bit surprised it was not previously known. For instance, if their Condition (ii) was replaced with a Hausdorff distance bound between K(Y) and K(Y'), I think this is essentially folklore (though of course Condition (ii) is more sophisticated than this).", "limitations": "The authors were clear about all assumptions and I don't anticipate any negative impacts.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The approach introduced in this paper appears very useful and flexible, and I anticipate future work building off this approach to address a variety of high-dimensional private estimation tasks\n- They make significant progress on two important problems in private estimation\n- Presentation is strong, prior work and guarantees are clearly delineated, their results are clearly stated, and, while there is a bit of technicality in the second application, the authors build up to it nicely with the first SBM results\n- The Gaussian mixture application requires some clever sum-of-squares analysis. More broadly, there is certainly technical depth to this work beyond the general framework", "weaknesses": "- I sometimes lost track of the role of different parameters. I think some may be obvious to those more familiar with this subfield of privacy, and the many special cases mentioned in the discussion were helpful. In any extended version of this work, I would suggest adding more discussion of terms appearing in bounds from the classic (non-private) settings.\n- The choice of objective function in the Gaussian mixture application wasn't clear to me - where does the all ones matrix come from?\n- A fair bit of technical detail was pushed to the supplement, which was a bit tricky to parse. In particular, while the high level approach was clear to me, I was unable to verify correctness for the Gaussian mixture proofs (though I mostly blame this on my unfamiliarity with the SoS program they consider)\n\nMinor nits:\n- On lines 207-208, it was unclear to me why $k \\geq \\sqrt{\\log d}$ should be considered a high-dimensional regime\n- On line 272, the objective function $f$ is not defined (though clear enough from context)\n- \"the basic\" repeated on line 360\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback and suggestions.\n\n**comment:**\n> The choice of objective function in the Gaussian mixture application wasn't clear to me - where does the all ones matrix come from?\n\n**response:** We choose the objective function $||W-J||\\_F^2$ (where $J$ is the $n\\times n$ all-one matrix) for both privacy and utility. Intuitively, $W_{ij}\\in[0,1]$ indicates whether the $i$-th and $j$-th datapoints are sampled from the same Gaussian component.) \nLet $W^*\\in\\set{0,1}^{n\\times n}$ be the ground-truth membership matrix. To recover $W^*$, we at least want $W$ to have (roughly) the same number of ones as $W^*$. One way to achieve this goal is to simply put an upper bound on $|\\sum_{i,j}W_{ij}-W^*_{ij}|$. However, such a constraint would put a lower bound on $\\sum_{i,j}W_{ij}$, which could cause privacy issues. Because there would be two neighboring datasets, such that the constraint set induced by one dataset is satisfiable, but the constraint set induced by the other dataset is not satisfiable. Thus we avoid lower bounding $\\sum_{i,j}W_{ij}$ and instead minimize the distance between $W$ and the all-one matrix $J$. In this way, the constraint set induced by any dataset is satisfiable and the utility is kept.\n\n**comment:**\n> Are the authors aware of their sensitivity analysis appearing in any previous convex programming literature? It is simple enough that I am a bit surprised it was not previously known. For instance, if their Condition (ii) was replaced with a Hausdorff distance bound between $K(Y)$ and $K(Y')$, I think this is essentially folklore (though of course Condition (ii) is more sophisticated than this).\n\n**response:** We are aware of similar sensitivity analysis in the literature (e.g. [1,2,3]) and we believe our sensitivity analysis is not known before. (There is one paragraph comparing our work with previous works in our manuscript, but was removed from our submission due to page limitation.) \nMore specifically, most previous results focus on the special case of unconstrained optimization of strongly convex functions. In contrast, our sensitivity bound applies to the significantly more general settings where both the objective functions and the constrained set may depend on the input.\nWe will add back such a comparison in an updated version of our paper.\n\n[1] Kamalika Chaudhuri, Claire Monteleoni, and Anand D Sarwate, Differentially private empirical risk minimization., Journal of Machine Learning Research 12 (2011), no. 3.\n\n[2] Raef Bassily, Adam Smith, and Abhradeep Thakurta, Private empirical risk minimization: Efficient algorithms and tight error bounds, 2014 IEEE 55th annual symposium on foundations of computer science, IEEE, 2014, pp. 464–473.\n\n[3] Di Wang, Minwei Ye, and Jinhui Xu, Differentially private empirical risk minimization revisited: Faster and more general, Advances in Neural Information Processing Systems 30 (2017).\n\n**comment:**\n> - In any extended version of this work, I would suggest adding more discussion of terms appearing in bounds from the classic (non-private) settings.\n> - A fair bit of technical detail was pushed to the supplement, which was a bit tricky to parse.\n> - On lines 207-208, it was unclear to me why  should be considered a high-dimensional regime\n> - On line 272, the objective function is not defined (though clear enough from context)\n> - \"the basic\" repeated on line 360\n\n**response:** We will fix them in an updated version of our paper. We thank the reviewer for these helpful suggestions!"}, {"review_id": "LI6be37pBL", "reviewer": "Reviewer_cXQY", "summary": "In this paper, the authors investigate graph clustering with the Stochastic Block Model (SBM) and metric clustering with the Gaussian Mixture Model (GMM) within a novel privacy-preserving framework. They propose differentially private (DP) algorithms for both problems, which offer significantly improved performance guarantees compared to state-of-the-art DP algorithms.\n\nSpecifically, the authors present efficient  $(\\epsilon,\\delta)$-differentially private SBM algorithms for both weak recovery and exact recovery. These algorithms operate in polynomial time, outperforming the previous quasi-polynomial time DP algorithms. Additionally, they provide a lower bound that demonstrates the optimality of the privacy guarantees from an information-theoretic perspective.\n\nFurthermore, the authors develop private algorithms for learning the mixture of $k$ spherical Gaussians under the condition that the minimum separation is at least $O(k^{1/t}\\sqrt{t})$. Notably, these algorithms do not require an additional additive term of $\\Omega(\\sqrt{\\log n})$ in the minimum separation or an explicit upper bound on the Euclidean norm of the centers, which were assumed in prior works.\n\nThe proposed algorithms are built upon a simple yet powerful connection between robustness and privacy. The authors formalize this connection as a framework for transforming robust estimation algorithms into private algorithms. They show that if two strongly convex functions over constrained sets, where both the function and the set may depend on the input, are point-wise close, their minimizers are also close. Conversely, they establish that projections of points that are close to each other onto convex sets that are point-wise close must also be close.", "questions": "No. ", "limitations": "It might be useful to briefly discuss why the current framework does not achieve $(\\log k)^{c+1/2}$ separation in polynomial time. ", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper is well-written and addresses fundamental clustering problems, including SBM and GMM, providing novel differentially private algorithms that make remarkable advancements in the field. The lower bound also demonstrates the optimality of the privacy guarantees of the SBM algorithm.\n\nMoreover, their techniques of translating robust estimation algorithms to DP algorithms seem readily extended to some other estimation algorithms. Though the ideas are intuitively simple, the formal connections and applications to clustering algorithms are highly nontrivial. ", "weaknesses": "There is no experiment in the paper (but it is a strong theory paper). ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback and suggestions.\n\n**comment:** \n> It might be useful to briefly discuss why the current framework does not achieve $\\log(k)^{1/2 + c}$ separation in polynomial time.\n\n**response:** On a high level, our framework converts non-private algorithms that are based on convex optimization into private ones. Currently, the only known (non-private) algorithm that can achieve separation $\\log(k)^{1/2 + c}$ in polynomial time is not based on such convex optimization techniques (but rather intricate moment calculations). We believe it is likely, that there also exists an algorithm based on convex optimization that (non-privately) matches these guarantees. Given such an algorithm, we could likely use this in our framework and achieve the same guarantees privately."}]

Author rebuttal:

Meta-review: This work develops sample efficient and (in some cases) computationally efficient algorithms for privately learning a number of probabilistic models, specifically mixtures of spherical Gaussians and stochastic block models. The reviewers agreed that the results and techniques are novel and interesting.  Overall, the paper is a solid contribution to this field.
Decision: Accept (spotlight)