SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hongjie Chen ETH Zürich
Vincent Cohen-Addad  
Google Research
Tommaso d'Orsi Bocconi*
Alessandro Epasto  
Google Research
Jacob Imola  
UC San Diego
David Steurer ETH Zürich
Stefan Tiegel ETH Zürich
We introduce general tools for designing efficient private estimation algorithms, in the high-dimensional settings, whose statistical guarantees almost match those of the best known non-private algorithms. To illustrate our techniques, we consider two problems: recovery of stochastic block models and learning mixtures of spherical Gaussians.
For the former, we present the first efficient  $(\varepsilon, \delta)$ -differentially private algorithms for both weak recovery and exact recovery. Previously known algorithms achieving comparable guarantees required quasi-polynomial time. We complement these results with an information-theoretic lower bound that highlights how the guarantees of our algorithms are almost tight.
For the latter, we design an  $(\varepsilon, \delta)$ -differentially private algorithm that recovers the centers of the  $k$ -mixture when the minimum separation is at least  $O(k^{1/t}\sqrt{t})$ . For all choices of  $t$ , this algorithm requires sample complexity  $n \geqslant k^{O(1)}d^{O(t)}$  and time complexity  $(nd)^{O(t)}$ . Prior work required either an additional additive  $\Omega(\sqrt{\log n})$  term in the minimum separation or an explicit upper bound on the Euclidean norm of the centers.
Computing a model that best matches a dataset is a fundamental question in machine learning and statistics. Given a set of  $n$  samples from a model, how to find the most likely parameters of the model that could have generated this data? This basic question has been widely studied for several decades, and recently revisited in the context where the input data has been partially corrupted (i.e., where few samples of the data have been adversarially generated—see for instance [37, 18, 22, 20]). This has led to several recent works shedding new lights on classic model estimation problems, such as the Stochastic Block Model (SBM) [28, 46, 44, 25, 21, 40] and the Gaussian Mixture Model (GMM) [30, 36, 9, 10] (see Definitions 1.1 and 1.2).
Privacy in machine learning and statistical tasks has recently become of critical importance. New regulations, renewed consumer interest as well as privacy leaks, have led the major actors to adopt privacy-preserving solutions for the machine learning [1, 2, 3]. This new push has resulted in a flurry of activity in algorithm design for private machine learning, including very recently for SBMs and GMMs [55, 32, 16, 60]. Despite this activity, it has remain an open challenge to fully understand how privacy requirements impact model estimation problems and in particular their recovery thresholds and the computational complexity. This is the problem we tackle in this paper.
*Much of this work was done while the author was at ETH Zürich.
While other notions of privacy exist (e.g.  $k$ -anonymity), the de facto privacy standard is the differential privacy (DP) framework of Dwork, McSherry, Nissim, and Smith [24]. In this framework, the privacy quality is governed by two parameters,  $\varepsilon$  and  $\delta$ , which in essence tell us how the probability of seeing a given output changes (both multiplicatively and additively) between two datasets that differ by any individual data element. This notion, in essence, quantifies the amount of information leaked by a given algorithm on individual data elements. The goal of the algorithm designer is to come up with differentially private algorithms for  $\varepsilon$  being a small constant and  $\delta$  being of order  $1 / n^{\Theta(1)}$ .
Differentially private analysis of graphs usually considers two notions of neighboring graphs. The weaker notion of edge-  $DP$  defines two graphs to be neighboring if they differ in one edge. Under the stronger notion of node-  $DP$ , two neighboring graphs can differ arbitrarily in the set of edges connected to a single vertex. Recently, there is a line of work on node-  $DP$  parameter estimation in random graph models, e.g. Erdős-Rényi models [61] and Graphons [12, 13]. However, for the more challenging task of graph clustering, node-  $DP$  is sometimes impossible to achieve. Thus it is a natural first step to study edge-  $DP$  graph clustering.
Very recently, Seif, Nguyen, Vullikanti, and Tandon [55] were the first to propose differentially private algorithms for the Stochastic Block Model, with edge-DP. Concretely, they propose algorithms achieving exact recovery (exact identification of the planted clustering) while preserving privacy of individual edges of the graph. The proposed approach either takes  $n^{\Theta (\log n)}$  time when  $\varepsilon$  is constant, or runs in polynomial time when  $\varepsilon$  is  $\Omega (\log n)$ .
Gaussian Mixture Models have also been studied in the context of differential privacy by [32, 16, 60] using the subsample-and-aggregate framework first introduced in [52] (see also recent work for robust moment estimation in the differential privacy setting [35, 8, 29]). The works of [32, 16] require an explicit bound on the euclidean norm of the centers as the sample complexity of these algorithms depends on this bound. For a mixture of  $k$  Gaussians, if there is a non-private algorithm that requires the minimum distance between the centers to be at least  $\Delta$ , then [16, 60] can transform this non-private algorithm into a private one that needs the minimum distance between the centers to be at least  $\Delta + \sqrt{\log n}$ , where  $n$  is the number of samples.
In this paper, we tackle both clustering problems (graph clustering with the SBM and metric clustering with the GMM) through a new general privacy-preserving framework that brings us significantly closer to the state-of-the-art of non-private algorithms. As we will see, our new perspective on the problems appear to be easily extendable to many other estimation algorithms.
From robustness to privacy In recent years a large body of work (see [19, 22, 40, 18, 37, 14, 36, 30] and references therein) has advanced our understanding of parameter estimation in the presence of adversarial perturbations. In these settings, an adversary looks at the input instance and modifies it arbitrarily, under some constraints (these constraints are usually meant to ensure that it is still information theoretically possible to recover the underlying structure). As observed in the past [23, 35, 41], the two goals of designing privacy-preserving machine learning models and robust model estimation are tightly related. The common objective is to design algorithms that extract global information without over-relaying on individual data samples.
Concretely, robust parameter estimation tends to morally follow a two-steps process:  $(i)$  argue that typical inputs are well-behaved, in the sense that they satisfy some property which can be used to accurately infer the desired global information,  $(ii)$  show that adversarial perturbations cannot significantly alter the quality of well-behaved inputs, so that it is still possible to obtain an accurate estimate. Conceptually, the analysis of private estimation algorithms can also be divided in two parts: utility, which is concerned with the accuracy of the output, and privacy, which ensures there is no leak of sensitive information. In particular, the canonical differential privacy definition can be interpreted as the requirement that, for any distinct inputs  $Y, Y'$ , the change in the output is proportional to the distance $^3$  between  $Y$  and  $Y'$ .
It is easy to see this as a generalization of robustness: while robust algorithm needs the output to be stable for typical inputs, private algorithms require this stability for any possible input. Then,
In particular, we cannot hope to achieve exact recovery. We could isolate a vertex by removing all of its adjacent edges. Then it is impossible to cluster this vertex correctly. See below for formal definitions.
stability of the output immediately implies adding a small amount of noise to the output yields privacy. If the added noise is small enough, then utility is also preserved.
Our work further tightens this connection between robustness and privacy through a simple yet crucial insight: if two strongly convex functions over constrained sets –where both the function and the set may depend on the input– are point-wise close (say in a  $\ell_2$ -sense), their minimizers are also close (in the same sense). The alternative perspective is that projections of points that are close to each other, onto convex sets that are point-wise close, must also be close. This observation subsumes previously known sensitivity bounds in the empirical risk minimization literature (in particular in the output-perturbation approach to ERM, see Section 2 for a comparison).
The result is a clean, user-friendly, framework to turn robust estimation algorithms into private algorithms, while keeping virtually the same guarantees. We apply this paradigm to stochastic block models and Gaussian mixture models, which we introduce next.
Stochastic block model The stochastic block model is an extensively studied statistical model for community detection in graphs (see [4] for a survey).
Model 1.1 (Stochastic block model). In its most basic form, the stochastic block model describes the distribution $^4$  of an  $n$ -vertex graph  $\mathbf{G} \sim \mathsf{SBM}_n(d, \gamma, x)$ , where  $x$  is a vector of  $n$  binary $^5$  labels,  $d \in \mathbb{N}$ ,  $\gamma > 0$ , and for every pair of distinct vertices  $i, j \in [n]$  the edge  $\{i, j\}$  is independently added to the graph  $\mathbf{G}$  with probability  $(1 + \gamma \cdot x_i \cdot x_j) \frac{d}{n}$ .
For balanced label vector  $x$ , i.e., with roughly the same number of  $+1$ 's and  $-1$ 's, parameter  $d$  roughly corresponds to the average degree of the graph. Parameter  $\gamma$  corresponds to the bias introduced by the community structure. Note that for distinct vertices  $i, j \in [n]$ , the edge  $\{i, j\}$  is present in  $\mathbf{G}$  with probability  $(1 + \gamma) \frac{d}{n}$  if the vertices have the same label  $x_i = x_j$  and with probability  $(1 - \gamma) \frac{d}{n}$  if the vertices have different labels  $x_i \neq x_j$ .<sup>6</sup>
Given a graph  $\mathbf{G}$  sampled according to this model, the goal is to recover the (unknown) underlying vector of labels as well as possible. In particular, for a chosen algorithm returning a partition  $\hat{x}(\mathbf{G}) \in \{\pm 1\}^n$ , there are two main objective of interest: weak recovery and exact recovery. The former amounts to finding a partition  $\hat{x}(\mathbf{G})$  correlated with the true partition. The latter instead corresponds to actually recovering the true partition with high probability. As shown in the following table, by now the statistical and computational landscape of these problems is well understood [17, 42, 48, 49, 28]:
Objective	can be achieved (and effi-ciently so) iff
weak recovery	PG~SBMn(d,γ,x) (1/n|{x,hat(G)}| ≥ Ωd,γ(1)) ≥ 1 - o(1)	γ2·d≥1
exact recovery	PG~SBMn(d,γ,x) (hat(G) ∈ {x,-x}) ≥ 1 - o(1)	d/√n (1 - √1-γ2) ≥ 1
Learning mixtures of spherical Gaussians The Gaussian Mixture Model we consider is the following.
Model 1.2 (Mixtures of spherical Gaussians). Let  $D_{1}, \ldots, D_{k}$  be Gaussian distributions on  $\mathbb{R}^{d}$  with covariance Id and means  $\mu_{1}, \ldots, \mu_{k}$  satisfying  $\| \mu_{i} - \mu_{j} \| \geqslant \Delta$  for any  $i \neq j$ . Given a set  $\mathbf{Y} = \{\mathbf{y}_{1}, \ldots, \mathbf{y}_{n}\}$  of  $n$  samples from the uniform mixture over  $D_{1}, \ldots, D_{k}$ , estimate  $\mu_{1}, \ldots, \mu_{k}$ .
It is known that when the minimum separation is  $\Delta = o(\sqrt{\log k})$ , superpolynomially many samples are required to estimate the means up to small constant error [54]. Just above this threshold, at separation  $k^{O(1 / \gamma)}$  for any constant  $\gamma$ , there exist efficient algorithms based on the sum-of-squares hierarchy recovering the means up to accuracy  $1 / \mathrm{poly}(k)$  [30, 36, 59]. In the regime
4We use bold characters to denote random variables.
5More general versions of the stochastic block model allow for more than two labels and general edge probabilities depending on the label assignment. However, many of the algorithmic phenomena of the general version can in their essence already be observed for the basic version that we consider in this work.
At times we may write  $d_{n}, \gamma_{n}$  to emphasize that these may be functions of  $n$ . We write  $o(1), \omega(1)$  for functions tending to zero (resp. infinity) as  $n$  grows.
where  $\Delta = O(\sqrt{\log k})$  these algorithms yield the same guarantees but require quasipolynomial time. Recently, [39] showed how to efficiently recover the means as long as  $\Delta = O(\log (k)^{1 / 2 + c})$  for any constant  $c > 0$ .
Stochastic block model We present here the first  $(\varepsilon, \delta)$ -differentially private efficient algorithms for exact recovery. In all our results on stochastic block models, we consider the edge privacy model, in which two input graphs are adjacent if they differ on a single edge (cf. Definition C.1).
Theorem 1.3 (Private exact recovery of SBM). Let  $x \in \{\pm 1\}^n$  be balanced<sup>7</sup>. For any  $\gamma, d, \varepsilon, \delta > 0$  satisfying
$$
\frac {d}{\log n} \left(1 - \sqrt {1 - \gamma^ {2}}\right) \geqslant \Omega (1) \quad a n d \quad \frac {\gamma d}{\log n} \geqslant \Omega \left(\frac {1}{\varepsilon^ {2}} \cdot \frac {\log (1 / \delta)}{\log n} + \frac {1}{\varepsilon}\right),
$$
there exists an  $(\varepsilon, \delta)$ -differentially edge private algorithm that, on input  $\mathbf{G} \sim \mathrm{SBM}_n(d, \gamma, x)$ , returns  $\hat{x}(\mathbf{G}) \in \{x, -x\}$  with probability  $1 - o(1)$ . Moreover, the algorithm runs in polynomial time.
For any constant  $\varepsilon > 0$ , Theorem 1.3 states that  $(\varepsilon, \delta)$ -differentially private exact recovery is possible, in polynomial time, already a constant factor close to the non-private threshold. Previous results [55] could only achieve comparable guarantees in time  $O(n^{O(\log n)})$ . It is also important to observe that the theorem provides a trade-off between signal-to-noise ratio of the instance (captured by the expression on the left-hand side with  $\gamma, d$ ) and the privacy parameter  $\varepsilon$ . In particular, we highlight two regimes: for  $d \geqslant \Omega(\log n)$  one can achieve exact recovery with high probability and privacy parameters  $\delta = n^{-\Omega(1)}, \varepsilon = O(1/\gamma + 1/\gamma^2)$ . For  $d \geqslant \omega(\log n)$  one can achieve exact recovery with high probability and privacy parameters  $\varepsilon = o(1), \delta = n^{-\omega(1)}$ . Theorem 1.3 follows by a result for private weak recovery and a boosting argument (cf. Theorem C.3 and Appendix C.2).
Further, we present a second, exponential-time, algorithm based on the exponential mechanism [43] which improves over the above in two regards. First, it gives pure differential privacy. Second, it provides utility guarantees for a larger range of graph parameters. In fact, we will also prove a lower bound which shows that its privacy guarantees are information theoretically optimal. All hidden constants are absolute and do not depend on any graph or privacy parameters unless stated otherwise. In what follows we denote by  $\operatorname{err}(\hat{x}, x)$  the minimum of the hamming distance of  $\hat{x}$  and  $x$ , and the one of  $-\hat{x}$  and  $x$ , divided by  $n$ .
Theorem 1.4 (Slightly informal, see Theorem C.18 in the supplements for full version). Let  $\gamma \sqrt{d} \geqslant \Omega(1)$ ,  $x \in \{\pm 1\}^n$  be balanced, and  $\zeta \geqslant \exp(-\Omega(\gamma^2 d))$ . For any  $\varepsilon \geqslant \Omega\left(\frac{\log(1 / \zeta)}{\gamma^d}\right)$ , there exists an algorithm which on input  $\mathbf{G} \sim \mathsf{SBM}_n(\gamma, d, x)$  outputs an estimate  $\hat{x}(\mathbf{G}) \in \{\pm 1\}^n$  satisfying err  $(\hat{x}(\mathbf{G}), x) \leqslant \zeta$  with probability at least  $1 - \zeta$ . In addition, the algorithm is  $\varepsilon$ -differentially edge private. Further, we can achieve error  $\Theta\left(1 / \sqrt{\log(1 / \zeta)}\right)$  with probability  $1 - e^{-n}$ .
A couple of remarks are in order. First, our algorithm works across all degree-regimes in the literature and matches known non-private thresholds and rates up to constants.<sup>9</sup> In particular, for  $\gamma^2 d = \Theta(1)$ , we achieve weak/partial recovery with either constant or exponentially high success probability. Recall that the optimal non-private threshold is  $\gamma^2 d > 1$ . For the regime, where  $\gamma^2 d = \omega(1)$ , it is known that the optimal error rate is  $\exp(-(1 - o(1))\gamma^2 d)$  [64] even non-privately which we match up to constants - here  $o(1)$  denotes a function that tends to zero as  $\gamma^2 d$  tends to infinity. Moreover, our algorithm achieves exact recovery as soon as  $\gamma^2 d = \Omega(\log n)$  since then  $\zeta < \frac{1}{n}$ . This also matches known non-private thresholds up to constants [5, 47]. We remark that [55] gave an  $\varepsilon$ -DP exponential time algorithm which achieved exact recovery and has inverse polynomial success probability in the utility case as long as  $\varepsilon \geqslant \Omega(\frac{\log n}{\gamma d})$ . We recover this result as a special case (with slightly worse constants). In fact, their algorithm is also based on the exponential mechanism, but their analysis only applies to the setting of exact recovery, while our result holds much more generally. Another
7A vector  $x\in \{\pm 1\} ^n$  is said to be balanced if  $\sum_{i = 1}^{n}x_{i} = 0$
It is optimal in the "small error" regime, otherwise it is almost optimal. See Theorem C.22 for more detail.
For ease of exposition we did not try to optimize these constants.
crucial difference is that we show how to privatize a known boosting technique frequently used in the non-private setting, allowing us to achieve error guarantees which are optimal up to constant factors.
It is natural to ask whether, for a given set of parameters  $\gamma, d, \zeta$  one can obtain better privacy guarantees than Theorem 1.4. Our next result implies that our algorithmic guarantees are almost tight.
Theorem 1.5 (Informal, see Theorem C.22 in the supplements for full version). Suppose there exists an  $\varepsilon$ -differentially edge private algorithm such that for any balanced  $x \in \{\pm 1\}^n$ , on input  $\mathbf{G} \sim \mathrm{SBM}_n(d, \gamma, x)$ , outputs  $\hat{x}(\mathbf{G}) \in \{\pm 1\}^n$  satisfying
$$
\mathbb {P} \left(\operatorname {e r r} (\hat {x} (\mathbf {G}), x) <   \zeta) \geqslant 1 - \eta . \right.
$$
$$
\varepsilon \geqslant \Omega \left(\frac {\log (1 / \zeta)}{\gamma d} + \frac {\log (1 / \eta)}{\zeta n \gamma d}\right). \tag {1.1}
$$
This lower bound is tight for  $\varepsilon$ -DP exact recovery. By setting  $\zeta = 1/n$  and  $\eta = 1/poly(n)$ , Theorem 1.5 implies no  $\varepsilon$ -DP exact recovery algorithm exists for  $\varepsilon \leqslant O\left(\frac{\log n}{\gamma d}\right)$ . There exist  $\varepsilon$ -DP algorithms (Algorithm C.19 in the supplements and the algorithm in [55]) exactly recover the community for any  $\varepsilon \geqslant \Omega\left(\frac{\log n}{\gamma d}\right)$ .
Notice Theorem 1.5 is a lower bound for a large range of error rates (partial to exact recovery). For failure probability  $\eta = \zeta$ , the lower bound simplifies to  $\varepsilon \geqslant \Omega \left( \frac{\log(1 / \zeta)}{\gamma d} \right)$  and hence matches Theorem 1.4 up to constants. For exponentially small failure probability,  $\eta = e^{-n}$ , it becomes  $\varepsilon \geqslant \Omega \left( \frac{1}{\zeta \gamma d} \right)$ . To compare, Theorem 1.4 requires  $\varepsilon \geqslant \Omega \left( \frac{1}{\zeta^2 \gamma d} \right)$  in this regime, using the substitution  $\sqrt{\log(1 / \zeta)} \to \zeta$ .
Further, while formally incomparable, this  $\varepsilon$ -DP lower bound also suggests that the guarantees obtained by our efficient  $(\varepsilon, \delta)$ -DP algorithm in Theorem 1.3 might be close to optimal. Note that setting  $\zeta = \frac{1}{n}$  in Theorem 1.5 requires the algorithm to exactly recover the partitioning. In this setting, Theorem 1.3 implies that there is an efficient  $(\varepsilon, n^{-\Theta(1)})$ -DP exact recovery algorithm for  $\varepsilon \leqslant O\left(\sqrt{\frac{\log n}{\gamma d}}\right)$ . Theorem 1.5 states any  $\varepsilon$ -DP exact recovery algorithm requires  $\varepsilon \geqslant \Omega\left(\frac{\log n}{\gamma d}\right)$ . Further, for standard privacy parameters that are required for real-world applications, such as  $\varepsilon \approx 1$  and  $\delta = n^{-10}$ , Theorem 1.3 requires that  $\gamma d \geqslant \Omega(\log n)$ . Theorem 1.5 shows that for pure-DP algorithms with the same setting of  $\varepsilon$  this is also necessary. We leave it as fascinating open questions to bridge the gap between upper and lower bounds in the context of  $(\varepsilon, \delta)$ -DP.
Learning mixtures of spherical Gaussians Our algorithm for privately learning mixtures of  $k$  spherical Gaussians provides statistical guarantees matching those of the best known non-private algorithms.
Theorem 1.6 (Privately learning mixtures of spherical Gaussians). Consider an instance of Model 1.2. Let  $t > 0$  be such that  $\Delta \geqslant O\left(\sqrt{tk^{1/t}}\right)$ . For  $n \geqslant \Omega\left(k^{O(1)} \cdot d^{O(t)}\right), k \geqslant (\log n)^{1/5}$ , there exists an algorithm, running in time  $(nd)^{O(t)}$ , that outputs vectors  $\hat{\mu}_1, \ldots, \hat{\mu}_k$  satisfying
$$
\max _ {\ell \in [ k ]} \left\| \hat {\boldsymbol {\mu}} _ {\ell} - \mu_ {\pi (\ell)} \right\| _ {2} \leqslant O (k ^ {- 1 2}),
$$
with high probability, for some permutation  $\pi : [k] \to [k]$ . Moreover, for  $\varepsilon \geqslant k^{-10}$ ,  $\delta \geqslant n^{-10}$ , the algorithm is  $(\varepsilon, \delta)$ -differentially private<sup>10</sup> for any input  $Y$ .
The conditions  $\varepsilon \geqslant k^{-10}$ ,  $\delta \geqslant n^{-10}$  in Theorem 1.6 are not restrictive and should be considered a formality. Moreover, setting  $\varepsilon = 0.01$  and  $\delta = n^{-10}$  already provides meaningful privacy guarantees in practice. The condition that  $k \geqslant (\log n)^{1/5}$  is a technical requirement by our proofs.
Prior to this work, known differentially private algorithms could learn a mixture of  $k$ -spherical Gaussian either if: (1) they were given a ball of radius  $R$  containing all centers [32, 16]; or (2) the minimum separation between centers needs an additional additive  $\Omega (\sqrt{\log n})$  term  $[16, 60]^{12}$ .
$^{10}$ Two input datasets are adjacent if they differ on a single sample. See Definition D.1 in the supplements.
In [32, 16] the sample complexity of the algorithm depends on this radius  $R$ .
12 For  $k \leqslant n^{o(1)}$  our algorithm provides a significant improvement as  $\sqrt{\log k} = o(\sqrt{\log n})$ .
To the best of our knowledge, Theorem 1.6 is the first to get the best of both worlds. That is, our algorithm requires no explicit upper bounds on the means (this also means the sample complexity does not depend on  $R$ ) and only minimal separation assumption  $O(\sqrt{\log k})$ . Furthermore, we remark that while previous results only focused on mixtures of Gaussians, our algorithm also works for the significantly more general class of mixtures of Poincaré distributions. Concretely, in the regime  $k \geqslant \sqrt{\log d}$ , our algorithm recovers the state-of-the-art guarantees provided by non-private algorithms which are based on the sum-of-squares hierarchy [36, 30, 59]:<sup>13</sup>