SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- If  $\Delta \geqslant k^{1 / t^*}$  for some constant  $t^*$ , then by choosing  $t \geqslant \Omega(t^*)$  the algorithm recovers the centers, up to a  $1/\mathrm{poly}(k)$  error, in time  $\mathrm{poly}(k,d)$  and using only  $\mathrm{poly}(k,d)$  samples.
- If  $\Delta \geqslant \Omega (\sqrt{\log k})$  then choosing  $t = O(\log k)$  the algorithm recovers the centers, up to a  $1 / \mathrm{poly}(k)$  error, in quasi-polynomial time  $\mathrm{poly}(k^{O(t)},d^{O(t^2)})$  and using a quasi-polynomial number of samples  $\mathrm{poly}(k,d^{O(t)})$ .
For simplicity of exposition we will limit the presentation to mixtures of spherical Gaussians. We reiterate that separation  $\Omega (\sqrt{\log k})$  is information-theoretically necessary for algorithms with polynomial sample complexity [54].
Subsequently and independently of our work, the work of [6] gives an algorithm that turns any non-private GMM learner into a private one based on the subsample and aggregate framework. They apply this reduction to the classical result of [45] to give the first finite-sample  $(\varepsilon, \delta)$ -DP algorithm that learns mixtures of unbounded Gaussians, in particular, the covariance matrices of their mixture components can be arbitrary.
2 Techniques
We present here our general tools for designing efficient private estimation algorithms in the high-dimensional setting whose statistical guarantees almost match those of the best know non-private algorithms. The algorithms we design have the following structure in common: First, we solve a convex optimization problem with constraints and objective function depending on our input  $Y$ . Second, we round the optimal solution computed in the first step to a solution  $X$  for the statistical estimation problem at hand.
We organize our privacy analyses according to this structure. In order to analyze the first step, we prove a simple sensitivity bound for strongly convex optimization problems, which bounds the  $\ell_2$ -sensitivity of the optimal solution in terms of a uniform sensitivity bound for the objective function and the feasible region of the optimization problem.
For bounded problems - such as recovery of stochastic block models - we use this sensitivity bound, in the second step, to show that introducing small additive noise to standard rounding algorithms is enough to achieve privacy.
For unbounded problems – such as learning GMMs – we use this sensitivity bound to show that on adjacent inputs, either most entries of  $X$  only change slightly, as in the bounded case, or few entries vary significantly. We then combine different privacy techniques to hide both type of changes.
Privacy from sensitivity of strongly convex optimization problems Before illustrating our techniques with some examples, it is instructive to explicit our framework. Here we have a set of inputs  $\mathcal{V}$  and a family of strongly convex functions  $\mathcal{F}(\mathcal{V})$  and convex sets  $\kappa (\mathcal{V})$  parametrized by these inputs. The generic non-private algorithm based on convex optimization we consider works as follows:
1. Compute  $\hat{X} \coloneqq \operatorname{argmin}_{X \in \mathcal{K}(Y)} f_Y(X)$ ;
2. Round  $\hat{X}$  into an integral solution.
For an estimation problem, a distributional assumption on  $\mathcal{Y}$  is made. Then one shows how, for typical inputs  $\mathbf{Y}$  sampled according to that distribution, the above scheme recovers the desired structured information.
<sup>13</sup>We remark that [39] give a polynomial time algorithm for separation  $\Omega (\log (k)^{1 / 2 + c})$  for constant  $c > 0$  in the non-private setting but for a less general class of mixture distributions.
We can provide a privatized version of this scheme by arguing that, under reasonable assumptions on  $\mathcal{F}(\mathcal{Y})$  and  $\mathcal{K}(\mathcal{Y})$ , the output of the function  $\mathrm{argmin}_{X \in \mathcal{K}(Y)} f_Y(X)$  has low  $\ell_2$ -sensitivity. The consequence of this crucial observation is that one can combine the rounding step 2 with some standard privacy mechanism and achieve differential privacy. That is, the second step becomes:
2. Add random noise  $\mathbf{N}$  and round  $\hat{X} +\mathbf{N}$  into an integral solution.
Our sensitivity bound is simple, yet it generalizes previously known bounds for strongly convex optimization problems (we provide a detailed comparison later in the section). For adjacent  $Y, Y' \in \mathcal{V}$ , it requires the following properties:
(i) For each  $X\in \mathcal{K}(Y)\cap \mathcal{K}(Y^{\prime})$  it holds  $|f_{Y}(X) - f_{Y^{\prime}}(X)|\leqslant \alpha$
(ii) For each  $X \in \mathcal{K}(Y)$  its projection  $Z$  onto  $\mathcal{K}(Y) \cap \mathcal{K}(Y')$  satisfies  $|f_{Y}(X) - f_{Y'}(Z)| \leqslant \alpha$ .
Here we think of  $\alpha$  as some small quantity (relatively to the problem parameters). Notice, we may think of  $(i)$  as Lipschitz-continuity of the function  $g(Y,X) = f_{Y}(X)$  with respect to  $Y$  and of  $(ii)$  as a bound on the change of the constrained set on adjacent inputs. In fact, these assumptions are enough to conclude low  $\ell_2$  sensitivity. Let  $\hat{X}$  and  $\hat{X}^{\prime}$  be the outputs of the first step on inputs  $Y,Y^{\prime}$ . Then using (i) and (ii) above and the fact that  $\hat{X}$  is an optimizer, we can show that there exists  $Z\in \mathcal{K}(Y)\cap \mathcal{K}(Y^{\prime})$  such that
$$
\left| f _ {Y} (\hat {X}) - f _ {Y} (Z) \right| + \left| f _ {Y ^ {\prime}} \left(\hat {X} ^ {\prime}\right) - f _ {Y ^ {\prime}} (Z) \right| \leqslant O (\alpha).
$$
By  $\kappa$ -strong convexity of  $f_{Y}, f_{Y'}$  this implies
$$
\left\| \hat {X} - Z \right\| _ {2} ^ {2} + \left\| \hat {X} ^ {\prime} - Z \right\| _ {2} ^ {2} \leqslant O (\alpha / \kappa)
$$
which ultimately means  $\| \hat{X} - \hat{X}' \|_2^2 \leqslant O(\alpha / \kappa)$  (see Lemma B.1). Thus, starting from our assumptions on the point-wise distance of  $f_{Y'}$ ,  $f_{Y'}$  we were able to conclude low  $\ell_2$ -sensitivity of our output!
A simple application: weak recovery of stochastic block models The ideas introduced above, combined with existing algorithms for weak recovery of stochastic block models, immediately imply a private algorithm for the problem. To illustrate this, consider Model 1.1 with parameters  $\gamma^2 d\geqslant C$  for some large enough constant  $C > 1$ . Let  $x\in \{\pm 1\} ^n$  be balanced. Here  $Y$  is an  $n$ -by- $n$  matrix corresponding to the rescaled centered adjacency matrix of the input graph:
$$
Y _ {i j} = \left\{ \begin{array}{l l} \frac {1}{\gamma d} \left(1 - \frac {d}{n}\right) & \text {i f} i j \in E (G) \\ - \frac {1}{\gamma n} & \text {o t h e r w i s e .} \end{array} \right.
$$
The basic semidefinite program [28, 46] can be recast<sup>14</sup> as the strongly convex constrained optimization question of finding the orthogonal projection of the matrix  $Y$  onto the set  $\mathcal{K} := \{X \in \mathbb{R}^{n \times n} \mid X \succeq \mathbf{0}, X_{ii} = \frac{1}{n} \forall i\}$ . That is
$$
\hat {X} := \operatorname * {a r g m i n} _ {X \in \mathcal {K}} \| Y - X \| _ {\mathrm {F}} ^ {2}.
$$
Let  $f_{Y}(X) \coloneqq \| X\|_{\mathrm{F}}^{2} - 2\langle X,Y\rangle$  and notice that  $\hat{X} = \operatorname*{argmin}_{X\in \mathcal{K}}f_{Y}(X)$ . It is a standard fact that, if our input was  $\mathbf{G} \sim \mathsf{SBM}_n(d,\gamma ,x)$ , then with high probability  $X(\mathbf{G}) = \operatorname*{argmin}_{X\in \mathcal{K}}f_{Y(\mathbf{G})}(X)$  would have leading eigenvalue-eigenvector pair satisfying
$$
\lambda_ {1} (\mathbf {G}) \geqslant 1 - O (1 / \gamma^ {2} d) \quad \text {a n d} \quad \langle v _ {1} (\mathbf {G}), x / \| x \| \rangle^ {2} \geqslant 1 - O \left(1 / \gamma^ {2} d\right).
$$
This problem fits perfectly the description of the previous paragraph. Note that since the constraint set does not depend on  $Y$ , Property (ii) reduces to Property (i). Thus, it stands to reason that the projections  $\hat{X}, \hat{X}'$  of  $Y, Y'$  are close whenever the input graphs generating  $Y$  and  $Y'$  are adjacent. By Hölder's Inequality with the entry-wise infinity and  $\ell_1$ -norm, we obtain  $|f_{Y}(X) - f_{Y'}(X)| \leqslant 2\| X\|_{\infty}\| Y - Y'\|_{1}$ . By standard facts about positive semidefinite matrices, we have  $\| X\|_{\infty} \leqslant \frac{1}{n}$
14The objective function in [28, 46] is linear in  $X$  instead of quadratic. However, both programs have similar utility guarantees and the utility proof of our program is an adaption of that in [28] (see Lemma C.8). We use the quadratic objective function to achieve privacy via strong convexity.
for all  $X \in \mathcal{K}$ . Also,  $Y$  and  $Y'$  can differ on at most 2 entries and hence  $\| Y - Y' \|_1 \leqslant O\left(\frac{1}{\gamma d}\right)$ . Thus,  $\|\hat{X} - \hat{X}'\|_{\mathrm{F}}^2 \leqslant O\left(\frac{1}{n\gamma d}\right)$ .
The rounding step is now straightforward. Using the Gaussian mechanism we return the leading eigenvector of  $\hat{X} +\mathbf{N}$  where  $\mathbf{N}\sim N(0,\frac{1}{n\gamma d}\cdot \frac{\log(1 / \delta)}{\varepsilon^2})^{n\times n}$ . This matrix has Frobenius norm significantly larger than  $\hat{X}$  but its spectral norm is only
$$
\| \mathbf {N} \| \leqslant \frac {\sqrt {n \log (1 / \delta)}}{\varepsilon} \cdot \sqrt {\frac {1}{n \gamma d}} \leqslant \frac {1}{\varepsilon} \cdot \sqrt {\frac {\log (1 / \delta)}{\gamma d}}.
$$
Thus by standard linear algebra, for typical instances  $\mathbf{G} \sim \mathsf{SBM}_n(d,\gamma,x)$ , the leading eigenvector of  $\hat{X}(\mathbf{G}) + \mathbf{N}$  will be highly correlated with the true community vector  $x$  whenever the average degree  $d$  is large enough. In conclusion, a simple randomized rounding step is enough!
Remark 2.1 (From weak recovery to exact recovery). In the non-private setting, given a weak recovery algorithm for the stochastic block model, one can use this as an initial estimate for a boosting procedure based on majority voting to achieve exact recovery. We show that this can be done privately. See Appendix C.2 in the supplements.
An advanced application: learning mixtures of Gaussians In the context of SBMs our argument greatly benefited from two key properties: first, on adjacent inputs  $Y - Y'$  was bounded in an appropriate norm; and second, the convex set  $\mathcal{K}$  was fixed. In the context of learning mixtures of spherical Gaussians as in Model 1.2, both this properties are not satisfied (notice how one of this second properties would be satisfied assuming bounded centers!). So additional ideas are required.
The first observation, useful to overcome the first obstacle, is that before finding the centers, one can first find the  $n$ -by- $n$  membership matrix  $W(Y)$  where  $W(Y)_{ij} = 1$  if  $i,j$  where sampled from the same mixture component and 0 otherwise. The advantage here is that, on adjacent inputs,  $\| W(Y) - W(Y')\|_{\mathrm{F}}^2 \leqslant 2n / k$  and thus one recovers the first property. $^{15}$  Here early sum-of-squares algorithms for the problem [30, 36] turns out to be convenient as they rely on minimizing the function  $\| W\|_{\mathrm{F}}^2$  subject to the following system of polynomial inequalities in variables  $z_{11}, \ldots, z_{1k}, \ldots, z_{nk}$ , with  $W_{ij} = \sum_{\ell} z_{i\ell} z_{j\ell}$  for all  $i,j \in [n]$  and a parameter  $t > 0$ .
$$
\left\{ \begin{array}{l l} z _ {i \ell} ^ {2} = z _ {i \ell} & \forall i \in [ n ], \ell \in [ k ] \quad (\text {i n d i c a t o r s}) \\ \sum_ {\ell \in [ k ]} z _ {i \ell} \leqslant 1 & \forall i \in [ n ] \quad (\text {c l u s t e r m e m b e r s h i p}) \\ z _ {i \ell} \cdot z _ {i \ell^ {\prime}} = 0 & \forall i \in [ n ], \ell \in [ k ] \quad (\text {u n i q u e m e m b e r s h i p}) \\ \sum_ {i} z _ {i \ell} = n / k & \forall \ell \in [ k ] \quad (\text {s i z e o f c l u s t e r s}) ^ {1 6} \\ \mu_ {\ell} ^ {\prime} = \frac {k}{n} \sum_ {i} z _ {i \ell} \cdot y _ {i} & \forall \ell \in [ k ] \quad (\text {m e a n s o f c l u s t e r s}) \\ \frac {k}{n} \sum_ {i} z _ {i \ell} \langle y _ {i} - \mu_ {\ell} ^ {\prime}, u \rangle^ {2 t} \leqslant (2 t) ^ {t} \cdot \| u \| _ {2} ^ {t} & \forall u \in \mathbb {R} ^ {d}, \ell \in [ k ] \quad (\text {s u b g a u s s i a n i t y o f t - m o m e n t}) \end{array} \right\}
$$
For the scope of this discussion, we may disregard computational issues and assume we have access to an algorithm returning a point from the convex hull  $\mathcal{K}(Y)$  of all solutions to our system of inequalities. Each indicator variable  $z_{i\ell} \in \{0,1\}$  is meant to indicate whether sample  $y_i$  is believed
15Notice for typical inputs  $\mathbf{Y}$  from Model 1.2 one expect  $\| W(\mathbf{Y})\|_{\mathrm{F}}^{2}\approx n^{2} / k$
Formally, we would replace the constraint on the size of the clusters by one which requires them to be of size  $(1 \pm \alpha) \frac{n}{k}$ , for some small  $\alpha$ .
17While this is far from being true, it turns out that having access to a pseudo-distribution satisfying  $\mathcal{P}(Y)$  is enough for our subsequent argument to work, albeit with some additional technical work required.
<sup>18</sup>We remark that a priori it is also not clear how to encode the subgaussian constraint in a way that we could recover a degree- $t$  pseudo-distribution satisfying  $\mathcal{P}(Y)$  in polynomial time. By now this is well understood, we discuss this in Appendix A in the supplements.
to be in cluster  $C_{\ell}$ . In the non-private setting, the idea behind the program is that – for typical  $\mathbf{Y}$  sampled according to Model 1.2 with minimum separation  $\Delta \geqslant k^{1 / t}\sqrt{t}$  any solution  $W(\mathbf{Y}) \in \mathcal{K}(\mathbf{Y})$  is close to the ground truth matrix  $W^{*}(\mathbf{Y})$  in Frobenius norm:  $\| W(\mathbf{Y}) - W^{*}(\mathbf{Y})\|_{\mathrm{F}}^2 \leqslant 1 / \mathrm{poly}(k)$ . Each row  $W(\mathbf{Y})_i$  may be seen as inducing a uniform distribution over a subset of  $\mathbf{Y}$ . Combining the above bound with the fact that subgaussian distributions at small total variation distance have means that are close, we conclude the algorithm recovers the centers of the mixture.
While this program suggests a path to recover the first property, it also possesses a fatal flaw: the projection  $W'$  of  $W \in \mathcal{K}(Y)$  onto  $\mathcal{K}(Y) \cap \mathcal{K}(Y')$  may be far in the sense that  $||W||_{\mathrm{F}}^2 - ||W'||_{\mathrm{F}}^2 | \geqslant \Omega(\|W\|_{\mathrm{F}}^2 + \|W'\|_{\mathrm{F}}^2) \geqslant \Omega(n^2 / k)$ . The reason behind this phenomenon can be found in the constraint  $\sum_{i} z_{i\ell} = n / k$ . The set indicated by the vector  $(z_{1\ell}, \ldots, z_{n\ell})$  may be subgaussian in the sense of  $\mathcal{P}(Y)$  for input  $Y$  but, upon changing a single sample, this may no longer be true. We work around this obstacle in two steps:
1. We replace the above constraint with  $\sum_{i}z_{il}\leqslant n / k$
2. We compute  $\hat{W} \coloneqq \operatorname*{argmin}_{W \text{ solving } \mathcal{P}(Y)} \| J - W \|_{\mathrm{F}}^2$ , where  $J$  is the all-one matrix.[20]
The catch now is that the program is satisfiable for any input  $Y$  since we can set  $z_{il} = 0$  whenever necessary. Moreover, we can guarantee property (ii) (required by our sensitivity argument) for  $\alpha \leqslant O(n / k)$ , since we can obtain  $W' \in \mathcal{K}(Y) \cap \mathcal{K}(Y')$  simply zeroing out the row/column in  $W$  corresponding to the sample differing in  $Y$  and  $Y'$ . Then for typical inputs  $\mathbf{Y}$ , the correlation with the true solution is guaranteed by the new strongly convex objective function.
We offer some more intuition on the choice of our objective function: Recall that  $W_{ij}$  indicates our guess whether the  $i$ -th and  $j$ -th datapoints are sampled from the same Gaussian component. A necessary condition for  $W$  to be close to its ground-truth counterpart  $W^{*}$ , is that they roughly have the same number of entries that are (close to) 1. One way to achieve this would be to add the lower bound constraining  $\sum_{\ell} z_{i\ell} \gtrsim \frac{n}{k}$ . However, such a constraint could cause privacy issues: There would be two neighboring datasets, such that the constraint set induced by one dataset is satisfiable, but the constraint set induced by the other dataset is not satisfiable. We avoid this issue by noticing that the appropriate number of entries close to 1 can also be induced by minimizing the distance of  $W$  to the all-one matrix. This step is also a key difference from [35], explained in more detail below.
From low sensitivity of the indicators to low sensitivity of the estimates For adjacent inputs  $Y, Y'$  let  $\hat{W}, \hat{W}'$  be respectively the matrices computed by the above strongly convex programs. Our discussion implies that, applying our sensitivity bound, we can show  $\| \hat{W} - \hat{W}' \|_{\mathrm{F}}^2 \leqslant O(n / k)$ . The problem is that simply applying a randomized rounding approach here cannot work. The reason is that even tough the vector  $\hat{W}_i$  induces a subgaussian distribution, the vector  $\hat{W}_i + v$  for  $v \in \mathbb{R}^n$ , might not. Without the subgaussian constraint we cannot provide any meaningful utility bound. In other words, the root of our problem is that there exists heavy-tailed distributions that are arbitrarily close in total variation distance to any given subgaussian distribution.
On the other hand, our sensitivity bound implies  $\| \hat{W} -\hat{W}^{\prime}\|_{1}^{2}\leqslant o(\| \hat{W}\|_{1})$  and thus, all but a vanishing fraction of rows  $i\in [n]$  must satisfy  $\| \hat{W}_i - \hat{W}_i'\| _1\leqslant o(\| \hat{W}_i\| _1)$ . For each row  $i$ , let  $\mu_i,\mu_i^\prime$  be the means of the distributions induced respectively by  $\hat{W}_i,\hat{W}_i^\prime$ . We are thus in the following setting:
1. For a set of  $(1 - o(1))\cdot n$  good rows  $\| \mu_i - \mu_i^{\prime}\| _2\leqslant o(1)$
2. For the set  $\mathcal{B}$  of remaining bad rows, the distance  $\| \mu_i - \mu_i'\| _2$  may be unbounded.
We hide differences of the first type as follows: pick a random subsample  $\mathcal{S}$  of  $[n]$  of size  $n^c$ , for some small  $c > 0$ , and for each picked row use the Gaussian mechanism. The subsampling step is useful as it allows us to decrease the standard deviation of the entry-wise random noise by a factor  $n^{1 - c}$ . We hide differences of the second type as follows: Note that most of the rows are clustered together in space. Hence, we aim to privately identify the regions which contain many of the rows.
19More generally, we may think of a vector  $v\in \mathbb{R}^n$  as the vector inducing the distribution given by  $v / \| v\| _1$  onto the set  $Y$  of  $n$  elements.
20We remark that for technical reasons our function in Appendix D.1 in the supplements will be slightly different. We do not discuss it here to avoid obfuscating our main message.
Formally, we use a classic high dimensional  $(\varepsilon, \delta)$ -differentially private histogram learner on  $S$  and for the  $k$  largest bins of highest count privately return their average (cf. Lemma A.13). The crux of the argument here is that the cardinality of  $B \cap S$  is sufficiently small that the privacy guarantees of the histogram learner can be extended even for inputs that differ in  $|B \cap S|$  many samples. Finally, standard composition arguments will guarantee privacy of the whole algorithm.
Comparison with the framework of Kothari-Manurangsi-Velingker Both Kothari-Manurangsi-Velingker [35] and our work obtained private algorithms for high-dimensional statistical estimation problems by privatizing strongly convex programs, more specifically, sum-of-squares (SoS) programs. The main difference between KMV and our work lies in how we choose the SoS program. For the problem of robust moment estimation, KMV considered the canonical SoS program from [30, 36] which contains a minimum cardinality constraint (e.g.,  $\sum_{l}z_{il}\gtrsim \frac{n}{k}$  in the case of GMMs). Such a constraint is used to ensure good utility. However, as alluded to earlier, this is problematic for privacy: there will always exist two adjacent input datasets such that the constraints are satisfiable for one but not for the other. KMV and us resolve this privacy issue in different ways.
KMV uses an exponential mechanism to pick the lower bound of the minimum cardinality constraint. This step also ensures that solutions to the resulting SoS program will have low sensitivity. In contrast, we simply drop the minimum cardinality constraint. Then the resulting SoS program is always feasible for any input dataset! To still ensure good utility, we additionally pick an appropriate objective function. For example, in Gaussian mixture models, we chose the objective  $\| W - J\| _F^2$ . Our approach has the following advantages: First, the exponential mechanism in KMV requires computing  $O(n)$  scores. Computing each score requires solving a large semidefinite program, which can significantly increase the running time. Second, proving that the exponential mechanism in KMV works requires several steps: 1) defining a (clever) score function, 2) bounding the sensitivity of this score function and, 3) showing existence of a large range of parameters with high score. Our approach bypasses both of these issues.
Further, as we show, our general recipe can be easily extended to other high dimensional problems of interest: construct a strongly convex optimization program and add noise to its solution. This can provide significant computational improvements. For example, in the context of SBMs, the framework of [35] would require one to sample from an exponential distribution over matrices. Constructing and sampling from such distributions is an expensive operation. However, it is well-understood that an optimal fractional solution to the basic SDP relaxation we consider can be found in near quadratic time using the standard matrix multiplicative weight method [7, 58], making the whole algorithm run in near-quadratic time. Whether our algorithm can be sped up to near-linear time, as in [7, 58], remains a fascinating open question.
Comparison with previous works on empirical risk minimization Results along the lines of the sensitivity bound described at the beginning of the section (see Lemma B.1 for a formal statement) have been extensively used in the context of empirical risk minimization [15, 34, 57, 11, 63, 50]. Most results focus on the special case of unconstrained optimization of strongly convex functions. In contrast, our sensitivity bound applies to the significantly more general settings where both the objective functions and the constrained set may depend on the input.[21] Most notably for our settings of interest, [15] studied unconstrained optimization of (smooth) strongly convex functions depending on the input, with bounded gradient. We recover such a result for  $X' = X$  in (ii). In [50], the authors considered constraint optimization of objective functions where the domain (but not the function) may depend on the input data. They showed how one can achieve differential privacy while optimize the desired objective function by randomly perturbing the constraints. It is important to remark that, in [50], the notion of utility is based on the optimization problem (and their guarantees are tight only up to logarithmic factors). In the settings we consider, even in the special case where  $f$  does not depend on the input, this notion of utility may not correspond to the notion of utility required by the estimation problem, and thus, the corresponding guarantees can turn out to be too loose to ensure the desired error bounds.
21The attentive reader may argue that one could cast convex optimization over a constrained domain as unconstrained optimization of a new convex function with the appropriate penalty terms. In practice however, this turns out to be hard to do for constraints such as Definition A.19.
This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No 815464).