SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xiao Hu*
University of Waterloo Waterloo, Canada xiaohu@uwaterloo.ca
Google Research Mountain View, CA, US ravi.k53@gmail.com
Pasin Manurangi
Given a dataset of  $n$  i.i.d. samples from an unknown distribution  $P$ , we consider the problem of generating a sample from a distribution that is close to  $P$  in total variation distance, under the constraint of differential privacy (DP). We study the problem when  $P$  is a multi-dimensional Gaussian distribution, under different assumptions on the information available to the DP mechanism: known covariance, unknown bounded covariance, and unknown unbounded covariance. We present new DP sampling algorithms, and show that they achieve near-optimal sample complexity in the first two settings. Moreover, when  $P$  is a product distribution on the binary hypercube, we obtain a pure-DP algorithm whereas only an approximate-DP algorithm (with slightly worse sample complexity) was previously known.
Differential privacy (DP) [18, 16] is a strong and rigorous notion of privacy that has been increasingly studied and deployed as protection against the leakage of personal data used to train ML models.
A basic setting widely studied in ML is distribution learning, where given samples drawn i.i.d. from an unknown distribution  $P$ , we seek to output a distribution  $Q$  that is as close to  $P$  as possible (formal definitions are given in Section 1.1). Recent works [27, 2, 29, 4] have studied DP distribution learning, whereby the output distribution is guaranteed to stay roughly the same when a single input sample is changed. A closely related setting is DP mean and covariance estimation studied by [38, 21, 36, 30, 7, 8, 11, 15, 28, 25, 9, 26]. Motivated by the fact that tasks often require much fewer samples than full-fledged learning, the very recent work of [35] studied the task of DP distribution sampling, where the goal is to generate a sample from a distribution  $Q$  that is as close as possible to the distribution  $P$  from which the input samples are drawn i.i.d. (In the non-private setting, variants of the sampling task have also been studied, e.g., in the work on sample amplification by [5].)
In this work, we study DP distribution sampling, and the quantitative gap with respect to the a priori more challenging task of DP distribution learning. For the case of multi-dimensional Gaussians, we consider three natural settings: known covariance, unknown bounded covariance, and unknown unbounded covariance; for the first two, we obtain near-tight bounds. This answers an open question of [35]. Moreover, we obtain near-tight bounds on sampling in the case of product distributions on  $\{0,1\}^d$  with pure-DP, also improving upon previous work.
*This work was done while the author was visiting Google Research.
Motivation. There are many natural settings where a sample from the underlying distribution would be sufficient (as an alternative to the possibly more expensive task of learning the distribution).
For example, if one is implementing an algorithm that would run on sensitive user data, then, as part of the usual software development cycle, unit tests [24] are important. Writing unit tests on real data can violate users' privacy (e.g., if the unit test code is public). On the other hand, (privately) learning the distribution of the user data could be a significant overkill for this application. A private sample of the underlying distribution would provide a sufficiently representative input for the unit testing algorithm, and this input could be revealed publicly without compromising the privacy of the users.
A second motivation arises in distributed settings. Consider the situation where algorithms are being developed on medical data provided by multiple hospitals. The algorithm designer would like a synthetic dataset that would work well for the algorithm development process. Such a dataset could be obtained if each hospital provides a few (private) samples from its underlying distribution.
Finally, we describe another distributed setting motivation where a central curator wishes to build a synthetic dataset from multiple users, each contributing multiple items, under the constraint of item-level DP (as opposed to user-level DP). A simple approach for generating such a synthetic dataset is to let each user send to the curator a private sample (satisfying item-level DP).
1.1 Formulation
Let  $\mathcal{D}$  be a class of distributions on some domain  $\mathcal{X}$ . We consider a setting where there is an unknown distribution  $D\in \mathcal{D}$  and an algorithm has sample access to  $D$ . There are two natural problems that can be posed in this setting. In the sampling problem, the goal is to design an algorithm that uses samples from  $D$  (which is unknown) and outputs an element in  $\mathcal{X}$ . We say that an algorithm  $\mathcal{A}$  is an  $\alpha$ -accurate sampler for  $\mathcal{D}$  iff  $d_{\mathrm{tv}}(D,Q_{\mathcal{A},D})\leq \alpha$  for all  $D\in \mathcal{D}$ , where  $Q_{\mathcal{A},D}$  denotes the distribution of  $\mathcal{A}(\mathbf{X})$  where  $\mathbf{X}\sim D^{n}$  and  $d_{\mathrm{tv}}(\cdot ,\cdot)$  denotes the total variation distance. Here  $n$  is said to be the sample complexity of the sampler. In the learning problem, the goal is to design an algorithm that outputs a distribution  $D^{\prime}\in \mathcal{D}$ . We say that an algorithm  $\mathcal{A}$  is an  $(\alpha,\beta)$ -accurate learner for  $\mathcal{D}$  if  $\operatorname*{Pr}_{\mathbf{X}\sim D^n,D'\sim \mathcal{A}(\mathbf{X})}[d_{\mathrm{tv}}(D',D)\leq \alpha ]\geq 1 - \beta$  for all  $D\in \mathcal{D}$ ; as before,  $n$  is the sample complexity of the learner.
The DP version of the learning problem is well studied, e.g., [27, 4]; in this work we study the DP version of the sampling problem. First, we recall the definition of DP. We consider the substitution notion, i.e., two datasets are neighbors iff they have the same number of samples and we can transform one to the other by changing a single sample.
Definition 1.1 (Differential Privacy [18, 16]). An algorithm  $M: \mathcal{V} \to \mathcal{O}$  is said to be  $(\epsilon, \delta)$ -differentially private  $((\epsilon, \delta)\text{-DP})$  for  $\epsilon > 0$ ,  $\delta \geq 0$  iff, for every  $S \subseteq \mathcal{O}$  and every neighboring datasets  $Y, Y' \in \mathcal{V}$ , we have  $\operatorname*{Pr}[\mathcal{M}(Y) \in S] \leq e^{\epsilon} \cdot \operatorname*{Pr}[\mathcal{M}(Y') \in S] + \delta$ .
We abbreviate  $(\epsilon, 0)$ -DP by  $\epsilon$ -DP, aka, pure-  $DP$ ; the  $\delta \neq 0$  case is approximate-  $DP$ . We assume that the privacy parameters satisfy  $\epsilon \leq 1$ ,  $\delta \leq 1/2$ , and the accuracy parameter satisfies  $0 < \alpha \leq 1/2$ .
1.2 Our Results for Gaussian Distributions in  $\mathbb{R}^d$
We focus on the  $d$ -dimensional Gaussian distribution, i.e.,  $D = \mathsf{N}(\mu, \Sigma)$  with  $\mu \in \mathbb{R}^d$ ,  $\Sigma \in \mathbb{R}^{d \times d}$ . Recall that  $\mathsf{N}(\mu, \Sigma)$  is supported on  $\mathbb{R}^d$  with  $f_{\mathsf{N}(\mu, \Sigma)}(x) \propto \exp\left(-\frac{1}{2}(x - \mu)^T\Sigma^{-1}(x - \mu)\right)$ . We study three cases that have been considered in the literature:
- Known Covariance:  $\Sigma$  is known, but  $\mu$  is not. More formally, the class of distributions is  $\mathcal{D}_{\Sigma}^{\mathbb{N}} := \{\mathsf{N}(\mu, \Sigma) \mid \mu \in \mathbb{R}^{d}\}$ .
- Unknown Bounded Covariance: Both  $\mu, \Sigma$  are unknown but with a promise that  $I \preceq \Sigma \preceq \kappa \cdot I$  for a known constant  $\kappa > 0$ , i.e., the class of distributions is  $\mathcal{D}_{\kappa}^{\mathbb{N}} := \{\mathsf{N}(\mu, \Sigma) \mid \mu \in \mathbb{R}^{d}, \Sigma \in \mathbb{R}^{d \times d}$  s.t.  $I \preceq \Sigma \preceq \kappa \cdot I\}$ .
- Unknown Unbounded Covariance: Both  $\mu, \Sigma$  are unknown, i.e., the class of distributions is  $\mathcal{D}^{\mathbb{N}} := \{\mathsf{N}(\mu, \Sigma) \mid \mu \in \mathbb{R}^d, \Sigma \in \mathbb{R}^{d \times d}\}$ .
A summary of our DP sampling results along with a comparison to known DP learning results is in Table 1. Before we describe our results in more detail, we highlight the following. (i) In all cases,
2We write  $A\preceq B$  to denote that  $B - A$  is positive semi-definite, for matrices  $A$  and  $B$
the dependence of our algorithms on the accuracy parameter  $\alpha$  is only polylogarithmic, whereas for DP learning algorithms, this dependence is polynomial. (ii) In the case of known covariance and unbounded known covariance, we obtain improvements over DP learning in terms of the dependence on the dimension  $d$ . (iii) All of our algorithms run in polynomial time, although we do not explicitly state the running time in the formal statements.
Known Covariance. While any DP learning algorithm requires  $\tilde{\Theta}\left(\frac{d}{\alpha^2} + \frac{d}{\alpha \epsilon}\right)$  samples in this setting [27], we show that, surprisingly, only  $\tilde{O}\left(\sqrt{d}/\epsilon\right)$  samples suffice for DP sampling.
Theorem 1.2. There is an  $\alpha$ -accurate  $(\epsilon, \delta)$ -DP sampler for Gaussian distributions with known covariance with sample complexity  $O\left(\frac{\sqrt{d}}{\epsilon} \cdot \mathrm{polylog}\left(\frac{d}{\delta \epsilon \alpha}\right)\right)$ .
We also show that the  $\sqrt{d}$  dependence is necessary, i.e., that our algorithm's sample complexity is tight up to polylogarithmic factors.
Theorem 1.3. Any 0.1-accurate  $(\epsilon, \delta)$ -DP sampler for Gaussian distributions with known covariance must have sample complexity  $\Omega(\sqrt{d}/\epsilon)$ .
Unknown Bounded Covariance. In this setting, we obtain an algorithm with sample complexity of  $\tilde{O}_{\kappa}\left(\frac{d}{\epsilon}\right)$ ; in contrast, the best known DP learning algorithm requires  $\tilde{\Theta}_{\kappa}\left(\frac{d^2}{\alpha^2} + \frac{d^2}{\alpha\epsilon}\right)$  samples [27, 29].
Theorem 1.4. There is an  $\alpha$ -accurate  $(\epsilon, \delta)$ -DP sampler for Gaussian distributions with unknown covariance, under the assumption  $I \preceq \Sigma \preceq \kappa \cdot I$ , with sample complexity  $O\left(\frac{d}{\epsilon} \cdot \kappa^2 \cdot \text{polylog}\left(\frac{d}{\delta \epsilon \alpha}\right)\right)$ .
Similar to before, we can show that the sample complexity dependence on  $d,\epsilon$  is near-optimal:
Theorem 1.5. Let  $\alpha >0$  be a sufficiently small constant, and  $\epsilon ,\delta$  be such that  $\delta \leq O\left(\frac{1}{nd^2}\right)$ . Then, any  $\alpha$ -accurate  $(\epsilon ,\delta)$ -DP sampler for Gaussian distributions with unknown covariance, under the assumption  $I\preceq \Sigma \preceq 2I$ , must have sample complexity  $n = \Omega \left(\frac{d}{\epsilon\sqrt{\log d}}\right)$ .
Unknown Unbounded Covariance. In this setting, the best known DP learning algorithm uses  $\tilde{\Theta}\left(\frac{d^2}{\alpha^2} + \frac{d^2}{\alpha\epsilon}\right)$  samples [4, 29, 28]. For DP sampling, we show that we can reduce the dependence on  $\alpha$  to polylogarithmic and the dependence on  $d$  to  $d^{1.5}$ .
Theorem 1.6. There exists an  $\alpha$ -accurate  $(\epsilon, \delta)$ -DP sampler for Gaussian distributions (without any assumption) with sample complexity  $O\left(\frac{d^{1.5}}{\epsilon}\log \left(\frac{d}{\alpha\epsilon\delta}\right)\right)$ .
Known Covariance	Bounded Covariance	Unbounded Covariance
Non-Private Learning (Folklore)	Θ(d/α2)	Θ(d2/α2)	Θ(d2/α2)
(ε,δ)-DP Learning	Θ(d/α2+d/αε)[27]	Θ(d2/α2+d2/αε)[27, 28]	Θ(d2/α2+d2/αε)[4, 28]
(ε,δ)-DP Sampling (Our results)	Θ(√d/ε)Theorems 1.2,1.3	Θ(d/ε)Theorems 1.4, 1.5	O(d1.5/ε)Theorem 1.6
Table 1: Sample complexity of private learning and sampling for Gaussian distributions. Here,  $\tilde{O},\tilde{\Theta}$  hide factors that are polylogarithmic in  $d,1 / \epsilon ,1 / \delta ,1 / \alpha$  (and  $1 / \beta$  in the case of learning).
1.3 Our Results for Product Distributions on  $\{0,1\}^d$
For  $p \in [0,1]$ , let  $\mathsf{Ber}(p)$  be the Bernoulli distribution supported on  $\{0,1\}$  with probability mass function  $f_{\mathsf{Ber}(p)}(0) = 1 - p$  and  $f_{\mathsf{Ber}(p)}(1) = p$ . We consider product distributions  $\mathsf{Ber}(p_1) \otimes \dots \otimes \mathsf{Ber}(p_d)$  where  $p_1, \ldots, p_d \in [0,1]$  are unknown. (In other words, the class of distributions is  $\mathcal{D}^{\mathrm{prod}} := \{\mathsf{Ber}(p_1) \otimes \dots \otimes \mathsf{Ber}(p_d) \mid p_1, \ldots, p_d \in [0,1]\}$ .)
We give a pure-DP sampler with sample complexity  $\tilde{O}\left(\frac{d}{\alpha\epsilon}\right)$ . Previously, only approximate-DP sampler with similar sample complexity was known from [35], which also provided a matching lower bound. In comparison, DP learning uses  $\tilde{\Theta}\left(\frac{d}{\alpha^2} + \frac{d}{\alpha\epsilon}\right)$  samples [27, 10].
Theorem 1.7. There exists an  $\alpha$ -accurate  $\epsilon$ -DP sampler for product distributions on  $\{0,1\}^d$  with sample complexity  $O\left(\frac{d\log\left(\frac{d}{\alpha}\right)}{\alpha\epsilon} + \frac{d\log^2\left(\frac{d}{\alpha}\right)}{\epsilon}\right)$ .
Non-Private Learning	Θ(d/α2)
ε-DP Learning	Θ(d/α2 + d/αε)[29, 27]
(ε, δ)-DP Sampling	Θ(d/αε)[35]
ε-DP Sampling (Our result)	Θ(d/αε) Theorem 1.7
Table 2: Sample complexity for private learning and sampling for product distributions on  $\{0,1\}^d$ . Here,  $\tilde{\Theta}$  hides factors that are polylogarithmic in  $d,1 / \alpha$  (and  $1 / \beta$  in the case of learning).
We also note that our result above improves upon even the approximate-DP sampler in [35] by logarithmic factors. Specifically, for  $\alpha \leq 1 / \log d$ , our sample complexity is  $O\left(\frac{d\log(d / \alpha)}{\alpha\epsilon}\right)$  whereas theirs is  $O\left(\frac{d\sqrt{\log(1 / \delta)}}{\alpha\epsilon}\left(\log^{9 / 4}d + \log^{5 / 4}(1 / \alpha)\right)\right)$ .
2.1 Gaussian Distributions: Algorithms
Known Covariance. When  $\Sigma$  is known, we may assume w.l.o.g. that  $\Sigma = I$ ; otherwise, we can transform each sample  $X$  into  $\Sigma^{-1/2} X$ . We start by using known algorithms [23, 40, 33] to find a "rough" estimate for the mean. In particular, we find an estimate  $\hat{\mu}$  such that  $\| \hat{\mu} - \mu \|_2 \leq R = \tilde{O}(\sqrt{d}/\epsilon)$  using  $n = \tilde{O}(\sqrt{d}/\epsilon)$  samples. By appropriately shifting the subsequent samples, this is equivalent to assuming that  $\| \mu \|_2 \leq R$ . We then focus on designing a DP sampler for this bounded mean case. It turns out, surprisingly, that the Gaussian mechanism suffices here. Specifically, for a parameter  $B > 0$  (chosen later), we truncate each sample so that its  $\ell_2$ -norm is at most  $B$ . We then output their average with a (spherical) Gaussian noise  $\mathsf{N}(0, \sigma^2 I)$  added. The description is given in Algorithm 1.
The analysis of the Gaussian mechanism [e.g., 20, Appendix A] shows that the algorithm is  $(\epsilon, \delta)$ -DP as long as we pick  $\sigma \geq \tilde{O}\left(\frac{B}{n\epsilon}\right)$ .
As for the accuracy, observe that if there were no truncation, then the output is exactly distributed as  $\mathsf{N}(\mu ,(\sigma^2 +1 / n)I)$ , which is precisely  $\mathsf{N}(\mu ,I)$  if we set  $\sigma^2 = (n - 1) / n$ . Therefore, by setting  $B = R + O(\sqrt{d + \log(1 / \alpha)})$  so that the truncation does not occur with probability  $1 - \alpha$ , we
Algorithm 1 SPHERICALGAUSSIANSAMPLER
Parameters:  $B, \sigma > 0$ , and  $n \in \mathbb{N}$ .
Sample  $X_{1},\ldots ,X_{n}\sim D$
for  $i = 1,\dots ,n$  do
Xtrunc=trunc(Bx) see (1)
Sample  $Z\sim \mathsf{N}(0,\bar{\sigma}^2 I)$
return  $Z + \frac{1}{n}\sum_{i\in [n]}X_i^{\mathrm{trunc}}$
ensure that the sampler is  $\alpha$ -accurate. The constraint that  $\sigma \geq \tilde{O}\left(\frac{B}{n\epsilon}\right)$  from privacy implies that we need  $n \geq \tilde{O}(\sqrt{d}/\epsilon)$  and hence yielding the sample complexity in Theorem 1.2.
Unknown Bounded Covariance. Recall that in this setting we know that  $I \preceq \Sigma \preceq \kappa \cdot I$ . While it might be tempting to use the above Gaussian mechanism for this setting as well, it turns out that this approach results in sample complexity that depends polynomially on  $1 / \alpha$ .<sup>3</sup>
To circumvent this, we first consider the case where  $\mu = 0$  (i.e., "centered" Gaussians). In this case, our algorithm originates from the following attempt: output  $\sum_{i\in [n]}a[i]\cdot X_i$ , where  $(a[1],\ldots ,a[n])\sim \mathrm{Uni}_n^{\mathbb{S}}$ , the uniform distribution over points on the unit sphere in  $\mathbb{R}^n$ . It follows from the 2-stability of the Gaussian distribution [41] that, when  $X_{1},\ldots ,X_{n}\sim \mathsf{N}(0,\Sigma)$ , this results<sup>4</sup> in an output that is distributed exactly as  $\mathsf{N}(0,\Sigma)$ .
Unfortunately, this algorithm is not DP: if  $X_{1} = \dots = X_{n - 1} = 0$ , then the output will reveal the direction of  $X_{n}$  in the clear. To remedy this, we build on the intuition that, if  $X_{1},\ldots ,X_{n}$  "sufficiently span all directions", then there should be "enough noise" to make this algorithm DP. In particular, using the bounded covariance property, we can show that if all the eigenvalues of  $\sum_{i\in [n]}X_iX_i^T$
3See Appendix C for a proof sketch of the sample complexity from such an approach.
4Note that this holds even for any fixed unit vector  $a\in \mathbb{R}^d$
are sufficiently large (and each  $X_{i}$  is truncated appropriately), then this algorithm is indeed "DP". This is perhaps the most technically challenging part of our work, as the noise is data-dependent and therefore poses significant hurdles in the privacy analysis (Section 4.2). Note that this is also the reason we need  $n \geq \Omega(d)$ , as otherwise  $X_{1}, \ldots, X_{n}$  cannot "sufficiently span all directions" in  $\mathbb{R}^{d}$ .
With the above, the last ingredient is a testing step (in the "propose-test-release" paradigm of [17]) that checks this eigenvalue condition. When this condition fails, we return  $\perp$ ; otherwise  $\sum_{i \in [n]} a[i] \cdot X_i$ .
To handle the case where  $\mu \neq 0$ , we take an output to be the sum of the average of  $n_1$  samples and  $\sqrt{1 - \frac{1}{n_1}} \cdot (\sum_{i \in [n_2]} a[i] \cdot U_i)$ , where each  $U_i$  is the difference between two fresh independent samples divided by  $\sqrt{2}$  and  $(a[1], \ldots, a[n_2]) \sim \mathrm{Uni}_{n_2}^{\mathbb{S}}$ . Notice here that each  $U_i \sim \mathsf{N}(0, \Sigma)$ , while the average over  $n_1$  samples is  $\sim \mathsf{N}(\mu, \frac{1}{n_1} \cdot \Sigma)$ ; thus, the sum is  $\sim \mathsf{N}(\mu, \Sigma)$  as desired. The full description is presented in Algorithm 2; the parameter setting and analysis can be found in Appendix B.4.
Unknown Unbounded Covariance.
We proceed by reducing this case to the previous setting. We do so by first applying the known DP "preconditioner" algorithm for Gaussians (with unknown unbounded covariance) from the work of [9] in order to obtain rough estimates  $\hat{\mu},\hat{\Sigma}$  of  $\mu ,\Sigma$  respectively. This allows us to transform any subsequent sample  $X$  into  $\hat{\Sigma}^{-1 / 2}(X - \hat{\mu})$  . This reduces us back to DP sampling for  $\mathsf{N}(\hat{\Sigma}^{-1 / 2}(\mu -$ $\hat{\mu}),\hat{\Sigma}^{1 / 2}\Sigma^{-1}\hat{\Sigma}^{1 / 2})$  . The guarantee of the DP preconditioner ensures that this Gaussian actually has bounded
Algorithm 2 BOUNDEDCOVGAUSSIANSAMPLER
Parameters:  $B, \Delta > 0$ , and  $n_1, n_2 \in \mathbb{N}$ .
Sample  $X_{1},\ldots ,X_{n_{1}},X_{n_{1} + 1},\ldots ,X_{n_{1} + 2n_{2}}\sim D$
for  $i = 1,\dots ,n_{1} + 2n_{2}$  do
$$
\left| \quad X _ {i} ^ {\text {t r u n c}} = \operatorname {t r u n c} _ {B} ^ {2} \left(X _ {i}\right) \right. \quad \triangleright \text {s e e (1)}
$$
for  $j = 1,\ldots ,n_{2}$  do
$$
\left| \quad U _ {i} = \frac {1}{\sqrt {2}} \left(X _ {n _ {1} + 2 i - 1} ^ {\text {t r u n c}} - X _ {n _ {1} + 2 i} ^ {\text {t r u n c}}\right) \right.
$$
Sample  $r\sim \mathrm{STLap}\left(\frac{\epsilon}{2},\frac{\delta}{2},\Delta\right)$   see Lemma 3.1
if  $\lambda_{\min}\left(\sum_{i\in [n_2]}U_iU_i^T\right) + r\geq 0.75n_2$  then
Sample  $a\sim \mathsf{Un}_{n_2}^{\mathbb{S}}$
$$
\text {r e t u r n} \frac {1}{n _ {1}} \left(\sum_ {i \in [ n _ {1} ]} X _ {i} ^ {\text {t r u n c}}\right) + \sqrt {1 - \frac {1}{n _ {1}}} \cdot \left(\sum_ {i \in [ n _ {2} ]} a [ i ] \cdot U _ {i}\right)
$$
covariance. Therefore, we can apply our previous algorithm. Note that a significant part of the sample complexity is due to the DP preconditioner of the Gaussian; it turns out that the sample complexity of this task is only  $\tilde{O}_{\epsilon,\alpha}(d^{3/2})$  (compared to  $\tilde{\Theta}_{\epsilon,\alpha}(d^2)$  for learning). Furthermore, since we only need the preconditioner to be a rough estimate, we can set the accuracy parameter for the learning to be  $\Theta(1)$  and thus avoid the polynomial dependence on  $1/\alpha$ . (Note that private preconditioner is a standard ingredient in the recipe for DP learning [e.g., 27].)
2.2 Gaussian Distributions: Lower Bounds
**Known Covariance.** Our lower bound in this setting builds on the following insight: if we take a constant number of samples from  $\mathsf{N}(\mu, I)$  (using the DP sampler) and use them to estimate  $\mu$ , then we incur an expected  $\ell_2^2$ -error that is  $O(d)$ . It turns out that known lower bounds for DP mean estimation of Gaussians with known covariance [27] hold for this setting and give a lower bound of  $\Omega\left(\frac{d}{\gamma\epsilon}\right)$ , where  $\gamma^2$  is the  $\ell_2^2$ -error. Plugging in  $\gamma = \sqrt{d}$  in our setting gives the desired lower bound of  $\Omega(\sqrt{d}/\epsilon)$ . In the actual proof, one complication stems from the fact that our DP sampler does not output a sample exactly from  $\mathsf{N}(\mu, I)$ . Nonetheless, we can quantify this in terms of the accuracy  $\alpha$ .
Unknown Bounded Covariance. In this setting, we reduce from a lower bound on DP covariance estimation [28]—rather than DP mean estimation earlier—of centered Gaussians. The challenge is that  $\Theta(1)$  samples from  $\mathsf{N}(0,\Sigma)$  do not provide a sufficiently high accuracy estimate for  $\Sigma$  so that we can apply the known lower bound<sup>5</sup>. Therefore, the above approach does not work directly.
To overcome this, we will have to use many samples to estimate the covariance. Recall that the sample complexity lower bound of  $\Omega(d^2/\epsilon)$  for covariance estimation requires the accuracy in the Frobenius distance (or the Mahalanobis distance) to be constant [28]. Due to this accuracy requirement, we need to use our DP sampler to generate  $L = \Omega(d^2)$  samples  $Y_1, \ldots, Y_L$  to achieve such an accuracy. We can draw a fresh batch  $X_1^i, \ldots, X_n^i$  of samples from the underlying distribution to generate each
This high accuracy requirement is inherent in the known DP covariance estimation lower bound; see [28, Remark 4.4] for more details.
$Y_{i}$ . However, since  $L = \Omega (d^{2})$ , this would use  $Ln$  samples in total. The covariance estimation lower bound would then yield  $Ln\geq \Omega (d^{2} / \epsilon)$ , implying  $n\geq \Omega (1 / \epsilon)$  even weaker than the mean estimation lower bound!
Fortunately, it turns out this can be overcome by using advanced composition of DP [19]. In particular, we may run our sampler  $L$  times on the same  $n$  samples to produce  $Y_{1},\ldots ,Y_{L}$ . In this case, the final covariance estimation algorithm has privacy loss parameter  $\tilde{O} (\sqrt{L}\cdot \epsilon) = \tilde{O} (d\cdot \epsilon)$  due to advanced composition (Theorem A.5). Therefore, we get a lower bound of  $\tilde{\Omega} (d^{2} / (d\cdot \epsilon)) = \tilde{\Omega} (d / \epsilon)$  as desired.