SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma A.3 (e.g., [14]). There exists a constant  $C > 0$  such that the following holds. For any  $\Sigma, \hat{\Sigma} \in \mathbb{R}^{d \times d}$ , we have  $d_{\mathrm{tv}}(\mathsf{N}(0,\Sigma), \mathsf{N}(0,\hat{\Sigma})) \leq C \cdot \| \Sigma - \hat{\Sigma} \|_{\Sigma}$ .
A.2.1 Amplification by Subsampling
Suppose that we have as an input  $N$  samples, and we draw  $n < N$  subsamples (without replacement) randomly from these  $N$  samples and run an  $(\epsilon, \delta)$ -DP algorithm on these  $n$  subsamples. Then, it is known that this results in an  $(\epsilon', \delta')$ -DP algorithm for  $\epsilon' < \epsilon, \delta' < \delta$ . Such a phenomenon is called amplification by subsampling and is often used in DP learning [e.g., 1]. We will use the following version of this for our lower bound proof.
Theorem A.4 (Amplification by Subsampling, [6]). Suppose that  $\mathcal{A}$  is an  $(\epsilon, \delta)$ -DP algorithm that takes in  $n$  samples. Then, the algorithm that draws  $n$  subsamples randomly without replacement out of  $N$  samples and runs  $\mathcal{A}$  on the  $n$  subsamples is  $(\epsilon', \delta')$ -DP where
$$
\epsilon^ {\prime} = \ln (1 + (n / N) (e ^ {\epsilon} - 1)), \quad \delta^ {\prime} = (n / N) \cdot \delta .
$$
A.3 Advanced Composition
We will also use the following advanced composition of DP.
Theorem A.5 (Advanced Composition, [19]). Let  $\epsilon_0 > 0$  and  $\delta_0, \delta_1 \in (0, 1/2]$ , and  $\mathcal{A}$  be any  $(\epsilon_0, \delta_0)$ -DP algorithm. Then, the algorithm that runs  $\mathcal{A}$  a total of  $k$  times is  $(\epsilon_1, k\delta_0 + \delta_1)$ -DP where
$$
\epsilon_ {1} = \left(\sqrt {2 k \ln \left(1 / \delta_ {1}\right)} + k \left(e ^ {\epsilon_ {0}} - 1\right)\right) \epsilon_ {0}.
$$
A.4 Lower Bounds for Mean and Covariance Estimations
As stated in our proof overview (Section 2), our lower bounds for Gaussian samplers (Theorems 1.3 and 1.5) are via reductions from previously known lower bounds for mean and covariance estimations. We state them below, starting with the mean estimation lower bound (aka "fingerprinting for Gaussian") result due to [27]
Theorem A.6 ([27]). For any  $\epsilon, \delta \in (0,1]$  such that  $\delta \leq O\left(\frac{\sqrt{d}}{n\sqrt{\log(n / d)}}\right)$ , if an  $(\epsilon, \delta)$ -DP algorithm  $\mathcal{M}$  that can take in  $N$  samples from  $D = \mathsf{N}(\mu, I)$  where  $-1 \leq \mu \leq 1$  and output an estimate  $\hat{\mu} \in \mathbb{R}^d$  that satisfies
$$
\mathbb {E} _ {\mathbf {X} \sim D ^ {N}, \hat {\mu} \sim \mathcal {M} (\mathbf {X})} [ \| \hat {\mu} - \hat {\mu} \| _ {2} ^ {2} ] \leq \gamma^ {2} \leq d / 6,
$$
then  $N\geq \Omega \left(\frac{d}{\gamma\epsilon}\right)$
Next is a similar lower bound but for covariance estimation. Notice that the lower bound on the sample complexity below is  $\Omega(d^2/\epsilon^2)$  whereas the lower bound above is only  $\Omega(d/\epsilon)$  (for constant accuracy parameter  $\gamma$ ).
Theorem A.7 ([28]). There exists a small constant  $\xi > 0$  such that the following holds: for any  $\epsilon, \delta \in (0,1]$  such that  $\delta \leq O\left(\min \left\{1 / n, d^2 / (n \log n)\right\}\right)$ , if an  $(\epsilon, \delta)$ -DP algorithm  $\mathcal{M}$  that can take in  $N$  samples from  $D = \mathsf{N}(0, \Sigma)$  where  $I \preceq \Sigma \preceq 2I$  and output an estimate  $\hat{\Sigma} \in \mathbb{R}^{d \times d}$  that satisfies
$$
\mathbb {E} _ {\mathbf {X} \sim D ^ {N}, \hat {\Sigma} \sim \mathcal {M} (\mathbf {X})} [ \| \hat {\Sigma} - \Sigma \| _ {\Sigma} ^ {2} ] \leq \xi^ {2},
$$
then  $N\geq \Omega \left(d^{2} / \epsilon\right)$
A.5 Agnostic Learner
In the main body of the paper, we considered the learner where the samples are drawn from  $D$  which belongs to a certain class  $\mathcal{D}$ . For our lower bound proofs, it will be convenient to consider the agnostic setting where  $D$  is not assumed to be from  $\mathcal{D}$ . The definition and accuracy guarantee of agnostic learner is given below. We remark that the definition coincides with the learner definition given earlier if we assume that  $D \in \mathcal{D}$ . (Note that the constant 3 is required for Theorem A.8.)
Agnostic Learning. Let  $\mathcal{D}$  be a class of distribution. An algorithm  $\mathcal{A}$ , which takes in  $n$  samples and output a distribution  $\hat{P} \in \mathcal{D}$ , is said to be an  $(\alpha, \beta)$ -accurate agnostic learner for a class  $\mathcal{D}$  of distributions iff, for any distribution  $P$ , we have
$$
\operatorname * {P r} _ {\mathbf {X} \sim P ^ {n}, \hat {P} \leftarrow \mathcal {A} (\mathbf {X})} \left[ d _ {\mathrm {t v}} (\hat {P}, P) \leq 3 \cdot \min  _ {P \in \mathcal {D}} d _ {\mathrm {t v}} (D, P) + \alpha \right] \geq 1 - \beta .
$$
We will use the following (well-known) result that the class of centered Gaussian distributions can be learned with sample complexity  $O(d^{2} / \gamma^{2})$ . For a full proof, see e.g., [3].
Theorem A.8. For any  $\gamma, \beta \in (0,1/2]$ , there exists a  $(\gamma, \beta)$ -accurate agnostic learner for centered Gaussian distributions in  $d$  dimensions with sample complexity
$$
O \left(\frac {d ^ {2} + \sqrt {\log (1 / \beta)}}{\gamma^ {2}}\right).
$$
B.1 Reduction to the Bounded Mean Case
The reduction will be done by applying a so-called "densest ball" algorithm; this is an algorithm that, when there is a ball of radius  $r$  that contains the majority of the input, can find a ball of radius  $O(r)$  containing the majority of the input. There are many algorithms known for this problem [23, 40, 33] that give similar guarantees<sup>10</sup>. We state such a guarantee below.
<sup>8</sup>We write  $-1\leq \mu \leq 1$  as a shorthand for  $-1\leq \mu [j]\leq 1$  for all  $j\in [d]$ .
Recall that centered simply means that  $\mu = 0$
<sup>10</sup>We note that our problem is in fact easier than those considered in the aforementioned work as we assume that  $\Sigma \preceq \kappa \cdot I$  and the problem can be solved "coordinate-by-coordinate". This means that many other algorithms e.g., truncated Laplace-based histogram [12] also work and give similar bounds. However, we note that some other algorithms [e.g., 7] require prior bounds on  $\| \mu \|$ , whereas Theorem B.1 does not require any such bound.
Theorem B.1 (e.g., [23]). For  $n_{\mathrm{dens}}(d,\beta ,\epsilon ,\delta) = \tilde{\Theta}\left(\frac{\sqrt{d}}{\epsilon}\right)$ , there exists an  $(\epsilon ,\delta)$ -DP algorithm  $\mathcal{A}_{\mathrm{dens}}$  that takes in  $X_{1},\ldots ,X_{n}\in \mathbb{R}^{d}$  together with a radius parameter  $r > 0$  such that: if there is an  $r$ -radius ball containing at least  $2 / 3$  fraction of the input points, then with probability  $1 - \beta$ , it outputs a ball of radius  $O(r)$  that contains at least half of the input points.
We can now give a reduction of an arbitrary mean case to the bounded mean case by first using Theorem B.1 to obtain a rough estimate of the mean and then shifting the remaining samples accordingly.
Proof. The new sampler works as follows:
- Let  $n_1 \coloneqq \max \{n_{\mathrm{dens}}(d, \alpha/4, \epsilon/2, \delta/2), 100 \log (4 / \alpha) \}$ ,  $r = 10C \kappa \sqrt{d}$  where  $C$  is the constant from Lemma 3.2, and  $n_2 \coloneqq n_{\mathrm{bm-sampler}}(\alpha/2, r, \epsilon/2, \delta/2)$ .
- Rough Mean Estimator Stage:
- Run  $(\epsilon /2,\delta /2)$  -DP  $\mathcal{A}_{\mathrm{dens}}$  from Theorem B.1 on  $n_1$  samples (with  $r$  as specified above) to get a ball centered at  $c\in \mathbb{R}^d$
- Sampling Stage:
- Draw  $n_2$  additional samples  $X_1, \ldots, X_{n_2}$ .
- Run  $(\epsilon / 2, \delta / 2)$ -DP  $(\alpha / 2)$ -accurate  $\mathcal{A}_{\mathrm{bm - sampler}}$  with  $R = r$  on  $(X_1 - c), \ldots, (X_{n_2} - c)$ ; let  $Y$  be its output.
- Output  $Y + c$ .
Privacy Analysis. Basic composition implies that the sampler is  $(\epsilon, \delta)$ -DP as desired.
Accuracy Analysis. Since  $\Sigma \preceq \kappa I$ , Lemma 3.2 implies that  $\operatorname{Pr}_{X \sim \mathbb{N}(\mu, \Sigma)}[\|X - \mu\|_2 \geq r] \leq 0.1$ . Thus, using standard concentration inequality (e.g., Theorem E.1) and since  $n_1 \geq 100 \log(4 / \alpha)$ , the probability that at least  $2/3$  of the  $n_1$  points sampled in the first stage contains in the ball  $B(\mu, r)$  is at least  $1 - \alpha / 4$ . The guarantee of the densest ball algorithm then implies that with probability  $1 - \alpha / 2$ , we have  $\| \mu - c \|_2 \leq r$ .
Let us consider a fixed  $c$  and consider the output distribution  $\mathcal{D}_Y^c$  of  $Y$  in the sampling stage and  $\mathcal{D}_{Y + c}^{c}$  of the final output  $Y + c$ . Notice that  $(X_{1} - c),\ldots ,(X_{n_{2}} - c)\sim \mathsf{N}(\mu -c,\Sigma)$ . Consequently, when  $\| \mu -c\| _2\leq r$ , we may apply the guarantee of  $\mathcal{A}_{\mathrm{bm - sampler}}$  to yield
$$
\alpha / 2 \geq d _ {\mathrm {t v}} \left(\mathcal {D} _ {Y} ^ {c}, \mathsf {N} (\mu - c, \Sigma)\right) = d _ {\mathrm {t v}} \left(\mathcal {D} _ {Y + c} ^ {c}, \mathsf {N} (\mu , \Sigma)\right).
$$
Combining these arguments, the sampler is  $\alpha$ -accurate as desired.
B.2 Unknown Bounded Covariance
Proof of Lemma 4.4. From Lemma 3.5, we have
$$
d _ {\epsilon} (\mathsf {U n i} _ {M, N} ^ {\mathbb {S}} | | \mathsf {U n i} _ {M, N} ^ {\mathbb {S}} + v) \leq \Pr_ {z \sim \mathsf {U n i} _ {M, N} ^ {\mathbb {S}}} [ f _ {\mathsf {U n i} _ {M, N} ^ {\mathbb {S}}} (z) > e ^ {\epsilon} \cdot f _ {\mathsf {U n i} _ {M, N} ^ {\mathbb {S}}} (z - v) ].
$$
For convenience, let us also define  $\eta = \tau \cdot \left(10\sqrt{\frac{\log(10 / \delta)}{cM}}\right)$ . Note that  $\eta \leq 0.01$ .
For  $z$  such that  $\| z\| \leq 0.9,|\langle z,v\rangle |\leq \eta$ , we also have  $\| z - v\| ^2\leq 0.9^2 +0.02 + 0.0001 < 0.9$  and thus
$$
\begin{array}{l} \ln \frac {\operatorname {U n i} _ {M , N} ^ {\mathbb {S}} (z)}{\operatorname {U n i} _ {M , N} ^ {\mathbb {S}} (z - v)} \stackrel {(2)} {=} \left(\frac {M - N}{2} - 1\right) \ln \left(\frac {1 - \| z \| ^ {2}}{1 - \| z - v \| ^ {2}}\right) \\ \leq \left(\frac {M - N}{2} - 1\right) \left(\frac {1 - \| z \| ^ {2}}{1 - \| z - v \| ^ {2}} - 1\right) \\ \leq \left(\frac {M - N}{2} - 1\right) \left(\frac {\| v \| ^ {2} - 2 \langle z , v \rangle}{1 - \| z - v \| ^ {2}}\right) \\ \leq M \left(1 0 \left(\tau^ {2} + 2 \eta\right)\right) \\ \leq \epsilon , \\ \end{array}
$$
where the last inequality follows from our choice of parameters (i.e.,  $M \geq \frac{10^4 \log(10 / \delta)}{c\epsilon}$ ).
Combining the previous two bounds, we have
$$
\begin{array}{l} d _ {\epsilon} \left(\operatorname {U n i} _ {M, N} ^ {\mathbb {S}} \mid \mid \operatorname {U n i} _ {M, N} ^ {\mathbb {S}} + v\right) \leq \Pr_ {z \sim \operatorname {U n i} _ {M, N} ^ {\mathbb {S}}} \left[ \| z \| > 0. 9 \vee \langle z, v \rangle > \eta \right] \\ \leq \Pr_ {z \sim \mathsf {U n i} _ {M, N} ^ {\mathbb {S}}} [ \| z \| > 0. 9 ] + \Pr_ {z \sim \mathsf {U n i} _ {M, N} ^ {\mathbb {S}}} [ \langle z, v \rangle > \eta ] \\ \leq \operatorname * {P r} _ {z \sim \mathbb {U n i} _ {M, N} ^ {\mathbb {S}}} [ \| z \| > 0. 9 ] + \operatorname * {P r} _ {z \sim \mathbb {U n i} _ {M, N} ^ {\mathbb {S}}} \left[ \left\langle z, \frac {v}{\| v \|} \right\rangle > 1 0 \sqrt {\frac {\log (1 0 / \delta)}{c M}} \right]. \\ \end{array}
$$
Next, recall that when  $z \sim \mathrm{Uni}_{M,N}^{\mathbb{S}}$ ,  $\|z\|^2 \sim \mathrm{Beta}(N/2, (M-N)/2)$ .<sup>11</sup> Thus, applying Lemma 3.3 and using the assumption that  $N \leq M/2$ , we can conclude that
$$
\operatorname * {P r} _ {z \sim \operatorname {U n i} _ {M, N} ^ {\mathbb {S}}} [ \| z \| > 0. 9 ] = \operatorname * {P r} _ {Z \sim \operatorname {B e t a} (N / 2, (M - N) / 2)} [ Z ^ {2} > 0. 8 1 ] \leq 2 \exp (- c \cdot 0. 0 1 M) \leq \delta / 2,
$$
where the first inequality is from Lemma 3.3 and the last inequality follows from our assumption that  $M \geq 100c\log (10 / \delta)$ .
Recall also that when  $z \sim \mathsf{Unl}_{M,N}^{\mathbb{S}}$ ,  $\left|\left\langle z, \frac{v}{\|v\|} \right\rangle \right|^2 \sim \mathsf{Beta}(1/2, (M - 1)/2)$ . Again, applying Lemma 3.3 yields
$$
\operatorname * {P r} _ {z \sim \mathrm {U n i} _ {M, N} ^ {\delta}} \left[ \left\langle z, \frac {v}{\| v \|} \right\rangle > 1 0 \sqrt {\frac {\log (1 0 / \delta)}{c M}} \right] \leq \operatorname * {P r} _ {Z \sim \mathrm {B e t a} (1 / 2, (M - 1) / 2)} \left[ Z ^ {2} > \frac {1 0 0 \log (1 0 / \delta)}{c M} \right] \leq \delta / 2,
$$
where the last inequality again follows from our choice of parameters.
Combining the three preceding inequalities, we can conclude that  $d_{\epsilon}(\mathsf{Uni}_{M,N}^{\mathbb{S}}||\mathsf{Uni}_{M,N}^{\mathbb{S}} + v)\leq \delta$  as desired.
B.3 Unknown Covariance
Throughout this section, we assume the standard assumption that the covariance matrix  $\Sigma$  is full rank. It is simple to extend the result to the more general case using an algorithm of [4], which can identify the span of  $\Sigma$  with probability one, using  $O(d\log (1 / \delta) / \epsilon)$  samples.
As stated earlier, we will use DP preconditioner from [9]. We say that an algorithm is an  $(\alpha, \beta)$ -accurate preconditioner for Gaussian distributions iff, when the input  $\mathbf{X}$  is drawn i.i.d. from  $\mathsf{N}(\mu, \Sigma)$  (for any  $\mu \in \mathcal{R}^d$ ,  $\Sigma \in \mathcal{R}^{d\times d}$ ), the algorithm outputs  $\hat{\mu}, \hat{\Sigma}$  that satisfies the following with probability at least  $1 - \beta$ :  $\| \hat{\Sigma}^{-1}(\hat{\mu} - \mu)\|_2 \leq \alpha$  and  $12\|\Sigma^{-1/2}\hat{\Sigma}\Sigma^{-1/2} - I\|_2 \leq \alpha$ . Below we provide a generic reduction from the unbounded covariance case to the bounded covariance case using preconditioner.
Lemma B.2. Suppose that the following hold:
- There exists an  $(\alpha, \beta)$ -accurate  $(\epsilon, \delta)$ -DP preconditioner  $\mathcal{A}_{\mathrm{precond}}$  for Gaussian distributions with sample complexity  $n_{\mathrm{precond}}(\alpha, \beta, \epsilon, \delta)$ .
- There exists an  $\alpha$ -accurate  $(\epsilon, \delta)$ -DP sampler  $\mathcal{A}_{\mathrm{bc - sampler}}$  for Gaussian distributions with  $I \preceq \Sigma \preceq \kappa \cdot I$  with sample complexity  $n_{\mathrm{bc - sampler}}(\kappa, \alpha, \epsilon, \delta)$ .
Then, there exists an  $\alpha$ -accurate  $(\epsilon, \delta)$ -DP sampler for Gaussian (without any assumption) with sample complexity
$$
n _ {\mathrm {p r e c n d}} (0. 0 0 1, \alpha / 2, \epsilon / 2, \delta / 2) + n _ {\mathrm {b c - s a m p l e r}} (4, \alpha / 2, \epsilon / 2, \delta / 2).
$$
Proof of Lemma B.2. The sampler is as follows:
- Let  $n_1 \coloneqq n_{\mathrm{precnd}}(0.001, \alpha/2, \epsilon/2, \delta/2)$  and  $n_2 \coloneqq n_{\mathrm{bc - sampler}}(4, \alpha/2, \epsilon/2, \delta/2)$ .
Estimation Stage:
<sup>11</sup>See, e.g., https://en.wikipedia.org/wiki/Beta_distribution
12The guarantee here is in the spectral norm, not Frobenius norm. However, this suffices for our purposes.
- Run  $(\epsilon /2,\delta /2)$  -DP  $(0.001,\alpha /2)$  -accurate  $\mathcal{A}_{\mathrm{pred}}$  on  $n_1$  samples to get  $\hat{\mu},\hat{\Sigma}$
- Draw  $n_2$  additional samples  $X_{1},\ldots ,X_{n_{2}}$ .
- Run  $(\epsilon /2,\delta /2)$ -DP  $(\alpha /2)$ -accurate  $\mathcal{A}_{\mathrm{bc - sampler}}$  on  $2\hat{\Sigma}^{-1 / 2}(X_1 - \hat{\mu}),\ldots ,2\hat{\Sigma}^{-1 / 2}(X_{n_2} - \hat{\mu})$  ; let  $Y$  be its output.
- Output  $0.5\hat{\Sigma}^{1/2}Y + \hat{\mu}$ .
Basic composition implies that the sampler is  $(\epsilon, \delta)$ -DP.
To see the accuracy guarantee, the guarantee of the learner implies that with probability  $1 - \alpha /2$  we have  $\| \hat{\Sigma}^{-1}(\hat{\mu} -\mu)\| _2\leq 0.001$  and  $\| \Sigma^{-1 / 2}\hat{\Sigma}\Sigma^{-1 / 2} - I\| _2\leq 0.001$ . The latter implies that all eigenvalues of  $\hat{\Sigma}^{1 / 2}\Sigma^{-1 / 2}$  lie in [0.998, 1.002]; in turn, this implies that the eigenvalues of  $\hat{\Sigma}^{-1 / 2}\Sigma^{1 / 2}$  all lie in [0.996, 1.004]. Finally, this implies that  $\| \Sigma^{1 / 2}\hat{\Sigma}^{-1}\Sigma^{1 / 2} - I\| _2\leq 0.01$ .
Let us consider a fixed  $\hat{\Sigma}$ ,  $\hat{\mu}$  and consider the output distribution of the sampling stage. When the above inequalities hold, the guarantee of  $\mathcal{A}_{\mathrm{bc - sampler}}$  implies that
$$
\alpha / 2 \geq d _ {\mathrm {t v}} (Y, \mathsf {N} (2 \hat {\Sigma} ^ {- 1 / 2} (\mu - \hat {\mu}), 4 \Sigma^ {1 / 2} \hat {\Sigma} ^ {- 1} \Sigma^ {1 / 2})) = d _ {\mathrm {t v}} (0. 5 \hat {\Sigma} ^ {1 / 2} Y + \hat {\mu}, \mathsf {N} (\mu , \Sigma)).
$$
Combining the above arguments, the sampler is  $\alpha$ -accurate as desired.
Combining the above reduction (Lemma B.2) together with a known result on DP preconditioner of Gaussians (Theorem B.3 below), we immediately arrive at Theorem 1.6.
Theorem B.3 ([8]). There is an  $(\alpha, \beta)$ -accurate  $(\epsilon, \delta)$ -DP preconditioner for Gaussian distributions with sample complexity  $O\left(\frac{d}{\alpha^2} + \frac{d^{3/2}}{\alpha \epsilon} \cdot \text{polylog}\left(\frac{d}{\alpha \beta \epsilon \delta}\right)\right)$ .
B.4 Proof of Theorem 4.3
We use Algorithm 2 with the following parameters:  $B = R + 10^{4}\kappa \sqrt{d + \log\left(\frac{2\log(2 / \delta)}{\alpha\epsilon}\right)}, \Delta = 2B^2$  and  $n_1 = n_2 = \left\lceil 10^4 C^2\left(B^2\cdot \frac{\log(10 / \delta)}{c\epsilon}\right)\right\rceil$ , where  $c, C$  are the constants from Lemmas 3.3 and 3.4.
Privacy Analysis. To analyze the privacy constraint, we first notice that the sensitivity of  $\lambda_{\min}\left(\sum_{i\in [n_2]}U_iU_i^T\right)$  is at most  $\Delta = (\sqrt{2}\cdot B)^2$  and Lemma 3.1 ensures that the check, which only uses  $\lambda_{\min}\left(\sum_{i\in [n_2]}U_iU_i^T\right) + r$ , is  $(\epsilon /2,\delta /2)$ -DP. Passing this check ensures that
$$
\lambda_ {\min } \cdot \left(\sum_ {i \in [ n _ {2} ]} U _ {i} U _ {i} ^ {T}\right) \geq 0. 7 5 n _ {2}. \tag {3}
$$
From Lemma 3.8, it suffices to show that the output of the algorithm is DP, under (3). Let  $\mathbf{X} = X_{1},\ldots ,X_{n_{1} + 2n_{2}}$
To do this, it will be more convenient to define  $V_{i} = \sqrt{1 - \frac{1}{n_{1}}} \cdot U_{i}$ . Condition (3) implies that
$$
\lambda_ {\min } \left(\sum_ {i \in [ n _ {2} ]} V _ {i} V _ {i} ^ {T}\right) \geq 0. 5 n _ {2} + \Delta . \tag {4}
$$
Let  $\mathcal{M}(\mathbf{X}) = \frac{1}{n_1}\left(\sum_{i\in [n_1]}X_i^{\mathrm{trunc}}\right) + \left(\sum_{i\in [n_2]}a[i]\cdot V_i\right)$ . Consider any neighboring inputs  $\mathbf{X}$  and  $\tilde{\mathbf{X}}$  that satisfy the condition. There are two cases, based on where they differ.
(i) Case I:  $\mathbf{X},\tilde{\mathbf{X}}$  differ on the first  $n_1$  samples. Let  $v = \frac{1}{n_1}\sum_{i\in [n_1]}(\tilde{X}_i^{\mathrm{trunc}} - X_i)$ ; notice that  $\| v\| \leq \frac{2B}{n_1}$ . Furthermore, (4) implies that  $\frac{1}{n_2}\left(\sum_{i\in [n_2]}V_iV_i^T\right)\succeq 0.5I$ . Letting  $W$  denote the matrix whose columns are  $V_{1},\ldots ,V_{n_{2}}$ ,  $d_{\frac{\epsilon}{2}}\left(\mathcal{M}(\mathbf{X})\parallel \mathcal{M}(\tilde{\mathbf{X}})\right) = d_{\frac{\epsilon}{2}}\left(\mathsf{Unil}_{W}^{\mathbb{S}}\parallel \mathsf{Unil}_{W}^{\mathbb{S}} + v\right)\leq \delta /2$ , where the inequality follows from Lemma 4.5.
(ii) Case II:  $\mathbf{X},\tilde{\mathbf{X}}$  differ on the last  $n_2$  samples; assume w.l.o.g.  $X_{n_1 + 2n_2}\neq \tilde{X}_{n_1 + 2n_2}$ . Let  $w = V_{n_2}^{\prime} - V_{n_2}$  and  $W$  denote the matrix whose columns are  $V_{1},\ldots ,V_{n_{2} - 1}$ . Furthermore, let  $a,a^{\prime}$  be independent samples from  $\mathrm{Uni}_q^{\S}$ .
$$
\begin{array}{l} d _ {\frac {\epsilon}{2}} \left(\mathcal {M} (\mathbf {X}) \parallel \mathcal {M} (\tilde {\mathbf {X}})\right) = d _ {\frac {\epsilon}{2}} \left(a [ n _ {2} ] \cdot V _ {n _ {2}} + \sum_ {i \in [ n _ {2} - 1 ]} a [ i ] \cdot V _ {i} \left\| a ^ {\prime} [ n _ {2} ] \cdot V _ {n _ {2}} ^ {\prime} + \sum_ {i \in [ n _ {2} - 1 ]} a ^ {\prime} [ i ] \cdot V _ {i}\right) \right. \\ \leq \int_ {y} d _ {\frac {\varepsilon}{2}} \left(\operatorname {U n i} _ {W} ^ {\mathbb {S}} \right\lvert   \left. \left| \frac {y \cdot w}{\sqrt {1 - y ^ {2}}} + \operatorname {U n i} _ {W} ^ {\mathbb {S}} \right.\right) \operatorname {U n i} _ {n _ {2}, 1} ^ {\mathbb {S}} (y) d y, \tag {5} \\ \end{array}
$$
where the inequality follows from the coupling  $a[n_2] = a'[n_2] = y$ ,  $a[n_2] \sim \mathrm{Uni}_{n_2,1}^{\mathbb{S}}$  and the observation that  $\left(\frac{a[1]}{\sqrt{1 - y^2}}, \ldots, \frac{a[n_2 - 1]}{\sqrt{1 - y^2}}\right)$  and  $\left(\frac{a'[1]}{\sqrt{1 - y^2}}, \ldots, \frac{a'[n_2 - 1]}{\sqrt{1 - y^2}}\right)$  are independently distributed as  $\mathrm{Uni}_{n_2 - 1}^{\mathbb{S}}$ .
When  $|y| \leq \sqrt{\frac{10\log(10 / \delta)}{cn_2}}$ , we have  $\left\| \frac{y \cdot w}{\sqrt{1 - y^2}} \right\| \leq \sqrt{\frac{20\log(10 / \delta)}{cn_2}} \cdot \sqrt{2} \cdot B \leq 20\sqrt{\frac{\log(10 / \delta)}{c}}$ . Furthermore, (4) implies that  $\frac{1}{n_2}\left(\sum_{i \in [n_2 - 1]} V_i V_i^T\right) \succeq 0.5I$ . Thus, Lemma 4.5 implies
$$
d _ {\epsilon / 2} \left(\operatorname {U n i} _ {W} ^ {\mathbb {S}} \right\lvert   \left| \frac {y \cdot w}{\sqrt {1 - y ^ {2}}} + \operatorname {U n i} _ {W} ^ {\mathbb {S}}\right) \leq \delta / 4.
$$
Combining this with (5), we arrive at  $d_{\epsilon /2}\left(\mathcal{M}(\mathbf{X})\parallel \mathcal{M}(\tilde{\mathbf{X}})\right) \leq \delta /4 + \operatorname*{Pr}_{y\sim \mathrm{Uni}_{n_2,1}^{\mathbb{S}}}\left[|y| > \sqrt{\frac{10\log(10 / \delta)}{cn_2}}\right]$ . Since  $y^{2} \sim \mathrm{Beta}(1 / 2,(n_{2} - 1) / 2)$ , we may apply the tail bound (Lemma 3.3) to get
$$
\Pr \left[ y ^ {2} > \frac {1 0 \log (1 0 / \delta)}{c n _ {2}} \right] \leq \delta / 4.
$$
Thus,  $d_{\epsilon /2}\left(\mathcal{M}(\mathbf{X})||\mathcal{M}(\tilde{\mathbf{X}})\right)\leq \delta /4 + \delta /4 = \delta /2.$
In both cases, we have  $d_{\epsilon /2}\left(\mathcal{M}(\mathbf{X})\parallel \mathcal{M}(\tilde{\mathbf{X}})\right)\leq \delta /2$  under (3). This concludes our privacy proof.
Accuracy Analysis. Let  $D = \mathbb{N}(\mu, \Sigma)$  for some unknown  $\mu, \Sigma$ . Consider the algorithm  $\mathcal{A}'$  where there is no truncation and no halting. Since  $I \preceq \Sigma \preceq \kappa \cdot I$ , standard concentration bounds (Lemmas 3.2 and 3.4) imply that the truncation and halting are not applied in  $\mathcal{A}$  with probability at least  $1 - \alpha / 2$ . Therefore, we have  $d_{\mathrm{tv}}(Q_{\mathcal{A},D}, Q_{\mathcal{A}',D}) \leq \alpha / 2$ .
Now, notice that the algorithm  $\mathcal{A}'$  just outputs  $Y := \frac{1}{n_1} \left( \sum_{i \in [n_1]} X_i \right) + \sqrt{1 - \frac{1}{n_1}} \cdot \left( \sum_{i \in [n_2]} a_i U_i \right)$ ; this output  $Y \sim \mathsf{N}(\mu, \Sigma)$ . In other words, we have  $Q_{\mathcal{A}', D} = \mathsf{N}(\mu, \Sigma)$ . Combining these, we can conclude that  $\mathcal{A}$  is  $\alpha$ -accurate.
C A Simpler but Worse Algorithm for Unknown Bounded Covariance
In this section, we give a simpler algorithm for the unknown bounded covariance setting. Its sample complexity bound of  $\tilde{O}\left(\frac{d^{3/2}}{\alpha\epsilon^2}\right)$  is worse than the one in Algorithm 2 and Theorem 4.3, but is still better than  $\Omega\left(\frac{d^2}{\alpha^2} + \frac{d^2}{\alpha\epsilon}\right)$  for DP learning [27] for constant  $\epsilon > 0$ .
The simpler sampler is given in Algorithm 3 and its guarantee is given in the theorem below. Note that this algorithm is once again just a form of (scaled) Gaussian mechanism, whereas the algorithm in the main body (Algorithm 2) adds noise that is input dependent.
Theorem C.1. Assume that  $I \preceq \Sigma \preceq \kappa \cdot I$  for some  $\kappa > 0$  and that  $\|\mu\| \leq R$ , there is an  $\alpha$ -accurate  $(\epsilon, \delta)$ -DP sampler with sample complexity
$$
n = O \left(\left(R ^ {2} + \kappa^ {2} d \left(\log \left(\frac {d}{\alpha \epsilon}\right) + \log \log (1 / \delta)\right)\right) \cdot \frac {d ^ {1 / 2}}{\alpha} \cdot \frac {\log (1 / \delta)}{\epsilon^ {2}}\right).
$$