SlowGuess/NeurIPS_2023_OCR huggingface dataset:
While the above overview seems intuitively plausible, there are certain difficulties that we need to overcome. First, the samples  $Y_{i}$  that our algorithm produces are not exactly drawn from  $\mathsf{N}(0,\Sigma)$ ; to fix this, we run an agnostic learner for Gaussians [e.g., 3] to recover the estimate of  $\Sigma$ . Second, since we run our DP sampler on the same  $n$  samples, the produced  $Y_{1},\ldots ,Y_{L}$  are not independent. We fix this by first drawing a larger number  $N$  of samples, and then produce each  $Y_{i}$  using  $n$ -out-of-  $N$  random samples; this reduces the correlation across the  $Y_{i}$ 's. Furthermore, using amplification-by-sampling of DP [6] gives us the desired privacy-vs-sample complexity lower bound guarantee.
2.3 Product Distributions: Algorithm
Our sampler follows the framework of [35], which is built upon the preconditioning procedure proposed by [27] for DP learning.
We start with a private preconditioner, which obtains a crude estimate of each  $p_j$  (up to a constant multiplicative factor). This is similar to that of [27], except that we use Laplace noise instead of Gaussian noise; this ensures that the resulting algorithm is pure-  $\mathrm{DP^6}$ . By suitably partitioning  $[0,1]$  into geometrically decreasing buckets in terms of  $d / \alpha$ , the goal then is to estimate  $p_j$  by placing it in one of these buckets. This can be done by an appropriate thresholding and the Laplace mechanism.
The next step is to obtain a refined estimate of the  $p_i$ 's using a fresh batch of samples. The algorithm then returns a sample randomly drawn from the product distribution given by these estimates. The earlier crude estimates are helpful in truncating and clipping the samples to make the produced sample DP, without adding any noise to the refined estimate.
For convenience, we use the notation  $\mathrm{trunc}_B^p (X)$  for "truncation" for all  $X\in \mathbb{R}^d,B > 0,p\geq 1$  ..
$$
\operatorname {t r u n c} _ {B} ^ {p} (X) := \left\{ \begin{array}{l l} X & \text {i f} \| X \| _ {p} \leq B, \\ X \cdot B / \| X \| _ {p} & \text {i f} \| X \| _ {p} > B. \end{array} \right. \tag {1}
$$
Let  $[k]$  denote  $\{1,\dots ,k\}$ . For any  $X\in \mathbb{R}^d$ , we use  $X[j]$  to denote the value of its  $j$ th coordinate.
3.1 Distributions and Tail Bounds
For a discrete distribution  $D$ , we use  $f_{D}$  to denote its probability mass function (PMF); for a continuous distribution  $D$ , we use  $f_{D}$  to denote its probability density function (PDF). Let  $\operatorname{supp}(D)$  denote the support of  $D$ . We let  $Z \sim D$  denote that the random variable  $Z$  is distributed according to  $D$ ; throughout, we may write the random variable in place of the distribution and vice versa when convenient. Finally, when  $D_{1}, \ldots, D_{d}$  are distributions, we use  $D_{1} \otimes \dots \otimes D_{d}$  as the product distribution, i.e., the distribution of  $(Z_{1}, \ldots, Z_{d})$  where  $Z_{1} \sim D_{1}, \ldots, Z_{d} \sim D_{d}$  are independent.
For a distribution  $D$  on  $\mathbb{R}^d$  and  $v\in \mathbb{R}^d$ , we write  $D + v$  as a shorthand for the distribution of  $X + v$  where  $X\sim D$ . Furthermore, for a distribution  $D$  and a (possibly randomized) function  $h$ , we write  $h(D)$  to denote the distribution of  $h(X)$  where  $X\sim D$ .
We will list a few distributions that will be useful for us.
<sup>6</sup>We remark that a similar analysis has also been done by [37], who uses such a pure-DP preconditioner to give pure-DP algorithms for learning product distributions.
(i) Shifted Truncated Discrete Laplace Distribution: For  $\Delta > 0$ , let  $s(\epsilon, \delta) = \lceil \Delta (1 + \log (1 / \delta) / \epsilon) \rceil$ . We define  $\mathsf{STLap}(\epsilon, \delta, \Delta)$  to be the discrete distribution supported on  $[-2s(\epsilon, \delta), 0]$  such that  $f_{\mathsf{STLap}(\epsilon, \delta, \Delta)}(x) \propto \exp(-\epsilon |x + s(\epsilon, \delta)|)$ .
It is known that adding STLap noise to a low-sensitivity function results in a DP estimate [e.g., 22].
Lemma 3.1. If  $g$  is a function with sensitivity  $\leq \Delta$ , then the algorithm that outputs  $g(\mathbf{X}) + \mathrm{STLap}(\epsilon, \delta, \Delta)$  is  $(\epsilon, \delta)$ -DP.
(ii) Beta Distribution: For  $\alpha, \beta > 0$ ,  $\operatorname{Beta}(\alpha, \beta)$  has the PDF  $f_{\operatorname{Beta}(\alpha, \beta)}(x) \propto x^{\alpha - 1}(1 - x)^{\beta - 1}$ .
(iii) Uniform Distribution over Unit Sphere: For  $d \in \mathbb{N}$ , let  $\mathrm{Uni}_d^{\mathbb{S}}$  denote the distribution of a random unit vector in  $\mathbb{R}^d$ .
(iv) Projection of Uniform Distribution over Unit Sphere: For any  $d \in \mathbb{N}$  and  $i \in [d]$  and  $z \in \mathbb{R}^d$ , let  $\Pi_{\leq i}(z)$  denote  $(z_1, \ldots, z_i)$ . Then, let  $\mathrm{Uni}_{d,i}^{\mathbb{S}}$  denote the distribution of  $\Pi_{\leq i}(Z)$  where  $Z \sim \mathrm{Uni}_d^{\mathbb{S}}$ . This distribution has the PDF (see, e.g., [34, Theorem 2]) given below.
$$
f _ {\operatorname {U n i} _ {d, i} ^ {\mathbb {S}}} (z) \propto \left\{ \begin{array}{l l} (1 - \| z \| ^ {2}) ^ {\frac {d - i}{2} - 1} & \text {i f} \| z \| ^ {2} <   1 \\ 0 & \text {o t h e r w i s e .} \end{array} \right. \tag {2}
$$
We need a tail bound on the  $\ell_2$ -norm of a Gaussian-distributed vector:
Lemma 3.2 ([42, Theorem 6.2]). There exists a constant  $c \geq 1$  such that, for any  $\mu \in \mathbb{R}^d$ ,  $\Sigma \in \mathbb{R}^{d \times d}$  where  $\Sigma \preceq \kappa \cdot I$  and any  $\beta \in (0, \frac{1}{2})$ ,  $\operatorname*{Pr}_{X \sim \mathsf{N}(\mu, \Sigma)}\left[\|X - \mu\|_2 > c\sqrt{\kappa}\left(\sqrt{d} + \sqrt{\log(1 / \beta)}\right)\right] \leq \beta$ .
We also need a tail bound for the beta distribution:
Lemma 3.3 ([43], Theorem 8). There exists a constant  $c \in (0,1)$  such that, for any  $0 < \alpha < \beta$  and any  $x \geq 0$ , we have  $\operatorname*{Pr}_{Z \sim \mathrm{Beta}(\alpha, \beta)} \left[ Z \geq \frac{\alpha}{\alpha + \beta} + x \right] \leq 2e^{-c \cdot \min \left\{ \frac{\beta^2 x^2}{\alpha}, \beta x \right\}}$ .
The following concentration bound on the empirical covariance will also be helpful.
Lemma 3.4 ([27, Fact 3.4]). There exists a constant  $c \geq 1$  such that for any  $\Sigma \succeq I$ , let  $U_1, \ldots, U_n \sim \mathsf{N}(0, \Sigma)$  and  $\hat{\Sigma} = \frac{1}{n} \sum_{i \in [n]} U_i U_i^T$ , we have  $\operatorname*{Pr}\left[\hat{\Sigma} \succeq \left(1 - c\sqrt{\frac{d + \log(1 / \beta)}{n}}\right) \cdot I\right] \geq 1 - \beta$ .
Hockey Stick Divergence. We also recall the definition of  $\epsilon$ -hockey stick divergence between two distributions  $P, Q$ :  $d_{\epsilon}(P||Q) \coloneqq \int_{y \in \operatorname{supp}(P)} [f_P(y) - e^{\epsilon} f_Q(y)]_+ dy$ , where  $[a]_+ \coloneqq \max \{a, 0\}$ .
The following standard fact about the hockey stick divergence is often useful in proving DP guarantees of algorithms [39, 31].
Lemma 3.5. For any  $\epsilon \geq 0$  and distributions  $P, Q$ ,  $d_{\epsilon}(P \parallel Q) \leq \operatorname{Pr}_{y \sim P}[f_P(y) > e^{\epsilon} f_Q(y)]$ .
It will be also useful to keep in mind the "post-processing" property of DP:
Lemma 3.6. For any distributions  $P, Q$  and any function  $h$ , we have  $d_{\epsilon}(h(P) \mid h(Q)) \leq d_{\epsilon}(P \mid Q)$ .
DP under Condition. Since we will use a "propose-test-release"-style algorithm [17], it will be convenient to use the notion of "DP under condition" together with its composition properties. The particular definition we use below is from [32]; similar notions have been used earlier, e.g., in [17].
Definition 3.7 (DP under Condition, [32]). Let  $\Psi : \mathcal{Y} \to \{0,1\}$  be a predicate. An algorithm  $M: \mathcal{Y} \to \mathcal{O}$  is  $(\epsilon, \delta)$ -DP under condition  $\Psi$  for  $\epsilon, \delta > 0$  iff, for every  $S \subseteq \mathcal{O}$  and every neighboring datasets  $Y, Y' \in \mathcal{Y}$  both satisfying  $\Psi$ , we have  $\operatorname*{Pr}[\mathcal{M}(Y) \in S] \leq e^{\epsilon} \cdot \operatorname*{Pr}[\mathcal{M}(Y') \in S] + \delta$ .
Lemma 3.8 (Composition for Algorithm with Halting, [32]). Let  $\mathcal{M}_1: \mathcal{Y} \to \mathcal{O}_1 \cup \{\bot\}, \mathcal{M}_2: \mathcal{O}_1 \times \mathcal{Y} \to \mathcal{O}_2$  be algorithms. Furthermore, let  $\mathcal{M}$  denote the following algorithm: Let  $o_1 = \mathcal{M}_1(Y)$  and, if  $o_1 = \bot$ , then halt and output  $\bot$  or else, output  $o_2 = \mathcal{M}_2(o_1,Y)$ .
Let  $\Psi$  be any condition such that, if  $Y$  does not satisfy  $\Psi$ , then  $\mathcal{M}_1(Y)$  always returns  $\perp$ . Suppose that  $\mathcal{M}_1$  is  $(\epsilon_1, \delta_1)$ - $DP$  and  $\mathcal{M}_2$  is  $(\epsilon_2, \delta_2)$ - $DP$  under condition  $\Psi$ . Then,  $\mathcal{M}$  is  $(\epsilon_1 + \epsilon_2, \delta_1 + \delta_2)$ - $DP$ .
4 Gaussian Distribution: Algorithms
Reduction to the Bounded Mean Case. As stated earlier, for the cases of known covariance and bounded covariance, we will need a preprocessing step that computes a rough private estimate for the mean. The properties of the reduction are stated below.
Lemma 4.1. Suppose that there is an  $\alpha$ -accurate  $(\epsilon, \delta)$ -DP sampler for Gaussian distributions under the assumption that  $I \preceq \Sigma \preceq \kappa \cdot I$ ,  $\|\mu\| \leq R$  with sample complexity  $n_{\mathrm{bm-sampler}}(\alpha, R, \epsilon, \delta)$ . Then, there exists an  $\alpha$ -accurate  $(\epsilon, \delta)$ -DP sampler for Gaussian distributions under the assumption that  $I \preceq \Sigma \preceq \kappa \cdot I$  with sample complexity  $\tilde{O}\left(\frac{\sqrt{d}}{\epsilon}\right) + n_{\mathrm{bm-sampler}}(\alpha/2, O(\kappa\sqrt{d}), \epsilon/2, \delta/2)$ .
4.1 Known Covariance
With the reduction in Lemma 4.1, we may assume that  $\| \Sigma^{-1 / 2}\mu \| \leq R$ . When the covariance is known, we give an algorithm with the following guarantees.
Theorem 4.2. Assuming  $\Sigma$  is known and  $\| \Sigma^{-1 / 2}\mu \| \leq R$ , there is an  $\alpha$ -accurate  $(\epsilon, \delta)$ -DP sampler for Gaussian distributions with sample complexity  $O\left(\left(R + \sqrt{d + \log\left(\frac{\log(1 / \delta)}{\alpha\epsilon}\right)}\right)\frac{\sqrt{\log(1 / \delta)}}{\epsilon}\right)$ .
We assume w.l.o.g. that  $\Sigma = I$ ; otherwise, we can consider  $X_{i}^{\prime} = \Sigma^{-1 / 2}X_{i}$ . Before we describe the algorithm, note that Theorem 4.2 together with Lemma 4.1 implies Theorem 1.2. As stated earlier, the algorithm (Algorithm 1) is simple: take the average of the truncated input samples and add to it (spherical) Gaussian noise.
Proof. Let  $C \geq 1$  be the constant from Lemma 3.2,  $B = R + 10^{4}C\sqrt{d + \log\left(\frac{2\log(2 / \delta)}{\alpha\epsilon}\right)}$ , and  $n = 1 + \lceil 10B\sqrt{\log(2 / \delta)} / \epsilon \rceil$ . Let  $\mathcal{A}$  be Algorithm 1 with  $B, n$  as specified and  $\sigma = \sqrt{(n - 1) / n}$ .
Privacy Analysis.  $\mathcal{A}$  is the Gaussian mechanism with noise multiplier  $n\sigma /B\geq 10\sqrt{\log(2 / \delta)} /\epsilon$  therefore,  $\mathcal{A}$  is  $(\epsilon ,\delta)$  -DP, using [20, Appendix A].
Accuracy Analysis. Let  $D = \mathsf{N}(\mu, I)$  for some unknown  $\mu$ . Consider the algorithm  $\mathcal{A}'$  where there is no truncation, i.e.,  $\mathcal{A}'$  simply outputs  $Y := Z + \frac{1}{n} \sum_{i \in [n]} X_i$ . Via Lemma 3.2 and a union bound, the truncation is not applied anyway in  $\mathcal{A}$  (i.e.,  $X_i = X_i^{\mathrm{trunc}}, \forall i \in [n]$ ) with probability at least  $1 - \alpha$ . Therefore,  $d_{\mathrm{tv}}(Q_{\mathcal{A},D}, Q_{\mathcal{A}',D}) \leq \alpha$ . Note that  $\mathcal{A}'$  just outputs  $Y := Z + \frac{1}{n} \sum_{i \in [n]} X_i$ , so we have  $Y \sim \mathsf{N}(\mu, I) = D$ , i.e.,  $Q_{\mathcal{A}',D} = D$ . Combining these bounds yields  $d_{\mathrm{tv}}(Q_{\mathcal{A},D}, D) \leq \alpha$ .
4.2 Unknown Bounded Covariance
We now move on to the case where both  $\mu, \Sigma$  are unknown but under the assumption  $I \preceq \Sigma \preceq \kappa \cdot I$ . The main result of this section is stated below<sup>7</sup>. Again, note that Theorem 4.3 and Lemma 4.1 immediately yield Theorem 1.4.
Theorem 4.3. Assuming  $I \preceq \Sigma \preceq \kappa \cdot I$  for some  $\kappa > 0$  and  $\| \mu \| \leq R$ , there is an  $\alpha$ -accurate  $(\epsilon, \delta)$ - $DP$  sampler for Gaussian distributions with sample complexity  $O\left(\left(R^2 + \kappa^2 \left(d + \log \left(\frac{\log(1 / \delta)}{\alpha \epsilon}\right)\right)\right) \cdot \frac{\log(1 / \delta)}{\epsilon}\right)$ .
As stated in the overview, the main challenge lies in the privacy analysis. It will be proved in three steps. First, in Section 4.2.1, we will show that if we add noise that is drawn from projection of random unit vector to the first  $M$  coordinates to a low-  $(\ell_2)$  -sensitivity function, then it is DP. Then, in Section 4.2.2, we use this to show that adding noise of the form  $\sum_{i\in [n]}a[i]\cdot w_i$  (where  $a\sim \mathrm{Un}\mathbf{i}_n^{\mathbb{S}}$ ) to a low-sensitivity function also suffices for privacy as long as the smallest eigenvalue of  $\sum_{i\in [n]}w_iw_i^T$  is sufficiently large. Here  $w_{1},\ldots ,w_{n}$  are assumed to be fixed vectors given beforehand. Note that the noises discussed so far are input independent. Finally, in Appendix B.4, we relate this to the
The dependence on  $\kappa$  can be reduced to  $\mathrm{polylog}(1 / \kappa)$  by applying the private preconditioning in [27], although this will increase the dependence on  $d$  to  $d^{1.5}$ ; it is an interesting question if this increase in the dependence on  $d$  can be avoided.
privacy of our algorithm—which is more intricate as these  $w_{i}$ 's are now  $X_{i}$ 's, i.e., the noise is input dependent. The accuracy analysis, which is simpler than the privacy analysis, is also in Appendix B.4.
4.2.1 Noise via Random Unit Vector Projection
We first show that adding noise drawn from  $\mathrm{Uni}_{M,N}^{\mathbb{S}}$  to a function with low  $\ell_2$ -sensitivity ensures DP.
Lemma 4.4. Let  $M, N$  be any positive integers such that  $M \geq \max \left\{2N, \frac{10^4 \log(10 / \delta)}{c\epsilon}\right\}$ , where  $c$  is the constant from Lemma 3.3. Furthermore, let  $\tau \in (0, 0.01)$  be such that  $\tau \leq 10\sqrt{\frac{\log(10 / \delta)}{cM}}$ . Then, for any vector  $v$  such that  $\| v \| \leq \tau$ , we have  $d_{\epsilon}(\mathsf{Unim}_{M,N}^{\mathbb{S}} \| \mathsf{Unim}_{M,N}^{\mathbb{S}} + v) \leq \delta$ .
The proof of this lemma proceeds similarly to those of independent noise addition (e.g., the Gaussian mechanism): using Lemma 3.5, it suffices to bound the probability that the privacy loss is large. We achieve such a bound via concentration properties of the beta distribution.
4.2.2 Noise via Spherical Linear Combinations
Next, we relate the previous lemma to privacy in the case where the noise added is of the form  $\sum_{i\in [q]}a[i]\cdot w_i$ , where  $w_{1},\ldots ,w_{q}$  are fixed beforehand and  $a\sim \mathsf{Uni}_q^{\mathbb{S}}$ . The guarantee of such a noise is stated below. The proof is based on a reduction argument to Lemma 4.4. For notational convenience, for every matrix  $W\in \mathbb{R}^{d\times q}$ , let  $\mathsf{Uni}_W^{\mathbb{S}}$  be the distribution of  $Wa$  where  $a\sim \mathsf{Uni}_q^{\mathbb{S}}$ .
Lemma 4.5. Let  $q, d \in \mathbb{N}$  and  $\tau_0 \in (0,1)$  be such that  $q \geq \max \left\{2d, \frac{10^4 \log(10 / \delta)}{c\epsilon} \right\}$  and  $\tau_0 \leq 5\sqrt{\log(10 / \delta) / c}$ . For any  $w_1, \ldots, w_q \in \mathbb{R}^d$  such that  $\frac{1}{q} \left( \sum_{i=1}^{q} w_i w_i^T \right) \succeq 0.5I$  and any  $v \in \mathbb{R}^d$  such that  $\|v\| \leq \tau_0$ , we have  $d_\epsilon(\mathrm{Unil}_W^{\mathbb{S}}) \| \mathrm{Unil}_W^{\mathbb{S}} + v) \leq \delta$ , where  $w_1, \ldots, w_q$  are the columns of  $W \in \mathbb{R}^{d \times q}$ .
Proof. Let  $W^{+} \in \mathbb{R}^{d \times q}$  denote the pseudoinverse of  $W$ . Note that  $WW^{T} = \sum_{i=1}^{q} w_{i} w_{i}^{T} \succeq 0.5qI$ . Therefore, all singular values of  $W$  are at least  $\sqrt{0.5q}$ . This means that all singular values of  $W^{+}$ are at most  $1/\sqrt{0.5q}$ . This in turn implies that  $\| W^{+}v \| \leq \| v \| / \sqrt{0.5q} \leq \tau_{0} / \sqrt{0.5q} =: \tau$ .
Since  $W^T W \succeq 0.5qI$ ,  $W^T$  is full rank and thus  $\dim(\operatorname{im}(W^+)) = d$ . Since  $d < q$ , pick any orthonormal basis  $b_1, \ldots, b_d \in \mathbb{R}^q$  for  $\operatorname{im}(W^+)$  and let  $\Pi: \mathbb{R}^q \to \mathbb{R}^d$  be such that  $\Pi(z) = (\langle z, b_1 \rangle, \ldots, \langle z, b_d \rangle)$ .
Furthermore, let  $h: \mathbb{R}^d \to \mathbb{R}^d$  be defined as  $h(x) := \sum_{i \in [d]} x[i] \cdot b_i$ . Observe that, for any  $a \in \mathbb{R}^q$ , we have  $a_1 u_1 + \dots + a_q u_q = h(\Pi(a))$  and  $a_1 u_1 + \dots + a_q u_q + v = h(\Pi(a) + \Pi(W^+ v))$ . Thus,
$$
\begin{array}{l} d _ {\epsilon} \left(\operatorname {U n i} _ {W} ^ {\mathbb {S}} \mid \mid \operatorname {U n i} _ {W} ^ {\mathbb {S}} + v\right) = d _ {\epsilon} \left(h \left(\Pi \left(\operatorname {U n i} _ {q} ^ {\mathbb {S}}\right)\right) \mid \mid h \left(\Pi \left(\operatorname {U n i} _ {q} ^ {\mathbb {S}}\right) + \Pi \left(W ^ {+} v\right)\right)\right) \\ \stackrel {\text {L e m m a 3 . 6}} {\leq} d _ {\epsilon} \left(\Pi \left(\operatorname {U n i} _ {q} ^ {\mathbb {S}}\right) \mid \mid \Pi \left(\operatorname {U n i} _ {q} ^ {\mathbb {S}}\right) + \Pi \left(W ^ {+} v\right)\right). \\ \end{array}
$$
Note that  $\Pi(\mathsf{Uni}_q^{\mathbb{S}}) \sim \mathsf{Uni}_{q,d}^{\mathbb{S}}$ . So, Theorem 4.4 implies  $d_{\epsilon}(\Pi(\mathsf{Uni}_q^{\mathbb{S}}) \mid \mid \Pi(\mathsf{Uni}_q^{\mathbb{S}}) + \Pi(W^{+}v)) \leq \delta$ .
5 Discussion and Open Questions
In this work, we give several algorithms for DP sampling. For Gaussian distributions, we demonstrate that the dependence on the accuracy parameter  $\alpha$  in the sample complexity can be only polylogarithmic in comparison to the polynomial dependence necessary in DP learning. Furthermore, we also reduce the dependence on  $d$  in the case of known variance and unknown bounded variance; in both cases, we show this dependence (of  $\sqrt{d}$  and  $d$  respectively) to be essentially tight. An immediate open question is whether the dependence of  $d^{1.5}$  in the case of unknown unbounded variance is tight. Lower bounds for Gaussian sampling in terms of  $\alpha$  is also an interesting question. Another—more general—direction is to extend the results to other families of distributions. In the case of DP learning, this has been explored by several works. Our algorithms very specifically use the property of Gaussian distributions (namely that a linear combination of Gaussians is another Gaussian); other distribution families might need different techniques.
Acknowledgment. We thank Shyam Narayanan for pointing us to [9], which improves dependency on  $d$  in the sample complexity for sampling Gaussians with unknown (unbounded) covariance from  $d^2$  to  $d^{1.5}$ .
[1] Martín Abadi, Andy Chu, Ian J. Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In CCS, pages 308-318, 2016.
[2] Ishaq Aden-Ali, Hassan Ashtiani, and Gautam Kamath. On the sample complexity of privately learning unbounded high-dimensional Gaussians. In ALT, pages 185-216, 2021.
[3] Hassan Ashtiani, Shai Ben-David, and Abbas Mehrabian. Sample-efficient learning of mixtures. In AAAI, pages 2679-2686, 2018.
[4] Hassan Ashtiani and Christopher Liaw. Private and polynomial time algorithms for learning Gaussians and beyond. In  $COLT$ , pages 1075-1076, 2022.
[5] Brian Axelrod, Shivam Garg, Vatsal Sharan, and Gregory Valiant. Sample amplification: Increasing dataset size even when learning is impossible. In ICML, pages 442-451, 2020.
[6] Borja Balle, Gilles Barthe, and Marco Gabordi. Privacy amplification by subsampling: Tight analyses via couplings and divergences. In NeurIPS, pages 6280-6290, 2018.
[7] Sourav Biswas, Yihe Dong, Gautam Kamath, and Jonathan R. Ullman. Coinpress: Practical private mean and covariance estimation. In NeurIPS, 2020.
[8] Gavin Brown, Marco Gaboardi, Adam D. Smith, Jonathan R. Ullman, and Lydia Zakynthinou. Covariance-aware private mean estimation without private covariance estimation. In NeurIPS, pages 7950-7964, 2021.
[9] Gavin Brown, Samuel Hopkins, and Adam Smith. Fast, sample-efficient, affine-invariant private mean and covariance estimation for subgaussian distributions. In  $COLT$ , pages 5578-5579, 2023.
[10] Mark Bun, Gautam Kamath, Thomas Steinke, and Zhiwei Steven Wu. Private hypothesis selection. IEEE Trans. Inf. Theory, 67(3):1981-2000, 2021.
[11] T. Tony Cai, Yichen Wang, and Linjun Zhang. The cost of privacy: Optimal rates of convergence for parameter estimation with differential privacy. Ann. Stat., 49(5):2825-2850, 2021.
[12] Damien Desfontaines, James Voss, Bryant Gipson, and Chinmoy Mandayam. Differentially private partition selection. PETS, 2022(1):339-352, 2022.
[13] Luc Devroye, Abbas Mehrabian, and Tommy Reddad. The total variation distance between high-dimensional Gaussians with the same mean. arXiv:1810.08693, 2018.
[14] Ilias Diakonikolas, Gautam Kamath, Daniel Kane, Jerry Li, Ankur Moitra, and Alistair Stewart. Robust estimators in high-dimensions without the computational intractability. SIAM J. Comput., 48(2):742-864, 2019.
[15] Wei Dong, Yuting Liang, and Ke Yi. Differentially private covariance revisited. In NeurIPS, 2022.
[16] Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. In EUROCRYPT, pages 486-503, 2006.
[17] Cynthia Dwork and Jing Lei. Differential privacy and robust statistics. In STOC, pages 371-380, 2009.
[18] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In TCC, pages 265-284, 2006.
[41] Vladimir V. Uchaikin and Vladimir M. Zolotarev. Chance and Stability. De Gruyter, Berlin, Boston, 1999.
[42] Roman Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science. Cambridge University Press, 2018.
[43] Anru R. Zhang and Yuchen Zhou. On the non-asymptotic and sharp lower tail bounds of random variables. Stat, 9(1):e314, 2020.
We consider a generalization of the truncation (1) where there is a weight vector  $w \in \mathbb{R}_{\geq 0}^{d}$ , and the truncation is w.r.t.  $w$ :
$$
\operatorname {t r u n c} _ {B, w} ^ {p} (X) := \left\{ \begin{array}{l l} X & \text {i f} \| X \circ w \| _ {p} \leq B, \\ X \cdot B / \| X \circ w \| _ {p} & \text {i f} \| X \circ w \| _ {p} > B, \end{array} \right.
$$
where  $\circ$  denotes the element-wise product. Clearly  $\mathrm{trunc}_B^p = \mathrm{trunc}_{B,w}^p$  when  $w$  is the all-ones vector.
For  $a, b \in \mathbb{R}$  where  $a \leq b$ , we also use  $\operatorname{clip}_{(a,b)}: \mathbb{R} \to \mathbb{R}$  to denote the function:
$$
\operatorname {c l i p} _ {(a, b)} (x) = \max  \{a, \min  \{b, x \} \}.
$$
A.1 Total Variation Distance between Gaussians
For our proofs, it will be useful to state the total variation distance bounds for Gaussian distributions. We will use the following notation:
Definition A.1. For any matrix  $A \in \mathbb{R}^{d \times d}$  and positive semi-definite matrix  $\Sigma \in \mathbb{R}^{d \times d}$ , we write  $\|A\|_{\Sigma} := \| \Sigma^{-1/2} A \Sigma^{-1/2}\|_F$ , where  $\| \cdot \|_F$  denotes the Frobenius norm.
It is well known that when  $\| \Sigma -\hat{\Sigma}\|_{\Sigma}\leq O(1)$ , the total variation distance between Gaussian with covariance matrices  $\Sigma$  and  $\hat{\Sigma}$  and the same mean is  $\Theta (\| \Sigma -\hat{\Sigma}\|_{\Sigma})$ , as stated more formally below.
Lemma A.2 ([13]). There exists a constant  $C \geq 1$  such that the following holds. For any  $\Sigma, \hat{\Sigma}$ , if  $d_{\mathrm{tv}}(\mathsf{N}(0,\Sigma), \mathsf{N}(0,\hat{\Sigma})) \leq 0.001$ , then  $\| \Sigma - \hat{\Sigma} \|_{\Sigma} \leq C \cdot d_{\mathrm{tv}}(\mathsf{N}(0,\Sigma), \mathsf{N}(0,\hat{\Sigma}))$ .