SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 3 BOUNDEDCOVARIANCEGAUSSIANSAMPLEER
Parameters:  $B,\sigma >0$  , and  $n_1,n_2\in \mathbb{N}$    
Sample  $X_{1},\ldots ,X_{n_{1}},X_{n_{1} + 1},\ldots ,X_{n_{1} + 2n_{2}}\sim D$    
for  $i = 1,\dots ,n_1 + 2n_2$  do  $X_{i} = \mathrm{trunc}_{B}^{2}(X_{i})$    
Sample  $Z\sim \mathsf{N}(0,\sigma^2 I)$    
return  $\frac{1}{n_1}\left(\sum_{i\in [n_1]}X_i\right) + \sqrt{\frac{1 - 1 / n_1}{2n_2}}\left(\sum_{i\in [n_2]}(X_{n_1 + 2i - 1} - X_{n_1 + 2i})\right) + Z$
Proof Sketch of Theorem C.1. Let  $B = R + 10^4 \kappa \sqrt{d\left(\log\left(\frac{2d}{\alpha\epsilon}\right) + \log \log(2 / \delta)\right)}$ ,  $\sigma = \frac{\sqrt{\alpha}}{2d^{1/4}}$  and  $n_1, n_2$  be such that  $n_1 = n_2 \geq \frac{100B^2 \log(2 / \delta)}{\sigma^2 \epsilon^2}$ . Let  $\mathcal{A}$  denote Algorithm 3 with parameters  $B, n, \sigma$  as specified.
Privacy Analysis.  $\mathcal{A}$  is the Gaussian mechanism with noise multiplier  $\frac{\sigma}{B\cdot\sqrt{\frac{1 - 1 / n_1}{2n_2}}}\geq 10\sqrt{\log(2 / \delta) / \epsilon}$  by the setting of our parameters; therefore,  $\mathcal{A}$  is  $(\epsilon ,\delta)$ -DP.
Accuracy Analysis. To understand the accuracy guarantee, let  $D = \mathsf{N}(\mu, \Sigma)$  for some unknown  $\mu, \Sigma$  where  $I \preceq \Sigma \preceq \kappa \cdot I$ . Let us consider the algorithm  $\mathcal{A}'$  where there is no truncation. Again, a standard concentration bound implies that the truncation is not applied anyway in  $\mathcal{A}$  with probability at least  $1 - \alpha / 2$ . Therefore, we have
$$
d _ {\mathrm {t v}} \left(Q _ {\mathcal {A}, P}, Q _ {\mathcal {A} ^ {\prime}, P}\right) \leq \alpha / 2.
$$
Now, notice that in algorithm  $\mathcal{A}'$ , we just output
$$
Y := \frac {1}{n _ {1}} \left(\sum_ {i \in [ n _ {1} ]} X _ {i}\right) + \sqrt {\frac {1 - 1 / n _ {1}}{2 n _ {2}}} \left(\sum_ {i \in [ n _ {2} ]} \left(X _ {n _ {1} + 2 i - 1} - X _ {n _ {1} + 2 i}\right)\right) + Z.
$$
Without truncation, this output  $Y$  has identical distribution as  $\mathsf{N}(\mu, \Sigma + \sigma^2 I)$ . In other words, we have
$$
d _ {\mathrm {t v}} (Q _ {\mathcal {A} ^ {\prime}, P}, \mathsf {N} (\mu , \Sigma)) = d _ {\mathrm {t v}} (\mathsf {N} (\mu , \Sigma + \sigma^ {2} I), \mathsf {N} (\mu , \Sigma)) \leq \| \sigma^ {2} I \| _ {\Sigma} \leq \| \sigma^ {2} I \| _ {F} = \sigma^ {2} \cdot \sqrt {d} \leq \alpha / 2,
$$
where the first inequality follows from Lemma A.3 and the last inequality follows from our parameter selection. Combining the above two inequalities, we can conclude that the sampler is  $\alpha$ -accurate as desired.
D Gaussian Distributions: Lower Bounds
D.1 Known Covariance
In this section, we prove a lower bound of  $\Omega (\sqrt{d} /\epsilon)$  that holds even for the simplest case of known covariance (Theorem 1.3), showing that the dependence on  $d$  in our sampler (Theorem 4.2) is nearly optimal.
We prove this by using a DP sampler to draw a large, but constant, number of samples and the use them to perform mean estimation; the lower bound for mean estimation (Theorem A.6) then gives the desired lower bound for the sample complexity of DP sampler.
Proof of Theorem 1.3. Given an  $\alpha$ -accurate  $(\epsilon, \delta)$ -DP sampler  $\mathcal{A}$ , we construct an algorithm  $\mathcal{M}$  for mean estimation as follows:
- For  $i = 1, \dots, 10^6$ :
- Run  $\mathcal{A}$  on  $n$  fresh samples from  $D$  to get  $Y_{i}$ .
Output  $\hat{\mu}\in \mathbb{R}^d$  with  $\hat{\mu} [j]\coloneqq \mathrm{clip}_{-1,1}(\mathrm{median}(Y_1[j],\dots ,Y_{10^6}[j]))$
Since  $\mathcal{A}$  is  $\alpha$ -accurate, each  $Y_{i}$  is sampled from a distribution  $D'$  that is  $\alpha$ -close (in total variation distance) to  $D = \mathsf{N}(\mu, I)$ . For  $\alpha = 0.1$ , this means that
$$
\Pr \left[ Y _ {i} (j) \leq \mu - 0. 3 \right] \leq \Phi (- 0. 3) + \alpha \leq 0. 4 9,
$$
$$
\Pr \left[ Y _ {i} (j) \geq \mu + 0. 3 \right] \leq (1 - \Phi (0. 3)) + \alpha \leq 0. 4 9.
$$
As a result, standard concentration bounds (e.g., Theorem E.1) imply that
$$
\Pr \left[ \operatorname {m e d i a n} \left(Y _ {1} (j), \dots , Y _ {1 0 ^ {\circ}} (j)\right) \in \left[ \mu - 0. 3, \mu + 0. 3 \right] \right] > 0. 9 9.
$$
$$
\mathbb {E} _ {\mathbf {X} \sim D ^ {N}, \hat {\mu} \sim \mathcal {M} (\mathbf {X})} [ \| \hat {\mu} - \mu \| _ {2} ^ {2} ] \leq d \cdot (0. 0 1 \cdot 2 ^ {2} + 0. 9 9 \cdot 0. 3 ^ {2}) \leq d / 6.
$$
Applying Theorem A.6 with  $\gamma = \sqrt{d / 6}$ , we can conclude that the sample complexity of  $\mathcal{M}$  (which is equal to  $10^{6}n$ ) must be at least  $\Omega\left(\frac{d}{\epsilon\sqrt{d / 6}}\right) = \Omega(\sqrt{d} / \epsilon)$ . Thus, we must have  $n \geq \Omega(\sqrt{d} / \epsilon)$  as claimed.
D.2 Unknown Bounded Covariance
Next, we will prove the lower bound for the unknown bounded covariance case (Theorem 1.5).
D.2.1 Reduction to Covariance Estimation
Algorithm 4 COVARIANCEESTIMATOR
Parameters:  $N,n,L\in \mathbb{N}$  , sampler  $\mathcal{A}_{\mathrm{sympler}}$  , agnostic learner Alearner for centered Gaussians Sample  $X_{1},\ldots ,X_{N}\sim P$    
for  $\ell = 1,\dots ,L$  do Randomly draw  $n$  subsamples  $X_{i_1^{\ell}},\ldots ,X_{i_n^{\ell}}$  without replacement from  $X_{1},\ldots ,X_{N}$ $Y_{\ell}\gets \mathcal{A}_{\mathrm{sympler}}(X_{i_1^\ell},\ldots ,X_{i_n^\ell})$ $\mathsf{N}(0,\hat{\Sigma})\leftarrow \mathcal{A}_{\mathrm{learner}}(Y_1,\dots ,Y_L)$    
if  $0.5I\succeq \hat{\Sigma}\succeq 2.5I$  then  $\begin{array}{rl}{\hat{\Sigma}^{\prime}} & {\leftarrow \hat{\Sigma}} \end{array}$    
else  $\begin{array}{rl}{\hat{\Sigma}^{\prime}} & {\leftarrow I} \end{array}$    
return  $\hat{\Sigma}^{\prime}$
As stated earlier in Section 2, we will prove this lower bound by reducing to covariance estimation (Theorem A.7). This is done by first taking  $N \gg n$  samples, and then generating each  $Y_{\ell}$  by subsampling the input to  $n$  samples and running our DP sampler. These  $Y_{\ell}$ 's are then feed into the agnostic learner to produce an estimate  $\hat{\Sigma}$  for  $\Sigma$ . The full reduction is given in Algorithm 4. Note that here  $\mathcal{A}_{\mathrm{sampler}}$  will be the  $(\epsilon, \delta)$ -DP sampler and  $\mathcal{A}_{\mathrm{learner}}$  will be the learner from Theorem A.8.
Privacy Analysis. The privacy guarantee of Algorithm 4 follows easily from the amplification by subsampling and advanced composition of DP. This is formalized below.
Lemma D.1. For any  $\epsilon^{*} \in (0,1], \delta^{*} \in (0,1)$ , if  $\mathcal{A}_{\mathrm{sampler}}$  is  $(\epsilon, \delta)$ - $DP$  for  $\epsilon \leq \min \left\{1, \frac{\epsilon^{*}N}{4n\sqrt{L\ln(2/\delta^{*})}}\right\}$ ,  $\delta \leq \left(\frac{0.5N}{Ln}\right) \delta^{*}$ , then Algorithm 4 is  $(\epsilon^{*}, \delta^{*})$ - $DP$ .
Proof. First, we apply Theorem A.4 which means that computing a single  $Y_{\ell}$  is  $(\epsilon', \delta')$ -DP where
$$
\epsilon^ {\prime} = \ln (1 + (n / N) (e ^ {\epsilon} - 1)),
$$
$$
\delta^ {\prime} = (n / N) \cdot \delta .
$$
Then, applying Theorem A.5 with  $\delta_1 = 0.5\delta^*$  implies that all  $(Y_{1},\ldots ,Y_{L})$  together is  $(\epsilon_{1},L\delta^{\prime} + \delta_{1})$  DP where
$$
\epsilon_ {1} = \left(\sqrt {2 L \ln \left(1 / \delta_ {1}\right)} + L \left(e ^ {\epsilon^ {\prime}} - 1\right)\right) \epsilon^ {\prime}.
$$
Combining the above expressions, we have
$$
\epsilon_ {1} = \left(\sqrt {2 L \ln (2 / \delta^ {*})} + L (n / N) \left(e ^ {\epsilon} - 1\right)\right) \ln (1 + (n / N) \left(e ^ {\epsilon} - 1\right))
$$
$$
\begin{array}{l} (\text {f r o m} \epsilon \leq 1) \leq \left(\sqrt {2 L \ln (2 / \delta^ {*})} + L (n / N) (2 \epsilon)\right) \ln (1 + (n / N) (2 \epsilon)) \\ \leq \left(\sqrt {2 L \ln (2 / \delta^ {*})} + L (n / N) (2 \epsilon)\right) \cdot ((n / N) (2 \epsilon)) \\ \end{array}
$$
$$
\left(\operatorname {f r o m} \epsilon \leq \frac {\epsilon^ {*} N}{4 n \sqrt {L \ln (2 / \delta^ {*})}}\right) \leq \epsilon^ {*},
$$
and  $L\delta' + \delta_1 \leq \delta^* / 2 + \delta^* / 2 \leq \delta^*$ . In other words,  $(Y_1, \ldots, Y_L)$  together is  $(\epsilon^*, \delta^*)$ -DP. Finally, applying  $\mathcal{A}_{\mathrm{learner}}$  on  $(Y_1, \ldots, Y_L)$  is simply a post-processing step. Thus, the entire algorithm is  $(\epsilon^*, \delta^*)$ -DP as desired.
D.2.2 Accuracy Analysis
Next, we argue that the accuracy of the covariance estimation algorithm, assuming the accuracy of the sampler and the agnostic learner.
Lemma D.2. Let  $\xi \in (0, 0.01]$  and  $n, N, L \in \mathbb{N}$  be such that  $N \geq \left(\frac{10nLd}{\xi}\right)^2$ , and suppose that  $I \preceq \Sigma \preceq 2I$ . Furthermore, suppose that
-  $\mathcal{A}_{\mathrm{sympler}}$  is an  $\left(\frac{\xi}{10C}\right)$ -accurate sampler for the class of Gaussians under the assumption  $I \preceq \Sigma \preceq 2I$ , and
-  $\mathcal{A}_{\mathrm{learner}}$  is an  $\left(\frac{\xi}{10C}, \frac{\xi^2}{200d^2}\right)$ -accurate agnostic learner for the class of centered Gaussians,
where  $C$  is the constant in Lemma A.2. Then,  $\mathbb{E}[\|\hat{\Sigma}' - \Sigma\|_{\Sigma}^2] \leq \xi^2$  where  $\hat{\Sigma}'$  denotes the output of Algorithm 3.
Proof. Let  $D = \mathsf{N}(0, \Sigma)$  denote the underlying distribution and for notational convenience, let  $\zeta := \left(\frac{\xi}{10C}\right)$ . Let  $\mathcal{E}_{\mathrm{disjoint}}$  denote the event that  $i_1^1, \ldots, i_n^L$  are all different. Note that we have
$$
\Pr \left[ \neg \mathcal {E} _ {\text {d i s j o i n t}} \right] \leq \sum_ {\ell \neq \ell^ {\prime} \in [ L ], j, j ^ {\prime} \in [ n ]} \Pr \left[ i _ {j} ^ {\ell} = i _ {j ^ {\prime}} ^ {\ell^ {\prime}} \right] = \sum_ {\ell \neq \ell^ {\prime} \in [ L ], j, j ^ {\prime} \in [ n ]} \frac {1}{N ^ {2}} \leq \frac {L ^ {2} n ^ {2}}{2 N} \leq \frac {\xi^ {2}}{2 0 0 d ^ {2}},
$$
where the last inequality follows from our choice of parameters.
Next, suppose that  $\mathcal{E}_{\mathrm{disjoint}}$  holds. Then, we have that  $Y_{1},\ldots ,Y_{L}$  are independently sampled from  $Q_{\mathcal{A}_{\mathrm{sampler}},D}$ . Applying the agnostic learning guarantee of  $\mathcal{A}_{\mathrm{learner}}$ , with probability  $1 - \frac{\xi^2}{200d^2}$ , we get
$$
d _ {\mathrm {t v}} \left(\mathsf {N} (0, \hat {\Sigma}), Q _ {\mathcal {A} _ {\text {s a m p l e r}}, D}\right) \leq 3 \cdot d _ {\mathrm {t v}} \left(\mathsf {N} (0, \Sigma), Q _ {\mathcal {A} _ {\text {s a m p l e r}}, D}\right) + \zeta .
$$
Recall also from the accuracy guarantee of the sampler that  $d_{\mathrm{tv}}(\mathsf{N}(0,\Sigma),Q_{\mathcal{A}_{\mathrm{sampler},D}})\leq \zeta$
Combining all of these together we have
$$
\Pr \left[ d _ {\mathrm {t v}} \left(\mathsf {N} (0, \hat {\Sigma}), Q _ {\mathcal {A} _ {\text {s a m p l e r}}, D}\right) > 4 \zeta \right] \leq 1 - \frac {\xi^ {2}}{1 0 0 d ^ {2}}.
$$
Applying Lemma A.2, we get
$$
\operatorname * {P r} [ \| \hat {\Sigma} - \Sigma \| _ {\Sigma} > \xi / 2 ] \leq 1 - \frac {\xi^ {2}}{1 0 0 d ^ {2}}.
$$
Notice that if  $\| \hat{\Sigma} - \Sigma \|_{\Sigma} \leq \xi / 2$ , then we have  $\hat{\Sigma}' = \hat{\Sigma}$ . Furthermore,  $I \preceq \Sigma \preceq 2I$  and  $0.5I \preceq \hat{\Sigma}' \preceq 2.5I$  imply that  $\| \hat{\Sigma}' - \Sigma \|_{\Sigma} \leq 6d$ . Thus, we have
$$
\mathbb {E} [ \| \hat {\Sigma} ^ {\prime} - \Sigma \| _ {\Sigma} ^ {2} ] \leq (\xi / 2) ^ {2} + (6 d) ^ {2} \cdot \Pr [ \| \hat {\Sigma} - \Sigma \| _ {\Sigma} > \xi / 2 ] \leq (\xi / 2) ^ {2} + (6 d) ^ {2} \cdot \frac {\xi^ {2}}{1 0 0 d ^ {2}} \leq \xi^ {2}.
$$
D.2.3 Putting Things Together
Combing the privacy and accuracy guarantees and plugging in the appropriate parameters immediately yields Theorem 1.5.
Proof of Theorem 1.5. Let  $\alpha = \xi / (10C)$  where  $\xi$  is as in Theorem A.7 and  $C$  is as in Lemma A.2. Suppose for the sake of contradiction that there exists an  $(\epsilon, \delta)$ -DP  $\alpha$ -accurate sampler with sample complexity  $n = o\left(\frac{d}{\epsilon \sqrt{\log d}}\right)$  under the assumption  $I \preceq \Sigma \preceq 2I$ . Then, let us select the parameters as follows:
-  $L \leq O\left(d^{2}\right)$  denote the sample complexity of the  $\left(\frac{\xi}{10C}, \frac{\xi^2}{200d^2}\right)$ -accurate agnostic learner for the class of centered Gaussians as given by Theorem A.8.
-  $N = \left\lceil \left(\frac{10nLd}{\xi}\right)^2\right\rceil \leq O\left(n^2 d^2\log d\right)$ .
-  $\delta^{*} = O\left(\min \left\{1 / N, d^{2} / (N \log N)\right\}\right)$ . Note that, when the constant in big-O notation is sufficiently large, our choice of parameters implies that  $\delta \leq \left(\frac{0.5N}{Ln}\right) \delta^{*}$ .
$\epsilon^{*} = \frac{4n\epsilon\sqrt{L\ln(2 / \delta^{*})}}{N} = o(d^{2} / N).$
By Lemma D.2, we have that  $\mathbb{E}[\|\hat{\Sigma}' - \Sigma\|_{\Sigma}^2] \leq \xi^2$ . Furthermore, by Lemma D.1, we have that the algorithm COVARIANCEESTIMATOR is  $(\epsilon^*, \delta^*)$ -DP. However, we also have  $N = o(d^2 / \epsilon^*)$ , which contradicts Theorem A.7.
E Product Distributions on  $\{0,1\}^d$
In this section, we describe and analyze our sampler for product distributions on  $\{0,1\}^d$  (Theorem 1.7). We may assume w.l.o.g. that  $p_1,\ldots ,p_d\leq 3 / 4$ . (Otherwise, we first privately estimate  $p_i$  using, e.g., the Laplace mechanism, and flip the  $i$ th bit in all samples if the estimate is more than  $3 / 4$ .)
Given a dataset element  $X_{i} \in \{0,1\}^{d}$ , we use  $X_{i}[j]$  to denote its  $j$ th coordinate, and  $X_{i}[S] = (X_{i}[j])_{j \in S}$  to denote the vector  $X_{i}$  restricted to the subset  $S \subseteq [d]$  of coordinates.
E.1 Additional Preliminaries
Here we list a few concentration inequalities that are useful. We start with the following version of the Bernstein inequality [see, e.g., 42, Theorem 2.8.4]:
Theorem E.1 (Bernstein's inequality). Let  $Y_{1}, \ldots, Y_{n}$  be independent real-valued random variables such that, with probability 1,  $Y_{i} \in [0, C]$  for all  $i \in [n]$ . Let  $V = \sum_{i \in [n]} \operatorname{var}(Y_{i})$ . Then, for any  $\Delta \geq 0$ , we have
$$
\operatorname * {P r} \left[ \left| \sum_ {i \in [ n ]} Y _ {i} - \sum_ {i \in [ n ]} \mathbb {E} [ Y _ {i} ] \right| > \Delta \right] \leq 2 \exp \left(\frac {- \Delta^ {2}}{2 V + C \Delta}\right).
$$
We now list a couple of versions of this inequality, which will be more useful in our setting.
Corollary E.2. Let  $P$  be a product distribution on  $\{0,1\}^d$ . For any  $\beta, \gamma \in (0,\frac{1}{2})$ , let  $n > \frac{50}{\gamma}\log \frac{d}{\beta}$  and  $X_{1},\ldots,X_{n} \sim P$ . Let  $\overline{X} := \frac{1}{n}\sum_{i\in [n]}X_{i}$ . Then, we have
$$
\Pr \left[ \forall j \in [ d ], \bar {X} [ j ] \in [ 0. 9 p _ {j} - \gamma , 1. 1 p _ {j} + \gamma ] \right] \geq 1 - \beta .
$$
Proof. For each  $j \in [d]$ , applying Theorem E.1 with  $\Delta = \max \{\gamma, 0.1p_j\} \cdot n$  and using  $p_j \leq 3/4$ , we obtain
$$
\operatorname * {P r} \left[ | \overline {{X}} [ j ] - p _ {j} | \leq \max  \{\gamma , 0. 1 p _ {j} \} \right] \leq 2 \exp \left(\frac {- \Delta^ {2}}{2 n p _ {j} + \Delta}\right) \leq 2 \exp \left(- \frac {\Delta}{2 1}\right) \leq \frac {\beta}{d}.
$$
Taking a union bound over all  $j \in [d]$  completes the proof.
Corollary E.3. Let  $P$  be a product distribution on  $\{0,1\}^d$ . For any  $\beta, \gamma \in (0,\frac{1}{2})$ , let  $n > \frac{50}{\gamma}\log \frac{d}{\beta}$  and  $X_{1},\ldots ,X_{n}\sim P$ . Then, we have
$$
\operatorname * {P r} \left[ \forall i \in [ n ], \sum_ {j \in [ d ]} \frac {X _ {i} [ j ]}{\max \{p _ {j} , \gamma \}} \leq 3 d + \frac {4}{\gamma} \log \frac {n}{\beta} \right] \geq 1 - \beta .
$$
Proof. Denote  $Y_{j} = X_{i}[j] / \max \{p_{j},\gamma \}$  for all  $j\in [d]$ . Here we have  $\mathbb{E}[Y_j] = p_j / \max \{p_j,\gamma \} \leq 1$  and  $\mathrm{var}(Y_j)\leq p_j / \max \{p_j,\gamma \}^2\leq 1 / \gamma$ . Moreover, with probability 1, we have  $Y_{j}\leq 1 / \gamma$ . As a result, we can apply Theorem E.1 with  $\Delta = 2d + 4\log (n / \beta) / \gamma$ , which gives
$$
\Pr \left[ \left| \sum_ {j \in [ d ]} \frac {X _ {i} [ j ]}{\operatorname* {m a x} \{p _ {j} , \gamma \}} - d \right| > \Delta \right] \leq 2 \exp \left(\frac {- t ^ {2}}{2 d / \gamma + t / \gamma}\right) \leq 2 \exp (- 0. 5 \Delta \gamma) \leq \frac {\beta}{n}.
$$
Taking a union bound over all  $i \in [n]$  completes the proof.
E.2 Private Preconditioner
We start with the following private preconditioner, which estimates each  $p_j$  up to a constant factor. This preconditioner is similar to that of [27], except that we add noise from the Laplace distribution<sup>13</sup> (instead of Gaussian noise) to achieve pure-DP and that we are looking for a coarser guarantee compared to [27].
Lemma E.4. There exists an  $\epsilon$ -DP algorithm that takes
$$
O \left(\left(d \log \left(\frac {d}{\alpha}\right) + \frac {d}{\alpha}\right) \frac {\log \left(\frac {d}{\alpha \beta}\right)}{\epsilon}\right),
$$
samples as input and output integers  $\ell_1, \ldots, \ell_d \in \{0, \ldots, \lceil \log \left(\frac{2d}{\alpha}\right) \rceil \}$  such that, with probability  $1 - \beta$ , the following hold for all  $j \in [d]$ :
- if  $\ell_j \neq \left\lceil \log \left(\frac{2d}{\alpha}\right) \right\rceil$ , then  $p_j \in [1/4 \cdot 2^{-\ell_j}, 3/4 \cdot 2^{-\ell_j}]$ , and
- if  $\ell_{j} = \lceil \log \left(\frac{2d}{\alpha}\right)\rceil$ , then  $p_j\leq \alpha /2d$
In the remaining of this section, we will focus on the proof of Lemma E.4 and Algorithm 5.
Sample Complexity. From Algorithm 5, the total sample complexity can be bounded by
$$
\sum_ {\ell \in [ L ]} n _ {\ell} = O \left(\sum_ {\ell \in [ L ]} (d + 2 ^ {\ell}) \frac {\log \left(\frac {d}{\alpha \beta}\right)}{\epsilon}\right) = O \left(\left(d \log \left(\frac {d}{\alpha}\right) + \frac {d}{\alpha}\right) \frac {\log \left(\frac {d}{\alpha \beta}\right)}{\epsilon}\right),
$$
Privacy Analysis. In each iteration  $\ell \in [L]$ , the samples  $X_{1}^{\ell},\ldots ,X_{n_{\ell}}^{\ell}$ , after truncation, are noised with a Laplace distribution; by the privacy guarantee of the Laplace mechanism, each iteration is thus  $\epsilon$ -DP. Furthermore, since each iteration uses a fresh set of samples, by the parallel composition property of DP, the entire algorithm is  $\epsilon$ -DP as desired.
Accuracy Analysis. We will prove this by induction. Specifically, let  $S_{\ell}^{\uparrow} \coloneqq \{j \in [d] \mid p_j \leq 3/4 \cdot 2^{-\ell}\}$  and  $S_{\ell}^{\downarrow} \coloneqq \{j \in [d] \mid p_j \leq 1/4 \cdot 2^{-\ell}\}$ . We will show that
$$
\Pr \left[ \forall \ell \in \{0, \dots , t \}, S _ {\ell - 1} ^ {\downarrow} \subseteq S _ {\ell} \subseteq S _ {\ell} ^ {\uparrow} \right] \geq 1 - \frac {t \beta}{L}, \tag {6}
$$
for all  $t \in [L]$  by an induction on  $t$ . Observe that this statement implies the desired accuracy claim in the lemma because  $j$  gets assigned  $\ell_j = \ell$  if and only if  $j \in S_\ell \setminus S_{\ell+1}$ . (Here we use the convention that  $S_{L+1} = \emptyset$ .)
13The Laplace distribution  $\operatorname {Lap}(b)$  is given by the PDF  $f_{\mathrm{Lap}(b)}(x)\propto \exp (-|x| / b)$
Algorithm 5 PRECONDITIONERPRODUCTDIST
Input: An unknown product distribution  $P$  over  $\{0,1\}^d$ , privacy parameter  $\epsilon$ , accuracy parameter  $\alpha$ , failure probability parameter  $\beta$
$$
\text {O u t p u t}: \ell_ {1}, \dots , \ell_ {d} \in \{0, 1, \dots , \lceil \log (\frac {2 d}{\alpha}) \rceil \}
$$
$$
L \leftarrow \lceil \log \left(\frac {2 d}{\alpha}\right) \rceil
$$
$$
\ell_ {j} \leftarrow 0 \text {f o r e v e r y} j \in [ d ]
$$
$$
S _ {0} \leftarrow [ d ]
$$
foreach  $\ell = 0,1,\dots ,L - 1$  do
Base Case. (2) trivially holds for  $t = 0$  as  $S_0^\uparrow = S_0 = [d]$ .
Inductive Step. Suppose that (6) holds for  $t - 1$  for some  $t \in \mathbb{N}$ . We will also show that it also holds for  $t$ . From a union bound, it suffices to show
$$
\Pr \left[ S _ {t - 1} ^ {\downarrow} \subseteq S _ {t} \subseteq S _ {t} ^ {\uparrow} \mid S _ {t - 1} \subseteq S _ {t - 1} ^ {\uparrow} \right] \geq 1 - \frac {\beta}{L}.
$$
To show this, we need the following claim:
Claim 1. Assume  $S_{t-1} \subseteq S_{t-1}^{\uparrow}$ . For each  $j \in S_t$ , with probability  $1 - \frac{0.5\beta}{Ld}$ , we have
$$
\frac {1}{n _ {t}} \left(\sum_ {i \in [ n _ {t} ]} \operatorname {t r u n c} _ {B _ {t}} ^ {1} \left(X _ {i} ^ {t} [ S _ {t} ]\right)\right) _ {j} \in \left[ 0. 8 p _ {j} - 0. 0 1 \cdot 2 ^ {- t}, 1. 1 p _ {j} + 0. 0 1 \cdot 2 ^ {- t} \right].
$$
Before we prove Claim 1, let us see how to use it to finish the proof. Let  $Z^t \sim \mathrm{Lap}(B_t / \epsilon)$  be the Laplace noise added. First, by a standard concentration bound for the Laplace distribution, we have
$$
\Pr \left[ | Z _ {j} ^ {t} | <   \frac {B _ {t}}{\epsilon} \log \left(\frac {4 L d}{\beta}\right) \right] \geq 1 - \frac {0 . 5 \beta}{L d}.
$$
Note also that by our choice of parameters, we have  $B_{t} / \epsilon \cdot \log (4Ld / \beta)\leq 0.01\cdot 2^{-t}\cdot n_{t}$
Hence, by a union bound, we have that with probability  $1 - \beta / L$  for all  $j \in S_t$ ,
$$
q _ {t} [ j ] \in \left[ 0. 8 p _ {j} - 0. 0 2 \cdot 2 ^ {- t}, 1. 1 p _ {j} + 0. 0 2 \cdot 2 ^ {- t} \right].
$$
Consider any  $j \in S_t \setminus S_t^\uparrow$ . Since  $p_j \geq 0.75 \cdot 2^{-t}$ , we have  $q_t[j] \geq 0.58 \cdot 2^{-t} > \tau_t$ . Hence,  $j$  will not be included in  $S_t$ , coming to a contradiction. In other words,  $S_t \setminus S_t^\uparrow = \emptyset$ , i.e.,  $S_t \subseteq S_t^\uparrow$ . Similarly, consider any  $j \in S_{t-1}^\downarrow$ . Since  $p_j \leq 0.25 \cdot 2^{-t+1}$ , we have  $q_t[j] \leq 0.57 \cdot 2^{-t} = \tau_t$ . By Algorithm 5,  $j$  will be included in  $S_t$ . Hence,  $S_{t-1}^\downarrow \subseteq S_t$ . These conclude the inductive step.
We are now left to prove Claim 1.
Proof of Claim 1. Let  $Y_{i} = \left(\mathrm{trunc}_{B_{t}}^{1}\big(X_{i}^{t}[S_{t}]\big)\right)_{j}$ .
First note that  $\sum_{i\in [n_t]}Y_i\leq \sum_{i\in [n_t]}X_i^t [j]$ . Therefore, we may apply Corollary E.2, which gives
$$
\Pr \left[ \frac {1}{n _ {t}} \cdot \sum_ {i \in [ n _ {t} ]} Y _ {i} \leq 1. 1 p _ {j} + 0. 0 1 \cdot 2 ^ {- t} \right] > 1 - \frac {0 . 2 5 \beta}{L d}.
$$
Next, to give a lower bound on  $Y_{i}$ , we also observe that
$$
\begin{array}{l} \Pr \left[ Y _ {i} = 1 \right] = \Pr \left[ X _ {i} ^ {t} [ j ] = 1 \text {a n d} \| X _ {i} ^ {\ell} \left[ S _ {t} \right] \| _ {1} \leq B _ {t} \right] \\ \geq \Pr \left[ X _ {i} ^ {t} [ j ] = 1 \text {a n d} \| X _ {i} ^ {t} \left[ S _ {t} \backslash \{j \} \right] \| _ {1} \leq B _ {t} - 1 \right] \\ = \Pr \left[ X _ {i} ^ {t} [ j ] = 1 \right] \cdot \Pr \left[ \| X _ {i} ^ {t} [ S _ {t} \backslash \{j \} ] \| _ {1} \leq B _ {t} - 1 \right] \\ = p _ {j} \cdot \Pr \left[ \| X _ {i} ^ {t} [ S _ {t} \backslash \{j \} ] \| _ {1} \leq B _ {t} - 1 \right], \\ \end{array}
$$
$$
\mathbb {E} [ \| X _ {i} ^ {t} [ S _ {t} \setminus \{j \} ] \| _ {1} ] = \sum_ {j ^ {\prime} \in S _ {t} \backslash \{j \}} p _ {j ^ {\prime}}.
$$
We note that  $S_{t} \subseteq S_{t-1}$  holds by Algorithm 5 and  $S_{t-1} \subseteq S_{t-1}^{\uparrow}$  follows the assumption. Hence,  $p_{j'} < 0.75 \cdot 2^{-(t-1)}$  for every  $j' \in S_{t-1}^{\uparrow}$ . Moreover,  $|S_{t-1}^{\uparrow}| \leq d$ . Putting everything together, we can further bound
$$
\mathbb {E} [ \| X _ {i} ^ {t} [ S _ {t} \setminus \{j \} ] \| _ {1} ] = \sum_ {j ^ {\prime} \in S _ {t} \setminus \{j \}} p _ {j ^ {\prime}} \leq \sum_ {j ^ {\prime} \in S _ {t - 1} ^ {\uparrow}} p _ {j ^ {\prime}} \leq d \cdot 0. 7 5 \cdot 2 ^ {- (t - 1)} \leq 0. 1 (B _ {t} - 1).
$$
Therefore, by Markov's inequality, we have
$$
\Pr \left[ \| X _ {i} ^ {t} \left[ S _ {t} \backslash \{j \} \right] \| _ {1} \leq B _ {t} - 1 \right] \geq 0. 9.
$$
Combining with the above, we have  $\operatorname*{Pr}[Y_i = 1] \geq 0.9p_j$ . Notice also that  $Y_{1}, \ldots, Y_{n_{t}}$  are i.i.d. and always lie between  $[0,1]$ . Thus, we can apply Theorem E.1 with  $\Delta = (0.1p_{j} + 0.01 \cdot 2^{-t})n_{t}$  to obtain