SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\operatorname * {P r} \left[ \frac {1}{n _ {t}} \cdot \sum_ {i \in [ n _ {t} ]} Y _ {i} <   0. 8 p _ {j} - 0. 0 1 \cdot 2 ^ {- t} \right] \leq 2 \exp \left(\frac {\Delta^ {2}}{2 p _ {j} n _ {t} + \Delta}\right) \leq 2 \exp \left(\frac {\Delta^ {2}}{2 1 \Delta}\right) \leq \frac {0 . 2 5 \beta}{L d}.
$$
Applying a union bound then yields the claim.
E.3 Sampler
Next, we give a DP sampler for an unknown product distribution, assuming that a rough estimate of each  $p_j$  is already computed via the private preconditioner in Section E.2. The exact guarantee is given below; combining this with Lemma E.4, we immediately arrive at Theorem 1.7.
Lemma E.5. Suppose that  $\ell_1, \ldots, \ell_d$  satisfy the conditions in Lemma E.4. Then, there is an  $\epsilon$ -DP algorithm taking  $\ell_1, \ldots, \ell_d$  as input, that is an  $\alpha$ -accurate sampler and has a sample complexity of
$$
O \left(\frac {d \log (d / \alpha)}{\alpha \epsilon}\right).
$$
Algorithm 6 PRODUCTDISTSAMPLEER
Input: An unknown product distribution  $P$  over  $\{0,1\}^d$ , privacy parameter  $\epsilon$ , accuracy parameter  $\alpha$ , and  $\ell_1, \ldots, \ell_d \in \mathbb{N}$  satisfy the conditions in Lemma E.4
Output: A random sample that is  $\alpha$ -accurate for  $P$
$$
B \leftarrow 1 0 0 0 (d / \alpha) \cdot \log (\bar {d} / \alpha)
$$
$$
n \leftarrow \lceil 1 6 B / \epsilon \rceil
$$
$$
\text {S a m p l e} X _ {1}, \dots , X _ {n} \sim P
$$
$$
w \leftarrow (2 ^ {\ell_ {1}}, \dots , 2 ^ {\ell_ {d}})
$$
$$
q \leftarrow \frac {1}{n} \cdot \sum_ {i \in [ n ]} \operatorname {t r u n c} _ {B, w} ^ {1} \left(X _ {i}\right)
$$
$$
\mathbf {f o r} j = 1, \dots , d \mathbf {d o}
$$
$$
\left| \tilde {p} _ {j} \leftarrow \operatorname {c l i p} _ {\frac {1}{8 w _ {j}}, \frac {7}{8 w _ {j}}} \left(q _ {j}\right) \right.
$$
Sample  $Z\sim \operatorname {Ber}(\tilde{p}_1)\otimes \dots \otimes \operatorname {Ber}(\tilde{p}_d)$
return Z
In the following, we will focus on the proof of Lemma E.5 with Algorithm 6.
Privacy Analysis. Consider any pair  $\mathbf{X},\mathbf{X}^{\prime}$  of neighboring datasets. We may assume w.l.o.g. that the two datasets agree except for  $X_{n}$  and  $X_{n}^{\prime}$ . We write  $q^{\prime},\tilde{p}^{\prime}$  to denote the values of  $q,\tilde{p}$  computed for  $\mathbf{X}^{\prime}$ . Now, consider any possible output  $y\in \{0,1\} ^d$ . For each  $j\in [d]$ , we claim that
$$
\frac {\Pr \left[ \mathcal {A} \left(\mathbf {X} ^ {\prime}\right) _ {j} = y _ {j} \right]}{\Pr \left[ \mathcal {A} (\mathbf {X}) _ {j} = y _ {j} \right]} \leq 1 + 8 w _ {j} \left| \tilde {p} _ {j} ^ {\prime} - \tilde {p} _ {j} \right|. \tag {7}
$$
To show this, we first mention two facts:
$\frac{\operatorname*{Pr}[\mathcal{A}(\mathbf{X}^{\prime})_{j} = 0]}{\operatorname*{Pr}[\mathcal{A}(\mathbf{X})_{j} = 0]} = \frac{1 - \tilde{p}_{j}^{\prime}}{1 - \tilde{p}_{j}}\leq 1 + \frac{|\tilde{p}_{j}^{\prime} - \tilde{p}_{j}|}{1 - \tilde{p}_{j}};$
$\frac{\operatorname*{Pr}[\mathcal{A}(\mathbf{X}')_j = 1]}{\operatorname*{Pr}[\mathcal{A}(\mathbf{X})_j = 1]} = \frac{\tilde{p}_j'}{\tilde{p}_j}\leq 1 + \frac{|\tilde{p}_j' - \tilde{p}_j|}{\tilde{p}_j}.$
It suffices to show  $1 / (8w_{j}) \leq \tilde{p}_{j} \leq 1 - 1 / (8w_{j})$  to prove (7). This follows since  $w_{j} > 1$  and  $1 / (8w_{j}) \leq \tilde{p}_{j} \leq 7 / (8w_{j})$  due to clipping.
We also note the following useful property:
$$
\begin{array}{l} \left\| w \circ \left(q ^ {\prime} - q\right) \right\| _ {1} = \frac {1}{n} \cdot \left\| w \circ \left(\operatorname {t r u n c} _ {B, w} ^ {1} \left(X _ {n} ^ {\prime}\right) - \operatorname {t r u n c} _ {B, w} ^ {1} \left(X _ {n}\right)\right) \right\| _ {1} \\ \leq \frac {1}{n} \cdot \| w \circ \left(\operatorname {t r u n c} _ {B, w} ^ {1} \left(X _ {n} ^ {\prime}\right) + \operatorname {t r u n c} _ {B, w} ^ {1} \left(X _ {n}\right)\right) \| _ {1} \leq \frac {2 B}{n}. \tag {8} \\ \end{array}
$$
Finally, we are ready to prove the privacy guarantee:
$$
\begin{array}{l} \frac {\operatorname* {P r} [ \mathcal {A} (\mathbf {X} ^ {\prime}) = y ]}{\operatorname* {P r} [ \mathcal {A} (\mathbf {X}) = y ]} = \prod_ {j \in [ d ]} \frac {\operatorname* {P r} [ \mathcal {A} (\mathbf {X} ^ {\prime}) _ {j} = y _ {j} ]}{\operatorname* {P r} [ \mathcal {A} (\mathbf {X}) _ {j} = y _ {j} ]} \leq \prod_ {j \in [ d ]} \left(1 + 8 w _ {j} \cdot | \tilde {p} _ {j} ^ {\prime} - \tilde {p} _ {j} |\right) \\ \leq \prod_ {j \in [ d ]} \exp \left(8 w _ {j} \cdot | \tilde {p} _ {j} ^ {\prime} - \tilde {p} _ {j} |\right) = \exp \left(8 w \circ | \tilde {p} ^ {\prime} - \tilde {p} |\right) \\ \leq \exp (8 w \circ | q ^ {\prime} - q |) \leq \exp \left(\frac {1 6 B}{n}\right) = \exp (\epsilon), \\ \end{array}
$$
where the third inequality is due to clipping, the penultimate step follows (8), and the last step follows from our choice of parameters. Thus, the algorithm is  $\epsilon$ -DP as claimed.
Accuracy Analysis. Let  $S = \{j \in [d] \mid p_j < 2^{-L}\}$ . We first consider the version of  $\mathcal{A}$  (Algorithm 6), where there is no truncation at all and there is no clipping for any  $j \notin S$ , denoted as  $\mathcal{A}'$ . Let  $Q_{\mathcal{A}',P} = \operatorname{Ber}(p_1^*) \otimes \dots \otimes \operatorname{Ber}(p_d^*)$ . Fix  $\beta = \alpha / 12d < \alpha / 4$ . We define the event  $\mathcal{E}$  as "no  $X_i$  is truncated for any  $i \in [n]$  and no  $p_j$  is clipped for any  $j \notin S$  by Algorithm 6". By concentration results (Corollaries E.2 and E.3 with  $\gamma = 2^{-L - 1}$ ) and our selection of parameters, we can conclude that  $\mathcal{E}$  happens with probability at least  $1 - \beta$ , and therefore
$$
d _ {\mathrm {t v}} \left(Q _ {\mathcal {A} ^ {\prime}, P}, Q _ {\mathcal {A}, P}\right) = d _ {\mathrm {t v}} \left(Q _ {\mathcal {A} ^ {\prime}, P}, Q _ {\mathcal {A}, P} \mid \mathcal {E}\right) \cdot \Pr [ \mathcal {E} ] + d _ {\mathrm {t v}} \left(Q _ {\mathcal {A} ^ {\prime}, P}, Q _ {\mathcal {A}, P} \mid \bar {\mathcal {E}}\right) \cdot \Pr [ \bar {\mathcal {E}} ]
$$
$$
= 0 \cdot (1 - \beta) + 1 \cdot \beta \leq \alpha / 2.
$$
Furthermore, we can see that  $\mathbb{E}[p_j^*] = p_j$  if  $j \notin S$ . For  $j \in S$ , clipping ensures us that  $p_j^* \leq 2^{-L}$  hence  $\mathbb{E}[p_j^*] \leq 2^{-L}$ . Together, we have
$$
d _ {\mathrm {t v}} (Q _ {\mathcal {A} ^ {\prime}, P}, P) = \sum_ {j \in [ d ]} | \mathbb {E} [ p _ {j} ^ {*} ] - p _ {j} | = \sum_ {j \in S} | \mathbb {E} [ p _ {j} ^ {*} ] - p _ {j} | \leq \sum_ {j \in S} 2 ^ {- L} \leq d \cdot 2 ^ {- L} \leq \alpha / 2.
$$
Combining the above two inequalities yields the  $\alpha$ -accuracy guarantee of  $\mathcal{A}$ :
$$
d _ {\mathrm {t v}} \left(Q _ {\mathcal {A}, P}, P\right) \leq d _ {\mathrm {t v}} \left(Q _ {\mathcal {A}, P}, Q _ {\mathcal {A} ^ {\prime}, P}\right) + d _ {\mathrm {t v}} \left(Q _ {\mathcal {A} ^ {\prime}, P}, P\right) \leq \alpha / 2 + \alpha / 2 = \alpha .
$$
Proof of Theorem 1.7. Finally, we are ready to prove Theorem 1.7. For simplicity, let  $\alpha'$  be the target accuracy parameter. Set  $\alpha = \alpha'/2$  and  $\beta = \alpha/12d < \alpha'/2$ . We define the event  $\mathcal{E}$  as "Algorithm 5 returns  $\ell_1, \ldots, \ell_d$  satisfying the properties in Lemma E.4". If  $\mathcal{E}$  holds, then the error of our sampler is at most  $\alpha'/2$ , which is implied by Lemma E.5. If  $\mathcal{E}$  fails, the error is at most 1, but this event happens with probability at most  $\beta < \alpha'/2$ . Together, the error of our sampler is
$$
\alpha^ {\prime} / 2 \cdot (1 - \beta) + 1 \cdot \beta <   \alpha^ {\prime} / 2 + \alpha^ {\prime} / 2 = \alpha^ {\prime},
$$
and the sampling complexity is (with  $\beta = \alpha' / 24d$ )
$$
O \left(\left(d \log \left(\frac {d}{\alpha^ {\prime}}\right) + \frac {d}{\alpha^ {\prime}}\right) \frac {\log \left(\frac {d}{\alpha^ {\prime} \beta}\right)}{\epsilon} + \frac {d}{\alpha^ {\prime} \epsilon} \log \left(\frac {d}{\alpha^ {\prime}}\right)\right) = O \left(\frac {d}{\epsilon} \log^ {2} \left(\frac {d}{\alpha^ {\prime}}\right) + \frac {d}{\alpha^ {\prime} \epsilon} \log \left(\frac {d}{\alpha^ {\prime}}\right)\right),
$$