SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem D.2 (Formal version of Theorem 3.4). Let  $S$  be an  $\epsilon$ -differentially private distribution sketch of a dataset  $\mathcal{D} = \{x_1, \ldots, x_N\}$  with  $\mathcal{D} \sim p(x)$  and let  $k(x, y)$  be an LSH kernel (see Definition 3.2). Let  $S$  define the function implemented by Algorithm 4. Then
with convergence rate  $O\left(\sqrt{\log R / R} + \sqrt{R \log R} / (N \epsilon)\right)$  when  $R = \omega(1)$  (e.g.  $R = \log N$ ).
Proof. As before, let  $\kappa(x) = \sum_{x_i \in \mathcal{D}} \sqrt{k(x, x_i)}$  be the (non-normalized) kernel density estimate and let  $\widetilde{\kappa}(x) = \sum_{x_i \in \mathcal{D}} \sqrt{k(x, x_i)}$ . For the  $\epsilon$ -differentially private version of the algorithm, Theorem 3.4 of [48] provides the following inequality for any  $\delta > 0$ .
$$
\operatorname * {P r} \left[ | N S (x) - \kappa (x) | > \left(\left(\frac {\widetilde {\kappa} ^ {2} (x)}{R} + 2 \frac {R}{\epsilon^ {2}}\right) 3 2 \log 1 / \delta\right) ^ {1 / 2} \right] <   \delta
$$
Again, we divide both sides of the inner inequality by  $N$ , and we also loosen (and simplify) the inequality with the observation that  $\widetilde{\kappa}(x)/N < 1$ .
$$
\operatorname * {P r} \left[ \left| S (x) - \frac {1}{N} \kappa (x) \right| > \left(\left(\frac {1}{R} + 2 \frac {R}{N ^ {2} \epsilon^ {2}}\right) 3 2 \log 1 / \delta\right) ^ {1 / 2} \right] <   \delta
$$
As before, we choose  $\delta = \frac{1}{R}$ , which yields the following inequality:
$$
\Pr \left[ \left| S (x) - \frac {1}{N} \kappa (x) \right| > \left(\left(\frac {1}{R} + 2 \frac {R}{N ^ {2} \epsilon^ {2}}\right) 3 2 \log R\right) ^ {1 / 2} \right] <   \frac {1}{R}
$$
This is the same as:
$$
\Pr \left[ \left| S (x) - \frac {1}{N} \kappa (x) \right| > \left(3 2 \frac {\log R}{R} + 6 4 \frac {R}{N ^ {2} \epsilon^ {2}} \log R\right) ^ {1 / 2} \right] <   \frac {1}{R}
$$
Here, we will loosen the inequality again (also for the sake of presentation). Because  $\sqrt{a + b} \leq \sqrt{a} + \sqrt{b}$ , the following inequality is also satisfied:
$$
\operatorname * {P r} \left[ \left| S (x) - \frac {1}{N} \kappa (x) \right| > 4 \sqrt {2} \sqrt {\frac {\log R}{R}} + 8 \frac {\sqrt {R \log R}}{N \epsilon} \right] <   \frac {1}{R}
$$
Here, the convergence rate is  $O(\sqrt{\log R / R} + \sqrt{R\log R} / (N\epsilon))$ . For us to have the error converge in probability to zero, we need for
$$
\sqrt {\frac {\log R}{R}} + \sqrt {2} \frac {\sqrt {R \log R}}{N \epsilon} \rightarrow 0
$$
as  $N \to \infty$ . A simple way to achieve this is for  $R$  to be weakly dependent on  $N$  (i.e. choose  $R = \omega(1)$ ). For example, choosing  $R = \log N$  satisfies the conditions.