SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \operatorname {p s c r} _ {\mathcal {P}, \mathcal {P} ^ {*}} ^ {\psi} (\mathcal {D}) = \mathcal {P} \left(\mathcal {W} _ {\mathcal {P}, \mathcal {P} ^ {*}}\right) - \mathcal {D} \left(\mathcal {W} _ {\mathcal {P}, \mathcal {P} ^ {*}}\right) \\ = \left(\mathcal {P} \left(\mathcal {W} _ {\mathcal {P}, \mathcal {P} ^ {*}}\right) - \mathcal {P} ^ {*} \left(\mathcal {W} _ {\mathcal {P}, \mathcal {P} ^ {*}}\right)\right) - \left(\mathcal {D} \left(\mathcal {W} _ {\mathcal {P}, \mathcal {P} ^ {*}}\right) - \mathcal {P} ^ {*} \left(\mathcal {W} _ {\mathcal {P}, \mathcal {P} ^ {*}}\right)\right) \\ = d _ {\mathrm {t v}} \left(\mathcal {P}, \mathcal {P} ^ {*}\right) - \left(\mathcal {D} \left(\mathcal {W} _ {\mathcal {P}, \mathcal {P} ^ {*}}\right) - \mathcal {P} ^ {*} \left(\mathcal {W} _ {\mathcal {P}, \mathcal {P} ^ {*}}\right)\right) \\ \geq d _ {\mathrm {t v}} (\mathcal {P}, \mathcal {P} ^ {*}) - d _ {\mathrm {t v}} (\mathcal {P} ^ {*}, \mathcal {D}) \\ \geq d _ {\mathrm {t v}} (\mathcal {P}, \mathcal {P} ^ {*}) - 0. 1 \alpha . \\ \end{array}
$$
Thus, applying the utility guarantee from Theorem 37, we can conclude that, with probability  $1 - \beta$ , the algorithm outputs  $\mathcal{P}$  such that  $d_{\mathrm{tv}}(\mathcal{P},\mathcal{D})\leq 0.5\alpha +0.1\alpha < \alpha$  as desired.
E.3.3 Distribution Learning
Private hypothesis selection has a number of applications, arguably the most prominent one being distribution learning. In distribution learning, there is a family  $\mathfrak{P}$  of distributions. The underlying distribution  $\mathcal{D}$  comes from this family. The task dlearn  $(\mathfrak{P};\alpha)$  is to output a distribution  $\mathcal{Q}$  such that  $d_{\mathrm{tv}}(\mathcal{Q},\mathcal{D})\leq \alpha$  , where  $\alpha >0$  denotes the accuracy parameter.
For convenient, let us defined packing and covering of a family of distributions:
- A family  $\mathfrak{Q}$  of distributions is an  $\alpha$ -cover of a family  $\mathfrak{P}$  of distributions under distance measure  $d$  iff, for all  $\mathcal{P} \in \mathfrak{P}$ , there exists  $\mathcal{Q} \in \mathfrak{Q}$  such that  $d(\mathcal{Q}, \mathcal{P}) \leq \alpha$ .
- A family  $\mathfrak{Q}$  of distributions is an  $\alpha$ -packing of a family  $\mathfrak{P}$  of distributions under distance measure  $d$  iff,  $\mathfrak{Q} \subseteq \mathfrak{P}$  and for all distinct  $\mathcal{Q}, \mathcal{Q}' \in \mathfrak{Q}$ , we have  $d(\mathcal{Q}, \mathcal{Q}') \geq \alpha$ .
The size of the cover / packing is defined as  $\mathrm{size}(\mathfrak{Q})\coloneqq \log (|\mathfrak{Q}|)$ . The diameter of  $\mathfrak{Q}$  under distance  $d$  is defined as  $\max_{\mathcal{Q},\mathcal{Q}'\in \mathfrak{Q}}d(\mathcal{Q},\mathcal{Q}')$ .
The following convenient lemma directly follows from Theorem 41 and Lemma 39.
Theorem 42 (Distribution Learning—Arbitrary Family). Let  $\mathfrak{P}$  be any family of distributions, and let  $\alpha, \beta, \varepsilon > 0$  be sufficiently small. If there exists an  $0.1\alpha$ -cover of  $\mathfrak{P}$  under the TV distance of size  $L$ , then
$$
n _ {m} ^ {\mathrm {d l e a r n} (\mathfrak {P}; \alpha)} (\varepsilon , \delta = 0) \leq \widetilde {O} \left(\frac {L}{\varepsilon} + \frac {L}{\alpha \varepsilon \sqrt {m}} + \frac {L}{\alpha^ {2} m}\right).
$$
Furthermore, if there exists a family  $\mathfrak{Q}$  of size  $L$  that is an  $3\alpha$ -packing of  $\mathfrak{P}$  under the TV distance and has diameter at most  $\beta$  under the KL-divergence, then
$$
n _ {m} ^ {\mathrm {d l e a r n} (\mathfrak {P}; \alpha)} (\varepsilon , \delta = 0) \geq \tilde {\Omega} \left(\frac {L}{\varepsilon} + \frac {L}{\varepsilon \sqrt {m \beta}} + \frac {L}{m \beta}\right).
$$
In theorems below, we abuse the notations  $\tilde{\Theta},\tilde{O}$  and also use them to suppress terms that are polylogarithmic in  $1 / \alpha ,d,R,$  and  $\kappa$
Discrete Distributions. First is for the task of discrete distribution learning on domain  $[k]$  (denoted by  $\mathrm{DD}k(\alpha)$ ), which has been studied before in  $[\mathrm{LSY}^{+}20]$ . In this case,  $\mathfrak{P}$  consists of all distributions over the domain  $[k]$ . We have the following theorem:
Theorem 43 (Distribution Learning—Discrete Distributions). For any sufficiently small  $\alpha, \varepsilon > 0$  and for all  $m \in \mathbb{N}$ , we have
$$
n _ {m} ^ {\mathrm {D D} k (\alpha)} (\varepsilon , \delta = 0) = \tilde {\Theta} \left(\frac {k}{\varepsilon} + \frac {k}{\alpha \varepsilon \sqrt {m}} + \frac {k}{\alpha^ {2} m}\right).
$$
Proof. Upper bound. It is simple to see that there exists an  $0.1\alpha$ -cover of the family of size  $O(k \cdot \log (k / \alpha))$ : We simply let the cover contain all distributions whose probability mass at each point is a multiple of  $\lfloor 10k / \alpha \rfloor$ . Plugging this into Theorem 42 yields the desired upper bound.
Lower bound. As for the lower bound, we may use a construction similar to that of [ASZ21] (and also similar to that in the proof of Theorem 40). Assume w.l.o.g. that  $k$  is even. From Theorem 24, there exist  $u_{1},\ldots ,u_{W}\in \{0,1\}^{k / 2}$  where  $W = 2^{\Omega (k)}$  such that  $\| u_i - u_j\| _0\geq \kappa \cdot k$  for some constant  $\kappa$ . For any sufficiently small  $\alpha < \kappa /16$ , we can define the distribution  $\mathcal{D}_i$  for  $i\in [W]$  by
$$
\mathcal {D} _ {i} (2 \ell - 1) = \frac {1}{2 k} \left(1 + \frac {8 \alpha}{\kappa} \cdot \mathbf {1} [ (u _ {i}) _ {\ell} = 0 ]\right)
$$
$$
\mathcal {D} _ {i} (2 \ell) = \frac {1}{2 k} \left(1 - \frac {8 \alpha}{\kappa} \cdot {\bf 1} [ (u _ {i}) _ {\ell} = 0 ]\right),
$$
for all  $\ell \in [k / 2]$ . By a similar calculation as in the proof of Theorem 40, we have that it is an  $(3\alpha)$ -packing under TV distance and its diameter under KL-divergence is at most  $O(\alpha^2)$ . Plugging this into Theorem 42 then gives the lower bound.
Interestingly, the user complexity matches those achieved via approximate-  $DP$  algorithms from  $\mathrm{[LSY^{+}20,NME22]}$ , except for the first term (i.e.,  $k / \varepsilon$ ) whereas the best known  $(\varepsilon ,\delta)$ -DP algorithm of [NME22] works even with  $\log (1 / \delta) / \varepsilon$  instead. In other words, when  $m$  is intermediate, the user complexity between the pure-  $DP$  and approximate-  $DP$  cases are the same. Only for large  $m$  that approximate-  $DP$  helps.
Product Distributions. In this case,  $\mathfrak{P}$  is a product distribution over the domain  $[k]^d$  (denoted by  $\mathrm{PD}(k,d;\alpha)$ ). We have the following theorem:
Theorem 44 (Distribution Learning—Product Distributions). For any sufficiently small  $\alpha, \varepsilon > 0$  and for all  $m \in \mathbb{N}$ , we have
$$
n _ {m} ^ {\mathrm {P D} (k, d; \alpha)} (\varepsilon , \delta = 0) = \tilde {\Theta} \left(\frac {k d}{\varepsilon} + \frac {k d}{\alpha \varepsilon \sqrt {m}} + \frac {k d}{\alpha^ {2} m}\right).
$$
Proof. Upper Bound. Similar to before, we simply take  $\mathfrak{Q}$  to be a  $0.1\alpha$ -cover (under TV distance) of the product distributions, which is known to have size at most  $O(kd \cdot \log(kd / \alpha))$  [BKSW19, Lemma 6.2].
Lower Bound. Acharya et al. [ASZ21, Proof of Theorem 11] showed that there exists a family  $\mathfrak{L}$  of product distributions over  $[k]^d$  that is a  $(3\alpha)$ -packing under TV distance and its diameter under KL-divergence is at most  $O(\alpha^2)$  of size  $\Omega(kd)$ . Theorem 42 then gives the lower bound.
Gaussian Distributions: Known Covariance. Next, we consider Gaussian distributions  $\mathcal{N}(\mu, \Sigma)$  where  $\mu \in \mathbb{R}^d$ ,  $\Sigma \in \mathbb{R}^{d \times d}$ , under the assumption that  $\| \mu \| \leq R$  and  $\Sigma = I^{14}$ . Let  $\mathrm{Gauss}(R,d;\alpha)$  denote this task, where  $\alpha$  is again the (TV) accuracy. For this problem, we have:
Theorem 45 (Distribution Learning—Gaussian Distributions with Known Covariance). For any sufficiently small  $\alpha, \varepsilon > 0$  and for all  $m \in \mathbb{N}$ , we have
$$
n _ {m} ^ {\mathrm {G a u s s} (R, d; \alpha)} (\varepsilon , \delta = 0) = \tilde {\Theta} \left(\frac {d}{\varepsilon} + \frac {d}{\alpha \varepsilon \sqrt {m}} + \frac {d}{\alpha^ {2} m}\right).
$$
Proof. Upper Bound. It is known that this family of distribution admits a  $(0.1\alpha)$ -cover (under TV distance) of size  $O(d \cdot \log(dR / \alpha))$  [BKSW19, Lemma 6.7]. This, together with Theorem 42, gives the upper bound.
Lower Bound. Again, Acharya et al. [ASZ21, Proof of Theorem 12] showed that there exists a family  $\mathfrak{Q}$  of isotropic Gaussian distributions with  $\| \mu \| \leq O(\sqrt{\log(1 / \alpha)})$  that is a  $(3\alpha)$ -packing under TV distance and its diameter under KL-divergence is at most  $O(\alpha^2)$  of size  $\Omega (d)$ . Theorem 42 then gives the lower bound.
We note that this problem has already been (implicitly) studied under user-level approximate-DP in [NME22], $^{15}$  who showed that  $n_m^{\mathrm{Gauss}(R,d;\alpha)}(\varepsilon, \delta > 0) \leq \widetilde{O}\left(\frac{d}{\alpha\varepsilon\sqrt{m}} + \frac{d}{\alpha^2m}\right)$ . Again, it is perhaps surprising that our pure-DP bound nearly matches this result, except that we have the extra first term (i.e.,  $d / \varepsilon$ ).
Gaussian Distributions: Unknown Bounded Covariance. Finally, we consider the case where the covariance is also unknown but is assumed to satisfy  $I \preceq \Sigma \preceq \kappa I$ . We use Gauss $(R, d, \kappa; \alpha)$  to denote this task.
Theorem 46 (Distribution Learning—Gaussian Distributions with Unknown Bounded Covariance). Let  $\kappa > 1$  be a constant. For any sufficiently small  $\alpha, \varepsilon > 0$  and for all  $m \in \mathbb{N}$ , we have
$$
n _ {m} ^ {\mathrm {G a u s s} (R, d, \kappa ; \alpha)} (\varepsilon , \delta = 0) = \tilde {\Theta} _ {\kappa} \left(\frac {d ^ {2}}{\varepsilon} + \frac {d ^ {2}}{\alpha \varepsilon \sqrt {m}} + \frac {d ^ {2}}{\alpha^ {2} m}\right).
$$
Proof. Upper Bound. This follows from Theorem 42 and a known upper bound of  $O(d^{2}\log (d\kappa /\alpha) + d\cdot \log (dR / \alpha))$  on the size of a  $(0.1\alpha)$ -cover (under TV distance) of the family [BKSW19, Lemma 6.8].
Lower Bound. Devroye et al. [DMR20] showed $^{16}$  that, for  $\kappa \geq 1 + O(\alpha)$ , there exists a  $3\alpha$ -packing under TV distance whose diameter under the KL-divergence is at most  $O(\alpha^2)$  of size  $\Omega(d^2)$ . This, together with Theorem 42, gives the lower bound.
We remark that it is simple to see that Gaussian with unbounded mean or unbounded covariance cannot be learned with pure (even user-level) DP using a finite number of examples.
<sup>14</sup>Or equivalently that  $\Sigma$  is known; since such an assumption is sufficient to rotate the example to isotropic positions.
15 Actually, Narayanan et al. [NME22] studied the mean estimation problem, but it is not hard to see that an algorithm for mean estimation also provides an algorithm for learning Gaussian distributions.
<sup>16</sup>Specifically, this is shown in [DMR20, Proposition 3.1]; their notation "m" denote the number of non-zero (non-diagonal) entries of the covariance matrix which can be set to  $\Omega(d^2)$  for our purpose.