SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof sketch. The full proof is given in Appendix D.1, and mirrors that of Theorem 4.5 from  $\mathrm{[ABDH^{+}20]}$  (albeit adapted to the public-private setting). Fix  $\alpha ,\beta \in (0,1]$  and  $\varepsilon >0$  . Let  $\tau = \tau (\frac{\alpha}{6},\frac{\beta}{2})$ $t = t(\frac{\alpha}{6},\frac{\beta}{2})$  , and  $m = m_{C}(\frac{\alpha}{6},\frac{\beta}{2})$  . We draw a public sample  $\tilde{\pmb{X}}$  of size  $m$  , and enumerate all combinations of a size  $\tau$  subset of  $\tilde{\pmb{X}}$  with a binary string of length  $t$  . Essentially, we "guess" the encoding of the unknown distribution  $p$  . Running the decoder on this set of possible encodings gives us a set of distributions; some of them are close to  $p$  . Then, with private samples only, we use the pure DP 3-agnostic learner for finite classes (Fact C.2) to pick one such out.
3.2 Public-private learning implies list learning
Next, we show that the existence of a public-private learner for a class of distributions implies the existence of a list learner for the class. A list learner for a class  $\mathcal{Q}$  takes input samples from any  $p\in \mathcal{Q}$  and outputs a finite list of distributions, one of which is close to  $p$
Definition 3.2 (List learner). Let  $\mathcal{Q} \subseteq \Delta(\mathcal{X})$ . For  $\alpha, \beta \in (0,1]$  and  $\ell \in \mathbb{N}$ , an  $(\alpha, \beta, \ell)$ -list learner for  $\mathcal{Q}$  is an algorithm  $\mathcal{L} \colon \mathcal{X}^m \to \{L \subseteq \Delta(\mathcal{X}) : |L| \leq \ell\}$ , such that for any  $p \in \mathcal{Q}$ , if we draw a dataset  $X = (X_1, \ldots, X_m)$  i.i.d. from  $p$ , then
$$
\underset {\boldsymbol {X} \sim p ^ {m}} {\mathbb {P}} \left\{\operatorname {d i s t} (p, \mathcal {L} (\boldsymbol {X})) \leq \alpha \right\} \geq 1 - \beta .
$$
Definition 3.3 (List learnable class). A class of distributions  $\mathcal{Q} \subseteq \Delta(\mathcal{X})$  is list learnable to list size  $\ell(\alpha, \beta)$  with  $m(\alpha, \beta)$  samples if for every  $\alpha, \beta \in (0,1]$ , letting  $\ell = \ell(\alpha, \beta)$  and  $m = m(\alpha, \beta)$ , there is an  $(\alpha, \beta, \ell)$ -list-learner for  $\mathcal{Q}$  that takes  $m$  samples.
If  $\mathcal{Q}$  satisfies the above, irrespective of the list size complexity  $\ell(\alpha, \beta)$ , we say  $\mathcal{Q}$  is list learnable with  $m(\alpha, \beta)$  samples.
Now, we state our reduction of list learning to public-private learning. The key step of our proof is showing that, upon receiving samples  $\tilde{\pmb{x}}$ , outputting a finite cover of the list of distributions that a public-private learner would succeed on given public data  $\tilde{\pmb{x}}$  is a successful strategy for list learning.
Proposition 3.4 (Public-private learning  $\Longrightarrow$  list learning). Let  $\mathcal{Q} \subseteq \Delta(\mathcal{X})$ . Suppose  $\mathcal{Q}$  is public-privately learnable with  $m_P(\alpha, \beta, \varepsilon)$  public and  $n(\alpha, \beta, \varepsilon)$  private samples. Then for all  $\varepsilon > 0$ ,  $\mathcal{Q}$  is list-learnable to list size  $\ell(\alpha, \beta) = \frac{10}{9} \exp(\varepsilon \cdot n(\frac{\alpha}{2}, \frac{\beta}{10}, \varepsilon))$  with  $m(\alpha, \beta) = m_P(\frac{\alpha}{2}, \frac{\beta}{10}, \varepsilon)$  samples.
Proof. Let  $\varepsilon > 0$  be arbitrary. Fix any  $\alpha, \beta \in (0,1]$ . By assumption,  $\mathcal{Q}$  admits a  $(\frac{\alpha}{2}, \frac{\beta}{10}, \varepsilon)$ -public-private learner  $\mathcal{A}$ , which uses  $m := m_P(\frac{\alpha}{2}, \frac{\beta}{10}, \varepsilon)$  public and  $n := n(\frac{\alpha}{2}, \frac{\beta}{10}, \varepsilon)$  private samples. We use  $\mathcal{A}$  to construct a  $(\alpha, \beta, \frac{10}{9} \exp(\varepsilon n))$ -list learner that uses  $m$  samples. Consider any  $\tilde{\boldsymbol{x}} = (\tilde{x}_1, \ldots, \tilde{x}_m) \in \mathcal{X}^m$  and the class
$$
\mathcal{Q}_{\tilde{\boldsymbol{x}}} = \left\{q\in \mathcal{Q}:\underset { \begin{array}{c}\boldsymbol {X}\sim q^{n}\\ Q\sim \mathcal{A}(\tilde{\boldsymbol{x}},\boldsymbol {X}) \end{array} }{\mathbb{P}}\left\{\mathrm{TV}(Q,q)\leq \frac{\alpha}{2}\right\} \geq \frac{9}{10}\right\} .
$$
Note that by definition,  $\mathcal{Q}_{\tilde{x}}$  has a  $(\frac{\alpha}{2},\frac{1}{10})$ -learner under  $\varepsilon$ -DP that takes  $n$  samples. Hence, by Fact C.1 it follows that any  $\alpha$ -packing of  $\mathcal{Q}_{\tilde{x}}$  must have size  $\leq \frac{10}{9}\exp (\varepsilon n) =: \ell$ . Let  $\widehat{Q}_{\tilde{x}}$  be such a maximal  $\alpha$ -packing, hence it is also an  $\alpha$ -cover of  $\mathcal{Q}_{\tilde{x}}$  with  $|\widehat{Q}_{\tilde{x}}| \leq \ell$ . We define our list learner's output,  $\mathcal{L}(\tilde{x}) = \widehat{Q}_{\tilde{x}}$ . It remains to show that for any  $p \in \mathcal{Q}$ , with probability  $\geq 1 - \beta$  over the sampling of  $\tilde{X} \sim p^m$ ,  $\mathrm{dist}(p,\mathcal{L}(\tilde{X})) \leq \alpha$ . Suppose otherwise, that is, there exists  $p_0 \in \mathcal{Q}$ , such that
$$
\underset {\tilde {\boldsymbol {X}} \sim p _ {0} ^ {m}} {\mathbb {P}} \left\{\operatorname {d i s t} (p _ {0}, \mathcal {L} (\tilde {\boldsymbol {X}})) > \alpha \right\} > \beta .
$$
Since  $\mathcal{L}(\tilde{\boldsymbol{X}})$  is a  $\alpha$ -cover of  $\mathcal{Q}_{\tilde{\boldsymbol{X}}}$ , we have that with probability  $> \beta$  over the sampling of  $\tilde{\boldsymbol{X}} \sim p_0^m$ ,  $p_0 \notin \mathcal{Q}_{\tilde{\boldsymbol{X}}}$ . This contradicts the success guarantee of  $\mathcal{A}$ :
$$
\begin{array}{l} \mathop{\mathbb{P}}_{\substack{\tilde{\boldsymbol{X}}\sim p_{0}^{n},\boldsymbol {X}\sim p_{0}^{n}\\ Q\sim \mathcal{A}(\tilde{\boldsymbol{X}},\boldsymbol {X})}}\left\{\mathrm{TV}(Q,p_{0}) > \frac{\alpha}{2}\right\} \geq \mathbb{P}\left\{\mathrm{TV}(Q,p_{0}) > \frac{\alpha}{2}\big|p_{0}\notin \mathcal{Q}_{\tilde{\boldsymbol{X}}} \right\} \cdot \mathbb{P}\left\{p_{0}\notin \mathcal{Q}_{\tilde{\boldsymbol{X}}} \right\} \\ > \frac {1}{1 0} \cdot \beta = \frac {\beta}{1 0}. \\ \end{array}
$$
The second inequality follows by the definition of  $Q_{\tilde{X}}$ : conditioned on the event  $p_0 \notin \mathcal{Q}_{\tilde{X}}$ , the probability, over the private samples  $X \sim p_0^n$  and the randomness of the algorithm  $\mathcal{A}$ , that the output  $Q$  of our algorithm satisfies  $\mathrm{TV}(Q, p_0) \leq \frac{\alpha}{2}$  is  $< \frac{9}{10}$ .
3.3 List learning implies compression
We state the final component of Theorem 1.1/D.1: the existence of a list learner for a class of distributions  $\mathcal{Q}$  implies the existence of a sample compression scheme for  $\mathcal{Q}$ . This follows from the definitions: given samples  $\tilde{\pmb{x}}$ , the encoder of the sample compression scheme runs a list learner  $\mathcal{L}$  on  $\tilde{\pmb{x}}$ . It passes along  $\pmb{x}$ , and, with knowledge of the target distribution  $q$ , the index  $i$  of the distribution in  $\mathcal{L}(\tilde{\pmb{x}})$  that is close to  $q$ . The decoder receives this information and outputs  $\mathcal{L}(\tilde{\pmb{x}})_i$ . The proof can be found in Appendix D.2.
Proposition 3.5 (List learning  $\Longrightarrow$  compression). Let  $\mathcal{Q} \subseteq \Delta(\mathcal{X})$ . Suppose  $\mathcal{Q}$  is list learnable to list size  $\ell(\alpha, \beta)$  with  $m_L(\alpha, \beta)$  samples. Then  $\mathcal{Q}$  admits  $(\tau(\alpha, \beta), t(\alpha, \beta), m(\alpha, \beta)) = (m_L(\alpha, \beta), \log_2(\ell(\alpha, \beta)), m_L(\alpha, \beta))$  realizable sample compression.
Here, we state a few applications of the connections obtained via Theorem 1.1/D.1. First, we recover and extend results on the public-private learnability of high-dimensional Gaussians and mixtures of Gaussians, using known results on sample compression schemes. Second, we describe the closure properties of public-private learnability: if a class  $\mathcal{Q}$  is public-privately learnable, the class of mixtures of  $\mathcal{Q}$  and the class of products of  $\mathcal{Q}$  are also public-privately learnable.
4.1 Public-private learnability of Gaussians and mixtures of Gaussians
There are known realizable sample compression schemes for the class of Gaussians in  $\mathbb{R}^d$ , as well as for the class of all  $k$ -mixtures of Gaussians in  $\mathbb{R}^d$  [ABDH+20]. Hence, these classes are public-privately learnable.
Fact 4.1 (Robust compression scheme for Gaussians  $\mathrm{[ABDH^{+}20}$ , Lemma 5.3]). The class of Gaussians over  $\mathbb{R}^d$  admits  $(O(d), O(d^{2}\log (\frac{d}{\alpha}))$ ,  $O(d\log (\frac{1}{\beta}))\frac{2}{3}$ -robust sample compression.
Fact 4.2 (Realizable compression scheme for mixtures of Gaussians [ABDH+20, Lemma 4.8 applied to Lemma 5.3]). The class of  $k$ -mixtures of Gaussians over  $\mathbb{R}^d$  admits  $(O(kd), O(kd^2 \log \left(\frac{d}{\alpha}\right) + \log \left(\frac{k}{\alpha}\right)), O\left(\frac{k d \log (k / \beta) \log (1 / \beta)}{\alpha}\right))$  realizable sample compression.
Gaussians. From Theorem 1.1/D.1 and Fact 4.1, we get a public-private learner for Gaussians over  $\mathbb{R}^d$  using  $O(d\log (\frac{1}{\beta}))$  public and  $\tilde{O} (\frac{d^2 + \log(1 / \beta)}{\alpha^2} +\frac{d^2 + \log(1 / \beta)}{\alpha\varepsilon})$  private samples (Corollary E.1). This recovers the result of [BKS22] on Gaussians up to a factor of  $O(\log (\frac{1}{\beta}))$  in public sample complexity, and improves the private sample complexity by a polylog  $(1 / \beta)$  factor.
Mixtures of Gaussians. From Theorem 1.1/D.1 and Fact 4.2, we get a public-private learner for the class of  $k$ -mixtures of Gaussians in  $\mathbb{R}^d$  using  $\tilde{O}\left(\frac{k d \log^{2}(1 / \beta)}{\alpha}\right)$  public and  $\tilde{O}\left(\frac{k d^{2} + \log(1 / \beta)}{\alpha^{2}} + \frac{k d^{2} + \log(1 / \beta)}{\alpha \varepsilon}\right)$  private samples (Theorem 1.2/Corollary E.2).
In terms of  $k$ ,  $d$ , and  $\alpha$ , the public sample complexity of this learner is less than the  $\tilde{\Theta}\left(\frac{kd^2}{\alpha^2}\right)$  necessary and sufficient sample complexity for the problem non-privately [ABDH+20]. This also implies that in the regime where  $\varepsilon > \alpha$ , having more public samples (but still  $\tilde{o}\left(\frac{kd^2}{\alpha^2}\right)$ ) cannot improve private sample complexity. Under pure DP, no finite sample size suffices; under approximate DP,  $\tilde{O}\left(\frac{k^2d^4\log(1 / \delta)}{\alpha^2\varepsilon}\right)$  has been shown to suffice [AAL23b].
Algorithms for parameter estimation, like the mixture of Gaussians estimators from [BKS22] cannot be directly compared as they target a strictly stronger success criteria under stronger assumptions on the underlying distribution. [BKS22] gives an estimator for separated Gaussian mixtures that uses  $\tilde{O} \left( \frac{d \log(k)}{w_{*}} \right)$  public and  $\tilde{O} \left( \frac{d^2}{w_{*} \alpha^2} + \frac{d^2}{w_{*} \alpha^2} \right)$  private samples satisfying  $\frac{\varepsilon^2}{2}$ -zCDP, where  $w_{*}$  is the minimum component mixing weight. In terms of the privacy guarantee and private sample complexity, our result is a strict improvement for density estimation, since  $k = O\left( \frac{1}{w_{*}} \right)$ . In the setting where  $w_{*} = \Omega \left( \frac{1}{k} \right)$ , [BKS22]'s algorithm uses  $\frac{1}{\alpha}$ -factor fewer public samples. Furthermore, their algorithm runs in time polynomial in the sample complexity.
4.2 Public-private learnability of mixture and product distributions
If a class is realizably compressible, the class of its  $k$ -mixtures and the class of its  $k$ -products are also realizably compressible. Being realizably compressible with  $O(m(\alpha, \beta))$  samples is equivalent to being public-privately learnable with  $O(m(\alpha, \beta))$  public samples. Hence, we have black-box reductions of public-private learnability of mixture/product classes to public-private learnability of their base classes.
Theorem 4.3 (Public-private learning for mixture and product distributions (Informal; see Theorems E.5 and E.7)). Let  $k \geq 1$ . If  $\mathcal{Q} \subseteq \Delta(\mathcal{X})$  is public-privately learnable with  $m(\alpha, \beta, \varepsilon)$  public samples, then for any  $\varepsilon_0 > 0$ :
1. The class of  $k$ -mixtures of  $\mathcal{Q}$  is public-privately learnable with  $O\left(\frac{k\log(k / \beta)}{\alpha} \cdot m(\alpha, \beta, \varepsilon_0)\right)$  public samples.
2. The class of  $k$ -products of  $\mathcal{Q}$  (over  $\mathcal{X}^k$ ) is public-privately learnable with  $O(\log \left(\frac{k}{\beta}\right)\cdot m(\alpha /k,\beta ,\varepsilon_0))$  public samples.
The full statements and proof can be found in Appendix E.2. Here, we use the non-constructive reduction of list learning to public-private learning, and hence this does not yield a finite time algorithm. Note also that the target privacy  $\varepsilon$  does not appear in the public sample complexity (it only affects the amount of private samples required).
5 Agnostic and distribution-shifted public-private learning
The setting we have examined thus far makes the following assumptions on the data generation process: (1) same distribution - public and private data are sampled from the same underlying distribution; and (2) realizability - public and private data are sampled from members of the class  $\mathcal{Q}$ .
[BKS22] shows that for Gaussians over  $\mathbb{R}^d$ , the first condition can be relaxed: they give an algorithm for the case where the public and the private data are generated from different Gaussians with bounded TV distance. However, they do not remove the second assumption.
We show that for general classes of distributions that robust compression schemes yield public-private learners, which: (1) can handle public-private distribution shifts (i.e., the setting where the public data and the private data distributions can be different); and (2) are agnostic, i.e., they do not require samples to come from a member of  $\mathcal{Q}$ , and instead, promise error close to the best approximation of the private data distribution by a member of  $\mathcal{Q}$ . Since Gaussians admit a robust compression scheme (Fact 4.1), we obtain public-private Gaussian learners that work under relaxed forms of these assumptions on the data generating process (Theorem 1.3/F.1).
We first formally define the notion of agnostic and distribution-shifted public-private learning, and then prove the main result of this section.
Definition 5.1 (Agnostic and distribution-shifted public-private learner). Let  $\mathcal{Q} \subseteq \Delta(\mathcal{X})$ . For  $\alpha, \beta \in (0,1], \varepsilon > 0, \gamma \in [0,1]$ , and  $c \geq 1$  a  $\gamma$ -shifted c-agnostic  $(\alpha, \beta, \varepsilon)$ -public-private learner for  $\mathcal{Q}$  is a public-private  $\varepsilon$ -DP algorithm  $\mathcal{A}: \mathcal{X}^m \times \mathcal{X}^n \to \Delta(\Delta(\mathcal{X}))$ , such that for any  $\tilde{p}, p \in \Delta(\mathcal{X})$  with  $\mathrm{TV}(\tilde{p}, p) \leq \gamma$ , if we draw datasets  $\tilde{\boldsymbol{X}} = (\tilde{X}_1, \ldots, \tilde{X}_m)$  i.i.d. from  $\tilde{p}$  and  $\boldsymbol{X} = (X_1, \ldots, X_n)$  i.i.d. from  $p$ , and then  $Q \sim \mathcal{A}(\tilde{\boldsymbol{X}}, \boldsymbol{X})$ ,
$$
\mathbb{P}_{\substack{\tilde{\boldsymbol{X}}\sim \tilde{p}^{m},\boldsymbol {X}\sim p^{n}\\ Q\sim \mathcal{A}(\tilde{\boldsymbol{X}},\boldsymbol {X})}}\left\{\mathrm{TV}(Q,p)\leq c\cdot \mathrm{dist}(p,\mathcal{Q}) + \alpha \right\} \geq 1 - \beta .
$$
Theorem 5.2 (Robust compression  $\Longrightarrow$  agnostic and distribution-shifted public-private learning). Let  $\mathcal{Q} \subseteq \Delta(\mathcal{X})$  and  $r > 0$ . If  $\mathcal{Q}$  admits  $(\tau(\alpha, \beta), t(\alpha, \beta), m_C(\alpha, \beta))$ $r$ -robust compression, then for every  $\alpha, \beta \in (0,1]$  and  $\varepsilon > 0$ , there exists a  $\frac{r}{2}$ -shifted  $\frac{2}{r}$ -agnostic  $(\alpha, \beta, \varepsilon)$ -public-private learner for  $\mathcal{Q}$  that uses  $m(\alpha, \beta, \varepsilon) = m_C\left(\frac{\alpha}{12}, \frac{\beta}{2}\right)$  public samples and  $n(\alpha, \beta, \varepsilon) = O\left(\left(\frac{1}{\alpha^2} + \frac{1}{\alpha\varepsilon}\right) \cdot \left(t\left(\frac{\alpha}{12}, \frac{\beta}{2}\right) + \tau\left(\frac{\alpha}{12}, \frac{\beta}{2}\right) \log\left(m_C\left(\frac{\alpha}{12}, \frac{\beta}{2}\right)\right) + \log\left(\frac{1}{\beta}\right)\right)$  private samples.
Proof. The proof again mirrors the proof of Theorem 4.5 in  $\mathrm{[ABDH^{+}20]}$ . The key observation (and difference from the proof in Appendix D.1) is the following: for the unknown distribution  $p\in \Delta (\mathcal{X})$  consider  $\mathrm{dist}(p,\mathcal{Q})$ . If  $\mathrm{dist}(p,\mathcal{Q})\geq \frac{r}{2}$ , the output  $Q$  of any algorithm satisfies  $\mathrm{TV}(p,Q)\leq 1\leq \frac{2}{r}\cdot \mathrm{dist}(p,\mathcal{Q})$ . Hence, we can assume  $\mathrm{dist}(p,\mathcal{Q}) < \frac{r}{2}$ , and let  $q_{*}\in \mathcal{Q}$  with  $\mathrm{TV}(p,q_{*}) < \min \left\{\frac{r}{2},\mathrm{dist}(p,\mathcal{Q}) + \frac{\alpha}{12}\right\}$  as guaranteed by such.
By triangle inequality,  $\mathrm{TV}(\tilde{p},q_{*}) < r$ . This implies that when we generate hypotheses  $\widehat{\mathcal{Q}}$  to choose from using the  $r$ -robust sample compression with samples from  $\tilde{p}$ , with high probability there will be some  $q\in \widehat{\mathcal{Q}}$  with  $\mathrm{TV}(q,q_{*})\leq \frac{\alpha}{12}$ . We have
$$
\operatorname {T V} (p, q) \leq \operatorname {T V} (p, q _ {*}) + \operatorname {T V} (q _ {*}, q) \leq \operatorname {d i s t} (p, \mathcal {Q}) + \frac {\alpha}{1 2} + \frac {\alpha}{1 2} = \frac {\alpha}{6}.
$$
Applying the 3-agnostic  $\varepsilon$ -DP learner for finite classes from [AAAK21] (Fact C.2) with the above setting of  $n$  gives us the result.
We give a lower bound on the number of public samples required to public-privately learn Gaussians in  $\mathbb{R}^d$ . We know that Gaussians in  $\mathbb{R}^d$  are public-privately learnable with  $d + 1$  public samples from [BKS22]. We show that this is within 1 of the optimal: the class of Gaussians in  $\mathbb{R}^d$  is not public-privately learnable with  $d - 1$  public samples. The following is the formal statement of Theorem 1.4.
Theorem 6.1. The class  $\mathcal{Q}$  of all Gaussians in  $\mathbb{R}^d$  is not public-private learnable with  $m_P(\alpha, \beta, \varepsilon) = d - 1$  public samples, regardless of the number of private samples. That is, there exists  $\alpha_d, \beta_d > 0$  such that for any  $n \in \mathbb{N}$ ,  $\mathcal{Q}$  does not admit a  $(\alpha_d, \beta_d, 1)$ -public-private learner using  $d - 1$  public and  $n$  private samples.
Our result leverages the connection between public-private learning and list learning. The existence of such a public-private learner described above would imply the existence of a list learner for  $d$ -dimensional Gaussians taking  $d - 1$  samples as input. We show, using a "no-free-lunch"-style argument (e.g. Theorem 5.1 from [SSBD14]) that such a list learner cannot exist. The proof of Theorem 6.1, given in Appendix G, goes through the following steps.
1. We reduce list learning to public-private learning, via Proposition 3.4;
2. We establish a technical lemma that relates the PAC guarantee of a list learner with its average performance over a set of problem instances, via a "no-free-lunch"-style argument (Lemma G.1);
3. For every  $d \geq 2$ , we find a sequence of hard subclasses of Gaussians over  $\mathbb{R}^d$ , which satisfy the conditions of Lemma G.1. This forms the set of hard problem instances that imply a lower bound on the error of any list learner for the class (does not receive enough samples);
4. Since list learning to arbitrary error with few samples is impossible, public-private learning to arbitrary error with few public samples must also be impossible.
7 Learning when the Yatracos class has finite VC dimension
In this section, we describe a public-private learner for classes of distributions whose Yatracos class has finite VC dimension. We start by defining the Yatracos class of a family of distributions.
Definition 7.1 (Yatracos class). For  $\mathcal{Q} \subseteq \Delta(\mathcal{X})$ , the Yatracos class of  $\mathcal{Q}$  is given by
$$
\mathcal {H} = \{\{x \in \mathcal {X}: p (x) > q (x) \}: p \neq q \in \mathcal {Q} \}. ^ {2}
$$
Theorem 7.2. Let  $\mathcal{Q} \subseteq \Delta(\mathcal{X})$ . Let  $\mathcal{H}$  be the Yatracos class of  $\mathcal{Q}$ , denote by  $\mathrm{VC}(\mathcal{H})$  and  $\mathrm{VC}^*(\mathcal{H})$  the VC and dual VC dimension of  $\mathcal{H}$ .  $\mathcal{Q}$  is public-privately learnable with  $m$  public and  $n$  private samples, where
$$
m = O \left(\frac {\operatorname {V C} (\mathcal {H}) \log \left(\frac {1}{\alpha}\right) + \log \left(\frac {1}{\beta}\right)}{\alpha}\right) \quad a n d \quad n = O \left(\frac {\operatorname {V C} (\mathcal {H}) ^ {2} \operatorname {V C} ^ {*} (\mathcal {H}) + \log \left(\frac {1}{\beta}\right)}{\varepsilon \alpha^ {3}}\right).
$$
Theorem 7.2/1.5 says that classes of distributions whose Yatracos class have finite VC dimension can be public-privately learned. Note that the number of public samples used is indeed fewer than the  $O\left(\frac{\mathrm{VC}(\mathcal{H})}{\alpha^2}\right)$  sample requirement in the non-private analogue of the result (Fact H.1).
The proof is given in Appendix H. The result is a consequence of a known public-private uniform convergence result [BCM $^{+}$ 20, Theorem 10]. To adapt it to our setting, we (1) modify their result for pure DP (rather than approximate DP); and (2) conclude that uniform convergence over the Yatracos sets of  $\mathcal{Q}$  suffices to implement the learner from Fact H.1.
In this work, we connect public-private distribution learning to the notions of sample compression and list learning. In doing so, for broad classes of distributions, we introduce approaches to: (1) design sample-efficient public-private learners; (2) prove lower bounds on how much public data is required for public-private learnability. In the following, we list several questions for future study.
Question 8.1. For a class  $\mathcal{Q}$ , our work examines the minimal amount of public data needed to render  $\mathcal{Q}$  pure privately learnable. How much do we need to render  $\mathcal{Q}$  approximate DP learnable?
Question 8.2. The VC bound of Theorem 1.5/7.2, although more general, is "qualitatively" loose. For Gaussians, it yields a  $O\left(\frac{d^2}{\alpha}\right)$  public sample complexity, though we know  $O(d)$  is possible, notably with no dependence on  $\alpha$ . On the other hand,  $\Omega(1 / \alpha)$  public samples are required for public-privately learning mixtures of Gaussians. What qualities of a distribution admit public-private learning with  $\alpha$ -independent public sample complexity? Stronger: can we find more illuminating characterizations of the sample complexity of list learning?
Question 8.3. Our work studies sample complexity improvements from using public data. Can public data lead to algorithmic improvements, that is, runtime efficiency or simpler algorithms?
We thank the anonymous reviewers for their helpful feedback.
AB, GK, and VS acknowledge funding from an NSERC discovery grant. AB is supported by a David R. Cheriton Graduate Scholarship. CC is supported by an ARC DECRA and an unrestricted gift from Google. GK is supported by a Canada CIFAR AI Chair, an unrestricted gift from Google, a University of Waterloo startup grant, and an unrestricted gift from Apple.
2This is for when the distributions in  $\mathcal{Q}$  are discrete. For classes of continuous distributions, we substitute  $p$  and  $q$  for their respective density functions.