SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[SSBD14] Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, 2014.
[TB21] Florian Tramér and Dan Boneh. Differentially private learning needs better features (or much more data). In Proceedings of the 9th International Conference on Learning Representations, ICLR '21, 2021.
[TCK+22] Eliad Tsfadia, Edith Cohen, Haim Kaplan, Yishay Mansour, and Uri Stemmer. FriendlyCore: Practical differentially private aggregation. In Proceedings of the 39th International Conference on Machine Learning (ICML'22), 2022.
[TKC22] Florian Tramèr, Gautam Kamath, and Nicholas Carlini. Considerations for differentially private learning with large-scale public pretraining. arXiv preprint arXiv:2212.06470, 2022.
[VW02] Santosh Vempala and Grant Wang. A spectral algorithm for learning mixtures of distributions. In Proceedings of the 43rd Annual IEEE Symposium on Foundations of Computer Science, FOCS '02, pages 113-123, Washington, DC, USA, 2002. IEEE Computer Society.
$\left[\mathrm{XZA}^{+}23\right]$  Zheng Xu, Yanxiang Zhang, Galen Andrew, Christopher A. Choquette-Choo, Peter Kairouz, H. Brendan McMahan, Jesse Rosenstock, and Yuanbo Zhang. Federated learning of Gboard language models with differential privacy. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL'23), 2023.
[Yat85] Yannis G. Yatracos. Rates of convergence of minimum distance estimators and Kolmogorov's entropy. The Annals of Statistics, 13(2):768-774, 1985.
$\left[\mathrm{YNB}^{+}22\right]$  Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang. Differentially private fine-tuning of language models. In Proceedings of the 10th International Conference on Learning Representations, ICLR '22, 2022.
[YZCL21] Da Yu, Huishuai Zhang, Wei Chen, and Tie-Yan Liu. Do not let privacy overbill utility: Gradient embedding perturbation for private learning. In Proceedings of the 9th International Conference on Learning Representations, ICLR '21, 2021.
[ZWB21] Yingxue Zhou, Zhiwei Steven Wu, and Arindam Banerjee. Bypassing the ambient dimension: Private SGD with gradient subspace identification. In Proceedings of the 9th International Conference on Learning Representations, ICLR '21, 2021.
Privately learning Gaussians. Our work studies the task of learning arbitrary, unbounded Gaussians while offering differential privacy guarantees. Basic private algorithms for the task (variants of "clip-and-noise") impose boundedness assumptions on the underlying parameters of the unknown Gaussian, since their sample complexities grow to infinity as the bounds widen to include more allowed distributions. Understanding these dependencies without public data has been a topic of significant study. [KV18] examined univariate Gaussians, showing that logarithmic dependencies on parameter bounds are necessary and sufficient in the case of pure DP, but can be removed under approximate DP. The same is true in the multivariate setting [AAAK21, KMS+22, TCK+22, AL22, KMV22, LKO22]. [BKS22] shows that instead of relaxing to approximate DP to handle arbitrary Gaussians, one can employ a small amount of public data; our lower bound tells us almost exactly how much is needed. Furthermore, our reductions between public-private learning and list learning offers the conclusion that the role of public data is precisely for bounding: distribution classes that can be privately learned with a small amount of public data are exactly the distributions that can be bounded with a small amount of public data.
Privately learning mixtures of Gaussians. Another line of related work is that on privately learning mixtures of Gaussian, but without any public data. [NRS07] provided a subsample-and-aggregate approach to learn the parameters of mixtures of spherical Gaussians based on the work by [VW02] under the weaker, approximate DP. Recently,  $\mathrm{[CCd^{+}23]}$  improved on this by weakening the separation condition required for the mixture components. [KSSU19] provided the first polynomial-time, approximate DP algorithms to learn the parameters of mixtures of non-spherical Gaussians under weak boundedness assumptions.  $\mathrm{[CKM^{+}21]}$  improved on their work both in terms of the sample complexity and the separation assumption. [AAL23b] provided a polynomial-time reduction for privately and efficiently learning mixtures of unbounded Gaussians from the approximate DP setting to its non-private counterpart (albeit at a polynomial overhead in the sample complexity).
Our work falls into the category of private density estimation, for which [AAL21] gave new algorithms for the special case of spherical Gaussians under approximate DP, while [BKSW19] gave (computationally inefficient) algorithms for bounded Gaussians under pure DP. For comparison, the latter would have infinite sample complexity for unbounded Gaussians, but our work provides finite private sample complexity even under pure DP using public data. On the other hand, [ASZ21] showed hardness results for privately learning mixtures of Gaussians with known covariances. The concurrent work of Azfali, Ashtiani, and Liaw [AAL23a] gives the first learnability result for general high-dimensional mixtures of Gaussians under approximate differential privacy.
Theory for private algorithms with public data. Beyond distribution learning, there is a lot more work that investigates how public data can be employed in private data analysis. Some specific areas include private query release, synthetic data generation, and classification [JE13, BNS16, ABM19, NB20,  $\mathrm{BCM}^{+}20$ , BMN20,  $\mathrm{LVS}^{+}21$ , KADV23], and the results are a mix of theoretical versus empirical. The definition of public-private algorithms that we adopt is from [BNS16], which studied classification in the PAC model. The VC dimension bound we give for public-private distribution learning relies on results from public-private classification [ABM19] and uniform convergence  $[\mathrm{BCM}^{+}20]$ .
A concurrent and independent work [LLHR23] also studies learning with public and private data, focusing on the problems of mean estimation, empirical risk minimization, and stochastic convex optimization. The focus of the two works is somewhat different, both in terms of the type of problems considered (we study density estimation) and the type of results targeted. That is, our objective is to draw connections between different learning concepts and exploring the resulting implications for public-private distribution learning, while their goal seems to be understanding the precise error rates for some fundamental settings.
Private machine learning with public data. Within the context of private machine learning, there has been significant interest in how to best employ public data. The most popular method is pretraining  $\left[\mathrm{ACG}^{+}16, \mathrm{PCS}^{+}19, \mathrm{TB21}, \mathrm{LWAFF21}, \mathrm{YZCL21}, \mathrm{LTLH22}, \mathrm{YNB}^{+}22, \mathrm{GHN}^{+}23, \mathrm{HASP22}\right]$  (though some caution about this practice [TKC22]), while other methods involve computing statistics about the private gradients [ZWB21, YZCL21, KRRT21,  $\mathrm{AGM}^{+}22, \mathrm{GKW23}$ ], or training a student
model [PAE $^{+}$ 17, PSM $^{+}$ 18, BTGT18]. For more discussion of public data for private learning, see Section 3.1 of [CDE $^{+}$ 23].
List learning We also use the notion of list learning in this work, which is a "non-robust" version of the well-known list-decodable learning [AAL21, RY20, BBV08, CSV17, DKS18, KS17], where the goal is still to output a list of distributions that contains one that is accurate with respect to the true distribution, but the sampling may happen from a corrupted version of the underlying distribution.
Hybrid differential privacy. Finally, a related setting is the hybrid model, in which samples require either local or central differential privacy  $\mathrm{[AKZ^{+}17]}$ . Some learning tasks studied in this model include mean estimation [ADK20] and transfer learning [KS22].
B Technical overview
New connections for public-private learning. Our first contribution is establishing connections between sample compression, public-private learning, and list learning, via reductions. We show that: (1) sample compression schemes yield public-private learners; (2) public-private learners yield list learners; and (3) list learners yield sample compression schemes.
(1) and (3) are straightforward to prove: (1) compression implies public-private learning follows from a modification of an analogous result of  $\left[\mathrm{ABDH}^{+}20\right]$ , where we observe that in their proof, the learner's two-stage process of drawing a small compression sample used to generate a finite set of hypotheses using the compression scheme, followed by hypothesis selection with a larger sample, can be cleanly divided into using public and private samples respectively. In the latter stage, we employ a known pure DP hypothesis selection algorithm [BKSW19, AAAK21]. (3) List learning implies compression follows immediately from the definitions.
For (2) public-private learning implies list learning, we show non-constructively that there exists a list learner for a class, given a public-private learner for that class, but do not provide an algorithmic translation of the public-private learner to a list learner. For a set of samples  $S$ , we show that outputting a finite cover of the list of distributions on which the public-private learner succeeds on, when using  $S$  as the public samples, is a correct output for a list learner. Hence the list learner we construct, on input  $S$ , outputs this finite cover as determined by (but not explicitly constructed from) the public-private learner.
Agnostic and distribution-shifted public-private learning. We identify some distributional assumptions that can be relaxed in the public-private learning setup. We obtain agnostic and distribution-shifted public-private learners via a connection to robust compression schemes. The reduction ideas are similar to those in the case of public-private learning and non-robust sample compression described above.
Lower bound on public-private learning of Gaussians. Our lower bound for public-privately learning high-dimensional Gaussians exploits our above connection between public-private learning and list learning, and applies a "no-free-lunch"-style argument. The latter uses the fact that an algorithm's worst-case performance cannot be better than its performance when averaged across all the problem instances. In other words, we have two main steps in our proof: (1) we first claim that due to our above reduction, a lower bound for list learning high-dimensional Gaussians would imply a lower bound on the public sample complexity for public-privately learning high-dimensional Gaussians; and (2) assuming certain accuracy guarantees and a sample complexity for our list learner for high-dimensional Gaussians; we show that across a set of adversarially chosen problem instances, our average accuracy guarantee fails, which is a contradiction to our assumed worst-case guarantees.
We observe that the lower bound from Theorem 6.1 establishes that at least  $d$  public samples are necessary for public-private learning to vanishingly small error as  $d$  increases (impossibility of learning to a target error, via the application of Lemma G.1, is related to the bound on  $\eta$  from Equation (6), which decreases exponentially with  $d$ ). A natural question is whether the result can be strengthened to say that there is a single target error, simultaneously for all  $d$ , for which learning is impossible without  $d$  public samples.
VC dimension upper bound for public-private learning. Our proof involves invoking the existing results for public-private binary classification [ABM19] and uniform convergence  $\left[\mathrm{BCM}^{+}20\right]$ . We use them to implement Yatracos' minimum distance estimator [Yat85, DL01] in a public-private way.
C Extended preliminaries
Covers and packings. For  $\alpha > 0$  and  $\mathcal{Q} \subseteq \Delta(\mathcal{X})$ , we say that  $\mathcal{C} \subseteq \Delta(\mathcal{X})$  is an  $\alpha$ -cover of  $\mathcal{Q}$  if for any  $q \in \mathcal{Q}$ , there exists a  $p \in \mathcal{C}$  with  $\mathrm{TV}(p, q) \leq \alpha$ . For  $\alpha > 0$  and  $\mathcal{Q} \subseteq \Delta(\mathcal{X})$ , we say that  $\mathcal{P} \subseteq \mathcal{Q}$  is an  $\alpha$ -packing of  $\mathcal{Q}$  if for any  $p \neq q \in \mathcal{P}$ ,  $\mathrm{TV}(p, q) > \alpha$ .
Class of  $k$ -mixtures. Let  $\mathcal{Q} \subseteq \Delta(\mathcal{X})$  be a class of distributions. For any  $k \geq 1$ , the class of  $k$ -mixtures of  $\mathcal{Q}$  is given by
$$
\mathcal {Q} ^ {\oplus k} := \left\{\sum_ {i = 1} ^ {k} w _ {i} q _ {i}: q _ {i} \in \mathcal {Q}, w _ {i} \geq 0 \text {f o r a l l} i \in [ k ] \text {a n d} \sum_ {i = 1} ^ {k} w _ {i} = 1 \right\}.
$$
Class of  $k$ -products. Let  $\mathcal{Q} \subseteq \Delta(\mathcal{X})$  be a class of distributions over  $\mathcal{X}$ . For any  $k \geq 1$ ,  $q = (q_1, \ldots, q_k)$  is a product distribution over  $\mathcal{X}^k$ , if  $q_i \in \mathcal{Q}$  for all  $i \in [k]$  and for  $X \sim q$ , the  $i$ -th component  $X_i$  of  $X$  is independently (of all the other coordinates) sampled from  $q_i$ . The class of  $k$ -products of  $\mathcal{Q}$  over  $\mathcal{X}^k$  is given by
$$
\mathcal {Q} ^ {\otimes k} := \left\{\left(q _ {1}, \dots , q _ {k}\right): q _ {i} \in \mathcal {Q} \text {f o r a l l} i \in [ k ] \right\}.
$$
C.2 Privacy
The following is a known hardness result on density estimation of distributions under pure differential privacy, and is based on the standard "packing lower bounds".
Fact C.1 (Packing lower bound [BKSW19, Lemma 5.1]). Let  $\mathcal{Q} \subseteq \Delta(\mathcal{X})$ ,  $\alpha \in (0,1]$ , and  $\varepsilon > 0$ . Let  $\widehat{\mathcal{Q}}$  be any  $\alpha$ -packing of  $\mathcal{Q}$ . Any  $\varepsilon$ -DP algorithm  $\mathcal{A} \colon \mathcal{X}^n \to \Delta(\Delta(\mathcal{X}))$  that, upon receiving  $n$  i.i.d. samples  $X_1, \ldots, X_n$  from any  $p \in \mathcal{Q}$ , outputs  $Q$  with  $\mathrm{TV}(Q,p) \leq \frac{\alpha}{2}$  with probability  $\geq \frac{9}{10}$  requires
$$
n \geq \frac {\log (| \widehat {\mathcal {Q}} |) - \log (\frac {1 0}{9})}{\varepsilon}.
$$
The next result guarantees the existence of agnostic learners for finite hypothesis classes under pure differential privacy.
Fact C.2 (Pure DP 3-agnostic learner for finite  $\mathcal{Q}$  [BKSW19], [AAAK21, Theorem 2.24]). Let  $\mathcal{Q} \subseteq \Delta(\mathcal{X})$  with  $|\mathcal{Q}| < \infty$ . For every  $\alpha, \beta \in (0,1]$  and  $\varepsilon > 0$ , there exists an  $\varepsilon$ -DP algorithm  $\mathcal{A} \colon \mathcal{X}^n \to \Delta(\Delta(\mathcal{X}))$ , such that for any  $p \in \Delta(\mathcal{X})$ , if we draw a dataset  $\mathbf{X} = (X_1, \dots, X_n)$  i.i.d. from  $p$  and then  $Q \sim \mathcal{A}(\mathbf{X})$ ,
$$
\mathop{\mathbb{P}}_{\substack{\boldsymbol {X}\sim p^{n}\\ Q\sim \mathcal{A}(\boldsymbol {X})}}\left\{\mathrm{TV}(Q,p)\leq 3\cdot \mathrm{dist}(p,\mathcal{Q}) + \alpha \right\} \geq 1 - \beta ,
$$
$$
n = O \left(\frac {\log (| \mathcal {Q} |) + \log (\frac {1}{\beta})}{\alpha^ {2}} + \frac {\log (| \mathcal {Q} |) + \log (\frac {1}{\beta})}{\alpha \varepsilon}\right).
$$
D Statements and proofs for Section 3 - The connection to sample compression schemes
The following is the full formal statement of Theorem 1.1.
Theorem D.1 (Sample complexity equivalence between sample compression, public-private learning, and list learning). Let  $\mathcal{Q} \subseteq \Delta(\mathcal{X})$ . Let  $m : (0,1]^2 \to \mathbb{N}$  be a sample complexity function, such that  $m(\alpha, \beta) = \mathrm{poly}\left(\frac{1}{\alpha}, \frac{1}{\beta}\right)$ . Then the following are equivalent.
3The reductions between the learners do not need this assumption, it is only used to state the sample complexity equivalence.
1.  $\mathcal{Q}$  is realizably compressible with  $m_C(\alpha, \beta) = O(m(\alpha, \beta))$  samples.
2.  $\mathcal{Q}$  is public-privately learnable with  $m_P(\alpha, \beta, \epsilon) = O(m(\alpha, \beta))$  public samples.
3.  $\mathcal{Q}$  is list learnable with  $m_{L}(\alpha ,\beta) = O(m(\alpha ,\beta))$  samples.
The functions  $m_{C}$ ,  $m_{P}$ , and  $m_{L}$  are related to one another as:  $m_{P}(\alpha, \beta, \varepsilon) = m_{C}(\frac{\alpha}{6}, \frac{\beta}{2})$ ;  $m_{L}(\alpha, \beta) = m_{P}(\frac{\alpha}{2}, \frac{\beta}{10}, \varepsilon)$  for any  $\varepsilon > 0$ ; and  $m_{C}(\alpha, \beta) = m_{L}(\alpha, \beta)$ . Hence, if there exists a polynomial  $m : (0,1]^{2} \to \mathbb{N}$ , such that  $m_{C}(\alpha, \beta) = O(m(\alpha, \beta))$ , then  $m_{C}(\alpha, \beta)$ ,  $m_{P}(\alpha, \beta)$ , and  $m_{L}(\alpha, \beta)$  are all within constant factors of each other.
D.1 Compression implies public-private learning
Proof of Proposition 3.1. The proof this proposition closely mirrors that of Theorem 4.5 from  $\mathrm{[ABDH^{+}20]}$ . We adapt their result to the public-private setting.
Fix  $\alpha, \beta \in (0,1]$  and  $\varepsilon > 0$ . Let  $\tau = \tau\left(\frac{\alpha}{6}, \frac{\beta}{2}\right)$ ,  $t = t\left(\frac{\alpha}{6}, \frac{\beta}{2}\right)$ , and  $m = m_C\left(\frac{\alpha}{6}, \frac{\beta}{2}\right)$ . We draw a public dataset  $\tilde{X}$  of size  $m$  i.i.d. from  $p$ . Consider
$$
\mathcal {S} := \left\{\left(\boldsymbol {S} ^ {\prime}, b\right): \boldsymbol {S} ^ {\prime} \subseteq \tilde {\boldsymbol {X}} \text {w h e r e} | \boldsymbol {S} ^ {\prime} | = \tau , \text {a n d} b \in \{0, 1 \} ^ {t} \right\}.
$$
Note that the encoding  $f_{p}(\tilde{\boldsymbol{X}}) \in S$ , so forming  $\widehat{\mathcal{Q}} = \{g(S', b) : (S', b) \in S\}$  means that with probability  $\geq 1 - \frac{\beta}{2}$  over the sampling of  $\tilde{\boldsymbol{X}}$ ,  $q = g(f_{p}(\tilde{\boldsymbol{X}})) \in \widehat{\mathcal{Q}}$  has  $\mathrm{TV}(q, p) \leq \frac{\alpha}{6}$ .
Now, we run the  $\varepsilon$ -DP 3-agnostic learner from Fact C.2 on  $\widehat{\mathcal{Q}}$ , targeting error  $\frac{\alpha}{2}$  and failure probability  $\frac{\beta}{2}$ , which is achieved as long as we have  $n$  private samples (given in the statement of Proposition 3.1), which is logarithmic in  $|S|$ . With probability  $\geq 1 - \beta$ , we approximately recover  $p$  with the compression scheme and the DP learner succeeds, and so the output  $Q$  satisfies
$$
\begin{array}{l} \operatorname {T V} (Q, p) \leq 3 \cdot \min  _ {q \in \mathcal {Q}} \operatorname {T V} (p, q) + \frac {\alpha}{2} \\ \leq 3 \cdot \frac {\alpha}{6} + \frac {\alpha}{2} = \alpha . \\ \end{array}
$$
D.2 List learning implies sample compression
Proof of Proposition 3.5. Fix any  $\alpha, \beta \in (0,1]$ . Let  $m = m_L(\alpha, \beta)$  and  $\ell = \ell(\alpha, \beta)$ . By assumption,  $\mathcal{Q}$  admits an  $(\alpha, \beta, \ell)$ -list learner  $\mathcal{L}: \mathcal{X}^m \to \{L \subseteq \Delta(\mathcal{X}) : |L| \leq \ell\}$  that takes  $m$  samples. Letting  $\tau = m$  and  $t = \log_2(\ell)$ , we define the compression scheme as follows.
- Encoder: for any  $q \in \mathcal{Q}$ , the encoder  $f_{q}: \mathcal{X}^{m} \to \mathcal{X}^{\tau} \times \{0,1\}^{t}$  produces the following, given an input  $\tilde{\boldsymbol{x}} \in \mathcal{X}^{m}$ . It first runs the list learner on  $\tilde{\boldsymbol{x}}$ , obtaining  $\mathcal{L}(\tilde{\boldsymbol{x}})$ . Then, it finds the smallest index  $i$  with  $\mathrm{TV}(q,\mathcal{L}(\tilde{\boldsymbol{x}})_i) = \mathrm{dist}(q,\mathcal{L}(\tilde{\boldsymbol{x}}))$ , where  $\mathcal{L}(\tilde{\boldsymbol{x}})_i$  denotes the  $i$ -th element of the list  $\mathcal{L}(\tilde{\boldsymbol{x}})$ . The output of the list learner is  $(\tilde{\boldsymbol{x}},i)$ . Note that  $\tilde{\boldsymbol{x}} \in \mathcal{X}^{\tau}$  and that  $i$  can be represented with  $\log_2(\ell) = t$  bits.
- Decoder: the fixed decoder  $g:\mathcal{X}^{\tau}\times \{0,1\}^{t}\to \Delta (\mathcal{X})$  takes  $\tilde{\pmb{x}}$  and  $i$ , runs the list learner  $\mathcal{L}$  on  $\tilde{\pmb{x}}$ , and produces  $\mathcal{L}(\tilde{\pmb{x}})_i$ .
By the guarantee of the list learner, we indeed have for any  $q \in \mathcal{Q}$ , with probability  $\geq 1 - \beta$  over the sampling of  $\tilde{\pmb{X}} \sim q^{m}$ ,  $\mathrm{TV}(q, g(f_{q}(S))) \leq \alpha$ .
E Statements and proofs for Section 4 - Applications
E.1 Public-private learnability of Gaussians and mixtures of Gaussians
Corollary E.1 (Public-private learning for Gaussians). Let  $d \geq 1$ . The class of Gaussians over  $\mathbb{R}^d$  is public-privately learnable with  $m(\alpha, \beta, \varepsilon)$  public samples and  $n(\alpha, \beta, \varepsilon)$  private samples, where
$$
\begin{array}{l} m (\alpha , \beta , \varepsilon) = O \left(d \log \left(\frac {1}{\beta}\right)\right), \\ n (\alpha , \beta , \varepsilon) = O \left(\frac {d ^ {2} \log \left(\frac {d}{\alpha}\right) + \log \left(\frac {1}{\beta}\right)}{\alpha^ {2}} + \frac {d ^ {2} \log \left(\frac {d}{\alpha}\right) + \log \left(\frac {1}{\beta}\right)}{\alpha \varepsilon}\right). \\ \end{array}
$$
Corollary E.2 (Public-private learning for mixtures of Gaussians). Let  $d, k \geq 1$ . The class of all  $k$ -mixtures of Gaussians over  $\mathbb{R}^d$  is public-privately learnable with  $m(\alpha, \beta, \varepsilon)$  public samples and  $n(\alpha, \beta, \varepsilon)$  private samples, where
$$
\begin{array}{l} m (\alpha , \beta , \varepsilon) = O \left(\frac {k d \log \left(\frac {k}{\beta}\right) \log \left(\frac {1}{\beta}\right)}{\alpha}\right), \\ n (\alpha , \beta , \varepsilon) = O \left(\left(\frac {1}{\alpha^ {2}} + \frac {1}{\varepsilon \alpha}\right) \cdot \left(k d ^ {2} \log \left(\frac {d}{\alpha}\right) + k d \log \left(\frac {k d \log \left(\frac {k}{\beta}\right)}{\alpha}\right) + \log \left(\frac {1}{\beta}\right)\right)\right). \\ \end{array}
$$
E.2 Public-private learnability of mixture and product distributions
Mixture distributions. We first mention a fact from  $\left[\mathrm{ABDH}^{+}20\right]$ , which says that if a compression scheme exists for a class of distributions  $\mathcal{Q}$ , then there exists a compression scheme for the class of  $k$ -mixtures of  $\mathcal{Q}$ .
Fact E.3 (Compression for mixture distributions [ABDH+20, Lemma 4.8]). If a class of distributions  $\mathcal{Q}$  admits  $(\tau (\alpha ,\beta),t(\alpha ,\beta),m(\alpha ,\beta))$  realizable sample compression, then for any  $k\geq 1$ , the class of  $k$ -mixtures of  $\mathcal{Q}$  admits  $(\tau_{k}(\alpha ,\beta),t_{k}(\alpha ,\beta),m_{k}(\alpha ,\beta))$  realizable sample compression, where  $\tau_k,t_k,m_k:(0,1]^2\to \mathbb{N}$  are as follows:
$$
\begin{array}{l} \tau_ {k} (\alpha , \beta) = k \cdot \tau \left(\frac {\alpha}{3}, \beta\right), \quad t _ {k} (\alpha , \beta) = k \cdot t \left(\frac {\alpha}{3}, \beta\right) + \log_ {2} \left(\frac {3 k}{\alpha}\right), \\ m _ {k} (\alpha , \beta) = \frac {4 8 k \log \left(\frac {6 k}{\beta}\right)}{\alpha} \cdot m \left(\frac {\alpha}{3}, \beta\right). \\ \end{array}
$$
Next, we state a corollary of Propositions 3.4 and 3.5, which describes the existence of a compression scheme, given the existence of a public-private learner.
Corollary E.4 (Public-private learning  $\Longrightarrow$  compression). Let  $\mathcal{Q} \subseteq \Delta(\mathcal{X})$  be a class of distributions. Suppose  $\mathcal{Q}$  is public-privately learnable with  $m_P(\alpha, \beta, \varepsilon)$  public samples and  $n(\alpha, \beta, \varepsilon)$  private samples. Then for any  $\varepsilon > 0$ ,  $\mathcal{Q}$  admits
$$
(\tau (\alpha , \beta), t (\alpha , \beta), m (\alpha , \beta)) = \left(m _ {P} \left(\frac {\alpha}{2}, \frac {\beta}{1 0}, \varepsilon\right), \frac {\log \left(\frac {1 0}{9}\right) + \varepsilon \cdot n \left(\frac {\alpha}{2} , \frac {\beta}{1 0} , \varepsilon\right)}{\log (2)}, m _ {P} \left(\frac {\alpha}{2}, \frac {\beta}{1 0}, \varepsilon\right)\right)
$$
realizable sample compression.