Title: Private Distribution Learning with Public Data: The View from Sample Compression
Venue: NeurIPS 2023
Accepted: yes
Keywords: differential privacy

distribution learning

gaussians

mixture of gaussians

compression schemes

robust compression schemes

privacy
Abstract: We study the problem of private distribution learning with access to public data. In this setup, which we refer to as *public-private learning*, the learner is given public and private samples drawn from an unknown distribution $p$ belonging to a class $\mathcal Q$, with the goal of outputting an estimate of $p$ while adhering to privacy constraints (here, pure differential privacy) only with respect to the private samples. 
    
We show that the public-private learnability of a class $\mathcal Q$ is connected to the existence of a sample compression scheme for $\mathcal Q$, as well as to an intermediate notion we refer to as \emph{list learning}. Leveraging this connection: (1) approximately recovers previous results on Gaussians over $\mathbb R^d$; and (2) leads to new ones, including sample complexity upper bounds for arbitrary $k$-mixtures of Gaussians over $\mathbb R^d$, results for agnostic and distribution-shift resistant learners, as well as closure properties for public-private learnability under taking mixtures and products of distributions. Finally, via the connection to list learning, we show that for Gaussians in $\mathbb R^d$, at least $d$ public samples are necessary for private learnability, which is close to the known upper bound of $d+1$ public samples.

Private Distribution Learning with Public Data: The View from Sample Compression ∗

Abstract

We study the problem of private distribution learning with access to public data. In this setup, which we refer to as public-private learning , the learner is given public and private samples drawn from an unknown distribution p belonging to a class Q , with the goal of outputting an estimate of p while adhering to privacy constraints (here, pure differential privacy) only with respect to the private samples. We show that the public-private learnability of a class Q is connected to the existence of a sample compression scheme for Q , as well as to an intermediate notion we refer to as list learning . Leveraging this connection: (1) approximately recovers previous results on Gaussians over R d ; and (2) leads to new ones, including sample complexity upper bounds for arbitrary k -mixtures of Gaussians over R d , results for agnostic and distribution-shift resistant learners, as well as closure properties for public-private learnability under taking mixtures and products of distributions. Finally, via the connection to list learning, we show that for Gaussians in R d , at least d public samples are necessary for private learnability, which is close to the known upper bound of d +1 public samples.

1 Introduction

Statistical analysis of sensitive data, and specifically parameter and density estimation, is a workhorse of privacy-preserving machine learning. To provide meaningful and rigorous guarantees on algorithm for these tasks, the framework of differential privacy (DP) [DMNS06] has been widely adopted by both algorithm designers and machine learning practitioners [App17, Abo18, XZA + 23], and is, by and large, one of the past decade's success stories in principled approaches to private machine learning, with a host of results and implementations [Goo19a, Goo19b, HBAL19, Ope20] for many of the flagship private learning tasks.

Yet, however usable the resulting algorithms may be, DP often comes at a steep price: namely, many estimation tasks simple without privacy constraints provably require much more data to be performed privately; even more dire, they sometimes become impossible with any finite number of data points, absent some additional strong assumptions.

The prototypical example in that regard is learning a single d -dimensional Gaussian distribution from samples. In this task, a learner receives i.i.d. samples from an unknown d -dimensional Gaussian p

and is tasked with finding an estimate q close to p in total variation ( TV ) distance. Without privacy constraints, it is folklore that this can be done with O ( d 2 ) samples; yet once privacy enters the picture, in the form of pure differential privacy, no finite sample algorithm for this task can exist , unless a bound on the mean vector and covariance matrix are known.

This 'cost of privacy' is, unfortunately, inherent to many estimation tasks, as the positive results (algorithms) developed over the years have been complemented with matching negative results (lower bounds). In light of these strong impossibility results, it is natural to wonder if one could somehow circumvent this often steep privacy cost by leveraging other sources of public data to aid the private learning process.

Recent work in finetuning machine learning models has tried to address the question whether, in situations where a vast amount of public data is available, one can combine the public data with a relatively small amount of private data to somehow achieve privacy guarantees for the private data and learning guarantees that would otherwise be ruled out by the aforementioned impossibility results. We, on the other hand, address the same question, but in the opposite setting, i.e., when the amount of public data available is much smaller than the amount of private data available. In other words, we answer the following question from new perspectives.

Can one leverage small quantities of public data to privately learn from sensitive data, even when private learning is impossible?

This question was the focus of a recent study of Bie, Kamath, and Singhal [BKS22], the starting point of our work. In this paper, we make significant strides in this direction, by obtaining new 'plug and play' results and connections in this public-private learning setting, and using them to obtain new sample complexity bounds for a range of prototypical density estimation tasks.

1.1 Our results

First, we establish a connection between learning (in the sense of distribution learning or density estimation ) with public and private data (Definition 2.4) and sample compression schemes for distributions (Definition 2.5; see [ABDH + 20, Definition 4.2]), as well as an intermediate notion we refer to as list learning (Definition 3.2).

Theorem 1.1 (Sample compression schemes, public-private learning, and list learning (Informal; see Theorem D.1)) . Let Q be a class of probability distributions and m ( α, β ) be a sample complexity function in terms of target error α and failure probability β . Then the following are equivalent.

Q has a sample compression scheme using O ( m ( α, β )) samples.

Q is public-privately learnable with O ( m ( α, β )) public samples.

Q is list learnable with O ( m ( α, β )) samples.

Despite its technical simplicity, this sample complexity equivalence turns out to be quite useful, and allows us to derive new public-private learners for an array of key distribution classes by leveraging known results on sample compression schemes. In particular, from the connection to sample compression schemes we are able to obtain new public-private learners for: (1) highdimensional Gaussian distributions (Corollary E.1); (2) arbitrary mixtures of high-dimensional Gaussians (Theorem 1.2/Corollary E.2); (3) mixtures of public-privately learnable distribution classes (Theorem 4.3/E.5); and (4) products of public-privately learnable distribution classes (Theorem 4.3/E.7). For instance, the following is a consequence of the above connection.

Theorem 1.2 (Public-private learning for mixtures of Gaussians (Informal; see Corollary E.2)) . The class of mixtures of k arbitrary d -dimensional Gaussians is public-privately learnable with m public samples and n private samples, where

in which α is the target error and ε the privacy parameter.

We also examine public-private distribution learning in a setting with relaxed distributional assumptions, in which the distributions underlying the public and private data: (1) may differ (the case of

public-private distribution shift ); and (2) may not be members of the reference class of distributions, and so we instead ask for error close to the best approximation of the private data distribution by a member of the class (the agnostic case). We show that robust sample compression schemes for a class of distributions can be converted into public-private learners in this agnostic and distributionshifted setting. As a consequence, we have the following result for learning distributions that can be approximated by Gaussians under public-private distribution shift.

Theorem 1.3 (Agnostic and distribution-shifted public-private learning for Gaussians (Informal; see Corollary F.1)) . There is a public-private learner that takes m public samples and n private samples from any pair of distributions ˜ p and p over R d respectively, with TV(˜ p, p ) ≤ 1 3 , where

in which α is the target error and ε the privacy parameter. With probability ≥ 9 10 , the learner outputs q with TV( q, p ) ≤ 3OPT+ α , where OPT is the total variation distance between p and the closest d -dimensional Gaussian to it.

Next, using the aforementioned connection to list learning, we are able to establish a fine-grained lower bound on the number of public data points required to privately learn high-dimensional Gaussians.

Theorem 1.4 (Almost tight lower bound on privately learning Gaussians with public data (Informal; see Theorem 6.1)) . The class of all d -dimensional Gaussians is not public-privately learnable with fewer than d public samples, regardless of the number of private samples.

[BKS22] showed that d -dimensional Gaussians are public-privately learnable with ˜ O ( d 2 α 2 + d 2 αε ) private samples, as soon as d +1 public samples are available . Thus, our result shows a very sharp threshold for the number of public data points necessary and sufficient to make private learning possible.

We also provide a general result for public-privately learning classes of distributions whose Yatracos class has finite VC dimension.

Theorem 1.5 (VC dimension bound for public-private learning (Informal; see Theorem 7.2)) . Let Q be a class of probability distributions over a domain X such that the Yatracos class of Q , defined as

has bounded VC dimension. Denote by VC( H ) and VC ∗ ( H ) the VC and dual VC dimensions of H respectively. Then Q is public-privately learnable with m public samples and n private samples, where

in which α is the target error and ε the privacy parameter.

The ˜ O ( VC( H ) α ) public sample requirement is less than the known O ( VC( H ) α 2 ) sample requirement to learn with only public data via the non-private analogue of this result [Yat85, DL01].

1.2 Related work

The most closely related work is that of Bie, Kamath, and Singhal [BKS22], which initiated the study of distribution learning with access to both public and private data. That work studied algorithms for specific canonical distribution classes, while the present paper aims to broaden our understanding of public-private distribution learning in general, via connections to other problems and providing more general approaches for devising sample efficient public-private learners. In addition, we prove the first lower bounds on the amount of public data needed for private distribution learning.

There is a long line of work on private distribution learning, especially with regards to Gaussians and mixtures of Gaussians. [KV18] studied univariate Gaussians, showing that logarithmic dependencies on parameter bounds are necessary and sufficient in the case of pure DP, but can be removed under approximate DP. The same is true in the multivariate setting [KLSU19, KMS + 22, TCK + 22, AL22,

KMV22, LKO22]. For mixtures of Gaussians, most studies have focused on parameter estimation of mixture components [NRS07, KSSU19, CKM + 21, CCd + 23, AAL23b], and employ component separation and mixing weight assumptions. For density estimation (the setting studied in this work) [BKSW19, AAL21] give learnability results under various structural assumptions. The concurrent work of Azfali, Ashtiani, and Liaw [AAL23a] gives the first learnability result for general highdimensional mixtures of Gaussians under approximate differential privacy.

Outside of distribution learning, there is a significant interest in using public data to improve private algorithms. Some problems studied include private query release, synthetic data generation, classification, mean estimation, empirical risk minimization, and stochastic convex optimization [JE13, BNS16, ABM19, NB20, BCM + 20, BMN20, LVS + 21, KADV23, LLHR23]. The definition of public-private algorithms that we adopt is from [BNS16], which studied classification in the PAC model. The VC dimension bound we give for public-private distribution learning relies on results from public-private classification [ABM19] and uniform convergence [BCM + 20].

On the technical side, we establish connections with distribution compression schemes as introduced by [ABDH + 20], and directly apply their results to establish new results for public-public learning. Related compression schemes for PAC learning for binary classification have been shown to be necessary and sufficient for learnability in those settings [LW86, MY16]. For further discussion of related work, please see Appendix A.

1.3 Limitations

Our work investigates the sample complexity of public-private learning, and does not give computationally efficient learners, or in some cases, algorithmic learners that run in finite time. 1 In particular, all public-private learners we obtain from sample compression run in time exponential in the sample complexity. Also, for our VC dimension upper bounds, we enumerate all realizable labellings of the input sample as per the relevant Yatracos class H , which is not computable for general H [AABD + 20]. Finally, dependence on VC ∗ ( H ) in sample complexity is not ideal, as VC ∗ ( H ) ≤ 2 VC( H )+1 -1 is the best possible upper bound in terms of VC( H ) for general H [Ass83].

2 Preliminaries

2.1 Notation

We denote by X the domain of examples . For a domain U , denote by ∆( U ) the set of all probability distributions over U . We refer to a set Q ⊆ ∆( X ) as a class of distributions over X .

We equip ∆( X ) with the total variation metric, which is defined as follows: for p, q ∈ ∆( X ) , TV( p, q ) := sup B ∈B | p ( B ) -q ( B ) | , where B are the measurable sets of X . For p ∈ ∆( X ) and a set of distributions L ⊆ ∆( X ) , we denote their point-set distance by dist( p, L ) := inf q ∈ L TV( p, q ) .

We will let ˜ x = (˜ x 1 , . . . , ˜ x m ) ∈ X m denote a public dataset and x = ( x 1 , . . . , x n ) ∈ X n denote a private dataset . Their respective capital versions ˜ X , X denote random variables for datasets realized by sampling from some underlying distribution. For p ∈ ∆( X ) , we denote by p m the distribution over X m obtained by concatenating m i.i.d. samples from p .

2.2 Public-private learning

Definition 2.1 (Differential privacy [DMNS06]) . Fix an input space X and an output space Y . Let ε, δ > 0 . A randomized algorithm A : X n → ∆( Y ) is ( ε, δ ) -differentially private (( ε, δ )-DP) , if for any private datasets x , x ′ ∈ X n differing in one entry,

The case where δ = 0 is referred to as pure differential privacy or ε -DP.

Our focus is on understanding the public data requirements for privately learning different classes of distributions. We seek to answer the following question:

For a class of distributions Q , how much public data is necessary and sufficient to render Q privately learnable?

To do so, we give the formal notion of 'public-private algorithms' - algorithms that take public data samples and private data samples as input, and guarantee differential privacy with respect to the private data - as studied previously in the setting of binary classification [BNS16, ABM19]. We restrict our attention to public-private algorithms that offer a pure DP guarantee to private data.

Definition 2.2 (Public-private ε -DP) . Fix an input space X and an output space Y . Let ε > 0 . A randomized algorithm A : X m ×X n → ∆( Y ) is public-private ε -DP if for any public dataset ˜ x ∈ X m , the randomized algorithm A ( ˜ x , · ) : X n → ∆( Y ) is ε -DP.

Definition 2.3 (Public-private learner) . Let Q ⊆ ∆( X ) . For α, β ∈ (0 , 1] and ε > 0 , an ( α, β, ε ) -public-private learner for Q is a public-private ε -DP algorithm A : X m ×X n → ∆(∆( X )) , such that for any p ∈ Q , if we draw datasets ˜ X = ( ˜ X 1 , ..., ˜ X m ) and X = ( X 1 , ..., X n ) i.i.d. from p and then Q ∼ A ( ˜ X , X ) ,

Definition 2.4 (Public-privately learnable class) . We say that a class of distributions Q ⊆ ∆( X ) is public-privately learnable with m ( α, β, ε ) public and n ( α, β, ε ) private samples if for any α, β ∈ (0 , 1] and ε > 0 , there exists an ( α, β, ε ) -public-private learner for Q that takes m = m ( α, β, ε ) public samples and n = n ( α, β, ε ) private samples.

When Q satisfies the above, we may omit the private sample requirement, and say that Q is publicprivately learnable with m ( α, β, ε ) public samples .

Denote by SC Q ( α, β ) the sample complexity of learning Q . Our primary interest lies in determining when non-privately learnable Q can be public-privately learned with m ( α, β, ε ) = o ( SC Q ( α, β )) public samples (at a target ε ).

2.3 Sample compression schemes

One of the main techniques that we use in this work to create public-private learners for various distribution families is the robust sample compression scheme from [ABDH + 20]. Roughly speaking, if every member q of a class of distributions Q admits a way to encode enough information about itself in a small number of samples from q and extra bits, such that it can be approximately reconstructed by a fixed and deterministic decoder, Q can be learned.

Definition 2.5 (Robust sample compression [ABDH + 20, Definition 4.2]) . Let r ≥ 0 . We say Q ⊆ ∆( X ) admits ( τ ( α, β ) , t ( α, β ) , m ( α, β )) r -robust sample compression if for any α, β ∈ (0 , 1] , letting τ = τ ( α, β ) , t = t ( α, β ) , m = m ( α, β ) , there exists a decoder g : X τ ×{ 0 , 1 } t → ∆( X ) , such that the following holds:

For any q ∈ Q there exists an encoder f q : X m →X τ ×{ 0 , 1 } t satisfying for all x ∈ X m that for all i ∈ [ τ ] , there exists j ∈ [ m ] with f q ( x ) i = x j , such that for every p ∈ ∆( X ) with TV( p, q ) ≤ r , if we draw a dataset X = ( X 1 , ..., X m ) i.i.d. from p , then

When Q satisfies the above, we may omit the compression size complexity function τ ( α, β ) and bit complexity function t ( α, β ) , and say that Q is r -robustly compressible with m ( α, β ) samples. When r = 0 , we say that Q admits ( τ ( α, β ) , t ( α, β ) , m ( α, β )) realizable compression and is realizably compressible with m ( α, β ) samples.

Both robust and realizable compression schemes satisfy certain useful properties that we use to develop public-private distribution learners in different settings. For example, the existence of a realizable compression scheme is closed under taking mixtures or products of distributions.

3 The connection to sample compression schemes

In this section we prove Theorem 1.1/D.1, the sample complexity equivalence between sample compression (Definition 2.5), public-private learning (Definition 2.4), and an intermediate notion we refer to as list learning (Definition 3.2). We do so by giving sample-efficient reductions between the three notions (Propositions 3.1, 3.4, and 3.5). The propositions state the quantitative translations between: compression size τ and number of bits t , number of private samples n , and list size ℓ .

3.1 Compression implies public-private learning

Westart by establishing that the existence of a sample compression scheme for Q implies the existence of a public-private learner for Q .

Proposition 3.1 (Compression = ⇒ public-private learning) . Let Q ⊆ ∆( X ) . Suppose Q admits ( τ ( α, β ) , t ( α, β ) , m C ( α, β )) realizable sample compression. Then Q is public-privately learnable with m ( α, β, ε ) = m C ( α 6 , β 2 ) public and n ( α, β, ε ) = O (( 1 α 2 + 1 αε ) · ( t ( α 6 , β 2 ) + τ ( α 6 , β 2 ) log( m C ( α 6 , β 2 )) + log( 1 β ))) private samples.

Proof sketch. The full proof is given in Appendix D.1, and mirrors that of Theorem 4.5 from [ABDH + 20] (albeit adapted to the public-private setting). Fix α, β ∈ (0 , 1] and ε > 0 . Let τ = τ ( α 6 , β 2 ) , t = t ( α 6 , β 2 ) , and m = m C ( α 6 , β 2 ) . We draw a public sample ˜ X of size m , and enumerate all combinations of a size τ subset of ˜ X with a binary string of length t . Essentially, we 'guess' the encoding of the unknown distribution p . Running the decoder on this set of possible encodings gives us a set of distributions; some of them are close to p . Then, with private samples only, we use the pure DP 3 -agnostic learner for finite classes (Fact C.2) to pick one such out.

3.2 Public-private learning implies list learning

Next, we show that the existence of a public-private learner for a class of distributions implies the existence of a list learner for the class. A list learner for a class Q takes input samples from any p ∈ Q and outputs a finite list of distributions, one of which is close to p .

Definition 3.2 (List learner) . Let Q ⊆ ∆( X ) . For α, β ∈ (0 , 1] and ℓ ∈ N , an ( α, β, ℓ ) -list learner for Q is an algorithm L : X m → { L ⊆ ∆( X ) : | L | ≤ ℓ } , such that for any p ∈ Q , if we draw a dataset X = ( X 1 , . . . , X m ) i.i.d. from p , then

Definition 3.3 (List learnable class) . A class of distributions Q ⊆ ∆( X ) is list learnable to list size ℓ ( α, β ) with m ( α, β ) samples if for every α, β ∈ (0 , 1] , letting ℓ = ℓ ( α, β ) and m = m ( α, β ) , there is an ( α, β, ℓ ) -list-learner for Q that takes m samples.

If Q satisfies the above, irrespective of the list size complexity ℓ ( α, β ) , we say Q is list learnable with m ( α, β ) samples .

Now, we state our reduction of list learning to public-private learning. The key step of our proof is showing that, upon receiving samples ˜ x , outputting a finite cover of the list of distributions that a public-private learner would succeed on given public data ˜ x is a successful strategy for list learning.

Proposition 3.4 (Public-private learning = ⇒ list learning) . Let Q ⊆ ∆( X ) . Suppose Q is publicprivately learnable with m P ( α, β, ε ) public and n ( α, β, ε ) private samples. Then for all ε > 0 , Q is list-learnable to list size ℓ ( α, β ) = 10 9 exp( ε · n ( α 2 , β 10 , ε )) with m ( α, β ) = m P ( α 2 , β 10 , ε ) samples.

Proof. Let ε > 0 be arbitrary. Fix any α, β ∈ (0 , 1] . By assumption, Q admits a ( α 2 , β 10 , ε ) -public-private learner A , which uses m := m P ( α 2 , β 10 , ε ) public and n := n ( α 2 , β 10 , ε ) private samples. We use A to construct a ( α, β, 10 9 exp( εn )) -list learner that uses m samples. Consider any ˜ x = (˜ x 1 , . . . , ˜ x m ) ∈ X m and the class

Note that by definition, Q ˜ x has a ( α 2 , 1 10 ) -learner under ε -DP that takes n samples. Hence, by Fact C.1 it follows that any α -packing of Q ˜ x must have size ≤ 10 9 exp( εn ) =: ℓ . Let ̂ Q ˜ x be such a maximal α -packing, hence it is also an α -cover of Q ˜ x with | ̂ Q ˜ x | ≤ ℓ . We define our list learner's output, L ( ˜ x ) = ̂ Q ˜ x . It remains to show that for any p ∈ Q , with probability ≥ 1 -β over the sampling of ˜ X ∼ p m , dist( p, L ( ˜ X )) ≤ α . Suppose otherwise, that is, there exists p 0 ∈ Q , such that

Since L ( ˜ X ) is a α -cover of Q ˜ X , we have that with probability > β over the sampling of ˜ X ∼ p m 0 , p 0 ̸∈ Q ˜ X . This contradicts the success guarantee of A :

The second inequality follows by the definition of Q ˜ X : conditioned on the event p 0 ̸∈ Q ˜ X , the probability, over the private samples X ∼ p n 0 and the randomness of the algorithm A , that the output Q of our algorithm satisfies TV( Q,p 0 ) ≤ α 2 is < 9 10 .

3.3 List learning implies compression

We state the final component of Theorem 1.1/D.1: the existence of a list learner for a class of distributions Q implies the existence of a sample compression scheme for Q . This follows from the definitions: given samples ˜ x , the encoder of the sample compression scheme runs a list learner L on ˜ x . It passes along x , and, with knowledge of the target distribution q , the index i of the distribution in L ( ˜ x ) that is close to q . The decoder receives this information and outputs L ( ˜ x ) i . The proof can be found in Appendix D.2.

Proposition 3.5 (List learning = ⇒ compression) . Let Q ⊆ ∆( X ) . Suppose Q is list learnable to list size ℓ ( α, β ) with m L ( α, β ) samples. Then Q admits ( τ ( α, β ) , t ( α, β ) , m ( α, β )) = ( m L ( α, β ) , log 2 ( ℓ ( α, β )) , m L ( α, β )) realizable sample compression.

4 Applications

Here, we state a few applications of the connections obtained via Theorem 1.1/D.1. First, we recover and extend results on the public-private learnability of high-dimensional Gaussians and mixtures of Gaussians, using known results on sample compression schemes. Second, we describe the closure properties of public-private learnability: if a class Q is public-privately learnable, the class of mixtures of Q and the class of products of Q are also public-privately learnable.

4.1 Public-private learnability of Gaussians and mixtures of Gaussians

There are known realizable sample compression schemes for the class of Gaussians in R d , as well as for the class of all k -mixtures of Gaussians in R d [ABDH + 20]. Hence, these classes are public-privately learnable.

Fact 4.1 (Robust compression scheme for Gaussians [ABDH + 20, Lemma 5.3]) . The class of Gaussians over R d admits ( O ( d ) , O ( d 2 log( d α )) , O ( d log( 1 β )) 2 3 -robust sample compression.

Fact 4.2 (Realizable compression scheme for mixtures of Gaussians [ABDH + 20, Lemma 4.8 applied to Lemma 5.3]) . The class of k -mixtures of Gaussians over R d admits ( O ( kd ) , O ( kd 2 log( d α ) + log( k α )) , O ( kd log( k/β ) log(1 /β ) α )) realizable sample compression.

Gaussians. From Theorem 1.1/D.1 and Fact 4.1, we get a public-private learner for Gaussians over R d using O ( d log( 1 β )) public and ˜ O ( d 2 +log(1 /β ) α 2 + d 2 +log(1 /β ) αε ) private samples (Corollary E.1). This recovers the result of [BKS22] on Gaussians up to a factor of O (log( 1 β )) in public sample complexity, and improves the private sample complexity by a polylog(1 /β ) factor.

Mixtures of Gaussians. From Theorem 1.1/D.1 and Fact 4.2, we get a public-private learner for the class of k -mixtures of Gaussians in R d using ˜ O ( kd log 2 (1 /β ) α ) public and ˜ O ( kd 2 +log(1 /β ) α 2 + kd 2 +log(1 /β ) αε ) private samples (Theorem 1.2/Corollary E.2).

In terms of k, d , and α , the public sample complexity of this learner is less than the ˜ Θ( kd 2 α 2 ) necessary and sufficient sample complexity for the problem non-privately [ABDH + 20]. This also implies that in the regime where ε > α , having more public samples (but still ˜ o ( kd 2 α 2 ) ) cannot improve private sample complexity. Under pure DP, no finite sample size suffices; under approximate DP, ˜ O ( k 2 d 4 log(1 /δ ) α 2 ε ) has been shown to suffice [AAL23b].

Algorithms for parameter estimation , like the mixture of Gaussians estimators from [BKS22] cannot be directly compared as they target a strictly stronger success criteria under stronger assumptions on the underlying distribution. [BKS22] gives an estimator for separated Gaussian mixtures that uses ˜ O ( d log( k ) w ∗ ) public and ˜ O ( d 2 w ∗ α 2 + d 2 w ∗ α 2 ) private samples satisfying ε 2 2 -zCDP, where w ∗ is the minimum component mixing weight. In terms of the privacy guarantee and private sample complexity, our result is a strict improvement for density estimation, since k = O ( 1 w ∗ ) . In the setting where w ∗ = Ω( 1 k ) , [BKS22]'s algorithm uses 1 α -factor fewer public samples. Furthermore, their algorithm runs in time polynomial in the sample complexity.

4.2 Public-private learnability of mixture and product distributions

If a class is realizably compressible, the class of its k -mixtures and the class of its k -products are also realizably compressible. Being realizably compressible with O ( m ( α, β )) samples is equivalent to being public-privately learnable with O ( m ( α, β )) public samples. Hence, we have black-box reductions of public-private learnability of mixture/product classes to public-private learnability of their base classes.

Theorem 4.3 (Public-private learning for mixture and product distributions (Informal; see Theorems E.5 and E.7)) . Let k ≥ 1 . If Q ⊆ ∆( X ) is public-privately learnable with m ( α, β, ε ) public samples, then for any ε 0 > 0 :

The class of k -mixtures of Q is public-privately learnable with O ( k log( k/β ) α · m ( α, β, ε 0 )) public samples.

The class of k -products of Q (over X k ) is public-privately learnable with O (log( k β ) · m ( α/k, β, ε 0 )) public samples.

The full statements and proof can be found in Appendix E.2. Here, we use the non-constructive reduction of list learning to public-private learning, and hence this does not yield a finite time algorithm. Note also that the target privacy ε does not appear in the public sample complexity (it only affects the amount of private samples required).

5 Agnostic and distribution-shifted public-private learning

The setting we have examined thus far makes the following assumptions on the data generation process: (1) same distribution - public and private data are sampled from the same underlying distribution; and (2) realizability - public and private data are sampled from members of the class Q .

[BKS22] shows that for Gaussians over R d , the first condition can be relaxed: they give an algorithm for the case where the public and the private data are generated from different Gaussians with bounded TV distance. However, they do not remove the second assumption.

We show that for general classes of distributions that robust compression schemes yield public-private learners, which: (1) can handle public-private distribution shifts (i.e., the setting where the public data and the private data distributions can be different); and (2) are agnostic, i.e., they do not require samples to come from a member of Q , and instead, promise error close to the best approximation of the private data distribution by a member of Q . Since Gaussians admit a robust compression scheme (Fact 4.1), we obtain public-private Gaussian learners that work under relaxed forms of these assumptions on the data generating process (Theorem 1.3/F.1).

We first formally define the notion of agnostic and distribution-shifted public-private learning , and then prove the main result of this section.

Definition 5.1 (Agnostic and distribution-shifted public-private learner) . Let Q ⊆ ∆( X ) . For α, β ∈ (0 , 1] , ε > 0 , γ ∈ [0 , 1] , and c ≥ 1 a γ -shifted c -agnostic ( α, β, ε ) -public-private learner for Q is a public-private ε -DP algorithm A : X m ×X n → ∆(∆( X )) , such that for any ˜ p, p ∈ ∆( X ) with TV(˜ p, p ) ≤ γ , if we draw datasets ˜ X = ( ˜ X 1 , . . . , ˜ X m ) i.i.d. from ˜ p and X = ( X 1 , . . . , X n ) i.i.d. from p , and then Q ∼ A ( ˜ X , X ) ,

Theorem 5.2 (Robust compression = ⇒ agnostic and distribution-shifted public-private learning) . Let Q ⊆ ∆( X ) and r > 0 . If Q admits ( τ ( α, β ) , t ( α, β ) , m C ( α, β )) r -robust compression, then for every α, β ∈ (0 , 1] and ε > 0 , there exists a r 2 -shifted 2 r -agnostic ( α, β, ε ) -public-private learner for Q that uses m ( α, β, ε ) = m C ( α 12 , β 2 ) public samples and n ( α, β, ε ) = O (( 1 α 2 + 1 αε ) · ( t ( α 12 , β 2 ) + τ ( α 12 , β 2 ) log( m C ( α 12 , β 2 )) + log( 1 β ))) private samples.

Proof. The proof again mirrors the proof of Theorem 4.5 in [ABDH + 20]. The key observation (and difference from the proof in Appendix D.1) is the following: for the unknown distribution p ∈ ∆( X ) , consider dist( p, Q ) . If dist( p, Q ) ≥ r 2 , the output Q of any algorithm satisfies TV( p, Q ) ≤ 1 ≤ 2 r · dist( p, Q ) . Hence, we can assume dist( p, Q ) < r 2 , and let q ∗ ∈ Q with TV( p, q ∗ ) < min { r 2 , dist( p, Q ) + α 12 } as guaranteed by such.

By triangle inequality, TV(˜ p, q ∗ ) < r . This implies that when we generate hypotheses ̂ Q to choose from using the r -robust sample compression with samples from ˜ p , with high probability there will be some q ∈ ̂ Q with TV( q, q ∗ ) ≤ α 12 . We have

Applying the 3-agnostic ε -DP learner for finite classes from [AAAK21] (Fact C.2) with the above setting of n gives us the result.

6 Lower bounds

We give a lower bound on the number of public samples required to public-privately learn Gaussians in R d . We know that Gaussians in R d are public-privately learnable with d + 1 public samples from [BKS22]. We show that this is within 1 of the optimal: the class of Gaussians in R d is not public-privately learnable with d -1 public samples. The following is the formal statement of Theorem 1.4.

Theorem 6.1. The class Q of all Gaussians in R d is not public-private learnable with m P ( α, β, ε ) = d -1 public samples, regardless of the number of private samples. That is, there exists α d , β d > 0 such that for any n ∈ N , Q does not admit a ( α d , β d , 1) -public-private learner using d -1 public and n private samples.

Our result leverages the connection between public-private learning and list learning. The existence of such a public-private learner described above would imply the existence of a list learner for d -dimensional Gaussians taking d -1 samples as input. We show, using a 'no-free-lunch'-style argument (e.g. Theorem 5.1 from [SSBD14]) that such a list learner cannot exist. The proof of Theorem 6.1, given in Appendix G, goes through the following steps.

We reduce list learning to public-private learning, via Proposition 3.4;

We establish a technical lemma that relates the PAC guarantee of a list learner with its average performance over a set of problem instances, via a 'no-free-lunch'-style argument (Lemma G.1);

For every d ≥ 2 , we find a sequence of hard subclasses of Gaussians over R d , which satisfy the conditions of Lemma G.1. This forms the set of hard problem instances that imply a lower bound on the error of any list learner for the class (does not receive enough samples);

Since list learning to arbitrary error with few samples is impossible, public-private learning to arbitrary error with few public samples must also be impossible.

7 Learning when the Yatracos class has finite VC dimension

In this section, we describe a public-private learner for classes of distributions whose Yatracos class has finite VC dimension. We start by defining the Yatracos class of a family of distributions.

Definition 7.1 (Yatracos class) . For Q ⊆ ∆( X ) , the Yatracos class of Q is given by

̸

Theorem 7.2. Let Q ⊆ ∆( X ) . Let H be the Yatracos class of Q , denote by VC( H ) and VC ∗ ( H ) the VC and dual VC dimension of H . Q is public-privately learnable with m public and n private samples, where

Theorem 7.2/1.5 says that classes of distributions whose Yatracos class have finite VC dimension can be public-privately learned. Note that the number of public samples used is indeed fewer than the O ( VC( H ) α 2 ) sample requirement in the non-private analogue of the result (Fact H.1).

The proof is given in Appendix H. The result is a consequence of a known public-private uniform convergence result [BCM + 20, Theorem 10]. To adapt it to our setting, we (1) modify their result for pure DP (rather than approximate DP); and (2) conclude that uniform convergence over the Yatracos sets of Q suffices to implement the learner from Fact H.1.

8 Conclusion

In this work, we connect public-private distribution learning to the notions of sample compression and list learning. In doing so, for broad classes of distributions, we introduce approaches to: (1) design sample-efficient public-private learners; (2) prove lower bounds on how much public data is required for public-private learnability. In the following, we list several questions for future study.

Question 8.1. For a class Q , our work examines the minimal amount of public data needed to render Q pure privately learnable. How much do we need to render Q approximate DP learnable?

Question 8.2. The VC bound of Theorem 1.5/7.2, although more general, is 'qualitatively' loose. For Gaussians, it yields a O ( d 2 α ) public sample complexity, though we know O ( d ) is possible, notably with no dependence on α . On the other hand, Ω(1 /α ) public samples are required for public-privately learning mixtures of Gaussians. What qualities of a distribution admit public-private learning with α -independent public sample complexity? Stronger: can we find more illuminating characterizations of the sample complexity of list learning?

Question 8.3. Our work studies sample complexity improvements from using public data. Can public data lead to algorithmic improvements, that is, runtime efficiency or simpler algorithms?

Acknowledgments and Disclosure of Funding

We thank the anonymous reviewers for their helpful feedback.

AB, GK, and VS acknowledge funding from an NSERC discovery grant. AB is supported by a David R. Cheriton Graduate Scholarship. CC is supported by an ARC DECRA and an unrestricted gift from Google. GK is supported by a Canada CIFAR AI Chair, an unrestricted gift from Google, a University of Waterloo startup grant, and an unrestricted gift from Apple.

References

[AAAK21] Ishaq Aden-Ali, Hassan Ashtiani, and Gautam Kamath. On the sample complexity of privately learning unbounded high-dimensional Gaussians. In Proceedings of the 32nd International Conference on Algorithmic Learning Theory (ALT'21) , 2021.

[AABD + 20] Sushant Agarwal, Nivasini Ananthakrishnan, Shai Ben-David, Tosca Lechner, and Ruth Urner. On learnability wih computable learners. In Proceedings of the 31st International Conference on Algorithmic Learning Theory (ALT'20) , 2020.

[AAL21] Ishaq Aden-Ali, Hassan Ashtiani, and Christopher Liaw. Privately learning mixtures of axis-aligned Gaussians. In Advances in Neural Information Processing Systems 34 (NeurIPS'21) , 2021.

[AAL23a] Mohammad Afzali, Hassan Ashtiani, and Christopher Liaw. Mixtures of gaussians are privately learnable with a polynomial number of samples. CoRR , abs/2309.03847, 2023.

[AAL23b] Jamil Arbas, Hassan Ashtiani, and Christopher Liaw. Polynomial time and private learning of unbounded Gaussian mixture models. In Proceedings of the 40th International Conference on Machine Learning (ICML'23) , 2023.

[ABDH + 20] Hassan Ashtiani, Shai Ben-David, Nicholas J. A. Harvey, Christopher Liaw, Abbas Mehrabian, and Yaniv Plan. Near-optimal sample complexity bounds for robust learning of Gaussian mixtures via compression schemes. J. ACM , 67(6), 2020.

[ABM19] Noga Alon, Raef Bassily, and Shay Moran. Limits of private learning with access to public data. In Advances in Neural Information Processing Systems 32 (NeurIPS'19) , 2019.

[Abo18] John M. Abowd. The U.S. Census Bureau adopts differential privacy. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , 2018.

[ACG + 16] Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In CCS'16: 2016 ACM SIGSAC Conference on Computer and Communications Security , 2016.

[ADK20] Brendan Avent, Yatharth Dubey, and Aleksandra Korolova. The power of the hybrid model for mean estimation. Proc. Priv. Enhancing Technol. , 2020(4):48-68, 2020.

[AGM + 22] Ehsan Amid, Arun Ganesh, Rajiv Mathews, Swaroop Ramaswamy, Shuang Song, Thomas Steinke, Vinith M. Suriyakumar, Om Thakkar, and Abhradeep Thakurta. Public data-assisted mirror descent for private model training. In Proceedings of the 39th International Conference on Machine Learning (ICML'22) , 2022.

[AKZ + 17] Brendan Avent, Aleksandra Korolova, David Zeber, Torgeir Hovden, and Benjamin Livshits. BLENDER: Enabling local search with a hybrid differential privacy model. In 26th USENIX Security Symposium (USENIX Security'17) , 2017.

[AL22] Hassan Ashtiani and Christopher Liaw. Private and polynomial time algorithms for learning Gaussians and beyond. In Proceedings of the 35th Annual Conference on Learning Theory (COLT'22) , 2022.

[AM18] Hassan Ashtiani and Abbas Mehrabian. Some techniques in density estimation. arXiv preprint arXiv:1801.04003 , 2018.

[App17] Differential Privacy Team, Apple. Learning with privacy at scale, 2017.

[Ass83] Patrick Assouad. Densit´ e et dimension. Annales de l'Institut Fourier , 33(3):233-282, 1983.

[ASZ21] Jayadev Acharya, Ziteng Sun, and Huanyu Zhang. Differentially private assouad, fano, and le cam. In Proceedings of the 32nd International Conference on Algorithmic Learning Theory , ALT '21, pages 48-78. JMLR, Inc., 2021.

[BBV08] Maria-Florina Balcan, Avrim Blum, and Santosh Vempala. A discriminative framework for clustering via similarity functions. In Proceedings of the Fortieth Annual ACM Symposium on Theory of Computing , STOC '08, page 671-680, New York, NY, USA, 2008. Association for Computing Machinery.

[BCM + 20] Raef Bassily, Albert Cheu, Shay Moran, Aleksandar Nikolov, Jonathan R. Ullman, and Steven Wu. Private query release assisted by public data. In Proceedings of the 37th International Conference on Machine Learning (ICML'20) , 2020.

[BKS22] Alex Bie, Gautam Kamath, and Vikrant Singhal. Private estimation with public data. In Advances in Neural Information Processing Systems 35 (NeurIPS'22) , 2022.

[BKSW19] Mark Bun, Gautam Kamath, Thomas Steinke, and Zhiwei Steven Wu. Private hypothesis selection. In Advances in Neural Information Processing Systems 32 , NeurIPS '19, pages 156-167. Curran Associates, Inc., 2019.

[BLR13] Avrim Blum, Katrina Ligett, and Aaron Roth. A learning theory approach to noninteractive database privacy. Journal of the ACM , 60(2):1-25, 2013.

[BMN20] Raef Bassily, Shay Moran, and Anupama Nandi. Learning from mixtures of private and public populations. In Advances in Neural Information Processing Systems 33 , NeurIPS '20, pages 2947-2957. Curran Associates, Inc., 2020.

[BNS16] Amos Beimel, Kobbi Nissim, and Uri Stemmer. Private learning and sanitization: Pure vs. approximate differential privacy. Theory of Computing , 12(1):1-61, 2016.

[BTGT18] Raef Bassily, Om Thakkar, and Abhradeep Guha Thakurta. Model-agnostic private learning. In Advances in Neural Information Processing Systems 31 , NeurIPS '18, pages 7102-7112. Curran Associates, Inc., 2018.

[CCd + 23] Hongjie Chen, Vincent Cohen-Addad, Tommaso d'Orsi, Alessandro Epasto, Jacob Imola, David Steurer, and Stefan Tiegel. Private estimation algorithms for stochastic block models and mixture models. arXiv preprint arXiv:2301.04822 , 2023.

[CDE + 23] Rachel Cummings, Damien Desfontaines, David Evans, Roxana Geambasu, Matthew Jagielski, Yangsibo Huang, Peter Kairouz, Gautam Kamath, Sewoong Oh, Olga Ohrimenko, Nicolas Papernot, Ryan Rogers, Milan Shen, Shuang Song, Weijie Su, Andreas Terzis, Abhradeep Thakurta, Sergei Vassilvitskii, Yu-Xiang Wang, Li Xiong, Sergey Yekhanin, Da Yu, Huanyu Zhang, and Wanrong Zhang. Challenges towards the next frontier in privacy. arXiv preprint arXiv:2304.06929 , 2023.

[CKM + 21] Edith Cohen, Haim Kaplan, Yishay Mansour, Uri Stemmer, and Eliad Tsfadia. Differentially-private clustering of easy instances. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 2049-2059. PMLR, 18-24 Jul 2021.

[CSV17] Moses Charikar, Jacob Steinhardt, and Gregory Valiant. Learning from untrusted data. In Proceedings of the 49th Annual ACM Symposium on the Theory of Computing , STOC '17, pages 47-60, New York, NY, USA, 2017. ACM.

[DKS18] Ilias Diakonikolas, Daniel M. Kane, and Alistair Stewart. List-decodable robust mean estimation and learning mixtures of spherical Gaussians. In Proceedings of the 50th Annual ACM Symposium on the Theory of Computing , STOC '18, pages 1047-1060, New York, NY, USA, 2018. ACM.

[DL01] Luc Devroye and G´ abor Lugosi. Combinatorial Methods in Density Estimation . Springer Series in Statistics. Springer New York, 2001.

[DMNS06] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Proceedings of the 3rd Conference on Theory of Cryptography (TCC'06) , 2006.

[DMR18] Luc Devroye, Abbas Mehrabian, and Tommy Reddad. The total variation distance between high-dimensional Gaussians. arXiv preprint arXiv:1810.08693 , 2018.

[DR14] Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science , 9(3-4):211-407, 2014.

[GHN + 23] Arun Ganesh, Mahdi Haghifam, Milad Nasr, Sewoong Oh, Thomas Steinke, Om Thakkar, Abhradeep Guha Thakurta, and Lun Wang. Why is public pretraining necessary for private model training? In Proceedings of the 40th International Conference on Machine Learning (ICML'23) , 2023.

[GKW23] Xin Gu, Gautam Kamath, and Zhiwei Steven Wu. Choosing public datasets for private machine learning via gradient subspace distance. arXiv preprint arXiv:2303.01256 , 2023.

[Goo19a] Google. Google's differential privacy libraries. https://github.com/google/ differential-privacy , 2019. Accessed 07/31/2023.

[Goo19b] Google. TensorFlow privacy. https://github.com/tensorflow/privacy , 2019. Accessed 07/31/2023.

[HASP22] Frederik Harder, Milad Jalali Asadabadi, Danica J. Sutherland, and Mijung Park. Differentially private data generation needs better features. CoRR , abs/2205.12900, 2022.

[HBAL19] Naoise Holohan, Stefano Braghin, P´ ol Mac Aonghusa, and Killian Levacher. Diffprivlib: The IBM differential privacy library. CoRR , abs/1907.02444, 2019.

[JE13] Zhanglong Ji and Charles Elkan. Differential privacy based on importance weighting. Machine Learning , 93(1):163-183, 2013.

[KADV23] Mikhail Khodak, Kareem Amin, Travis Dick, and Sergei Vassilvitskii. Learningaugmented private algorithms for multiple quantile release. In Proceedings of the 40th International Conference on Machine Learning (ICML'23) , 2023.

[KLSU19] Gautam Kamath, Jerry Li, Vikrant Singhal, and Jonathan R. Ullman. Privately learning high-dimensional distributions. In Proceedings of the 32nd Annual Conference on Learning Theory (COLT'19) , 2019.

[KMS + 22] Gautam Kamath, Argyris Mouzakis, Vikrant Singhal, Thomas Steinke, and Jonathan R. Ullman. A private and computationally-efficient estimator for unbounded Gaussians. In Proceedings of the 35th Annual Conference on Learning Theory (COLT'22) , 2022.

[KMV22] Pravesh K. Kothari, Pasin Manurangsi, and Ameya Velingker. Private robust estimation by stabilizing convex relaxations. In Proceedings of the 35th Annual Conference on Learning Theory (COLT'22) , 2022.

[KRRT21] Peter Kairouz, M´ onica Ribero, Keith Rush, and Abhradeep Thakurta. (nearly) dimension independent private ERM with adagrad rates via publicly estimated subspaces. In Proceedings of the 34th Annual Conference on Learning Theory , COLT '21, pages 2717-2746, 2021.

[KS17] Pravesh K. Kothari and Jacob Steinhardt. Better agnostic clustering via relaxed tensor norms, 2017.

[KS22] Refael Kohen and Or Sheffet. Transfer learning in differential privacy's hybrid-model. In Proceedings of the 39th International Conference on Machine Learning (ICML'22) , 2022.

[KSSU19] Gautam Kamath, Or Sheffet, Vikrant Singhal, and Jonathan Ullman. Differentially private algorithms for learning mixtures of separated Gaussians. In Advances in Neural Information Processing Systems 32 , NeurIPS '19, pages 168-180. Curran Associates, Inc., 2019.

[KV18] Vishesh Karwa and Salil P. Vadhan. Finite sample differentially private confidence intervals. In 9th Innovations in Theoretical Computer Science Conference (ITCS'18) , 2018.

[Li10] Shengqiao Li. Concise formulas for the area and volume of a hyperspherical cap. Asian Journal of Mathematics & Statistics , 4(1):66-70, 2010.

[LKO22] Xiyang Liu, Weihao Kong, and Sewoong Oh. Differential privacy and robust statistics in high dimensions. In Proceedings of the 35th Annual Conference on Learning Theory , COLT '22, pages 1167-1246, 2022.

[LLHR23] Andrew Lowy, Zeman Li, Tianjian Huang, and Meisam Razaviyayn. Optimal differentially private learning with public data. CoRR , abs/2306.15056, 2023.

[LTLH22] Xuechen Li, Florian Tram` er, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners. In Proceedings of the 10th International Conference on Learning Representations , ICLR '22, 2022.

[LVS + 21] Terrance Liu, Giuseppe Vietri, Thomas Steinke, Jonathan Ullman, and Steven Wu. Leveraging public data for practical private query release. In Proceedings of the 38th International Conference on Machine Learning , ICML '21, pages 6968-6977. JMLR, Inc., 2021.

[LW86] Nick Littlestone and Manfred Warmuth. Relating data compression and learnability, 1986.

[LWAFF21] Zelun Luo, Daniel J Wu, Ehsan Adeli, and Li Fei-Fei. Scalable differential privacy with sparse network finetuning. In Proceedings of the 2021 IEEE Computer Society Conference on Computer Vision and Pattern Recognition , CVPR '21, pages 5059-5068, Washington, DC, USA, 2021. IEEE Computer Society.

[MY16] Shay Moran and Amir Yehudayoff. Sample compression schemes for VC classes. Journal of the ACM , 63(3):1-10, 2016.

[NB20] Anupama Nandi and Raef Bassily. Privately answering classification queries in the agnostic PAC model. In Proceedings of the 31st International Conference on Algorithmic Learning Theory , ALT '20, pages 687-703. JMLR, Inc., 2020.

[NRS07] Kobbi Nissim, Sofya Raskhodnikova, and Adam Smith. Smooth sensitivity and sampling in private data analysis. In Proceedings of the 39th Annual ACM Symposium on the Theory of Computing , STOC '07, pages 75-84, New York, NY, USA, 2007. ACM.

[Ope20] OpenDP Team. The OpenDP white paper, 2020.

[PAE + 17] Nicolas Papernot, Mart´ ın Abadi, Ulfar Erlingsson, Ian Goodfellow, and Kunal Talwar. Semi-supervised knowledge transfer for deep learning from private training data. In Proceedings of the 5th International Conference on Learning Representations , ICLR '17, 2017.

[PCS + 19] Nicolas Papernot, Steve Chien, Shuang Song, Abhradeep Thakurta, and Ulfar Erlingsson. Making the shoe fit: Architectures, initializations, and tuning for learning with privacy. https://openreview.net/forum?id=rJg851rYwH , 2019.

[PSM + 18] Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and ´ Ulfar Erlingsson. Scalable private learning with PATE. In Proceedings of the 6th International Conference on Learning Representations , ICLR '18, 2018.

[RY20] Prasad Raghavendra and Morris Yau. List decodable learning via sum of squares. In Shuchi Chawla, editor, Proceedings of the 2020 ACM-SIAM Symposium on Discrete Algorithms, SODA 2020, Salt Lake City, UT, USA, January 5-8, 2020 , pages 161-180. SIAM, 2020.

[SSBD14] Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning: From Theory to Algorithms . Cambridge University Press, 2014.

[TB21] Florian Tram` er and Dan Boneh. Differentially private learning needs better features (or much more data). In Proceedings of the 9th International Conference on Learning Representations , ICLR '21, 2021.

[TCK + 22] Eliad Tsfadia, Edith Cohen, Haim Kaplan, Yishay Mansour, and Uri Stemmer. FriendlyCore: Practical differentially private aggregation. In Proceedings of the 39th International Conference on Machine Learning (ICML'22) , 2022.

[TKC22] Florian Tram` er, Gautam Kamath, and Nicholas Carlini. Considerations for differentially private learning with large-scale public pretraining. arXiv preprint arXiv:2212.06470 , 2022.

[VW02] Santosh Vempala and Grant Wang. A spectral algorithm for learning mixtures of distributions. In Proceedings of the 43rd Annual IEEE Symposium on Foundations of Computer Science , FOCS '02, pages 113-123, Washington, DC, USA, 2002. IEEE Computer Society.

[XZA + 23] Zheng Xu, Yanxiang Zhang, Galen Andrew, Christopher A. Choquette-Choo, Peter Kairouz, H. Brendan McMahan, Jesse Rosenstock, and Yuanbo Zhang. Federated learning of Gboard language models with differential privacy. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL'23) , 2023.

[Yat85] Yannis G. Yatracos. Rates of convergence of minimum distance estimators and Kolmogorov's entropy. The Annals of Statistics , 13(2):768-774, 1985.

[YNB + 22] Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang. Differentially private fine-tuning of language models. In Proceedings of the 10th International Conference on Learning Representations , ICLR '22, 2022.

[YZCL21] Da Yu, Huishuai Zhang, Wei Chen, and Tie-Yan Liu. Do not let privacy overbill utility: Gradient embedding perturbation for private learning. In Proceedings of the 9th International Conference on Learning Representations , ICLR '21, 2021.

[ZWB21] Yingxue Zhou, Zhiwei Steven Wu, and Arindam Banerjee. Bypassing the ambient dimension: Private SGD with gradient subspace identification. In Proceedings of the 9th International Conference on Learning Representations , ICLR '21, 2021.

A Extended related work

Privately learning Gaussians. Our work studies the task of learning arbitrary, unbounded Gaussians while offering differential privacy guarantees. Basic private algorithms for the task (variants of 'clip-and-noise') impose boundedness assumptions on the underlying parameters of the unknown Gaussian, since their sample complexities grow to infinity as the bounds widen to include more allowed distributions. Understanding these dependencies without public data has been a topic of significant study. [KV18] examined univariate Gaussians, showing that logarithmic dependencies on parameter bounds are necessary and sufficient in the case of pure DP, but can be removed under approximate DP. The same is true in the multivariate setting [AAAK21, KMS + 22, TCK + 22, AL22, KMV22, LKO22]. [BKS22] shows that instead of relaxing to approximate DP to handle arbitrary Gaussians, one can employ a small amount of public data; our lower bound tells us almost exactly how much is needed. Furthermore, our reductions between public-private learning and list learning offers the conclusion that the role of public data is precisely for bounding: distribution classes that can be privately learned with a small amount of public data are exactly the distributions that can be bounded with a small amount of public data.

Privately learning mixtures of Gaussians. Another line of related work is that on privately learning mixtures of Gaussian, but without any public data. [NRS07] provided a subsample-andaggregate approach to learn the parameters of mixtures of spherical Gaussians based on the work by [VW02] under the weaker, approximate DP. Recently, [CCd + 23] improved on this by weakening the separation condition required for the mixture components. [KSSU19] provided the first polynomialtime, approximate DP algorithms to learn the parameters of mixtures of non-spherical Gaussians under weak boundedness assumptions. [CKM + 21] improved on their work both in terms of the sample complexity and the separation assumption. [AAL23b] provided a polynomial-time reduction for privately and efficiently learning mixtures of unbounded Gaussians from the approximate DP setting to its non-private counterpart (albeit at a polynomial overhead in the sample complexity).

Our work falls into the category of private density estimation, for which [AAL21] gave new algorithms for the special case of spherical Gaussians under approximate DP, while [BKSW19] gave (computationally inefficient) algorithms for bounded Gaussians under pure DP. For comparison, the latter would have infinite sample complexity for unbounded Gaussians, but our work provides finite private sample complexity even under pure DP using public data. On the other hand, [ASZ21] showed hardness results for privately learning mixtures of Gaussians with known covariances. The concurrent work of Azfali, Ashtiani, and Liaw [AAL23a] gives the first learnability result for general high-dimensional mixtures of Gaussians under approximate differential privacy.

Theory for private algorithms with public data. Beyond distribution learning, there is a lot more work that investigates how public data can be employed in private data analysis. Some specific areas include private query release, synthetic data generation, and classification [JE13, BNS16, ABM19, NB20, BCM + 20, BMN20, LVS + 21, KADV23], and the results are a mix of theoretical versus empirical. The definition of public-private algorithms that we adopt is from [BNS16], which studied classification in the PAC model. The VC dimension bound we give for public-private distribution learning relies on results from public-private classification [ABM19] and uniform convergence [BCM + 20].

A concurrent and independent work [LLHR23] also studies learning with public and private data, focusing on the problems of mean estimation, empirical risk minimization, and stochastic convex optimization. The focus of the two works is somewhat different, both in terms of the type of problems considered (we study density estimation) and the type of results targeted. That is, our objective is to draw connections between different learning concepts and exploring the resulting implications for public-private distribution learning, while their goal seems to be understanding the precise error rates for some fundamental settings.

Private machine learning with public data. Within the context of private machine learning, there has been significant interest in how to best employ public data. The most popular method is pretraining [ACG + 16, PCS + 19, TB21, LWAFF21, YZCL21, LTLH22, YNB + 22, GHN + 23, HASP22] (though some caution about this practice [TKC22]), while other methods involve computing statistics about the private gradients [ZWB21, YZCL21, KRRT21, AGM + 22, GKW23], or training a student

model [PAE + 17, PSM + 18, BTGT18]. For more discussion of public data for private learning, see Section 3.1 of [CDE + 23].

List learning We also use the notion of list learning in this work, which is a 'non-robust' version of the well-known list-decodable learning [AAL21, RY20, BBV08, CSV17, DKS18, KS17], where the goal is still to output a list of distributions that contains one that is accurate with respect to the true distribution, but the sampling may happen from a corrupted version of the underlying distribution.

Hybrid differential privacy. Finally, a related setting is the hybrid model, in which samples require either local or central differential privacy [AKZ + 17]. Some learning tasks studied in this model include mean estimation [ADK20] and transfer learning [KS22].

B Technical overview

New connections for public-private learning. Our first contribution is establishing connections between sample compression, public-private learning, and list learning, via reductions. We show that: (1) sample compression schemes yield public-private learners; (2) public-private learners yield list learners; and (3) list learners yield sample compression schemes.

(1) and (3) are straightforward to prove: (1) compression implies public-private learning follows from a modification of an analogous result of [ABDH + 20], where we observe that in their proof, the learner's two-stage process of drawing a small compression sample used to generate a finite set of hypotheses using the compression scheme, followed by hypothesis selection with a larger sample, can be cleanly divided into using public and private samples respectively. In the latter stage, we employ a known pure DP hypothesis selection algorithm [BKSW19, AAAK21]). (3) List learning implies compression follows immediately from the definitions.

For (2) public-private learning implies list learning , we show non-constructively that there exists a list learner for a class, given a public-private learner for that class, but do not provide an algorithmic translation of the public-private learner to a list learner. For a set of samples S , we show that outputting a finite cover of the list of distributions on which the public-private learner succeeds on, when using S as the public samples, is a correct output for a list learner. Hence the list learner we construct, on input S , outputs this finite cover as determined by (but not explicitly constructed from) the public-private learner.

Agnostic and distribution-shifted public-private learning. We identify some distributional assumptions that can be relaxed in the public-private learning setup. We obtain agnostic and distributionshifted public-private learners via a connection to robust compression schemes. The reduction ideas are similar to those in the case of public-private learning and non-robust sample compression described above.

Lower bound on public-private learning of Gaussians. Our lower bound for public-privately learning high-dimensional Gaussians exploits our above connection between public-private learning and list learning, and applies a 'no-free-lunch'-style argument. The latter uses the fact that an algorithm's worst-case performance cannot be better than its performance when averaged across all the problem instances. In other words, we have two main steps in our proof: (1) we first claim that due to our above reduction, a lower bound for list learning high-dimensional Gaussians would imply a lower bound on the public sample complexity for public-privately learning high-dimensional Gaussians; and (2) assuming certain accuracy guarantees and a sample complexity for our list learner for high-dimensional Gaussians; we show that across a set of adversarially chosen problem instances, our average accuracy guarantee fails, which is a contradiction to our assumed worst-case guarantees.

We observe that the lower bound from Theorem 6.1 establishes that at least d public samples are necessary for public-private learning to vanishingly small error as d increases (impossibility of learning to a target error, via the application of Lemma G.1, is related to the bound on η from Equation (6), which decreases exponentially with d ). A natural question is whether the result can be strengthened to say that there is a single target error, simultaneously for all d , for which learning is impossible without d public samples.

VCdimension upper bound for public-private learning. Our proof involves invoking the existing results for public-private binary classification [ABM19] and uniform convergence [BCM + 20]. We use them to implement Yatracos' minimum distance estimator [Yat85, DL01] in a public-private way.

C Extended preliminaries

C.1 Notation

Covers and packings. For α > 0 and Q ⊆ ∆( X ) , we say that C ⊆ ∆( X ) is an α -cover of Q if for any q ∈ Q , there exists a p ∈ C with TV( p, q ) ≤ α . For α > 0 and Q ⊆ ∆( X ) , we say that P ⊆ Q is an α -packing of Q if for any p = q ∈ P , TV( p, q ) > α .

̸

Class of k -mixtures. Let Q ⊆ ∆( X ) be a class of distributions. For any k ≥ 1 , the class of k -mixtures of Q is given by

Class of k -products. Let Q ⊆ ∆( X ) be a class of distributions over X . For any k ≥ 1 , q = ( q 1 , . . . , q k ) is a product distribution over X k , if q i ∈ Q for all i ∈ [ k ] and for X ∼ q , the i -th component X i of X is independently (of all the other coordinates) sampled from q i . The class of k -products of Q over X k is given by

C.2 Privacy

The following is a known hardness result on density estimation of distributions under pure differential privacy, and is based on the standard 'packing lower bounds'.

Fact C.1 (Packing lower bound [BKSW19, Lemma 5.1]) . Let Q ⊆ ∆( X ) , α ∈ (0 , 1] , and ε > 0 . Let ̂ Q be any α -packing of Q . Any ε -DP algorithm A : X n → ∆(∆( X )) that, upon receiving n i.i.d. samples X 1 , . . . , X n from any p ∈ Q , outputs Q with TV( Q,p ) ≤ α 2 with probability ≥ 9 10 requires

The next result guarantees the existence of agnostic learners for finite hypothesis classes under pure differential privacy.

Fact C.2 (Pure DP 3 -agnostic learner for finite Q [BKSW19], [AAAK21, Theorem 2.24]) . Let Q ⊆ ∆( X ) with |Q| < ∞ . For every α, β ∈ (0 , 1] and ε > 0 , there exists an ε -DP algorithm A : X n → ∆(∆( X )) , such that for any p ∈ ∆( X ) , if we draw a dataset X = ( X 1 , ..., X n ) i.i.d. from p and then Q ∼ A ( X ) ,

where

D Statements and proofs for Section 3 - The connection to sample compression schemes

The following is the full formal statement of Theorem 1.1.

Theorem D.1 (Sample complexity equivalence between sample compression, public-private learning, and list learning) . Let Q ⊆ ∆( X ) . Let m : (0 , 1] 2 → N be a sample complexity function, such that m ( α, β ) = poly( 1 α , 1 β ) . 3 Then the following are equivalent.

Q is realizably compressible with m C ( α, β ) = O ( m ( α, β )) samples.

Q is public-privately learnable with m P ( α, β, ϵ ) = O ( m ( α, β )) public samples.

Q is list learnable with m L ( α, β ) = O ( m ( α, β )) samples.

The functions m C , m P , and m L are related to one another as: m P ( α, β, ε ) = m C ( α 6 , β 2 ) ; m L ( α, β ) = m P ( α 2 , β 10 , ε ) for any ε > 0 ; and m C ( α, β ) = m L ( α, β ) . Hence, if there exists a polynomial m : (0 , 1] 2 → N , such that m C ( α, β ) = O ( m ( α, β )) , then m C ( α, β ) , m P ( α, β ) , and m L ( α, β ) are all within constant factors of each other.

D.1 Compression implies public-private learning

Proof of Proposition 3.1. The proof this proposition closely mirrors that of Theorem 4.5 from [ABDH + 20]. We adapt their result to the public-private setting.

Fix α, β ∈ (0 , 1] and ε > 0 . Let τ = τ ( α 6 , β 2 ) , t = t ( α 6 , β 2 ) , and m = m C ( α 6 , β 2 ) . We draw a public dataset ˜ X of size m i.i.d. from p . Consider

Note that the encoding f p ( ˜ X ) ∈ S , so forming ̂ Q = { g ( S ′ , b ) : ( S ′ , b ) ∈ S} means that with probability ≥ 1 -β 2 over the sampling of ˜ X , q = g ( f p ( ˜ X )) ∈ ̂ Q has TV( q, p ) ≤ α 6 .

Now, we run the ε -DP 3 -agnostic learner from Fact C.2 on ̂ Q , targeting error α 2 and failure probability β 2 , which is achieved as long as we have n private samples (given in the statement of Proposition 3.1), which is logarithmic in |S| . With probability ≥ 1 -β , we approximately recover p with the compression scheme and the DP learner succeeds, and so the output Q satisfies

D.2 List learning implies sample compression

Proof of Proposition 3.5. Fix any α, β ∈ (0 , 1] . Let m = m L ( α, β ) and ℓ = ℓ ( α, β ) . By assumption, Q admits an ( α, β, ℓ ) -list learner L : X m →{ L ⊆ ∆( X ) : | L | ≤ ℓ } that takes m samples. Letting τ = m and t = log 2 ( ℓ ) , we define the compression scheme as follows.

Encoder: for any q ∈ Q , the encoder f q : X m → X τ ×{ 0 , 1 } t produces the following, given an input ˜ x ∈ X m . It first runs the list learner on ˜ x , obtaining L ( ˜ x ) . Then, it finds the smallest index i with TV( q, L ( ˜ x ) i ) = dist( q, L ( ˜ x )) , where L ( ˜ x ) i denotes the i -th element of the the list L ( ˜ x ) . The output of the list learner is ( ˜ x , i ) . Note that ˜ x ∈ X τ and that i can be represented with log 2 ( ℓ ) = t bits.

Decoder: the fixed decoder g : X τ ×{ 0 , 1 } t → ∆( X ) takes ˜ x and i , runs the list learner L on ˜ x , and produces L ( ˜ x ) i .

By the guarantee of the list learner, we indeed have for any q ∈ Q , with probability ≥ 1 -β over the sampling of ˜ X ∼ q m , TV( q, g ( f q ( S ))) ≤ α .

E Statements and proofs for Section 4 - Applications

E.1 Public-private learnability of Gaussians and mixtures of Gaussians

Corollary E.1 (Public-private learning for Gaussians) . Let d ≥ 1 . The class of Gaussians over R d is public-privately learnable with m ( α, β, ε ) public samples and n ( α, β, ε ) private samples, where

Corollary E.2 (Public-private learning for mixtures of Gaussians) . Let d, k ≥ 1 . The class of all k -mixtures of Gaussians over R d is public-privately learnable with m ( α, β, ε ) public samples and n ( α, β, ε ) private samples, where

E.2 Public-private learnability of mixture and product distributions

Mixture distributions. We first mention a fact from [ABDH + 20], which says that if a compression scheme exists for a class of distributions Q , then there exists a compression scheme for the class of k -mixtures of Q .

Fact E.3 (Compression for mixture distributions [ABDH + 20, Lemma 4.8]) . If a class of distributions Q admits ( τ ( α, β ) , t ( α, β ) , m ( α, β )) realizable sample compression, then for any k ≥ 1 , the class of k -mixtures of Q admits ( τ k ( α, β ) , t k ( α, β ) , m k ( α, β )) realizable sample compression, where τ k , t k , m k : (0 , 1] 2 → N are as follows:

Next, we state a corollary of Propositions 3.4 and 3.5, which describes the existence of a compression scheme, given the existence of a public-private learner.

Corollary E.4 (Public-private learning = ⇒ compression) . Let Q ⊆ ∆( X ) be a class of distributions. Suppose Q is public-privately learnable with m P ( α, β, ε ) public samples and n ( α, β, ε ) private samples. Then for any ε > 0 , Q admits

realizable sample compression.

Proof. Fix ε > 0 . From Proposition 3.4, if Q is public-privately learnable, then it is list learnable to list size ℓ ( α, β ) with m L ( α, β ) samples, where

Proposition 3.5 implies Q admits ( τ ( α, β ) , t ( α, β ) , m C ( α, β )) sample compression, where

This completes the proof.

As a consequence of Corollary E.4, Fact E.3, and Proposition 3.1, we have the following result about the public-private learnability of mixture distributions.

Theorem E.5 (Public-private learning for mixture distributions) . Suppose Q ⊆ ∆( X ) is publicprivately learnable with m ( α, β, ε ) public samples and n ( α, β, ε ) private samples. Then for any k ≥ 1 , Q ⊕ k , the class of k -mixtures of Q , is public-privately learnable with m k ( α, β, ε ) public samples and n k ( α, β, ε ) private samples, where

for any choice of ε 0 > 0 .

Wegive an example of an application of this result. Consider the class of Gaussians over R d , for which there exists a public-private learner that uses m = O ( d ) public samples and n = O ( d 2 α 2 + d 2 εα ) · polylog ( d, 1 α , 1 β ) private samples [BKS22]. Then Theorem E.5 implies that there exists a publicprivate learner for the class of k -mixtures of Gaussians that uses m k = O ( kd log( k/β ) α ) public samples and

private samples for any ε 0 > 0 .

With the choice of ε 0 = α , we get a private sample complexity of n k = O ( kd 2 α 3 + kd 2 α 2 ε ) · polylog ( d, k, 1 α , 1 β ) . Notably, this private sample complexity, obtained by specializing the general result of Theorem E.5, suffers some loss compared to our learner for mixtures of Gaussians from Corollary E.2.

Product distributions. We start by mentioning a fact from [ABDH + 20], which says that if a compression scheme exists for a class of distributions Q , then there exists a compression scheme for the class of k -products of Q .

Fact E.6 (Compression for product distributions [ABDH + 20, Lemma 4.6]) . If a class of distributions Q admits ( τ ( α, β ) , t ( α, β ) , m ( α, β )) r -robust sample compression, then for any k ≥ 1 , the class of k -products of Q admits ( τ k ( α, β ) , t k ( α, β ) , m k ( α, β )) r -robust sample compression, where τ k , t k , m k : (0 , 1] → N are as follows:

As a consequence of Corollary E.4, Fact E.6, and Proposition 3.1, we have the following result about the public-private learnability of product distributions.

Theorem E.7 (Public-private learning for mixture distributions) . Suppose Q ⊆ ∆( X ) is publicprivately learnable with m ( α, β, ε ) public samples and n ( α, β, ε ) private samples. Then for any k ≥ 1 , Q ⊗ k , the class of k -products of Q over X k , is public-privately learnable with m k ( α, β, ε ) public samples and n k ( α, β, ε ) private samples, where

for any choice of ε 0 > 0 .

As an example, for the class of Gaussians over R , there exists a public-private learner that requires m = O (1) public samples and n = O ( 1 α 2 + 1 εα ) · polylog ( 1 α , 1 β ) private samples [BKS22]. Then Theorem E.7 implies that there exists a public-private learner for the class of k -products of Gaussians that requires m k = O (log( k/β )) public samples and n k = ( ( 1 α 2 + 1 εα ) · ( ε 0 k ( 1 α 2 + 1 ε 0 α ) + k )) · polylog ( k, 1 α , 1 β ) private samples, for any choice of ε 0 > 0 . Note that if were to apply Fact E.6 to Fact 4.1 after setting d = 1 in the latter, and then apply Proposition 3.1, we would obtain a better sample complexity in terms of the private data than what we would after combining Corollary E.1 (setting d = 1 ) and Theorem E.7 here. However, Theorem E.7 is a more versatile framework, so some loss is to be expected again.

F Statements and proofs for Section 5 - Agnostic and distribution-shifted public-private learning

Theorem 5.2 gives us an agnostic and a distribution-shifted learner for Gaussians over R d , as stated in the following corollary.

Corollary F.1 (Agnostic and distribution-shifted public-private learner for Gaussians) . Let d ≥ 1 . For any α, β ∈ (0 , 1] and ε > 0 , there exists 1 3 -shifted 3 -agnostic public-private learner for the class of Gaussians in R d that uses m public samples and n private samples, where

G Statements and proofs for Section 6 - Lower bounds

Lemma G.1. Let Q ⊆ ∆( X ) and m ∈ N . For a subclass C ⊆ Q , denote by U ( C ) the uniform distribution over C . Suppose there exists a sequence of distribution classes ( Q k ) ∞ k =1 , with each Q k ⊆ Q , and a set B ⊆ X m such that following holds:

There exists η ∈ (0 , 1] and k η ∈ N with

for all k ≥ k η .

and

There exist c > 0 and α ∈ (0 , 1] such that, defining ( u k ) ∞ k =1 , ( r k ) ∞ k =1 , and ( s k ) ∞ k =1 as

Then for any ℓ ∈ N , there does not exist any ( αη 4 , αη 4 , ℓ ) -list learner for Q that uses m samples.

Proof. We provide a proof by contradiction. Suppose for some ℓ ∈ N , we have an ( αη 4 , αη 4 , ℓ ) -list learner for Q using m samples, denoted by L : X m →{ L ⊆ ∆( X ) : | L | ≤ ℓ } . Then for all k ∈ N , we have that

Now, since lim k →∞ r k s k = 0 , there exists k 0 ≥ k η ∈ N , such that

Fix any x ∈ B , and let R = { q ∈ Q k 0 : dist( q, L ( x )) ≤ α } and S = { q ∈ Q k 0 : q m ( x ) ≥ cu k 0 } (uote that both R and S depend on x ).

For i ∈ [ ℓ ] , further let R i = { q ∈ Q k 0 : TV( q, L ( x ) i ) ≤ α } , so that R = ∪ ℓ i =1 R i .

̸

Now, fix i ∈ [ ℓ ] . Assuming that R i = ∅ , consider any p ∈ R i . For any q ∈ R i , we have TV( p, q ) ≤ 2 α . Hence, R i ⊆ { q ∈ Q k 0 : TV( p, q ) ≤ 2 α } . Regardless of whether R i is empty,

Moreover, we can conclude that

Observe that this implies, since u k 0 ≥ q m ( x ) ,

we have that

an inequality we will use momentarily. We can now write

Plugging (4) in, along with the definition of s k 0 , we have,

Integrating over all x ∈ B and using Inequality 3,

If P { dist( Q, L ( X )) ≤ α | X ∈ B } ≥ 1 10 , then E [dist( Q, L ( X ))] ≥ αη , contradicting (1). Otherwise,

also contradicting Equation 1.

Proof of Theorem 6.1. To prove Theorem 6.1, it suffices to find, for every d ≥ 2 , a sequence of subclasses ( Q k ) ∞ k =1 and a set B ∈ ( R d ) d -1 that indeed satisfy the conditions of Lemma G.1. In what follows, we fix an arbitrary d ≥ 2 .

The construction of the sequence of hard subclasses. Let e d = [0 , 0 , . . . , 1] ⊤ ∈ R d . We define the following sets:

That is, T is a 1 2 -disk (a disk with radius 1 2 ) in R d -1 embedded onto the ( d -1) -dimensional hyperplane in R d spanning the first ( d -1) dimensions (axes), centered at the origin. C is a cylinder of unit length and radius 1 2 placed unit distance away from T in the positive e d -direction.

Let S d -1 = { x ∈ R d : ∥ x ∥ 2 = 1 } be the unit-sphere, centered at the origin, in R d , and let

That is, N is the set of vectors u on the unit hypersphere with angle ≥ π 6 from e d . For u ∈ N , define the 'rotatiou' matrix

where { v 2 , . . . , v d } is any orthonormal basis for { u } ⊥ (where { u } ⊥ denotes the subspace orthogonal to the subspace spanned by the set of vectors { u } ). 4

Now, for σ > 0 , t ∈ T , and u ∈ N , define the Gaussian

For all k ≥ 1 , let

That is, each Q k is a class of 'flat' (i.e., near ( d -1) -dimensional) Gaussians in R d , with σ 2 = 1 k 2 variance on a single thin direction u and unit variance in all other directions. Their mean vectors come from a point on the hyperplanar disk T (which we recall is a ( d -1) -dimensional disk orthogonal to e d ), and the thin direction u comes from N (which is S d -1 excluding points that form angle < π 6 with e d ). As k →∞ , the Gaussians get flatter.

Lower bounding the weight of B . We start with the following claim, which shows the probability that d -1 samples drawn the uniform mixture of Q d -1 k all fall into the cylinder C can be uniformly lower bounded by an absolute constant, independent of k .

Claim G.2. Let B be the set of all possible vectors of d -1 points in the cylinder C , i..e, B = C d -1 ∈ ( R d ) d -1 . There exists η > 0 such that for k ≥ 10 ,

Proof of Claim G.2. Consider the inscribed cylinder C ′ ⊆ C

Also, consider T ′ ⊆ T and N ′ ⊆ N :

Now, fix u ∈ N ′ and t ∈ T ′ . Define the plane going through t with normal vector u as,

First, we show P ( u, t ) ∩ C ′ contains a ( d -1) -dimensional region. Consider,

The projection onto P ( u, t ) of y is given by,

where | c | = | ( y -t ) · u | ≤ 3 2 · 1 36 = 1 24 . Since ∥ t ∥ 2 ≤ 1 4 , the norm of the first ( d -1) dimensions of y ′ is ≤ 1 4 + 1 24 ≤ 1 3 and y ′ d ∈ [ 3 2 -1 24 , 3 2 + 1 24 ] , and so y ′ ∈ C ′ . Moreover, adding any z with z · u = 0 and ∥ z ∥ 2 ≤ 1 24 results in y ′ + z with the norm of the first d -1 dimensions being at most 1 4 + 1 24 + 1 24 ≤ 1 3 and ( y ′ + z ) d ∈ [ 3 2 -1 12 , 3 2 + 1 12 ] . Hence, y ′ + z ∈ C ′ . This shows that P ( u, t ) ∩ C ′ contains a ( d -1) -dimensional subspace, since it contains a ( d -1) -dimensional disk of radius 1 24 .

Next, let

That is, M is a rectangular 'extrusion' of C ′ ∩ P ( u, t ) along both its normal vectors. Indeed, we have M ⊆ C , since adding a vector of length ≤ 1 6 cannot take a point in C ′ outside of C . We also have that M is a d -dimensional region, so

Note that for σ ≤ 1 10 , we have

This is because any x ∈ M can be written as t + x + cu , where x is such that x · u = 0 , and | c | ≤ 1 6 . Plugging in this decomposition of x into the densities of G (1 / 10 , u, t ) and G ( σ, u, t ) , and simplifying yields the above.

To conclude, for k ≥ 10 , we have

where c = f T ( t ) · f N ( u ) > 0 is the uniform density over T × N . Note that the final integral is non-zero since T ′ × N ′ has non-zero measure in T × N and that P X ∼ G (1 / 10 ,t,u ) { X ∈ M } is indeed non-zero for all t ∈ T ′ , u ∈ N ′ .

Upper bounding r k , the weight of α -TV balls. We prove the following.

Then we have,

Note that

We use the following three facts regarding total variation distance, Gaussians, and the surface area of hyperspherical caps.

Fact G.4 (Data-processing inequality for TV distance) . Let p, q ∈ ∆( X ) . For any measurable f : X → Y ,

where for p ∈ ∆( X ) , f ( p ) denotes the push-forward distribution assigning for all measurable A ⊆ Y , f ( p )( A ) = p ( f -1 ( A )) .

Fact G.5 (TV Distance between 1 -Dimensional Gaussians [DMR18, Theorem 1.3]) . Let N ( µ 1 , σ 2 1 ) and N ( µ 2 , σ 2 2 ) be Gaussians over R . Then

Fact G.6 (Surface area of hyperspherical caps [Li10]) . For u ∈ S d -1 and θ ∈ [0 , π 2 ] , define

where for u, v ∈ S d -1 , ∠ ( u, v ) := cos -1 ( u · v ) . We have

Proof of Claim G.3. Let σ > 0 . Let t 1 , t 2 ∈ T and u 1 , u 2 , ∈ N . We will compare the total variation distance of the Gaussians defined by these parameters. Let

By Fact G.4, taking f : R d → R to be f ( x ) = u ⊤ 1 ( x -t 1 ) ,

where ∆ t = t 2 -t 1 . For the last line above, we take R u 2 = [ u 2 , v 2 , . . . , v d ] , where { v 2 , . . . , v d } is an orthonormal basis for { u 2 } ⊥ . Then the equality in the last line for the variance of the second Gaussian uses,

where R u 2 being unitary implies that ( u 1 · u 2 ) 2 +( u 1 · v 2 ) 2 + · · · +( u 2 · v d ) 2 = 1 , yielding the second equality in the above.

We show that if ∠ ( u 1 , u 2 ) ∈ [ √ 2 π 2 σ, π -√ 2 π 2 σ ] , TV( G ( σ, t 1 , u 1 ) , G ( σ, t 2 , u 2 )) ≥ 1 200 . First, we consider the case where ∠ ( u 1 , u 2 ) ∈ [ √ 2 π 2 σ, π 2 ] . Using that on [0 , π 2 ] , we have sin( x ) ≥ 2 π x and cos( x ) ≥ 0 , we get

Therefore,

This implies that

and by Fact G.5, we can conclude that TV( G ( σ, t 1 , u 1 ) , G ( σ, t 2 , u 2 )) ≥ 1 200 . Now, consider the case where ∠ ( u 1 , u 2 ) ∈ [ π 2 , π -√ 2 π 2 ] . Note that in this case, there exists u ′ 2 = -u 2 ∈ [ √ 2 π 2 , π 2 ] with G ( σ, t 2 , u 2 ) = G ( σ, t 2 , u ′ 2 ) , bringing us back to the previous case.

Next, note that since ∥ u 1 ∥ 2 = 1 and | u ( d ) 1 | = | u 1 · e d | ≤ √ 3 2 (by the definition of N ), letting r = [ u (1) 1 , . . . , u ( d -1) 1 ] ⊤ ∈ R d -1 , we have ∥ r ∥ 2 ≥ 1 2 . Let ˆ r = r ∥ r ∥ 2 . We have that if [∆ t 1 , . . . , ∆ t d -1 ] ⊤ · ˆ r ≥ 1 20 σ , then,

and by Fact G.5, we can conclude TV( G ( σ, t 1 , u 1 ) , G ( σ, t 2 , u 2 )) ≥ 1 200 . Therefore, for any u ∈ N , t ∈ T ,

For the first term, note that the event

which under U ( T ) , for some c d > 0 depending only on d , has probability ≤ c d · 1 20 k .

For the second term, note that ∠ ( u, u ′ ) ∈ [0 , √ 2 π 2 k ) ∪ ( π -√ 2 π 2 k , π ] means u ′ ∈ C ( u, √ 2 π 2 k ) ∪ C ( -u, √ 2 π 2 k ) . By Fact G.6, we know that under U ( N ) , for some c d depending only on d ,

The bound is the same for C ( -u, √ 2 π 2 k ) . Plugging these into the above, we can conclude that

This proves the claim.

Lower bounding s k , the weight of alternative hypotheses. First, we note that

which is achieved by G ( 1 k , 0 , e 1 ) (where 0 ∈ R d is the origin) and x = ( e d , . . . , e d ) . Let

Claim G.7. For k ≥ 1 , letting ( u k ) ∞ k =1 and c be defined as above, define

Proof of Claim G.7. Let k ≥ 1 . Fix any x = ( x 1 , . . . , x d -1 ) ∈ B . For every t ∈ T , there exists u ∈ { x 1 -t, x 2 -t, . . . , x d -1 -t } ⊥ . We show ∠ ( u, e d ) ≥ π 4 . Suppose otherwise, that is, ∠ ( u, e d ) < π 4 = ⇒ | u · e d | = | u ( d ) | > √ 2 2 . Then,

By our assumption on u ( d ) , and by the fact that x 1 ∈ C , we have that | u ( d ) x ( d ) 1 | > √ 2 2 . By Cauchy-Schwarz in R d -1 , we have that,

Then we have,

The last inequality uses that ∥ u ∥ 2 = 1 and ( u ( d ) ) 2 > 1 2 , so the norm of the first ( d -1) coordinates is < √ 2 2 , and also the fact that the first ( d -1) coordinates of x and t are in the 1 2 -disk. This inequality, combined with the fact that | u ( d ) x ( d ) 1 | > √ 2 2 contradicts that that ( x 1 -t ) · u = 0 .

Now, for the t and the u from above, consider an arbitrary u ′ with ∠ ( u, u ′ ) ≤ 1 k , and the Gaussian with mean t and normal vector u ′ , G ( 1 k , t, u ′ ) . We will show that any such Gaussian assigns high mass to the point x , and furthermore that there is a high density of such Gaussians. Note that for k ≥ 5 , 1 k ≤ π 3 -π 4 = ⇒ ∠ ( u ′ , e d ) ≥ π 3 = ⇒ u ′ ∈ N . We compute the minimum density this Gaussian assigns to x . Consider, for i ∈ [ d -1] ,

where ˆ r = ( x i -t ) / ∥ x i -t ∥ and { v 2 , . . . , v d } is an orthonormal basis of { u ′ } ⊥ . We have that,

Hence, the density of G ( 1 k , t, u ′ ) on x is lower bounded by,

For every t ∈ T , we found a set of u ′ ∈ N such that the density G ( 1 k , t, u ′ ) assigns to x is greater than cu k . Since for some constant c d > 0 depending only on d ,

and since x ∈ B was arbitrary, we indeed have,

This completes the proof of the claim.

With the three claims, applying Lemma G.1 allows us to conclude that the class of all Gaussians in R d is not list learnable with m ( α, β ) = d -1 samples. This implies that the class is also not public-privately learnable with m ( α, β, ε ) = d -1 public samples.

H Statements and proofs for Section 7 - Learning when the Yatracos class has finite VC dimension

When the VC dimension of the Yatracos class of Q is finite, the following gives an upper bound on the number of samples required to non-privately learn Q .

Fact H.1 ([Yat85], [DL01, Theorem 6.4]) . Let Q ⊆ ∆( X ) . Let H be the Yatracos class of Q , and let d = VC( H ) . Q is learnable with

samples.

For some classes of distributions, the above bound is tight. For example, it recovers the Θ( d 2 α 2 ) sample complexity for learning Gaussians in R d [AM18].

To prove Theorem 7.2 we employ the following result on generating distribution-dependent covers for binary hypothesis classes with public data.

Fact H.2 (Public data cover [ABM19, Lemma 3.3 restated]) . Let H ⊆ 2 X and VC( H ) = d . There exists A : X ∗ →{ H ⊆ 2 X : | H | < ∞} , such that for any α, β ∈ (0 , 1] there exists

such that for any p ∈ ∆( X ) , if we draw X = ( X 1 , . . . , X m ) i.i.d. from p , with probability ≥ 1 -β , A ( X ) outputs ̂ H ⊆ 2 X and a mapping f : H → ̂ H with

(where for A,B ⊆ X , A △ B denotes the symmetric set difference ( A \ B ) ∪ ( B \ A )) . Furthermore, we have | ̂ H| ≤ ( em d ) 2 d .

We also use the following pure DP algorithm for answering counting queries on finite domains. Fact H.3 (SmallDB [BLR13], [DR14, Theorem 4.5]) . Let X be a finite domain. Let H ⊆ 2 X . Let α, β ∈ (0 , 1] and ε > 0 , There is an ε -DP randomized algorithm, that on any dataset x = ( x 1 , . . . , x n ) with

outputs estimates ˆ g : H → R such that with probability ≥ 1 -β ,

Proof of Theorem 7.2. We use our m public samples from the unknown p ∈ Q to generate a public data cover ̂ H and mapping f : H → ̂ H courtesy of Fact H.2, selecting m to target error α 6 and failure probability β 3 . Note that this implies that with probability ≥ 1 -β 3 , for every h ∈ H , | p ( h ) -p ( f ( h )) | ≤ | p ( h △ f ( h )) | ≤ α 6 .

Next, we consider the representative domain of X with respect to ̂ H , denoted by X ̂ H . In other words, for every unique behaviour ( 1 ˆ h ( x )) ˆ h ∈ ̂ H ∈ { 0 , 1 } | ̂ H| induced by a point x ∈ X on ̂ H , we include exactly one representative [ x ] in X ̂ H . By Sauer's lemma we can conclude that

Then, we take our n private samples X = ( X 1 , . . . , X n ) and map each point X i to its representative [ X i ] ∈ X ̂ H , yielding a dataset of n examples [ X ] on the finite domain X ̂ H . Note that for any ˆ h ∈ ̂ H , 1 n ∑ n i =1 1 ˆ h ( X i ) = 1 n ∑ n i =1 1 ˆ h ([ X i ]) . Hence when we run SmallDB (Fact H.3) on the input [ X ] over the finite domain X ̂ H with finite class ̂ H , choosing n large enough, we obtain ˆ g : ̂ H → R such that with probability ≥ 1 -β 3 , | ˆ g ( ˆ h ) -1 n ∑ n i =1 1 ˆ h ( X i ) | ≤ α 6 for all ˆ h ∈ ̂ H .

We also ensure n is large enough so that we get the uniform convergence property on ̂ H , which has VC dimension d , with the private samples. That is, for all ˆ h ∈ ̂ H , with probability ≥ 1 -β 3 , | p ( ˆ h ) -1 n ∑ n i =1 1 ˆ h ( X i ) | ≤ α 6 .

As a post-processing of ˆ g , our learner outputs

By the union bound, with probability ≥ 1 -β , all of our good events occur. In this case, we have for all h ∈ H ,

which implies | p ( h ) -ˆ g ( f ( h )) | ≤ α 2 . So for any q ∈ Q ,

We have that

Therefore,

It can be verified that the choices of m and n in the statement of Theorem 7.2 suffice.

Peer reviews:
comments: SUMMARY:
This paper studies differentially private learnability of probability distributions.
The results put forward are very interesting!
For example, they show that the public-private learnability of a class of distributions is connected to the existence of sample compression schemes as well as to an intermediate notion of learning they define.

STRENGTHS:
Originality:
This work is original, to the best of my knowledge.

Quality:
Quality of results and topic choice are very good; see questions.

Clarity:
Writing is clear; see questions.

Significance:
The results are very significant for the learning community.

WEAKNESSES:
Not significant weaknesses detected.

QUESTIONS:
Are there any complexity-theoretic implications of your work?
Learning is connected to complexity lower bounds.
Is this the case for private learning?
Maybe some stronger connection exists?

Line 43:
Nice question!
Can you please discuss similar questions?

Theorem 1.1:
You should emphasize this result more, as it is a main important result.

Line 118:
Please define TV.
(This is in the appendix...)

Line 142:
Please define $SC$ (as "sample complexity?").

General comment:
Since you are not including proofs in the main body, maybe simplify the bounds you are presenting?
This will save space and thus allow for more explanations.

Section 5:
Can you please better explain the notion of distribution shifted?

Section 6 and Section 7:
Can you please add some more intuition here?

Maybe put limitations in the main body?

There is no conclusion section!
Why?
You could put open problems there.

This paper has so many novel results (proofs in appendix).
Impressive!

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-24

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider the problem of learning a distribution in a known class using private samples under pure DP, as well as a number of public samples drawn from the same distribution. Some distribution classes, such as Gaussians, are not learnable using a finite number of private samples, or even with a finite number of private samples and a small number of public samples, but are learnable with private samples and a medium number of public samples.\n\nThe authors make progress towards understanding what classes of distributions are learnable with what number of public samples. Their main contribution is to establish that the following three are equivalent (up to constants): (1) A class of distributions is public-private learnable, i.e. learnable with m public samples and some finite number of private samples (2) A class of distributions is realizably compressible with m samples, i.e. given a finite number of samples from a distribution, one can encode the distribution into m samples and some finite number of auxiliary bits such that some decoder can learn the distribution using only the encoded information (3) A class is list learnable with m samples, i.e. there is an algorithm that takes m samples and outputs a finite list of distributions such that one distribution in this list is close to the target distribution. The authors also give concrete/non-trivial bounds on the finite number of samples/finite list size in the equivalence statement.\n\nThe above equivalence has a number of applications. For example, since mixtures of Gaussians are known to be compressible, the above equivalence immediately shows they are public-private learnable. More generally, since the class of mixtures (resp. products) of distributions from compressible classes is also compressible, mixtures (resp. products) of distributions from public-private learnable classes are also public-private learnable. While BKS22 established public-private learnability of Gaussians, the authors' result has several qualitative advantages, such as a stronger notion of DP and allowing arbitrarily small mixture probabilities. Similarly, the authors show that d-dimensional Gaussians are not learnable with d-1 public samples and any number of private samples, by showing that this implies list-learnability of d-dimensional Gaussians with d-1 public samples, which violates a lower bound. \n\nThe authors also extend the reduction from compressible classes to public-private learnable classes, to a reduction from robust compressible classes to classes that are public-private learnable, with out-of-distribution public samples and a private distribution that is only close to a member of the class. \n\nFinally, the authors show that classes of distributions whose Yatracos class has VC dimension d and dual VC dimension d* are learnable with d private samples and d^2 times d* private samples.\n\nSTRENGTHS:\nOverall I feel the paper is quite strong. The main contribution of the paper, the equivalence of three properties of a class of distribution, is elegant and likely to have high impact, as distribution learning is obviously a very central and fundamental problem in learning theory, and using public data for private learning is becoming increasing popular. As an example of the potential for impact, just from the equivalence of these three classes, the authors are able to show a lower bound on the number of public samples needed to learn a Gaussian that is optimal up to an additive difference of 1, which was an open problem beforehand. I am not very confident in my knowledge of the field of distribution learning, but I suspect the elegance of the result could easily lend itself to understanding the public-private learnability of other important classes of distributions. In addition to this main contribution, there are a number of other contributions that are independently interesting. The paper is also technically interesting and novel. e.g, this paper is the first to use list-learning (which is related to, but not the same as list-decodable learning) in establishing upper and lower bounds for other definitions of learnability. The paper was enjoyable and easy to read - while there are several involved definitions needed to fully understand the results, the authors do a good job distilling the results down into a form that's easier to parse, without sacrificing too much detail.\n\nWEAKNESSES:\nI felt the main \"weakness\" of the paper is that there are many results the authors provide, and so the authors had to save most of the discussion of proof techniques for the appendix. In particular, I felt Sections 6 and 7 did not offer much more insight than the related sections of the introduction, and could be moved to the appendix in favor of e.g. a proof sketch for Prop 3.2, which would help the reader better understand the equivalence between the different definitions of learnability. Of course, such a weakness is likely to not exist in a camera-ready or arxiv version.\n\nQUESTIONS:\n-You mention that you do not optimize the private sample complexity - do you have a sense for how large the slack in the private sample complexities might be? (i.e., are they optimal up to log factors, small polynomials, large polynomials?)\n\nLIMITATIONS:\nYes; in Appendix A the authors nicely state the limitations of their work. \nN/A on negative societal impact, since it is a theory work on improving privacy-preserving algorithms.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nContinuing a line of work recently initiated by Bie, Kamath, and Singhal (NeurIPS '22), this submission studies differentially private distribution learning (a.k.a. hypothesis selection) in the presence of public data. The main result is a collection of quantitative equivalences between this task and the problems of sample compression and list learning. More specifically, the paper shows that for any class of distributions Q, the following are (roughly) equivalent:\n1) Q admits a sample compression scheme using m samples and compression size t\n2) Q is pure $\\varepsilon$-DP learnable using m public samples and t private samples\n3) Q is list learnable using m samples and list size exp(t).\n\nAs applications of this set of equivalences, the paper proves new upper and lower bounds on the sample complexity of pure public-private learning. These include new algorithms for learning Gaussians, mixture distributions, and product distributions via sample compression, as well as a lower bound on the public sample complexity of pure public-privately learning a single Gaussian that follows from a lower bound for list learning. The paper also works out a connection between robust sample compression and agnostic/distribution-shifted public-private learning, and a pure public-private learner for classes whose Scheffe sets have finite VC dimension.\n\nSTRENGTHS:\n- The equivalences shown in this paper provide a nice set of perspectives on the public-private distribution learning problem, both conceptually and \"practically\" in terms of offering new avenues for proving upper and lower bounds.\n\n- The proofs of these equivalences are simple, conceptually clear, and illuminating.\n\n- I'd say the main technical contribution in this paper is a lower bound showing that it is impossible to list-learn d-dimensional Gaussians using fewer than d samples. (This in turn implies a lower bound of public-private learning.) This is a sharp result that matches an upper bound from BKS'22. The proof goes through a general lower bound technique for list-learning, called a \"no-free-lunch\" theorem in this paper, roughly based on exhibiting collections of distributions that are not close in TV distance, but easily confused using a small number of samples.\n\nWEAKNESSES:\n- The characterizations shown in this work all apply to pure differential privacy. This is good for upper bounds, but is a major limitation of the most technically-involved contribution of the paper on lower bounding list- / public-private learnability of Gaussians. In particular, the conversion from a public-private learner to a list-learner depends crucially on pure differential privacy; while it can probably be extended to a quantitatively weaker result for concentrated DP, it doesn't say anything about, say, Renyi or approximate DP.\n\n- The paper's conversion from a public-private learner to a list-learner is not quite constructive, as it relies on a covering-based characterization of pure private distribution learning. However, this characterization can probably be unrolled into an algorithmic construction of a cover using the usual technique of running the DP algorithm repeatedly on a fixed dataset using many sequences of coin tosses. Also, none of the equivalences proved preserve computational efficiency in general.\n\nQUESTIONS:\n- Lines 235-239 offer a comparison to the previous result of BKS'22 on learning mixtures of Gaussians, but isn't easy to make sense of without a brief (informal) statement of what their result actually is.\n\n- Line 500: Should be $\\tau_0 = \\tau(\\alpha/6, \\beta/2)$\n\n- While ultimately short, I found the contradiction-based proof of Proposition 3.5 unnecessarily hard to follow for this reason. Why not just do a direct proof along the following lines, using roughly the same calculations? For every distribution $p \\in \\mathcal{Q}$, accuracy of $\\mathcal{A}$ implies that whp over $\\widetilde{X} \\sim p^m$, we have $p \\in Q_{\\widetilde{X}}$. This latter event implies $p$ is close to $\\mathcal{L}(\\widetilde{X})$, which is the success condition of the list learner.\n\n- I didn't understand why Lemma F.1 is called a \"no free lunch\" theorem. To my mind, a no free lunch result rules out the existence of learning algorithms that are simultaneously optimal for a sufficiently broad class of problems. Did the authors have in mind a different context in which this name makes sense to describe this result?\n\nLIMITATIONS:\nSome limitations (overlapping with weaknesses described above) are discussed in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn public-private learning, an algorithm gets two sets of i.i.d. samples from an unknown distribution $q$ from a class $\\cal{Q}$ (say, the first set of size $m$ and the second one of size $n$), but is required to preserve DP only with respect to the second set. \nThis research direction is about understanding what are the (minimal) regimes of $m$ and $n$ that allows to learn a distribution from $\\cal{Q}$. The interesting regimes are when $m$ is much smaller than the sample complexity required to learn without privacy, and $n$ is smaller than what is required for learning under (full) DP. \nThis research direction has been initiate by Bie, Kamath and Singhal (NeurIPS 22’, BKS22), who showed that a relatively small amount of public samples suffices to improve the sample complexity of privately learning a single Gaussian and a mixture of Gaussians. \nThis work makes a significant step towards characterizing public-private learning by proving that it is equivalent (in terms of sample complexity) to two other notions: (1) Compressible Learning (Ashtiani et al., J.ACM 2020, ABDH+20) and (2) List-Learning. In more detail (ignoring running times and low order terms and constants in the sample complexity parameters):\n\nProposition 3.2: $(\\tau, t, m)$-compressible learning implies $(m,n = t + \\tau \\log m)$-public-private learning, where the former means that we encode $m$ samples using a $\\tau$ size subset plus an auxiliary string of size $t$, and this information suffices for estimating the distribution. \n\nProposition 3.5: $(m,n)$-public-private learning implies $(m,\\ell=\\exp(n))$-list learning, where the latter means that using $m$ samples we can output a list of size $\\ell$ of candidate distributions that at least one of them is a good estimation to the true distribution.\n\nProposition 3.6: $(m,\\ell)$-list learning implies $(\\tau=m, t = \\log(\\ell), m)$-compressible learning. \n\nThe main applications comes from the equivalence to compressible learning, since [ABDH+20] proved that learning a Gaussian or a mixture of Gaussians is compressible learnable with relatively small amount of samples, so using this connection they deduce an improved sample complexity upper bounds for these tasks under private-public learning, as well as a tight lower bound of the number of public samples that is required for learning a single Gaussian. The upper bound on public-private learning a mixture of Gausians is actually an upper bound for learning any mixture of distributions for a given class $\\cal{Q}$ as a function of the sample complexity that is required to learn a single distribution from $\\cal{Q}$, as [ABDH+20] proved that for compressible learning, learning a $k$ mixture requires (esseitially) $k$ times the sample complexity for a single distribution.\n\nIn section 5 they also handle the agnostic case where the samples comes from a distribution that is only close to a one in the class, a setting which was also considered in the previous work of [BKS22]. They prove (Theorem 5.2) that robust compression implies agnostic public-private learning, and deduce an upper bound on learning a Gaussian in this case.\n\nSTRENGTHS:\nThe equivalences between public-private learning to the other notions of learning are very interesting, and the progress from the work of [BKS22] (both quantitative and qualitative) is significant. Despite the significant limitations (described below), in my opinion, this is a good submission which is above the acceptance threshold. Therefore, I recommend acceptance.\n\nWEAKNESSES:\nAs mentioned in the limitations section (Section A, Supplementary material), the reductions between the different notions are inefficient, and sometimes not even constructive (e.g., in the reduction from list learning to public-private learning). So all the new upper bounds of this paper are inefficient, and it is still left open whether we can construct efficient public-private learning algorithms with the improved samples complexities achieved in this paper.\nWhile the results of [BKS22] are more restricted and weaker in terms of sample complexity, they are computationally efficient (unlike this work), so if we are only interested in computationally efficient learners, this work does not make any progress except the lower bound which holds (in particular) for efficient learners.\n\nQUESTIONS:\nNo questions.\n\n\nOne suggestion:\n\nI think it is worth adding short proof sketches of the reductions (Propositions 3.2, 3.5, 3.6) in the main body. The proofs are not long, and I'm sure you can explain the idea in each proof using a few sentences.\n\nMinor comments:\n\n(1) Theorem 1.1, item 2: \"$\\cal{Q}$ is has\".\n\n(2) Theorem 1.3: robust compression is not defined at this point.\n\n(3) Definition 2.5: In ABDH+20, $f_q$ is explicitly defined by outputting an $\\tau_0$-subset of samples, and a short string. Here, you just claiming the existence of such $f_q$, which is strange to me since, for example, if $f_q$ is randomized and depends on $q$, it could simply ignore the input samples (which are sampled from a distribution that is only close to $q$) and just use fresh samples from $q$.\n\n(4) Corollary 4.2: I think it is worth comparing the sample complexity to existing efficient and fully DP methods like https://arxiv.org/abs/2303.04288 and https://arxiv.org/pdf/2112.14445.pdf (BKS22 only compared their result to KSSU19 https://arxiv.org/abs/1909.03951 which is significantly inefficient compared to the first two results I mentioned).\n\nLIMITATIONS:\nThe authors present the limitations in Section A (supplementary material).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-09"}