Title: On Differentially Private Sampling from Gaussian and Product Distributions
Venue: NeurIPS 2023
Accepted: yes
Keywords: privacy

sampling

Gaussian distribution

product distributions
Abstract: We study the problem, where given a dataset of $n$ i.i.d. samples from an unknown distribution $P$, we seek to generate a sample from a distribution that is close to $P$ in total variation distance, under the constraint of differential privacy. We study the settings where $P$ is a multi-dimensional Gaussian distribution with different assumptions: known covariance, unknown bounded covariance, and unknown unbounded covariance. We present new differentially private sampling algorithms, and show that they achieve near-optimal sample complexity in the first two settings. Moreover, when $P$ is a product distribution on the binary hypercube, we obtain a pure-DP algorithm whereas only an approximate-DP algorithm (with slightly worse sample complexity) was previously known.

On Differentially Private Sampling from Gaussian and Product Distributions

Abstract

Given a dataset of n i.i.d. samples from an unknown distribution P , we consider the problem of generating a sample from a distribution that is close to P in total variation distance, under the constraint of differential privacy (DP). We study the problem when P is a multi-dimensional Gaussian distribution, under different assumptions on the information available to the DP mechanism: known covariance, unknown bounded covariance, and unknown unbounded covariance. We present new DP sampling algorithms, and show that they achieve near-optimal sample complexity in the first two settings. Moreover, when P is a product distribution on the binary hypercube, we obtain a pure-DP algorithm whereas only an approximateDP algorithm (with slightly worse sample complexity) was previously known.

1 Introduction

Differential privacy (DP) [18, 16] is a strong and rigorous notion of privacy that has been increasingly studied and deployed as protection against the leakage of personal data used to train ML models.

A basic setting widely studied in ML is distribution learning , where given samples drawn i.i.d. from an unknown distribution P , we seek to output a distribution Q that is as close to P as possible (formal definitions are given in Section 1.1). Recent works [27, 2, 29, 4] have studied DP distribution learning , whereby the output distribution is guaranteed to stay roughly the same when a single input sample is changed. A closely related setting is DP mean and covariance estimation studied by [38, 21, 36, 30, 7, 8, 11, 15, 28, 25, 9, 26]. Motivated by the fact that tasks often require much fewer samples than full-fledged learning, the very recent work of [35] studied the task of DP distribution sampling , where the goal is to generate a sample from a distribution Q that is as close as possible to the distribution P from which the input samples are drawn i.i.d. (In the non-private setting, variants of the sampling task have also been studied, e.g., in the work on sample amplification by [5].)

In this work, we study DP distribution sampling, and the quantitative gap with respect to the a priori more challenging task of DP distribution learning. For the case of multi-dimensional Gaussians, we consider three natural settings: known covariance, unknown bounded covariance, and unknown unbounded covariance; for the first two, we obtain near-tight bounds. This answers an open question of [35]. Moreover, we obtain near-tight bounds on sampling in the case of product distributions on { 0 , 1 } d with pure-DP, also improving upon previous work.

Motivation. There are many natural settings where a sample from the underlying distribution would be sufficient (as an alternative to the possibly more expensive task of learning the distribution).

For example, if one is implementing an algorithm that would run on sensitive user data, then, as part of the usual software development cycle, unit tests [24] are important. Writing unit tests on real data can violate users' privacy (e.g., if the unit test code is public). On the other hand, (privately) learning the distribution of the user data could be a significant overkill for this application. A private sample of the underlying distribution would provide a sufficiently representative input for the unit testing algorithm, and this input could be revealed publicly without compromising the privacy of the users.

A second motivation arises in distributed settings. Consider the situation where algorithms are being developed on medical data provided by multiple hospitals. The algorithm designer would like a synthetic dataset that would work well for the algorithm development process. Such a dataset could be obtained if each hospital provides a few (private) samples from its underlying distribution.

Finally, we describe another distributed setting motivation where a central curator wishes to build a synthetic dataset from multiple users, each contributing multiple items, under the constraint of item-level DP (as opposed to user-level DP). A simple approach for generating such a synthetic dataset is to let each user send to the curator a private sample (satisfying item-level DP).

1.1 Formulation

Let D be a class of distributions on some domain X . We consider a setting where there is an unknown distribution D ∈ D and an algorithm has sample access to D . There are two natural problems that can be posed in this setting. In the sampling problem , the goal is to design an algorithm that uses samples from D (which is unknown) and outputs an element in X . We say that an algorithm A is an α -accurate sampler for D iff d tv ( D,Q A ,D ) ≤ α for all D ∈ D , where Q A ,D denotes the distribution of A ( X ) where X ∼ D n and d tv ( · , · ) denotes the total variation distance. Here n is said to be the sample complexity of the sampler. In the learning problem , the goal is to design an algorithm that outputs a distribution D ′ ∈ D . We say that an algorithm A is an ( α, β ) -accurate learner for D if Pr X ∼ D n ,D ′ ∼A ( X ) [ d tv ( D ′ , D ) ≤ α ] ≥ 1 -β for all D ∈ D ; as before, n is the sample complexity of the learner.

The DP version of the learning problem is well studied, e.g., [27, 4]; in this work we study the DP version of the sampling problem. First, we recall the definition of DP. We consider the substitution notion, i.e., two datasets are neighbors iff they have the same number of samples and we can transform one to the other by changing a single sample.

Definition 1.1 (Differential Privacy [18, 16]) . An algorithm M : Y → O is said to be ( ϵ, δ ) -differentially private ( ( ϵ, δ ) -DP) for ϵ > 0 , δ ≥ 0 iff, for every S ⊆ O and every neighboring datasets Y, Y ′ ∈ Y , we have Pr[ M ( Y ) ∈ S ] ≤ e ϵ · Pr[ M ( Y ′ ) ∈ S ] + δ .

̸

We abbreviate ( ϵ, 0) -DP by ϵ -DP, aka, pure-DP ; the δ = 0 case is approximate-DP . We assume that the privacy parameters satisfy ϵ ≤ 1 , δ ≤ 1 / 2 , and the accuracy parameter satisfies 0 < α ≤ 1 / 2 .

1.2 Our Results for Gaussian Distributions in R d

We focus on the d -dimensional Gaussian distribution, i.e., D = N ( µ, Σ) with µ ∈ R d , Σ ∈ R d × d . Recall that N ( µ, Σ) is supported on R d with f N ( µ, Σ) ( x ) ∝ exp ( -1 2 ( x -µ ) T Σ -1 ( x -µ ) ) . We study three cases that have been considered in the literature:

Known Covariance: Σ is known, but µ is not. More formally, the class of distributions is D N Σ := { N ( µ, Σ) | µ ∈ R d } .

Unknown Bounded Covariance: Both µ, Σ are unknown but with a promise that I ⪯ Σ ⪯ κ · I for a known constant κ > 0 , i.e., the class of distributions is D N κ := { N ( µ, Σ) | µ ∈ R d , Σ ∈ R d × d s.t. I ⪯ Σ ⪯ κ · I } . 2

Unknown Unbounded Covariance: Both µ, Σ are unknown, i.e., the class of distributions is D N := { N ( µ, Σ) | µ ∈ R d , Σ ∈ R d × d } .

A summary of our DP sampling results along with a comparison to known DP learning results is in Table 1. Before we describe our results in more detail, we highlight the following. (i) In all cases,

the dependence of our algorithms on the accuracy parameter α is only polylogarithmic, whereas for DP learning algorithms, this dependence is polynomial. (ii) In the case of known covariance and unbounded known covariance, we obtain improvements over DP learning in terms of the dependence on the dimension d . (iii) All of our algorithms run in polynomial time, although we do not explicitly state the running time in the formal statements.

Known Covariance. While any DP learning algorithm requires ˜ Θ ( d α 2 + d αϵ ) samples in this setting [27], we show that, surprisingly, only ˜ O ( √ d/ϵ ) samples suffice for DP sampling.

Theorem 1.2. There is an α -accurate ( ϵ, δ ) -DP sampler for Gaussian distributions with known covariance with sample complexity O ( √ d ϵ · polylog ( d δϵα ) ) .

We also show that the √ d dependence is necessary, i.e., that our algorithm's sample complexity is tight up to polylogarithmic factors.

Theorem 1.3. Any 0.1-accurate ( ϵ, δ ) -DP sampler for Gaussian distributions with known covariance must have sample complexity Ω( √ d/ϵ ) .

Unknown Bounded Covariance. In this setting, we obtain an algorithm with sample complexity of ˜ O κ ( d ϵ ) ; in contrast, the best known DP learning algorithm requires ˜ Θ κ ( d 2 α 2 + d 2 αϵ ) samples [27, 29].

Theorem 1.4. There is an α -accurate ( ϵ, δ ) -DP sampler for Gaussian distributions with unknown covariance, under the assumption I ⪯ Σ ⪯ κ · I , with sample complexity O ( d ϵ · κ 2 · polylog ( d δϵα )) .

Similar to before, we can show that the sample complexity dependence on d, ϵ is near-optimal:

Theorem 1.5. Let α > 0 be a sufficiently small constant, and ϵ, δ be such that δ ≤ O ( 1 nd 2 ) . Then, any α -accurate ( ϵ, δ ) -DP sampler for Gaussian distributions with unknown covariance, under the assumption I ⪯ Σ ⪯ 2 I , must have sample complexity n = Ω ( d ϵ √ log d ) .

Unknown Unbounded Covariance. In this setting, the best known DP learning algorithm uses ˜ Θ ( d 2 α 2 + d 2 αϵ ) samples [4, 29, 28]. For DP sampling, we show that we can reduce the dependence on α to polylogarithmic and the dependence on d to d 1 . 5 .

Theorem 1.6. There exists an α -accurate ( ϵ, δ ) -DP sampler for Gaussian distributions (without any assumption) with sample complexity O ( d 1 . 5 ϵ polylog ( d αϵδ ) ) .

1.3 Our Results for Product Distributions on { 0 , 1 } d

For p ∈ [0 , 1] , let Ber ( p ) be the Bernoulli distribution supported on { 0 , 1 } with probability mass function f Ber ( p ) (0) = 1 -p and f Ber ( p ) (1) = p . We consider product distributions Ber ( p 1 ) ⊗ · · · ⊗ Ber ( p d ) where p 1 , . . . , p d ∈ [0 , 1] are unknown. (In other words, the class of distributions is D prod := { Ber ( p 1 ) ⊗··· ⊗ Ber ( p d ) | p 1 , . . . , p d ∈ [0 , 1] } .)

We give a pure-DP sampler with sample complexity ˜ O ( d αϵ ) . Previously, only approximateDP sampler with similar sample complexity was known from [35], which also provided a matching lower bound. In comparison, DP learning uses ˜ Θ ( d α 2 + d αϵ ) samples [27, 10].

Theorem 1.7. There exists an α -accurate ϵ -DP sampler for product distributions on { 0 , 1 } d with sample complexity O ( d log ( d α ) αϵ + d log 2 ( d α ) ϵ ) .

Table 2: Sample complexity for private learning and sampling for product distributions on { 0 , 1 } d . Here, ˜ Θ hides factors that are polylogarithmic in d, 1 /α (and 1 /β in the case of learning).

We also note that our result above improves upon even the approximate-DP sampler in [35] by logarithmic factors. Specifically, for α ≤ 1 / log d , our sample complexity is O ( d log( d/α ) αϵ ) whereas

2 Technical Overview

2.1 Gaussian Distributions: Algorithms

Known Covariance. When Σ is known, we may assume w.l.o.g. that Σ = I ; otherwise, we can transform each sample X into Σ -1 / 2 X . We start by using known algorithms [23, 40, 33] to find a 'rough' estimate for the mean. In particular, we find an estimate ˆ µ such that ∥ ˆ µ -µ ∥ 2 ≤ R = ˜ O ( √ d/ϵ ) using n = ˜ O ( √ d/ϵ ) samples. By appropriately shifting the subsequent samples, this is equivalent to assuming that ∥ µ ∥ 2 ≤ R . We then focus on designing a DP sampler for this bounded mean case. It turns out, surprisingly, that the Gaussian mechanism suffices here. Specifically, for a parameter B > 0 (chosen later), we truncate each sample so that its ℓ 2 -norm is at most B . We then output their average with a (spherical) Gaussian noise N (0 , σ 2 I ) added. The description is given in Algorithm 1.

The analysis of the Gaussian mechanism [e.g., 20, Appendix A] shows that the algorithm is ( ϵ, δ ) -DP as long as we pick σ ≥ ˜ O ( B nϵ ) .

As for the accuracy, observe that if there were no truncation, then the output is exactly distributed as N ( µ, ( σ 2 +1 /n ) I ) , which is precisely N ( µ, I ) if we set σ 2 = ( n -1) /n . Therefore, by setting B = R + O ( √ d +log(1 /α )) so that the truncation does not occur with probability 1 -α , we

ensure that the sampler is α -accurate. The constraint that σ ≥ ˜ O ( B nϵ ) from privacy implies that we need n ≥ ˜ O ( √ d/ϵ ) and hence yielding the sample complexity in Theorem 1.2.

Unknown Bounded Covariance. Recall that in this setting we know that I ⪯ Σ ⪯ κ · I . While it might be tempting to use the above Gaussian mechanism for this setting as well, it turns out that this approach results in sample complexity that depends polynomially on 1 /α . 3

To circumvent this, we first consider the case where µ = 0 (i.e., 'centered' Gaussians). In this case, our algorithm originates from the following attempt: output ∑ i ∈ [ n ] a [ i ] · X i , where ( a [1] , . . . , a [ n ]) ∼ Uni S n , the uniform distribution over points on the unit sphere in R n . It follows from the 2 -stability of the Gaussian distribution [41] that, when X 1 , . . . , X n ∼ N (0 , Σ) , this results 4 in an output that is distributed exactly as N (0 , Σ) .

Unfortunately, this algorithm is not DP: if X 1 = · · · = X n -1 = 0 , then the output will reveal the direction of X n in the clear. To remedy this, we build on the intuition that, if X 1 , . . . , X n 'sufficiently span all directions', then there should be 'enough noise' to make this algorithm DP. In particular, using the bounded covariance property, we can show that if all the eigenvalues of ∑ i ∈ [ n ] X i X T i

Algorithm 1 SPHERICALGAUSSIANSAMPLER

are sufficiently large (and each X i is truncated appropriately), then this algorithm is indeed 'DP'. This is perhaps the most technically challenging part of our work, as the noise is data-dependent and therefore poses significant hurdles in the privacy analysis (Section 4.2). Note that this is also the reason we need n ≥ Ω( d ) , as otherwise X 1 , . . . , X n cannot 'sufficiently span all directions' in R d .

With the above, the last ingredient is a testing step (in the 'propose-test-release' paradigm of [17]) that checks this eigenvalue condition. When this condition fails, we return ⊥ ; otherwise ∑ i ∈ [ n ] a [ i ] · X i .

̸

To handle the case where µ = 0 , we take an output to be the sum of the average of n 1 samples and √ 1 -1 n 1 · ( ∑ i ∈ [ n 2 ] a [ i ] · U i ) , where each U i is the difference between two fresh independent samples divided by √ 2 and ( a [1] , . . . , a [ n 2 ]) ∼ Uni S n 2 . Notice here that each U i ∼ N (0 , Σ) , while the average over n 1 samples is ∼ N ( µ, 1 n 1 · Σ) ; thus, the sum is ∼ N ( µ, Σ) as desired. The full description is presented in Algorithm 2; the parameter setting and analysis can be found in Appendix B.4.

Unknown Unbounded Covariance. We proceed by reducing this case to the previous setting. We do so by first applying the known DP 'preconditioner' algorithm for Gaussians (with unknown unbounded covariance) from the work of [9] in order to obtain rough estimates ˆ µ, ˆ Σ of µ, Σ respectively. This allows us to transform any subsequent sample X into ˆ Σ -1 / 2 ( X -ˆ µ ) . This reduces us back to DP sampling for N ( ˆ Σ -1 / 2 ( µ -ˆ µ ) , ˆ Σ 1 / 2 Σ -1 ˆ Σ 1 / 2 ) . The guarantee of the DP preconditioner ensures that this Gaussian actually has bounded

Algorithm 2 BOUNDEDCOVGAUSSIANSAMPLER

covariance. Therefore, we can apply our previous algorithm. Note that a significant part of the sample complexity is due to the DP preconditioner of the Gaussian; it turns out that the sample complexity of this task is only ˜ O ϵ,α ( d 3 / 2 ) (compared to ˜ Θ ϵ,α ( d 2 ) for learning). Furthermore, since we only need the preconditioner to be a rough estimate, we can set the accuracy parameter for the learning to be Θ(1) and thus avoid the polynomial dependence on 1 /α . (Note that private preconditioner is a standard ingredient in the recipe for DP learning [e.g., 27].)

2.2 Gaussian Distributions: Lower Bounds

Known Covariance. Our lower bound in this setting builds on the following insight: if we take a constant number of samples from N ( µ, I ) (using the DP sampler) and use them to estimate µ , then we incur an expected ℓ 2 2 -error that is O ( d ) . It turns out that known lower bounds for DP mean estimation of Gaussians with known covariance [27] hold for this setting and give a lower bound of

Ω ( d γϵ ) , where γ 2 is the ℓ 2 2 -error. Plugging in γ = √ d in our setting gives the desired lower bound of Ω( √ d/ϵ ) . In the actual proof, one complication stems from the fact that our DP sampler does not output a sample exactly from N ( µ, I ) . Nonetheless, we can quantify this in terms of the accuracy α .

Unknown Bounded Covariance. In this setting, we reduce from a lower bound on DP covariance estimation [28]-rather than DP mean estimation earlier-of centered Gaussians. The challenge is that Θ(1) samples from N (0 , Σ) do not provide a sufficiently high accuracy estimate for Σ so that we can apply the known lower bound 5 . Therefore, the above approach does not work directly.

To overcome this, we will have to use many samples to estimate the covariance. Recall that the sample complexity lower bound of Ω( d 2 /ϵ ) for covariance estimation requires the accuracy in the Frobenius distance (or the Mahalanobis distance) to be constant [28]. Due to this accuracy requirement, we need to use our DP sampler to generate L = Ω( d 2 ) samples Y 1 , . . . , Y L to achieve such an accuracy. We can draw a fresh batch X i 1 , . . . , X i n of samples from the underlying distribution to generate each

Y i . However, since L = Ω( d 2 ) , this would use Ln samples in total. The covariance estimation lower bound would then yield Ln ≥ Ω( d 2 /ϵ ) , implying n ≥ Ω(1 /ϵ ) -even weaker than the mean estimation lower bound!

Fortunately, it turns out this can be overcome by using advanced composition of DP [19]. In particular, we may run our sampler L times on the same n samples to produce Y 1 , . . . , Y L . In this case, the final covariance estimation algorithm has privacy loss parameter ˜ O ( √ L · ϵ ) = ˜ O ( d · ϵ ) due to advanced composition (Theorem A.5). Therefore, we get a lower bound of ˜ Ω( d 2 / ( d · ϵ )) = ˜ Ω( d/ϵ ) as desired.

While the above overview seems intuitively plausible, there are certain difficulties that we need to overcome. First, the samples Y i that our algorithm produces are not exactly drawn from N (0 , Σ) ; to fix this, we run an agnostic learner for Gaussians [e.g., 3] to recover the estimate of Σ . Second, since we run our DP sampler on the same n samples, the produced Y 1 , . . . , Y L are not independent. We fix this by first drawing a larger number N of samples, and then produce each Y i using n -out-ofN random samples; this reduces the correlation across the Y i 's. Furthermore, using amplification-by-sampling of DP [6] gives us the desired privacy-vs-sample complexity lower bound guarantee.

2.3 Product Distributions: Algorithm

Our sampler follows the framework of [35], which is built upon the preconditioning procedure proposed by [27] for DP learning.

We start with a private preconditioner, which obtains a crude estimate of each p j (up to a constant multiplicative factor). This is similar to that of [27], except that we use Laplace noise instead of Gaussian noise; this ensures that the resulting algorithm is pure-DP 6 . By suitably partitioning [0 , 1] into geometrically decreasing buckets in terms of d/α , the goal then is to estimate p j by placing it in one of these buckets. This can be done by an appropriate thresholding and the Laplace mechanism.

The next step is to obtain a refined estimate of the p i 's using a fresh batch of samples. The algorithm then returns a sample randomly drawn from the product distribution given by these estimates. The earlier crude estimates are helpful in truncating and clipping the samples to make the produced sample DP, without adding any noise to the refined estimate.

3 Preliminaries

For convenience, we use the notation trunc p B ( X ) for 'truncation' for all X ∈ R d , B > 0 , p ≥ 1 :

Let [ k ] denote { 1 , . . . , k } . For any X ∈ R d , we use X [ j ] to denote the value of its j th coordinate.

3.1 Distributions and Tail Bounds

For a discrete distribution D , we use f D to denote its probability mass function (PMF); for a continuous distribution D , we use f D to denote its probability density function (PDF). Let supp( D ) denote the support of D . We let Z ∼ D denote that the random variable Z is distributed according to D ; throughout, we may write the random variable in place of the distribution and vice versa when convenient. Finally, when D 1 , . . . , D d are distributions, we use D 1 ⊗··· ⊗ D d as the product distribution, i.e., the distribution of ( Z 1 , . . . , Z d ) where Z 1 ∼ D 1 , . . . , Z d ∼ D d are independent.

For a distribution D on R d and v ∈ R d , we write D + v as a shorthand for the distribution of X + v where X ∼ D . Furthermore, for a distribution D and a (possibly randomized) function h , we write h ( D ) to denote the distribution of h ( X ) where X ∼ D .

We will list a few distributions that will be useful for us.

(i) Shifted Truncated Discrete Laplace Distribution: For ∆ > 0 , let s ( ϵ, δ ) = ⌈ ∆(1 + log(1 /δ ) /ϵ ) ⌉ . We define STLap ( ϵ, δ, ∆) to be the discrete distribution supported on [ -2 s ( ϵ, δ ) , 0] such that f STLap ( ϵ,δ, ∆) ( x ) ∝ exp( -ϵ | x + s ( ϵ, δ ) | ) .

It is known that adding STLap noise to a low-sensitivity function results in a DP estimate [e.g., 22]. Lemma 3.1. If g is a function with sensitivity ≤ ∆ , then the algorithm that outputs g ( X ) + STLap ( ϵ, δ, ∆) is ( ϵ, δ ) -DP.

(iii) Uniform Distribution over Unit Sphere: For d ∈ N , let Uni S d denote the distribution of a random unit vector in R d .

(iv) Projection of Uniform Distribution over Unit Sphere: For any d ∈ N and i ∈ [ d ] and z ∈ R d , let Π ≤ i ( z ) denote ( z 1 , . . . , z i ) . Then, let Uni S d,i denote the distribution of Π ≤ i ( Z ) where Z ∼ Uni S d . This distribution has the PDF (see, e.g., [34, Theorem 2]) given below.

We need a tail bound on the ℓ 2 -norm of a Gaussian-distributed vector:

Lemma 3.2 ([42, Theorem 6.2]) . There exists a constant c ≥ 1 such that, for any µ ∈ R d , Σ ∈ R d × d where Σ ⪯ κ · I and any β ∈ (0 , 1 2 ) , Pr X ∼ N ( µ, Σ) [ ∥ X -µ ∥ 2 > c √ κ ( √ d + √ log(1 /β ) )] ≤ β.

We also need a tail bound for the beta distribution:

Lemma 3.3 ([43], Theorem 8) . There exists a constant c ∈ (0 , 1) such that, for any 0 < α < β and any x ≥ 0 , we have Pr Z ∼ Beta ( α,β ) [ Z ≥ α α + β + x ] ≤ 2 e -c · min { β 2 x 2 α ,βx } .

The following concentration bound on the empirical covariance will also be helpful.

3.2 Differential Privacy

Hockey Stick Divergence. We also recall the definition of ϵ -hockey stick divergence between two distributions P, Q : d ϵ ( P || Q ) := ∫ y ∈ supp( P ) [ f P ( y ) -e ϵ f Q ( y )] + dy, where [ a ] + := max { a, 0 } .

The following standard fact about the hockey stick divergence is often useful in proving DP guarantees of algorithms [39, 31].

Lemma 3.5. For any ϵ ≥ 0 and distributions P, Q , d ϵ ( P || Q ) ≤ Pr y ∼ P [ f P ( y ) > e ϵ f Q ( y )] .

It will be also useful to keep in mind the 'post-processing' property of DP:

Lemma 3.6. For any distributions P, Q and any function h , we have d ϵ ( h ( P ) || h ( Q )) ≤ d ϵ ( P || Q ) .

DP under Condition. Since we will use a 'propose-test-release'-style algorithm [17], it will be convenient to use the notion of 'DP under condition' together with its composition properties. The particular definition we use below is from [32]; similar notions have been used earlier, e.g., in [17].

Definition 3.7 (DP under Condition, [32]) . Let Ψ : Y → { 0 , 1 } be a predicate. An algorithm M : Y → O is ( ϵ, δ ) -DP under condition Ψ for ϵ, δ > 0 iff, for every S ⊆ O and every neighboring datasets Y, Y ′ ∈ Y both satisfying Ψ , we have Pr[ M ( Y ) ∈ S ] ≤ e ϵ · Pr[ M ( Y ′ ) ∈ S ] + δ.

Lemma 3.8 (Composition for Algorithm with Halting, [32]) . Let M 1 : Y → O 1 ∪ {⊥} , M 2 : O 1 ×Y → O 2 be algorithms. Furthermore, let M denote the following algorithm: Let o 1 = M 1 ( Y ) and, if o 1 = ⊥ , then halt and output ⊥ or else, output o 2 = M 2 ( o 1 , Y ) .

Let Ψ be any condition such that, if Y does not satisfy Ψ , then M 1 ( Y ) always returns ⊥ . Suppose that M 1 is ( ϵ 1 , δ 1 ) -DP and M 2 is ( ϵ 2 , δ 2 ) -DP under condition Ψ . Then, M is ( ϵ 1 + ϵ 2 , δ 1 + δ 2 ) -DP.

4 Gaussian Distribution: Algorithms

Reduction to the Bounded Mean Case. As stated earlier, for the cases of known covariance and bounded covariance, we will need a preprocessing step that computes a rough private estimate for the mean. The properties of the reduction are stated below.

Lemma 4.1. Suppose that there is an α -accurate ( ϵ, δ ) -DP sampler for Gaussian distributions under the assumption that I ⪯ Σ ⪯ κ · I, ∥ µ ∥ ≤ R with sample complexity n bm -sampler ( α, R, ϵ, δ )

. Then, there exists an α -accurate ( ϵ, δ ) -DP sampler for Gaussian distributions under the assumption that I ⪯ Σ ⪯ κ · I with sample complexity ˜ O ( √ d ) + n bm -sampler ( α/ 2 , O ( κ √ d ) , ϵ/ 2 , δ/ 2) .

4.1 Known Covariance

With the reduction in Lemma 4.1, we may assume that ∥ Σ -1 / 2 µ ∥ ≤ R . When the covariance is known, we give an algorithm with the following guarantees.

We assume w.l.o.g. that Σ = I ; otherwise, we can consider X ′ i = Σ -1 / 2 X i . Before we describe the algorithm, note that Theorem 4.2 together with Lemma 4.1 implies Theorem 1.2. As stated earlier, the algorithm (Algorithm 1) is simple: take the average of the truncated input samples and add to it (spherical) Gaussian noise.

Privacy Analysis. A is the Gaussian mechanism with noise multiplier nσ/B ≥ 10 √ log(2 /δ ) /ϵ ; therefore, A is ( ϵ, δ ) -DP, using [20, Appendix A].

Accuracy Analysis. Let D = N ( µ, I ) for some unknown µ . Consider the algorithm A ′ where there is no truncation, i.e., A ′ simply outputs Y := Z + 1 n ∑ i ∈ [ n ] X i . Via Lemma 3.2 and a union bound, the truncation is not applied anyway in A (i.e., X i = X trunc i , ∀ i ∈ [ n ] ) with probability at least 1 -α . Therefore, d tv ( Q A ,D , Q A ′ ,D ) ≤ α . Note that A ′ just outputs Y := Z + 1 n ∑ i ∈ [ n ] X i , so we have Y ∼ N ( µ, I ) = D , i.e., Q A ′ ,D = D . Combining these bounds yields d tv ( Q A ,D , D ) ≤ α .

4.2 Unknown Bounded Covariance

We now move on to the case where both µ, Σ are unknown but under the assumption I ⪯ Σ ⪯ κ · I . The main result of this section is stated below 7 . Again, note that Theorem 4.3 and Lemma 4.1 immediately yield Theorem 1.4.

Theorem 4.3. Assuming I ⪯ Σ ⪯ κ · I for some κ > 0 and ∥ µ ∥ ≤ R , there is an α -accurate ( ϵ, δ ) -DP sampler for Gaussian distributions with sample complexity O (( R 2 + κ 2 ( d +log ( log(1 /δ ) αϵ ))) · log(1 /δ ) ϵ ) .

As stated in the overview, the main challenge lies in the privacy analysis. It will be proved in three steps. First, in Section 4.2.1, we will show that if we add noise that is drawn from projection of random unit vector to the first M coordinates to a low-( ℓ 2 -)sensitivity function, then it is DP. Then, in Section 4.2.2, we use this to show that adding noise of the form ∑ i ∈ [ n ] a [ i ] · w i (where a ∼ Uni S n ) to a low-sensitivity function also suffices for privacy as long as the smallest eigenvalue of ∑ i ∈ [ n ] w i w T i is sufficiently large. Here w 1 , . . . , w n are assumed to be fixed vectors given beforehand. Note that the noises discussed so far are input independent . Finally, in Appendix B.4, we relate this to the

privacy of our algorithm-which is more intricate as these w i 's are now X i 's, i.e., the noise is input dependent. The accuracy analysis, which is simpler than the privacy analysis, is also in Appendix B.4.

4.2.1 Noise via Random Unit Vector Projection

We first show that adding noise drawn from Uni S M,N to a function with low ℓ 2 -sensitivity ensures DP.

Lemma 4.4. Let M,N be any positive integers such that M ≥ max { 2 N, 10 4 log(10 /δ ) cϵ } , where c is the constant from Lemma 3.3. Furthermore, let τ ∈ (0 , 0 . 01) be such that τ ≤ 10 √ log(10 /δ ) cM . Then, for any vector v such that ∥ v ∥ ≤ τ , we have d ϵ ( Uni S M,N || Uni S M,N + v ) ≤ δ .

The proof of this lemma proceeds similarly to those of independent noise addition (e.g., the Gaussian mechanism): using Lemma 3.5, it suffices to bound the probability that the privacy loss is large. We achieve such a bound via concentration properties of the beta distribution.

4.2.2 Noise via Spherical Linear Combinations

Next, we relate the previous lemma to privacy in the case where the noise added is of the form ∑ i ∈ [ q ] a [ i ] · w i , where w 1 , . . . , w q are fixed beforehand and a ∼ Uni S q . The guarantee of such a noise is stated below. The proof is based on a reduction argument to Lemma 4.4. For notational convenience, for every matrix W ∈ R d × q , let Uni S W be the distribution of Wa where a ∼ Uni S q .

Lemma 4.5. Let q, d ∈ N and τ 0 ∈ (0 , 1) be such that q ≥ max { 2 d, 10 4 log(10 /δ ) cϵ } and τ 0 ≤ 5 √ log(10 /δ ) /c . For any w 1 , . . . , w q ∈ R d such that 1 q (∑ q i =1 w i w T i ) ⪰ 0 . 5 I and any v ∈ R d such that ∥ v ∥ ≤ τ 0 , we have d ϵ ( Uni S W || Uni S W + v ) ≤ δ, where w 1 , . . . , w q are the columns of W ∈ R d × q .

Proof. Let W + ∈ R d × q denote the pseudoinverse of W . Note that WW T = ∑ q i =1 w i w T i ⪰ 0 . 5 qI . Therefore, all singular values of W are at least √ 0 . 5 q . This means that all singular values of W + are at most 1 / √ 0 . 5 q . This in turn implies that ∥ W + v ∥ ≤ ∥ v ∥ / √ 0 . 5 q ≤ τ 0 / √ 0 . 5 q =: τ .

Since W T W ⪰ 0 . 5 qI , W T is full rank and thus dim(im( W + )) = d . Since d < q , pick any orthonormal basis b 1 , . . . , b d ∈ R q for im( W + ) and let Π : R q → R d be such that Π( z ) = ( ⟨ z, b 1 ⟩ , . . . , ⟨ z, b d ⟩ ) .

Furthermore, let h : R d → R d be defined as h ( x ) := ∑ i ∈ [ d ] x [ i ] · b i . Observe that, for any a ∈ R q , we have a 1 u 1 + · · · + a q u q = h (Π( a )) and a 1 u 1 + · · · + a q u q + v = h (Π( a ) + Π( W + v )) . Thus,

Note that Π( Uni S q ) ∼ Uni S q,d . So, Theorem 4.4 implies d ϵ (Π( Uni S q ) || Π( Uni S q ) + Π( W + v )) ≤ δ .

5 Discussion and Open Questions

In this work, we give several algorithms for DP sampling. For Gaussian distributions, we demonstrate that the dependence on the accuracy parameter α in the sample complexity can be only polylogarithmic in comparison to the polynomial dependence necessary in DP learning. Furthermore, we also reduce the dependence on d in the case of known variance and unknown bounded variance; in both cases, we show this dependence (of √ d and d respectively) to be essentially tight. An immediate open question is whether the dependence of d 1 . 5 in the case of unknown unbounded variance is tight. Lower bounds for Gaussian sampling in terms of α is also an interesting question. Another-more general-direction is to extend the results to other families of distributions. In the case of DP learning, this has been explored by several works. Our algorithms very specifically use the property of Gaussian distributions (namely that a linear combination of Gaussians is another Gaussian); other distribution families might need different techniques.

Acknowledgment. We thank Shyam Narayanan for pointing us to [9], which improves dependency on d in the sample complexity for sampling Gaussians with unknown (unbounded) covariance from d 2 to d 1 . 5 .

References

Mart´ ın Abadi, Andy Chu, Ian J. Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In CCS , pages 308-318, 2016.

Ishaq Aden-Ali, Hassan Ashtiani, and Gautam Kamath. On the sample complexity of privately learning unbounded high-dimensional Gaussians. In ALT , pages 185-216, 2021.

Hassan Ashtiani, Shai Ben-David, and Abbas Mehrabian. Sample-efficient learning of mixtures. In AAAI , pages 2679-2686, 2018.

Hassan Ashtiani and Christopher Liaw. Private and polynomial time algorithms for learning Gaussians and beyond. In COLT , pages 1075-1076, 2022.

Brian Axelrod, Shivam Garg, Vatsal Sharan, and Gregory Valiant. Sample amplification: Increasing dataset size even when learning is impossible. In ICML , pages 442-451, 2020.

Borja Balle, Gilles Barthe, and Marco Gaboardi. Privacy amplification by subsampling: Tight analyses via couplings and divergences. In NeurIPS , pages 6280-6290, 2018.

Sourav Biswas, Yihe Dong, Gautam Kamath, and Jonathan R. Ullman. Coinpress: Practical private mean and covariance estimation. In NeurIPS , 2020.

Gavin Brown, Marco Gaboardi, Adam D. Smith, Jonathan R. Ullman, and Lydia Zakynthinou. Covariance-aware private mean estimation without private covariance estimation. In NeurIPS , pages 7950-7964, 2021.

Gavin Brown, Samuel Hopkins, and Adam Smith. Fast, sample-efficient, affine-invariant private mean and covariance estimation for subgaussian distributions. In COLT , pages 5578-5579, 2023.

Mark Bun, Gautam Kamath, Thomas Steinke, and Zhiwei Steven Wu. Private hypothesis selection. IEEE Trans. Inf. Theory , 67(3):1981-2000, 2021.

T. Tony Cai, Yichen Wang, and Linjun Zhang. The cost of privacy: Optimal rates of convergence for parameter estimation with differential privacy. Ann. Stat. , 49(5):2825-2850, 2021.

Damien Desfontaines, James Voss, Bryant Gipson, and Chinmoy Mandayam. Differentially private partition selection. PETS , 2022(1):339-352, 2022.

Luc Devroye, Abbas Mehrabian, and Tommy Reddad. The total variation distance between high-dimensional Gaussians with the same mean. arXiv:1810.08693 , 2018.

Ilias Diakonikolas, Gautam Kamath, Daniel Kane, Jerry Li, Ankur Moitra, and Alistair Stewart. Robust estimators in high-dimensions without the computational intractability. SIAM J. Comput. , 48(2):742-864, 2019.

Wei Dong, Yuting Liang, and Ke Yi. Differentially private covariance revisited. In NeurIPS , 2022.

Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. In EUROCRYPT , pages 486-503, 2006.

Cynthia Dwork and Jing Lei. Differential privacy and robust statistics. In STOC , pages 371-380, 2009.

Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In TCC , pages 265-284, 2006.

Cynthia Dwork, Moni Naor, and Salil P. Vadhan. The privacy of the analyst and the power of the state. In FOCS , pages 400-409, 2012.

Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci. , 9(3-4):211-407, 2014.

Cynthia Dwork, Kunal Talwar, Abhradeep Thakurta, and Li Zhang. Analyze gauss: Optimal bounds for privacy-preserving principal component analysis. In STOC , pages 11-20, 2014.

Badih Ghazi, Neel Kamal, Ravi Kumar, Pasin Manurangsi, and Annika Zhang. Private aggregation of trajectories. PETS , 2022(4):626-644, 2022.

Badih Ghazi, Ravi Kumar, and Pasin Manurangsi. Differentially private clustering: Tight approximation ratios. In NeurIPS , 2020.

Paul Hamill. Unit Test Frameworks: Tools for High-Quality Software Development . ' O'Reilly Media, Inc.', 2004.

Samuel B. Hopkins, Gautam Kamath, and Mahbod Majid. Efficient mean estimation with pure differential privacy via a sum-of-squares exponential mechanism. In STOC , pages 1406-1417, 2022.

Samuel B. Hopkins, Gautam Kamath, Mahbod Majid, and Shyam Narayanan. Robustness implies privacy in statistical estimation. In STOC , pages 497-506, 2023.

Gautam Kamath, Jerry Li, Vikrant Singhal, and Jonathan R. Ullman. Privately learning highdimensional distributions. In COLT , pages 1853-1902, 2019.

Gautam Kamath, Argyris Mouzakis, and Vikrant Singhal. New lower bounds for private estimation and a generalized fingerprinting lemma. In NeurIPS , 2022.

Gautam Kamath, Argyris Mouzakis, Vikrant Singhal, Thomas Steinke, and Jonathan R. Ullman. A private and computationally-efficient estimator for unbounded Gaussians. In COLT , pages 544-572, 2022.

Vishesh Karwa and Salil P. Vadhan. Finite sample differentially private confidence intervals. In ITCS , pages 44:1-44:9, 2018.

Antti Koskela, Joonas J¨ alk¨ o, and Antti Honkela. Computing tight differential privacy guarantees using FFT. In AISTATS , pages 2560-2569, 2020.

Pravesh Kothari, Pasin Manurangsi, and Ameya Velingker. Private robust estimation by stabilizing convex relaxations. In COLT , pages 723-777, 2022.

Shyam Narayanan, Vahab S. Mirrokni, and Hossein Esfandiari. Tight and robust private mean estimation with few users. In ICML , pages 16383-16412, 2022.

Hedvika Ranosova. Spherically Symmetric Measures. Bachelor's thesis, Department of Probability and Mathematical Statistics, Charles University, 2021.

Sofya Raskhodnikova, Satchit Sivakumar, Adam Smith, and Marika Swanberg. Differentially private sampling from distributions. In NeurIPS , pages 28983-28994, 2021.

Or Sheffet. Differentially private ordinary least squares. In ICML , pages 3105-3114, 2017.

Vikrant Singhal. A polynomial time, pure differentially private estimator for binary product distributions. CoRR , abs/2304.06787, 2023.

Adam Smith. Privacy-preserving statistical estimation with optimal convergence rates. In STOC , pages 813-822, 2011.

David M. Sommer, Sebastian Meiser, and Esfandiar Mohammadi. Privacy loss classes: The central limit theorem in differential privacy. PETS , 2019(2):245-269, 2019.

Eliad Tsfadia, Edith Cohen, Haim Kaplan, Yishay Mansour, and Uri Stemmer. Friendlycore: Practical differentially private aggregation. In ICML , pages 21828-21863, 2022.

Vladimir V. Uchaikin and Vladimir M. Zolotarev. Chance and Stability . De Gruyter, Berlin, Boston, 1999.

Roman Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science . Cambridge University Press, 2018.

Anru R. Zhang and Yuchen Zhou. On the non-asymptotic and sharp lower tail bounds of random variables. Stat , 9(1):e314, 2020.

A Additional Preliminaries

We consider a generalization of the truncation (1) where there is a weight vector w ∈ R d ≥ 0 , and the truncation is w.r.t. w :

where ◦ denotes the element-wise product. Clearly trunc p B = trunc p B,w when w is the all-ones vector.

For a, b ∈ R where a ≤ b , we also use clip ( a,b ) : R → R to denote the function:

A.1 Total Variation Distance between Gaussians

For our proofs, it will be useful to state the total variation distance bounds for Gaussian distributions. We will use the following notation:

Definition A.1. For any matrix A ∈ R d × d and positive semi-definite matrix Σ ∈ R d × d , we write ∥ A ∥ Σ := ∥ Σ -1 / 2 A Σ -1 / 2 ∥ F , where ∥ · ∥ F denotes the Frobenius norm.

It is well known that when ∥ Σ -ˆ Σ ∥ Σ ≤ O (1) , the total variation distance between Gaussian with covariance matrices Σ and ˆ Σ and the same mean is Θ( ∥ Σ -ˆ Σ ∥ Σ ) , as stated more formally below.

Lemma A.2 ([13]) . There exists a constant C ≥ 1 such that the following holds. For any Σ , ˆ Σ , if d tv ( N (0 , Σ) , N (0 , ˆ Σ)) ≤ 0 . 001 , then ∥ Σ -ˆ Σ ∥ Σ ≤ C · d tv ( N (0 , Σ) , N (0 , ˆ Σ)) .

Lemma A.3 (e.g., [14]) . There exists a constant C > 0 such that the following holds. For any Σ , ˆ Σ ∈ R d × d , we have d tv ( N (0 , Σ) , N (0 , ˆ Σ)) ≤ C · ∥ Σ -ˆ Σ ∥ Σ .

A.2 Differential Privacy

A.2.1 Amplification by Subsampling

Suppose that we have as an input N samples, and we draw n < N subsamples (without replacement) randomly from these N samples and run an ( ϵ, δ ) -DP algorithm on these n subsamples. Then, it is known that this results in an ( ϵ ′ , δ ′ ) -DP algorithm for ϵ ′ < ϵ, δ ′ < δ . Such a phenomenon is called amplification by subsampling and is often used in DP learning [e.g., 1]. We will use the following version of this for our lower bound proof.

Theorem A.4 (Amplification by Subsampling, [6]) . Suppose that A is an ( ϵ, δ ) -DP algorithm that takes in n samples. Then, the algorithm that draws n subsamples randomly without replacement out of N samples and runs A on the n subsamples is ( ϵ ′ , δ ′ ) -DP where

A.3 Advanced Composition

We will also use the following advanced composition of DP.

Theorem A.5 (Advanced Composition, [19]) . Let ϵ 0 > 0 and δ 0 , δ 1 ∈ (0 , 1 / 2] , and A be any ( ϵ 0 , δ 0 ) -DP algorithm. Then, the algorithm that runs A a total of k times is ( ϵ 1 , kδ 0 + δ 1 ) -DP where

A.4 Lower Bounds for Mean and Covariance Estimations

As stated in our proof overview (Section 2), our lower bounds for Gaussian samplers (Theorems 1.3 and 1.5) are via reductions from previously known lower bounds for mean and covariance estimations. We state them below, starting with the mean estimation lower bound (aka 'fingerprinting for Gaussian') result due to [27]

Theorem A.6 ([27]) . For any ϵ, δ ∈ (0 , 1] such that δ ≤ O ( √ d n √ log( n/d ) ) , if an ( ϵ, δ ) -DP algorithm 8

M that can take in N samples from D = N ( µ, I ) where -1 ≤ µ ≤ 1 and output an estimate ˆ µ ∈ R d that satisfies

Next is a similar lower bound but for covariance estimation. Notice that the lower bound on the sample complexity below is Ω( d 2 /ϵ 2 ) whereas the lower bound above is only Ω( d/ϵ ) (for constant accuracy parameter γ ).

Theorem A.7 ([28]) . There exists a small constant ξ > 0 such that the following holds: for any ϵ, δ ∈ (0 , 1] such that δ ≤ O ( min { 1 /n, d 2 / ( n log n ) }) , if an ( ϵ, δ ) -DP algorithm M that can take in N samples from D = N (0 , Σ) where I ⪯ Σ ⪯ 2 I and output an estimate ˆ Σ ∈ R d × d that satisfies

then N ≥ Ω ( d 2 /ϵ ) .

A.5 Agnostic Learner

In the main body of the paper, we considered the learner where the samples are drawn from D which belongs to a certain class D . For our lower bound proofs, it will be convenient to consider the agnostic setting where D is not assumed to be from D . The definition and accuracy guarantee of agnostic learner is given below. We remark that the definition coincides with the learner definition given earlier if we assume that D ∈ D . (Note that the constant 3 is required for Theorem A.8.)

Agnostic Learning. Let D be a class of distribution. An algorithm A , which takes in n samples and output a distribution ˆ P ∈ D , is said to be an ( α, β ) -accurate agnostic learner for a class D of distributions iff, for any distribution P , we have

We will use the following (well-known) result that the class of centered 9 Gaussian distributions can be learned with sample complexity O ( d 2 /γ 2 ) . For a full proof, see e.g., [3].

Theorem A.8. For any γ, β ∈ (0 , 1 / 2] , there exists a ( γ, β ) -accurate agnostic learner for centered Gaussian distributions in d dimensions with sample complexity

B Proofs for Section 4

B.1 Reduction to the Bounded Mean Case

The reduction will be done by applying a so-called 'densest ball' algorithm; this is an algorithm that, when there is a ball of radius r that contains the majority of the input, can find a ball of radius O ( r ) containing the majority of the input. There are many algorithms known for this problem [23, 40, 33] that give similar guarantees 10 . We state such a guarantee below.

Theorem B.1 (e.g., [23]) . For n dens ( d, β, ϵ, δ ) = ˜ Θ ( √ d ϵ ) , there exists an ( ϵ, δ ) -DP algorithm A dens that takes in X 1 , . . . , X n ∈ R d together with a radius parameter r > 0 such that: if there is an r -radius ball containing at least 2 / 3 fraction of the input points, then with probability 1 -β , it outputs a ball of radius O ( r ) that contains at least half of the input points.

We can now give a reduction of an arbitrary mean case to the bounded mean case by first using Theorem B.1 to obtain a rough estimate of the mean and then shifting the remaining samples accordingly.

Proof. The new sampler works as follows:

Let n 1 := max { n dens ( d, α/ 4 , ϵ/ 2 , δ/ 2) , 100 log(4 /α ) } , r = 10 Cκ √ d where C is the constant from Lemma 3.2, and n 2 := n bm -sampler ( α/ 2 , r, ϵ/ 2 , δ/ 2) .

Rough Mean Estimator Stage:

-Run ( ϵ/ 2 , δ/ 2) -DP A dens from Theorem B.1 on n 1 samples (with r as specified above) to get a ball centered at c ∈ R d .

Sampling Stage:

-Draw n 2 additional samples X 1 , . . . , X n 2 .

-Run ( ϵ/ 2 , δ/ 2) -DP ( α/ 2) -accurate A bm -sampler with R = r on ( X 1 -c ) , . . . , ( X n 2 -c ) ; let Y be its output.

-Output Y + c .

Privacy Analysis. Basic composition implies that the sampler is ( ϵ, δ ) -DP as desired.

Accuracy Analysis. Since Σ ⪯ κI , Lemma 3.2 implies that Pr X ∼ N ( µ, Σ) [ ∥ X -µ ∥ 2 ≥ r ] ≤ 0 . 1 . Thus, using standard concentration inequality (e.g., Theorem E.1) and since n 1 ≥ 100 log(4 /α ) , the probability that at least 2 / 3 of the n 1 points sampled in the first stage contains in the ball B ( µ, r ) is at least 1 -α/ 4 . The guarantee of the densest ball algorithm then implies that with probability 1 -α/ 2 , we have ∥ µ -c ∥ 2 ≤ r .

Let us consider a fixed c and consider the output distribution D c Y of Y in the sampling stage and D c Y + c of the final output Y + c . Notice that ( X 1 -c ) , . . . , ( X n 2 -c ) ∼ N ( µ -c, Σ) . Consequently, when ∥ µ -c ∥ 2 ≤ r , we may apply the guarantee of A bm -sampler to yield

Combining these arguments, the sampler is α -accurate as desired.

B.2 Unknown Bounded Covariance

Proof of Lemma 4.4. From Lemma 3.5, we have

For convenience, let us also define η = τ · ( 10 √ log(10 /δ ) cM ) . Note that η ≤ 0 . 01 .

For z such that ∥ z ∥ ≤ 0 . 9 , | ⟨ z, v ⟩ | ≤ η , we also have ∥ z -v ∥ 2 ≤ 0 . 9 2 +0 . 02 + 0 . 0001 < 0 . 9 and thus

where the last inequality follows from our choice of parameters (i.e., M ≥ 10 4 log(10 /δ ) cϵ ). Combining the previous two bounds, we have

Next, recall that when z ∼ Uni S M,N , ∥ z ∥ 2 ∼ Beta ( N/ 2 , ( M -N ) / 2) . 11 Thus, applying Lemma 3.3 and using the assumption that N ≤ M/ 2 , we can conclude that

where the first inequality is from Lemma 3.3 and the last inequality follows from our assumption that M ≥ 100 c log(10 /δ ) .

Recall also that when z ∼ Uni S M,N , ∣ ∣ ∣ 〈 z, v ∥ v ∥ 〉∣ ∣ ∣ 2 ∼ Beta (1 / 2 , ( M -1) / 2) . Again, applying Lemma 3.3 yields

where the last inequality again follows from our choice of parameters.

Combining the three preceding inequalities, we can conclude that d ϵ ( Uni S M,N || Uni S M,N + v ) ≤ δ as desired.

B.3 Unknown Covariance

Throughout this section, we assume the standard assumption that the covariance matrix Σ is full rank. It is simple to extend the result to the more general case using an algorithm of [4], which can identify the span of Σ with probability one, using O ( d log(1 /δ ) /ϵ ) samples.

As stated earlier, we will use DP preconditioner from [9]. We say that an algorithm is an ( α, β ) -accurate preconditioner for Gaussian distributions iff, when the input X is drawn i.i.d. from N ( µ, Σ) (for any µ ∈ R d , Σ ∈ R d × d ), the algorithm outputs ˆ µ, ˆ Σ that satisfies the following with probability at least 1 -β : ∥ ˆ Σ -1 (ˆ µ -µ ) ∥ 2 ≤ α and 12 ∥ Σ -1 / 2 ˆ ΣΣ -1 / 2 -I ∥ 2 ≤ α . Below we provide a generic reduction from the unbounded covariance case to the bounded covariance case using preconditioner.

Lemma B.2. Suppose that the following hold:

There exists an ( α, β ) -accurate ( ϵ, δ ) -DP preconditioner A precnd for Gaussian distributions with sample complexity n precnd ( α, β, ϵ, δ ) .

There exists an α -accurate ( ϵ, δ ) -DP sampler A bc -sampler for Gaussian distributions with I ⪯ Σ ⪯ κ · I with sample complexity n bc -sampler ( κ, α, ϵ, δ ) .

Then, there exists an α -accurate ( ϵ, δ ) -DP sampler for Gaussian (without any assumption) with sample complexity

Proof of Lemma B.2. The sampler is as follows:

Let n 1 := n precnd (0 . 001 , α/ 2 , ϵ/ 2 , δ/ 2) and n 2 := n bc -sampler (4 , α/ 2 , ϵ/ 2 , δ/ 2) .

Estimation Stage:

-Run ( ϵ/ 2 , δ/ 2) -DP (0 . 001 , α/ 2) -accurate A precnd on n 1 samples to get ˆ µ, ˆ Σ .

Sampling Stage:

-Draw n 2 additional samples X 1 , . . . , X n 2 .

-Run ( ϵ/ 2 , δ/ 2) -DP ( α/ 2) -accurate A bc -sampler on 2 ˆ Σ -1 / 2 ( X 1 -ˆ µ ) , . . . , 2 ˆ Σ -1 / 2 ( X n 2 -ˆ µ ) ; let Y be its output.

-Output 0 . 5 ˆ Σ 1 / 2 Y + ˆ µ .

Basic composition implies that the sampler is ( ϵ, δ ) -DP.

To see the accuracy guarantee, the guarantee of the learner implies that with probability 1 -α/ 2 , we have ∥ ˆ Σ -1 (ˆ µ -µ ) ∥ 2 ≤ 0 . 001 and ∥ Σ -1 / 2 ˆ ΣΣ -1 / 2 -I ∥ 2 ≤ 0 . 001 . The latter implies that all eigenvalues of ˆ Σ 1 / 2 Σ -1 / 2 lie in [0.998, 1.002]; in turn, this implies that the eigenvalues of ˆ Σ -1 / 2 Σ 1 / 2 all lie in [0 . 996 , 1 . 004] . Finally, this implies that ∥ Σ 1 / 2 ˆ Σ -1 Σ 1 / 2 -I ∥ 2 ≤ 0 . 01 .

Let us consider a fixed ˆ Σ , ˆ µ and consider the output distribution of the sampling stage. When the above inequalities hold, the guarantee of A bc -sampler implies that

Combining the above arguments, the sampler is α -accurate as desired.

Combining the above reduction (Lemma B.2) together with a known result on DP preconditioner of Gaussians (Theorem B.3 below), we immediately arrive at Theorem 1.6.

Theorem B.3 ([8]) . There is an ( α, β ) -accurate ( ϵ, δ ) -DP preconditioner for Gaussian distributions with sample complexity O ( d α 2 + d 3 / 2 αϵ · polylog ( d αβϵδ )) .

B.4 Proof of Theorem 4.3

We use Algorithm 2 with the following parameters: B = R +10 4 κ √ d +log ( 2 log(2 /δ ) αϵ ) , ∆ = 2 B 2

, and n 1 = n 2 = ⌈ 10 4 C 2 ( B 2 · log(10 /δ ) cϵ )⌉ , where c, C are the constants from Lemmas 3.3 and 3.4.

Privacy Analysis. To analyze the privacy constraint, we first notice that the sensitivity of λ min ( ∑ i ∈ [ n 2 ] U i U T i ) is at most ∆ = ( √ 2 · B ) 2 and Lemma 3.1 ensures that the check, which only uses λ min ( ∑ i ∈ [ n 2 ] U i U T i ) + r , is ( ϵ/ 2 , δ/ 2) -DP. Passing this check ensures that

From Lemma 3.8, it suffices to show that the output of the algorithm is DP, under (3). Let X = X 1 , . . . , X n 1 +2 n 2 .

To do this, it will be more convenient to define V i = √ 1 -1 n 1 · U i . Condition (3) implies that

Let M ( X ) = 1 n 1 ( ∑ i ∈ [ n 1 ] X trunc i ) + ( ∑ i ∈ [ n 2 ] a [ i ] · V i ) . Consider any neighboring inputs X and ˜ X that satisfy the condition. There are two cases, based on where they differ.

(i) Case I: X , ˜ X differ on the first n 1 samples. Let v = 1 n 1 ∑ i ∈ [ n 1 ] ( ˜ X trunc i -X i ) ; notice that ∥ v ∥ ≤ 2 B n 1 . Furthermore, (4) implies that 1 n 2 ( ∑ i ∈ [ n 2 ] V i V T i ) ⪰ 0 . 5 I . Letting W denote the matrix whose columns are V 1 , . . . , V n 2 , d ϵ 2 ( M ( X ) || M ( ˜ X ) ) = d ϵ 2 ( Uni S W || Uni S W + v ) ≤ δ/ 2 , where the inequality follows from Lemma 4.5.

̸

(ii) Case II: X , ˜ X differ on the last n 2 samples; assume w.l.o.g. X n 1 +2 n 2 = ˜ X n 1 +2 n 2 . Let w = V ′ n 2 -V n 2 and W denote the matrix whose columns are V 1 , . . . , V n 2 -1 . Furthermore, let a, a ′ be independent samples from Uni S q .

where the inequality follows from the coupling a [ n 2 ] = a ′ [ n 2 ] = y , a [ n 2 ] ∼ Uni S n 2 , 1 and the observation that ( a [1] √ 1 -y 2 , . . . , a [ n 2 -1] √ 1 -y 2 ) and ( a ′ [1] √ 1 -y 2 , . . . , a ′ [ n 2 -1] √ 1 -y 2 ) are independently distributed as Uni S n 2 -1 .

Furthermore, (4) implies that 1 n 2 ( ∑ i ∈ [ n 2 -1] V i V T i ) ⪰ 0 . 5 I . Thus, Lemma 4.5 implies

Combining this with (5), we arrive at d ϵ/ 2 ( M ( X ) || M ( ˜ X ) ) ≤ δ/ 4 Pr y ∼ Uni S n 2 , 1 [ | y | > √ 10 log(10 /δ ) cn 2 ] . Since y 2 ∼ Beta (1 / 2 , ( n 2 -1) / 2) , we may apply the tail bound (Lemma 3.3) to get

Thus, d ϵ/ 2 ( M ( X ) || M ( ˜ X ) ) ≤ δ/ 4 + δ/ 4 = δ/ 2 .

In both cases, we have d ϵ/ 2 ( M ( X ) || M ( ˜ X ) ) ≤ δ/ 2 under (3). This concludes our privacy proof.

Accuracy Analysis. Let D = N ( µ, Σ) for some unknown µ, Σ . Consider the algorithm A ′ where there is no truncation and no halting. Since I ⪯ Σ ⪯ κ · I , standard concentration bounds (Lemmas 3.2 and 3.4) imply that the truncation and halting are not applied in A with probability at least 1 -α/ 2 . Therefore, we have d tv ( Q A ,D , Q A ′ ,D ) ≤ α/ 2 .

Now, notice that the algorithm A ′ just outputs Y := 1 n 1 ( ∑ i ∈ [ n 1 ] X i ) + √ 1 -1 n 1 · ( ∑ i ∈ [ n 2 ] a i U i ) ; this output Y ∼ N ( µ, Σ) . In other words, we have Q A ′ ,D = N ( µ, Σ) . Combining these, we can conclude that A is α -accurate.

C A Simpler but Worse Algorithm for Unknown Bounded Covariance

In this section, we give a simpler algorithm for the unknown bounded covariance setting. Its sample complexity bound of ˜ O ( d 3 / 2 αϵ 2 ) is worse than the one in Algorithm 2 and Theorem 4.3, but is still better than Ω ( d 2 α 2 + d 2 αϵ ) for DP learning [27] for constant ϵ > 0 .

The simpler sampler is given in Algorithm 3 and its guarantee is given in the theorem below. Note that this algorithm is once again just a form of (scaled) Gaussian mechanism, whereas the algorithm in the main body (Algorithm 2) adds noise that is input dependent.

+

Theorem C.1. Assume that I ⪯ Σ ⪯ κ · I for some κ > 0 and that ∥ µ ∥ ≤ R , there is an α -accurate ( ϵ, δ ) -DP sampler with sample complexity

Algorithm 3 BOUNDEDCOVARIANCEGAUSSIANSAMPLER

Parameters: B,σ > 0 , and n 1 , n 2 ∈ N .

Sample

X

1

, . . . , X

n

1

, X

n

1

+1

for i = 1 , . . . , n 1 +2 n 2 do

X

i

=

trunc

2

B Sample Z ∼ N (0 , σ 2 I )

(

X

i

)

n 1 , n 2 be such that n 1 = n 2 ≥ σ 2 ϵ 2 . Let A denote Algorithm 3 with parameters B,n,σ as specified.

Proof Sketch of Theorem C.1. Let B = R + 10 4 κ √ d ( log ( 2 d αϵ ) +loglog(2 /δ ) ) , σ = α 2 d 1 / 4 and 100 B 2 log(2 /δ )

Privacy Analysis. A is the Gaussian mechanism with noise multiplier σ B · √ 1 -1 /n 1 2 n 2 ≥ 10 √ log(2 /δ ) /ϵ

by the setting of our parameters; therefore, A is ( ϵ, δ ) -DP.

Accuracy Analysis. To understand the accuracy guarantee, let D = N ( µ, Σ) for some unknown µ, Σ where I ⪯ Σ ⪯ κ · I . Let us consider the algorithm A ′ where there is no truncation. Again, a standard concentration bound implies that the truncation is not applied anyway in A with probability at least 1 -α/ 2 . Therefore, we have

Now, notice that in algorithm A ′ , we just output

Without truncation, this output Y has identical distribution as N ( µ, Σ+ σ 2 I ) . In other words, we have

where the first inequality follows from Lemma A.3 and the last inequality follows from our parameter selection. Combining the above two inequalities, we can conclude that the sampler is α -accurate as desired.

D Gaussian Distributions: Lower Bounds

D.1 Known Covariance

In this section, we prove a lower bound of Ω( √ d/ϵ ) that holds even for the simplest case of known covariance (Theorem 1.3), showing that the dependence on d in our sampler (Theorem 4.2) is nearly optimal.

We prove this by using a DP sampler to draw a large, but constant, number of samples and the use them to perform mean estimation; the lower bound for mean estimation (Theorem A.6) then gives the desired lower bound for the sample complexity of DP sampler.

Proof of Theorem 1.3. Given an α -accurate ( ϵ, δ ) -DP sampler A , we construct an algorithm M for mean estimation as follows:

, . . . , X

n

1

+2

n

2

∼

D

For i = 1 , . . . , 10 6 :

-Run A on n fresh samples from D to get Y i .

Output ˆ µ ∈ R d with ˆ µ [ j ] := clip -1 , 1 ( median ( Y 1 [ j ] , . . . , Y 10 6 [ j ]))

Since A is α -accurate, each Y i is sampled from a distribution D ′ that is α -close (in total variation distance) to D = N ( µ, I ) . For α = 0 . 1 , this means that

As a result, standard concentration bounds (e.g., Theorem E.1) imply that

This in turn implies that

Applying Theorem A.6 with γ = √ d/ 6 , we can conclude that the sample complexity of M (which is equal to 10 6 n ) must be at least Ω ( d ϵ √ d/ 6 ) = Ω( √ d/ϵ ) . Thus, we must have n ≥ Ω( √ d/ϵ ) as claimed.

D.2 Unknown Bounded Covariance

Next, we will prove the lower bound for the unknown bounded covariance case (Theorem 1.5).

D.2.1 Reduction to Covariance Estimation

Algorithm 4 COVARIANCEESTIMATOR

Parameters: N,n,L ∈ N , sampler A sampler , agnostic learner A learner for centered Gaussians Sample X 1 , . . . , X N ∼ P

Randomly draw n subsamples X i ℓ 1 , . . . , X i ℓ n without replacement from X 1 , . . . , X N

if

0

else

ˆ Σ

′

return

⪯

ˆ Σ

.

5

I

⪯

2

.

5

I

then

ˆ Σ

′

′

ˆ Σ

As stated earlier in Section 2, we will prove this lower bound by reducing to covariance estimation (Theorem A.7). This is done by first taking N ≫ n samples, and then generating each Y ℓ by subsampling the input to n samples and running our DP sampler. These Y ℓ 's are then feed into the agnostic learner to produce an estimate ˆ Σ for Σ . The full reduction is given in Algorithm 4. Note that here A sampler will be the ( ϵ, δ ) -DP sampler and A learner will be the learner from Theorem A.8.

Privacy Analysis. The privacy guarantee of Algorithm 4 follows easily from the amplification by subsampling and advanced composition of DP. This is formalized below.

Proof. First, we apply Theorem A.4 which means that computing a single Y ℓ is ( ϵ ′ , δ ′ ) -DP where

ˆ Σ

I

and similarly,

Then, applying Theorem A.5 with δ 1 = 0 . 5 δ ∗ implies that all ( Y 1 , . . . , Y L ) together is ( ϵ 1 , Lδ ′ + δ 1 ) -DP where

Combining the above expressions, we have

and Lδ ′ + δ 1 ≤ δ ∗ / 2 + δ ∗ / 2 ≤ δ ∗ . In other words, ( Y 1 , . . . , Y L ) together is ( ϵ ∗ , δ ∗ ) -DP. Finally, applying A learner on ( Y 1 , . . . , Y L ) is simply a post-processing step. Thus, the entire algorithm is ( ϵ ∗ , δ ∗ ) -DP as desired.

D.2.2 Accuracy Analysis

Next, we argue that the accuracy of the covariance estimation algorithm, assuming the accuracy of the sampler and the agnostic learner.

Lemma D.2. Let ξ ∈ (0 , 0 . 01] and n, N, L ∈ N be such that N ≥ ( 10 nLd ξ ) 2 , and suppose that I ⪯ Σ ⪯ 2 I . Furthermore, suppose that

A sampler is an ( ξ 10 C ) -accurate sampler for the class of Gaussians under the assumption I ⪯ Σ ⪯ 2 I , and

A learner is an ( ξ 10 C , ξ 2 200 d 2 ) -accurate agnostic learner for the class of centered Gaussians,

where C is the constant in Lemma A.2. Then, E [ ∥ ˆ Σ ′ -Σ ∥ 2 Σ ] ≤ ξ 2 where ˆ Σ ′ denotes the output of Algorithm 3.

Proof. Let D = N (0 , Σ) denote the underlying distribution and for notational convenience, let ζ := ( ξ 10 C ) . Let E disjoint denote the event that i 1 1 , . . . , i L n are all different. Note that we have

̸

̸

where the last inequality follows from our choice of parameters.

Next, suppose that E disjoint holds. Then, we have that Y 1 , . . . , Y L are independently sampled from Q A sampler ,D . Applying the agnostic learning guarantee of A learner , with probability 1 -ξ 2 200 d 2 , we get

Recall also from the accuracy guarantee of the sampler that d tv ( N (0 , Σ) , Q A sampler ,D ) ≤ ζ .

Combining all of these together we have

Applying Lemma A.2, we get

Notice that if ∥ ˆ Σ -Σ ∥ Σ ≤ ξ/ 2 , then we have ˆ Σ ′ = ˆ Σ . Furthermore, I ⪯ Σ ⪯ 2 I and 0 . 5 I ⪯ ˆ Σ ′ ⪯ 2 . 5 I imply that ∥ ˆ Σ ′ -Σ ∥ Σ ≤ 6 d . Thus, we have

D.2.3 Putting Things Together

Combing the privacy and accuracy guarantees and plugging in the appropriate parameters immediately yields Theorem 1.5.

Proof of Theorem 1.5. Let α = ξ/ (10 C ) where ξ is as in Theorem A.7 and C is as in Lemma A.2. Suppose for the sake of contradiction that there exists an ( ϵ, δ ) -DP α -accurate sampler with sample complexity n = o ( d ϵ √ log d ) under the assumption I ⪯ Σ ⪯ 2 I . Then, let us select the parameters as follows:

L ≤ O ( d 2 ) denote the sample complexity of the ( ξ 10 C , ξ 2 200 d 2 ) -accurate agnostic learner for the class of centered Gaussians as given by Theorem A.8.

δ ∗ = O ( min { 1 /N,d 2 / ( N log N ) }) . Note that, when the constant in big-O notation is sufficiently large, our choice of parameters implies that δ ≤ ( 0 . 5 N Ln ) δ ∗ .

By Lemma D.2, we have that E [ ∥ ˆ Σ ′ -Σ ∥ 2 Σ ] ≤ ξ 2 . Furthermore, by Lemma D.1, we have that the algorithm COVARIANCEESTIMATOR is ( ϵ ∗ , δ ∗ ) -DP. However, we also have N = o ( d 2 /ϵ ∗ ) , which contradicts Theorem A.7.

E Product Distributions on { 0 , 1 } d

In this section, we describe and analyze our sampler for product distributions on { 0 , 1 } d (Theorem 1.7). We may assume w.l.o.g. that p 1 , . . . , p d ≤ 3 / 4 . (Otherwise, we first privately estimate p i using, e.g., the Laplace mechanism, and flip the i th bit in all samples if the estimate is more than 3 / 4 .)

Given a dataset element X i ∈ { 0 , 1 } d , we use X i [ j ] to denote its j th coordinate, and X i [ S ] = ( X i [ j ]) j ∈ S to denote the vector X i restricted to the subset S ⊆ [ d ] of coordinates.

E.1 Additional Preliminaries

Here we list a few concentration inequalities that are useful. We start with the following version of the Bernstein inequality [see, e.g., 42, Theorem 2.8.4]:

Theorem E.1 (Bernstein's inequality) . Let Y 1 , . . . , Y n be independent real-valued random variables such that, with probability 1, Y i ∈ [0 , C ] for all i ∈ [ n ] . Let V = ∑ i ∈ [ n ] var( Y i ) . Then, for any ∆ ≥ 0 , we have

We now list a couple of versions of this inequality, which will be more useful in our setting.

Corollary E.2. Let P be a product distribution on { 0 , 1 } d . For any β, γ ∈ (0 , 1 2 ) , let n > 50 γ log d β and X 1 , . . . , X n ∼ P . Let X := 1 n ∑ i ∈ [ n ] X i . Then, we have

Proof. For each j ∈ [ d ] , applying Theorem E.1 with ∆ = max { γ, 0 . 1 p j } · n and using p j ≤ 3 / 4 , we obtain

Taking a union bound over all j ∈ [ d ] completes the proof.

Corollary E.3. Let P be a product distribution on { 0 , 1 } d . For any β, γ ∈ (0 , 1 2 ) , let n > 50 γ log d β and X 1 , . . . , X n ∼ P . Then, we have

Proof. Denote Y j = X i [ j ] / max { p j , γ } for all j ∈ [ d ] . Here we have E [ Y j ] = p j / max { p j , γ } ≤ 1 and var( Y j ) ≤ p j / max { p j , γ } 2 ≤ 1 /γ . Moreover, with probability 1, we have Y j ≤ 1 /γ . As a result, we can apply Theorem E.1 with ∆ = 2 d +4log( n/β ) /γ , which gives

Taking a union bound over all i ∈ [ n ] completes the proof.

E.2 Private Preconditioner

We start with the following private preconditioner, which estimates each p j up to a constant factor. This preconditioner is similar to that of [27], except that we add noise from the Laplace distribution 13 (instead of Gaussian noise) to achieve pure-DP and that we are looking for a coarser guarantee compared to [27].

Lemma E.4. There exists an ϵ -DP algorithm that takes

samples as input and output integers ℓ 1 , . . . , ℓ d ∈ { 0 , . . . , ⌈ log( 2 d α ) ⌉} such that, with probability 1 -β , the following hold for all j ∈ [ d ] :

̸

if ℓ j = ⌈ log( 2 d α ) ⌉ , then p j ≤ α/ 2 d .

if ℓ j = ⌈ log( 2 d α ) ⌉ , then p j ∈ [ 1 / 4 · 2 -ℓ j , 3 / 4 · 2 -ℓ j ] , and

In the remaining of this section, we will focus on the proof of Lemma E.4 and Algorithm 5.

Sample Complexity. From Algorithm 5, the total sample complexity can be bounded by

as desired.

Privacy Analysis. In each iteration ℓ ∈ [ L ] , the samples X ℓ 1 , . . . , X ℓ n ℓ , after truncation, are noised with a Laplace distribution; by the privacy guarantee of the Laplace mechanism, each iteration is thus ϵ -DP. Furthermore, since each iteration uses a fresh set of samples, by the parallel composition property of DP, the entire algorithm is ϵ -DP as desired.

Accuracy Analysis. Wewill prove this by induction. Specifically, let S ↑ ℓ := { j ∈ [ d ] | p j ≤ 3 / 4 · 2 -ℓ } and S ↓ ℓ := { j ∈ [ d ] | p j ≤ 1 / 4 · 2 -ℓ } . We will show that

for all t ∈ [ L ] by an induction on t . Observe that this statement implies the desired accuracy claim in the lemma because j gets assigned ℓ j = ℓ if and only if j ∈ S ℓ \ S ℓ +1 . (Here we use the convention that S L +1 = ∅ .)

Algorithm 5 PRECONDITIONERPRODUCTDIST

Input: An unknown product distribution P over { 0 , 1 } d , privacy parameter ϵ , accuracy parameter α , failure probability parameter β

Inductive Step. Suppose that (6) holds for t -1 for some t ∈ N . We will also show that it also holds for t . From a union bound, it suffices to show

To show this, we need the following claim:

Claim 1 . Assume S t -1 ⊆ S ↑ t -1 . For each j ∈ S t , with probability 1 -0 . 5 β Ld , we have

Before we prove Claim 1, let us see how to use it to finish the proof. Let Z t ∼ Lap ( B t /ϵ ) be the Laplace noise added. First, by a standard concentration bound for the Laplace distribution, we have

Note also that by our choice of parameters, we have B t /ϵ · log (4 Ld/β ) ≤ 0 . 01 · 2 -t · n t . Hence, by a union bound, we have that with probability 1 -β/L for all j ∈ S t ,

Consider any j ∈ S t \ S ↑ t . Since p j ≥ 0 . 75 · 2 -t , we have q t [ j ] ≥ 0 . 58 · 2 -t > τ t . Hence, j will not be included in S t , coming to a contradiction. In other words, S t \ S ↑ t = ∅ , i.e., S t ⊆ S ↑ t . Similarly, consider any j ∈ S ↓ t -1 . Since p j ≤ 0 . 25 · 2 -t +1 , we have q t [ j ] ≤ 0 . 57 · 2 -t = τ t . By Algorithm 5, j will be included in S t . Hence, S ↓ t -1 ⊆ S t . These conclude the inductive step.

We are now left to prove Claim 1.

Proof of Claim 1. Let Y i = ( trunc 1 B t ( X t i [ S t ]) ) j .

First note that ∑ i ∈ [ n t ] Y i ≤ ∑ i ∈ [ n t ] X t i [ j ] . Therefore, we may apply Corollary E.2, which gives

Next, to give a lower bound on Y i , we also observe that

and

We note that S t ⊆ S t -1 holds by Algorithm 5 and S t -1 ⊆ S ↑ t -1 follows the assumption. Hence, p j ′ < 0 . 75 · 2 -( t -1) for every j ′ ∈ S ↑ t -1 . Moreover, | S ↑ t -1 | ≤ d . Putting everything together, we can further bound

Therefore, by Markov's inequality, we have

Combining with the above, we have Pr[ Y i = 1] ≥ 0 . 9 p j . Notice also that Y 1 , . . . , Y n t are i.i.d. and always lie between [0 , 1] . Thus, we can apply Theorem E.1 with ∆ = (0 . 1 p j + 0 . 01 · 2 -t ) n t to obtain

Applying a union bound then yields the claim.

E.3 Sampler

Next, we give a DP sampler for an unknown product distribution, assuming that a rough estimate of each p j is already computed via the private preconditioner in Section E.2. The exact guarantee is given below; combining this with Lemma E.4, we immediately arrive at Theorem 1.7.

Lemma E.5. Suppose that ℓ 1 , . . . , ℓ d satisfy the conditions in Lemma E.4. Then, there is an ϵ -DP algorithm taking ℓ 1 , . . . , ℓ d as input, that is an α -accurate sampler and has a sample complexity of

Algorithm 6 PRODUCTDISTSAMPLER

Input: An unknown product distribution P over { 0 , 1 } d , privacy parameter ϵ , accuracy parameter α , and ℓ 1 , . . . , ℓ d ∈ N satisfy the conditions in Lemma E.4

Output: A random sample that is α -accurate for P

In the following, we will focus on the proof of Lemma E.5 with Algorithm 6.

Privacy Analysis. Consider any pair X , X ′ of neighboring datasets. We may assume w.l.o.g. that the two datasets agree except for X n and X ′ n . We write q ′ , ˜ p ′ to denote the values of q, ˜ p computed for X ′ . Now, consider any possible output y ∈ { 0 , 1 } d . For each j ∈ [ d ] , we claim that

To show this, we first mention two facts:

It suffices to show 1 / (8 w j ) ≤ ˜ p j ≤ 1 -1 / (8 w j ) to prove (7). This follows since w j > 1 and 1 / (8 w j ) ≤ ˜ p j ≤ 7 / (8 w j ) due to clipping.

We also note the following useful property:

Finally, we are ready to prove the privacy guarantee:

where the third inequality is due to clipping, the penultimate step follows (8), and the last step follows from our choice of parameters. Thus, the algorithm is ϵ -DP as claimed.

Accuracy Analysis. Let S = { j ∈ [ d ] | p j < 2 -L } . We first consider the version of A (Algorithm 6), where there is no truncation at all and there is no clipping for any j / ∈ S , denoted as A ′ . Let Q A ′ ,P = Ber ( p ∗ 1 ) ⊗ · · · ⊗ Ber ( p ∗ d ) . Fix β = α/ 12 d < α/ 4 . We define the event E as 'no X i is truncated for any i ∈ [ n ] and no p j is clipped for any j / ∈ S by Algorithm 6'. By concentration results (Corollaries E.2 and E.3 with γ = 2 -L -1 ) and our selection of parameters, we can conclude that E happens with probability at least 1 -β , and therefore

Furthermore, we can see that E [ p ∗ j ] = p j if j / ∈ S . For j ∈ S , clipping ensures us that p ∗ j ≤ 2 -L , hence E [ p ∗ j ] ≤ 2 -L . Together, we have

Combining the above two inequalities yields the α -accuracy guarantee of A :

Proof of Theorem 1.7. Finally, we are ready to prove Theorem 1.7. For simplicity, let α ′ be the target accuracy parameter. Set α = α ′ / 2 and β = α/ 12 d < α ′ / 2 . We define the event E as 'Algorithm 5 returns ℓ 1 , . . . , ℓ d satisfying the properties in Lemma E.4'. If E holds, then the error of our sampler is at most α ′ / 2 , which is implied by Lemma E.5. If E fails, the error is at most 1 , but this event happens with probability at most β < α ′ / 2 . Together, the error of our sampler is

and the sampling complexity is (with β = α ′ / 24 d )

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper discusses the problem of generating a sample from a distribution that is close to an unknown distribution P, under the constraint of differential privacy. The authors present new differential privacy sampling algorithms for multi-dimensional Gaussian distributions, with different assumptions on the available information. They also obtain a pure-DP algorithm for product distributions on the binary hypercube. The paper aims to bridge the quantitative gap between DP distribution sampling and DP distribution learning.\n\nSTRENGTHS:\nThe paper proposes a new DP sampling algorithm for multivariate Normal and the product Bernoulli distribution.\nIt also aims to bridge the gap between DP distribution sampling and DP distribution learning. It could lead to more effective and efficient methods for privacy-preserving machine learning.\n\nWEAKNESSES:\nThe scope of the paper is quite limited as it only focuses on Gaussian and binary product distributions. While these are important classes of distributions, the applicability of the proposed methods to other types of distributions remains unclear. It would be beneficial if the authors could extend their analysis to a broader range of distributions or discuss the potential challenges and strategies for doing so.\n\nThe paper indicates that fewer samples are needed for estimation under these specific parameter families compared to sampling. It's not clear whether this is due to the geometry of the parameter space and output space, or if it's more related to the properties of differential privacy itself, perhaps due to the use of the composition theorem. It would be interesting to see a discussion on whether similar situations occur in non-DP settings. This could help clarify whether the observed phenomena are intrinsic to differential privacy or if they are more general statistical phenomena.\n\nQUESTIONS:\nSee weakness\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the problem of differentially private sampling. In this setting, the goal is the following: Given $n$ iid samples from an unknown distribution (e.g., a Gaussian distribution with an unknown mean), design a private algorithm that produces a fresh sample from the distribution the original samples came from. This framework was recently introduced with the following motivation: Sometimes it would be useful to have access to a synthetic dataset, i.e., a dataset that behaves in the same way as the original one but doesn't reveal any information about it.\n\nThe authors systematically investigate this problem for various cases of the gaussian distribution. They give algorithms and nearly matching lower bounds. They also have some results for the binary setting.\n\nSTRENGTHS:\nIn my eyes, this paper makes important contributions to a new interesting line of work. In particular, it shows that, at least for the Gaussian distribution, the private sampling task is inherently easier than the associated learning task, e.g., learning the mean. For the former, the dependence on the accuracy parameters is only logarithmic, while for the latter it is polynomial.\n\nThe authors investigate this task in a systematic way, in the gaussian case, they study three different natural settings (known covariance, unknown and bounded covariance, and unknown and unbounded covariance) and give upper and lower bounds.\n\nWEAKNESSES:\nFor the discrete setting, the dependence on the accuracy parameter is qualitatively worse (polynomial vs. logarithmic). It would be nice to give some intuition if this necessary and if so, why. For instance, is there any lower bound that you can prove? Even one that is not necessarily tight.\n\nQUESTIONS:\nRelated to my comment above: Do you believe the discrete setting is inherently harder? Could it be that the difference between learning and sampling is less stark in this setting?\n\nLIMITATIONS:\nThe authors discuss limitations and open questions related to their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem of sampling from distributions under differential privacy. In particular, the specific formulation presented in the paper assumes the existence of a dataset that has been drawn i.i.d. from a distribution $P$ that belongs in a family of distributions $\\mathcal{F}$ and the goal is to a generate a sample from a distribution $Q \\in \\mathcal{F}$ such that $d_{\\mathrm{TV}}(P, Q) \\le \\alpha$ in a way that protects the privacy of the input dataset. This problem is considered for various choices of $\\mathcal{F}$, which include i) Gaussians with uknown mean but known covariance, ii) Gaussians with unknown mean and unknown covariance $\\Sigma$ which satisfies $I \\preceq \\Sigma \\preceq \\kappa I$ for some known $\\kappa$, iii) Gaussians with unknown mean and covariance without any a-priori bounds on $\\Sigma$, iv) binary product distributions. The results include upper bounds for the aforementioned families, as well as lower bounds for families (i) and (ii).\n\nSTRENGTHS:\nThe writing is fairly accessible and it is relatively easy to grasp the results and follow the exposition. In terms of contributions, I feel the main strength of the paper lies in the results about binary product distributions, which constitute significant improvements over prior work. Additionally, the arguments via which the lower bounds are established are quite elegant.\n\nWEAKNESSES:\nI have two main concerns with this work.\n\nThe first has to do with the bounds given for Gaussian distributions. In particular, while the guarantees given in the paper are tight (or nearly tight) with respect to the dimension $d$, it seems like the guarantees are sub-optimal in terms of the range parameters. For the case of Gaussians with known covariance but unknown mean, the dependence in the sample complexity for sampling under approximate differential privacy is polynomial with respect to $R$ (Theorem 4.2), where $R$ is a positive real number such that $|| \\Sigma^{- \\frac{1}{2}} \\mu ||_2 \\le R$. In [KV18], the authors show that, under approx-DP, you do not need any a-priori bounds on the mean to do parameter/density estimation for univariate Gaussians. This immediately extends to $d$-dimensional Gaussians with known covariance, since one can rescale the datapoints by $\\Sigma^{- \\frac{1}{2}}$ to make the Gaussian spherical and then apply the algorithm of [KV18] component-wise. Thus, it feels out of place for sampling to require a polynomial dependence on $R$, despite sampling being an easier task than estimation. Similarly, I have a similar concern for the guarantees of Theorem 4.3, both related to the dependence in $R$ and in $\\kappa$.\n\nThe second concern (which is not as significant as the first one) has to do with the model considered. The standard formulation of the sampling problem involves having a known target distribution that you want to draw a sample from and defining a process that randomly generates points that come from a distribution that's close to the target (e.g., via a random walk). In this formulation, there is no input dataset to work with, thus not leaving much room for formulations that involve DP. In light of this, the variant of the standard sampling problem considered in this paper feels somewhat artificial. It might make sense in the context of e.g., synthetic data generation (a topic on which there is a broad literature), but it feels a bit odd to market this as ``sampling with DP\". However, to be fair, this is a criticism that one could make about [RSSS21], which is the paper that introduced the model that the present paper considers. For that reason, this is not something I hold against this work and I consider it minor compared the points raised in the previous paragraph.\n\nOverall, if the guarantees were to be optimized in terms of the range parameters, I believe a polished version of this paper could stand a chance to be accepted. However, in its current form, I feel this work is not ready for publication.\n\n==== Update ====\n\nThe rebuttal addressed my main concern. For that reason, I raised my score from 3 to 5. Depending on how discussion among reviewers and the AC goes, I might further increase my score to 6. The reason I'm not considering giving the paper a higher score is due to the issue I raised regarding the fact that the model might be of somewhat limited interest.\n\n[KV18] Vishesh Karwa and Salil P. Vadhan. Finite sample differentially private confidence intervals. In\nITCS, pages 44:1–44:9, 2018\n\n[RSSS21] Sofya Raskhodnikova, Satchit Sivakumar, Adam Smith, and Marika Swanberg. Differentially\nprivate sampling from distributions. In NeurIPS, pages 28983–28994, 2021.\n\nQUESTIONS:\nI guess the one question I have has to do with the main issue I raised in the section about weaknesses. Is there something that I have missed or misjudged in terms of the dependencies in the range parameters? If I have misunderstood something and the authors can clarify things to me, I am open to raising my score.\n\nLIMITATIONS:\nThe work is purely theoretical and doesn't have any obvious prospects for societal impact (positive or negative).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nRaskhodnikova, Sivakumar, Smith, and Swanberg (NeurIPS '21) recently introduced the problem of privately approximate sampling: Given i.i.d. draws from a member D of a known family of distributions, produce one sample whose distribution is close to D in TV distance. This is an easier task than the well-studied problem of private distribution learning, while isolating some of its algorithmic difficulty, and RSSS'21 gave several upper and lower bounds for discrete distributions and product distributions.\n\nThis submission addresses several open directions from RSSS'21 by 1) thoroughly investigating the sample complexity of privately sampling from Gaussians, and 2) giving a pure DP sampler for Bernoulli product distributions with optimal sample complexity (the sample-optimal algorithm from RSSS'21 only guaranteed approximate DP). In more detail, this paper establishes tight (up to polylog factors) sample complexity upper and lower bounds for sampling from Gaussians with either known or a prior bounded covariance, as well as giving an upper bound in the unbounded covariance case that separates DP sampling from DP learning.\n\nSTRENGTHS:\n- The problem of private sampling is reasonably well-motivated from the standpoint of \"applications\" (i.e., minimizing the overhead of privacy when full distribution learning is not necessary) as well as from the standpoint of doing theory, i.e., developing a finer understanding of what makes private statistical analysis challenging, establishing new algorithmic primitives that can be used elsewhere, and more generally forcing us to develop new techniques.\n\n- Within the context above, this paper addresses two of the most fundamental classes of distributions. In several cases, it identifies nearly matching upper and lower bounds.\n\n- The paper makes the interesting finding that, at least for Gaussian sampling, the dependence on the error parameter $\\alpha$ is at most polylogarithmic. In contrast, the dependence is at least polynomial for learning Gaussians, as well as for the discrete sampling tasks studied previously.\n\n- Both the algorithms and lower bounds exhibit interesting (and in my opinion, surprising) technical insights. For instance, Algorithm 2 for learning Gaussians with unknown bounded covariance describes and analyzes a new sampling distribution based on taking linear combinations of spherical noise. As another example, the lower bound for the same setting implements a conceptually very natural reduction, but has to overcome a technical issue of breaking the correlation between samples generated by running a private sampler multiple times on the same dataset (which addresses a more general issue with DP sampling and could be used in other contexts).\n\nWEAKNESSES:\n- The algorithms for Gaussian sampling are quite specialized to Gaussians, and it's quite difficult to imagine how they might generalize to other classes of distributions.\n\n- This is sort of a nitpick, but I think the writing could highlight some of the new ideas described under \"strengths\" above, e.g., with lemma statements that would make them easier to apply elsewhere.\n\nQUESTIONS:\n- On the topic of generalizing to other distributions: Many algorithms for learning distributions (including Gaussians) readily generalize to the semi-agnostic case, incurring additional error that scales with the amount of model misspecification. Does such a ready generalization apply to the algorithms in this paper?\n\n- Lines 81-82 mention that the paper's algorithms run in polynomial time. In what computational model? Since the problems considered require TV-close sampling from continuous distributions, it's not clear that taking a fixed precision discretization suffices.\n\n- Is it possible to incorporate some polylogarithmic in $\\alpha$ dependence on the error in the lower bounds for Gaussian sampling? (As things stand, it could be the case that the right dependence on $\\alpha$ is, say, doubly logarithmic.)\n\nLIMITATIONS:\nThe paper's conclusion raises the question about extending beyond Gaussian distributions mentioned above, as well as the problem of closing the quantitative gap between upper and lower bounds for the unbounded covariance case.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-11"}

comments: SUMMARY:
This paper considers a very interesting problem: Design a $\mathit{differentially~private}$ algorithm that given samples from unknown distribution $D$ from a class of distributions ${\mathcal D}$, output a sample from a distribution $D'$ where $D'$ is close to $D$ in total variation distance. One direct approach is to first learn the distribution $D$ in a differentially private manner and output a sample from the distribution. However, the paper argues that learning is an overkill. The paper shows that the sample complexity of DP-sampling problem is substantially less than the DP-learning problem in many cases. In particular, they design algorithms for DP-sampling algorithms of $d$-dimensional Gaussian distributions with optimal sample complexities. They consider three subclasses of Gaussian distributions: (1) known covariance (2) unknown but bounded covariance (3) unknown unbounded covariance -- and sample optimal DP-sampling algorithms for these class of distributions. They also consider the class of product distributions and give optimal pure DP-samplers. Earlier, only approximate DP-samplers where known.

STRENGTHS:
The problem they consider is very interesting to me: producing samples from unknown distribution in a differentially private manner appears to be a very basic task. Investigating this problem for Gaussian distributions should be of broad interest.

WEAKNESSES:
I am not sure whether it is a weakness, all proofs are in the appendix and are for specialist so may not be accessible to general audience.

QUESTIONS:
I do not have any major questions at this point.

LIMITATIONS:
This is a theoretical paper and does not seem to have any negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-12