Private Estimation with Public Data
Authors: Alex Bie
Gautam Kamath
Vikrant Singhal
Venue: NeurIPS 2022 Accept (2022)
Keywords: Differential Privacy
Learning
Machine Learning
Data Privacy
Statistics
Gaussians
Mixtures of Gaussians
Covariance Estimation

Abstract: We initiate the study of differentially private (DP) estimation with access to a small amount of public data. For private estimation of $d$-dimensional Gaussians, we assume that the public data comes from a Gaussian that may have vanishing similarity in total variation distance with the underlying Gaussian of the private data. We show that under the constraints of pure or concentrated DP, $d+1$ public data samples are sufficient to remove any dependence on the range parameters of the private data distribution from the private sample complexity, which is known to be otherwise necessary without public data. For separated Gaussian mixtures, we assume that the underlying public and private distributions are the same, and we consider two settings: (1) when given a dimension-independent amount of public data, the private sample complexity can be improved polynomially in terms of the number of mixture components, and any dependence on the range parameters of the distribution can be removed in the approximate DP case; (2) when given an amount of public data linear in the dimension, the private sample complexity can be made independent of range parameters even under concentrated DP, and additional improvements can be made to the overall sample complexity.

Private Estimation with Public Data
Alex Bie University of Waterloo yable@uwaterloo.. ca
Gautam Kamath University of Waterloo @@csail.mic
Vikrant Singhal University Waterloo
vikrant. wwwghaaluuaaaeroooooooooo
Abstract
We initiate the study differentially private (DP) estimation with access to a small amount of public data. For private estimation of d-dimensional Gaussians, we assume that the public data comes from a Gaussian that may have vanishing similarity in total variation distance with the underlying Gaussian of the private data. We show that under the constraints of pure or concentrated DP, + public data samples
sufficient to remove any dependence the private data distribution from the private sample complexity, which is known to be otherwise necessary without public data. For separated Gaussian mixtures, we assume that the underlying public and private distributions are the same, and we consider two settings: (1) when given diimeniion-inependder amount of public data, the private sample complexity can be improved polynomially in terms the number of mixture components, and any dependence on the range parameters of the distribution can be removed in the approximate DP case; (2) when given amount of public data linear in the dimension, the private sample complexity can be made independent of range parameters even under concentrated DP, and additional improvements can be made to the overall sample complexity.
the range parameters of
Introduction
Differential privacy (DP) [DMNS06] guarantees the privacy of every point in a dataset. This is strong requirement, and often gives rise to qualitatively new requirements when performing private data analysis. For instance, for the problem of private mean estimation (under pure or concentrated DP), the analyst must specify a guess for the unknown parameter, and needs more data to get an accurate estimate depending on how good their guess is. This cost can be prohibitive in cases where the data domain may be unfamiliar.
Fortunately, in many cases, it is natural to assume that there exists an additional public dataset. This public dataset may vary in both size and quality. For example, one could imagine that fraction of users opt out of privacy considerations, giving a small set of public in-distribution data. Alternatively, it is common pretrain models large amounts of public data from the web, which may be orders of magnitude larger than the private data but significantly out distribution. In a variety of such settings, this public data can yield dramatic theoretical and empirical improvements utility in private data analysis (see discussion in Section 1.3). We seek to answer the following question:
How can one take advantage of public data for privale estimation?
We initiate the study of differentially private statistical estimation with a supplementary public dataset. In particular, our goal understand when small amount public data can significantly reduce the cost of private estimation.
AAthhors listed in alphahctical order.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

1.1 Problem Formulation
We investigate private estimation with public data through the canonical problems of estimating Gaussians and mixtures of Gaussians. We focus on parameter estimation from i..d. samples drawn from public and private data distributions.
We wish to design estimation algorithms that take as input both public data and private data, and are private with respect to private data. Formally, say randomized algorithm A Xm X y taking as input public data X E n and private data X X is private with respect to private data for any setting of the public data X. A(X,)) X + y is differentially private.
Gaussians. Let Hi E R2d and positive definite Σ, E E Rdxd be unknown, with drv(N (j, Σ)) < for some known 0 I < 1. Then for 0 < o, < 1, given
public samples from W 2) and private samples from (1,, E), we would like to output μ. Σ such
that with probability 2 B. dWv (p, S a.
Mixtures of Gaussians. We represent a Gaussian mixture D by sequence of k tuples ((1, wi)...... ... (peg, ER, wx)), each representing the mean, covariance matrix, and mixing weight of a component. Then for 0 < 3 < 1, given public and private samples drawn from D, we would like to output Gaussian mixture D [( Σi, @ )....... (Ma; Ex, (x)) such that with probability 1 11-3, 3, for each [k], there exists distinct j [k:), such that drv (Ni,,, 2x), N (ū, Σ,)) S a and Iw; أان I a/k. This implies drv(D, D) s O(a), and we refer to this stronger notion of component- wise estimation as (o, -elaariing (Definition 10).).
1.2 Results
Gaussians. In the case of pure and concentrated DP, exactly d public samples are sufficient to remove any dependence "range parameters" in the private sample complexity, even when the underlying Gaussian distribution of the public data bears almost no similarity to that of the private data. We start with following result for the scenario when the two distributions are the same.
Theorem 1.1. For all a, 3, > 0 and 2 1 there exists a computationally efficient 2CDP (and computationally inefficient s-DP) algorithm M that takes d d 1 public samples and n private samples from a Gaussian D over Rd, which is private with respect to private data, such that if then M estimates the parameters of to within TV error a pollyog (d. 3 a
12 de 0
ac with probability 2 B.
Without public data, the analyst would be required to specify a bound on the & norm of the mean and bound K on the condition number of the covariance, and the private sample complexity would scale logarithmically in terms of these parameters (which we denote as range parameters). Thus, this cost could be arbitrarily large, even infinite. given poor priori knowledge of the problem. Note that these requirements are not due to suboptimality of current private algorithms. Packing lower bounds for estimation under pure and concentrated DP show that dependence on range parameters is an inherent cost of privacy. However, this dependence can be entirely eliminated at the price of + 1 public data points, which is less public data than the non- non-private O d² sample complexity of the problem. For estimating Gaussians with identity or known covariance (that is, mean estimation) turns out that just one public sample suffices to remove all dependence on range parameters.
The next natural question to ask is: what if our public data does not come from the same distribution as our private data? Indeed, in practical settings, there may be significant domain shift between data that is publicly available and the private data of interest. Our next result shows that we can relax this assumption: it suffices that our public data comes from another Gaussian that is at most y < 1 away in TV distance from the underlying private data distribution.
Theorem 1.2. For all a, 3, > 0, 1, and 0 S Y there exists computationally efficient 2 -CDDP (and computationally inefficient s-DP) algorithm M that takes + public samples from
'rrevious work exploring private algorithms with additional public data, such [BNS16a; BCMNUW20J, adopt this privacy requirement for their algorithms.

---

Gaussian D over Bd and n private samples from Gaussian D over Rd with drv Sev(D,D) S
which is private with respect to private data, such that if = O ( + + arE -polylog d, B I-7
then M estimates the parameters of D within TV error a with probability 2 1 3.
The upper bound on the TV gap y results in the sample complexity picking up an extra polylog( و factor. For instance, even when the public and private distributions have total variation distance 1 1/d (that is, they are almost entirely dissimilar), the sample complexity. In particular, note that the public dataset may be simultaneously quite small and significantly out of distribution with respect to the private data.
incur only an extra poly log d factor in
Mixtures of Gaussians. We focus on estimation of mixtures of k separated Gaussians in d dimensions. Specifically, we assume that, given mixture components N M E,) Ex) with respective mixing weights wy for E [k:] (with wi 2 Wmin), then for each i + j.
|144 12 Ω
Vk
max
E 2. 12
(1)
W
Private estimation of such mixtures of Gaussians (without public data) was previously studied by [KSSU19]. Their algorithms proceed in two stages: first, clustering step to isolate the samples from each component, and an estimation step, which estimates the parameters of each individual Gaussian using the isolated samples from each component. Our algorithms follow similar structure, but employ the public data at appropriate points in the procedure to enable substantial savings.
I In all the results that follow, Ô hides the polylog factors of d, k, the inverse of error parameters a 3 1. We provide two flavours of results for learning mixtures and the inverse of privacy parameters 1. 8 p'
of Gaussians. First, we investigate the setting when have very small, dimension-indeeendeen amount of public data.
Theorem 1.3. For all a, e,,, > O, there exists an DD algorithm M that takes n private samples and m public samples from Gaussian mixture D. satisfying (1), which is private with respect to private data, such that if m = =0 then M (a, By-learss D.
log(k/88 and n n==0 Ō Wmin
d²
d?
d' g2.
Wimmn a
Winin Du
Theorem 1.4. For all a. 3. > 0, there exists p-zCDP algorithm M that takes n private samples and m public samples from Gaussian mixture D satisfying (1), which is private with respect
to private data, such that if Vi, ||pill2 I R and I 3 E. 1 KIL. and
O
log(k/3) and
Wawin
d²
d²
d=223..5) poolyoog (K, then M (a, B)-lears D. vp
Wanin
Wiin aminayp
Our results differ from the prior work of [KSSU19] in the following ways: (1) we have results for both approximate DP and zCDP, as opposed to just approximate DP in [KSSU19]; (2) the private sample complexity due to the clustering step does not involve any dependence on the range parameters; and (3) the sample complexity due to the clustering step is now O(d'-622.b), which is a significant improvement, and shows up only due to applications of private PCA under better sensitivity guarantees. As far as the private sample complexity in the component estimation process is concerned, in the case of approximate DP, we have expressions independent of distribution parameters owing the recent Gaussian learner by [AL.21].
opposed to 0(d1-5g9.00)
Finally, we investigate Gaussian mixture estimation with an amount of public data which is linear in the dimension.
Theorem 1.5. For all a, B,5,5 > O, there exists an (E, 5)DP algorithm M that takes n private samples and public samples from Gaussian mixture D satisfying (1), which is private with
respect to private data, such that if m = O (k33) and n = Ō Wming a + (a., -learss D.
d²
then M
Wanio
"iiin
Theorem 1.6. For all a, B.P. > O, there exists P-ZCDP algorithm M that takes private samples and m public samples from Gaussian mixture D satisfying (1), which is private with respect to private data, such that if m O learns D.
d dlog(k/B) and n Ō O +
wuainn/p)) then M (a,, B)-
d²
Main
3

---

Using (dd/un Wmin) public samples establishes all the advantages that using O(1/ //min) public samples gives. but additionally improves the private sample complexity even further. As we will describe the technical overview, the clustering process is done entirely using the public data, and ends up removing the term due to private PCA from the private sample complexity in both approximate DP and zCDP regimes. Additionally, in the zCDP case, we do not have any dependence on distribution range parameters even in the component estimation process because we can use results from Theorem 1.1 this time.
1.3 Related Work
large part of our work focuses on using public data to obviate the dependence on range parameters in private estimation. Understanding these dependences without public data has been topic of significant study. [KV18] investigated private estimation of univariate Gaussians, showing that loga- rithmic dependences on the range parameters were both necessary and sufficient for pure differential privacy, but by using stability-based histograms [KKMN09; BNS1 6b)] they could be removed under approximate differential privacy. Similar results have been shown in the multivariate setting: while logarithmic dependences are necessary and sufficient under pure or concentrated DP [KLSU19], they can be removed under approximate DP [AAK21; KMSSU21; TCKMS21; AL21; KMV21; LKO21]. More broadly, a common theme is that the dependences which are intrinsic to pure or concentrated DP be eliminated by relaxing approximate DP. Our results demonstrate that instead of relaxing the privacy definition for all the data, one can achieve the same goal by employing a much smaller amount of public data.
Our investigation fits more broadly into a line of work employing public data for private data analysis. Works, both theoretical and empirical, investigate the role of public data in private query release, synthetic data generation, and prediction [JE13; BNS 1 6a; ABM19; NB20; BCMNUW20; BMN20; LVSUW21]. For distribution- free classification with public and private data, it was shown in [ABM19] that, roughly speaking, a hypothesis class is privately learnable with small amount of public data (o(1 (1)))), it is privately learnable with no public data. For density estimation, we show that the family of unbounded Gaussians learnable with a small amount of public data (a-independent), despite the fact that it is not learnable without public data.
Additionally, a number of empirical works study the efficacy of public data in private machine learn- ing, including methods such public pre-training IACGMMTZI6; PCSTE19; TB21; LWAFF21; YZCL21; LTLH22; Yu+22], using public data to compute useful statistics about the private gra- dients IZWB21: YZCL21; KRRT21; IGMRSSSTT21], or using unlabeled public data to train student model [PAEGTI7; PSMRTE18; BTGTI8].
Another setting with mixed privacy guarantees is when different users may require local versus central differential privacy [AKZHL11]. Mean estimation has also been studied this setting [ADK19] 19].
1.4 Organization
The remainder of the paper is organized as follows. In Section 2, we discuss our results for estimating multivariate Gaussians. After that Section 3, we give a technical overview of our results for estimating mixtures of Gaussians. We present all the relevant notations and preliminaries about dif- ferential privacy. Gaussian mixtures, and techniques like PCA, along with few useful concentration inequalities in Appendix A. In Appendix B, provide further technical details that missing from Section 2. We provide all our technical results about mixtures of Gaussians from Section 3, including algorithms, theorems, and proofs, in Appendix C. Finally, in Appendix D we present some proof-of -o-cceet- numerical simulations demonstrating the effectivenss of public data for private estimation.
2 Estimating Gaussians
We discuss our results for estimating multivariate Gaussians with access to limited amount of public data. First, as a warm-up, we consider the case of estimating the mean of an identity (or known) covariance Gaussian, and give p-zCDP private algorithm that uses 1 public sample. We then generalize the high-level approach taken there to arrive at private algorithm that uses (d+ + 1) public samples for general Gaussians. Furthermore, we show that can relax the condition that the

---

public and the private data come from the same distribution the public data can come from another Gaussian that is at most y far away TV distance.
2.1 Warmup: Identity Covariance Gaussians with Public Sample
In this setting, we are given a single public sample X, along with private samples X1...., X,, where X and the Xj are drawn from d-dimensional Gaussian N(,, T) independently. Our main observation that a single public sample is sufficient to get "good enough" coarse estimate of μ., allowing to apply existing private algorithms for finer estimate. This is captured in the following claim, which is a standard fact regarding norm concentration of Gaussian vectors (Lemma A.3).
Claim 2.1. Suppose get sample X NN, I). Then with probability at least 1 - 3, 1|u X|12 S
Vd+ ( + 2og (1/3).
We restate the guarantees an existing folklore p-zCDP "clip-and-noise" algorithm via the Gaussian mechanism for estimating identity covariance Gaussians.
Lemma 2.2 (7CDP Private Gaussian Mean Estimation). For all R, >>>>> O, there exists P-zCDP algorithm that takes n = Ö [+Hog(1/2)
(o private samples from N (p,,I) over avP
a²
Rd where |llll < R. and outputs an estimate ÿ E Rd, such that with probability at least 1 - B, (NV(H I). 1)) I a.
The above "clip-and- mechanism simply clips all the points in the dataset to within radius A R + d 2 llgg(m/B)) + 2 log (n/B) around the origin, and outputs the noisy empirical mean of the clipped points via the Gaussian mechanism.
Claim 2.1, combined with the result above, yields a i-public-sample private mean estima- tion algorithm via the following three-step process: (1) form shifted private dataset Vi,..., Yn by taking each Y,
X, X; (2) run "clip-and-nosee" on YI,..., Yn. setting Rt yd d 2 (lo((23)) + 2 (og(2/3) = O( dlogll/g)/)) and target failure probability 3/2, to obtain jy: and (3) output = Jyy X.
By union bound, with probability 2 1-13. 3, the distribution of YI...., Y. satisfies tight boundedness independent of the range parameters, and furthermore, succeeds in outputting good estimate joy for Jy = EY, The last step undoes the shift to yield good estimate fu for the original μ. Privacy is preserved with respect to private data, since we are doing entry-wise pre-processing and post- processing with public data. For full version of this argument (which includes the s-DP case), see Theorem B.2.
This yields a private sample complexity that no longer depends on the range bound R, allowing us to use fixed sample size to estimate the family of all identity covariance Gaussians over RRd
Theorem 2.3 (ZCDP Gaussian Mean Estimation with Public Data). For all a, B.P > 0, there exists a p-zCDP algorithm that is private with respect to the private samples, takes public sample and Ō d+log(1/8) d'log- private samples from N(s,, T) over Rd, and outputs an estimate av/
Rdd, such that with probability at least 1 - B. drv (N, N 11) S
2.2 General Gaussians with d + Public Samples
In this section, we discuss our main result for privately estimating Gaussians with public data.
Theorem 2.4 (Private Gaussian Estimation with Public Data), For all a, e.B.e.p > there exist p>cCDP and E-DP algorithms that take d + public samples "3.....X44,, and n private samples X1....,X, from unknown, d-dimensional Gaussians N N(p, Σ) and N N(H,E), respectively, with 0 S drv(N(4, (T, Σ)) & < 1, are private with respect to the private samples, and return 7 E Rd and Ex E Rdxd, such that (N (fxx, Ex), Σ)) S with probability at least 1 B, as long as the following bounds on n hold.
1. For p-zCDP (with computational efficiency) via [KLSU19] and d public samples,
5

---

d + log
1
d² polylog
d aBp
+ dlog
dl.s polylog Ba((->)
(3p(1->)
n = =
a²
avp
Vp
2. For S-DP (without computational efficiency) via [BKSW19] and d + public samples,
d + log (3)
d log (18(1-7) aE
d
= 0
a²
For simplicity, the rest of this section examines the case, that is, when the public data and the private data distributions are the same. Later, we address what modifications are needed for the >> 0 case.
Public Data Preconditioning. We extend the same high-level idea from the identity covariance case. We use public data to do coarse estimation, and then use that coarse estimate to transform the private data, reducing to the bounded case that can be solved using existing private algorithms (Lemma A.22). Specifically, we use the d + public samples for "public data preconditioning".
Algorithm 1: Public Data Preconditioner PubP Pecornccndi (X)
Public samples X = (X),.... .....,,, X Failure probability 0. Output: j Rdd Σ E Rdxd L RR U TR.
Input:
// Compute the empirical mean and covariance of X.
d+1
X and Σ + 7 Σ
Xx -) 72)
d
1=1
1=1
// Compute L and U.
U + 9d2
L+ +
and
4d 4 2d log 3
log
Return L,)
Public-Private Gaussian Estimator. We obtain public-private Gaussian estimator via the fol- lowing process. (1) We use the preconditioning parameters output by Algorithm to recenter, then rescale private data X X X,, Specifically, the transformed private data, denoted by Yı...., Yn, are given by Y, == 1 5x1 - P). (2) The Yı...., Y. are then fed as input to an ex- isting DPGaussianESsimaator (from Lemma A.2 for bounded Gaussians. It outputs estimates çoy and Ey. (3) For our final estimates, we output Σx = Ly S1/2 and fax = VLE + p. See Algorithm 2 in Appendix for an exact description.
VL
To establish correctness, we must ensure that, with high probability over the sampling of public data, the parameters of the true distribution underlying Yi,... Y, indeed satisfy tight range bounds required by OPGaassiinnEStimatto to provide the desired success guarantee with parameter-free private sample complexity. other words, the mean of the transformed Gaussian must lie in known poly(d, 1/3) ball, and the condition number of its covariance must be poly 1/3).
Lemma 2.5 (Public Data Preconditioning). For all 3 > O, there exists an algorithm that takes d ++ 1 independent samples X1,...,XA Xa+1 from Gaussian N(n, E) over Rdd and outputs ğ E Ted
E E addd L E IR and U R. such that Ey 1/2 and pv J/ 5-11/2 ( p.) then with probability at least 1 over the sampling of the data, (1) - Ey 3 41 and (2)
|| S V /5log(3/3), where 4. =
log(1/18)

---

Proof. We prove the theorem by proving the utility of Algorithm 1. Note that the quantities L, U. as defined in Algorithm 1, satisfy U/L = (og(/)))) To prove part (I), where we wish to bound the eigenvalues of Σy, by the the properties of Loewner ordering, and since Σ, Σ are symmetric, we have
1 3 I
1/2
U L
+ LU = LE-1/2EE 1/2 31 US 1/-
L
Therefore, is sufficient to prove the final inequality. Let Eg =====~11 - Then
d+1
d+1
Ez - 1=1
1))² = Σ im1
(Z: -
where for each i E --=1, ~ (O, 1) independently and u2 = --11/2 n) / Id+1 Zg. Therefore, it suffices to show L.2, 3 and 3 U. For i E [d]. let W; II) independently. Then Cochran's theorem (see Fact B.4) says that Ez identically distributed to à Eis1 W.W.". From here, we can apply the bounds from Fact B.3 by noting that aa(522) j ([ 1...... Wa)) and (E.z) 1 ([W.....Waa)).
With probability 2 B/3, a (W1,... Wal)² > (8/3)2
USIZ II. II.
(2+++ / 221x(3388)) d
With probability 2 3/3, 0 (Wi,..., Wa))
S.S.2 II.
Taking the union bound, (1) holds with probability at least in (2). We have
2 3.
Next, we prove the bound on juy
5-12 1 E 1/1 - 7 VI. by VT.
Σ Σ
12) =- 1 1/1/2/1//22
11
d+1
VI.
==1
Since fiz is identically distributed /d4i Z1, by concentration of the norm for multivariate Gaussians (Lemma A.3), we have that with probability 2 1-3/3,
d 2 Vd (3/3 log (3/3)
Iizl22 S 1
d+1
From (1), we know that implies that
5 1 Hence, || - 1 -1/2|
S VUTT. This
I.
luyl12 S -
1-125/21/22
|1 S V
V TT d 2 d log log(3/3) (3/3)
VI
d+1
which implies (2). Applying the union bound again, we have the claim.
Hence the process described above is indeed (d + 1) --public-sampll private algorithm satisfying privacy with respect private data Xi...., X,, and the required accuracy guarantees. This follows from the guarantees of public data preconditioning (as stated in the aforementioned Lemma 2.5) combined with the guarantees of existing DPGaussianFstimatt Theorem B.5 for full details. Again, we note that our sample complexity no longer depends on the priori bounds on the mean and the covariance of the unknown private data distribution.
(stated in Lemma A.24). See
Different Public and Private Distributions. Thus far, we have only considered the case where the public and the private data come from the same distribution. In the more general case where the public data XI...., X4+1 comes from Gaussian D within TV distance 7 < 1 of the private data distribution D, it be shown that Algorithm (public data preconditioning) with slight modification will work similarly to map our unbounded problem onto a bounded one. particular, outputting U, 4U/(1 -2)² and L, (1 * L/4 instead of U and L leads to covariance 7))) and R O(dlog(1/8)//(L
and norm bounds K O(d log(1/8)/
7,.) on the
7

---

underlying parameters of the transformed private data. This result is the consequence of a technical Lemma that translates the TV bound y into a bound on Gaussian parameters. Full details are in Appendix B.2.
With modified Algorithm 1, we can perform the same process of: (1) transforming private data with our public data preconditioner; (2) running an existing private algorithm requiring bounded range parameters; and (3) undoing the transformation. The guarantees of such process are stated in Theorem 2.4. We pick up an extra polylog(1/11-3) ()) factor in our sample complexity.
3 Estimating Gaussian Mixtures
The general process of estimating the parameters of Gaussian mixtures involves separating the mixture components by using techniques like PCA. followed by their isolation, and finally by individual component estimation. In other words, the components are first projected onto a low-dimensional subspace of RRd, which separates the large mixture components from the rest, given that the separation in the original space is enough. The subspace is computed with respect to the available data, that is, it is chosen to be the top subspace of the data matrix, which preserves the original distances between the individual components, but shrinks down each component so that the intra-component distances are small but inter-component distances are large still. This allows us to separate a group of components from the rest. Repeating this process on each group, Gaussian in each group, and estimate its parameters.
narrow down to a single
This has been the high-level idea in a lot of prior work, both non non-private [VW02; AM05] and private [KSSU19]. In our setting, we utilise the public data available in different settings at various points to establish these tasks, while having low cost in the sample complexity of the private data. Given lower bound on the mixing weights (Wmin), we consider two settings in terms of the availability of the public data: (I) when there is very little available ((1 //min)) ); and (2) when there is a lot more
available (dd (/dd//in). The key challenge, as indicated, is accurate clustering of data. Due to space constraints, the complete technical details for this section are deferred Appendix C.
3.1 Public Data Sample Complexity: Õ(1/mmin)
Clustering. This is the more challenging setting of the two because there is not enough public data for techniques such as PCA to give accurate results. We therefore have to rely on the private data for these tasks, and use their private counterparts to get the desired results. The entire procedure is described in detail in Algorithm and its corresponding result in Theorem C.21.
1. Superclustering to reduce sensitivity for Private PCA: The main goal, as in most private estimation tasks, is to reduce the amount of noise added for privacy in order to get better accuracy. This boils down to limiting the sensitivity of the target empirical function of the data. For this, we would like to make sure that the range of the data (to which, we scale the noise) is very tight- - in our context, it means that we would like to have a ball that contains group of mixture components, which are very close to another (a "supercluster") having close to the smallest possible radius. We can limit the data to within this ball, and the sensitivity for the next step would be reduced. To get such ball, we use the public data, and come up with a "superclustering" algorithm (Algorithm 3, Theorem C.14), which does exactly that. This algorithm essentially relies the concentration properties of high- dimensional Gaussians it finds the radius of the largest Gaussian and a point (as the centre) within that Gaussian, and additively grows the ball until it stops finding more points. This gives ball whose radius is an O(k) approximation to the radius of the largest Gaussian. We use this ball to isolate the same components in the private data, and perform private PCA to separate the components within. Getting this tight ball is crucial to reduce the cost of private PCA in terms of the sample complexity compared to the prior work of [KSSU19].
2. Private PCA: We use the well-known private PCA algorithm (Algorithm 4, Corollary C.1 that has been previously analysed in [DTTZ14; KSSU19] for this.
3. Partitioning in low dimensions: The next task is to partition the components in the low- dimensional subspace way that there no overlap. For this, we use an algorithm (Algorithm 5, Theorem C.20) that is similar to the "terrific ball" algorithm of [KSSU19].
8

---

and relies upon the accuracy guarantees of the private PCA step. It uses both public and private data, and looks for a terrific ball within them. If there is one, indicating that there is more than one component in that subset of the datasets, it partitions the datasets (and the components), and queues the partitions for further work. Otherwise, it just adds that subset of the private dataset to the set of discovered clusters.
4. The whole process is repeated on all the remaining partitions.
Here, we provide some intuition about the clustering process. To explain the implication of step (1), we state an informal version of Theorem C.14 only for the first application of Algorithm 3 when it is being run initially on the whole public dataset (of size m))
( (In(k/3) samples from a Gaussian
Theorem 3.1. There exists an algorithm, which if given m 2 O
Wmin
mixture D, it outputs E Fd and R E R, such that with probability 2 1-11- B: (1) BR (c) is non-empty
and contains either 2 1 1 probability mass of a Gaussian component or < 3/ kk and (2) R is an Okk multiplicative approximation of the radius of the largest Gaussian within Bn (c).
The above result essentially says that a supercluster is captured by this algorithm in tight ball. Because with high probability, the points from high-dimensionn Gaussians are highly concentrated, BR (c) would capture all points from both public and private datasets only corresponding to the captured components. Note that this process is performed entirely using the public dataset.
In step (2), where perform private PCA, the radius of the hall we're limiting the points to (Bn (c)) is small, that is, R is an (k) approximation of the radius of the largest mixture component contained within it. Therefore, while computing the noisy empirical covariance, we add relatively small amount of noise that preserves the signal of the points from that Gaussian. This ensures that projection matrix we get preserves the inter-Gaussian distances appropriately, but significantly reduces the Gara-Gaassiian distances. In other words, it preserves the distance between the mean of that largest Gaussian from the means of the rest of Gaussians within Bn (c). while shrinking down all the Gaussians. This helps to isolate at least one component in step (3).
Step (3) relies the following observation about Gaussians, along with the result from step (2). In the case when there are at least two components within Bn (c), step (2) ensures that in the low- dimensional subspace, there is least one Gaussian that is far from all the others. Thus, there must be a ball that contains a large number of points, such that expanding it by a constant factor would not give us any new points (because of the available empty space), and the complement of that expanded ball would contain a large number of points. On the other hand, if there is only one component within BR (c). then projecting it low-dimensional subspace would never give such a ball. Using this observation, Algorithm 5 uses both public and private datasets to determine whether there are one or more components in BR (c), and in the first case, returns a ball that partitions the datasets in a non-trivial way that no component has points in both subsets (a "clean partition"), " or in the second case, returns 1. The number of public samples is enough, such that in the first case, querying the projected public dataset for the existence of such a ball would give us one, but not enough to correctly reject the second case, which why we use the much larger private dataset to confirm whether the positive answer we get by querying the public dataset is correct or not.
In the end, the idea of the clustering algorithm is that in each iteration, it would either isolate both public and private points from a component or partition the datasets further in a clean way. In either case, we move one step closer towards obtaining k clean subsets of the two datasets.
Component Estimation. The final task is estimate the parameters of the isolated components (Algorithm 7, Theorem C.22). In the (c, 5)-P setting (Theorem C.23), we simply apply the private Gaussian learner of [AL21] to each component, which enables us to learn with no dependence in the sample complexity on the range parameters of the Gaussian itself (among other available choices, were the (e, 8)-DP learners from [KMSSU21; KMV21; TCKMS21J, but this one had the joint best sample complexity with [TCKMS21). In the p-2CDP setting (Theorem C.24), we use the Gaussian learner from [KLSU19], instead. Note that this setting, the private sample complexity does depend on the range parameters of the Gaussians, themselves.

---

3.2 Public Data Sample Complexity: O(d/Wmin)
Clustering. In the second setting, clustering is easier in terms of the tasks that need to be done privately - can be done entirely by using the public dataset itself (Algorithm 9, Theorem C.26). This time, the number of public samples enough be able to accurately do PCA on the public data, hence, perform clustering using just the public data itself. This improves the private data sample complexity by removing the term due to private PCA altogether.
1. No supereustering involved: The first change in the clustering algorithm is eliminating the superclustering step.
2. PCA: We essentially get the projection matrix for the top & subspace of the data using non-private PCA, and work within that subspace using the public data further partition the two datasets (and the components).
3. Partitioning in low dimensions: The algorithm for partitioning in the low-dimensional subspace (Algorithm 8, Theorem C.25) is similar to the private partitioner used in the previous case, except that there is no need for privacy in this case.
4. The whole process is repeated on all the remaining partitions.
Component Estimation. Once the components are separated, as before, we individually estimate them using the partitions of the private dataset privately (Algorithm 10. Theorem C.27). In the (e, 6)- DP setting (Theorem C.28), we again use the learner from [AI.21], which ensures parameter-free private data sample complexity in this process. the p-zCDP setting though (Theorem C.29),, we use our own zCDP learner from this text (Theorem 1.1), which uses the public data, as well, to ensure parameter-free private data sample complexity for this process.
Conclusion
We investigate the task of private estimation with access to small amount of public data. Availability of some public data is reasonable assumption in practice, therefore, we demonstrate ways to take advantage of it. We show that significant improvements can be made in terms of the private sample complexity for the tasks of estimating multivariate Gaussians and Gaussian mixtures, given a small amount of public data. For the first problem, we show that even in the case where the public data distribution differs from the private data distribution significantly, we can still estimate the private data distribution using small number of public samples, and improve the private sample complexity by removing any dependence on the range parameters of the private data distribution, which is not possible without any public data. For the problem of estimating Gaussian mixtures under privacy constraints, our new algorithms take advantage of public data to obtain sample complexity bounds, which are significantly better than those of the best known private algorithms for this problem.
Limitations and Open Problems. One question in the context of Gaussian estimation is whether we could obtain computationally efficient, pure DP algorithms that use public data, and match the best known private sample complexity for the problem. Another question is whether we could polynomially improve the private sample complexity, given a much larger, but o(d), number of public samples. A natural question to ask is whether our results for Gaussian estimation could be extended to mean and covariance estimation of heavy-tailed distributions. For estimation of Gaussian mixtures, our work focuses on the case where the public data and the private data distributions are the same, so studying this problem for the case where they are different would be an interesting challenge. Another related question is understanding other instances of clustering problems, where the results of the existing private algorithms could be improved using some additional public data.
Acknowledgments and Disclosure of Funding
AB, GK, and VS are supported by an NSERC Discovery Grant, University of Waterloo Startup Grant, an unrestricted gift from Google, and an unrestricted gift from Apple. AB is supported by Vector Scholarship in Artificial Intelligence.
10

---

References
[AAK21]
Ishaq Aden-Ali, Hassan Ashtiani, and Gautam Kamath. "On the Sample Com- plexity of Privately Learning Unbounded Sigh-Dimensiooal Gaussians", In: Proceedings of the 32nd International Conference on Algorithmic Learning Theory. ALT 21. JMLR, Inc., 2021, pp. 185-216 (cit. on 4).
[ABM19]
Noga Alon, Raef Bassily, and Shay Moran. "Limits of Private Learning with Access to Public Data". In: Advances in Neural Information Processing Systems 32. NeurlPS '19. Curran Associates, Inc., 2019, pp- 10342-10352 (cit. on p. 4). Martin Abadi, Andy Chu, Ian Goodfellow, Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. "Deep Learning with Differential Privacy".. In: Proceedings of the 2016 ACM Conference on Computer and Communications Security. CCS 16. New York, NY, USA: ACM, 2016, pp. 308-318 (cit. on p. 4).
[ACGMMTZI6]
[9DK19]
Brendan Avent, Yatharth Dubey, and Aleksandra Korolova. "The Power of the Hybrid Model for Mean Estimation". In: Proceedings Privacy Enhancing Technologies 2020.4 (2019), pp. 48-68 (cit. on 4).
[AGMRSSSTT21]
Ehsan Amid, Arun Ganesh, Rajiv Mathews, Swaroop Ramaswamy, Shuang Song, Thomas Steinke, Vinith M Suriyakumar, Om Thakkar, and Abhradeep Thakurta. "Public Data-Assisted Mirror Descent for Private Model Training". In: arXiv preprint arXiv:21 12.00193 (2021) (cit. on p. 4).
[AKO1]
Sanjeev Arora and Ravi Kannan. "Learning mixtures of arbitrary Gaussians". In: Proceedings of the 33rd Annual ACM Symposium the Theory of Computing. STOC '01. New York, NY, USA: ACM, 2001, pp. 247-257 (cit. on p. 16).
[AKZHLI7]
Brendan Avent, Aleksandra Korolova, David Zeber, Torgeir Hovden, and Ben- jamin Livshits. "BLENDER: Enabling Local Search with Hybrid Differential Privacy Model". In: 26th USENIX Security Symposium. USENIX Security 17..
USENIX Association, 2017, pp. 747-764 (cit. on p. 4).
[AL21]
Hassan Ashtiani and Christopher Liaw. "Private and polynomial time algorithms for learning Gaussians and beyond". In: arXiv preprint arXiv:2111. 11320111100 (2021) (cit. on pp. 3, 4, 9, 10, 19).
[AM05]
Dimitris Achlioptas and Frank McSherry. "On Spectral Learning of Mixtures of Distributions". In: Proceedings of the 18th Annual Conference on Learning Theory. COLT '05. Springer, 2005, pp. 458-469 (cit. on pp. 8, 27). Raef Bassily, Albert Cheu, Shay Moran, Aleksandar Nikolov, Jonathan Ullman, and Steven Wu. "Private Query Release Assisted by Public Data". In: Proceed- ings of the 37th International Conference on Machine Learning. ICML '20. JMLR, Inc., 2020, pp. 695-703 (cit. on pp. 2, 4).
[BCMNUW20]
[BDKU20]
Sourav Biswas, Yihe Dong, Gautam Kamath, and Jonathan Ullman. "CoinPress: Practical Private Mean and Covariance Estimation". In: Advances in Neural Information Processing Systems 33. NeurlPS '20. Curran Associates, Inc., 2020, pp. 14475-14485 (cit. pp. 48, 49).
[BSWWI9 19]
Mark Bun, Gautam Kamath, Thomas Steinke, and Zhiwei Steven Wu. "Private Hypothesis Selection". In: Advances in Neural Information Processing Systems 32. NeurlPS '19. Curran Associates, Inc., 2019, pp. 156-167 (cit. on pp. 6, 19, 20, 22, 27).
[BM220]
Raef Bassily, Shay Moran, and Anupama Nandi. "Learning from Mixtures of Private and Public Populations". In: Advances in Neural Information Processing Systems 33, NeurIPS '20. Curran Associates, Inc., 2020, pp. 2947-2957 (cit. on 4)
[BNS16a]
Amos Beimel, Kobbi Nissim, and Uri Stemmer. "Private Learning and Sanitiza- tion: Pure vs. Approximate Differential Privacy".. In: Theory of Computing 12.1 (2016), pp. 1-61 (cit. on pp. 2, 4).
[BNS 16b]
Mark Bun, Kobbi Nissim, and Uri Stemmer. "Simultancous Private Learning of Multiple Concepts". In: Proceedings of the 7th Conference on Innovations in Theoretical Computer Science. ITCS '16. New York, NY, USA: ACM, 2016, pp. 369-380 (cit. on 4).

---

[BS16]
Mark Bun and Thomas Steinke. "Concentrated Differential Privacy: Simplifica- tions, Extensions, and Lower Bounds". In: Proceedings of the 14th Conference Theory of Cryptograhhy TCC. '16-B. Berlin, Heidelberg: Springer, 2016 6,
pp. 635-658 (cit. on pp. 17, 18).
[BTGT18]
Raef Bassily, Om Thakkar, and Abhradeep Guha Thakurta. "Model-Agnosee
Private Learning.. In: Advances in Neural Information Processing Systems 31. NeurIPS 18. Curran Associates, Inc., 2018, pp. 7102-7112 12 (cit. on 4). Ilias Diakonikolas, Gautam Kamath, Daniel M. Kane, Jerry Li, Ankur Moitra, and Alistair Stewart. "Robust Estimators in High Dimensions without the Com- putational Intractability". In: Proceedings of the 57th Anmual IEEE Symposium on Foundations of Computer Science. FOCS '16. Washington, DC, USA: IEEE Computer Society, 2016, pp. 655-664 (cit. on 15).
[DKKLMSI6]
[DMNS06]
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. "Calibrating Noise to Sensitivity in Private Data Analysis". In: Proceedings of the 3rd Conference Theory of Cryptography. TCC' '06. Berlin, Heidelberg: Springer, 2006, pp. 265-284 (cit. on pp. 1, 17, 18).
[DRV 1O]
Cynthia Dwork, Guy N. Rothblum, and Salil Vadhan. "Boosting and Differential Privacy". In: Proceedings of the 51st Annual IEEE Symposium on Foundations of Computer Science. FOCS 10. Washington, DC, USA: IEEE Computer Society, 2010, pp. 51-60 (cit. on p. 18).
[TTZZ44]
Cynthia Dwork, Kunal Talwar, Abhradeep Thakurta, and Li Zhang. "Analyze Gauss: Optimal Bounds for Privacy-Preservie Preserving Principal Component Analysis". In: Proceedings of the 46th Annual ACM Symposium on the Theory of Com- puting. STOC 14. New York, NY, USA: ACM, 2014, pp. 11-20 (cit. on pp. 33).
[HKM22]
Samuel B Hopkins, Gautam Kamath, and Mahbod Majid. "Efficient Mean Estimation with Pure Differential Privacy via a Lfn-of-Squaree Exponential Mechanism". In: STOC '22 (2022) (cit. on p. 20).
[HW71]
David Lee Hanson and Farroll Tim Wright. "A Bound on Tail Probabilities for Quadratic Forms Independent Random Variables". In: The Annals of Mathematical Statistics 42.3 (1971), pp. 1079-1083 (cit. on p. 15).
[JE13]
Zhanglong Ji and Charles Elkan. *Differential Privacy Based on Importance Weighting" In: Machine Learning 93.1 (2013), pp. 163-183 (cit. on 4). Aleksandra Korolova, Krishnaram Kenthapadi, Nina Mishra, and Alexandros Ntoulas. "Releasing Search Queries and Clicks Privately". In: Proceedings of the 1 8th International World Wide Web Conference. WWW '09. New York, NY, USA: ACM, 2009, pp- 171-180 (cit. on p. 4).
[KKMN09]
[KLSU19]
Gautam Kamath, Jerry Li, Vikrant Singhal, and Jonathan Ullman. "Privately Learning Digen-iimnsi Distributions" In: Proceedings of the 32nd Annual Conference on Learning Theory, COLT '19. 2019, pp. 1853-1902 (cit. on pp. 4, 5, 19, 27).
[KMSSU21]
Gautam Kamath, Argyris Mouzakis, Vikrant Singhal, Thomas Steinke, and Jonathan Ullman. A Private and Computationally-Ffficc Estimator for Un- bounded Gaussians. 2021. URL: https: /ttp///aaivvv.//.v org/ abs /111 04609
(cit. on pp. 4, 9)..
[KMV21]
Pravesh K Kothari, Pasin Manurangsi, and Ameya Velingker. "Private Ro- bust Estimation by Stabilizing Convex Relaxations". In: arXiv preprint arXiv:2112.03548 (2021) (cit. on pp. 9).
[KRRT21]
Peter Kairouz, Mónica Ribero, Keith Rush, and Abhradeep Thakurta. "(Neally Dimension Independent Private ERM with AdaGrad Rates via Publicly Esti- mated Subspaces". In: Proceedings of the 34th Annual Conference on Learning Theory. COLT 21. 2021, pp. 2717-2746 (cit. on p. 4).
[KSSU19]
Gautam Kamath, Or Sheffet. Vikrant Singhal, and Jonathan Ullman. "Differen- tially Private Algorithms for Learning Mixtures of Separated Gaussians". In: Advances in Neural Information Processing Systems 32. NeurIPS '19. Curran Associates, Inc., 2019, pp. 168-180 (cit. on pp. 3, 8, 17, 27, 34).
12

---

[KV18]
Vishesh Karwa and Salil Vadhan. "Finite Sample Differentially Private Con- fidence Intervals". In: Proceedings of the 9th Conference on Innovations in Theoretical Computer Science. ITCS '18. Dagstuhl, Germany: Schloss Dagstuhl- Leibniz-7entrumm fuer Informatik, 2018, 44:1-44:9 (cit. on p. 4).
[LKO21]
Xiyang Liu, Weihao Kong. and Sewoong Oh. "Differential privacy and robust statistics in high dimensions". In: arXiv preprint arXiv:2111.06577 (2021) (cit.
on p. 4).
[LTLH22]
Xuechen Li, Florian Tramèr, Percy Liang, and Tatsunori Hashimoto. "Large Language Models Can Be Strong Differentially Private Learners". In: Proceed- ings of the 10th International Conference on Learning Representations. ICLR '22. 2022 (cit. on p. 4).
[LVSUW21]
Terrance Liu, Giuseppe Vietri, Thomas Steinke, Jonathan Ullman, and Steven Wu. "Leveraging Public Data for Practical Private Query Release". In: Proceed- ings of the 38th International Conference on Machine Learning. ICML '21.
JMLR, Inc., 2021, pp. 6968 6977 (cit. on p. 4).
[LWAFF21]
Zelun Luo, Daniel Wu, Ehsan Adeli, and Li Fei-Fei. "Scalable Differential
Privacy with Sparse Network Finetuning". In: Proceedings of the 2021 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 21. Washington, DC, USA: IEEE Computer Society, 2021, pp. 5059- 5068 (cit. on p. 4).
[NB20]
Anupama Nandi and Raef Bassily. "Privately Answering Classification Queries in the Agnostic PAC Model". In: Proceedings the 31st International Confer- ence on Algorithmic Learning Theooy. ALT '20. JMLR, Inc., 2020, pp. 687-703 (cit. p. 4).
[PAEGTI7]
Nicolas Papernot, Martín Abadi, Ulfar Erlingsson, Ian Goodfellow, and Kunal Talwar. "Semi-superviee knowledge transfer for deep learning from private training data". In: Proceedings of the 5th International Conference on Learning Representations. ICLR 17. 2017 (cit. on p. 4).
[PCSTE19]
Nicolas Papernot, Steve Chien, Shuang Song, Abhradeep Thakurta, and Ulfar Erlingsson. Making the Shoe Fit: Architectures, Initializations, and Tuning for Learning with Privacy. https: http:///pppeeeveeeeeee wett:/frommdddrrrggeett 2019 (cit. on p. 4).
[PSMRTEI8]
Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Úlfar Erlingsson. "Scalable Private Learning with PATE". In: Pro- ceedings of the 6th International Conference on Learning Representations. ICLR '18. 2018 (cit. on p. 4).
[SST06]
Arvind Sankar, Daniel A. Spielman, and Shang- Hhang-Huatre Teng. "Smoothed Analysis of the Condition Numbers and Growth Factors Matrices". In: SIAM J. Matrix Anal. Appl. 28.2 (2006), 446-476 (cit. on p. 22).
[TB21]
Florian Tramèr and Dan Boneh. "Differentially Private Learning Needs Bet- ter Features (or Much More Data)". In: Proceedings of the 9th International Conference Learning Representations. ICLR '21. 2021 (cit. on p. 4).
[TCKMS21]
Eliad Tsfadia, Edith Cohen, Haim Kaplan, Yishay Mansour, and Uri Stemmer. "FriendlyCore: Practical Differentially Private Aggregation". In: arXiv preprint arXiv:2 10. 10132 (2021) (cit. on pp. 4, 9).
[VW02]
Santosh Vempala and Grant Wang. "A Spectral Algorithm for Learning Mix- tures of Distributions". In: Proceedings of the 43rd Annual IEEE Symposium on Foundations of Computer Science. FOCS '02. Washington, DC, USA: IEEE Computer Society, 2002, pp. 113-123
(cit. on p. 8).
[Yu+22]
Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang. "Differentially Private Fine-tuning of Language Models". In: Proceedings of the 10th International Conference on Learning Representations. ICLR 222. 2022 (cit. on p. 4).
13

---

[YZCL2]]
Da Yu, Huishuai Zhang, Wei Chen, and Tie- Yan Liu. "Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learning.. In: Proceedings of the 9th International Conference Learning Representations. ICLR '21. 2021 (cit. on p. 4).
[ZWB21]
Yingxue Zhou, Zhiwei Steven Wu, and Arindam Banerjee. "Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identification", In: Proceedings of the 9th International Conference on Learning Representations.
ICLR 21. 2021 (cit. p. 4).
[ZZ18]
Anru R. Zhang and Yuchen Zhou. On the Non-asymptotic and Sharp Lower Tail Bounds of Random Variables. 2018. URL: https: / /arxiv. org/ abs 1810. 09006 (cit. on p. 15).
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See Section 2 and Appendices and C.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A] No license in the repository (https:// //tp://gghttbb/t coom/wwisteedui wwweccuiicccooipreeeeeeeeeeeeeee press), however received per-
mission from the authors to use their code.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "_JQKTqJsO6U", "reviewer": "Reviewer_Hm4a", "summary": "This paper studies differential private estimation by taking advantage of some public data. It has been shown that some improvements on the sample complexity can be made when given a certain amount of public data.", "questions": "(1) This paper simply assumes the public comes from a Gaussian distribution. This assumption is too strong in realism. For example, the data in the economy and biology domain can usually be heavy-tailed. Is that possible to extend the current assumption to be a general one?\n\n(2) In Algorithm 1, using only d+1 samples $\\tilde{X}_i$ to obtain the covariance $\\widehat{\\Sigma}$ leads to inconsistent estimation, especially when the dimension d is large. Would this make the output inefficient in a certain case?\n\n(3) Similarly, in Lemma 2.5, how could you make sure the matrix $\\Sigma_Y=\\frac{1}{L}\\widehat{\\Sigma}^{-1/2}\\Sigma\\widehat{\\Sigma}^{-1/2}$ is positive definite?\n\n(4) In Lemma 2.5, is the probability at least 1-$\\beta$ going to 1 with the increasing of the dimension d? In the current presentation, it seems $\\beta$ is an arbitrary constant greater than 0.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The targeted problem is interesting. The idea of taking advantage of public data is good. This paper is easy to follow. \n\nWeakness: \n(1) The organization of this paper should be improved. There are more than thirty pages appendix. The appendix is too long.\n\n(2) This paper simply assumes the public comes from a Gaussian distribution. This assumption is too strong in realism. For example, the data in the economy and biology domain can usually be heavy-tailed. Is that possible to extend the current assumption to be a general one?\n\n(3) In Algorithm 1, using only d+1 samples $\\tilde{X}_i$ to obtain the covariance $\\widehat{\\Sigma}$ leads to inconsistent estimation, especially when the dimension d is large. Would this make the output inefficient in a certain case?\n\n(4) Similarly, in Lemma 2.5, how could you make sure the matrix $\\Sigma_Y=\\frac{1}{L}\\widehat{\\Sigma}^{-1/2}\\Sigma\\widehat{\\Sigma}^{-1/2}$ is positive definite?\n\n(5) In Lemma 2.5, is the probability at least 1-$\\beta$ going to 1 with the increasing of the dimension d? In the current presentation, it seems $\\beta$ is an arbitrary constant greater than 0.\n\n(6) No numerical performance is conducted to demonstrate the effectiveness of the proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XSKys1JaYGQ", "reviewer": "Reviewer_cFnJ", "summary": "The paper studies estimation a setting in which there are two sets of samples available: A \"public\" dataset, and a \"private\" dataset whose elements must be protected with differential privacy. This is studied for two classes of distributions: 1) d-dimensional Gaussians, and 2) mixtures of d-dimensional Gaussians. In case 2) the public dataset must come from the same distribution as the private one, but for 1) it suffices that it is a Gaussian that is \"not too far\" from the same distribution. In both settings it is shown that a small amount of public data (much less than what is needed for estimation) can be used to gear the utility of the private sample, lowering the sampling complexity.\nIn terms of techniques the algorithm for 1) uses the public data to do a parameter range estimation and then apply a private estimator that needs these parameters, a fairly natural (and generally applicable) idea. For 2) the public data is used to improve various steps in an existing algorithm [KSSU19], which is more involved.", "questions": "- Given the prior work, what do you mean more precisely by \"we initiate the study\" in the abstract?\n- In Theorem 1.2, does $\\gamma$ need to be known to the algorithm, or can the sample size somehow be adapted to how well the public distribution approximates the private one?", "limitations": "There is an adequate discussion of limitations. Potential negative societal impacts are unlikely, so not discussed.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Addresses a timely and significant question (how to leverage public data) in two classical settings\n- Adds to the theory of an approach that has shown great promise empirically, but where there is still only a limited theoretical understanding\n- The fact that public samples from an \"almost entirely dissimilar\" distribution can help is intriguing and worth investigating in other settings (could lead to follow-up work)\n- As far as I can tell, the technical contribution is solid (I am not familiar with all the past work on which this builds)\n- The writing is clear and of high quality\n\nWeaknesses:\n- In setting 1) it was already known how to get results of a similar flavor without public data in the setting of *approximate* DP, so the advance lies only in obtaining pure DP, which is mainly of theoretical interest\n- In setting 2) the public data must come from the same distribution as the private data, which is arguably a strong assumption. (I would guess it can be relaxed along the lines of case 1), but this is not worked out in the paper.)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WTalcKPar82", "reviewer": "Reviewer_2n6x", "summary": "This work studies differentially private estimation of Gaussian distributions and mixtures of Gaussians with additional public non-private data. For Gaussian estimation, the authors demonstrate that with a small amount of public data, the number of private samples required no longer depends on the bound on $\\ell_2$ norm of the mean and condition number of the covariance matrix, and instead depends logarithmically on the discrepancy between public and private data distributions. For Gaussian mixtures, similar results are shown when the public and private distributions are the same.", "questions": "Questions regarding the results have already been stated in the Weakness section.\n\nSuggestion on writing:\n- The algorithm description for Gaussian mixture only consists of high-level ideas. Perhaps the technical contribution could stand out more if the authors could highlight a few important intermediate results or claims?\n- Add a brief description about the problem formulation of estimating high-dimensional Gaussian and Gaussian mixtures.", "limitations": "Limitations and negative impacts are adequately addressed.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Understanding the performance of DP algorithms with public data is an important problem both in theory and in practice. The authors proved meaningful results in the context of estimating Gaussian distributions, which is a fundamental statistical estimation problem.\n- For Gaussian mixtures, the improvement on $k$ is significant.\n- The results are comprehensive. The paper studies both standard Gaussian estimation and mixture of Gaussian. Algorithms are designed for various scenarios.\n- Overall, the writing is clear and well structured. The authors did a good job by motivating the algorithm ideas with a simple Gaussian mean estimation problem.\n\nWeakness:\n- For Gaussian estimation: considering that the dependence on dimensionality remains the same, removing the logarithmic dependence on range parameters might seem a weak improvement. The improvement can also be achieved by using approximate DP, which is often used in practice, so the practical implication of the result seems limited.\n- I wonder if at least some of the results can be implied by the range-independent results for approximate DP. For $(\\epsilon, \\delta)$-DP, in the worst case roughly $\\delta$ fraction of the dataset could be released as-is, which might be viewed equivalently as \"public dataset\". It would be great if the authors could discuss the connections.\n\n---\nUpdate: raised my score to 6 because the authors addressed the concern about the relation to $(\\epsilon, \\delta)$-DP.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept