Dimension-free Private Mean Estimation for Anisotropic Distributions
Authors: Yuval Dagan
Michael Jordan
Xuelin Yang
Lydia Zakynthinou
Nikita Zhivotovskiy
Venue: NeurIPS 2024 poster (2024)
Keywords: differential privacy
mean estimation
anisotropic
covariance-adaptive error
Primary area: privacy
TL;DR: We present private mean estimators for anisotropic distributions with dimension-free sample complexity, which we prove is optimal. We also give an estimator under unknown covariance, with a dimension-dependence that is milder than in prior work.

Abstract: We present differentially private algorithms for high-dimensional mean estimation. Previous private estimators on distributions over $\mathbb{R}^d$ suffer from a curse of dimensionality, as they require $\Omega(d^{1/2})$ samples to achieve non-trivial error, even in cases where $O(1)$ samples suffice without privacy. This rate is  unavoidable when the distribution is isotropic, namely, when the covariance is a multiple of the identity matrix. Yet, real-world data is often highly anisotropic, with signals concentrated on a small number of principal components. We develop estimators that are appropriate for such signals---our estimators are $(\varepsilon,\delta)$-differentially private and have sample complexity that is dimension-independent for anisotropic subgaussian distributions.  Given $n$ samples from a distribution with known covariance-proxy $\Sigma$ and unknown mean $\mu$, we present an estimator $\hat{\mu}$ that achieves error, $\|\hat{\mu}-\mu\|_2\leq \alpha$, as long as $n\gtrsim \text{tr}(\Sigma)/\alpha^2+ \text{tr}(\Sigma^{1/2})/(\alpha\varepsilon)$. We show that this is the optimal sample complexity for this task up to logarithmic factors. Moreover, for the case of unknown covariance, we present an algorithm whose sample complexity has improved dependence on the dimension, from $d^{1/2}$ to $d^{1/4}$.

Dimension-free Private Mean Estimation for Anisotropic Distributions
Yuval Dagan
Michael I. Jordan
School of Computer Science
Department of EECS and Statistics University of California Berkeley mi chael_ jaaeelsooonnnnbeeeeeeg edu
Tel Aviv University ydaganStauex. taweeexttau.e. ac. il
Xuelin Yang
Lydia Zakynthinou Department of EECS
Department of Statistics University of California Berkeley xuelin@berkeley. edu
University of California Berkeley
1ydiazak@berkeley edu
Nikita Zhivotovskiy
Department of Statisties
University of California Berkeley
zhivotorskiy@berrelley edu
Abstract
We present differentially private algorithms for high-dimensional mean estima- tion. Previous private estimators on distributions over IR suffer from a curse dimensionality, as they require (2d2/2) samples to achieve non-trivial error, even in cases where ((1) samples suffice without privacy. This rate is unavoidable when the distribution is isotropic, namely, when the covariance is multiple of the identity matrix. Yet, real-world data is often highly anisotropic, with signals con- centrated on a small number principal components. We develop estimators that are appropriate for such signals-our estimators are (e, 8)-differentially private and have sample complexity that is dimension-indepennee inneeeedennneeeennttt for anisotropic subgaussian distributions. Given samples from distribution with known covariance-prox, E and unknown mean μ4, we present an estimator f that achieves error, , |l 1|راار/2 S a, as long as Z tr(2)// /a² + (r(=)))))((aE)) We show that this is the optimal sam- ple complexity for this task up to logarithmic factors. Moreover, for the case unknown covariance, we present an algorithm whose sample complexity has improved dependence on the dimension, from d22/2 d!/1,
Introduction
Machine learning increasingly deployed in real-world settings learn about properties of pop- ulations, both large and small. When the data comes from human populations, it is essential that algorithm design allows inferring properties of populations without revealing potentially sensitive information about specific individuals in the population. That sensitive information can be revealed, inadvertently or adversarially, has been demonstrated numerous ways, including via reconstruction attacks [56]. often targeting sensitive genomic To mitigate the risk of privacy violations in general database theory, Dwork, McSh- erry, Nissim, and Smith 25 proposed the rigorous guarantee of differential privacy (DP), which has
attacks [2 24
as well as bembership-inferent
data 34 55
Authors ordered alphabetically.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

been widely adopted in industry 229 11 33 62, 53 are differentially private are guaranteed not leak too much information about the individuals in database.
and government [30] 1 2 Algorithms that [30
In the machine learning setting, there is tension between differential privacy and inferential and predictive accuracy. It is an ongoing challenge to capture that tension mathematically. in a way that applicable a wide variety of problems and is sufficiently quantitative so as to provide guide for real users and real systems designers, A particularly salient theoretical challenge is to obtain results that capture dimenssion-deeeddenee giveen that machine learning data are often of high dimensionality and involve significant correlation among dimensions, and given that privacy difficult to guarantee in high dimensions, particularly so when there are correlations. Indeed, differentially private inference suffers from a curse of dimensionality- the sample size n that is required to obtain a non-trivial DP learner often polynomial in the dimension of the data.
Significant progress has been made in addressing this challenge in recent years by focusing on a relatively simple inferential task, that of high-dimensional mean estimation. Formally, given data set of points, X (X(1)
X()))= R drawn i.idd. from a multivariate distribution P with unknown mean j E IR4, the goal is to learn μ..
Obtaining low-error private mean estimators in the high-dimensiona regime is not always possible. For example, consider a Gaussian distribution P = NN,, oaLa), where La is the d x d identity matrix. Here, the sample complexity of any private estimator á achieving error || م S a is 2(do /a2do"/a² + do (az)) 339,, where is the privacy parameter The first term corresponds to the non-private sample complexity and the second term to the additional samples required due to privacy. Although both depend on d, note that for non-trivial error a 0.01o /d and 0.1, the non-private term is O(1, whereas the dimension-dependenee persists in the cost of privacy which is
O(vd).
In spite of this lower bound, there is still hope for obtaining better dependence on the dimension in certain cases. This is due to the fact that the lower bound instance assumes that the covariance is isotropic: a multiple of the identity matrix. However, real-world data are far from being isotropic. Often, the signal is concentrated in few directions, while it is significantly weaker in others, as can be revealed via Singular Value Decomposition (SVD). In these cases, there are several examples of non-private estimators for variety of tasks which exploit the structure of the data to achieve lower sample complexity. Specifically for mean estimation of Gaussian distributions, as in our example above, only ((tr(2)/a²) (a²) samples are required [48] (this number of samples is sufficient even for robust estimators under the strong contamination model (50). This bound instance-adaptive, as the trace of the covariance matrix tr(E) equals its upper bound, di d|[E]|2, in the isotropic case, but can be much smaller for anisotropic data. Exploiting the non non-isotrope structure of the covariance matrix is also central to the covariance estimation problem with respect to the operator norm (namely, when the error between the true covariance matrix Σ and its estimate E is measured in terms of ||Σ El|2) A more recent focus is on overparametrized linear regression (9). where again the highly non-isotropic structure of the covariance matrix allows for inference under certain assumptions on the decay of eigenvalues of the covariance matrix. In all the mentioned results, non-private estimation is possible when & d, including even unfinite-dimensione
43 65
Hilbert spaces.
Returning to private estimation, prior work has obtained optimal bounds for learning the mean of high-dimensional (sub) Gaussian distributions in the affine-invariean Mahalanobis distance [18 [3 3 46 13 47). These imply an upper bound for learning the mean in Euclidean distance in the order of n = (d cases. A folklore estimator, hased on [1 achieves
+ d /
which is optimal for isotropic, but loose for anisotropic
22 (dr)drr(E (az)) while ( are the first to focus on the anisotropic case and obtain improved bounds for diagonal covariance, achieving error n = ((er)) + vd/e). Thus, all previous work requires that the sample complexity is at least (((d)), which excludes the high-dimensional scenario we are interested in. We are led to pose the following question.:
We focus on appreximate (c, 6)-P, as opposed to pure (s, DD. However, we omit any dependence on 8 in the introduction.

---

Is it possible to obtain good private mean estimators with sample size that grows slower with the dimension, or is even dimension-indeye independent, when the covariance of the data is far from isotropic? What is the optimal sample complexity in the case of known and unknown covariance?
Question 1.
1.1 Our contributions
First, note that no improved bounds are possible for pure DP, follows directly from the so-called packing technique [32 [8] and specifically applying [18] Lemma 5.1]: any E-DP algorithm which
estimates the mean of Gaussian distribution up to constant accuracy requires Q (d/s) samples.
This negative result motivates us to focus on (E, 5) -ddiffeeret privacy.
In order to make progress, one would like to utilize the fact that when the covariance is far from being isotropic, the data closer to being low-dimensional. Concretely, let Σ be the covariance matrix of d and of 2 ... 2 o its singular values. If the covariance is far from isotropic, there are only few directions with non-trivial variance. For illustration, if 01 =... og = 1. whereas ak+1
dd 1/d, then the distribution is, in some sense, close to being k-dimensional. Here, would like sample complexity to be of order k rather than Vd.
We start by presenting a result in the case where the covariance matrix known. Here, the bound depends only on Ei 0; = tr(E!/?), quantity allowing less contribution from small singular values:
Theorem 1.1 (Upper bound, known covariance, informal). Set E, E (0,1), a a O. Let X (u, 2)" with known covariance. There exists an (e, (), ddiffeerree private algorithm which, with probability 0.99, returns an estimate ji such that || - ull2 S a, and has sample complexity
tr ( (r(E1/2) n Ō
(1/( log(1/0)
as
The first term corresponds to the non-private sample complexity, whereas the remaining two terms due to privacy. The result extends to subgaussian distributions. In the example illustrated above, this bound indeed yields diimensoon-indeeendee
complexity of n Os(k/a² + k/ k/(a)).
We show that the sample complexity of Theorem 1.1 is nearly optimal. Indeed, the first summand is optimal due to [19 Theorem 4]. while the last summand is optimal by a lower bound in the univariate case [4I. We show the optimality of the intermediate summand in ① up to polylogarithmic terms.
Theorem 1.2 (Lower bound, informal). Any (e, 8)-DP algorithm which estimates the mean μ E
[-1, 1]d of a Gaussian up to a with probability 0.99 has sample complexity n S
or log
We now move to the case of unknown covariance. A first approach would be to learn the covariance approximately, namely, find a matrix A such that A ل < CAA, for some C' A instead of Σ in our known-covariance estimator. However, learning such a matrix A privately Another approach would be to learn only the diagonal elements of the covariance [41] this would require n = O ((Vd/e) samples. Below, we obtain a sample complexity whose dependence in the dimension is d²/4, together with a dependendence on the diagonal elements of the covariance matrix:
and then use
requires sample size n = ( 39 40
Theorem 1.3 (Upper bound, unknown covariance, informal). Let parameters e, E (0, 1). Let X (n, Σ)" with unknown covariance Σ. There exists (e, DDP algorithm which, with probability 0.99, returns an estimate ja such that 1| - pll2 I a, and has sample complexity
(r(E) a
E Vlog(1/4)) (18)
dl/4 Ed Σ// log(1/6)
n Ō
ò
-i=1
(2)
vas
a
In general, tr(
and if Σ is diagonal, the two quantities coincide. Our theorem is in fact more adaptable to casier cases of covariance structure. As a special case, when the covariance is diagonal and the singular values exhibit an exponential decay, that is, o, = oie (i-1) then n =
i==1 1// >
tr(2)
((EL/2) (lo((1/8)
(2) ((1/8) samples suffice even under unknown 8
as
covariance.
3

---

1.2 Techniques
Known covariance. A folklore (e, 6DDP algorithm, based on techniques for the univariate case developed by [4T] is to filter outliers by privately estimating each individual coordinate of the mean, Ms, up to an additive error of Ö(E) for all i, clipping any sample point to within that range, and out- putting the mean of the modified data set with added spherical Gaussian noise (0, la/
tr(23) A standard analysis of this procedure yields a sample complexity of n. 2 a² + Va vdu(z) where the dependence on omitted for clarity. For constant e, the folklore estimator achieves privacy for free, that is, the error due to privacy is lower than the error of statistical estimation, when n Z d.
ae
An improvement to this simple analysis, proposed recently by Aumüller et al. [ for matrices of diagonal covariance, suggests adding noise '' (0, (r(5)//))5//2 /e/////((nn))))))))))))))))))/ instead, which introduces more noise in the directions of larger variance. Slightly simplifying their result and additionally
ignoring logarithmic factors in d, and the range of p. their sample complexity is n 2 xx(++ v
(< This estimator achieves privacy for free as long as 2 max |li /1//||= vd), where o² denotes the vector of singular values of Σ. While this removes the dimension dependence in the third term compared the naïve sample complexity, the second term still requires S2( Vd) samples. This due to the first step of the algorithm (inherited from [37,, which performs d independent estimation tasks. In both approaches, the prepprocsssing step a form of coarse mean estimation which ensures that the data will not include outliers, and it is the source of sample-ineffiiiency.
Thus, in our work, we remove outliers, namely vectors too far away from the true mean in one of the coordinates, using only ((1/E) samples, thus completely removing the dependence on d in the final sample complexity bounds. (Indeed, our estimator achieves privacy for free for 2 ||o / |o| ( Next, we generalize the approach of (8) general covariance, rather than diagonal. Finally, we show that the sample complexity is nearly optimal. Specifically:
Our pre- pre-processing is realized by using a polynomial-time filtering algorithm of Tsfadia et al. [63] Given a predicate computed for two data points, so-called FriendlyCore returns a subset X' of the input, such that all pairs of the remaining, unfiltered data points satisfy the predicate. Its sample complexity is ((1/)) for any predicate, hence it has the potential to yield dimensioonindeeendeen bound. For our purposes, X' needs to satisfy some sensitivity properties. follows from our analysis that the filtering should be such that for any two points X X(e E X', X'.1E--//(XU) - -x S Ô (1r())/2)).
The lower bound for (ePD) is an application of the standard fingerprinting 15 28
technique for isotropic Gaussians. A straightforward modification of the technique to anisotropic covariance Σ gives a weaker bound than Theorem 1.2 Instead one needs choose an appropriate set of almosst-sstroopc coordinates whose size scales with tr(E1/2), and apply the technique to that set.
Unknown covariance. Moving to the case of unknown covariance, for illustration, we focus on the simpler, yet fundamental, case where the covariance matrix is diagonal, so that Σ = First, the folklore algorithm described in the covorrcoveei setting, which adds spherical Gaussian noise, does not require knowledge of the covariance but only of its trace. The trace can be privately learned with ((1/)) samples. Second, we note that with
((Vd/E) samples, it is possible to learn each o; up to a multiplicative constant [11. This allows us to apply the algorithm with known covariance from Theorem 1.1 However, the first step in this approach still requires 2(vd) samples.
Our approach is to combine these two methods. We privately learn the largest k R s2n² variances, and their indices. This is done using the sparse vector technique (X and can be achieved with n samples. We use the known-covarianc algorithm to estimate the mean in these top & coordinates, with the same error bound as in the known-covariance setting. For the mean at the remaining coordinates, we use the algorithm that only requires knowledge of the trace of the covariance. The error of the latter estimate is abot v (la((n)), where boo is the vector containing the lowest d k variances. The first observation is that or contains at least entrics as large as ||abot hence, ||ا||| 2 klloobotloo Then, by Hölder's inequality, ||aboolla I V/ I [llohoollooo Substituting yields Abot ss o Ila (=)))) which implics the desired sample complexity bound in Theorem 1.3

---

1.3 Related work
proposed estimators for asymp- totically normal statistics with optimal convergence rates under certain range of parameters. The optimal sample complexity for learning the mean of a Gaussian with known covariance in Maha- lanobis norm under (a, 5)-DP is 2 d/a² + d// d/(oe) + 1og(1/6)/E and has been established in series of works [8 39 46], starting from 41 in the univariate setting. Given the covariance matrix Σ, the Mahalanobis distance between the estimate ji and the true mean fu is defined as: - 14|1=-1---1000011-1-- = 2 (z- 12) When V = La. the Mahalanobis and Euclidean norms coincide. The Mahalanobis distance yields an affine-invariant accuracy guarantee, and || Eاا S immediately implies || /l/2 S / / However, the power of the Mahalanobis guarantee is overshadowed by the fact that even for a = vd, a large sample size, namely n = ((vd), is required, which excludes the high-dimensional scenario we are interested in Furthermore, confidence sets induced by guarantees in the Euclidean distance have the pleasant property of being more easily constructible.
Differentially private Gaussian mean estimation. Smith [58]
Beyond global sensitivity. There are several lines of work within differential privacy which aim to satisfy some form of instance-adaptive accuracy guarantee, as we do. General purpose frameworks which aim to privately estimate a statistic of the data, with error which adapts to "good" data sets, include propose-test-reless Our method follows the same high-level structure propose-test-reless
(22). smooth-sensitivity [52], and Lipschitz extensions [2 42
The latter has been combined with robust estimators to yield optimal private learners for several tasks [13 47.. Even more generally, 6 335 give black-box method which transforms robust estimators to private ones via the sersitivey mechanism SS (see (59] for a discussion on inverse-sensitivity). As there exist optimal robust estimators for the mean of anisotropic Gaussians [50], this would be viable approach, but the volumetric analysis of the transformation involves terms which depend on the dimension. Tsfadia al. [63] propose a filtering method which yields private aggregators whose error adapts to the diameter of the input data set. It is their method that we utilize for our upper bounds. series of works formalize instance-optin opptimllyy for private estimation of empirical [ 37 20 or population [4] 7 quantities. These are all generally well-suited to our setting but either do not adapt to high dimensions, or a direct application would require n 2 /dtr(E)/((es))
Nikolov and Tang [51] study instance-optimaliig specifically for Gaussian noise mechanisms, albeit for data that belong in a bounded convex set. Although this is not the case for Gaussian data, is worth noting that our error rates match those of [5]], which hold for arbitrary distributions over K, when the bounded set is K μ + (d) Privately learning K however would require more samples.
Privately learning nuisance parameters. Karwa and Vadhan 41 learn (a constant multiple of) the variance of univariate Gaussian using = Ô (log(1/6)/e samples. In high dimensions, privately learning the covariance matrix of Gaussian in spectral norm requires 2 d3/2 samples 33 40)
which is more than one needs to learn the mean under known covariance. Brown et al. [13 avoid the bottleneck of private covariance estimation, showing that the sample complexity of Gaussian mean estimation under known covariance with respect to Mahalanobis distance can in fact be matched, even when the covariance is unknown. Their tools also follow the propose-test-releaee approach and could be modified to fit our setting, but the privacy analysis would still require n Z d. Singhal and Steinke (57) learn subspace in which the majority of the data lie, which could be used pre- rre-processing step. followed by projection. However, to recover the set of top ls eigenvectors, they require that there exists a large gap between the two consecutive variances, that is, σ 2 poly(d)ok+1-
The paper by Aumüller et al. 8 is the closest work to ours, aiming to find sample-efficient mean estimators with respect to the Euclidean norm in the anisotropic case. Their work focuses on the less general case of diagonal, (almost) known covariance. The sample complexity of their estimator requires n. 2 Vd, whereas our estimator for the known covariance case ciimension-indepeeddee
Comparison with 8
and, as we prove, optimal. However, the focus [8 is on estimators that satisfy the stricter privacy guarantee of p-zCDP, which forces the need for dimension-dependeee sample size. This is the key contrast with our dimension- free philosophy. As an interesting distinction,
"This limitation is due to the fact that the Mahalanobis distance equalizes the variance across all directions and forces make inferences even in directions where the distribution has particularly small variance.
5

---

Aumüller et al. 8 provide accuracy guarantees with respect to the l. norm (the upper bounds) for slightly more general classes of so-called well-coneentrated distributions, which include subgaussians. It would be interesting to establish optimal private mean estimation bounds with respect to general ep norms. In fact, the optimal non-private sample complexity of Gaussian mean estimation, with matching upper and lower bounds, with respect to general norms has been established only recently, and it depends on the Gaussian mean width of the set induced by the unit dual ball of the norm [19]
Preliminaries
We write [n] = {1......)) ,n), log denotes the natural logarithm, and B3t (c, denotes the d-dimensional Euclidean ball with radius r and center c. We omit c c = O.
We introduce differential privacy here. We say that X, are neighboring data sets if either EjE E [[X]] such that X" = X X or 3j E [X'I]] such that X =XX(( Differentially private algorithms have indistinguishabbe output distributions on neighboring data sets.
Two distributions P, Q over domain W are (e, 0) indistinguishable, denoted by =e, Q, for any measurable subset W SW,
Definition 2.1 ((s, (5,5))-inn d0-indistinguishabiitt
Pr w E W] e Pr w E w]+ + and Wew
Pr Pr E W] S Ef Pr, W1]6. ui
uin
min
Definition 2.2 (Differential Privacy (25) A randomized algorithm A: X - W is (c, 6)-
differentially private if for all neighboring data sets X' we have A(X) 25,5 A(X)). We say that algorithm A satisfies pure differential privacy it satisfies the definition for ò 0.
Differential privacy satisfies several useful properties, such as post-proceste post-proeessing and composition 25 27. For further details and guarantees of standard DP mechanisms, see Section
Our estimators will use the BasicFilter procedure of Tsfadia al. [63], whose detailed definition is presented in Section 3 They provide a framework which allows us to extend an algorithm which private only for "easy" pairs of data sets, to an algorithm that is private for any worst-case pair. "Easy" pairs of data sets are modelled with respect to predicate f between two data points:
Definition 2.3 (f-friendly, Def. 1.1 (3)). Let X be data set over X and let X² + {0. 13 be predicate. We say X is J-friendly for all y E X there exists z E X such that fxx, [=,,y)=1
Definition 2.4 (f-friendly DP, Def. 1.3 (33). An algorithm is called f-friendly (e, 8) for any neighboring data sets X,X', such that X X' is f-friendly, A(X) =e,5 A(X').
Theorem 2.5 (Theorem 4.11 (3).. Let A be an f-friendly (e, 6)DP algorithm. Given data
X, let BasicFilter J 0) and C(X) (X()) =1 Then B(X) A(C(X)) is (2(e - 1)s, 2(( D)-
We assume data are drawn from subgaussian distributions, which include Gaussians. Definition 2.6 (Subgaussian distributions). The random vector X with mean is subgaussian with a p.s.d. covariance matrix proxy E for any À and any v E Rd Ex -n,)) S e
Yo/22 No/2
Lemma 2.7 (Norm of the subgaussian vector [36, (51). Let X (X ,x,,, X()) drawn i.i.d.
from the subgaussian distribution with mean μ and covariance-proxy E. With probability at least 1 8, Σド= X 1ll2 & Vr((E)/n + / Σ (BB)/B/nn.
3 Nearly-matching upper and lower bounds under known covariance
Algorithm [ proceeds two simple steps. The first step filters out outliers so that remaining pairs of data points satisfy the re-scaled distance predicate disL,, and, assuming enough data points remain, the second step releases their empirical mean along with appropriate Gaussian noise.
We retrieve the folklore result, by taking M Ia / tr()), which is known (otherwise, can be easily privately estimated as in Section 4. The filtering then guarantees that all pairs of points
'This is the so-called add/remove model of DP, which will be convenient for our use of prior work. The same privacy guarantees will also hold for the swap model, where dıamm (x) S 1, up constant factors.

---

are within distance /E), and adds spherical Gaussian noise with covariance tr tr( To retrieve the optimal bound, take M Σ. which splits the privacy budget unevenly among coordinates. Then, vtr(>1//2 and the Gaussian noise has covariance /333///)////((""2 as in (8.
Algorithm Private Re-scaled Averaging: AVgM.....
Require: Data set X (x(1) (()) E Raxd Privacy parameters: e, 6 > 0. Failure
probability 8 > 0. Symmetric invertible matrix M. Parameter X.
Let d.st (z, y) = 1 1[ 1/4///1/4(/- 1)ll $ A}.
2: v = BasicFilters [X, dista,2, 0).
3: Let C = (x (j: m-=1)-
4: Compute cic = |C| log(1/4) + 2 where 2 ~ Lap(
5: if [C| = or û then
6: return 1.
7:
return j = C Exeo where 17 N (o, (1.5(122//8)2 52
M
8: procedure BasicFilterrX,, a,
Algorithm 4.3 from
63
9: for j = 1,.... n do
10:
Let 2, Σ (xx))xx( n122
0,
z S 0, if 2 (1/2 - a)n, otherwise.
11:
Sample Vi Bern(p;), where Pj-
1,
Ej (1/2-a)n
12:
return v (Ul....,,,) Un)
Theorem 3.1. Let E (0, 10), 5 E (0,1), 0, (0, Algorithm (c, J)diffferentially private. Let X be data set of size n, drawn from subgaussian distribution with covariance-pa proxy
and mean p.. Given M = E, X 2 /2 1/4) + 2 M |2 log( with probability at least 1 B, Algorithm returns jo such that A| Hll2 S a, as long as
(r() + 1112 1 n1 =
1r(22//) log
12illa log log #
log 33
(3)
a²
OE
aE
where ÿt hides constants and a log factor of the third term multiplied with itself.
The theorem holds more generally for any symmetric invertible M, and À satisfying the assumptions. We sketch the proof of Theorem next. All remaining details are in Appendix
Proof sketch. We start with the accuracy analysis. First we show that the original dataset X passes
through BasicFilter (i.e., C = X) with high probability. It suffices to show that each pair j of
k & [n]. satisfies d, (X()), (()) = 1 with probability 1 - 3/n². Observe that for j * k. M 1/4 M-1/4 (X)) - (()) is subgaussian with mean O and covariance proxy 2M-1/4 1/4 By
Lemma 2.7 and our setting of A, indeed || (x(())))()_)/)))) xx) S A for each pair with probability 1 B/n". We condition on X. With high probability by the CDF of the Laplace distribution and since |C| = = Q(log (1/53)/E). âC = 2(n). Thus, the algorithm does not abort, and returns estimate f.. remains to upper bound the total error of j.. This is at most the error of the empirical mean plus the error due to noise ||/]|22 By Lemma 2.7 with high probability, the former is O(vtr(E)/n) and the latter Õ (M (en)) Substituting M E, and the value for X, the total error becomes Ō( (()tr(Z)/n +
which yiclds the stated sample complexity.
The privacy analysis follows the steps of [63 Claim 3.4]. By Theorem 2.5 it suffices to show that lines 4-7 of Algorithm namely, A, are
DP. Consider neighboring inputs X, X', differing the nith data point w.l.o.g.. g.. that X' X (n), Since || | = 1, by the guarantees of Moreover, they both satisfy ñxx fi'x < X with probability - 5/2 > 1/2. Conditioning on this event for the remainder of the sketch, can fix fix f''x ń < [X], for some value n. If < 0. both runs abort. Otherwise.
the Laplace mechanism, the r.v. îx Åx in Line 4 are (8, ())-inndstinguushabbe
7

---

it suffices to show that Line 7 adds sufficient noise to maintain privacy. By post- prosesrrcessing... since M is not data-dependent, this is equivalent to ensuring that (M-1// 1/4
[ X(), v Ta 55
N( 1 Σ! 1 x (a)a where V - (ol/))((22og(1.222)
()) This is true by the
guarantees of the Gaussian mechanism applied to f(X) M- sensitivity for -fffs..-ffeddl X, XX can be upper bounded by 2 < 2X/h (since 0 < á < [X] by assumption). By composition, A indeed satisfies distM, -fffennndlyy (O(e), (
|X] xO ||X whose b2- -i=1
We show that the sample complexity of Theorem 3.1 is optimal. We briefly explain our lower bound construction here. All remaining details are in Appendix C
Proof Sketch of Theorem 7.2 Let 5 dlagg(o2). Assume w.l.o.g. that of 2 ... 2 oz. Partition
the set of coordinates into buckets S% = {i E [d] or E 01 (2- k 2 [+] Vk [log(d)) [d) and Slog(d)+1 = [d] \ Uke(log(d)) Sx. We have that [xe(log(d)+ Lies, or -|o011 Consider the bucket S which contributes the most to this sum and let øs be the maximum variance in this |a]|=
bucket. It must be that |S| 2 (log(d)+1)org The lower bound of 39 Theorem 6.5] for isotropic
Gaussians, implies that any (s, 0)-rivatee mean estimator which returns, with constant probability, an estimate üs with error for the coordinates in S (note that they are all within a factor of 2), requires n S ( log(d)) = 2 /(ke log((d)) samples. As an estimator for the d-dimensional Gaussian mean restricted to S, would give us such a pss, the statement follows. O
4 Handling unknown covariance
In this section we consider the case of unknown covariance. First, recall that (2d8/2) samples are required to privately learn the covariance matrix in spectral norm [40] which is prohibitive. The lower bound instance is an -isootiisotropic Gaussian, which means that anisotropic distributions may circumvent it. Still, the superlinear dependence on d implies that this approach will yield suboptimal sample complexity for mean estimation. Avoiding private covariance estimation, Brown et al. [13 propose "covariance-aware which scales with the empirical covariance matrix of the data set Σx, as ((1,, X Ex Ex((==n)) for appropriate factor Ai Since adding data-dependent noise can break privacy, pre- pre-processing step is required to ensure that no outliers exist in the data set with respect to the empirical covariance, roughly
private mean estimator which returns the mean with Gaussian noise
ensuring that | 1/211/222 (x(() x6) V Aas, for all j * k E [n]. In our case, to maintain the accuracy guarantee the known-covariance case, the Gaussian noise should be NN0, x² /(="n>²)
x()) S X. Note that 2 tr(E)/|[E)) samples suffice for the empirical covariance to be close to the true covariance Σ in spectral norm [43), so applying the algorithm from [33 could maintain accuracy while still allowing dimension-free sample complexity. Unfortunately, we still cannot use this approach because n 2 d samples are required for the privacy analysis to go though, namely, for neighboring data sets X, X' it holds that (0, 11/3 Re,5 (O, EX( for g zo d/n, which forces us to take n 2 d/e samples. The same is true for the follow-up works of [14 44 which give polynomial time versions of this algorithm with slightly better statistical guarantees.
and all data points should satisfy ||
Luckily, our accuracy guarantee does not require the variance estimate in all directions to be accurate. For example, consider all directions with variance at most |/]|//d. Adding spherical Gaussian noise to these directions maintains a dimension-free error, without requiring tighter estimates for their variance. Thus, on high level, our approach for mean estimation in the unknown covariance case is to identify and estimate many of the top variances k z e2n,, while adding spherical Gaussian noise to the remaining ones.
sample size allows, which turns out to be
We sketch the proof of the following theorem. All remaining details are in Appendix D
Theorem 4.1. Let parameters e, 6 E 10, Let X N(u, E))= with unknown covariance E. Algorithm 2 is (e,8) diffferrenllll private and, with probability 1 B, returns estimate j such
8

---

Algorithm 2 Private Re-scaled Averaging with Unknown Covariance
Require: Data set X = (x ......,..1))).) X(zn)yz E R2nxd, Privacy parameters: e, e > 0. Failure
probability B > 0.
1: Require Ω (d) +o log( /lg(()) log(d)/))
2: Let k +- ==2n// (log" (d) (1)) log (1/63) + log(en)) and <- ((log(d)).
3: Split the dataset into two equal halves: X and X X
4: Split X into
2 groups of size 26. Define X X the r-th sample in the j-th group.
5: for each group j = to m and each dimension i to do
6: Define v2(0) 27 E (x(12r-1) - )²
7: R 4- FindKuhLarges
8: Lop + TopVar (V, and bot + [d] hop
9: for each Iop do
10:
Estimate Σ₁; t- VarianceSum, ,0 (V, (i}) for e' +- V
8' t- B. t k.
11: Compute Shot VarianceSSmm,.. bot))
@VVrehhon 1/12
12: Prop + AVgMA (X (xmea((eop))) where M diag( dlig((Eis)) (eiisheophp0. x
13: jbot + A.gM, m (bort) where M = -á, x = ((VSSbnt)
14: return (Prop: Phot)
that |1 - -4ll2 S a, as long as
E
Vlog? E/
21/4 log 5/4 (t)) log(d)
log
n ===0 log"(d)=
tr(E) a²
+
(4)
as
vas
where the symbol Q hides multiplicative logarithmic factors in 1/3.
Next, we describe Algorithm 2 and introduce some of its subroutines along with their guarantees. All omitted proofs are in Appendix D Our algorithm receives data set X()).... X(n), where each ( is a d-dimensional vector distributed as (N, )). The algorithm starts by splitting the dataset into [2(2 groups each of size 26, where O(lg d). Denote the elements of each group j by X (j.1)
(.20), Within each group j. for each coordinate i, we compute an estimate v,()) for Σ.: v.(1) 1/2 Σ
2()22r))² For convenience, we define what means for the v-()) variables to provide good estimate of the set of {Ea)ee(4
(x(12+)) (
28
Definition 4.2. Given variances E1....., Edd and given a set of estimates, V = (v,5)) j(mm)
we say that V is valid if {j: Vi E [d], En/2 < v.61 < 2E 2 4m/5.
For ease of notation, we assume that Σ diag( a We start from the accuracy analysis. Assume satisfies the sample complexity bound of Eq. (4 By Chernoff bound, since (log(d)) and m = (1 18)),, with probability 1 - B, V is valid. We use the estimates v2(6) inputs to private procedures: FindKthLarges Meeaaarrreeaaianeeeee( Variance(V, k)) (compute the k-th largest variance up to multiplicative constant), VarianceSumiV, /) (compute the sum Σier of up to multiplicative constant), and TopVar' V, R, ke) (identify the indices of the at-most-k largest variances o²² > R). The first two tasks can be implemented by the Stable Histogram algorithm [6 m 52ooog (1))/e)) Top can be implemented via the Sparse Vector technique 226 m = 2 /k / (1g)) log(d/3)/ VEn). Both are satisfied for the given m = n//2f. With these procedures, the algorithm privately learns the k-th largest variance R. identifies the top & coordinates, Loop. and learns estimates [o that are accurate up to a multiplicative constant.
Proof sketch of Theorem 4.1
Next, estimate the mean f in the coordinates Toop separately from I [d] Tupp. denoted by Prop and phot, respectively. Denote vector doop = (fa) and obot. = (eo.) To estimate Ptop. Algorithm given input vectors (1), ..... X (n), restricted to coordinates Irop-

---

diagonal matrix M, with Ma = o7 (assume that the rows and columns of M are indexed by Лtpp). and A Rs / / returns estimate frop with error (since the sample complexity of Eq. is larger than the one required by Theorem 3.1
To estimate Abot, we do not know the variances, so we use the naive approach. We first call VarianceSum once again, to provide an estimate i of ||ahotll2 up to a multiplicative constant. Given f, we again call Algorithm 1 now for a (d - k) k)ddimensional estimation problem. Given input vectors ((), ..(.,, restricted to coordinates hot. matrix M = La-k, and X = f, Algo- rithm returns an estimate jbbot with error a as long as n = ō bool///
By Hölder's inequality, blo 1l2 S || l S <V|lorl11 bot ol By the guarantees of Top Var and FirdKthLareest Variance, boo loe is smaller than the k-th largest variance of up to a multi- plicative constant, which, in turn, must be smaller than ||=|| //k. Substituting this above, we obtain that it suffices for the stated sample complexity to additionally satisfy n $2( (D(Vd]le (Vkk)))) which can be confirmed by substituting the definition for k.
The privacy guarantee follows directly by composition of O(1) (c, 8)-DP mechanisms.
Remark 1. We note that the sample complexity of Algorithm 2 in fact depends on the decay of the diagonal elements of Σ, and can yield improved bounds for easier instances. In particular, the error of the algorithm due to privacy is in the order of I|or [en) + /Too | hoor (lz) Thus, if a follows an exponential decay, i.e., the i-th largest variance is proportional to e e(--1), or all hhot. variances are smaller than ||a|||/ /d, then it suffices to learn only the top k = log( (d) variances and the error almost matches that of the known-covariance case, up to additional logarithmic factors in 1/5. Moreover, identifying easier instances is possible by computing private histogram over log(d) ()
buckets of the form (2 1+ |||or loo given (log(e)) (c)/E) samples [16 41).
Thus, we can determine special cases where the decay of Σ allows us to achieve the optimal rate of Theorem I.T even with unknown diagonal covariance. But without further assumptions, our algorithm has sample complexity that depends on d//4, The question of the optimal sample complexity for mean estimation in the case of unknown covariance, which captures anisotropic distributions, remains open.
5 Conclusion and future work
We present (e, -iiifferrnttillyy private mean estimators for subgaussian distributions with error as measured in Euclidean distance, with high probability, as long the sample size is Ô (tr(E)/a² + tr(x1/2) (ae)). The sample complexity is thus dimension-independee covariance is highly anisotropic. We show that this is the optimal sample complexity for this task up to logarithmic factors. We also present an algorithm in the more challenging case of unknown covariance, whose sample complexity has improved dependence on the dimension, that is, d'/4.
when the
In the known covariance case, the dependence on 1og(1/8) could possibly be decoupled from the tr(E)/2)/((a)) term. This is an artifact of the Gaussian noise added for privacy and can possibly be avoided using mean estimators based on the exponential mechanism, as in the spherical Gaussian case 13 35), but the volumetric arguments involved in their analysis incur factors dependent on which seem hard to overcome.
A more interesting direction for future work is the case of unknown covariance. We can determine special cases where the decay of Σ allows us to achieve the optimal rate of Theorem [.] with unknown diagonal covariance. What is the appropriate norn in which one needs learn Σ for the current known-covariann covariance approach to be accurate, and how many samples are needed for this task privately? More generally, the optimal sample complexity of mean estimation in the unknown (even diagonal) covariance case for anisotropic distributions (possibly achieved by an algorithm which doesn't 't follow the same structure) is an open question.
Acknowledgments and Disclosure of Funding
We thank NeurIPS reviewers for suggestions on improving the clarity of this manuscript. This work was done while both LZ and YD were postdoctoral fellows in the Simons Institute for the Theory
10

---

of Computing, funded by FODSI. We also wish to acknowledge funding from the European Union 21--2222-SYG-OCEA---OOO
Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.
References
[1] John M. Abowd. The US Census Bureau adopts differential privacy. In ACM International Conference Knowledge Discovery & Data Mining, KDD '18, pages 2867-2867, 2018.
[2] John M. Abowd, Robert Ashmead, Ryan Cumings-Menon, Simson Garfinkel, Micah Heineck, Christine Heiss, Robert Johns, Daniel Kifer, Philip Leclere, Ashwin Machanavajhhal, Brett Moran, William Sexton, Matthew Spence, and Pavel Zhuravlev. The 2020 Census Disclosure Avoidance System TopDown Algorithm. Harvard Data Science Review, Special Issue 2, June 2022.
[3] Ishaq Aden-Ali, Hassan Ashtiani, and Gautam Kamath. On the sample complexity of privately learning unbounded high-dimensional Gaussians. In Proceedings of the 32nd International Conference on Algorithmic Learning Theory, ALT '21, March 2021.
[4] Apple Differential Privacy Team. Learning with privacy at scale. Apple Machine Learning Jour- nal, 1(8), 2017, httpp::/ dewelooprr apple www.moommmmeeeeee com//nl-reesaaccchpaaeeeeee learning with www.hhh--ivvay-at-scalala. pdf
[5] Hilal Asi and John C. Duchi. Instance-optimality differential privacy via approximate inverse sensitivity mechanisms. In Advances in Neural Information Processing Systems 33, NeuRIPS
200 pages 14106-14117, Dec 2020.
[6] Hilal Asi, Jonathan Ullman, and Lydia Zakynthinou. From robustness privacy and back. In Proceedings of the 40th International Conference on Machine Learning, ICML '23, pages 1121-1146. PMLR, Jul 2023.
[7] Hilal Asi, John C. Duchi. Saminul Haque, Zewei Li, and Feng Ruan. Universally instance- optimal mechanisms for private statistical estimation. In Proceedings of 37th Conference on Learning Theory, COLT 24, pages 221-259. PMLR, Jul 2024.
[8] Martin Aumüller, Christian Janos Lebeda, Boel Nelson, and Rasmus Pagh. PLAN: Variance- 08745.
aware private mean estimation, June 2023.
// http://wwwwww...aa/aoo///sao
[9] Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117748):30033-30077 2020.
[10] Amos Beimel, Hai Brenner, Shiva Kasiviswanathan, and Kobbi Nissim. Bounds the sample complexity for private learning and private data release. Machine Learning, 94:401-437, 2014. doi: 10.1007/s 00900/01110-4-1-4444400
[11] Andrea Bittau, Úlfar Erlingsson, Petros Maniatis, Ilya Mironov, Ananth Raghunathan, David Lie, Mitch Rudominer, Ushasree Kode, Julien Tinnes, and Bernhard Seefeld. PROCHLO: Strong privacy for analytics in the crowd. In ACM Symposium on Operating Systems Principles, SOSP 17, pages 441-459, 2017.
[12] Jeremiah Blocki, Avrim Blum, Anupam Datta, and Or Sheffet. Differentially private data analysis of social networks via restricted sensitivity. In 4th ACM Conference on Innovations in Theoretical Computer Science, ITCS 133 pages 87-96, 2013.
[13] Gavin Brown, Marco Gaboardi, Adam Smith, Jonathan Ullman, and Lydia Zakynthinou. Covariance-aware private mean estimation without private covariance estimation. In Advances in Neural Information Processing Systems 34, NeurIPS '21, pages 7964,7964,.0 2021.
[14] Gavin Brown, Samuel B. Hopkins, and Adam Smith. Fast, sample-efficient, affine-invariant private mean and covariance estimation for subgaussian distributions. Proceedings of the 36th Conference on Learning Theory, COLT '23, pages 5578-5579. PMLR, Jul 2023.

---

[15] Mark Bun, Jonathan Ullman, and Salil Vadhan. Fingerprinting codes and the price of approxi- mate differential privacy. In ACM Symposium on the Theory of Computing, STOC '44, pages 1-10, 2014.
[16] Mark Bun, Kobbi Nissim, and Uni Stemmer. Simultaneous private learning of multiple concepts. In Proceedings of the 7th ACM Conference on Innovations in Theoretical Computer Science, ITCS 16, pages 369-380. ACM, 2016.
[17] Mark Bun, Thomas Steinke, and Jonathan Ullman. Make up your mind: The price of online queries in differential privacy. In Proceedings of the 28th Annual ACM-SIAM Symposium on Discrete Algorithmss, SODA $17, pages 1306-1325. SIAM, 2017.
[18] Mark Bun, Gautam Kamath, Thomas Steinke, and Zhiwei Steven Wu. Private hypothesis selection. In Advances in Neural Information Processing Systems 32, NeurlPS '19, pages 156-167, 2019.
[19] Jules Depersin and Guillaume Lecué. Optimal robust mean and location estimation via convex programs with respect to any pseudo-norms. Probability Theory and Related Fields, 183(3): 997-1025, 2022.
[20] Travis Dick, Alex Kulesza, Ziteng Sun, and Ananda Theertha Suresh. Subset-based instance optimality in private estimation. In Proceedings of the 40th International Conference on Machine Learning, ICML'23, JMLR, 2023.
[21] Irit Dinur and Kobbi Nissim. Revealing information while preserving privacy. In Proceedings of the 22nd ACM Symposium on Principles of Database Systems, PODS '03, pages 202-210. ACM, 2003.
[22] Cynthia Dwork and Jing Lei. Differential privacy and robust statistics. In Proceedings of the 41st ACM Symposium on Theory of Computing, STOC '09, pages 371-380. ACM, 2009.
[23] Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Founda- tions and Trends in Theoretical Computer Science, 93--4:2111-00) 2014.
[24] Cynthia Dwork and Sergey Yekhanin. New efficient attacks on statistical disclosure control mechanisms. In Annual International Cryptology Conference, pages 469-480. Springer, 2008.
(25] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Conference on Theory of Cryptography, TCC '06, pages 265-284, 2006.
[26] Cynthia Dwork, Moni Naor, Omer Reingold, Guy N. Rothblum, and Salil P. Vadhan. On the complexity of differentially private data release: efficient algorithms and hardness results. In Proceedings of the 41 ACM Symposium on Theory of Computing, STOC '09, pages 381-390. ACM, 2009.
[27] Cynthia Dwork, Guy N. Rothblum, and Salil Vadhan. Boosting and differential privacy. In Proceedings of the 51st IEEE Annual Symposium on Foundations of Computer Science, FOCS '10, pages 51-60. IEEE, 2010.
[28] Cynthia Dwork, Adam Smith, Thomas Steinke, Jonathan Ullman, and Salil Vadhan. Robust traceability from trace amounts. In Proceedings of the 56th IEEE Annual Symposium on Foundations of Computer Science, FOCS 15, 2015.
[29] Úlfar Erlingsson, Vasyl Pihur, and Aleksandra Korolova. RAPPOR: Randomized aggregatable privacy-preserving ordinal response. In ACM Conference on Computer and Communications Security, CCS 14, 2014.
[30] Samuel Haney, Ashwin Machanavajjhala, John Abowd, Matthew Graham, Mark Kutzbach, and Lars Vilhuber. Utility cost of formal privacy for releasing national employer-employee statisties. In Proceedings of the 2017 ACM International Conference on Management of Data, SIGMOD 17, pages 339--355. ACM, 2017.
12

---

[31] Moritz Hardt and Guy Rothblum. A multiplicative weights mechanism for privacy privacy-preserneeeee preserving data analysis. In IEEE Symposium on Foundations of Computer Science, FOCS 10, pages 61-70, 2014.
[32] Moritz Hardt and Kunal Talwar. On the geometry of differential privacy. In Proceedings of the 42nd ACM Symposium on Theory of Computing, STOC '10 2010.
[33] Florian Hartmann and Peter vacy for federated learning, distributed otgpaaaaa....a...ooooooo.eeeaaaa httpa///reeearcchhgggoo//o/go google/blog/
Kairouz.
Distributed differential pri-
2023.
https://research.
[34] Nils Homer, Szaboles Szelinger, Margot Redman, David Duggan, Waibhav Tembe, Jill Muehling, John V. Pearson, Dietrich A. Stephan, Stanley F. Nelson, and David W. Craig. Resolving individuals contributing trace amounts of DNA highly complex mixtures using high-density SNP genotyping microarrays, PLoS Genetics, (8)ee100010) 2008.
[35] Samuel B. Hopkins, Gautam Kamath, Mahbod Majid, and Shyam Narayanan. Robustness implies privacy in statistical estimation. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing, STOC '23, page 497-506, 2023.
[36] Daniel Hsu, Sham Kakade, and Tong Zhang. A tail inequality for quadratic forms of subgaussian random vectors. Electronic Communications in Probability, 17:1 - 6, 2012.
[37] Ziyue Huang, Yuting Liang, and Ke Yi. Instance-optimal mean estimation under differential privacy. In Advances in Neural Information Processing Systems 34, pages 25993-26004, 2021.
[38] Peter Kairouz, Sewoong Oh, and Pramod Viswanath. The composition theorem for differential privacy. In Proceedings of the 32nd International Conference on Machine Learning, ICML '15 pages
2015.
[39] Gautam Kamath, Jerry Li, Vikrant Singhal, and Jonathan Ullman. Privately learning high dimensional distributions. In Proceedings of the 32nd Annual Conference on Learning Theory, COLT 19. JMLR, 2019.
[40] Gautam Kamath, Argyris Mouzakis, and Vikrant Singhal. New lower bounds for private estima- tion and generalized fingerprinting lemma. Advances in Neural Information Processing Systems 35, NeurlPS '23, pages 24405-24418, 2022.
[41] Vishesh Karwa and Salil Vadhan. Finite sample differentially private confidence intervals. In Proceedings of the 9th Conference on Innovations Theoretical Computer Science, ITCS 118 pages 44:1 44:9, 2018.
[42] Shiva Kasiviswanathan, Kobbi Nissim, Sofya Raskhodnikova, and Adam Smith. Analyzing graphs with node differential privacy. In 10th IACR Theory of Cryptography Conference, TCC '13 pages 457 476.. Springer, 2013.
[43] Vladimir Koltchinskii and Karim Lounici. Concentration inequalities and moment bounds for sample covariance operators. Bernoulli, pages 110-133, 2017.
[44] Rohith Kuditipudi, John C. Duchi, and Saminul Haque. A pretty fast algorithm for adaptive private mean estimation. In Proceedings of the 36th Conference on Learning Theory, COLT '23, pages 2511-2551 PMLR, Jul 2023.
[45] Beatrice Laurent and Pascal Massart. Adaptive estimation of a quadratic functional by model selection. Annals of Statistics, pages 1302-1338, 2000.
[46] Xiyang Liu, Weihao Kong, Sham Kakade, and Sewoong Oh. Robust and differentially private mean estimation. In Advances in Neural Information Processing Systems 34, NeurlPS '21, pages 3887-3901, 2021.
[47] Xiyang Liu, Weihao Kong, and Sewoong Oh. Differential privacy and robust statistics in high dimensions. In Proceedings of the 35th Annual Conference on Learning Theory, COLT pages 1167-1246. PMLR, Jul 2022.
13

---

[48] Gábor Lugosi and Shahar Mendelson. Mean estimation and regression under heavy-tailed distributions: survey. Foundations of Computational Mathematics, 1965):145--1100, 2019.
[49] Audra McMillan, Adam Smith, and Jon Ullman. Instance-optimal differentially private estima- tion, 2022. hattps://arxi:i waxxi.oorgr. hts//bbs/22111 15819
[50] Arshak Minasyan and Nikita Zhivotovskiy. Statistically optimal robust mean and covariance estimation for anisotropic Gaussians, 2023. https:///axiis httpaarwwww....graaa//a//a 2a0t..
[51] Aleksandar Nikolov and Haohua Tang. General Gaussian noise mechanisms and their optimality for unbiased mean estimation. 15th ACM Conference on Innovations in Theoretical Computer Science, ITCS '24, 2024.
[52] Kobbi Nissim, Sofya Raskhodnikova, and Adam Smith. Smooth sensitivity and sampling in private data analysis. In Proceedings of the 30th ACM Symposium Theory of Computing, STOC, STOC *07, pages 75-84, 2007.
[53] Ryan Rogers, Subbu Subramaniam, Sean Peng, David Durfee, Seunghyun Lee, Santosh Kumar Kancha, Shraddha Sahay, and Parvez Ahammad. LinkedIn's audience engagements API: A privacy preserving data analytics system at scale, 2020. https://arxivvorg//aae 05839
[54] Aaron Roth and Tim Roughgarden. Interactive privacy via the median mechanism. In Proceed- ings of the 42nd ACM Symposium on Theory of Computing, STOC '10, pages 765-774. ACM, June 2010.
[55] Sriram Sankararaman, Guillaume Obozinski, Michael I. Jordan, and Eran Halperin. Genomic privacy and limits of individual detection in pool. Nature Genetics, 41 2009.
[56] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In IEEE Symposium on Security and Privacy (S&P), Oakland, 2017.
[57] Vikrant Singhal and Thomas Steinke. Privately learning subspaces. In Advances in Neural Information Processing Systems 34, NeurIPS '21, pages 1312-1324, 2021.
[58] Adam Smith. Privacy-preserving statistical estimation with optimal convergence rates. In Proceedings of the 43rd Annual ACM Symposium on the Theory of Computing, STOC '11, pages 813-822. ACM, 2011.
[59] Thomas Steinke. Beyond global sensitivity via inverse sensitivity. DifferentialPrivacy.or Sept 2023. http:///wwaaat..aapprrarrr//r///
[60] Thomas Steinke and Jonathan Ullman. Interactive fingerprinting codes and the hardness of preventing false discovery. In Proceedings of the 28th Annual Conference on Learning Theory, COLT 15, 2015.
[61] Thomas Steinke and Jonathan Ullman. Tight lower bounds for differentially private selection. In Proceedings of the 58th IEEE Symposium on Foundations of Computer Science, FOCS '17, 2017.
[62] David speed library for training PyTorch vacy.
Tastuggine and Ilya Mironov.
Introducing Opacus: A high- models with differential pri-
https: facebook. com/blog/ prrroducinn-oppccss. a--ighs saa--igh-sppap sin-ssped---e sioodd-ibbrry-foo training swwraining--ytorcc.. pytorch- with-diifferomtaal--iiiee
Facebook AI Blog, 2020.
[63] Eliad Tsfadia, Edith Cohen, Haim Kaplan, Yishay Mansour, and Uri Stemmer. FriendlyCore: Practical differentially private aggregation. In Proceedings of the 39th International Conference on Machine Learning, ICML. '22, pages (828-21863. PMLR, Jul 2022.
[64] Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In IEEE Computer Security Foundations Symposium, CSF '18 pages 268-282, 2018.
[65] Nikita Zhivotovskiy. Dimension-free bounds for sums of independent matrices and simple tensors via the variational principle. Electronic Journal of Probability, 29:1-28, 2024.
14

---

A Standard DP mechanisms and properties
Definition A.1 (Laplace distribution). For v 2 0, let Lap(t) denote the Laplace distribution over R.
which has probability density function p(=) = 2n
_ -///// From the CDF of the Laplace distribution,
we get that Piz=Lap(()) 2 v [log(1 3.
Definition A.2 (Laplace Mechanism, 25 Let f X - R. data set over X, and privacy
parameter c. The Laplace Mechanism returns
f(x)= = f(x) + Lap(v), where u A,/E
and A max (S(X) (((X)-((XX)) f(X')) X-xx
Lemma A.3 (25) The Laplace Mechanism is E-differentially private.
Definition A.4 (Gaussian Mechanism, (25 Let f X - TR.,, data set X over X, and privacy
parameters e, The Gaussian Mechanism returns
f(x) f(X) + NOO Ia)) where D
and A
max [If(X) f(X) is the global éz-sensitivity of f. X-X
Lemma A.5
25
The Gaussian Mechanism (s, -ddiffereent private.
Differential privacy maintained under post-processing and degrades mildly under composition.
Lemma A.6 (Composition, 25 27 38 ). Let M be an adaptive composition of M Mr,, that is, on input X, M() = M1 X, M M2(X, M1 (X)))). Then
1. (Basic composition) M1...., Mr are (E1, 81). ... (sT, diffefrrrnttl private respec- tively, then M is (e, 6) differrentially private for Σt Et and ö Σt= 82.
2. (Advanced composition) Let & > 8 E [0. 1] for E TT....,T,, and & [0, 1]. If My..., MT are (81, 6,)...., (ET, 8τ) drifeefereeti private respectively, then M (Ez, 8-
Dt 1= 6i)-differyntially private where Eg is given by:
c 1)ee
Σ
2 log
E +1
7
Fact A.7 (Fact 2.17 633 reduced to pure DP). Let Y 2e Y' random variables over y and let the event E C y be such that Pr[Y E Pr\Pr E E] q. Then Y Rely Y
B Omitted details of Section 3
We state here again the general theorem which holds for any M.
Theorem B.1. Let & E (0, 10), 5 E (0,1),a a > 3 E (0. 1) Algorithm 7 (c, ( -)dffferrenti private. Let X be data set of size n, drawn from a subgaussian distribution with covariance-pron
E and mean p. Given M E, X /2 1/4) + / 2 with probability at least 1 3. Algorithm 1
1/4 | log( }3
returns such that Í Hl12 S long
20
tr(E)+ + 11 E log 2
A (Vir(MrrE)) + /1111//112 log
1
v
log
log
3g
aE
'We require to be smaller than some constant, due to approximations we take in the privacy analysis. The theorem may hold for 10 but did not optimize the choice of constant, this range already wide.
15

---

below and for some universal constant C. I is known, choosing M Σ and substituting X, the sample complexity becomes
tr(2) + || log n 2
tr(E log
+
a²
ae
21 log
log
| log 1/ log
log
log
(6)
Although the theorem holds for any M, X, choosing M gives us the optimal bound
Remark 2. If M is such that 2 3 M, we may assume without loss of generality that M is invertible. Indeed, this not the case, then we know that the distribution of the data is supported on lower-dimen liverdimensional subspace along with its mean μ. Using M, we can project onto this subspace. In this context, we can refocus our analysis on the scenario where M an invertible matrix.
Computational complexity We note that Algorithm has time complexity O(n²), which can be further improved to On log rt) as in 633.
The guarantees of Theorem B. follow by combining Theorem B.3 and Theorem
B.5
re-scaling parameters E, 6, 3 with appropriate constants.
B.1 Accuracy analysis
We start with the accuracy analysis. We first prove that for subgaussian data sets, all data points pass the filter with high probability.
Lemma B.2. Let X = (X(1,, ((.,,.,),()))))
X()) be data set drawn from subgaussian distribution with covariance proxy E. Let B E (0, 1), M invertible matrix and some X 2 (2 M
log(n/ given as inputs to Algorithm Then the BasicFilter procedure oulputs C =X, with probability B.
Proof. It suffices to show that in BasicFilter we have Pj for j E [n]] (so that by and thus C. =X)). For each j, [n], we want to show aist (x (X))), (k)) with probability at least 1 B/n2.. For k. it trivial. What is left is to show for j # k.
|| 1/M-1/((X)) )2 S 221(MM-/AMMMMV)) + 2 2 M / (7)
with probability least 3/n2. First observe that - (A) is subgaussian vector independent of ( with mean - and covariance proxy Σ. Hence. ( - (x) is subgaussian vector with mean 0 and covariance proxy 2 and so M -1 - X (A+) is subgaussian with mean zero and B/n².
covariance proxy 2M -M
By Lemma 2.7 with probability
|M-1/4(X00) x( 2tr 1/4) + 2 2/1M-1/AMMM//1 log(n//
Then we union bound all n(n 1) pairs of j. E [n].j + such that Eq. 7 holds with probability of at least
B.
'For intuition, consider adding noise (0, of to each coordinate be distributed unevenly among coordinates.) We would like to minimize the & norm of this noise, which is approximately Σi= of The average sensitivity of each coordinate of Gaussian is a;, so to achieve total privacy loss e, by advanced composition, we require the c,'s satisfy Σd of/C < E², Solving problem, we find that e x |||a||110,. which corresponds noise \000, 2yy1/22) for A ||]| The same reasoning be applied to the of the Mahalanobis error metric 13] where adding noise (O A for
clear the privacy budget should
optimization
d A x 2a. gives us the optimal bound. Here the minimization objective is roughly x:d - /of so the optimal solution requires
16

---

Theorem B.3. Let > 0, E (0.1). > 0, E (0,1). Suppose 2 2log(1/68)/E. Let
X = "x.,,..., x(1) be a data set drawn from subgaussian distribution with mean and
covariance proxy E. Then, given invertible matrix M and X 2 /
2/2 1/4 22 log(n/3), Algorithm
1
with probability 7 B, returns ji such that
/tr(E)
2
2.
log(1/B)
1l Hll2
Vn
Vn
log(1 25/0JA
(VirMrri2)) + + 221/2//3122 1og(1/3)
en
Proof. Let pc. p. be the sample mean of and X, respectively. By the triangle inequality, we decompose it into
1 rll2 - 11 Malla + Ilec - Mxll2 + Ilxx 1ll2
(8)
By Lemma B.2
C = X with probability 1 - B. Condition on this event for the rest of the proof.
Then, NC 1og(1/6) satisfies àec 0.5n 0 with probability B/22 because
log(1/8)
log (1/6) log(1/63)
log(1/B) 8
Pr
0.5m
Pr
Pr
8
by Definition A. and our assumption that 2 2 log (1/63) Conditioning this assumption, do not abort and with probability 3, by Lemma 2.7
4v (12
(Vo((ML/2) + / (1/3)
1 Hall2 ||lla -
en
Again, by Lemma 2.7 with probability 1- 3,
n 1
tr Σ
2 log(l//)
M1xx - 2l2
(
Vn
j=1
Moreover, since C' X, it holds that μx p.c. Combining these results into Eq. (8. the algorithm does not abort and we retrieve the stated error bound, with probability 1 3.3
B.2 Privacy analysis
We now move the privacy analysis.
Lemma B.4. In Algorithm 7 fic < [C}, with probability 1 5/2.
Proof. It follows that by Definition A.I
Pr Jñc 2 |C|] Pr [|C| - log(18)))
|C]] = Pr 2 log(1/8)/E]= = =2
Note that this holds regardless of whether C' is distsy, or whether X is subgaussian.
The privacy analysis follows the steps of [63 Claim 3.4).
Theorem B.5. Let E E (0,1/2), 8 E (0, 1/2). For any input parameters M, X, Algorithm 0 satisfies (21s, e 108)
Proof. It suffices show that lines 4-7 of Algorithm are dirienl (6',8')-DP, such that by Algorithm is (2(e='- - 1)e', 2 +2(e" -1)8')-DP.
Theorem 2.5
Denote lines 4-7 of Algorithm 1 algorithm A. Consider neighhoring inputs X, X' such that X is disty, frrienly Assume without loss of generality X' X \ x so that X = || Let A(X), A(X') represent the outputs of two independent executions of A and let Nx, Nx be the random variable in line 4 of the algorithm. We want to show A(X) Re A(X').
17

---

Note that |X| > 0. If | = O. then |X| 1 and Pr [A(X)) = [1 = 1. We then show that
Pr [r[A(X) 1] 2 1 8/22. This holds since by Definition A.1
e
Pr[Nx < 0] Pr(2 log(1/0)/e
Pr(z log (e)((/
2
Therefore, in this case. A(X) 6// A(X') That is, if S 1/2, A is (0, 8D.P).
Now consider [X > O. By Lemma
B.4. We know Pr[Nx < X = Pr[Nxx < [X']] 1 8/2.
Hence, Pr[Nxx < | = Pr[Nxx
<
X
+ 1] 2 1 - 8/2. Then what is left is to compare
x A(X') X XX--
By Lemma
A.3
Nx Re, Nx as X X 1. By Fact A.7
Nx
Nx<|x (1
Nx N x <|X]' In order to perform composition by Lemma A.6 now show that for each fixed N as follows:
[X], A(X)lx== R,,88 A(X'))
Choose < [X]. A S O, then A(X)l - A(X)) = and we are done. If <
< [ it suffices to show M/{x x0, (1 Re,6 N Mpxi-1 E X M/3)
where 22 log({ 1,25/81 1,25/8)A²
which, by post-processing is equivalent
=²
X
I Σ i=1iss
N 1/4
(0)
ReS N
M-1/1
(9)
X
X
Define vector = X
X
EI
X As XUX dist afriendly, for every
i [|XI]\ (7), there exists some (() Rd such that disty,, (x(1) y()) dist Y(0)- =
1. We have
((E)
-x ) Σ x()
| -M-1//D1la
M M-1/4
| 1)
[X|
(1 M 1-4 1/4
(x(()-x())) r(i) + +1|11- -||00-144r000 |1,)
(x(0) -800-y
| 1 1)
i=1,85
(by triangle inequality)
$ 1 XXX[X|
2x 2A
Σ
2A
X
(by
assumption and since < [X)
We know Equation 9 holds by applying the guarantees of the Gaussian mechanism (Lemma A.5 where we set f(X) = M 11/4
[ - ( and D, = 22/
Combining these results, have A is (e + 1-87 Se (1--12)) 5/2) -DP in this case. For s 1/2,6 < 1/2, this becomes at most (3z, -DP.
Therefore, overall, by Theorem 2.5 Algorithm is (2(e 1) 2e 12(( 3,, 8' = 26. So for e < 1/2, the algorithm is (21e, e DDD overall.
12(061111 -1)8'--DP with e'
C Lower bounds
C.1 Dimension-dependent lower bound under pure DP
The so-called packing lower bound technique 32 10 number samples required by any pure DP algorithm learning the mean Gaussian distribution, even in the anisotropic case consider in this paper.
implies a lower bound on the order of d for the
18

---

There exist several statements in prior works which establish the lower bound for learning Gaussian distribution with known covariance in TV distance, which equivalent learning the mean in Mahalanobis distance, or to learning the mean in E2 norm in the isotropic case [88 Lemma 5.1]. It trivial to observe that the dependence on the dimension d persists in the anisotropic case, yet we include the proof here for completeness.
Theorem C.1. For any a R/2, any E-DP algorithm which estimates the mean μ. E (R) of a Gaussian distribution with known covariance Σ, up to accuracy a in E2 norm with probability 9/ 10, requires ) samples.
e
Proof. Consider a 2a-packing of the d-dimensional R-radius ball, denoted by P20 (R).. That is, Yu, v E P. 1|u vill > 20, so that the balls with centers u, and radius a are disjoint: (u, n (e, a)) = We consider the family of Gaussian distributions {W(u, E)
Suppose A is an e-DP algorithm with the stated accuracy requirement. This implies that vu Pan:
Pr (u,)-
[A(X) (u, 2 9/10.
(10)
At the same time, for any pair of samples X, No of size n, and any measurable set B C rA)g by the privacy guarantee, Pr (x)) E BJ S e Pr (Ax,) E B). This implies specifically that for uo P2a.
Pr [A(X) E S T AXxN(u,
Pr (A(X)) E B).
(11)
A. EYn
We have
12
(A(X)) U (u) A.Xo~NNu..N M.
Pr
uEPso
Σ
Pr
(A(X)) (u,)))
(u, disjoint)
Σ en
Pr (x
[A(X) E (u,a)
(by Eq.
NEP
9
iPza
l sn
(by Eq. 10
10
We conclude that log Paol
Since P2al 2 (8) it follows that (log(R/20)
This lower bound makes e-DP prohibitive for the regime we consider in our setting. To compare with upper bounds for (c, DD, suppose that we want to learn μ with accuracy eσ1 R, where 0
is a small constant. Then our main result implics that this is achievable with n I &allllla samples for
some constant > 0, whereas under e-DP, we would need at least n 2 consider in this paper.
| for the regime we eo1
C.2 Lower bound for approximate DP
The so-called tracing or fingerprinting lower-bound technique [3 technique used yield lower bounds for mean estimation under (e, DP Kamath et al. 39 40) apply it to give lower bounds for the problem of learning a Gaussian in TV distance (which is equivalent learning the Gaussian in Mahalanobis distance for the known covariance case, or the isotropic case).
60 17
61
28 is the main
Theorem C.2 (Theorem 6.5 39). Af RRdx - [-Ro, Rojá is (c, for 5 = Ō Vd and
for every Gaussian distribution with mean μ -Ro, Royd and known covariance matrix o o²I Ia. with probability 2/3, [[A(()) all < a Vda R/3, then 24a log(dR)
do
Following exactly the same steps as the proof of the theorem under the slightly more general case of known covariance Σ diag(o") a²) gives us a weak lower bound for our setting, on the order of |all
24co Toy
However, more careful application of the same theorem directly gives us the following stronger lower bound, which implics that our algorithm for the known covariance case is near-optimal.
19

---

Theorem C.3. If A: Raxx - Rd is (E, 6) å =
and every Gaussian
distribution with mean μt E [-1, 11 and known covariance proxy Σ (iag((o)), with probability 2/3. [IA(X) mll v a== 0 o /11/ (og(d)), then n 52
|| h
lox
Proof. Assume w.l.o.g. that of 2 2 oz. Consider a partition of the set of coordinates [d] into buckets Sy = ii E [d] or (2k 22.22--11
22-11. Ak E [[og(d) and Siog(d)+1 = [d] Uke[lot(e) Sg- We
have that Σx=1 -log(d) Eies o = ||o111. Consider the bucket Sm which contributes most to this sum, that is m = arg max Lies oz. Let os.. = max : Sm}. It must be that
|
|mm| 2 (log(d) 1)0s..
Otherwise, ||-|| = k=1
log(d)+1 Eies, S (log(d) + 1)mmm os.. 1|0ll11. which is contradiction.
All the variances of the coordinates in Sn are within factor of two from os... We apply Theorem C.2 to the lmmmmennnso Gaussian with = 1. Consider the Gaussian distribution with mean JAS.. E 1,,1]|5m and known covariance matrix o3 Is- We have that any (e, 5)-DP algorithm
for 5
0 (oggn))
which returns, with probability 2/3, an estimate üss, with error Ilãs,.
Msm. 122 S a v S Tsm /3, requires
SSm Sm 24as log(d) 48cx log²(d)
|
n 2
(12)
samples.
Now assume that there exists (e, 8)-P algorithm A Rdxx - IRd for 5 =0 ( such that,
(>Vlog(n)
for every Gaussian distribution with mean / E [], and known covariance proxy Σ diag(o²). with probability 2/3, ||A(()) S S 3(log()++1) Restricting the Autput_A(X) the coordinates in Sm. would give us a mean estimate for Sm with error at most a. Combined with Eq. (12), this completes the proof of the theorem.
||
D Omitted details of Section
We state the main theorem in more detail.
Theorem D.1. Let parameters e, 5 E (0, 1). Let X N(,, Σ)" with unknown covariance There exists an (s, diifferrent private algorithm which, with probability 1 - 3, returns estimate such that ja /al2 S a, as long as
log(d) log
log 1
Ω log2(d)
83
tx(2) + |>11|2 12 log 1
n Ω
(13)
a²
Ed 1/2 Li=1 24
log 1
Ω
(14)
aE
and
dd//4
d Ed
1 1/4
5/4
log(d)
Ω
is
(15)
Vae
where the symbol Q hides multiplicative logarithmic factors in 1/3 and the term in parentheses.
20

---

Recall the definition of set V. Within each group j, for each coordinate i, we compute an estimate V2 for En as follows:
VO)
28 Σ
(x((2x-1)- x(((2r)) 2
(16)
We show that the variance estimates are valid in the subsequent sections.
Lemma D.2. Let x(),..., X()) be d-dimensional i.i.d. samples from Ni,, E). Let
(V,0) [Je(m), be the estimates defined in Eq. (I6). Then, there exist universal constants C, C' >
such that if é Colog and m C" 1og(1/ with probability least 1 3, the V of estimates
is valid.
Next, use the estimates v:0) inputs multiple procedures. We introduce the following estimation tasks.
Definition D.3. For a covariance matrix Σ Raxd consider the following:
1. k-th largest variance: Approximate the k-th largest value among the diagonal of E, namely, the k-th largest value among (Em1,-..; Eda)-
2. Sum of variances: given subset I S [d]. approximate the sum Eiel Ei-
We have the following algorithms for these tasks. The proofs of Lemmas D.4 D.5 and D.6 are the subsequent subsections.
Lemma DA. Let e, E, 6,3e 1/2). There exists an algorithm FindkthLargest Variance which
receives variance estimates V(1), ...., V(m) Rd and an integer k E [d]. and satisfies the following, provided that
1
Ω log
83
Privacy: FindKthLargest wew.laaaaggeeaaaieeeee..mma Variance,, is (s, 0)-P with respect to changing each input vector V())
Accuracy: denote the k-th largest entry of (Em..... Σda) by Q and the algoritlm'ss output by Q. If the estimates (V(),... ..... V(ml) are valid wrt then there exists a universal constant C' > 0 such that with probability at least 1 - B,
Q/8 S $
Lemma D.5. Let E, E (0.1). There exists an algorithm VarianceSume,6 which receives variance estimates V(1), "....,,(m),, Rd and a subset I C [d]. It has the exact same guarantees as FindKthLareest Variance from Lemma D.4 except that it provides an estimate for Eier Ei; instead of estimate for the k-th largest diagonal entry E.
Assume for that the estimates V(O) valid. With these procedures hand, we first compute R such that (by rescaling) Q/64 < R < Q. where Q is the k-th largest diagonal entry of Σ. Then, we call procedure that finds k entries i such that Ex 2 R. Its guarantees are listed below:
Lemma D.6. Let E, 5, E (0, 1). There exists an (e, 5)DD algorithm TopVar, (V. R), such that, if V is valid.
(v
(llg((1/0)
(1/8) eT
log 2
m 2 Ω
and (1:: Ene 2 R)| k then the algorithm outputs a set Trop of size such that for all i Ttop- Ea 2 R/4.
At the next step, we would like find, up constant factor, the variances corresponding these coordinates: the values Σ,; for i E hop We use the algorithm VarianceSum & times, providing the sets (i) for E hop-- We obtain estimates E,; that approximate Σn up constant factor.
Next, we estimate the mean j the coordinates Topp denoted (tops separately from T := [d]] Teap- denoted Abot since we approximately know the variances in Itop. obtain better estimate.
21

---

Both for estimating Ptop, and for estimating Mbot, we use AVgM, A.c.á (Algorithm I from Section 3) with appropriate choices of parameters M, X. Recall that Algorithm satisfies the guarantees of Theorem B.1
For estimating Mtop- we A. for estimating the mean of k-dimensional Gaussian, with input vectors restricted to coordinates Tepp- x 2.... x
the k k-dimensional diagonal
matrix M, with Ma Ees (we assume that the rows and columns of M are indexed by 4оp).
and À =0 (VVV D/ log B
Denote the output by Prop-
Theorem B.1 shows that with
probability 1 - 3, llpoop - Propple S a, if
tr(E) a
Vlog(1/0) Eishop 2i
12/2
log(1/6)
n 2 Ω
(17)
QE
where Ω hides multiplicative logarithmic factors in 1/3 and the second term.
For estimating Toot, we do know the variances. In order to perform the estimation, first call the algorithm VarianceSun to provide an estimate Sbot such that ? Eiehon Ens S Shot I C Evethot Ei for a constant C. Given that estimate, we again will call Avgm.Ae. now for (d-k)-- dimensional estimation problem. We input the samples X Ibou ................................ X(1) Aboe replace the matrix M with
the identity of dimension (d k) (d - k), and let À = O (vS Sbor log 3) Denote the output by Abnt The guarantees of Theorem B.1 provide that with probability B, | Phoell2 I a, if, additionally to Eq. (17). we have
V dlog 1/2
EiEloot Σii
n 2 1
(18)
where Q hides multiplicative logarithmic factors of 1/3 and of the term in parentheses. As we prove below, combining these guarantees would yield the desired result. Additionally, we that in order for the proof to go through, we split the sample into two groups. One group is used for estimating the variances and the other group is given as input to the two invocations of VarianceSum. We provide the formal pseudocode in Algorithm 2
Accuracy analysis. We put together the statements of the lemmas above, establish the overall accuracy guarantee of Algorithm| By Lemma D.2 the estimates v(O) are valid (i.e., at least 4m/5 of the groups have approximation up to 2 for every coordinate), with probability 1 8 as long as m = 2(log 4). Consequently, Lemma D.4 implics that as long as m 2
_ log 88 ).
FindKthLargest Variance outputs an estimate of the Æ-th largest variance, which accurate up to a constant factor C = 8, with probability 1 33. By scaling, we can assume that, ft/8, is upper bounded by the k-th largest variance. Under this assumption, and as long as m = Ω log(11
log 4
en
Lemma impliis that w.p. 1 - 3, the output of Var, Trops is set of size k. containing indices of elements whose variances
least R/32.. By Lemma D.5 long
Ω log 8
A the estimates E. to the variances in the indices in Toop accurate up to Ω (olog(1/8) log 83
constant factor, with failure probability of B/k for each invocation of this lemma, which sums up to failure probability of B. Similarly, the estimate Sbot has the same guarantee. If n is large enough to satisfy the requirement of Line then all previous constraints on m are satisfied.
Lastly, the two estimates from A Avg & suffer an approximation of a, each with failure probability of H. provided that the conditions on the sample complexity n, that are given in Eq. (17 and Eq. (88
We could additionally privately learn the largest variance among Inot, denoted by and set (VStott + v /olo ;) to decouple Shot from the logarithmic factor, but we choose not to for simplicity, and since did not optimize for logarithmic factors overall.
22

---

hold. By assumption on the sample complexity (Eq. 13
14
the guarantee of Eq. 17 indeed holds.
remains to prove that the guarantee of Eq. 18 holds as well. We analyze the term Eieloo Ej.-
Denote vector bot (foi)iElnor where o; E Then Eiethor En
||
By Hölder's
inequality,
o 22 S /I Oot |l Oot S v Vloll oot
By the guarantees of Top Var and FindKthLarges Maraaaerttaaaainneee Variance, except for a failure probability of (3),, there exists universal constant C > 1 such that
max 2// S CCR/2 iEo
Further, by assumption, R is up to constant the k-th largest diagonal element of Σ, hence,
d <
C
k
im
Substituting this above, we obtain that
d 1 1/2 is
Σ E
Vk i=1
iE
Thus, it suffices for the stated sample complexity to additionally satisfy n Eim 2//² )
(1))
Substituting the definition for k, we obtain Eq. 15 which
Vkae
completes the proof.
Privacy analysis. Notice that the output of the algorithm is obtained by composing multiple differentially private mechanisms. Some of these mechanisms access the estimates V ...... ((m) instead of the original dataset. Yet, since each input datapoint ( influences only one vector VG), this implies that any DP guarantees for algorithms that use the VO) estimates, directly translate to DP guarantees on the original input dataset.
Notice that the algorithm has 0(1) calls to (s, 6)-DP mechanisms, and & calls to 8')DD mecha- nisms: these are the calls to VarianceSum. By Lemma (advanced composition), the concate- nation of all the calls to VarianceSun are together, (O(E). O())). By basic composition of the same lemma, composing the resulting composion with the other calls to DP mechanisms, yields an (O(s), (D mechanism.
D.1 The variance estimates are valid: Proof Lemma D.2
The random variable (x: (xx()2r-1)
x ((22))) //12///222/ 2 is standard normal. Thus,
(1.2--1 x (1.2r)
follows chi-squared distribution with e degrees of freedom. We use the following concentration property of a Chi-squared random variable [45) Lemma 1]: Z is Chi-squared with degrees of freedom,
Pr[E[Z]/22 < Z 2 2 1 -e
for some constant e 0. Consequently,
(x 1) -x(-))xxx111)) x62221)2
Pr
< 2l
2 - -1-2e" 2e
2>1
for some constant c
0.
By a union bound over all dimensions i, the probability that all dimensions' variance estimates fall within the specified bounds in single group is least 1 2d exp(-ce) Assuming é 2 C'log d ensures this probability is very high (e.g., at least 7/8 for suitable constant C)).
Using the Chernoff bound for the binomial distribution, if each group independently satisfies the variance bounds with probability at least 7 7/8, then the probability that at least 4/5 of the groups satisfy the variance bounds is at least 1 for Q(log(1/8)).
23

---

D.2 Finding the indices of the largest variances: Proof of Lemma D.6
We propose an algorithm which, receives estimates v- for the variances and a threshold R, and outputs k indices E [d] whose variance is at least R/C for some universal constant C. To do so, we use the sparse vector algorithm, which receives dataset D, queries (D). Qa(D), , a threshold T' and natural number k. It outputs & indices such that Q; 2 T (approximately), In order to use the sparse vector to identify the largest variances, our dataset D will V, the collection of estimates. The query Q:(V) will capture whether the i'th variance is (( We define the query
Q.(V) m {5: v( 2 R/2)|1
and the threshold T = 1/2. Intuitively, if QV() 2 1/2 this means that at least half of the values of j: vo5) 2 R/2, which implies that Ea 2 R)) provided that the estimates V are valid. Otherwise, it implies En I R.
We now formally define the sparse vector algorithm 26 54 31, and review its guarantees. See 23 Section 3.6] for a detailed analysis of the sparse vector technique.
Algorithm (Q.), e, 8), from [23
Require: Input is private database D, adaptively chosen stream of sensitivity 1/n queries Q threshold T, cutoff point k, and privacy parameters e, 6.
1: T t T
Lap
32k (1(/8))
2: a
en
3: count + 0
4: I+-
5: for each query do
6:
Us + Lap(or)
7:
if Qf
+
f then
8:
Iu
9:
count count +1
10: if count > 1 then
11: return 7
12:
return
Lemma D.7 (Sparse guarantees). Sparse (Algorithm is (e, diifffereetl private. Let 3 E (0,
and define
) 128k In(1/8)
2 3
(1og d log 2)
a 2a log + log
3
EN
For any sequence queries QI.... Qa if there are at least queries i such that QD()) T + a. then the following holds with probability 1 B3: the output of Algorithm I, is a set of size k; and for each E I, (D( 2 a.
Next, we formally define the algorithm Var, find the indices of the largest variances.
Algorithm 4 Top (V, RV, k))
Require: Variance estimates V (v,()) (0, 1), number of indices k E N.
Jjelm) threshold R I. privacy parameters E,
1: Define queries (D) for each i [d] as:
Q((D)= 1 {s: v(6) > R//2
2: T <- 1/2
3: return Sparse(V V, (Q.), T,
24

---

The privacy guarantees of Vor follow directly from the guarantees of the sparse vector. Next, we describe how to derive the accuracy guarantees. Notice that if the v,0) are valid, then, for any such that Ea 2 R: for at least 4m/5 values of j. it holds that V 2 R/22, hence, Q:(D) 2 4/5. Further, for any such that En R/4, for least 4mn/ values j holds that vi() < R/2, hence Qi(D) S 1/5. Hence, if we set the threshold at T = 1/2,, and a 1/4, then, for any output by the algorithm, En 2 R/4. Further, there are at least & indices i such that Qi 2 R. the algorithm will output k indices.
D.3 Finding the k-th largest variance: Proof of Lemma D.4
We propose an algorithm, Algorithm 5 that receives pre-computed variance estimates V. for each group j and coordinate i. The algorithm uses them to compute an estimate for the k-th largest variance for each vO);
Mj k-th largest of {v(o..... ..... VG1
sE|4
Our algorithm combines all of these estimates in a differentially private manner, using a stable histogram: Algorithm| 6. That algorithm splits the real line into buckets, { {Bohezut(-so receives the estimates M1, M,..... Mm E IR and outputs the index b of the bucket that contains the largest number of estimates M, (approximately).
In our application, we would like to estimate the k-th largest variance up to a multiplicative constant consann factor, hence, define the buckets
be b b= 000
BB = {0}}
Denote by b" index of the bucket that contains the k-th largest diagonal entry of E. If the estimates v())..., (mm) are valid then, by definition of validity (Definition 4.2 it follows that at least 4yn /5 of the estimates M, fall into the union By- U BB- U Bg Under this assumption, Algorithm is
guaranteed to output one of b* 1, b* b* 1. with probability 1 6.
The algorithm for k-th largest variance, Algorithm 5
presented here:
Algorithm 5 WinnKKthLargesttariieee ((V. je(m)
Require: Pre-computed variance estimates v(5) for each group j and each coordinate i. Privacy parameters s, 0. Integer k S d. Number of groups m.
1: for j E [m] do
2: My < k-th largest value among (V1(), v,()) ...., v
3: Define bins (Boleez((-xi) by:
[40,a001]
b E
B,
[10}
b= 00
4: b 15-Stann (MM (Ba))
5: return M
We proceed by defining StableFistogram as introduced [66 and providing its guarantees, and then conclude with the proof of Lemma D.4 The presentation of StablelHistogram is from [[3]
25

---

Algorithm 6 Stableffistogram, ((M)), (Ba)), from [16
Require: Items MI..... Mm EU. Bins (BoJeez. Privacy parameters e, >>>>> 0.
1: for E 2 do
2: Cb f {i E BbHl
3: for b with c > do
4:
Gb - + Lap(2/e)
5: T 7+1+ 21og(1/8)
6: Let Umax- arg max; Cb with arbitrary tie breaks
7: if Chres then
8:
return max
9:
else
10: return T
We use its privacy and accuracy guarantees, proved as Lemma C.I in 13: Lemma D.8 (Stable Histogram Guarantees). StableHistogram.. (Algorithm (c, 6)-
differentially private. Suppose that there exists b* E Z such that
I,, Mm n (B Bb- U B1)+ 2 3m/4.
There exists constant C' > 0 such that, for all E, 3,8 < 1, if
C
m 2 log 53*
then with probability least 1 - 3, the algorithm's 's output lies in {b 1,6, 1
The privacy guarantees of Algorithm f directly from the privacy guarantees of Algorithm 6 For the accuracy guarantees, notice that if the estimates V()) are valid then at least 4m/5 of the values My fall into the bucket By- that contains the true value the k-th largest entry of the diagonal of Σ. Under this assumption, Algorithm 6 is guaranteed to output, with probability B, one of 6" or 6* + This implies that the output of Algorithm is approximates the target quantity up constant, required.
D.4 Finding a sum of variances: Proof of Lemma D.5
We propose algorithm that is similar Algorithm with single difference: given each estimate ((), the algorithm computes
Mj
Σ YEl
The algorithm is summarized below:
Algorithm VarianceSuu,, (
Require: Pre-computed variance estimates v.6) for each group j and each coordinate Privacy parameters E, 5 > 0. Subset I C [d]. Number of groups m.
1: for j E [m]] do
M, - EiEr
2:
3: Define bins {BoJoEzu(-oo) by:
[14 1b-14
be b b= 00
B₁
[10}
4: b 3 SStableHistogramm.... ({M, je(m), (Bbt)
5: return M 4
The proof is identical to the proof of Lemma D.4 order to carry that proof, one has to notice that is the bucket that contains Σier Ex and the estimates VQ) are valid, then least 4m/5 of the estimates M, fall within By- U By- U B
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The introduction includes informal theorem statements for each of our main claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: We discuss limitations of this work in the introduction, in Section 4 and when outlining directions for future work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
27

---

Answer: [Yes]
Justification: All theorems in the technical Sections 3 4
as well as the Appendix include
their assumptions in the theorem statement.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [NA]
Justification:
The paper does not include experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
28

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [NA]
Justification:
The paper does not include experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurIPS code and data submission guidelines (https
/ "pipsscc/puullc//uuu /ttee//ggudddeeeedSuuuummbbbbi ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [NA]
Justification: The paper does not include experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: The paper does not include experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors Is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [NA]
Justification: The paper does not include experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We believe that the presented research conforms with the NeurIPS Code of Ethics, as it is theoretical and does not enable harmful consequences.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our paper theoretical and contributes to a line work that aims to design privacy-preserving statistical estimators, which use few samples, even in high-dimensional settings. We believe that in the long-term, the societal impact of this line of work will be methods, and possibly
positive, in the sense that it will enable the use of even mitigate the need for collection of large data sets.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
30

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The paper does not contribute data or models with high risk misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
31

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "yPC2u72o4O", "reviewer": "Reviewer_JTxB", "summary": "The paper studies mean estimation for multivariate Gaussian distributions. It's well known that this problem under differential privacy suffers from a curse of dimensionality- the sample complexity of estimating the mean (in expected $\\ell_2^2$ error) scales with $\\sqrt{d}$ where $d$ is the dimension. However, the lower bounds are for isotropic covariance matrices, whereas in the real world, covariance matrices are often far from isotropic. The main intuition is that when covariance matrices are far from isotropic, there is a gap between large and small singular values, and hence the number of 'important dimensions' is relatively small- the object behaves 'more lower-dimensional' than the dimension $d$ indicates.\n\nThey consider two settings, the known and unknown covariance cases.\n\n1) In the known covariance case, they show a dimension-independent bound (for approximate DP) that depends instead on the sum of (square roots of) singular values. They argue that this is optimal up to logarithmic factors (and also that no pure DP algorithm can achieve such a dimension-independent bound). \n\n2) For the unknown covariance case, one approach is estimating the covariance and then applying the known-covariance algorithm, but this might be prohibitive since estimating the covariance is known to require at least $d^{3/2}$ samples asymptotically. The authors instead show that it's possible to achieve a bound with a $d^{1/4}$ dependence instead.\n\nFor the known covariance case, the authors use the known FriendlyCore algorithm with a carefully specified predicate to remove outliers (points far from the mean in any coordinate), which allows them to bound the sensitivity and then add dimension-independent noise. The lower bounds are adaptations of the packing and fingerprinting approaches used for pure and approximate DP. In the unknown covariance case, they treat large and small singular values differently. Firstly, they learn the identities and values of the top $k$ singular values up to multiplicative factors and use the known covariance algorithm to estimate the mean restricted to these coordinates. For the other coordinates, they use Holder's inequality to bound the $\\ell_2$ norm of the singular values. For small singular values, the variance does not need to be as accurately estimated since these directions are less important. Balancing $k$ to optimize the cumulative error gives the sample complexity bound.", "questions": "1) Line 829, I believe the authors mean $\\|\\| \\sigma_{bot} \\|\\|_2$. \n\n2) Do known lower bounds for estimating the covariance matrix depend on $d$ even with large singular value gaps? Could one hope to get bounds for this problem that depend similarly on the sum of singular values? I am curious if existing results truly rule out the approach of estimating the covariance matrix before using a known covariance algorithm.", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1) Gaussian mean estimation is a fundamental problem, and this paper suggests a way to deal with the curse of dimensionality that privacy imposes for this problem, by considering non-worst case instances (anisotropic covariances). This is an interesting direction that will likely spawn future work. The paper also does a nice job of suggesting future directions of research in this space.\n\n2) The paper combines known techniques in privacy in clever ways to obtain their upper and lower bounds. The problem of private Gaussian mean estimation has seen lots of prior investigation, and they also do a good job of explaining how a lot of these techniques inherently give dimension-dependent bounds.", "weaknesses": "1) While the results are interesting, they mostly follow from applying known techniques from the privacy literature- the additional technical insight of this paper is rather limited.\n\n2) The unknown covariance case as presented was confusing to me- is there an assumption that the covariance matrix is diagonal; this seems to be used in the author's proofs and approaches? This wasn't clearly described in the paper and could use clarification. I don't see how to extend the techniques of the authors to deal with general covariance matrices.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and interest in our results. \n\n- **The case of unknown general covariance**: Our main result for unknown covariance (Theorem 1.3) includes the sum of the square-roots of the diagonal elements of the covariance matrix, $\\sum_{i=1}^d \\Sigma_{ii}^{½}$. This way, we present a result for general covariance without making any assumptions on the covariance matrix. However, this sum is more easily interpretable when the covariance is indeed diagonal: the sum is equal to $\\mathrm{tr}(\\Sigma^{½})$, when the covariance matrix is diagonal, and can be larger otherwise. In the paper, we focus on the case of diagonal covariance in places, for ease of exposition when describing our techniques, to compare with other baseline approaches which only work for diagonal covariance matrices, or to compare with our result for the case of known covariance which includes the $\\mathrm{tr}(\\Sigma^{½})$ term. We understand how this can be confusing and we will clarify that we make this assumption for simplicity when we do. Thank you for the suggestion. \n\n- Indeed, Line 829 has a typo and this is the correct term. Thank you for noting it!\n\n- **On the approach of privately estimating the covariance matrix before using a known-covariance mean estimation algorithm**: We do not know of any lower bounds specifically aimed at privately estimating a covariance matrix with large singular value gaps. The existing lower bounds for covariance estimation in spectral norm which apply to Gaussian data ([Kamath Mouzakis Singhal 2022], extended to the low-accuracy regime later by [Narayanan 2023]) use almost-isotropic covariance matrices as the lower bound instance, so they do not go through under a large singular value gap assumption. But we would not expect them to: there are cases [Singhal Steinke 2021] where assuming a known large singular value multiplicative gap in the order of O(d^2) allows us to learn the subspace of the top eigenvectors (and subsequently their singular values) with a sample complexity roughly polynomial in the dimension of the subspace, which does not depend on d. \nHowever, we note that in any case, we expect that going through covariance estimation to achieve our goal would be a lossy step. The reason is that for the case of (almost) identity covariance, the sample complexity of covariance estimation in spectral norm would be $d^{1.5}/\\alpha\\varepsilon$, whereas our current upper bound for the unknown covariance case is $d/\\alpha\\varepsilon+d^{0.75}/\\alpha^{½}\\varepsilon$, which is smaller (and optimal for the regime $\\alpha\\leq \\sqrt{d}$ which is usually of interest). We will update our manuscript to include the discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zl5v2qnv1g", "reviewer": "Reviewer_XFCk", "summary": "This paper presents a new method for estimating the mean of a subgaussian distribution, such that differential privacy is guaranteed (i.e., the final result does not provide too much identifying information about any individual sample).  For the proposed method, in the case of known covariance,  the sample complexity depends only on the trace of the (1/2) power of the covariance matrix. This means that, if the distribution is highly anisotropic (with the variance mostly concentrated in only a few dimensions) then the sample complexity does not depend explicitly on the full dimensionality of the space *at all*, only on the dimensionality of the lower-dimensional space where \"most\" of the variance lies. This improves on the prior SOTA result, from (Aumüller et al 2023), where the sample complexity explicitly contains a factor of sqrt(d), where d is the full dimensionality of the space. The core algorithm proceeds by first filtering outliers, using a differentially-private filtering mechanism proposed by (Tsfadia et al 2022), and then simply taking the mean of the remaining samples and adding Gaussian noise.\n\nA matching lower bound is also proven, showing that we can't in general do any better asymptotically than O(trace(Σ^(1/2))), up to logarithmic factors.\n\n\nA variant of the algorithm is proposed for the case of unknown covariance as well, which has sample complexity with an explicit dependence on d^(1/4), which is still an impovement overt the prior SOTA d^(1/2).", "questions": "- Is it possible to derive a separate, tighter lower bound for the the case of unknown covariance? Or is it possible that we may in the future improve from the d^(1/4) result in this paper?", "limitations": "Limitations are addressed adequately in the Future Work section.", "rating": 7, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The problem being addressed is important, given the increasing relevance of privacy guarantees in machine learning.\n- The improvements from prior work are clearly contextualized. Sufficient background information is provided to understand the major points of the paper.\n- To my first estimation, the technique appears sound, although this is not my area of expertise and I am not qualified to rigorously evaluate the technical correctness of the proposed algorithm.", "weaknesses": "- The order of the presentation could be improved. The preliminaries section could be presented earlier in the paper, before the definitions given are referred to.\n- No empirical tests are performed. Including a numerical simulation of the algorithm would increase the reader's confidence in the correctness of the theoretical  results, as well as give an idea of the tightness of the bound in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and suggestions to improve the presentation of our paper. \n\n- Thank you for the suggestion on the presentation! We will consider including definitions from the preliminaries earlier in the introduction to formalize the intuition we want to convey. Please let us know if there was a specific definition that would have been helpful to see earlier.\n- **Lower bound for the case of unknown covariance**: We are not certain that our results are optimal in the unknown covariance case, even assuming a diagonal covariance matrix, and we agree that this is a very interesting question. We know of special cases (e.g., diagonal matrices whose singular values follow an exponential decay or isotropic matrices) for which our results are optimal up to logarithmic factors. Our current intuition is that there is a case for which the trace term is constant and the $d^{¼}$ term is necessary. Therefore, we think that it is possible that the $d^{¼}$ term cannot be improved.\n- **Simulations**: We focus on the theoretical analysis of our algorithms and prove their privacy and accuracy guarantees, but we do expect that the known-covariance algorithm will perform well, since the constants involved in our error are provably small. The results for the unknown covariance case are less conclusive and we agree that some experimentation might be interesting in this case. However, we would like to point out that ours are the first methods that are applicable for $n\\leq \\sqrt{d}/\\varepsilon$, which makes it difficult to choose an appropriate baseline to compare to for smaller sample sizes.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g65D2V42Dh", "reviewer": "Reviewer_HUbD", "summary": "This paper considers the problem of DP mean estimation and the focus is on the high dimensional settings where the distribution is nearly low rank (or tr(Sigma)<<d), and the error metric is l_2. Prior work in this setting still requires sqrt(d) samples to achieve any non-trivial error which is sub-optimal. In the known covariances setting this paper achieves sample complexity n = tr(Sigma)/alpha^2 + tr(Sigma^1/2)/(alpha*eps) which can be significantly smaller than sqrt(d) in the high dimension setting. The algorithm filters out the outlier using the FriendlyCore algorithm [62] and leverages the propose-test-release framework. For the unknown covariance, one will need to first have a rough estimate of the covariance such that appropriate noise can be added to guarantee privacy. However dimension free estimation of the covariance is impossible, so the idea is to estimate the top-K coordinate with the largest variances, and simply adding small and isotropic gaussian noise to the remaining directions. Combining the error incurred in the top and bottom coordinate gives a d^(1/4) dependency in the dimensionality. On the lower bound side, they prove their known covariance result is optimal up to logarithmic factors. It is unclear what the optimal sample complexity for the unknown covariance should be.", "questions": "I don't have question for the authors.", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "DP mean estimation problem in the high dimensional setting is a fundamental problem in differential privacy, and the paper achieves optimal results for known covariance setting and makes improvement for the unknown covariance meeting. The ideas for unknown covariance seem novel to me. The sample complexity of mean estimation in the unknown covariance setting, posed in this paper, remains an interesting open problem.", "weaknesses": "The techniques for the known covariance setting have been developed previously.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and interest in our results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gRARMcPXSp", "reviewer": "Reviewer_Tkrg", "summary": "The paper presents new differentially private algorithms for estimating the mean of high-dimensional data, addressing the inefficiencies of traditional methods that suffer from the \"curse of dimensionality.\" The proposed estimators are tailored for anisotropic subgaussian distributions, where data signals are concentrated in fewer dimensions. These estimators achieve optimal sample complexity that is independent of dimensionality when the covariance is known and improve the sample complexity for unknown covariance cases from $d^{1/2}$ to $d^{1/4}$.", "questions": "1. The lower bound does not totally match the upper bound. Is it possible to improve the gap?\n2. In Theorem 3.1., the range of $\\epsilon$ is from 0 to 10. Actually, 10 is a weak privacy guarantee in DP in practice. Why do you take this value?", "limitations": "There is no experiment.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper studied interesting problems and the writing is clear. The authors give both upper bound and lower bound for the problem.", "weaknesses": "1. I understand the space of the main text is limited but there is no conclusion.\n2. There is no experimental design to verify their theoretical findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and suggestions to improve the presentation of our paper. \n\n- We agree that having a conclusion would be preferable and since the final manuscript can be a page longer we will certainly add one (and move discussion for future work from the appendix) in the main body. \n- **Lower bound for the case of unknown covariance**: We are not certain that our results are optimal in the unknown covariance case, even assuming a diagonal covariance matrix, and we agree that this is a very interesting question. We know of special cases (e.g., diagonal matrices whose singular values follow an exponential decay or isotropic matrices) for which our results are optimal up to logarithmic factors. Our intuition is that there is a case for which the trace term is constant and the $d^{¼}$ term is necessary. Therefore, we think that it is possible that the $d^{¼}$ term cannot be improved.\n- **Range of epsilon**: Thank you for pointing this out – we understand that the range $(0,10)$ seems somewhat arbitrary, and we will clarify this in the updated version. Please note that the theorem still holds for $\\varepsilon\\in(0,1)$, which would correspond to a reasonably strong privacy guarantee. However, we wanted to point out that the theorem holds more generally, for $\\varepsilon\\in(0,c)$ where $c$ is some constant larger than $1$, since $\\varepsilon>1$ is still encountered in real-world applications, and might be a regime of interest for those. The choice of constant is due to approximations we take in the privacy proof and it has not been optimized (so the theorem could possibly hold for $\\varepsilon\\in(0,c)$, where $c>10$). However, as you mentioned, 10 seems to be large enough already, so we did not find it necessary to explore the exact constant $>10$ for which our proof goes through.\n- **Simulations**: We focus on the theoretical analysis of our algorithms and prove their privacy and accuracy guarantees, but we do expect that the known-covariance algorithm will perform well, since the constants involved in our error are provably small. The results for the unknown covariance case are less conclusive and we agree that some experimentation might be interesting in this case. However, we would like to point out that ours are the first methods that are applicable for $n\\leq \\sqrt{d}/\\varepsilon$, which makes it difficult to choose an appropriate baseline to compare to for smaller sample sizes.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MddgCmPhvU", "reviewer": "Reviewer_PrhZ", "summary": "This paper tackles the problem of DP mean estimation for high-dimensional distributions exhibiting anisotropy, meaning the variances along different directions are highly non-equal. Prior works on this problem were plagued by a \"curse of dimensionality\", requiring sample complexities at least on the order of the square root of the ambient dimension d, even in cases where the non-private setting permits much lower sample sizes. The authors make two main contributions to address this limitation.\n\nFirstly, for the case when the covariance matrix $\\Sigma$ is known, they provide a DP algorithm achieving sample complexity independent of the dimension d. Instead, the sample complexity depends solely on $tr(\\Sigma^{1/2})$, which can be substantially smaller than d when the distribution is highly anisotropic. The bound matches the optimal non-private sample complexity up to logarithmic factors. Secondly, the authors develop a DP algorithm for the unknown covariance setting that improves upon prior work by reducing the dimension dependence from $1/2$ to $1/4$, while also depending on properties of the diagonal entries of $\\Sigma$.", "questions": "Major questions with potential score raising:\n- Is it possible to establish a lower bound strictly higher than Theorem 1.2 for the unknown covariance case? This could help identify an intrinsic gap between the known and unknown covariance settings beyond logarithmic factors.\n- The authors mention that for special cases of eigenvalue/variance decay, the unknown covariance bound can be improved to nearly match the known case up to log factors. What are the key bottlenecks or assumptions preventing a fully dimension-independent bound for unknown covariance? The remaining $d^{1/4}$ dependence seems to stem from coordinates with smaller variances - is this avoidable under some assumptions on the decay behavior?\n\nMinor questions:\n- In line 113, is the last term in the sample complexity bound for unknown covariance correct?", "limitations": "Limitations are well addressed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Paper is clear and well written, though it is mathematically heavy. The analysis, supported by proof sketches, appears technically sound.\n- The motivations of improved rates with certain covariance structure are intuitive.", "weaknesses": "- The results involving known covariance are straightforward. The benefits comes from (1) application of Tsfadia et al. and (2) rescaled noise adding, which is similar to Aumüller et al.\n- There is still a $d^{1/4}$ left.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and interest in our results.\n\n- **Lower bound for the case of unknown covariance**: We are not certain that our results are optimal in the unknown covariance case, even assuming a diagonal covariance matrix, and we agree that identifying an intrinsic gap between the known and unknown covariance would be very interesting (additionally since this gap does not exist for privately learning the mean in the Mahalanobis norm). Known lower bound techniques for private statistical estimation (in particular, the most popular fingerprinting technique for exponential families) does not apply as is to our problem. However, our current intuition is that there is a case for which the trace term is constant and the $d^{¼}$ term is necessary. Therefore, we think that it is possible that the $d^{¼}$ term cannot be improved.\n\n- **When can we achieve a dimension-independent bound under unknown covariance**: Indeed, the additional error in our results for the unknown covariance case stems from the $d-k$ coordinates of smaller variance, where $k$ is the number of top-variance coordinates we choose to learn, and we show that we can always learn up to roughly $\\varepsilon^2n^2$ coordinates without increasing the sample complexity. In particular, the error incurred in the mean estimation step is $(\\lVert\\sigma_{1:k}\\rVert_1 + \\sqrt{d-k} \\lVert \\sigma_{k+1:d}\\rVert_2) / \\varepsilon n$. The first term is already smaller than the optimal $\\lVert\\sigma_{1:d}\\rVert_1$, and in our result we show that the second term can always be upper-bounded by $\\sqrt{d}\\lVert\\sigma_{1:d}\\rVert_1/\\sqrt{k}$. When this inequality is tight for some $k$, our analysis of this algorithm is tight. \nMore generally though, there are interesting cases when the term $\\sqrt{d-k}\\lVert\\sigma_{k+1:d}\\rVert_2$ is smaller than the optimal $\\lVert\\sigma_{1:d}\\rVert_1$, for $k$ at most $\\varepsilon^2n^2$, and then our results match the ones for known covariance, up to logarithmic factors in $d$. One example is the case of exponentially decaying variances: if $\\sigma_i=\\sigma_1 e^{i-1}$, then it suffices to learn only the top $k=\\log(d)$ variances, because the error incurred by the last term $\\sqrt{d-k}\\lVert\\sigma_{k+1:d}\\rVert_2\\approx 1/\\sqrt{d}$, which is smaller than the optimal $\\lVert\\sigma_{1:d}\\rVert_1=O(1)$. Learning the top $\\log(d)$ variances using the sparse-vector technique can be achieved with a polylogarithmic in $d$ number of samples, adding only logarithmic factors to the optimal sample complexity.\n- Line 113 omits several logarithmic factors with respect to $\\delta$, $\\beta$, but indeed there is a typo in this particular term: the exact dependence is $\\log^2(d)+\\log^{1.5}(d)/\\varepsilon\\gamma + \\log(d)/\\varepsilon\\gamma^2$. We will add the full calculation for this case of interest in the appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors consider the problem of differentially private mean estimation for anisotropic distributions (i.e., assuming the data has high variance only in a few directions or similarly, assuming data has small effective rank). For non-private setting, simple mean gives dimension-independent bound but doing this privately is non trivial. This is an important problem since mean estimation is a subroutine in many private algorithms. The authors consider the known and unknown covariance settings (results for unknown covariance is not tight). Although the paper may not be technically very novel, the results are interesting, non-trivial, and of broad interest in the privacy community. I recommend acceptance.
Decision: Accept (poster)