Covariance-Aware Private Mean Estimation Without Private Covariance Estimation

1 Introduction . Although the goal of statistics and machine learning is to infer properties of a population , there is a growing awareness that many statistical estimators and trained models reveal a concerning amount of information about their data set , which leads to significant concerns about the privacy of the individuals who have contributed sensitive information to that data set . These privacy violations have been demonstrated repeatedly via reconstruction attacks [ 21 , 30 , 27 , 46 ] , membership-inference attacks [ 37 , 52 , 10 , 32 , 53 , 59 ] , and instances of unwanted memorization of training data [ 13 , 14 , 34 , 7 ] . In order to realize the benefits of analyzing sensitive data sets , it is crucial to develop statistical 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . estimators and machine learning algorithms that make accurate inferences about the population but also protect the privacy of the individuals who contribute data . In this work we study statistical estimators that satisfy a condition called differential privacy [ 29 ] , which has become the standard criterion for individual privacy in statistics and machine learning . Informally , a differentially private algorithm guarantees that no attacker , regardless of their background knowledge or resources , can infer much more about any individual than they could have learned had that individual never contributed to the data set [ 45 ] . A long body of work shows that differential privacy is compatible with a wide range of tasks in statistics and machine learning , and it is now seeing deployment at companies like Google [ 33 , 6 , 57 ] , Apple [ 3 ] , Facebook [ 55 ] and LinkedIn [ 51 ] , as well as statistical agencies like the U.S. Census Bureau [ 1 , 36 ] . Background : Differentially Private Mean Estimation . We revisit differentially private estimators for one of the most fundamental tasks in all of statistics and machine learning—given x1 , . . . , xn ∈ Rd sampled i.i.d . from a distribution with mean µ ∈ Rd and covariance Σ ∈ Rd×d , estimate the mean µ . Mean estimation is not only an essential summary statistic in its own right , but also a building block for more sophisticated tasks like regression and stochastic optimization . Without privacy constraints , the natural solution is to output the empirical mean µx = 1n ∑ i xi . The natural way to state the sample-complexity guarantee of the empirical mean is n & d α2 =⇒ ‖µx − µ‖Σ ≤ α , where & hides a universal multiplicative constant , and the accuracy guarantee holds with large constant probability ( say , 0.99 ) . Importantly , ‖µx − µ‖Σ = ‖Σ−1/2 ( µx − µ ) ‖2 is the error in Mahalanobis distance scaled to the covariance Σ. Bounding the error in Mahalanobis distance implies that in every direction v , the squared error is proportional to the variance vTΣv in that direction . The Mahalanobis distance is the natural way to measure the error for mean estimation , since it tightly captures the uncertainty about the true mean and is preserved under affine transformations . Unfortunately , in high dimensions , releasing the empirical mean leads to concrete privacy breaches [ 32 , 41 ] . A natural question is thus : can we design a differentially private mean estimator that performs nearly as well as the empirical mean ? Without making additional assumptions , the answer turns out to be no—every differentially private estimator incurs a large overhead in sample complexity compared to the empirical mean [ 42 ] . However , it is known that if we further assume that the distribution satisfies some additional concentration properties , we can do much better [ 54 , 44 , 41 , 42 ] . In this work , we focus on one class of well-concentrated distributions , those that are Gaussian ( or , in some of our results , subgaussian ) . Although assuming Gaussian data is restrictive , it is a natural starting point for understanding the complexity of estimation when the distribution is not pathological . Moreover , even in the Gaussian case , one can not obtain error comparable to that of the empirical mean unless n & d [ 10 , 32 , 41 ] , so we will also focus on the case where the sample size is at least as large as the dimension . For Gaussian data , if the analyst has prior information about the covariance matrix in the form of a matrix A and bound κ ≥ 1 such that A Σ κA,1 then there is a folklore private estimator A ( x ) , based on a line of work initiated by Karwa and Vadhan [ 44 , 41 , 5 , 2 ] , that finds an approximate range for the data , truncates the points to within that range , and runs the Gaussian mechanism on the resulting empirical mean . This estimator achieves n & d α2 + d √ κ αε =⇒ ‖A ( x ) − µ‖Σ ≤ α , ( 1 ) where ε is the privacy parameter controlling the level of privacy ( see Definition 2.2 ) , with stronger privacy as ε→ 0.2 Here & hides a universal multiplicative constant and polylogarithmic factors of d , 1δ , 1 ε and 1 α ; the accuracy guarantee holds with large constant probability . We can interpret this result as showing that the additional cost of privacy is small provided that the user has a strong a priori 1Given two covariance matrices A , B ∈ Rd×d , the notation A B indicates that in every direction v ∈ Rd the variance vTBv is at least as large as vTAv . More generally , A B iff B −A is positive semidefinite . 2To simplify the discussion , we focus only on the ε parameter , although our results and many of those we discuss require relaxations of differential privacy such as approximate [ 28 ] or concentrated [ 26 , 8 ] differential privacy , which have different parameterizations . bound on the covariance so that κ is small ( e.g . κ is a constant ) , and also that the privacy guarantee is not too strong ( e.g . ε ≥ α ) . In particular , setting κ = 1 corresponds to the known-covariance setting , where the guarantee in ( 1 ) is known to be minimax optimal up to polylogarithmic factors [ 32 , 41 ] among all differentially private estimators . However , the sample complexity in ( 1 ) grows asymptotically with √ κ , a large price to pay for the user ’ s uncertainty . Intuitively , this degradation arises because the algorithm perturbs the empirical mean µx with noise from a spherical Gaussian distribution , whose magnitude must be proportional to the largest variance in any direction , so the noise is unnecessarily large in the directions with small variance . In contrast , when the user is very uncertain about the covariance , there are estimators with a weaker dependence on κ but a superlinear dependence on the dimension . In particular , there is an estimator [ 41 , Theorem 4.3 ] with an error guarantee of the form n & d α2 + d αε + d3/2 log1/2 κ ε =⇒ ‖A ( x ) − µ‖Σ ≤ α . ( 2 ) Here & hides a universal multiplicative constant and logarithmic factors of d , 1ε , 1 α , log κ , and ρ , where ‖µ‖2 ≤ ρ ; the accuracy guarantee holds with large constant probability . Without any prior information about the covariance , the best known approach is to estimate the mean by learning the entire distribution—both mean and covariance—which is the more difficult task considered in [ 2 , Theorem 4.6 ] . Doing so incurs an even worse dependence on the dimension : n & d2 α2 + d2 αε =⇒ ‖A ( x ) − µ‖Σ ≤ α . ( 3 ) Here & hides a universal multiplicative constant and logarithmic factors of 1δ and 1 α ; the accuracy guarantee holds with large constant probability . The Covariance-Estimation Bottleneck . The bottleneck in the algorithms above is privately obtaining a good spectral approximation to the covariance , i.e . a matrix A such that A Σ 2A . With such an estimate , one can apply the known-covariance approach in ( 1 ) . Without privacy constraints , the empirical covariance will have this spectral-approximation property when the sample size is n & d. However , all known private covariance estimators require n = Ω ( d3/2 ) samples , and there is evidence that this is an inherent limitation , as Ω ( d3/2 ) samples are necessary for solving this task for a worst-case data distribution [ 31 ] . Our Work : Sample-Efficient Private Mean Estimation . We circumvent this apparent difficulty of covariance estimation by designing an algorithm that adapts the noise it adds to the distribution ’ s covariance without actually providing an explicit covariance estimate , nearly matching the optimal sample complexity ( 1 ) for the known-covariance setting . Theorem 1.1 ( Informal ) . For α ≤ 1 , there is an ( ε , δ ) -differentially private estimator A ( · ) such that if x = ( x1 , . . . , xn ) are sampled from N ( µ , Σ ) for unknown µ and Σ of full rank , n & d α2 + d αε + log ( 1/δ ) ε =⇒ ‖A ( x ) − µ‖Σ ≤ α . The above guarantee holds with high probability over the sample x and the randomness of A . Here & hides a universal multiplicative constant and a logarithmic factor of 1α . For the formal statement , see Theorem 3.2 . Our estimator is based on privately sampling a point of large Tukey depth . Tukey depth generalizes the notion of quantiles to multiple dimensions ; it is known to be a good robust estimator of the Gaussian mean . The natural way to sample such a point privately is to use the exponential mechanism ( as in the concurrent work of [ 47 ] ) , but sampling from a distribution over the entire domain Rd will not have finite sample complexity . Our innovation is to sample from a data-dependent domain consisting only of points of large Tukey depth , which necessitates careful preprocessing and privacy analysis . We emphasize that the sample complexity of this estimator is optimal up to polylogarithmic factors . However , the estimator is not computationally efficient . An interesting open problem is to design an estimator matching the guarantee of Theorem 1.1 with running time polynomial in the dimension . Beyond Gaussian Distributions . A natural question is how much the assumption of Gaussian data can be relaxed without blowing up the sample complexity . Our second result is an alternative estimator , based on a completely different technique , that will give a similar guarantee for any distribution with subgaussian tails . For our purposes , we say that P with mean µ and covariance Σ is subgaussian if , for every direction u ∈ Rd , the tails of the distribution decay as fast as a univariate normal distribution with mean uTµ and variance CuTΣu for some constant C. That is , for every λ , E [ eλuT ( P−µ ) ] ≤ eCλ2 ( uTΣu ) /2 . More generally , our estimator works for any distribution such that the empirical covariance matrix converges rapidly to the population covariance matrix and typical samples are close to the mean in Mahalanobis distance . Theorem 1.2 ( Informal ) . For α ≤ 1 , there is an ( ε , δ ) -differentially private estimator A ( · ) such that if x = ( x1 , . . . , xn ) are sampled from any subgaussian distribution with unknown mean µ and unknown covariance Σ of full rank , n & d α2 + d log ( 1/δ ) αε2 =⇒ ‖A ( x ) − µ‖Σ ≤ α . The above guarantee holds with high probability over the sample x and the randomness of A . Here & hides a universal multiplicative constant and polylogarithmic factors of d , 1δ , 1 ε , and 1 α . For the formal statement , see Theorem 4.4 . This estimator is based on another simple approach—we perturb the empirical mean µx with noise scaled to Σx , where Σx is the exact ( not private ) empirical covariance . We show that this approach satisfies differential privacy if the data set satisfies certain concentration properties , which we enforce using a careful preprocessing step . Both of our estimators generalize beyond Gaussian distributions in different directions , not fully captured by our theorems . Although the Tukey depth estimator will only return an approximation to the mean when the distribution is symmetric and will not generalize to arbitrary subgaussian distributions , it returns an approximate median for distributions satisfying some natural regularity conditions . In contrast , the empirically rescaled estimator generalizes to distributions that are well-concentrated , in the sense that typical samples from the distribution are close to the mean in Mahalanobis distance with respect to the empirical covariance , which captures much more than just subgaussian distributions . Exploring the extent to which each estimator can be generalized is an interesting direction for future work . Limitations . Our estimators are not computationally efficient . We provide finite implementations which construct and search over a sufficiently fine grid , resulting in running time exponential in the dimension d and sample size n. We also stress that the error guarantees of our estimators only hold for families of distributions which satisfy certain regularity conditions , as described above . We leave these two interesting directions of improvement to future work . Societal Impact . Since our algorithms are inefficient , they will not see practical use without further research . More broadly , our work serves the goal of increasing the array of differentially private algorithms , and thereby enables safer access to privacy-sensitive data . Although such tools might support corporations and governments in harmful data-collection efforts , we believe that the existence of these tools is beneficial on balance .

Summary: This paper investigates mean estimation of high dimensional Gaussian (and sub-Gaussian) distributions under the constraint of differential privacy (DP). More specifically, the authors aim at privately estimating the mean of a high-dimensional Gaussian in the Mahalanobis distance (w.r.t. to the covariance matrix).   One simple approach for this task is to first privately estimating the covariance matrix (say up to a constant factor in spectral distance) after which the problem has known solutions. Unfortunately, this approach is suboptimal as estimating the covariance matrix is harder task. Thus, the authors provide a way to circumvent the issue and construct simple estimators whose sample complexity scales only linearly with the dimension.  The first algorithm is based on combination of the propose-test-release (PTR) framework, exponential mechanism, and the concept of Tukey depth. Roughly speaking, the authors show that by sampling from the distribution defined by the exponential mechanism with a score function based on the Tukey depth restricted to a data dependent set of possible outputs (some points with high Tukey depth), one can get an accurate and private algorithm for mean estimation. In order to maintain privacy while restricting the output to be a data-dependent output set, the authors make use of the PTR framework to test whether the specific dataset is ``safe’’ for use. They then show that when sampled from Gaussian data, a dataset will be safe with high probability, proving that the algorithm will often output an estimate of the mean. Finally, upon success the output will w.h.p. be a point of high Tukey depth, so the estimator is also accurate.   The second algorithm relaxes the requirement of strict Gaussian data to sub-Gaussian data. This algorithm is also based on PTR framework, with a clever application of "skewed" Gaussian noise addition. The standard Gaussian mechanism used in DP adds Gaussian noise scaled equally in all directions. The authors propose to use an additive Gaussian mechanism (to the empirical mean) that adds noise proportional to the empirical covariance of the data instead. Intuitively, this preserves the scale of the data (and thus preserves accuracy w.r.t. to the Mahalanobis distance). While this is not private for worst case datasets, the authors circumvent this by using the PTR to first test whether the dataset is ``good’’ (appropriately defined), and if need be, project the dataset to the set of good datasets before running the algorithm. Finally, they show that with high probability, sub-Gaussian data will be good and there will be no projection, thus the algorithm will output the empirical mean (perturbed with skewed Gaussian noise), that will be accurate w.r.t. the Mahalanobis distance.