Covariance-Aware Private Mean Estimation Without Private Covariance Estimation

1 Introduction . Although the goal of statistics and machine learning is to infer properties of a population , there is a growing awareness that many statistical estimators and trained models reveal a concerning amount of information about their data set , which leads to significant concerns about the privacy of the individuals who have contributed sensitive information to that data set . These privacy violations have been demonstrated repeatedly via reconstruction attacks [ 21 , 30 , 27 , 46 ] , membership-inference attacks [ 37 , 52 , 10 , 32 , 53 , 59 ] , and instances of unwanted memorization of training data [ 13 , 14 , 34 , 7 ] . In order to realize the benefits of analyzing sensitive data sets , it is crucial to develop statistical 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . estimators and machine learning algorithms that make accurate inferences about the population but also protect the privacy of the individuals who contribute data . In this work we study statistical estimators that satisfy a condition called differential privacy [ 29 ] , which has become the standard criterion for individual privacy in statistics and machine learning . Informally , a differentially private algorithm guarantees that no attacker , regardless of their background knowledge or resources , can infer much more about any individual than they could have learned had that individual never contributed to the data set [ 45 ] . A long body of work shows that differential privacy is compatible with a wide range of tasks in statistics and machine learning , and it is now seeing deployment at companies like Google [ 33 , 6 , 57 ] , Apple [ 3 ] , Facebook [ 55 ] and LinkedIn [ 51 ] , as well as statistical agencies like the U.S. Census Bureau [ 1 , 36 ] . Background : Differentially Private Mean Estimation . We revisit differentially private estimators for one of the most fundamental tasks in all of statistics and machine learning—given x1 , . . . , xn ∈ Rd sampled i.i.d . from a distribution with mean µ ∈ Rd and covariance Σ ∈ Rd×d , estimate the mean µ . Mean estimation is not only an essential summary statistic in its own right , but also a building block for more sophisticated tasks like regression and stochastic optimization . Without privacy constraints , the natural solution is to output the empirical mean µx = 1n ∑ i xi . The natural way to state the sample-complexity guarantee of the empirical mean is n & d α2 =⇒ ‖µx − µ‖Σ ≤ α , where & hides a universal multiplicative constant , and the accuracy guarantee holds with large constant probability ( say , 0.99 ) . Importantly , ‖µx − µ‖Σ = ‖Σ−1/2 ( µx − µ ) ‖2 is the error in Mahalanobis distance scaled to the covariance Σ. Bounding the error in Mahalanobis distance implies that in every direction v , the squared error is proportional to the variance vTΣv in that direction . The Mahalanobis distance is the natural way to measure the error for mean estimation , since it tightly captures the uncertainty about the true mean and is preserved under affine transformations . Unfortunately , in high dimensions , releasing the empirical mean leads to concrete privacy breaches [ 32 , 41 ] . A natural question is thus : can we design a differentially private mean estimator that performs nearly as well as the empirical mean ? Without making additional assumptions , the answer turns out to be no—every differentially private estimator incurs a large overhead in sample complexity compared to the empirical mean [ 42 ] . However , it is known that if we further assume that the distribution satisfies some additional concentration properties , we can do much better [ 54 , 44 , 41 , 42 ] . In this work , we focus on one class of well-concentrated distributions , those that are Gaussian ( or , in some of our results , subgaussian ) . Although assuming Gaussian data is restrictive , it is a natural starting point for understanding the complexity of estimation when the distribution is not pathological . Moreover , even in the Gaussian case , one can not obtain error comparable to that of the empirical mean unless n & d [ 10 , 32 , 41 ] , so we will also focus on the case where the sample size is at least as large as the dimension . For Gaussian data , if the analyst has prior information about the covariance matrix in the form of a matrix A and bound κ ≥ 1 such that A Σ κA,1 then there is a folklore private estimator A ( x ) , based on a line of work initiated by Karwa and Vadhan [ 44 , 41 , 5 , 2 ] , that finds an approximate range for the data , truncates the points to within that range , and runs the Gaussian mechanism on the resulting empirical mean . This estimator achieves n & d α2 + d √ κ αε =⇒ ‖A ( x ) − µ‖Σ ≤ α , ( 1 ) where ε is the privacy parameter controlling the level of privacy ( see Definition 2.2 ) , with stronger privacy as ε→ 0.2 Here & hides a universal multiplicative constant and polylogarithmic factors of d , 1δ , 1 ε and 1 α ; the accuracy guarantee holds with large constant probability . We can interpret this result as showing that the additional cost of privacy is small provided that the user has a strong a priori 1Given two covariance matrices A , B ∈ Rd×d , the notation A B indicates that in every direction v ∈ Rd the variance vTBv is at least as large as vTAv . More generally , A B iff B −A is positive semidefinite . 2To simplify the discussion , we focus only on the ε parameter , although our results and many of those we discuss require relaxations of differential privacy such as approximate [ 28 ] or concentrated [ 26 , 8 ] differential privacy , which have different parameterizations . bound on the covariance so that κ is small ( e.g . κ is a constant ) , and also that the privacy guarantee is not too strong ( e.g . ε ≥ α ) . In particular , setting κ = 1 corresponds to the known-covariance setting , where the guarantee in ( 1 ) is known to be minimax optimal up to polylogarithmic factors [ 32 , 41 ] among all differentially private estimators . However , the sample complexity in ( 1 ) grows asymptotically with √ κ , a large price to pay for the user ’ s uncertainty . Intuitively , this degradation arises because the algorithm perturbs the empirical mean µx with noise from a spherical Gaussian distribution , whose magnitude must be proportional to the largest variance in any direction , so the noise is unnecessarily large in the directions with small variance . In contrast , when the user is very uncertain about the covariance , there are estimators with a weaker dependence on κ but a superlinear dependence on the dimension . In particular , there is an estimator [ 41 , Theorem 4.3 ] with an error guarantee of the form n & d α2 + d αε + d3/2 log1/2 κ ε =⇒ ‖A ( x ) − µ‖Σ ≤ α . ( 2 ) Here & hides a universal multiplicative constant and logarithmic factors of d , 1ε , 1 α , log κ , and ρ , where ‖µ‖2 ≤ ρ ; the accuracy guarantee holds with large constant probability . Without any prior information about the covariance , the best known approach is to estimate the mean by learning the entire distribution—both mean and covariance—which is the more difficult task considered in [ 2 , Theorem 4.6 ] . Doing so incurs an even worse dependence on the dimension : n & d2 α2 + d2 αε =⇒ ‖A ( x ) − µ‖Σ ≤ α . ( 3 ) Here & hides a universal multiplicative constant and logarithmic factors of 1δ and 1 α ; the accuracy guarantee holds with large constant probability . The Covariance-Estimation Bottleneck . The bottleneck in the algorithms above is privately obtaining a good spectral approximation to the covariance , i.e . a matrix A such that A Σ 2A . With such an estimate , one can apply the known-covariance approach in ( 1 ) . Without privacy constraints , the empirical covariance will have this spectral-approximation property when the sample size is n & d. However , all known private covariance estimators require n = Ω ( d3/2 ) samples , and there is evidence that this is an inherent limitation , as Ω ( d3/2 ) samples are necessary for solving this task for a worst-case data distribution [ 31 ] . Our Work : Sample-Efficient Private Mean Estimation . We circumvent this apparent difficulty of covariance estimation by designing an algorithm that adapts the noise it adds to the distribution ’ s covariance without actually providing an explicit covariance estimate , nearly matching the optimal sample complexity ( 1 ) for the known-covariance setting . Theorem 1.1 ( Informal ) . For α ≤ 1 , there is an ( ε , δ ) -differentially private estimator A ( · ) such that if x = ( x1 , . . . , xn ) are sampled from N ( µ , Σ ) for unknown µ and Σ of full rank , n & d α2 + d αε + log ( 1/δ ) ε =⇒ ‖A ( x ) − µ‖Σ ≤ α . The above guarantee holds with high probability over the sample x and the randomness of A . Here & hides a universal multiplicative constant and a logarithmic factor of 1α . For the formal statement , see Theorem 3.2 . Our estimator is based on privately sampling a point of large Tukey depth . Tukey depth generalizes the notion of quantiles to multiple dimensions ; it is known to be a good robust estimator of the Gaussian mean . The natural way to sample such a point privately is to use the exponential mechanism ( as in the concurrent work of [ 47 ] ) , but sampling from a distribution over the entire domain Rd will not have finite sample complexity . Our innovation is to sample from a data-dependent domain consisting only of points of large Tukey depth , which necessitates careful preprocessing and privacy analysis . We emphasize that the sample complexity of this estimator is optimal up to polylogarithmic factors . However , the estimator is not computationally efficient . An interesting open problem is to design an estimator matching the guarantee of Theorem 1.1 with running time polynomial in the dimension . Beyond Gaussian Distributions . A natural question is how much the assumption of Gaussian data can be relaxed without blowing up the sample complexity . Our second result is an alternative estimator , based on a completely different technique , that will give a similar guarantee for any distribution with subgaussian tails . For our purposes , we say that P with mean µ and covariance Σ is subgaussian if , for every direction u ∈ Rd , the tails of the distribution decay as fast as a univariate normal distribution with mean uTµ and variance CuTΣu for some constant C. That is , for every λ , E [ eλuT ( P−µ ) ] ≤ eCλ2 ( uTΣu ) /2 . More generally , our estimator works for any distribution such that the empirical covariance matrix converges rapidly to the population covariance matrix and typical samples are close to the mean in Mahalanobis distance . Theorem 1.2 ( Informal ) . For α ≤ 1 , there is an ( ε , δ ) -differentially private estimator A ( · ) such that if x = ( x1 , . . . , xn ) are sampled from any subgaussian distribution with unknown mean µ and unknown covariance Σ of full rank , n & d α2 + d log ( 1/δ ) αε2 =⇒ ‖A ( x ) − µ‖Σ ≤ α . The above guarantee holds with high probability over the sample x and the randomness of A . Here & hides a universal multiplicative constant and polylogarithmic factors of d , 1δ , 1 ε , and 1 α . For the formal statement , see Theorem 4.4 . This estimator is based on another simple approach—we perturb the empirical mean µx with noise scaled to Σx , where Σx is the exact ( not private ) empirical covariance . We show that this approach satisfies differential privacy if the data set satisfies certain concentration properties , which we enforce using a careful preprocessing step . Both of our estimators generalize beyond Gaussian distributions in different directions , not fully captured by our theorems . Although the Tukey depth estimator will only return an approximation to the mean when the distribution is symmetric and will not generalize to arbitrary subgaussian distributions , it returns an approximate median for distributions satisfying some natural regularity conditions . In contrast , the empirically rescaled estimator generalizes to distributions that are well-concentrated , in the sense that typical samples from the distribution are close to the mean in Mahalanobis distance with respect to the empirical covariance , which captures much more than just subgaussian distributions . Exploring the extent to which each estimator can be generalized is an interesting direction for future work . Limitations . Our estimators are not computationally efficient . We provide finite implementations which construct and search over a sufficiently fine grid , resulting in running time exponential in the dimension d and sample size n. We also stress that the error guarantees of our estimators only hold for families of distributions which satisfy certain regularity conditions , as described above . We leave these two interesting directions of improvement to future work . Societal Impact . Since our algorithms are inefficient , they will not see practical use without further research . More broadly , our work serves the goal of increasing the array of differentially private algorithms , and thereby enables safer access to privacy-sensitive data . Although such tools might support corporations and governments in harmful data-collection efforts , we believe that the existence of these tools is beneficial on balance .

Summary: This paper considered the DP mean estimation problem with unknown covariance under Mahalanobis norm. In the known covariance setting, previous work can estimate gaussian mean near optimally. But in the unknown covariance setting, previous algorithms either need to pay an additional conditional number factor, or require d^3/2 samples to estimate the covariance which are both unsatisfactory. The key difficulty of the problem is getting Mahalanobis norm guarantee without first estimating the covariance privately.   The authors present two different (exponential time) algorithms for Gaussian and sub-Gaussian settings. The gaussian algorithm is based on an exponential mechanism with Tukey depth score. Although this approach has been used in [45] before, they need to choose the sampling region adaptively based on the data in order to achieve a good Mahalanobis guarantee. This brings additional difficulty in making the algorithm private, and they use the classical propose-test-release framework to obtain privacy guarantees. The second estimator uses the empirically rescaled Gaussian mechanism, where the gaussian noise is scaled by the empirical covariance such that Mahalanobis guarantee is naturally achieved. This estimator can be applied more generally on sub-gaussian distribution, though the sample complexity has a d/\alpha\eps^2 instead of the optimal d/\alpha\eps.