Instance-optimal Mean Estimation Under Differential Privacy

1 Introduction . Mean estimation is one of the most fundamental problems in statistics , optimization , and machine learning . However , privacy concerns forbid us from using the exact mean in these applications , and the problem of how to achieve the smallest error under a given privacy model has received considerable attention in the literature . Differential privacy ( DP ) is a rigorous mathematical definition for protecting individual privacy and has emerged as the golden standard in privacy-preserving data analysis nowadays , which has been deployed by Apple [ 16 ] , Google [ 22 ] , and Microsoft [ 17 ] . Given a data set D : = { xi } i∈ [ n ] ⊂ Ud , where U = [ u ] , i.e. , each coordinate of the input vector is an integer ( real-valued coordinates can be handled by quantization ; see remark 1 ) , our goal is to obtain a differentially private estimation M ( D ) for the mean f ( D ) = 1n ∑n i=1 xi with small ` 2 error ‖M ( D ) − f ( D ) ‖2 . Because f ( · ) has global ` 2 sensitivity GSf = √ du/n , the standard DP mechanism just adds Gaussian noise scaled to GSf to each coordinate of f ( D ) , which results in an ` 2 error proportional to du/n . This simple mechanism is worst-case optimal [ 28 ] , but it is certainly undesirable in practice , as people often conservatively use a large u ( e.g. , u = 232 ) but the actual dataset D may have much smaller coordinates . Instead , the clipped-mean estimator [ 1 ] ( see Section 3.1 for details ) has been widely used as an effective heuristic , but two questions remain unresolved : ( 1 ) how to choose the clipping threshold C ; and ( 2 ) if it can yield any optimality guarantees . We answer these questions in a fairly strong sense in this paper . 1.1 Instance Optimality . As worst-case optimality is theoretically trivial and practically meaningless for the mean estimation problem when the global sensitivity is too large , one may aim at instance optimality . More precisely , letM be the class of DP mechanisms and let Rins ( D ) : = inf M ′∈M inf { ξ | Pr [ ‖M ′ ( D ) − f ( D ) ‖2 ≤ ξ ] ≥ 2/3 } 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . be the smallest error anyM ′ can achieve ( with constant probability ) onD , then the standard definition of instance optimality requires us to design an M such that Pr [ ‖M ( D ) − f ( D ) ‖2 ≤ c · Rins ( D ) ] ≥ 2/3 ( 1 ) for every D , where c is called the optimality ratio . Unfortunately , for any D , one can design a trivial M ′ ( · ) ≡ f ( D ) that has 0 error on D ( but fails miserably on other instances ) , soRins ( · ) ≡ 0 , which rules out instance-optimal DP mechanisms by a standard argument [ 21 ] . SinceRins ( · ) is unachievable , relaxed versions can be considered . The above trivialM ′ exists because it is only required to work well on one instance D. Imposing higher requirements on M ′ would yield relaxed notions of instance optimality . One natural requirement is that M ′ should work well not just on D , but also on its neighbors , i.e. , we raise the target error fromRins ( D ) to Rnbr ( D ) : = inf M ′∈M sup D′ : dham ( D , D′ ) ≤1 inf { ξ | Pr [ ‖M ′ ( D′ ) − f ( D′ ) ‖2 ≤ ξ ] ≥ 2/3 } . Vahdan [ 35 ] observes thatRnbr ( D ) is exactly LSf ( D ) , the local sensitivity of f at D , up to constant factors . However , LSf ( · ) may not be an appropriate target to shoot at , depending on what f is . For the MEDIAN problem , LSf ( D ) = 0 for certain D ’ s and no DP mechanisms can achieve this error [ 33 ] , while for mean estimation , LSf ( D ) = Θ ( GSf ) = Θ ( √ du/n ) for all D , so this relaxation turns instance optimality into worst-case optimality . The reason why the above relaxation is “ too much ” for the mean estimation problem is that D′ may change one vector of D arbitrarily , e.g. , from ( 0 , . . . , 0 ) to ( u , . . . , u ) . We restrict this . More precisely , letting supp ( D ) denote the set of distinct vectors in D , we consider the target error Rin-nbr ( D ) : = inf M ′∈M sup D′ : dham ( D , D′ ) ≤1 , supp ( D′ ) ⊆supp ( D ) inf { ξ | Pr [ ‖M ′ ( D′ ) −f ( D′ ) ‖2 ≤ ξ ] ≥ 2/3 } , namely , we require M ′ to work well only on D and its in-neighbors , in which a vector can only be changed to another one already existing in D. Correspondingly , an instance-optimal M ( w.r.t . the in-neighborhood ) is one such that ( 1 ) holds whereRins is replaced byRin-nbr . We make a few notes on this notion of instance optimality : ( 1 ) This optimality is only about the utility of the mechanism , not its privacy . We still require the mechanism to satisfy the DP requirement between any D , D′ such that dham ( D , D′ ) = 1 , not necessarily one and its in-neighbors . ( 2 ) In general , a smaller neighborhood leads to a stronger notion of instance optimality . Thus , the optimality using in-neighbors is stronger than that using all neighbors , which is in turn stronger than worst-case optimality ( i.e. , D′ can be any instance ) , while the latter two are actually the same for the mean estimation problem . ( 3 ) For an instance-optimal M ( by our notion ) , there still exist D , M ′ such that M ′ does better on D than M , but it is not possible for M ′ to achieve a smaller error than the error of M on D over all in-neighbor of D. This is more meaningful than ranging over all neighbors of D , some of which ( e.g. , one with ( u , . . . , u ) as a datum ) are unlikely to be the actual instances encountered in practice . 1.2 Our Results . To design an M ( D ) for the mean function f ( D ) = 1n ∑n i=1 xi that achieves an error w.r.t.Rin-nbr ( D ) for allD , we need an upper bound and a lower bound . For the lower bound , we show thatRin-nbr ( D ) = Ω ( w ( D ) /n ) , where w ( D ) : = max1≤i < j≤n ‖xi − xj‖2 is the diameter of D. Thus , from the upper bound side , it suffices to show that the mechanism ’ s error is bounded by c ·w ( D ) /n . This is achieved in two steps . First , we use the clipped-mean estimator , but find the clipping threshold C that optimizes its bias-variance trade-off , which is a certain quantile of the norms of the vectors in D. However , we can not use the optimal C directly , as it would violate DP . Thus , we use a simple binary search based algorithm that can find any specific quantile privately with an optimal rank error . This results in a DP mechanism with error Õ ( √ d/ρ ) ·r ( D ) /n , where r ( D ) : = maxi ‖xi‖2 and ρ is the privacy parameter ( formal definition given in Section 2 ) . To reduce the error from r ( D ) to w ( D ) , in the second step , we rotate and shift D into a D̃ such that r ( D̃ ) = O ( w ( D ) ) w.h.p. , and apply the clipped-mean estimator ( with our privatized optimal clipping threshold ) on D̃ , leading to an error of Õ ( √ d/ρ ) · w ( D ) /n for n = Ω̃ ( √ d/ρ ) . We also show that the optimality ratio c = Õ ( √ d/ρ ) is optimal , i.e. , any mechanism M ( D ) having error c · w ( D ) /n for all D must have c = Ω̃ ( √ d/ρ ) for ρ < Õ ( √ d/n ) . Our mechanism has the following applications : ( 1 ) It can be applied directly to statistical mean estimation , where the vectors in D are i.i.d . samples from a certain distribution and one would like to estimate the mean of the distribution ( in contrast , the version defined above is referred to as empirical mean estimation ) . For concreteness , we show how this is done for the multivariate Gaussian distributions N ( µ , Σ ) . For the case Σ = I , our algorithm achieves an ` 2 error of α using n = Õ ( dα2 + d α √ ρ ) samples for α ≤ O ( 1 ) , matching the optimal bound in the statistical setting [ 8 ] . For a non-identity , unknown Σ , the error is proportional to ‖Σ1/2‖2 as in [ 8 ] . Our mechanism requires only crude a priori bounds on µ and Σ ( i.e. , the error depends on these bounds logarithmically ) , while [ 8 ] needs a constant-factor approximation of Σ , which can be obtained using n = Ω̃ ( d3/2/ √ ρ ) samples [ 26 ] . Note that this can be a √ d-factor higher than the sample complexity of mean estimation . Fundamentally , estimating Σ is harder than estimating µ , and we bypass the former so as to retain the same sample complexity of the latter . In practice , estimating Σ first would consume the privacy budget from the mean estimation problem itself . On the other hand , the benefit of estimating Σ first is that one can obtain an error guarantee under the Mahalanobis distance [ 26 ] , which can not be achieved by our method . ( 2 ) By simply changing the primitive operations , our mechanism easily extends to the local and shuffle model of differential privacy . In doing so , we also extend the one-dimensional summation/mean estimation protocol in the shuffle model [ 5 ] to high dimensions . In addition to the theoretical optimality , our mechanism is also simple and practical . Most importantly , there is no ( internal ) parameter to tune . Yet , our experimental results demonstrate that our mechanism outperforms the state-of-the-art algorithm [ 8 ] with the best parameters tuned for each specific setting . 1.3 Related Work . Asi and Duchi [ 4 ] recently initialized the study on instance optimality under DP . They propose two ways to relax ( equivalently , strengthen the requirement on M ′ ) the strict instance optimality , which is unachievable . The first is to require M ′ to be unbiased . This is not appropriate for mean estimation , since many estimators , including clipped-mean , is not unbiased . The second is to require M ′ to work well over all the r-distance neighbors of D for r ≥ 1 . Thus , their optimality is weaker than using Rnbr ( · ) , hence not appropriate for the mean estimation problem ( i.e. , their optimality is the same as worst-case optimality ) . Instance optimality has not been studied in the local or shuffle model ; existing protocols in these two models [ 7 , 19 , 5 ] all have errors proportional to the global sensitivity . How to choose the clipping threshold C for the clipped mean estimator has been extensively studied [ 2 , 3 , 34 , 32 ] , but existing methods do not offer any optimality guarantees . In particular , Andrew et al . [ 3 ] also use a quantile ( actually , median ) as C , but as we shall see , median is actually not the optimal choice . Furthermore , they use online gradient descent to find a privatized quantile , which does not have any theoretical error guarantees . Amin et al . [ 2 ] attempt to select an optimal quantile as the clipping threshold to truncate the number of contributions from each user , instead of clipping the actual samples in high dimensions as in our paper . In the statistical setting , where the data are i.i.d . samples from some specific distribution , there are numerous methods [ 26 , 8 , 27 , 29 ] that can avoid an error proportional to the global sensitivity , by exploiting the concentration property of the distribution . In particular , Biswas et al . [ 8 ] provide a simple and practical mechanism for multivariate Gaussian data . Levy et al . [ 30 ] propose a private mean estimator with error scaling with the concentration radius τ of the distribution rather than the entire range , but their algorithm requires τ to be publicly known in advance . In the local model , the algorithm in [ 23 ] uses a quantile estimation procedure based on binary search as a subroutine for one-dimensional Gaussian data . Very recently , the relationship between the error of mean estimation and the diameter of the dataset has been exploited in [ 15 ] for low-communication protocols , but they do not consider privacy . Our DP protocols in the local and shuffle models have communication cost Õ ( d ) per user ( we do not state the communication costs in the theorems as they are not our major concern ) ; it would be interesting to see if ideas from [ 15 ] can be used to reduce it further .

Summary: This paper does differentially private mean estimation for empirical estimation, as opposed to statistical estimation (statistical estimation is an application of their work). It focusses on instance optimal mean estimation by introducing a new target error function of the dataset, $\mathcal{R}_{\text{in-nbr}}$. It shows both and upper bound and a lower bound for this problem. It has both theoretical and empirical results.  The theoretical results say that for this measure of error, the optimal error scales according to the diameter of the dataset in question. To prove the upper bound, the authors first use a clipped-mean-estimator, with the clipping threshold decided via a simple derivative of the error. Since that clipping estimate is a function of the dataset, it could not be used directly. So, they came up with a simple quantile estimator to do the same. They, however, first performed a random rotation, followed by a translation to fix scaling issues. Then they finally use the aforementioned clipped-mean-estimator.  They finally showed application in statistical mean estimation of high-dimensional Gaussians. They showed that they require very weak prior knowledge of the covariance matrix of the Gaussian for this. They finally compared their empirical results with the Coinpress algorithm of Biswas et al.  They extend these results to the local and shuffle models of DP, too.