Instance-optimal Mean Estimation Under Differential Privacy

1 Introduction . Mean estimation is one of the most fundamental problems in statistics , optimization , and machine learning . However , privacy concerns forbid us from using the exact mean in these applications , and the problem of how to achieve the smallest error under a given privacy model has received considerable attention in the literature . Differential privacy ( DP ) is a rigorous mathematical definition for protecting individual privacy and has emerged as the golden standard in privacy-preserving data analysis nowadays , which has been deployed by Apple [ 16 ] , Google [ 22 ] , and Microsoft [ 17 ] . Given a data set D : = { xi } i∈ [ n ] ⊂ Ud , where U = [ u ] , i.e. , each coordinate of the input vector is an integer ( real-valued coordinates can be handled by quantization ; see remark 1 ) , our goal is to obtain a differentially private estimation M ( D ) for the mean f ( D ) = 1n ∑n i=1 xi with small ` 2 error ‖M ( D ) − f ( D ) ‖2 . Because f ( · ) has global ` 2 sensitivity GSf = √ du/n , the standard DP mechanism just adds Gaussian noise scaled to GSf to each coordinate of f ( D ) , which results in an ` 2 error proportional to du/n . This simple mechanism is worst-case optimal [ 28 ] , but it is certainly undesirable in practice , as people often conservatively use a large u ( e.g. , u = 232 ) but the actual dataset D may have much smaller coordinates . Instead , the clipped-mean estimator [ 1 ] ( see Section 3.1 for details ) has been widely used as an effective heuristic , but two questions remain unresolved : ( 1 ) how to choose the clipping threshold C ; and ( 2 ) if it can yield any optimality guarantees . We answer these questions in a fairly strong sense in this paper . 1.1 Instance Optimality . As worst-case optimality is theoretically trivial and practically meaningless for the mean estimation problem when the global sensitivity is too large , one may aim at instance optimality . More precisely , letM be the class of DP mechanisms and let Rins ( D ) : = inf M ′∈M inf { ξ | Pr [ ‖M ′ ( D ) − f ( D ) ‖2 ≤ ξ ] ≥ 2/3 } 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . be the smallest error anyM ′ can achieve ( with constant probability ) onD , then the standard definition of instance optimality requires us to design an M such that Pr [ ‖M ( D ) − f ( D ) ‖2 ≤ c · Rins ( D ) ] ≥ 2/3 ( 1 ) for every D , where c is called the optimality ratio . Unfortunately , for any D , one can design a trivial M ′ ( · ) ≡ f ( D ) that has 0 error on D ( but fails miserably on other instances ) , soRins ( · ) ≡ 0 , which rules out instance-optimal DP mechanisms by a standard argument [ 21 ] . SinceRins ( · ) is unachievable , relaxed versions can be considered . The above trivialM ′ exists because it is only required to work well on one instance D. Imposing higher requirements on M ′ would yield relaxed notions of instance optimality . One natural requirement is that M ′ should work well not just on D , but also on its neighbors , i.e. , we raise the target error fromRins ( D ) to Rnbr ( D ) : = inf M ′∈M sup D′ : dham ( D , D′ ) ≤1 inf { ξ | Pr [ ‖M ′ ( D′ ) − f ( D′ ) ‖2 ≤ ξ ] ≥ 2/3 } . Vahdan [ 35 ] observes thatRnbr ( D ) is exactly LSf ( D ) , the local sensitivity of f at D , up to constant factors . However , LSf ( · ) may not be an appropriate target to shoot at , depending on what f is . For the MEDIAN problem , LSf ( D ) = 0 for certain D ’ s and no DP mechanisms can achieve this error [ 33 ] , while for mean estimation , LSf ( D ) = Θ ( GSf ) = Θ ( √ du/n ) for all D , so this relaxation turns instance optimality into worst-case optimality . The reason why the above relaxation is “ too much ” for the mean estimation problem is that D′ may change one vector of D arbitrarily , e.g. , from ( 0 , . . . , 0 ) to ( u , . . . , u ) . We restrict this . More precisely , letting supp ( D ) denote the set of distinct vectors in D , we consider the target error Rin-nbr ( D ) : = inf M ′∈M sup D′ : dham ( D , D′ ) ≤1 , supp ( D′ ) ⊆supp ( D ) inf { ξ | Pr [ ‖M ′ ( D′ ) −f ( D′ ) ‖2 ≤ ξ ] ≥ 2/3 } , namely , we require M ′ to work well only on D and its in-neighbors , in which a vector can only be changed to another one already existing in D. Correspondingly , an instance-optimal M ( w.r.t . the in-neighborhood ) is one such that ( 1 ) holds whereRins is replaced byRin-nbr . We make a few notes on this notion of instance optimality : ( 1 ) This optimality is only about the utility of the mechanism , not its privacy . We still require the mechanism to satisfy the DP requirement between any D , D′ such that dham ( D , D′ ) = 1 , not necessarily one and its in-neighbors . ( 2 ) In general , a smaller neighborhood leads to a stronger notion of instance optimality . Thus , the optimality using in-neighbors is stronger than that using all neighbors , which is in turn stronger than worst-case optimality ( i.e. , D′ can be any instance ) , while the latter two are actually the same for the mean estimation problem . ( 3 ) For an instance-optimal M ( by our notion ) , there still exist D , M ′ such that M ′ does better on D than M , but it is not possible for M ′ to achieve a smaller error than the error of M on D over all in-neighbor of D. This is more meaningful than ranging over all neighbors of D , some of which ( e.g. , one with ( u , . . . , u ) as a datum ) are unlikely to be the actual instances encountered in practice . 1.2 Our Results . To design an M ( D ) for the mean function f ( D ) = 1n ∑n i=1 xi that achieves an error w.r.t.Rin-nbr ( D ) for allD , we need an upper bound and a lower bound . For the lower bound , we show thatRin-nbr ( D ) = Ω ( w ( D ) /n ) , where w ( D ) : = max1≤i < j≤n ‖xi − xj‖2 is the diameter of D. Thus , from the upper bound side , it suffices to show that the mechanism ’ s error is bounded by c ·w ( D ) /n . This is achieved in two steps . First , we use the clipped-mean estimator , but find the clipping threshold C that optimizes its bias-variance trade-off , which is a certain quantile of the norms of the vectors in D. However , we can not use the optimal C directly , as it would violate DP . Thus , we use a simple binary search based algorithm that can find any specific quantile privately with an optimal rank error . This results in a DP mechanism with error Õ ( √ d/ρ ) ·r ( D ) /n , where r ( D ) : = maxi ‖xi‖2 and ρ is the privacy parameter ( formal definition given in Section 2 ) . To reduce the error from r ( D ) to w ( D ) , in the second step , we rotate and shift D into a D̃ such that r ( D̃ ) = O ( w ( D ) ) w.h.p. , and apply the clipped-mean estimator ( with our privatized optimal clipping threshold ) on D̃ , leading to an error of Õ ( √ d/ρ ) · w ( D ) /n for n = Ω̃ ( √ d/ρ ) . We also show that the optimality ratio c = Õ ( √ d/ρ ) is optimal , i.e. , any mechanism M ( D ) having error c · w ( D ) /n for all D must have c = Ω̃ ( √ d/ρ ) for ρ < Õ ( √ d/n ) . Our mechanism has the following applications : ( 1 ) It can be applied directly to statistical mean estimation , where the vectors in D are i.i.d . samples from a certain distribution and one would like to estimate the mean of the distribution ( in contrast , the version defined above is referred to as empirical mean estimation ) . For concreteness , we show how this is done for the multivariate Gaussian distributions N ( µ , Σ ) . For the case Σ = I , our algorithm achieves an ` 2 error of α using n = Õ ( dα2 + d α √ ρ ) samples for α ≤ O ( 1 ) , matching the optimal bound in the statistical setting [ 8 ] . For a non-identity , unknown Σ , the error is proportional to ‖Σ1/2‖2 as in [ 8 ] . Our mechanism requires only crude a priori bounds on µ and Σ ( i.e. , the error depends on these bounds logarithmically ) , while [ 8 ] needs a constant-factor approximation of Σ , which can be obtained using n = Ω̃ ( d3/2/ √ ρ ) samples [ 26 ] . Note that this can be a √ d-factor higher than the sample complexity of mean estimation . Fundamentally , estimating Σ is harder than estimating µ , and we bypass the former so as to retain the same sample complexity of the latter . In practice , estimating Σ first would consume the privacy budget from the mean estimation problem itself . On the other hand , the benefit of estimating Σ first is that one can obtain an error guarantee under the Mahalanobis distance [ 26 ] , which can not be achieved by our method . ( 2 ) By simply changing the primitive operations , our mechanism easily extends to the local and shuffle model of differential privacy . In doing so , we also extend the one-dimensional summation/mean estimation protocol in the shuffle model [ 5 ] to high dimensions . In addition to the theoretical optimality , our mechanism is also simple and practical . Most importantly , there is no ( internal ) parameter to tune . Yet , our experimental results demonstrate that our mechanism outperforms the state-of-the-art algorithm [ 8 ] with the best parameters tuned for each specific setting . 1.3 Related Work . Asi and Duchi [ 4 ] recently initialized the study on instance optimality under DP . They propose two ways to relax ( equivalently , strengthen the requirement on M ′ ) the strict instance optimality , which is unachievable . The first is to require M ′ to be unbiased . This is not appropriate for mean estimation , since many estimators , including clipped-mean , is not unbiased . The second is to require M ′ to work well over all the r-distance neighbors of D for r ≥ 1 . Thus , their optimality is weaker than using Rnbr ( · ) , hence not appropriate for the mean estimation problem ( i.e. , their optimality is the same as worst-case optimality ) . Instance optimality has not been studied in the local or shuffle model ; existing protocols in these two models [ 7 , 19 , 5 ] all have errors proportional to the global sensitivity . How to choose the clipping threshold C for the clipped mean estimator has been extensively studied [ 2 , 3 , 34 , 32 ] , but existing methods do not offer any optimality guarantees . In particular , Andrew et al . [ 3 ] also use a quantile ( actually , median ) as C , but as we shall see , median is actually not the optimal choice . Furthermore , they use online gradient descent to find a privatized quantile , which does not have any theoretical error guarantees . Amin et al . [ 2 ] attempt to select an optimal quantile as the clipping threshold to truncate the number of contributions from each user , instead of clipping the actual samples in high dimensions as in our paper . In the statistical setting , where the data are i.i.d . samples from some specific distribution , there are numerous methods [ 26 , 8 , 27 , 29 ] that can avoid an error proportional to the global sensitivity , by exploiting the concentration property of the distribution . In particular , Biswas et al . [ 8 ] provide a simple and practical mechanism for multivariate Gaussian data . Levy et al . [ 30 ] propose a private mean estimator with error scaling with the concentration radius τ of the distribution rather than the entire range , but their algorithm requires τ to be publicly known in advance . In the local model , the algorithm in [ 23 ] uses a quantile estimation procedure based on binary search as a subroutine for one-dimensional Gaussian data . Very recently , the relationship between the error of mean estimation and the diameter of the dataset has been exploited in [ 15 ] for low-communication protocols , but they do not consider privacy . Our DP protocols in the local and shuffle models have communication cost Õ ( d ) per user ( we do not state the communication costs in the theorems as they are not our major concern ) ; it would be interesting to see if ideas from [ 15 ] can be used to reduce it further .

Summary: The paper studies instance-optimal mean estimation in the central model of differential privacy, with straightforward extensions to the shuffle and local models. The specific notion of instance optimality used is, I believe, original to this paper: the goal is to find an algorithm minimizing error over a given database D and a restricted neighborhood of databases that are neighbors and only include elements also present in D. The algorithm consists of four steps:  1) randomly rotate the database, 2) center it using a privately estimated component-wise median, 3) privately estimate a certain quantile of the L_2 norm of the resulting database and clip data to that estimated L_2 norm, 4) take a noisy empirical mean of the clipped data and de-bias.  The paper gives instance-specific upper and lower bounds that are nearly matching, up to logarithmic factors in the dimension and a discretization parameter. Experiments suggest that the algorithm performs well against existing methods.