A Central Limit Theorem for Differentially Private Query Answering

α p , where ‖x‖p is the standard ` p-norm , satisfies the conditions . Taking this perspective , we make use of the Cramer–Rao inequality and show a “ uncertainty principle ” -style result : the product of privacy parameter and the ` 2-loss of the mechanism is lower bounded by the dimension . Furthermore , the Gaussian mechanism achieves the constant-sharp optimal privacy-accuracy trade-off among all such noises . Our findings are corroborated by numerical experiments . 1 Introduction . Introduced in [ DMNS06 ] , to date differential privacy ( DP ) is perhaps the most popular privacy definition . One of the most important applications of differential privacy is to answer numeric queries . Given a function f of interest , which is also termed a query , our goal is to evaluate this ( potentially vector-valued ) query f on the sensitive data . To preserve privacy , a DP mechanism M working on a dataset D , in its simplest form , is defined as M ( D ) = f ( D ) + tX . ( 1 ) Above , X denotes the noise term and t is a scalar , which together are selected depending on the properties of the query f and the desired privacy level . Among these , perhaps the most popular examples are the Laplace mechanism and the Gaussian mechanism where the noise X follows the Laplace distribution and the Gaussian distribution , respectively . ∗the Institute for Data , Econometrics , Algorithms , and Learning 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Aside from privacy considerations , the most important criterion of an algorithm is arguably the estimation accuracy in the face of choosing , for example , between the Laplace mechanism or its Gaussian counterpart for a given problem . To be concrete , consider a real-valued query f with sensitivity 1—that is , ∆f = supD , D′ |f ( D ) − f ( D′ ) | = 1 , where the supremum is over all neighboring datasets D and D′ . Assuming ( ε , 0 ) -DP for the mechanism M , we are interested in minimizing its ` 2 loss defined as err ( M ) : = E ( M ( D ) − f ( D ) ) 2 = E ( tX ) 2 = t2EX2 . This question is commonly2 addressed by setting X to a standard Laplace random variable and t = ε−1 [ DMNS06 ] . This gives err ( M ) = 2ε−2 . Moving forward , we relax the privacy constraint from ( ε , 0 ) -DP to ( ε , δ ) -DP for some small δ . The canonical way , which was born together with the notion of ( ε , δ ) -DP , is to add Gaussian noise [ DKM+06 ] . A well-known result demonstrates that Gaussian mechanism withX being the standard normal and t = 1ε √ 2 log ( 1.25δ−1 ) is ( ε , δ ) -DP ( see , e.g. , [ DR14 ] ) . The ` 2-loss is err ( M ) = t2 = 2ε−2 · log ( 1.25δ−1 ) . A quick comparison between the two errors reveals a surprising message . The latter error 2ε−2 · log ( 1.25δ−1 ) is larger than the former 2ε−2 . In fact , the extra factor log ( 1.25δ−1 ) is already greater than 10 when δ = 10−5 . At least on the surface , this observation contradicts the fact that ( ε , δ ) -DP is a relaxation of ( ε , 0 ) -DP . Put differently , moving from Laplace to Gaussian , both privacy and accuracy get worse . Nevertheless , this contradiction suggests that we need a better alternative to the Gaussian mechanism instead of giving up the notion of ( ε , δ ) -DP . Indeed , the truncated Laplace mechanism has been proposed as a better alternative to achieve ( ε , δ ) -DP [ GDGK20 ] , which outperforms the Laplace mechanism in terms of estimation accuracy 3 . Motivated by these facts concerning the Laplace , Gaussian , and truncated Laplace mechanisms , one can not help asking : ( Q1 ) Why was the truncated Laplace mechanism not considered in the first place ? Are there any insights behind the design of such mechanisms ? ( Q2 ) More importantly , are these insights inherent for answering one-dimensional queries , or can we extend them to high-dimensional setting ? In this paper , we tackle these fundamental questions , beginning with explaining ( Q1 ) in Section 2 from the decision-theoretic perspective of DP [ WZ10 , KOV17 , DRS21 ] . However , our main focus is ( Q2 ) . In addressing this question , we uncover a seemingly surprising phenomenon — it is impossible to utilize the ( ε , δ ) privacy budget in high-dimensional problems the same way as the truncated Laplace mechanism utilizes it in the one-dimensional problem . More specifically , we show a central limit behavior of the noise-addition mechanism in high dimensions , which , roughly speaking , says that for general noise distributions , the corresponding mechanisms all behave like a Gaussian mechanism . The formal language of “ a mechanism behaves like the Gaussian mechanism ” has been set up in [ DRS21 ] , where a notion called Gaussian Differential Privacy ( GDP ) was proposed . Roughly speaking , a mechanism is µ-GDP if it offers as much privacy as adding N ( 0 , µ−2 ) noise to a sensitivity-1 query . As in the ( ε , δ ) -DP case , the smaller µ is , the stronger privacy is offered . To state our first main contribution , let f be an n-dimensional query and assume that its ` 2-sensitivity is 1 . Consider the noise addition mechanism M ( D ) = f ( D ) + tX where X has a log-concave density ∝ e−ϕ ( x ) on Rn . Let IX : = E [ ∇ϕ ( X ) ∇ϕ ( X ) T ] be the n × n Fisher information matrix and ‖IX‖2 be its operator norm . Theorem 1.1 ( Central Limit Theorem ( Informal version of Theorem 3.1 ) ) . Under certain conditions on ϕ , for t = µ−1 · √ ‖IX‖2 , the corresponding noise addition mechanism M defined in Eq . ( 1 ) is asymptotically µ-GDP as the dimension n → ∞ except for an o ( 1 ) fraction of directions of f ( D ) − f ( D′ ) . In particular , the norm power functions ϕ ( x ) = ‖x‖αp ( p , α > 1 ) satisfy these technical conditions . Note that this class already contains correlated noise , so the results in [ DRS21 ] do not apply here . Numerical results in Figure 1 shows that the convergence occurs for a dimension as small as 30 . 2If f is integer-valued , then the doubly geometric distribution is a better choice and yields an ` 2-loss of 1 2 sinh−2 ε 2 < 2ε−2 . In the so-called high privacy regime , i.e . ε → 0 , the two ` 2-losses have the same order in the sense that their ratio goes to 1 . 3One may blame the sub-optimality of the choice of t , but the problem remains even if the smallest possible t from [ BW18 ] is applied . We then elaborate on the condition “ o ( 1 ) fraction of f ( D ) −f ( D′ ) ” . Following the original definition , DP or GDP is a condition that needs to hold for arbitrary neighboring datasets D and D′ . This worst case perspective is exactly what prevents us to observe the central limit behavior . For example , consider a certain pair of datasets with f ( D ) = ( 0 , 0 , . . . , 0 ) and f ( D′ ) = ( 1 , 0 , . . . , 0 ) , then privacy is completely determined by the first marginal distribution of X , and the dimension n plays no role here . The “ o ( 1 ) fraction of f ( D ) − f ( D′ ) ” rules out the essentially low-dimension cases and reveals the truly high-dimensional behavior . In summary , Theorem 1.1 suggests that when the dimension is high , a large class of noise addition mechanisms behave like the Gaussian mechanism , and hence are doomed to a poor use of the given ( ε , δ ) privacy budget , in the same fashion as we have seen in the one-dimensional example . However , admitting the central limit phenomenon , our second theorem turns the table and characterizes the optimal privacy-accuracy trade-off and justifies the Gaussian mechanism . To see this , recall that the noise addition mechanism defined in Equation ( 1 ) is determined by the pair ( t , X ) . Both privacy and accuracy are jointly determined by t and X . Adopting the central limit theorem 1.1 , it is convenient to take an equivalent parametrization , which is ( µ , X ) , where µ is the desired ( asymptotic ) GDP parameter . Given X , the two parametrizations are related by t = µ−1 · √ ‖IX‖2 . Using parameters ( µ , X ) , the corresponding mechanism Mµ , X is given by Mµ , X ( D ) = f ( D ) + µ −1 · √ ‖IX‖2 ·X By Theorem 1.1 , it is asymptotically µ-GDP . The following theorem states in an “ uncertainty principle ” fashion that the privacy parameter and the error can not be small at the same time . Theorem 1.2 . As long as the Fisher information of X is defined , we have µ2 · err ( Mµ , X ) > n. The equality holds if X is n-dimensional standard Gaussian . Combining Theorems 1.1 and 1.2 , among all the noise that satisfies the conditions of Theorem 1.1 , Gaussian yields the constant-sharp optimal privacy-accuracy trade-off . As far as we know , this is the first result characterizing optimality with the sharp constant when the dimension is high . The privacy conclusion of Theorem 1.1 does not work for every pair of neighboring datasets , so it is worth noting that we do NOT intend to suggest this as a valid privacy guarantee . Instead , we present it as an interesting phenomenon that has been largely overlooked in the literature . Furthermore , this central limit theorem admits an elegant characterization of privacy-accuracy trade-off that is sharp in constant . From a theoretical point of view , the proof of Theorem 1.1 , as we shall see in later sections , involves non-linear functionals of high dimensional distributions . This type of results are , to the best of our knowledge , quite underexplored compared to linear functionals , so our results may serve as an additional motivation to study this type of questions . Related work There is a large body of literature on the characterization of privacy-accuracy tradeoff for query answering mechanisms . For the one-dimensional case , the constant-sharp optimal noise for ( ε , 0 ) -DP was shown to have a piece-wise constant density by [ GV16 ] . This complements our discussion in Figure 2 . When the dimension is high , only up-to-constant-factor optimality was known . In particular , [ BUV18 , SU17 ] confirm that Gaussian mechanism is minimax rate optimal under ( ε , δ ) DP by a novel lower bound technique . In addition , [ ENU20 ] also confirms the minimax optimality of Gaussian mechanism for linear queries with a refined notion of sensitivity . Our work extends this direction by taking the CLT perspective and providing an elegant constant-sharp optimality result . There are also works studying the up-to-constant-factor minimax optimality in other models , such as the ( sparse ) linear regression [ CWZ21 ] , generalized linear models [ CWZ20 , SSTT21 ] , Gaussian mixtures [ KSSU20 , ZZ21 ] and so on . In our work , we initialize the investigation in the ( simpler ) mean estimation problem , and leave the constant-sharp optimality in other problems for future work .

Summary: This paper studies the privacy-accuracy trade-off of answering a single n-dimensional query with ell-2 sensitivity 1, under differential privacy. This problem has been studied and the optimal noise distribution the Gaussian, and its upper and lower error bounds match up to constant factors. This paper however studies optimality in the limit n->infty and shows that the Gaussian mechanism is optimal even with respect to constant factors.   A central quantity to the approach and intuition in this work is the ROC function: using a known hypothesis testing view of DP, the ROC function T[M(D),M(D′)] measures the optimal error distinguishing M(D) and M(D’) for two neighboring datasets D, D’ and we say that M is (epsilon, delta)-DP if this function is above a fixed function f_(epsilon, delta) pointwise. Let G be the ROC function of the Gaussian mechanism (omitting privacy parameters).  Starting from the 1-dimensional case, the authors demonstrate that the optimal curve f_(epsilon, delta) and G are far whereas for example the ROC function of the truncated Laplace mechanism is closer to the optimal tradeoff, which they use to stress that the Gaussian mechanism is wasteful in 1-dimension and ask whether this is also the case for n->infty.  The main contribution of the paper is a central limit type of theorem that demonstrates that this is not the case: for all distributions of noise that satisfy a certain set of conditions, as n->infty, inf_{D,D’} T[M_n(D),M_n(D’)] -> G pointwise. That is, the sequence of mechanisms with the specified noise distributions approach the tradeoff of the Gaussian mechanism. This holds for almost all neighboring pairs of datasets, with the probability of the event that it does not happen going to 0, as n->infty. These conditions are satisfied for example by noise distributions of the form exp(-||x||^a_p) for any ell-p norm and power a. For these and more generally noise distributions of the form exp(-φ) where φ has finite second moments and Fisher information matrix, the authors also prove that the privacy parameter times the ell-2 error of any mechanism is lower bounded by the dimension n, with the inequality being tight when we choose the n-dimensional Gaussian as the noise distribution.