A Central Limit Theorem for Differentially Private Query Answering

α p , where ‖x‖p is the standard ` p-norm , satisfies the conditions . Taking this perspective , we make use of the Cramer–Rao inequality and show a “ uncertainty principle ” -style result : the product of privacy parameter and the ` 2-loss of the mechanism is lower bounded by the dimension . Furthermore , the Gaussian mechanism achieves the constant-sharp optimal privacy-accuracy trade-off among all such noises . Our findings are corroborated by numerical experiments . 1 Introduction . Introduced in [ DMNS06 ] , to date differential privacy ( DP ) is perhaps the most popular privacy definition . One of the most important applications of differential privacy is to answer numeric queries . Given a function f of interest , which is also termed a query , our goal is to evaluate this ( potentially vector-valued ) query f on the sensitive data . To preserve privacy , a DP mechanism M working on a dataset D , in its simplest form , is defined as M ( D ) = f ( D ) + tX . ( 1 ) Above , X denotes the noise term and t is a scalar , which together are selected depending on the properties of the query f and the desired privacy level . Among these , perhaps the most popular examples are the Laplace mechanism and the Gaussian mechanism where the noise X follows the Laplace distribution and the Gaussian distribution , respectively . ∗the Institute for Data , Econometrics , Algorithms , and Learning 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Aside from privacy considerations , the most important criterion of an algorithm is arguably the estimation accuracy in the face of choosing , for example , between the Laplace mechanism or its Gaussian counterpart for a given problem . To be concrete , consider a real-valued query f with sensitivity 1—that is , ∆f = supD , D′ |f ( D ) − f ( D′ ) | = 1 , where the supremum is over all neighboring datasets D and D′ . Assuming ( ε , 0 ) -DP for the mechanism M , we are interested in minimizing its ` 2 loss defined as err ( M ) : = E ( M ( D ) − f ( D ) ) 2 = E ( tX ) 2 = t2EX2 . This question is commonly2 addressed by setting X to a standard Laplace random variable and t = ε−1 [ DMNS06 ] . This gives err ( M ) = 2ε−2 . Moving forward , we relax the privacy constraint from ( ε , 0 ) -DP to ( ε , δ ) -DP for some small δ . The canonical way , which was born together with the notion of ( ε , δ ) -DP , is to add Gaussian noise [ DKM+06 ] . A well-known result demonstrates that Gaussian mechanism withX being the standard normal and t = 1ε √ 2 log ( 1.25δ−1 ) is ( ε , δ ) -DP ( see , e.g. , [ DR14 ] ) . The ` 2-loss is err ( M ) = t2 = 2ε−2 · log ( 1.25δ−1 ) . A quick comparison between the two errors reveals a surprising message . The latter error 2ε−2 · log ( 1.25δ−1 ) is larger than the former 2ε−2 . In fact , the extra factor log ( 1.25δ−1 ) is already greater than 10 when δ = 10−5 . At least on the surface , this observation contradicts the fact that ( ε , δ ) -DP is a relaxation of ( ε , 0 ) -DP . Put differently , moving from Laplace to Gaussian , both privacy and accuracy get worse . Nevertheless , this contradiction suggests that we need a better alternative to the Gaussian mechanism instead of giving up the notion of ( ε , δ ) -DP . Indeed , the truncated Laplace mechanism has been proposed as a better alternative to achieve ( ε , δ ) -DP [ GDGK20 ] , which outperforms the Laplace mechanism in terms of estimation accuracy 3 . Motivated by these facts concerning the Laplace , Gaussian , and truncated Laplace mechanisms , one can not help asking : ( Q1 ) Why was the truncated Laplace mechanism not considered in the first place ? Are there any insights behind the design of such mechanisms ? ( Q2 ) More importantly , are these insights inherent for answering one-dimensional queries , or can we extend them to high-dimensional setting ? In this paper , we tackle these fundamental questions , beginning with explaining ( Q1 ) in Section 2 from the decision-theoretic perspective of DP [ WZ10 , KOV17 , DRS21 ] . However , our main focus is ( Q2 ) . In addressing this question , we uncover a seemingly surprising phenomenon — it is impossible to utilize the ( ε , δ ) privacy budget in high-dimensional problems the same way as the truncated Laplace mechanism utilizes it in the one-dimensional problem . More specifically , we show a central limit behavior of the noise-addition mechanism in high dimensions , which , roughly speaking , says that for general noise distributions , the corresponding mechanisms all behave like a Gaussian mechanism . The formal language of “ a mechanism behaves like the Gaussian mechanism ” has been set up in [ DRS21 ] , where a notion called Gaussian Differential Privacy ( GDP ) was proposed . Roughly speaking , a mechanism is µ-GDP if it offers as much privacy as adding N ( 0 , µ−2 ) noise to a sensitivity-1 query . As in the ( ε , δ ) -DP case , the smaller µ is , the stronger privacy is offered . To state our first main contribution , let f be an n-dimensional query and assume that its ` 2-sensitivity is 1 . Consider the noise addition mechanism M ( D ) = f ( D ) + tX where X has a log-concave density ∝ e−ϕ ( x ) on Rn . Let IX : = E [ ∇ϕ ( X ) ∇ϕ ( X ) T ] be the n × n Fisher information matrix and ‖IX‖2 be its operator norm . Theorem 1.1 ( Central Limit Theorem ( Informal version of Theorem 3.1 ) ) . Under certain conditions on ϕ , for t = µ−1 · √ ‖IX‖2 , the corresponding noise addition mechanism M defined in Eq . ( 1 ) is asymptotically µ-GDP as the dimension n → ∞ except for an o ( 1 ) fraction of directions of f ( D ) − f ( D′ ) . In particular , the norm power functions ϕ ( x ) = ‖x‖αp ( p , α > 1 ) satisfy these technical conditions . Note that this class already contains correlated noise , so the results in [ DRS21 ] do not apply here . Numerical results in Figure 1 shows that the convergence occurs for a dimension as small as 30 . 2If f is integer-valued , then the doubly geometric distribution is a better choice and yields an ` 2-loss of 1 2 sinh−2 ε 2 < 2ε−2 . In the so-called high privacy regime , i.e . ε → 0 , the two ` 2-losses have the same order in the sense that their ratio goes to 1 . 3One may blame the sub-optimality of the choice of t , but the problem remains even if the smallest possible t from [ BW18 ] is applied . We then elaborate on the condition “ o ( 1 ) fraction of f ( D ) −f ( D′ ) ” . Following the original definition , DP or GDP is a condition that needs to hold for arbitrary neighboring datasets D and D′ . This worst case perspective is exactly what prevents us to observe the central limit behavior . For example , consider a certain pair of datasets with f ( D ) = ( 0 , 0 , . . . , 0 ) and f ( D′ ) = ( 1 , 0 , . . . , 0 ) , then privacy is completely determined by the first marginal distribution of X , and the dimension n plays no role here . The “ o ( 1 ) fraction of f ( D ) − f ( D′ ) ” rules out the essentially low-dimension cases and reveals the truly high-dimensional behavior . In summary , Theorem 1.1 suggests that when the dimension is high , a large class of noise addition mechanisms behave like the Gaussian mechanism , and hence are doomed to a poor use of the given ( ε , δ ) privacy budget , in the same fashion as we have seen in the one-dimensional example . However , admitting the central limit phenomenon , our second theorem turns the table and characterizes the optimal privacy-accuracy trade-off and justifies the Gaussian mechanism . To see this , recall that the noise addition mechanism defined in Equation ( 1 ) is determined by the pair ( t , X ) . Both privacy and accuracy are jointly determined by t and X . Adopting the central limit theorem 1.1 , it is convenient to take an equivalent parametrization , which is ( µ , X ) , where µ is the desired ( asymptotic ) GDP parameter . Given X , the two parametrizations are related by t = µ−1 · √ ‖IX‖2 . Using parameters ( µ , X ) , the corresponding mechanism Mµ , X is given by Mµ , X ( D ) = f ( D ) + µ −1 · √ ‖IX‖2 ·X By Theorem 1.1 , it is asymptotically µ-GDP . The following theorem states in an “ uncertainty principle ” fashion that the privacy parameter and the error can not be small at the same time . Theorem 1.2 . As long as the Fisher information of X is defined , we have µ2 · err ( Mµ , X ) > n. The equality holds if X is n-dimensional standard Gaussian . Combining Theorems 1.1 and 1.2 , among all the noise that satisfies the conditions of Theorem 1.1 , Gaussian yields the constant-sharp optimal privacy-accuracy trade-off . As far as we know , this is the first result characterizing optimality with the sharp constant when the dimension is high . The privacy conclusion of Theorem 1.1 does not work for every pair of neighboring datasets , so it is worth noting that we do NOT intend to suggest this as a valid privacy guarantee . Instead , we present it as an interesting phenomenon that has been largely overlooked in the literature . Furthermore , this central limit theorem admits an elegant characterization of privacy-accuracy trade-off that is sharp in constant . From a theoretical point of view , the proof of Theorem 1.1 , as we shall see in later sections , involves non-linear functionals of high dimensional distributions . This type of results are , to the best of our knowledge , quite underexplored compared to linear functionals , so our results may serve as an additional motivation to study this type of questions . Related work There is a large body of literature on the characterization of privacy-accuracy tradeoff for query answering mechanisms . For the one-dimensional case , the constant-sharp optimal noise for ( ε , 0 ) -DP was shown to have a piece-wise constant density by [ GV16 ] . This complements our discussion in Figure 2 . When the dimension is high , only up-to-constant-factor optimality was known . In particular , [ BUV18 , SU17 ] confirm that Gaussian mechanism is minimax rate optimal under ( ε , δ ) DP by a novel lower bound technique . In addition , [ ENU20 ] also confirms the minimax optimality of Gaussian mechanism for linear queries with a refined notion of sensitivity . Our work extends this direction by taking the CLT perspective and providing an elegant constant-sharp optimality result . There are also works studying the up-to-constant-factor minimax optimality in other models , such as the ( sparse ) linear regression [ CWZ21 ] , generalized linear models [ CWZ20 , SSTT21 ] , Gaussian mixtures [ KSSU20 , ZZ21 ] and so on . In our work , we initialize the investigation in the ( simpler ) mean estimation problem , and leave the constant-sharp optimality in other problems for future work .

Summary: Within the framework of f-DP, the paper establishes a CLT for high-dimensional additive noise mechanims, demonstrating that as the dimension increases, adding independent noise to the coordinates has a privacy loss similar to Gaussian noise. This result addresses the question of what noise mechanism is best applied, indicating that Gaussian noise offers the best privacy-utility tradeoff. The privacy-utility tradeoff is communicates through an "uncertainty principle", which shows that the product of the privacy parameter and a measure of utility are bounded, indicating that both quantities cannot be simultaneously small. They show that Gaussian noise makes this uncertainty inequality tight.