Privately Publishable Per-instance Privacy

1 Introduction . An explosion of data has fueled innovation in machine learning applications and demanded , in equal turn , privacy protection for the sensitive data with which machine learning practitioners train and evaluate models . Differential privacy ( DP ) ( Dwork et al. , 2006 , 2014a ) has become a mainstay of privacy-preserving data analysis , replacing less robust privacy definitions such as k-anonymity which fail to protect against sufficiently powerful de-anonymization attacks ( Narayanan & Shmatikov , 2008 ) . In contrast , DP offers provable privacy guarantees that are robust against an arbitrarily strong adversary . The data curator could trivially protect against privacy loss by reporting a constant function , or by releasing only data-independent noise . The key challenge of DP is to release privatized output that retains utility to the data analyst . A desired level of utility in a machine learning application might necessitate a high value of , but the privacy guarantees degrade quickly past = 1 . ( Triastcyn & Faltings , 2020 ) construct an example whereby a differentially private algorithm with = 2 allows an attacker to use a maximum-likelihood estimate to conclude with up to 88 % accuracy that an individual is in a dataset . For = 5 , the theoretical upper bound on the accuracy of an optimal attack is 99.3 % . Moreover , practical applications of differential privacy commonly use large values of . A study of Apple ’ s deployment of differential privacy revealed that the overall daily privacy loss permitted by the system was as high as = 6 for Mac OS 10.12.3 and = 14 for iOS 10.1.1 ( Tang et al. , 2017 ) – offering only scant privacy protection ! Recent work ( Yu et al. , 2021 ) has empirically justified large privacy parameters by conducting membership inference attacks to demonstrate that these seemingly tenuous privacy guarantees are actually much stronger in practice . These results are unsurprising from the perspective that DP gives a data-independent bound on the worst-case privacy loss which is likely to be a conservative estimate of the risk to a particular individual when a DP algorithm is applied to a particular input dataset . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Per-instance differential privacy provides a theoretically sound alternative to the empirical approach for revealing the gap between the worst-case DP bound and the actual privacy loss in practice . The privacy loss to a particular individual relative to a fixed dataset might be orders of magnitude smaller than the worst-case bound guaranteed by standard DP . In this case , an algorithm meeting a desired level of utility but providing weak DP guarantees may , for the same level of utility , achieve drastically more favorable per-instance DP guarantees . The remaining challenge is that the per-instance privacy loss is a function of the entire dataset ; publishing it directly would negate the purpose of privately training a model in the first place ! In this paper , we propose a methodology to privately release the per-instance privacy losses associated with private empirical risk minimization . Our contributions are as follows : • We introduce ex-post per-instance differential privacy to provide a sharp characterization of the privacy loss to a particular individual that adapts to both the input dataset and the algorithm ’ s output . • We present a novel analysis of the ex-post per-instance privacy losses incurred by the objective perturbation mechanism , demonstrating that these ex-post pDP losses are orders of magnitude smaller than the worst-case guarantee of differential privacy . • We propose a group of methods to privately and accurately release the ex-post pDP losses . In the particular case of generalized linear models , we show that we can accurately publish the private ex-post pDP losses using a dimension- and dataset-independent bound . • One technical result of independent interest is a new DP mechanism that releases the Hessian matrix by adding a Gaussian Orthogonal Ensemble matrix , which improves the classical “ AnalyzeGauss ” ( Dwork et al. , 2014b ) by roughly a constant factor of 2 . 1.1 Related Work . This paper builds upon ( Wang , 2019 ) , which proposed the per-instance DP framework and left as an open question the matter of publishing the pDP losses . We extend the pDP framework to an ex-post setting to provide privacy guarantees that adapt even more fluidly to data-dependent properties of our algorithms . Another fundamental ingredient in our privacy analysis is the objective perturbation algorithm ( Obj-Pert ) of ( Chaudhuri et al. , 2011 ) , further analyzed by ( Kifer et al. , 2012 ) , which privately releases the minimizer of an empirical risk by adding a linear perturbation to the objective function before optimizing . Per-instance DP and ex-post per-instance DP belong to a growing family of DP definitions that provide a more fine-grained characterization of the privacy loss . Among these are data-dependent DP ( Papernot et al. , 2018 ) , which conditions on a fixed dataset ; personalized DP ( Ghosh & Roth , 2011 ; Ebadi et al. , 2015 ; Liu et al. , 2015 ) , which conditions on a fixed individual ’ s datapoint ; and ex-post DP ( Ligett et al. , 2017 ) , which conditions on the realized output of the algorithm . Per-instance DP conditions on both a fixed dataset and a fixed individual ’ s datapoint , and ex-post per-instance DP adapts even further to the realized output of the algorithm . A more detailed comparison of these DP variants is included in the supplementary materials . Other data-adaptive methodologies include propose-test-release ( Dwork & Lei , 2009 ) and local sensitivity ( Nissim et al. , 2007 ) . In addition , Bayesian differential privacy ( Triastcyn & Faltings , 2020 ) provides data-dependent privacy guarantees that afford strong protection to “ typical ” data by making distributional assumptions about the sensitive data . The Rényi-DP-based privacy filters of ( Feldman & Zrnic , 2020 ) are also closely related to our work ; the authors study composition of personalized ( but not per-instance ) privacy losses using adaptively-chosen privacy parameters . 2 Preliminaries . 2.1 Symbols and Notation . We write the output of a randomized algorithm A as A ( · ) , and for continuous distributions we take Pr [ A ( D ) = o ] to be the value of the probability density function at output o . We will let z ∈ Z refer to both an individual and their data ; for example , individual z holds data z = ( x , y ) in a supervised learning problem . We take Z∗ = ∪∞n=0Zn to be the space of datasets with an unspecified number of data points . D±z ∈ Z∗ denotes the fixed dataset D = { z1 , . . . , zn } ∈ Z∗ with the data point z removed from D if z ∈ D , or added to D if z /∈ D. In our mathematical expressions , we use “ ± ” to mean “ add if z /∈ D , subtract otherwise ” . Similarly . “ ∓ ” means “ subtract if z /∈ D , add otherwise ” . We distinguish between as fixed input to a DP algorithm , and ( · ) as a function parameterized according to a particular DP relaxation — e.g. , ( o , D , D±z ) means the ex-post per-instance privacy loss conditioned on output o , dataset D , and data point z . 2.2 Differential Privacy . Let Z denote the data domain , andR the set of all possible outcomes of algorithm A . Fix , δ ≥ 0 . Definition 1 . ( Differential privacy ) A randomized algorithm A : Z∗ → R satisfies ( , δ ) -DP if for all datasets D ∈ Z∗ and data points z ∈ Z , and for all measurable sets S ⊂ R , Pr [ A ( D ) ∈ S ] ≤ e Pr [ A ( D±z ) ∈ S ] + δ . Differential privacy guarantees that the presence or absence of any particular data record has little impact on the output distribution of a randomized algorithm . In this paper we use the “ add/remove ” notion of DP , by which we construct neighboring dataset D±z by adding or removing an individual z from dataset D. DP is powerful and universal in that its guarantee applies to any D , z and set of output events . However , there are often situations where the privacy losses of A vary drastically depending on its input data , and the privacy loss bound ( protecting even the worst-case pair of neighboring datasets ) may not be informative of the privacy loss incurred to individuals when the input toA is typical . This motivated ( Wang , 2019 ) to consider a per-instance version of the DP definition . Definition 2 . ( Per-instance differential privacy ) A randomized algorithm A : Z∗ → R satisfies ( ( D , D±z ) , δ ) -pDP if for dataset D and data point z , and for all measurable sets S ⊂ R , Pr [ A ( D ) ∈ S ] ≤ e Pr [ A ( D±z ) ∈ S ] + δ , Pr [ A ( D±z ) ∈ S ] ≤ e Pr [ A ( D ) ∈ S ] + δ . The pDP definition can be viewed as using a function ( D , D±z ) that more precisely describes the privacy guarantee in protecting a fixed data point z when A is applied to dataset D. As it turns out , it is most convenient for us to work with an even more instance-specific description of the privacy loss that is further parameterized by the realized output of A ex-post — after the random coins of A are flipped and the outcome released . Definition 3 . ( Ex-post per-instance differential privacy ) A randomized algorithm A satisfies ( · ) - ex-post per-instance differential privacy for an individual z and a fixed dataset D at an outcome o ∈ Range ( A ) if ∣∣∣∣∣log ( Pr [ A ( D ) = o ] Pr [ A ( D±z ) = o ] ) ∣∣∣∣∣ ≤ ( o , D , D±z ) . This definition generalizes the ex-post DP definition ( Ligett et al. , 2017 ) ( introduced for a different purpose ) to a per-instance version that depends on a given pair of neighboring datasets . The above quantity is essentially the absolute value of the log-odds ratio , used extensively in hypothesis testing . Intuitively , the ex-post per-instance privacy loss ( o , D , D±z ) describes how confidently an attacker could infer , given the output of algorithm A , whether or not individual z is in dataset D. Despite ( or perhaps because of ) its precise accounting for privacy , ex-post pDP could reveal sensitive information about the dataset , as the following example explicitly illustrates . Example 4 ( The privacy risk of exposing ex-post pDP ) . Consider a standard Gaussian mechanism A that adds noise to a counting query Q applied to dataset D , i.e . A ( D ) = Q ( D ) +N ( 0 , σ2 ) . Q has global sensitivity ∆Q = 1 . We will show that an attacker , knowing only the output o of algorithm A , her ex-post pDP loss and that her individual data is not contained in dataset D , can conclusively uncover the sensitive quantity Q ( D ) protected by algorithm A . Following the proof of Theorem ? ? , the ex-post pDP can be directly calculated as ( o , D , D±z ) = |Q ( D ) −Q ( D±z ) ||2o−Q ( D ) −Q ( D±z ) | 2σ2 . Enter attacker z , who has auxiliary information : she knows that her own individual data is not contained in D. After algorithm A is applied to D , attacker z receives output o = 1 and is informed of her ex-post pDP ( o , D , D+z ) . Since Q ( D+z ) = Q ( D ) + 1 is known , attacker z can solve for Q ( D ) and obtain Q ( D ) = o − 0.5 ± σ2 ( o , D , D+z ) . With probability 1 , only one of the two possibilities is an integer1 . Therefore , exposing ex-post pDP in this case completely reveals Q ( D ) . Problem statement . The lesson of Example 4 is that we can not directly reveal the ex-post pDP losses without potentially nullifying the algorithm ’ s privacy benefits . How , then , can we privately and accurately publish the ex-post pDP losses ? The goal of this paper is to develop an algorithm that publishes a function ̃ : Z → R whose output estimates the ex-post pDP loss to an individual z of releasing the output θ̂P from the objective perturbation mechanism . Any individual ( not just those whose data is contained in the dataset ) can plug her own data z into this function in order to receive a high-probability bound on her ex-post pDP loss which does not depend directly on any sensitive data except her own . This requirement offers the same type of privacy protection as joint differential privacy ( Kearns et al. , 2014 ) , which relaxes the standard DP definition by allowing an algorithm ’ s output to individual z to be sensitive only in her own private data . Our notion of privacy is slightly more general in that it holds for individuals both in and out of the dataset . The difference lies in how the algorithm ’ s output space is defined ; whereas a joint DP algorithm produces a fixed-length tuple partitioning the output to each individual in the dataset , our algorithm outputs a function whose domain includes any data point z ∈ Z . As a result , our methods are robust against collusion by arbitrary coalitions of adversaries , allowing repeated queries by any group of individuals without invalidating the privacy guarantees promised by the pDP losses .

Summary: The paper considers the problem of bridging the gap between central DP, where overall epsilon is released, versus personalized epsilon, that is privacy of individual data points that may or may not have been part of a dataset analyzed with central DP. The paper considers central DP of specific function: objective perturbation mechanism, this setting of achieving central DP is less common that others. The paper makes an observation that naively computing personalized epsilon and releasing it will violate privacy of central DP (as this computation depends on private data) as anyone can use such an oracle to determine the real value that central DP is trying to hide. Since the paper considers a specific DP mechanism its results are applied only to private empirical risk minimization problems (ie function computed on the private dataset).  The proposed mechanism works by computing personalizes privacy loss (es-post pDP) directly (since it considers convex problems it is possible). It then adds noise to computed result because otherwise this metric cannot be released. However the paper also derives data-independent upper bounds for pDP that do not incur additional noise.