Privately Publishable Per-instance Privacy

1 Introduction . An explosion of data has fueled innovation in machine learning applications and demanded , in equal turn , privacy protection for the sensitive data with which machine learning practitioners train and evaluate models . Differential privacy ( DP ) ( Dwork et al. , 2006 , 2014a ) has become a mainstay of privacy-preserving data analysis , replacing less robust privacy definitions such as k-anonymity which fail to protect against sufficiently powerful de-anonymization attacks ( Narayanan & Shmatikov , 2008 ) . In contrast , DP offers provable privacy guarantees that are robust against an arbitrarily strong adversary . The data curator could trivially protect against privacy loss by reporting a constant function , or by releasing only data-independent noise . The key challenge of DP is to release privatized output that retains utility to the data analyst . A desired level of utility in a machine learning application might necessitate a high value of , but the privacy guarantees degrade quickly past = 1 . ( Triastcyn & Faltings , 2020 ) construct an example whereby a differentially private algorithm with = 2 allows an attacker to use a maximum-likelihood estimate to conclude with up to 88 % accuracy that an individual is in a dataset . For = 5 , the theoretical upper bound on the accuracy of an optimal attack is 99.3 % . Moreover , practical applications of differential privacy commonly use large values of . A study of Apple ’ s deployment of differential privacy revealed that the overall daily privacy loss permitted by the system was as high as = 6 for Mac OS 10.12.3 and = 14 for iOS 10.1.1 ( Tang et al. , 2017 ) – offering only scant privacy protection ! Recent work ( Yu et al. , 2021 ) has empirically justified large privacy parameters by conducting membership inference attacks to demonstrate that these seemingly tenuous privacy guarantees are actually much stronger in practice . These results are unsurprising from the perspective that DP gives a data-independent bound on the worst-case privacy loss which is likely to be a conservative estimate of the risk to a particular individual when a DP algorithm is applied to a particular input dataset . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Per-instance differential privacy provides a theoretically sound alternative to the empirical approach for revealing the gap between the worst-case DP bound and the actual privacy loss in practice . The privacy loss to a particular individual relative to a fixed dataset might be orders of magnitude smaller than the worst-case bound guaranteed by standard DP . In this case , an algorithm meeting a desired level of utility but providing weak DP guarantees may , for the same level of utility , achieve drastically more favorable per-instance DP guarantees . The remaining challenge is that the per-instance privacy loss is a function of the entire dataset ; publishing it directly would negate the purpose of privately training a model in the first place ! In this paper , we propose a methodology to privately release the per-instance privacy losses associated with private empirical risk minimization . Our contributions are as follows : • We introduce ex-post per-instance differential privacy to provide a sharp characterization of the privacy loss to a particular individual that adapts to both the input dataset and the algorithm ’ s output . • We present a novel analysis of the ex-post per-instance privacy losses incurred by the objective perturbation mechanism , demonstrating that these ex-post pDP losses are orders of magnitude smaller than the worst-case guarantee of differential privacy . • We propose a group of methods to privately and accurately release the ex-post pDP losses . In the particular case of generalized linear models , we show that we can accurately publish the private ex-post pDP losses using a dimension- and dataset-independent bound . • One technical result of independent interest is a new DP mechanism that releases the Hessian matrix by adding a Gaussian Orthogonal Ensemble matrix , which improves the classical “ AnalyzeGauss ” ( Dwork et al. , 2014b ) by roughly a constant factor of 2 . 1.1 Related Work . This paper builds upon ( Wang , 2019 ) , which proposed the per-instance DP framework and left as an open question the matter of publishing the pDP losses . We extend the pDP framework to an ex-post setting to provide privacy guarantees that adapt even more fluidly to data-dependent properties of our algorithms . Another fundamental ingredient in our privacy analysis is the objective perturbation algorithm ( Obj-Pert ) of ( Chaudhuri et al. , 2011 ) , further analyzed by ( Kifer et al. , 2012 ) , which privately releases the minimizer of an empirical risk by adding a linear perturbation to the objective function before optimizing . Per-instance DP and ex-post per-instance DP belong to a growing family of DP definitions that provide a more fine-grained characterization of the privacy loss . Among these are data-dependent DP ( Papernot et al. , 2018 ) , which conditions on a fixed dataset ; personalized DP ( Ghosh & Roth , 2011 ; Ebadi et al. , 2015 ; Liu et al. , 2015 ) , which conditions on a fixed individual ’ s datapoint ; and ex-post DP ( Ligett et al. , 2017 ) , which conditions on the realized output of the algorithm . Per-instance DP conditions on both a fixed dataset and a fixed individual ’ s datapoint , and ex-post per-instance DP adapts even further to the realized output of the algorithm . A more detailed comparison of these DP variants is included in the supplementary materials . Other data-adaptive methodologies include propose-test-release ( Dwork & Lei , 2009 ) and local sensitivity ( Nissim et al. , 2007 ) . In addition , Bayesian differential privacy ( Triastcyn & Faltings , 2020 ) provides data-dependent privacy guarantees that afford strong protection to “ typical ” data by making distributional assumptions about the sensitive data . The Rényi-DP-based privacy filters of ( Feldman & Zrnic , 2020 ) are also closely related to our work ; the authors study composition of personalized ( but not per-instance ) privacy losses using adaptively-chosen privacy parameters . 2 Preliminaries . 2.1 Symbols and Notation . We write the output of a randomized algorithm A as A ( · ) , and for continuous distributions we take Pr [ A ( D ) = o ] to be the value of the probability density function at output o . We will let z ∈ Z refer to both an individual and their data ; for example , individual z holds data z = ( x , y ) in a supervised learning problem . We take Z∗ = ∪∞n=0Zn to be the space of datasets with an unspecified number of data points . D±z ∈ Z∗ denotes the fixed dataset D = { z1 , . . . , zn } ∈ Z∗ with the data point z removed from D if z ∈ D , or added to D if z /∈ D. In our mathematical expressions , we use “ ± ” to mean “ add if z /∈ D , subtract otherwise ” . Similarly . “ ∓ ” means “ subtract if z /∈ D , add otherwise ” . We distinguish between as fixed input to a DP algorithm , and ( · ) as a function parameterized according to a particular DP relaxation — e.g. , ( o , D , D±z ) means the ex-post per-instance privacy loss conditioned on output o , dataset D , and data point z . 2.2 Differential Privacy . Let Z denote the data domain , andR the set of all possible outcomes of algorithm A . Fix , δ ≥ 0 . Definition 1 . ( Differential privacy ) A randomized algorithm A : Z∗ → R satisfies ( , δ ) -DP if for all datasets D ∈ Z∗ and data points z ∈ Z , and for all measurable sets S ⊂ R , Pr [ A ( D ) ∈ S ] ≤ e Pr [ A ( D±z ) ∈ S ] + δ . Differential privacy guarantees that the presence or absence of any particular data record has little impact on the output distribution of a randomized algorithm . In this paper we use the “ add/remove ” notion of DP , by which we construct neighboring dataset D±z by adding or removing an individual z from dataset D. DP is powerful and universal in that its guarantee applies to any D , z and set of output events . However , there are often situations where the privacy losses of A vary drastically depending on its input data , and the privacy loss bound ( protecting even the worst-case pair of neighboring datasets ) may not be informative of the privacy loss incurred to individuals when the input toA is typical . This motivated ( Wang , 2019 ) to consider a per-instance version of the DP definition . Definition 2 . ( Per-instance differential privacy ) A randomized algorithm A : Z∗ → R satisfies ( ( D , D±z ) , δ ) -pDP if for dataset D and data point z , and for all measurable sets S ⊂ R , Pr [ A ( D ) ∈ S ] ≤ e Pr [ A ( D±z ) ∈ S ] + δ , Pr [ A ( D±z ) ∈ S ] ≤ e Pr [ A ( D ) ∈ S ] + δ . The pDP definition can be viewed as using a function ( D , D±z ) that more precisely describes the privacy guarantee in protecting a fixed data point z when A is applied to dataset D. As it turns out , it is most convenient for us to work with an even more instance-specific description of the privacy loss that is further parameterized by the realized output of A ex-post — after the random coins of A are flipped and the outcome released . Definition 3 . ( Ex-post per-instance differential privacy ) A randomized algorithm A satisfies ( · ) - ex-post per-instance differential privacy for an individual z and a fixed dataset D at an outcome o ∈ Range ( A ) if ∣∣∣∣∣log ( Pr [ A ( D ) = o ] Pr [ A ( D±z ) = o ] ) ∣∣∣∣∣ ≤ ( o , D , D±z ) . This definition generalizes the ex-post DP definition ( Ligett et al. , 2017 ) ( introduced for a different purpose ) to a per-instance version that depends on a given pair of neighboring datasets . The above quantity is essentially the absolute value of the log-odds ratio , used extensively in hypothesis testing . Intuitively , the ex-post per-instance privacy loss ( o , D , D±z ) describes how confidently an attacker could infer , given the output of algorithm A , whether or not individual z is in dataset D. Despite ( or perhaps because of ) its precise accounting for privacy , ex-post pDP could reveal sensitive information about the dataset , as the following example explicitly illustrates . Example 4 ( The privacy risk of exposing ex-post pDP ) . Consider a standard Gaussian mechanism A that adds noise to a counting query Q applied to dataset D , i.e . A ( D ) = Q ( D ) +N ( 0 , σ2 ) . Q has global sensitivity ∆Q = 1 . We will show that an attacker , knowing only the output o of algorithm A , her ex-post pDP loss and that her individual data is not contained in dataset D , can conclusively uncover the sensitive quantity Q ( D ) protected by algorithm A . Following the proof of Theorem ? ? , the ex-post pDP can be directly calculated as ( o , D , D±z ) = |Q ( D ) −Q ( D±z ) ||2o−Q ( D ) −Q ( D±z ) | 2σ2 . Enter attacker z , who has auxiliary information : she knows that her own individual data is not contained in D. After algorithm A is applied to D , attacker z receives output o = 1 and is informed of her ex-post pDP ( o , D , D+z ) . Since Q ( D+z ) = Q ( D ) + 1 is known , attacker z can solve for Q ( D ) and obtain Q ( D ) = o − 0.5 ± σ2 ( o , D , D+z ) . With probability 1 , only one of the two possibilities is an integer1 . Therefore , exposing ex-post pDP in this case completely reveals Q ( D ) . Problem statement . The lesson of Example 4 is that we can not directly reveal the ex-post pDP losses without potentially nullifying the algorithm ’ s privacy benefits . How , then , can we privately and accurately publish the ex-post pDP losses ? The goal of this paper is to develop an algorithm that publishes a function ̃ : Z → R whose output estimates the ex-post pDP loss to an individual z of releasing the output θ̂P from the objective perturbation mechanism . Any individual ( not just those whose data is contained in the dataset ) can plug her own data z into this function in order to receive a high-probability bound on her ex-post pDP loss which does not depend directly on any sensitive data except her own . This requirement offers the same type of privacy protection as joint differential privacy ( Kearns et al. , 2014 ) , which relaxes the standard DP definition by allowing an algorithm ’ s output to individual z to be sensitive only in her own private data . Our notion of privacy is slightly more general in that it holds for individuals both in and out of the dataset . The difference lies in how the algorithm ’ s output space is defined ; whereas a joint DP algorithm produces a fixed-length tuple partitioning the output to each individual in the dataset , our algorithm outputs a function whose domain includes any data point z ∈ Z . As a result , our methods are robust against collusion by arbitrary coalitions of adversaries , allowing repeated queries by any group of individuals without invalidating the privacy guarantees promised by the pDP losses .

Summary: The paper considers the problem of publishing the per-instance privacy loss of a trained learning model to each individual while protecting the privacy of other participants. The paper focuses on training a convex, smooth model using objective perturbation. By deriving bounds on the per-instance ex-post privacy loss, the paper develops algorithms that publish the Hessian matrix privately, which can be used by each individual to evaluate their per-instance privacy loss.  The paper shows that under certain regimes, the algorithm can provide an approximation of the per-instance privacy loss up to constant factors. The paper also provides empirical experiments which show how the derived method provides an approximation of the per-instance privacy cost and how it compares to the worst-case bound.