Bounding Membership Inference

1 INTRODUCTION . Differential Privacy ( DP ) ( Dwork , 2006 ) is employed extensively to reason about privacy guarantees in a variety of different settings ( Dwork , 2008 ) . Recently , DP started being used to give privacy guarantees for the training data of deep neural networks ( DNNs ) learned through stochastic gradient descent ( Abadi et al. , 2016 ) . However , even though DP gives privacy guarantees and bounds the worst case privacy leakage , it is not immediately clear how these guarantees bound the accuracy of known existing forms of privacy infringement attacks . At the time of writing , the most practical attack on the privacy of DNNs is Membership Inference ( MI ) ( Shokri et al. , 2017 ) , where an attacker predicts whether or not a model used a particular data point for training ( note that this is quite similar to the hypothetical adversary at the core of the game instantiated in the definition of DP ) ; membership inference attacks saw a strong interest by the community , and several improvements and renditions were proposed since its inception ( Sablayrolles et al. , 2019 ; Choquette-Choo et al. , 2021 ; Maini et al. , 2021 ; Hu et al. , 2021 ) . Having privacy guarantees would desirably defend against MI , and in fact current literature highlights that DP does indeed give an upper bound on the performance of MI adversaries ( Yeom et al. , 2018 ; Erlingsson et al. , 2019 ; Sablayrolles et al. , 2019 ; Jayaraman et al. , 2020 ) . In this paper , we first propose a tighter bound on MI accuracy for training algorithms that provide -DP . Our approach uses a lemma about an equivalence of certain sets of datasets . Furthermore , in obtaining our bound , we also show how this bound can benefit from a form of privacy amplification where the training dataset itself is sub-sampled from a larger dataset . Amplification is a technique pervasively found in work improving the analysis of DP learners like DP-SGD ( Abadi et al. , 2016 ) , and we observe the effect of our amplification on lowering MI accuracy is significantly stronger than the effect of batch sampling , a common privacy amplification scheme for training DNNs . Our bound also has consequences for the problem of unlearning ( or data forgetting in ML ) introduced by Cao & Yang ( 2015 ) . In particular the MI accuracy on the point to be unlearned is a popular measure for how well a model has unlearned ( Baumhauer et al. , 2020 ; Graves et al. , 2020 ; Golatkar et al. , 2020b ; a ) . However empirical verification of the MI accuracy can be open ended , as it is subjective to the attack employed . Theoretical bounds on all MI attacks , such as the one proposed in this work , circumvent this issue ; a bound on the accuracy of MI attacks , in particular the probability a data point was used in the training dataset , indicates a limitation for any entity to discern if the model had trained on the data point . In the case when this is sufficiently low ( where sufficiently is defined apriori ) , one can then claim to have unlearned by achieving a model sufficiently likely to have not come from training with the data point . Our analysis shows that , if dataset subsampling is used , one can unlearn under this definition by training with a relatively large ( and thus have less cost to performance ) . To summarize , our contributions are : • We present a tighter general bound on MI accuracy for -DP ; • We further demonstrate how to lower this bound using a novel privacy amplification scheme built on dataset subsampling ; • We discuss the benefits of such bounds to Machine Unlearning as a rigorous way to use MI as a metric for unlearning . 2 BACKGROUND . 2.1 DIFFERENTIAL PRIVACY . Differential privacy ( DP ) ( Dwork , 2006 ) bounds how different the outputs of a function on adjacent inputs can be in order to give privacy guarantees for the inputs . More formally a function F is -DP if for all adjacent inputs x and x′ ( i.e . inputs with hamming distance 1 ) we have for all sets S in the output space : P ( F ( x ) ∈ S ) ≤ e P ( F ( x′ ) ∈ S ) ( 1 ) We also have a more relaxed notion of ( , δ ) -DP where in the same setup as above , but with a parameter δ ∈ ( 0 , 1 ] , we have P ( F ( x ) ∈ S ) ≤ e P ( F ( x′ ) ∈ S ) + δ . Notably , ( , δ ) -DP is used for functions where it is more natural to work with ` 2 metrics on the input space , which has to do with how DP guarantees are obtained . To achieve DP guarantees one usually introduces noise to the output of the function F . The amount of noise is calibrated to the maximal ` 2 or ` 1 difference between all possible outputs of the function on adjacent datasets ( also called sensitivity ) . Significant progress was achieved on minimizing the amount of noise needed for a given sensitivity ( Balle & Wang , 2018 ) , and on how DP guarantees scale when composing multiple DP functions ( Dwork et al. , 2010 ; Kairouz et al. , 2015 ) . Abadi et al . ( 2016 ) demonstrated a method to make the final model returned by mini-batch SGD ( , δ ) -DP with respect to its training dataset by bounding the sensitivity of gradient updates during mini-batch SGD and introducing Gaussian noise to each update . This approach became the de-facto standard for DP guarantees in DNNs . However , the adoption is still greatly limited because of an observed trade-off between privacy guarantees and model utility . At the time of writing there is still no feasible ways to learn with low and high accuracy , and past work ( Jagielski et al. , 2020 ) have suggested that DP-analysis may be too loose and provides more privacy than is expected . However , more recently Nasr et al . ( 2021 ) showed ( using statistical tests and stronger MI adversaries ) that the current state of the art approaches to achieving ( , δ ) -DP bounds for deep learning are tight , in contrast to Jagielski et al . ( 2020 ) results suggesting that they were loose . That is , there is not much more improvement to be gained by studying how to improve the ( , δ ) -DP bound from a given amount of noise or improving composition rules . Facing this , future improvements in DP training would lie in understanding the guarantees that DP bounds provide against the performance of relevant privacy attacks . This would allow us to be more informed about the guarantees required during training to defeat practical attacks and enable the use of looser guarantees if one is only interested in defending against a specific set of attacks . 1 . 1It is worth noting that Nasr et al . ( 2021 ) showed that current analytic upper bounds on DP guarantees are tight , measuring them empirically with various strong privacy adversaries . Although results do suggest that bounds match , the paper did not investigate how DP guarantees limit performance of the adversary . 2.2 MEMBERSHIP INFERENCE . Shokri et al . ( 2017 ) introduced a MI attack against DNNs , which leveraged shadow models ( models with the same architecture as the target model ) trained on similar data in order to train a classifier which , given the outputs of a model on a data point , predicts if the model was trained on that data point or not . Since the introduction of this initial attack , the community has proposed several improved and variations of the original MI attack ( Yeom et al. , 2018 ; Salem et al. , 2018 ; Sablayrolles et al. , 2019 ; Truex et al. , 2019 ; Jayaraman et al. , 2020 ; Maini et al. , 2021 ) , such as an attack that only look at predicted labels of the target model ( Choquette-Choo et al. , 2021 ) . MI attacks are currently the main practical threat to the privacy of a user ’ s data used to train DNNs . Especially in the context of DP DNN learning , it would be beneficial to know how DP bounds translate to bounds on the accuracy of MI attacks . This would provide an understanding on what sort of DP guarantees an entity requires to ensure a sufficiently low maximum accuracy of an adversary trying to discern their training data from their deployed model . The bounds would hence help in guiding decisions regarding setting these parameters and their corresponding guarantees . In our paper we will work with the following ( abstract ) notion of MI in giving our bounds . In particular we define our MI adversary as a function f which takes a set of models S and a data point x∗ to deterministically output either 1 or 0 , corresponding to whether x∗ was in the dataset D used to obtain the models in S or not respectively . Note the generality of this adversary , as we do not consider how the adversary obtains the function , i.e . it can have arbitrary strength . Any such adversary will then satisfy the upper and lower bounds we derive later in the paper . With that definition of our adversary , we have the following definition of positive and negative MI accuracy ( which is what we focus on in this paper ) ; note that here D is the training dataset and P ( x∗ ∈ D|S ) is the probability a data point x∗ was in the training dataset used to obtain the models in the set S ( i.e . this is a probability over datasets ) . We explain more about where the randomness is introduced ( in particular the probability involved in obtaining a training dataset ) in Section 3.1 . Definition 1 ( MI accuracy ) . The positive accuracy of f ( x∗ , S ) , the accuracy if f outputs 1 which we define asA ( f ( x∗ , S ) = 1 ) , is P ( x∗ ∈ D|S ) and the negative accuracy , the accuracy if f outputs 0 which we define as A ( f ( x∗ , S ) = 0 ) , is P ( x∗ /∈ D|S ) = 1− P ( x∗ ∈ D|S ) 2.3 PREVIOUS BOUNDS . Before giving bounds on MI accuracy , we have to formally define the attack setting . Two of the main bounds ( Yeom et al. , 2018 ; Erlingsson et al. , 2019 ) focused on an experimental setup first introduced by Yeom et al . ( 2018 ) . To summarize the experiment , and in particular the situation the adversary is operating in , an adversary f is given a datapoint x∗ that is 50 % likely to have been used to train a model S or not . The adversary then either predicts 1 if they think it was used , or 0 otherwise . Let b = 1 or 0 indicate if the datapoint was or was not used for training respectively . We say the adversary was correct if their prediction matches b . We then define the adversary ’ s advantage as improvement in accuracy over the 50 % baseline of random guessing , or more specifically 2 ( A ( f ) − 0.5 ) where A ( f ) is the accuracy of f . For such an adversary operating in a scenario where data is equally likely to be included or not in the training dataset , Yeom et al . ( 2018 ) showed that they could bound the advantage of the adversary by e − 1 when training with -DP . In other words , they showed that they could bound the accuracy of the MI adversary by e /2 . Their proof used the fact that the true positive rate ( TPR ) and false positive rate ( FPR ) of their adversary could be represented as expectations over the different data points in a dataset , and from that introduced the DP condition to obtain their MI bound , noting that MI advantage is equivalent to TPR - FPR . Erlingsson et al . ( 2019 ) improved on the bound developed by Yeom et al . ( 2018 ) for an adversary operating under the same condition by utilizing a proposition given by Hall et al . ( 2013 ) on the relation between TPR and FPR for an ( , δ ) -DP function . Using these facts , Erlingsson et al . ( 2019 ) bounded the membership advantage by 1−e− +δe− , which is equivalent to bounding the accuracy of the adversary by 1 − e− /2 when δ = 0 ( i.e . in -DP ) . This is , to the best of our knowledge , the previous state-of-the-art bound for high . Other work have considered more general setups where the probability of sampling a datapoint in the dataset can vary , similar to what we consider in Section 3.1 . For -DP , Sablayrolles et al . ( 2019 ) bounded the probability of a datapoint x∗ being used in the training set of a model ( i.e. , the accuracy of an attacker who predicted the datapoint was in the dataset of the model ) by Px∗ ( 1 ) + 4 where Px∗ ( 1 ) is the probability of the datapoint being in the dataset . This is , to the best of our knowledge , the previous state-of-the-art bound for low ( when reduced to case of Yeom et al . ( 2018 ) by setting Px∗ ( 1 ) = 0.5 as to compare with Erlingsson et al . ( 2019 ) ) . Finally , Jayaraman et al . ( 2020 ) bounded the positive predictive value of an attacker ( i.e . its precision ) on a model trained with ( , δ ) -DP when the FPR is fixed . It is worth noting that although the considered setup is similar to works covered above , it assumes an unbalanced sampling procedure . Similarly , Jayaraman et al . further bounded membership advantage under the experiment described by Yeom et al . ( 2018 ) for a fixed FPR . Erlingsson et al . ( 2019 ) followed a similar argument for their bound but were also able to remove the need for an explicit knowledge of the FPR .

Summary: This paper presents a tighter analysis for membership inference attacks on algorithms that satisfy $\varepsilon$-differential privacy. It also talks about a better form of privacy amplification based on that. Finally, it discusses the role of membership inference in the task of machine unlearning.