Privacy Auditing of Machine Learning using Membership Inference Attacks

1 INTRODUCTION . Machine learning systems have come under intense scrutiny of the regulatory authorities in the past few years . Veale et al . ( 2018 ) argue that machine learning models could be considered personal data due to their susceptibility to inference attacks that can recover sensitive information about training data just from the models . Membership inference attacks ( Homer et al. , 2008 ; Dwork et al. , 2015 ; Shokri et al. , 2017 ) and reconstruction attacks ( Dinur & Nissim , 2003 ; Song et al. , 2017 ; Carlini et al. , 2020 ) are the main inference attacks that highlight , and can quantify , the privacy risk of releasing aggregate information computed on sensitive data ( Dwork et al. , 2017 ) . The focus of this paper will be on membership inference attacks for measuring privacy risk . Organizations such as the ICO ( UK ) and NIST ( US ) have highlighted membership inference as a potential confidentiality violation and privacy threat to the training data ( Murakonda & Shokri , 2020 ) . This has lead to the development of open-source tools 1 and capabilities in widely-used ML libraries 2 for measuring privacy risk from machine learning models using membership inference attacks . Although the approach of quantifying privacy risk through membership inference attacks is gaining traction , the attack success , as measured by a lot of works , can not be completely attributed to information leakage from the models and hence their privacy risk . Various factors such as the distribution of training data , difference in distributions of the train and test data may provide an over-estimate or under-estimate of the actual privacy risk from the model ( Erlingsson et al. , 2019 ; Humphries et al. , 2020 ) . Theoretical analyses that connect the success of membership inference to privacy risk through the framework of differential privacy avoid this issue by slightly modifying how the attack performance is measured ( Yeom et al. , 2018 ; Jagielski et al. , 2020 ; Nasr et al. , 2021 ; Malek et al. , 2021 ) . Instead of measuring the leakage from a particular model , these works aim at measuring the worst-case leakage of the training algorithm ( for a given model architecture ) , and construct multiple models with and without one training point and keep the rest of training set fixed . The performance of the attack ( false positive and false negative errors ) is then computed over these sampled models . 1https : //github.com/privacytrustlab/ml_privacy_meter 2https : //blog.tensorflow.org/2020/06/introducing-new-privacy-testing-library . html However , in practice and during auditing , the performance needs to be computed based on the points in the training set versus the population data ( test set ) for a given fixed model . Various works rely on measuring privacy risk through the performance of membership inference attacks , but the subtle differences in how the attack is formulated mean that they might associate different causes of attack success to privacy loss . When auditing machine learning models , we need to pay attention to what exactly we are measuring and how it relates to the information leakage of the machine learning algorithm and not other factors such as the prior knowledge of the attacker . Previous membership inference attacks are designed for high overall performance , i.e . to succeed for most member ( or non-member ) data points of most target models . With few exception , they train shadow ( reference ) models on datasets randomly sampled from a population , to mimic the general behavior of models on member ( or non-member ) data points . This general behavior , however , does not capture the behavior of specific models on specific samples . As a result , the limited performance of shadow model attack does not precisely measure the model ( sample ) -specific information leakage . In this work , we focus on designing membership inference attacks that more precisely measure what a target model leaks about each individual training data , in a binary hypothesis testing framework . We start from the ( constant ) loss threshold attack , and study how to further design different attack strategy ( loss threshold ) for different target model ( sample ) . We derive multiple attack strategies ( existing and new ) from this hypothesis test formulation via different approximations for the null hypothesis . We also formulate the performance of an attack inside the hypothesis testing framework , via the trade-off between its false positive rate and true positive rate . Following the methodology , we not only derive and explain existing shadow ( reference ) model attacks ( Attack S and R ) , but also design new attacks ( Attack P and D ) that offer more accurate privacy auditing for machine learning models , through model-dependent and ( or ) sample-dependent attack strategies . We empirically evaluate and compare the attack performance ( TPR-FPR curve and AUC score ) and computation cost of our new attacks ( notably Attack D ) , and the prior attacks on multiple datasets . 2 ATTACK FRAMEWORK . Our objective is to design a framework that enables auditing the privacy loss of machine learning models in the black-box setting ( where only model outputs —and not their parameters or internal computations— are observable ) . This framework needs to have three elements : ( i ) the inference game as the evaluation setup ; ( ii ) the indistinguishably metric to measure the privacy risk , and ( iii ) the construction of membership inference attack as hypothesis testing . The notion of privacy underlying our framework is primarily based on differential privacy , and multiple pieces of this framework are generalizations of existing inference attacks against machine learning algorithms . Instead of focusing on designing new attacks , we present the important design choices for constructing and evaluating membership inference attacks , for the purpose of having a precise privacy auditing . We identify different sources of uncertainty that influence the error of inference attacks , and can lead to miscalculation of the privacy loss of a model . We quantify privacy loss of a model in a hypothetical inference game between a challenger and an adversary . We are given a private training set D , and a model θ which is trained on D using a training algorithm T . The challenger samples a random data point ( x1 , y1 ) from the training set ( a member ) , and a random data point ( x0 , y0 ) from the data population outside the training set ( a non-member ) . He then randomly selects one of the two ( member or non-member ) b ∼ { 0 , 1 } with probability 12 , and shares the selected data point ( xb , yb ) and the model ’ s output f ( xb ; θ ) with the adversary . The adversary ’ s task is to determine if the data point is a member or not ( i.e. , guess b ) .3 3We need to emphasize that there is a difference between the privacy loss of a machine learning algorithm and that of the specific models trained with the algorithm . A model is a given instance of a training algorithm , thus its leakage needs to be computed with respect to the individual data records in its specific training set . This subsequently means that the privacy loss of an algorithm varies depending on the randomness in sampling of its training data , and the randomness of the training algorithm . One can analyze the privacy loss of an algorithm as , for example , its worst case privacy loss ( as in differential privacy ) . Differentially private algorithms enforce an upper bound on the privacy loss of an algorithm over all models with respect to all possible training data . Thus , in the inference game for differential privacy , the privacy loss of the training algorithms , would be the worst case privacy loss over all choices of D , ( x0 , y0 ) , and ( x1 , y1 ) in our inference game for model privacy . We use an indistinguishability measure ( which is the basis of differential privacy ) to define privacy of individual training data of a model . According to this measure , the privacy loss of the model with respect to its training data is the adversary ’ s success in distinguishing between the two possibilities b = 0 vs b = 1 over multiple repetitions of the inference game . Naturally , the inference attack is a hypothesis test , and adversary ’ s error is composed of the false positive ( i.e. , inferring a non-member as member ) and false negative of the test . In practice , the error of adversary in each round of the inference game depends of multiple factors : • The true leakage of the model about the target data ( xb , yb ) when b = 1 . • The uncertainty ( belief or background knowledge ) of attack algorithm about the population data • The adversary ’ s uncertainty about the training algorithm T • The uncertainty about all training data except the target data ( xb , yb ) • The attack dependency on the target data ( xb , yb ) , and the model θ In the ideal setting , we only want the attack error to be dependent on the true leakage of the model about the target data ( i.e. , whether the same model trained with and without ( xb , yb ) are distinguishable from each other ) . To this end , and to cancel out the effect of other uncertainties , an attack algorithm and the evaluation setup for the inference game need to be designed based on the following principle : The population data used for constructing the attack algorithm , and evaluating the inference game , need to be similar , in distribution , to the training data . This is to minimize the impact of prior belief ( what could have been sampled for the training set ) in the performance of the inference attack . This is not hard to achieve as all the process ( of constructing the hypothesis testing attack , and evaluating it ) is controlled by the auditor . By violating this principle , we might overestimate the privacy loss ( by making the test dependent on a distinct prior knowledge ) or underestimate the privacy loss ( by evaluating the inference attack on a population data distribution for which it was not constructed ) . Another crucial requirement is that the privacy audit needs to output a detailed report , which captures the uncertainty of the attack . Reporting one number as the accuracy of the attack , as it is mostly reported in the literature , is not an informative report . Given the attack being a hypothesis test , the audit report needs to include the analysis of the error versus power of the test : if we can tolerate a certain level of false positive rate in the inference attack , how much would be the true positive rate of the attack , over the random samples from member and non-member data ? The area under the curve for such an analysis reflects the chance that the membership of a random data point from the population or the training set can be inferred correctly . 3 CONSTRUCTING MEMBERSHIP INFERENCE ATTACKS . The adversary in the membership inference game can observe the output of a target machine learning model θ , trained on unknown dataset D. He also gets a precise target data point z as input , and is expected to output 0 or 1 to guess whether the sample z is in the dataset D or not . We use likelihood ratio test ( LRT ) as the most powerful criterion for choosing among membership hypotheses , under the following assumptions . 1 . The adversary knows , and can sample from the underlying data distribution π ( z ) over population z = ( xz , yz ) ∈ Dpop . 2 . A randomized training algorithm T : D 7→ θ takes in a training dataset D , which consists of i.i.d . samples from the data population , and produces a model θ that incurs a low loss ∑ ( x , y ) ∈D ` ( θ , x , y ) on dataset D. We denote P ( θ|D ) to be the posterior distribution of trained model θ given training dataset D. Definition 3.1 ( Approximated LRT for Membership Inference ) Let ( θ , z ) be random samples from the joint distribution of target model and target data point , specified by one of the following membership hypotheses . H0 : D n i.i.d.samples∼π ( z ) ←−−−−−−−−−−−−− Dpop , θ ∼ T ( D ) , z sample∼π ( z ) ←−−−−−−−− Dpop ( 1 ) H1 : D n i.i.d.samples∼π ( z ) ←−−−−−−−−−−−−− Dpop , θ ∼ T ( D ) , z sample←−−−− D ( 2 ) The likelihoods function of hypothesis H0 and H1 , given observed target model θ and target data point z , is as follows ( detailed derivations are in the Appendix ) . L ( H0|θ , z ) = PH0 ( θ , z ) = π ( z ) ·ED∼πn [ P ( θ|D ) ] ( 3 ) L ( H1|θ , z ) = PH1 ( θ , z ) = π ( z ) ·ED′∼πn−1 [ P ( θ|D′ ∪ z ) ] ( 4 ) Now we follow the previous construction of Bayes-optimal membership inference attack Sablayrolles et al . ( 2019 ) , and model the posterior distribution P ( θ|D ) of trained model as follows . P ( θ|D ) = e − 1T ∑ ( x , y ) ∈D ` ( θ , x , y ) ∫ e− 1 T ∑ ( x , y ) ∈D ` ( θ , x , y ) dθ , ( 5 ) where T is a temperature constant that allows some randomness in the training algorithm T . The equation 5 holds for Bayesian learning algorithms , such as stochastic gradient descent Polyak & Juditsky ( 1992 ) , deterministic MAP ( Maximum A Posteriori ) inference ( for T → 0 ) , and Bayesian posterior sampling Welling & Teh ( 2011 ) ( for T = 1 ) . Therefore , the LRT statistics can be computed as follows ( detailed derivations are in the Appendix ) . LR ( θ , z ) = L ( H0|θ , z ) L ( H1|θ , z ) = ED∼πn [ P ( θ|D ) ] ED′∼πn−1 [ P ( θ|D′ ∪ z ) ] ( 6 ) ≈ e 1T ` ( θ , xz , yz ) , ( 7 ) The LRT hypothesis test rejects H0 when the LRT statistic is small . By equation 7 , the rejection region { ( θ , z ) : LR ( θ , z ) ≤ c } can be approximated as follows . { ( θ , z ) : ` ( θ , xz , yz ) ≤ T · log c } ( 8 ) The above approximated LRT strategy compares a constant threshold T · log c with the loss ` ( θ , xz , yz ) of the target model θ on the target data point z . This recovers the commonly used ( constant ) loss threshold attacks in literature Nasr et al . ( 2019 ) ; Sablayrolles et al . ( 2019 ) . However , by derivations in the appendix ( equation 48 ) , a more accurate LRT attack strategy would compare the loss ` ( θ , xz , yz ) with a threshold function T · log ( c·ED′∼πn−1 [ P ( θ|D ′ ) ] ED∼πn [ P ( θ|D ) ] ) that depends on the tar- get model θ . Therefore , the attack in equation 8 with constant threshold cα can overly simplify the LRT , thus limiting its performance . This motivates our design of attacks with model-dependent and sample-dependent thresholds , with the objective of improving the attack performance . Our general template for attack construction . Building on the approximate LRT equation 8 , we derive the following variant of sample-dependent and ( or ) model dependent attack strategy . If ` ( θ , xz , yz ) ≤ cα ( θ , xz , yz ) , reject H0 , ( 9 ) where cα ( θ , xz , yz ) is a threshold function chosen by the attacker under α tolerance of false positive rate ( FPR ) , i.e. , cα ( θ , xz , yz ) satisfies the following equation which controls false positive rate . EPH0 ( θ , xz , yz ) [ 1 ` ( θ , xz , yz ) ≤cα ] = α ( 10 ) In the following attacks , we approximate the joint distribution PH0 ( θ , z ) with empirical distribution over its samples . This facilitates solving equation 10 , which give valid attack threshold cα ( θ , xz , yz ) .

Summary: Membership inference attacks are attacks that infer whether a given record is in the dataset of a given machine learning model or not. By definition, (the success of) such attacks is in contrast with differential privacy.  The paper proposes multiple attacks to infer membership in the data sets of machine learning models. The envisioned goal of the paper is that these attacks can provide a more complete picture about how private a machine learning algorithm is, when compared with other membership inference attacks. Hence, the ultimate goal is to provide “reports” that go beyond “just a number” to better audit the privacy of learning algorithms.