Year: 2021

Abstract:
Membership inference attacks are used as an auditing tool to quantify the private information that a model leaks about the individual data points in its training set. Membership inference attacks are influenced with different uncertainties that an attacker has to resolve about training data, training algorithm, and the underlying data distribution. Thus attack success rates, of many attacks in the literature, do not precisely capture the information leakage of models about their data, as they also reflect other uncertainties that the attack algorithm has. In this paper, we present a hypothesis testing framework that can explain the implicit assumptions and also the simplifications made in the prior work. We also derive new attack algorithms from our framework that can achieve a high AUC score while also highlighting the different factors that affect their performance. Our algorithms capture a very precise approximation of privacy loss in models, and can be used as a tool to perform an accurate and informed estimation of privacy risk in machine learning models. We provide a thorough empirical evaluation of our attack strategies on various machine learning tasks trained on benchmark datasets.

heading: 1 INTRODUCTION
text: Machine learning systems have come under intense scrutiny of the regulatory authorities in the past few years. Veale et al. (2018) argue that machine learning models could be considered personal data due to their susceptibility to inference attacks that can recover sensitive information about training data just from the models. Membership inference attacks (Homer et al., 2008; Dwork et al., 2015; Shokri et al., 2017) and reconstruction attacks (Dinur & Nissim, 2003; Song et al., 2017; Carlini et al., 2020) are the main inference attacks that highlight, and can quantify, the privacy risk of releasing aggregate information computed on sensitive data (Dwork et al., 2017). The focus of this paper will be on membership inference attacks for measuring privacy risk. Organizations such as the ICO (UK) and NIST (US) have highlighted membership inference as a potential confidentiality violation and privacy threat to the training data (Murakonda & Shokri, 2020). This has lead to the development of open-source tools 1 and capabilities in widely-used ML libraries 2 for measuring privacy risk from machine learning models using membership inference attacks.
Although the approach of quantifying privacy risk through membership inference attacks is gaining traction, the attack success, as measured by a lot of works, cannot be completely attributed to information leakage from the models and hence their privacy risk. Various factors such as the distribution of training data, difference in distributions of the train and test data may provide an over-estimate or under-estimate of the actual privacy risk from the model (Erlingsson et al., 2019; Humphries et al., 2020). Theoretical analyses that connect the success of membership inference to privacy risk through the framework of differential privacy avoid this issue by slightly modifying how the attack performance is measured (Yeom et al., 2018; Jagielski et al., 2020; Nasr et al., 2021; Malek et al., 2021). Instead of measuring the leakage from a particular model, these works aim at measuring the worst-case leakage of the training algorithm (for a given model architecture), and construct multiple models with and without one training point and keep the rest of training set fixed. The performance of the attack (false positive and false negative errors) is then computed over these sampled models.
1https://github.com/privacytrustlab/ml_privacy_meter 2https://blog.tensorflow.org/2020/06/introducing-new-privacy-testing-library.
html
However,in practice and during auditing, the performance needs to be computed based on the points in the training set versus the population data (test set) for a given fixed model. Various works rely on measuring privacy risk through the performance of membership inference attacks, but the subtle differences in how the attack is formulated mean that they might associate different causes of attack success to privacy loss. When auditing machine learning models, we need to pay attention to what exactly we are measuring and how it relates to the information leakage of the machine learning algorithm and not other factors such as the prior knowledge of the attacker.
Previous membership inference attacks are designed for high overall performance, i.e. to succeed for most member (or non-member) data points of most target models. With few exception, they train shadow (reference) models on datasets randomly sampled from a population, to mimic the general behavior of models on member (or non-member) data points. This general behavior, however, does not capture the behavior of specific models on specific samples. As a result, the limited performance of shadow model attack does not precisely measure the model (sample)-specific information leakage.
In this work, we focus on designing membership inference attacks that more precisely measure what a target model leaks about each individual training data, in a binary hypothesis testing framework. We start from the (constant) loss threshold attack, and study how to further design different attack strategy (loss threshold) for different target model (sample). We derive multiple attack strategies (existing and new) from this hypothesis test formulation via different approximations for the null hypothesis. We also formulate the performance of an attack inside the hypothesis testing framework, via the trade-off between its false positive rate and true positive rate. Following the methodology, we not only derive and explain existing shadow (reference) model attacks (Attack S and R), but also design new attacks (Attack P and D) that offer more accurate privacy auditing for machine learning models, through model-dependent and (or) sample-dependent attack strategies. We empirically evaluate and compare the attack performance (TPR-FPR curve and AUC score) and computation cost of our new attacks (notably Attack D), and the prior attacks on multiple datasets.

heading: 2 ATTACK FRAMEWORK
text: Our objective is to design a framework that enables auditing the privacy loss of machine learning models in the black-box setting (where only model outputs —and not their parameters or internal computations— are observable). This framework needs to have three elements: (i) the inference game as the evaluation setup; (ii) the indistinguishably metric to measure the privacy risk, and (iii) the construction of membership inference attack as hypothesis testing. The notion of privacy underlying our framework is primarily based on differential privacy, and multiple pieces of this framework are generalizations of existing inference attacks against machine learning algorithms. Instead of focusing on designing new attacks, we present the important design choices for constructing and evaluating membership inference attacks, for the purpose of having a precise privacy auditing. We identify different sources of uncertainty that influence the error of inference attacks, and can lead to miscalculation of the privacy loss of a model.
We quantify privacy loss of a model in a hypothetical inference game between a challenger and an adversary. We are given a private training set D, and a model θ which is trained on D using a training algorithm T . The challenger samples a random data point (x1, y1) from the training set (a member), and a random data point (x0, y0) from the data population outside the training set (a non-member). He then randomly selects one of the two (member or non-member) b ∼ {0, 1} with probability 12 , and shares the selected data point (xb, yb) and the model’s output f(xb; θ) with the adversary. The adversary’s task is to determine if the data point is a member or not (i.e., guess b).3
3We need to emphasize that there is a difference between the privacy loss of a machine learning algorithm and that of the specific models trained with the algorithm. A model is a given instance of a training algorithm, thus its leakage needs to be computed with respect to the individual data records in its specific training set. This subsequently means that the privacy loss of an algorithm varies depending on the randomness in sampling of its training data, and the randomness of the training algorithm. One can analyze the privacy loss of an algorithm as, for example, its worst case privacy loss (as in differential privacy). Differentially private algorithms enforce an upper bound on the privacy loss of an algorithm over all models with respect to all possible training data. Thus, in the inference game for differential privacy, the privacy loss of the training algorithms, would be the worst case privacy loss over all choices of D, (x0, y0), and (x1, y1) in our inference game for model privacy.
We use an indistinguishability measure (which is the basis of differential privacy) to define privacy of individual training data of a model. According to this measure, the privacy loss of the model with respect to its training data is the adversary’s success in distinguishing between the two possibilities b = 0 vs b = 1 over multiple repetitions of the inference game. Naturally, the inference attack is a hypothesis test, and adversary’s error is composed of the false positive (i.e., inferring a non-member as member) and false negative of the test. In practice, the error of adversary in each round of the inference game depends of multiple factors:
• The true leakage of the model about the target data (xb, yb) when b = 1.
• The uncertainty (belief or background knowledge) of attack algorithm about the population data
• The adversary’s uncertainty about the training algorithm T • The uncertainty about all training data except the target data (xb, yb)
• The attack dependency on the target data (xb, yb), and the model θ
In the ideal setting, we only want the attack error to be dependent on the true leakage of the model about the target data (i.e., whether the same model trained with and without (xb, yb) are distinguishable from each other). To this end, and to cancel out the effect of other uncertainties, an attack algorithm and the evaluation setup for the inference game need to be designed based on the following principle: The population data used for constructing the attack algorithm, and evaluating the inference game, need to be similar, in distribution, to the training data. This is to minimize the impact of prior belief (what could have been sampled for the training set) in the performance of the inference attack. This is not hard to achieve as all the process (of constructing the hypothesis testing attack, and evaluating it) is controlled by the auditor. By violating this principle, we might overestimate the privacy loss (by making the test dependent on a distinct prior knowledge) or underestimate the privacy loss (by evaluating the inference attack on a population data distribution for which it was not constructed).
Another crucial requirement is that the privacy audit needs to output a detailed report, which captures the uncertainty of the attack. Reporting one number as the accuracy of the attack, as it is mostly reported in the literature, is not an informative report. Given the attack being a hypothesis test, the audit report needs to include the analysis of the error versus power of the test: if we can tolerate a certain level of false positive rate in the inference attack, how much would be the true positive rate of the attack, over the random samples from member and non-member data? The area under the curve for such an analysis reflects the chance that the membership of a random data point from the population or the training set can be inferred correctly.

heading: 3 CONSTRUCTING MEMBERSHIP INFERENCE ATTACKS
text: The adversary in the membership inference game can observe the output of a target machine learning model θ, trained on unknown dataset D. He also gets a precise target data point z as input, and is expected to output 0 or 1 to guess whether the sample z is in the dataset D or not. We use likelihood ratio test (LRT) as the most powerful criterion for choosing among membership hypotheses, under the following assumptions.
1. The adversary knows, and can sample from the underlying data distribution π(z) over population z = (xz, yz) ∈ Dpop.
2. A randomized training algorithm T : D 7→ θ takes in a training dataset D, which consists of i.i.d. samples from the data population, and produces a model θ that incurs a low loss ∑ (x,y)∈D `(θ, x, y) on dataset D. We denote P (θ|D) to be the posterior distribution
of trained model θ given training dataset D.
Definition 3.1 (Approximated LRT for Membership Inference) Let (θ, z) be random samples from the joint distribution of target model and target data point, specified by one of the following membership hypotheses.
H0 :D n i.i.d.samples∼π(z)←−−−−−−−−−−−−− Dpop, θ ∼ T (D), z sample∼π(z)←−−−−−−−− Dpop (1)
H1 :D n i.i.d.samples∼π(z)←−−−−−−−−−−−−− Dpop, θ ∼ T (D), z sample←−−−− D (2)
The likelihoods function of hypothesis H0 and H1, given observed target model θ and target data point z, is as follows (detailed derivations are in the Appendix).
L(H0|θ, z) = PH0(θ, z) = π(z) ·ED∼πn [P (θ|D)] (3) L(H1|θ, z) = PH1(θ, z) = π(z) ·ED′∼πn−1 [P (θ|D′ ∪ z)] (4)
Now we follow the previous construction of Bayes-optimal membership inference attack Sablayrolles et al. (2019), and model the posterior distribution P (θ|D) of trained model as follows.
P (θ|D) = e − 1T ∑ (x,y)∈D `(θ,x,y)∫
e− 1 T ∑ (x,y)∈D `(θ,x,y)dθ , (5)
where T is a temperature constant that allows some randomness in the training algorithm T . The equation 5 holds for Bayesian learning algorithms, such as stochastic gradient descent Polyak & Juditsky (1992), deterministic MAP (Maximum A Posteriori) inference (for T → 0), and Bayesian posterior sampling Welling & Teh (2011) (for T = 1). Therefore, the LRT statistics can be computed as follows (detailed derivations are in the Appendix).
LR(θ, z) = L(H0|θ, z) L(H1|θ, z) = ED∼πn [P (θ|D)] ED′∼πn−1 [P (θ|D′ ∪ z)] (6)
≈ e 1T `(θ,xz,yz), (7) The LRT hypothesis test rejects H0 when the LRT statistic is small. By equation 7, the rejection region {(θ, z) : LR(θ, z) ≤ c} can be approximated as follows.{
(θ, z) : `(θ, xz, yz) ≤ T · log c }
(8)
The above approximated LRT strategy compares a constant threshold T · log c with the loss `(θ, xz, yz) of the target model θ on the target data point z. This recovers the commonly used (constant) loss threshold attacks in literature Nasr et al. (2019); Sablayrolles et al. (2019). However, by derivations in the appendix (equation 48), a more accurate LRT attack strategy would compare the loss `(θ, xz, yz) with a threshold function T · log ( c·ED′∼πn−1 [P (θ|D ′)]
ED∼πn [P (θ|D)]
) that depends on the tar-
get model θ. Therefore, the attack in equation 8 with constant threshold cα can overly simplify the LRT, thus limiting its performance. This motivates our design of attacks with model-dependent and sample-dependent thresholds, with the objective of improving the attack performance.
Our general template for attack construction. Building on the approximate LRT equation 8, we derive the following variant of sample-dependent and (or) model dependent attack strategy.
If `(θ, xz, yz) ≤ cα(θ, xz, yz), reject H0, (9) where cα(θ, xz, yz) is a threshold function chosen by the attacker under α tolerance of false positive rate (FPR), i.e., cα(θ, xz, yz) satisfies the following equation which controls false positive rate.
EPH0 (θ,xz,yz) [ 1`(θ,xz,yz)≤cα ] = α (10)
In the following attacks, we approximate the joint distribution PH0(θ, z) with empirical distribution over its samples. This facilitates solving equation 10, which give valid attack threshold cα(θ, xz, yz).

heading: 3.1 ATTACK S: MIA VIA SHADOW MODELS
text: We first formalize a shadow model membership inference attack S based on Shokri et al. (2017), that effectively uses label-dependent attack threshold cα(yz), as follows.
If `(θ, xz, yz) ≤ cα(yz), reject H0, (11) where the threshold function cα(yz) satisfies equation 10 and ensures false positive rate α. To approximate the joint distribution PH0(θ, xz, yz) in equation 3, the attacker samples the following set S of shadow models and shadow data points from the joint distribution PH0(θ, z).
S = ∪i=1,2,··· {(θi, zi1), (θi, zi2), · · · } (12)
∀i = 1, 2, · · · , θi ∼ T (Di) , Di n i.i.d.samples∼π(z)←−−−−−−−−−−−−− Dpop (13)
∀i = 1, 2, · · · , zi1, zi2, · · · i.i.d. ∼ π(z) (14)
The shadow models θs = {θi}i=1,2,··· trained on population datasets approximate the distribution ED∼πn [P (θ|D)] in equation 3, and the shadow points zi1, zi2, · · · for the i-th shadow model approximate the population data distribution π(z) in equation 3. Therefore, we approximate the joint distribution PH0(θ, z) in equation 10 with the empirical distribution over set S, as follows.
|{(θ, z) ∈ S : `(θ, xz, yz) ≤ cα(yz)}| |S| = α (15)
One sufficient condition for equation 15 to hold is that for any fixed data label y ∈ Y , we have |{(θ, z) ∈ S : `(θ, xz, yz) ≤ cα(yz) and yz = y}|
|{(θ, z) ∈ S : yz = y}| = α (16)
By solving equation 16 for every y ∈ Y , we compute that cα(y) equals the the α-percentile for the histogram of loss values `(θ, xz, yz) over samples {(θ, z) ∈ S : yz = y}. This recovers the class-dependent attack threshold function cα(yz) that we use in Attack S.

heading: 3.2 ATTACK P: MODEL-DEPENDENT MIA VIA POPULATION DATA
text: We design a new membership inference Attack P that uses attack threshold function cα(θ) dependent on the target model θ. The rationale for this design is to construct an inference attack which exploits the similar statistics as in Attack S, in a more accurate way by computing it on the target model, yet with less computations (without the need to train shadow models). The hypothesis test is as follows.
If `(θ, xz, yz) ≤ cα(θ), reject H0, (17) where the threshold function cα(θ) satisfies equation 10 and ensures false positive rate α. Using equation 23, we rewrite the false positive rate in equation 10 as follows.
EPH0 (θ) [ Eπ(z) [ 1`(θ,xz,yz)≤cα(θ) ]] = α (18)
One sufficient condition for equation 18 to hold is that, for any fixed target model θ, we have Eπ(z) [ 1`(θ,xz,yz)≤cα(θ) ] = α (19)
To approximate the data distribution π(z) in equation 19, the attacker samples the following set P of population data points from distribution π(z).
P = {zi}i=1,2,···, where z1, z2, · · · i.i.d. ∼ π(z) (20) Therefore, we approximate π(z) in equation 19 with the empirical distribution over samples z ∈ P .
|{z ∈ P : `(θ, xz, yz) ≤ cα(θ)}| |P | = α (21)
By solving equation 21 for every fixed target model θ, we compute that cα(θ) equals the the α− percentile for the histogram of loss values `(θ, xz, yz) over population data samples z = (xz, yz) ∈ P . This recovers the model-dependent attack threshold cα(θ) for Attack P.

heading: 3.3 ATTACK R: SAMPLE-DEPENDENT MIA VIA REFERENCE MODELS
text: The privacy loss of the model with respect to the target data could be directly related to how susceptible the target data is to be memorized (e.g., being an outlier) Feldman (2020). Based on this finding, we design the membership inference Attack R that uses attack threshold function cα(xz, yz) which depends on the target data (both its input features xz and the label yz). This attack is very similar to the membership inference attacks designed for summary statistics, graphical models and machine learning, which use reference models to compute the probability of the null hypothesis Sankararaman et al. (2009); Murakonda et al. (2021); Long et al. (2020). The hypothesis test is as follows.
If `(θ, xz, yz) ≤ cα(xz, yz), reject H0, (22) where the threshold function cα(xz, yz) satisfies equation 10 and ensures false positive rate α. Using the chain rule of joint distribution, and the independence of θ and z = (xz, yz) under null hypothesis H0 in equation 1, we prove that
PH0(θ, xz, yz) = PH0(xz, yz) · PH0(θ|xz, yz) = π(z) · PH0(θ), (23)
where PH0(θ) = ED∼πn [P (θ|D)] is the target model distribution specified by the null hypothesis H0. Using equation 23, we rewrite the false positive rate constraint in equation 10 as follows.
Eπ(z) [ EPH0 (θ) [ 1`(θ,xz,yz)≤cα(xz,yz) ]] = α (24)
One sufficient condition for equation 24 to hold is that, for any fixed data point z = (xz, yz), EPH0 (θ) [ 1`(θ,xz,yz)≤cα(xz,yz) ] = α (25)
To approximate the target model distribution PH0(θ) in equation 25, the attacker then samples the following set R of reference models.
R = {θi}i=1,2,···, where θi ∼ T (Di) , Di n i.i.d.samples∼π(z)←−−−−−−−−−−−−− Dpop (26)
Because the reference models θr = {θi}i=1,2,··· are trained on population datasets, they serve as samples from the distribution PH0(θ) = ED∼πn [P (θ|D)]. Therefore, we replace PH0(θ, z) in equation 25 with the empirical distribution over samples θi ∈ R, as follows.
|{θi ∈ R : `(θi, xz, yz) ≤ cα(xz, yz)}| |R| = α (27)
By solving equation 27 for fixed target sample z = (xz, yz), we compute that cα(xz, yz) equals the the α-percentile for the histogram of loss values `(θi, xz, yz) over reference models samples θi ∈ R. This computes the sample-dependent attack threshold cα(xz, yz) that in Attack R.

heading: 3.4 ATTACK D: MODEL-DEPENDENT AND SAMPLE DEPENDENT MIA VIA DISTILLATION
text: Can we design an attack that takes advantage of all the information available in the target model and the target data that can increase the chance of identifying the right hypothesis? We design a membership inference Attack D whose threshold function cα(θ, xz, yz) depends on both the target sample z and the target model θ, as follows.
If `(θ, xz, yz) ≤ cα(θ, xz, yz), reject H0, (28)
where the threshold function cα(θ, xz, yz) satisfies equation 10 and ensures false positive rate α. However, the degree of freedom in the threshold function cα(θ, xz, yz) is still too large for us to directly solve equation 10. Therefore, we restrict cα(θ, xz, yz) to take the following form.
cα(θ, xz, yz) = cα(Dθ, xz, yz), (29)
where Dθ = T −1(θ) is the training dataset for target model θ. For the simplicity of derivation, let us first assume that the randomized training algorithm T has a deterministic inverse mapping T −1 : θ → D, i.e. the training dataset for a given model θ is uniquely specified. (Later we also show how to approximate the training datasets Dθ for model θ when the training algorithm T is not invertible.) Plugging equation 29 into equation 10, we rewrite the FPR constraint as follows.
EPH0 (θ,xz,yz) [ 1`(θ,xz,yz)≤cα(Dθ,xz,yz) ] = α (30)
By deterministic mapping from θ to Dθ, the chain rule of joint distribution, and the independence between random variables θ and z = (xz, yz) under null hypothesis H0 in equation 1, we prove that
PH0(θ, xz, yz) = PH0(θ, xz, yz) · 1 = PH0(θ, xz, yz) · P (Dθ|θ) (31) = PH0(Dθ, θ, xz, yz) = PH0(Dθ, xz, yz) · PH0(θ|Dθ, xz, yz) (32) = PH0(Dθ, xz, yz) · P (θ|Dθ) (33)
where P (θ|Dθ) is distribution of trained model under a fixed training dataset Dθ, as specified in equation 5. Plugging equation 31 into equation 30, we rewrite the false positive rate constraint as
EPH0 (Dθ,xz,yz) [ EP (θ|Dθ) [ 1`(θ,xz,yz)≤cα(Dθ,xz,yz) ] ] = α (34)
One sufficient condition for equation 34 to hold is that, for any fixed sample z and target model θ, EP (θ′|Dθ) [ 1`(θ′,xz,yz)≤cα(Dθ,xz,yz) ] = α, (35)
whereDθ is an implicitly fixed training dataset for the target model θ, and the distribution P (θ′|Dθ) captures retrained models on the training dataset Dθ = T −1(θ) for given target model θ, as follows.
T ( T −1(θ) ) = θ′ with probability P (θ′|Dθ). (36)
To approximate this distribution of retrained model T ( T −1(θ) ) ∼ P (θ′|Dθ), the attacker generates the following set of self-distilled models using the target model θ.
M = {θi}i=1,2,···, where θi ∼ T (Di) , Di soft-labeled with θ←−−−−−−−−− D′i i.i.d.samples∼π(z)←−−−−−−−−−−−− Dpop (37)
These distilled models M = {θi}i=1,2,··· approximate samples from the retrained model distribution T (T −1(θ)) ∼ P (θ|Dθ). This is because θi is trained on distillation dataset Di consisting of population data points which are soft-labeled with the target model θ. This roughly recovers the target model θ trained on Dθ, however without its potential dependence on z. Therefore, the attacker approximate P (θ|Dθ) in the false positive rate constraint equation 35 with the empirical distribution over distilled models samples in M , as follows.
|{θi ∈M : `(θi, xz, yz) ≤ cα(Dθ, xz, yz)}| |M | = α (38)
By solving equation 38 for fixed target model θ and target point z, we compute that cα(Dθ, xz, yz) equals the the α-percentile for the histogram of loss values `(θi, xz, yz) on distilled models θi ∈M . This recovers the model-dependent and sample-dependent attack threshold cα(xz, yz) in Attack D.

heading: 3.5 ATTACK L: LEAVE ONE OUT ATTACK
text: An ideal attack, that removes the randomness over the training data (except the target data that could potentially be part of the training set) would be the leave one out attack. In this attack, the adversary trains reference models θ′ on D \ {(xb, yb)}. The attack would be in the same class of attacks as in Attack D, as it will be a model-dependent and data-dependent attack. It also runs a similar hypothesis test, however the attack requires assuming the adversary already knows the n − 1 data records in D \ {(xb, yb)}. This is a totally acceptable assumption in the setting of privacy auditing. Note that Attack D aims at reproducing the results of the leave-one-out attack without assuming the knowledge of n− 1 data records in D \ {(xb, yb)}.

heading: 3.6 SUMMARY AND COMPARISON OF DIFFERENT ATTACKS
text: For identifying whether a data point z has been part of the training set of θ, here are the main underlying questions for the attacks we present in this section:
• How likely is the loss `(θ, z) to be a sample from the distribution of loss of random samples from the population on (Attack P: the same model) (Attack S: models trained on the population data)? Depending on the tolerable false positive rate α and the estimated distribution of loss, we reject the null hypothesis.
• How likely is the loss `(θ, z) to be a sample from the distribution of loss of z on (Attack R: models trained on population data) (Attack D: models trained to be as close as possible to the target model, using distillation) (Attack L: models trained on n − 1 records from D excluding z)? Depending on the tolerable false positive rate α and the estimated distribution of loss, we reject the null hypothesis.
Effectively, these questions cover different types of hypothesis tests that could be designed for performing membership inference attacks. We expect these attacks to have a different error due to the uncertainties that can influence their performance.
Attacks S and P are of the same nature. However, attack S could potentially have a higher error due to its imprecision in using other models to approximate the loss distribution of the target model on population data. Attacks R, D, and L are also of the same nature. However, we expect attacks D to have more confidence in the tests due to reducing the uncertainty of other training data that can influence the model’s loss. Thus, we expect attack D to be the closest to the strongest attack which is the leave-one-out attack.

heading: 4 EXPERIMENTS
text: In this section, we empirically measure and study the performance of different attack strategies that we derived from our framework. A detailed analysis of the internal mechanics of these attacks and how they differ from each other is provided in the Appendix A.2.
Evaluation of attacks performance Which one of the attacks S, P, R, D has the best performance? How to evaluate the strength of attack besides attack accuracy? We quantify the attacker’s performance using two metrics: its true positive rate (TPR), and its false positive rate (FPR), over the random samples from member and non-member data and random target models. The ROC curve captures the tradeoff between the TPR and FPR of an attack, as its threshold cα is varied across different FPR tolerance α. The AUC (area under the ROC curve) score measures the strength of an attack. Therefore, we plot the ROC curves of all attacks on the Purchase100 dataset, and computes their AUC (area under the ROC curve) score in Figure 3 (in the appendix). The attack with the highest AUC score on Purchase100 is Attack D, which has the least level of uncertainty, as discussed Section 3. Table 1 and Table 4 shows more detailed AUC score results for different attacks under more settings.
Comparison between the Attacks Besides attack strength, how differently are the attacks performing on random input target models and target points? How do the different internal mechanics of attacks changing the attacker’s guess qualitatively? Do the attacks succeed on similar or different samples of member data and target model? Do the attacks have different level of confidence on the same input? How far away are the attack performance from the most ideal leave-one-out attacks described in Section 3? Answers to these questions require understanding how and why the attacks perform differently, for which we do detailed comparisons between attacks as follows.
• Similarity of attacks with each other. The scatter plot comparing Attack S and P in Figure 5 is roughly linearly centered around the diagonal, with slightly more points in NorthWest then in SouthEast. This shows that Attack S and Attack P almost always guesses membership of train points similarly, while Attack P performs slightly better than Attack S. Meanwhile, Attack D dominates Attack R for correctly guessing membership of train points because their comparison scatter plot is biased towards the NorthWest direction.
• Gap between attacks to the ground truth. From Table 2, among all attacks, Attack D agrees with the ground truth the most on train points, while the least on test points. This matches our observation that Attack D has a larger threshold under given α from Figure 1, which causes it to guess both more points as members. Besides that, we see that the agreement rate between Attack S and Attack P is as high as 0.94 both on train points and test points, this matches their linear comparison scatter plot in Figure 5. This may be because Attack P and Attack R both reduces one degree of uncertainty for the joint distribution under null hypothesis H0, as discussed in Section 3.
• Closeness of attacks to ideal leave-one-out attack. One interesting observation from Table 2 is that, among all the attacks, Attack D agrees with the Attack L the most, with agreement rate 0.98 on train points and 0.82 on test points. We believe this is because Attack D is highly similar in nature with Attack L, by approximating the training dataset of a target model and performing retraining, as discussed in Section 3.

heading: 5 SUMMARY
text: We provide a framework for auditing the privacy risk from machine learning models through membership inference attacks. The framework is used to derive attack strategies and also highlight the factors beyond leakage from the models that affect the attack performance. We also empirically analyze the performance of these attack strategies against models trained on benchmark datasets.

heading: A APPENDIX
text: A.1 DETAILED DERIVATION OF APPROXIMATED LRT FOR MEMBERSHIP INFERENCE
We first prove two useful approximation inequalities about the posterior distribution P (θ|D) as follows.
1. For arbitrary data point z = (zx, zy), and arbitrary dataset D, we have
P (θ|D ∪ z) ≥ e− `(θ,zx,zy) T · P (θ|D). (39)
2. Let z1, z2, · · · , zn be i.i.d. samples from the data distribution π(z). Then when n is large enough, for any model parameter θ, we have
Ez1,··· ,zn∼π[P (θ|z1, · · · , zn)] ≈ Ez1,··· ,zn−1∼π[P (θ|z1, · · · , zn−1)] (40)
We now offer details for deriving the approximated likelihood ratio test (LRT) for membership inference.
Definition A.1 (Approximated LRT for membership inference) Let (θ, z) be random samples from the joint distribution of target model and target data point, specified by one of the following membership hypotheses.
H0 :D n i.i.d.samples∼π(z)←−−−−−−−−−−−−− Dpop, θ ∼ T (D), z sample∼π(z)←−−−−−−−− Dpop (41)
H1 :D n i.i.d.samples∼π(z)←−−−−−−−−−−−−− Dpop, θ ∼ T (D), z sample←−−−− D (42)
The likelihoods function of hypothesis H0 and H1 given observed target model θ and target data point z is as follows.
L(H0|θ, z) = PH0(θ, z) = π(z) ·ED∼πn [P (θ|D)] (43) L(H1|θ, z) = PH1(θ, z) = ∑ D PH1(D, θ, z) = ∑ D π(z) · PH1(D|z) · P (θ|D) (44)
= π(z) ·ED′∼πn−1 [P (θ|D′ ∪ z)] (45)
(By equation 39) ≥ π(z) · e− 1T `(θ,zx,zy) ·ED′∼πn−1 [P (θ|D′)] (46)
(By equation 40) ≈ π(z) · e− 1T `(θ,zx,zy) ·ED∼πn [P (θ|D)] (47)
Therefore the LRT statistics is
λ(θ, z) = L(H0|θ, z) L(H1|θ, z) ≤ e 1T `(θ,zx,zy) ED∼π n [P (θ|D)] ED′∼πn−1 [P (θ|D′)] ≈ e 1T `(θ,zx,zy) (48)
The LRT hypothesis test rejects H0 when the LRT statistic is small. By equation 6, the rejection region {(θ, z) : λ(θ, z) ≤ c} can be approximated as follows.{
(θ, z) : `(θ, zx, zy) ≤ T · log c }
(49)
Datasets
• The Purchase100 dataset is based on Kaggle’s ”Acquire Valued Shoppers Challenge” that contains shopping histories for thousands of individuals4. We use a simplified preprocessed purchase dataset provided in Shokri et al. (2017). There are 197, 324 data points in this dataset, where each data point has 600 binary features. The data points are clustered into 100 classes to represent different shopping style.
• The CIFAR10 and CIFAR100 datasets are widely used benchmark datasets for image classification. The CIFAR10 dataset consists of 60, 000 data points in 10 classes, whereas the CIFAR100 dataset consists of 60, 000 data points in 100 classes. Here each data point is a 32 × 32 color image, and there are 50, 000 training images and 10, 000 test images in total.
• The MNIST dataset is a widely used benchmark dataset for image classification. The MNIST dataset consists of 70, 000 data points in 10 classes, where each data point is a 28×28 handwritten digit image. In total, there are 60, 000 training images and 10, 000 test images.
Experiment setup
1. For each configuration, we train up to n = 1000 models on random IID splits of the data. For the Purchase100 configurations, we use a 4 layer MLP with layer units = [512, 256,
4https://www.kaggle.com/c/acquire-valued-shoppers-challenge
128, 64]. For CIFAR100 and MNIST configurations, we use a 2 layer CNN with filters = [32, 64] and max pooling. For CIFAR10 configurations, we use AlexNet and a 3-block VGGNet. For all target models and datasets, we use SGD as the optimization function and categorical crossentropy loss. For Configuration IV of Purchase100, we use gradient clipping while training the models. For CIFAR10, we add momentum while training the models.
2. Target models: We use the first 10 models as target models on each of the Purchase100, CIFAR10, CIFAR100 and MNIST datasets. The performance of Attacks S, P, and R are evaluated on all 10 models, whereas Attack D is evaluated on the first 3 models.
3. Shadow models for Attack S and Reference models for Attack R: Given a target model, we use the remaining (n− 1) = 999 models as shadow models or reference models for Attack S and Attack R respectively.
4. Distilled models for Attack D: Given a target model, we use the same random IID splits used to train n = 1000 models to distill (n − 1) = 999 models using the soft labels from the target model.
5. Using n < 1000 models: For Configuration IV of Purchase100 and Configuration III of CIFAR10, we report the attack performance results using n = 30 shadow, reference, and distilled models. For Configurations I and II of CIFAR10 we report the attack performance results using n = 400 shadow, reference, and distilled models.
A.2 ADDITIONAL EMPIRICAL EVALUATION RESULTS
In this section, we provide a detailed analysis of the internal workings of the attack strategies. We specifically compare how the thresholds on loss values derived from these strategies differ on various target points and target models.
Illustration of Attack threshold How differently does the attack threshold in Attack S, P, R, D depend on the target model θ and the target data point z? How is the different dependence of threshold on θ and z in different attacks affecting their success? To investigate these important questions, we plot the loss histograms that different attacks use to compute the thresholds, on different input θ1, θ2 and z1, z2 in Figure 1. The loss histogram approximates the LRT statistics distribution under null hypothesis H0 in equation 1, with different uncertainty for different attacks, as discussed in 3. We see that model-dependence and sample-dependence of attack threshold reduces the uncertainty in the loss histogram. The attacks (Attack D) with more concentrated loss histogram are more likely to succeed.

heading: B RELATED WORK
text: The current work in this domain can be broadly grouped into three categories: 1) Empirical works for improving the existing attack strategies or adapting them to different settings and models 2) Theoretically/empirically analyzing the privacy risk in various systems using the existing attack strategies 3) Exploring the connections with differential privacy and using them to establish lower bounds on leakage and/or select privacy parameters. Below, we provide a brief summary of works in all the three categories.
Empirical attack strategies for membership inference: Shokri et al. (2017) demonstrated the vulnerability of machine learning models to membership inference attacks in the black-box setting, where the adversary has only a query access to the target model. The attack algorithm is based on the concept of shadow models, which are models trained on some attacker dataset that is similar to that of the training data. Membership inference is modeled as a binary classification task for an attack model that is trained on the predictions of shadow models on the attacker dataset. A huge literature followed this work extending the attacks to different setting like white box access (Nasr et al., 2019; Sablayrolles et al., 2019; Leino & Fredrikson, 2020), label-only access (Li & Zhang, 2020; Choquette-Choo et al., 2021), federated learning (Melis et al., 2019), transfer learning (Zou et al., 2020) and different types of data, models such as aggregate location data (Pyrgelis et al., 2017), generative models (Hayes et al., 2019), language models (Carlini et al., 2019; Song & Shmatikov, 2019; Carlini et al., 2020), sentence embeddings (Song & Raghunathan, 2020), and speech recognition models (Shah et al., 2021). Multiple works have looked at improving the attack methodology through a more fine grained analysis or by reducing the background knowledge and the compute power required to execute the attack (Long et al., 2018; Song & Mittal, 2021; Salem et al., 2018). See (Hu et al., 2021) for a more comprehensive list of membership inference attacks against machine learning models. All these works follow the same attack framework for membership inference but they either exploit a slightly different signal that is correlated with membership of a point in the training set or find an efficient way to exploit the already known signals.
Privacy risk analysis with membership inference attacks: Homer et al. (2008) performed the first membership inference attack on genome data to identify the presence of an individual’s genome in a mixture of genomes. Sankararaman et al. (2009) provided a formal analysis of this risk of detecting the presence of an individual from aggregate statistics computed on independent and binary attributes. Murakonda et al. (2021) extended this analysis to the case of releasing Discrete Bayesian networks learned from data with dependent attributes. Backes et al. (2016) perform an analysis similar to that of (Sankararaman et al., 2009) but for MicroRNA data (aggregate statistics computed on independent and continuous attributes). Dwork et al. (2015) provide a more extensive analysis when the released statistics are noisy and the attacker has only one reference sample to perform the attack. The key results of all these works establish the privacy risk of releasing aggregate statistics by quantifying the success of membership inference attacks as a function of the number of statistics released and the number of individuals in the dataset. Similar attempts were made to analyze the privacy risk of machine learning models through membership inference via the lens of mutual information (Farokhi & Kaafar, 2020) and generalization error (Yeom et al., 2018; Del Grosso et al., 2021). Beyond these theoretical analyses, membership inference attacks are also used to empirically study the trade-offs between privacy and other desirable characteristics for machine learning models such as fairness (Chang & Shokri, 2020), robustness to adversarial examples (Song et al., 2019), and providing explanations (Shokri et al., 2021).
Differential privacy and Membership inference: The definitions of differential privacy (Dwork et al., 2006) and membership inference (Homer et al., 2008; Dwork et al., 2015; Shokri et al., 2017) are very closely connected and the hypothesis testing interpretation of differential privacy provides a clear view of the relationship between them. Satisfying differential privacy is equivalent to imposing a bound on the ability to distinguish any two neighboring datasets that differ by the presence of one individual i.e., inferring about the presence/absence of the individual. The bound is stated as a tradeoff between the type-I and type-II errors in distinguishing the two neighboring datasets. Hypothesis testing interpretation of differential privacy is very useful in deriving tight compositions (Kairouz et al., 2015) and has even motivated a new relaxed notion of differential privacy called f-DP (Dong et al., 2019). By definition, differentially private algorithms bound the success of membership inference attacks. Multiple works (Yeom et al., 2018; Erlingsson et al., 2019; Humphries et al., 2020), each improving on the previous work, have provided upper bounds on the success of membership inference attacks as a function of the parameters in differential privacy. Jayaraman & Evans (2019) evaluated the performance of membership inference attacks on machine learning models trained
with different values of epsilon under different relaxed notions of differential privacy. Rahman et al. (2018) also use membership inference attacks to measure the privacy loss on models trained with differentially private algorithms. The empirical performance of membership inference attacks has also been used to provide lower bounds on the privacy guarantees achieved by various differentially private algorithms (Jagielski et al., 2020; Nasr et al., 2021; Malek et al., 2021). The key difference between the empirical analysis of membership inference in the previous three works and other works is that they simulate the exact adversary in differential privacy i.e., they train multiple models with and without one particular training point and keep the rest of training set fixed. The performance of the attack (type I and type II errors) is averaged over these models, whereas in the previous works the model is fixed and the performance is computed as an average over points in the training and test sets. This simulation of the DP adversary helps in removing the effects of other points in the dataset when measuring the leakage through the model about a particular point of interest.

Review summaries:
Membership inference attacks are attacks that infer whether a given record is in the dataset of a given machine learning model or not. By definition, (the success of) such attacks is in contrast with differential privacy.  The paper proposes multiple attacks to infer membership in the data sets of machine learning models. The envisioned goal of the paper is that these attacks can provide a more complete picture about how private a machine learning algorithm is, when compared with other membership inference attacks. Hence, the ultimate goal is to provide “reports” that go beyond “just a number” to better audit the privacy of learning algorithms. 

The focus of this paper is on membership inference attacks. In particular, the paper aims at providing a framework that can help to access how different factors beyond information leakage from the model affect the performance of membership inference attacks, and how to design attacks that cancel out the effects off other factors. More specifically, a framework for understanding the relationship between, success of membership inference attacks and information leakage is introduced. Experiments using well-known datasets on visual tasks are provided to evaluate the performance  of the proposed attacks(in terms of AUC scores. Various frameworks are discussed in Section 3.   

This paper proposed a hypothesis testing framework for membership inference attacks (MIAs). The framework involves different algorithms for finding the loss threshold for MIAs, i.e. if a target data point with loss less than the threshold value then it will be inferred as a member of training data. The notation of membership privacy loss is the same as in differential privacy and the proposed algorithms improves the utility of the attacks compared to prior works on benchmark datasets.