Privacy Auditing of Machine Learning using Membership Inference Attacks
Authors: Jiayuan Ye
Aadyaa Maddi
Sasi Kumar Murakonda
Reza Shokri
Venue: ICLR 2022 Submitted (2022)

Abstract: Membership inference attacks determine if a given data point is used for training a target model. Thus, this attack could be used as an auditing tool to quantify the private information that a model leaks about the individual data points in its training set. In the last five years, a variety of membership inference attacks against machine learning models are proposed, where each attack exploits a slightly different clue. Also, the attacks are designed under different implicit assumptions about the uncertainties that an attacker has to resolve. Thus attack success rates do not precisely capture the information leakage of models about their data, as they also reflect other uncertainties that the attack algorithm has (for example, about data distribution or characteristics of the target model). In this paper, we present a framework that can explain the implicit assumptions and also the simplifications made in the prior work. We also derive new attack algorithms from our framework that can achieve a high AUC score while also highlighting the different factors that affect their performance. Thus, our algorithms can be used as a tool to perform an accurate and informed estimation of privacy risk in machine learning models. We provide a thorough empirical evaluation of our attack strategies on various machine learning tasks trained on benchmark datasets.

Under review as a conference paper at ICLR 2022
PRIVACY AUDITING OF MACHINE LEARNING USING MEMBERSHIP INFERENCE ATTACKS
Anonymous authors Paper under double-blind review
ABSTRACT
Membership inference attacks are used as an auditing tool to quantify the private information that model leaks about the individual data points in its training set. Membership inference attacks are influenced with different uncertainties that an attacker has resolve about training data, training algorithm, and the underlying data distribution. Thus attack success rates, of many attacks in the literature, do not precisely capture the information leakage of models about their data, as they also reflect other uncertainties that the attack algorithm has. In this paper, we present a hypothesis lesting framework that can explain the implicit assumptions and also the simplifications made in the prior work. We also derive new attack algorithms from our framework that can achieve a high AUC score while also highlighting the different factors that affect their performance. Our algorithms capture a very precise approximation of privacy loss models, and can be used as tool perform an accurate and informed estimation of privacy risk in ma- chine learning models. We provide a thorough empirical evaluation of our attack strategies on various machine learning tasks trained on benchmark datasets.
INTRODUCTION
Machine learning systems have come under intense scrutiny of the regulatory authorities in the past few years. VVeale et al. (2018) argue that machine learning models could be considered personal data due to their susceptibility to inference attacks that can recover sensitive information about training
data just from the models. Membership inference attacks (Homer et al. 2008 Dwork et al. 2015 Shokri et al. 2017) and reconstruction attacks (Dinur & Nissim 2003 et al. 2020) are the main inference attacks that highlight, and can quantify, the privacy risk of releasing aggregate information computed sensitive data (Dwork et al., 2017). The focus of this paper will be on membership inference attacks for measuring privacy risk. Organizations such as the ICO (UK) and NIST (US) have highlighted membership inference as a potential confidentiality violation and privacy threat to the training data (Murakonda & Shokri 2020). This has lead to the development of open-source tools and capabilities in widely-used ML libraries for measuring privacy risk from machine learning models using membership inference attacks.
Song et al. 2017| Carlini
Although the approach of quantifying privacy risk through membership inference attacks is gaining traction, the attack success, as measured by a lot of works, cannot be completely attributed to infor- mation leakage from the models and hence their privacy risk. Various factors such as the distribution of training data, difference in distributions of the train and test data may provide over-estimate or under-estimate of the actual privacy risk from the model (Erlingsson et al. 2019 Humphries et al. 2020). Theoretical analyses that connect the success of membership inference to privacy risk through the framework of differential privacy avoid this issue by slightly modifying how the attack performance is measured (Yeom et al. 2018 Jagielski et al. 2020 Nasr et al.) 202 Malek et al.) 2021]. Instead of measuring the leakage from a particular model, these works aim at measuring the worst-case leakage of the training algorithm (for a given model architecture), and construct multiple models with and without one training point and keep the rest of training set fixed. The performance of the attack (false positive and false negative errors) is then computed over these sampled models.
www
tensor www.:wooog/.oo0. 06/ wtt.://nnnnnttttuooooo roducing ng- new pri vacy-test ng- 11 brary.
html

---

Under review as a conference paper at ICLR 2022
However,in practice and during auditing, the performance needs to be computed based the points in the training set versus the population data (test set) for a given fixed model. Various works rely on measuring privacy risk through the performance of membership inference attacks, but the subtle differences in how the attack is formulated mean that they might associate different causes of attack success to privacy loss. When auditing machine learning models, we need to pay attention to what exactly we are measuring and how it relates to the information leakage of the machine learning algorithm and not other factors such as the prior knowledge of the attacker.
Previous membership inference attacks are designed for high overall performance, i.e. to succeed for most member (or non- non-member) data points of most target models. With few exception, they train shadow (reference) models on datasets randomly sampled from a population, to mimic the general behavior of models on member (or non-member) data points. This general behavior, however, does not capture the behavior of specific models on specific samples. As result, the limited performance of shadow model attack does not precisely measure the model ssample)-ppccifee information leakage.
In this work, we focus on designing membership inference attacks that more precisely measure what a target model leaks about each individual training data, in a binary hypothesis testing framework. We start from the (constant) loss threshold attack, and study how to further design different attack strategy (loss threshold) for different target model (sample). We derive multiple attack strategies (existing and new) from this hypothesis test formulation via different approximations for the null hypothesis. We also formulate the performance of an attack inside the hypothesis testing framework, via the trade-off between its false positive rate and true positive rate. Following the methodology, we not only derive and explain existing shadow (reference) model attacks (Attack S and R), but also design new attacks (Attack P and D) that offer more accurate privacy auditing for machine learn- ing models, through model-dependent and (or) sample-dependent attack strategies. We empirically evaluate and compare the attack performance (TPR-FPR curve and AUC score) and computation cost of our new attacks (notably Attack D), and the prior attacks on multiple datasets.
2 ATTACK FRAMEWORK
Our objective is to design a framework that enables auditing the privacy loss of machine learning models in the black-box setting (where only model outputs and not their parameters or internal computations- are observable). This framework needs to have three elements: (i) the inference game the evaluation setup; (ii) the indistinguishably metric measure the privacy risk, and (iii) the construction of membership inference attack as hypothesis testing. The notion of privacy under- lying our framework primarily based on differential privacy, and multiple pieces of this frame- work are generalizations of existing inference attacks against machine learning algorithms. Instead of focusing on designing new attacks, we present the important design choices for constructing and evaluating membership inference attacks, for the purpose of having precise privacy auditing. We identify different sources of uncertainty that influence the error of inference attacks, and can lead to miscalculation of the privacy loss of model.
We quantify privacy loss of a model in a hypothetical inference game between challenger and an adversary. We are given a private training set D, and a model @ which is trained on D using training algorithm T. The challenger samples a random data point (I v1) from the training set (a member), and a random data point (zo, Mo) from the data population outside the training set (a mon-member). He then randomly selects one of the two (member or non-membery b ~ (0,1) with probability } and shares the selected data point (It,, yb) and the model's output fæz; 6) with the adversary. The adversary's 's task is to determine if the data point is member or not (i.c., guess b) F
We need to emphasize that there is difference between the privacy loss of a machine learning algorithm and that of the specific models trained with the algorithm. A model is a given instance of a training algorithm, thus its leakage needs to be computed with respect to the individual data records in its specific training set. This subsequently means that the privacy loss of an algorithm varies depending on the randomness in sampling of its training data, and the randomness of the training algorithm. One can analyze the privacy loss of an algorithm as, for example, its worst case privacy loss (as in differential privacy). Differentially private algorithms enforce an upper bound on the privacy loss of an algorithm over all models with respect to all possible training data. Thus, in the inference game for differential privacy, the privacy loss of the training algorithmss, would be the worst case privacy loss over all choices of D, (xo, 20), and (x1 y1) in our inference game for model privacy.
2

---

Under review as a conference paper at ICLR 2022
We use an indistinguishabiitt measure (which is the basis of differential privacy) to define privacy of individual training data of model. According to this measure, the privacy loss of the model with respect to its training data is the adversary's success in distinguishing between the two possibilities b = 0 vs 1 = over multiple repetitions of the inference game. Naturally, the inference attack is a hypothesis test, and adversary's error composed of the false positive (i.e., inferring non-member as member) and false negative of the test. In practice, the error of adversary in each round of the inference game depends of multiple factors:
The true leakage of the model about the target data (2a, Wo) when = 1.
The uncertainty (belief background knowledge) of attack algorithm about the population data
The adversary's uncertainty about the training algorithm T
The uncertainty about all training data except the target data (I' yb)
The attack dependency on the target data (#p. 3b). and the model 0
In the ideal setting, we only want the attack error be dependent on the true leakage of the model about the target data (i.e., whether the same model trained with and without (ra, yb) are distinguish- able from each other). To this end, and to cancel out the effect of other uncertainties, an attack algorithm and the evaluation setup for the inference game need to be designed based on the fol- lowing principle: The population data used for constructing the attack algorithm, and evaluating the inference game, need to be similar, in distribution, to the training data. This is to minimize the impact of prior belief (what could have been sampled for the training set) in the performance of the inference attack. This is not hard to achieve as all the process (of constructing the hypothesis testing attack, and evaluating it) is controlled by the auditor. By violating this principle, we might overesti- mate the privacy loss (by making the test dependent on distinct prior knowledge) or underestimate the privacy loss (by evaluating the inference attack on a population data distribution for which it was not constructed).
Another crucial requirement is that the privacy audit needs to output a detailed report, which captures the uncertainty of the attack. Reporting one number as the accuracy of the attack, as it is mostly reported in the literature, is not an informative report. Given the attack being a hypothesis test, the audit report needs to include the analysis of the error versus power of the test: if we can tolerate certain level of false positive rate the inference attack, how much would be the true positive rate of the attack, over the random samples from member and non-member data? The area under the curve for such an analysis reflects the chance that the membership of a random data point from the population the training set can be inferred correctly.
3 CONSTRUCTING MEMBERSHIP INFERENCE ATTACKS
The adversary in the membership inference game can observe the output of a target machine learning model 0, trained on unknown dataset D. He also gets a precise target data point z as input, and expected to output 0 or to guess whether the sample is in the dataset D or not. We use likelihood ratio test (LRT) as the most powerful criterion for choosing among membership hypotheses, under the following assumptions.
1. The adversary knows, and can sample from the underlying data distribution T(2) over population 2 = (xz, y=) E D pap--
2. A randomized training algorithm T D + 0 takes in training dataset D, which con- sists of i.i.d. samples from the data population, and produces model 0 that incurs a low loss ((nw)) E(0, Ir y) on dataset D. We denote P)) be the posterior distribution of trained model 0 given training dataset D.
Definition 3.1 (Approximated LRT for Membership Inference) Lei (0,z) be random samples from the joint distribution of target model and target data point, specified by one of the following membership hypotheses.
Hg :D f..d i.d.d. (smpieaws)))
D Dpop, ~ T(D),, samplewar(s) D Dpop
(1)
H : (id.ecappeewwr(a)
D Ppopy T(D), sample D
(2)
3

---

Under review as a conference paper at ICLR 2022
The likelihoods function of hypothesis Hu and H1. given observed target model 0 and target data point 2, is as follows (detailed derivations are in the Appendix).
L 2) = PHo, (0, 2) = (2)) ED [P
(3)
L L () = PH, (0, = (2)) En (P( U2)
(4)
Now we follow the previous construction of Bayes-optimal membership inference attack Sablayyollllal et al. (2019) and model the posterior distribution P((]D) trained model as follows.
+ DCCnDED ((0...3))
P) = f 8(0...2008
(5)
where T a temperature constant that allows some randomness in the training algorithm T. The equation [3 holds for Bayesian learning algorithms, such as stochastic gradient descent Polyak & Juditsky (992), deterministic MAP (Maximum A Posteriori) inference (for T + 0), and Bayesian posterior sampling Welling & Teh (2 (for 1). Therefore, the LRT statistics can be computed as follows (detailed derivations are in the Appendix).
LR(0, 2) L(Ho)0,22) 2)
ED [P(O
(6)
(.2)
EDi D
Re e (00,a.00)))
The LRT hypothesis test rejects Ho when the LRT statistic is small. By equation 7 the rejection region (00,2) LR(0,2) S eJ can be approximated as follows.
(0.2)) : E(0, 22. Uz) S T llge
(8)
The above approximated LRT strategy compares a constant threshold T -log with the loss E(, U2. V2) of the target model 0 on the target data point z. This recovers the commonly used (constant) loss threshold attacks in literature Nast et al. (2019): Sablayrolles et al. (2019). However, by derivations in the appendix (equation 48). a more accurate LRT attack strategy would compare
the loss E(A, Iz. Uz) with a threshold function T llog (o.p p> (P() that depends on the tar- Ep-." D))
get model 0. Therefore, the attack in equation g with constant threshold ea can overly simplify the LRT, thus limiting its performance. This motivates our design of attacks with model-dependent and sample-dependent thresholds, with the objective of improving the attack performance.
Our general template for attack construction. Building on the approximate LRT equation g we derive the following variant of sample-dependent and (or) model dependent attack strategy.
If E(0,, x2. y=) S (6, yz), reject Ho.
(9)
where co 10. Uz) is a threshold function chosen by the attacker under tolerance of false positive rate (FPR), i.., Ca (0, wz: y=) satisfies the following equation which controls false positive rate.
E Erno (0.)...)) [_( =
(10)
In the following attacks, we approximate the joint distribution PH, (0, 2) with empirical distribution over its samples. This facilitates solving equation 10 which give valid attack threshold ca,00,
(0, Fz, vs).
3.1 ATTACK S: MIA VIA SHADOW MODEL.S
We first formalize shadow model membership inference attack S based on Shokri et al. (017), that effectively uses label-dependent attack threshold ca (uz), as follows.
If E(0, x., y=) S c. (y=). reject Ho.
(11)
where the threshold function c. (y=) satisfies equation 10 and ensures false positive rate a. To
approximate the joint distribution PHo (0, xz, Vz) in equation 3 the attacker samples the following set S of shadow models and shadow data points from the joint distribution (H, (0,
S U {(0,, 2(), (0,, ..)....
(12)
0, 0~~ T(D:) D; t p.t.s.ampllammss D pop
Vi = 1,2, ...,
(13)
Wi 1,2,
2/, ......iidd. T2))
(14)
4

---

Under review as a conference paper at ICLR 2022
The shadow models 0g
{0, (8.)i=1 1=1,2, trained population datasets approximate the distribu- tion ED- ((0) P]] in equation 3 and the shadow points 22. ...... for the i-th shadow model approximate the population data distribution T(2) equation 3
Therefore, we approximate the joint distribution PH. (0, in equation with the empirical distribution over set S, as follows.
K(0, x) E 2(0, T2,
T2, ya) S co (y=)H
(15)
One sufficient condition for equation 15 to hold is that for any fixed data label E Y, we have
[(0, 2) E S ((0 22, Vz) S ca (w=) and y: = y}
(16)
[(0,, 2) E S ys =
By solving equation 16 for every y Y, compute that (y) equals the the a-percentile for the histogram of loss values ((0, x2 yz) over samples {(0, 2) E S y/ = y This recovers the class-dependent attack threshold function Ca (V=) that we use in Attack S.
3.2 ATTACK P: SEDDEL--EPENEETT MIA VIA POPULATION DATA
We design a new membership inference Attack P that uses attack threshold function (0) dependent on the target model 0. The rationale for this design is to construct an inference attack which exploits the similar statistics as in Attack S, in more accurate way by computing it on the target model, yet with less computations (without the need to train shadow models). The hypothesis test is as follows.
If (0, Fz, U2) S (o), reject Ho,
(17)
where the threshold function ca (0) satisfies equation I0 and ensures false positive rate a. Using equation 23 we rewrite the false positive rate in equation 10 follows.
EPH,t0 (0)
E Ex(=)
[I.00.....))cc.00))] a
(18)
One sufficient condition for equation 18 to hold is that, for any fixed target model 6 we have
Ex(=) [1(0.0...)))..00))
(19)
To approximate the data distribution (2)) in equation 19 the attacker samples the following set P of population data points from distribution πt (2).
p= i=,,,., where 2 22, .2,.....i.... x(2)
(20)
Therefore, we approximate 7 (2) in equation 19 with the empirical distribution over samples 2 E P.
H P ((), x2 U2) ca))))
a
(21)
P
By solving equation 21 for every fixed target model 6, we compute that (0) equals the the a - -percentile for the histogram of loss values E(A, Zz, yz) over population data samples (Zz) E P. This recovers the model-dependent attack threshold c. (0) for Attack P.
3.3 ATTACK R: SAMPLE-DEPENDER MIA VIA REFERENCE MODELS
The privacy loss of the model with respect to the target data could be directly related to how suscepti- ble the target data is to be memorized (e.g., being outlier) Feldman (2020). Based on this finding, we design the membership inference Attack R that uses attack threshold function co (x Mz) which depends the target data (both its input features x, and the label yz). This attack is very similar to the membership inference attacks designed for summary statistics, graphical models and machine learning, which use reference models to compute the probability of the null hypothesis Sankarara- man et al. (2009); Murakonda et al. (2021); Long et al. (2020). The hypothesis test is as follows.
If E 2. ye) S ca (IF. reject Ho.
(22)
where the threshold function e, (Ig, y2) satisfies equation 10 and ensures false positive rate a. Us- ing the chain rule of joint distribution, and the independence of 0 and (æ2, under null hypothesis Ho in equation [ we prove that
PHo 10, Wz)
Pito (Z2, yz) Pro (0) yz) (=)) Prro (0),
(23)
5

---

Under review as a conference paper at ICLR 2022
where PHo (0) D
P(]ID)] is the target model distribution specified by the null hypothesis
Hy- Using equation 23
we rewrite the false positive rate constraint in equation 10 as follows.
E (x)
(p)
Ds (v.... a
(24)
One sufficient condition for equation 24 to hold that, for any fixed data point (Zzg Vz),
E (p) ( U(0 (w.. = a
(25)
To approximate the target model distribution PHo (0) in equation 25 following set of reference models.
the attacker then samples the
s.d sampieentt
R = 10,,,1,,,.... where 02 ~ T(D;) . DD;
D pop
(26)
Because the reference models 0, 11,2,- are trained on population datasets, they serve as samples from the distribution PH. (6) ED-r- [P(0 D)). Therefore, we replace PH, (0, in equation 25 with the empirical distribution over samples 0, R, as follows.
10,- E R E(0 Ue) S ca 2=)H
(27)
[R]
By solving equation 27 for fixed target sample (I', yz), compute that Car (I2, Us) equals the the aepercentile for the histogram of loss values T2. y/) over reference models samples 0g E R. This computes the sample-dependent attack threshold ca (22; V=) that in Attack R.
3.4 ATTACK D: DEDEE--DEENDENT AND SAMPLE DEPENDENT MIA VIA DISTILLATION
Can we design attack that takes advantage of all the information available in the target model and the target data that can increase the chance of identifying the right hypothesis? We design membership inference Attack D whose threshold function ca (0,2 22 Uz) depends on both the target sample z and the target model 0, as follows.
If 8(0, 22,, W2) co (0, reject Ho:
(28)
where the threshold function Ca (0. xz, V=) satisfies equation 10 and ensures false positive rate a. However, the degree of freedom the threshold function ca (8, Fg. Y2) is still too large for us to directly solve equation 10 Therefore, we restrict c. (0, 22, y2) to take the following form.
co (o, 12, M2) = ca (Do, X2, 1z)
(29)
where Da (0) is the training dataset for target model 0. For the simplicity of derivation, let us first assume that the randomized training algorithm T has deterministic inverse mapping T-1, 0 + D. i.e. the training dataset for a given model is uniquely specified. (Later we also show how to approximate the training datasets Dg for model 0 when the training algorithm T is not invertible.) Plugging equation
29 into equation 10
we rewrite the FPR constraint as follows.
E EPH. (0.w..0))
1e. (Do.....)) a
(30)
By deterministic mapping from f to Dg. the chain rule joint distribution, and the independence between random variables and = (Is, Us) under null hypothesis Ho in equation we prove that
Pho (0, I2 Nz) = PHo (0, Nz, (2) 1 PHo (0, xz, x0, Nz) P(Da
(31)
PH, (Do, 0, Te, v2) PH. (D,, (Do, N2) PH. (0) De.
(32)
PHo (Dg. 22. yz) Pa)
(33)
where Pg) is distribution of trained model under fixed training dataset Do. as specified in equation Plugging equation 31 into equation 30 we rewrite the false positive rate constraint as
E PHo (De.
E Ep(e)De)
[1e (.le. So. (Do weoe = a
(34)
One sufficient condition for equation 34 to hold is that, for any fixed sample z and target model 0,
E 1
(Do.tt (st
a,
(35)

---

Under review as a conference paper at ICLR 2022
where Dg is implicitly fixed training dataset for the target model 6, and the distribution P(6' D&) captures retrained models on the training dataset Dg = T (0) for given target model 8, as follows.
T (T) = 6' with probability P(0' (D)).
(36)
To approximate this distribution retrained model 7 (T) ~ P(O')D)) the attacker generates the following set of self-distilled models using the target model 0.
M = [0,) where 02 ~ T(D:) Dr laheled with
D i.d. (.d.sammtesswx(a)
D Doo
(37)
These distilled models M = {0, 1,1,,,..... approximate samples from the retrained model distribu- tion T(T-(()) Pg) This is because 0g is trained on distillation dataset D, consisting of population data points which are soft-labeled with the target model 6. This roughly recovers the tar- get model 0 trained on Do. however without its potential dependence on Therefore, the attacker approximate P(0) Da) in the false positive rate constraint equation 35 with the empirical distribution over distilled models samples M, as follows.
K0,- E M E(O,,, tes We) S ca (Do.
(38)
[M]
By solving equation 38 for fixed target model 9 and target point 2, we compute that ca (Da, 22 M/ equals the the a-percentile for the histogram of loss values {(,,, Tz, V=) on distilled models Og E M. This recovers the model-dependent and sample-dependent attack threshold ca (x2, Uz) in Attack D.
3.5 ATTACK L: LEAVE ONE OUT ATTACK
An ideal attack, that removes the randomness over the training data (except the target data that could potentially be part of the training set) would be the leave out attack. In this attack, the adversary trains reference models 8' on D \ {(Ir, y6)}- The attack would be in the same class of attacks in Attack D, as it will be a model-dependent and data-dependent attack. It also runs similar hypothesis test, however the attack requires assuming the adversary already knows the n 1 data records in D {1 v6)]. This is a totally acceptable assumption in the setting of privacy auditing.
Note that Attack D aims at reproducing the results of the leave-one-out attack without assuming the knowledge of n data records in D {(T., y6)].
3.6 SUMMARY AND COMPARISON OF DIFFERENT ATTACKS
For identifying whether data point z has been part of the training set of 0, here the main underlying questions for the attacks we present in this section:
How likely the loss E(0, z) be sample from the distribution of loss of random samples from the population on (Attack P: the same model) (Attack S: models trained on the population data)? Depending on the tolerable false positive a and the estimated distribution of loss, reject the null hypothesis.
How likely the loss E(0,2) 2) be a sample from the distribution of loss of on (Attack R: models trained on population data) (Attack D: models trained to be as close as possible to the target model, using distillation) (Attack models trained on records from excluding 2)? De- pending on the tolerable false positive rate a and the estimated distribution of loss, we reject the null hypothesis.
Effectively. these questions cover different types of hypothesis tests that could be designed for per- forming membership inference attacks. We expect these attacks to have a different error due to the uncertainties that can influence their performance.
Attacks S and P are of the same nature. However, attack could potentially have a higher error due to its imprecision in using other models to approximate the loss distribution of the target model on population data. Attacks R, D. and L are also of the same nature. However, we expect attacks D to have more confidence in the tests due to reducing the uncertainty of other training data that can influence the model's loss. Thus, we expect attack to be the closest to the strongest attack which the leave-one-out attack.
7

---

Under review as a conference paper at ICLR 2022
Point
Abtack
Attack
Distibution Attock 5
0.07
0.07
Atrack
Attock
Attack
0.04
0.04
0.01
0.01
10.5
10-1
10
10
10-1
10
101
10a
10-3
10-1
10
10"2
10"h
10P
10:
102
Loss Value
LosG Volue
LIsS Print
Abbock
Ottack
0.08
0.08
Loss Datt butio
Loss Distr tontio
0.07
Altack R.
0.07
Attack
Attack D
Attack
0.05
0.05
0.04
0.04
0.02
0.02
0.01
0.01
10-4
10-1
10-3
10-2
10
10
10a
10a
10-4
10-1
10-2
10-2
10
10P
10-
102
Value
Loss
(21, O2)
(22g θ1)
Figure/ 1: Loss distributions used by Attack P, R, D for points 21, 22 and target model 0, in Purchase100 Configuration Ila. = 0.1. Note that both 21 and are datapoints from the same class. We also show the loss distributions for 21, 22 on another target model 0g the appendix.
EXPERIMENTS
In this section, we empirically measure and study the performance of different attack strategies that derived from our framework. A detailed analysis of the internal mechanics of these attacks and how they differ from each other is provided the Appendix A.2
Evaluation of attacks performance Which of the attacks S, P. R, D has the best performance? How to evaluate the strength of attack besides attack accuracy? We quantify the attacker's s perfor- mance using metries: its true positive rate (TPR), and its false positive rate (FPR), random samples from member and non-member data and random target models. The ROC curve captures the tradeoff between the TPR and FPR of an attack, as its threshold ca is varied across different FPR tolerance a. The AUC (area under the ROC curve) score measures the strength of an attack. Therefore, their AUC (area under the ROC curve) score in Figure 3 (in the appendix). The attack with the high- est AUC score on Purchase 100 is Attack D, which has the least level of uncertainty, as discussed Section 3 Table 1 and Table 4 shows more detailed AUC score results for different attacks under more settings.
the
100 plot the ROC curves of all attacks on the Purchase 100 dataset, and computes
Comparison between the Attacks Besides attack strength, how differently are the attacks perform- ing on random input target models and target points? How do the different internal mechanics of attacks changing the attacker's guess qualitatively? Do the attacks succeed on similar or different samples of member data and target model? Do the attacks have different level confidence on the same input? How far away the attack performance from the most ideal leave-one-on attacks described in Section 3 Answers to these questions require understanding how and why the attacks perform differently, for which do detailed comparisons between attacks as follows.
8

---

Under review as a conference paper at ICLR 2022
Train Acc.
Test Acc.
Áttack
Atfack P
Attack K
Áttack
la
96.2 = 0.031
52.5 = 0.026
0.809 1 0.017
0.822 0.014
0.84 = 0.023
0.876 + 0.009
lb 51.3.00115
35.0 = 0.095
0.628 + 0.035
0.646 + 0.026
0,643 + 0.045
0.652 + 0.045
lla
99.5
1 0.004
65.4 = 0.009
0.752 + 0.008
0.755
0.006
0.799 0.009
0.821 = 0.004
11b
04.1
49.9 = 0.019
0.599 + 0.009
0.609 - 0.009
0.613 = 0.012
0.635 + 0.004
III
100.0
+ 0.0
75.5 = 0.004
0.687 + 0.003
0.687 + 0.003
0.755 + 0.004
0.768 + 0.002
IV
95.74 = 0.01
71.71 0.009
0.647 + 0.004
0.55 + 0.005
0.682 0.009
0.70 0.005
Table 1: AUC Scores of all attacks Purchase 100 Dataset. Configurations la, Ih are trained 2500 data- points, configurations Ia, IIb trained 5000 datapoints, and configurations III, IV trained 10000 datapoints. Configurations
Ilb are trained using L2 regularization with regularization penalty = 0.01. Configuration IV is trained with a gradient clipping norm of 2.0. Configurations la. 1b, Ila, Ilb, and III use 1000 (shadow. reference, distilled) models. whereas configuration IV uses n 30 (shadow, reference, distilled) models.
Similarity of attacks with each other. The scatter plot comparing Attack S and P in Figure 5 roughly linearly centered around the diagonal, with slightly more points in NorthWest then in SouthEast. This shows that Attack S and Attack P almost always guesses membership of train points similarly, while Attack P performs slightly better than Attack S. Meanwhile, Attack D dominates Attack R for correctly guessing membership of train points because their comparison scatter plot is biased towards the North direction.
Gap between attacks to the ground truth. From Table among all attacks, Attack D agrees with the ground truth the most on train points, while the least on test points. This matches our observation that Attack D has larger threshold under given a from Figure T which causes it to guess both more points as members. Besides that, we see that the agreement rate between Attack S and Attack P is as high as 0.94 both on train points and test points, this matches their linear comparison scatter plot in Figure 5 This may be because Attack P and Attack R both reduces one degree of uncertainty for the joint distribution under null hypothesis Ho. as discussed in Section
Closeness of attacks to ideal leave-one-out attack. One interesting observation from Table that, among all the attacks, Attack D agrees with the Attack L the most, with agreement rate 0.98 on train points and 0.82 on test points. We believe this is because Attack D is highly similar in nature with Attack L. by approximating the training dataset of a target model and performing retraining, as discussed in Section
GT
L
S
GT
0.968
0.696
0.662
0.772
0.992
L
0.41
0.692
0.654
0.796
0.968
S
0.656
0.73
0.874
0.592
0.696
P
0.664
0.706
0.948
0.586
0.662
R
0.656
0.722
0.756
0.732
0.78
D
0.372
0.822
0.716
0.708
0.692
Table Agreement rate between ground truth (GT) membership values, and Attacks L, S, S,P.R. R. D for 500 train and 500 test datapoints. The upper triangle of the table corresponds to the agreement rates of train datapoints, whereas the lower triangle corresponds to the agreement rates of test datapoints. The experimental setup is Purchase 100 Configuration Iaa, with 0.3.
5 SUMMARY
We provide a framework for auditing the privacy risk from machine learning models through mem- bership inference attacks. The framework is used to derive attack strategies and also highlight the factors beyond leakage from the models that affect the attack performance. We also empirically analyze the performance of these attack strategies against models trained benchmark datasets.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Michael Backes, Pascal Berrang, Mathias Humbert, and Praveen Manoharan. Membership privacy in microrna-based studies. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 319-330, 2016.
Nicholas Carlini, Chang Liu, Ülfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th {USENIX) Security Symposium ( (USENIX) Security 19), pp. 267-284, 2019.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voor Katherine Lec, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. arXiv preprint arXiv:2012.07805, 2020.
Hongyan Chang and Reza Shokri. On the privacy risks of algorithmic fairness. arXiv preprint arXiv:2011.03731, 2020.
Christopher Choquette-Choo Choo, Florian Tramer, Nicholas Carlini, and Nicolas Papernot. Label-only membership inference attacks, In International Conference on Machine Learning, pp. 1964-1974,
2021.
Ganesh Del Grosso, Georg Pichler, Catuscia Palamidessi, and Pablo Piantanida. Bounding informa- tion leakage in machine learning. arXiv preprint (xXi::2105008//1 2021.
Irit Dinur and Kobbi Nissim. Revealing information while preserving privacy. In Proceedings of the !wenty-second ACM SIGMOD-SIGACT-SSGARR symposium on Principles of database systems, pp. 202-210, 2003.
Jinshuo Dong, Aaron Roth, and Weijie Su. Gaussian differential privacy. arXiv preprint arXiv:1905.02388 2019.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pp. 265-284, 2006.
Cynthia Dwork, Adam Smith, Thomas Steinke, Jonathan Ullman, and Salil Vadhan. Robust trace- ability from trace amounts. Foundations of Computer Science (FOCS), 2015 IEEE 56th Annual Symposium on, pp. 650-669, 2015.
Cynthia Dwork, Adam Smith, Thomas Steinke, and Jonathan Ullman. Exposed! survey of attacks on private data. Annual Review of Statistics and Its Application, pp. 61-84, 2017.
Úlfar Erlingsson, Ilya Mironov, Ananth Raghunathan, and Shuang Song. That which we call private. arXiv preprint orXiv:1908 03566, 2019.
Farhad Farokhi and Mohamed Ali Kaafar. Modelling and quantifying membership information leakage machine learning. arXiv preprint 1006642000.106488 2020.
Vitaly Feldman. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pp. 954-959, 2020.
Jamie Hayes, Luca Melis, George Danezis, and Emiliano De Cristofaro. Logan: Membership in- ference attacks against generative models. Proceedings on Privacy Enhancing Technologies, pp- 133-152, 2019.
Nils Homer, Szaboles Szelinger, Margot Redman, David Duggan, Waibhav Tembe, Jill Muchling, John Pearson, Dietrich A Stephan, Stanley F Nelson, and David W Craig. Resolving individuals contributing trace amounts of dna to highly complex mixtures using high-density snp genotyping microarrays. PLoS genetics, pp. e 10001677, 2008
Hongsheng Hu, Zoran Salcie, Gillian Dobbie, and Xuyun Zhang. Membership inference attacks on machine learning: A survey. arXiv preprint arXiv: /03.0783.. 2021.
Thomas Humphries, Matthew Rafuse, Lindsey Tulloch, Simon Oya, Ian Goldberg, Urs Hengartner, and Florian Kerschbaum. Differentially private learning does not bound membership inference. arXiv preprint arXiv:2010./2112. 2020.
10

---

Under review as a conference paper at ICLR 2022
Matthew Jagielski, Jonathan Ullman, and Alina Oprea. Auditing differentially private machine learning: How private is private sgd? arXiv preprint arXiv:2006.07709, 2020.
Bargav Jayaraman and David Evans. Evaluating differentially private machine learning in practice.
In 28th {USENIX] Security Symposium ({ ({USENIX) Security 19), pp. 1895-1912, 2019.
Peter Kairouz, Sewoong Oh, and Pramod Viswanath. The composition theorem for differential privacy. In International conference machine learning, pp. 1376-1385, 2015.
Klas Leino and Matt Fredrikson. Stolen memories: Leveraging model memorization for calibrated white-box membership inference. In 29th {USENIX) Security Symposium 20), pp. 1605-1622, 2020.
((taaiiimmmmmMEAA ((USENIX) Security
Zheng Li and Yang Zhang. arXiv:2007. 15528, 2020.
Membership leakage in label-only exposures. arXiv preprint
Yunhui Long, Vincent Bindschaedler, Lei Wang, Diyue Bu, Xiaofeng Wang, Haixu Tang, Carl A Gunter, and Kai Chen. Understanding membership inferences on well-generalized learning mod- els. arXiv preprint arXiv: 04889, 2018.
Yunhui Long, Lei Wang, Diyue Bu, Vincent Bindschaedler, Xiaofeng Wang, Haixu Tang, Carl A Gunter, and Kai Chen. A pragmatic approach to membership inferences on machine learning models. In 2020 IEEE Europeun Symposium Security and Privacy (EuroS&P), pp. 521-534. IEEE, 2020.
Mani Malek, Ilya Mironov, Karthik Prasad, Igor Shilov, and Florian Tramèr. Antipodes of label differential privacy: Pate and alibi. arXiv preprint arXiv:2106. 10X.::2006.03488, 2021.
Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. Exploiting unintended feature leakage in collaborative learning. In 2019 IEEE Symposium on Security and Privacy (SP), pp. 691-706, 2019.
Sasi Kumar Murakonda and Reza Shokri. MI privacy meter: Aiding regulatory compliance by quantifying the privacy risks of machine learning. arXiv preprint arXiv:2007.09339, 2020.
Sasi Kumar Murakonda, Reza Shokri, and George Theodorakopoulos. Quantifying the privacy risks of learning high-dimensional graphical models. In International Conference on Artificial Intelligence and Statistics, pp. 2287-2295, 2021.
M. Nasr, R. Shokri, and A. Houmansadr. Comprehensive privacy analysis of deep learning: Pas- sive and active white-box inference attacks against centralized and federated learning. In IEEE Symposium on Security and Privacy (SP), pp. 1022-1036, 2019.
Milad Nasr, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, and Nicholas Carlini. Adver- sary instantiation: Lower bounds for differentially private machine learning. arXiv preprint arXiv:2101.04535 2021.
Boris Polyak and Anatoli Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal control and optimization, 30(4):838 -85555888-85, 1992.
Apostolos Pyrgelis, Carmela Troncoso, and Emiliano De Cristofaro. Knock knock, who's there? membership inference on aggregate location data. arXiv preprint 17X1::7708.06145 2017.
Md Atiqur Rahman, Tanzila Rahman, Robert Laganiere, Noman Mohammed, and Yang Wang. Membership inference attack against differentially private deep learning model. Transactions on Data Privacy, pp. 61-79, 2018.
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Yann Ollivier, and Hervé Jégou. White- box vs blxc Bayes optimal strategies for membership inference. In International Confer- ence on Machine Learning, pp. 5558-5567, 2019.
Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. MI-leaks: Model and data independent membership inference attacks and defenses on machine learning models. arXiv preprint 1806.00 2018.

---

Under review as a conference paper at ICLR 2022
Sriram Sankararaman, Guillaume Obozinski, Michael Jordan, and Eran Halperin. Genomic privacy and limits of individual detection in a pool. pp. 965, 2009.
Muhammad A Shah, Joseph Szurley, Markus Mueller, Athanasios Mouchtaris, and Jasha Droppo. Evaluating the vulnerability of end-to-end automatic speech recognition models to membership inference attacks. Proc. Interspeech 2021, pp. 991-895, 2021.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference at- tacks against machine learning models. In Security and Privacy (SP), 2017 IEEE Symposium on, pp. 3-18, 2017.
Reza Shokri, Martin Strobel, and Yair Zick. On the privacy risks of model explanations. In Pro- ceedings of the 2021 AAAI/ACM Conference AI, Ethics, and Sociery, pp. 231-2411 2211, 2021.
Congzheng Song and Ananth Raghunathan. Information leakage in embedding models. In Pro- ceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security, pp- 377-390, 2020.
Congzheng Song and Vitaly Shmatikov. Auditing data provenance in text- nett-gnneration models. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 196-206, 2019.
Congzheng Song, Thomas Ristenpart, and Vitaly Shmatikov. Machine learning models that remem- ber much. In Proceedings of the 2017 ACM SIGSAC Conference computer and communi- cations security, 587-601, 2017.
Liwei Song and Prateek Mittal. Systematic evaluation of privacy risks of machine learning models. In 30th {USENIX} Security Symposium (UUSENIX) Security 21), 2021.
Liwei Song, Reza Shokri, and Prateek Mittal. Privacy risks of securing machine learning models against adversarial examples. Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pp. 241-257, 2019.
Michael Veale, Reuben Binns, and Lilian Edwards. Algorithms that remember: model inversion attacks and data protection law. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, pp. 20180083, 2018.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pp. 6888.. Citeseer, 2011.
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learn- ing: Analyzing the connection to overfitting. In 2018 IEEE 3 31 Computer Security Foundations Symposium (CSF), pp.
2018.
Yang Zou, Zhikun Zhang, Michael Backes, and Yang Zhang. Privacy analysis of deep learning in the wild: Membership inference attacks against transfer learning. arXiv preprint arXiv:2009.04877 2020.
A APPENDIX
A.1 DETAILED DERIVATION OF APPROXIMATED LRT FOR MEMBERSHIP INFERENCE
We first prove two useful approximation inequalities about the posterior distribution P(O as follows.
1. For arbitrary data point = (Zay Zy), and arbitrary dataset D, we have
P(O U2) 2 ((o.5g.*y) P((
(39)
12

---

Under review as a conference paper at ICLR 2022
2. Let 21.22, 22, 2n be i..d. samples from the data distribution (2).. Then when n is large enough, for any model parameter 8, we have
E ~ (1 ,2)) s - [P(0]21, -1 2n-1)
(40)
We now offer details for deriving the approximated likelihood ratio test (LRT) for membership inference.
Definition A.1 (Approximated LRT for membership inference) Let (0, random samples from the joint distribution of target model and target data point, specified by one of the following membership hypotheses.
n w.i.d miidssampeewwxt)
D pop. 0 ~ T(D), samplentT(s)
Ho :D &
D pop
(41)
Hy: :D midddsampieswwta) D Prop: T(D), sample D
(42)
The likelihoods function of hypothesis Ho and H1 given observed target model O and target data point z as follows.
2(Ho0,, (0,22) T2)) ED [P{
(43)
2(H1 = (0, Σ (D,00) = Σ T(2) PH. (112)- P(ODD)
(44)
D
= =(2) - 1P U2]
(45)
(By equation
39
2 x(2) )E( E E ~ - [P(0DD')
(46)
(By equation 40
π(2) - (100)20))))) E [P(ODD)
(47)
Therefore the LRT statistics is
X(0 =) 2(Hol0,2) e ((() I(H
En- [P( x ((0,22)))))
(48)
E (P D*)
The LRT hypothesis test rejects Ho when the LRT statistic is small. By equation o the rejection region {(0, z): A(0, 2) S e} can be approximated as follows.
(0, E(0, 2x, 2x) S -..--oog. loge
(49)
Datasets
The Purchase 100 dataset is based on Kaggle's "Acquire Valued Shoppers Challenge" that contains shopping histories for thousands of individual We use simplified preprocessed purchase dataset provided in Shokri et al. (2017). There are 197, 324 data points in this dataset, where each data point has 600 binary features. The data points are clustered into 100 classes to represent different shopping style.
The CIFARIO and CIFARI00 datasets are widely used benchmark datasets for image classification. The CIFAR10 10 dataset consists of 60, 000 data points in 10 classes, whereas the CIFARIO0 dataset consists of 60, 000 data points in 100 classes. Here each data point is 32 32 color image, and there are 50, 000 training images and 10, 000 test images in total.
The MNIST dataset is a widely used benchmark dataset for image classification. The MNIST dataset consists of 70, 000 data points in 10 classes, where each data point is 28 x 28 handwritten digit image. In total, there are 60, 000 training images and 10, 000 test images.
Experiment setup
For each configuration, we train up to n = 1000 models on random IID splits of the data. For the Purchase configurations, we use 4 layer MLP with layer units [512, 256,
13

---

Under review as a conference paper at ICLR 2022
2 layer CNN with filters = [32, 64] and max pooling. For CIFARIO configurations, we use AlexNet and a 3-block VGGNet. For all target models and datasets, and categorical crossentropy loss. For Configuration IV of Purchase 100, we use gradient clipping while training the models. For CIFARIO, we add momentum while training the models.
128, 64]. For CIFARIO0 and MNIST configurations,
SGD the optimization function
2. Target models: We use the first 10 models as target models on each of the Purchase 100, CIFARIO, CIFAR 100 and MNIST datasets. The performance Attacks S, and R are evaluated all 10 models, whereas Attack D is evaluated on the first 3 models.
3. Shadow models for Attack S and Reference models for Attack R: Given target model, use the remaining (n - = 999 models as shadow models or reference models for Attack S and Attack respectively.
4. Distilled models for Attack D: Given a target model, we use the same random IID splits used to train 1000 models to distill (n 1) 999 models using the soft labels from the target model.
5. Using < 1000 models: For Configuration IV of Purchase 100 and Configuration III of CIFAR10, we report the attack performance results using 30 shadow, reference, and distilled models. For Configurations I and II of CIFARIO we report the attack performance results using n 400 shadow, reference, and distilled models.
A.2 ADDITIONAL EMPIRICAL EVALUATION RESULTS
In this section, we provide a detailed analysis of the internal workings of the attack strategies. We specifically compare how the thresholds on loss values derived from these strategies differ various target points and target models.
Illustration of Attack threshold How differently does the attack threshold in Attack S, P. R, D depend on the target model and the target data point How is the different dependence of thresh- old on 0 and in different attacks affecting their success? To investigate these important questions, we plot the loss histograms that different attacks use to compute the thresholds, on different input 01, O2 and æ1, 22 in Figure [ The loss histogram approximates the LRT statistics distribution under null hypothesis Ho in equation [ with different uncertainty for different attacks, as discussed in We that model-dependence and sample-dependens of attack threshold reduces the uncertainty in the loss histogram. The attacks (Attack D) with more concentrated loss histogram are more likely
succeed.
Train Acc.
Test Acc.
Attack S
Attack P
Attack R
Attack D
96.2 + 0.046
40.9 0.029
0.870 0.018
0.857 0.023
0.874 0.018
0.871 = 0.007
97.8 + 0.012
45.9 0.010
0.860 0.014
0.868 0.008
0.858
0.019
0.889 = 0.011
III
97.4 1: 0.004
68.2 1 0.011
0.706 = 0.011
0.708 1 0.009
0.737 1 0.014
0.742 = 0.003
Table 3: AUC Scores of attacks on CIFARIO Datasct. Configuration I trained on 2500 datapoints, and configuration II is trained on 5000 datapoints. Configurations and II are trained using AlexNet. Configuration III is trained on 10000 datapoints using a 3-block VGGNet. Here report the results of the attacks for Configurations I and using 400 (shadow, distilled, reference) models, and 30 for Configuration III. All models have been trained using the SGD optimizer with momentum, an L2 regularization penalty 0.001.
B RELATED WORK
The current work in this domain can be broadly grouped into three categories: 1) Empirical works for improving the existing attack strategies or adapting them to different settings and models 2) Theoretically/emprriaalll analyzing the privacy risk in various systems using the existing attack strategies 3) Exploring the connections with differential privacy and using them to establish lower bounds on leakage and/or select privacy parameters. Below, we provide brief summary of works in all the three categories.
14

---

Under review as a conference paper at ICLR 2022
Loos of Poins
Loss of Poins
Attook
Attack
Attack
0.08
0.08
Loss Distribution
Loss Distribution Attack
0.06
0.06
Feactton
0.03
0.03
0.00
0.00
Valae
Attark of hoint
Attock D
Alleck D
0.08
0.08
Loss Distribtion Attack
Loss Distributon
Attack
0.06
0.06
0.05
0.05
0.02
0.03
0.02
0.00
0.00
Value
(21. 02)
(22, 02)
Figure 2: Loss distributions used by Attack S, P. R, D for points 21, 22 and target models A,, 02 in Purchase Configuration Πa,, class and are members of both the target models' training datasets. We show the loss distributions for 21, 22 on target model 0₃ in the main paper, and on 0g here.
0.1. Note that both 21 and z2 datapoints from the same
Train Acc.
Test Acc.
Attack S
Attack P
Attack R
Attack D
97.4 1 0.013
14.9 0.009
0.959 1 0.005
0.960
0.004
0.964 1 0.006
0.957 1 0.0003
89.8 0.035
13.9 = 0.011
0.937 + 0.007
0.941
0.004
0.938 = 0.012
0.941 + 0.004
laa
97.9 1 0.006
20.4 1 0.006
0.944 1 0.004
0.945 1 0.003
0.945 1 0.006
0.936 1 0.0
Ilb
87.9 0.020
19.63 0.007
0.905 + 0.007
0.906 0.007
0.904 0.008
0.893 0.002
Table 4: AUC Scores of all attacks on CIFAR100 Dataset. Configurations Ia, lb are trained on 2500 data- points, and configurations Ia,, Ilb trained 5000 datapoints. Configurations Ib, Ilb trained using L2
regularization with regularization penalty À 0.005.
Train Acc.
Test Acc.
Attack S
Attack P
Attack
Attack D
97.9 0.004
95.8 0.007
0.50 = 0.004
0.50 + 0.005
0,557 0.009
0.549 0.006
94.2 0.005
93.4 = 0.006
0.497
0.005
0.496 + 0.004
0.544 0.014
0.522 0.005
Ma
98.6 + 0.001
97.1 + 0.002
0.496
0.005
0.496 + 0.006
0,551 = 0.011
0.544 + 0.004
11b
94.6 0.003
94.5 = 0.003
0.491
0.005
0.49 0.005
0.52 + 0.011
0.497
0.004
Table 5: AUC Scores of all attacks on MNIST Dataset. Configurations Ia, are trained on 2500 datapoints, and configurations Il, IIb are trained on 5000 datapoints. Configurations Ib, Ilb are trained using 1.2 regularization with regularization penalty À 0.05.
15

---

Under review as a conference paper at ICLR 2022
1.0
1.0
0.8
0.8
TPR
0.6
TPc
0.4
Attack S. AUC 0.809
Attack S, AUC = 0.752
Attack P. PUAUC 0.822
Attack AUC 0.755
Attack AUC 0.839
Attack AUC 0.799
Attack D, AUC 0.576
Attack D AUC 0.821
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
FPR
Purchase100 (la)
Purchase (Ila)
Figure 3: FPR TPR with AUC scores for all attacks on Configurations Ia and Ila of the Pur- chase100 100 dataset. Note that models in Configuration la have been trained using 2500 points, and models in Configuration Ila have been trained using 5000 points.
1.0
0.8
0.8
0.6
TPP
0.6
0.4
0.4
0.648
Attack P,AUC 0.708
Attack P. AUC 0.655
0.2
Attack AUC 0.737
0.2-
Attack R. AUC 0.682
Attack AUC 0.742
Attack D. AUC 0.7
0.0
0.0
0.0
1.0
0.6
0.8
FPR.
CIFARIO (III)
Purchase 100 (IV)
Figure 4: FPR vs TPR with AUC scores for all attacks on Configuration III of the CIFARIO dataset and Configuration IV of the Purchase 100 dataset. Note that models in Configuration III of CIFAR10 use the 3-block VGGNet architecture. Models in Configuration IV of Purchase 100 have been trained using gradient clipping with an 1.2 clipping norm of 2.0. All attacks here use n = 30 (shadow, reference, distilled) models.
Empirical attack strategies for membership inference: Shokri et al. (2017) demonstrated the vulnerability of machine learning models to membership inference attacks in the black-box setting, where the adversary has only a query access to the target model. The attack algorithm is based on the concept of shadow models, which are models trained on some attacker dataset that is similar to that of the training data. Membership inference is modeled as a binary classification task for an attack model that is trained the predictions of shadow models the attacker dataset. A huge litera- ture followed this work extending the attacks to different setting like white box access (Nasr et al. 2019 Sablayrolles al. 2019: Leino & Fredrikson 2020), label-only access (Li & Zhang 2020 Choquette-Choo et al. 2021). federated learning (Melis et al. 2019). transfer learning (Zou et al. 2020) and different types of data, models such as aggregate location data (Pyrgelis al.) 2017).
generative models (Hayes et al.] 2019), language models (Carlini et al.] 2019;[Song & Shmatikov 2019 Carlini al. 2020), sentence embeddings (Song & Raghunathan, 2020). and speech recogni- tion models (Shah et al. (02T). Multiple works have looked at improving the attack methodology through a more fine grained analysis or by reducing the background knowledge and the compute power required to execute the attack (Long et al. 2018; Song & Mittal 2021 Salem et al. 2018) See (Hu et al. 2021) for a more comprehensive list of membership inference attacks against ma- chine learning models. All these works follow the same attack framework for membership inference but they either exploit a slightly different signal that is correlated with membership a point in the training set or find efficient way exploit the already known signals.
16

---

Under review as a conference paper at ICLR 2022
1.00
0.25
Adbock
Ntack
0.00
-0.25
-0.50
33,9156
5.57%
12.31%
0.63%
-1.00
-0.75 -0.50
-0.25
0.00
0.25
0.50
0.75
1.00
Attack
Attock R
(a)
(b)
Figure 5: Scatterplots comparing Attacks S and P. and Attacks R and D in Purchase 100 Configura- tion Ia.. Each dot on scatterplot corresponds a particular train datapoint. The two coordinates of the dot are for the two attacks being compared. Each coordinate of the dot corresponds to the differ- ence between the loss of the point on the target model and the loss threshold used by the particular attack on the point. Plot (a) compares Attacks S and P. whereas plot (b) compares Attacks R and D.
Privacy risk analysis with membership inference attacks: Homer et al. (2008) performed the first membership inference attack genome data to identify the presence of an individual's genome in mixture of genomes. (aankararaaa et al. (2009) provided a formal analysis of this risk of de- tecting the presence of an individual from aggregate statistics computed on independent and binary attributes. Murakonda et al. (202T) extended this analysis to the case of releasing Discrete Bayesian networks learned from data with dependent attributes. |Backes al. (2016) perform an analysis sim- ilar to that of (Sankararaman et al., 2009) but for MicroRNA data (aggregate statistics computed on independent and continuous attributes). [work et al. (2155) provide a more extensive analysis when the released statistics are noisy and the attacker has only one reference sample to perform the attack. The key results of all these works establish the privacy risk of releasing aggregate statistics by quantifying the success of membership inference attacks a function of the number of statistics released and the number of individuals in the dataset. Similar attempts were made to analyze the privacy risk of machine learning models through membership inference via the lens of mutual in- formation (Farokhi & Kaafar (2020) and generalization error (Yeom et al. 2018 Del Grosso et al.) 2021]. Beyond these theoretical analyses, membership inference attacks are also used to empirically study the trade-offs between privacy and other desirable characteristies for machine learning models such as fairness (Chang Shokri 2020), robustness to adversarial examples (Song et 2019 and providing explanations (Shokri et al. 2021).
Differential privacy and Membership inference: The definitions of differential privacy (Dwork
et al.] 2006) and membership inference (Homer et al. 2008 Dwork et al. [2015] Shokri et al. (017)) very closely connected and the hypothesis testing interpretation of differential privacy provides clear view of the relationship between them. Satisfying differential privacy is equivalent to imposing a bound on the ability distinguish any two neighboring datasets that differ by the presence of one individual i.c., inferring about the presence/absence of the individual. The bound is stated as trade- off between the type-I and type-Il errors in distinguishing the two neighboring datasets. Hypothesis testing interpretation of differential privacy is very useful in deriving tight compositions (Kairouz) ct al. 2015) and has even motivated new relaxed notion of differential privacy called f-DP (Dong al. 2019). By definition, differentially private algorithms bound the success of membership infer- ence attacks. Multiple works (Yeom et al., 2018 Erlingsson et al. 2019 Humphries et al. 2020), each improving the previous work, have provided upper bounds the success of membership inference attacks as a function of the parameters in differential privacy. Jayaraman & Evans] (2019)
evaluated the performance of membership inference attacks machine learning models trained
17

---

Under review as a conference paper at ICLR 2022
with different values of epsilon under different relaxed notions of differential privacy. Rahmaan al. (0188) also use membership inference attacks to measure the privacy loss on models trained with differentially private algorithms. The empirical performance of membership inference attacks has also been used to provide lower bounds on the privacy guarantees achieved by various differentially private algorithms (Jagielski et al., 2020 Nasr al. 202T Malek al. 202T). The key difference between the empirical analysis of membership inference in the previous three works and other works that they simulate the exact adversary differential privacy i.e., they train multiple models with and without one particular training point and keep the rest of training set fixed. The performance of the attack (type and type errors) is averaged over these models, whereas the previous works the model is fixed and the performance is computed an average over points in the training and test sets. This simulation of the DP adversary helps in removing the effects of other points the dataset when measuring the leakage through the model about particular point of interest.
18


Reviews:
[{"review_id": "lqDQYaN0TRv", "reviewer": "Reviewer_Sgo6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "** Strengths ** \n- The investigated problem is relevant and the paper is well written;\n- The notation is convient enough and all important concepts are well introduced; \n- The central idea remains interesting but it is not novel enough. \n\n** Weaknesses **\n- The contributions are not clearly stated in the introduction section. Therefore, it appears difficult to identify which are the main contributions in this paper. According the introduction section, the main contribution is the proposed framework via the connection with binary hypothesis testing (i.e., the tradeoff between Type I and Type II errors) seem to be the main contribution.  \n- The main misconception underlaying this work is the lack of novelty. The proposed framework via the connection with binary hypothesis testing is not novel and has been already proposed in a series of works (e.g., among others, see https://arxiv.org/abs/2105.03875). \n- All results in the paper rely strongly on the assumption introduced in eq. (1) which is indeed to characterize the outcome of the algorithm obtained via SGD training. However, this has not been properly justified to be considered as being strong enough to validate a possible privacy audit of ML  algorithms. The authors do not provide any valide proof or formal justification for this assumption. \n- Another central problem is the lack of numerical results and in particular, comparison with competing methods for building attacks in the literature. In addition the tradeoffs between Type I and Type II error are not always reported. On the other hand, there is very little data sets and relevant architectures to validate the results. \n", "summary_of_the_paper": "The focus of this paper is on membership inference attacks. In particular, the paper aims at providing a framework that can help to access how different factors beyond information leakage from the model affect the performance of membership inference attacks, and how to design attacks that cancel out the effects off other factors. More specifically, a framework for understanding the relationship between, success of membership inference attacks and information leakage is introduced. Experiments using well-known datasets on visual tasks are provided to evaluate the performance  of the proposed attacks(in terms of AUC scores. Various frameworks are discussed in Section 3.   ", "summary_of_the_review": "Given the above comments, I believe that the paper lacks of technical novelty, numerical results and relevant comparisons with several existent methods in the literature. To conclude, I believe that the work contains some valuable ideas which deserve to be further investigated (both theoretically and practically)  but in its current state, the contributions and the presentation of the results are too short in terms of exceptions to be accepted to be published in ICLR conference. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hmQIK_jZQR", "reviewer": "Reviewer_BGkr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\nThe hypothesis testing framework is well-motivated and the formulation is solid which also covers prior work such as shadow model MIAs. The attacks based on distillation also seem noval and not discussed before. The authors also showed that the newly proposed attacks are better than shadow model MIAs. \n\nWeaknesses and questions:\n1. Many of the attacks could still be inefficient for auditing purposes as one might need to train 1000 reference models to perform the attacks. The numbers also are only high for models that are extremely overfitting (which is also true for previous work). How could we audit models that are well-generalized?\n2. The comparison and difference between this work and prior work is vaguely discussed. No in detailed comparison with [1], which also formalized MIA with indistinguishability based framework. The hypothesis testing framework in this work is very similar to the threshold adversary in [1]. I hope authors could discuss the differences in detail. No comparison to white-box models (e.g. [2]). How would this compare to the proposed framework?\nMinor: related works should be discussed in the main text instead of supplementary materials.\n3. No connection made between this work and differential privacy, although the indistinguishability setup is highly related. It would be nice to see how the framework translates to privacy loss in DP.\n4. Minor: column names overflowed in Table 4.\n\nReferences\n\n[1] Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting. Samuel Yeom, Irene Giacomelli, Matt Fredrikson, Somesh Jha\n\n[2]  Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning. Milad Nasr, Reza Shokri, Amir Houmansadr\n", "summary_of_the_paper": "This paper proposed a hypothesis testing framework for membership inference attacks (MIAs). The framework involves different algorithms for finding the loss threshold for MIAs, i.e. if a target data point with loss less than the threshold value then it will be inferred as a member of training data. The notation of membership privacy loss is the same as in differential privacy and the proposed algorithms improves the utility of the attacks compared to prior works on benchmark datasets.\n", "summary_of_the_review": "This paper is well-motivated and the proposed framework could potentially be impactful. However, there are many empirical limitations on how to apply the attacks for auditing state of the art models. Furthermore, the evaluation and comparison of baselines methods are not thorough. I therefore recommend a weak reject.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3VgDGf895vl", "reviewer": "Reviewer_XT1N", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Please list both the strengths and weaknesses of the paper. When discussing weaknesses, please provide concrete, actionable feedback on the paper.\n\nOn the positive side, the paper aims to make “auditing privacy” more modular. This goal is pursued by proposing multiple different attacks for membership inference. Then, many different factors, beyond just the success rate of the attacks are compared and reported with experiments.\n\nOn the weakness side: it is not really clear how having multiple attacks of specific form can give a full picture about whether or not an algorithm is private. There is of course the possibility that none of these attacks work on a learning algorithm and another attack works. That is why the only way to really “audit” privacy is to prove it. Alternatively, one might envision being able to prove that some class of attacks are “complete” in some sense, but I doubt that we are there yet, and certainly this paper does not provide any justification as to why their set of attacks are “complete”. In general, the paper’s plans are cryptic and each page has many ambiguous sentences with unclear goals. That is also the case for the introduction. The paper also does not justify clearly why they pick certain criteria and focus on them. For example, the bullets in page 3 are not clearly motivated/explained.\n\nOther comments:\n\nThere are quite a few typos, plz make a pass. Example “confidentiality confidentiality” “explaining about the uncertainties” etc.\n\nThe introduction says “\"Below, we first briefly review the related work in this domain, explain how different works measure privacy risk, and then…”\nBut then the paper moves on and explains the contributions of our work. I guess that is maybe due to the change of the paper for the submission.\n\nThe paper says “The notion of privacy underlying our framework is primarily based on differential privacy,  I would say computational DP”\nNote that your definition is rather an “average case, computational, indistinguishability-based” variation of DP. That is because: you pick the data set and z at random, you work with efficient (poly time) attacks and your goal is just to distinguish 2 cases. Standard DP asks for more conditions in all three categories.\n\nThe first attack writes probabilities P(theta|D) as if we are aware of the learning algorithm. Can you please be more clear that you are basically *assuming* P(theta|D) to be what you wrote in Equation 1? In other words, you are assuming to work with a specific learner, because you sate the distribution of theta given D, no?\n\nPage 3: “The population data used for constructing the attack algorithm, and evaluating the inference game, need to be similar, in distribution, to the training data”\nI don’t get this (and many similar firm judgements). Why? I agree that natural algorithms might fall into this category, but simply stating it as a general rule needs proof (and I don’t think this is true in general).\n\n“By violating this principle, we might overestimate the privacy loss”\nI think this is wrong. The whole point of DP is that the adversary might have *arbitrary* auxiliary information. So, I see no reason why having a particular auxiliary information, and winning the security game, can be a misleading indication that the scheme was not DP; quite the contrary.\n\nAs another example of sentences that I found unclear:\n“The adversary knows the underlying data distribution\nWhat does that mean? Can I sample from it?\n\nMaybe I am missing something here, but I don't think pi(z) should show up in Equation (4). Is not z independent of \\theta and D in that case?\n\nIn your experiments of the main body: you say the data set (Purchase100) but what is the learner?\n", "summary_of_the_paper": "Membership inference attacks are attacks that infer whether a given record is in the dataset of a given machine learning model or not. By definition, (the success of) such attacks is in contrast with differential privacy.\n\nThe paper proposes multiple attacks to infer membership in the data sets of machine learning models. The envisioned goal of the paper is that these attacks can provide a more complete picture about how private a machine learning algorithm is, when compared with other membership inference attacks. Hence, the ultimate goal is to provide “reports” that go beyond “just a number” to better audit the privacy of learning algorithms.\n", "summary_of_the_review": "\nI agree with the general sentiment of the paper, that having a framework for auditing privacy could be helpful. But such efforts need a much more detailed and justified approach, arguing for “complete” attacks that at least “capture known attack techniques so far”. Otherwise, just focusing on 5 attacks might be dangerous as that might suggest that schemes are private while they are not. I also think that the paper needs to be much more clear in its criteria that it proposes (in addition to the success rate of the attacks) to be part of the “report” on privacy.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work provides a formal framework for discussing membership inference attacks (MIA). It then examines existing attacks and proposes some new ones. The attacks are evaluated on several datasets.  The framework mostly formalizes the types of information and error types that an attack may use and is presented as the main contribution of this work. However the presented formalizations do not appear to contribute significantly beyond the existing work on MIAs. The new attacks may be of interest and, according to the presented experiments, (mildly) improve on some of the existing MIAs. At the same time, as presented, the discussion of the the benefits of the new attacks is relatively short and reviewers did not find the results to be sufficiently convincing. Therefore I cannot recommend acceptance for this work in its current form.
Decision: Reject