Mace: A flexible framework for membership privacy estimation in generative models
Authors: Yixi Xu
Sumit Mukherjee
Xiyang Liu
Shruti Tople
Rahul M Dodhia
Juan M Lavista Ferres
Venue: Accepted by TMLR ()

Abstract: Generative machine learning models are being increasingly viewed as a way to share sensitive data between institutions. While there has been work on developing differentially private generative modeling approaches, these approaches generally lead to sub-par sample quality, limiting their use in real world applications. Another line of work has focused on developing generative models which lead to higher quality samples but currently lack any formal privacy guarantees. In this work, we propose the first formal framework for membership privacy estimation in generative models.  We formulate the membership privacy risk as a statistical divergence between training samples and hold-out samples, and propose sample-based methods to estimate this divergence. Compared to previous works, our framework makes more realistic and flexible assumptions. First, we offer a generalizable metric as an alternative to the accuracy metric (Yeom et al., 2018; Hayes et al., 2019) especially for imbalanced datasets. Second, we loosen the assumption of having full access to the underlying distribution from previous studies (Yeom et al., 2018; Jayaraman et al., 2020), and propose sample-based estimations with theoretical guarantees. Third, along with the population-level membership privacy risk estimation via the optimal membership advantage, we offer the individual-level estimation via the individual privacy risk. Fourth, our framework allows adversaries to access the trained model via a customized query, while prior works require specific attributes (Hayes et al., 2019; Chen et al., 2019; Hilprecht et al., 2019).

Published in Transactions on Machine Learning Research (10/2022)
Mace: A flexible framework for membership privacy estimation in generative models
Yixi Xu" Microsoft
yimm@microsoft.com
Sumit Mukherjee" Insitro
sumammasherjee22@@maal.com
Xiyang Liu University of Washington
sigengg@@cs..usshhggtoo...t
Shruti Tople Microsoft Research
shruut.toglae@micrrssff.ccmm
Rahul Dodhia Microsoft
vabul.doltia@@miccroof.tomm
Juan Lavista Ferres Microsoft
jlaiisa@@microooft.com
Reviewed on OpenReview:
https: h/:p:/ooppee openreview. net/ forum? ZamOkNNe3u7
Abstract
Generative machine learning models are being increasingly viewed as a way to share sensitive data between institutions. While there has been work on developing differentially private generative modeling approaches, these approaches generally lead to sub-par sample quality, limiting their use in real world applications. Another line of work has focused on developing generative models which lead to higher quality samples but currently lack any formal privacy guarantees. In this work, we propose the first formal framework for membership privacy estimation in generative models. We formulate the membership privacy risk as a statistical divergence between training samples and hold-out samples, and propose sample- based methods to estimate this divergence. Compared to previous works, our framework makes more realistic and flexible assumptions. First, we offer generalizable metric as an alternative to the accuracy metrie (Yeom al. 2018 Hayes et al. 2019) especially for imbalanced datasets. Second, loosen the assumption of having full access to the underlying distribution from previous studies (Yeom el al. 2018 Jayaraman el al. 2020), and propose sample-base based estimations with theoretical guarantees.
Third, along with the population-level membership privacy risk estimation via the optimal membership advantage, we offer the individual-level estimation via the individual privacy risk. Fourth, our framework allows advessriee to access the trained model via a customized query, while prior works require specific attributes (Hayes al. 2019: Chen al. 2019 Hilprecht al. 2019).
1 Introduction
The past. decade has seen much progress in machine learning, largely due to the rapid growth in the number of large-scale datasets. However, concerns about the privacy of individuals being represented in datasets have led to a variety of regulations that made it increasingly difficult to share sensitive data across institutions
"equll contribution

---

Published in Transactions on Machine Learning Research (10/2022)
especially in healthcare (Viggt & Von dem Bussche 2017). Recent progress in the area of generative machine learning has made it possible to share synthetic data, which reflects the statistical properties of the original datasets Seeorges-Filteau & Cirillo 2020). Such synthetic data has been shown to allow for the development of downstream machine learning models with limited loss of performance compared to the original data (Rajolte et al. )021). This has led to synthetic data sharing being increasingly viewed as a potentially privacy preserving alternative to sharing the raw data (Tom et al., 2020) between institutions.
Despite the appeal of using synthetic data generated by generative models an alternative to traditional data sharing, recent work has shown common generative modeling approaches are often vulnerable to a variety of privacy attacks (Hilprecht et al. 2019: Hayes et al.] 2019 Chen ct al., 2019). This led to the development of differentially private generative modeling approaches which allow for the generation of synthetic data while 2018). However, in the case of high dimensional datasets (such as images), such approaches have been shown to produce synthetic samples of very poor quality for any reasonable level of guaranteed privacy (Xic et al.) 2018: Mukherjee et al., 2019)) For example, Mukherjee et al. (2019) generated 50,000 synthetic CIFAR1O samples using the differentially private GAN (Xie et al. 2018 with a large privacy budget = 100. Then, classifier was trained the synthetic data, however the accuracy was below 20% when validated on the test set. It was initially assumed that the poor performance was the result of loose privacy accounting (leading to an overestimation of E), but this assumption is recently challenged by a work indicating that moments accountant-base approaches lead to tight estimates of e (Nasr et al.) (021). Thus greatly reducing the possibility of developing a differentially private generative modeling approach that could generate samples good enough to train a strong ML model in practical high-dimensional data settings.
providing strong formal privacy guarantees (Xie et al. 2018 Jordon et al.
More recent work has focused on developing novel generative learning methods that have been empirically shown to be protected against certain types of privacy attacks, such as membership inference attacks (Mukherjee et al., 2019 Chen et al. 2021). Despite these advancements in empirically improving the privacy of generative models in a few scttings, there is currently no approach to provide formal privacy certificates for such models. This in turn has limited the usability of such models in real-world applications where the complete lack of formal certificates would pose a problem with regulators. A promising line of related work has been in using membership inference attacks to audit the privacy of trained machine learning models (primarily discriminative models) with theoretical justifications (Yeom et al., 2018 Jayaraman et al. 2020). More specifically, these works estimate the membership privacy risk of model against a specific adversary. As a result, it would be computationally expensive to estimate the maximum risk when there is a large group of adversarics, or even impossible given an infinite set of adversaries. As a comparison, MACE is able to estimate the maximum privacy risk via the Bayes optimal classifier. Another limitation is that these frameworks assume a full access to the underlying data distribution. For example, Yeom et al. (2018) used a dataset including 4819 patients who were prescribed warfarin, collected by the International Warfarin Pharmacogenntics Consortium to demonstrate their methodology. However, the method could hardly generalize if the population of interest includes all the patients who were prescribed warfarin instead of the specific 4819 patients. While the above case is more common in practice, it is usually not feasible to get full access to this kind of sensitive data. Thus, existing methods (Yeom et al. 2018: Jayaraman et al.) 2020) are not applicable, and this leaves a gap between theory and practice. To overcome this restriction, MACE allows for not only a full access but also a limited access to the data via a simple random sample. Furthermore, MACE is able provide consistent estinators of membership privacy risks at both individual and population level. This allows us to estimate the membership privacy risks of different subgroups, which is usually different as shown by Feldman (2020) on long-tailed distributions. Last but not least, much of this line of work has focused on auditing differentially private discriminative models. As comparison, we focus on trained generative models, which may or may not be differentially private.
To motivate our paper, let us look at a real-world application scenario. A clinical research institution wants to publicly release a medical imaging dataset to enable machine learning model development using the data. However, due to concerns about the personal health information (PHT) in the datasel, they look into synthetic data generation. Having identified a viable generative modeling approach, the institution is faced with few questions prior to dala/ mola/model release: i) should they release the synthetic dalasel or the trained generative model?, ti) if they just release synthetic data, does it matter how many synthetic samples they release?, iii)
2

---

Published in Transactions on Machine Learning Research (10/2022)
how vulnerable is the synthetic data or the trained model to membership inference attacks. Currently, there is no answer to these questions, unless making strong assumption of the adversaries i.e. only considering a few specific heuristic membership inference attacks (Hayes et al. 2019 Hilprecht et al. 2019). which is not realistic in practice.
In this paper, we begin to answer these questions through the development flexible statistical framework to measure the membership privacy risk in generative models (MACE: Membership privACy Estimation). Our framework is built the formulation of the membership privacy risk (given query access) as statistical divergence between the distribution of training-set and non-training-set samples. We show the utility of our framework using many SOTA queries from the literature and some new ones against common computer vision as well as medical imaging datasets.
Our primary contributions are as follows:
We develop framework to estimate the maximum membership privacy risk against adversarics that have query access to the model. Our framework can not only estimate membership privacy risks that are defined as the accuracy of the membership inference attack as in (Yeom et al. 2018 but also those from a more general risk class (Koyejo et al., 2014j. This gives the users ficxibility to measure the ability of a membership inference attack to distinguish members from non non-members from different angles, especially when the training set is a small part of the total available dataset. In addition, MACE is capable of estimating the membership privacy risk given any scalar or vector valued attributes from learned model, while prior works (Hayes et al. 2019: Chen et al. 2019
Hilprecht et al. 2019) restrict to a set of specific attributes.
Our framework is able to measure the individual-level membership privacy risk. This measures the risk of each individual sample against specific modes of membership inference attacks, allowing users to identify those high risk individuals and decide whether to exclude high risk samples and re-train their model.
We loosen the assumption of having full access to the training and non-training set from previous studies (Yeom et al. 2018 Jayaraman et al. 2020), and extend to the case where only a simple random sample is feasible. Furthermore, we derive consistent estimators for both the maximum membership privacy risk and the individuul-level membership privacy risk with theoretical justifications.
We demonstrate the usability of MACE by experiments that analyze the membership privacy risks with regards to various query types and generative model architectures three real-world datasets via the membership advantage and the individual privacy risk under both the accuracy-based and generalized metrics.
2 Background
In this section, we present a brief background on query functions, membership inference attacks, attack experiments, and the Bayes optimal classifier. Then we briefly discuss the limitations of current membership inference approaches. We assumne readers already possess a general understanding of generative models and Differential Privacy but provide a short background section in the Appendix for the sake of completeness.
2.1 Notation
We introduce notations that will be used in the rest of the paper.
Let = = (x. E 2 X x y be a data point from a data distribution D. Note that is some extra information for generative models such as conditional GANs. In other words, = x X for normal generative models.
We assume an adversary A would have query access to a model via a query function Qs(+) : X' y R1, where q is an integer. Examples of adversaries and query functions are given in Sections 2.3 and
2.2 respectively.
3

---

Published in Transactions on Machine Learning Research (10/2022)
Let ~~ D be ordered list of n points. It is referred to training set, sampled from D. We will assume the training set S to be fixed in this paper.
~~ S denotes uniformly sampling from a training set S. Also, ~ D denotes uniformly sampling from the data distribution not including the training set S, which is referred to as sampling from a hold-out sel.
For set of samples {z1, 22, -.. , .N), we define their associated membership labels
{m1, [m1,m2,"" M2,"" ,,y, where m, = 2; is in the training set and m; = -1 otherwise for i = ==
For a given condition C, let I(C) = the condition C holds, otherwise 0.
2.2 Query functions
In this subsection, we introduce some representative query functions Qs(2) considered in or motivated by prior work. Following Chen al. (2019); Hilprecht et al. (2019); Hayes et al. (2019), we divide our attack
settings based on the accessibility of model components: (1) access only to generated synthetic data and (2)
access to models.
2.2.1 Accessible synthetic datasets
In the common practice of synthetic data releasing, researchers or data providers may consider releasing only generated datasets or just the generator. However, prior works (Chen et al. 2019 Hilprecht et al., 2019) have shown releasing generator/syuthett datasets can cause privacy leakage. Specifically, for the case where generative model released, Chen al. (2019) consider the following query function Qs(2) min L2(2, G(w)), where G is the generator released the public. Alternatively, Hilprecht et al. (2019) first generate large synthetic datasct n1, 12.22... -Nn using the generator and then the following query function:
Qs(2) - Σ T(g U+(2)) log d (gi, 2),
(1)
where d is some distance metric and U, e-ball defined on distance metric d.
Similar to these approaches, assume that the generator memorizes the training data thus it generates synthetic dataset close to the training data. Under this assumption, if a sample is closer to the synthetic datasot, it is more likely that belongs to the training data. Hence for a sample 2, we consider using the nearest neighbor distance to synthetic datasets as the query function:
min d(2,g)). Qs(=)
(2)
jE(a
where d is a distance metric.
2.2.2 Accessible models
In this setting, assume the adversary has query accoss to the model (the discriminator and the generator in the case of GANs). Such a situation commonly arises when researchers open source model parameters, share model parameters insecurely.
For generative modols, especially GANs, the most successful attack known Hayes et al. 2019 assumes adversaries to access the model via the following query:
= D(())=((2))
(3)
where (22) is the output of the discriminator corresponding to input sample z. Intuitively, if sample is in the training set, the discriminator would be more likely to output high values. While the adversary could
4

---

Published in Transactions on Machine Learning Research (10/2022)
solely access the discriminator via the above query, we introduce query below allowing accessing both the generator and the discriminator.
Qs(=) - (D(=), min d(=,9)))
(4)
jein]
This attack is a combination attacks described in Equations and 2
While the discriminator score has been shown to very effective query for generalive models with one discriminator, many recent privacy preserving generative modeling approaches often have multiple discriminators (Jordon et al.) 2018 Mukherjoe et al. 2019), with cach discriminator being exposed to part of the training dataset. Our previous query will not be useful in such situations. Here, consider the recent work privGAN DrivGGN((Muuhhrr) ((Mukherjee et al. 2019) and present two queries (one single dimensional and one multi-dimeenional), The single dimensional query used in (Mukherjee et al. 2019 is as follows:
Qs(z) = max (z))),
(5)
where D(2) is the output of the discriminator i corresponding to input sample 2, We propose new multi-dimennsinal query which is stated as follows:
Qs(=) = (D) Dr(=))
(6)
For the purposes of demonstration, in this paper we use r = 2.
2.3 Membership inference attack adversaries
The goal of a membership inference attack (MIA) (Li et al. 2013 Shokri et al., 2017 Truex et al. 2018 Long ct al. 2017), is to infer whether a sample is a part the training set S. In our paper, we assume an MIA access the model via query function Qs- Thus, an MIA adversary equivalent as a classifier given Qs(2) as the input to predict whether 2 belongs to S.
In this paper, we focus on generative machine learning models (such GANs). The study of MIAs against generative models is relatively new research area. Hayes al. (2019) first demonstrated MIAs against GANs. They propose:
black-box adversary that trains shadow GAN model using the released synthetic data, ii) a white-box adversary that uses a threshold on the discriminator score of a released GAN model. Hilprecht et al. (2019) demonstrates a black-box MIA adversary that uses only the generator of the GANs (or synthetic samples) and operates by thresholding the L2 distance between the query sample and the closest synthetic sample. These cxisting works focus on the construction of strong binary classifier as the MIA adversary, given different query functions. The details of these query functions have been discussed in Section 2.2.
2.4 The attack experiment
The membership privacy risk that arises from a query the model usually evaluated through a membership privacy experiment (Yeom et al.) 2018 Jayaraman et al. 2020p. The experiment assumes we have sampled training set S with size |S| = from the data distribution D. Then learning algorithm is trained on S, and an adversary would have access to the trained model through query function Qs : Z + Q. To be specific, an adversary is provided with the query output of a randomly sampled point z from either (with probability p) or D (with probability 1 1-p). The adversary would then get a utility if it guesses the membership correctly or an utility -1 otherwise.
2.5 The Bayes optimal classifier
Since membership identification essentially a binary classification task, a membership adversary can then be seen simply as binary classification model. Indood, many existing papers on membership inference explicitly train binary classifiers for the purpose of membership inference (Shokri al.) (2017). The performance of such classifiers is often also used to empirically measure the membership privacy risk of different. models
5

---

Published in Transactions on Machine Learning Research (10/2022)
[Mukherjee et al. 2019). As the binary classifiers used in such papers are heuristically chosen, there is no guarantee that a better classification model does not exist for the task. The classifier that minimizes the expected error rate (maximizes accuracy) is called the Bayes optimal classifier (Devroye et al. 2013). Given, samples (z, y) E Rd x (-1,1), the Bayes optimal classifier is:
Bayye (x)
arg max P(Y = = = x).
rE(-1,1)
Note: the Bayes optimal classifier can only be approximated in practical scenarios, since we rely on estimates of IP = r|XX x))
While the Bayes optimal classifier in Equation 7 was originally designed to maximize the classification accuracy,
Koyejo et al. (2014) has exlended it. to a family of the generalized metrics. We first. define TP, FP, FN
and TN as IP (X() L,Y 1), P(A(X)= = 1, = -1), ((A(X) = -1, 1) and P(A(X) 1, -1)
rospectively. Next, we show the definition of the generalized metrics as follows:
Go + 011 TP + FP + FN + Too
(AA P)
(8)
bo + b1 TP + b1oFP + FN TNN
where A is the adversary, P is the distribution, Go bu. asj and by are pre-defined scalars for i = 0, 1 and j = (X,)) ~ P. The generalized metric can be used to represent several commonly used metrics such as accuracy, PPV, TPR, TNR, WA etc. (see Appendix). It then demonstrated that the Bayes optimal classifier for this family of generalized metrics takes the forms:
(9) where n7(x)= =- IPYY- 1AA(X) - A(x)) and 4e (0, 1) is constant depending on the metric 6, when the marginal distribution of X is absolutely contimuous with respect to the dominating measure on X (Koyejo et al.] 2014). It is worth noting that the Bayes optimal classifier for the generalized metric can only be approximated in practical scenarios, due to the lack of closed-form expressions of n)z)) and te.
sgn(n(z) - tt) or sgn(te- - -x))).
2.6 Limitations of current membership inference approaches
There are several limitations in the existing literature membership inference. First, most papers (Hayes et al. 2019 Hilprecht et al.) 2019) focus on developing novel heuristic membership inference attacks, which are often limited in scope and can hardly be extended to another query. This is particularly problematic as much of these heuristic approaches cannot readily generalize to the generative modeling setting. Second, the current formal membership privacy estimation frameworks Yeom et al. (2018); Jayaraman et al.) (2020) require full access to the underlying data distribution, while this is not always possible in practice. Third, paper has yet provided rigorous approach to estimate the membership privacy risk at the individual level. Fourth, for most of the current membership inference methods (Hayes et al. 2019 Hilprecht et al.
2019) probability of Experiment | is usually set as 0.5 to form a balanced binary classification problem.
However, in practice, p is usually much smaller than 0.5, as pointed out in prior work (Jayaraman et al.
2020
Rezaei & Liu 2020). In this work we seek to address all these issues.
3 The Membership Privacy Risk Quantification
In this section, we first introduce the membership advantage of a given adversary. Then, we define the optimal membership advantage as the maximun membership advantage. It is the maximum expected membership privacy risk of any adversary for the whole population. Furthermore, we present the optimal membership inference adversary. Finally, the individual privacy risk would be proposed to estimate the membership privacy risk at the individual level. The first subsection focuses on the accuracy-based metric, and the second subsection extends to the generalized metrics. We first propose an experiment formalizing membership inference attacks, adapted from Yeom et al. (2018).
Experiment 1. Let DD be the data distribution on Z. We first have a fired training set S ~ D with size SS] = and have a trained model. An adversary A: Q + (-1,1} would access the trained model via the query function Qs Z + Q. The membership experiment proceeds follows:
6

---

Published in Transactions on Machine Learning Research (10/2022)
1. Randomiy sample E {-1,1} such that m with probability p.
Ifm = 1, then wniformly sample 2 ~ S; otherwise sumple a ~ DVS uniformly.
Let (D, S. Qs) denote the distribution of m) in Experiment 7
3.1 For the accuracy-based metric
To define the optimal membership advantage, we first introduce the membership advantage of an adversary A given the query Qs, as given in Definition of Yeom et al. (2018). We define the membership advantage of the query Qs by an adversary A as the rescaled expected accuracy of the membership inference attack adversary A. When p= p=p00 0.5, the membership advantage is equal to the difference between the adversary's 's true and false positive rates.
Definition The membership advandage of Qs by an adversury A defined as
Adv,(Qs, A) (A(Qs(=)) = m)) 11
(10)
where the probabilily is laken over the rondom sample (z, m) drawn from E, Qs)).
If the adversary A random guessing, Adv,(Qs. 0. If the advensary always gets the membership right, then Adv,(Qs. = 1. Further, note that IP (=((s(=))= is the expected accuracy of the adversary's predictions. Hence this definition of membership advantage is directly related to accuracy. After introducing the membership advantage, we define the optimal membership advantage as the maximum membership advantage of all possible adversaries.
Definition 2. The optimal membership advantage is defined as
Adv,(Qs) max AQs;
(11)
A
The following lemma obtains the optimal adversary for the accuracy-based metric.
Lemma 1. Given the query Qs, the data distribution D, the training set S and the prior probability p, the Bayes optimal classifier A mazimizing membership advantage is given by
A²(ss())) sgn (P(m=
sgn
(12)
where the probability is taken over the rondom sample (z, m) drawn from ((D,SQQs) Purthermore, the optimal membership advantage be re-written
Ady)(Qs) E, [2)]
(13)
where
= P 1)(1 -p) P(Qs(2)]m 1)p+ 1)(1 -p)
[x(=)
(14)
Proof. See the complete proof in Appendix D.I
Now we introduce the individual privacy risk at sample zp- It is proportional to the accuracy of the Bayesian optimal classifier A conditioning on Qs(z) = Qs(z)).
Definition 3. The individual privacy risk of a sample for a query Qs under the aecuracy-based metric is defined as:
Advl,(Qs, =0) 2P(AA' m[Qs(=) = Qs(2o)) - 1, where the probabilities laken over the random sample (z, m) drawn from E(D,S,Q9

---

Published in Transactions on Machine Learning Research (10/2022)
We then provide several convenient properties of the individual privacy risk AdvI,(Qs: 20) under the accuracy- based metric at sample 20. First, we show that it could be rewritten as 15p(=0)), where Jp is defined in Equation 14 This can be used in the following sections to derive a consistent estimator and confidence interval. Secondly, we connect the individual privacy risk to the optimal membership advantage. Thirdly, we show that AdvIp(Qs. =o) proportional lo the highest accuracy conditioning on Qs(za). Fourthly, we establish the connection between the optimal membership ad advantage, the individual privacy risk and Differential Privacy.
Theorem 1. Let Qs the query function and 2g a fixed sample E Z.
1. The individual privacy risk at 2o can be re-written
AdvIp(Qs: zu) (zp(2))
2. The oplimal membership advanlage is the expectation of the individual privucy risk given the sample
2 as random variable.
Adv, [A([AvI,(Qss.))
(15)
The Bayesion optimal classifier A optimal at the individualllevel, as Advlp(Qs, z0) = 2 max ((AA(P((()s))))))) = mm]smmm)))))) Q
If training algorithm is e-differentially private, then for anyy choice of 2o; have both the optimal membership advantage and the individual privacy risk bounded by constant determined by e:
Ady1, S mmxx { (aanh (-+
tanh (=E+^) 2 }
where À log -
Advi, When p 0.5 and & 1, we have Adv, (Qd))(s),.))))))))))))))))) AdvI, AdvI, =0) S 0.462 for ony
20 E Z.
Proof. See the complete proof in Appendix D.3
3.2
For the generalized metrics
As previously mentioned, membership privacy leakage is a highly imbalanced problem (Jayaraman et al., 2020 Rezaci & Liu 2020). For example, the training set in medical dataset may consist of data from the palients admitted to a clinical study with a particular health condition and the distribution D may represent data from all patients (in the world). Notably, previous works (Hayes et al.) 2019 Hilprecht et al. 2019 Mukherjee et al., 2019) used metrics such as accuracy, precision and recall to measure the privacy risks even for the highly imbalanced setting where p = 0.1 (Hayes al. 2019 Although these attacks result in high accuracy, precision or recall during the privacy evaluation stage, it has been shown to suffer from a high false positive rate & Liu 2020), thus is less useful in practice.
To overcome these issues, prior work (Jayaraman & Evans 2019 Jayaraman et al. 2020 proposes to use the positive predictive value (PPV), which defined as the ratio of true members predicted among all the positive membership predictions made by an adversary A. Here, we allow users even more flexibility by adopting the generalized metric defined Equation g Through Experiment [ we define the following metric to measure the membership privacy risk under the generalized metric.
Definition The membership advantage of an adversary A that has access a trained model via query Qs under the generalized metric & is defined as
Adve, ((A0))))))(()))))))0......
where E(D,S, Qs) is the distribution generated by Experiment
8

---

Published in Transactions on Machine Learning Research (10/2022)
After introducing the membership advantage under the generalized metric we define the optimal membership advantage as the maximum membership advantage. Definition 5. The optimal membership advantage is defined
Adve,((s)) = max Advep(Qs.))))
A
Similar to the accuracy-based metric, the optinal adversary for the generalized metric is the Bayes optimal classifier. While we mention that the exact function depends on the dataset and the metric, in some cases there exists a closed form solution for it.
Lemma 2. Given the query Qs, the data distribution D, the training set S and the prior probability p, if b1 = bo1 and bi0 = buo, then the Bayes optimal classifier A marimizing membership advantage under the generulized metric asymptotically is given by
(Qs(zz) sgn (P (m 1[Qs(z)) tt)
(16)
where the probability is laken over the sample pair (z, m) drawm from E(D, S,D,S,Qs) Qs) and l4 an -a10 apt +
Ano -10
Proof. See the complete proof Appendix D.2
As a sanity check, for accuracy, we have facc = 1. A wide range of performance measure under class imbalance settings can be seen as instances of the family of metrics under Lemma 2 For example, AM measure (Mennn al.) 2013) defined as AM :- 1/2 (TPR TNR) has optimal threshold fAM = p. As another example, TPR or recall has the optimal threshold tIpr 0, which means in practice, can always
predict as positive to get the highest recall.
Similar to the case of the accuracy based metric, we define the individual privacy risk for the generalized metric as follows:
Definition 6. The indinidual privacy risk of sample zo for query Qs under generalized metric E is defined as
Advli, = (0 (00
where A'' is the Bayes Optimal Adversary under the generalized metris and EDD,S,Q5 QsQQ is the distribution generated by Experiment 1 given Qs(=) = Q=(2))
Remark 1. While we focus on a single query function in this section, the result can be easily extended to set of queries (Q', ' 233 by compuling the marimuum risk of all the queries Q's, 1,...
4 Estimation of the Individual Privacy Risk
In the ideal world, we have full access to the data distribution D. As a result, we could get the exact numbers to describe the individual privacy risk and the optimal membership advantage by Definition 2 6 However, this could hardly be true in practice. For example, it is almost impossible to get access to the health records of all the patients around the world, while a simple random sample could be feasible. Previous studs (Yoom et al. 2018; Jayaraman et al., 2020) dealt with this by assuming the existence of a subset to fully represent a distribution. For instance, Jayaraman al. (2020) used Texas hospital data set consisting of 67,000 patient records as the data distribution to validate their framework. However, their framework could hardly be adapted to the case when the 67,000 patient records are only part of the population of interest.
To conquer this limitation, we exlend to the situation when one can only access the training set S and the data distribution D via a simple random sample. Moreover, we will provide a practical and principled approach estimate the individual privacy risk for both the accuracy-basod and the generalized metrics, In order to achieve this, we first propose an experiment. Note that this experiment would also be used istimate the optimal membership advantage in the following section.
9

---

Published in Transactions on Machine Learning Research (10/2022)
Experiment 2. Let D be the data distribution on 2. We first have training set S ~ DD with size |S| and have brained model. An adversary would access the trained model via the query function Qs 2 + Q. The rutio is the prior probability. The sampling process proceeds as follows:
Uniformly sample N1 = Np points #1," NN from S. Assign 2, = Ii and m, 1, for = 1, ,, ,N
Uniformly sample N2 = (1 - p)N points yı," UN, from DVS. Assign 2, = Vi-Ns and m; = -1, for i N1 + +1,.... .N2.
3. Create a consolidated set of samples {(2,, m,)) where m, = -1 if N1 and m; = 0 if 2 Ni.
We assume that Np and N p) are always integers for simplicity in this paper. But keep in mind that all the algorithms, lemmas and thcorems could be casily extended to the case when this assumption does not hold.
4.1 For the accuracy-based metric
By Theorem | have AdvIp(Qs, [f)())) Thus, it is sufficient to estimate fp(2) to estimate AdvIp(Qs.2) In the following sub-sections, we describe the construction of consistent estimators for fp(z).
4.1.1 Discrete queries
When Qs(z) Q is discrete, for particular output j Q, let r, P(Qs(=) j]mm = and q, = P(Qs(2)
Settings(Muukeejjee j'm -1). Frequency-basee plug-in methods have been used empirically in similar settings Mukherjee et al.
2019 Yaghini et al. 2019). We simply collect samples from Experiment 2 and plug-in the fraction to estimate Tj and Qj- Then we account for the estimation error of this process by using a Clopper-Pearson confidence interval (Clopper Pearson 1934). We find the 2 3-Clopper Pearson lower bound for ry denoted as flowerr and the 2 CCCooperr Pearson upper bound for rý denoted as (uppeer Finally we have the following theorem.
Theorem 2. Let (X,,""" YI,'.,, YN,} randomly sampled by Experiment Qs())= = j. then
consistent eslimator of So(z) is given as follows:
f,p - 43 p)
83p + +1 p))
where Pj = Ni E. TX,s(X,))= = j), and  = 1 En', T(Qs(Vi) j).
No
2. The - (1- conffience interval (1))(2) of fp(z) is
f Folowerr Qjupper (1 p) fjupperP G3 lower (1 p) F F.JowwrP + uyppeer (1 p) P.upperr dowo p)
Proof.
See the complete proof in Appendix D.4
4.1.2 Continuous queries
Consider when Qs a continuous query. Let r(x) P(Qs(z) x/m 1) and g(x) P(Qs(=) x/m 11). We first use Kernel Density Estimators (KDE) for both r(x) and g(x).
Recall that for samples U1, U2, uN E Ed from unknown distribution R defined by probability density function KDE with bandwidth h and kernel function K is given by
K
Uz h
(17)
Nha
Additionally, we have a plug-in confidence interval for KDE (Chen 2017 (see Lemma consistent estimator for fp(2) with confidence interval to estimate the uncertainty.
Using this, we have
10

---

Published in Transactions on Machine Learning Research (10/2022)
Theorem 3. Let Qs be continuous query and E Z. Let {Xi, .NN .N..V... ... .Nat randomly sampled from Pxperiment
A consistent estimator of fo(z) is given as follows:
PNy -
1 where Fx((s(z)) = Nihd ΣM K (ssl)))(s(X.)) and IN (Qs(z)) = Night
1 NNo K (Os())-(o(X))
2. The (1 8)-confidence interval Ci of 5p(z) is
Flower Flower(Qs( (Qs(=))p Tupper (Qs(2))( p) Puppor (Qs(z))p - Tlower (Qs(=))))=)))))))))))))))))))))) - p) Flower (Qs(2))p Jupper (s(2)) - p) " Fupper ( (Q (2))) pp+pt Nower (Qs(z))(1 (p)
where (Fower, Fupper]] , [lower, Tupper] are - 6/2) confidence intervals of r(Qs(=)) and ((Qs(=))
spectively. Both confidence intervals are derived by Lemma 4
Proof. See the complete proof in Appendix D.5
4.2 For the generalized metrics
Similar to the case of the accuracy-based metric, we propose a consistent estimator to estimate the individual privacy risk for the generalized metric. Though there is not always closod-form expression for the individual privacy risk for the generalized metric, we are able to provide an explicit formula given a few conditions. The following theorem provides a way to calculate the confidence interval for the generalized metric given a known tz-
Theorem 4. Let = (ao + ao1), C2 = (oo + awo), (an - aw1), (a10 aoo), d1 = (bu + boi), d2 = (bn - boo), d3 = (b11 - , ds = (bio - boo), and zo be sample E Z. Under the following conditions:
1. The Bayes optimal classifier A for the generalized metric can be written in the form of sgn(n(x) -te), and le is knowm constant;
Pmm 11Qs(2) = Qs(2o)) # tz:
8. pdz 11
4 - (=)((((=)) = -1) + (d)F(Qs(20)mm = 1)I(P(m = 1[Q)(z0)) > (z0)()))) #0;
4) + =
have
consistent estimator of AdvIr, =0) can be given as follows:
c1fp + c cappl(( te)pp te(1 - p)g) + - (1)((11))((1166)) tt te(1 p)g)
(18)
dipp + d24(1 - + dsfpI((1 te)pp > te(1 - p)4) + d1q(1 (it((1pp(((( - -t))pp > te(1 - (p))
If Q is discrete and Q (2)) = j, then f; and q aj. as defined in Theorem continuous query, then f = fN (Qs(22))) and q (Nx()z2)), as defined in Theorem
If Qs is
2. The (1 - 11-8)-confidence interval of Adv (n) follows by plugging the (1-8/2) confudence intervals of p and [flower, i Pupper] and (Qowerr [upper] into Equation 18. the case of discrete queries, [Rower, P Pupper] = [o), fj,upper), where Q2(20) J. as defined in Theorem [ In the case of continuous queries, [Flowrr, Pupper] Foowrr (Qs(2)))) Pupper (Qs(z))),, as defined in Theorem qlower and Pupper are defined in similar manner.
Proof- See the complete proof in Appendix D.6
11

---

Published in Transactions on Machine Learning Research (10/2022)
A wide range of performance measures satisfy Condition 1. For example, AM measure (Menon et al. 2013 has the optimal threshold AM p, and TPR has the optimal threshold /TPR 0. Condition 2 can be waived if c3 = C4 = d3 = d4 O. Metrics such as accuracy, recall and specificity meet the above condition. Condition assumes a nonzero expected value of the denominator Equation [8] As for the confidence interval 2, there could exist a closed-form expression if for example, Ad,I(Qs, =o) is a monotonic function in terms of P(Qs(2+))[m 1)))(Qs(20)) m -1). Otherwise, an optimization problem would need to be solved, as outlined in 2.
5 Estimation of the Optimal Membership Advantage
In the above section, we proposed to estimate individual privacy risk for both the accuracy-based and generalized metrics with theoretical justifications. Tn this section, we further develop methods to estimate the optimal membership advantage (Qs).
5.1 For the accuracy-based metric
We first propose consistent estimator for the optimal membership advantage given discrete query.
Theorem 5. Let XI.... ,......,N NN,, be ii.d. random variables drown from S and Y.,".., Yn, be i...d random variables drawn from D in Eaperiment 2 where Ni = =p= and N2 = (1 -0--p p)N. Assume that Q is a finile set. Define
Wv W(X1,"" NN,,,,," ,Nn))
N₂
1 N₂
= Σ
M I(Qs(X,)) = N
2
Σ N₂
I(Qs(M) = j)1
JE
Then (1) W is consistent estimator of the oplimal membership advantage Adv (Qs); (2) P([W EW > V log( ()) < 8.
Proof. See the complete proof in Appendix D.7
We then propose a consistent estimator for the optimal membership advantage given a continuous query.
Thcorem 6. Let X., XN,, be i.i.d. random variables drawn from S, and Y1, .YN, be i.i.d. random variables drawn from DVSS in Experiment 2 where N1 Np and N₂ N(1 p)- Assume that Q = Rd and h + 0, as N - 0o. Define
UN U(X1, XN. YI....,,,, .YNz) N.
Σ
K r
(x(
1
N
K
F
Qs(Y)) )dz.
Nihh
h
Then (1) UN is a consistent estimator of the optimal membership advantage Adv,(Qs): (2) P(UUN - EUN V 2 ()) < 8.
Proof. See the complete proof Appendix D.8
The practical estimation of the optimal membership advantage for the accuracy- based metric is described in
A Algorithm T using Monte Carlo integration.
12

---

Published in Transactions on Machine Learning Research (10/2022)
Algorithm Practical estimation of the optimal membership advantage AQ))(Qs)
Input: number of samples N, prior of membership training set S, query function Qs, confidence level 8
Output: the optimal membership advantage estimate Advp(Qs)
1: Perform Experiment
2 and draw one set of samples {ai with membership {m J. respectively. if Qs discrete then
Use the tuples m,))) to calculate WN
4: Adv)(Qs) + WN
5:
clse
6: Use the tuples ())) m.) to calculate UN
7: (Qs) t UN
8:
end if
5.2 For the generalized metrics
In this subsection, we propose consistent estimators for the optimal membership advantage when there exists closed form solution for the optimal adversary.
Under some specific condition, Lemma 2 shows that the Bayes optimal classifier A maximizing membership advantage under the generalized metric / is given by
(Qs(())) = sgn (n(Qs(=)) - m(Q((z)) = Pp = 1[Qs(z))
(19)
where tę (Aoo (100/(aa))) (an 010 001 + ago). Hence, we propose Algorithm 2 when the Bayes optimal first split the N samples into two partitions, and use the first partition to obtain the estimator n(Qs(z)) for n)Qs(z)). Following this,
classifier can be written as Equation and 4 known. As shown in Algorithm 2
we estimate the Bayes optimal classifier A" by
A(Qs(z)) = sgn (?(Qs(z)) - tz)
Next, we use the second partition obtain an empirical mcasure of Adve,p More specifically, we
Algorithm 2 Practical estimation of the optimal membership advantage (Qs,,p(Qs) under a generalized
metric {
Input: number of samples N, prior of membership p, training set S, query function Qs, generalized metric Output: the empirical membership privacy (Qvr,((Qs) estimate
1:
Perform Experiment 2 and split the N samples into two sets S and S2.
Estimate n2))(z)) by (()s(z)) using S1.
Let A())(2)) = sgn (f()s(2)) tz), where tę (a0o a11)/(a11 010 G01 + ago).
4: Calculate Adve,p (Qs. A) using S2-
adopt the empirical measure defined by Koyejo et al. 2014). For any adversary A, assuming that we sample {2i, m by Experiment 2 we first define
TP, = Σ ( A(Qs(z)) + 1) () mh + 1 2) (A) = Σ ==1
( 2
A(Qs((=)+ + 2
2
(=1
as the empirical estimate of TP= P(M = 1, L,A((s(Z)) = 1) and P(A(Qs(Z)) = 1). After this, we define the empirical measure of Adv,,, A) follows:
eu + TP + (A)
Adv
Adv (Qs,
(20)
ho + TT (A) (A)
with the constants
e0 0010 + ag0
4000 (gg e1 .11 010 0g1
ao0; e2 0110
400:
13

---

Published in Transactions on Machine Learning Research (10/2022)
hg buip + boo- boop + bu: h b11 b1u buz + buo, h2 b10 buo-
2 the empirical measure A (Qs, Adve,p |521
In Algorithm
A).
Next, we introduce the following lemma giving the form of the Bayes optimal classifier when the query Qs is continuous.
Lemma 3 (Koyejo et al. (014)). Assume that the marginal distribulion of (2(2) in Experiment 1 is absolutely continuous with respect to the dominating measure on Qs(Z). Given the constants co. C1, e2, ho, h1, h2 defined in Equation 20 define
h2Adve, (Qs) e2
62 h1 Adve.
When ei hi Adve, (Qs), the Bayes optimal classifter A macimizzng membership advanlage under the generalized metric 8 is given by sgu(n(Qs(z)) - ty);
When e₁ h1 Adve,(Qs), the Bayes optimal classifier A mazimizing membership advantage under the generalized metric is given by sgn(t; m(Qs(z)))
By Lemma 3. the specific form of the Bayes optimal classifier relies on the unknown optimal membership advantage (Qs). Koyejo al. (2014) suggested to estimate loose upper and lower bounds of Adve,p(Ss) to determine the classilier. In the rest of this subsection, assume el >> hi pp (Qs(((z)) sgn(n(Qs(z)) - tt) The case where e1 h1Adve,,(Qs can be solved similarly.
Based on Lemma B we propose Algorithm when the query Qs is contimuuus. In Algorithm 3, we first split. the N samples into three partitions, and use the first partition to obtain the estimator n(Qs(=)) for ((Qs)))). Next, we use the second partition to estimate tz. Combing these two steps, we obtain the empirical Bayes optimal classifier A(Qs(z)) sgn (eofff((Qr) Qs(2)) fz). We use the third partition calculate the empirical
[Sal
measure of Advep(Qss.4 by Equation 20 Especially in Algorithm 3 AQv = Adve,p (Qs,
Algorithm 3 Practical estimation of the optimal membership advantage Adve,p under generalized metric & when l is unknown
Input: number of samples N,, prior of membership p, training set S, query function Qs, generalized metric { Output: the empirical membership privacy Advr,((Ss) estimate
1: Perform Experiment
2 and draw three sets of samples S1, S2 and Sz.
2: Estimate ())()))) by ((Qs(z)) using St.
Compute & arg ((.1) Advv.p(0s.suu(7((ss0)
z)) S2.
4: Let A()s(=)) sgn (7))s(=) - (t). Calculate Adve,p(Qs.. A) using S3-
We now introduce the following nice propertics of the two algorithms. First, Theorem shows that Advz,p (Qs, in Algorithm is a consistent estimate of the optimal membership advantage Adv if E - 7/2 + 0. Using a suitable strongly proper loss function, we can obtain an estimator f satisfying Eiń - 2/2 +> by the proof of Theorem (Menon et al., 2013
Theorem 7. Assume that b = bot, b1o = boo, a1 > 4oi and 060 010- Let A outputted by Algorithm If - (=))(=)) (()
2, for some a 2 1,
Adve P. Adve,
Proof.
See the complete proof Appendix D.9
Second, we show the consistency of the empirical measure of the membership advantage of any adversary by the proof of Lemma 8 (Koyejo et al. 2014).
Theorem [Koyejo et al. (2014)). For each adversary A, Adve, A) E, Adve,(Qss.
14

---

Published in Transactions on Machine Learning Research (10/2022)
Finally, show the following nice property of fz the estimate of in Algorithm 3
Thcorem 9 (oyyeoo al. (2014)). Assume that the marginal distribution of Qs(Z) Experiment absolulely conlinuous with respect to the dominaling measure on Qs( Let le be oulpulted by Algoritio
If f) 4
Adv.,pQ2s,sgn(/((ssss [e)) 2, AQs)
6
Experiments
In this section, we demonstrate how to use MACE by performing practical membership privacy estimation on trained generative models.
6.1 Setup
The different GAN architectures used were WGAN-GP (Gulajjaii et al. 2017 (on CIFAR-10 & MNIST),
JS-GAN [Goodfellow et al.) 2014 (on skin-cancer MNIST) and privGAN (Mukherjee et al. 2019 (on MNIST). A JS-GAN is the original GAN formulation which uses Censen-Shammen divergence based loss. A WGAN-GP is an improved GAN formulation with Wasserstein distance based loss with a gradient penalty. privGAN is a GAN formulation that utilizes multiple generator-discriminat pairs and has been empirically shown to provide membership privacy.
Our experiments would be based on the following three real-world datasets: MNIST, CIFAR-10 and skin cancer MNIST. MNIST contains gray scale images of handwritten digits with 70000 digits from 0 to 9. CIFAR-10 comprises of 10 classes of 32 x32 RGB colored images with 60000 images in the whole dataset. Both of them are commonly used in the GAN literature. Additionally, to demonstrate a real world use case in healthcare, we use the skin cancer MNIST dataset (Tschandl et al. 2018) which comprises of 10,000 64 x 64 RGB images of skin lesions (both cancerous and benign).
Following the common practice of membership inference attacks on generative models (Hayes et al. 2019 Mukherjee al. 2019 Chen et al. 2019), we choose random 10% subset of the entire dataset as training set to show overfitting. In sub-section 6.3 10% of the training images were corrupted by placing white box at the center of images (with no changes to the non-training set images). For all the experiments, we set the confidence level 8 = 0.05. To create discrete queries, we bin the continuous interval into 100d bins (where d is the dimension of the query).
Discrete vs Continuous queries
B
Discriminator vs Generator query
Companison heunstic attacks
Membership Advantage
1.0
Membershhp Advantage
Opt Ady- Discrmmnater
Membership Advantage
1.0
0.8
Hearidiic Generbor (MCMC)
Heurtai--icciinnvte (LOGAN)
06-
0.6
0.4
84-
02
02
heustiic Opt Apv Contruuus
08
80
5090
10003
3300
1000
1250
3300
1858
Epochs
Epochs
Epochs
Figure 1: a) Comparison of discrete continuous queries against the WGAN-GP on the CIFAR-10 dataset. b) Comparison of queries against the generator and the discriminator the skin-cancer MNIST dataset (for JS JS-GAN), In all cases 8 is set to 0.05. c) Comparison of the optimal membership advantage and heuristic attacks' membership advantage for both queries against the generator and the discriminator on the skin-cancer MNIST datasel (for JS-GAN).
15

---

Published in Transactions on Machine Learning Research (10/2022)
6.2 Estimation of the optimal membership advantage
6.2.1 For the accuracy-based metric
We first demonstrate the utility of our estimators of the optimal membership advantage under the accuracy- based metric with regard to different query types. In Figure D we show the applicability of our discrete and continuous estimators using the discriminator score as the query. Additionally, we compare the estimated performance of the optimal adversaries against a hcuristic adversary that uses the same query (Hayes et al., 2019). We find that for both discrete and continuous queries, the estimated membership advantage is higher for the optimal adversary compared to the heuristic adversary. We note that the continuous query is seen to yield somewhat poorer performance than the discrete query, most likely due to sub-optimal selection of the KDE bandwidth. Optimal choice of the binning/ DKD KDE hyperparameters
beyond the scope of this paper but we direct readers to existing papers on this topic (Chen 2015 Knuth 2019). In Figure D we show the utility of our estimators on queries against accessible models and accessible datasets using query of each type. We use the discriminator score as an example of queries on an accessible model and the query described in Equation 2 as an example of queries on an accessible synthetic dataset. Additionally, we compare our optimal membership advantage with the membership advantage of SOTA heuristic adversaries that use the similar queries as in (Hayes al.) 2019 Hilprecht al. 2019 As widely reported (Hlayes et al. 2019 Mukherjee al. 2019), find among our experiments that the optimal membership advantage is lot smaller when adversaries gain access to the datasets compared to when they get direct access to the model. Furthermore, our optimal membership advantage estimates are higher than the SOTA heuristic adversaries in both settings. This validates Theorem 5 and 6 and demonstrates that our estimators are good estimators
for the optimal membership advantage that would bound all the membership advantages including those due to the SOTA heuristic adversaries.
A
Single Multi Nultt-Dimensionlllc querles against privGAN
B
Single Multt-Dimensional querles JSGGNN
1.0
Membership Advantage
Optimal Adversary
Advantage
1d Optimal Adversary
0.6
0.6
Opt Adv- generator discriminator
Membership
Opt generaerr
0.4
0.4
Opt Adv discriminator
0.-
0.2
500
1000
1500
2000
2500
3000
500
750
1000
1250
1500
1750
Epochs
Epochs
Figure? 2: Comparison of single dimensional vs multi-dimensional queries against (a) privGAN on the MNIST dataset and (b) WGAN-GP on the skin-cancer MNIST dataset.
Next, we demonstrate the applicability of our estimators to multi-dimenniioal queries. In Figure 2 we compare the estimated membership advantage for a single and a multi-dimensional queries against privGAN (Mukherjee et al. 2019). We see that the estimated membership advantage with the multii-dimensional query is much higher than that of the 1-d query used in the privGAN paper. This indicates that while the PrivGAN is less likely to suffer from overfitting than the JS-GAN, relcasing multiple discriminators could potentially increase it's privacy risk. In Figure 2b, we compare the estimated membership advantage for two single and one multi-dimensional queries against the JS-GAN. The multi-dimennional query indeed a hybrid query that is the combination of the two 1-d queries (Equation 4.. Intuitively, a hybrid query should impose a higher privacy risk than an individual query. Our experiment has validated this assumption and shown that the hybrid query has a higher estimated membership ad advantage than each individual 1-d query.
16

---

Published in Transactions on Machine Learning Research (10/2022)
MNIST
CIFAR-10
1.0
1.0
0.8-
0.8
2xAccuracy-1
2xAccuracy-1
0.4
0.4
0.2
estimated advantage by binning
0.-
estimated advantage binning
0.05 confidence interval binning
0.05 confidence interval by binning
whitebox
0.0
0.0
4000
6000
12000
2000
4000
10000
12000
Epochs
Epochs
1.0
0.8
0.8
AM metric
AM metric
0.6
0.4
0.2
estimated advantage by binning
estimated by binning
confidence interval by binning whitebox attack
0.05 confidence interval by binning whitebox attack
0.0
0.0
2000
4000
6000
12000
2000
4000
6000
8000
10000
12000
Epochs
Epochs
Figure 3: The membership advantages of the optimal adversary vs. the SOTA adversary against WGAN-GP under different metrics on the MNIST and CIFAR-10 datasets. For the sake of consistency, the membership advantage estimation for both metries done using the method used for the generalized metric.
6.2.2 Generalized metric
To demonstrate how to apply MACE under the generalized metrics, we qualitatively compare the membership advantages under AM and the accuracy-based metric. We set. p p= here to construct. an imbalanced dataset. In Figure under both the accuracy-based metric and the AM metric, we compare the optimal membership advantage (the membership advantage of the optimal adversary) with the membership advantage of the heuristic adversary defined in (Hayes et al., 2019) using the discriminator score as the query. We find the heuristic adversary has comparable performance to the optimal adversary under both metrics. It is important to note here that while the optimal adversary is asymptotically optimal, on any set of samples there may be a stronger adversary possible.
6.3 Estimation of individual privacy risks
Having demonstrated the utility of our estimators for estimating the optimal membership advantage, here we demonstrate the utility of our estimators for the individual privacy risk. is well known that samples from minority sub-groups can often be more vulnerable to membership inference attacks (Yaghini et al. 2019 Thus it is necessary reveal the membership privacy at the individual level to reflect the true risks faced by the minority group. For the purposes of simple demonstration, we constructed toy dataset where 10% of the images were corrupted (described in section 6.1 After that, we trained JS-GAN this dataset and estimated the individual privacy risk of samples against the discriminator score query using the individual privacy risk estimator described in section 4.1.1 In Figure 4 we show (both qualitatively and quantitatively) that the corrupted images on average have a higher individual privacy risk than the uncorrupted images. This shows that outlier samples or cortain sub-groups can be more vulncrable than the rest of the population, and the optimal membership advantage alone may not capture this information. It is worth noting here that there are particularly severe ramifications of higher privacy leakage risks of minorities in healthcare settings,
17

---

Published in Transactions on Machine Learning Research (10/2022)
A
f_hat =1.0
hat =1.0
B
Avg. Individual Privacy Risk
1.00
Corrupted
0.99
f_hat =0.0
hat =0.6
Uncorrupted
Comuupted
0.96
Uncorrupted
600
800
1000
1200
1400
1600
1800
2000
Epochs
Figure 4: Application of individual privacy risk estimation. a) Example corrupted and uncorrupted images from skin-cancer MATST, along with their estimated individual privacy risk. b) Comparison of average individual privacy risk between corrupted and uncorrupted images.
specially if such information points to the disease status. Data and model owners in such cases can consider retraining their generative model by omitting such sub-groups of samples.
7 Conclusion and Remarks
We developed the first formal framework that provides a certificate for the membership privacy risk of a trained generative model posed by adversaries having query access to the model at both the individual and population level. While our theory works regardless of the query dimension, we do not have practical way to generate a mcaningful certificate for the high-dimensional case. This would be a focus our future work. Our framework works for a large farily of metrics, allowing users flexible ways to measure the membership risk. Through experiments on multiple datasets, queries, model types and metrics, we show the practical applicability of our framework in measuring the optimal membership advantage well the individual privacy risk. Finally, to wrap up the paper, re-visit fictional example from the introduction to explain how MACE can help such data/model owners:
By comparing the optimal membership advantage against given trained model of queries that access the model via different practical queries, the model owners can determine the relative risk of releasing: i) the complete trained model, ii) parts the trained model, iii) the synthetic dataset.
For model owners who are only interested releasing synthetic data, MACE can help identify how much data can be released for a desired level of membership advantage (see Appendix for example).
For datasets containing sensitive groups of samples, MACE allows to estimate the group-level membership privacy risk through the estimation of the individual privacy risk and identify the most. vulnerable minorities.
While we focus generative models in this paper, framework can be applied to discriminative models well. An example application of our framework to discrminative model is shown in the Appendix. While we focus on the thcoretical aspects of membership privacy estimation in this paper, future work could look at using MACE to design new queries which can lead to stronger MIAs and a better understanding of membership privacy risks of different generative models. Another important direction is to derive a consistent estimator the optimal membership advantage for the generalized metric by adapting Koyejo et al. (2014). It would also be interesting to extend our work to high-dimensional queries such certain layers of the generator and discriminator.
18

---

Published in Transactions on Machine Learning Research (10/2022)
References
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan. ar preprint ar 101.:70771.777 2017.
Tharindu Bandaragoda, Kai Ming Ting, David Albrocht, Fei Tony Liu, and Jonathan R Wells. Efficient: anomaly detection by isolation using nearest neighbour ensemble. In 2014 IEEE International Conference Data Mining Workshop; pp. 698-705. TEFE, 2014.
David Berthelot, Thomas Schumm,, and Luke Metz. Began: Boundary equilibrium generative adversarial networks. Xiviv preprint (eXic:1033.1017 2017.
Dingfan Chen, Ning Yu, Yang Zhang, and Mario Fritz. Gan-lcaks: A taxonomy of membership inference attacks against gans. arXiv preprint arXiv:1909. 01119999099933 2019.
Junjie Chen, Wendy Hui Wang, Hongchang Gao, and Xinghua Shi. Par-gan: Improving the generalization of generative adversarial networks against membership inference attacks. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 127 127-137, 2021.
Su Chen. Optimal bandwidth selection for kernel density functionals estimation. Journal of Probability and Statistics, 2015, 2015.
Yen-Chi Chen. A tutorial kernel density estimation and recent advances. Biostatistics & Epidemiology, (1):161-187, 2017.
Charles J Clopper and Egon Pearson. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26(4):404 413, 1934.
L. Devroye and L. Gyorfi. Nonpurametric Densily Estimation: The L1 View. Wiley Interscience Series in Discrete Mathematics. Wiley, 1985. ISBN 9780471816461. URL https://books. wowples.gooog...ooo www.gooooooooooooooooooooooooooo com.bokks/obccooocooooooooa
GGpCo)
Lue Devroye, László Györfi, and Gábor Lugosi. A probabilistic theory of pattern recognition, volume 31. Springer Science & Business Media, 2013.
Charles Elkan. The foundations of cost-sensitive learning. In International joint conference on artificial intelligence, volume 17, pp. 973-978. Lawrence Erlbaum Associates Ltd, 2001.
Vitaly Feldman. Does learning require memorization? short tale about long tail. In Proceedings of the 52nd Annuual ACM SIGACT Symposium on Theory of Compuling, pp. 954-959, 2020.
Jeremy Georges-Filtest and Elisa Cirillo. Synthetic observational health data with gans: from slow adoption boom in medical research and ultimately digital twins? arXiv preprint arXiv:2005. 13510, 2020.
Tan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing syslems, pp. 2672 2680, 2014.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. In Advances in neurol information processing systems, pp. 5767-5777, 2017.
Jamie Hayes, Luca Melis, George Danezis, and Emiliano De Cristofaro. Logan: Membership inference attacks against generative models. Proceedings on Privacy Enhancing Technologies, 2019(1):133 152, 2019.
Benjamin Hilprecht, Martin Härterich, and Daniel Bernau. Monte carlo and reconstruction membership inference attacks against generative models. Proceedings on Privacy Enhancing Technologies, 2019(4): 232 249, 2019.
Bargav Jayaraman and David Evans. Evaluating differentially private machine learning in practice. In 28th { USENIX) Security Symposium (( (UUSENIX) Security 19), pp. 1895-1913 2019.
19

---

Published in Transactions on Machine Learning Research (10/2022)
Bargav Jayaraman, Lingxiao Wang, David Evans, and Quanquan Gu. Revisiting membership inference under realistic assumptions. ar arXiv preprint orXie:2005. 000:2005.1888.. 10881, 2020.
James Jordon, Jinsung Yoon, and Mihaela van der Schaar. Pate-gan: Generating synthetic data with differential privacy guarantees. In International Conference on Learning Representations, 2018.
Kevin H Knuth. Optimal lasa- binning for histograms and distogram-based probability density models.
Digital Signal Processing, 95:102581, 2019.
Oluwasanmi O Koyejo, Nagarajan Natarajan, Pradeep K Ravikumar, and Inderjit S Dhillon. Consistent binary classification with generalized performance metrics. In Advonces in Neural Information Processing Systems, pp. 2744-2753 2014.
Ninghui Li, Wahbeh Qardaji, Dong Su, Yi Wu, and Weining Yang. Membership privacy: a unifying framework for privacy definitions. In Proceedings of the 2013 ACM SIGSAC conference on Computer & communications security, pp. 889 900, 2013.
Yunhui Long, Vincent Bindschacdler, and Carl A Gunter. Towards measuring membership privacy. ar preprinl Xiv: 772.09136, 2017.
Aditya Menon, Harikrishna Narasimhan, Shivani Agarwal, and Sanjay Chawla. On the statistical consistency of algorithms for binary classification under class imbalance. In International Conference on Machine Learning, pp. 603 611. PMLR, 2013.
Mchdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXi::1411.11 2014.
Sumit Mukherjee, Yixi Xu, Anusua Trivedi, and Juan Lavista Ferres. Protecting gans against privacy attacks by preventing overfitting, arXiv preprint arXiv:2001.0007. 2019.
Milad Nasr, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, and Nicholas Carlini. Adversary in- stantiation: Lower bounds for differentially private machine learning, arXiv preprint arXiv:2101.04535, 2021.
Francoss Rajotte, Sumit Mukherjee, Caleb Robinson, Anthony Ortiz, Christopher West, Juan M. Lavista Ferres, and Raymond T. Ng. Reducing bias and increasing utility by federated generative modeling of medical images using a centralized adversary. GoodIT '21, pp. 74,84, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450384780.
Shahbaz Rezaci and Xin Liu. Towards the infcasibility of membership inference on deep models. arXiv preprinl ar Xrx::20005. 13702, 2020.
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Yann Ollivier, and Hervé Jégou. White-box vs black-box: Bayes optimal strategies for membership inference. In International Conference on Machine Learning, pp. 5558 5567, 2019.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 3 18. IEEE, 2017.
Elysse Tom, Pearse A Keane, Marian Blazes, Louis R Pasquale, Michael F Chiang, Aaron Y Lee, Cecilia S Lee, and AAO Artificial Intelligence Task Force. Protecting data privacy in the age ai-enabled ophthalmology. Translational Vision Science & Technology, 9(2):36-33 36, 2020.
Reihaneh Torkzadehmahani, Peter Kairouz, and Benedict Paten. Dp-ogan: Differentially private synthetic data and label generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 0 0, 2019.
Stacey Truex, Ling Liu, Mchmet Emre Gursoy, Lei Yu, and Wenqi Wei. Towards demystifying membership inference attacks. arXiv preprint Xiv: 1ett1097707 09173,11111911 2018.
20

---

Published in Transactions on Machine Learning Research (10/2022)
P. Tschandl, C Rosendahl, and H Kittler. The ham10000 dataset, large collection of multi-source dermato- scopic images of common pigmented skin lesions. scientific data 5, 180161 (aug 2018), 2018.
Paul Voigt and Axel Von dem Bussche. The eu general data protection regulation (gdpr). A Practical Guide, 1st Ed., Cham: Springer International Publishing, 10:3152676, 2017.
Liyang Xie, Kaixiang Lin, Shu Wang, Fei Wang, and Jiayu Zhou. Differentially private generative adversarial 2018.
network. arXiv preprint arXi::1802.0000
Mohammad Yaghini, Bogdan Kulynych, and Carmela Troncoso. Disparate vulnerability: On the unfairness of privacy attacks against machine learning. arXiv preprint arXiv: 00089, 2019.
Samuel Yoom, Irene Giacomelli, Matt Fredrikson, and Somesh Tha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st Compuler Securily Foundalions Symposium (CSF), pp. 268 282. IEEE, 2018.
A
Background Generative Adversarial Networks
Generative Adversarial Networks the most common class of generative models. The original GAN algorithm [Goodfellow et al. 2014) learns distribution of dataset by adversarially training two modules, namely, generator and a discriminator. The goal of the generator G(w) is to learn a transformation that would convert a random vector w to a realistic data sample. The goal of the discriminator module is to reliably distinguish synthetic samples (generated by the generator) from real samples. The mathematical formulation this problem is as follows:
min maxe blog [xx]]+
E (opett)loog( - D(z))
Here, Pr is the real data distribution, and Pg is the distribution of G(w). There have been many GAN variants proposed since (Arjovsky et al. 2017 Mirza & Osindero, 2014 Berthelot et al. 2017). In this work, we examine our framework on the original GAN and some of its variations.
B Examples of common metrics that can be derived from the generalized metric
TP + TN
ACC TP + FP + TN FN TP
PPV or Precision = TP FP TP
TPR Recall
TP FN FN
TN
TNR FP + TN
wi TP W2 TN
WA
w TP + W2 TN + FtFP + FN
C Background on differential privacy
The definition of e-differential privacy is given follows:
Definition (s-Differential Privacy). We say randomized algorithm M e-differentially private if for any pair of neighbouring databases S and S that differ by one record and any output event E, have
P(M(S) F. eFP(M(S) E F.
(21)
21

---

Published in Transactions on Machine Learning Research (10/2022)
D Detailed Proofs
D.1 Proof for Lemma 1
Proof. As we can see from Definition 1 the membership advantage is defined as 2Accuracy (A) 1. This means the Bayes classifier given by sgu (IP (m = [[)s(=)) Ž)) (Sablayrolles et al. 2019 We get the optimal membership advantage by plugging this Bayes classifier.
D.2 Proof for Lemma 2
Proof. When bn = boz and b1o boo, our loss
to TPtaaaPP FN +o bo+bb1 (NT++N))bo(TTNAA
aoten ta FN It becomes linear combination of TP, TN, FP and FN, which is also by to
called cost sensitive classification as defined in (Elkan 2001 As outlined in Elkan (0011) the optimal classifier would predict 1 when
(=)oa + (1 (1-11 (l=n(QQ(=)))am 2 (QQs()))n + +1 - n(Qs(=)))an
Rearranging the terms completes the proof.
D.3 Proof for Theorem 1
Proof. By Equation 16 and Definition [z)(), and 1 follows.
3
we have AdvIp(Qs. m0) = [P(m = 1(Qs(z0)) - P(m = - -1/Qs(z0))
follows from Equation 15 and Equation
Then, we prove 3. If A equals Q2(2a), 2P 2F[A((s(()) m))() = P(m 1[Qs(=n) PP(m
-1=
2m = - [=0) -=))===o)]- Tf A equals -1 at Qs(=0), 2P(A(Qs(=)) m|Qs(=) = Qs(zo)
Pmmm - 1[0s(z))] Pmmm = 1[Qs(zo)). Thus, the maximum equals to [mm - 1|Qs(=o)) - Pmmm = - -10)s=zo) = AdvI, 20).
Finally we show 4. By the post-processing property, the e-differential privacy indicates that the query output satisfies for any record z, we have
P(Qs(z))m= 1) PPQs((2)m = 1]
log
S
(22)
Then 4 directly follows from 1, 2 and the fact that
x 1
tanhl ln(z)).
(23)
0
D.4 Proof of Theorem 2
Proof. By the law of large numbers, we have f3 E ry and 9J 2 qj. Then 1 follows from Slutsky's theorem
(Corollary 1 and pr,
+ (1
p)q 0. To prove 2, we first derive the (1 1- 5/2) confidence intervals of ry and
Øj by Clopper & Pearson (934). Then we could divide the nominator and the denominator by qj and 2
follows from the fact. that 2+1 is monotonically increasing function for x > 0.
D.5 Proof of Theorem 3
Proof. By Chen 2017); we have fx (x)()z)) 2 r(Qs(=)) and 'v (m(()(=)) 2, g(Qs(=)). Then follows from Slutsky's theorem (Corollary D and pr(Qs(2) + (1 p)s()s(z) > 0. To prove 2, we first derive the confidence intervals of r(Qs(z)) and (=)s(=)) by Chen (2017) Then we could divide the nominator and the denominator by q(Qs(2)), and follows from the fact that s-1 is a monotonically increasing function for x+1
0.
22

---

Published in Transactions on Machine Learning Research (10/2022)
D.6
Proof of Theorem 4
Proof. Define
r(Qs(zo)) ((Qss)))=(((()))m=m==))))))))) ((Qs(z))) = P(Qs(z)))== I(Qs(z))) = I ([0=)mmm1100(00)) te).
We will first show that
+ +2(1 prtp
AdvI,p pdir(Qs(2))) de(1 potp://00000oaaaaaaaaaaaaooaaaa0
(24)
Note that the individual privacy risk at sample is written as
Advli.((ss: 20) do 01 + + ((oP((())))))))))))))))))))))))) + an FN(((((()) + Aoo (2))s
(25)
bo + TP + (zooP(Qs(z))) + b0s FN((2(()())) (Nos
where du, bu, dij and by are pre-defined scalars for 0, 1 and j = 0, as in Equation TP(Qs(z)))
FP(Qs(zo)), FN((s(z0)) and (N))s(zo)) are the conditional versions of TP, FP, FN and TN on Qs(z)
Qs(zo). The four terms be written
TP(Qs(20) F(A)(Q5()) 1,m 1(Qs(z)))
FP(Qs(2)) = ((Ar(Qs(2)) 1,m m = 1..m==11Qssaam 1(Qs(20)).
FN((s(20)) = ((dr(Qs())) = -11,
1,m = 1(Qs(2))),
= - TN(Qs(=0)) = P(A4" (Qs(z) = =-
-100(=)))
By
FP(Qs(za)) = P(m = -1(Qs(za)) - TN(Qs(20)) & FN((s(=0)) P(m 1[Qs(z)))) TP(Qs(z))),
Equation 25 can be re-written as
0o + = 10 (011 ( + (400 - (1o))(((((((((
bn Pm 0
bb P(m 1[Qs(=o)) (b,
(boo - b1o) (No)s)(()
(26)
We first re-write
TP(Qs(20)) P(m -1)0(=100()))(mmm 1[Qs(z0)) tτ)
and
TN(Qs(zo)) Pmm =1m==](((()))mmmmmm -000)((mmmm10((0)))) <b)
by plug-in Equation 16
Then, by
[[Qs())=Qs(z))))) 1)p
P(m 1(Qs(z)))
(27)
and
P(m = -1Qs=1((s(aa)) = P(Qs(z)= = 11 {m=
we have Equation 26 equal to
+ c2 ora
+ + (x(_1l1))(((()))))))))))))) ||tp://ohaaaaaaooo/////////////0
Hence, have proved Equation 24
23

---

Published in Transactions on Machine Learning Research (10/2022)
In the next step, will show the consistency of the proposed estimator. Note that, have f(Qs(zo)) &
r(Qs(zo)), ((Qs(za)) & ((Qs(z))) by the proof of Theorems and By Equation 27
pr(Qs(z0))
I(Qs(z))) = pr(QQs=o)) ppr((((2=))))
(1 p)g(Qs(z0)) te
I[(1 te)pr(Qs(z0)) te(1 p)g((s(z))))
It is function of and q, and it is continuous except when P(m 1[Qs(z0)) ty. By the second condition, we have P(m (z))( o lp. Thus, ((11-))p) > 41 - () 2 I(Qs(z))) follows from the continuous
mapping theorem (Theorem
By applying Slutsky's theorem (Corollary we show that the nominator of Equation 18 converges probability lo the nominator in Equation 24 Similarly, we can prove that the denominator Equation [8] converges in probability to the denominator in Equation 24 Because of the third condition, the denominator
in Equation 24 is nonzero. Thus, 1 follows by Slutsky's theorem (Corollary D
Finally, we observe that given (1 11-6 8/2))cooniieence intervals for P(Qs(=)|m= = 1), (=)s(=)lm== = -1), and the independence ((Qs(=))m = 1) and P(Qs(=))m = -1), the joint (1 confidncee interval of ((Qs(=) 1) and (P(Qs(z) = -1) is simply the union of the two (using union bound), and follows.
D.7 Proof of Theorem 5
Proof. First note that
A (Qs)s) = Ez]5x(Z)
P(Qs(Z)|M= = -1p -1p-P(Qs(2))M) = -1)(1 p)
= Ez
P(Qs(Z))
= Σ (Z) -IM 1)p PQQs(Z) = =JIM -1) p)
JEQ
It is sufficient to show that Vj 2
Ni
N₂ I(Qs(M)) - -1) 2 (1((s((2)= = jIM = (()s()) jjM = -1)(1 -)]1-
((ss(x)) 5) 1-
Σ
N₂
(28)
and then (1) follows by Slutsky's Theorem (Corollary
To prove Equation
28
we first show that
N;
(Qs(X:)))) 2, ( jIM =1),
N
1
I(Qs(Y)) =) 3, P(Qs(Z) -JMM -1)
N₂
by the law of large numbers. Then Equation (Theorem 11
28 follows by applying the continuous mapping theorem
Next, we prove (2). It is easy to verify that Vi E (1, ,N) Ver." ,", EN V1. ... UNN. 24 E 2, we always have
2p
W ,,,,, " ,N,, U1, ,UN) W (Z , " ,N, U1. .U.,,, w S
N₂ N
Similarly, for Vi E (1, N2) and
"..,""., CN,, y1, Vi. ..... UNa. Di 2, we always have
24

---

Published in Transactions on Machine Learning Research (10/2022)
W( N 31."" " ,NN,) W( ((1, ,N,, .N, M1 .b.. .... 2(1 - 2
N2 N
Then by McDiarmid's inequality (Theorem 13 P( EWwl 2 1) S 2exp (- Let 8 = exx (- N and we have (2).
D.8 Proof of Theorem 6
Proof. (1) We first prove the consistency. Note that
(As)s)
[Z))
P(Qs(Z))M = 1)p PD)s((() - 11((1 p)
Ez
P(Qs(Z))
| - g(x)(1 p}] dr,
where r(z) = =() = x/m= = 1) and g(x) = (z) = z/m = -1).
Rewrite UN as
UN = /
P
N
K(
(Xi)
1-p
Σ
N₂
K(
x. - Q.((Y.)
)dx
ha
h
Nohd
h
1=1
1=1
/
N₂
(x( -(())
[g(x) (alz)
1
N₂
Qs dr.
K(
+ (pr((x) - (1--100 (1)p(((z)) + p)
K
Nihd
h
Nohd
i=1
i=1
Then
UN - Ar(ss) S 1
1
K Qx(())) -((s)
(1 pp) (((x) -p)(a(a))
Nghd Σ K( Qs(Vi)
dx
S p /
Ni
(x())) r(0))
do + do+ p) / Nah
(((x))
N₂
Qs(Yi) da
K(
K
Naha
i=1
2 0.
where the last step follows from Theorem 12 and Slutsky's theorem (Corollary 1
(2) It is casy to verify that Vi E .Ni. ;
Vzi, Fy, INN.81 V1. MN.. E Z, we always have
2p N, N
U0 . NN W1. .NN)) U'x,,' ". ... ,N,, W1. "...",,
.NNN)
Similarly, for Vi E {1,-.. N2} and
Vel."" NN: M1, .. Ubs ... UNa: Vi 2, we always have
2(1 -1--) pp)
2
U .N, 0..... Ue. UN.) Ux1.......
.N, 0/. ....... .NN.) 5
N₂ N
Then by McDiarmid's inequality (Theorem 13 P(UN and we have (2).
EUN 2 t) s 2exp (- P).. Let 5 2exp (- 22)
25

---

Published in Transactions on Machine Learning Research (10/2022)
D.9 Proof of Theorem 7
Proof. First, would bound Adve,p(9)) Advep(Qs, A). By the definition of ED, Qs), we have
P(m 11 1) P(m 1, (.A((s(=)))
-1)= =
and
P(m -1m== = + P(m = -1, (2)
= -1) =1 11
Thus, we could rewrite the membership advantage under the generalized metric Adve, (Qs; A) as follows:
ag + 0i a1ip + u0o(1 - p) (a11 a01) = -, A(Qs((())) = -1) - (400 - 410)PP(m - (1,)(Q((=)) =1)
bo + bui + boo - p)
Define c == (000 a10)/ 410 + a11 a01), 3o (ag + app + a00 p)) (bo + +b1n++ +000+ pp)), and
81 = (Oou - a1o + 011 - -a01) (bo + b11 + buu (1 - (1)))) We have
(Qv A) = 30
= -1,A((Qs()) = =)) - 1, = -1)]. (29)
Because doo aıo and d1 401, (0, and 31 > 0. Define θ = (A: A(Qs(z)) sgn
sgn (d(Qs(z)) to)
R + [0, 1]}. Next, we would bound (Qs) Adve, A).
A (Qs) Adve, (Qs, = max Advz (Qs, Adv , (Qs.
A
max Adve, A) Adve, Ace
(30a)
= 81 cP(m -1, A(Qs(z)) = 1) + (1 - cm A(Qs(()))) (1,00()= =
-1] --1)
81 inf [cP(m 11, -1, = =) +1 (11-c)(((m 1, --1] (30b)
I B ((ocs)(((((s) m(Qs(=)) -
(30c)
The step in Equation 30a follows from Lemma
2
The step in Equation 30b follows from Equation 29 and
B1 > O. The step in Equation 30c follows from B 0, E (0, 1) and Lemma. 4 (Menon et al. 2013)
Note
that AQve,p(Ss.A) < (Qve,,(Qs)) Hence we have (Qv E (Qs)..
E Model architectures and hyper-parameters
Here we outline the different layers used in the model architectures for different datasets. The last layers of discrininators for WGAN experiments do not have sigmoid activation functions. The hyper-parameters are chosen the same same [Goodfellow et al. 2014 Gulrajani et al. 2017
E.1 MNIST
E.1.1
Generator layers
Dense(units= 512, input size= 100)
LeakyReLU(a= = 0.2)
Dense(units= 512)
LeakyReLU(a= 0.2)
Dense(units= 1024)
LeakyReL.UR 0.2)
Dense(units= 784, activation 'tanh")
26

---

Published in Transactions on Machine Learning Research (10/2022)
E.1.2
Discriminator layers
Dense(units= 2048)
Lea 0.2)
Dense(units= 512)
Lea 0.2)
Dense(units- 256)
Lea = 0.2)
Dense(units= 1, activation = 'sigmoid')
E.2
CIFAR-10
E.2.1
Generator layers
Dense(units=2 x 2 x 512)
Reshape(target shape= 22,2,)12)
(iur: 128, kernel size= 4, strides= 1)
ReLU(
Conv22Trrassooe(lllees-- 64, kernel size= 4, strides= 2, padding- 1)
ReLU()
(iur22rrraasspose(llltees=- 32, kernel size= 4, strides= 2, padding- 1)
ReLU()
Crnn22Dransposs(ltttssss-- 3, kernel size= 4, strides= 2, 2,padding- 1, activation = 'tanh')
E.2.2
Discriminator layers
Conv2D(fiters= 64, kernel size= 5, strides= 2)
Cony2D(fiters= 128, kernel size= 5, strides=
LeakyReLU(a= 0.2)
Conv2D(fiters= 128, kernel size= 5, strides= 2)
LeakyReLU(a= 0.2)
ConveD(fiters= 256, kernel size= 5, strides= 2)
LeakyReLU(a= 0.2)
Densc(units= 1, activation 'sigmoid')
E.3
Skin-cancer MNIST
E.3.1
Generator layers
x x 5122
Reshape(target shape= (4, 4, 512))
(any2DTTannsssseitteeee==== 256, kernel size= 5, strides= 2)
ReLU()
(fnv22DTaansossslllllles=== 128, kernel size= strides= ReI.U()
Conv2DTranssoss(lllesss 64, kernel size= 5, strides= 2) ReLUU)
Sonv2DTranspose((( 3, kernel size= 5, strides= 2,activation 'tanh')
27

---

Published in Transactions on Machine Learning Research (10/2022)
E.3.2
Discriminator layers
CfnteD(filters= 64, kernel size= 5, strides= 2)
Conv2D(filters= 128, kernel size= 5, strides=
LeakyReLU(a= 0.2)
ConveD(filters= 128, kernel size= 5, strides= 2)
LeakyRel.U(a= 0.2)
Conv2D(filters= 256, kernel size= 5, strides= 2)
LeakyReLU(a= 0.2)
Dense(units= 1, activation = 'sigmoid')
E.3.3 Toy binary classifier layers
(fiteD(filter== 512, kernel size= 5, strides= 2)
LoakyReLU(a= 0.2)
Denso(units= 2, activation 'relu')
Dense(units= 1, activation = 'sigmoid')
F
Auxiliary lemmas and theorems
Theorem 10 (Slutsky's theorem). Let Xn 4 X and Yn 4 where e is constant. Then
Xn 4 X +
X, 4 cX;
3. Xn/Yn 4 X/e if c
Corollary 1 (Slutsky's theorem). Let X, 2 ca and Yn 8g c1, where co and ei constants. Then
1. X, + Yn E co c1
2. X,Yn 2 coc1;
3. Xn/Yn 2 co/c1 if a # 0.
Proof. When is constant, Zn 2. is equivalent to Zn 4 Thus have Xn 4 co and Y, 4 Cı. By applying Theorem 10 (1), we have
Xn + Yn 4 cu C1.
Since + ej is constant, it follows
Xn + Yn & co C1.
Similarly, we can prove 2 and
Theorem 11 (Continuous mapping theorem). Let f Rm + R be a measuarable function. Define
C fx: : f is continuous at x}.
If Xn. & X and P(X E Cy) 1, then
((X,) 2s S(X).
28

---

Published in Transactions on Machine Learning Research (10/2022)
Theorem 12 Devroye & Gyorfi 1985
Let pn be automotic kernel estimate with arbitary density K, defined in Equation 17). If + (nha)- - 0 completely (almost surely, in probability), then J |P p) -> 0
falmost surely, in probability), for all density on IR.
Theorem 13 (MeDiarmid's inequality). Let f 2 + IR be function satisfying
[f(21,... Ri
,,,,, ;i,...,,.m) ) [(21,.... ((=,,....2.... ; - Ci
(Vi, V21 ... Zm, 2q 2).
Denole
m
Σ
Let -1,... Zm be independent variables with support 2. Then
P(J(Z,,... ( Zm)) 2 exx((11((2)))
and
PP -mm -m) -E]((Z,, Zm)i S -t) S exp (- (-12/(2))
Lemma 4 (Chen 2017
With probability (1 - 8/2), we have
r(z) E [Fooer (1)), Pupper (x)]
(31)
where
PK (N(
Flower (x) = FN(()) 2L-8/4
Nhd
(x(
Pupper (x)) = FN (x) + #1-6/41
Nhd
and pk: == J (2(x)de, 21-6/4 is the (1 61-6/4) quantile of a slandard normal distribution.
G Connection to differential privacy
As shown in Theorem 1 both the optimal membership advantage and the individual privacy risk are bounded by differential privacy guarantees. We construct a toy dataset from MNIST by forming new imbalanced dataset with 6900 digit zeros and 700 digit sixes. This dataset is also used for anomaly detection [Bandaragdda et al.) 2014). We set p = 0.5 for simplicity. Figure [ shows the optimal membership advantage of the DP-cGAN (Torkzadehmahani et al. (219)) with different choices of the privacy budget e. As we can see here, the theoretic upper bound given by tanh(e/2) much larger than the estimated optimal membership advantage. Figure shows the individual privacy risks for both 2 and highest individual privacy risk is strictly bounded by the upper bound derived from the privacy budget The upper bound (fanh(?)= = 0.762 for e = 2, and tanh( ( = 0.999 for E = 10). These demonstrations scem to indicate that if membership privacy is desired, using differentially private methods can lead to far too conservative models (which may lead to poorer model utility). However, it may also be that the privacy accounting in CP-NGAN loose, thereby leading to an overestimation of
10. As expected, even the
The optimal membership advantage as a function of synthetic dataset size
To explore the effect the size of synthetic datasets on the membership advantage, we first. trained a JS-GAN on the skin-cancer MNIST dataset. The JS-GAN was trained for 2000 epochs and several synthetic datasets of sizes varying from 10 10 samples was generated. The adversary used was the one described in Equation It can be seen in Figure that as the synthetic dataset size increases, so does the optimal membership advantage.
29

---

Published in Transactions on Machine Learning Research (10/2022)
Membership Advantage
1.0
0.6
advantage
upper bound
0.2
(s, 10 -e,1105)-diffeeeee differrenti privacy
Figure 5: The optimal membership advantage and its upper bound estimated versus the privacy budget for DP-cGANs.
Fraction
Fraction
Individual privacy risk
Individual privacy risk
(a)
(b)
Figure 6: The individual privacy risks for DP-cGAN with privacy budget (e, 10-9) (a) individual privacy risks for E = 2 (b)individual privacy risks for 10.
Estimation of optimal membership advantage for discriminative models
To demonstrate that our estimators of optimal membership advantage also applicable to discrininative models, trained simple binary classifier (architecture described in section E.3.3) the skin-cancer MNIST dataset (same experimental sctting as the generative model). We chose two queries: query the final output of the classifier, ii) a white-box query the output of the penultimate layer of the classifier. We discretized the outputs and used our discrete estimator to estimate the optimal membership advantage in each case, as seen in Figure As expected, the white-box query has a higher optimal membership advantage than the black-box query.
black-box
Membership privacy vs Synthetic dataset size
0.300
Membership Advantage
0.755
0250
0125
0.00
2.5
3.0
3.5
50
Synthetic dataset size (10*x)
Figure 7: The optimal membership advantage the synthetic dataset size.
30

---

Published in Transactions on Machine Learning Research (10/2022)
Attacks against discriminative models
Membership Advantage
Optimal adversary - Black-box
0.20
Optimal adversary White
010
0.00
100
200
300
400
Epochs
Figure 8: Estimation of the optimal membership advantage with white-box and black-box queries against binary classification model on skin-cancer MNIST.
31


Reviews:
[{"review_id": "C14xuNOOcZ", "reviewer": "Reviewer_6r1B", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed framework is very general and can be used to estimate privacy risk under different assumptions, i.e., releasing a model or synthetic dataset.\n2. The proposed method takes into account the unbalanced membership in the data.\n3. It can be used to estimate individual-level risk.\n4. Experiments show the effectiveness of the proposed framework.\n\nWeakness:\n1. The motivation of the proposed query function is not clear.\n2. The accuracy based metric need to be further explained.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the problem of membership privacy estimation in generative models. More specifically, the authors propose a framework to estimate the maximum membership privacy risk for generative models. The proposed framework is more general than the previous heuristic methods and is able to characterize the individual-level membership privacy risk. Experiments validate the advantage of the proposed method.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "My main concerns of the current paper are as follows:\n1. What is the motivation of your proposed query function in equation (2)? What is the advantage of the proposed query function compared with the previous one?\n2. Is there a scaling issue in equation (4)? What is the output range of $D$ and $d$?\n3. In section 3.1, the Adv definition in equation (1) is based on $p=0.5$ in Experiment 1. Why can it be used to derive the proposed optimal advantage since you are considering a general $p$?\n4. In Experiment 2, your method also needs access to the distribution D, and why is this better than the previous work, e.g., Yeom et al. and Jayaraman et al.?\n", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "g1yflMk42a", "reviewer": "Reviewer_3Yd4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The entire use of the framework allows the authors to look at different studies of membership inference attacks using the same lens. Furthermore, it allows them to go beyond the previous studies by considering individual level privacy risks as well as when one does not have access to the entire training data; the latter makes it a more practical attack because it is not often the case that we have access to te entire training data. \n\nI have a few questions. Reading through your definition of the adversarial model, it seems like it is not restricted to GAN. It resembles a lot of what we usually see in cryptography. In cryptography, one would usually write such definitions as an adversarial game where the adversary is given auxiliary inputs. From the definition of the paper, I do not see that to be the case. Is it by choice? If not, then clarify. If there is auxiliary input available to the adversary, then mention it. Also, is the adversary passive or active, adaptive, or non-adaptive, honest-but-curious or malicious? None of that is clear in the paper. \n\nFor the weakness, I found the writing (especially, the proofs) very sloppy. This makes it hard to verify the correctness of the paper. A journal paper should ideally have a much higher editor quality than a conference paper and I feel the paper falls short of that. I will use this opportunity to give some suggestions so that the authors can submit a version that I can verify before pointing to specific instances:\n1. Please be consistent in the use of different citation techniques available in natbib. If you are saying, Following Chen et al. 2018, maybe, use the one that does not put parenthesis to be consistent with the later usage. \n2. If you are using two or more items, then separate them with a conjunction. Like end of page 3, (1) <text>; and (2).\n3. Use space between phrases and citations. Like, Hayes et al., 2019 assumes. assumes -> assume.\n4. Change the phrase We then to We next or something like that. It is a very repetitive choice of phrase. \n5. Please use active voice instead of passive voice. One example is line 3 and 6 in the description of Algorithm 1. \n\nSome specific instances:\n1. After Definition 3. we -> We\n2. Do not start a new sentence with a conjunction. A specific example is right after equation (17).\n3. What is N in theorem statements? Define its value. I am guessing it is N=N_1+N_2 from Experiment 2. A theorem statement should be self-contained. \n4. Which theorem belongs to Algorithm 1?\n5. Algorithm 2: the line numbers are missing.  \n6. Appendix D.4: What is p? Is p(\\cdot) a function? It seems like the authors have overloaded the symbol p and it makes the reviewer's job very difficult. The same issue is in Appendix D.4.\n7. AdvI should be in math font to be consistent. \n8. FP, etc are said to be that they are conditional forms, and the expression is given later. I would rather have the mathematical expression first before the description of what it means. \n9. Proof of Theorem 4: I would elaborate on all the steps in the proof. There is no page limit and every mathematical expression should be spelled out. It might be helpful to readers if the authors do not write the mathematical expression as a paragraph but in the math environment. \n10. \\hat p tends in probability to the expression is only used in Theorem 2's proof unless the authors have overloaded p_N to denote p.\n11. Please use a preliminaries section to state the version of Slutsky's theorem and the continuous mapping theorem that you use. It is very important given the repeated use of the aforementioned theorems. \n12. Elaborate on how 1 follows by Slutsky theorem and the third condition. \n13. Why is 2p/N_1 = 2/N? Again, what is N? \n14. I cannot find Theorem 7. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The main contribution of the paper is to develop a framework to estimate the risk of membership inference attacks when the adversary is given oracle access to the model. The framework in the paper is general in the sense that it extends to general risk class considered in previous works and can measure individual level membership privacy risks, ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Please take a look at the weakness section above. ", "broader_impact_concerns": "N/!", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sRJNMYxrnX", "reviewer": "Reviewer_p24z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. The paper is well-written and organized, and it clearly stated its contribution and differences to prior works.\n2. The paper is technically sound where the proposed theoretical framework for membership privacy provides a convincing lower bound of risks. \n3. The framework is very flexible and can be used for multiple settings and purposes. The individual privacy risk estimation is particularly important for us to understand the privacy of the vulnerable samples.\n4. The empirical evaluation demonstrated its usability and how the framework provides a tighter privacy estimation than heuristic adversaries.\n\nWeaknesses\n\n1. The datasets and models in the experiments are all small scaled. This is acceptable given the main contributions in the paper are in the theoretical aspects. \n2. The individual risk estimation is very interesting and potentially important, however I did not find the discussion or experiments on this to be illustrative enough. Perhaps a distribution of the individual risks or more examples of the vulnerable samples would be helpful to highlight their characteristics.  Also the observation is closely related to “memorization” in neural networks discussed in Feldman 2020, which is not cited. \n3. Although the authors stated the connection to differential privacy in Theorem 1, there are no empirical experiments showing this connection. It would be very helpful to show a plot similar to Figure 1 in Erlingsson et al 2020. \n\nReferences:\n1. Ulfar Erlingsson, Ilya Mironov, Ananth Raghunathan, Shuang Song. That which we call private. 2020.\n2. Vitaly Feldman. Does Learning Require Memorization? A Short Tale about a Long Tail. 2020.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a framework for estimating membership privacy in generative models (e.g. GANs). The paper shared some of the setups with prior works such as Yeom et al 2018, and extended with more flexible assumptions to be applied in more general settings. Specifically this paper addresses the imbalance issue in prior membership privacy frameworks by utilizing a general metric class studied in Koyejo et al 2014. Then the authors made a novel connection between individual-level and population-level membership privacy. Finally the authors proposed another setting with looser assumptions than Yeom et al where only a random sample is accessible instead of the data distribution and provided practical estimators for membership risks  in this setting. The framework was evaluated on three benchmark image datasets (MNIST, CIFAR10 and skin-cancer MNIST) and on different GAN architectures. \n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. In Algorithm 2, the authors proposed to estimate $\\eta_(Q_S(z))$ by the first set of samples, while no further explanation is provided. Would appreciate the author to elaborate more on this estimation or provide a reference.\n2. In Section 6.2.1, the authors stated “our optimal membership advantage estimates are higher than the SOTA heuristic adversaries in both settings.” However, I could only find comparison with SOTA heuristics in Figure 1a for 1 setting instead of both settings. Would appreciate the authors to complete the plots for the statement.\n3. In Section 6.2.2, the authors stated “In this case, the performance of the optimal adversary is consistently better than the heuristic adversary.” This is the case for the MNIST AM metric plot but not very obvious in the CIFAR10 plots where the two curves overlap from epoch 5,000 or so. Would appreciate the authors to reword or explain the CIFAR10 plot in more detail.\n4. In Figure 2a, why does the 2d query curve (red) fluctuate regularly? E.g. membership advantage could drop from 0.8 to 0.3. Would appreciate authors to explain this behavior.\n5. Seems like for one comparison setting, one model and one dataset are used. The paper could be strengthened by showing the comparison on all models and all datasets. \n\nMinor and questions:\n1. In Section 2.4, An adversary -> an adversary\n2. All the plots in Section 6 are small. Bigger plots might be easier to read.\n3. In algorithm 2, do the three sampled sets need to be disjoint? How would this impact the estimation of optimal membership advantage?\n4. As the authors acknowledged in the conclusion that the framework could be extended to discriminative models. I also did not find that the framework specific to generative models, yet the title and motivation all focused on generative models. Why not emphasize that the framework is suitable for both discriminative and generative models, which covers a wider range of ML and could be more impactful.\n", "broader_impact_concerns": "No concern on the ethical implications", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper makes a contribution in the area of estimating privacy of generative models. The reviewers initially had some feedback and comments on aspects of the work, which the authors addressed in an updated revision. Beyond this, there was little additional discussion among the reviewers, who felt their questions and concerns were addressed by the authors, and thus deemed this paper ready for publication.
Decision: Accept (poster)