Gaussian Membership Inference Privacy
Authors: Tobias Leemann
Martin Pawelczyk
Gjergji Kasneci
Venue: NeurIPS 2023 poster (2023)
Keywords: Privacy
Membership Inference Attacks
TL;DR: We propose a novel privacy notion termed $f$-Membership Inference Privacy and show how it can be achieved through noisy gradient descent.

Abstract: We propose a novel and practical privacy notion called $f$-Membership Inference Privacy ($f$-MIP), which explicitly considers the capabilities of realistic adversaries under the membership inference attack threat model.  Consequently, $f$-MIP offers interpretable privacy guarantees and improved utility (e.g., better classification accuracy). In particular, we derive a parametric family of $f$-MIP guarantees that we refer to as $\mu$-Gaussian Membership Inference Privacy ($\mu$-GMIP) by theoretically analyzing likelihood ratio-based membership inference attacks on stochastic gradient descent (SGD). Our analysis highlights that models trained with standard SGD already offer an elementary level of MIP.  Additionally, we show how $f$-MIP can be amplified by adding noise to gradient updates. Our analysis further yields an analytical membership inference attack that offers two distinct advantages over previous approaches. First, unlike existing state-of-the-art attacks that require training hundreds of shadow models, our attack does not require any shadow model.  Second, our analytical attack enables straightforward auditing of our privacy notion $f$-MIP. Finally, we quantify how various hyperparameters (e.g., batch size, number of model parameters) and specific data characteristics determine an attacker's ability to accurately infer a point's membership in the training set. We demonstrate the effectiveness of our method on models trained on vision and tabular datasets.

Gaussian Membership Inference Privacy
Tobias Leemann' University Tübingen Technical University of Munich
Martin Pawelczyk' Harvard University
Gjergji Kasneci
Technical University Munich
Abstract
We propose a novel and practical privacy notion called f-Membership Inference Privacy (f-MIP), which explicitly considers the capabilities of realistic adversaries under the membership inference attack threat model. Consequently, f-MIP offers interpretable privacy guarantees and improved utility (e.g., better classification accuracy). In particular, we derive parametric family of f-MIP guarantees that we refer to as j-Gaussian Membership Inference Privacy (μ-GMIP) by theoreti- cally analyzing likelihood daii- membership inference attacks stochastic gradient descent (SGD). Our analysis highlights that models trained with standard show how f-MIP
SGD already offer elementary level of MIP. Additionally, can be amplified by adding noise to gradient updates. Our analysis further yields analytical membership inference attack that offers two distinet advantages over previous approaches. First, unlike existing state-of-the of-t----rrtttttt attacks that require training hundreds of shadow models, our attack does not require any shadow model. Second, our analytical attack enables straightforward auditing of our privacy notion [-MIP. Finally, we quantify how various hyperparameters (e.g., batch size, number of model parameters) and specific data characteristics determine an attacker's 's abil- ity accurately infer point's 's membership in the training set. We demonstrate the effectiveness of our method on models trained on vision and tabular datasets.
Introduction
Machine learning (ML) has application across various domains. However, some of these domains, such as finance and healthcare, deal with sensitive data that cannot be publicly shared due ethical or regulatory concerns. Therefore, ensuring data privacy becomes crucial at every stage of the ML process, including model development and deployment. In particular, the trained model itself [5, 31] or explanations computed to make the model more interpretable [29, 32] may leak information about the training data if appropriate measures
surge in popularity and effectiveness, leading its widespread
taken. For example, this is problem for recent generative Diffusion Models [7] and Large Language models, where the data leakage seems to be amplified by model size [6].
Differential privacy (DP) [14] is widely acknowledged as the benchmark for ensuring provable privacy in academia and industry [10]. DP utilizes randomized algorithms during training and guarantees that the output of the algorithm will not be significantly influenced by the inclusion exclusion of any individual sample in the dataset. This provides information-theorrtiicc protection against the maximum amount of information that attacker extract about any specific sample in the dataset, even when an attacker has full access to and full knowledge about the predictive model.
While DP is an appealing technique for ensuring privacy, DP's broad theoretical guarantees often come at the expense of significant loss in utility for many MI algorithms. This utility loss cannot be further reduced by applying savvier algorithms: Recent work [26, 27] confirms that attacker can
"Equal contribution. Corresponding authors: tobias hwwpemaannnnnntuuieeeeee de and martin.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

be implemented whose empirical capacity to differentiate between neighboring datasets D and D' when having access to privatized models matches the theoretical upper bound. This finding suggests that to improve a model's utility, we need to take step back and inspect the premises underlying DP. For example, previous work has shown that privacy attacks are much weaker when one imposes additional realistic restrictions on the attacker's capabilities [26].
In light of these findings, we revisit the DP threat model and identify three characteristies of an attacker that might be overly restrictive in practice. First, DP grants the attacker full control over the dataset used in training including the capacity to poison all samples the dataset. For instance, DP's protection includes pathological cases such as an empty dataset and a dataset with single, adversarial instance [27]. Second, in many applications, is more likely that the attacker only has access to an API to obtain model predictions [13, 31] or to model gradients [20]. Finally, one may want to protect typical samples from the data distribution. As argued by Triastcyn & Faltings [38], may be over-constraining to protect images of dogs in model that is conceived and trained with images of cars.
Such more realistic attackers have been studied in the extensive literature on Membership Inference (MI) attacks (e.g.. [5, 41D, where the attacker attempts determine whether a sample from the data distribution was part of the training dataset. Under the MI threat model, Carlini et al. [5] observe that ML models with very lax (c > 5000) or no (c oo) DP- ou-guraatt still provide some defense against membership inference attacks [5, 41]. Hence, we hypothesize that standard ML models trained with low or no noise injection may already offer some level of protection against realistic threats such as MI, despite resulting in very large provable DP bounds.
To build solid groundwork for our analysis, we present hypothesis testing interpretation of MI attacks. We then derive f-Membership Inference Privacy (f-MIP), which bounds the trade-off between an MI attacker's false positive rate (i.e., FPR, type errors) and false negative rate (i.e., FNR, type II errors) in the hypothesis testing problem by some function f. We then analyze the privacy leakage of a gradient update step in stochastic gradient descent (SGD) and derive the first analytically optimal attack based on a likelihood ratio test. However, for f-MIP to cover practical scenarios, post-processinn and composition operations need to be equipped with tractable privacy guarantees as well. Using f-MIP's handy composition properties, we analyze full model training via SGD and derive explicit f-MIP guarantees. We further extend our analysis by adding carefully calibrated noise to the SGD updates to show that f-MIP may be guaranteed without any noise or with less noise than the same parametric level of f-DP [11], leading to a smaller loss of utility.
Our analysis comes with a variety of novel insights: We confirm our hypothesis that, unlike for DP, no noise (72 0) needs be added during SGD guarantee f-MIP. Specifically, we prove that the trade-off curves of single SGD step converge to the family of Gaussian trade-off's identified by Dong al. [11] and result in the more specific μ G-Gusssian Membership Inference Privacy (J-GMIP). The main contributions this research offers to the literature on privacy preserving ML include:
1. Interpretable and practical privacy notion: We suggest the novel privacy notion of fMMIP that addresses the realistic threat of MI attacks. f-MIP considers the MI attacker's 's full trade-off curve between false positives and false negatives. Unlike competing notions, f-MIP offers appealing composition and post-processing properties.
2. Comprehensive theoretical analysis: We provide (tight) upper bounds on any attacker's ability to run successful MI attacks, i.c., we bound any MI attacker's ability to identify whether points belong to the training set when ML models are trained via gradient updates.
3. Verification and auditing through novel attacks: As side product of theoretical analysis, which leverages the Neyman-Pearson lemma, we propose a novel set of attacks for auditing privacy leakages. An important advantage of our analytical Gradient Likelihood- Ratio (GLiR) attack is its computational efficiency. Unlike existing attacks that rely on training hundreds of shadow models to approximate the likelihood ratio, our attack does not require any additional training steps.
4. Privacy amplification through noise addition: Finally, our analysis shows how one can use noisy SGD (also known as Differentially Private SGD (1]) to reach fMMIP while maintaining worst-case DP guarantees. Thereby our work establishes a theoretical connection between [MIP and f-DP 1], which allows to translate an fDDP guarantee into an fMPP guarantee and vice versa.

---

2 Related Work
Privacy notions. DP and its variants provide robust, information-theoreti privacy guarantees by ensuring that the probability distribution of an algorithm's 's output remains stable even when one sample of the input dataset is changed [ [14]. For instance, a DP algorithm is e-DP if the probability of the algorithm outputting a particular subset E for a dataset S is not much higher than the probability of outputting E' for a dataset So that differs from S in only one element. DP has several appealing features, such as the ability to combine DP algorithms without sacrificing guarantees.
A few recent works have proposed to carefully relax the attacker's capabilities in order to achieve higher utility from private predictions [4, 13, 17, 38.. For example, Dwork Feldman [13] suggest prediction" to make private model predictions through an API interface. Their work focuses on PAC learning guarantees of any class of Boolean functions. Similarly, Triastcyn & Faltings [38] suggest "Bayesian DP", which is primarily based on the definition of DP, but restricts the points in which the datasets S and So may differ to those sampled from the data distribution. In contrast, Izzo et al. [17] introduces notion based on MI attacks, where their approach guarantees that an adversary A does not gain a significant advantage in terms of accuracy when distinguishing whether an element was in the training data set compared to just guessing the most likely option. However, they only constrain the accuracy of the attacker, while we argue that is essential to bound the entire trade-off curve, particularly in the low FPR regime, to prevent certain re-identification of a few individuals [5]. Our work leverages a hypothesis testing formulation that covers the entire trade-off curve thereby offering protection also to the most vulnerable individuals. Additionally, our privacy notion maintains desirable properties such as composition and privacy amplification through subsampling. which previous notions did not consider.
the notion of pprvacc-preserving
Privacy Attacks on ML Models. Our work is also related to auditing privacy Icakages through a common class of attacks called MI attacks. These attacks determine if a given instance is present in the training data of a particular model [5, 7, 9, 15, 23, 29, 30, 31. 32, 35, 36, 40, 41]. Compared to these works, work suggests new much stronger class of MI attacks that is analytically derived and uses information from model gradients. An important advantage of our analytically derived
attack is its computational efficiency, as it eliminates the need to train any additional shadow models.
3 Preliminaries
The classical notion of (c, dddfffeerettial privacy [14] is the current workhorse of private ML and can be described as follows: An algorithm is DP if for any two neighboring datasets S, S" (that differ by one instance) and any subset of possible outputs, the ratio of the probabilities that the algorithm's 's output lies in the subset for inputs S, S' is bounded by a constant factor. DP is rigid guarantee, that covers every pair of datasets S and S", including pathologically crafted datasets (for instance, Nasr et al. [27] use empty dataset) that might be unrealistic in practice. For this reason, consider different attack model in this work: The MI game [41]. This attack mechanism on ML models follows the goal of inferring individual's 's membership in the training set of a learned ML model. We will formulate this problem using the language of hypothesis testing and rrade-off functions, a concept from hypothesis testing theory [11]. We will close this section by giving several useful properties of trade-off functions which we leverage in our main theoretical results presented in Sections 4 and 5.
3.1 Membership Inference Attacks
The overarching goal of this end, will show that alternative notion of privacy can be defined through the success of a MI attack which attempts to infer whether given instance was present in the training set or not. Following Yeom et al. [41] we define the standard MI experiment as follows:
machine lcarning lies in protecting personal data. To
Definition 3.1 (Membership Inference Experiment [41]). Let A be an attacker, be a learning algorithm,, N be positive integer, and 'D be distribution over data points æ E D, where the vector x may also be a tuple of data and labels. The MI experiment proceeds as follows: The model and data owner O sumples S ~ Dx (iee sumple n points i.i.d. from D) and trains As A(S). They choose be E (0, 1} uniformly at random and draw x' DD b = O, or ae S if = 1. Finally, the attacker is successful if A(x', As. N, D) b. A must output either 0 or 1.
3

---

We note that the membership inference threat model features several key differences to the threat model underlying DP, which are listed in Table 1. Most notably, in MI attacks. the datasets are sampled from the distribution D, whereas DP protects all datasets. This corresponds to granting the attacker the capacity of full dataset manipulation. Therefore, the MI attack threat model is sensible in cases where the attacker cannot manipulate the dataset through injection of malicious samples also called "canaries". This may be realistic for financial and healthcare applications, where the data is often collected from actual events (e.g.. past trades) or only a handful of people (trusted hospital staff) have access to the records. In such scenarios, it might be overly restrictive to protect against worst-case canary attacks as attackers cannot freely inject arbitrary records into the training datasets. Furthermore, MI attacks are handy as a fundamental ingredient in crafting data extraction attacks [6]. Hence we expect privacy notion based on the MI threat model to offer protection against broader class of reconstruction attacks. Finally, being an established threat in the literature [5, 9, 31, 40, 41) ]. MI can be audited through variety of existing attacks.
[-DP threat model
f-MIP threat model (this work)
Goal
Distinguish between S and S' for any S' that differ in most one instance.
Distinguish whether or' S (training data set) or not.
Dataset access
Attacker has full data access. For example, the attacker can poison or adversarially con- struct datasets on which ML models could be trained; e.g. S = {} and S' {10").
Attacker has no access to the training data set; i.e., the model owner privately trains their model free of adversarially poisoned samples.
Protected Instances
The instance in which S and S' differ is arbitrary. This includes OOD samples and extreme outliers.
The sample x' for which memhership is to be inferred is drawn from the data distribu- tion D. Therefore, MI concerned with typical samples that can occur in practice.
Best used
When the specific attack model is unknown. Offers form of general protection.
When datasct access (e.g. canary injection) of attacker can ruled and the main attack goal lies revealing private training data (e.g., membership inference, data re- construction).
Model knowledge
The attacker knows the model architecture and has full access to the model in form of its parameters, hyperparameters and its model outputs.
Table 1: Comparing the threat models underlying f-DP and f-MIP.
3.2 Membership Inference Privacy as a Hypothesis Testing Problem
While DP has been studied through the perspective of a hypothesis testing formulation for while [3, 11, 19, 39], we adapt this route to formulate membership inference attacks. To this end, consider the following hypothesis test:
Hn a & S vs. H1 a S.
Rejecting the null hypothesis corresponds to detecting the presence of the individual x' in S, whereas failing to reject the null hypothesis means inferring that æ' was not part of the dataset S. The formulation (1) is a natural vehicle to think about any attacker's capabilities in detecting members of train set in terms of false positive and true positive rates. The motivation behind these measures is that the attacker wants to reliably identify the subset of data points belonging to the training set (i.e., true positives) while incurring as few false positive errors possible [5]. In other words, the attacker wants to maximize their true positive rate at any chosen and ideally low false positive rate (e.g., 0.001). From this perspective, the formulation in (1) allows to define membership inference privacy via trade-off functions f which exactly characterize the relation of false negative rates (i.c., 1-TPR) and false positive rates that an optimal attacker can achieve.
Definition 3.2. (Trade-off function [11]) For any two probability distributions P and Q on the same space, denote the trade-off function Test( P: Q)= [0:1] [0; 1]
Test(P: (a) = inf (FNR) FPR =a
(2)
where the infinum is taken over all (measurable) rejection rules ("tests") which lead to a FPR of a between distributions P, Q.

---

Not every function makes for a valid trade-off function. Instead, rrae-ooff functions possess certain characteristics that are handy in their analysis.
Definition 3.3 (Characterization of trade-off functions [II]). A function f [0. + [0. 1 is fractioffuumctinn if f is convex. continuous at zero, non-increasing, and f(r) S for E [0, 1|.
We additionally introduce a semi-order on the space of trade-off functions to make statements on the hardness of different trade-offss in relation each other.
Definition 3.4 (Comparing trade-offs). A trade-off off function f is uniformly at least as hard another trade-off function 9. if ((r) 2 g(r) for all O I r r 1. We write f 2 g-
If Test( P: Q) 2 Test(P", Q'), testing P vs Q is uniformly at least as hard as testing P' vs Q' Intuitively, this means that for a given FPR o, the best test possible test on (P; Q) will result in an equal or higher FNR than the best test on (P': Q').
3.3 Noisy Stochastic Gradient Descent (Noisy SGD)
Most recent large-scale ML models are trained via stochastic gradient descent (SGD). Noisy SGD (also known as DP-SGD) is a variant of classical SGD that comes with privacy guarantees. We consider the algorithm as in the work by Abadi et al. [1], which we restate for convenience in Appendix A. While its characteristics with respect to DP have been extensively studied, we take fundamentally different perspective in this work and study the capabilities of this algorithm to protect against membership inference attacks. In summary, the algorithm consists of three fundamental steps: gradient clipping (i.e., 0: == g( Us) max(1, C/I,///ll/ y:)ll) where g (C yi = Li yi) is the gradient with respect to the loss function L), aggregation (i.e., m + Y where Y ~ "N with variance parameter 7²). To obtain
2i=1 O.) and adding
Gaussian noise (i.e., m privacy bounds for this algorithm, we study MI attacks for means of random variables. This allows us to bound the MI vulnerability of SGD.
4 Navigating Between Membership Inference Privacy and DP
In this section, we formally define our privacy notion f-MIP. To this end, it will be handy view MI attacks as hypothesis tests.
4.1 Membership Inference Attacks from Hypothesis Testing Perspective
Initially, we define the following distributions of the algorithm's 's output
Ao = A(X U (x))) with X ~ D=-1, x ~ DD and Ai (ze') = A(X (2r)) with X ~ D=-1 (3)
where we denote other randomly sampled instances that go into the algorithm by X =
Here Ao represents the output distribution under the null hypothesis (Ho) where the sample x' not part of the training dataset. On the other hand, A is the output distribution under the alternative hypothesis (H₂) where x' was part of the training dataset. The output contains randomness due to the instances drawn from the distribution D and due to potential inherent randomness in A.
We observe that the distribution A₁ depends the sample x'' which is known the attacker. The attacker will have access to samples for which Ag and A (xx) are simpler to distinguish and others where the distinction is harder. To reason about the characteristics of such stochastically composed test, we define a composition operator that defines an optimal test in such a setup. To obtain a global FPR of a, an attacker can target different FPRs a(x') for each specific test. We need to consider the optimum over all possible ways of choosing a(x'), which we refer to as test-specific FPR function, giving rise to the following definition.
Definition 4.1 (Stochastic composition of trade-off functions). Let F be a family of trade-off functions, h: D C IRd + F be a function that maps an instance of the data domain to a corresponding trade-off function, and D be a probability distribution on D. The set of valid test-specific FPR functions ā D + [0, 1] that result in a global FPR of a E [0, 1] given the distribution D is defined through
E(o, D) = (ā: D
[0, 1]
E - [ā(x')] =a.
(4)
5

---

For given test-specific FPR function. ā the global false negative rate (type II error) 3 is given by
Ba(a) (M
(5)
where ((x) is the argument to the trade-off function h(x)) E F. For global or E [0, 1] the stochastic
composition of these trade-function is defined as
X h((a)) (a) (a)=
min aEE(a,
{Bx(a)}
(6)
(supposing the minimum exists), the smallest global false negative rate possible at global FPR of a.
This definition specifies the trade-off function aanD h(a): [0, 1] [] of a stochastic composi- tion of several trade-offs. While it reminiscent of the "most powerful test" (MPT) [28], there are several differences to the MPT that are important in our work. Most prominently, a straightforward construction of the MPT to MI problems does not work since the adversary does not only run one hypothesis test guess whether one sample belongs the training data set not; instead, the adversary draws multiple samples and runs sample-dependent and (potentially) different hypotheses tests for each drawn sample. This is necessary due to the form of the alternative hypotheses in the formulation of the test in (3), which depends on the sample x'. We therefore require a tool to compose the results from different hypothesis tests. Finally, prove that the trade-off of the stochastic composition has the properties
trade-off function (see App. D.1):
Theorem 4.1 (Stochastic composition of trade-off functions). The stochastic composition
2anD h(x) of trade-ofi off functions h(x) maintains the characteristics of a trade-off function, i.e., convex, non-increasing,
(8 a h())) (r) < - for all E [0, 1]. and it is continuous at zero.
4.2 /-Membership Inference Privacy (f-MIP)
This rigorous definition of the stochastic composition operator allows us to define membership inference privacy from hypothesis testing perspective.
Definition 4.2 (f-Membership Inference Privacy). Let f be a trade-off function. An algorithm' A : D + Rd is said to be f-membership inference private (f-MIP) with respect to data distribution if
Test (Ao; A1(x)) 2 f,
where x.' ~ and 2 denotes the stochastic composition built from individual trade-off functions of the MI hypotheses tests for random draws of x'.
In this definition, both sides are functions dependent on the false positive rate a. A prominent special case of trade-off function is the Gaussian trade-off, which stems from testing one-dimensional normal distributions of unit variance that are spaced apart by p. E Rzo. Therefore, defining the following special case of fMMIP will be useful.
Definition 4.3 (p-Gaussian Membership Inference Privacy). Let & be the cumulative distribution μ)) to be the trade-off function derived from testing two Gaussians; one with mean 0 and one with mean p. An algorithm A is μ- u-Gaussian Membership Inference private (μ-GMIP) with privacy parameter if is 9n MIP, i.e., it is MI private with trade-off function 9p-
function (CDF) of a standard normal distribution. Define ga(a) == 1b(0=1(1-
Remark 4.1. DP can also be defined via the Gaussian trade-off function, which results u-Gaussian Differential Privacy (H-GDP, [11]). While the trade-off curves for both μ-GDP and y-GMIP have the same parametric form, they have different interpretations: p-GDP describes the trade-ofi off function that an attacker with complete knowledge (left column in Table 1) could achieve while y-GMIP describes the trude-off function that an attacker with MI attack capability can achieve (right column in Table 1). In the next section we will quantify their connection further.
When using the term "algorithm"", we also include randomized mappings.

---

4.3 Relating f-MIP and f-DP
We close this section by providing first results regarding the relation between f-DP and f-MIP. As expected, f-DP is strictly stronger than f-MIP, which can be condensed in the following result:
Theorem 4.2 (f-DP implies f-MIP), Let an algorithm A : D + Rd be f-differentially private [11]. Then, algorithm A will also be J-membership inference private.
We proof this result Appendix F.I. This theorem suggests one intuitive, simple and yet actionable approach to guarantee Membership Inference Privacy. This approach involves the of DP learning algorithms such as DP-SGD [1], which train models using noised gradients. However, as we will see in the next section, using noise levels to guarantee f-DP is usually suboptimal to guarantee f-MIP.
5 Implementing f-MIP through Noisy SGD
We would now like to obtain a practical learning algorithm that comes with f-MIP guarantees. As the dependency between the final model parameters and the input data is usually hard to characterize, follow the common approach and trace the information flow from the data to the model parameters through the training process of stochastic gradient descent [1, 33]. Since the gradient updates the only path where information flows from the data into the model, it suffices to privatize this step.
5.1 f-MIP for One Step of Noisy SGD
We start by considering single SGD step. Following prior work [1, 33], make the standard assumption that only the mean over the individual gradients m = Σm= 0, is used to update the model (or is published directly) where 0; E _Rd is sample gradient. Consistent with the definition of the membership inference game, the attacker tries predict whether specific gradient O' was part of the set {0:}; that was used to compute the model's mean gradient m or not. We are interested in determining the shape of the attacker's trade-off function. For the sake of conciseness, we directly consider one step of noisy SGD (i.e., one averaging operation with additional noising, see Algorithm from the Appendix), which subsumes result for the case without noise by setting 72 O. We establish the following theorem using the Central Limit Theorem (CLT) for means of adequately large batches sample gradients, which proven in Appendix E.
Theorem 5.1 (One-step noisy SGD is f f-membership inference private). Denote the cunulative distribution function of the non-central chi- chi-squared distribution with degrees of freedom and non- centrality parameter 7 by Fx2(r)- Let the gradients 0' E Rdd of the test points follow distribution
with mean μ and covariance E, let K 2 0'/200 [3 and define Neffective n C
=²n² For sufficiently
large batch sizes n, one step of noisy SGD is f-membership inference private with trade-off function given by:
B(a) R= 1-11 F Fx 1)K)
Peeffective
F-1 x
on) (2))))
(8)
Teeffective
1.0
The larger the number of parameters and the batch size grow, the more the trade-off curve approaches the y-GMIP curve, which we show next (see Figure 1).
0.8
now
10.0
22,
15
Pale segunte
0.6
Corollary 5.1 (One step noisy SGD is approx. J-GMIP). For large d, n, noisy SGD is approximately 9Hspp-GMIP. -GMIP. in particular, 3(a) Rs 1b(0-1(11 a) s(Step) with privacy parameter:
0.4
0.2
0.0
d - (2neglective 1)K
0.00
0.75
1.000
Mstep neffective v 2d + 4N K
(9)
False positive rall (e)
Figure 1: Trade-off function
convergence. The trade-off func- tion from Theorem 5.1 converges to the one from Corollary 5.1 where -2=0 and K=d.
This result striking in its generality as it also covers models trained without additional noise or gradient cropping (Neffective n in that case). Unlike for DP, even standard models trained with non-noisy SGD offer elementary level of MIP. Our result further
explicitly quantifies four factors
7

---

that lead to attack success: the batch size n, the number of parameters d. the strength of the noise T² and the worst-case data-dependent gradient susceptibility ||X- 13. The closeness of the trade-off function the diagonal, which is equivalent the attacker randomly guessing whether a gradient O' was part of the training data or not, is majorly determined by the ratio of d to n. The higher the value of d relative to n, the easier it becomes for the attacker to identify training data points. Furthermore, a higher gradient susceptibility K, which measures the atypicality of gradient with respect to the gradient distribution, increases the likelihood of MI attacks succeeding in identifying training data membership. It is worth noting that if we do not restrict the gradient distribution or its support, then there might always exist gradient samples that significantly distort the mean, revealing their membership in the training dataset. This phenomenon is akin to the ó parameter in DP, which also allows exceptions for highly improbable events.
Remark 5.1 (Magnitude of Ustep). When the dimensions of the uncorrelated components in Σ O' are also independent, we expect K to follow a x2 -ddittribtttion with d degrees of freedom and thus K E ((d). In the standard SGD- (T² = 0) with d, n >> 1, we obtain J. E (((d//))
Remark 5.2 (On Optimality). The dependency on when 72 > O is a consequence of our intention- ally broad proving strategy. Our proof approach consists of two key steps: First, we establish the optimal LRT under general gradient distributions, without adding noise or imposing any cropping construints (See Appendix E. This initial step serves as the foundation for our subsequent analysis and is (1) as general as possible covering all distributions with finite vuriance and is (2) optimal in the sense of the Neyman-Pearson Lemma, i.e., it cannot be improved. This means that our result covers all models trained with standard SGD (T² = and C' = oc) and is remarkable in its generality as it is the first to suggest clear conditions when adding noise is not required to reach f-MIP. Second, we specialize our findings to cropped random variables with added noise (See Appendix E.2). This analysis could potentially be improved by considering individual gradient dimensions independently.
5.2 Composition and Subsampling
In the previous section, we have derived the trade-off function for single step of SGD. Since SGD run over multiple rounds, we require an understanding of how the individual trade-off functions can be composed when sequence of f-MIP operations is conducted, and a random subset of the entire data distribution is used as an input for the privatized algorithm. The next lemma provides such a result for μ-GMIP and follows from a result that holds for hypotheses tests between Gaussian random variables due to Dong et al. [11] (see Appendix D.3 for details and more results).
Lemma 5.1 (Asymptotic convergence of infinite DP-SGD), Let n be the batch size in SGD, and N be the entire size of the dataset. If a single SGD Step is at least as hard as Ustep-GMi -MIP with respect
to the samples that were part of the batch and nw/i decreased), then the noisy SGD algorithm will be μ- H-GMIP with
N + e as lim (the batch size is gradually
F1 = /20g feee/exp(pig )xp ) (1.5pstee) + 3 (- 0.55pt)))) 2.
(10)
Note that this result also provides a (loose) bound for the case where exactly T iterations are run with a batch size of n' with c = " (through using ntt) = n' if T, else n(t)= = z'vT), ). With this result in place, we can defend against MI attacks using the standard noisy SGD algorithm.
6 Experimental Evaluation
Datasets and Models. We use three datasets that were previously used in works on privacy risks of ML models [32]: The CIFAR-IO dataset which consists of 60k small images [21], the Purchase tabular classification dataset [25] and the Adult income classification dataset from the UCI machine learning repository [12]. Following prior work by Abadi al. [1], we use a model pretrained on CIFAR-100 and finetune the last layer on CIFAR- 10 using a ResNet-56 model for this task [16] where the number of fine-tuned parameters equals
650. We follow a similar strategy on the Purchase dataset, where we use a three-layer neural network. For finetuning, we use the 20 most common classes and d = 2580 parameters while the model is pretrained on 80 classes. On the adult dataset, use a two-layer network with 512 random features in the first layer trained from scratch on the
8

---

dataset such that d = 1026. We refer to Appendix C.I for additional training details. We release our code online."
6.1 Gradient Attacks Based on the Analytical LRT
To confirm our theoretical anal- ysis for one step of SGD and its composition, we implement the gradient attack based on the like- lihood ratio test derived in the proof of Theorem 5.1. We pro- vide a sketch of the implemen-
Algorithm 1:
Gradient Likelihood Ratio (GLiR) Attack
Require:
Training data distribution D. batch size n, number of pa- rameters d, query point D, averaged gradients of cach batch m. E R for training steps t 1, ..... parameter gradient com- putation function Vw,C C: D + IR of training, threshold yj
1: Puta t-
2: for t 1...., do
tation in Algorithm and addi- tional details in Appendix C.3.
3:
B (bi...., bm D"
Sample background data Compute background gradients
4:
g: (..((b,), 1...m
An essential requirement in the construction of the empirical test is the estimation of the true gradient mean l and the true inverse covariance matrix E-1 since these quantities are essen- tial parts of both the test statistic 10:
5:
Σ Cov (g),... 9m E R
Approximate covariance E Rd
6:
i Mcan {gr. "..,, gml E
> Approximate mean Compute gradients for the query point
7:
0 Vw, C(x)
8:
$ = (n-1) (m: 0)
-f) b Compute test statistic Σ
9:
= | - 1/2 (0-
A)
b. Estimate gradient susceptibility
Pump log (m28) ($)
D. -Compute log p-value under Ho
S and the true gradient suscepti-
11: Photal f-
Piotal + Pstep
12:
end for
bility term K' needed for the an- alytical attack. The attacker uses
13:
return Train if Punal < 17. else Test
their access to the gradient distribution (which is standard for membership inference attacks [5, 29] and realistic in federated learning scenarios [20]), to estimate the distribution parameters. In practice, however, the empirical estimates of μ., Σ and thus Æ will be noisy and therefore we do not expect that the empirical trade-off curves match the analytical curves exactly.
Using our novel Gradient Likelihood Ratio (GLiR) attack we can audit our derived guarantees and their utility. First, we audit our one-step guarantees from Theorem 5. To compare the models, we adapt the batch size n such that all models reach the same level of Ju-GMIP. In Figure 2a, we use simulated gradient distribution with known parameters μ., E and d. In this case, we can estimate K accurately and observe that our bounds are tight when the distribution parameters and thus the respective gradient susceptibilities can be computed accurately. We provide additional ablation studies that gauge the approximation quality of with small values for d, n and different simulated distributions in Appendix C.2. When the parameters parameters, our attacks become weaker and do not match the analytical prediction (see Figure 2b).
unknown and have to estimate the
We also audit our composition guarantees. We do five SGD-steps in Figure 2c. While there is a small gain in attack performance on the CIFAR-10 dataset (e.g., at FPR=0.25), the attack performance on the other datasets remains largely unaffected. This mismatch occurs since the theoretical analysis is based on the premise that the attacker gains access to independently sampled gradient means for each step separate training and non-training points, but in practice do not gain much new information as the model updates are not statistically independent and too incremental to change the gradient means significantly between two subsequent steps. Therefore, a practical attacker does not gain much additional information through performing several steps instead of one. Future work is required to model these dependencies and potentially arrive at tighter composition result under incremental parameter updates. We provide results for additional existing membership inference attacks, for instance the recent loss-based likelihood attack by Carlini et al. [5] in Appendix C.4, which all show weaker success rates than the gradient-based attack that proved most powerful in our setting.
6.2 Comparing Model Utility under pDGDP and G-GMIP
Here we compare the utility under our privacy notion to the utility under differential privacy. We sample 20 different privacy levels ranging from l E [0.4,...,50 50] and calibrate the noise in the SGD iteration to reach the desired value of μ. We can do both for j-GMIP using the result in Equation (10) and using the result by Dong et al. [11, Corollary 4] for J-GDP, which result in the same
chttps://gitaee httpl/.ccommeeeeeeeeeeeeeeeeeeee wtttttteeeennneeeee wwwp//ggsssssiaa snnmsssiimmmpppppp

---

1.00
TPR.
purchase
TPR
purchase
TPR.
purchase
citarto
eitart0
citarto
adult
adult
adult
antlytical
anslytical
1.14)
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
(a) Single step simulated gradient (b) Single step with real model gra- distribution with known parameters.
(c) As (b), now composition of 5 steps for real model gradients.
dients and estimated parameters.
Figure 2: Auditing /-MIP with our gradient attack (GLiR) when 7² = We show trade-off curves when the gradient distribution is known (a) and when the gradients are obtained from a trained model that was finetuned on various data sets (b, c). The analytical solutions are computed with a value of K = d and using the composition result for l steps in Appendix D.3 for (c).
70
83.0
oxcunay (8)
60
60
Dift. Privacy (DP)
aocurac
.0
60
Mem. Inf. Privacy (MIP)
82.0
40
Did Privacy
Mem. Inf. Privacy (MIP)
20
Didt. Privacy (DP)
Mem. Inf. Privacy (MIP)
0
81.0
10P
privacy (p)
privacy (p)
privacy
(a) CIFAR-10
(b) Purchase
(c) Adult
Figure 3: Utility of DP versus MIP. Model performance on three datasets across different privacy lev- els μ (small y denotes high privacy) using the notions of j-Gaussian Differential Privacy (parametric form of f-DP, [1]J) and p-Gaussian Membership Inference Privacy (parametric form of f-MIP, ours) on three datasets. GMIP usually allows for substantially increased accuracy over the corresponding GDP guarantee with the same attack success rates controlled by μ.. However, the attacker under GMIP runs membership inference (MI) attacks while GDP allows for wider of privacy threat models. For more details on differences in the underlying threat models see Table 1.
attack success rates while G-GDP allows for stronger privacy threat models. Due to Theorem 4.2, never need to add more noise for G-IMII than for μDDP. Further details are provided in Appendix C. Figure 3 shows a comparison of the accuracy that the models obtain. We observe that the model under GMIP results in significantly higher accuracy for most values of μr. As -> 0 both privacy notions require excessive amounts of noise such that the utility decreases towards the random guessing accuracy. On the other hand, for higher values of μr, there is no need to add any noise to the gradient obtain μ-GMIP, allowing obtain the full utility of the unconstrained model. This indicates that useful GMIP-bounds do not necessarily require noise. For instance, on the CIFAR-10 model, no noise required for μ 2 0.86 which a reasonable privacy level [11]. Overall, these results highlight that useful and interpretable privacy guarantees can often be obtained without sacrificing utility.
7 Conclusion and Future Work
In the present work, we derived the general notion of /-Membership Inference Privacy (f-MIP) by taking hypothesis testing perspective on membership inference attacks. We then studied the noisy SGD algorithm as a model-agnostic tool to implement f f-Membership Inference Privacy, while maintaining Differential Privacy (DP) as worst-case guarantce. Our analysis revealed that significantly less noise may be required to obtain f-MIP compared to DP resulting in increased utility. Future work required to better model the dependencies when composing subsequent SGD steps which could lead to improved bounds in practice. Furthermore, our analysis shows that when the capacity of the attacker further restricted, e.g., to API access of predictions, there remains gap between our theoretical bounds and loss-based membership inference attacks that can be implemented for real models. More work required to either produce more sophisticated attacks or derive theoretical bounds for even less powerful attackers to close this gap.
10

---

References
[1] Abadi, M., Chu, A., Goodfellow, I,, McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308-31 318, 2016. 2, 5, 7, 8, 14, 16
[2] Andrew, G., Kairouz, P., Oh, S., Oprea, A., McMahan, H. B., and Suriyakumar, V. One-shot empirical privacy estimation for federated learning. arXiv preprint arXiv:2302.03098, 2023. 14
[3] Balle, B., Barthe, G., Gaboardi, M., Hsu, J., and Sato, T. Hypothesis testing interpretations and renyi differential privacy. In International Conference on Artificial Intelligence and Statistics, pp. 2506. PMLR, 2020. 4
[4] Bassily, R., Thakkar, O., and Guha Thakurta, A. Model-agnostic private learning. Advances in Neural Information Processing Systems (NeurIPS), 31, 2018." 3
[5] Carlini, N., Chien, S., Nasr, M., Song, S., Terzis, A., and Tramer, F. Membership inference attacks from first principles. arXiv preprint arXiv:21
2021. I, 2, 3, 4, 9, 14, 15, 17, 18
[6] Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert- Voss, A., Lee, K., Roberts, A., Brown, B., Song, D., Erlingsson, U., et al. Extracting training data from large language models. In USENIX Security Symposium, volume 6, 2021. 1, 4, 16
[7] Carlini, N., Hayes, J., Nasr, M., Jagielski, M., Sehwag, V., Tramer, F., Balle, B.,, Ippolito, D., and Wallace, E. Extracting training data from diffusion models. arXiv preprint arXiv:2301. 1888
2023. 1, 14
[8] Chen, D., Yu, N., Zhang, Y., and Fritz, M. Gan-leaks: A taxonomy of membership inference attacks against generative models. In Ligatti, J., Ou, X., Katz, J., and Vigna. G. (eds.), CCS '20: 2020 ACM SIGSAC Conference on Computer and Communications Security, Virtual Event, USA, November 9-13, 2020, pp. 343-362. ACM, 2020. doi: 10. 10.1145/372297..41122 URL https://doi. org/ 10rg//0..0 1145/3372297 3417238. 14
[9] Choquette-Choo, C. A., Tramèr, F., Carlini, N., and Papernot, N. Label-only membership inference attacks. In Proceedings of the 37th International Conference on Machine Learning (ICML), volume abs/2007. 14422007..1421,, 2020. 3, 4,
[10] Cummings, R., Desfontaines, D., Evans, D., Geambasu, R., Jagielski, M., Huang, Y., Kairouz, P., Kamath, G., Oh, S., Ohrimenko, O., al. Challenges towards the next frontier in privacy. arXiv preprint urXiv:2304.06929 2023, 1
[11) Dong, J., Roth, A., and Su, J. Gaussian differential privacy. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(1):3-37, 2022. 2, 3, 4, 6, 9, 10, 14, 16, 18, 20, 21, 33, 34
[12] Dua, D. and Graff, C. UCI machine learning repository, 2017. URL UtLAttpp:/aacchive..ss uci. edu/ml. 8
[13] Dwork, C. and Feldman, V. Privacy-preserving prediction. In Conference On Learning Theory, pp. 1693-1702. PMLR, 2018. 2.
[14] Dwork, C., McSherry, F., Nissim, K., and Smith, A. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pp. 265-284. Springer, 2006. 1,
[15] Haim, N., Vardi, G., Ychudai, G., Shamir, O., et al. Reconstructing training data from trained neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 3, 14, 16
[16] He. K., Zhang, X., Ren, S., and Sun, Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
[17] Izzo, 7.,, Yoon, J., Arik, O., and Zou, J. Provable membership inference privacy. arXiv preprint arXiv:2211.06582 2022. 3

---

[18] Jagielski, M., Ullman, J., and Oprea, A. Auditing differentially private machine learning: How private private sgd? Advances Neural Information Processing Systems (NeurIPS), 33: 22205-22211 2020. 14
[19] Kairouz, P., Oh, S., and Viswanath, P. The composition theorem for differential privacy. In Proceedings of the 32nd International Conference on International Conference on Machine Learning (ICML), pp. 1376-1385, 2015,
[20] Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennís, M., Bhagoji, A. N., Bonawitz, K.,
Charles, Z., Cormode, G., Cummings, R., et al. Advances and open problems in federated learning. Foundations and Trends@ Machine Learning, 14(1-2):1-216 2021. 2, 9, 15
[21] Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009. 8
[22] Lehmann, E. L.. Romano, J. P., and Casella, G. Testing statistical hypotheses, volume 3. Springer, 2005. 24
[23] Long, Y., Bindschaedler, V. Wang, L., Bu, D., Wang, X., Tang, H., Gunter, C. A., and Chen, K. Understanding membership inferences on well nell-generalized learning models. arXiv preprint arXiv:1802.04889) 2018. 3, 14
[24] Maddock, S., Sablayrolles, A., and Stock, Canife: Crafting canaries for empirical privacy measurement in federated learning. In International Conference on Learning Representations (ICLR), 2023. 14
[25] Nasr, M., Shokri, R., and Houmansadr, A. Machine learning with membership privacy using adversarial regularization. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, pp. 334-646.77 2018. 8
[26] Nasr, M., Songi, S., Thakurta, A., Papernot, N., and Carlin, N. Adversary instantiation: Lower bounds for differentially private machine learning. In 2021 IEEE Symposium on security and privacy (SP), pp. 866-882. IEEE, 2021. 2, 14
[27] Nasr, M., Hayes, J., Steinke, T., Balle, B., Tramer, F., Jagielski, M., Carlini, N., and Terzis, A. Tight auditing of differentially private machine learning. In 32nd USENIX Security Symposium (USENIX Security 23). pp. 163 1-1648. USENIX Association, 2023. 3, 14
[28] Neyman, and Pearson, E. S. Ix. on the problem of the most efficient tests statistical hypotheses. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, 23 (694-7770))
1933.
[29] Pawelczyk, M., Lakkaraju, H., and Neel, S. On the Privacy Risks of Algorithmic Recourse. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2023. 1, 3, 9, 14, 15, 17
[30] Sablayrolles, A., Douze, M., Schmid, C., Ollivier, Y., and Jegou, H. White-box black-box: Bayes optimal strategies for membership inference. In Proceedings of the 36th International Conference on Machine Learning (ICML), 2019. 14
[31] Shokri, R., Stronati, M., Song, C., and Shmatikov, V. Membership inference attacks against machine learning models. In 2017 IEEE symposium security and privacy (SP), pp. 3-18. IEEE, 2017.
14
[32] Shokri, R., Strobel, M., and Zick, Y. On the privacy risks of model explanations. In Proceedings of the 2021 AAAT/ACM Conference on AI, Ethics, and Society (AIES), pp. 231-241, 2021. 1, 3. 8, 14, 15
[33] Song, S., Chaudhuri, K., and Sarwate, A. D. Stochastic gradient descent with differentially private updates. In 2013 IEEE global conference on signal and information processing, pp. 245-248. IEEE, 2013. 7, 16
[34] Steinke, T., Nasr, M., and Jagielski, M. Privacy auditing with one (1) training run. In Advances in Neural Information Processing Systems (NeurIPS), 2023. 14
12

---

[35] Tan, J., Mason, B., Javadi, H., and Baraniuk, R. Parameters privacy: A provable tradeoff between overparameterrization and membership inference. Advances Neural Information Processing Systems (NeurIPS), 15:17488-17000 2022. 3, 14
[36] Tan, J., LeJeune. D., Mason, B., Javadi, H., and Baraniuk, R. G. A blessing of dimensionality in membership inference through regularization. In International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 10968-10993, PMLR, 2023. 3, 14
[37] Thudi, A., Shumailov, I., Boenisch, F., and Papernot, N. Bounding membership inference. arXiv preprini urXiv:2202. 2022. 14
[38] Triastcyn, A. and Faltings, B. Bayesian differential privacy for machine learning, In Interna- tional Conference on Machine Learning (ICML), pp. 9583-9592, PMLR, 2020. 2,
[39] Wasserman, and Zhou, statistical framework for differential privacy. Journal of the American Statistical Association, 105(489):375-389) 2010. 4
[40] Ye, J., Maddi, A., Murakonda, S. K., Bindschaedler, V., and Shokri, R. Enhanced membership inference attacks against machine learning models. In Proceedings of the 2022 ACM SIGSAC Conference Computer and Communications Security (CCS), 2022. 3, 4, 14
[41] Yeom, S., Giacomelli, I., Fredrikson, M., and Jha, S. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31 computer security foundations symposium (CSF), pp. 268-282. IEEE, 2018. 3, 14, 15
[42] Ganella-Béguelie S., Wutschitz, L., Tople, S., Salem, A., Rühle, V., Paverd, A., Naseri, M., Köpf, B., and Jones, D. Bayesian estimation of differential privacy. In International Conference on Machine Learning (ICML). PMLR, 2023. 14
13


Reviews:
[{"review_id": "OTpjSX5sVA", "reviewer": "Reviewer_DVNo", "summary": "The paper proposes a notion of Gaussian membership inference privacy (GMIP) to capture the information leakage of a training algorithm about a data point $x$ when all the remaining training datasets are randomly drawn from a data distribution. The new GMIP definition has two main benefits compared to the prior leakage definition.\n1. It captures the entire trade-off curve of membership inference as a hypothesis test and is thus more informative than prior average-case privacy definitions such as membership inference advantage. \n2. It stochastically composes the trade-off curves over the randomly drawn remaining training dataset (other than the target data). It, therefore, captures leakage against a more realistic adversary that does not have control over the remaining training dataset (compared to the worst-case f-DP definition). \n\nFollowing this new GMIP definition, the paper analytically proves GMIP for the DP-SGD algorithm via an analytical likelihood ratio attack on an observed noisy gradient (under assumptions on the gradient distribution, model dimension, and dataset size). The proved GMIP bound has interesting dependencies on the model dimension and a data-dependent constant associated with the gradient distribution. To further illustrate the tightness of this GMIP bound, the authors evaluate the performance of various attacks for a mu-GMIP noisy SGD algorithm and show when the attack performances are close to the GMIP upper bound. Finally, the authors evaluate model accuracy under mu-GMIP and show that it improves over the model accuracy under mu-GDP, which indicates a privacy-utility trade-off gain due to relaxed adversary assumption.", "questions": "[Q1] What exactly are the approximations and assumptions used for the proved GMIP bound, for the comparison figure 1? How much would possible issues with assumptions (such as the one mentioned in weakness [W2] and insufficiently large dataset size n and model dimension d) break the proved GMIP bound? \n\n[Q2] Is the proved GMIP bound less tight than the standard mu-GDP bound for DP-SGD, when the model dimension is large? See weakness [W3] for more information.", "limitations": "The limitations regarding assumptions made for the analysis should be discussed more. See weakness [W1] and [W2] for more details.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper studies an important problem of analytically bounding informative information leakage of training algorithms against realistic adversaries.\n- The critical component is a novel analytical derivation of the likelihood ratio test, assuming that the aggregated noisy gradients follow a multivariate Gaussian distribution. \n- The proved GMIP bound is novel and has interesting dependencies on various factors about the model and data distribution. The discussion about the tightness of the bound is detailed and supported by empirical evaluations.", "weaknesses": "[W1] The main weakness is that the proved GMIP bound is based on several approximation arguments and relies on assumptions about the gradient distribution, the model dimension, and the dataset size. The authors should clarify these approximations and assumptions in the comparison in Figure 1. Otherwise, the comparison to the mu-GDP algorithm is unfair or misleading.\n\n[W2] Specifically, one of the assumptions used in approximating the LRT (line 618 appendix D.1) is that the distribution of averaged gradient follows a Gaussian distribution, provided that the number of averaged samples is large enough. This assumption is invalid because the gradients are clipped before averaging, so the distribution of averaged gradient is bounded and is not close to a Gaussian distribution. \n\n[W3] The proved GMIP bound Theorem 4.2. grows indefinitely with regard to model dimension d. Such dimension dependency does not exist in the standard mu-GDP bound for DP-SGD (which implies GMIP). This suggests that the GMIP bound may be less tight than the standard mu-GDP bound for the high-dimensional problem, which is the case for training a deep neural network.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for their thoughtful review and the questions raised. We clarify the individual points below.\n\n> [W1] The main weakness is that the proved GMIP bound is based on several approximation arguments and relies on assumptions about the gradient distribution, the model dimension, and the dataset size.The authors should clarify these approximations and assumptions [...]\n\nWe clarify that we do not make parametric approximation assumptions about the gradient distribution. Our theoretical results stem from the application of the central limit theorem (CLT) to averages formed over mini-batches of gradients and do *not* use parametric assumptions on the gradient distribution. Loosely speaking, the CLT states that the distribution of sample means converges to a Gaussian for large enough sample sizes regardless of the distribution of individual gradients. Therefore, we dont need to make assumptions about the shape of the gradient distribution. For a discussion on other parameters see the response to [Q1].\n\nWe provide a full discussion of threat models of other privacy notions in the general comment due to space constraints, and will incorporate them into the final version of our manuscript. \n\n> [W2] Specifically, one of the assumptions used in approximating the LRT (line 618 appendix D.1) is that the distribution of averaged gradient follows a Gaussian distribution [...] This assumption is invalid because the gradients are clipped before averaging, so the distribution of averaged gradients is bounded and is not close to a Gaussian distribution.\n\nWhile it is correct that the gradients are clipped, we stress that, in non-technical terms, the Central Limit Theorem states that the **distribution of sample means converges to a normal distribution for large enough sample sizes, regardless of the shape of the distribution of the individual samples** [1, p.66-68]. This means that the individual gradients may be clipped and that the CLT can still be applied to averages over clipped gradients. We demonstrate this empirically through an ablation study (see answer to [Q1])\n\n> [W3] The proved GMIP bound Theorem 4.2. grows indefinitely with regard to model dimension d. [..] This suggests that the GMIP bound may be less tight than the standard mu-GDP bound for the high-dimensional problem, which is the case for training a deep neural network.\n\n> [Q2] Is the proved GMIP bound less tight than the standard mu-GDP bound for DP-SGD, when the model dimension is large? [...]\n\nIndeed there might exist settings where our results require more noise for $\\mu$-MIP than $\\mu$-DP. In these cases one should resort to $mu$-DP as it implies $mu$-MIP. This is however not the case for the realistic models considered in this work. \n\nThe dependency on the parameter $d$  is a consequence of our intentionally broad proving strategy. Our proof approach consists of two key steps: First, we establish an optimal LRT framework under general gradient distributions, without imposing any cropping constraints (App. D1). This initial step serves as the foundation for our subsequent analysis and is (1) as general as possible, i.e., no distribution assumptions and (2) optimal in the sense of the Neyman-Pearson Lemma, i.e., it cannot be improved. Our result covers all models trained with standard SGD and is remarkable in its generality as it is the first to suggest clear conditions when adding noise is not required to reach $mu$-MIP.\n\nSecond, we specialize our findings to noise addition on cropped random variables (App D2). This analysis may potentially be improved. We offer the following intuitive rationale: Since variance of noise is considered fixed across dimensions, the introduction of additional dimensions naturally leads to an overall increase in norms, approx. on the order of $O(sqrt(d))$. It's important to note that the dependence on $d$ may thus be removed through full incorporation of the fixed cropping threshold $C$ as is common in DP. We leave this improvement for future work as it requires a substantially more complicated analysis \n\nFinally, we emphasize that our work already achieves a significant milestone by being the first to analytically characterize the entire trade-off curve for membership inference attacks.\n\n> [Q1] What exactly are the approximations and assumptions used for the proved GMIP bound, for the comparison figure 1? How much would possible issues with assumptions [...] break the proved GMIP bound?\n\nWhile we require that the gradient dim. $d$ and batch size $n$ are sufficiently large, we put these aspects into perspective. Following the suggestion of the Reviewer, we ran four ablation studies in the rebuttal PDF to investigate the effects of the params. on the GMIP bound. We do this by averaging gradients that follow a Gaussian or a uniform distribution and observe that the $\\mu$-GMIP bound closely reflects the empirically observed trade-off curves for:\n\n*   Realistic values of batch size $n$ ($n >= 10$) both when the Gradients follow a Gaussian (Figure 3a, e) or a Unit distribution (Figure 3b, f)\n*   A wide range of cropping thresholds C from 1 to 10 (in this example, the expected value of $\\lVert \\theta_i \\rVert = 5$, Figure 3c, g). We see no effect on the validity of our results (see CLT argument).\n*   Small values of $d$. Our bound is an extremely good approximation even for values as small as $d=2$ (Figure 3d, h)\n\n---\n\nWe thank the Reviewer again for the feedback, which will certainly help improve our manuscript. We sincerely hope that we have particularly addressed your main concern of why the CLT is in fact applicable and why no distributional assumptions are required, which are empirically confirmed in the new ablation experiment provided. We are happy to include these remarks in the final version of the paper. We would kindly request the Reviewer to reconsider the review score in light of this response.\n\nReference\n\n[1] Y. Dodge. The Concise Encyclopedia of Statistics. Springer, 2008"}, {"review_id": "60O4TUasRz", "reviewer": "Reviewer_qYbt", "summary": "Authors came up with a privacy definition that is more relaxed than DP. It is called Gaussian Membership Inference Privacy (GMIP) which consists of a hypothesis testing which is supposed to decide whether a single instance is present in the training data. DP implies GMIP. The proposed privacy framework is applied with SGD and in experiments found to be much better than DP in terms of utility.", "questions": "* Definition 4.1 is basically the most powerfull test. $\\mathcal{E}$ is the significance and $\\beta$ is the power function, or more concretely 1 - power function?\n* Regarding Theorem 4.1: according to the Neyman-Pearson fundamental lemma the risk function is always convex when the null and alternative consist of a single distribution. And the testing problem presented in (3) is like that. See Lehamnn-Romano: Testing Statistical Hypotheses, Sec 3.2. \n* Testing are applied in many rounds and i guess only union bound is used over the individual SGD steps. Some sequential hypthesis testing might be worth to consider?\n", "limitations": "I believe that this research direction is interesting and promising. However, convexity of risk function seems not novel observation which is the key of having most uniformly powerful test. And this is what the paper relies on. I do not understand why to stick to Gaussion distribution, since any distribution from the exponential family can be used in a very similar way. Furthermore, FNR are controlled in each SGD step independently from each other however sequential testing approaches might applied here which would make this paper much more interesting. So technical contribution is somewhat limited.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* There is a need to come up with some privacy definition that is more practical than DP.\n* Research direction is promising.", "weaknesses": "* I believe that standard notions from statistical hypothesis testing are reinvented, and the results seems not so surprising taken into account results already published in testing. I would suggest to reuse those results (if it is possible). Nevertheless, please address my questions!", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for their thoughtful review and the questions raised. We will clarify the individual points below.\n\n> I believe that standard notions from statistical hypothesis testing are reinvented, and the results seems not so surprising taken into account results already published in testing. [...]\n\nWe would like to highlight that we only *leverage* standard notions from statistical hypothesis testing. However, we frame the problem of membership inference privacy as a hypothesis testing problem to be able to use testing tools to develop a constructive theory. This theory helps us set up a privacy notion geared towards the realistic and empirically well studied threat model of membership inference attacks and a constructive algorithm which effectively protects against such attacks. Such results are highly sought after in the privacy literature, where the need to reconsider threat models has recently been proclaimed by leading researchers in the field [1, sections 2.1, 4.1, 4.3].\n\n> Definition 4.1 is basically the most powerfull test. $\\mathcal{E}$ is the significance and $\\beta$ is the power function, or more concretely 1 - power function?\n\nThe reviewer is right in that Definition. 4.1 is reminiscent of the most powerful test. However, there are several differences to the standard definition of the most powerful test that are important in our work and that motivate the need for Definition 4.1. Most prominently, we adjust the definition of the most powerful test (i.e., hypothesis tests) to be applicable to the membership inference problem (this is what definition 4.1 accomplishes). Please note that a straightforward construction of the most powerful test does not work in this setup. This is the case because the adversary does not only run one hypothesis test to figure out whether one sample belongs to the training data set or not; instead, the adversary draws samples $x$ and runs individual, sample-dependent and different hypotheses tests for each drawn sample. This is necessary due to the formulation of the distribution $A_1(x)$ under the alternative hypotheses in the formulation of the test (Eqn. 3), which depends on the sample $x$. The value of $x$ is known to the adversary. We therefore require a tool to compose the results from the different hypothesis tests, which we carefully craft in Definition 4.1.We can then compute the expected trade-off curve for an adversary that runs tests with different powers according to the observed samples $x$.\n\n> Regarding Theorem 4.1: according to the Neyman-Pearson fundamental lemma the risk function is always convex when the null and alternative consist of a single distribution. And the testing problem presented in (3) is like that. [...]\n\nIn Definition 4.1., the distributions are not simple, but instead depend on a stochastic sample $x$. In particular, the distribution under the alternative hypotheses, $A_1(x)$ in Eqn. (3), depends on the individual sample $x$, which is known to the attacker and can be used to run sample-specific tests. Therefore, to compute the expected trade-off curve that can be reached by an attacker who samples $x$, we use the trade-off defined in Definition 4.1. While its properties shown in Theorem 4.1 may seem intuitive, the proof is not trivial, which is why we decided to include it for completeness. Note that we consider our main contributions to be Theorem 5.1 and Corollary 5.1, which are the first of its kind and precisely quantify the factors that lead to successful Membership Inference.\n\n> Testing are applied in many rounds and i guess only union bound is used over the individual SGD steps. Some sequential hypthesis testing might be worth to consider?\n\nWhile we do not use the union bound for the composition as suggested by the reviewer, our result composing multiple steps of SGD given in Lemma 5.1 and follows from Theorem 11 and Theorem 4  in Dong et al. [2]. These results already provide tight composition bounds for hypothesis tests that satisfy all our needs. We are however confident that it would be possible to derive similar results using union bounds as well.\n\n> I do not understand why to stick to Gaussion distribution, since any distribution from the exponential family can be used in a very similar way\n\nPlease note that we don’t *assume* the Gaussian distribution. Instead, we consider averages over parameter gradients commonly used in minibatch stochastic gradient descent, which then follow a Gaussian distribution by the central-limit theorem (CLT). We put no distributional assumption's on the gradients at all.\n\n> Furthermore, FNR are controlled in each SGD step independently from each other however sequential testing approaches might applied here which would make this paper much more interesting. So technical contribution is somewhat limited.\n\nNote that we use tight results from the testing literature, which consider the full trade-off curve of a sequence of hypothesis tests (see, e.g., Theorem 4 of Dong et al. [2]). These bounds are optimal for any overall FNR and also cover cases where, e.g., different FNRs at targeted at the individual tests or the statistics of different tests are combined in non-trivial ways.\n\nOverall, we would like to stress that we *use* tools from the hypotheses testing literature, but this is not our key contribution. The key contribution of our work lies in defining MI privacy through the composed trade-off curve and bounding the attack risk of DP-SGD with respect to membership inference attacks. We will adjust our manuscript to better differentiate our contributions from the existing tools that we use. We hope that our replies have clarified the matter and are happy to answer any follow-up questions.\n\n**References** \n\n[1] Challenges towards the Next Frontier in Privacy, Cummings et al. (2023), arXiv:2304:06929\n\n[2] Dong, J., Roth, A., and Su, W. J. Gaussian differential privacy. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(1):3–37, 2022."}, {"review_id": "MtLWA16KXT", "reviewer": "Reviewer_PQQ4", "summary": "the submission extends the work on privacy guarantees specifically for protecting data membership inference attacks, and the extension is on using the Gaussian distribution to characterize the trade-off function. The proposed Gaussian Membership Inference Privacy has the same parametrization as the Gaussian Differential Privacy, but provides weaker privacy guarantees in the sense that an attacker only has access to the learned machine learning model and true data distribution or the public data. \n\nThe submission further shows that a single step of noisy SGD is approximately Gaussian membership Inference privacy with a privacy parameter $\\mu$, and empirically shows that the theoretical trade-off curve is the upper bound of the ROC curves of practical attacks with the given public knowledge.", "questions": "please see the weaknesses.", "limitations": "please see the weaknesses.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. the membership inference attack captures a class of realistic attacks against machine learning models, where the attacker can't manipulate the private training data but has access to the data distribution and the learned model through an API. It indeed provides weaker privacy guarantees than Differential Privacy does, but studying it might lead us to a more practical privacy definition for protecting individuals in machine learning systems.\n\n2.  one-step noisy SGD can be captured by f-membership inference privacy, and can be captured approximately by Gaussian Membership Inference privacy.\n\n3. a very important observation is that averaging over norm-clipped gradients satisfies GMIP with a bounded privacy parameter, which means that it already protects, to some extent, against membership inference attacks on gradients.", "weaknesses": "1. Gaussian Differential Privacy exactly captures the trade-off function of the Gaussian mechanism, which is used to release noisy gradients. However, Gaussian Membership Inference Privacy only approximately captures so, and I am wondering what 'approximately' entails, and whether it is actually useful compared to f-membership inference privacy.\n\n2. the assumption for the algorithm that an attacker would use is that the algorithm has to return binary answers, but we've seen successful attacks using ranking, for example, the watchdog experiments in [1]. Thus, I am questioning this assumption on top of an attacker's algorithm.\n\n3. I was hoping to see experiments with real machine learning models with only API access to them, e.g. an attacker can only have access to the trained classification neural network through the final probabilistic predictions over classes. The distributional assumption over the susceptivity does not hold anymore, and it is interesting to see how the proposed privacy definition captures the success of membership inference attacks.\n\n\n\n[1] https://www.pnas.org/doi/epdf/10.1073/pnas.2218605120 ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are very grateful for the positive comments, which highlight the relevance of the studied Membership Inference threat model, our ability to analytically capture SGD steps with f-MIP, and which highlight the finding that we can obtain MIP through averaging of gradients only. We will address the remaining points below.\n\n> Gaussian Membership Inference Privacy only approximately captures so, and I am wondering what 'approximately' entails, and whether it is actually useful compared to f-membership inference privacy.\n\nFirst note that our exact formulation from Theorem 5.1 contains the CDF of a $\\chi^2$ distribution, while our approximation from Corollary 5.1 uses the Gaussian CDF. In general, it is well known that for large $d$, the $\\chi^2$ distribution converges to a Gaussian distribution at a convergence rate of $\\mathcal{O}(1/\\sqrt{d})$ (see Theorem 1 in [7], see Figure 2a of the paper for an illustration). Regarding $n$, we can apply the Berry-Esseen Theorem to get a finite sample bound for the error between the true averaged gradient distribution and the Gaussian distribution. The finite sample error depends on a universal constant (some number depending on third moments), and scales with $\\mathcal{O}(1/\\sqrt{n})$. Hence, as we increase the number of gradient vectors in a batch, we expect the error to decrease. To demonstrate that this is in fact the case, we have added ablation results in Figure 3 of the rebuttal PDF file for small dimensions (d) and for small batch sizes (n). In these experiments, we start with small values of $d$ and $n$ and increase these which empirically demonstrates that our approximations are accurate even for small $d$ and $n$ despite our theory being carried out for larger sample sizes.\n\n> the assumption for the algorithm that an attacker would use is that the algorithm has to return binary answers, but we've seen successful attacks using ranking, for example, the watchdog experiments in [1]. Thus, I am questioning this assumption on top of an attacker's algorithm.\n\nMembership Inference attacks are a common standard in the literature (e.g., [1-6]) and are one of the simplest attacks that can be carried out by an adversary. The assumption about the output of the membership inference attack follows the standard definition originally introduced by [4], which has since been adopted in many popular prior works (e.g., [1,5]). The reference provided by the reviewer considers reconstruction attacks, which are different in the sense that full samples are reconstructed. As opposed to reconstruction attacks, membership inference attacks attempt to only verify whether a given sample was part of the training dataset. Due to the simplicity of the MI attack, protecting against MI should also offer protection against other, more complex attacks as well. For example, one could cast the data reconstruction attack as a series of sequentially applied membership inference attacks where the task consists of verifying whether a given token was part of the training data set. We will gladly include a discussion of reconstruction attacks such as the one mentioned in our manuscript.\n\n> I was hoping to see experiments with real machine learning models with only API access [...] it is interesting to see how the proposed privacy definition captures the success of membership inference attacks.\n\nThe membership inference threat model in our work uses access to gradients, which is a common scenario in federated learning setups. Due to the use of the gradients our attack is stronger than attacks that rely solely on API access. To showcase the superiority of our attack, we run the state-of-the-art API-only attack suggested by Carlini et al. [1] on real machine learning models and obtain the following results shown in Figure 5 of the Appendix. These results show that our attack (Figure 4) is 2-10 times stronger than the state-of-the-art attack based on API access only. Thus, adding the noise required to protect under our optimal attack under gradient access, also protects against attacks that rely on API access only. \n\n-------\n\nWe appreciate the positive comments by the reviewer and are glad to hear that the Reviewer appreciated the “Soundness”, “Presentation”, and “Contribution” of our work by awarding a rating of \"Good\". We hope that we have addressed the remaining concerns with the new ablation studies and the results on the state-of-the-art API-only attack by Carlini et al. [1]. Given the positive evaluations and our clarifications, we would kindly ask the Reviewer to reconsider their overall score.\n\n-----\n\n**References**\n\n[1] Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramèr. Membership inference attacks from first principles. IEEE Symposium on Security and Privacy (SP), 2022.\n\n[2] Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, and Reza Shokri. Enhanced membership inference attacks against machine learning models. ACM CCS, 2022.\n\n[3] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), IEEE, 2017.\n\n[4] Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 31st IEEE Computer Security Foundations Symposium, 2018..\n\n[5] Martin Pawelczyk, Himabindu Lakkaraju, Seth Neel. On the Privacy Risks of Algorithmic Recourse. Proceedings of The 26th International Conference on Artificial Intelligence and Statistics (AISTATS), 2023.\n\n[6] Christopher A. Choquette-Choo, Florian Tramèr, Nicholas Carlini, and Nicolas Papernot. Label-only membership inference attacks. In Proceedings of the 37th International Conference on Machine Learning (ICML), 2020. \n\n[7] Donagh Horgan and Colin C. Murphy. On the Convergence of the Chi Square and Noncentral Chi Square Distributions to the Normal Distribution. IEEE Communications Letters, Vol. 17, No. 12, 2013 \n"}, {"review_id": "0lS2hKb0OJ", "reviewer": "Reviewer_Uu8E", "summary": "The authors introduce a new privacy notion called f-membership inference privacy (f-MIP), which relaxes strict Differential Privacy (DP) assumptions thereby promising better model utility. The paper proposes a theoretical analysis of membership inference attacks on DP-SGD based on trade-off curves (similar to f-DP) and introduces a family of f-MIP guarantees called µ-Gaussian Membership Inference Privacy (µ-GMIP)(similar to GDP).  The analysis follows a similar approach to the original DP-SGD analysis: first the privacy budget is derived for single step with subsampling which is then  composed over the training run.\n\nThe paper then verifies the theoretical analysis by introducing gradient attacks based on likelihood ratio tests.\nThe attack requires to know the underlying gradient distribution parameters. They present results for a single DP-SGD step with known parameters and the privacy guarantees seem tight. They also present results for estimated distribution parameters where the guarantees look loser.", "questions": "- Is it assumed that $\\tau$ in the noise parameter includes the batch size? Algorithm 1 does not scale the noise by the batch size which is different to typical DP-SGD [Abadi et al 16].\n- Figure 4a. Purchase and CIFAR10 seem to exceed the analytical bound for low FPRs. Is this because of the earlier mentioned assumption that the challenge points are sampled from the distribution which for low FPRs may be already distributional outliers?\n- Are the techniques for composing DP guarantees numerically also applicable in this setting?", "limitations": "As discussed in weaknesses. I believe the discussion about the validity of the initial assumptions is limited.\n", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The derivations seem sound and follow the well regarded hypothesis testing interpretation in DP\n- Investigating the tightness and potential relaxation of DP-SGD is an important problems and the authors make a solid contribution.\n- The paper is clearly structured and easy to follow.", "weaknesses": "- It would be nice to see an investigation on the validity of the initial assumptions. The paper was motivated that DP is overly conservative since it also holds for pathological datasets such as empty datasets and singletons with an adversarial sample. While this is true, it has been recently shown that a simple canary insertion in an otherwise natural dataset may be sufficient to produce tight lower bounds for DP accountants [Nasr et al 2023].\n- It would be great to extend the FPR and TPR ranges in the plots in figure 4 to smaller values. Ideally all the way to the first data point. These ranges capture very relevant adversary objectives where an adversary only cares about identifying one sample but that with high confidence.\n- Minor:\n  - Only asymptotic guarantees for composition. Recently, there has been significant progress in numerical composition of DP guarantees.\n\n\n**References**\n\nNasr, Milad, et al. \"Tight Auditing of Differentially Private Machine Learning.\" arXiv preprint arXiv:2302.07956 (2023).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for their thoughtful review and were pleased to hear that the Reviewer found our paper to be *clearly structured* and to be a *solid contribution*. We will answer the individual points raised below.\n\n> It would be nice to see an investigation on the validity of the initial assumptions. [...] it has been recently shown that a simple canary insertion in an otherwise natural dataset may be sufficient to produce tight lower bounds for DP accountants [Nasr et al 2023].\n\nThe reviewer’s observation is accurate. The problem of finding a tight lower bound for DP is considered solved as canary insertions (i.e., outlier sample insertions to datasets) make the DP bound tight (Nasr et al 2023). Therefore, to improve the utility over DP-trained models, new takes on privacy are in demand where one maps the exact threat model to a suitable privacy notion, which is precisely what our work accomplishes. If we can rule out pathological canary insertions (e.g., because the adversary, by construction, has no dataset access), other notions than standard differential privacy can be considered. This is what has recently been identified as a fundamental open problem by a large group of leading researchers from the field of privacy preserving ML ([1, sections 2.1, 4.1, 4.3]. \n\nWe explicitly compare the assumptions underlying DP and MIP in new table in the general comment. Regarding the validity of our theoretical assumptions, we conduct 4 ablation studies in the rebuttal PDF file (Figure 3). In all these cases, we find that our theoretical results accurately predict attack success.\n\n> It would be great to extend the FPR and TPR ranges in the plots in figure 4 to smaller values\n\nWe thank the reviewer for this constructive suggestion. We provide the curves in Figures 1 and 2 in the uploaded rebuttal PDF (Figure 1 shows individual runs up to the first data point). Our bounds hold very well in the low FPR regime. At very low values for the FPR, the standard errors (across multiple runs) become very large due to the small number of samples, but the empirical estimates always lie well within one standard error.\n\n> Is it assumed that $\\tau$ in the noise parameter includes the batch size? Algorithm 1 does not scale the noise by the batch size which is different to typical DP-SGD [Abadi et al 16].\n\nThis is correct. Different works use different parameterization of the noise (e.g., Abadi et al. and Dong et al. [3] also use different parameterizations). As in our paper, we denote the dimension-wise variance of the total noise that is added to the average gradient by $\\tau^2$. Thus, our parameter $\\tau$  can be converted to the $\\sigma$ used by Abadi et al. as follows: $\\tau^2 =  \\frac{\\sigma^2_{\\text{Abadi}}C^2}{n}$. If it would make the manuscript more accessible to the reviewer, we will be happy to express the added noise through the parameterization used by Abadi et al. in our manuscript.\n\n> Figure 4a. Purchase and CIFAR10 seem to exceed the analytical bound for low FPRs. Is this because of the earlier mentioned assumption that the challenge points are sampled from the distribution which for low FPRs may be already distributional outliers?\n\nOur results cover the entire trade-off curve including the low-FPR regimes. Our theoretical predictions are probabilistic and our empirical results are always within the statistical standard error. To make this more clear, we provide plots using more samples in the rebuttal PDF file. On the setup corresponding to Figure 4a in the main paper (Figure 2a in the rebuttal PDF) our results always reside within the standard error and match the theoretical prediction almost exactly at the earlier limit of $10^{-2}$. At very low values of FPR of $10^{-4}$ the standard errors (across multiple runs) become very large due to the small number of available samples to estimate the FPRs and TPRs.\n\n> Are the techniques for composing DP guarantees numerically also applicable in this setting?\n\nWe would like to highlight that our privacy notion already comes with powerful composition and post-processing results that were transferred from the work of Dong et al. [3]. We agree that the asymptotic result under subsampling and composition presented in Lemma 4.1. may potentially be improved using numeric estimation techniques similar to those proposed by Gopi et al. [2], which is a great suggestion for follow-up work.\n\n>  I believe the discussion about the validity of the initial assumptions is limited.\n\nTo further address this point, we provide a table detailing the different assumptions underlying various attacks and privacy notions in the general comment, which we will gladly include in our final manuscript.\n\n-------------------\n\nWe were happy to hear that the Reviewer found our work to be solid and sound in general, as mentioned in your review. We sincerely hope that we have addressed your remaining concerns regarding the low-FPR regime and the discussion of the initial threat models, leaving no major concerns unaddressed. In light of this response and your positive assessment, we would kindly ask you to consider updating the review score in light of this rebuttal and the changes listed in the general response.\n\n----------------\n\n**References**\n\n[1] Challenges towards the Next Frontier in Privacy, Cummings et al (2023),  arXiv:2304:06929\n\n[2] Sivakanth Gopi, Yin Tat Lee, and Lukas Wutschitz: Numerical Composition of Differential Privacy, arXiv:2106.02848\n\n[3] Dong, J., Roth, A., and Su, W. J. Gaussian differential privacy. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(1):3–37, 2022.\n\n"}, {"review_id": "fMqJHJtQbP", "reviewer": "Reviewer_vdUQ", "summary": "The proposed f-MIP method incorporates a practical membership inference attack threat model, offering easily interpretable privacy guarantees. This approach improves utility, especially when the attacker's capabilities are realistically constrained. Through a theoretical analysis of likelihood ratio-based membership inference attacks on noisy stochastic gradient descent (SGD), μ-Gaussian Membership Inference Privacy (μ-GMIP) is introduced. μ-GMIP requires less noise compared to the corresponding Gaussian differential privacy (GDP) guarantees, resulting in higher utility.", "questions": "As mentioned in the paper, it is necessary to focus on cases where the membership is accurately determined among those classified as members, which corresponds to achieving a high true positive rate (TPR) at low false positive rate (FPR) in threshold-type tests. Therefore, it would be logical to concentrate on scenarios with low FPR. However, the trade-off function is compared across the entire range [0, 1] rather than solely focusing on these low FPR cases.", "limitations": "-\tThe iid assumption is not realistic in practical scenarios, which suggests that the actual privacy guarantees would be lower than those stated in the paper.\n-\tSince the f-MIP method is specifically designed for SGD, it raises questions about its generalizability to other problem domains.\n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tThe concept of applying control over type I and type II errors to MIP is interesting.\n-\tThe proposed approach holds promise for enhancing privacy protection in SGD.\n-\tThe numerical study provides evidence that the proposed method outperforms noisy SGD in terms of utility, highlighting its potential for practical applications.\n", "weaknesses": "-\tThe paper lacks a theoretical exploration of the relationship between f-MIP and f-DP, which hinders a comprehensive understanding of the proposed method. For instance, while mu-GMIP and mu-GDP are compared at the same mu in Figure 1, it remains unclear whether they actually share the same mu value.\n-\tThe absence of a discussion on post-processing or the composition rule (in terms of mu) limits the practical applicability and usefulness of the proposed approach.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their review and for referring to our approach as interesting and promising. We will answer the specific questions below.\n\n> The paper lacks a theoretical exploration of the relationship between f-MIP and f-DP [...]  in Figure 1, it remains unclear whether they actually share the same mu value.\n\nThe value of $\\mu$ defines the possible trade-off curve attainable by an attacker. This can be seen as a proxy for susceptibility of a model to specific types of attack. Regarding Figure 1, we compute noise levels $\\tau$, such that the privacy notions share the same nominal value of $\\mu$. The required noise levels across both f-DP and f_MIP are different and shown in Table 4 (App.), where we show that more noise is required for $\\mu$-GDP than for $\\mu$-GMIP.  The premises underlying MIP and DP are different however. We refer to the general response where we explicitly compare the threat models more clearly.\n\nThe relationship between the $\\mu$ values from G-MIP and G-DP can also be made explicit through the following Corollary:\n\n**Corollary: Converting Trade-Offs between Gaussian-DP and Gaussian-MIP**\n\nUnder the usual setting of $K=d$ and the conditions stated in Corollary 4.1., the following conversions between the privacy parameter $\\mu$ of a single step of noisy SGD hold:\n\n1. Converting $\\mu_\\text{DP}$ into $\\mu_\\text{MIP}$\n\n$\\mu_{\\text{MIP}} = \\sqrt{\\frac{d}{n +\\frac{4C^2}{\\mu_{\\text{DP}}^2}+\\frac{1}{2}}}$\n\n2. Converting $\\mu_\\text{MIP}$ into $\\mu_\\text{DP}$\n\n$\\mu_{\\text{DP}} = \\begin{cases}\n    \\frac{2}{\\sqrt{\\frac{d}{\\mu_{\\text{MIP}}^2}-n -\\frac{1}{2}}}, & \\text{if } \\mu_{\\text{MIP}} < \\sqrt{\\frac{2d}{2n+1}} \\\\\\\\\n   \\infty, & \\text{else}\n  \\end{cases}$\n\n**Proof Sketch.** This result follows from solving Corollary 4.2 and the result from Dong et al. [1, last line on page 30] for differential privacy of a single SGD step, $\\mu_{\\text{DP}} = \\frac{1}{\\sigma} = \\frac{2C}{\\tau n}$ (where Dong et al.'s $\\sigma= \\frac{\\tau n}{2C}$ in our notation) for $\\tau$, setting the two terms equal, and solving for $\\mu_{\\text{MIP}}$ or $\\mu_{DP}$ correctly. The same strategy can be applied for composed results.\n\nWe observe that, as expected, $\\mu_{\\text{DP}}$ and $\\mu_{\\text{MIP}}$ increase and decrease mutually. For high values of $\\mu_{\\text{MIP}}$ (weak privacy without any additional noise added), no DP-guarantee can be found, while some level of GMIP privacy can be guaranteed.\n\n> The absence of a discussion on post-processing or the composition rule (in terms of mu) limits the practical applicability and usefulness of the proposed approach.\n\nWe would like to highlight that our notion of privacy enjoys powerful post-processing and composition results as discussed in Appendix C.3 of our paper. Due to the hypothesis formulation, these follow from Theorem 4 and Theorem 11 of Dong et al. [1] and are used in Lemma 5.1 of this work. For example, Dong et al. [1, Corollary 2] provide a result which explicitly states that the n-fold composition of $i=1,\\ldots, n$ steps which are $\\mu_i$-GDP each is $\\sqrt{\\mu_1^2+ \\ldots + \\mu_n^2}$-GDP. This result also holds for $\\mu$-GMIP.\n\n>  it would be logical to concentrate on scenarios with low FPR. However, the trade-off function is compared across the entire range [0, 1] rather than solely focusing on these low FPR cases\n\nOur theory covers the entire trade-off curve, including arbitrary low FPRs. Following the suggestion of the reviewer, we investigate this regime further and provide the observed trade-off curves in Figures 1 and 2 of the uploaded rebuttal PDF. We see that our bounds hold very well in the low FPR regime. At very low FPR values of $10^{-4}$ the standard errors (across multiple runs) become very large due to the small number of samples, but the empirical estimates always lie well within one standard error.\n\n> The iid assumption is not realistic in practical scenarios, which suggests that the actual privacy guarantees would be lower than those stated in the paper.\n\nNote that the assumption that the distribution of the gradients in subsequent gradient descent steps are independent makes our bounds more conservative (we are thus more cautious which is as desired from a privacy perspective and apply more noise than required and obtain more privacy). This is so as the attacker gets some redundant information about the samples in subsequent steps,  because the sample gradients are correlated as the model and the sample gradients stay very similar through small model updates. This is confirmed in Figure 3b, 3c, where the lower curves indicate that the attack is harder to execute for the attacker than predicted by our theory. \n\n> Since the f-MIP method is specifically designed for SGD, it raises questions about its generalizability to other problem domains.\n\nDP SGD is the workhorse to implement Differential privacy in machine learning [2] and the standard object of study in recent works in privacy-preserving ML literature (e.g., [1,3]). The derivation of potential other mechanisms to implement GMIP is a fruitful avenue for future work.\n\n------\n\nThank you for your thoughtful comments and constructive feedback, which we will gladly incorporate into our manuscript. We have addressed the key concerns including the relation between f-DP and f-MIP, composition theorems, and the low-FPR regimes. In light of our response, we kindly ask the reviewer to reconsider the overall assessment of this work.\n\n------\n\n[1] Dong, J., Roth, A., and Su, W. J. Gaussian differential privacy. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(1):3–37, 2022. \n\n[2] Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308–318, 2016.\n\n[3] Nasr, Milad, et al. \"Tight Auditing of Differentially Private Machine Learning.\" arXiv preprint arXiv:2302.07956 (2023)."}]

Author rebuttal:
We thank all reviewers for their constructive feedback, which allows for direct improvements of our work. Inspired by the reviewer comments, we plan to make the following changes and intend to use the additional page provided in the camera-ready version for the following additions:

   * We add a table comparing the assumptions and premises of  f-DP and our f-MIP privacy notion (shown below). We will additionally compare our suggested class of new membership inference attacks with other attacks from literature.
   * We empirically investigate the low-FPR regime using additional plots up to a FPR of $10^{-4}$ and find our empirical results to match our theory well.
   * We ran 4 new empirical studies on the effect of the size of the parameter vector $d$, the number of samples $n$, the cropping threshold $C$ and the type of gradient distribution. They confirm that 1) our theoretical bounds are highly accurate across parameter ranges, 2) that the CLT can in fact be applied to means of cropped random variables (addressing Reviewer ``DVNo``s concern) and 3) that our theory holds with minor ramifications when $n$ and $d$ are extremely small. To explain this (surprising) behavior we provide theoretical insights on the convergence of the errors  in the response to Reviewer ``qYbt``.
   * In Response to Reviewer ``vdUQ``, we provide a new Corollary relating the values of the parameter $\mu$ between $\mu$-GDP and $\mu$-MIP. 

We would additionally like to stress that our work makes *several impactful contributions*:

1. We are the first to formulate membership inference as a hypothesis testing problem and use the trade-off function of the test to define a versatile notion of f-Gaussian Membership Inference Privacy. Such results are highly sought after in the privacy literature, where the need to reconsider threat models has recently been proclaimed by leading researchers in the field [1, sections 2.1, 4.1, 4.3].
2. Our hypothesis test formulation is valuable, as it allows for a fine-grained theoretical analysis of membership inference attacks. Our main contributions are Theorem 5.1 and Corollary 5.1, which precisely quantify the factors that lead to membership inference attack success in a step of stochastic gradient descent. Our results rely on constructing the most powerful test and are remarkably general: They also cover all ML models trained with standard gradient-based optimization, even without noise or gradient cropping. 
3. Notably, through this formulation we can transfer composition and post-processing results from existing literature. We apply those to steps of noisy stochastic gradient descent to bound the attack success for training of an entire model
4. Based on our theoretical insights, our final contribution consists of a constructive algorithm that quantifies the required noise level in SGD to defend against membership inference attacks.

Thank you again for your feedback. We will be happy to answer any further questions.



-------------------------


**Table: Comparing $f$-Differential Privacy and $f$-Membership Inference Privacy**

|                     | $f$-Differential Privacy                                                                                                                                                                    | $f$-Membership Inference Privacy                                                                                                                                                 |
|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Adversary goal      | Distinguish between $D$ and $D^\prime$ for any $D$, $D^\prime$ that differ in at most one instance                                                                                          | Distinguish whether $x^\prime \in S$ (training data set) or not.                                                                                                                 |
| Data access         | Attacker has full dataset access. For example, the attacker can poison or adversarially construct datasets on which ML models could be trained; e.g., $D = \\{ \\}$ and $D^\prime = \\{100000\\}$ | Attacker has no access to the training data set; i.e., the model owner privately trains their model free of adversarially poisoned samples.                                      |
| Protected instances | The instance in which $D$ and $D^\prime$ differ is arbitrary. This includes OOD samples and extreme outliers                                                                                | The sample $x^\prime$ for which membership is to be inferred is drawn from the data distribution $D$. Therefore, MI is concerned with typical samples that can occur in practice |
| Model knowledge     | The attacker knows the full model architecture including hyperparameters and has full access to samples from the distribution and the parameters during training                             | The attacker knows the full model architecture including hyperparameters and has full access to samples from the distribution and the parameters during training                  |


**References** 

[1] Challenges towards the Next Frontier in Privacy, Cummings et al (2023),  arXiv:2304:06929


Meta-review: The reviewers appreciate the paper's insightful approach to analyzing information leakage in training algorithms, particularly praising the new GMIP bound as a significant addition to the field. Although there seems to be some overlap with existing concepts from statistical hypothesis testing, the fresh perspectives and thorough empirical evaluations give the paper a strong position. The meta-reviewer recommends acceptance as a poster, anticipating that a few tweaks could further amplify the paper's impact within the community.
Decision: Accept (poster)