SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tobias Leemann*
University of Tübingen Technical University of Munich
Martin Pawelczyk*
Gjergji Kasneci
We propose a novel and practical privacy notion called  $f$ -Membership Inference Privacy ( $f$ -MIP), which explicitly considers the capabilities of realistic adversaries under the membership inference attack threat model. Consequently,  $f$ -MIP offers interpretable privacy guarantees and improved utility (e.g., better classification accuracy). In particular, we derive a parametric family of  $f$ -MIP guarantees that we refer to as  $\mu$ -Gaussian Membership Inference Privacy ( $\mu$ -GMIP) by theoretically analyzing likelihood ratio-based membership inference attacks on stochastic gradient descent (SGD). Our analysis highlights that models trained with standard SGD already offer an elementary level of MIP. Additionally, we show how  $f$ -MIP can be amplified by adding noise to gradient updates. Our analysis further yields an analytical membership inference attack that offers two distinct advantages over previous approaches. First, unlike existing state-of-the-art attacks that require training hundreds of shadow models, our attack does not require any shadow model. Second, our analytical attack enables straightforward auditing of our privacy notion  $f$ -MIP. Finally, we quantify how various hyperparameters (e.g., batch size, number of model parameters) and specific data characteristics determine an attacker's ability to accurately infer a point's membership in the training set. We demonstrate the effectiveness of our method on models trained on vision and tabular datasets.
Machine learning (ML) has seen a surge in popularity and effectiveness, leading to its widespread application across various domains. However, some of these domains, such as finance and healthcare, deal with sensitive data that cannot be publicly shared due to ethical or regulatory concerns. Therefore, ensuring data privacy becomes crucial at every stage of the ML process, including model development and deployment. In particular, the trained model itself [5, 31] or explanations computed to make the model more interpretable [29, 32] may leak information about the training data if appropriate measures are not taken. For example, this is a problem for recent generative Diffusion Models [7] and Large Language models, where the data leakage seems to be amplified by model size [6].
Differential privacy (DP) [14] is widely acknowledged as the benchmark for ensuring provable privacy in academia and industry [10]. DP utilizes randomized algorithms during training and guarantees that the output of the algorithm will not be significantly influenced by the inclusion or exclusion of any individual sample in the dataset. This provides information-theoretic protection against the maximum amount of information that an attacker can extract about any specific sample in the dataset, even when an attacker has full access to and full knowledge about the predictive model.
While DP is an appealing technique for ensuring privacy, DP's broad theoretical guarantees often come at the expense of a significant loss in utility for many ML algorithms. This utility loss cannot be further reduced by applying savvier algorithms: Recent work [26, 27] confirms that an attacker can
*Equal contribution. Corresponding authors: tobias.leemann@uni-tuebingen.de and martin.pawelczyk.1@gmail.com.
be implemented whose empirical capacity to differentiate between neighboring datasets  $D$  and  $D'$  when having access to privatized models matches the theoretical upper bound. This finding suggests that to improve a model's utility, we need to take a step back and inspect the premises underlying DP. For example, previous work has shown that privacy attacks are much weaker when one imposes additional realistic restrictions on the attacker's capabilities [26].
In light of these findings, we revisit the DP threat model and identify three characteristics of an attacker that might be overly restrictive in practice. First, DP grants the attacker full control over the dataset used in training including the capacity to poison all samples in the dataset. For instance, DP's protection includes pathological cases such as an empty dataset and a dataset with a single, adversarial instance [27]. Second, in many applications, it is more likely that the attacker only has access to an API to obtain model predictions [13, 31] or to model gradients [20]. Finally, one may want to protect typical samples from the data distribution. As argued by Triastcyn & Faltings [38], it may be over-constraining to protect images of dogs in a model that is conceived and trained with images of cars.
Such more realistic attackers have been studied in the extensive literature on Membership Inference (MI) attacks (e.g., [5, 41]), where the attacker attempts to determine whether a sample from the data distribution was part of the training dataset. Under the MI threat model, Carlini et al. [5] observe that ML models with very lax ( $\epsilon > 5000$ ) or no ( $\epsilon = \infty$ ) DP-guarantees still provide some defense against membership inference attacks [5, 41]. Hence, we hypothesize that standard ML models trained with low or no noise injection may already offer some level of protection against realistic threats such as MI, despite resulting in very large provable DP bounds.
To build a solid groundwork for our analysis, we present a hypothesis testing interpretation of MI attacks. We then derive  $f$ -Membership Inference Privacy ( $f$ -MIP), which bounds the trade-off between an MI attacker's false positive rate (i.e., FPR, type I errors) and false negative rate (i.e., FNR, type II errors) in the hypothesis testing problem by some function  $f$ . We then analyze the privacy leakage of a gradient update step in stochastic gradient descent (SGD) and derive the first analytically optimal attack based on a likelihood ratio test. However, for  $f$ -MIP to cover practical scenarios, post-processing and composition operations need to be equipped with tractable privacy guarantees as well. Using  $f$ -MIP's handy composition properties, we analyze full model training via SGD and derive explicit  $f$ -MIP guarantees. We further extend our analysis by adding carefully calibrated noise to the SGD updates to show that  $f$ -MIP may be guaranteed without any noise or with less noise than the same parametric level of  $f$ -DP [11], leading to a smaller loss of utility.
Our analysis comes with a variety of novel insights: We confirm our hypothesis that, unlike for DP, no noise  $(\tau^2 = 0)$  needs to be added during SGD to guarantee  $f$ -MIP. Specifically, we prove that the trade-off curves of a single SGD step converge to the family of Gaussian trade-offs identified by Dong et al. [11] and result in the more specific  $\mu$ -Gaussian Membership Inference Privacy ( $\mu$ -GMIP). The main contributions this research offers to the literature on privacy preserving ML include:
1. Interpretable and practical privacy notion: We suggest the novel privacy notion of  $f$ -MIP that addresses the realistic threat of MI attacks.  $f$ -MIP considers the MI attacker's full trade-off curve between false positives and false negatives. Unlike competing notions,  $f$ -MIP offers appealing composition and post-processing properties.
2. Comprehensive theoretical analysis: We provide (tight) upper bounds on any attacker's ability to run successful MI attacks, i.e., we bound any MI attacker's ability to identify whether points belong to the training set when ML models are trained via gradient updates.
3. Verification and auditing through novel attacks: As a side product of our theoretical analysis, which leverages the Neyman-Pearson lemma, we propose a novel set of attacks for auditing privacy leakages. An important advantage of our analytical Gradient Likelihood-Ratio (GLiR) attack is its computational efficiency. Unlike existing attacks that rely on training hundreds of shadow models to approximate the likelihood ratio, our attack does not require any additional training steps.
4. Privacy amplification through noise addition: Finally, our analysis shows how one can use noisy SGD (also known as Differentially Private SGD [1]) to reach  $f$ -MIP while maintaining worst-case DP guarantees. Thereby our work establishes a theoretical connection between  $f$ -MIP and  $f$ -DP [11], which allows to translate an  $f$ -DP guarantee into an  $f$ -MIP guarantee and vice versa.
Privacy notions. DP and its variants provide robust, information-theoretic privacy guarantees by ensuring that the probability distribution of an algorithm's output remains stable even when one sample of the input dataset is changed [14]. For instance, a DP algorithm is  $\varepsilon$ -DP if the probability of the algorithm outputting a particular subset  $E$  for a dataset  $S$  is not much higher than the probability of outputting  $E$  for a dataset  $S_0$  that differs from  $S$  in only one element. DP has several appealing features, such as the ability to combine DP algorithms without sacrificing guarantees.
A few recent works have proposed to carefully relax the attacker's capabilities in order to achieve higher utility from private predictions [4, 13, 17, 38]. For example, Dwork & Feldman [13] suggest the notion of "privacy-preserving prediction" to make private model predictions through an API interface. Their work focuses on PAC learning guarantees of any class of Boolean functions. Similarly, Triastcyn & Faltings [38] suggest "Bayesian DP", which is primarily based on the definition of DP, but restricts the points in which the datasets  $S$  and  $S_0$  may differ to those sampled from the data distribution. In contrast, Izzo et al. [17] introduces a notion based on MI attacks, where their approach guarantees that an adversary  $\mathcal{A}$  does not gain a significant advantage in terms of accuracy when distinguishing whether an element  $x$  was in the training data set compared to just guessing the most likely option. However, they only constrain the accuracy of the attacker, while we argue that it is essential to bound the entire trade-off curve, particularly in the low FPR regime, to prevent certain re-identification of a few individuals [5]. Our work leverages a hypothesis testing formulation that covers the entire trade-off curve thereby offering protection also to the most vulnerable individuals. Additionally, our privacy notion maintains desirable properties such as composition and privacy amplification through subsampling, which previous notions did not consider.
Privacy Attacks on ML Models. Our work is also related to auditing privacy leakages through a common class of attacks called MI attacks. These attacks determine if a given instance is present in the training data of a particular model [5, 7, 9, 15, 23, 29, 30, 31, 32, 35, 36, 40, 41]. Compared to these works, our work suggests a new much stronger class of MI attacks that is analytically derived and uses information from model gradients. An important advantage of our analytically derived attack is its computational efficiency, as it eliminates the need to train any additional shadow models.
The classical notion of  $(\varepsilon, \delta)$ -differential privacy [14] is the current workhorse of private ML and can be described as follows: An algorithm is DP if for any two neighboring datasets  $S, S'$  (that differ by one instance) and any subset of possible outputs, the ratio of the probabilities that the algorithm's output lies in the subset for inputs  $S, S'$  is bounded by a constant factor. DP is a rigid guarantee, that covers every pair of datasets  $S$  and  $S'$ , including pathologically crafted datasets (for instance, Nasr et al. [27] use an empty dataset) that might be unrealistic in practice. For this reason, we consider a different attack model in this work: The MI game [41]. This attack mechanism on ML models follows the goal of inferring an individual's membership in the training set of a learned ML model. We will formulate this problem using the language of hypothesis testing and trade-off functions, a concept from hypothesis testing theory [11]. We will close this section by giving several useful properties of trade-off functions which we leverage in our main theoretical results presented in Sections 4 and 5.
3.1 Membership Inference Attacks
The overarching goal of privacy-preserving machine learning lies in protecting personal data. To this end, we will show that an alternative notion of privacy can be defined through the success of a MI attack which attempts to infer whether a given instance was present in the training set or not. Following Yeom et al. [41] we define the standard MI experiment as follows:
Definition 3.1 (Membership Inference Experiment [41]). Let  $\mathcal{A}$  be an attacker,  $A$  be a learning algorithm,  $N$  be a positive integer, and  $\mathcal{D}$  be a distribution over data points  $\pmb{x} \in D$ , where the vector  $\pmb{x}$  may also be a tuple of data and labels. The MI experiment proceeds as follows: The model and data owner  $\mathcal{O}$  samples  $S \sim \mathcal{D}_N$  (i.e., sample  $n$  points i.i.d. from  $\mathcal{D}$ ) and trains  $A_S = A(S)$ . They choose  $b \in \{0,1\}$  uniformly at random and draw  $\pmb{x}' \sim \mathcal{D}$  if  $b = 0$ , or  $\pmb{x}' \sim S$  if  $b = 1$ . Finally, the attacker is successful if  $\mathcal{A}(\pmb{x}', A_S, N, \mathcal{D}) = b$ .  $\mathcal{A}$  must output either 0 or 1.
We note that the membership inference threat model features several key differences to the threat model underlying DP, which are listed in Table 1. Most notably, in MI attacks, the datasets are sampled from the distribution  $\mathcal{D}$ , whereas DP protects all datasets. This corresponds to granting the attacker the capacity of full dataset manipulation. Therefore, the MI attack threat model is sensible in cases where the attacker cannot manipulate the dataset through injection of malicious samples also called "canaries". This may be realistic for financial and healthcare applications, where the data is often collected from actual events (e.g., past trades) or only a handful of people (trusted hospital staff) have access to the records. In such scenarios, it might be overly restrictive to protect against worst-case canary attacks as attackers cannot freely inject arbitrary records into the training datasets. Furthermore, MI attacks are handy as a fundamental ingredient in crafting data extraction attacks [6]. Hence we expect a privacy notion based on the MI threat model to offer protection against a broader class of reconstruction attacks. Finally, being an established threat in the literature [5, 9, 31, 40, 41], MI can be audited through a variety of existing attacks.
f-DP threat model	f-MIP threat model (this work)
Goal	Distinguish between S and S' for any S, S' that differ in at most one instance.	Distinguish whether x' ∈ S (training data set) or not.
Dataset access	Attacker has full data access. For example, the attacker can poison or adversarially construct datasets on which ML models could be trained; e.g., S = {} and S' = {10^6}.	Attacker has no access to the training data set; i.e., the model owner privately trains their model free of adversarially poisoned samples.
Protected Instances	The instance in which S and S' differ is arbitrary. This includes OOD samples and extreme outliers.	The sample x' for which membership is to be inferred is drawn from the data distribution D. Therefore, MI is concerned with typical samples that can occur in practice.
Best used	When the specific attack model is unknown. Offers a form of general protection.	When dataset access (e.g. canary injection) of an attacker can be ruled out and the main attack goal lies in revealing private training data (e.g., membership inference, data reconstruction).
Model knowledge	The attacker knows the model architecture and has full access to the model in form of its parameters, hyperparameters and its model outputs.
Table 1: Comparing the threat models underlying  $f$  -DP and  $f$  -MIP.
3.2 Membership Inference Privacy as a Hypothesis Testing Problem
While DP has been studied through the perspective of a hypothesis testing formulation for a while [3, 11, 19, 39], we adapt this route to formulate membership inference attacks. To this end, consider the following hypothesis test:
$$
\mathrm {H} _ {0}: \boldsymbol {x} ^ {\prime} \notin S \text {v s .} \mathrm {H} _ {1}: \boldsymbol {x} ^ {\prime} \in S. \tag {1}
$$
Rejecting the null hypothesis corresponds to detecting the presence of the individual  $x'$  in  $S$ , whereas failing to reject the null hypothesis means inferring that  $x'$  was not part of the dataset  $S$ . The formulation in (1) is a natural vehicle to think about any attacker's capabilities in detecting members of a train set in terms of false positive and true positive rates. The motivation behind these measures is that the attacker wants to reliably identify the subset of data points belonging to the training set (i.e., true positives) while incurring as few false positive errors as possible [5]. In other words, the attacker wants to maximize their true positive rate at any chosen and ideally low false positive rate (e.g., 0.001). From this perspective, the formulation in (1) allows to define membership inference privacy via trade-off functions  $f$  which exactly characterize the relation of false negative rates (i.e., 1-TPR) and false positive rates that an optimal attacker can achieve.
Definition 3.2. (Trade-off function [11]) For any two probability distributions  $P$  and  $Q$  on the same space, denote the trade-off function  $\operatorname{Test}(P;Q) : [0;1] \to [0;1]$
$$
\operatorname {T e s t} (P; Q) (\alpha) = \inf  \left\{F N R \mid F P R = \alpha \right\}, \tag {2}
$$
where the infimum is taken over all (measurable) rejection rules ("tests") which lead to a FPR of  $\alpha$  between distributions  $P, Q$ .
Not every function makes for a valid trade-off function. Instead, trade-off functions possess certain characteristics that are handy in their analysis.
Definition 3.3 (Characterization of trade-off functions [11]). A function  $f:[0,1]\to [0,1]$  is a trade-off function if  $f$  is convex, continuous at zero, non-increasing, and  $f(r)\leq 1 - r$  for  $r\in [0,1]$ .
We additionally introduce a semi-order on the space of trade-off functions to make statements on the hardness of different trade-offs in relation to each other.
Definition 3.4 (Comparing trade-offs). A trade-off function  $f$  is uniformly at least as hard as another trade-off function  $g$ , if  $f(r) \geq g(r)$  for all  $0 \leq r \leq 1$ . We write  $f \geq g$ .
If  $\operatorname{Test}(P;Q) \geq \operatorname{Test}(P';Q')$ , testing  $P$  vs  $Q$  is uniformly at least as hard as testing  $P'$  vs  $Q'$ . Intuitively, this means that for a given FPR  $\alpha$ , the best test possible test on  $(P;Q)$  will result in an equal or higher FNR than the best test on  $(P';Q')$ .
3.3 Noisy Stochastic Gradient Descent (Noisy SGD)
Most recent large-scale ML models are trained via stochastic gradient descent (SGD). Noisy SGD (also known as DP-SGD) is a variant of classical SGD that comes with privacy guarantees. We consider the algorithm as in the work by Abadi et al. [1], which we restate for convenience in Appendix A. While its characteristics with respect to DP have been extensively studied, we take a fundamentally different perspective in this work and study the capabilities of this algorithm to protect against membership inference attacks. In summary, the algorithm consists of three fundamental steps: gradient clipping (i.e.,  $\theta_{i} \coloneqq g(\pmb{x}_{i}, y_{i}) \cdot \max(1, C / \|g(\pmb{x}_{i}, y_{i})\|)$  where  $g(\pmb{x}_{i}, y_{i}) = \nabla \mathcal{L}(\pmb{x}_{i}, y_{i})$  is the gradient with respect to the loss function  $\mathcal{L}$ ), aggregation (i.e.,  $m = \frac{1}{n} \sum_{i=1}^{n} \theta_{i}$ ) and adding Gaussian noise (i.e.,  $\tilde{m} = m + Y$  where  $Y \sim \mathcal{N}(0, \tau^2 I)$  with variance parameter  $\tau^2$ ). To obtain privacy bounds for this algorithm, we study MI attacks for means of random variables. This allows us to bound the MI vulnerability of SGD.
4 Navigating Between Membership Inference Privacy and DP
In this section, we formally define our privacy notion  $f$ -MIP. To this end, it will be handy to view MI attacks as hypothesis tests.
4.1 Membership Inference Attacks from a Hypothesis Testing Perspective
Initially, we define the following distributions of the algorithm's output
$$
A _ {0} = A \left(\boldsymbol {X} \cup \left\{\boldsymbol {x} \right\}\right) \text {w i t h} \boldsymbol {X} \sim \mathcal {D} ^ {n - 1}, \boldsymbol {x} \sim \mathcal {D} \text {a n d} A _ {1} \left(\boldsymbol {x} ^ {\prime}\right) = A \left(\boldsymbol {X} \cup \left\{\boldsymbol {x} ^ {\prime} \right\}\right) \text {w i t h} \boldsymbol {X} \sim \mathcal {D} ^ {n - 1}, \tag {3}
$$
where we denote other randomly sampled instances that go into the algorithm by  $\mathbf{X} = \{\pmb{x}_1, \dots, \pmb{x}_{n-1}\}$ . Here  $A_0$  represents the output distribution under the null hypothesis  $(H_0)$  where the sample  $\pmb{x}'$  is not part of the training dataset. On the other hand,  $A_1$  is the output distribution under the alternative hypothesis  $(H_1)$  where  $\pmb{x}'$  was part of the training dataset. The output contains randomness due to the instances drawn from the distribution  $\mathcal{D}$  and due to potential inherent randomness in  $A$ .
We observe that the distribution  $A_{1}$  depends on the sample  $x'$  which is known to the attacker. The attacker will have access to samples for which  $A_{0}$  and  $A_{1}(x')$  are simpler to distinguish and others where the distinction is harder. To reason about the characteristics of such a stochastically composed test, we define a composition operator that defines an optimal test in such a setup. To obtain a global FPR of  $\alpha$ , an attacker can target different FPRs  $\bar{\alpha}(x')$  for each specific test. We need to consider the optimum over all possible ways of choosing  $\bar{\alpha}(x')$ , which we refer to as test-specific FPR function, giving rise to the following definition.
Definition 4.1 (Stochastic composition of trade-off functions). Let  $\mathcal{F}$  be a family of trade-off functions,  $h: D \subset \mathbb{R}^d \to \mathcal{F}$  be a function that maps an instance of the data domain to a corresponding trade-off function, and  $\mathcal{D}$  be a probability distribution on  $D$ . The set of valid test-specific FPR functions  $\bar{\alpha}: D \to [0,1]$  that result in a global FPR of  $\alpha \in [0,1]$  given the distribution  $\mathcal{D}$  is defined through
$$
\mathcal {E} (\alpha , \mathcal {D}) = \left\{\bar {\alpha}: D \rightarrow [ 0, 1 ] \mid \mathbb {E} _ {\boldsymbol {x} ^ {\prime} \sim \mathcal {D}} \left[ \bar {\alpha} \left(\boldsymbol {x} ^ {\prime}\right)\right] = \alpha \right\}. \tag {4}
$$
For a given test-specific FPR function,  $\bar{\alpha}$  the global false negative rate (type II error)  $\beta$  is given by
$$
\beta_ {h} (\bar {\alpha}) = \mathbb {E} _ {\boldsymbol {x} ^ {\prime} \sim \mathcal {D}} [ h (\boldsymbol {x}) (\bar {\alpha} (\boldsymbol {x})) ], \tag {5}
$$
where  $\bar{\alpha} (\pmb {x})$  is the argument to the trade-off function  $h(\pmb {x})\in \mathcal{F}$  . For a global  $\alpha \in [0,1]$  the stochastic composition of these trade-functions is defined as
$$
\left(\bigotimes_ {\boldsymbol {x} \sim \mathcal {D}} h (\boldsymbol {x})\right) (\alpha) = \min  _ {\bar {\alpha} \in \mathcal {E} (\alpha , \mathcal {D})} \left\{\beta_ {h} (\bar {\alpha}) \right\}, \tag {6}
$$
(supposing the minimum exists), the smallest global false negative rate possible at a global FPR of  $\alpha$ .