SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Georgios Kaissis *
AI in Medicine and Healthcare
g.kaissis@tum.de
Alexander Ziller
alex.ziller@tum.de
Stefan Kolek
Mathematical Foundations of AI
kolek@math.lmu.de
Anneliese Riess
Machine Learning in Biomedical Imaging
Helmholtz Munich
Neuherberg, Germany
anneliese.riess@helmholtz-munich.de
Daniel Rueckert
daniel.rueckert@tum.de
Differentially private mechanisms restrict the membership inference capabilities of powerful (optimal) adversaries against machine learning models. Such adversaries are rarely encountered in practice. In this work, we examine a more realistic threat model relaxation, where (sub-optimal) adversaries lack access to the exact model training database, but may possess related or partial data. We then formally characterise and experimentally validate adversarial membership inference capabilities in this setting in terms of hypothesis testing errors. Our work helps users to interpret the privacy properties of sensitive data processing systems under realistic threat model relaxations and choose appropriate noise levels for their use-case.
Machine learning (ML) offers a powerful set of techniques for addressing complex problems across various fields, including fields where sensitive data is processed, e.g. medicine or finance. However, the use of sensitive data in ML comes with privacy challenges: It is known that ML models memorise their training data [1], and that some degree of memorisation may be necessary for the best model performance [2]. When a model memorises private information, it can reveal this information when attacked by adversaries. A very important class of attack is Membership Inference (MI) [3], which aims to determine whether the data of a specific individual was used to train a machine learning model. MI can lead to the disclosure of highly sensitive information about the individual, e.g. that
*GK is also with the Institute of Machine Learning in Biomedical Imaging, Helmholtz Munich and the Department of Computing, Imperial College London.
they are part of a database of cancer patients which was used to train a predictive medical ML system. The importance of studying MI is furthermore derived from the fact that a successful MI attack signifies that the model is also vulnerable to all other attacks such as data reconstruction or attribute inference ("most significant bit" property [4]).
Differential Privacy (DP) [5] is a formal framework and collection of techniques to furnish objective privacy guarantees for ML model training. It is a natural counterpart to MI, and it is possible to interpret DP entirely through the MI lens, which is known as the hypothesis testing interpretation of DP [6, 7]. Given a database  $D$  and a database  $D'$  which differ in exactly one record, hypothesis testing DP quantifies the trade-offs between the Type-I and Type-II errors of an adversary using a hypothesis test to determine whether a model was trained on  $D$  or on  $D'$ . This hypothesis test is considered to be optimal in the sense of the Neyman-Pearson (NP) lemma [8], which makes the DP Threat Model (DPTM) very strong: It assumes that the adversary has full access to the training database, knowledge of the specifications of the DP mechanism and can even deeply manipulate the process of ML model training itself to their advantage [9]. Although this has the benefit that DP is a true "worst-case" guarantee and thus holds for all weaker adversaries, it may be unnecessarily conservative in practice. For instance, the assumption that the adversary has full access to the training data may be unrealistic in settings where this data is tightly access-restricted. As a consequence, previous works also considered threat models with weaker adversaries, i.e. Relaxed Threat Models (RTMs), and experimentally showed that the empirical protection against MI offered by DP mechanisms is much stronger than the upper bound predicted by the DP analysis [9, 10, 11]. While such empirical investigations provide valuable initial insights, we contend that an additional formal analysis of specific, practically relevant RTMs can greatly benefit stakeholders in conducting a thorough privacy analysis of ML systems. For example, a "privacy certificate" for an ML model can include both (1) the MI risk under the DPTM (which holds when "all bets are off") but also (2) the more realistic MI risk under whichever RTM is applicable to the actual model training conditions.
In this work, we study MI in the context of a specific RTM with high practical relevance, in which the adversary has incomplete access to the model's training data. An example of this RTM is the training of a predictive ML model on medical data which is generated and stored at a single hospital and never released, whereas the model is shared with third parties. Here, an MI adversary must resort to leveraging auxiliary data from the same distribution as the hospital's target database. A similar attack (offline MI) was studied in [12, 13, 14], and also does not use the full database for the attack (albeit for reasons of computational efficiency). As MI is equivalent to a hypothesis testing problem, the aforementioned works demonstrate that the Type-I and Type-II errors (i.e. the false positive/false negative rates) of offline MI attacks are higher than MI attacks assuming the DPTM. In this work, we formally derive and validate exact bounds on the error rates of MI attacks against DP noise mechanisms in the aforementioned RTM (which we will refer to as "the RTM" from now on).
Contributions Our contributions are as follows: (1) Since MI can be fully characterised using hypothesis testing, we begin by formalising the RTM as a hypothesis testing security game and show that the RTM adversary is strictly sub-optimal in terms of error rates compared to the DPTM adversary; (2) Next, we formally derive the "best sub-optimal" hypothesis testing MI strategy in the RTM; (3) We then analyse the error rates associated with this strategy against common additive noise mechanisms used in DP under the RTM. (4) We express these error rates using trade-off functions, which parameterise all achievable Type-I/Type-II error combinations, similar to the approach taken by [7] in defining f-DP. (5) Additionally, we introduce a new metric,  $\lambda$ -Detection Resilience ( $\lambda$ -DR), which allows us to compare the hypothesis testing capabilities of the RTM and DPTM adversaries, despite their differing threat models. (6) We conclude our study by empirically auditing the proposed bounds and demonstrating accuracy advantages in the training of deep neural networks with DP-SGD.
Related work Our work contributes to the ongoing research in three distinct, yet related areas of privacy-preserving ML. Firstly, we aim to augment empirical investigations which use MI to establish lower bounds on the privacy guarantees of ML systems, with formal guarantees. For instance, [9] experimentally evaluate several types of adversaries against DP ML workflows, [10, 11] present empirical privacy analyses of DP-SGD [15], while [16] focus on federated learning. Such empirical privacy auditing techniques have found their way into software tools such as ML-Doctor [17] or ML
Privacy Meter [14]. As mentioned above, our RTM considers an adversary performing MI without access to the (complete) model's training data. This type of MI attack has been studied in several prior works, most notably by Watson et al. [13], Ye et al. [14] and Carlini et al. [12], who find the adversary's hypothesis testing capabilities to be substantially diminished compared to the DPTM. The second line of work related to our paper are formal analyses of MI in the DPTM, such as [18], who consider a Bayesian adversary or [4], who bypass the DP analysis and directly study the properties of the underlying distributions. Moreover, [19] and [20] are rooted in hypothesis testing theory like our work, but only consider threat models equivalent to the DPTM, while [21] consider restricted adversaries, albeit not through the hypothesis testing lens. Lastly, we adopt the recent development of using functions (rather than numbers) to parameterise privacy guarantees, which is becoming common practice in the DPTM [7, 22, 23], but has not yet been used to study threat model relaxations. Our paper unifies these strands of work by introducing a formal guarantee in the RTM, which allows for a more nuanced analysis of the privacy risks associated with machine learning models.
Differential privacy DP [5] is a stability condition on randomised mechanisms requiring that their outcomes should be approximately equally likely when the data of a single individual is added or removed from the input database. Throughout this work, we will consider the scenario of ML model training, where DP is realised through an additive noise mechanism. We consider a deterministic model training function  $q$ . The global sensitivity  $\Delta > 0$  of  $q$  is defined as  $\Delta = \sup_{D, D': D \simeq D'} \| q(D) - q(D') \|$  where the sup is taken over all pairs of neighbouring databases  $D \simeq D'$  which differ in a single record.  ${}^2\Delta$  is measured in some norm, which we will indicate by subscript wherever relevant (e.g.  $\Delta_1$ ). An additive noise mechanism (or just mechanism)  $\mathcal{M}$  maps databases and probability distributions to a (model) parameter space  $\Theta$ . Its outputs are of the form  $\theta = q(X) + z, z \sim \mathcal{Z}$ , where  $\mathcal{Z}$  is a noise distribution. Since  $q$  and the database are considered deterministic,  $\theta$  is a random variable induced by the randomness of  $\mathcal{Z}$ . Unless otherwise indicated,  $\mathcal{M}$  will be realised through distributions  $\mathcal{Z}(\mu, \xi)$  supported on the real line which admit a density. We assume that  $\xi$  is fixed and public information and that  $\mathcal{Z}$  is shift invariant. In other words,  $\mathcal{M}(D) = (q(D) + \mathcal{Z}(0, \xi)) \sim \mathcal{Z}(q(D), \xi)$  and  $\mathcal{M}(D') = (q(D') + \mathcal{Z}(0, \xi)) \sim \mathcal{Z}(q(D'), \xi)$ .
In the hypothesis testing interpretation of DP [6, 7, 24], the adversary attempts to determine whether  $\theta$  was trained on  $D$  or  $D'$  using a classical/frequentist hypothesis test. The adversary considers the hypotheses  $\mathcal{H}_0: \theta \sim \mathcal{M}(D)$  vs.  $\mathcal{H}_1: \theta \sim \mathcal{M}(D')$  and their reverse, i.e.  $\mathcal{H}_0: \theta \sim \mathcal{M}(D')$  vs.  $\mathcal{H}_1: \theta \sim \mathcal{M}(D)$ .  $\mathcal{H}_0$  and  $\mathcal{H}_1$  are called the null and alternative hypothesis, respectively. The aim of the hypothesis testing problem is to distinguish between  $\mathcal{M}(D)$  and  $\mathcal{M}(D')$  using a pair of decision rules (tests). Let  $\phi, \phi'$  be tests with Type-I error rate  $\alpha_{\phi} = \mathbb{E}_{\mathcal{M}(D)}(\phi)$  and Type-II error rate  $\beta_{\phi} = 1 - \mathbb{E}_{\mathcal{M}(D')}(\phi)$  (identical for  $\phi'$ ). The adversary's goal will be to maximise their power  $1 - \beta$  at a fixed level  $\alpha$ , since the fundamental trade-off in hypothesis testing is that  $\alpha$  and  $\beta$  cannot be minimised simultaneously [25]. All achievable  $\alpha/\beta$  pairs for a given adversary/test combination are expressed by a pair of functions on the unit square, called the trade-off functions  $T$  and  $T^{-1}$ .
$T$  is defined through the test  $\phi$ :  $\mathcal{M}(D)$  vs.  $\mathcal{M}(D')$  as  $T(\mathcal{M}(D), \mathcal{M}(D'))(\alpha) = \inf_{\phi} \{\beta_{\phi} : \alpha_{\phi} \leq \alpha\}$ , whereas  $T^{-1} = T(\mathcal{M}(D'), \mathcal{M}(D))$  is defined using  $\phi'$ . The inf is taken over the set of tests which can be constructed in a given threat model, in this case the DPTM. For mechanisms with symmetric noise density,  $T = T^{-1}$ , allowing us to consider only a single trade-off function. In general, one takes the symmetrisation/convexification of  $T$  and  $T^{-1}$ , denoted  $\mathrm{C}(T, T^{-1})$  which creates a single, symmetric trade-off function. Since trade-off functions fully characterise the adversary's error rates, they can be used to compare the privacy guarantees of different mechanisms and/or different threat models. In particular, the closer the trade-off function's graph is to the line  $\beta(\alpha) = 1 - \alpha$ , i.e. the off-diagonal of the unit square, the stronger the implied privacy guarantee of the mechanism is. The notion of expressing DP guarantees based on a comparison of  $\mathrm{C}(T, T^{-1})$  with a "reference" trade-off function  $f$  is called f-DP [7]:
Definition 1. A mechanism  $\mathcal{M}$  satisfies  $f$ -DP, if  $\forall D, D': D \simeq D'$  it holds that  $\mathrm{C}(T, T^{-1})(\alpha) \geq f(\alpha) \forall \alpha \in [0,1]$  for a reference trade-off function  $f$  (see Figure 2 of [7] for an example).
A pair of distributions  $\mathcal{Z}(\mu_1,\xi),\mathcal{Z}(\mu_2,\xi)$  is called a dominating pair [22] for  $\mathcal{M}$  if,  $\forall D,D^{\prime}:D\simeq$ $D^{\prime}$ $\mathrm{C}(T,T^{-1})(\alpha)\geq \mathrm{C}(T(\mathcal{Z}(\mu_1,\xi),\mathcal{Z}(\mu_2,\xi)),T(\mathcal{Z}(\mu_2,\xi),\mathcal{Z}(\mu_1,\xi)))(\alpha)\forall \alpha \in [0,1].$
2In particular, we will assume databases are true sets and that  $D^{\prime}$  is constructed from  $D$  by adding or removing a single record. The neighbouring relation is denoted  $\simeq$ .
For example, the dominating pair for the Gaussian Mechanism (GM) is  $(\mathcal{N}(0,\sigma^2),\mathcal{N}(\Delta_2,\sigma^2))$  and for the Laplace Mechanism (LM), it is  $(\mathrm{Lap}(0,b),\mathrm{Lap}(\Delta_1,b))$ . The dominating pair distributions thus exhibit the greatest effect size (in this case  $\Delta_1 / b$  or  $\Delta_2 / \sigma$ ), allowing for the construction of the trade-off functions whose graphs are farthest from the off-diagonal of the unit square. In other words: the construction of the trade-off functions is determined by the hypothesis tests with the greatest power  $1 - \beta$  under a specific threat model, evaluated at the dominating pair distributions.
In the DPTM,  $\mathcal{H}_0$  and  $\mathcal{H}_1$  are both simple hypotheses of the form  $\mathcal{H}_0: \theta \in \Theta_0 \subset \Theta$  vs.  $\mathcal{H}_1: \theta \in \Theta_1 \subset \Theta$  with  $\| \Theta_0 \| = \| \Theta_1 \| = 1$  and  $\Theta_0 \cap \Theta_1 = \emptyset$ . The reason is that the distributions of  $\mathcal{M}(D)$  and  $\mathcal{M}(D')$  are fully specified because the adversary can evaluate  $q(D)$  and  $q(D')$ . The NP lemma [8] states that, for simple vs. simple hypothesis tests such as the ones in the DPTM, the uniformly most powerful (UMP) level  $\alpha$  tests  $\phi$  and  $\phi'$  (with the highest power  $1 - \beta$  at a specified level  $\alpha$  over the entire range of  $\theta$ , in this case  $q(D)$  or  $q(D')$ ) can be constructed by thresholding the (log-) Likelihood Ratio (LRs)  $\Lambda / \Lambda'$  of the mechanism's outputs  $\theta$  under  $D$  and  $D'$ :
$$
\Lambda (\theta) = \log \frac {\mathcal {L} (\theta \mid \mathcal {M} (D ^ {\prime}))}{\mathcal {L} (\theta \mid \mathcal {M} (D))} \leqslant c \text {a n d} \Lambda^ {\prime} (\theta) = \log \frac {\mathcal {L} (\theta \mid \mathcal {M} (D))}{\mathcal {L} (\theta \mid \mathcal {M} (D ^ {\prime}))} \leqslant c,
$$
where  $c \in \mathbb{R}$  is the critical value. The trade-off functions in the DPTM are then constructed from the distributions of  $\Lambda / \Lambda'$  evaluated at the dominating pairs under the respective  $\mathcal{H}_0$  and  $\mathcal{H}_1$  (also called privacy loss distributions [23]). Since the DPTM adversary enjoys the optimality properties of the NP lemma, we will refer to this adversary as  $\mathcal{A}^{\mathrm{NP}}$ .
3 Hypothesis testing in the RTM
The trade-off function's form for a given threat model depends only on the test statistic's distributions under  $\mathcal{H}_0$  and  $\mathcal{H}_1$ . Our strategy for the rest of the paper is thus: (1) Derive the optimal tests the adversary can construct in the RTM; (2) Specify the test statistic distributions for these tests, and (3) Construct the trade-off functions for a given mechanism using these test statistics, which will allow us to tightly bound the adversary's error rates and compare them to the DPTM. We first recall a general result, specifying how trade-off functions are constructed from test statistics.
Definition 2 (Trade-off function construction). Let  $P, Q$  be the distributions of a test statistic under  $\mathcal{H}_0$  and  $\mathcal{H}_1$ , respectively and let  $\Phi_{P/Q}$  denote their cumulative distribution function (CDF),  $\Psi_{P/Q}$  their survival function (SF) and  $\Phi_{P/Q}^{-1}, \Psi_{P/Q}^{-1}$  their respective inverses (iCDF, iSF). Then, we define:
$$
T (\alpha) = \Phi_ {Q} \left(\Psi_ {P} ^ {- 1} (\alpha)\right) a n d T ^ {- 1} (\alpha) = \Psi_ {P} \left(\Phi_ {Q} ^ {- 1} (\alpha)\right). \tag {1}
$$
For example, in experimental studies, empirical proxies of the test statistic distributions are evaluated by training shadow models [3]. Then, the empirical trade-off functions are constructed, which parameterise the error rates of the empirical adversary under a specific threat model [11, 12, 14]. We will utilise this strategy later to validate our theoretical bounds empirically. For our formal analysis on the other hand, we will derive exact expressions for  $T$  and  $T^{-1}$ .
RTM security game For reasons which will soon become clear, we will refer to the adversary under our RTM as a sub-optimal adversary  $(\mathcal{A}^{\mathrm{SO}})$  and use a formal security game, similar to the one used to define the DPTM (see e.g. Section 3.1. of [4]), which proceeds between a neutral/trusted curator  $\mathcal{C}$  and  $\mathcal{A}^{\mathrm{SO}}$ . We consider a single-round, non-interactive protocol.
Step 1: The adversary  $\mathcal{A}^{\mathrm{SO}}$  constructs a database  $D = \{x_{1},\ldots x_{n}\} \subseteq \mathbb{D}$ , decides on a training function  $q$ , a DP mechanism  $\mathcal{M}$  and sends them to the curator  $\mathcal{C}$ ;
Step 2:  $\mathcal{C}$  flips a bit  $b$ . If  $b = 0$ , they fix  $X = D$ . If  $b = 1$ , they choose a singleton  $\{x^{*}\} \subset \mathbb{D}^{\prime}$  where  $\mathbb{D} \subset \mathbb{D}'$  and  $\mathbb{D} \cap \mathbb{D}' = \{x^{*}\}$  and fix  $X = D \cup \{x^{*}\}$ . They then train a model  $\theta \in \Theta$  on  $X$  using  $q$  and  $\mathcal{M}$  ( $\theta$  may also be a gradient) and send  $\theta$  to  $\mathcal{A}^{\mathrm{SO}}$ ;
Step 3:  $\mathcal{A}^{\mathrm{SO}}$  decides if  $\mathcal{H}_0:\theta \sim \mathcal{M}(D)$  or  $\mathcal{H}_1:\theta \sim \mathcal{M}(D')$ ;
Step 4: If  $\mathcal{A}^{\mathrm{SO}}$  is correct, the MI attack is successful, privacy is breached and the game is won.
The key difference between the RTM and the DPTM is step 2:  $\mathcal{A}^{\mathrm{SO}}$  has no access to the point  $x^{*}$  which may be added by  $\mathcal{C}$ , and by extension, no knowledge of the exact value of  $q(D')$ . In step 3,  $\mathcal{A}^{\mathrm{SO}}$  can therefore only decide  $\theta \sim \mathcal{M}(D')$  by rejecting that  $\theta \sim \mathcal{M}(D)$ , but not by directly confirming that  $\theta \sim \mathcal{M}(D')$ .
The hypothesis testing capabilities of the RTM adversary are identical to those of an offline MI adversary. Offline MI attacks are used in previous works to audit the privacy of ML model training [12, 13, 14] and decide whether  $x^{*}$  is part of the training data or not without actively training shadow models on databases containing it. Equivalently, offline MI is MI by exclusion (i.e. inferring  $\theta \sim \mathcal{M}(D')$  by rejecting  $\mathcal{H}_0: \theta \sim \mathcal{M}(D)$ ). In offline MI, this is done for reasons of practicality and computational efficiency [12], while in the RTM, it by design. Nonetheless, the resulting error rates are identical and our bounds thus can be used to characterise offline MI.
RTM: Formal analysis  $\mathcal{M}$  is an additive noise mechanism based on the distribution  $\mathcal{Z}$ , thus  $\mathcal{H}_0:\theta \sim \mathcal{M}(D)$  is equivalent to  $\mathcal{H}_0:\theta \sim \mathcal{Z}(q(D),\xi)$ . Since  $q(D)$  is computable by  $\mathcal{A}^{\mathrm{SO}}$  through access to  $D$ , the likelihood  $\mathcal{L}(\theta \mid \mathcal{Z}(q(D),\xi))$  is also computable and  $\mathcal{H}_0$  is a simple hypothesis of the form  $\theta \in \Theta_0, \| \Theta_0\| = 1$ , exactly like the DPTM. However, the likelihood  $\mathcal{L}(\theta \mid \mathcal{Z}(q(D'),\xi))$  is not computable without access to  $D'$  and  $\mathcal{H}_0$  is a composite hypothesis of the form  $\theta \in \Theta \setminus \Theta_0$  with  $\| \Theta \setminus \Theta_0\| > 1$  because it depends on the unknown value of  $q(D')$ . The following is a direct consequence of the NP lemma, which holds only for simple vs. simple hypotheses [25].
Lemma 1. In the RTM, no UMP level  $\alpha$  test exists for all possible values of  $q(D')$ , i.e.  $\forall \theta \in \Theta \setminus \Theta_0$ . Thus,  $\mathcal{H}_0$  vs.  $\mathcal{H}_1$  (and  $\mathcal{H}_1$  vs.  $\mathcal{H}_0$ ) are each not decidable by a single test.
This implies that  $\mathcal{A}^{\mathrm{SO}}$  is necessarily weaker than  $\mathcal{A}^{\mathrm{NP}}$ , i.e. cannot achieve the same power  $1 - \beta$  at a given level  $\alpha$  for all values of the parameter of interest; to thus establish the optimal tests in this sub-optimal setting,  $\mathcal{A}^{\mathrm{SO}}$  must consider specific value ranges of  $q(D')$  separately. In the following, we will call two tests equivalent if they have the same power at a level  $\alpha$  for all values of the tested parameter. Moreover, we will limit our description to  $\mathcal{H}_0: \theta \sim \mathcal{M}(D)$  vs.  $\mathcal{H}_1: \theta \sim \mathcal{M}(D')$ , as  $\mathcal{H}_1$  vs.  $\mathcal{H}_0$  is handled identically.
Lemma 2. The composite  $LR$  test  $\mathcal{H}_0: \theta \sim \mathcal{Z}(q(D), \xi)$  vs.  $\mathcal{H}_1: \theta \sim \mathcal{Z}(q(D'), \xi)$  is equivalent to letting  $\theta \sim \mathcal{Z}(\mu, \xi)$  and simultaneously conducting two individual one-sided  $LR$  tests  $r$  and  $r'$  with null hypothesis  $\mu = 0$  and alternative hypotheses  $-\Delta \leq \mu < 0$  for  $r$  and  $0 < \mu \leq \Delta$  for  $r'$ , where  $\mu = q(D') - q(D)$  and  $\Delta$  is the global sensitivity of  $q$ .
Thus,  $\mathcal{A}^{\mathrm{SO}}$  can leverage known facts about  $q$  and  $\mathcal{M}$  to limit the value ranges for the alternative hypothesis, but, as  $q(D')$  is unknown, must "split the problem" into two individual tests. The individual tests are then UMP level  $\alpha$  under the following pre-condition.
Lemma 3. The individual LR tests  $r$  and  $r'$  are UMP level  $\alpha$  if the noise distribution  $\mathcal{Z}$  has the monotone likelihood ratio property (MLRP). The Laplace Mechanism (LM), Gaussian Mechanism (GM) and the Poisson-Subsampled Gaussian Mechanism (SGM) all have the MLPR. Moreover, the power of both tests is maximised when  $\| \mu \| = \| q(D') - q(D) \| = \Delta$ .
This lemma intuitively states that, while there is an optimality condition associated with the tests that can be constructed in the RTM, it is not the same optimality condition as the NP lemma guarantees in the DPTM. In particular, although  $\mathcal{A}^{\mathrm{SO}}$  can construct a pair of tests which are optimal (UMP level  $\alpha$ ), they are only optimal for specific parameter value ranges rather than for all possible parameter values. The lemma also states that it suffices to consider the tests  $\mu = 0$  vs.  $\mu = \pm \Delta$  when constructing the trade-off functions, as these correspond to the highest effect size at a fixed  $\xi$ . In other words, dominating pairs remain dominating pairs in the RTM. Next, we provide two convenience lemmata, which allow  $\mathcal{A}^{\mathrm{SO}}$  to conduct both tests simultaneously.