SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Martin Bertran
Amazon AWS AI/ML
Shuai Tang
University of Pennsylvania Amazon AWS AI/ML
Jamie Morgenstern
Aaron Roth
Membership inference attacks are designed to determine, using black box access to trained models, whether a particular example was used in training or not. Membership inference can be formalized as a hypothesis testing problem. The most effective existing attacks estimate the distribution of some test statistic (usually the model's confidence on the true label) on points that were (and were not) used in training by training many shadow models—i.e. models of the same architecture as the model being attacked, trained on a random subsample of data. While effective, these attacks are extremely computationally expensive, especially when the model under attack is large.
We introduce a new class of attacks based on performing quantile regression on the distribution of confidence scores induced by the model under attack on points that are not used in training. We show that our method is competitive with state-of-the-art shadow model attacks, while requiring substantially less compute because our attack requires training only a single model. Moreover, unlike shadow model attacks, our proposed attack does not require any knowledge of the architecture of the model under attack and is therefore truly "black-box". We show the efficacy of this approach in an extensive series of experiments on various datasets and model architectures. Our code is available at github.com/amazon-science/quantile-mia.
The basic goal of privacy-preserving machine learning is to find models that are predictive on some underlying data distribution, without being discursive of the particular data points on which they were trained. The simplest kind of attack that can be launched on a trained model—falsifying privacy guarantees—is a membership inference attack. A membership inference attack, informally, is a statistical test that is able to reliably determine whether a particular data point was included in the training set used to train the model or not.
Almost all membership inference attacks are based on the observation that models tend to overfit their training sets in different ways. In particular, they tend to systematically predict higher confidence in the true labels of data points from their training set, compared to points drawn from the same distribution not in their training set. The confidence that a model places on the true label of a data-point is thus a natural test statistic to build a membership-inference hypothesis test around. A
<sup>0</sup>Martin and Shuai are the lead authors, and other authors are ordered alphabetically. {maber-lop,shuat}@amazon.com
variety of recent methods [Shokri et al., 2017, Long et al., 2020, Sablayrolles et al., 2019, Song and Mittal, 2021, Carlini et al., 2022] are based around this idea, and aim to estimate the distribution of the test statistic (the confidence assigned to the true label of a datapoint) over the distribution of datapoints that were not used in training (and sometimes, also over the distribution of datapoints that were used in training) for the purpose of designing tests that can reject the null hypothesis—that a data point under attack was not used in training—with the desired level of confidence.
The efficacy of this class of attacks depends in large part on the granularity to which the distribution of the test statistic can be estimated. The simplest (and most computationally efficient) approach, originally proposed by Yeom et al. [2018], is to estimate this distribution marginally — i.e. without conditioning on the covariates  $x$  of the example being attacked. This reduces the problem to a simple one-dimensional estimation problem, and—under mild assumptions—the optimal hypothesis test (by the Neyman-Pearson Lemma) is simply a fixed threshold  $\tau$  on the test statistic—examples are declared to have been used in training if the confidence the model places on their true label exceeds  $\tau$ , and are declared to have been not used in training otherwise. More sophisticated methods attempt to estimate the distribution of the test statistic conditional on the inclusion of a target point  $x$  in the training data (over the randomness of the selection of the other points used in training). Our method and others follow this approach, where the confidence score produced by each example  $x$  by the target model is compared to a sample-dependent threshold  $\tau(x)$ —points  $x$  with scores exceeding this threshold are declared to be used in training. The most common method under this approach is to train shadow models [Shokri et al., 2017, Long et al., 2020, Sablayrolles et al.,
2019, Song and Mittal, 2021, Carlini et al., 2022]. Informally, shadow models are trained with the same architecture as the model being attacked, using random subsets of training data, that either include or do not include the target point  $x$ . As a result, each shadow model gives a sample of the test statistic conditional on  $x$ 's inclusion (or non-inclusion) in the training set, where the randomness is over the other examples in the training set and any randomness involved in training. Because many samples from this distribution on the test statistic are needed to estimate it, membership inference attacks based on shadow models generally require training many shadow models of the same architecture as the model under attack; between 64 and 256 shadow models were used in Carlini et al. [2022] (Figure 1 compares the receiver operating characteristic (ROC) of the attack on ImageNet against our proposed approach). Especially for large models, this makes shadow model attacks prohibitive, for at least two reasons:
1. Training Cost: Widely used commercial models, on which membership inference attacks would be most damaging, are extremely large and expensive to train. An attacker launching a membership inference attack based on shadow models must train many (dozens to hundreds) models of the same architecture. Thus the computational costs can be hundreds of times larger than the costs of training the model under attack, which for commercial models is prohibitive for attackers without enormous resources.
2. Knowledge of the Model Under Attack: As argued in Carlini et al. [2022], and also shown in Appendix A, when the shadow model is of the same complexity as or more complex than the target model, the LiRA attack performs well, and when less complex shadow models are deployed, the success rate of the attack drops precipitously. Hence the success of the attack depends on knowledge of the model under attack. But many aspects of the architecture and
training process for large commercial models are not publicly known, making this style of attack less effective in realistic settings.
We introduce a new class of membership inference attacks, based on quantile regression, that is able to mitigate these issues:
1. Like attacks based on shadow models, it is a conditional attack, subjecting different examples  $x$  to different thresholds  $\tau(x)$ . However,
2. It only requires training a single model, and
3. The architecture of the model used in the attack need not be related to the architecture of the model under attack, and so no knowledge of the model architecture or training algorithm used to train the model under attack is needed.
Our quantile regression attack. Given a model  $f$  that we intend to attack, we collect a dataset of labelled examples  $\{(x_i, y_i)\}_{i=1}^n$  from the underlying data distribution, known to not have been used in training. For each example  $(x_i, y_i)$  in our dataset, we evaluate the model  $f$  on example  $x_i$ , and record the (real-valued) confidence score  $s(x_i, y_i)$  that the model places on the correct label  $y$ . We then train a quantile regression model  $q$  on the dataset  $\{(x_i, s(x_i, y_i))\}$  consisting of examples  $x$  labeled with their confidence scores  $s(x, y)$ . Informally, the quantile regression model is trained to predict  $q(x)$ , a target  $1 - \alpha$  quantile of the conditional distribution on  $s(x_i, y_i)$ , given  $x_i$ . Intuitively, a score  $s(x_i, y_i)$  larger than the  $1 - \alpha$  quantile  $q(x_i)$  indicates that  $f$  assigns a confidence on the true label that is higher than a  $1 - \alpha$  fraction of the examples not used in training — giving us evidence that the example in question was part of the training set. Thus, given a new target point  $(x, y)$ , we reject the null hypothesis that  $x$  was not used in training (i.e., we declare  $x$  to have been used in training) whenever  $s(x, y)$  exceeds  $q(x)$ —i.e., when  $s(x, y) \geq q(x)$ . Similarly, whenever  $s(x, y) < q(x)$ , we do not reject the null hypothesis, and declare the point  $(x, y)$  to have not been used in training.
This attack by design has a false positive rate of  $\alpha$  — the probability that it incorrectly declares a randomly selected point  $(x, y)$  that was not used in training to have been used in training is  $\alpha$ . The ability of  $q$  to correctly label those examples used in training as such, measured by its true positive rate or precision or related statistics, will vary with  $\alpha$  (the higher  $\alpha$ , the larger the number of positive labels our test will assign). So, in varying our target  $\alpha$ , we can sweep out our test's tradeoff between false positive and true positive rates.
The primary strength of our attack is that we need only a single quantile regression model  $q$ , rather than a large number of shadow models. Furthermore, because the success of our attack depends only on how well  $q$  predicts the quantiles of the confidence score distribution of  $f$  (rather than producing confidence scores drawn from the same distribution as  $f$ ),  $q$  need not have any relationship to the architecture of  $f$  or any knowledge of it—the only access to  $f$  that is needed is the ability to evaluate confidence scores  $s(x, y)$  produced by  $f$  given examples  $x$ . Our attack is, therefore, more "black-box" than those which use shadow models of the same architecture as  $f$ .
We derive a basic theory for our approach based on quantile regression, which trains a model to predict quantiles by minimizing pinball loss. We run an extensive series of experiments and find that our quantile regression approach is competitive with (and sometimes more effective than) much more computationally expensive shadow model approaches. The relative effectiveness of our approach appears to grow the more complex the classification task and model under attack are. For example, when attacking a ResNet-50 model trained on ImageNet-1k, our attack (which trains only a single model) outperforms shadow model approaches trained much more expensively at every false positive rate. On simpler and less data rich tasks (like CIFAR-10), the accuracy of our approach dominates the marginal baseline of Yeom et al. [2018], but falls short of shadow model approaches. Thought provokingly, however, we find that when this occurs, it is because the shadow model approach has
<sup>1</sup>Under the presumption that only a small fraction of data sampled from the distribution were used in training, then we may simply take a random sample from the underlying distribution, and be confident that it is representative of data not used in the training procedure for  $f$ .
2This is an informal description, as in realizable settings, conditioning on  $x_{i}$  in its entirety leaves a point mass distribution on  $s(x_{i},y_{i})$  — i.e. the deterministic confidence score for  $y_{i}$  predicted by the model  $f(x_{i})$ . See Section 3 for precise guarantees.
found thresholds that correspond to a quantile model with lower pinball loss than our trained quantile regression model. This suggests that our fundamental approach of pinball loss minimization is sound, and that our attempts to directly optimize for it are less successful when data is less plentiful. Across all experiments, we find that the best quantile regression method (as measured by pinball loss) is uniformly the best membership inference attack.
Starting with the seminal work of Homer et al. [2008], membership inference has become one of the most widely studied classes of privacy attacks. Most approaches for membership inference determine whether an example is part of the training set via some score function, which can be loss [Yeom et al., 2018, Sablayrolles et al., 2019], confidence [Salem et al., 2018], entropy [Song and Mittal, 2021], or difficulty calibration [Watson et al., 2021] among others. Another common approach is to query the model on similar or related examples to the target point [Wen et al., 2023, Jayaraman et al., 2020, Long et al., 2018, Li and Zhang, 2020].
We focus the remainder of our discussion on related work on shadow model approaches to membership inference since they are our main benchmarks. Most work on shadow models considers a setup where there is a private dataset  $D^{\mathrm{private}}$  (unknown to the attacker) drawn from a distribution  $Q$ , and an algorithm  $\mathcal{A}$  for training a model  $f = \mathcal{A}(D^{\mathrm{private}})$ . The attacker has access to a set of data  $D^{\mathrm{public}}$  drawn from the same distribution  $Q$  and partial query access to the model  $f$  from which the attacker can compute scores  $s(x, y)$  given target examples  $(x, y)$ . The attacker aims to predict, for a data point  $(x, y)$ , whether  $(x, y) \in D^{\mathrm{private}}$ .
Likelihood Ratio Attacks. Membership inference attacks are fundamentally hypotheses tests between two competing hypotheses  $(H_0:(x,y)\notin D^{\mathrm{private}},H_1:(x,y)\in D^{\mathrm{private}})$ . By the Neyman-Pearson lemma [Neyman and Pearson, 1933], the optimal hypothesis test based on a test statistic  $s(x,y)$  computes the likelihood ratio of the score under the null and alternative hypothesis, and subjects the likelihood ratio to a threshold  $\tau$ . The choice of the threshold  $\tau$  determines the trade-off between precision (the fraction of examples labeled as belonging to the private dataset which did belong to the dataset) and recall (or true positive rate) of the resulting classifier. We call a membership inference attack carried out with this classifier a likelihood ratio attack (LiRA), introduced by Carlini et al. [2022]. LiRA was designed to achieve very high precision (very few false positives relative to the number of positive predictions), as they noted high precision corresponds to a high degree of confidence that the data points accused of being part of the training set were, in fact, part of the training set. Prior work had looked at global notions of inference attack quality, at possibly much lower degrees of precision [Ye et al., 2021, Jayaraman et al., 2021].
The main difficulty with implementing LiRA directly is that the density functions of the score under the null and alternative hypothesis are unknown. Instead, the literature aims to estimate these density functions, primarily by training a collection of shadow models [Shokri et al., 2017, Long et al., 2020, Sablayrolles et al., 2019, Song and Mittal, 2021, Carlini et al., 2022]. Shadow model attacks split the attacker's dataset  $D^{\mathrm{public}}$  into several pairs of shadow public/private datasets  $D_{i}^{\mathrm{private}}$ ,  $D_{i}^{\mathrm{public}}$ , and for each of these shadow datasets, a shadow model  $f_{i}$  is trained on  $D_{i}^{\mathrm{private}}$ . The shadow model  $f_{i}$  and corresponding datasets  $D_{i}^{\mathrm{private}}$ ,  $D_{i}^{\mathrm{public}}$  are used to generate private and public score samples  $s_{i}^{\mathrm{private}}$ ,  $s_{i}^{\mathrm{public}}$  from which to estimate the likelihood ratio function given parametric assumptions. Carlini et al. [2022] used a large number of shadow models to achieve high precision. This approach works well—but it is computationally demanding because it requires training many shadow models.
We study attacks on models  $f$  that solve a supervised learning problem defined over a distribution  $\mathcal{D} \in \Delta(\mathcal{X} \times \mathcal{Y})$  of labeled examples  $(x, y)$ , consisting of feature vectors  $x \in \mathcal{X}$  and labels  $y \in \mathcal{Y}$ . We make no assumptions about  $\mathcal{X}$  or  $\mathcal{Y}$  (e.g.  $\mathcal{Y}$  could be a discrete set in a multi-class classification problem, or we could have  $\mathcal{Y} = \mathbb{R}$  in a regression problem). We assume that the model  $f$  outputs a confidence score in  $[0, 1]$  for each possible label  $\hat{y} \in \mathcal{Y}$ : in other words,  $f: \mathcal{X} \to [0, 1]^{\mathcal{Y}}$ , and for each  $\hat{y} \in \mathcal{Y}$ , we write  $f_{\hat{y}}(x) \in [0, 1]$  for the confidence score that  $f$  assigns to label  $\hat{y}$  given input  $x$ . Such models are often used to make point predictions by predicting the label  $\hat{y} = \arg \max_{y \in \mathcal{Y}} f_y(x)$  on input  $x$  — but we will interact with such models  $f$  only at the level of confidence score predictions.
A model  $f$  is derived from a training process that did not have direct access to  $\mathcal{D}$ , but rather to a finite sample  $D^{\mathrm{private}}$  called the training set. The training process correlates  $f$  with  $D^{\mathrm{private}}$ . A membership inference attack is a hypothesis test that must use a test statistic derived only from  $f$  that aims to determine whether a labeled example  $(x, y)$  is a member of the training set  $D^{\mathrm{private}}$  or not. Formally, we model this as a hypothesis test that aims to solve the following simple hypothesis testing problem:
$$
H _ {0}: (x, y) \sim \mathcal {D} \qquad H _ {1}: (x, y) \sim D ^ {\mathrm {p r i v a t e}}
$$
Here,  $(x,y)\sim D^{\mathrm{private}}$  denotes sampling a point  $(x,y)$  uniformly at random from  $D^{\mathrm{private}}$ . Observe that since we derive the test statistic from  $f$ , even if  $D^{\mathrm{private}}$  was itself sampled i.i.d. from  $\mathcal{D}$ ,  $H_0$  and  $H_{1}$  are distinct hypotheses since the training process has correlated  $f$  with  $D^{\mathrm{private}}$ . In particular, we will base our attack on the presumption that  $f$  will tend to be over-confident on examples  $(x,y)\in D^{\mathrm{private}}$ . Towards this, we choose as our test statistic  $s(x,y) = z_y(x) - \max_{y'\neq y}z_{y'}(x)$  the logit difference between the true label and its most likely alternative, where  $z(x)$  denotes the logits (unnormized features before the softmax nonlinearity) of the model. This choice follows the scoring rule used in Carlini et al. [2022] and will be useful for experimental comparisons. However, the remainder of our theoretical treatment will be agnostic as to this choice.
In this paper we restrict attention to membership inference attacks (hypothesis tests) that apply a threshold function to  $s(x, y)$ , with a threshold that may depend on  $x$ . Given a function  $q: \mathcal{X} \to \mathbb{R}$  that maps examples  $x$  to thresholds, the corresponding membership inference attack is given by:
$$
\mathcal {A} _ {q} (x, y) = \left\{ \begin{array}{l l} \top       (x \sim \mathcal {D}) & \text {i f} s (x, y) <   q (x) \\ \bot       (x \sim D ^ {\text {p r i v a t e}}) & \text {i f} s (x, y) \geq q (x). \end{array} \right.
$$
Here  $\bot$  is shorthand for "We reject the null hypothesis that  $(x,y)\sim \mathcal{D}$  (and thus declare  $(x,y)$  of being in the training set), and  $\top$  is shorthand for "we do not reject the null hypothesis (and thus do not accuse  $(x,y)$  of being in the training set)".
A natural baseline is to set  $q(x,y) = \tau$  to be a constant. This is the attack proposed by Yeom et al. [2018], and we write this baseline as  $\mathcal{A}_{\tau}$ . If  $\tau$  is set to be a  $1 - \alpha$  quantile of the marginal distribution on  $s(x,y)$  when  $(x,y) \sim \mathcal{D}$ , then this attack can easily be seen to have false positive rate  $\alpha$ . Below we define quantiles assuming (for simplicity) that the distribution in question is continuous—but it is also possible to define quantiles without this assumption.
Definition 1. Fix a continuous distribution  $\mathcal{P} \in \Delta \mathbb{R}$ . A number  $\tau \in \mathbb{R}$  is a  $(1 - \alpha)$ -quantile of  $\mathcal{P}$  if:
$$
\operatorname * {P r} _ {s \sim \mathcal {P}} [ s \leq \tau ] = 1 - \alpha
$$
We can evaluate the performance of a membership inference attack by evaluating its false positive rate, true positive rates, and precision<sup>3</sup>:
Definition 2. Fix an arbitrary membership inference attack  $\mathcal{A}:\mathcal{X}\times \mathcal{Y}\to \{\top ,\bot \}$ . We define the following performance metrics
$$
F P R (\mathcal {A}) = \operatorname * {P r} _ {(x, y) \sim \mathcal {D}} [ \mathcal {A} (x, y) = \bot ], \qquad T P R (\mathcal {A}) = \operatorname * {P r} _ {(x, y) \sim D ^ {\text {p r i v a t e}}} [ \mathcal {A} (x, y) = \bot ],
$$
$$
P r e c (\mathcal {A}) = \frac {F P R (\mathcal {A})}{F P R (\mathcal {A}) + T P R (\mathcal {A})}.
$$
It is immediate that the baseline membership inference attack achieves its target false positive rate; the true positive rate and precision of the attack can be evaluated empirically:
Lemma 1. Let  $\tau$  be a  $1 - \alpha$  quantile of  $\mathcal{P}$ , the distribution on confidence scores  $s(x,y)$  that results from sampling  $(x,y) \sim \mathcal{D}$ . Then the baseline membership inference attack  $\mathcal{A}_{\tau}$  has FPR  $(\mathcal{A}_{\tau}) = \alpha$ .
Proof. This follows from the definitions:  $\mathrm{FPR}(\mathcal{A}_{\tau}) = \mathrm{Pr}_{(x,y)\sim \mathcal{D}}[s(x,y)\geq \tau ] = \alpha$
3 Precision is equivalent to the accuracy of the attack conditioned on a positive prediction  $\perp$  when  $\mathrm{Pr}[\perp ] = 0.5$
3 Our Attack
Our attack is  $\mathcal{A}_q(x,y)$ , where  $q$  is derived from a quantile regression model trained to predict quantiles of our test statistic  $s(x,y)$  on a dataset of points  $(x,y)$  drawn from our null hypothesis distribution  $(x,y) \sim \mathcal{D}$ . A popular non-parametric quantile regression method is to minimize pinball loss, which elicits quantiles (just as squared loss elicits means):
Definition 3. The pinball loss function defined for a  $1 - \alpha$  quantile is:
$$
P B _ {1 - \alpha} (\hat {y}, y) = \max  \left\{\alpha (\hat {y} - y), (1 - \alpha) (y - \hat {y}) \right\}
$$
Pinball loss is a useful objective function because it elicits quantiles:
Lemma 2. Fix any distribution  $\mathcal{P} \in \Delta \mathbb{R}$ . Let:
$$
\tau \in \operatorname *{arg  min}_{\hat{y}\in \mathbb{R}}\underset {y\sim \mathcal{P}}{\mathbb{E}}\left[PB_{1 - \alpha}(\hat{y},y)\right]
$$
Then  $\tau$  is a  $(1 - \alpha)$ -quantile of  $\mathcal{P}$ .
Viewed through this lens, the baseline attack can be thought of as the end result of the following simple pipeline:
1. Select a target false positive rate  $\alpha$
2. Choose a threshold  $\tau$  by solving the minimization problem
$$
\tau \in \arg \min  _ {\tau^ {\prime} \in \mathbb {R}} \underset {(x, y) \sim \mathcal {D}} {\mathbb {E}} \left[ \mathrm {P B} _ {1 - \alpha} (\tau^ {\prime}, s (x, y)) \right]
$$
3. Instantiate the baseline membership inference attack  $\mathcal{A}_{\tau}$ .
Our attack departs from this baseline attack by training a model  $q:\mathcal{X}\to \mathbb{R}$  on feature/confidence score pairs to optimize pinball loss, rather than a single threshold  $\tau$ :
1. Select a target false positive rate  $\alpha$  and a class of model architectures  $\mathcal{H}$  consisting of models  $q:\mathcal{X}\to \mathbb{R}$ .
2. Train a model  $q \in \mathcal{H}$  by solving the following risk minimization problem:
$$
q \in \arg \min  _ {q ^ {\prime} \in \mathcal {H}} \underset {(x, y) \sim \mathcal {D}} {\mathbb {E}} [ \mathrm {P B} _ {1 - \alpha} (q (x), s (x, y)) ] \tag {1}
$$
3. Instantiate the membership inference attack  $\mathcal{A}_q$
We train our quantile regression model on a dataset consisting of points  $(x,y)$  drawn from the underlying distribution (of points not used in training), labeled by the confidence scores  $s(x,y)$  derived from the model. Thus our attack assumes only that we have API access to the model under attack  $f$ , and are able to query it on a finite set of points to obtain confidence scores.