Scalable Membership Inference Attacks via Quantile Regression
Authors: Martin Andres Bertran
Shuai Tang
Aaron Roth
Michael Kearns
Jamie Heather Morgenstern
Steven Wu
Venue: NeurIPS 2023 poster (2023)
Keywords: machine learning
privacy
membership inference
TL;DR: A scalable approach to membership inference attacks based on quantile regression

Abstract: Membership inference attacks are designed to determine, using black box access to trained models, whether a particular example was used in training or not. Membership inference can be formalized as a hypothesis testing problem. The most effective existing attacks estimate the distribution of some test statistic (usually the model's confidence on the true label) on points that were (and were not) used in training by training many \emph{shadow models}---i.e. models of the same architecture as the model being attacked, trained on a random subsample of data. While effective, these attacks are extremely computationally expensive, especially when the model under attack is large. \footnotetext[0]{
Martin and Shuai are the lead authors, and other authors are ordered alphabetically. \{maberlop,shuat\}@amazon.com}

We introduce a new class of attacks based on performing quantile regression on the distribution of confidence scores induced by the model under attack on points that are not used in training. We show that our method is competitive with state-of-the-art shadow model attacks, while requiring substantially less compute because our attack requires training only a single model. Moreover, unlike shadow model attacks, our proposed attack does not require any knowledge of the architecture of the model under attack and is therefore truly ``black-box". We show the efficacy of this approach in an extensive series of experiments on various datasets and model architectures. Our code is available at \href{https://github.com/amazon-science/quantile-mia}{github.com/amazon-science/quantile-mia.}

Scalable Membership Inference Attacks via Quantile Regression
Martin Bertran * Amazon AWS AI/ML
Shuai Tang Amazon AWS AI/ML
Michael Kearns University of Pennsylvania Amazon AWS AI/L
Jamie Morgenstern University of Washington Amazon AWS AI/ML
Aaron Roth University of Pennsylvania Amazon AWS AI/ML
Zhiwei Steven Wu
Carnegie Mellon University Amazon AWS AI/M
Abstract
Membership inference attacks are designed determine, using black box access trained models, whether particular example was used in training or not. Mem- bership inference can be formalized as a hypothesis testing problem. The most effective existing attacks estimate the distribution of some test statistic (usually the model's confidence on the true label) on points that were (and were not) used training by training many shadow models-iee. models of the same architecture the model being attacked, trained on a random subsample of data. While effective, these attacks are extremely computationally expensive, especially when the model under attack is large.
We introduce a new class attacks based on performing quantile regression on the distribution of confidence scores induced by the model under attack on points that are not used in training. We show that our method is competitive with state-of- the-art shadow model attacks, while requiring substantially less compute because our attack requires training only a single model. Moreover, unlike shadow model attacks, our proposed attack does not require any knowledge of the architecture the model under attack and is therefore truly "black-box" We show the efficacy of this approach in an extensive series of experiments on various datasets and model architectures. Our code is available at gutthh/ccommmmmooosscceeeeeearge
Introduction
The basic goal of privacy-preserving machine learning is to find models that are predictive on some underlying data distribution, without being disclosive of the particular data points on which they were trained. The simplest kind of attack that can be launched guarantees- is a membership inference attack. membership inference attack, informally, is a statistical test that is able reliably determine whether particular data point included in the training set used to train the model or not.
trained model-falsifying privacy
Almost all membership inference attacks are based on the observation that models tend to overfit their training sets different ways. particular, they tend to systematically predict higher confidence in the true labels of data points from their training set, compared to points drawn from the same distribution not in their training set. The confidence that a model places on the true label data-point is thus natural test statistic to build membershiipiinference hypothesis test around. A
Martin and Shuai the lead authors, and other authors ordered alphabetically.
{maber-
lop.shuat) @ @amazon.com
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

variety of recent methods [Shokri et al. 2017, Long et al., 2020, Sablayrolles et al., 2019 Song and Mittal 2021 Carlini et al. 2022 are based around this idea, and aim to estimate the distribution of the test statistic (the confidence assigned to the true label of a datapoint) over the distribution of datapoints that were not used training (and sometimes, also over the distribution of datapoints that were used in training) for the purpose of designing tests that can reject the null hypothesis- that data point under attack was not used in training- with the desired level of confidence.
The efficacy of this class of attacks depends in large part on the granularity to which the distri- bution of the test statistic can be estimated. The simplest (and most computationally efficient) approach, originally proposed by Yeom et al.
10a
True Positive Rate
10-1
2018), is to estimate this distribution marginally i.e. without conditioning on the covariates a of the example being attacked. This reduces the problem to simple one-dimensional estimation problem, and-under mild assumptions op- timal hypothesis test (by the Neyman-Pearson Lemma) is simply a fixed threshold 7 on the test statistic-exampll examples are declared to have been used in training if the confidence the model places their true label exceeds and are declared to have been not used in training oth- erwise. More sophisticated methods attempt to estimate the distribution of the test statistic con- ditional on the inclusion of a target point æ in the training data (over the randomness of the se- lection of the other points used in training). Our method and others follow this approach, where the confidence score produced by each example by the target model is compared to a sample- dependent threshold f(x) points with scores exceeding this threshold are declared to be used in training. The most common method under this approach is to train shadow models [Shokri iet al. 2017 [ et al. 2020 Sablayrolles et al.
10-²
Ours
LIRA (n=2)
LIRA (1=4)
LÍRA
10
LIRA (n=0)
LIRA (n-64 Visual)
Marginal Baseline
10
10-1
10-3
10-2
10-1
10
False Positive Rate
Comparing the true positive rate vs. false positive rate of our membership inference attack with the marginal baseline proposed in Yoom et al. [2018] and the state-of-the- thethe-rrt LiRA proposed in Carlini et al. 2022 evaluated at 2, 4. 6, and 8 shadow models. We also pro- vide visual readout of their 64 shadow model results, as reported in their paper (we did not have the compute necessary to reproduce this). We faithfully replicated LIRA's attack setup and produced hetter results than their reported values. Our single-model quantile regres- sion attack can reliably identify training samples on a ResNet-50 ImageNet target model (67.5% test accuracy) without knowledge of the target architecture.
Figure 1:
2019 Song and Mittal 2021 Carlini et al. 2022). Informally, shadow models are trained with the same architecture as the model being attacked, using random subsets of training data, that either include or do not include the target point æ. As result, each shadow model gives sample of the test statistic conditional on z's 's inclusion (or non- con-inclusion) in the training set, where the randomness is over the other examples in the training set and any randomness involved in training. Because many samples from this distribution on the test statistic are needed to estimate it, membership inference attacks based shadow models generally require training many shadow models of the same architec- ture as the model under attack; between 64 and 256 shadow models were used in Carlini et al. [2022] (Figure [ the receiver operating characteristic (ROC) of the attack on ImageNet against proposed approach). Especially for large models, this makes shadow model attacks prohibitive, for at least two reasons:
1. Training Cost: Widely used commercial models, on which membership inference attacks would be most damaging, are extremely large and expensive to train. An attacker launching a membership inference attack based on shadow models must train many (dozens to hundreds) models of the same architecture. Thus the computational costs can be hundreds of times larger than the costs of training the model under attack, which for commercial models is prohibitive for attackers without enormous resources.
2. Knowledge of the Model Under Attack: As argued in Carlini et al. (2022) and also shown in Appendix A when the shadow model is of the same complexity as or more complex than the target model, the LiRA attack performs well, and when less complex shadow models are deployed, the success rate of the attack drops precipitously. Hence the success of the attack depends on knowledge of the model under attack. But many aspects of the architecture and

---

training process for large commercial models are not publicly known, making this style of attack less effective in realistic settings.
1.1 Our Results
We introduce a new class of membership inference attacks, based on quantile regression, that is able to mitigate these issues:
1. Like attacks based on shadow models, is a conditional attack, subjecting different examples x to different thresholds r(z). However,
2. It only requires training a single model, and
3. The architecture of the model used in the attack need not be related to the architecture of the model under attack, and so no knowledge of the model architecture or training algorithm used to train the model under attack is needed.
Our quantile regression attack. Given a model f that we intend to attack, we collect a dataset of labelled examples (1) }/ from the underlying data distribution, known to not have been used in training For each example (Ir; yi) in our dataset, we evaluate the model f on example z4, and record the (real-valued) confidence score s(x., ye) that the model places on the correct label y. We then train a quantile regression model on the dataset {(=;, s(1;, }:))} consisting of examples labeled with their confidence scores s( (xx, y). Informally, the quantile regression model is trained to predict q(xr), a target - a quantile of the conditional distribution s(x;, 14), given I; Intuitively, a score s( 3/3 larger than the 1 a quantile ((z;) indicates that f assigns a confidence on the true label that is higher than a 1 fraction of the examples not used in training
2/5
giving us evidence that the example in question was part of the training set. Thus, given a new target point (x: y). we reject the null hypothesis that was not used in training (i.e. we declare to have been used in training)
whenever s(x y) exceeds (2) - ie. when s(z y) 2 q(x). Similarly, whenever s(r, y) < q(z), we do not reject the null hypothesis, and declare the point (Ir, have not been used in training.
This attack by design has false positive rate of a- the probability that incorrectly declares a randomly selected point (x. y) that was not used in training to have been used in training is a. The ability of g to correctly label those examples used in training as such, measured by its true positive rate precision or related statistics, will vary with a (the higher a, the larger the number of positive labels our test will assign). So, in varying our target a, we can sweep out our test's tradeoff between false positive and true positive rates.
The primary strength of our attack is that we need only a single quantile regression model q. rather than large number of shadow models. Furthermore, because the success of our attack depends only on how well g predicts the quantiles of the confidence score distribution of f (rather than producing confidence scores drawn from the same distribution as f), need not have any relationship to the architecture of f or any knowledge of it the only access to f that is needed the ability to evaluate confidence scores szz, y) produced by f given examples x. Our attack is, therefore, more "black-box" than those which use shadow models of the same architecture as f.
We derive a basic theory for our approach based on quantile regression, which trains a model to predict quantiles by minimizing pinball loss. We run an extensive series of experiments and find that our quantile regression approach is competitive with (and sometimes more effective than) much more computationally expensive shadow model approaches. The relative effectiveness of our approach appears to grow the more complex the classification task and model under attack are. For example, when attacking a ResNet-50 model trained on ImageNet- 1k, our attack (which trains only a single model) outperforms shadow model approaches trained much more expensively at every false positive rate. On simpler and less data rich tasks (like 10), the accuracy of our approach dominates the marginal baseline of Yeom et al. (2088), ). but falls short of shadow model approaches. Thought provokingly, however, we find that when this occurs, it is because the shadow model approach has
'Under the presumption that only a small fraction of data sampled from the distribution were used in training, then we may simply take random sample from the underlying distribution, and be confident that it is representative of data not used in the training procedure for f-
"This is an informal description, as in realizable settings, conditioning on x; in its entirety leaves a point mass distribution on s(z., y)) - i.e. the deterministic confidence score for y predicted by the model f(x). See Section 3 for precise guarantees.
3

---

found thresholds that correspond to quantile model with lower pinball loss than our trained quantile regression model. This suggests that our fundamental approach of pinball loss minimization sound. and that our attempts to directly optimize for it are less successful when data is less plentiful, Across all experiments, we find that the best quantile regression method (as measured by pinball loss) is uniformly the best membership inference attack.
1.2 Additional Related Work
Starting with the seminal work of Homer et al. 2008 J, membership inference has become one of the most widely studied classes of privacy attacks. Most approaches for membership inference determine whether an example is part of the training set via some score function, which can be loss Yeom et al., 2018, Sablayrolle et al.) 2019]. confidence [Salem et al. 2018), entropy [Song and Mittal| [202]]. or difficulty calibration Watson et al. 2021] among others. Another common approach is query the model on similar or related examples to the target point (Wen et al. 2023 Jayaraman et al. 2020
Long et al. 2018 Li and Zhang 2020].
We focus the remainder of our discussion on related work on shadow model approaches to membership inference since they are main benchmarks. Most work shadow models considers a setup where there a private dataset D (unknown to the attacker) drawn from a distribution Q. and an algorithm A for training a model f A(Drivate). The attacker has access drawn from the same distribution Q and partial query access to the model f from which the attacker can compute scores s( y) given target examples (z. v). The attacker aims to predict, for data point (2r, y), whether (2., E Dprivare
set of data Drublic
Likelihood Ratio Attacks. Membership inference attacks are fundamentally hypotheses tests between two competing hypotheses (Hg : (x. & Dprivate H1 : (x, y) E Dprivate), By the Neyman- Pearson lemma [Neyman and Pearson| 1933]. the optimal hypothesis test based on test statistic s(x, y) computes the likelihood ratio of the score under the null and alternative hypothesis, and subjects the likelihood ratio to a threshold r. The choice of the threshold r determines the trade-off between precision (the fraction of examples labeled as belonging to the private dataset which did belong to the dataset) and recall (or true positive rate) of the resulting classifier. We call membership inference attack carried out with this classifier likelihood ratio attack (LiRA), introduced by Carlini et al. [2022] LiRA was designed to achieve very high precision (very few false positives relative to the number of positive predictions), as they noted high precision corresponds to a high degree of confidence that the data points accused of being part of the training set were, in fact, part of the training set. Prior work had looked at global notions of inference attack quality, at possibly much lower degrees of precision YYe et al. 2021
Jayaraman et al. 2021).
The main difficulty with implementing LiRA directly that the density functions of the score under the null and alternative hypothesis are unknown. Instead, the literature aims to estimate these density functions, primarily by training a collection of shadow models (Shokri et al. 2017] I Long et al.) 2020 Sablayrolles et al. 2019 Song and Mittal 2021 [arlini et al.) 2022]. Shadow model attacks split the attacker's 's dataset Drmblic into several pairs of shadow public/private datasets Derivate DFuullcc and
for each of these shadow datasets, a shadow model f. is trained on D.piiaatee The shadow model f., and corresponding datasets Derivatee Dopolic are used to generate private and public score samples privatte publi: from which to estimate the likelihood ratio function given parametric assumptions. Carlini et al. (2022) used a large number of shadow models to achieve high precision. This approach works well but is computationally demanding because it requires training many shadow models.
2 Preliminaries
We study attacks on models f that solve supervised learning problem defined over a distribution D E A(X' x Y) of labeled examples (æz, v), consisting of feature vectors E X and labels y E y. We make no assumptions about X' or ) (e.g. Y could be discrete set in a multi-class classification problem, or we could have y IR in regression problem). We assume that the model f outputs a
confulence score in [0, 1] for each possible label ý E Y: in other words, ʃ X - [0. 1]²,, and for each E Y. we write (x) E [0. 1] for the confidence score that ʃ assigns to label given input x. Such models are often used to make point predictions by predicting the label ý = arg max fy(x) on input - but we will interact with such models f only at the level of confidence score predictions.
4

---

A model f is derived from training process that did not have direct access to D, but rather to finite sample Drrivate called the training set. The training process correlates with D A membership inference attack is a hypothesis test that must use test statistic derived only from f that aims to determine whether a labeled example (æ, y) a member of the training set Dirivate or not. Formally, we model this as a hypothesis test that aims to solve the following simple hypothesis testing problem:
Ho (I, D
H1 (z, M) D
Here, (a, y) ~ Dprivate denotes sampling point (æ, y) uniformly at random from Dprivate, Observe that since we derive the test statistic from f. even if Derivate was itself sampled i.idd from D, Ho and H, are distinct hypotheses since the training process has correlated f with Drrivate In particular, will base our attack on the presumption that ʃ will tend be over-confident on examples (3c, y) E Dpavatee Towards this, we choose as our test statistic s(1, y) = (2) - maxy'yy Zy (4) the logit difference between the true label and its most likely alternative, where =(xr) denotes the logits (unnormalized features before the softmax nonlinearity) of the model. This choice follows the scoring rule used in Carlini al. [2022) and will be useful for experimental comparisons. However, the remainder of our theoretical treatment will be agnostic as to this choice.
In this paper we restrict attention membership inference attacks (hypothesis tests) that apply threshold function to s(z, y), with a threshold that may depend on æ. Given a function q : R that maps examples thresholds, the corresponding membership inference attack is given by:
[T (x D) Az v) = [1 (x Derivate)
sf, y) q(z) if 3) 2 g(z).
Here 1 is shorthand for We reject the null hypothesis that (x, v) ~ D (and thus declare (z. y) of being in the training set)", and T is shorthand for "we do not reject the null hypothesis (and thus do not accuse (x, y) of being in the training set)".
A natural baseline is to set ((x, y) = - to be constant. This is the attack proposed by Yeom et al. [2018), and we write this baseline as A,. T is set to 1 1-a a quantile of the marginal distribution on s(x, y) when (z, y) D. then this attack we define quantiles assuming (for simplicity) that the distribution in question is continuous- but is also possible to define quantiles without this assumption.
easily be seen to have false positive rate Below
Definition 1. Fix continuous distribution E AR. number T R is (1 a---qanntll of if:
Pr 8 S = -
We can evaluate the performance of a membership inference attack by evaluating its false positive rate, true positive rates, and precision R
Fix an arbitrary membership inference attack A X' x y (T, We define the following performance metrics
Definition 2.
FPR(A) = Pr [A(x, =1], (z.g)~D
TPR(A)
Pr (z.g))-
[A(x, v) = -1],
FPR(A) Prec(A) FPR(A) + TPR(A)
It is immediate that the baseline membership inference attack achieves its target false positive rate; the true positive rate and precision of the attack can be evaluated empirically:
Lemma 1. Let be a 1 - a quantile of P. the distribution confidence scores s(x, y) that results from sampling (z. y) D. Then the baseline membership inference attack A, has FPR(A,) a.
Proof. This follows from the definitions: (PR(A,) = Pi(r.a)~D(((E))) v) 2
Preciison is equivalent the accuracy of the attack conditioned positive prediction 1 when Pr[1] 0.5
5

---

3 Our Attack
Our attack is Az y). where q is derived from quantile regression model trained to predict quantiles of our test statistic s(x, y) on dataset of points (z, y) drawn from our null hypothesis distribution (I,
D. A popular non- con-aaametric quantile regression method minimize pinball loss, which elicits quantiles (just as squared loss elicits means):
Definition 3. The pinball loss function defined for a a quantile is:
PB (0 y) max - v), (1 a)(y - 0)}
Pinball loss is a useful objective function because it elicits quantiles:
Lemma 2. Fix any distribution P E AR. Let:
T E arg min E (P yER youp"
Then is a - a:) quaantie P.
Viewed through this lens, the baseline attack can be thought of as the end result of the following simple pipeline:
1. Select target false positive rate a,
2. Choose threshold by solving the minimization problem
E arg min E
P (
7'ER (R.y)~
3. Instantiate the baseline membership inference attack A,.
Our attack departs from this baseline attack by training a model q : X - R on feature/confidence score pairs to optimize pinball loss, rather than single threshold 7:
1. Select target false positive rate a and a class of model architectures H consisting of models
q X + R.
2. Train a model E by solving the following risk minimization problem:
arg min E PBi-a (g(x), s(z v))]
d'EH (..))~
3. Instantiate the membership inference attack Ag
We train our quantile regression model on a dataset consisting of points (x, drawn from the underlying distribution (of points not used in training), labeled by the confidence scores s(x; y) derived from the model. Thus our attack assumes only that we have API access to the model under attack f. and are able to query it finite set of points obtain confidence scores.
We now establish some basic properties of our attack. The first is that, like the baseline attack, actually achieves its target false positive rate. Unlike the baseline attack, this is no longer immediate, but can be derived from properties of the pinball loss:
Theorem 1. Fix distribution D E A(X x V) over labeled examples and a model Suppose that the marginal distribution over s(z, y) for (z. y) ~ D is continuous. Let be any class of models that closed under additive shifts- iee. such that for each 9 E H and A E R. then also have of E for o' (x:) q(x) + A. Then for the membership inference attack Ag produced by our method, (PR(A.)
We defer the proof to Appendix C.I trading off our target false positive rate with our (empirically measured) true positive rate, just can for the baseline attack AA.
Thus by varying a, we can use our attack to sweep out a curve
Is this guarantee stronger than the baseline attack, and if so, in what sense? To give one perspective on this, it will be helpful to define group conditional quantile consistency, which related to multicalibration, concept originating from the fairness in machine learning literature (Hébert-
Johnson et al., 2018, Gupta et al.) 2022 2023
Bastani et al. 2022 Jung et al.) 2023 Noarov and Roth

---

Definition 4. Fix a collection G of group indicator functions g X + {0, and a model q: : x.= + R. q satisfies group conditional quantile consistency with respect to a distribution P A(X R., larget quantile 1 a, and the collection of groups 9 for every g E G:
Pr (xx) S slg(2)) = =1 1 =1-
(2,8))>p
Group conditional quantile consistency asks that our quantile predictions be correct not just marginally over the data, but also (simultaneously) conditionally on membership in large number of potentially intersecting groups. If we optimize pinball loss over a richer set of models (that are closed under shifts by a class of group indicator functions, rather than just constant functions), then our attack will achieve its target false positive rate even when conditioning on membership in each of the groups specified by the functions g E 9, rather than just marginally. This is a stronger guarantee, as a marginal false positive rate on its own need not hold subject to additional conditioning events.
Theorem 2. Fix a distribution D E A(X x y) over labeled examples and a model f. Fix collection of group indicator functions G and class of models H such that:
1. H. is closed under shifts from G: for every h. E H. E g, and v) E R. the function h/ (x:) h(x) ng(zz) is such that h' E H.
2. The conditional distribution over s(x,y) for (z,z) D, conditional on g(x) 1 is continuous for all E g.
Then for the membership inference attack Ag produced by our method. its false positive rate is conditional on membership in each group g E G: (((.)))=(AA -1g(()) = 1] = 1110 aa.
We defer the proof to Appendix C.2
4 Experiments
We present two sets of experiments on two different data domains, including images and tabular data. Here, we mainly focus attacking widely- classification models in these two domains, however, our theoretical claims generalizes to other data domains as well.
4.1 Image Classification Experiments
We evaluate the effectiveness of our proposed approach on four image classification datasets: CIFAR- 10 [Krizhessky et al.) 2009]. J a standard image classification dataset with 10 target classes; CIFAR-I00
Krizhevsky et al. 2009] another image classification dataset with 100 target classes; ImageNet-Ik
Russakovsky et al. [005], a substantially larger image classification task with 1000 target classes; and CINIC-10 DDrlow et al. 2018). J. an extension of CIFAR-10 10 that additionally uses images from ImageNet-Ik Ik corresponding to the original 10 target classes. To provide a realistic evaluation, we ensure our base models use common, well-performing architectures and follow standard guidelines for hyperparameter selection [He et al., 2015). including data augmentation, learning rate schedule, decay). For CIFAR-10 and CIFAR-100. target (classification) models include ResNet- 10, ResNet-18 18, ResNet-34, and ResNet-50 (He et al. 2013). For ImageNet- Ik and CINIC-10, the target model is ResNet-50. In all experiments, 50% of the dataset is used for training the target model, and, following the common standards, the resolution of the target model is 32x32 for CIFAR and CINIC datasets, and 224x224 for the ImageNet-1K dataset. The accuracy of each target model is presented in Appendix B
and 12 regularization
To perform our membership inference attack, train single quantile regression model following our proposal in [T. One of the advantages of our attack is that it is model-agnostic: since it does not require the knowledge of the model architecture of the target the knowledge of the training algorithm, we use the same model architecture for our quantile regression model in all settings a pretrained ConvNext-Ting Tiny model [Liu et al.) [022]. On CINIC-10 additionally experimented with a ResNet-50 model as our quantile model architecture.
For the scoring function, we use the hinge score proposed in Carlini et al. 2022 Shinge (I. v) (x))) - maxy'/y Zy (1r). This scoring rule closely related to the logit function of the model's confidence s(x,)) y) (x)) log(1 fy (x)) for models with high label confidence, and has
7

---

been empirically shown to be approximately normally distributed. On the largest dataset, ImageNet- Ik, we directly train quantile regression model predict (log-spaced) quantiles An alternative is to learn a parameterized model so that, for each sample, the model predicts the mean (a)) and the log of the standard deviation log a (ar), and the quantiles of a sample can generated from the Gaussian distribution N (z((z), (e (2))). Due to the fact that CIFAR and CINIC datasets much smaller (25000 samples available for training CIFAR), rather than directly learning quantiles, we opt to learn the parametrized models with Gaussian parameters on these datasets.
[0.9996. 1]
All images are processed at 224x224 resolution by the quantile model, which is trained on the remaining 50% of the training samples that were not used to train the target model. Since there is smaller body of literature on stable hyperparameters for regression models, we use Ray Tune [iaw al. 2018 for hyperparameter tuning (tuning is used to minimize validation pinball loss in held out dataset). The compute budget for our attack was approximately 30 GPU minutes per quantile regression attack (4 hours including hyperparameter optimization) on CAAAR-O/CIIAR-III minutes (4 hours 40 minutes including hyperparameter optimization) on CINIC-10. 10, and 16 hours (128 hours including hyperparameter optimization) on ImageNet-Ik, Ik. Final hyperparameters were found to be consistent across all tasks sharing an architecture; more information is provided in Appendix D
18
Figures T and 2 shows ROC curves of proposed approach on ImageNet-IK and CINIC-10 respectively; FPR is computed on a held-out dataset that was not used to train the target or the quantile regression model. Both the marginal quantile approach from Yeom et [2018) and the shadow model approach LIRA from Carlini et al. [2022) are also shown for reference. In these experiments, our quantile regression approach dominates the shadow model approach at all comparison points for ImageNet- 1k, and is roughly comparable to 4 shadow models on CINIC-10. Appendix E shows the same comparison against all CIFAR- 10/100 target models (ResNet- 10, ResNet18, ResNet34, ResNet-50). In these latter experiments, our attack outperforms the marginal baseline but falls short of the shadow model approach. We note that in this case, the shadow models actually produce thresholds that have lower pinball loss than our quantile regression algorithm, suggesting that on the smaller CIFAR datasets, our optimization heuristic was unable to sufficiently minimize test pinball loss. We find this observation to be consistent across experiments, that the attack with smaller pinball loss on public data is generally the attack with the best TPR at that FPR level a
10
True Positive Rate
10-1
10-2
Ours (Gaussian)
Marginal Baseline
LIRA (n=2)
LIRA (n=4)
LIRA (n-6)
LIRA (n=8)
10
10
10-2 False Positive Rate
10
100
Figure 2: Comparing the true positive rate vs. false positive rate of our membership inference attack (with parametric Gaussian loss) on the CINIC-10 dataset (an extension of the CIFAR- 10 dataset with 270,000 images, 4.5 times that of CIFAR-10) 10) with the marginal baseline proposed by Yeom et al. (2018) and the state-of-the-an LiRA proposed in Carlini et al. 2022 evaluated at 2, 4, 6, and shadow models. Both the model under attack and our quantile regression model use a ResNet-50 architecture. Both LiRA and our method are able to reliably identify train samples at very low false positive rates.
8

---

Table Precision of all membership inference attack 1% and 0.1% false positive rates on ResNet- 50 architectures. Our attack consistently dominates the marginal baseline and produces excellent results on ImageNet-Il Ik compared to shadow model approaches. We do not beat the shadow model approach the smaller CIFAR datasets, potentially owing to their small dataset size. On CINIC-10, a larger dataset than CIFAR, we obtain comparable results to shadow models. Additional results for the remaining architectures are presented in Appendix E
Precision @ 1% FPR
Precision @ 0.1% FPR
Method
C-10
C-100
CINIO
IN-IK
C-10
C-100
CINIO
IN-Ik
Marginal
48.56%
58.81% %
49.02%
47.62%
60.94%
65.75%
45.76%
46.81
LIRA (n=2)
78.55%
95.21% %
55.43%
62.70%
83. 18%
98.65%
54.07%
56.04%
LIRA (n=4)
80.52%
95.87%
78.71%
89.11%
91.48%
98.94%
86.99%
95.18%
LIRA (n=6)
83.19%
96.20%
88.75%
93.74%
93.17%
99.02%
96.40%
98.38%
LIRA (n=8)
83.00%
96.07%
91.86%
94.57%
93.70%
98.98%
97.94%
98.73%
Ours
62.95%
79.57%
76.67%
97.45%
64.48%
85.41%
85.46%
99.64%
Table [ shows precision of the proposed membership inference attack 1% and 0.1% false positive
rate on the ResNet-50 target networks (additional results shown in Appendix F Our attack robustly predicts membership in the private dataset with high precision even at low FPR. The proposed approach works on all target architectures and datasets, but works particularly well on the more complex and data intensive ImageNet-II Ik task.
Since LIRA produces an explicit score distribution ((s(1, y): plx, y). y)) based on the shadow model's 's predictions, we compare all 3 methods (Ours. LIRA, marginal baseline) in terms of pinball loss on public data (x ~ D). We find that the attack with the smallest pinball loss on public data is the better membership inference attack across all datasets. This shows that strong quantile predictor on public data is a strong membership inference attack; which validates the core premise of approach in Section
A possible explanation for the relative lack of success of directly learning a quantile regression model on CIFAR datasets could be the relatively low number of available samples (25000). In such low data scenarios, training shadow models is also computationally affordable. The opposite holds true for the much larger ImageNetlk k dataset, which the computational cost of training single target model, much less multiple shadow models far exceeds the cost of a single quantile regression model.
4.2 Tabular Classification Experiments
In addition to experiments on image datasets, we here demonstrate the effectiveness of our member- ship inference attack on tabular datasets, including large datasets from derived from the US Census' American Community Survey (ACS) (Ding et al. 2021 and small ones from OpenML (Grinsztajn al. 2022]. Gradient boosting with decision trees widely-used for classification tasks with tabular data, so in our experiments, to achieve reasonable performance, we train a gradient boosting model with 5-fold cross validation for hyperparameter tuning on the private portion of the data. For our attack model, gradient boosting with regression trees is applied, but now with regression targets as mentioned above. Hyperparameters for regression tasks are also tuned using public portion of the data to avoid overfitting. In our experiments, cathoost is used for model training, and Optuna Akiba ct al. 2019 is used for hyperparameter tuning.
Table shows that our attack, which involves learning a single regression model, performs on par with the LiRA attack, which requires learning at least 16 models on some tasks and more on other tasks. Since each model, including our regression model and shadow model in LiRA, has the same latency in terms of hyperparameter tuning and model training, our attack requires significantly less compute (equivalent to a single shadow model), and it reduces successful attack from training many models to only one model. Figure shows ROC curves of our proposed approach on OpenML.
5 Discussion
We have introduced a new family of membership inference attacks that are competitive with the state of the art (and in our ImageNet experiments, substantially and uniformly better), while requiring
hatps//wwww.pennl.org

---

Table 2: Precision @ 1% FPR on OpenML datasets and 0.5% on ACS datasets. Evaluating at different FPR levels is due to the fact that OpenML datasets have around 5,000 samples, and ACS datasets
have around 20,000 samples. n here denotes the number of shadow models trained for LIRA.
Precision @ 1% FPR (OpenMI.)
Precision @ 0.5% FPR (ACS NY)
Task
361057
361064
361067
361070
Coverage
Income
Travel
Mobility
LIRA (n=16)
70.33%
85.44%
85.52%
73.33%
66.07%
50.71%
66.07%
72.74%
LIRA (n=32)
76.22%
88.52%
89.62%
78.35%
66.28%
52.53%
67.52%
68.84%
L.RA (n=64)
82.73%
90.31%
89.46%
79.57%
69.23%
50.24%
65.64%
65.26%
Ours
83.35%
88.05%
87.35%
86.54%
67.31%
56.35%
63.98%
85.27%
Task ID 361057
Task ID = 381084
Task ID 38 106777
Task ID 361070
10
Positve Rete
Method
10
Quantile
LIRA (n=18)
LIRA (n=32)
LIRA (n=04)
10
10
10"
10"10"
10"
10'10"
10"
10'00
10"
10"
False Positive Rate
False Positive Rate
False Positive Rate
False Positive Rate
OpenML Datasets
Task coverage
Task travel
Task mobility
Task Income
10
Method
Positive
10
Quantile
LIRA (n=18)
LIRA (n=32)
(n=84)
10
10"
10
10"
10" 10'10"
10"
10 '10"
10"
10"
False Posltive Rate
Felse Positive Rate
False Positie Rate
False Positive Rete
(b) ACS Task NY data
Figure 3: Comparing the true positive rate vs. false positive rate of our membership inference attack against LIRA evaluated 16, 32, and 64 shadow models on OpenML and ACS datasets. single quantile regression model can produce similar results as multiple shadow models at a fraction of the compute cost for gradient boosting models.
substantially fewer computational resources and less knowledge of the target model. Moreover, we have identified pinball loss as a key target objective: uniformly across all of our experiments, the methods that produce thresholds minimizing pinball loss are the most effective attacks. Together, this brings membership inference closer practicality highlight a growing risk to privacy but also provides a more efficient means to audit models by subjecting them to our attacks. We hope that our methods encourage and enable a more widespread practice of auditing models for privacy violations by subjecting them to membership inference attacks before deployment.
large commercial models. This serves to
References
T. Akiba, Sano, T. Yanase, T. Ohta, and M. Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, 2019.
O. Bastani, V. Gupta, C. Jung, G. Noarov, R. Ramalingam, and A. Roth. Practical adversarial multivalid conformal prediction. In Neural Information Processing Systems (NeurIPS), 2022.
J. Bergstra, D. Yamins, and D. Cox. Making science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In International conference on machine learning, pages 115-123. PMLR, 2013.
10

---

N. Carlini, S. Chien, M. Nasr. S. Song, A. Terzis, and F. Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium Security and Privacy (SP), pages 1897-1914. IEEE, 2022.
L. N. Darlow, E. J. Crowley, A. Antoniou, and A. J. Storkey. Cinic-10 10 is not imagenet or cifar- 10. arXiv preprint
2018.
Ding, M. Hardt, J. Miller, and L.. Schmidt. Retiring adult: New datasets for fair machine learning. Advances in Neural Information Processing Systems, 34, 2021.
L. Grinsztajn, E. Oyallon, and G. Varoquaux. Why do tree-based models still outperform deep learning on tabular data? arXiv preprint arXiv:2207.088/55 2022.
V. Gupta, C. Jung, G. Noarov, M. M. Pai, and Roth. Online multivalid learning: Means, moments, and prediction intervals. In 13th Innovations in Theoretical Computer Science Conference (ITCS 2022). Schloss Dagstuhl-Leeinizz-i
für Informatik, 2022.
K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. arxiv 2015. arXiv preprint arXiv:15/2.03388; 14, 2015.
U. Héhert-Johnson, M. Kim, O. Reingold, and G. Rothblum. Multicalibration: Calibration for the (computationall--iennffffaa masses. In International Conference on Machine Learning, pages 1939-1944 PMLR, 2018
N. Homer, S. Szelinger, M. Redman, D. Duggan, W. Tembe, J. Muchling, J. V. Pearson, D. A. Stephan, S. F. Nelson, and D. W. Craig. Resolving individuals contributing trace amounts of dna to highly complex mixtures using high-density snp genotyping microarrays. PLOS Genetics, 4(8): http:p/p://aol..r/10011.111////
1-9, 08 2008. doi: /o.r37//ourmal..gen...ooo 1000167.
pgen 1000167
B. Jayaraman, L. Wang, K. Knipmeyer, Q. Gu, and D. Evans. Revisiting membership inference under realistic assumptions. arXiv preprint arXiv:2005. 10881, 2020.
B. Jayaraman, Wang, K. Knipmeyer, Q Gu, and D. Evans. Revisiting membership inference under realistic assumptions. Proceedings on Privacy Enhancing Technologies, 2021:348-368, 04 2021.
C. Jung, G. Noarov, R. Ramalingam, and A. Roth. Batch multivalid conformal prediction. In International Conference on Learning Representations (ICLR), 2023.
A. Krizhevsky, G. Hinton, et Learning multiple layers of features from tiny images. 2009.
L. Li, K. Jamieson, A. Rostamizadeh, E. Gonina, J. Ben-Tzur M. Hardt, B. Recht, and Talwalkar.
A system for massively parallel hyperparameter tuning. Proceedings of Machine Learning and Systems, 2:230-246, 2020.
and Y. Zhang. Membership leakage in label-only exposures. Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, 2020.
R. Liaw, E. Liang, R. Nishihara, P. Moritz, J. E. Gonzalez, and I. Stoica. Tune: A research platform for distributed model selection and training. arXiv preprint arXiv: 1807.05 18, 2018.
Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, Darrell, and S. Xie. convnet for the 2020s. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pages 11976-1199 1986, 2022.
Y. Long, V. Bindschaedler, L. Wang, D. Bu, X. Wang, H. Tang, C. Gunter, and Chen. Under- standing membership inferences well-generalized learning models. CoRR, abs/ 0ts//802.0444 180.00488, 2018.
//axxvv./ www./bg////aa.eeee..oo.o 1802. 04889
Y. Long, Wang, D. Bu, V. Bindschaedler, X. Wang, H. Tang, C. A. Gunter, and K. Chen. A pragmatic approach to membership inferences on machine learning models. In 2020 IEEE European Symposium on Security and Privacy (EuroS&P), pages 521-534. IEEE, 2020.
Neyman and S. Pearson. On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society A, 231:289- 1933.

---

G. Noarov and A. Roth. The scope of multicalibration: Characterizing multicalibration via property elicitation. International Conference on Machine Learning (ICML), 2023.
A. Roth.
Uncertain:
Modern topics
uncertainty
estimation.
http://www.cis.upem. eedeenneedu///ee paroot//ncertaiint--nteee. 2022.
O. Russakovsky, J. Deng, H. Su, Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recog- nition Challenge. International Journal of Computer Vision (IJCV), 115(3):211-222 2015. doi: 10. 1263-015-08166..
A. Sablayrolles, M. Douze, C. Schmid, Y. Ollivier, and H. Jégou. White-box vs black-box: Bayes optimal strategies for membership inference. In International Conference Machine Learning, pages
PMLR, 2019.
A. Salem, Y. Zhang, M. Humbert, P. Berrang, M. Fritz, and M. Backes. MI-leaks: Model and data independent membership inference attacks and defenses on machine learning models. arXiv preprint arXiv: 01246,, 2018.
R. Shokri, M. Stronati, C. Song, and V. Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3-18. IEEE, 2017.
L. Song and P. Mittal. Systematic evaluation of privacy risks of machine learning models. In USENIX Security Symposium, volume 1, page 4, 2021.
L. Watson, C. Guo, Cormode, and Sablayrolles. On the importance of difficulty calibration in membership inference attacks. arXiv preprint arXiv:2111.00 1.0840, 2021.
Y. Wen, A. Bansal, H. Kazemi, E. Borgnia, M. Goldblum, J. Geiping, and 'T. Goldstein. Canary in a coalmine: Better membership inference with ensembled adversarial queries. In The Eleventh International Conference on Learning Representations, 2023. URL forum?id=b7SBTEBEn
net/
J. Ye, A. Maddi, S. K. Murakonda, and R. Shokri. Enhanced membership inference attacks against machine learning models. CoRR, abs/2111.00679, 2021. URL htt.:/ttps///ariv.oo//////////// 09679]
S. Yeom, I. Giacomelli, M. Fredrikson, and S. Jha. Privacy risk in machine learning: Analyzing the connection overfitting. 2018 IEEE 31st computer security foundations symposium (CSF), pages 268-282. IEEE, 2018.
S. Zagoruyko and N. Komodakis. Wide residual networks. arXiv preprint arXiv: 1605.07146, 2016.
12

---

A Shadow Model Architecture Mismatch
Here explore how shadow model membership inference attack is affected by the lack of knowledge the target model. For this, we vary the shadow model's and target model's architecture between different configurations: 3 CNN models with 32 filters each and varying number of pooling layers between and 3, Wide Residual Networks Zagoruyko and Komodakis (0166 of depth 28 and varying width (11, 10)). We train single target model and shadow models; all models trained with SGD with momentum and random augmentations. The results are shown in Figure 4 where small architecture mismatches (i.e, model family of target and test architecture) generally degrade performance very little, but larger architecture mismatches can cause significant performance degradation.
Presision 20 LISERR cifar100
Procision 010 15SEPR,
cifar100
992%
1.%%
1.5%
9%
90.4%
44.6%
04.4%
44%%
1.9%
60.9%
00.4%
P4.5%
DA.4%
Drpee Mode Sac
16.5%
1..%%
80.5%
84.5%
9% $3.5%
$6.8%
84.%
6.5%
78.1%
80.6%
$4.5%
$3.5%
$6.8%
Jrgeet
6..5%
59.3%
55.0%
$1.8%
55.%
94.5%
75.00
55.0%
$1.5%
51.2%
$8.40 4%
34.5 5455 5%
9%
G41.3%
5%
91.2%
9%
S 5%
@41.3%
66.5%
GM4.0 00%
31.1%
96.5%
@11.1%
10.
R4 5%
77.8%
KKS. 13
10.9%
$2.4%
1%
880.3%
92.4
mare
wm288.5
wm28--1
cmd Lmas
cm Frm
wm225
Shadow Model Siz
Shadow Model Sze
CIFAR-10
(b) CIFAR-100
Figure 4: Comparing the precision at 0.1% false positive rate of the LIRA attack on CIFAR- 10 and CIFAR- 100 on mismatched target and shadow model architectures. Red circles are used denote scenarios where target and shadow model architectures match.
Target Model Accuracies
Here we summarize the accuracy of all target model architectures.
Architecture
Dataset
ResNet-10 Acc
ResNet-18 Acc ResNet-34 Acc ResNet-50 Acc
CIFAR-T0
90.3%
90.5%
90.7%
91.0%
CIFAR-100 100
66.7%
68.6%
68.3%
68.6%
ImageNet-Ik
67.5%
Table Accuracy of target classifiers. All target classifiers are trained on 50% of the usual training data split using the training setup described in He al. [2015). Due the reduced ammount of training data, these target networks have lower accuracies than those initially reported in He et al. 2015).
C Proofs
C.1 Proof of Theorem
Proof. To establish this theorem, the following lemma will be useful, which informally states that we can shift model's quantile predictions to get "marginal coverage rate" be closer to the target then we will also decrease the pinball loss of that model:
13

---

Lemma 3 (Jung et al. [2023], Roth 2022)). ), Fix any continuous distribution P E AR with density
bounded by p. Suppose q a model such that Pfy ly S [[z)]
= a', and let A IR be such
that Pry-plu y S ((x)) + A = a. Let ((x) =(x)) + A. Then:
(a o2)2
1 (0(2), v\1
E (PB1-a(((E)) v)1 v~p
y~p
2p
Recall that g is chosen such that:
q E arg min ( s(,
PB
E'EH (z.8)
For point of contradiction, suppose that (PR(Ag) a' for some a' + a. Expanding out the definition of the false positive rate, we have that:
a' FPR(A,)
Pr (2 = =1]
(aN))D
Pr [s(2, y) 2 9(x)]
(a
= 1 Pr [s(x,)) v) S {()))
(z.g)~
So ((x.))/=(s(x,, y) I ([x)] = 1 a' Let A E R be such that PT(z.))D((((,, y) S g(x) + A =
- a Note that such a A is guaranteed to exist by continuity of the distribution on s(r, y). Let (xxx) ((x) + A. By Lemma 3
E
(B s(x,))))
E (2.9)~D
PBI-a PB)-a(((2), s(r,
(a.y)~D
But because H. is closed under additive shifts, we also have that o' E H. Together, these contradict the optimality of as measured by pinball loss, which completes the proof.
C.2 Proof of Theorem 2
Proof. Recall that is chosen such that:
q arg min E [BB (PBI-a ((()), E'EH (2.8)~D
For point of contradiction, suppose that there is some g and some a' A a such that A, false positive rate conditional on g(x) 1 is a'. Let D2 be the conditional distribution on D conditional on g(x) = 1. and let D3 be the conditional distribution on conditional on g(z) = O. Expanding out definitions, we have that:
a'
Pr (g( = -(((t) = =1]
Pr [s(I,)) 2 {(x)]
(n.y)>D,
Pr [s v) S {(x))
(a,y)-D,
So (, 3) S [(x)] = a'. 1e E R be such that Pr(ma)) [s(x, v) S (()) + -
1 - a-Note that such an v is guaranteed to exist by continuity of the distribution on 8(x,1) v) conditional on (xx) = 1. Let g'xx) g(xr) nz(x). By Lemma 3
E
[ (d' (() v)) < E
[PB (P))
(m.
(a.9)~D,
We can relate this decrease in pinball loss conditional on g(z) = 1 to the decrease in pinball loss on the underlying distribution D:
E PBB PPBi-a s(, (a
Prla(x) = 1]
PB] (BI-a(d'x)) + = 0) E
[B (1')), (x), y))
(m,w)~Dg
<
D Prio(z) = E
s0, ( PF(a(z) = 0] E
P (g(x), s
(x,9)>D,
(a.
Pr(g(z)= = (PBi-a s(2 Pr(g(=) = o E
P (()
(x,
(a,y)~D,
14

---

But because H. is closed under additive shifts by all E G, we also have that g' E H. Together, these contradict the optimality of q as measured by pinball loss, which completes the proof.
D Hyperparameters
We use Ray Tune Liaw al. 2018 for hyperparameter tuning image datasets. All experiments use Asyne Hyperband Scheduler Li et al. [020] and the Hyperopt search package Bergstra et al. 2013 Table 4 summarizes the hyperparameters that were tuned and their configurations
Table 4: Summary of hyperparameters optimized for our quantile regressor model on all image experiments
Hyperppraaeeeer
Configuration
Description
Ir
Inguniform( 10 6,
Learning rate
Weight Decay Hidden dims
loguniform(2 10-6, 10-3)
12 weight regularization (excluding biases)
choicc([]. (512,512])
size and number of hidden dimensions of MLP number batches accumulate (base batch sizc=32) number training epochs
Accumulate gradient batches Epochs
choice([1, 2, 8, 16, 32, 32, 64)
randint(stari=5, stop=50, step=5)
We ran 32 trials to find the optimal configuration per experiment. After examining the results, found that a hidden dim of [512, 512] and accumulate gradient batches (Effective batch size 64) were consistently chosen across all experiments. Similarly, epochs mostly chosen the 10 - 20 range, learning rates and weight decays were consistently chosen near the middle of the value range. This indicates that hyperparameter tuning may not be especially task sensitive and can be shared across attacks.
For tabular data, eter tuning. (aalle5]prseen the hyperparameters that were tuned and their corresponding ranges. Each model was tuned with 300 trials with 5-fold cross-validation.
Catboost for model training, and Optuna Akiba et al. 2019 for hyperparam-
Table 5: Summary of hyperparameters optimized for our quantile regressor model on tabular data Hyperparameter
Configuration
Description
depth 12 1caf reg (carning_rate rate subsample iterations
uniform(1,10)
Depth of tree
loguniform( (eetrmmmmc21111e1 (c-2,1c+6)
Strength of L2 regularization
loguniform( lc-6,1)
Learning rate of gradient boosting
loguniform( 1e-2,1)
Subsampling ratio each leave node
loguniform(1,1 1000)
Number of boosting iterations
E
Additional Results
Here we show extended results on membership inference attacks on CIFAR-1O and CIFAR-100 for
ResNet- 10, -18,- -34, and -50 architectures. Tables 6 and 7 show precision and pinball loss at 1% and 0.1% FPR for target architectures on CIFAR- 10 and CIFAR-100 respectively. We observe strong correlation between lowest pinball loss test samples and highest precision across the majority of experiments. Figure 5 additionally presents a visual comparison of the true positive rate and false positive rate trade-off for all tested methods on CIFAR-10 and CIFAR-100 for the ResNet-50 architecture.
15

---

Table Precision and pinball loss of all membership inference attack 1% and 0.1 11 false positive rates on CIFARIO 10 for ResNet-10, -18, -34, and -50 architectures. Pinball losses are computed on held out test set. Lower pinball losses consistently predict better membership inference performance.
Method
PB1x
Precision @ 1% FPR PBo.1%
Precision @ 0.1% FPR
CIFAR- 10
ResNet-10
LIRA (n=2)
0.1454
80.43%
0.0194
89.20%
LIRA (n=4)
0.1439
83.45%
0.0193
93.15%
LIRA (n=6)
0,1441
84.46%
0.0193
94.47%
LIRA (n=8)
0.1442
84.79%
0.0193
94.67%
Marginal Baseline
0.2157
47.84 47.84%
0.0262
46.81 %
Ours
0.1543
62.69%
0.0262
57.83%
CIFAR- 10 ResNet-
LIRA (n=2)
0.1637
82.17%
0.0222
94.09%
LIRA (n=4)
0.1607
84.91%
0.0216
93.90%
LIRA (n=6)
0.1603
85.33%
0.0215
94.68%
LIRA (n=8)
0.1603
85.26%
0.0215
95.04%
Marginal Baseline
0.1853
43.43%
0.0223
50.00%
Ours
0.1707
63.08%
0.0213
65.65%
CIFAR- 10 ResNet-34
LIRA (n=2)
0.1674
82.75%
0.0225
92.45%
LIRA (n=4)
0.1672
84.16%
0,0223
94,45%
LIRA (n=6)
0.1669
85.65%
0.0222
95.40%
LIRA (n=8)
0.1668
85.79%
0.0222
95.30%
Marginal Baseline
0.1893
51.42%
0.0230
47.92%
Ours
0.1743
73.06%
0,0230
80.39%
CIFAR- 10 ResNet-50
LIRA (n=2)
0.1913
78.55%
0.0262
83.18%
LIRA (n=4)
0.1926
80.52%
0.0261
91.48%
LIRA (n=6)
0.1930
83.19%
0.0260
93.17%
LIRA (n=8)
0.1933
83.00%
0.0260
93.70%
Marginal Baseline
0.2163
48.70%
0.0287
60.94%
Ours
0.2070
62.95%
0.0277
59.68%
10a
True Positive Rate
True Positive Rate
10-1_
10-1
Ours
Ours
LIR (nm2)
URA (nm2)
(n-4)
(n-4)
10-2
LRRA (1=6)
10-2
URA (n=6]
LIRA
LIRA (n=256 Visual)
URA (n=256 Visual)
Marginal Baseline
Marginal Baseline
10-3
10-1
10
10-2
10
10a
10-3
10-2
10
False Positive
Positive
(a) CIFAR-10
(b) CIFAR-100
Figure 5: Comparing the positive rate false positive rate of membership inference attack against the state-of-the-art shadow model approach LIRA proposed in Carlini et al. [2022] evaluated 2, 6, and 8 shadow models, and the marginal baseline proposed in Yeom al. [2018] Our single-model quantile regression attack can reliably perform membership inference attacks on ResNet-50 LIFAR-1O/CIER
100 target model (91% and 68.6% test accuracies respectively) without relying on any knowledge of the target architecture. We include a visual readout of the 256-shadow model attack shown in Carlini al. [202] for reference. Our attack's effectiveness is dominates the marginal baseline but falls short of LIRA in this scenario. We find pinball loss to be strong predictor of performance for membership inference attacks.
16

---

Table 7: Precision and pinball loss of all membership inference attack at 1% and 0.1 % false positive rates on CIFARI00 for ResNet- 10, 18, -34, and -50 architectures. Pinball losses are computed on held out test set. Lower pinball losses consistently predict better membership inference performance. Method
PB1x
Precision @ 1% FPR PB..1%
Precision 0.1% FPR
CIFAR-100 ResNet-10
LIRA (n=2)
0.1486
95.44%
0.0188
99.06%
LIRA (n=4)
0.1386
95.95%
0.0170
98.97%
LIRA (n=6)
0.1385
96.19%
0.0170
99.10%
LIRA (n=8)
0.1385
96.14%
0.0170
99.06%
Marginal Baseline
0.2749
50.84%
0.0390
47.92%
Ours
0.1775
83.30%
0.0390
77.37%
CIFAR-TOU 100
ResNet-18
LIRA (n=2)
0.1612
96.29%
0,0193
99.1 11%
LIRA (n=4)
0.1523
96.74%
0.0179
99.46%
LIRA (n=6)
0.1524
97.02%
0.0181
99.51%
LIRA (n=8)
0.1525
96.94%
0.0181
99.48%
Marginal Baseline
0.2754
51.71% %
0.0364
59.02%
Ours
0.2492
90.37%
0.0364
89.63%
CIFAR 100
ResNet-34
LIRA (n=2)
0.1840
95.73%
0.0213
99.10%
LIRA (n=4)
0.1732
96.34%
0.0203
99.17%
LIRA (n=6)
0.1726
96.45%
0,0202
99.31%
LIRA (n=8)
0.1727
96.42%
0.0202
99.28%
Marginal Baseline
0.2065
62.13%
0.0270
66.67%
Ours
0.1913
81.80%
0.0270
79.64%
CIFAR- 100 ResNet-50
LIRA (n=2)
0.1926
95.21 %
0.0239
98.65%
LIRA (n=4)
0.1832
95.87%
0.0227
98.94%
LIRA (n=6)
0.1828
96.20%
0.0228
99.02%
LIRA (n=8)
0.1828
96.07%
0.0228
98.98%
Marginal Baseline
0.2188
58.81%
0,0272
65.75%
Ours
0.2006
79.57%
0.0272
85.41%
17


Reviews:
[{"review_id": "oeTkP0ZJqV", "reviewer": "Reviewer_Fy4T", "summary": "This paper studies the question of membership inference attack (MIA), which can be formalized as a hypothesis testing (HT) problem.\nThe main contribution of this paper is introducing a new class of MIA. \nThe authors claim that the proposed method is competitive with SOTA MIA methods while being more computationally efficient.\nThe authors explain the motivation behind their method and theoretically prove the method's effectiveness (in terms of false positive rate).\nThis paper also conducts experiments on various datasets to evaluate their method.", "questions": "1. It seems unintuitive that the proposed method outperforms Carlini et al [2022]. Could you please explain this phenomenon?\n2. Theorem 1 assumes that $\\mathcal{H}$ is closed under additive shifts. Are there any examples of such hypothesis classes?", "limitations": "No", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper explains the relationship between MIA and HT, which is friendly to readers unfamiliar with MIA.\n2. This paper compares the proposed method with those methods in Yeom et al [2018] and Carlini et al [2022]. The proposed methods seem to be a direct generalization of Yeom et al [2018]. \n", "weaknesses": "My major concerns lie in the theoretical part of this paper:\n1. Theorems 1 and 2 are established at the population level. The approximation error of $q$ is not discussed.\n2. This paper does not include any non-asymptotic results.\n2. The theoretical part of this paper only compares the proposed method with the baseline attack (Yeom et al [2018]). The difference between the proposed method and Carlini et al [2022] is not discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful reading and useful feedback — below we address your specific questions. We’re happy to further discuss any of these points if there are any remaining questions or confusion!\n\n> *Theorems 1 and 2 are established at the population level. The approximation error of is not discussed.*\n\nThe reviewer is correct that our theorems are stated at the population level/without approximation error. That said, under the assumption that the learning of $q$ is done from some family of functions with bounded fat-shattering dimension (or other analogous quantity), standard sample complexity and uniform convergence results will imply guarantees in the finite sample setting. The reviewer is correct to note that to get worst-case theoretical bounds from finite data, one would want generalization theorems: but since the learning portion of our attack is simply solving a bounded, Lipschitz, convex optimization problem, generalization issues (and their solutions) are standard and not the focus of our work. We never rely on our theorems when evaluating our method: all results are empirical, and computed on a holdout set. The theorems are meant to give guiding intuition. We are happy to include a discussion of these points and a pointer to textbook generalization theorems for convex optimization. \n\n> *The theoretical part of this paper only compares the proposed method with the baseline attack (Yeom et al [2018]). The difference between the proposed method and Carlini et al [2022] is not discussed.*\n\nThe baseline attack and our method both have guarantees that hold in the worst case, without making parametric assumptions. Carlini et al. [2022] do not have comparable theoretical results. Their approach estimates score distributions parametrically, so any comparable guarantees would require making corresponding parametric assumptions about the distribution of scores over the randomness of the dataset and model training. We can add a discussion of this point in the revision. \n\n> *It seems unintuitive that the proposed method outperforms Carlini et al [2022]. Could you please explain this phenomenon?*\n\nThere are two primary reasons that our approach outperforms Carlini et al. **in some scenarios**.\n\nThe first, as mentioned above, is that their method fits a simple parametric model to the scores sampled from the shadow model, and to the extent that their parametric model is ill-specified, the success of their approach will degrade; in contrast our quantile regression method (when implemented via pinball loss minimization) is non-parametric. \n\nSecond, we formulate our hypothesis testing problem differently from Carlini et al., in a way that better captures the attack scenario. The hypothesis testing formulation in Carlini et al. operates on distributions of shadow models that are generated by running the training algorithm on random input datasets that purposefully include or not include the target attack example.\nThus their hypothesis test is designed to attack a random model sampled from some specified training distribution, rather than the particular (realized) model that the membership inference attack is being launched on. In contrast, in the way we cast the hypothesis testing problem, the model under attack is fixed, and the randomness is entirely over the selection of the point under attack. Thus our hypothesis test is targeted at the specific model under attack, rather than a random model sampled from the same distribution. This better fits the actual threat model. We will add a discussion of this to the paper. \n\n> *Theorem 1 assumes that is closed under additive shifts. Are there any examples of such hypothesis classes?*\n\nYes! Many hypothesis classes are closed under additive shifts. This includes linear and polynomial regression models, regression tree models, and any neural network architecture that has a bias/offset term. More generally, any model architecture can be made to be closed under additive shifts by adding a single additional parameter (a bias/offset term, when one does not already exist). Thus we view the additive shift assumption to be extremely mild, and easily enforceable if necessary. "}, {"review_id": "2gn2kLewwb", "reviewer": "Reviewer_bLLW", "summary": "The authors propose a novel class of membership inference attacks based on quantile regression applied to confidence score distributions. The proposed 'black-box' algorithm does not require knowledge of the model's architecture and performs competitively with state-of-the-art shadow model attacks while being computationally more efficient. The paper presents several experiments showcasing the effectiveness of the approach across various datasets.\n", "questions": "1. Could you provide a sensitivity study regarding the architecture of q? Is it an important factor? Will the method perform better when q uses the same architecture as f? Will it lead to more accurate quantile estimation?\n\n2. I mentioned my concerns about the theoretical part above.\n\n3. The experimental results do not perform as well on small datasets. While the computation of LIRA when n=2 seems acceptable and achieves much better performance than your method, could you explain the observed gap?", "limitations": "I did not find the potential negative societal impact.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written, with a clear and understandable motivation.\n\n2. The technical aspects are solid and coherent.", "weaknesses": "1. In my opinion, the main contribution of this paper, which involves using the quantile regression model q to predict the quantiles of the score s(x,y), is not a novel idea within quantile regression. [a] has previously utilized pinball loss in conformal prediction for estimating quantiles of output. Combining quantile regression with membership inference might not present a significant technical challenge and may seem trivial.\n\n2. The claim that the method does not require any knowledge of the model's architecture appears overstated. The method's performance heavily depends on the quantile regression model q. If q is not well-trained, the method will fail. The paper only employs the ConvNext-Tiny model in the main paper, and the sensitivity to different model architectures remains unknown.\n\n3. I find the theoretical part confusing. In my understanding, Theorems 1 and 2 do not establish the theoretical superiority of the proposed method compared to the baseline method using constant quantiles. The paper introduces group validity, which has been extensively studied in the papers mentioned in Line 253, resulting in a lack of theoretical novelty. Additionally, even if the quantile regression model q is learned from a richer set of models, there is no concept of a 'regression model q' for the baseline method. Therefore, it is unclear how the authors concluded that the proposed method outperforms the baseline attack. It seems more intuitive to conclude that a richer model complexity for q may benefit the proposed method.\n\n[a] Romano et al., Conformalized Quantile Regression, NeurIPS 2019", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful reading and useful feedback — below we address your specific questions. We’re happy to further discuss any of these points if there are any remaining questions or confusions!\n\n> *In my opinion, the main contribution of this paper, which involves using the quantile regression model q to predict the quantiles of the score s(x,y), is not a novel idea within quantile regression...*\n\nQuantile regression is a classical statistical method for estimating the quantiles of a score distribution. It has been applied in many domains over the course of decades, including recently within conformal prediction, as noted by the reviewer. We do not claim to be developing novel quantile regression techniques; we use quantile regression as a tool to develop a new, scalable membership inference attack, which gives substantial improvements over the state-of-the-art on large datasets. \n\n> *The claim that the method does not require any knowledge of the model's architecture appears overstated...*\n\nWe agree that this point can be elaborated on in our paper to give further substantiation. We note at the outset that the results we report use the same quantile regression architecture to attack vision models of different architectures; in contrast the results we report for shadow models use custom architectures for each model under attack. Here are the primary rationales for our claim. \n- From a theoretical perspective, because pinball loss minimization is a non-parameteric method for quantile regression, the regression function $q$ need not be from a class containing the target model (in fact, they can be completely unrelated classes). Of course the model class needs to be sufficiently expressive to obtain low loss, but this is qualitatively different than the use of shadow models, which are used to sample from the same distribution on scores as the original model did. \n- Our experiments found that large image datasets and a variety of architectures were attackable with convolutional architectures like ConvNext or ResNet; smaller, tabular datasets and models were attackable with gradient-boosting trees.\n\nAs an example of architecture robustness, we were able to achieve $76.67\\%$ and $ 85.46\\%$ precision at $1\\%$ and $0.1\\%$ FPR, respectively, attacking a ResNet-50 model with a ResNet-50 quantile regression model on the CINIC10 dataset. This performance is roughly comparable to $4$ shadow models in the same conditions ($78.71\\%$ and $86.99\\%$ Precision respectively) with perfect knowledge of the target model's training parameters. These results are shown in more detail in the accompanying pdf.\n\n> *I find the theoretical part confusing. In my understanding, Theorems 1 and 2 do not establish the theoretical superiority of the proposed method compared to the baseline method using constant quantiles...*\n\nAlthough this is not how Yeom et al. describe their method, if one wants to view our attack as a generalization of Yeom et al, one can view their attack as solving a pinball loss minimization problem over the class of constant threshold models. This is because, over the class of constant functions, pinball loss is minimized at the threshold that corresponds to the target quantile of the score distribution. In this framing, our attack is a generalization in that it solves a quantile regression problem over a strictly larger model class. As we optimize over richer model classes, we get strictly stronger guarantees---the most obvious are that we find models that have lower pinball loss. As we highlight in the theory section of the paper, our guarantees are also stronger in the sense that we provide group conditional coverage--that is, the false-positive rate is no more than a target level for a large collection of subgroups. In contrast, Yeom et al. only provide a marginal guarantee that ensures a target false-positive rate over the entire distribution. Of course, the main demonstration of the superiority of our attack is given in the empirical results on large datasets. \n\n> *Could you provide a sensitivity study regarding the architecture of q?...*\n\nOur empirical results do not use the same architectures for the models we attack and our quantile regression functions. We find, empirically, that the ability of the model class class to minimize pinball loss is an excellent predictor of how well the attack will work. Just as with other learning tasks, more expressive classes will perform better if there is sufficient training data. \n\n> *The experimental results do not perform as well on small datasets...*\n\nFor small datasets (and small models), you are correct that LIRA's performs quite well. As we discuss in the paper, we find that in these cases, we can extract a quantile predictor from LIRA's shadow models that has lower pinball loss than the one we train directly, and so the gap appears to be due to the difficulty of directly optimizing for pinball loss in relatively data poor settings. We note that for relatively small datasets and models, shadow model attacks like LIRA are not extremely expensive to run: the running time benefits of our approach are most apparent for learning problems corresponding to large datasets and models, which fortunately is also the setting in which our attack performs the best. This is further supported by our additional experiments on CINIC10, a generalization of the CIFAR10 dataset, where we achieve $76.67\\%$ and $ 85.46\\%$ precision at $1\\%$ and $0.1\\%$ FPR respectively against a ResNet-50 model with a ResNet-50 quantile regression model. This performance is roughly comparable to $4$ shadow models in the same conditions ($78.71\\%$ and $86.99\\%$ precision respectively). The computational cost of our attack, including extensive hyper-parameter tuning, is roughly $80\\%$ that of the comparable shadow model attack. See the attached pdf for additional details on these results."}, {"review_id": "maKqTaeW2V", "reviewer": "Reviewer_NGxz", "summary": "The main focus of this paper is about the membership inference attack problem, i.e., determining whether  a particular example was used in training or not. Most existing such attacks estimate the distribution of some test statistics, which are usually computationally expensive. In contrast to existing approaches, this paper proposes to perform quantile regression on the distribution of confidence scores induced by the model under attack on points that are not used in training. For implementation, the authors first collect a dataset known to be never used in training. Given a well trained model $f$, the corresponding confidence scores on collected data can be obtained. Thereafter, a quantile regression model $q$ is trained to predict the  target quantile of the data-label pair.", "questions": "All the current MIAs, including this submission, are based on the observation that models tend to overfit their training sets. Will it still be true for current big models trained on big data? My understanding is that the assumption under MIAs is mainly caused by the generalisation ability of most models.\n\nWill the conclusion still be true in other domains, e.g., natural language and audio?", "limitations": "Please refer to the weaknesses shown above.\n\nMoreover, the contribution of this work appears to be rather limited in scope. To put it simply, the essence of this study involves training a regression model using only a subset of the available training data. It therefore makes that the significance and potential impact of this work may be somewhat restricted.\n\nIt would be better to show experiments on other image models, such as based on VIT, SwinT, BEIT, image encoder in CLIP, .etc.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Reducing many (around hundred) shadow models to a single one is interesting.\n\nExperiment studies on different image datasets, including CIFAR-10, CIFAR-100, ImageNet-1K.", "weaknesses": "-- Paper writing still needs some improvements. It requires efforts to follow. \n\n-- The parameter $\\alpha$ is important yet hard to adjust manually. \n\n--  Currently, I have not observed any significant influence of the ongoing work on industry-level applications. Despite this, it is important to note that through additional analysis and exploration, there may be a possibility of uncovering substantial potential for impact in the future. As of now, the ongoing work does not appear to have made any noticeable waves in the industry. Nevertheless, by delving deeper into its intricacies and exploring its possibilities, we might unveil its capacity to bring about significant changes that could shape the industry landscape in the coming years.\n\n-- The authors fail to provide a comprehensive explanation of the significance of membership inference attacks (MIAs). It remains unclear why it is crucial to prevent the disclosure of specific data points that were used in training a model. If we assume that we have already inferred the utilization of certain data during the training process, what kind of significant problems could arise as a result? While consulting the references cited in the submission may shed some light on these matters, the paper itself should strive to be self-contained and inclusive, providing a thorough understanding of the subject matter without relying heavily on external sources.\n\n-- In order to facilitate better comprehension, it is necessary to reorganize the related work more effectively. By improving the organization of the related work, we can enhance its accessibility and ensure that readers can easily grasp its significance. A well-structured presentation of the related work will contribute to a clearer understanding of the subject matter.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful reading and useful feedback --- below we address your specific questions. We're happy to further discuss any of these points if there are any remaining questions or confusions!\n\n> *The parameter $\\alpha$ is important yet hard to adjust manually.*\n\n$\\alpha$ is the desired false positive rate of the attack, not a hyper-parameter that needs to be optimized. Pinball loss minimization with target quantile $1-\\alpha$ explicitly produces a quantile regression model with false positive rate $\\alpha$. Moreover, if we want an ensemble of attacks with multiple false positive rates, this doesn't require learning multiple quantile regression models from scratch. It can be efficiently done via multi-task learning with a shared neural network representation (and a separate ``head`` layer per target quantile).\n\n> *Currently, I have not observed any significant influence of the ongoing work on industry-level applications. Despite this, it is important to note that through additional analysis and exploration, there may be a possibility of uncovering substantial potential for impact in the future. As of now, the ongoing work does not appear to have made any noticeable waves in the industry. Nevertheless, by delving deeper into its intricacies and exploring its possibilities, we might unveil its capacity to bring about significant changes that could shape the industry landscape in the coming years.... -- The authors fail to provide a comprehensive explanation of the significance of membership inference attacks (MIAs). It remains unclear why it is crucial to prevent the disclosure of specific data points that were used in training a model. If we assume that we have already inferred the utilization of certain data during the training process, what kind of significant problems could arise as a result? While consulting the references cited in the submission may shed some light on these matters, the paper itself should strive to be self-contained and inclusive, providing a thorough understanding of the subject matter without relying heavily on external sources*\n\nThanks for the suggestion -- we are happy to devote more space towards motivating membership inference attacks. There are several different kinds of ''privacy attacks'' on trained models, and membership inference attacks are the simplest. There are broadly two main reasons to be interested in membership inference attacks: \n\nFirst, membership inference attacks are building blocks that are used to launch stronger kinds of attacks, like data extraction attacks, which extract training data from the models given API access --- see e.g. Extracting Training Data from Diffusion Models, Carlini et al. 2023. Improvements in membership inference lead to improvements across the entire stack of attacks based on membership inference.\n\nSecond, the guarantee of differential privacy (a strong notion of privacy that has been adopted by companies including Apple, Google, and Microsoft) is exactly that membership inference attacks can have True Positive/False positive rate curves that lie boundedly above the random guessing baseline (the diagonal on our plots). See e.g. the paper ''[Gaussian Differential Privacy](https://rss.org.uk/RSS/media/Training-and-events/Events/2020/Dong-et-al-jrssb-final.pdf)'' by Dong, Roth, and Su for an overview of the hypothesis testing view of differential privacy. So launching a successful membership inference attack falsifies a differential privacy guarantee, and is thus a form of privacy auditing that is gaining attention in industry --- see e.g. ''[Privacy Auditing with One (1) Training Run](https://arxiv.org/abs/2305.08846)'' by Steinke et al.  Improving the scalability of membership inference attacks (as our paper does) makes this form of privacy auditing more tractable. \n\n> *All the current MIAs, including this submission, are based on the observation that models tend to overfit their training sets. Will it still be true for current big models trained on big data? My understanding is that the assumption under MIAs is mainly caused by the generalisation ability of most models.*\n\nYes: The trend we see in this work is that our attacks work better on larger training sets and larger models. This suggests that these issues may get worse, not better, in larger data and larger model regimes. Other work [2] has shown that MIAs are effective on even larger datasets and architectures than those we used here (i.e. large language models), suggesting that membership inference attacks continue to be problematic in these regimes. \n\n[2] Extracting Training Data from Large Language Models, Carlini et al. 2021\n\n> *Will the conclusion still be true in other domains, e.g., natural language and audio?*\n\nThere is no reason a priori our approach would not work on these other domains, though the relevant tasks there are generally not classification. Carlini et al. 2021 and Carlini et al. 2022 both have demonstrated that it was entirely possible to attack large language models, and it is reasonable to assume that our attack would also be effective there.\n\n> *Moreover, the contribution of this work appears to be rather limited in scope. To put it simply, the essence of this study involves training a regression model using only a subset of the available training data. It therefore makes that the significance and potential impact of this work may be somewhat restricted.*\n\nWe would like to clarify our regression model is trained on **validation** data, not training data, meaning this attack does not require that the attackers have access to any training data: the attack can be launched by anyone with API access to the model."}, {"review_id": "9fjTHUHvqA", "reviewer": "Reviewer_5DZs", "summary": "The paper proposes a new membership inference attack based on training an attack model with the pinball loss. This avoids the use of shadow models, a common technique for membership inference, while often outperforming attacks which do use shadow models. They evaluate their attacks on a variety of image and tabular data, and show it is competitive with or outperforms the state of the art LiRA attack on these datasets.", "questions": "Is the pinball loss used at all when attacking the CIFAR models? If it is, could you try to explain in more detail how it is used?\n\nDo you have thoughts on why the attacks start to perform worse than other attacks at higher FPR?", "limitations": "The paper does not discuss limitations. It has minor overclaiming in some places, such as when saying the attack requires only a single model, when all results are presented after hyperparameter tuning. Another place is in claiming existing shadow model attacks require knowledge of the target model architecture, which is untrue — the LiRA paper evaluates with different architectures.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed approach only requires the training time of a single shadow model.\n\nThe proposed attack often reaches very high precision at low false positive rates. Precision is often competitive with attacks that require many shadow models.\n\nThe evaluation considers several datasets, including both image and tabular data.\n\nThe paper is easy to read.\n", "weaknesses": "Some magic happens in the experiment section in the paragraph starting at line 290 (page 7). Here, the authors appear to introduce a different objective for their attack model, to directly learn a sample’s mean and deviation. I would like to see some ablation of the choices made in this paragraph.\n\nThe proposed attack has a higher “online” computation cost than the offline variant of LiRA. LiRA can train shadow models independently of the target model, while the quantile attack requires the target model to train its attack model.\n\nFrom plots, it looks like the attack performs poorly at higher false positive rates. In settings where attack accuracy or attacker advantage is preferred to TPR and FPR, this attack might end up worse than existing attacks.\n\nResults are only reported for a single target model, rather than averaging over multiple target models.\n\nThe attack model requires hyperparameter tuning, which can be amortized over multiple shadow models when running LiRA.\n\nSmall comment: The definition of precision should have TPR in the numerator", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful reading and useful feedback --- below we address your specific questions. We're happy to further discuss any of these points if there are any remaining questions or confusions!\n\n> *Some magic happens in the experiment section in the paragraph starting at line 290 (page 7). Here, the authors appear to introduce a different objective for their attack model, to directly learn a sample’s mean and deviation. I would like to see some ablation of the choices made in this paragraph.*\n\nApologies for any confusion! The general method is the same: we solve a quantile regression problem with the goal of minimizing pinball loss. We find that for smaller datasets, and simpler tasks, doing parametric quantile regression (by fitting a parametric model, rather than non-parametrically minimizing pinball loss), leads to lower pinball loss than our direct minimization used for larger datasets. This may have to do with generalization issues in these more data scarce regimes. Thanks for flagging the confusion --- we will elaborate in the next revision.\n\n> *The proposed attack has a higher “online” computation cost than the offline variant of LiRA...*\n\nIt is true that to train our quantile regression model, we first need to evaluate the model we are going to attack on the points in a validation set, whereas the offline variant of LiRA does not need this evaluation step before training its shadow models. We will mention this in the next revision --- but note that this is a low-order cost compared to training LiRA's shadow models (and our own quantile regression training): Each evaluation of the model under attack requires a single forward pass. On the other hand, training a model of the same architecture requires multiple forward and backward passes per data point. \n\n> *From plots, it looks like the attack performs poorly at higher false positive rates. In settings where attack accuracy or attacker advantage is preferred to TPR and FPR, this attack might end up worse than existing attacks.*\n    \nWe follow the same principle in Carlini 2022 by focusing on TPR at low FPR and agree with their point that attacker advantage is a poor indicator of performance at low FPR. For reference, see the accompanying pdf with results on the CINIC10 dataset which includes attacker advantage; all tested methods perform similarly on this metric.\n\n> *Results are only reported for a single target model, rather than averaging over multiple target models.*\n\nAgreed that it would be useful to show this method works well over the randomness of SGD, if that is what you mean! We're happy to show averages/coverage intervals of this approach over that randomness in future drafts.\n\n> *Is the pinball loss used at all when attacking the CIFAR models? If it is, could you try to explain in more detail how it is used?*\n\nWe found that using empirical pinball loss in hyperparameter tuning for CIFAR attacks was effective.  As mentioned above, we achieve lower test pinball loss for  small data/model regimes by using another objective function. Why this takes place is something we definitely hope to explore further.\n\nIn all of our experiments, our goal is to find a quantile regression model that minimizes pinball loss. On ImageNet, we found that the most effective way to do this was to directly minimize pinball loss \"non-parametrically\" -- i.e. without trying to fit a parametric probability distribution to the scores. To do this, we create multiple outputs for a single quantile regression model, each with its own target quantile $\\alpha$ (e.g. logarithmically spaced $\\alpha$ values) and the entire network is trained to minimize pinball loss simultaneously across all outputs.\n\nFor the smaller datasets like CIFAR, CINIC10, and OpenML tabular datasets, we are able to get lower test pinball loss by instead fitting a parametric Gaussian model to the score distributions; \nthe model computes negative log-likelihood (NLL) of the score under the Gaussian distribution defined by the predicted mean and variance pair given a sample, and the objective function of the minimization problem is the averaged NLL over samples. After learning, based on the mean and the variance produced by the model given a sample under attack, quantiles with specific $\\alpha$ values are computed from the fit Gaussian distribution.\n\nWhy parametric quantile regression methods outperform direct pinball loss minimization (as measured by test pinball loss) on smaller datasets is something we hope to explore further.\n\n> *Do you have thoughts on why the attacks start to perform worse than other attacks at higher FPR?*\n\nWe don't know; non-parametric quantile regression via pinball loss minimization appears to work less well (in our setting at least) for lower target quantiles. But we believe that the low false positive rate regime is the most interesting/important from the point of view of an attacker.\n\n\n> *The paper does not discuss limitations. It has minor overclaiming in some places, such as when saying the attack requires only a single model, when all results are presented after hyperparameter tuning. Another place is in claiming existing shadow model attacks require knowledge of the target model architecture, which is untrue — the LiRA paper evaluates with different architectures.*\n\nThanks for the comments: we'll look out for instances of overclaiming and try and dial them back. We can clarify that our attack only requires a single regression model trained on a holdout $S$, though performance of the attack can be optimized using hyperparameter tuning which involves training several models and using the best one.\n\nWe also highlight that difference in architectures is a sensitive parameter for shadow model attacks, as evidenced in the LiRA paper on Fig 11. Our approach doesn’t require knowledge of the architecture, since the quantile regression is essentially a different task than the original classification task.\n"}, {"review_id": "2xmGfoHYAT", "reviewer": "Reviewer_648h", "summary": "This work presents a novel membership inference attack that offers computational efficiency compared to state-of-the-art (SOTA) approaches. While the paper addresses a topical issue and provides an alternative attack method, there are several weaknesses that need to be addressed for a more comprehensive and convincing study, including the lack of evaluation of TPR@lowFPR, concerning empirical results, insufficient support for computational efficiency claims, and a lack of empirical evidence for the theoretical results. Addressing these weaknesses is essential to enhance the paper's credibility and provide a more comprehensive evaluation of the proposed attack method. Finally, I would be happy to increase my score if the authors could alleviate some of my concerns.", "questions": "Could the authors please elaborate on why the read-off attack results from [2] that use 64 shadow models is performing worse than the authors' implemented attacks with 4 and 8 shadow models? There must be some discrepancies in the underlying models that have been trained!?", "limitations": "This point should be ok.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "**Topical and relevant**:\nThe paper addresses a relevant topic in the field of membership inference attacks, which is of interest to conference readers. The proposed attack method introduces a novel approach that offers computational efficiency compared to existing SOTA methods.\n\n**Competitive performance without training multiple shadow models**:\nThe authors suggest an attack method that competes with state-of-the-art approaches while being computationally more efficient. By eliminating the need to train several shadow models, their method offers a practical advantage, which is a significant contribution to the field.", "weaknesses": "Regarding the theoretical results:\n\n**Lack of TPR@lowFPR evaluation**: The theoretical results presented in the paper focus on controlling the false positive rate (FPR). However, what truly matters in membership inference attacks (see [1]) is the true positive rate (TPR) at low FPR. The paper lacks analysis of TPR@lowFPR, which is crucial for assessing the attack's effectiveness. (This recent paper [2] could help the authors with this kind of analysis).\n\nRegarding the empirical results:\n\n**Concerning empirical results** \n\n- The quantile attack outperforms the Likelihood Ratio Test-based Inference Attack (LIRA) in terms of TPR@lowFPR on ImageNet. This is surprising, as the Neyman-Pearson Lemma suggests that the LRT is the most powerful test. The discrepancy may be due to the limited computational resources used in training shadow models or differences in training setups when training the underlying imagenet classifier as batch size, # epochs, etc impact the succeptibility to membership inference attacks (e.g., see [1]).\n\n- The replicated LIRA attacks with 8 and 4 shadow models outperform the read-off LIRA attack with 64 shadow models [2]. This observation questions the authors' conclusion that their attack is more favorable. It suggests that if the authors had trained 64 shadow models on ImageNet, the LIRA attack would perform better, as indicated by the evaluation on tabular datasets.\n\n**Computational intensity claims**: The authors argue that their method is computationally more efficient by eliminating the need for training shadow models. However, there is no empirical evidence supporting this claim. It is crucial to provide comparative plots showing the tradeoff between computational cost and TPR@fixed FPR to substantiate this argument.\n\n**Lack of empirical support for theoretical results**: The theoretical results regarding FPR control lack empirical evaluation. Conducting simulation studies or additional experiments would provide empirical evidence to support these claims.\n\n----- \n**References**\n\n[1] Membership Inference Attacks From First Principles, https://arxiv.org/abs/2112.03570\n\n[2] Gaussian Membership Inference Privacy, https://arxiv.org/abs/2306.07273\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful reading and useful feedback --- below we address your specific questions. We're happy to further discuss any of these points if there are any remaining questions or confusions!\n\n\n> *The theoretical results presented in the paper focus on controlling the false positive rate (FPR). However, what truly matters in membership inference attacks (see [1]) is the true positive rate (TPR) at low FPR. The paper lacks analysis of TPR@lowFPR, which is crucial for assessing the attack's effectiveness. (This recent paper [2] could help the authors with this kind of analysis).*\n\nOur theory concerns the false positive rate of our attacks, which is something that can be established in a model-agnostic way (i.e. without making any assumptions about the model that we are attacking). It is not possible to prove similar theorems about the attack having high true positive rates at low false positive rates, for the simple reason that not all models are vulnerable to membership inference attacks. For example, models that are trained in a differentially private way for small values of $\\epsilon$ have the provable property that no membership inference attack with low false positive rate can have a high true positive rate. Thus, as with much work on hypothesis testing and membership inference, we prove one-sided theoretical guarantees, and then evaluate the tradeoff between false positive rates and true positive rates empirically. We note that our empirical analyses are indeed focused on the low false positive rate regime: e.g. the false positive rate axis in figure 1 is on a log scale, and goes as low as $10^{-4}$, and tables in our submission present precision values at low false positive rates.\n    \n> *The quantile attack outperforms the Likelihood Ratio Test-based Inference Attack (LIRA) in terms of TPR@lowFPR on ImageNet....*\n\nNeyman-Pearson implies that thresholding likelihood ratios gives an optimal test _under the assumption that the likelihood ratios are known exactly_. However, in the context of membership inference attacks, these likelihood ratios are never known, as they arise from a complex empirical process. LiRA uses a simple parametric family of models to attempt to approximate the likelihoods from samples, but this approach comes with no guarantees, as in general the true likelihood ratios need not be well approximated by anything in the parametric class.\n\nUnder the assumption that the likelihood ratios are monotonic in the test statistic (which is true for the parametric family of distributions used by LiRA), a likelihood ratio test also corresponds to a thresholding of the test statistic (which is what our attack does). Thus both our attack and LiRA can be viewed as consistent with the structure of an optimal hypothesis test by Neyman Pearson: we differ in what properties about the test statistic distribution we choose to estimate, and whether we perform those estimates parametrically or non parametrically. We can elaborate on this in the revision.\n\n\n> *The replicated LIRA attacks with 8 and 4 shadow models outperform the read-off LIRA attack with 64 shadow models [2]. This observation questions the authors' conclusion that their attack is more favorable. It suggests that if the authors had trained 64 shadow models on ImageNet, the LIRA attack would perform better, as indicated by the evaluation on tabular datasets.*\n\nThe description in LiRA in the original paper is not completely specified, so we did our best to implement the described method; our implementation outperformed their stated results. Unfortunately, we didn't have the computational power to replicate their 64-model results (which is part of the motivation for our work, which is about reducing computational costs), but did provide their stated results for comparison. We expect that were we able to run it, our implementation of LIRA with 64 models would out-perform our implementation with 8 models, but it is not clear doing so would outperform our attack since there are diminishing marginal returns to using more models. \n\n> *Computational intensity claims: The authors argue that their method is computationally more efficient by eliminating the need for training shadow models. However, there is no empirical evidence supporting this claim. It is crucial to provide comparative plots showing the tradeoff between computational cost and TPR@fixed FPR to substantiate this argument.*\n\nWe will add final clock times to the paper in our next revision. For reference, the base model ImageNet-1k took 18 hours to train using our setup, while the quantile MIA attack took 40 hours on the same server including hyper-parameter tuning; roughly the cost of $2.2$ shadow models; each individual quantile regression trial took 75 minutes. On our recently added CINIC10 results, it took 85 minutes to train the base network (and each subsequent shadow model) and 4 hours 40 minutes for hyper-parameter tuning; roughly the cost of $3.3$ shadow models with a performance comparable to 4 shadow models. The time cost for a single quantile regression trial was 18 minutes On our tabular examples, we perform hyper-parameter tuning on both the base and target model and thus our approach has the same compute cost as a single shadow model. \n\nIt is worth noting that the performance of LiRA is rather sensitive to the architecture choice and the data augmentation strategies, which all are considered as hyperparameters. In the scenarios where only API access to the model is available to the attacker, then our algorithm is favorable since ours doesn't rely on any information regarding the architecture, the training process, or the augmentation strategies of the underlying model, whilst LiRA would suffer.\n"}, {"review_id": "wjMm9nBf3O", "reviewer": "Reviewer_kgzg", "summary": "This paper introduces a new class of membership inference attacks based on performing quantile regression on the distribution of confidence scores induced by the model under attack on points that are not used in training. The approach is computationally efficient and does not require knowledge of the model's architecture, making it truly \"black-box\". The paper discusses the efficacy of this approach through extensive experiments on various datasets and model architectures. The experiments show that this approach is competitive with (and sometimes more effective than) much more computationally expensive shadow model approaches. Overall, the paper presents a more scalable and efficient membership inference attack.", "questions": "- I appreciate it if the authors could also compare the proposed method to the model-based attacks (https://arxiv.org/abs/1610.05820) and metric-based attacks. (https://arxiv.org/abs/2003.10595)\n\n- Why the proposed method performs worse on C-10 and C-100 but better on IN-1k? I would suggest the authors elaborate more on it.", "limitations": "- Besides accessing the performance, it is also important to report the time/computational cost of different methods to further demonstrate the efficiency of the proposed method.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- interesting topic\n- novel method\n- well-written paper\n", "weaknesses": "- more evaluation metrics needed\n- compare with different attacks", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful reading and useful feedback --- below we address your specific questions. We're happy to further discuss any of these points if there are any remaining questions or confusions!\n\n> *I appreciate it if the authors could also compare the proposed method to the model-based attacks (https://arxiv.org/abs/1610.05820) and metric-based attacks. (https://arxiv.org/abs/2003.10595)*\n\nThese are methods that pre-date LIRA, and focus on global metrics of success (rather than performance of the attacks at low FPRs). We choose LiRA as our main point of comparison as it is the state-of-the-art for membership inference when looking at performance on the presented datasets at low FPR. The original LiRA paper directly compares LiRA to the approaches you reference and shows that LiRA outperforms them, which is our basis for choosing LiRA as a state of the art method. Both methods are appropriately cited in our related work section.\n    \n\n\n> *Why the proposed method performs worse on C-10 and C-100 but better on IN-1k? I would suggest the authors elaborate more on it.*\n\nWe discuss this a bit in the paper, but are happy to elaborate further in the next revision. We find that uniformly across all of our experiments, the attack that obtains the lowest pinball loss has the best performance. In the smaller data regimes like C-10, we find that we can extract a quantile regression model from the LiRA shadow models that has lower pinball loss than the regression model we can obtain by directly optimizing for pinball loss (and correspondingly, out-performs). Direct optimization of pinball loss appears to work better in the large data regime, which is also where the computational benefits of our approach are most pronounced.\n\n\n> *Besides accessing the performance, it is also important to report the time/computational cost of different methods to further demonstrate the efficiency of the proposed method.*\n    \nWe will add final clock times to the paper in our next revision. For reference, the base model ImageNet-1k took 18 hours to train using our setup, while the quantile MIA attack took 40 hours on the same server including hyper-parameter tuning; roughly the cost of $2.2$ shadow models. On our recently added CINIC10 results, it took 85 minutes to train the base network (and each subsequent shadow model and 4 hours 40 minutes for hyper-parameter tuning; roughly the cost of $3.3$ shadow models with a performance comparable to 4 shadow models. The time cost for a single quantile regression trial was 18 minutes. On our tabular examples, we perform hyper-parameter tuning on both the base and target model and thus our approach has the same compute cost as a single shadow model. "}]

Author rebuttal:
We thank the reviewers for their careful reading and useful feedback. In the attached pdf we include an additional experiment over the CINIC10 dataset to address some of the reviewers' specific concerns. We're happy to further discuss any of these points if there are any remaining questions or confusions!


Meta-review: There is a consensus amongst all reviewers that this work achieves SOTA results, and has good novelties: the paper has competitive results while reducing shadow model count to a single one (reviewers NGxz, 648h, kgzg, 5DZs, bLLW).
Reviewer bLLW refers to similarity wrt. quantile regression and states that this "does not yield high novelty". Adequate references has been given by authors in the paper regarding this. Other reviewers find the paper novel enough for the MIA community.
Reviewer NGxz does not see the direct application, which is not a question particular to this work, but related to MIA in general. MIA is well established field and has applications even extending to legal domain.

Therefore the paper is recommended for acceptance.
Decision: Accept (poster)