Title: Scalable Membership Inference Attacks via Quantile Regression
Venue: NeurIPS 2023
Accepted: yes
Keywords: machine learning

privacy

membership inference
Abstract: Membership inference attacks are designed to determine, using black box access to trained models, whether a particular example was used in training or not. Membership inference can be formalized as a hypothesis testing problem. The most effective existing attacks estimate the distribution of some test statistic (usually the model's confidence on the true label) on points that were (and were not) used in training by training many \emph{shadow models}---i.e. models of the same architecture as the model being attacked, trained on a random subsample of data. While effective, these attacks are extremely computationally expensive, especially when the model under attack is large. \footnotetext[0]{
Martin and Shuai are the lead authors, and other authors are ordered alphabetically. \{maberlop,shuat\}@amazon.com}

We introduce a new class of attacks based on performing quantile regression on the distribution of confidence scores induced by the model under attack on points that are not used in training. We show that our method is competitive with state-of-the-art shadow model attacks, while requiring substantially less compute because our attack requires training only a single model. Moreover, unlike shadow model attacks, our proposed attack does not require any knowledge of the architecture of the model under attack and is therefore truly ``black-box". We show the efficacy of this approach in an extensive series of experiments on various datasets and model architectures. Our code is available at \href{https://github.com/amazon-science/quantile-mia}{github.com/amazon-science/quantile-mia.}

Scalable Membership Inference Attacks via Quantile Regression

Abstract

Membership inference attacks are designed to determine, using black box access to trained models, whether a particular example was used in training or not. Membership inference can be formalized as a hypothesis testing problem. The most effective existing attacks estimate the distribution of some test statistic (usually the model's confidence on the true label) on points that were (and were not) used in training by training many shadow models -i.e. models of the same architecture as the model being attacked, trained on a random subsample of data. While effective, these attacks are extremely computationally expensive, especially when the model under attack is large.

We introduce a new class of attacks based on performing quantile regression on the distribution of confidence scores induced by the model under attack on points that are not used in training. We show that our method is competitive with state-ofthe-art shadow model attacks, while requiring substantially less compute because our attack requires training only a single model. Moreover, unlike shadow model attacks, our proposed attack does not require any knowledge of the architecture of the model under attack and is therefore truly 'black-box". We show the efficacy of this approach in an extensive series of experiments on various datasets and model architectures. Our code is available at github.com/amazon-science/quantile-mia.

1 Introduction

The basic goal of privacy-preserving machine learning is to find models that are predictive on some underlying data distribution, without being disclosive of the particular data points on which they were trained. The simplest kind of attack that can be launched on a trained model-falsifying privacy guarantees-is a membership inference attack. A membership inference attack, informally, is a statistical test that is able to reliably determine whether a particular data point was included in the training set used to train the model or not.

Almost all membership inference attacks are based on the observation that models tend to overfit their training sets in different ways. In particular, they tend to systematically predict higher confidence in the true labels of data points from their training set, compared to points drawn from the same distribution not in their training set. The confidence that a model places on the true label of a data-point is thus a natural test statistic to build a membership-inference hypothesis test around. A

variety of recent methods [Shokri et al., 2017, Long et al., 2020, Sablayrolles et al., 2019, Song and Mittal, 2021, Carlini et al., 2022] are based around this idea, and aim to estimate the distribution of the test statistic (the confidence assigned to the true label of a datapoint) over the distribution of datapoints that were not used in training (and sometimes, also over the distribution of datapoints that were used in training) for the purpose of designing tests that can reject the null hypothesis-that a data point under attack was not used in training-with the desired level of confidence.

The efficacy of this class of attacks depends in large part on the granularity to which the distribution of the test statistic can be estimated. The simplest (and most computationally efficient) approach, originally proposed by Yeom et al. [2018], is to estimate this distribution marginally -i.e. without conditioning on the covariates x of the example being attacked. This reduces the problem to a simple one-dimensional estimation problem, and-under mild assumptions-the optimal hypothesis test (by the Neyman-Pearson Lemma) is simply a fixed threshold τ on the test statistic-examples are declared to have been used in training if the confidence the model places on their true label exceeds τ , and are declared to have been not used in training otherwise. More sophisticated methods attempt to estimate the distribution of the test statistic conditional on the inclusion of a target point x in the training data (over the randomness of the selection of the other points used in training). Our method and others follow this approach, where the confidence score produced by each example x by the target model is compared to a sampledependent threshold τ ( x ) -points x with scores exceeding this threshold are declared to be used in training. The most common method under this approach is to train shadow models [Shokri et al., 2017, Long et al., 2020, Sablayrolles et al.,

100

10-1

itive

e

Ours

Posi

10-2

LIRA (n=2)

LIRA (n=4)

True

LIRA (n=6)

10-3

LIRA (n=8)

LIRA (n=64 Visual)

Marginal Baseline

10-4

10-4

10-3

10-2

10-1

100

FalsePositiveRate

2019, Song and Mittal, 2021, Carlini et al., 2022]. Informally, shadow models are trained with the same architecture as the model being attacked, using random subsets of training data, that either include or do not include the target point x . As a result, each shadow model gives a sample of the test statistic conditional on x 's inclusion (or non-inclusion) in the training set, where the randomness is over the other examples in the training set and any randomness involved in training. Because many samples from this distribution on the test statistic are needed to estimate it, membership inference attacks based on shadow models generally require training many shadow models of the same architecture as the model under attack; between 64 and 256 shadow models were used in Carlini et al. [2022] (Figure 1 compares the receiver operating characteristic (ROC) of the attack on ImageNet against our proposed approach). Especially for large models, this makes shadow model attacks prohibitive, for at least two reasons:

Training Cost : Widely used commercial models, on which membership inference attacks would be most damaging, are extremely large and expensive to train. An attacker launching a membership inference attack based on shadow models must train many (dozens to hundreds) models of the same architecture. Thus the computational costs can be hundreds of times larger than the costs of training the model under attack, which for commercial models is prohibitive for attackers without enormous resources.

Knowledge of the Model Under Attack: As argued in Carlini et al. [2022], and also shown in Appendix A, when the shadow model is of the same complexity as or more complex than the target model, the LiRA attack performs well, and when less complex shadow models are deployed, the success rate of the attack drops precipitously. Hence the success of the attack depends on knowledge of the model under attack. But many aspects of the architecture and

training process for large commercial models are not publicly known, making this style of attack less effective in realistic settings.

1.1 Our Results

We introduce a new class of membership inference attacks, based on quantile regression, that is able to mitigate these issues:

Like attacks based on shadow models, it is a conditional attack, subjecting different examples x to different thresholds τ ( x ) . However,

It only requires training a single model, and

The architecture of the model used in the attack need not be related to the architecture of the model under attack, and so no knowledge of the model architecture or training algorithm used to train the model under attack is needed.

Our quantile regression attack . Given a model f that we intend to attack, we collect a dataset of labelled examples { ( x i , y i ) } n i =1 from the underlying data distribution, known to not have been used in training. 1 For each example ( x i , y i ) in our dataset, we evaluate the model f on example x i , and record the (real-valued) confidence score s ( x i , y i ) that the model places on the correct label y . We then train a quantile regression model q on the dataset { ( x i , s ( x i , y i )) } consisting of examples x labeled with their confidence scores s ( x, y ) . Informally, the quantile regression model is trained to predict q ( x ) , a target 1 -α quantile of the conditional distribution on s ( x i , y i ) , given x i . 2 Intuitively, a score s ( x i , y i ) larger than the 1 -α quantile q ( x i ) indicates that f assigns a confidence on the true label that is higher than a 1 -α fraction of the examples not used in training - giving us evidence that the example in question was part of the training set. Thus, given a new target point ( x, y ) , we reject the null hypothesis that x was not used in training (i.e. we declare x to have been used in training) whenever s ( x, y ) exceeds q ( x ) -i.e. when s ( x, y ) ≥ q ( x ) . Similarly, whenever s ( x, y ) < q ( x ) , we do not reject the null hypothesis, and declare the point ( x, y ) to have not been used in training.

This attack by design has a false positive rate of α -the probability that it incorrectly declares a randomly selected point ( x, y ) that was not used in training to have been used in training is α . The ability of q to correctly label those examples used in training as such, measured by its true positive rate or precision or related statistics, will vary with α (the higher α , the larger the number of positive labels our test will assign). So, in varying our target α , we can sweep out our test's tradeoff between false positive and true positive rates.

The primary strength of our attack is that we need only a single quantile regression model q , rather than a large number of shadow models. Furthermore, because the success of our attack depends only on how well q predicts the quantiles of the confidence score distribution of f (rather than producing confidence scores drawn from the same distribution as f ), q need not have any relationship to the architecture of f or any knowledge of it- the only access to f that is needed is the ability to evaluate confidence scores s ( x, y ) produced by f given examples x . Our attack is, therefore, more 'black-box" than those which use shadow models of the same architecture as f .

We derive a basic theory for our approach based on quantile regression, which trains a model to predict quantiles by minimizing pinball loss . We run an extensive series of experiments and find that our quantile regression approach is competitive with (and sometimes more effective than) much more computationally expensive shadow model approaches. The relative effectiveness of our approach appears to grow the more complex the classification task and model under attack are. For example, when attacking a ResNet-50 model trained on ImageNet-1k, our attack (which trains only a single model) outperforms shadow model approaches trained much more expensively at every false positive rate. On simpler and less data rich tasks (like CIFAR-10), the accuracy of our approach dominates the marginal baseline of Yeom et al. [2018], but falls short of shadow model approaches. Thought provokingly, however, we find that when this occurs, it is because the shadow model approach has

found thresholds that correspond to a quantile model with lower pinball loss than our trained quantile regression model. This suggests that our fundamental approach of pinball loss minimization is sound, and that our attempts to directly optimize for it are less successful when data is less plentiful. Across all experiments, we find that the best quantile regression method (as measured by pinball loss) is uniformly the best membership inference attack.

1.2 Additional Related Work

Starting with the seminal work of Homer et al. [2008], membership inference has become one of the most widely studied classes of privacy attacks. Most approaches for membership inference determine whether an example is part of the training set via some score function, which can be loss [Yeom et al., 2018, Sablayrolles et al., 2019], confidence [Salem et al., 2018], entropy [Song and Mittal, 2021], or difficulty calibration [Watson et al., 2021] among others. Another common approach is to query the model on similar or related examples to the target point [Wen et al., 2023, Jayaraman et al., 2020, Long et al., 2018, Li and Zhang, 2020].

Wefocus the remainder of our discussion on related work on shadow model approaches to membership inference since they are our main benchmarks. Most work on shadow models considers a setup where there is a private dataset D private (unknown to the attacker) drawn from a distribution Q , and an algorithm A for training a model f = A ( D private ) . The attacker has access to a set of data D public drawn from the same distribution Q and partial query access to the model f from which the attacker can compute scores s ( x, y ) given target examples ( x, y ) . The attacker aims to predict, for a data point ( x, y ) , whether ( x, y ) ∈ D private .

Likelihood Ratio Attacks. Membership inference attacks are fundamentally hypotheses tests between two competing hypotheses ( H 0 : ( x, y ) ̸∈ D private , H 1 : ( x, y ) ∈ D private ). By the NeymanPearson lemma [Neyman and Pearson, 1933], the optimal hypothesis test based on a test statistic s ( x, y ) computes the likelihood ratio of the score under the null and alternative hypothesis, and subjects the likelihood ratio to a threshold τ . The choice of the threshold τ determines the trade-off between precision (the fraction of examples labeled as belonging to the private dataset which did belong to the dataset) and recall (or true positive rate) of the resulting classifier. We call a membership inference attack carried out with this classifier a likelihood ratio attack (LiRA), introduced by Carlini et al. [2022]. LiRA was designed to achieve very high precision (very few false positives relative to the number of positive predictions), as they noted high precision corresponds to a high degree of confidence that the data points accused of being part of the training set were, in fact, part of the training set. Prior work had looked at global notions of inference attack quality, at possibly much lower degrees of precision [Ye et al., 2021, Jayaraman et al., 2021].

The main difficulty with implementing LiRA directly is that the density functions of the score under the null and alternative hypothesis are unknown. Instead, the literature aims to estimate these density functions, primarily by training a collection of shadow models [Shokri et al., 2017, Long et al., 2020, Sablayrolles et al., 2019, Song and Mittal, 2021, Carlini et al., 2022]. Shadow model attacks split the attacker's dataset D public into several pairs of shadow public/private datasets D private i , D public i , and for each of these shadow datasets, a shadow model f i is trained on D private i . The shadow model f i , and corresponding datasets D private i , D public i are used to generate private and public score samples s private i , s public i from which to estimate the likelihood ratio function given parametric assumptions. Carlini et al. [2022] used a large number of shadow models to achieve high precision. This approach works well-but it is computationally demanding because it requires training many shadow models.

2 Preliminaries

We study attacks on models f that solve a supervised learning problem defined over a distribution D ∈ ∆( X × Y ) of labeled examples ( x, y ) , consisting of feature vectors x ∈ X and labels y ∈ Y . We make no assumptions about X or Y (e.g. Y could be a discrete set in a multi-class classification problem, or we could have Y = R in a regression problem). We assume that the model f outputs a confidence score in [0 , 1] for each possible label ˆ y ∈ Y : in other words, f : X → [0 , 1] Y , and for each ˆ y ∈ Y , we write f ˆ y ( x ) ∈ [0 , 1] for the confidence score that f assigns to label ˆ y given input x . Such models are often used to make point predictions by predicting the label ˆ y = arg max y ∈Y f y ( x ) on input x -but we will interact with such models f only at the level of confidence score predictions.

A model f is derived from a training process that did not have direct access to D , but rather to a finite sample D private called the training set. The training process correlates f with D private . A membership inference attack is a hypothesis test that must use a test statistic derived only from f that aims to determine whether a labeled example ( x, y ) is a member of the training set D private or not. Formally, we model this as a hypothesis test that aims to solve the following simple hypothesis testing problem:

Here, ( x, y ) ∼ D private denotes sampling a point ( x, y ) uniformly at random from D private . Observe that since we derive the test statistic from f , even if D private was itself sampled i.i.d. from D , H 0 and H 1 are distinct hypotheses since the training process has correlated f with D private . In particular, we will base our attack on the presumption that f will tend to be over-confident on examples ( x, y ) ∈ D private . Towards this, we choose as our test statistic s ( x, y ) = z y ( x ) -max y ′ = y z y ′ ( x ) the logit difference between the true label and its most likely alternative, where z ( x ) denotes the logits (unnormalized features before the softmax nonlinearity) of the model. This choice follows the scoring rule used in Carlini et al. [2022] and will be useful for experimental comparisons. However, the remainder of our theoretical treatment will be agnostic as to this choice.

̸

In this paper we restrict attention to membership inference attacks (hypothesis tests) that apply a threshold function to s ( x, y ) , with a threshold that may depend on x . Given a function q : X → R that maps examples x to thresholds, the corresponding membership inference attack is given by:

Here ⊥ is shorthand for 'We reject the null hypothesis that ( x, y ) ∼ D (and thus declare ( x, y ) of being in the training set)', and ⊤ is shorthand for 'we do not reject the null hypothesis (and thus do not accuse ( x, y ) of being in the training set)'.

A natural baseline is to set q ( x, y ) = τ to be a constant. This is the attack proposed by Yeom et al. [2018], and we write this baseline as A τ . If τ is set to be a 1 -α quantile of the marginal distribution on s ( x, y ) when ( x, y ) ∼ D , then this attack can easily be seen to have false positive rate α . Below we define quantiles assuming (for simplicity) that the distribution in question is continuous-but it is also possible to define quantiles without this assumption.

Definition 1. Fix a continuous distribution P ∈ ∆ R . A number τ ∈ R is a (1 -α ) -quantile of P if:

We can evaluate the performance of a membership inference attack by evaluating its false positive rate, true positive rates, and precision 3 :

Definition 2. Fix an arbitrary membership inference attack A : X × Y → {⊤ , ⊥} . We define the following performance metrics

It is immediate that the baseline membership inference attack achieves its target false positive rate; the true positive rate and precision of the attack can be evaluated empirically:

Lemma 1. Let τ be a 1 -α quantile of P , the distribution on confidence scores s ( x, y ) that results from sampling ( x, y ) ∼ D . Then the baseline membership inference attack A τ has FPR ( A τ ) = α .

3 Our Attack

Our attack is A q ( x, y ) , where q is derived from a quantile regression model trained to predict quantiles of our test statistic s ( x, y ) on a dataset of points ( x, y ) drawn from our null hypothesis distribution ( x, y ) ∼ D . A popular non-parametric quantile regression method is to minimize pinball loss , which elicits quantiles (just as squared loss elicits means):

Definition 3. The pinball loss function defined for a 1 -α quantile is:

Pinball loss is a useful objective function because it elicits quantiles:

Lemma 2. Fix any distribution P ∈ ∆ R . Let:

Then τ is a (1 -α ) -quantile of P .

Viewed through this lens, the baseline attack can be thought of as the end result of the following simple pipeline:

Select a target false positive rate α ,

Choose a threshold τ by solving the minimization problem

Instantiate the baseline membership inference attack A τ .

Our attack departs from this baseline attack by training a model q : X → R on feature/confidence score pairs to optimize pinball loss, rather than a single threshold τ :

Select a target false positive rate α and a class of model architectures H consisting of models q : X → R .

Train a model q ∈ H by solving the following risk minimization problem:

Instantiate the membership inference attack A q

We train our quantile regression model on a dataset consisting of points ( x, y ) drawn from the underlying distribution (of points not used in training), labeled by the confidence scores s ( x, y ) derived from the model. Thus our attack assumes only that we have API access to the model under attack f , and are able to query it on a finite set of points to obtain confidence scores.

We now establish some basic properties of our attack. The first is that, like the baseline attack, it actually achieves its target false positive rate. Unlike the baseline attack, this is no longer immediate, but can be derived from properties of the pinball loss:

Theorem 1. Fix a distribution D ∈ ∆( X × Y ) over labeled examples and a model f . Suppose that the marginal distribution over s ( x, y ) for ( x, y ) ∼ D is continuous. Let H be any class of models that is closed under additive shifts - i.e. such that for each q ∈ H and ∆ ∈ R , then we also have q ′ ∈ H for q ′ ( x ) = q ( x ) + ∆ . Then for the membership inference attack A q produced by our method, FPR ( A q ) = α .

We defer the proof to Appendix C.1. Thus by varying α , we can use our attack to sweep out a curve trading off our target false positive rate with our (empirically measured) true positive rate, just as we can for the baseline attack A τ .

Is this guarantee stronger than the baseline attack, and if so, in what sense? To give one perspective on this, it will be helpful to define group conditional quantile consistency, which is related to multicalibration, a concept originating from the fairness in machine learning literature [HébertJohnson et al., 2018, Gupta et al., 2022, Bastani et al., 2022, Jung et al., 2023, Noarov and Roth, 2023].

Definition 4. Fix a collection G of group indicator functions g : X → { 0 , 1 } and a model q : X → R . q satisfies group conditional quantile consistency with respect to a distribution P ∈ ∆( X × R ) , a target quantile 1 -α , and the collection of groups G if for every g ∈ G :

Group conditional quantile consistency asks that our quantile predictions be correct not just marginally over the data, but also (simultaneously) conditionally on membership in a large number of potentially intersecting groups. If we optimize pinball loss over a richer set of models (that are closed under shifts by a class of group indicator functions, rather than just constant functions), then our attack will achieve its target false positive rate even when conditioning on membership in each of the groups specified by the functions g ∈ G , rather than just marginally. This is a stronger guarantee, as a marginal false positive rate on its own need not hold subject to additional conditioning events.

Theorem 2. Fix a distribution D ∈ ∆( X ×Y ) over labeled examples and a model f . Fix a collection of group indicator functions G and a class of models H such that:

H is closed under shifts from G : for every h ∈ H , g ∈ G , and η ∈ R , the function h ′ ( x ) = h ( x ) + ηg ( x ) is such that h ′ ∈ H .

The conditional distribution over s ( x, y ) for ( x, y ) ∼ D , conditional on g ( x ) = 1 is continuous for all g ∈ G .

Then for the membership inference attack A q produced by our method, its false positive rate is 1 -α conditional on membership in each group g ∈ G : Pr ( x,y ) ∼D [ A q ( x, y ) = ⊥| g ( x ) = 1] = 1 -α .

4 Experiments

We present two sets of experiments on two different data domains, including images and tabular data. Here, we mainly focus on attacking widely-used classification models in these two domains, however, our theoretical claims generalizes to other data domains as well.

4.1 Image Classification Experiments

We evaluate the effectiveness of our proposed approach on four image classification datasets: CIFAR10 [Krizhevsky et al., 2009], a standard image classification dataset with 10 target classes; CIFAR-100 [Krizhevsky et al., 2009] another image classification dataset with 100 target classes; ImageNet-1k [Russakovsky et al., 2015], a substantially larger image classification task with 1000 target classes; and CINIC-10 [Darlow et al., 2018], an extension of CIFAR-10 that additionally uses images from ImageNet-1k corresponding to the original 10 target classes. To provide a realistic evaluation, we ensure our base models use common, well-performing architectures and follow standard guidelines for hyperparameter selection [He et al., 2015], including data augmentation, learning rate schedule, and l2 regularization(weight decay). For CIFAR-10 and CIFAR-100, target (classification) models include ResNet-10, ResNet-18, ResNet-34, and ResNet-50 [He et al., 2015]. For ImageNet-1k and CINIC-10, the target model is a ResNet-50. In all experiments, 50% of the dataset is used for training the target model, and, following the common standards, the resolution of the target model is 32x32 for CIFAR and CINIC datasets, and 224x224 for the ImageNet-1k dataset. The accuracy of each target model is presented in Appendix B.

To perform our membership inference attack, we train a single quantile regression model following our proposal in Eq.(1). One of the advantages of our attack is that it is model-agnostic: since it does not require the knowledge of the model architecture of the target or the knowledge of the training algorithm, we use the same model architecture for our quantile regression model in all settings : a pretrained ConvNext-Tiny model [Liu et al., 2022]. On CINIC-10 we additionally experimented with a ResNet-50 model as our quantile model architecture.

̸

For the scoring function, we use the hinge score proposed in Carlini et al. [2022] s hinge ( x, y ) = z y ( x ) -max y ′ = y z y ′ ( x ) . This scoring rule is closely related to the logit function of the model's confidence s ( x, y ) = log( f y ( x )) -log(1 -f y ( x )) for models with high label confidence, and has

been empirically shown to be approximately normally distributed. On the largest dataset, ImageNet1k, we directly train a quantile regression model to predict (log-spaced) quantiles at α ∈ [0 . 9996 , 1] An alternative is to learn a parameterized model so that, for each sample, the model predicts the mean µ ( x ) and the log of the standard deviation log σ ( x ) , and the quantiles of a sample can be generated from the Gaussian distribution N ( µ ( x ) , ( e log σ ( x ) ) 2 ) . Due to the fact that CIFAR and CINIC datasets are much smaller ( 25000 samples available for training on CIFAR), rather than directly learning quantiles, we opt to learn the parametrized models with Gaussian parameters on these datasets.

All images are processed at 224x224 resolution by the quantile model, which is trained on the remaining 50% of the training samples that were not used to train the target model. Since there is a smaller body of literature on stable hyperparameters for regression models, we use Ray Tune [Liaw et al., 2018] for hyperparameter tuning (tuning is used to minimize validation pinball loss in a held out dataset). The compute budget for our attack was approximately 30 GPU minutes per quantile regression attack (4 hours including hyperparameter optimization) on CIFAR-10/CIFAR-100, 18 minutes (4 hours 40 minutes including hyperparameter optimization) on CINIC-10, and 16 hours (128 hours including hyperparameter optimization) on ImageNet-1k. Final hyperparameters were found to be consistent across all tasks sharing an architecture; more information is provided in Appendix D

Figures 1 and 2 shows ROC curves of our proposed approach on ImageNet-1k and CINIC-10 respectively; FPR is computed on a held-out dataset that was not used to train the target or the quantile regression model. Both the marginal quantile approach from Yeom et al. [2018] and the shadow model approach LIRA from Carlini et al. [2022] are also shown for reference. In these experiments, our quantile regression approach dominates the shadow model approach at all comparison points for ImageNet-1k, and is roughly comparable to 4 shadow models on CINIC-10. Appendix E shows the same comparison against all CIFAR-10/100 target models (ResNet-10, ResNet18, ResNet34, ResNet-50). In these latter experiments, our attack outperforms the marginal baseline but falls short of the shadow model approach. We note that in this case, the shadow models actually produce thresholds that have lower pinball loss than our quantile regression algorithm, suggesting that on the smaller CIFAR datasets, our optimization heuristic was unable to sufficiently minimize test pinball loss.We find this observation to be consistent across experiments, that the attack with smaller pinball loss on public data is generally the attack with the best TPR at that FPR level α

100

10-1

Positive

True

10-2

Ours (Gaussian)

Marginal Baseline

LIRA (n=2)

LIRA (n=4)

LIRA (n=6)

LIRA (n=8)

10-3

10-3

10-2

10-1

100

FalsePositiveRate

Table 1 shows precision of the proposed membership inference attack at 1% and 0.1% false positive rate on the ResNet-50 target networks (additional results shown in Appendix E). Our attack robustly predicts membership in the private dataset with high precision even at low FPR. The proposed approach works on all target architectures and datasets, but works particularly well on the more complex and data intensive ImageNet-1k task.

Since LIRA produces an explicit score distribution N ( s ( x, y ); µ ( x, y ) , σ ( x, y )) based on the shadow model's predictions, we can compare all 3 methods (Ours, LIRA, marginal baseline) in terms of pinball loss on public data ( x ∼ D ). We find that the attack with the smallest pinball loss on public data is the better membership inference attack across all datasets. This shows that a strong quantile predictor on public data is a strong membership inference attack; which validates the core premise of our approach in Section 3.

A possible explanation for the relative lack of success of directly learning a quantile regression model on CIFAR datasets could be the relatively low number of available samples (25000). In such low data scenarios, training shadow models is also computationally affordable. The opposite holds true for the much larger ImageNet1k dataset, on which the computational cost of training a single target model, much less multiple shadow models far exceeds the cost of a single quantile regression model.

4.2 Tabular Classification Experiments

In addition to experiments on image datasets, we here demonstrate the effectiveness of our membership inference attack on tabular datasets, including large datasets from derived from the US Census' American Community Survey (ACS) [Ding et al., 2021] and small ones from OpenML 4 [Grinsztajn et al., 2022]. Gradient boosting with decision trees is widely-used for classification tasks with tabular data, so in our experiments, to achieve a reasonable performance, we train a gradient boosting model with 5-fold cross validation for hyperparameter tuning on the private portion of the data. For our attack model, gradient boosting with regression trees is applied, but now with regression targets as mentioned above. Hyperparameters for regression tasks are also tuned using a public portion of the data to avoid overfitting. In our experiments, catboost is used for model training, and Optuna [Akiba et al., 2019] is used for hyperparameter tuning.

Table 2 shows that our attack, which involves learning a single regression model, performs on par with the LiRA attack, which requires learning at least 16 models on some tasks and more on other tasks. Since each model, including our regression model and a shadow model in LiRA, has the same latency in terms of hyperparameter tuning and model training, our attack requires significantly less compute (equivalent to a single shadow model), and it reduces a successful attack from training many models to only one model. Figure 3 shows ROC curves of our proposed approach on OpenML.

5 Discussion

We have introduced a new family of membership inference attacks that are competitive with the state of the art (and in our ImageNet experiments, substantially and uniformly better), while requiring

(a) OpenML Datasets

TaskID=361057

TaskID =361064

TaskID=361067

TaskID=361070

True Positive Rate

100

Method

Quantile

10

LIRA (n=16)

LIRA (n=32)

LIRA (n=64)

10~2

10~2

10-1

10°10-2

10-1

10°0-2

10~1

10°10-2

10-1

10°

False Positive Rate

False Positive Rate

False Positive Rate

False Positive Rate

Task=coverage

Task = travel

Task = mobility

Task= income

True Positive Rate

10

-1

Method

10

-2

Quantile

LIRA (n=16)

LIRA (n=32)

LIRA (n=64)

10~3

10~3

10-2

10~10-3

10~2

10-10-3

10-2

10-10-3

10~2

10~1

FalsePositiveRate

False Positive Rate

FalsePositiveRate

False Positive Rate

(b) ACS Task on NY data

substantially fewer computational resources and less knowledge of the target model. Moreover, we have identified pinball loss as a key target objective: uniformly across all of our experiments, the methods that produce thresholds minimizing pinball loss are the most effective attacks. Together, this brings membership inference closer to practicality on large commercial models. This serves to highlight a growing risk to privacy-but also provides a more efficient means to audit models by subjecting them to our attacks. We hope that our methods encourage and enable a more widespread practice of auditing models for privacy violations by subjecting them to membership inference attacks before deployment.

References

Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , 2019.

Bastani, V. Gupta, C. Jung, G. Noarov, R. Ramalingam, and A. Roth. Practical adversarial multivalid conformal prediction. In Neural Information Processing Systems (NeurIPS) , 2022.

Bergstra, D. Yamins, and D. Cox. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In International conference on machine learning , pages 115-123. PMLR, 2013.

Carlini, S. Chien, M. Nasr, S. Song, A. Terzis, and F. Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP) , pages 1897-1914. IEEE, 2022.

N. Darlow, E. J. Crowley, A. Antoniou, and A. J. Storkey. Cinic-10 is not imagenet or cifar-10. arXiv preprint arXiv:1810.03505 , 2018.

Ding, M. Hardt, J. Miller, and L. Schmidt. Retiring adult: New datasets for fair machine learning. Advances in Neural Information Processing Systems , 34, 2021.

Grinsztajn, E. Oyallon, and G. Varoquaux. Why do tree-based models still outperform deep learning on tabular data? arXiv preprint arXiv:2207.08815 , 2022.

Gupta, C. Jung, G. Noarov, M. M. Pai, and A. Roth. Online multivalid learning: Means, moments, and prediction intervals. In 13th Innovations in Theoretical Computer Science Conference (ITCS 2022) . Schloss Dagstuhl-Leibniz-Zentrum für Informatik, 2022.

He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. arxiv 2015. arXiv preprint arXiv:1512.03385 , 14, 2015.

Hébert-Johnson, M. Kim, O. Reingold, and G. Rothblum. Multicalibration: Calibration for the (computationally-identifiable) masses. In International Conference on Machine Learning , pages 1939-1948. PMLR, 2018.

Homer, S. Szelinger, M. Redman, D. Duggan, W. Tembe, J. Muehling, J. V. Pearson, D. A. Stephan, S. F. Nelson, and D. W. Craig. Resolving individuals contributing trace amounts of dna to highly complex mixtures using high-density snp genotyping microarrays. PLOS Genetics , 4(8): 1-9, 08 2008. doi: 10.1371/journal.pgen.1000167. URL https://doi.org/10.1371/journal. pgen.1000167 .

Jayaraman, L. Wang, K. Knipmeyer, Q. Gu, and D. Evans. Revisiting membership inference under realistic assumptions. arXiv preprint arXiv:2005.10881 , 2020.

Jayaraman, L. Wang, K. Knipmeyer, Q. Gu, and D. Evans. Revisiting membership inference under realistic assumptions. Proceedings on Privacy Enhancing Technologies , 2021:348-368, 04 2021.

Jung, G. Noarov, R. Ramalingam, and A. Roth. Batch multivalid conformal prediction. In International Conference on Learning Representations (ICLR) , 2023.

Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Li, K. Jamieson, A. Rostamizadeh, E. Gonina, J. Ben-Tzur, M. Hardt, B. Recht, and A. Talwalkar. A system for massively parallel hyperparameter tuning. Proceedings of Machine Learning and Systems , 2:230-246, 2020.

Li and Y. Zhang. Membership leakage in label-only exposures. Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security , 2020.

Liaw, E. Liang, R. Nishihara, P. Moritz, J. E. Gonzalez, and I. Stoica. Tune: A research platform for distributed model selection and training. arXiv preprint arXiv:1807.05118 , 2018.

Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11976-11986, 2022.

Long, V. Bindschaedler, L. Wang, D. Bu, X. Wang, H. Tang, C. A. Gunter, and K. Chen. Understanding membership inferences on well-generalized learning models. CoRR , abs/1802.04889, 2018. URL http://arxiv.org/abs/1802.04889 .

Long, L. Wang, D. Bu, V. Bindschaedler, X. Wang, H. Tang, C. A. Gunter, and K. Chen. A pragmatic approach to membership inferences on machine learning models. In 2020 IEEE European Symposium on Security and Privacy (EuroS&P) , pages 521-534. IEEE, 2020.

Neyman and E. S. Pearson. On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society A , 231:289-337, 1933.

Noarov and A. Roth. The scope of multicalibration: Characterizing multicalibration via property elicitation. International Conference on Machine Learning (ICML) , 2023.

Roth. Uncertain: Modern topics in uncertainty estimation. https://www.cis.upenn.edu/ aaroth/uncertainty-notes.pdf, 2022.

Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) , 115(3):211-252, 2015. doi: 10.1007/s11263-015-0816-y.

Sablayrolles, M. Douze, C. Schmid, Y. Ollivier, and H. Jégou. White-box vs black-box: Bayes optimal strategies for membership inference. In International Conference on Machine Learning , pages 5558-5567. PMLR, 2019.

Salem, Y. Zhang, M. Humbert, P. Berrang, M. Fritz, and M. Backes. Ml-leaks: Model and data independent membership inference attacks and defenses on machine learning models. arXiv preprint arXiv:1806.01246 , 2018.

Shokri, M. Stronati, C. Song, and V. Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP) , pages 3-18. IEEE, 2017.

Song and P. Mittal. Systematic evaluation of privacy risks of machine learning models. In USENIX Security Symposium , volume 1, page 4, 2021.

Watson, C. Guo, G. Cormode, and A. Sablayrolles. On the importance of difficulty calibration in membership inference attacks. arXiv preprint arXiv:2111.08440 , 2021.

Wen, A. Bansal, H. Kazemi, E. Borgnia, M. Goldblum, J. Geiping, and T. Goldstein. Canary in a coalmine: Better membership inference with ensembled adversarial queries. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/ forum?id=b7SBTEBFnC .

Ye, A. Maddi, S. K. Murakonda, and R. Shokri. Enhanced membership inference attacks against machine learning models. CoRR , abs/2111.09679, 2021. URL https://arxiv.org/abs/2111. 09679 .

Yeom, I. Giacomelli, M. Fredrikson, and S. Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF) , pages 268-282. IEEE, 2018.

Zagoruyko and N. Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146 , 2016.

A Shadow Model Architecture Mismatch

Here we explore how a shadow model membership inference attack is affected by the lack of knowledge of the target model. For this, we vary the shadow model's and target model's architecture between 6 different configurations: 3 CNN models with 32 filters each and varying number of pooling layers between 1 and 3, 4 Wide Residual Networks Zagoruyko and Komodakis [2016] of depth 28 and varying width ( [1 , 2 , 5 , 10] ). We train a single target model and 4 shadow models; all models are trained with SGD with momentum and random augmentations. The results are shown in Figure 4, where small architecture mismatches (i.e, same model family of target and test architecture) generally degrade performance very little, but larger architecture mismatches can cause significant performance degradation.

90.6%1

(a) CIFAR-10

(b) CIFAR-100

Precision @0.1%FPR, cifar100

81.3%

87.

6%4.8%P

96.9%

%7196

81.3%

96.9%

%7796

107

77.0%

9.4%

94.6%

94.4%

94.4%

107

77.0%

84.8

4.9.9

0.9%

90.4%

94.6%

94.4%

94.4%

.6%

76.6%

86.8%

%/08

76.6%

86.8%

%5/t8

106

106

66.5%

65.0%

61.5%

65/1%

66.5%

65.0%

61.5%

65/1%

3.4%0

61.2%

71.

41.3%

C47/5%

71.

66.5%

031.1%

38/3%

44.0%

031.1%

105

62.4%

49.9%

C47,7%

105

62.4%

C44.

C48.8%

49.9%

C47,7%

wrn28-10

wrn28-5

x-T-

cn32-m2-1

cn32-m2-

Shadow ModelSize

B Target Model Accuracies

Here we summarize the accuracy of all target model architectures.

C Proofs

C.1 Proof of Theorem 1

Proof. To establish this theorem, the following lemma will be useful, which informally states that if we can shift a model's quantile predictions to get its 'marginal coverage rate' to be closer to the target 1 -α , then we will also decrease the pinball loss of that model:

Lemma 3 (Jung et al. [2023], Roth [2022]) . Fix any continuous distribution P ∈ ∆ R with density bounded by ρ . Suppose ˆ q is a model such that Pr y ∼P [ y ≤ ˆ q ( x )] = 1 -α ′ , and let ∆ ∈ R be such that Pr y ∼P [ y ≤ ˆ q ( x ) + ∆] = 1 -α . Let q ( x ) = ˆ q ( x ) + ∆ . Then:

Recall that q is chosen such that:

For point of contradiction, suppose that FPR ( A q ) = α ′ for some α ′ = α . Expanding out the definition of the false positive rate, we have that:

So Pr ( x,y ) ∼D [ s ( x, y ) ≤ q ( x )] = 1 -α ′ . Let ∆ ∈ R be such that Pr ( x,y ) ∼D [ s ( x, y ) ≤ q ( x ) + ∆] = 1 -α -Note that such a ∆ is guaranteed to exist by continuity of the distribution on s ( x, y ) . Let q ′ ( x ) = q ( x ) + ∆ . By Lemma 3,

But because H is closed under additive shifts, we also have that q ′ ∈ H . Together, these contradict the optimality of q as measured by pinball loss, which completes the proof.

C.2 Proof of Theorem 2

Proof. Recall that q is chosen such that:

For point of contradiction, suppose that there is some g ∈ G and some α ′ = α such that A q 's false positive rate conditional on g ( x ) = 1 is α ′ . Let D g be the conditional distribution on D conditional on g ( x ) = 1 , and let D ¯ g be the conditional distribution on D conditional on g ( x ) = 0 . Expanding out definitions, we have that:

So Pr ( x,y ) ∼D g [ s ( x, y ) ≤ q ( x )] = 1 -α ′ . Let η ∈ R be such that Pr ( x,y ) ∼D g [ s ( x, y ) ≤ q ( x ) + η ] = 1 -α -Note that such an η is guaranteed to exist by continuity of the distribution on s ( x, y ) conditional on g ( x ) = 1 . Let q ′ ( x ) = q ( x ) + ηg ( x ) . By Lemma 3,

We can relate this decrease in pinball loss conditional on g ( x ) = 1 to the decrease in pinball loss on the underlying distribution D :

̸

̸

But because H is closed under additive shifts by all g ∈ G , we also have that q ′ ∈ H . Together, these contradict the optimality of q as measured by pinball loss, which completes the proof.

D Hyperparameters

We use Ray Tune Liaw et al. [2018] for hyperparameter tuning on image datasets. All experiments use Async Hyperband Scheduler Li et al. [2020] and the Hyperopt search package Bergstra et al. [2013]. Table 4 summarizes the hyperparameters that were tuned and their configurations

We ran 32 trials to find the optimal configuration per experiment. After examining the results, we found that a hidden dim of [512 , 512] and accumulate gradient batches of 2 (Effective batch size 64) were consistently chosen across all experiments. Similarly, epochs were mostly chosen on the 10 -20 range, learning rates and weight decays were consistently chosen near the middle of the value range. This indicates that hyperparameter tuning may not be especially task sensitive and can be shared across attacks.

For tabular data, we use Catboost for model training, and Optuna Akiba et al. [2019] for hyperparameter tuning. Table 5 presents the hyperparameters that were tuned and their corresponding ranges. Each model was tuned with 300 trials with 5-fold cross-validation.

E Additional Results

Here we show extended results on membership inference attacks on CIFAR-10 and CIFAR-100 for ResNet-10, -18, -34, and -50 architectures. Tables 6 and 7 show precision and pinball loss at 1% and 0 . 1% FPR for all target architectures on CIFAR-10 and CIFAR-100 respectively. We observe a strong correlation between lowest pinball loss on test samples and highest precision across the majority of experiments. Figure 5 additionally presents a visual comparison of the true positive rate and false positive rate trade-off for all tested methods on CIFAR-10 and CIFAR-100 for the ResNet-50 architecture.

(a) CIFAR-10

(b) CIFAR-100

100

100

Rate

10-1

Rat

10-1

Positive

sitive

Ours

LIRA (n=2)

Posi

LIRA (n=2)

LIRA (n=4)

LIRA (n=4)

True

10-2

LIRA (n=6)

e

10-2

LIRA (n=6)

LIRA (n=8)

LIRA (n=8)

LIRA (n=256 Visual)

LIRA (n=256Visual)

Marginal Baseline

MarginalBaseline

10-3

10-3

10-3

10-2

10-1

100

10-3

10-2

10-1

100

FalsePositiveRate

FalsePositiveRate

Peer reviews:
comments: SUMMARY:
This paper introduces a new class of membership inference attacks based on performing quantile regression on the distribution of confidence scores induced by the model under attack on points that are not used in training. The approach is computationally efficient and does not require knowledge of the model's architecture, making it truly "black-box". The paper discusses the efficacy of this approach through extensive experiments on various datasets and model architectures. The experiments show that this approach is competitive with (and sometimes more effective than) much more computationally expensive shadow model approaches. Overall, the paper presents a more scalable and efficient membership inference attack.

STRENGTHS:
- interesting topic
- novel method
- well-written paper

WEAKNESSES:
- more evaluation metrics needed
- compare with different attacks

QUESTIONS:
- I appreciate it if the authors could also compare the proposed method to the model-based attacks (https://arxiv.org/abs/1610.05820) and metric-based attacks. (https://arxiv.org/abs/2003.10595)

- Why the proposed method performs worse on C-10 and C-100 but better on IN-1k? I would suggest the authors elaborate more on it.

LIMITATIONS:
- Besides accessing the performance, it is also important to report the time/computational cost of different methods to further demonstrate the efficiency of the proposed method.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a novel membership inference attack that offers computational efficiency compared to state-of-the-art (SOTA) approaches. While the paper addresses a topical issue and provides an alternative attack method, there are several weaknesses that need to be addressed for a more comprehensive and convincing study, including the lack of evaluation of TPR@lowFPR, concerning empirical results, insufficient support for computational efficiency claims, and a lack of empirical evidence for the theoretical results. Addressing these weaknesses is essential to enhance the paper's credibility and provide a more comprehensive evaluation of the proposed attack method. Finally, I would be happy to increase my score if the authors could alleviate some of my concerns.\n\nSTRENGTHS:\n**Topical and relevant**:\nThe paper addresses a relevant topic in the field of membership inference attacks, which is of interest to conference readers. The proposed attack method introduces a novel approach that offers computational efficiency compared to existing SOTA methods.\n\n**Competitive performance without training multiple shadow models**:\nThe authors suggest an attack method that competes with state-of-the-art approaches while being computationally more efficient. By eliminating the need to train several shadow models, their method offers a practical advantage, which is a significant contribution to the field.\n\nWEAKNESSES:\nRegarding the theoretical results:\n\n**Lack of TPR@lowFPR evaluation**: The theoretical results presented in the paper focus on controlling the false positive rate (FPR). However, what truly matters in membership inference attacks (see [1]) is the true positive rate (TPR) at low FPR. The paper lacks analysis of TPR@lowFPR, which is crucial for assessing the attack's effectiveness. (This recent paper [2] could help the authors with this kind of analysis).\n\nRegarding the empirical results:\n\n**Concerning empirical results** \n\n- The quantile attack outperforms the Likelihood Ratio Test-based Inference Attack (LIRA) in terms of TPR@lowFPR on ImageNet. This is surprising, as the Neyman-Pearson Lemma suggests that the LRT is the most powerful test. The discrepancy may be due to the limited computational resources used in training shadow models or differences in training setups when training the underlying imagenet classifier as batch size, # epochs, etc impact the succeptibility to membership inference attacks (e.g., see [1]).\n\n- The replicated LIRA attacks with 8 and 4 shadow models outperform the read-off LIRA attack with 64 shadow models [2]. This observation questions the authors' conclusion that their attack is more favorable. It suggests that if the authors had trained 64 shadow models on ImageNet, the LIRA attack would perform better, as indicated by the evaluation on tabular datasets.\n\n**Computational intensity claims**: The authors argue that their method is computationally more efficient by eliminating the need for training shadow models. However, there is no empirical evidence supporting this claim. It is crucial to provide comparative plots showing the tradeoff between computational cost and TPR@fixed FPR to substantiate this argument.\n\n**Lack of empirical support for theoretical results**: The theoretical results regarding FPR control lack empirical evaluation. Conducting simulation studies or additional experiments would provide empirical evidence to support these claims.\n\n----- \n**References**\n\n[1] Membership Inference Attacks From First Principles, https://arxiv.org/abs/2112.03570\n\n[2] Gaussian Membership Inference Privacy, https://arxiv.org/abs/2306.07273\n\nQUESTIONS:\nCould the authors please elaborate on why the read-off attack results from [2] that use 64 shadow models is performing worse than the authors' implemented attacks with 4 and 8 shadow models? There must be some discrepancies in the underlying models that have been trained!?\n\nLIMITATIONS:\nThis point should be ok.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new membership inference attack based on training an attack model with the pinball loss. This avoids the use of shadow models, a common technique for membership inference, while often outperforming attacks which do use shadow models. They evaluate their attacks on a variety of image and tabular data, and show it is competitive with or outperforms the state of the art LiRA attack on these datasets.\n\nSTRENGTHS:\nThe proposed approach only requires the training time of a single shadow model.\n\nThe proposed attack often reaches very high precision at low false positive rates. Precision is often competitive with attacks that require many shadow models.\n\nThe evaluation considers several datasets, including both image and tabular data.\n\nThe paper is easy to read.\n\nWEAKNESSES:\nSome magic happens in the experiment section in the paragraph starting at line 290 (page 7). Here, the authors appear to introduce a different objective for their attack model, to directly learn a sample’s mean and deviation. I would like to see some ablation of the choices made in this paragraph.\n\nThe proposed attack has a higher “online” computation cost than the offline variant of LiRA. LiRA can train shadow models independently of the target model, while the quantile attack requires the target model to train its attack model.\n\nFrom plots, it looks like the attack performs poorly at higher false positive rates. In settings where attack accuracy or attacker advantage is preferred to TPR and FPR, this attack might end up worse than existing attacks.\n\nResults are only reported for a single target model, rather than averaging over multiple target models.\n\nThe attack model requires hyperparameter tuning, which can be amortized over multiple shadow models when running LiRA.\n\nSmall comment: The definition of precision should have TPR in the numerator\n\nQUESTIONS:\nIs the pinball loss used at all when attacking the CIFAR models? If it is, could you try to explain in more detail how it is used?\n\nDo you have thoughts on why the attacks start to perform worse than other attacks at higher FPR?\n\nLIMITATIONS:\nThe paper does not discuss limitations. It has minor overclaiming in some places, such as when saying the attack requires only a single model, when all results are presented after hyperparameter tuning. Another place is in claiming existing shadow model attacks require knowledge of the target model architecture, which is untrue — the LiRA paper evaluates with different architectures.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe main focus of this paper is about the membership inference attack problem, i.e., determining whether  a particular example was used in training or not. Most existing such attacks estimate the distribution of some test statistics, which are usually computationally expensive. In contrast to existing approaches, this paper proposes to perform quantile regression on the distribution of confidence scores induced by the model under attack on points that are not used in training. For implementation, the authors first collect a dataset known to be never used in training. Given a well trained model $f$, the corresponding confidence scores on collected data can be obtained. Thereafter, a quantile regression model $q$ is trained to predict the  target quantile of the data-label pair.\n\nSTRENGTHS:\nReducing many (around hundred) shadow models to a single one is interesting.\n\nExperiment studies on different image datasets, including CIFAR-10, CIFAR-100, ImageNet-1K.\n\nWEAKNESSES:\n-- Paper writing still needs some improvements. It requires efforts to follow. \n\n-- The parameter $\\alpha$ is important yet hard to adjust manually. \n\n--  Currently, I have not observed any significant influence of the ongoing work on industry-level applications. Despite this, it is important to note that through additional analysis and exploration, there may be a possibility of uncovering substantial potential for impact in the future. As of now, the ongoing work does not appear to have made any noticeable waves in the industry. Nevertheless, by delving deeper into its intricacies and exploring its possibilities, we might unveil its capacity to bring about significant changes that could shape the industry landscape in the coming years.\n\n-- The authors fail to provide a comprehensive explanation of the significance of membership inference attacks (MIAs). It remains unclear why it is crucial to prevent the disclosure of specific data points that were used in training a model. If we assume that we have already inferred the utilization of certain data during the training process, what kind of significant problems could arise as a result? While consulting the references cited in the submission may shed some light on these matters, the paper itself should strive to be self-contained and inclusive, providing a thorough understanding of the subject matter without relying heavily on external sources.\n\n-- In order to facilitate better comprehension, it is necessary to reorganize the related work more effectively. By improving the organization of the related work, we can enhance its accessibility and ensure that readers can easily grasp its significance. A well-structured presentation of the related work will contribute to a clearer understanding of the subject matter.\n\nQUESTIONS:\nAll the current MIAs, including this submission, are based on the observation that models tend to overfit their training sets. Will it still be true for current big models trained on big data? My understanding is that the assumption under MIAs is mainly caused by the generalisation ability of most models.\n\nWill the conclusion still be true in other domains, e.g., natural language and audio?\n\nLIMITATIONS:\nPlease refer to the weaknesses shown above.\n\nMoreover, the contribution of this work appears to be rather limited in scope. To put it simply, the essence of this study involves training a regression model using only a subset of the available training data. It therefore makes that the significance and potential impact of this work may be somewhat restricted.\n\nIt would be better to show experiments on other image models, such as based on VIT, SwinT, BEIT, image encoder in CLIP, .etc.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a novel class of membership inference attacks based on quantile regression applied to confidence score distributions. The proposed 'black-box' algorithm does not require knowledge of the model's architecture and performs competitively with state-of-the-art shadow model attacks while being computationally more efficient. The paper presents several experiments showcasing the effectiveness of the approach across various datasets.\n\nSTRENGTHS:\n1. The paper is well-written, with a clear and understandable motivation.\n\n2. The technical aspects are solid and coherent.\n\nWEAKNESSES:\n1. In my opinion, the main contribution of this paper, which involves using the quantile regression model q to predict the quantiles of the score s(x,y), is not a novel idea within quantile regression. [a] has previously utilized pinball loss in conformal prediction for estimating quantiles of output. Combining quantile regression with membership inference might not present a significant technical challenge and may seem trivial.\n\n2. The claim that the method does not require any knowledge of the model's architecture appears overstated. The method's performance heavily depends on the quantile regression model q. If q is not well-trained, the method will fail. The paper only employs the ConvNext-Tiny model in the main paper, and the sensitivity to different model architectures remains unknown.\n\n3. I find the theoretical part confusing. In my understanding, Theorems 1 and 2 do not establish the theoretical superiority of the proposed method compared to the baseline method using constant quantiles. The paper introduces group validity, which has been extensively studied in the papers mentioned in Line 253, resulting in a lack of theoretical novelty. Additionally, even if the quantile regression model q is learned from a richer set of models, there is no concept of a 'regression model q' for the baseline method. Therefore, it is unclear how the authors concluded that the proposed method outperforms the baseline attack. It seems more intuitive to conclude that a richer model complexity for q may benefit the proposed method.\n\n[a] Romano et al., Conformalized Quantile Regression, NeurIPS 2019\n\nQUESTIONS:\n1. Could you provide a sensitivity study regarding the architecture of q? Is it an important factor? Will the method perform better when q uses the same architecture as f? Will it lead to more accurate quantile estimation?\n\n2. I mentioned my concerns about the theoretical part above.\n\n3. The experimental results do not perform as well on small datasets. While the computation of LIRA when n=2 seems acceptable and achieves much better performance than your method, could you explain the observed gap?\n\nLIMITATIONS:\nI did not find the potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-19"}

comments: SUMMARY:
This paper studies the question of membership inference attack (MIA), which can be formalized as a hypothesis testing (HT) problem.
The main contribution of this paper is introducing a new class of MIA. 
The authors claim that the proposed method is competitive with SOTA MIA methods while being more computationally efficient.
The authors explain the motivation behind their method and theoretically prove the method's effectiveness (in terms of false positive rate).
This paper also conducts experiments on various datasets to evaluate their method.

STRENGTHS:
1. This paper explains the relationship between MIA and HT, which is friendly to readers unfamiliar with MIA.
2. This paper compares the proposed method with those methods in Yeom et al [2018] and Carlini et al [2022]. The proposed methods seem to be a direct generalization of Yeom et al [2018].

WEAKNESSES:
My major concerns lie in the theoretical part of this paper:
1. Theorems 1 and 2 are established at the population level. The approximation error of $q$ is not discussed.
2. This paper does not include any non-asymptotic results.
2. The theoretical part of this paper only compares the proposed method with the baseline attack (Yeom et al [2018]). The difference between the proposed method and Carlini et al [2022] is not discussed.

QUESTIONS:
1. It seems unintuitive that the proposed method outperforms Carlini et al [2022]. Could you please explain this phenomenon?
2. Theorem 1 assumes that $\mathcal{H}$ is closed under additive shifts. Are there any examples of such hypothesis classes?

LIMITATIONS:
No
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-25