Year: 2021

Abstract:
The vulnerability of machine learning models to membership inference attacks has received much attention in recent years. However, existing attacks mostly remain impractical due to having high false positive rates, where non-member samples are often erroneously predicted as members. This type of error makes the predicted membership signal unreliable, especially since most samples are non-members in real world applications. In this work, we argue that membership inference attacks can benefit drastically from difficulty calibration, where an attack’s predicted membership score is adjusted to the difficulty of correctly classifying the target sample. We show that difficulty calibration can significantly reduce the false positive rate of a variety of existing attacks without a loss in accuracy.

text: The vulnerability of machine learning models to membership inference attacks has received much attention in recent years. However, existing attacks mostly remain impractical due to having high false positive rates, where non-member samples are often erroneously predicted as members. This type of error makes the predicted membership signal unreliable, especially since most samples are non-members in real world applications. In this work, we argue that membership inference attacks can benefit drastically from difficulty calibration, where an attack’s predicted membership score is adjusted to the difficulty of correctly classifying the target sample. We show that difficulty calibration can significantly reduce the false positive rate of a variety of existing attacks without a loss in accuracy.

heading: 1 INTRODUCTION
text: Modern applications of machine learning often involve training models on sensitive data such as health records and personal information. Unfortunately, recent studies have found that these models can memorize their training data to a large extent, compromising the privacy of participants in the training dataset (Fredrikson et al., 2014; 2015; Shokri et al., 2017; Carlini et al., 2019). One prominent category of privacy attacks against machine learning is the so-called membership inference attack (Shokri et al., 2017; Yeom et al., 2018), where the adversary aims to infer the participation of an individual in the target model’s training set. Such an attack is undoubtedly damaging when the status of participation itself is considered sensitive, e.g., if the training dataset consists of health records of cancer patients. Moreover, the ability to infer membership can be viewed as a lower bound for the model’s degree of memorization (Yeom et al., 2018), which is useful in itself as an empirical quantifier of privacy loss (Jagielski et al., 2020; Nasr et al., 2021).
The efficacy of membership inference attacks has been improved significantly since the first attempts (Salem et al., 2018; Sablayrolles et al., 2019; Leino & Fredrikson, 2020). However, the most common evaluation metric, attack accuracy, overlooks the crucial factor of the false positive rate (FPR) of non-members (Rezaei & Liu, 2021). Indeed, most attacks operate by first defining a membership score s(h, z) for the model h and a target input-label pair z = (x, y) that measures how much h memorized the sample z. The attack subsequently selects a threshold τ and predicts that z is a member if and only if s(h, z) > τ . For typical choices of the membership scoring function, there is usually a large overlap in the distribution of s(h, z) between members and non-members (see Figure 1a). As a result, an attack that determines the membership of z by thresholding on s(h, z) will have a high FPR. This drawback renders most existing attacks unreliable since the vast majority of samples likely belong to the non-member class.
In this study, we identify the lack of difficulty calibration as a core contributor to the high FPR of existing attacks. Specifically, a non-member sample may have a high membership score simply because it is over-represented in the data distribution. Consequently, an attack that determines a sample is likely to be a member due to having a high score will inevitably fail on these over-represented samples. To remedy this problem, we make the acute observation that if the membership score is measured in comparison to a typical model trained on data drawn from the same data distribution, this difference in behavior can serve as a much more reliable membership signal. Indeed, Figure 1b shows the histogram of scores s(h, z) after difficulty calibration, where the member and non-member scores have significantly better separation and low FPR is now attainable.
We propose difficulty calibration as a general technique for improving score-based membership inference attacks, and modify several membership scoring functions such as confidence (Salem et al., 2018), loss (Yeom et al., 2018), and gradient norm (Nasr et al., 2019) to construct their calibrated variants. Evaluated on a comprehensive suite of benchmark datasets, we show that calibrated attacks achieve a significantly lower FPR compared to prior work. In particular, we measure the trade-off between true positives and false positives using the area under ROC curve (AUC) metric, and show that difficulty calibration drastically improves this trade-off compared to uncalibrated attacks, by up to 0.10 AUC on common ML benchmarks. In addition, calibrated attacks also drastically improve the precision-recall trade-off, while remaining on-par with or better than uncalibrated attacks in terms attack accuracy. Our results suggest that it may be important for future work to apply difficulty calibration to design more reliable and practical membership inference attacks.

heading: 2 BACKGROUND
text: Membership inference attacks are concerned with determining whether a given sample was part of a target model’s training set. Homer et al. (2008) showed in their pioneering study that it is possible to infer an individual’s presence in a complex genomic DNA mixture, which led to increased caution around releases of DNA data (Zerhouni & Nabel, 2008). Recent interest in member inference attacks was sparked by the work of Shokri et al. (2017), who introduced the shadow models method: An adversary trains substitute models (called shadow models) to mimic the behavior of the model under attack (called the target model). The adversary then observes the behavior of the shadow models when exposed to member and non-member samples, and uses this observation to train an attack meta-model for predicting membership on any given sample. Shokri et al. (2017) evaluated this attack on ML models trained on cloud APIs, and showed that the shadow models approach attains high levels of accuracy.
Score-based attacks. Yeom et al. (2018) discovered a connection between membership inference attacks and overfitting, arguing that in principle, attack accuracy can be determined by how much the target model memorizes (or overfits to) the given sample (x, y). This discovery led to a series of work on quantifying the degree of memorization via membership scores, which can be used for predicting that the given sample is a member when the score is high. The membership score can be computed using the loss (Yeom et al., 2018), the gradient norm (GN) (Nasr et al., 2019) or the confidence of the model’s prediction (Salem et al., 2018), often yielding state-of-the-art results (Salem et al., 2018; Choquette-Choo et al., 2021). We define these scores for the cross-entropy loss `:
sloss(h, (x, y)) = −`(h(x), y) := log(h(x)y), (1) sGN(h, (x, y)) = −‖∇`(h(x), y)‖2, (2)
sconfidence(h, (x, y)) = −max y′ `(h(x), y′) = max i log(h(x)i). (3)
Label-only attacks. The above score-based attacks rely on continuous-valued predictions from the model in order to define the membership score. To counter these attacks, prior work considered obfuscating the model’s output by returning only the top label or modifying the predicted values (Shokri et al., 2017; Jia et al., 2019). However, subsequent studies showed that even with only hard-label output, it is still possible to define membership scores that attain close to state-of-the-art accuracy (Li & Zhang, 2020; Choquette-Choo et al., 2021).
High-precision attacks. Various forms of difficulty calibration have been considered in the context of high-precision attacks. Long et al. (2018) selected samples that differ the most in loss between the target and a set of reference models, and showed that the resulting attack has high precision even for well-generalized target models. Carlini et al. (2020) showed that such privacy attacks are also possible on large-scale language models such as GPT-2 (Radford et al., 2019). Their attack operates by first generating a large number of sentences from the language model and then ranking these sentences by the (log) perplexity, with lower perplexity indicating a more plausible memorized training sample. These perplexity values are then divided by either the z-lib entropy or the perplexity given by a smaller language model to account for the sentence’s rarity. In effect, only rare sentences with low perplexity can minimize the resulting score. Both attacks leverage a form of difficulty calibration by comparing the target model’s loss with that of reference models, predicting membership only when the difference (or ratio) is large.
Differential privacy as a mitigation. Differential privacy (Dwork et al., 2006) (DP) is a powerful mathematical framework for privacy-preserving data analysis. A randomized algorithmM satisfies ( , δ)-differential privacy if, given any two datasets D and D′ that differ in at most one sample, and for any subset R of the output space, we have:
P(M(D) ∈ R) ≤ exp( )P(M(D′) ∈ R) + δ. (4)
Under mild assumptions, DP provably protects against a variety of privacy attacks, in particular membership inference (Yeom et al., 2018; Sablayrolles et al., 2019). Abadi et al. (2016) proposed a differentially private version of stochastic gradient descent (SGD), called DP-SGD, to enable differentially private training of generic ML models. Their analysis has further been refined (Mironov, 2017; Mironov et al., 2019) and has been shown experimentally to be tight (Nasr et al., 2021).

heading: 3 DIFFICULTY CALIBRATION
text: As depicted in Figure 1, prior works on score-based membership inference attack are very unreliable for separating easy-to-predict non-members from hard-to-predict members since both can attain a high membership score. We argue that a simple modification to the score, which we call difficulty calibration, can drastically improve the attack’s reliability. This approach has been applied to the loss score for high-precision attack against well-generalized models (Long et al., 2018).
Let s(h, (x, y)) be the membership score, where higher score indicates a stronger signal that the sample (x, y) is a member. Instead of computing the membership score only on the target model h, we sample multiple “typical” models trained on the same data distribution as h and evaluate the membership score on these models. Doing so calibrates s(h, (x, y)) to the difficulty of the sample. For instance, if (x, y) is an easy-to-predict non-member, then s(h, (x, y)) is high but typical models g would also perform well on (x, y), hence s(g, (x, y)) would also be high. The small gap between s(h, (x, y)) and s(g, (x, y)) suggests that (x, y) is likely a non-member.
Formally, let Dshadow be a shadow dataset drawn from the same data distribution as the training set of h, and let A be a randomized training algorithm that samples from a distribution over models trained on Dshadow. We define the calibrated score as:
scal(h, (x, y)) = s(h, (x, y))− Eg←A(Dshadow)[s(g, (x, y))], (5)
where the expectation is approximated by sampling one or more models from A(Dshadow). The membership inference attack proceeds by thresholding on the score scal(h, (x, y)).
Efficient difficulty calibration via forgetting. Faithfully executing the difficulty calibration algorithm in Equation 5 requires training multiple models g using the randomized training algorithm
A. This can be prohibitively expensive if the model is large, which is the case with modern neural networks such as large-scale transformers (Brown et al., 2020).
An alternative, more efficient approach if given white-box access to the target model h is to warmstart training on h. In fact, this approach establishes an explicit connection between membership inference attacks and catastrophic forgetting in neural networks (Goodfellow et al., 2013; Kirkpatrick et al., 2017). Toneva et al. (2018) observed that when a trained model resumes training on a separate dataset drawn from the same data distribution, it is very likely to “forget” about its original training samples. The training examples that are most likely forgotten are ones with abnormal or distinctive features, and this set of examples is stable across different training runs. We leverage this phenomenon to define a more efficient variant of difficulty calibration.
Connection to posterior inference. Under a simplifying assumption of the distribution A(D) induced by the training algorithm, it is possible to derive difficulty calibration as an approximation to an optimal white-box attack. Sablayrolles et al. (2019) proposed a formal analysis of membership inference, where they assumed that the density function of A(D) has the form:
p(h;D) ∝ e− ∑ (x,y)∈D `(h(x),y), (6)
which can be formally derived for optimization using SGD under certain conditions (Sato & Nakagawa, 2014). The implied threat model in differential privacy assumes that the adversary has knowledge that the training dataset is either D or D′ = D ∪ {(x, y)} for some inference target (x, y), and D is known to the adversary. It can be shown that the Bayes-optimal attack strategy under this setting is to threshold on the calibrated score in Equation 5, where s = sloss andD = Dshadow (Sablayrolles et al., 2019). This analysis can be generalized to any scoring function s(h, (x, y)), such as gradient norm (Equation 2) and confidence (Equation 3).

heading: 4 EXPERIMENTS
text: To demonstrate the effect of difficulty calibration, we perform a comprehensive evaluation of several score-based attacks on standard benchmark datasets. We evaluate both variants of difficulty calibration: (i) calibration by training reference models from scratch, and (ii) calibration via forgetting.

heading: 4.1 EXPERIMENTAL SETUP
text: Datasets. We perform experiments on several benchmark classification datasets: German Credit, Hepatitis and Adult datasets from the UCI Machine Learning Repository (Dua & Graff, 2017), MNIST (LeCun et al., 1998), CIFAR10/100 (Krizhevsky et al., 2009), and ImageNet (Deng et al., 2009). The datasets vary in size from 155 to 1,281,167 points and represent both image and nonimage classification tasks.
Attack setup. We split the data into two sets: a private set, known only to the trainer, and a public set, which is used for training reference models and selecting the decision threshold τ . The trainer trains their model h on half of the private set, keeping the other half as non-members. At evaluation time, a sample (x, y) is selected from the private set. The attacker computes a membership score s(h, (x, y)) and predicts (x, y) as a member if s(h, (x, y)) > τ . For ImageNet, we sample 10,000 member and non-member samples from the member and non-member sets, while for all other datasets we use the full private set for evaluation. With such a setup, the accuracy of a random adversary is 50%, as is the precision at all levels of recall.
We repeat this attack under the same setup T times, each time performing a different random split into the private/public and member/non-member sets. We set T = 5 for all datasets except for ImageNet, where T = 3. Thus, in all our results, the reported averages and standard deviations reflect randomness in both model optimization and in the data splits.
Model Architectures. The target model for the German Credit, Adult and Hepatitis datasets is a multi-layer perceptron (MLP) with one hidden layer and ReLU activation followed by a softmax layer. For a dataset withm features, the hidden layer has 2m hidden units. For MNIST, we followed the architecture used in Leino & Fredrikson (2020), which is a small convolutional network with two
convolutional layers with 20 and 50 output channels, respectively, and a kernel size of 5 followed by max pooling layers. The classification head consists of one fully connected layer with 500 hidden units and ReLU activation. Dropout at rate 0.25 is applied after the the max pooling layers and at rate 0.5 after the fully connected layer. For CIFAR10 and CIFAR100, we used a CNN architecture with 4 convolutional and average pooling layers and ReLU activations using 32, 64, 64 and 128 output channels respectively with kernel size 3. For ImageNet, we used the ResNet18 (He et al., 2016) model architecture.
Model training. The target models were trained for between 50 and 200 epochs, with batch sizes varying from 4 (for very small datasets) to 1024. For optimization, we used SGD with a learning rate of 0.1, Nesterov momentum of 0.9 and a cosine learning rate schedule for the CIFAR10/100 and ImageNet datasets. Smaller datasets such as the German Credit dataset also used weight decay of 1× 10−4. Differentially private models were trained using the Opacus1 library. Table 1 presents the target model accuracy statistics.
Reference models for difficulty calibration are trained using the same hyperparameters as the target model. For most experiments, we train a single reference model for calibration, and show the effect of the number of reference models in subsection 4.3. For calibration via forgetting, reference models were tuned over a set of learning rates and batch sizes to ensure that their training and test accuracies were similar to the target model for their largest number of training epochs. For both calibration methods, the reference models were trained until their validation accuracy reached its maximum value, and stopped before their training accuracy converged to
its maximum. Due to its warm start, calibration-via-forgetting generally used fewer training epochs for reference models.
Threshold selection. When computing AUC and precision/recall metrics, we sweep over a range of values for the threshold τ and measure the resulting attack’s FPR/TPR and precision/recall tradeoffs. To find a threshold for optimal accuracy, we first split the public set of examples in half again, and treat one half as members, with the rest as non-members. We can then choose the best threshold τ based on membership inference accuracy for this simulated setup. Frequently, the optimal threshold for calibrated attacks was a value only slightly greater than 0 (e.g., 0.0001), which can be used as a default threshold if threshold selection is not performed.
1https://github.com/pytorch/opacus

heading: 4.2 MAIN RESULTS
text: We first evaluate attacks that rely on continuous-valued output from the target model: loss score (Equation 1), gradient norm (GN) score (Equation 2), and confidence score (Equation 3). We also consider the gap attack (Yeom et al., 2018) as a baseline to measure progress against.
AUC and accuracy. We compute two primary metrics: area under ROC curve (AUC) for evaluating the FPR/TPR trade-off, and attack accuracy. Table 2 shows the computed AUC for various attacks before and after difficulty calibration. Overall, calibrated attacks often perform drastically better than their uncalibrated counterparts, and in some instances the performance increase can exceed 0.10, such as on the German Credit and CIFAR10 datasets. Table 3 shows the corresponding attack accuracy numbers. Evidently, calibration does not reduce accuracy in most cases, and sometimes brings a modest improvement.
We observe that the confidence score attacks under-perform other methods due to the model predicting non-member points incorrectly with high confidence. For this reason, calibrated confidence did not improve the attack performance on many datasets, as training points that were correctly classified on the target model but were incorrectly classified on the reference model became indistinguishable.
Effect of data augmentation. The last two rows in Tables 2 and 3 show various attacks’ performance on the CIFAR10/100 target model trained with data augmentation. As expected, attacks are less effective on models with data augmentation due to reduced overfitting. Nevertheless, calibration still improves both the AUC and accuracy of loss and gradient norm based attacks most of the time.
ROC and precision-recall curves. Figure 2 shows the ROC and precision-recall curves for the loss score attack on CIFAR10, which demonstrates the effect of difficulty calibration in closer detail. In the left plot, at low FPR values, the uncalibrated attack has close to zero advantage in TPR, behaving essentially the same as random guessing. In stark contrast, calibrated attacks exhibit a large difference in the FPR and TPR values. Furthermore, as shown in the right plot, calibrated attacks can detect a group of member points with very high precision at low recall values, which is not possible for the uncalibrated counterpart. This pattern can also be clearly seen in the membership score distributions of calibrated and uncalibrated attacks shown in Figure 1.
We highlight another interesting phenomenon by computing the threshold τ that achieves optimal accuracy for each attack, shown by the red dot in Figure 2. For all attacks, this threshold lies towards the tail of the ROC and precision-recall curves, where the attack achieves high FPR and high recall but low precision. This observation concurs with prior work, which shows that existing attacks do not predict membership reliably when optimizing for accuracy (Rezaei & Liu, 2021).
Label-only attacks. When the target model returns only a discrete label rather than a continuousvalued output, membership inference can still be performed using label-only attacks. We show that difficulty calibration also improves such attacks. Table 4 reports the performance of calibrated and uncalibrated label-only attacks using the HopSkipJump boundary distance method (Choquette-Choo et al., 2021). We excluded the MNIST and ImageNet datasets as the HopSkipJump attack required too many queries to successfully change the label. Similar to the results in Tables 2 and 3, labelonly attacks also benefit from difficulty calibration with increased AUC and improved or similar accuracy. However, the performance increase is noticeably smaller for CIFAR10/100 compared to other score-based attacks.

heading: 4.3 ABLATION STUDIES
text: Training set size. Since vulnerability to membership inference attack is linked to overfitting (Yeom et al., 2018), training set size for the target model is a crucial factor in attack performance. Indeed, Figure 3 shows that for the gap attack and calibrated/uncalibrated loss score attacks, both AUC and accuracy increase when the training set size is reduced. More importantly, difficulty calibration can effectively leverage overfitting to improve the attack’s performance across the entire range of training set sizes.
Ratio of member to non-member samples. The experimental setup so far uses an equal number of member and non-member samples. In real world scenarios, the majority of samples are likely non-members, which drastically affects the accuracy and precision-recall trade-off of membership inference attacks. In the following experiment, we simulate this scenario by subsampling the member set for evaluation to decrease the ratio of member to non-member samples.
Figure 4 shows the accuracy and precision-recall curves for different member to non-member ratios. In the left plot, as the number of non-member samples increases, attack accuracy also drastically increases. This is due to the threshold τ for optimal accuracy favoring high recall and low precision (see Figure 2), effectively performing “non-membership inference”. In contrast, the right plot shows that precision-recall trade-off worsens when the number of non-member samples increases, which reflects the fact that attacks are less reliable in the real world when most samples are non-members. Nevertheless, for the calibrated attack, it is still possible to achieve a high precision at low values of recall even for low member to non-member ratios.
Number of reference models. Results reported in Section 4.2 use a single reference model for difficulty calibration. Figure 5 shows the effect of increasing the number of reference models on AUC and attack accuracy. To train each reference model, we subsample a different subset of the public dataset without replacement to create the shadow dataset. We then apply calibration-viaforgetting to calibrate the loss score, while varying the shadow dataset size. Increasing the number of reference models and increasing the shadow dataset size strictly improve both AUC and accuracy. The difference between 1 and 10 reference models is up to 0.05 for AUC and 0.03 for accuracy.
Differential privacy is by far the most common defense against membership inference attacks, and provably limits the accuracy of these attacks (Yeom et al., 2018) when is small. With a higher value of , DP still empirically protects against uncalibrated membership inference attacks on average; however, calibrated attacks provide a significant edge against outliers for high values of the privacy parameter .
Figure 6 shows the ROC and precision-recall curves of calibrated/uncalibrated loss score attacks on CIFAR10 models trained with DP-SGD. As expected, DP clearly reduces the effectiveness of attacks as indicated by the gap between attacks against models with finite (blue and orange lines) and attacks against an undefended model (dotted line). However, for = 100 and 1000, calibrated attacks can still attain a non-trivial FPR-TPR and precision-recall trade-off. Only when gets close to 1 do the AUC and accuracy approach that of uncalibrated attacks, and both attacks converge to the effectiveness of random guessing.
A qualitative inspection of member images from CIFAR10 that are successfully inferred at a high precision of > 0.9 for the DP model at = 1000, but are not identified for ≤ 10, showed that vulnerable images were often atypical samples from their respective classes. They did not appear to be extreme outliers. A further manual inspection of examples did not suggest any general pattern as to which ones are most vulnerable under differential privacy.

heading: 5 CONCLUSION
text: We showed that difficulty calibration is a powerful post-processing technique for improving existing membership inference attacks. Most score-based attacks can greatly benefit from calibration, making the attack much more reliable in low false positive rate and high precision regimes. Our study also showed that prior work that focused on evaluating attack accuracy do not reveal the full picture, and detailed analysis using the ROC and precision-recall curves is crucial for designing reliable membership inference attacks.
When given white-box access to the target model, calibration via forgetting can be an efficient alternative to training reference models from scratch. This alternative method also serves as an explicit connection between membership inference and the well-studied phenomenon of catastrophic forgetting in neural networks. We hope that future work can further explore this connection and leverage new findings to improve the state-of-the-art in membership inference attacks.

Review summaries:
This paper focuses on the analysis of membership inference attacks (MIAs), in  particular addressing the problem of high false positive rate (FPR) that still affects the  state-of-the-art solutions. The authors suggest an approach to MIA that is based on difficulty calibration, i.e., a method where the predicted membership score is adjusted  to the difficulty of obtaining the correct classification for the target sample, in order to reduce the FPR. As illustrated in Figure 1, previous work on the score-based membership inference attack is highly unreliable for separating easy-to-predict non-members from hard-to-predict members, as both can achieve high membership scores. This technique appears to be particularly beneficial in helping to distinguish between  member and non-member samples that are overrepresented in the data distribution. The authors support the proposed method with a large body of mostly favorable experimental results.

The paper aims to solve the issue of high false positive rates in modern membership inference attack methods by computing the difference in model prediction metrics (e.g., loss, confidence) between the target model and a reference model. The intuition is that “easy-to-predict non-members” will have a small difference between the target and the reference model, while “hard-to-predict members” will have a large difference. Experimental results on multiple datasets also validate the effectiveness of proposed method.

This work proposes a simple output calibration method to suppress false positive errors of membership attacks. The core idea is to define a calibrated decision score as the difference between the original membership score and the expected membership score trained by a randomized training algorithm using a shadow dataset (a dataset following the same distribution as the true training data set). The calibrated membership decision score can provide a lower false positive rate than the original membership score.