On the Importance of Difficulty Calibration in Membership Inference Attacks
Authors: Lauren Watson
Chuan Guo
Graham Cormode
Alexandre Sablayrolles
Venue: ICLR 2022 Poster (2022)
Keywords: membership inference attack
privacy

Abstract: The vulnerability of machine learning models to membership inference attacks has received much attention in recent years. However, existing attacks mostly remain impractical due to having high false positive rates, where non-member samples are often erroneously predicted as members. This type of error makes the predicted membership signal unreliable, especially since most samples are non-members in real world applications. In this work, we argue that membership inference attacks can benefit drastically from difficulty calibration, where an attack's predicted membership score is adjusted to the difficulty of correctly classifying the target sample. We show that difficulty calibration can significantly reduce the false positive rate of a variety of existing attacks without a loss in accuracy.

Published as a conference paper at ICLR 2022
ON THE IMPORTANCE OF DIFFICULTY CALIBRATION IN MEMBERSHIP INFERENCE ATTACKS
Lauren Watson' University of Edinburgh
Chuan Guo
Graham Cormode Meta Al
Alexandre Sablayrolles
ABSTRACT
The vulnerability of machine learning models to membership inference attacks has received much attention in recent years. Existing attacks mostly remain impracti- cal due having high false positive rates, where non-member samples often erroneously predicted as members. This type of error makes the predicted mem- bership signal unreliable, especially since most samples are non-members in real world applications. In this work, we argue that membership inference attacks can benefit drastically from difficulty calibration, where an attack's i predicted member- ship score is adjusted to the difficulty of correctly classifying the target sample. We show that difficulty calibration can significantly reduce the false positive rate of variety of existing attacks without a loss in accuracy.
INTRODUCTION
Modern applications of machine learning often involve training models on sensitive data such as health records and personal information. Unfortunately, recent studies have found that these models can memorize their training data to a large extent, compromising the privacy of participants in the training dataset (Fredrikson et al. 2014 2015) Shokri et al. 2017 Carlini et al. 2019). One promi- nent category of privacy attacks against machine learning is the so-called membership inference attack (Shokri et al. 2017; Yeom et al. 2018), where the adversary aims to infer the participation of an individual in the target model's training set. Such an attack undoubtedly damaging when the status of participation itself is considered sensitive, e.g., if the training dataset consists of health records of cancer patients. Moreover, the ability to infer membership can be viewed as a lower bound for the model's degree of memorization (Yeom et al., 2018). which is useful in itself as an empirical quantifier of privacy loss (Jagielski et al. 2020 Nast et al. 2021).
The efficacy of membership inference attacks has been improved significantly since the first at- tempts (Salem et al. 2018 Sablayrolles et al. 2019: Leino & Fredrikson) 2020). However, the most common evaluation metric, attack accuracy, overlooks the crucial factor of the false positive rate (FPR) of non-members (Rezaei & Liu| [021]. Indeed, most attacks operate by first defining mem- bership score s(h, 2) for the model h and a target input-label pair z = (x, y) that measures how The attack subsequently selects threshold and predicts that
much h memorized the sample # is a member if and only if sh,, x) > т. For typical choices of the membership scoring function, there is usually large overlap in the distribution of s(h, x) between members and non-members (see Figure Ta). As a result, an attack that determines the membership of z by thresholding on s(h, 2) will have high FPR. This drawback renders most existing attacks unreliable since the vast majority of samples likely belong to the non-member class.
In this study, we identify the lack of difficulty calibration as a core contributor to the high FPR of existing attacks. Specifically, a non-member sample may have a high membership score simply because it is over-represented in the data distribution. Consequently, an attack that determines sam- ple is likely to be a member due to having a high score will inevitably fail on these over-represented samples. To remedy this problem, we make the acute observation that if the membership score is measured in comparison to a typical model trained on data drawn from the same data distribu- tion, this difference in behavior can serve as a much more reliable membership signal. Indeed,
Work done during an internship Facebook. htt.:/aarren.attstteee.e......c {chuanguo, geormode, asablayrolles) fb..com

---

Published as a conference paper at ICLR 2022
10a
Member Non-Member
Member Non-Member
10a
Log Frquency
10a
Log Frequency
10a
10?
104
109
10
-0
-30
-20
Calibrated
Score
(a) Before calibration
(b) After calibration
Figure 1: Histogram of the negative loss score (cf. [Equation 1b before and after difficulty calibration. Without calibration, the member and non-member scores overlap significantly, and it is impossible determine threshold that results low FPR. After calibration, the highest scored samples mostly belong to the member class, enabling high precision and low FPR attacks.
Figure 1b shows the histogram of scores s(h, z) after difficulty calibration, where the member and non-member scores have significantly better separation and low FPR is now attainable.
We propose difficulty calibration a general technique for improving score-hased membership inference attacks, and modify several membership scoring functions such as confidence (Salem al.) 2018), loss (Yeom al. 2018), and gradient norm (Nast et al., 2019) to construct their calibrated variants. Evaluated on comprehensive suite of benchmark datasets, we show that calibrated attacks achieve significantly lower FPR compared to prior work. In particular, we measure the trade-off' between true positives and false positives using the area under ROC curve (AUC) metric, and show that difficulty calibration drastically improves this trade-off compared to uncalibrated attacks, by up to 0. 1010 AUC on common MI benchmarks. In addition, calibrated attacks also drastically improve the precision-recall trade-off, while remaining on-par with better than uncalibrated attacks in terms of attack accuracy. Our results suggest that it may be important for future work to apply difficulty calibration design more reliable and practical membership inference attacks.
2 BACKGROUND
Membership inference allacks are concerned with determining whether a given sample was part of target model's training set. Homer et al. (2008) showed in their pioneering study that it is possible to infer individual's 's presence in complex genomic DNA mixture, which led to increased caution around relcases of DNA data Zeerhouni 2008). Recent interest in member inference attacks was sparked by the work of Shhkri et al. (2017). who introduced the shadow models method: an adversary trains substitute models (called shadow models) to mimic the behavior of the model under attack (called the target model). The adversary then observes the behavior of the shadow models when exposed to member and non-member samples, and uses this observation to train an attack meta-model for predicting membership on any given sample. [Shokri et al. (017) evaluated this attack on ML models trained on cloud APIs, and showed that the shadow models approach attains high levels of accuracy.
Score-based attacks. Yeom et al. (2018) discovered a connection between membership inference attacks and overfitting, arguing that in principle, attack accuracy can be determined by how much the target model memorizes (or overfits to) the given sample (x, v). This discovery led to series of work on quantifying the degree of memorization via membership scores, which can be used for predicting that the given sample is member when the score is high. The membership score computed using the loss (Yeom et al. 2018). the gradient norm (GN) (Nasr et al.) 2019) or the con- fidence of the model's prediction (Salem et al. 2018), often yielding state-of-the-ane results (Salem)
be
2

---

Published as a conference paper at ICLR 2022
al. 2018 Choquette- Choquette-Choo et al. )021). We define these scores for the cross-entropy loss &
Stoss (h, (z, y)) (h(())) y) log(h(x))))
(1)
8GN (h, (2, y)) = [[Ve(h(=), v)l2,,
(2)
Sconfdence (h, (x. 3)) = max (((())./) = max log(h(x)).)
(3)
y'
Label-only attacks. The above score-based attacks rely the model in order to define the membership score. To counter these attacks, prior work consid- ered obfuscating the model's output by returning only the top label or modifying the predicted values (Shokri et al., 2017 Jia et al. 2019). However, subsequent studies showed that even with only hard-label output, it possible to define scores attaining close to state-of-the-arr accuracy LLi & Zhang 2020: Choquette- Craqueete-Choo et al. (021))
pantinuous-vallee predictions from
High-precision attacks. Various forms of difficulty calibration have been considered in the context of high-precision attacks. [ongg et al. (2018) selected samples that differ the most loss between the target and set of reference models, and showed that the resulting attack has high precision even for well-generalized target models. Carlini et al. (2020) showed that privacy attacks are also possible on large-scale language models such as GPT-2 (Radford et al.) 2019). Their attack operates by first generating large number of sentences from the language model and then ranking these sentences by the (log) perplexity, with lower perplexity indicating more plausible memorized training sample. These perplexity values are then divided by either the z-lib entropy or the perplexity given by a smaller language model account for the sentence's rarity. In effect, only rare sentences with low perplexity can minimize the resulting score, Both attacks leverage form of difficulty calibration by comparing the target model's loss with that of reference models, predicting membership only when the difference (or ratio) is large.
Differential privacy as a mitigation. Differential privacy (Dwork et al. 2006 (DP) is powerful
mathematical framework for privacy-preservnn data analysis. A randomized algorithm M satisfies (e, dddiffferential privacy if, given any two datasets D and D) that differ in at most one sample, and for any subset R of the output space, we have:
PPM(D) E R) S exp(c)P(M(D') E R) +
(4)
Under mild assumptions, DP provably protects against a variety of privacy attacks, in particular membership inference (Yeom et al. 2018 Sablayrolles et al. 2019 Abadi et al. (2016) proposed a differentially private version of stochastic gradient descent (SGD), called DPDSGD, to enable dif- ferentially private training of generic ML models. Their analysis has further been refined (Mironov)
2017 Mironov et al. 2019) and has been shown experimentally to be tight (Nasr et al. 2021).
3 DIFFICULTY CALIBRATION
As depicted in Figure prior works on score-based membership inference attack are very unreliable for separating easy-to-predict non-members from hard-to-predict members since both attain high membership score. We argue that a simple modification to the score, which we call difficulty calibration, can drastically improve the attack's reliability. This approach has been applied to the loss score for high-precision attack against well-generalized models (Long et al. 2018).
Let s(h, (x.y)) be the membership score, where higher score indicates stronger signal that the sample (x, y) is a member. Instead of computing the membership score only on the target model h, we sample multiple "typical" models trained on the same data distribution as h and evaluate the membership score on these models. Doing so calibrates s(h, (æ,, y)) to the difficulty of the sample. For instance, if (ar, y) is an easy-to-predict non- non-member, then shh, (æ., y)) high but typical models g would also perform well on (x, y), hence s(g, (x, y)) would also be high. The small gap between s(h, (x, 1)) and s(g, (x, y)) suggests that (x, 3) is likely a non-member.
Formally, let Dshadow be a shadow dataset drawn from the same data distribution as the training set of h, and let A be a randomized training algorithm that samples from a distribution over models trained on D Dehadow. We define the calibrated score as: soll (h, (xx, u)) s(h, (x. v)) Byye A(tahadon [s(g, (ar, v))}
(5)
3

---

Published as a conference paper at ICLR 2022
where the expectation is approximated by sampling one models from A (Dshadoo) The membership inference attack proceeds by thresholding on the score s (h, (a. y)).
Efficient difficulty calibration via forgetting. Faithfully executing the difficulty calibration algo- rithm in Equation 5) requires training multiple models 9 using the randomized training algorithm A. This can be prohibitively expensive if the model is large, which is the case with modern neural networks such as large-scale transformers (Brown al. 2020).
An alternative, more efficient approach if given white-box access to the target model h is to warm- start training on h. In fact, this approach establishes an explicit connection between membership inference attacks and catastrophic forgetting in neural networks (Goodfellow et al. 2013 Kirk- patrick et al.] (017). [Toneva et al. (2018) observed that when trained model resumes training on separate dataset drawn from the same data distribution, it is very likely to "forget" about its original training samples. The training examples that are most likely forgotten are ones with abnormal or distinctive features, and this set of examples is stable across different training runs. We leverage this phenomenon to define a more efficient variant of difficulty calibration.
Connection to posterior inference. Under a simplifying assumption of the distribution A(D) in- duced by the training algorithm, it is possible to derive difficulty calibration as an approximation to an optimal white-box attack. [Sablayrolles et al. (2019) proposed formal analysis of membership inference, where they assumed that the density function of A(D) has the form:
p(h; D) x E(..yie {(h(a)).))
which can be formally derived for optimization using SGD under certain conditions (Sato Naka- gawa 2014). The implied threat model in differential privacy assumes that the adversary has knowl- edge that the training dataset is either or D' = 'D UU {(x, y)} for some inference target (æc, y), and D is known to the adversary. The Bayes-optimal attack strategy under this setting can be approached by thresholding on the calibrated score in [qqaaiio 5) where = $loes and 'D = DD Dahadow (Sablay- rolles et al. 2019). This analysis can be extended any scoring function sh,, (x, y)), such as gradient norm or confidence (Equations)
4 EXPERIMENTS
To demonstrate the effect of difficulty calibration, we perform a comprehensive evaluation of several score-based attacks standard benchmark datasets! We evaluate both variants of difficulty cali- bration: (i) calibration by training reference models from scratch, and (ii) calibration via forgetting.
4.1 EXPERIMENTAL SETUP
Datasets. We perform experiments several benchmark classification datasets: German Credit, Hepatitis and Adult datasets from the UCI Machine Learning Repository (Dua Graff 2017). MNIST (LeCun et al., 1998), CIFAR10/100 (Krizhevsky et al.) 2009), and ImageNet (Deng et al.) 2009). The datasets vary in size from 155 to 1,281,167 points and represent both image and non- image classification tasks.
Attack setup. We split the data into two sets: private set, known only the trainer, and public set, which is used for training reference models and selecting the decision threshold 7. The trainer trains their model h on half of the private set, keeping the other half non memmeeeers. At evalu- ation time, sample (z, y) is selected from the private set. The attacker computes a membership score s(h, (æe, y)) and predicts (xr, y) as a member s(h, (æ,, y)) > т. For ImageNet, we sample 10,000 member and non-member samples from the member and non-member sets, while for all other datasets we use the full private set for evaluation. With such a setup, the accuracy of a random adversary is 50%, is the precision at all levels of recall.
We repeat this attack under the same setup T' times, each time performing a different random split into the private/public and nember/non-memble sets. We set T 5 for all datasets except for
AAn implementation of these attacks is available at ca 1ilbbaa braat oon eo__emberrship
4

---

Published as a conference paper at ICLR 2022
1.0
1.0
Lass
Cal. Loss
True Positive Rate
0.8-
0.9
Cal.-F Loss
Best Accuracy
0.6
Precision
0.8
0.4
0.7-
Loss
Loss
0.2
0.6-
Cal.-F Loss
Accuracy
0.0
0.5
0.0
0.2
0.4 0.5
0.8
1.0
0.0
0.2
0.6
0.8
1.0
False Positive Rate
Recall
Figure 2: Left: ROC (left) and precision-recal (right) curves of calibratedduucaalibraeed loss score attacks on CIFAR1O. The threshold that optimizes accuracy is shown as red dot. Calibration yields a higher TPR for the same value of FPR, or equivalently a higher precision low levels of recall. The precision-recall trade-off surfaces very different behaviors for two methods that have otherwise very similar accuracy.
ImageNet, where T 3. Thus, in all our results, the reported averages and standard deviations reflect randomness in both model optimization and in the data splits.
Model Architectures. The target model for the German Credit, Adult and Hepatitis datasets is multi-layer perceptron (MLP) with one hidden layer and ReLU activation followed by a softmax layer. For dataset with features, the hidden layer has 2m hidden units. For MNIST, the architecture used in Leino & Fredrikson (2020), which is a small convolutional network with two convolutional layers with 20 and 50 output channels respectively, and kernel size of 5, followed by max pooling layers. The classification head consists of one fully connected layer with 500 hidden units and ReLU activation. Dropout rate 0.25 is applied after the max pooling layers and rate 0.5 after the fully connected layer. For CIFARIO and CIFAR 100, we used CNN architecture with convolutional and average pooling layers and ReLU activations using 32, 64, 64 and 128 output channels respectively with kernel size 3. For ImageNet, we used the ResNet18 (He al.) 2016) model architecture.
followed
Model training. The target models are trained for between 50 and 200 epochs, with batch sizes varying from 4 (for very small datasets) 1024. For optimization, we use SGD with a learning rate of 0.1, Nesterov momentum of 0.9 and cosine learning rate schedule for the CIFAR 10/10000000 and ImageNet datasets. Smaller datasets such as the German Credit dataset also used weight decay of
1 x 10-4 Differentially private training is done using the Opacus YYousefpour et al. 2021 library.
Table presents the target model accuracy statistics.
Reference models for difficulty calibration trained us- ing the same hyperparameters as the target model. For most experiments,
Task
Train Acc.
Test Acc.
train single reference model for calibration, and show the effect of the number refer- ence models in Isubsection 4.3) For calibration via for- getting, reference models were tuned over a set of learn- ing rates and batch sizes ensure that their training and test accuracies were similar to the target model for their largest number of training epochs. For both cali- bration methods, the reference models were trained un- til their validation accuracy reached its maximum value, and stopped before their training accuracy converged to its maximum. Due to its warm start, calibration ceaibbrttioovvi--ooreaaae for reference models.
Credit
0.921
0.751
Hep.
0.998
0.871
Adult
0.904
0.842
MNIST
0.990
0,989
CIFARI0
0.946
0.740
CIFAR100
0.938
0.373
ImageNet 0.773
0.637
Table Average train and test accuracy statistics for the target models.
via-forgetting generally used fewer training epochs
Threshold selection. When computing AUC and precision/recall metrics, we sweep over a range of values for the threshold and measure the resulting attack's 's FPR/TPR and precision/recall trade- offs. To find a threshold for optimal accuracy, we first split the public set examples in half again, and treat one half as members, with the rest as non-members. We can then choose the best threshold
5

---

Published as a conference paper at ICLR 2022
Loss
GN
Confidence
Dataset
Gap Atk.
Orig
Cal.
Cal,-F
Orig
Cal.
Cal.-F
Orig
Cal.
Cal.-F
Credit
0.549
0.542
0.648
0.594
0.513
0.579
0.559
0.501
0.553
0.543
Hep.
0.535
0,514
0,558
0.559
0.515
0.553
0.548
0.513
0.525
0.533
Adult
0.514
0.516
0.554
0.529
0.510
0.523
0.515
0.507
0.510
0.512
MNIST
0.506
0.505
0.519
0.509
0.504
0.514
0.509
0.503
0.510
0.505
CIFARIO
0.663
0.676
0.731
0.708
0.678
0.787
0.762
0.629
0.635
0.631
CIFARI00
0.854
0,911
0,903
0.886
0,912
0.924
0.912
0.852
0.707
0.760
ImageNet
0.536
0.547
0,557
0.551
0.540
0.511
0.520
0.544
0.515
0.512
CIFARIO (aug.)
0.603
0.603
0.690
0.674
0,603
0.666
0.707
0,560
0.568
0.575
100 (aug.)
0.784
0.812
0.844
0.831
0.811
0.856
0.843
0.679
0.672
0.647
Table AUC metric for score-based membership attacks before and after difficulty calibration. The standard deviation is up to 0.024 for Credit, up to 0.034 for Hep., up to 0.003 for MNIST and to 0.015 for all other datasets. Calibration (Cal.) consistently improves the AUC of attacks by signif- icant margin, while dlibration-via-forgeen for better efficiency.
(Cal.-F) sacrifices a modest amount of improvement
Loss
GN
Confidence
Dataset
Gap Atk.
Orig
Cal.
Cal.-F
Orig
Cal.
Cal.-F
Orig
Cal.
Cal.-F
Credit
0.589
0.617
0.618
0.577
0.569
0,573
0.567
0.557
0,554
0,555
Hep.
0.561
0.574
0.575
0.591
0.574
0.593
0.585
0.574
0.577
0.576
Adult
0.534
0.536
0.534
0.522
0.518
0.518
0.515
0.512
0.511
0.511
MNIST
0.506
0.508
0.513
0.509
0.508
0,511
0.508
0.507
0.508
0.506
CIFARIO
0.664
0.712
0.657
0.662
0.719
0.720
0.702
0.642
0.623
0.622
CIFARI00
0.854
0.911
0.829
0.862
0.915
0.876
0.898
0.820
0.731
0.711
ImageNet
0.536
0.542
0.545
0.542
0.538
0.518
0.521
0.540
0.541
0.553
CIFAR1O (aug-)
0.602
0.609
0.626
0.625
0.610
0.610
0.651
0.562
0.542
0.558
CIFAR 100 (aug.)
0.784
0.785
0.765
0.775
0.788
0.793
0.789
0.647
0.641
0.621
Table 3: Attack accuracy for score-based membership attacks before and after difficulty calibration. The standard deviation is up to 0.018 for Credit, up to 0.026 for Hep., up to 0.002 for MNIST and up to 0.005 for all other datasets. In almost all settings, calibrated attacks on-par with or better than their uncalibrated version in terms of accuracy.
based on membership inference accuracy for this simulated setup. Frequently, the optimal thresh- old for calibrated attacks was a value only slightly greater than 0 (e.g., 0.0001), which can be used as default threshold if necessary.
4.2 MAIN RESULTS
We first evaluate attacks that rely on continuous-valued output from the target model: loss score [qqaaioo 1). gradient norm (GN) score [Equation 2), and confidence score (Equation 3). We also consider the gap attack (Yeom et al. 2018) as a baseline to measure progress against. See Ap- pendix[C additional experiments using multiple calibration models and other membership scores.
AUC and accuracy. We compute two primary metrics: area under ROC curve (AUC) for evaluating the FPR/TPR trade-off, and attack accuracy. Table sssows the computed AUC for various attacks before and after difficulty calibration. Overall, calibrated attacks often perform drastically better than their uncalibrated counterparts, and in some instances the performance increase can exceed 0.10, such the German Credit and CIFARIO datascts. Table 3 shows the corresponding attack accuracy numbers. Evidently, calibration does not reduce accuracy in most cases, and sometimes brings a modest improvement.
We observe that the confidence score attacks under-perform other methods due to the model predict- ing non-member points incorrectly with high confidence. For this reason, calibrated confidence did not improve the attack performance on many datasets, as training points that were correctly classified on the target model but were incorrectly classified the reference model became indistinguishable.

---

Published as a conference paper at ICLR 2022
0.85
Gap Attack
Loss
0.80
Gap Attack
Loss
Cal. Loss
Cal. Loss
0.75
Accuracy
0.70
0.70
0.65
0.65
20
40
60
80
100
20
40
60
80 100
% Total Training Data
% Total Training
Figure 3: AUC (left) and accuracy (right) of the gap attack and calibrateed/nnaaiibraeed loss score attacks against target model trained on varying training set sizes on CIFARIO. Both AUC and accuracy increase as the training dataset size decreases due to more severe overfitting of the target model. Difficulty calibration can effectively leverage this improve the attack's s AUC and accuracy across all training set sizes.
Effect of data augmentation. The last two rows in Tables 2 and 3 show various attacks' perfor- mance on the CIFAR 10/100 target model trained with data augmentation. As expected, attacks are less effective on models with data augmentation due to reduced overfitting. Nevertheless, calibration still improves both the AUC and accuracy loss and gradient norm based attacks most of the time.
ROC and precision-recall curves. Figure shows the ROC and precision-recall curves for the loss score attack on CIFAR1O, which demonstrates the effect of difficulty calibration in closer detail. In the left plot, at low FPR values, the uncalibrated attack has close zero advantage in TPR, behaving essentially the same as random guessing. In stark contrast, calibrated attacks exhibit large difference in the FPR and TPR values. Furthermore, as shown in the right plot, calibrated attacks can detect group of member points with very high precision at low recall values, which is not possible for the uncalibrated counterpart. This pattern is also seen in the membership scores of calibrated and uncalibrated attacks shown in Figure I
We highlight another interesting phenomenon by computing the threshold t that achieves optimal accuracy for each attack, shown by the red dot in FFigure 2) For all attacks, this threshold lies towards the tail of the ROC and precision-recal curves, where the attack achieves high FPR and high recall but low precision. This observation concurs with prior work, which shows that existing attacks do not predict membership reliably when optimizing for accuracy (Rezaci & Liu 2021).
Label-only attacks. When the target model returns only a discrete label rather than a continuous- valued output, membership inference still be performed using label-only attacks. We show that difficulty calibration also improves such attacks. Table 4epprrts the performance of calibrated and uncalibrated label-only attacks using the HopSkipJump boundary distance method (Choquette-Chan Choo et al.) 2021). We excluded the MNIST and ImageNet datasets as the HopSkipJump attack required too many queries to successfully change the label. Similar to the results Tables 2 and 3] label- only attacks also benefit from difficulty calibration with increased AUC and improved or similar accuracy. However, the performance increase noticeably smaller for CIFAR10/10 compared to other score-based attacks.
4.3
A ABLATION STUDIES
Training set size. Since vulnerability to membership inference attack is linked to overfitting (Yeom) ct al. 2018), 3 the training set size of the target model is crucial factor attack performance. Indeed, Figure shows that for the gap attack and calibrated/uncalibbaeed loss score attacks, both AUC and accuracy increase when the training set size is reduced. More importantly, difficulty calibration effectively leverage overfitting to improve the attack's performance across the entire range of training set sizes.
Ratio of member to non-member samples. The experimental setup so far uses an equal number of member and non-member samples. In real world scenarios, the majority of samples are likely

---

Published as a conference paper at ICLR 2022
AUC
Accuracy
Privacy
Orig.
Cal.
Cal.-F
Orig.
Cal.
Cal.-F
Credit
0.552
0.640
0.599
0.616
0.620
0.638
Hep.
0.547
0.585
0.594
0.596
0.588
0.591
Adult
0.512
0.541
0.537
0,533
0.549
0.539
CIFARIO
0.660
0.686
0.620
0.665
0.666
0.623
CIFARTOO
0.849
0.841
0.856
0.847
0.841
0.844
Table 4: Average accuracy and AUC of the HopSkipJump label-only attack using min(n, 1000)
datapoints for each of the training and heldout sets, where is their original datasize. CIFAR models were trained without data augmentation.
Loss
1.0-
Loss
#Member Non-member
Cal. Loss
Cal. Loss
0.85
1:7
1:2
1:5
1:1
Accuraci
0.80
Predision
0.6
0.75
0.2
0.0
0.8
1.0
#Member:# Non-member
Recall
Figure 4: Accuracy (left) and precision-recall curve (right) of the loss score attack when changing the member to non-memee non-member ratio. At low ratios, the calibrated attack can still identify members with high precision, whereas the uncalibrated attack is unable do so despite high accuracy.
non-members, which drastically affects the accuracy and precision-recall trade-off of membership inference attacks. In the following experiment, we simulate this scenario by subsampling the mem- ber set for evaluation decrease the ratio of member to non-member samples.
Figure 4surws the accuracy and precision- -recall curves for different member to non- non-member ratios. In the left plot, as the number of non-member samples increases, attack accuracy also drastically increases. This is due to the threshold for optimal accuracy favoring high recall and low precision (see 2}. effectively performing "non- membership inference". contrast, the right plot shows that precision-recall trade-off worsens when the number of non-member samples increases, which reflects the fact that attacks are less reliable in the real world when most samples are non-members. Nevertheless, for the calibrated attack, it is possible to achieve high precision at low values of recall even for low member to non- non-member ratios.
Number of reference models. Results reported in Section 4.2 use single reference model for difficulty calibration. Figure 5 shows the effect of increasing the number of reference models on AUC and attack accuracy. To train each reference model, we subsample a different subset of the public dataset without replacement to create the shadow dataset. We then apply calibration-vis -va- forgetting to calibrate the loss score, while varying the shadow dataset size. Increasing the number of reference models and increasing the shadow dataset size strictly improve both AUC and accuracy. The difference between and 10 reference models is up 0.05 for AUC and 0.03 for accuracy.
Differential privacy is by far the most common defense against membership inference attacks, and provably limits the accuracy of these attacks (Yeom et al. (2018) when is small. With higher value of e, DP still empirically protects against uncalibrated membership inference attacks on average; however, calibrated attacks provide significant edge against outliers for high values of the privacy parameter c.
Figure 6 shows the ROC and reccallnn-recall curves of culibrated//mccliiraaed loss score attacks on CIFARTO models trained with DP-SGD. As expected, DP clearly reduces the effectiveness of attacks indicated by the gap between attacks against models with finite (blue and orange lines) and at-
8

---

Published as a conference paper at ICLR 2022
0.63
0.68
0.62
Accuracy
AUC
0.66
0.64
10 Datasize
Datasize
0.62
0.59
10
10
# of Reference Models
# of Reference Models
Figure 5: AUC (left) and attack accuracy (right) for the calibrated attack with varying number of reference models on CIFAR- 10. Using more reference models and training reference models with a higher shadow dataset size strictly improve performance at a cost of more computation.
1.0
1.0
True Positive Rate
0.8
0.8
0.6-
Precision
0.7
0.4
0.5
0.2
1000
0.4
Cal. Loss
No Privacy
0.0
False Positive Rate
Recall
Figure 6: ROC (left) and precision-recall (right) curves for calibratedd/nncalibrreeed loss attack on CIFARIO against differentially private models. For large values of DP parameter the calibrated attack can still successfully infer member samples with high precision low recall.
tacks against undefended model (dotted line). However, for 100 and 1000, calibrated attacks can still attain a non-trivial FPR-TPR and precision-rece recaal trade-off. Only when approaches do the AUC and accuracy approach that of uncalibrated attacks, and both converge to the effectiveness of random guessing.
A qualitative inspection of member images from CIFAR10 that successfully inferred high precision of > 0.9 for the DP model at 6 = 1000, but are not identified for < S 10, showed that vulnerable images were often atypical samples from their respective classes. They did not appear to be extreme outliers. A further manual inspection of examples did not suggest any general pattern for the most vulnerable points.
CONCLUSION
We showed that difficulty calibration is powerful post-processing technique for improving existing membership inference attacks. Most score-based attacks can greatly benefit from calibration, mak- ing the attack much more reliable low false positive rate and high precision regimes. Our study also showed that prior work that focused on evaluating attack accuracy do not reveal the full picture, and detailed analysis using the ROC and precision-recall curves is important for designing reliable membership inference attacks.
When given white-box access to the target model, calibration via forgetting can be an efficient alternative to training reference models from scratch. This alternative method also serves as an explicit connection between membership inference and the well-studied phenomenon of catastrophic forgetting in neural networks. We hope that future work further explore this connection and leverage new findings to improve the state-of of-the-art in membership inference attacks.

---

Published as a conference paper at ICLR 2022
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. CCS, 2016.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. arXiv preprint arXiv:2005. /4/65, 2020.
Nicholas Carlini, Chang Liu, Ülfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium, pp. 267-284. 2019.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, al. Extracting training data from large language models. arXiv preprint arXiv:20/2.07805, 2020.
Christopher A Choquette-Choe Florian Tramer, Nicholas Carlini, and Nicolas Papernot. Label-only membership inference attacks. In International Conference on Machine Learning, pp. 1964-1974. PMLR, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 248-255. leee, 2009.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. ics. uci. edu/ml
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pp. 265-284. Springer, 2006.
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confi- dence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC confer- ence on computer and communications security, pp. 1322-1333, 2015.
Matthew Fredrikson, Eric Lantz, Somesh Jha, Simon Lin, David Page, and Thomas Ristenpart. Privacy in pharmacogenetics: An end-to-end case study of personalized warfarin dosing. In 23rd USENIX Security Symposium, pp. 17-32, 2014.
lan Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empiri- investigation of catastrophic forgetting gradient-based neural networks. arXiv preprint 132iv:1312.6211, 2013.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp- 770-778, 2016.
Nils Homer, Szaboles Szelinger, Margot Redman, David Duggan, Waibhay Tembe, Jill Muehling, John V Pearson, Dietrich A Stephan, Stanley F Nelson, and David W Craig. Resolving indi- viduals contributing trace amounts of DNA to highly complex mixtures using high-density SNP genotyping microarrays. PLoS genetics, 4(8)=e100016) 2008
Matthew Jagielski, Jonathan Ullman, and Alina Oprea. Auditing differentially private machine learning: How private is private SGD? arXiv preprint arXiv:2006.07709, 2020.
Bargav Jayaraman, Lingxiao Wang, Katherine Knipmeyer, Quanquan Gu, and David Evans. Re- visiting membership inference under realistic assumptions. Proceedings on Privacy Enhancing Technologies. 2021(2). 2021.
Jinyuan Jia, Ahmed Salem, Michael Backes, Yang Zhang, and Neil Zhenqiang Gong. Memguard: Defending against black-box membership inference attacks via adversarial examples. In Pro- ceedings of the 2019 ACM SIGSAC conference on computer and communications security, pp- 259-274, 2019.
10

---

Published as a conference paper at ICLR 2022
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska Barkinkka et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3511-3526, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-222) 1998.
Klas Leino and Matt Fredrikson. Stolen memories: Leveraging model memorization for calibrated white-box membership inference. In 29th USENIX Security Symposium, pp. 1605-1622, 2020.
Zheng Li and Yang Zhang. Label-leaks: Membership inference attack with label. arXiv e-prints,
pp. arXiv-2007, 2020.
Yunhui Long, Vincent Bindschaedler, Lei Wang, Diyue Bu, Xiaofeng Wang, Haixu Tang, Carl A Gunter, and Kai Chen. Understanding membership inferences on well-generalized learning mod- els. arXiv preprint 1802. 04889, 2018.
Ilya Mironov. Rényi differential privacy. In 2017 IEEE 30th Computer Security Foundations Sym- posium (CSF), pp. 263-275. IEEE, 2017.
Πly Mironov, Kunal Talwar, and Li Zhang. Rényi differential privacy of the sampled gaussian mechanism. arXiv preprint arXiv: 1908. 10530, 2019.
Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive privacy analysis of deep learning: Passive and active chite-box inference attacks against centralized and federated learning. In 2019 IEEE symposium on security and privacy (SP), pp. 739-753. IEEE, 2019.
Milad Nasr, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, and Nicholas Carlini. Adver- sary instantiation: Lower bounds for differentially private machine learning. arXiv preprint arXiv:2101.04533 2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9, 2019.
Shahbaz Rezaei and Xin Liu. On the difficulty of membership inference attacks. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 7892-7900, 2021.
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Yann Ollivier, and Hervé Jégou. White- box vs black-box: Bayes optimal strategies for membership inference. In International Confer- ence on Machine Learning, pp. 5558-5567, PMLR, 2019.
Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. MI-leaks: Model and data independent membership inference attacks and defenses on machine learning models. arXiv preprint arXiv: /006vv1080..1/// 1806.0/26,, 2018.
Issei Sato and Hiroshi Nakagawa. Approximation analysis of stochastic gradient Langevin dynam- ics by using Fokker-Plamn Planck equation and Ito process. In International Conference Machine Learning. pp. 982-990. PMLR, 2014.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference at- tacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 3-18. IEEF, 2017
Liwei Song and Prateek Mittal. Systematic evaluation of privacy risks of machine learning models. In 30th USENIX Security Symposium (USENIX Security 21). pp. 2615-2632, 2021.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler. Yoshua Bengio, and Geoffrey Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint 18 2018.

---

Published as a conference paper at ICLR 2022
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learn- ing: Analyzing the connection to overfitting. In 2018 IEEE 31st Computer Security Foundations Symposium (CSF). pp. 268-282. IEEE, 2018.
Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj, Jessica Zhao, et al. Opacus: User-friendly differential privacy library pytorch. 2021.
Elias A Zerhouni and Elizabeth G Nabel. Protecting aggregate genomic data. Science, 2008
12

---

Published as a conference paper at ICLR 2022
A CALIBRATED GAP ATTACK
In this section, we explore the effects of using difficulty calibration on the gap attack Yeom et al. (2018). This analysis provides an intuitive explanation as why calibration improves the AUC of many attacks while retaining similar accuracy statistics in comparison to their uncalibrated counter- parts.
calib \\ target
correct
incorrect
calib
target
correct
incorrect
correct incorrect
Piest
0
correct
Phest
0
Prrin
Piest
Pirain
incorrect
Piest
Table 5: Performance of the calibrated and target models the train (left) and test (right) sets.
Let us assume that have two models, target and calib. The calibrated gap attack predicts that sample comes from the training set if the target model predicts it correctly and the calibration model predicts incorrectly.
Simplified analysis.
We simplify the analysis by making assumptions:
the train set, if the target model is wrong, the calibrated model is wrong too
on the test set, the models are either both right or both wrong about a sample
Table 5 shows that will predict all of the test elements correctly, and portion Prain Piest of the train set correctly. The accuracy of such method will thus be
(1 + Purann
Prest)
(7)
2
calib \ target
correct
incorrect calib target
correct
incorrect
correct
Prest E1
E1
correct incorrect
Prest
E2
incorrect
Parin
Piest +
1
Purin
Piest
Table 6:
Performance of the calibrated and target models on the train (left) and test (right) sets.
Full analysis. Now let us get rid of this assumption. In order to fill Table 6 we have equations: all probabilities should sum one, and the rows and columns where the model is correct should sum to Prain (respectively Pres). Given that we have four unknowns, this leaves only one undetermined quantity for each table, ei and E2. The accuracy of the calibrated gap attack is thus
(1 Purin
Ptest + E1).
Based on experiments using the calibrated gap attack on CIFARIO and CIFAR100 with and without data augmentation, we observed that is often very small, with values ranging from 0% 2%. On the other hand, e2 is often larger at around 8% 12%.
Accuracy: As implied by Equation g the accuracy of the calibrated gap attack is then lower than the uncalibrated gap attack as <2 exceeds é1- However, note that accuracy of the uncalibrated gap attack can be trivially recovered in this context by simply predicting that all points predicted correctly by the target model are members, as opposed to only those predicted correctly by the target model and incorrectly by the calibration model.
AUC: On the other hand, the calibrated gap attack improves the precision- reccall trade-off of the gap attack. As shown in Figures 7 and 8 calibration makes it possible identify subset of member points with a significantly lower false positive rate. For this reason, the AUC of the gap attack on CIFAR10 10 improved from average of 0.66 0.71.
13

---

Published as a conference paper at ICLR 2022
Gap
Gap
12000
Member Member
8000
Member
Member
Frequency
6000
8000
6000
4000
2000
Target (Com)
Target linoorr.)
Target (Conc)
[Croct(Com)) Callb (Com.
Target (Incom:)
Target lincom) Calib (ConT)
Callb (Inoor.)
Calle (Inoo..)
Figure 7: Gap Attack vs. Calibrated Gap Attack score distributions. Target (Corr.) indicates that the predicted class for the datapoint was correct using the target model. Calib (Incorr. indicates that the predicted class was incorrectly predicted on the calibration model. Note that the two left-most categories in the calibrated plot correspond to division of the left hand category in the original gap attack plot. The same relationship exists between the right-most categories. The left-most category in the calibrated gap attack shows the improvement in TPR that be attained via calibration, resulting in the increased AUC seen in Figure g
1.0
0.8
0.6
TPR
0.4
Gap
Calibrated Gap
0.0
0.0
0.2
0.6
0.8
FPR
Figure 8: The ROC curves for the gap attack and calibrated gap attack. Note that the area under the ROC curve (AUC) is higher for the calibrated attack due to the increased TPR for low FPR.
B GENERALIZATION AND MEMBERSHIP INFERENCE
Calibrated membership inference looks at the quantity E(0,, =) E(0, z) where is the target model, to the calibration model and sample included in the training of 0. Generalization, hand, is concerned with L(O, 2²) E(O, 2) where z' is sample that does not belong to the training of 0. We argue that ((0,, 2) and ((O, z') are different quantities but they both represent the loss that is incurred on sample outside of the training set and should thus behave similarly statistically.
the other
C ADDITIONAL EXPERIMENTS
Tables 7 and demonstrate the impact of using multiple calibration models instead of a single calibration model. Tables / and 10 show the effects of difficulty calibration on other membership scores not included in the main body of this paper. Specifically, we considered the entropy and modified entropy scores proposed by [ong & Mittal (202T) as well as the Merlin and Morgan attacks proposed by Jayaraman et al. (2021). As suggested by Song & Mittal (202T). Figure 9 shows that their entropy metries are closely related the confidence and loss scores respectively. This relationship explains why the AUC and accuracy results show the same behaviours the confidence and loss scores under calibration.
The Merlin and Morgan attacks target high positive predictive value (PPV) as opposed to attack accuracy. For this reason, their accuracy and AUC results are lower than that of other attacks. In these attacks, the member and mommerrerr distributions except for the small number of points identified with very high PPV. In particular, the Morgan attack outputs binary values chosen maximize PPV. When successful this results in very skewed
Merlin and Morgan:
often indistinguishable
14

---

Published as a conference paper at ICLR 2022
distribution identifying a few datapoints with PPV close to 1 (and value 1) and assigning all other points value O. This results in accuracy and AUC results close to that of random guessing. Table [T] reports the PPV of uncalibrated and calibrated versions of the Merlin and Morgan attacks, in many cases calibration improves the maximum achieved PPV.
Loss
GN
Confidence
Dataset
Gap Atk.
Orig
Cal.
Cal.-F
Orig
Cal.
Cal.-F
Orig
Cal.
Cal.-F
Credit
0.589
0.542
0.668
0.650
0.513
0.597
0.596
0.501
0.569
0.550
Hep.
0.561
0,514
0.562
0.534
0.515
0.557
0.568
0.513
0.582
0.573
Adult
0.534
0.516
0.559
0.539
0.510
0.527
0.520
0.507
0.518
0.513
MNIST
0.506
0.505
0.527
0.510
0.504
0.525
0.517
0.503
0.517
0.509
CIFARIO
0.663
0.676
0.739
0.696
0.678
0.796
0.759
0.629
0.658
0.646
CIFARI00
0.854
0.911
0.914
0.916
0.912
0.933
0.918
0.852
0.721
0.746
CIFARTO TO (aug-)
0.603
0.603
0.704
0.693
0.603
0.746
0.725
0.560
0.581
0.578
CIFAR 100 (aug.)
0.784
0.812
0.878
0.873
0.811
0.891
0.877
0.679
0.665
0.635
Table 7: AUC metric for score-based membership attacks before and after difficulty calibration with 10 calibration models. Calibration (Cal.) consistently improves the AUC of attacks by significant margin, while calibration-via-foorgttnn (Cal.-F) sacrifices a modest amount of improvement for better efficiency.
Loss
GN
Conf
Datasel
Gap Atk.
Orig
Cal.
Cal.-F
Orig
Cal.
Cal.-F
Orig
Cal.
Cal.-F
Credit
0.589
0.617
0.624
0.6005
0.569
0.586
0.584
0.557
0.563
0.553
Hep.
0.561
0.574
0.587
0.574
0.574
0.590
0.593
0.574
0.601
0.594
Adult
0.534
0.536
0.537
0.525
0.518
0,520
0.516
0.512
0.514
0.511
MNIST
0.506
0,508
0.517
0.510
0.508
0,516
0.512
0.507
0,513
0.508
CIFARIO
0.664
0.712
0.657
0.663
0.719
0.707
0.703
0.642
0.614
0.623
CIFARTOR 100
0.854
0.911
0.832
0.846
0.915
0.885
0.896
0.820
0.675
0.691
CIFARIO (aug.)
0.602
0.609
0.631
0.639
0.610
0.668
0.657
0.562
0.567
0.564
CIFAR100 100 (aug.)
0.784
0.785
0.791
0.801
0.788
0.831
0.822
0.647
0.616
0.606
Table 8: Attack accuracy for score-based membership attacks before and after difficulty calibration with 10 calibration models. In almost all settings, calibrated attacks are on-par with or better than their uncalibrated version in terms of accuracy.
Entropy
Modified Entropy
Merlin
Morgan
Dataset
Gap
Orig.
Cal.
Cal.-F
Orig.
Cal.
Cal.-F
Orig.
Cal.
Cal.FF
Orig.
Cal.
Cal.-F
Credit
0,588
0.525
0.591
0.597
0.562
0.637
0.621
0.514
0.523
0.527
0.505
0.502
0.508
Hep.
0.544
0.516
0.520
0.534
0.534
0.581
0.587
0.472
0.476
0.484
0.524
0.544
0.526
Adult
0,514
0.509
0,512
0.513
0.519
0.557
0.530
0,500
0.502
0.507
0,500
0,500
0.500
MNIST
0.506
0.507
0.512
0.510
0.510
0.512
0,513
0.497
0.503
0.501
0.503
0.502
0.503
CIFAR10
0.662
0.627
0.634
0.635
0.675
0.730
0.710
0.506
0,504
0.504
0,500
0,500
0.500
CIFARIOK
0.858
0.849
0.701
0.750
0.910
0.912
0.902
0.373
0.530
0.510
0.500
0.500
0.500
CIFARTO (aug.)
0.602
0.563
0.598
0,581
0.603
0.691
0.688
0,500
0.516
0.514
0.555
0.500
0.500
CIFAR 100 (ang.)
0.785
0.664
0.700
0.706
0.798
0.851
0.868
0.495
0.507
0.509
0.644
0.500
0.500
Table 9: AUC results. As suggested by (Song Mittal (2021). Figure P shows that the entropy and modified entropy metrics are closely related to the confidence and loss scores respectively. This relationship explains why the AUC results are similar to those of the confidence and loss scores under calibration. The Merlin and Morgan attacks were proposed to maximize positive predictive value (PPV) instead of accuracy, which is reflected in their results. Note that there was a large standard deviation for the Hepatitis dataset using the Merlin and Morgan attacks of up 0.09 making the results difficult to compare. For the Morgan attack, a value close to 0.500 results from the attack identifying a small number of points with FPR- 0 and achieving the attack's objective of maximizing PPV. Higher values indicate when this was not possible, and the intended behaviour was then improved by calibration due to the creation of high precision region e.g. see CIFARI (100()g)))))))) and Table II
15

---

Published as a conference paper at ICLR 2022
Entropy
Modified Entropy
Merlin
Morgan
Dataset
Gap
Orig.
Cal.
Cal.-F
Orig.
Cal.
Cal--F
Orig.
Cal.
Cal.-F
Orig.
Cal.
Cal--F
Credit
0.588
0.525
0.591
0.597
0.622
0.633
0.629
0.537
0.537
0.543
0.505
0,502
0.508
Hep.
0.544
0.516
0.520
0.534
0.597
0.597
0.544
0.571
0.524
0.544
Adult
0.514
0,509
0,512
0.513
0.537
D.535
0.533
0,505
0.505
0.507
0.500
01500)
0.500
MNIST
0,506
0.507
0.512
0,510
0.510
0.509
0.509
0,502
0.505
0.504
0.503
0,502
0.503
CIFARIO
0.662
0.627
0.634
0.635
0.710
0.661
0.664
0.515
0.504
0.511
0.500
0.500
0.500
CIFAR 100
0.858
0.849
0.701
0.750
0.910
0.850
0.839
0.501
0.532
0.525
0.500
0,500
0.500
CIFARIO (ang-)
0.602
0.563
0.581
0.608
0.501
0.516
0.555
0.500
CIFAR 100 (aug.)
0.785
0.664
0.700
0.706
0.771
0.801
0.793
0,502
0.522
0,524
0.644
0,500
0.500
Table 10: Accuracy results. As suggested by (Sog & Mittal (2021), J. Figure shows that the entropy and modified entropy metrics are closely related to the confidence and loss scores respectively. This relationship explains why the accuracy results are similar to those of the confidence and loss scores under calibration. The Merlin and Morgan attacks were proposed maximize positive predictive value (PPV) instead of accuracy, which is reflected in their results. Note that there was a large standard deviation for the Hepatitis dataset using the Merlin and Morgan attacks up to 0.07 making the results difficult to compare. For the Morgan attack, value close to 0.500 results from the attack identifying a small number of points with FPR = 0 and achieving the attack's 's objective of maximizing PPV. Higher values indicate when this not possible, and the intended behaviour was then improved by calibration due to the creation a high precision region e.g. see CIFAR (00(ag)) and Table 11
Dataset
Merlin Cal.
Morgan Cal.
Orig.
Cal.-F
Cal.-F
Credit
0.92 (2.80= 2.04)
0.71 (1.0=1
(2002..)))
0.64 (0.4=0.8)
0.80 (0)8
0.92 (1:40.8)))
Hop.
(1.0+..22)
1.0 (1.000000)
0.81 (1.0 0.89)
(2.7522222055) (.(5558.))7)
(1.00.00
Adult
0,72 (0.2 =0.40)
0.76 = 1.55)
0.77 0.49)
0.78 (0.2=004)
0.80 (0.6=0049)
0.80 (1.2+ 1.16)
MNIST
0.51 (0.0 10.0)
0.64 (0.0 0.0)
0.60 (0.0 10.0)
0.52 (0.0:0.0)
0.64 (0.0::0.0
0.66 (0.0=:00)
CIFARIO
0.83 01.2 11600
0.79 (0.6 = 0,55)
0.78 (0.4 0.55)
0.83 (1.2 =1 (0)
1.0 (1.410.88
1.0 (288.224)
CIFAR100
12.4 11.14
L.0 (1.0 0.0)
0.94 (0.6 0.55)
(2.4+11114
1.0 (4.0+11667)
(3.02111,.
CIFARIO (aug.)
0.30 (0.0 = 0.D)
0.85 (0:0210.0)
0.84 E117)
0.58 (000 10.0)
(1.0000))
1. (1,2:020
CIFAR100 (mug)
0.50 (0.0 0.0)
1.0 (2.0+-1.55)
(1.44.00
0.72 (0.0 0.0)
0.98 (1.2 +0998)
(3.4 1.36)
Table 11: Positive Predictive Value (PPV) results and number of points identified with FPR 0. In many cases calibration improves PPV. The thresholding technique for the attacks maximized PPV, as used by Jayaraman et al. 22021)
Training Points Cifar10
Heldout Points Cifar10
1.5
Loss
1.0-
Loss
20
0.0
0.0
0.1
0.2
0.3
0.4
0.5
10
20
30
40
50
Modified Entropy
Modified Entropy
Training Points Cifar10
Heldout Points Cifar10
0.0
0.0
Confidence
Canfidence
-0.4
-0.5
-0.6
-1.0
0.0
0.5 Entropy
1.0
0.0
0.5
1.0
1.5
Entropy
Figure 9; Comparison of entropy and modified entropy with confidence and loss scores respectively for CIFARIO without data augmentation. Song & Mittal (202T) note that entropy is similar to confidence. Modified entropy is similar to cross entropy loss.
16


Reviews:
[{"review_id": "nkXKmHpPu1r", "reviewer": "Reviewer_32Pd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: Controlling false positives of membership attacks is important for practical membership inference scenarios. Research efforts along  this direction should be encouraged. \n\nWeakness: \nThe theoretical rationality of the proposed calibration method is not provided in the study. Though it is mentioned that the proposed method is associated with posterior inference, more formal study establishing the link is still needed. The discussion at the end of Section.3 is not convincing. \n", "summary_of_the_paper": "This work proposes a simple output calibration method to suppress false positive errors of membership attacks. The core idea is to define a calibrated decision score as the difference between the original membership score and the expected membership score trained by a randomized training algorithm using a shadow dataset (a dataset following the same distribution as the true training data set). The calibrated membership decision score can provide a lower false positive rate than the original membership score. \n\n ", "summary_of_the_review": "The theoretical reasoning the performance of the proposed calibration method is not provided yet necessary. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Xnuc8ECSAN6", "reviewer": "Reviewer_2TZc", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a natural and simple idea to improve membership inference attacks: train a few reference models on the same distribution, and then use these as a \"prior\" for the distribution of losses of non-members.\nThis idea is not entirely new: it was proposed and analyzed both in Sablayrolles et al. and Long et al. which the paper references. It would be helpful to clarify this paper's contribution with respect to these two papers. In particular, it seems to me that the attack proposed in this paper is strictly *weaker* than the one of Sablayrolles et al.\nIn \"Connection to posterior inference\", this paper claims that the proposed approach is equivalent to that of Sablayrolles et al. But this isn't quite true: Sablayrolles et al. train reference models on subsamples of the data so that they can approximate both the average loss of an example when it is a non-member *and when it is a member*. Their attack then essentially determines whether an example's observed loss is closer to the average non-member loss or the average member loss of that sample.\nThe approach in this paper is similar, but only considers reference models that do not contain a victim point. So this attack is necessarily weaker.\n\nWhile the attack idea thus doesn't seem particularly novel, the analysis and experiments are however very thorough and nicely complement the more theoretical paper of Sablayrolles et al. The ablation studies are overall very solid and give a good idea of what factors influence the attack's strength. A few minor comments:\n- Some of the model accuracies in Table 1 are fairly low (e.g., CIFAR-10/100). Do you get similar results if you use a better model or training setup?\n- The finding that data augmentation leads to less privacy leakage is contrary to what was found by Choquette-Choo et al. Did you also use augmentations for the attack as in that paper?\n- Figure 4 (right) and Figure 6 are hard to read as they display a 3-dimensional relationship. Instead of a full PR curve, it might be better to extract a single scalar for each experiment (e.g., the AUC, or the precision at a recall of 50% or something like that) and then plot that scalar for varying ratios in Figure 4, and for varying epsilon in Figure 6.\n- The AUC curves in Figure 5 do not seem to reach a plateau. Can you further improve it by training more reference models? ", "summary_of_the_paper": "This paper proposes to improve membership inference attacks by calibrating the attack threshold on each example's hardness.", "summary_of_the_review": "A nice paper with an idea that is not entirely novel but with a rigorous empirical analysis.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Mdyn6BNwcL8", "reviewer": "Reviewer_QzDd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "** Strength ** \n- The paper is overall well written and easy to understand. The contributions are clearly stated and explained. \n- The main contribution is the application of difficulty calibration (inspired by Long et al., 2018) to MIA. Wide body of experimental  results using different datasets (only images though) which mostly support the proposed approach by showing favorable results. \n- The analysis with different member-non member ratios is definitely interesting and well done.\n\n** Weaknesses** \n* It would be highly beneficial to better and more formally support the intuition about easy-to-predict  non-member samples page 3 section 3. Indeed, the intuition lacks a deep analysis and is only supported by most of the experimental (empirical) results. Fors instance, several fundamental questions remain unsolved such us:\n- Are there further connection to the generalization capabilities of the network ? Several results have been mentioned in the literature but this aspect has not been discussed in this work. \n- It is not clear of the condition to ensure the validity of expression (6) are satisfied. This assumption should be justified and further investigated. \n- What about possible context with the accuracy of the network? Calibration and accuracy are two different concepts but they may also be related to the potential weakness to MIA. \n", "summary_of_the_paper": "This paper focuses on the analysis of membership inference attacks (MIAs), in  particular addressing the problem of high false positive rate (FPR) that still affects the  state-of-the-art solutions. The authors suggest an approach to MIA that is based on difficulty calibration, i.e., a method where the predicted membership score is adjusted  to the difficulty of obtaining the correct classification for the target sample, in order to reduce the FPR. As illustrated in Figure 1, previous work on the score-based membership inference attack is highly unreliable for separating easy-to-predict non-members from hard-to-predict members, as both can achieve high membership scores. This technique appears to be particularly beneficial in helping to distinguish between  member and non-member samples that are overrepresented in the data distribution. The authors support the proposed method with a large body of mostly favorable experimental results.", "summary_of_the_review": "In order to better understand the improvement coming with the proposed method, it would be useful to have a deeper understanding of the reason for which it is indeed beneficial. More details on the differences between the models f and g are to be provided: different random\ninitializations are probably involved as well as different batch shuffling. I guess that mainly the g models are going to converge faster on the training data, presenting more variance in the performance for non member data, which in turn makes it easier to find an effective threshold for the MIA problem. However, this does not seem effective on some datasets. Could the authors investigate those issues and shed more light on the relation between the type of data and the success/insuccess of their method?", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4pqtteDfCT-", "reviewer": "Reviewer_QoUg", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[strengths]\n1. The idea of avoiding high false positive rates is indeed important for membership inference research.\n2. Experimental results on multiple datasets indicate significant improvement than prior approaches.\n\n[weaknesses]\n1. The idea of difficulty calibration is not new. As the authors already pointed out, Long et al. (2018) and Carlini et al. (2020) already used similar technique to compare the difference (or ratio) between target models and reference models. What are new things in this paper?\n2. The paper fails to compare with existing attacks which are also designed to decrease false positive rates, specifically, the following paper. --- Jayarama et al., “Revisiting Membership Inference under Realistic Assumptions”, PETS 2021.\n3. Besides evaluated metrics (loss, gradient norm, confidence), recent works also leverage entropy for membership inference. It would be great if the authors can also include entropy metric as well.  --- Song & Mittal, “Systematic Evaluation of Privacy Risks of Machine Learning Models”, USENIX 2021.\n", "summary_of_the_paper": "The paper aims to solve the issue of high false positive rates in modern membership inference attack methods by computing the difference in model prediction metrics (e.g., loss, confidence) between the target model and a reference model. The intuition is that “easy-to-predict non-members” will have a small difference between the target and the reference model, while “hard-to-predict members” will have a large difference. Experimental results on multiple datasets also validate the effectiveness of proposed method.", "summary_of_the_review": "I like this paper’s focus on reducing high false positive rates. However, unless the authors clearly explain the difference with prior works (Long et al. (2018), Carlini et al. (2020)) and show the advantage over other attacks (Jayarama et al. (2021)), I suggest not accepting this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a technique to improve membership inference attacks by
carefully applying "difficulty calibration" to improve the attack success
rate. The reviewers are split on this paper. They all generally agree on
the facts: the paper introduces a (somewhat) new technique and performs a
solid evaluation, but the novelty on top of prior work isn't all that high.

On the whole I believe this paper should be accepted. This paper has identified
a very clear problem with existing attacks (poor performance at low false
positive rate) and has carefully developed a way to improve on this metric. A
thorough evaluation has convinced the reviewers that this paper does what it
set out to do.

It comes down to a question of novelty then. And here the question is this:
does someone who reads this paper learn something new that wasn't obvious
before? Part of this can be novelty in the method---and I agree with the
reviewers that this paper lacks novely in the method. However the paper does
not lack novelty in the ideas on the whole. While Long et al., Sablayrolles et
al., and Carlini et al. do all use some kind of low false positive rate
evaluation and calibrate for low loss, none of these papers actually go out
of their way to evaluate this fact explicitly. And so even if this paper
had no technical contribution at all, the simple measurement study in and
of itself would be a useful insight.

Machine learning research at present focuses fairly heavily on novelty
of the techniques. While this is good, it's also important to go back and
actually evaluate what we have. That's what this paper does, and it does
it well enough to be worth accepting.

The paper would definitely be improved by following some of the advice of
the reviewers and including comparisons to prior work (e.g., especially
clarifying the relationship to Sablayrolles et al. and if it is true that
this attack is a simplification of this prior one and is thus less effective)
and I hope the authors will take the opportunity to do this.
Decision: Accept (Poster)