On the Importance of Difficulty Calibration in Membership Inference Attacks

The vulnerability of machine learning models to membership inference attacks has received much attention in recent years . However , existing attacks mostly remain impractical due to having high false positive rates , where non-member samples are often erroneously predicted as members . This type of error makes the predicted membership signal unreliable , especially since most samples are non-members in real world applications . In this work , we argue that membership inference attacks can benefit drastically from difficulty calibration , where an attack ’ s predicted membership score is adjusted to the difficulty of correctly classifying the target sample . We show that difficulty calibration can significantly reduce the false positive rate of a variety of existing attacks without a loss in accuracy . 1 INTRODUCTION . Modern applications of machine learning often involve training models on sensitive data such as health records and personal information . Unfortunately , recent studies have found that these models can memorize their training data to a large extent , compromising the privacy of participants in the training dataset ( Fredrikson et al. , 2014 ; 2015 ; Shokri et al. , 2017 ; Carlini et al. , 2019 ) . One prominent category of privacy attacks against machine learning is the so-called membership inference attack ( Shokri et al. , 2017 ; Yeom et al. , 2018 ) , where the adversary aims to infer the participation of an individual in the target model ’ s training set . Such an attack is undoubtedly damaging when the status of participation itself is considered sensitive , e.g. , if the training dataset consists of health records of cancer patients . Moreover , the ability to infer membership can be viewed as a lower bound for the model ’ s degree of memorization ( Yeom et al. , 2018 ) , which is useful in itself as an empirical quantifier of privacy loss ( Jagielski et al. , 2020 ; Nasr et al. , 2021 ) . The efficacy of membership inference attacks has been improved significantly since the first attempts ( Salem et al. , 2018 ; Sablayrolles et al. , 2019 ; Leino & Fredrikson , 2020 ) . However , the most common evaluation metric , attack accuracy , overlooks the crucial factor of the false positive rate ( FPR ) of non-members ( Rezaei & Liu , 2021 ) . Indeed , most attacks operate by first defining a membership score s ( h , z ) for the model h and a target input-label pair z = ( x , y ) that measures how much h memorized the sample z . The attack subsequently selects a threshold τ and predicts that z is a member if and only if s ( h , z ) > τ . For typical choices of the membership scoring function , there is usually a large overlap in the distribution of s ( h , z ) between members and non-members ( see Figure 1a ) . As a result , an attack that determines the membership of z by thresholding on s ( h , z ) will have a high FPR . This drawback renders most existing attacks unreliable since the vast majority of samples likely belong to the non-member class . In this study , we identify the lack of difficulty calibration as a core contributor to the high FPR of existing attacks . Specifically , a non-member sample may have a high membership score simply because it is over-represented in the data distribution . Consequently , an attack that determines a sample is likely to be a member due to having a high score will inevitably fail on these over-represented samples . To remedy this problem , we make the acute observation that if the membership score is measured in comparison to a typical model trained on data drawn from the same data distribution , this difference in behavior can serve as a much more reliable membership signal . Indeed , Figure 1b shows the histogram of scores s ( h , z ) after difficulty calibration , where the member and non-member scores have significantly better separation and low FPR is now attainable . We propose difficulty calibration as a general technique for improving score-based membership inference attacks , and modify several membership scoring functions such as confidence ( Salem et al. , 2018 ) , loss ( Yeom et al. , 2018 ) , and gradient norm ( Nasr et al. , 2019 ) to construct their calibrated variants . Evaluated on a comprehensive suite of benchmark datasets , we show that calibrated attacks achieve a significantly lower FPR compared to prior work . In particular , we measure the trade-off between true positives and false positives using the area under ROC curve ( AUC ) metric , and show that difficulty calibration drastically improves this trade-off compared to uncalibrated attacks , by up to 0.10 AUC on common ML benchmarks . In addition , calibrated attacks also drastically improve the precision-recall trade-off , while remaining on-par with or better than uncalibrated attacks in terms attack accuracy . Our results suggest that it may be important for future work to apply difficulty calibration to design more reliable and practical membership inference attacks . 2 BACKGROUND . Membership inference attacks are concerned with determining whether a given sample was part of a target model ’ s training set . Homer et al . ( 2008 ) showed in their pioneering study that it is possible to infer an individual ’ s presence in a complex genomic DNA mixture , which led to increased caution around releases of DNA data ( Zerhouni & Nabel , 2008 ) . Recent interest in member inference attacks was sparked by the work of Shokri et al . ( 2017 ) , who introduced the shadow models method : An adversary trains substitute models ( called shadow models ) to mimic the behavior of the model under attack ( called the target model ) . The adversary then observes the behavior of the shadow models when exposed to member and non-member samples , and uses this observation to train an attack meta-model for predicting membership on any given sample . Shokri et al . ( 2017 ) evaluated this attack on ML models trained on cloud APIs , and showed that the shadow models approach attains high levels of accuracy . Score-based attacks . Yeom et al . ( 2018 ) discovered a connection between membership inference attacks and overfitting , arguing that in principle , attack accuracy can be determined by how much the target model memorizes ( or overfits to ) the given sample ( x , y ) . This discovery led to a series of work on quantifying the degree of memorization via membership scores , which can be used for predicting that the given sample is a member when the score is high . The membership score can be computed using the loss ( Yeom et al. , 2018 ) , the gradient norm ( GN ) ( Nasr et al. , 2019 ) or the confidence of the model ’ s prediction ( Salem et al. , 2018 ) , often yielding state-of-the-art results ( Salem et al. , 2018 ; Choquette-Choo et al. , 2021 ) . We define these scores for the cross-entropy loss ` : sloss ( h , ( x , y ) ) = − ` ( h ( x ) , y ) : = log ( h ( x ) y ) , ( 1 ) sGN ( h , ( x , y ) ) = −‖∇ ` ( h ( x ) , y ) ‖2 , ( 2 ) sconfidence ( h , ( x , y ) ) = −max y′ ` ( h ( x ) , y′ ) = max i log ( h ( x ) i ) . ( 3 ) Label-only attacks . The above score-based attacks rely on continuous-valued predictions from the model in order to define the membership score . To counter these attacks , prior work considered obfuscating the model ’ s output by returning only the top label or modifying the predicted values ( Shokri et al. , 2017 ; Jia et al. , 2019 ) . However , subsequent studies showed that even with only hard-label output , it is still possible to define membership scores that attain close to state-of-the-art accuracy ( Li & Zhang , 2020 ; Choquette-Choo et al. , 2021 ) . High-precision attacks . Various forms of difficulty calibration have been considered in the context of high-precision attacks . Long et al . ( 2018 ) selected samples that differ the most in loss between the target and a set of reference models , and showed that the resulting attack has high precision even for well-generalized target models . Carlini et al . ( 2020 ) showed that such privacy attacks are also possible on large-scale language models such as GPT-2 ( Radford et al. , 2019 ) . Their attack operates by first generating a large number of sentences from the language model and then ranking these sentences by the ( log ) perplexity , with lower perplexity indicating a more plausible memorized training sample . These perplexity values are then divided by either the z-lib entropy or the perplexity given by a smaller language model to account for the sentence ’ s rarity . In effect , only rare sentences with low perplexity can minimize the resulting score . Both attacks leverage a form of difficulty calibration by comparing the target model ’ s loss with that of reference models , predicting membership only when the difference ( or ratio ) is large . Differential privacy as a mitigation . Differential privacy ( Dwork et al. , 2006 ) ( DP ) is a powerful mathematical framework for privacy-preserving data analysis . A randomized algorithmM satisfies ( , δ ) -differential privacy if , given any two datasets D and D′ that differ in at most one sample , and for any subset R of the output space , we have : P ( M ( D ) ∈ R ) ≤ exp ( ) P ( M ( D′ ) ∈ R ) + δ . ( 4 ) Under mild assumptions , DP provably protects against a variety of privacy attacks , in particular membership inference ( Yeom et al. , 2018 ; Sablayrolles et al. , 2019 ) . Abadi et al . ( 2016 ) proposed a differentially private version of stochastic gradient descent ( SGD ) , called DP-SGD , to enable differentially private training of generic ML models . Their analysis has further been refined ( Mironov , 2017 ; Mironov et al. , 2019 ) and has been shown experimentally to be tight ( Nasr et al. , 2021 ) . 3 DIFFICULTY CALIBRATION . As depicted in Figure 1 , prior works on score-based membership inference attack are very unreliable for separating easy-to-predict non-members from hard-to-predict members since both can attain a high membership score . We argue that a simple modification to the score , which we call difficulty calibration , can drastically improve the attack ’ s reliability . This approach has been applied to the loss score for high-precision attack against well-generalized models ( Long et al. , 2018 ) . Let s ( h , ( x , y ) ) be the membership score , where higher score indicates a stronger signal that the sample ( x , y ) is a member . Instead of computing the membership score only on the target model h , we sample multiple “ typical ” models trained on the same data distribution as h and evaluate the membership score on these models . Doing so calibrates s ( h , ( x , y ) ) to the difficulty of the sample . For instance , if ( x , y ) is an easy-to-predict non-member , then s ( h , ( x , y ) ) is high but typical models g would also perform well on ( x , y ) , hence s ( g , ( x , y ) ) would also be high . The small gap between s ( h , ( x , y ) ) and s ( g , ( x , y ) ) suggests that ( x , y ) is likely a non-member . Formally , let Dshadow be a shadow dataset drawn from the same data distribution as the training set of h , and let A be a randomized training algorithm that samples from a distribution over models trained on Dshadow . We define the calibrated score as : scal ( h , ( x , y ) ) = s ( h , ( x , y ) ) − Eg←A ( Dshadow ) [ s ( g , ( x , y ) ) ] , ( 5 ) where the expectation is approximated by sampling one or more models from A ( Dshadow ) . The membership inference attack proceeds by thresholding on the score scal ( h , ( x , y ) ) . Efficient difficulty calibration via forgetting . Faithfully executing the difficulty calibration algorithm in Equation 5 requires training multiple models g using the randomized training algorithm A . This can be prohibitively expensive if the model is large , which is the case with modern neural networks such as large-scale transformers ( Brown et al. , 2020 ) . An alternative , more efficient approach if given white-box access to the target model h is to warmstart training on h. In fact , this approach establishes an explicit connection between membership inference attacks and catastrophic forgetting in neural networks ( Goodfellow et al. , 2013 ; Kirkpatrick et al. , 2017 ) . Toneva et al . ( 2018 ) observed that when a trained model resumes training on a separate dataset drawn from the same data distribution , it is very likely to “ forget ” about its original training samples . The training examples that are most likely forgotten are ones with abnormal or distinctive features , and this set of examples is stable across different training runs . We leverage this phenomenon to define a more efficient variant of difficulty calibration . Connection to posterior inference . Under a simplifying assumption of the distribution A ( D ) induced by the training algorithm , it is possible to derive difficulty calibration as an approximation to an optimal white-box attack . Sablayrolles et al . ( 2019 ) proposed a formal analysis of membership inference , where they assumed that the density function of A ( D ) has the form : p ( h ; D ) ∝ e− ∑ ( x , y ) ∈D ` ( h ( x ) , y ) , ( 6 ) which can be formally derived for optimization using SGD under certain conditions ( Sato & Nakagawa , 2014 ) . The implied threat model in differential privacy assumes that the adversary has knowledge that the training dataset is either D or D′ = D ∪ { ( x , y ) } for some inference target ( x , y ) , and D is known to the adversary . It can be shown that the Bayes-optimal attack strategy under this setting is to threshold on the calibrated score in Equation 5 , where s = sloss andD = Dshadow ( Sablayrolles et al. , 2019 ) . This analysis can be generalized to any scoring function s ( h , ( x , y ) ) , such as gradient norm ( Equation 2 ) and confidence ( Equation 3 ) .

Summary: This work proposes a simple output calibration method to suppress false positive errors of membership attacks. The core idea is to define a calibrated decision score as the difference between the original membership score and the expected membership score trained by a randomized training algorithm using a shadow dataset (a dataset following the same distribution as the true training data set). The calibrated membership decision score can provide a lower false positive rate than the original membership score.