On the Importance of Difficulty Calibration in Membership Inference Attacks

The vulnerability of machine learning models to membership inference attacks has received much attention in recent years . However , existing attacks mostly remain impractical due to having high false positive rates , where non-member samples are often erroneously predicted as members . This type of error makes the predicted membership signal unreliable , especially since most samples are non-members in real world applications . In this work , we argue that membership inference attacks can benefit drastically from difficulty calibration , where an attack ’ s predicted membership score is adjusted to the difficulty of correctly classifying the target sample . We show that difficulty calibration can significantly reduce the false positive rate of a variety of existing attacks without a loss in accuracy . 1 INTRODUCTION . Modern applications of machine learning often involve training models on sensitive data such as health records and personal information . Unfortunately , recent studies have found that these models can memorize their training data to a large extent , compromising the privacy of participants in the training dataset ( Fredrikson et al. , 2014 ; 2015 ; Shokri et al. , 2017 ; Carlini et al. , 2019 ) . One prominent category of privacy attacks against machine learning is the so-called membership inference attack ( Shokri et al. , 2017 ; Yeom et al. , 2018 ) , where the adversary aims to infer the participation of an individual in the target model ’ s training set . Such an attack is undoubtedly damaging when the status of participation itself is considered sensitive , e.g. , if the training dataset consists of health records of cancer patients . Moreover , the ability to infer membership can be viewed as a lower bound for the model ’ s degree of memorization ( Yeom et al. , 2018 ) , which is useful in itself as an empirical quantifier of privacy loss ( Jagielski et al. , 2020 ; Nasr et al. , 2021 ) . The efficacy of membership inference attacks has been improved significantly since the first attempts ( Salem et al. , 2018 ; Sablayrolles et al. , 2019 ; Leino & Fredrikson , 2020 ) . However , the most common evaluation metric , attack accuracy , overlooks the crucial factor of the false positive rate ( FPR ) of non-members ( Rezaei & Liu , 2021 ) . Indeed , most attacks operate by first defining a membership score s ( h , z ) for the model h and a target input-label pair z = ( x , y ) that measures how much h memorized the sample z . The attack subsequently selects a threshold τ and predicts that z is a member if and only if s ( h , z ) > τ . For typical choices of the membership scoring function , there is usually a large overlap in the distribution of s ( h , z ) between members and non-members ( see Figure 1a ) . As a result , an attack that determines the membership of z by thresholding on s ( h , z ) will have a high FPR . This drawback renders most existing attacks unreliable since the vast majority of samples likely belong to the non-member class . In this study , we identify the lack of difficulty calibration as a core contributor to the high FPR of existing attacks . Specifically , a non-member sample may have a high membership score simply because it is over-represented in the data distribution . Consequently , an attack that determines a sample is likely to be a member due to having a high score will inevitably fail on these over-represented samples . To remedy this problem , we make the acute observation that if the membership score is measured in comparison to a typical model trained on data drawn from the same data distribution , this difference in behavior can serve as a much more reliable membership signal . Indeed , Figure 1b shows the histogram of scores s ( h , z ) after difficulty calibration , where the member and non-member scores have significantly better separation and low FPR is now attainable . We propose difficulty calibration as a general technique for improving score-based membership inference attacks , and modify several membership scoring functions such as confidence ( Salem et al. , 2018 ) , loss ( Yeom et al. , 2018 ) , and gradient norm ( Nasr et al. , 2019 ) to construct their calibrated variants . Evaluated on a comprehensive suite of benchmark datasets , we show that calibrated attacks achieve a significantly lower FPR compared to prior work . In particular , we measure the trade-off between true positives and false positives using the area under ROC curve ( AUC ) metric , and show that difficulty calibration drastically improves this trade-off compared to uncalibrated attacks , by up to 0.10 AUC on common ML benchmarks . In addition , calibrated attacks also drastically improve the precision-recall trade-off , while remaining on-par with or better than uncalibrated attacks in terms attack accuracy . Our results suggest that it may be important for future work to apply difficulty calibration to design more reliable and practical membership inference attacks . 2 BACKGROUND . Membership inference attacks are concerned with determining whether a given sample was part of a target model ’ s training set . Homer et al . ( 2008 ) showed in their pioneering study that it is possible to infer an individual ’ s presence in a complex genomic DNA mixture , which led to increased caution around releases of DNA data ( Zerhouni & Nabel , 2008 ) . Recent interest in member inference attacks was sparked by the work of Shokri et al . ( 2017 ) , who introduced the shadow models method : An adversary trains substitute models ( called shadow models ) to mimic the behavior of the model under attack ( called the target model ) . The adversary then observes the behavior of the shadow models when exposed to member and non-member samples , and uses this observation to train an attack meta-model for predicting membership on any given sample . Shokri et al . ( 2017 ) evaluated this attack on ML models trained on cloud APIs , and showed that the shadow models approach attains high levels of accuracy . Score-based attacks . Yeom et al . ( 2018 ) discovered a connection between membership inference attacks and overfitting , arguing that in principle , attack accuracy can be determined by how much the target model memorizes ( or overfits to ) the given sample ( x , y ) . This discovery led to a series of work on quantifying the degree of memorization via membership scores , which can be used for predicting that the given sample is a member when the score is high . The membership score can be computed using the loss ( Yeom et al. , 2018 ) , the gradient norm ( GN ) ( Nasr et al. , 2019 ) or the confidence of the model ’ s prediction ( Salem et al. , 2018 ) , often yielding state-of-the-art results ( Salem et al. , 2018 ; Choquette-Choo et al. , 2021 ) . We define these scores for the cross-entropy loss ` : sloss ( h , ( x , y ) ) = − ` ( h ( x ) , y ) : = log ( h ( x ) y ) , ( 1 ) sGN ( h , ( x , y ) ) = −‖∇ ` ( h ( x ) , y ) ‖2 , ( 2 ) sconfidence ( h , ( x , y ) ) = −max y′ ` ( h ( x ) , y′ ) = max i log ( h ( x ) i ) . ( 3 ) Label-only attacks . The above score-based attacks rely on continuous-valued predictions from the model in order to define the membership score . To counter these attacks , prior work considered obfuscating the model ’ s output by returning only the top label or modifying the predicted values ( Shokri et al. , 2017 ; Jia et al. , 2019 ) . However , subsequent studies showed that even with only hard-label output , it is still possible to define membership scores that attain close to state-of-the-art accuracy ( Li & Zhang , 2020 ; Choquette-Choo et al. , 2021 ) . High-precision attacks . Various forms of difficulty calibration have been considered in the context of high-precision attacks . Long et al . ( 2018 ) selected samples that differ the most in loss between the target and a set of reference models , and showed that the resulting attack has high precision even for well-generalized target models . Carlini et al . ( 2020 ) showed that such privacy attacks are also possible on large-scale language models such as GPT-2 ( Radford et al. , 2019 ) . Their attack operates by first generating a large number of sentences from the language model and then ranking these sentences by the ( log ) perplexity , with lower perplexity indicating a more plausible memorized training sample . These perplexity values are then divided by either the z-lib entropy or the perplexity given by a smaller language model to account for the sentence ’ s rarity . In effect , only rare sentences with low perplexity can minimize the resulting score . Both attacks leverage a form of difficulty calibration by comparing the target model ’ s loss with that of reference models , predicting membership only when the difference ( or ratio ) is large . Differential privacy as a mitigation . Differential privacy ( Dwork et al. , 2006 ) ( DP ) is a powerful mathematical framework for privacy-preserving data analysis . A randomized algorithmM satisfies ( , δ ) -differential privacy if , given any two datasets D and D′ that differ in at most one sample , and for any subset R of the output space , we have : P ( M ( D ) ∈ R ) ≤ exp ( ) P ( M ( D′ ) ∈ R ) + δ . ( 4 ) Under mild assumptions , DP provably protects against a variety of privacy attacks , in particular membership inference ( Yeom et al. , 2018 ; Sablayrolles et al. , 2019 ) . Abadi et al . ( 2016 ) proposed a differentially private version of stochastic gradient descent ( SGD ) , called DP-SGD , to enable differentially private training of generic ML models . Their analysis has further been refined ( Mironov , 2017 ; Mironov et al. , 2019 ) and has been shown experimentally to be tight ( Nasr et al. , 2021 ) . 3 DIFFICULTY CALIBRATION . As depicted in Figure 1 , prior works on score-based membership inference attack are very unreliable for separating easy-to-predict non-members from hard-to-predict members since both can attain a high membership score . We argue that a simple modification to the score , which we call difficulty calibration , can drastically improve the attack ’ s reliability . This approach has been applied to the loss score for high-precision attack against well-generalized models ( Long et al. , 2018 ) . Let s ( h , ( x , y ) ) be the membership score , where higher score indicates a stronger signal that the sample ( x , y ) is a member . Instead of computing the membership score only on the target model h , we sample multiple “ typical ” models trained on the same data distribution as h and evaluate the membership score on these models . Doing so calibrates s ( h , ( x , y ) ) to the difficulty of the sample . For instance , if ( x , y ) is an easy-to-predict non-member , then s ( h , ( x , y ) ) is high but typical models g would also perform well on ( x , y ) , hence s ( g , ( x , y ) ) would also be high . The small gap between s ( h , ( x , y ) ) and s ( g , ( x , y ) ) suggests that ( x , y ) is likely a non-member . Formally , let Dshadow be a shadow dataset drawn from the same data distribution as the training set of h , and let A be a randomized training algorithm that samples from a distribution over models trained on Dshadow . We define the calibrated score as : scal ( h , ( x , y ) ) = s ( h , ( x , y ) ) − Eg←A ( Dshadow ) [ s ( g , ( x , y ) ) ] , ( 5 ) where the expectation is approximated by sampling one or more models from A ( Dshadow ) . The membership inference attack proceeds by thresholding on the score scal ( h , ( x , y ) ) . Efficient difficulty calibration via forgetting . Faithfully executing the difficulty calibration algorithm in Equation 5 requires training multiple models g using the randomized training algorithm A . This can be prohibitively expensive if the model is large , which is the case with modern neural networks such as large-scale transformers ( Brown et al. , 2020 ) . An alternative , more efficient approach if given white-box access to the target model h is to warmstart training on h. In fact , this approach establishes an explicit connection between membership inference attacks and catastrophic forgetting in neural networks ( Goodfellow et al. , 2013 ; Kirkpatrick et al. , 2017 ) . Toneva et al . ( 2018 ) observed that when a trained model resumes training on a separate dataset drawn from the same data distribution , it is very likely to “ forget ” about its original training samples . The training examples that are most likely forgotten are ones with abnormal or distinctive features , and this set of examples is stable across different training runs . We leverage this phenomenon to define a more efficient variant of difficulty calibration . Connection to posterior inference . Under a simplifying assumption of the distribution A ( D ) induced by the training algorithm , it is possible to derive difficulty calibration as an approximation to an optimal white-box attack . Sablayrolles et al . ( 2019 ) proposed a formal analysis of membership inference , where they assumed that the density function of A ( D ) has the form : p ( h ; D ) ∝ e− ∑ ( x , y ) ∈D ` ( h ( x ) , y ) , ( 6 ) which can be formally derived for optimization using SGD under certain conditions ( Sato & Nakagawa , 2014 ) . The implied threat model in differential privacy assumes that the adversary has knowledge that the training dataset is either D or D′ = D ∪ { ( x , y ) } for some inference target ( x , y ) , and D is known to the adversary . It can be shown that the Bayes-optimal attack strategy under this setting is to threshold on the calibrated score in Equation 5 , where s = sloss andD = Dshadow ( Sablayrolles et al. , 2019 ) . This analysis can be generalized to any scoring function s ( h , ( x , y ) ) , such as gradient norm ( Equation 2 ) and confidence ( Equation 3 ) .

Summary: This paper focuses on the analysis of membership inference attacks (MIAs), in  particular addressing the problem of high false positive rate (FPR) that still affects the  state-of-the-art solutions. The authors suggest an approach to MIA that is based on difficulty calibration, i.e., a method where the predicted membership score is adjusted  to the difficulty of obtaining the correct classification for the target sample, in order to reduce the FPR. As illustrated in Figure 1, previous work on the score-based membership inference attack is highly unreliable for separating easy-to-predict non-members from hard-to-predict members, as both can achieve high membership scores. This technique appears to be particularly beneficial in helping to distinguish between  member and non-member samples that are overrepresented in the data distribution. The authors support the proposed method with a large body of mostly favorable experimental results.