Demystifying the Membership Inference Attack
By: Paul Irolla
Tags: Machine Learning, Artificial Intelligence, Security, Privacy, Explainable Ai

Understanding the factors that influence the MIA

Let’s start by studying the MIA on the standard MNIST dataset. This is the Hello World! of machine learning, a very simple problem to solve. We train the target model with 10 epochs on a simple convolutional neural network:

It is PyTorch code to define a convolutional neural network. For instance nn.Conv2d(1, 10, 3, 1)) defines a 2D convolutional layer accepting images over 1 channel in input (gray scale), it has 10 trainable filters of size 3x3, and filters move over images with a stride of 1.

The target model gets 99.89% accuracy on its training set and 98.82% accuracy on its test set. The shadow models are copies of the target model, trained on 10 epochs on their shadow dataset. We then generate training samples for our 10 attack models with the trained shadow models. Attack models are simple multilayer perceptrons:

To measure the success of our attack we compute the mean accuracy of the attack models on their test set (train and test samples executed by the target model). We get 51.97% accuracy, hence this attack is not significantly better than a coin toss. The original paper [4] states that the MIA is linked with overfitting. The authors measure the overfit with the gap between the training set accuracy and the test set accuracy. In this case the gap is close to 0. Hence, we carry on with a harder problem on which it is harder to generalize to unseen samples.

CIFAR-10 is known to be a harder problem to solve. It is composed of miniature images of 10 object classes. We use a more complex convolutional model for the target and shadow models:

In this experiment we do not modify the architecture of the attack models and we train the target and shadow models with 15 epochs. The target model has 98% accuracy on its training set, 66% on its test set and the attack models have a mean accuracy of 67.44%. We start to see some success for the MIA. Effectively the train-test accuracy is 32% and attack models succeed to differentiate partially the in from out confidence levels. Let’s investigate the attack models samples distributions to understand how it works:

Confidence distributions for in (left) and out (right) test samples (generated by the target model).

This figure is a violin plot [15] of the confidence distributions of in and out attack model test samples (samples output by the target model). We plot only the distribution of the confidence level for the class 0 for the attack model for the class 0 here. The difference between in and out confidence level is clear, but the out confidence distribution seems to be the addition of two distinct distributions. To investigate this phenomenon we differentiate samples that have a correct prediction (true positives) from the target model class 0 and that have a wrong prediction (false negative). We do not have here false positives and true negatives because we only consider samples of label 0 for the test “does the model predicted label 0?”.

Distributions of in and out samples, false negatives only

Distributions of in and out samples, true positives only

With this sample separation we are able to single out the two different distributions that compose the total distribution. The clear differentiation that we have been able to glimpse is now is no longer relevant here. Actually, the difference in form in the total distributions is due to the differences in the number of true positive samples and the false negative samples in the in and out samples. The common hypothesis for the MIA success is that in samples exhibit a higher confidence level for their class compared to out samples [4]. This hypothesis does not hold here. In fact, a better hypothesis is: the attack models learn to single out samples that get a correct prediction from samples that get an incorrect prediction. To summarize, we could replace the whole membership inference attack process by a simple rule “If the target model predicts the label correctly for a sample, then it comes from the training set, otherwise it does not”. We refer later to this rule as the correct classification rule (CCR).

Let’s measure the accuracy of the CCR. We assume that to measure accuracy, there is as many sample in as out to test. This is indeed the case in our implementation of the MIA. On the MNIST experiment, the CCR accuracy is 50.53% (99.89 + (100–98.82))/2) and on the CIFAR-10 experiment the CCR accuracy is 66%. It seems very correlated with the attack models mean accuracy, respectively 51.97% and 67.44%. In fact, with these experiments, the rule is as effective to infer sample membership than the MIA with shadow models. Does this mean that, finally, the hypothesis of MIA success based on the confidence level is purely an illusion? It seems that the success of the attack is mainly based the train-test accuracy gap and the CCR captures this phenomenon.

Does this mean that, finally, the hypothesis of MIA success based on the confidence level is purely an illusion?

We studied all the public implementations that we could find, and reproduced as many experiments as we could from the literature. Some experiments presented in research articles expose a low train-test accuracy gap, but show a high MIA accuracy. Unfortunately we have not been able to reproduce those experiments. Either the dataset was not public, or the processing steps was too vaguely described to allow for reproduction. Moreover, a majority of public implementations are currently wrong, and show unrealistically good results because of induced biases. Those biases are often a bad dataset splitting strategy leading to biased results or the testing of the attack models on samples produced by the shadow models instead of the target model. In definitive, we could not find counterexamples to our hypothesis which is alarming because it questions the reality of the MIA with shadow model. Lastly, our hypothesis explains some curious assertions like: “our attacks are robust even if the attacker’s assumptions about the distribution of the target model’s training data are not very accurate” [4], “even restricting the prediction vector to a single label (most likely class), which is the absolute minimum a model must output to remain useful, is not enough to fully prevent membership inference” [4]. Indeed, the CCR does not need confidence levels.