Title: Students Parrot Their Teachers: Membership Inference on Model Distillation
Venue: NeurIPS 2023
Accepted: yes
Keywords: model distillation

membership inference

privacy

dark knowledge
Abstract: Model distillation is frequently proposed as a technique to reduce the privacy leakage of machine learning. These empirical privacy defenses rely on the intuition that distilled ``student'' models protect the privacy of training data, as they only interact with this data indirectly through a ``teacher'' model. In this work, we design membership inference attacks to systematically study the privacy provided by knowledge distillation to both the teacher and student training sets. Our new attacks show that distillation alone provides only limited privacy across a number of domains. We explain the success of our attacks on distillation by showing that membership inference attacks on a private dataset can succeed even if the target model is never queried on any actual training points, but only on inputs whose predictions are highly influenced by training data. Finally, we show that our attacks are strongest when student and teacher sets are similar, or when the attacker can poison the teacher set.

Students Parrot Their Teachers: Membership Inference on Model Distillation

Abstract

Model distillation is frequently proposed as a technique to reduce the privacy leakage of machine learning. These empirical privacy defenses rely on the intuition that distilled 'student' models protect the privacy of training data, as they only interact with this data indirectly through a 'teacher' model. In this work, we design membership inference attacks to systematically study the privacy provided by knowledge distillation to both the teacher and student training sets. Our new attacks show that distillation alone provides only limited privacy across a number of domains. We explain the success of our attacks on distillation by showing that membership inference attacks on a private dataset can succeed even if the target model is never queried on any actual training points, but only on inputs whose predictions are highly influenced by training data. Finally, we show that our attacks are strongest when student and teacher sets are similar, or when the attacker can poison the teacher set.

1 Introduction

Model distillation [HVD+15] is a common framework for knowledge transfer, where knowledge learned by a 'teacher model' is transferred to a 'student model' via the teacher's predictions. Distillation is helpful because the teacher's predictions are a more useful guide for the student model than hard labels; this phenomenon has been explained by the teacher's predictions containing some useful 'dark knowledge'. Variants of model distillation have been proposed for, e.g., model compression [HVD+15; BC14; PPA18; KPK18; SCGL19] or training more accurate models [ZK16; XLHL20]. Within the privacy-preserving machine learning community, distillation has been adapted to protect the privacy of a training dataset [PAEGT16; TMSSNHM22; SH21; MHHVEHP22].

Many of these approaches rely on the intuition that distilling the teacher model serves as a privacy barrier that protects the teacher's training data. Informally, restricting the student to learn only from the teacher's predictions is a form of data minimization , which should result in less private information being fed into, and memorized by, the student. This privacy barrier around the teacher also allows the teacher model to be trained with strong, non-private, training approaches, improving both the teacher model's and student model's accuracy.

Because model distillation does not provide a rigorous privacy guarantee (such as those offered by differential privacy [DMNS06]), in our work we evaluate the empirical privacy provided by these schemes. We show that distillation is vulnerable to membership inference attacks-a well-studied class of privacy attacks on machine learning [SSSS17; YGFJ18].

We adapt the state-of-the-art Likelihood Ratio Attack (LiRA) [CCNSTT22] to the distillation setting, and find that this attack works surprisingly well at inferring membership of the teacher's training data. For some training examples, model distillation fails to appropriately protect against mem-

bership inference. To explain this finding, we show for the first time that the membership presence of some examples can be inferred based on the model's predictions on other , seemingly unrelated examples. This observation provides new insights into how membership information is transmitted from the teacher to the student.

Figure 1 gives an example of such cross-example leakage. A teacher model trained on the red parrot in Figure 1a (labeled as 'bird'), never seeing the student queries on the right in Figure 1b, encodes membership information about the parrot in predictions on these student queries. Interestingly, the most informative student queries are not birds-they are images of other classes with similar red hues that the model confuses as being 'bird-like' because of the influence of the parrot in the teacher's training set. In other words, the student model manages to 'parrot' its teacher using the peculiarities transferred through distillation.

We systematically evaluate a number of factors which impact the empirical privacy of model distillation. We find that similarity between the training datasets of the teacher and student models lead to increased privacy risk, along with higher temperature parameters. Moreover, an adversary capable of poisoning the teacher training set can amplify privacy risk for teacher examples, in line with recent work [TZJRR16; CSSWZ22]. We also find that distillation provides little privacy protection to student examples. We hope our attacks can assist experts in properly evaluating the privacy risks resulting from model distillation. Our work also highlights two possible approaches for mitigating privacy risks in distillation: deduplicating the teacher and student datasets, and reducing the leakage from the teacher model, e.g., using provable guarantees such as differential privacy [DMNS06].

2 Background and Related Work

2.1 Machine Learning Privacy

Machine learning models are known to be vulnerable to a variety of privacy attacks, including membership inference attacks [SSSS17], attribute inference attacks [FJR15], property inference attacks [GWYGB18], and training data extraction [CTWJHVLRBSE+21]. Each of these captures a different type of leakage about the training data or individual examples.

In this work, we focus on membership inference, as it is the most widely studied privacy attack. In a membership inference attack, an adversary tries to determine whether or not a particular example was used to train a model. There have been a number of membership inference attacks proposed in the literature, which generally compute some 'membership score', which is designed to be informative of a target example's membership. Most of these scores make use of the model's prediction on the target example, perhaps to compute the example's loss [YGFJ18; SDSOJ19] or to compute some other score [WGCS21; SSSS17; CCNSTT22]. Other attacks rely on querying the model with examples nearby or derived from the target [JWKGE20; LBWBWTGC18; LZ21; CCTCP21; WBKBGGG22]. One of our contributions is to design an attack which performs well despite relying on the model's predictions on entirely different examples from the target.

Our attacks are based on the state-of-the-art Likelihood Ratio Attack (LiRA) [CCNSTT22]. In LiRA, the adversary first trains many shadow models , such that half of these models will contain a given target example ( x, y ) (the IN models), and half will not (the OUT models). Next, the adversary queries each shadow model f , to compute the logits corresponding to the correct class, f ( x ) y , and fits a Gaussian N ( µ in , σ 2 in ) to the logits for all models containing the example (and similarly for the

OUT models). To attack a new model f ′ , the adversary computes the probability density function (PDF) of each Gaussian (which we write as p in ( f ′ ( x ) y ) and p out ( f ′ ( x ) y ) ), and then computes the likelihood ratio, which serves as the membership score. Our attacks in this paper will be derived from LiRA but adapted to the variety of distillation settings we consider.

2.2 Knowledge Distillation

Knowledge distillation [HVD+15; BC14] is a technique for transferring knowledge from a 'teacher' model to a 'student' model. There are two datasets in knowledge distillation: the teacher dataset D T = { x i , y i } n t i =1 , with n t examples, and the student dataset D S = { x i , y i } n s i =1 , with n s examples. Distillation begins by training a teacher model f T on the teacher dataset D T . The teacher model is used to generate (soft) labels for the student dataset, producing a query dataset D Q = { x i , S ( f T ( x i )) } n s i =1 , where the softmax function S converts logits into a probability vector. Training on this query dataset then produces a student model f S , the output of distillation. Sometimes, distillation includes 'temperature' scaling, where the logits f T ( x i ) are scaled by a temperature parameter H , before the softmax is applied. The query dataset will instead be D Q = { x i , S ( f T ( x i ) /H ) } n s i =1 ; setting H = 1 recovers the original distillation procedure. We illustrate this process in Figure 2.

Knowledge distillation in private machine learning. Prior work has suggested that distillation mitigates prevent privacy attacks. Perhaps the most well known example is the PATE framework [PAEGT16], where distillation is used to reduce an ensemble of teacher models into a single model, in such a way that the final model has provable differential privacy guarantees [DMNS06]. Zheng, Cao, and Wang [ZCW21] and Tang, Mahloujifar, Song, Shejwalkar, Nasr, Houmansadr, and Mittal [TMSSNHM22] construct ensembles of models that are designed to be private and use distillation to condense these models. Mireshghallah, Backurs, Inan, Wutschitz, and Kulkarni [MBIWK22] enforces a differential privacy guarantee by training the teacher and student models with differential privacy. In each of these approaches, distillation is only one component; an underlying ensemble, or provable differential privacy guarantees, may also improve the privacy of the overall approach. In our work, we focus on the distillation procedure itself, and leave to future work the task of designing attacks on these more complicated approaches. Importantly, also, those works which offer provable privacy guarantees will have hard limits on attack effectiveness.

Indeed, other prior work has already suggested that using distillation alone to protect privacy. Shejwalkar and Houmansadr [SH21] propose a defense that relies on a sufficient glyph[lscript] 2 distance between teacher and student examples and limited entropy of the queries. We evaluate this defense in Section 6.4 and Appendix D.1. Mazzone, Heuvel, Huber, Verdecchia, Everts, Hahn, and Peter [MHHVEHP22] consider using repeated distillation to prevent membership inference attacks. In our work, we will design stronger membership inference attacks to adaptively evaluate the privacy provided by distillation.

3 Threat Model and Experimental Setting

3.1 Threat Model

We investigate the ability of distillation to protect against membership inference attacks in three threat models:

Private Teacher. The teacher dataset D T is sensitive and the student dataset D S is nonsensitive. We assume the adversary has knowledge of the student dataset. This threat model

is used in most private machine learning approaches. We evaluate this threat model in the main body of the paper.

Private Student. The teacher dataset is nonsensitive and the student dataset is sensitive. We assume the adversary has access to the teacher dataset. This threat model may be used to transfer knowledge from a foundation model used as a teacher to a sensitive downstream task [LZZHCZ22]. Because this threat model is less well considered in the privacy literature [SH21], we evaluate this threat model in Appendix D.1.

Self-Distillation. Self-distillation is commonly used to refer to the setting where the teacher and student datasets are identical. Self-distillation is commonly used when distillation is used to improve model performance or during model compression. We evaluate this threat model in Section D.2.

For each of these threat models, we measure the success of the adversary at performing membership inference on the sensitive dataset, both by measuring the true positive rate (TPR) and false positive rate (FPR) as suggested by Carlini, Chien, Nasr, Song, Terzis, and Tramer [CCNSTT22], and also by investigating the membership inference accuracy on each individual example, as done in Carlini, Jagielski, Zhang, Papernot, Terzis, and Tramer [CJZPTT22]. In general, we will report this perexample membership inference accuracy when presenting the performance of one or two attacks, due to the amount of information this metric conveys, and use ROC curves to compare between more than two.

Beyond distillation's potential uses in privacy, our attacks on distillation have implications for private information leaked during learning-based model extraction attacks [TZJRR16; OSF19; PGSKSG20; JCBKP20], which often resemble model distillation. In model extraction, an adversary uses API access to a target model to reproduce its functionality into the weights of a local model. The target model is analogous to the teacher model in distillation, and the local model is analogous to the student. Then attacks in the Private Teacher threat model can be cast instead as allowing an adversary to inspect their local model to perform membership inference attacks on the target model's training data, without directly querying it. The Private Student threat model has implications for a defensive setting in model extraction, where the target model's owner wants to link queries made to the target model with a model they believe has been extracted from their model.

3.2 Experimental Setting

We study four standard datasets for our analysis: CIFAR-10, WikiText103, Purchase100, and Texas100. On CIFAR-10 , we start with code from the DAWNBench benchmark [CNKZZNBORZ17] which trains an accurate ResNet-9 model in under 15 seconds; we adapt this code to support model distillation and the subsampling required for LiRA variants. We remove 5275 duplicates from CIFAR-10, using the imagededup library [JLJT19], and split the remaining dataset into a teacher set of 30,000 examples and a student set of 14,725 examples. On WikiText103 , we used the GPT-2 architecture (with a context window of 256 tokens, 4 heads, 4 layers and an embedding size of 256 dimensions). We split WikiText103 into a teacher set of 500,000 records, and use the remaining records to train the student models. On Purchase100 and Texas100 , we train single-layer neural networks with hidden layer sizes of 256 and 512, respectively, and subsample the datasets to produce teacher and student sets of 20000 examples each. On Purchase100, student models reach 74-75% accuracy, and on Texas100, they reach 54-55% accuracy. On all datasets , we train our models with the cross entropy loss: teacher models are trained with the standard sparse cross entropy loss on the teacher dataset, and student models are trained with a dense cross entropy loss to mimic the soft labels predicted by the teacher. Unless otherwise stated, all LiRA-based attacks use 100 shadow models for calibration, and all figures are produced by running the attack on over 1000 models. Needing to train a large number of shadow models is one limitation of our attacks, and any shadow model-based membership inference attacks; we comment on training efficiency and code for our experiments in Appendix B.

4 Evaluating Privacy of the Sensitive Teacher Training Set

The most common way prior work in privacy has used distillation is to improve the privacy of the teacher set. Intuitively, distillation protects the teacher set because the adversary can only interact

(a) CIFAR-10

(b) WikiText103

(c) Texas-100

(d) Purchase-100

1.0

+

R=0.81

1.0

+

R=0.90

1.0

+

R=0.88

1.0

+

R=0.95

0.9

一

y=x

0.9

y=x

0.9

y=x

0.9

y=x

0.8

0.8

0.8

0.8

LiRA

d-to-End

0.7

0.7

0.7

0.7

0.6

0.6

0.6

0.6

End-

0.5

0.5

0.6

0.7

0.8

0.9

1.0

0.5

0.5

0.6

0.7

0.8

0.9

1.0

0.5

0.5

0.6

0.7

0.8

0.9

1.0

0.5

0.5

Teacher MI Accuracy

0.6

0.7

0.8

0.9

1.0

with the student model, which never sees any teacher examples. In fact, because the student model can be seen as a 'post-processing' of the teacher model, the data processing inequality provably implies that attacks can be no more powerful on the student model than they are on the teacher.

Stepping through the distillation process can help us anticipate how distillation can impact the privacy of the teacher set. The first step, training the teacher model, is the most well-understood from a privacy perspective, as the large literature on membership inference applies to the teacher model. In particular, recent work has found that state-of-the-art membership inference attacks are better at attacking some 'outlier' examples than other 'inlier' examples [CCNSTT22], which we expect to be true in the teacher dataset as well, making these 'inlier' examples less vulnerable in later steps, as well.

Subsequent steps of distillation are less well-explored by the privacy literature. The second step of distillation creates the query dataset, which can be seen as 'compressing' the teacher model into its responses on these queries. Intuitively, this step is the most important at reducing private information leakage. However, we hypothesize that some queries will capture information about teacher examples, perhaps due to some similarity between the queries and teacher examples. In the final step, the student model is trained on the query set. While this step cannot contain more sensitive information than the queries themselves, it is possible that this step makes that information easier to discover , perhaps by interpolating between the queries.

4.1 Performing Membership Inference on Teacher Examples

We now show that membership inference attacks work surprisingly well on distilled student models.

We first propose the 'Transfer LiRA' attack, a simple extension of LiRA to the distillation setting. In this attack, we train shadow teacher models. We calibrate the IN and OUT Gaussians for LiRA on these shadow teacher models, and then run the attack by applying it directly to the student models. Notice that in this attack, distillation never happens in training shadow models-the success of this attack relies on the similarity between teacher and student models' predictions.

To capture the information loss because Transfer LiRA does not use distillation in any way, we also propose a second attack, which we call 'End-to-End LiRA', where we instead train shadow models with the entire distillation procedure, first training shadow teacher models, and then distilling these teacher models into shadow student models. We then calibrate LiRA using these shadow student models. Note that performing this attack requires knowledge of the student training set to query the shadow teacher models and train the shadow student models. This is in contrast to Transfer LiRA which does not, although it still requires access to in-distribution data to train teacher models.

Distillation provides limited privacy. In Figure 3, we plot the change in per-example attack success rate on distilled models. We compare each example's vulnerability to LiRA on the teacher model (i.e. vulnerability without distillation) on the x-axis, to each example's vulnerability to End-to-End LiRA (i.e. vulnerability after distillation) on the y-axis. In the Appendix, we provide plots for

(a) CIFAR-10

(b) Texas-100

(c) Purchase-100

1.0

1.0

1.0

+

R=0.77

+

R=0.73

+

R=0.76

Accur

0.9

y=x

0.9

y=x

0.9

y=x

0.8

0.8

0.8

M

0.7

0.7

0.7

Query

0.6

0.6

0.6

0.5

0.5

0.5

0.6

0.8

1.0

0.5

0.6

0.7

0.8

0.9

1.0

0.5

0.6

0.7

0.8

0.9

1.0

TeacherMIAccuracy

TeacherMlAccuracy

Transfer LiRA (Figure 7), and ROC curves for each strategy (Figure 8). We also find our attack significantly outperforms a simple logit threshold baseline, similar to the weaker attacks used in prior work to evaluate distillation (Figure 12).

For a large fraction of teacher examples on each dataset, we find End-to-End LiRA achieves nearly the same membership inference performance as directly attacking the teacher model. In other words, many teacher examples do not observe any privacy benefits from distillation, despite student models never directly seeing these teacher examples! While the average membership inference accuracy (and TPR at low FPR) do decrease, 5% of examples' vulnerability drop by less than 8 percentage points on CIFAR-10, 5 points on Purchase-100, and 4 points on Texas-100, and these examples have a variety of teacher vulnerabilities. Because privacy is a worst-case guarantee, distillation provides limited privacy benefits.

We note two other interesting takeaways from these results. First, the per-example student attack success rate has a high variance. This variance is partly due to statistical uncertainty (although each example's attack success rate is computed with over 1000 models, giving each coordinate a standard deviation of 1.5 percentage points for both the x and y axes), but more interestingly, some examples do see significant privacy benefits from distillation, even controlling for the original model's vulnerability. We investigate duplication as one potential cause for this variance in Section 6.1. Second, each plot has a positive correlation, meaning that examples that are more vulnerable to attack on the teacher model attack also tend to be more vulnerable after distillation. As a result, reducing the vulnerability of the teacher model, perhaps using techniques such as differential privacy, are likely to improve the student's privacy.

5 Tracing Teacher Privacy Leakage Through Student Queries

It is surprising that membership inference still performs well on distilled student models, despite these models never directly using any of the teacher data. We now investigate why . For a membership inference attack to be successful on the student model, it must be the case that the student queries reveal some membership information about the teacher examples. However, to rigorously evaluate how this information is encoded in the student queries, we will turn our attention to designing a new attack that only has access to the student query dataset D Q (and, in particular, has no knowledge of the teacher model's predictions on the teacher examples). To the best of our knowledge, this attack is also the only membership inference attack in the literature which does not require querying a model directly on an example (or on algorithmically derived examples) to predict its membership status.

An indirect attack using student queries. We adapt LiRA to a setting where only the student query scores are available to the adversary. Because of this limitation, the adversary can only rely on information about teacher examples that is indirectly contained in the student query scores. To adapt the attack to this setting, we use the same approach that LiRA uses to combine queries on

multiple 'augmentations' of an example: multiplying their likelihood ratios to produce an aggregate membership score. Concretely, for each teacher example z T j = ( x T j , y T j ) , we fit a Gaussian distribution to the logits of each student example z S i = ( x S i , y S i ) , when z T j is either IN or OUT. We write the PDFs of these distributions as p IN j,i and p OUT j,i , so that the joint PDF of the IN Gaussians is p IN j = Π i p IN j,i and similarly for the OUT Gaussians. This natural adjustment allows us to infer membership of the teacher examples using the student examples.

However, we find that this direct adaptation of LiRA tends to have poor performance, so we propose two modifications which significantly improve the attack. First, we find that a teacher example tends to have more influence on the logit corresponding to the teacher label y T j than the student label y S i , and so we choose to calibrate LiRA using the teacher label logit rather than the student label logit. Second, there tend to be many uninformative student examples for each teacher example, so we filter the student queries for all teacher examples. We filter these student queries by selecting only those with the largest mean gap | µ IN j,i -µ OUT j,i | , which are most informative about the teacher. This removes most of the noise from our LiRA adaptation, improving it especially when we train few shadow models.

Student queries leak private information. In Figure 4, we plot the per-example membership inference accuracy after distillation, using our student query attack with 4000 shadow models. We consider all datasets except WikiText103, which we omit due to the high computational cost of our attack. Our results are similar to when we attacked the student models: MIA vulnerability reduces on average, but many examples maintain nearly identical membership inference attack success rate using only the predictions on these indirect student queries. Cases where this attack achieves high membership inference accuracy indicate that a large amount of information about the teacher set is encoded in the student queries.

Observations we noticed for our student model attacks persist for this query-based attack. Examples which are more vulnerable to attack in the teacher model also tend to be more vulnerable using teacher model predictions. This is true on all datasets, although we also find the decay in vulnerability with queries to be dataset-dependent and example-dependent. Within each dataset, there is a high variance in student model membership inference accuracy.

We also remark on the counterintuitive fact that our prior attack with access to the student models outperformed this attack, despite the fact that the student model is just a post-processing of these student queries. This implies that the process of training a student model makes it easier to extract private information from these student queries, by interpolating between the student queries. In the following subsections, we will investigate some factors that can impact the success of the student query attack, and what this tells us about distillation's dark knowledge.

Ablation. In Figure 5a, we ablate the modifications we made for our student query attack. We show both LiRA scores (logits for the student label) and Label scores (logits for the teacher label). We also show attacks with 'All', without filtering, and 'Filtered' scores, with filtering to 10 student examples. Our modifications significantly improve membership inference: without either modification, the attack is no better than random chance. Applying both modifications achieves a TPR of 10 -3 at a FPR of 10 -4 .

Qualitative analysis. Our ablations also shed light on where membership information is contained in distillation's dark knowledge: predominantly in the logit corresponding to the teacher label within only a few student examples. This corroborates our intuition from the parrot in Figure 1. There, only two student queries were also in the bird class, emphasizing the importance of label scores. We inspect other vulnerable examples in Figure 9 in Appendix C, finding some teacher examples whose most informative students mostly belong to their same class.

6 Factors Influencing Teacher Privacy Leakage

So far, we have designed two types of attack: the attack from Section 4 which uses the student model, and the attack from Section 5 which uses the student queries. These attacks find a surprising amount of leakage which could not have been identified by prior attacks. Now, we investigate more deeply the factors which may influence the leakage of the teacher.

10

3

10

2

10

1

10 0

False Positive Rate

10

3

10

2

10

1

10 0

True Positive Rate

Filtered

All

Filtered

All

(a) Ablation

(b) Duplicates

(c) Data Poisoning

(d) Temperature

1.0

No Dups

1.0

Clean

100

sdna

+

Pois 1

0.9

y=x

Accurac

2

0.9

+

Pois 4

Pois 16

Rate

10-1

0.8

0.8

一

y=x

Positive

H=4

Query

0.7

IW

Query

0.7

102

H=2

H=1

0.6

0.6

True

H=0.5

10-3

H=0.1

0.5

0.5

y=x

TeacherMIAccuracy

0.6

0.8

1.0

0.6

0.8

1.0

10-3

10-2

10-1

100

False Positive Rate

6.1 Student-Teacher Duplication

Unlike the preceding indirect queries, where student examples only share high-level similarities with their teacher examples, we now investigate how increased student-teacher similarity impacts the performance of our student query attack. To do so, we reintroduce the near-duplicates we removed from CIFAR-10, and compare the attack success rate between the duplicated and deduplicated student query attacks, on those teacher examples with duplicate student examples. Intuitively, student examples which are duplicates of teacher examples will carry membership information in their queries, which is what we find in Figure 5b ( p < 10 -15 using a Chow Test). This indicates that deduplication significantly reduces privacy risk, although indirect queries can still carry membership information. In the Appendix, we show student attacks are also worsened by duplication in Figure 11.

6.2 Teacher Set Poisoning

Tram` er, Shokri, San Joaquin, Le, Jagielski, Hong, and Carlini [TSSJLJHC22] and Chen, Shen, Shen, Wang, and Zhang [CSSWZ22] find that data poisoning attacks added into a training set can amplify membership inference vulnerability for other examples in the training set. Here, we investigate how distillation interacts with this effect; i.e., we measure whether poisoning in the teacher set can increase an example's vulnerability in the teacher predictions. In Figure 5c, we evaluate this using our student query attack on CIFAR-10 using the label flipping poisoning strategy from prior work.

With higher poisoning counts, many examples have higher teacher membership inference accuracy, i.e., they shift to the right on the x -axis; this is exactly in line with prior work. However, interestingly, we see that poisoning does not impact the relationship between teacher vulnerability and student vulnerability-they move upward on the y -axis identically to examples which were not poisoned.

We remark that our poisoning attack does not specifically target the student query set; it is an interesting open question whether poisoning attacks exist that could increase membership leakage on the student model, but with less (or more!) impact on the teacher model's vulnerability. Section 6.1 hints at such a strategy, if the adversary can poison the student set instead: if an adversary adds duplicates of target examples to the student set, the resulting student queries will increase risk on those target teacher examples.

6.3 Temperature Scaling

When introducing knowledge distillation, Hinton, Vinyals, Dean, et al. [HVD+15] proposed a modification known as temperature scaling. Temperature scaling makes two simple changes to distillation: (1) introducing a temperature hyperparameter H , which when increased, rescales the logits and flattens the resulting probability distribution of student queries, and (2) rescaling gradients by a 1 /H 2 factor. Though normally set to 1 , modifying this parameter can improve performance. It is natural to consider whether such manipulation of the teacher outputs might improve empirical

privacy; indeed, Shejwalkar and Houmansadr [SH21] evaluate their distillation-based defense with various temperatures, and find that high temperature reduces private information.

We use our strong End-to-End LiRA to evaluate the empirical privacy of student models trained with H ∈ [0 . 1 , 4] , shown in Figure 5d. We find lower temperatures are mildly less vulnerable than higher temperatures, reducing TPR by a factor of roughly 4 at an FPR of 10 -3 when H decreases from 4 to 0.1; model accuracy at all of these temperatures is similar to the baseline of H = 1 . While this trend is small, it is also intuitive: in Section 5, we found that all logit values carry important membership information. When H is small, the entropy of query probabilities decreases, reducing information captured by these probabilities. We hypothesize that the reverse effect observed by Shejwalkar and Houmansadr [SH21] may be a result of the accuracy decrease they found resulted from high temperatures (i.e. up to H = 10 ), which we do not observe.

Finally, we remark on an interesting gap between an adversary with access to student queries and one who only has access to a student model. With access to temperature-scaled student logits, an adversary can multiply by H to reverse the scaling-temperature scaling cannot make attacks harder in this threat model.

However, it does appear that low temperatures lead to weaker attacks on the student model, which we do not know how to reverse. Indeed, a natural way to adaptively attack a temperature-scaled student model is to rescale the student model's logits, but our LiRA accounts for this, as LiRA is scale-invariant. Adaptively attacking temperature-scaled models is an interesting open question.

6.4 Student Distribution Drift

In cases where the teacher dataset is sensitive, nonsensitive student data from the same distribution may be difficult to find, leading to distribution drift between the teacher and student datasets. We now investigate whether this impacts attack success. From our results on studentteacher duplication, we may expect the signal to diminish as distributions become more different. To evaluate this, we consider a CIFAR-10 teacher model distilled with a CIFAR-100 student set; CIFAR-10 and CIFAR-100 have no class overlap but have the same size. Interestingly, we find that distilling with this different dataset actually improves attack performance, despite the resulting model being somewhat less accurate than after distilling on indistribution data. We find that this is due to a higher entropy of the teacher model predictions, which can encode more membership information about the teacher examples. We run an experiment in which we train on the lowest entropy and highest entropy predictions from CIFAR100, as well as random student queries. All experiments

10

3

10

2

10

1

10 0

False Positive Rate

10

3

10

2

10

1

10 0

Teacher Model

CIFAR-10

C100 Low

C100 High

y=x

True Positive Rate

use the same student set sizes. We present the results in Figure 6. The CIFAR-10 student still results in the lowest leakage, and increasing entropy of the CIFAR-100 examples also increases the leakage monotonically. Note that the CIFAR-10 student predictions have a mean entropy of 0.24, while the lowest leakage CIFAR-100 examples have a mean entropy of 0.39. Our results here can also be taken as an attack on the full Shejwalkar and Houmansadr [SH21] defense, which removes high entropy and duplicate student examples. Our strong attacks allow us to better evaluate their defense.

6.5 Accuracy of the Student Model

It is common for privacy preserving training algorithms to exhibit a 'privacy-utility' tradeoff: the stronger the privacy enforced by the model, the worse the model's predictive performance. By piecing together the accuracies of models trained throughout Section 6, we find that differences in attack performance cannot be explained by changes in model accuracy. For example, we find in Section 6.3 that increasing temperature increases vulnerability to our attacks but does not significantly impact model accuracy (maintaining roughly 88% accuracy for the student). Our results with distribution drift in Section 6.4 show that distribution shift leads to higher vulnerability to our attacks,

but reduces model accuracy (from 88% to roughly 85% student model accuracy with a CIFAR-100 distilled student). Finally, we find in Section 5b that deduplication reduces vulnerability to our attacks, but actually increases model accuracy slightly (from roughly 87.5% to 88% student accuracy)! In summary, depending on the source of a model performance change, increased vulnerability can coincide with increased, decreased, or unchanged model performance!

7 Conclusion

In this work, we have used membership inference attacks to empirically evaluate to what extent knowledge distillation protects the privacy of training data. For those interested in using model distillation to improve privacy, our work offers three main design considerations. First, deduplicating the teacher set with respect to the student set is necessary to reduce risk on duplicated examples. Second, while average-case privacy can significantly improve after distillation, the worst-case vulnerable examples often see only marginal benefits. As a result, techniques which make the teacher model more private, such as differentially private training, should be seen as complementary to distillation. And third, the student training set should not be seen to have improved privacy as a result of model distillation. Our work also offers insights into the privacy properties of distillation's dark knowledge, which may be of broader interest. Our results also imply privacy attacks on model extraction attacks [TZJRR16; JCBKP20] which rely on similar algorithms to knowledge distillation.

Limitations

The main limitation of our work is the strong threat model under which our attacks work. We use the same threat model as the 'online' version of the LiRA [CCNSTT22] attack. This attack assumes access to the target examples before training a model, and, for the End-to-End LiRA attack, access to the student dataset. We use this strong threat model to assess worst-case vulnerability, as prior work has evaluated distillation under weaker attacks. Another limitation of the attacks is the large running time of the student query attack from Section 5, which requires thousands of shadow models to obtain good performance. While generally impractical, we prefer to position this attack as a way of explaining why distillation propagates membership information, and leave future work to attempt to improve the attack's efficiency.

References

[OSF19]

[PAEGT16]

[PGSKSG20]

[PPA18]

[SCGL19]

[SDSOJ19]

[SH21]

[SSSS17]

[TMSSNHM22]

[TSSJLJHC22]

[TZJRR16]

[WBKBGGG22]

13-24. ISBN: 9781450398800. URL: https://doi.org/10.1145/ 3560830.3563721 (cit. on pp. 1, 3).

T. Orekondy, B. Schiele, and M. Fritz. 'Knockoff nets: Stealing functionality of black-box models'. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 2019, pp. 4954-4963 (cit. on p. 4).

N. Papernot, M. Abadi, U. Erlingsson, I. Goodfellow, and K. Talwar. 'Semi-supervised knowledge transfer for deep learning from private training data'. In: arXiv preprint arXiv:1610.05755 (2016) (cit. on pp. 1, 3).

S. Pal, Y. Gupta, A. Shukla, A. Kanade, S. Shevade, and V. Ganapathy. 'Activethief: Model extraction using active learning and unannotated public data'. In: Proceedings of the AAAI Conference on Artificial Intelligence . Vol. 34. 2020, pp. 865-872 (cit. on p. 4).

A. Polino, R. Pascanu, and D. Alistarh. 'Model compression via distillation and quantization'. In: arXiv preprint arXiv:1802.05668 (2018) (cit. on p. 1).

S. Sun, Y. Cheng, Z. Gan, and J. Liu. 'Patient knowledge distillation for bert model compression'. In: arXiv preprint arXiv:1908.09355 (2019) (cit. on p. 1).

A. Sablayrolles, M. Douze, C. Schmid, Y. Ollivier, and H. J´ egou. 'Whitebox vs black-box: Bayes optimal strategies for membership inference'. In: International Conference on Machine Learning . PMLR. 2019, pp. 55585567 (cit. on p. 2).

V. Shejwalkar and A. Houmansadr. 'Membership privacy for machine learning models through knowledge transfer'. In: Proceedings of the AAAI Conference on Artificial Intelligence . Vol. 35. 2021, pp. 9549-9557 (cit. on pp. 1, 3, 4, 9, 13).

R. Shokri, M. Stronati, C. Song, and V. Shmatikov. 'Membership inference attacks against machine learning models'. In: 2017 IEEE symposium on security and privacy (SP) . IEEE. 2017, pp. 3-18 (cit. on pp. 1, 2).

X. Tang, S. Mahloujifar, L. Song, V. Shejwalkar, M. Nasr, A. Houmansadr, and P. Mittal. 'Mitigating membership inference attacks by { Self-Distillation } through a novel ensemble architecture'. In: 31st USENIX Security Symposium (USENIX Security 22) . 2022, pp. 14331450 (cit. on pp. 1, 3).

F. Tram` er, R. Shokri, A. San Joaquin, H. Le, M. Jagielski, S. Hong, and N. Carlini. 'Truth Serum: Poisoning Machine Learning Models to Reveal Their Secrets'. In: Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security . CCS '22. Los Angeles, CA, USA: Association for Computing Machinery, 2022, 2779-2792. ISBN: 9781450394505. URL: https://doi.org/10.1145/3548606. 3560554 (cit. on p. 8).

F. Tram` er, F. Zhang, A. Juels, M. K. Reiter, and T. Ristenpart. 'Stealing machine learning models via prediction { APIs } '. In: 25th USENIX security symposium (USENIX Security 16) . 2016, pp. 601-618 (cit. on pp. 2, 4, 10).

Y. Wen, A. Bansal, H. Kazemi, E. Borgnia, M. Goldblum, J. Geiping, and T. Goldstein. 'Canary in a Coalmine: Better Membership Inference with Ensembled Adversarial Queries'. In: arXiv preprint arXiv:2210.10750 (2022) (cit. on p. 2).

L. Watson, C. Guo, G. Cormode, and A. Sablayrolles. 'On the importance of difficulty calibration in membership inference attacks'. In: arXiv preprint arXiv:2111.08440

[WGCS21] (2021) (cit. on p. 2).

Q. Xie, M.-T. Luong, E. Hovy, and Q. V. Le. 'Self-training with noisy student improves imagenet classification'. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 2020, pp. 10687-

[XLHL20] 10698 (cit. on pp. 1, 14).

A Broader Impact

Our work designs privacy attacks, which have the potential to cause harm. However, by making the vulnerabilities in existing approaches known, and more rigorously evaluating the risk to users, our work is a necessary step to designing stronger mitigations in the future.

B More Experiment Details

All of our results on CIFAR-10 make use of fewer than 30000 trained models. While a very large number of models, the fast, publicly available training code we use allows us to train this number of models in fewer than 1 GPU-week (although we decrease the wall-clock time by parallelizing over 4 GPUs). Our results on Purchase-100 and Texas-100 also use simple models, taking under 1 minute to train (we train all models for 20 epochs with SGD with a learning rate of 0.01 and momentum parameter of 0.99, which we found to maximize performance over our hyperparameter sweep). We train 8000 of these models for our analysis, taking fewer than 1 GPU-week for each of these datasets. Our most expensive attack, relying on only student queries, starts to outperform random guessing with as few as 100 models, which can be trained on 1 GPU in two hours on all three of these datasets. Unfortunately, we are unable to make our code public at this time due to organizational constraints.

C Extended Results on Teacher Dataset Privacy

Weplot the effectiveness of Transfer LiRA in Figure 7. ROC curves for our student attacks are found in Figure 8. Further qualitative examples can be found in Figure 9. Ablation of score information with and without duplicates is plotted in Figure 10. Per-example student attack success rates for CIFAR-10 with duplicates are found in Figure 11. In Figure 12, we compare our student model attacks against a simple logit threshold baseline, similar to the loss thresholding attack designed by Yeom, Giacomelli, Fredrikson, and Jha [YGFJ18], which was used to evaluate distillation privacy in Shejwalkar and Houmansadr [SH21].

D Privacy of Student Training Set

Having evaluated the Private Teacher threat model, we now turn to the Private Student and SelfDistillation threat models, which we will consider simultaneously. The Private Student threat model can be used to perform knowledge transfer from large, general purpose models to task-specific models, by querying on (sensitive) task-specific student data. Self-distillation is often used in applications of distillation to compress models and improve their performance.

D.1 Private Student

The private student threat model does not involve data minimization, unlike the private teacher threat model; the empirical privacy we investigate here comes instead from an adversary having limited knowledge of the specifics of the teacher model. That is, the question we investigate is: how much

(a) CIFAR-10

(b) WikiText

(c) Texas-100

(d) Purchase-100

1.0

1.0

1.0

1.0

+

R=0.29

+

R=0.90

+

R=0.46

+

R=0.63

0.9

y=x

0.9

y=x

0.9

y=x

0.9

y=x

IW

0.8

0.8

0.8

0.8

LiRA I

0.7

0.7

0.7

0.7

Transfer

0.6

0.6

0.6

0.6

0.5

0.5

0.6

0.7

0.8

0.9

1.0

0.5

0.5

0.6

TeacherMlAccuracy

0.7

0.8

0.9

1.0

0.5

0.5

0.6

0.7

0.8

0.9

1.0

0.5

0.5

0.6

0.7

0.8

0.9

1.0

10

3

10

2

10

1

10 0

False Positive Rate

10

3

10

2

10

1

10 0

True Positive Rate

LiRA on Teacher

Transfer LiRA

End-to-End LiRA

Random Guess

(a) CIFAR-10

10

3

10

2

10

1

10

0

False Positive Rate

10

3

10

2

10

1

10

0

(b) WikiText103

10

3

10

2

10

1

10 0

False Positive Rate

10

3

10

2

10

1

10 0

(c) Texas-100

10

3

10

2

10

1

10 0

False Positive Rate

10

3

10

2

10

1

10 0

(d) Purchase-100

does the adversary need to know about the teacher model to get reliable attacks on the private student dataset?

Weconsider three levels of adversarial knowledge: Known Teacher , where the adversary knows the precise teacher model used to query the student examples; Unknown Teacher , where the adversary knows the teacher model is one of a small subset of models; and Surrogate , where the adversary can only collect similar data, to train their own surrogate teacher models. Both the Known and Unknown Teacher settings reflect a world where the teacher model is one of a small number of general purpose public models, such as a large language model. The Surrogate setting requires the adversary to train their own copy.

We run the LiRA variants in a number of these settings on the CIFAR-10 dataset, calibrated to the knowledge the adversary has (for example, in the Surrogate threat model, the adversary trains their own teacher models, and trains a number of shadow student models to calibrate LiRA). We plot our results in Figure 13a, and find that, as expected, less knowledge about the teacher model reduces the adversary's success at membership inference. However, even the weakest threat model, Surrogate, allows for powerful attacks, with a TPR as large as 10 -2 at a FPR of 10 -3 .

D.2 Privacy of Self-Distillation

Having considered the privacy of the student and teacher datasets independently, we now investigate the common self-distillation setting [FLTIA18; XLHL20], where the student and teachers are identical. Given that duplicate examples in the student set carry membership information of teacher examples (Section 6.1), and student examples themselves are not well protected by distillation (Section D.1), we do not expect self-distillation to reduce privacy risk significantly. However, a common technique in self-distillation is to train the student on a loss function which combines the cross entropy loss on the query dataset glyph[lscript] Q with the cross entropy loss on the student examples' original 'hard labels' glyph[lscript] S . We write glyph[lscript] α = αglyph[lscript] Q +(1 -α ) glyph[lscript] S , so that α = 1 recovers the standard distillation

(a) Target

(b) Student Queries

行

(c) Target

(d) Student Queries

10

4

10

2

10 0

False Positive Rate

10

4

10

3

10

2

10

1

10 0

True Positive Rate

Dups

Original

Denoised

No Dups

Original

Denoised

No Dups

Original

Denoised

objective, while α = 0 recovers the standard cross entropy loss (as if there was never a teacher model).

To evaluate self-distillation, we run LiRA by training shadow student models with the entire selfdistillation algorithm, using identical datasets for each pair of teacher and student shadow models. We perform calibration on these shadow student models, and plot our results at a range of α values in Figure 13b. While we don't observe a large effect, it appears that larger α (that is, heavier reliance on the distillation loss function) results in better attacks. This is likely because relying on the distillation loss function reinforces the memorization from the teacher even further in the second round of training on the student.

MI Accuracy

1.0

+

R=0.81

0.9

y=x

0.8

LiRA

0.7

End-to-End

0.6

0.5

0.5

0.6

0.7

0.8

0.9

1.0

TeacherMlAccuracy

10

3

10

2

10

1

10 0

False Positive Rate

10

3

10

2

10

1

10 0

True Positive Rate

Known

Unknown

Surrogate

y=x

(a) Private Student

10

3

10

2

10

1

10 0

False Positive Rate

10

3

10

2

10

1

10 0

True Positive Rate

=1

=0.5

=0

y=x

(b) Self-Distillation

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMultiple previous works have proposed knowledge distillation techniques to distill the knowledge of a teacher trained on sensitive data into a student model which is supposedly protected against membership inference attacks. This paper proposes a new membership inference attack to perform membership inference attacks on the student model. Attacking the student models, it is shown that knowledge distillation provides only limited privacy against membership inference attacks. Having access to the dataset used to train the student model but not the student model itself, it is shown that this suffices to attack the private data of the teacher model.\n\nSTRENGTHS:\n- **Originality:** The paper is novel and proposes a novel approach for membership inference attacks on student models.\n- **Quality:** The experimental results seem sound and the claims of the paper are well supported.\n- **Clarity:** Overall, the paper is mostly well written and well organized. \n- **Significance:** With previous work having proposed knowledge distillation as a defense technique against membership inference, the results of the paper are important and most likely of high impact to other researchers.\n\nWEAKNESSES:\n**Clarity:**  \n- The setting and the assumptions made for the proposed attacks are not very clear, and it would be helpful for the reader to explain the assumptions and information available to the attacker in more detail.\n- unfortunately, the code used to run the experiments will not be made public, which hinders reproducibility\n\nMisc:\n- Line 94: \"mitigates prevent\". There seems to be some missing or additional word.\n- maybe you could mention in the captions that “+” in the plots means the positive correlation. Otherwise, it is a bit confusing, if the reader is searching for a blue plus in the plots.\n\nQUESTIONS:\n**Q1:** The per example membership inference attack success rate is calculated over 1000 models. However, how do these models differ? Were they trained on the same data but with different seeds? If so, aren't these models behaving very similar?  \n**Q2:** In line 241 the indirect attack is described. However, the procedure of the attack is not quite clear to me. As far as I understood, the attacker has no knowledge about the teacher model's predictions on the teacher examples (line 231). Could you clarify what the assumptions made for this attack are and on how the attacker is fitting the Gaussians in this case?\n\nLIMITATIONS:\n- the limitations are appropriately addressed in the supplementary material", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper examines the effectiveness of model distillation in protecting the privacy of training data. Through the use of membership inference attacks, the authors demonstrate that distillation alone provides limited privacy across various domains. The authors also suggest several design considerations for improving privacy in model distillation, such as deduplicating the teacher set and considering complementary techniques like differentially private training.\n\nSTRENGTHS:\n- Trendy topic\n- Well-organized paper\n- Extensive experiments\n\nWEAKNESSES:\n- Attack detail in Section 5 is not clear\n- Additional discussion might be helpful\n\nQUESTIONS:\nThis paper provides valuable insights by addressing the misconception that model distillation can effectively protect the privacy of training data. The paper is well-organized and easy to follow, and a thorough examination of potential influencing factors contributes to the strength of the research.\n\nHowever, I have some concerns regarding the details of the attacks and the interpretation of results:\n\n- Regarding the experimental setting for End-to-End LiRA, it would be helpful to clarify how the corresponding student dataset was chosen. This information would enhance the reproducibility and validity of the experiments.\n\n- The paper mentions that attacks on the student model should not be more powerful than those on the teacher model, as implied by the data processing inequality. However, Figure 3 reveals that some samples remain more vulnerable after distillation. It would be beneficial to provide a possible explanation for this observation, as it appears to contradict the expected outcome.\n\n- Section 5, which introduces a new attack that only utilizes the student query dataset, is difficult to understand. The authors claim that “for each teacher example $z^T_j$, we fit a Gaussian distribution to the logits of each student example $z^S_i$, when $z^T_j$ is either IN or OUT.” However, it is unclear how the authors establish a link between student examples and arbitrary teacher examples, and how the student examples can indicate the membership status of the teacher example. Further clarification and additional illustrations would greatly enhance understanding in this section.\n\nOverall, this paper makes significant contributions and offers valuable insights. Addressing the aforementioned concerns would further strengthen the research and improve the clarity of the presented findings.\n\nLIMITATIONS:\nThe authors have adequately claimed their limitations in Appendix B.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores the privacy implications of model distillation, a technique used to transfer knowledge from a teacher model to a student model. The authors investigate membership inference attacks on both the teacher and student training sets to evaluate the privacy provided by distillation. \nThe authors extend the LiRA attack to the distillation setting in two ways. The first attack calibrates the attack only on the teacher models and the second attack uses the whole distillation procedure. They find that distillation alone provides limited privacy protection. The attacks on distilled models succeed even though the distilled models have never seen the teacher's data directly.\nFinally, they also highlight the importance of considering factors such as data duplication, teacher set poisoning, temperature scaling and distribution shifts when evaluating the privacy risks of distillation.\n\nSTRENGTHS:\n- The authors address an important problem. As machine learning capabilities grow and models are deployed directly to personal devices with limited computing capability, model compression algorithms such as distillation are often used. Understanding the privacy risk of these algorithms is therefore an increasingly important topic.\n- The paper is very easy to read and well motivated. Particularly, the visualizations in figure 3 and 4 are very intuitive and help tell the story\n- The paper considers many experimental evaluation settings (albeit limited detail about the concrete experiments)\n\nWEAKNESSES:\n- An obvious mitigation of membership inference attacks is Differential Privacy, it would be interesting to see how effective DP mitigates the attack as by Mireshghallah et al [2022].\n- Limited detail of experimental setup. What is the utility of the models?\n- Some conclusions seem not deeply thought through. E.g. Line 213, the authors claim that since there are examples for which vulnerability drops by only  single digit percentage points and since privacy is a worst case guarantee the authors conclude that distillation provides limited privacy. While I agree that distillation provides limited privacy, it is possible that least vulnerable points see the single digit drop in vulnerability whereas the most vulnerable points see a more significant drop.\n\n\n**References**\n\nMireshghallah, Fatemehsadat, et al. \"Differentially private model compression.\" Advances in Neural Information Processing Systems 35 (2022): 29468-29483.\n\nQUESTIONS:\nWhat is the utility of the models? Is privacy simply correlated with utility? Do we see the same percentage point drops in model utility when going from teacher to student?\n\nLIMITATIONS:\nThe authors have accurately described limitations albeit only in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors in this paper investigate the efficacy of membership inference attacks (MIA) in model distillation. Their novel attack(s) show that MIA is possible even when the teacher model is only queried on the most influential points in the student inputs. Finally, they also demonstrate how their attacks are the strongest when the teacher set is poisoned OR when the student set ~= teacher set.\n\nSTRENGTHS:\n1. The novelty is strong in this paper. No one has previously tackled MIA in model distillation as the authors here do. \n2. The paper is also very easy to read and understand. \n3. I really like the novel idea present in this paper (specifically Figure 1): membership of a target example in the teacher’s training set can be derived through querying the teacher model on various student set examples (and these are entirely different examples from the target!). \n4. Figure 3 is a great result as well. It might not be as obvious as it is, because the data points are essentially “post-processed” and there should be some sense of privacy after the distillation process. The authors here, show that distillation doesn’t show much privacy benefits. \n5. Their investigation on why MIA works well for student models is sound. I really like Figure 5 as well, which shows the MIA effects through duplicates, data poisoning and temperature scaling and the section on student-teacher data drift was an amazing read :-)\n\nWEAKNESSES:\n1. I don’t really have any major weaknesses to point out!\n\nQUESTIONS:\n1. Could this happen because of correlated features present between the student queries and the target example? Acc to Figure 1, the “red” colour present in the student examples are essential for the MIA to work. My intuition is that a higher correlation in features between the student inputs and the teacher target example, should lead to higher success in MIA. Of course, this may be written down as Future Work and I’m not quite sure if such work has been done in this space. I think it might be nice to investigate such correlation effects between the student and teacher examples. \n2. I noticed that all the attacks mentioned in this paper, were based off LiRA. Is there a specific reason why the authors succumbed to using LiRA and not other attacks. (It's totally okay to use LiRA but I wonder if there could be other attacks you could have tried/did try out and measure the efficacy of such attacks against the LiRA-based ones; provided you have time to run these expts).\n\nLIMITATIONS:\n1. Need to run a number of shadow models for running a successful MIA (not really a limitation of this approach, but this is a limitation of MIA itself!).", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-09"}