Students Parrot Their Teachers: Membership Inference on Model Distillation
Authors: Matthew Jagielski
Milad Nasr
Katherine Lee
Christopher A. Choquette-Choo
Nicholas Carlini
Florian Tramèr
Venue: NeurIPS 2023 oral (2023)
Keywords: model distillation
membership inference
privacy
dark knowledge
TL;DR: Model distillation is not resistant to strong membership inference attacks, and we investigate why.

Abstract: Model distillation is frequently proposed as a technique to reduce the privacy leakage of machine learning. These empirical privacy defenses rely on the intuition that distilled ``student'' models protect the privacy of training data, as they only interact with this data indirectly through a ``teacher'' model. In this work, we design membership inference attacks to systematically study the privacy provided by knowledge distillation to both the teacher and student training sets. Our new attacks show that distillation alone provides only limited privacy across a number of domains. We explain the success of our attacks on distillation by showing that membership inference attacks on a private dataset can succeed even if the target model is never queried on any actual training points, but only on inputs whose predictions are highly influenced by training data. Finally, we show that our attacks are strongest when student and teacher sets are similar, or when the attacker can poison the teacher set.

Students Parrot Their Teachers: Membership Inference on Model Distillation
Matthew Jagielski! Christopher Choquette-Choo
Milad Nasr' Nicholas Carlini'
Katherine Lee'
Florian Tramèr²
Google DeepMind 2 ETH Zurich
Abstract
Model distillation is frequently proposed as technique to reduce the privacy leak- age of machine learning. These empirical privacy defenses rely on the intuition that distilled "student" models protect the privacy of training data, as they only interact with this data indirectly through a "teacher" model. In this work, we de- sign membership inference attacks to systematically study the privacy provided by knowledge distillation to both the teacher and student training sets. Our new attacks show that distillation alone provides only limited privacy across number domains. We explain the success our attacks on distillation by showing that membership inference attacks private dataset can succeed even if the target model is never queried on any actual training points, but only on inputs whose predictions highly influenced by training data. Finally, show that at- tacks are strongest when student and teacher sets are similar, or when the attacker can poison the teacher set.
Introduction
Model distillation [HVD+15] is a common framework for knowledge transfer, where knowledge learned by a "teacher model" is transferred to a "student model" via the teacher's predictions. Dis- tillation is helpful because the teacher's predictions are a more useful guide for the student model than hard labels; this phenomenon has been explained by the teacher's predictions containing some useful "dark knowledge". Variants of model distillation have been proposed for, e.g., model com- pression [HDD+15] BC14 PPA18 KPKI8, SCGL19] or training more accurate models [ZK16] XLHL20 Within the privacy pereeccireerriinggg machine learning community, distillation has been adapted to protect the privacy of a training dataset [PAEGT16] TMSSNHM22 (S2T MHHVEHP2]]
Many of these approaches rely on the intuition that distilling the teacher model serves as privacy barrier that protects the teacher's training data. Informally, restricting the student learn only from the teacher's predictions is form of data minimization, which should result in less private information being fed into, and memorized by, the student. This privacy barrier around the teacher also allows the teacher model to be trained with strong, non-private, training approaches, improving both the teacher model's 's and student model's 's accuracy.
Because model distillation does not provide a rigorous privacy guarantee (such as those offered by differential privacy (DMNS06]), in our work we evaluate the empirical privacy provided by these schemes. We show that distillation is vulnerable to membership inference attacks-2 well-studied class of privacy attacks on machine learning (SSSS17 YGHIT8).
We adapt the state-of-the-ar Likelihood Ratio Attack (LiRA) [CCNSTT22 to the distillation set- ting, and find that this attack works surprisingly well inferring membership of the teacher's train- ing data. For some training examples, model distillation fails to appropriately protect against mem-
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(a) Target
(b) Indirect Student Queries
Figure I: We can predict the membership status of a target example (a) in the teacher model's train- ing set by querying the teacher model on different student training examples (b). Our membership inference attack uses only the student model's predictions on the ten images on the right to reach 73% accuracy at predicting membership of the target image. Interestingly, while the target example a bird, only two of the most informative student queries are birds.
bership inference. To explain this finding. we show for the first time that the membership presence of some examples can be inferred based the model's predictions on other, seemingly unrelated examples. This observation provides new insights into how membership information is transmitted from the teacher to the student.
Figure gives an example of such cross-example leakage. A teacher model trained on the red parrot in Figure la (labeled as "bird"), never seeing the student queries on the right in Figure [bb encodes membership information about the parrot in predictions on these student queries. Interestingly, the most informative student queries are not bhrds-they are images of other classes with similar red hues that the model confuses as being "bird-like" because of the influence of the parrot in the teacher's 's training set. In other words, the student model manages to "parrot" its teacher using the peculiarities transferred through distillation.
We systematically evaluate a number of factors which impact the empirical privacy of model distil- lation. We find that similarity between the training datasets of the teacher and student models lead to increased privacy risk, along with higher temperature parameters. Moreover, an adversary capa- ble of poisoning the teacher training set can amplify privacy risk for teacher examples, in line with recent work (TZJRRI6: CSSWW22).. We also find that distillation provides little privacy protection to student examples. We hope our attacks can assist experts in properly evaluating the privacy risks resulting from model distillation. Our work also highlights two possible approaches for mitigating privacy risks in distillation: deduplicating the teacher and student datasets, and reducing the leakage from the teacher model, e.g., using provable guarantees such as differential privacy (DMNS06).
2 Background and Related Work
2.1 Machine Learning Privacy
Machine learning models are known to be vulnerable to a variety of privacy attacks, including membership inference attacks (SSSS17), attribute inference attacks (FJR15), property inference at- tacks [GWYGB18), and training data extraction [CTWJHVLRBSEA42] Each of these captures a different type of leakage about the training data or individual examples.
In this work, we focus on membership inference, as it is the most widely studied privacy attack. In membership inference attack, an adversary tries to determine whether or not particular example was used to train a model. There have been a number of membership inference attacks proposed in the literature, which generally compute some "membership score". which is designed to be infor- mative of a target example's 's membership. Most of these scores make use of the model's 's prediction on the target example, perhaps compute the example's ' loss (YGFJT8) SDSOJ19] to compute some other score [WGCS21 SSSS17 CCNSTT22) Other attacks rely on querying the model with examples nearby or derived from the target [WK2GH20 LBWBWIGG18) L721 CCTCP2T WBKBBGG22). One of our contributions is to design an attack which performs well despite relying on the model's predictions on entirely different examples from the target.
Our attacks are based on the state-of-the-art Likelihood Ratio Attack (LiRA) (CNSST222) In LiRA, the adversary first trains many shadow models, such that half of these models will contain given target example (z, y) (the IN models), and half will not (the OUT models). Next, the adversary queries each shadow model f. to compute the logits corresponding to the correct class, f(x and fits Gaussian N (Pin o? ) to the logits for all models containing the example (and similarly for the

---

Train model
Query fr
Train model
Dr
fr
DQ
fs
on Dr
with
Ds
on DQ
Figure 2: Knowledge distillation is a multi-step training process. A teacher model fr is first trained on a teacher dataset Dr. A dataset of images annotated by the teacher Dę is generated by querying the teacher fr on an (unlabeled) student training set Ds. Finally, a student model fs is trained on Do.
OUT models). To attack a new model f', the adversary computes the probability density function (PDF) of each Gaussian (which we write Pin (f y and Pout (x))))) and then computes the likelihood ratio, which serves as the membership score. Our attacks in this paper will be derived from LiRA but adapted the variety of distillation settings we consider.
2.2 Knowledge Distillation
Knowledge distillation (HVD+15 BC14] is a technique for transferring knowledge from a "teacher" model to "student" model. There two datasets in knowledge distillation: the teacher dataset Dr = {z,, J. with n, examples, and the student dataset Ds = 3e with n, ex- amples. Distillation begins by training teacher model fr the teacher dataset Dr. The teacher model is used to generate (soft) labels for the student dataset, producing a query dataset DQ {z,, ((:()) where the softmax function S converts logits into probability vec- tor. Training on this query dataset then produces a student model fs. the output of distilla- tion. Sometimes, distillation includes "temperature" scaling, where the logits fr( are scaled by temperature parameter H, before the softmax is applied. The query dataset will instead be Do {zi, (()(())) }- setting H recovers the original distillation procedure. We illustrate this process in Figure
Knowledge distillation in private machine learning. Prior work has suggested that distillation mitigates prevent privacy attacks. Perhaps the most well known example is the PATE frame- work [6] where distillation is used to reduce an ensemble of teacher models into a single model, in such way that the final model has provable differential privacy guarantees (DMNS06). Zheng, Cao, and Wang (ZCW2I) and Tang, Mahloujifar, Song, Shejwalkar, Nasr, Houmansadr, and Mittal [TMSSNHM22) construct ensembles of models that are designed be private and use distillation to condense these models. Mireshghallah, Backurs, Inan, Wutschitz, and Kulka- mi [MBIWK22] enforces differential privacy guarantee by training the teacher and student models with differential privacy. In each of these approaches, distillation is only one component; an un- derlying ensemble, or provable differential privacy guarantees, may also improve the privacy of the overall approach. In our work, we focus on the distillation procedure itself, and leave to future work the task of designing attacks on these more complicated approaches. Importantly, also, those works which offer provable privacy guarantees will have hard limits attack effectiveness.
Indeed, other prior work has already suggested that using distillation alone protect privacy. She- jwalkar and Houmansadr (SH21) propose defense that relies on a sufficient b distance between teacher and student examples and limited entropy the queries. We evaluate this defense Sec- tion 6.4 and Appendix D.I Mazzone, Heuvel, Huber, Verdecchia, Everts, Hahn, and Peter [MH-] HVEHP22] consider using repeated distillation to prevent membership inference attacks. In our work, we will design stronger membership inference attacks to adaptively evaluate the privacy pro- vided by distillation.
3 Threat Model and Experimental Setting
3.1 Threat Model
We investigate the ability of distillation to protect against membership inference attacks in three threat models:
Private Teacher. The teacher dataset Dr is sensitive and the student dataset Dg nonsen- sitive. We assume the adversary has knowledge of the student dataset. This threat model
3

---

is used in most private machine learning approaches, We evaluate this threat model in the main body of the paper.
2. Private Student. The teacher dataset nonsensitive and the student dataset sensitive. We assume the adversary has access to the teacher dataset. This threat model may be used to transfer knowledge from a foundation model used as a teacher to a sensitive downstream task [LZZHCZ22]. Because this threat model is less well considered in the privacy litera- ture SH21J, we evaluate this threat model in Appendix D.I]
3. Self-Distillation.
Self-distillation is commonly used to refer to the setting where the teacher and student datasets are identical. Self-distillation is commonly used when dis- tillation is used to improve model performance or during model compression. We evaluate this threat model in Section D.2
For each of these threat models, we measure the success of the adversary at performing membership inference on the sensitive dataset, both by measuring the true positive rate (TPR) and false positive rate (FPR) as suggested by Carlini, Chien, Nasr, Song, Terzis, and Tramer [CCNSTT22), and also by investigating the membership inference accuracy on each individual example, as done in Carlini, Jagielski, Zhang, Papernot, Terzis, and Tramer [CIZPTT22]. In general, we will report this per- example membership inference accuracy when presenting the performance of one two attacks, due to the amount of information this metric conveys, and use ROC curves to compare between more than two.
Beyond distillation's 's potential uses in privacy, our attacks on distillation have implications for private information leaked during learning- based model extraction attacks [TZJRRI6 PGSKSG22 JCBKP20], which often resemble model distillation. model extraction, an adversary uses API access to target model to reproduce its functionality into the weights of local model. The target model is analogous to the teacher model in distillation, and the local model is analogous to the student. Then attacks in the Private Teacher threat model can be cast instead allowing an adversary to inspect their local model to perform membership inference attacks on the target model's s training data, without directly querying it. The Private Student threat model has implications for defensive setting in model extraction, where the target model's owner wants to link queries made to the target model with model they believe has been extracted from their model.
3.2 Experimental Setting
We study four standard datasets for our analysis: CIFAR-10, WikiText103, Purchase100, and Tex On CIFAR-10, we start with code from the DAWNBench benchmark [CNKZZN- BORZ17) which trains an accurate ResNet-9 model in under 15 seconds; we adapt this code to support model distillation and the subsampling required for LiRA variants. We remove 5275 dupli- cates from CIFAR-10, 100, using the imagededup library [LJT11., and split the remaining dataset into a teacher set of 30,000 examples and a student set of 14,725 examples. On WikiText103, we used the GPT-2 architecture (with context window of 256 tokens, 4 heads, 4 layers and embedding size of 256 dimensions). We split WikiText103 into a teacher set of 500,000 records, and use the re- maining records to train the student models. On Purchase100 and Texas100, we train single-layer neural networks with hidden layer sizes of 256 and 512, respectively, and subsample the datasets to produce teacher and student sets of 20000 examples each. On Purchase 100, student models reach 74-75% accuracy, and on Texas100 they reach 54-55% accuracy. On all datasets, we train our models with the cross entropy loss: teacher models are trained with the standard sparse cross en- tropy loss on the teacher dataset, and student models are trained with a dense cross entropy loss to mimic the soft labels predicted by the teacher. Unless otherwise stated, all LiRA- attacks use 100 shadow models for calibration, and all figures are produced by running the attack on over 1000 models. Needing train a large number of shadow models one limitation of our attacks, and any shadow model-based membership inference attacks; we comment on training efficiency and code for our experiments in Appendix
4 Evaluating Privacy of the Sensitive Teacher Training Set
The most common way prior work in privacy has used distillation to improve the privacy of the teacher set. Intuitively, distillation protects the teacher set because the adversary can only interact

---

End USA accuracy
1.0
13
10
0.9
yoX
0.8
0.5
08
0.5
0.6
35
0.6
0.5
3.5
0.7
0.8
0.9
1.0
0.5
Bn
A8
1.0
0.
0.7 C/ Teachar Accuracy
Teacher Accuracy
Teacher Accuuancy
Teacher KI Accuracy
(a) CIFAR-10
(b) WikiText103
(c) Texas- 100
(d) Purchase- 100
Figure 3: Many data points get privacy benefits from distillation. On the axis, plot the vulnerability of each teacher example to membership inference before distillation, using teacher models. On the y axis, plot the vulnerability of each teacher example attack after distillation, using the End-to-End LiRA strategy. Observe that many data points (very small blue plus signs) lie near the y = line, which indicates no reduction in vulnerability from distillation.
with the student model, which never sees any teacher examples. In fact, because the student model can be seen as a ipost-processing of the teacher model, the data processing inequality provably implies that attacks be no more powerful the student model than they
the teacher.
Stepping through the distillation process can help us anticipate how distillation can impact the pri- vacy of the teacher set. The first step, training the teacher model, is the most well-understood from a privacy perspective, as the large literature on membership inference applies to the teacher model. In particular, recent work has found that state-of-the-art membership inference attacks are better at attacking some "outlier" examples than other "inlier" examples [CCNSTT22), which we expect to be true the teacher dataset as well, making these "inlier" examples less vulnerable in later steps, as well.
Subsequent steps of distillation are less well-explored by the privacy literature. The second step of distillation creates the query dataset, which can be seen as "compressing" the teacher model into its responses on these queries. Intuitively, this step is the most important at reducing private information leakage. However, we hypothesize that some queries will capture information about teacher examples, perhaps due to some similarity between the queries and teacher examples. In the final step, the student model is trained on the query set. While this step cannot contain more sensitive information than the queries themselves, it is possible that this step makes that information easier to discover, perhaps by interpolating between the queries.
4.1 Performing Membership Inference on Teacher Examples
We now show that membership inference attacks work surprisingly well on distilled student models.
We first propose the "Transfer LiRA" attack, simple extension of LiRA to the distillation setting. In this attack, we train shadow teacher models. We calibrate the IN and OUT Gaussians for LiRA on these shadow teacher models, and then run the attack by applying directly to the student models. Notice that in this attack, distillation never happens in training shadow models the success of this attack relies the similarity between teacher and student models' predictions.
To capture the information loss because Transfer LiRA does not use distillation in any way, we also propose a second attack, which we call "End-to-Fnn LiRA", where we instead train shadow models with the entire distillation procedure, first training shadow teacher models, and then distilling these teacher models into shadow student models. We then calibrate LiRA using these shadow student models. Note that performing this attack requires knowledge of the student training set to query the shadow teacher models and train the shadow student models. This is in contrast to Transfer LiRA which does not, although still requires access to in-distribution data to train teacher models.
Distillation provides limited privacy. In F we plot the change in per-example attack success rate on distilled models. We compare each example's vulnerability to LiRA on the teacher model (i.e. vulnerability without distillation) on the x-axis, to each example's 's vulnerability to End-to-Eni LiRA (i.e. vulnerability after distillation) on the y-axis. In the Appendix, we provide plots for
5

---

1.0
1.0
1.0
Accuracy
R=0.77
Rm0.73
Rm0.75
0.9
0.9
0.7
0.7
0.7
0.6
0.6
0.6
0.6
1.0
0.5
0.7
0.8
0.9
1.0
0.5
0.6
0.7
1.0
Teacher MI Accuracy
Teacher MI Accuracy
Teacher MI Accuracy
(a) CIFAR-10
(b) Texas-100
(c) Purchase-100
Per-example membership inference accuracy using only student queries, compared to LiRA accuracy on the teacher model. Membership inference accuracies are non-trivial, and remain high for some examples. Effectiveness is dataset- and seample-dependee with Texas-100 having the most vulnerable examples in the teacher model remaining vulnerable to attacks based on teacher predictions.
Figure 4:
We also find our attack significantly outperforms a simple logit threshold baseline, similar to the weaker attacks used in prior work to evaluate distillation (Figure 12
Transfer LiRA (Figure 7. and ROC curves for each strategy (Figure g
For large fraction of teacher examples each dataset, we find End-to-End LiRA achieves nearly the same membership inference performance as directly attacking the teacher model. other words, many teacher examples do not observe any privacy benefits from distillation, despite student models never directly seeing these teacher examples! While the average membership inference accuracy (and TPR at low FPR) do decrease, 5% of examples' vulnerability drop by less than percentage points on CIFAR-10, 5 points on Purchase- 100, and 4 points on Texas-100 and these examples have a variety of teacher vulnerabilities. Because privacy worst-case guarantee, distillation provides limited privacy benefits.
We note two other interesting takeaways from these results. First, the per-example student attack success rate has a high variance. This variance is partly due to statistical uncertainty (although each example's i attack success rate computed with over 1000 models, giving each coordinate standard deviation 1.5 percentage points for both the x and y axes), but more interestingly, some examples do significant privacy benefits from distillation, even controlling for the original model's vulner- ability. We investigate duplication as one potential cause for this variance in Section 6. Second, each plot has positive correlation, meaning that examples that more vulnerable to attack on the teacher model attack also tend to be more vulnerable after distillation. As a result, reducing the vulnerability of the teacher model, perhaps using techniques such as differential privacy, are likely to improve the student's ' privacy.
Tracing Teacher Privacy Leakage Through Student Queries
It is surprising that membership inference still performs well on distilled student models, despite these models never directly using any of the teacher data. We now investigate why. For a mem- bership inference attack to be successful on the student model, it must be the case that the student queries reveal some membership information about the teacher examples. However, to rigorously evaluate how this information is encoded in the student queries, we will turn our attention to de- signing new attack that only has access to the student query dataset Do (and, in particular, has no knowledge of the teacher model's predictions on the teacher examples). To the best of our knowledge, this attack is also the only membership inference attack in the literature which does not require querying model directly its membership status.
example (or on algorithmically derived examples) to predict
An indirect attack using student queries. We adapt LiRA to setting where only the student query scores are available to the adversary. Because of this limitation, the adversary can only rely on information about teacher examples that is indirectly contained in the student query scores. To adapt the attack to this setting, we use the same approach that LiRA uses to combine queries on

---

multiple "augmentations" of an example: multiplying their likelihood ratios to produce an aggre- gate membership score. Concretely, for each teacher example 2/ (a"". .v]" ,,"), we fit a Gaussian distribution to the logits of each student example of (a v,"), when 2/ is either IN or OUT. We IN write the PDFs of these distributions Pj.a and Pj.+ p..TT so that the joint PDF of the IN Gaussians Df I. and similarly for the OUT Gaussians. This natural adjustment allows us to infer membership of the teacher examples using the student examples.
However, find that this direct adaptation of LiRA tends to have poor performance, two modifications which significantly improve the attack. First, we find that a teacher example tends to have more influence the logit corresponding to the teacher label vf than the student label y and so we choose to calibrate LiRA using the teacher label logit rather than the student label logit. Second, there tend be many uninformative student examples for each teacher example, so we filter the student queries for all teacher examples. We filter these student queries by selecting only those with the largest mean gap L - 11 1OU'", which most informative about the teacher. This removes most of the noise from our LiRA adaptation, improving it especially when we train few shadow models.
propose
Student queries leak private information. In Figure 4 we plot the per-example membership inference accuracy after distillation, using our student query attack with 4000 shadow models. We consider all datasets except WikiText103, which we omit due to the high computational cost of our attack. Our results are similar to when we attacked the student models: MIA vulnerability reduces on average, but many examples maintain nearly identical membership inference attack success rate using only the predictions on these indirect student queries. Cases where this attack achieves high membership inference accuracy indicate that a large amount of information about the teacher set encoded in the student queries.
Observations we noticed for our student model attacks persist for this query-based attack. Examples which are more vulnerable to attack in the teacher model also tend to be more vulnerable using teacher model predictions. This is true all datasets, although we also find the decay in vulnera- bility with queries to be dataset-dependent and example-dependent. Within each dataset, there is a high variance in student model membership inference accuracy.
We also remark on the counterintuitive fact that our prior attack with access to the student models outperformed this attack, despite the fact that the student model is just a post-processing of these student queries. This implies that the process of training a student model makes it easier to extract private information from these student queries, by interpolating between the student queries. In the following subsections, we will investigate some factors that can impact the success of the student query attack, and what this tells about distillation's 's dark knowledge.
Ablation. In Figure 5a ablate the modifications we made for our student query attack. We show both LiRA scores (logits for the student label) and Label scores (logits for the teacher label). We also show attacks with "All", without filtering, and "Filtered" scores, with filtering 10 student examples. Our modifications significantly improve membership inference: without either modifica- tion, the attack is no better than random chance. Applying both modifications achieves a TPR of 10-3 3 at a FPR of 10-4
Qualitative analysis. Our ablations also shed light on where membership information is contained in distillation's 's dark knowledge: predominantly in the logit corresponding to the teacher label within only a few student examples. This corroborates our intuition from the parrot in Figure T There, only two student queries were also in the bird class, emphasizing the importance of label scores. We inspect other vulnerable examples in Figure in Appendix C. finding some teacher examples whose most informative students mostly belong to their same class.
6 Factors Influencing Teacher Privacy Leakage
So far, we have designed two types of attack: the attack from Section 4 which uses the student model, and the attack from Section 5 which uses the student queries. These attacks find a surprising amount of leakage which could not have been identified by prior attacks. Now, we investigate more deeply the factors which may influence the leakage of the teacher.
7

---

Ro mape DMps
307
PoS3
Pasilue Itate
10m1
Query Mi Accurscy
peu
Querry Accusicy
Thee Postive Rate
URA Sconee
Hed
20-2
He1
Laadl Scrrc
M=0.5
Birend
H-0.1
y-X
10a False Posilvve Rate
20-1
Teacher Accuracy
Teacher NI Accuracy
Falis Positiiv Rate
(a) Ablation
(b) Duplicates
(c) Data Poisoning
(d) Temperature
Figure 5: Additional investigations into the success of attacks reveal: label scoring and score filtering are important for improving attack success; b) duplication between the teacher and stu- dent datasets increases privacy risk; c) data poisoning attacks amplify the performance of our indi- rect attack; d) temperature scaling causes mild changes in privacy vulnerability. All results are on CIFAR-10.
6.1 Student-Teacher Duplication
Unlike the preceding indirect queries, where student examples only share high-level similarities with their teacher examples, now investigate how increased student-teacher similarity impacts the performance of our student query attack. To so, we reintroduce the near-duplicates we re- moved from CIFAR-10, and compare the attack success rate between the duplicated and dedupli- cated student query attacks, on those teacher examples with duplicate student examples. Intuitively, student examples which are duplicates of teacher examples will carry membership information in their queries, which what we find in Figure 5b (p < 10 10-15 using a Chow Test). This indicates that deduplication significantly reduces privacy risk, although indirect queries still carry bership information. In the Appendix, we show student attacks are also worsened by duplication in Figure 11
6.2 Teacher Set Poisoning
Tramer, Shokri, San Joaquin, Le, Jagielski, Hong, and Carlini [TSS)LJHC22 and Chen, Shen, Shen, Wang, and Zhang [CSSWZ22] find that data poisoning attacks added into training set amplify membership inference vulnerability for other examples in the training set. Here, we investigate how distillation interacts with this effect; i.e., we measure whether poisoning in the teacher set can increase an example's 's vulnerability in the teacher predictions. In Figure 5c we evaluate this using student query attack CIFAR-10 using the label flipping poisoning strategy from prior work.
With higher poisoning counts, many examples have higher teacher membership inference accuracy, ie.. they shift the right on the z-axis; this is exactly in line with prior work. However, interestingly, we see that poisoning does not impact the relationship between teacher vulnerability and student vulnerability they move upward the y-axis identically to examples which were not poisoned.
We remark that our poisoning attack does not specifically target the student query set; an teresting open question whether poisoning attacks exist that could increase membership leakage on the student model, but with less (or more!) impact on the teacher model's vulnerability. Section 6. hints at such a strategy, if the adversary can poison the student set instead: if an adversary adds duplicates of target examples to the student set, the resulting student queries will increase risk on those target teacher examples.
6.3 Temperature Scaling
When introducing knowledge distillation, Hinton, Vinyals, Dean, el. [HVD+15) proposed a mod- ification known as temperature scaling. Temperature scaling makes two simple changes to distil- lation: (1) introducing a temperature hyperparameter H, which when increased, rescales the logits and flattens the resulting probability distribution of student queries, and (2) rescaling gradients by 1/H2 factor. Though normally set to 1, modifying this parameter can improve performance. is natural to consider whether such manipulation of the teacher outputs might improve empirical

---

privacy; indeed, Shejwalkar and Houmansadr (SH2T) evaluate their distillation bassd defense with various temperatures, and find that high temperature reduces private information.
We use our strong End-to-End LiRA to evaluate the empirical privacy of student models trained with H E [0.1, 4], shown in Figure 5d We find lower temperatures mildly less vulnerable than higher temperatures, reducing TPR by a factor of roughly 4 at an FPR of 10-3 when H decreases from 4 to 0.I; model accuracy at all of these temperatures is similar the baseline of H = While this trend is small, it is also intuitive: in Section [ we found that all logit values carry important membership information. When H is small, the entropy of query probabilities decreases, reducing information captured by these probabilities. We hypothesize that the reverse effect observed by Shejwalkar and Houmansadr (SH21) may be result of the accuracy decrease they found resulted from high temperatures (i.e. up to H = 10), which we do not observe.
interesting gap between adversary with access to student queries and one who only has access to a student model. With access to temperature-sealed student logits, an adversary can multiply by H to reverse the temperttte scaling cannot make attacks harder in this threat model.
Finally, we remark
However, it does appear that low temperatures lead to weaker attacks on the student model, which we do not know how to reverse. Indeed, a natural way to adaptively attack a temperature-scau student model is to rescale the student model's logits, but our LiRA accounts for this, LiRA is scale-invariant Adaptively attacking semperature-scalle models is an interesting open question.
6.4 Student Distribution Drift
In cases where the teacher dataset is sensitive, nonsensi- tive student data from the same distribution may be dif- ficult to find, leading to distribution drift between the teacher and student datasets. We now investigate whether this impacts attack success. From our results on student- teacher duplication, may expect the signal to diminish as distributions become more different. To evaluate this, consider a CIFAR-10 teacher model distilled with CIFAR- 100 student set; CIFAR-10 and CIFAR-100 have no class overlap but have the same size. Interestingly, we find that distilling with this different dataset actually improves attack performance, despite the resulting model being somewhat less accurate than after distilling on in- distribution data. We find that this is due to a higher en- tropy of the teacher model predictions, which can encode more membership information about the teacher exam- plcs. We run an experiment in which we train on the low- est entropy and highest entropy predictions from CIFAR- 100, as well as random student queries. All experiments use the same student set sizes. We present the results in Figure The CIFAR- 10 student still results in the lowest leakage, and increasing entropy of the CIFAR- 100 examples also increases the leakage monotonically. Note that the CIFAR-IO student predictions have a mean entropy of 0.24, while the lowest leakage CIFAR- 100 examples have a mean entropy of 0.39. Our results here can also be taken as an attack on the full Shejwalkar and Houmansadr (SH2T) defense, which removes high entropy and duplicate student examples. Our strong attacks allow us to better evaluate their defense.
100
10-1
Teacher Model
CIFAR-10
10-2
C100 Low
C100 Randam
C100 High
y=x
10
10
10"
False Positive Rate
Figure 6: A CIFAR-100 student set re- sults in more privacy leakage than a CIFAR-10 10 student set, result of the in- creased entropy of the CIFAR-100 pre- dictions.
6.5 Accuracy of the Student Model
It is common for privacy preserving training algorithms to exhibit a "privacy-utility" tradeoff: the stronger the privacy enforced by the model, the worse the model's predictive performance. By piecing together the accuracies of models trained throughout Section 6 we find that differences in attack performance cannot be explained by changes in model accuracy. For example, we find in Sec- tion 6.3 that increasing temperature increases vulnerability to our attacks but does not significantly impact model accuracy (maintaining roughly 88% accuracy for the student). Our results with dis- tribution drift in Section 6.4 show that distribution shift leads to higher vulnerability to our attacks,

---

but reduces model accuracy (from 88% to roughly 85% student model accuracy with CIFAR-100 distilled student). Finally, we find in Section 5b that deduplication reduces vulnerability our tacks, but actually increases model accuracy slightly (from roughly 87.5% to 88% student accuracy)! In summary, depending on the source of model performance change, increased vulnerability can coincide with increased, decreased, or unchanged model performance!
7 Conclusion
In this work, we have used membership inference attacks to empirically evaluate to what extent knowledge distillation protects the privacy of training data. For those interested in using model dis- tillation to improve privacy, our work offers three main design considerations. First, deduplicating the teacher set with respect the student set is necessary to reduce risk duplicated examples. Second, while average-case privacy can significantly improve after distillation, the worst-case vul- nerable examples often only marginal benefits. As result, techniques which make the teacher model more private, such as differentially private training, should be seen as complementary to distillation. And third, the student training set should not be seen to have improved privacy as result of model distillation. Our work also offers insights into the privacy properties of distillation' 's dark knowledge, which may be of broader interest. Our results also imply privacy attacks on model extraction attacks (TZJRRI6 JCBKP20] which rely on similar algorithms to knowledge distillation.
Limitations
The main limitation of our work is the strong threat model under which our attacks work. We use the same threat model as the "online" version of the LiRA [CCNSTT22] attack. This attack assumes access to the target examples before training model, and, for the End-to-End LiRA attack, access to the student dataset. We use this strong threat model to assess worst-case vulnerability, as prior work has evaluated distillation under weaker attacks. Another limitation of the attacks is the large running time of the student query attack from Section which requires thousands of shadow models to obtain good performance. While generally impractical, we prefer to position this attack way of explaining why distillation propagates membership information, and leave future work to attempt improve the attack's 's efficiency.
Acknowledgements
We would like to thank Andreas Terzis and Andreas Schou for helpful comments on our work.
References
[BC14]
J. Ba and R. Caruana. "Do deep nets really need to be deep?" In: Advances in neural information processing systems 27 (2014) (cit. pp. 3).
[CCNSTT22]
N. Carlini, S. Chien, M. Nasr, S. Song. A. Terzis, and F. Tramer. "Mem- bership inference attacks from first principles". In: 2022 IEEE Symposium
on Security and Privacy (SP). IEEE, 2022, pp. 1897-1914 (cit. on pp. I
4
4 5 10).
[CCTCP21]
C. A. Choquette-Choo F. Tramer, N. Carlini, and N. Papernot "Label- only membership inference attacks". In: International conference on ma- chine learning. PMLR. 2021, pp. 1964-1974 (cit. on P.2..
[CIZPTT22]
N. Carlini, M. Jagielski, C. Zhang, N. Papernot, A. Terzis, and F. Tramer. "The Privacy Onion Effect: Memorization is Relative". In: Advances in Neural Information Processing Systems. Ed. by A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho. 2022. URL: httpps://openrevie: net/forum?
Ed=ErULLrGaVEU (cit. on p. 4
[CNKZZNBORZI7]
C. Coleman, D. Narayanan, D. Kang. T. Zhao, J. Zhang, L. Nardi, P. Bailis, K. Olukotun, C. Ré, and M. Zaharia. "Dawnbench: An end-to-end deep learning benchmark and competition". In: Training 100.101 (2017), p. 102 (cit. on 4
10

---

[CSSWZ22]
Y. Chen, C. Shen, Y. Shen, C. Wang, and Y. Zhang. "Amplifying Member- ship Exposure via Data Poisoning" In: Advances in Neural Information Processing Systems. Ed. by A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho. 2022 (cit. on pp. 28.
[CTWJHVLRBSE+21]
Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert- Voss, K. Lee, A. Roberts, T. Brown, D. Song. U. Erlingsson, al. "Extracting training data from large language models". In: 30th USENIX Security Symposium (USENIX Security 2/). 2021, pp. 2633-2650 (cit. on p.2).
[DMNS06]
C. Dwork, F. McSherry, K. Nissim, and A. Smith. "Calibrating noise to sensitivity in private data analysis". In: Theory of cryptography confer- ence. Springer. 2006, pp. 265 284 (cit. on pp. D
[FJR15]
M. Fredrikson. S. Jha, and T. Ristenpart. "Model inversion attacks that ex- ploit confidence information and basic countermeasures" In: Pr Proceedings of the 22nd ACM SIGSAC conference on computer and communications security. 2015, pp.
(cit. p. 2).
[FLTIA88]
T. Furlanello, Z. Lipton. M. Tschannen, L. Itti. and A. Anandkumar. "Born again neural networks". In: International Conference on Machine Learn- ing. PMLR. 2018, pp. 1607-1616 (cit. on p. [4).
[GWYGB18]
K. Ganju, Q. Wang, W. Yang. C. A. Gunter, and N. Borisov. "Property inference attacks fully connected neural networks using permutation invariant representations". In: Proceedings of the 2018 ACM SIGSAC con- ference computer and communications security. 2018, pp. 619-633
(cit. on p. 2
[HVD+15]
G. Hinton, O. Vinyals, J. Dean, et al. "Distilling the knowledge in a neural network". In: arXiv preprint arXiv: 02531 2.7 (2015) (cit. on pp.
3
[JCBK220]
M. Jagielski, N. Carlini, D. Berthelot, A. Kurakin, and N. Paperot. "High accuracy and high fidelity extraction of neural networks". In: 29th USENIX security symposium (USENIX Security 20). 2020, pp. 1345-1362 (cit. on pp. 10P.
[JLJT19]
T. Jain, C. Lennan, Z. John, and D. Tran. Imagededup http::p/tttt::::/ttt:te //gitbub com/idea comm/ddealo/imageddaup 2019 (cit. on p. (4.
[JWKGE20]
B. Jayaraman, L. Wang, K. Knipmeyer, Q. Gu, and Evans. "Revisit- ing membership inference under realistic assumptions". In: arXiv preprint arXiv:2005. 1088/ (2020) (cit. on p.Z).
[KPK18]
J. Kim, S. Park, and N. Kwak. "Paraphrasing complex network: Network compression via factor transfer". In: Advances in neural information pro- cessing systems 31 (2018) (cit. on p.
[LBWBWTGC18]
Y. Long, V. Bindschaedler, L. Wang, D. Bu, X. Wang. H. Tang, C. A. Gunter, and K. Chen. "Understanding membership inferences on well- generalized learning models". . In: arXiv preprint arXiv: 04889 (2018) (cit. on p. 2
[LZ21]
Z. Li and Y. Zhang. "Membership leakage in label-only exposures". In: Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security. 2021, pp. 880-895 (cit. on p.D
[LZZHCZ22]
C. LLang, S. Zuo. Q. Zhang, P. He, W. Chen, and T. Zhao. "Less is More: Task-aware Layer-wise Distillation for Language Model Compression". In: arXiv preprint 2xXiv:220000333 (2022) (cit. on F.
[MBIWK22]
F. Mireshghallah, A. Backurs, H. A. Inan, L. Wutschitz, and J. Kulka- mi. "Differentially Private Model Compression". In: arXiv preprint ar arXiv:2206.0 01888 (2022) (cit. on P.
[MHVVEH22]]
F. Mazzone, L. van den Heuvel, M. Huber, C. Verdecchia, M. Everts. F. Hahn, and A. Peter. "Repeated Knowledge Distillation with Confidence Masking Mitigate Membership Inference Attacks". In: Proceedings of the 15th ACM Workshop on Artificial Intelligence and Security. AlSec'22 Los Angeles, CA, USA: Association for Computing Machinery, 2022,

---

13-24. ISBN: 978 9781450398800. URL: https: doi org 10 1145 3560830 3563721 (cit. on pP. DB.
[OSF19]
T. Orekondy, B. Schiele, and M. Fritz. "Knockoff nets: Stealing function- ality of black-box models". In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019, pp. 4954 4963 (cit. on 4.
[PAEGTI6]
N. Papernot, M. Abadi. U. Erlingsson, Goodfellow, and K. Talwar. "Semi-supervised knowledge transfer for deep learning from private train- ing data". In: arXiv preprint arXiv:1610.0577 (2016) (cit. on pp. [ S. Pal, Y. Gupta, A. Shukla, Kanade, Shevade, and Ganapathy. "Activethief: Model extraction using active learning and unannotated pub- data". In: Proceedings of the AAAI Conference on Artificial Intelli- gence. Vol. 34. 2020, pp. 865-872 (cit. on 4)
[PGSKSG20]
[PPA18]
A. Polino, Pascanu, and D. Alistarh. "Model compression via distilla- tion and quantization". In: arXiv preprint 1rXiv:1802.0566 (2018) (cit.
p.
[SCGL19]
S. Sun, Cheng, Z. Gan, and Liu. "Patient knowledge distillation for bert model compression". In: arXiv preprint
(2019)
(cit. on p. D.
[SDSOJ19]
Sablayrolles, M. Douze, C. Schmid, Y. Ollivier, and H. Jégou. "White- box black-box: Bayes optimal strategies for membership inference". In: International Conference on Machine Learning. PMLR. 2019, pp. 5558- 5567 (cit. p.Z)
[SH21]
V. Shejwalkar and A. Houmansadr. "Membership privacy for machine learning models through knowledge transfer". In: Proceedings the AAAI Conference on Artificial Intelligence. Vol. 35. 2021, pp. 9549-9557 (cit.
PP-
4 13).
[SSSS17]
R. Shokri, M. Stronati, C. Song, and V. Shmatikov. "Membership infer- ence attacks against machine learning models". In: 2017 IEEE symposium security and privacy (SP). IEEE. 2017, pp. 3-18 (cit. P 2).
[TMSSNHM22]
X. Tang, S. Mahloujifar, L. Song, V. Shejwalkar, M. Nasr, A. Houmansadr, and P. Mittal. "Mitigating membership inference attacks by [Self-Distillation) through novel ensemble architecture". In: 31st USENIX Security Symposium (USENIX Security 22). 2022, pp. 1433 1450 (cit. on Pp. B
[TSSJLJHC22]
F. Tramèr, R. Shokri, A. San Joaquin, H. Le, M. Jagielski, Hong, and N. Carlini. "Truth Serum: Poisoning Machine Learning Models to Re- veal Their Secrets". In: Proceedings of the 2022 ACM SIGSAC Confer- ence Computer and Communications Security. CCS '22. Los Ange- les, CA, USA: Association for Computing Machinery, 2022,
ISBN: 9781450394500 URL: https :/oo::::///o... org/ 10 1145/3548608 3560554 (cit. on p....
[TZJRR66]
F. Tramèr, F. Zhang, A. Juels, M. K. Reiter, and T. Ristenpart. "Stealing
machine learning models via prediction (APIs)" In: 25th USENIX secu-
rity symposium (USENIX Security 16). 2016, pp. 601-618 (cit. on pp.
10p.
[WBKBGGG22]
Y. Wen, A. Bansal, H. Kazemi, E. Borgnia, M. Goldblum, J. Geiping, and
T. Goldstein. "Canary in Coalmine: Better Membership Inference with Ensembled Adversarial Queries". In: arXiv preprint arXiv:2210.101 10750 (2022) (cit. p. 2)..
[WGCS21]
I. Watson, C. Guo, G. Cormode, and A. Sablayrolles. "On the impor- tance of difficulty calibration in membership inference attacks". In: arXiv preprint arXiv:2111 1..0440 (2021) (cit. on p.
[XLHL20]
Q. Xie, M.-T. Luong, E. Hovy, and Q. V. Le. "Self-training with noisy stu- dent improves imagenet classification" In: Proceedings of the IEEF/CVF conference on computer vision and pattern recognition. 2020, pp. 10687- 10698 (cit. on pp.
12

---

[YGF]18]
S. Yeom, I. Giacomelli, M. Fredrikson, and S. Jha. "Privacy risk in machine learning: Analyzing the connection to overfitting.. In: 2018 IEEE 31 computer security foundations symposium (CSF). IEEF. 2018, pp. 268-26222222 (68-282 (cit. on pp. 1 13
[ZCW21]
J. Zheng, Y. Cao, and H. Wang. "Resisting membership inference at- tacks through knowledge distillation". In: Neurocomputting 452 (2021), pp. 114-126. ISSN: 0925-2312. URL: https: / www wwwssciiiieo.... sciencedirect. com http://cccceeaatttcccce//////2//
(cit. on p. 3)
[ZK16]
S. Zagoruyko and N. Komodakis. "Paying more attention to attention: Im- proving the performance of convolutional neural networks via attention transfer". In: arXiv preprint arXiv: (2016) (cit. on p.
A Broader Impact
Our work designs privacy attacks, which have the potential cause harm. However, by making the vulnerabilities in existing approaches known, and more rigorously evaluating the risk to users, our work is necessary step designing stronger mitigations in the future.
B More Experiment Details
All of our results on CIFAR-10 make use of fewer than 30000 trained models. While a very large number of models, the fast, publicly available training code of models in fewer than 1 GPU-week (although we decrease the wall-clock time by parallelizing over 4 GPUs). Our results on Purchase- 100 and Texas- 100 also use simple models, taking under minute to train (we train all models for 20 epochs with SGD with learning rate of 0.01 and momentum parameter of 0.99, which we found maximize performance over our hyperparameter sweep). We train 8000 of these models for our analysis, taking fewer than GPU-week for each of these datasets. Our most expensive attack, relying on only student queries, starts outperform random guessing with as few as 100 models, which can be trained on 1 GPU in two hours on all three of these datasets. Unfortunately, we are unable make our code public this time due to organizational constraints.
allows us train this number
C Extended Results on Teacher Dataset Privacy
We plot the effectiveness of Transfer LiRA in Figure 7 ROC curves for our student attacks are found in Figure g Further qualitative examples can be found in Figure 9 Ablation of score information with and without duplicates is plotted Figure 10 Per-example student attack success rates for CIFAR-10 10 with duplicates are found in Figure [[] In Figure 12 we compare our student model attacks against a simple logit threshold baseline. similar the loss thresholding attack designed by Yeom, Giacomelli, Fredrikson, and Jha [YGFJ18), which was used to evaluate distillation privacy Shejwalkar and Houmansadr [SH2T].
D Privacy of Student Training Set
Having evaluated the Private Teacher threat model, we now turn to the Private Student and Self- Distillation threat models, which we will consider simultaneously. The Private Student threat model can be used to perform knowledge transfer from large. general purpose models to task-specifi models, by querying on (sensitive) task-specific student data. Self-distillation often used in appli- cations of distillation to compress models and improve their performance.
D.1 Private Student
The private student threat model does not involve data minimization, unlike the private teacher threat model; the empirical privacy we investigate here comes instead from an adversary having limited knowledge of the specifics of the teacher model. That is, the question we investigate is: how much
13

---

1.0
13
10
URA Accurazy
Bid 201
0.9
yoX
0.8
0.5
08
0.5
0.6
35
0.6
0.5
3.5
0.7
088
0.9
1.0
0.5
8n
68 1.0
0.7 C/ Teachar Accuracy
Teacher Accuracy
Teacher Accuunscy
Teacher MI Accuracy
(a) CIFAR-10
(b) WikiText
(c) Texas- 100
(d) Purchase- 100
Figure 7: Many data points do not get privacy benefits from distillation. With the axis, we plot the vulnerabilityy of each teacher example to attack before distillation, using teacher models. With the y axis, we plot the vulnerability to attack after distillation, using the Transfer LiRA strategy to attack student models, Observe that many data points lie near the = x line, which indicates no reduction vulnerability from distillation.
10-1
16
10"
31-4
Poskive
LIAA Racher
10-1
End-to-End LRM
Tranaferr URA
12"
10
Randem Guess
False Positive
Falte Postiive Rate
False Positive Rale
13-1
1g-1
22
10"
False Pasilive Rata
10-11
10-0
(a) CIFAR-10 10
(b) WikiText103
(c) Texas-100
(d) Purchase- 100
Figure 8: ROC curves for our attacks on student models.
does the adversary need to know about the teacher model to get reliable attacks the private student dataset?
We consider three levels of adversarial knowledge: Known Teacher, where the adversary knows the precise teacher model used to query the student examples; Unknown Teacher, where the adversary knows the teacher model is one of a small subset of models; and Surrogate, where the adversary can only collect similar data, to train their own surrogate teacher models. Both the Known and Unknown Teacher settings reflect world where the teacher model is one of a small number of general purpose public models, such as a large language model. The Surrogate setting requires the adversary to train their own copy.
We run the LiRA variants in a number of these settings on the CIFAR-IO dataset, calibrated to the knowledge the adversary has (for example, in the Surrogate threat model, the adversary trains their own teacher models, and trains a number of shadow student models to calibrate LiRA). We plot our results in Figure 13a, and find that, as expected, less knowledge about the teacher model reduces the adversary's 's success at membership inference. However, even the weakest threat model, Surrogate, allows for powerful attacks, with TPR as large 10 FPR of 10
D.2 Privacy of Self-Distillatie
Having considered the privacy of the student and teacher datasets independently, we now investi- gate the common self-distillation setting [FLTIA18] XLHL20), where the student and teachers are identical. Given that duplicate examples in the student set carry membership information of teacher examples (Section 6.I), and student examples themselves are not well protected by distillation (Sec- tion DD we do not expect self-distillation to reduce privacy risk significantly. However, a common technique in self-distillation is to train the student a loss function which combines the cross en- tropy loss on the query dataset to with the cross entropy loss on the student examples' original "hard labels" E.. We write fa alq + (1 a)ls, that = 1 recovers the standard distillation
14

---

(a) Target
(b) Student Queries
(c) Target
(d) Student Queries
Figure 9: Two examples of target examples for which the most informative student queries are predominantly in the same class. The only exception is the eighth student query in (d) for the yellow truck in (c), which is an airplane. The filtered attack using the displayed student queries reaches 78% accuracy the yellow automobile in (a), and 74% accuracy the yellow truck in (c).
Dups Original Densited
20
No Dups Origiaal Deniieed
80
10 4 Fabse Positive Rate
Figure 10: The impact of denoising on duplicated and deduplicated teacher attacks.
objective, while 0 recovers the standard cross entropy loss (as if there was never teacher model).
To evaluate self-distillation, we run LiRA by training shadow student models with the entire self- distillation algorithm, using identical datasets for each pair of teacher and student shadow models. We perform calibration on these shadow student models, and plot our results at range of values Figure [36] While we don't observe a large effect, it appears that larger a (that is, heavier reliance on the distillation loss function) results in better attacks. This is likely because relying on the distillation loss function reinforces the memorization from the teacher even further in the second round of training on the student.
End-to-End LiRA MI Accuracy
1.0
R=0.81
0.9
0.8
0.7
0.6
0.5
0.5
0.6
0.7
0.8
0.9
1.0
Teacher Accuracy
Figure 11: Duplication also has an impact on CIFAR-10 student attacks. Compare with Figure 3a
15

---

End Bnd LIA
Teacher LiKs Tacher Model Logit thrrashlld
37"
Figure 12: Our attacks outperform simple logit threshold baseline attack, used by prior work.
100
10
True Positive Rate
True Positive Rate
10
10
10:
10
Known
a=1
Unknown Surrogate
a= 0.5
10-3
10-3
a=
y=x
y=x
10-3
10-2
10-1
100
10-3
10-2
10-1
10
False Positive Rate
False Positive Rate
Private Student
(b) Self-Distillation
Figure 13: Distillation has limited ability to prevent membership inference either a) on sensitive stu- dent examples, b) in self-distillation. However, reducing the knowledge available the adversary seems to help in the Private Student threat model. Results for both on CIFAR- 10.
16


Reviews:
[{"review_id": "WB4x9jZmt5", "reviewer": "Reviewer_odTu", "summary": "The authors in this paper investigate the efficacy of membership inference attacks (MIA) in model distillation. Their novel attack(s) show that MIA is possible even when the teacher model is only queried on the most influential points in the student inputs. Finally, they also demonstrate how their attacks are the strongest when the teacher set is poisoned OR when the student set ~= teacher set.", "questions": "1. Could this happen because of correlated features present between the student queries and the target example? Acc to Figure 1, the “red” colour present in the student examples are essential for the MIA to work. My intuition is that a higher correlation in features between the student inputs and the teacher target example, should lead to higher success in MIA. Of course, this may be written down as Future Work and I’m not quite sure if such work has been done in this space. I think it might be nice to investigate such correlation effects between the student and teacher examples. \n2. I noticed that all the attacks mentioned in this paper, were based off LiRA. Is there a specific reason why the authors succumbed to using LiRA and not other attacks. (It's totally okay to use LiRA but I wonder if there could be other attacks you could have tried/did try out and measure the efficacy of such attacks against the LiRA-based ones; provided you have time to run these expts).", "limitations": "1. Need to run a number of shadow models for running a successful MIA (not really a limitation of this approach, but this is a limitation of MIA itself!). ", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The novelty is strong in this paper. No one has previously tackled MIA in model distillation as the authors here do. \n2. The paper is also very easy to read and understand. \n3. I really like the novel idea present in this paper (specifically Figure 1): membership of a target example in the teacher’s training set can be derived through querying the teacher model on various student set examples (and these are entirely different examples from the target!). \n4. Figure 3 is a great result as well. It might not be as obvious as it is, because the data points are essentially “post-processed” and there should be some sense of privacy after the distillation process. The authors here, show that distillation doesn’t show much privacy benefits. \n5. Their investigation on why MIA works well for student models is sound. I really like Figure 5 as well, which shows the MIA effects through duplicates, data poisoning and temperature scaling and the section on student-teacher data drift was an amazing read :-)", "weaknesses": "1. I don’t really have any major weaknesses to point out!", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time in reading our submission and interesting questions!\n\n>**Feature correlation:** This is an interesting intuition, and we agree that the example in Figure 1 does seem to be due to this “red”ness (and one might draw similar conclusions about the examples in Figure 9 in the supplement). However, it is unclear how to define and quantify correlation for systematic study: we believe this is very interesting future work. This seems to have implications beyond privacy in understanding distillation’s dark knowledge, and we were unable to find related results in this literature.\n\n>**Why use LiRA:** LiRA represents the current state-of-the-art for membership inference. Thus, it represents the strongest baseline and also the strongest starting point for our research. Prior work using distillation as a defense has all predated the LiRA paper and so only evaluated with weaker attacks. This is also a motivation for our work: to understand whether stronger membership inference attacks could challenge this use of distillation. We also see this in Figure 12 in the supplementary material: a simple logit-gap membership inference attack (akin to simpler attacks such as Yeom et al. 2018) is unable to get beyond random guessing on CIFAR-10. "}, {"review_id": "8tDkUqvhbV", "reviewer": "Reviewer_4Gun", "summary": "The paper explores the privacy implications of model distillation, a technique used to transfer knowledge from a teacher model to a student model. The authors investigate membership inference attacks on both the teacher and student training sets to evaluate the privacy provided by distillation. \nThe authors extend the LiRA attack to the distillation setting in two ways. The first attack calibrates the attack only on the teacher models and the second attack uses the whole distillation procedure. They find that distillation alone provides limited privacy protection. The attacks on distilled models succeed even though the distilled models have never seen the teacher's data directly.\nFinally, they also highlight the importance of considering factors such as data duplication, teacher set poisoning, temperature scaling and distribution shifts when evaluating the privacy risks of distillation.", "questions": "What is the utility of the models? Is privacy simply correlated with utility? Do we see the same percentage point drops in model utility when going from teacher to student?", "limitations": "The authors have accurately described limitations albeit only in the appendix.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- The authors address an important problem. As machine learning capabilities grow and models are deployed directly to personal devices with limited computing capability, model compression algorithms such as distillation are often used. Understanding the privacy risk of these algorithms is therefore an increasingly important topic.\n- The paper is very easy to read and well motivated. Particularly, the visualizations in figure 3 and 4 are very intuitive and help tell the story\n- The paper considers many experimental evaluation settings (albeit limited detail about the concrete experiments)", "weaknesses": "- An obvious mitigation of membership inference attacks is Differential Privacy, it would be interesting to see how effective DP mitigates the attack as by Mireshghallah et al [2022].\n- Limited detail of experimental setup. What is the utility of the models?\n- Some conclusions seem not deeply thought through. E.g. Line 213, the authors claim that since there are examples for which vulnerability drops by only  single digit percentage points and since privacy is a worst case guarantee the authors conclude that distillation provides limited privacy. While I agree that distillation provides limited privacy, it is possible that least vulnerable points see the single digit drop in vulnerability whereas the most vulnerable points see a more significant drop.\n\n\n**References**\n\nMireshghallah, Fatemehsadat, et al. \"Differentially private model compression.\" Advances in Neural Information Processing Systems 35 (2022): 29468-29483. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time in reading the submission and writing the review!\n\n>**Differential privacy:** This is an interesting question. Part of the goal of our work was to consider distillation *without privacy* because there exist past works that attempt to show distillation can achieve a strong notion of privacy. Thus, our aim was to show this is not the case. Investigating differential privacy’s impact on these attacks is interesting future work.  We also remark that the Mireshghallah et al work considers a slightly different threat model where the teacher and student datasets are identical, which we show in the Supplemental material to be much more vulnerable to attack than the threat model where the teacher is sensitive and the student is public, nonsensitive data. We can also be more clear when we cite the Mireshghallah et al work in Section 2.2 that it would offer a provable guarantee against our attacks.\n\n>**Model utility:** Our CIFAR-10 models reach roughly 88% accuracy. This training setup reaches >94% accuracy on the full CIFAR-10 training set. On Purchase100 the student models reach 74-75% accuracy, and on Texas100 the models reach 54-55% accuracy (note these are 100 class tasks). Unfortunately, we didn’t save test predictions on WikiText, so we need to retrain a model to get its utility, please bear with us. We’ll put these numbers in the paper.\n\n>**Privacy as a worst case:** It’s true that the scenario you propose could offer worst-case privacy protection, but only if this were true for *all very vulnerable points*; we broadly find that this is not the case in Figure 3. We will be more careful about our wording here and ensure we highlight that not all vulnerable data see significant reductions in membership inference vulnerability.\n\n>**Privacy and utility?:** Though utility and vulnerability are often correlated, out attacks significantly outperform what can be inferred by simple attacks based on the accuracy of the model (i.e., the Yeom et al. 2018 attack, see Figure 12). Further, our attacks are not always well correlated with utility: there exist other factors that influence its success. For example, we find that increasing temperature (Figure 5d) does not change utility significantly, but does increase vulnerability. Distilling with CIFAR-100 examples (Figure 6) leads to less utility compared to distilling with CIFAR-10 examples (roughly 85% compared to 88%), but interestingly leads to higher vulnerability. Deduplicating increases utility slightly (88% instead of 87.5% accuracy), but reduces vulnerability (Figure 5b). We can add a discussion of this to the paper.\n"}, {"review_id": "xT96U2zObN", "reviewer": "Reviewer_E9Bb", "summary": "The paper examines the effectiveness of model distillation in protecting the privacy of training data. Through the use of membership inference attacks, the authors demonstrate that distillation alone provides limited privacy across various domains. The authors also suggest several design considerations for improving privacy in model distillation, such as deduplicating the teacher set and considering complementary techniques like differentially private training.", "questions": "This paper provides valuable insights by addressing the misconception that model distillation can effectively protect the privacy of training data. The paper is well-organized and easy to follow, and a thorough examination of potential influencing factors contributes to the strength of the research.\n\nHowever, I have some concerns regarding the details of the attacks and the interpretation of results:\n\n- Regarding the experimental setting for End-to-End LiRA, it would be helpful to clarify how the corresponding student dataset was chosen. This information would enhance the reproducibility and validity of the experiments.\n\n- The paper mentions that attacks on the student model should not be more powerful than those on the teacher model, as implied by the data processing inequality. However, Figure 3 reveals that some samples remain more vulnerable after distillation. It would be beneficial to provide a possible explanation for this observation, as it appears to contradict the expected outcome.\n\n- Section 5, which introduces a new attack that only utilizes the student query dataset, is difficult to understand. The authors claim that “for each teacher example $z^T_j$, we fit a Gaussian distribution to the logits of each student example $z^S_i$, when $z^T_j$ is either IN or OUT.” However, it is unclear how the authors establish a link between student examples and arbitrary teacher examples, and how the student examples can indicate the membership status of the teacher example. Further clarification and additional illustrations would greatly enhance understanding in this section.\n\nOverall, this paper makes significant contributions and offers valuable insights. Addressing the aforementioned concerns would further strengthen the research and improve the clarity of the presented findings.\n", "limitations": "The authors have adequately claimed their limitations in Appendix B.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Trendy topic\n- Well-organized paper\n- Extensive experiments", "weaknesses": "- Attack detail in Section 5 is not clear\n- Additional discussion might be helpful", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time in reading the submission and writing the review!\n\n>**End-to-End LiRA:** We use the same student set as used to train the target student model. This is because distillation can be done on public, nonsensitive data.\n\n>**Figure 3 & Data Processing Inequality:** The data processing inequality bounds the information contained in the student model about the data’s membership on the teacher. Thus, this only bounds the performance of an optimal attack, which LiRA on the teacher model may not be. We also note that the x and y coordinate of each point in Figure 3 is the average over a sample of target models and so are noisy values that can randomly appear above the dotted line.\n\n>**Clarification for Section 5:** The LiRA strategy trains many shadow models on a random 50% subsample of the teacher set. As a result, each teacher example will be present in roughly half of the shadow models, and will not be present in the other half. Then if we train 100 shadow models, we can look at their predictions, on each student example, of the ~50 shadow models containing the target teacher example, as well as the predictions of the ~50 shadow models without the target teacher example. Let us know if this explanation helps, and we can add it to the paper, or help clarify more."}, {"review_id": "Iln4GWna17", "reviewer": "Reviewer_xTVE", "summary": "Multiple previous works have proposed knowledge distillation techniques to distill the knowledge of a teacher trained on sensitive data into a student model which is supposedly protected against membership inference attacks. This paper proposes a new membership inference attack to perform membership inference attacks on the student model. Attacking the student models, it is shown that knowledge distillation provides only limited privacy against membership inference attacks. Having access to the dataset used to train the student model but not the student model itself, it is shown that this suffices to attack the private data of the teacher model.", "questions": "**Q1:** The per example membership inference attack success rate is calculated over 1000 models. However, how do these models differ? Were they trained on the same data but with different seeds? If so, aren't these models behaving very similar?  \n**Q2:** In line 241 the indirect attack is described. However, the procedure of the attack is not quite clear to me. As far as I understood, the attacker has no knowledge about the teacher model's predictions on the teacher examples (line 231). Could you clarify what the assumptions made for this attack are and on how the attacker is fitting the Gaussians in this case?", "limitations": "- the limitations are appropriately addressed in the supplementary material", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- **Originality:** The paper is novel and proposes a novel approach for membership inference attacks on student models.\n- **Quality:** The experimental results seem sound and the claims of the paper are well supported.\n- **Clarity:** Overall, the paper is mostly well written and well organized. \n- **Significance:** With previous work having proposed knowledge distillation as a defense technique against membership inference, the results of the paper are important and most likely of high impact to other researchers.", "weaknesses": "**Clarity:**  \n- The setting and the assumptions made for the proposed attacks are not very clear, and it would be helpful for the reader to explain the assumptions and information available to the attacker in more detail.\n- unfortunately, the code used to run the experiments will not be made public, which hinders reproducibility\n\nMisc:\n- Line 94: \"mitigates prevent\". There seems to be some missing or additional word.\n- maybe you could mention in the captions that “+” in the plots means the positive correlation. Otherwise, it is a bit confusing, if the reader is searching for a blue plus in the plots.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and effort in reading our submission and writing your review! \n\n>**Line 94:** Sorry about this! Actually, every point is a small blue plus sign! We will explain this clearly in the caption.\n\n>**Q1:** We follow the evaluation strategy used to evaluate LiRA, where each model is trained on a different 50% split of the dataset. That ensures these models differ in their membership inference behavior.\n\n>**Q2:** The LiRA strategy trains many shadow models on a random 50% subsample of the teacher set. As a result, each teacher example will be present in roughly half of the shadow models, and will not be present in the other half. Then if we train 100 shadow models, we can look at their predictions, on each student example, of the ~50 shadow models containing the teacher, as well as the predictions of the ~50 shadow models without the teacher. We can do this for each student query, because we assume here the adversary sees the student query outputs (but cannot query on arbitrary points, which would be necessary to observe the model outputs for the teacher examples). This attack has the additional assumption that the adversary knows the student queries, as all of our End-to-End attacks do, but otherwise has no extra assumptions beyond LiRA’s. Let us know if this explanation helps, and we can add it to the paper, or help clarify more."}]

Author rebuttal:

Meta-review: The paper discusses a crucial issue of privacy in the face of model distillation.  Reviewers unanimously agree on the significance of the contributions made. Strengths highlighted by the reviewers include the paper's remarkable clarity and the thoroughness of experiments.

The reviewers also highlight the paper's contribution by dispelling a misconception that distillation safeguards data privacy. By spotlighting factors such as data duplication, teacher set poisoning, temperature scaling, and distribution shifts, the authors enrich the discourse on the topic.
Decision: Accept (oral)