SlowGuess/NeurIPS_2023_OCR huggingface dataset:
However, we find that this direct adaptation of LiRA tends to have poor performance, so we propose two modifications which significantly improve the attack. First, we find that a teacher example tends to have more influence on the logit corresponding to the teacher label  $y_{j}^{T}$  than the student label  $y_{i}^{S}$ , and so we choose to calibrate LiRA using the teacher label logit rather than the student label logit. Second, there tend to be many uninformative student examples for each teacher example, so we filter the student queries for all teacher examples. We filter these student queries by selecting only those with the largest mean gap  $|\mu_{j,i}^{IN} - \mu_{j,i}^{OUT}|$ , which are most informative about the teacher. This removes most of the noise from our LiRA adaptation, improving it especially when we train few shadow models.
Student queries leak private information. In Figure 4, we plot the per-example membership inference accuracy after distillation, using our student query attack with 4000 shadow models. We consider all datasets except WikiText103, which we omit due to the high computational cost of our attack. Our results are similar to when we attacked the student models: MIA vulnerability reduces on average, but many examples maintain nearly identical membership inference attack success rate using only the predictions on these indirect student queries. Cases where this attack achieves high membership inference accuracy indicate that a large amount of information about the teacher set is encoded in the student queries.
Observations we noticed for our student model attacks persist for this query-based attack. Examples which are more vulnerable to attack in the teacher model also tend to be more vulnerable using teacher model predictions. This is true on all datasets, although we also find the decay in vulnerability with queries to be dataset-dependent and example-dependent. Within each dataset, there is a high variance in student model membership inference accuracy.
We also remark on the counterintuitive fact that our prior attack with access to the student models outperformed this attack, despite the fact that the student model is just a post-processing of these student queries. This implies that the process of training a student model makes it easier to extract private information from these student queries, by interpolating between the student queries. In the following subsections, we will investigate some factors that can impact the success of the student query attack, and what this tells us about distillation's dark knowledge.
Ablation. In Figure 5a, we ablate the modifications we made for our student query attack. We show both LiRA scores (logits for the student label) and Label scores (logits for the teacher label). We also show attacks with "All", without filtering, and "Filtered" scores, with filtering to 10 student examples. Our modifications significantly improve membership inference: without either modification, the attack is no better than random chance. Applying both modifications achieves a TPR of  $10^{-3}$  at a FPR of  $10^{-4}$ .
Qualitative analysis. Our ablations also shed light on where membership information is contained in distillation's dark knowledge: predominantly in the logit corresponding to the teacher label within only a few student examples. This corroborates our intuition from the parrot in Figure 1. There, only two student queries were also in the bird class, emphasizing the importance of label scores. We inspect other vulnerable examples in Figure 9 in Appendix C, finding some teacher examples whose most informative students mostly belong to their same class.
6 Factors Influencing Teacher Privacy Leakage
So far, we have designed two types of attack: the attack from Section 4 which uses the student model, and the attack from Section 5 which uses the student queries. These attacks find a surprising amount of leakage which could not have been identified by prior attacks. Now, we investigate more deeply the factors which may influence the leakage of the teacher.
(a) Ablation
(b) Duplicates
(c) Data Poisoning
(d) Temperature
Figure 5: Additional investigations into the success of our attacks reveal: a) label scoring and score filtering are important for improving attack success; b) duplication between the teacher and student datasets increases privacy risk; c) data poisoning attacks amplify the performance of our indirect attack; d) temperature scaling causes mild changes in privacy vulnerability. All results are on CIFAR-10.
6.1 Student-Teacher Duplication
Unlike the preceding indirect queries, where student examples only share high-level similarities with their teacher examples, we now investigate how increased student-teacher similarity impacts the performance of our student query attack. To do so, we reintroduce the near-duplicates we removed from CIFAR-10, and compare the attack success rate between the duplicated and deduplicated student query attacks, on those teacher examples with duplicate student examples. Intuitively, student examples which are duplicates of teacher examples will carry membership information in their queries, which is what we find in Figure 5b ( $p < 10^{-15}$  using a Chow Test). This indicates that dedduplication significantly reduces privacy risk, although indirect queries can still carry membership information. In the Appendix, we show student attacks are also worsened by duplication in Figure 11.
6.2 Teacher Set Poisoning
Tramér, Shokri, San Joaquin, Le, Jagielski, Hong, and Carlini [TSSJLJHC22] and Chen, Shen, Shen, Wang, and Zhang [CSSWZ22] find that data poisoning attacks added into a training set can amplify membership inference vulnerability for other examples in the training set. Here, we investigate how distillation interacts with this effect; i.e., we measure whether poisoning in the teacher set can increase an example's vulnerability in the teacher predictions. In Figure 5c, we evaluate this using our student query attack on CIFAR-10 using the label flipping poisoning strategy from prior work.
With higher poisoning counts, many examples have higher teacher membership inference accuracy, i.e., they shift to the right on the  $x$ -axis; this is exactly in line with prior work. However, interestingly, we see that poisoning does not impact the relationship between teacher vulnerability and student vulnerability—they move upward on the  $y$ -axis identically to examples which were not poisoned.
We remark that our poisoning attack does not specifically target the student query set; it is an interesting open question whether poisoning attacks exist that could increase membership leakage on the student model, but with less (or more!) impact on the teacher model's vulnerability. Section 6.1 hints at such a strategy, if the adversary can poison the student set instead: if an adversary adds duplicates of target examples to the student set, the resulting student queries will increase risk on those target teacher examples.
6.3 Temperature Scaling
When introducing knowledge distillation, Hinton, Vinyals, Dean, et al. [HVD+15] proposed a modification known as temperature scaling. Temperature scaling makes two simple changes to distillation: (1) introducing a temperature hyperparameter  $H$ , which when increased, rescales the logits and flattens the resulting probability distribution of student queries, and (2) rescaling gradients by a  $1 / H^2$  factor. Though normally set to 1, modifying this parameter can improve performance. It is natural to consider whether such manipulation of the teacher outputs might improve empirical
privacy; indeed, Shejwalkar and Houmansadr [SH21] evaluate their distillation-based defense with various temperatures, and find that high temperature reduces private information.
We use our strong End-to-End LiRA to evaluate the empirical privacy of student models trained with  $H \in [0.1, 4]$ , shown in Figure 5d. We find lower temperatures are mildly less vulnerable than higher temperatures, reducing TPR by a factor of roughly 4 at an FPR of  $10^{-3}$  when  $H$  decreases from 4 to 0.1; model accuracy at all of these temperatures is similar to the baseline of  $H = 1$ . While this trend is small, it is also intuitive: in Section 5, we found that all logit values carry important membership information. When  $H$  is small, the entropy of query probabilities decreases, reducing information captured by these probabilities. We hypothesize that the reverse effect observed by Shejwalkar and Houmansadr [SH21] may be a result of the accuracy decrease they found resulted from high temperatures (i.e. up to  $H = 10$ ), which we do not observe.
Finally, we remark on an interesting gap between an adversary with access to student queries and one who only has access to a student model. With access to temperature-scaled student logits, an adversary can multiply by  $H$  to reverse the scaling—temperature scaling cannot make attacks harder in this threat model.
However, it does appear that low temperatures lead to weaker attacks on the student model, which we do not know how to reverse. Indeed, a natural way to adaptively attack a temperature-scaled student model is to rescale the student model's logits, but our LiRA accounts for this, as LiRA is scale-invariant. Adaptively attacking temperature-scaled models is an interesting open question.
6.4 Student Distribution Drift
In cases where the teacher dataset is sensitive, nonsensitive student data from the same distribution may be difficult to find, leading to distribution drift between the teacher and student datasets. We now investigate whether this impacts attack success. From our results on student-teacher duplication, we may expect the signal to diminish as distributions become more different. To evaluate this, we consider a CIFAR-10 teacher model distilled with a CIFAR-100 student set; CIFAR-10 and CIFAR-100 have no class overlap but have the same size. Interestingly, we find that distilling with this different dataset actually improves attack performance, despite the resulting model being somewhat less accurate than after distilling on indistribution data. We find that this is due to a higher entropy of the teacher model predictions, which can encode more membership information about the teacher examples. We run an experiment in which we train on the lowest entropy and highest entropy predictions from CIFAR-100, as well as random student queries. All experiments
Figure 6: A CIFAR-100 student set results in more privacy leakage than a CIFAR-10 student set, a result of the increased entropy of the CIFAR-100 predictions.
use the same student set sizes. We present the results in Figure 6. The CIFAR-10 student still results in the lowest leakage, and increasing entropy of the CIFAR-100 examples also increases the leakage monotonically. Note that the CIFAR-10 student predictions have a mean entropy of 0.24, while the lowest leakage CIFAR-100 examples have a mean entropy of 0.39. Our results here can also be taken as an attack on the full Shejwalkar and Houmansadr [SH21] defense, which removes high entropy and duplicate student examples. Our strong attacks allow us to better evaluate their defense.
6.5 Accuracy of the Student Model
It is common for privacy preserving training algorithms to exhibit a "privacy-utility" tradeoff: the stronger the privacy enforced by the model, the worse the model's predictive performance. By piecing together the accuracies of models trained throughout Section 6, we find that differences in attack performance cannot be explained by changes in model accuracy. For example, we find in Section 6.3 that increasing temperature increases vulnerability to our attacks but does not significantly impact model accuracy (maintaining roughly  $88\%$  accuracy for the student). Our results with distribution drift in Section 6.4 show that distribution shift leads to higher vulnerability to our attacks,
but reduces model accuracy (from  $88\%$  to roughly  $85\%$  student model accuracy with a CIFAR-100 distilled student). Finally, we find in Section 5b that dedduplication reduces vulnerability to our attacks, but actually increases model accuracy slightly (from roughly  $87.5\%$  to  $88\%$  student accuracy)! In summary, depending on the source of a model performance change, increased vulnerability can coincide with increased, decreased, or unchanged model performance!
In this work, we have used membership inference attacks to empirically evaluate to what extent knowledge distillation protects the privacy of training data. For those interested in using model distillation to improve privacy, our work offers three main design considerations. First, deduplicating the teacher set with respect to the student set is necessary to reduce risk on duplicated examples. Second, while average-case privacy can significantly improve after distillation, the worst-case vulnerable examples often see only marginal benefits. As a result, techniques which make the teacher model more private, such as differentially private training, should be seen as complementary to distillation. And third, the student training set should not be seen to have improved privacy as a result of model distillation. Our work also offers insights into the privacy properties of distillation's dark knowledge, which may be of broader interest. Our results also imply privacy attacks on model extraction attacks [TZJRR16; JCBKP20] which rely on similar algorithms to knowledge distillation.
The main limitation of our work is the strong threat model under which our attacks work. We use the same threat model as the "online" version of the LiRA [CCNSTT22] attack. This attack assumes access to the target examples before training a model, and, for the End-to-End LiRA attack, access to the student dataset. We use this strong threat model to assess worst-case vulnerability, as prior work has evaluated distillation under weaker attacks. Another limitation of the attacks is the large running time of the student query attack from Section 5, which requires thousands of shadow models to obtain good performance. While generally impractical, we prefer to position this attack as a way of explaining why distillation propagates membership information, and leave future work to attempt to improve the attack's efficiency.
We would like to thank Andreas Terzis and Andreas Schou for helpful comments on our work.
[BC14] J. Ba and R. Caruana. “Do deep nets really need to be deep?” In: Advances in neural information processing systems 27 (2014) (cit. on pp. 1, 3).
[CCNSTT22] N. Carlini, S. Chien, M. Nasr, S. Song, A. Terzis, and F. Tramer. "Membership inference attacks from first principles". In: 2022 IEEE Symposium on Security and Privacy (SP). IEEE. 2022, pp. 1897-1914 (cit. on pp. 1, 2, 4, 5, 10).
[CCTCP21] C. A. Choquette-Choo, F. Tramer, N. Carlini, and N. Papernot. "Label-only membership inference attacks". In: International conference on machine learning. PMLR. 2021, pp. 1964-1974 (cit. on p. 2).
[CJZPTT22] N. Carlini, M. Jagielski, C. Zhang, N. Papernot, A. Terzis, and F. Tramer. "The Privacy Onion Effect: Memorization is Relative". In: Advances in Neural Information Processing Systems. Ed. by A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho. 2022. URL: https://openreview.net/forum? id=ErUlLrGaVEU (cit. on p. 4).
[CNKZZNBORZ17] C. Coleman, D. Narayanan, D. Kang, T. Zhao, J. Zhang, L. Nardi, P. Bailis, K. Olokotun, C. Ré, and M. Zaharia. "Dawnbench: An end-to-end deep learning benchmark and competition". In: Training 100.101 (2017), p. 102 (cit. on p. 4).
[YGFJ18] S. Yeom, I. Giacomelli, M. Fredrikson, and S. Jha. "Privacy risk in machine learning: Analyzing the connection to overfitting". In: 2018 IEEE 31st computer security foundations symposium (CSF). IEEE. 2018, pp. 268-282 (cit. on pp. 1, 2, 13).
[ZCW21] J. Zheng, Y. Cao, and H. Wang. “Resisting membership inference attacks through knowledge distillation”. In: Neurocomputing 452 (2021), pp. 114–126. ISSN: 0925-2312. URL: https://www.sciencedirect.com/science/article/pii/S0925231221006329 (cit. on p. 3).
[ZK16] S. Zagoruyko and N. Komodakis. "Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer". In: arXiv preprint arXiv:1612.03928 (2016) (cit. on p. 1).
Our work designs privacy attacks, which have the potential to cause harm. However, by making the vulnerabilities in existing approaches known, and more rigorously evaluating the risk to users, our work is a necessary step to designing stronger mitigations in the future.
B More Experiment Details
All of our results on CIFAR-10 make use of fewer than 30000 trained models. While a very large number of models, the fast, publicly available training code we use allows us to train this number of models in fewer than 1 GPU-week (although we decrease the wall-clock time by parallelizing over 4 GPUs). Our results on Purchase-100 and Texas-100 also use simple models, taking under 1 minute to train (we train all models for 20 epochs with SGD with a learning rate of 0.01 and momentum parameter of 0.99, which we found to maximize performance over our hyperparameter sweep). We train 8000 of these models for our analysis, taking fewer than 1 GPU-week for each of these datasets. Our most expensive attack, relying on only student queries, starts to outperform random guessing with as few as 100 models, which can be trained on 1 GPU in two hours on all three of these datasets. Unfortunately, we are unable to make our code public at this time due to organizational constraints.
C Extended Results on Teacher Dataset Privacy
We plot the effectiveness of Transfer LiRA in Figure 7. ROC curves for our student attacks are found in Figure 8. Further qualitative examples can be found in Figure 9. Ablation of score information with and without duplicates is plotted in Figure 10. Per-example student attack success rates for CIFAR-10 with duplicates are found in Figure 11. In Figure 12, we compare our student model attacks against a simple logit threshold baseline, similar to the loss thresholding attack designed by Yeom, Giacomelli, Fredrikson, and Jha [YGFJ18], which was used to evaluate distillation privacy in Shejwalkar and Houmansadr [SH21].
D Privacy of Student Training Set
Having evaluated the Private Teacher threat model, we now turn to the Private Student and Self-Distillation threat models, which we will consider simultaneously. The Private Student threat model can be used to perform knowledge transfer from large, general purpose models to task-specific models, by querying on (sensitive) task-specific student data. Self-distillation is often used in applications of distillation to compress models and improve their performance.
D.1 Private Student
The private student threat model does not involve data minimization, unlike the private teacher threat model; the empirical privacy we investigate here comes instead from an adversary having limited knowledge of the specifics of the teacher model. That is, the question we investigate is: how much
(b) WikiText
(c) Texas-100
Figure 7: Many data points do not get privacy benefits from distillation. With the x axis, we plot the vulnerability of each teacher example to attack before distillation, using teacher models. With the y axis, we plot the vulnerability to attack after distillation, using the Transfer LiRA strategy to attack student models. Observe that many data points lie near the  $y = x$  line, which indicates no reduction in vulnerability from distillation.
(b) WikiText103
(c) Texas-100
(d) Purchase-100
Figure 8: ROC curves for our attacks on student models.
does the adversary need to know about the teacher model to get reliable attacks on the private student dataset?
We consider three levels of adversarial knowledge: Known Teacher, where the adversary knows the precise teacher model used to query the student examples; Unknown Teacher, where the adversary knows the teacher model is one of a small subset of models; and Surrogate, where the adversary can only collect similar data, to train their own surrogate teacher models. Both the Known and Unknown Teacher settings reflect a world where the teacher model is one of a small number of general purpose public models, such as a large language model. The Surrogate setting requires the adversary to train their own copy.
We run the LiRA variants in a number of these settings on the CIFAR-10 dataset, calibrated to the knowledge the adversary has (for example, in the Surrogate threat model, the adversary trains their own teacher models, and trains a number of shadow student models to calibrate LiRA). We plot our results in Figure 13a, and find that, as expected, less knowledge about the teacher model reduces the adversary's success at membership inference. However, even the weakest threat model, Surrogate, allows for powerful attacks, with a TPR as large as  $10^{-2}$  at a FPR of  $10^{-3}$ .
D.2 Privacy of Self-Distillation
Having considered the privacy of the student and teacher datasets independently, we now investigate the common self-distillation setting [FLTIA18; XLHL20], where the student and teachers are identical. Given that duplicate examples in the student set carry membership information of teacher examples (Section 6.1), and student examples themselves are not well protected by distillation (Section D.1), we do not expect self-distillation to reduce privacy risk significantly. However, a common technique in self-distillation is to train the student on a loss function which combines the cross entropy loss on the query dataset  $\ell_{Q}$  with the cross entropy loss on the student examples' original "hard labels"  $\ell_{S}$ . We write  $\ell_{\alpha} = \alpha \ell_{Q} + (1 - \alpha)\ell_{S}$ , so that  $\alpha = 1$  recovers the standard distillation
(b) Student Queries
(c) Target
Figure 9: Two examples of target examples for which the most informative student queries are predominantly in the same class. The only exception is the eighth student query in (d) for the yellow truck in (c), which is an airplane. The filtered attack using the displayed student queries reaches  $78\%$  accuracy on the yellow automobile in (a), and  $74\%$  accuracy on the yellow truck in (c).
(d) Student Queries
Figure 10: The impact of denoising on duplicated and deduplicated teacher attacks.
objective, while  $\alpha = 0$  recovers the standard cross entropy loss (as if there was never a teacher model).
To evaluate self-distillation, we run LiRA by training shadow student models with the entire self-distillation algorithm, using identical datasets for each pair of teacher and student shadow models. We perform calibration on these shadow student models, and plot our results at a range of  $\alpha$  values in Figure 13b. While we don't observe a large effect, it appears that larger  $\alpha$  (that is, heavier reliance on the distillation loss function) results in better attacks. This is likely because relying on the distillation loss function reinforces the memorization from the teacher even further in the second round of training on the student.
Figure 11: Duplication also has an impact on CIFAR-10 student attacks. Compare with Figure 3a.
Figure 12: Our attacks outperform a simple logit threshold baseline attack, used by prior work.
(a) Private Student
(b) Self-Distillation
Figure 13: Distillation has limited ability to prevent membership inference either a) on sensitive student examples, or b) in self-distillation. However, reducing the knowledge available to the adversary seems to help in the Private Student threat model. Results for both on CIFAR-10.