SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Matthew Jagielski<sup>1</sup> Milad Nasr<sup>1</sup> Katherine Lee<sup>1</sup> Christopher Choquette-Choo<sup>1</sup> Nicholas Carlini<sup>1</sup> Florian Tramère<sup>2</sup>
Google DeepMind ETH Zurich
Model distillation is frequently proposed as a technique to reduce the privacy leakage of machine learning. These empirical privacy defenses rely on the intuition that distilled "student" models protect the privacy of training data, as they only interact with this data indirectly through a "teacher" model. In this work, we design membership inference attacks to systematically study the privacy provided by knowledge distillation to both the teacher and student training sets. Our new attacks show that distillation alone provides only limited privacy across a number of domains. We explain the success of our attacks on distillation by showing that membership inference attacks on a private dataset can succeed even if the target model is never queried on any actual training points, but only on inputs whose predictions are highly influenced by training data. Finally, we show that our attacks are strongest when student and teacher sets are similar, or when the attacker can poison the teacher set.
Model distillation [HVD+15] is a common framework for knowledge transfer, where knowledge learned by a "teacher model" is transferred to a "student model" via the teacher's predictions. Distillation is helpful because the teacher's predictions are a more useful guide for the student model than hard labels; this phenomenon has been explained by the teacher's predictions containing some useful "dark knowledge". Variants of model distillation have been proposed for, e.g., model compression [HVD+15; BC14; PPA18; KPK18; SCGL19] or training more accurate models [ZK16; XLHL20]. Within the privacy-preserving machine learning community, distillation has been adapted to protect the privacy of a training dataset [PAEGT16; TMSSNHM22; SH21; MHHVEHP22].
Many of these approaches rely on the intuition that distilling the teacher model serves as a privacy barrier that protects the teacher's training data. Informally, restricting the student to learn only from the teacher's predictions is a form of data minimization, which should result in less private information being fed into, and memorized by, the student. This privacy barrier around the teacher also allows the teacher model to be trained with strong, non-private, training approaches, improving both the teacher model's and student model's accuracy.
Because model distillation does not provide a rigorous privacy guarantee (such as those offered by differential privacy [DMNS06]), in our work we evaluate the empirical privacy provided by these schemes. We show that distillation is vulnerable to membership inference attacks—a well-studied class of privacy attacks on machine learning [SSSS17; YGFJ18].
We adapt the state-of-the-art Likelihood Ratio Attack (LiRA) [CCNSTT22] to the distillation setting, and find that this attack works surprisingly well at inferring membership of the teacher's training data. For some training examples, model distillation fails to appropriately protect against mem
(a) Target
Figure 1: We can predict the membership status of a target example (a) in the teacher model's training set by querying the teacher model on different student training examples (b). Our membership inference attack uses only the student model's predictions on the ten images on the right to reach  $73\%$  accuracy at predicting membership of the target image. Interestingly, while the target example is a bird, only two of the most informative student queries are birds.
(b) Indirect Student Queries
bership inference. To explain this finding, we show for the first time that the membership presence of some examples can be inferred based on the model's predictions on other, seemingly unrelated examples. This observation provides new insights into how membership information is transmitted from the teacher to the student.
Figure 1 gives an example of such cross-example leakage. A teacher model trained on the red parrot in Figure 1a (labeled as "bird"), never seeing the student queries on the right in Figure 1b, encodes membership information about the parrot in predictions on these student queries. Interestingly, the most informative student queries are not birds—they are images of other classes with similar red hues that the model confuses as being "bird-like" because of the influence of the parrot in the teacher's training set. In other words, the student model manages to "parrot" its teacher using the peculiarities transferred through distillation.
We systematically evaluate a number of factors which impact the empirical privacy of model distillation. We find that similarity between the training datasets of the teacher and student models lead to increased privacy risk, along with higher temperature parameters. Moreover, an adversary capable of poisoning the teacher training set can amplify privacy risk for teacher examples, in line with recent work [TZJRR16; CSSWZ22]. We also find that distillation provides little privacy protection to student examples. We hope our attacks can assist experts in properly evaluating the privacy risks resulting from model distillation. Our work also highlights two possible approaches for mitigating privacy risks in distillation: deduplicating the teacher and student datasets, and reducing the leakage from the teacher model, e.g., using provable guarantees such as differential privacy [DMNS06].
2.1 Machine Learning Privacy
Machine learning models are known to be vulnerable to a variety of privacy attacks, including membership inference attacks [SSSS17], attribute inference attacks [FJR15], property inference attacks [GWYGB18], and training data extraction [CTWJHVLRBSE+21]. Each of these captures a different type of leakage about the training data or individual examples.
In this work, we focus on membership inference, as it is the most widely studied privacy attack. In a membership inference attack, an adversary tries to determine whether or not a particular example was used to train a model. There have been a number of membership inference attacks proposed in the literature, which generally compute some "membership score", which is designed to be informative of a target example's membership. Most of these scores make use of the model's prediction on the target example, perhaps to compute the example's loss [YGFJ18; SDSOJ19] or to compute some other score [WGCS21; SSSS17; CCNSTT22]. Other attacks rely on querying the model with examples nearby or derived from the target [JWKGE20; LBWBWTGC18; LZ21; CCTCP21; WBKBGGG22]. One of our contributions is to design an attack which performs well despite relying on the model's predictions on entirely different examples from the target.
Our attacks are based on the state-of-the-art Likelihood Ratio Attack (LiRA) [CCNSTT22]. In LiRA, the adversary first trains many shadow models, such that half of these models will contain a given target example  $(x,y)$  (the IN models), and half will not (the OUT models). Next, the adversary queries each shadow model  $f$ , to compute the logits corresponding to the correct class,  $f(x)_y$ , and fits a Gaussian  $\mathcal{N}(\mu_{\mathrm{in}},\sigma_{\mathrm{in}}^2)$  to the logits for all models containing the example (and similarly for the
Figure 2: Knowledge distillation is a multi-step training process. A teacher model  $f_{T}$  is first trained on a teacher dataset  $D_{T}$ . A dataset of images annotated by the teacher  $D_{Q}$  is generated by querying the teacher  $f_{T}$  on an (unlabeled) student training set  $D_{S}$ . Finally, a student model  $f_{S}$  is trained on  $D_{Q}$ .
OUT models). To attack a new model  $f'$ , the adversary computes the probability density function (PDF) of each Gaussian (which we write as  $p_{\mathrm{in}}(f'(x)_y)$  and  $p_{\mathrm{out}}(f'(x)_y)$ ), and then computes the likelihood ratio, which serves as the membership score. Our attacks in this paper will be derived from LiRA but adapted to the variety of distillation settings we consider.
2.2 Knowledge Distillation
Knowledge distillation [HVD+15; BC14] is a technique for transferring knowledge from a "teacher" model to a "student" model. There are two datasets in knowledge distillation: the teacher dataset  $D_{T} = \{x_{i},y_{i}\}_{i = 1}^{n_{t}}$ , with  $n_t$  examples, and the student dataset  $D_{S} = \{x_{i},y_{i}\}_{i = 1}^{n_{s}}$ , with  $n_s$  examples. Distillation begins by training a teacher model  $f_{T}$  on the teacher dataset  $D_{T}$ . The teacher model is used to generate (soft) labels for the student dataset, producing a query dataset  $D_{Q} = \{x_{i},S(f_{T}(x_{i}))\}_{i = 1}^{n_{s}}$ , where the softmax function  $S$  converts logits into a probability vector. Training on this query dataset then produces a student model  $f_{S}$ , the output of distillation. Sometimes, distillation includes "temperature" scaling, where the logits  $f_{T}(x_{i})$  are scaled by a temperature parameter  $H$ , before the softmax is applied. The query dataset will instead be  $D_{Q} = \{x_{i},S(f_{T}(x_{i}) / H)\}_{i = 1}^{n_{s}}$ ; setting  $H = 1$  recovers the original distillation procedure. We illustrate this process in Figure 2.
Knowledge distillation in private machine learning. Prior work has suggested that distillation mitigates prevent privacy attacks. Perhaps the most well known example is the PATE framework [PAEGT16], where distillation is used to reduce an ensemble of teacher models into a single model, in such a way that the final model has provable differential privacy guarantees [DMNS06]. Zheng, Cao, and Wang [ZCW21] and Tang, Mahloujifar, Song, Shejwalkar, Nasr, Houmansadr, and Mittal [TMSSNHM22] construct ensembles of models that are designed to be private and use distillation to condense these models. Mireshghallah, Backurs, Inan, Wutschitz, and Kulkarni [MBIWK22] enforces a differential privacy guarantee by training the teacher and student models with differential privacy. In each of these approaches, distillation is only one component; an underlying ensemble, or provable differential privacy guarantees, may also improve the privacy of the overall approach. In our work, we focus on the distillation procedure itself, and leave to future work the task of designing attacks on these more complicated approaches. Importantly, also, those works which offer provable privacy guarantees will have hard limits on attack effectiveness.
Indeed, other prior work has already suggested that using distillation alone to protect privacy. Shejwalkar and Houmansadr [SH21] propose a defense that relies on a sufficient  $\ell_2$  distance between teacher and student examples and limited entropy of the queries. We evaluate this defense in Section 6.4 and Appendix D.1. Mazzone, Heuvel, Huber, Verdecchia, Everts, Hahn, and Peter [MH-HVEHP22] consider using repeated distillation to prevent membership inference attacks. In our work, we will design stronger membership inference attacks to adaptively evaluate the privacy provided by distillation.
3 Threat Model and Experimental Setting
We investigate the ability of distillation to protect against membership inference attacks in three threat models:
1. Private Teacher. The teacher dataset  $D_{T}$  is sensitive and the student dataset  $D_{S}$  is nonsensitive. We assume the adversary has knowledge of the student dataset. This threat model
is used in most private machine learning approaches. We evaluate this threat model in the main body of the paper.
2. Private Student. The teacher dataset is nonsensitive and the student dataset is sensitive. We assume the adversary has access to the teacher dataset. This threat model may be used to transfer knowledge from a foundation model used as a teacher to a sensitive downstream task [LZZHCZ22]. Because this threat model is less well considered in the privacy literature [SH21], we evaluate this threat model in Appendix D.1.
3. Self-Distillation. Self-distillation is commonly used to refer to the setting where the teacher and student datasets are identical. Self-distillation is commonly used when distillation is used to improve model performance or during model compression. We evaluate this threat model in Section D.2.
For each of these threat models, we measure the success of the adversary at performing membership inference on the sensitive dataset, both by measuring the true positive rate (TPR) and false positive rate (FPR) as suggested by Carlini, Chien, Nasr, Song, Terzis, and Tramer [CCNSTT22], and also by investigating the membership inference accuracy on each individual example, as done in Carlini, Jagielski, Zhang, Papernot, Terzis, and Tramer [CJZPTT22]. In general, we will report this perexample membership inference accuracy when presenting the performance of one or two attacks, due to the amount of information this metric conveys, and use ROC curves to compare between more than two.
Beyond distillation's potential uses in privacy, our attacks on distillation have implications for private information leaked during learning-based model extraction attacks [TZJRR16; OSF19; PGSKSG20; JCBKP20], which often resemble model distillation. In model extraction, an adversary uses API access to a target model to reproduce its functionality into the weights of a local model. The target model is analogous to the teacher model in distillation, and the local model is analogous to the student. Then attacks in the Private Teacher threat model can be cast instead as allowing an adversary to inspect their local model to perform membership inference attacks on the target model's training data, without directly querying it. The Private Student threat model has implications for a defensive setting in model extraction, where the target model's owner wants to link queries made to the target model with a model they believe has been extracted from their model.
3.2 Experimental Setting
We study four standard datasets for our analysis: CIFAR-10, KiText103, Purchase100, and Texas100. On CIFAR-10, we start with code from the DAWNbench benchmark [CNKZZN-BORZ17] which trains an accurate ResNet-9 model in under 15 seconds; we adapt this code to support model distillation and the subsampling required for LiRA variants. We remove 5275 duplicates from CIFAR-10, using the imagededup library [JLJT19], and split the remaining dataset into a teacher set of 30,000 examples and a student set of 14,725 examples. On KiText103, we used the GPT-2 architecture (with a context window of 256 tokens, 4 heads, 4 layers and an embedding size of 256 dimensions). We split KiText103 into a teacher set of 500,000 records, and use the remaining records to train the student models. On Purchase100 and Texas100, we train single-layer neural networks with hidden layer sizes of 256 and 512, respectively, and subsample the datasets to produce teacher and student sets of 20000 examples each. On Purchase100, student models reach  $74 - 75\%$  accuracy, and on Texas100, they reach  $54 - 55\%$  accuracy. On all datasets, we train our models with the cross entropy loss: teacher models are trained with the standard sparse cross entropy loss on the teacher dataset, and student models are trained with a dense cross entropy loss to mimic the soft labels predicted by the teacher. Unless otherwise stated, all LiRA-based attacks use 100 shadow models for calibration, and all figures are produced by running the attack on over 1000 models. Needing to train a large number of shadow models is one limitation of our attacks, and any shadow model-based membership inference attacks; we comment on training efficiency and code for our experiments in Appendix B.
4 Evaluating Privacy of the Sensitive Teacher Training Set
The most common way prior work in privacy has used distillation is to improve the privacy of the teacher set. Intuitively, distillation protects the teacher set because the adversary can only interact
(b) WikiText103
(c) Texas-100
Figure 3: Many data points get no privacy benefits from distillation. On the x axis, we plot the vulnerability of each teacher example to membership inference before distillation, using teacher models. On the y axis, we plot the vulnerability of each teacher example to attack after distillation, using the End-to-End LiRA strategy. Observe that many data points (very small blue plus signs) lie near the  $y = x$  line, which indicates no reduction in vulnerability from distillation.
(d) Purchase-100
with the student model, which never sees any teacher examples. In fact, because the student model can be seen as a "post-processing" of the teacher model, the data processing inequality provably implies that attacks can be no more powerful on the student model than they are on the teacher.
Stepping through the distillation process can help us anticipate how distillation can impact the privacy of the teacher set. The first step, training the teacher model, is the most well-understood from a privacy perspective, as the large literature on membership inference applies to the teacher model. In particular, recent work has found that state-of-the-art membership inference attacks are better at attacking some "outlier" examples than other "inlier" examples [CCNSTT22], which we expect to be true in the teacher dataset as well, making these "inlier" examples less vulnerable in later steps, as well.
Subsequent steps of distillation are less well-explored by the privacy literature. The second step of distillation creates the query dataset, which can be seen as "compressing" the teacher model into its responses on these queries. Intuitively, this step is the most important at reducing private information leakage. However, we hypothesize that some queries will capture information about teacher examples, perhaps due to some similarity between the queries and teacher examples. In the final step, the student model is trained on the query set. While this step cannot contain more sensitive information than the queries themselves, it is possible that this step makes that information easier to discover, perhaps by interpolating between the queries.
4.1 Performing Membership Inference on Teacher Examples
We now show that membership inference attacks work surprisingly well on distilled student models.
We first propose the "Transfer LiRA" attack, a simple extension of LiRA to the distillation setting. In this attack, we train shadow teacher models. We calibrate the IN and OUT Gaussians for LiRA on these shadow teacher models, and then run the attack by applying it directly to the student models. Notice that in this attack, distillation never happens in training shadow models—the success of this attack relies on the similarity between teacher and student models' predictions.
To capture the information loss because Transfer LiRA does not use distillation in any way, we also propose a second attack, which we call "End-to-End LiRA", where we instead train shadow models with the entire distillation procedure, first training shadow teacher models, and then distilling these teacher models into shadow student models. We then calibrate LiRA using these shadow student models. Note that performing this attack requires knowledge of the student training set to query the shadow teacher models and train the shadow student models. This is in contrast to Transfer LiRA which does not, although it still requires access to in-distribution data to train teacher models.
Distillation provides limited privacy. In Figure 3, we plot the change in per-example attack success rate on distilled models. We compare each example's vulnerability to LiRA on the teacher model (i.e. vulnerability without distillation) on the x-axis, to each example's vulnerability to End-to-End LiRA (i.e. vulnerability after distillation) on the y-axis. In the Appendix, we provide plots for
(b) Texas-100
Figure 4: Per-example membership inference accuracy using only student queries, compared to LiRA accuracy on the teacher model. Membership inference accuracies are non-trivial, and remain high for some examples. Effectiveness is dataset- and example-dependent, with Texas-100 having the most vulnerable examples in the teacher model remaining vulnerable to attacks based on teacher predictions.
(c) Purchase-100
Transfer LiRA (Figure 7), and ROC curves for each strategy (Figure 8). We also find our attack significantly outperforms a simple logit threshold baseline, similar to the weaker attacks used in prior work to evaluate distillation (Figure 12).
For a large fraction of teacher examples on each dataset, we find End-to-End LiRA achieves nearly the same membership inference performance as directly attacking the teacher model. In other words, many teacher examples do not observe any privacy benefits from distillation, despite student models never directly seeing these teacher examples! While the average membership inference accuracy (and TPR at low FPR) do decrease,  $5\%$  of examples' vulnerability drop by less than 8 percentage points on CIFAR-10, 5 points on Purchase-100, and 4 points on Texas-100, and these examples have a variety of teacher vulnerabilities. Because privacy is a worst-case guarantee, distillation provides limited privacy benefits.
We note two other interesting takeaways from these results. First, the per-example student attack success rate has a high variance. This variance is partly due to statistical uncertainty (although each example's attack success rate is computed with over 1000 models, giving each coordinate a standard deviation of 1.5 percentage points for both the  $x$  and  $y$  axes), but more interestingly, some examples do see significant privacy benefits from distillation, even controlling for the original model's vulnerability. We investigate duplication as one potential cause for this variance in Section 6.1. Second, each plot has a positive correlation, meaning that examples that are more vulnerable to attack on the teacher model attack also tend to be more vulnerable after distillation. As a result, reducing the vulnerability of the teacher model, perhaps using techniques such as differential privacy, are likely to improve the student's privacy.
5 Tracing Teacher Privacy Leakage Through Student Queries
It is surprising that membership inference still performs well on distilled student models, despite these models never directly using any of the teacher data. We now investigate why. For a membership inference attack to be successful on the student model, it must be the case that the student queries reveal some membership information about the teacher examples. However, to rigorously evaluate how this information is encoded in the student queries, we will turn our attention to designing a new attack that only has access to the student query dataset  $D_{Q}$  (and, in particular, has no knowledge of the teacher model's predictions on the teacher examples). To the best of our knowledge, this attack is also the only membership inference attack in the literature which does not require querying a model directly on an example (or on algorithmically derived examples) to predict its membership status.
An indirect attack using student queries. We adapt LiRA to a setting where only the student query scores are available to the adversary. Because of this limitation, the adversary can only rely on information about teacher examples that is indirectly contained in the student query scores. To adapt the attack to this setting, we use the same approach that LiRA uses to combine queries on
multiple "augmentations" of an example: multiplying their likelihood ratios to produce an aggregate membership score. Concretely, for each teacher example  $z_{j}^{T} = (x_{j}^{T},y_{j}^{T})$ , we fit a Gaussian distribution to the logits of each student example  $z_{i}^{S} = (x_{i}^{S},y_{i}^{S})$ , when  $z_{j}^{T}$  is either IN or OUT. We write the PDFs of these distributions as  $p_{j,i}^{IN}$  and  $p_{j,i}^{OUT}$ , so that the joint PDF of the IN Gaussians is  $p_{j}^{IN} = \Pi_{i}p_{j,i}^{IN}$  and similarly for the OUT Gaussians. This natural adjustment allows us to infer membership of the teacher examples using the student examples.