SLaM: Student-Label Mixing for  Distillation with Unlabeled Examples
Authors: Vasilis Kontonis
Fotis Iliopoulos
Khoa Trinh
Cenk Baykal
Gaurav Menghani
Erik Vee
Venue: NeurIPS 2023 poster (2023)
Keywords: Distillation
teacher
student
TL;DR: We propose SLaM: a novel method for distillation with unlabeled examples.

Abstract: Knowledge distillation with unlabeled examples is a powerful training paradigm for generating compact and lightweight student models in applications where the amount of labeled data is limited but one has access to a large pool of unlabeled data. In this setting, a large teacher model generates "soft" pseudo-labels for the unlabeled dataset which are then used for training the student model. Despite its success in a wide variety of applications, a  shortcoming of this approach is that the teacher's pseudo-labels are often noisy, leading to impaired student performance. In this paper, we present a principled method for knowledge distillation with unlabeled examples that we call Student-Label Mixing (SLaM) and we show that it consistently improves over prior approaches by evaluating it on several standard benchmarks. 
Finally, we show that SLaM comes with theoretical guarantees; along the way we give an algorithm improving the best-known sample complexity for learning halfspaces with margin under random classification noise, 
and provide the first convergence analysis for so-called ``forward loss-adjustment" methods.

SLaM: Student-Label Mixing for Distillation with Unlabeled Examples
Vasilis Kontonis
Fotis Iliopoulos Google Research fotisi@google com
Khoa Trinh Google Research
UT Austin
vasilis@cs. utexas. edu
khoatrinh@@oogle com
Cenk Baykal
Gaurav Menghani Google Research umenghani@googee com
Erik Vee
Google Research
Google Research erikvee@google. com
baykalcogoogle. com
Abstract
Knowledge distillation with unlabeled examples is powerful training paradigm for generating compact and lightweight student models applications where the amount of labeled data is limited but one has access to a large pool of unlabeled data. In this setting, large teacher model generates "soft" pseudo-labels for the unlabeled dataset which are then used for training the student model. Despite its success a wide variety of upplications, a shortcoming of this approach that the teacher's 's pseudo-labels are often noisy, leading impaired student performance. In this paper, we present a principled method for knowledge distillation with unlabeled examples that call Student-LLabel Mixing (SLaM) and show that it consistently improves over prior approaches by evaluating it on several standard benchmarks. Finally, show that SLaM comes with theoretical guarantees; along the way we give an algorithm improving the best-known sample complexity for learning halfspaces with margin under random classification noise, and provide the first convergence analysis for so-called "forward loss-adjustment" methods.
Introduction
While good quality human-labeled data are often hard to obtain, finding huge amounts of unlabeled data is relatively easy. Therefore, in modern machine learning applications, often face the situation where we have small "golden" dataset with human labels and large unlabeled dataset. In Distillation with Unlabeled Examples 12] 26 [6] large teacher model is first trained (or fine-tuned) on the human-labeled data and is then used to generate "soft" pseudo-lade -landd for the unlabeled dataset. Then the (typically smaller) student model, i.e., the model that will be deployed for the purposes of the application, is trained on the combined dataset that contains both the labels generated by humans and the pseudo-labels generated by the teacher model. This general-purpose training paradigm has been applied in a wide variety of contexts [16 46 53 54, 58 57) including but not limited to distilling knowledge from large-scale foundational models like BERT [8] and GPT-3 [ We remark that in such settings one does not have access to the teacher model but only on its pseudo-labels (which were generated during some previous "bulk-inference" phase). This "bulk-inference" step typically computationally expensive and happens once: one cannot modify the teacher network (or even use it for inference) during the training process of student.
Despite its widespread success in practice, the effectiveness of this powerful approach generally depends the quality of the pseudo-labels generated by the teacher model. Indeed, training the student model on noisy pseudo-lahels often leads to significant degradation of its generalization
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

performance, and this is well- phenomenon that has been observed and studied in plethora of papers in the literature, e.g., 6 36
44
51
53 g 27
In this work, we propose Student-Laael Mixing (SLaM), a principled method for knowledge dis- tillation with unlabeled examples that accounts for the teacher's noise and consistently improves over prior approaches. At the heart of our method lies the observation that the noise introduced by the teacher is neither random nor adversarial, in the sense that it correlates well with metrics of "confidence" such as the margin score or the entropy of the teacher's predictions. We exploit this empirical fact to our benefit in order to introduce a model for the teacher's 's noise, which we use to appropriately modify the student's 's loss function. At high level, for any given example during the student's training process, we evaluate the student's loss function on a convex combination of the student's 's current prediction and another (soft- Jlabll that we estimate using our model for the teacher's s noise (hence the name "student-label mixing").
Our contributions can be summarized as follows:
1. We propose SLaM: a principled method for improving knowledge distillation with unlabeled examples. The method is efficient, data-agnostic and simple implement.
2. We provide extensive experimental evidence and comparisons which show that our method consistently outperforms previous approaches on standard benchmarks. Moreover, we show that SLaM can be combined with standard distillation techniques such temperature scaling and confidence- based weighting schemes.
3. We give theoretical guarantees for SLaM under standard assumptions. As a byproduct of our analysis we obtain simple "forward loss-adjustment" iteration that provably learns halfspaces with 7-margin under Random Classification Noise with ((1)(<²))) samples improving over prior works that had worse dependence on either the margin y or the generalization error e (see Theorem 5.1 and Remark 5.2)
2 Related Work
Knowledge Distillation. Most of the literature on knowledge distillation has been focused on the fully supervised/labeled setting, i.e., when distillation is performed on the labeled training data of the teacher model rather than on new, unlabeled data- see e.g. the original paper of [26). Naturally, in this setting the pseudo-labels generated by the teacher are almost always accurate and so many follow-up works 22 14
41 [22 have developed advanced distillation techniques that aim to enforce greater consistency between the teacher's 's and the student's predictions, or even between the intermediate representations learned by the two models. Applying such methods our setting where the training dataset contains mainly unlabeled examples is still possible but, in this case, known [5T] 27 that fully trusting the teacher model can be actually harmful to the student model, making these methods less effective. (In fact, when the teacher is highly noisy these methods even underperform vanilla distillation with unlabeled examples.) In Section 4.2 we present results that show the improved effectiveness of SLaM relative to the Mfte-offtthe-ar supervised knowledge distillation methods like the Variational Information Distillation for Knowledge Transfer (VID) framework (2]. Moreover, in Appendix D.3 we show that our method can be combined with (i.e., provide an additional improvement) the most simple, yet surprisingly effective, methods of improving knowledge distillation, namely the pepperature-sclling idea introduced by (26].
15
For distillation with unlabeled examples, many approaches 17 33
29
propose filtering-out or
reweighting the teacher's pseudo-labels based on measures of teacher's uncertainty, such as dropout variance, entropy, margin-score or the cut-statistic. These methods are independent of the student model and can be synergistically combined with our technique. For instance, in Section D.4 we demonstrate that combining our method with sacche--nncertaitty-beee based reweighting schemes leads to improved student performance relative to applying the reweighting scheme alone.
Much more closely related to our approach is the recently introduced approach of (27 There, the authors design model for the teacher's 's noise and utilize it in order to modify the student's loss function so that, expectation, the loss simulates the loss with respect to noise-free pseudo-labels. One of the main advantages of our method compared to that of 27 is that our model for the teacher's noise is more structured and easier to learn, which - as our experiments in Section show leads to consistently better student performance.
2

---

Learning From Noisy Labels. Learning from noisy labels is an important and well-studied problem
see [50] for recent survey. The fundamental difference between our setting and papers in this literature is that the noise introduced by the teacher is structured, and this is crucial observation we utilize in our design. Specifically, our approach is inspired by the so-called forward loss-adjustment methods, e.g. [43] but it is specifically tailored to the structure of the distillation with unabeled examples setting. Indeed, forward methods typically attempt to estimate noise transition matrix whose (i, entry is the probability of the true label being flipped into a corrupted label j. which can be rather problematic when dealing with general, instance specific noise like in the case of distillation with unlabeled examples. On the other hand, we exploit that (i) we have access to confidence metrics of the teacher's 's predictions; and (ii) that often times, when the teacher model's 's top-1 prediction is inaccurate the true label is within its top-k predictions for some appropriate k to design and estimate a much more refined model for the teacher's noise that we use to inform the design of the student's loss function.
with a vast literature 7 21
23 28
37
40
45 47
Another related technique for dealing with noisy data is using "robust" loss functions 4 20 35 [66] such that they achieve a small risk for new clean examples even under the presence of noise in the training dataset. In Section 4.2 we compare our method with the general framework of (20) for designing robust loss functions and we show that our approach, when applied to the standard cross-entropy loss, consistently outperforms (20) in the setting of distillation with unlabeled examples. That said, stress that method is not tied to the cross-entropy loss and, in fact, it often gives better results when combined with more sophisticated loss functions. We demonstrate this in Appendix D.6 where we apply our method in cases where the student loss function comes from the families of Tosses introduced in (20) and 35
Semi-Superrisee Learning. Akin to our setting, in semi-supervised learning (SSL) (see e.g. [55] for a recent survey) the learner is presented with a small labeled dataset and a typically much larger unlabeled dataset B. Unlike to our setting though, there is typically no distinction between the student and teacher: the model of interest generates pseudo-labels on B which are utilized by using appropriate loss functions or preprocessing procedures (e.g. "filtering" or "correcting") often times in an iterative fashion with the goal of improving the quality of the newly- generated pseudo-labels. It is also worth noting that in many real-world applications of distillation with unlabeled examples either the teacher model is unavailable or it is too expensive to retrain it and create fresh pseudo-labels on the data (e.g., when we request labels from pretrained large language model). Therefore, SSL approaches that either (i) update the "teacher" model (e.g.. [34), or (ii) on
require several fresh teacher-generated pseudo-labels (e-g., by requesting random data-auggmentations or perturbed version of the unlabeled examples of B e.g., 99) are not applicable in our setting. We implement the recent SSL technique of (48) and show that our method outperforms it in the context of distillation with unlabeled examples. Besides performing on par with saute-of-the-ar SSL approaches like [9] the method of (88 is free of inherent limitations like using domain-specific data augmentations - which is also an important feature of our approach.
Learning Halfspaces with Random Classification Noise. The theoretical study of classification with Random Classification Noise (RCN) was initiated by (5). For the fundamental class of linear
classifiers (halfspaces) the first polynomial time algorithms for the problem where given in B and
[0]. The iteration proposed in [[3] is a "backward adjssadjutmmett method [33 for which it is
known that resulting optimization landscape is convex (for linear classifiers). In [19 improved analysis of the method of [13 was given, showing that SGD on this convex loss learns 7-margin halfspaces with RCN with ((1((7462) samples. On the other hand, forward loss-adjustment methods for dealing with RCN are known to result in an inherently non-convex landscape, see (3 and Figure 99. Our theoretical result for SLaM (see Theorem [.1p is the first convergence result for a "forward loss-adjustment" method and, at the same time, achieves a sample complexity of 0(1((?2e2) improving over the prior work.
3 SLaM: Student-Label Mixing Distillation
In this section, we describe our distillation with unlabeled examples setting and present SLaM. In what follows, we assume that examples are represented by feature-vectors some space X. We shall denote by X the distribution over examples. We consider multi-class classification with L classes and assume that the ground-truth label of an example ar is represented by a one-hot vector in y = {0, 1)2 given by some unknown function g(z) X + y. In multi-class classification the learning algorithm typically
3

---

optimizes parametric family of classification models F= [f(::w] w) [f;(;;; X + RA : w E W), i... for every parameter W. f(; w) is an L-dimensional "score vector". where f(z; w); corresponds to the probability that the model assigns to the class for the example v. We shall denote by ((::)) R x + R. the classification loss function used by the learning algorithm. During training the algorithm considers a set of labeled examples S = {(x()) (o(z()))...... (a(m), (a(m).(((()) g(x(1))) and optimizes the loss ((.,)) over S, i.e., solves the problem min TST E (x.a())) ((g(z), [ w))
For two vectors v, u E R we denote by crr(u, u) = 1 1(argmax(v) & argmax(u)) the indicator of the event that the positions of the maximum elements of v, u agree. Similarly, for two classifiers h()), f(x): Rd + R we can use err(h(x) [()) to denote whether their top-1 predictions for the example x agree. Our goal to train classifier over the sample S so that its generalization error, i.e., E (err(eeeerrrreee [x; w), (x))), is small.
Distillation with Unlabeled Examples. We assume that are given a (usually small) dataset A of correctly labeled examples (I, (r.)( and a set of unlabeled data U. A "teacher" model (+(+) : X + R% is first trained on the labeled dataset A and then provides soft-labels for the examples of dataset i.c., we create a dataset B = {(x, (z)) : E L/} containing examples labeled with the corresponding probability distribution over classes (soft-labels) of the teacher model. We then train a (typically smaller) student model using both the original labeled data A and the teacher-labeled dataset B, i.e., minwew | E(s.))EAUB" E(2, (a; w)). In what follows, we shall call the above training procedure as vraiila-diitiillttio
Remark 3.1 ("Hard-" vs "Soft-" Distillation). We remark that the process where instead of using the soft-labels provided by the teacher model on the unlabeled dataset U, we use one-hot vectors representing the class with maximum score according to the teacher, is known as hard-distillation. We will denote by ys (x) the soft-label of the teacher and by y(z) the corresponding hard-label, i.e., v(2) is the one-hot representation of argmax y. (z). When it is clear from the context we may simply write y instead of y(z).
Modelling the Teacher as a "Noisy" Label Oracle. In the distillation setting described in the previous paragraph, it is known [51 27 g 44) that the teacher model often generates incorrect predictions on the unlabeled examples, impairing the student's 's performance. Given any æ E U, we model the teacher's 's prediction y as random variable. Similarly to 227 we assume that, for every unlabeled datapoint x E U, the provided teacher label is correct with probability a(r) and incorrect with probability 1 (z)). However, contrast with [77) our noise model prescribes a non-advsersarial (semi-random) behavior of the teacher when its top-1 prediction is incorrect.
A first step towards more benign noisy teachers is to assume that, conditionally on being wrong, the teacher label is a uniformly random class of the remaining L- classes We remark that this model is already enough to give improvements in datasets with moderately large number of classes (e.g., up to 100). In particular, it perfectly captures the noisy teacher in binary classification: when the teacher label is different than the ground-truth g(x) then it has to be equal the "flipped" ground-truth 1 (x)).
We now further refine our model so that it is realistic for datasets with thousands of classes. Even though the top-1 accuracy of the teacher model may not be very high on the unlabeled data U, the true label is much more likely to belong in the top-5 top-10 predictions of the teacher rather than being completely arbitrary. For example, training a ResNet50 network on 10% of ImageNet 49 yields average 1 accuracy about 52.78% on the test dataset whereas the top-10 accuracy of the same model is about 83.55%. In datasets with a large number of classes, this observation significantly reduces the number of potential correct classes of the examples where the teacher label is incorrect. Motivated by the above, we assume the following structured, semi-random noise model for the teacher, tailored to multi-clas settings.
Definition 3.2 (Noisy Teacher Model). Let x be any example of the unlabeled data U and denote by g(x) its ground-truth label. Let ys (zr) resp. y(ι) be the random variable corresponding to the soft resp. hard prediction of the teacher model for the example x. We assume that for every there exist (unknown to the learner) (xx))) E [0, 1] and k(x) E (2...., . I.) such that the teacher's 's top-1 prediction y agrees with the ground-truth g(z) with probability a(x) and, with probability 1 - a(x): (i) the
4

---

ground truth belongs in the top-k(æ:) predictions of the teacher; and (ii) the teacher's (hard) prediction a uniformly random incorrect class out of the top-k(a) predictions of the teacher soft-label y. (xx)
Remark 3.3. We remark that the model of Definition 3.2 captures having a "perfect" teacher model by
setting (xx) = for all x and also generalizes the binary case described above by taking k(x) - for all x E X.
Given the above noise model for the teacher, the problem of improving hhowledge-distillatee consists of two main tasks: (i) obtaining estimates for accuracy statistics azx)), (xx) for each example U: and (ii) using those estimated values to improve the training of the student model so that it is affected less by the mistakes of the teacher on dataset B.
Training Better Students Using a(x), k(x) We first assume that for every x we have oracle access to the values (z)), k(x) and present our Student-Label Mixing loss function. Instead of using (xx), k(x) to "denoise" the teacher's 's label, we use them to add noise to the student's 's predictions. To make notation more compact, in what follows, given a vector
RL we denote by top(2; k) the vector that has the value 1 in the positions of the of the 1-st up to k-th largest elements of 2 and 0 in all
other positions, e.g., top((1, 2, 3); 1) (0. O, 1) and top((-1, 1, 0, 2); 3) (0, 1, 1, 1). Assuming
that the student-label for some E U is f(z; w) we "mix" it (hence the name Student-Label Mixing) using (xx), (xx)) to obtain the mixed prediction
mix(f((=))) k(x)) = (x)) + (1 (1-afe a())) (o)(v((t); * 1- f(x; w)
(1)
k(x) -
where 9 * is the element wise multiplication of the vectors p, q. We then train the mixed student model, on the "noisy" dataset B:
min NEW LA U BI
(12 + (H,2)EA
Σ
Σ Ely, mix(f w): (())
(N.Y)EB
The main intuition behind the mixing of the student's 's labels is that by training the "noisy" student to match the "noisy" teacher label y on dataset B, the underlying (non-mixed) student [(z; w)) will eventually learn the ground-truth. In particular, when E(.,,) is the Cross-Entropy loss we have that the expected mixed loss conditioned any is
E[e(y; mix(f (f; w), a(z))
x] = {mmix(g(x); (zx), k())), mix(f(;;; (() (x)), (()))
where we used the fact that the pross-entropy is linear in its first argument, and that by the definition of our noise model (Definition 3.2) it holds that E byy x]
mix(g(x); (xx), k(z)). Therefore, when the student is equal to the ground-truth f(x; w) = g(x), we obtain that the mixed student- model will satisfy mix(g(2); (x)), k(x)) = mix(J It (æ; w); axxx), k(x)) for all x E X, and (by Gibb's s inequality), we obtain that g(x) is a minimizer of the SLaM loss. We show the following proposition, see Appendix for the formal statement and proof.
Proposition 3.4 (SLaM Consistency (Informal)). examples of dataset B, i.e., we first draw
Let D be the distribution of the teacher-labeled X and then label it using the noisy teacher of Definition 3.2 Moreover, assume that there exists some parameter w" E W such that the ground-truth g(x)) fax; w²) Then w" is the minimizer of the (population) SLaM objective: min, E( cely, mix(f ((; (x;w)); o(x)), k(x)))), where ce(,;)) is the Cross-Entropy loss.
Estimating the Teacher's Accuracy Statisties alz). k(z) via Isotonic Regression We first show how we estimate a(x) for each x of dataset B, i.e., the dataset labeled by the teacher model. In (2 the authors empirically observed that axxx) correlates with metries of teacher's confidence such as the "margin", i.e., the difference between the probabilities assigned in the top-1 class and the second largest class according to the teacher's soft label y.- In particular, the larger the margin is the more likely is that the corresponding teacher label is correct. We exploit this monotonicity by employing isotonic regression on a small validation dataset to learn the mapping from the teacher's margin at an example z to the corresponding teacher's 's accuracy a(x). For more details, see Appendix B.I
To perform this regression task we use a small validation dataset V with correct labels that the teacher has not seen during training. For every example x E V we compute the corresponding
Giien that the teacher's 's prediction is incorrect and that the ground-trutt belongs in the top-k(x) predictions of the teacher, assumption (ii) describes uniform distribution on k(x) 1 labels.
5

---

soft-teacher label (z) and compute its margin margin(x) = max, (z)) max (I))) For every z E V we also compute the hard-prediction of the teacher and compare with the ground-truth. i.e., for every E V the covariate and responce pair is (margin(x). 1 - err(g(x), v(r))). We then use isotonic regression to fit a piecewise constant, increasing function to the data. We remark that isotonic regression can be implemented very efficiently in On log n) time (where n is the size of the validation dataset).
For k(æ) we consider two different options: (i) using the same value for all examples (e.g., using k so that the top-k accuracy of teacher is above some threshold on the validation data); and (ii) using a "data-dependent" k(z)) that we estimate by solving L (recall that L is the number of classes) rserenie-regression problems (similar to that for estimating (x) above). We refer Appendix B.1 for more details.
4 Experimental Evaluation
In this section, we present our experimental results. In Section 4.1 we describe our experimental setup and in
Tenthers Accurccy
Section 4.2 4.2 we compare the performance of our method with previous approaches on standard benchmarks. In Sec- tion [oonDDawe we show that our method can be combined with teacher-uncertaintt- based reweighting techniques. Finally, due to space limitations, we provide additional empirical results in the Appendix: in Appendix D.5] we show that SLaM can effectively be used with distillation temperature, and in Appendix [D6 we consider using SI SLaM with other losses beyond the Cross-Entroph
botone Pepressice WIM Repression
verto, senenepott
Learning a(x) via iso- tonic regression. The data were generated by a ResNet 110 teacher trained 5000 examples of CIFAR- 100 and evaluated on a validation dataset V of 500 examples. The regression data ((mar err(y,(x) (2), g(x))) x V) are shown in gray (the response is binary 0/1). By enforcing monotonicity, isotonic regres- sion yields a more stable and robust curve than, for example, the KNN pre- dictor.
Figure 1:
4.1 The Setup
Here, we describe our procedure for simulating knowl- edge distillation with unlabeled examples on academic datasets. We start by splitting the training dataset in two parts: dataset A and dataset C. We then train the teacher and student models on dataset A (using the standard cross- entropy loss) 2 Then we perform multiple independent trials where, for each trial, we randomly split dataset C into a small (e.g.. 500 examples validation dataset V and an unlabeled training dataset U. For each trial we (i) use the teacher model to label the points on dataset U to obtain the leaeher-labeled dataset B (ii) initialize the weights of the student to those of the student model that was pre-trained on dataset A; (iii) train the student model (using each distillation method) on the combined labeled data of A, V (that have true labels) and the data of B (that have teacher labels). We remark here that we include the validation data V during the training of the student be fair towards methods that do not use validation dataset. However, while it is important that the teacher has not seen the validation data during training, the performance of no method was affected significantly by including (or excluding) the validation data from the training dataset.
4.2 Comparison with Previous Approaches
The Baselines A natural question is whether a more sophisticated distillation method that enforces greater consistency between the teacher and the student, would improve distillation with unlabeled examples: we use the VID method [2) that incorporates the penultimate layer of the student model (after a suitable trainable projection) in the loss. We also compare distillation method of (27) that reweights the examples of dataset B in order to "correct" the effect of the noisy pseudo-labels provided by the teacher. The Taylor cross-entropy method of [20 is
method against the weighted
"e remark that our method does not require pre-training the student on dataset A, however, since 27 requires pre- pre-training the student, we do the same for all methods that compare.

---

CINRTO Sart D- Bili
CIAA3OO Sart Mawa
CAAAA SARLO SiBIEe
In
136
25%
254
30% 35%
15%
30%
34%
3/4
X
Comparison of distillation methods on CIFAR-10,100 100 and CelebA. On the horizontal axis we plot the size of Dataset A as a percentage of the whole training dataset. On the vertical axis we plot the accuracy of the trained student-model on the test dataset.
Figure 2:
Table 1: Experiments on CIFAR-10 10 (soft-distillation). See Section 4.2 for details.
Labeled Examples
5000
7500
10000
12500
15000
17500
Teacher
61.30
68.98
72.42
73.92
76.63
78.63
Vanilla
63.53
0.29
70.39 1: 0.11
73.23 1 0.15
0.25
76.84
75.63
0.16
Taylur CE 20
64.07
0.26
71.19
+: 0.17
74.18
0.25
74.65
0.24
77.17 4 0.04
75.67
0.13
UPS 4R
64.56 14.56:2..1 0.13
71.10
0.34
74.17
0.06
75.05
0.24
77.64 1 0.12
0.22
VID 1
63.76 0.13
70.58
+ 0.17
73.77
0.40
+ 0.40
74.95
0.21
77.25 +: 0.06
78.23
0.09
Weightod 27
63.85
71.04
0.22
73.64
4: 0.36
75.00
17.40::0011
78.93
SLaM (Oua))
66.82
0.61
72.61 + 0.30
75.01
+ 0.26
76.72
+ 0.17
78.04 + 0.16
79.22 0.11
a modification of CE that truncates the taylor-series of the CE loss. In 120] it was shown that it offers significant improvements when the labels are corrupted by random classification noise. The fact that the teacher's 's noise is much closer to random than to adversarial makes this approach a natural baseline. The UPS loss of 48 a semi-supervised technique that takes into account the variance (uncertainty) of the teacher model on the examples of dataset B in order to transform the soft pseudo-labels provided by the teacher to more "robust" binary vectors and then use a modified binary CE loss. To estimate the uncertainty of the teacher model, we used either dropout with Monte-Carlo estimation or random data-augmentattonn as suggested in [48]. We remark that, as we discussed in Section
2 and Section [ strictly speaking, this method is not applicable in our setting because it requires multiple forward passes of the teacher model to estimate its variance but we implement it as it is relevant approach that aims to improve the pscudo-labels of the teacher.
CIFAR-(10,100 and CelebA Here we present our results on CIFAR- (10, 100) 30 and
CelebA (22]. CIFAR-10 and 110 are image classification datasets with 10 and 100 classes respectively. They contain 60000 labeled images, which are split to a training set of 50000 images, and a test set of 10000 images. From the 50000 images of the train set we use the 10%, 15%, 20%, 25%, 30%, 35% (or 5000, 7500, 10000, 12500, 15000, and 17500 examples) as the labeled dataset where we train the teacher and pre-train the student models. For each size of dataset A, we perform a random split on the remaining training data and use 500 labeled examples as the validation dataset and the remaining examples as the unlabeled dataset U. For the CIFAR- 10 experiments, we use a Mobilenet with depth multiplier 2 as the teacher, and a Mobilenet with depth multiplier 1 as the student. For CIFAR- 100, we use a ResNet- 10 as a teacher, and ResNet-56 as the student. We compare the methods both on soft- and hard-distillation For each trial we train the student model for 200 epochs and keep the best test accuracy over all epochs. We perform 3 trials and report the average of each method and the variance of the achieved accuracies over the trials. The results of our experiments for doft-distillation can be found in Table [ and Table The corresponding plots are given inFigure 2
We include our results for hard-distillation in Appendix D.2
We consider the male/female binary classification task using the CelebA dataset (2 consisting of a training set of 162770 images and a test set of 19962 images. We use MobileNet with depth multiplier as the teacher, and ResNet- 11 as the student. As the labeled dataset A we used 2%, 3%, 4%, 5%, 6% percent (or 3256, 4883, 6510, 8138, 9766. 11394 examples) of the training dataset and split the remaining data in validation dataset of 500 examples and an unlabeled dataset U. Our results for CelebA can be found in Table 3 -sott-distillation) and in Table 7 hhadd-distillation).
7

---

Table 2: Experiments on CIFAR-100 (soft-distillation) See Section 4.2 for details.
Labelol Examples
5000
7600
10000
12500
15000
17500
Teacher
35.97
44.66
49.62
55.68
59.19
62.05
Vanilla
37.94
0.30
4G.42
0.24
02.17 # 0.21
57.72
= 0.17
60.91 + 0.07
63.47
0,23
Taylur CE
40.18
0.07
48.05
1: 0.29
54.08
0:24
58.45
0.17
61.13 1 0.10
63.54
0.26
UPS [8
39.02
0.23
48.48 + 0.15
54.43
+ 0.27
58.17
+ 0.07
60.74 4 0.10
02.13
0.12
VID 1
38.93 2: 0.39
46.76 1: 0.10
52.56
1 0.17
$7.94
:: 0.37
61.14
1 0.28
63.56
0.18
Weighted 2
38.63
0.32
47.11
53.16
0.25
0.11
61.29
0.15
63.58
0.07
SLaM (Ous)
42.72 0.30
49.89
1 0.23
54.73
0.27
58.78
0.15
61.30
0.09
63.98
0.19
Table 3: Experiments on CelebA (soft-distillationn) See Section 4.2 for details.
Labeled Examples
2%
3%
4%
6%
6%
7%
Teacher
68.25
91.31
92.62
Vanilla
89.96 0.08
91.55
0.14
02.16:00.10
03.42 = 0.06
95.98 + 0.04
94.29 1: 0.03
TaylurCCE 20
90.80 0.00
92.23
0.1
92.56 1 0.14
93.80
0.20
94.17 1 0.07
94.47
0.0.
UPS 48
80.96
92.03
0.00
92.44
0.04
93.9 0.05
94.28 + 0.07
94.68 4 0.03
VID 1
80.91
91.75
0.22
02.21
94.15
94.33
0.11
Weighted Er
89.92 = 0.12
91.73 + 0.09
92.31
+ 0.22
93.64 = 0.10
93.93
+ 0.14
94.23
#
SLaM (Ouas)
90.37
0.17
92.25 +: 0.11
92.74 +: 0.17
94.06
0.07
94.39
0.10
94.75
+ 0.08
The corresponding plots are given in Figure Due to space limitations our results for hard-distillation can be found in Appendix [D.2]
Taken together, our comparisons show that SLaM consistently outperforms the baselines, often by a large margin. The reader is referred to Appendix D. for additional details.
Remark 4.1 (Soft-Distillation and Temperature Scaling). We remark that in the comparisons we performed soft-distillation with temperature set to 1, i.c., for every example do not scale the corresponding teacher and student logits. In Appendix [D.5] we show that our method can readily be used together with temperature scaling improve the accuracy of the student model.
ImageNet Here we present the re- sults on ImageNet [49]. ImageNet is an image classification dataset with 1000 classes consisting of training set of approximately 1.3 million images, and a test set of 50000 images. From the 1.3 mil- lion images of the training set we use the 5%, 10%, 15%. 20% percent (or 64058, 128116, 192174, 256232 examples) the labeled dataset A where we train the teacher and pre- train the student models. For each size of dataset A, we perform a ran- dom split on the remaining training data and use 10000 labeled examples the validation dataset and the re- maining examples as the unlabeled dataset U. We use ResNet-50 as the teacher, and ResNet-18 18 as the student. We compare the methods on coft-distillation. For each trial, we train the student model for 100 epochs
ImageNet: Soft Distilation
TyyorCE
(Durs)
20
45-
40
10%
20%
30%
Examples
Figure 3: Comparison of distillation methods on ImageNet. On the horizontal axis we plot the size of Dataset A percentage of the whole training dataset. On the vertical axis we plot the accuracy of the trained student-model on the test dataset.
and keep the best test accuracy over all epochs. We perform 4 trials and report the average of each method and the variance of the achieved accuracies over the trials. Our results for ImageNet can be found in Table We remark that we do not include the results of the UPS method in Table did not improve over the accuracy achieved after pre-training the student model on dataset A. The reader is referred to Appendix D.I for additional details.
8

---

Table 4: Experiments ImageNet (soft (soft-distillationn. See Section 4.2 for details.
Laheled Exumples
6%
10%
15%
20%
25%
30%
Toacher
39.48
52.96
59.64
63.62
66.00
67.85
Vinilla
41.67
0.05
55.9
0.06
62.3
0.09
65.91
0.05
07.98
0.07
69.12 + 0.08
Taylor CE 2
41:61
0,06
56.43
0,06
62.38
0.11
65.88
1: 0.08
67.70
0.22
88, 62 1: 0.07
VID E
40.12 + 0.04
52.79
+ 0.04
58.01
# 0.03
61.21
+: 0.06
02.37
+ 0.00
63.05 # 0.07
Weighted 22
41.67 2: 0.01
55,96
1: 0.07
62.29
1: 0.08
65.91
1: 0.05
67.96
0.06
69.16
1: 0.08
SLaM (Ouns
48.1
0.05
59.51
0.07
64.08
0.06
0.11
68.17
0.07
69.07
Large Movies Reviews Dataset Here we present results on the Large Movies Reviews Dataset 39 This is dataset for binary sentiment classification containing 25000 movie reviews for training and 25000 for testing. We use an ALBERT-large model [32 as teacher, and an ALBERT -Lase model as a student. We use 2%, 4%, 8%, 40% percent (or 500, 1000, 2000, 10000 examples) from the training dataset and split the remaining data in validation dataset of 500 examples and an unlabeled dataset U. Our results and more experimental details can be found in Appendix D33
Performance Gains of SL.MM as a Function of The Number of Labeled Examples In our experiments, the fraction of examples we consider "labeled" controls two things at the same time: (i) the accuracy of the teacher model - as the teacher is trained on the labeled examples available; and (ii) the number of unlabeled examples the teacher model provides pseudo-labels for. The more inaccurate the teacher model is, the better the improvements provided by our method. (Given a "perfect" teacher that never generates incorrect pseudo-labels for the unlabeled examples, our method is mathematically equivalent to the "vanilla" approach (see the mixing operation in Equation (I). Therefore, the smaller the number of labeled examples available, the bigger the performance gains of SLaM as (i) the teacher will be less accurate; and (ii) it has to generate labels for more unlabeled examples (and therefore the absolute number of inaccurate predictions that SL.M "corrects" increases statistically). It is worth emphasizing that the main reason behind the enormous success of distillation is exactly that the teacher network can blow up the size of the student's training dataset: in practice, the ratio of labeled examples to unlabeled examples is typically (much) less than 1%.
Distilling Linear Models and Learning Noisy Halfspaces
In this section we show that, when the dataset is separable by a halfspace, i.e., for every example æ, the ground-truth is g(x) = (1(w" >> 0}, 1(w" S 0}) for some unknown weight vector w", then using SLaM with a linear model as the student will recover the ground truth classifier. We make the standard assumption that the ground-truth halfspace has y-margin, i.c., that || ||2 1 and that holds Iw= x 2 y for all examples x. For a fixed example x, the observed noisy teacher- label y satisfies Definition 3.2 i.c., y = =(x)) w.p. a(x) and y
9(x) w.p. 1 - a(x) (since k = 2 for binary classification). Our approach consists of using the standard cross-entropy loss ce(p, 9) and training moudent-model consisting of linear layer plus soft-max activation, i.e.,
f(x; w) = I+e-
Theorem 5.1 (SLaM Convergence). Let X be a distribution on Rd and g(x) be the ground-truth halfspace with normal vector w* E Rd. Let D be the distribution over (noisy) teacher-labeled examples (z, y) whose x--arginal is X. Assume that there exist B. >> O such for all examples
x in the support of X it holds that Iw" x| 2 y and 11/2 a(x) Z
Let > O. After
T = (32 2)) SGD iterations the SLaM objective (see Algorithm 3) with probability at
least 99%, there exists an iteration t < T where P [err(f (x; w(t)) g.g(z)) Sec.
Remark 5.2 (Learning Halfspaces with RCN). The problem of learning halfspaces with Random Classification Noise (RCN) can be modeled as having a teacher with constant accuracy probability, i.c., a(r) a > 1/2 for all x. As a corollary of Theorem 5.1 we obtain an efficient learning algorithm for m-magn halfspaces under RCN achieving a sample complexity of O(1((?2)2), Prior to our work, the best known sample complexity for provably learning halfspaces with RCN was
(() [19 where the "backward loss- adjustmmttmm"" of [3 was used.

---

Conclusion, Limitations, and Broader Impact
In this work we propose SLaM, a novel and principled method for improving distillation with unlabeled examples. We empirically show that SLaM consistently outperforms the baselines, often by a large margin. We also showed that SLaM can used with and improve (i) knowledge distillation with temperature scaling: (ii) loss functions beyond the standard Cross-Entropy loss; and (iii) confidence-based weighting schemes that down-weight examples where the teacher model is not very confident. Apart from extensive experimental evaluation, establishing the consistency and optimality of SLaM. As a byproduct of our theoretical analysis, obtain iteration for learning ? margin halfispaces with RCN that improves the best known sample complexity for this problem.
provide strong theoretical guarantees
A limitation of SLaM is that it does not necessarily improve over vanilla distillation when the teacher model makes only a few mistakes (this is to be expected as our method designed for the case where the sracher-model is imperfect).
is very popular deep learning method, and therefore, potentially malicious usage of our work is an important societal issue, as deep learning has far-reaching applications from NLP Robotics and Self- Self-Driving cars.
chowledge-distillaeee
References
[1] Martín Abadi, Ashish Agarwal, Paul Barham, Fugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv: 2016.
[2] Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Vari- ational information distillation for knowledge transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9163 9171, 2019.
[3] Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D. Lawrence, and Zhenwen Dai. Vari- ational information distillation for knowledge transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[4] Ehsan Amid, Manfred KK Warmuth, Rohan Anil, and Tomer Koren. Robust bi-tempered logistic loss based on bregman divergences. Advances in Neural Information Processing Systems, 32, 2019.
[5] D. Angluin and P. Laird. Learning from noisy examples. Machine Learning, 2(4):343-370, 1988.
[6] Eric Arazo, Diego Ortego, Paul Albert, Noel O'Connor, and Kevin McGuinness. Pseudo- labeling and confirmation bias in deep semi-supervised learning. In 2020 International Joint Conference on Neural Networks (IJCN), pages 1-8. IEEE, 2020.
[7] Noga Bar, Tomer Koren, and Raja Giryes. Multiplicative reweighting for robust neural network optimization. arXiv preprint 0rXiv:2102.12/990 2021.
[8] Cenk Baykal, Khoa Trinh, Fotis Iliopoulos, Gaurav Menghani. and Erik Vee. Robust active distillation. International Conference on Learning Representations (ICLR), 2023.
[9] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach semi-supervised learning. Advances in neural information processing systems, 32, 2019.
[10] A. Blum, A. Frieze, R. Kannan, and S. Vempala. A polynomial-time time algorithm for learning noisy linear threshold functions. Algorithumica, 22(1):35-55) 52, 1998.
[11] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1 2020.
10

---

[12] Cristian Buciluä, Rich Caruana, and Alexandru Niculescu-Mizil Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 535-541, 2006.
[13] T. Bylander. Learning linear threshold functions in the presence of classification noise. In Proceedings of the seventh annual conference on Computational learning theory, pages 340-347, 1994.
[14] Hanting Chen, Yunhe Wang, Chang Xu, Chao Xu, and Dacheng Tao. Learning student networks via feature embedding. IEEE Transactions on Neural Networks and Learning Systems, 32(1):25-
35, 2020.
[15] Liqun Chen, Dong Wang. Zhe Gan, Jingjing Liu, Ricardo Henao, and Lawrence Carin. Wasser- stein contrastive representation distillation. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 16296-16305, 2021.
[16] Ting Chen, Simon Korbllith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models strong semi-supervised learners. Advances in neural information processing systems, 33:22243-22255, 2020.
[17] Mostafa Dehghani, Arash Mehrjou, Stephan Gouws, Jaap Kamps, and Bernhard Schölkopf. Fidelity- learning. arXiv preprint arXiv:1711.02799 2017.
[18] Jacob Devlin, Ming-Woc Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800 2018.
[19] I. Diakonikolas, T. Gouleakis, and C. Tzamos. Distribution-indeeendee pac learning of halfspaces with massart noise. Advances in Neural Information Processing Systems, 32, 2019.
[20] Lei Feng, Senlin Shu, Zhuoyi Lin, Fengmao Lv, Li Li, and Bo An. Can cross entropy loss be robust to label noise? In Proceedings of the Twenty-Ninth International Conference International Juint Conferences on Artificial Intelligence, pages 2206-2212, 2021.
[21] Benoft Frénay and Michel Verleysen. Classification in the presence of label noise: a survey. IEEE transactions neural networks and learning systems, 25(5):845 869, 2013.
[22] Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti. and Anima Anandkumar. Born again neural networks. In International Conference on Machine Learning, pages 1607- 1616. PMLR, 2018 8.
[23] Dragan Gamberger, Nada Lavrac, and Ciril Groselj. Experiments with noise filtering in a medical domain. In /CML, volume 99, pages 143-151, 1999.
[24] Aritra Ghosh, Himanshu Kumar, and P Shanti Sastry, Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
(25) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[26] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. CoRR, abs/1503.02531, 2015.
[27] Fotis Iliopoulos, Vasilis Kontonis, Cenk Baykal, Gaurav Menghani, Khoa Trinh, and Erik Vee. Weighted distillation with unlabeled examples. In NeurlPS, 2022.
[28] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International Conference Machine Learning, pages 2304-2313, PMLR, 2018.
[29] Akisato Kimura, Zoubin Ghahramani, Koh Takeuchi, Tomoharu Iwata, and Naonori Ueda. Few-shot learning of neural networks from scratch by pseudo example optimization, arXiv preprint arXiv: 1802.03802.00039 2018
11

---

[30] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[31] Abhishek Kumar and Ehsan Amid. Constrained instance and class reweighting for robust learning under label noise. arXiv preprint arXiv:2/11.053: 2021.
[32] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv: 19942, 2019.
[33] Hunter Lang, Aravindan Vijayaraghavan, and David Sontag. Training subset selection for weak supervision. arXiv preprint arXiv:2206.02911, 2022.
[34] Mong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning. ICML, volume 3, page 896, 2013.
[35] Zhaoqi Leng, Mingxing Tan, Chenxi Liu, Ekin Dogus Cubuk, Jay Shi, Shuyang Cheng, and Dragomir Anguelov. Polyloss: A polynomial expansion perspective classification loss functions. In International Conference on Learning Representations, 2022.
[36] Lu Liu and Robby T Tan. Certainty driven consistency loss multi-teache networks for semi-supervised learning. Pattern Recognition, 120: (40, 2021.
[37] Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on pattern analysis and machine intelligence, 38(3):447-461, 2015.
[38] Michal Lukasik, Srinadh Bhojanapalli, Aditya Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? In International Conference on Machine Learning, pages 6448-6458 PMLR, 2020.
[39] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguisties: Human Language Technologies, pages 142-150 150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics.
[40] Negin Majidi, Ehsan Amid, Hossein Talebi, and Manfred K Warmuth. Exponentiated gradient reweighting for robust training under label noise and beyond. arXiv preprint arXiv:2104.01493, 2021.
[41] Rafael Müller, Simon Kornblith, and Geoffrey Hinton. Subclass distillation. arXiv preprint arXiv:2002.039336 2020.
[42] Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. Advances neural information processing systems, 26, 2013.
[43] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: loss correction approach. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1944-1952, 2017.
[44] Hieu Pham, Zihang Dai, Qizhe Xie, and Quoc V Le. Meta pseudo labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11557-11568,
2021.
[45] Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q Weinberger. Identifying mislabeled data using the area under the margin ranking. Advances in Neural Information Processing Systems, 33:1704441170000 2020.
[46] Ilija Radosavovic, Piotr Dollár, Ross Girshick, Georgia Gkioxari, and Kaiming He. Data distillation: Towards supervuuer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4119-4128, 2018.
[47] Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International conference on machine learning, pages 4334 4343. PMLR, 2018
12

---

[48] Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo-labeling: An uncertaint--aware pseudo-label selection framework for supervvie learning. In International Conference on Learning Representations (ICIRR), 2021.
[49] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115(3):211-255 2015.
[50] Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae- Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Systems, 2022.
[51] Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew G Wilson. Does knowledge distillation really work? Advances in Neural Information Processing Systems, 34:6906 6919, 2021.
[52] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv preprint arXiv:I910. 10699, 2019.
[53] Qizhe Xie, Minh-Thang Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10687-10698, 2020.
[54] I Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi-supervised learning for image classification. arXiv preprint arKiv: arXiv: 1905.00546, 2019.
[55] Xiangli Yang, Zixing Song, Irwin King, and Zenglin Xu. A survey on deep semi-supervised learning. IEEE Transactions on Knowledge and Data Engineering, 2022.
[56] Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018.
[57] Barret Zoph, Golnaz Ghiasi, Tsung- Yi Lin, Yin Cui, Hanxiao Liu, Ekin Dogus Cubuk, and Quoc Le. Rethinking pre-training and self-training. Advances in neural information processing systems, 33:3833-388 3845, 2020.
[58] Yang Zou, Zhiding Yu, Xiaofeng Liu, BVK Kumar, and Jinsong Wang. Confidence regularized self-training, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5982-5991, 2019.
13

---

A Notation
For two vectors p. q E Rd we denote by p = Di=1 P:q; their inner product. We use p* q to denote their element- product, i.e. (p q): Piq. We use the notation max; p to denote the i-th largest element of the vector p. We use margin (p) to denote the difference between the top-2 elements of p. i.., margin(p) = max1 p maXę p. Moreover, we marging (p) to denote the top-k margin, ie.. marging (p) Σ> max; p - max p. Given function f(w) R <> R. we denote by (ww) the gradient of f with respect to the parameter w.
B Detailed Description of SLaM
B.1 Estimating the Teacher's Accuracy Parameters: a(x), k(z)
Estimating the Teacher's Accuracy a(x) via Isotonic Regression We now turn our attention to the problem of estimating (xx) for each x of dataset B, i.e., the dataset labeled by the teacher model. In [27] the authors empirically observed that a(x) correlates with metrics of teacher's confidence such as the "margin", i.e., the difference between the probabilities assigned in the top- 1 class and the second largest class according to the teacher's soft label ys- In particular, the larger the margin is the more likely is that the corresponding teacher label is correct. We exploit (and enforce) this monotonicity by employing isotonic regression on a small validation dataset to learn the mapping from the teacher's margin an example x to the corresponding teacher's accuracy a<x)
To perform this regression task we use a small validation dataset V with correct labels that the teacher has not seen during training. For every example E V we compute the corresponding soft-teacher label (x) and compute its margin margin(x)) = max (x)) max2(y (rr). For every V we also compute the hard-prediction of the teacher and compare it with the ground- truth, i.e., for every x the covariate and responce pair is amargin(2), 1 err(g(x), y(z)).. We then use isotonic regression to fit piecewise constant, increasing function to the data. Sorting the regression data ((margin(x)) - err(g(z), v(z))) E V) by increasing margin to obtain a list (c()) (...,,(()) (()),, ),..,, (c(m), p())), isotonic regression solves the following task
Σ (n - 200
min
(01) (Em)
subject to lb S p(0) S p(++1) S,,
where the parameter lb is a lower bound on the values ,(() and is a hyper-parameter that we tune. On the other hand, the upper bound for the values can be set since know that the true value (I) is at most for every x (since it corresponds to the probability that the teacher-label is correct). After we compute the values f(1) (m)) for any given E [0, 1] the output of the regressor is the value of f()) corresponding to the smallest c(()) that is larger-than or equal to c. This is going to be our estimate for (z). We remark that finding the values r()) can be done efficiently in O(n) time after sorting the data (which has runtime of O(n log n)) so the whole isotonic regression task can be done very efficiently.
Estimating k(z). We now describe our process for estimating the values of axx) and k(x) for every example of dataset B. Similarly to the binary classification setting, we estimate the accuracy probability (ar) using isotonic regression on a small validation dataset. The value of k(x)) can be set to be equal to a fixed value of l for all data, so that the top-k accuracy of the teacher on the validation data reasonable (say above 60%). For example, in our ImageNet experiments, we used k 5. We also provide a data-dependent method to find different values k(() for every example x. To do this we adapt the method for estimating the top-1 accuracy a<x) of the teacher from the validation dataset.
For every value of k = 2.... L- 1 we compute the top-k margin of the teacher's predictions on the validation data which is equal to the sum of the top-k probabilities of the teacher soft-label minus the probability assigned to the i + 1-th class, i.c.
marging(v.(T)) =
max vo(x)) max ye (x).
k+1
14

---

Using the top-k margin as the covariate and the top-k accuracy as the response we solve the cor- responding regression task using isotonic regression to obtain the value ag(x)) representing the probability that the true label belongs in the top-k predictions of the teacher soft-label. For some threshold, say 90%, for every a we set k(x) to the smallest value of so that og (x) 2 90%. We empirically observed that using larger thresholds for the top-k accuracy (e.g., 90% 95%). is better. We remark that while using the top-k margin as the covariate in the regression task is reasonable, our method can be used with other "uncertainty metrics" of the teacher's 's soft-labels, e.g. the entropy of
the distribution of (z) after grouping together the top-k elements. The higher this entropy metric is the more likely that the top-k accuracy probability a(z)) of the teacher is low.
B.2 SLaM for Distillation with Unlabeled Examples: Pseudocode
In this section present pseudo-code describing the distillation with unlabeled examples setting and the SLaM method, Algorithm
Remark B.I. We remark that in experiments, we observed that not normalizing the mixing operation with k(æ) - resulted in better results overall. Therefore, the mixing operation used in our experimental evaluation of SLaM is mix(f w); k()) fx; w) + (1 a<x)))(1 f(; w)) typ (x): k(x)). For more details we refer the reader to the code provided in the supplementary material.
Algorithm 1 Student Label Mixing (SLaM) Distillation Input: Labeled Dataset A, Labeled Validation dataset V, Unlabeled Dataset U Output: A trained Student model f(a; w)
Train Teacher model on Labeled Dataset A Pre-train Student model on Labeled Dataset A
Label examples of Dataset U using the Teacher B + 0
for each x E U do
Add (I, Us (I)) to B # For hard-distillation use y(x) end for
Learn Teacher Accuracy Statistics <<x)), k(()) Algorithm 2 á(r), k(x),((x) t- SaaamnAcrraaSSStttttiisssss// B) Train student {x; w) using the SLaM loss:
Σ
Σ (..N)EB
Ely. mix(f(x;; (): k(x))
(2.))AAVV
C SLaM Consistency
In the following proposition we show that any minimizer of the SLaM loss over the noisy teacher-data must agree with the ground-truth for all (that have positive density). To keep the presentation simple and avoid measurability issues (e.g.. considering measure zero sets under X) the following we will assume that the example distribution X is supported finite set. We remark that one can easily adapt the proof to hold for any distribution X (but the result will hold after excluding measure-zero sets under X).
Proposition C.1 (SLaM Consistency). Let D be the distribution of the teacher-labeled examples of dataset B, i.e., we first draw x X. and then label using the noisy teacher of Definition 3.2 Moreover, assume that there exists some parameter w" E W such that the ground- truth g(x) (x; w²), Denote by SLaM (w) E(g pl(y, mix(f((;; w); (()), the SLaM objective. The following hold true.
1. w" minimizes the SLaM objective.
15

---

Algorithm 2 Estimating Teacher's Accuracy Statistics a(x),,(x)
Input: (Noisy) Teacher Model y (z:), Labeled Validation dataset V. Cootonic-Regreseee lower-boume lb E [o, 1], and top-k accuracy threshold E [0, 1].
Output: Estimates ((z), (xx) of the actual a<x)), k(x).
Create Soft-labels for the Validation dataset using the teacher model (ya(x) : : for j = 1 to L 1 do
# Map ys (x) to top-j margin and accuracy pairs on the Validation V
{{{
C+
Σ
max v.(x)
max (x(),, 1- (x) (1,2))
++1
Set (x) be the output of Isotonic Regression with lower-bound lb on the (covariate, responce) pairs in C.
# See Appendix B.7
end for
á(x) < (2) (x) +
# The top-L. accuracy is always (Irivially) equal to
Given example x for some threshold t set k((z) to be the smallest integer r E (l..... L} so that a, 2t.
2. Assuming further that for all x it holds that a (xx))(())))))) + 1, we have that any minimizer w of the SLaM objective satisfies: fa; w) = g(x) for all x.
Proof. Fix any example x E X. By Definition 3.2 we have that the corresponding teacher label is correct with probability (xx) and a uniformly random incorrect label out of the top-k labels according to the teacher soft-label ys (z). Recall for an L-dimension score vector p. by (pp(p: k) E {0, 1 we denote the vector that has 1 on the positions of the top-k elements of p. e.g., top (pp((1, (1, 3, 4, 5):
(0, O, 0, 1). Conditional on σ, the corresponding expected noisy teacher label is
B æ² = Ply g(z) xlg(() + Ply + g(x) Ely 2, 9(x)) = (x))((())) (1 a(x)) Ely y + g(r),
We know that the expected teacher label conditional on it being wrong Ely * g(x), x] is uniformly random incorrect label from the top-k labels of the corresponding teacher soft-label (z). Assume first that &- = L, since the ground-trun is represented by one- hot vector, the distribution of uniformly random incorrect labels conditional on x can be written (1- (L(()))((L- 1). For example, if the ground-truth label is g(x) = (1, 0, O, O, 0) then uniformly random incorrect label has probability distribution (0. 1/4, 1/ 1/4, 1/4). Assume now that k(x) and top (#): = (1, 1, 1, O, 0). Then the distribution of the (incorrect) teacher label becomes (0, 1/2, 1 /2, 0, 0). Using
to denote clement-wise multiplication of two vectors, we have
1 1 g(x) k(z) 1 top(u<(z):
Ely x. + g(x)]
Therefore, obtain
Ely 2] (x))(()) + + a(x)) g())
g(z) k(x)
to)( mix(
Therefore, by using the fact that Cross-Entropy is linear its first argument, we obtain that the expected SLaM loss some example is
Efce(y,
æ] = ce (Ely 2) mix(f w); (xx) k(z)))
-ce(mi((g (x): (I)), k(r)), mix(f((;; ();
We first have to show that there exist some parameter E W that matches the (expected) ob- served labels Ely xx.. Observe first that by using the realizability assumption, i.e,that there exists w² so that [x;:"*) g(x) we obtain that, for every æ, it holds (())) k(x)) = mix(f(x w (z)) In fact, by Gibb's inequality (convexity of Cross-Entropy) we have that w" is (global) minimizer of the SLaM objective.
16

---

We next show that any (global) minimizer of the SLaM objective must agree with the ground-truth for every I. Since we have shown that w" able match the (expected) labels Ely xj any other minimizer must also satisfy (ix()(1);; k(=)) mix w); a())) k(r))) Assume without loss of generality that 90 1, i.e., the ground-truth label is We observe that by using that mix(((2; w); (x)), k(z)) - (())g(()) + (1 - a(x)) k
1_و( * top(y. (2); k(x)) and the fact that the
ground-truth belongs in the top-k(x:) of the teacher's predictions conditional that the teacher's top-1 prediction is incorrect (thus (us (x)) = 1). we obtain that
((1 (1-k())))))) = a<))
Using the fact that 9o = 1 we can simplify the above expression to
(1 - ((; (a(x))
1- -=x h(x)
Using the assumption that (<)))(x)) # 1 we obtain that the term (a(x) 1-a(x) is not vanishing and therefore it must hold that f(; w) = = 9o- i.e., the student model must be equal to the ground-truth.
D Extended Experimental Evaluation
We implemented all algorithms in Python and used the TensorFlow deep learning library [ We ran our experiments on 64 Cloud TPU v4s each with cores.
D.1 Implementation Details: Vision Datasets
Here present the implementation details for the vision datasets we considered. Remark D.I. We note that in all our experiments, "VID" corresponds to the implementation of the loss described in equation (2), (4) and (6) of [2) (which requires appropriately modifying the student model so that we have access to its embedding layer).
Experiments on CIFAR- 10/100) and CelebA For the experiments on CIFAR- and CelebA we use the Adam optimizer with initial learning rate lr 0.001. We then proceed according to the following learning rate schedule (see, e.g., (5)):
Ir 0.5 10-3
if #epochs 180
Ir 10-3
if #epochs 160
Ir
Ir 10-2
if #epochs 120
10-10-1
if #epochs > 80
Finally, data autaantgmenttiion. In particular,
random horizontal flipping and random width and height translations with width and height factor, respectively, equal to 0.1.
The hyperparameters of each method are optimized as follows. For SLaM we always use 0.5 as the lower bound for isotonic regression (i.e., the parameter Ib in Algorithm 2). As CelebA is binary classification benchmark k(x:) is naturally set 2 for all examples. For CIFAR-10/10 we used the data-dependent method for estimating k(xx) (see Algorithm 2 with threshold pa- rameter /. 0.9. For weighted distillation we do a grid search over updating the weights every (1,25, 25, 50, 100, 200) epochs and we report the best average accuracy achieved. Finally, for VID we search over {0.001, 0.1, 0.2, 0.5, 0.8, 1.0. 2.0, 10.0, 50.0, 100.0) for the coefficient of the VID-related term of the loss function, and for the Taylor cross-entropy method we optimize its hyperparameter over {1.0, 2.0, 3.0, 4.0, 5.0, 6.0}.
Experiments on ImageNet For the ImageNet experiments we use SGD with momentum 0.9 the optimizer. For data-augmentation we use random horizontal flipping and random cropping. Finally, the learning rate schedule is as follows. For the first epochs the learning rate Ir is increased from
17

---

ImageNet: Distillaionn
79 TaylenCC
MDD
65- Welghtte
StaMt rOurs]
60
50-
595
10%
15%
20%
25%
30%
Percentage of Labsled Examples
Figure 4: Comparison of distillation methods on ImageNet. On the horizontal axis plot the size of Dataset as a percentage of the whole training dataset. On the vertical axis we plot the accuracy of the trained student-model on the test dataset.
Table 5: Experiments on CIFAR-10 (hard See Section 4.2 for details.
Tabelol Examples
5000
7500
10000
12500
15000
17500
Teacher
61.30
68.98
72.42
73.92
76.63
78.63
Vanilla
72.09
76.93
77.43
0.15
Taylar-C 21
64..14 #: 0.0T
69.98
+ 0.11
72.72 # 0.36
73.77
+ 0.28
76.26
4 0.29
77.88 + 0.20
UPS [48
64:27 = 0.08
70.93
+ 0.26
73.78
2 0.16
74.06
0.29
77.38
+ 0.37
78.96 + 0.08
0.22
69.39
0.47
74.53
75.55
0.21
Weighted hi
63:22
0.45
71.04 1: 0.26
72.84 1 0.12
74.20
0.16
76.56
4 0.24
78.23
0.15
SLaM (Ouns)
66.40
0.31
72.44 a: 0.17
74.77
1 0.13
75.64
1 0.19
77.99
1: 0.36
79.26
1: 0.26
0.0 to 0.1 linearly. After that, the learning rate changes as follows:
0.01,
if #epochs > 30
Ir
0.001,
if #epochs > 60
0.0001,
if #epochs > 80
The hyperparameters of each method are optimized as follows. For SL.M we hyperparameter search over {0.55, 0.60, 0.65, 0.70} for the lower bound for isotonic regression, and we keep the best performing value for each potential size of dataset A. We used the fixed value 5 for k(x), as the top-5 accuracy of the teacher model was satisfactory (much higher than its top-1 accuracy) on the validation dataset. For Taylor- we did a hhper-yppaneeee parameter search for the Taylor series truncation values in {1, 2, 3, 6, 10, 20, 50, 80, 100}. For weighted distillation compute the weights in one-shot fashion using the pre-traine pre-trained student (as in the ImageNet experiments in (27). For VID we search over (0.1, 0.3, 0.5) for the coefficient of the VID-related term of the loss function.
D.2 Hard-Distillation
Here we present results on hard-distillation. The hyper-parameters of all methods are chosen the same way as in our soft-distillation experiments, see Appendix D.I Tables 5, 6 and 7 contain our results on CIFAR-IO, CIFAR-I00 and CelebA, respectively. We observe that in almost all cases, SLaM consistently outperforms the other baselines. Moreover, for CIFAR- 10 and CIFAR-100 dard-distillation performs worse than ooft-issiilltion (as it is typical the case) but in CelebA hard- distillation seems to be performing on par with (sometimes even outperforming) soft-distillation. A plausible explanation for latter outcome is that our CelebA experiments the teacher and student have different architectures (MobileNet and ResNet, respectively) so that soft-labels from the teacher are not so informative for the student. (This is also binary classification task where the information passed from the teacher to the student through its soft-labels is limited.)
18

---

Table 6: Experiments CIFAR-100 (hard-distillation).. See Section 4.2 for details.
Labelol Examples
5000
7500
10000
12500
15000
17500
Teacher
25.97
44,05
49.62
55.08
39.19
02.05
Vanilla
36.36 = 0.04
44.15 + 0.10
00.22 : 0.07
55.65
= 0.24
58.85
0.1
61.43
0.19
Taylur CE 20
39.12 0.14
46.87
1 0.10
52.64
0:22
57.19
0.28
39.95
1 0.11
62.36
0.21
UPS [8
39.49 + 0.13
48.36
+ 0.44
03.93
+ 0.10
07.56
# 0.10
60.09
+ 0,29
62.09
0.28
VID
37.19 0.09
44.67
1: 0.16
50.63
1: 0.35
54.78
1: 0.07
59.27
1 0.14
62.01
0.05
Weightod
38.04
62.33
57.43
60.81
0.06
SLaM (Ou))
42.01 0.29
49.08
- 0.14
54.49
1 0.17
58.53
1 0.04
61.12 = 0.15
63.21 0.18
Table 7: Experiments on CelebA
See Section for details.
Labeled Examples
2%
3%
4%
5%
7%
Teacher
86.19
58,25
88.95
91.31
92.09
92.62
Vanilla
89.73 #: 0.08
91.61 + 0.09
92.05 + 0.11
93.41 + 0.13
94.02
+ 0.15
94.06
+ 0.04
Taylur-CL 20
90.62
92.19 1:
92.66
0.11
0.14
94,00
0.04
94.38
1 0.10
UPS 4N
0.04
91.30 +
91.95
0.12
93.18
0.07
93.71
+ 0.04
94.18
+ 0.08
VID X
89.92 2: 0.21
91.60 1: 0.11
92.20 1 0.12
93.51
0.15
94.08
1 0.15
94.27
0.110
Weighted bir
90.06 = 0.06
91.97 + 0.13
92.45
+ 0.10
93.60
= 0.07
93.94 + 0.12
94.25
StaM (Ous)
0.05
92.25
92.71
0.008
0:8::000
94.52
CENILE land
MMA200 Hard Clss
Cokak Harz diliiio
15c
15c
&
Comparison of distillation methods on CI0AR-100100 and CelebA. On the horizontal axis we plot the size of Dataset A as a percentage of the whole training dataset. On the vertical axis we plot the accuracy of the trained student-model on the test dataset.
Figure 5:
D.3 Large Movies Reviews Dataset Results
Here we present the results and the implementation details regarding the experiments on the Large Movies Reviews dataset. Recall that we use an ALBERT-large model as teacher, and an ALBERT- base model as student. We also use 2%, 4%, 8%, 40% percent (or 500, 1000, 2000, I 10000 examples) from the training dataset and split the remaining data in validation dataset of 500 examples and an unlabeled dataset U. We compare the methods on the soft-distillation. For each trial we train the student model for 40 epochs and keep the best test accuracy over all epochs. We perform 3 trials and report the average of each method and the variance of the achieved accuracies over the trials. The results of our experiments method for this dataset as the data- augmentattonoo method for estimating the teacher's s accuracy could not be readily used for this NLP dataset. Moreover, using dropout and Monte Carlo estimation for the uncertainty was also not compatible with the Albert model used in this experiment.
be found in Table g We remark that we did not implement the UPS
Since we are dealing with ALBERT-modee (which are already pre-trained), we do not pre-train the " [27], where we pre-train the student model on dataset A just for 1 epoch. The teacher model is trained using the Adam optimizer for 20 epochs with initial learning rate 2 10-5 The student model is trained also using the Adam optimizer but for 40 epochs and with learning rate 10
student model on dataset A except in the case of
The hyperparameters of each method are optimized as follows. For SLaM we do hyperparameter search over (0.5, 0.6, 0.7, 0.8, 0.9} for the lower bound for isotonic regression, and we keep the best performing value for each potential size of dataset A. As this a binary classification benchmark we naturally set k(x)= = 2 for all examples. For weighted distillation we do a grid search over updating the weights every (1, 10, 20, 40) epochs and, similarly, we report the best average accuracy achieved.
Finally, for VID (recall also Remark D.I
we search over {0.1, 0.5, 1.0, 2.0) for the cocfficient of
19

---

Table 8: Experiments the Large Movies Reviews Dataset soft-distillation).. See Section D.3 for details.
Labeled Examples
2%
4%
85%
40%
Teacher
77.52
84.04
85.44
88.3
80,93
TaylorCC
20
79.5 0.38
85.14 4 0.13
85,98
= 0.14
87.57
0.3
VID
81.76 = 0.32
86.33 + 0.35
86.17
0.06
87.71
0.01
Weightel
81.1 -0.1
85.2 = 0.05
56.13
0.28
SLaM
+ 0.28
0.09
86.23
0.13
87.73 0.38
OFAR 100 Composabiiby of Staft with the Fibellty Weighting Scheme
St.M
45
Labecked Dutcoos
Figure 6: Composability the fidelity-based weighting scheme of [17] The æ-axis shows the different values of the fidelity hyper-parameter 3 and the size of dataset A. From left to right we increase the size of dataset A from 10% to 35% and for each size we try different values of B. We observe that SLaM on its own (shown in green) is usually much better than the fidelity weighting scheme (shown in orange). Moreover, using SLaM on top of the fidelity weighting scheme (shown in blue) consistently improves its performance.
the VID-related term of the loss function, and for the PolyLoss we opitmize its hyperparameter over {-1.0, 0.8, -066, -0.4, -.2, 0.0, 2.0).
D.4 Combining with Teacher- Canttter-Unneetait----aa-ae Reweighting Techniques
As discussed in Section 2 method can in principle be combined with leacher-uncertaiin filter- ing and weighting schemes as these can be seen as preprocessing steps. To demonstrate this, we com- bine our method with the so-called fidelity-based weighting scheme of [17]. The fidelity weighting scheme reweights examples using some uncertainty measure for teacher's labels, e.g.. by performing random aata-auumenttttions and estimating the variance of the resulting teacher labels using dropout and Monte Carlo estimation. More precisely, for every example in the teacher-labeled dataset B, the fidelity weigghtn scheme assigns the weight w Fid (x) = cxp(-3 uncertainty teacher (x)) for some hyper-parameter O. In our experiments we performed 10 random data augmentations (random crop and resize), estimated the coordinate-wise variance of the resulting teacher soft-labels, and finally computed the average of the variances of the k-classes, as proposed in [17. We normalized the above uncertainty of each example by the total uncertainty of the teacher over the whole dataset
B. The weights of examples in dataset A are 1 and the reweighted objective is optimized over the combination of the datasets A, B.
fid (w)
1
Σ (E.))E
f(y,f(x;w)) + Σ w Fid(x) Cl,, ((; (x.N)EB
(3)
TAUB
To demonstrate the composability of our method with such uncertainty- based weighting schemes, we use CIFAR1O0 100 and the percentage of the labeled dataset A (as a fraction of the whole training set) 10,, 15%, 20%, 25%, 30%, 35%, similar to the setting of Section 4.2 The teacher ResNetl 10 and the student is a ResNet56. We first train the student using only the fidelity weighting scheme, i.e., optimize the loss function of Equation (4) using different values for the hyperparameter
3 {0.1, 0.2, 1.0, 1.2, 2.0, 5.0, 10.0, 20.0), i.c., ranging from mildly reweighting the examples of
20

---

CIFAR100: Distillaioon with Tempereture Sceling
Stakt + Temperature Scaling
Varilla Tempersture Scaling
60
55
50-
45
10%
15%6
20%
23%
30%
35%
Peicersage of Labeled Examciiss
Figure 7: CIFAR100: Temperature Ablation. On the x-axis we have the size of the labeled dataset (as percentage of the whole training dataset) that the teacher model uses for training.
dataset to more agressively "removing" examples where the teacher's entropy is large. For the same values of 3 we then train the student using the reweighted SLaM objective:
CFid+SLaM (w) = LAUB
1
Σ (N.E))
f(;
Σ
wid Ely, miy.mi((flr a k(x))
(a,y)E
For the combined SLaM + Fidelity method we did not perform hyper-parameter search and used the same parameters for the isotonic regression
did in the "standard" SLaM experiment in CIFARIO0 of Appendix D.I] We present our comprehensive results for all sizes of dataset and values of the hyper-parameter 3 in Figure 6 Our results show that, regardless of the value of the hyperparameter 3 and the size of the labeled dataset A, using SLaM together with the fidelity weighting scheme provides consistent improvements. Moreover, in Figure [6 observe that by using SL.M the achieved accuracy depends less on the hyper-parameter 3: since SLaM takes into account the fact that some of the teacher's predictions are incorrect, it is not crucial down-weight them or filter them out.
D.5 Using Distillation Temperature
In this section we show that our approach can be effectively combined with temperature-scaling [26] Choosing the right distillation temperature often provides significant improvements. In our setting, the teacher provides much more confident predictions (e.g., soft-labels with high-margin) on dataset (where the teacher was trained) compared to the teacher soft-labels of dataset where the teacher is, on average, less confident. Given this observation, it is reasonable to use different distillation temperatures for dataset and dataset B. We try different temperatures for dataset A and dataset B and perform vanilla distillation with temperature and also consider applying the temperature scaling before applying SLaM. For each size of dataset A we try pairs temperatures LA E (0.01, 0.1, 0.5, 0.8. 1., 2., 5., 10., 100.) and report the best accuracy achieved by vanilla distillation and the best achieved by first applying temperature scaling and then SLaM. Figure 7 observe that SLaM with temperature scaling consistently improves vanilla distillation with temperature.
D.6 Using SLaM with other loss functions beyond cross-entropy
In this section, we demonstrate that our method can be successfully applied when the student loss function comes from the families of losses introduced in (20 and [35 We perform experiments on CIFAR- 100 and ImageNet following the setting of Section
In particular, we compare vanilla
distillation with unlabeled examples using the Taylor-CE loss of 20 and the PolyLoss of (35), with combining SLaM with these losses. For the Taylor-CE loss we set the "degree" hyperparameter to be (as suggested (00 and we set the hyperparameter the PolyLoss to 2.0 (as suggested in (35). The corresponding results can be found in Figure
21

---

CHARUO9: thon SLARt With Trylar
CIAKEEE: (Rang SEVV with PSW4005
Pracete: FC S.AM wf MARIS
Ryver
Stat Po
<
1279
220
22%
175
.05 24
Figure 8: Using SLaM with PolyLoss [35 and Taylor CE [20]. On the x-axis we have the size of the labeled dataset (as a percentage the whole training dataset) that the teacher model uses for training. See Appendix D.6 for more details.
D.7 Performance of SLaM with (even) fewer labels
In this section we investigate more extensively the effect of the range of the the size of the labeled- dataset. In particular, we provide experiments with even fewer labeled examples available. Experi- ments in which the number of available labeled examples is small are of greater importance since this is the typical scenario where "distillation with unlabeled examples" applies. For larger dataset sizes, as seen in the plots of the previous experiments, all methods converge to roughly the same performance.
Labelel Examples
CIFARIO,
CIFARIO,
CIFARTOO,
CIFARIOO,
Tencher
10.07
51.67
9.45
23.61
Vanilla
11.75 0.1
54.2 34.2.00011
10.08 2 0.06
25.15 0.111 0.11
JaylarCCl
10.00 0.01
55.14
9.79
20.14
0.39
UFS [1
12.74 = 0.94
56.21
2: 0.25
10:40 1 0.05
26.41 1 0.13
VID
14:25 0.26
644.82
+: 0,06
0.13
24..3 + 0.19
Weighted 27
17.67 0.01
54.38 1: 0.1
10.07
0.06
25.36
1. 0.14
SLaM
26.73 + 0.02
67.40 + 0.06
10.87
4: 0.07
27.70
+ 0,19
Table 9: Comparison of methods on CIFARIO and CIFAR 100 datasets.
D.8 SLaM
Ablation
We investigate the effect of the
used in SLaM, see Appendix D.I
In Table TO we investigate using different values for the isotonic regression lower-bound parameter, (lb in Appendix (.I). We observe that SL.M is rather robust to this hyperpameter and usually simply setting Ib- = 0.5 yields good results.
Labolol Examples
15%
20%
35%
35%
1620
65.80 1: 0.14
71.32
0.11
74.02
E 0.19
76.71 1 0.08
77.36
1 0.12
18.9 1
0.12
1b=0.1
65.7
0.0G
0.17
74.18
0.14
78.07
78.99
[b=0.3
71.94
70.76
0.13
78.19
0.18
79.01
Ib-0.5
66.13 = 0.18
71.96
11.96.0
2 0:21
74.27
4 0.12
76.78
0.00
78.01
4 0.14
78.98
1 0.09
Ib-0.7
64.09 + 0.09
71.19
0.13
73.74
4 0.18
76.38
0:11
78.2
00.2200111
78.71
0.16
1b=0.9
0.14
72.57
0.20
0.17
77.35
0.09
Ib-0.99
62.86
60.84
0.18
72.34
0.36
75.16
0.21
77.37
+ 0.20
78.13
0.13
[b-1
62.78
4: 0.11
69.84
0.09
72.63
42 0.34
76.1
0.19
77.19
4 0.08
78.17 +
0.16
Table 10: CIFAR 10: a(x) ablation results for the lower bound lb of isotonic regression. The value of
lb ranges from 0 to 1.
We next compare using a fixed value for the top-k threshold value, k(x) versus the data-depenent method described in Appendix D.I] We again observe that SLaM is robust to the value of k used since outperforms vanilla distillation for reasonable values of k. Overall, we found that using fixed values of k(z) (after some hyper-paramete search for k) and using the data-dependent method yield comparable results. The advantage of the fixed-value method is that it is easier to implement (and slightly more efficient) and the advantage of the data-dependent method is that its hyperparameter (threshold in Algorithm 2) is easier to tune (in all our experiments 0.9 achieved good performance).
22

---

Labeled Examples
10%
15%
20%
Vamila
37.91
0.10
16.42
0.24
52.17
0.21
41.25
0.25
k-5
40.71
0.29
49.41
0.28
54.41
0.2
k=10
41.2
0.8
49.31
0.12
54.42
2 0.19
Desa-rpemeent k(s) 0.9)
42.7 0.30
49.89
0 0.23
54.
73.73-2
0.27
Table 11: CIFARI00: Fixed-Value vs Data Gata-Dependent k(() Ablation.
D.9 Robustness of SLaM to inaccuracies in k(z),
We perform a "controlled" experiment by adding noise to the estimates of a(x)) and k(z) to test
the robustness of SLaM inaccurate predictions of (xx) and k(z) is a valuable addition to our experimental evaluation. To do this, we start from the oracle values for (x)) and k(z),, i.c., a (z) = 1 the teacher prediction is correct on x and 0 if it is incorrect and k'()) is equal to the smallest integer value l so that the ground-truth label is contained in the top é predictions of the teacher. We then introduce random noise to the oracle predictions: for (x) we perform a random perturbation
a(x) a ax))) (1 2a 2a"())) E.
where Ç is uniformly distributed in (). Hence, when a² (ar) 0 we increase it by adding random we decrease by subtracting the same random variable E E [0, or.. We clip the resulting value in the interval [0, 1J. To create noisy values for k(x) we simply
variable 5 E [0, a) and when (x)
add a random integer to the optimal value k" (z). i... k(x) = (=) + Z, where is a random
integer in {-t....,4f {.. We clip the resulting value of k(x) in the range {0, ..... num mlasssss.).
We test our method on CIFARI00 and observe that the accuracy of SLaM gracefully decays as the predictions for (xx) and k(x:) become worse (bottom right corner is the noisiest o 0.5, l = 90 and top left the noiseless 0, l 0), see Table 12
61.69 + 0.3
39.71
4 0.49
59.7
+ 0.55
39.63
+ 0.44
39.4
0.05
80.04:100
39.55
1 0.33
59.66
0.35
59.79
0.2
60.03
0.441
69.39
0.1
39.23
0.25
59.5
0.39
39.31
0.31
0.5
57.71
0.15
57.6
= 0.25
57.56
0.28
57.46
+ 0.25
57.29 + 0.21
Table 12: CIFAR 100 with 20% labeled data performance with different levels of noise added to the predictions for k(z) and (x))
D.10 Different Regression Algorithms for a(x), k(x)
In our main experimental evaluation of SLaM, we chose isotonic regression enforce the mono- tonicity in the learned accuracy estimates for axx) based on the empirical observation that a(x) often approximately monotone as function of the margin of the teacher. Moreover, the lower threshold (denoted by lb in Appendix D.I) in isotonic regression gives "agressive" the mixing operation is going be. That said, SLaM does not hinge on some particular regression method and other methods can be used. We investigate using different regression methods (Knn, Logistic regression) for estimating (xx),, k(x) with SLaM. Table 13 and Table we see that Isotonic regression typically outperfoms other methods. Moreover, SLaM provides consistent improvements regardless of regression method used.
way to control how
Labeled
20%
30%
KNN 1-
67.1 +
0.15
70.56
+ 0.21
74.6 0.11
76.68
+ 0.17
78 +
0.16
79.26 1 0.12
RNN 1-20
4 0.15
71.09
74.47
0.16
77.03
78.03
0.11
0.18
LNN k=
87.51
0.11
71.27
1 0.13
74.66
03
0.13
0.2 08.07
70.2
0.18
L-NI-40
0.21 67.64
71.08
+ 0.22
74.5
+ 0.09
76.64 + 0.12
77.92
0.11
79.41
+ 0.22
Logistic
0.12
Iontonie 16-85
66.82
1:222:00 0.61
72.61
- 0.30
75.01 - 0.25
75.72
0.16
79.22
0.11
Table 13: CIFAR10: Using different regression methods for estimating k(x)
23

---

Labeled Data
10%
15%
20%
2596
30%
35%
ENN1=J0
30.75
1 0.00
49.07
1 0.15
$4.36
57.87 0.11
61.9
0.2
63.06 1 0.22
X-20
49.19
0.12
54.9
61.76
0.2
63.45
ENNI=10
41.04
1 0.07
49.55
1 0.13
55.14
1 0.15
57 17.96
0.21
61.9
1 0.19
63.2
1 0.23
LNN J-40
41,23
0.03
49.76 +
54.78
0.1
0.21 61.98
68.47
Logistic
39.68
+. 0.03
38.17
+ 0.1
53.56
0.11
57.45
+ 0.09
61.77
0.19
63.24
+ 0.18
Isotonic 16.0.5
42.72
+ 0.30
49.89
+ 0.23
64.73 # 0.27
58.78
+ 0.15
61.30
0.09
63.98
+ 0.19
Table 14: CIFARI00: Using different regression methods for estimating k())
D.11 Validation dataset size ablation
In this section we investigate the effect of the size of the validation dataset required by SLaM, As we have already showed in previous experiments, SLaM requires only rough estimates of a(x) and k(z) and thus even very small validation datasets suffice. We observe that SLaM is able to provide improvements even with very small validation datasets (e.g., with 128 labels).
In Table 15 we use different validation sizes for the CIFAR-100 experiment described in our manuscript and and show that the performance of SLaM improves when the validation dataset is larger but the gaps are not very significant especially for larger sizes of the labeled dataset. We show that SLaM is able to provide improvements even with very small validation datasets (e.g., with 128 labels).
Tabelad Dama
10%
15%
10%
25%
50%
35%
128
40.67 1 0
48.05
1
0.18
54.27
0.21
55.87
81
12
63,65 1 0.21
256
40.97
49
0.25
512
41.06 6: 0.30
49
0.19
54.36
0.12
58.57
0.26
85
0.31
63.38
0.11
1024
41.83 = 0.32
49.35
+ 0.25
54.71
0.18
68.90
0.32
61.28
+ 0.46
63.62
+ 0.28
Table
15: CIFAR-100 validation dataset size ablation results.
E Distilling Linear Models and Learning Noisy Halfspaces
In this section we state and prove our convergence result for the SLaM method when applied to linear models. Our assumption is that the ground-truth g(x) corresponds to a halfspace, i.e., g(z) = (1(w" x > 0}. 1(w" x S 0}} for some unknown weight vector w". We show that using SLaM with a linear model as the student will recover the ground truth classifier. We make the standard assumption that the ground-truth halfspace has 7-margin, i.e., that ||w"|
and that it holds |wo x| 2 7 for all examples x. For fixed example x, the observed noisy teacher-label y satisfies Definition 3.2 i.e., y = g(x) w.p. a(x) and y
g(z) w.p. - a(z) (since k = 2 for binary classification). Our approach consists of using the standard cross-entropy loss ce(p, q) and training a moudent-mode consisting of linear layer plus soft-max activation, i.c.,
fx; w) = (fo (2; w), fi (x) lit--ar w. Ite
Recall, that for binary classification, we define the mixing operation as
(xx = a()) +1 (0)))a)))1-))0)) - f(x;
Theorem E.1 (Student Label Mixing Convergence). Let X be a distribution on Rd and g(x)) be the ground-truth halfspace with normal vector w" Rd. Let D be the distribution over (noisy) teacher-labeled examples (x, y) whose x-marginal is X. We denote by a(x) the probability that the teacher label y [0, 1|2 is correct, ie., axx P (aa))= (aran-/(agmmx(())
g(x) x). Assume that there exist B, 7 > 0 such that for all examples x in the support of X it holds that xx) y and |1/2- - a(>) < 3. Let e > O. After T = ((1 ()) iterations of SLaM (Algorithm 3) with probability at least 99%, there exists an iteration t < T where P P (x;;),()) g(())))()))))))))))))))))))))))) S.
Remark E.2 -High-Probability Result). We remark that constant probability (at least %99) we can amplify its success probability to 1 - 8 by standard amplification techniques (i.e., by repeating the algorithm (loo(11/0) times and keeping the best result). To achieve success probability 5 the total sample complexity is ((1og(1/5)/(22)
though our learner succeeds with
Proof. We first provide simplified expressions for the gradient of the SLaM objective and the update vectors (0) (t)) used in Algorithm 3 In what follows we remark that for any binary classification
24

---

Algorithm 3 SLaM for Linear Models
Initialiaze weight vector of student w.(0) 0
for I. = 1,....7 do Draw example x(() ~ X.
Label z(() with (noisy) teacher to obtain y()
Compute the gradient of the SLaM loss at (x(0) y()):
gl + Co(ol) mf w w ( (==w((-))
Compute step size: A 1/r(f(x(0) w (oll-1)), a(x(())) (see Lemma E.3 for the definition of r)::))) Update the student model: w(t) t- w(t-1) 100) g(() end for
model fa; w) = (fo (z; w). (a;; w)) we have the following identities: (i) (mix w): a()))) mix(fo(x; w); (x)), where to simplify notation we overload the mixing operation to also act the scalar fo(2; w), i.e. mix(So(x; w); ((()so(x;;)))) w) + (1 - axx)) fo(æ; w)); and (ii)
fi w) = 1 fo(r;
Lemma E.3 (SLaM Gradient). The gradient of the SLaM objective is equal to
Cy mxx = r(fo(z;; where
w); a); sgn(2a(x) - 1) (fmix(fo(; - y,)
(1-;:))1-f((am)))
r((();));(=)))
[2x(x)
Let L(r;w) y col(,, mixm w) a(x)) be be the expected student label mixing loss con- ditional on some example E R.d. holds 0, Mxx; w) r(f(x;)) a(x)) [2a(x) 1| (fo(2; w) (o())) x.
Proof. We first show the formula
cely mix(f(x; w) axx)) = r(fo(; sgn2a((() 1) ((mxx(f((;; y0)xr.
(5)
Using the chain rule, we obtain
a (f)) =
mnix(Jo(x;;) w) (zz)) aw (mix(((((5
yo
mix (z; w), a ((z)) Ow (mxx
y1
Now we observe that that for binary classification, it holds that y1 = - mix mix a(x)) =
1 mix(fo(x;; w); (x))) and therefore, also mix( (z; w); (x)))) mix a(r))))
to obtain the simplified expression:
ceyy (x; ();
(xx arm (mix(f((;;;))) (x)
yo
1 y0
mix(fo((x w) a(x)) On ((())
Further simplifying the above expression, obtain:
in.cely (ix
fo a()) yo
mix(Jo(r;w), a (x) (1 mix(So(;; a())) &n (mix(fo(;;;
25

---

Using again the chain rule we obtain that
Ore (mixx(f((:;; w); a(x))
Using the fact that the derivative of the sigmoid function r(t)= -
-- = (1 r(t)), and the chain rule, we obtain that
-1
Putting everything together we obtain the claimed formula for ce mix( w): az)))
To obtain the gradient formula for the expected loss conditional on some fixed example x, we can use the fact that an Efce(y, mfx(f(ar w); (())) x:] ceoy mix(f (x; w); a(x))) z]. Now using the formula of Equation (5) and the fact that Elyo xe] mix(go(x); (xx)) by the definition of our noise model, we obtain that
Ox: w)
r ffo( ()) 1)(mix(f6(1; u)) mix((o()):a())))
r
- 1)(f((;;)) go(2))
We first show the following claim proving that after roughly T = 1/(32y222) gradient iterations the student parameter vector w(t) will have good correlation with the ground-truti vector w". Claim 1.
Fix any T larger than a sufficiently large constant multiple of log(1 (og()/))/((²))))) and assume that for all t < T it holds that P P (or((((;;))) w(()), g(x)) > Then, we have (T) w" (Bxx) T, with probability at least 5.
Proof. Denote by (t = -x(0) the update vector used in Algorithm We observe that the weight vector at round T is equal to w(() = Ei= (() In what follows we denote by F(() the filtration corresponding to the randomness of the updates of Algorithm [3 We define the martingale
((T) = (E [[(-1)] (())) with (0)) = 0. We first show that under the assumption that P far/margaax((f(. w()))) # g(x)] e, for all t = T, it holds that Σin E(u(() [F(+))) w" 2 (ery3/2) T. Using the SLaM gradient expression of Lemma E.3 and the definition of the step size (
we obtain that ( Fit-1)]- Ex [[a((x) 1| (90(x) fo(æ; (((-1))) z]. Take any step t.
We have that
(t)( [((-11]- w" Ex~x E 11 (go(x) folz; (((-l1)) (x ww'l
Ex ((2((x) - 1/ [90(z)) fo(æ; (1(x::((()))) (1) |r w"ll.
where we used the fact that (g0(z) - fo(:; ((t-)))) sgn(x w²) = 190(x)) fo(æ; w(t-l))] Now, using the "y-margin assumption of the distribution D and the fact that (2a(x) - 1 2 B we obtain
E(((() | w0
2 By E (190())) - So w (c=)))
2 By E - fo(æ; w ((-11) err(((x)) f(x; w D))]
2 (Bx/2) Pawx (err(g(x), f(;; w ((-1))) 2 Byc/2,
where for the penultimate inequality we used the fact that when g(ø) and f(; ((l-1)) disagree holds that [go(=) - fo(æ; (-(- 2 1/2. Take, for example, the case where go(x) = 1. Then fo(a; (tt-))) must be smaller than 1/2 otherwise the prediction of the model argmax f faz; (t-1)) would also be 0 (and would agree with the prediction of g(x)). Finally, for the last inequality we used the fact that, by our assumption, it holds that P P (err(g(()))) [x; (t-))) 2 Therefore, we conclude that DT= ((u(() _ = > (ey3/2) T. Next, we shall show that (T) also achieves good correlation with the optimal direction w" with high probability. We will use the fact that g()) is martingale and the Azuma-Houu Hoeffiding incquality to show that (T) w" will not be very far from its expectation.
Lemma E.4 (Azuma-Hoeffding Let g(t) be a martingale with bounded increments, i.e., (5(0) <(t-1)| < M. It holds that PT (()) + >] (2MMP)
26

---

Recall that from Lemma fo(2; w() æ] and
E.3 we have that Efu(*) | F(111) = [x>x(12a(xx) - 11 (go(z)
sgn (2a(x(0) 1) (v6 - fo (ll a(x(())) x01
Observe that since ||a]|/ I 1 for all x it holds that ||x(0) 2|2 S 1. Therefore, the difference || B(u()) [[-11]]
1 < 2 with probability 1. Since | = 1, using Cauchy-Schward we also obtain
that E(u(9) wt [(-1)] u() w01 S 2.
Using Lemma E.4 and the fact that g0) = 0 we obtain that P(g(0) =² 2 (Bye/4) T) s
2 Therefore we conclude that for any T larger than 128 (5/(11///822/ with probability at least 1 - 5, it holds that g( -= 2 (3ye/4)T or equivalently w(T) -= 2 (8ye/4) T, where we used our previously obtained bound for the expected updates > E(u(0) F(l-11]. 2 (Bye/2) T.
Claim 2. Fix any T 2 1. Then, we have | O(VT), with probability at least 99%.
Proof. We have that || [1 = we obtain that
+ 2u(T) -. + (T 12. Unrolling the iteration,
T ( (0(t-1) Σ 10 < Σ ( w (t-1) + TT,
( = 3
(6)
1=1
where we used the fact that, since ||o((0/12 112 & 1, it holds that |/u S 1 (see the proof of Claim
We first show that Ei= E(u(() F((-1)] (t-11) 1) = O(T). We have
(1)( [=(112)] w(t-1) Exax (22((x) 11 (90(2x) fo(æ; ())--11) (x (1111))) E Ex~xx folæ; (((-1))) (o . w((t1))].
We will show that for æ it holds that
go(x) - f(;; w (t (x) (1( S -
Fix some æ and let = w(t 1) Assume first that go(x) = L. Then, we have
go(x)) [xx; w (t- (x=))))) (t( (1
1 T+e-s)
e ite-
1
where we used the fact that 5 --e-- S 0 for s < and Ite- -* S 1/e for s 2 0 (using the
elementary inequality ze_ S 1/e for all 2 E R). When go(x)) we similarly have that
(x(x)= - f(x; (t (x)x))))(x (t- 1
where we used the fact that when s 2 0 it holds that "Ite-x S and when 8 S 0, -+e-- S -s/e-== = -sc.. For the final inequality, we used again the inequality =0= < 1/e for all 2 E R (where we replaced z with -s).
Therefore, we obtain that E (1)( | ([(1)) (t-1) 1) < and EL E(u(() F((-1) (o(1-)) < <T/e. Using the decomposition of Equation (6) linearity of expectation, and the tower rule for conditional
expectations, we conclude that B[] < (2/e+ + 1)T. Using Markov's inequality we obtain that
with probability at least 99% it holds that [Txx = O(T) or equivalently [T = O(VT).
We can now finish the proof of Theorem 5.1
Assume, in order to reach a contradiction, that
for all < 7 it holds that X (err ([x [(r))] > 6. Now picking to be larger than sufficiently large constant multiple of 1/(e 2 82) and using Claim I and Claim 2 we obtain that,
27

---

Figure 9: The landscape and gradient field of the population student label mixing loss for a simple dimensional feature problem with ground truth corresponding to halfspace. We observe that the landscape is non-convex; however the optimal direction" and therefore gradient descent converges to the global minimizer. A potential issue is the fact that the landscape contains regions where the gradients may almost vanish and this could lead to the gradient iteration of the student getting trapped there. To handle this issue, in Algorithm we multiply the gradient of SLaM with appropriate step-size.
that the corresponding gradient field "points towards
with probability at least 99%, holds that w(T) w² /f 2 ((BrrevT)) which can be made to be larger than by our choice of T. However, this is contradiction by Cauchy-Schwarz we have w w0 /l] < ||"" la < 1. Therefore, with probability at least 99%, it must be that for some t s T it holds that P P (fr w()) )xx))) <.
28


Reviews:
[{"review_id": "KTBc1TfZ8d", "reviewer": "Reviewer_E5Wk", "summary": "The paper attempts to resolve the problem of noisy teacher labels for unlabeled examples in knowledge distillation. Proposed Student-Label Mixing (SLaM) to introduce noise into student prediction when matching to teacher’s  pseudo-label on unlabeled data. Experiments on standard benchmarks were used to demonstrate the effectiveness of the proposed method. Also, the paper provided theoretical guarantees for SLaM for learning noisy halfspaces.\n\nrating updated after author rebuttal. ", "questions": "The proposed algorithm has the additional cost of needing a validation dataset, seems a fair baseline would be to incorporate the validation set for vanilla distillation, and on top that add vanilla unlabeled distillation or other state of the art unlabeled distillation? \n\nWhat is the intuition of the second term on formula (1)?\n\nHow important is the design \\alpha(x) and k(x) is? How does the size of the validation set affect the overall performance? \n", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper studied the critical problem about how to learn unlabeled examples for the student model with knowledge distillation when the teacher model is not perfect.\n\nThe paper proposed to resolve it from a novel angle by introducing noise to the student model’s prediction, and also provided a theoretical guarantee for the proposed solution.\n\nExtensive experiments based on common datasets and multiple baselines were provided to demonstrate its effectiveness.\n", "weaknesses": "The paper lacks the explanation on the intuition of the mixing solution on formula (1), where there is no clarification why it is essential for the second term, i.e. the element-wise multiplication term.\n\nEven though the paper developed a solution to consider the noise for student model prediction, it seems not robust enough to learn \\alpha(x) and k(x). As it requires an additional validation dataset, it is easy to make \\alpha overfit to those data. Also it has another strong assumption that \\alpha is ​​isotonic regression. All of these make the learning of \\alpha easily overfit the data which cannot be robust enough, which also make the solution not that elegant.\n\nIt lacks the result details about \\alpha and k setup for each experiment. Also it lacks the related ablation study to demonstrate how \\alpha and k affect the model performance.\n\nno comparison with a baseline where the validation dataset is used for vanilla distillation, and on top that add vanilla unlabeled distillation or other state of the art unlabeled distillation. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The paper lacks the explanation on the intuition of the mixing solution on formula (1), w ... *What is the intuition of the second term on formula (1)?\n\n**Short Answer** \n* The high-level intuition of equation (1) appears in Lines 212-226 of the manuscript.\n* In Appendix C of the supplementary material we have provided a **formal proof** that the minimizer of the SLaM objective using equation (1) is the ground-truth labeling (given noisy teacher predictions).\n* The second term is needed to ensure that the minimizer of the SLaM objective is the ground-truth labeling, see Appendix C. The expected noisy teacher label is equal to $\\alpha(x) g(x) + (1-\\alpha(x)) (1-g(x))$ (where g(x) is the ground-truth label of x).The noisy student is equal to $\\alpha(x) f(x) + (1-\\alpha(x)) (1-f(x))$. When the noisy teacher matches the noisy (mixed) student it must be that $f(x) = g(x)$ (the student matches the ground-truth). See  Appendix C for more details.\n\n> Even though ... not robust enough to learn $\\alpha(x)$ and $k(x)$. A... it is easy to make $\\alpha$ overfit to those data.  -- How important is the design $\\alpha(x)$ and $k(x)$ is?\n\n* As we mention in our manuscript, the parameter $\\alpha(x)$ that we estimate is only a function of the teacher's margin so it can be learned from very few data via a **simple one dimensional** regression task. \n* SLaM is robust to noise in the estimates of $\\alpha(x)$ and $k(x)$; only rough estimates are needed for SLaM to provide improvements. In fact, in many cases, **even using a single value for $k(x)$ for all examples (say k(x)=5) is enough for SLaM to achieve good performance** (see our Imagenet experiments in Table 4 and also our response below). We also test the robustness of SLaM to noise in the predictions of $\\alpha(x)$ and $k(x)$ in the ablation provided to Reviewer PQzZ.\n\n>Also it has another strong assumption that \\alpha is isotonic regression.\n\n**Using isotonic regression is not an assumption about $\\alpha(x)$**: our regression algorithm works regardless of whether $\\alpha(x)$ is monotone as a function of the margin.  \n As we mention in Figure 1 and Lines 564-571, using isotonic regression is just a way to enforce the monotonicity of the resulting estimator — a property we have empirically observed that enhances the robustness of SLaM and leads to more stable estimates of $\\alpha(x)$ (*harder to overfit*) -- see also our general response and the response to Reviewer PQzZ.\n\n>It lacks the result details about \\alpha and k setup for each experiment. Also it lacks the related ablation study ....\n\n**Short Answer** Due to space limitations, the details for $\\alpha(x)$ and $k(x)$ were included in section D of the supplementary material.  See Lines 677-680 for Celeb-A, Cifar10/100, Lines 689-693 for Imagenet, Lines 726-727 for Large Movies Reviews Dataset.\n\n**More Details** \n\nBinary Classification. Our method only requires tuning a single hyperparameter: the accuracy lower bound for a(x) in isotonic regression, see the value lb in the first paragraph in Section B.1. \n\nThe effect of the parameter lb is clear from the mixing operation (see the mixing operation above Equation 2 in Section 3). As the accuracy lower bound (lb) converges to 1, SLaM converges to “vanilla” distillation as we do not mix the student’s predictions. When the accuracy lower bound is closer to 0, SLaM more aggressively mixes student’s predictions. \n\nMulticlass Classification. For the isotonic regression lower bound in CIFAR-10 and CIFAR100, we used lb=0.5. We found out that SLaM is robust in the setting of lb in those datasets and the values {0.5,0.6,0.7} usually yield good performance. See ablation in Table 2 of the pdf.\n\nWe use two methods to obtain k(x).\n\nThe first one uses the same value k(x) = k for all examples. We found that it suffices to achieve good performance gains. We used this method in our ImageNet experiments and used the value k=5 as the top-5 accuracy of the teacher model was satisfactory (much higher than its top-1 accuracy) on the validation dataset. \n\nThe second way uses a (data-dependent) value of k(x) for each example (see Sections B.1. and B.2.). This requires tuning a hyperparameter t (see the top-k accuracy threshold in Algorithm 2). We used this method in CIFAR10/100 and used t=0.9. \n\nIn Table 3 of the rebuttal pdf, we present our results for CIFAR100 using a fixed value for k(x). We observe that SLaM is rather robust to the value of k used since it outperforms vanilla distillation for some reasonable values of k. Overall, we found that using fixed values of $k(x)$ (after some hyper-parameter search for $k$) and using the data-dependent method yield comparable results. The advantage of the fixed-value method is that it is easier to implement (and slightly more efficient) and the advantage of the data-dependent method is that its hyperparameter (threshold $t$ in Algorithm 2) is easier to tune (in all our experiments $t=0.9$ achieved good performance).\n\n>no comparison with a baseline where the validation dataset is used ... incorporate the validation set for vanilla distillation ...?\n\nΑs we mention in Lines 274-278 of the main body of the paper, to be fair to methods not using validation data, we have included the validation data in the training dataset of all methods compared.\n\n>How does the size of the validation set affect the overall performance?\n\nAs we already discussed SLaM requires only rough estimates of $\\alpha(x)$ and $k(x)$ and thus even very small validation datasets suffice.  In Table 1 of the rebuttal pdf we use different validation sizes for the CIFAR-100 experiment described in our manuscript and and show that the performance of SLaM improves when the validation dataset is larger but the gaps are not very significant especially for larger sizes of the labeled dataset. We show that SLaM is able to provide improvements even with very small validation datasets (e.g., with 128 labels).  See also Item 2 of Additional Experiments in the general response.\n"}, {"review_id": "wULCrV142Y", "reviewer": "Reviewer_PQzZ", "summary": "In their paper, the authors propose a knowledge distillation procedure that augments pseudo-labels predicted by a teacher models by its uncertainty, leading to a uncertainty-aware student learning routine. This is realized by a novel loss formulation that mixes student predictions with quantified uncertainty associated with the pseudo-labeling prediction. Together with convincing empirical results demonstrating improvements in terms of generalization performance, theoretical results characterize the convergence of the proposed solution.", "questions": "-", "limitations": "I do not see too much about limitations, this could certainly be deepened.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Strong empirical results against recent baselines\n- Profound theoretical results\n- Well-written manuscript with a clear and valid motivation, addressing a relevant problem\n- The simplicity of the approach is appealing, promoting the applicability of this method\n", "weaknesses": "Major:\n\n- While the empirical evaluation itself is certainly convincing from a generalization point of view, a more rigorous ablation study of the influence of the uncertainty quantification would be insightful. Namely, it is hard to judge how sensible this procedure is towards a good quality of the isotonic regression solution. One possibility to address this concern could be by obtaining $\\alpha(x)$ and $k(x)$ from an oracle with “perfect” estimations, and gradually contaminating these estimates. Then, one could see how the performance evolves.\n- Relating to the previous comment, I am wondering why exactly isotonic regression is chosen as a means to quantify $\\alpha(x)$. I see little explanations on why exactly this method is chosen, and not others, like conformal prediction. Of course, it is completely valid to make an arbitrary design choice that works well, but I would love to see a more thorough elaboration on the motivation of this specific construction. An ablation study as described before could be used to support this decision.\n\nMinor:\n\n- The figures are relatively small and hard to read, enlarging them would be appreciated\n- The orthography could be improved, many missing commas (e.g., when starting sentences like “In Section xyz, …”)\n- The references are inconsistent (e.g., the conference identifiers: NeurIPS, ICML, … vs. International Conference on Learning Representations)\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n\n>.. a more rigorous ablation study of the influence of the uncertainty quantification would be insightful. ... obtaining $\\alpha(x)$ and $k(x)$ from an oracle with “perfect” estimations, and gradually contaminating these estimates.\n\nWe agree with the reviewer that a \"controlled\" experiment by adding noise to test the robustness of SLaM to inaccurate predictions of $\\alpha(x)$ and $k(x)$ is a valuable addition to our experimental evaluation.  To do this, as the reviewer suggested, we start from the oracle values for $\\alpha(x)$ and $k(x)$, i.e., $\\alpha^*(x) = 1$ if the teacher prediction is correct on $x$ and $0$ if it is incorrect and $k^*(x)$ is equal to the smallest integer value $\\ell$ so that the ground-truth label is contained in the top $\\ell$ predictions of the teacher. We then introduce random noise to the oracle predictions: for $\\alpha(x)$ we perform a random perturbation $$\\alpha(x) = \\alpha^*(x) + (1-2\\alpha^*(x)) ~ \\xi, $$ where $\\xi$ is uniformly distributed in $[0,\\sigma]$. Hence, when $\\alpha^*(x) = 0$ we increase it by adding a random variable $\\xi \\in [0,\\sigma]$ and when $\\alpha^*(x) = 1$ we decrease it by subtracting the same random variable $\\xi \\in [0,\\sigma]$. We clip the resulting value in the interval $[0,1]$.\nTo create noisy values for $k(x)$ we simply add a random integer to the optimal value $k^*(x)$, i.e., $k(x) = k^*(x) + Z$, where $Z$ is a random integer in $\\{-\\ell,\\ldots,\\ell\\}$. We clip the resulting value of $k(x)$ in the range $\\{0,\\ldots, numClasses\\}$.\n\nThe table with our results on CIFAR100 can be found in our general response (Item 2 of Additional Experiments).\n\n>... why exactly isotonic regression is chosen ...  An ablation study as described before could be used to support this decision.\n\nWe chose isotonic regression to enforce the monotonicity in the learned accuracy estimates for $\\alpha(x)$ based on the empirical observation that $\\alpha(x)$ is often approximately monotone as a function of the margin of the teacher. Moreover, the lower threshold (denoted by lb in Section B.1. in the Supplementary Material) in isotonic regression gives us a way to control how \"agressive\" the mixing operation is going to be.    We agree with the reviewer that SLaM does not hinge on some particular regression method and other methods can be used.\nWe used SLaM with the k-nearest neighbors regression and with logistic regression and found that isotonic regression typically outperfoms logistic regression and its hyper-parameter is easier to tune than the number of neighbors in kNN.  That said, SLaM is agnostic to the methods for estimating $\\alpha(x), k(x)$ and more sophisticated methods (such as conformal prediction) could also be used.  We will include this discussion and ablations in our updated manuscript.\n\n| CIFAR10              | 10%           | 15%           | 20%           | 25%           | 30%           | 35%           |\n|-|-|-|-|-|-|-|\n|kNN  k=10  | $67.1 \\pm 0.15$ | $70.56 \\pm 0.21$ | $74.6 \\pm 0.11$ | $76.68 \\pm 0.17$ | $78 \\pm 0.16$    | $79.26 \\pm 0.12$ |\n| kNN  k=20  | $67.5 \\pm 0.15$ | $71.09 \\pm 0.19$ | $74.47 \\pm 0.15$ | $77.03 \\pm 0.1$  | $78.03 \\pm 0.17$ | $79.21 \\pm 0.13$ |\n|kNN  k=30  | $67.51 \\pm 0.11$| $71.27 \\pm 0.13$| $74.66 \\pm 0.12$| $77.03 \\pm 0.13$| $78.07 \\pm 0.2$ | $79.2 \\pm 0.18$  |\n|kNN  k=40  | $67.64 \\pm 0.21$| $71.08 \\pm 0.22$| $74.5 \\pm 0.09$ | $76.64 \\pm 0.12$| $77.92 \\pm 0.11$| $79.41 \\pm 0.22$ |\n|Logistic | $65.26 \\pm 0.05$ | $68.85 \\pm 0.08$ | $73.35 \\pm 0.12$ | $76.17 \\pm 0.15$ | $76.87 \\pm 0.25$ | $78.76 \\pm 0.35$|\n|Isotonic lb=0.5| $66.82 \\pm 0.61$ | $72.61 \\pm 0.30$ | $75.01 \\pm 0.25$ | $75.72 \\pm 0.17$ | $78.04 \\pm 0.16$ | $79.22 \\pm 0.11$|\n\n| CIFAR100 |  10%    |  15%    |  20%    |  25%    |  30%    |  35%    |\n|-|-|-|-|-|-|-|\n| kNN k=10  | $40.75 \\pm 0.02$ | $49.07 \\pm 0.15$ | $54.86 \\pm 0.11$ | $57.87 \\pm 0.17$ | $61.9 \\pm 0.2$  | $63.06 \\pm 0.22$ |\n| kNN k=20  | $41.03 \\pm 0.05$ | $49.19 \\pm 0.12$ | $54.9 \\pm 0.1$   | $57.85 \\pm 0.18$ | $61.76 \\pm 0.2$  | $63.45 \\pm 0.16$ |\n| kNN k=30  | $41.04 \\pm 0.07$ | $49.55 \\pm 0.13$ | $55.14 \\pm 0.15$ | $57.96 \\pm 0.21$ | $61.9 \\pm 0.19$  | $63.2 \\pm 0.23$  |\n| kNN k=40  | $41.23 \\pm 0.03$ | $49.76 \\pm 0.15$ | $54.78 \\pm 0.1$  | $58.15 \\pm 0.17$ | $61.98 \\pm 0.21$ | $63.47 \\pm 0.2$  |\n| Logistic | $39.68 \\pm 0.03$ | $48.17 \\pm 0.1$  | $53.56 \\pm 0.11$ | $57.45 \\pm 0.09$ | $61.77 \\pm 0.19$ | $63.24 \\pm 0.18$ |\n|Isotonic lb=0.5 | $42.72 \\pm 0.30$  |  $49.89 \\pm 0.23$ | $54.73 \\pm 0.27$ | $58.78 \\pm 0.15$ | $61.30 \\pm 0.09$ | $63.98 \\pm 0.19$|\n\n>.. Minor comments regarding presentation\n\nWe will implement all the reviewer's suggestions regarding the presentation of our paper."}, {"review_id": "BfCYEiwVy9", "reviewer": "Reviewer_Un32", "summary": "The paper introduces SLaM, a method for knowledge distillation using unlabeled examples. Its main contribution lies in the development of an efficient and data-agnostic approach that enhances student model performance in scenarios with limited labeled data but abundant unlabeled data. SLaM accomplishes this by incorporating information from both labeled and unlabeled examples during the distillation process. The paper extensively presents experimental results showcasing SLaM's superiority over previous methods on standard benchmarks. Furthermore, the authors provide theoretical guarantees and insights into SLaM's effectiveness. Overall, the paper delivers a valuable contribution to the field of knowledge distillation, showcasing a novel approach supported by empirical evidence.", "questions": "1. Can the proposed method ensure that the pseudo-labels generated by SLaM are noise-free, thereby addressing the limitation discussed in the paper? How and why?\n\n2. Although not specifically designed for knowledge distillation, the baseline method Taylor-CE shows competitive or even superior performance on celebA even with only 2% labeled data. Can you elaborate on the reasoning behind this, especially highlighting the advantages of the proposed method in dealing with sparsely labeled data, which Taylor-CE lacks?\n\n3. It is better to plot a figure to demonstrate the performance of the comparison method in different percents of the labeled training data with a large span. Eg: 5%, 10%, 30%, 50%, 70%.\n\n4. The baseline method Taylor-CE, which focuses on noise label problem, also achieve competitive or even superior performance. I wonder if the proposed method can work in noise label learning that also need reliable pseudo-labels? For example, on clothing1m dataset.", "limitations": "Please refer to weakness and question.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel approach to knowledge distillation with unlabeled examples that leverages the information from both labeled and unlabeled examples during the distillation process.\n\n2. The experimental results show the effectiveness of the proposed method in knowledge distillation.\n\n3. Theoretical results further guarantee the effectiveness of the proposed method.\n", "weaknesses": "The proposed method argues that existing methods suffer from the shortcoming of noisy pseudo-label, but it also seems hard to guarantee the proposed method can get the pseudo-labels without noisy. Moreover, the proposed method can be limited to semi-supervised learning problem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n>... but it also seems hard to guarantee the proposed method can get the pseudo-labels without noisy. Can the proposed method ensure that the pseudo-labels generated by SLaM are noise-free, thereby addressing the limitation discussed in the paper? How and why?*\n\n SLaM effectively improves the quality (reduces the noise) of the pseudo-labels produced by the student. We remark that we \"add\" noise in the output of the student **only during training** -- during testing/inference no noise is added to the pseudo-labels of the student. While we cannot guarantee that the pseudo-labels generated by a student model trained with SLaM will be perfect (and no other method can guarantee that) we argue that student models trained with SLaM produce better predictions than the available baselines. To show that SLaM effectively reduces the noise of the produced pseudo-labels we perform a self-distillation experiment. We first train a ResNet56 (teacher) on the small labeled Dataset A (its size is shown in the first row of the table below) and then use SLaM to train another ResNet56.  Observe that SLaM produces a student ResNet56 that is more accurate than the teacher ResNet56 for all dataset sizes. Therefore, in this self-distillation experiment the pseudo-labels produced by the student trained with SLaM are **less noisy** than the labels provided initially by the teacher. \n\n\n| CIFAR100 (Resnet 56->56) |   1%    |   5%   |   10%   |   15%   |  20%   |  25%   |  30%   |  35%   |  50%   |  60%    |\n|-|-|-|-|-|-|-|-|-|-|-|\n| Teacher                  |   8.78   |   21.06  |   34.15  |   42.72  |   47.9   |   53.47  |   56.41  |   58.73  |   64.4   |   66.35  |   69.18  |\n| Vanilla                  | $9.36 \\pm 0.03$  | $22.42 \\pm 0.11$ | $36.34 \\pm 0.30$ | $44.93 \\pm 0.34$ | $50.74 \\pm 0.36$ | $55.8 \\pm 0.11$  | $58.72 \\pm 0.33$ | $61.24 \\pm 0.15$ | $66.43 \\pm 0.04$ | $68.4 \\pm 0.02$  |\n| Taylor-CE                   | $9.36 \\pm 0.05$  | $23.58 \\pm 0.15$ | $38.41 \\pm 0.19$ | $46.45 \\pm 0.09$ | $52.53 \\pm 0.11$ | $56.67 \\pm 0.31$ | $59.79 \\pm 0.23$ | $61.98 \\pm 0.31$ | $66.53 \\pm 0.35$ | 68.43+/-0.30 |\n| SLaM            | $\\mathbf{10.52 \\pm 0.4}$ | $\\mathbf{25.66 \\pm 0.2}$ | $\\mathbf{39.72 \\pm 0.3}$ | $\\mathbf{48.46 \\pm 0.4}$ | $\\mathbf{53.53 \\pm 0.15}$ | $\\mathbf{57.38 \\pm 0.31}$ | $\\mathbf{60.64 \\pm 0.13}$ | $\\mathbf{62.27 \\pm 0.32}$ | $\\mathbf{66.60 \\pm 0.15}$ | $\\mathbf{68.47 \\pm 0.24}$  |\n\n\n>... Taylor-CE shows competitive or even superior performance on celebA even with only 2% labeled data. Can you elaborate on the reasoning behind this, especially highlighting the advantages of the proposed method ... ?*\n\n* Taylor-CE outperforms SLaM only for the 2\\% case of the Celeb-A dataset.  In contrast, SLaM outperforms Taylor-CE in all other experiments and in some cases by a large margin (see, e.g., the 5\\% case of **Imagenet at Table 4 where SLaM outperforms Taylor-CE by ~6.5\\%** and the CIFAR100 ablation of the previous answer).\n    \n    \n* SLaM almost always outperforms Taylor-CE because the noise in the teacher's labels is structured (not random nor adversarial) and SLaM exploits this structure while Taylor-CE does not (was designed for generic noisy labels).\n    \n* SLaM can be combined with different loss functions (we used the cross-entropy in our experiments because it is the most commonly used one).  In fact, in Section D.6 and Figure 7 of the supplementary material we show that SLaM can be used with the Taylor-CE loss function and improve its performance.\n\n\n> It is better to plot a figure to demonstrate the performance of the comparison method in different percents of the labeled training data with a large span. Eg: 5%, 10%, 30%, 50%, 70%.\n\n* We included smaller labelled training data (1%, 5%) for CIFAR10/100.  We observe that SLaM consistently outperfoms the baselines. For CIFAR10 with 1% labelled datas SLaM achieved an improvement of over 10% over other baselines. For the results, we refer to Item 1 of  Additional Experiments in our general response.\n\n* For larger dataset sizes, as seen in the plots of  our manuscript, all methods converge to roughly the same performance (the setting is more similar to standard knowledge distillation where the teacher model has access to the full labelled dataset). See also our ablation in the previous question.\n\n* Experiments in which the the number of available labeled examples is small are of greater importance since this is the typical scenario where \"distillation with unlabeled examples\" applies.  (See also the \"performance gains of SlaM as a Function of the Number of Labeled Examples\" section -- bottom of page 8 of our manuscript -- for more details, and also the additional experiments we performed to answer the remark of Reviewer 7BQA.)\n\n\n>I wonder if the proposed method can work in noise label learning that also need reliable pseudo-labels? For example, on clothing1m dataset.\n\nSLaM is related to forward methods for dealing with label noise (see our Related Work section for more details).  However, the fundamental difference between our setting and the general learning under label noise setting is that the noise introduced by the teacher is structured, and this is a crucial observation we utilize in our design. Specifically, our approach is specifically tailored to the structure of the distillation with unabeled examples setting by exploiting that (i) we have access to confidence metrics of the teacher's predictions; and (ii) that often times, when the teacher model's top-$1$ prediction is inaccurate the true label is within its top-$k$ predictions for some appropriate $k$, to design and estimate a much more refined model for the teacher's noise that we use to inform the design of the student's loss function.  Given the above SLaM cannot be applied to general learning from noisy-labels tasks but it is interesting to investigate whether adaptations of it could be helpful in those settings.\n"}, {"review_id": "IszcKk3OF8", "reviewer": "Reviewer_7BQA", "summary": "This paper aims to systematically utilize the noise of teacher predictions in a knowledge distillation setting comprised of both labeled and unlabeled samples. The proposed method, SLaM linearly mixes the student predictions with the noise of the teacher predictions, thereby hoping that the student will learn the noise-free true underlying label. The method is supported by both theoretical results and empirical experiments on different datasets.", "questions": "See the above.", "limitations": "Sufficiently addressed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written, easy to follow, and nicely structured.\n- The proposed method, SLaM is relatively simple and general which is a nice feature.\n- The method is supported by both theoretical and empirical results.", "weaknesses": "I am generally positive about the paper, but my biggest concern is the empirical evaluation of the methods. I am open to changing my score if the following is addressed.\n\nComments and questions:\n- All empirical results presented are based on \"[...] the best test accuracy over all epochs.\". I.e. the methods are continuously evaluated on the test set (effectively making it a validation set), and no real test set performance is available - this is bad practice. Thus, I am concerned whether the reported scores are actually representative of the real test performance of the methods, and thereby if one can, in fact, rely on the conclusions drawn. I would strongly suggest reporting something like the test performance at the last epoch to avoid overfitting the test set. And even so, the experiments might be biased towards having evaluated the methods on the test set previously, so ideally evaluations on completely independent test sets would be included.\n- In e.g. L230 SLaM is reported to \"[...] consistently outperform the baselines, often by a large margin\". I believe this is too bold a claim. In particular, only 7 of 24 experiments yield improvements over $1\\%$ compared to the second-best baseline, while 8 other of 24 have improvements $<0.2\\%$ of which 3 have improvements $<0.02\\%$. Furthermore, there are 2 of 24 experiments where the method does not improve over the baselines. All of these observations are without consideration of the variance, which likely makes some additional improvements statistically indistinguishable. I would suggest reducing the claims slightly, especially given the evaluation protocol.\n- It is unclear why this method shouldn't be considered a semi-supervised learning method, as both labeled and unlabeled samples are provided and used for the distillation procedure. In fact, the availability of a labeled validations set, $V$ appears to be a necessity for this method, and I believe it should be presented as a semi-supervised setting more explicitly.\n- A lot of results and details are deferred to the appendix. While I fully understand the limitations a 9-page restriction can put on a paper, I believe some of the space in the paper could be better utilized to present at least some key results from the appendix.\n\nMinor comments:\n- L148: \"agree\" -> \"disagree\"\n- L166: \"etting\" -> \"setting\"\n- L311: \"inFigure 2\" -> \"in Figure 2\"\n- Figure 1 is never referenced in the paper\n- Figure 2 is badly formatted; i.e. too small a font and hard to distinguish the results/lines of different methods (consider rescaling the y-axis to e.g. improvement over baseline)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n\n>All empirical results presented are based on \"[...] the best test accuracy over all epochs... I would strongly suggest reporting something like the test performance at the last epoch to avoid overfitting the test set. ... completely independent test sets would be included.\n\nWe agree with the reviewer that reporting the last-epoch accuracy is a better metric and are willing to change our tables to report that. We used the best test accuracy over all epochs to be consistent with the way the baselines of the prior works were evaluated.  In the following table we report the final epoch accuracy of all methods for CIFAR100. We observe that SLaM again consistently outperforms the baselines. \n\nComparison of Last-Epoch Accuracy \n| CIFAR100 |  10%     |  15%     |  20%    |  25%    |  30%    |  35%    |\n|-|-|-|-|-|-|-|\n| Vanilla                           | $37.97 \\pm 0.1$  | $46.37 \\pm 0.12$ | $51.9 \\pm 0.13$  | $57.85 \\pm 0.17$ | $60.86 \\pm 0.18$ | $63.56 \\pm 0.35$ |\n| Taylor-CE                         | $40.12 \\pm 0.03$ | $47.99 \\pm 0.21$ | $54.18 \\pm 0.21$ | $57.76 \\pm 0.06$ | $61.22 \\pm 0.11$ | $63.50 \\pm 0.20$ |\n| UPS                               | $39.47 \\pm 0.18$ | $48.29 \\pm 0.20$ | $53.27 \\pm 0.26$ | $57.83 \\pm 0.29$ | $61.16 \\pm 0.13$ | $62.58 \\pm 0.28$ |\n| VID                               | $37.76 \\pm 0.13$ | $46.3 \\pm 0.18$  | $52.07 \\pm 0.12$ | $57.91 \\pm 0.31$ | $60.57 \\pm 0.21$ | $63.43 \\pm 0.18$ |\n| Weighted                          | $38.39 \\pm 0.07$ | $46.91 \\pm 0.08$ | $52.48 \\pm 0.13$ | $57.7 \\pm 0.15$  | $\\mathbf{61.32 \\pm 0.31}$ | $63.66 \\pm 0.46$ |\n| SLaM                              | $\\mathbf{41.16 \\pm 0.16}$ | $\\mathbf{49.11 \\pm 0.17}$ | $\\mathbf{54.31 \\pm 0.19}$ | $\\mathbf{58.59 \\pm 0.24}$ | $\\mathbf{61.30 \\pm 0.26}$ | $\\mathbf{63.72 \\pm 0.43}$ |\n\n\n> In e.g. L230 SLaM is reported to \"[...] consistently outperform the baselines, often by a large margin\". I believe this is too bold a claim. In particular, only 7 of 24 experiments yield improvements over 1 compared to the second-best baseline, while 8 other of 24 have improvements < 0.2 of which 3 have improvements < 0.02. Furthermore, there are 2 of 24 experiments where the method does not improve over the baselines. All of these observations are without consideration of the variance, which likely makes some additional improvements statistically indistinguishable. I would suggest reducing the claims slightly, especially given the evaluation protocol.\n\nWe are certainly open to changing the exact phrasing. However, it is worth mentioning that not all experiments are of the same \"importance\":\n\n* Experiments in which the the number of available labeled examples is small are of greater importance since this is the typical scenario where \"distillation with unlabeled examples\" applies.  In these cases, our method indeed often outperforms other methods by a large magin. (See also the \"performance gains of SlaM as a Function of the Number of Labeled Examples\" section for more details -- bottom of page 8 of our manuscript -- and also the additional experiments we performed to answer the remark of Reviewer Un32.)\n* Our method significantly outperforms the other baselines on Imagenet which is arguably the most \"difficult\" dataset. As prominent examples: when given labels for only 5% of the Imagenet examples our method is better by **more than 6%** from every other baseline; when given labels for only 10% of Imagenet our method is better by **more than 3%** from every other baseline.\n\n\nGiven the above we are willing to change the phrasing to \"[...] often by a large margin in the important cases where (i) only few labeled examples are available; (ii) one deals with large-scale problems with many classes like the Imagenet dataset\". \n\nWe are also happy to adopt any specific suggestion by the reviewers.\n\n\n\n>It is unclear why this method shouldn't be considered a semi-supervised learning method, as both labeled and unlabeled samples are provided and used for the distillation procedure. In fact, the availability of a labeled validations set, V appears to be a necessity for this method, and I believe it should be presented as a semi-supervised setting more explicitly.\n\n* We definitely agree that distillation with unlabeled examples and semi-supervised learning have a lot in common — see the Semi-Supervised Learning paragraph in the \"Related Work section\" where we explain the similaraties and differences.  It is also true that our method could potentially be applied in a purely semi-supervised learning setting and, moreover, potentially be combined with other semi-supervised learning techniques  — especially given the performance of our methond on Imagenet (a typically difficult dataset for SOTA semi-supervised learning techniques). \n* However, we believe that distillation with unlabeled examples is an important problem of its own (given the popularity of the approach in practical applications) and our goal in this paper is to focus on it and, in particular, to study ways to deal with \"teacher's noise\" both practically and theoretically. \n\n* We are open to any suggestions by the reviewers for highlighting the connection to semi-supervised learning more explicitly.\n\n> A lot of results and details are deferred to the appendix. While I fully understand the limitations a 9-page restriction can put on a paper, I believe some of the space in the paper could be better utilized to present at least some key results from the appendix.\n\nThis is certainly true — and we indeed had a hard time deciding what results we should defer to the appendix. We promise to make an effort to include more results from the appendix to the main body of the paper, and we are open to any suggestions by the reviewers for what they think we should include.\n"}]

Author rebuttal:

## General Response 
We want to thank all reviewers for taking the time to read our manuscript carefully and for providing constructive and insightful feedback. We are very encouraged by the positive comments of the reviewers on **tackling a well-motivated and important problem** (Reviewers E5Wk, PQzZ) the **simplicity, generality, and novelty of SLaM** (Reviewers E5Wk, PQzZ, Un32, 7BQA), **its extensive experimental evaluation and performance gains** (Reviewers E5Wk, PQzZ, Un32), **its strong theoretical guarantees** (Reviewers PQzZ, Un32, 7BQA), **the writing quality and the clarity of the presentation of the ideas** (Reviewers PWzZ, 7BQA).

Here we give an update about our theoretical contribution and provide an overview of the additional experiments included in the rebuttal. We provide detailed responses to each reviewer separately.

### Additional Experiments

1. **SLaM with (even) fewer labels**
As proposed by reviewer Un32, we extended the range of the the size of the labeled-dataset considered in our experiments. In particular, we provide experiments with even fewer labeled examples available. (Experiments in which the number of available labeled examples is small are of greater importance since this is the typical scenario where "distillation with unlabeled examples" applies. For larger dataset sizes, as seen in the plots of  our manuscript, all methods converge to roughly the same performance.) See response to Reviewer Un32 for more details.

|Method||CIFAR10 1%|CIFAR10 5%||CIFAR100 1%|CIFAR100 5%|
|-|-|-|-|-|-|-|
|Teacher | |$10.07$| $51.67$ || $9.45$ | $23.61$||
|Vanilla || $11.75 \pm 0.1$| $54.2 \pm 0.16$|| $10.08 \pm 0.06$| $25.15 \pm 0.11$|
|Taylor-CE || $10.00 \pm 0.01$| $55.14 \pm 0.28$|| $9.79 \pm 0.13$| $26.14 \pm 0.39$|
|UPS|| $12.74 \pm 0.94$| $56.21 \pm 0.23$| | $10.40 \pm 0.05$| $26.41 \pm 0.13$|
|VID|| $13.25 \pm 0.26$| $54.32 \pm 0.05$|| $10.02 \pm 0.13$| $24.93 \pm 0.19$|
|Weighted || $12.67 \pm 0.01$| $54.58 \pm 0.1$ || $10.07 \pm 0.06$| $25.36 \pm 0.14$|
|SLaM || $\mathbf{26.73 \pm 0.02}$| $\mathbf{57.40 \pm 0.05}$|| $\mathbf{10.87 \pm 0.07}$| $\mathbf{27.76 \pm 0.19}$|

2. **Robustness of SLaM to inaccuracies in $\alpha(x), k(x)$** 
We tested the robustness of SLaM to inaccurate predictions for $\alpha(x), k(x)$ as proposed by Reviewer PQzZ.  As the reviewer suggested, we start from the oracle values for $\alpha(x)$ and $k(x)$, i.e., $\alpha^*(x) = 1$ if the teacher prediction is correct on $x$ and $0$ if it is incorrect and $k^*(x)$ is equal to the smallest integer value $\ell$ so that the ground-truth label is contained in the top $\ell$ predictions of the teacher. We then introduce random noise to the oracle predictions: for $\alpha(x)$ we add random perturbations with variance roughly $\sigma^2$ and for $k(x)$ we add a random integer in the interval $[-\ell, \ell]$ to the oracle prediction $k^*(x)$.  For more details on the noise operation we refer to our response to Reviewer PQzZ. We test our method on CIFAR100 and **observe that the accuracy of SLaM gracefully decays as the predictions for $\alpha(x)$ and $k(x)$ become worse** (bottom right corner is the noisiest $\sigma = 0.5, \ell=90$ and top left is the noiseless $\sigma=0, \ell =0$).
 

|CIFAR100 20% Labeled Data | $\ell$ = 0|$\ell$ = 5| $\ell$ = 10 | $\ell$ = 50 | $\ell$ = 90|
|-|-|-|-|-|-|
|$\sigma$ = 0| $61.69 \pm 0.3$| $59.71 \pm 0.49$ | $59.7 \pm 0.55$  | $59.63 \pm 0.44$ | $59.4 \pm 0.65$ |
|$\sigma$ = 0.1| $60.04 \pm 0.29$| $59.55 \pm 0.33$ | $59.66 \pm 0.35$ | $59.79 \pm 0.2$  | $60.03 \pm 0.41$|
|$\sigma$ = 0.2| $59.39 \pm 0.1$ | $59.23 \pm 0.25$ | $59.5 \pm 0.39$  | $59.16 \pm 0.29$ | $59.31 \pm 0.31$|
|$\sigma$ = 0.5| $57.71 \pm 0.15$ | $57.6 \pm 0.23$  | $57.56 \pm 0.28$ | $57.46 \pm 0.25$ | $57.29 \pm 0.21$|

3. Ablations for hyperparameters of $\alpha(x)$ and $k(x)$ as requested by reviewer E5Wk.  We investigated the effect of the isotonic-regression lower bound hyper-parameter and the two methods for estimating $k(x)$. For more we refer to our response to Reviewer E5Wk. The corresponding tables are Table 2 and Table 3 in the pdf of the rebuttal.

4. We investigated using different regression methods (knn, Logistic regression) for estimating $\alpha(x), k(x)$ with SLaM as proposed by Reviewer PQzZ.  **We see that Isotonic regression typically outperfoms other methods. Moreover, SLaM provides consistent improvements regardless of regression method used**.  We refer to our response to Reviewer PQzZ for more details.

5. An ablation investigating the effect of the validation size, see Response to Reviewer E5Wk. **We show that SLaM is able to provide improvements even with very small validation datasets** (e.g., with 128 labels). This is consistent with the fact that SLaM is robust to inaccuracies in the parameters $\alpha(x), k(x)$, see Item 2 of Additional Experiments in the general response.

6. We include a table reporting the last-epoch accuracy of all methods on CIFAR100 as proposed by Reviewer 7BQA. The last-epoch accuracy of SLaM is again better than the baselines. See our response to Reviewer 7BQA for more details.

### Theoretical Contribution

We would like to share with the reviewers and ACs an exciting recent result presenting strong evidence about the optimality of our theoretical result for SLaM.

In the very recent paper [DKK+23] a Statistical Query (SQ) lower bound of $\Omega(1/(\gamma^{1/2} \epsilon^2))$ was given for learning halfspaces with random classification noise. This implies that there is now strong evidence (SQ algorithms is very broad class that contains gradient-based optimization methods) that **our theoretical result for SLaM is, in fact, optimal in its dependence on the accuracy parameter $\epsilon$.**  We will update our conclusion section to include this discussion.

[DKK+23]: Diakonikolas I., Diakonikolas J., Daniel K., Wang P., Zarifis N. Information-Computation Tradeoffs for Learning Margin Halfspaces with Random Classification Noise. COLT 2023

Meta-review: This paper proposes a method for improving knowledge distillation with unlabeled examples called Student-Label Mixing (SLAM). Knowledge distillation is widely used, and this paper seeks to improve the process by modeling the noise from the teacher model. By evaluating the student model using both its prediction and an estimated label from the noise model, SLAM can reduce the effect of noise in the training pseudo-labels. Experiments show that SLAM provides consistent performance improvements on standard benchmarks and beneficial combination with other standard techniques for distillation. A theoretical analysis also shows that SLAM can provably learn halfspaces with noise with sample complexity based on the size of the margin and amount of noise.

During the review and discussion period, the reviewers identified the contributions to knowledge distillation as novel and valuable. The principled approach was viewed as a strength. The main concerns of the reviewers generally focused on the experimental evaluation. Two areas that came up were evaluating SLAM on even smaller amounts of labeled data and evaluating its robustness by introducing additional noise. The authors provided additional results addressing these concerns and the reviewers were all in favor of acceptance. The authors are strongly encouraged to add these additional results to the final version of the paper.
Decision: Accept (poster)