Title: SLaM: Student-Label Mixing for  Distillation with Unlabeled Examples
Venue: NeurIPS 2023
Accepted: yes
Keywords: Distillation

teacher

student
Abstract: Knowledge distillation with unlabeled examples is a powerful training paradigm for generating compact and lightweight student models in applications where the amount of labeled data is limited but one has access to a large pool of unlabeled data. In this setting, a large teacher model generates "soft" pseudo-labels for the unlabeled dataset which are then used for training the student model. Despite its success in a wide variety of applications, a  shortcoming of this approach is that the teacher's pseudo-labels are often noisy, leading to impaired student performance. In this paper, we present a principled method for knowledge distillation with unlabeled examples that we call Student-Label Mixing (SLaM) and we show that it consistently improves over prior approaches by evaluating it on several standard benchmarks. 
Finally, we show that SLaM comes with theoretical guarantees; along the way we give an algorithm improving the best-known sample complexity for learning halfspaces with margin under random classification noise, 
and provide the first convergence analysis for so-called ``forward loss-adjustment" methods.

SLaM: Student-Label Mixing for Distillation with Unlabeled Examples

Abstract

Knowledge distillation with unlabeled examples is a powerful training paradigm for generating compact and lightweight student models in applications where the amount of labeled data is limited but one has access to a large pool of unlabeled data. In this setting, a large teacher model generates 'soft' pseudo-labels for the unlabeled dataset which are then used for training the student model. Despite its success in a wide variety of applications, a shortcoming of this approach is that the teacher's pseudo-labels are often noisy, leading to impaired student performance. In this paper, we present a principled method for knowledge distillation with unlabeled examples that we call Student-Label Mixing (SLaM) and we show that it consistently improves over prior approaches by evaluating it on several standard benchmarks. Finally, we show that SLaM comes with theoretical guarantees; along the way we give an algorithm improving the best-known sample complexity for learning halfspaces with margin under random classification noise, and provide the first convergence analysis for so-called 'forward loss-adjustment' methods.

1 Introduction

While good quality human-labeled data are often hard to obtain, finding huge amounts of unlabeled data is relatively easy. Therefore, in modern machine learning applications, we often face the situation where we have a small 'golden' dataset with human labels and a large unlabeled dataset. In Distillation with Unlabeled Examples [12, 26, 16] a large teacher model is first trained (or fine-tuned) on the human-labeled data and is then used to generate 'soft' pseudo-labels for the unlabeled dataset. Then the (typically smaller) student model, i.e., the model that will be deployed for the purposes of the application, is trained on the combined dataset that contains both the labels generated by humans and the pseudo-labels generated by the teacher model. This general-purpose training paradigm has been applied in a wide variety of contexts [16, 46, 53, 54, 58, 57] including but not limited to distilling knowledge from large-scale foundational models like BERT [18] and GPT-3 [11]. We remark that in such settings one does not have access to the teacher model but only on its pseudo-labels (which were generated during some previous 'bulk-inference' phase). This 'bulk-inference' step is typically computationally expensive and happens once: one cannot modify the teacher network (or even use it for inference) during the training process of student.

Despite its widespread success in practice, the effectiveness of this powerful approach generally depends on the quality of the pseudo-labels generated by the teacher model. Indeed, training the student model on noisy pseudo-labels often leads to significant degradation of its generalization

performance, and this is a well-known phenomenon that has been observed and studied in a plethora of papers in the literature, e.g., [6, 36, 44, 51, 53, 8, 27].

In this work, we propose Student-Label Mixing (SLaM), a principled method for knowledge distillation with unlabeled examples that accounts for the teacher's noise and consistently improves over prior approaches. At the heart of our method lies the observation that the noise introduced by the teacher is neither random nor adversarial, in the sense that it correlates well with metrics of 'confidence' such as the margin score or the entropy of the teacher's predictions. We exploit this empirical fact to our benefit in order to introduce a model for the teacher's noise, which we use to appropriately modify the student's loss function. At a high level, for any given example during the student's training process, we evaluate the student's loss function on a convex combination of the student's current prediction and another (soft-)label that we estimate using our model for the teacher's noise (hence the name 'student-label mixing').

Our contributions can be summarized as follows:

We propose SLaM: a principled method for improving knowledge distillation with unlabeled examples. The method is efficient, data-agnostic and simple to implement.

We provide extensive experimental evidence and comparisons which show that our method consistently outperforms previous approaches on standard benchmarks. Moreover, we show that SLaM can be combined with standard distillation techniques such as temperature scaling and confidence-based weighting schemes.

We give theoretical guarantees for SLaM under standard assumptions. As a byproduct of our analysis we obtain a simple 'forward loss-adjustment' iteration that provably learns halfspaces with γ -margin under Random Classification Noise with O (1 / ( ϵ 2 γ 2 )) samples improving over prior works that had worse dependence on either the margin γ or the generalization error ϵ (see Theorem 5.1 and Remark 5.2).

2 Related Work

Knowledge Distillation. Most of the literature on knowledge distillation has been focused on the fully supervised/labeled setting , i.e., when distillation is performed on the labeled training data of the teacher model rather than on new, unlabeled data - see e.g. the original paper of [26]. Naturally, in this setting the pseudo-labels generated by the teacher are almost always accurate and so many follow-up works [2, 14, 15, 41, 52] have developed advanced distillation techniques that aim to enforce greater consistency between the teacher's and the student's predictions, or even between the intermediate representations learned by the two models. Applying such methods in our setting where the training dataset contains mainly unlabeled examples is still possible but, in this case, it is known [51, 27] that fully trusting the teacher model can be actually harmful to the student model, making these methods less effective. (In fact, when the teacher is highly noisy these methods even underperform vanilla distillation with unlabeled examples.) In Section 4.2 we present results that show the improved effectiveness of SLaM relative to the state-of-the-art supervised knowledge distillation methods like the Variational Information Distillation for Knowledge Transfer (VID) framework [2]. Moreover, in Appendix D.5 we show that our method can be combined with (i.e., provide an additional improvement) the most simple, yet surprisingly effective, methods of improving knowledge distillation, namely the temperature-scaling idea introduced by [26].

For distillation with unlabeled examples, many approaches [17, 33, 29] propose filtering-out or reweighting the teacher's pseudo-labels based on measures of teacher's uncertainty, such as dropout variance, entropy, margin-score, or the cut-statistic. These methods are independent of the student model and can be synergistically combined with our technique. For instance, in Section D.4 we demonstrate that combining our method with teacher-uncertainty-based reweighting schemes leads to improved student performance relative to applying the reweighting scheme alone.

Much more closely related to our approach is the recently introduced approach of [27]. There, the authors design a model for the teacher's noise and utilize it in order to modify the student's loss function so that, in expectation, the loss simulates the loss with respect to noise-free pseudo-labels. One of the main advantages of our method compared to that of [27] is that our model for the teacher's noise is more structured and easier to learn, which - as our experiments in Section 4.2 show - leads to consistently better student performance.

Learning From Noisy Labels. Learning from noisy labels is an important and well-studied problem with a vast literature [7, 21, 23, 28, 31, 37, 40, 42, 45, 47] - see [50] for a recent survey. The fundamental difference between our setting and papers in this literature is that the noise introduced by the teacher is structured, and this is a crucial observation we utilize in our design. Specifically, our approach is inspired by the so-called forward loss-adjustment methods , e.g. [43], but it is specifically tailored to the structure of the distillation with unabeled examples setting. Indeed, forward methods typically attempt to estimate a noise transition matrix whose ( i, j ) entry is the probability of the true label i being flipped into a corrupted label j , which can be rather problematic when dealing with general, instance specific noise like in the case of distillation with unlabeled examples. On the other hand, we exploit that (i) we have access to confidence metrics of the teacher's predictions; and (ii) that often times, when the teacher model's top1 prediction is inaccurate the true label is within its topk predictions for some appropriate k , to design and estimate a much more refined model for the teacher's noise that we use to inform the design of the student's loss function.

Another related technique for dealing with noisy data is using 'robust' loss functions [4, 20, 24, 35, 56] such that they achieve a small risk for new clean examples even under the presence of noise in the training dataset. In Section 4.2 we compare our method with the general framework of [20] for designing robust loss functions and we show that our approach, when applied to the standard cross-entropy loss, consistently outperforms [20] in the setting of distillation with unlabeled examples. That said, we stress that our method is not tied to the cross-entropy loss and, in fact, it often gives better results when combined with more sophisticated loss functions. We demonstrate this in Appendix D.6 where we apply our method in cases where the student loss function comes from the families of losses introduced in [20] and [35].

Semi-Supervised Learning. Akin to our setting, in semi-supervised learning (SSL) (see e.g. [55] for a recent survey) the learner is presented with a small labeled dataset A and a typically much larger unlabeled dataset B . Unlike to our setting though, there is typically no distinction between the student and teacher: the model of interest generates pseudo-labels on B which are utilized by using appropriate loss functions or preprocessing procedures (e.g. 'filtering' or 'correcting') often times in an iterative fashion with the goal of improving the quality of the newly-generated pseudo-labels. It is also worth noting that in many real-world applications of distillation with unlabeled examples either the teacher model is unavailable or it is too expensive to retrain it and create fresh pseudo-labels on the data (e.g., when we request labels from a pretrained large language model). Therefore, SSL approaches that either (i) update the 'teacher' model (e.g., [34]), or (ii) require several fresh teacher-generated pseudo-labels (e.g., by requesting teacher-predictions on random data-augmentations or perturbed version of the unlabeled examples of B e.g., [9]) are not applicable in our setting. We implement the recent SSL technique of [48] and show that our method outperforms it in the context of distillation with unlabeled examples. Besides performing on par with state-of-the-art SSL approaches like [9], the method of [48] is free of inherent limitations like using domain-specific data augmentations - which is also an important feature of our approach.

Learning Halfspaces with Random Classification Noise. The theoretical study of classification with Random Classification Noise (RCN) was initiated by [5]. For the fundamental class of linear classifiers (halfspaces) the first polynomial time algorithms for the problem where given in [13] and [10]. The iteration proposed in [13] is a 'backward loss-adjustment' method [43] for which it is known that resulting optimization landscape is convex (for linear classifiers). In [19] an improved analysis of the method of [13] was given, showing that SGD on this convex loss learns γ -margin halfspaces with RCN with ˜ O (1 / ( γ 4 ϵ 2 )) samples. On the other hand, forward loss-adjustment methods for dealing with RCN are known to result in an inherently non-convex landscape, see [38] and Figure 9). Our theoretical result for SLaM (see Theorem 5.1) is the first convergence result for a 'forward loss-adjustment' method and, at the same time, achieves a sample complexity of O (1 / ( γ 2 ϵ 2 )) improving over the prior work.

3 SLaM: Student-Label Mixing Distillation

In this section, we describe our distillation with unlabeled examples setting and present SLaM. In what follows, we assume that examples are represented by feature-vectors in some space X . Weshall denote by X the distribution over examples. We consider multi-class classification with L classes and assume that the ground-truth label of an example x is represented by a one-hot vector in Y = { 0 , 1 } L given by some unknown function g ( x ) : X ↦→ Y . In multi-class classification the learning algorithm typically

optimizes a parametric family of classification models F = { f ( · ; w ) : X ↦→ R L : w ∈ W} , i.e., for every parameter w ∈ W , f ( x ; w ) is an L -dimensional 'score vector', where f ( x ; w ) i corresponds to the probability that the model assigns to the class i for the example x . We shall denote by ℓ ( · , · ) : R L × R L ↦→ R the classification loss function used by the learning algorithm. During training the algorithm considers a set of labeled examples S = { ( x (1) , g ( x (1) )) , . . . , ( x ( n ) , g ( x ( n ) ) } and optimizes the loss ℓ ( · , · ) over S , i.e., solves the problem min w ∈W 1 | S | ∑ ( x,g ( x )) ∈ S ℓ ( g ( x ) , f ( x ; w )) .

̸

For two vectors v, u ∈ R L we denote by err( v, u ) = 1 { argmax( v ) = argmax( u ) } the indicator of the event that the positions of the maximum elements of v, u agree. Similarly, for two classifiers h ( x ) , f ( x ) : R d ↦→ R L we can use err( h ( x ) , f ( x )) to denote whether their top-1 predictions for the example x agree. Our goal is to train a classifier over the sample S so that its generalization error, i.e., E x ∼ X [err( f ( x ; w ) , g ( x ))] , is small.

Distillation with Unlabeled Examples. We assume that we are given a (usually small) dataset A of correctly labeled examples ( x, g ( x )) and a set of unlabeled data U . A 'teacher' model y s ( · ) : X ↦→ R L is first trained on the labeled dataset A and then provides soft-labels for the examples of dataset U , i.e., we create a dataset B = { ( x, y s ( x )) : x ∈ U } containing examples labeled with the corresponding probability distribution over classes (soft-labels) of the teacher model. We then train a (typically smaller) student model using both the original labeled data A and the teacher-labeled dataset B , i.e., min w ∈W 1 | A ∪ B | ∑ ( x,z ) ∈ A ∪ B ℓ ( z, f ( x ; w )) . In what follows, we shall call the above training procedure as 'vanilla-distillation'.

Remark 3.1 ('Hard-' vs 'Soft-' Distillation) . We remark that the process where instead of using the soft-labels provided by the teacher model on the unlabeled dataset U, we use one-hot vectors representing the class with maximum score according to the teacher, is known as hard-distillation. We will denote by y s ( x ) the soft-label of the teacher and by y ( x ) the corresponding hard-label, i.e., y ( x ) is the one-hot representation of argmax y s ( x ) . When it is clear from the context we may simply write y instead of y ( x ) .

Modelling the Teacher as a 'Noisy' Label Oracle. In the distillation setting described in the previous paragraph, it is known [51, 27, 8, 44] that the teacher model often generates incorrect predictions on the unlabeled examples, impairing the student's performance . Given any x ∈ U , we model the teacher's prediction y as a random variable. Similarly to [27] we assume that, for every unlabeled datapoint x ∈ U , the provided teacher label y is correct with probability α ( x ) and incorrect with probability 1 -α ( x ) . However, in contrast with [27], our noise model prescribes a non-advsersarial (semi-random) behavior of the teacher when its top-1 prediction is incorrect.

A first step towards more benign noisy teachers is to assume that, conditionally on being wrong, the teacher label is a uniformly random class of the remaining L -1 classes. We remark that this model is already enough to give improvements in datasets with a moderately large number of classes (e.g., up to 100). In particular, it perfectly captures the noisy teacher in binary classification: when the teacher label is different than the ground-truth g ( x ) then it has to be equal to the 'flipped' ground-truth 1 -g ( x ) .

We now further refine our model so that it is realistic for datasets with thousands of classes. Even though the top-1 accuracy of the teacher model may not be very high on the unlabeled data U , the true label is much more likely to belong in the top-5 or top-10 predictions of the teacher rather than being completely arbitrary. For example, training a ResNet50 network on 10% of ImageNet [49] yields an average top-1 accuracy about 52 . 78% on the test dataset whereas the top-10 accuracy of the same model is about 83 . 55% . In datasets with a large number of classes, this observation significantly reduces the number of potential correct classes of the examples where the teacher label is incorrect. Motivated by the above, we assume the following structured, semi-random noise model for the teacher, tailored to multi-class settings.

Definition 3.2 (Noisy Teacher Model) . Let x be any example of the unlabeled data U and denote by g ( x ) its ground-truth label. Let y s ( x ) resp. y ( x ) be the random variable corresponding to the soft resp. hard prediction of the teacher model for the example x . We assume that for every x there exist (unknown to the learner) α ( x ) ∈ [0 , 1] and k ( x ) ∈ { 2 , . . . , L } such that the teacher's top-1 prediction y agrees with the ground-truth g ( x ) with probability α ( x ) and, with probability 1 -α ( x ) : (i) the

ground-truth belongs in the topk ( x ) predictions of the teacher; and (ii) the teacher's (hard)-prediction is a uniformly random incorrect class out of the topk ( x ) predictions of the teacher soft-label y s ( x ) 1 .

Remark 3.3 . We remark that the model of Definition 3.2 captures having a 'perfect' teacher model by setting α ( x ) = 1 for all x and also generalizes the binary case described above by taking k ( x ) = 2 for all x ∈ X .

Given the above noise model for the teacher, the problem of improving knowledge-distillation consists of two main tasks: (i) obtaining estimates for accuracy statistics α ( x ) , k ( x ) for each example x ∈ U ; and (ii) using those estimated values to improve the training of the student model so that it is affected less by the mistakes of the teacher on dataset B .

Training Better Students Using α ( x ) , k ( x ) We first assume that for every x we have oracle access to the values α ( x ) , k ( x ) and present our Student-Label Mixing loss function. Instead of using α ( x ) , k ( x ) to 'denoise' the teacher's label, we use them to add noise to the student's predictions . To make notation more compact, in what follows, given a vector z ∈ R L we denote by top( z ; k ) the vector that has the value 1 in the positions of the of the 1 -st up to k -th largest elements of z and 0 in all other positions, e.g., top((1 , 2 , 3); 1) = (0 , 0 , 1) and top(( -1 , 1 , 0 , 2); 3) = (0 , 1 , 1 , 1) . Assuming that the student-label for some x ∈ U is f ( x ; w ) we 'mix' it (hence the name Student-Label Mixing) using α ( x ) , k ( x ) to obtain the mixed prediction

where q ∗ p is the element-wise multiplication of the vectors p, q . We then train the mixed student model, on the 'noisy' dataset B :

The main intuition behind the mixing of the student's labels is that by training the 'noisy' student to match the 'noisy' teacher label y on dataset B , the underlying (non-mixed) student f ( x ; w ) will eventually learn the ground-truth . In particular, when ℓ ( · , · ) is the Cross-Entropy loss we have that the expected mixed loss conditioned on any x is

where we used the fact that the cross-entropy is linear in its first argument, and that by the definition of our noise model (Definition 3.2) it holds that E [ y | x ] = mix( g ( x ); α ( x ) , k ( x )) . Therefore, when the student is equal to the ground-truth f ( x ; w ) = g ( x ) , we obtain that the mixed studentmodel will satisfy mix( g ( x ); α ( x ) , k ( x )) = mix( f ( x ; w ); α ( x ) , k ( x )) for all x ∈ X , and (by Gibb's inequality), we obtain that g ( x ) is a minimizer of the SLaM loss. We show the following proposition, see Appendix C for the formal statement and proof.

Proposition 3.4 (SLaM Consistency (Informal)) . Let D be the distribution of the teacher-labeled examples of dataset B , i.e., we first draw x ∼ X and then label it using the noisy teacher of Definition 3.2. Moreover, assume that there exists some parameter w ∗ ∈ W such that the ground-truth g ( x ) = f ( x ; w ∗ ) . Then w ∗ is the minimizer of the (population) SLaM objective: min w E ( x,y ) ∼ D [ce( y, mix( f ( x ; w ); α ( x ) , k ( x )))] , where ce( · , · ) is the Cross-Entropy loss.

Estimating the Teacher's Accuracy Statistics α ( x ) , k ( x ) via Isotonic Regression We first show how we estimate α ( x ) for each x of dataset B , i.e., the dataset labeled by the teacher model. In [27] the authors empirically observed that α ( x ) correlates with metrics of teacher's confidence such as the 'margin', i.e., the difference between the probabilities assigned in the top-1 class and the second largest class according to the teacher's soft label y s . In particular, the larger the margin is the more likely is that the corresponding teacher label is correct. We exploit this monotonicity by employing isotonic regression on a small validation dataset to learn the mapping from the teacher's margin at an example x to the corresponding teacher's accuracy α ( x ) . For more details, see Appendix B.1.

To perform this regression task we use a small validation dataset V with correct labels that the teacher has not seen during training. For every example x ∈ V we compute the corresponding

soft-teacher label y s ( x ) and compute its margin margin( x ) = max 1 ( y s ( x )) -max 2 ( y s ( x )) . For every x ∈ V we also compute the hard-prediction of the teacher and compare it with the ground-truth, i.e., for every x ∈ V the covariate and responce pair is (margin( x ) , 1 -err( g ( x ) , y ( x ))) . We then use isotonic regression to fit a piecewise constant, increasing function to the data. We remark that isotonic regression can be implemented very efficiently in O ( n log n ) time (where n is the size of the validation dataset).

For k ( x ) we consider two different options: (i) using the same value for all examples (e.g., using k so that the top-k accuracy of teacher is above some threshold on the validation data); and (ii) using a 'data-dependent' k ( x ) that we estimate by solving L (recall that L is the number of classes) isotonic-regression problems (similar to that for estimating α ( x ) above). We refer to Appendix B.1 for more details.

4 Experimental Evaluation

In this section, we present our experimental results. In Section 4.1 we describe our experimental setup and in Section 4.2 we compare the performance of our method with previous approaches on standard benchmarks. In Section D.4 we show that our method can be combined with teacher-uncertainty-based reweighting techniques. Finally, due to space limitations, we provide additional empirical results in the Appendix: in Appendix D.5 we show that SLaM can effectively be used with distillation temperature, and in Appendix D.6 we consider using SLaM with other losses beyond the Cross-Entropy.

4.1 The Setup

Here, we describe our procedure for simulating knowledge distillation with unlabeled examples on academic datasets. We start by splitting the training dataset in two parts: dataset A and dataset C. We then train the teacher and student models on dataset A (using the standard crossentropy loss). 2 Then we perform multiple independent trials where, for each trial, we randomly split dataset C into a small (e.g., 500 examples validation dataset V and an unlabeled training dataset U. For each trial we (i) use the teacher model to label the points on dataset U to obtain the teacher-labeled dataset B (ii) initialize the weights of

the student to those of the student model that was pre-trained on dataset A; (iii) train the student model (using each distillation method) on the combined labeled data of A, V (that have true labels) and the data of B (that have teacher labels). We remark here that we include the validation data V during the training of the student to be fair towards methods that do not use a validation dataset. However, while it is important that the teacher has not seen the validation data during training, the performance of no method was affected significantly by including (or excluding) the validation data from the training dataset.

4.2 Comparison with Previous Approaches

The Baselines A natural question is whether a more sophisticated distillation method that enforces greater consistency between the teacher and the student, would improve distillation with unlabeled examples: we use the VID method [2] that incorporates the penultimate layer of the student model (after a suitable trainable projection) in the loss. We also compare our method against the weighted distillation method of [27] that reweights the examples of dataset B in order to 'correct' the effect of the noisy pseudo-labels provided by the teacher. The Taylor cross-entropy method of [20] is

CIFAR10: Soft-Distillation

CIFAR100: Soft-Distillation

CelebA: Soft-Distillation

80 -

78 -

76

UPS

VID

SLaM (Ours)

Weighted

60 -

65

UPS

Weighted

VID

SLaM (Ours)

94 -

- S6

→

+

UPS

VID

Weighted

SLaM (Ours)

(%)

Test Accuracy

74

Test Accuracy

55

93 -

72

70 -

50 -

92 -

68 -

45 -

91 -

66 -

64 -

40 -

- 06

10%

15%

20%

25%

30%

35%

10%

15%

20%

25%

30%

35%

2%

3%

4%

5%

6%

7%

Percentage of Labeled Examples

Percentage of Labeled Examples

Percentage of Labeled Examples

a modification of CE that truncates the taylor-series of the CE loss. In [20] it was shown that it offers significant improvements when the labels are corrupted by random classification noise. The fact that the teacher's noise is much closer to random than to adversarial makes this approach a natural baseline. The UPS loss of [48] is a semi-supervised technique that takes into account the variance (uncertainty) of the teacher model on the examples of dataset B in order to transform the soft pseudo-labels provided by the teacher to more 'robust' binary vectors and then use a modified binary CE loss. To estimate the uncertainty of the teacher model, we used either dropout with Monte-Carlo estimation or random data-augmentations as suggested in [48]. We remark that, as we discussed in Section 2 and Section 1, strictly speaking, this method is not applicable in our setting because it requires multiple forward passes of the teacher model to estimate its variance but we implement it as it is a relevant approach that aims to improve the pseudo-labels of the teacher.

CIFAR-{ 10,100 } and CelebA Here we present our results on CIFAR-{ 10, 100 } [30] and CelebA [22]. CIFAR-10 and CIFAR-100 are image classification datasets with 10 and 100 classes respectively. They contain 60000 labeled images, which are split to a training set of 50000 images, and a test set of 10000 images. From the 50000 images of the train set we use the 10% , 15% , 20% , 25% , 30% , 35% (or 5000, 7500, 10000, 12500, 15000, and 17500 examples) as the labeled dataset A where we train the teacher and pre-train the student models. For each size of dataset A, we perform a random split on the remaining training data and use 500 labeled examples as the validation dataset and the remaining examples as the unlabeled dataset U. For the CIFAR-10 experiments, we use a Mobilenet with depth multiplier 2 as the teacher, and a Mobilenet with depth multiplier 1 as the student. For CIFAR-100, we use a ResNet-110 as a teacher, and a ResNet-56 as the student. We compare the methods both on soft- and hard-distillation. For each trial we train the student model for 200 epochs and keep the best test accuracy over all epochs. We perform 3 trials and report the average of each method and the variance of the achieved accuracies over the trials. The results of our experiments for soft-distillation can be found in Table 1 and Table 2. The corresponding plots are given inFigure 2. We include our results for hard-distillation in Appendix D.2.

We consider the male/female binary classification task using the CelebA dataset [22] consisting of a training set of 162770 images and a test set of 19962 images. We use a MobileNet with depth multiplier 2 as the teacher, and a ResNet-11 as the student. As the labeled dataset A we used 2% , 3% , 4% , 5% , 6% percent (or 3256, 4883, 6510, 8138, 9766, 11394 examples) of the training dataset and split the remaining data in a validation dataset of 500 examples and an unlabeled dataset U. Our results for CelebA can be found in Table 3 (soft-distillation) and in Table 7 (hard-distillation).

The corresponding plots are given in Figure 2. Due to space limitations our results for hard-distillation can be found in Appendix D.2.

Taken together, our comparisons show that SLaM consistently outperforms the baselines, often by a large margin. The reader is referred to Appendix D.1 for additional details.

Remark 4.1 (Soft-Distillation and Temperature Scaling) . We remark that in the comparisons we performed soft-distillation with temperature set to 1 , i.e., for every example we do not scale the corresponding teacher and student logits. In Appendix D.5 we show that our method can readily be used together with temperature scaling to improve the accuracy of the student model.

ImageNet Here we present the results on ImageNet [49]. ImageNet is an image classification dataset with 1000 classes consisting of a training set of approximately 1 . 3 million images, and a test set of 50000 images. From the 1 . 3 million images of the training set we use the 5% , 10% , 15% , 20% percent (or 64058, 128116, 192174, 256232 examples) as the labeled dataset A where we train the teacher and pretrain the student models. For each size of dataset A , we perform a random split on the remaining training data and use 10000 labeled examples as the validation dataset and the remaining examples as the unlabeled dataset U . We use a ResNet-50 as the teacher, and a ResNet-18 as the student. We compare the methods on soft-distillation. For each trial, we train the student model for 100 epochs

ImageNet:Soft-Distillation

70 -

★

I1

VID

65 -

SLaM (Ours)

Weighted

Test Accuracy (%)

60 -

55

50

45 -

40 -

5%

10%

15%

20%

25%

%08

Percentage of Labeled Examples

and keep the best test accuracy over all epochs. We perform 4 trials and report the average of each method and the variance of the achieved accuracies over the trials. Our results for ImageNet can be found in Table 4. We remark that we do not include the results of the UPS method in Table 4 because it did not improve over the accuracy achieved after pre-training the student model on dataset A . The reader is referred to Appendix D.1 for additional details.

Large Movies Reviews Dataset Here we present results on the Large Movies Reviews Dataset [39]. This is a dataset for binary sentiment classification containing 25000 movie reviews for training and 25000 for testing. We use an ALBERT-large model [32] as a teacher, and an ALBERT-base model as a student. We use 2% , 4% , 8% , 40% percent (or 500, 1000, 2000, 10000 examples) from the training dataset and split the remaining data in a validation dataset of 500 examples and an unlabeled dataset U . Our results and more experimental details can be found in Appendix D.3.

Performance Gains of SLaM as a Function of The Number of Labeled Examples In our experiments, the fraction of examples we consider 'labeled' controls two things at the same time: (i) the accuracy of the teacher model - as the teacher is trained on the labeled examples available; and (ii) the number of unlabeled examples the teacher model provides pseudo-labels for. The more inaccurate the teacher model is, the better the improvements provided by our method. (Given a 'perfect' teacher that never generates incorrect pseudo-labels for the unlabeled examples, our method is mathematically equivalent to the 'vanilla' approach (see the mixing operation in Equation (1)). Therefore, the smaller the number of labeled examples available, the bigger the performance gains of SLaM as (i) the teacher will be less accurate; and (ii) it has to generate labels for more unlabeled examples (and therefore the absolute number of inaccurate predictions that SLaM 'corrects' increases statistically). It is worth emphasizing that the main reason behind the enormous success of distillation is exactly that the teacher network can blow up the size of the student's training dataset: in practice, the ratio of labeled examples to unlabeled examples is typically (much) less than 1%.

5 Distilling Linear Models and Learning Noisy Halfspaces

In this section we show that, when the dataset is separable by a halfspace, i.e., for every example x , the ground-truth is g ( x ) = ( 1 { w ∗ · x > 0 } , 1 { w ∗ · x ≤ 0 } ) for some unknown weight vector w ∗ , then using SLaM with a linear model as the student will recover the ground truth classifier. We make the standard assumption that the ground-truth halfspace has γ -margin, i.e., that ∥ w ∗ ∥ 2 = 1 and that it holds | w ∗ · x | ≥ γ for all examples x . For a fixed example x , the observed noisy teacherlabel y satisfies Definition 3.2, i.e., y = g ( x ) w.p. α ( x ) and y = 1 -g ( x ) w.p. 1 -α ( x ) (since k = 2 for binary classification). Our approach consists of using the standard cross-entropy loss ce( p, q ) and training a student-model consisting of a linear layer plus a soft-max activation, i.e., f ( x ; w ) = ( 1 1+ e -w · x , e -w · x 1+ e -w · x ) .

Theorem 5.1 (SLaM Convergence) . Let X be a distribution on R d and g ( x ) be the ground-truth halfspace with normal vector w ∗ ∈ R d . Let D be the distribution over (noisy) teacher-labeled examples ( x, y ) whose x -marginal is X . Assume that there exist β, γ > 0 such that for all examples x in the support of X it holds that | w ∗ · x | ≥ γ and | 1 / 2 -α ( x ) | ≥ β . Let ϵ > 0 . After T = O (1 / ( β 2 γ 2 ϵ 2 )) SGD iterations on the SLaM objective (see Algorithm 3), with probability at least 99% , there exists an iteration t ≤ T where P x ∼ X [err( f ( x ; w ( t ) ) , g ( x ))] ≤ ϵ .

Remark 5.2 (Learning Halfspaces with RCN) . The problem of learning halfspaces with Random Classification Noise (RCN) can be modeled as having a teacher with constant accuracy probability, i.e., α ( x ) = α > 1 / 2 for all x . As a corollary of Theorem 5.1 we obtain an efficient learning algorithm for γ -margin halfspaces under RCN achieving a sample complexity of O (1 / ( γ 2 ϵ 2 )) . Prior to our work, the best known sample complexity for provably learning halfspaces with RCN was ˜ O (1 / ( γ 4 ϵ 2 )) [19] where the 'backward loss-adjustment' of [13] was used.

6 Conclusion, Limitations, and Broader Impact

In this work we propose SLaM, a novel and principled method for improving distillation with unlabeled examples. We empirically show that SLaM consistently outperforms the baselines, often by a large margin. We also showed that SLaM can be used with and improve (i) knowledge distillation with temperature scaling; (ii) loss functions beyond the standard Cross-Entropy loss; and (iii) confidence-based weighting schemes that down-weight examples where the teacher model is not very confident. Apart from extensive experimental evaluation, we provide strong theoretical guarantees establishing the consistency and optimality of SLaM. As a byproduct of our theoretical analysis, we obtain a new iteration for learning γ -margin halfspaces with RCN that improves the best known sample complexity for this problem.

A limitation of SLaM is that it does not necessarily improve over vanilla distillation when the teacher model makes only a few mistakes (this is to be expected as our method is designed for the case where the teacher-model is imperfect).

Knowledge-distillation is a very popular deep learning method, and therefore, potentially malicious usage of our work is an important societal issue, as deep learning has far-reaching applications from NLP to Robotics and Self-Driving cars.

References

Mart´ ın Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467 , 2016.

Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9163-9171, 2019.

Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D. Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019.

Ehsan Amid, Manfred KK Warmuth, Rohan Anil, and Tomer Koren. Robust bi-tempered logistic loss based on bregman divergences. Advances in Neural Information Processing Systems , 32, 2019.

D. Angluin and P. Laird. Learning from noisy examples. Machine Learning , 2(4):343-370, 1988.

Eric Arazo, Diego Ortego, Paul Albert, Noel E O'Connor, and Kevin McGuinness. Pseudolabeling and confirmation bias in deep semi-supervised learning. In 2020 International Joint Conference on Neural Networks (IJCNN) , pages 1-8. IEEE, 2020.

Noga Bar, Tomer Koren, and Raja Giryes. Multiplicative reweighting for robust neural network optimization. arXiv preprint arXiv:2102.12192 , 2021.

Cenk Baykal, Khoa Trinh, Fotis Iliopoulos, Gaurav Menghani, and Erik Vee. Robust active distillation. International Conference on Learning Representations (ICLR) , 2023.

David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in neural information processing systems , 32, 2019.

A. Blum, A. Frieze, R. Kannan, and S. Vempala. A polynomial-time algorithm for learning noisy linear threshold functions. Algorithmica , 22(1):35-52, 1998.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Cristian Buciluˇ a, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 535-541, 2006.

T. Bylander. Learning linear threshold functions in the presence of classification noise. In Proceedings of the seventh annual conference on Computational learning theory , pages 340-347, 1994.

Hanting Chen, Yunhe Wang, Chang Xu, Chao Xu, and Dacheng Tao. Learning student networks via feature embedding. IEEE Transactions on Neural Networks and Learning Systems , 32(1):2535, 2020.

Liqun Chen, Dong Wang, Zhe Gan, Jingjing Liu, Ricardo Henao, and Lawrence Carin. Wasserstein contrastive representation distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16296-16305, 2021.

Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. Advances in neural information processing systems , 33:22243-22255, 2020.

Mostafa Dehghani, Arash Mehrjou, Stephan Gouws, Jaap Kamps, and Bernhard Sch¨ olkopf. Fidelity-weighted learning. arXiv preprint arXiv:1711.02799 , 2017.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

I. Diakonikolas, T. Gouleakis, and C. Tzamos. Distribution-independent pac learning of halfspaces with massart noise. Advances in Neural Information Processing Systems , 32, 2019.

Lei Feng, Senlin Shu, Zhuoyi Lin, Fengmao Lv, Li Li, and Bo An. Can cross entropy loss be robust to label noise? In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence , pages 2206-2212, 2021.

Benoˆ ıt Fr´ enay and Michel Verleysen. Classification in the presence of label noise: a survey. IEEE transactions on neural networks and learning systems , 25(5):845-869, 2013.

Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International Conference on Machine Learning , pages 16071616. PMLR, 2018.

Dragan Gamberger, Nada Lavrac, and Ciril Groselj. Experiments with noise filtering in a medical domain. In ICML , volume 99, pages 143-151, 1999.

Aritra Ghosh, Himanshu Kumar, and P Shanti Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI conference on artificial intelligence , volume 31, 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. CoRR , abs/1503.02531, 2015.

Fotis Iliopoulos, Vasilis Kontonis, Cenk Baykal, Gaurav Menghani, Khoa Trinh, and Erik Vee. Weighted distillation with unlabeled examples. In NeurIPS , 2022.

Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International Conference on Machine Learning , pages 2304-2313. PMLR, 2018.

Akisato Kimura, Zoubin Ghahramani, Koh Takeuchi, Tomoharu Iwata, and Naonori Ueda. Few-shot learning of neural networks from scratch by pseudo example optimization. arXiv preprint arXiv:1802.03039 , 2018.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Abhishek Kumar and Ehsan Amid. Constrained instance and class reweighting for robust learning under label noise. arXiv preprint arXiv:2111.05428 , 2021.

Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 , 2019.

Hunter Lang, Aravindan Vijayaraghavan, and David Sontag. Training subset selection for weak supervision. arXiv preprint arXiv:2206.02914 , 2022.

Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML , volume 3, page 896, 2013.

Zhaoqi Leng, Mingxing Tan, Chenxi Liu, Ekin Dogus Cubuk, Jay Shi, Shuyang Cheng, and Dragomir Anguelov. Polyloss: A polynomial expansion perspective of classification loss functions. In International Conference on Learning Representations , 2022.

Lu Liu and Robby T Tan. Certainty driven consistency loss on multi-teacher networks for semi-supervised learning. Pattern Recognition , 120:108140, 2021.

Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on pattern analysis and machine intelligence , 38(3):447-461, 2015.

Michal Lukasik, Srinadh Bhojanapalli, Aditya Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? In International Conference on Machine Learning , pages 6448-6458. PMLR, 2020.

Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , pages 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics.

Negin Majidi, Ehsan Amid, Hossein Talebi, and Manfred K Warmuth. Exponentiated gradient reweighting for robust training under label noise and beyond. arXiv preprint arXiv:2104.01493 , 2021.

Rafael M¨ uller, Simon Kornblith, and Geoffrey Hinton. Subclass distillation. arXiv preprint arXiv:2002.03936 , 2020.

Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. Advances in neural information processing systems , 26, 2013.

Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1944-1952, 2017.

Hieu Pham, Zihang Dai, Qizhe Xie, and Quoc V Le. Meta pseudo labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11557-11568, 2021.

Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q Weinberger. Identifying mislabeled data using the area under the margin ranking. Advances in Neural Information Processing Systems , 33:17044-17056, 2020.

Ilija Radosavovic, Piotr Doll´ ar, Ross Girshick, Georgia Gkioxari, and Kaiming He. Data distillation: Towards omni-supervised learning. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4119-4128, 2018.

Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International conference on machine learning , pages 4334-4343. PMLR, 2018.

Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. In International Conference on Learning Representations (ICLR) , 2021.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision , 115(3):211-252, 2015.

Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Systems , 2022.

Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew G Wilson. Does knowledge distillation really work? Advances in Neural Information Processing Systems , 34:6906-6919, 2021.

Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv preprint arXiv:1910.10699 , 2019.

Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10687-10698, 2020.

I Zeki Yalniz, Herv´ e J´ egou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi-supervised learning for image classification. arXiv preprint arXiv:1905.00546 , 2019.

Xiangli Yang, Zixing Song, Irwin King, and Zenglin Xu. A survey on deep semi-supervised learning. IEEE Transactions on Knowledge and Data Engineering , 2022.

Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems , 31, 2018.

Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin Dogus Cubuk, and Quoc Le. Rethinking pre-training and self-training. Advances in neural information processing systems , 33:3833-3845, 2020.

Yang Zou, Zhiding Yu, Xiaofeng Liu, BVK Kumar, and Jinsong Wang. Confidence regularized self-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5982-5991, 2019.

A Notation

For two vectors p, q ∈ R d we denote by p · q = ∑ d i =1 p i q i their inner product. We use p ∗ q to denote their element-wise product, i.e., ( p ∗ q ) i = p i q i . We use the notation max i p to denote the i -th largest element of the vector p . We use margin( p ) to denote the difference between the top-2 elements of p , i.e., margin( p ) = max 1 p -max 2 p . Moreover, we use margin k ( p ) to denote the top-k margin, i.e., margin k ( p ) = ∑ k i =1 max i p -max k +1 p . Given a function f ( w ) : R d ↦→ R we denote by ∂ w f ( w ) the gradient of f with respect to the parameter w .

B Detailed Description of SLaM

B.1 Estimating the Teacher's Accuracy Parameters: α ( x ) , k ( x )

Estimating the Teacher's Accuracy α ( x ) via Isotonic Regression We now turn our attention to the problem of estimating α ( x ) for each x of dataset B , i.e., the dataset labeled by the teacher model. In [27] the authors empirically observed that α ( x ) correlates with metrics of teacher's confidence such as the 'margin', i.e., the difference between the probabilities assigned in the top-1 class and the second largest class according to the teacher's soft label y s . In particular, the larger the margin is the more likely is that the corresponding teacher label is correct. We exploit (and enforce) this monotonicity by employing isotonic regression on a small validation dataset to learn the mapping from the teacher's margin at an example x to the corresponding teacher's accuracy α ( x ) .

To perform this regression task we use a small validation dataset V with correct labels that the teacher has not seen during training. For every example x ∈ V we compute the corresponding soft-teacher label y s ( x ) and compute its margin margin( x ) = max 1 ( y s ( x )) -max 2 ( y s ( x )) . For every x ∈ V we also compute the hard-prediction of the teacher and compare it with the groundtruth, i.e., for every x the covariate and responce pair is (margin( x ) , 1 -err( g ( x ) , y ( x ))) . We then use isotonic regression to fit a piecewise constant, increasing function to the data. Sorting the regression data { (margin( x ) , 1 -err( g ( x ) , y ( x ))) x ∈ V } by increasing margin to obtain a list ( c (1) , . . . , r (1) ) , . . . , ( c ( m ) , r ( m ) ) , isotonic regression solves the following task

where the parameter lb is a lower bound on the values ˆ r ( i ) and is a hyper-parameter that we tune. On the other hand, the upper bound for the values can be set to 1 since we know that the true value α ( x ) is at most 1 for every x (since it corresponds to the probability that the teacher-label is correct). After we compute the values ˆ r (1) , . . . , ˆ r ( m ) for any given c ∈ [0 , 1] the output of the regressor is the value of ˆ r ( i ) corresponding to the smallest c ( i ) that is larger-than or equal to c . This is going to be our estimate for α ( x ) . We remark that finding the values r ( i ) can be done efficiently in O ( n ) time after sorting the data (which has a runtime of O ( n log n ) ) so the whole isotonic regression task can be done very efficiently.

Estimating k ( x ) . We now describe our process for estimating the values of α ( x ) and k ( x ) for every example of dataset B . Similarly to the binary classification setting, we estimate the accuracy probability α ( x ) using isotonic regression on a small validation dataset. The value of k ( x ) can be set to be equal to a fixed value of k for all data, so that the top-k accuracy of the teacher on the validation data is reasonable (say above 60% ). For example, in our ImageNet experiments, we used k = 5 . We also provide a data-dependent method to find different values k ( x ) for every example x . To do this we adapt the method for estimating the top-1 accuracy α ( x ) of the teacher from the validation dataset. For every value of k = 2 , . . . , L -1 we compute the top-k margin of the teacher's predictions on the validation data which is equal to the sum of the top-k probabilities of the teacher soft-label minus the probability assigned to the k +1 -th class, i.e.,

Using the top-k margin as the covariate and the top-k accuracy as the response we solve the corresponding regression task using isotonic regression to obtain the value α k ( x ) representing the probability that the true label belongs in the top-k predictions of the teacher soft-label. For some threshold, say 90% , for every x we set k ( x ) to be the smallest value of k so that α k ( x ) ≥ 90% . We empirically observed that using larger thresholds for the top-k accuracy (e.g., 90% or 95% ), is better. We remark that while using the top-k margin as the covariate in the regression task is reasonable, our method can be used with other 'uncertainty metrics' of the teacher's soft-labels, e.g., the entropy of the distribution of y s ( x ) after grouping together the top-k elements. The higher this entropy metric is the more likely that the top-k accuracy probability α ( x ) k of the teacher is low.

B.2 SLaM for Distillation with Unlabeled Examples: Pseudocode

In this section we present pseudo-code describing the distillation with unlabeled examples setting and the SLaM method, Algorithm 1.

Remark B.1 . We remark that in our experiments, we observed that not normalizing the mixing operation with k ( x ) -1 resulted in better results overall. Therefore, the mixing operation used in our experimental evaluation of SLaM is mix( f ( x ; w ); α ( x ) , k ( x )) = α ( x ) f ( x ; w ) + (1 -α ( x ))(1 -f ( x ; w )) ∗ top( y s ( x ); k ( x )) . For more details we refer the reader to the code provided in the supplementary material.

Algorithm 1 Student Label Mixing (SLaM) Distillation

Input: Labeled Dataset A, Labeled Validation dataset V, Unlabeled Dataset U Output: A trained Student model f ( x ; w )

Train Teacher model on Labeled Dataset A

C SLaM Consistency

In the following proposition we show that any minimizer of the SLaM loss over the noisy teacher-data must agree with the ground-truth for all x (that have positive density). To keep the presentation simple and avoid measurability issues (e.g., considering measure zero sets under X ) in the following we will assume that the example distribution X is supported on a finite set. We remark that one can easily adapt the proof to hold for any distribution X (but the result will hold after excluding measure-zero sets under X ).

Proposition C.1 (SLaM Consistency) . Let D be the distribution of the teacher-labeled examples of dataset B , i.e., we first draw x ∼ X and then label it using the noisy teacher of Definition 3.2. Moreover, assume that there exists some parameter w ∗ ∈ W such that the ground-truth g ( x ) = f ( x ; w ∗ ) . Denote by L SLaM ( w ) = E ( x,y ) ∼ D [ ℓ ( y, mix( f ( x ; w ); α ( x ) , k ( x ))] . the SLaM objective. The following hold true.

1. w ∗ minimizes the SLaM objective.

Algorithm 2 Estimating Teacher's Accuracy Statistics α ( x ) , k ( x )

Input: (Noisy) Teacher Model y s ( x ) , Labeled Validation dataset V,

Isotonic-Regression lower-bound lb ∈ [0 , 1] , and top-k accuracy threshold t ∈ [0 , 1] .

Output: Estimates ˆ α ( x ) , ˆ k ( x ) of the actual α ( x ) , k ( x ) .

Create Soft-labels for the Validation dataset using the teacher model { y s ( x ) : x ∈ V } .

for j = 1 to L -1 do

# Map y s ( x ) to top-j margin and accuracy pairs on the Validation V

Set ˆ α j ( x ) to be the output of Isotonic-Regression with lower-bound lb on the (covariate, responce) pairs in C . # See Appendix B.1

end for

ˆ a ( x ) ← ˆ a 1 ( x )

ˆ a L ( x ) ← 1 # The top-L accuracy is always (trivially) equal to 1

Given example x for some threshold t set ˆ k ( x ) to be the smallest integer r ∈ { 1 , . . . , L } so that a r ( x ) ≥ t .

̸

Assuming further that for all x it holds that α ( x ) k ( x ) = 1 , we have that any minimizer w of the SLaM objective satisfies: f ( x ; w ) = g ( x ) for all x .

Proof. Fix any example x ∈ X . By Definition 3.2 we have that the corresponding teacher label y is correct with probability α ( x ) and a uniformly random incorrect label out of the top-k labels according to the teacher soft-label y s ( x ) . Recall for an L -dimension score vector p , by top( p ; k ) ∈ { 0 , 1 } L we denote the vector that has 1 on the positions of the top-k elements of p , e.g., top((1 , 2 , 3 , 4 , 5); 2) = (0 , 0 , 0 , 1 , 1) . Conditional on x , the corresponding expected noisy teacher label is

̸

̸

̸

̸

We know that the expected teacher label conditional on it being wrong E [ y | y = g ( x ) , x ] is a uniformly random incorrect label from the top-k labels of the corresponding teacher soft-label y s ( x ) . Assume first that k = L , since the ground-truth is represented by a one-hot vector, the distribution of uniformly random incorrect labels conditional on x can be written as (1 -g ( x )) / ( L -1) . For example, if the ground-truth label is g ( x ) = (1 , 0 , 0 , 0 , 0) then a uniformly random incorrect label has probability distribution (0 , 1 / 4 , 1 / 4 , 1 / 4 , 1 / 4) . Assume now that k ( x ) = 3 and top( y s ( x ); 3) = (1 , 1 , 1 , 0 , 0) . Then the distribution of the (incorrect) teacher label becomes (0 , 1 / 2 , 1 / 2 , 0 , 0) . Using ∗ to denote element-wise multiplication of two vectors, we have

̸

Therefore, we obtain

Therefore, by using the fact that Cross-Entropy is linear in its first argument, we obtain that the expected SLaM loss on some example x is

We first have to show that there exist some parameter w ∈ W that matches the (expected) observed labels E [ y | x ] . Observe first that by using the realizability assumption, i.e.,that there exists w ∗ so that f ( x ; w ∗ ) = g ( x ) we obtain that, for every x , it holds mix( g ( x ); α ( x ) , k ( x )) = mix( f ( x ; w ∗ ); α ( x ) , k ( x )) . In fact, by Gibb's inequality (convexity of Cross-Entropy) we have that w ∗ is a (global) minimizer of the SLaM objective.

We next show that any (global) minimizer of the SLaM objective must agree with the ground-truth for every x . Since we have shown that w ∗ is able to match the (expected) labels E [ y | x ] any other minimizer w must also satisfy mix( g ( x ); α ( x ) , k ( x )) = mix( f ( x ; w ); α ( x ) , k ( x ))) . Assume without loss of generality that g 0 = 1 , i.e., the ground-truth label is 0 . We observe that by using that mix( g ( x ; w ); α ( x ) , k ( x )) = α ( x ) g ( x ) + (1 -α ( x )) 1 -g ( x ) k ( x ) -1 ∗ top( y s ( x ); k ( x )) and the fact that the ground-truth belongs in the topk ( x ) of the teacher's predictions conditional that the teacher's top-1 prediction is incorrect (thus top( y s ( x )) 0 = 1 ), we obtain that

Using the fact that g 0 = 1 we can simplify the above expression to

̸

Using the assumption that a ( x ) k ( x ) = 1 we obtain that the term ( α ( x ) -1 -α ( x ) k ( x ) -1 ) is not vanishing and therefore it must hold that f ( x ; w ) 0 = 1 = g 0 , i.e., the student model must be equal to the ground-truth.

D Extended Experimental Evaluation

We implemented all algorithms in Python and used the TensorFlow deep learning library [1]. We ran our experiments on 64 Cloud TPU v4s each with two cores.

D.1 Implementation Details: Vision Datasets

Here we present the implementation details for the vision datasets we considered.

Remark D.1 . We note that in all our experiments, 'VID' corresponds to the implementation of the loss described in equation (2), (4) and (6) of [2] (which requires appropriately modifying the student model so that we have access to its embedding layer).

Experiments on CIFAR-{ 10/100 } and CelebA For the experiments on CIFAR-10/100 and CelebA we use the Adam optimizer with initial learning rate lr = 0 . 001 . We then proceed according to the following learning rate schedule (see, e.g., [25]):

Finally, we use data-augmentation. In particular, we use random horizontal flipping and random width and height translations with width and height factor, respectively, equal to 0 . 1 .

The hyperparameters of each method are optimized as follows. For SLaM we always use 0 . 5 as the lower bound for isotonic regression (i.e., the parameter lb in Algorithm 2). As CelebA is a binary classification benchmark k ( x ) is naturally set to 2 for all examples. For CIFAR-10/10 we used the data-dependent method for estimating k ( x ) (see Algorithm 2) with threshold parameter t = 0 . 9 . For weighted distillation we do a grid search over updating the weights every { 1 , 25 , 50 , 100 , 200 } epochs and we report the best average accuracy achieved. Finally, for VID we search over { 0 . 001 , 0 . 1 , 0 . 2 , 0 . 5 , 0 . 8 , 1 . 0 , 2 . 0 , 10 . 0 , 50 . 0 , 100 . 0 } for the coefficient of the VID-related term of the loss function, and for the Taylor cross-entropy method we optimize its hyperparameter over { 1 . 0 , 2 . 0 , 3 . 0 , 4 . 0 , 5 . 0 , 6 . 0 } .

Experiments on ImageNet For the ImageNet experiments we use SGD with momentum 0 . 9 as the optimizer. For data-augmentation we use random horizontal flipping and random cropping. Finally, the learning rate schedule is as follows. For the first 5 epochs the learning rate lr is increased from

ImageNet:Soft-Distillation

70 -

I1

VID

65 -

Weighted

SLaM (Ours)

Test Accuracy (%)

60 -

55

50

45 -

40 -

5%

10%

15%

20%

25%

30%

Percentage of Labeled Examples

0 . 0 to 0 . 1 linearly. After that, the learning rate changes as follows:

The hyperparameters of each method are optimized as follows. For SLaM we do a hyperparameter search over { 0 . 55 , 0 . 60 , 0 . 65 , 0 . 70 } for the lower bound for isotonic regression, and we keep the best performing value for each potential size of dataset A . We used the fixed value 5 for k ( x ) , as the top-5 accuracy of the teacher model was satisfactory (much higher than its top-1 accuracy) on the validation dataset. For Taylor-CE we did a hyper-parameter search for the Taylor series truncation values in { 1 , 2 , 3 , 4 , 5 , 6 , 10 , 20 , 50 , 80 , 100 } . For weighted distillation we compute the weights in a one-shot fashion using the pre-trained student (as in the ImageNet experiments in [27]). For VID we search over { 0 . 1 , 0 . 3 , 0 . 5 } for the coefficient of the VID-related term of the loss function.

D.2 Hard-Distillation

Here we present results on hard-distillation. The hyper-parameters of all methods are chosen the same way as in our soft-distillation experiments, see Appendix D.1. Tables 5, 6 and 7 contain our results on CIFAR-10, CIFAR-100 and CelebA, respectively. We observe that in almost all cases, SLaM consistently outperforms the other baselines. Moreover, for CIFAR-10 and CIFAR-100 hard-distillation performs worse than soft-distillation (as it is typical the case) but in CelebA harddistillation seems to be performing on par with (sometimes even outperforming) soft-distillation. A plausible explanation for the latter outcome is that in our CelebA experiments the teacher and student have different architectures (MobileNet and ResNet, respectively) so that soft-labels from the teacher are not so informative for the student. (This is also a binary classification task where the information passed from the teacher to the student through its soft-labels is limited.)

CIFAR10: Hard-Distillation

CIFAR100: Hard-Distillation

CelebA: Hard-Distillation

80.0 -

77.5 -

+

UPS

VID

60 -

UPS

VID

94 -

+

UPS

VID

75.0 -

★

SLaM (Ours)

Weighted

Weighted

SLaM (Ours)

*

Weighted

SLaM (Ours)

(%)

Test Accuracy (%)

55

Test Accuracy (%)

- E6

72.5 -

70.0 -

50

- Z6

67.5 -

45 -

91 -

65.0 -

40 -

90 -

62.5 -

10%

15%

20%

25%

30%

35%

10%

15%

20%

25%

30%

35%

2%

3%

4%

5%

6%

7%

Percentage of Labeled Examples

Percentage of Labeled Examples

Percentage of Labeled Examples

D.3 Large Movies Reviews Dataset Results

Here we present the results and the implementation details regarding the experiments on the Large Movies Reviews dataset. Recall that we use an ALBERT-large model as a teacher, and an ALBERTbase model as a student. We also use 2% , 4% , 8% , 40% percent (or 500, 1000, 2000, 10000 examples) from the training dataset and split the remaining data in a validation dataset of 500 examples and an unlabeled dataset U. We compare the methods on the soft-distillation. For each trial we train the student model for 40 epochs and keep the best test accuracy over all epochs. We perform 3 trials and report the average of each method and the variance of the achieved accuracies over the trials. The results of our experiments can be found in Table 8. We remark that we did not implement the UPS method for this dataset as the data-augmentation method for estimating the teacher's accuracy could not be readily used for this NLP dataset. Moreover, using dropout and Monte Carlo estimation for the uncertainty was also not compatible with the Albert model used in this experiment.

Since we are dealing with ALBERT-models (which are already pre-trained), we do not pre-train the student model on dataset A except in the case of 'weighted-distillation' [27], where we pre-train the student model on dataset A just for 1 epoch. The teacher model is trained using the Adam optimizer for 20 epochs with initial learning rate 2 · 10 -5 . The student model is trained also using the Adam optimizer but for 40 epochs and with learning rate 10 -7 .

The hyperparameters of each method are optimized as follows. For SLaM we do a hyperparameter search over { 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 } for the lower bound for isotonic regression, and we keep the best performing value for each potential size of dataset A . As this is a binary classification benchmark we naturally set k ( x ) = 2 for all examples. For weighted distillation we do a grid search over updating the weights every { 1 , 10 , 20 , 40 } epochs and, similarly, we report the best average accuracy achieved. Finally, for VID (recall also Remark D.1) we search over { 0 . 1 , 0 . 5 , 1 . 0 , 2 . 0 } for the coefficient of

CIFAR100:Composability of SLaMwith the FidelityWeighting Scheme

65 -

★

SLaM+Fidelity

★

Fidelity

★

SLaM

60 -

Test Accuracy (%)

55

50

45 -

40-

10%

15%

20%

25%

30%

35%

111111111111111

1202000022020000720200007207

111111111111

Fidelity Hyper-parameter/Size of Labeled Dataset

the VID-related term of the loss function, and for the PolyLoss we opitmize its hyperparameter over {-1 . 0 , -0 . 8 , -0 . 6 , -0 . 4 , -0 . 2 , 0 . 5 , 1 . 0 , 2 . 0 } .

D.4 Combining with Teacher-Uncertainty-Based Reweighting Techniques

As we discussed in Section 2, our method can in principle be combined with teacher-uncertainty filtering and weighting schemes as these can be seen as preprocessing steps. To demonstrate this, we combine our method with the so-called fidelity-based weighting scheme of [17]. The fidelity weighting scheme reweights examples using some uncertainty measure for teacher's labels, e.g., by performing random data-augmentations and estimating the variance of the resulting teacher labels or using dropout and Monte Carlo estimation. More precisely, for every example x in the teacher-labeled dataset B , the fidelity-weighting scheme assigns the weight w Fid ( x ) = exp( -β uncertainty teacher ( x )) for some hyper-parameter β > 0 . In our experiments we performed 10 random data augmentations (random crop and resize), estimated the coordinate-wise variance of the resulting teacher soft-labels, and finally computed the average of the variances of the k -classes, as proposed in [17]. We normalized the above uncertainty of each example by the total uncertainty of the teacher over the whole dataset B . The weights of examples in dataset A are set to 1 and the reweighted objective is optimized over the combination of the datasets A,B .

To demonstrate the composability of our method with such uncertainty-based weighting schemes, we use CIFAR100 and the percentage of the labeled dataset A (as a fraction of the whole training set) is 10% , 15% , 20% , 25% , 30% , 35% , similar to the setting of Section 4.2. The teacher is a ResNet110 and the student is a ResNet56. We first train the student using only the fidelity weighting scheme, i.e., optimize the loss function of Equation (4) using different values for the hyperparameter β ∈ { 0 . 1 , 0 . 2 , 1 . 0 , 1 . 2 , 2 . 0 , 5 . 0 , 10 . 0 , 20 . 0 } , i.e., ranging from mildly reweighting the examples of

CIFAR100:Distillation withTemperatureScaling

65 -

SLaM+TemperatureScaling

Vanilla+TemperatureScaling

60 -

(%)

Test Accuracy

55 -

50 -

45 -

10%

15%

I

20%

25%

I

30%

35%

Percentage of Labeled Examples

dataset B to more agressively 'removing' examples where the teacher's entropy is large. For the same values of β we then train the student using the reweighted SLaM objective:

For the combined SLaM + Fidelity method we did not perform hyper-parameter search and used the same parameters for the isotonic regression as we did in the 'standard' SLaM experiment in CIFAR100 of Appendix D.1. We present our comprehensive results for all sizes of dataset A and values of the hyper-parameter β in Figure 6. Our results show that, regardless of the value of the hyperparameter β and the size of the labeled dataset A, using SLaM together with the fidelity weighting scheme provides consistent improvements. Moreover, in Figure 6, we observe that by using SLaM the achieved accuracy depends less on the hyper-parameter β : since SLaM takes into account the fact that some of the teacher's predictions are incorrect, it is not crucial to down-weight them or filter them out.

D.5 Using Distillation Temperature

In this section we show that our approach can be effectively combined with temperature-scaling [26]. Choosing the right distillation temperature often provides significant improvements. In our setting, the teacher provides much more confident predictions (e.g., soft-labels with high-margin) on dataset A (where the teacher was trained) compared to the teacher soft-labels of dataset B where the teacher is, on average, less confident. Given this observation, it is reasonable to use different distillation temperatures for dataset A and dataset B. We try different temperatures for dataset A and dataset B and perform vanilla distillation with temperature and also consider applying the temperature scaling before applying SLaM. For each size of dataset A we try pairs of temperatures t A , t B ∈ { 0 . 01 , 0 . 1 , 0 . 5 , 0 . 8 , 1 ., 2 ., 5 ., 10 ., 100 . } and report the best accuracy achieved by vanilla distillation and the best achieved by first applying temperature scaling and then SLaM. In Figure 7 we observe that SLaM with temperature scaling consistently improves over vanilla distillation with temperature.

D.6 Using SLaM with other loss functions beyond cross-entropy

In this section, we demonstrate that our method can be successfully applied when the student loss function comes from the families of losses introduced in [20] and [35]. We perform experiments on CIFAR-100 and ImageNet following the setting of Section 4.2. In particular, we compare vanilla distillation with unlabeled examples using the Taylor-CE loss of [20] and the PolyLoss of [35], with combining SLaM with these losses. For the Taylor-CE loss we set the 'degree' hyperparameter to be 2 (as suggested in [20]) and we set the hyperparameter of the PolyLoss to be 2 . 0 (as suggested in [35]). The corresponding results can be found in Figure 8.

CIFAR100: Using SLaM with Taylor CE

CIFAR100: Using SLaM with PolyLoss

ImageNet: Using SLaM with PolyLoss

65

SLaM + Taylor CE

SLaM + PolyLoss

70 -

SLaM + PolyLoss

60 -

60 -

PolyLoss

65 -

PolyLoss

55

Test Accuracy

55

Test Accuracy

60 -

50 -

50 -

55

50 -

45

45

40 -

40 -

45

10%

15%

20%

25%

30%

35%

10%

15%

20%

25%

30%

35%

5%

10%

15%

20%

25%

Percentage of Labeled Examples

Percentage of Labeled Examples

Percentage of Labeled Examples

D.7 Performance of SLaM with (even) fewer labels

In this section we investigate more extensively the effect of the range of the the size of the labeleddataset. In particular, we provide experiments with even fewer labeled examples available. Experiments in which the number of available labeled examples is small are of greater importance since this is the typical scenario where 'distillation with unlabeled examples' applies. For larger dataset sizes, as seen in the plots of the previous experiments, all methods converge to roughly the same performance.

D.8 SLaM Hyper-parameter Ablation

We investigate the effect of the hyper-parameters used in SLaM, see Appendix D.1.

In Table 10 we investigate using different values for the isotonic regression lower-bound parameter, ( lb in Appendix D.1). We observe that SLaM is rather robust to this hyperpameter and usually simply setting lb = 0 . 5 yields good results.

We next compare using a fixed value for the top-k threshold value, k ( x ) versus the data-depenent method described in Appendix D.1. We again observe that SLaM is robust to the value of k used since it outperforms vanilla distillation for reasonable values of k . Overall, we found that using fixed values of k ( x ) (after some hyper-parameter search for k ) and using the data-dependent method yield comparable results. The advantage of the fixed-value method is that it is easier to implement (and slightly more efficient) and the advantage of the data-dependent method is that its hyperparameter (threshold t in Algorithm 2) is easier to tune (in all our experiments t = 0 . 9 achieved good performance).

Labeled Examples

10%

15%

20%

Vanilla

37

.

94

±

0

.

10

46

.

42

±

0

.

24

52

.

17

±

0

.

21

k=2

41

.

25

±

0

.

36

49

.

18

±

0

.

19

54

.

48

±

0

.

25

k=5

40

.

71

±

0

.

29

49

.

41

±

0

.

23

54

.

41

±

0

.

2

k=10

41

.

2

±

0

.

8

49

.

31

±

0

.

12

54

.

42

±

0

.

19

Data-Dependent k(x) (

t

= 0

.

9

)

42

.

7

±

0

.

30

49

.

89

±

0

.

23

54

.

73

±

0

.

27

D.9 Robustness of SLaM to inaccuracies in α ( x ) , k ( x )

We perform a 'controlled' experiment by adding noise to the estimates of α ( x ) and k ( x ) to test the robustness of SLaM to inaccurate predictions of α ( x ) and k ( x ) is a valuable addition to our experimental evaluation. To do this, we start from the oracle values for α ( x ) and k ( x ) , i.e., α ∗ ( x ) = 1 if the teacher prediction is correct on x and 0 if it is incorrect and k ∗ ( x ) is equal to the smallest integer value ℓ so that the ground-truth label is contained in the top ℓ predictions of the teacher. We then introduce random noise to the oracle predictions: for α ( x ) we perform a random perturbation

where ξ is uniformly distributed in [0 , σ ] . Hence, when α ∗ ( x ) = 0 we increase it by adding a random variable ξ ∈ [0 , σ ] and when α ∗ ( x ) = 1 we decrease it by subtracting the same random variable ξ ∈ [0 , σ ] . We clip the resulting value in the interval [0 , 1] . To create noisy values for k ( x ) we simply add a random integer to the optimal value k ∗ ( x ) , i.e., k ( x ) = k ∗ ( x ) + Z , where Z is a random integer in {-ℓ, . . . , ℓ } . We clip the resulting value of k ( x ) in the range { 0 , . . . , numClasses } .

We test our method on CIFAR100 and observe that the accuracy of SLaM gracefully decays as the predictions for α ( x ) and k ( x ) become worse (bottom right corner is the noisiest σ = 0 . 5 , ℓ = 90 and top left is the noiseless σ = 0 , ℓ = 0 ), see Table 12.

D.10 Different Regression Algorithms for α ( x ) , k ( x )

In our main experimental evaluation of SLaM, we chose isotonic regression to enforce the monotonicity in the learned accuracy estimates for α ( x ) based on the empirical observation that α ( x ) is often approximately monotone as a function of the margin of the teacher. Moreover, the lower threshold (denoted by lb in Appendix D.1) in isotonic regression gives us a way to control how 'agressive' the mixing operation is going to be. That said, SLaM does not hinge on some particular regression method and other methods can be used. We investigate using different regression methods (knn, Logistic regression) for estimating α ( x ) , k ( x ) with SLaM. In Table 13 and Table 14, we see that Isotonic regression typically outperfoms other methods. Moreover, SLaM provides consistent improvements regardless of regression method used.

D.11 Validation dataset size ablation

In this section we investigate the effect of the size of the validation dataset required by SLaM. As we have already showed in our previous experiments, SLaM requires only rough estimates of α ( x ) and k ( x ) and thus even very small validation datasets suffice. We observe that SLaM is able to provide improvements even with very small validation datasets (e.g., with 128 labels).

In Table 15 we use different validation sizes for the CIFAR-100 experiment described in our manuscript and and show that the performance of SLaM improves when the validation dataset is larger but the gaps are not very significant especially for larger sizes of the labeled dataset. We show that SLaM is able to provide improvements even with very small validation datasets (e.g., with 128 labels).

E Distilling Linear Models and Learning Noisy Halfspaces

In this section we state and prove our convergence result for the SLaM method when applied to linear models. Our assumption is that the ground-truth g ( x ) corresponds to a halfspace, i.e., g ( x ) = ( 1 { w ∗ · x > 0 } , 1 { w ∗ · x ≤ 0 } ) for some unknown weight vector w ∗ . We show that using SLaM with a linear model as the student will recover the ground truth classifier. We make the standard assumption that the ground-truth halfspace has γ -margin, i.e., that ∥ w ∗ ∥ 2 = 1 and that it holds | w ∗ · x | ≥ γ for all examples x . For a fixed example x , the observed noisy teacher-label y satisfies Definition 3.2, i.e., y = g ( x ) w.p. α ( x ) and y = 1 -g ( x ) w.p. 1 -α ( x ) (since k = 2 for binary classification). Our approach consists of using the standard cross-entropy loss ce( p, q ) and training a student-model consisting of a linear layer plus a soft-max activation, i.e.,

Recall, that for binary classification, we define the mixing operation as

Theorem E.1 (Student Label Mixing Convergence) . Let X be a distribution on R d and g ( x ) be the ground-truth halfspace with normal vector w ∗ ∈ R d . Let D be the distribution over (noisy) teacher-labeled examples ( x, y ) whose x -marginal is X . We denote by α ( x ) the probability that the teacher label y ∈ [0 , 1] 2 is correct, i.e., α ( x ) = P ( x,y ) ∼ D [argmax( y ) = g ( x ) | x ] . Assume that there exist β, γ > 0 such that for all examples x in the support of X it holds that | w ∗ · x | ≥ γ and | 1 / 2 -α ( x ) | ≤ β . Let ϵ > 0 . After T = O (1 / ( β 2 γ 2 ϵ 2 )) iterations of SLaM (Algorithm 3), with probability at least 99% , there exists an iteration t ≤ T where P x ∼ X [err( f ( x ; w ( t ) ) , g ( x ))] ≤ ϵ .

Remark E.2 (High-Probability Result) . We remark that even though our learner succeeds with constant probability (at least %99 ) we can amplify its success probability to 1 -δ by standard amplification techniques (i.e., by repeating the algorithm O (log(1 /δ )) times and keeping the best result). To achieve success probability 1 -δ the total sample complexity is O (log(1 /δ ) / ( ϵ 2 γ 2 β 2 )) .

Proof. We first provide simplified expressions for the gradient of the SLaM objective and the update vectors λ ( t ) g ( t ) used in Algorithm 3. In what follows we remark that for any binary classification

Algorithm 3 SLaM for Linear Models

(0)

model f ( x ; w ) = ( f 0 ( x ; w ) , f 1 ( x ; w )) we have the following identities: (i) (mix( f ( x ; w ); α ( x ))) 0 = mix( f 0 ( x ; w ); α ( x )) , where to simplify notation we overload the mixing operation to also act on the scalar f 0 ( x ; w ) , i.e., mix( f 0 ( x ; w ); α ( x )) = α ( x ) f 0 ( x ; w ) + (1 -α ( x ))(1 -f 0 ( x ; w )) ; and (ii) f 1 ( x ; w ) = 1 -f 0 ( x ; w ) .

Lemma E.3 (SLaM Gradient) . The gradient of the SLaM objective is equal to

∂ w ce( y, mix( f ( x ; w ); α ( x )) = r ( f 0 ( x ; w ); α ( x )) sgn(2 α ( x ) -1) ((mix( f 0 ( x ; w ); α ( x )) -y 0 ) x, where

Let L ( x ; w ) = E ( x,y ) ∼ D [ce( y, mix( f ( x ; w ) , α ( x )) | x ] be the expected student label mixing loss conditional on some example x ∈ R d . It holds ∂ w L ( x ; w ) = r ( f ( x ; w ) , α ( x )) | 2 α ( x ) -1 | ( f 0 ( x ; w ) -g 0 ( x )) x.

Proof. We first show the formula

Using the chain rule, we obtain

Now we observe that that for binary classification, it holds that y 1 = 1 -y 0 , mix( f 1 ( x ; w ); α ( x )) = 1 -mix( f 0 ( x ; w ); α ( x )) , and therefore, also ∂ w mix( f ( x ; w ); α ( x )) 1 ) = -∂ w mix( f ( x ; w ); α ( x )) 0 ) to obtain the simplified expression:

Further simplifying the above expression, we obtain:

Using again the chain rule we obtain that

∂ w (mix( f 0 ( x ; w ); α ( x )) = α ( x ) ∂ w ( f 0 ( x ; w ))+(1 -α ( x )) ∂ w (1 -f 0 ( x ; w )) = (2 α ( x ) -1) ∂ w f 0 ( x ; w ) .

Using the fact that the derivative of the sigmoid function r ( t ) = 1 / (1 + e -t ) , is r ′ ( t ) = e -t / (1 -e -t ) 2 = r ( t )(1 -r ( t )) , and the chain rule, we obtain that ∂ w f 0 ( x ; w ) = f 0 ( x ; w )(1 -f 0 ( x ; w )) x . Putting everything together we obtain the claimed formula for ∂ w ce( y, mix( f ( x ; w ); α ( x ))) .

To obtain the gradient formula for the expected loss conditional on some fixed example x , we can use the fact that ∂ w E [ce( y, mix( f ( x ; w ); α ( x ))) | x ] = E [ ∂ w ce( y, mix( f ( x ; w ); α ( x ))) | x ] . Now using the formula of Equation (5) and the fact that E [ y 0 | x ] = mix( g 0 ( x ); α ( x )) by the definition of our noise model, we obtain that

We first show the following claim proving that after roughly T = 1 / ( β 2 γ 2 ϵ 2 ) gradient iterations the student parameter vector w ( t ) will have good correlation with the ground-truth vector w ∗ .

Claim 1 . Fix any T larger than a sufficiently large constant multiple of log(1 /δ ) / ( ϵ 2 γ 2 β 2 ) , and assume that for all t ≤ T it holds that P x ∼ X [err( f ( x ; w ( t ) ) , g ( x ))] > ϵ . Then, we have w ( T ) · w ∗ = Ω( βγϵ ) T , with probability at least 1 -δ .

̸

Proof. Denote by u ( t ) = -λ ( t ) g ( t ) the update vector used in Algorithm 3. We observe that the weight vector at round T is equal to w ( T ) = ∑ T t =1 u ( t ) . In what follows we denote by F ( t ) the filtration corresponding to the randomness of the updates of Algorithm 3. We define the martingale q ( T ) = ∑ T t =1 ( E [ u ( t ) | F ( t -1) ] -u ( t ) ) with q (0) = 0 . We first show that under the assumption that P x ∼ X [argmax( f ( x ; w ( t ) )) = g ( x )] > ϵ , for all t ≤ T , it holds that ∑ T t =1 E [ u ( t ) | F ( t -1) ] · w ∗ ≥ ( ϵγβ/ 2) T . Using the SLaM gradient expression of Lemma E.3 and the definition of the step size λ ( t ) we obtain that E [ u ( t ) | F ( t -1) ] = E x ∼ X [ | 2 α ( x ) -1 | ( g 0 ( x ) -f 0 ( x ; w ( t -1) )) x ] . Take any step t . We have that

where we used the fact that ( g 0 ( x ) -f 0 ( x ; w ( t -1) )) sgn( x · w ∗ ) = | g 0 ( x ) -f 0 ( x ; w ( t -1) ) | . Now, using the γ -margin assumption of the distribution D and the fact that | 2 α ( x ) -1 | ≥ β we obtain

where for the penultimate inequality we used the fact that when g ( x ) and f ( x ; w ( t -1) ) disagree it holds that | g 0 ( x ) -f 0 ( x ; w ( t -1) ) | ≥ 1 / 2 . Take, for example, the case where g 0 ( x ) = 1 . Then f 0 ( x ; w ( t -1) ) must be smaller than 1 / 2 otherwise the prediction of the model argmax f ( x ; w ( t -1) ) would also be 0 (and would agree with the prediction of g ( x ) ). Finally, for the last inequality we used the fact that, by our assumption, it holds that P x ∼ X [err( g ( x ) , f ( x ; w ( t -1) ))] ≥ ϵ . Therefore, we conclude that ∑ T t =1 E [ u ( t ) | F ( t -1) ] · w ∗ ≥ ( ϵγβ/ 2) T . Next, we shall show that w ( T ) also achieves good correlation with the optimal direction w ∗ with high probability. We will use the fact that q ( t ) is a martingale and the Azuma-Hoeffding inequality to show that w ( T ) · w ∗ will not be very far from its expectation.

Lemma E.4 (Azuma-Hoeffding) . Let ξ ( t ) be a martingale with bounded increments, i.e., | ξ ( t ) -ξ ( t -1) | ≤ M . It holds that P [ ξ ( T ) ≥ ξ (0) + λ ] ≤ e -λ 2 / (2 M 2 T ) .

Recall that from Lemma E.3 we have that E [ u ( t ) | F ( t -1) ] = E x ∼ X [ | 2 α ( x ) -1 | ( g 0 ( x ) -f 0 ( x ; w ( t -1) )) x ] and

Observe that since ∥ x ∥ 2 ≤ 1 for all x it holds that ∥ u ( t ) ∥ 2 ≤ 1 . Therefore, the difference ∥ E [ u ( t ) | F ( t -1) ] -u ( t ) ∥ ≤ 2 with probability 1 . Since ∥ w ∗ ∥ 2 = 1 , using Cauchy-Schwarz, we also obtain that | E [ u ( t ) · w ∗ | F ( t -1) ] -u ( t ) · w ∗ | ≤ 2 .

Using Lemma E.4, and the fact that q (0) = 0 we obtain that P [ q ( t ) · w ∗ ≥ ( βγϵ/ 4) T ] ≤ e -β 2 γ 2 ϵ 2 T/ 128 . Therefore we conclude that for any T larger than 128 log(1 /δ ) / ( β 2 γ 2 ϵ 2 ) , with probability at least 1 -δ , it holds that q ( T ) · w ∗ ≥ ( βγϵ/ 4) T or equivalently w ( T ) · w ∗ ≥ ( βγϵ/ 4) T , where we used our previously obtained bound for the expected updates ∑ T t =1 E [ u ( t ) | F ( t -1) ] · w ∗ ≥ ( βγϵ/ 2) T .

Claim 2 . Fix any T ≥ 1 . Then, we have ∥ w ( T ) ∥ 2 = O ( √ T ) , with probability at least 99% .

Proof. We have that ∥ w ( T ) ∥ 2 2 = ∥ w ( T -1) ∥ 2 2 +2 u ( T ) · w ( T -1) + ∥ u ( T ) ∥ 2 2 . Unrolling the iteration, we obtain that

where we used the fact that, since ∥ x ( t ) ∥ 2 ≤ 1 , it holds that ∥ u ( t ) ∥ 2 ≤ 1 (see the proof of Claim 1). We first show that ∑ T t =1 E [ u ( t ) | F ( t -1) ] · w ( t -1) = O ( T ) . We have

We will show that for x it holds that

Fix some x and let s = w ( t -1) · x . Assume first that g 0 ( x ) = 1 . Then, we have

where we used the fact that s e -s 1+ e -s ≤ 0 for s ≤ 0 and s e -s 1+ e -s ≤ se -s ≤ 1 /e for s ≥ 0 (using the elementary inequality ze -z ≤ 1 /e for all z ∈ R ). When g 0 ( x ) = 0 we similarly have that

where we used the fact that when s ≥ 0 it holds that -s 1+ e -s ≤ 0 and when s ≤ 0 , -s 1+ e -s ≤ -s/e -s = -se s . For the final inequality, we used again the inequality ze -z ≤ 1 /e for all z ∈ R (where we replaced z with -s ).

Therefore, we obtain that E [ u ( t ) | F ( t -1) ] · w ( t -1) ≤ 1 /e and ∑ T t =1 E [ u ( t ) | F ( t -1) ] · w ( t -1) ≤ T/e . Using the decomposition of Equation (6), linearity of expectation, and the tower rule for conditional expectations, we conclude that E [ ∥ w ( T ) ∥ 2 2 ] ≤ (2 /e +1) T . Using Markov's inequality we obtain that with probability at least 99% it holds that ∥ w ( T ) ∥ 2 2 = O ( T ) or equivalently ∥ w ( T ) ∥ 2 = O ( √ T ) .

We can now finish the proof of Theorem 5.1. Assume, in order to reach a contradiction, that for all t ≤ T it holds that P x ∼ X [err( f ( x ; w ( t ) ) , g ( x ))] > ϵ . Now picking T to be larger than a sufficiently large constant multiple of 1 / ( ϵ 2 γ 2 β 2 ) and using Claim 1 and Claim 2 we obtain that,

10

2.5

2.0

10

-5

-10

10

10

with probability at least 99% , it holds that w ( T ) · w ∗ / ∥ w ( T ) ∥ 2 ≥ Ω( βγϵ √ T ) , which can be made to be larger than 1 by our choice of T . However, this is a contradiction as by Cauchy-Schwarz we have w ( T ) · w ∗ / ∥ w ( T ) ∥ 2 ≤ ∥ w ∗ ∥ 2 ≤ 1 . Therefore, with probability at least 99% , it must be that for some t ≤ T it holds that P x ∼ X [err( f ( x ; w ( t ) ) , g ( x ))] ≤ ϵ .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to systematically utilize the noise of teacher predictions in a knowledge distillation setting comprised of both labeled and unlabeled samples. The proposed method, SLaM linearly mixes the student predictions with the noise of the teacher predictions, thereby hoping that the student will learn the noise-free true underlying label. The method is supported by both theoretical results and empirical experiments on different datasets.\n\nSTRENGTHS:\n- The paper is well-written, easy to follow, and nicely structured.\n- The proposed method, SLaM is relatively simple and general which is a nice feature.\n- The method is supported by both theoretical and empirical results.\n\nWEAKNESSES:\nI am generally positive about the paper, but my biggest concern is the empirical evaluation of the methods. I am open to changing my score if the following is addressed.\n\nComments and questions:\n- All empirical results presented are based on \"[...] the best test accuracy over all epochs.\". I.e. the methods are continuously evaluated on the test set (effectively making it a validation set), and no real test set performance is available - this is bad practice. Thus, I am concerned whether the reported scores are actually representative of the real test performance of the methods, and thereby if one can, in fact, rely on the conclusions drawn. I would strongly suggest reporting something like the test performance at the last epoch to avoid overfitting the test set. And even so, the experiments might be biased towards having evaluated the methods on the test set previously, so ideally evaluations on completely independent test sets would be included.\n- In e.g. L230 SLaM is reported to \"[...] consistently outperform the baselines, often by a large margin\". I believe this is too bold a claim. In particular, only 7 of 24 experiments yield improvements over $1\\%$ compared to the second-best baseline, while 8 other of 24 have improvements $<0.2\\%$ of which 3 have improvements $<0.02\\%$. Furthermore, there are 2 of 24 experiments where the method does not improve over the baselines. All of these observations are without consideration of the variance, which likely makes some additional improvements statistically indistinguishable. I would suggest reducing the claims slightly, especially given the evaluation protocol.\n- It is unclear why this method shouldn't be considered a semi-supervised learning method, as both labeled and unlabeled samples are provided and used for the distillation procedure. In fact, the availability of a labeled validations set, $V$ appears to be a necessity for this method, and I believe it should be presented as a semi-supervised setting more explicitly.\n- A lot of results and details are deferred to the appendix. While I fully understand the limitations a 9-page restriction can put on a paper, I believe some of the space in the paper could be better utilized to present at least some key results from the appendix.\n\nMinor comments:\n- L148: \"agree\" -> \"disagree\"\n- L166: \"etting\" -> \"setting\"\n- L311: \"inFigure 2\" -> \"in Figure 2\"\n- Figure 1 is never referenced in the paper\n- Figure 2 is badly formatted; i.e. too small a font and hard to distinguish the results/lines of different methods (consider rescaling the y-axis to e.g. improvement over baseline)\n\nQUESTIONS:\nSee the above.\n\nLIMITATIONS:\nSufficiently addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces SLaM, a method for knowledge distillation using unlabeled examples. Its main contribution lies in the development of an efficient and data-agnostic approach that enhances student model performance in scenarios with limited labeled data but abundant unlabeled data. SLaM accomplishes this by incorporating information from both labeled and unlabeled examples during the distillation process. The paper extensively presents experimental results showcasing SLaM's superiority over previous methods on standard benchmarks. Furthermore, the authors provide theoretical guarantees and insights into SLaM's effectiveness. Overall, the paper delivers a valuable contribution to the field of knowledge distillation, showcasing a novel approach supported by empirical evidence.\n\nSTRENGTHS:\n1. The paper introduces a novel approach to knowledge distillation with unlabeled examples that leverages the information from both labeled and unlabeled examples during the distillation process.\n\n2. The experimental results show the effectiveness of the proposed method in knowledge distillation.\n\n3. Theoretical results further guarantee the effectiveness of the proposed method.\n\nWEAKNESSES:\nThe proposed method argues that existing methods suffer from the shortcoming of noisy pseudo-label, but it also seems hard to guarantee the proposed method can get the pseudo-labels without noisy. Moreover, the proposed method can be limited to semi-supervised learning problem.\n\nQUESTIONS:\n1. Can the proposed method ensure that the pseudo-labels generated by SLaM are noise-free, thereby addressing the limitation discussed in the paper? How and why?\n\n2. Although not specifically designed for knowledge distillation, the baseline method Taylor-CE shows competitive or even superior performance on celebA even with only 2% labeled data. Can you elaborate on the reasoning behind this, especially highlighting the advantages of the proposed method in dealing with sparsely labeled data, which Taylor-CE lacks?\n\n3. It is better to plot a figure to demonstrate the performance of the comparison method in different percents of the labeled training data with a large span. Eg: 5%, 10%, 30%, 50%, 70%.\n\n4. The baseline method Taylor-CE, which focuses on noise label problem, also achieve competitive or even superior performance. I wonder if the proposed method can work in noise label learning that also need reliable pseudo-labels? For example, on clothing1m dataset.\n\nLIMITATIONS:\nPlease refer to weakness and question.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn their paper, the authors propose a knowledge distillation procedure that augments pseudo-labels predicted by a teacher models by its uncertainty, leading to a uncertainty-aware student learning routine. This is realized by a novel loss formulation that mixes student predictions with quantified uncertainty associated with the pseudo-labeling prediction. Together with convincing empirical results demonstrating improvements in terms of generalization performance, theoretical results characterize the convergence of the proposed solution.\n\nSTRENGTHS:\n- Strong empirical results against recent baselines\n- Profound theoretical results\n- Well-written manuscript with a clear and valid motivation, addressing a relevant problem\n- The simplicity of the approach is appealing, promoting the applicability of this method\n\nWEAKNESSES:\nMajor:\n\n- While the empirical evaluation itself is certainly convincing from a generalization point of view, a more rigorous ablation study of the influence of the uncertainty quantification would be insightful. Namely, it is hard to judge how sensible this procedure is towards a good quality of the isotonic regression solution. One possibility to address this concern could be by obtaining $\\alpha(x)$ and $k(x)$ from an oracle with “perfect” estimations, and gradually contaminating these estimates. Then, one could see how the performance evolves.\n- Relating to the previous comment, I am wondering why exactly isotonic regression is chosen as a means to quantify $\\alpha(x)$. I see little explanations on why exactly this method is chosen, and not others, like conformal prediction. Of course, it is completely valid to make an arbitrary design choice that works well, but I would love to see a more thorough elaboration on the motivation of this specific construction. An ablation study as described before could be used to support this decision.\n\nMinor:\n\n- The figures are relatively small and hard to read, enlarging them would be appreciated\n- The orthography could be improved, many missing commas (e.g., when starting sentences like “In Section xyz, …”)\n- The references are inconsistent (e.g., the conference identifiers: NeurIPS, ICML, … vs. International Conference on Learning Representations)\n\nQUESTIONS:\n-\n\nLIMITATIONS:\nI do not see too much about limitations, this could certainly be deepened.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper attempts to resolve the problem of noisy teacher labels for unlabeled examples in knowledge distillation. Proposed Student-Label Mixing (SLaM) to introduce noise into student prediction when matching to teacher’s  pseudo-label on unlabeled data. Experiments on standard benchmarks were used to demonstrate the effectiveness of the proposed method. Also, the paper provided theoretical guarantees for SLaM for learning noisy halfspaces.\n\nrating updated after author rebuttal.\n\nSTRENGTHS:\nThe paper studied the critical problem about how to learn unlabeled examples for the student model with knowledge distillation when the teacher model is not perfect.\n\nThe paper proposed to resolve it from a novel angle by introducing noise to the student model’s prediction, and also provided a theoretical guarantee for the proposed solution.\n\nExtensive experiments based on common datasets and multiple baselines were provided to demonstrate its effectiveness.\n\nWEAKNESSES:\nThe paper lacks the explanation on the intuition of the mixing solution on formula (1), where there is no clarification why it is essential for the second term, i.e. the element-wise multiplication term.\n\nEven though the paper developed a solution to consider the noise for student model prediction, it seems not robust enough to learn \\alpha(x) and k(x). As it requires an additional validation dataset, it is easy to make \\alpha overfit to those data. Also it has another strong assumption that \\alpha is ​​isotonic regression. All of these make the learning of \\alpha easily overfit the data which cannot be robust enough, which also make the solution not that elegant.\n\nIt lacks the result details about \\alpha and k setup for each experiment. Also it lacks the related ablation study to demonstrate how \\alpha and k affect the model performance.\n\nno comparison with a baseline where the validation dataset is used for vanilla distillation, and on top that add vanilla unlabeled distillation or other state of the art unlabeled distillation.\n\nQUESTIONS:\nThe proposed algorithm has the additional cost of needing a validation dataset, seems a fair baseline would be to incorporate the validation set for vanilla distillation, and on top that add vanilla unlabeled distillation or other state of the art unlabeled distillation? \n\nWhat is the intuition of the second term on formula (1)?\n\nHow important is the design \\alpha(x) and k(x) is? How does the size of the validation set affect the overall performance?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}