Weighted Distillation with Unlabeled Examples
Authors: Fotis Iliopoulos
Vasilis Kontonis
Cenk Baykal
Gaurav Menghani
Khoa Trinh
Erik Vee
Venue: NeurIPS 2022 Accept (2022)
Keywords: distillation
importance reweighting

Abstract: Distillation with unlabeled examples is a popular and powerful method for training deep neural networks in settings where the amount of labeled data is limited: A large “teacher” neural network is trained on the labeled data available, and then it is used to generate labels on an unlabeled dataset (typically much larger in size). These labels are then utilized to train the smaller “student” model which will actually be deployed. Naturally, the success of the approach depends on the quality of the teacher’s labels, since the student could be confused if trained on inaccurate data. This paper proposes a principled approach for addressing this issue based on a “debiasing" reweighting of the student’s loss function tailored to the distillation training paradigm. Our method is hyper-parameter free, data-agnostic, and simple to implement. We demonstrate significant improvements on popular academic datasets and we accompany our results with a theoretical analysis which rigorously justifies the performance of our method in certain settings.


Weighted Distillation with Unlabeled Examples
Fotis Iliopoulos
Vasilis Kontonis Google Research kontoni crtttomss@goggle.com
Cenk Baykal Google Research
Google Research fot Notisi@ee ootiss@@oogle.coo
baykalc@google... com
Gaurav Menghani
Khoa Trinh
Erik Vee Google Research e erikvee@google com
Google Research
Google Research khoatrinh@@google.. com
gmenghani&googgle com
Abstract
Distillation with unlabeled examples is popular and powerful method for training deep neural networks in settings where the amount of labeled data is limited: A large "teacher" neural network trained the labeled data available, and then it is used to generate labels on an unlabeled dataset (typically much larger in size). These labels then utilized to train the smaller "student" model which will actually be deployed. Naturally, the success of the approach depends on the quality of the teacher's labels, since the student could be confused if trained inaccurate data. This paper proposes a principled approach for addressing this issue based on "debiasing" reweighting of the student's 's loss function tailored to the distillation training paradigm. Our method is typer-parameter free, data- aag--sgnostic, and simple implement. We demonstrate significant improvements on popular academic datasets and we accompany our results with a theoretical analysis which rigorously justifies the performance of our method in certain settings.
Introduction
In many modern applications of deep neural networks, where the amount of labeled examples for training is limited, distillation with unlabeled examples [8, 20] has been enormously successful. In this two-stage training paradigm larger and more sophisticated "teacher model"-- typically non-deployable for the purposes of the application - is trained learn from the limited amount of available training data in the first stage. In the second stage, the teacher model is used to generate labels on an unlabeled dataset, which is usually much larger in size than the original dataset the teacher itself was trained on. These labels are then utilized to train the "student model", namely the model which will actually be deployed. Notably, distillation with unlabeled examples is the most commonly used training-paradign paradigm in applications where one finetunes and distills from very large-scale foundational models such as BERT [13] and GPT-3 [5] and, additionally, it can be used to significantly improve distillation supervised examples only (see e.g. [47]).
While this has proven be a very powerful approach in practice, its success depends on the quality of labels provided by the teacher model. Indeed, often times the teacher model generates inaccurate labels on non non-neginn neglligiee portion of the unlabeled dataset, confusing the student. As deep neural networks are susceptible to overfitting to corrupted labels [50], training the student on the teacher's s generalization performance.
noisy labels can lead to degradation depicts instance based on CIFAR-10 where filtering out the teacher's noisy labels is quite beneficial for the student's performance.
an example, Figure
We address this shortcoming by introducing a natural "noise model" for the teacher which allows us modify the student's loss function in principled way in order produce unbiased estimate
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

of the student's s clean objective. From a practical standpoint, this produces a fully ardug-and-play method which adds minimal implementation overhead, and which composable with essentially every other known distillation technique.
The idea of compressing a teacher model into smaller student model by matching the predictions of the teacher was initially introduced by Buciluă, Caruana and Niculescu-Mizil [6] and, since then, variations of this method [20, 26, 31, 35, 39, 42, 49] have been applied in a wide variety of contexts [37, 48, 51]. (Notably, some of these applications go beyond compression - see e.g. [8, 47, 48, 52] for reference.) In the simplest form of the method [26], and using classification as a canonical example, the labels produced by the teacher are one- hot vectors that represent the class which has the maximum predicted probability - this method is often referred to as "hard" -istillaation. More generally. Hinton et. al. [8, 20] have shown that it is often beneficial to train the student so that it minimizes the cross-entropy (or KI.-divergence) with the probability distribution produced by the teacher while also potentially using a temperature higher than in the softmax of both models poott-dditilllation) (Temperatures higher than are used in order to emphasize the difference between the probabilities of the classes with lower likelihood of corresponding to the correct label according to the teacher model.)
The main contribution of this work is a principled method for improving distillation with unlabeled examples by reweighting the loss function of the student. That is, we assign importance weights to the examples labeled by the teacher so that each weight reflects (i) how likely it is that the teacher has made an inaccurate prediction regarding the label of the example and (ii) how "distorted" the unweighted loss function we use to train the student is (measured with respect to using the ground- truth label for the example instead of the teacher's 's label). More concretely, our reweighting strategy is based on introducing and analyzing a certain noise model designed to capture the behavior of the teacher in distillation. In this setting, we are able to come up with a closed-form solution for weights that "de-noise" the objective in the sense that (on expectation) they simulate having access to clean labels. Crucially, we empirically observe that the key characteristics of our noise model for the teacher can be effectively estimated through small validation dataset, since in practice the teacher's noise is neither random nor adversarial, and it typically correlates well with its "confidence" " see e.g. Figure 1. In particular, we use the validation dataset to learn map that takes as input the teacher's s and student's confidence for the label of a certain example and outputs estimates for the quantities mentioned in items (i) and (ii) above. Finally, we plug in these estimates to our closed-form solution for the weights so that, overall, we obtain an automated way of computing the student's reweighted objective in practice. A detailed description of our method can be found in Section 2.3.
Our main findings and contributions can be summarized as follows:
We propose a principled and hyperparameter-ffee reweighting method for knowledge distillation with unlabeled examples. The method is efficient, data-agnostic and simple to implement.
We present extensive experimental results which show that our method provides significant improvements when evaluated in standard benchmarks.
Our reweighting technique comes with provable guarantees: (i) it is information enformatio-thhorrttialll theoretically optimal; and (ii) under standard assumptions SGD optimization of the reweighted objective learns solution with nearly optimal generalization.
1.1 Related work
Fidelity vs Generalization in knowledge distillation. Conceptually, our work is related to the paper of Stanton et al. [43] where the main message is that "good student accuracy does not imply good distillation fidelity". i.e.. that more closely matching the teacher does not necessarily lead to better student generalization. In particular, [43] demonstrates that when it comes to enlarging the distillation dataset beyond the teacher training data, there is a trade-off between optimization complexity and distillation data quality. Our work can be seen as a principled way of improving this trade-off.
Advanced distillation techniques. Since the original paper of Hinton et. al. [20], there have been several follow-up works [2, 7,1, 45] which develop advanced distillation techniques which aim to enforce greater consistency between the teacher and the student (typically in the context of distillation on labeled examples). These methods enforce consistency not only between the teacher's 's predictions
2

---

0.65
Sort-Das llatoo: 7500 uboled oxamples, 40500 unabeled examples
0.60
100 Accurcy (86)
72
0.50
0.45
70
02
0.25
Crucke
Conventional distilarion
0.30
Teacher
Figure 1: Left: Performance comparison between student trained using conventional distillation with unlabeled examples (orange) and a student trained only on the examples which are labeled correctly by the teacher (blue). Here assume access to 7500 labeled examples and 40500 unlabeled examples of CIFAR-10. 10. The teacher is a MobileNet of depth multiplier 2, while the student is a MobileNet of depth multiplier 1. Right: Plot of teacher's accuracy function of the margin score.
and student's predictions, but also between the representations learned by the two models. However, in the context of distillation with unlabeled examples, forcing the student match the teacher's inaccurate predictions is still harmful, and therefore weighting the corresponding loss functions via our method is still applicable and beneficial. We demonstrate this fact by considering instances the Variational Information Distillation for Knowledge Transfer (VID) [2] framework, and showing how it is indeed beneficial to combine with our method in Section 3.1.3. We also show that our approach provides benefits top of any improvement obtained via temperature-scaling in Appendix B.2.
Learning with noisy data techniques. As we have already discussed, the main conceptual contribution of our work is viewing the teacher model in distillation with unlabeled examples as the source of stochastic noise with certain characteristics. Naturally, one may wonder what is the relationship between our method and works from the vast literature of learning with noisy data (e.g. [4, 14, 16, 23, 25, 27, 29, 33, 36, 38]. The answer that our method does not attempt to solve the generic "learning with noisy data" problem, which is a highly non-trivial task in the absence of assumptions (both in theory and in practice). Instead, our contribution to observe and exploit the fact that the noise introduced by the teacher in distillation has structure, as it correlates with several metrics of confidence such as the margin-score, the entropy of the predictions etc. We use and quantify this empirical observation to our advantage in order to formulate principled method for developing a debiasing reweighting scheme (an approach inspired by the "learning with noisy which comes with theoretical guarantees. An additional difference is that works from the learning with noisy data literature typically assume that the training dataset consists of corrupted hard labels, and often times their proposed method is not compatible with coft-distillation (in the sense that they degrade its performance, making it much less effective) - see c.g. [32] for a study of this phenomenon in the case of label smoothing [28, 44).
Uncertainty-based weighting schemes. Related to our method are approaches for semi-supervised learning where examples are downweighted or filtered out when the teacher is "uncertain." (A plethora of ways for defining and measuring "uncertainty" have been proposed the literature, e.g. entropy, margin-score, dropout variance etc.) These methods are independent of the student model (they only depend on the teacher model), and so they cannot hope to remove the bias from the student's ss loss function. In fact, these methods can be viewed as preprocessing steps that can be combined with the student- trainitiining process we propose. To demonstrate this we both combine and compare method to the fidelity-based weighting scheme of [12] in Section 3.3.
1.2 Organization of the paper.
In Section 2, we present our method in detail, while Section 3 we present our key experimental results. In Section 4, we discuss the theoretical aspects of our work. In Section 5, we summarize our results, we discuss the benefits and limitations of our method, and future work, Finally, we present extended experimental and theoretical results in the Appendix.
3

---

2 Weighted distillation with unlabeled examples
In this section we present method. In Section 2.1, we review multiclass classification. In Section 2.2, we introduce the noise model for the teacher in distillation, which motivates our approach. And finally, in Section 2.3, we describe our method in detail.
2.1 Multiclass classification
In multiclass classification, we are given a training sample (( yı), (I22, ....... (In yn)) drawn from F where IP is an unknown distribution over instances X' and labels y [7.] 21,2.... L}. Our goal is to learn a predictor f X' - Rt, namely to minimize the risk of f. The
latter is defined the expected loss of f:
R(f) E[y,,, f(x)))
(I)
where (a, y) is drawn from F. and & [L] x R + R+ is loss function such that, for a label Y E [L] and prediction vector f(x) E RR,, Eyy. f()) is the loss incurred for predicting f(z) when the true label is y. The most common way to approximate the risk of predictor f is via the so-called empirical risk:
Rs(f) = S
Σ
Ely, f(x)).
(2)
(s.
That is, given a hypothesis class of predictors F, our goal is typically to find minjer Rs(f) as way to estimate min R(f).
2.2 Debiasing weights
As we have already discussed, the main drawback of distillation with unlabeled examples is essentially that the empirical risk minimizer corresponding the dataset labeled by the teacher cannot be trusted, since the teacher may generate inaccurate labels. To quantify this phenomenon and guide our algorithmic design, we consider the following simple but natural noise model for the teacher.
Let X be an unknown distribution over instances X. We assume the existence of a ground-truth classifier that each E X' is associated with ground-truth label ftrue()) E [4] [ 21, L}. In other words, a clean labeled example is of the form (x, ftrue()) ~ P (and x ~ X). Additionally, consider stochastic adversary that given instance E X, outputs "corrupted" label Vadv (x) p(z). Let denote
with probability p(z), and the ground- tround-truth label furee (z) with probability the induced adversarial distribution over instances and labels.
not hard to see that the empirical risk with respect a predictor f and sample from not an unbiased estimator of the risk
R(S) = = [x])
see Proposition 2.1. On the other hand, the following weighted empirical risk (5) is indeed an unbiased estimator of R()). For each let
wy(x) 1 p(x) (distortion/ (z) - -1)' where distortion = E(Yaav (1r), f(x))
(4)
(2),
and define
R =
Σ (ENIEE
wy ( f(y,f))),,,
(5)
where S = (:)
D" Observe that the weight for each instance r depends on (i) how likely it the adversary corrupts its label; and on (ii) how this corrupted label "distorts" the loss we observe at instance æ. In the following proposition we establish that the standard (unweighted) empirical risk with respect to distribution and predictor f a biased estimator of the risk of under the clean distribution P. while the weighted empirical risk (5) is an unbiased one.
Proposition 2.1 (Debiasing Weights). Let = ((x,, ~ TD be sample from the adversarial distribution. Defining Bfas(f) = Ez-x (p(x) (distortion (x) - -1) {(ftrue(z) fx))] we have: E R (f) R()) vs "standard"
"reweighted"
[[Rs(f)] R(f) + Bias(f).

---

Notice that, as expected, the bias of the unweighted predictor is function of the "power" of the adversary, i.e., a function of how often they can corrupt the label of an instance. and the "distortion" this corruption causes to the loss we observe. The proof of Proposition 2.1 follows from simple, direct calculations and it can be found Appendix D.3.
Intuitively, given a sufficiently large sample S ~ DD*, optimizing an unbiased estimator for the risk should provide a better approximation for minser R(f) compared to optimizing an estimator with constant bias. We formalize this intuition in Section 4 and in Appendix D.
2.3 Our method
We consider the standard setting for distillation with unlabeled examples where we are given a dataset S₁ = [x ) of m labeled examples from an unknown distribution IP, and dataset
S. { i=++1 of n. unlabeled examples typically, 2 m. We also assume the existence of (small) clean validation dataset S, = {( ye) N=m+ttt of size q. Finally, let é I2 x R + R., be a loss function that takes as input two vectors over the set of labels [L]. We describe our method below and more formally in Algorithm 1 in Appendix A.
Remark 2.1. The only assumption we need to make about the validation set S. is that it is not in the train set of the teacher model. That is, set S.e can be used the train set of the student model if needed - we chose to present Sy as completely independent hold out set to make our presentation as conceptually clear as possible.
Training the teacher. The teacher model is trained on dataset St, and then it is used to generate labels for the instances in Su. The labels can be one-hot vectors or probability distributions on [L], depending on whether we apply "hard" or "soft" distillation, respectively.
Training the student. We start by pretraining the student model on dataset Sy. Then, the idea is to think of the teacher model as the source of noise in the setting of Section 2.2, compute a weight wy(at (1) for each example x based on (4), and finally train the student on the union of labeled and teacher-labeled examples by minimizing the weighted empirical risk (examples from St are assigned unit weight).
We point out two remarks. First, in order to apply (4) to compute the weight of an example x, we need have estimates of p(x) and distortion, (z). To obtain these estimates we use the validation dataset S. as we describe in the next paragraph. Second, observe that, according to (4), the weight of an example is function of the predictor, namely the parameters of the model in the case of neural networks. This means that ideally we should be updating our weights assignment every time the parameters of the student model get updated during training. However, we empirically observe that even estimating the weight assignment only once during the whole training process (right after training the student model on S₂) suffices for our purposes, and so our method adds minimal overhead to the standard training process. More generally, the fact that our process of computing the weights is simple and incxpensive allows us to recompute them during training (say every few epochs) to improve our approximation of the theoretically optimal weights. We explore the effect of updating the weights during training in Section 3.2.
Estimating the weights. We estimate the weights using the Nearest Neighbors method on S. to learn a map that takes as input the teacher's and student's "confidence" for the label of certain example a, and outputs estimates for p(x:) and distortion (ar) so we can apply (4). In our experiments we measure the confidence of a model either via the margin-score, i.e., the difference between the largest two predicted classes for the label of a given example (see e.g. the so-called "margin" uncertainty sampling variant [40]), or via the entropy of its prediction. However, one could use any metric (not necessarily confidence) that correlates well with the accuracy of the corresponding models.
More coneretely, we reduce the task of estimating the weights to two-dimensional (i.c., two inputs and two outputs) regression task over the validation set which we solve using the Nearest Neighbors method. In particular, our Nearest Neighbor data structure is constructed as follows. Each example of the validation set is assigned the two following pairs of points: (i) (teacher confidence at z, student
confidence at x)- - this the covariate of the regression task; (ii) (1, distortion at (a:) if the teacher correctly predicts the label of xr or (0, distortion at 2). if the teacher does not correctly predict the
label of x - this is the response of the regression task. The query corresponding to an unlabeled example z' is of the form (teacher confidence at x', student confidence at x'). The Nearest Neighbors
5

---

data structure returns the average response over the & closest in euclidean distance pairs (teacher confidence at x, student confidence at æ) the validation set. The value of lo is specified in the next paragraph. The pseudocode for our method can be found in Algorithm 2 in Appendix A.
We point out two remarks. First, the number i of neighbors we use for our weights-
is always & = v This is because choosing k = 0(q2/(2+dmm) where q is the size of the 2
validation dataset and dim is the dimension of the underlying metric space (dim = 2 in our case), is asymptotically optimal, and 1/2 is a popular choice for the hidden constant used in practice, see e.g. [9, 18]. Second, notice that (4) implies that the weight of an example could be larger than 1 if (and only if) the corresponding distortion value (4) at that example is less than 1. This could happen for example if both the student and teacher have the same (or very similar) inaccurate prediction for certain example. In such a case, the value of the weight in (4) informs us that the loss at this example should be larger than the (low) value the unweighted loss function suggests. However, since we do not have the ground-trutt label for point during training - but only the inaccurate prediction of the teacher-having having weight larger than in this case would most likely guide our student model fit an inaccurate label. To avoid this phenomenon, we always project our weights onto the [o, 1] interval (Line 16 of Algorithm 2). In Appendix D.2 we discuss an additional reason why is beneficial to project the weights of examples of low distortion onto [o, 1] based on a MSE analysis.
Experimental results
In this section we present our experimental results. In Section 3.1 we consider an experimental setup according to which the weights are estimated only once during the whole training process. In Section 3.2 we study the effect of updating the weights during training. Finally, in Section 3.3 we demonstrate that our method can be combined with uncertainty- weighting techniques by both combining and comparing our method to the fidelity- -baeed weighting scheme of [12].
Spt.Ceelluttn:: 7533 libild 63757 uniabeler cxampic
Seft Disi Butoor: 1552 lubker crampic. 40500 unldeeted cramppo
Sal Didillaiio can 2002) sbeled euargtint, 2543 starlid
800
Ruc we' accencc
55
Sun 2922 arion Kumbbr irbele examgrrs es Accuracy
Soft Cecal stone Number of lebeed examples Accuracy
Salt Bonaan Humbor of totileed examples Accuncy
50.0
Welghted 10unt
Welghted 10un)
mee Meiightd (Cum))
10.5
60.0
94.
500
2030
12993
14300
1500 D800 20C09
2030
12000
15300
1900E
35000
20300
25330
38593
Mamber
Mamber lubeld cRaTpiic
Namoor (abeoa
SVHN
CIFAR-10
CelebA
Figure 2: The student's ' test accuracy over the training trajectory (first row) and student's s best test- accuracy achieved over all (second row) when applying one-shot estimation of the weights. The teacher model is a MobileNet with depth multiplier 2. In the cases of SVHN and CIFAR-10 the student model is MobileNet of depth multiplier 1, and in the case of CelebA is ResNet-11. Our approach leads to consistently better models in terms of test-accuaacy and convergence speed. Shaded regions correspond to values within one standard deviation of the mean.
3.1 Improvements via one-shot estimation of the weights
Here we show how applying our reweighting scheme provides consistent improvements on several popular benchmarks even when the weights are estimated only once during the whole training process. We compare our method against conventional distillation with unlabeled examples, but we also show that our method can provide benefits when combined with more advanced distillation techniques such the framework of [2] (see Figure 9).

---

Soft- Dett-Deeiliition: Number labeled examples Accuracy
Soft Distilation: Number of labeled examoles Accuracy
$0.3
19
76
85.0
74
Weighted
94.0
70
Ricelyybeseed weighting Composition
10000
15000
20000
25000
30020
2000
10530
13000
15500
18000
Numbier of labaled snamples
Number of labeled usarpples
CelebA
CIFAR-10
Figure 3: Left: We compare conventional distillation (unweighted conventional), standard Variational Information Distillation (VID) [2] (unweighted VID), and reweighting the conventional loss function (weighted conventional) and the VID loss function with our method (weighted VID) on CelebA. (These results correspond to one-shot estimation of the weights.) We see that our method can be combined effectively with more advanced distillation techniques such as VID. Right: We combine and compare our method with the -daeedty-based reweighting scheme of [12] on CIFAR-IO. We see that our method can be combined effectively with weighting schemes based only on the teacher's 's uncertainty. (These results correspond updating our weights at the end of every epoch.)
More concretely, we evaluate our method on benchmark vision datasets. We compare our method to conventional distillation with unlabeled examples both in terms of the best test accuracy achieved by the student, and in terms of convergence speed (see Figure 2). We also evaluate the comparative advantage of our method as a function of the number of labeled examples available (size of dataset St). We always choose the temperature in the softmax of the models to be 1 for simplicity and consistency, and our metric for confidence is always the margin-score. Implementation details for our experiments and additional results can be found in Appendices and C. We implemented all algorithms in Python making use of the TensorFlow deep learning library [1]. We use 64 Cloud TPU v4s each with two cores.
3.1.1 Experimental setup
Our experiments are of the following form. The academic dataset we use each time is first split into two parts A and B. Part A, which is typically smaller, is used as the labeled dataset S where the teacher model is trained on (recall the setting we described in Section 2.3). Part B is randomly split again into two parts which represent the unlabeled dataset S., and validation dataset S.,, respectively. Then, (i) the teacher and student models are trained once on the labeled dataset Sy; (ii) the teacher model is used to generate soft-labels for the unlabeled dataset S.y (iii) we train the student model on the union of Se and S,, using our method and conventional distillation with unlabeled examples. We repeat step (iii) a number of times: each trial we partition part B randomly and independently, and then the student model is trained using the (student-) wweghhts reached after completing the training on dataset Sy in step (i) as initialization, both for our method and the competing approaches.
3.1.2 CIFAR-(10, 100) and SVHN experiments
SVHN [34] an image classification dataset where the task is to classify street view numbers (10 classes). The train set of SVHN contains 73257 labeled images and its test set contains 26032 images. We use a MobileNet [21 with depth multiplier as the teacher, and a MobileNet with depth multiplier 1 as the student'. The tables in Figure 4 contain the results of our experiments (averages over 3 trials). In each experiment we use the first N E (7500, 10000, 12500, 15000, 1 17500, 20000) examples as the labeled dataset Se, and then the rest 73257 N images are randomly split to labeled validation dataset S, of size 2000, and an unlabeled dataset S, of size 71257 N.
CIFAR-10 and CIFAR- 100 [24] are image classification datasets with 10 and 100 classes respectively. Each of them consists of 60000 labeled images, which we split to a training set of 50000 images, and a test set of 10000 images. For CIFAR-10, we use Mobilenet with depth multiplier 2 as the teacher,
Note that we see the student outperform the teacher here and in other experiments, as can often happen with distillation with unlabeled examples, particularly when the teacher is trained on limited data.
7

---

and a Mobilenet with depth multiplier as the student. For CIFAR-100, we use ResNet- 10 as teacher, and a ResNet-56 as the student. We use validation set of 2000 examples. The results of our experiments (averages over 3 trials) can be found in the tables of Figures 5, 6.
Labeled Examples
7500
10000
12500
15000
17500
Toucher
%7
88.5%
88.45%
91.38%
91.32%
Weighted (Ouns
90. 41 2 0.12%
91.21 1 O. 0.09%
92.4 1 O 121%
92.64 6. 0.14%
82.85 0.1%
Unweightd
0.00%
0.11%
91.90
0.1%
92.31 0.07%
92.45 0.07%
Figure 4: Experiments on SVHN. See Section 3.1.2 for details.
Labeled Examples
7500
10000
12500
15000
17500
Teacher
67.17%
71.39%
74.69%
77%
78.46%
Weighted (Ours)
69.01 4 0 25%
72. 96
0.1%
75.33 1 O 0.43%
77.69 =: 0. 0.35%
78.34 1 0.16%
Unweighted
0.22%
72.48
0.26%
74.95 + 0.2%
76.85 + 0.32%
77.92
0.067
Figure 5: Experiments on CIFAR-10. See Section 3.1.2 for details.
Labeled Exammiles
7500
10000
12500
15000
17500
Teacher
44.1%
51.31%
65.54%
69.06%
02.17%
Weighted (Ours)
46.81 4 U 0.34%
53.31
E 0.60%
57.5 +: 0.3%
60. 94 =: 0. 3.3%
62.86 4 0. 0.41%
Unweighted
0.04%
62.83 + 0.53%
+ 0.63%
60.73 + 0.03%
0.09%
Figure 6: Experiments on CIFAR-100. See Section 3.1.2 for details.
3.1.3 CelebA experiments: considering more advanced distillation techniques
As we have already discussed in Section 1,1, our method can be combined with more advanced distillation techniques which aim to enforce greater consistency between the teacher and the student. We demonstrate this fact by implementing the method of Variational Information Distillation for Knowledge Transfer (VID) [2] and showing how it is indeed beneficial to combine it with our method. We chose the gender binary classification task of CelebA [15] as our benchmark (see details in the next paragraph), because it is known (see e.g. [31]) that the more advanced distillation techniques tend to be more effective when applied to classification tasks with few classes. In the table below, "Unweighted VID" corresponds to the implementation of loss described in equations (2), (4) and (6) of [2], and "Weighted VID" corresponds to the reweighting of the latter loss using our method.
CelebA [15] is a large-scale face attributes dataset with more than 200000 celebrity images, each with forty attribute annotations. Here consider the binary male/female classification task. The train set of CelebA contains 162770 images and its test set contains 19962 images. We use a MobileNet with depth multiplier 2 as the teacher, and a ResNet-11 [19] as the student. The tables in Figure 7 contain the results of our experiments (averages over 3 trials). In each experiment we use the first N {10000, 15000, 20000, 25000, 30000, 35000) examples as the labeled dataset Se, and then the rest 162770- - N images are randomly split to labeled validation dataset S. of size 2000, and an unlabeled dataset S, of size 160770 N.
Labeled Examples
10000
15000
20000
25000
30000
Jeacher
51.59%
96.
94. 41%
94.86%
94.92%%
Weighted VID
94.35
0.11%
95.01
0.17%
95.73
0 0.0%%
93.89
0.08%
11 1: 0.08% %
Weigltted Conventional
94.06
0.04%
96.
0.1%
95.00
0.03%
95.86
0.08%
95. 92
0.01%
Unweighned VID [2]
94.11
1 1.11%%
94.76
2 1.14%
95.46
= 0.11%
95.69
4 0.05%
93. 88
1: 0.03%
Lawogllted Conventicaal
93.68
0.01%
94.92
0.02%
95.38
0.07%
= 0.05%
95.73
0.09%
Figure 7: Experiments on CelebA. See Section 3.1.3 for details and also Figure 3.
3.1.4 ImageNet experiments
ImageNet [41] is a large-scale image classification dataset with 1000 classes consisting of approximately I ~ 1.3M images. For our experiments, we use a ResNet-50 as the teacher, and a ResNet-18 as the student.
each experiment we use the first N E (64055, 128116) labeled examples (5% and 10% of 7, respectively) as the labeled dataset Sr, and the rest I - N examples are randomly split to a labeled validation dataset S, of size 10000, and an unlabeled dataset S, of size I !-N - 10000. The results of our experiments (averages over 10 trials) can be found in Figure 8
8

---

Labeled Exsuugles
3% of ImageNett
10% of ImageNet
Labeled Excampies
3% of ImageNet
10% or Imagehet
Teacher (sot))
38.74%
51.88%
Teacher (hand)
36. 74%
51.68%
Weighted (Ours)
60 = 0.07%
53.59 = 0.09%
Wrighted (Cums)
36.56 0.07%
53.34 Z 0.11%
Unweighted
38.44 1 0.08%
53.43 1 0.08%
Unweighitd
38. 18.42 0.08%
58.18 d: 0.07%
Figure 8: Experiments with soft-distillation (left) and hard-distillation (right) on ImageNet.
3.2 Updating the weights during training
In this section we consider the effect of updating our estimation of the optimal weights during training. For each dataset we consider, the experimental setup is identical to the corresponding setting in Section 3.1, except for that we always use a validation set of size 500 and the entropy of model's prediction as the metric for its confidence. We note that the time required for computing the weight for each example is insignificant compared to total training time (less than I % of the total training time in all of our experiments), which allows us to conduct experiments in which update our estimation at the end of every epoch. We see that doing this typically significantly improves the resulting student's ' performance (however, in CIFAR- 100 we do not observe substantial benefits).
SVt (1-51 biton Mamber ebeled maaisiee es Accurccy
Sunt (NSI bnton Number sbeled exanstes Accuracy
Silt Diss loban: Number of Eslieted exarplees vo Accunccy
06.00
apech
16.00
09.25
94.75
$2.5
94.2
96.00
30000
1500
2005
23030
33900
1530
30300
12300
15300
175CE
1552
1000S
32500
17390
CelebA
CIFAR-10
CIFAR-100
Figure 9: The effect of updating the weights during training. We compare our method when (i) weights are estimated only once and; (ii) when we update our estimation at the end of every epoch.
3.3 Combining and comparing with uncertainty- weighting schemes
As we have already discussed in Section 1.1, our method can be combined with uncertainty- weighting schemes as these are independent of the student model and, therefore, they can be seen as preprocessing step (modifying the loss function) before applying our method. We demonstrate this by combining and comparing our method with the filelit--bseed weighting scheme of [12] on CIFAR-10 10 and CIFAR-100. Details about our experimental setup can be found in Appendix C.3.
Laboled Examples
7500
10000
12500
15000
17500
Teacher (sof))
67.55%
72.85%
74.85%
77.63%
78.40%
Method
70.59
0.06%
74.50
0.07%
76.75 + 0.32%
78.66 = 0.14%
79.21 = 0.17%
Tideliy-bssed weightngg (12)
69.31 2 0.41%
73.39 2 0.44%
74,69 1 0.31%
77.10 2 1.6
76.19 2 1.1
Composiion
71.20 1 0.097%
75.54
1.221%
76.86 86.80 0.29%
79.55
0.22%
80.28
0.28%
Labeled lxtamples
10000
12300
15000
17600
Teacher (sol))
44. 70%
51.45%
56.00%
58, 70%
61.8%%
Method
0.46%
53.15
0.16%
0.4%
59.66
0.4%
02.02 + 0.26%
Fideliyy based we weighting [12]
46.32 1: 0.23%
52.92
0.20%
37.40
0.4%%
59.45
0.17%
61.39 1: 0.09%
Compostioon
46.62 1 0.44%
$3.39
0.16%
37.11 1.41%
59.63
0.15%
62.30 1 0.26%
Figure 10: Combining and comparing our method with the weighting scheme of [12] on CIFAR-10 (top table) and CIFAR-100 (bottom table). See also Figure 3.
4 Theoretical motivation
In this section we show that our debiasing reweighting method described Section 2.2 comes with provable statistical and optimization guarantees. We first show that the method is statistically consistent in the sense that, with a sufficiently large dataset, the reweighted risk converges to the true or "clean" risk. We show (see Appendix D) the following convergence guarantec.

---

Theorem 4.1 (Uniform Convergence of Reweighted Risk), Assume that S is a dataset of i.i.d. "noisy" samples from Under standard capacity assumptions for the class of models and regularity assumptions for the loss ((.) for every J E F it holds that
lim ( R(J) and
lim Rs(J) = R()) + BBBass))
18|+00
To prove our optimization guarantees we analyze the reweighted objective in the fundamental case where the model f(;; ②) is linear, i.e., [(x; θ) = Or IR= and the loss fly, z) is convex in z for every y. In this case, the composition of the loss and the model f f2; θ) is convex
function of the parameter @ E LRd Recall that we denote by (ar R2d + IR the ground truth classifier and by P the "clean" distribution, i.e., sample from P has the form (x. ftrue(:)) where x drawn from distribution X supported on subset of) Rd Finally, we denote by ID the "noisy" labeled distribution on I_ x I and assume that the z-marginal of ID is also X.
We next give general definition of debiasing weight functions, i.e., weighting mechanisms that make the corresponding objective function an unbiased estimator of the clean objective R(O) for every parameter vector è E Rdd Recall that the weight function defined in Section 2.2 is debiasing.
Definition 4.2 (Debiasing Weights), We say that a weight function w(x, yadv; Q) is a debiasing weight if it holds that
Rd (+) (. Yadvi () -
Since the loss convex in O, one could try to optimize the naive objective that does not reweight and simply minimizes E(-) over the noisy examples, (e) - >ore.aaa>>D(((aadd Oz)] We show (unsurprisingly) that doing this is a bad idea: there are instances where optimizing the naive objective
produces classifiers with bad generalization error over clean examples. For the formal statement and proof we refer the reader to Appendix E.
SGD on the naive objective (-aive..) learns parameters with arbitrarily bad generalization error over the "clean" " data.
Our main theoretical insight is that optimizing linear models with the reweighted loss leads to parameters with almost optimal generalization.
Given debiasing weight function w(-), SGD the reweighted objective R²(-)) learns parameter with almost optimal generalization error over the "clean" " data.
The main issue with optimizing the reweighted objective is that, in general, we have no guarantees that the weight function preserves its convexity (recall that it depends on the parameter +). However, we know that its population version corresponds to the clean objective R((-) which is a convex objective. We show that we can use the convexity of the underlying clean objective to show results for stochastic gradient descent, by proving the following key structural property.
Proposition 4.3 (Stationary Points of the Reweighted Objective Suffice (Informal)). Let be dataset of poly (dl./c) i.i.d. samples from the noisy distribution D. Let θ be any stationary point of the weighted objective Rg (O) constrained the unit ball (with respect to the Frobenious norm || p). Then, with probability at least 99%, it holds that
R())= < min R(O)= +e. 15l
Conclusion
We propose a principled reweighting scheme for distillation with unlabeled examples. Our method is hyper-paramete free, adds minimal implementation overhead, and comes with theoretical guarantees. We evaluated our method on standard benchmarks and we showed that consistently provides significant improvements. We note that investigating improved data-driven ways of estimating the weights (4) could be of interest, since potential inaccurate estimation of the weights is the main limitation of our work. We leave this question open for future work.
10

---

References
[1] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems, arXiv preprint 10.3.600.400 2016.
[2] Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9163-9171, 2019.
[3] Martin Anthony and Peter Bartlett. Neural network learning: Theoretical foundations. cambridge university press, 1999.
[4] Noga Bar, Tomer Koren, and Raja Giryes. Multiplicative reweighting for robust neural network optimization. arXiv preprint arXiv:2102.12/92, 2021.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1190 2020.
[6] Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 535-541, 2006.
[7] Liqun Chen, Dong Wang. Zhe Gan, Jingjing Liu, Ricardo Henao, and Lawrence Carin. Wasserstein contrastive representation distillation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16296-16308 2021.
[8] Ting Chen, Simon Korbllihh, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi- supr-sspereiied learners. Advances in neural information processing systems, 33:22243-222 22255,, 2020.
[9] Thomas Cover and Peter Hart. Nearest neighbor pattern classification. IEEE transactions information theory, 13(1):21-27, 1967.
[10] Sanjoy Dasgupta, Adam Tauman Kalai, and Claire Montelconi. Analysis of perceptron-bac based active learning. COLT'05, 05, page 249-263. Berlin, Heidelberg, 2005. Springer-Verlag.
[11] Damek Davis and Dmitriy Drusvyatskiy. Stochastic subgradient method converges at the rate o(k-1/1) on weakly convex functions. arXiv preprint arXiv: /802002988, 2018.
[12] Mostafa Dehghani, Arash Mehrjou, Stephan Gouws, Jaap Kamps, and Bernhard Schölkopf. Fidehity-weighted learning. arXiv preprint
2017.
[13] Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0488 2018.
[14] Benoît Frénay and Michel Verleysen. Classification in the presence of label noise: a survey. IEEE transactions neural networks and learning systems, 25(5):845-866 2013.
[15] Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International Conference on Machine Learning, pages 1607- 1616. PMLR, 2.18.
[16] Dragan Gamberger, Nada Lavrac, and Ciril Groselj. Experiments with noise filtering in medical domain. In /CML, volume 99, pages 143-151, 1999.
[17] Ying Guo, Peter I. Bartlett, John Shawe- Shawe-Taylor, and Robert C Williamson. Covering numbers for support vector machines. In Proceedings of the twelfth annual conference on Computational learning theory, pages 267-277, 1999.
[18) Trevor Hastie, Robert Tibshirani, Jerome Friedman, and Jerome H Friedman. The elements of statistical learning: data mining. inference, and prediction, volume 2. Springer, 2009.
11

---

[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[20] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. CoRR, abs/1503.0255 02530,251,12 2015.
[21] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 04861, 2017.
[22] Prateek Jain, Dheeraj M. Nagaraj, and Praneeth Netrapalli. Making the last iterate of sgd information theoretically optimal. SIAM Journal on Optimization, 31(2):1108-1133 108-1130, 2021.
[23] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International Conference Muchine Learning, pages 2304-2313, PMLR, 2018.
[24] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[25] Abhishek Kumar and Ehsan Amid. Constrained instance and class reweighting for robust learning under label noise. arXiv preprint orXiv:2//1.05424 2021.
[26] Dong-Hyun Lee et al. Pseudo-label: The simple and efficient sumi-supe learning method for deep neural networks. In Workshop on challenges in representation learning, ICMI., volume 3, page 896, 2013.
[27] Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on pattern analysis and machine intelligence, 38(3):447-461, 2015.
[28] Michal Lukasik, Srinadh Bhojanapalli, Aditya Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? In International Conference on Machine Learning, pages 6448 -458.. PMLR, 2020.
[29] Negin Majidi, Ehsan Amid, Hossein Talebi, and Manfred K Warmuth. Exponentiated gradient reweighting for robust training under label noise and beyond. arXiv preprint arXiv:2 2021.
[30] Andreas Maurer and Massimiliano Pontil. Empirical bernstein bounds and sample variance penalization. arXiv preprint x:Xivv0907.37400, 2009.
[31] Rafael Müller, Simon Korbblith, and Geoffrey Hinton. Subclass distillation. arXiv preprint MrXiv:2002.03936 2020.
[32] Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When does label smoothing help? Advances in neural information processing systems, 32, 2019.
[33] Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. Advances neural information processing systems, 26, 2013.
[34] Yuval Netzer, Tao Wang. Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[35] Hieu Pham, Zihang Dai, Qizhe Xie, and Quoc V Le. Meta pseudo labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11557-11568.
2021.
[36] Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q Weinberger. Identifying mislabeled data using the area under the margin ranking. Advances in Neural Information Processing $ystems, 33:17044-117556 2020.
[37] Ilija Radosavovic, Piotr Dollár, Ross Girshick, Georgia Gkioxari, and Kaiming He. Data distillation: Towards ompi-super learning. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4119-4128, 2018.
12

---

[38] Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International conference on machine learning, pages 4334 4343. PMLR, 2018.
[39] Ellen Riloff. Automatically generating extraction patterns from untagged text. In Proceedings of the national conference on artificial intelligence, pages 1044-1049, 1996.
[40] Dan Roth and Kevin Small. Margin- based active learning for structured output spaces, In European Conference Machine Learning, pages 413-424. Springer, 2006.
[41] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-25 2015.
[42] Henry Scudder. Probability of error of some adaptive pattern-recogniition machines. IEEE Transactions on Information Theory, 11(3)363-377, 1965.
[43] Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew G Wilson. Does knowledge distillation really work? Advances in Neural Information Processing Systems, 14:6906-6919, 2021.
[44] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826, 2016.
[45] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv preprint arXiv: 19.0. 10699, 2019.
[46] Roman Vershynin. High-dimensional probabiliy: An introduction with applications data science, volume 47. Cambridge university press, 2018.
[47] Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10687-10698, 2020.
[48] I Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi-sut semi-supervised learning for image classification. arXiv preprint 1rXXiv:/905.005.00 2019.
[49] David Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. In 33rd annual meeting of the association for computational linguistics, pages 189-196, 1995.
[50] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107
115, 2021.
[51] Barret Zoph, Golnaz Ghiasi, Tsung- Yi Lin, Yin Cui, Hanxiao Liu, Ekin Dogus Cubuk, and Quoc Le. Rethinking pre-training and self-training, Advances in neural information processing systems, 33:3833-3845. 2020.
[52] Yang Zou, Zhiding Yu, Xiaofeng Liu, BVK Kumar, and Jinsong Wang. Confidence regularized self-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5982-5991, 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See Section 3 for an empirical evaluation of our method. See Section 4 for theoretical justification for the performance of our approach.
(b) Did you deseribe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 5
13

---

(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results....
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See theoretical results in Section and the Supplementary Material.
(b) Did you include complete proofs of all theoretical results? [Yes] We provide complete
proofs of all theoretical results in the Supplementary Material.
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 3 and the Supplementary Material.
(c) Did you report error bars (e.g., with respect the random seed after running experiments multiple times)? [Yes] All plots in Section 3 contain shaded regions that correspond the values within one standard deviation of the mean.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Sec. 3
4. If you are using existing assets (c.g., code, data, models) or curating/relcasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section and the Supplementury Material.
(b) Did you mention the license of the assets? [Yes] See the Supplementary Material.
(C) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you'l 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "Z75DP7x0gGm", "reviewer": "Reviewer_C5ER", "summary": "The authors propose a knowledge distillation technique for semi supervised settings. For the labeled data, they uniformly assign equal weights for the cross entropy loss between the predicted output and the supervision. On the other hand, the proposed method distributes adaptive weights for each data, which is theoretically and rigorously justified. The experimental results also support their theoretical analysis.\n", "questions": "1. It is hard to understand k-NN(U, Query) in the algorithm part 1. Could you explain it in detail? I can not find the definition of the  function.  ", "limitations": "The authors adequately addressed the limitations and potential negative social impact of their work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. The method is well-motivated and theoretically justified. \n2. Even though the accuracy improvements are marginal compared to the baselines, the proposed method consistently outperforms them. \n\nWeaknesses:\n1. The method is hard to follow due to the missing details and confusing notations. For example, D has different meanings in algorithm 1 and line 183.\n2. The experiments were insufficient. The previous standard knowledge distillation techniques such as [1,2,3] can still be applied for the problem settings used in the main paper. Could you compare with the previous algorithms [1,2,3]?\n\n[1] Wasserstein Contrastive Representation Distillation, CVPR 2021.\n[2] Contrastive Representation Distillation, ICLR 2020.\n[3] Variational Information Distillation for Knowledge Transfer, CVPR 2019.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "RXc2SE7dX1-", "reviewer": "Reviewer_1d5d", "summary": "Distillation is very popular technique both for preparing smaller student model for inference or doing some sort of pseudo-labeling including usage of unlabeled data. Current paper continues study in distillation topic and proposed a new theoretically motivated loss reweighting method to improve distillation of the teacher. The method is mainly studied in context of distillation on huge amount of unlabeled data. Here first student model is trained on supervised data, then the weights for every unlabeled sample are computed once from the results on validation data (we take KNN for distillation samples among validation data to tell what weight value will be), and finally we finetune model with weighted loss on mix of labeled and unlabeled data. Idea behind the weights is to have unbiased estimation for the risk minimization in distillation setting and there is an interesting discussion in the paper on noisy labels relation. Authors perform experiments on various datasets for image classification both with hard labels and soft labels with standard distillation or with a proposed method. Additional ablations on the supervised dataset size are conducted. Proposed method is hyper-parameter free, data-agnostic, and simple to implement. Experimental results are supported by theoretical analysis which rigorously justifies the method in certain settings.", "questions": "**Main paper**\n\nThe paper is very well written, thanks authors for this careful formulations and style of writing! Please find below my small suggestions on improving the text and typos as well as some extra questions on the results.\n- I would suggest authors to include [1] into introduction / related works section and discuss this paper in the context of usage unlabeled data and getting unbiased predictions from teacher. Also would be a bit simpler to track references if they are ordered by their appearance in the text.\n- line 27 \"overparametrized networks\" - here I wonder about this formulation (and do not agree with it) as student model is often much smaller (and authors also mentioned this before), so hard to say how this small student models will be overparametrized. Maybe reformulate this a bit?\n- In the whole paper (including appendix) there is a bit mess with usage of space only for inputs or for pair (input, target), e.g. lines 129-130, equation (3), etc. Would be simpler to have a unified way on these definitions and usages in expectation formulas. Maybe having notation of $X$ and $\\tilde{X}$ for input data saying that these spaces are the same for original data and adversarial.\n- line 182 formula for $k$: fix $2 / 2+D$ to $2 / [2+D]$\n- With respect to recent results in [1] I wonder what is happening with proposed method in case we have only labeled data and distillation is done as the original knowledge distillation paper (on training data of teacher model only)? Does reweighting scheme work? Also does unlabeled data improves distillation compared to distillation on supervised data only? [1] showed interesting results in this direction. \n- [1] also introduced discrepancy between training quality (how we transfer knowledge from teacher to student) and generalization (about which we care). Would be interesting to see training curves (training loss and accuracy) for student models with standard distillation and weighted one as maybe proper weights change the optimization and simplify it even.\n- Could authors provide what is approximate amount of time on computing weights for every sample? I guess it is negligible overall compared to the training, but curious to see confirmation.\n- Do authors do any ablations on recomputing the weights for every sample say every 50 epochs? At least to know if it changes a lot or not during the training.\n- Could authors perform ablation with temperature for the soft distillation case? Here we know (see e.g. in [1]) that temperature is giving a lot for optimization and generalization, even outweights other techniques to improve distillation. I wonder how it can change the optimization procedure and affect the proposed method. Also in practice we most likely use temperature not equal to 1.\n- Could authors provide numbers on the teacher performance pre-trained on supervised data before starting the distillation?\n- In Section 4 notation on spaces and distributions is not consistent with Section 2.\n\n**Appendix**\n\nI went through almost all appendix except last two proofs for sgd. General comment - Appendix needs a revision, it is less clear and careful prepared compared to the main text. For the rest of the text comments are below\n- Appendix A plots are strange: teacher accuracies between hard and soft distillation are not consistent (e.g. Fig 8 leftmost top and bottom plots). Why are they different? We start distillation from the same teacher for both hard and soft distillation so teacher accuracy should be exactly the same.\n- line 547 \"if the sample\" -> \"is the sample\"\n- line 558-559 usage of $M$ while it should be $\\mathcal{M}$\n- Equations (7) and (8): what about variance estimation? Do we have any assumption on it? Or it is bounded for some reason?\n- line 572: I repeated the math several times and got another range for $p$. It is $[0, (1-2d) / (1-d)^2]$ where $d$ is distortion. This does not affect the main conclusion and restricting weights to $[0, 1]$.\n- C3 section should be moved earlier before C1\n- in Equations after line 586 I don't understand how $l(f_{true}, f)$ disappears in $Bias(f)$. I think $Bias(f)=E_{x \\sim\\mathbb{P}} \\\\,p(x) (d - 1) * l(f_{true}, f)$ where $d$ is distortion.\n- line 577 type \"than\" -> \"that\"\n- notation is Section C3 and beginning of D should be fixed as again it is messed up with notation for input only and for (input, target) pairs.\n- lines 600-606: usage of notation $z$ should be changed. It is used as $x$ context and later as $y$ context.\n- line 629 - seems English language should be fixed here.\n- It will be simpler to read if all proofs from Section D.* will be placed in the first statements formulation in section D itself, not reformulating this several times.\n- Lemma D.7 - please add proper reference to the proof of this property (or via which theorem it can be proved).\n\n\n[1] Stanton, S., Izmailov, P., Kirichenko, P., Alemi, A.A. and Wilson, A.G., 2021. Does knowledge distillation really work?. Advances in Neural Information Processing Systems, 34, pp.6906-6919.", "limitations": "Limitations are listed in the conclusion section.\n\n\n**Discussion period updates**: I changed my rating score from 7 to 8 during the discussion period, details see in the thread below.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- A new reweighting method for knowledge distillation with unlabeled data which is based not on the uncertainty estimation (as people often do in pseudo-labeling) but on unbiased reweighting. \n- The method is simple to implement \n- Strong experimentation results across different datasets for image classification task (including ImageNet and smaller datasets): both for hard and soft distillation / with different amount of supervised data proposed method has statistically better results. [Sometimes however improvement is still marginal though statistically better]. \n- Interesting connection to noisy labels but here authors formulate the problem that noise is very specific and we can play with it.\n- Detailed instructions on reproduction experiments which are simple to follow.\n- Theoretical justifications of the proposed method for simpler forms of functions and standard regularity conditions: (i) it is information-theoretically optimal; and (ii) SGD optimization of the reweighted objective learns a solution with nearly optimal generalization.\n\n**Weaknesses**\n- Some mistakes in the proofs and formulations are found from Appendix section, however these findings should not affect the result and main statements.\n- Additional results on distillation on supervised data and training curves for current results would be great to have for the full picture (details see in the Questions section).\n- Ablation on recomputing the weights during the training (with some simple schedule at least, say every 50 epochs).\n- Ablation on the temperature for soft distillation as it could be the main contribution for the generalization and maybe proposed approach doesn't change much. Temperature usage is very standard setting in distillation works.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QGflXliYrZ", "reviewer": "Reviewer_gJJP", "summary": "This paper deals with the problem of few-shot knowledge distillation and proposes a method for weighing unlabeled examples based on the gaps between the true and distilling teachers. The obtained weights are employed for computing the loss function.", "questions": "I have no questions about this paper. ", "limitations": "See the Strengths and Weaknesses part.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n\nS1. The proposed method is simple and intuitive, which would draw the attention of people who are interested in applying it to their own tasks or applications.\n\nS2. The current manuscript is basically well written and easy to follow.\n\n[Weaknesses]\n\nW1. The current experimental evaluation fails to demonstrate the effectiveness of the proposed method due to the following two reasons:\n\n  (1) It lacks empirical comparisons with other related methods. In particular, [Dehghani+ ICLR2018] and [Kimura+ BMVC2018] incorporated example weighting for knowledge distillation. \n\n  (2) According to the experimental setup presented in Section 3.1, both the teacher and student models have access to whole the training dataset, which is not a typical setting in the context of knowledge distillation. The teacher is usually trained with the full size of supervised training examples and the student is able to employ the same dataset as the teacher (fully supervised), a part of the supervised dataset plus additional unsupervised examples (few-shot) or only unsupervised examples (zero-shot). Due to this experimental setting, it is very difficult to understand whether the current baseline sufficiently works well or not.\n\nW2. The proposed method requires additional supervised validation examples for computing the weights for unsupervised examples, which is a critical limitation of the proposed method. Existing methods for weighing examples do not rely on the availability of validation examples. If they are available for training, it seems better to employ them as additional training examples.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5DoQSBadcgL", "reviewer": "Reviewer_HUb8", "summary": "In this paper, the authors proposed a reweighting method to improve the performance of data distillation on the unlabeled dataset. Experiments on various datasets (SVHN, CIFAR-10/100, CelebA, and ImageNet) demonstrate the performance of the proposed method.", "questions": "I'm wondering why or how likely the pseudo labels are not reliable in the considered setting of the paper. In many cases in distillation with unlabeled data, the unlabeled dataset is different from the labeled data (or comes from a different distribution) used to train the teacher model. Then, it is likely that the pseudo labels given from the teacher model are unreliable. But when the labeled data and the unlabeled data come from the same distribution (i.e., the setting used in the experiments), this may be not a big issue.", "limitations": "It seems that the authors haven't addressed the potential negative societal impact of their work.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Advantages:\n1. The research topic, distillation with unlabeled examples, is very close to many real-world applications.\n2. The proposed method has some provable guarantees.\n\nDisadvantages:\n1. The proposed method, especially the procedure for weight estimation is quite complex and computationally expensive. If the labels from the teacher model are not reliable (can be regarded as label noise), why not apply some label noise methods to tackle this problem?\n2. In the experiments, the performance gain is not that much (e.g., on CelebA and ImageNet) when compared with the unweighted method.\n3. The paper does not show how and why using K-nn to learn the mapping between label confidence and the estimates of p(x) and the distortion(x). Is there any other method that can be used? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept