Rethinking Soft Labels for Knowledge Distillation: A Bias–Variance Tradeoff Perspective
Authors: Helong Zhou
Liangchen Song
Jiajie Chen
Ye Zhou
Guoli Wang
Junsong Yuan
Qian Zhang
Venue: ICLR 2021 Poster (2021)
Keywords: Knowledge distillation
soft labels
teacher-student model

Abstract: Knowledge distillation is an effective approach to leverage a well-trained network or an ensemble of them, named as the teacher, to guide the training of a student network.  The outputs from the teacher network are used as soft labels for supervising the training of a new network.  Recent studies (M ̈uller et al., 2019; Yuan et al., 2020) revealed an intriguing property of the soft labels that making labels soft serves as a good regularization to the student network.   From the perspective of statistical learning,  regularization aims to reduce the variance,  however how bias and variance change is not clear for training with soft labels.   In this paper, we investigate the bias-variance tradeoff brought by distillation with soft labels.   Specifically,  we observe that during training the bias-variance tradeoff varies sample-wisely. Further, under the same distillation temperature setting, we observe that the distillation performance is negatively associated with the number of some specific samples, which are named as regularization samples since these samples lead to bias increasing and variance decreasing.  Nevertheless, we empirically find that completely filtering out regularization samples also deteriorates distillation performance.  Our discoveries inspired us to propose the novel weighted soft labels to help the network adaptively handle the sample-wise bias-variance tradeoff.  Experiments on standard evaluation benchmarks validate the effectiveness of our method. Our code is available in the supplementary.

Published as a conference paper at ICLR 2021
RETHINKING SOFT LABELS FOR KNOWLEDGE DISTIL- LATION: A BIAS- VARIANCE TRADEOFF PERSPECTIVE
Helong Zhou' Liangchen Song2*i Jiajie Chen'r, Ye Zhou', Guoli Wang Junsong Yuan', Qian Zhang'
Hoorizon Robotics ?University at Buffalo TTinnguua University
(helong. zhou, jiajie. chen, ye. zhou, guoli wwwggnaoniizonn..... ai [lsong8, jsyuan fobuf falo. edu, qian01. zhangehorizon. ai
ABSTRACT
Knowledge distillation is an effective approach to leverage well-trained network an ensemble of them, named as the teacher, to guide the training of a student network. The outputs from the teacher network are used as soft labels for super- vising the training
new network. Recent studies (Müller al. 2019 Yuan et al.] (220) revealed an intriguing property of the soft labels that making labels soft serves as a good regularization to the student network. From the perspec- tive of statistical learning, regularization aims to reduce the variance, however how bias and variance change is not clear for training with soft labels. In this paper, we investigate the bari-nare tradeoff brought by distillation with soft labels. Specifically, we observe that during training the bias-variance tradeoff varies sample-wised Further, under the same distillation temperature setting, we observe that the distillation performance is negatively associated with the num- ber of some specific samples, which are named regularization samples since these samples lead to bias increasing and variance decreasing. Nevertheless, we empirically find that completely filtering out regularization samples also deterio- rates distillation performance. Our discoveries inspired us to propose the novel weighted soft labels to help the network adaptively handle the sample-wise bias- variance tradeoff. Experiments on standard evaluation benchmarks validate the effectiveness of our method. Our code is available at pst::s:/ com/ be Le_Lymonsterr/Weiihheee Soft Label Disti llation
INTRODUCTION
For deep neural networks (Goodfellow al.) 2016), knowledge distillation (KD) (Ba Caruana 2014] [Hinton al.] (2015) refers to the technique that uses well-trained networks to guide the train- ing of another network. Typically, the well-trained network is named as the teacher network while the network to be trained is named the student network. For distillation, the predictions from the teacher network are leveraged and referred to as the soft labels (Balan et al. 2015 Müller et al,)
2019). Soft labels generated by the teacher network have been proven effective large-scale em-
pirical studies (Liang et al. 2019 Tian et al. 2020 Zagoruyko & Komodakis 2017 Romero et al.
2015) as well as recent theoretical studies (Phuong & Lampert 2019).
However, the reason why soft labels are beneficial to the student network is still not well explained. Giving a clear theoretical explanation is challenging: The optimization details of deep network with the common one-hot labels are still not well-studied (Nagarajan & Kolter 2019)) not to mention training with the soft labels. Nevertheless, two recent studies (Müller et [2019] Yuan et al., 2020) shed light on the intuitions about how the soft labels work. Specifically, label smoothing, which special case of soft labels based training, shown regularize the activations of the penultimate layer to the network (Müller et al. 2019). The regularization property of soft labels is further explored in (Yuan et al., 2020). They hypothesize that in KD, one main reason why the soft labels work is the regularization introduced by soft labels. Based on the assumption, the authors
"These authors contributed equally to this work.
Work done while the author was research intern Horizon Robotics.

---

Published as a conference paper at ICLR 2021
design teacher-free distillation method by turning the predictions of the student network into soft labels.
Considering that soft labels are targets for distillation, the evidence of the regularization brought by soft labels drives us to rethink soft labels for KD: Soft labels are both supervisory signals and regularizers. Meanwhile, it is known that there is tradeoff between fitting the data and imposing regularizations, i.e., the bias-variance dilemma (Kohavi & Wolpert 1996 Bishop 2006), but unclear how bias and variance change for distillation with soft labels. Since the bias-variance tradeoff is an important issue in statistical learning, we investigate whether the bias-variance tradeoff' exists for soft labels and how the tradeoff affects distillation performance.
We first compare the bias and variance decomposition of direct training with that of distillation with soft labels, noticing that distillation results in a larger bias error and a smaller variance. Then, we rewrite distillation loss into the form of a regularization loss adding the direct training loss. Through inspecting the gradients of the two terms during training, we notice that for soft labels, the bias- variance tradeoff varies sample-wisely. Moreover, by looking into a conclusion from (Müller al.) 2019), we observe that under the same temperature setting, the distillation performance is nega- tively associated with the number of some certain samples. These samples lead to bias increase and variance decrease and we name them as regularization samples. To investigate how regularization samples affect distillation, first examine if we can design ad hoc filters for soft labels to avoid training with regularization samples. But completely filtering out regularization samples also de- teriorates distillation performance, leading us to speculate that regularization samples are not well handled by standard KD. In the light of these findings, we propose weighted soft labels for distil- lation to handle the sample-wist bias-variance tradeoff, by adaptively assigning a lower weight to regularization samples and larger weight to the others. To sum up, our contributions are:
For knowledge distillation, we analyze how the soft labels work from a perspective of bias- variance tradeoff.
We discover that the bias-variance tradeoff' varies sample-wisely. Also, we discover that if we fix the distillation temperature, the number of regularization samples is negatively associated with the distillation performance.
We design straightforward schemes to alleviate negative impacts from regularization sam- ples and then propose the novel weighted soft labels for distillation. Experiments on large scale datasets validate the effectiveness of the proposed weighted soft labels.
2 RELATED WORKS
Knowledge distillation. Hinton et al. (2015) proposed to distill outputs from large and cumber- some models into smaller and faster models, which is named as knowledge distillation. The outputs for large networks are averaged and formulated as soft labels. Also, other kinds of soft labels have
been widely used for training deep neural networks (Szegedy et al., 2016
Pereyra et al. 2017).
Treating soft labels as regularizers were pointed out in (Hinton et al. 2015 since lot of helpful
information can be carried soft labels. More recently, Müller et al. (2019) showed the adverse effect of label smoothing upon distillation. It is thought-provokiee both label smoothing and distillation are exploiting the regularization property behind soft labels. Yuan et al. (2020) further investigated the regularization property of soft labels and then proposed teacher free distillation scheme.
discovery for the reason that
One of our main contributions is that we improve the distillation loss. For adap- tively adjusting the distillation loss, Tang al. (2019) pays attention hard-to-learn and hard-to- mimic samples, and the latter is weighted based on the prediction gap between teacher and student. However, it does not consider that the teacher may give an incorrect guide the student, under which the prediction gap is still large and such a method may lead to the performance being hurt. Saputra al. (2019) transfers teacher's guidance only on the samples where the performance of the teacher surpasses the student, while Wen et al. (2019) deals with the incorrect guidance by probabil- ity shifting strategy. Our approach is different from the above methods, in terms of motivations as well as the proposed solutions.
Distillation loss
Bias-variance tradeoff. Birs-nariance tradeoff is a well-studied topic in machine learning (Kohavi
& Wolpert 1996. Domingos 2000; Valentini & Dietterich] 2004 Bishopp 2006) and for neural
2

---

Published as a conference paper at ICLR 2021
networks (Geman et al. 1992 Neal et al. 2018 Belkin et al. 2019 Yang al. 2020). Existing methods are mainly concemed with the variance brought by the choice of network models. Our perspective is different from the previous methods since focus the behavior of samples during training. In our work, based on the results from Heskes (1998). we present the decomposition of distillation loss, which is defined by Kullback-Leibler divergence. Besides, main contribution is not to study how to theoretically analyze the tradeoff, but how to adaptively tune the sample-wise tradcoff during training.
3 BIAS- TRADEOFF FOR SOFT LABELS
One-hot Label Bias
Soft labels play the role of supervisory signals and regularizations the same time, which inspires us to rethink soft labels from the perspective of the bias-varianc, tradeoff. We begin our analysis with some mathematical descriptions. For sample x labeled as i-th class, let the ground-truth label be one-hot vector y where = 1 and other entries are O. Then for æ and softmax output temperature the soft label predicted by the teacher network is denoted Y and the output from the student is denoted as D.- The soft label i² is then used for training the student by the distillation loss, i.e.
Label set A
Variance
Label Soft Label
Figure 1: Bias and variance.
Lka -- Σx & iker log ii where Di oh,-- means the k-th element of the student's output D and the teacher's output V" respectively. With the above notations, the cross-entropy loss for training with one-hot labels is Lee log V
We now present the bias-variance decomposition for Lce and Lidd based on the definition and no- tations from Heskes (1998). First, we denote the train dataset as 'D and the output distribution a sample x of the network trained without distillation as yc. = fce (x; D). For the network trained with distillation, the model also depends the teacher network, so define the output on as Ykd (x; D, 7), where T is the selected teacher network. Then, let the averaged output of ýıd and Yce be Yka and Yce. that is,
1
Yce
exp(Enllog'e ycel,, Ykd
exp(Ex,7 [log kkal),
Zee
Zua
where Zec, Zkd are two normalization constant. following decomposition for the expected error on the sample x and y f(x) is the ground truth label:
Then according to Heskes 1998), we have the
y
yce
errorGe = Ex.D l-y log Ycel = Ex,D -y logy + log
+
yy log
Yce
yce
y
/ce Pco_
Exl-y yo y| + Ex
y log
ED
Ex
y log
(2)
yen.
Ex| log + DKL (y. Fee) + ED DKL
intrinsic noise + bias + variance,
where DKL is the Kullback-Leiblerr divergence. The derivation of the variance term is based on the Yce facts that Eop Yoo is constant and [x[y] [x[yce] 1. Detailed derivations can be found from Eq. (4) in Heskes 1998). Next, analyze the bias-variance decomposition of Lud- As mentioned above, when training with soft labels, extra randomness is introduced for the selection of teacher network. In Fig. [ we illustrate the corresponding bias and variance for the selection process of a set of soft labels, which are generated by a teacher network. In this case, a high variance model indicates the model (grey point) closer to the one-hot trained model (black point), while a low variance model indicates that the model is closer to other possible models trained with soft labels (red points). Although for KD there are more sources introducing randomness, the overall variance brought by Lud is not necessarily higher than Lee. In fact, existing empirical results strongly suggest that the overall variance is smaller with KD. For example, students trained with soft labels better calibrated than one-hot baselines (Müller al. 2019) and KD makes the predictions of students more consistent when facing adversarial noise (Papernot et al. 2016). Here, we present these empirical evidence assumption:
3

---

Published as a conference paper at ICLR 2021
Assumption The variance brought by KD is smaller than direct training. that is, (P.r([ (xa)) (Fee
Similar to Eq. 2 write the decomposition for Lkd as
yce Ykd
errorkd Ex[ y log y + DKL (y, Fce) Ex log
ED, [DRL (Fkd, Ska]]-
(3)
An observation here is that Yce converges to one-hot labels while Vlot converges to soft labels, so Yoe is closer to the one-hot ground-truth distribution y than Ykd, i.e., Ex | y yka
Ess
> 0. If
we rewrite Lid Lidd Lid Lee Lees then Lud Lee causes that the bias increases by
Ex y log
Yeu and the variance decreases by (Foo. - [DRI (F
From the above analysis, we separate Lid into terms, and Lid Lee leads to variance reduction, and Loe leads to bias reduction. In the following sections, we first analyze how Lkd Lee links to the bias-variance tradeoff during training. Then we analyze the changes in the relative importance between bias reduction and variance reduction during training with soft labels.
3.1 THE MAS-VVARANCE TRADEOFF DURING TRAINING
is known that bias reduction and variance reduction are often in conflict and we cannot minimize bias and variance together. However, if we consider the change of bias and variance during the training process, the importance of tuning the tradeoff also changes during training. Specifically, shortly after the training of the network starts, the bias error dominates the total error and the variance less important. As training goes on, gradients of reducing the bias error (induced by Ioee) and reducing the variance (induced by Lid Lee) can be of the same scale for some samples, then need to balance the tradeoff because reducing one term is likely to increase another one. Therefore for soft labels, we need handle the bias-variance tradeoff in sample-wise manner and take the training process into consideration.
To study the bias- variance tradeoff during training, we consider the gradients of bias and variance reduction. Let z be the logits output of the student on input x and 2; is i-th element of it, then are interested in o(Lng-Lca) For simplifying analysis, we concerned with the gradients dz: the ground- truth related logit, that is, the sample is labeled as i-th class. Mathematically. for the gradients of variance reduction, we have
a(Lkd= - Lec) (Wi - )i, (07, 11)1--))- Ui) -T
Mir (
c
1/
(4)
0z,
where of.r denotes the i-th element of the teacher's prediction, i.e., Df. The term Lia Lee is easy to understand when 1 since the gradient now becomes y Vf Mcanwhile, for the bias reduction, Ø(Lua Len) always have different signs, leading to a tradeoff. 82,
we have aL. 0f V/. so
OLee and
Gz.
If aLes is much higher than ((Laa-La)) the bias reduction dominates the overall optimization direc- 0z, tion. Instead, if -Lon) becomes higher, the sample is used for variance reduction. Interestingly, 82, we discover that under a fixed distillation temperature, the final performance is worse when more training samples are used for variance reduction, which will be introduced in the next section.
0z:
3.2 REGULARIZATION SAMPLES
Our analysis starts with conclusion from Müller al. (2019): if teacher network is trained with label smoothing, knowledge distillation into a student network is much less effective. Inspired by the phenomenon, gather the impact of bias and variance during training with different distillation settings. Let a 8z,
0Lee and b a(Lza then as introduced before, we use a and represent 82, the impact of bias and variance, respectively. If we have [b] > |al for a sample, we name the sample regularization sample since the variance dominates the optimization direction. From the collected data, we find that the number of regularization samples is closely related to distillation performance.
4

---

Published as a conference paper at ICLR 2021
Table 1: We count the number of regularization samples with different distillation settings on CIFAR- 100. The teacher-student network pair is WRN-40-2 (Zagoruyko & Komodakis 2017) and WRN-16-2. Results are averaged over repeated runs. The temperature column means the temper- ature for distillation and the label smoothing column means whether the teacher network is trained with label smoothing trick.
Teacher: 76.55 label smoothing, 75.61 w/o label smoothing
Baseline Top-1 Acc
Student: 73.26
Temperature
Label smoothing?
Student Top-1 Ace
Number of regularization samples
74.79
15379
T T= 2
74.62
25235
x
74.92
17709
T = 4
74.59
24775
75.10
17408
7 6
74.46
24538
w: ladi smoothin
ny lubel smootting amoothing
Lidd smouthing smonthing
smonthing
50000
.0000
10000
5000
0003
3800
250
200
Epech
Eaoch
(a) Temperature r
(b) Temperature = 4
(c) Temperature T =
Figure 2: The number of regularization samples with respect training epochs. The distillation
settings are the same as the settings in Tab.
In Tab. [ we present the count of regularization samples for student network trained by knowledge distillation. For distillation with temperature higher than which is the common setting, we observe that if the teacher network is trained with label smoothing, more samples will be involved in variance reduction. Also, distillation from a teacher trained with label smoothing performs worse, which is consistent with Müller et al. (2019). Therefore, we conclude that for distillation with soft labels, the regularization samples during training affect the final distillation performance.
Moreover, plot the number of regularization samples with respect to different training epochs in Fig. As demonstrated the plots, the number of such samples increases much faster when using the teacher trained with label smoothing for distillation. For regularization samples, the gap of their number between with and without label smoothing becomes larger for more training epochs. These observations verify our motivation that the bias-variance tradcoff varies sample-wisch and evolves during the training process.
From the above results, conclude that hias-variance tradeoff for soft labels varies sample-wisely, therefore the strategy for tuning the tradeoff should also be sample-wise. In the next section, we set up ad hoc filters for soft labels and further investigate how regularization samples affect distillation.
3.3 HOw REGULARIZATION SAMPLES AFFECT DISTILLATION
The results presented in the last section suggest that should avoid training with regularization samples. Hence, we design two straightforward solutions and then find that totally filtering out regularization samples deteriorates the distillation performance.

---

Published as a conference paper at ICLR 2021
No backward gradient
Ln --² E, Nh,r 12,
Lic
log
Lice
log $7
Input
Teacher
T
Tempereture
Softmax
T(X,7)
Dreeder Laber
Lioo
Studient
Saftmax
S(X,7)
86 1-th
Jan
cxp
Lige
Lya
10
1 00)
Logits
Figure 3:
Computational graph of knowledge distillation with our proposed weighted soft labels.
Table 2: Study of the impact on distillation for regularization samples. Loss function presented here for the loss on a specific sample. Results are classification Top-I accuracy. We follow the settings used in Tab. and set t = 4. Results are averaged over 5 runs.
Teacher: 75.61;
Student with direct training: 73.26; Student with standard KD: 74.92
Performance gap
Setting
Loss function
Student performance
to direct training to KD
Mask KD loss on the label related logit
lice + Lid
73.51
+0.25
-1.41
Locn if |a| [6]
Excluding regularization samples
74.59
+1.33
-0.33
Loe + Lud,; if [a] > [6]
Towe if |a| T [6]
Only on regularization samples
73.86
+0.60
-1.06
Loo + Lud, if [a] [6]
The first experiment conduct is manually resolve the conflicting gradient the label related logit, as defined section 3.2 Specifically, we apply a mask to the distillation loss Lıd such that aLga
dz. = 0 where is the label. Consequently, the loss for this sample now becomes Lia = Exs, 0k, log
log oh.,. The motivation behind the masked distillation loss is that we only transfer the knowledge of resemblances among the labels. Another experiment is to figure out what role in distillation those regularization samples will play. To investigate this, we carry out knowledge distillation on two subsets of samples: 1) Lkd is not valid on regularization samples, and 2) Lxa is valid only on regularization samples.
The results of the two experiments are presented in Tab. We can observe that all of the three approaches are not as good as the baseline knowledge distillation performance, but are better than the direct training baseline. First, since masking Lkd loss on the label related logit results in worse performance compared to standard KD, we cannot resolve the tradeoff by applying a mask on ground truth related logit. Then, from the second experiment, we can that filtering out regulariza- tion samples deteriorates the distillation performance. Moreover, the result of the third experiment is higher than the direct training baseline, indicating that regularization samples are still valuable for distillation. The above results motivate us to think that regularization samples are not fully exploited by standard KD and we can tune the tradeoff to fulfill the potential of regularization samples.
WEIGHTED SOFT LABELS
From the last section, we realize that the bias-variance tradeoff varies sample-wisely during training and under fixed distillation settings, the number of regularization samples is negatively associated with the final distillation performance. Yet, discarding regularization samples deteriorates distil- lation performance and distilling knowledge from these samples better than the direct training baseline. The above evidence inspires us to lower the weight of regularization samples.
Recall that regularization samples are defined by the relative value of and we propose to assign importance weight to sample according to a and b. However, since Lud is computed with the hyperparameter temperature, a and b are correlated with the temperature and thus bring difficulty to tuning the hyperparameter. To make the weighting scheme independent of the temperature hy- perparameter, we compare a and with temperature T 1. Note that when 7 = 1, a Ø( - y: and = y; T so we compare 0 and gf instead. Finally, ,, in the light of previous works that assign sample-wist weights (Lin al. 2017 Tang et al. 2019), we propose weighted soft labels for

---

Published as a conference paper at ICLR 2021
Table 3: Top-1 classification accuracy results on CIFAR-100. Comparison results are quoted from
Tian et al. (2020). We report our results over repeated runs.
Same architecture style
Different architecture style
Teacher
WRN-40-2
resnet56
1eseet10
resnet1
10500132x4
resnet32x4
resnet32x4
WRN-40-2
Student
WRN-40-1
resnet20
resaet20
resnet32
resnet8x4
SbuffeeNetV
ShuffleNetV2
ShuffleNetVi
Teacher
75.61
72.34
74.31
74.31
79.42
79.42
79.42
75.61
Student
71.98
69.06
69.06
71.14
72.50
70.5
71.82
70.5
FitNet
72.24
69.21
68.99
71.06
73.50
73.59
73.54
73.73
AT
72.77
70.55
70.22
72.31
73.44
71.73
72.73
73.32
SP
72.43
69.67
70.04
72.69
72.94
73,48
74.56
74.52
CC
72.21
69.63
69.48
71.48
72.97
71.14
71.29
71.38
VID
73.30
70.38
70.16
72.61
73.09
73.38
73.40
73.61
RKD
72.22
69.61
69.25
71.82
71.90
72.28
73.21
72.21
PKT
73.45
70.34
70.25
72.61
73.64
74.10
74.69
73.89
AB
72.38
69.47
69.53
70.98
73.17
73.55
74.31
73.34
FT
71.59
69.84
70.22
72.37
72.86
71.75
72.50
72.03
FSP
n/a
69.95
70.11
71.89
72.62
n/a
n/a
n/a
NST
72.24
69.60
69.53
71.96
73.30
74.12
74.68
74.89
KD
73.54
70.66
70.67
73.08
73.33
74.07
74.45
74.83
CRD
74.14
71.16
71.46
73.48
75.51
75.11
75.65
76.05
Ours
74.48
72.15
72.19
74.12
76.05
75.46
75.93
76.21
knowledge distillation, which is formally defined
I Lowll exp (-
(1
log if,1
) Lkd (11---1000
(1-cxp exp
Lege
Lidd
(5)
log of,
L
where the ground truth class of the sample. The above equation means that weighting factor is assigned each sample's Lkd according the predictions of the teacher and the student. In this way, if compared to the teacher, student network relatively better trained on a sample, we have 0f,1 Vh,1 then smaller weight is assigned to this sample. In Fig. 3 the whole computational graph of knowledge distillation with the proposed weighted soft labels is demonstrated. Finally, add Lust and Loe together to supervise the network, i.e., Leooal Lee + alwaai, where is balancing hyperparameter.
EXPERIMENTS
To evaluate weighted soft labels comprehensively. we first conduct experiments with various teacher-student pair settings on CIFAR-100 (Krizhevsky et al. 2009). Next, we compare our method with current state-of-le -fe-----t-ht art distillation methods ImageNet (Deng et al. 2009). To validate the effectiveness our method terms handling the bias- variance tradeoff, we conduct ablation experiments by applying weighted soft labels different subsets.
5.1 DATASET AND HYPERPARAMETER SETTINGS
The datasets used in our experiments are CIFAR- 100 (Krizhevsky et al. 2009 and ImageNet (Deng et al. 2009). CIFAR- 100 contains 50K training and 10K test images of size 32 x 32. ImageNet contains 1.2 million training and 50K validation images. Except the loss function, training settings like learning rate or training epochs the same with Tian al. (2020) for CIFAR-100 and Heo ct al. (2019) for ImageNet. For distillation, we set the temperature T = 4 for CIFAR and r = 2 for ImageNet. For loss function, set 2.25 for distillation CIFAR and = 2.5 for ImageNet via grid search. The teacher network well-trained previously and fixed during training.
For comparison, the following recent state-of-the-art methods chosen: FitNet (Romero et al. 2013), AT (Zagoruyko Komodakis 2017). SP (Tung & Mori 2019). CC (Peng et al. 2019), VID Ahh et al. 2019), RKD (Park al. 2019). PKT (Passalis & Tefas 2018), AB (Heo et al. 2019

---

Published as a conference paper at ICLR 2021
Table 4: Top-1 and Top-5 classification accuracy results on ImageNet validation set. All training hyperparameter like learning rate and training epochs are in accordance with (Heo et al. 2019
Teacher:
ResNer-34 + Student: ResNet-18
Teacher: ResNet-50 + Student: MobileNet-vl
Method
Top-1 Acc
Top-5 Acc
Method
Top-1 Acc
Top-5 Acc
Teacher Student
73.31
91.42
Teacher Student
76.16
92.87
69.75
89.07
68.87
88.76
KD
70.67
90.04
KD
70.49
89.92
AT
71.03
90.04
AT
70.18
89.68
NST
70.29
89.53
FT
69.88
89.5
FSP
70.58
89.61
AB
68.89
88.71
RKD
70.40
89.78
RKD
68.50
88.32
Overhaul
71.03
90.15
Overhaul
71.33
90.33
CRD
71.17
90.13
CRD
69.07
88.94
Ours
72.04
90.70
Ours
71.52
90.34
(Kim et al.
2018), FSP (Yim et al.
2017). NST Huang Wang 2017). Overhaul (Heo et al,
2019 and CRD (Tain et al. 2020).
5.2
MODEL COMPRESSION
Results on CIFAR-100 In Tab. B] we present the Top-1 classification accuracy of our method and comparison methods. The results of comparison methods are quoted from Tian et al. (2020). Teacher-student pairs of the same and different architecture styles are considered. For pairs same architecture style, we use wide residual networks (Zagoruyko & Komodakis (017) and residual networks (He et al. 2016). For pairs of different architecture style, residual networks and ShuffleNet (Zhang al. 20188 pairs arc chosen for experiments. As shown in the table, for distillation with both same and different architecture style, our method reached new state-of-the-arr results. Specifically, method outperforms standard KD by large margin, which verifies the effectiveness of our method.
Results on ImageNet In Tab. we compare our method with current SOTA methods on Ima- geNet. Note that for the ResNet34 > ResNet-18 18 distillation setting, the result of CRD is trained 10 more extra epochs while ours is the same as other methods. For ResNet-50 - MobileNet-vi distillation setting, NST and FSP are not chosen for comparison as the two methods require too large GPU memories, so we include the accuracy of FT and AB reported in Heo et al. (2019) for comparison. Our results outperform all the existing methods, verifying the practical value of our method.
5.3 ABLATION STUDIES
Weighted soft labels on different subsets. Recall that we propose weighted soft labels for tuning sample- wise bias-variance tradeoff, it is still unclear whether the improvements come from well-handled sample- wise vias-variance tradeoff. investigate this issue, we compare the performance gain of weighted soft la- bels on different training subsets. Similar to the settings used in Tab. 2 apply weighted soft labels on two different subsets: only the regularization samples and excluding regularization samples. In Tab. 5 show the results on subsets of only regularization samples and excluding regularization samples. From only improve performance on the
Table 5: Performance on different subsets with soft labels and weighted soft la- bels. RS means regularization samples. Results averaged over 5 runs.
Subsets
Standard KD
Weighted
Only RS 73.86
74.46
Excluding
74.59
75.35
the significant improvements, RS subset, the improvements on excluding RS subset is also significant. We conclude that weighted soft labels can tune sample-wise bias-variance tradeoff globally and lead an improved distillation performance.
can that our method
8

---

Published as a conference paper at ICLR 2021
Distillation with label smoothing trained teacher. Our exploration of birs-variance tradeoff starts with the conclusion made in Mäller et al. (2019): teacher net- work trained with the label smoothing trick is less effec- tive for distillation. It is worthwhile to study whether the conclusion remains true for distillation with our weighted soft labels. As discussed before, we hold the opinion that too many regularization samples make the distillation less effective. Since our weighted soft label is proposed mitigate the negative effects of the regularization sam-
Table 6: Distillation using weighted soft labels and teacher trained with label smoothing (denoted as LS?). Results are averaged over runs.
T
IS?
Standard KD
Weighted
4
X
74.92
75.78
4
v
74.59
75.60
6
x
75.10
75.74
6
/
74.46
75.42
ples, with the same settings from Tab. [ we conduct com-
parison experiments in Tab. [6 to see if the negative effects still exist. It is evident that weighted soft labels significantly improve the distillation performance, especially for distillation from the teacher trained with label smoothing. Besides, using the teacher trained with label smoothing still performs worse than that without label smoothing, which again verifies the conclusion drawn by Müller et al. 2019).
CONCLUSION
Recent studies (Müler al. 2019 Yuan et al. 2020) point out that important reason behind the effectiveness of distillation is the regularization effect brought by being soft. In this paper, we rethink the soft labels for distillation from bias- vias-variance tradeoff perspective. The tradeoff varies sample-wisely and we propose weighted soft labels to handle the tradeoff, of which the effectiveness is verified with experiments standard evaluation benchmarks.
ACKNOWLEDGEMENTS
This work supported in part by gift grant from Horizon Robotics and National Science Founda- tion Grant CNS-195199 1951952. We thank Yichen Gong, Chuan Tian, Jiemin Fang and Yuzhu Sun for the discussion and assistance.
REFERENCES
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In Proceedings of the IEFE Conference Com- puter Vision and Pattern Recognition, pp. 9163-9171, 2019.
Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In Advances in Neural Informa- fion Processing Systems, pp. 2654-2662, 2014.
Anoop Korattikara Balan, Vivek Rathod, Kevin P Murphy, and Max Welling. Bayesian dark knowl- edge. In Advances Neural Information Processing Systems, pp. 3438-3444444 2015.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical bias variance trade-off. Proceedings of the National Academy of Sciences, 156632)
2019.
Christopher M Bishop. Pattern recognition and machine learning. springer, 2006.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database, In The IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255. IEEE, 2009.
Pedro Domingos, unified variane decomposition. International Conference on Machine Learning, pp. 231-28 231-238, 2000.
Stuart Geman, Elie Bienenstock, and René Doursat. Neural networks and the bias/variance dilemma. Neural computation, 4(1):1-58, 1992.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.

---

Published as a conference paper at ICLR 2021
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In The IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A com- prehensive overhaul of feature distillation. In Proceedings of the IEEE International Conference Computer Vision, pp. 1921-1930, 2019.
Tom Heskes. Bias/variance decompositions for likelihood-based estimators. Neural Computation, 10(6):1425-1433, 1998.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. CoRR, abs/1503.025311 2015.
Zehao Huang and Naiyan Wang. Like what you like: Knowledge distill via neuron selectivity transfer. arXiv preprint arXiv:1707.01219, 2017.
Jangho Kim, SeongUk Park, and Nojun Kwak. Paraphrasing complex network: Network compres- sion via factor transfer. In Advances in neural information processing systems, pp. 2760-2769, 2018.
Ron Kohavi and David H Wolpert. Bias plus variance decomposition for zero-one loss functions. In International Conference Machine Learning, volume 96, pp. 275-83, 1996.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ruofan Liang, Tianlin Li, Longfei Li, Jing Wang, and Quanshi Zhang. Knowledge consistency between neural networks and beyond. In International Conference on Learning Representations, 2019.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick. Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2980-2988, 2017.
Charles Loop. Smooth subdivision surfaces based on triangles. Master's 's thesis, University of Utah, Department of Mathematics, 1987.
Rafael Müller, Simon Komblith, and Geoffrey E Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems, pp. 4694-4703, 2019.
Vaishnavh Nagarajan and J. Zico Kolter. Uniform convergence may be unable to explain generaliza- tion in deep learning. Advances in Neural Information Processing Systems, pp. 11615-11626. Curran Associates, Inc., 2019.
Brady Neal, Sarthak Mittal, Aristide Baratin, Vinayak Tantia, Matthew Scicluna, Simon Lacoste- Julien, and Ioannis Mitliagkas. A modern take on the bias-variance tradeoff in neural networks. arXiv preprint arXiv:1810.08555 2018
Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as defense to adversarial perturbations against deep neural networks. IEEE Symposiun on Security and Privacy, pp. 582-597. IEEE, 2016.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Pro- ceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 3967-3976. 2019.
Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In Proceedings of the European Conference on Computer Vision, pp. 268-284, 2018.
Ted Pedersen, Siddharth Patwardhan, and Jason Michelizzi. Wordnet:: Similarity-maasuring the relatedness of concepts. In AAAI Conference on Artificial Intelligence, volume 4, pp. 25-29, 2004.
10

---

Published as a conference paper at ICLR 2021
Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang. Correlation congruence for knowledge distillation. Proceedings of the IEEE International Conference Computer Vision, pp. 5007-5016, 2019.
Gabriel Pereyra, George Tucker, Jan Chorowski, Éukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:/701 17700066555. 2017.
Mary Phuong and Christoph Lampert. Towards understanding knowledge distillation. In Interna- tional Conference on Machine Learning, pp. 5142-5151, 2019.
PlotOptiX. http:////pppp.oooo/ooooope/ppppp
(Accessed on 09/08/2020).
Adriana Romero,
Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In International Conference on Learning
Representations, 2015.
Muhamad Risqi U Saputra, Pedro PB de Gusmao, Yasin Almalioglu, Andrew Markham, and Niki Trigoni. Distilling knowledge from deep pose regressor network. In Proceedings of the IEEE International Conference Computer Vision, pp. 263-272, 2019.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 4314-4323, 2019.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In The IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818 2826, 2016.
Shitao Tang, Litong Feng, Wenqi Shao, Zhanghui Kuang, Wei Zhang, and Yimin Chen. Learning efficient detector with semi-supervised adaptive distillation. arXiv preprint 2019.
Yonglong Tian, Dilip Krishman, and Phillip Isola. Contrastive representation distillation. In Inter- national Conference on Learning Representations, 2020.
Frederick Tung and Greg Mori. Similarity-preserving knowledge distillation. In Proceedings of the IEEE International Conference Computer Vision, pp. 1365-13744 2019.
Giorgio Valentini and Thomas G Dietterich. Bias-variance analysis of support vector machines for the development of svm-based ensemble methods. Journal of Machine Learning Research. 5(Jul): 725-775, 2004.
Tiancheng Wen, Shenqi Lai, and Xueming Qian. Preparing lessons: Improve knowledge distillation with better supervision. arXiv preprint 171.4.19 2019.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad croad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pp. 1112-1122. Association for Computational Linguisties, 2018. URL
http: http///alwwwbbbe org/ anthologyooooo.oo /N18 11c1
Zitong Yang, Yaodong Yu, Chong You, Jacob Steinhardt, and Yi Ma. Rethinking bias-variance trade-off for generalization neural networks. arXiv preprint arXiv:2002.11328, 2020.
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4133-4141, 2017.
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In The IEEE Conference on Computer Vision and Pattern Recognition, pp. 3903-3911, 2020.
11

---

Published as a conference paper at ICLR 2021
(b)
(d)
Figure 4: Visualization of the resemblances introduced by soft label regularizers: (a) VGG-19 (Teacher) + VGG-16 (Student), (b) ResNet-50 (Teacher) + ResNet-18 (Student). And seman- tic similarity between label names: (c) LCH similarity (Pedersen et al. 2004), (d) WUP similarity (Pedersen et al. 2004). Darker areas denote larger values.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the per- formance of convolutional neural networks via attention transfer. International Conference on Learning Representations, 2017.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6848-6856, 2018.
A
APPENDIX
A.1
VISUALIZATION OF THE RESEMBLANCES INTRODUCED BY SOFT LABEL REGULARIZER
In Sec 3.1 we propose to study 8(L2a Lee) during the training process. When т 1, we show 0z:
Lee) equals y4 - As vÊ,- is the output from the teacher network and computed by a linear mapping of the activations in the teacher's penultimate layer, the regularization indicates that the student should follow the learned the resemblances between classes (Hinton et al. 2015) Müller et al. 2019). Still, two questions are unclear: 1) what the resemblances are and 2) whether the regularization still indicates resemblances if r is set to 4, widely adopted hyperparameter (Tian) et al. 2020).
that 0z;
Towards answering the questions, we visualize the value of gradient vector 8(Laa Inee) concerning dz
each class. Specifically, on ImageNet (Deng al. 2009) training set and 7 4. we caleulate the average value of A[Lsg- Lae) for each class. Let M be the matrix of values with the ij-th entry dz ol - for class j. Since E; Mij 0. diagonal elements are ignored for 0s,
Mij means averaged
visualization. The results are visualized in Fig. We find that plotting the common correlation
12

---

Published as a conference paper at ICLR 2021
Table 7: Intermediate states between excluding and only on regularization samples
(a) CIFAR100 (WRN-40-2 with KD)
Percentage of excluded regularization samples.
100%
75%
50%
25%
74.59
74.63
74.72
74.87
Percentage of adding non reguulaaaarzziionn samples
0%
25%
50%
75%
73.86
74.12
74.47
74.71
(b) CIFARIO0 (WRN-40-2- with weighted soft labels)
Percentage of excluded regularization samples
100%
75%
50%
25%
75.35
75.48
75.61
75.72
Percentage of adding non-regularization samples
0%
25%
50%
75%
74.46
74.79
75.18
75.53
matrix heatmap is ambiguous, because the matrix to be visualized is of large size (1000 x 1000) with a large variance. By treating each entry Mij as vertex and then constructing a mesh for the matrix, apply subdivision (Loop) 1987) to the mesh for smoothing the extreme points and finally rendering the mesh by ray-tracing package PlotOptiX] We can observe the several facts from the figures: 1) Comparing the sub-figure (a) and (b), implied by regularizers are similar across different teacher-student pairs. 2) Comparing (ab) with (cd),
that for distillation resemblances
see that the resemblances consistent with the semantic similarity of image class names.
In word, for 4, the variance reduction brought by soft labels still implies resemblances among labels, which are consistent with the semantic distance of class names. In the next section, we will analyze how bias-variance tradeoff changes when training with soft labels.
A.2 INTERMEDIATE STATES BETWEEN EXCLUDING AND ONLY ON REGULARIZATION SAMPLES
To further investigate the phenomenon about regularization samples, we conduct experiments to show the intermediate states between excluding and only regularization samples. Two settings are considered here: First, we gradually exclude regularization samples during training, from ex- cluding all regularization samples to excluding 25% regularization samples; Second, we keep all regularization samples and then gradually add non-regularization samples. Since we judge a sample regularization not according the training loss, we cannot pre-define sample set such that certain percentage samples are kept or dropped. Therefore, we propose to conduct these experiments by assigning probability to whether backward the loss computed with regularization samples. For example, if during training, a sample is marked as regularization sample according to the value of and b, we backward the loss of this sample by probability p 0.5. In this way, can get the performance of excluding 75% regularization samples. In Tab. 77] we first present result with KD in (a) and then present result with weighted soft labels applied in (b). We can observe that weighted soft labels are indeed balancing the sample-wise, not on dataset scale, bias and variance.
A.3 COMBINING WITH RKD PARK ET AL. 2019
To investigate how the weighted soft labels can be applied to the variants of KD, we conduct an experiment of combining RKD (Park al. 2019) with weighted soft labels. Relational knowl- edge distillation measures the L2 distance of features between two samples or the angle formed by three samples as knowledge to transfer. In other words, the knowledge in RKD is measured by
13

---

Published as a conference paper at ICLR 2021
Table 8: Combining weighted soft labels with RKD
Park et al. 2019)
Distillation settings
WRN-40-2 WRN-16-2
WRN-40-2 + WRN-40-1
resnet56 resnet20
Teacher
75.61
75.61
72.34
Student
73.26
71.98
69.06
RDK WSL + RKD
74.12
73.34
70.25
74.65
73.89
70.73
Table 9: Comparison to other weighting forms. (Setting: CIFARI00, WRN-4022-WRN-4N
2.0
3.0
4.0
Sigmoid baseline
74.13
73.97
73.29
Ours
74.38
74.12
73.46
the relations between sample pairs. It longer sample-independent, which different from the weighted soft labels applied to KD which can assign the weights sample-wisely. We currently take the averaged weighting factors of the involved sample pairs when calculating the distance/angle ma- trix. The results on CIFAR-100 are presented in Tab. 8 (averaged over 5 runs). As can be observed from the table, the weighted soft label applied to RKD still brings improvements, though not that big compared with WSL applied to KD. Also, we believe that it is an important future direction to explore the applications to more variants of KD.
A.4 OTHER VARIANTS OF WEIGHTING.
In the work, the weighting scheme is defined as exp
which is inspired by Lin et al.
2017 Tang et al. (2019). The basic idea to convert regularization samples are lower than those non-regularization samples. A straightforward baseline
L into value [0, 1], so that weights of
Li
4.s that use the Sigmoid function to convert into value in [0, 1]. Note that 4 is always
I.f.
bigger than 1, so the weight needs scaling and can be defined as 1+exp(- ( 1. In Tab. 9 we
2
present the comparison between adopted weighting form and the Sigmoid bascline. We can see that as long as we can adaptively tune the sample-wise bias-variance tradeoff, the performance is better than KD, i.e., without weighted soft labels. Therefore, although the proposed weighting form is not mathematically optimal, the not-too-big or not-too-smo not hard to tune. These results verify our main contribution that there is sample-wir wise bias- variance tradeoff and we need to assign weights to the regularization examples.
weights for these regularization examples
A.5 ABLATION ON a.
In Tab. I0 We first the value of a on CIFAR100, with four values 31,2,3,4) tested. Then we test with three values in [2,3) in (b). Finally, we tune on ImageNet in (c). As a conclusion, the results are not very sensitive to a and the cost of searching a in our work not expensive.
A.6 RESULTS ON MULTINLI
To further validate our method, we conduct experiments an NLP dataset MultiNL (Williams) et al. 2018). In this setting, the teacher is BERT-base-cased with 12 layers, 768 Hidden and 108M params. The student is T3 with layers, 768 Hidden and 44M params. Besides, we follow the training setting in (Sun et al. (201)). In Tab. [I we present the result comparisons of standard KD and our weighted soft labels.
14

---

Published as a conference paper at ICLR 2021
Table 10: Ablation on a.
(a) CIFARI00 (WRN-40--WRN--0JJ
a
2
3
Top1
73.67
74.38
74.12
73.46
(b) CIFARIO0 (WRN-40-2 WRR--0-1)
2.25
2.5
2.75
Top1
74.48
74.34
74.21
(c) ImageNet (ResNet-34 ResNet-18)
2.25
2.5
Top1
71.91
71.96
72.04
Table 11: Results on MultiNLI.
Teacher (BERT-12)
Student (BERT-3)
KD (BERT-3)
Ours (BERT-3)
Results reported by Sun et al. 2019
83.7 83.57
74.8
75.4
Our replications
75.06
75.50
76.28
15


Reviews:
[{"review_id": "t7mbShCWTJ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors studied the soft labels for knowledge distillation from a bias-variance tradeoff perspective. Specifically, the authors first provide a mathematically descriptions of the bias-variance decomposition in knowledge distillation. Then, based on the theoretically analysis and experiments, the authors proposed an novel weighted soft labels to help the network adaptively handle the sample-wise bias-variance tradeoff.\n\nStrength:\n1. The paper is well written and easy to follow.\n\n2. The authors provide many mathematically proof in the paper, which could serve as a theoretically foundation of this topic. \n\n3. The authors claim that their code is available in the supplementary, which makes it easier for other researchers to reproduce this work.\n\nWeakness:\n1. In addition to computer vision, Knowledge distillation is also very popular in NLP area recently. Maybe the authors can add more experiments for NLP to prove that the solution can be widely-adopted.\n\n2. I am not very familiar with computer vision area, but it seems that ResNet-50/ResNet-34 has a huge gap with the current SOTA (Ref: https://paperswithcode.com/sota/image-classification-on-imagenet). Experimenting with stronger baselines would make the conclusion even stronger.  \n\nOverall comments:\nI think this is a good paper and I'd like to see it to be accepted.", "extra_scores": {}, "extra_text": {}}, {"review_id": "lNnmoxxb_cF", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper analyzes the distillation from the bias-variance perspective. Beyond this, the regularization samples affect the performance. Based on the observation, a novel weighted mechanism is proposed to distill knowledge from teacher networks.\n\nStrengths:\n\n+) This paper is clear and easy to follow, the organization is good. The bias-variance analysis, the regularization samples, the weighted soft labels all make sense. I felt comfortable when I was reading this paper.\n\n+) The analysis is clear and reasonable. The deduction seems correct. The figures are clear.\n\n+) The experiments are enough to examine the effectiveness of the proposed weighted distillation (see below).\n\n+) The code is submitted to contribute to the community. I appreciate the submission.\n\n\nWeaknesses & Concerns:\n\n-) Sec. 3, first paragraph, $T(x, \\tau)$ -> $\\hat{y}^t = T(x, \\tau)$, $S(x, \\tau)$ -> $\\hat{y}^s = S(x, \\tau)$ to make it more clear. \n\n-) ' For loss function, we set α = 2.25 for distillation on CIFAR and α = 2.5 for ImageNet via grid search.' How many $\\alpha$s have been tested? What are the results? The main concern is that the grid search is costly in practice.  Therefore, I appreciate the analysis in this paper that helps us understanding KD better. However, the grid searched hyper-parameters makes Sec. 4 costly in practice.\n\nBased on the quality of the paper, I select 6 as the initial score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Q2IEO7ThO2v", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper shows a new perspective of tackling the knowledge distillation problem. The author(s) have decomposed the expected student's training error into the bias, variance, and irreducible noise parts. This decomposition is further rewritten as two parts: one for bias reduction and another for variance reduction. The motivation is clearly explained and the experimental results show that this new approach can improve the model training performance of the student on both CIFAR100 and Imagenet. \n\npros:\n- A novel perspective of performing knowledge distillation.\n- The motivation is clear and the framework is straightforward to understand.\n- The code is provided, therefore, I believe the reproducibility is high.\n\ncons:\n- The definition of variance is flawed. The formulation doesn't match with the figure illustration and code implementation.\n- The observed phenomenon regarding the regularization examples is not well explained.\n- The experimental details are not sufficient.\n\nconcerns:\n- My major concern comes from the definition of variance and its corresponding implementation. From my understanding, the variance should be estimated from multiple teachers, just as shown in Fig 1, which matches my thoughts. But it seems like the author(s) are trying to fade this concept intentionally. First, the expectation's definition for Equation 1 is not clear. It should take the expectation over the space of infinite teachers, or at least empirically multiple teachers with a substantial number. Second, after checking on the code, I only find one teacher is utilized in the implementation. Would you please elaborate on why this is the design? \n\n- Second concern is that the mathematical reasons behind the not-too-big or not-too-small weights for these regularization examples are unclear. This reweight scheme is purely empirically, and it would be a much solid paper with mathematical interpretations. \n\n- The bias-variance decomposition on classification has been well studied in multiple papers, e.g., [Pedro Domingos 2002], and [Giorgio Valentini 2004]. The relation (similarity and difference) between your employed BVD from [Heskes 1998] and [Pedro Domingos 2002] is not discussed. The latter [Giorgio Valentini 2004] is not even cited, which seems inappropriate. Also, comparing with the variance definition of [Zitong Yang, 2020], have you tried multiple loss functions and observe the same phenomenon? \n\n- minor issue: the sample-wise error for one example x should be explicitly listed and the notation should be consistent through the whole paper, y(x), and y should not be mixed together. I know the original [Heskes 1998] paper did the same, but you should be consistent on this notation. \n\n- other minor things: Could you please elaborate a bit more of how you preprocess the imagenet data into 1.2 million images and how many classes are remained? This piece of information seems missing and the code doesn't reveal much about it. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "FK3TtnlDSje", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides a way to research the effect of soft labels in knowledge distillation from the perspective of sample-wise bias-variance tradeoff.  Based on some observations about regularization samples, the authors propose the weighted soft labels to handle the tradeoff.  Experiments on standard datasets show that the proposed method can improve the standard knowledge distillation.\n\nThe motivation and logic of the article are clear. Based on the form of rewriting distillation loss into a regularization loss adding the direct training loss, this work introduces the sample-wise bias-variance tradeoff, which help to understand the regularization effect of KD.  Despite the weighted soft label method is heuristic, given the simplicity and effectiveness of the proposed method, I think this paper is satisfactory.\n\nStill, I would like to give the authors some suggestions:\n1.Section 3.4 should do more experiments about the intermediate state between excluding regularization samples and only on regularization samples to clearly show the tradeoff.\n2.Can the authors give some quantitative results in the Table 5 experiments to show how weighted soft labels tune sample-wise bias-variance?\n3.Can the weighted soft labels be applied to the variants of KD?  It would be better to show some results about this respect. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates the effect of soft labels in knowledge distillation from the perspective of sample-wise bias-variance tradeoff. They observe that during training the bias-variance tradeoff
varies sample-wisely. and under the same distillation temperature setting, we
 distillation performance is negatively associated with the number of regularization samples. But removing them altogether hurts the performance (the authors show empirical evidence of this). Based on some observations about regularization samples, the authors propose the weighted soft labels to handle the tradeoff. Experiments on standard datasets show that the proposed method can improve the standard knowledge distillation.

pros.
-the paper is written clearly.
-through the review period the authors added additional experiments suggested by the reviewers and enhances experimental results. The experiment results are convincing and the authors have now added explanations on hyperparameter choices.
-the mathematical setting is now clear after incorporating reviewer's comments.
-the missing related work as suggested by reviewers is added

cons.
-comparison with results of Zitong Yang et al 2020[1] is missing.

I thank the authors for incorporating the changes requested by reviewers. Please add comparison with result of [1] in the final version.

[1] Rethinking Bias-Variance Trade-off for Generalization of Neural Networks
Zitong Yang, Yaodong Yu, Chong You, Jacob Steinhardt, Yi Ma

Decision: Accept (Poster)