Rethinking Soft Labels for Knowledge Distillation: A Bias–Variance Tradeoff Perspective

1 INTRODUCTION . For deep neural networks ( Goodfellow et al. , 2016 ) , knowledge distillation ( KD ) ( Ba & Caruana , 2014 ; Hinton et al. , 2015 ) refers to the technique that uses well-trained networks to guide the training of another network . Typically , the well-trained network is named as the teacher network while the network to be trained is named as the student network . For distillation , the predictions from the teacher network are leveraged and referred to as the soft labels ( Balan et al. , 2015 ; Müller et al. , 2019 ) . Soft labels generated by the teacher network have been proven effective in large-scale empirical studies ( Liang et al. , 2019 ; Tian et al. , 2020 ; Zagoruyko & Komodakis , 2017 ; Romero et al. , 2015 ) as well as recent theoretical studies ( Phuong & Lampert , 2019 ) . However , the reason why soft labels are beneficial to the student network is still not well explained . Giving a clear theoretical explanation is challenging : The optimization details of a deep network with the common one-hot labels are still not well-studied ( Nagarajan & Kolter , 2019 ) , not to mention training with the soft labels . Nevertheless , two recent studies ( Müller et al. , 2019 ; Yuan et al. , 2020 ) shed light on the intuitions about how the soft labels work . Specifically , label smoothing , which is a special case of soft labels based training , is shown to regularize the activations of the penultimate layer to the network ( Müller et al. , 2019 ) . The regularization property of soft labels is further explored in ( Yuan et al. , 2020 ) . They hypothesize that in KD , one main reason why the soft labels work is the regularization introduced by soft labels . Based on the assumption , the authors ∗These authors contributed equally to this work . †Work done while the author was a research intern at Horizon Robotics . design a teacher-free distillation method by turning the predictions of the student network into soft labels . Considering that soft labels are targets for distillation , the evidence of the regularization brought by soft labels drives us to rethink soft labels for KD : Soft labels are both supervisory signals and regularizers . Meanwhile , it is known that there is a tradeoff between fitting the data and imposing regularizations , i.e. , the bias-variance dilemma ( Kohavi & Wolpert , 1996 ; Bishop , 2006 ) , but it is unclear how bias and variance change for distillation with soft labels . Since the bias-variance tradeoff is an important issue in statistical learning , we investigate whether the bias-variance tradeoff exists for soft labels and how the tradeoff affects distillation performance . We first compare the bias and variance decomposition of direct training with that of distillation with soft labels , noticing that distillation results in a larger bias error and a smaller variance . Then , we rewrite distillation loss into the form of a regularization loss adding the direct training loss . Through inspecting the gradients of the two terms during training , we notice that for soft labels , the biasvariance tradeoff varies sample-wisely . Moreover , by looking into a conclusion from ( Müller et al. , 2019 ) , we observe that under the same temperature setting , the distillation performance is negatively associated with the number of some certain samples . These samples lead to bias increase and variance decrease and we name them as regularization samples . To investigate how regularization samples affect distillation , we first examine if we can design ad hoc filters for soft labels to avoid training with regularization samples . But completely filtering out regularization samples also deteriorates distillation performance , leading us to speculate that regularization samples are not well handled by standard KD . In the light of these findings , we propose weighted soft labels for distillation to handle the sample-wise bias-variance tradeoff , by adaptively assigning a lower weight to regularization samples and a larger weight to the others . To sum up , our contributions are : • For knowledge distillation , we analyze how the soft labels work from a perspective of biasvariance tradeoff . • We discover that the bias-variance tradeoff varies sample-wisely . Also , we discover that if we fix the distillation temperature , the number of regularization samples is negatively associated with the distillation performance . • We design straightforward schemes to alleviate negative impacts from regularization sam- ples and then propose the novel weighted soft labels for distillation . Experiments on large scale datasets validate the effectiveness of the proposed weighted soft labels . 2 RELATED WORKS . Knowledge distillation . Hinton et al . ( 2015 ) proposed to distill outputs from large and cumbersome models into smaller and faster models , which is named as knowledge distillation . The outputs for large networks are averaged and formulated as soft labels . Also , other kinds of soft labels have been widely used for training deep neural networks ( Szegedy et al. , 2016 ; Pereyra et al. , 2017 ) . Treating soft labels as regularizers were pointed out in ( Hinton et al. , 2015 ) since a lot of helpful information can be carried in soft labels . More recently , Müller et al . ( 2019 ) showed the adverse effect of label smoothing upon distillation . It is a thought-provoking discovery for the reason that both label smoothing and distillation are exploiting the regularization property behind soft labels . Yuan et al . ( 2020 ) further investigated the regularization property of soft labels and then proposed a teacher free distillation scheme . Distillation loss . One of our main contributions is that we improve the distillation loss . For adaptively adjusting the distillation loss , Tang et al . ( 2019 ) pays attention to hard-to-learn and hard-tomimic samples , and the latter is weighted based on the prediction gap between teacher and student . However , it does not consider that the teacher may give an incorrect guide to the student , under which the prediction gap is still large and such a method may lead to the performance being hurt . Saputra et al . ( 2019 ) transfers teacher ’ s guidance only on the samples where the performance of the teacher surpasses the student , while Wen et al . ( 2019 ) deals with the incorrect guidance by probability shifting strategy . Our approach is different from the above methods , in terms of motivations as well as the proposed solutions . Bias-variance tradeoff . Bias-variance tradeoff is a well-studied topic in machine learning ( Kohavi & Wolpert , 1996 ; Domingos , 2000 ; Valentini & Dietterich , 2004 ; Bishop , 2006 ) and for neural networks ( Geman et al. , 1992 ; Neal et al. , 2018 ; Belkin et al. , 2019 ; Yang et al. , 2020 ) . Existing methods are mainly concerned with the variance brought by the choice of network models . Our perspective is different from the previous methods since we focus on the behavior of samples during training . In our work , based on the results from Heskes ( 1998 ) , we present the decomposition of distillation loss , which is defined by Kullback-Leibler divergence . Besides , our main contribution is not to study how to theoretically analyze the tradeoff , but how to adaptively tune the sample-wise tradeoff during training . 3 BIAS-VARIANCE TRADEOFF FOR SOFT LABELS One-hot Label Soft LabelLabel set B Label set A Bias Variance Figure 1 : Bias and variance . Soft labels play the role of supervisory signals and regularizations at the same time , which inspires us to rethink soft labels from the perspective of the bias-variance tradeoff . We begin our analysis with some mathematical descriptions . For a sample x labeled as i-th class , let the ground-truth label be a one-hot vector y where yi = 1 and other entries are 0 . Then for x and softmax output temperature τ , the soft label predicted by the teacher network is denoted as ŷtτ and the output from the student is denoted as ŷsτ . The soft label ŷtτ is then used for training the student by the distillation loss , i.e . Lkd = −τ2 ∑ k ŷ t k , τ log ŷ s k , τ , where ŷsk , τ , ŷ t k , τ means the k-th element of the student ’ s output ŷ s τ and the teacher ’ s output ŷ t τ , respectively . With the above notations , the cross-entropy loss for training with one-hot labels is Lce = −yk log ŷsk,1 . We now present the bias-variance decomposition for Lce and Lkd , based on the definition and notations from Heskes ( 1998 ) . First , we denote the train dataset as D and the output distribution on a sample x of the network trained without distillation as ŷce = fce ( x ; D ) . For the network trained with distillation , the model also depends on the teacher network , so we define the output on x as ŷkd = fkd ( x ; D , T ) , where T is the selected teacher network . Then , let the averaged output of ŷkd and ŷce be ȳkd and ȳce , that is , ȳce = 1 Zce exp ( ED [ log ŷce ] ) , ȳkd = 1 Zkd exp ( ED , T [ log ŷkd ] ) , ( 1 ) where Zce , Zkd are two normalization constant . Then according to Heskes ( 1998 ) , we have the following decomposition for the expected error on the sample x and y = t ( x ) is the ground truth label : errorce = Ex , D [ −y log ŷce ] = Ex , D [ −y logy + y log y ȳce + y log ȳce ŷce ] = Ex [ −y logy ] + Ex [ y log y ȳce ] + ED [ Ex [ y log ȳce ŷce ] ] = Ex [ −y logy ] +DKL ( y , ȳce ) + ED [ DKL ( ȳce , ŷce ) ] = intrinsic noise + bias + variance , ( 2 ) where DKL is the Kullback-Leibler divergence . The derivation of the variance term is based on the facts that log ȳceED [ log ŷce ] is a constant and Ex [ y ] = Ex [ ȳce ] = 1 . Detailed derivations can be found from Eq . ( 4 ) in Heskes ( 1998 ) . Next , we analyze the bias-variance decomposition of Lkd . As mentioned above , when training with soft labels , extra randomness is introduced for the selection of a teacher network . In Fig . 1 , we illustrate the corresponding bias and variance for the selection process of a set of soft labels , which are generated by a teacher network . In this case , a high variance model indicates the model ( grey point ) is closer to the one-hot trained model ( black point ) , while a low variance model indicates that the model is closer to other possible models trained with soft labels ( red points ) . Although for KD there are more sources introducing randomness , the overall variance brought by Lkd is not necessarily higher than Lce . In fact , existing empirical results strongly suggest that the overall variance is smaller with KD . For example , students trained with soft labels are better calibrated than one-hot baselines ( Müller et al. , 2019 ) and KD makes the predictions of students more consistent when facing adversarial noise ( Papernot et al. , 2016 ) . Here , we present these empirical evidence as an assumption : Assumption 1 The variance brought by KD is smaller than direct training , that is , ED , T [ DKL ( ȳkd , ŷkd ) ] 6 ED [ DKL ( ȳce , ŷce ) ] . Similar to Eq . ( 2 ) , we write the decomposition for Lkd as errorkd = Ex [ −y logy ] +DKL ( y , ȳce ) + Ex [ y log ( ȳce ȳkd ) ] + ED , T [ DKL ( ȳkd , ŷkd ) ] . ( 3 ) An observation here is that ȳce converges to one-hot labels while ȳkd converges to soft labels , so ȳce is closer to the one-hot ground-truth distribution y than ȳkd , i.e. , Ex [ y log ( ȳce ȳkd ) ] > 0 . If we rewrite Lkd as Lkd = Lkd − Lce + Lce , then Lkd − Lce causes that the bias increases by Ex [ y log ( ȳce ȳkd ) ] and the variance decreases by ED [ DKL ( ȳce , ŷce ) ] − ED , T [ DKL ( ȳkd , ŷkd ) ] . From the above analysis , we separate Lkd into two terms , and Lkd−Lce leads to variance reduction , and Lce leads to bias reduction . In the following sections , we first analyze how Lkd − Lce links to the bias-variance tradeoff during training . Then we analyze the changes in the relative importance between bias reduction and variance reduction during training with soft labels .

Summary: The paper shows a new perspective of tackling the knowledge distillation problem. The author(s) have decomposed the expected student's training error into the bias, variance, and irreducible noise parts. This decomposition is further rewritten as two parts: one for bias reduction and another for variance reduction. The motivation is clearly explained and the experimental results show that this new approach can improve the model training performance of the student on both CIFAR100 and Imagenet.