Rethinking Soft Labels for Knowledge Distillation: A Bias–Variance Tradeoff Perspective

1 INTRODUCTION . For deep neural networks ( Goodfellow et al. , 2016 ) , knowledge distillation ( KD ) ( Ba & Caruana , 2014 ; Hinton et al. , 2015 ) refers to the technique that uses well-trained networks to guide the training of another network . Typically , the well-trained network is named as the teacher network while the network to be trained is named as the student network . For distillation , the predictions from the teacher network are leveraged and referred to as the soft labels ( Balan et al. , 2015 ; Müller et al. , 2019 ) . Soft labels generated by the teacher network have been proven effective in large-scale empirical studies ( Liang et al. , 2019 ; Tian et al. , 2020 ; Zagoruyko & Komodakis , 2017 ; Romero et al. , 2015 ) as well as recent theoretical studies ( Phuong & Lampert , 2019 ) . However , the reason why soft labels are beneficial to the student network is still not well explained . Giving a clear theoretical explanation is challenging : The optimization details of a deep network with the common one-hot labels are still not well-studied ( Nagarajan & Kolter , 2019 ) , not to mention training with the soft labels . Nevertheless , two recent studies ( Müller et al. , 2019 ; Yuan et al. , 2020 ) shed light on the intuitions about how the soft labels work . Specifically , label smoothing , which is a special case of soft labels based training , is shown to regularize the activations of the penultimate layer to the network ( Müller et al. , 2019 ) . The regularization property of soft labels is further explored in ( Yuan et al. , 2020 ) . They hypothesize that in KD , one main reason why the soft labels work is the regularization introduced by soft labels . Based on the assumption , the authors ∗These authors contributed equally to this work . †Work done while the author was a research intern at Horizon Robotics . design a teacher-free distillation method by turning the predictions of the student network into soft labels . Considering that soft labels are targets for distillation , the evidence of the regularization brought by soft labels drives us to rethink soft labels for KD : Soft labels are both supervisory signals and regularizers . Meanwhile , it is known that there is a tradeoff between fitting the data and imposing regularizations , i.e. , the bias-variance dilemma ( Kohavi & Wolpert , 1996 ; Bishop , 2006 ) , but it is unclear how bias and variance change for distillation with soft labels . Since the bias-variance tradeoff is an important issue in statistical learning , we investigate whether the bias-variance tradeoff exists for soft labels and how the tradeoff affects distillation performance . We first compare the bias and variance decomposition of direct training with that of distillation with soft labels , noticing that distillation results in a larger bias error and a smaller variance . Then , we rewrite distillation loss into the form of a regularization loss adding the direct training loss . Through inspecting the gradients of the two terms during training , we notice that for soft labels , the biasvariance tradeoff varies sample-wisely . Moreover , by looking into a conclusion from ( Müller et al. , 2019 ) , we observe that under the same temperature setting , the distillation performance is negatively associated with the number of some certain samples . These samples lead to bias increase and variance decrease and we name them as regularization samples . To investigate how regularization samples affect distillation , we first examine if we can design ad hoc filters for soft labels to avoid training with regularization samples . But completely filtering out regularization samples also deteriorates distillation performance , leading us to speculate that regularization samples are not well handled by standard KD . In the light of these findings , we propose weighted soft labels for distillation to handle the sample-wise bias-variance tradeoff , by adaptively assigning a lower weight to regularization samples and a larger weight to the others . To sum up , our contributions are : • For knowledge distillation , we analyze how the soft labels work from a perspective of biasvariance tradeoff . • We discover that the bias-variance tradeoff varies sample-wisely . Also , we discover that if we fix the distillation temperature , the number of regularization samples is negatively associated with the distillation performance . • We design straightforward schemes to alleviate negative impacts from regularization sam- ples and then propose the novel weighted soft labels for distillation . Experiments on large scale datasets validate the effectiveness of the proposed weighted soft labels . 2 RELATED WORKS . Knowledge distillation . Hinton et al . ( 2015 ) proposed to distill outputs from large and cumbersome models into smaller and faster models , which is named as knowledge distillation . The outputs for large networks are averaged and formulated as soft labels . Also , other kinds of soft labels have been widely used for training deep neural networks ( Szegedy et al. , 2016 ; Pereyra et al. , 2017 ) . Treating soft labels as regularizers were pointed out in ( Hinton et al. , 2015 ) since a lot of helpful information can be carried in soft labels . More recently , Müller et al . ( 2019 ) showed the adverse effect of label smoothing upon distillation . It is a thought-provoking discovery for the reason that both label smoothing and distillation are exploiting the regularization property behind soft labels . Yuan et al . ( 2020 ) further investigated the regularization property of soft labels and then proposed a teacher free distillation scheme . Distillation loss . One of our main contributions is that we improve the distillation loss . For adaptively adjusting the distillation loss , Tang et al . ( 2019 ) pays attention to hard-to-learn and hard-tomimic samples , and the latter is weighted based on the prediction gap between teacher and student . However , it does not consider that the teacher may give an incorrect guide to the student , under which the prediction gap is still large and such a method may lead to the performance being hurt . Saputra et al . ( 2019 ) transfers teacher ’ s guidance only on the samples where the performance of the teacher surpasses the student , while Wen et al . ( 2019 ) deals with the incorrect guidance by probability shifting strategy . Our approach is different from the above methods , in terms of motivations as well as the proposed solutions . Bias-variance tradeoff . Bias-variance tradeoff is a well-studied topic in machine learning ( Kohavi & Wolpert , 1996 ; Domingos , 2000 ; Valentini & Dietterich , 2004 ; Bishop , 2006 ) and for neural networks ( Geman et al. , 1992 ; Neal et al. , 2018 ; Belkin et al. , 2019 ; Yang et al. , 2020 ) . Existing methods are mainly concerned with the variance brought by the choice of network models . Our perspective is different from the previous methods since we focus on the behavior of samples during training . In our work , based on the results from Heskes ( 1998 ) , we present the decomposition of distillation loss , which is defined by Kullback-Leibler divergence . Besides , our main contribution is not to study how to theoretically analyze the tradeoff , but how to adaptively tune the sample-wise tradeoff during training . 3 BIAS-VARIANCE TRADEOFF FOR SOFT LABELS One-hot Label Soft LabelLabel set B Label set A Bias Variance Figure 1 : Bias and variance . Soft labels play the role of supervisory signals and regularizations at the same time , which inspires us to rethink soft labels from the perspective of the bias-variance tradeoff . We begin our analysis with some mathematical descriptions . For a sample x labeled as i-th class , let the ground-truth label be a one-hot vector y where yi = 1 and other entries are 0 . Then for x and softmax output temperature τ , the soft label predicted by the teacher network is denoted as ŷtτ and the output from the student is denoted as ŷsτ . The soft label ŷtτ is then used for training the student by the distillation loss , i.e . Lkd = −τ2 ∑ k ŷ t k , τ log ŷ s k , τ , where ŷsk , τ , ŷ t k , τ means the k-th element of the student ’ s output ŷ s τ and the teacher ’ s output ŷ t τ , respectively . With the above notations , the cross-entropy loss for training with one-hot labels is Lce = −yk log ŷsk,1 . We now present the bias-variance decomposition for Lce and Lkd , based on the definition and notations from Heskes ( 1998 ) . First , we denote the train dataset as D and the output distribution on a sample x of the network trained without distillation as ŷce = fce ( x ; D ) . For the network trained with distillation , the model also depends on the teacher network , so we define the output on x as ŷkd = fkd ( x ; D , T ) , where T is the selected teacher network . Then , let the averaged output of ŷkd and ŷce be ȳkd and ȳce , that is , ȳce = 1 Zce exp ( ED [ log ŷce ] ) , ȳkd = 1 Zkd exp ( ED , T [ log ŷkd ] ) , ( 1 ) where Zce , Zkd are two normalization constant . Then according to Heskes ( 1998 ) , we have the following decomposition for the expected error on the sample x and y = t ( x ) is the ground truth label : errorce = Ex , D [ −y log ŷce ] = Ex , D [ −y logy + y log y ȳce + y log ȳce ŷce ] = Ex [ −y logy ] + Ex [ y log y ȳce ] + ED [ Ex [ y log ȳce ŷce ] ] = Ex [ −y logy ] +DKL ( y , ȳce ) + ED [ DKL ( ȳce , ŷce ) ] = intrinsic noise + bias + variance , ( 2 ) where DKL is the Kullback-Leibler divergence . The derivation of the variance term is based on the facts that log ȳceED [ log ŷce ] is a constant and Ex [ y ] = Ex [ ȳce ] = 1 . Detailed derivations can be found from Eq . ( 4 ) in Heskes ( 1998 ) . Next , we analyze the bias-variance decomposition of Lkd . As mentioned above , when training with soft labels , extra randomness is introduced for the selection of a teacher network . In Fig . 1 , we illustrate the corresponding bias and variance for the selection process of a set of soft labels , which are generated by a teacher network . In this case , a high variance model indicates the model ( grey point ) is closer to the one-hot trained model ( black point ) , while a low variance model indicates that the model is closer to other possible models trained with soft labels ( red points ) . Although for KD there are more sources introducing randomness , the overall variance brought by Lkd is not necessarily higher than Lce . In fact , existing empirical results strongly suggest that the overall variance is smaller with KD . For example , students trained with soft labels are better calibrated than one-hot baselines ( Müller et al. , 2019 ) and KD makes the predictions of students more consistent when facing adversarial noise ( Papernot et al. , 2016 ) . Here , we present these empirical evidence as an assumption : Assumption 1 The variance brought by KD is smaller than direct training , that is , ED , T [ DKL ( ȳkd , ŷkd ) ] 6 ED [ DKL ( ȳce , ŷce ) ] . Similar to Eq . ( 2 ) , we write the decomposition for Lkd as errorkd = Ex [ −y logy ] +DKL ( y , ȳce ) + Ex [ y log ( ȳce ȳkd ) ] + ED , T [ DKL ( ȳkd , ŷkd ) ] . ( 3 ) An observation here is that ȳce converges to one-hot labels while ȳkd converges to soft labels , so ȳce is closer to the one-hot ground-truth distribution y than ȳkd , i.e. , Ex [ y log ( ȳce ȳkd ) ] > 0 . If we rewrite Lkd as Lkd = Lkd − Lce + Lce , then Lkd − Lce causes that the bias increases by Ex [ y log ( ȳce ȳkd ) ] and the variance decreases by ED [ DKL ( ȳce , ŷce ) ] − ED , T [ DKL ( ȳkd , ŷkd ) ] . From the above analysis , we separate Lkd into two terms , and Lkd−Lce leads to variance reduction , and Lce leads to bias reduction . In the following sections , we first analyze how Lkd − Lce links to the bias-variance tradeoff during training . Then we analyze the changes in the relative importance between bias reduction and variance reduction during training with soft labels .

Summary: In this paper, the authors studied the soft labels for knowledge distillation from a bias-variance tradeoff perspective. Specifically, the authors first provide a mathematically descriptions of the bias-variance decomposition in knowledge distillation. Then, based on the theoretically analysis and experiments, the authors proposed an novel weighted soft labels to help the network adaptively handle the sample-wise bias-variance tradeoff.