Year: 2021

Abstract:
Knowledge distillation aims to obtain a small and effective deep model (student) by learning the output from a larger model (teacher). Previous studies found a severe degradation problem, that student performance would degrade unexpectedly when distilled from oversized teachers. It is well known that larger models tend to have sharper outputs (Guo et al., 2017). Based on this observation, we found that the sharpness gap between the teacher and student output may cause this degradation problem. To solve this problem, we first propose a metric to quantify the sharpness of the model output. Based on the second-order Taylor expansion of this metric, we propose Adaptive Temperature Knowledge Distillation (ATKD), which automatically changes the temperature of the teacher and the student, to reduce the sharpness gap Gs gap. We conducted extensive experiments on CIFAR100 and ImageNet and achieved significant improvements. Specifically, ATKD trained the best ResNet18 model on ImageNet as we knew (73.0% accuracy).

heading: 1 INTRODUCTION
text: Deep neural networks have achieved remarkable success in most of computer vision tasks (He et al., 2016; Deng et al., 2009), while the increasing network capacity in the current state-of-the-art models also results in severe computational burdens and high inference time (Tian et al., 2020; Yim et al., 2017). One direction to reduce the model size is knowledge distillation (KD), which trains small efficient models (student) by learning from the output of large models (teacher) (Tang et al., 2020). In 2015, Hinton proposed to soften the model output by raising the temperature of the last softmax layer(Hinton et al., 2015). Since then, this temperature-based kowledge distillation has drawn the main stream attentions and achieved many successes (Zagoruyko & Komodakis, 2017; Kim et al., 2020; Furlanello et al., 2018; Fu et al., 2020).
However, recent researches found that knowledge distillation suffers from a mysterious performance degradation problem(Cho & Hariharan, 2019; Mirzadeh et al., 2019). Specifically, since the idea of KD is transferring the knowledge of teacher into student, a nature assumption is larger teachers would train better students. But recent researches refuted this assumption and found that the student often performs worse with oversized teachers(Cho & Hariharan, 2019).
This paper studies this degradation problem by considering the sharpness (or softness) of the model output. It is well known that networks with more parameters tend to produce sharper output (Guo et al., 2017; Lee et al., 2018), which would create the gap of sharpness between teacher and student, i.e. the teacher is usually much sharper than the student. It should be noted that temperature technology in vanilla knowledge distillation does not reduce this gap. This is because in vanilla knowledge distillation, the teacher and student are softened with the same temperature, which maintains the large gap of sharpness. When the teacher grows larger, this gap will further increases, making it more difficult for the student to learn from the teacher.
To solve this problem, let us consider a modified softer teacher model with the same accuracy. In this way, the sharpness gap between the student and the teacher models would be narrowed, making learning more accessible. One way to build this softer teacher is to use higher temperatures for the teacher model, but it would require labor-intensive searches on the validation data set.
With this insight, we propose to adaptively change the temperatures of the teacher and the student by their sharpness. Our contribution is three-fold:
1. We propose a metric to quantify the sharpness of models, allowing us to precisely control the sharpness of model output. Formally, we use the realsoftmax function (Nielsen & Sun, 2016) on the logits vector as the sharpness metric, which approximates a smoother version of the max function. If we denote student logits as z, the teacher logits as v, teacher temperature and student temperature as τT and τS respectively (τT is equal to τS in the vanilla knowledge distillation), the sharpness metric and the gap of sharpness are defined as follows:
Ssharpness = log ∑ j ezj/τ
Gs gap = log ∑ j evj/τT − log ∑ j ezj/τS (1)
2. We propose Adaptive Temperature Knowledge Distillation (ATKD), which automatically changes the temperatures of the teacher and the student, to reduce the sharpness gapGs gap. ATKD relieves the burden of searching for the appropriate temperature on the validation set. Specifically, by Taylor second-order expansion, our method can be implemented easily by normalizing the logits with the standard deviation.
3. Our Ssharpness metric can be used to explain why two existing methods work. Specifically, the Early Stopped method and the Teacher Assistant method both reduce the sharpness gap between the teacher and the student.
We present comprehensive experiments on CIFAR-100 Krizhevsky et al. (2009) and ImageNet datasets (Deng et al., 2009) to evaluate our method. The experiment results show that: 1) The proposed ATKD method tremendously mitigates the performance degradation problem and trains the best ResNet18 model on ImageNet as we know (73.01% accuracy). 2) The proposed ATKD method is easy to optimize, exhibits much lower training/test loss with oversized teachers than KD.

heading: 2 METHODOLOGY

heading: 2.1 BACKGROUND
text: Vanillay Knowledge Distillation During training, we minimize the negative log likelihood of the ground truth class to update model parameters. After the model is properly trained, the probability of the ground truth would be close to 1, while the rest wrong predictions are near zeros.
In 2015, Hinton noticed that these small wrong probabilities are useful to unveil “dark knowledge” (Hinton et al., 2015). Take a picture of “cat” for example, the model are more likely to output higher probability for class “dog” than class “airplane”. These wrong probabilities imply the relationship between the two classes and unveil how a model tends to generalize. This observation inspired to use the output of large models as soft targets to train efficient small models.
However, modern deep networks tend to produce peaky probabilities (Guo et al., 2017; Lee et al., 2018), that the numbers of those wrong classes (near zero values) would be negligible compared to the ground truth (near one). Thus Hinton proposed to raise the temperature of the last softmax to soften the output probabilities, which can be used as soft targets to train small networks.
We denote logits of the teacher as v, and logits of the student as z, temperature as τ , i and j denote the ith and jth value of logits (i.e. the ith and jth category of K classes). the loss of knowledge distillation is as follows:
LKD = − ∑ i pTi log p S i pSi = ezi/τ∑ j e zj/τ , pTi = evi/τ∑ j e vj/τ
(2)
The final loss for the student is then the weighted sum of the cross entropy loss Lcls and the knowledge distillation loss LKD:
L = λLKD + (1− λ)Lcls (3)
The popular choice of the temperature τ is in {3, 4, 5} and the weight λ = 0.9 (Hinton et al., 2015; Cho & Hariharan, 2019; Tian et al., 2020).
Performance Degradation Problem While knowledge distillation achieved success in many fields, a mysterious performance degradation problem was spotted in 2019. Since the idea of knowledge distillation is transferring teacher knowledge into students, one natural hypothesis is that a larger and more accurate teacher would capture more knowledge and thus train better students. Unfortunately, previous studies invalidate this hypothesis by showing that the student performance would degenerate unexpectedly with larger teachers.
Table 1 shows our experiment result. The student is ResNet14. With larger teachers, the student accuracy degrades, and the KD loss increase. Cho & Hariharan (2019) hypothesizes that the mismatch of capacity causes this problem. We will discuss this problem in the rest of the section.

heading: 2.2 ADAPTIVE TEMPERATURE KNOWLEDGE DISTILLATION
text: We investigated the degradation problem and proposed to use adaptive temperatures during training to mitigate this problem:
1. We found that the mismatch sharpness between the teacher and the student may cause the degradation problem.
2. We propose a metric to quantify the sharpness of a model.
3. We propose the adaptive temperature knowledge distillation (ATKD) methods.
The sharpness of neural networks Two main theories explain why knowledge distillation is effective. The first theory originated from Hinton’s first paper, which argued that teachers are more accurate in capturing category similarities. This information of category similarity helps students to generalize better on unseen data. On the other hand, some contend that the soft output of the teacher prevents the student network from overconfidence (ie. label smoothing technique).
However, when the teacher gets larger, these two theories will contradict each other. This is because that larger models are usually more accurate and peaky at the same time. On the one hand, a more accurate teacher is beneficial to the student model. On the other hand, a sharper teacher tends to make the student overconfident. Therefore, the larger teacher capacity is a double-edged sword for distillation. This analysis shows that the sharpness of teachers should be investigated carefully, while previous studies have overlooked this critical attribute of teachers.
To this end, we propose to use realsoftmax (Nielsen & Sun, 2016) to quantify the sharpness of the model output, which is defined as the logarithm of the sum of the exponentials of the logits:
Ssharpness = log ∑ j ezj (4)
Such metric has at least two advantages, therefore reasonably measure the sharpness of the model output: 1) This metric is a smooth approximation to the maximum function maxj(z) 2) It is also differentiable.
Meanwhile, we quantify the sharpness gap between the teacher and the student as the difference of the sharpness metric:
Gs gap = log ∑ j evj − log ∑ j ezj (5)
If we consider the temperatures, the sharpness gap is:
Gs gap = log ∑ j evj/τ − log ∑ j ezj/τ (6)
Table 2 shows sharpness metric value for models of different capacity. It shows that Ssharpness increases with network capacity, and there is a sharpness gap between networks of different sizes.
Adaptive Temperature One idea is using higher temperatures for the teacher to reduce the sharpness gap between teachers and students. However, searching on the validation data set requires a lot of computational costs. Instead, we propose adaptive temperature knowledge distillation, which automatically tunes the temperature according to the Ssharpness.
By Taylor second expansion:
Gs gap = log ∑ j evj/τ − log ∑ j ezj/τ
≈ log(K + ∑ j vj/τ + 1 2 ∑ j v2j /τ 2)
− log(K + ∑ j zj/τ + 1 2 ∑ j z2j /τ 2)
(7)
We follow the assumption from Hinton (Hinton et al., 2015), that the logits have been zero-meaned separately for each training example so that ∑ j zj = ∑ j zj = 0. We found from experiments that the sum of logits is indeed very small numbers close to zero. The experiment results and further theoretical analysis are provided in the appendix.
Given the above assumption ∑ j zj = ∑ j zj = 0, We can get:
Gs gap = log(K + 1
2 ∑ j (vj/τ) 2)− log(K + 1 2 ∑ j (zj/τ) 2)
= log(1 + 1
2τ2K ∑ j v2j − log(1 + 1 2τ2K ∑ j z2j ))
(8)
The 1K ∑ j v 2 j in the equation is the variance of logits, which can be represented by the square of the standard deviation std2:
Gs gap = log(1 + 1 2 ∗ (stdT /τ)2)− log(1 + 1 2 ∗ (stdS/τ)2) (9)
This result shows the relationship between the temperature and sharpness gap is pretty straightforward, that the standard deviation controls the sharpness gap between the teacher and the student. Therefore, we set the temperature of the teacher and the student as the standard deviation of the logits vector respectively. These adaptive temperatures would automatically change during training, reducing the sharpness gap between the teacher and the student. It is worth noting that there is no need to search for the proper temperature on the validation set like the vanilla KD.
Formally, if we denote standard deviation function as Std, we propose Adaptive Temperature Knowledge Distillation as follows:
τTi = Std(vi), τ S i = Std(zi) LATKD = − ∑ i pTi log p S i
pSi = ezi/τ S i∑
j e zj/τSi
, pTi = evi/τ T i∑
j e vj/τTi
L = λLATKD + (1− λ)Lcls
(10)

heading: 2.3 SHARPNESS GAP ANALYSIS
text: We investigated the relationship between the sharpness gap and the temperature, and found that the Gs gap of ATKD decrease with 1τ3 , while the vanilla KD decrease with 1 τ2 . Therefore, If the temperature of KD and adaptive temperatures of ATKD are in the same temperature range, the sharpness gap of ATKD Gs gap would be smaller than KD in a considerable margin.
If we use K to denote K classes, the sharpness gap would decrease with 1τ2 under vanilla KD: Gs gap = log ∑ j evj/τ − log ∑ j ezj/τ
= log(1 +O( 1 τ2 ))− log(1 +O( 1 τ2 )) + logK − logK ≈O( 1 τ2 )
(11)
This shows that when the temperature is very high and the linear function (1+x) can approximate ex, Gs gap would be reduced. However, with this linear approximation, the KD loss would degenerate into MSE loss, as Hinton points out (Hinton et al., 2015):
∂L ∂zi = 1 T ( ezi/τ∑ j e zj/τ − e vi/τ∑ j e vj/τ ) (12)
≈ 1 T
( zi/τ + 1
K − vi/τ + 1 K
) (13)
≈ 1 Kτ2 (zi − vi) (14)
However, high temperatures could be harmful to student, because it would encourage the student to learn more from those negative logits of the teacher (Hinton et al., 2015), and as Hinton pointed out, these negative values of logits could be very noisy.
On the other hand, the sharpness gap of ATKD would decrease faster with 1τ3 , and enable the ATKD to achieve much lower sharpness gap than KD. Because we can use second-order Taylor approximation (1+x+ 12x
2), which is a better approximation than linear function (1+x) in a wide range.
Gs gap = log ∑ j evj/τ T − log ∑ j ezj/τ S
= log(1 + 1 2 ∗ (stdS/τS)2 +O(( 1 τS )3))
− log(1 + 1 2 ∗ (stdT /τT )2 +O(( 1 τT )3))
≈O(( 1 τT )3)−O(( 1 τS )3)
(15)

heading: 3 EXPERIMENTS
text: In this section, we will show comprehensive experiment results to validate the effectiveness of ATKD from several perspectives. Specifically, we first conducts experiments on two popular CV datasets
to demonstrate the performance of ATKD. Then we focused on evaluating whether it could alleviate the performance degradation problem.
Dataset 1) CIFAR-100 (Krizhevsky et al., 2009) is a relatively small data set and is widely used for testing various of deep learning methods. CIFAR-100 contains 50,000 images in the training set and 10,000 images in the dev set, divided into 100 fine-grained categories. 2) ImageNet (Deng et al., 2009) is a much larger one than CIFAR-100. ImageNet contains 1.2M images for training and 50K for validation, that distributes in 1000 classes.
CIFAR Experimental settings We run a total of 240 epochs for all methods. The learning rate is initialized as 0.05, then decay by 0.1 every 30 epochs after 150 epochs. Temperature is 4 for vanilla KD, and the weight of ATKD or KD and cross-entropy is 0.9 and 0.1 for all the settings.
ImageNet Experimental settings Here we use ResNet18 as student for all of methods. Training settings like learning rate or training epochs are the same with Heo et al. (2019) for ImageNet. The teacher network is well-trained previously and fixed during training.

heading: 3.1 CIFAR-100 AND IMAGENET
text: Baselines. We selected many SOTA KD methods to evaluate the performances of ATKD. Knowledge defined from intermediate layers: FitNet (Romero et al., 2015), AT (Zagoruyko & Komodakis, 2017), SP (Tung & Mori, 2019), PKT (Passalis & Tefas, 2018), FT (Kim et al., 2020), FSP (Yim et al., 2017) . 1) Knowledge defined via mutual information: CC (Peng et al., 2019), VID (Ahn et al., 2019), CRD (Tian et al., 2020). 2) Structured Knowledge: RKD (Park et al., 2019). 3) Knowledge from logits: KD (Hinton et al., 2015), NST (Huang & Wang, 2017), ES (Cho & Hariharan, 2019), TA (Mirzadeh et al., 2019)
Results in CIFAR-100. Table 3 shows that ATKD always has an outstanding improvement compared with all other methods. In some situations (e.g. those where teacher/student is WRN-402/WRN-40-1 or ResNet110/ResNet32), the performances of ATKD are even very close to those of teacher.
Results in ImageNet. All experiments used ResNet34 as the teacher and ResNet18 as the student. Table 4 (the Top1 accuracy) shows that ATKD exceeds all of the previous SOTA by a large margin. Before ATKD, the improvement of this task is limited. Fig. 1 shows the training process of
vanilla KD and ATKD. It is worth noting that ATKD provides comparable performance to KD’s final performance after first 30th epoch training.

heading: 3.2 PERFORMANCE DEGRADATION EXPERIMENTS
text: CFIFAR-100 On CIFAR-100 task, We trained the ResNet14 with multiple teachers on the CIFAR100 dataset, and the result is shown in Table 5. Experimental details follows the above settings. Under ATKD, student performance continues to increase as the teacher gets bigger. The training loss of ATKD is also much lower than the vanilla KD. In addition, the Gs gap (computed with temperatures) metric shows that ATKD significantly reduces the sharpness gap.
In the analysis of section 2.3, we concluded that larger temperatures would have smaller sharpness gap. Therefore the comparison between KD and ATKD need to take temperature values into account. The temperature of KD is 4, while most of the mean adaptive temperatures used by ATKD are between (3, 4). We provide more details in the appendix.
ImageNet Table 6 shows the degradation problem in ImageNet with ResNet18 as the student, and ’-’ denote that this specific experiment was not conducted in the cited paper. We compared ATKD with two previous methods that aim to alleviate the degradation problem, Early Stop (Cho & Hariharan, 2019) (ES) and Teacher Assistant (Mirzadeh et al., 2019) (TA). Both of these two methods explicitly regularizing the teacher capacity: 1) TA proposed to distill the large teacher to an intermediate teacher and then distill to the student, so that each knowledge distillation step has a better match between student and teacher capacity. 2) ES methods use the early stopped teacher, the teacher capacity would be regularized by fewer training steps.
We can see that ATKD exceeds Early Stop and TA methods with a large margin in all teacher settings. For example, when distilled by ResNet50 and ResNet152, the performance exceeds other methods by 2%. We obtained 73.01% accuracy, which is the best ResNet18 results as we know.
An interesting finding of the early stopped teacher method is that these regularized teachers have smaller Ssharpness, which can reduce the sharpness gap between the teacher and the student. The results is in Table 7. Similarly, the Teacher Assistant method also reduces the sharpness gap by using medium-sized teachers. This finding indicates that Ssharpness metric could provide a universal framework to mitigate the degradation problem.
Table 7: The Ssharpness of Early Stopped models, temperature set to 1.
Network ResNet14 ResNet20 ResNet32 ResNet44 ResNet56 ResNet110
Ssharpness (240 epochs) 12.20 13.11 13.84 14.45 15.37 16.13 Ssharpness (60 epochs) - 13.03 13.68 14.29 14.77 15.69
1. CIFAR-100 Training Loss 2. ImageNet Test Accuracy
Figure 1: 1. The training/test loss on CIFAR-100. With the teacher size grows, the loss of ATKD increases slower than KD, which shows that ATKD alleviates the Degradation Performance Problem. 2. The training process of KD and ATKD on ImageNet. ATKD achieves comparable accuracy with KD at the 30th epoch.

heading: 4 RELATED WORK
text: Knowledge Distillation Buciluǎ et al. (2006) first proposed compressing a trained cumbersome model into a smaller model by matching the logits between them. Then Hinton et al. (2015) advanced this idea and formed a more widely used framework known as knowledge distillation (KD). Knowledge distillation tries to minimize the KL divergence between the soft output probabilities generated by the logits through softmax. Xu et al. (2020) proposed to normalize the feature, the penultimate layer of the network, to perform distillation. This method is similar to our methods except that we perform on the logits layer. Knowledge distillation is also a kind of soft label training method. Similar to label smoothing, previous studies have found that knowledge distillation helps to regularize the training of network.
Performance Degradation Problem: Although distillation has shown a great potential in many tasks, a mysterious problem is found that larger teachers often harm the distillation performance, despite its more powerful ability (Cho & Hariharan, 2019; Mirzadeh et al., 2019). This problem is particularly severe on ImageNet, resulting in poor performance for KD. It was widely accepted that the mismatch of capacity caused this problem. Previous research proposed to regularize the teacher capacity to alleviate this problem heuristically. Cho & Hariharan (2019) proposed to early stop the training of the teacher. Moreover, Mirzadeh et al. (2019) proposed to use a medium-size teacher assistant (TA) to perform a sort of sequence distillation. This TA first learns from the teacher, then the student can learn from this TA. However, the accuracy of the early stopped teacher or TA is also harmed, which is lower than the original teacher.

heading: 5 CONCLUSION
text: The vanilla knowledge distillation overlooks the sharpness gap between the teacher and the student, which may cause the performance degradation problem. In this paper, we propose to use a metric to measure the sharpness of neural networks, allowing us to control the sharpness of models by using adaptive temperatures. Recently some papers have focused on combining label smoothing and knowledge distillation to train neural networks (Shen et al., 2021; Müller et al., 2019). An appealing point of view is that ATKD is inherently consistent with these methods. ATKD smooths teachers by adapting temperatures, while these methods let students learn from teachers trained by label smoothing. The deeper relationship between knowledge distillation and label smoothing remains to be explored.

heading: A APPENDIX

heading: A.1 EXPERIMENTS OF DIFFERENT FIXED TEMPERATURES
text: We conducted experiments of different fixed temperatures for vanilla KD (Table 8), which shows no observable performance improvement. We distilled ResNet14 by ResNet56 on CIFAR-100 dataset, and the rest details followed the setting in section 3.

heading: A.2 EXPERIMENTS VARIES WITH WIDTH
text: We add degradation problem experiments where teacher models vary with width here (Table 9). We use Wide ResNet to train on the CIFAR-100 dataset, the rest experiment details follow the setting in section 3.

heading: A.3 ENTROPY SCORE OF MODELS
text: Entropy was used as confidence score at Pereyra et al. (2017), that is related to the sharpness of model output. We measured the entropy values of different models (Table 10). Experimental details follow the setting in section 3.

heading: A.4 SHARPNESS GAP DURING TRAINING
text: Figure 2 shows the sharpness gap changes during the training. ResNet20 and WRN-16-1 are used as students, and ResNet56 and WRN-16-3 are used as teachers respectively.

heading: A.5 EXPERIMENT ON SVHN
text: We add another dataset SVHN here (Goodfellow et al., 2013). Results show in table 11. SVHN is similar to MNIST (e.g., the images are of small cropped digits), but comes from real house numbers in Google Street View images. We use ResNet14 as the student. The rest experiment settings use the setting of CIFAR-100 in section 3. Further exploration on parameter tuning might get better results.

heading: A.6 THE SUM OF LOGITS
text: In the paper of Hinton et al. (2015), they presume that the ∑ j zj equals to zero during training without further analysis (In page 3, Eq. 3). Here, we analyze this assumption theoretically and experimentally.
It is well known that the gradient to logits zj is:
∂L ∂zj = pj − qj (16)
Where pj is the jth class probability of the student and the qj is the probability of the teacher. We can get the gradient to ∑ zj by adding these gradients:∑
j
∂L ∂zj = ∑ j (pj − qj) = 1− 1 = 0 (17)
Therefore, the gradient to the sum of logits is zero.
Besides, we know that the logits is the product of the weight W and the feature vector f in the penultimate layer:
z =Wf = ∑ i Wi ∗ fi (18)
Where Wi is the ith column vector of W , considering that the initial weights of W are sampled from the zero-meaned normal distribution in the popular initialization settings (He et al., 2015), we assume that the initialized ∑ jWij (the sum of column vector) equals to zero. Therefore, the ∑ zj should be zero all the time given the above assumptions.
We conducted experiments to verify this assumption. We train these models on the CIFAR-100 dataset. Table 12 shows that all these models logits are close to zero.

heading: A.7 THE ADAPTIVE TEMPERATURES
text: We provide the mean value of the temperatures in table 13.

Review summaries:
The paper proposes to mitigate the performance degradation by controlling the sharpness gap between a large teacher and a student model. The sharpness is defined as the real softmax function (the logarithm of the sum of the exponentials of the logits). During the training, the temperature is set according to the sharpness of the logits. For the teacher model which has a sharp output, the temperature is larger so that the output becomes more smooth.  

This work explores how the sharpness gap between a teacher model and a student model affects knowledge distillation performance. The paper argues that a large sharpness gap harms KD and proposes an adaptive temperature strategy to speed up the decrease of the sharpness gap in training. The experiments demonstrate a high correlation between the sharpness gap and student performance. It also shows that the proposed adaptive temperature strategy can achieve SOTA results on the image classification tasks.

This paper is to improve the vanilla knowledge distillation method. Based on the observation in Cho&Hariharan that student degrades by oversized teachers, this paper proposed an adaptive temperature solution. A new metric called sharpness is introduced to quantify the teacher-student gap. Experiments are done on CIFAR100 and ImageNet.

This work aims to alleviate the performance degradation problem of the student model during knowledge distillation. The authors argue that the degradation problem may come from the sharpness gap of the model outputs.  They first demonstrate the relation of the sharpness gap (between teacher and student) and the degradation degree, then introduce the realsoftmax function to measure the sharpness of the model output, and finally propose ATKD to adaptively change the temperatures of the teacher and the student for reducing the sharpness gap.  Besides, the authors also analyze previous methods (e.g., Early Stop and TS) from the perspective of the sharpness of models.