Reducing the Teacher-Student Gap via Adaptive Temperatures
Authors: Jia Guo
Venue: ICLR 2022 Submitted (2022)
Keywords: Soft Labels
Knowledge distillation

Abstract: Knowledge distillation aims to obtain a small and effective deep model (student) by learning the output from a larger model (teacher). Previous studies found a severe degradation problem, that student performance would degrade unexpectedly when distilled from oversized teachers. It is well known that larger models tend to have sharper outputs. Based on this observation, we found that the sharpness gap between the teacher and student output may cause this degradation problem. To solve this problem, we first propose a metric to quantify the sharpness of the model output. Based on the second-order Taylor expansion of this metric, we propose Adaptive Temperature Knowledge Distillation (ATKD), which automatically changes the temperature of the teacher and the student, to reduce the sharpness gap. We conducted extensive experiments on CIFAR100 and ImageNet and achieved significant improvements. Specifically, ATKD trained the best ResNet18 model on ImageNet as we knew (73.0% accuracy).

Under review as a conference paper at ICLR 2022
REDUCING THE TEACHER-STUDENT GAP VIA ADAPTIVE TEMPERATURES
Anonymous authors Paper under double-blind review
ABSTRACT
Knowledge distillation aims to obtain small and effective deep model (student) by learning the output from a larger model (teacher). Previous studies found se- vere degradation problem, that student performance would degrade unexpectedly when distilled from oversized teachers. It is well known that larger models tend have sharper outputs (Guo et al. 2017). Based on this observation, we found that the sharpness gap between the teacher and student output may cause this degra- dation problem. To solve this problem, we first propose metric to quantify the sharpness of the model output. Based on the second-order Taylor expansion of this metric, we propose Adaptive Temperature Knowledge Distillation (ATKD), which automatically changes the temperature of the teacher and the student, to reduce the sharpness gap G. gop. We conducted extensive experiments on CIFAR 100 and ImageNet and achieved significant improvements. Specifically, ATKD trained the best ResNet18 model on ImageNet as we knew (73.0% accuracy).
INTRODUCTION
Deep neural networks have achieved remarkable success in most of computer vision tasks (He et al., 2016 [Deng al.) 2009). while the increasing network capacity in the current state-of-the-an models also results in severe computational burdens and high inference time (Tian et al.) [2200] Yim et al.)
2017]. One direction to reduce the model size is knowledge distillation (KD), which trains small efficient models (student) by learning from the output of large models (teacher) (Tang et al. 2020). In 2015, Hinton proposed to soften the model output by raising the temperature of the last softmax layer(Hinton et al. 2015). Since then, this remperature-basee kowledge distillation has drawn the main stream attentions and achieved many successes (Zagoruyko & Komodakis 2017
Kim et al.
2020 Furlanello et al. 2018 Fu et al. 2020).
However, recent researches found that knowledge distillation suffers from a mysterious performance
degradation probleméCho & Hariharan] 2019; Mirzadeh al., (019). Specifically, since the idea of KD is transferring the knowledge of teacher into student, a nature assumption is larger teachers would train better students. But recent researches refuted this assumption and found that the student often performs worse with oversized teachers(Cho & Hariharan 2019)
This paper studies this degradation problem by considering the sharpness (or softness) of the model output. It is well known that networks with more parameters tend to produce sharper output (Guo al. 2017 Lee et al., 2018), which would create the gap of sharpness between teacher and student, Lee. the teacher is usually much sharper than the student. It should be noted that temperature technol- ogy in vanilla knowledge distillation does not reduce this gap. This is because in vanilla knowledge distillation, the teacher and student are softened with the same temperature, which maintains the large gap of sharpness. When the teacher grows larger, this gap will further increases, making more difficult for the student to learn from the teacher.
To solve this problem, let us consider a modified softer teacher model with the same accuracy. In this way, the sharpness gap between the student and the teacher models would be narrowed, making learning more accessible. One way to build this softer teacher is to use higher temperatures for the teacher model, but it would require iebor-nteensve searches the validation data set.
With this insight, we propose to adaptively change the temperatures of the teacher and the student by their sharpness. Our contribution is three-fold:

---

Under review as a conference paper at ICLR 2022
1. We propose metric to quantify the sharpness of models, allowing us to precisely control the sharpness of model output. Formally, we use the realsoftmax function (Nielsen Sun 2016) on the logits vector as the sharpness metric, which approximates smoother version of the max function. If we denote student logits as 2, the teacher logits as v, teacher temperature and student temperature as -" and S respectively (+3 is equal to 7 in the vanilla knowledge distillation), the sharpness metric and the gap of sharpness are defined follows:
Ssharpness = =log) Σ E
G Ga.gop log Σ
E'SIiT log Σ 1/11
2. We propose Adaptive Temperature Knowledge Distillation (ATKD), which automatically changes the temperatures of the teacher and the student, to reduce the sharpness gap G gs. ATKD relieves the burden of searching for the appropriate temperature on the validation set. Specifically, by Taylor second-order expansion, our method can be implemented easily by normalizing the logits with the standard deviation.
3. Our Ssharpmess metric can be used to explain why two existing methods work. Specifically, the Early Stopped method and the Teacher Assistant method both reduce the sharpness gap between the teacher and the student.
We present comprehensive experiments on CIFAR-100 Krizhevsky et al. (2009) and ImageNet datasets (Deng et al.) (009) to evaluate our method. The experiment results show that: 1) The proposed ATKI method tremendously mitigates the performance degradation problem and trains the best ResNet18 8 model on ImageNet as we know (73.01% % accuracy). 2) The proposed ATKI method is easy to optimize, exhibits much lower training/test loss with oversized teachers than KD.
2 METHODOLOGY
2.1 BACKGROUND
Vanillay Knowledge Distillation During training, we minimize the negative log likelihood of the ground truth class to update model parameters. After the model is properly trained, the probability of the ground truth would be close to I, while the rest wrong predictions are near zeros.
In 2015, Hinton noticed that these small wrong probabilities useful to unveil "dark knowledge" (Hinton et al. 2015). Take a picture of "cat" for example, the model are more likely to output higher probability for class "dog" than class "airplane". These wrong probabilities imply the relationship between the two classes and unveil how a model tends to generalize. This observation inspired to use the output of large models soft targets to train efficient small models.
However, modern deep networks tend to produce peaky probabilities (Guo et al. 2017 Lee et al.
2018), that the numbers of those wrong classes (near zero values) would be negligible compared to the ground truth (near one). Thus Hinton proposed to raise the temperature of the last softmax to soften the output probabilities. which can be used as soft targets to train small networks.
We denote logits of the teacher as u, and logits of the student as z, temperature as т, i and j denote the ith and jth value of logits (i.e. the ith and jth category of classes). the loss of knowledge distillation is as follows:
SKD=-> - ΣDTT logp,
(2)
ealT
e/
8
The final loss for the student is then the weighted sum of the cross entropy loss Lat and the knowl- edge distillation loss LKD:
- =CADD + (1- Lcts
(3)
2

---

Under review as a conference paper at ICLR 2022
Table 1: The Performance Degradation Problem.
Teacher
ResNer20
ResNer32
ResNet44
ResNet56
ResNetl 10
Teacher Acc
69.57
70.9
71.9
72.8
73.8
Student Acc
67.4
68.2
68
67.5
67.1
KD loss
1.1
1.7
2.1
2.5
3.3
The popular choice the temperature is in 53,4,5 and the weight À = 0.9 (Hinton et al. 2015
Cho & Hariharan 2019: Tian et al. 2020
Performance Degradation Problem While knowledge distillation achieved success in many fields, mysterious performance degradation problem spotted in 2019. Since the idea of knowledge distillation is transferring teacher knowledge into students, one natural hypothesis is that a larger and more accurate teacher would capture more knowledge and thus train better students. Unfortu- nately, previous studies invalidate this hypothesis by showing that the student performance would degenerate unexpectedly with larger teachers.
Table T shows our experiment result. The student is ResNet14. 4. With larger teachers, the student ac- curacy degrades, and the KD loss increase. Cho Hariharan (2019) hypothesizes that the mismatch of capacity causes this problem. We will discuss this problem in the rest of the section.
2.2 ADAPTIVE TEMPERATURE KNOWLEDGE DISTILLATION
We investigated the degradation problem and proposed to use adaptive temperatures during training to mitigate this problem:
1. We found that the mismatch sharpness between the teacher and the student may cause the degradation problem.
2. We propose metric to quantify the sharpness of model.
3. We propose the adaptive temperature knowledge distillation (ATKD) methods.
The sharpness of neural networks Two main theories explain why knowledge distillation ef- fective. The first theory originated from Hinton's first paper, which argued that teachers are more accurate in capturing category similarities. This information of category similarity helps students to generalize better on unseen data. On the other hand, some contend that the soft output of the teacher prevents the student network from overconfidence (ic. label smoothing technique).
However, when the teacher gets larger, these two theories will contradict each other. This is because that larger models are usually more accurate and peaky at the same time. On the one hand, a more accurate teacher is beneficial the student model. On the other hand, sharper teacher tends to make the student overconfident. Therefore, the larger teacher capacity is a double-edged sword for distillation. This analysis shows that the sharpness of teachers should be investigated carefully, while previous studies have overlooked this critical attribute of teachers.
To this end, propose realsoftmax (Nielsen & Sun 2016) quantify the sharpness of the model output, which is defined as the logarithm of the sum of the exponentials of the logits:
S Ssharpness log Σ
(4)
Such metric has at least two advantages. therefore reasonably measure the sharpness of the model output: 1) This metric smooth approximation the maximum function (2) 2) It is also differentiable.
Meanwhile, we quantify the sharpness gap between the teacher and the student as the difference of the sharpness metric:
Gagap log Σe " log Σ
(5)
3

---

Under review as a conference paper at ICLR 2022
Table 2: The Sahorpness of different models with temperature set one.
Network
ResNet14
ResNer20
ResNer32
ResNet44
ResNet56
ResNet1
Ssharpness
12.20
13.11
13.84
14.45
15.37
16.13
If consider the temperatures, the sharpness gap is:
Gs gap = log Σ cdslT - log Σ eslTT
Table 2 shows sharpness metric value for models of different capacity. It shows that Ssharpness increases with network capacity, and there is sharpness gap between networks of different sizes.
Adaptive Temperature One idea is using higher temperatures for the teacher to reduce the sharp- ness gap between teachers and students. However, searching on the validation data set requires lot of computational costs. Instead, we propose adaptive temperature knowledge distillation, which automatically tunes the temperature according to the S Shhrpnesss
By Taylor second expansion:
Gz.gop = log Σ
enilT - llog Σ esstT
2 log(K + Σ U//T T + 2
29/179)
loog K + Σ + Σ 27/77)
We follow the assumption from Hinton (Hinton et al. 2015), that the logits have been zero-meaned
separately for each training example that Σ; 2j Σj 2j O. We found from experiments that
the sum of logits is indeed very small numbers close zero. The experiment results and further theoretical analysis are provided in the appendix.
Given the above assumption E3 2j = -, 23 O. We can get:
1 [(1/T)} log(K + Σ (2//7)3 Ga.gop = log(K +
(8)
Σ 0タ log(1 + 2r2K = =log(1 + 2
1 Σ 2f))
3
The & E3 in the equation is the variance of logits, which can be represented by the square of the standard deviation stdf:
1 (sld" /T)²) log(1 + (s/dd/// /7)²)
1
Gs log(1 2 (sld"/T)²)
This result shows the relationship between the temperature and sharpness gap pretty straightfor- ward, that the standard deviation controls the sharpness gap between the teacher and the student. Therefore, we set the temperature of the teacher and the student as the standard deviation of the logits vector respectively. These adaptive temperatures would automatically change during training, reducing the sharpness gap between the teacher and the student. is worth noting that there is no need search for the proper temperature the validation like the vanilla KD.
4

---

Under review as a conference paper at ICLR 2022
Formally, if denote standard deviation function as Std, propose Adaptive Temperature Knowledge Distillation as follows:
T. Std(u;), Ti Szd(zz)
CATKD =- -- logp,
(10)
6n1/1?
p.
Σ, Eeessis.PP E, e0s/T
L AATRKD + (1-A)6as
2.3 SHARPNESS GAP ANALYSIS
We investigated the relationship between the sharpness gap and the temperature, and found that the gop of ATKD decrease with 73, while the vanilla KD decrease with f2- Therefore, the temperature of KD and adaptive temperatures of ATKD in the same temperature range, the sharpness gap of ATKD Ge.gap would be smaller than KD considerable margin.
we use to denote K classes, the sharpness gap would decrease with f under vanilla KD:
Gs.gop log Σ e "//T log Σ Es/T
loog(1 0 ((
log(1 + +(())) 1
+ logk logK
(11)
This shows that when the temperature is very high and the linear function (1++a)) can approximate e", Gs Ge. would be reduced. However, with this linear approximation, the KD loss would degenerate into MSE loss, as Hinton points out (Hinton et al. 2015):
0L
ex//T
0//T
(12)
0z, T
Exet/ EjenlT
/+/++1 vi/T+1
(13)
K
K
(2+ v4)
(14)
Kr²
However, high temperatures could be harmful to student, because would encourage the student to leamn more from those negative logits of the teacher (Hinton et al. 2015), and as Hinton pointed out, these negative values of logits could be very noisy.
On the other hand, the sharpness gap of ATKD would decrease faster with f and enable the ATKD to achieve much lower sharpness gap than KD. Because we can use second-order Taylor approximation (1 x }x²), which is better approximation than linear function (1 + x)) in wide range.
G G. = log Σ c3stt" -log Desstt
1
(std / + Oil 0((
llog(1
2
(15)
- log(11 + 2 * (std" 17 + 011 O(( )))
00 0 0({ )
3 EXPERIMENTS
In this section, we will show comprehensive experiment results to validate the effectiveness of ATKD from several perspectives. Specifically, first conducts experiments popular CV datasets
5

---

Under review as a conference paper at ICLR 2022
Table CIFAR-I00 experiments.
Teacher Student
WRN-40-2
WRN-40-2 WRN-40-1
ResNet56 ResNer20
ResNet1 ResNet20
ResNetl ResNet32
ResNet32*4 ResNet8*4
VGG13
WRN-16-2
VGG8
Teacher
75.61
75.61
72.34
74.31
74.31
79.42
74.64
Student
73.26
71.98
69.06
69.06
71.14
72.50
70.36
KD
74.92
73.54
70.66
70.67
73.08
73.33
72.98
FitNet
73.58
72.24
69.21
68.99
71.06
73.50
71.02
AT
74.08
72.77
70.55
70.22
72.31
73.44
71,43
SP
73.83
72.43
70.04
72.69
72.94
72.68
CC
73.56
72.21
69.63
69.48
71.48
72.97
70.71
VID
74.11
73.30
70.38
70.16
72,61
73.09
71.23
RKT
73.35
72.22
69.61
69.25
71.82
71.90
71,48
PKT
74.54
73.45
70.34
70.25
72.61
73.64
72.88
72.50
72.38
69.47
69.53
70.98
73.17
70.94
FT
73.25
71.59
69.84
70.22
72.37
72.86
70.58
FSP
72.91
69.95
70.11
71.89
72.62
70.23
NST
73.68
72.24
69.60
69.53
71.96
73.30
71.53
CRD
75.48
74.14
71.16
71.46
73.48
75.51
73.94
ATKD
75.75
75.06
72.08
72.12
74.09
76.40
74.17
Table 4: ImageNet experiment with Top1 accuracy.
CE
KD
ES
SP
CC
CRD
AT
ATKD)
69.8
69.20
71.40
70.62
69.96
71.38
70.70
72.80
to demonstrate the performance of ATKD. Then we focused on evaluating whether it could alleviate the performance degradation problem.
Dataset 1) CIFAR- 100 (Krizhevsky et al. 2009) is relatively small data set and is widely used for testing various of deep learning methods. CIFAR-100 contains 50,000 images in the training set and 10,000 images in the dev set, divided into 100 fine-grained categories. 2) ImageNer (Deng et al., 2009) is much larger one than CIFAR- 100. ImageNet contains 1.2M images for training and 50K for validation, that distributes in 1000 classes.
CIFAR Experimental settings We run total of 240 epochs for all methods. The learning rate is initialized 0.05, then decay by 0.1 every 30 epochs after 150 epochs. Temperature is for vanilla KD, and the weight of ATKD or KD and cross-entropy is 0.9 and 0.1 for all the settings.
ResNet18 student for all of methods. Training settings like learning rate or training epochs are the same with Heo et al. (2019) for ImageNet. The teacher network is well-trained previously and fixed during training.
ImageNet Experimental settings Here
3.1 CIFAR- 100 AND IMAGENET
Baselines. We selected many SOTA KD methods evaluate the performances of ATKD. Knowl- edge defined from intermediate layers: FitNet (Romero et al., (015), AT (Zagoruyko & Komodakis
20171. SP (Tung Mori 2019). PKT (Passalis & Tefas 2018), FT (im et al. 2020). FSP (Ymm et al. 2017). 1) Knowledge defined via mutual information: CC (Peng et al. 2019), VID (Ahn et al.] 2019). CRD (Tian et al. 2020). 2) Structured Knowledge: RKD (Park et al. 2019)). 3) Knowledge
from logits: KD (Hinton al. 2015), NST (Huang & Wang 2017). ES (Cho & Hariharan 2019),
TA
(Mizzadeh et al. 2019)
Results in CIFAR-100. Table 3 shows that ATKD always has outstanding improvement com- pared with all other methods. In some situations (e.g. those where teacher/student is WRN-40- 2/WRN-40-1 ResNetl 10/ResNet32), the performances of ATKD are very close to those of teacher.
Results in ImageNet. All experiments used ResNet34 the teacher and ResNet18 the student. Table (the Top1 accuracy) shows that ATKD exceeds all of the previous SOTA by a large mar- gin. Before ATKD, the improvement of this task is limited. Fig. [ shows the training process of

---

Under review as a conference paper at ICLR 2022
Table 5: Performance Degradation Problem on CIFAR-100. Student ResNet14. ATKD archives lower training loss and higher accuracy. The G x.gap between the distilled student and teacher is also reduced significantly. Temperature is set in vanilla KD.
ResNet20
ResNet32
ResNet44
ResNet56
ResNet110
Training loss
Vanilla KD ATKD
1.7
2.1
2.5
3.3
0.9
1.2
1.3
1.4
1.6
Test
Vanilla KD
67.4
68.2
68
67.5
67.1
ATKD
68.2
68.7
68.9
68.8
69.2
Gg gap
Vanilla KD ATKD
0.09
0.16
0.21
0.30
0.38
0.05
0.11
0.14
0.20
0.25
Table 6: Performance Degradation Problem ImageNet.
Teacher
Method
Accuracy
Teacher
Method
Accuracy
ResNet34
KD
69.43
ResNet101
ES
70.98
ATKD
72.80
ATKD
72.85
ResNet50
KD
69.05
ResNet152
TA
70.65
TA
70.59
ES 70.95
ES
70.74
ATKD
73.01
ATKD
72.70
vanilla KD and ATKD. It worth noting that ATKD provides comparable performance KD's final performance after first 30th epoch training.
3.2 PERFORMANCE DEGRADATION EXPERIMENTS
CFIFAR-I00 On CIFAR-IO0 task, We trained the ResNet14 with multiple teachers on the CIFAR- 100 dataset, and the result is shown in Table 5 Experimental details follows the above settings. Under ATKD, student performance continues to increase as the teacher gets bigger. The training loss of ATKD is also much lower than the vanilla KD. In addition, the G gap (computed with temperatures) metric shows that ATKD significantly reduces the sharpness gap.
In the analysis of section 2.33 concluded that larger temperatures would have smaller sharpness gap. Therefore the comparison between KD and ATKD need to take temperature values into account. The temperature of KD is while most of the mean adaptive temperatures used by ATKD between (3, 4). We provide more details in the appendix.
ImageNet Tsole the degradation problem in ImageNet with ResNet18 the student, and denote that this specific experiment was not conducted in the cited paper. We compared ATKD with two previous methods that aim alleviate the degradation problem, Early Stop (Cho Hariharan, 2019) (ES) and Teacher Assistant (Mirzadch et al. 2019) (TA). Both of these two methods explicitly regularizing the teacher capacity: 1) TA proposed to distill the large teacher an intermediate teacher and then distill to the student, so that each knowledge distillation step has better match between student and teacher capacity. 2) ES methods use the early stopped teacher, the teacher capacity would be regularized by fewer training steps.
We can see that ATKD exceeds Early Stop and TA methods with a large margin in all teacher settings. For example, when distilled by ResNet50 and ResNet152, the performance exceeds other methods by 2%. We obtained 73.01 % accuracy, which the best ResNet18 results as we know.
An interesting finding of the early stopped teacher method is that these regularized teachers have smaller Ssharpnese: which can reduce the sharpness gap between the teacher and the student. The results is in Table ? Similarly, the Teacher Assistant method also reduces the sharpness gap by using medium medium-sized teachers. This finding indicates that S Ssharpness metric could provide a universal framework mitigate the degradation problem.

---

Under review as a conference paper at ICLR 2022
Table The Sehorpness of Early Stopped models, temperature set to 1.
Network
ResNet14
ResNet20
ResNet32
ResNet44
ResNet56
ResNet1
S Sharppesss (240 epochs)
12.20
13.11
13.84
14.45
15.37
16.13
S Saharpness (60 epochs)
13.03
13.68
14.29
14.77
15.69
Truining Loss
Test
ACC
5.8
2.9
2.4
5.8
1.9
4.4
Aw
mm
1.4
3.2
-TTK KD
0.9
20
32
44
56
110
depth
depth
41
*KD&ATKD
1. CIFAR-100 Training Loss
2. ImageNet Test Accuracy
Figure 1: 1. The training/test loss on CIFAR-100 100. With the teacher size grows, the loss ATKD increases slower than KD, which shows that ATKD alleviates the Degradation Performance Problem.
2. The training process KD and ATKD on ImageNet. ATKD achieves comparable accuracy with KD at the 30th epoch.
4 RELATED WORK
Knowledge Distillation Buciluă al. (2006) first proposed compressing a trained cumbersome model into smaller model by matching the logits between them. Then Hinton et al. (2015) ad- vanced this idea and formed more widely used framework known as knowledge distillation (KD). Knowledge distillation tries to minimize the KL divergence between the soft output probabilities generated by the logits through softmax. Xu et al. (2020) proposed to normalize the feature, the penultimate layer of the network, to perform distillation. This method is similar to methods except that we perform on the logits layer. Knowledge distillation is also a kind of soft label training method. Similar to label smoothing, previous studies have found that knowledge distillation helps to regularize the training of network.
Performance Degradation Problem: Although distillation has shown great potential in many tasks, mysterious problem is found that larger teachers often harm the distillation performance, despite its more powerful ability (Cho & Hariharan, 2019 Mirzadeh al. 2019). This problem is particularly severe on ImageNet, resulting in poor performance for KD. It was widely accepted that the mismatch of capacity caused this problem. Previous research proposed regularize the teacher capacity to alleviate this problem heuristically. Cho & Hariharan (2019) proposed to early stop the training of the teacher. Moreover, MMizadeh et al. (2019) proposed to use medium-size teacher assistant (TA) perform sort of sequence distillation. This TA first learns from the teacher, then the student can learn from this TA. However, the accuracy of the early stopped teacher or TA also harmed, which is lower than the original teacher.
CONCLUSION
The vanilla knowledge distillation overlooks the sharpness gap between the teacher and the student, which may cause the performance degradation problem. this paper, we propose to use metric
to measure the sharpness of neural networks, allowing to control the sharpness of models by using adaptive temperatures. Recently some papers have focused on combining label smoothing and knowledge distillation to train neural networks (Shen et al. 2021 Müller al. [201)) An appealing point of view is that ATKD is inherently consistent with these methods. ATKD smooths teachers by adapting temperatures, while these methods let students learn from teachers trained by label smoothing. The deeper relationship between knowledge distillation and label smoothing remains to be explored.
8

---

Under review as a conference paper at ICLR 2022
REFERENCES
Sungsoo Ahn, Shell Xu Hu, Andreas C. Damianou, Neil D. Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9155-9163, 2019.
Cristian Buciluă. Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. Proceed- ings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '06, pp. 535-541, New York, NY, USA, 2006.
Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4793-4801, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. 2009 IEEE Conference Computer Vision and Pattern Recognition, pp. 248-255, 2009.
Jie Fu, Xue Geng, Zhijian Duan, Bohan Zhuang, Xingdi Yuan, Adam Trischler, Jie Lin, Chris Pal, and Hao Dong. Role-wise data augmentation for knowledge distillation. arXiv preprint 20X0: 2020.
Tommaso Furlanello. Zachary C. Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks, 2018
lan Goodfellow, Yaroslav Bulatov, Julian Ibarz, Sacha Arnoud, and Vinay Shet. Multi-digit number recognition from street view imagery using deep convolutional neural networks. arXiv preprint arXiv:1312.6082, 2013.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp.
2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770- 778, 2016.
Byeongho Heo, Jecsoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A comprehensive overhaul of feature distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1921-1930, 2019.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. ArXiv, abs/ abs/1503.00253, 2015.
Zehao Huang and Naiyan Wang. Like what you like: Knowledge distill via neuron selectivity transfer. ArXiv, abs/1 707.01219, 2017.
Jangho Kim, ScongUk Park, and Nojun Kwak. Paraphrasing complex network: Network compres- sion via factor transfer, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence--allibrated classifiers for detecting out-of-distribution samples, 2018.
Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadch. Improved knowledge distillation via teacher assistant, 2019.
Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When does label smoothing help? arXiv
preprint arXiv: /906. wwt.: 02629, 2019.

---

Under review as a conference paper at ICLR 2022
Frank Nielsen and Ke Sun. Guaranteed bounds the kullback-leible divergence of univariate mixtures using piecewise log-sum-expp inequalities. arXiv preprint arXiv:1606.05880, 2016.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 3962-3971, 2019.
Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In ECCV, 2018.
Baoyun Peng. Xiao Jin, Jiaheng Liu, Shunfeng Zhou, Yichao Wu, Yu Liu, Dong sheng Li, and Zhaoning Zhang. Correlation congruence for knowledge distillation. 2019 IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), pp.
2019.
Gabriel Pereyra, George Tucker, Jan Chorowski, Fukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.06545 2017.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. CoRR, abs/ 1412.6500, 2015.
Zhiqiang Shen, Zechun Liu, Dejia Xu, Zitian Chen, Kwang- Ting Cheng, and Marios Savvides. Is label smoothing truly incompatible with knowledge distillation: An empirical study, arXiv preprint arXiv:21044 00676, 2021.
Jiaxi Tang, Rakesh Shivanna, Zhe Zhao, Dong Lin, Anima Singh, Ed H. Chi, and Sagar Jain. Un- derstanding and improving knowledge distillation, 2020.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. ArXiv, abs/1910.10699 2020.
Frederick Tung and Greg Mori. Similarity-preservinn knowledge distillation. 2019.
2019 IEEE/CVF
International Conference on Computer Vision (ICCV), pp.
Kunran Xu, Lai Rui, Yishi Li, and Lin Gu. Feature normalized knowledge distillation for image clas- sification. In Computer Fision-ECCY 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings, Part XXV 16, pp. 664-680. Springer, 2020.
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7130-7138, 2017.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer. ArXiv, abs/1612.03928, 2017.
A APPENDIX
A.1 EXPERIMENTS OF DIFFERENT FIXED TEMPERATURES
We conducted experiments of different fixed temperatures for vanilla KD (Table8) which shows no observable performance improvement. We distilled ResNet14 by ResNet56 on CIFAR-100 dataset, and the rest details followed the setting in section 3.
A.2 EXPERIMENTS VARIES WITH WIDTII
We add degradation problem experiments where teacher models vary with width here (Table We use Wide ResNet to train on the CIFAR-100 dataset, the rest experiment details follow the setting in section
10

---

Under review as a conference paper at ICLR 2022
Table 8: Experiments of different fixed temperatures
7
4
4
4
ATKD
7
3.5
3.7
4
4.3
4.5
ATKD
Accuracy
68.27
68.1 11
68.06
68.22
68.30
69.0
S. gap
0.23
0.25
0.30
0.35
0.37
0.19
7
3.5
3.7
4
4.3
4.5
5
7
4
4
4
4
4
4
Accuracy
67.79
68.01
68.06
68.10
10
68.07
68.01
Sp.gop
0.40
0.33
0.30
0.27
0.24
0.21
Table 9: Experiments varies with width
WRN-16-2
WRN-16-3
WRN-16-4
WRN-16-5
WRN-16-6
Test acc
Vanilla KD ATKD
68.22
67.88
68.27
67.80
67.2
69.39
69.33
69.39
69.40
69.21
Gs gap
Vanilla KD
0.37
0.49
0.59
0.71
0.83
ATKD
0.10
0.22
0.34
0.41
0.55
A.3 ENTROPY SCORE OF MODELS
Entropy was used as confidence score at Pereyra et al. (2017), that is related to the sharpness of model output. We measured the entropy values of different models (Table I0). Experimental details follow the setting in section 3.
Table 10: Logits Entropy
R
ResNer20
ResNet32
ResNet44
ResNet56
ResNet1 10
0.97
0.74
0.45
0.35
0.22
0.09
A.4 SHARPNESS GAP DURING TRAINING
Figure shows the sharpness gap changes during the training. ResNet20 and WRN-16-1 used as students, and ResNet56 and WRN-16-3 are used as teachers respectively.
A.5 EXPERIMENT ON SVHN
We add another dataset SVHN here (Goodiellow et al. 2013). Results show in table IT SVHN is similar MNIST (e.g., the images are of small cropped digits), but comes from real house numbers in Google Street View images. We use 4 as the student. The rest experiment settings use the setting of CIFAR-100 in section Further exploration on parameter tuning might get better results.
A.6 THE SUM LOGITS
In the paper of Hinton et al. (2055), they presume that the E, 2j equals to zero during training without further analysis (In page 3, Eq. 3). Here, we analyze this assumption theoretically and experimentally.
It is well known that the gradient to logits zj is:
0L
Pj 9i
(16)
0zj
11

---

Under review as a conference paper at ICLR 2022
1) ResNet ACC
2) ResNet Ge.gap
3) WRN ACC
4) WRN Gegapp
Figure 2: The ResNet20 accuracy plot during training The ResNet20 sharpness gap plot during
training. 3) The WRN-16-1 accuracy plot during training 4) The WRN-16-1 sharpness gap plot during training
Table 11:
Experiments on SVHN
ResNet20 96.40
ResNet32 96.68
ResNet44 96.73
ResNet56
Teacher acc
96.89
Test acc
Vanilla KD ATKD
96.57
96.54
96.61
96.59
96.70
96.83
96.73
96.77
Gagap
Vanilla KD
0.05
0.05
0.07
0.07
ATKD
0.03
0.04
0.04
0.04
Where Py is the jth class probability of the student and the qj is the probability of the teacher. We can get the gradient to Σ 2j by adding these gradients:
OL
Σ
= Σ (p, 3
-1
(17)
3 Ozg
Therefore, the gradient to the sum of logits is zero.
Besides, know that the logits is the product of the weight W and the feature vector f in the penultimate layer:
2 = = Σ W
(18)
Where W; is the ith column vector W, considering that the initial weights of W are sampled from the Pero-meaned normal distribution in the popular initialization settings (He et al. 2015), assume that the initialized Σ, Wij (the sum of column vector) equals to zero. Therefore, the _ zj should be zero all the time given the above assumptions.
We conducted experiments to verify this assumption. We train these models the CIFAR-100 dataset. Table [2] shows that all these models logits are close to zero.
Table 12: The value of Σ 2j
ResNet20
ResNet32
ResNet44
ResNet56
ResNet110
Logits Sum
-5e-5
-4.7e-5
-5.7e-5
-7.9e-4
-6.1e-5
WRN-16-1
WRN-16-2
WRN-16-3
WRN-16-4
VGG13
Logits Sum
-.88e-5
-6.le-6
-5.5e-5
-1.2e-5
-3.ee-5
Table 13: Mean Value of Adaptive Temperatures
ResNet14
ResNer20
ResNer32
ResNet44
ResNet56
ResNetl
3.34
3.43
3.49
3.58
3.71
3.87
12

---

Under review as a conference paper at ICLR 2022
A.7 THE ADAPTIVE TEMPERATURES
We provide the mean value of the temperatures in table 13
13


Reviews:
[{"review_id": "k81OtzSeda1", "reviewer": "Reviewer_Yenx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The proposed method can effectively reduce the sharpness gap between the teacher model and the student model and achieve better performance when a large teacher model is used. \n- The work reveals a new perspective to investigate knowledge distillation. \n\nWeakness:\n- In table 5, please clarify that what temperature is used during training and computing the sharpness gap in the table. It's not fair if the sharpness is compared with different temperatures.\n- Please explain in more detail that why the sum of student logits is a constant during training even if the input and parameters are zero-mean initialized.\n- The logits 'std' defined on page 4 should be variance. And, why the mean of the output is assumed to be 0?\n\n", "summary_of_the_paper": "The paper proposes to mitigate the performance degradation by controlling the sharpness gap between a large teacher and a student model. The sharpness is defined as the real softmax function (the logarithm of the sum of the exponentials of the logits). During the training, the temperature is set according to the sharpness of the logits. For the teacher model which has a sharp output, the temperature is larger so that the output becomes more smooth. \n", "summary_of_the_review": "In summary, the paper proposes an interesting perspective to investigate the performance degradation in knowledge distillation using a large teacher model. However, some detail and assumption are not well-presented in the paper. \n\n-- post-rebuttal\n\nThe response addressed some of my concerns. I will keep my score. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WexBsnSlySA", "reviewer": "Reviewer_6R6z", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\nThe authors:\n\n(1) Provide a new perspective to explain the degradation problem of KD.\n\n(2) Introduce the realsoftmax function to reflect the sharpness of models, and propose ATKD to adaptively narrow the sharpness gap for the steady knowledge distillation.\n\nWeaknesses\n\nThe authors should provide more analysis:\n\n(1) What is the best sharpness gap between the teacher model and the student model? A theoretical analysis?\n\n(2) Can you manually adjust the temperature of the teacher (fix the temperature of the student) and report the performance changes of the student? It would be better to contrast the sharpness gap of the best student with that of ATKD.\n\n(3) Whether the best sharpness gap varies on different datasets or tasks?\n", "summary_of_the_paper": "This work aims to alleviate the performance degradation problem of the student model during knowledge distillation. The authors argue that the degradation problem may come from the sharpness gap of the model outputs.\n\nThey first demonstrate the relation of the sharpness gap (between teacher and student) and the degradation degree, then introduce the realsoftmax function to measure the sharpness of the model output, and finally propose ATKD to adaptively change the temperatures of the teacher and the student for reducing the sharpness gap.\n\nBesides, the authors also analyze previous methods (e.g., Early Stop and TS) from the perspective of the sharpness of models.\n", "summary_of_the_review": "The idea is novelty and interesting. The authors should provide more analysis to demonstrate the effectiveness of this method comprehensively. \n\nI would recommend this paper for ICLR2022.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SR51kPR15uP", "reviewer": "Reviewer_iBXp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The sharpness gap analysis and observations are interesting. It shows a strong correlation between the sharpness gap and KD results, providing new insight into the KD strategy. The proposed ATKD method is simple and has a reasonable theoretical justification. Its result achieves SOTA performance, especially the large gain in the ImageNet experiment (Tables 4 and 6); such a performance gain is non-trivial and considered significant.\n\nThis work raised several interesting questions. Some of them probably should be answered in the paper to conclude the observations about sharpness better:\n1. Would other sharpness functions/scores lead to the same conclusion? For example, would the entropy of output probability a reasonable sharpness score, and how does it compare to the realsoftmax function?\n2. Any guess about why a student with vanilla KD loss can not learn to match the sharpness of the teacher? The limited capacity of the student may not be a candidate cause since the student model can multiply a large number to the logits to make the sharpness score large, thus reducing the sharpness gap. \n3. Section 2.3 suggests that the classical KD loss converges slower than the proposed ATKD loss. This indicates that the classical KD converges slower but does not have a worse error bound. However, in practice, would training longer with the classical KD loss converge to a similar sharpness gap and classification performance obtained by ATKD? \n4. Would the proposed sharpness score and observation generalize to other cases? The observations in the paper are mainly based on varying the depth of the model. Does the same trend hold when changing the width of the teacher model?\n5. One additional observation that could be made is to have a line plot that has training epochs in the x-axis and has the sharpness gap and the student classification accuracy in the y-axis. \n", "summary_of_the_paper": "This work explores how the sharpness gap between a teacher model and a student model affects knowledge distillation performance. The paper argues that a large sharpness gap harms KD and proposes an adaptive temperature strategy to speed up the decrease of the sharpness gap in training. The experiments demonstrate a high correlation between the sharpness gap and student performance. It also shows that the proposed adaptive temperature strategy can achieve SOTA results on the image classification tasks.", "summary_of_the_review": "The work provides an interesting insight into KD by investigating the sharpness gap between the teacher and student model. The ATKD method built based on this insight is simple but can significantly improve the KD results. However, this work still leaves several important questions around the justification of sharpness discussed above. The mixed pros and cons lead to my current rating, subject to be updated.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4HSl9sbdl0r", "reviewer": "Reviewer_iJMG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strengths\n\nThis paper attempted to improve the vanilla knowledge distillation method's performance from the fixed temperature side. Intuitively, this solution is reasonable and practical.\n\nweaknesses\n\n1. This paper is poorly written. Equations are confusing without the description of each component. For example, i and j in Eq(2). L_{KD} loss is also wrongly described. Also, the tables and captions are very unclear. For example, table1, which metrics are used? Top1 or Top5. In Table 6, what does '-' represent? What is TS? Figure 1, what is SKD?\n\n2. Table 4 is supposed to be resnet18 as student and resnet34 as a teacher but ATKD with 73.01 is from teacher resent50. Check table 6. \n", "summary_of_the_paper": "This paper is to improve the vanilla knowledge distillation method. Based on the observation in Cho&Hariharan that student degrades by oversized teachers, this paper proposed an adaptive temperature solution. A new metric called sharpness is introduced to quantify the teacher-student gap. Experiments are done on CIFAR100 and ImageNet.", "summary_of_the_review": "The intuition of this paper is clear but the method proposed is not well presented, making understanding this work is hard. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors study the degradation problem observed in KD for large teacher networks and propose to address it by quantifying and adapting to a *sharpness gap* between the student and the teacher. The reviewers generally appreciated the proposed approach in handling larger teachers and found it effective within the scope of the numerical results provided in the paper. That said, the reviewers raised several critical issues concerning the writing and the presentation of several crucial parts of the paper, in particular those related to the sharpness measure and the proposed training method ATKD. Thus, given this, and the exchanges between the reviewers and the authors, in its present form, the paper cannot be recommended for acceptance. The authors are encouraged to incorporate the valuable feedback provided by the knowledgeable reviewers.
Decision: Reject