Learning Student-Friendly Teacher Networks for Knowledge Distillation
Authors: Dae Young Park
Moon-Hyun Cha
Changwook Jeong
Daesin Kim
Bohyung Han
Venue: NeurIPS 2021 Poster (2021)
Keywords: deep learning
transfer learning
knowledge distillation

Abstract: We propose a novel knowledge distillation approach to facilitate the transfer of dark knowledge from a teacher to a student. Contrary to most of the existing methods that rely on effective training of student models given pretrained teachers, we aim to learn the teacher models that are friendly to students and, consequently, more appropriate for knowledge transfer. In other words, at the time of optimizing a teacher model, the proposed algorithm learns the student branches jointly to obtain student-friendly representations. Since the main goal of our approach lies in training teacher models and the subsequent knowledge distillation procedure is straightforward, most of the existing knowledge distillation methods can adopt this technique to improve the performance of diverse student models in terms of accuracy and convergence speed. The proposed algorithm demonstrates outstanding accuracy in several well-known knowledge distillation techniques with various combinations of teacher and student models even in the case that their architectures are heterogeneous and there is no prior knowledge about student models at the time of training teacher networks

Learning Student-Friendly Teacher Networks for Knowledge Distillation
Dae Young Park*.l, Moon-Hyun Cha', Changwook Jeong', Dae Sin Kim!, and Bohyung Han*
DITT Center, Samsung Electronics, Korea
ECE & ASRI, Seoul National University, Korea
(p30. daeyoung, moonhyun. cha, thris.jeone daesin. kim Asamsung. com bhhan@snu.c ac.kr
Abstract
We propose novel knowledge distillation approach facilitate the transfer of dark knowledge from a teacher to student. Contrary to most of the existing methods that rely effective training of student models given pretrained teachers, we aim to learn the teacher models that are friendly to students and, consequently, more appropriate for knowledge transfer. In other words, the time of optimizing a teacher model, the proposed algorithm learns the student branches jointly to obtain student-friende representations. Since the main goal of approach lies in training teacher models and the subsequent knowledge distillation procedure is straightforward, most of the existing knowledge distillation methods can adopt this technique to improve the performance of diverse student models in terms of accuracy and convergence speed. The proposed algorithm demonstrates outstanding accuracy in several well-known knowledge distillation techniques with various combinations of teacher and student models even in the case that their architectures are heterogeneous and there is no prior knowledge about student models at the time of training teacher networks.
1 Introduction
Knowledge distillation [ is well-known technique to learn compact deep neural network models with competitive accuracy, where smaller network (student) is trained simulate the representations of larger one (teacher). The popularity of knowledge distillation is mainly due to its simplicity and generality; straightforward to learn a student model based on teacher and there no restriction about the network architectures of both models. The main goal of most approaches is how to transfer dark knowledge to student models effectively, given predefined and pretrained teacher networks.
Although knowledge distillation is a promising and convenient method, it sometimes fails to achieve satisfactory performance in terms of accuracy. This partly because the model capacity of student is too limited compared to that of teacher and knowledge distillation algorithms are suboptimal 2 3 In addition to this reason, we claim that the consistency of teacher and student features is critical to knowledge transfer and the inappropriate representation learning of a teacher often leads to the suboptimality of knowledge distillation.
We are interested in making a teacher network hold better transferable knowledge by providing the teacher with snapshot of the student model the time of its training. We take advantage of the typical structures of convolutional neural networks with multiple blocks and make the representations of each block in teachers easy be transferred to students. The proposed approach aims to train
"Equal contribution
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Teacher Softmax
Student Dranch
Teacher
Softmax
Softmax
Teacher Network
Studeur Branch
Loss
Teacher
Tencher Softnux
Netwerk
Hard
Student
Easy
Knowledge
Knowledge
Teacher
Nesork
Studens Nerwork
Loos
Teacher
Loss
Studem: Netw
Tosss
Network
Label
Sandent
Teacher
Smidett
Sottmax
Label
Softmax
Label
Sothmax
Libel
Training Teacher
Distillation
Training Student-Aware Teacher
Distillation
(a) Standard knowledge distillation
(b) Student-friendly teacher network
Figure 1: Comparison between the standard knowledge distillation and our approach. (a) The standard knowledge distillation trains teachers alone and then distill knowledge students. (b) The proposed student-frient friendly teacher network trains teachers along with student branches, and then distill more casy-to-transfer knowledge students.
teacher models friendly to students for facilitating knowledge distillation; we call the teacher model trained by this strategy student-friendly teacher network (SFTN). SFTN deployed in arbitrary distillation algorithms easily due to its generality for training models and transferring knowledge.
SFTN is partly related collaborative learning methods 4 5 6) which may suffer from the
correlation between the models trained jointly and fail to fully exploit knowledge in teacher models. On the other hand, SFTN is free from the limitation since it performs knowledge transfer from teacher to a student in one direction via a two-stage learning procedure-ssuueet---n teacher network followed by knowledge distillation from a teacher a student. Although the structure of a teacher network depends on target student models, it is sufficiently generic to be adopted by [ demonstrates the main difference between the proposed
training of
students with various architectures. Figure algorithm and the standard knowledge distillation methods.
The following the list of main contributions:
We adopt student-and aware teacher learning procedure before knowledge distillation, which enables teacher models to transfer their representations to students more effectively.
The proposed approach applicable diverse architectures teacher and students while can be incorporated into various knowledge distillation algorithms.
We demonstrate that the integration of SFTN into various baseline algorithms and models improve accuracy consistently with substantial margins.
The rest of the paper is organized as follows. We first discuss the existing knowledge distillation techniques in Section Section Seetionndescribee the details the proposed SFTN including the knowledge distillation algorithm. The experimental results with in-depth analyses are presented Section and we make the conclusion in Section
2 Related Work
Although deep learning has shown successful outcomes in various fields, it is still difficult to apply deep neural networks to real-world tasks due their excessive requirement for computation and memory. There have been many attempts to reduce the computational cost of deep learning models, and knowledge distillation one of the examples. Various computer vision 7 g 10] and natural language processing [II
B3 [4] tasks often employ knowledge distillation to obtain efficient models. Recently, some cross mooss-moaal tasks 15 16 transfer knowledge across domains. This section summarizes the research efforts to improve performance of models via knowledge distillation.
12
2.1
What to distill
Since Hinton al. [ introduce the basic concept knowledge distillation, where the dark knowledge in teacher models is given by the cemperature-scaled representations of the softmax function, various

---

kinds of information have been employed as the sources of knowledge for distillation from teachers to students. FitNets [88 distills intermediate features of teacher network, where the student network transforms the intermediate features using guided layers and then calculates the difference between the guided layers and the intermediate features of teacher network. The position of distillation is shifted to the layers before the ReLU operations in [19], which also proposes the novel activation function and the partial Le loss function for effective knowledge transfer. Zagoruyko and Komodakis (20) argue importance of attention and propose an attention transfer (AT) method from teachers to students while Kim et al. [2] compute the factor information of the teacher representations using an autoencoder, which is decoded by students for knowledge transfer. Relational knowledge distillation (RKD) [2] introduces a technique to transfer relational information such as distances and angles of features.
maximizes mutual information between teacher and a student via contrastive learning. There exist a couple of methods to perform knowledge distillation without teacher models. For example, ONE [4] distills knowledge from ensemble of multiple students while BYOT [25] transfers knowledge from deeper layers to shallower ones. Besides, SSKD [26] distills self-supervised features of teachers students for transferring richer knowledge.
CRD 23
2.2 How to distill
Several recent knowledge distillation methods focus on the strategy of knowledge distillation. Born again network (BAN) [27] presents the effectiveness of sequential knowledge distillation via the networks with an identical architecture. A curriculum learning method (28) employs the optimization
trajectory of teacher model to train students. Collaborative learning approaches 4 5 6 attempt to
learn multiple models with distillation jointly, but their concept is not well-suited for asymmetric teacher-student relationship, which may lead to suboptimal convergence of student models.
The model capacity gap between a teacher and a student addressed in employs extra network to reduce model capacity gap between teacher and student models, where teacher transfers knowledge to a student via a teaching assistant network with an intermediate size. An carly stopping technique for training teacher networks is proposed to obtain better transferable representations and a neural architecture search is employed to identify a student model with the optimal size [2] Our work proposes a novel student-friendle learning technique of teacher network to facilitate knowledge distillation.
3 TAKD K
3 Student-Friendly Knowledge Distillation
This section describes the details of the student- frienndy teacher network (SFTN), which transfers the features of teacher models student networks more effectively than the standard distillation. Figure Suuure@illuee Dillustrates the main idea of our method.
3.1 Overview
The conventional knowledge distillation approaches attempt to find the way of teaching student networks given the architecture of teacher networks. The teacher network is trained with the loss with respect the ground-truth, but the objective is not necessarily beneficial for knowledge distillation to students. To the contrary, SFTN framework aims to improve the effectiveness of knowledge distillation from the teacher to the student models.
Modularizing teacher and student networks We modularize teacher and student networks into multiple blocks based on the depth of layers and the feature map sizes. This is because knowledge distillation is often performed at every 3 or 4 blocks for accurate extraction and transfer of knowledge in teacher models. Figure 2 presents the case that both networks are modularized into blocks, denoted by (B), BR, B and (B), B3, B3) for teacher and a student, respectively.
Adding student branches SFTN augments student branches to teacher model for the joint training of both parts. Each student branch is composed of teacher network feature transform layer T and student network blocks. Note that T is similar to a guided layer in FitNets [18] and transforms the dimensionality of the channel in Fi into that of B3+11 Depending on the configuration of teacher and student networks, the transformation need to increase or decrease the size of the feature maps. We employ 3 3x3 convolutions to reduce the size of FG while 4x4 transposed convolutions
3

---

Student Brunch
Teacher Network
Bi
Student Branch
7
Image
F
8
Inage
F7
D1
B3
Label
Student Network
Label
Tencher Network
(a) Student-aware training a teacher network
(b) Knowledge distillation
Figure 2: Overview of the student-friendly teacher network (SFTN), In this figure, F, B, T, and denote feature map, a network block, teacher network feature transform layer, and softmax output, respectively, where the superscript means the network block index and the subscript S, T, and R respectively indicate the student network, the teacher network, and the student branch in the teacher model. The loss for teacher network Cr is given by 4 while Kullback-Leible loss C and cross entropy loss f are defined (5) and (6), respectively. (a) When training a teacher, SFTN optimizes Ff and qr for better knowledge transfer student networks. (b) In the distillation stage, the features the teacher network, F1 and qr. are distilled to student networks with existing knowledge distillation algorithms straightforwardly.
used to increase its size. Also, 1x convolutions is used when we do not need to change the size of Ff. The features transformed to a student branch is forwarded separately to compute the logit of the branch. For example, shown in Figure 2(a). F1 in the teacher stream is transformed to fit B3. which initiates a student branch to derive q while another student branch starts from the transformed features of F.. Note that F has no trailing teacher network block in the figure and has no associated student branch because it is directly utilized to compute the logit of the main teacher network.
Training SFTN The teacher network is trained along with multiple student branches corresponding to individual blocks the teacher, where we minimize the differences in the representations between the teacher and the student branches. Our loss function is composed of three terms: loss in the teacher network LT. Kullback-Leibler loss LKL in the student branch, and cross-entropy loss L in the student branch. The main loss term, Lr. minimizes the error between qr and the ground-trutt while CKI. enforces q and qr to be similar to each other and D makes qi fit the ground-tre
Distillation using SFTN As shown in Figure b), the conventional knowledge distillation tech- nique is employed to simulate Fj and qr by F'S and qs respectively. The actual knowledge distillation step straightforward because the representations of Fý and qr have already been learned properly at the time of training SFTN. We expect the performance of the student network distilled from the SFTN be better than the one obtained from the conventional teacher network.
3.2 Network Architecture
SFTN consists of a teacher network and multiple student branches. The teacher and student networks divided into N blocks, where set of blocks in the teacher is given by Br (Bi while the blocks in the student is denoted by By {B² N Note that the last block in the teacher network does not have the associated student branch.
Given an input of the network, x, the output of the softmax function for the main teacher network, qr, is given by
Fr(x)
qx(x; = softmax
(I)

---

where Fr denotes the logit of the teacher network and 7 is the temperature of the softmax function. On the other hand, the output of the softmax function in the ith student branch, q given by
OR (Fi;7) softmax F3(T((F)))
(2)
T
where F3 denotes the logit of the 8t student branch.
3.3 Loss Functions
The teacher network in the conventional knowledge distillation framework is traned only with Cr- However, SFTN has additional loss terms such as LR and LCF as described in Section 3.1 The total loss function of SFTN, denoted by LSFTN, is given by
CSFIN = XrTr + CR + CE LR
(3)
where År, L and C are the weights of individual loss terms.
Each loss term is defined as follows. First, LT given by the cross-entropy between the teacher's prediction qr and the ground-truth label y as
LT CrossEntropy((nn (q
The knowledge distillation loss, denoted by Lk employs the KL divergence between q and qr- where N 1 student branches except for the last block in the teacher network are considered together as
N-1
CKL
Kkk (qr).
(5)
N
where q and qr denote smoother softmax function outputs with larger temperature, F. The cross-entropy loss the student network, C is obtained by the average cross-entropy loss from all the student branches, which is given by
N-1 Σ
CR =
CrossEntropy(ake y).
(6)
N
Note that we set r to for both the cross-entropy loss, Lr and CCB.
4 Experiments
We evaluate the performance of SFTN in comparison to existing methods and analyze the charac- teristics of SFTN in various aspects. We first describe our experiment setting Section 4.T Then, compare results between SFTNs and the standard teacher networks with respect classification accuracy in various knowledge distillation algorithms Section 4.2 The results from ablative experiments for SFTN and transfer learning discussed in the rest of this section.
4.1 Experiment Setting
We perform evaluation on multiple well-known datasets including ImageNet 30 and CIFAR-100
using several different backbone networks such as ResNet [32], WideResNet [33], VGG (34), Shuf- fleNetV1
335 and ShuffleNetV2 [36]. For comprehensive evaluation, adopt various knowledge distillation techniques, which include KD [I] FitNets [8]] AT 20). SP (37]. VID 38 RKD [22].
PKT 39
AB 40) FT 21.. CRD 23, SSKD 26 and OH 19 Among these methods, the feature
distillation methods [8 20 37
38,
22 39
40
21 19 conduct joint distillation with conventional
KD [ during student training, which results higher accuracy in practice than the feature distilla- tion only. We also include comparisons with collaborative learning methods such as DML 4 and KDCL. [) and curriculum learning technique, RCO 28). We have reproduced the results from the existing methods using the implementations provided by the authors of the papers.
5

---

Table 1: Comparisons between SFTN and the standard teacher models CIFAR-100 dataset when the architectures of the teacher-student pairs are homogeneous. In all the tested algorithms. the students distilled from the teacher models given by SFTN outperform the ones trained from the standard teacher models. All the reported results are based on the outputs of 3 independent runs.
Teacher/Student
WRN40-2/WRNI622
WRN40-2/W6 1
resnet 2x4ressseee
resnet32xdresnee8s2
Teacher training
Stan.
SFTN
Stan.
SFTN
Stun.
SFTN
A
Stun.
SFTN
A
Teacher Acc.
76.30
78.20
76.30
77.02
79.25
79,41
79.25
77.89
Student Acc. w/o KD
73.41
72.16
72.38
68.19
75,46
76.25
+0.79
73.73
75.09
+1.36
73.39
76.09
+-2.70
43
69.17
+1.74
FitNets 18
75.72
76.73
+1.01
74.14
75.54
+1.40
75.34
76.89
+1.55
69.80
71.07
+1.27
AT 20
75.85
76.82
+0.97
74.56
75.86
+1.30
74.98
76.91
+1.93
68.79 79
70.90
+2.11
SP
75.43
74.51
+0.80
76.37
+2.31
68.39
70.03
VID 58
75.63
76.79
+1.16
74.21
75.76
+1.55
74.86
77.00
+-2.14
69.53
71.08
+1.55
RKD
75.48
76.49
+1.01
73.86
75.11
+1.25
74.12
76.62
+2.50
68.54
70.91
+2,36
PKT 39
75.71
76.57
+0.66
74.43
75.49
+1.06
76:57
+1.87
69.29
70.75
AB
(40
70.12
70.76
+0.64
74.38
75.51
+1.13
74.73
76.51
+-1.78
69.76
71.05
+1.29
FT 21
75.6
76.51
-0.11
74.49
75.11
+-0.62
74.89
77.02
+-2.13
69.70
71.11
+1.40
CRD 25
75.91
77.23
+1.32
74.93
76.09
+1.16
75.54
76.95
+1.41
70.34
71.34
SSKD 26
75.96
76.80
+0.84
75.72
76.03
+0.31
75.95
76.85
+0.90
69.34
70.29
+0.96
OH 19
76.00
76.39
0.39
74.79
75.62
+-0.3
75.04
76.65
+1.61
68.10
69.69
+1.59
Best
76.00
77.23
1.25
75.72
76.09
+0.37
75.95
77.02
+1.07
70.34
71.34
+1.00
Table 2: Comparisons between SFTN and the standard teacher models CIFAR-10 100 dataset when the architectures of the teacher-student pairs are heterogeneous. In all the tested algorithms, the student models distilled from the teacher models given by SFTN outperform the ones trained from the standard teacher models. All the reported results are based on the outputs of 3 independent runs.
Teacher/Student
resnet32xd/Shuflle2 i
ResNar50/VGG8
REN40-2/ShuutleV
Teacher training
Stan.
SFTN
Stan.
SFTN
A
Stan.
SFTN
A
Stan.
SPTN
A
Teacher Acc.
79.25
80.03
79.25
79.58
78.70
82.52
76.30
78.21
Student Acc. w/o KD
71.95
73.21
71.12
73.21
KD [
74.26
77.93
+3.67
75.25
78.07
+2.82
73.82
74.92
+1.10
76.68
78.06
+1.38
PitNets 18
75.95
78.75
-.880
77.00
79.68
+.2.68
73.22
74.80
+11.58
77.31
79.21
+1.90
AT 20
76.12
78.63
+2.51
76.57
78.79
+2.22
73.56
74.05
+-0.49
77
78.29
+0.88
SP
75.80
2..56
76.11
78.38
+-2.7
74.02
75.37
+11.35
76.93
78.12
+1.19
VID 38
75.16
78.03
+2.87
75.70
78.49
+2.79
73.59
74.76
+1.17
77.27
78.78
+1.51
RKD
74.84
77.72
+2.88
75.48
77.77
+-229
73.54
74.70
+1.16
76.69
78.11
+1.42
PKT 39
75.05
77.46
+2.41
75.79
78.28
+-2.49
75.17
+1.38
78.28
AB 40
75.95
78.53
+2.58
76.25
78.68
+2.43
73.72
74.77
+1.05
77.28
78.77
+1.49
FT 21
75.58
77.84
+2.26
76.42
78.37
+1.95
73.34
74.77
+1.43
76.80
77.65
+0.85
CRD 25
75.60
78.43
+2208
75.41
+0.89
78.81
SSKD 26
78.05
79.10
+1.05
78.66
79.65
+-0.99
76.03
76.95
+0092
77.81
78.34
+0.53
OH [19
77.51
79.56
+2.05
78.08
79.98
+1.90
74.55
75.95
+1.40
77.82
79.14
+1.32
78.05
79.98
76.95
+0.92
77.82
4.2 Main Results
To show effectiveness of SFTN, we incorporate SFTN into various existing knowledge distillation algorithms and evaluate accuracy. We present implementation details and experimental results on CIFAR-100 100 [1 and ImageNet 33 datasets.
4.2.1 CIFAR-100
CIFAR-100 31 consists of 50K training images and 10K testing images in 100 classes. We select 12 state-of-the the art distillation methods to compare accuracy of SFTNs with the standard teacher networks. To show the generality of the proposed approach, pairs of teacher and student models have been tested in our experiment. The experiment setup for CIFAR- is identical to the one performed in CRD most experiments employ the SGD optimizer with learning rate 0.05, weight decay 0.0005 and momentum 0.9 while learning rate is set to 0.01 in the ShuffleNet experiments. The hyperparameters for the loss function are set as AT = 1, A student-aware training while = 4 in knowledge distillation.
XKL 3, and F
Table ! and 2 demonstrate the full results on the CIFAR-100 dataset. Table I presents the distillation performance of all the compared algorithms when teacher and student pairs have the same architecture type while Table2]show the results from teacher-student pairs with heterogeneous architecture styles. Both tables clearly demonstrate that SFTN is consistently better than the standard teacher network

---

82
Student
Beet Student of Standard
Best Student of SFTN
Standard Teacher
78
76
72
70
68
66
Teacher: WFRN40
WRRN4O 2
Tesnel3234
resnet52xd
resnet32x4
resnetS2x4 ShuttleV2
ResNe:50 VISO8
VMRN40 Sihuttee22
Student: WKNN16-2
WHN4D-11
resnatiik2
Figure 3: Accuracy comparison of the best students from SFTN with standard teacher on CIFAR- 100. The four best student models of SFTN (blue) outperform the standard teachers (gray) while the only one best student of the standard teacher (red) achieves higher aceuracy than its teacher (gray).
Table 3: Top-1 and Top-5 validation accuracy on ImageNet of SFTN in comparison to other methods.
Teacher/Student
ResNet50/Resaar3
Top-1
Top-5 SFTN
Teacher training
Standard
SFTN
Standard
Teacher Acc.
76.45
77.43
93.15
93.75
Student Acc. w/o KD
73.79
91.74
KD
73.55
74.14
+0.59
91.81
92.21
+0.40
FirNets [8
74.56
75.01
+11.45
92.31
92.51
+0.20
SP 377
74.95
75.53
+0.58
92.54
92.69
+0.15
CRD 23
75.01
75.39
+-0.88
92.56
92.67
+0.11
OH [
74.56
75.01
+00.45
92.36
92.56
+0.20
75.01
75.33
+0.52
92.56
92.69
+0.13
all algorithms. The average difference between SFTN and the standard teacher is 1.58% points. and the average difference between best student accuracy of SFTN and the standard teacher is 1. 1.10% points. We note that the outstanding performance of SFTN not only driven by the higher accuracy of teacher models achieved by student-aware learning technique. As observed in Table [ and proposed approach often presents substantial improvement compared to the standard distillation methods despite similar lower teacher accuracies. Refer to Section 4.4 for the further discussion about the relation of accuracy between teacher and student networks.
Figure 3 iillustrates the accuracies of the best student models of the standard teacher and SFTN given teacher and student architecture pairs. Despite the small capacity of the students, the best student models of SFTN sometimes outperform the standard teachers while the only one best student of the standard teacher shows higher accuracy than its teacher.
4.2.2 ImageNet
ImageNet 30 consists of 1.2M training images and 50K validation images for 1K classes. We adopt the standard Pytorch set-up for ImageNet training for this experiment" The optimization given by SGD with learning rate 0. weight decay 0.0001 and momentum 0.9.
The coefficients of individual loss terms We conduct the ImageNet experiment for 5 different knowledge distillation methods, where teacher models based ResNet50 transfer knowledge to student networks with ResNet34.
ÀT 1, A 1, and X 1, where 7 1.
As presented in Table 3 SFTN consistently outperforms the standard teacher network in all settings. The best student accuracy of SFTN achieves the higher top-1 I accuracy than the standard teacher model by approximately 0.5% points. This results implies that the proposed algorithm has great potential on large datasets as well.
4.3 Comparison with Collaborative and Curriculum Learning Methods
Contrary to traditional knowledge distillation methods based on statie pretrained teachers, collab- orative learning approaches employ dynamic teacher networks trained jointly with students and curriculum learning methods keep track the optimization history teachers for distillation. Table
shows that SFTN outperforms the collaborative learning techniques such as DML [4] and KDCL [5]

---

Table 4: Comparision with collaborative and curriculum learning approaches CIFAR-100 100. We employ KDCL-Naive for ensemble logits of KDCL. RCO is based on the one-stage EEI (equal epoch interval) while RCO-EEI-4 adopts 4 anchor points selection with the EEI strategy. Note that both RCO-EEI-4 and SFTN-4 are trained for 240 x 4 epochs.
Teacher
WRN40-2 WRN16-2
WRN40-2
resnet32x4
resnet32x4 ShuffleV1
res
ResNet50 VGG8
Student
WRN40-1 1
resmet8x4
Shuflle
Standard teacher Acc.
76.30
76.30
79,25
79.25
79.25
78, 70
Student Acc. w/o KD
73.41
72.16
72.38
71.95
73.21
71.12
Stu.
Stu.
Stu.
A
Stu.
Stu.
A
Stu.
Standard
75.46
-2.05
13:73
+1.57
73.39
+1.01
74.26
+2.31
73.25
+2.04
73.82
+2.70
DML 4
75.30
-1.89
74.08
+1.92
74.34
+-196
73.37
+1.42
73.80
+ 0.59
73.01
+1.89
KDCL 5
75.45
+2.04
74.65
+2.49
75.21
+2.83
73.98
+ 2.03
74.30
+1.09
73.48
+2.36
RCO 28
75.36
+1.95
74.29
+2.13
74.06
+1.68
76.62
+4.67
77.40
+4419
74,30
+3.18
SFTN
76.25
-.884
75.09
+2293
76.09
+3.71
77.93
78.07
74.92
+3.80
RCO-EEL4 28
75.69
2.28
74.87
+2.71
73.73
+1.35
76.97
+5.02
76.89
+3.68
74.24
+3.12
SFTN-4
76.96
+3.55
76.31
+4.15
76.67
+4.29
79.11
+7.16
78.95
+5.74
75.52
+4.40
Table 5: Effect of varying F in the KL-divergence loss of the student-aware training tested CIFAR- 100, where for the knowledge distillation is set to 4. The student accuracy is fairly stable over a wide range of the hyperparameter. Note that the accuracies of SFTNs and the student model are rather inversely correlated, which implies that the maximization of teacher models not necessarily ideal for knowledge distillation.
Accuracy SFTN
accuracy
Teacher Student
2es
WRN40-2
WRN40-2
ShuffleV1
ShuffleV2
WRN16-2
WRN40-1
Avg.
ShuffleV1
ShufileV2
WRN16-2
WRN40
Avg.
==1
81.19
80.26
78.23
78.14
78.85
76.05
77.18
76.30
74.75
75.58
f= 5
81.23
81.56
79.22
78.31
79.54
75.36
75.59
76,31
73.64
75.10
10
81.27
78.81
78.38
79.58
74.47
75.93
75.85
73.62
74.76
f 15
81.89
81.74
79.27
78.63
79.74
74.78
75.65
75.79
73.49
74.81
F 20
81.60
81.70
78.84
78.45
79.59
74.62
75.88
75.82
74.03
74.95
the heterogeneous architectures turn out to be effective for mutual learning. On the other hand, the accuracy of SFTN is consistently higher than that of the curriculum learning method, RCO [28], under the same and even harsher training condition in terms of the number of epochs. Although the identification the optimal checkpoints may be challenging in the trajectory-based learning. SFTN improves its accuracy substantially with more iterations shown in the results for SFTN-4.
4.4 Effect of Hyperparameters
SFTN computes the KL-divergence loss, CKL, to minimize the difference between the softmax outputs of teacher and student branches, which involves two hyperparameters, temperature of the softmax function, т, and weight for KL-divergence loss term, A We discuss the impact and trade-off issue of the two hyperparameters. In particular, we present our observations that the student-aware learning is indeed helpful to improve the accuracy of student models while maximizing performance of teacher models may be suboptimal for knowledge distillation.
Temperature softmax function The temperature parameter of the KL-divergence loss in (5) denoted by F, controls the softness of द and q as F gets higher, the output of the softmax function becomes smoother. Despite the fluctuation in teacher accuracy, student models given by knowledge distillation via SFTN maintain fairly consistent results. Table alsso shows that the performance of SFTNs and the student models is rather inversely correlated. This result implies that a loosely optimized teacher model may be more effective for knowledge distillation according this ablation study.
Weight for KL-divergence loss The hyperparameter KL facilitates knowledge distillation by making dr similar to d However, it affects the accuracy of teacher network negatively. Table 6 shows that the average accuracy gaps between SFTNs and the corresponding student models drop gradually as A increases. One interesting observation the student accuracy via SFTN with JKL = 10 in comparison to its counterpart via the standard teacher; even though the standard teacher network is more accurate than SFTN by large margin, corresponding student accuracy lower than that of SFTN.
AR
8

---

Table 6: Effect of varying A in the knowledge distillation via SFTN tested on CIFAR-I00, The accuracies of SFTNs and the corresponding students are not correlated while the accuracy gaps of the Iwo models drop as AR increases.
Accuracy of SFTN
Student accuracy by KD
Teacher Student
ShufteVI
resnet32x4
ShuffleV2
WRN16-2
WRN40-2
WRN40-1
Avg.
ShuffeeV1
resnet32x4
Shuffee22
WRN16-2
WRN40-2
WRN40-1
Avg.
81.19
80.26
78.23
78.14
79.46
76.05
77.18
76.30
74.75
76.07
Ag
78.70
79.80
77.83
77.57
78.48
77.36
78.56
76.20
74.71
76.71
78.29
78.29
77.28
76.05
77.48
77.33
77.70
76.02
74.67
76.43
10
73.02
75.01
75.03
73.51
74.14
75.57
76.62
74.19
73.08
74.87
Standard
79.25
79.25
76.30
76.30
77.78
74.31
75.25
73.28
73.56
74.60
Table 7: Effectiveness of knowledge distillation via SFTN when student models have different capacity compared to the used in the student-aware training. This experiment is conducted on CIFAR- 100. The numbers in bold and red denote the best and the second-best results. SB denotes student branch. (MI: resmet8x2, M2: resnet8x4, M3: resnet32x4, M4: WRN16-1, M5:WRNI6-2,
M6: WRN40-2, M7: ShuffleV2)
Teacher/Student
WRN4O-2/VRN 16-2 (M6/MS)
Nesee (M3/M))
SB cupacity
N/A
Smaller
Equal/similar
Larger
NIA
Smaller
Equal/similar
Larger
SB model
SB Acc.
68.19
67.10
73.41
73.21
79.25
76.30
67.10
68.19
72.38
73.21
79.25
76.30
Teacher
76.30
76.22
75.98
78.20
78.21
78.82
78.69
79.25
76.53
77.89
79.41
79.58
80.85
80.30
KD
75.46
74.67
74.73
76.25
75.68
75.56
75.63
73.39
74.71
75.19
76.09
75.82
75.19
75.03
SP
37
75.43
74.75
75.29
76.77
76.56
76.13
76.08
74.06
75.31
75.76
76.37
76.09
75.62
75.36
FT 21
75.60
74.61
75.23
76.51
76.66
76.47
76.23
74.89
75.79
76.54
77.02
76.62
76.48
76.63
CRD
23
75.91
76.14
76.07
77.23
77.45
77.06
76,70
75.54
76.38
76.72
76,95
76.64
76.54
76.46
SSKD
75.96
74.35
74.41
76.80
76.73
76.22
76.67
76.24
OH
[9
76.00
74.95
74.97
76.39
76.49
76.27
76.15
75.04
75.65
75.69
76.65
76.48
76.38
76.44
Average
75.73
74.91
75.12
76.66
76.56
76.37
76.25
74.81
75.48
75.95
76.66
76.31
76.15
76.03
Best
76.00
76.14
76.07
77.23
77.45
77.06
76.72
75.95
77.02
76.64
76.67
76.63
4.5
Versatility of SFTN
Although our teacher network obtained from the student-aware training procedure is specialized for a specific student model, it also effective to transfer knowledge to the students models with substantially different architectures. Table shows that the benefit of method is also preserved well as long as the student branch has similar capacity to the student models, where the model capacity defined by the achievable accuracy via independent training without distillation. In addition, presents that larger students branches are often effective to enhance distillation performance while smaller student branches are not always helpful. In summary. these results imply that a teacher network in SFTN trained for a specific architecture of student network has the potential to transfer its knowledge other types of student networks.
4.6 Use of Pretrained Teachers
The main goal of knowledge distillation is maximize the benefit in student networks, and the additional training cost may not be critical in many real applications. However, the increase of training cost originated from the student branch of the teacher network is still undesirable. We can sidestep this limitation by adopting pretrained teacher networks in the student-aware training stage. The training cost of SFTN teacher networks is reduced significantly by using pretrained models, and T presents the tendency clearly. Compared to 240 epochs for the standard student-aware training, fine-tuning pretrained teacher networks only needs 60 epochs for convergence; we train the student branches only for the first 30 epochs and fine-tune the whole network for the remaining 30 epochs. Table / shows that fine-tuned pretrained teacher networks have potential to enhance distillation performance. They achieve almost same accuracy with the full SFTN in 6 knowledge distillation algorithms.
4.7 Similarity between Teacher and Student Representations
The similarity between teacher and student models is an important measure for knowledge distillation performance in the sense that student network aims to resemble the output representations of

---

Table 8: Training time of SFTN teachers with pretrained models. The additional training time in each SFTN with pretrained teacher model presented the last column; is significantly reduced compared to the standard SFTN teacher (the second-last column).
Training time (s2c)
Models (leacher/student)
Teacher
Student
SFTN teacher SFTN teacher with pretrained model
6,005
5,624
10,910
2,298
resnet32x4/Shuttlee2
6,005
6,221
10.949
2,370
WRRNA-2/WRNING-
3,940
1,745
6,028
1,178
WRNNO-2/WRN40-1
3,940
3,698
7,431
1,621
Table 9: Performance of SFTN fine-tuned from a pretrained teacher (SFTN-FT) on CIFAR-100.
Table 10: Similarity measurements between
teachers and students on CIFAR-100.
Teacher/Student
resnet32xd/ShufteV2
Teacher training method
Standard
SFTN
SFTN+FT
Models (tcacher/studeet)
4esncc22x4Shufft222
Teacher
79.25
80.03
80.41
Simularity metric
KL-divergence
CKA
Student Acc. wo KD
71.95
Teacher training method
Standard
SPTN
Standard
SPTN
KD
75.25
78.07
78.12
KD
D
1.10
0.47
0.88
0.95
SP
37
76.11
78.38
78.51
FaiNets
FT A]
76.42
78.37
77.90
SP 17
0.95
0.45
0.89
0.95
CRD 23
76.35
78.43
78.88
SSKD
26
78.66
79.65
79.15
CRD 23
0.81
0.43
0.88
0.95
OH
[9]
78.08
79.98
79.68
SSKD 26
0.54
0.26
0.92
Averuge Best
76.81
78.81
78.71
OH 19
0.85
0.37
0.90
0.96
78.66
79.98
79.68
AVG
0.84
0.89
teacher network. We employ KL-divergence and CKA 41 as similarity metrics, where lower KL-divereence and higher CKA indicate higher similarity.
Table TO presents the similarities between the representations ResNet32x4 and S2, respectively, which given by various algorithms the CIFAR-100 test set. The results show that the distillations from SFTNS always give higher similarity to the student models with respect to the corresponding teacher networks; SFTN reduces the KL-divergence by more than 50% in average while improving the average CKA by 7% points compared to the standard teacher network. The improved similarity of SFTN is natural since it is trained with student branches to obtain student-friendly representations via the KL-divergence loss.
teacher and a student based on
5 Conclusion
We proposed simple but effective knowledge distillation approach by introducing the novel student- friendly teacher network (SFTN). Our strategy sheds light in new direction to knowledge distilla- tion by focusing on the stage to train teacher networks. We train teacher networks along with their student branches, and then perform distillation from teachers to students. The proposed strategy turns out achieve outstanding performance, and can be incorporated into various knowledge distillation algorithms easily. For the demonstration of the effectiveness of our strategy, we conducted compre- hensive experiments in diverse environments, which show consistent performance gains compared to the standard teacher networks regardless of architectural and algorithmic variations.
The proposed approach is effective for achieving higher accuracy with reduced model sizes, but it is not sufficiently verified in the unexpected situations with out-of ontoo-ddistriuttinn inputs, domain shifts, lack of training examples, etc. Also, still rely on large number of training examples and still has the limitation of high computational cost and potential privacy issue.
6 Acknowledgments and Disclosure of Funding
This work was partly supported by Samsung Electronics Co., Ltd. (I0200626-07474-000 and Institute for Information & Communications Technology Promotion (IITP) grant funded by the Korea government (MSIT) [2017-0-01779; XAI, 2021-0-01343 Artificial Intelligence Graduate School Program (Seoul National University)).
References
[1] Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the Knowledge in a Neural Network. In NeurIPS Deep Learning and Representation Learning Workshop. 2015.
10

---

[2] Minsoo Kang, Jonghwan Mun, and Bohyung Han. Towards oracle knowledge distillation with neural architecture search. In AAAI, 2020.
[3] Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant: Bridging the gap between student and teacher. In AAAI, 2020.
[4] Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep Mutual Learning. In CVPR, 2018.
[5] Qiushan Guo, Xinjiang Wang, Yichao Wu, Zhipeng Yu, Ding Liang, Xiaolin Hu, and Ping Luo. Online knowledge distillation via collaborative learning. In CVPR, 2020.
[6] Guile Wu and Shaogang Gong. Peer collaborative learning for online knowledge distillation. In arXiv 2006.04147, 2020.
[7] Guobin Chen, Wongun Choi, Xiangdong Chen, Tony X. Han, and Manmohan Krishna Chandraker. Learning Efficient Object Delection Models with Knowledge Distillation. In NeurIPS, 2017.
[8] Ping Luo, Zhenyao Zhu, Ziwei Liu, Xiaogang Wang, and Xiaoou Tang. Face Model Compression by Distilling Knowledge from Neurons. In AAAI, 2016.
[9] Ancong Wu, Wei-Shi Zheng. Xiaowei Guo, and Jian-Huang Lai. Distilled person re-identification: Towards more scalable system. In CVPR, 2019.
[10] Huan Wang. Yijun Li, Yuehai Wang, Haoji Hu, and Ming-Hsuan Yang. Collaborative distillation for ultra-resolution universal style transfer. In CVPR, 2020.
[11] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling BERT for natural language understanding. In 19X9vv1909.1033. 2019.
[12] Jonghwan Mun, Kimin Lee, Jinwoo Shin, and Bohyung Han. Learning to Specialize with Knowledge Distillation for Visual Question Answering. In NeurIPS, 2018.
[13] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, 2019.
[14] Siddhartha Arora, Mitesh M. Khapra, and Harish G. Ramaswamy. On knowledge distillation from complex networks for response prediction. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, NAACL, 2019.
[15] Fida Mohammad Thoker and Juergen Gall. Cross-modal knowledge distillation for action recognition. In ICIP, 2019.
[16] Mingmin Zhao, Tianhong Li, Mohammad Abu Alsheikh, Yonglong Tian, Hang Zhao, Antonio Torralba, and Dina Katabi. Through-wall human pose estimation using radio signals. In CVPR, 2018.
[17] Brady Zhou, Nimit Kalra, and Philipp Krähhenühh.. Domain adaptation through task distillation. In ECCV, 2020.
[18] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. FitNets: Hints for Thin Deep Nets. In ICLR, 2015.
[19] Bycongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi, comprehen- sive overhaul of feature distillation. In ICCV, 2019.
[20] Sergey Zagoruyko Nikos Komodakis. Paying More Attention to Attention: Improving Performance of Convolutional Neural Networks via Attention Transfer. In ICLR, 2017.
[21] Jangho Kim, Seong Uk Park, and Nojun Kwak. Paraphrasing Complex Network: Network Compression via Factor Transfer. In NeurlPS, 2018
[22] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational Knowledge Distillation. CVPR, 2019.
[23] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR. 2020.
[24] Xu Lan, Xiatian Zhu, and Shaogang Gong. Knowledge Distillation by On-the-Fly Native Ensemble. In NeurlPS, 2018.
[25] Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher; Improve the performance of convolutional neural networks via self distillation. In ICCV, 2019.
11

---

[26] Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. Knowledge distillation meets self-supervision. In ECCV, 2020.
[27] Tommaso Furlanello, Zachary Chase Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born-again neural networks. In ICML, 2018.
[28] Xiao Jin, Baoyun Peng, Yichao Wu, Yu Liu, Jiaheng Liu, Ding Liang, Junjie Yan, and Xiaolin Hu. Knowledge distillation via route constrained optimization. In ICCV, 2019.
[29] Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In ICCV, 2019.
[30] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis., 155():211-2522 2015.
[31] Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images, Technical report, Citescer, 2009.
[32] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CVPR, 2016.
[33] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
[34] Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. In ICLR, 2015.
[35] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In CVPR, 2018.
[36] Mingxing Tan, Bo Chen, Ruoming Pang. Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe V. Lc. Mnasnet: Platform- aware neural architecture scarch for mobile. In CVPR, 2019.
[37] Frederick Tung and Greg Mori. Similarity- preserving knowledge distillation. In ICCV, 2019.
[38] Sungsoo Ahn, Shell Xu Hu, Andreas C. Damianou, Neil D. Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In CVPR, 2019.
[39] Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In ECCV, 2018
[40] Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. In AAAI, 2019.
[41] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey E. Hinton. Similarity of neural network representations revisited. In ICML, 2019.
12


Reviews:
[{"review_id": "lEtFHbeAunm", "reviewer": "Reviewer_2smh", "summary": "This paper proposes a new KD framework named SFTN that optimizes both the teacher and the student at the same time for better knowledge transfer. The experiments show that the proposed method can improve the performance of many KD methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Pros\n- The paper is well-written.\n- The performance is pretty good - for different teacher-student pairs and different compression methods.\n- The authors have made improvements since their ICML rejection and the added results are convincing.\n\n### Cons\n- Designing student branches is non-trivial. I would like to see more discussion about the impact of different student branch designs. This is also a weakness of the proposed method since it requires human expertise for this job.\n- I'm concerned about the computational overheads. The student branches should have a large impact on the training speed and memory etc., especially when there are multiple branches (when one tries to compress a huge model). If I'm not mistaken, the overhead should be $O(n \\log n)$, where $n$ is the number of intermediate features/student branches. Meanwhile, a traditional knowledge distillation method can easily scale because its overhead is $O(n)$. This is clearly a limitation that should be addressed in the paper.\n\n### Questions for the Authors\n- How to distill when the teacher and the student have the same basic blocks? For example, compressing a 12-layer BERT model to a 6-layer one (which is a classic model compression setting in NLP).\n- I'm not sure if the authors heard about module replacing for model compression [1] which employs a very similar idea of training the student with the teacher. While the difference is obvious, I wonder is there any shared mechanism between these two methods?\n\n### References\n[1] https://arxiv.org/abs/2002.02925", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, but I hope to see more discussion about the additional computational overheads of the proposed method.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "Q6ygdY8PsEO", "reviewer": "Reviewer_6fou", "summary": "1. \nThis submission proposes that large models intended for distillation be regularized with auxiliary loss terms that represent a compromise between full collaborative learning, where independent teacher and student networks are trained jointly, and completely disjointed conventional knowledge distillation. The submission is an empirical paper, and considers a variety of network architectures and distillation procedures on CIFAR-100 and ImageNet. \n\n2. I think the main idea has the potential to be very impactful. Since this paper is entirely empirical, everything hinges on flawless execution of the experiments in order for the results to have any credence. Although the authors have clearly gone to great lengths to compare a broad variety of architectures and methods, they seem to have neglected some of the most fundamental priorities when comparing methods empirically. I would rather see a comparison between a handful of carefully tuned baselines than a large table of untuned comparisons. If my concerns can be addressed, I think this could be a very good paper. However in its current state I have enough doubt about the results that I cannot recommend acceptance.\n\nUPDATE 09/02/2021\nDue to a very strong response from the authors, I am increasing my score to 7, good paper.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "3. (Relevance/Significance)\nGiven recent work on distillation that seems to identify optimization as a crucial bottleneck in distillation, this submission is very relevant to the NeurIPS community. In the context of self-distillation, we *know* that the teacher representations can be perfectly emulated by an identical network, yet in practice self-distillation produces students that are very different from their teachers. Furthermore there is little evidence to suggest changing the distillation dataset, the augmentation procedures, or the optimizer can really resolve the issue  [2, 3]. Therefore papers that investigate interventions that alter the distillation loss surface in some way, either by giving the student information about intermediate activations (like FitNet) or make the teacher labels easier to match (like this paper) are very interesting. \n\n4. (Motivation)\nThe motivation to use auxiliary heads to regularize the teacher rather than an independent student network is reasonable. It is not hard to imagine that if a teacher is regularized to be similar to a small student then a ceiling on the teacher's performance is imposed that potentially negates the effects of having a large teacher in the first place. Furthermore the ceiling cannot be raised except by making the student larger, which may not be feasible if the student size is dictated by hardware constraints. Treating the student branches as part of the teacher gives more flexibility, although obviously the correspondence of the capacity of the student branches and the student network is probably still a relevant factor.\n\n5. (Correctness)\nI am concerned that the logit temperature is not treated correctly. If equations (1) and (4) are taken at face value, then the main teacher head logits are tempered not just for the auxiliary distillation loss term, but also for the standard cross-entropy loss term. I believe this is either a mistake in the implementation or an oversight in the writing; unfortunately since the code was not included I cannot tell which. Obviously for \\tau=1 the mistake is fairly benign, but I think it potentially completely compromises the temperature ablation experiment. The result of the temperature ablation (\\tau^* = 1) is completely opposed to similar results in distillation literature, which heightens my concern. In nearly every other similar experiment in the literature the best temperature is not 1 (see, for example, the ablations in [2] or [4]). \nThis also raises a concern about what temperature was used in the vanilla KD baseline. If the vanilla KD baseline used a temperature of 1 it was almost certainly undertuned, which would be a significant error. \n\n6. (Empirical Rigor)\n\n6.a) The number of architectures considered is commendable, along with the experiment with fine-tuned pre-trained teacher networks. \n\n6.b) The loss term coefficient ablations may be compromised by the apparent decision not to constrain the coefficients to a simplex (i.e. take the total loss to be a convex combination of the individual losses). Unless the learning rate is being adjusted, changing the scale of the loss by making unnormalized changes to the coefficients is potentially a serious confounder. In general it is simplest to simply constrain the coefficients to a simplex instead. \n\n6.c) Error bars appear to be given for only one experiment in line 32 of the appendix, in contradiction to the author’s response to the relevant checklist question. I would like to emphasize that section 3 of the checklist is not a mere formality. It explicitly states common expectations of empirical rigor for NeurIPS submissions. Large tables of experiments are not helpful if the reader cannot be confident that they represent the whole picture.\n\n7. (Completeness)\n\nThe greatest omission from this submission is any attempt to explain or understand why SFTN works. Table 9 is arguably the most important experiment in the paper if the authors wish to argue that SFTN is doing something more profound than simply regularizing the teacher in some clever way, which indeed is exactly what is claimed in the first sentence of the abstract. Figure 1 in the appendix (note that the figure numbering should not be reset to avoid confusion with the main text) is also helpful by providing evidence that SFTN is not improving student accuracy simply by regularizing the teacher to generalize better. Here the authors have a natural opportunity to provide an explanation for why SFTN works, an opportunity that is not taken up. I would encourage the authors to carefully design a self-distillation experiment that gives insight into why SFTN teachers are easier to distill than normal teachers. One possible explanation is that SFTN does something similar to tempering, in that it artificially raises or lowers the entropy of the teacher labels. I would also like to see top-1 agreement reported in Table 9, since it is a simple, interpretable measure of functional correspondence between teacher and student. \n\n8. (Understanding of Related Work)\n\nThe idea is very similar to the auxiliary classifier heads proposed in the original Inception paper [1] (also known as auxiliary towers), a connection which is worth noting in the discussion of related work. \n\n---------------\nReferences\n\n[1] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., ... & Rabinovich, A. (2015). Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1-9).\n\n[2] Stanton, S., Izmailov, P., Kirichenko, P., Alemi, A. A., & Wilson, A. G. (2021). Does Knowledge Distillation Really Work?. arXiv preprint arXiv:2106.05945.\n\n[3] Beyer, L., Zhai, X., Royer, A., Markeeva, L., Anil, R., & Kolesnikov, A. (2021). Knowledge distillation: A good teacher is patient and consistent. arXiv preprint arXiv:2106.05237.\n\n[4] Cho, J. H., & Hariharan, B. (2019). On the efficacy of knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 4794-4802).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "9. \nThe work only explores distillation in the context of image classification.\n\n10. \nThis work does not report the sensitivity of their method to dataset shift, which is very relevant in the context of distillation, because distillation is usually motivated as a means to deploy ML models to small edge devices where dataset shift is almost inevitable. Hence it is extremely important to at least quantify the sensitivity to shift, rather than simply identify it as an unknown risk in the broader impacts section.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "9iP3iS-JXXA", "reviewer": "Reviewer_sdwv", "summary": "The author presents a novel knowledge distillation approach to assist the learning of better teacher models that are friendly to students and consequently a better-performing student network. For this purpose, the purposed method is a two-stage learning schema where the teacher architecture is firstly trained jointly with the Student, and then for distillation. The learning objective during the two phases is easy while looks surprisingly effective. The experiments presented in the paper are comprehensive and sufficient. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method presented in the paper is easy and not investigated (as far as claimed by the author). The proposed two-stage training schema is simple while yet surprisingly effective. \n\n- Despite so, my major concern lies in intuitive explanations about why student-specific distillation makes the distillation more effective. In essence, the training of the Teacher is not only optimized by the original CE losses, but also to the same `\"hidden space\" of the Student because of an additional regularization term. I am curious about what exactly leads the Teacher better representations suitable for Students: why this KL regularization leveraging the Student block can be helpful? \nThe paper indeed gives an explanation intuitively that more consistent Teacher&Student features will be approached, but in such a case, shouldn't the online distillation achieve the same effect? (i.e., allowing Teacher network to be further optimized during distillation stage for longer epochs). \n\n- Following my previous comment, an experiment can be: consider attaching more visualizations, or ablations studying the consistency of features across the Teacher/Student before/after SFTN, e.g.: T-SNE visualizations or statistical distance of features between T-S, comparing with the T-S without the SFTN, also the baseline of online distillation.\n\n- From another step, adopting the two-stage training can potentially lead to further computational burdens as well as a limited applicable scope. As for most of the real scenarios, it will remain unknown to what particular student architectures will be leveraged. And choosing and training a new Teacher per student case can be luxury and costly. This also raises my concern about the applicability of the proposed student-specific learning paradigm.\n\n- Discuss more recent ongoings bring a broader scope of the works, and is strongly suggested: 1. the neural-architecture-searching (NAS). As now the Teacher needs to be student-specific and training, it would be interesting to see if the NAS can be applied to the Teacher to search for an optimal Teacher architecture for a specific Student. 2. In addition, the recently emerging self-supervised distillation looks more appropriate&challenging for the student-specific distillation schema as the self-supervised efforts somehow reveal that consistency in self-supervised learning is critical. Some relevant works include BYOL[1], SEED[2] and etc. \n\nAs I'm less familiar with supervised distillation, I'd much like to hear more from other reviewers & responses from authors then adjust my ratings accordingly.\n\n[1]. Grill, Jean-Bastien, et al. \"Bootstrap your own latent: A new approach to self-supervised learning.\" arXiv preprint arXiv:2006.07733 (2020).\n\n[2]. Fang, Zhiyuan, et al. \"Seed: Self-supervised distillation for visual representation.\" arXiv preprint arXiv:2101.04731 (2021).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative social impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "0hJ1ts-FKk", "reviewer": "Reviewer_o6Fn", "summary": "The authors propose student friendly teacher networks (SFTN) that can facilitate better knowledge transfer. Specifically, during the training of the teacher network, the snapshot of the student model is also provided. The teacher network is trained along with the student branches to learn student-friendly representations. Thereafter this trained student-aware teacher network is used for knowledge distillation. They show the efficacy of their method on CIFAR-100 and ImageNet datasets over Resnet, VGG and ShuffleNet architectures. They also do sensitivity analysis of the hyperparameters like temperature of softmax and weight of KL divergence loss.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- As the student is involved during training of the teacher network, the representations from these two networks are similar as evidenced by low kl divergence and high CKA values. \n\n- After the teacher training, any distillation method can be used on top of it. The method obtains improvements over the several distillation methods.\n\n- The writing is clear and easy to read.\n\nWeakness/Limitations:\n\n- One of the key drawbacks is that the teacher network is tied to a particular student architecture. The major advantage of the knowledge distillation over other compression techniques like pruning is its architectural independence i.e. one can transfer knowledge from any architecture to any other architecture, which is being compromised here. In section 4.5, line 245, they limit their method to the student with either the same architecture or same capacity. \n\n- Their method is more suitable when a-priori knowledge of the student network is there during the teacher training. However, this is a very limited use case.\n\n- In Table 3, the improvements are minor, not significant.\n\nQueries:\n\n- There is no BS1 block in Figure 2(a). Also, how to determine how many layers to make a block ?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "One of the key limitations which authors also point out is the increase in training cost due to the inclusion of the student branch of the teacher training stage. They fix it up through the use of pretrained teacher networks. But still, the teacher networks require additional finetuning.\n\nSuggestions:\n\n- Mention systematic ways to modularize teacher and student networks.  Checking the impact of different modularization on distillation performance.\n\n- Perform experiments to check the performance of SFTN (trained for a specific student architecture) on other student networks that neither have the same architecture nor the same capacity.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an auxiliary loss that balances independent teacher and student training and joint training. It is largely an empirical submission, but helps address an interesting open question --- since it has been shown recently that the agreement between student and teacher are often very poor. The authors also put a heroic effort in their responses to reviewers. Please carefully account for all reviewer comments in your revisions. 
Decision: Accept (Poster)