Learning Student-Friendly Teacher Networks for Knowledge Distillation
Authors: Dae Young Park
Moon-Hyun Cha
Changwook Jeong
Daesin Kim
Bohyung Han
Venue: NeurIPS 2021 Poster (2021)
Keywords: deep learning
transfer learning
knowledge distillation

Abstract: We propose a novel knowledge distillation approach to facilitate the transfer of dark knowledge from a teacher to a student. Contrary to most of the existing methods that rely on effective training of student models given pretrained teachers, we aim to learn the teacher models that are friendly to students and, consequently, more appropriate for knowledge transfer. In other words, at the time of optimizing a teacher model, the proposed algorithm learns the student branches jointly to obtain student-friendly representations. Since the main goal of our approach lies in training teacher models and the subsequent knowledge distillation procedure is straightforward, most of the existing knowledge distillation methods can adopt this technique to improve the performance of diverse student models in terms of accuracy and convergence speed. The proposed algorithm demonstrates outstanding accuracy in several well-known knowledge distillation techniques with various combinations of teacher and student models even in the case that their architectures are heterogeneous and there is no prior knowledge about student models at the time of training teacher networks

Learning Student-Friendly Teacher Networks for Knowledge Distillation
Dae Young Park*.l, Moon-Hyun Cha', Changwook Jeong', Dae Sin Kim!, and Bohyung Han*
DITT Center, Samsung Electronics, Korea
ECE & ASRI, Seoul National University, Korea
(p30. daeyoung, moonhyun. cha, thris.jeone daesin. kim Asamsung. com bhhan@snu.c ac.kr
Abstract
We propose novel knowledge distillation approach facilitate the transfer of dark knowledge from a teacher to student. Contrary to most of the existing methods that rely effective training of student models given pretrained teachers, we aim to learn the teacher models that are friendly to students and, consequently, more appropriate for knowledge transfer. In other words, the time of optimizing a teacher model, the proposed algorithm learns the student branches jointly to obtain student-friende representations. Since the main goal of approach lies in training teacher models and the subsequent knowledge distillation procedure is straightforward, most of the existing knowledge distillation methods can adopt this technique to improve the performance of diverse student models in terms of accuracy and convergence speed. The proposed algorithm demonstrates outstanding accuracy in several well-known knowledge distillation techniques with various combinations of teacher and student models even in the case that their architectures are heterogeneous and there is no prior knowledge about student models at the time of training teacher networks.
1 Introduction
Knowledge distillation [ is well-known technique to learn compact deep neural network models with competitive accuracy, where smaller network (student) is trained simulate the representations of larger one (teacher). The popularity of knowledge distillation is mainly due to its simplicity and generality; straightforward to learn a student model based on teacher and there no restriction about the network architectures of both models. The main goal of most approaches is how to transfer dark knowledge to student models effectively, given predefined and pretrained teacher networks.
Although knowledge distillation is a promising and convenient method, it sometimes fails to achieve satisfactory performance in terms of accuracy. This partly because the model capacity of student is too limited compared to that of teacher and knowledge distillation algorithms are suboptimal 2 3 In addition to this reason, we claim that the consistency of teacher and student features is critical to knowledge transfer and the inappropriate representation learning of a teacher often leads to the suboptimality of knowledge distillation.
We are interested in making a teacher network hold better transferable knowledge by providing the teacher with snapshot of the student model the time of its training. We take advantage of the typical structures of convolutional neural networks with multiple blocks and make the representations of each block in teachers easy be transferred to students. The proposed approach aims to train
"Equal contribution
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Teacher Softmax
Student Dranch
Teacher
Softmax
Softmax
Teacher Network
Studeur Branch
Loss
Teacher
Tencher Softnux
Netwerk
Hard
Student
Easy
Knowledge
Knowledge
Teacher
Nesork
Studens Nerwork
Loos
Teacher
Loss
Studem: Netw
Tosss
Network
Label
Sandent
Teacher
Smidett
Sottmax
Label
Softmax
Label
Sothmax
Libel
Training Teacher
Distillation
Training Student-Aware Teacher
Distillation
(a) Standard knowledge distillation
(b) Student-friendly teacher network
Figure 1: Comparison between the standard knowledge distillation and our approach. (a) The standard knowledge distillation trains teachers alone and then distill knowledge students. (b) The proposed student-frient friendly teacher network trains teachers along with student branches, and then distill more casy-to-transfer knowledge students.
teacher models friendly to students for facilitating knowledge distillation: we call the teacher model trained by this strategy student-friendly teacher network (SFTN). SFTN deployed in arbitrary distillation algorithms easily due to its generality for training models and transferring knowledge.
SFTN is partly related collaborative learning methods 4 5 6) which may suffer from the
correlation between the models trained jointly and fail to fully exploit knowledge in teacher models. On the other hand, SFTN is free from the limitation since it performs knowledge transfer from teacher to a student in one direction via a two-stage learning procedure-ssuueet---n teacher network followed by knowledge distillation from a teacher a student. Although the structure of a teacher network depends on target student models, it is sufficiently generic to be adopted by [ demonstrates the main difference between the proposed
training of
students with various architectures. Figure algorithm and the standard knowledge distillation methods.
The following the list of main contributions:
We adopt student-and aware teacher learning procedure before knowledge distillation, which enables teacher models to transfer their representations to students more effectively.
The proposed approach applicable diverse architectures teacher and students while can be incorporated into various knowledge distillation algorithms.
We demonstrate that the integration of SFTN into various baseline algorithms and models improve accuracy consistently with substantial margins.
The rest of the paper is organized as follows. We first discuss the existing knowledge distillation techniques in Section Section Seetion@desccibee the details the proposed SFTN including the knowledge distillation algorithm. The experimental results with in-depth analyses are presented Section and we make the conclusion in Section
2 Related Work
Although deep learning has shown successful outcomes in various fields, it is still difficult to apply deep neural networks to real-world tasks due their excessive requirement for computation and memory. There have been many attempts to reduce the computational cost of deep learning models, and knowledge distillation one of the examples. Various computer vision 7 g 10] and natural language processing [II
B3 [4] tasks often employ knowledge distillation to obtain efficient models. Recently, some cross mooss-moaal tasks 15 16 transfer knowledge across domains. This section summarizes the research efforts to improve performance of models via knowledge distillation.
12
2.1
What to distill
Since Hinton al. [ introduce the basic concept knowledge distillation, where the dark knowledge in teacher models is given by the cemperature-scaled representations of the softmax function, various

---

kinds of information have been employed as the sources of knowledge for distillation from teachers to students. FitNets [88 distills intermediate features of teacher network, where the student network transforms the intermediate features using guided layers and then calculates the difference between the guided layers and the intermediate features of teacher network. The position of distillation is shifted to the layers before the ReLU operations in [19], which also proposes the novel activation function and the partial Le loss function for effective knowledge transfer. Zagoruyko and Komodakis (20) argue importance of attention and propose an attention transfer (AT) method from teachers to students while Kim et al. [2] compute the factor information of the teacher representations using an autoencoder, which is decoded by students for knowledge transfer. Relational knowledge distillation (RKD) [2] introduces a technique to transfer relational information such as distances and angles of features.
maximizes mutual information between teacher and a student via contrastive learning. There exist a couple of methods to perform knowledge distillation without teacher models. For example, ONE [4] distills knowledge from ensemble of multiple students while BYOT [25] transfers knowledge from deeper layers to shallower ones. Besides, SSKD [26] distills self-supervised features of teachers students for transferring richer knowledge.
CRD 23
2.2 How to distill
Several recent knowledge distillation methods focus on the strategy of knowledge distillation. Born again network (BAN) [27] presents the effectiveness of sequential knowledge distillation via the networks with an identical architecture. A curriculum learning method (28) employs the optimization
trajectory of teacher model to train students. Collaborative learning approaches 4 5 6 attempt to
learn multiple models with distillation jointly, but their concept is not well-suited for asymmetric teacher-student relationship, which may lead to suboptimal convergence of student models.
The model capacity gap between a teacher and a student addressed in employs extra network to reduce model capacity gap between teacher and student models, where teacher transfers knowledge to a student via a teaching assistant network with an intermediate size. An carly stopping technique for training teacher networks is proposed to obtain better transferable representations and a neural architecture search is employed to identify a student model with the optimal size [2] Our work proposes a novel student-friendle learning technique of teacher network to facilitate knowledge distillation.
3 TAKD K
3 Student-Friendly Knowledge Distillation
This section describes the details of the student- frienndy teacher network (SFTN), which transfers the features of teacher models student networks more effectively than the standard distillation. Figure Suuure@illuee Dillustrates the main idea of our method.
3.1 Overview
The conventional knowledge distillation approaches attempt to find the way of teaching student networks given the architecture of teacher networks. The teacher network is trained with the loss with respect the ground-truth, but the objective is not necessarily beneficial for knowledge distillation to students. To the contrary, SFTN framework aims to improve the effectiveness of knowledge distillation from the teacher to the student models.
Modularizing teacher and student networks We modularize teacher and student networks into multiple blocks based on the depth of layers and the feature map sizes. This is because knowledge distillation is often performed at every 3 or 4 blocks for accurate extraction and transfer of knowledge in teacher models. Figure 2 presents the case that both networks are modularized into blocks, denoted by (B), BR, B and (B), B3, B3) for teacher and a student, respectively.
Adding student branches SFTN augments student branches to teacher model for the joint training of both parts. Each student branch is composed of teacher network feature transform layer T and student network blocks. Note that T is similar to a guided layer in FitNets [18] and transforms the dimensionality of the channel in Fi into that of B3+11 Depending on the configuration of teacher and student networks, the transformation need to increase or decrease the size of the feature maps. We employ 3 3x3 convolutions to reduce the size of FG while 4x4 transposed convolutions
3

---

Student Brunch
Teacher Network
Bi
Student Branch
7
Image
F
8
Inage
F7
D1
B3
Label
Student Network
Label
Tencher Network
(a) Student-aware training a teacher network
(b) Knowledge distillation
Figure 2: Overview of the student-friendly teacher network (SFTN), In this figure, F, B, T, and denote feature map, a network block, teacher network feature transform layer, and softmax output, respectively, where the superscript means the network block index and the subscript S, T, and R respectively indicate the student network, the teacher network, and the student branch in the teacher model. The loss for teacher network Cr is given by 4 while Kullback-Leible loss C and cross entropy loss f are defined (5) and (6), respectively. (a) When training a teacher, SFTN optimizes Ff and qr for better knowledge transfer student networks. (b) In the distillation stage, the features the teacher network, F1 and qr. are distilled to student networks with existing knowledge distillation algorithms straightforwardly.
used to increase its size. Also, 1x convolutions is used when we do not need to change the size of Ff. The features transformed to a student branch is forwarded separately to compute the logit of the branch. For example, shown in Figure 2(a). F1 in the teacher stream is transformed to fit B3. which initiates a student branch to derive q while another student branch starts from the transformed features of F.. Note that F has no trailing teacher network block in the figure and has no associated student branch because it is directly utilized to compute the logit of the main teacher network.
Training SFTN The teacher network is trained along with multiple student branches corresponding to individual blocks the teacher, where we minimize the differences in the representations between the teacher and the student branches. Our loss function is composed of three terms: loss in the teacher network LT. Kullback-Leibler loss LKL in the student branch, and cross-entropy loss L in the student branch. The main loss term, Lr. minimizes the error between qr and the ground-trutt while CKI. enforces q and qr to be similar to each other and D makes qi fit the ground-tre
Distillation using SFTN As shown in Figure b), the conventional knowledge distillation tech- nique is employed to simulate Fj and qr by F'S and qs respectively. The actual knowledge distillation step straightforward because the representations of Fý and qr have already been learned properly at the time of training SFTN. We expect the performance of the student network distilled from the SFTN be better than the one obtained from the conventional teacher network.
3.2 Network Architecture
SFTN consists of a teacher network and multiple student branches. The teacher and student networks divided into N blocks, where set of blocks in the teacher is given by Br (Bi while the blocks in the student is denoted by By {B² N Note that the last block in the teacher network does not have the associated student branch.
Given an input of the network, x, the output of the softmax function for the main teacher network, qr, is given by
Fr(x)
qx(x; = softmax
(I)

---

where Fr denotes the logit of the teacher network and 7 is the temperature of the softmax function. On the other hand, the output of the softmax function in the ith student branch, q given by
OR (Fi;7) softmax F3(T((F)))
(2)
T
where F3 denotes the logit of the 8t student branch.
3.3 Loss Functions
The teacher network in the conventional knowledge distillation framework is traned only with Cr- However, SFTN has additional loss terms such as LR and LCF as described in Section 3.1 The total loss function of SFTN, denoted by LSFTN, is given by
CSFIN = XpTr + CR + CE LR
(3)
where År, L and C are the weights of individual loss terms.
Each loss term is defined as follows. First, LT given by the cross-entropy between the teacher's prediction qr and the ground-truth label y as
LT CrossEntropy((nn (q
The knowledge distillation loss, denoted by Lk employs the KL divergence between q and qr- where N 1 student branches except for the last block in the teacher network are considered together as
N-1
CKL
Kkk (qr).
(5)
N
where q and qr denote smoother softmax function outputs with larger temperature, F. The cross-entropy loss the student network, C is obtained by the average cross-entropy loss from all the student branches, which is given by
N-1 Σ
CR =
CrossEntropy(ake y).
(6)
N
Note that we set r to for both the cross-entropy loss, Lr and CCB.
4 Experiments
We evaluate the performance of SFTN in comparison to existing methods and analyze the charac- teristics of SFTN in various aspects. We first describe our experiment setting Section 4.T Then, compare results between SFTNs and the standard teacher networks with respect classification accuracy in various knowledge distillation algorithms Section 4.2 The results from ablative experiments for SFTN and transfer learning discussed in the rest of this section.
4.1 Experiment Setting
We perform evaluation on multiple well-known datasets including ImageNet 30 and CIFAR-100
using several different backbone networks such as ResNet [32], WideResNet [33], VGG (34), Shuf- fleNetV1
335 and ShuffleNetV2 [36]. For comprehensive evaluation, adopt various knowledge distillation techniques, which include KD [I] FitNets [8]] AT 20). SP (37]. VID 38 RKD [22].
PKT 39
AB 40) FT 21. CRD 23, SSKD 26 and OH 19] Among these methods, the feature
distillation methods [8 20 37
38,
22 39
40
21 19 conduct joint distillation with conventional
KD [ during student training, which results higher accuracy in practice than the feature distilla- tion only. We also include comparisons with collaborative learning methods such as DML 4 and KDCL. [) and curriculum learning technique, RCO 28). We have reproduced the results from the existing methods using the implementations provided by the authors of the papers.
5

---

Table 1: Comparisons between SFTN and the standard teacher models CIFAR-100 dataset when the architectures of the teacher-student pairs are homogeneous. In all the tested algorithms. the students distilled from the teacher models given by SFTN outperform the ones trained from the standard teacher models. All the reported results are based on the outputs of 3 independent runs.
Teacher/Student
WRN40-2/WRNI622
WRN40-2/W6 1
resnet 2x4ressseee
resnet32xdresnee8s2
Teacher training
Stan.
SFTN
Stan.
SFTN
Stun.
SFTN
A
Stun.
SFTN
A
Teacher Acc.
76.30
78.20
76.30
77.02
79.25
79,41
79.25
77.89
Student Acc. w/o KD
73.41
72.16
72.38
68.19
75,46
76.25
+0.79
73.73
75.09
+1.36
73.39
76.09
+-2.70
43
69.17
+1.74
FitNets 18
75.72
76.73
+1.01
74.14
75.54
+1.40
75.34
76.89
+1.55
69.80
71.07
+1.27
AT 20
75.85
76.82
+0.97
74.56
75.86
+1.30
74.98
76.91
+1.93
68.79 79
70.90
+2.11
SP
75.43
74.51
+0.80
76.37
+2.31
68.39
70.03
VID 58
75.63
76.79
+1.16
74.21
75.76
+1.55
74.86
77.00
+-2.14
69.53
71.08
+1.55
RKD
75.48
76.49
+1.01
73.86
75.11
+1.25
74.12
76.62
+2.50
68.54
70.91
+2,36
PKT 39
75.71
76.57
+0.66
74.43
75.49
+1.06
76:57
+1.87
69.29
70.75
AB
(40
70.12
70.76
+0.64
74.38
75.51
+1.13
74.73
76.51
+-1.78
69.76
71.05
+1.29
FT 21
75.6
76.51
-0.11
74.49
75.11
+-0.62
74.89
77.02
+-2.13
69.70
71.11
+1.40
CRD 25
75.91
77.23
+1.32
74.93
76.09
+1.16
75.54
76.95
+1.41
70.34
71.34
SSKD 26
75.96
76.80
+0.84
75.72
76.03
+0.31
75.95
76.85
+0.90
69.34
70.29
+0.96
OH 19
76.00
76.39
0.39
74.79
75.62
+-0.3
75.04
76.65
+1.61
68.10
69.69
+1.59
Best
76.00
77.23
1.25
75.72
76.09
+0.37
75.95
77.02
+1.07
70.34
71.34
+1.00
Table 2: Comparisons between SFTN and the standard teacher models CIFAR-10 100 dataset when the architectures of the teacher-student pairs are heterogeneous. In all the tested algorithms, the student models distilled from the teacher models given by SFTN outperform the ones trained from the standard teacher models. All the reported results are based on the outputs of 3 independent runs.
Teacher/Student
resnet32xd/Shuflle2 i
ResNar50/VGG8
REN40-2/ShuutleV
Teacher training
Stan.
SFTN
Stan.
SFTN
A
Stan.
SFTN
A
Stan.
SPTN
A
Teacher Acc.
79.25
80.03
79.25
79.58
78.70
82.52
76.30
78.21
Student Acc. w/o KD
71.95
73.21
71.12
73.21
KD [
74.26
77.93
+3.67
75.25
78.07
+2.82
73.82
74.92
+1.10
76.68
78.06
+1.38
PitNets 18
75.95
78.75
-.880
77.00
79.68
+.2.68
73.22
74.80
+11.58
77.31
79.21
+1.90
AT 20
76.12
78.63
+2.51
76.57
78.79
+2.22
73.56
74.05
+-0.49
77
78.29
+0.88
SP
75.80
2..56
76.11
78.38
+-2.7
74.02
75.37
+11.35
76.93
78.12
+1.19
VID 38
75.16
78.03
+2.87
75.70
78.49
+2.79
73.59
74.76
+1.17
77.27
78.78
+1.51
RKD
74.84
77.72
+2.88
75.48
77.77
+-229
73.54
74.70
+1.16
76.69
78.11
+1.42
PKT 39
75.05
77.46
+2.41
75.79
78.28
+-2.49
75.17
+1.38
78.28
AB 40
75.95
78.53
+2.58
76.25
78.68
+2.43
73.72
74.77
+1.05
77.28
78.77
+1.49
FT 21
75.58
77.84
+2.26
76.42
78.37
+1.95
73.34
74.77
+1.43
76.80
77.65
+0.85
CRD 25
75.60
78.43
+2208
75.41
+0.89
78.81
SSKD 26
78.05
79.10
+1.05
78.66
79.65
+-0.99
76.03
76.95
+0092
77.81
78.34
+0.53
OH [19
77.51
79.56
+2.05
78.08
79.98
+1.90
74.55
75.95
+1.40
77.82
79.14
+1.32
78.05
79.98
76.95
+0.92
77.82
4.2 Main Results
To show effectiveness of SFTN, we incorporate SFTN into various existing knowledge distillation algorithms and evaluate accuracy. We present implementation details and experimental results on CIFAR-100 100 [1 and ImageNet 33 datasets.
4.2.1 CIFAR-100
CIFAR-100 31 consists of 50K training images and 10K testing images in 100 classes. We select 12 state-of-the the art distillation methods to compare accuracy of SFTNs with the standard teacher networks. To show the generality of the proposed approach, pairs of teacher and student models have been tested in our experiment. The experiment setup for CIFAR- is identical to the one performed in CRD most experiments employ the SGD optimizer with learning rate 0.05, weight decay 0.0005 and momentum 0.9 while learning rate is set to 0.01 in the ShuffleNet experiments. The hyperparameters for the loss function are set as AT = 1, A student-aware training while = 4 in knowledge distillation.
XKL 3, and F
Table ! and 2 demonstrate the full results on the CIFAR-100 dataset. Table I presents the distillation performance of all the compared algorithms when teacher and student pairs have the same architecture type while Table2]show the results from teacher-student pairs with heterogeneous architecture styles. Both tables clearly demonstrate that SFTN is consistently better than the standard teacher network

---

82
Student
Beet Student of Standard
Best Student of SFTN
Standard Teacher
78
76
72
70
68
66
Teacher: WFRN40
WRRN40 2
Tesnel3234
resnet52xd
resnet3224
resnetS2x4 ShuttleV2
ResNe:50 VISO8
VMRN40 Sihuttee22
Student: WKNN16-2
WHN4D-11
resnatiik2
Figure 3: Accuracy comparison of the best students from SFTN with standard teacher on CIFAR- 100. The four best student models of SFTN (blue) outperform the standard teachers (gray) while the only one best student of the standard teacher (red) achieves higher aceuracy than its teacher (gray).
Table 3: Top-1 and Top-5 validation accuracy on ImageNet of SFTN in comparison to other methods.
Teacher/Student
ResNet50/Resaar3
Top-1
Top-5 SFTN
Teacher training
Standard
SFTN
Standard
Teacher Acc.
76.45
77.43
93.15
93.75
Student Acc. w/o KD
73.79
91.74
KD
73.55
74.14
+0.59
91.81
92.21
+0.40
FirNets [8
74.56
75.01
+11.45
92.31
92.51
+0.20
SP 377
74.95
75.53
+0.58
92.54
92.69
+0.15
CRD 23
75.01
75.39
+-0.88
92.56
92.67
+0.11
OH [
74.56
75.01
+00.45
92.36
92.56
+0.20
75.01
75.33
+0.52
92.56
92.69
+0.13
all algorithms. The average difference between SFTN and the standard teacher is 1.58% points. and the average difference between best student accuracy of SFTN and the standard teacher is 1. 1.10% points. We note that the outstanding performance of SFTN not only driven by the higher accuracy of teacher models achieved by our student-aware learning technique. As observed in Table [ and proposed approach often presents substantial improvement compared to the standard distillation methods despite similar lower teacher accuracies. Refer to Section 4.4 for the further discussion about the relation of accuracy between teacher and student networks.
Figure B iillustrates the accuracies of the best student models of the standard teacher and SFTN given teacher and student architecture pairs. Despite the small capacity of the students, the best student models of SFTN sometimes outperform the standard teachers while the only one best student of the standard teacher shows higher accuracy than its teacher.
4.2.2 ImageNet
ImageNet 30 consists of 1.2M training images and 50K validation images for 1K classes. We adopt the standard Pytorch set-up for ImageNet training for this experiment" The optimization given by SGD with learning rate 0. weight decay 0.0001 and momentum 0.9.
The coefficients of individual loss terms We conduct the ImageNet experiment for 5 different knowledge distillation methods, where teacher models based ResNet50 transfer knowledge to student networks with ResNet34.
ÀT 1, A 1, and X 1, where 7 1.
As presented in Table 3 SFTN consistently outperforms the standard teacher network in all settings. The best student accuracy of SFTN achieves the higher top-1 I accuracy than the standard teacher model by approximately 0.5% points. This results implies that the proposed algorithm has great potential on large datasets as well.
4.3 Comparison with Collaborative and Curriculum Learning Methods
Contrary to traditional knowledge distillation methods based on statie pretrained teachers, collab- orative learning approaches employ dynamic teacher networks trained jointly with students and curriculum learning methods keep track the optimization history teachers for distillation. Table
shows that SFTN outperforms the collaborative learning techniques such as DML [4] and KDCL [5]

---

Table 4: Comparision with collaborative and curriculum learning approaches CIFAR-100 100. We employ KDCL-Naive for ensemble logits of KDCL. RCO is based on the one-stage EEI (equal epoch interval) while RCO-EEI-4 adopts 4 anchor points selection with the EEI strategy. Note that both RCO-EEI-4 and SFTN-4 are trained for 240 x 4 epochs.
Teacher
WRN40-2 WRN16-2
WRN40-2
resnet32x4
resnet32x4 ShuffleV1
res
ResNet50 VGG8
Student
WRN40-1 1
resmet8x4
Shuflle
Standard teacher Acc.
76.30
76.30
79,25
79.25
79.25
78, 70
Student Acc. w/o KD
73.41
72.16
72.38
71.95
73.21
71.12
Stu.
Stu.
Stu.
A
Stu.
Stu.
A
Stu.
Standard
75.46
-2.05
13:73
+1.57
73.39
+1.01
74.26
+2.31
73.25
+2.04
73.82
+2.70
DML 4
75.30
-1.89
74.08
+1.92
74.34
+-196
73.37
+1.42
73.80
+ 0.59
73.01
+1.89
KDCL 5
75.45
+2.04
74.65
+2.49
75.21
+2.83
73.98
+ 2.03
74.30
+1.09
73.48
+2.36
RCO 28
75.36
+1.95
74.29
+2.13
74.06
+1.68
76.62
+4.67
77.40
+4419
74,30
+3.18
SFTN
76.25
2.884
75.09
+2293
76.09
+3.71
77.93
78.07
74.92
+3.80
RCO-EEL4 28
75.69
2.28
74.87
+2.71
73.73
+1.35
76.97
+5.02
76.89
+3.68
74.24
+3.12
SFTN-4
76.96
+3.55
76.31
+4.15
76.67
+4.29
79.11
+7.16
78.95
+5.74
75.52
+4.40
Table 5: Effect of varying F in the KL-divergence loss of the student-aware training tested CIFAR- 100, where for the knowledge distillation is set to 4. The student accuracy is fairly stable over a wide range of the hyperparameter. Note that the accuracies of SFTNs and the student model are rather inversely correlated, which implies that the maximization of teacher models not necessarily ideal for knowledge distillation.
Accuracy SFTN
accuracy
Teacher Student
2es
WRN40-2
WRN40-2
ShuffleV1
Shuffle22
WRN16-2
WRN40-1
Avg.
ShuffleV1
ShufileV2
WRN16-2
WRN40
Avg.
==1
81.19
80.26
78.23
78.14
78.85
76.05
77.18
76.30
74.75
75.58
f= 5
81.23
81.56
79.22
78.31
79.54
75.36
75.59
76,31
73.64
75.10
10
81.27
78.81
78.38
79.58
74.47
75.93
75.85
73.62
74.76
f 15
81.89
81.74
79.27
78.63
79.74
74.78
75.65
75.79
73.49
74.81
F 20
81.60
81.70
78.84
78.45
79.59
74.62
75.88
75.82
74.03
74.95
the heterogeneous architectures turn out to be effective for mutual learning. On the other hand, the accuracy of SFTN is consistently higher than that of the curriculum learning method, RCO [28], under the same and even harsher training condition in terms of the number of epochs. Although the identification the optimal checkpoints may be challenging in the trajectory-based learning. SFTN improves its accuracy substantially with more iterations shown in the results for SFTN-4.
4.4 Effect of Hyperparameters
SFTN computes the KL-divergence loss, CKL, to minimize the difference between the softmax outputs of teacher and student branches, which involves two hyperparameters, temperature of the softmax function, т, and weight for KL-divergence loss term, A We discuss the impact and trade-off issue of the two hyperparameters. In particular, we present our observations that the student-aware learning is indeed helpful to improve the accuracy of student models while maximizing performance of teacher models may be suboptimal for knowledge distillation.
Temperature softmax function The temperature parameter of the KL-divergence loss in (5) denoted by F, controls the softness of द and q as F gets higher, the output of the softmax function becomes smoother. Despite the fluctuation in teacher accuracy, student models given by knowledge distillation via SFTN maintain fairly consistent results. Table alsso shows that the performance of SFTNs and the student models is rather inversely correlated. This result implies that a loosely optimized teacher model may be more effective for knowledge distillation according this ablation study.
Weight for KL-divergence loss The hyperparameter KL facilitates knowledge distillation by making dr similar to d However, it affects the accuracy of teacher network negatively. Table 6 shows that the average accuracy gaps between SFTNs and the corresponding student models drop gradually as A increases. One interesting observation the student accuracy via SFTN with JKL = 10 in comparison to its counterpart via the standard teacher; even though the standard teacher network is more accurate than SFTN by large margin, corresponding student accuracy lower than that of SFTN.
AR
8

---

Table 6: Effect of varying A in the knowledge distillation via SFTN tested on CIFAR-I00, The accuracies of SFTNs and the corresponding students are not correlated while the accuracy gaps of the Iwo models drop as AR increases.
Accuracy of SFTN
Student accuracy by KD
Teacher Student
ShufteVI
resnet32x4
ShuffleV2
WRN16-2
WRN40-2
WRN40-1
Avg.
ShuffeeV
resnet32x4
Shuffee22
WRN16-2
WRN40-2
WRN40-1
Avg.
81.19
80.26
78.23
78.14
79.46
76.05
77.18
76.30
74.75
76.07
Ag
78.70
79.80
77.83
77.57
78.48
77.36
78.56
76.20
74.71
76.71
78.29
78.29
77.28
76.05
77.48
77.33
77.70
76.02
74.67
76.43
10
73.02
75.01
75.03
73.51
74.14
75.57
76.62
74.19
73.08
74.87
Standard
79.25
79.25
76.30
76.30
77.78
74.31
75.25
73.28
73.56
74.60
Table 7: Effectiveness of knowledge distillation via SFTN when student models have different capacity compared to the used in the student-aware training. This experiment is conducted on CIFAR- 100. The numbers in bold and red denote the best and the second-best results. SB denotes student branch. (MI: resmet8x2, M2: resnet8x4, M3: resnet32x4, M4: WRN16-1, M5:WRNI6-2,
M6: WRN40-2, M7: ShuffleV2)
Teacher/Student
WRN4O-2/VRN 16-2 (M6/MS)
Nesee (M3/M))
SB cupacity
N/A
Smaller
Equal/similar
Larger
NIA
Smaller
Equal/similar
Larger
SB model
SB Acc.
68.19
67.10
73.41
73.21
79.25
76.30
67.10
68.19
72.38
73.21
79.25
76.30
Teacher
76.30
76.22
75.98
78.20
78.21
78.82
78.69
79.25
76.53
77.89
79.41
79.58
80.85
80.30
KD
75.46
74.67
74.73
76.25
75.68
75.56
75.63
73.39
74.71
75.19
76.09
75.82
75.19
75.03
SP
37
75.43
74.75
75.29
76.77
76.56
76.13
76.08
74.06
75.31
75.76
76.37
76.09
75.62
75.36
FT 21
75.60
74.61
75.23
76.51
76.66
76.47
76.23
74.89
75.79
76.54
77.02
76.62
76.48
76.63
CRD
23
75.91
76.14
76.07
77.23
77.45
77.06
76,70
75.54
76.38
76.72
76,95
76.64
76.54
76.46
SSKD
75.96
74.35
74.41
76.80
76.73
76.22
76.67
76.24
OH
[9
76.00
74.95
74.97
76.39
76.49
76.27
76.15
75.04
75.65
75.69
76.65
76.48
76.38
76.44
Average
75.73
74.91
75.12
76.66
76.56
76.37
76.25
74.81
75.48
75.95
76.66
76.31
76.15
76.03
Best
76.00
76.14
76.07
77.23
77.45
77.06
76.72
75.95
77.02
76.64
76.67
76.63
4.5
Versatility of SFTN
Although our teacher network obtained from the student-aware training procedure is specialized for a specific student model, it also effective to transfer knowledge to the students models with substantially different architectures. Table shows that the benefit of method is also preserved well as long as the student branch has similar capacity to the student models, where the model capacity defined by the achievable accuracy via independent training without distillation. In addition, presents that larger students branches are often effective to enhance distillation performance while smaller student branches are not always helpful. In summary. these results imply that a teacher network in SFTN trained for a specific architecture of student network has the potential to transfer its knowledge other types of student networks.
4.6 Use of Pretrained Teachers
The main goal of knowledge distillation is maximize the benefit in student networks, and the additional training cost may not be critical in many real applications. However, the increase of training cost originated from the student branch of the teacher network is still undesirable. We can sidestep this limitation by adopting pretrained teacher networks in the student-aware training stage. The training cost of SFTN teacher networks is reduced significantly by using pretrained models, and T presents the tendency clearly. Compared to 240 epochs for the standard student-aware training, fine-tuning pretrained teacher networks only needs 60 epochs for convergence; we train the student branches only for the first 30 epochs and fine-tune the whole network for the remaining 30 epochs. Table / shows that fine-tuned pretrained teacher networks have potential to enhance distillation performance. They achieve almost same accuracy with the full SFTN in 6 knowledge distillation algorithms.
4.7 Similarity between Teacher and Student Representations
The similarity between teacher and student models is an important measure for knowledge distillation performance in the sense that student network aims to resemble the output representations of

---

Table 8: Training time of SFTN teachers with pretrained models. The additional training time in each SFTN with pretrained teacher model presented the last column; is significantly reduced compared to the standard SFTN teacher (the second-last column).
Training time (s2c)
Models (leacher/student)
Teacher
Student
SFTN teacher SFTN teacher with pretrained model
6,005
5,624
10,910
2,298
resnet32x4/Shuttlee2
6,005
6,221
10.949
2,370
WRRNA-2/WRNING-
3,940
1,745
6,028
1,178
WRNNO-2/WRN40-1
3,940
3,698
7,431
1,621
Table 9: Performance of SFTN fine-tuned from a pretrained teacher (SFTN-FT) on CIFAR-100.
Table 10: Similarity measurements between
teachers and students on CIFAR-100.
Teacher/Student
resnet32xd/ShufteV2
Teacher training method
Standard
SFTN
SFTN+FT
Models (tcacher/studeet)
Teacher
79.25
80.03
80.41
Simularity metric
KL-divergence
CKA
Student Acc. wo KD
71.95
Teacher training method
Standard
SPTN
Standard
SPTN
KD
75.25
78.07
78.12
KD
D
1.10
0.47
0.88
0.95
SP
37
76.11
78.38
78.51
FaiNets
FT A]
76.42
78.37
77.90
SP 17
0.95
0.45
0.89
0.95
CRD 23
76.35
78.43
78.88
SSKD
26
78.66
79.65
79.15
CRD 23
0.81
0.43
0.88
0.95
OH
[9]
78.08
79.98
79.68
SSKD P6
0.54
0.26
0.92
Averuge Best
76.81
78.81
78.71
OH 19
0.85
0.37
0.90
0.96
78.66
79.98
79.68
AVG
0.84
0.89
teacher network. We employ KL-divergence and CKA 41 as similarity metrics, where lower KL-divereence and higher CKA indicate higher similarity.
Table TO presents the similarities between the representations ResNet32x4 and S2, respectively, which given by various algorithms the CIFAR-100 test set. The results show that the distillations from SFTNS always give higher similarity to the student models with respect to the corresponding teacher networks; SFTN reduces the KL-divergence by more than 50% in average while improving the average CKA by 7% points compared to the standard teacher network. The improved similarity of SFTN is natural since is trained with student branches to obtain student-friendly representations via the KL-divergence loss.
teacher and a student based on
5 Conclusion
We proposed simple but effective knowledge distillation approach by introducing the novel student- friendly teacher network (SFTN). Our strategy sheds light in new direction to knowledge distilla- tion by focusing on the stage to train teacher networks. We train teacher networks along with their student branches, and then perform distillation from teachers to students. The proposed strategy turns out achieve outstanding performance, and can be incorporated into various knowledge distillation algorithms easily. For the demonstration of the effectiveness of our strategy, we conducted compre- hensive experiments in diverse environments, which show consistent performance gains compared to the standard teacher networks regardless of architectural and algorithmic variations.
The proposed approach is effective for achieving higher accuracy with reduced model sizes, but it is not sufficiently verified in the unexpected situations with out-of ontoo-ddistriuttinn inputs, domain shifts, lack of training examples, etc. Also, still rely on large number of training examples and still has the limitation of high computational cost and potential privacy issue.
6 Acknowledgments and Disclosure of Funding
This work was partly supported by Samsung Electronics Co., Ltd. (I0200626-07474-000 and Institute for Information & Communications Technology Promotion (IITP) grant funded by the Korea government (MSIT) [2017-0-01779; XAI, 2021-0-01343 Artificial Intelligence Graduate School Program (Seoul National University)).
References
[1] Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the Knowledge in a Neural Network. In NeurIPS Deep Learning and Representation Learning Workshop. 2015.
10

---

[2] Minsoo Kang, Jonghwan Mun, and Bohyung Han. Towards oracle knowledge distillation with neural architecture search. In AAAI, 2020.
[3] Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant: Bridging the gap between student and teacher. In AAAI, 2020.
[4] Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep Mutual Learning. In CVPR, 2018.
[5] Qiushan Guo, Xinjiang Wang, Yichao Wu, Zhipeng Yu, Ding Liang, Xiaolin Hu, and Ping Luo. Online knowledge distillation via collaborative learning. In CVPR, 2020.
[6] Guile Wu and Shaogang Gong. Peer collaborative learning for online knowledge distillation. In arXiv 2006.04147, 2020.
[7] Guobin Chen, Wongun Choi, Xiangdong Chen, Tony X. Han, and Manmohan Krishna Chandraker. Learning Efficient Object Delection Models with Knowledge Distillation. In NeurIPS, 2017.
[8] Ping Luo, Zhenyao Zhu, Ziwei Liu, Xiaogang Wang, and Xiaoou Tang. Face Model Compression by Distilling Knowledge from Neurons. In AAAI, 2016.
[9] Ancong Wu, Wei-Shi Zheng. Xiaowei Guo, and Jian-Huang Lai. Distilled person re-identification: Towards more scalable system. In CVPR, 2019.
[10] Huan Wang. Yijun Li, Yuehai Wang, Haoji Hu, and Ming-Hsuan Yang. Collaborative distillation for ultra-resolution universal style transfer. In CVPR, 2020.
[11] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling BERT for natural language understanding. In 19X9vv1909.1033. 2019.
[12] Jonghwan Mun, Kimin Lee, Jinwoo Shin, and Bohyung Han. Learning to Specialize with Knowledge Distillation for Visual Question Answering. In NeurIPS, 2018.
[13] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, 2019.
[14] Siddhartha Arora, Mitesh M. Khapra, and Harish G. Ramaswamy. On knowledge distillation from complex networks for response prediction. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, NAACL, 2019.
[15] Fida Mohammad Thoker and Juergen Gall. Cross-modal knowledge distillation for action recognition. In ICIP, 2019.
[16] Mingmin Zhao, Tianhong Li, Mohammad Abu Alsheikh, Yonglong Tian, Hang Zhao, Antonio Torralba, and Dina Katabi. Through-wall human pose estimation using radio signals. In CVPR, 2018.
[17] Brady Zhou, Nimit Kalra, and Philipp Krähhenühh. Domain adaptation through task distillation. In ECCV, 2020.
[18] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. FitNets: Hints for Thin Deep Nets. In ICLR, 2015.
[19] Bycongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi, comprehen- sive overhaul of feature distillation. In ICCV, 2019.
[20] Sergey Zagoruyko Nikos Komodakis. Paying More Attention to Attention: Improving Performance of Convolutional Neural Networks via Attention Transfer. In ICLR, 2017.
[21] Jangho Kim, Seong Uk Park, and Nojun Kwak. Paraphrasing Complex Network: Network Compression via Factor Transfer. In NeurlPS, 2018
[22] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational Knowledge Distillation. CVPR, 2019.
[23] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR. 2020.
[24] Xu Lan, Xiatian Zhu, and Shaogang Gong. Knowledge Distillation by On-the-Fly Native Ensemble. In NeurlPS, 2018.
[25] Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher; Improve the performance of convolutional neural networks via self distillation. In ICCV, 2019.
11

---

[26] Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. Knowledge distillation meets self-supervision. In ECCV, 2020.
[27] Tommaso Furlanello, Zachary Chase Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born-again neural networks. In ICML, 2018.
[28] Xiao Jin, Baoyun Peng, Yichao Wu, Yu Liu, Jiaheng Liu, Ding Liang, Junjie Yan, and Xiaolin Hu. Knowledge distillation via route constrained optimization. In ICCV, 2019.
[29] Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In ICCV, 2019.
[30] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis., 155():211-2522 2015.
[31] Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images, Technical report, Citescer, 2009.
[32] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CVPR, 2016.
[33] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
[34] Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. In ICLR, 2015.
[35] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In CVPR, 2018.
[36] Mingxing Tan, Bo Chen, Ruoming Pang. Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe V. Lc. Mnasnet: Platform- aware neural architecture scarch for mobile. In CVPR, 2019.
[37] Frederick Tung and Greg Mori. Similarity- preserving knowledge distillation. In ICCV, 2019.
[38] Sungsoo Ahn, Shell Xu Hu, Andreas C. Damianou, Neil D. Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In CVPR, 2019.
[39] Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In ECCV, 2018
[40] Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. In AAAI, 2019.
[41] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey E. Hinton. Similarity of neural network representations revisited. In ICML, 2019.
12


Reviews:
[{"review_id": "SrdMvCqRBq9", "reviewer": "Reviewer_9tKz", "summary": "This paper proposes a student friendly teacher network (SFTN) strategy for knowledge distillation. In experiments, the proposed method boost performance for various current knowledge distillation algorithms on CIFAR-100 and ImageNet. \n", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The authors propose a student friendly teacher network (SFTN) for knowledge distillation. The proposed method is orthogonal to the current KD methods. \n2. The experimental results are strong. SFTN demonstrates the performance improvements for various KD methods on CIFAR-100 and ImageNet.\n3. The writing and presentation are clear and well-organized. \n\nWeakness:\n1. The propose SFTN is a little inefficient. According to the Sec. 3.1, for each student model, SFTN needs to train a specialized teacher model. Although Sec. 4.6 and Table 8 give some analyses on CIFAR-100, I still think the proposed SFTN may not be flexible and efficient, especially for the large-scale training data (i.e. ImageNet) and the huge teacher models (i.e. ResNet-101/ResNeXt-101). It is necessary to discuss the training time cost for SFTN. \n2. According to the experiments, SFTN got significant performance improvements (1~2%) on CIFAR-100, but marginal improvements (0.5%) on ImageNet. Can the authors give some theoretical or empirical explanations for this phenomenon? Or it means that, with the increasing number of training data, SFTN may not be effect to keep the consistency between teacher and student features. \n3. The experiments on ImageNet are not sufficient. Considering the network structures and the capacity, ResNet-50 is similar to ResNet-34. The authors should give more experiments on the large gap of teacher-student models for homogeneous architecture (ResNet101->ResNet18) and heterogeneous architecture (ResNet101->VGG-16) to support the authors' claim (L24-26).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See weakness. I am pleased to improve the final rating if the authors can address all my concerns. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "SIQG_aVvjK", "reviewer": "Reviewer_6tca", "summary": "This work proposes Student-Friendly Teacher Network (SFTN), an approach that aims to learn teacher models that are friendly to student models, thereby facilitating the subsequent knowledge distillation process resulting in superior student models (studied image classification problems).\n\n", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The work is more of an engineering work supported by large scale empirical study. The ideas and results are not particularly novel nor exciting. Considering this as an engineering work, the practicality/ usability of the proposed technique may be limited due to the large computational overhead (i.e.: using student branches in SFTN training).\n\nPaper Strengths:\n\nThe paper has a clear definition of the problem, proposed methodology and extensive empirical studies.\n\nPaper Weaknesses:\n\nIs $L_{R}^{CE}$ term required during SFTN training? Given that the objective of SFTN training is to make the teacher and student representations similar to each other (this is what is meant by friendly), do the authors think that the use of $L_T$ and $L_{R}^{KL}$ is sufficient? \n\nFor feature-based KD, how did the authors decide on the design for modularizing the teacher and student networks since there are multiple design choices and this will also vary for every pair of (Teacher, Student) setups. \n\nKD is done on the training set. So for Table 9, the similarities between the representations must be measured using the training set.\n\nDid the authors observe identical behaviour with respect to similarities between representations (Table 9) in ImageNet experiments (I don’t see these results). I emphasize this point as this work focuses on representation learning and ImageNet is more representative of challenging learning problems.\n\nGiven that authors propose a KD framework, during distillation (training of student) do authors learn only from the teacher (soft-targets) or from both the ground truth (hard-targets) and the teacher (soft-targets)? I think this is an important detail to clarify as I expect that the authors only use the soft-targets for distillation.\n\nWhy should the student branches learn multiple representations ( $q_{R}^{1}, q_{R}^{2} )$? How does the student's accuracy change if only $q_{R}^{1}$ is used?\n\nGiving more context into the term \"friendly representations\" could be better. I.e.: In these setups can we say that KL divergence between the teacher and student representations can be used to measure this degree of friendliness? \n\n\nClarity:\n\nThe paper is ok. One suggestion is to move some hyper-parameter details to Supplementary so that it can be less distracting to read (i.e.: learning rates, optimizer details etc).\n\n\nRelation to prior work:\n\nYes, the contribution of this work is discussed\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, limitations are briefly discussed in Sec 5.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "DS6KWpFQUiL", "reviewer": "Reviewer_rkNh", "summary": "This paper proposes a simple method to facilitate knowledge transfer from a teacher to a student. Unlike traditional knowledge distillation methodology that aims to effectively train student networks given pre-trained teachers, the goal of the work is training teachers which are friendly to knowledge transfer to students. The authors propose a framework in which the target student network is incorporated into teacher networks through a set of student branches. The teacher network and associated student branches are trained jointly before distillation. Extensive experiments on CIFAR and ImageNet datasets are conducted for evaluation. ", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well presented and organized. The motivation of the work is interesting. Unlike effectively training students given pretrained teachers, the authors take a different perspective that expects to learn a student-friendly teacher which facilitates knowledge transfer to the target student. Extensive experiments and ablation studies, including comparison to collaborative learning methods, on CIFAR and ImageNet datasets are conducted for evaluation. \n\nHowever, the proposed solution is less convincing as the target student network architecture needs to be given ahead, which may reduce the generalization of knowledge distillation. Given a pretrained large model, the distillation typically performs without any assumption or restriction on target network architectures and can work on any target networks in practice. It may happen that the proposed method can only work well on given pairs of teachers and students but its benefit may degrade when the student networks for training SFTN and distillation are dramatically different. The authors need to analyze and provide empirical studies for the problem. \n\nThe technical novelty of the work is incremental as the network design and loss functions are based on existing methods. The design of adding multiple (typically shallow) branches into deep networks is highly related to deeply supervised learning [1] which introduces extra branches to facilitate the training of deep networks. The relation to the work needs to be discussed although the motivation of the two works are different.\n\nSome questions about the experimental section:\n1)\tI think resnet32x4 and resnet8x4 denotes the network configuration of ResNext rather than ResNet. Otherwise, the definition of the two networks needs to be clarified in the paper. \n2)\tThe explanation of experimental results are insufficient. For example, lack of analyzing the phenomenon that the teacher performance of SFTN in the setting of WRN40-2/WRN16-2 is much better than WRN40-2/WRN40-1. I think the reason is that using shallow branches may benefit the training of deep networks. The authors need to improve the analysis in the experimental section.\n\n[1] Chen-Yu Lee et al., Deeply-Supervised Nets. In AISTATS 2015.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have discussed some limitations of the work in the conclusion section.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "3xskuNSC5F3", "reviewer": "Reviewer_Bfa6", "summary": "This paper proposes SFTN to learn teacher models that are better at transferring their knowledge to the student models (i.e., student-friendly). The proposed method makes the teacher model aware of the student models during their training process by jointly learning the student branches. The proposed method is versatile to different kinds of KD algorithms. Experiments on image classification benchmarks show that SFTN successfully improves the performance of different KD methods. The authors also conduct ablation studies to better understand the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Pros\n\n- The paper is well written and the problem of the student-teacher capacity gap is an important problem and the idea of learning student-friendly teacher models is well motivated. \n\n- Experiments are well conducted and the results seem to be promising. SFTN consistently improves various KD methods in many different settings.\n\n## Cons\n\n- The novelty of the proposed paper is limited because the line of online knowledge distillation research is conceptually very similar to the SFTN method. They learn teacher and student models together and the objective function is very similar to the case of SFTN, except that they do not add the second stage KD because they focus on one-stage KD, and they do not introduce branches of student models. A baseline that trains the student model jointly with the teacher model, but separates the students and teachers (independent students rather than student branches) may be helpful.\n\n- The performance of the teacher model trained by SFTN is higher than a conventional teacher in many cases, therefore it is hard to tell if the SFTN model is student-friendly or simply stronger and generalizes better. \n\n- The computational overhead of training the teacher model by SFTN should be mentioned. Since most KD method assumes a pre-trained teacher model, the fact that SFTN requires re-train the teacher model in a more sophisticated way is obviously a drawback. What if we train a new powerful teacher with the same computational budget compared to that of SFTN teacher training?\n\n-The choice of student branches is non-trivial. Also, how are the student branches possible if the student and teacher have completely different architectures?\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limited discussion about very related work on online KD literature and limited discussion about the computational cost for training the teacher model.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers are in broad agreement, finding the paper below the bar for NeurIPS. Reviewers found the problem itself interesting, and the experiments reasonably thorough. Criticisms centered around issues of novelty, efficiency, and additional components that should be added to the experiments to make them more convincing. None of these individually seemed like deal-breakers, but collectively put the paper somewhat below the threshold.
Decision: Accept (Poster)