Knowledge Transfer via Student-Teacher Collaboration
Authors: Tianxiao Gao
Ruiqin Xiong
Zhenhua Liu
Siwei ma
Feng Wu
Tiejun Huang
Wen Gao
Venue:  (2020)
Keywords: Network Compression and Acceleration
Knowledge Transfer
Student-Teacher Collaboration
Deep Learning.

Abstract: Accompanying with the flourish development in various fields, deep neural networks, however, are still facing with the plight of high computational costs and storage. One way to compress these heavy models is knowledge transfer (KT), in which a light student network is trained through absorbing the knowledge from a powerful teacher network. In this paper, we propose a novel knowledge transfer method which employs a Student-Teacher Collaboration (STC) network during the knowledge transfer process. This is done by connecting the front part of the student network to the back part of the teacher network as the STC network. The back part of the teacher network takes the intermediate representation from the front part of the student network as input to make the prediction. The difference between the prediction from the collaboration network and the output tensor from the teacher network is taken into account of the loss during the train process. Through back propagation, the teacher network provides guidance to the student network in a gradient signal manner. In this way, our method takes advantage of the knowledge from the entire teacher network, who instructs the student network in learning process. Through plentiful experiments, it is proved that our STC method outperforms other KT methods with conventional strategy.

Under review as a conference paper at ICLR 2020
KNOWLEDGE TRANSFER VIA STUDENT-TEACHER
COLLABORATION
Anonymous authors Paper under double-blind review
ABSTRACT
Accompanying with the flourish development in various fields, deep neural net- works, however, are still facing with the plight of high computational costs and storage. One way to compress these heavy models is knowledge transfer (KT)), in which light student network is trained through absorbing the knowledge from a powerful teacher network. In this paper, propose novel knowledge transfer Student-Teacher the knowledge transfer process. This is done by connecting the front part of the student network to the back part of the teacher network as the STC network. The back part of the teacher network takes the intermediate representation from the front part of the student network as input to make the prediction. The difference between the prediction from the collaboration network and the output tensor from the teacher network is taken into account of the loss during the train process. Through back propagation, the teacher network provides guidance the student network in a gradient signal manner. In this way, our method takes advantage the knowledge from the entire teacher network, who instructs the student net- work in learning process. Through plentiful experiments, it is proved that our STC method outperforms other KT methods with conventional strategy.
method which employs a Student- Teacher Collaboration (STC) network during
INTRODUCTION
Deep neural networks have produced breakthrough results in various fields, such as computer vi- sion (Krizhessky et al. 2012 He et al. (016) and natural language processing (Mikolov et al. 2010 in recent years. Through a mass of studies (Neyshabur al.] [0177] [anziani et al. 2016 Novak et al. 2018), researchers have proved that DNN with larger capacity will have a better gen- eralizability, which leads to a better performance. However, larger capacity will also cause heavier computational costs and storage, making these powerful models difficult to meet real-time require- ments on embedded systems.
One way to compress these heavy models is knowledge transfer (KT). As can be seen in Figure ((a) KT is a method to improve the performance of a light student network by absorbing the know lddge from strong teacher network. In the early studies of KT such knowledge distillation (KD) (Hin
ton et al. 2015), researchers took advantage of the output vector from teacher networks, converted into "soft target" and trained the student network with the soft target and the ground-truth. KD
can only be applied in classification task since the "soft target" is produced by the softmax function with temperature In recent studies, many methods focused on the intermediate representation of the teacher network, in which the feature map (Romero et al., 2015). attention map (Zagoruyko & Komodakis 2016a) or the factor (Kim et al., 2018) extracted from student network are induced to mimic the corresponding one from teacher network by minimizing the difference between them. Figure [(b) and Figure [(C] are the overview of AT and FT. As can be seen, the role of the teacher network in these two methods is simply to provide intermediate representation for imitation while does not give extra help during the training process. Moreover, due to divergence of the structure between the student and teacher networks, the student network usually cannot generate the same intermediate representation as the teacher network. In this case, an intermediate representation with the smallest difference does not equal to accurate prediction.

---

Under review as a conference paper at ICLR 2020
Teater Network
H
Teadhor Prodiches
Newwork
Teacker Prodectie
Truddr
Suaket
Stadcal
Studect Prodiction
Predictio
(a) Student and teacher network
(b) Attention transfer
Tracher
Collater eice Prediction
Colaberston Nework
Netwok
Network
Naato Subvestuet
(c) Factor transfer
(d) Student-teacher Collaboration
Figure 1: Overview of the proposed student-teacher collaboration method compared with other methods. (a) Student and teacher network. (b) Attention transfer (c) Factor transfer (d) Student- teacher collaboration. Different from the previous methods, we employ a collaboration network which is connection of the front part of the student network and the back part of the teacher network. The difference between the predictions from the collaboration network and the teacher network is taken into account of loss during the training process. It can be clearly seen that our STC method additionally utilizes the knowledge from the top part of the teacher network.
To address the above problems, propose novel knowledge transfer method illustrated in Figure [[d]] Different from the previous methods, we employ a collaboration network which is a connection of the front part of the student network and the back part of the teacher network during the training process. Specifically, we select a set of corresponding layers from the student and teacher networks. The front part and the back part of the selected layers of student and teacher networks are called student sub-network and teacher sub-network respectively. The teacher sub-network takes the intermediate representation from the student aub-network as input to make the prediction. Unlike KD using the soft target, in our method, the output tensor from the teacher network is directly treated as the target of the collaboration network. In this manner, our method can be applied to different tasks. During the training process, the difference between the predictions from the collaboration network and the teacher network taken into account of the loss. Through back propagation, the gradient signal in the back part of the teacher network can be transferred to the student network and supervises the training process. In this way, teacher network in our method instructs student network on how get the "answer", rather than just give an intermediate representation for mimicking as in previous methods. It can be clearly seen from Figure [ that our method additionally utilizes the knowledge from the teacher sub-network, compared with the previous methods. It is worth noting that the collaboration network is only used during the training process, the student network with the original structure is used for prediction during the inference time.
Our contributions can be summarized as follows:
We propose a novel knowledge transfer method, additionally utilizing the knowledge from the back part of the teacher network by employing a collaboration network structure. To the best of our knowledge, the training strategy of insturcting the student network with collaboration network has never been used.
The teacher network in our method instructs student network on how get the right "answer", rather than just give intermediate representation for mimicking as in previous methods.
We take the output tensor from the teacher network the target of the collaboration network, which brings good generalizability to our method on different tasks.
We experimentally show that our method outperforms other methods with conventional strategy various datasets in different tasks.
2

---

Under review as a conference paper at ICLR 2020
2 RELATED WORKS
To reduce the model size as well as the computational costs of deep neural networks, a variety of methods have been proposed. These methods can be summarized into five categories: network pruning, parameter quantization, tensor decomposition, efficient architecture design and knowledge transfer. Network pruning way reduce the redundancy in the neural networks(LeCun et al.) 1990). HHan et al. (2015) pruned the network by removing the unimportant connections. Later net- work pruning methods operate at channel or filter levels(Han et al. 2015; Yamamoto & Maeno, 2018 Liu et al., [0188] He et al., 2017). Parameter quantization aims at compressing the model by reducing the number of bits occupied by the weights or neurons. In Gupta et al. (2013). Vanhoucke ct al. (201] Zhuang et al. and Rastegari et al. (2016), authors trained convolutional neural networks using 16-bit, 8-bit, 4-bit and 1-bit weights respectively. HHan et al. (2016) optimized the model com- bined with network pruning. quantization and huffman coding. Tensor decomposition compresses the networks by decomposing dense convolutional kemels with low-rank approximations, including Dnacomposittion[Leeedde
al.] 2014), Tucker decomposition(Kim et al., (016) and tensor ring decomposition (Zha et al.] (0166). Efficient network architecture design is an interesting approach accelerate the model, such as SqueezeNet (landola al.] 2017). Mobilenet (Howard al. 2017).
ShuffleNet (Zhang et al. 2018)) and Xception (aoolle([201))
In addition to the above four strategies that will partially change the components of the network, knowledge transfer is another way to compress the model by improving the performance of lighter student network with the knowledge from stronger teacher network. To the best of our knowledge, the earliest work of knowledge transfer is Bucilua et al. (2006), which to train a small model with data labeled by an ensemble of large model. Li et al. (2014) took the KL divergence between the posterior probabilities produced by the softmax operation from student and teacher model as loss function for knowledge transfer. Hinton et al. (2015) proposed a method called knowledge distilla- tion (KD), converting the posterior probabilities extracted from teacher network into "soft targets". In Fitnets (Romero et al. 2015), they regarded the feature map extracted from the teacher network as hints, and trained the student network by mimicking the corresponding feature maps of student and teacher networks. Attention transfer (AT) (Zagoruyko & Komodakis, (016a) computed the sum- mations of the feature map across the channel to generate the attention map and trained the student network by minimizing the difference between the attention map of corresponding blocks. Factor transfer (FT) (Kim et al., 2018) employed a paraphraser and a translator to translate the knowledge in the feature maps into factors. The student network is optimized by minimizing the la loss between student and teacher factors. In Ding et al. (2019), authors compressed the CNN-DBLSTM model on OCR task with Tucker decomposition and the knowledge in the teacher's BLSTM and inner product layers. There are also studies on knowledge transfer utilizing the adversarial networks, such as Xu et al. Belagiannis al. (2018) and Wang et al. (2018).
3
METHOD
An eminent teacher should not only give a referenced answer to a student, but also teach student how to get the answer. Analogously, during the training process of the student network, a teacher network should teach the student network how to make the right prediction eminent teacher. In this section, we will firstly explain some concerns in the previous knowledge transfer methods and then describe our method in detail explain how the teacher network instructs the student network's learning process in our method.
play role as an
3.1 CONCERNS IN THE PREVIOUS METHODS
Previous knowledge transfer methods, such as AT ZZaoruykoo & Komodakis 20) and FT (Kim al. 2018), train the student network by minimizing the difference between the intermediate rep- resentations extracted from the corresponding layers of student and teacher networks. However, this training strategy contains the following problems.
The first problem is that the teacher network in the previous methods simply provided an interme- diate representation for imitation but did not teach student network how to get it. Therefore. the guidance from teacher network is limited in the previous methods. Another problem is the inter- mediate representation from teacher network is treated as the target in these method. However, due
3

---

Under review as a conference paper at ICLR 2020
Teacher Sub network
Teacher Collaboraation
Tencher Network
Collaboration Prediction
Collaboration Network
Teacher
Inpet Image
Ground truth Label
Studeat Network
Student Sub-network
Student Prediction
Figure 2: Training process of the proposed sudentt-eacche collaboration method. We take the out- put tensor from the teacher network as the target of the collaboration network. The student network is trained with the collaboration loss and the classification loss. During the training process, the weights of the teacher network and teacher sub-network are fixed, only the weights of student net- work are updated.
to the divergence of the structure between the student and teacher networks, the student network usually cannot generate the same intermediate representation as the teacher network. In this case, an intermediate representation with the smallest difference does not equal to accurate prediction. What's ' more, the intermediate representation is extracted from the hidden layer of the network, so the knowledge in the subsequent layers of the teacher network cannot be utilized.
3.2 STUDENT-TEACHER COLLABORATION
To address the above problems, we proposed a novel training strategy for knowledge transfer in our method which is insturcting the student network with collaboration network during the training process. As illustrated in Figure our method consists of three steps. Firstly, we forward propagate pretrained teacher network 7 with the input data I to get the target O,, Secondly, connect all the front part of selected layer of the student network which we called student sub-network to the corresponding laye of the teacher network of the teacher network which we called teacher sub-network takes the intermediate representation from the student sub-network as input to produce the prediction O.. The forward propagation the collaboration network can be formulated as:
collaboration network. The subsequent layers
O, T aub aub( = Tuu a(=-1
where e and are the symbol for collaboration and student network respectively. T sub denotes the calculations of the teacher sub-network. f denotes the activation function. 1, and w denotes the layer number, intermediate representation and weights of the network respectively. Biases are omitted for simplifying notations.
After forward propagate the collaboration network, compute the difference between the predic- tions from the collaboration network and the teacher network the STC loss Lsrc- The back propagation of STC loss can be formulated as:
(LsTo(O.,O., O.) @Os Ot) 00. Ox Orr..
80,
dxy dw:y
where O, denotes the output tensor from teacher network. Biases are omitted for simplifying nota- tions.
'Take He et al. 2016 an example. The spatial size of the feature maps extracted from some layers are downsampled by 2x in ResNet architecture. In this paper, the corresponding layers of student and teacher networks the layers that have output spatial size after downsampling.
4

---

Under review as a conference paper at ICLR 2020
As can be seen in Eq. 2 the teacher network instructs the training process of the student network in a gradient signal manner. To be specific, the gradient signal of the teacher sub-network 80.- plays ăw² a role of weight parameters the backward propagation formula, which indicates that the teacher network provides guidance to the student network on which element in the weights w' should be paid more attention to during the training process. Since we directly take the output tensor of the teacher network the target of the collaboration network, this training strategy is more accurate than minimizing the difference between the intermediate representations as previous methods did. Moreover, it can be clearly seen from Figure 2 that our method additionally utilizes the back part of the teacher network, which will bring more knowledge for student network during the training process.
There are also optional selections of target in our method, which are soft target as in Hinton et al. 2013)) and the ground-truth target. However, since there is no knowledge of the teacher network in the ground-truth target, it is not good choice for our method. As for the soft target, it can only be applied to classification task, because it generated by the softmax function with temperature T. We will demonstrate the experimental results of different selections of the target in Sec.
3.3 LOss FUNCTION
The loss function Liotat for training student network in classification task can separated into two items, i.e. the classification loss LCF and the student-teacher collaboration (STC) loss LSrc
Liotal = aLsTe + (1 LCF
(3)
LSTC C(T sub -
(4)
LCF = C(S
(5)
where C denotes the cross entropy loss, St, denotes the front part calculations of the student network and G denotes the ground-truth label respectively.
We train the student network by minimizing the weighted sum of two loss shown in Eq. 3 where weight a is a pyper-parameter. Specifically, the STC loss is the cross-entropy between the predic- tions from collaboration and teacher networks, the classification loss is the cross-entropy between the prediction from student network and ground-truth. During the training process, the weights of the teacher sub-network are fixed, only the weights of the student network updated.
For other types of tasks, the training process can be achieved by replacing the cross-entropy loss with the corresponding loss, such smooth-L, loss for bounding-box regression (Girshick] 2015).
3.4 INTEGRATING WITH KD
Knowledge distillation (KD) Hinton et al. (2015) trains the student network with the soft target from teacher network. The definition of soft target is p sof softmax(( f), where o is the output tensor of teacher logits (pre-softmax activations) and T is temperature. Since KD takes no additional storage and computational costs during training, it can integrated with other knowledge transfer methods, such as AT Zagoruyko & Komodakis (2016a) and FT Kim et al. (2018) in the classification task. However, these methods result in performance degradation when integrating with KD some cases, because of the fact that mimicking the intermediate representations does not equal good prediction of student network.
In contrast, method takes the output tensor from teacher network as the target, which is consistent with KD. During experiments, our method shows good synergy integrating with KD, thus further improves the performance of student network. We will demonstrate these results in Sec. 4
EXPERIMENTS
In this section, we demonstrate the performance of proposed STC method in classification task on CIFAR-10 10 (Krizhevsky et al.) 20144, CIFAR-106 (Krizhevsky et al. 2009). ImageNet LSVRC 2012 (Deng et al. (209)) datasets and object detection task on PASCAL VOC 2007 (Everingham & Winn 2006) dataset. Firstly, we evaluate the STC performance integrated with or without KD (Hin- ton et al. 2015) of various student and teacher models CIFAR-10, CIFAR- 100 and ImageNet
5

---

Under review as a conference paper at ICLR 2020
Table 1: Top-1 classification (%) on CIFAR-10 dataset.
The first 7 columns from
manuscript of Kim et al. (2018).
Student
Teacher
KD
AT
+KD
FT
+KD
STC
+KD
ResNet-20 (7.78)
ResNet-56 (6.36)
7.19
7.13
6.89
6.85
7.04
6,53
6.3
ResNet-20 (7.78)
WRN-40-1 (6.52)
7.09
7.34
7.00
6.85
6.95
6.30
6.22
VGG- 13 (5.99)
WRN-46-4 (4.22)
5.71
5.54
5.30
4.84
4.65
4.82
4.63
(RN-16-118.62)
WRN-1662 (5.99)
7.64
8.10
7.52
7.64
7.59
7.15
7.05
Table 2: Top-1 classification error (%) CIFAR-100 dataset. manuscript Kim et al. (2018).
The first 7 columns are from
Student
Teacher
KD
AT'
+KD)
FT
+K1)
STC
+KD
ResNet-20 (31.24) ResNet- (26.99)
33.14
31.04
34.78
29.08
32.19
28.75
28.53
ResNet-56 (28.04) ResNet- (26.99)
27.96
27.28
28.01
25.62
26.93
25.33
24.92
ILSVRC2012 datasets, compared with AT (Zaorrykoo & Komodakis 2016a) and FT (Kim et al.) 2018 Secondly, we evaluate our method on PASCAL VOC 2007 dataset for object detection task compared with FT to show the generalizability of our method. Finally, we illustrate the result of different types of target on CIFAR-10 dataset.
4.1 CIFAR
CIFAR-10 and CIFAR-100 both are the basic image classification datasets and are used to evaluate the performance in many knowledge transfer methods (Zagoruyko & Komodakis) 2016a Kim et al.) 2018 Huang & Wang 2017). For these two sets, we train the student network for 500 epochs with a mini-batch size of 128 and weight decay of 10-4, The learning rate starts from 0.1 and divided by 10 150, 300 and 400 epochs. is set to 0.3 on CIFAR-10 and 0.5 on CIFAR- 100 empirically. For data augmentation, we following the policy in He et al. (2016). In order to make the experimental results more convincing, we use the same student and teacher networks as FT (Kim et al., 2018) did, including ResNet (He et al., 2016). Wide ResNet (WRN) (Zagoruyko & Komodakis 2016b) and VGG (Simonyan & Zisserman (014). For CIFAR-10, 10, the corresponding student and teacher networks are 1) ResNet-20 and ResNet-56 which have same width (number of channels) and different depth (number of layers). 2) ResNet-20 and WRN-40-1, which have different width and depth. 3) WRN-16-1 and WRN-16-2, which have same depth and different width. 4) VGG-13 and WRN-46-4, where residual block exists in WRN but not in VGG. For CIFAR-100, the corresponding student and teacher networks are 1) ResNet-56 and ResNet-110. 2) Resnet-20 and R 100. For cases that the intermediate representation from the student sub-network has different number of channels with the teacher sub-network, a simple convolutional layer is employed to transform the dimension.
In Table and Table 2 "Student" column shows the type of student network and the number in parentheses is the performance of student network trained from scratch. "Teacher" column provides the type of teacher network and the performance of pretrained teacher network by our implementa- tion. "+KD" column provides the performance of the corresponding method combined with KD.
Performance on CIFAR-10 10 is shown in Table I for all the cases our STC method outperforms other knowledge transfer methods no matter with or without KD. We can find another result that AT and our method show good synergy when integrating with KD on CIFAR-10 dataset, while FT has conflicts with KD in some cases.
As shown in Table 2 on CIFAR-100 100 dataset, our method achieves the best performance among all the methods in both cases of integrated with or without KD. Different from CIFAR- dataset, after combining with KD, both AT and FT have an obvious performance degradation on CIFAR- 100. This is because the data of CIFAR-100 are more complex than CIFAR-10 and the teacher network (ResNet- 1 10) is deeper than that CIFAR-10. In this case, the knowledge from the teacher sub- network is vital for the student network.

---

Under review as a conference paper at ICLR 2020
4.2 IMAGENET
To demonstrate the performance of our method on large dataset, we choose ResNet-18 as student network and ResNet-34 as teacher network training on ImageNet II.SVRC2012 dataset, which con- sists of 1.2 million training images and 50 thousand validation images. We optimize the student network with a mini-batch size of 512 and weight decay 10 on 4 GPUs. The a are set to 0.5 empirically. The learning rate starts from 0.1 and is divided by 10 when the enror plateaus. A 224x crop is randomly sampled from an image or its horizontal flip for data augmentation. We evaluate the performance of KD,, AT, FT and proposed method STC. Top-1 and Top-5 error rates as shown in Table
As be seen, STC method consistently outperforms other methods both Top-1 and Top- error rates on ImageNet dataset. KD method suffers from the gap of depths between teacher and student network, leads KT and AT, again, show the conflicts when combined with KD, since they target at mimicking the intermediate representation but not the prediction. In contrast, our STC method improve the performance of student network by 1.36% Top-1 error rate without KD and 1.61% % Top-1 error rate combined with KD, consistently shows good synergy integrated with KD.
performance than training the student from seratch.
Table 3: Top-1 1 and Top-5 classification error (%) on ImageNet ILSRR2012 dataset. Pretrained student and teacher network are from PyTorch (Paszke al. 2017 model zoo. Other statistics are gathered from our implementations.
error (%)
Student
Teacher
KD
AT
+KD
+KD
STC
+KD
Top-1
ResNet-18 (30.24)
ResNet-34 (26.69)
33.77
29.52
32.80
29.08
30.30
28.88
28.63 9.66 9.54
Top-5
ResNet-18 (10.92)
ResNet-34 (8.58)
12.29
9.95
11.89
9.75
10.47
4.3 PASCAL VOC 2007
To verify the generalizability of method different tasks, we evaluate the performance of our method on PASCAL VOC 2007 detection dataset. We use Faster-RCNN (Ren et al., (015)) pipeline for evaluation as Kim et al. (2018) did, which are VGG-16 backbone for student network and ResNet-101 backbone for teacher network specifically. We train the student network for 15 epochs with a mini-batch size of 16. The learning rate starts from 0.01 and divided by at 6 and 11 epochs. The a are set to 0.5 empirically. Since the proposal boxes from RPN module teacher and student network are of different spatial positions, we extract the RPN and detector module of the teacher network the teacher sub-network and the backbone of student network as student sub-network, which can be seen in Figure
Referved Preficioon
Callbweation Prodictien
Green muth Tirget
Stuket Predicioo
Lae
frncher seb-astwarkd
Sach extnert
Detector
pooling
pooling
RoI pooling
Mudile
RPN Module
Sindkee
Sal-actoon
Teutury Magg
Studlent BackBons
RackBune
Tracher Natwork
Input
Student Netwrr
Figure 3: STC method on object detection task. We take the backbone of the student network as student sub-network and RPN, ROI pooling, detector module of teacher network as teacher sub- network.
7

---

Under review as a conference paper at ICLR 2020
Table 4 is the performance of method compared with FT on PASCAL VOC 2007 dataset. As can be seen, the mAP of the student network is promoted by 1.6% in our method, while that in is 0.8%. This result proves that our method can be applied to various tasks and is superior to the previous methods.
Table 4: Mean average precision (%) on PASCAL VOC 2007 dataset. Both student and teacher network follow the Faster-RCNN pipeline. The backbone of student and teacher networks are VGG- 16 and ResNet-101 respectively.
Method
Student
Teacher
MAP
FT
VGG-16 backbone(66.4) ResNet- 101 backbone (75.2)
70.3
STC
VGG-16 backbone(69.4)
ResNet-101 backbone (75.2)
71.0
4.4 ABLATION STUDY
In this paragraph, we evaluate the performance of different selections of the target in our method on CIFAR- 10 dataset, using the same student and teacher network as Sec. 4.1
The performance of various target can be seen in Table 5 In Table 5 "output tensor (t)" and "soft target (t)" denote the output tensor and the soft target are generated from the teacher networks. "ground-truth (d)" denotes the ground-truth target from the dataset. For the generating of the soft target, we follow the fashion in Hinton et al. (2015).
As can be seen, the selection of ground-truth target gets the worst results among all the cases. This is because the ground-truth target does not contain knowledge in the teacher network, so there is limited help in improving the performance the student network.
For the selection of soft target, even though it can get about the same performance, it can only be employed on classification task, because the soft target is produced by the softmax function with temperature T. Considering the generalizability of our method on different tasks, choose the output tensor from teacher network as the target.
Table 5: Top-1 classification error( %) of different selections of the target on CIFAR-10 10 dataset.
Student
Teacher
output tensor
soft target (1)
ground-truth (d)
ResNet-20 (7.78)
ResNet-56 (6.39)
6.53
6.59
6.78
20 (7.78)
WRN-40-1 (6.84)
6.30
6.55
6.83
VGG-13 (5.99)
WRN-46-4 (4.44)
4.82
4.80
5.17
(8N-16-1
(6.216-2(6.27))
7.15
7.39
8.01
5 CONCLUSION
In this paper, we proposed novel knowledge transfer method called student-teacher collaboration (STC). Different from previous methods, our method employs collaboration network by connect- ing the front part of the student network and the back part of the teacher network during the training process. We take the difference between the output predictions from the collaboration network and the teacher network into account of the loss to train the student network. Through back propagation, the knowledge of the teacher sub-network can be additionally utilized gradient signal manner. Specifically, the teacher network provides guidance to the student network on which element in the weights should be paid more attention to during the training process.
Through plentiful experiments, it is proved that our STC method outperforms previous knowledge transfer methods on various datasets and has good generalizability on different tasks. What's 's more, method has good synergy integrated with KD to further improve the performance of the student network, while other methods result in accuracy degradation in some cases.
To the best of knowledge, the training strategy which is insturcting the student network with collaboration network has never been used in other knowledge transfer methods. We believe that this novel idea will further promote the development of knowledge transfer.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Vasileios Belagiannis, Azade Farshad, and Fabio Galasso. Adversarial network compression. 03 2018.
Cristian Bucilua, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Pro- ceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining. pp. 531-541. ACM, 2006.
Alfredo Canziani, Adam Paszke, and Eugenio Culurciello. An analysis of deep neural network models for practical applications. arXiv preprint arXiv:1605.07678 2016.
Francois Chollet. Xception: Deep learning with depthwise separable convolutions. computer vision and pattern recognition, pp. 18000-107,, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Haisong Ding, Kai Chen, and Qiang Huo. Compressing con-dblstm models for ocr with teacher- student learning and tucker decomposition. Pattern Recognition, 96, 07 2019. doi: 10.1016/j. patcog.2019.07.00
Mark Everingham and John Winn. The pascal visual object classes challenge 2007 (voc2007) de- velopment kit. International Journal of Computer Vision, 111(1))98-136, 2006.
Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 1440-1448, 2015.
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. Computer Science, 2015.
Han, Pool, J. Tran, and W. Dally, Learning both weights and connections for efficient neural networks. neural information processing systems, pp. 1135-1143, 2015.
S. Han, H. Mao, and W. J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. international conference on learning repre- sentations, 2016.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In The IEEE International Conference Computer Vision (ICCV) Oct 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 1503. 2015.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv: Computer Vision and Pattern Recognition, 2017.
Zehao Huang and Naiyan Wang. Like what you like: Knowledge distill via neuron selectivity transfer. arXiv preprint (rXivv1707.0122 2017.
Forrest N landola, Song Han, Matthew Moskewicz, Khalid Ashraf, William Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and 10.5mb model size. arXiv: Computer Vision and Pattern Recognition, 2017.
Jangho Kim, Scounguk Park, and Nojun Kwak. Paraphrasing complex network: Network compres- sion via factor transfer. neural information processing systems, pp. 2760-2766 2018.
Yong- Yong-Deok Kim. Eunhyeok Park, Sungjoo Yoo. Taclim Choi, Lu Yang, and Dongjun Shin. Com- pression of deep convolutional neural networks for fast and low power mobile applications. 05 2016.

---

Under review as a conference paper at ICLR 2020
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar- 10 and cifar- 10far-100 datasets. https://wwe: cs. toronto. edu/krizccifar html, 6, 2009.
URI:
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The difar-10 dataset. online: http://www. cs. toronto. edu/kriz/cifar, html, 55, 2014.
Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba. Ivan Oseledets, and Victor Lempitsky. Speeding-up convolutional neural networks using fine-tuned cp-decomposition, 2014. URL
htttp://axiv..ott
6553 cite arxiv: 141.65533...
Y. LeCun, Denker, and A. Solla. Optimal brain damage. In Advances in neural information processing systems, pp. 598-605, 1990.
J. Li, R. Zhao, J.-T Huang, and Y. Gong. Learning small-size dnn with output-distriiutiin--aass criteria. Proceedings of the Annual Conference of the International Speech Communication As- sociation, INTERSPEECH, pp. 1910-1914, 01 2014.
Zhenhua Liu, Jizheng Xu, Xiulian Peng, and Ruiqin Xiong. Frequency-domain dynamic pruning for convolutional neural networks. In Advances in Neural Information Processing Systems, pp- 1043-1053, 2018
Tomáš Mikolov, Martin Karafíát, Lukáš Burget, Jan Černocký, and Sanjeev Khudanpur. Recurrent neural network based language model. In Eleventh annual conference of the international speech communication association, 2010.
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring general- ization in deep learning. In Advances in Neural Information Processing Systems, pp. 5947-5956, 2017.
Roman Novak, Yasaman Bahri. Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl- Dickstein. Sensitivity and generalization in neural networks: an empirical study. arXiv preprint 2018
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
M. Rastegari, V. Ordonez, Redmon, and A. Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European Conference Computer Vision, pp. 525- 542. Springer, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99, 2015.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. international conference on learning represen- tations, 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Vanhoucke, A. Senior, and M.Z. Mao. Improving the speed of neural networks on cpus. 2011.
Xiaojie Wang, Rui Zhang, Yu Sun, and Jianzhong Qi. Kdgan: Knowledge distillation with generative adversarial networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Bianchi, and R. Gareett (eds.), Advances in Newral Information Processing Systems 31, pp. 775-786. Curran Associates, Inc., 2018. URL http: hattp:/ppapers wwtp::pppppppppppppp.pppppppppe. works.
7358 - xdgan-knowweed wwweddee--iiitiie htt.::/wow........aaaaaaaeeeeera
pdf
10

---

Under review as a conference paper at ICLR 2020
Zheng Xu, Yen-Chang Hsu, and Jiawei Huang. Training shallow and thin networks for acceleration via knowledge distillation with conditional adversarial networks.
Kohei Yamamoto and Kurato Maeno. Pcas: Pruning channels with attention statistics. 2018.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer. arXiv preprint arXiv:16/2.03922, 2016a.
Sergey Zagoruyko and Nikos Komodakis. arXivv1605.07146. 2016b.
Wide residual networks.
arXiv preprint
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. computer vision and pattern recognition, pp- 6848-6856, 2018.
Qibin Zhao, Guoxu Zhou, Shengli Xie, Liqing Zhang, and Andrzej Cichocki. Tensor ring decom- position. 06 2016.
Bohan Zhuang, Chunhua Shen, Mingkui Tan, Lingqiao Liu, and Ian Reid. Towards effective low- bitwidth convolutional neural networks.


Reviews:
[{"review_id": "rkxVAH9LtB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper proposed a new method for knowledge distillation, which transfers knowledge from a large teacher network to a small student network in training to help network compression and acceleration. The proposed method concatenate the first a few layers of the student network with the last a few layers of the teacher network, and claims the gradient directly flows from teacher to student, instead of through a KL or L2 similarity loss between teacher and student logits. \n\nThe experimental results look good, and extensive experiments have been done on CIFAR, ImageNet and PASCAL VOC. \n\nHowever, the description of the proposed method looks rather unclear. First, the ‘front’ and ‘back’ part of networks are very vague. I have to guess that is the first a few layers of student and last a few layers of teacher. And it is still unclear how many layers in student and teacher are concatenated to form the ‘collaboration network’. How could the authors connect the two subnetwork with different structures?\n\nIt is unclear to me why proposed method is better than AT, FT or FitNets. It looks to me the proposed method use an ad-hoc selected layer to transfer knowledge from teacher to student, and the transfer is indirect because it has to go through the pre-trained subnetwork in teacher.\n\nMinor issue, FT and AT are not defined when they first appear in page 1. \n\nCould the authors show the student and teacher accurayunder standard supervised training in the result tables?\n\nSeveral related works are not discussed, such as\nXu et al. 2018 https://arxiv.org/abs/1709.00513\nBelagiannis et al. 2018 https://arxiv.org/abs/1803.10750\nWang et al. 2018 https://papers.nips.cc/paper/7358-kdgan-knowledge-distillation-with-generative-adversarial-networks\n\n\n============ after rebuttal ================\nI updated my rate to weak accept. Though it is a borderline or below paper to me. The paper has really good empirical results. However, I cannot understand the intuition behind the paper why concatenating teacher to student is better than use l2 for intermediate layers. The choice of the transferring layer seems to be rather ad-hoc, and it is hard to say how much tuning needed to get the empirical benefits.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "rkezle-NtH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper suggests a new method for knowledge transfer from teacher neural network to student: student-teacher collaboration (STC). The idea is that teacher not only provides student with the correct answer or correct intermediate representation, but also instructs student network on how to get the right answer. The paper suggests to merge the front part of the student network and back part of the teacher network into the collaboration network. In this way, the weights in the student subnetwork are learned from the loss backpropagated from the teacher subnetwork. The target labels for collaboration network are outputs of teacher network. The method is adapted for different tasks, classification and bounding box regression are presented in the experiments. It outperforms previous methods on various datasets. Furthermore, the method shows good results when integrated with traditional knowledge distillation.\n\nOverall, the paper is a significant algorithmic contribution. The related work section provides thorough review of different methods to decrease computational costs, including not only knowledge distillation, but also pruning, compressing and decomposition approaches. The idea is elegant and, to the best of my knowledge, has never been suggested in other works. Considering the theoretical part, it is clearly shown how the gradient signal from the teacher sub-network guides the student network on which part of the weights should be paid attention on. All derivations are easy to follow. The paper also considers how the suggested idea is aimed to solve the problems of previous knowledge transfer methods. The experimental section is consistent and clearly shows the advantage of the suggested method. Teacher and student networks used are different sizes of ResNet, Wide ResNet and VGG. The paper presents classification experiments on CIFAR-10, CIFAR-100, ImageNet and object detection experiment on PASCAL VOC 2007 dataset. STC outperforms previous methods, both with KD integration and without. The performance is always better than pure student training (which was not always the case for previous methods) and sometimes the results are even better than teacher performance. Finally, the choice of teacher output as target over soft target and ground truth, which was previously motivated in the theoretical section, is shown to be superior in the experiment.\n\nPossible improvement of the paper is the instruction on how to choose the intermediate layer from where to teach the representation, i.e. where the student sub-network ends and teacher sub-network begins. For object detection experiment the choice of the border is naturally defined by the architecture of the network in Faster-RCNN approach. Could the choice be different? May be somewhere inside the BackBone part of the networks? For classification, it could be interesting to study how this choice influences the results. However, this question didn’t affect my score of the paper, and, as far as I know, it is also not considered in the previous works on knowledge distillation.\n\nMinor comments\n1.\tIn the context of accelerating the models using decomposition, Lebedev et al., ICLR 2015 could be cited.\n2.\tPage 2: difference tasks -> different tasks\n3.\tPage 2 the first bullet point: additionally utilizes -> additionally utilizing/which additionally utilizes\n4.\tPage 2 the third bullet point: brings good generalizability -> which brings good generalizability\n5.\tPage 5: “training strategy is more accelerate than” – odd phrase\n6.\tPage 6: while KT has conflicts with KD in some cases -> while FT has conflicts with KD in some cases.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "HyxdAeIpFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall the method proposed in this paper is simple but effective, and adequate experimental results are given to show its performance improvements.  However, the literature survey of this paper is not satisfactory.\n\n1. To reduce model size, there are several different ways including efficient architecture design, parameter pruning, quantization, tensor decomposition and knowledge distillation. The authors forgot to mention tensor decomposition and mixed it with efficient architecture design. As for parameter pruning and quantization,  many important papers are missing.\n\n2. Utilizing the \"soft targets\" to transfer knowledge from teacher to student model is not first proposed by Hinton et al. (2015). To the best of my knowledge, it is first proposed in  \nJ. Li, R. Zhao, J.-T. Huang, Y. Gong, “Learning small-size DNN with output-distribution-based criteria,” Proc. Interspeech-2014, pp.1910-1914.\n\n3. Leveraging back part of teacher model's guidance to improve student performance has been investigated by other researchers on OCR tasks in \nDing H, Chen K, Huo Q. Compressing CNN-DBLSTM models for OCR with teacher-student learning and Tucker decomposition[J]. Pattern Recognition, 2019, 96: 106957.\nThey combine student's CNN with teacher's DBLSTM to learn better representations.\n\nIn conclusion, I will give a weak reject currently, unless the authors improve their literature survey and modify their claims.\n\n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper has been assessed by three reviewers scoring it as follows: 6, 3, 8. The submission however attracted some criticism post-rebuttal from the reviewers e.g., why concatenating teacher to student is better than the use l2 loss or how the choice of transf. layers has been made (ad-hoc). Similarly, other major criticism includes lack of proper referencing to parts of work that have been in fact developed earlier in preceding papers. On balance, this paper falls short of the expectations of ICLR 2020, thus it cannot be accepted at this time. The authors are encouraged to work through major comments and resolve them for a future submission.
Decision: Reject