Knowledge Transfer via Student-Teacher Collaboration

Accompanying with the flourish development in various fields , deep neural networks , however , are still facing with the plight of high computational costs and storage . One way to compress these heavy models is knowledge transfer ( KT ) , in which a light student network is trained through absorbing the knowledge from a powerful teacher network . In this paper , we propose a novel knowledge transfer method which employs a Student-Teacher Collaboration ( STC ) network during the knowledge transfer process . This is done by connecting the front part of the student network to the back part of the teacher network as the STC network . The back part of the teacher network takes the intermediate representation from the front part of the student network as input to make the prediction . The difference between the prediction from the collaboration network and the output tensor from the teacher network is taken into account of the loss during the train process . Through back propagation , the teacher network provides guidance to the student network in a gradient signal manner . In this way , our method takes advantage of the knowledge from the entire teacher network , who instructs the student network in learning process . Through plentiful experiments , it is proved that our STC method outperforms other KT methods with conventional strategy . 1 INTRODUCTION . Deep neural networks have produced breakthrough results in various fields , such as computer vision ( Krizhevsky et al. , 2012 ; He et al. , 2016 ) and natural language processing ( Mikolov et al. , 2010 ) in recent years . Through a mass of studies ( Neyshabur et al. , 2017 ; Canziani et al. , 2016 ; Novak et al. , 2018 ) , researchers have proved that a DNN with larger capacity will have a better generalizability , which leads to a better performance . However , larger capacity will also cause heavier computational costs and storage , making these powerful models difficult to meet real-time requirements on embedded systems . One way to compress these heavy models is knowledge transfer ( KT ) . As can be seen in Figure 1 ( a ) , KT is a method to improve the performance of a light student network by absorbing the knowledge from a strong teacher network . In the early studies of KT such as knowledge distillation ( KD ) ( Hinton et al. , 2015 ) , researchers took advantage of the output vector from teacher networks , converted it into “ soft target ” and trained the student network with the soft target and the ground-truth . KD can only be applied in classification task since the “ soft target ” is produced by the softmax function with temperature T. In recent studies , many methods focused on the intermediate representation of the teacher network , in which the feature map ( Romero et al. , 2015 ) , attention map ( Zagoruyko & Komodakis , 2016a ) or the factor ( Kim et al. , 2018 ) extracted from student network are induced to mimic the corresponding one from teacher network by minimizing the difference between them . Figure 1 ( b ) and Figure 1 ( c ) are the overview of AT and FT. As can be seen , the role of the teacher network in these two methods is simply to provide an intermediate representation for imitation while does not give extra help during the training process . Moreover , due to divergence of the structure between the student and teacher networks , the student network usually can not generate the same intermediate representation as the teacher network . In this case , an intermediate representation with the smallest difference does not equal to an accurate prediction . To address the above problems , we propose a novel knowledge transfer method as illustrated in Figure 1 ( d ) . Different from the previous methods , we employ a collaboration network which is a connection of the front part of the student network and the back part of the teacher network during the training process . Specifically , we select a set of corresponding layers from the student and teacher networks . The front part and the back part of the selected layers of student and teacher networks are called student sub-network and teacher sub-network respectively . The teacher sub-network takes the intermediate representation from the student sub-network as input to make the prediction . Unlike KD using the soft target , in our method , the output tensor from the teacher network is directly treated as the target of the collaboration network . In this manner , our method can be applied to different tasks . During the training process , the difference between the predictions from the collaboration network and the teacher network is taken into account of the loss . Through back propagation , the gradient signal in the back part of the teacher network can be transferred to the student network and supervises the training process . In this way , teacher network in our method instructs student network on how to get the “ answer ” , rather than just give an intermediate representation for mimicking as in previous methods . It can be clearly seen from Figure 1 that our method additionally utilizes the knowledge from the teacher sub-network , compared with the previous methods . It is worth noting that the collaboration network is only used during the training process , the student network with the original structure is used for prediction during the inference time . Our contributions can be summarized as follows : • We propose a novel knowledge transfer method , additionally utilizing the knowledge from the back part of the teacher network by employing a collaboration network structure . To the best of our knowledge , the training strategy of insturcting the student network with a collaboration network has never been used . • The teacher network in our method instructs student network on how to get the right “ answer ” , rather than just give an intermediate representation for mimicking as in previous methods . • We take the output tensor from the teacher network as the target of the collaboration network , which brings good generalizability to our method on different tasks . • We experimentally show that our method outperforms other methods with conventional strategy on various datasets in different tasks . 2 RELATED WORKS . To reduce the model size as well as the computational costs of deep neural networks , a variety of methods have been proposed . These methods can be summarized into five categories : network pruning , parameter quantization , tensor decomposition , efficient architecture design and knowledge transfer . Network pruning is a way to reduce the redundancy in the neural networks ( LeCun et al. , 1990 ) . Han et al . ( 2015 ) pruned the network by removing the unimportant connections . Later network pruning methods operate at channel or filter levels ( Han et al. , 2015 ; Yamamoto & Maeno , 2018 ; Liu et al. , 2018 ; He et al. , 2017 ) . Parameter quantization aims at compressing the model by reducing the number of bits occupied by the weights or neurons . In Gupta et al . ( 2015 ) , Vanhoucke et al . ( 2011 ) , Zhuang et al . and Rastegari et al . ( 2016 ) , authors trained convolutional neural networks using 16-bit , 8-bit , 4-bit and 1-bit weights respectively . Han et al . ( 2016 ) optimized the model combined with network pruning , quantization and huffman coding . Tensor decomposition compresses the networks by decomposing dense convolutional kernels with low-rank approximations , including CP-decomposition ( Lebedev et al. , 2014 ) , Tucker decomposition ( Kim et al. , 2016 ) and tensor ring decomposition ( Zhao et al. , 2016 ) . Efficient network architecture design is an interesting approach to accelerate the model , such as SqueezeNet ( Iandola et al. , 2017 ) , Mobilenet ( Howard et al. , 2017 ) , ShuffleNet ( Zhang et al. , 2018 ) and Xception ( Chollet , 2017 ) . In addition to the above four strategies that will partially change the components of the network , knowledge transfer is another way to compress the model by improving the performance of a lighter student network with the knowledge from a stronger teacher network . To the best of our knowledge , the earliest work of knowledge transfer is Bucilua et al . ( 2006 ) , which to train a small model with data labeled by an ensemble of large model . Li et al . ( 2014 ) took the KL divergence between the posterior probabilities produced by the softmax operation from student and teacher model as loss function for knowledge transfer . Hinton et al . ( 2015 ) proposed a method called knowledge distillation ( KD ) , converting the posterior probabilities extracted from teacher network into “ soft targets ” . In Fitnets ( Romero et al. , 2015 ) , they regarded the feature map extracted from the teacher network as hints , and trained the student network by mimicking the corresponding feature maps of student and teacher networks . Attention transfer ( AT ) ( Zagoruyko & Komodakis , 2016a ) computed the summations of the feature map across the channel to generate the attention map and trained the student network by minimizing the difference between the attention map of corresponding blocks . Factor transfer ( FT ) ( Kim et al. , 2018 ) employed a paraphraser and a translator to translate the knowledge in the feature maps into factors . The student network is optimized by minimizing the l2 loss between student and teacher factors . In Ding et al . ( 2019 ) , authors compressed the CNN-DBLSTM model on OCR task with Tucker decomposition and the knowledge in the teacher ’ s BLSTM and inner product layers . There are also studies on knowledge transfer utilizing the adversarial networks , such as Xu et al. , Belagiannis et al . ( 2018 ) and Wang et al . ( 2018 ) . 3 METHOD . An eminent teacher should not only give a referenced answer to a student , but also teach student how to get the answer . Analogously , during the training process of the student network , a teacher network should teach the student network how to make the right prediction to play a role as an eminent teacher . In this section , we will firstly explain some concerns in the previous knowledge transfer methods and then describe our method in detail to explain how the teacher network instructs the student network ’ s learning process in our method . 3.1 CONCERNS IN THE PREVIOUS METHODS . Previous knowledge transfer methods , such as AT ( Zagoruyko & Komodakis , 2016a ) and FT ( Kim et al. , 2018 ) , train the student network by minimizing the difference between the intermediate representations extracted from the corresponding layers of student and teacher networks . However , this training strategy contains the following problems . The first problem is that the teacher network in the previous methods simply provided an intermediate representation for imitation but did not teach student network how to get it . Therefore , the guidance from teacher network is limited in the previous methods . Another problem is the intermediate representation from teacher network is treated as the target in these method . However , due to the divergence of the structure between the student and teacher networks , the student network usually can not generate the same intermediate representation as the teacher network . In this case , an intermediate representation with the smallest difference does not equal to an accurate prediction . What ’ s more , the intermediate representation is extracted from the hidden layer of the network , so the knowledge in the subsequent layers of the teacher network can not be utilized .

Summary: The paper suggests a new method for knowledge transfer from teacher neural network to student: student-teacher collaboration (STC). The idea is that teacher not only provides student with the correct answer or correct intermediate representation, but also instructs student network on how to get the right answer. The paper suggests to merge the front part of the student network and back part of the teacher network into the collaboration network. In this way, the weights in the student subnetwork are learned from the loss backpropagated from the teacher subnetwork. The target labels for collaboration network are outputs of teacher network. The method is adapted for different tasks, classification and bounding box regression are presented in the experiments. It outperforms previous methods on various datasets. Furthermore, the method shows good results when integrated with traditional knowledge distillation.