Knowledge Distillation from Few Samples

Current knowledge distillation methods require full training data to distill knowledge from a large `` teacher '' network to a compact `` student '' network by matching certain statistics between `` teacher '' and `` student '' such as softmax outputs and feature responses . This is not only time-consuming but also inconsistent with human cognition in which children can learn knowledge from adults with few examples . This paper proposes a novel and simple method for knowledge distillation from few samples . Taking the assumption that both `` teacher '' and `` student '' have the same feature map sizes at each corresponding block , we add a 1× 1 conv-layer at the end of each block in the student-net , and align the block-level outputs between `` teacher '' and `` student '' by estimating the parameters of the added layer with limited samples . We prove that the added layer can be absorbed/merged into the previous conv-layer to formulate a new conv-layer with the same size of parameters and computation cost as the previous one . Experiments verify that the proposed method is very efficient and effective to distill knowledge from teacher-net to student-net constructing in different ways on various datasets . 1 INTRODUCTION . Deep neural networks ( DNNs ) have demonstrated extraordinary success in a variety of fields such as computer vision ( Krizhevsky & Hinton , 2012 ; He et al. , 2016 ) , speech recognition ( Hinton et al. , 2012 ) , and natural language processing ( Mikolov et al. , 2010 ) . However , DNNs are resource-hungry which hinders their wide deployment to some resource-limited scenarios , especially low-power embedded devices in the emerging Internet-of-Things ( IoT ) domain . To address this limitation , extensive works have been done to accelerate or compress deep neural networks . Putting those works on designing ( Chollet , 2016 ) or automatically searching efficient network architecture aside ( Zoph & Le , 2016 ) , most studies try to optimize DNNs from four perspectives : network pruning ( Han et al. , 2016 ; Li et al. , 2016 ) , network decomposition ( Denton et al. , 2014 ; Jaderberg et al. , 2014 ) , network quantization ( or low-precision networks ) ( Gupta et al. , 2015 ; Courbariaux et al. , 2016 ; Rastegari et al. , 2016 ) and knowledge distillation ( Hinton et al. , 2015 ; Romero et al. , 2015 ) . Among these method categories , knowledge distillation is somewhat different due to the utilization of information from the pre-trained teacher-net . The concept was proposed by ( Bucila et al. , 2006 ; Ba & Caruana , 2014 ; Hinton et al. , 2015 ) for transferring knowledge from a large `` teacher '' model to a compact yet efficient `` student '' model by matching certain statistics between `` teacher '' and `` student '' . Further research introduced various kinds of matching mechanisms in the field of DNN optimization . The distillation procedure designs a loss function based on the matching mechanisms and enforces the loss during a full training process . Hence , all these methods usually require time-consuming training procedure along with fully annotated large-scale training dataset . Meanwhile , some network pruning ( Li et al. , 2016 ; Liu et al. , 2017 ) and decomposition ( Zhang et al. , 2016 ; Kim et al. , 2016 ) methods can produce extremely small networks , but with large accuracy drops so that time-consuming fine-tuning is required for possible accuracy recovery . Usually , it may still not be able to recover the accuracy drops with the original cross-entropy loss due to its low representation capacity . Hence , knowledge distillation may be used to alleviate the problem , since the compact student-net can sometimes be trained to match the performance of the teacher-net . For instance , Crowley et al . ( 2017 ) uses cheap group convolutions and pointwise convolutions to build a small student-net and adopts knowledge distillation to transfer knowledge from a full-sized `` teacher-net '' to the `` student-net '' . However , it still suffers from high training cost . As is known , children can learn knowledge concept from adults with few examples . This cognition phenomenon has motivated the development of the few-shot learning ( Fei-Fei et al. , 2006 ; Bart & Ullman , 2005 ) , which aims to learn information about object categories from a few training samples , and focuses more on image classification task . Nevertheless , it inspires people to consider the possibility of knowledge distillation from few samples . Some recent works on knowledge distillation address this problem by constructing `` pseudo '' training data ( Kimura et al. , 2018 ; Lopes et al. , 2017 ) with complicated heuristics and heavy engineering , which are still costly . This paper proposes a novel and simple three-step method for few-sample knowledge distillation ( FSKD ) as illustrated in Figure 1 , including student-net design , teacher-student alignment , and absorbing added conv-layer . We assume that both `` teacher '' and `` student '' nets have the same feature map sizes at each corresponding block . However , the relatively small student-net can be obtained in various ways , such as pruning/decomposing the teacher-net , and fully redesigned network with random initialization . We add a 1×1 conv-layer at the end of each block of the student-net and align the block-level outputs between `` teacher '' and `` student '' , which is done by estimating the parameters of the added layer with few samples using least square regression . Since the added 1×1 conv-layers have relatively few parameters , we can get a good approximation from a small number of samples . We further prove that the added 1×1 conv-layer can be absorbed/merged into the previous conv-layer when certain conditions fulfill , so that the new conv-layer has the same number of parameters and computation cost as the older/previous one . We argue that FSKD has many potential applications , especially when fine-tuning or full training is not feasible in practice . We just name a few such cases below . First , edge devices have limited computing resources , while FSKD offers the possibility of on-device learning to compress deep models with a limited number of samples . Second , FSKD may help software/hardware vendors optimizing the deep models from their customers when full training data is unavailable due to privacy or confidential issues . Third , FSKD enables fast model deployment optimization when there is a strict time budget . Our major contributions can be summarized as follows : ( 1 ) To the best of our knowledge , we are the first to show that knowledge distillation can be done with few samples within minutes on a desktop PC . ( 2 ) The proposed FSKD method is widely applicable not only for fully redesigned student-nets but also for compressed networks from pruning and decomposition-based methods . ( 3 ) We demonstrate significant performance improvement of the student-net by FSKD , comparing to existing distillation techniques on various datasets and network structures . 2 RELATED WORK . Knowledge Distillation ( KD ) transfers knowledge from a pre-trained large `` teacher '' network ( or even an ensemble of networks ) to a small `` student '' network , for facilitating the deployment at test time . Originally , this is done by regressing the softmax output of the teacher model ( Hinton et al. , 2015 ) . The soft continuous regression loss used here provides richer information than the label based loss , so that the distilled model can be more accurate than training on labeled data with cross-entropy loss . Later , various works have extended this approach by matching other statistics , including intermediate feature responses ( Romero et al. , 2015 ; Chen et al. , 2016 ) , gradient ( Srinivas & Fleuret , 2018 ) , distribution ( Huang & Wang , 2017 ) , Gram matrix ( Yim et al. , 2017 ) , etc . Deep mutual learning ( Zhang et al. , 2018 ) trains a cohort of student-nets and teaches each other collaboratively with model distillation throughout the training process . All these methods require a large amount of data ( known as the “ transfer set ” ) to transfer the knowledge , whereas we aim to provide a solution with a limited number of samples . We need emphasize that FSKD has a quite different philosophy on aligning intermediate responses to the closest knowledge distillation method FitNet ( Romero et al. , 2015 ) . FitNet re-trains the whole student-net with intermediate supervision using a larger amount of data , while FSKD only estimates parameters for the added 1× 1 conv-layer with few samples . We will verify in experiments that FSKD is not only more efficient but also more accurate than FitNet . Network Pruning methods obtain a small network by pruning weights from a trained larger network , which can keep the accuracy of the larger model if the prune ratio is set properly . Han et al . ( 2015 ) proposes to prune the individual weights that are near zero . Recently , channel pruning has become increasingly popular thanks to its better compatibility with off-the-shelf computing libraries , compared with weights pruning . Different criteria have been proposed to select the channel to be pruned , including norm of weights ( Li et al. , 2016 ) , scales of multiplicative coefficients ( Liu et al. , 2017 ) , statistics of next layer ( Luo et al. , 2017 ) , etc . It is usually required iterative loop between pruning and fine-tuning for achieving better pruning ratio and speedup . Similar gradually adjusting trick is also applied to train very-deep neural networks ( Smith et al. , 2016 ) . Meanwhile , Network Decomposition methods try to factorize heavy layers in DNNs into multiple lightweight ones . For instance , it may adopt low-rank decomposition to fully-connection layers ( Denton et al. , 2014 ) , and different kinds of tensor decomposition to conv-layers ( Zhang et al. , 2016 ; Kim et al. , 2016 ) . However , aggressive network pruning or network decomposition usually lead to large accuracy drops , thus fine-tuning is required to alleviate those drops ( Li et al. , 2016 ; Liu et al. , 2017 ; Zhang et al. , 2016 ) . As aforementioned , KD is more accurate than directly training on labeled data , it is of great interest to explore KD on extremely pruned or decomposed networks , especially under the few-sample setting . Learning with few samples has been extensively studied under the concept of one-shot or few-shot learning . One category of methods directly model few-shot samples with generative models ( Fei-Fei et al. , 2006 ; Lake et al. , 2011 ) , while most others study the problem under the notion of transfer learning ( Bart & Ullman , 2005 ; Ravi & Larochelle , 2017 ) . In the latter category , meta-learning methods ( Vinyals et al. , 2016 ; Finn et al. , 2017 ) solve the problem in a learning to learn fashion , which has been recently gaining momentum due to their application versatility . Most studies are devoted to the image classification task , while it is still less-explored for knowledge distillation from few samples . Recently , some works tried to address this problem . Kimura et al . ( 2018 ) constructs pseudo-examples using the inducing point method , and develops a complicated algorithm to optimize the model and pseudo-examples alternatively . Lopes et al . ( 2017 ) records per-layer meta-data for the teacher-net in order to reconstruct a training set , and then adopts a standard training procedure to obtain the student-net . Both are very costly due to the complicated and heavy training procedure . On the contrary , we aim for a simple solution for knowledge distillation from few samples .

Summary: In this paper, an efficient re-training algorithm for neural networks is proposed. The essence is like Hinton's distillation, but in addition to use the output of the last layer, the outputs of intermediate layers are also used. The core idea is to add 1x1 convolutions to the end of each layer and train them by fixing other parameters. Since the number of parameters to train is small, it performs well with the small number of samples such as 500 samples.