# Layer-Level Knowledge Distillation for Deep Neural Network Learning
## 1 Introduction  
Convolutional neural networks (CNN) of deep learning have proved to be very successful in many applications in the field of computer vision, such as image classification [1], object detection [2; 3; 4], semantic segmentation [5], and others. However, the large network structure also brings high computational complexity and memory cost. This problem motivates researchers to develop model compression methods for neural networks. Model compression methods can be roughly divided into two categories: network pruning [6; 7; 8; 9; 10] and knowledge distillation (KD) methods [11; 12; 13; 14; 15]. Network pruning methods reduce the number of weights or neurons during neural network training, whereas knowledge distillation methods train a small model linked to a large model, which is also called a teacher-student network. The core idea of knowledge distillation is to consider not only the supervision of information from the large model that can be transferred to the neurons of the small model, but also the output of the small network, which should be as similar as possible to the output of the large network.  
Applications of network pruning and knowledge distillation include reducing power consumption on mobile devices [16; 17; 18], object detection [19], visual relationship detection [20; 21], and character recognition [22]. One-shot whole network compression [16] compresses the CNN network to deploy deep CNNs on mobile devices. However, model compression methods learn compact models, but with reduced accuracy. In [19], a new framework was proposed to learn compact and fast object detection networks with improved accuracy using knowledge distillation and hint learning.  
In this paper, we propose a new Layer Selectivity Learning (LSL) framework to tackle the knowledge distillation method to transfer information from a well-trained large CNN model to asmaller CNN model in a layer-level manner. This leads to a compact and small model with higher accuracy than one that is trained alone. We exploit a teacher-network framework called Auxiliary Structure Learning (ASL) to link a well-trained large model, such as AlexNet [23], with a small new model within one unified learning framework. This can be achieved by projection layers and alignment layers. Then, two novel operators are proposed to select the corresponding intermediate layers automatically to link both models: the layered inter-class Gram matrix and the inter-layered Gram matrix. The former matrix uses feature maps within the same layer to obtain the relationships between classes. Each entry in the layered inter-class Gram matrix indicates the relationship of two classes to be highly correlated or irrelevant. The latter matrix calculates the Gram matrix using all feature vectors of two adjacent layers. The matrix represents whether the network can learn more diverse features than those in the previous layer. The contribution of the LSL framework is two-fold:  
* Auxiliary Structure Learning (ASL) is proposed to improve the performance of a target model by distilling knowledge from a well-trained model;
* The Layer Selectivity Procedure (LSP) includes two novel operators, the layered inter-class Gram matrix and the inter-layered Gram matrix, which are proposed for the selection and linkage of the intermediate layers of both target and source models.  
The experimental results are demonstrated on three publicly available datasets, and show significant improvement in classification accuracy compared to other recently developed transfer learning methods. The rest of paper is organized as follows. The literature review of model compression and knowledge distillation is introduced in Section 2. In Section 3, the proposed Auxiliary Structure Learning (ASL) and the Layer Selectivity Procedure (LSP) are presented. The experimental settings and the results are demonstrated in Section 4. Lastly, Section 5 concludes our paper.