# Layer-Level Knowledge Distillation for Deep Neural Network Learning
## 3 Proposed Method
### Auxiliary Structure Learning (ASL)  
Inspired by [PERSON] et al. [25], we propose the Auxiliary Structure Learning framework [14]. It is an asymmetric dual-path learning framework, consisting of four components: (1) teacher network, (2) student network, (3) alignment layers, and (4) projection layers.  
* _Teacher network_. The teacher network is also known as the source network, and is a well-trained deep learning model. In the ASL framework, the teacher network is supplied with pre-trained weights before the student network starts to train.  
Figure 1: Complete architecture of the Layer Selectivity Learning (LSL) framework.  
* _Student network_. The student network is also known as the target network. A student network is a new deep learning model without pre-trained weights.
* _Alignment layer_. The alignment layer is used to bridge the dual path deep learning models. It is designed to be the loss function that enforces similarity between two corresponding intermediate layers of both models.
* _Projection layer_. Since any two intermediate layers of both the teacher and student models are not likely to have the same feature dimensions, they cannot align directly. Before the representation layer connects to the alignment layer, it needs to connect to one additional fully connected layer, which is called the projection layer. This ensures that the two intermediate layers are projected to same dimension.  
In the ASL framework, the teacher network can be considered as auxiliary architecture to train the student network. The choice of the teacher network is flexible, since we can choose any well-known deep model according to the requirement of our task, such as AlexNet [11] or VGGNet [16]. For the student network, it can be any new model without pre-trained weights. To link the layer of the teacher model to the layer of the student model, projection layers are required to project both layers to the same dimension. In fact, the fully connected layer is exploited as the projection layer in the ASL framework. After connecting to the projection layers, the alignment layer is introduced to build dependencies of both models. Alignment loss is in following form:  
\[\mathcal{L}_{Align}^{(k)}(t,\;s)=\|X_{t}^{(i)}-X_{s}^{(j)}\|_{2} \tag{1}\]  
where \(t\) and \(s\) represent the teacher network and the student network, respectively. \(X_{t}^{(i)}\) and \(X_{s}^{(j)}\) are the feature representations from the \(i\)-th and \(j\)-th layers of the teacher network and the student network, respectively. \(k\) is the index of the alignment layer. By minimizing \(\mathcal{L}_{Align}^{(k)}(t,\;s)\), it enforces the \(j\)-th intermediate layer of the student model to be similar to the \(i\)-th intermediate layer of the teacher model. This transfers knowledge from the teacher network to the student network. The benefits to the student network are accuracy improvement and fast convergence, compared to the network when it is trained alone.  
The total loss function of the ASL framework can be defined as follows:  
\[\mathcal{L}_{total}=\sum\
olimits_{k=1}^{n}\mathcal{L}_{Align}^{(k)}(t,\;s)+ \sum\limits_{  
the intermediate layer from the teacher network and one corresponding intermediate layer from the student network to link both models? In this section, a novel scheme, the Layer Selectivity Procedure (LSP), is proposed to determine the intermediate layers and bind the two models. Two matrices, the layered inter-class Gram matrix and the inter-layered Gram matrix, are proposed to help select the appropriate layers from the teacher and student models. Then, the projection layers and alignment layers can be later deployed to build the ASL framework.