# Layer-Level Knowledge Distillation for Deep Neural Network Learning
## 3 Proposed Method
### Auxiliary Structure Learning (ASL)
#### 3.2.1 Inter-Layered Gram Matrix  
Inspired by [PERSON] et al. [10], the Gramian matrix is capable of representing the texture information of the input image, because it is generated by calculating the inner product of feature vectors. The directionality between features can be considered as the similarity of texture information. The Gramian matrix that computes the inner product of features from adjacent layers is referred to as the _inter-layered Gram matrix_. The value in the entry of the inter-layered Gram matrix \((i,j)\) is equal to the inner product of the \(i\)-th channel and the \(j\)-th channel in the deep learning model. The large value of the inner product means two feature vectors are more aligned to the same direction. Features extracted from that layer are considered to carry less information. We use the inter-layered Gram matrix to represent the relationship between feature maps between two layers and consider the layer diversity. The layer with more diversity, between the inter-layered Gram matrix and its previous layer, is more likely to be chosen as the alignment layer. The illustration of the procedure employed to determine the inter-layered Gram matrix is depicted in Figure 2.  
For a given trained CNN model, feature maps can be extracted from the convolutional and fully-connected layers. Then, we compute the inner product of feature vectors from two layers to obtain the inter-layered Gram matrix \(C_{m,\ n}^{l}\in\mathbb{R}^{M\times N}\). For a given layer \(l\) and its previous layer \(l-1\), the feature maps can be denoted as \(F^{l-1}\in\mathbb{R}^{K\times M}\) and \(F^{l}\in\mathbb{R}^{K\times N}\), where \(M\) and \(N\) are the number of channels in layer \(l-1\) and layer \(l\). \(m\) and \(n\) are the index of the channels in layer \(l-1\) and layer \(l\), respectively. \(K\) is the 1D feature dimension of the feature map. In practice, reshaping the feature maps into one dimension with zero padding is required to ensure that the feature vectors are in the same dimension. The \((m,n)\) entry of the inter-layered Gram matrix \(G_{m,\ n}^{l}\) can be defined as follows:  
\[G_{m,\ n}^{l}=\frac{F_{m}^{l-1},\ \Gamma_{n}^{l}}{K}. \tag{4}\]  
Figure 2: Illustration of the calculation for the inter-layered Gram matrix.  
The feature diversity can be calculated for a given layer \(l\) by:  
\[G^{l}_{diversity}=\frac{\sum_{m}\sum_{n}G^{l}_{m,\ n}}{M\times N}. \tag{5}\]  
The entry of the inter-layered Gram matrix represents the relationship between two layers. A value close to zero indicates that features learned through the adjacent layers are not aligned and thus more diverse. A smaller value of \(G^{l}_{diversity}\) indicates that layer \(l\) contains more distinctive features. Note that the values of feature maps obtained by the activation function ReLu are non-negative. Therefore, the values of the inter-layered Gram matrix range from 0 to 1. Then, the layer with minimal values of \(G^{l}_{diversity}\) is chosen as the representative intermediate layer. The optimization can be represented as follows:  
\[l^{*}=\ argmin_{l}G^{l}_{diversity}. \tag{6}\]