# Layer-Level Knowledge Distillation for Deep Neural Network Learning
## 4 Experimental Results
### Results of ASL Framework and LSP  
In this section, we would like to verify: (1) whether or not the ASL framework helps the student model to improve the model performance and (2) if the layered inter-class Gram matrix and the inter-layered Gram matrix are able to select the best intermediate layers to bind the teacher model and the student model. We firstly trained the teacher network and the student network separately on the PPMI dataset, Willow dataset, and UIUC-Sports dataset as the baseline method. The LSP loss (defined by Equation (7)) was calculated for the teacher network and the student network using the PPMI dataset. As shown in Table 3, the best convolutional layer is marked in blue and the best fully connected layer is marked in green. In the teacher network, Conv5 of the teacher network had the minimal LSP value and Conv3 reached the minimal LSP in student network. For the fully connected layer, FC1 in both models reached the minimal LSP. These results somehow followed the concept of deep learning models in which the first fully connected layer contains the richest features and the deeper convolutional layers carry more high-level and discriminative features. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Layer Name** & **Type** & **Kernel** & **Stride** & **\# Channels** \\ \hline \multirow{2}{*}{Conv1} & conv & 11 & 4 & 96 \\  & maxpool & 3 & 2 & 96 \\ \hline \multirow{2}{*}{Conv2} & conv & 5 & 1 & 256 \\  & maxpool & 3 & 2 & 256 \\ \hline \multirow{2}{*}{Conv3} & conv & 3 & 1 & 384 \\ \hline \multirow{2}{*}{Conv4} & conv & 3 & 1 & 384 \\ \hline \multirow{2}{*}{Conv5} & conv & 3 & 1 & 256 \\  & maxpool & 3 & 2 & 256 \\ \hline \multirow{2}{*}{FC1} & & & 4096 & \\ \hline \multirow{2}{*}{FC2} & & & 4096 & \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Configuration of teacher network using AlexNet. Each convolutional layer and fully connected layer was followed by ReLU, which is not shown in the table for brevity. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Layer Name** & **Type** & **Kernel** & **Stride** & **\# Channels** \\ \hline \multirow{2}{*}{Conv1} & conv & 3 & 1 & 96 \\  & maxpool & 3 & 2 & 96 \\ \hline \multirow{2}{*}{Conv2} & conv & 3 & 2 & 384 \\ \hline \multirow{2}{*}{Conv3} & conv & 3 & 1 & 256 \\  & maxpool & 3 & 2 & 256 \\ \hline \multirow{2}{*}{FC1} & & & 512 & \\ \hline \multirow{2}{*}{FC2} & & & 512 & \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Configuration of student network. Each convolutional layer and fully connected layer was followed by ReLU, which is not shown for brevity. Following the results of Table 3, Conv5 of the teacher network is a better choice to bind Conv3 of the student network for training. Based on the selection of Conv3 of the student network, we tested the binding of each layer of the teacher network to verify if the layer binding configuration (teacher, student) = (Conv5, Conv3) is the best choice. Table 3 shows other alternatives by (X, Conv3). The variable X indicates other possible convolutional layers. It was found that the highest classification accuracies of the PPMI and UIUC-Sports datasets support the configuration (Conv5, Conv3). It was also interesting to see that all results of the ASL framework outperform the baseline method for the student network learning. The classification accuracies are reported in the first two rows of Table 4. Compared to the baseline method--Student Only, at most, 12.6%, 5.1%, and 6.5% improvement of classification accuracy was achieved using the three datasets, respectively. In fact, all classification accuracies were boosted by ASL in this experiment. Note that although the accuracies of the teacher network decreased, it does not matter at all, since our goal was to train the student network. Once the training by ASL is complete, the teacher network is discarded. Only student network is used. Table 5 shows the results of another configuration of binding Conv3 and FC1 of the student network.