# Improving Deep Mutual Learning via Knowledge Distillation
## 3 Results and Discussion
### Experiment on CIFAR-100  
We use three experimental scenarios to demonstrate experimental results on the CIFAR-100 dataset. The first scenario uses the same type of architecture network for the student network, while for a teacher network, we use a slightly more complex type of architecture in terms of the number of parameters, as shown in Table 2, the accuracy results of our approach outperform the existing methods, except the case where Resnet \(32\times 4\) as a teacher and Resnet\(8\times 4\) as a student shows better accuracy in CRD+KD for being 75.53%. Meanwhile, the comparison between the two methods that we propose shows that FDDML dominates over HDDML. On the CIFAR-100 dataset, learning between two students together improves accuracy with the support of directions from knowledge transfer by a teacher to increase the accuracy of these students. When compared with KD and DML, the accuracy of our method far exceeds them, and even the accuracy of the students can exceed the accuracy of the teacher. Next, in the second scenario, we use a method similar to the one in the original DML test, namely using different student network architectures whose accuracy results are shown in Table 3, it indicates a significant increase in accuracy, such as in the test using WRN-40-2 as a teacher and MobileNetv2 and Resnet32 as the first and second students, with an accuracy of 73.85% for students. Note that the accuracy of the second student \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline  & & \multicolumn{6}{c}{**Network Architecture**} \\ \hline Teacher & Resnet32 & Resnet56 & Resnet110 & Resnet110 & WRN-40-2 & Resnet32x4 & WRN-40-2 & Resnet56 \\ Student 1 & Resnet20 & Resnet20 & Resnet56 & Resnet20 & MobileNetv2 & Resnet8x4 & WRN-16-2 & Resnet44 \\ Student 2 & Resnet20 & Resnet20 & Resnet56 & Resnet20 & MobileNetv2 & Resnet8x4 & WRN-16-2 & Resnet44 \\ \hline \hline \multicolumn{10}{c}{Test Accuracy (\%)} \\ \hline Teacher & 70.12 & 72.04 & 73.79 & 73.79 & 75.3 & 76.95 & 75.3 & 72.04 \\ Student 1 & 69.05 & 69.05 & 72.04 & 69.05 & 64.55 & 71.14 & 73.15 & 71.54 \\ Student 2 & 69.05 & 69.05 & 72.04 & 69.05 & 64.55 & 71.14 & 73.15 & 71.54 \\ KD [8] & 70.5 & 70.33 & 74.41 & 70.88 & 68.51 & 75.24 & 74.76 & 73.87 \\ AT [19] & 69.42 & 70.91 & 74.54 & 69.95 & 59.27 & 72.46 & 73.02 & 71.46 \\ SP [13] & 70.16 & 69.71 & 73.83 & 69.65 & 64.82 & 73.12 & 73.74 & 73.54 \\ CC [14] & 68.78 & 68.45 & 71.89 & 69.06 & 65.91 & 72.4 & 72.97 & 72.06 \\ CRD [10] & 70.95 & 70.63 & 75.1 & 70.86 & 69.62 & 75.14 & 75.42 & 74.36 \\ CRD + KD [10] & 71.16 & 71 & 74.99 & 70.92 & 69.78 & **75.53** & 75.54 & 74.43 \\ \hline DML: [22] & & & & & & & & \\ \hline Student 1 & 69.98 & 69.98 & 73.39 & 69.98 & 67.31 & 71.58 & 73.74 & 72.14 \\ Student 2 & 69.71 & 69.71 & 73.26 & 69.71 & 66.88 & 71.72 & 73.96 & 72.8 \\ \hline FDDML: & & & & & & & & \\ \hline Student 1 & 70.6 & 70.9 & 75 & **71.04** & **70.63** & 75.27 & 75.21 & 74.31 \\ Student 2 & **71.17** & **71.16** & 74.9 & 70.85 & 69.88 & 74.99 & **75.75** & **74.49** \\ \hline HDDML: & & & & & & & & \\ \hline Student 1 & 70.71 & 70.87 & 73.7 & 69.98 & 68.75 & 73.81 & 74.56 & 74.31 \\ Student 2 & 70.18 & 68.46 & **75.11** & 70.79 & 70.14 & 74.52 & 75.43 & 72.88 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Comparing resultstop-1 accuracy on the CIFAR-100 datasets with twin student network. outperforms the accuracy of FDDML and DML, where only the maximum accuracy for the second student is shown, to be 73.0% and 71.84%, respectively. This test also shows better accuracy when compared with the same network architecture of students in Table 2, where WRN-40-2 as a teacher and MobileNetv2 as the first and second students can only obtain the highest accuracy on the first and second FDDML students, namely 70.63% and 69.88%.