# Improving Deep Mutual Learning via Knowledge Distillation
## 3 Results and Discussion
### Experiment on TinyImageNet  
Then, to see if our method is robust or not, we conducted a test with a larger dataset using TinyImageNet with the original size of 64 \(\times\) 64 images. In this test, we use two training scenarios. In the first scenario, we use the dataset of the original size, as shown in the test results as follows. In contrast to the testing using the CIFAR-100 dataset, in the second scenario, we show the accuracy of our approach in Table 4, where the highest accuracy is generally by HDDML, and one CRD+KD [10] process is dominant over other tests, that is, when VGG16 is the teacher and VGG8 is a student, which is marked in bold, followed by the second-highest accuracy marked with underline. It can be seen that the performance of our proposed methods can exceed the accuracy produced by the teacher network, also shown in the existing method [10], which is able to beat the accuracy of the teacher network, but it is not better than HDDML, one of the two methods we propose. Then, to ensure the performance of our proposed method, we tested it by downsampling the TinyImageNet datasets to become 32 \(\times\) 32. The downsampled images carried out in this experiment are intended to demonstrate the reliability of our proposed method in adapting to different conditions and to illustrate the results of comparisons with similar methods [11] as a baseline. Although it appears that the student network accuracy performance has decreased due to the changes in image size and image resolution compared with the cases when trained with the original image size, it can adapt well when compared with the baseline. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multicolumn{7}{c}{**Network Architecture**} \\ \hline Teacher & Resnet50 & Resnet50 & Resnet50 & Vgg13 & Vgg16 & Vgg16 \\ Student 1 & Resnet34 & Resnet18 & Vgg16 & Vgg8 & Vgg13 & Vgg8 \\ Student 2 & Resnet34 & Resnet18 & Vgg16 & Vgg8 & Vgg13 & Vgg8 \\ \hline \multicolumn{7}{c}{Test Accuracy (\%)} \\ \hline Teacher & 55.34 & 55.34 & 55.34 & 48.1 & 49.2 & 49.2 \\ Student 1 & 37.94 & 36.04 & 49.2 & 41.68 & 48.1 & 41.68 \\ Student 2 & 37.94 & 36.04 & 49.2 & 41.68 & 48.1 & 41.68 \\ \hline KD [8] & 55.26 & 55.72 & 53.92 & 44.42 & 52.4 & 44.68 \\ AT [19] & 50.13 & 50.22 & 45.68 & 44.84 & 48.92 & 43.18 \\ SP [13] & 55.38 & 55.68 & 54.28 & 45.06 & 51.88 & 44.66 \\ CC [14] & 50.9 & 49.82 & 48.72 & 42.1 & 48.08 & 41.68 \\ CRD [10] & 57.48 & 55.92 & 55.82 & 45.12 & 51.1 & 45.38 \\ CRD + KD [10] & 57.14 & 56.94 & 55.74 & 46.08 & 51.98 & **46.88** \\ \hline DML: [11] & & & & & & \\ Student 1 & 55.1 & 54.86 & 53.88 & 44.94 & 52.84 & 44.94 \\ Student 2 & 55.13 & 54.92 & 53.26 & 45.16 & 53.76 & 45.16 \\ \hline FDDML: & & & & & & \\ Student 1 & 57.33 & 56.58 & 56.12 & 45.02 & 53.22 & 46.3 \\ Student 2 & 57.98 & 55.88 & 56.54 & 44.52 & 53.1 & 44.36 \\ \hline HDDML: & & & & & & \\ Student 1 & 57.05 & 55.1 & 53,66 & 46.06 & 52.32 & 46.26 \\ Student 2 & **58.27** & **57.48** & **56.68** & **46.26** & **54.24** & 45.74 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Comparing results top-1 accuracy on the TinyImageNet datasets with 64-Image size. Table 5 shows that the method we propose is still superior to the accuracy of teachers in general. The last scenario in the experiment is that we adapt our proposed method to the transfer of knowledge of student network trained using a downsampled image from the original image for being an example of a previously trained exemplary teacher. This test involves variations in temperatures and variations in batch sizes. In Table 6, the highest accuracy is marked in bold for each method. Here, we use two types of networks, namely, Resnet50 as a teacher and WRN-16-2 as a student, with an accuracy of 55.34% and 28.32%, respectively. The results are the accuracy of the teacher model using the original image size of 64 \(\times\) 64 and the accuracy of the independent student using the downsampled image size of 32 \(\times\) 32 image.