# Efficient Knowledge Distillation for Brain Tumor Segmentation
## 5 Ablation Study
### Comparison with AT  
We compared our approach with AT. We performed this part of the experiment using the UNet and the DeepResUNet, and extracted feature maps at the same locations in the network to ensure the fair implementation of different distillation methods. We visualized their output, as shown in Figure 3. It can be seen intuitively that CD corrected some errors of AT when segmenting.  
Table 3 shows the comparison results. Obviously, in contrast to AT, our distillation method CD improved results in every metric. It is worth mentioning that the Dice score of CD in TC improved from 85.66 to 87.34, and the Hausdorff distance of CD in TC improved from 1.64 to 1.55.  
\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Method** & \multicolumn{3}{c}{**Dice Score (\%)**} & \multicolumn{3}{c}{**Hausdorff Distance**} & \multicolumn{1}{c}{**\#Params (M)**} \\ \hline  & **ET** & **WT** & **TC** & **ET** & **WT** & **TC** & \\ \hline \multicolumn{8}{c}{Teachers} \\ \hline T1: UNet & 78.83 & 85.69 & 85.81 & 2.77 & 2.57 & 1.66 & 39.4 \\ T2: AttUNet & 78.79 & 85.31 & 85.82 & 2.83 & 2.65 & 1.64 & 39.7 \\ \hline \multicolumn{8}{c}{Students and their performances by our approach} \\ \hline S1: DeepResUNet & 78.44 & 85.19 & 85.36 & 2.79 & 2.60 & 1.68 & \\ S1+T1 (ours) & 79.69 & 85.91 & **88.20** & **2.71** & **2.53** & **1.52** & 32.6 \\ S1+T2 (ours) & 80.22 & **86.49** & 87.14 & - & - & 1.61 & \\ \hline S2: UNet++ & 78.47 & 85.16 & 85.60 & 2.78 & 2.60 & 1.61 & \\ S2+T1 (ours) & 79.13 & 85.62 & 87.64 & 2.73 & 2.55 & - & 9.2 \\ S2+T2 (ours) & **80.48** & 86.46 & 87.57 & - & - & - & \\ \hline \hline \end{tabular}
\end{table}
Table 1: The results of our experiments on different combinations of networks on BraTS2018. The bold values are the highest Dice scores and lowest Hausdorff distances in their columns. When the segmentation index of the teacher was lower than that of the student, theoretically, knowledge distillation was not applicable to this situation.  
\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline
**Models** & \multicolumn{3}{c}{**GFLOPs \#Params**} & \multicolumn{3}{c}{**Dice Score (\%)**} & \multicolumn{3}{c}{**Hausdorff Distance**} \\ \hline  & & & **ET** & **WT** & **TC** & **ET** & **WT** & **TC** \\ \hline UNet & 18.03 & 39.4M & 78.83 & 85.69 & 85.81 & 2.77 & 2.57 & 1.66 \\ DeepResUNet & 23.27 & 32.6M & 78.44 & 85.19 & 85.36 & 2.79 & 2.60 & 1.68 \\ AttUNet & 18.47 & 39.7M & 78.79 & 85.31 & 85.82 & 2.83 & 2.65 & 1.64 \\ UNet++ & 13.52 & 9.2M & 78.47 & 85.16 & 85.60 & 2.78 & 2.60 & 1.61 \\ HybridResUNet & 22.0 & 31.6M & 76.81 & 84.08 & 86.88 & 2.83 & 2.64 & 1.62 \\ UNet++ (ours) & **13.52** & **9.2M** & **80.48** & **86.46** & **87.57** & - & - & - \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparison with current popular methods.  
\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline
**Method** & \multicolumn{3}{c}{**Dice Score (\%)**} & \multicolumn{3}{c}{**Hausdorff Distance**} \\ \hline  & **ET** & **WT** & **TC** & **ET** & **WT** & **TC** \\ \hline T: UNet & 78.83 & 85.69 & 85.81 & 2.77 & 2.57 & 1.66 \\ S: DeepResUNet & 78.44 & 85.19 & 85.36 & 2.79 & 2.60 & 1.68 \\ \hline S+AT & 78.01 & 84.95 & 85.66 & 2.86 & 2.66 & 1.64 \\ S+CD (ours) & **78.83** & **85.44** & **87.34** & **2.78** & **2.60** & **1.55** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparison with AT on the BraTS2018.