# Reverse Knowledge Distillation with Two Teachers for Industrial Defect Detection
## 4 Experiments
### Ablation Study  
Teacher networks are important for feature extraction of images, and in Table 5, we verified the effect of using pretraining and nonpretraining networks on the image-level and pixel-level anomaly detection results. From Table 5, it can be seen that using pretraining can improve the results by about 20 points, indicating that using pretraining can extract the features of the image well.  
In the attention mechanism module, we investigated the image feature channel division parameter r to see its effect on the anomaly detection results at the image-level and pixel-level, and the results are shown in Table 4. As can be seen from Table 6, the pixel-level results have a relatively small change in value as the r parameter changes. At the image-level anomaly detection results have some variation as the r parameter changes. It indicates that the division of image feature channels has some influence on the image-level anomaly detection results.  
In Table 7, the baseline represents the adoption of the original network, including one teacher and student network. New Student Network represents the replacement of the original student network with a new student network, which is designed by ourselves. Two Teachers Network means replacing a single teacher network with two teacher networks. Mscam represents the addition of two teacher network feature channels to the attention  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Div** & **2** & **4** & **6** & **8** \\ \hline Texture & 99.68/97.3 & 99.96/97.46 & 98.46/97.40 & 99.68/97.46 \\ Object & 97.58/97.99 & 99.12/98.20 & 99.15/98.27 & 99.18/98.27 \\ Average & 98.63/97.65 & 99.44/97.83 & 98.81/97.84 & 99.43/97.87 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Image-level and pixel-level anomaly detection results for the attention mechanism module with channel division parameter r.  
Figure 5: Histogram of sample abnormal scores. Blue indicates normal samples and yellow indicates abnormal samples. The horizontal axis indicates the abnormal scores and the vertical axis indicates the counts.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Two Teachers Network** & **Unpretrained** & **Pretained** \\ \hline Texture & 86.42/77.52 & 99.68/97.46 \\ Object & 71.55/79.83 & 99.18/98.27 \\ Average & 78.99/78.68 & 99.43/97.87 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Image-level and pixel-level anomaly detection results with teacher network using pretraining and nonpretraining.  
mechanism module separately. Iaff represents the feature fusion of the two-teacher network feature channels again.  
Both image-level and pixel-level anomaly detection results are improved by using different architectures for the teacher and student networks. This suggests that the powerful performance student network may cause overfitting and recover the anomalies as well.  
Better feature representation can be extracted using two teacher networks. In Table 6, it can observe that the two teacher network achieves good results in image-level anomaly detection, but the pixel-level anomaly detection results are almost unchanged.  
The use of an attention mechanism and feature fusion module enables better extraction of useful features. As can be seen from Table 6, the image-level and pixel-level anomaly detection results achieve a good result. The attention mechanism can extract better feature channels according to their importance, and the feature fusion module can assign higher weights to the teacher network that provides valuable features.