# Vision Transformer in Industrial Visual Inspection
## 4 Experiments and Results
### Results  
Figure 9 shows the development of the mAP with at least 50% IoU between predicted bounding boxes and ground truth on the validation set (\(mAP_{50,val}\)) of the sheet metal use case. The strong oscillation in the performance of the transformer models is present during training of all full image use cases and makes it hard to determine if the models have converged. This instability is caused by the low batch size utilized with these models due to their high memory requirements, and further amplified by the large size of the input images.  
Figure 10 visualizes the \(mAP_{50,val}\) during training of the sliding window version of the wooden load space use case. It is clearly visible that the stability of the training process is positively influenced by the possibility to use higher batch sizes due to smaller input images. Retina Net and Retina Swin converge the fastest approximately at epoch 50, followed by DDETR, Yolo, and F-RCNN around epoch 110 and DETR at epoch 120. Retina Next seems to keep improving until epoch 150 and could possibly still benefit from continued training based on the curve's upward gradient. This shows on the one hand how effective the improvements of DDETR on DETR are in low-data environments, and on the other hand that DDETR has a comparable convergence speed to commonly used CNN models, even  
\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline  & **Model** & **Epoch** & **Batch Size** & \(mAP_{50,test}\) & \(F1\)-\(Score_{initial}\) & \(F1\)-\(Score_{adjusted}\) & **IoU** \\ \hline \multirow{5}{*}{
\begin{tabular}{c} **V1-Score** \\ **V2-Score** \\ **V3-Score** \\ **V4-Score** \\ **V5-Score** \\ **V6-Score** \\ **V7-Score** \\ **V8-Score** \\ **V9-Score** \\ **V10-Score** \\ **V11-Score** \\ **V12-Score** \\ **V13-Score** \\ **V14-Score** \\ **V15-Score** \\ **  
with lower batch size due to high memory requirements. Retina Swin converges as fast as the fastest pure CNN in Retina Net with even better precision.  
In order to determine the best weights for each model, checkpoints were saved every 10 epochs from epoch 50 onwards and evaluated on the test set. The checkpoint with best \(mAP_{50,test}\) at the lowest epoch is selected for the following comparison. As neither recall nor precision can be neglected in industrial visual inspection, we choose their harmonic mean, also known as \(F1\)-\(Score\), as the performance metric.  
For the initial \(F1\)-\(Score\) (\(F1\)-\(Score_{initial}\)) a prediction is considered to be correct, if it achieves 50% IoU and prediction confidence, which are the usual values used on benchmark datasets. Compared to benchmarks such as COCO, perfect alignment of predicted bounding boxes is not as important in real world applications as long as the targets are recognized.  
Figure 10: \(mAP_{50,val}\) development during training of the wooden load space use case.  
Figure 9: \(mAP_{50,val}\) development during training of the sheet metal use case.  
Therefore, the reduction in the models' IoU threshold for the given task can improve \(F1\)-\(Score_{initial}\), in many cases without drawbacks to \(F1\)-\(Score_{adjusted}\). The adjusted IoU values for each model are listed in Table 3 in the last column. The prediction confidence correctness threshold is still kept at 50% to retain the same prediction quality as for \(F1\)-\(Score_{initial}\). In the following, we will analyze the performance of all models summarized in Table 2, use case by use case regarding \(mAP_{50,test}\), \(F1\)-\(Score_{initial}\) and \(F1\)-\(Score_{adjusted}\).