# Bolt Loosening Detection Using Key-Point Detection Enhanced by Synthetic Datasets
## 3 Results
### Training Results
#### 3.1.3 Error Analysis  
We collected the results of the five models that detected the correct objects and computed the error of key-point detection. We visualized the manually labeled correct detection boxes and key points, along with the predicted boxes and predicted points detected by the models, on one image at the same time. The results in Figure 20 can be obtained in the case of larger and smaller detection errors, respectively. The hollow dots represent the manually labeled key points, and the solid dots represent the model-detected key points; the purple ones are the manually labeled detection boxes, and the blue ones are the model-detected boxes with the model-detected confidence levels on them.  
Figure 19: A wrong detection result obtained by Keypoint-RCNN.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Training Set** & **Bolts** & **Detect** & **Kpts** & **Detect** \\ \hline
200 real + 1200 synthetic & & 50 & & 50 \\
100 real & 50 & 49 & 50 & 49 \\
100 real + 1200 synthetic & & 50 & & 50 \\
1200 synthetic & & 24 & & 21 \\ \hline
200 real + 1200 synthetic (Keypoint-RCNN) & 50 & 50 & 50 & 45 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Evaluation results of bolt and bolt corner point detection for the test set.  
In order to be able to evaluate quantitatively, we used the normalized mean error (NME) to evaluate the performance of different models. The error was calculated as Equation (9). To represent the error more intuitively, we also used Equation (10) to convert the NME into an angular form.  
\[\text{NME }=\frac{\sum\sqrt{\left(x_{i}^{*}-\ x_{i}^{\prime}\right)^{2}+\left(y_{i }^{*}-\ y_{i}^{\prime}\right)^{2}}}{\sum\ r_{i}} \tag{9}\]  
where \(\left(x_{i}^{*},y_{i}^{*}\right)\) and \(\left(x_{i}^{\prime},y_{i}^{\prime}\right)\) are the manually labeled and model-detected coordinates of the corner points; \(r_{i}\) is the distance from the center of the bolt to each corner point.  
\[\delta=\frac{\text{NME}}{\pi}\times 180^{\circ} \tag{10}\]  
After calculating the recognition errors for individual images, we calculated the average error of the model for the overall test set, as shown in Table 6. From this, the following conclusions can be drawn:  
1. The model trained on the \"200 real + 1200 synthetic\" dataset showed the best performance with not only the smallest error but also the smallest error variance, which is a significant performance improvement compared to the \"200 real\" model with only real images;
2. The performance of the \"100 real + 1200 synthetic\" model was close to that of the \"200 real\" model, while the \"100 real\" model showed poor recognition accuracy;
3. Even though the \"1200 synthetic\" model performed poorly in terms of overall precision, the key-point detection in the images where the bolts were successfully detected did not show large errors, and the accuracy was still relatively good.
4. Compared with Keypoint-RCNN, YOLOv7-pose showed better performance with a smaller error and much smaller error variance.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Training Set** & **NME** & **\$(\text{\textasciitif{}})** & **Var** \\ \hline
200 real & 0.043785 & 2.510472 & 1.951742 \\
200 real + 1200 synthetic & 0.033903 & 1.943473 & 1.304070 \\
100 real & 0.061144 & 3.514130 & 9.047428 \\
100 real + 1200 synthetic & 0.047353 & 2.717504 & 3.644367 \\
1200 synthetic & 0.070147 & 4.038014 & 15.20069 \\ \hline
200 real + 1200 synthetic & & & \\ (Keypoint-RCNN) & 0.055395 & 3.247408 & 24.88260 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Evaluation results of bolt and bolt corner point detection for the test set.  
Figure 20: Detection box and key-points visualization (**a**) Detection results of the “100 real” model; (**b**) Detection results of the “200 real + 1200 synthetic” model.