# Automated Car Damage Assessment Using Computer Vision: Insurance Company Use Case
## 5 Results and Discussion  
### Isolated YOLOv5 vs. YOLOv5 Ensemble  
Once we obtained the results of Experiment 1 shown in Table 2, we focused on using the precision value to compare all the trained models and the proposed set. Although the recall and AUC values allowed us to determine the performance of a model, the precision value in an industrial setting is more important because low precision generates costs to companies in terms of time and effort when checking for false positives. In Figure 5, we \begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Dataset** & **Size** & **Categories** & **Task** & **Appraiser** \\ \hline [PERSON] et al. [48] & 326 & N/A & D & N \\ \hline [PERSON] et al. [49] & 533 & 2 & C & N \\ \hline [PERSON] et al. [50] & 600 & 3 & C & N \\ \hline [PERSON] [51] & 1007 & 4 & C & N \\ \hline [PERSON]. [52] & 1077 & 7 & C, D & N \\ \hline [PERSON] [2] & 1150 & N/A & C & N \\ \hline [PERSON] [47] & 1150 & 8 & C & N \\ \hline [PERSON] [27] & 1439 & 4 & D, C & N \\ \hline [PERSON] [53] & 2100 & N/A & S, C & N \\ \hline [PERSON]. [46] & 2822 & 5 & S & N \\ \hline [PERSON] [26] & 3092 & 7 & D, C & N \\ \hline [PERSON] et al. [45] & 3818 & 3 & D & N \\ \hline [PERSON] [4] & 4000 & 6 & C, D, S, SOD & N \\ \hline Ours private dataset & 10,360 & 8 & C, D & Y \\ \hline Ours public dataset & 108 & 8 & C, D & Y \\ \hline \hline \end{tabular} \(\mathbb{C}\): damage classification, D: damage detection, S: damage segmentation, SOD: salient object detection, N/A: information not provided by the authors, N/Y: whether the dataset was created in the context of an appraisal.  
\end{table} 
Table 3: Comparison of vehicle damage datasets. can see the precision of all the models used together with the ensemble of models in order to highlight the precision value obtained by the ensemble. We performed an additional evaluation utilizing YOLOv5l_1280 (YOLOv5l), since it obtained one of the best AUC values among all models, and it was the proposed ensemble to identify the false positive rate. For this, we evaluated these models on our false positive set, and we show the results in Table 4 including the AUC vs. IoU value, which is the average of the AUC values over the set IoU range. We can observe that although the YOLOv5l model has a higher AUC compared to the ensemble, the false positive rate of YOLOv5l is very high. For this reason, this model is not feasible for the damage verification process in a real scenario. In contrast to YOLOv5l, the proposed ensemble manages to keep the false positive rate at 9%. In addition, we evaluate YOLOv5l using different confidence score thresholds against the ensemble model. To gain detailed insight into the behavior of each model, each metric is derived based on a range of IoU values at the time we performed the evaluation. We visualize the results in Figure 6, where we can see that the ensemble implementation achieves intermediate results across all the metrics, thus maintaining a balance between precision and recall. In order to extend the use of our methodology, we applied the ensemble to the YOLOv8 models and carried out the evaluation presented in Figure 7 on dataset 2. This evaluation allows us to present an alternative to the use of YOLOv5 in case a lower accuracy is required. Figure 5: Precision achieved by the models individually vs. the ensemble. Figure 6: YOLOv5l thresholds vs. ensemble. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Models** & **FP Rate** & **AUC vs. IoU** & \(mAP_{\text{30}}\) & **Inference Time (CPU)** \\ \hline YOLOv5l 10\% & 69\% & 0.2779 & 7.20\% & 1700 ms \\ \hline DCN+ 64\% & 30\% & 0.1541 & 4.33\% & 12,000 ms \\ \hline DCN+ 76\% & 8\% & 0.067 & 2.17\% & 12,000 ms \\ \hline Ensemble faster R-CNN & 3.3\% & 0.0784 & 1.36\% & 3540 ms \\ \hline Ensemble YOLOv8 (ours) & 16\% & 0.1849 & 4.14\% & 1110 ms \\ \hline Ensemble YOLOv5 (ours) & 9\% & 0.1507 & 3.77\% & 2820 ms \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Metrics summary of the models for dataset 2.