## 4 Evaluation
\begin{table}
\begin{tabular}{c c c c c} \hline \hline  & \
\begin{tabular}{c} **Mold 1** \\ **(Original Dataset)** \\ \end{tabular} &
\begin{tabular}{c} **Model 2** \\ **(Augmented Dataset)** \\ \end{tabular} \\ \hline mAP @ IoU = [0.5:0.05:0.95] & 0.486 & 0.524 \\ AP @ IoU = 0.5 & 0.691 & 0.776 \\ AP @ IoU = 0.75 & 0.527 & 0.562 \\ AP @ distant & 0.305 & 0.402 \\ AP @ medium & 0.473 & 0.509 \\ AP @ near & 0.645 & 0.633 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Evaluation results. Figure 5: Example of improved sensitivity and specificity. (**a**) Original image with marked region containing the considered objects (red rectangular). (**b**) Magnified predictions of Model 1 including one FP and two FN predictions. (**c**) Magnified predictions of Model 2 with perfect predictions. Even though the more precise detection of bounding boxes may not seem particularly relevant at first, it is of great importance for the distance estimation based on the bounding box sizes, as it is carried out in the SMART/SMART2 system using the artificial neural network DisNet [13]. Accordingly, more precise bounding boxes have a direct influence on the accuracy of distance estimation of objects. For example, the ground truth distance between the camera mounted on the train and detected object (car) in Figure 6 was 93 m. The estimated distance based on the imprecise car bounding box resulting from Model 1 was 125 m, while the estimated distance based on the precise bounding box resulting from Model 2 was 104 m.  
Overall, the newly trained Model 2 was able to correctly detect 221 of the 252 objects present in the test dataset. In all these cases, the IoU was above 0.5 and the object was classified as true positive (TP). Accordingly, 31 FN occurred in the test dataset. Furthermore, 35 FP occurred, resulting in a improved precision of 0.863 and a recall of 0.877. Thus, the harmonic mean (F1 score) for this model is 0.870, as can be seen in the following Table 3.  
In the railway application, bearing in mind the long braking distances of a train, early obstacle detection in long-range distances is of great importance. Based on the assumption that small objects in SMART test dataset images correspond to distant objects (small with respect to the number of pixels they are covering in the image), the evaluation was performed on how the object detection model re-trained on the augmented dataset, with respect to the ability to detect distant objects. First, all objects were assigned to one of the three categories, \"near\", \"medium\" and \"distant\", so that three equally distributed classes were created. The threshold between \"near\" and \"medium\" category was \(135^{2}\) pixels per object, which corresponds to a distance of approximately 100 m. The second threshold between \"medium\" and \"distant\" categories was \(46^{2}\) pixels per object, which corresponds to a distance of approximately 250 m. As Table 2 shows, training with the augmented dataset resulted in an improved AP for the \"distant\" and the \"medium\" objects. While in the categories \"medium\" and \"near\", only a minimal improvement or worsening of the AP value emerges, in the category \"distant\" a significant increase of the AP by 9.7% has been achieved. Due to its high importance for obstacle detection in railways, this improvement of detection of distant objects by augmentation of training dataset with synthetic images was one of the primary goals of the presented work. The decrease of 0.012 in the AP value for near objects is probably due to the fact that most of the objects in the synthetic images were inserted in the medium or distant distance range. This resulted in  
Figure 6: Example of improved object bounding box coordinates. (**a**) Original image with marked region containing the considered object (red rectangular). (**b**) Magnified predictions of Model 1. (**c**) Magnified predictions of Model 2.