## 4 Training and Evaluation of Threat-Detection AI Model
### Results  
The training and validation results for the object-detection models (i.e., YOLOv4, DINO) to detect the external interference threat are reported quantitatively and graphically. The training loss and validation mAP curves against the number of epochs for each model are presented in Figure 5. For both models, the training loss follows the negative exponential curve (i.e., the loss is decreasing as the number of epochs increases), which suggests a normal training process to learn the detection task. Similarly, the validation mAP follows the positive exponential trend (i.e., the mAP increases as the number of epochs increases), indicating that there is no overfitting during the training process. For the YOLOv4 model, the training loss was reported to be settled at 5.3, while the validation mAP was reported to be 0.631 after 1000 epochs. For DINO, the training loss converged at 9.15, while the validation mAP settled at 0.671 at the end of 200 epochs. In addition to the validation performance, the trained models were also evaluated against an unseen test dataset, and the results were compared with the validation to ensure that the models learned generalized features and no overfitting occurred during the training process. Table 4 presents the detailed categorywise quantitative results for the validation and test mAP for the last training epoch models. The results indicate that the validation and test performances of the trained models were relative to each other, demonstrating generalized training. DINO emerged as the best model among all with a validation mAP of \begin{table}
\begin{tabular}{l c c} \hline \hline
**Parameter** & **YOLOv4** & **DINO** \\ \hline Backbone & CSPDarkNet53 & Fan Tiny \\ Training Epochs & 1000 & 200 \\ Batch Size & 8 & 4 \\ Base Learning Rate & 1e\({}^{-4}\) & 2e\({}^{-4}\) \\ Number of Classes & 13 & 13 \\ Optimizer & Adam & – \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Model training parameters. 0.671 and a test mAP of 0.712. In terms of separate categories, the best validation mAP was observed for the tractor (i.e., 0.829), boring_rig (i.e., 0.823), and auger (i.e., 0.822) categories. On the other hand, the lowest mAP was observed for the clay_delver (i.e., 0.256) and bobcat (i.e., 0.392). One potential reason for the lower mAPs for clay_delver could be associated with images related to these classes in the dataset. Enhancing the dataset with more images related to these categories in the future will potentially increase the performance. Although the superior performance was achieved by the DINO model, given the requirement of deploying a solution on a limited-computational-power edge computer (i.e., NVIDIA Jetson Nano), a simpler model of YOLOv4 optimized with TensorRT for deployment was selected. The YOLOv4 model achieved a validation mAP of 0.631 and a test mAP of 0.618. Figure 6 graphically compares the categorywise mAPs for the validation and test datasets. Figure 5: Training loss and validation mAP curves for the trained object-detection models. Figure 7 presents a few selected images from the test dataset, where the YOLOv4 model was able to correctly predict the threat type and where the model failed or had missing predictions. It can be observed that the model was able to accurately predict for the first row of images with the boring rig, truck, tractor, and post driver categories. In terms of false/missing predictions, the model resulted in multiple bbox predictions for the same object, which is a common issue and can be resolved by simply picking the best scoring bbox and rejecting the others. Further, the model, sometimes, partially detected the objects in the image and missed the other object, for example, in Figure 7, \"tractor\" and \"post driver\" were accurately predicted, while the \"person\" category was missed. This observation suggests adding more images related to missed categories in the training dataset.