## 2 Materials and Methods
### Performance Evaluation Metrics  
We use precision (P), recall (R), and mean average precision at the threshold of 50 (mAP@50) to evaluate the performance of the proposed models as follows.  
Figure 4: Flowchart of the proposed ensemble model.  
Figure 5: Flowchart of the proposed test time augmentation.  
Precision (P) is the ratio of correctly predicted positive observations to the total predicted positives as given in Equation (2):  
\[\text{Precision }(\text{P})=\frac{\text{TP}}{\text{TP}+\text{FP}}, \tag{2}\]  
where TP (true positive) is the number of correctly recognized bad traffic signs. FP (false positive) is the number of instances where the model incorrectly identifies an object as a bad traffic sign when it is not.  
Recall (R) is the ratio of correctly predicted positive observations to all the observations in the actual class. It is given in Equation (3):  
\[\text{Recall }(\text{R})=\frac{\text{TP}}{\text{TP}+\text{FN}}, \tag{3}\]  
where FN (false negative) is the number of bad traffic signs that are present but not identified by the model.  
Mean average precision mAP@50 is calculated by averaging the precision values at different recall levels across all object classes. It is calculated in Equation (4):  
\[\text{mAP}= \frac{1}{\text{C}}\text{$\sum_{i=1}^{\text{c}}$}\text{AP}_{i}, \tag{4}\]  
where AP\({}_{i}\) is the average precision (AP) value for all class i. C is the total number of classes and average precision (AP) summarizes the precision-recall performance of the model across all the recall levels ranging from 0 to 1.  
In addition to the above, we use the k-fold cross-validation technique to evaluate the stability of the metrics and provide a reliable performance evaluation. In k-fold cross-validation [29; 30], a dataset is split into k subsets of equal size. The model is then trained on k\(-\)1 folds and validated on the remaining fold. The process is repeated k times with a different fold, which serves as the validation set. The performance metrics are averaged across all folds to obtain the cross-validated performance. The mean of these performance metrics provides an estimate of the model's general performance. The standard deviation indicates the variability or consistency of the model's performance across different data splits. This technique helps to assess how well the model is likely to generalize to unseen data and reduces bias in the performance evaluation.  
In this study, we use k = 5 and split the dataset into five (5) folders. The test set has 210 images while the remaining 1890 images are randomly split into five (5) folders of 378 images each. One folder is used for validation and the remaining folders are used for training. Performance metrics (precision, recall, and mAP@50) are averaged over all folds. In addition, we calculate the mean, variance, and standard deviation to gain further insight into the behavior of the models.  
Mean, also called average, is the sum of all the values in a dataset divided by the number of values. It is calculated in Equation (5).  
\[\text{Mean}(\text{\mu})=\frac{1}{\text{N}}\text{$\sum_{i}^{\text{N}}$}\text{ x}_{i} \tag{5}\]  
where (N) is the number of observations and xi represents each individual observation.  
Variance measures how much the values in a dataset differ from the mean. In k-fold cross-validation, it measures how consistent a model's performance is across different data splits (folds). It is calculated in Equation (6).  
\[\text{Variance }\left(\sigma^{2}\right)=\frac{1}{\text{N}}\text{$\sum_{i=1}^{ \text{N}}$}(\text{x}_{i}-\text{\mu})^{2} \tag{6}\]Standard deviation measures the amount of variation or dispersion in a dataset. It offers a clear measure of how much a model's performance varies or remains consistent across the different folds. It is the square root of the variance as given in Equation (7).  
\[\text{Standard deviation}= \sqrt{\frac{1}{\text{N}}\sum_{i=1}^{\text{N}}\left(\text{x}_{i}- \mu\right)^{2}} \tag{7}\]