# Identification of NLOS and Multi-Path Conditions in UWB Localization Using Machine Learning Methods
## 7 Evaluation Results
### Performance Comparison of the Three Classifiers Using the Macro-Averaging F1-Score as a Metric  
To give an overview of the actual state in each trial conducted 10 times, we use the macro-averaging F1-score to compare the performance of the three classifiers in this section. The F1-score, i.e., in contrast to the overall accuracy score in the confusion matrix (Section 7.2), is extensively used to quantify the classifier's performance in ML because it takes into account both the precision and recall to compute the decisive score [11; 49]. It is the harmonic mean of the precision and recall, which can be expressed for a binary classification as:  
\[F1=2\cdot\frac{Precision\cdot Recall}{Precision+Recall} \tag{4}\]  
For multi-class classification, there are two typical ways (macro-averaging and micro-averaging) to compute the overall F1-score for a classifier [49]. We applied the macro-averaging technique in our evaluation, which treated all the classes equally. Based on the mentioned macro-averaging F1-score, Figure 7 compares the experimental evaluation results of the three classifiers in two test environments (the scenario that was the same as vs. different from the training state) defined in Section 4.2.2. The solid lines denote the results of the test dataset when the data in the test state were collected in a different environment from the training state. The dotted lines show the results of the test dataset achieved from the same scenario as the training state. In general, a significant gap between the two test scenarios was discovered in the experimental evaluation results of the three classifiers (Figure 7). The figure reveals that impressive outcomes  
Figure 6: Comparison of the overall accuracy, training, and test times for the scaled vs. unscaled features in the preprocessing phase. (**a**) Overall accuracy comparison for scaled Vs. unscaled features (**b**) Processing time comparison for scaled and unscaled features. were achieved in RF and MLP classifiers when the test and training states were conducted in the same environments. However, the performance of SVM was relatively low in this scenario compared to RF and MLP. In contrast, the performance of all classifiers was notably degraded when the test environment was different from the training state. Specifically, the resultant mean of the SVM classifier based on the macro-averaging F1-score was reduced from 0.83 (when the training and test scenarios were the same) to 0.75 (when the training and test scenarios were different). Similarly, the performance of the RF decreased from 0.92 to 0.73. The MLP classifier showed a degradation from 0.91 to 0.72. The results showed that an immediate conclusion and judgment on the choice of classifiers based on a single test scenario or environment could be misleading. The core reason was that the estimated precision and characteristics of the measured wireless signal, i.e., the UWB ranging data in our evaluation, are affected by a variety of physical impacts in indoor environments, as previously mentioned several times in this manuscript. It is interesting to see that SVM stood out to be the best classifier in our evaluation when the test scenario was different from the training state (Figure 7). However, it had the poorest performance among the three classifiers when the same environments of training and test were applied. Moreover, the outcomes of SVM were consistent in all of the evaluated trials in both of the scenarios. Indeed, the outcomes of RF were also quite stable across the whole trials compared to the MLP classifier. However, many fluctuations were evident in the predicted outputs of the MLP, especially in the condition where the training and test environments were different. In all experiments, the lowest F1-score was 0.69 in Trial No. 4 when MLP was used as a classifier, and the highest score reached 0.92 using RF.