## 4 Performance Evaluation Results
### Confusion Matrices for the Combined Scenario
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Algorithm** & **Accuracy** & **Precision** & **Recall** & **Specificity** & **F1-Score** \\ \hline KNN & 0.7275 & 0.7102 & 0.7098 & 0.9494 & 0.7100 \\ SVM & 0.5418 & 0.5288 & 0.5032 & 0.8918 & 0.5157 \\ DT & 0.7280 & 0.7119 & 0.7107 & 0.9494 & 0.7113 \\ NB & 0.4693 & 0.4849 & 0.4297 & 0.8620 & 0.4556 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Evaluation Metric Results for 8-beam codebook size. Figure 11: KNN Confusion Matrix for the Combined Scenario with \(M=32\) and 80:20% split (Blue color represents correct classifications while orange represents misclassifcations; the darker the color tone, the higher the value). Figure 12a,b show the AUC-ROC curves for \(M=8\) for DT and NB, respectively. As highlighted in Section 3.4, the closer the ROC curve is to the upper left corner of the plot (or the closer the AUC value to 1), the higher the accuracy of the model's predictions. Also, an AUC = 0.5 (i.e., the back line on the ROC plots in Figure 12a,b with TPR = FPR) means that the model is only making random guesses. Therefore, comparing Figure 12a,b shows that DT is a better classifier than NB for the considered scenario/dataset. In Figure 12a for DT, the curves are closer to the upper left corner than those of NB in Figure 12b. Similarly, the AUCs and model operating points for beams 1-8 of DT are higher than those of NB. The DT's average AUC is 0.89596, while that of NB is 0.82487. We have only shown AUC-ROC curves for DT (best-performing algorithm) and NB (worst-performing algorithm) and for \(M=8\) only, due to space constraints. The analyses follow the same trends for KNN and SVM, and for \(M=16\) and \(M=32\). On the comparison with the state-of-the-art, in Table 7, we compare our KNN results with the Top-1 accuracy results of Table III in [16]. The results are comparable, and the differences are due to the following:  
1. The number of samples or datapoints used in both works are slightly different. For example, the datapoints used in [16] for Scenarios 1, 6, and 7 are 2667, 1011, and 897, respectively (as in Figure 1 of [16]), as against the corresponding 2441, 915, and 854 samples in the publicly-released dataset used in this work, as presented in Table 2 and available in [19]. The number of samples used for Scenarios 2 and 5 are unavailable in [16]. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Algorithm** & **Accuracy** & **Precision** & **Recall** & **Specificity** & **F1-Score** \\ \hline KNN & 0.4158 & 0.3828 & 0.3623 & 0.9552 & 0.3723 \\ SVM & 0.2260 & 0.1686 & 0.1527 & 0.8765 & 0.1603 \\ DT & 0.4360 & 0.4035 & 0.3798 & 0.9587 & 0.3913 \\ NB & 0.1873 & 0.2013 & 0.1798 & 0.8632 & 0.1900 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Evaluation Metric Results for 32-beam codebook size. Figure 12: AUC-ROC curves for (**a**) DT and (**b**) NB for the Combined Scenario with \(M=8\). \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Algorithm** & **Accuracy** & **Precision** & **Recall** & **Specificity** & **F1-Score** \\ \hline KNN & 0.5820 & 0.5462 & 0.5407 & 0.9543 & 0.5435 \\ SVM & 0.3571 & 0.3369 & 0.2998 & 0.8823 & 0.3175 \\ DT & 0.6032 & 0.5753 & 0.5716 & 0.9582 & 0.5735 \\ NB & 0.3201 & 0.3000 & 0.2739 & 0.8564 & 0.2863 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Evaluation Metric Results for 16-beam codebook size. 2. The combined results in [16] are averaged over nine scenarios, while the combined results in this work are averaged over five scenarios. 3. In [16], the GPS coordinates are employed directly as ML features, while the coordinates are converted first to cartesian coordinates in this work as part of the preprocessing steps before being used as the ML features. As also noted in [16], the performance results depend to a large extent on the specific scenario under consideration as well as on the number of samples, as can be observed in Table 7, and also depend on the employed algorithm.