# Exploring Multi-Channel GPS Receivers for Detecting Spoofing Attacks on UAVs Using Machine Learning
## 5 Evaluation and Results
### All Missions Combined  
To further assess the generalization of our models, the datasets of all the missions are consolidated into a single one. The resulting distribution comprises 76.8% authentic signals and 23.2% attack signals, with 6.5% simplistic, 7.5% intermediate, and 9.2% sophisticated attacks; see Figure 12.  
The investigation of feature importance on the dataset comprising all the missions allows us to observe from Figure 13 that the decision tree model relies on the PD feature as the most relevant, followed by CP, DO, TCD, PRN, and C/N0, while the remaining features converge to zero. The random forest model leverages PD as the main feature, accompanied by TCD, CP, DO PRN, and C/N0; however, the rest of the features could be ignored. For the XGBoost model, it depends mainly on PD, as in the two previous models, followed by PRN, DO, TCD, CP, and C/N0. The discussed scores for each model are presented in Table 12. \begin{table}
\begin{tabular}{l c c c} \hline \hline
**Feature** & **Decision Tree** & **Random Forest** & **XGBoost** \\ \hline PRN & 0.009924 & 0.079273 & 0.250672 \\ DO & 0.130057 & 0.164093 & 0.076245 \\ PD & 0.533118 & 0.337536 & 0.381509 \\ DELTA\_RX & 0.000000 & 0.000000 & 0.000000 \\ DELTA\_TOW & 0.000091 & 0.000057 & 0.000702 \\ CP & 0.182235 & 0.155711 & 0.085856 \\ EC & 0.000000 & 0.006048 & 0.000690 \\ LC & 0.000000 & 0.007628 & 0.000829 \\ PC & 0.000000 & 0.008709 & 0.000758 \\ PIP & 0.000000 & 0.001583 & 0.000545 \\ PQP & 0.000389 & 0.001531 & 0.000568 \\ TCD & 0.130549 & 0.191832 & 0.174175 \\ C/N\_0 & 0.013637 & 0.045999 & 0.027451 \\ \hline \hline \end{tabular}
\end{table} 
Table 11: Feature importance scores for Mission 3. Figure 12: Class distribution of the three missions combined. Figure 13: Feature importance ranking from most to least significant—all missions combined. Through 50 independent runs of our models on the dataset comprising all the missions, we examine the behavior of each model across varying training and testing sets. Accordingly, we illustrate the results using curves that demonstrate the values of the evaluation metrics in each test. Initially, the analysis of the decision tree model, as illustrated in Figure 13(a), shows that the accuracy remains stable, with a small standard deviation (STD) of \(\pm\)0.006. The worst accuracy observed was 99.951%, the best was 99.988%, and the average was 99.972%. The precision values, shown in Figure 13(b), range from a minimum of 99.909% to a maximum of 99.987%, with an average of 99.953% and an STD of \(\pm\)0.015. For recall, the model achieved a minimum of 99.90% and a maximum of 99.969%, with an average of 99.941% and a low STD of \(\pm\)0.017, as shown in Figure 13(c). Meanwhile, the F1-score, which represents the balance between precision and recall, achieved an average of 99.947%, with a worst value of 99.904%, a best of 99.978%, and a stable performance indicated by an STD of \(\pm\)0.015, as shown in Figure 13(d). For the random forest model, Figure 14(a) shows an impressive accuracy, reaching a maximum of 99.996% and a minimum of 99.982%, averaging 99.99% with a very low STD of \(\pm\)0.002. Figure 14(b) shows the precision values, ranging from 99.971% to 99.996%, with an average of 99.985% and a low STD of \(\pm\)0.005. The recall, shown in Figure 14(c), reached a maximum of 99.992% and a minimum of 99.965%, with a mean of 99.979% and a stable STD of \(\pm\)0.006. Furthermore, the F1-score achieved a best value of 99.993%, a worst value of 99.97%, and an average of 99.982%, with an STD of \(\pm\)0.004, as shown in Figure 14(d). Lastly, the XGBoost model was found to be the least effective. As shown in Figure 15(a), it achieved a best accuracy of 99.939% and a worst of 99.842%, with a mean of 99.894% and an STD of \(\pm\)0.02. Precision, shown in Figure 15(b), ranged from 99.731% to 99.886%, with an average of 99.814% and an STD of \(\pm\)0.04. Figure 15(c) shows the recall values, with a maximum of 99.915%, a minimum of 99.742%, a mean of 99.828%, and an STD of \(\pm\)0.03.