# Exploring Multi-Channel GPS Receivers for Detecting Spoofing Attacks on UAVs Using Machine Learning
## 5 Evaluation and Results
### Feature Selection  
In this section, we apply feature selection on the dataset comprising all the missions in order to extract the relevant features that reduce model complexity while maintaining the performance. Therefore, for each model, we set a threshold of 0.05 as the feature relevancy score and created Table 13 to describe the status of each feature with respect to this threshold. Consequently, the features PRN, DO, PD, CP, and TCD exceed the threshold score and will therefore be preserved. Additionally, we observe that C/N0 is considered important for both the random forest and XGBoost models but not decision tree. Thus, we retained it as a relevant feature. We ran 50 independent tests of our ML models on the all-missions dataset after the application of feature selection. The results obtained were subsequently summarized. The decision tree model acquired an average accuracy of 99.976%, a precision of 99.962%, a recall value of 99.947%, and F1-score of 99.954%. Random forest achieved 99.99% accuracy, 99.984% precision, 99.98% recall, and 99.982% F1-score. Finally, XGBoost obtained an average accuracy of 99.894% and precision of 99.817%; for the recall value, it obtained 99.828%, and F1-score of 99.822%. Figures 17-19 demonstrate the evaluation metric behavior of our proposed decision tree, random forest, and XGBoost models, respectively. This information was captured at each iteration of the 50 independent tests before and after applying feature selection on the combined missions dataset. As a consequence, this experiment proves that the performance of the three models remains consistent even after applying feature selection. Figure 16: Performance metrics of the XGBoost model on all-missions dataset. \begin{table}
\begin{tabular}{l c c c} \hline \hline
**Feature** & **Decision Tree** & **Random Forest** & **XGBoost** \\ \hline PRN & YES & YES & YES \\ DO & YES & YES & YES \\ PD & YES & YES & YES \\ DELTA\_RX & NO & NO & NO \\ DELTA\_TOW & NO & NO & NO \\ CP & YES & YES & YES \\ EC & NO & NO & NO \\ LC & NO & NO & NO \\ PC & NO & NO & NO \\ PIP & NO & NO & NO \\ PQP & NO & NO & NO \\ TCD & YES & YES & YES \\ C/N\_0 & NO & YES & YES \\ \hline \hline \end{tabular}
\end{table} 
Table 13: Feature relevancy. Figure 17: Evaluation metric comparison before and after applying feature selection of the decision tree model. Figure 19: Evaluation metric comparison before and after applying feature selection of the XG-Boost model. Figure 18: Evaluation metric comparison before and after applying feature selection of the random forest model. For additional investigation, we compare the time required for each model to fit the training data before and after feature selection using a system with Python 3.9 running on an Intel Core i5-8365U CPU with 16 GB RAM. Therefore, It is clear from Table 14 that the training time is reduced after discarding irrelevant features. However, among the models, random forest exhibits the longest training time, whereas decision tree demonstrates the fastest convergence. To summarize, these experimental results prove the efficiency of our models despite the presence of class imbalance and various dataset sizes. The random forest ensemble model based on bagging emerged as the best-performing approach, consistently achieving the best scores across all the dataset parameters, succeeded by the basic decision tree model, while the XGBoost ensemble method achieved the worst results. The results breakdown is presented in Table 15. However, the random forest model showed the longest convergence time, in contrast to decision tree, which was faster. The feature importance parameter provided by tree-based ML models allowed the extraction of the most significant features in detecting GPS spoofing attacks, namely PRN, DO, PD, CP, TCD, and C/N0, consequently reducing the processing time.