# Securing the Smart City Airspace: Drone Cyber Attack Detection through Machine Learning
## 5 Simulations and Analysis
### Analysis  
Results obtained through experiments yielded an outcome that showed superior classification accuracy of the random forest classifier in detecting the three UAV attacks presented in this work. Table 4 shows the impact of choosing different values for the max-depth parameter and the resulting best detection results were obtained with a setting, max depth = 6. In contrast, the results obtained for varying the number of estimators presented in Table 5 and Figure 10 show that the accuracy of the model first increases with the number of estimators and then reduces after the value of nine is reached. Hence, we can conclude that the optimal number of estimators for random forest in detecting UAV attacks is _nine_. Similar observations can be made from the receiver operating characteristics (ROC) curve that illustrate the model performance, as shown in Figures 11 and 12. The ROC curves plot the false positive rate against the true positive rate indicating the discrimination capability between classes for the generated models. A higher area under the curve indicates better capability of models to distinguish between UAV attacks and normal UAV flight operations. Comparing the two hyper-parameters, namely, the max depth and the number of estimators, it is found that the max-depth increases the ability of the generated/trained model to differentiate between the attack and normal classes but may also \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**max\_depth** & **Accuracy** & **Precision** & **Recall** & **Training Time** \\ \hline
2 & 0.9517 & 0.8767 & 0.7518 & 0.2288 \\ \hline
3 & 0.9796 & 0.9818 & 0.8666 & 0.2599 \\ \hline
4 & 0.9782 & 0.9979 & 0.8421 & 0.3401 \\ \hline
5 & 0.9914 & 0.9930 & 0.9436 & 0.3846 \\ \hline
6 & 0.9947 & 0.9960 & 0.9653 & 0.4455 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Impact of max_depth on accuracy, precision and recall rates for the random forest classifier. Figure 10: Impact on random forest accuracy for increasing numbers of estimators. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**n\_estimators** & **Accuracy** & **Precision** & **Recall** & **Training Time** \\ \hline
5 & 0.9636 & 0.9847 & 0.7448 & 0.1424 \\ \hline
7 & 0.9651 & 0.9841 & 0.7567 & 0.1862 \\ \hline
9 & 0.9784 & 0.9759 & 0.8631 & 0.2544 \\ \hline
11 & 0.9615 & 0.8843 & 0.8264 & 0.3371 \\ \hline
13 & 0.9763 & 0.9848 & 0.8393 & 0.3557 \\ \hline
15 & 0.9757 & 0.9896 & 0.8309 & 0.4062 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Impact of number of estimators on accuracy, precision and recall rates for the random forest classifier. lead to overfitting. However, the number of estimators with a value of nine showed the best differentiation capability between normal and malicious drone flight logs. The random forest classifier also outperformed other classifiers, namely, Naive Bayes (NB), linear regression (LR) and support vector machines (SVM). Table 6 presents the comparison of individual classifier performances against the random forest scheme. The results show that random forest with a max-depth set to three and number of estimators set to nine obtained better accuracy, precision and reasonably good recall rates. Especially with large differences noted between the normal and attack categories, the precision and recall results are essential metrics to justify classifier performance in terms of the number of UAV attacks and normal instances to be correctly labeled as belonging to attack and normal classes,  
Figure 11: ROC curve comparing the performances with varying values of max_depth. Figure 12: ROC curve comparing the performances with varying values of numbers of estimators. respectively. The results also show that random forest had very high precision values when compared to other models indicating it was able to detect most attacks and normal instances correctly with low false positives. On the other hand, the other classifiers had higher recall rates when compared to random forest, indicating that certain attack instances of the UAV flight were classified as normal.