# Secure Aviation Control through a Streamlined ADS-B Perception System
## 5 Analysis and Discussion
### Model Evaluation  
This section presents the performance evaluation results for the proposed detection model based on random forest in various indicators. A confusion matrix is used to confirm the performance of the suggested machine learning model. The confusion matrix uses the following criteria to assess the model's prediction:  
* True positives (TP): as anticipated by the model; an injected message is true (injected).
* True negatives (TN): although true, the model anticipated that it would be negative (genuine).
* False positives (FP): although negative, the model anticipated it to be positive (genuine).
* False negatives (FN): although positive, the model predicted it to be negative (injected).  
The confusion matrix can be used to build several equations that can be used to understand and assess the performance of the proposed model, including the following ones:  
* Accuracy indicates the percentage of real positives and real negatives that the model correctly identified.  
\[\text{Accuracy}=\frac{\text{TP }+\text{ TN}}{\text{TP }+\text{ TN }+\text{ FP }+\text{ FN}} \tag{1}\]
* Precision indicates the percentage of real positives for each model's successful predictions. Precision measures how probable a method is to produce accurate results.  
\[\text{Precision}=\frac{\text{TP}}{\text{TP }+\text{ FP}} \tag{2}\]  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Feature** & **Value** \\ \hline Operating System & Windows 11 Pro \\ \hline Processor & AMD RyzenTM 5 5600X \\ \hline Frequency & 4.6 Ghz \\ \hline \# Cores/\# Threads & 6 Cores/12 Threads \\ \hline GPU & RTX 2070 Super Advanced Edition/8 GB GDDR6 \\ \hline Memory/Memory Frequency & 16 GB GDDR4/3200 Mgz \\ \hline ML Framework & Scikit-Learn \\ \hline Language & Python \\ \hline Tool & Jupyter Notebook \\ \hline \hline \end{tabular}
\end{table}
Table 6: Experimental environment.  
* **Recall** calculates the total number of correct predictions produced over the entire dataset, including correct predictions the model missed. Therefore, a high recall value for an IDS model is desired. \[\text{Recall}=\frac{\text{TP}}{\text{TP }+\text{ FN}}\] (3)
* F1-score calculates the percentage of correctly predicted events. It is a weighted harmonic mean of recall and precision. \[\text{F1}-\text{Score}=2\;\times\;\frac{\text{Precision }\times\text{ Recall}}{\text{Precision }+\text{ Recall}}\] (4)
* Matthews correlation coefficient (MCC) is one of the best measures to summarize a confusion matrix and the quality of a binary classification. Simply, it is a correlation coefficient between observed and predicted binary classification, and it will only have good results if all four categories inside a confusion matrix have good results. \[\text{MCC}=\frac{\left(\text{TP }\times\text{TN}\right)\;-\;\left(\text{FP }\times\text{ FN}\right)}{\sqrt{\left(\left(\text{TP}+\text{FP}\right)\left( \text{TP}+\text{FN}\right)\left(\text{TN}+\text{FP}\right)\left(\text{TN}+ \text{FN}\right)\right)}}\] (5)  
We will use accuracy, F1-score, and Matthews correlation coefficient to compare the findings of this study with the previous related literature since the accuracy statistic is frequently used to assess the effectiveness of machine learning algorithms. Accuracy could be misleading when working with imbalanced datasets because it would predict based on the majority class, but this was addressed using different sampling techniques to remove bias. The F1-score and MCC metrics were used due to their ability to showcase the quality of the results.