# Automated Anomaly Detection and Causal Analysis for Civil Aviation Using QAR Data
## 6 Experimental Evaluations
### Evaluation Metrics  
This article uses multiple evaluation metrics to represent the performance of the model, including accuracy, precision, recall, and F1 score. The calculation formulas are as follows:  
\[Accuracy=\frac{TP+TN}{TP+TN+FP+FN} \tag{9}\]  
\[Precision=\frac{\text{TP}}{\text{TP}+\text{FP}} \tag{10}\]  
\[Recall=\frac{\text{TP}}{\text{TP}+\text{FN}} \tag{11}\]  
\[F1=2\times\frac{Precision\times Recall}{Precision+Recall} \tag{12}\]  
In the above formulas, _TP_ represents _true positive_, _TN_ represents _true negative_, _FN_ represents _false negative_, and _FP_ represents _false positive_. In the multi-class prediction results in this paper, _TP_ represents the number of correctly classified predicted values, _FN_ represents the number of other categories incorrectly predicted as a certain category, and _FP_ represents the number of incorrectly predicted values as a certain category. Now, the data before and after balancing are used as training sets for experimental comparison, and the results are presented in the form of a heatmap, as shown in Figure 6.  
The heatmap can be seen as a confusion matrix, where the sum of each row represents the number of true values for a category, and the sum of each column represents the number of predicted values for each category. Therefore, it can be inferred that the data on the diagonal of the confusion matrix is _TP_. According to Figure 6, the evaluation indicators can be calculated. Based on the binary classification, the average of the multiple classifications is taken as the overall result, as shown in Table 7.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Dataset** & **Unbalanced** & **Balanced** \\ \hline Training sample points & 48,858 & 50,000 \\ Testing sample points & 10,251 & 10,251 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Information of datasets.  
\begin{table}
\begin{tabular}{l c c l l} \hline \hline
**Parameter** & **Default** & **Optimum** & **Description** & **Function** \\ \hline learning\_rate & 0.1 & 0.02 & Learning rate & Help converge stably \\ max\_depth & None & 13 & Maximum depth of tree & Prevent over fitting \\ min\_child\_weight & 1 & 4 & The smallest sample weight sum & Control the stop of splitting \\ subsample & 1 & 0.8 & Subsampling rate & Balance the variance and bias \\ colsample\_bytes & 1 & 0.7 & Sampling columns while spanning trees & Improve the generalization ability \\ alpha & 0 & 0.080 & L1 regularization coefficient & Make more conservative \\ lambda & 0 & 0.086 & L2 regularization coefficient & Make more conservative \\ \hline \hline \end{tabular}
\end{table}
Table 5: Results of automated adjustment of parameters.  
From the above results, it can be seen that the model can predict different abnormal events in the approach phase well. Compared with the prediction model using imbalanced samples as the training set, the balanced dataset greatly improved in recall rate. The overall improvement in accuracy and recall rate can be seen from the F1 score, which increased by 1.3%, and the accuracy also improved. To display as many types of anomalies as possible in the test set data, multiple routes were selected, with a total of 10,251 sample data. But the probability of all the above anomalies occurring simultaneously on a single route is extremely small. We selected the approach phase data of a certain flight as the validation set, and the results are shown in Table 8. The experimental results indicate that using balanced data as the training set is effective and stable in improving model performance.