# Automatic False Alarm Detection Based on XAI and Reliability Analysis
## 4 Experimental Results
### IDS Dataset Experiment

Table 2: XGBoost parameters. Figure 5: Summary plots by label in training dataset: (**a**) instances labeled normal in the training dataset; (**b**) instances labeled attack in the training dataset. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Accuracy** & **Precision** & **Recall** & **F1 Score** \\ \hline
0.8064 & 0.8523 & 0.8064 & 0.8055 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: XGBoost classification results in the test dataset. The first reliability analysis indicator is SSSF. The SSSF distribution of the data predicted by AI as normal in the test dataset is shown in the graph on the top left in Figure 7. In the graph, the x-axis represents the proportion of the entire dataset, and 0.1 is the data distribution corresponding to 10% of the entire dataset. The current state is a random state that is not sorted based on an indicator and just represents the order in which it was input. The SSSF of most data is distributed below 0. If the AI prediction for the data is accurate, the contribution to the attack of important features would be negative, so the SHAP value would be negative, and the SSSF would also be negative. Therefore, it can be expected that SSSF is calculated at a position less than 0 in the graph. However, on the contrary, if the SSSF is rather positive, then AI is predicted to be normal, but this means that it is considered an attack for important features. It is appropriate to judge these cases as normal based on important features that the analyst considers important but that AI judges as attacks. If an analyst re-analyzes them first, the false AI prediction can be detected in an efficient and explainable way. In Figure 6, if the analyst analyzes the data in the order of high SSSF, it is possible to detect an error with high probability. The graph on the right side of Figure 6 shows the prediction success/failure distribution. Black indicates data that are correctly predicted, and red indicates data that failed to be predicted. In fact, in the case of the data whose labels were predicted to be normal, it can be seen that prediction failure occurs more in the region where the SSSF is higher than expected. Conversely, the data predicted as an attack are expected to have a high SHAP value for important features. The corresponding SSSF will also come out high. If instances with small SSSF are analyzed first, false AI predictions can be detected with high probability. Figure 6: Prediction success/failure distribution in SSSF of IDS dataset: (**a**) success/failure distribution of the data predicted as normal in the test dataset; (**b**) group of SSSF success/failure distribution of the data predicted as normal in the test dataset; (**c**) success/failure distribution of the data predicted as an attack in the test dataset; (**d**) group of SSSF success/failure distribution of the data predicted as an attack in the test dataset. Next, a reliability analysis is performed based on SSSF and the outlier score. The k-NN algorithm is used to calculate the outlier score by representing minute differences in the data. Based on all the features of the data, the degree of difference in other data is measured. The outlier score is the Euclidean distance between the input data and the k-th closest data in the training dataset. In this paper, k = 5 was set for the experiment. The degree of anomaly is measured through SSSF, and the outlier score and is used as a reliability analysis indicator. The reliability score is calculated using the k-NN algorithm based on 2D data composed of SSSF and an outlier score and is not based on all features. The left graph of Figure 7 is the result of calculating the SSSF and outlier score of the normal data of the training dataset. We calculated SSSF and outlier score of the new input data. Based on these scores, the reliability score of the input data was calculated. This is to determine reliability by analyzing the general trend of the training dataset and analyzing different degrees.