# Automatic False Alarm Detection Based on XAI and Reliability Analysis
## 5 Discussion  
Although advanced cyber threats are increasing day by day, it is difficult to provide an accurate analysis of and response to most threats due to the lack of effective systems. AI technology is being introduced to solve these problems. However, because of opacity and reliability problems, the AI decision-making process is difficult to understand. As a result, there is a limit to introducing AI technology because it is difficult to fully trust AI predictions. Given the reliability problems of AI, analysts must analyze and process data  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Dataset** & **Data Rate** & **FOS** & \
\begin{tabular}{c} **Reliability Score** \\ **(Compared with FOS)** \\ \end{tabular} &
\begin{tabular}{c} **SSSF** \\ **(Compared with FOS)** \\ \end{tabular} \\ \hline \multirow{4}{*}{IDS Dataset} & 10\% & 103\% & 207\% (51\%) & 262\% (78\%) \\  & 20\% & 115\% & 153\% (18\%) & 189\% (35\%) \\  & 30\% & 95\% & 142\% (24\%) & 140\% (23\%) \\  & 40\% & 80\% & 112\% (18\%) & 100\% (11\%) \\ \hline \multirow{4}{*}{Malware Dataset} & 10\% & 68\% & 118\% (30\%) & 127\% (36\%) \\  & 20\% & 122\% & 144\% (10\%) & 178\% (25\%) \\ \cline{1-1}  & 30\% & 82\% & 126\% (24\%) & 144\% (34\%) \\ \cline{1-1}  & 40\% & 52\% & 92\% (26\%) & 90\% (24\%) \\ \hline \hline \end{tabular}
\end{table}
Table 7: Error detection comparison of unordered datasets and reliability analysis results.  
Figure 12: Error detection results when we analyzed the data in the order of highest analysis priority: (a) results of the experiment with the IDS dataset; (b) results of the experiment with the malware dataset.  
directly after predictions by AI models. However, for a massive cyber threat, the amount of data that must be analyzed is huge, and there are far too few analysts. This makes it difficult for analysts to directly analyze the predictions of cyberthreat detection AI models and respond in a timely manner.  
In this paper, we propose a method for measuring the contribution of each feature through XAI and calculating the reliability by measuring the statistical outliers of new data based on the training dataset. It was actually possible to analyze effective false alarms when using confidence indicators during analysis. The judgment of the AI model was analyzed based on the statistical outlier of the SHAP value indicating the contribution. Therefore, it is important that the XAI scores, including the SHAP value, accurately measure the contribution. It is necessary to evaluate how accurately the SHAP value actually contains the degree to which the feature contributed to AI prediction.  
Currently, several XAI evaluation methods are being studied, but no definitive standard exists. It is impossible to measure the accuracy of direct contribution, but indirect measurement is possible through other XAI technologies. However, the results differ depending on the measurement method.  
False alarms were able to be analyzed with higher probability compared to previous studies, but there are still disadvantages because the objective verification of the XAI score has not been performed. Further research on XAI evaluation methods is necessary to expand the effectiveness of the XAI-based framework.