# Automatic False Alarm Detection Based on XAI and Reliability Analysis
## 4 Experimental Results
### Comparative Analysis of AI Reliability Evaluation  
We compared the results using the same dataset as in the previous paper. [PERSON] et al. proposed a method for the automatic screening of valuable alerts based on XAI [25]. After the method measures the contribution of each feature of the data used for AI prediction, it selects 10 important contributing features. Then, by measuring the contribution of the features to the new input data, the method decides whether to doubt whether AI's judgment about important features is reliable. AI judgment is analyzed using a confidence indicator called the feature outlier score (FOS). FOS is calculated based on the cumulative probability distribution of the contribution of each feature in the training dataset, and then the suspicion score for the input data is calculated and valuable alerts are selected. Experiments were conducted using the IDS dataset, NSL-KDD, the malware dataset, and the 2019 KISA Data Challenge Dataset. Figure 11 is the result of comparing the number of errors detected by the AI model with those of the proposed method. Compared to the existing AI model, the proposed method showed a 114% improvement in the IDS dataset and a 95% improvement in the malware dataset. This method has the advantage of being able to check the suspicion score for each feature, and the improved performance is insufficient for application in real environments.  
Figure 12 represents a graph that compares the experimental results of the IDS dataset and the results calculated based on the FOS. Table 7 is the result of comparing the false  
Figure 11: The number of errors detected by the AI model and the proposed method (IDS and malware datasets) [25].  
Figure 10: Process of calculating the reliability score of the test dataset with the training dataset of malware dataset: (**a**) SSSF/ outlier score distribution calculated from the training dataset labeled normal; (**b**) reliability score distribution of the data predicted as normal in the test dataset.  
alarms that were detected when we analyzed the upper data with high analysis priorities with data that were not ordered with priority. The performance improvement ratio according to the proposed method was calculated by comparing data that were not ordered by priority and data that were ordered by priority using each indicator. The \"data rate\" represents the proportion of the entire dataset. If the dataset is sorted based on the reliability score, then \"data rate = 0.1\" represents the top 10% of data. At the same data rate, if 10 false alarms occurred in unordered data, 16 occur in data sorted by FOS, and 20 occur in data sorted by SSSF, the improvement rate can be calculated as follows: FOS improved performance by 60% (\((16-10)/10\)) over unordered data, SSSF improved performance by 100% (\((20-10)/10\)) over unordered data, and SSSF improved performance by 25% (\((20-16)/16\))) over FOS. Based on the top 10% of data with high analysis priority in the IDS dataset, it was confirmed that the results calculated based on the FOS were 103%, the results calculated with the reliability score were 207%, and the results calculated with SSSF were 262%. For the malware dataset, it was confirmed that the results calculated based on FOS were 68%, the results calculated with the reliability score were 118%, and the results calculated with the SSSF were 127%.