# Automatic False Alarm Detection Based on XAI and Reliability Analysis
## 4 Experimental Results
### Malware Dataset Experiment  
To verify the validity, an experiment was conducted using the 2019 KISA Data Challenge Dataset. The data composition of the 2019 KISA Data Challenge Dataset is the same as shown in Table 4, with the exception of a file whose features could not be extracted because the format of the PE header did not match. The malware features were extracted using PE static analysis and YARA rules [28]. YARA rules are used to classify and identify malware samples by creating descriptions of malware families based on textual or binary patterns. Through the PE static analysis, 677 features were extracted from the DLL/API, section, entropy, and PE Header, and 104 features were extracted through YARA rules matching information. A total of 781 malware features were used. \begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Dataset**} & \multicolumn{3}{c}{**Total No. of Instance**} \\  & **Instance** & **Normal** & **Malware** \\ \hline Train & 29,130 & 11,568 & 17,562 \\ Test & 9301 & 4518 & 4513 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Composition of the 2019 KISA Data Challenge Dataset. Figure 7: Process of calculating the reliability score of the test dataset with the training dataset of IDS dataset: (**a**) SSSF/outlier score distribution calculated from the training dataset labeled normal; (**b**) reliability score distribution of the data predicted as normal in the test dataset. The AI learning algorithm used LightGBM, which is an ensemble algorithm similar to XGBoost. The LightGBM parameters used in the experiment are shown in Table 5. Table 6 summarizes the results generated by predicting the test dataset. The AI that learned the malware dataset also used TreeExplainer. Figure 8 shows the results of generating a summary plot for each label after calculating the SHAP value of the training dataset. The left and right graphs are summary plots of data whose labels are normal and malware. According to the criteria for selecting the important features described above, the following 10 features were selected for each label. The important features of normal were \"CheckSum\", \"IMAGE_SCN_MEM_WRITE\", \"Subsystem\", \"Characteristics\", \"AddressOfEntryPoint\", \"ImageBase\", \"without_urls\", \"Text_VirtualSize_por\", \"Major-SubsystemVersion\", and \"section_49.\" The important features of malware were \"CheckSum\", \"Characteristics\", \"section_93\", \"Str_Win32_Winosock2_Library\", \"inject_thread\", \"create_process\", \"IMAGE_FILE_RELOCS_STRIPPED\", \"network_dropper\", \"persistence\", and \"PointerToLineumbers\". Based on the features selected in this way, the SSSF was calculated with features fitting the label corresponding to the data predicted by AI. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Accuracy** & **Precision** & **Recall** & **F1 Score** \\ \hline
0.9909 & 0.9910 & 0.9909 & 0.9909 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: LightGBM classification results on the test dataset. Figure 8: Summary plots by label in the training dataset (**a**) instances labeled normal in the training dataset; (**b**) instances labeled attack in the training dataset. The SSSF distribution of the data predicted by the AI as normal in the test dataset is shown in the graph on the top-left of Figure 9. It can also be seen that the SSSF of most data is distributed below 0 for the malware dataset. As with the IDS dataset, if you analyze the upper data with a high SSSF, you would be able to detect false predictions with a high probability. In fact, as shown in the graph on the top right of Figure 9, it can be seen that false predictions mainly occurred in the data with a high SSSF even though the prediction was normal. As in the previous experiment with the IDS dataset, the k of the k-NN algorithm was set to 5 to measure the outlier score. The left graph of Figure 10 is the result of calculating the SSSF and the outlier score of the normal data on the training dataset.