# Automatic False Alarm Detection Based on XAI and Reliability Analysis
## 4 Experimental Results
### IDS Dataset Experiment  
The KDD'99 dataset, which has been widely used in IDS construction, has been a popular dataset since its launch in 1999. The NSL-KDD is a dataset that has been proposed to supplement some problems in the original dataset, such as removing duplicate records from KDD'99. The labels in the NSL-KDD dataset consist of normal and four main types of attacks. This experiment combines the four attack types into one and predicts the attack types and the normal instances. The detailed composition of the NSL-KDD dataset is shown in Table 1.  
Among the 41 features of the NSL-KDD dataset, the protocol type, flag, and service are configured in the form of a string. There are three types of string values for the protocol type, 11 types of string values for the flag, and 70 types of string values for the service. A one-hot encoder is used for three features to convert strings into numbers to train the AI models. Any feature to which the one-hot encoder is applied has a unique feature value. Finally, the 41 features are extended to 122. Next, for the optimization, min-max scaling is applied to all features except for six features that have binary values (\"land\", \"login_in\", \"root_shell_su_attempted\", \"is_host_login\", \"is_guest_login\"). This reduces the deviation between feature values and reduces the error of prediction. After applying min-max scaling, the feature is converted to a value within the range of 0 and 1. The AI model was learned with a preprocessed training dataset. For learning, XGBoost, an ensemble algorithm that uses a combination of several decision trees, was used. The XGBoost parameters used in the experiment are shown in Table 2. Table 3 summarizes the results generated by predicting the test dataset. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Dataset**} & \multirow{2}{*}{**Instance**} & \multicolumn{5}{c}{**Total No. of Instances**} & \multirow{2}{*}{**R2L**} \\  & & & **DoS** & **Probe** & **U2R** & **R2L** \\ \hline Train & 125,973 & 67,343 & 45,927 & 11,656 & 52 & 995 \\ Test & 22,544 & 9711 & 7460 & 2421 & 67 & 2885 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Composition of the NSL-KDD. Figure 4: Reliability score calculation method. (i) Test data calculated with SSSF and outlier score. (ii) Reliability score calculated based on two-dimensional data, SSSF, and outlier score. We used TreeExplainer, which quickly and accurately calculated the SHAP value for the tree and tree emblems. Figure 5 is the result of generating a summary plot for each label after calculating the SHAP value of the training dataset. The summary plot shows the SHAP value for each feature value of each instance. The color is determined by the feature value, the x-axis shows the SHAP value, and the y-axis shows the distribution of instances. The left and right graphs are summary plots of the data whose labels are normal and attacks, respectively. According to the criteria described above for selecting the important features, the following 10 features were selected for each label. The important features of normal are \"dst_host_srv_count\", \"service_http\", \"dst_host_same_srv_rate\", \"logged_in\", \"service_domain_u\", \"Protocol_type_udp\", \"sv_diff_host_rate\", \"service_smtp\", \"flag_RE\", and \"flag_S1.\" The important features for attack are \"count\", \"dst_host_s_error_rate\", \"service_private\", \"dst_host_same_src_port_rate\", \"Protocol_type_icmp\", \"dst_host_error_rate\", \"hot\", \"service_ecr_i\", \"dst_host_diff_srv_rate\", and \"dst_host_srv_diff_host_rate\". Based on the features selected in this way, SSSF is calculated with features that fit the label corresponding to the data predicted by AI. The SSSF distribution for each label is shown in Figure 6. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Parameter** & **Value** & **Parameter** & **Value** \\ \hline Booster & Gbtree & Subsample & 1 \\ Objective & Binary:logistic & Colsample\_bytree & 1 \\ Max\_depth & 4 & Learning\_rate & 0.1 \\ \hline \hline \end{tabular}
\end{table}