# AI-Assisted Security Alert Data Analysis with Imbalanced Learning Methods
## 4 Experiments
### Performance Evaluation
#### 4.4.2 Oversampled Data  
To reduce the impact of the majority class on the detection performance of the security alert data analysis system, minority class data augmentation algorithms were implemented to balance the data distribution and improve the performance. Table 4 presents the results of applying the SMOTE, ADASYN, and SVMSMOTE oversampling techniques to the enterprise dataset. The positive alert detection performance was improved for all algorithms, with LR yielding the highest recall of 99.760% when SMOTE and SVMSMOTE were used for oversampling, but it had a high FPR of 1.063% and 1.090%, respectively. When \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Algorithms** & **Accuracy (\%)** & **Recall (\%)** &
\begin{tabular}{c} **Precision** \\ **(\%)** \\ \end{tabular} & **FPR (\%)** & **TNR (\%)** & **F-Score (\%)** & **BCR (\%)** \\ \hline \hline \multicolumn{8}{c}{Imbalanced security alert data} \\ \hline LR & 90.545 & 7.704 & 73.944 & 0.300 & 99.700 & 13.953 & 53.702 \\ \(k\)-NN & 96.753 & 71.313 & 94.768 & 0.435 & 99.565 & 81.384 & 85.439 \\ DT & 98.895 & 94.180 & 94.689 & 0.584 & 99.416 & 94.434 & 96.798 \\ LightGBM & 99.426 & 95.818 & **98.368** & **0.176** & **99.824** & 97.076 & 97.821 \\ XGBoost & **99.435** & **95.940** & 98.345 & 0.178 & 99.822 & **97.128** & **97.881** \\ \hline \multicolumn{8}{c}{Oversampled security alert data} \\ \hline \multicolumn{8}{c}{SMOTE} \\ \hline LR & 85.205 & 73.955 & 37.621 & 13.551 & 86.449 & 49.872 & 80.202 \\ \(k\)-NN & 88.868 & 79.384 & 46.524 & 10.084 & 89.916 & 58.666 & 84.650 \\ DT & 98.866 & 95.158 & 93.556 & 0.724 & 99.276 & 94.350 & 97.217 \\ LightGBM & 99.392 & **96.674** & 97.197 & 0.308 & 99.692 & 96.935 & 98.183 \\ XGBoost & **99.438** & 96.625 & **97.700** & **0.251** & **99.749** & **97.160** & **98.187** \\ \hline \multicolumn{8}{c}{ADASYN} \\ \hline LR & 44.725 & 96.209 & 14.850 & 60.965 & 39.035 & 25.729 & 67.622 \\ \(k\)-NN & 82.302 & 83.639 & 34.122 & 17.846 & 82.154 & 48.469 & 82.897 \\ DT & 98.642 & 95.500 & 91.260 & 1.011 & 98.989 & 93.332 & 97.245 \\ LightGBM & 99.323 & **97.188** & 96.060 & 0.441 & 99.559 & 96.620 & **98.374** \\ XGBoost & **99.384** & 96.870 & **96.941** & **0.338** & **99.662** & **96.905** & 98.266 \\ \hline \multicolumn{8}{c}{SVMSMOTE} \\ \hline LR & 83.713 & 75.079 & 35.114 & 15.332 & 84.668 & 47.849 & 79.874 \\ \(k\)-NN & 89.289 & _77.867_ & 47.665 & 9.449 & 90.551 & 59.133 & 84.209 \\ DT & 98.698 & 95.133 & 92.049 & 0.908 & 99.092 & 93.566 & 97.113 \\ LightGBM & 99.333 & 96.698 & 96.604 & 0.376 & 99.624 & 96.651 & 98.161 \\ XGBoost & **99.384** & **96.723** & **97.079** & **0.322** & **99.678** & **96.901** & **98.201** \\ \hline \multicolumn{8}{c}{Ensemble resampled security alert data} \\ \hline LR & 89.459 & 58.547 & 47.594 & 7.124 & 92.876 & 52.506 & 75.711 \\ \(k\)-NN & 90.883 & 78.234 & 52.832 & 7.719 & 92.281 & 63.072 & 85.258 \\ DT & 98.644 & 96.014 & 90.880 & 1.065 & 98.935 & 93.376 & 97.474 \\ LightGBM & 99.413 & **96.992** & 97.111 & 0.319 & 99.681 & 97.051 & **98.337** \\ XGBoost & **99.440** & 96.698 & **97.654** & **0.257** & **99.743** & **97.174** & 98.221 \\ \hline \multicolumn{8}{c}{MAS-OSS} \\ \hline LR & 89.503 & 58.596 & 47.767 & 7.081 & 92.919 & 52.630 & 75.758 \\ \(k\)-NN & 92.716 & 76.889 & 60.555 & 5.535 & 94.465 & 67.751 & 85.677 \\ DT & 98.747 & 95.525 & 92.166 & 0.897 & 99.103 & 93.815 & 97.314 \\ LightGBM & 99.409 & 96.552 & 97.481 & 0.276 & 99.724 & 97.014 & 98.138 \\ XGBoost & **99.469** & **96.772** & **97.873** & **0.232** & **99.768** & **97.319** & **98.270** \\ \hline \multicolumn{8}{c}{LR, \(k\)-NN, DT, LightGBM, and XGBoost stand for logistic regression, \(k\)-nearest neighbors, decision trees, light gradient boosting machine, and extreme gradient boosting algorithms, respectively. The highest evaluation performance scores are shown in bold font.} \\ \end{tabular}
\end{table} 
Table 5: Performance evaluation with different sampling settings on the UNSW-NB15 dataset. ADASYN was used for oversampling, LR achieved a recall of 92.067%.