## 4 Results
### Evaluation Results
This experiment highlights the importance of meticulously selecting features to enhance the performance of AI models and underscores that such a feature selection process depends on both the dataset's characteristics and the type of AI model employed. We emphasize that while these results showing high-level performance provide valuable insights, they do not offer detailed information about each AI model's performance on different classes of attacks, which we explore further in the subsequent sections. **Variation in Best AI Model across Attack Types:** Table 9 displays the accuracy for each attack type across the three datasets. The table reveals that most AI models do not consistently achieve superior accuracy across all different attack classes within each dataset (except for SVM on the first dataset and ADA on the second dataset, although SVM exhibits the drawback of a very long training time). This contradicts claims suggesting that a single AI algorithm can outperform others for different datasets with varying intrusion types, underscoring the necessity for an effective approach to understanding how each AI model makes decisions for different attack types, which we elucidate in our current work. In analyzing the variation in AI model performance across different attack types, as shown in Table 9, it is evident that no single model consistently excels across all conditions (i.e., no model achieves 0.99 in all metrics). SVM stands out for the RoEduNet-SIMARGL2021 dataset, achieving perfect accuracy scores in categories like Normal, DoS, and PS attacks, followed closely by RF, KNN, and MLP. This highlights these models' capability to effectively classify various attack types. Conversely, ADA demonstrates robust performance on the CICIDS-2017 dataset, with high accuracy across all attack types, suggesting its potential as a versatile and reliable model for intrusion detection in this case. Nonetheless, RF, SVM, and KNN come in close second, with 0.99 accuracy in most attacks. For NSL-KDD, RF is the best performer, achieving the highest score for three attack types out of five. On the other end of the spectrum, SVM and LightGBM show weaker performance in specific scenarios. SVM struggles with Normal and DoS classification on the NSL-KDD dataset, indicating potential limitations in handling complex \begin{table}
\begin{tabular}{c c c c c c c c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{**RoEduNet-SIMARGL2021**} & \multicolumn{3}{c}{**CICIDS-2017**} & \multicolumn{3}{c}{**NSL-KDD**} \\ \hline
**AI Model** & **Normal** & **DoS** & **PS** & **Normal** & **DoS** & **B. Force** & **Web** & **Inflit.** & **Bot** & **PS** & **Normal** & **DoS** & **Probe** & **R2L** & **U2R** \\ \hline RF & 0.99 & **0.99** & **0.99** & **0.99** & **0.99** & 0.98 & 0.94 & 0.94 & **0.99** & **0.80** & **0.89** & 0.83 & 0.87 & **0.99** \\ ADA & 0.89 & 0.89 & **0.99** & **0.99** & **0.99** & **0.99** & **0.99** & **0.99** & **0.99** & 0.72 & 0.82 & 0.90 & 0.87 & 0.95 \\ DNN & 0.96 & **0.99** & 0.89 & 0.96 & **0.99** & 0.89 & 0.84 & **0.99** & 0.94 & **0.99** & 0.73 & 0.84 & **0.91** & 0.87 & **0.99** \\ SVM & **1.00** & **0.99** & **0.99** & **0.99** & 0.91 & **0.99** & **0.99** & **0.99** & 0.91 & **0.99** & 0.63 & 0.73 & 0.86 & 0.86 & **0.99** \\ KNN & 0.99 & **0.99** & **0.99** & 0.97 & **0.99** & **0.99** & 0.97 & **0.99** & **0.99** & 0.72 & 0.74 & 0.77 & 0.87 & **0.99** \\ MLP & 0.99 & **0.99** & **0.99** & 0.96 & **0.99** & **0.99** & 0.98 & 0.93 & 0.94 & **0.99** & 0.74 & 0.75 & 0.84 & **0.88** & **0.99** \\ LightGBM & 0.89 & 0.64 & 0.55 & **0.99** & 0.95 & **0.99** & 0.95 & **0.99** & 0.94 & **0.99** & 0.75 & 0.87 & 0.82 & 0.87 & **0.99** \\ \hline \hline \end{tabular}
\end{table} 
Table 9: Accuracy per attack type for the RoEduNet-SIMARGL2021, CICIDS-2017, and NSL-KDD datasets with 15 features. The best results are in bold text. or infrequent attack vectors.