## 3 Experiments
### Experimental Analysis
#### 3.4.3 Experiment 3 Comparison of Multiple Optimization Algorithms  
The classical machine learning algorithms and the models developed in this study were utilized to perform binary and multiclassification experiments on three distinct datasets. Subsequently, a comprehensive comparison and analysis were conducted based on the results.  
* Binary Classification  
Using the NSL-KDD dataset to train the model, it can be seen from Table 14 that the accuracy, recall, precision and F1 of the Mrmr-SMOTE-Optuna-Catboost algorithm designed in this paper for the detection of the NSL-KDDtest+ dataset are 99.2623%, 99.6350%, 99.2219% and 99.4280% respectively. All the metrics are higher than those of several machine learning models. Figure 9 is its confusion matrix.  
Using the UNSW-NB15 dataset to train the model, Table 15 shows that the t algorithm designed in this paper is also superior to the comparison model in all indicators, which is better than most machine learning methods. Figure 10 is its confusion matrix.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline  & \multicolumn{2}{c}{**NSL-KDD Dataset Performance Comparison**} & & \\
**Method** & **Accuracy** & **Precision** & **Recall** & **F-Measure** \\ \hline SVM [44; 45; 46] & 93.4780\% & 94.2598\% & 95.5015\% & 94.8766\% \\ LR [47; 48; 49] & 96.1362\% & 96.3387\% & 97.6362\% & 96.9831\% \\ XGBoost [50; 51; 52] & 97.9111\% & 98.1793\% & 98.5726\% & 98.3756\% \\ LightGBM [53; 54] & 98.2312\% & 98.5115\% & 98.7397\% & 98.6254\% \\ Catboost [55; 56; 57] & 98.5390\% & 99.0700\% & 98.6616\% & 98.8654\% \\ Algorithm of this paper & 99.2623\% & 99.6350\% & 99.2219\% & 99.4280\% \\ \hline \hline \end{tabular}
\end{table}
Table 14: Comparison results of the proposed model with traditional methods in multiple classifications of the NSL-KDD dataset.  
Figure 8: Optuna Optimization Search Process Diagram.  
The integrated model proposed through the analysis of the results surpasses alternative strategies in the context of binary classification. It demonstrates remarkable generalization capabilities, making it suitable for datasets of varying complexity. The model consistently outperforms others across multiple datasets, underscoring its versatility and adaptability. Notably, this work emphasises feature selection and hyperparameter optimization more than previous studies. Our method was rigorously tested during validation on various benchmark datasets, yielding averaged and reliable results for a comprehensive comparison.  
* Multi classification  
Intrusion detection entails the accurate detection of intrusion behaviors and the precise identification of specific attack types. While some existing intrusion detection algorithms exhibit strong performance in binary classification tasks, they encounter challenges in multi-classification tasks, particularly in accurately categorizing the minority classes within abnormal data, resulting in decreased accuracy. This study leveraged information from three datasets to consolidate the smaller categories into larger categories. We then classified the categorical items into 5, 10, and 6 categories, including normal traffic. Tables 17-19 present a performance comparison between the intrusion detection model based on feature extraction and various machine learning algorithms for multi-classification tasks on the three datasets [58; 59; 60].  
It can be inferred from the table that the proposed integrated model outperforms other strategies in terms of overall performance metrics after completing feature engineering on the dataset, as well as hyperparameter tuning, overcoming the challenges of high latitude and class imbalance encountered in intrusion detection, and efficiently detecting intrusions on the three most well-represented datasets.