## 4 Experiment and Results
### Binary Classification  
#### 4.1.1 Model Training and Selection  
In the case of binary classification, the goal is to identify whether the traffic is legitimate or an attack, without differentiating between specific attack types. As previously stated, we reorganized the data into two categories: legitimate (0) and attack (1). To effectively train models, we followed a standardized approach. We first separated the dataset into features and target variables. The features were scaled using a StandardScaler (Z-score normalization) to normalize the data, which is vital for many machine learning models to perform effectively [36; 37]. Following the scaling process, the dataset was divided into training and testing sets, with 30% allocated for testing. A stratified split was used to preserve the class distribution of the target variable in both the training and testing sets. This stratified split helps preserve each class's proportion, producing more balanced and representative subsets. Balanced and representative subsets enhance model performance and prevent bias in evaluation metrics [38]. To optimize the performance of the models, we employed grid search with cross-validation (GridSearchCV) [39] for hyperparameter tuning and cross-validating. Hyperparameters are settings that control how an ML model learns from data. Tuning involves systematically adjusting these settings (e.g., tree depth, learning rate) to find the combination that provides optimal performance. Thus, selecting appropriate values significantly impacts model performance [38]. For all employed learning algorithms except DNNs, we defined a parameter grid that specifies a range of values for each hyperparameter. Subsequently, GridSearchCV systematically explores multiple combinations of hyperparameter values to identify the optimal set of hyperparameters. To mitigate the risk of overfitting, grid search is combined with cross-validation, a technique that divides the dataset into multiple subsets to train and test the model iteratively. In k-fold cross-validation, the training set is split into k equally sized, non-overlapping subsets. This approach creates multiple training-validation pairs for hyperparameter tuning and model selection, ensuring that the test set remains independent for the final evaluation of the model. For each iteration, k \(-\) 1 subsets are used for training, while the remaining subset is used to evaluate the model. This process is repeated k times, allowing each subset to be used as the validation set once. Consequently, different models and \begin{table}
\begin{tabular}{c l l l} \hline \hline
**No.** & **Name** & **Description** & **Protocol Layer** \\ \hline
1 & tcp.time\_delta & Time TCP stream & TCP \\
2 & tcp.len & TCP Segment Len & TCP \\
3 & mqtt.kalive & Keep-Alive & MQTT \\
4 & mqtt.msgid & Message Identifier & MQTT \\
5 & mqtt.retain & Retain & MQTT \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Retained features. performance metrics are obtained for k. The final performance metric is calculated as the average of these k values. The choice of k often depends on the data size. However, 5 (utilized in this study) and 10 are the commonly used values of k in the literature [40]. After identifying the optimal hyperparameters and selecting the best model, we refit the model using the entire training set. Table 4 summarizes the optimized hyperparameters for each employed algorithm. For DNN, we utilized Keras Tuner, a framework for hyperparameter optimization [41]. Keras Tuner simplifies the process of defining search spaces with a define-by-run syntax and supports multiple search algorithms, including Bayesian Optimization, Hyperband, and Random Search [41; 42; 43]. Our study employed Random Search to explore a range of hyperparameters such as dropout and learning rates. The tuner was configured to perform five trials with three executions per trial, optimizing for validation accuracy.