## 5 Dataset Manipulation
### Dataset Balancing  
An imbalanced training dataset refers to a scenario where the distribution of examples across different classes is uneven or skewed. This can range from a slight bias to a severe imbalance, where the majority class has a significantly larger number of examples compared to the minority class(es). Such an imbalance poses a challenge for classification tasks because many machine learning algorithms assume an equal number of examples for each class, leading to poor predictive performance, particularly for the minority class. The issue is compounded by the fact that the minority class is often more important, and misclassification errors for this class have a higher impact on the problem at hand than errors for the majority class [60]. In supervised machine learning methods, having a balanced dataset during the training phase is crucial to achieve good classification performance. Considering the specific datasets analyzed in this work, Table 7 presents the percentage distribution of attacks for binary classification. It is evident that only the KDD 99 dataset is balanced in terms of attack-normal observations. On the other hand, the UNSW-NB15 and CSE-CIC-IDS 2018 datasets exhibit a significant imbalance, with a strong bias towards normal traffic observations. To address the imbalance issue in the datasets (see Figure 5), a solution was implemented by considering the total number of attacks as the upper limit for selecting normal observations randomly. In the UNSW-NB15 dataset, the total number of attacks is 321,283. Thus, the final dataset is constructed to have an equal number of attacks (321,283) and normal observations. Similarly, in the CSE-CIC-IDS 2018 dataset, which has a total of 2,748,294 attacks, an equal number of attacks and normal observations is chosen. As a result, two reduced datasets are obtained, where the number of normal traffic observations is lower than the original dataset, but the datasets are perfectly balanced with a 50% representation of normal observations and 50 % attacks for the training phase. The same imbalance challenge arises in the multi-class case. However, it should be noted that the KDD 99 dataset is excluded from the multi-class analysis as it only supports binary classification. Tables 8 and 9 provide the percentages of each individual attack type in the UNSW-NB15 and CSE-CIC-IDS 2018 datasets, respectively, compared to the total number of attacks. These tables demonstrate the existing imbalance in the datasets, which necessitates balancing the classes for effective analysis and classification. As observed in the UNSW-NB15 and CSE-CIC-IDS 2018 datasets, the distribution of attacks across different categories is highly unbalanced. Some attack categories have a very small number of observations, which is statistically insignificant for the analysis conducted in this work. For instance, the \"Worms\" attack category in UNSW-NB15 constitutes only 0.05% of the dataset, equivalent to 174 observations. Similarly, the \"SQL Injection\" category in CSE-CIC-IDS 2018 comprises a mere 0.003% of the dataset, corresponding to 87 observations. Having such low absolute numbers poses a challenge for machine learning algorithms to effectively learn the distinct characteristics of these classes during the supervised training phase. Insufficient samples hinder the algorithm's ability to accurately classify these attack categories. To address this issue, a solution was proposed to set a lower limit threshold of 10,000 observations, ensuring an adequate number of samples for machine learning training. Attack categories with observation counts below this threshold were removed. For the remaining attack categories, the number of observations was set equal to the category with the lowest count above the threshold. In UNSW-NB15,