## 3 Methodology
### Dataset and Data Preprocessing
#### 3.1.2 Data Preprocessing  
Data preprocessing is a critical aspect of building a robust and accurate intrusion detection system (IDS). In this section, we present our meticulous approach to prepare the datasets for training and evaluating our AI-based IDS. Through essential steps, we aim to optimize the IDS's performance in detecting cyber threats and enhancing network security. We address challenges such as class imbalance, diverse attack types, and feature selection using cutting-edge techniques. By curating a well-balanced dataset and selecting informative features, we empower the IDS to discern between benign and malicious network activities effectively. In the following subsections, we detail each preprocessing step, ensuring the IDS's readiness to combat modern cybersecurity challenges. Binary LabelingTo streamline our focus on binary classification, we transform the multi-class datasets into a binary format. This approach involves categorizing all distinct attack categoriesinto a unified \"attack\" class, while maintaining normal network traffic as the \"BENIGN\" class. This binary setting facilitates a clear discrimination between malicious activities and benign network behavior, enabling our model to concentrate on detecting potential threats effectively. Addressing Data Imbalance  
To ensure the reliability and effectiveness of our intrusion detection system (IDS) in handling real-world cyber threats, we meticulously addressed the issue of data imbalance in the training dataset. Imbalanced data, where one class is significantly more prevalent than the other, can lead to biased model performance, as the classifier may prioritize the majority class and overlook crucial instances of the minority class. In our research, after conducting a thorough analysis of the dataset, we observed a substantial class imbalance problem, with a scarcity of instances representing certain network attacks compared to the abundance of benign network traffic samples. To mitigate this challenge, we employed the synthetic minority over-sampling technique (SMOTE) [8], a widely recognized approach that effectively balances the class distribution. SMOTE works by generating synthetic instances for the minority class based on the existing minority samples. By synthesizing new attack instances that closely resemble genuine attack patterns, SMOTE strategically expands the number of minority class samples, bringing the class distribution closer to balance. Additionally, to prevent overfitting and improve generalization, we applied SMOTE in conjunction with random under-sampling [8; 9; 26]. Random under-sampling selectively reduces the number of majority class instances, ensuring a more balanced and representative dataset while preserving the essential characteristics of both classes. The combination of SMOTE and random under-sampling leads to an adequately balanced training dataset, empowering our IDS to learn from rare attack instances and detect potential cyber threats with greater accuracy. By introducing synthetic attack instances that align with genuine attack patterns, our model becomes more resilient and adaptive, capable of effectively handling the intricacies of real-world intrusion scenarios. In summary, the application of SMOTE and random under-sampling in our training data sampling strategy serves as a crucial step in overcoming the data imbalance challenge. By creating a well-balanced dataset that adequately represents both normal and attack instances, our AI-based IDS demonstrates enhanced performance and robustness in detecting and mitigating cybersecurity threats in IoT networks. Feature selection is a crucial step in the development of an effective intrusion detection system (IDS). It involves identifying and retaining the most informative attributes from the dataset while discarding less relevant ones. The goal is to reduce the dimensionality of the data and enhance the IDS's performance, interpretability, and efficiency.