# Machine Learning for Network Intrusion Detection--A Comparative Study
## 5 Methodology
### Preprocessing  
The main goal of preprocessing in machine learning is to facilitate the training/testing process by appropriately transforming and scaling the entire dataset. The preprocessing of data before applying them to an ML algorithm is a crucial step in the ML workflow. Preprocessing removes outliers and scales the features to an equivalent range. It helps to  
Figure 2: Methodology.  
improve the accuracy, reduce the time and resources required to train the model, prevent overfitting, and improve the interpretability of the model.  
In the preprocessing phase, we first merge the eight files into one containing all samples. The process starts by loading the CICIDS-2017 merged file containing 3,119,345 rows and 79 columns. An important step in preprocessing is cleaning the dataset. The cleaning of the data involves finding incomplete, incorrect, inaccurate, or unnecessary data, followed by replacing, modifying, or deleting these data from the dataset. Rows with null values, duplicates, and empty cells, such as infinity (Inf) and not a number (NaN), are dropped. The initial CICIDS-2017 dataset contains categorical features (e.g., labels) that need to be converted into numerical values to prepare them for the ML algorithms.  
Machine learning algorithms assume an even distribution among classes, and class imbalance is a frequent problem, especially in datasets used for network intrusion detection. Handling imbalance is important in real-world applications, where it is crucial to detect minority classes reliably--in our case, attacks. Different solutions to tackle this problem exist, including random undersampling and oversampling. As stated above, a major issue with the CICIDS-2017 dataset is the class imbalance problem. To tackle this problem, we implement a technique that combines different classes to minimize biasing towards the majority classes [14]. This technique considers the characteristics of the attacks and combines similar attacks. The implementation of this technique reduces the number of classes from 15 to 7. We also apply the SMOTE technique to increase the number of samples in the minority classes to 1000 samples each. After this, MinMaxScaler is used to scale the data for the algorithms that are sensitive to the magnitudes of different features (i.e., SVM and LR). Without scaling features, the algorithm may be biased towards the features with values higher in magnitude. The scaling of the data therefore enables these models to learn better and improves their performance. We use the MinMaxScaler technique to transform all feature values in the cleaned CICIDS-2017 data to a range between zero and one.  
The resulting models have to be tested against unseen data to produce reliable outputs. To finalize the preprocessing section, therefore, the dataset is split into three sub-datasets: training (80%), validation (10%), and testing (10%). The split is performed randomly and stratified, meaning that the class distribution percentage of the initial dataset is retained in the training, validation, and testing sets. The split is created in this way to ensure that there are sufficient samples to train the models, while also ensuring sufficient samples for the testing and validation of the results. The training and validation sub-datasets are used for model training and cross-validation, while the testing dataset is used for the final evaluation of the models.