## 3 Preprocessing and Data Analysis
### Data Preprocessing  
Data Pre-processing is a technique used in data mining for the transformation of raw data into a format that can be utilized efficiently. It encompasses different techniques for data cleaning, data transformation, and data reduction. Data Cleaning is an essential step in pre-processing that removes the irrelevant and missing information from the dataset that could lead to the incorrect estimation of the target class.  
There were a total of 8 different files that were merged into one single file and the entire work presented was relying on this file. The different types of attacks noticed are displayed in Table 2. In total, they were 2,830,743 examples included in 15 classes as shown in Table 3. On the other hand, the number of instances per class is presented in Table 4.  
\begin{table}
\begin{tabular}{l c c} \hline
**File Name** & **Day** & **Attack Type** \\ \hline Monday- & & Benign (Normal human activities) \\ WorkingHours.pcap\_ISCX.csv & & Tuesday & Benign, FTP-Patator, SSH-Patator \\ \hline Wednesday- & & Benign, DoS GoldenEye, DoS Hulk, DoS Slowhttptest, DoS slowloris, Heartbleed \\ \hline Thursday-WorkingHours- & & Benign, Web Attack—Brute Force, Web Attack—SQL Injection, Web Attack—XSS \\ ISCX.csv & & \\ \hline Thursday-WorkingHours- & & Benign, Infiltration \\ Afternoon-Infiltration.pcap\_ISCX.csv & & \\ \hline Friday-WorkingHours- & & Benign, Bot \\ Morning.pcap\_ISCX.csv & & \\ \hline Friday-WorkingHours- & & Benign, PortScan \\ FortScan.pcap\_ISCX.csv & & \\ \hline Friday-WorkingHours- & & \\ Afternoon- & & \\ DDoS.pcap\_ISCX.csv & & \\ \hline \end{tabular}
\end{table}
Table 2: Description of CICIDS2017 Dataset.  
\begin{table}
\begin{tabular}{l c c} \hline
**Dataset Name** & **CICIDS2018** \\ \hline Dataset Type & Multi-class \\ \hline Year of release & 2017 \\ \hline Total number of distinct instances & 2,830,540 \\ \hline Number of features & 83 \\ \hline \end{tabular}
\end{table}
Table 3: CICIDS2017 Dataset characteristics.  
Each row in the dataset contained nearly 83 features. The CICI Flow Meter is also responsible for generating bidirectional flow. In this flow, the first packet is accountable for both forward and backward directions. The forward direction is from source to direction whereas the backward direction is from destination to source. Therefore, it can be demonstrated that in total there are 83 statistical features which include both the data from back and reverse directions.  
In the research, the subset of the original 83 features was utilized excluding a few features. Some of these excluded features were IPs of the source to destination, timestamp, the ID of flow, etc. As a result, nearly 79 features were left. Out of the remaining features, the last feature, i.e., the 79 th feature represents the category of traffic prescribed in the present Bi-flow; this is denoted as the label.  
Upon investigation, we identified that the dataset contains 135,000 values that are either missing or contain infinite floating-point values that can't be processed. So, we removed those data instances from the dataset. The dataset used contains 83 distinct features for the classification of network activity. Table 5 shows some of the most discriminative features along with their variance ratios. From Table 3, we can analyze that the dataset is highly imbalanced, and applying the machine learning algorithms to the imbalanced dataset could lead to more bias toward the majority class [32]. So, we removed the classes that are highly imbalanced and utilized the random under-sampling technique to decrease the size of the majority class. Table 4 shows the number of class-wise instances that we are using for this research.