# A Complete EDA and DL Pipeline for Softwarized 5G Network Intrusion Detection
## 4 Dataset Description, Exploration, and Discussion
### Exploratory Data Analysis Discussion  
Exploratory data analysis (EDA) can be defined as the process of \"using descriptive statistics and graphical tools to better understand data\" [70]. Its goal is to gain additional insights into the nature of a dataset and facilitate the detection of any outliers and anomalies within it [70]. Other than helping visualize the data, EDA can help us create hypotheses that can later be tested [71]. That is why EDA is considered as a crucial first step in any data mining or ML/DL framework development, as it can guide the process and choice of models within the framework [71]. It is worth noting that the EDA acronym in this context (within AI/ML applications and frameworks) should not be confused with the commonly used acronym in electronic design (specifically electronic design automation). In this context, it specifically refers to the process of exploring the data and analyzing to uncover underlying patterns within the data. As part of the exploratory data analysis process, several steps were taken. Starting off with a simple counting, Figures 4 and 5 outline the number and percentage distribution of malicious attack instances. It can be seen that the top attack types were UDPFlood and HTTPFlood, which are common distributed denial-of-service (DDoS) attacks. This mimics real-life attack patterns, with a recent report conducted by QRATOR labs indicating that UDP flood attacks constituted 60.2% of DDoS attacks in 2023 [72]. Similarly, the report stated that other flooding attack patterns such as IP flood (similar to HTTP flood) and SYN flood accounted for roughly 23% of attacks. This reinforces the realistic nature of the 5G-NIDD dataset generated in [65]. Figure 4: Number of benign and malicious attack instances—categorized per attack type. The next step was calculating the mutual information score of the different features with respect to the class label. Figures 6 and 7 plot the mutual information score for the binary (i.e., benign or malicious) and multi-class (i.e., categorized by attack type) cases, respectively. More specifically, Figures (a)a and (a)a represent the top 25 features, while Figures (b)b and (b)b represent the remaining features for the binary and multi-class cases, respectively. It can be seen from these figures that the top features for both cases were destination time-to-live (dTtl), number of hops to destination (dHops), destination window (DstWin), and source window (SrcWin). The only difference was is the order. This implies that, from a holistic point of view, features such as dTtl and dHops provided more insights into whether the instance was an attack or not. However, when delving deeper into the data and trying to determine the attack type, features such as DstWin and SrcWin were better capable of determining the attack type, as they provide more information about it. These visualizations give us insights into the potential impact of these different features on the overall performance of the underlying DL models that are used for intrusion detection and attack classification. To further explore the data, a principal component analysis was conducted to obtain insights into the overall distribution of the data per class. Figures 8 and 9 plot the 1 st and 2 nd principal components for the binary and multi-class cases, respectively, while Figure 10 plots the importance of each principal component. As can be seen from Figures 8 and 9, the data were highly non-linear. This was expected, since network traffic data often exhibit non-linear trends, as observed in multiple previous works [15; 16; 17; 27]. This again further highlights and emphasizes the need for ML and DL models that can effectively and efficiently deal with non-linear data. Thus, the choice of using DL models such as DNN with autoencoders, CNNs, and RNNs was justified. Additionally, the choice of plotting only the first two principal components was justified by the results shown in Figure 10, which illustrate that these two components explained nearly 80% of the variance within the data.