# Semi-Supervised Learning for Intrusion Detection in Large Computer Networks
## 3 Methodology
### Data Collection and Preprocessing  
#### 3.1.1 Description of the Dataset  
The CIC-DDoS2019 dataset [26] is a substantial and diverse collection of network traffic data that closely resembles real-world network environments. It contains a wide range of network activities, including both normal and anomalous behavior, making it an ideal foundation for our intrusion detection research. The dataset has the following key characteristics:  
* **Size and Scope:** The CIC-DDoS2019 dataset is substantial in size, containing a 30 gigabyte collection of network traffic records. It spans a wide range of network activities, making it suitable for both research and real-world application. * **Variety of Network Traffic:** It comprises diverse network traffic data, encompassing legitimate network transactions and various forms of network attacks. The dataset includes BENIGN, DNS, LDAP, MSSQL, NetBIOS, NTP, SNMP, SSDP, UDP, Portmap, Syn, TFTP, and UDPLag data labels. This diversity reflects the complexity of real-world network environments. * **Real-World Relevance:** The dataset is curated to mirror real-world scenarios, capturing network behaviors observed in operational networks. This real-world relevance ensures that the research findings are applicable to practical network security situations. * **Network Protocols:** It covers a wide range of network protocols, including HTTP, TCP, UDP, ICMP, and others. This protocol diversity reflects the complexity of modern network communications. * **Traffic Features:** Each network traffic record is associated with a set of features, which include flow characteristics, packet attributes, and temporal information. In total, the standard dataset before preprocessing contains 85 features. These features are essential for building effective intrusion detection models. * **Imbalanced Data:** To mirror common real-world intrusion detection scenarios, the dataset exhibits class imbalance in which normal traffic significantly outweighs malicious traffic. Addressing class imbalance is a crucial aspect of our research. The UNSW-NB15 dataset [27] is a widely recognized benchmark dataset for network intrusion detection that is designed to simulate real-world network environments and cyberattack scenarios. It serves as an excellent resource for evaluating intrusion detection models, and has been used extensively in research due to its comprehensive coverage of various attack types. The dataset has the following key characteristics:  
* **Size and Scope:** The UNSW-NB15 dataset contains over 2.5 million network traffic records. The dataset's large size ensures that it encompasses a broad spectrum of network behaviors, providing ample data for training and testing intrusion detection models. * **Variety of Network Traffic:** The dataset includes a mix of legitimate (normal) network traffic and a wide range of attack vectors. In addition to normal traffic, it contains nine categories of attack labels: Fuzzers, Analysis, Backdoor, DoS, Exploits, Generic, Reconnaissance, Shellcode, and Worms. This variety offers a diverse set of data to evaluate the effectiveness of different detection techniques. * **Real-World Relevance:** UNSW-NB15 was designed to replicate realistic network environments and modern-day cyber threats. It reflects the behavior of real-world traffic, making it highly relevant for evaluating security models aimed at practical deployment in current network security systems. * **Network Protocols:** The dataset captures network flows that involve various protocols such as TCP, UDP, ICMP, and HTTP. The inclusion of multiple protocols makes it possible to explore a wide range of attack surfaces and defense mechanisms. * **Traffic Features:** Each record in the UNSW-NB15 dataset includes 49 features, which cover both network flow characteristics and packet-level details. These features are critical for building robust machine learning models for intrusion detection. Key features include flow duration, service type, packet size, and content-based attributes.