# Combining Unsupervised Approaches for Near Real-Time Network Traffic Anomaly Detection
## 4 Experiment
### Datasets  
One of the main challenges in anomaly-based intrusion detection is the lack of appropriate datasets for evaluating and comparing intrusion detection systems. This gap is caused by the nature of the data, as the inspection of network traffic may expose sensitive information and confidential or personal communications. In addition, publicly sharing raw internet traffic may violate privacy laws. To solve this problem, network traffic is captured in a simulated context. The advantage of using simulation as a network traffic  
Figure 1: Description of the training phase and recognition phase of the proposed approaches.  
generator is that it is free of privacy or sensitivity issues, but since real traffic is very complex, simulating real traffic is a challenge.  
Evaluating the effectiveness of the network traffic anomaly detection system requires datasets that contain data on network traffic monitored over a given time interval in a context in which activities related to regular network use and activities related to cyberattacks that generate anomalous traffic follow one another.  
The Canadian Institute of Cybersecurity (CIC) provides several network traffic datasets used by researchers and companies to evaluate their security systems. The primary datasets identified to evaluate the effectiveness of the proposed Anomaly Detection algorithms are as follows:  
* **KDDCUP99**: The KDD Cup 1999 [26] dataset is one of the most widely used datasets in system evaluation for network security. Since the proportion of data belonging to the class identifying attacks is much larger than the proportion of data belonging to the class defining normal traffic, the class relating to attacks is subsampled to a proportion of 20%. A detailed description of the KDDCUP99 dataset has been published in [27]. The KDDCUP99 dataset consists of approximately 4,900,000 single connection vectors, each of which contains 41 features. Each sample in the dataset is labeled with either the class related to normal traffic or the classes identifying the simulated attack;
* It does not include redundant records in the training dataset, so the classifiers will not be skewed towards more frequent records;
- No duplicate records are present in the test datasets;
- The number of records selected from each set of difficulty levels is inversely proportional to the percentage of records in the original KDDCUP99 dataset. As a result, the classification rates of different machine learning methods vary over a broader range, which makes it more efficient to have an accurate evaluation of different learning techniques;
- The number of records in the training and testing dataset is reasonable, making running the experiments on the entire set convenient without randomly selecting a small portion. As a result, the evaluation results of the different research works will be consistent and comparable.
* **CIC-IDS2017**: The CIC-IDS2017 [30; 31] dataset contains network traffic recorded in a controlled network environment for five days and includes traffic related to normal activities performed by network users and traffic generated by cyberattacks such as BruteForce, DoS, and Botnet. The dataset is provided as a set of PCAP files, which can be played on a computer network interface. The dataset also includes a schedule for attacks and a spreadsheet that contains more than 80 features extracted from network traffic.