# Models versus Datasets: Reducing Bias through Building a Comprehensive IDS Benchmark
## 2 Literature Review
### Benchmark Datasets  
The dataset has a critical role in building and testing a holistic IDS solution. Deep learning models need a large volume of data for better classification and improved performance. Some of our goals in this study, as stated earlier, are (a) to gather, extract, and analyze aggregate knowledge from various benchmark network datasets and look at the commonalities to avoid any possible bias in produced results and (b) mitigate dataset quality and reliability issues. To achieve these goals, we gathered various benchmark network datasets to capture a diverse set of attacks, devices used for data collection, different traffic capture duration, and actual and simulated traffic patterns. Some of these datasets, such as KDD CUP 99 and NSL-KDD, are very old and not IoT specific; however, researchers have still used them extensively to build their models. Others similar to BoT_IoT are recent and provide IoT-specific traffic. Table 1 provides details of the datasets used in our empirical analysis.  
\begin{table}
\begin{tabular}{l l l l l l} \hline \hline
**Dataset** & **Attacks Captured** & **Total Features** & **Total Benign** & **Total Malicious** & **Description** \\  & & & **Records** & **Records** & \\ \hline Bot-IoT & DeS, DDoS, Reconnaissance, Theft & 45 & 477 & 3,668,045 & The dataset consists of both legitimate and simulated IoT traffic and various attack types. It provides full packet capture information with corresponding labels [35]. The dataset is highly imbalanced, with very few benign and a large volume of malicious records. The full dataset consists of 73,300,900 rows and has a size of over 69 GB. Unsily provides a scaled-down 5 percent dataset to make data handling easy. \\ \hline N-BaloT & Gafyot combo, Gafyot junk, Gafyot & 115 & 555,932 & 6,506,676 & The dataset consists of real IoT network traffic collected from 9 commercial IoT devices. Dataset is infected with two famous and harmful IoT botnets Mirai and BASHLITE (also known as Gafyot) [36]. Dataset is unbalanced, with benign records much smaller than malicious records [37]. \\ \hline CICIDS-2017 & DoS Hulk, ForScan, DDoS, DDoS & 85 & 2,271,320 & 556,556 & This dataset consists of complex features that were not available in previous datasets, such as NSL-KDD, KDDCUP 9981. \\  & GeldenEye, FIT-Patator, SSH-Patator, DDoS and Slowhrtplets, Bot, Web Attack Brute, Force, Web Attack XSS, Infiltration, Web Attack SQL Injection, Heartbleed & & & \\ \hline UNSW-NB15 & Generic, Exploits, Fuzzers, DDoS, Reconnaissance, Analysis, Backdoor, Shellcode, Worms & 49 & 2,218,764 & 321,283 & Dataset is designed based on a synthetic environment for generating attack activities [42]. The dataset is not IoT specific and generated by collecting the real benign network traffic and synthetically generated attacks. It contains approximately one hour of anonymized traffic traces from a DDoS attack in 2007 [43]. The overall classification accuracy has a mitigating effect on this dataset; it is due to the greater number of classes (10 in NB15 vs. 5 in KDD) and a higher Null Error Rate (55.06% in NB15 vs. 26.1% in KDD) [44]. \\ \hline NSL-KDD & DoS, Probe, R2L, U2R & 43 & 77,054 & 71,463 & Dataset is an extension of the dataset “KDDCUP 99”. It is not IoT specific, contains no duplicate records, and lacks in modern large-scale attacks [51,24]. The training dataset provides 22 attack types, and the test dataset provides 37 attack types, which are categorized into four attack classes [46]. \\ \hline KDD CUP 99 & Dos, Probe, R2L, U2R & 41 & 1,033,372 & 4,176,086 & Dataset is not specific to the IoT environment. Lacks the latest attack data and contains unbalanced labels [51,24,13]. Excessive duplication in records leads to skewed label distribution, and the classifier generates biased results towards most occurring records and cannot thoroughly learn the least occurring records [47] \\ \hline \hline \end{tabular}
\end{table}
Table 1: Benchmark datasets analysis.