dataesr/scientific-paragraphs-categorization huggingface dataset:
B. RELATED WORK In the past decade, numerous IDSs were developed and evaluated against a range of published available datasets. Diverse review and comparative studies have been published tackling the design of IDS for various applications, as well as, the machine learning techniques used to build IDS, however, the dataset challenges are not discussed. Current Network IDS surveys often focus on a single aspect of IDS evaluation. Buczak and Guven  [4]  focus on the different ML and DL algorithms used to build IDS. They explain the different algorithms, mention their time complexity and list notable papers that employ each algorithm to IDS. Hodo et al. [8]  extend the ML discussion, the authors focus on the role that feature selection plays in the overall training and performance evaluation of ML techniques.
For example, Ismail Butun et al. [2]  discusses Wireless Sensor Networks (WSN), Chunjie Zhou et al. [24]  highlights IDS in industrial process automation while Ghaffarian and Shahriari  [16]  study ML and Data Mining (DM) techniques for software vulnerability. While these surveys provide valuable information on the design and the accuracy, none provide a detailed overview of the shortcomings of available datasets nor do they provide information on tools used to carry out attacks. In this manuscript, we address these shortcomings and provide detailed complementary work to build datasets that reflect current network threats. This manuscript is complementary to prior surveys by highlighting the shortcomings of current datasets and the claims of numerous studies on their abilities to detect deprecated attacks.
A. DATASETS Researchers depended on benchmark datasets to evaluate their results. However, currently available datasets lack reallife characteristics of recent network traffic. This is the reason that made most of the anomaly IDSs not applicable for production environments  [25]. Furthermore, IDS is unable to adapt to constant changes in networks (i. e. new nodes, changing traffic loads, changing topology, etc. ). Networks are constantly changing, for this reason depending solely on old datasets doesn't help the advancement of IDS. The process of generating new datasets should consider this constant change fact. For example, proposing a standard dataset generation platform with extendable functionality, would remove the burden of generating datasets from scratch and cope with concept drift in network patterns.
This recommendation and others are further discussed in Section V. Datasets could either be real (i. e. recorded from a network set-up) or synthetic (i. e. simulated or injected traffic). Synthetic attack injection could be used to either introduce attacks to an existing dataset or balance the attack classes present in a dataset. Viegas et al. [25]  mentioned that for a dataset to be considered, it has to cover the following properties. (a) Real network traffic (similar to production ones), (b) valid, such that it has complete scenarios. (c) Labeled, classifying each record as normal or attack, (d) variant, (e) correct, (f) can be updated easily. (g) Reproducible in order to give researchers space to compare across different datasets, and finally, (h) shareable, hence it should not contain any confidential data.
Additionally, Sharafaldin et al. [56]  mentions that (i) having an appropriate documentation for the feature and dataset collection environment is an important aspect of IDS dataset. Cordero et al. [57]  adds (j) having high quality normal and (k) excluding any disturbance or defects as further requirements for evaluation datasets. Furthermore, for NIDS evaluation dataset, functional and non-functional requirements are elaborated in  [58]. In this manuscript, we also identify two problems that impact research domains using datasets, whether they are synthetic or not. i) Sharing datasets is sometimes prohibited due to the data contained, hence, the research in the area is limited. ii) Simulating real-life scenarios and associated attacks is difficult due to the number of parameters required for the model to be viable.
However, this manuscript provides a list of the most used and recent datasets. TABLE  1  summarizes the available datasets and categorizes them based on the domain they belong to. Moreover, attacks found in each are presented. Extra remarks, including the publication year, institute and attack classes details are listed in TABLE A. 2. These datasets cover mobile applications, Virtual Private Networks (VPN), Tor Networks, IDS, Botnet, Network Flows and IoT. Some of the mentioned datasets are presented in  [56]. The evaluation includes DEF-CON  [59] , CAIDA  [60] , LBNL  [61] , CDX  [62] , Kyoto  [63] , Twente  [64] , UMASS  [65]  and ADFA  [30]. Ring et al. [66]  comprehensively overview of NIDS datasets covering their main features, data format, anonymity, size, availability, recording environment, balancing, etc.
The authors list the datasets and their corresponding values in each of the aforementioned criteria, leaving the choice for researchers to make based on their use-case and scenario. On the contrary, Gharib et al. [67]  propose datasets score based on the attacks' coverage, protocols' coverage, metadata availability, anonymity, heterogeneity and labeling. While the authors evaluate attacks in the datasets and present a scientific comparison, the authors fail to provide a detailed analysis of the broader impact of their analysis. Furthermore, due to the sparsity of the details supplementing the available datasets, the task of evaluating and ranking datasets would introduce unfair results. For example, a dataset that realistically represents background and attack - - - - - - - CICIDS2017 [27] - - - - - - - CIC DoS dataset [28] - - - - - - - - - - - - - - - - - & ADFA-IDS [29], [30] - - - * + - - - - - - - - - Unified Network Dataset [31] - - - - - - - - - - - - - - - - - DDoSTB [32] - - - - - - - - - - - - - - - - - Booters [33] - - - - - - - - - - - - - - - - - - TUIDS Coordinated Scan [34] - - - - - - - - - - - - - - - - TUIDS DDoS [34] - - - - - - - - - - - - - - - - - TUIDS Intrusion [34] - - - - - - - - - - - - - - - - Botnet dataset [35] - - - - - - - - - - - - - - - - - STA2018 [36] - - - - - - - - - - - - - - CTU-13 [37] - - - - - - - - - - - - - - - - - ISCXIDS2012 [38] - - - - - - - - - - - - - - Waikato [39] - - - - - - - - - - - - - - - - - - CAIDA DDoS [40] - - - - - - - - - - - - - - - - - - NSL-KDD [41] - - - - - - - - - - - - - - KDD'99 [42] - - - - - - - - - - - - - - 1998, 1999, DARPA [43] - - - - - - - - - - - - - - Special Purpose Networks Year Dataset IoT VPN Tor SCADA Bot-IoT [44] - - - Anomalies Water System [45], [46] - - - IoT devices captures [47] - - - Tor-nonTor dataset [48] - - - VPN-nonVPN dataset [49] - - - 4SICS ICS [50] - - - Mobile Applications Year Dataset Benign Botnet Adware Malware Kharon Malware Dataset [51] - - - - Android Adware and General Malware Dataset [52] - - Android Botnet dataset [53] - - - - Android Malware Genome [54] - - - - AndroMalShare [55] - - - *: Adding new Superuser, +: C100 VOLUME 4, 2016 traffic is better than a dataset that doesn't.
B. IDS AND ASSOCIATED DATASETS ANALYSIS In this section, a survey of recent ML IDS is provided, analyzing the associated datasets, and their shortcomings. IEEE Xplore and Google Scholar queries were made using "Intrusion Detection System*" OR "IDS*" filtering the dates to include manuscripts published in the last decade. The filtration was made to have a wide coverage of datasets, ML techniques and detected attacks. A total of 85 published manuscripts in the period of  [2008 -2020]  are analyzed. Analysis of older IDS ML techniques and used features for the period 2004 -2007 was previously conducted by Nguyen and Armitage  [11]. They discuss the limitations of port-based and payload-based classification and the emerging use of ML techniques to classify IP traffic. TABLE  A. 1 summarizes the pre-eminent (i.
e. , most cited) IDS research from the past decade. Each IDS is mentioned with a list of the algorithms used and the datasets that the IDS was evaluated against. Moreover, the attacks detected are also listed. The algorithmic trends are then discussed alongside the attacks included in the datasets used. FIGURE  1  shows the distribution of datasets used for research in the last decade. Only 11% of the mentioned IDSs used generated or simulated datasets. It is also clear through this analysis that most datasets lack real-life properties, which were previously mentioned in Section III-A. FIGURE 1 also highlights the use of KDD-99 as the dataset of choice. Amjad Al Tobi and Ishbel Duncan  [68]  provide a comprehensive analysis of the drawbacks of the KDD'99 dataset. Moreover, Siddique et al. [69]  provide a timeline for KDD datasets family.
The provided timeline show both the different criticism points and the UCI Lab warning not to use KDD Cup'99 dataset, which further emphasizes the drawbacks of using KDD Cup'99 in the current IDS research. The second most used dataset is the DARPA datasets. DARPA datasets fail to accurately represent current attacks due to their age. Moreover, the use of the KDD'99 and DARPA datasets lead to an endemic situation, numerous results reported in literature claim detection results which are not applicable in real-world scenarios. The shortcomings of the DARPA dataset are analyzed by M. Mahoney and P. Chan  [70]  and John McHugh  [71]. Alongside the limitations of each dataset, they are also deprecated, hence, demonstrating the inability of the IDSs presented in.
This section classifies network threats based on the layers of the OSI model, provides examples of attacks for different threat types and presents a taxonomy associating network threats and the tools used to carry out attacks. The taxonomies aim at helping researchers building IDSs, but more importantly by associating the threats to the OSI model and benchmarking the threats to the tools used to carry attack or take advantage of specific vulnerabilities, the taxonomies aim to achieve higher accuracies and reduce the number of false positives of current IDS  [80]  and build better datasets. As shown, attacks can target a single layer of the OSI model, but it is important to highlight that other layers may also be affected. The taxonomy presented in this manuscript focuses on the main target layer of attack.
An attack is also described to be active if it affects information, performance, or any aspect of the media on which it is running. In contrast to active attacks, during passive attacks the attacker is concerned with either gathering information or monitoring the network. These can be identified by their shape in FIGURE  4. Active attacks are represented by a rectangle shape, whilst passive attacks are represented by an oval shape. Attacks like adware (FIGURE  4 -  2) are forms of active attacks. However, some attacks cannot be considered active or passive until their usage is known. An example of this case is SQL-injection, if it is used for querying data from a database then it is passive. However, if it is used to alter data, drop tables or relations then the attack can be considered as active.
A. LIMITATIONS AND CHALLENGES The limitations and challenges in datasets used in IDSs can be summarized in the following: • Attacks Coverage: As shown in this work only 33. 3% of known attacks are covered in publicly available datasets reviewed. This is considered one of the biggest challenges preventing IDSs to be used in real-life environments. • Real-life Simulation: Only 11% of the past decade IDSs use recent and/or real-life generated or simulated datasets. This demonstrates a flaw in the development of IDSs but highlights their limited ability to cope with the emerging needs. • Zero-Day Attacks Handling: Attacks evolve at a pace that datasets are not currently coping with. New dataset.
B. RECOMMENDATIONS Guided by the critical impact of datasets on the evolvement of IDSs and the importance of robust and accurate IDS models, the following recommendations help build the next generation IDSs. The research direction should focus equally on building complex models for IDSs and gathering/generating data that represent benign and attack scenarios accurately. This will result in IDSs suitable for real-life deployments. • ML-First Vs Data-First: As discussed in Section III-A, obtaining valid, representative, and accurate data should be considered as the primary focus of research for the creation of IDSs. Building IDSs based on skewed and biased data only produces models unfit for exploitation, hence, Data-First models must be considered before ML-First. • Using precise evaluation metrics: As discussed in Section II, metrics -other than accuracy -should be considered to precisely reflect an IDS performance.
For example, FP and Recall should be reported. Furthermore, the geometric mean should be used with imbalanced datasets, as well as, networking metrics such as throughput. Conventional ML models report loss and accuracy by default unless other parameters are defined. Relying on the recorded loss and accuracy without measuring proposed evaluation metrics may result in misleading assessments of the overall IDS efficiency. • Introduce modular and extendable datasets: As aforementioned, special purpose datasets are demanded, either to cover bespoke networks and architectures (e. g. IoT, SCADA, Tor, etc. ) or to introduce new and zeroday attacks. To increase the impact of datasets, they are required to be easily extendable and capable of integrating with other datasets. As a result, datasets would be adaptable to the continuous network changes.
Also, dataset generation could be rendered in the IDS pipeline, therefore, not requiring the generation of a new dataset with every introduced change. To this end, • Standardize attack dataset generation/collection method: One of the main challenges forcing researchers to work with outdated datasets is the lack of documentation associated with newly available datasets. Moreover, publishing raw packet data, not only the computed features, is needed to expand the use of datasets. One of the ways to generate datasets relies on α (using descriptive language to describe attacks) and β (using behavior and statistical measures to describe attacks) profiles, as described by Shiravi et al. [94]. Ring et al. [66]  recommends being careful with anonymization and choosing which fields to be discarded.
Privacy is depicted as one of the main obstacles against the collection of new attacks data. Furthermore, the lack of standard tools for data collection, anonymization, documentation and publication demand researchers to use their own tailored methods. • Introduce models to inject realistic attacks: Flow-Level Anomaly Modeling Engine FLAME  [95]  was one of the first tools to inject attacks that leave traces into network flows. ID2T  [57] ,  [96] ,  [97]  is a proposed flexible model to inject scenarios to existing datasets. The absence of thorough documentation of datasets makes it harder to map one dataset to another, rendering it impractical to add new attacks to existing datasets. Moreover, assuming that there exists a standard method to export realistic traffic, there isn't enough information about how to inject newly collected data to existing ones.
Furthermore, the existing traffic injection proposals are limited to a single usage/prototype. • Generated dataset resilience: To ensure dataset resilience, variations of the dataset should be generated. This could include variation of attack scenarios, different attacks, diverse benign traffic load (different time of day load). This, with other measures, would guarantee an extended dataset lifetime. Moreover, the complexity of dataset variation generation should be kept minimal. It is key for a dataset to be provided in a raw state to allow researchers to make a choice between using stateless or stateful analysis. If a dataset is provided only in a pre-processed form, researchers may lose the ability to use stateless or stateful detection methods and hinder the detection and accuracy of their algorithms.
• Leveraging network monitoring to create real traffic: Since most of the available benchmark datasets lack real-life properties, new datasets generation should benefit from network monitoring to generate realistic background traffic  [34]. Furthermore, if real traffic could not be included in the dataset due to privacy concerns, real traffic should act as the ground truth for further traffic generation/simulation. Moreover, for special purpose networks, relying on released IoT/Critical infrastructure network architecture case studies should act as a guidance for network simulation. This should reduce the gap in terms of accurate realistic datasets. Different validation approaches should be used for IDS, however, with the advancement of IDS research, realistic experiments should be the main focus as demonstrated in  [6].
• Dataset Validation: Newly generated datasets should be validated based on network traffic validation techniques. Molnár et al. [98]  list the various metrics that could be used for this purpose. Furthermore, the similarities between real and synthetic traffic should also be evaluated as proposed in  [99] ,  [100]. • Updated Threats Taxonomy: The networking threat taxonomy presented at this work aims at helping create datasets that cover a wider range of attacks. Maintaining the taxonomy in a timely manner will keep it an upto-date reference for future IDSs research. Furthermore, the taxonomy is made available for public contribution through a GitHub repository to encourage contributions from other researchers to extend, revise and update it. While these recommendations might appear trivial at first, the majority of recent and old datasets proposed online do not conform to these guidelines as demonstrated within the previous sections.
VI. CONCLUSION This research aims at tackling the problem of having a generic taxonomy for network threats. A proposed taxonomy is presented for categorizing network attacks based on the source, OSI model layer and whether the threat is active or passive. The prominent IDS research over the past decade (2008 -2020) is analyzed. The analysis results in three main findings. First, benchmark datasets lack real-world properties and fail to cope with constant changes in attacks and network architectures, thus, limiting the performance of IDS. Second, we present a taxonomy of tools and associated attacks, and demonstrate that current IDS research only covers around 33. 3% of threats presented in the taxonomy. Third, we highlight that -whilst ML is used by 97. 25% of the examined IDS -ANN, k-means and SVM represent the majority of the algorithms used.
Active attacks are represented by a rectangle shape, whilst passive attacks are represented by an oval shape. Attacks like adware (FIGURE4 -2. 1. 3), spyware (FIGURE 4 -2. 1. 4) and information gathering (FIGURE 4 -3. 1) are considered passive attacks. DoS (FIGURE 4 -1. 1), Impersonation (FIG-URE 4 -1. 4) and Virus (FIGURE 4 -2. 1. 2) are forms of active attacks. However, some attacks cannot be considered active or passive until their usage is known. An example of this case is SQL-injection, if it is used for querying data from a database then it is passive. However, if it is used to alter data, drop tables or relations then the attack can be considered as active.
6% 2. 1. 2 Worm 0. 7% FIGURE 2: Covered Attacks in Discussed IDS (85 IDSs Manuscripts listed in Table A. 1) could be used to train machine learning models used for anomaly detection. By employing extendable datasets and a standardized method for dataset generation, alongside the advancement in ML [72], [73], zero-day detection could be integrated into anomaly-based IDSs. Later in Section IV, our presented threat taxonomy highlights the percentage of attack coverage achieved by current IDSs. To further analyze the last decade research on IDSs, it is important to consider the algorithms used. Anomaly-based IDSs are based on identifying patterns that define normal and abnormal traffic. These IDSs can be classified into subcate- gories based on the training method used as aforementioned in Section II.
These categories are identified respectively as statistical, knowledge-based and ML based. Statistical includes univariate, multivariate and time series. Knowledge- based uses finite state machines and rules like case-based, n-based, expert systems and descriptor languages. ML algo- rithms include artificial neural networks, clustering, genetic algorithms, Deep Learning (DL), etc. namely; DoS/DDoS, Probing, R2L, U2R. Moreover, only 12 attacks are listed in FIGURE2which highlights potential limitations of these IDS to cope with the broad range of attacks and zero-day attacks. To tackle the detection of zeroday attacks, there is a need to build extendable datasets that FIGURE3(a) highlights the dominance of ML algorithms employed when building an IDS. As shown, both statistical and knowledge-based algorithms are less represented.
5 -NB -RF -k-means -SVDD -NB -ANN -SVM APD -Bayesian Network Likelihood -Conditional Anomaly Detection -WSARE -AdaBoost -ABC -Fuzzy Association Rules -Fuzzy Association Rules -Genetic-based -C4. 5 BSPNN using: -AdaBoost -Semi-parametric NN -RBF -Elman NN -SNORT -Non-Parametric CUSUM -EM based Clustering FC-ANN based on: -Fuzzy Clustering -ANN -Logistic Regression -FCM Clustering -NN -OCSVM -AdaBoost -NB -Genetic Algorithm -Weighted k-NN Genetic Fuzzy Systems based on: -Michigan -Pittsburgh -IRL -DT -Ripper Rule -BON -RBF NN -Bayesian Network -NB -K-means clustering -SOM -Rule-Based -BON -ART Network -Hidden NB -SVM -DT -ANN -Mapping -BayesNet TABLE A. 2: Datasets Attacks Remarks Probing, DoS, R2L, U2R Probing, DoS, R2L, U2R Normal and abnormal Detected Attacks Probing, DoS, R2L, U2R Normal and abnormal Probing, DoS, R2L, U2R Probing, DoS, R2L, U2R Attack and Non-Attack Probing, DoS, R2L, U2R DoS Probing, DoS, R2L, U2R Probing, DoS, R2L, U2R Probing, DoS, R2L, U2R U2R 1) Illegal activity in imported containers 2) Anthrax 3) DoS and R2L Probing, DoS, R2L, U2R Probing, DoS, R2L, U2R Credit Card Fraud Probing, DoS, R2L, U2R Probing, DoS, R2L, U2R Probing, DoS, R2L, U2R Probing, DoS, R2L, U2R 13 Attack Types Probing, DoS, R2L, U2R Probing, DoS, R2L, U2R Probing, DoS, R2L, U2R -Nachi / Netbios scan -DDoS UDP/ TCP flood -Stealthy DDoS UDP flood -DDoS UDP flood + traffic deletion Popup spam -SSH scan + TCP flood Probing, DoS, R2L, U2R DoS / DDoS Probing, DoS, R2L, U2R -Probing -DoS Probing, DoS, R2L, U2R Probing, DoS, R2L, U2R Probing, DoS, R2L, U2R Probing, DoS, R2L, U2R DoS/DDoS -SQL Injection -XSS Probing, DoS, R2l, U2R -DT (Enhanced J48, J48, ADTree, DecisionStump, RandomTree, Institute Attacks Remarks -SA Ensemble DTs: -Decision Stump -C4.