Poster: Pillars of Sand: The current state of Datasets in the ﬁeld of Network Intrusion Detection Gints Engelen imec-DistriNet KU Leuven Leuven, Belgium gints.engelen@kuleuven.be Robert Flood University of Edinburgh Edinburgh, UK s1784464@ed.ac.uk Lisa Liu School of Engineering and IT University of New South Wales Canberra, Australia l.liuthorrold@adfa.edu.au Vera Rimmer imec-DistriNet KU Leuven Leuven, Belgium vera.rimmer@kuleuven.be Henry Clausen University of Edinburgh Edinburgh, UK henry.clausen@ed.ac.uk David Aspinall University of Edinburgh London, UK Wouter Joosen imec-DistriNet KU Leuven Leuven, Belgium wouter.joosen@kuleuven.be Abstract—Network Intrusion Detection Systems play a crit- ical role in protecting network architectures from harm. In the past decade, Machine Learning has moved to the fore- front of research in this ﬁeld, with many approaches resulting in great performance on benchmark NIDS datasets. The rel- evance of these performance results is however directly tied to the quality of the benchmark datasets used for training, which have so far not been subjected to thorough analysis. As part of our work, we have performed a large-scale manual investigation of the most commonly used publicly available NIDS datasets, where we have uncovered numerous errors due to problems in data pre-processing, attack simulation and labelling. We also highlight the lack of variability in both benign and malicious trafﬁc, which often renders the classiﬁcation task trivial. To quantify this variability, we have devised an automated methodology that can be applied without requiring expert domain knowledge. Nevertheless, we believe it is vital for any NIDS benchmark datasets to undergo a thorough manual analysis before being widely adopted. As a follow-up of our previous work where we provided an improved version of the CICIDS 2017 dataset, we are also actively working on improving the CSE-CIC- IDS 2018 dataset, which we intend to release to the research community. Index Terms—network intrusion detection, machine learning, benchmark dataset, data collection. 1. Introduction Network Intrusion Detection Systems (NIDS) are de- vices that are placed at strategic locations within a network infrastructure in order to protect it from internal and external threats. These threats range from attempts to gain unauthorised access to the network, to large-scale DDoS attacks that aim to disrupt the services of the network’s hosts. With attackers becoming more sophisticated, new threats are emerging on a daily basis, and traditional rule- based Intrusion Detection Systems are at risk of being overwhelmed by the sheer number of zero-day attacks. This is why, over the past decade, NIDS research has gravitated towards Machine Learning (ML), which does not rely as much on manual updates in order to detect new attacks. Research in this ﬁeld has shown a lot of promise. High performance results on benchmark NIDS datasets [1]–[3] seem to indicate that using ML for NIDS is a solved prob- lem. However, the obtained results are heavily dependent on the quality of the used datasets. While some research has highlighted the disparity between trafﬁc found in these datasets and that of a real-world environment [4], the network trafﬁc found in these datasets has generally not been subjected to a thorough manual investigation. In earlier work, we have shown that the CICIDS 2017 dataset [5] suffered from a multitude of issues which made its use as a benchmark dataset questionable [6]; as part of that work we released an improved version of this dataset. Following up on that, we have performed a large- scale manual analysis of ﬁve modern and widely-used NIDS datasets, where we uncovered a wide range of issues pertaining to labelling, attack simulation, documentation, and network trafﬁc realism. As part of this work, we are working on improving the CSE-CIC-IDS 2018 dataset, speciﬁcally making sure that the ground truth of all labels is as accurate as possible. Usi