# Models versus Datasets: Reducing Bias through Building a Comprehensive IDS Benchmark
## 1 Introduction  
Growing consumer, business, and industrial demand for advanced Internet of Things (IoT) solutions creates unique challenges to securing these devices. IoT devices are limited in resources such as memory, disk space, and processing power. Therefore, traditional security solutions (firewall, antivirus, software patches) applicable to personal computers do not adequately secure the growing number of IoT devices [1]. Unlike personal computers, these small devices not only lack built-in robust security features, but manufacturers also do not consistently deliver security patches and upgrades after selling the devices [2]. Naive users either do not change the default passwords or set strong or difficult-to-remember passwords; this makes the devices vulnerable to various cyber-attacks. By using advanced mining tools, adversaries can easily takeover these devices remotely by finding many access points and penetrating deep into the network [3]. Once compromised, thousands or possibly millions of IoT devices can be controlled by a command and control (C&C) server to launch various large-scale attacks such as Distributed Denial of Service (DDoS) on critical national infrastructure and websites. In order to secure IoT devices, a drastic change in security approaches and a paradigm shift are required [4]. A common and widely adopted network attack detection approach is known as anomaly-based (also known as behavior-based) Intrusion Detection System(IDS). This approach provides robust capabilities to detect various known and zero-day cyber-attacks by analyzing network traffic [5]. Researchers have been proposing anomaly-based IDS solutions for IoT environments using different Machine Learning (ML) and Deep Learning (DL) models trained on benchmark network datasets. Even though anomaly-based IDS provides good attack detection results, it may generate high false-positive detection results if not appropriately built [6]. An increase in false-positive results would undermine the system's performance in detecting attacks. Unfortunately, many cyber-attacks and their variants are not so easy for an IDS to detect. Several critical factors must be considered to create a proper and comprehensive security solution for IoTs to prevent large-scale attacks. The most important factors involve the quality and reliability of the benchmark datasets and their various features that contribute the most to providing a high attack detection rate. Regarding data quality and reliability, many recent IoT IDS research studies have been proposed based on very old benchmark datasets such as KDD CUP 99 [7; 8], or NSL-KDD [9; 10; 11]. These datasets lack the modern day's network traffic patterns and the various current attack information [5; 12; 13]. There are other important factors that most of the research studies lack in their proposed solution, such as (a) Many researchers report their DL model's settings but nothing about explaining the model's behavior on making specific predictions. (b) Researchers propose a DL-based IDS solution by reporting their findings on a single dataset, which produces biased results. Any model trained on a single benchmark dataset represents limited or restricted training; such models produce biased results when detecting different attack classes [14]. The goals of this research study are many-fold, including (a) to gather, extract, and analyze aggregate knowledge from various benchmark network datasets and look at the commonalities to avoid any possible bias in produced results; (b) mitigate dataset quality and reliability issues; (c) improve DL model transparency; (d) extract top features that influence the model predictions the most. Our results will provide insights and several benchmarks to future researchers when building a comprehensive IDS for IoTs. Some of the benchmarks would include finding which DL models perform well on which dataset; similarly, finding which DL models converge quickly and are suitable for a resource-constrained IoT environment.