# Models versus Datasets: Reducing Bias through Building a Comprehensive IDS Benchmark
## 2 Literature Review
### Related Work  
The performance of conventional ML approaches plateau and remains stagnant once a certain computational threshold has reached; in contrast, DL performance grows rapidly as the dataset size grows [10]. Therefore, our focus in this study is only on deep learning-based models. Due to the continuous changes in network attacks every year, it is important to train models on diverse datasets consisting of different attacks and also provide the ability to quickly retrain models on new traffic patterns to evaluate IDS performance [20]. The security experts need to be able to interpret the model's output consistently and understand the factors and features that cause the model to make certain predictions [21]. Features have a significant impact on ML model predictions and their quality [22]. Many research studies have recently been performed on explaining model outputs [23; 24; 25; 26]. The two common approaches to explain model output are post hoc, also known as model-agnostic and model-specific approaches. The model-agnostic approach has received much attention in the research community [27]. Model-agnostic approaches to explain machine learning models provide many advantages over model-specific interpretation approaches. One of the important ones is to allow researchers to build any machine learning model they desire without worrying about the model's complexity and interpretability [28]. For a model-specific interpretation, it is difficult to switch to a new or existing model with a slight change in the model settings.  
Many researchers have proposed deep learning-based IDS for IoT environments; however, few have explained their model's output. For example, in [23], the authors tried to explain their model's output, but their solution is not comprehensive. They only looked at a single dataset NSL-KDD to train the model and predict attacks. In another study, the authors only investigated the robustness of interpretable models [29]. Some researchers used Convolutional Neural Networks (CNN) to detect diseases using localized model interpretation only [30]. In another research, the authors used the NSL-KDD dataset to train a deep neural network and applied LIME and SHAP to explain the model behavior [31]. However, their solution only focuses on explaining a single classifier and using a single dataset, \"NSL-KDD,\" which is old and lacks recent attacks [32]; they have not used their research to find optimal classifier settings across various datasets. In [33], the authors used a localized explanation to interpret the model using an old dataset and Multilayer Perceptron (MLP).  
This paper is a step forward from our previously published paper [34] which proves the bias issues in models that are trained on a single dataset. In our previous work, first, we replicated various DL-based models from many researchers using the same dataset, classifier settings, and the preprocessing steps; second, we proved the bias issue by switching the dataset to a new benchmark dataset that caused the good performing classifier to drop their performance considerably. In this paper, we overcome this limitation. Instead of building a model on a single dataset and a single DL algorithm, we incorporated multiple datasets and multiple algorithms to diversify the input data and the processing algorithm for optimal performance.