# Models versus Datasets: Reducing Bias through Building a Comprehensive IDS Benchmark
## 2 Literature Review
### Feature Importance  
Over the past many years, extensive research has been performed on building an optimal IDS for an IoT environment. However, researchers have been challenged to optimally handle the changing network traffic patterns, technology shifts, and large data volume management [67]. An essential operation in machine learning (ML) is finding the best features before training the models. Unfortunately, there are no commonly agreed-upon approaches on feature selection that can be applied to every problem on hand. Researchers sometimes use their judgment, experience, or statistical techniques such as Principal Component Analysis (PCA) to find the least number of optimal features. Each of these approaches aims to effectively handle large data sets with a large number of features without compromising model performance and accuracy. Once the representational features are extracted, they are used to train the ML classifiers and are applied to test the anonymous traffic patterns [68]. Selecting the important features is commonly performed manually in traditional ML approaches. The process is challenging, labor-intensive, time-consuming, and is prone to errors [57]. The feature selection computation time increases when the input dataset has a large number of features [69]. On the other hand, deep learning algorithms perform well on large datasets without explicitly performing feature selection [70]. However, some researchers still find benefits in performing feature selection in deep learning-based models [56]. For example, in one study, the authors used the NSGA-ii-aJG feature selection method with the CNN + LSTM model and achieved 99.03% accuracy [71]. On the other hand, in another study, the authors used DL's capabilities to auto-select features to detect anomalies; their proposed LSTM model achieved a 99.98% precision score compared to the SVM model, which could only achieve an 88.18% precision score [57]. Deep learning models are inherently complex with a large number of hidden parameters, complex settings, multiple layers, and hidden nodes. Because of the complex nature of DL models, many researchers only report model settings and performance results but nothing about the output and findings of the model itself. Some researchers try to justify their classifier performance using techniques such as cross-validation, but DL models could still fail to learn important hidden representations that an expert human in the field might consider necessary. Similarly, a model may consider certain features important, which an expert human may not consider necessary. Explaining model output is essential for various reasons, including (a) human curiosity about specific predictions [72], (b) making the model explainable to naive humans, (c) enhancing and redesigning the model by analyzing its output to gain optimal performance, (d) understanding why specific wrong predictions were made, and (e) improving trust and confidence in the model's decisions. Once a model is analyzed as a whole, or when individual predictions are deeply analyzed, it is assumed that models can be redesigned with a limited set of features for better results and performance. When a model is explainedlocally, its scope is limited to single-input data instance only. Individual probabilistic scores are calculated to express the predictions for an individual instance compared to all predictions. On the other hand, models can be explained as a whole with a global scope. The approach to explaining models can be built within a DL model itself or can be applied as a post hoc approach. However, model explanation within a model has a restrictive scope and can lead to significant changes with a slight change in model settings. On the other hand, researchers have shown interest in the post hoc approach that uses ad hoc methods and techniques to explain any model already built [73]. It is not easy to interpret complex deep learning models such as ensemble or hybrid models.