# A Deep Learning Method for Lightweight and Cross-Device IoT Botnet Detection +
## 6 Limitations and Threats to Validity  
The evaluation of an IDS is a complex subject, and it depends on many factors including--but not limited to--the underlying network devices, topology and speed, the nature of the attacks and the quality of the legitimate traffic being used to infer the \"normal profile\" of the devices. For example, the normal traffic might be fraught with spurious, out-of-the-crowd behaviors and by imperfect labels. The proposed outlier-based threshold selection method mitigates the effect of the imperfections of the normative traffic. As much related work in the area, the findings of this paper must be contextualized with respect to the attacks and data of the adopted dataset. In this respect, many public security datasets have been proposed over the years; some of them have gained significant popularity and become widely consolidated benchmarks for intrusion detection algorithms and tools, such as N-BaloT. We are aware that a \"lab-made\" dataset, such as N-BaloT and many others, might be a simplification of real-life production networks; however, adopting a reference benchmark makes it possible to compare the results obtained with those of the related proposals (as discussed in Section 2). As for the application of the autoencoder and outlier-based method to a further dataset, we used the 2021 version of the CICIDS2017 dataset ([[https://downloads.distrinet-research.be/WTMC2021/tools_datasets.html](https://downloads.distrinet-research.be/WTMC2021/tools_datasets.html)]([https://downloads.distrinet-research.be/WTMC2021/tools_datasets.html](https://downloads.distrinet-research.be/WTMC2021/tools_datasets.html)), accessed on 3 January 2023) in a previous paper that investigates the issues of training effective IDS models by a single autoencoder [25]. Besides the use of a synthetic dataset, another simplification is in the format of data addressed by our study. N-BaloT is based on a network made of different IoT devices, which range from a thermostat, to doorbells and security cameras--possibly from different vendors--and network equipment (e.g., WiFi access point, switch and router); the \"nature\" of the IoT devices in real-life network environments is even more exacerbated when compared to a synthetic dataset. The model proposed in this paper can detect threats in a network made of different devices under the assumption that all the traffic is transformed into a _standard_ data format consisting of fixed-length records irrespective of the device. Nowadays, there exist many products for capturing network traffic and generating fixed-length records suited for machine and deep learning purposes. The availability of a standard data format simplifies the use of typical IDS approaches applied to common networks. As a final remark, the proposed model is not meant to cover the entire life cycle of an infection; rather, our work focuses on the last stage of botnet detection that pertains to IoT bots launching the attacks. As for any data-driven, as those involving deep learning methods, there may be concerns regarding the validity and generalizability of the results. We discuss them based on four aspects of validity listed in [38]. **Construct validity**. The study builds around the intuition that it is possible to learn a cross-device IDS model only on top of normative training traffic records related to several IoT devices. Our study, based on deep autoencoders, has the potential to drive scalable and maintainable IDS solutions that can cope with the ever-growing complexity of IoT networks. This _construct_ has been investigated with normal and attack traffic from a widely accepted benchmark by the related IoT literature. Experiments are based on the ubiquitous Keras-tensorflow deep learning framework. Overall, the study is supported by well-founded theory and practice, and the typical evaluation metrics of recall, precision, false positive rate and F1 score for comparative purposes. **Internal validity**. Our study implements several countermeasures aiming to mitigate internal validity threats.