## 6 Conclusions and Future Work  
This work introduced a comprehensive study of the current NIDSs proposals pointing out the lack of standard ways to perform a reliable ML-based NIDS assessment. Most of the state-of-the-art approaches do not appropriately describe or avoid necessary steps followed in the methodology used to evaluate their proposals, making it hard to perform a fair comparison and evaluation of ML-based NIDSs, and to be confident about the results published by different authors addressing similar types of problems. To overcome this issue, a methodology comprising what the authors consider are mandatory steps in NIDSs evaluation has been introduced, covering aspects in a wide range varying from the raw network dataset to the recommended performance metrics. The framework suitability has been tested with classical ML algorithms and a real and updated network dataset. Moreover, a comparison to state-of-the-art ML-based NIDS approaches has also been performed.  
The results showed that the methodology proposed can help the research community to build better NIDS solutions to allow honest comparisons. However, defining a definitive evaluation guide or framework is not a trivial task, as there could be more than one valid methodology per application field or objective, e.g., to fight against one kind of attack in specific network environments. Besides, not all the steps proposed in the methodology of this paper could always be necessary, i.e., some of them can be counter-productive depending on the context of the application. Furthermore, it is not clear what the impact is of the network dataset on the NIDS performance, where real and updated ones may improve the performance and feasibility of the solutions so that, later on, they can be deployed in real production environments.  
In summary, extra work should be done in order to propose common and standard frameworks to compare and evaluate ML-based NIDS solutions that are entirely accepted by the research community. In this sense, future work will focus on determining the impact of the network dataset and the considered methodology steps on the ML-based or DL-based NIDSs performance, with particular  
Figure 4: Comparison of the AUC performance result among the ML-based solutions proposed in this work and a previously published NIDS based on MSNM techniques.  
attention on the feature engineering procedures allowing NIDS generalization and, consequently, its real deployment in production environments.  
**Supplementary Materials:** The source code of the proposed framework is available online on [[https://github.com/ucadatalab/ff4](https://github.com/ucadatalab/ff4) ml]([https://github.com/ucadatalab/ff4](https://github.com/ucadatalab/ff4) ml). We encourage readers to contribute to the project for adding new functionalities and improvements.  
**Author Contributions:** Conceptualization, R.M.-C.; Methodology, R.M.-C. and D.U.; Software, R.M.-C., I.D.-C. and D.U.; Validation, R.M.-C., I.D.-C. and D.U.; Formal Analysis, R.M.-C., D.U. and B.D.; Investigation, R.M.-C., D.U. and I.D.-C.; Resources, R.M.-C. and I.D.-C.; Data curation, R.M.-C. and I.D.-C.; writing-original draft preparation, R.M.-C.; writing-review and editing, R.M.-C., D.U. and B.D. All authors have read and agreed to the published version of the manuscript.  
**Funding:** The authors would like to acknowledge the Spanish Ministerio de Ciencia, Innovacion y Universidades and ERDF for the support provided under contracts RTI2018-100754-B-I00 (iSUN) and RTI2018-098160-B-I00 (DEEPAPFORE).  
**Acknowledgments:** The authors acknowledge support given by the supercomputing center at the University of Cadiz as well as [PERSON] (University of Granada) whose support on anomaly detection methods based on the use of multivariate statistical techniques was greatly appreciated.  
**Conflicts of Interest:** The authors declare no conflict of interest.