## 6 Discussion and Conclusions
As the model and the current state of the art diverge, the model suffers from hidden technical debt. Therefore, the model must be retrained to reflect the current state-of-the-art attacks and new network traffic patterns [149]. Researchers might develop and use more up-to-date datasets. Further avenues for research include semi-supervised/unsupervised ML and active learning methods that continuously update the underlying model and do not rely on labelled datasets. We identify that data-level techniques such as resampling, balancing datasets, and cross validation could have effects on robustness against adversarial examples. Further research is required to explore how the bias-variance trade-off can effect robustness. We prioritise the areas of future research, setting the agenda for research in this area. Critical areas of research include breaking the transferability of adversarial examples that would hopefully be applicable across domains. Non-visual domains including cybersecurity and cyber physical systems have been under-explored and this oversight should be rectified urgently. Further research on transformations in non-visual domains could provide useful knowledge. Detection of adversarial examples and pushing the fields of cybersecurity, intrusion detection, and cyber-physical systems will yield benefits beyond cybersecurity and may be applicable in other non-visual domains. Moreover, research is required in areas beyond instance classifiers. Areas of RNNs and reinforcement learning have been under-explored. More research is required to understand the use of domain constraints and functionality-preserving adversarial examples. Further research is needed towards effective countermeasures. Additionally, we consider that more research attention could be given to dataset resampling strategies as a defence against adversarial examples. There is a need for better robustness metrics. Some researchers simply state accuracy, and others might state the better F1-score; however, the F1-score is biased by unbalanced datasets that are widespread in intrusion detection, partly due to large numbers of benign samples. Using F1-score could lead to a false sense of security. Researchers should adopt stronger metrics such as CLEVER [59] or empirical robustness [62]. Adversarial machine learning is a critical area of research. If not addressed, there is increasing potential for novel attack strategies that seek to exploit the inherent weaknesses that exist within machine learning models; however, few works consider \"_realisable_\" perturbations that take account of domain and/or real-world constraints. Successful adversarial examples must be crafted to comply with domain and real-world constraints. This may be challenging since even small modifications may corrupt network packets that are likely to be dropped by firewalls. This necessitates functionality preservation in adversarial learning. We propose that human perception may not be the best criterion for analyzing adversarial examples. In cybersecurity domains we propose that adversarial examples must preserve functionality. Traditionally, adversarial examples are thought of as having imperceptible noise. That is, that humans cannot perceive the difference between the original and perturbed inputs. Indeed, human perception in some domains might be immaterial. For example, strategic attacks triggered at crucial moments might cause damage to CPS before any human could reasonably act. In cyber-security domains traditional gradient descent algorithms may be insufficient, although JSMA may be reasonable because it perturbs few features. Stringent constraints exist in the cyber-security domain and extreme care must be taken to create valid adversarial examples. We offer some guidelines for generating functionality-preserving adversarial examples.