## 6 Fuzzing Frameworks  
There are several open-source frameworks available to test an AI/ML against existing adversarial example generation attack algorithms. These frameworks offer robust implementations of several standardised, state-of-the-art adversarial attacks, defence and detection schemes, robustness certifications, metrics, and formal verifications. As it was mentioned in the introduction, crafting adversarial examples using one of such frameworks as a rule requires specifying a target model that takes an input and makes a prediction, a criterion that defines what an adversarial perturbation is, a distance measure of the perturbation size, and an attack algorithm that takes the input and its label as well as the model, the adversarial criterion and the distance measure to generate an adversarial perturbation. All the frameworks we managed to find are implemented in Python which is the most popular programming language for developing AI/ML applications. As with most of the Python libraries, one of the aforementioned adversarial frameworks can be easily installed via pip which is a standard package installer for Python. Most of the frameworks found are able to interface with most popular AI/ML libraries such as PyTorch, Keras, TensorFlow, Theano and several others. Furthermore, many frameworks also provide a straight-forward way to add support for other AI/ML libraries. Study [172] describes Cleverhans which is a Python library that provides standard adversarial example generation technology and reference implementations for adversarial training, which can be used to develop more robust AI/ML models and provide standard model performance benchmarks for confrontation settings. The target model can be either a Keras Sequential or a raw TensorFlow model. A custom target model can also be implemented by inheriting from the certain model class which includes the minimum amount of properties and methods the custom target model should implement. The attacks implemented include L-BFGS attack [19], FGSM attack [137], C&W attack [144], ElasticNet attack [145], BIM attack [138], PGD attack [139], MIM attack [140], JSMA attack [20], DeepFool attack [142], fast feature attack [141], and SPSA attack [151]. As one can notice, all of the attacks implemented are white-box adversarial example generation based attacksagainst neural networks. In addition to the attack implementations, the framework also implements adversarial training which is essentially adding adversarial examples to the training data to allow the target model to generalise more efficiently [137]. The target model is then suggested to be tested using the dataset that includes both the legitimate and adversarial samples. The resulting accuracy value can be used as a metric to benchmark the target model robustness. There are however no metrics implemented to benchmark the attack algorithms themselves. Adversarial Robustness Toolbox (ART) is another Python library which aims to defend AI/ML models against adversarial threats [173]. In addition to the support of Keras and Tensorflow, ART contains a functional API enabling the integration of models from various ML libraries such as Pytorch, MXNet, Sklearn, XGBoost, LightGBM, CatBoost, and GPy. Combining several classifier models into an ensemble is also supported. In order to test black-box attacks the most basic classifier properties and methods should be implemented using the corresponding abstract class provided. In addition, there are specific interface implementations that should be employed when implementing a custom neural network or a decision tree classifier. In addition to the attack algorithms implemented in Cleverhans, the attack list in ART includes UAP attack [147], NewtonFool attack [143], spatial transformation attack [174], ZOO attacks [159], boundary attack [156], SimBA attack [153], GeoDA attack [160], adversarial patch [175], decision tree attack [133], high confidence low uncertainty (HCLU) attack [176] and HopSkipJump attack [158].