# ExtendAIST: Exploring the Space of AI-in-the-Loop System Testing
## 3 Extendaist
### Design
\[\text{Minimization }c\cdot\parallel x-x^{{}^{\prime}}\parallel_{2}^{2}+loss_{f,t}(x^{{}^{ \prime}})\] (7)
* FGSM. To generate adversarial examples in a quick way, [PERSON] et al. [23] presented the _Fast Gradient Sign Method_ (FGSM) based on the norm \(L_{\infty}\). As shown in Equation (8), the adversarial example \(x^{{}^{\prime}}\) depends on the single-step \(\epsilon\) and the gradient sign of loss function \(loss_{f,t}(x)\), which determines the direction that increases the probability of the targeted class \(t\). \[x^{{}^{\prime}}=x-\epsilon\cdot sign(\
abla loss_{f,t}(x))\] (8)* BIM. To improve the accuracy of adversarial examples, [PERSON] et al. [24] proposed the _Basic Iterative Method_ (BIM) by replacing single-step \(\epsilon\) in FGSM with multiple smaller steps \(\alpha\) and minimizing the loss function for directing at the targeted label \(t\). Equation (9) indicates that the adversarial example generated from the \(i\)th step depends on that from the last step iteratively. \[x^{{}^{\prime}}_{i}=x^{{}^{\prime}}_{i-1}-clip_{\epsilon}(\alpha\cdot sign( \
abla loss_{f,t}(x^{{}^{\prime}}_{i-1}))),x^{{}^{\prime}}_{0}=0\] (9)
* ILCM. [PERSON] [25] further proposed the _Iterative Least-likely Class Method_ (ILCM) by taking the target label with the least likelihood that is the most difficult to attack rather than the label with the most possibility in BIM. * JSMA. [PERSON] et al. [26] proposed the _Jacobian-based Saliency Map Attack_ (JSMA) to obtain a saliency map, which indicates the influence of each pixel on target label classification. Therefore, adversarial example can be generated with small perturbation by changing pixels with the greatest influence. * One pixel attack. The one pixel attack [77] is presented extremely to generate adversarial image by changing only one pixel of the seeded image. * C&W attack. To get an adversarial image with less distortion on the seeded image, [PERSON] and [PERSON] [78] proposed applying three distance metrics--\(L_{0}\), \(L_{2}\), and \(L_{\infty}\) norms--to have a target attack on neural network. * Universal perturbation. The _universal perturbation_[79; 80; 81] is proposed for the misclassification of entire test dataset instead of misclassifying one specific test case. In other words, the adversarial examples are generated from the same perturbation and recognized as the same target label. #### 4.2.3 Generative Adversarial Examples (EP_GAE)

This sub-extension point EP_GAE is provided to allow researchers to generate test data with the generative adversarial nets, which aims at delivering data that cannot be distinguished from the training data or generator. * Generative adversarial nets. The Generative Adversarial Nets (GAN) [19], including a generative model \(G\) and a discriminative model \(D\), are proposed to generate samples by learning and try to fool the discriminative model. The purpose of generative model is producing samples by learning the training data distribution that causes the discriminative model making a mistake. The discriminative model must determine whether the input sample is from training data or generative model \(G\). #### 4.2.4 Metamorphic Testing Based Strategy (EP_MT)

Metamorphic testing (MT) [82; 83; 84] is an approach to alleviate the test oracle problem by checking whether the certain property of program is satisfied instead of comparing the expected output with the actual output since some expected outputs are expensive to compute. Such a property is referred to as the _metamorphic relation_ (MR) of the program function. MT is also used as a strategy of test case generation since it generates follow-up test cases according to source test cases and related MRs. Therefore, metamorphic testing based strategy is regarded as a sub-extension point in ExtendAlST, namely EP_MT, to generate test data for AIS. The existing applications of MT on AIS are as follows. * Classifier testing.