# A Self-Adaptive Approximated-Gradient-Simulation Method for Black-Box Adversarial Sample Generation
## 4 Experiment and Evaluation  
### Experimental Setting  
To evaluate the performance of the proposed SAGM, different experiments were conducted on the two image datasets, namely MNIST and CIFAR10, which have different image dimensions and category numbers. MNIST is a dataset of 28 \(\times\) 28 gray-scale handwritten digits in the range of 0 to 9, containing 60,000 training samples and 10,000 testing samples. CIFAR-10 is a 32 \(\times\) 32 color image containing 10 categories, with 6000 images per category. One DNN model, namely LeNet5 [44], was pre-trained in the MNIST dataset, while three DNN models, namely ResNet18 [45], AlexNet [46], and GoogLeNet [47], were selected as the CIFAR-10 dataset. For MNIST and CIFAR-10, all training samples were used to train the accuracy of the DNN used in this paper, and all test samples were used to verify the robustness of the DNN. Moreover, for this DNN model, parameters were consistent with the original framework. All the experiments were carried out on a computer with an Intel Core i-7 CPU and 16 GB RAM. LeNet5 on the MNIST achieved an accuracy of 98.44%, and ResNet18, AlexNet, and GoogLeNet on the Cifar-10 achieved accuracies of 98.90%, 96.46%, and 98.38%, respectively. Furthermore, four attack algorithms, namely FGM [48], PGD, CW, and BMI-FGSM were selected for comparison with our attack algorithm. The curves of training accuracy and loss are shown in Figures 5 and 6, respectively. are presented in Table 1. The results in Table 1 show that the SAGM has the highest value on ASR; therefore, the proposed SAGM has better performance. For the CIFAR-10 dataset, the ASRs of five attack methods on ResNet18 and AlexNet are shown in Table 2. PGD achieved a 100% success rate on ResNet18 and the SAGM, but slightly lower than the SAGM on AlexNet. Therefore, the SAGM and BMI-FGSM a have better performance among these attack algorithms. Transferability AEs generated for one ML model can be used to misclassify another model, even if both models have different architecture and training data. This property is called transferability. The adversarial examples of the SAGM for MNIST and Cifar10 are shown in Figures 7 and 8, respectively. The proposed algorithm was found to have a good effect. Robustness is a metric used to evaluate the flexibility of DNNs to adversarial examples. The model robustness to adversarial perturbations is defined as the following Equation (22). \[\Delta_{adv}(x,p)=\underset{\delta\in[0,1]^{n}}{\text{min}}\parallel \delta\parallel_{p} \tag{22}\] \[s.t.\ p(x,t)p(x+\delta,t)\leq 0\]  
The robustness of the three DNNs on the three norms is shown in Table 3. The smaller the value, the smaller the distance between the adversarial examples and the real samples. The best mean values are shown in bold. The results show that the samples generated based on the white-box attack have a better adversarial distance than the black-box attack because it utilizes the DNN gradient. To further discuss the sensitivity of each class of images, we selected the testing samples from the CIFAR-10 dataset on AlexNet and ResNet18 to verify the sensitivity. The  
Figure 8: The adversarial examples of SAGM for Cifar10.