SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Firstly, we would like to emphasize that certain gradient computation innovations (e.g., IR [60], VT [57], and TAIG [21]) incorporate input augmentations for averaging the gradients obtained from multiple randomly augmented inputs at each iteration. These methods require high computational cost, and comparing them with other methods in the same category directly is unfair, as the other methods can also apply random augmentation and gradient averaging to boost their performance. Taking VT [57] as an example, it introduces random noise to the input and perform backpropagation for 20 times at each iteration. Similar for IR and TAIG. We compare them with their corresponding baselines that employ the same input augmentations and keep the same number of backpropagation operations at each iteration. That is, the corresponding baselines of VT and IR (i.e., VT-baseline and IR-baseline) perform multiple rounds of backpropagation with UN and DP augmentations, respectively, at each
(a) IR
(b) VT
Figure 3: Comparing IR, VT, and TAIG with their corresponding baselines. The dotted lines indicate AAA (lower indicates more powerful attack).
(c) TAIG
iteration of optimization, and the corresponding baseline of TAIG (i.e., TAIG-baseline) perform both input scaling and UN augmentations. Figure 3 reports the comparison results. It can be seen that the simple baselines achieve even lower AAA, indicating that the most effective factor of these methods may be input augmentation and gradient averaging, instead of what were claimed. Based on these findings, we strongly recommend that newly designed methods, which incorporate input augmentations, either explicitly or implicitly, should be carefully compared with reasonable baselines that adopt the same mechanisms to show their effectiveness.
There are still  $10+$  methods to be evaluated and compared. Previous work often compares using a simple optimization back-end, e.g., I-FGSM or MI-FGSM. As we have obtained a combination of optimization strategies (i.e., UN-DP-  $\mathrm{DI}^2$  -TI-PI-FGSM) which has been proven to be more powerful in Section 4.2, we further introduce it as a more advanced optimization back-end. It aids in better exploring true advantages of compared methods. To ensure optimal performance across different substitute models, we employed a validation set consisting of 500 samples that were distinct from the test examples tune hyper-parameters of compared methods. The hyper-parameters that yielded the best results on the validation set were then adopted for testing. The hyper-parameters with each substitute model are tuned using models whose name have been mentioned in Section 3.3. The attack performance is evaluated not only on these "validation" models, but also on 15 more victim models which are distinct from them. The detailed hyper-parameters are reported in Section F.
In Table 1 (column 2-11), we report the AA achieved by utilizing each model as the substitute model to attack the 9 "validation" models under the  $\ell_{\infty}$  constraint. AAA (which is the average AA over all substitute models) is provided in the 12-th column in the table. Note that some methods are unable to be performed on some architectures. For instance, ConBP [72] suggests replacing ReLU with a softplus function in the backward pass to ensure smooth gradient backpropagation, making it not suitable to substitute models that are equipped with smooth activations only. SGM [65] is not applicable to substitute architectures without skip-connections (e.g., VGG-19). PNA [62] and SE [38] focus on vision transformers, and thus they are not suitable to most convolutional substitute models. For "substitute model training" methods, ResNet-50 is commonly chosen as the substitute model, as is adopted in the official GitHub implementations of these methods, and we leave the exploration of training with more advanced substitute architectures to future work. In addition to the  $\ell_{\infty}$  experiment, we also conduct evaluate with the  $\ell_{2}$  constraint and the results are provided in Section A.
When I-FGSM [24] is simply applied as the optimization back-end, NAA [74] beats the other gradient computation innovations and achieves the lowest AAA with a value of  $72.65\%$  (see the upper half of Table 1), while RFA demonstrates the best results among all "substitute model training" methods. Table 1 shows that the most suitable gradient computation strategy can be different for issuing attacks from different substitute models. In particular, if ResNet-50 or VGG-19 is chosen as the substitute model, FIA [61] seems even superior to NAA in the sense of achieving higher attack success rate and lower victim accuracy, however, NAA is the best if any other model is chosen as the substitute model.
When UN-DP-  $\mathrm{DI}^2$  -TI-PI-FGSM is introduced as the new optimization back-end, almost all "gradient computation" methods show improved performance (see the lower half of Table 1). Yet, in combination with the new baseline, the advantage of most methods becomes less obvious. SGM [65], PNA [62], and SE [38] still produce performance gains when being combined with UN-DP-  $\mathrm{DI}^2$  -TIPI-FGSM. In particular, with the new optimization back-end, SE on the DeiT-B substitute model leads to the optimal attack performance among all concerned options, in the sense of BAA, fooling the victim models to show an accuracy of only  $18.61\%$  (on average). PNA obtains the lowest WAA among all, which is  $31.50\%$ . As for substitute model training, we see that the MoreBayesian method significantly outperforms the other methods, and it fools the victim models to show an average accuracy of  $27.09\%$  using a ResNet-50 substitute model.
Table 1: Comparing the obtained AA and AAA of some "gradient computation" and "substitute model training" methods. Smaller values indicate more powerful attacks. The adversarial examples were generated under an  $\ell_{\infty}$  constraint with  $\epsilon = 8 / 255$ .
ResNet-50	VGG-19	Inception v3	EffNetV2	ConvNeXt-B	ViT-B	DeiT-B	BEiT-B	Swin-B	Mixer-B	AAA
I-FGSM Back-end
- Baseline
I-FGSM	87.79%	91.21%	93.71%	95.46%	88.32%	90.28%	90.28%	89.56%	94.81%	94.37%	91.58%
- Gradient Computation
TAP (2018) [77]	81.75%	89.80%	91.01%	93.84%	90.20%	91.90%	92.86%	92.11%	95.08%	93.93%	91.25%
NRDM (2018) [37]	82.19%	87.62%	85.29%	96.12%	94.36%	94.70%	95.02%	95.23%	95.01%	90.25%	91.58%
FDA (2019) [11]	85.11%	93.91%	89.91%	98.00%	96.27%	96.60%	95.52%	96.67%	97.56%	97.66%	94.72%
ILA (2019) [20]	74.76%	77.21%	83.38%	90.20%	84.13%	77.91%	80.62%	78.29%	89.18%	85.30%	82.10%
SGM (2020) [65]	72.56%	-	-	79.64%	71.37%	85.72%	87.04%	83.67%	90.55%	91.01%	-
ILA++ (2020) [26]	71.80%	73.60%	80.07%	88.01%	83.12%	74.50%	80.19%	77.02%	88.08%	82.08%	79.85%
LinBP (2020) [14]	75.84%	86.66%	92.87%	96.96%	89.05%	91.74%	91.26%	92.62%	95.65%	96.07%	90.87%
ConBP (2021) [72]	73.46%	85.49%	91.00%	-	-	-	-	-	-	-	-
SE (2021) [38]	-	-	-	-	-	90.15%	89.18%	89.34%	-	92.34%	-
FIA (2021) [61]	68.48%	71.86%	83.84%	89.66%	80.35%	76.06%	80.13%	82.42%	88.75%	79.13%	80.07%
PNA (2022) [62]	-	-	-	-	-	88.13%	87.14%	87.97%	93.62%	-	-
NAA (2022) [74]	70.34%	78.41%	76.37%	83.56%	63.93%	65.04%	69.02%	66.24%	79.26%	74.33%	72.65%
- Substitute Model Training
RFA (2021) [47]	47.49%	-	-	-	-	-	-	-	-	-	-
LGV (2022) [13]	74.84%	-	-	-	-	-	-	-	-	-	-
DRA (2022) [78]	48.55%	-	-	-	-	-	-	-	-	-	-
MoreBayesian (2023) [27]	63.40%	-	-	-	-	-	-	-	-	-	-
New Optimization Back-end
- Baseline
UN-DP-DI2-TI-PI-FGSM	35.70%	48.33%	58.62%	52.98%	33.64%	32.74%	36.58%	33.72%	45.24%	46.60%	42.42%
- Gradient Computation
TAP (2018) [77]	63.34%	54.64%	68.02%	68.90%	27.26%	41.48%	46.78%	34.45%	56.02%	54.49%	51.54%
NRDM (2018) [37]	51.78%	63.14%	70.76%	61.81%	40.04%	52.12%	60.89%	48.98%	77.87%	57.84%	58.52%
FDA (2019) [11]	42.62%	52.83%	60.25%	89.48%	69.01%	94.83%	83.99%	78.26%	83.19%	94.97%	74.94%
ILA (2019) [20]	37.80%	45.66%	54.99%	48.86%	30.72%	28.98%	33.28%	29.40%	56.98%	45.80%	41.25%
SGM (2020) [65]	31.97%	-	-	27.82%	20.96%	28.80%	24.77%	25.42%	24.27%	38.57%	-
ILA++ (2020) [26]	37.26%	44.77%	54.24%	48.62%	31.15%	29.49%	34.68%	29.80%	59.30%	45.66%	41.50%
LinBP (2020) [14]	38.00%	48.18%	80.28%	90.10%	18.81%	41.44%	36.89%	45.51%	70.28%	82.75%	55.22%
ConBP (2021) [72]	36.10%	48.15%	69.73%	-	-	-	-	-	-	-	-
SE (2021) [38]	-	-	-	-	-	35.93%	18.61%	24.70%	-	35.58%	-
FIA (2021) [61]	38.22%	52.38%	58.38%	76.93%	51.21%	42.31%	42.28%	59.02%	62.00%	58.65%	54.14%
PNA (2022) [62]	-	-	-	-	-	31.50%	19.76%	28.88%	29.80%	-	-
NAA (2022) [74]	38.04%	49.58%	54.66%	54.42%	32.04%	31.45%	33.34%	41.10%	50.07%	41.28%	42.60%
- Substitute Model Training
RFA (2021) [47]	43.00%	-	-	-	-	-	-	-	-	-	-
LGV (2022) [13]	31.82%	-	-	-	-	-	-	-	-	-	-
DRA (2022) [78]	51.10%	-	-	-	-	-	-	-	-	-	-
MoreBayesian (2023) [27]	27.09%	-	-	-	-	-	-	-	-	-	-
According to Table 1, vision transformers should be preferable when choosing the substitute model, as the best attack performance (i.e., BAA) is often obtained on vision transformers for many attacks. To compare the transfer performance from vision transformers to convolutional networks and from the opposite direction, we report the accuracy of victim models in predicting SGM adversarial examples generated on ResNet-50 and ViT-B as the substitute model, respectively. The results are shown in Table 2. It can be seen that transferring from vision transformers to convolutional networks seems easier. When utilizing ViT-B as the substitute model, the accuracy of convolutional networks shows a range in  $[28.32\%, 37.24\%]$ , while, with ResNet-50, the accuracy of vision transformers lies in  $[36.82\%, 48.32\%]$ . Overall, using ViT-B as the substitute model leads to lower average accuracy  $(28.80\%$  vs  $31.97\%)$  and the worst accuracy  $(37.24\%$  vs  $48.32\%)$  on victim models, which means better average and worst-case attack performance, respectively. ConvNeXt [32] that follows some designing principles of the vision transformers is also a favorable choice of the substitute model, according to our results. When performing LinBP on ConvNeXt-B, the generated adversarial examples are capable of fooling the victim models to show an average accuracy of only  $18.81\%$ , which is super close to the best attack performance that could be achieved in Table 1. Additionally, it
Table 2: The accuracy of victim models in predicting adversarial examples crafted via SGM using ResNet-50 and ViT-B as the substitute model, respectively. Smaller values indicate more powerful attacks. The optimization back-end is UN-DP-DI²-TI-PI-FGSM, and the adversarial examples were generated under an  $\ell_{\infty}$  constraint with  $\epsilon = 8/255$ .
Substitute model	ResNet -50	VGG -19	Inception v3	EffNetV2 -M	ConvNeXt -B	ViT -B	DeiT -B	BEiT -B	Swin -B	Mixer -B	AA
ResNet-50	-	2.72%	7.92%	29.42%	28.52%	48.32%	47.64%	36.82%	47.66%	38.70%	31.97%
ViT-B	30.00%	28.32%	36.40%	37.24%	33.66%	-	28.76%	15.60%	23.26%	25.92%	28.80%
is worth noting that even though ViT-B, DeiT-B, and BEiT-B share the same network architecture, they exhibit considerably different performance when acting as the substitute model. This suggests that the training procedure can also play a crucial role in improving the adversarial transferability.
If we still focus on ResNet/Inception and test adversarial examples only on these traditional models, as in many previous papers, then different conclusions will be drawn. Likewise, if we only focus on the simple optimization back-end (i.e., I-FGSM) without introducing UN-DP-  $\mathrm{DI}^2$  -TI-PI-FGSM, the conclusions will also be different, since NAA seems to be the best solution with I-FGSM.
In summary, some key takeaways are provided as follows. (i) It is essential to evaluate on a variety of substitute and victim models to gain a comprehensive understanding of the performance of a developed method. (ii) Evaluations using a more advanced optimization back-end (e.g., UN-DP-  $\mathrm{DI}^2$  -TI-PI-FGSM) should be considered. (iii) Generally, using vision transformers as substitute models yields superior attack performance comparing with using the traditional convolutional models. (iv) Among the "substitute model training" methods, the MoreBayesian method consistently enhances adversarial transferability, and it outperforms other methods when using UN-DP-  $\mathrm{DI}^2$  -TI-PI-FGSM as the back-end.
Except the results on the "validation" models, we further simulate a more practical attack scenario where 15 more victim models that are distinct from those "validation" models in Table 1 are considered. The conclusion remains consistent on attacking these models, except performing LinBP on ConvNeXt achieves the best AA, and we report the results in Section B.
4.4 Evaluation of "Generative Modeling" Methods
Another series of transfer-based attacks use generative modeling. They train a generative model to craft adversarial examples and adopt the substitute model as a discriminator. In Table 3, we compare these methods (including CDA [35], GAPF [44], BIA [75], TTP [36], and C-GSP [69]). As the BIA paper also introduce two additional modules (call RN and DA) that could be beneficial under certain circumstances, we also evaluate BIA+RN and BIA+DA and report the results in Table 3. We follow their official implementations and adopted a ResNet-152 model [16] as the substitute model. GAPF performs the best according to our results. For a comprehensive evaluation, we further compare these methods to UN-DP-DI²-TI-PI-FGSM, which is the newly developed optimization back-end. Table 3 shows that it outperforms these generative modeling methods significantly. Moreover, since these methods craft adversarial examples using a generative model, it is infeasible to directly adopt UN-DP-DI²-TI-PI-FGSM for boosting their performance.
Table 3: The performance of generative modeling methods on attacking different victim models. Their substitute model is the same ResNet-152 model. The adversarial examples were generated under an  $\ell_{\infty}$  constraint with  $\epsilon = 8 / 255$ . Smaller values indicate more powerful attacks.
ResNet -50	VGG -19	Inception v3	EffNetV2 -M	ConvNeXt -B	ViT -B	DeiT -B	BEiT -B	Swin -B	Mixer -B	AA
CDA (2019) [35]	35.20%	31.50%	57.22%	78.64%	71.96%	85.08%	90.58%	81.80%	86.98%	80.62%	69.96%
GAPF (2021) [44]	7.22%	6.20%	14.56%	47.12%	59.80%	85.76%	86.12%	72.80%	80.28%	69.26%	52.91%
TTP (2021) [36]	44.00%	32.06%	59.14%	89.68%	91.92%	94.48%	95.60%	92.16%	94.78%	85.36%	77.92%
BIA (2022) [75]	31.26%	19.16%	28.40%	75.24%	87.12%	91.24%	93.90%	87.12%	91.78%	77.20%	68.24%
BIA+RN (2022) [75]	28.04%	20.28%	41.16%	84.90%	88.52%	94.02%	95.28%	91.10%	92.64%	80.56%	71.65%
BIA+DA (2022) [75]	44.36%	28.74%	43.42%	82.74%	88.82%	90.52%	93.78%	85.90%	92.40%	76.22%	72.69%
C-GSP (2022) [69]	52.48%	43.70%	70.64%	91.58%	91.78%	90.24%	95.82%	84.66%	94.82%	86.42%	80.21%
UN-DP-DI2-TI-PI-FGSM	3.10%	7.14%	10.22%	32.38%	29.20%	49.48%	48.46%	35.06%	50.24%	46.14%	31.14%
In this paper, we have presented a benchmark for transfer-based attacks, called TA-Bench. On TA-Bench, we have implemented  $30+$  advanced transfer-based attack methods, including those focused on input augmentation and optimizer innovation, those "gradient computation" methods, those "substitute model training" methods, and those applying generative modeling. With TA-Bench, we are capable of evaluating and comparing transfer-based attacks systematically, practically, and fairly. Given comprehensive experimental results on TA-Bench, we have provided new insights about the effectiveness of these attacks, including but not limited to useful combinations of input augmentations and optimizers, reasonable choices of substitute/victim models, etc. Hoping to offer a sagacious judge of the state of transfer attacks and help future innovations in this field.
[1] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. In International Conference on Learning Representations, 2021.
[2] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[3] Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: a standardized adversarial robustness benchmark. arXiv preprint arXiv:2010.09670, 2020.
[4] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pages 2206-2216. PMLR, 2020.
[5] Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vgg-style convnets great again. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13733-13742, 2021.
[6] Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In CVPR, 2018.
[7] Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Evading defenses to transferable adversarial examples by translation-invariant attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4312-4321, 2019.
[8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
[9] Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. arXiv preprint arXiv:2303.11331, 2023.
[10] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19358-19369, 2023.
[11] Aditya Ganeshan, Vivek BS, and R Venkatesh Babu. Fda: Feature disruptive attack. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8069-8079, 2019.
[12] Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph. Simple copy-paste is a strong data augmentation method for instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2918-2928, 2021.
[13] Martin Gubri, Maxime Cordy, Mike Papadakis, Yves Le Traon, and Koushik Sen. Lgv: Boosting adversarial example transferability from large geometric vicinity. arXiv preprint arXiv:2207.13129, 2022.
[74] Jianping Zhang, Weibin Wu, Jen-tse Huang, Yizhan Huang, Wenxuan Wang, Yuxin Su, and Michael R Lyu. Improving adversarial transferability via neuron attribution-based attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14993-15002, 2022.
[75] Qilong Zhang, Xiaodan Li, YueFeng Chen, Jingkuan Song, Lianli Gao, Yuan He, et al. Beyond imagenet attack: Towards crafting adversarial examples for black-box domains. In International Conference on Learning Representations, 2022.
[76] Zhengyu Zhao, Hanwei Zhang, Renjue Li, Ronan Sicre, Laurent Amsaleg, and Michael Backes. Towards good practices in evaluating transfer adversarial attacks. arXiv preprint arXiv:2211.09565, 2022.
[77] Wen Zhou, Xin Hou, Yongjun Chen, Mengyun Tang, Xiangqi Huang, Xiang Gan, and Yong Yang. Transferable adversarial perturbations. In ECCV, 2018.
[78] Yao Zhu, Yuefeng Chen, Xiaodan Li, Kejiang Chen, Yuan He, Xiang Tian, Bolun Zheng, Yaowu Chen, and Qingming Huang. Toward understanding and boosting adversarial transferability from a distribution perspective. IEEE Transactions on Image Processing, 31:6487-6501, 2022.
A  $\ell_2$  Results
Table 4: Comparing the obtained AA and AAA of some "gradient computation" and "substitute model training" methods. Smaller values indicate more powerful attacks. The adversarial examples were generated under an  $\ell_2$  constraint with  $\epsilon = 5$ .
ResNet -50	VGG -19	Inception v3	EffNetV2 -M	ConvNeXt -B	ViT -B	DeiT -B	BEiT -B	Swin -B	Mixer -B	AAA
I-FGSM Back-end
- Baseline
I-FGSM	88.21%	92.46%	94.91%	97.49%	89.34%	91.21%	90.81%	90.46%	95.76%	95.47%	92.61%
- Gradient Computation
TAP (2018) [77]	89.06%	94.55%	95.42%	98.39%	94.51%	95.26%	95.68%	94.90%	97.19%	96.77%	95.17%
NRDM (2018) [37]	91.41%	92.36%	96.00%	98.94%	95.28%	97.00%	97.14%	97.63%	97.26%	95.43%	95.85%
FDA (2019) [11]	92.64%	96.62%	96.10%	99.16%	96.95%	97.72%	96.66%	97.00%	98.15%	98.54%	96.95%
ILA (2019) [20]	83.62%	84.54%	92.61%	96.41%	90.97%	89.22%	88.45%	88.50%	94.31%	93.80%	90.24%
SGM (2020) [65]	79.14%	-	-	89.40%	82.09%	89.48%	90.00%	90.06%	94.21%	95.26%	-
ILA++ (2020) [26]	81.01%	81.77%	91.44%	95.83%	90.42%	87.86%	88.90%	87.40%	93.78%	92.41%	89.08%
LinBP (2020) [14]	84.02%	90.56%	97.53%	98.81%	91.52%	92.99%	92.60%	93.61%	96.43%	98.18%	93.63%
ConBP (2021) [72]	82.17%	89.70%	96.71%	-	-	-	-	-	-	-	-
SE (2021) [38]	-	-	-	-	-	93.67%	91.12%	92.79%	-	95.93%	-
FIA (2021) [61]	74.04%	75.87%	90.49%	95.44%	84.89%	82.60%	85.25%	86.39%	92.47%	85.90%	85.33%
PNA (2022) [62]	-	-	-	-	-	90.56%	89.32%	90.18%	95.17%	-	-
NAA (2022) [74]	79.03%	85.49%	88.38%	94.84%	72.61%	75.96%	77.56%	75.04%	86.56%	84.52%	82.00%
- Substitute Model Training
RFA (2021) [47]	67.24%	-	-	-	-	-	-	-	-	-	-
LGV (2022) [13]	74.86%	-	-	-	-	-	-	-	-	-	-
DRA (2022) [78]	64.29%	-	-	-	-	-	-	-	-	-	-
MoreBayesian (2023) [27]	70.24%	-	-	-	-	-	-	-	-	-	-
New Optimization Back-end
- Baseline
\( UN-DP-DI^2-TI-PI-FGSM \)	43.09%	55.86%	72.13%	75.73%	45.74%	43.36%	51.06%	43.58%	63.74%	60.27%	55.46%
- Gradient Computation
TAP (2018) [77]	77.26%	65.26%	82.02%	91.82%	52.38%	70.49%	78.21%	53.32%	83.40%	72.93%	72.71%
NRDM (2018) [37]	71.28%	78.55%	86.54%	81.93%	65.57%	81.32%	85.54%	67.78%	93.14%	79.44%	79.11%
FDA (2019) [11]	58.39%	65.94%	78.50%	96.43%	79.18%	98.23%	95.88%	83.26%	95.79%	96.87%	84.85%
ILA (2019) [20]	47.80%	57.54%	73.57%	74.58%	48.88%	47.89%	64.93%	40.11%	75.82%	65.62%	59.67%
SGM (2020) [65]	38.56%	-	-	57.71%	32.25%	38.47%	36.07%	33.57%	32.94%	54.31%	-
\( ILA++ \)(2020) [26]	47.58%	56.46%	72.96%	74.80%	48.95%	48.26%	65.76%	40.88%	85.21%	65.55%	60.64%
LinBP (2020) [14]	48.62%	56.32%	89.79%	97.82%	30.87%	54.97%	50.57%	55.38%	82.27%	88.77%	65.54%
ConBP (2021) [72]	46.34%	56.39%	83.41%	-	-	-	-	-	-	-	-
SE (2021) [38]	-	-	-	-	-	53.60%	32.28%	37.83%	-	51.99%	-
FIA (2021) [61]	44.83%	58.65%	72.43%	89.03%	60.46%	51.32%	54.84%	64.98%	75.76%	68.91%	64.12%
PNA (2022) [62]	-	-	-	-	-	42.10%	29.33%	38.68%	51.56%	-	-
NAA (2022) [74]	47.22%	59.99%	72.75%	75.64%	41.47%	42.20%	47.26%	47.91%	65.69%	55.74%	55.59%
- Substitute Model Training
RFA (2021) [47]	57.76%	-	-	-	-	-	-	-	-	-	-
LGV (2022) [13]	41.28%	-	-	-	-	-	-	-	-	-	-
DRA (2022) [78]	64.14%	-	-	-	-	-	-	-	-	-	-
MoreBayesian (2023) [27]	38.90%	-	-	-	-	-	-	-	-	-	-
We evaluate the "gradient computation" methods and "substitute model training" methods under  $\ell_{2}$  constraint and provide the results in Table 4. Some  $\ell_{2}$  results are provided in this section. When I-FGSM is applied as the optimization back-end, same as the  $\ell_{\infty}$  results in Table 1 in our main paper, NAA achieves the lowest AAA (i.e.,  $82.00\%$  ) compared with the other "gradient computation" methods, while FIA beats it when ResNet-50 or VGG-19 is chosen as the substitute model. However, unlike in the  $\ell_{\infty}$  setting, SE shows consistently inferior performance when compared with the I-FGSM baseline in the  $\ell_{2}$  setting, and DRA instead of RFA achieves the best performance among "substitute model training" methods.
When UN-DP-  $\mathrm{DI}^2$  -TI-PI-FGSM is applied as the new optimization back-end, same as in the  $\ell_{\infty}$  setting, SGM, PNA, and SE provide favorable attack performance, while PNA on the DeiT-B substitute model turns out to be the best (in the sense of achieving lower BAA) and the generated adversarial examples fools victim models to show an accuracy of only  $29.33\%$  . The lowest WAA (which is  $51.56\%$  ) is obtained by PNA. For the "substitute model training" methods, the MoreBayesian method still outperforms the other methods by a large margin.