SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Qizhang Li $^{1,2}$ , Yiwen Guo $^{3*}$ , Wangmeng Zuo $^{1}$ , Hao Chen $^{4}$
<sup>1</sup>Harbin Institute of Technology, <sup>2</sup>Tencent Security Big Data Lab, <sup>3</sup>Independent Researcher, <sup>4</sup>UC Davis {liqizhang95,guoyiwen89}@gmail.com wMZuo@hit.edu.cn chen@ucdavis.edu
The adversarial vulnerability of deep neural networks (DNNs) has drawn great attention due to the security risk of applying these models in real-world applications. Based on transferability of adversarial examples, an increasing number of transfer-based methods have been developed to fool black-box DNN models whose architecture and parameters are inaccessible. Although tremendous effort has been exerted, there still lacks a standardized benchmark that could be taken advantage of to compare these methods systematically, fairly, and practically. Our investigation shows that the evaluation of some methods needs to be more reasonable and more thorough to verify their effectiveness, to avoid, for example, unfair comparison and insufficient consideration of possible substitute/victim models. Therefore, we establish a transfer-based attack benchmark (TA-Bench) which implements  $30+$  methods. In this paper, we evaluate and compare them comprehensively on 25 popular substitute/victim models on ImageNet. New insights about the effectiveness of these methods are gained and guidelines for future evaluations are provided. Code at: https://github.com/qizhangli/TA-Bench.
In recent years, deep neural networks (DNNs) have demonstrated unprecedented success in various applications. However, the success comes at a price: DNNs are vulnerable to adversarial examples crafted by adding imperceptible perturbations to inputs (e.g., images). The existence of adversarial examples poses a significant threat to the security and reliability of DNNs, especially in safety-critical applications such as autonomous driving, biometrics, and medical image analysis.
There are many different ways of generating adversarial examples and performing attacks. Transfer-based attacks, that are capability of compromising DNNs without having access to their network architecture and parameters, have been widely studied over the past few years. To issue such transfer-based attacks, an attacker first collects a substitute model or a set of substitute models, then computes gradients on the substitute model(s) and perform optimization based on the gradients. Performance of the attacks largely relies on the transferability of the generated adversarial examples. Over the years, a large number of methods have been proposed to improve the adversarial transferability. The methods innovate various aspects of the attack procedure, ranging from substitute model training [47, 78, 13, 27] to gradient computation (that modifies loss or forward/backward architectures given well-trained substitute models) [77, 11, 20, 14, 65, 26, 60, 57, 61, 74, 21, 15, 74] to input augmentation [67, 7, 28, 58] and the optimizer that applies the gradients [6, 28, 59]. Some methods further propose to train a generative model using additional data for obtaining transferable adversarial examples [35, 44, 75, 36, 69].
* Yiwen Guo leads the project and serves as the corresponding author.
Despite all the progress, there still lacks a standardized benchmark that could be taken advantage of to compare these methods systematically, fairly, and practically. Existing evaluations in the literature suffer from several limitations. First, there is a paucity of variety in the tested models, while certain methods may exhibit superior performance only with specific substitute/victim architectures. Second, many methods verified their efficacy only with a basic optimization back-end, e.g., I-FGSM [24], despite the existence of more advanced optimization including input augmentations. It is hence unclear whether the benefits of an innovation in these methods can be similarly brought by incorporating these more advanced optimization techniques, which could undermine the reliability of the evaluations.
To address these problems, we create a transfer-based attack benchmark (TA-Bench), allowing researchers to compare a variety of methods in a fair and reliable manner. We believe that TA-Bench will foster the development of adversarial machine learning and inspire effective ways of generating adversarial examples for evaluating the robustness of DNNs. Our main contributions include (but not limited to) the following items.
A More Advanced Optimization Back-end. We have evaluated various combinations of input augmentation mechanisms and optimizers on our TA-Bench. Our results convey that, with a few exceptions, combining more types of augmentations leads to more powerful attacks, and it could be more reasonable to evaluate the effectiveness of input augmentation mechanisms and optimizers in combination rather than in isolation. Besides, the obtained combination is suggested to be a more advanced optimization back-end for evaluating the "gradient computation" methods and "substitute model training" methods.
Insightful Observations from Comprehensive Evaluations. We consider various substitute/victim models that are considered popular, including CNNs (e.g., ResNet-50 [16], VGG-19 [49] with batch normalization, Inception v3 [49], EfficientNetV2-M [51], and ConvNeXt-B [32]), vision transformers (e.g., ViT-B [8], DeiT-B [54], Swin-B [31], and BEiT-B [1]), and a MLP (MLP-Mixer [53]). Comprehensive results on our TA-Bench systematically show how the performance of transfer-based attacks varies with different choices of substitute/victim models. These results demonstrate that adopting transformers as the substitute model generally yields superior attack performance for "gradient computation" methods, comparing with using traditional convolutional models (as the substitute model).
A Unified Codebase. We offer an open-source codebase for TA-Bench, featuring a well-organized code structure that can effectively accommodate a diverse range of transfer-based attacks, as well as various substitute/victim models. It provides a unified setting for evaluations, ensuring consistency and reproducibility in experimental results. The code is at https://github.com/qizhangli/TA-Bench.
Transfer-based Attacks. Transfer-based attacks emerged as a result of the discovery that adversarial examples are not only powerful against the model they were generated on but also effective against other models. By exploiting such a discovery, attackers can use adversarial examples created on some substitute models to attack the victim model, and a series of transfer-based attacks have been proposed. These methods have innovated various aspects of the attack procedure, and we will carefully discuss them in Section 3.2.
Related Benchmarks. There have been several libraries or benchmarks for generating adversarial examples, such as CleverHans [39], Foolbox [41, 42], RobustART [52], Torchattacks [23], etc. However, these benchmarks only cover a very limited number of transfer-based methods, as they were developed for evaluating the robustness of DNNs against not only black-box attacks but also white-box attacks. In particular, most of the methods are outdated and only considered as baseline methods due to the rapid development in this field. By contrast, our TA-Bench focuses on transfer-based attacks, and it implements  $30+$  methods in order to perform systematical, practical, and fair comparisons. A contemporary benchmark [76] also evaluates many transfer-based attacks and it draws attention to the stealthiness of the adversarial examples. Our contributions are mostly orthogonal to theirs, as we focus on fair and practical comparison between different methods. We will show some previous conclusions might be overthrown in practice. We provide new insights and guidance for evaluation to assist the future work in the community.
In a black-box scenario, the adversary has limited access to the victim model. Transfer-based attack aims to compromise the victim model by generating adversarial examples on a substitute model or a set of substitute models. Recent transfer-based methods, with few exceptions [25, 48], tested in a setting where: 1) training data of the victim model is accessible and training/fine-tuning a model on such data is possible for the adversary [47, 78, 13, 27, 40, 35, 44, 75, 36], or 2) the adversary is able to collect at least one substitute model trained using the same dataset that the victim model learned from [37, 77, 22, 11, 20, 14, 65, 26, 60, 57, 61, 74, 21, 15, 74, 24, 34, 67, 7, 58, 6, 28]. We follow this assumption for setting up the benchmark. Specifically, nothing except for the training data is known about the victim model for performing attacks in our threat model, i.e., the adversary has no idea about the pre-processing pipeline, architecture, and parameters.
Adversarial examples on the benchmark are all obtained by performing pixel-wise perturbations to benign images under an  $\ell_p$  constraint.
3.2 Methodologies
In order to compare different methods more reasonably, we roughly divide existing methods, based on their main innovations, into four categories highlighted as follows.
Input Augmentation and Optimizer. To craft an adversarial example on any given substitute model, the perturbation can be optimized as in the white-box setting. Gradient-based iterative optimization is commonly utilized, in which the perturbation is initialized to a zero tensor (e.g., in I-FGSM [24]) or a random tensor whose entries are sampled from a distribution (e.g., in PGD [34]). Image data augmentation has been considered for generating transformation-robust perturbations to each benign example, for example in diverse inputs I-FGSM  $(\mathrm{DI}^2$ -FGSM) [67], translation-invariant I-FGSM (TI-FGSM) [7], scale-invariant I-FGSM (SI-FGSM) [28], and Admix [58]. Several attacks also innovate by taking advantage of the momentum optimizer, e.g., momentum I-FGSM (MI-FGSM) [6], Nesterov I-FGSM (NI-FGSM) [28], and pre-gradient guided momentum I-FGSM (PI-FGSM) [59]. In general, these methods are all architecture-independent.
Gradient Computation (DNN-Specific). There is a belief that improved adversarial transferability can be achieved by modifying the loss or the backpropagation process. For backpropagation, both the forward and the backward pass can be altered to achieve powerful attacks, and a series of methods, including SGM [65], LinBP [14], ConBP [72], PNA [62], and SE [38] have been proposed. Some methods advocate loss terms obtained on a middle layer of the substitute model (e.g., NRDM [37], TAP [77], FDA [11], ILA [20], ILA++ [26], FIA [61], NAA [74]), while other methods stick with loss computed on the final layers (e.g., IR [60], VT [57], and TAIG [21]).
Substitute Model Training. Though most prior work uses off-the-shelf models (that could be collected on the Internet) directly as substitute models, some methods advocate fine-tuning these models or even training new ones to better suit the goal of achieving transferable adversarial examples. For instance, RFA [47] suggests adopting adversarial training to obtain substitute models. DRA [78] fine-tunes the substitute model to push the adversarial examples away from the distribution of their benign counterparts during performing attacks. LGV [13] fine-tunes the substitute model with a high learning rate and to collect a set of models on the training trajectory. A very recent method proposed by Li et al. performs fine-tuning in order to obtain Bayesian substitute models [27].
Generative Modeling. In addition to the above mentioned attacks, there are another line of transfer-based attacks that use generative models to craft adversarial examples. These methods often require additional data for training the generative model. Once properly trained, the model can generate transferable adversarial examples across different victim models. Some effective methods concerned in this line include CDA [35], GAPF [44], BIA [75], TTP [36], C-GSP [69], etc.
All these mentioned methods have been implemented on our benchmark. That is, we implemented  $30+$  methods from these four categories for comprehensive evaluation and comparison of transfer-based attacks.
3.3 Victim Models and Substitute Models
With a surge of innovation in deep learning, there exists a variety of image classification DNNs. Each has its own advantages. In practice, an engineer can choose any of them to train and deploy according to his or her specific requests. An adversary whose aim is to compromise a computer vision service developed by the engineer then anticipate the generated adversarial examples transfer well to all these possible models. Hence, to make the evaluation comprehensive and practical, we consider various victim models that are considered popular, including CNNs (e.g., ResNet-50 [16], VGG-19 [49] with batch normalization, Inception v3 [49], EfficientNetV2-M [51], and ConvNeXt-B [32]), vision transformers (e.g., ViT-B [8], DeiT-B [54], Swin-B [31], and BEiT-B [1]), and a MLP (MLP-Mixer-B [53]). It is worth noting that all of these models were obtained directly from an open-source repository timm [63] on GitHub, and our benchmark can be easily updated to include new models in the future.
Having witnessed the development of image classification architectures, it is unwise for the adversary to stick with conventional models (e.g., ResNets, VGG-Nets, and Inceptions), since it is possible that generating adversarial examples on a more advanced substitute model leads to superior attack success rates. Thus, on this benchmark, we employ the 10 victim models named above to generate adversarial examples and evaluate the attack performance of all options. As will be shown in Section 4.3, many new insights can be gained from the evaluation results.
3.4 Experimental Settings and Implementation
All evaluations on our benchmark are conducted on ImageNet [43]. We randomly selected 5,000 benign examples that could be correctly classified by all the victim models, from the ImageNet validation set, to craft adversarial examples. Filenames of these benign examples will be provided, for reproducing results and testing new methods in the future. A distance metric is required to measure the magnitude of perturbations. We adopt the popular  $\ell_p$  distance for  $p\in \{\infty ,2\}$  and set the perturbation budget under  $\ell_{\infty}$  and  $\ell_2$  constraints to  $\epsilon = 8 / 255$  and  $\epsilon = 5$ , respectively, to guarantee that the adversarial perturbations are almost imperceptible. The optimization process of each compared method runs 100 iterations with a step size of 1/255 and 1 for  $\ell_{\infty}$  constraint and  $\ell_2$  constraint, respectively. For each victim model, we pre-process its input in exactly the same way as their official implementation to ensure a practical evaluation of attack performance, and note that the adversary has no idea about the detailed implementation of this pre-processing. For instance, when evaluating the performance of attacking a ResNet-50 victim, we first resize an adversarial example to  $256\times 256$  by bilinear interpolation, then crop the image to  $224\times 224$  in the center, and finally feed the  $224\times 224$  image into the victim model and evaluate whether the attack is successful or not. Implementation details about all the supported methods are provided in Section F in the Appendix. All experiments are performed on an NVIDIA V100 GPU.
For evaluating the transferability of adversarial examples, we use the accuracy of victim models for classifying the adversarial examples as a measure. Using the prediction accuracy of victim models instead of the attack success rate makes it easier to incorporate other victim models in the future, as a reasonable calculation of attack success rate often requires the benign counterparts of the adversarial examples be classified correctly by all victim models. With a specific choice of the substitute model, prior work often evaluates the average accuracy (AA) over all victim models for comparing different attacks. However, since our benchmark studies a variety of substitute models, we further evaluate the average AA (AAA), the worst AA (WAA), and the best AA (BAA) over all choices of these substitute models. Lower AAA, WAA, and BAA indicate stronger attacks and more vulnerable victim models.
We have built a codebase consisting of modular components that serve as the basis of TA-Bench. By leveraging modular design principles, the substitute and victim models, back-end methods, and hyper-parameters can be easily adapted to help the future work of the community.
4 Evaluations and Analyses
In Section 4.1, we identify a pre-processing pipeline that is more practical. In Section 4.2, we investigate an improved back-end method by evaluating the possible combinations of iterative optimization methods. We then re-evaluate state-of-the-arts under a comprehensive and unified
benchmark, which incorporates various substitute and victim models, to assess the effectiveness of these methods in Section 4.3 and Section 4.4.
4.1 Pre-processing in Practice
Modern image classification models (especially CNNs) can take images of various sizes. After investigating experimental settings of previous transfer-based attacks, we found that the adversarial examples were often evaluated by feeding them into the victim models directly (without taking pre-processing operations of the victim models into account, e.g., resize and crop) [28, 57, 59, 58] or just resize to the input size of the victim models [61, 74]. However, these victim models, when deployed, are often equipped with inference time data pre-processing to improve their effectiveness and efficiency. Getting rid of it may lead to over-estimation of adversarial vulnerability.
As mentioned in Section 3.4, on our benchmark, we follow the default pre-processing pipeline of each victim model and evaluate under the circumstances where pre-processing often exists. We observed degraded attack performance under such circumstances (see Figure 1 for results). Obviously, the adversarial examples seem less vulnerable when the default test-time preprocessing is re-introduced to each victim model. Given the same I-FGSM adversarial examples, the average accuracy of victim models increases to  $87.79\%$  (from  $86.16\%$ ) with pre-processing, confirming that ignoring pre-processing operations of victim models indeed leads to overestimation of their adversarial vulnerability. Similar observations can be made with other substitute models and other attack methods. Considering that the pre-processing pipeline of the victim model is inaccessible to the adversary, it is infeasible for the adversary to follow the same
pipeline when generating adversarial examples. On the substitute models, a safe choice is to resize each of their inputs to the default size without cropping them.
Figure 1: Attack performance with different preprocessing strategies in the target environment, and the AAA is indicated by dotted lines (lower means more vulnerable to the attack). The adversarial examples were generated using I-FGSM on a ResNet-50 substitute model under the  $\ell_{\infty}$  constraint with  $\epsilon = 8 / 255$
4.2 Evaluation of Input Augmentation and the Optimizer
To verifying the effectiveness of a newly developed computer vision architecture (e.g., vision transformers), it is common to show that its performance surpasses previous state-of-the-arts on a challenging benchmark dataset (e.g., ImageNet), and test of the newly developed architecture may utilize a combination of advanced optimization strategies (e.g., AdamW [33] + mixup [73] + cutmix [71] + stochastic depth [19]) if such a combination is beneficial [2, 55]. Novel optimization strategies are also often tested in combination with existing ones, to show their consistent effectiveness [12].
Likewise, the transferability of adversarial examples can also benefit from appropriate choices of input augmentations and the optimizer, however, ways of innovating these optimization strategies are often evaluated in isolation in this setting. On our benchmark, we, for the first time, evaluate combinations of different choices of input augmentations and the optimizer systematically. Specifically, we performed a grid search to seek the optimal combination of I-FGSM [24], PGD [34],  $\mathrm{DI}^2$ -FGSM [67], TI-FGSM [7], SI-FGSM [28], Admix [58], NI-FGSM [28], MI-FGSM [6], PI-FGSM [59], etc. Since Admix inherently includes SI-FGSM, we have SI-FGSM by default when Admix is chosen. NI-FGSM, MI-FGSM, and PI-FGSM seem not orthogonal and thus they are tested in a mutually exclusive way in our experiment. The same for I-FGSM and PGD. For TI-FGSM, an alternative implementation where inputs are translated is adopted, as it is more effective than its suggested approximation which convolves gradients. Besides the augmentations in  $\mathrm{DI}^2$ -FGSM, TI-FGSM, SI-FGSM, and Admix, we consider several other input augmentation mechanisms including adding uniform noise to the input (which was used in VT [57] and TAIG [21]) and randomly dropping some patches of the perturbation (which was used in IR [60]). We call the two methods as UN and DP, respectively. Note that in a previous work [66], Gaussian noise is added to the input in each attack iteration. In this study, we
Figure 2: Comparing different combinations of the optimization strategies. The red solid circles indicate AAA, while the grey triangles show BAA and WAA (lower indicates more powerful attack).
opt for uniform noise, as it is now more commonly adopted. It is also worth noting that the original implementation of SI-FGSM and Admix uses several augmented copies of an input and averages the gradients computed on these copies for optimization. Such an approach increases the computational complexity of performing attacks, and, in fact, all input augmentation mechanisms in this category can be improved by such an approach [76], and, for reducing computational complexity, we only craft one augmented input at each iteration.
Each possible combination is tested with every possible substitute model whose name has been mentioned in Section 3.3 to evaluate the attacking performance over other models which are considered as the victim models. The results are sorted by AAA, in a descending order from left to right, and demonstrated in Figure 2. It illustrates not only AAA but also the range between BAA and WAA (as "error bars"). We see that the optimal AAA is achieved by UN-DP-  $\mathbf{D}\mathbf{I}^2$  -TI-PI-FGSM, which is  $42.42\%$  . The detailed AA when each substitute model is chosen for crafting adversarial examples by the method is reported in Section E. Inspecting the obtained results, we found that the performance gap between MI-FGSM, NI-FGSM, and PI-FGSM is marginal. For instance, the top 3 solutions (with PI-FGSM, NI-FGSM, and MI-FGSM, respectively) lead to similar AAA  $(42.42\%)$ $42.45\%$  and  $42.46\%$  ). The performance of I-FGSM and PGD is also similar. In most cases, PGD leads to slightly inferior performance than that of I-FGSM, accordingly to our experimental results.
In general, more input augmentation mechanisms leads to more powerful attacks. Yet there are exceptions. With DP adopted, SI-FGSM and Admix that apply input scaling and mixing fail to manifest their gains regarding the adversarial transferability. In particular, the AAA increases to  $43.12\%$  and  $44.11\%$  when further adding SI and Admix to UN-DP-  $\mathrm{DI}^2$  -TI-PI-FGSM, respectively. This is in contrast to the previous belief that these two methods are effective, and a possible explanation is that, when adopting DP and Admix/SI-FGSM simultaneously, the augmentation becomes too strong to keep the input a in-distribution sample. We adopted the default hyper-parameters for all combined methods and it is possible (yet computationally very intensive since the number of combinations is huge) to carefully tune hyper-parameters to achieve even better combinations. We will leave it to future work.
4.3 Evaluation of "Gradient Computation" Methods and "Substitute Model Training" Methods
The previous section evaluates input augmentations and optimizers, and, in this section, we shall focus on "gradient computation" and "substitute model training" methods.