Title: Towards Evaluating Transfer-based Attacks Systematically, Practically, and Fairly
Venue: NeurIPS 2023
Accepted: yes
Keywords: adversarial examples

adversarial transferability

black-box attack
Abstract: The adversarial vulnerability of deep neural networks (DNNs) has drawn great attention due to the security risk of applying these models in real-world applications. Based on transferability of adversarial examples, an increasing number of transfer-based methods have been developed to fool black-box DNN models whose architecture and parameters are inaccessible. Although tremendous effort has been exerted, there still lacks a standardized benchmark that could be taken advantage of to compare these methods systematically, fairly, and practically. Our investigation shows that the evaluation of some methods needs to be more reasonable and more thorough to verify their effectiveness, to avoid, for example, unfair comparison and insufficient consideration of possible substitute/victim models. Therefore, we establish a transfer-based attack benchmark (TA-Bench) which implements 30+ methods. In this paper, we evaluate and compare them comprehensively on 10 popular substitute/victim models on ImageNet. New insights about the effectiveness of these methods are gained and guidelines for future evaluations are provided.

Towards Evaluating Transfer-based Attacks Systematically, Practically, and Fairly

Abstract

The adversarial vulnerability of deep neural networks (DNNs) has drawn great attention due to the security risk of applying these models in real-world applications. Based on transferability of adversarial examples, an increasing number of transfer-based methods have been developed to fool black-box DNN models whose architecture and parameters are inaccessible. Although tremendous effort has been exerted, there still lacks a standardized benchmark that could be taken advantage of to compare these methods systematically, fairly, and practically. Our investigation shows that the evaluation of some methods needs to be more reasonable and more thorough to verify their effectiveness, to avoid, for example, unfair comparison and insufficient consideration of possible substitute/victim models. Therefore, we establish a transfer-based attack benchmark (TA-Bench) which implements 30+ methods. In this paper, we evaluate and compare them comprehensively on 25 popular substitute/victim models on ImageNet. New insights about the effectiveness of these methods are gained and guidelines for future evaluations are provided. Code at: https://github.com/qizhangli/TA-Bench.

1 Introduction

In recent years, deep neural networks (DNNs) have demonstrated unprecedented success in various applications. However, the success comes at a price: DNNs are vulnerable to adversarial examples crafted by adding imperceptible perturbations to inputs ( e.g. , images). The existence of adversarial examples poses a significant threat to the security and reliability of DNNs, especially in safety-critical applications such as autonomous driving, biometrics, and medical image analysis.

There are many different ways of generating adversarial examples and performing attacks. Transferbased attacks, that are capability of compromising DNNs without having access to their network architecture and parameters, have been widely studied over the past few years. To issue such transferbased attacks, an attacker first collects a substitute model or a set of substitute models, then computes gradients on the substitute model(s) and perform optimization based on the gradients. Performance of the attacks largely rely on the transferability of the generated adversarial examples. Over the years, a large number of methods have been proposed to improve the adversarial transferability. The methods innovate various aspects of the attack procedure, ranging from substitute model training [47, 78, 13, 27] to gradient computation (that modifies loss or forward/backward architectures given well-trained substitute models) [77, 11, 20, 14, 65, 26, 60, 57, 61, 74, 21, 15, 74] to input augmentation [67, 7, 28, 58] and the optimizer that applies the gradients [6, 28, 59]. Some methods further propose to train a generative model using additional data for obtaining transferable adversarial examples [35, 44, 75, 36, 69].

Despite all the progress, there still lacks a standardized benchmark that could be taken advantage of to compare these methods systematically, fairly, and practically. Existing evaluations in the literature suffer from several limitations. First, there is a paucity of variety in the tested models, while certain methods may exhibit superior performance only with specific substitute/victim architectures. Second, many methods verified their efficacy only with a basic optimization back-end, e.g. , I-FGSM [24], despite the existence of more advanced optimization including input augmentations. It is hence unclear whether the benefits of an innovation in these methods can be similarly brought by incorporating these more advanced optimization techniques, which could undermine the reliability of the evaluations.

To address these problems, we create a transfer-based attack benchmark (TA-Bench), allowing researchers to compare a variety of methods in a fair and reliable manner. We believe that TA-Bench will foster the development of adversarial machine learning and inspire effective ways of generating adversarial examples for evaluating the robustness of DNNs. Our main contributions include (but not limited to) the following items.

A More Advanced Optimization Back-end. We have evaluated various combinations of input augmentation mechanisms and optimizers on our TA-Bench. Our results convey that, with a few exceptions, combining more types of augmentations leads to more powerful attacks, and it could be more reasonable to evaluate the effectiveness of input augmentation mechanisms and optimizers in combination rather than in isolation. Besides, the obtained combination is suggested to be a more advanced optimization back-end for evaluating the 'gradient computation' methods and 'substitute model training' methods.

Insightful Observations from Comprehensive Evaluations. We consider various substitute/victim models that are considered popular, including CNNs ( e.g. , ResNet-50 [16], VGG-19 [49] with batch normalization, Inception v3 [49], EfficientNetV2-M [51], and ConvNeXt-B [32]), vision transformers ( e.g. , ViT-B [8], DeiT-B [54], Swin-B [31], and BEiT-B [1]), and a MLP (MLPMixer [53]). Comprehensive results on our TA-Bench systematically shows how the performance of transfer-based attacks varies with different choices of substitute/victim models. These results demonstrate that adopting transformers as the substitute model generally yields superior attack performance for 'gradient computation' methods, comparing with using traditional convolutional models (as the substitute model).

A Unified Codebase. We offer an open-source codebase for TA-Bench, featuring a well-organized code structure that can effectively accommodate a diverse range of transfer-based attacks, as well as various substitute/victim models. It provides a unified setting for evaluations, ensuring consistency and reproducibility in experimental results. The code is at https://github.com/qizhangli/TA-Bench.

2 Related Work

Transfer-based Attacks. Transfer-based attacks emerged as a result of the discovery that adversarial examples are not only powerful against the model they were generated on but also effective against other models. By exploiting such a discovery, attackers can use adversarial examples created on some substitute models to attack the victim model, and a series of transfer-based attacks have been proposed. These methods have innovated various aspects of the attack procedure, and we will carefully discuss them in Section 3.2.

Related Benchmarks. There have been several libraries or benchmarks for generating adversarial examples, such as CleverHans [39], Foolbox [41, 42], RobustART [52], Torchattacks [23], etc. However, these benchmarks only cover a very limited number of transfer-based methods, as they were developed for evaluating the robustness of DNNs against not only black-box attacks but also white-box attacks. In particular, most of the methods are outdated and only considered as baseline methods due to the rapid development in this field. By contrast, our TA-Bench focuses on transfer-based attacks, and it implements 30+ methods in order to perform systematical, practical, and fair comparisons. A contemporary benchmark [76] also evaluates many transfer-based attacks and it draws attention to the stealthiness of the adversarial examples. Our contributions are mostly orthogonal to theirs, as we focus on fair and practical comparison between different methods. We will show some previous conclusions might be overthrown in practice. We provide new insights and guidance for evaluation to assist the future work in the community.

3 Our Benchmark

3.1 Threat Model

In a black-box scenario, the adversary has limited access to the victim model. Transfer-based attack aims to compromise the victim model by generating adversarial examples on a substitute model or a set of substitute models. Recent transfer-based methods, with few exceptions [25, 48], tested in a setting where: 1) training data of the victim model is accessible and training/fine-tuning a model on such data is possible for the adversary [47, 78, 13, 27, 40, 35, 44, 75, 36], or 2) the adversary is able to collect at least one substitute model trained using the same dataset that the victim model learned from [37, 77, 22, 11, 20, 14, 65, 26, 60, 57, 61, 74, 21, 15, 74, 24, 34, 67, 7, 58, 6, 28]. We follow this assumption for setting up the benchmark. Specifically, nothing except for the training data is known about the victim model for performing attacks in our threat model, i.e. , the adversary has no idea about the pre-processing pipeline, architecture, and parameters.

Adversarial examples on the benchmark are all obtained by performing pixel-wise perturbations to benign images under an ℓ p constraint.

3.2 Methodologies

In order to compare different methods more reasonably, we roughly divide existing methods, based on their main innovations, into four categories highlighted as follows.

Input Augmentation and Optimizer. To craft an adversarial example on any given substitute model, the perturbation can be optimized as in the white-box setting. Gradient-based iterative optimization is commonly utilized, in which the perturbation is initialized to a zero tensor ( e.g. , in I-FGSM [24]) or a random tensor whose entries are sampled from a distribution ( e.g. , in PGD [34]). Image data augmentation has been considered for generating transformation-robust perturbations to each benign example, for example in diverse inputs I-FGSM (DI 2 -FGSM) [67], translation-invariant I-FGSM (TI-FGSM) [7], scale-invariant I-FGSM (SI-FGSM) [28], and Admix [58]. Several attacks also innovate by taking advantage of the momentum optimizer, e.g. , momentum I-FGSM (MI-FGSM) [6], Nesterov I-FGSM (NI-FGSM) [28], and pre-gradient guided momentum I-FGSM (PI-FGSM) [59]. In general, these methods are all architecture-independent.

Gradient Computation (DNN-Specific). There is a belief that improved adversarial transferability can be achieved by modifying the loss or the backpropagation process. For backpropagation, both the forward and the backward pass can be altered to achieve powerful attacks, and a series of methods, including SGM [65], LinBP [14], ConBP [72], PNA [62], and SE [38] have been proposed. Some methods advocate loss terms obtained on a middle layer of the substitute model ( e.g. , NRDM [37], TAP [77], FDA [11], ILA [20], ILA++ [26], FIA [61], NAA [74]), while other methods stick with loss computed on the final layers ( e.g. , IR [60], VT [57], and TAIG [21]).

Substitute Model Training. Though most prior work uses off-the-shelf models (that could be collected on the Internet) directly as substitute models, some methods advocate fine-tuning these models or even training new ones to better suit the goal of achieving transferable adversarial examples. For instance, RFA [47] suggests adopting adversarial training to obtain substitute models. DRA [78] fine-tunes the substitute model to push the adversarial examples away from the distribution of their benign counterparts during performing attacks. LGV [13] fine-tunes the substitute model with a high learning rate and to collect a set of models on the training trajectory. A very recent method proposed by Li et al. performs fine-tuning in order to obtain Bayesian substitute models [27].

Generative Modeling. In addition to the above mentioned attacks, there are another line of transferbased attacks that use generative models to craft adversarial examples. These methods often require additional data for training the generative model. Once properly trained, the model can generate transferable adversarial examples across different victim models. Some effective methods concerned in this line include CDA [35], GAPF [44], BIA [75], TTP [36], C-GSP [69], etc.

All these mentioned methods have been implemented on our benchmark. That is, we implemented 30+ methods from these four categories for comprehensive evaluation and comparison of transferbased attacks.

3.3 Victim Models and Substitute Models

With a surge of innovation in deep learning, there exists a variety of image classification DNNs. Each has its own advantages. In practice, an engineer can choose any of them to train and deploy according to his or her specific requests. An adversary whose aim is to compromise a computer vision service developed by the engineer then anticipate the generated adversarial examples transfer well to all these possible models. Hence, to make the evaluation comprehensive and practical, we consider various victim models that are considered popular, including CNNs ( e.g. , ResNet-50 [16], VGG-19 [49] with batch normalization, Inception v3 [49], EfficientNetV2-M [51], and ConvNeXtB [32]), vision transformers ( e.g. , ViT-B [8], DeiT-B [54], Swin-B [31], and BEiT-B [1]), and a MLP (MLP-Mixer-B [53]). It is worth noting that all of these models were obtained directly from an open-source repository timm [63] on GitHub, and our benchmark can be easily updated to include new models in the future.

Having witnessed the development of image classification architectures, it is unwise for the adversary to stick with conventional models ( e.g. , ResNets, VGG-Nets, and Inceptions), since it is possible that generating adversarial examples on a more advanced substitute model leads to superior attack success rates. Thus, on this benchmark, we employ the 10 victim models named above to generate adversarial examples and evaluate the attack performance of all options. As will be shown in Section 4.3, many new insights can be gained from the evaluation results.

3.4 Experimental Settings and Implementation

All evaluations on our benchmark are conducted on ImageNet [43]. We randomly selected 5,000 benign examples that could be correctly classified by all the victim models, from the ImageNet validation set, to craft adversarial examples. Filenames of these benign examples will be provided, for reproducing results and testing new methods in the future. A distance metric is required to measure the magnitude of perturbations. We adopt the popular ℓ p distance for p ∈ {∞ , 2 } and set the perturbation budget under ℓ ∞ and ℓ 2 constraints to ϵ = 8 / 255 and ϵ = 5 , respectively, to guarantee that the adversarial perturbations are almost imperceptible. The optimization process of each compared method runs 100 iterations with a step size of 1 / 255 and 1 for ℓ ∞ constraint and ℓ 2 constraint, respectively. For each victim model, we pre-process its input in exactly the same way as their official implementation to ensure a practical evaluation of attack performance, and note that the adversary has no idea about the detailed implementation of this pre-processing. For instance, when evaluating the performance of attacking a ResNet-50 victim, we first resize an adversarial example to 256 × 256 by bilinear interpolation, then crop the image to 224 × 224 in the center, and finally feed the 224 × 224 image into the victim model and evaluate whether the attack is successful or not. Implementation details about all the supported methods are provided in Section F in the Appendix. All experiments are performed on an NVIDIA V100 GPU.

For evaluating the transferability of adversarial examples, we use the accuracy of victim models for classifying the adversarial examples as a measure. Using the prediction accuracy of victim models instead of the attack success rate makes it easier to incorporate other victim models in the future, as a reasonable calculation of attack success rate often requires the benign counterparts of the adversarial examples be classified correctly by all victim models. With a specific choice of the substitute model, prior work often evaluates the average accuracy (AA) over all victim models for comparing different attacks. However, since our benchmark studies a variety of substitute models, we further evaluate the average AA (AAA), the worst AA (WAA), and the best AA (BAA) over all choices of these substitute models. Lower AAA, WAA, and BAA indicate stronger attacks and more vulnerable victim models.

We have built a codebase consisting of modular components that serve as the basis of TA-Bench. By leveraging modular design principles, the substitute and victim models, back-end methods, and hyper-parameters can be easily adapted to help the future work of the community.

4 Evaluations and Analyses

In Section 4.1, we identify a pre-processing pipeline that is more practical. In Section 4.2, we investigate an improved back-end method by evaluating the possible combinations of iterative optimization methods. We then re-evaluate state-of-the-arts under a comprehensive and unified

benchmark, which incorporates various substitute and victim models, to assess the effectiveness of these methods in Section 4.3 and Section 4.4.

4.1 Pre-processing in Practice

Modern image classification models (especially CNNs) can take images of various sizes. After investigating experimental settings of previous transfer-based attacks, we found that the adversarial examples were often evaluated by feeding them into the victim models directly (without taking pre-processing operations of the victim models into account, e.g. , resize and crop) [28, 57, 59, 58] or just resize to the input size of the victim models [61, 74]. However, these victim models, when deployed, are often equipped with inference time data pre-processing to improve their effectiveness and efficiency. Getting rid of it may lead to over-estimation of adversarial vulnerability.

As mentioned in Section 3.4, on our benchmark, we follow the default pre-processing pipeline of each victim model and evaluate under the circumstances where pre-processing often exists. We observed degraded attack performance under such circumstances (see Figure 1 for results). Obviously, the adversarial examples seem less vulnerable when the default test-time preprocessing is re-introduced to each victim model. Given the same I-FGSM adversarial examples, the average accuracy of victim models increases to 87.79% (from 86.16%) with pre-processing, confirming that ignoring pre-processing operations of victim models indeed leads to overestimation of their adversarial vulnerability . Similar observations can be made with other substitute models and other attack methods. Considering that the pre-processing pipeline of the victim model is inaccessible to the adversary, it is infeasible for the adversary to follow the same

100

+resize

Morevulnerableto theattack

90

+resize&crop(default)

Accuracy (%)

80

70

60

50

VGG-19

ViT-B

DeiT-B

BEiT-B

Swin-B

Mixer-B

Inceptionv3

EffNetV2-M

ConvNeXt-B

Victim model

pipeline when generating adversarial examples. On the substitute models, a safe choice is to resize each of their inputs to the default size without cropping them.

4.2 Evaluation of Input Augmentation and the Optimizer

To verifying the effectiveness of a newly developed computer vision architecture ( e.g. , vision transformers), it is common to show that its performance surpasses previous state-of-the-arts on a challenging benchmark dataset ( e.g. , ImageNet), and test of the newly developed architecture may utilize a combination of advanced optimization strategies ( e.g. , AdamW [33] + mixup [73] + cutmix [71] + stochastic depth [19]) if such a combination is beneficial [2, 55]. Novel optimization strategies are also often tested in combination with existing ones, to show their consistent effectiveness [12].

Likewise, the transferability of adversarial examples can also benefit from appropriate choices of input augmentations and the optimizer, however, ways of innovating these optimization strategies are often evaluated in isolation in this setting. On our benchmark, we, for the first time, evaluate combinations of different choices of input augmentations and the optimizer systematically. Specifically, we performed a grid search to seek the optimal combination of I-FGSM [24], PGD [34], DI 2 -FGSM [67], TI-FGSM [7], SI-FGSM [28], Admix [58], NI-FGSM [28], MI-FGSM [6], PI-FGSM [59], etc. Since Admix inherently includes SI-FGSM, we have SI-FGSM by default when Admix is chosen. NIFGSM, MI-FGSM, and PI-FGSM seem not orthogonal and thus they are tested in a mutually exclusive way in our experiment. The same for I-FGSM and PGD. For TI-FGSM, an alternative implementation where inputs are translated is adopted, as it is more effective than its suggested approximation which convolves gradients. Besides the augmentations in DI 2 -FGSM, TI-FGSM, SI-FGSM, and Admix, we consider several other input augmentation mechanisms including adding uniform noise to the input (which was used in VT [57] and TAIG [21]) and randomly dropping some patches of the perturbation (which was used in IR [60]). We call the two methods as UN and DP, respectively. Note that in a previous work [66], Gaussian noise is added to the input in each attack iteration. In this study, we

100

Average accuracy (%)

9

More powerful attack

0010605040

30

12-TI-NI-FGSM

2la-d

P-I-T-M-FGSM

I-DP-

S

UN-

UN-DP-

UN-DP-

UN-DP-

UN-DP-A

opt for uniform noise, as it is now more commonly adopted. It is also worth noting that the original implementation of SI-FGSM and Admix uses several augmented copies of an input and averages the gradients computed on these copies for optimization. Such an approach increases the computational complexity of performing attacks, and, in fact, all input augmentation mechanisms in this category can be improved by such an approach [76], and, for reducing computational complexity, we only craft one augmented input at each iteration.

Each possible combination is tested with every possible substitute model whose name has been mentioned in Section 3.3 to evaluate the attacking performance over other models which are considered as the victim models. The results are sorted by AAA, in a descending order from left to right, and demonstrated in Figure 2. It illustrates not only AAA but also the range between BAA and WAA (as 'error bars'). We see that the optimal AAA is achieved by UN-DP-DI 2 -TI-PI-FGSM, which is 42 . 42 % . The detailed AA when each substitute model is chosen for crafting adversarial examples by the method is reported in Section E. Inspecting the obtained results, we found that the performance gap between MI-FGSM, NI-FGSM, and PI-FGSM is marginal. For instance, the top 3 solutions (with PI-FGSM, NI-FGSM, and MI-FGSM, respectively) lead to similar AAA ( 42 . 42% , 42 . 45% , and 42 . 46% ). The performance of I-FGSM and PGD is also similar. In most cases, PGD leads to slightly inferior performance than that of I-FGSM, accordingly to our experimental results.

In general, more input augmentation mechanisms leads to more powerful attacks. Yet there are exceptions. With DP adopted, SI-FGSM and Admix that apply input scaling and mixing fail to manifest their gains regarding the adversarial transferability. In particular, the AAA increases to 43 . 12% and 44 . 11% when further adding SI and Admix to UN-DP-DI 2 -TI-PI-FGSM, respectively. This is in contrast to the previous belief that these two methods are effective, and a possible explanation is that, when adopting DP and Admix/SI-FGSM simultaneously, the augmentation becomes too strong to keep the input a in-distribution sample. We adopted the default hyper-parameters for all combined methods and it is possible (yet computationally very intensive since the number of combinations is huge) to carefully tune hyper-parameters to achieve even better combinations. We will leave it to future work.

4.3 Evaluation of 'Gradient Computation' Methods and 'Substitute Model Training' Methods

The previous section evaluates input augmentations and optimizers, and, in this section, we shall focus on 'gradient computation' and 'substitute model training' methods.

Firstly, we would like to emphasize that certain gradient computation innovations ( e.g. , IR [60], VT [57], and TAIG [21]) incorporate input augmentations for averaging the gradients obtained from multiple randomly augmented inputs at each iteration. These methods require high computational cost, and comparing them with other methods in the same category directly is unfair, as the other methods can also apply random augmentation and gradient averaging to boost their performance. Taking VT [57] as an example, it introduces random noise to the input and perform backpropagation for 20 times at each iteration. Similar for IR and TAIG. We compare them with their corresponding baselines that employ the same input augmentations and keep the same number of backpropagation operations at each iteration. That is, the corresponding baselines of VT and IR ( i.e. , VT-baseline and IR-baseline) perform multiple rounds of backpropagation with UN and DP augmentations, respectively, at each

100

100

100

(%)

90

IR-baseline

IR

(%)

90

VT-baseline

VT

(%)

90

TAIG-baseline

TAIG

More powerful attack

Average accuracy

accuracy

Average accuracy

80

80

80

70

70

70

Average

60

60

60

50

50

50

40

40

40

ResNet-50

VGG-19

Inception v3

EffNetV2-M

ConvNeXt-B

ViT-B

DeiT-B

BEIT-B

Swin-B

Mixer-B

ResNet-50

VGG-19

Inception v3

EffNetV2-M

ConvNeXt-B

ViT-B

DeiT-B

BEIT-B

Swin-B

Mixer-B

ResNet-50

VGG-19

Inception v3

EffNetV2-M

ConvNeXt-B

ViT-B

DeiT-B

BEiT-B

Swin-B

Mixer-B

Substitute model

Substitute model

Substitute model

iteration of optimization, and the corresponding baseline of TAIG ( i.e. , TAIG-baseline) perform both input scaling and UN augmentations. Figure 3 reports the comparison results. It can be seen that the simple baselines achieve even lower AAA, indicating that the most effective factor of these methods may be input augmentation and gradient averaging, instead of what were claimed. Based on these findings, we strongly recommend that newly designed methods, which incorporate input augmentations, either explicitly or implicitly, should be carefully compared with reasonable baselines that adopt the same mechanisms to show their effectiveness.

There are still 10+ methods to be evaluated and compared. Previous work often compares using a simple optimization back-end, e.g. , I-FGSM or MI-FGSM. As we have obtained a combination of optimization strategies ( i.e. , UN-DP-DI 2 -TI-PI-FGSM) which has been proven to be more powerful in Section 4.2, we further introduce it as a more advanced optimization back-end. It aids in better exploring true advantages of compared methods. To ensure optimal performance across different substitute models, we employed a validation set consisting of 500 samples that were distinct from the test examples tune hyper-parameters of compared methods. The hyper-parameters that yielded the best results on the validation set were then adopted for testing. The hyper-parameters with each substitute model are tuned using models whose name have been mentioned in Section 3.3. The attack performance is evaluated not only on these 'validation' models, but also on 15 more victim models which are distinct from them. The detailed hyper-parameters are reported in Section F.

In Table 1 (column 2-11), we report the AA achieved by utilizing each model as the substitute model to attack the 9 'validation' models under the ℓ ∞ constraint. AAA (which is the average AA over all substitute models) is provided in the 12-th column in the table. Note that some methods are unable to be performed on some architectures. For instance, ConBP [72] suggests replacing ReLU with a softplus function in the backward pass to ensure smooth gradient backpropagation, making it not suitable to substitute models that are equipped with smooth activations only. SGM [65] is not applicable to substitute architectures without skip-connections ( e.g. , VGG-19). PNA [62] and SE [38] focus on vision transformers, and thus they are not suitable to most convolutional substitute models. For 'substitute model training' methods, ResNet-50 is commonly chosen as the substitute model, as is adopted in the official GitHub implementations of these methods, and we leave the exploration of training with more advanced substitute architectures to future work. In addition to the ℓ ∞ experiment, we also conduct evaluate with the ℓ 2 constraint and the results are provided in Section A.

When I-FGSM [24] is simply applied as the optimization back-end, NAA [74] beats the other gradient computation innovations and achieves the lowest AAA with a value of 72 . 65% (see the upper half of Table 1), while RFA demonstrates the best results among all 'substitute model training' methods. Table 1 shows that the most suitable gradient computation strategy can be different for issuing attacks from different substitute models. In particular, if ResNet-50 or VGG-19 is chosen as the substitute model, FIA [61] seems even superior to NAA in the sense of achiving higher attack success rate and lower victim accuracy, however, NAA is the best if any other model is chosen as the substitute model.

When UN-DP-DI 2 -TI-PI-FGSM is introduced as the new optimization back-end, almost all 'gradient computation' methods show improved performance (see the lower half of Table 1). Yet, in combination with the new baseline, the advantage of most methods becomes less obvious. SGM [65], PNA [62], and SE [38] still produce performance gains when being combined with UN-DP-DI 2 -TIPI-FGSM. In particular, with the new optimization back-end, SE on the DeiT-B substitute model leads to the optimal attack performance among all concerned options, in the sense of BAA, fooling the victim models to show an accuracy of only 18 . 61% (on average). PNA obtains the lowest WAA among all, which is 31 . 50% . As for substitute model training, we see that the MoreBayesian method significantly outperforms the other methods, and it fools the victim models to show an average accuracy of 27 . 09% using a ResNet-50 substitute model.

According to Table 1, vision transformers should be preferable when choosing the substitute model, as the best attack performance ( i.e. , BAA) is often obtained on vision transformers for many attacks. To compare the transfer performance from vision transformers to convolutional networks and from the opposite direction, we report the accuracy of victim models in predicting SGM adversarial examples generated on ResNet-50 and ViT-B as the substitute model, respectively. The results are shown in Table 2. It can be seen that transferring from vision transformers to convolutional networks seems easier. When utilizing ViT-B as the substitute model, the accuracy of convolutional networks shows a range in [28 . 32% , 37 . 24%] , while, with ResNet-50, the accuracy of vision transformers lies in [36 . 82% , 48 . 32%] . Overall, using ViT-B as the substitute model leads to lower average accuracy ( 28 . 80% vs 31 . 97% ) and the worst accuracy ( 37 . 24% vs 48 . 32% ) on victim models, which means better average and worst-case attack performance, respectively. ConvNeXt [32] that follows some designing principles of the vision transformers is also a favorable choice of the substitute model, according to our results. When performing LinBP on ConvNeXt-B, the generated adversarial examples are capable of fooling the victim models to show an average accuracy of only 18 . 81% , which is super close to the best attack performance that could be achieved in Table 1. Additionally, it

is worth noting that even though ViT-B, DeiT-B, and BEiT-B share the same network architecture, they exhibit considerably different performance when acting as the substitute model. This suggests that the training procedure can also play a crucial role in improving the adversarial transferability.

If we still focus on ResNet/Inception and test adversarial examples only on these traditional models, as in many previous papers, then different conclusions will be drawn. Likewise, if we only focus on the simple optimization back-end ( i.e. , I-FGSM) without introducing UN-DP-DI 2 -TI-PI-FGSM, the conclusions will also be different, since NAA seems to be the best solution with I-FGSM.

In summary, some key takeaways are provided as follows. (i) It is essential to evaluate on a variety of substitute and victim models to gain a comprehensive understanding of the performance of a developed method. (ii) Evaluations using a more advanced optimization back-end ( e.g. , UN-DP-DI 2 -TI-PI-FGSM) should be considered. (iii) Generally, using vision transformers as substitute models yields superior attack performance comparing with using the traditional convolutional models. (iv) Among the 'substitute model training' methods, the MoreBayesian method consistently enhances adversarial transferability, and it outperforms other methods when using UN-DP-DI 2 -TI-PI-FGSM as the back-end.

Except the results on the 'validation' models, we further simulate a more practical attack scenario where 15 more victim models that are distinct from those 'validation' models in Table 1 are considered. The conclusion remains consistent on attacking these models, except performing LinBP on ConvNeXt achieves the best AA, and we report the results in Section B.

4.4 Evaluation of 'Generative Modeling' Methods

Another series of transfer-based attacks use generative modeling. They train a generative model to craft adversarial examples and adopt the substitute model as a discriminator. In Table 3, we compare these methods (including CDA [35], GAPF [44], BIA [75], TTP [36], and C-GSP [69]). As the BIA paper also introduce two additional modules (call RN and DA) that could be beneficial under certain circumstances, we also evaluate BIA+RN and BIA+DA and report the results in Table 3. We follow their official implementations and adopted a ResNet-152 model [16] as the substitute model. GAPF performs the best according to our results. For a comprehensive evaluation, we further compare these methods to UN-DP-DI 2 -TI-PI-FGSM, which is the newly developed optimization back-end. Table 3 shows that it outperforms these generative modeling methods significantly. Moreover, since these methods craft adversarial examples using a generative model, it is infeasible to directly adopt UN-DP-DI 2 -TI-PI-FGSM for boosting their performance.

5 Conclusion

In this paper, we have presented benchmark for transfer-based attacks, called TA-Bench. On TABench, we have implemented 30+ advanced transfer-based attack methods, including those focus on input augmentation and optimizer innovation, those 'gradient computation' methods, those 'substitute model training' methods, and those applying generative modeling. With TA-Bench, we are capable of evaluating and comparing transfer-based attacks systematically, practically, and fairly. Given comprehensive experimental results on TA-Bench, we have provided new insights about the effectiveness of these attacks, including but not limited to useful combinations of input augmentations and optimizers, reasonable choices of substitute/victim models, etc. Hoping to offer a sagacious judge of the state of transfer attacks and help future innovations in this field.

Acknowledgment

This material is based upon work supported by the National Science Foundation under Grant No. 1801751 and 1956364.

References

Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. In International Conference on Learning Representations , 2021.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650-9660, 2021.

Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: a standardized adversarial robustness benchmark. arXiv preprint arXiv:2010.09670 , 2020.

Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning , pages 2206-2216. PMLR, 2020.

Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vgg-style convnets great again. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13733-13742, 2021.

Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In CVPR , 2018.

Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Evading defenses to transferable adversarial examples by translation-invariant attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4312-4321, 2019.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. arXiv preprint arXiv:2303.11331 , 2023.

Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19358-19369, 2023.

Aditya Ganeshan, Vivek BS, and R Venkatesh Babu. Fda: Feature disruptive attack. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8069-8079, 2019.

Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph. Simple copy-paste is a strong data augmentation method for instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2918-2928, 2021.

Martin Gubri, Maxime Cordy, Mike Papadakis, Yves Le Traon, and Koushik Sen. Lgv: Boosting adversarial example transferability from large geometric vicinity. arXiv preprint arXiv:2207.13129 , 2022.

Yiwen Guo, Qizhang Li, and Hao Chen. Backpropagating linearly improves transferability of adversarial examples. In NeurIPS , 2020.

Yiwen Guo, Qizhang Li, Wangmeng Zuo, and Hao Chen. An intermediate-level attack framework on the basis of linear regression. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , 2016.

Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In CVPR , 2018.

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR , 2017.

Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14 , pages 646-661. Springer, 2016.

Qian Huang, Isay Katsman, Horace He, Zeqi Gu, Serge Belongie, and Ser-Nam Lim. Enhancing adversarial example transferability with an intermediate level attack. In ICCV , 2019.

Yi Huang and Adams Wai-Kin Kong. Transferable adversarial attack based on integrated gradients. arXiv preprint arXiv:2205.13152 , 2022.

Nathan Inkawhich, Wei Wen, Hai Helen Li, and Yiran Chen. Feature space perturbations yield more transferable adversarial examples. In CVPR , 2019.

Hoki Kim. Torchattacks: A pytorch repository for adversarial attacks. arXiv preprint arXiv:2010.01950 , 2020.

Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale. In ICLR , 2017.

Qizhang Li, Yiwen Guo, and Hao Chen. Practical no-box adversarial attacks against dnns. In NeurIPS , 2020.

Qizhang Li, Yiwen Guo, and Hao Chen. Yet another intermediate-leve attack. In ECCV , 2020.

Qizhang Li, Yiwen Guo, Wangmeng Zuo, and Hao Chen. Making substitute models more bayesian can enhance transferability of adversarial examples. In International Conference on Learning Representations , 2023.

Jiadong Lin, Chuanbiao Song, Kun He, Liwei Wang, and John E Hopcroft. Nesterov accelerated gradient and scale invariance for adversarial attacks. arXiv preprint arXiv:1908.06281 , 2019.

Chang Liu, Yinpeng Dong, Wenzhao Xiang, Xiao Yang, Hang Su, Jun Zhu, Yuefeng Chen, Yuan He, Hui Xue, and Shibao Zheng. A comprehensive study on robustness of image classification models: Benchmarking and rethinking. arXiv preprint arXiv:2302.14301 , 2023.

Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12009-12019, 2022.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10012-10022, 2021.

Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11976-11986, 2022.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR , 2018.

Muhammad Muzammal Naseer, Salman H Khan, Muhammad Haris Khan, Fahad Shahbaz Khan, and Fatih Porikli. Cross-domain transferability of adversarial perturbations. Advances in Neural Information Processing Systems , 32, 2019.

Muzammal Naseer, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Fatih Porikli. On generating transferable targeted perturbations. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7708-7717, 2021.

Muzammal Naseer, Salman H Khan, Shafin Rahman, and Fatih Porikli. Task-generalizable adversarial attack based on perceptual metric. arXiv preprint arXiv:1811.09020 , 2018.

Muzammal Naseer, Kanchana Ranasinghe, Salman Khan, Fahad Shahbaz Khan, and Fatih Porikli. On improving adversarial transferability of vision transformers. arXiv preprint arXiv:2106.04169 , 2021.

Nicolas Papernot, Fartash Faghri, Nicholas Carlini, Ian Goodfellow, Reuben Feinman, Alexey Kurakin, Cihang Xie, Yash Sharma, Tom Brown, Aurko Roy, Alexander Matyasko, Vahid Behzadan, Karen Hambardzumyan, Zhishuai Zhang, Yi-Lin Juang, Zhi Li, Ryan Sheatsley, Abhibhav Garg, Jonathan Uesato, Willi Gierke, Yinpeng Dong, David Berthelot, Paul Hendricks, Jonas Rauber, and Rujun Long. Technical report on the cleverhans v2.1.0 adversarial examples library. arXiv preprint arXiv:1610.00768 , 2018.

Omid Poursaeed, Isay Katsman, Bicheng Gao, and Serge Belongie. Generative adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4422-4431, 2018.

Jonas Rauber, Wieland Brendel, and Matthias Bethge. Foolbox: A python toolbox to benchmark the robustness of machine learning models. In Reliable Machine Learning in the Wild Workshop, 34th International Conference on Machine Learning , 2017.

Jonas Rauber, Roland Zimmermann, Matthias Bethge, and Wieland Brendel. Foolbox native: Fast adversarial attacks to benchmark the robustness of machine learning models in pytorch, tensorflow, and jax. Journal of Open Source Software , 5(53):2607, 2020.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. IJCV , 2015.

Mathieu Salzmann et al. Learning transferable adversarial perturbations. Advances in Neural Information Processing Systems , 34:13950-13962, 2021.

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR , 2018.

Naman D Singh, Francesco Croce, and Matthias Hein. Revisiting adversarial training for imagenet: Architectures, training and generalization across threat models. arXiv preprint arXiv:2303.01870 , 2023.

Jacob Springer, Melanie Mitchell, and Garrett Kenyon. A little robustness goes a long way: Leveraging robust features for targeted transfer attacks. Advances in Neural Information Processing Systems , 34, 2021.

Chenghao Sun, Yonggang Zhang, Wan Chaoqun, Qizhou Wang, Ya Li, Tongliang Liu, Bo Han, and Xinmei Tian. Towards lightweight black-box attacks against deep neural networks. arXiv preprint arXiv:2209.14826 , 2022.

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR , 2016.

Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning , pages 6105-6114. PMLR, 2019.

Mingxing Tan and Quoc Le. Efficientnetv2: Smaller models and faster training. In International conference on machine learning , pages 10096-10106. PMLR, 2021.

Shiyu Tang, Ruihao Gong, Yan Wang, Aishan Liu, Jiakai Wang, Xinyun Chen, Fengwei Yu, Xianglong Liu, Dawn Song, Alan Yuille, Philip H.S. Torr, and Dacheng Tao. Robustart: Benchmarking robustness on architecture design and training techniques. https://arxiv.org/pdf/2109.05211.pdf , 2021.

Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. Advances in Neural Information Processing Systems , 34:24261-24272, 2021.

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning , volume 139, pages 10347-10357, July 2021.

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning , pages 10347-10357. PMLR, 2021.

Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In European conference on computer vision , pages 459-479. Springer, 2022.

Xiaosen Wang and Kun He. Enhancing the transferability of adversarial attacks through variance tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1924-1933, 2021.

Xiaosen Wang, Xuanran He, Jingdong Wang, and Kun He. Admix: Enhancing the transferability of adversarial attacks. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 16158-16167, 2021.

Xiaosen Wang, Jiadong Lin, Han Hu, Jingdong Wang, and Kun He. Boosting adversarial transferability through enhanced momentum. arXiv preprint arXiv:2103.10609 , 2021.

Xin Wang, Jie Ren, Shuyun Lin, Xiangming Zhu, Yisen Wang, and Quanshi Zhang. A unified approach to interpreting and boosting adversarial transferability. In International Conference on Learning Representations , 2020.

Zhibo Wang, Hengchang Guo, Zhifei Zhang, Wenxin Liu, Zhan Qin, and Kui Ren. Feature importanceaware transferable adversarial attacks. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7639-7648, 2021.

Zhipeng Wei, Jingjing Chen, Micah Goldblum, Zuxuan Wu, Tom Goldstein, and Yu-Gang Jiang. Towards transferable adversarial attacks on vision transformers. In Proceedings of the AAAI Conference on Artificial Intelligence , 2022.

Ross Wightman. Pytorch image models. https://github.com/rwightman/ pytorch-image-models , 2019.

Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16133-16142, 2023.

Dongxian Wu, Yisen Wang, Shu-Tao Xia, James Bailey, and Xingjun Ma. Rethinking the security of skip connections in resnet-like neural networks. In ICLR , 2020.

Lei Wu, Zhanxing Zhu, Cheng Tai, et al. Understanding and enhancing the transferability of adversarial examples. arXiv preprint arXiv:1802.09707 , 2018.

Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan L Yuille. Improving transferability of adversarial examples with input diversity. In CVPR , 2019.

Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR , 2017.

Xiao Yang, Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Boosting transferability of targeted adversarial examples via hierarchical generative networks. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IV , pages 725-742. Springer, 2022.

Weihao Yu, Chenyang Si, Pan Zhou, Mi Luo, Yichen Zhou, Jiashi Feng, Shuicheng Yan, and Xinchao Wang. Metaformer baselines for vision. arXiv preprint arXiv:2210.13452 , 2022.

Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision , pages 6023-6032, 2019.

Chaoning Zhang, Philipp Benz, Gyusang Cho, Adil Karjauv, Soomin Ham, Chan-Hyun Youn, and In So Kweon. Backpropagating smoothly improves transferability of adversarial examples. In CVPR 2021 Workshop Workshop on Adversarial Machine Learning in Real-World Computer Vision Systems and Online Challenges (AML-CV) , volume 2, 2021.

Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 , 2017.

Jianping Zhang, Weibin Wu, Jen-tse Huang, Yizhan Huang, Wenxuan Wang, Yuxin Su, and Michael R Lyu. Improving adversarial transferability via neuron attribution-based attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14993-15002, 2022.

Qilong Zhang, Xiaodan Li, YueFeng Chen, Jingkuan Song, Lianli Gao, Yuan He, et al. Beyond imagenet attack: Towards crafting adversarial examples for black-box domains. In International Conference on Learning Representations , 2022.

Zhengyu Zhao, Hanwei Zhang, Renjue Li, Ronan Sicre, Laurent Amsaleg, and Michael Backes. Towards good practices in evaluating transfer adversarial attacks. arXiv preprint arXiv:2211.09565 , 2022.

Wen Zhou, Xin Hou, Yongjun Chen, Mengyun Tang, Xiangqi Huang, Xiang Gan, and Yong Yang. Transferable adversarial perturbations. In ECCV , 2018.

Yao Zhu, Yuefeng Chen, Xiaodan Li, Kejiang Chen, Yuan He, Xiang Tian, Bolun Zheng, Yaowu Chen, and Qingming Huang. Toward understanding and boosting adversarial transferability from a distribution perspective. IEEE Transactions on Image Processing , 31:6487-6501, 2022.

A ℓ 2 Results

We evaluate the 'gradient computation' methods and 'substitute model training' methods under ℓ 2 constraint and provide the results in Table 4. Some ℓ 2 results are provided in this section. When I-FGSM is applied as the optimization back-end, same as the ℓ ∞ results in Table 1 in our main paper, NAA achieves the lowest AAA ( i.e. , 82.00%) compared with the other 'gradient computation' methods, while FIA beats it when ResNet-50 or VGG-19 is chosen as the substitute model. However, unlike in the ℓ ∞ setting, SE shows consistently inferior performance when compared with the IFGSM baseline in the ℓ 2 setting, and DRA instead of RFA achieves the best performance among 'substitute model training' methods.

When UN-DP-DI 2 -TI-PI-FGSM is applied as the new optimization back-end, same as in the ℓ ∞ setting, SGM, PNA, and SE provide favorable attack performance, while PNA on the DeiT-B substitute model turns out to be the best (in the sense of achieving lower BAA) and the generated adversarial examples fools victim models to show an accuracy of only 29 . 33% . The lowest WAA (which is 51 . 56% ) is obtained by PNA. For the 'substitute model training' methods, the MoreBayesian method still outperforms the other methods by a large margin.

VIT-B

100

100

100

Average accuracy (%)

IR-baseline

IR

Average accuracy (%)

VT-baseline

VT

Average accuracy (%)

TAIG-baseline

TAIG

More powerful attack

90

90

90

80

80

80

70

70

70

60

60

60

ResNet-50

VGG-19

Inception v3

EffNetV2-M

ConvNeXt-B

8-1

DeiT-B

BEiT-B

Swin-B

Mixer-B

ResNet-50

VGG-19

Inception v3

EffNetV2-M

ConvNeXt-B

DeiT-B

BEIT-B

Swin-B

Mixer-B

ResNet-50

VGG-19

Inception v3

EffNetV2-M

ConvNeXt-B

ViT-B

DeiT-B

BEiT-B

Swin-B

Mixer-B

Substitute model

Substitute model

Substitute model

Figure 4 compares TAIG, VT, and IR with their corresponding baselines under the ℓ 2 constraint. It can be seen that the they exhibit higher AAA and worse attack performance compared with their baselines, as in the ℓ ∞ setting.

B Results of Attacking Other Victim Models

We collected 15 additional victim models, including 7 CNNs (EfficientNet-L2 [50], ConvNeXt V2L [64], MobileNet V2 [45], DenseNet-161 [18], ResNeXt-101 [68], SENet-154 [17], and RepVGGB3 [5]) and 8 vision transformers (ViT-L [8], DeiT-L [54], Swin V2-L [30], BEiT-L [1], CAFormerB36 [70], MaxViT-L [56], EVA-L [10], EVA02-L [9]), and conducted an experiment on attacking these victim models. For the 'substitute model training' methods, the conclusion remains consistent with the observations from Table 1. Specifically, when employing I-FGSM as the back-end, RFA achieves the best AA ( i.e. , 63.93%), and when applying UN-DP-DI 2 -TI-PI-FGSM as the backend, MoreBayesian attains the best AA ( i.e. , 47.13%). For the 'gradient computation' methods, when I-FGSM is applied as the optimization back-end, the conclusion aligns with the findings in Table 1 of the paper. NAA consistently outperforms other methods on most choices of the substitute model, achieving the lowest AAA ( i.e. , 79.27%). When introducing UN-DP-DI 2 -TI-PI-FGSM as the optimization back-end, the top four lowest AAs are achieved using ConvNeXt-B, DeiT-B, BEiT-B, and Swin-B as the substitute models, as in Table 1. The best AA is obtained by performing LinBP on the ConvNeXt-B substitute model ( i.e. , 30.81%, which stands as the second-best in Table 1 and is only 0.20% higher than the best AA), due to slight distribution shift of the tested victim models.

C Results of Attacking Robust Models

We also evaluated the performance of different methods in attacking 3 defensive models obtained via adversarial training, i.e. , a robust ConvNext-B [29], a robust Swin-B [29], and a robust ViT-BCvSt [46]. They are all collected from RobustBench [3] and exhibit high robust accuracy against AutoAttack [4]. The results are given in Table 6. It can be seen that when I-FGSM is used as the optimization back-end, NAA and DRA achieve the best AAs among the methods of 'gradient computation' and 'substitute model training' categories, respectively. When UN-DP-DI 2 -TI-PIFGSM is employed as the optimization back-end, SGM achieves the best AAs among the 'gradient computation' methods for the most of substitute models, and the best AA is achieved by using DeiT-B as the substitute model, i.e. , 92.05%. For the 'substitute model training' methods, RFA instead of MoreBayesian achieves the best AA, i.e. , 86.37%, since it shares the same training scheme ( i.e. , adversarial training) with the victim models.

D Results of different ϵ

We conducted the evaluation of 'gradient computation' methods and 'substitute model training' methods under an ℓ ∞ constraint with ϵ = 16 / 255 since it is a common setting that many previous work [6, 28, 58, 74] adopted. The results are shown in Table 7. The conclusion is consistent with

observations from Table 1, except when I-FGSM is applied as the optimization back-end, DRA instead of RFA achieves the lowest AA, i.e. , 13.43%.

E Detailed Results of Input Augmentations and Optimizers

We show the detailed results of different combinations of input augmentations and optimizers in Table 8. It can be seen that UN-DP-DI 2 -TI-PI-FGSM achieves the best performance on average, despite the optimal solution on different substitute models are different. ℓ 2 results are given in Table 9.

F Implementation Details

Augmentations and Optimizer. For PGD, DI 2 -FGSM, MI-FGSM, NI-FGSM, and PI-FGSM, we use the default hyper-parameters. For TI-FGSM, we randomly translate the input with a range of [-3, +3] since its performance is better than the approximation using a 7 × 7 Gaussian kernel in many implementations [28, 58, 59, 27]. For SI-FGSM and Admix, both of them average the gradients obtained by feeding different augmented inputs into the substitute model, which may lead to unfair comparisons. Therefore, we randomly select one input from the augmented copies, and the hyper-parameters remain the same as in their original papers. For UN, the noise added to the input follows U ( -ϵ, ϵ ) and U ( -ϵ √ HW , ϵ √ HW ) (the dimension of inputs is 3 × H × W ) for attacks under ℓ ∞ and ℓ 2 constraints, respectively. For DP, we divide the perturbation into 16 × 16 patches and randomly drop 50% of the patches at each iteration.

Gradient Computation. For TAIG, VT, IR, TAP, FDA, SE, and PNA, we set the same hyperparameters as in their original papers. For NRDM, ILA, ILA++, LinBP, ConBP, FIA, and NAA, the main hyper-parameter which significantly impacts the performance is the choice of the middle layer. The scaling factor of SGM is also related to the selection of the substitute model. We tune these hyper-parameters by evaluating on a validation set consisting of 500 samples that do not overlap with the samples in the test set, and the best choices are shown in Table 10.

Substitute Model Training. In this category of methods, ResNet-50 is commonly chosen as the substitute model, and we collect the models from the GitHub repositories of these methods. For LGV and the MoreBayesian method, we only sample once at each iteration.

Generative Modeling. In this category of methods, all the generators are collected from the GitHub repositories of these methods.

Peer reviews:
comments: SUMMARY:
This paper introduces a benchmark, called TA-Bench, for transfer-based attacks. The authors implement 30+ transfer-based attack methods that are mostly proposed in the last 3 years. This paper takes several aspects of transfer-based attacks into consideration, including augmentation, optimizer, substitute model training, and generative modeling.

STRENGTHS:
1. Solid work with a considerable amount of experiments. I believe that this work, as its topic suggests, will bring new insights in systematically, practically, and fairly evaluating transfer-based attacks.
2. Apart from the benchmark, this paper provides some useful takeaways in Line 338-345.

WEAKNESSES:
This paper would better be submitted to the benchmark track.

QUESTIONS:
1. The recent paper "Reliable Evaluation of Adversarial Transferability" by Yu et al. also provides a benchmark for evaluating the transferability of adversarial examples. Could you please compare your work with theirs?

LIMITATIONS:
No.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-29

comments: SUMMARY:
This paper establishes a transfer-based attack benchmark (TA-Bench) so that researchers could take advantage of this to comare different methods systematically, fairly and practically. TA-Bench implements 30+ methods and evaluate on 10 popular victim models (architecture) on ImageNet.

STRENGTHS:
1.The paper is well-written and easy to follow.

2.The TA-Bench has practical value for transfer-based attacks, since it implements 30+ methods and provides comprehensive evaluations.

WEAKNESSES:
1.The benchmark focuses on ImageNet. Is it possible to extend to other datasets, smaller (MNIST) or larger (JFT300)?

2.Typos:
1)The term “TA-bench” and “TA-Bench” should be consistent.

QUESTIONS:
1.This paper did a very good job. TA-Bench is comprehensive. I score 6 for this paper. But this paper may fit better towards “NeurIPS Tack Datasets and Benchmarks”.
2.In table 1, what is the meaning of different colors, e.g., grey, black, bald black?

LIMITATIONS:
The authors adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the problem of adversarial transferability evaluation on image classification tasks. The authors feel that there are a large number of migration attacks, but this community lacks a standard benchmark. Therefore, this paper establishes a transfer-based attack benchmark (TA-Bench) which implements 30+ methods and then evaluate and compare them comprehensively on 10 popular substitute/victim models on ImageNet.\n\nSTRENGTHS:\n1. The benchmark of this paper integrates many transfer attacks.\n\n2. TA-bench helps researchers to carry out research more easily and in-depth.\n\nWEAKNESSES:\n1. We acknowledge the workload and potential of this paper, but compared to previous work, the proposed TA-bench does not consider adversarial defense methods at all, which leads to the wrong robustness assessment. We think that some of the latest pre-processing defense and adversarial training models should be taken into account, at least ens3-adv-Inception-v3, ens4-adv-Inception-v3, ens-adv-Inception-ResNet-v2, HGD, R&P, NIPS-r3, JPEG, FD, ComDefend, NRP, RS, Bit-Red, DiffPure). We will not give references here, because it is very common in this community.\n\n2. The scalability of TA-bench is not strong enough. Transfer attacks have recently paid more attention to the research of targeted attacks and are a future research direction, but TA-bench lacks corresponding evaluations, which restricts its contribution. We believe that the category relationship of images should be considered in the screening of data sets, and an evaluation system for targeted transfer attacks should be constructed.\n\n3. The motivation for this paper is not very sufficient. The set of models in the NIPS2017 adversarial competition is indeed relatively limited, but I-FGSM does show the most powerful attack performance as an optimized backend (after integrating with other methods), and the experiments in this paper also illustrate this point. We think it is best for the authors to start with the challenge of transfer attacks to build benchmarks, including but not limited to targeted attacks, ViT to CNN.\n\n4. More and more latest foundation models adopt the transformer architecture, but in this paper, the discussion of the adversarial transferability of ViT and CNN is not very sufficient. This is also a recent research hotspot.\n\n5. Also, we are a little bit less sympathetic to the taxonomy of this paper. We feel that it can be divided into more detailed classifications such as advanced optimization, data enhancement, model integration, feature attack, and network structure, rather than just \"Gradient Computation\" Methods and \"Substitute Model Training\" Methods\".\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nThe authors did not discuss limitation and ethical risks in the main body.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new benchmark of techniques designed to increase the transferability of adversarial examples. The paper implements more than 30 of these techniques to compare the success rate of the corresponding attacks. The paper identifies several flaws of current evaluation protocols, for example, not considering the pre-processing pipeline.\n\nSTRENGTHS:\n- The engineering work done is impressive. Benchmarking 30 techniques for transferability is a testimony of good view of the field of transferability, and of a strong software development work.\n- The paper correctly and fairly evaluates techniques, regarding best practices developed recently in \\[59\\]. In particular, the paper takes a particular care to control the effect of the number of gradients computed per iteration, which should be controlled for fairness.\n- Some choices in categorisation are sound. For example, evaluating separately techniques based on generative modelling makes sense.\n- The related work is sensible and extensive. For example, the paper correctly acknowledges and iterate about the most related work \\[59\\], another benchmark paper on transferability. Unfortunately, the review of techniques is not exhaustive. And the paper would benefit from a clear explanation on how the 30 methods were selected. For example, the paper could exhaustively list all the techniques for transferability published at top-conferences. \n- The paper proposes a more consistent way to evaluate the complementary of techniques than what is used currently. But there are some limitations (see below).\n\nWEAKNESSES:\n- An exhaustive review of techniques published at top-conferences (as done for review papers) would clarify why some techniques are included and not other. For example, the paper titled \"Understanding and Enhancing the Transferability of Adversarial Examples\" was the first to average gradients over (Gaussian) noise in 2018. Moreover, \"Learning Transferable Adversarial Examples via Ghost Networks\" (AAAI-20), \"On Success and Simplicity: A Second Look at Transferable Targeted Attacks\" (NeurIPS 2021), \"Efficient and Transferable Adversarial Examples from Bayesian Neural Networks\" (UAI 2022), \"Boosting the Transferability of Adversarial Attacks with Reverse Adversarial Perturbation\" (NeurIPS 2022) could be considered.\n- No code is provided as supplementary materials. It is impossible to evaluate the claimed contribution of a unified codebase. Open-sourcing the code is particularly important for benchmark papers, where its value depends on the cleanness and extensibility of its code base.\n- The usage of cross-validation is weak and not systematic. I think that the benchmark should provide a standardised protocol to tune hyperparameters and select any elements with cross-validation. For example, the paper should select the best combination of augmentation and optimizer by cross-validation (section 4.2). This is a significant flaw of the current paper. Some tuning of the hyperparameters is currently done by on a set of hold-out target models (Section 4.3). But these models are removed from the test set of targets models. Therefore, there is an overlap between validation and targets between the experiments. The test evaluation is weaker because performed on fewer models. It would be better to use additional models that would be used only for validation. As the code is not shared, I cannot check if the codebase includes an easy way to tune the hyperparameters of any techniques implemented in the benchmark. From my experience, tuning the hyperparameters is of first importance when changing some experimental settings. In particular, selecting the SGM hyperparameter when changing the source architecture. \n- A limitation of the current benchmark is that the training dataset is supposed to be known. I think this overestimates significantly the success rates (I suspect that the effect is larger than the one of the pre-processing). I understand removing this hypothesis is very costly computationally. But the paper should therefore not position itself as a more realistic evaluation. In addition, I think training and distributing source and target models without this hypothesis would add a high value to the benchmark (in practice, splitting ImageNet in two or introducing some distributional shift between the datasets).\n- The paper does not extend the evaluation of some techniques to other source architecture. For example, in Table 1, most columns are empty for substitute model training. I understand that some techniques are too costly to train from scratch (RFA for example), but LGV and MoreBayesian should be feasible since they require only a few additional epochs from a pretrained model. \n- The categories of techniques used are debatable. For example, I think that LGV and MoreBayesian should be categorised as gradient computation. Both techniques are training-based, but their primary objective is *not* to train a better single base substitute model, but to augment an existing base pretrained model and obtain several slightly modified models (to be used one per gradient). The \"augmentation\" category should be renamed \"input augmentation\" or \"data augmentation\" to clarify. I am not sure why input augmentation techniques and optimizers techniques are evaluated together as a single category of techniques, and \"gradient computation\" techniques are not evaluated together with optimizers too. In fact, numerous \"gradient computation\" techniques also augment the surrogate model (LinBP, GN, SGM, etc.). I think that a more granular evaluation of the category of techniques, category per category, as done in \\[59\\], would be more advisable.\n- I think that this paper would have been a perfect fit for the benchmark and datasets track of NeurIPS. I feel that the impact of the paper is slightly too limited for the main track. The conclusions of the benchmark have a somehow limited significance for the community.\n- The paper evaluates the gradient computation category on the best combination of augmentation and optimizer techniques. It would be good to evaluate the relation in both ways: the gradient computation category on I-FSGM vs. the gradient computation category on the best surrogate models.  \n    \n  **Minor comments**\n\n- The paper would benefit from polished writing and improved formatting. Bold should be used with care, only for sparse important keywords. It would be better to highlight entire sentences (or paragraph) with italic instead of bold. The writing should be improved overall. For example,  \"Similar for IR and TAIG.\" (l.272) is not a valid sentence, \"state-of-the-arts\" (l.183) is not a valid word. Exaggeration must be avoided. For example, please avoid familiar and exaggerated formulations like \"super close\" (l.329). The paper would benefit from a more precise writing. For example, the paper states that \"30+\" techniques are compared. The exact number of techniques is never mentioned. Some legends are missing some descriptions. For example, the legend of Table 1 does not specify what the grey colour means, and does not specify that the first row is the source model.\n- No evaluation of defended target models is performed. This is not necessarily a major issue, but it is advisable to discuss this limitation. Same for targeted/untargeted perturbations.\n- I think that the paper would be more clear if it reports success rates instead of accuracies.\n- Specifying the step size of I-FGSM relatively to the epsilon norm of the perturbation would be better and improve consistency. For example, use alpha = eps/10 for both Linf and L2 norms.\n\nQUESTIONS:\n- Could you share the code base (anonymously) to review it?\n- Which hyperparameters are currently selected by cross-validation? More details should be provided on how did you tune the hyperparameters (paragraph l.228-l.241).\n- Will you publish the code to sample the 5K subset of examples? This code is needed since this subset would need to be rebuilt for each new model training techniques, to ensure fairness. The 5K examples are correctly classified by all existing surrogate models, so it should be the case for    surrogate models that will be added later. It is important to specify this in the documentation.\n- Which number of random restarts did you use for PGD? I did not find this information in the paper.\n- Line 324 states that transformers are better surrogate architecture than CNN. Is it true for all types of targets? I.e. does this observation hold for the BAA (best-case accuracy)? Or is it simply related to the fact that there are more transformers in the set of target models (and we are confused by the average AAA)?\n- Can you describe more precisely the training differences, briefly mentioned line 332?\n\nI am ready to increase my score depending on how the authors answer the weaknesses and questions.\n\nLIMITATIONS:\nSome limitations are not discussed (see above weakness section)", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is a benchmark paper on transfer-based attack in the area of adversarial machine learning. The paper benchmarks 30+ methods on ImageNet, grouped into four principal categories: augmentation and optimizer; gradient computation; substitute model training, and generative models. The extensive evaluation results uncover several interesting novel insights. For instance, employing Vision Transformer (ViT) architectures in the training of substitute models can generally enhance the efficacy of transfer attacks. Additionally, the MoreBayesia method has consistently demonstrated an improved transferability of adversarial examples.\n\nSTRENGTHS:\n1.\tThe evaluation is comprehensive, encompassing over 30 implemented and evaluated methods, thereby offering a thorough comparison of state-of-the-art transfer-based attacks.\n2.\tAdditionally, this benchmark investigates the robustness of transfer-based attacks on ViT, an aspect often overlooked in other papers.\n3. The evaluation results provide novel insights into the design of more potent transfer-based attacks.\n\nWEAKNESSES:\nThe paper conducts an extensive and systematic study on transfer-based attacks, comparing state-of-the-art approaches. However, one concern is the absence of significant novel insights. While the paper does offer interesting findings, such as the improved performance of transfer attacks when using ViT as substitute models, it could benefit from additional unique insights to meet the standards expected at NeurIPS.\n\n\nFurthermore, the paper lacks a discussion on other types of black-box attacks, despite transfer-based attacks being a specific form of such attacks. It would be valuable to include a discussion on related work concerning different black-box attack techniques, highlighting the similarities and differences between various approaches. This would provide a more comprehensive understanding of transfer-based attacks in the broader context of black-box adversarial attacks.\n\nQUESTIONS:\nSee the weakness part.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors present a new transfer-based attack benchmark (TA-Bench) to evaluate the transferability of adversarial attacks. TA-Bench implements 30+ adversarial attacks with 10 substitute models and introduces more advanced optimization back-ends that incorporate augmentation and different choices of optimizers. The benchmark provides a means to compare different adversarial attacks systematically, fairly and practically. Evaluation results bring new and interesting insights about existing attacks.\n\nSTRENGTHS:\n1. This is a solid paper with clear motivation and comprehensive experimental results.\n2. The proposed TA-Benchmark provides a systematical, practical and fair way to compare different adversarial attacks.\n3. TA-benchmark covers attacks with various mechanisms,  including the implementation of many more recent and advanced attacks.\n4. The authors bring new insights about existing adversarial attacks such as choices of augmentation and optimizer have impacts on effectiveness and transformers are better substitute models for gradient computation-based attacks.\n5. The paper is well written and easy to follow. I enjoyed reading this paper.\n\nWEAKNESSES:\n1. The codebase is not provided (maybe for anonymity). Thus it is hard to evaluate the accessibility, portability, scalability and usability of the benchmark. For example, how difficult would it to evaluate a new attack using TA-bench? Is there any interface for users to quickly implement their own attacks or select their own substitute models?\n2. Since the main contribution of this work is providing a fair and practical benchmark to evaluate different attacks. It is worth discussing the standards and metrics for the evaluation. However, I don't find an explicit explanation of such standards and metrics.\n3. The adversarial perturbations ($\\epsilon$) are set to be fixed in the experiments. However, attackers may use different $\\epsilon$ in practice.\n\nQUESTIONS:\nPlease see weaknesses for details.\n\nLIMITATIONS:\nNo potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-16"}

comments: SUMMARY:
In this paper, the authors have presented benchmark for transfer-based attacks, in which they have implemented 30+ advanced transfer-based attack methods, including those focus on augmentation and optimizer innovation, those “gradient computation” methods, those “substitute model training” methods, and those applying generative modeling.  And by evaluating and comparing transfer-based attacks systematically, they have some new insights.

STRENGTHS:
1. This paper is the first transfer-based attack benchmark, it will help people in this field.

2. The authors have implemented 30+ methods, including augmentation, optimizer, gradient computation, substitute model training, and generative model. Various methods make sure its fairness and comprehensiveness.

3. It is helpful that the results of the experiment show some beneficial conclusions.

WEAKNESSES:
1. There is only one type of dataset used in the experiment, which makes results lack credibility.

2. This paper mainly evaluates transfer-based methods on the base of two back-ends, and the results are quite different. It lacks more explanation.

3. The authors have maked a bilinear interpolation and resize in the pre-processing, which has an influence on following evaluations.

QUESTIONS:
1. Why is attack performance of ILA, ILA++, and NAA using I-FGSM quite different with that using new back-end?

2. The analysis of Figure 3 in Section 4.3 is too short to make readers understand why the most effective factor of these methods may be input augmentation and gradient averaging.

LIMITATIONS:
See Weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-20