<figure><img src="image_1.png" /></figure>

HAL
open science

# Revisiting Transferable Adversarial Images: Systemization, Evaluation, and New Insights

Zhengyu Zhao, Hanwei Zhang, Renjue Li, Ronan Sicre, Laurent Amsaleg,

Michael Backes, Qi Li, Qian Wang, Chao Shen

► To cite this version:

Zhengyu Zhao, Hanwei Zhang, Renjue Li, Ronan Sicre, Laurent Amsaleg, et al. Revisiting Transferable Adversarial Images: Systemization, Evaluation, and New Insights. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025, pp.1-16. 10.1109/TPAMI.2025.3610085. hal-05267252

HAL Id: hal-05267252

https://inria.hal.science/hal-05267252v1

Submitted on 8 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Revisiting Transferable Adversarial Images:
Systemization, Evaluation, and New Insights

Zhengyu Zhao, Member, IEEE, Hanwei Zhang, Renjue Li, Ronan Sicre, Laurent Amsaleg, Member, IEEE, Michael Backes, Fellow, IEEE, Qi Li, Senior Member, IEEE, Qian Wang, Fellow, IEEE, Chao Shen, Senior Member, IEEE

**Abstract**—Transferable adversarial images raise critical security concerns for computer vision systems in real-world, black-box attack scenarios. Although many transfer attacks have been proposed, existing research lacks a systematic and comprehensive evaluation. In this paper, we systemize transfer attacks into five categories around the general machine learning pipeline and provide the first comprehensive evaluation, with 23 representative attacks against 11 representative defenses, including the recent, transfer-oriented defense and the real-world Google Cloud Vision. In particular, we identify two main problems of existing evaluations: (1) for attack transferability, lack of intra-category analyses with fair hyperparameter settings, and (2) for attack stealthiness, lack of diverse measures. Our evaluation results validate that these problems have indeed caused misleading conclusions and missing points, and addressing them leads to new, consensus-challenging insights, such as (1) an early attack, DI, even outperforms all similar follow-up ones, (2) the state-of-the-art (white-box) defense, DiffPure, is even vulnerable to (black-box) transfer attacks, and (3) even under the same $L_p$ constraint, different attacks yield dramatically different stealthiness results regarding diverse imperceptibility metrics, finer-grained measures, and a user study. We hope that our analyses will serve as guidance on properly evaluating transferable adversarial images and advance the design of attacks and defenses.

# I. INTRODUCTION

DEEP Neural Networks (DNNs) have achieved great success in various machine learning tasks. However, they are known to be vulnerable to adversarial examples [20], [74], which are intentionally perturbed model inputs to induce prediction errors. Adversarial examples have been studied in the white-box scenario to explore the worst-case model robustness but also in the black-box scenario to understand the real-world threats to deployed models.

An intriguing property of adversarial examples that makes them threatening in the real world is their transferability. Transfer attacks assume a realistic scenario the adversarial examples generated on a (local) surrogate model can be directly transferred to the (unknown) target model [41], [54].

Such attacks require no “bad” query [13] interaction with the target model, and as a result, they eliminate the possibility of being easily flagged by the target model under attack [11]. Due to their real-world implications, transfer attacks have attracted great attention, and an increasing number of new methods have been rapidly developed.

However, existing research still lacks a systematic evaluation of transferable adversarial images in a common, comprehensive threat model. Specifically, we identify two major problems of common evaluation practices as follows.

**Problem 1.** Existing evaluations on transferability are often *unsystematic* and sometimes *unfair*. Specifically, they are unsystematic since there lacks intra-category attack analyses. This is problematic particularly when both attacks are based on the same category of techniques (see our definition in Section II). In this case, it is unclear whether a newly proposed method can be a good replacement for a previous similar method. For example, the original work of TI [16] does not directly compare the proposed TI to the previous method, DI [94], although both of them are based on input augmentations. However, we find that actually, TI is consistently worse than DI. Moreover, unfair hyperparameter settings are sometimes adopted. For example, recent input augmentation-based attacks (i.e., Admix [79], SI [40], and VT [78]) adopt multiple input copies but are directly compared to previous similar attacks that adopt only one input copy (i.e., DI [94] and TI [16]). However, we find that when adopting the same number of input copies, the previous attacks perform much better.

**Problem 2.** Existing evaluations solely compare attack strength, i.e., transferability, but largely overlook another key attack property, stealthiness. Instead, they pre-fix the stealthiness based on simply imposing a single $L_p$ norm on the perturbations. This is problematic since it is well known that a successful attack requires a good trade-off between attack strength and stealthiness. More generally, (black-box) transfer attacks normally require larger perturbations than white-box attacks for success, but using a single $L_p$ norm is known to be insufficient when perturbations are not very small [10], [67], [75]. In addition, other stealthiness measures beyond the image distance have not been explored.

In this paper, we address the above two major problems by establishing new evaluation guidelines from both the perspectives of transferability and stealthiness. To address Problem 1, we first systemize existing transfer attacks into five categories (Section II). Based on this attack systematization, we then conduct systematic, intra-category attack comparisons (Section IV). Our intra-category analyses also help determine

Zhengyu Zhao and Chao Shen are with Xi'an Jiaotong University, Xi'an, China. Hanwei Zhang is with the Institute of Intelligent Software, Guangzhou, China, and Saarland University, Saarbrücken, Germany. Renjue Li is with the Institute of AI for Industries/CAS, Nanjing, China, and the Institute of Software/CAS, Beijing, China. Ronan Sicre is with LIS - Ecole Centrale Marseille, Marseille, France. Laurent Amsaleg is with Inria/Univ Rennes/CNRS/IRISA, Rennes, France. Michael Backes is with CISPA Helmholtz Center for Information Security, Saarbrücken, Germany. Qi Li is with Tsinghua University, Beijing, China. Qian Wang is with Wuhan University, Wuhan, China.

The first three authors make equal contributions.

Zhengyu Zhao is the corresponding author.
<table><caption>TABLE I: Evaluation practices in existing work compared to ours regarding attack transferability and stealthiness.</caption><thead><tr><th>Evaluation</th><th>Transferability</th><th>Stealthiness</th></tr></thead><tbody><tr><td>Existing</td><td>Unsystematic: Lack of intra-category comparisons (half the time) X<br>Unfair: Inconsistent hyperparameter settings for similar attacks X</td><td>Pre-fixed L<sub>∞</sub> norm without comparisons X</td></tr><tr><td>Ours</td><td>Systematic: Intra-category comparisons with detailed analyses ✓<br>Fair: Consistent hyperparameter settings for similar attacks ✓</td><td>Pre-fixed L<sub>∞</sub> norm with diverse comparisons ✓<br>(1) Five popular imperceptibility metrics<br>(2) Two new fine-grained stealthiness measures from the perspective of attack traceback<br>(3) A user study on the image quality</td></tr></tbody></table>

<table><caption>TABLE II: Overview of our evaluated transfer attacks.</caption><thead><tr><th>Gradient Stabilization</th><th>Input Augmentation</th><th>Feature Disruption</th><th>Surrogate Refinement</th><th>Generative Modeling</th></tr></thead><tbody><tr><td>MI [15] (CVPR'18)</td><td>DI [94] (CVPR'19)</td><td>TAP [113] (ECCV'18)</td><td>SGM [89] (ICLR'20)</td><td>GAP [57] (CVPR'18)</td></tr><tr><td>NI [40] (ICLR'20)</td><td>TI [16] (CVPR'19)</td><td>AA [28] (CVPR'19)</td><td>LinBP [21] (NeurIPS'20)</td><td>CDA [50] (NeurIPS'19)</td></tr><tr><td>PI [80] (BMVC'21)</td><td>SI [40] (ICLR'20)</td><td>ILA [24] (ICCV'19)</td><td>RFA [71] (NeurIPS'21)</td><td>TTP [49] (ICCV'21)</td></tr><tr><td></td><td>VT [79] (CVPR'21)</td><td>FIA [83] (ICCV'21)</td><td>IAA [114] (ICLR'22)</td><td>GAPF [30] (NeurIPS'21)</td></tr><tr><td></td><td>Admix [79] (ICCV'21)</td><td>NAA [106] (CVPR'22)</td><td>DSM [96] (ICTAF'23)</td><td>BIA [107] (ICLR'22)</td></tr></tbody></table>

<table><caption>TABLE III: Overview of our evaluated defenses.</caption><thead><tr><th>Input Pre-processing</th><th>Purification Network</th><th>Adversarial Training</th><th>Practical Systems</th></tr></thead><tbody><tr><td>BDR [95] (NDSS'18)</td><td>HGD [39] (CVPR'18)</td><td>AT<sub>∞</sub> [93] (CVPR'19)</td><td>PubDef [69] (ICLR'24)</td></tr><tr><td>PD [58] (CVPR'18)</td><td>NRP [48] (CVPR'20)</td><td>FD<sub>∞</sub> [93] (CVPR'19)</td><td>Google Cloud</td></tr><tr><td>R&P [92] (ICLR'18)</td><td>DiffPure [53] (ICML'22)</td><td>AT<sub>2</sub> [62] (NeurIPS'20)</td><td>Vision API [1]</td></tr></tbody></table>

the optimal settings of key attack hyperparameters and ensure fairness in our comprehensive attack evaluations (Section V). To address Problem 2, we compare different attacks regarding diverse stealthiness measures. Specifically, we adopt five different imperceptibility metrics. Moreover, we move beyond imperceptibility and investigate other aspects of stealthiness, e.g., attack traceback with (input) image features or (output) misclassification features. We also conducted a user study on the visual quality of adversarial images from different attacks.

Overall, we present the first large-scale evaluation of transferable adversarial images on ImageNet, covering 23 representative attacks (in 5 categories) against 11 representative defenses. These attacks and defenses have been published in the past 5 years (i.e., 2018-2023) at top-tier machine learning and security conferences (e.g., ICLR, ICML, NeurIPS, CVPR, ICCV, ECCV, NDSS), which can represent the current state of the art in transferable adversarial examples. In particular, the defenses include the state-of-the-art, transferability-oriented method, PubDef [69], and the real-world Google Cloud Vision API [1]. Table I compares our evaluation to those in existing work in terms of attack transferability and stealthiness. More detailed comparisons can be found in Table IX of Appendix A. As can be seen, existing work considers 11 attacks at most and has not conducted intra-category analyses. In addition, only *half* of transferability comparisons are considered to be fair (see detailed discussion in Section IV).

The most relevant previous work is the recent [36], which focuses on achieving an optimal combination of different transfer attacks through empirical comparisons. In contrast, we focus on systemizing existing transfer attacks and address major problems in existing evaluations. To the best of our knowledge, most existing work does not consider attack systemization, and a few recent attempts [81], [104] are not as complete and detailed as ours and do not provide intra-category analyses. Another study [47] has provided a large-scale evaluation of attack transferability against private MLaaS platforms but only

considers conventional attacks published before 2017, e.g., FGSM, CW, and DeepFool.

We draw new insights that complement or even challenge existing knowledge about transfer attacks. Key insights into both attack transferability and stealthiness are:

**New insights into transferability.** In general, we find that the performance of different methods is highly contextual, i.e., the optimal (category of) attack/defense for one (category of) defense/attack may not be optimal for another. In particular, we reveal that a defense may largely overfit specific attacks that generate adversarial examples similar to those used in the optimization of that defense. For example, DiffPure [53] claims the state-of-the-art white-box robustness and even beats adversarial training [46] in their original evaluation. On the contrary, we demonstrate that *DiffPure is surprisingly vulnerable to (black-box) transfer attacks* that produce semantic perturbations (within the claimed norm bounds). See Section V for detailed results. Our intra-category attack analyses also lead to surprising findings regarding transferability. For example, for gradient stabilization attacks, *using more iterations may even decrease the performance*. Another example is that the earliest input augmentation-based attack, *DI*, *actually outperforms all subsequent attacks* when they are compared fairly with the same number of input copies.

**New insights into stealthiness.** In general, we find that stealthiness may be at odds with transferability. Specifically, *almost all transfer attacks yield worse imperceptibility than the PGD baseline*, in terms of five metrics. This suggests that it is problematic to solely constrain all attacks with the same $L_∞$ norm bound without more comprehensive comparisons. Moreover, different attacks yield dramatically different stealthiness characteristics, in terms of imperceptibility and other aspects. For example, the adversarial images generated from different attacks are well separable solely based on their visual features. In addition, different attacks lead to different misclassification patterns regarding the model output distribution. These new
<figure><img src="image_3.png" /><figcaption>Fig. 1: Illustration of our attack systemization around the general machine learning pipeline, containing T: input, f: model, L: loss, and g: gradient. Note that the generative modeling attack with the generator G is independent.</figcaption></figure>

explorations suggest the possibility of *attack traceback* based on (input) image features or (output) misclassification features.

Overall, our extensive evaluation results validate that the existing problematic evaluations have indeed caused *misleading conclusions and missing points*, and as a result, hindered the assessment of the actual progress in this field.<fnref n="1" /> Our new insights could potentially advance the design of new transfer attacks and defenses (see details in Section V). Based on our analyses, we recommend the following actionable practices:

* (Evaluation Guideline) Generate a transferability vs. iteration curve, to avoid the “no convergence” or “over-fitting” of an attack.

* (Evaluation Guideline) Adopt fair hyperparameter settings when comparing attacks from the same category.

* (Evaluation Guideline) Measure the stealthiness from diverse aspects beyond just a single $L_p$ norm.

* (Attack Design) Examine individual attacks in their corresponding category before integrating them but do not just use the new attacks.

* (Defense Design) Consider diverse potential attacks when designing a defense for better generalizability.

## II. SYSTEMIZATION OF RELATED WORK

In this section, we introduce our systemization of transfer attacks. First of all, these attacks can be generally divided into two types: iterative and generative attacks. For iterative attacks, we derive four categories: gradient stabilization, input augmentation, feature disruption, and surrogate refinement. Each of these four categories operates on one of the four major components in the general machine learning pipeline: gradient, (image) input, (loss) output, and (surrogate) model. The relation between these four categories and the pipeline is illustrated in Fig. 1. Due to the specific nature of generative attacks, we leave it as an independent category: generative modeling.

In general, these five categories are disjoint to each other, i.e., each attack falls into *only one* category according to the technique that was claimed as the key contribution in its

<table><caption>TABLE IV: Key ideas of different attack categories.</caption><thead><tr><td>Attack Categories</td><td>Key Ideas</td></tr></thead><tbody><tr><td>Gradient Stabilization</td><td>Stabilize the gradient update</td></tr><tr><td>Input Augmentation</td><td>Augment the input image</td></tr><tr><td>Feature Disruption</td><td>Disrupt intermediate-layer features</td></tr><tr><td>Surrogate Refinement</td><td>Refine the surrogate model</td></tr><tr><td>Generative Modeling</td><td>Learn a perturbation generator</td></tr></tbody></table>

original work. Different attack methods that share the same key idea fall into the same category and can be compared systematically in a single dimension. Table IV summarizes the key ideas of the five categories. Our attack systemization is also suggested to be reliable based on a perturbation classification task (see Section V-B).

In the following, we present these five categories in turn. Before that, we briefly formulate the optimization process of adversarial attacks in image classification. Given a classifier $f(x): x \in \mathcal{X} \to y \in \mathcal{Y}$ that predicts a label y for an original image x, an attacker aims to create an adversarial image $x'$ by perturbing x. For iterative attacks, which fall into our first four categories, the adversarial image is iteratively optimized based on gradients:

$$x'_{0} = x, \quad x'_{i+1} = x'_{i} + \eta \cdot \text{sign}(g_{i}), \quad \text{where } g_{i} = \nabla_{x} \mathcal{L}. \quad (1)$$

Here, $x'_{i}$ is the updated image at the i-th iteration, $g_{i}$ is the intermediate gradient vector, and $\mathcal{L}$ is the adversarial loss function. In order to speed up the optimization and ensure the image is in the original quantized domain (e.g., with 8-bit), a sign operation is commonly added to keep only the direction of the gradients but not their magnitude [20]. Specifically, the gradient calculation in the baseline attack, PGD [32], [46], can be formulated as follows:

$$g_i = \nabla_x \mathcal{L}_{CE}(x'_i, y, f), \quad (2)$$

where the loss function $\mathcal{L}_{CE}$ is the cross-entropy loss, which takes as inputs the image, label, and model. In each of the following four attack categories, only one of the four major components (i.e., gradient update, input, loss function, and model) in Equation 7 would be modified.

### A. Gradient Stabilization

Different DNN architectures tend to yield radically different decision boundaries, yet similar test accuracy, due to their high non-linearity [41], [70]. For this reason, the attack gradients calculated on a specific (source) model may cause the adversarial images to trap into local optima, resulting in low transferability to a different (target) model. To address this issue, popular machine learning techniques that can stabilize the update directions and help the optimization escape from poor local maxima during the iterations are useful. Existing work on transfer attacks has adopted two such techniques, momentum [56] and Nesterov accelerated gradient (NAG) [52], by adding a corresponding term into the attack optimization.

In this way, the attack is optimized based on gradients from not only the current iteration but also other iteration(s)

<footnote n="1">¹Code and a curated list of transferability-related papers are available at https://github.com/ZhengyuZhao/TransferAttackEval.</footnote>
determined by the additional term. Specifically, the momentum is to accumulate previous gradients, i.e., looking back [15], and the Nesterov accelerated gradient (NAG) is an improved momentum that additionally accumulates future gradients, i.e., looking ahead [40]. In particular, NAG is later examined in [80], where using only the single previous gradient for looking ahead is found to work better than using all previous gradients. In terms of the formulation, the gradient calculation in Equation 7 becomes:

$$
\begin{align}
\text{MI [15]:} \quad \mathbf{g}_i &= \mu \cdot \mathbf{g}_{i-1} + \nabla_x \mathcal{L}_{CE}(\mathbf{x}'_i, y, f), \nonumber \\
\text{NI [40]:} \quad \mathbf{g}_i &= \mu \cdot \mathbf{g}_{i-1} + \nabla_x \mathcal{L}_{CE}(\mathbf{x}'_i + \alpha \cdot \mathbf{g}_{i-1}, y, f), \tag{3} \\
\text{PI [80]:} \quad \mathbf{g}_i &= \mu \cdot \mathbf{g}_{i-1} + \nabla_x \mathcal{L}_{CE}(\mathbf{x}'_i + \alpha \cdot \hat{\mathbf{g}}_{i-1}, y, f), \nonumber \\
\text{where } \hat{\mathbf{g}}_{i-1} &= \nabla_x \mathcal{L}_{CE}(\mathbf{x}'_{i-1}, y, f), \nonumber
\end{align}
$$

where the first, momentum term in each equation is for looking back, and the second is for looking ahead.

## B. Input Augmentation

Machine learning is a process where a model is optimized on a local dataset such that it can generalize to unseen test images. Following the same principle, learning transfer attacks can be treated as a process where an adversarial image is optimized on a local surrogate model such that it can generalize/transfer to unseen target models [38]. Due to this conceptual similarity, various techniques that are used for improving model generalizability can be exploited to improve attack transferability. Data augmentation is one such technique that is commonly explored in existing work on transfer attacks. To this end, the adversarial images are optimized to be invariant to a specific image transformation. In terms of the formulation, the gradient calculation in Equation 7 becomes:

$$ \mathbf{g}_i = \nabla_x \mathcal{L}_{CE}(T(\mathbf{x}'_i), y, f), \quad (4) $$

where the input image is first transformed by a specific image transformation $T$.

According to the idea of data augmentation, the transformation is required to be sematic-preserving. For example, in image classification, a transformed image should still be correctly classified. Existing work has explored various types of image transformations, such as geometric transformations (e.g., resizing & padding [94] and translation [16]), pixel value scaling [40], random noise [78], and mixing images from incorrect classes [79]. Recent studies also explore more complex methods that refine the above transformations [59], [99], [115], leverage regional (object) information [6], [37], [86], or rely on frequency-domain image transformations [43].

## C. Feature Disruption

It is understandable that adversarial attacks in image classification commonly adopt cross-entropy loss because this loss is also used for classification by default. However, this choice may not be optimal for transfer attacks since the output-layer, class information is often model-specific. In contrast, features extracted from the intermediate layers are known to be more generic [31], [98]. Based on this finding, existing work on

transfer attacks has proposed to replace the commonly used cross-entropy loss with some feature-based loss. These studies aim to disrupt the feature space of an image such that the final class prediction would change. Specifically, the distance between the adversarial image and its corresponding original image in the feature space is maximized. In terms of the formulation, the gradient calculation in Equation 7 becomes:

$$ \mathbf{g}_i = \nabla_x d(f_l(\mathbf{x}'_i), f_l(\mathbf{x})), \quad (5) $$

where $f_l$ calculates the feature output at a specific intermediate layer $l$ of the classifier, and $d(\cdot, \cdot)$ calculates the feature difference between the original and adversarial images.

Early methods of feature disruption attacks simply adopt the $L_2$ distance for $d(\cdot, \cdot)$ [18], [24], [35], [42], [51], [113]. In this way, the resulting feature disruption is indiscriminate at any image location. However, these methods are sub-optimal because the model decision normally only depends on a small set of important features [112]. To address this limitation, later methods [83], [90], [106] instead calculate the feature difference $d(\cdot, \cdot)$ based on only important features. In this case, the importance of different features is normally determined based on model interpretability results [64], [72]. Feature disruption is also explored for the relatively challenging, targeted attacks. In this case, rather than maximizing the distance between the original image and the adversarial image, the attack minimizes the distance between the original image and a target image [28] or a target distribution determined by a set of target images [26], [27]. Beyond feature disruption, other losses are also explored specifically for targeted attacks, such as the triplet loss [34] or logit loss [110].

## D. Surrogate Refinement

The current CNN models are commonly learned towards high prediction accuracy but with much less attention given to transferable representations. Several recent studies find that adversarial training could yield a model that better transfers to downstream tasks, although it inevitably trades off the accuracy of a standard model in the source domain [14], [62], [77]. This finding encourages researchers to explore how to refine the surrogate model to improve the transferability of attacks. In terms of the formulation, the gradient calculation in Equation 7 becomes:

$$ \mathbf{g}_i = \nabla_x \mathcal{L}_{CE}(T(\mathbf{x}'_i), y, f'), \quad (6) $$

where $f'$ denotes a refined version of the original model $f$.

The underlying assumption of surrogate refinement is a smooth surrogate model leads to transferable adversarial images. To this end, existing studies have explored three different ways, in which a standard model is modified in terms of training procedures [71], [96], [102] and architectural designs [21], [60], [89], [101], [114]. Specifically, in terms of training procedures, using an adversarially-trained surrogate model is helpful since robust features are known to be smoother than non-robust features [25]. Similarly, early stopping [102] or training with soft annotations [96] are also shown to yield a smoother model. In terms of architectural designs, it is helpful to replace the non-linear, ReLU activation function with a
smoother one [21], [101], [114]. For ResNet-like architectures, backpropagating gradients through skip connections can lead to higher transferability compared to residual modules [89]. This line of research also leads to rethinking the relation between the accuracy of the surrogate model and the transferability [102], [114].

### E. Generative Modeling

In addition to the above iterative attacks, existing work also relies on Generative Adversarial Networks (GANs) [19] to directly generate transferable adversarial images. Generative modeling attacks can efficiently generate perturbations for any given image with only one forward pass. The GAN framework consists of two networks: a generator and a discriminator. The generator captures the data distribution, and the discriminator estimates the probability that a sample came from the training data rather than the generator. In the typical GAN training, these two networks are simultaneously trained in a minimax two-player game.

In the context of adversarial attacks, only the generator is trained on additional (clean) data, while a pre-trained image classifier is fixed as the discriminator. The training objective of the generator is to cause the discriminator to misclassify the generated image. A clipping operation is normally applied after the generation to constrain the perturbation size. Specifically, the perturbations output from the generator are initially unbounded and then clipped by a specific norm bound. This means that once trained, the generator $G$ can be used to generate perturbations under any target norm bound $\epsilon$:

$$x' = \text{Clip}(G(x), \epsilon). \qquad (7)$$

Existing studies have explored different loss functions and generator architectures. Specifically, the earliest generative attack [57] adopts the widely-used cross-entropy loss. Later studies further adopt a relativistic cross-entropy loss [50] or intermediate-level, feature losses [30], [107]. Class-specific [49] and class-conditional [97] generators are designed to particularly improve targeted transferability.

## III. EVALUATION METHODOLOGY

### A. Threat Model

Following the common practice, we specify our threat model from three dimensions: attacker's knowledge, attacker's goal, and attacker's capability [5], [7], [55]. We make sure our threat model follows the most common settings in existing work regarding five aspects, as summarized in Table X of Appendix A. This threat model focuses on *untargeted* attacks against *public ImageNet* models that are trained on the *same* data with the convolutional neural networks (CNNs) or Vision Transformers (ViTs).

**Attacker's knowledge.** An attacker can have various levels of knowledge about the target model. In the ideal, white-box case, an attacker has full control over the target model. In the realistic, black-box case, an attacker has either query access to the target model or in our transfer setting, no access but only leverages a surrogate model. In this work, we follow

most existing work to explore the attack transferability between image classifiers that are trained on the same public dataset (here ImageNet) but with different architectures. Cross-dataset transferability is rarely explored [50], [107] and beyond the scope of this work.

**Attacker's goal.** In general, an attacker aims at either untargeted or targeted misclassification. An untargeted attack aims to fool the classifier into predicting any other class than the original one, i.e., $f(x') \neq y$. A targeted attack aims at a specific incorrect class $t$, i.e., $f(x') = t$. Technically, an untargeted attack is equal to running a targeted attack for each possible target and taking the closest [8]. Achieving the targeted goal is strictly more challenging because a specific (target) direction is required, while many (including random) directions suffice for untargeted success. Targeted success and non-targeted success follow fundamentally different mechanisms. Specifically, targeted success requires representation learning of the target class while untargeted success is relevant to model sensitivity to small variations [49], [110]. In this work, we follow most existing work to focus on untargeted transferability while leaving the more challenging, targeted transferability for future work.

**Attacker's capability.** In addition to the adversarial effects, an attack should be constrained to stay stealthy. Most existing work addresses this by pursuing the imperceptibility of perturbations, based on the simple, $L_p$ norms [8], [20], [32], [55], [74] or more advanced metrics [2], [12], [29], [44], [61], [88], [91], [103], [109]. There are also recent studies on "perceptible yet stealthy" adversarial modifications [4], [66], [111]. It is worth noting that the constraint on the attack budget should also be carefully set. In particular, unreasonably limiting the iteration number of attacks is shown to be one of the major pitfalls in evaluating adversarial robustness [7], [76], [110]. In this work, we follow most existing work to constrain the perturbations to be imperceptible by using the $L_\infty$ norm. We also consider more advanced perceptual metrics for measuring imperceptibility. Beyond imperceptibility, we further look into other aspects of attack stealthiness, such as perturbation characteristics and misclassification patterns. For the constraint on the attack budget, we ensure the attack convergence by using sufficient iterations for iterative attacks and sufficient training epochs for generative attacks.

### B. Experimental Setups

**Attacks and defenses.** For each of the above five attack categories presented in Section II, we select 5 representative attacks<fnref n="2" />, resulting in a total number of 23 attacks, as summarized in Table II. Following the common practice, the perturbations are constrained by the $L_\infty$ norm bound with $\epsilon = 16/255$. Beyond directly feeding the adversarial images into the (standard) model, we consider 9 representative defenses from 3 different categories as well as 2 practical defenses, as summarized in Table III. Note that these defenses were not specifically developed against any single attack method but are generally applicable. Detailed descriptions of these attacks and defenses are provided in Appendix D.

<footnote n="2">²We only select 3 gradient stabilization attacks since they are less studied.</footnote>
**Dataset and models.** We follow the common threat model shown in Table X to use ImageNet and four CNNs with diverse architectures, i.e., InceptionV3 [73], ResNet50 [22], DenseNet121 [23], and VGGNet19 [68], as well as the Vision Transformer (ViT), i.e., ViT-B-16-224 [17], as the target model. We randomly select 5000 images (5 per class<fnref n="3" />) from the validation set that are correctly classified by all the above four models. All original images are resized and then cropped to the size of 299×299 for Inception-V3 and 224×224 for the other four models. Note that we do not pre-process any adversarial image since input pre-processing is discussed as a specific category of defense in our evaluation.

### C. Evaluation Metrics

**Transferability.** The transferability is measured by the (untargeted) attack success rate on the target model. Given an attack $\mathcal{A}$ that generates an adversarial image $\mathbf{x}'_n$ for its original image with the true label $y_n$ and target classifier $f$, the transferability over $N$ test images is defined as:

$$ \mathrm{Suc}(\mathcal{A}) = \frac{1}{N} \sum_{n=1}^{N} 1(f(\mathbf{x}'_n) \neq y_n), \quad (8) $$

where $1(\cdot)$ is the indicator function. Note that here the attack success rate on the (white-box) source model is not relevant to transferability.

**Stealthiness.** The imperceptibility is measured using a variety of perceptual similarity metrics: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM) [84], $\Delta E$ [45], [109], Learned Perceptual Image Patch Similarity (LPIPS) [108], and Frechet Inception Distance (FID) [73]. Detailed formulations of these metrics can be found in Appendix F. Beyond the above five imperceptibility measures, we also look into other, finer-grained stealthiness characteristics regarding the perturbation and misclassification patterns. Detailed descriptions of these characteristics can be found in Section V-B.

## IV. INTRA-CATEGORY ANALYSES

In this section, we conduct systematic, one-to-one analyses of similar attacks in each of the five categories. In this way, similar attacks can be compared along a single dimension with fair settings of key hyperparameters. In particular, each attack is implemented with only its core technique, although it may integrate others by default when it was originally proposed.

### A. Analysis of Gradient Stabilization Attacks

Fig. 2 Top shows the transferability of the three gradient stabilization attacks under various iterations. As can be seen, all three attacks converge very fast, within 10 iterations, since they accumulate gradients with the momentum. However, using more iterations does not necessarily improve, but may even cause the “overfitting” phenomenon. However, this “overfitting” phenomenon was previously overlooked since existing

<figure><img src="image_4.png" /><figcaption>Fig. 2: Results for gradient stabilization attacks. Top: Transferability vs. iteration curve. Additional results on another three surrogates with consistent patterns are in Fig. 14 of Appendix B. Bottom: Impact of the number of looking-ahead iterations.</figcaption></figure>

studies directly adopt only a few iterations (for all different categories of attacks). Such an “overfitting” phenomenon is also found to be contrastive to “no convergence” of the other categories of attacks (see detailed discussion in the following subsections).

Since the above “overfitting” phenomenon is newly found, we further explore whether it also applies to the more challenging, targeted attacks. We find that although the results of MI and NI are always lower than 3%, they still suffer from the same “overfitting” problem. In contrast, PI performs better, and its targeted transferability finally reaches 9.4% on average. This difference between PI for non-targeted and targeted transferability might be because targeted transferability especially requires an accurate direction [110], and PI achieves this through additional looking ahead [80].

When comparing different attacks, we can observe that PI and NI perform better than MI due to looking ahead, but NI is better only at the beginning. The only difference between PI and NI is the maximum number of previous iterations used to look ahead. To figure out the impact of the number of look-ahead iterations, we repeat the experiments with values from 1 (i.e., PI) to 100 (i.e., NI). Fig. 2 Bottom shows that the performance drops more as more previous iterations are incorporated to look ahead, and the optimal performance is actually achieved by PI.

### B. Analysis of Input Augmentation Attacks

Fig. 3 Top shows the transferability of the five input augmentation attacks under various iterations. Previous evaluations of input augmentation attacks often conduct unfair comparisons where different attacks may leverage a different number of random input copies [79]. In contrast, our comparisons are under the same number of random input copies. We find that, surprisingly, the earliest attack methods, DI and TI, always perform the best. However, the latest method, Admix, achieves very low transferability.

<footnote n="3">³Several classes contain fewer than 5 eligible images.</footnote>
<figure><img src="image_5.png" /><figcaption>Fig. 3: **Results for input augmentation attacks.** Top: Transferability vs. iteration curve. All attacks use five input copies. Additional results on another three surrogates with consistent patterns are in Fig. 15 of Appendix B. **Bottom:** Impact of the number of input copies.</figcaption></figure>

We explain the above new finding by comparing the input diversity caused by different attacks since higher input diversity is commonly believed to yield higher transferability [43], [94]. Specifically, we quantify the input diversity of the five attacks by the change of the logit value of the top-1 model prediction caused by their specific augmentations. We calculate the input diversity over 5000 images with 10 repeated runs. We get 0.23 for DI, 0.0017 for TI, 0.00039 for SI, 0.000091 for VT, and -0.035 for Admix. The highest input diversity of DI explains its highest transferability and the lowest input diversity of Admix explains its lowest transferability. In particular, Admix even decreases the original logit value, denoting that it is not a suitable choice for the required label-preserving augmentation.

We further explore the impact of the number of random input copies. Fig. 3 Bottom shows that the transferability of all attacks is improved with more copies. Therefore, we would get misleading conclusions if we follow existing studies (e.g., Admix, SI, and VT) to (unfairly) set different numbers of copies for different attacks. For example, for IncV3 → RN50, VT with 20 copies is shown to be better than DI with only one copy (68.62% vs. 65.38%), but when they are compared fairly with 20 copies, VT is far behind (68.62% vs. 76.46%).

Specifically, DI and TI consistently perform the best in all settings. This finding suggests the global superiority of spatial transformations (i.e., resizing&padding in DI and translation in TI) over other types of image transformations, such as pixel scaling in SI, additive noise in VT, and image composition in Admix. Note that more copies generally consume more computational resources.

### C. Analysis of Feature Disruption Attacks

Fig. 4 Top shows the transferability of the five feature disruption attacks under various iterations. We see that FIA and NAA, which exploit feature importance, perform the best. In addition, ILA and TAP perform well, by incorporating the CE loss into their feature-level optimizations. Finally, AA

<figure><img src="image_6.png" /><figcaption>Fig. 4: **Results for feature disruption attacks.** Top: Transferability vs. iteration curve. TAP uses all layers and the others use “conv3_x”. **Bottom:** Impact of the layers.</figcaption></figure>

performs even worse than the PGD baseline since it only uses a plain feature loss but ignores the CE loss.

Since all these attacks, except TAP, only disrupt features in a specific layer, we explore the impact of layer choice. As can be seen from Fig. 4 Bottom, the last layer (“conv5_x”) performs much worse than the early layers. This might come from the last layer being too complex and model-specific, which results in poor generalizability to unseen models. Moreover, the mid layer (“conv3_x”) always achieves the best performance since it can learn more semantic features than earlier layers, which are known to capture low-level visual attributes, e.g., colors and textures [100]. This is consistent with existing conclusions on layer selection.

### D. Analysis of Surrogate Refinement Attacks

Fig. 5 shows the transferability of the five surrogate refinement attacks under various iterations. Here ResNet50 is used as the surrogate model for a fair comparison since SGM [89] can only be applied to architectures with skip connections. We see that IAA achieves the best results mainly because it does not simply integrate specific components (i.e., skip connections and continuous activation functions) but optimizes their hyperparameters towards high transferability. In addition, $RFA_\infty$ achieves lower performance (sometimes even lower than the baseline attack) since it solely uses a robust surrogate model, which is known to have different representation characteristics from the (target) standard model [25], [62], [63].

Model refinement opens a way to understand the impact of specific properties of the surrogate model on transferability. Thus we look into model properties in terms of two common metrics: accuracy and interpretability, as well as another specific measure, i.e., the model similarity between the surrogate and the target models. Existing work has not analyzed such impact but solely focuses on improving transferability. Specifically, interpretability is measured by Average Increase (AI) and Average Drop (AD) [9] based on GradCAM [64]. Detailed definitions of these interpretability measures are provided in Appendix G. For model similarity, we calculate the Kullback-Leibler (KL) divergence of output features following [26]. For
<figure><img src="image_7.png" /><figcaption>Fig. 5: Results for surrogate refinement attacks (1/2): Transferability vs. iteration curve.</figcaption></figure>

<table><caption>TABLE V: Results for surrogate refinement attacks (2/2): Impact of the surrogate property. DN121 is the target model.</caption><thead><tr><th>Attacks</th><th>Transfer↑</th><th>Acc↑</th><th>AI↑</th><th>AD↓</th><th>KL↓</th></tr></thead><tbody><tr><td>Original</td><td>79.6</td><td>100.0</td><td>37.4</td><td>8.4</td><td>14.8</td></tr><tr><td>SGM</td><td>90.3</td><td>100.0</td><td>37.4</td><td>8.4</td><td>14.8</td></tr><tr><td>LinBP</td><td>98.1</td><td>100.0</td><td>37.4</td><td>8.4</td><td>14.8</td></tr><tr><td>RFA</td><td>99.1</td><td>96.4</td><td>33.8</td><td>14.2</td><td>16.0</td></tr><tr><td>IAA</td><td>99.5</td><td>34.9</td><td>4.4</td><td>29.5</td><td>22.4</td></tr><tr><td>DSM</td><td>95.7</td><td>97.6</td><td>41.0</td><td>7.6</td><td>15.9</td></tr></tbody></table>

LinBP and SGM, we refine the surrogate model not only during backpropagation as in their original implementations but also during the forward pass. This makes sure that the refinement can also potentially influence the model properties besides transferability.

As can be seen from Table V, there is a clear negative correlation between transferability and other model properties. In particular, IAA ranks the highest in terms of transferability but the lowest in terms of all the other four measures of model properties. This is aligned with the previous finding in transfer learning that a pre-trained model that performs well in the target domain does not necessarily perform well in the source domain [62]. In addition, the refinement in LinBP and SGM has little impact on all model properties.<fnref n="4" /> This indicates that modifying only the activation function or the hyperparameters of the residual block does not influence the model properties. Specifically, the GradCAM calculations in AI and AD only involve the gradients with respect to the last layer and the ReLU function is applied by default. In addition, for KL, since both the surrogate model and the target model achieve an accuracy of 100%, the above modifications yield small changes in the logit values for the (true) prediction.

## E. Analysis of Generative Modeling Attacks

As discussed above, generative modeling attacks can be used to generate perturbations under various constraints by simply specifying the corresponding norm bound in the clipping operation. Fig. 6 Top shows the transferability of the five generative modeling attacks under various perturbation bounds. For the targeted attack TTP, we calculate its untargeted transferability over 5000 targeted adversarial images that are generated following the 10-Targets setting [49].

<figure><img src="image_8.png" /><figcaption>Fig. 6: Results for generative modeling attacks. Top: Transferability vs. perturbation budget curve. Bottom: Impact of the training perturbation bound, for GAP attack.</figcaption></figure>

As can be seen, the transferability generally increases as the perturbation constraint is relaxed. Consistent with our discussion about feature disruption attacks, feature-level losses (i.e., GAPF and BIA) outperform output-level losses (i.e., GAP and CDA). More specifically, GAPF outperforms BIA probably because GAPF leverages the original, multi-channel feature maps but BIA compresses the feature maps by cross-channel pooling. CDA outperforms GAP since the Relative CE loss in CDA is known to be better than the CE loss in GAP [50]. However, although TTP also adopts a feature-level loss, it falls far behind GAPF and BIA, especially for a small perturbation size. This might be because TTP, as a targeted attack, relies on embedding semantic features (of the target) in the perturbations, which are shown to be difficult when the perturbations are constrained to be small [49], [110].

In addition to the generation/testing stage, the clipping operation is also applied when training the generator. Existing work commonly adopts the perturbation bound $\epsilon_{train} = 10$ without exploring the impact of this training perturbation bound on the transferability. To fill this gap, we train the GAP generator with various $\epsilon_{train}$ and test these generators across different perturbation bounds $\epsilon_{test}$. Fig. 6 Bottom shows that in general, adjusting $\epsilon_{train}$ has a substantial impact on the transferability. For example, when the $\epsilon_{test}$ is set to 16, varying the $\epsilon_{train}$ from 4 to 32 yields a difference in transferability by about 25%. In addition, using a moderate $\epsilon_{train}$ (i.e. 8-16) consistently leads to the best results for all possible $\epsilon_{test}$. This suggests that it makes sense to set $\epsilon_{train} = 10$ as in existing work. However, this finding is somewhat unexpected since choosing a matched parameter (here, $\epsilon_{train} = \epsilon_{test}$) performs the best in normal machine learning.

Additional experimental results in Appendix B confirm that our main intra-category insights generalize to more advanced models and practical scenarios. For example, as shown in Fig. 12, for gradient stabilization attacks, “using more iterations may even decrease the performance” still holds ViT and PubDef. As shown in Fig. 13, the relative performance of attacks within a category remains largely unchanged after integrating another category of attack, which is practically

<footnote n="4">⁴There are very small differences after the second decimal place.</footnote>
<table><caption>TABLE VI: Attack transferability in terms of success rate (%). ResNet-50 is used as the surrogate. Additional results under $L_{\infty} = 8/255$ in Table XI of Appendix B show consistent patterns. Note that GCV is not directly comparable to the others because it adopts a different label set from the standard ImageNet-1K. A darker cell means a better attack/worse defense.</caption><thead><tr><th rowspan="2">Attacks</th><th colspan="4">Without Defenses</th><th colspan="4">Input Pre-processing</th><th colspan="4">Purification Network</th><th colspan="4">Adversarial Training</th><th colspan="2">Real-World Model</th></tr><tr><th>DN121</th><th>VGG19</th><th>IncV3</th><th>ViT</th><th>BDR</th><th>PD</th><th>R&P</th><th>HGD</th><th>NRP</th><th>DiffPure</th><th>AT<sub>∞</sub></th><th>FD<sub>∞</sub></th><th>AT<sub>2</sub></th><th>PubDef</th><th>Def</th><th>Def</th><th>Def</th></tr></thead><tbody><tr><td>Clean Acc</td><td>100.0</td><td>100.0</td><td>100.0</td><td>94.7</td><td>89.1</td><td>97.3</td><td>94.1</td><td>98.0</td><td>90.2</td><td>91.7</td><td>77.8</td><td>81.4</td><td>77.0</td><td>95.4</td><td>100</td><td>100</td><td>100</td></tr><tr><td>PGD</td><td>79.6</td><td>72.7</td><td>43.6</td><td>21.2</td><td>100.0</td><td>100.0</td><td>98.2</td><td>84.5</td><td>78.9</td><td>13.1</td><td>22.1</td><td>18.6</td><td>61.2</td><td>48.7</td><td>55.0</td><td>55.0</td><td>55.0</td></tr><tr><td>MI [15]</td><td>85.7</td><td>78.1</td><td>55.8</td><td>31.4</td><td>100.0</td><td>100.0</td><td>98.6</td><td>87.5</td><td>49.4</td><td>21.4</td><td>22.6</td><td>18.9</td><td>64.9</td><td>60.8</td><td>61.0</td><td>61.0</td><td>61.0</td></tr><tr><td>NI [40]</td><td>87.2</td><td>82.7</td><td>60.4</td><td>31.1</td><td>100.0</td><td>100.0</td><td>99.2</td><td>88.2</td><td>83.3</td><td>20.0</td><td>22.4</td><td>19.1</td><td>64.3</td><td>64.9</td><td>63.0</td><td>63.0</td><td>63.0</td></tr><tr><td>PI [80]</td><td>92.1</td><td>87.7</td><td>66.0</td><td>33.3</td><td>100.0</td><td>100.0</td><td>99.7</td><td>81.3</td><td><b>93.3</b></td><td>20.4</td><td>22.8</td><td>19.2</td><td>64.4</td><td>73.1</td><td>59.0</td><td>59.0</td><td>59.0</td></tr><tr><td>DI [94]</td><td>99.0</td><td>99.1</td><td>69.8</td><td>36.9</td><td>100.0</td><td>100.0</td><td>100.0</td><td>98.9</td><td>81.4</td><td>16.1</td><td>22.9</td><td>19.5</td><td>62.3</td><td>85.5</td><td>65.0</td><td>65.0</td><td>65.0</td></tr><tr><td>TI [16]</td><td>96.9</td><td>96.1</td><td>63.0</td><td>34.5</td><td>100.0</td><td>100.0</td><td>100.0</td><td>97.9</td><td>78.4</td><td>16.1</td><td>22.9</td><td>19.1</td><td>62.3</td><td>78.4</td><td>62.0</td><td>62.0</td><td>62.0</td></tr><tr><td>SI [40]</td><td>93.8</td><td>85.6</td><td>61.8</td><td>29.5</td><td>100.0</td><td>100.0</td><td>99.2</td><td>95.2</td><td>72.6</td><td>14.6</td><td>22.6</td><td>19.2</td><td>62.6</td><td>70.9</td><td>66.0</td><td>66.0</td><td>66.0</td></tr><tr><td>VT [78]</td><td>95.4</td><td>92.5</td><td>67.1</td><td>35.9</td><td>100.0</td><td>100.0</td><td>99.8</td><td>97.8</td><td>80.5</td><td>19.1</td><td>22.8</td><td>19.1</td><td>63.3</td><td>79.1</td><td>61.0</td><td>61.0</td><td>61.0</td></tr><tr><td>Admix [79]</td><td>86.7</td><td>83.5</td><td>53.4</td><td>26.5</td><td>100.0</td><td>95.4</td><td>95.0</td><td>89.9</td><td>80.0</td><td>14.1</td><td>22.7</td><td>19.0</td><td>62.1</td><td>63.7</td><td>67.0</td><td>67.0</td><td>67.0</td></tr><tr><td>TAP [113]</td><td>77.0</td><td>77.6</td><td>50.3</td><td>28.0</td><td>100.0</td><td>100.0</td><td>95.6</td><td>80.5</td><td>40.9</td><td>15.8</td><td>22.6</td><td>19.4</td><td>64.4</td><td>56.2</td><td>67.0</td><td>67.0</td><td>67.0</td></tr><tr><td>AA [28]</td><td>61.8</td><td>64.1</td><td>43.5</td><td>26.1</td><td>91.0</td><td>88.4</td><td>81.4</td><td>67.8</td><td>14.7</td><td>20.6</td><td>24.9</td><td>21.4</td><td>65.1</td><td>47.0</td><td>67.0</td><td>67.0</td><td>67.0</td></tr><tr><td>ILA [24]</td><td>94.5</td><td>92.6</td><td>72.3</td><td>38.3</td><td>100.0</td><td>100.0</td><td>99.1</td><td>94.8</td><td>83.5</td><td>18.2</td><td>22.6</td><td>19.1</td><td>63.9</td><td>86.0</td><td>62.0</td><td>62.0</td><td>62.0</td></tr><tr><td>FIA [83]</td><td>97.5</td><td>97.1</td><td>88.4</td><td>61.8</td><td>100.0</td><td>99.9</td><td>99.6</td><td>98.1</td><td>71.6</td><td>28.5</td><td>25.1</td><td>21.3</td><td>68.2</td><td>92.6</td><td>74.0</td><td>74.0</td><td>74.0</td></tr><tr><td>NAA [106]</td><td>96.7</td><td>95.3</td><td>85.0</td><td>55.0</td><td>99.9</td><td>99.9</td><td>99.1</td><td>97.2</td><td>76.8</td><td>29.2</td><td>24.8</td><td>21.3</td><td>66.2</td><td>89.2</td><td>73.0</td><td>73.0</td><td>73.0</td></tr><tr><td>SGM [89]</td><td>79.1</td><td>83.0</td><td>55.4</td><td>31.8</td><td>90.4</td><td>100.0</td><td>97.6</td><td>94.5</td><td>83.1</td><td>14.9</td><td>22.7</td><td>18.9</td><td>62.2</td><td>65.8</td><td>60.0</td><td>60.0</td><td>60.0</td></tr><tr><td>LinBP [21]</td><td>98.1</td><td>97.4</td><td>82.8</td><td>40.0</td><td>96.6</td><td>100.0</td><td>99.2</td><td>98.4</td><td>66.8</td><td>17.7</td><td>23.1</td><td>19.4</td><td>63.8</td><td>89.4</td><td>68.0</td><td>68.0</td><td>68.0</td></tr><tr><td>RFA<sub>2</sub> [71]</td><td>99.1</td><td>96.7</td><td>89.2</td><td>63.1</td><td>98.9</td><td>99.5</td><td>97.9</td><td>99.4</td><td>34.8</td><td>43.8</td><td>24.8</td><td>21.2</td><td>67.6</td><td>93.9</td><td>66.0</td><td>66.0</td><td>66.0</td></tr><tr><td>RFA<sub>∞</sub> [71]</td><td>77.7</td><td>68.7</td><td>81.1</td><td>73.3</td><td>81.7</td><td>80.4</td><td>77.5</td><td>72.8</td><td>21.0</td><td><b>69.4</b></td><td><b>62.0</b></td><td><b>53.4</b></td><td><b>87.3</b></td><td>62.7</td><td>77.0</td><td>77.0</td><td>77.0</td></tr><tr><td>IAA [114]</td><td>99.5</td><td>99.2</td><td>83.9</td><td>50.2</td><td>98.9</td><td>100.0</td><td>99.9</td><td><b>99.9</b></td><td>65.4</td><td>20.6</td><td>23.0</td><td>19.5</td><td>65.5</td><td>95.7</td><td>79.0</td><td>79.0</td><td>79.0</td></tr><tr><td>DSM [96]</td><td>95.7</td><td>90.8</td><td>57.3</td><td>27.3</td><td>97.4</td><td>99.1</td><td>92.6</td><td>98.1</td><td>38.2</td><td>14.9</td><td>22.7</td><td>19.1</td><td>62.3</td><td>74.6</td><td>64.0</td><td>64.0</td><td>64.0</td></tr><tr><td>GAP [57]</td><td>82.1</td><td>87.4</td><td>65.1</td><td>34.8</td><td>85.1</td><td>87.0</td><td>83.5</td><td>93.9</td><td>2.0</td><td>16.8</td><td>21.7</td><td>18.2</td><td>64.2</td><td>66.3</td><td>80.0</td><td>80.0</td><td>80.0</td></tr><tr><td>CDA [50]</td><td>99.2</td><td>99.2</td><td>97.8</td><td><b>82.1</b></td><td>97.6</td><td>100.0</td><td>98.3</td><td>99.9</td><td>3.9</td><td>62.0</td><td>26.0</td><td>22.6</td><td>73.2</td><td>98.1</td><td>86.0</td><td>86.0</td><td>86.0</td></tr><tr><td>GAPF [30]</td><td><b>99.6</b></td><td><b>99.6</b></td><td><b>99.2</b></td><td>72.5</td><td>99.5</td><td>99.6</td><td>99.8</td><td>99.8</td><td>14.3</td><td>28.9</td><td>23.4</td><td>20.2</td><td>67.7</td><td><b>99.3</b></td><td><b>94.0</b></td><td><b>94.0</b></td><td><b>94.0</b></td></tr><tr><td>BIA [107]</td><td>97.1</td><td>97.8</td><td>88.2</td><td>50.3</td><td>96.9</td><td>95.4</td><td>96.6</td><td>98.7</td><td>2.6</td><td>20.0</td><td>25.3</td><td>22.2</td><td>66.7</td><td>94.9</td><td>88.0</td><td>88.0</td><td>88.0</td></tr><tr><td>TTP [49]</td><td>97.1</td><td>97.3</td><td>89.0</td><td>64.3</td><td>97.6</td><td>98.2</td><td>97.9</td><td>95.4</td><td>36.2</td><td>37.4</td><td>25.3</td><td>21.3</td><td>68.7</td><td>90.0</td><td>67.0</td><td>67.0</td><td>67.0</td></tr></tbody></table>

implemented for further attack improvement.

## V. COMPREHENSIVE EVALUATION

Through the above systematic intra-category analyses, we can determine the optimal choices of the key attack hyper-parameters to achieve a fair large-scale evaluation of attack transferability and stealthiness. Specifically, the key attack hyperparameters refer to the number of iterations for all except generative model attacks, the number of input copies for input augmentation attacks, the choice of layers for feature disruption attacks, and the training perturbation bound $\epsilon_{train}$ for generative model attacks.

In general, different attacks under the same category have the same optimal hyperparameters. Specifically, for gradient stabilization attacks, we set the iteration number to 10 to avoid the “overfitting” phenomenon, and for the other three categories of iterative attacks, we set the iteration number to 50 to ensure attack convergence. For input augmentation attacks, we set the number of copies to 5 for a good trade-off between attack strength and efficiency. For feature disruption attacks, we use the mid layer (“conv3_x”), which always yields the best performance. For generative modeling attacks, we set $\epsilon_{train} = 10$, which is commonly adopted in existing work and validated to be optimal by our intra-category analyses. For other attack hyperparameters not discussed in our intra-category analyses and defense hyperparameters, we follow the common settings as summarized in Table XIII of Appendix C.

We make sure our findings are stable by repeating our experiments 5 times on 1000 images, each of which is selected from each of the 1000 ImageNet classes. The results confirm that 16 attacks always yield identical numbers because they are deterministic by design. For the other 7 attacks, i.e., the 5 input augmentation attacks, AA, and FIA, the standard deviations are always very small, i.e., standard deviations within [0.001,

0.013] for attack success rates within [0.464, 0.993]. Note that in contrast, there is no statistical analysis in any original work of the 23 attacks.

### A. New Insights into Transferability

Table VI summarizes the transferability results in both the standard and defense settings. These results either lead to new findings or complement existing findings that were only validated in a limited set of attacks and defenses.

**General conclusion:** The transferability of an attack against a defense is highly contextual. An attack/defense that is the best against one specific defense/attack may perform very badly against another. For example, the transferability of CDA to 9 different defenses varies largely, from 3.9% to 100.0%. Similarly, a defense that achieves the highest robustness to one specific attack may be very vulnerable to another attack. For example, the robustness of NRP to 24 different attacks varies largely, from 2.0% to 93.3%.

In particular, we reveal that this contextual performance is related to the overfitting of a defense to the type of perturbations used in its optimization. Specifically, the purification network-based and adversarial training-based defenses rely on training defensive models with additional adversarial data. In this way, the resulting defensive model only ensures high robustness against the perturbations that have been used for generating the adversarial data. We take NRP vs. DiffPure as an example to explain this finding. Specifically, NRP is trained with semantic perturbations [48] and so it is very effective in mitigating low-frequency, semantic perturbations, e.g., those generated by generative modeling attacks (see perturbation visualizations in Fig. 8). However, it performs much worse against high-frequency perturbations, e.g., those generated by gradient stabilization and input augmentation attacks. In
<figure><img src="image_9.png" /><figcaption>Fig. 7: Purifying CDA adversarial images. NRP works well but DiffPure does not due to the defense overfitting.</figcaption></figure>

contrast, DiffPure relies on the diffusion process with (high-frequency) Gaussian noise. Therefore, it is not capable of purifying low-frequency, semantic perturbations. Fig. 7 shows that NRP can purify the CDA adversarial images but DiffPure cannot do it well.

The substantial vulnerability of DiffPure to black-box (transferable) attacks is indeed very surprising since DiffPure was initially claimed to provide state-of-the-art robustness to white-box (adaptive) attacks, even surpassing adversarial training [53]. Our finding suggests that *DiffPure provides a false sense of security and requires a rightful evaluation following recommendations from* [3], [7], [76]. In addition, adversarial training is not effective against RFA since RFA has learned how to distort robust features by adopting an adversarially trained surrogate model. Note that the norm bound used in RFA is 2× larger than that in $AT_∞$ and $FD_∞$.

**Specific observations.** Adversarial training (AT) is effective but input pre-processing is not. Both the two $L_∞$ AT defenses, $AT_∞$ and $FD_∞$, are consistently effective against all attacks except $RFA_∞$, which uses an AT surrogate model for generating robust perturbations. The performance of $AT_2$ is also not sensitive to the attack method but sub-optimal due to the use of a different, $L_2$ norm. In contrast, input pre-processing defenses are generally not useful, although they are known to be effective against (non-adaptive) white-box attacks where the perturbations are much smaller [92], [95]. However, adversarial training is known to be computationally expensive [65], [87].

*Changing the model architecture is more useful than applying certain defenses* (although may not be practically desired). Previous work has noticed that transferring to Inception-v3 is especially hard [26], [27], [110]. This might be because the Inception architecture contains relatively complex components, e.g., multiple-size convolution and two auxiliary classifiers. Our results extend this finding from $≤ 3$ to 23 attacks. Furthermore, transferring from CNNs to ViT is also known to be hard [82], [85], [105]. Based on our results, we can observe that adopting ViT as the target model ranks about 4th on average among the 9 defenses.

*New model design largely boosts transferability.* The generative modeling attacks and some of the surrogate refinement attacks are based on designing new surrogate or generative

models, while the other attacks adopt off-the-shelf surrogate models. In almost all cases, the former attacks outperform the latter. However, they inevitably consume additional computational and data resources.

**Real-world attacks.** Beyond the single CNN or ViT models, we further evaluate attacks against two real-world models: a specifically designed defensive model against transfer-based attacks, PubDef [69], and a production model, Google Cloud Vision (GCV) [1], of which the architecture and training data are completely unknown to the attacker. Specifically, PubDef trains a defense model with adversarial examples generated on a heuristically selected set of 24 public pre-trained models, including CNNs, ViTs, Swin, BeiT, ConvMixer, zero-shot CLIP, etc. The results on PubDef follow a similar pattern with those on other models.<fnref n="5" /> For example, generative attacks are still the overall best, and the earliest method, DI, is still the best input augmentation attack.

For GCV, since it adopts a different label set from the standard ImageNet-1K, we treat an image to be successfully attacked when it yields a different top-1 prediction from the original image. In this case, the clean accuracy of GCV is 100%. Considering practical budget constraints, we test with 100 randomly selected adversarial images per attack. The results on GCV still confirm the best performance of generative attacks. However, compared to other models (that are trained on ImageNet), GCV yields relatively stable robustness against other attacks.

## B. New Insights into Stealthiness

Here we compare different attacks on their imperceptibility and also other finer-grained stealthiness characteristics. In particular, we conduct a user study to validate our findings. **General conclusion: Stealthiness may be at odds with transferability.** Fig. 8 visualizes the adversarial images with their corresponding perturbations for different attacks. Solely looking at these examples, we notice that different attacks yield dramatically different perturbations, although they are constrained by the same $L_∞$ norm bound with $ϵ = 16$. In the following, we will demonstrate in more detail the distinct stealthiness characteristics of different attacks.

**Imperceptibility.** Table VII reports the imperceptibility of different attacks regarding five diverse perceptual similarity metrics. First of all, almost all the 23 transfer attacks are more perceptible than the PGD baseline although their perturbations are constrained in the same way. This means that the current transfer attacks indeed sacrifice imperceptibility for improving transferability. One exception is the SSIM performance of RFA, which is even better than the PGD baseline. This might be because RFA mainly introduces smooth perturbations but does not break image structures too much, as confirmed by the visualizations in Fig. 8. We further calculate the luminance, contrast, and structure components of SSIM for those adversarial images, and the results confirm that SSIM yields a structure score of 0.916 (and contrast score of 0.893), which is 8.2% (13.6%) higher than the second best.

<footnote n="5">⁵For PubDef, the absolute numbers are not directly comparable to those in their original paper because we attack with $L_∞ = 16$ rather than 4.</footnote>
<figure><img src="image_10.png" /><figcaption>Fig. 8: Visualizations of adversarial examples generated by the 24 different attacks with their corresponding perturbations. Additional examples can be found in Appendix H.</figcaption></figure>

<table><caption>TABLE VII: Imperceptibility regarding five metrics. Additional results under $L_{\infty} = 8/255$ with consistent patterns are in Table XII of Appendix B.</caption><thead><tr><th>Attacks</th><th>PSNR↑</th><th>SSIM↑</th><th>ΔE↓</th><th>LPIPS↓</th><th>FID↓</th></tr></thead><tbody><tr><td>PGD</td><td><strong>28.112</strong></td><td><strong>0.714</strong></td><td><strong>0.662</strong></td><td><strong>0.175</strong></td><td><strong>33.823</strong></td></tr><tr><td>MI [15]</td><td>24.853</td><td>0.576</td><td>0.901</td><td>0.290</td><td>47.570</td></tr><tr><td>NI [40]</td><td>26.030</td><td>0.619</td><td>0.742</td><td>0.300</td><td>47.637</td></tr><tr><td>PI [80]</td><td>26.015</td><td>0.619</td><td>0.741</td><td>0.303</td><td>50.051</td></tr><tr><td>DI [94]</td><td>27.987</td><td>0.713</td><td>0.666</td><td>0.176</td><td>66.075</td></tr><tr><td>TI [16]</td><td>27.903</td><td>0.712</td><td>0.671</td><td>0.178</td><td>54.687</td></tr><tr><td>SI [40]</td><td>27.712</td><td>0.704</td><td>0.668</td><td>0.187</td><td>51.098</td></tr><tr><td>VT [78]</td><td>27.264</td><td>0.693</td><td>0.710</td><td>0.200</td><td>51.858</td></tr><tr><td>Admix [79]</td><td>27.817</td><td>0.703</td><td>0.670</td><td>0.187</td><td>43.052</td></tr><tr><td>TAP [113]</td><td>25.710</td><td>0.560</td><td>0.743</td><td>0.314</td><td>84.705</td></tr><tr><td>AA [28]</td><td>26.613</td><td>0.665</td><td>0.793</td><td>0.246</td><td>71.301</td></tr><tr><td>ILA [24]</td><td>26.222</td><td>0.625</td><td>0.665</td><td>0.223</td><td>119.047</td></tr><tr><td>FIA [83]</td><td>26.533</td><td>0.647</td><td>0.755</td><td>0.225</td><td>139.522</td></tr><tr><td>NAA [106]</td><td>26.634</td><td>0.653</td><td>0.757</td><td>0.207</td><td>109.383</td></tr><tr><td>SGM [89]</td><td>27.586</td><td>0.704</td><td>0.663</td><td>0.180</td><td>40.637</td></tr><tr><td>LinBP [21]</td><td>26.884</td><td>0.680</td><td>0.708</td><td>0.196</td><td>87.930</td></tr><tr><td>RFA<sub>2</sub> [71]</td><td>27.113</td><td>0.720</td><td>0.701</td><td>0.185</td><td>70.065</td></tr><tr><td>RFA<sub>∞</sub> [71]</td><td>24.542</td><td><strong>0.751</strong></td><td>0.906</td><td>0.226</td><td>60.349</td></tr><tr><td>IAA [114]</td><td>26.480</td><td>0.671</td><td>0.773</td><td>0.205</td><td>145.231</td></tr><tr><td>DSM [96]</td><td>27.947</td><td>0.712</td><td>0.667</td><td>0.177</td><td>45.909</td></tr><tr><td>GAP [57]</td><td>25.184</td><td>0.611</td><td>0.847</td><td>0.285</td><td>145.964</td></tr><tr><td>CDA [50]</td><td>24.200</td><td>0.605</td><td>1.022</td><td>0.337</td><td>1098.624</td></tr><tr><td>GAPF [30]</td><td>25.423</td><td>0.641</td><td>0.846</td><td>0.250</td><td>215.241</td></tr><tr><td>BIA [107]</td><td>24.590</td><td>0.537</td><td>0.883</td><td>0.400</td><td>230.470</td></tr><tr><td>TTP [49]</td><td>25.813</td><td>0.678</td><td>0.764</td><td>0.270</td><td>209.802</td></tr></tbody></table>

Moreover, among the 23 transfer attacks, imperceptibility varies greatly and negatively correlates with the transferability reported in Table VI. In particular, certain generative modeling attacks are the most transferable but also the most perceptible. In addition, generative modeling attacks yield especially large FID scores. This can be explained by the dramatic distribution shift between their adversarial and original images due to their image-agnostic perturbations [30], [49]. More generally, the poor imperceptibility performance suggests that using only the $L_{\infty}$ constraint is not sufficient. This conclusion is consistent

<figure><img src="image_11.png" /><figcaption>Fig. 9: Attack traceback with image features.</figcaption></figure>

with that for white-box attacks [67], [109] and calls for a more advanced design of attack constraints [33].

The above results demonstrate that different attacks indeed yield dramatically different imperceptibility. In the following, we further compare their stealthiness at finer-grained levels. To this end, we conduct “attack traceback”, which aims to trace back the specific attack based on certain information. In general, an attack is less stealthy when it is easier to trace back. The “attack traceback” is basically a classification problem, and in our case, with 24 classes. Specifically, we randomly select 500 original images and then perturb them using the 24 different attacks, resulting in 12000 adversarial images in total. For each attack, 400 adversarial images are used for training and the rest 100 for testing. we conduct the attack classification with two types of features: (input) image features and (output) misclassification features.

**Attack traceback with image features.** We use ResNet-18 as the backbone for the 24-class classifier with image features.
<figure><img src="image_12.png" /><figcaption>Fig. 10: Distributions of five most frequently predicted classes with their names.</figcaption></figure>

As can be seen from Fig. 9, different attacks can be well differentiated solely based on image features, with an overall accuracy of 60.43%, much higher than a random guess, i.e., 1/24. Moreover, the category-wise accuracy is even higher, i.e., 75.61%. This suggests that our attack systemization is reliable since it captures the signature of each specific category.

More specifically, we note low accuracy for PI and input augmentation attacks but perfect accuracy for generative modeling attacks. For PI, the low accuracy can be explained by its technical similarity to NI/MI and their resulting similar perturbations, as shown in Fig. 8. For generative modeling attacks, the perfect accuracy might be explained by their special perturbations. For input augmentation attacks, the perturbations are somewhat a mixture of low- and high-frequency components. So they are harder to distinguish from multiple other attacks. In general, it should be noted that perturbation patterns are hard to characterize, compared to objects that are semantically meaningful to humans.

**Attack traceback with misclassification features.** We train an SVM classifier with an RBF kernel and adopt the top-N output class IDs as the misclassification features. Table VIII shows the classification accuracy as a function of N. As can be seen, too small N is insufficient to represent the attack and too large N may introduce noisy information. In particular, a moderate choice of N = 5 yields the optimal accuracy, i.e., 18.5%, which is also much higher than a random guess, i.e., 1/24. However, it is still far behind the image features, which have much higher dimensions, i.e., $224 \times 224 = 50176$.

We further examine the class distributions of the output predictions. To this end, for each attack, we calculate the frequencies of different mispredicted classes over 5000 adversarial images. As can be seen from Fig. 10, different attack categories lead to different class distributions. Specifically, gradient stabilization and input augmentation attacks lead to uniform class distributions. In contrast, the other attacks yield more concentrated distributions. Moreover, the most frequent

<table><caption>TABLE VIII: Attack traceback with misclassification features.</caption><thead><tr><th>N</th><th>1</th><th>3</th><th>5</th><th>10</th><th>100</th><th>1000</th></tr></thead><tbody><tr><td>Acc (%)</td><td>14.6</td><td>17.9</td><td>18.5</td><td>18.1</td><td>17.4</td><td>17.0</td></tr></tbody></table>

class usually corresponds to semantics that contain rich textures, e.g., “brain coral” and “jigsaw puzzle”. This somewhat confirms the high-frequency nature of perturbations. Specifically, a few classes dominate the prediction of generative modeling attacks. For example, almost all CDA adversarial images are predicted into “harp”. These findings suggest that generative modeling attacks are very easy to trace back, even with only the Top-1 prediction.

**User study.** We conducted user studies to validate our findings on stealthiness with 3000 pairwise attack comparisons judged by 10 graduate students with sufficient knowledge about adversarial examples. We pay $5/hour per participant, which is higher than the minimum payout (about $3/hour) in our local region. We obtained informed consent from all participants for reporting their scores in this research, and the Human Research Ethics Committee of the first author’s affiliation determined that the study was exempt from further human subjects review. To our knowledge, well-known papers conducting user studies on the quality of adversarial images also do not involve ethics review [4], [91], [111].

Specifically, for each of the 10 original images shown in Fig. 16 of Appendix H, we generate $C(24,2) + 24 = 300$ pairs of adversarial images, involving all 24 attacks. These 3000 pairs of adversarial images are given to 10 users for judgment. For each pair, they are asked which of the two images are of higher visual quality and should choose from “left”, “right”, and “similar”. Statistics in Fig. 11 further support our findings on stealthiness. Specifically, Fig. 11 Left confirms that generative attacks are the least stealthy, and Fig. 11 Right confirms that the similarity between intra-category attacks is
<figure><img src="image_13.png" /><figcaption>Fig. 11: User study results that further support our findings on stealthiness. **Left:** Visual quality comparison between any two attacks. Each cell at (Row, Column) means the frequencies of the attack in that Row with higher visual quality than the attack in that Column. Note that all diagonal values are supposed to be zeros since the same attack yields the same quality. **Right:** Similarity between any two attacks.</figcaption></figure>

generally higher than that of inter-category attacks.

## VI. DISCUSSION AND OUTLOOK

**Forward compatibility of our evaluation.** Our attack system-ization corresponds to the major components in the common attack pipeline, as shown in Fig. 1. Therefore, we believe that future attacks would also fall into those categories unless there appears a completely new design of adversarial attacks. Our evaluation methodology can also be directly extended to other threat models. In the current stage, we have followed the most common threat model as described in Section III-A. This threat model covers basic settings regarding models, datasets, constraints, attack goals, etc. Therefore, specific practical considerations would not affect our scientific conclusions. In the future, exploring more realistic threat models would make more sense with the progress of transfer attacks. For example, targeted attacks pose more realistic threats than non-targeted attacks but most of the current transfer attacks cannot achieve substantial targeted success [49], [110].

**Further explanations of our findings.** We have drawn several new findings about transferable adversarial images through our systematic evaluation. However, some of them still need further exploration to be fully understood. For example, we find that for generative modeling attacks, using a moderate $\epsilon_{train}$ leads to consistently optimal performance for all $\epsilon_{test}$, although in normal machine learning, choosing a matched hyperparameter (here $\epsilon_{test} = \epsilon_{train}$) performs the best. This uncommon phenomenon may be specifically related to generative perturbations and requires further study in the future. In addition, different attacks have dramatically different properties regarding

perturbation and misclassification patterns. For future work, it is important to figure out how these properties are related to the specific attack design. In particular, the reason why generative modeling attacks tend to produce image-agnostic perturbations falling into a dominant class should be explored.

## VII. CONCLUSION

In this paper, we have provided the first systematic evaluation of recent transferable attacks, involving 23 representative transferable attacks (in five categories) against 11 representative defenses (in four categories) on ImageNet. In particular, we have established new evaluation guidelines that address the two common problems in existing work. Our evaluation focuses on both attack transferability and stealthiness, and our experimental results lead to new insights that complement or even challenge existing knowledge. In particular, for stealthiness, we move beyond diverse imperceptibility measures and explore the potential of attack traceback based on image or misclassification features. User study results on image quality further support our findings.

Our work provides a thorough picture of the current progress of transfer attacks and demonstrate that existing problematic evaluations have indeed hindered the assessment of the actual progress in transferable adversarial images. We hope that our analyses can guide future research towards good practices in transfer attacks.

## ACKNOWLEDGMENTS

This research is supported by the National Key Research and Development Program of China (2023YFE0209800), the
National Natural Science Foundation of China (62406240, U244120060). Thanks to the New Cornerstone Science Foundation and the Xplorer Prize.

## REFERENCES

[1] Google Cloud Vision API. https://cloud.google.com/vision.

[2] Rima Alaifari, Giovanni S Alberti, and Tandri Gauksson. ADef: an iterative algorithm to construct adversarial deformations. In *ICLR*, 2019.

[3] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In *ICML*, 2018.

[4] Anand Bhattad, Min Jin Chong, Kaizhao Liang, Bo Li, and David A Forsyth. Unrestricted adversarial examples via semantic manipulation. In *ICLR*, 2020.

[5] Battista Biggio and Fabio Roli. Wild patterns: Ten years after the rise of adversarial machine learning. *PR*, 84:317–331, 2018.

[6] Junyoung Byun, Seungju Cho, Myung-Joon Kwon, Hee-Seon Kim, and Changick Kim. Improving the transferability of targeted adversarial examples through object-based diverse input. In *CVPR*, 2022.

[7] Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. In *arXiv*, 2019.

[8] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In *S&P*, 2017.

[9] A. Chattopadhyay, A. Sarkar, P. Howlader, and V. N. Balasubramanian. Grad-CAM++: Generalized gradient-based visual explanations for deep convolutional networks. In *WACV*, 2018.

[10] Sizhe Chen, Qinghua Tao, Zhixing Ye, and Xiaolin Huang. Measuring $L_\infty$ attacks by the $L_2$ norm. In *ICASSP*, 2023.

[11] Steven Chen, Nicholas Carlini, and David Wagner. Stateful detection of black-box adversarial attacks. In *SPAI*, 2020.

[12] Francesco Croce and Matthias Hein. Sparse and imperceivable adversarial attacks. In *ICCV*, 2019.

[13] Edoardo Debenedetti, Nicholas Carlini, and Florian Tramèr. Evading black-box classifiers without breaking eggs. In *ICML-AdvML*, 2023.

[14] Zhun Deng, Linjun Zhang, Kailas Vodrahalli, Kenji Kawaguchi, and James Y Zou. Adversarial training helps transfer learning via better representations. In *NeurIPS*, 2021.

[15] Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In *CVPR*, 2018.

[16] Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Evading defenses to transferable adversarial examples by translation-invariant attacks. In *CVPR*, 2019.

[17] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weisenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In *ICLR*, 2021.

[18] Aditya Ganeshan, Vivek BS, and R Venkatesh Babu. FDA: Feature disruptive attack. In *ICCV*, 2019.

[19] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In *NeurIPS*, 2014.

[20] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In *ICLR*, 2015.

[21] Yiwen Guo, Qizhang Li, and Hao Chen. Backpropagating linearly improves transferability of adversarial examples. In *NeurIPS*, 2020.

[22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In *CVPR*, 2016.

[23] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In *CVPR*, 2017.

[24] Qian Huang, Isay Katsman, Horace He, Zeqi Gu, Serge Belongie, and Ser-Nam Lim. Enhancing adversarial example transferability with an intermediate level attack. In *ICCV*, 2019.

[25] Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In *NeurIPS*, 2019.

[26] Nathan Inkawhich, Kevin Liang, Lawrence Carin, and Yiran Chen. Transferable perturbations of deep feature distributions. In *ICLR*, 2020.

[27] Nathan Inkawhich, Kevin J Liang, Binghui Wang, Matthew Inkawhich, Lawrence Carin, and Yiran Chen. Perturbing across the feature hierarchy to improve standard and strict blackbox attack transferability. In *NeurIPS*, 2020.

[28] Nathan Inkawhich, Wei Wen, Hai Helen Li, and Yiran Chen. Feature space perturbations yield more transferable adversarial examples. In *CVPR*, 2019.

[29] Can Kanbak, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. Geometric robustness of deep networks: analysis and improvement. In *CVPR*, 2018.

[30] Krishna kanth Nakka and Mathieu Salzmann. Learning transferable adversarial perturbations. In *NeurIPS*, 2021.

[31] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In *ICML*, 2019.

[32] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In *ICLR*, 2017.

[33] Cassidy Laidlaw, Sahil Singla, and Soheil Feizi. Perceptual adversarial robustness: Defense against unseen threat models. In *ICLR*, 2021.

[34] Maosen Li, Cheng Deng, Tengjiao Li, Junchi Yan, Xinbo Gao, and Heng Huang. Towards transferable targeted attack. In *CVPR*, 2020.

[35] Qizhang Li, Yiwen Guo, and Hao Chen. Yet another intermediate-level attack. In *ECCV*, 2020.

[36] Qizhang Li, Yiwen Guo, Wangmeng Zuo, and Hao Chen. Towards evaluating transfer-based attacks systematically, practically, and fairly. In *NeurIPS*, 2023.

[37] Yingwei Li, Song Bai, Cihang Xie, Zhenyu Liao, Xiaohui Shen, and Alan L Yuille. Regional homogeneity: Towards learning transferable universal adversarial perturbations against defenses. In *ECCV*, 2020.

[38] Kaizhao Liang, Jacky Y Zhang, Boxin Wang, Zhuolin Yang, Sanmi Koyejo, and Bo Li. Uncovering the connections between adversarial transferability and knowledge transferability. In *ICML*, 2021.

[39] Fangzhou Liao, Ming Liang, Yinpeng Dong, Tianyu Pang, Xiaolin Hu, and Jun Zhu. Defense against adversarial attacks using high-level representation guided denoiser. In *CVPR*, 2018.

[40] Jiadong Lin, Chuanbiao Song, Kun He, Liwei Wang, and John E Hopcroft. Nesterov accelerated gradient and scale invariance for adversarial attacks. In *ICLR*, 2020.

[41] Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attacks. In *ICLR*, 2017.

[42] Zhuoran Liu, Zhengyu Zhao, and Martha Larson. Who's afraid of adversarial queries? the impact of image modifications on content-based image retrieval. In *ICMR*, 2019.

[43] Yuyang Long, Qilong Zhang, Boheng Zeng, Lianli Gao, Xianglong Liu, Jian Zhang, and Jingkuan Song. Frequency domain model augmentation for adversarial attack. In *ECCV*, 2022.

[44] Bo Luo, Yannan Liu, Lingxiao Wei, and Qiang Xu. Towards imperceptible and robust adversarial example attacks against neural networks. In *AAAI*, 2018.

[45] Ming Ronnier Luo, Guihua Cui, and B. Rigg. The development of the CIE 2000 colour-difference formula: CIEDE2000. *CRA*, 26:340–350, 2001.
[46] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In *ICLR*, 2018.

[47] Yuhao Mao, Chong Fu, Saizhuo Wang, Shouling Ji, Xuhong Zhang, Zhenguang Liu, Jun Zhou, Alex X Liu, Raheem Beyah, and Ting Wang. Transfer attacks revisited: A large-scale empirical study in real computer vision settings. In *S&P*, 2022.

[48] Muzammal Naseer, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Fatih Porikli. A self-supervised approach for adversarial robustness. In *CVPR*, 2020.

[49] Muzammal Naseer, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Fatih Porikli. On generating transferable targeted perturbations. In *ICCV*, 2021.

[50] Muzammal Naseer, Salman H Khan, Harris Khan, Fahad Shahbaz Khan, and Fatih Porikli. Cross-domain transferability of adversarial perturbations. In *NeurIPS*, 2019.

[51] Muzammal Naseer, Salman H Khan, Shafin Rahman, and Fatih Porikli. Task-generalizable adversarial attack based on perceptual metric. In *arXiv*, 2018.

[52] Yurii Evgen'evich Nesterov. A method of solving a convex programming problem with convergence rate $o(1/k^2)$. In *Doklady Akademii Nauk*, 1983.

[53] Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. Diffusion models for adversarial purification. In *ICML*, 2022.

[54] Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In *AsiaCCS*, 2017.

[55] Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In *EuroS&P*, 2016.

[56] Boris T Polyak. Some methods of speeding up the convergence of iteration methods. *Ussr computational mathematics and mathematical physics*, 1964.

[57] Omid Poursaeed, Isay Katsman, Bicheng Gao, and Serge Belongie. Generative adversarial perturbations. In *CVPR*, 2018.

[58] Aaditya Prakash, Nick Moran, Solomon Garber, Antonella DiLillo, and James Storer. Deflecting adversarial attacks with pixel deflection. In *CVPR*, 2018.

[59] Yuchen Ren, Zhengyu Zhao, Chenhao Lin, Bo Yang, Lu Zhou, Zhe Liu, and Chao Shen. Improving adversarial transferability on vision transformers via forward propagation refinement. In *CVPR*, 2025.

[60] Yuchen Ren, Zhengyu Zhao, Chenhao Lin, Bo Yang, Lu Zhou, Zhe Liu, and Chao Shen. Improving integrated gradient-based transferable adversarial examples by refining the integration path. In *AAAI*, 2025.

[61] Andras Rozsa, Ethan M Rudd, and Terrance E Boult. Adversarial diversity and hard positive generation. In *CVPRW*, 2016.

[62] Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversarially robust imagenet models transfer better? In *NeurIPS*, 2020.

[63] Shibani Santurkar, Dimitris Tsipras, Brandon Tran, Andrew Ilyas, Logan Engstrom, and Aleksander Madry. Computer vision with a single (robust) classifier. In *NeurIPS*, 2019.

[64] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In *ICCV*, 2017.

[65] Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! In *NeurIPS*, 2019.

[66] Ali Shahin Shamsabadi, Ricardo Sanchez-Matilla, and Andrea Cavallaro. ColorFool: Semantic adversarial colorization. In *CVPR*, 2020.

[67] Mahmood Sharif, Lujo Bauer, and Michael K. Reiter. On the suitability of $L_p$-norms for creating and preventing adversarial examples. In *CVPRW*, 2018.

[68] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In *ICLR*, 2015.

[69] Chawin Sitawarin, Jaewon Chang, David Huang, Wesson Altoyan, and David Wagner. Defending against transfer attacks from public models. In *ICLR*, 2024.

[70] Gowthami Somepalli, Liam Fowl, Arpit Bansal, Ping Yeh-Chiang, Yehuda Dar, Richard Baraniuk, Micah Goldblum, and Tom Goldstein. Can neural nets learn the same model twice? investigating reproducibility and double descent from the decision boundary perspective. In *CVPR*, 2022.

[71] Jacob M Springer, Melanie Mitchell, and Garrett T Kenyon. A little robustness goes a long way: Leveraging robust features for targeted transfer attacks. In *NeurIPS*, 2021.

[72] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In *ICML*, 2017.

[73] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In *CVPR*, 2016.

[74] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In *ICLR*, 2014.

[75] Florian Tramèr, Jens Behrmann, Nicholas Carlini, Nicolas Papernot, and Jörn-Henrik Jacobsen. Fundamental tradeoffs between invariance and sensitivity to adversarial perturbations. In *ICML*, 2020.

[76] Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. In *NeurIPS*, 2020.

[77] Francisco Utrera, Evan Kravitz, N Benjamin Erichson, Rajiv Khanna, and Michael W Mahoney. Adversarially-trained deep nets transfer better: Illustration on image classification. In *ICLR*, 2021.

[78] Xiaosen Wang and Kun He. Enhancing the transferability of adversarial attacks through variance tuning. In *CVPR*, 2021.

[79] Xiaosen Wang, Xuanran He, Jingdong Wang, and Kun He. Admix: Enhancing the transferability of adversarial attacks. In *ICCV*, 2021.

[80] Xiaosen Wang, Jiadong Lin, Han Hu, Jingdong Wang, and Kun He. Boosting adversarial transferability through enhanced momentum. In *BMVC*, 2021.

[81] Xiaosen Wang, Zeliang Zhang, and Jianping Zhang. Structure invariant transformation for better adversarial transferability. In *ICCV*, 2023.

[82] Yuxuan Wang, Jakai Wang, Zixin Yin, Ruihao Gong, Jingyi Wang, Aishan Liu, and Xianglong Liu. Generating transferable adversarial examples against vision transformers. In *ACMMM*, 2022.

[83] Zhibo Wang, Hengchang Guo, Zhifei Zhang, Wenxin Liu, Zhan Qin, and Kui Ren. Feature importance-aware transferable adversarial attacks. In *ICCV*, 2021.

[84] Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: from error visibility to structural similarity. *IEEE Transactions on Image Processing*, 13:600–612, 2004.

[85] Zhipeng Wei, Jingjing Chen, Micah Goldblum, Zuxuan Wu, Tom Goldstein, and Yu-Gang Jiang. Towards transferable adversarial attacks on vision transformers. In *AAAI*, 2022.

[86] Zhipeng Wei, Jingjing Chen, Zuxuan Wu, and Yu-Gang Jiang. Enhancing the self-universality for transferable targeted attacks. In *CVPR*, 2023.

[87] Eric Wong, Leslie Rice, and J Zico Kolter. Fast is better than free: Revisiting adversarial training. In *ICLR*, 2020.

[88] Eric Wong, Frank Schmidt, and Zico Kolter. Wasserstein adversarial examples via projected sinkhorn iterations. In *ICML*, 2019.

[89] Dongxian Wu, Yisen Wang, Shu-Tao Xia, James Bailey, and Xingjun Ma. Skip connections matter: On the transferability of adversarial examples generated with ResNets. In *ICLR*, 2020.
[90] Weibin Wu, Yuxin Su, Xixian Chen, Shenglin Zhao, Irwin King, Michael R Lyu, and Yu-Wing Tai. Boosting the transferability of adversarial samples via attention. In CVPR, 2020.

[91] Chaowei Xiao, Jun-Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. Spatially transformed adversarial examples. In ICLR, 2018.

[92] Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan Yuille. Mitigating adversarial effects through randomization. In ICLR, 2018.

[93] Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan L Yuille, and Kaiming He. Feature denoising for improving adversarial robustness. In CVPR, 2019.

[94] Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan L Yuille. Improving transferability of adversarial examples with input diversity. In CVPR, 2019.

[95] Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. In NDSS, 2018.

[96] Dingcheng Yang, Zihao Xiao, and Wenjian Yu. Boosting the adversarial transferability of surrogate model with dark knowledge. In ICTAI, 2023.

[97] Xiao Yang, Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Boosting transferability of targeted adversarial examples via hierarchical generative networks. In ECCV, 2022.

[98] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In NeurIPS, 2014.

[99] Zheng Yuan, Jie Zhang, and Shiguang Shan. Adaptive image transformations for transfer-based adversarial attack. In ECCV, 2022.

[100] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV, 2014.

[101] Chaoning Zhang, Philipp Benz, Gyusang Cho, Adil Karjauv, Soomin Ham, Chan-Hyun Youn, and In So Kweon. Backpropagating smoothly improves transferability of adversarial examples. In CVPR-AML, 2021.

[102] Chaoning Zhang, Gyusang Cho, Philipp Benz, Kang Zhang, Chen-shuang Zhang, Chan-Hyun Youn, and In So Kweon. Early stop and adversarial training yield better surrogate model: Very non-robust features harm adversarial transferability. In OpenReview, 2021.

[103] Hanwei Zhang, Yannis Avrithis, Teddy Furon, and Laurent Amsaleg. Smooth adversarial examples. EURASIP-JIS, 2020.

[104] Jianping Zhang, Jen-tse Huang, Wenxuan Wang, Yichen Li, Weibin Wu, Xiaosen Wang, Yuxin Su, and Michael R Lyu. Improving the transferability of adversarial samples by path-augmented method. In CVPR, 2023.

[105] Jianping Zhang, Yizhan Huang, Weibin Wu, and Michael R Lyu. Transferable adversarial attacks on vision transformers with token gradient regularization. In CVPR, 2023.

[106] Jianping Zhang, Weibin Wu, Jen-tse Huang, Yizhan Huang, Wenxuan Wang, Yuxin Su, and Michael R Lyu. Improving adversarial transferability via neuron attribution-based attacks. In CVPR, 2022.

[107] Qilong Zhang, Xiaodan Li, Yuefeng Chen, Jingkuan Song, Lianli Gao, Yuan He, and Hui Xue. Beyond imagenet attack: Towards crafting adversarial examples for black-box domains. In ICLR, 2022.

[108] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.

[109] Zhengyu Zhao, Zhuoran Liu, and Martha Larson. Towards large yet imperceptible adversarial image perturbations with perceptual color distance. In CVPR, 2020.

[110] Zhengyu Zhao, Zhuoran Liu, and Martha Larson. On success and simplicity: A second look at transferable targeted attacks. In NeurIPS, 2021.

[111] Zhengyu Zhao, Zhuoran Liu, and Martha Larson. Adversarial image color transformations in explicit color filter space. IEEE TIFS, 2023.

[112] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In CVPR, 2016.

[113] Wen Zhou, Xin Hou, Yongjun Chen, Mengyun Tang, Xiangqi Huang, Xiang Gan, and Yong Yang. Transferable adversarial perturbations. In ECCV, 2018.

[114] Yao Zhu, Jiacheng Sun, and Zhenguo Li. Rethinking adversarial transferability from a data distribution perspective. In ICLR, 2022.

[115] Junhua Zou, Zhisong Pan, Junyang Qiu, Xin Liu, Ting Rui, and Wei Li. Improving the transferability of adversarial examples with resized-diverse-inputs, diversity-ensemble and region fitting. In ECCV, 2020.

**Zhengyu Zhao** (Member, IEEE) received the Ph.D. degree from Radboud University, The Netherlands. He is currently a Professor at Xi'an Jiaotong University, China. His general research interests include Security of Machine Learning and Machine Learning for Security.

**Hanwei Zhang** received the Ph.D. degree from École normale supérieure de Rennes, France, and East China Normal University, China. She is currently a Postdoc at the Institute of Intelligent Software, China, and Saarland University, Germany. Her general research interests focus on trustworthy AI, especially machine learning security and interpretability.

**Renjue Li** received the Ph.D. degree from the Institute of Software, Chinese Academy of Sciences. He is currently a Research Associate at the Institute of AI for Industries, Chinese Academy of Sciences. His main research interests are Adversarial Attacks and Defences on Neural Networks and Neural Network Verification.

**Ronan Sicre** received the Ph.D. degree from the University of Bordeaux, France. He is currently an Assistant Professor at LIS - Ecole Centrale Marseille, Marseille, France. His general research interests include machine learning and computer vision.

**Laurent Amsaleg** (Member, IEEE) received the Ph.D. degree from the University of Paris 6, France. He is now a senior researcher at CNRS. He leads the Linkmedia research team at the IRISA-INRIA lab in Rennes, France. His research interests include high-dimensional indexing at scale and multimedia analytics, as well as the many facets of security issues in relation to the processing of extremely large collections of multimedia material.

**Michael Backes** (Fellow, IEEE) received the Ph.D. degree from Saarland University, Germany. He is the founding director of the CISPA Helmholtz Center for Information Security, Germany. His research focuses primarily on trustworthy machine learning, personal data protection, and software and system security. He has published over 300 papers in renowned international journals and conference proceedings and earned the ERC Synergy Grant, which is Europe's most highly endowed EU group research award, the Karl Heinz Beckurts Prize, an ERC Starting Grant, the Caspar Bowden Privacy Award, the IEEE and ACM Fellowship, as well as various Career Awards and Best Paper Awards.

**Qi Li** (Senior Member, IEEE) received the Ph.D. degree from Tsinghua University, where he is currently an Associate Professor with the Institute for Network Sciences and Cyberspace. He heads the Internet and Cloud Security Group with a group of graduate students working on different topics in security. His research interests are network and system security, particularly in Internet security, mobile security, and big data security.

**Qian Wang** (Fellow, IEEE) received the Ph.D. degree in Electrical and Computer Engineering from Illinois Institute of Technology. He is currently a Professor in the School of Cyber Science and Engineering, and an Adjunct Professor in the School of Computer Science at Wuhan University. His research interests include AI Security, Data Storage, Search and Computation Outsourcing Security and Privacy, Wireless Systems Security, Big Data Security and Privacy, and Applied Cryptography, etc.

**Chao Shen** (Senior Member, IEEE) received the Ph.D. degree in control theory and engineering from Xi'an Jiaotong University, China, where he is currently a Professor with the Faculty of Electronic and Information Engineering. His research interests include AI security, insider/intrusion detection, behavioral biometrics, and measurement/experimental methodology.
# APPENDIX

## A. Comparisons between Existing and Our Evaluations

We compare the Detailed statistics about the transferability evaluations in existing work are summarized in Table IX. As can be seen, existing work considers 11 attacks at most and has not conducted intra-category analyses. We make sure our threat model follows the most common settings in existing work regarding five aspects, as summarized in Table X.

<table><caption>TABLE IX: Comparisons between existing and our evaluations.</caption><thead><tr><th>Studies</th><th># of attacks (categories)</th><th>Fair comparisons</th><th>Intra-category analyses</th></tr></thead><tbody><tr><td>MI [15]</td><td>1 (1)</td><td></td><td></td></tr><tr><td>NI [40]</td><td>4 (2)</td><td></td><td></td></tr><tr><td>PI [80]</td><td>5 (2)</td><td>✓</td><td></td></tr><tr><td>DI [94]</td><td>2 (2)</td><td>✓</td><td></td></tr><tr><td>TI [16]</td><td>3 (2)</td><td></td><td></td></tr><tr><td>SI [40]</td><td>4 (2)</td><td></td><td></td></tr><tr><td>VT [78]</td><td>6 (2)</td><td></td><td></td></tr><tr><td>Admix [79]</td><td>4 (1)</td><td></td><td></td></tr><tr><td>TAP [113]</td><td>2 (2)</td><td>✓</td><td></td></tr><tr><td>AA [28]</td><td>2 (2)</td><td>✓</td><td></td></tr><tr><td>ILA [24]</td><td>4 (3)</td><td></td><td></td></tr><tr><td>FIA [83]</td><td>6 (3)</td><td></td><td></td></tr><tr><td>NAA [106]</td><td>5 (2)</td><td>✓</td><td></td></tr><tr><td>SGM [89]</td><td>4 (3)</td><td></td><td></td></tr><tr><td>LinBP [21]</td><td>5 (3)</td><td>✓</td><td></td></tr><tr><td>RFA [71]</td><td>1 (1)</td><td></td><td></td></tr><tr><td>IAA [114]</td><td>7 (4)</td><td>✓</td><td></td></tr><tr><td>DSM [96]</td><td>4 (3)</td><td></td><td></td></tr><tr><td>GAP [57]</td><td>1 (1)</td><td></td><td></td></tr><tr><td>CDA [50]</td><td>9 (4)</td><td>✓</td><td></td></tr><tr><td>GAPF [30]</td><td>3 (1)</td><td>✓</td><td></td></tr><tr><td>BIA [107]</td><td>5 (3)</td><td>✓</td><td></td></tr><tr><td>TTP [49]</td><td>11 (5)</td><td>✓</td><td></td></tr><tr><td>Ours</td><td>23 (5)</td><td>✓</td><td>✓</td></tr></tbody></table>

## B. Additional Experimental Results

In Fig. 14, Fig. 15, Fig. 13, Table XI, and Table XII, we present additional experimental results under the common settings in to support our findings about transferability and stealthiness.

Furthermore, we validate that our new insights drawn from the intra-category analyses still hold beyond the common settings in . Specifically, Fig. 12 shows that for gradient stabilization attacks, “using more iterations may even decrease the performance” still holds on more advanced models, i.e., ViT and PubDef. In addition, we look into a practical scenario with attack integration. Following the common practice, we integrate the well-known attacks, DI or MI, and Fig. 13 demonstrates that the relative attack performance remains largely unchanged after integration. This is also consistent with findings in existing work that their new method can achieve consistent superiority across different integrations. However, it should be noted that integrating more attacks incurs more computation costs and yields less stealthy images.

## C. Hyperparameters of Attacks and Defenses

We follow the settings in the existing work for attack and defense hyperparameters, as summarized in Table XIII.

<table><caption>TABLE X: Threat models in existing work regarding five aspects. For each aspect (A/B), ● denotes only considering A, ○ denotes only considering B, and ● denotes both.</caption><thead><tr><th>Attacks</th><th>Scenario (public model/ private system)</th><th>Attack goal (untarget/ target)</th><th>Dataset (ImageNet/ others)</th><th>Model (CNNs/ others)</th><th>Source/target training data (same/ disjoint)</th></tr></thead><tbody><tr><td>MI [15]</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><td>NI [40]</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><td>PI [80]</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><td>DI [94]</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><td>TI [16]</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><td>SI [40]</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><td>VT [78]</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><td>Admix [79]</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><td>TAP [113]</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><td>AA [28]</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><td>ILA [24]</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><td>FIA [83]</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><td>NAA [106]</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><td>SGM [89]</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><td>LinBP [21]</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><td>RFA [71]</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><td>IAA [114]</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><td>DSM [96]</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><td>GAP [57]</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><td>CDA [50]</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><td>GAPF [30]</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><td>BIA [107]</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><td>TTP [49]</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr><tr><td>Ours</td><td>●</td><td>●</td><td>●</td><td>●</td><td>●</td></tr></tbody></table>

<figure><img src="image_14.png" /><figcaption>Fig. 12: Transferability vs. iteration curve on ViT and PubDef target models for gradient stabilization attacks.</figcaption></figure>

<figure><img src="image_15.png" /><figcaption>Fig. 13: Integrating one category of attacks with a well-known attack (i.e., DI or MI) from another category does not change the relative attack performance.</figcaption></figure>
<figure><img src="image_16.png" /><figcaption>Fig. 14: Transferability vs. iteration curve on three different surrogates for gradient stabilization attacks. Our findings about transferability in Fig. 2 still hold.</figcaption></figure>

<figure><img src="image_17.png" /><figcaption>Fig. 15: Transferability vs. iteration curve on three different surrogates for input augmentation attacks. Our findings about transferability in Fig. 3 still hold.</figcaption></figure>

## D. Descriptions of Attacks

**Momentum Iterative (MI)** [15] integrates the momentum term into the iterative optimization of attacks, in order to stabilize the update directions and escape from poor local maxima. This momentum term accumulates a velocity vector in the gradient direction of the loss function across iterations. **Nesterov Iterative (NI)** [40] is based on an improved momentum method that also leverages the look ahead property of Nesterov Accelerated Gradient (NAG) by making a jump in the direction of previously accumulated gradients before computing the current gradients. This makes the attack escape

<table><caption>TABLE XI: Attack transferability (%) under $L_\infty = 8/255$. Our findings about transferability in Table VI still hold. Note that $RFA_\infty$ with $L_\infty = 8/255$ is weak since its surrogate is an adversarially robust model also under $L_\infty = 8/255$.</caption><thead><tr><th>Attacks</th><th>DN121</th><th>VGG19</th><th>IncV3</th><th>ViT</th></tr></thead><tbody><tr><td>Clean Acc</td><td>100.0</td><td>100.0</td><td>100.0</td><td>94.7</td></tr><tr><td>PGD</td><td>53.9</td><td>50.7</td><td>28.0</td><td>12.7</td></tr><tr><td>MI</td><td>67.2</td><td>59.9</td><td>39.4</td><td>17.3</td></tr><tr><td>NI</td><td>67.8</td><td>63.3</td><td>40.0</td><td>17.6</td></tr><tr><td>PI</td><td>76.2</td><td>70.7</td><td>44.2</td><td>19.0</td></tr><tr><td>DI</td><td>91.6</td><td>92.6</td><td>46.4</td><td>20.7</td></tr><tr><td>TI</td><td>85.6</td><td>84.8</td><td>41.5</td><td>19.3</td></tr><tr><td>SI</td><td>74.4</td><td>64.0</td><td>40.4</td><td>16.3</td></tr><tr><td>VT</td><td>76.7</td><td>72.7</td><td>41.1</td><td>18.0</td></tr><tr><td>Admix</td><td>71.7</td><td>70.1</td><td>36.2</td><td>15.7</td></tr><tr><td>TAP</td><td>56.0</td><td>56.3</td><td>34.0</td><td>15.1</td></tr><tr><td>AA</td><td>27.7</td><td>31.6</td><td>24.2</td><td>13.1</td></tr><tr><td>ILA</td><td>77.8</td><td>76.4</td><td>45.0</td><td>19.3</td></tr><tr><td>FIA</td><td>84.1</td><td>83.8</td><td>58.4</td><td>26.2</td></tr><tr><td>NAA</td><td>82.7</td><td>79.6</td><td>59.9</td><td>28.4</td></tr><tr><td>SGM</td><td>69.8</td><td>68.1</td><td>35.1</td><td>17.4</td></tr><tr><td>LinBP</td><td>82.6</td><td>83.3</td><td>44.2</td><td>18.9</td></tr><tr><td>RFA<sub>2</sub></td><td>92.9</td><td>86.8</td><td>73.1</td><td>41.6</td></tr><tr><td>RFA<sub>∞</sub></td><td>22.7</td><td>19.4</td><td>32.5</td><td>17.6</td></tr><tr><td>IAA</td><td>91.2</td><td>90.9</td><td>54.0</td><td>22.9</td></tr><tr><td>DSM</td><td>81.2</td><td>75.3</td><td>36.1</td><td>15.8</td></tr><tr><td>GAP</td><td>39.4</td><td>50.8</td><td>32.0</td><td>14.7</td></tr><tr><td>CDA</td><td>45.4</td><td>69.4</td><td>34.4</td><td>15.9</td></tr><tr><td>GAPF</td><td>93.5</td><td>96.6</td><td>49.2</td><td>22.9</td></tr><tr><td>BIA</td><td>72.3</td><td>82.3</td><td>41.6</td><td>18.7</td></tr><tr><td>TTP</td><td>62.1</td><td>71.9</td><td>42.2</td><td>21.0</td></tr></tbody></table>

<table><caption>TABLE XII: Imperceptibility regarding five metrics under $L_\infty = 8/255$. Our findings about stealthiness in Table VII still hold.</caption><thead><tr><th>Attacks</th><th>PSNR↑</th><th>SSIM↑</th><th>ΔE↓</th><th>LPIPS↓</th><th>FID↓</th></tr></thead><tbody><tr><td>PGD</td><td>33.651</td><td>0.866</td><td>0.343</td><td>0.058</td><td>15.227</td></tr><tr><td>MI</td><td>32.13</td><td>0.816</td><td>0.365</td><td>0.104</td><td>20.245</td></tr><tr><td>NI</td><td>31.971</td><td>0.809</td><td>0.372</td><td>0.112</td><td>22.398</td></tr><tr><td>PI</td><td>31.918</td><td>0.809</td><td>0.374</td><td>0.112</td><td>25.015</td></tr><tr><td>DI</td><td>33.415</td><td>0.864</td><td>0.349</td><td>0.062</td><td>35.828</td></tr><tr><td>TI</td><td>33.285</td><td>0.864</td><td>0.354</td><td>0.063</td><td>29.333</td></tr><tr><td>SI</td><td>33.106</td><td>0.857</td><td>0.350</td><td>0.068</td><td>24.483</td></tr><tr><td>VT</td><td>32.777</td><td>0.850</td><td>0.370</td><td>0.074</td><td>25.213</td></tr><tr><td>Admix</td><td>33.225</td><td>0.855</td><td>0.350</td><td>0.070</td><td>22.596</td></tr><tr><td>TAP</td><td>31.875</td><td>0.810</td><td>0.365</td><td>0.101</td><td>44.118</td></tr><tr><td>AA</td><td>31.949</td><td>0.842</td><td>0.421</td><td>0.090</td><td>25.691</td></tr><tr><td>ILA</td><td>31.511</td><td>0.812</td><td>0.448</td><td>0.089</td><td>51.048</td></tr><tr><td>FIA</td><td>31.886</td><td>0.833</td><td>0.405</td><td>0.086</td><td>85.818</td></tr><tr><td>NAA</td><td>31.962</td><td>0.837</td><td>0.408</td><td>0.078</td><td>54.370</td></tr><tr><td>SGM</td><td>33.020</td><td>0.855</td><td>0.349</td><td>0.066</td><td>19.759</td></tr><tr><td>LinBP</td><td>32.337</td><td>0.840</td><td>0.376</td><td>0.073</td><td>44.614</td></tr><tr><td>RFA<sub>2</sub></td><td>32.024</td><td>0.858</td><td>0.391</td><td>0.086</td><td>40.841</td></tr><tr><td>RFA<sub>∞</sub></td><td>30.416</td><td>0.884</td><td>0.455</td><td>0.074</td><td>14.753</td></tr><tr><td>IAA</td><td>31.807</td><td>0.829</td><td>0.417</td><td>0.081</td><td>109.173</td></tr><tr><td>DSM</td><td>33.337</td><td>0.862</td><td>0.349</td><td>0.062</td><td>22.101</td></tr><tr><td>GAP</td><td>30.171</td><td>0.769</td><td>0.460</td><td>0.125</td><td>56.566</td></tr><tr><td>CDA</td><td>30.455</td><td>0.796</td><td>0.483</td><td>0.103</td><td>75.639</td></tr><tr><td>GAPF</td><td>30.800</td><td>0.810</td><td>0.473</td><td>0.095</td><td>146.506</td></tr><tr><td>BIA</td><td>30.347</td><td>0.751</td><td>0.462</td><td>0.161</td><td>123.255</td></tr><tr><td>TTP</td><td>30.932</td><td>0.829</td><td>0.430</td><td>0.107</td><td>69.490</td></tr></tbody></table>
<table><caption>TABLE XIII: Hyperparameters of attacks and defenses.</caption><thead><tr><th>Attacks</th><th>Hyperparameter</th></tr></thead><tbody><tr><td>MI [15]</td><td>decay factor μ = 1</td></tr><tr><td>NI [40]</td><td>decay factor μ = 1</td></tr><tr><td>PI [80]</td><td>decay factor μ = 1</td></tr><tr><td>DI [94]</td><td>resize&pad range R = [1, 1.1]</td></tr><tr><td>TI [16]</td><td>transformation probability p = 0.7</td></tr><tr><td>SI [40]</td><td>translation range R = [-2, 2]</td></tr><tr><td>SI [40]</td><td>scale range R = [0.1, 1]</td></tr><tr><td>VT [78]</td><td>noise range R = [-1.5ε, 1.5ε]</td></tr><tr><td>Admix [79]</td><td>mixing factor η = 0.2</td></tr><tr><td>TAP [113]</td><td>λ = 0.005, η = 0.01, α = 0.5</td></tr><tr><td>AA [28]</td><td>20 random target images from 4 classes</td></tr><tr><td>ILA [24]</td><td>ILA projection loss</td></tr><tr><td>FIA [83]</td><td>N = 30, P<sub>drop</sub> = 0.3</td></tr><tr><td>NAA [106]</td><td>N = 30, γ = 1.0, linear transformation</td></tr><tr><td>SGM [89]</td><td>decay parameter γ = 0.5</td></tr><tr><td>LinBP [21]</td><td>first residual unit in the third meta block</td></tr><tr><td>RFA [71]</td><td>L<sub>2</sub> ε = 0.1 or L<sub>∞</sub> ε = 8 PGD-AT</td></tr><tr><td>IAA [114]</td><td>β = 15</td></tr><tr><td>DSM [96]</td><td>CutMix augmentation</td></tr><tr><td>GAP [57]</td><td>ResNet152 discriminator</td></tr><tr><td>CDA [50]</td><td>ResNet152 discriminator</td></tr><tr><td>GAPF [30]</td><td>ResNet152 discriminator</td></tr><tr><td>BIA [107]</td><td>ResNet152 discriminator, RN module</td></tr><tr><td>TTP [49]</td><td>ResNet50 discriminator</td></tr><tr><td>Defenses</td><td>Hyperparameters</td></tr><tr><td>BDR [95]</td><td>bit depth D = 2</td></tr><tr><td>PD [58]</td><td>R-CAM: k = 5, denoising: σ = 0.04</td></tr><tr><td>R&P [92]</td><td>resize&pad range R = [1, 1.1]</td></tr><tr><td>HGD [39]</td><td>ResNet152-Wide</td></tr><tr><td>NRP [48]</td><td>ResNet (1.2M parameters)</td></tr><tr><td>DiffPure [53]</td><td>noise level t = 150 WideResNet-50-2</td></tr><tr><td>AT<sub>∞</sub> [93]</td><td>L<sub>∞</sub> ε = 8, PGD iters n = 30, Ir=α = 1</td></tr><tr><td>FD<sub>∞</sub> [93]</td><td>L<sub>∞</sub> ε = 8, PGD iters n = 30, Ir=α = 1</td></tr><tr><td>AT<sub>2</sub> [62]</td><td>L<sub>2</sub> ε = 3.0, PGD iters n = 7, Ir=α = 0.5</td></tr></tbody></table>

from poor local maxima more easily and faster.

**Pre-gradient guided Iterative (PI) [80]** follows a similar idea as NI, but it makes a jump based on only the gradients from the last iteration instead of all previously accumulated gradients.

**Diverse Inputs (DI) [94]** applies random image resizing and padding to the input image before calculating the gradients in each iteration of the attack optimization. This approach aims to prevent attack overfitting (to the white-box, source model), inspired by the data augmentation techniques used for preventing model overfitting.

**Translation Invariant (TI) [16]** applies random image translations for input augmentation. It also introduces an approximate solution to improve the attack efficiency by directly computing locally smoothed gradients on the original image through the convolution operations rather than computing gradients multiple times for all potential translated images.

**Scale Invariant (SI) [40]** applies random image scaling for input augmentation. It scales pixels with a factor of $1/2^i$. In particular, in each iteration, it takes an average of gradients on multiple augmented images rather than using only one augmented image as in previous input augmentation attacks.

**Variance Tuning (VT) [78]** applies uniformly distributed additive noise to images for input augmentation and also calculates average gradients over multiple augmented images in each iteration.

**Adversarial mixup (Admix) [79]** calculates the gradients on

a composite image that is made up of the original image and another image randomly selected from an incorrect class. The original image label is still used in the loss function.

**Transferable Adversarial Perturbations (TAP) [113]** proposes to maximize the distance between original images and their adversarial examples in the intermediate feature space and also introduces a regularization term for reducing the variations of the perturbations and another regularization term with the cross-entropy loss.

**Activation Attack (AA) [28]** drives the feature-space representation of the original image towards the representation of a target image that is selected from another class. Specifically, AA can achieve targeted misclassification by selecting a target image from that specific target class.

**Intermediate Level Attack (ILA) [24]** optimizes the adversarial examples in two stages, with the cross-entropy loss used in the first stage to determine the initial perturbations, which will be further fine-tuned in the second stage towards larger feature distance while maintaining the initial perturbation directions.

**Feature Importance-aware Attack (FIA) [83]** proposes to only disrupts important features. Specifically, it measures the importance of features based on the aggregated gradients with respect to feature maps computed on a batch of transformed original images that are achieved by random image masking.

**Neuron Attribution-based Attacks (NAA) [106]** relies on an advanced neuron attribution method to measure the feature importance more accurately. It also introduces an approximation approach to conducting neuron attribution with largely reduced computations.

**Skip Gradient Method (SGM) [89]** suggests using ResNet-like architectures as the source model during creating the adversarial examples. Specifically, it shows that backpropagating gradients through skip connections lead to higher transferability than through the residual modules.

**Linear BackPropagation (LinBP) [21]** is proposed based on the new finding that the non-linearity of the commonly-used ReLU activation function substantially limits the transferability. To address this limitation, the ReLU is replaced by a linear function during only the backpropagation process.

**Robust Feature-guided Attack (RFA) [71]** proposes to use an adversarially-trained model (with $L_2$ or $L_∞$ bound) as the source model based on the assumption that modifying more robust features yields more generalizable (transferable) adversarial examples.

**Intrinsic Adversarial Attack (IAA) [114]** finds that disturbing the intrinsic data distribution is the key to generating transferable adversarial examples. Based on this, it optimizes the hyperparameters of the Softplus and the weights of skip connections per layer towards aligned attack directions and data distribution.

**Dark Surrogate Model (DSM) [96]** is trained from scratch with additional “dark” knowledge, which is achieved by training with soft labels from a pre-trained teacher model and using data augmentation techniques, such as Cutout, Mixup, and CutMix.

**Generative Adversarial Perturbations (GAP) [57]** proposes a new attack approach that is based on generative modeling. Specifically, it uses the source classifier as the discriminator
and trains a generator using the cross-entropy loss. Once trained, the generator can be used to generate an adversarial example for each input original image with only one forward pass.

**Cross-Domain Attack (CDA)** [50] follows the GAP pipeline but uses a more advanced loss (i.e., relativistic cross entropy) to train the generator. This new loss explicitly enforces the probability gap between the clean and adversarial images, boosting the transferability, especially in cross-domain scenarios.

**Transferable Targeted Perturbations (TTP)** [49] is focused on improving transferability of targeted attacks. It is based on learning target-specific generators, each of which is trained with the objective of matching the distribution of targeted perturbations with that of data from a specific target class. Specifically, input augmentation and smooth perturbation projection are used to further boost the performance.

**Generative Adversarial Feature Perturbations (GAFP)** [30] follows the general pipeline of GAP but trains the generator using a loss that maximizes the feature map distance between adversarial and original images at mid-level CNN layers.

**Beyond ImageNet Attack (BIA)** [107] also follows the general pipeline of GAP and specifically introduces a random normalization module to simulate different training data distributions and also a feature distance loss that is only based on important/generalizable features.

## E. Descriptions of Defenses

**Bit-Depth Reduction (BDR)** [95] pre-processes input images by reducing the color depth of each pixel while maintaining the semantics. This operation can eliminate pixel-level adversarial perturbations from adversarial images but has little impact on model predictions of clean images.

**Pixel Deflection (PD)** [58] pre-processes input images by randomly replacing some pixels with randomly selected pixels from their local neighborhood. It is specifically designed to happen more frequently to non-salient pixels, and a subsequent wavelet-based denoising operation is used to soften the corruption.

**Resizing and Padding (R&P)** [92] pre-processes input images by random resizing, which resizes the input images to a random size, and then random padding, which pads zeros around the resized input images.

**High-level representation Guided Denoiser (HGD)** [39] learns a purification network that can be used to purify/denoise the adversarial perturbations. Specifically, different from previous methods that focus on image-space denoising, HGD minimizes the difference between the clean image and the denoised image at intermediate feature layers.

**Neural Representation Purifier (NRP)** [48] learns a purification network using a combined loss that calculates both the image- and feature-space differences. Specifically, the adversarial images used for training the purification network are generated by feature loss-based adversaries, which are shown to be more effective in handling unseen attacks.

**Diffusion Purification (DiffPure)** [53] uses a diffusion model as the purification network. It diffuses an input image by gradually adding noise in a forward diffusion process and then

recover the clean image by gradually denoising the image in a reverse generative process. The reverse process is shown to be also capable of removing adversarial perturbations.

**$L_\infty$-Adversarial Training ($AT_\infty$)** [93] follows the basic pipeline of adversarial training, which is to train the robust model on adversarial images. Specifically, these adversarial images are generated using the PGD targeted attacks with the $L_\infty$ distance, and the model is trained distributedly on 128 Nvidia V100 GPUs.

**$L_\infty$-Adversarial Training with Feature Denoising ($FD_\infty$)** [93] modifies the standard model architecture by introducing new building blocks that are designed for denoising feature maps based on non-local means or other filters. This modification can help suppress the potential disruptions caused by the adversarial perturbations, and the modified model is trained end-to-end.

**$L_2$-Adversarial Training ($AT_2$)** [62] trains the model on adversarial images that are generated using the PGD non-targeted attacks with the $L_2$ distance.

## F. Imperceptibility Metrics

**Peak Signal-to-Noise Ratio (PSNR)** measures the ratio of the maximum possible power of a signal (image $x$) to the noise (perturbations $\sigma$) power. It is calculated by:

$$ \text{PSNR}(\mathbf{x}, \sigma) = 10 \cdot \log_{10} \frac{\max(\mathbf{x})^2}{\text{MSE}(\mathbf{x}, \sigma)}, \quad (9) $$

where the Mean Squared Error (MSE) measures the average squared difference between the two inputs.

**Structural Similarity Index Measure (SSIM)** [84] is used for measuring the perceptual quality of digital images. In our case, it measures the structural similarity between the original image $x$ and adversarial image $x'$, which is considered to be a degraded version of $x$. It is calculated by:

$$ \text{SSIM}(\mathbf{x}, \mathbf{x}') = \frac{(2\mu_{\mathbf{x}}\mu_{\mathbf{x}'} + c_1) + (2\sigma_{\mathbf{x}\mathbf{x}'} + c_2)}{(\mu_{\mathbf{x}}^2 + \mu_{\mathbf{x}'}^2 + c_1)(\sigma_{\mathbf{x}}^2 + \sigma_{\mathbf{x}'}^2 + c_2)}. \quad (10) $$

Descriptions of $c_1$ and $c_1$ and more technical details can be found in [84].

**$\Delta E$** [45] refers to the CIE $\Delta E$ standard formula used for measuring the perceptual color distance between two pixels. We follow [109] to use the latest variant, CIEDE2000, since it is shown to align very well with human perception. It is calculated in the CIELCH space by:

$$ \begin{aligned} \Delta E &= \sqrt{\left(\frac{\Delta L'}{k_L S_L}\right)^2 + \left(\frac{\Delta C'}{k_C S_C}\right)^2 + \left(\frac{\Delta H'}{k_H S_H}\right)^2 + \Delta R}, \\ \Delta R &= R_T\left(\frac{\Delta C'}{k_C S_C}\right)\left(\frac{\Delta H'}{k_H S_H}\right), \end{aligned} \quad (11) $$

where $\Delta L'$, $\Delta C'$, $\Delta H'$ denotes the distance between two pixels in their lightness, chroma, and hue channels, respectively. $\Delta R$ is an interactive term between chroma and hue differences. Detailed definitions and explanations of the weighting functions ($S_L$, $S_C$, $S_H$ and $R_T$) and hyperparameters $k_L$, $k_C$ and $k_H$ can be found in [45]. We use the averaged $\Delta E$ over all pixels.
**Learned Perceptual Image Patch Similarity (LPIPS) [108]** is developed for measuring the perceptual similarity between two images. It computes the cosine distance (in the channel dimension) between features at each given convolutional layer *l* and averages the results across spatial dimensions *H* × *W* and layers of a specific network *f*:

$$ \text{LPIPS} = \sum_{l} \frac{1}{H_{l}W_{l}} \sum_{h,w} \cos(f_{hw}^{l}(\mathbf{x}), f_{hw}^{l}(\mathbf{x}')) \quad (12) $$

where AlexNet is adopted as *f* for computational efficiency. **Frechet Inception Distance (FID)** [73] is originally used to assess the quality of generated images and can also be used to assess the quality of adversarial images. It compares the distribution of adversarial images to that of original images. Specifically, the output features from the pool3 layer of an Inception-V3 for original and adversarial images are used to fit two multidimensional Gaussian distributions $N(\mu, \Sigma)$ and $N(\mu', \Sigma')$, and the FID score is calculated by:

$$ \text{FID}(\mathcal{N}(\mu, \Sigma), \mathcal{N}(\mu', \Sigma')) = \|\mu - \mu'\|_2^2 + \text{tr}(\Sigma + \Sigma' - 2\sqrt{\Sigma\Sigma'}) \quad (13) $$

where tr denotes the trace of the matrix.

## G. Interpretability Measures

For model interpretability, we adopt two metrics, Average Increase (AI) and Average Drop (AD), from [9]. Let $p_i^c$ and $o_i^c$ be the predicted probability for the ground-truth class *c* given as input the *i*-th image $\mathbf{x}_i$ and its masked version, and let *n* be the number of test images. AI measures the percentage of images where the masked image yields a higher class probability than the original (higher is better):

$$ \text{AI} (\%) := \frac{1}{n} \sum_{i}^{n} \mathbb{1}_{p_i^c < o_i^c} \cdot 100 \quad (14) $$

AD quantifies how much predictive power, measured as class probability, is lost when only the masked regions of the image are used (lower is better):

$$ \text{AD} (\%) := \frac{1}{n} \sum_{i=1}^{n} \frac{[p_i^c - o_i^c]_+}{p_i^c} \cdot 100. \quad (15) $$

## H. Additional Visualizations
<figure><img src="image_18.png" /><figcaption>Fig. 16: Adversarial examples generated by gradient stabilization attacks and the PGD baseline, along with original images.</figcaption></figure>
<figure><img src="image_19.png" /><figcaption>Fig. 17: Adversarial examples generated by input augmentation attacks.</figcaption></figure>
<figure><img src="image_20.png" /><figcaption>Fig. 18: Adversarial examples generated by feature disruption attacks.</figcaption></figure>
<figure><img src="image_21.png" /><figcaption>Fig. 19: Adversarial examples generated by surrogate refinement attacks.</figcaption></figure>
<figure><img src="image_22.png" /><figcaption>Fig. 20: Adversarial examples generated by generative modeling attacks.</figcaption></figure>