A Survey on Transferability of Adversarial Examples Across Deep Neural Networks
Authors: Jindong Gu
Xiaojun Jia
Pau de Jorge
Wenqian Yu
Xinwei Liu
Avery Ma
Yuan Xun
Anjun Hu
Ashkan Khakzar
Zhijiang Li
Xiaochun Cao
Philip Torr
Venue: Accepted by TMLR ()

Abstract: The emergence of Deep Neural Networks (DNNs) has revolutionized various domains by enabling the resolution of complex tasks spanning image recognition, natural language processing, and scientific problem-solving. However, this progress has also brought to light a concerning vulnerability: adversarial examples. These crafted inputs, imperceptible to humans, can manipulate machine learning models into making erroneous predictions, raising concerns for safety-critical applications. An intriguing property of this phenomenon is the transferability of adversarial examples, where perturbations crafted for one model can deceive another, often with a different architecture. This intriguing property enables ``black-box'' attacks which circumvents the need for detailed knowledge of the target model. This survey explores the landscape of the adversarial transferability of adversarial examples. We categorize existing methodologies to enhance adversarial transferability and discuss the fundamental principles guiding each approach. While the predominant body of research primarily concentrates on image classification, we also extend our discussion to encompass other vision tasks and beyond. Challenges and opportunities are discussed, highlighting the importance of fortifying DNNs against adversarial vulnerabilities in an evolving landscape.

Published in Transactions on Machine Learning Research (05/2024)
A Survey on Transferability of Adversarial Examples Across Deep Neural Networks
Jindong Gu', Xiaojun ; Pau de Jorge', Wenqain Yu², Xinwei Lin', Avery Ma², Yuan Xun*, Anjun Hu', Ashkan Khakzar', Zhijiang Li Xiaochun Cao Philip Torr'
Torr Vision Croup, University of Orford, Ouford, United Kingdom
Nangang Technological University, Singapore
Wuhan University, Wuhan, China
University of Chinese Academy of Sciences, Beijing, China
University of Toronto, Torvnlo, Canada
Sun Yal-sen Universily, Shenzhen, China
Reviewed on OpenReview: https: // openreview. net/ forun? id= AYJ3m
Abstract
The emergence of Deep Neural Networks (DNNs) has revolutionized various domains by enabling the resolution of complex lasks spanning image recognition, natural language pro- cossing, and scientific problem-solving. However, this progress has also brought to light concerning vulnerability: adversarial examples. These crafted inputs, imperceptible to humans, can manipulate machine learning models into making erroncous predictions, rais- ing concerns for safety-critical applications. An intriguing property of this phenomenon is the transferability of adversarial examples, where perturbations crafted for one model can deceive another, oflen with different architecture. This intriguing property enables "black box" attacks which circumvents the need for detailed knowledge of the target model. This survey explores the landscape of the adversarial transfcrability of adversarial examples. We categorize existing methodologies enhance adversarial transferability and discuss the fundamental principles guiding each approach. While the predominant body of research primarily concentrates image classification, also extend our discussion to encompass other vision tasks and beyond. Challenges and opportunities are discussed, highlighting the importance of fortifying DNNs against adversarial vulnerabilities in an evolving landscape.
1 Introduction
In recent years, Deep Neural Network (DNN) has evolved as powerful tool for solving complex tasks, ranging from image recognition (He et al.
Dosovitskiy et al. 2020) and natural language processing (Kenton to natural science problems (Wang et al. 2023). Since the advent of neural networks, an intriguing and disconcerting phenomenon known as adversarial examples has come into focus (Szegedy et al.) 2013 Goodfellow et al.] 2014). Adversarial examples are specially crafted inputs that lead machine learning models make incorrect predictions. These inputs are imperceptibly different from correctly predicted inputs. The existence of adversarial examples poses potential thrcats to real-world safety-critical DNN-based applications, e.g., medical image analysis (Bortsova al. 2021 and autonomous
2016
& Toutaanoa| 2019: Brown et al.
2020a
driving systems (Kim & Canny 2017 Kim et al. 2018).
While the existence of adversarial examples has raised concerns about the robustness and reliability of ma- chine learning systems, researchers have uncovered an even more intriguing phenomenon: the trunsferabilily of adversorial ezamples (Goodfellow et al. 2014 Papernot et al. 2016). Transferability refers to the ability of an adversarial example designed for one model successfully deccive a different model, often one with a distinct architecture. With such property, a successful attack can be implemented without accessing any detail of the target model, such model architecture, model parameters, and training data.

---

Published in Transactions on Machine Learning Research (05/2024)
Table 1: Categorization of trannferability--hhannnng methods.
Xic et al.
2019
Dong ct al.
2019
Lin et al. (2019) Zou al. (2020)
Data Augmentation
Wu et a.
(2021) Li al. 2020bb
Byun et al. (0220) Wang et al.
[2021a]
Huang Kong 2022
Goodfellow et al. [0044] Dong al. (2018): Nesterov (1983) Zou et al.
Optimization Technique
[2022
Wang & He (2021): Xiong et al. (2022):
Zhu et al.
22233b
Li
et al.
2020a] Qin et al. (2022) Ma et al. (2023)
Gubri et al. (2022b)
Optimization- Based
Zhang et al. (2022a Xiao et al.
2021 ;
Li ct al.
2020a
Zhao et al.
Loss Objective
2021 Fang et al. [2022):
Xu et al.
(022bb
Li et
2023
Qlan al.
2023 Chen
(2023a
Zhou et al. (2018): Naseer et al. 2018
Hashemi al. (2022): Salzmann
ct al. (2021): :: Ganeshan et a.. (2019)
Wang et al. (2021c) Zhang al.
Model Component
[2022c]: Wu et al. (2020b);
hich al. (2019) 2020b
Waseday
(2033) Wu
(2020a
Guo et al. 2020 Nascer et al.
2022
Poursaced ct al.
2018 ; Xiao ct al
(008): ; Nascer ct al
2019
202
Gencration-
Unconditional Generation
Kim ct al. [2022
Peng ct al. (2022]
Zhao ot al. (2023
Based
Yang al. (2019
(2022):
Han et al.
2019
Mao et al. 2020
: Han et al.
Class-conditoonal Goneration
hhan et al.
2020 Chen et al.
2023b
2024
The recent surge in interest surrounding the transfcrability of adversarial examples is attributed to its po- tential application in executing black-box attacks (Papernot et al., 2016 Liu et al., 2017). Delving into the reasons behind the capacity of adversarial examples tailored for model to deceive others provides researchers with opportunity to acquire profound comprehension of the fundamental mechanisms un- derpinning the susceptibility of DNNs (Wu & Zhu 2020). Moreover, a comprehensive understanding of transferability offers the possibility of fostering the creation of adversarially robust models, capable of effec-
tively countering adversarial attacks (Jiia et al. 2022b Waseda et al. 2023 Πyas et al. 2019
Given the growing volume of publications on adversarial examples, several survey papers (Sun et al. 2018
Zhang & Li 2019 Wiyatno al. 2019 Serban al.) 2020 Han et al. 2023b) have emerged, seeking to en- capsulate these phenomena from diverse viewpoints. Yet, comprehensive survey specifically addressing the transferability of advensarial examples remains absent. This study endeavors to bridge that gap, embarking an extensive investigation into the realm of adversarial transferability, To this end, we thoroughly review the latest research on transferability assessment, methods to enhance transferability, and the associated chal- lenges and prospects. Specifically, as shown Table D we categorize the current
methods into two major calegories: (1) optimization- based methods where one directly optimizes for the adversarial perturbations based on one or more surrogate models inference time, without introducing ad- ditional generative models, and (2) generation-based methods that introduce generative models dedicated for adversary synthesis. Moreover, we also examine adversarial transferability beyond the commonly studied misclassification attacks and provide a summary of such phenomenon in other tasks (e.g. image reevil object detection, segmentation, etc.). Upon assessing the current advancements in adversarial transferability rescarch, we then outline a few challenges and potential avenues for future investigations.
The organization of this paper is as follows: Section first provides the terminology and mathematical nota- tions used across the paper, and then introduces the formulation and evaluation metrics of adversarial trans- ferability. Sections 3-5 present various techniques to improve the adversarial transferability of adversarial examples. Section 3 examines transferabilityy eihheeeniiiityyeeennnnnng methods that are applicable to optimisation-basee transferable attacks. These techniques are categorized into four perspectives: data processing, optimiza- tion, loss objective, and model architectures. In Section 4, various generation-based transferable allacks are presented. Section describes the research on adversarial transferability beyond image classification. Coneretely, transferability-enhnaee
techniques various vision tasks, natural language processing tasks as well as the ones across tasks are discussed. The current challenges and future opportunities in adversarial transferability are discussed in Section 6. The last section concludes our work.
2

---

Published in Transactions on Machine Learning Research (05/2024)
In order to facilitate the literature search, we also built and released project page where the related papers are organized and listed The page will be maintained and updated regularly. As the landscape of DNN continues to evolve, understanding the intricacies of adversarial examples and their transferability is of paramount importance. By shedding light on these vulncrabilitics, we hope to contribute to the development of more secure and robust DNN models that can withstand the challenges posed by adversarial atlacks.
2 Preliminary
In this section, first introduce the terminology and mathematical notations used across the paper. Then, introduce the formulation of adversarial transferability. In the last part, the evaluation of adversarial transferability is presented.
2.1 Terminology and Notations
The terminologies relevant to the topic of adversarial transferability and mathematical annotations are listed in Tab. 2 and Tab. 3 respectively.
Table 2:
The used terminologies are listed. They are followed across the paper.
adversarial
A small artificial perturbation that can cause misclassification of a neural network when added to the input image.
perturbation
target model
The target model (e.g. deep neural network) attack.
surrogate model
The model built to approximate the target model for generating adversarial examples.
white / black-boo
White attacks can access the target model (e.g. architecture and parameters), while black-box allacks cannot.
allacks
untargeted/targe attack
The goal of untargeted attacks to cause misclassifications of a target model, while targeted attacks aim to mislead the target model to classify an image into a specific target class.
clean accurocy
The model performance on the original clean test dataset.
fooling rate
The percentage of images that are misclassified the target model.
Table 3: The used mathematical notations are listed. They are followed across the paper.
I
clean input image
A ground-truth label of an image
8
Adversarial perturbation
Range of adversarial perturbation
X
Input distribution
zadv
Adversarial example + 8 of the input æ
y
Target class of adversarial attack
zadv(t)
Adversarial input at the end of th iteration
ft
Target model under attack
fz
Surrogate (source) model for AE ercation 4thh activation in the tt layer of the target model
f'(x)
the tth model output probability for the input
H²
H
the tth layer of the target network
z²
Model output logits
AE
Adversarial Example
AT
Adversarial Transferability of AE
Acc
Clean accuracy on clean dataset
FR
Fooling rate on target model
transferability
3

---

Published in Transactions on Machine Learning Research (05/2024)
2.2 Formulation of Adversarial Transferability
Given adversarial example æade of the input image z with the label y and two models s+(+) and Se(-), ad- versarial transferability describes the phenomenon that the adversarial example that is able to fool the model fs(+) can also fooling another model ft()). Formally speaking, the adversarial transferability untargetod attacks can formulated as follows:
argmax, f(( & y, given argmax, f { (dde) ) +
(1)
Similarly, the targeted transforable attacks can be doscribed as:
argmax, (z y", given argmax, f2 (2ade) y' and ff(
(2)
The process to create zadv for a given example is detailed in Sec. 3 and Sec. 4.
2.3 Evaluation of Adversarial Transferability
Fooling Rate (FR). The most popular metric used evaluate adversarial transferability is FR. We denote P as the number of adversarial examples that successfully fool source model. Among them, the number of examples that are also able to fool target model is Q. The Fooling Rate is then defined FR = $. The higher the FR is, the more transferable the adversarial examples are.
Interest Class Rank (ICR). According to Zhang et al. (022a)). an interest class is the ground-truth class in untargeted attacks or the target class targeted attacks. FR only indicates whether the interest class ranks top-1, which may not be an in-depth indication of transferability. To gain deeper insights into transferability, is valuable consider the ICR, which represents the rank of the interest class after the attack. In untargeted attacks, higher ICR indicates higher transfcrability, while in targeted attacks, higher ICR suggests lower transferability.
Knowledge Transfer-based Metrics. Liang et al. (021) considered all potential adversarial perturbation vectors and proposed two practical motrics for transfcrability evaluation. The transferability of dataset x ~ D from source model fs to larget model ft are defined as follows:
an (. ((l( (,,c(()))
((f((z). fo(x (x),)
where ,, (x)) the adversrial perturbation generated on surrogate model fs while (x)=(x) is that on target model ft. The first metric an measures the difference between two loss functions which can indicate the performance of attacks: black-box attack from fs to fe and white-box attack on f.. Tranferabilty from fs to fe is high when an is high.
a2/0 boti= (x A
(4)
where
- 6),, fx), A),-)(()) fi( (),, fo(x)
(5)
operation denotes the corresponding unit-length vector and ||:|| denotes the Frobenius norm. The second metric 02 measures the relationship between two deviation directions, indicating white-box attacks on and black-box attacks from fa to respectively.
Liang et al. 2021 h argue that these two metrics represent complementary perspectives of transferability: ai represents how often the adversarial attack transfers, while 02 encodes directional information of the output deviation.
3 Optimization-Based Transferable Attacks
In this section, we introduce optimization-basee transforabillt--chaanni methods: class of methods that seeks adversarial perturbation at test time based more surrogate models without introducing
4

---

Published in Transactions on Machine Learning Research (05/2024)
training additional models (e.g. generative model). Based on our formulation in Section 2.2 to obtain transferable adversarial perturbations with this class of methods can be expressed as:
the process
8 arg Er ((fx(T(+++)) 8)),8), s.t. 110|/cc S
(6)
where 5 is an adversarial perturbation of the same size as the input, T(-) is data augmentation operations, ((+) is designed loss, and f.(-) is a model used in the optimization process, which could be slightly modified version of a surrogate model. The examples of the modifications are linearizing the surrogate model by
removing the non-linear activation functions and highlighting skip connections in backpropagation passes.
The problem in Equation 6 is approximately solved with Projected Gradient Descent (Madry et al. 2017
Multi-step attacks (e.g. (Madry et al. 2017 update the perturbation iteratively as follows
g(++1) Coi +a. siiniigg[[.)
(7)
where a the stop size to update the perturbation, [=de((+1)) = x + 8', and Clip'(.) is a clipping function to make the constraint || S e satisfied. In contrast, single-step attacks (e.g. (Szegedy et al., 2013
Goodfellow et al. 2014) update the adversarial perturbation with only one attack iteration.
Given the expression in Equation D we categorize the transforablitty methods into four categories: data augmentation-basse optimization--aeee ion-based, model-based, and loss-based.
3.1 Data Augmentation-Baset Transferability Enhancing Methods
The family of methods discussed in this section, referred to as Data Augmentatioon-aase methods, are all based the rationale of applying data augmentation strategics. In these strategics, when computing the adversaries on the surrogate model Sa, input transformation T(-) with stochastic component is applied to increase adversarial transferability by preventing overfitting to the surrogate model. In the following, we will discuss the specific instance of T.) for each method.
Diverse Inputs Method (DIM). Xie et al. (209)) are the first to suggest applying differentiable input transformations to the clean image. In particular, their DTM consists of applying a different. transformation at each iteration of multi-step adversarial attacks. They perform random resizing and padding with a certain probability p. that is:
x with probability 1 p T(x)= =f
(8)
pad(resize(z)) with probability p
and as p increases, the transferability of iterative attacks improves most significantly. They also notice that although this is tied to a drop in the white-box performance, the latter is much milder.
Translation Invariance Method (TIM). Dong et al. 2019 study the discriminative image regions of different models, i.c. the image regions more important to classifiers output. They observe different models leverage different regions of the image, especially if adversarially trained. This motivates them to propose the Translation Invariance Method (TIM). That is, they want to compute an adversarial perturbation that works for the original image and any of its translated versions (where the position of all pixels is shifted by fixed amount), therefore:
(9) where la and by define the shift. Moreover, Dong shows that such perturbations can be found with little extra cost by applying kernel matrix on the gradients. This method can be combined with other attacks or augmentation techniques (e.g. DIM) to further improve transfcrability.
T(x)[i,]] = z/i + tz.j + +ty
Scale Invariance Method (SIM). When atlacking in a black-box setting, Dong el al. 2018) showed that computing an attack for an ensemble of models improves transferability albeit at an increased computational cost. Motivated by this fact, Lin et al. (2019) introduce the concept of loss-preserving transformation (any input transformation T that satisfies & ((((Tz))) y) ze {(f(x). y) Vx) and of model augmentation (given a loss- preserresg transformation (T), then f' S(T(-)) would be an augmented model of f)- One can then
5

---

Published in Transactions on Machine Learning Research (05/2024)
use different model augmentations and treat them ensemble. In particular, Lin et al. 2019 find downscaling the input image tends to preserve the loss,
T(x, 1) = St(x) = x/22,
(10)
where (1) scales the pixels of x and the number of predefined scales. In a similar fashion as TIM, the authors propose the Scale Invariance Method (SIM) to find perturbations that work for any downscaled version of the input. Note that this is different from DIM since SIM optimizes the perturbation for different scales at once while DIM applies different single resizing and padding at each gradient ascent step.
Resized Diverse Inputs (RDI). Zou et al. (2020) Introduce a variant of DIM where the inputs are resized back to the original size after applying DIM augmentations, i.c. Resized Diverse Inputs (RDI). Resizing the imputs allows them to test much more aggressive resizing and padding augmentations which boost performance. Similarly to SIM, they also propose to ensemble the gradients from different inputs, however, instead of multiple-scale images, they use RDI augmentations with varying strength. They also observe that keeping the attack optimizer step-size attacks.
constant, further improves the success rate of the
T(x)= = resize(resize(z, H/s, H/ss,Ws),,H
(11)
Adversarial Transformation Transfer Attack (ATTA). A common theme in previous methods has been that combining multiple image transformations (DIM + TIM + SIM) usually leads better results than using just one set of angmentations. However, all previous methods have focused on fixed set of augmentations, which limits the diversity of augmentations even if combined. Wu et al. (202T) introduce the Adversarial Transformation Transfer Attack where input transformations are parametrized by a network that is optimized alongside the adversarial perturbations to minimize the impact of adversarial attacks. Thus, improving the resilience of the final attacks to input perturbations.
T(x) (z), 0 arg min +
(12)
Li ct al. (020b observe that adversarial perturba- tions lend to have high regional homogeneity (i.e. gradients corresponding to neighboring pixels tend to be similar), especially when models are adversarially robust. Based on this observation they propose to ap- ply a parametrized normalization layer to the computed gradicnts that encourages Regionally Homogeneous Perturbations(RHP) Their objective can be written as:
Regionally Homogencous Perturbations (RHP).
ad = = e sign(
(13)
where 7' = (-(-) is a parametrized transformation on the gradients rather than the input. This transformation is optimized to maximize the loss of the perturbed sample addv Interestingly, they observe that when optimizing the normalization parameters on a large mumber of images, the generated perturbations converge to an input-independent perturbation. Hence, although not explicitly enforced, they find RHP leads to universal perturbation (i.e. perturbations that can fool the model for many different inputs).
Object-based Diverse Input (ODI). Motivated by the ability of humans to robustly recognize images even when projected over 3D objects (e.g. an image printed on a mug, or a t-shirt), Byun et al. (2022) present a method named Object-basee Diverse Input (ODI), variant of DIM which renders images on a set of 3D objects when computing the perturbations as a more powerful tochnique to perform data augmentation technique.
T(x) = O(,,O,
(14)
where IT is projection operation onto the surface of a 3D mesh and O represents the 3D object.
Admix. Inspired by the success of Mixup (training models with convex combinations of pairs of examples
and their labels) in the context of data augmentation for classification model training (Zhang et al. 2017).
Wang et al.] (2021a) study this technique the context of fostering transfcrability of adversarial example. However, they find that mixing the labels of two images significantly degrades the white-box performance of adversarial attacks and brings little improvement in terms of transferability. Hence, they present variation
6

---

Published in Transactions on Machine Learning Research (05/2024)
of Mixup (Admix) where small portion of different randomly sampled images is added to the original one, but the label remains unmodified. This increases the diversity of inputs, and thus improves transferability but does not harm white-box performance. In this case,
T(x) mut ra', where v < 1 and < 7/.
(15)
The blending parameters 7, are randomly sampled and the restriction < 17 ensures the resulting image does not differ too much from the original one. The additional images x' are randomly sampled from other categorics.
Transferable Attack based on Integrated Gradients (TAIG). Huang & Kong 2022 leverage the concept of integrated gradients (i.e. a line integral of the gradients between two images) introduced by [Sun- dararajan et al. (2017a). Instead of optimizing the attack based on the sign of the gradients (e.g. with FGSM) they the sign of the integraled gradients from the origin to the larget image. Moreover, they show that following a piece-wise linear random path improves results further. We can formalize their objective
follows:
T =
(16)
and [f,,,, x') is the integrated gradient between x and another image x'.
3.2
Optimization Technique- Based Transferability Enhancing Methods
The generation of transferable adversarial examples can be formulated as optimization problem of Equa- tion 6 In the last section, presented how the imput data augmentations influence the transferability of the created adversarial perturbations. In this section, we describe how the current work improves adversarial transferability from the perspective of the optimization technique itself.
In this section, we focus on perturbations constrained by an Loo ball with radius e, that is, ||ænd - x11300 S e. To understand the rest of this section, we begin by formalizing the itcrative variant of the fast gradient sign method (I-FGSM) (Goodfellow al. 2014p, which serves as the basis for the development of other methods. The I-FGSM has the following update rule:
(1+11) Vl(xade(tt) g).
adu((++1) = Clip (2 + a- sign(g
(17)
where (()) is the gradient of the loss function with respect the input, a denotes the step size at each iteration, and Clip,, ensures that the perturbation satisfies the norm constraints.
Momentum (MI-FGSM). One of the simplest and most widely used techniques improve the gener- alizability of neural networks is to incorporate momentum in the gradient direction (Polyak) 1964 Duch & Korczak 1998). Motivated by this, Dong et al. 2018) propose a momentum iterative fast gradient sign method (MI-FGSM) to improve the vanilla iterative FGSM methods by integrating the momentum term in the input gradient. At each iteration, MI-FGSM updates g(t+1) by using
V((wadv(() .3)
(+++) = (0) + [[Ve(zdd)() v)ll1
(18)
where (+)) now represents the accumulated gradients at itcration f, p denotes the docay factor of (0)), and the formulation for g.dd((++1) remains the same (17). By integrating the momentum term into the iterative attack process, MI-FGSM can help escape from poor local maxima, leading to higher transferability for adversarial examples.
More variants of I-FGSM have been proposed to make the created adversarial perturbation more transferable. Similar to the development of the DNN optimization techniques, the first-order moment, the second-order moment, and more components are integrated successively to improve adversarial transfcrability. Those variants include Nesterov (NI-FGSM) (Lin et al. 2019), Adam (AI-FGTM) (Zou al. 2022), and Variance Tuning (VNI/VMI-FGSM) (Wang & He 2021 ) the details of which can be found in Appendix
7

---

Published in Transactions on Machine Learning Research (05/2024)
Stochastic Variance Reduced Ensemble (SVRE). Xiong et al. (2022) propose a variance-tuning strat- egy to generate transferable adversarial attacks. Conventional ensemble-based approaches leverage gradients from multiple models to increase the transferability of the perturbation. Xiong et al. (2022) analogize this process as a stochastic gradient descent optimization process, in which the variance of the gradients on different models may lead to poor local optima. As such, they propose to reduce the variance of the gradient by using an additional iterative procodure to compute an unbiasod estimate of the gradient of the ensemble. SVRE can be generalized to any iterative gradient-based attack.
Gradient Relevance Attack (GRA). Zhu et al. 2023b) introduce GRA, an attack strategy built upon MI- FGSM and VMI-FGSM. This approach incorporates two key techniques to further increase transferability. First, during (88, the authors notice that the sign of the perturbation fluctuates frequently. Given the constant step size in the iterative process, this could suggest the optimization getting trapped in local optima. To address this, they introduced a decay indicator, adjusting the step size response to these fluctuations, thus refining the optimization procedure. Additionally, they argue that in VMI-FGSM, the luning strategy used during the last iteration might not accurately represent the loss variance at the current iteration. Therefore, they propose to use dot-product attention (Vaswani et al. 2017) to compute the gradient relevance between xadd(tt and its neighbors, providing more precise reprosentation of the regional information surrounding x zade(1), This relevance framework is then used to fine-tune the update direction.
Adaptive Gradient Method (AGM). While the previous methods focus on improving the optimization algorithm, another angle of attack is through the optimization objective. Li et al. (2020a) demonstrate that the cross-entropy loss, commonly utilized in iterative gradient- based methods, is not suitable for generating transferable perturbations in the targeted scenario. Analytically, they demonstrate the problem of vanishing gradients when computing the input gradient with respect to the cross-entropy loss. Although this problem is circumvented by projecting the gradient to unit & ball, they empirically show in the targeted setting that this normalized gradient is overpowered by the momentum accumulated from early iterations. Notice that because historical momentum dominates the update, the effect of the gradient at every itcration diminishos, and thus the update is not optimal in finding the direction loward the largeted class. To circumvent the vanishing gradient problem, they propose to incorporate the Poincaré metric into the loss function. They empirically demonstrate that the input gradient with rospect to the Poincaré metric can better capture the relative magnitude between the gradient magnitude and the distance from the perturbed data point to the target class, leading to more effective iterative updates during the attack process.
Reverse Adversarial Perturbation (RAP). Improving the adversarial robustness of neural networks by training with perturbed examples has been studied under the robust optimization framework, which is also known as a min-max optimization problem. Similarly, Qin al. (2022) propose to improve the transferability of adversarial examples under such a min-max bi-level optimization framework. They introduce RAP that encourages points in the vicinity of the perturbed data point to have similar high-loss values. Unlike the conventional I-FGSM formulation, the maximizaiinn procedure of RAP first finds perturbations with the goal of minimizing the loss, whereas the outer-minimization process updates the perturbed data point to find a new point added with the provided reverse perturbation that leads to a higher loss.
Momentum Integrated Gradients (MIG). The integrated gradient (IG) is a model-agnostic inter- protability method that attributions the prediction of a neural network to its inputs (Sundararajan et al., 2017b). The gradient derived from this method can be understood as saliency scores assigned to the input pixels. Notably, distinct feature of IG is its implementation invariance, meaning that the gradients only depend on the input and output of the model and are not affected by the model structure. Such a charac- teristic can be especially beneficial when improving the transferability of perturbation across different model architectures. Inspired by these, Ma et al. (0023) propose MIG which incorporates IG in the MI process.
3.3 Loss Objective-Basee Transferability Enhancing Methods
The loss objective used to create adversarial examples on the surrogate models the cross-cntropy loss in Equation 0 Various designs have also been explored to improve the transferability of the created adversarial examples.
8

---

Published in Transactions on Machine Learning Research (05/2024)
Normalized CE Loss. To increase the attack strength, Zhang et al. (2022a) identify that the weakness of the commonly used losses lies in prioritizing the speed to fool the network instead of maximizing its strength. With an intuitive interpretation of the logit gradient from the geometric perspective, they propose a now normalized CE loss that guides the logit to be updated in the direction of implicitly maximizing its rank distance from the ground-truth class. For boosting the lop-k strength, the loss function consists of two parts: the commonly used CE, and a normalization part, averaging the CE calculated for each class. The loss function they term Relative CE loss or RCE loss in short is formulated as follows:
RCE (sodu(t), v)
CE (2ado(t), .2)
Σ
CE
x
adoltt) ykk)
(19)
Their proposed RCE loss in the equation above achieved a strong top-k attack in both white-box and transferable black-box settings.
Generative Model as Regularization. Focusing on black-box attacks in restricted-access scenarios, Xiao ct al. (202)) propose to generate transferable adversarial patches (TAPs) to evaluate the robustness of face recognition models. Some adversarial attacks based on transferability show that certain adversarial examples of white-box alternative models g can be maintained adversarial to black-box target models f. Suppose g is a white-box face recognition model that accessible to the al Lacker, the optimization problem to generate the adversarial patch on the substitute model can be described as follows:
max Lg xi), s.t. x x (1 M) = x, @ (1-M),
(20)
where Cg is differentiable adversarial objective, O is the element- product, and M E {0, 1)" is a binary mask. However, when solving this optimization problem, even the sfatc-of-the-art optimization algorithms are still struggling to get rid of local optima with unsatisfactory transferability. To solve this optimization challenge, Xiao et al.] (2021) propose to optimize the adversarial patch on low-dimensional manifold as a regularization. Since the manifold poses specific structure on the optimization dynamics, they consider good manifold should have two properties: sufficient capacity and well regularized. In order to balance the requirements of capacity and regularity, they learn the manifold using generative model, which is pre- trained on natural human face data and can combine different face features by manipulating latent vectors to generate diverse, unseen faces, e.g., eye color, eyebrow thickness, etc. They propose to use the generative model to generate adversarial patches and optimize the patches by means of latent vectors. Thus the above optimization problem is improved as:
maxC, o (x, Xt), BES
s.t. x M) x, ① (1 (1-M),
x M = h(s) O
(21)
where the second constrain restricts the adversarial patch on the low-dimensional manifold represented by the generative model, h(s) : S -> Ip denote the pre-trained generative model and S is its latent space. When constrained on this manifold, the adversarial perturbations resemble face-like features. For different face recognition models, they expect that the responses to the face-like features are effectively related, which will improve the transferability of the adversarial patches.
Metric Learning Regularization. Most of the previous research on transferability has been conducted in non-targeted contexts. However, targeted adversarial examples are more difficult to transfer than non- targeted examples. Li et al. 2020aa find that there are two problems that can make it difficult to produce transferable examples: (1) During an iterative attack, the size of the gradient gradually decrcases, Icading to excessive consistency between two consecutive noises during momentum accumulation, which is referred to as noise curing. (2) Targeted adversarial examples must not only approach the target class but also stray from the ground-truth class. To overcome these two problems, they discard the Euclidean standard space and introduce Poincaré ball as a metric space for the first time to solve the noise curing problem, which makes the magnitude of gradient adaptive and the noise direction more flexible during the iterative attack
9

---

Published in Transactions on Machine Learning Research (05/2024)
process. Instead of the traditional cross-entropy loss, they propose Poincaré Distance Metric loss Lpos which makes the gradient increase only when it is close to the target class. Since all the points of the Poincaré ball are inside a n-dimensional unit lg ball, the distance between two points can be defined as:
d (u, v) arccosh (1 + 8 u)),
(22)
where u and are two points in n-dimensional Euclid space R" with &2 norm less than one, and 8 (2, u) is an isometric invariant defined as follow:
|/u 21l
8 8(u =
(23)
|u|| )1 ||]l3))
However, the fused logits are not satisfied [1(x=)]12 < 1, so they normalize the logits by the & distance. And they subtract the one hot target label from a small constant & = 0.0001 because the distance from any point to y is too. Thus, the final Poincar' e distance metric loss can be described as follows:
(po (x, = d (u, v) arccosh (1 + (u, u)),
(24)
where v ===x(<///|aa(>)))) == marly- -,,0-,,0),, E, and ((x) is the fused logits.
In targeted attacks, the loss function usually only considers the desired target label. But sometimes, the generated adversarial examples are too similar to the original class, causing some still be classified correctly by the target model. Therefore they also utilize the real label information as an addition during the iterative attack, using triplet loss, to help the adversarial examples stay away from the real labels to get better transferability. They use the logits of clean images (xteean), one-hot larget label and true label yhar, Vtrue as the triplet input:
Earip (yhar; Mx.), Utrue) = (D (1(z;), Uhar) - D (Maz), Utrue) + +7
(25)
Since the ((I is not normalized, so they use the angular distance D(-) as a distance metric, which eliminates the influence of the norm on the loss value. The distance calculation be described as follows:
D (1(x"do), Star) = 1 ((zadu). Ytar
(26)
[I Ila allittalll
Therefore, after adding the triplet loss term, their overall loss function:
lall = Epo (I(x), Star) + X Lerip (Ytar, Mxss, Mtrue).
(27)
Simple Logit Loss. Zhao et al. (0021 review transferable targeted attacks and find that their difficul- ties are overestimated due the blind spots in traditional evaluation procedures since current works have unreasonably restricted attack optimization to few iterations. Their study shows that with enough itera- tions, even conventional I-FGSM integrated with simple transfer methods can easily achieve high targeted transfcrability. They also demonstrate that attacks utilizing simple logit loss can further improve the tar- geted transferability by a very large margin, leading to results that are competitive with stale-of-the-art techniques. The simple logit loss can be expressed as:
= max (z)),,
(28)
where Z+(-) denotes the logit output before the softmax layer with respect to the target class.
Meta Loss. Instead of simply generating adversarial perturbations directly in these lasks, Fang el al. b optimize them using meta-learning concepts so that the perturbations can be better adapted to various conditions. The meta-lcarning method is implemented
each iteration of the perturbation update. In each iteration, they divide the task into support set S and query set Q, perform meta-training (training the support set) and meta-testing (finettuning on the query set) multiple times, and finally update the
10

---

Published in Transactions on Machine Learning Research (05/2024)
adversarial perturbations. In each meta-learning iteration, they sample subset S E S and calculate the average gradient with respect to input as:
Japt
Σ (z),)
(29)
S (*.:7,)ES
where G denotes the gradient updation of adversarial perturbations as in I-FGSM: G(æadur f) V Vaca ((((dd))): y). The = I,, 72,....?1ll E [0. denotes the set of decay factors during model aug- mentation, and the factor "/ defaults for i-th residual layer. Since the optimization of represents the augmentation of the model f, for ease of writing, they replaced 7 with f. ie, G(Eodur 7a) = G( f) = V Vando [[f (.dd)) y)-
Then, similar to FGSM, they obtain the temporary perturbation with single-step update:
8' sign(gspt)-
(30)
Then they finetune the query set Q and compute the query gradient gory by adding the temporary perturbation so that it can adapt more tasks:
Jury |
Σ
G(ary + 8', 7g)-
(31)
(.):7))E
Finally, they update the actual adversarial perturbation with the gradient from both the support set and the query set for maximum utilization:
I1TH0 (x + a..asa sign (gspt 9wry)
(32)
adv
where Jspt and Tery denote the average gradient over meta-lcarning itcrations, respectively.
Domain transferability Through Regularization. Xu et al. (2022b) propose theoretical framework to analyze the sufficient conditions for domain transferability from the view of function class regularization. They prove that shrinking the function class of feature extractors during training monotonically decreases a tight upper bound on the relative domain transferability loss. Therefore, it reasonable to expect that imposing regularization on the feature extractor during training can lead to better relative domain transfer- ability.
More Bayesian Attack. Many existing works enhance attack transferability by increasing the diversity in inputs of some substitute models. Li et al. (023) propose attack Bayesian model for achieving desirable transferability. By introducing probability measures for the weights and biases of the alternative models, all these parameters can be represented under the assumption of some to-be-learned distribution. In this way, an infinite number of ensembles of DNNs (which appear to be jointly trained) can be obtained in a single training session. And then by maximizing the average predictive loss of this model distribution, adversarial examples are produced, which referred as the posterior learned in Bayesian manner. The atlacks performing on the ensemble of the set of M models can be formulated as:
arg min ||x|| M
1 Eplux E
x arg max | Se M
1
Σ I( Ax, s.t. we p(w D), i=1
(33)
where L(-,, w.) is function evaluating prediction loss of DNN model parameterized by w, Using iterative optimization methods, different sets of models can be sampled at different itcration stages, as if an infinite number of substitute models existed.
Lightweight Ensemble Attack (LEA). Qian et al. 2023 notice three models with oon-overlag vulnerable frequency regions that can cover sufficiently large vulnerable subspaco. Based on this finding, they propose LEA2, a lightweight ensemble adversarial attack consisting of standard, weakly robust, and robust. models. Furthermore, they analyze Gaussian noise from frequency view and find that Gaussian
11

---

Published in Transactions on Machine Learning Research (05/2024)
noise occurs in the vulnerable frequency regions of standard models. As result, they replace traditional models with Gaussian noise to ensure that high-frequency vulnerable regions are used while lowering attack time consumption. They first define the vulnerable frequency regions and the adversarial example x' is generated by the following optimization:
arg max log Σ Syouutt (a tr + 00 + Σ w/ S weak (x (x+
(34)
where r (N, (2) is the Gaussian noise, M1 and M2 are the number of robust models and weakly ro- bust models respectively, Srobust and Sweak represent the corresponding softmax outputs, ly is the one-hot encoding of y, and Ei= + - 1. Together with the constraints of || xl1.0 S e, their adversarial examples updating process can be described as follows:
2/411 = Clip,,e {24 + a sign (Vol (x(,))11)
(35)
M2
C (z4,1v)
Σ w.F (brobust (z)
Σ wyL (N (z()
(36)
where the brobuet and h represent robust models and weakly robust models respestively, w is the core- sponding ensemble weights.
Adaptive Model Ensemble Attack (AdaEA). Existing ensemble attacks simply fuse the outputs of agent models uniformly, and thus do not effectively capture and amplify the intrinsic transfer information of the adversarial examples. Chen et al. (2023a) propose AdaEA that adaptively controls the fusion of each model's output, via monitoring the discrepancy ratio of their contributions towards the adversarial objective, Then, an extra disparity-reducer filter is introduced to further synchronize the update direction. The basic idea of an ensemble attack is to utilize the output of multiple white box models to obtain an average model loss and then apply a gradient-based attack to generate adversarial examples. In their work, they propose AdaEA equipped with adaptive gradient modulation (AGM) and a disparity-reduced filter (DRF) to amend the gradient optimization process for boosting the transferable information in the generated adversarial examples. In detail, The AGM strategy can adaptively combine the outputs of each model through an adversarial ratio, thus increasing the strength of the transferable information in the generated adversarial examples. The DRF can decrease the differences weew the agent models by calculating a discrepancy map and synchronizing the update direction. The process of AdaEA can be represented in short by the following equation:
odv Clip, { (agdo a sign (gf
(37)
K
Σ wiih (zgdi) .11 8 B,
ens 9++1
V agd L
(38)
where the g represents the ensemble gradient of models, the B represents a filter that can clean the disparity part in the ensemble gradient, and & denotes the element-wise multiplication.
3.4 Model Component-Band Based Transferability Enhancing Methods
In this subsection, introduce the transferabllty--ehnnii approaches based various model compo- nents. Typically, the components are from the surrogate model in Equation 6
Features. Several methods have aimed to improve transfer attacks, which focus on considering the feature space of the source model to generate noise that less overfits the specific architecture. Zhoo et al. (2018) first demonstrated that maximizing the distance of intermediate feature maps between clean images and
adversarial examples can enhance the transfer attack across models. They introduced two additional penalty terms in the loss function to efficiently guide the search directions for traditional untargeted attacks. The optimization problem can be described as follows:
adv arg max I[adoo 1) ||z=zado||?
[[([(r T(L(aado d))l dED
Σ
(a)(aa +
E absR (2 x, Ws),
(39)
12

---

Published in Transactions on Machine Learning Research (05/2024)
where the first term represents the traditional untargeted attack loss, and L(x, d) denotes the intermediate feature map in layer d E D. Here, T(L(1,))) ()) signifies the power normalization (Perronnin et al. 2010 of L(r, d). The regularization serves as a form of low-pass filter, enforcing the continuity of neighboring pixels and reducing the variations of adversarial perturbations. Nasecr et al. (2018) and Hashomi et al. (2022) followed this idea and generated adversarial examples that exhibited transferability across different network architectures and various vision tasks, including image segmentation, classification, and object detection.
Huang et al. 2019a) proposed the Intermediate Level Attack (ILA), which involves fine-tuning an existing adversarial example by magnifying the impact of the perturbation on a pre-specified layer of the source model. Given an adversarial example x' generated by basclinc attack, it serves as a hint. ILA aims to find a x" such that the optimized direction matches the direction of Ayí = Fi(²²) Fx(x) while maximizing the norm of the disturbance in this direction 43/" = =i(²") - Fi()). Within this framework, they propose two variants, ILAA and ILAFF The ILAP simply adopts the dot product for the maximization problem, and the ILAF augments the losses by separating out the maintenance of the adversarial direction from the magnitude and controls the trade-off with the additional parameter a.
CILAP (vls v()) -Ayk Avi"
(40)
LILAF (Nl.vl' = a Av'll2
AvA' 1
Avi A
(41)
A
A
Salzmann et al. (2021) introduced a transferable adversarial perturbation generator that employs a feature separation loss, with the objective of maximizing the L2 distance between the normal feature map f(())) and the adversarial feature map (indde) at layer l. This is defined as:
featt (2i, rdd = [[)((z))) - f1
(42)
The above methods usually trap into a local optimum and tend to overfit to the source model by indiscrim- inately distorting features, without considering the intrinsic characteristies of the images. To overcome this limitation, Ganeshan et al. (2019) proposed the Feature Disruptive Attack (FDA), which disrupts features at every layer of the network and causes deep features to be highly corrupt. For given 7th layer 4, they increase the layer objective L:
C (1.)) = D ((I(TON,, $ - NN E.)),
(43)
where 4()),, denotes the N2th value of 4()), S; denotes the set of activations that contribute to the current prediction. While this set is not straightforward to find, it can be approximated using a measure of central tendency, such as the median or the suter-uartill-mean D monotonically increasing function of activations (F). They perform it at each non-linearity in the network and combine the per-layer objectives for the goal. FDA treats all neurons as important neurons by differentiating the polarity of neuron importance by mean activation values.
In contrast, Wang et al. 2021c proposed a Feature Importance-aware Attack (FIA) to improve the trans- fcrability of adversarial examples by disrupting the critical object-aware features that dominate the decision of different models. FIA leverages feature importance, obtained by averaging the gradients with respect to feature maps from the source model, to guide the search for adversarial examples.
Zhang et al. 2022c introduced the Neuron Attribution-base Attack (NAA), which is feature-level attack based on more accurate neuron importance estimations. NAA altributes the model's output to each neuron and devises an approximation scheme for neuron attribution, significantly reducing the computation cost. Subscquently, NAA minimizes the weighted combination of positive and negative ncuron attribution values to generate transferable adversarial examples.
Wu et al. (2020b) proposed to alleviate overfitting through model attention. They consider an entire feature map as fundamental feature detector and approximate the importance of feature map Ag (the e-th feature map in layer k) to class t with spatially pooled gradients:
(x)
af =
(44)
7
A m n|
m
13

---

Published in Transactions on Machine Learning Research (05/2024)
They scale different feature maps with corresponding model attention weights af[t] and perform channel-wise summation of all feature maps in the same layer. Then derive the attention map for the label prediction as follows:
H = ReLU
( af -4.) A4)
(45)
Finally, they combine the original goal which aims to mislead the final decision of the target model, and the attention goal which aims to destroy the vital intermediate features.
mag 8
C (S (xxdd) + > Σ HH (xede) - H
(46)
The above transfer attack methods are only for un-targeted attacks. Rozsa et al. (017)) and Inkawhich et al. (2019) first describe transfer-based targeted adversarial attack that manipulates feature space rep- resentations to resemble those of target image. The Activation Attack (AA) loss is defined to make the source image I, closer to image of the target class I, in feature space.
JAA (4,, 1.) = 115z (4e) - fr (4.)l2 | 14f--11122 A4112:
(47)
where JAA is the Euclidean distance between the vectorized source image activations and vectorized target image activations at layer L, and fr. be truncated version of a white-box model fw- However, this method is challenging to scale to larger models and datasets due to the lack of modeling for the target class and its sensitivity to the chosen target sample.
Inkawhich et al. (2020b al propose to model the class-wise ura distributions at multiple layers of the white- box model, aiming for a more comprehensive representation of the target class in targeted attacks. Initially, they modeled the feature distribution of DNN using an auxiliary Neural Network gt.e to learn p(y = (x)), which represents the probability that the features of layer 1 of the white-box model, extracted from input image z, belong to class Subsequently, the attack employed these learned distributions to generate targeted adversarial examples by maximizing the probability that the adversarial example originates from a specific class's feature distribution. Additionally, they developed a flexible framework that could extend from single layer to multiple layers, emphasizing the explainability and interpretability of the attacking process.
Some other works have also explored the properties of adversarial examples in the feature space. Wang et al. (2021b) discovered a negative correlation between transferability and perturbation interaction units and provided a new perspective to understand the transferability of adversarial perturbations. They demon- strated that multi-stop attacks tend to generate adversarial perturbations with significant interactions, while classical moth of enhancing transferability essentially decrease interactions between perturbation units. Therefore, they proposed a new loss to directly penalize interactions between perturbation units during an attack, significantly improving the transfcrability of previous methods.
Waseda el al. (2023) demonstrated that adversarial examples tend to cause the same mistakes for non- robust features. Different mistakes could also occur between similar models regardless of the perturbation size. Both different and the same mistakes can be explained by non-robust features, providing novel insight into developing transfer adversarial examples based on non-robust features.
Batch Normalization (BN). Benz et al. (2021) | investigate the effect of BN on deep neural networks from a non-robust feature perspective. Their empirical findings suggest that BN causes the model to rely more heav- ily on non-robust features and increases susceptibility to adversarial attacks. Further, they demonstrate that a substitute model trained without BN outperforms its BN-cquipped counterpart and that carly-stopping the training of the substitute model can also boost transferable attacks
Skip Connections. Wu et al. (2000a) find that skip connections facilitate the generation of highly trans- ferable adversarial examples. Thus, they introduced the Skip Gradient Method (SGM), which involves using more gradients from skip connections rather than residual ones by applying a decay factor on gradients. Combined with existing techniques, SGM can drastically boost state-of-the-art transferability.
ReLU activation Guo et al. (2020) propose to boost transfcrability by enhancing the lincarity of deep neural networks in an appropriate manner. To achieve this goal, they propose simple yet very effective method technique dubbed linear backpropagation (LinBP), which performs backpropagation in more linear
14

---

Published in Transactions on Machine Learning Research (05/2024)
fashion using off-the-shelf attacks that exploit gradients. Specifically, LinBP computes the forward pass normal but backpropagates the loss linearly as if there is no ReLU activation encountered.
Patch Representation. Naseer et al. (2022) propose the Self-Ensemble (SE) method to find multiple discriminative pathways by dissecting a single ViT model into an ensemble of networks. They also introduce a Token Refinement (TR) module to refine the tokens and enhance the discriminative capacity at each block of ViT. While this method shows promising performance, it has limited applicability since many VIT models lack enough class tokens for building ensemble, and TR requires is time-consuming. Wei et al. (2018) find that ignoring the gradients of attention units and perturbing only a subset of the patches at each iteration can prevent overfitting and create diverse input patterns, thereby increasing transfcrability. They propose a dual attack framework consisting of a Pay No Attention attack and a PatchOut attack to improve the transferability of adversarial samples across different ViTs.
Ensemble of Models. Liu et al. (2017) first propose transferable generating adversarial examples by utilizing an ensemble of multiple models with varying architectures, Gubri et al. (2022b) presents a geometric approach to enhance the transferability of black-box box adversarial attacks by introducing Large Geometric Vicinity (LGV). LGV constructs a surrogate model by collecting weights along the SGD trajectory with high constant learning rates, starting from conventionally trained deep neural network. Gubri et al. (2022a) develop a highly efficient method for constructing a surrogate based on state-of-the-at Bayesian Deep Learning techniques. Their approach involves approximating the posterior distribution of neural network weights, which represents the belief about the value of each parameter. Similarly, Li et al. (2023) adopt a Bayesian formulation in their method to develop a principled strategy for possible fine-tuning, which can be combined with various off-the-shelf Gaussian posterior approximations over deep neural network parameters. Huang et al. (2023) focus on the single-model transfer-based black-box attack in object detection. They propose an enhanced attack framework by making slight adjustments to its training strategies and draw an analogy between patch optimization and regular model optimization. In addition, they propose series of self-ensemble approaches on the input data, the attacked model, and the adversarial patch to efficiently utilize the limited information and prevent the patch from overfitting.
4 Generation-Base Transferable Attacks
Optimization-baase adversarial attacks discussed in the previous use gradients from surrogate models to iteratively optimize bounded perturbations for each clean image at test time. In this section, we introduce generation-based transferabllit--enancii by directly synthesizing the adversarial example (or the adversarial perturbation) with generative models. Generation-ben attacks comprise two stages: Training and the attack stages. During the training stage, the attacker trains a generative model Go(-), a function parameterized by that outputs either the adver- sarial example zadv or an adversarial perturbation 6. The optimization of the generator parameters can be formulated follows:
methods. This class of methods takes an alternative approach
max (e 0
(48)
where fa()) is a surrogate model, and G-(-) directly generates the adversarial example. If the generator predicts the perturbation 8 instead, the loss becomes (9f(( + v)) In the case of targeted attacks, the optimization is described as:
min E (O...
(49)
Note that a surrogate model fa()) involved the first step of the generative model-based attack. During the attack stage, adversarial examples are obtained directly with single forward inference of the learned generator Go(.).
The input to the generator varies depending on the problem formulation. For input-dependent Pour- saeed et al. 2018 Naseer et al. 2019) adversarial perturbation generation, where the goal is to generate a perturbation specific to the given input x, we have:
adv ()
(50)
15

---

Published in Transactions on Machine Learning Research (05/2024)
where any smoothing operations additive and clipping operations can be absorbed into the mapping g
(A (Alternatively, we can generate the perturbation instead of the æddv that is 5 = Go(x)). For universal adversarial perturbations (Poursaeed et al. 2018), where the perturbation can be added to any we input
a fixed noise & to the generator function:
== -=(2
(51)
Generative models are believed to possess several properties that can help achieve improved imperceptibility and transferability. Firstly, only one single model forward pass is required at test time once the generator is trained. This avoids the costly iterative perturbation process and thus, allows highly efficient adversarial attacks to be performed in an online fashion. Secondly, generators are less reliant on class- boundary infor- mation from the surrogate classifier since they can model latent distributions (Naseer et al., 2021 Finally, generative models provide latent spaces from which perturbations can be injected. This enables the search for adversarics in the domer-dimensional latent space rather than directly within the imput data space, resulting in smoother perturbations with improved photorealism, diversity, and imperceptibilitt.
4.1 Methods Based on Unconditional Generation
Generative Adversarial Pertruabtions (GAP) Poursaeed et al. (0018) introduce generative models to the task of adversarial sample synthcsis. In this work, the gencrator generates the perturbation (6 = (o))) as opposed to generating the _dds They investigate two different variations: generating input-dependent adversarial perturbations and universal adversarial perturbations. For the former, they use
max Ee ((sCC(())) + x),
(52)
as generator training objective where is defined as the cross entropy loss. They also investigate generating universal perturbations, where the perturbation can be added to any input image. In this case, the generator is given a fixed noise z as input 51 Thus, the optimization term becomes:
max E (I. +2,
(53)
Their work demonstrates the high efficiency of learned generators for creating both targeted and untargeted adversarial examples.
AdvGAN, Xiao et al. (0018) proposed to incorporate adversarial training for the generator by introducing a discriminator D& and solving min-max game (equation 54). In addition to equation 48 they incorporated
a GAN loss to promote the realism of synthesized samples and a soft hinge loss on the L22 norm, where e denotes a user-specified perturbation budget.
min max (m) (I(f(Go(1)) + log (1 - (x(())) + log (Do(Go(>))) - max(00 - -c)
(54)
Cross Domain Adversarial Perturbation. Nascer et al. (209)) investigate the usage of generative models in generating adversarial attacks that transfer across different input domains. They propose to use relativistic loss in the generator training objective to enhance the transferability of cross-domain targeted attacks. The relativistic cross entropy (equation 55) objective is believed to provide a "contrastive" supervision signal that is agnostic to the underlying data distribution and hence achieves superior cross-domain transfcrability.
L = (E(f,()) - ( f.(((()))
(55)
Distribution and Neighbourhood Similarity Matching. To achieve good transferability for cross- domain largeted attacks, Naseer el al. (2021) propose a novel objective that considers both global distribution matching as well as sample local neighborhood structures. In addition to solving the optimization problem in equation 48 they propose to add two loss terms (1) one that minimizes the (scaled) Jensen-Shannon Divergence between the distribution of perturbed adversarial samples from the source domain (=)( and the distribution of real samples from the target class in the target domain p¹ (xlux); (2) a second term
16

---

Published in Transactions on Machine Learning Research (05/2024)
that aligns source and target similarity matrices S and S defined as Sig == (al)-fla.t))) which serves the purpose of matching the local structures based on neighborhood connectivity. /
(<)))/(<))
and Si ==
Laug = (KI(p" ((oo((((( + Prule (xll))
(56)
Sui = Dxx
(57)
Estentive-Diversit Attack (ADA). Kim ct al. 2022 propose method that stochasticly perturbs various salient features to enhance adversarial sample transferability. By manipulating image attention, their method is able to disrupt common features shared by different models and hence achieve better transferability. Their generator takes an image along with a random latent code ~ as input Ga(a, z). They propose two losses in addition to the classification loss in equation 48 (1) Latts that maximizes the distance between the attention maps of the original and the adversarial images for class-specific feature disruption and (2) Laie that promotes samples diversity by encouraging the generator to exploit the information in the latent code. They also argue that the stochasticity in 2 can help circumvent poor local optima and extend the
search space for adversarial samples.
Lattn |A(GG((, 2)) A(x)]12
(58)
AA(9 (I1, (111221) 21) ( 22))12
Lase
(59)
|| 22l
Conditional Adversarial Distribution (CAD) Feng et al. (2022 propose a treusferability-enaaadn approach that emphasizes robustness against surrogate biases. They propose to transfer subset of pa-
rameters based on CAD (i.e., the distribution of adversarial perturbations conditioned on benign examples) of surrogate models and learn the remainder of parameters based on queries to the target model while dynamically adjusting the CAD of the target model on new benign samples.
Model Discrepancy Minimisation. Zhao el al. 20223 propose an objective based on the hypothesis discrepancy principle that can be used to synthesize robust and transfcrable targeted adversarial examples with multiple surrogate models. In an adversarial training fashion, they jointly optimize the generator and the surrogate models (used as discriminators) to minimize the maximum model discrepancy (M3D) between surrogate models (equation 60 transform the image into a target class (equation 61 while maintaining the quality of surrogate models to provide accurate classification results on the original images (equation 62
max La Eewxdlfi o fa fi-fa
(60)
min fa B XPn (o) Go(
(61)
max L. En (E) () fifa
(62)
4.2 Methods Based on
Generation
Early generative targeted attack methods (Poursaced et al. 2018 Naseer et al. 2021 suffer from low parameter efficiency as they require training separate generator for each class. To address this issue, various approaches have been proposed to construct conditional generative models that handle targeted attacks of different classes with single unified model. While many different actualizations exist, these methods share the same mathematical formulation:
min (Ao
(63)
Conditional Generators. Yang et al. (0022) propose Conditional Generative model for a targeted attack, which can craft a strong Semantic Pattern (CG-SP). Coneretely, the target class information was processed through network before being taken as the condition of the generator MMirza & Osindero 2014). Claiming
17

---

Published in Transactions on Machine Learning Research (05/2024)
that it is difficult for single generator to learn distributions of all target classes, C-GSP divided all classes into a feasible number of subsets. Namely, only one generator is used for subset of classes instead of each.
Various ways to inject the condition into the synthesis process have been explored. For example, some authors propose to add trainable embeddings (Han et al. 2019 that can add target class information to the input tensor. In a similar spirit, GAP++ (Mao et al. 2020) extends GAP by taking target class encodings as model input and thereby only requires one model for both targeted and untargeted attacks. Multi-target Adversarial Network (MAN) (Han et al. 2019 propose method that enables multi-target adversarial attacks with a single model by incorporating category information into the intermediate features. To further improve the adversarial transferability, Phan et al. (2020) propose Content-Aware adversarial attack Generator (CAG) to integrate class activation maps (CAMs) (Zhou et al. 2016) information into the input, making adversarial noise more focused on objects.
Diffusion Models. Recent works have started investigate the usage of diffusion models for enhancing adversarial transferability. DiffAttack (Chen et al.) 2023b) is the first adversarial attack based on diffusion models (Ho et al.) 2020), whose properties can help achieve imperceptibility. Concretely, the perturbations were optimized in the latent space after encoder and DDIM (Ho et al. 2020). utilized in the loss function to distract attention from the labelled objects and disrupt the semantic relation- ship. Besides, self-attention maps are used for imperceptibility. keeping the original structure of images. Tn a similar spirit, AdvDiffuser (Chen et al. 2023d and Adversarial Content Attack (ACA) (Chen et al. 2024 also leverage pre-trained diffuusion models to craft highly transferable unrestricted adversarial examples.
maps are
5 Adversary Transferability Beyond Image Classification
In this section, we present transfer attacks beyond image classification tasks, such as varions vision tasks and NLP tasks. Furthermore, the transferability across lasks is also summarized.
5.1 Transfer Attacks in Vision Tasks
Image Retrieval. Previous works (Yang et al., 2018 Tolias et al. 2019) have shown that image retrieval is also vulnerable to adversarial examples. Xiao & Wang (2021) explore the transferability of adversarial examples in image retricval. In detail, they establish relationship between the transferability of adversarial examples and the adversarial subspace by using random noise proxy. Then, they propose an adversarial attack method to generate highly transferable adversarial examples by being both adversarial and robust to noise. Xiao & Wang (2021) point out the relationship between adversarial subspace and black-box transfer- ability. They propose to use additive Gaussian noise to estimate the generated adversarial region, thereby identifying adversarial perturbations that are both transferable and robust. to additive noise corruption.
Object Detection. Wei et al. (2018) find that existing image object detection attack methods suffer from weak transferability, i.e., the generated adversarial examples usually have an attack success rate in attacking oth detection methods. Then they propose a generative attack met to enhance the transferability of adversarial examples by using the feature maps extracted by a feature network. Specifically, they adopt the Generative Adversarial Network (GAN) framework, which is trained by the high-level class loss and low-level feature loss. Cai et al. (2022b) propose an object detection attack approach to generate context-aware attacks for object detectors. Specifically, they adopt the co-occurrence of objects, their relative locations, and sizes as context information to generate highly transferable adversarial examples. Moreover, Staff et al. (2021) explore the impact of transfer attacks on object detection. Specifically, they conduct objectness gradient attacks on the advanced object detector, é.e., YOLO V3. Then, they find increasing attack strength can significantly enhance the transferability of adversarial examples. They also study the transferability when the datasets for the attacking and larget models intersect. They find the size of the intersection has a direct relationship with the transfer attack performance. Additionally, Cai et al. (2022a) have indicated that existing adversarial attacks could not effectively attack the context-aware object detectors. To address that, They propose a zero-query context-aware attack method that can generate highly transferable adversarial scenes to fool context-aware object detectors effectively.
18

---

Published in Transactions on Machine Learning Research (05/2024)
Segmentation. Gu et al. b2021b explore the transferability of adversarial examples on image segmentation models. In detail, they investigate the overfitting phenomenon of adversarial examples on both classification and segmentation models and propose a simple yet effective attack method with imput diversity to generate highly transferable adversarial examples for segmentation models. Hendrik Metzen et al. (2017) explore the transferability of adversarial examples to attack the model of semantic image segmentation by generating universal adversarial perturbations. Specifically, they propose method to generate universal adversarial examples that can change the semantic segmentation of images in arbitrary ways. The proposed adversarial perturbations are optimized on the whole training set.
3D Tasks, Previous works (Niang et al. 2019: Zhou et al. 2019 Tsai et al. 2020 have developed several adversarial attack methods for 3D point clouds. Hamdi et al. (020) discover that existing adversarial attacks of 3D point clouds lack transferability across different networks. Then, they propose an effective 3D point cloud adversarial attack method that takes advantage of the input data distribution by including an adversarial loss in the objective following Auto-Encoder reconstruction. [Pestana et al. (2022) study the transferability of 3D adversarial examples generated by 3D adversarial textures and propose to use end-to- end optimization for the generation of adversarial textures for 3D models. Specifically, they adopt neural rendering to generate the adversarial texture and ensemble non-robust and robust models to improve the transferability of adversarial examples.
Person Re-Identiffcation. Previous works (Gou et al.) 2016 Xue et al. 2018 Huang et al. 2019b)
have indicated that person re-identification (ReID), which inherits the vulnerability of deep neural networks (DNNs), can be fooled by adversarial examples Wang al. (2020) explore the transferability of adversarial examples on ReID systems. Specifically, they propose learning-to-mis-te method to generate adversarial examples. They also adopt a multi-stage network to improve the transferability of adversarial examples by extracting transferable features.
Face Recognition. Jia et al. (022a) have indicated that previous face recognition adversarial attack methods rely on generating adversarial examples on pixels, which limits the transferability of adversarial examples. Then, they propose a unified, flexible adversarial attack method, which generates adversarial for perturbations of different attributes based on target-specific face recognition features to boost the attack transferability.
Video Classification. Wei et al. (2022a) have found that existing video attack met have only lim- ited transferability. Then, they propose a transferable adversarial attack method based on the temporal translation of the vidco, which generates adversarial perturbations over temporally translated video clips to enhance the attack transferability.
5.2 Transfer Attacks in NLP Tasks.
Yuan et al. (2020) introduce a comprehensive investigation into the transferability of adversarial examples for text classification models. In detail, they thoroughly study the impact of different factors, such as network architocture, on the transfcrability of adversarial examples. Morcover, they propose to adopt a generic algorithm to discover an ensemble of models capable of generating adversarial examples with high transferability. Furthermore, He et al. (2021) demonstrate the ability of an adversary to compromise BERT- based API service. With the available model, they can generate highly transferable adversarial examples. Wang et al. 2022) show that the adversarial examples are also transferable across the topic models, which are important statistical models. To further improve the transferability, they propose to use a generator to generate effective adversarial examples and an ensemble method, which finds the optimal model ensemble.
With the rise of large language models (LLM) like BERT (Kenton & Toutanova 2019), GPT (Brown et al, 2020b), and their variants (Li et al. 2019), the adversarial examples them have also received attention. Recently, Zou et al.] (2023) have demonstrated it is possible to induce aligned language models to generate inappropriate content, dubbed jailbreak allack. They also propose a simple way to make the jailbreak attack more transferable to other LLMs. Specifically, a jailbreak attack aims to maximize the likelihood of the language model generating an affirmative response instcad of declining to answer. They implement the attack by identifying a suffix that, when appended to various queries given to a language model, encourages generating undesirable content. They improve the transferability by attacking multiple surrogate LLMs with
19

---

Published in Transactions on Machine Learning Research (05/2024)
single suffix. Zou et al. (2023) show that the adversarial sufix is even transferable several mainstream close-sourced language models, e.g. ChatGPT (Achiam et al. 2023
5.3 Cross- Task Transfer Attacks.
Nascer et al. (2018) propose a novel adversarial attack method, which adopts the neural representation distor- tion to generate adversarial examples. They have demonstrated the remarkable transferability of adversarial examples across different neural network architectures, datasets, and tasks. Naseer et al. (2019) propose novel concept of domain-invariant adversaries, which demonstrates the existence of shared adversarial space among different datasets and models. They introduce new generative framework that creates strong adversarial examples with relativistic discriminator, outperforming traditional instance-specific attacks with universal adversarial function. Morcover, they propose to exploit the adversarial patterns capable of deceiving nelworks trained on completely different domains to improve atlack I transferability. [u el al. (2020) investigate the transferability of adversarial examples across diverse computer vision tasks, which include object detection, image classification, semantic segmentation, ctc. They propose a Dispersion Re- duction (DR) adversarial attack method which minimizes the standard deviation of intermediate feature maps to disturb features that are used by models intended for various tasks. WWi al. (2022b) study the transferability of adversarial perturbations across different modalities. In detail, they apply the adversarial examples on white-box image-based based models to attacking black-box ridco-based models by exploiting the similarity of low-level feature spaces between images and video frames. Naseer et. al. (0223) propose to adopt task-spocific prompts to incorporate spatial (image) and temporal (video) cues into the same source model, which can enhance the transferability of attacks from image-to-vided and image-lo-image models. They propose method to add dynamic cues to pre-trained image models through a simple rised-based transformation. Lu et al. (2023) study the adversarial transferability of some vision-language pre-training models. They propose set-level guidance adversarial attack to improve the trausferability of adversarial examples vision-language pre-training models, which makes full use of cross-modal guidance. Han et al. (2023a) propose adopting optimal transport optimization to enhance the adversarial transferability of vision- language models, which uses optima transmission theory to find the most effective mapping between image and lext features. Hu el al. (2024) propose to attack intermediate features of an encoder pre-trained on vision-language data for cross task transferability. They adopt a patch-wise approach that independently diverts the representation of each adversarial image patch from the corresponding clean one by minimising the cosine similarity between the two, thereby producing highly transferable adversarics that fool various vision-language understanding tasks.
6 Challenges, Opportunities and Connections to Broader Topics
This section delves into the intricacies of the challenges and illustrates the promising avenues for better Haanserabillty-bbsee attacks, and their evaluation and understanding.
6.1 Challenges and Opportunities
Adversarial Transferability is Far from Perfect. Adversarial examples are far from achieving perfection when transferred other models due to several inherent challenges. First, the performance of adversarial transferability tends degrade when evaluated on a variety of neural network architectures, highlighting the inconsistency in transferability across different models (Yu et al. 2023). Secondly, the task of transferring the adversarial perturbations created by targeted adversarial attacks remains challenging. Misleading to specific class is much more difficult than a simple fool in the case of adversarial transferability (Yu ct al. 2023). Finally, the current models with predefined classes. The current lision-language models (Lu et al.) 2023 Radford et al. 2021b Alayrac et al. 2022 Chen et al.] 2023c), which extract visual information from distinct perspective, pose unique challenges for transferability. These issues indicate that more hansferabllity-nnnnnee methods should be explored for better defense evaluation.
methods are mainly developed to larget visual classi in
20

---

Published in Transactions on Machine Learning Research (05/2024)
Natural, Unrestricted and Non-Additive Attacks. Albeit out of the scope of this survey, note the alternative, relaxed definition of adversarial attacks does exist. Adversarial perturbation does not need to be constrained by any norm-ball Hendrycks et al. 20021 ; Zhao et al. 2017)) and can be constructed through mcans other than additive noise (o.g. through transformations) (Brown ct al. 2018 Several studics have explored the transferability of natural adversarial attacks Chen el al. (2023d unconstrained (unrestricted)
adversarial attacks (Liu et al. 2023: Chen et al. 2023b Gao et al., 2022) and non-additive attacks (Zhang et al. 2020 Nonetheless, the community has not yet reached a consensus on how effectively evaluate such attacks. For example, perceptual metrics other than Ly distances may be required to evaluate stealthiness.
Source Model for Better Transfcrability. Current transferability methods are typically post hoc ap- proaches that involve enhancing the ability of adversarial examples generated on pre-trained models to deceive other models. When considering the source model, the question arises: How can we train a model to improve the transferability of adversarial examples created on them? For instance, one promising avenue for achieving this is to learn the model from the perspoctive of knowledge transferability. A model with high knowledge transferability inherently becomes a superior source model, adversarial examples generated on it exhibit a greater capacity to successfully deceive other models (Liang et al. 2021 Xu et al. 2022b). A follow-up question is which model architectures transfer better to others, CNNs, Vision Transformers Nasscrr
ct al.
2022 Wu et al. 2021 Ma et al. 2023 Wu et al. 2022). Capsule Networks
Gu et al. 2021a). or Spiking
Neural Networks (Xu et al. 2022a
Relation to Transferability Across Image Samples. In this work, we focus on adversarial transferability across models, namely, the ability of an adversarial perturbation crafted for one model or set of data to successfully fool another model. The community has also found that adversarial perturbation that effectively fools one image can also be applied to different image to achieve a similar adversarial effect, which is referred to as adversarial transferability across image samples (i.e. Universal Adversarial Image) (Moosavi- Dezfooli et al.) 2017). Understanding the interplay between transferability across images and transferability across models is essential for comprchending the broader landscape of adversarial attacks and defences. These two dimensions logether define the versalility and robustness of adversarial perturbations.
Theoretical Perspectives on Adversarial Transferability. The root causes behind transferability receive continued rescarch interest. [Dmontis et al. (2019) examine the effect of two factors on attack transferability: the intrinsic adversarial vulnerability of the larget model and the complexity of the surrogate model. [lyas et al. (2019) attributes adversarial transferability to the presence of non-robust features and points out the potential misalignment between robustness and inherent data geometry. Waseda et al. (2023) extends the theory of non-robust features by examining "class-aware transferability". In particular, they differentiate between the cases which a. target model predicts the same wrong class as the source model or a different wrong class, drawing connections between adversarial vulnerabilities and models' tendency of learning superficial cues (Jo & Bengio 2017) and shorteuts (Geirhos et al. 2020). Charles et al. (2019) examine adversarial transferability from a geometric point of view and prove the existence of transferable adversarial directions for simple network architectures. Based on observations that AEs tend to occur in contiguous regions within which all points can similarly fool the model (referred to as adversarial subspaces) (Tanay & Griffin, 2016). Tramer et al. (2017) explains the transferability as a result of intersection of models' adversarial subspaces: a higher mumber of orthogonal adversarial directions within these subspaces often implics higher transfcrability. Building on these works, Gubri et al. (2022b) highlights the role of weight space geometry in adversarial transferability; showing that adding random Gaussian noise to the weight space of DNNs increases their potential surrogates for crafting more transferable adversaries. A contemporary work by Zhu et al. (2021) makes an analogous obscrvation regarding the effect on adversarial transferability of adding random Gaussian noise in the output space. They propose Intrinsic Adversarial Attack (IAA) to diminish the impact of the deeper model layers. By doing so, they effectively exploit low-density regions of the data distribution where many highly transferable AEs can be found.
Evaluation Metrics. The assessment of adversarial transferability is a complex undertaking that demands thorough and extensive set of metrics. The Fooling Rate a popular choice is often used to quantify the trausferability of adversarial examples. It gauges the effectiveness of adversarial perturbations by measuring the percentage of these perturbations that can successfully deceive a specified target model. However, it's important to emphasize that the Fooling Rate metric is highly contingent on the choice of target models,
21

---

Published in Transactions on Machine Learning Research (05/2024)
which introduces considerable source of variability into the evaluation process. Recent research, as high- lighted in (Yu et al. 2023). has illuminated the profound impact that the selection of larget models can have on the relative rankings of different transferability -ellitt-ehhaciingg methods. Consequently, there is a pressing need for even more comprchensive evaluation metrics.
Consequently, there is a pressing need for an even more comprehensive benchmark that can encompass a wider range of model architectures and configurations. In addition to empirical evaluations, there is also a growing recognition of the necessity for theoretical characterizations of transferability. Such theoretical analyses can provide valuable insights into the underlying principles governing the transferability of adversarial attacks.
Benchmarking Adversarial Transferability. Various benchmarks have been developed to evaluate ad- versarial transferability. Initial robusiness benchmarks include transfer-based black-box attacks to evaluate the adversarial robustness of models (Croce et al.) 2020; Dong et al.) 2020). (Zhao et al. 2022) evalu- ates the transfcrability of adversarial examples, considering the perspectives of stcalthiness of adversarial perturbations. (Mao et al. 2022] evaluates Transferability-beee
attacks in real-world environments. Fur- thermore, [Zhoo et al. 2022 builds a more reliable evaluation benchmark by including various architectures.
Hybrid Approaches Combining Optimization-based and Generation-based. Both optimization- based and generation-based approaches have been intensively studied. While each of them has both ad- vantages and limitations, the pursuit of a well-designed hybrid method that combines both approaches to achieve better transferability is a promising direction for future endeavours. For example, certain methods leverage generative models while also going through optimization iterations during inference (Chen et all, 2.23d). Such hybrid methods have the potential to leverage the strengths of each approach to enhance the robustness and generalization of adversarial examples across various models and scenarios.
Adversarial Transferability Across Large Multimodal Models. The transferability of adversarial
examples across language models has been studied by various works introduced in section 5.2 Given the provalence of large language models (LLM) and multimodal foundation modcls, the adversarial transforability across such foundation models also become increasingly relevant to the community. The pioneering work of Zhao et al. (2024) shows that adversarial examples crafted against pre-trained models such as CLIP (Radford
ct al. 2021a) and BLIP (Li ct al.) 2022) can be transferred to other multimodal foundation models such MiniGPT-4 Zhu et al. 2023a) and LLaVA (Liu et al. 2024b. Similarly, Dong et al. (0023) demonstrates the feasibility of attacking Google's Bard with vision encoders of open-sourced models. Meanwhile, various works demonstrate that adversarial examples created on CLIP can be transferred to various CLIP-based systems (Lu et al.
2023 Zhang et al.
2022b Han et al. 2023a Hu et al. 2024). As reported in the current 2024 Dong et al.
work (Zhao et al.
2023 Luo et al.
2024h. the translerability across large multimodal
models is still very limited. Exploring the root causes behind such limited transfcrability and identifying
strategies for enhancing it could be an interesting direction for future research.
6.2 Connections to Broader Topics
Relation to Adversarial Transferability Prior To Deep Learning Era The concept of adversarial examples holds relevance both in deep learning contexts and in earlier eras of machine learning. Prior research shows that traditional machine learning algorithms also suffer from adversarial examples, e.g., support vector machine (Papernot et al., 2016) and decision tree (Papernot et al. 2016 Biggio et al.) 2013). Furthermore, Papernot et al. (2016) show that the adversarial examples can be translerred to different classes of machine learning classifiers. Specifically, the adversarial examples crcated on traditional machine learning models can be transferred others, even deep neural network-based classifiers. Similarly, the ones created on deep neural networks can also be transferred to traditional machine learning classifiers. However, the transferability is only shown on toy datasets. The experiments on large-scale datasets (e.g. ImageNet-1k (Russakovsky et al. 2015 are infeasible since the traditional algorithiss are not scalable to large datasets.
Relation to Trustworthy AI. Adversarial transferability is closely linked to Trustworthy AI, which aims to make AI systems reliable, fair, transparent, and accountable (Kaur et al., (022). When adversarial examples fool one AI model and then trick other models too, it highlights how vulnerable AI systems can be. Studying transferability can help us understand the cause of the adversarial example, namely, better understanding
22

---

Published in Transactions on Machine Learning Research (05/2024)
of failure cases of AI systems (Ilyas et al. 2019 Waseda et al. 2023 In addition, understanding this link helps improve AT's reliability by developing better defenses against such tricks (Madry et al. 2017
Relation to Adversarial ML. Adversarial ML focuses on understanding and mitigating vulnerabilities in machine learning models (Oprea & Vassiley 2023). In adversarial ML, the goal is to investigate how malicious actors can manipulate or deceive ML systems by modifying inputs (i.e. adversarial example) cause incorrect predictions or behaviors. Adversarial transferability is a crucial idea in Adversarial ML. It shows how adversarial examples, which are inputs crafted to fool one ML model, can also fool other models, even if they are different (Goodfellow et al.) 2014 Papernot et al. 2016). This reveals how vulnerabilities in ML systems can spread widely. Understanding adversarial transferability helps researchers grasp the complexities of attacks on ML models and develop stronger defenses against them.
7 Conclusion
In this comprehensive survey, we embarked on a journey through the intricate world of adversarial transfer- ability. Transferability allows adversarial examples designed for one model to successfully deceive different model, often with distinet architecture, opening the door to black-box attacks The adversarial transfer- ability across DNNs raises concerns about the reliability of DNN-based systems in safety-critical applications like medical image analysis and autonomous driving.
Throughout this survey, we navigated through the terminology, mathematical notations, and evaluation metrics crucial to understanding adversarial transferability. We explored a plethora of techniques designed to enhance transferability, categorizing them into two main groups: surrogate model-based and generative mosel-based methods. Moreover, we extended our investigation beyond image classification tasks, delving into
techniques in various vision and natural language processing tasks, as well those that Iranscend lask boundaries.
As the DNN landscape continues to advance, the comprehension of adversarial examples and their transfer- ability remains crucial. By illuminating the vulnerabilitics inherent in these systems, we aim to contribute to the development of more resilient, secure, and trustworthy DNN models, ultimately paving the way for their safe deployment in real-world applications. In this ever-cvolving journey towards adversarial resilience, we hope that this survey will serve as a valuable resource for researchers, practitioners, and enthusiasts alike.
Broader Impact Statement. At the intersection of machine learning and security, the study of adversarial examples and their transfcrability not only illuminates the vulnerabilities of modern learning systems but also opens avenues for strengthening their robustness and reliability. By comprehensively surveying the landscape of adversarial transferability, this paper contributes to a deeper understanding of the challenges posed by adversarial attacks across diverse domains, from image classification to various tasks. As researchers and practitioners strive fortify machine learning models against adversarial manipulation, the insights gleaned from this survey serve as a compass, guiding the development of more resilient algorithms and informing strategies for defending against emerging threats. Overall, this paper aims to better understand and safeguard against adversarial exploitation.
Acknowledgments
This work is partially supported by the UKRI grant: Turing AI Fellowship EP/ EP/WW02981/1 and EP- SRC/MURI grant: EP/N019474/1. We would also like to thank the Royal Academy of Engineering.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint NFT::2333008711 2023.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lone, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716 25-23716-33736, 2022.
23

---

Published in Transactions on Machine Learning Research (05/2024)
Philipp Benz, Chaoning Zhang, and In So Kweon. Batch normalization increases adversarial vulnerability and decreases adversarial transferability: A non-robust feature perspective. Tn Proceedings of the IEEE/CV International Conference on Computer Vision, pp. 7818 7827, 2021.
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šindić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Machine Learning and Knowledge Discovery in Dalabases: European Conference, ECML PKDD 2013, Prague, Czech Regublic, Seplember 23-27, 2013, Proceedings, Port III 13, pp. 387 402. Springer, 2013.
Gerda Bortsova, Cristina González- Gonzalo,
González-Gonzalo, Suzanne C Wetstein, Florian Dubost, loannis Katramados, Laurens Hogeweg, Bart Licfors, Bram van Ginneken, Josion PW Pluim, Mitko Veta, et al. Adversarial attack vulnerability of medical image analysis systems: Unexplored factors. Medical Image Analysis, 73: 102141, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020a.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neurol information processing systems, 33:1877 1901, 2020b.
Tom B Brown, Nicholas Carlini, Chiyuan Zhang, Catherine Olsson, Paul Christiano, and lan Goodfellow. Unrestricted adversarial examples. arXiv preprint ar Xio: 08352, 2018.
Junyoung Byun, Seungju Cho, Myung-Joon Kwon, Hee-Seon Kim, and Changick Kim. Improving the transferability of targeted adversarial examples through object-based diverse input. Tn Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15244-15253, 2022.
Zikui Cai, Shantanu Rane, Alejandro E Brito, Chengyu Song, Srikanth V Krishnamurthy, Amit K Roy- Chowdhury, and M Salman Asif. Zero-query transfer allacks on conlext-aware object detectors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15024-15033 2022a.
Zikui Cai, Xinxin Xic, Shasha Li, Mingjun Yin, Chengyu Song, Srikanth Krishnamurthy, Amit K Roy- Chowdhury, and M Salman Asif. Context-aware transfer atlacks for object detection. In Proceedings of the AAAI Conference Artificial Intelligence, pp. 149 149-157, 2022b.
Zachary Charles, Harrison Rosenberg, and Dimitris Papailiopoulos. A geometric perspective on the trans- forability of adversarial directions. The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1960 1968. PMLR, 2019.
Bin Chen, Jiali Yin, Shukai Chen, Bohao Chen, and Ximeng Liu. An adaptive model ensemble adversarial attack for boosting adversarial transferability. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 4489 4498, 2023a.
Jianqi Chen, Hao Chen, Keyan Chen, Yilan Zhang, Zhengxia Zou, and Zhenwei Shi. Diffusion models for imperceptible and transferable adversarial attack. arXiv preprint 2335 2023b.
Shuo Chen, Jindong Gu, Zhen Han, Yunpu Ma, Philip Torr, and Volker Tresp. Benchmarking robustness of adaptation methods on pre-trained vision-language models. arXiv preprint arXiw:2306..04 02080, 2023c.
Xinquan Chen, Xitong Gao, Juanjuan Zhao, Kejiang Ye, and
Xu. Advdiffuser: Natural adver-
sarial example synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4562-4572, October 2023d.
Zhaoyu Chen, Bo Li, Shuang Wu, Kaixun Jiang, Shouhong Ding, and Wenqiang Zhang. Content-baase based unrestricted adversarial attack. Advances in Neural Information Processing Systems, 36, 2024.
24

---

Published in Transactions on Machine Learning Research (05/2024)
Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: standardized adversarial robustness benchmark. arXiv preprint arXiv:2010. 09670, 2020.
Ambra Demontis, Marco Melis, Maura Pintor, Matthew Jagielski, Battista Biggio, Alina Oprea, Cristina Nita-Rotaru, and Fabio Roli. Why do adversarial attacks transfer? explaining transferability of evasion and poisoning attacks. In 28th USENIX Security Symposium (USENIX Security 19), pp. 321-338, Santa Clara, CA, August 2019. USENTX Association. ISBN 978-1-939133-06-99 URL https: //www usenix.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In Proceedings of the IEEE conference computer vision and pattern recognition, pp.
2018.
Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Evading defenses to transferable adversarial examples by translation-invariant attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4312 4321, 2019.
Yinpeng Dong, Qi-An Fu, Xiao Yang, Tianyu Pang, Hang Su, Zihao Xiao, and Jun Zhu. Benchmarking adversarial robustness on image classification. In proceedingg of the IEEE/CVE conference on compuler vision and pattern recognition, pp.
2020.
Yinpeng Dong, Huanran Chen, Jiawei Chen, Zhengwei Fang, Xiao Yang, Yichi Zhang, Yu Tian, Hang Su, and Jun Zhu. How robust is google's bard to adversarial image attacks? arXie preprint or Xrx:22300 1 11751, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words:
Transformers for image recognition scale. In arXiv:2010.1192) 2020.
Wlodzistaw Duch and Jerzy Korczak. Optimization and global minimization methods suitable for neural networks. Neural computing surveys, 2:163-212, 1998.
Shuman Fang, Jie Li, Xianming Lin, and Rongrong Ji. Learning to learn transferable attack. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 571-579, 2022.
Yan Feng, Baoyuan Wu, Yanbo Fan, Li Liu, Zhifeng Li, and Shu-Tao Xia. Boosting black-bor attack with partially transferred conditional adversarial distribution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15095-15104, 2022.
Aditya Ganeshan, Vivek BS, and R Venkatesh Babu. Fda: Feature disruptive attack. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8069 8079, 2019.
Xiangbo Gao, Cheng Luo, Qinliang Lin, Weicheng Xie, Minmin Liu, Linlin Shen, Keerthy Kusumam, and Siyang Song. Scale-free and task-agnostic attack: Generating photo-realistic adversarial patterns with patch quilting generalor. Xix preprint, 2208, 2022.
Robert Geirhos, Jörn-Henrit Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shorteut learning in deep neural networks. Nature Machine Intelli- gence, 2(11):665 673, 2020.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint 14f::141..6777 2014.
Mengran Gou, Xikang Zhang, Angels
Sadjad Asghari-Esfeden Mario Sznaier, and Octavia Camps. Person re-identification in appearance impaired scenarios. ar preprint arXiw:1604.00367 2016.
Jindong Gu, Baoyuan Wu, and Volker Tresp. Effective and efficient vote attack on capsule networks. International Conference on Learning Representations (ICI.R), 2021a.
25

---

Published in Transactions on Machine Learning Research (05/2024)
Jindong Gu, Hengshuang Zhao, Volker Tresp, and Philip Torr. Adversarial examples segmentation models can be easy to transfer. orXiv preprint arXiv:2111. 1121111.11100 1368, 2021b.
Martin Gubri, Maxime Cordy, Mike Papadakis, Yves Le Traon, and Koushik Sen. Efficient and transferable adversarial examples from bayesian neural networks. In Uncertainty in Artificial Intelligence, pp. 738 748.
PMLR, 2022a.
Martin Gubri, Maxime Cordy, Mike Papadakis, Yves Le Traon, and Koushik Sen. Lgv: Boosting adversarial example transferability from large geometric vicinity. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviu, Israel, October 23 23-27, 2022, Proceedings, Part IV, pp. 603-618 318. Springer, 2022b.
Yiwen Guo, Qishang Li, and Hao Chen. Backpropagating linearly improves transferability of adversarial examples. Advances in neural information processing systems, 33:85 95, 2020.
Abdullah Hamdi, Sara Rojas, Ali Thabet, and Bernard Ghanem. Advpe: Transferable adversarial pertur- bations on 3d point clouds. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XII 16, pp. 241-257. Springer, 2020.
Dongehen Han, Xiaojun Jia, Yang Bai, Jindong Gu, Yang Liu, and Xiaochun Cao. Ot-attack: Enhancing adversarial transferability of vision-language models via optimal transport optimization. arXie preprinl arXiv:2312.04403, 2023a.
Jiangfan Han, Xiaoyi Dong, Ruimao Zhang, Dongdong Chen, Weiming Zhang, Nenghai Yu, Ping Luo, and Xiaogang Wang. Once a man: Towards multi-target attack via learning multi-target adversarial network once. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2158-5167, 2019.
Sicong Han, Chenhao Lin, Chao Shen, Qian Wang, and Xiaohong Guan. Interpreting adversarial examples in doep learning: A review. ACM Computing Surveys, 2023b.
Atiye Sadat Hashemi, Andreas Bär, Saced Mozaffari, and Tim Fingscheidt. Improving transferability of generated universal adversarial perturbations for image classification and segmentation. In Deep Newral Networks and Data for Automated Driving: Robustness, Uncertainty Quantification, and Insights Towards Safety, pp. 171-196. Springer International Publishing Cham, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE International Conference on Computer Vision, pp. 770-778, 2016.
Xuanli He, Lingjuan Lyu, Qiongkai Xu, and Lichao Sun. Model extraction and adversarial transfcrability, your bert is vulnerable! arxiv preprint 1031033103.10013, 2021.
Jan Hendrik Metzen, Munmadi Chaithanya Kumar, Thomas Brox, and Volker Fischer. Universal adversarial perturbations against semantic image segmentation. In Proceedings of the IEEE internalional conference on computer vision, pp. 2755 2764, 2017.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial ex- amples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15262 15271, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840- (00166 2020.
Anjun Hu, Jindong Gu, Francesco Pinto, Konstantinos Kamnitsas, and Philip Torr. As firm as their founda- tions: Can open-sourced foundation models be used to create adversarial examples for downstream tasks? arXiv preprint ar Xrx:::240S. 1 12698, 2024.
Hao Huang, Ziyan Chen, Huanran Chen, Yongtao Wang, and Kevin Zhang. T-sca: Transfer-based self- ensemble attack on object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 20514-20522 2023.
26

---

Published in Transactions on Machine Learning Research (05/2024)
Qian Huang, Isay Katsman, Horace He, Zeqi Gu, Serge Belongie, and Ser-Nam Lim. Enhancing adversarial example transferability with an intermediate level attack. In Proceedings of the IEEE/CV international conference on computer vision, pp. 4733 4742, 2019a.
Yan Huang, Qiang Wu, Jingsong Xu, and Yi Zhong. Celebritics-ridd: A benchmark for clothes variation in long-term person re-identification. In 2019 International Joinl Conference on Neural Nelworks (IJCNN), pp. 1-8. IEEE, 2019b.
Yi Huang and Adams Wai-Kin Kong- Transferable adversarial attack based on integrated gradients. arXiv preprint (rXic:2205.1111 2022.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Alcksander Madry. Adversarial examples are not bugs, they are features. Adoances in neural informalion processing systems, 32, 2019.
Nathan Inkawhich, Wei Wen, Hai Helen Li, and Yiran Chen. Feature space perturbations yield more transferable adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 7066 7074, 2019.
Nathan Inkawhich, Kevin Liang, Binghui Wang, Matthew Inkawhich, Lawrence Carin, and Yiran Chen. Perturbing across the feature hierarchy to improve standard and strict blackbox attack transferability. Advances in Newral Information Processing Systems, 33:20791 20801, 2020a.
Nathan Inkawhich, Kevin Liang Lawrence Carin, and Yiran Chen. Transferable perturbations of deep feature distributions. ar preprint arXiv:2004. 12519, 2020b.
Shuai Jia, Bangjie Yin, Taiping Yao, Shouhong Ding, Chunhua Shen, Xiaokang Yang, and Chao Ma. Adv-attribute: Inconspicuous and transferable adversarial attack face recognition. ar Xiv preprint SEfi:2221000877. 2022a.
Xiaojun Jia, Yong Zhang, Baoyuan Wu, Ke Ma, Jue Wang, and Xiaochun Cao. Las-at: adversarial training with learnable attack strategy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13398-13408, 2022b.
Jason and Yoshua Bengio. Measuring the tendency cnns to learn surface statistical regularities. arXiv preprint arXto:1711.11661 2017.
Rie Johnson and Tong Zhang- Accelerating stochastic gradient descent using predictive variance reduction. Advances in neural information processing systems, 26, 2013.
Davinder Kaur, Suleyman Uslu, Kaley J Rittichier, and Arjan Durresi. Trustworthy artificial intelligence: a review. ACM compuling surveys (CSUR), 55(2):1 38, 2022.
Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of no..I--H.T, volume 1, pp. 2, 2019.
Jinkyu Kim and John F Canny. Interpretable learning for self- self-driving cars by visualizing causal attention. In International Conference on Computer Vision (ICCV) pp. 2961-2969, 2017.
Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John Canny, Zeynep Akata, et al. Textual explanations for self-driving vehicles. In ECCV, pp. 577-593. Springer, 2018.
Woo Jae Kim, Seunghoon Hong, and Sung-Fui Yoon. Diverse generative perturbations on attention space for transferable adversarial attacks. In 2022 IEEE International Conference on Image Processing (ICIP), pp. 281 285. IEEE, 2022.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
27

---

Published in Transactions on Machine Learning Research (05/2024)
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip:
Bootstrapping language-image pre-training for unified vision-laguuage understanding and generation. In ICML, 2022.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsich, and Kai-Wei Chang. Visualbert: A simple and performant bascline for vision and language. arXiv preprint arxiv: 03537, 2019.
Maosen Li, Cheng Deng, Tengjiao Li, Junchi Yan, Xinbo Gao, and Heng Huang. Towards transferable tar- geted attack. In Proceedings of the IERE/CVE Conference on Computer Vision and Pottern Recognition, pp. 641-649, 2020a.
Qizhang Li, Yiwen Guo, Wangmeng Zuo, and Hao Chen. Making substitute models more bayesian can enhance transferability of adversarial examples. ar preprint urXin:2302. 05066, 2023.
Yingwei Li, Song Bai, Cihang Xie, Zhenyu Liao, Xiaohui Shen, and Alan Yuille. Regional homogeneity: Towards learning transferablc universal adversarial perturbations against defenses. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part XI 16, pp. 795 813. Springer, 2020b.
Kaizhao Liang, Jacky Y Zhang, Boxin Wang, Zhuolin Yang, Sanmi Koyejo, and Bo Li. Uncovering the connections between adversarial transferability and knowledge transferability. In International Conference on Machine Learning, pp. 6577 -777- PMLR, 2021.
Jiadong Lin, Chuanbiao Song, Kun He, Liwei Wang, and John E Hopcroft. Nesterov accelerated gradient and scale invariance for adversarial attacks. arXiv preprint arXiv: 06281, 2019.
Fangcheng Liu, Chao Zhang, and Hongyang Zhang. Towards transferable unrestricted adversarial examples with minimum changes. In 2023 IEEE Conference on Secure and Trustworthy Machine Learning (Sa TML), pp. 327 338. IEEE, 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lec. Visual instruction tuning. Advances in neural information processing syslems, 36, 2024.
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attacks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track. Proceedings. OpenReview.ned 2017.
Dong Lu, Zhiqiang Wang, Teng Wang, Weili Guan, Hongchang Gao, and Feng Zheng. Set-level guidance attack: Boosting adversarial transferability of vision-language pre-training models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 102 111, 2023.
Yantao Lu, Yunhan Jia, Jianyu Wang, Bai Li, Weiheng Chai, Lawrence Carin, and Senem Velipasalar. Enhancing cross-task black-box transferability of adversarial examples with dispersion reduction. In Pro- ceedings of the IEEE/CV conference on Computer Vision and Pattern Recognition, pp. 940 9009949,44 2020.
Haochen Luo, Jindong Gu, Fengyuan Liu, and Philip Torr. An image is worth 1000 lies: Transferability of adversarial images across prompls on vision-language models. In The Twelfth International Conference on Learning Representations, 2024. URL / htw.peenrrreiieeeee/ourruuraa
Wenshuo Ma, Yidong Li, Xiaofeng Jia, and Woi Xu. Transferable adversarial attack for both vision trans- formers and convolutional networks via momentun integrated gradients. In International Conference on Computer Vision (ICCV), pp. 4630 4639, 2023.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXio: 06083, 2017.
Xiaofeng Mao, Yuefeng Chen, Yuhong Li, Yuan He, and Hui Xue. Gap++: Learning to generate target- conditioned adversarial examples. XrXio preprint ar Xiv:20000.0000 050997, 2020.
28

---

Published in Transactions on Machine Learning Research (05/2024)
Yuhao Mao, Chong Fu, Saizhuo Wang, Shouling Ji, Xuhong Zhang, Zhenguang Liu, Jun Zhou, Alex X Liu, Raheem Beyah, and Ting Wang. Transfer attacks revisited: A large-scale empirical study in real computer vision settings. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1423 1439. IEEE, 2022.
Mchdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXio:1411.170 2014.
Seyed-Mohsen Moosavi-Dezfooli, Alhusscin Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In Proceedings of the IEEE conference on compuler vision and pallern recognilion, pp.
2017.
Muhammad Muzammal Naseer, Salman H Khan, Muhammad Haris Khan, Fahad Shahbaz Khan, and Fatih Porikli. Cross-domain transferability of adversarial perturbations. Advances in Neural Information Pro- cessing Systems, 32, 2019.
Muzammal Naseer, Salman H. Khan, Shafin Rahmaan, and Fatih Porikli. Task-generalizable adversarial attack based on perceptual metric. arXiv preprint Xrx:::1811. 181.00900000000, 2018.
Muzammal Naseer, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Fatih Porikli. On general- ing transferable targeted perturbations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7708 7717, 2021.
Muzammal Naseer, Kanchana Ranasinghe, Salman Khan, Fahad Shahbaz Khan, and Fatih Porikli. On improving adversarial transferability of vision transformers. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
Muzammal Naseer, Ahmad Mahmood, Salman Khan, and Fahad Khan. Boosting adversarial transferability using dynamic cues. In International Conference on Learning Representations (ICLR), 2023.
Yurii Nesterov. A method for unconstrained convex minimization problem with the rate of convergence (1/k" 2). In Doklady ussr, volume 269, pp. 543 547, 1983.
Alina Oprea and Apostol Vassilev. Adversarial machine learning: taxonomy and terminology of attacks and mitigations. Technical report, National Institute of Standards and Technology, 2023.
Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learning: from phe- nomena to black-box attacks using adversarial samples. arXie preprint 1rXix:6655.022) 07277, 2016.
Florent Perronnin, Jorge Sánchez, and Thomas Mensink. Improving the fisher kernel for large-scale im- age classification. In Computer Vision-ECCC -Ccin 2010: 11th European Conference on Computer Vision, Heraklion, Crele, Greece, Seplember 5-11, 2010, Proceedings, Part IV 11, pp. 143 156. Springer, 2010.
Camilo Pestana, Naveed Akhtar, Nazanin Rahnavard, Mubarak Shah, and Ajmal Mian. Transferable 3d adversarial textures using end-to-end optimization. In Proceedings of the IEFE/CVF Winter Conference on Applications of Computer Vision, pp. 88 97, 2022.
Huy Phan, Yi Xic, Siyu Liao, Jie Chen, and Bo Yuan. Cag: real-time low-cost enhanced-robustnes high--tranferrabiiity content-aware adversarial attack generator. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 5412-541 2020.
Boris Polyak. Some methods of speeding up the convergence of iteration methods. User compulational mathematics and mathematical physics, 4(5):1-17, 1964.
Omid Poursaced, Tsay Katsman, Bicheng Gao, and Serge Belongie. Generative adversarial perturbations. Tn Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4422-4431, 2018.
Yaguan Qian, Shuke Hc, Chenyu Zhao, Jiaqiang Sha, Wei Wang, and Bin Wang. Lea2: A lightweight ensem- ble adversarial attack via non-overlapping vulnerable frequency regions. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 4510-4521 2023.
29

---

Published in Transactions on Machine Learning Research (05/2024)
Zeyu Qin, Yanbo Fan, Yi Liu, Li Shen, Yong Zhang, Jue Wang, and Baoyuan Wu. Boosting the transferability of adversarial attacks with reverse adversarial perturbation. arXiv preprint arXiv:2200.00 0.05968, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Tlya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021a.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PML.R, 2021b.
Nicolas Roux, Mark Schmidt, and Francis Bach. stochastic gradient method with an exponential conver- gence -rat for finite training sets, Advances neural information processing systems, 25, 2012.
Andras Rozsa, Manuel Günther, and Terrance E Boult. Lots about attacking deep features. In 2017 IEEE Internaiional Joint Conference on Biometrics (IJCB), pp. 168 176. IEEE, 2017.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Sathcesh, Scan Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-252, 2015.
2015.
Mathieu Salzmann et al. Learning transferable adversarial perturbations. Advances in Neural Information Processing Systems, 34:13950-13962 2021.
Alex Serban, Erik Poll, and Joost Visser. Adversarial examples on object recognition: A comprehensive survey. ACM Computing Surveys (CSUR), 53(3):1 38, 2020.
Alexander Michael Staff, Jin Zhang, Jingyue Li, Jing Xie, Elizabeth Ann Traiger, Jon Arne Glomsrud, and Kristian Bertheussen Karolius. An empirical study on cross-data transferability of adversarial attacks on object detectors. In Al-Cybersec@ SGAT, pp. 38 52, 2021.
Lu Sun, Mingtian Tan, and Zhe Zhou. A survey of practical adversarial example attacks. Cybersecurity, 1: 1-9, 2018.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic altribution for deep networks. In International conference on machine learning, pp. 3319 3328. PMLR, 2017a.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 2319-3328, PMLR, 2017b.
Christian Szegedy, Wojciech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, lan Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: 6199, 2013.
Thomas Tanay and Lewis Griffin. boundary tilting persepective on the phenomenon of adversarial exam- ples. arXiv preprinl NKK::6608007660 2016.
Giorgos Tolias, Filip Radenovic, and Ondrej Chum. Targeted mismatch adversarial attack: Quory with a flower to retrieve the tower. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 5037-5046, 2019.
Florian Tramer, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. The space of trans- ferable adversarial examples. arXiv preprint Xiv: 704.03453. 2017.
Tzungyu Tsai, Kaichen Yang, Tsung-Yi Yi Ho, and Yier Jin. Robust adversarial objects against deep learning models. In Proceedings of the AAAI Conference Artificial Intelligence, pp. 954 962, 2020.
Ashish Vaswani, Noam Shazoer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
30

---

Published in Transactions on Machine Learning Research (05/2024)
Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, et al. Scientific discovery in the age of artificial intelligence. Nature, 620(7972):47 60, 2023.
Hongjun Wang, Guangrun Wang, Ya Li, Dongyu Zhang, and Liang Lin. Transferable, controllable, and inconspicuous adversarial attacks on person re-identification with deep mis-ranking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 342 342-351, 2020.
Xiaosen Wang and Kun He. Enhancing the transferability of adversarial attacks through variance tuning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 1924-1933,
2021.
Xiaosen Wang, Xuanran He, Jingdong Wang, and Kun He. Admix: Enhancing the transferability of ad- versarial attacks. In Proceedings of the IEEE/CVP International Conference on Computer Vision, pp. 16158 16167, 2021a.
Xin Wang, Jie Ren, Shuyun Lin, Xiangming Zhu, Yisen Wang, and Quanshi Zhang. A unified approach to interpreting and boosting adversarial transferability. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.nee net, 2021b.
Zhen Wang, Yitao Zheng, Hai Zhu, Chang Yang, and Tianyi Chen. Transferable adversarial examples can efficiently fool topic models. Computers & Security, 118:102749, 2022.
Zhibo Wang, Hengchang Guo, Zhifei Zhang, Wenxin Liu, Zhan Qin, and Kui Ren. Feature importance-aware transferable adversarial attacks. In Proceedings of the IEEE/CVF international conference computer 2021c
vision, pp.
Futa Waseda, Sosuke Nishikawa, Trung-Nghia Le, Huy H Nguyen, and Isao Echizen. Closer look at the transferability of adversarial examples: How they fool different models differently. In Proceedings of the IEEE/CVF Winter Conference on Applications of Compuler Vision, pp. 1360 1368, 2023.
Xingxing Wei, Siyuan Liang, Ning Chen, and Xiaochun Cao. Transferable adversarial attacks for image and video object detection. arxiv preprint orXin: 111,,1241,, 2018.
Zhipeng Wei, Jingjing Chen, Zuxuan Wu, and Yu-Gang Jiang. Boosting the transferability of video adver- sarial examples via temporal translation. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 2659 2667, 2022a.
Zhipeng Wei, Jingjing Chen, Zuxuan Wu, and Yu-Gang Jiang- Cross-modal transferable adversarial attacks from images to videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15064 15073, 2022b.
Rey Reza Wiyatno, Anqi Xu, Ousmane Dia, and Archy De Berker. Adversarial examples in modern machine learning: A review. arXiv preprint ar
2019.
Boxi Wu, Jindong Gu, Zhifeng Li, Deng Cai, Xiaofei He, and Wei Liu. Towards efficient adversarial training on vision transformers. In European Conference on Computer Vision, pp. 407-325. Springer, 2022.
Dongxian Wu, Yisen Wang, Shu-Tao Xia, James Bailey, and Xingiun Ma. Skip connections matter: On the transfcrability of adversarial examples generated with resnets. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net net, 2020a.
Lei Wu and Zhanxing Zhu. Towards understanding and improving the transferability of adversarial examples in deep neural networks. In Asian Conference on Machine Learning, pp. $37-850. PMLR, 2020.
Weibin Wu, Yuxin Su, Xixian Chen, Shenglin Zhao, Irwin King, Michael R Lyu, and Yu-Wing Tai. Boosting the transferability of adversarial samples via attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1161-1170, 2020b.
31

---

Published in Transactions on Machine Learning Research (05/2024)
Weibin Wu, Yuxin Su, Michael R Lyu, and Irwin King. Improving the transferabilityy of adversarial samples with adversarial transformations ions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9024 9033, 2021.
Chong Xiang, Charles R Qi, and Bo Li. Generating 3d adversarial point clouds. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Paltern Recognition, pp. 9136 9144, 2019.
Chaowei Xiao, Bo Li, Jun-Yan Zhu, Warren He, Mingyan Liu, and Dawn Song. Generating adversarial examples with adversarial networks. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pp. 3905-3911, 2018.
Yanru Xiao and Cong Wang. You see what i want you to see: Exploring targeted black-box transferability attack for hash-based image retrieval systems. In Proceedings of the SGEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 1934-1943, 2021.
Zihao Xiao, Xianfeng Gao, Chilin Fu, Yinpeng Dong, Wei Gao, Xiaolu Zhang, Jun Zhou, and Jun Zhu, Improving transfcrability of adversarial patches face recognition with generative models. In Proceedings of the IEEE/CVE Conference on Compuler Vision and Pallern Recognition, pp. 11845 11854, 2021.
Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan L Yuille. Improving transferability of adversarial examples with input diversity. In Proceedings of the TEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2730 2739, 2019.
Yifeng Xiong, Jiadong Lin, Min Zhang, John E Hopcroft, and Kun He Stochastic variance reduced ensemble adversarial attack for boosting the adversarial transferability. In Proceedings of the IEEE/CVV Conference on Computer Vision and Pattern Recognition, pp. 14983 14992, 2022.
Nuo Xu, Kaleel Mahmood, Haowen Fang, Ethan Rathbun, Caiwen Ding, and Wujie Wen. Securing the spike: On the transferabilty and security spiking neural networks to adversarial examples. arXiv preprint ar X2v: 03358, 2022a.
Xiaojun Xu, Jacky Y Zhang, Evelyn Ma, Hyun Ho Son, Sanmi Koyejo, and Bo Li. Adversarially robust mod- els may not transfer better: Sufficient conditions for domain transferability from the view of regularization. In International Conference on Machine Learning, pp. 24770 24802. PMLR, 2022b.
Jia Xue, Zibo Meng, Karthik Katipally, Haibo Wang, and Kees Van Zon. Clothing change aware per- son identification. In Proceedings of the IEEE Conference on Computer Vision and Pallern Recognation Workshops, pp. 2112 2120, 2018
Erkun Yang, Tongliang Liu, Cheng Deng, and Dacheng Tao. Adversarial examples for hamming space search. IEEE transactions on cybernetics, 50(4):1473 1484, 2018
Xiao Yang, Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Boosting trausferability of targeted ad- versarial examples via hierarchical generative networks. In Computer -CCon-ECCV 2022: 17th European Conference, Tel Aviu, Israel, October 23-27, 2022, Proceedings, Port IV, pp. 725-742. Springer, 2022.
Wenqian Yu, Jindong Gu, Zhijiang Li, and Philip Torr. Reliable evaluation of adversarial transferability. arXiv preprint xrXi:22306.. 08565.,88556.. 2023.
Liping Yuan, Xiaoqing Zheng, Yi Zhou, Cho-Jui Hsich, and Kai-Wei Chang. On the transferability of adversarial attacksagainst neural text classifier. arXiv preprint arXiv:2011.085.1 2020.
Chaoning Zhang, Philipp Benz, Adil Karjauv, Jae Won Cho, Kang Zhang, and In So Kweon. Investigating top-k white-box and transferable black-box attack. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15085 15094, 2022a.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. Xix preprint 1rx00 2017.
32

---

Published in Transactions on Machine Learning Research (05/2024)
Jiaming Zhang, Qi Yi, and Jitao Sang. Towards adversarial attack on vision-language pre-training models. In Proceedings of the 30th ACM International Conference on Mullimedia, pp. 5005-5013, 2022b.
Jianping Zhang, Weibin Wu, Jen-tse Huang, Yizhan Huang, Wenxuan Wang, Yuxin Su, and Michael R. Lyu. Improving adversarial transferability via neuron attribution-based attacks In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 14993 15002, 2022c.
Jiliang Zhang and Chen Li. Adversarial examples: Opportunities and challenges. IEEE transactions on neural networks and learning systems, 31(7):2578 2593, 2019.
Yanghao Zhang, Wenjic Ruan, Fu Wang, and Xiaowei Huang. Generalizing universal adversarial attacks beyond additive perturbations. In 2020 IEEE International Conference on Data Mining (ICDM), pp- 1412 1417. IEEE, 2020.
Anqi Zhao, Tong Chu, Yahao Liu, Wen Li, Jingjing Li, and Lixin Duan. Minimizing maximum model discrepancy for transferable black box targeted attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognation, pp. 8153-8162, 2023.
Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan Li, Ngai-Man Man Cheung, and Min Lin. On evaluating adversarial robustness of large vision-language models. Advances in Newral Information Processing Systems, 36, 2024.
Zhengli Zhao, Dheeru Dua, and Sameer Singh. Generating natural adversarial examples. arXiv preprint arXiv:1710. 11342, 2017.
Zhengyu Zhao, Zhuoran Liu, and Martha Larson. On success and simplicity: A second look at transferable targeted attacks. Advances in Neural Information Processing Systems, 34:6115-6122 2021.
Zhengyu Zhao, Hanwei Zhang, Renjue Li, Ronan Sicre, Laurent Amsaleg, and Michael Backes. Towards good practices in evaluating transfer adversarial attacks. arXiv preprint 2=XXx:2211.09500 2022.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921 2929, 2016.
Hang Zhou, Kejiang Chen, Weiming Zhang, Han Fang, Wenbo Zhou, and Nenghai Yu. Dup-net: Denoiser and upsampler network for 3d adversarial point clouds defense. In Proceedings of the IEEE/CVD International Conference on Computer Vision, pp. 1961-1970, 2019.
Wen Zhou, Xin Hou, Yongjun Chen, Mengyun Tang, Xiangqi Huang, Xiang Gan, and Yong Yang. Transfer- able adversarial perturbations. bations. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 452 467, 2018
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv preprint arXiv:2304. 105922 2023a.
Hegui Zhu, Yuchen Ren, Xiaoyan Sui, Lianping Yang, and Wuming Jiang, Boosting adversarial transferability via gradient relevance attack. In International Conference on Computer Vision (ICCV), pp. 4741-4750, 2023b.
Yao Zhu, Jiacheng Sun, and Zhenguo Li. Rethinking adversarial transferability from data distribution perspective. In International Conference Learning Representations, 2021.
Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023.
Junhua Zou, Zhisong Pan, Junyang Qiu, Xin Liu, Ting Rui, and Wei Li. Improving the transferability of adversarial examples with resizel-diverse--nntt, diversity-ensenbl and region fitting. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK August 23- 23-28, 2020, Proceedings, Part XXII, pp. 563 579. Springer, 2020.
33

---

Published in Transactions on Machine Learning Research (05/2024)
Junhua Zou, Yexin Duan, Boyu Li, Wu Zhang, Yu Pan, and Zhisong Pan. Making adversarial examples more transferable and indistinguishabl. In Proceedings of the AAAT Conference on Arlificial Intelligence, pp. 3662 -670,, 2022.
A More Variants of I-FGSM
In this appendix, we first recall some background information on I-FGSM and present more variants of I-FGSM Dong et al.
We focus on perturbations constrained by an lo ball with radius that is, || - =allo Se. To understand the rest this section, we begin by formalizing the iterative variant of the fast gradient sign method (I- FGSM) (Goodfellow al., 2014), which serves as the basis for the development of other methods. The I-FSSM has the following update rule:
g(+1)) = Vl(x=du(1)) y).
(64) where g() is the gradient of the loss function with respect to the input, a denotes the step size at each iteration, and Clip', ensures that the perturbation satisfies the fo -orm constraints. More variants of 1- FGSM are as follows:
( Clip, (z a (iggnn((((++))
Nesterov (NI-FGSM). Nesterov Accelerated Gradient (NAG) is another popular extension of the vanilla gradient descent algorithm that incorporates momentum to accelerate convergence and improve the general- ization of neural networks (Nesterov) 1983). On top of the momentum mechanism, the most distinct feature of NAG is that the gradient evaluated a lookahead position based on the momentum term. Lin et al.) propose NI-FGSM (Nesterov Iterative Fast Gradient Sign Method), which integrates NAG in the iterative gradient-based attack to leverage its looking-ahead property to help escape from poor local optima. At each. iteration, NI-FGSM first moves the data point based on the accumulated update g(t)
nes(f) addv(t) a H (t)
then we have
(l(x y)
(1+1) p (t)
(2)ue(()) y)ll1
17 The author argues that the anticipatory update of NAG helps to circunvent getting stuck at the local optimal easier and faster, thereby improving the transferability of the perturbation.
and the formulation for 1) remains the same
Adam (AI-FGTM). Adam a popular adaptive gradient method that combines the first- and second- order momentum of the gradients (Kingma & Ba 2015). Zou el al.] introduce the Adam Iterative Fast Gradient Tanh Method (AL-FGTM), which adapts Adam the process of generating adversarial examples. In addition to using Adam opposed to the momentum formulation, key feature of AI-FGTM is the replacement of the sign function with the tanh function, which has the advantage of smaller perturbation size.
((+1) =0 + fı Vl(1)))(1) .1)) ((+1) = (0()) + μ2 (Vl(zadv(1) y))²
where (1) denotes the first moment vector, p()) represents the second moment vector, (o and #2 are the first and second-order momentum factors, respectively. Instead of using a fixed step size of a in each ileration, AI-FGTM computes an adaptive step size based on
1 g((+1)
5
Σ
1- ((
V (1-ss(t1)) g((t1))
=
V
/1-
34

---

Published in Transactions on Machine Learning Research (05/2024)
where 81 and 32 exponential decay rates, À denotes scale factor, and we have Ee a the update rule of AI-FGTM is
Finally,
ml++1) Vu(++j + 6"
2odv(t+1)= Clips fzade(t) + ta tanh(A
)
where 8 is to avoid division-by-zero.
Variance Tuning (VNI/VM--GGSM). Previous work shows that the stochastic nature of the mini- batch gradient introduces a large variance in the gradient estimation, resulting in slow convergence and poor generalivation; and this gives rise various variance reduction methods (Roux et al. 2012 Johnson & Zhang 2013). In the context of generating adversarial examples, Wang He presents variance-tuning technique that adopts the gradient information in the neighborhood of the previous data point tune the gradient of the current data point each iteration. Given an input Rd, they propose to approximate the variance of its gradient using
N V el' Vl(z,)), 21
V(x)
(65)
where x² + T; and each dimension of r; independently sampled from a uniform distribution between BB and 3 with being a hyper parameter. They introduce Variance-tuning MI-FGSM (VMI-FGSM) and Variance-tuning NI-FGSM (VNI-FGSM) as improvement over the original formulation. To integrate gradient variance in the iterative process, we can modify the following update rule for g(++)) by using
g(++1) v((x
g g(++1) + v(1) [[gh++1) + u(9)
g (1+)) p (())
(1+1) V(x(())
(66)
where 20 add((() in MI-FGSM and (0 mee(()) VNI-FGMM.
35


Reviews:
[{"review_id": "5l7cDmMiJn", "reviewer": "Reviewer_r2cH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- A comprehensive survey covering most impactful and recent works in developing transferabiltiy-based attacks.\n- A systematic and clear taxonomy. For example, for attacks via surrogate models, the survey classifies them into three classes: data augmentation, loss, and optimization.\n\nWeaknesses:\n- The submission may be improved in terms of summarization. Several parts of the submission read more like pure enumeration (xxx et al present xxx; xxx et al present xxx; ...) instead of categorizing and summarizing the literature. It would benefit the readers more if the core methodology is revealed and used to narrate the literature top-down. Furthermore, some taxonomy figures and comparison tables would improve the accessibility.\n- The granularity assignment may be improved. For attacks vis surrogate models, many works are introduced in detail, with core equations listed but without a complete explanation. For example, $m$ and $v$ are not defined in AI-FGTM. Eqns. (27) and (28) are presented without detailed explanation. On the other hand, in Section 5.3, only the goal of the attacks are revealed (present an attack achieving xxx) rather than core methodology. It can be better if the core methodology of the representative methods is introduced and other details are omitted.\n- The discussion can be further enhanced. For example, I would expect the authors to contextualize the research of adversarial transferability in a larger scope - their impact and connections with general adversarial ML, the theoretical point of views of adversarial transferability, their applications in more general trustworthy AI, the transferability study prior to the DL era, the transferability study in LLMs, etc. However, the discussion of the submission seems to mainly focus on how to leverage transferability to derive stronger adversarial attacks itself.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission is a survey paper. It provides a comprehensive review of research progress on achieving and improving the transferability of adversarial examples. The paper mainly focuses on transferability-based attacks in the image classification task, covers a broad range of these attacks, and classifies these attacks into two categories: via surrogate model, and via generative model. Then the paper further discusses the transferability-based attacks in other domains. The paper is concluded with a discussion of future works.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See \"weaknesses\" above for requested changes.\n\nMinor:\n- Table 2: define \"AE\" - I guess it is \"adversarial example\"?\n- Page 3 last line: \"pertrubations\" typo\n- Page 4 above Eqn. (7) \"in Equation 6 is often solved\" -> \"is often approximately solved\"\n- Page 6: \"Lin et al highlights\": \"Lin et al\" is in plural form, so it should be \"Lin et al highlight\" Same to several other occurrences.\n- Page 10: \" It is not enough for targeted adversarial examples to be close to the target class but not far from the true class.\" seems not very clear.\n- Eqn. (41): abs R_i seems to be a typo\n- Below Eqn. (45), $S_i$ denotes the set of activations. What is \"the set of activations\"?\n- Last line in Page 21, $L - p$ -> $L_p$", "broader_impact_concerns": "The work may need to add a Broader Impact Statement to summarize the potential malicious use of adversarial transferability-based attacks and potential mitigations.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ydjNvbQkjK", "reviewer": "Reviewer_7rTq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n1. To the best of my knowledge, this is the first survey on adversarial example transferability.\n2. The authors introduce many research works about enhancing adversarial transferability, which will benefit other researchers interested in the transferability of adversarial examples.\n\n# Weaknesses\n1. The authors mainly focus on introducing transferability-enhancing techniques. While I’m unaware of all the literature on adversarial transferability, it would be better to include more discussions on theoretical findings or explanations about what causes adversarial transferability.\n2. The authors should spend more time proofreading the writing and correcting mistakes in citation, spelling, typos, etc. In particular, typos in mathematical notations must be corrected; a survey paper should convey information as accurately as possible. Please take a look at** Requested Changes** for more detailed comments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper summarizes the existing literature on adversarial example transferability. Specifically, the authors cover a massive amount of research on transferability, organized by various topics: how to assess the transferability, two categories (surrogate model-based and generative model-based) of transferability-enhancing methods, and transferability-enhancing methods in diverse domains. The authors also discuss the current challenges and research opportunities in adversarial transferability.\n\nThe paper introduces preliminary materials to cover the basic knowledge of adversarial attack transferability. Additionally, the authors summarize the three existing evaluation metrics that measure adversarial transferability: Fooling rate, interest class rank, and knowledge transfer-based metrics.\n\nThe next two sections (Section 3 and Section 4) are the main parts that cover transferability-enhancing methods, mainly on image classification tasks. Specifically, the authors divide the existing transferability-enhancing techniques into two categories: One is based on the surrogate model, and the other is based on the generative model. The surrogate model-based method uses a surrogate model that approximates the target model, and the attacks described in Section 3 aim to improve the transferability of adversarial examples when attacking the surrogate model. The authors introduce an optimization form of the surrogate model-based attack. Then, the authors further divide this category into four subcategories according to the part on which the methods concentrate: data augmentation, optimization, model, and loss. The generative model-based method uses a generative model to generate adversarial examples (or adversarial perturbations). The authors introduce generative model-based methods in Section 4 from two perspectives: The methods' effectiveness and efficiency.\n\nSection 5 covers transferability-enhancing methods in machine learning domains other than image classification. Specifically, the authors briefly go through transfer attacks in vision tasks (e.g., object detection, segmentation, 3D point clouds, video, etc.), transfer attacks in NLP tasks, and even domain-invariant adversarial attacks.\n\nFinally, the authors list some existing challenges in adversarial transferability research, e.g., better choices of surrogate model, how to evaluate or benchmark transferability, etc.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Consider renaming the two categories of transferability-enhancing methods, i.e., surrogate model-based and generative model-based.\n   * The current category names might look like generative model-based methods have nothing to do with the surrogate model, but they also use a surrogate model in the formula.\n   * The generative model is a method that generates adversarial examples. However, the surrogate model is not an attack method, but the optimization generates adversarial examples for the methods in Section 3. I’d name this category optimization-based methods. (This may require renaming the subcategory in Section 3.2, though.)\n2. I can see a few typos in mathematical notations/formulae. I’ll list some of those mistakes, but I cannot guarantee that I covered all the formulae. Please review all the formulae and ensure there are no typos.\n   * In Table 2, $H_k^i$ should be $H_k^l$ because it is about $l$-th layer.\n   * Equation 4 should be corrected to $\\alpha_2^{f_s\\rightarrow f_t}=\\|\\mathbb{E}_{x\\sim D}[\\widehat{\\Delta _{f_s\\rightarrow f_t}(x)} \\cdot \\widehat{\\Delta _{f_s\\rightarrow f_t}(x)}^\\top]\\|$. (i.e., move $\\top$ inside the brackets.\n   * In Equation 8, exchange the probability $p$ and $1-p$. I checked Xie et al. paper, and $p$ is the probability of applying the transformation.\n   * In **Natural, Unrestricted and Non-Additive Attacks** (Page 21), $L-p$ in the last sentence looks like a typo of $L_p$\n3. Sometimes, the authors use terms or notations without explaining what they mean. Please describe the missing parts.\n   * In **ADMIX** description (Page 6), the authors used the term \"Mixup\" without explaining it. Please explain Mixup briefly with a citation.\n   * In **AI-FGTM** description (Page 7), the adaptive step size uses an unseen notation $T$. I believe that the author wanted to write $\\alpha^{(t)} = \\frac{\\varepsilon}{ \\sum_{s=0}^t \\frac{1-\\beta_1^{s+1}}{\\sqrt{1-\\beta_2^{s+1}}}} \\frac{1-\\beta_1^{t+1}}{\\sqrt{1-\\beta_2^{t+1}}}$ (I changed the running index to be $s$, running from 0 to $t$)?\n   * In Equation (31), the authors used an unseen notation $\\gamma$ without explanation. I had to read Fang et al. to understand what it means. It looks like this $\\gamma$ is some decay factor that determines the augmented model $f$, so Fang et al. replaced $f$ by $\\gamma$. I think the authors tried to clarify this in the next line but made another mistake in the formula. Correct the formula to $G(\\mathbf{x} _{adv}, \\gamma)=G(\\mathbf{x} _{adv}, f) = \\nabla _{\\mathbf{x} _{adv}} \\mathcal{L}(f _{\\mathbf{x} _{adv}}, y)$ and explain the context and clarify the notation.\n   * In **Conditional Adversarial Distribution (CAD)** (Page 17), the authors mention that Feng et al. proposed to transfer a subset of parameters based on CAD, but the authors do not explain anything about CAD. I can only see that it is an abbreviation of Conditional Adversarial Distribution, but what does it even mean? Please describe the concept to make it more understandable.\n4. In Table 1, the untargeted attack success is determined by the misclassification of a model. However, Equation 1 uses \"change of prediction\" as an attack success. Either change the description of the untargeted attack in Table 1 or change Equation 1 to $\\arg\\max_i f_t^i(x^{adv})\\ne y,\\text{ given }\\arg\\max_i f_s^i(x^{adv})\\ne y$.\n5. The first paragraph in Section 3.2 contains an interesting perspective. Can we interpret other methods (out of Section 3.2) from this viewpoint, i.e., data augmentation as another approach to enhance the generalization?\n6. In Section 3.2, separate the iterative variants of FGSM from the methods for adversarial transferability. You can add a `\\subsubsection` summarizing the variants or move them to other sections, e.g., Appendix.\n7. Consider addressing the following minor issues regarding the paper writing.\n   * According to the [resource about how to format the paper](https://github.com/JmlrOrg/tmlr-style-file/archive/refs/heads/main.zip) in the [Author guidelines](https://jmlr.org/tmlr/author-guide.html), we distinguish in-text citations and in-parenthesis citations. In particular,\n\n    > When the authors or the publication are included in the sentence, the citation should not be in parenthesis,\n\n      You should use `citet{}` in this case, and you should use `citep{}` otherwise. I see too many exceptions to this rule. Please fix all the citations properly.\n\n   * Run a spell checker or proofread to catch minor typos, e.g., “generaator”.\n   * In **RDI** description (Page 5), \"Similarly to SI\" should be changed to \"Similarly to SIM\".\n   * In **Relation to Transferability across Images** (Page 21), the paragraph's first sentence is redundant.", "broader_impact_concerns": "I don’t see a particular broader impact concern regarding this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4Zk1q5RJVB", "reviewer": "Reviewer_ifNX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n* The survey paper is well-organized, effectively structuring the existing body of research on adversarial transferability.\n* It includes an extensive survey of adversarial transferability, commendably encompassing both NLP tasks and a variety of vision tasks.\n* The paper does an excellent job in clearly outlining the current challenges faced in the field of adversarial transferability.\n\n**Weaknesses:**\n* While the survey provides a comprehensive overview, it would be greatly enhanced by a more general conclusions and common findings. Offering insights or perspectives on the potential future directions of adversarial transferability research would add significant value to the paper.\n* The paper, although informative, is somewhat marred by numerous grammatical errors. A thorough review and editing for language accuracy would enhance its clarity and professional impact.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper offers a comprehensive summary of prior research in the field of adversarial transferability. The author skillfully categorizes the existing approaches into two distinct methodologies: first, the surrogate model-based adversarial transferability, and second, the adversarial transferability utilizing generative models. Moreover, the author broadens the scope of discussion by introducing a variety of research contributions in natural language processing (NLP) and various vision tasks beyond the conventional realm of image classification. This inclusive approach enriches our understanding of the topic and its diverse applications.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Major comments\n* Offering insights or perspectives on the potential future directions of adversarial transferability research would add significant value to the paper.\nMinor comments\n* There appears to be an inconsistency in the use of `\\citet` and `\\citep`. For improved clarity, it's advisable to use `\\citet` when the citation is a subject of the sentence and `\\citep` in other cases. Consistent application of these citation formats would enhance the paper's readability and scholarly accuracy.\n* In Section 3.1, there's a mismatch between the paragraph titles and their subjects. For example, DIM is mentioned as the subject in the first sentence, but SIM is used as the paragraph title. A similar issue is noticed with the usage of RDI, which might be intended as a subject but seems to have a grammatical error. Aligning the paragraph titles with the subjects discussed within each would improve the structure and coherence of this section.\n* A typo error is present in the section titled \"Challenges and Opportunities.\"\n* The paper contains numerous grammatical errors. A comprehensive proofreading and editing process is recommended to enhance readability and professional presentation.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While the paper does not offer any new or novel approaches to adversarial transferability, the consensus is that the paper provides an extensive literature review and will benefit future research. Concerns were raised regarding the comprehensiveness, depth of analysis and comparison, and presentation quality. These concerns were addressed by the authors during the discussion phase, significantly enhancing the paper's quality.
Decision: Accept (poster)