Title: Perturbation Towards Easy Samples Improves Targeted Adversarial Transferability
Venue: NeurIPS 2023
Accepted: yes
Keywords: Adversarial Attacks; Generative Attack; Transferable Targeted Attack
Abstract: The transferability of adversarial perturbations provides an effective shortcut for black-box attacks. Targeted perturbations have greater practicality but are more difficult to transfer between models. In this paper, we experimentally and theoretically demonstrated that neural networks trained on the same dataset have more consistent performance in High-Sample-Density-Regions (HSDR) of each class instead of low sample density regions. Therefore, in the target setting, adding perturbations towards HSDR of the target class is more effective in improving transferability. However, density estimation is challenging in high-dimensional scenarios. Further theoretical and experimental verification demonstrates that easy samples with low loss are more likely to be located in HSDR. Perturbations towards such easy samples in the target class can avoid density estimation for HSDR location. Based on the above facts, we verified that adding perturbations to easy samples in the target class improves targeted adversarial transferability of existing attack methods. A generative targeted attack strategy named Easy Sample Matching Attack (ESMA) is proposed, which has a higher success rate for targeted attacks and outperforms the SOTA generative method. Moreover, ESMA requires only $5\%$ of the storage space and much less computation time comparing to the current SOTA, as ESMA attacks all classes with only one model instead of seperate models for each class. Our code is available at https://github.com/gjq100/ESMA

Perturbation Towards Easy Samples Improves Targeted Adversarial Transferability

Abstract

The transferability of adversarial perturbations provides an effective shortcut for black-box attacks. Targeted perturbations have greater practicality but are more difficult to transfer between models. In this paper, we experimentally and theoretically demonstrated that neural networks trained on the same dataset have more consistent performance in High-Sample-Density-Regions (HSDR) of each class instead of low sample density regions. Therefore, in the target setting, adding perturbations towards HSDR of the target class is more effective in improving transferability. However, density estimation is challenging in high-dimensional scenarios. Further theoretical and experimental verification demonstrates that easy samples with low loss are more likely to be located in HSDR. Perturbations towards such easy samples in the target class can avoid density estimation for HSDR location. Based on the above facts, we verified that adding perturbations to easy samples in the target class improves targeted adversarial transferability of existing attack methods. A generative targeted attack strategy named Easy Sample Matching Attack ( ESMA ) is proposed, which has a higher success rate for targeted attacks and outperforms the SOTA generative method. Moreover, ESMA requires only 5% of the storage space and much less computation time comparing to the current SOTA, as ESMA attacks all classes with only one model instead of seperate models for each class. Our code is available at https://github.com/gjq100/ESMA

1 Introduction

Deep learning models exhibits substantial computational capacity in many downstream tasks, but are vulnerable to adversarial attacks [1, 2]. Such attacks tend to be transferable [3, 4] as well as real-world achievable [5], which makes it implementable in black-box scenarios. Targeted attacks are known to be more difficult to transfer [4, 6] compared with non-targeted attacks.

Intuitively, directions and transferability of adversarial attacks are closely related, but their relationship is rarely discussed. [7] found that samples in low density regions of ground truth distribution are more susceptible to adversarial attacks. They also verified that adversarial perturbations that aligned with the low density direction of ground truth distribution can lead to better transferability. For targeted attack scenarios, the direction that can bring more transferability has not been fully researched yet is not clear.

Findings and arguments. In non-targeted scenarios, directing towards low-density regions of the ground-truth distribution can improve adversarial transferability [7]. However, this approach is not direct enough for targeted attacks. As shown in Figure 1, perturbations pointing at High-Sample-Density-Regions ( HSDR ) of the target domain are more effective than that pointing at low-density regions of the ground-truth distribution, it perturbates samples more directly to the target discrimination region. Moreover, we demonstrate and theoretically prove that neural networks tend to have more consistent outputs in the HSDR of each class, which means that perturbations pointing to the HSDR of the target class are also transferable between different models. However, density estimation for samples in high dimension is challenging.

Fortunately, by the definitions of hard samples and easy samples in [8], we found a fact that helps, in each category, easy samples with smaller losses are more likely to locate in HSDR. We provide theoretical and experimental assurance for this fact. Based on this fact, we can directly perturbate towards such easy samples of the target domain to improve transferability without density estimation.

Related works. Adversarial attacks can be divided into white-box attacks and black-box attacks. In the white-box setting, attackers have access to the model's structure and parameters, while in the black-box setting, they have no such information but only access to the input and output of the model. This is typically the scenario encountered in real-world situations. Black-box attacks include querybased attacks [9, 10, 11] and transfer-based attacks [12, 13, 14]. Conducting too many queries is impractical in real-world applications. In contrast, transfer-based attacks are more feasible as they do not require queries. Transferbased attacks often require the use of a white-box surrogate model to produce adversarial perturbations. In terms of the way adversarial perturbations are generated, there are

TowardsHSDRof

Q

target

8

domain

0

o

o

Towardslow densityregion

ofground-truthdistribution

。。

two approaches: iterative instance-specific methods and generative methods. Iterative instancespecific methods utilize model gradients to iteratively add perturbations to specified samples (e.g., FGSM [3], C&W [5], PGD [15]). To enhance the transferability of adversarial samples, subsequent work combines these methods with various techniques, such as introducing momentum [12, 13] and considering input transformations [13, 14, 16, 17] during iterations, training auxiliary classifiers [18, 19], or substituting different loss functions [20, 21, 22]. However, instance-specific methods are primarily designed for non-targeted scenarios, often lacking effectiveness in targeted settings. Although relatively good results have been achieved for targeted attacks [20, 21, 23], instance-specific methods still require iteratively creating perturbations for each specified sample, while generators trained for generating adversarial perturbations can be generalized on more samples after training [24, 25, 26, 27].

The leading perturbation generative method currently is TTP [24], which employs a target-specific GANtoalign clean and augmented data from the source domain with the target domain data. However, TTP necessitate training a dedicated generator for each class. This significantly increases storage requirements and training time. In our work, we design a generator for simultaneous attacks on all target classes, which uses class embedding information for each target class. To construct these embeddings, we adopt techniques similar to SNE [28] to align the surrogate model's output logits for target classes with the generator's embeddings. This enables latent features learned by the surrogate model to guide the construction of well-structured class embeddings. Building on our findings, we train a multi-class perturbation generator to simultaneously perturb the source domain samples towards easy samples with low loss of each target class. Experiments on the ImageNet dataset shown that our method can obtain a better target transfer success rate than TTP, while requiring much less storage. Our method also has certain advantages in training time.

Therefore, our contribution can be summarized as follows:

We found that deep learning models have more consistent outputs in HSDR of each class, as demonstrated theoretically and through experiments (Section 2.1). This implies that adding

adversarial perturbations pointing to the HSDR of the target class results in better targeted adversarial transferability.

We experimentally and theoretically verified that easy samples with low loss in earlystopping models are likely to be located in HSDR, which allows us to directly add perturbations pointing to such samples of the target class to improve targeted adversarial transferability (Section 2.2). This avoids density estimation of high-dimensional samples, which is challenging and computationally expensive.

We introduced the Easy Sample Matching Attack (ESMA) (Section 3). ESMA achieves a higher targeted transfer success rate compared to SOTA generative attacks TTP. Furthermore, it only needs one model to perform attacks for all target classes (Section 4), which requires much less storage space than TTP (only about 1/20 of the storage space in 10 targets case), and requires less training time.

Our findings and conclusions can not only provide guidance for target transfer attacks, but more importantly, they reveal the consistency of deep learning models in the HSDR, and the correlation between sample difficulty and local sample density. Not only that, the design of our multi-target perturbation generative model can provide a new reference for subsequent related research.

2 Main conclusions

In this section, we combine illustrative experiments and theoretical proofs to illustrate our two conclusions in turn, and we use the following notations and definitions. Z = X × Y ∈ R d × R is the sample space. The i.i.d. dataset S = { x i , y i } n i =1 consists of n sample pairs ( x i , y i ) , 1 ≤ i ≤ n . Given y , the conditional distribution of x is D x | y . Feature mapping f : X → R K , where K is the number of class, the class of feature mapping f ∈ F . Specifically, the parametrized class F w := { f w : X → R K , w ∈ W } , where W is the parameter space. Define F S w := { f S w = S ◦ f w : f w ∈ F w , S ◦ f w ( x ) = S ( f w ( x )) } , S denotes Softmaxtransformation. The Softmax-Cross-Entropy loss is denoted as glyph[lscript] sce ( · , · ) : R K × R → R . Let S j := { ( x, y ) ∈ S : y = j } , I j := { i : ( x i , y i ) ∈ S j } , C j := { x ∈ X : ( x, y ) ∈ Z , y = j } .

Definition 1 ( ( j, x 0 , r ) -Local sample density) Given a class j ∈ [ K ] , ( x 0 , y 0 ) ∈ S j , the ( j, x 0 , r ) -Local sample density:

where B ( x 0 , r ) := { x ∈ R d : ‖ x -x 0 ‖ ≤ r } , vol B ( x 0 , r ) denotes the volume of B ( x 0 , r ) .

Now we use the notation I ( j,x 0 ,r ) = { i : ( x i , y i ) ∈ S j , x i ∈ B ( x 0 , r ) , ( x 0 , y 0 ) ∈ S j ) } , C ( j,x 0 ,r ) = { x : x ∈ C j , x i ∈ B ( x 0 , r ) , ( x 0 , y 0 ) ∈ S j ) } . For simplicity, we denote glyph[lscript] sce ( f w ( x ) , y ) as glyph[lscript] ( w , x ) and define the local empirical risk R ( j,x 0 ,r ) ( w ) = 1 | I ( j,x 0 ,r ) | ∑ i ∈I ( j,x 0 ,r ) glyph[lscript] ( w , x i ) .

2.1 The Output Consistency of Different Deep Learning Models in HSDR

We conduct an experiment to explain why samples in low-density regions of ground-truth distribution are vulnerable to attacks, and verify the consistency of the output of the deep learning model in HSDR. A dataset consist of 200 samples is constructed by sampling from two 2 -d Gaussian distributions with equal probability, which is then used to train a neural network for classification. Subsequently, we plot the discriminant region of the Bayes' criterion (which has the minimum error rate) for the known ground truth prior distribution and compared it with the discriminant region of the trained classifier. In addition, we train two other neural networks with different structures and parameter quantities, and plot the output differences of the three neural networks (Figure 2). The discriminant region in Figure 2(a) can reach the Bayesian error rate. The intersection of the two population distributions in the middle of the probability density curve (as shown in Figure 2(b)) belongs to the low-density region of the ground-truth distribution and also to the misclassified region. In such a region, even with minimal expected error, the Bayesian discriminant criterion will classify samples with relatively small probability density of a single population into another class. These samples can be thought of as "outliers". The trained NN classifier discriminates samples into their original categories, causing a

(a)

(b)

(c)

(d)

4

0.875

1.000

0.875

1.000

0.875

1.000

3

HSDR

0.4375

0.5000

2

0.750

0.625

2

0.750

0.625

2

0.750

0.625

Z

1

00

0.3750

0.3125

0

00S'0

0

005'0

0

0.500

0

0.2500

0.375

0.375

0.375

-1

HSDR

0.1875

-2

0.125

0.250

-2

0.125

0.250

-2

0.250

0.125

Z-

E-

Sample domain

0.0625

0.1250

0000

0.000

0.000

0.0000

2

0

2

-2

0

2

-2

0

2

4

-3

2

-1

0

1

2

3

4

difference in decision boundaries from the Bayesian prior classifier and creating small pits as shown in Figure 2(a). Most of the samples around the pit (Figure 2(c)) belong to another class due to its lower population density compared to the other class. Hence, perturbing the samples from this pit (i.e. outliers) towards the discriminant region of the other class becomes easier. If other trained neural networks can accurately classify such outliers, they will also generate similar pits. By adding perturbations pointing these pits to samples from another class, they will be perturbed into such pits, enabling the transfer of adversarial samples between different classifiers, which explained the findings of [7].

However, as we stated in Figure 1, perturbations towards the low-density regions of the ground-truth distribution are not direct enough for targeted attacks, while perturbations towards the HSDR of the target class are more direct. Combining this with Figure 2(d), we observe that different classifiers have more consistent outputs in the HSDR. With Theorem 1, we theoretically verified this, which also implies that perturbing samples to the HSDR of the target class leads to better targeted adversarial transferability.

Theorem 1 (Local output consistency) For a target class j ∈ [ K ] , and two different parametrized class F w 1 := { f w 1 : w 1 ∈ W 1 } , F w 2 := { f w 2 : w 2 ∈ W 2 } , assume that 1 | I ( j,x 0 ,r ) | ∣ ∣ ∣ ∑ i ∈I ( j,x 0 ,r ) ( f S k w 1 ( x i ) -f S k w 2 ( x i ) ) ∣ ∣ ∣ ≤ γ , then for any sample ( x 0 , y 0 ) ∈ S j , in the neighborhood B ( x 0 , r ) , with probability at least 1 -δ , the following holds:

Remark. Theorem 1 suggests that different models have a more consistent output near the samples in HSDR, specifically, the difference between the outputs has a bound of order ˜ O (√ Kd d/ 2 ρ ( j,x 0 ,r ) 2 d r d ) . When the number of classes and dimensions are fixed, a larger sample density results in more consistent performance, a weaker consistency within smaller neighborhoods. Note that this local consistency is weakened as the dimension increases, but the relative output consistency between HSDR and LSDR does not change.

However, a very practical problem is that sample points in high-dimensional space tend to be very discrete due to the curse of dimentionality [29], which makes it difficult to find a suitable local neighborhood size to calculate the local density of samples, moreover, in the case of large datasets, calculating local density can be computationally expensive. But the conclusion of the next section can help us find the sample located in the HSDR using early-stopping models without directly calculating the local density of the sample.

2.2 Correlation between Local Sample Density and Sample Difficulty

In this section we illustrate the correlation between sample density and sample difficulty. Hard samples have attracted much attention in various tasks because of their significance for training

convergence and model generalization [8, 30, 31, 32]. [8] proposed that for the convergent model, the difficulty of the sample can be measured by the loss gradient norm of the sample, easy samples has a relatively small loss gradient norm, while the loss gradient norm of difficult samples is relatively large, especially, those with too large gradient norm may be outliers.

We use the following theoretical analysis to illustrate that for a trained classifier, the samples in the HSDR tend to have a smaller local empirical risk. At the same time, smaller loss gradient norms ensure that losses are more consistent in small neighborhoods, which implies those samples with both smaller loss and loss gradient norms guarantee less local empirical risk in the neighborhood where the sample is located. To illustrate the following conclusions, we make several mild assumptions:

Assumption 1 (Smoothness assumption) glyph[lscript] ( w , x ) satisfies the following conditions of Lipschitz continuous gradient:

Assumption 2 ‖∇ w glyph[lscript] ( w , x ) ‖ ≤ G for all w ∈ W .

Assumption 3 (Polyak-Łojasiewicz Condition) R ( j,x 0 ,r ) ( w ) satisfies the PL-condition:

Assumption 4 For ∀ i ∈ I ( j,x 0 ,r ) , the following holds:

Assumption 1 and 2 were made in [33], [34] and [35], where Assumption 2 can actually be derived from Assumption 1 when the input space is bounded, which is usually satisfied. Even a non-convex function can still satisfy Assumption 3 [34], the inequality in Assumption 3 implies that all stationary points are global minimum [36], which is proved in recently works [37, 38] for over-parameterized DNNs. Assumption 4 is reasonable when local neighborhood radius r is small.

Algorithm 1 Mini-batch SGD

Require: Initialized weights w 1 , total steps T , sample set S , batch size M and step size η t .

1: for t = 1 ← T do

2: Randomly sample M different samples B t from S , where batch size | B t | = M , corresponding indicator set denote as I B t .

3: w t +1 = w t -1 M ∑ i ∈I Bt ∇ w glyph[lscript] ( w t , x i )

4: end for

Return:

w

T

+1

Consider the local empirical risk under Algorithm 1, we use Theorem 2 to illustrate the convergence rate relies on local density.

Theorem 2 (Optimization relies on local density) Given a learnable parametrized class F w := { f w : X → R K , w ∈ W } , let w t updated by Algorithm 1, under Assumption 1, 2, 3 and 4, set η t = 1 βµt and T ≤ L 1 2 βµ , then with probability at least 1 -δ , holds

According to theorem 2, the local empirical risk R ( j,x 0 ,r ) ( w ) will reach a more faster convergence rate in HSDR and, relatively, a relatively slower convergence rate in LSDR, thus for early-stopping models, they has a lower local empirical risk in HSDR. Combined with the following Proposition 1, we show that a smaller gradient norm guarantees a smaller local empirical risk. For overfitting models, the local empirical risk of each neighborhood where samples are located may be very small, but for early-stopping models, this relativity of local empirical risk with respect to local sample density can be maintained.

Proposition 1 Under Assumption 1, given any ( x i , y i ) ∈ S , for any x ∈ X that satisfies ‖ x i -x ‖ ≤ r , the following holds:

Remark. Proposition 1 suggests that minimizing the loss leads to a smaller norm of the loss gradient. However, this constraint becomes less tight as the neighborhood radius r decreases. Compared with the loss itself, the loss gradient norm further guarantees the proximity of local loss, especially when r is small. Therefore, samples with smaller loss and smaller loss gradient norm are more likely to be in a neighborhood with smaller local empirical risk.

Using the above conclusion, for an early-stopping model, samples with smaller losses and loss gradient norms tend to be located in HSDR. Still in the example in the previous section, we train three models with early-stopping, and the learning rate adjusted with the number of steps, and then plotted Figure 3. The model has more consistent outputs in HSDR. When the loss and gradient norms of a sample are small, the region where the sample is located has smaller local empirical risk. Samples in HSDR have smaller local empirical risk, which is consistent with our theoretical analysis. Additionally, the rightmost graph of Figure 3 shows that samples with smaller loss and gradient norms often locate in HSDR, validating our conclusions.

Therefore, we can determine whether a sample is more likely to located in HSDR or LSDR by evaluating whether it has smaller loss and gradient norms simultaneously. This eliminates the need to calculate local sample densities to find samples in HSDR.

ifference of Model Output

0.8

1.0

Median

Mean

Local Empirical Risk

1.0

0.8

Median

Mean

Local Empirical Risk

1.0

0.8

Median

Mean

1.0

0.8

Median

Mean

Local Density

0.6

90

0.6

0.6

0.4

0.4

0.4

0.4

D

0.2

0.2

0.2

0.2

0.0

0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1

0.0

0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1

0.0

7'0-0

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1

0.0

0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1

Local Density

Loss+Gradnorm

Local Density

Loss+Gradnorm

We conduct transfer attack experiments on three baselines on the CIFAR10 dataset, three different models are chosen as victims. Combining our perspectives above, we use algorithm 2 to select the anchor of each target class for guiding the addition of adversarial perturbations. We implement our strategy by simply using squared loss to match anchor points, i.e. minimizing ∥ ∥ f ( x adv i ) -a target i ∥ ∥ 2 (choosing q = 10 and glyph[epsilon1] = 16 pixels). For comparison, we also use cross-entropy (CE) loss to calculate adversarial examples in the vanilla way, as well as using squared error without a screening mechanism (i.e., randomly selecting the target anchor in the target class) to exclude the influence of different losses. The results are shown in Table 1, our strategy indeed helps to enhance the transferability of target attacks, which further confirms our viewpoints.

Algorithm 2 Target Anchor Screening

Require: Early-stopping classifier f w , screening parameter q and sample set S .

1: for i = 1 ← n do

2: loss i = glyph[lscript] sce ( f ( x i ) , y i ) , gradnorm i = ‖∇ x glyph[lscript] sce ( f ( x i ) , y i ) ‖ .

3: end for

4: For each class k ∈ [ K ] , select the q -th smallest loss and the gradient norm among the samples in that class as thresholds thr loss k and thr gradnorm k

5: for k = 1 ← K do

Attack

Src:Res34

(95.44%)

Src:VGG16

(94.27%)

Src:Dense121

(95.47%)

→

VGG16

→

Dense121

→

Res34

→

Dense121

→

Res34

→

VGG16

MIM 14.32%/12.94%/

14.44

% 19.81%/19.00%/

20.09

% 14.80%/13.78%/

15.13

% 13.41%/12.11%/

13.83

% 17.13%/14.17%/

17.17

%11.23%/10.27%/

11.25

%

13.53%/12.50%/

13.88

% 15.23%/14.56%/

15.71

% 12.49%/11.78% /

12.88

%11.14%/ 10.17% /

11.40

%15.74%/14.61%/

15.89

%11.93%/10.94%/

12.09

%

DIM 15.46%/14.39%/

15.97

%18.10%/ 17.61%/

18.96

% 14.56%/13.28%/

15.12

% 12.97%/12.11%/

13.47

% 18.11%/16.83%/

18.33

%13.18%/11.72%/

13.26

%

3 Training Strategy of ESMA

We present our training strategy in this section, as shown in Figure 4. Our training strategy is carried out in two steps, the first step we pre-train the generator's embedding representations, and the second step is to find easy samples of each target class based on our previous conclusions, and then use these samples to guide the generator to generate perturbations from the source domain to the target domain.

Pre-trained Embeddings Guided by Latent Features To obtain better embeddings that more effectively represents inter-class information, since the latent space of deep learning models often extracts enough class information [39], we treat the output features of the local model as a set of a priori embedding. In order to make the generator embedding have a similar structure to such a priori embedding, we refer to the idea of manifold learning, using a strategy similar to the SNE algorithm [28]. Let µ j = 1 |I j | ∑ i ∈I j l i , where l i = f ( x i ) , and then we design the following manifold

TargetDomain

Feature

Samples

Cis

Source model

Embeddings

Extraction

eis

C.M

Samples

o

业

Easy

Hard

Pretrain

Embeddings

Generator

Screening

Sample

f(atarget,)s

W *g(ci)s

Generator

Perturbate, projection

and Clip

arad

Source model

LEM

BackPropagation

(xadu)

SourceDomain

matching loss. Generator embeddings of various class were pulled to the manifold that output features in to obtain embeddings with a better structure.

Next, we denote the generator embedding of class j as e j . The four matrices M S euc , M E euc , M S cos , M E cos satisfy M S euc i,j = ‖ µ i -µ j ‖ , M E euc i,j = ‖ e i -e j ‖ , M S cos i,j = µ i · µ j ‖ µ i ‖‖ µ j ‖ , M E cos i,j = e i · e j ‖ e i ‖‖ e j ‖ , respectively. Let

then our manifold matching loss is as follows:

The last regular term is to prevent losses from collapsing. λ 1 and λ 2 are hyperparameters. The pretrained embedding with our strategy has a larger Euclidean distance and a smaller cosine similarity, which greatly alleviates the previous clustering phenomenon. More detailed analysis and discussion are provided in Appendix B.

Training of Multi-target Adversarial Perturbation Generators After pre-training embedding, we freeze the parameters of the embedding layer, combined with our previous conclusions, we select several easy samples in each class to form target anchor sets A i , and match them with the output of our generator in feature space, the generator we use is a Unet with Resblocks. We propose the following objective easy sample feature matching loss to train the multi-class adversarial generator.

where clip glyph[epsilon1] ( x ) = clip (min ( x + glyph[epsilon1], max( x, x -glyph[epsilon1] ))) , W is a differentiable Gaussian kernel with size 3 ∗ 3 , such smoothing operations have been demonstrated to further improve transferability[24]. The measure of the distance between the two features d ( · , · ) is Smooth L1 loss, which has a unique optimal solution that is not sensitive to exceptional values [40]. Then, our final training strategy can be represented by algorithm 3.

glyph[negationslash]

4 Experiments

In this section, we verify the effectiveness of our method through experiments on ILSVRC2012 dataset [41]. To evaluate the effectiveness of different components in our strategy, we conduct ablation experiments in Section 4.3. Other ablation experiments can be found in Appendix C.

4.1 Experiment Setup

Dataset The dataset we used comes from the ILSVRC2012 dataset [41], in which we selected ten classes as the training set, which refer to the ten classes used for TTP training in [24], they are 24 , 99 , 198 , 245 , 344 , 471 , 661 , 701 , 802 , 919 . We train generators using images of these ten classes in the training set ( 1300 images per class), and use the images in the validation set ( 50 images per class) as the validation dataset for the targeted attack.

Models We use the four networks used in [24] as source models -ResNet-50 [42] (Res50), VGG19-bn [43] (VGG19bn), DenseNet-121 [44] (Dense121), ResNet-152 [42] (Res152). Except the above four models, we also select three models from the Inception series: Inception-v3 [45] (Inc-v3), Inception-v4 [46] (Inc-v4), Inception-ResNet-v2 [46] (IncRes-v2) and a transformer vision model, VIT [47]. As the analysis and fundamental assumptions of this paper are based on the condition of training data from the same distribution, in order to explore the transferability between models trained on training data with different distribution, we conduct additional transfer attack on two

adversarially trained models, Inc-v3-adv [48] and IncRes-v2-ens [49]. Results are shown in E.2. In addition, we also test the adversarial transferability of ESMA on the scenario where the source model is an ensemble of different models, corresponded results are reported in E.1.

Baselines We select many iterative instance-specific attack benchmarks, MIM [12], SI-NIM [13], TIM [14], DIM [16], and advanced iterative instance-specific attacks that are competitive in target setting, Po-TI-Trip [20], Logit [21], Rap-LS [23], FGS 2 M[50], DMTI-Logit-SU [51], S 2 I-SI-TIDIM [52]. Generative adversarial attacks HGN [53] and TTP [24] also included in our comparision, where TTP is the current SOTA generative method. As a generative attack, TTP requires training a class-dependent generator specifically for each class.

Parameter Setting For the parameter settings of different attack methods, we refer to the default settings in [20], total iteration number T = 20 , step size α = glyph[epsilon1]/T , where the glyph[lscript] ∞ perturbation restriction glyph[epsilon1] is set to 16 . Momentum factors µ is set to 1 , for the stochastic input diversity in DIM, we set the probability of applying input diversity as 0 . 7 . For TIM, the kernel-length is set to 7 , which is more suitable for targeted attacks. For Po-TI-Trip, the weight of triplet loss λ is set to 0 . 01 , while the margin γ is set to 0 . 007 . Referring to [21], the number of iteration steps of Logit is set to 300 , and for RAP-LS, we choose 400 iteration steps, K LS is set to 100 , and glyph[epsilon1] n is set to 12 / 255 [23]. Then for TTP, since we used a relatively small training set, we added 10 epochs to the original paper [24] settings to ensure the performance of the model, and the learning rate of Adam optimizer is 1 e -4 ( β 1 = . 5 , β 2 = . 999 ). Finally, for our model, we used the AdamW optimizer to train 300 epochs with a learning rate of 1 e -4 ( 350 for cases where the source model is VGG19bn or Dense121), the value of q used for sample screening is set to 2 .

Evaluation Setting We train the generator on the training set of the selected ten classes, verify the targeted transferality on the validation set, for each target class, we use the 450 images of the remaining classes as the source data and perturb them to the target class. For the iterative instancespecific attacks, we directly attack these instances and test their targeted transfer success rate. All methods (including training) were implemented on a single NVIDIA RTX A5000 GPU.

4.2 Results

The results of our experiments, reported in Table 3, ESMA outperforms current SOTA generative attack TTP in targeted transfer success rates. Our approach demonstrates significant advantages in terms of efficiency and effectiveness, with TTP having a parameter count of 7 . 84 M per model compared to ESMA's 4 . 40 M. Additionally, ESMA achieves an average training time that is 78 . 4% of TTP. The experimental results also strongly supports our viewpoint.

4.3 Ablation Studies

To validate the effectiveness of the two components in our strategy design, we compared the cases with (w/) and without (w/o) pretrained embeddings and target anchor screening. The results are shown in Figure 5. In the case without pre-trained embeddings, the performance is significantly weaker than the case with pre-trained embeddings across different training durations. Moreover, the case with target anchor screening shows a noticeable improvement compared to randomly selecting target anchors.

To verify that ESMA can indeed increase the sample density of the target class for the original samples, we use the samples used for adversarial testing in table 3. First, we calculate the sample density

of the target class for the clean samples ρ ( target ,x i ,r ) (here r is set to 600 ). Then, we apply ESMA to generate adversarial samples x adv i , and calculate the sample density of the target class for these adversarial samples ρ ( target ,x adv i ,r ) . In both cases, the density calculation results were averaged across all target classes for each sample. We normalize the results by dividing them by the maximum value among all results in both cases. Additionally, we further bin and count the number of test samples in different intervals under different perturbation constraints. The results are shown in Table 2. The

target class sample density of the samples perturbated using ESMA has increased in bins with larger values, which confirms that ESMA can indeed enhance the target class sample density.

5 Conclusion

In this work, we provided a novel perspective on the transferability of targeted adversarial attacks. The study theoretically and experimentally demonstrated that adding perturbations towards HSDR of the target class can further enhance the transferability of targeted attacks. We also proposed a method for identifying samples within HSDR, which avoided the impracticality of density estimation in high-dimensional data. Building upon these insights, we introduced an improved generative multi-target attack strategy ESMA, which surpassed previous generative targeted attacks in both effectiveness and efficiency. We believe that our insights not only provide guidance for targeted attacks but also offer insights for dataset selection. Further discussion on these points would be included in our future work.

References

Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR , 2014.

Biqing Qi, Bowen Zhou, Weinan Zhang, Jianxing Liu, and Ligang Wu. Improving robustness of intent detection under adversarial attacks: A geometric constraint perspective. IEEE transactions on neural networks and learning systems , PP, 2023.

Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR , 2015.

Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attacks. In ICLR . OpenReview.net, 2017.

Nicholas Carlini and David A. Wagner. Towards evaluating the robustness of neural networks. In IEEE S&P , 2017.

Nathan Inkawhich, Wei Wen, Hai (Helen) Li, and Yiran Chen. Feature space perturbations yield more transferable adversarial examples. In CVPR , 2019.

Yao Zhu, Jiacheng Sun, and Zhenguo Li. Rethinking adversarial transferability from a data distribution perspective. In ICLR , 2022.

Buyu Li, Yu Liu, and Xiaogang Wang. Gradient harmonized single-stage detector. In AAAI , 2019.

Nicolas Papernot, Patrick D. McDaniel, Ian J. Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In ACM , 2017.

Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. In ICML , 2018.

Jiawei Su, Danilo Vasconcellos Vargas, and Kouichi Sakurai. One pixel attack for fooling deep neural networks. IEEE Trans. Evol. Comput. , 23(5):828-841, 2019.

Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In CVPR , 2018.

Jiadong Lin, Chuanbiao Song, Kun He, Liwei Wang, and John E. Hopcroft. Nesterov accelerated gradient and scale invariance for adversarial attacks. In ICLR , 2020.

Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Evading defenses to transferable adversarial examples by translation-invariant attacks. In CVPR , 2019.

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR , 2018.

Junhua Zou, Zhisong Pan, Junyang Qiu, Xin Liu, Ting Rui, and Wei Li. Improving the transferability of adversarial examples with resized-diverse-inputs, diversity-ensemble and region fitting. In ECCV , 2020.

Junyoung Byun, Seungju Cho, Myung-Joon Kwon, Heeseon Kim, and Changick Kim. Improving the transferability of targeted adversarial examples through object-based diverse input. In CVPR , 2022.

Nathan Inkawhich, Kevin J. Liang, Lawrence Carin, and Yiran Chen. Transferable perturbations of deep feature distributions. In ICLR , 2020.

Nathan Inkawhich, Kevin J. Liang, Binghui Wang, Matthew Inkawhich, Lawrence Carin, and Yiran Chen. Perturbing across the feature hierarchy to improve standard and strict blackbox attack transferability. In NeurIPS , 2020.

Maosen Li, Cheng Deng, Tengjiao Li, Junchi Yan, Xinbo Gao, and Heng Huang. Towards transferable targeted attack. In CVPR , 2020.

Zhengyu Zhao, Zhuoran Liu, and Martha A. Larson. On success and simplicity: A second look at transferable targeted attacks. In NeurIPS , 2021.

Chaoning Zhang, Philipp Benz, Adil Karjauv, Jae-Won Cho, Kang Zhang, and In So Kweon. Investigating top-k white-box and transferable black-box attack. In CVPR , 2022.

Zeyu Qin, Yanbo Fan, Yi Liu, Li Shen, Yong Zhang, Jue Wang, and Baoyuan Wu. Boosting the transferability of adversarial attacks with reverse adversarial perturbation. In NeurIPS , 2022.

Muzammal Naseer, Salman H. Khan, Munawar Hayat, Fahad Shahbaz Khan, and Fatih Porikli. On generating transferable targeted perturbations. In ICCV , 2021.

Omid Poursaeed, Isay Katsman, Bicheng Gao, and Serge J. Belongie. Generative adversarial perturbations. In CVPR , 2018.

Konda Reddy Mopuri, Utkarsh Ojha, Utsav Garg, and R. Venkatesh Babu. NAG: network for adversary generation. In CVPR , 2018.

Muzammal Naseer, Salman H. Khan, Muhammad Haris Khan, Fahad Shahbaz Khan, and Fatih Porikli. Cross-domain transferability of adversarial perturbations. In NeurIPS , 2019.

Geoffrey E. Hinton and Sam T. Roweis. Stochastic neighbor embedding. In Suzanna Becker, Sebastian Thrun, and Klaus Obermayer, editors, NeurIPS , 2002.

Trevor Hastie, Robert Tibshirani, and Jerome H. Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd Edition . 2009.

Abhinav Shrivastava, Abhinav Gupta, and Ross B. Girshick. Training region-based object detectors with online hard example mining. In CVPR , 2016.

Xiaolong Wang, Abhinav Shrivastava, and Abhinav Gupta. A-fast-rcnn: Hard positive generation via adversary for object detection. In CVPR , 2017.

Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV , 2017.

Aman Sinha, Hongseok Namkoong, and John C. Duchi. Certifying some distributional robustness with principled adversarial training. In ICLR , 2018.

Maher Nouiehed, Maziar Sanjabi, Tianjian Huang, Jason D. Lee, and Meisam Razaviyayn. Solving a class of non-convex min-max games using iterative first order methods. In NeurIPS , 2019.

Yisen Wang, Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. On the convergence and robustness of adversarial training. In ICML , 2019.

Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximal-gradient methods under the polyak-łojasiewicz condition. In ECML/PKDD , 2016.

Simon S. Du, Jason D. Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In ICML , 2019.

Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via overparameterization. In ICML , 2019.

Bolei Zhou, Aditya Khosla, Àgata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In CVPR , 2016.

Ross B. Girshick. Fast R-CNN. In ICCV , 2015.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR , pages 248-255, 2009.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , 2016.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR , 2015.

Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In CVPR , 2017.

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR , 2016.

Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A. Alemi. Inception-v4, inceptionresnet and the impact of residual connections on learning. In AAAI , 2017.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2021.

Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian J. Goodfellow, Dan Boneh, and Patrick D. McDaniel. Ensemble adversarial training: Attacks and defenses. In ICLR , 2018.

Alexey Kurakin, Ian J. Goodfellow, Samy Bengio, Yinpeng Dong, Fangzhou Liao, Ming Liang, Tianyu Pang, Jun Zhu, Xiaolin Hu, Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, Alan L. Yuille, Sangxia Huang, Yao Zhao, Yuzhe Zhao, Zhonglin Han, Junjiajia Long, Yerkebulan Berdibekov, Takuya Akiba, Seiya Tokui, and Motoki Abe. Adversarial attacks and defences competition. CoRR , abs/1804.00097, 2018.

Lianli Gao, Qilong Zhang, Xiaosu Zhu, Jingkuan Song, and Heng Tao Shen. Staircase sign method for boosting adversarial attacks. CoRR , abs/2104.09722, 2021.

Zhipeng Wei, Jingjing Chen, Zuxuan Wu, and Yu-Gang Jiang. Enhancing the self-universality for transferable targeted attacks. In CVPR , 2023.

Yuyang Long, Qilong Zhang, Boheng Zeng, Lianli Gao, Xianglong Liu, Jian Zhang, and Jingkuan Song. Frequency domain model augmentation for adversarial attack. In ECCV , 2022.

Xiao Yang, Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Boosting transferability of targeted adversarial examples via hierarchical generative networks. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, ECCV , 2022.

Dylan J. Foster and Alexander Rakhlin. glyph[lscript] ∞ vector contraction for rademacher complexity. CoRR , abs/1911.06468, 2019.

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning . MIT Press, 2012.

Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning - From Theory to Algorithms . Cambridge University Press, 2014.

Yunwen Lei, Ürün Dogan, Ding-Xuan Zhou, and Marius Kloft. Data-dependent generalization bounds for multi-class classification. IEEE Trans. Inf. Theory , 65(5):2995-3021, 2019.

Yong Liu. Refined learning bounds for kernel and approximate k -means. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , 2021.

Wassily Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association , 58(301):13-30, 1963.

Supplementary Material :

Perturbation Towards Easy Samples Improves Targeted Adversarial Transferability

A Proofs

A.1 Proof of Theorem 1

First, we need the definition below:

Definition 2 ( ( j, k, x 0 , r ) -Local Output Rademacher Complexity) Let f S w = ( f S 1 w , . . . , f S K w ) ∈ [0 , 1] K , F S k w := { f S k w : w ∈ W } , given B ( x 0 , r ) = { x : ‖ x -x 0 ‖ ≤ r } , ( x 0 , y 0 ) ∈ S j , then the empirical ( j, k, x 0 , r ) -local output Rademacher Complexity is defined by:

where σ i , i ∈ I ( j,x 0 ,r ) are i.i.d random variables satisfies P ( σ i = 1) = P ( σ i = -1) = 1 2 . Correspondingly, the expected ( j, k, x 0 , r ) -local output Rademacher complexity is defined as R ( j,x 0 ,r ) ( F S k w ) = E S ( j,x 0 ,r ) [ R ( j,x 0 ,r ) n ( F S k w ) | x i ∈ C ( j,x 0 ,r ) ] , where S ( j,x 0 ,r ) = { ( x, y ) ∈ S j : x ∈ B ( x 0 , r ) , ( x 0 , y 0 ) ∈ S } .

We first prove the lemma 1 and use it to prove theorem 1, and we need lemma 2 and 3 before proving the lemma 1.

Lemma 1 Given B ( x 0 , r ) , ( x 0 , y 0 ) ∈ S j , S ( j,x 0 ,r ) , and f S k w ∈ F S k w , then with probability at least 1 -δ , the following holds for any f S k w ∈ F S k w and k ∈ [ K ] :

where K is the number of classes, C w and b w are constants.

Lemma 2 ( l ∞ Contraction Inequality [54] ) Let F ⊆ { f : X → R K } , and let φ : R K → R be L -Lipschitz with respect to the l ∞ norm, i.e. ‖ φ ( v ) -φ ( v ′ ) ‖ ∞ ≤ L · ‖ v -v ′ ‖ ∞ , ∀ v, v ′ ∈ R K . For any a > 0 , there exists a constant C > 0 such that if | φ ( f ( x )) | ∨ ‖ f ( x ) ‖ ∞ ≤ ζ , then

where R n ( φ ◦ F ) = E σ [ sup f ∈F | ∑ n i =1 σ i φ ( f ( x i )) | ] , ˜ R n ( F i ) = sup S ∈X n R n ( F i ) .

Lemma 3 Given B ( x 0 , r ) , ( x 0 , y 0 ) ∈ S j , S ( j,x 0 ,r ) and F S k w , with probability at least 1 -δ , holds:

Proof of Lemma 3 Let S ′ ( j,x 0 ,r ) be a dataset that has at most one element different with S ( j,x 0 ,r ) , its elements are x ′ i , i ∈ I ( j,x 0 ,r ) , then

the last inequality is because S ′ ( j,x 0 ,r ) differs from S ( j,x 0 ,r ) with at most one element, and f S k w ∈ [0 , 1] . Then use McDiarmid's inequality ([55], Theorem D.3), we have

holds with probability at least 1 -δ .

In the following we use the notation: ˆ E S ( j,x 0 ,r ) [ f S k w ] = 1 | I ( j,x 0 ,r ) | ∑ i ∈I ( j,x 0 ,r ) f S k w ( x i ) , then define:

Lemma 4 Given B ( x 0 , r ) , ( x 0 , y 0 ) ∈ S j and f S k w ∈ F S k w , with probability at least 1 -δ , the following holds for any f S k w ∈ F S k w :

Proof of Lemma 4 Similarly, let S ′ ( j,x 0 ,r ) be a dataset that has at most one element different with S ( j,x 0 ,r ) , then we have

then use McDiarmid's inequality, with probability at least 1 -δ , the following holds:

then we could use the standard symmetrization technique ([56], Theorem 3.1) to derive the following:

combine with (1), we can get:

holds with probability at least 1 -δ .

Proof of Lemma 1 Let φ k : R K → R satisfies φ k ( v ) = v k , where v = ( v 1 , . . . , v k ) ∈ R K . Then

since | φ k ( v ) -φ k ( v ′ ) | = | v k -v ′ k | ≤ ‖ v -v ′ ‖ ∞ , ∀ v, v ′ ∈ R K , i.e. φ k is 1 -Lipschitz. For f S w ∈ F S w , we have ∥ ∥ f S w ∥ ∥ ∞ ≤ 1 , therefore, ∣ ∣ φ k ( f S w ( x ) )∣ ∣ ∨ ∥ ∥ f S w ( x ) ∥ ∥ ∞ ≤ 1 , then use Lemma 2, choose L = 1 , ζ = 1 , a = 1 2 , there exists a constant C w k > 0 such that

where ˜ R ( j,x 0 ,r ) n ( F S k w ) = sup S ( j,x 0 ,r ) ∈Z | I ( j,x 0 ,r ) | ( j,x 0 ,r ) R ( j,x 0 ,r ) n ( F S k w ) , and we denote { ( x, y ) ∈ Z : x ∈ C ( j,x 0 ,r ) } as Z ( j,x 0 ,r ) .

the penultimate inequality uses the Khintchine-Kahane Inequality [57]. Therefore,

which means ˜ R ( j,x 0 ,r ) n ( F S k w ) n .

Choose (˜ x, ˜ y ) ∈ Z ( j,x 0 ,r ) , f S k w ∈ F S k w satisfies

then, similar to Lemma 3 in [58], we have

similarly, the penultimate inequality also uses the Khintchine-Kahane Inequality [57]. Below we denote ∣ ∣ ∣ ˜ f S k w (˜ x ) ∣ ∣ ∣ as b w ,k . Choose b w = max 1 ≤ k ≤ K b w ,k , which satisfies max k ˜ R ( j,x 0 ,r ) n ≥ √ 2 | I ( j,x 0 ,r ) | b w 2 , and C w = max 1 ≤ k ≤ K C w k , combine with (2), we have

holds for all k ∈ [ K ] . Then use Lemma 3 and Lemma 4, with probability at least 1 -δ , holds

then combine with (3), we have

holds with probability at least 1 -δ .

Proof of Theroem 1 For the inequality in Lemma 1, by taking δ ′ = δ K for each k ∈ [ K ] , we can guarantee with probability at least 1 -δ , the inequality

simultaneously holds for each k ∈ [ K ] . Then for two different parametrized class F S k w 1 := { f S k w 1 : w 1 ∈ W 1 } and F S k w := { f S k w : w ∈ W } , just take C = max { C w 1 , C w } and b = min { b w 1 , b w } , the following simultaneously holds for any i ∈ { 1 , 2 } and k ∈ [ K ] :

thus,

By the definition, we have ∣ ∣ I ( j,x 0 ,r ) ∣ ∣ = ρ ( j,x 0 ,r ) · vol B ( x 0 , r ) , and note that

therefore we can derive that with probability at least 1 -δ , the following holds:

which completes the proof.

A.2 Proof of Theorem 2

First, note that

the second inequality holds because of Assumption 1. Then due to the Lipschitz continuous gradient of R ( j,x 0 ,r ) ( w ) , we have

the inequality holds because of Assumption 2, prescribe notation ξ t = ∣ ∣ I B t ∩ I ( j,x 0 ,r ) ∣ ∣ , combined with Assumption 3, 4, we have

choose η t = 1 βµt , due to T ≤ L 1 2 βµ , η t G 2 ≤ L 1 η 2 t G 2 2 .

Note that ξ t ∼ Hypergeometric ( n, ∣ ∣ I ( j,x 0 ,r ) ∣ ∣ , M ) , by Hoeffding's inequality [59], we have

holds simultaneously for all t ∈ { 1 , 2 , . . . , T } with probability at least 1 -δ , which means

Therefore,

holds simultaneously for all t ∈ { 1 , 2 , . . . , T } with probability at least 1 -δ .

A.3 Proof of Proposition 1

Under Assumption 1, due to the Lipschitz continuous gradient, for ∀ ( x i , y i ) ∈ S , we have

holds for ∀ x ∈ X . By (4), choose x = x i -r ∇ x glyph[lscript] ( w ,x ) ‖∇ x glyph[lscript] ( w ,x ) ‖ , which satisfies ‖ x i -x ‖ ≤ r , then we have

take δ T = e -2 Mε 2 , we have

which means

Use (5), we have

then use (6), we have

Thus

Combining (7) and (8), we get

B Details of Pre-training Embeddings

In actual experiments, it was found that if the embedding is not pre-trained first, the model convergence speed will be quite slow. We tried to freeze the parameters of the embedding layer after pre-training multiple epochs, but found that even with the same number of epochs each pre-training, the internal structures between the embedding vectors are very different, and there may be clustering between different class embedding (as shown in Figure6), the distance between multiple classes is very close, which leads to redundant information from many other classes in embedding.

To address this issue and enable the generator's embeddings to better reflect class information, we employ the following loss, utilizing the feature outputs of the source model as guidance for embedding pretraining:

However, directly optimizing the L M leads to the issue of loss collapse. We analyse each component of the loss. First, note that for ∀ i, j ∈ [ K ] , -1 ≤ M E cos i,j ≤ 1 , -1 ≤ M S cos i,j ≤ 1 . Then glyph[negationslash]

glyph[negationslash]

Under the settings in this paper, K = 10 , so both terms analyse above are positive with lower bounds. In the optimization process, their ratio will not approach zero. As for M S euc i,j , it is a deterministic positive constant. Therefore, when it serves as the numerator, it will not lead to a ratio close to zero inside the logarithm. However, for M E euc i,j , if it becomes close to zero when used as the numerator, it will cause loss collapse. Note that glyph[negationslash]

Euclidean

Cosine

Cosine

Euclidean

1.00095

0.94

0.91-0.96 0.960.96  0.91 0.96

1.0

15.0

1.00

0.99

-0.97

-1.00

0.99-0.98

SL0

8.1

46

12.5

-0.94

-1.00

0.99

1.98

0.99

0.99

0.99097

0.960.99

0.50

025

EE

2.7

1.84

10.0

160

40.99

0.96

0.98

1.00

0.00

2.7

81

190

0.64

1.2

1.1

77

1.6

060

7.5

5.0

-0.96

-0.96

0.90

-0.97

-1.00

-1.00

0.97

0.99

0.99

-1.00

0.99

0.95

0.98

1.000.99

0.99

1.00

1.00

0.99

1.00

0.94

0.98

0:98

0.99

0.99

L00

0.25

0.50

-0.5

1.2

1.84

1.97

25

16Y

66°0

1.00

0.95

0.75

1.0

1.0

-0.5

0.0

0.5

Euclidean

Cosine

Cosine

Euclidean

1.00

160

0.91-0.830.950.930.700.87

17.5

60

0.99

0.840.960.990.870.84

0.75

15.0

-1.00

LO0

t60

L00

0.78

260

0.91

0.50

12.5

1.94

0.94-0.91-0.99

0.98

0.720.93

0.25

.6

17

10.0

-0.84

L00

1.78

0.94

0.91

L00

0.79

1.00

0.93

0.95

160

60

0.91

0.99

0.00

0.0

7

2.9

1.5

193

66°0

0.95

1.00

0.99

0.70

0.96

0.25

0.5

5.3

5.0

F6Y

-40.99

197

86°0

0.91

0.99

60

0.91

0.50

25

0.87

0.72

0.93

9

0.99

196

60

SL'0-

1.0

1.0

0.5

0.5

1.0

To prevent M E euc i,j from becoming too small, we need to ensure:

which means we need to ensure that

glyph[negationslash]

since M E euc i,k , M E euc i,j are non-negative. Also, note that M E euc i,k = ∣ ∣ e i -e k ∣ ∣ ≤ ∣ ∣ e i ∣ ∣ + ∣ ∣ e k ∣ ∣ . Therefore, by imposing the following constraint in the optimization:

We can ensure that M E euc i,k ≤ 2 U , which in turn guarantees max 1 ≤ k ≤ K exp M E euc i,k = + ∞ , ∀ 1 ≤ i ≤ K , glyph[negationslash]

thus avoiding the issue of M E euc i,j becoming too small and causing loss collapse.

By explicitly incorporating the above constraint as a regularizer into the optimization objective, we obtain the final manifold matching loss:

In our experiments, we use the AdamW optimizer with a learning rate of 1 . 5 e -5 for 15000 epochs to optimize the embeddings, while setting λ 1 to 5 and λ 2 to 0 . 01 . The results, as shown in Figure 7, demonstrate a significant improvement in the clustering of embeddings between different classes.

The trained embedding is denoted as e j ∈ R d 1 , j ∈ [ K ] , where e j ∈ R d 1 , j ∈ [ K ] . For the utilized Resblock, let the input and output channels be C in and C out respectively. The fully connected layer

Euclidean

Cosine

Cosine

Euclidean

11

12

11

0

12

12

12

0

7.2

12

IH

13

1

9

11

16

15

19

19

3

13

13

11

13

17

17

17

16

14

- 17.5

20.0

ra

1.00

0.89

0.72

0.59

0.76

1.00

0.72

0.76

1.00

0.77

0.79

0.60

0.56

0.67

0.86

0.61

0.67

083

0.53

0.60

0.82

0.65

0.74

0.79

0.46

0.58

0.74

0.55

0.65

0.74

0.9

- 0.8

1.0

0.5

1.0 -

5

12

9

7.2

12

4

0

15

17

22

17

20

20

- 15.0

620

077

0.60

L00

0.41

0.47

0.31

0.40

021

025

0.7

6

6

19

13

17

13

16

17

14

I5

19

13

17

9

11

13

11

13

H

15

17

22

17

20

20

88

7.4

0

10

11

11

9.6

10

11

0

10

10

8.8

11

11

8.9

0

11

9.6

7.8

7.4

11

7.8

0

7.8

8.8

8.9

6.9

10

0

69

8.8

7.8

11

10

0

- 7.5

- 12.5

- 10.0

- 5.0

- 25

+

r

0.56

0.61

0.53

0.65

0.46

0.55

0.67

0.67

0.60

0.74

0.58

065

0.74

0.86

0.83

0.82

60

0.74

0.21

0.41

0.47

0.31

0.40

0.25

0.85

0.91

0.90

0.89

L00

0.84

085

0.87

0.89

0.87

0.87

1.00

160

0.89

1.00

0.89

0.92

0.92

0.90

0.87

0.89

160

0.92

0.89

0.87

0.92

0.91

1.00

0.94

0.84

0.87

092

0.92

0.94

100

- 05

- 0.4

= 0.3

- 0.6

F 5'0-

1.0

0.0

8

6

0

6

8

6

5

b

6

00 -

0

E

6

1.0

0.5

0.0

0.5

1.0

9

3

3

6

9

12

MLP( · ) ∈ R d 1 → R C out . Firstly, we employ the initial convolution kernel Conv 1 to map the raw data from the input h of the current layer, transitioning it from [ C in , H, W] to [ C out , H, W]. Then we perform an addition operation between this mapped result and the embedding transformation obtained through the MLP. Formally expressed as:

where y is the label of the input. Subsequently, we apply a second convolution to this output, followed by a LayerNorm operation, and finally, pass it through an attention operation before adding it to the residual. Formally stated as:

C Additional Ablation Studies

To investigate the impact of different settings during the training phase, we conduct several experiments. First, we investigate the output consistency of the model for different values of q used for sample screening, as shown in Figure 8. A smaller q implies a stricter selection, which leads to an increase in output consistency (in terms of mean), but also results in a larger standard deviation. In this case, q = 5 is a relatively suitable choice.

Then we validate the impact of different choices of d ( · , · ) on targeted transferability. The results are shown in Figure 9, where d ( · , · ) was chosen as the Smooth L1 loss, which significantly improves the targeted transferability compared to L1 and L2 losses.

D Implementation Details

In the exploratory experiments conducted in Section 2, the three different network architectures we employed are depicted in Table 4.

For the experiments on CIFAR10 mentioned in Section 2, we used the PyTorch framework. The optimizer is SGD, with an initial learning rate of 0 . 1 , weight decay of 5 e -4 , and momentum of 0 . 9 . We apply cosine annealing learning rate decay, with a maximum number of epochs ( T max ) set to 200 . Tolerance of Early-stopping is set to 30 . Additionally, we used a batch size of 128 during training and applied the following data augmentations:

Randomly crops the input image to a size of 32 × 32 pixels with padding of 4 pixels.

Mean:0.19,stdi0j10

q=2

12

q=10

10

q=50

Mean:0.2istd:0.10

q=100

Density

8

Me

h：0.24$td:0.10

q=all

Mean:0.24 std:0.10

6

Meai

: 0.pn.i std:010

1101

4

Mea

: 0:26, std/0.1

101

0:27,s10

0

0.0

0.2

0.4

0.6

0.8

Cosine Similarity

1.0

0.9

L2

。。。

L1

0.8

Rate

0.7

0.6

SuccessR

0.5

0.4

0.3

0.2

0.1

0.0

Res152

Inc-v3

Inc-v4

IncRes-v2

Target Model

Randomly flips the input image horizontally with a probability of 0.5.

After data augmentation, the input is then transformed into a tensor and normalized with a mean of (0 . 4914 , 0 . 4822 , 0 . 4465) and a standard deviation of (0 . 2023 , 0 . 1994 , 0 . 2010) .

For the experiments in Section 4, we used the structure of following models: ResNet-50, VGG19bn, DenseNet-121, ResNet-152 from the torchvision 1 library, and others from the timm 2 library. The torchattacks 3 library is utilized for MIM, SI-NIM, TIM, and DIM.

E Additional Experimental Results

E.1 Ensemble Model as Source

In Table 5, we report the results of the case of an ensemble model as the source model. The selected ensemble model are consist of three models: Res-50, Inc-v3, and Inc-v4, while the other models

not involved in the ensemble are treated as target models. ESMA still demonstrates outstanding adversarial transferability.

Attack

Src:Ensemble

→

VGG19bn

→

Dense121

→

Res152

→

IncRes-v2

→

ViT

AVG

MIM

1.76%

3.40%

3.29%

1.07%

0.22%

1.95%

9.96%

16.13%

14.76%

8.27%

1.11%

10.05%

DIM

10.27%

16.31%

15.22%

8.20%

0.87%

10.17%

SI-NIM

1.16%

1.67%

1.49%

0.93%

0.13%

1.08%

Po-TI-Trip

14.22%

21.76%

18.69%

12.09%

1.62%

13.68%

SI-FGS

2

M

15.72%

24.84%

20.79%

12.20%

1.53%

15.02%

S

2

I-SI-TI-DIM

17.82%

21.07%

21.36%

16.38%

2.38%

15.80%

Logit

16.09%

27.56%

17.51%

36.96%

2.20%

20.06%

DTMI-Logit-SU

18.11%

30.16%

18.73%

38.64%

2.64%

21.71%

RAP-LS

48.98%

72.91%

70.67%

26.53%

4.22%

44.66%

HGN

55.10%

67.37%

63.54%

49.85%

26.78%

52.53%

TTP(129257.2)

60.02%

71.33%

67.44%

55.42%

30.04%

56.85%

ESMA(112303.5)

62.51

%

74.11

%

72.61

%

62.33

%

32.70

%

60.85

%

E.2 Models Trained on Different Distributions as Victim

Given that our discussion is based on the condition that the source and target models are trained on the same distribution data, to explore the transferability between models trained on training data with different distribution, we conducted additional transfer attacks on two adversarially trained models, Inc-v3-adv and IncRes-v2-ens. The results are reported in Table 6. ESMA's performance is slightly weakened but still competitive. Considering that this validation is not directly related to the viewpoints, conclusions, and statements of our paper, we will further discuss the transferability on target models trained on data with distribution shift in our future work.

F Fooling Google Lens

We show that adversarial examples perturbated using ESMA can deceive Google Lens in image search, as shown in Figure 10 and 11.

PP7963

PP7963

PP7963

StartYourCombatGame

withrealisticprojectiles!

Peer reviews:
comments: SUMMARY:
The paper theoretically and experimentally demonstrated that adding perturbations towards High-Sample-Density-Regions (HSDR) of the target class can further enhance the transferability of targeted attacks. To avoided the impracticality of density estimation in high-dimensional data, authors propose a method for identifying samples within HSDR. The experimental results demonstrate the effectivenss of  the proposed method.

STRENGTHS:
- The experiments demonstrate the effectiveness of the proposed ESMA, which can outperform baselines by a large margin.
- The idea of adding perturbations towards HSDR is intersting and effective.
- To avoided the impracticality of density estimation in high-dimensional data, this paper propose a method for identifying samples within HSDR, which is practical.

WEAKNESSES:
- The recent transfer-based attack [1,2,3] is missing. I suggest authors discuss or compare propsed method with these paper. 
- The experiments are not extensive, i.e., only two tables. More ablation studies should be presented. For example, 
   - Will the results be really poor if a difficult sample is selected to form target anchor sets? Or why author don't report the results of ESMA in Table 1.
   - Does ESMA still outperform the existing algorithm if the source model is an ensemble of models.



[1] Zhang et al. Staircase sign method for boosting adversarial attacks, 2021

[2] Zhao et al. On Success and Simplicity: A Second Look at Transferable Targeted Attacks, NeurIPS 2021

[3] Long et al. Frequency Domain Model Augmentation for Adversarial Attack, ECCV2022

QUESTIONS:
Please refer to weakness.

LIMITATIONS:
This paper only considers CNNs and it is doubtful whether it is valid on ViTs.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a hypothesis that pushing adversarial samples towards high-density regions can enhance the transferability of target attacks. The author provides theoretical analysis and experimental validation to support the rationale behind this hypothesis. Based on this hypothesis, the author designs a new generative attack method that improves the adversarial transferability of target attacks.\n\nSTRENGTHS:\n* The idea of pushing ushing adversarial samples towards high-density regions in target attack is well-motivated\n* The theoretical analysis presented in this paper is insightful and solid.\n\nWEAKNESSES:\n* The lack of persuasive experimental evidence is a major concern.\n    1. Insufficient ablation study: The paper introduces multiple modules in the proposed method, but the author fails to include an ablation study. Without such an analysis, it becomes challenging to ascertain whether the proposed method genuinely enhances adversarial transferability. For example, it would be valuable to investigate if the transferability improvement persists when pretrained embeddings are not utilized.\n    2. Limited impact of easy example-based adversarial attacks: The experiments presented in Table 1 demonstrate that the adversarial attacks based on easy examples have limited impact on transferability. Additionally, the experimental description in this section lacks sufficient detail. It would be helpful to clarify whether the baseline methods in Table 1 are based on cross-entropy. If they are, I recommend the author redefine the baselines, such as using target anchors generated from randomly selected samples of the target class.\n    3. Lack of validation experiments:  Experiments are required to assess whether the adversarial examples generated by the proposed method exhibit higher density/smaller loss and loss gradients compared to those generated by baseline methods. It is essential for the author to include such validation experiments to provide empirical evidence supporting the claims regarding the improved density and reduced loss characteristics of the generated adversarial examples.\n\n* The analysis and proposed solution in the paper exhibit a gap. The conclusion drawn in the paper suggests that samples with lower loss and smaller gradients tend to have stronger transferability. A straightforward approach to address this would be to minimize not only the loss but also the gradient of the loss during the attack. It raises the question of why the author did not employ this direct and simple solution. Instead, the author chose to modify the loss to make the output of adversarial examples align with the target anchor. This approach introduces a gap between the previous analysis and the proposed solution.\n\n* Section 3 lacks clarity in its description. The role of pretrained embedding is not well-defined. I recommend the author to provide a detailed formula or explanation of how pretrained embedding is utilized in the generator.\n\nQUESTIONS:\n*  I suggest the authors conduct ablation studies to verify their claim.\n*  Does making the output of adversarial examples match that of the target samples improve the density of adversarial examples?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents both theoretical and experimental evidence showing that incorporating perturbations towards the High-Sample-Density-Regions (HSDR) of the target class can significantly enhance the transferability of targeted attacks. To address the challenges of density estimation in high-dimensional data, a novel method for identifying samples within HSDR is proposed. Leveraging these insights, the authors introduce ESMA, which outperforms existing generative targeted attacks in terms of effectiveness and efficiency.\n\nSTRENGTHS:\nThe paper provides theoretical and experimental evidence to support the observation that deep learning models exhibit more consistent outputs in the High-Sample-Density-Regions (HSDR) of each class (Section 2.1). This finding establishes a foundation for the proposed approach. The paper demonstrates through experimental and theoretical analysis that easy samples with low loss in early-stopping models are commonly found in HSDR. Leveraging this insight, the proposed method directly adds perturbations targeting these samples of the target class, resulting in improved targeted adversarial transferability. This approach circumvents the need for computationally expensive density estimation of high-dimensional samples. Finally the paper introduces the Easy Sample Matching Attack (ESMA)  which surpasses state-of-the-art generative attacks such as TTP in terms of targeted transfer success rate. Notably, ESMA only requires one model to perform attacks for all target classes significantly reducing storage space requirements and training time compared to TTP.\n\nWEAKNESSES:\nThere are some grammar mistakes (in lines 231-238, the sentences in the whole paragraph are all connected with commas, making it difficult to read) and the notion is a little bit confusing (e.g. in Algorithm 2, $a_k$ is in the feature space, but in the equation in line 251, $a_j$ is in the input space).\n\nQUESTIONS:\n1 According to Algorithm 3,  the target class is randomly sampled from a uniform distribution in the training stage. How does the generator generate a target adversarial example given a specific target class in the reference stage? \n\n2 Is the model employed to select target anchors different from the source model in ESMA? This information is not clear.\n\nLIMITATIONS:\nThe writing and the usage of notation need to be enhanced.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper finds that optimizing perturbations towards easy examples located in HSDR can improve targeted adversarial transferability. The authors provide toy experiments and theoretical proof to illustrate these conclusions. They further utilize easy examples to guide the targeted attack in the GAN-like framework. This framework makes it feasible to perform the multi-target attack.\n\nSTRENGTHS:\n1. Exploring targeted adversarial transferability in terms of High-Sample-Density-Regions is interesting.\n2. The proposed framework support multi-target adversarial perturbation generation.\n3. The motivation of this paper is clear.\n\nWEAKNESSES:\nThe experiments are insufficient. They miss previous works, hierarchical generative networks for multi-target attack [A], and iterative attacks [B,C,D, E]. Besides, the authors only conduct experiments on naturally trained models, and neglect to consider adversarially trained models and other defense methods. And the authors should attack more diverse network architectures, such as ViTs.\n\n\n[A] Yang X, Dong Y, Pang T, et al. Boosting transferability of targeted adversarial examples via hierarchical generative networks[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 725-742.    \n[B] Zhao Z, Liu Z, Larson M. On success and simplicity: A second look at transferable targeted attacks[J]. Advances in Neural Information Processing Systems, 2021, 34: 6115-6128.   \n[C] Qin Z, Fan Y, Liu Y, et al. Boosting the transferability of adversarial attacks with reverse adversarial perturbation[J]. Advances in Neural Information Processing Systems, 2022, 35: 29845-29858.    \n[D] Wei Z, Chen J, Wu Z, et al. Enhancing the Self-Universality for Transferable Targeted Attacks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 12281-12290.    \n[E] Byun J, Cho S, Kwon M J, et al. Improving the transferability of targeted adversarial examples through object-based diverse input[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 15244-15253.\n\nQUESTIONS:\nPlease conduct more experiments as discussed above.\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}