Title: Rethinking the Backward Propagation for Adversarial Transferability
Venue: NeurIPS 2023
Accepted: yes
Keywords: Adversarial examples

Convolutional neural networks

Adversarial transferability

Backward propagation
Abstract: Transfer-based attacks generate adversarial examples on the surrogate model, which can mislead other black-box models without access, making it promising to attack real-world applications. Recently, several works have been proposed to boost adversarial transferability, in which the surrogate model is usually overlooked. In this work, we identify that non-linear layers (e.g., ReLU, max-pooling, etc.) truncate the gradient during backward propagation, making the gradient w.r.t. input image imprecise to the loss function. We hypothesize and empirically validate that such truncation undermines the transferability of adversarial examples. Based on these findings, we propose a novel method called Backward Propagation Attack (BPA) to increase the relevance between the gradient w.r.t. input image and loss function so as to generate adversarial examples with higher transferability. Specifically, BPA adopts a non-monotonic function as the derivative of ReLU and incorporates softmax with temperature to smooth the derivative of max-pooling, thereby mitigating the information loss during the backward propagation of gradients. Empirical results on the ImageNet dataset demonstrate that not only does our method substantially boost the adversarial transferability, but it is also general to existing transfer-based attacks. Code is available at https://github.com/Trustworthy-AI-Group/RPA.

Rethinking the Backward Propagation for Adversarial Transferability

Abstract

Transfer-based attacks generate adversarial examples on the surrogate model, which can mislead other black-box models without access, making it promising to attack real-world applications. Recently, several works have been proposed to boost adversarial transferability, in which the surrogate model is usually overlooked. In this work, we identify that non-linear layers ( e.g . ReLU, max-pooling, etc .) truncate the gradient during backward propagation, making the gradient w.r.t . input image imprecise to the loss function. We hypothesize and empirically validate that such truncation undermines the transferability of adversarial examples. Based on these findings, we propose a novel method called Backward Propagation Attack (BPA) to increase the relevance between the gradient w.r.t . input image and loss function so as to generate adversarial examples with higher transferability. Specifically, BPA adopts a non-monotonic function as the derivative of ReLU and incorporates softmax with temperature to smooth the derivative of max-pooling, thereby mitigating the information loss during the backward propagation of gradients. Empirical results on the ImageNet dataset demonstrate that not only does our method substantially boost the adversarial transferability, but it is also general to existing transfer-based attacks. Code is available at https://github.com/Trustworthy-AI-Group/RPA .

1 Introduction

Deep Neural Networks (DNNs) have been widely applied in various domains, such as image recognition [37, 13, 16], object detection [34], face verification [52, 41], etc . However, their susceptibility to adversarial examples [39, 10], which are carefully crafted by adding imperceptible perturbations to natural examples, has raised significant concerns regarding their security. In recent years, the generation of adversarial examples, aka adversarial attacks, has garnered increasing attention [31, 21, 6, 55, 44, 24] in the research community. Notably, there has been a significant advancement in the efficiency and applicability of adversarial attacks [19, 1, 47, 6, 57, 59, 51], making them increasingly viable in real-world scenarios.

By exploiting the transferability of adversarial examples across different models [29], transfer-based attacks generate adversarial examples on the surrogate model to fool the target models [6, 57, 12, 43, 58]. Unlike other types of attacks [2, 19, 1, 47], transfer-based attacks do not require direct access to the victim models, making them particularly applicable for attacking online interfaces. Consequently, transfer-based attacks have emerged as a prominent branch of adversarial attacks. However, it is worth noting that the early white-box attacks [10, 31, 21] often exhibit poor transferability despite demonstrating superior performance within the white-box setting.

To this end, different techniques have been proposed to enhance adversarial transferability, such as momentum-based attacks [6, 27, 43, 46, 9, 59], input transformations [57, 7, 45, 30, 8, 48, 42], advanced objective functions [17, 54, 49], and model-related attacks [23, 53, 12, 50]. Among these techniques, model-related attacks are particularly valuable due to their ability to exploit the characteristics of surrogate models. Model-related attacks offer a unique perspective on adversarial attacks by leveraging the knowledge gained from surrogate models, which can also shed new light on the design of more robust models. In despite of their potential significance, model-related attacks have been somewhat overlooked compared to other types of transfer-based attacks.

Since transfer-based attacks mainly design various gradient ascend methods to generate adversarial examples on the surrogate model, in this work, we first revisit the backward propagation procedure. We find that non-linear layers ( e.g ., activation function, max-pooling, etc .) often truncate the gradient of loss w.r.t . the feature map, which diminishes the relevance of the gradient between the loss and input image. We assume and empirically validate that such gradient truncation undermines the adversarial transferability. Based on this finding, we propose Backward Propagation Attack (BPA), which modifies the calculation for the derivative of ReLU activation function and max-pooling layers during the backward propagation process. With these modifications, BPA mitigates the negative impact of gradient truncation and improves the transferability of adversarial attacks.

Our main contribution can be summarized as follows:

To our knowledge, this is the first work that proposes and empirically validates the detrimental effect of gradient truncation on adversarial transferability. This finding sheds new light on improving adversarial transferability and might provide new directions to boost the model robustness.

We propose a model-related attack called BPA, that adopts a non-monotonic function as the derivative of the ReLU activation function and incorporates softmax with temperature to calculate the derivative of max-pooling. With these modifications, BPA mitigates the negative impact of gradient truncation and enhances the relevance of gradient between the loss function and the input.

Extensive experiments on ImageNet dataset demonstrate that BPA could significantly boost various untargeted and targeted transfer-based attacks and outperform the baselines with a substantial margin, emphasizing the effectiveness and superiority of our proposed approach.

2 Related Work

In this section, we provide a brief overview of the existing adversarial attacks and defenses.

2.1 Adversarial Attacks

Existing adversarial attacks can be categorized into two groups based on the access to target model, namely white-box attacks and black-box attacks. In the white-box setting [10, 33, 31, 2], attackers have complete access to the structure and parameters of the target model. In the black-box setting, the attacker has limited or no information about the target model, making it applicable in the physical world. Black-box attacks can be further grouped into three classes, i.e ., score-based attacks [4, 19], query-based attacks [3, 22, 47], and transfer-based attacks [6, 57, 43]. Among the three types of black-box attacks, transfer-based attacks generate adversarial examples on the surrogate model without accessing the target model, drawing increasing interest recently.

Since MI-FGSM [6] integrates momentum into I-FGSM [21] to stabilize the update direction and achieve improved transferability, various momentum-based attacks have been proposed to generate transferable adversarial examples. For instance, NI-FGSM [27] leverages Nesterov Accelerated Gradient for better transferability. VMI-FGSM [43] refines the current gradient using the gradient variance from the previous iteration, resulting in more stable updates. EMI-FGSM [46] enhances the momentum by averaging the gradient of several data points sampled in the previous gradient direction.

On the other hand, input transformations that modify the input image prior to gradient calculation have proven highly effective in enhancing adversarial transferability, such as DIM [56], TIM [7], SIM [27], Admix [45], SSA [30] and so on. Among these attacks, Admix introduces a small segment of an image from different categories, while SSA applies frequency domain transformations to the input image, both of which have demonstrated superior performance in generating transferable adversarial examples.

Several studies have explored the utilization of more sophisticated objective functions to enhance transferability in adversarial attacks. ILA [17] employs fine-tuning techniques to increase the similarity of feature differences between the original or current adversarial example and a benign sample. ATA [54] maximizes the disparity of attention maps between a benign sample and an adversarial example. FIA [49] minimizes a weighted feature map in an intermediate layer to disrupt significant object-aware features.

Afew works have emphasized the significance of the surrogate model in generating highly transferable adversarial examples. Ghost network [23] attacks a set of ghost networks generated by densely applying dropout at the intermediate features. On the other hand, another line of work focuses on the gradient during backward propagation. SGM [54] adjusts the decay factor to incorporate more gradients from the skip connections of ResNet to generate more transferable adversarial examples. LinBP [12] performs backward propagation in a more linear fashion by setting the gradient of ReLU as a constant of 1 and scaling the gradient of residual blocks. In this work, we find that the gradient truncation introduced by non-linear layers undermines the transferability and modify the backward propagation so as to generate more transferable adversarial examples.

2.2 Adversarial Defenses

The existence of adversarial examples poses a significant security threat to deep neural networks (DNNs). To mitigate this impact, researchers have proposed various methods, among which adversarial training has emerged as a widely used and effective approach [10, 21, 31]. By augmenting the training data with adversarial examples, this method enhances the robustness of trained models against adversarial attacks. For instance, Tramèr et al. [40] introduce ensemble adversarial training, a technique that generates adversarial examples using multiple models simultaneously, which shows superior performance against transfer-based attacks.

Although adversarial training is effective, it comes with high training costs, particularly for large-scale datasets and complex networks. Consequently, researchers have proposed innovative defense methods as alternatives. Guo et al. [11] utilize various input transformations such as JPEG compression and total variance minimization to eliminate adversarial perturbations from input images. Xie et al. [56] mitigate adversarial effects through random resizing and padding of input images. Liao et al. [25] propose training a high-level representation denoiser (HGD) specifically designed to purify input images. Nasser [32] introduce a neural representation purifier (NRP) by a self-supervised adversarial training mechanism to purify the input sample. Various certified defenses aim to provide a verified guarantee in a specific radius, such as randomized smoothing (RS) [5].

3 Methodology

In this section, we analyze the backward propagation procedure and identify that the gradient truncation introduced by non-linear layers undermines the adversarial transferability. Based on this finding, we propose Backward Propagation Attack (BPA) to mitigate such negative effect and gain more transferable adversarial examples.

3.1 Backward Propagation for Adversarial Transferability

̸

Given an input image x with ground-truth label y , a classifier f with l successive layers ( e.g ., z i +1 = ϕ i ( f i ( z i )) , z 0 = x ) predicts the label f ( x ) = f l ( z l ) = y with high probability. Here ϕ ( · ) is a non-linear activation function ( e.g ., ReLU) or identity function if there is no activation function after i -th layer f i . The attacker aims to find an adversarial example x adv adhering the constraint of ∥ x adv -x ∥ p ≤ ϵ , but resulting in f ( x adv ) = f ( x ) = y for untargeted attack and f ( x adv ) = y t for targeted attack. Here ϵ is the maximum perturbation magnitude, y t is the target label, and ∥ · ∥ p denotes the p -norm distance. For brevity, the following description will focus on non-targeted attacks with p = ∞ . Let J ( x, y ; θ ) denote the loss function of classifier f ( e.g ., the cross-entropy loss). Existing white-box attacks often solve the following constrained maximization problem using the gradient ∇ x J ( x, y ; θ ) :

Based on the chain rule, we can calculate the gradient as follows:

where 0 < k < l is the index of an arbitrary layer. Without loss of generality, we explore the backward propagation when passing the k -th layer as follows:

A fully connected or convolutional layer followed by a non-linear activation function . Taking ReLU activation ( i.e ., ϕ k ) for example, the j -th element in the gradient w.r.t . the k -th feature, [ ∂z k +1 ∂z k ] j , will be one if z k,j > 0 and otherwise, [ ∂z k +1 ∂z k ] j will be zero. These zero gradients in ∂z k +1 ∂z k can lead to the truncation of gradient of the loss function ∂J ( x,y ; θ ) ∂z k w.r.t . the input image. As a result, the gradient is effectively limited or weakened to some extent.

Max-pooling layer . As shown in Fig. 1, max-pooling calculates the maximum value (orange block) within a specific patch. Hence, the derivative ∂z k +1 ∂z k will be a binary matrix, containing only ones at locations corresponding to the orange blocks. In this case, approximately 3 / 4 of the elements in the given sample will be zeros. This means that max-pooling tends to discard a significant portion of the gradient information contained in ∂z k ∂x , resulting in a truncated gradient.

The truncation of gradient caused by non-linear layers ( e.g ., activation function, max-pooling) can limit or dampen the flow of gradients during backward propagation, which decays the relevance among the gradient between the loss and input. Considering that many existing attacks rely on maximizing the loss by leveraging the gradient information, we make the following assumption:

Assumption 1 The truncation of gradient ∇ x J ( x, y ; θ ) introduced by non-linear layers in the backward propagation process decays the adversarial transferability.

To validate Assumption 1, we conduct several experiments using FGSM, I-FGSM and MI-FGSM. The detailed experimental settings are summarized in Sec. 4.1.

0.1 -0.2 1.9

1.4

0.0 -0.5 2.3

0.7

-0.4 0.9

1.0 -2.0

0.7

0.6

0.5

1.7

Randomly mask the gradient . To investigate the impact of gradient truncation on adversarial transferability, we introduce a random masking operation to increase the probability of gradient truncation between stage 3 and stage 2 of ResNet-50. Fig. 2a illustrates the attack performance with various mask probabilities. As the mask probability increases, more zeros appear in the derivative, indicating a higher degree of gradient truncation. Consequently, the larger truncation probability renders the gradient less relevant to the loss function, decreasing the attack performance of the three evaluated methods. These findings validate our hypothesis that the truncation of gradient negatively impacts adversarial transferability and highlight the importance of preserving gradient information to maintain the effectiveness of adversarial attacks across various models.

Recover the gradient of ReLU or max-pooling layers . In contrast, it is expected that mitigating the truncation of gradient can improve the adversarial transferability. To explore this, we randomly replaced the zeros in the derivative of ReLU or max-pooling operations with ones, using various replacement probabilities. a) In Fig. 2b, as the probability of replacement increases, fewer gradients are truncated across ReLU, resulting in improved adversarial transferability on all the three attacks. Notably, these attacks achieve their best performance when the derivative consists entirely of ones, which aligns with LinBP [12]. b) As illustrated in Fig. 2c, when the ratio of ones in the derivative of max-pooling increases ( i.e ., the replacement probability increases), the attack performance initially improves, reaching a peak around 0 . 3 . Subsequently, the attack performance gradually decreases but remains superior to vanilla backward propagation. This hightlights that we need a more suitable approximation for the derivative calculation of max-pooling, which is detailed in Sec. 3.2. These results suggest that decreasing the probability of gradient truncation in max-pooling is beneficial for enhancing adversarial transferability.

Overall, these findings validate Assumption 1 that the truncation of gradients negatively impacts adversarial transferability. By preserving gradient information and carefully adjusting the replacement probabilities, it is possible to improve the effectiveness of adversarial attacks across different models.

FGSM

I-FGSM

MI-FGSM

0

0

.

2

0

.

4

0

.

6

0

.

8

1

0

10

20

Prob.

Attack success rates (%)

(a) Randomly mask the gradient

0

0

.

2

0

.

4

0

.

6

0

.

8

1

15

20

25

30

Prob.

Attack success rates (%)

(b) Recover the gradient of ReLU

0

0

.

2

0

.

4

0

.

6

0

.

8

1

15

20

25

30

Prob.

Attack success rates (%)

(c) Recover the gradient of max-pooling

3.2 Mitigating the Negative Impact of Gradient Truncation

In Sec. 3.1, we demonstrate that reducing the probability of gradient truncation in non-linear layers can enhance adversarial transferability. However, setting all elements in the corresponding derivative to one is not optimal for generating transferable adversarial examples. Here we investigate how to modify the backward propagation process of non-linear layers to further enhance the transferability.

Within the standard backward propagation procedure, the elements comprising the derivative depend on the magnitudes of the associated feature map. This observation provides an impetus for considering the intrinsic characteristics of the underlying features when diminishing the probability of gradient truncation. To this end, we modify the

-

5

0

5

0

0

.

5

1

Ours

LinBP

vallina

gradient calculation for the ReLU activation function and max-pooling in the backward propagation procedure as follows:

Gradient calculation for ReLU . To ensure precise gradient calculation, it is important to exclude extreme values from consideration when calculating the gradient, while still maintaining the relationship between the elements in the derivative and the magnitude of the feature map. Among the family of ReLU activation functions, SiLU [14] provides a smooth and continuous gradient across the entire input range and is less susceptible to gradient saturation issues. Hence, we propose using the derivative of SiLU to calculate the gradient of ReLU during the backward propagation process, i.e ., ∂z i +1 ∂z i = σ ( z i ) · (1 + z i · (1 -σ ( z i ))) , where σ ( · ) is the Sigmoid function. This formulation allows our gradient calculation to reflect the input magnitude mainly within the input range around [ -5 , 5] , while closely resembling the behavior of ReLU when the input is outside this range. As shown in Fig. 3, our proposed gradient calculation method demonstrates improved alignment with the input's magnitude compared to both the original derivative of ReLU and the derivative used in LinBP. By leveraging the smoothness and non-monotonicity of SiLU, we can obtain more accurate and reliable gradient information for ReLU.

Gradient calculation for max-pooling . Similar to the gradient calculation for ReLU, it is essential to exclude extreme values and ensure that the gradient remains connected to the magnitude of the feature map. Furthermore, in the case of max-pooling, the summation of gradients within each window should remain at one to minimize modifications to the gradient. To address these considerations, we propose using the softmax function to calculate the gradient within each window w of the max-pooling operation:

where t is the temperature coefficient to adjust the smoothness of the gradient. If the feature z k,i,j is related to multiple windows ( i.e ., the stride is smaller than the size of max-pooling), we sum its gradient calculated by Eq. 3 in each window as the final gradient.

In practice, we adopt the above two strategies to calculate the gradient of ReLU and max-pooling during the backward propagation process. This approach allows us to circumvent the issue of gradient truncation introduced by these non-linear layers. We refer to this modified backward propagation technique as Backward Propagation Attack (BPA), which can be applied to existing CNNs to adapt to various transfer-based attack methods.

4 Experiments

In this section, we conduct extensive experiments on standard ImageNet dataset [35] to validate the effectiveness of the proposed BPA. We first specify our experimental setup, then we conduct a series of experiments to compare BPA with existing state-of-the-art attacks under different settings. Additionally, we provide ablation studies to further investigate the performance and behavior of BPA.

4.1 Experimental Setup

Dataset. Following LinBP [12], we randomly sample 5,000 images pertaining to the 1,000 categories from ILSVRC 2012 validation set [35], which could be classified correctly by all the victim models.

Models. We select ResNet-50 [13] and VGG-19 [37] as our surrogate model for generating adversarial examples. As for the victim models, we consider six standardly trained networks, i.e ., Inception-v3 (Inc-v3) [13], Inception-Resnet-v2 (IncRes-v2) [38], DenseNet [16], MobileNet-v2 [36], PNASNet [28], and SENet [15]. Additionally, we adopt three ensemble adversarially trained models, namely ens3-adv-Inception-v3 (Inc-v3 ens 3 ), ens4-Inception-v3 (Inc-v3 ens 4 ), and ens-adv-InceptionResNet-v2 (IncRes-v2 ens ) [40]. To address the issue of different input shapes required by these models, we adhere to the official pre-processing pipeline, including resizing and cropping techniques.

Baselines. We adopt three model-related methods as our baselines, i.e ., SGM [53], LinBP [12] and Ghost [23], and evaluate their performance to boost adversarial transferability of iterative attacks (PGD [31]), momentum-based attacks (MI-FGSM [6], VMI-FGSM [43]), advanced objective functions (ILA [17]) and input transformation-based attacks (SSA [30]).

Hyper-parameters. We adopt the maximum magnitude of perturbation ϵ = 8 / 255 to align with existing works. We run the attacks in T = 10 iterations with step size α = 1 . 6 / 255 for untargeted attacks and T = 300 iterations with step size α = 1 / 255 for targeted attacks. We set the momentum decay factor µ = 1 . 0 and sample 20 examples for VMI-FGSM. The number of spectrum transformations and tuning factor is set to N = 20 and ρ = 0 . 5 , respectively. The decay factor for SGM is

γ = 0 . 5 and the random range of Ghost network is λ = 0 . 22 . We follow the setting of LinBP to modify the backward propagation of ReLU in the last eight residual blocks of ResNet-50. We set the temperature coefficient t = 10 for ResNet-50 and t = 1 for VGG-19.

4.2 Evaluation on Untargeted Attacks

To validate the effectiveness of our proposed method, we compare BPA with several other modelrelated methods ( i.e ., SGM, LinBP, Ghost) on ResNet-50 to boost various adversarial attacks, namely PGD, MI-FGSM, VMI-FGSM, ILA and SSA. Here we adopt ResNet-50 as the surrogate model since SGM is specific to ResNets. However, it is worth noting that BPA is general to various surrogate models with non-linear layers and we also report the results on VGG-19 in Appendix A.2. To further validate the effectiveness of BPA, we also consider more input transformation based attacks, different perturbation budgets and conduct evaluations on CIFAR-10 dataset [20] in Appendix A.3-A.5. We measure the attack success rates by evaluating the misclassification rates of the nine different target models on the generated adversarial examples.

Evaluations on the single baseline . We can observe from Table 1 that the model-related strategies can consistently boost performance of the five typical attacks on nine models. Among the baseline methods, LinBP generally achieves the best performance, except for VMI-FGSM where SGM surpasses LinBP. By addressing the issue of gradient truncation, BPA consistently improves the performance of all the five attack methods and achieves the best overall performance. On average, BPA outperforms the runner-up attack by a significant margin of 7 . 84% , 11 . 19% , 5 . 08% , 9 . 17% , 2 . 25% , respectively. These results highlight the effectiveness and generality of BPA in generating transferable adversarial examples compared with existing model-related strategies. The performance improvement achieved by BPA on SGM and LinBP, which also modify the backward propagation, validates our hypothesis that reducing the gradient truncation introduced by non-linear layers is beneficial for enhancing the adversarial transferability. This emphasizes the importance of carefully considering the backward propagation procedure when generating transferable adversarial examples.

Evaluations by combining BPA with the baselines . The primary objective of BPA is to mitigate the negative impact of gradient truncation on adversarial transferability, which is not considered by the baselines. Hence, it is expected that BPA can also boost the performance of these baselines. For validation, we integrate BPA with the baseline methods to enhance the performance of PGD and MI-FGSM attacks. The results of these combinations are presented in Table 2. We can observe that BPA can effectively

boost the adversarial transferability of various baselines. On average, BPA can boost the best baseline ( i.e ., LinBP) with a remarkable margin of 13 . 23% and 20 . 94% for PGD and MI-FGSM, highlighting

the high effectiveness and superiority of BPA. Such high performance also validates its excellent generality to various architectures and supports our hypothesis about gradient truncation.

Evaluations on defense methods . To further evaluate the effectiveness of BPA, we also assess its performance on six defense methods using PGD and MI-FGSM, namely HGD [25], R&P [56], NIPS-r3 3 , JPEG [11], RS [5] and NRP [32]. The results are presented in Table 3. We can observe that our BPA method successfully enhances both the PGD and MI-FGSM attacks, leading to higher attack performance against the defense methods. The results suggest that BPA can effectively enhance adversarial attacks against a range of defense techniques, reinforcing its potential as a powerful tool for generating transferable adversarial examples.

In summary, BPA exhibits superior transferability compared to various baseline methods when evaluated using a range of transfer-based attacks. It also exhibits good generality to further boost existing model-related approaches and achieves remarkable performance on several defense models, highlighting its effectiveness and versatility in generating highly transferable adversarial examples.

4.3 Evaluation on Targeted Attacks

To further evaluate the effectiveness of BPA, we also investigate its performance in boosting targeted attacks. Zhao et al . [60] identified that logit loss can yield better results than most resource-intensive attacks regarding targeted attacks. Here we adopt PGD and MI-FGSM to optimize the logit loss on ResNet-50 w/wo various model-related methods. The results are summarized in Table 4. Without the model-related methods, both PGD and MI-FGSM exhibit poor attack performance. However, when these methods are applied, the attack performance improves significantly. Notably, our BPA method achieves the best attack performance among all the baselines. This highlights the high effectiveness and excellent versatility of our proposed method in boosting targeted attacks and exhibits its potential to improve adversarial attacks in a wide range of scenarios. We also provide the results on VGG-19 in Appendix A.6.

4.4 Ablation Study

To gain further insights into the effectiveness of BPA, we perform parameter studies on two crucial aspects: the position of the first ReLU layer to be modified and the temperature coefficient t for max-pooling. Additionally, we conduct ablation studies to investigate the impact of diminishing the gradient truncation of ReLU and max-pooling separately. We also provide more discussions about BPA in Appendix A.7-A.9.

On the position of the first ReLU layer to be modified . ReLU activation functions are densely applied in existing neural networks. For instance, there are totally 17 ReLU activation functions in ResNet-50. Intuitively, the truncation in the latter layers has a greater impact on gradient relevance compared to the earlier layers. As BPA aims to recover the truncated gradients by injecting imprecise gradients into the backward propagation, it is essential to focus on the more critical layers. To identify these important layers and evaluate their impact on transferability, we conduct the BPA attack using MI-FGSM by modifying the ReLU layers starting from the i -th layer, where 1 ≤ i ≤ 17 . As shown in Fig. 4a, modifying the last ReLU layer alone significantly improves the transferability of the attack, showing its high effectiveness. As we modify more ReLU layers, the transferability further

Inc-v3

IncRes-v2

DenseNet

MobileNet

PNASNet

SENet

Inc-v3

ens

3

Inc-v3

ens

4

IncRes-v2

ens

0

1-0

1-1

1-2

2-0

2-1

2-2

2-3

3-0

3-1

3-2

3-3

3-4

3-5

4-1

4-2

4-3

20

40

60

80

The position of the first ReLU layer to be modified

Attack success rates (%)

0

3

6

9

12

15

20

40

60

80

Temperature coefficient

t

Attack success rates (%)

(a) Attack success rate (%) of BPA using MI-FGSM by modifying the ReLU layers starting from the i -th layer. Here 3 -0 indicates the first ReLU layer in the third stage

(b) Attack success rate (%) of BPA using MI-FGSM with various temperature coefficients ( 0 ≤ t ≤ 15 ) in Eq. (3) for the max-pooling layer

improves and remains consistently high for most models. However, for a few models ( e.g ., PNASNet), modifying more ReLU layers leads to a slight decay on performance. To maintain a high level of performance across all nine models, we modify the ReLU layers starting from 3 -0 ReLU layer, which is also adopted in LinBP [12].

On the temperature coefficient t for max-pooling . The temperature coefficient t plays a crucial role in determining the distribution of relative gradient magnitudes within each window. For example, when t = 0 , the gradient distribution becomes a normalized uniform distribution. To find an appropriate temperature coefficient, we conduct the BPA attack using MI-FGSM with various temperatures. As shown in Fig. 4b, when t = 0 , the attack exhibits the poorest performance but still outperforms the vanilla MI-FGSM. As we increase the value of t , the attack's performance consistently improves and reaches a high level of performance after t = 10 . By selecting a suitable temperature coefficient, we ensure that the gradient distribution within each window is well-balanced and contributes effectively to the adversarial perturbation. Thus, we adopt t = 10 in our experiments.

Ablation studies on ReLU and max-pooling . As stated in Sec. 3.1, we hypothesize that the gradient truncation caused by non-linear layers, such as ReLU and max-pooling in ResNet-50, has a detrimental effect on adversarial transferability. To further validate this hypothesis, we conduct ablation studies by comparing the performance of PGD and MI-FGSM attacks using the vallina backward propagation, the backward propagation modified by either ReLU or max-pooling, and both modifications combined. As shown in Table 5, adopting the modified backward propagation with either ReLU or max-pooling results in a significant improvement in adversarial transferability for both PGD and MI-FGSM attacks. Considering the presence of only one max-pooling layer in ResNet-50, the average performance improvement of 4 . 07% and 7 . 58% for PGD and MI-FGSM highlights the high effectiveness of BPA and underscores the efficacy of BPA in addressing the issue of gradient truncation. Furthermore, when both ReLU and max-pooling layers are modified in backward propagation, PGD and MI-FGSM exhibit the best performance. This finding supports the rational design of BPA and highlights the importance of mitigating gradient truncation in both ReLU and max-pooling layers to achieve optimal adversarial transferability.

5 Conclusion

In this work, we analyzed the backward propagation procedure and identified that non-linear layers ( e.g ., ReLU and max-pooling) introduce gradient truncation, which undermined the adversarial transferability. Based on this finding, we proposed a novel attack called Backward Propagation Attack (BPA) to mitigate the gradient truncation for more transferable adversarial examples. In particular, BPA addressed gradient truncation by introducing a non-monotonic function as the derivative of the ReLU activation function and incorporating softmax with temperature to calculate the derivative of max-pooling. These modifications helped to preserve the gradient information and prevented significant truncation during the backward propagation process. Empirical evaluations on ImageNet dataset demonstrated that BPA can significantly enhance existing untargeted and targeted attacks and outperformed the baselines by a remarkable margin. Our findings identified the vulnerability of model architectures and raised a new challenge in designing secure deep neural network architectures.

6 Limitation

Our proposed BPA modifies backpropagation process for gradient calculation, making it only suitable for gradient-based attacks. Besides, BPA modifies the derivatives of non-linear layers, such as ReLU and max-pooling. Consequently, it may not be directly applicable to models lacking these specific components, such as transformers. In the future, we will investigate how to generalize our BPA to such transformers by refining the derivatives of some components, e.g ., softmax. This endeavor to enhance the generality and versatility of BPA will be an essential aspect of ongoing research, paving the way for the broader applicability of the proposed method and facilitating its adoption in various deep learning models beyond those with ReLU and max-pooling layers.

References

Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models. In Proceedings of the International Conference on Learning Representations , 2018.

Nicholas Carlini and David Wagner. Towards Evaluating the Robustness of Neural Networks. In 2017 IEEE symposium on security and privacy (sp) , pages 39-57, 2017.

Jianbo Chen, Michael I Jordan, and Martin J Wainwright. Hopskipjumpattack: A Query-efficient Decisionbased Attack. In 2020 ieee symposium on security and privacy (sp) , pages 1277-1294, 2020.

Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models. In Proceedings of the 10th ACM workshop on artificial intelligence and security , pages 15-26, 2017.

Jeremy M. Cohen, Elan Rosenfeld, and J Zico Kolter. Certified Adversarial Robustness via Randomized Smoothing. International Conference on Machine Learning , pages 1310-1320, 2019.

Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting Adversarial Attacks with Momentum. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9185-9193, 2018.

Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Evading Defenses to Transferable Adversarial Examples by Translation-invariant Attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4312-4321, 2019.

Zhijin Ge, Fanhua Shang, Hongying Liu, Yuanyuan Liu, Liang Wan, Wei Feng, and Xiaosen Wang. Improving the Transferability of Adversarial Examples with Arbitrary Style Transfer. In Proceedings of the ACM International Conference on Multimedia , 2023.

Zhijin Ge, Fanhua Shang, Hongying Liu, Yuanyuan Liu, and Xiaosen Wang. Boosting Adversarial Transferability by Achieving Flat Local Maxima. In Proceedings of the Advances in Neural Information Processing Systems , 2023.

Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and Harnessing Adversarial Examples. In Proceedings of the International Conference on Learning Representations , 2015.

Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van der Maaten. Countering Adversarial Images using Input Transformations. In Proceedings of the International Conference on Learning Representations , 2018.

Yiwen Guo, Qizhang Li, and Hao Chen. Backpropagating Linearly Improves Transferability of Adversarial Examples. In Proceedings of the Advances in Neural Information Processing Systems , pages 85-95, 2020.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 770-778, 2016.

Dan Hendrycks and Kevin Gimpel. Gaussian Error Linear Units (GELUs). arXiv preprint arXiv:1606.08415 , 2016.

Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation Networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7132-7141, 2018.

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely Connected Convolutional Networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4700-4708, 2017.

Qian Huang, Isay Katsman, Horace He, Zeqi Gu, Serge Belongie, and Ser-Nam Lim. Enhancing Adversarial Example Transferability with an Intermediate Level Attack. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4733-4742, 2019.

Tianjin Huang, Vlado Menkovski, Yulong Pei, YuHao Wang, and Mykola Pechenizkiy. Directionaggregated attack for transferable adversarial examples. ACM Journal on Emerging Technologies in Computing Systems (JETC) , 18(3):1-22, 2022.

Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box Adversarial Attacks with Limited Queries and Information. In Proceedings of the International Conference on Machine Learning , pages 2137-2146, 2018.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning Multiple Layers of Features from Tiny Images. 2009.

Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial Machine Learning at Scale. In Proceedings of the International Conference on Learning Representations , 2017.

Huichen Li, Xiaojun Xu, Xiaolu Zhang, Shuang Yang, and Bo Li. QEBA: Query-Efficient Boundarybased Blackbox Attack. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1221-1230, 2020.

Yingwei Li, Song Bai, Yuyin Zhou, Cihang Xie, Zhishuai Zhang, and Alan Yuille. Learning Transferable Adversarial Examples via Ghost Networks. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 11458-11465, 2020.

Siyuan Liang, Xingxing Wei, Siyuan Yao, and Xiaochun Cao. Efficient adversarial attacks for visual object tracking. In European Conference on Computer Vision , pages 34-50, 2020.

Fangzhou Liao, Ming Liang, Yinpeng Dong, Tianyu Pang, Xiaolin Hu, and Jun Zhu. Defense against Adversarial Attacks using High-level Representation Guided Denoiser. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1778-1787, 2018.

Jiadong Lin, Chuanbiao Song, Kun He, Liwei Wang, and John E Hopcroft. Nesterov accelerated gradient and scale invariance for adversarial attacks. arXiv preprint arXiv:1908.06281 , 2019.

Jiadong Lin, Chuanbiao Song, Kun He, Liwei Wang, and John E. Hopcroft. Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attacks. In Proceedings of the International Conference on Learning Representations , 2020.

Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive Neural Architecture Search. In Proceedings of the European Conference on Computer Vision , pages 19-34, 2018.

Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into Transferable Adversarial Examples and Black-box Attacks. In Proceedings of the International Conference on Learning Representations , 2017.

Yuyang Long, Qilong Zhang, Boheng Zeng, Lianli Gao, Xianglong Liu, Jian Zhang, and Jingkuan Song. Frequency Domain Model Augmentation for Adversarial Attack. In Proceedings of the European Conference on Computer Vision , pages 549-566, 2022.

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Resistant to Adversarial Attacks. In Proceedings of the International Conference on Learning Representations , 2018.

Muzammal Naseer, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Fatih Porikli. A Selfsupervised Approach for Adversarial Robustness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 262-271, 2020.

Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. The Limitations of Deep Learning in Adversarial Settings. In 2016 IEEE European symposium on security and privacy (EuroS&P) , pages 372-387, 2016.

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards Real-Time Object Oetection with Region Proposal Networks. In Proceedings of the Advances in Neural Information Processing Systems , 2015.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet Large Scale Visual Recognition Challenge. International Journal of Computer Vision , 115:211-252, 2015.

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted Residuals and Linear Bottlenecks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4510-4520, 2018.

Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. In Proceedings of the International Conference on Learning Representations , 2015.

Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander Alemi. Inception-v4, Inception-resnet and the Impact of Residual Connections on Learning. In Proceedings of the AAAI Conference on Artificial Intelligence , 2017.

Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing Properties of Neural Networks. In Proceedings of the International Conference on Learning Representations , 2014.

Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. Ensemble Adversarial Training: Attacks and Defenses. In Proceedings of the International Conference on Learning Representations , 2018.

Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large Margin Cosine Loss for Deep Face Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2018.

Kunyu Wang, Xuanran He, Wenxuan Wang, and Xiaosen Wang. Boosting Adversarial Transferability by Block Shuffle and Rotation. arXiv preprint arXiv:2308.10299 , 2023.

Xiaosen Wang and Kun He. Enhancing the Transferability of Adversarial Attacks through Variance Tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1924-1933, 2021.

Xiaosen Wang, Kun He, Chuanbiao Song, Liwei Wang, and John E. Hopcroft. AT-GAN: An Adversarial Generator Model for Non-Constrained Adversarial Examples. arXiv preprint arXiv:1904.07793 , 2019.

Xiaosen Wang, Xuanran He, Jingdong Wang, and Kun He. Admix: Enhancing the Transferability of Adversarial Attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16158-16167, 2021.

Xiaosen Wang, Jiadong Lin, Han Hu, Jingdong Wang, and Kun He. Boosting Adversarial Transferability through Enhanced Momentum. In Proceedings of the British Machine Vision Conference , page 272, 2021.

Xiaosen Wang, Zeliang Zhang, Kangheng Tong, Dihong Gong, Kun He, Zhifeng Li, and Wei Liu. Triangle Attack: A Query-efficient Decision-based Adversarial Attack. In Proceedings of the European Conference on Computer Vision , pages 156-174, 2022.

Xiaosen Wang, Zeliang Zhang, and Jianping Zhang. Structure Invariant Transformation for better Adversarial Transferability. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023.

Zhibo Wang, Hengchang Guo, Zhifei Zhang, Wenxin Liu, Zhang Qin, and Kui Ren. Feature Importanceaware Transferable Adversarial Attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7639-7648, 2021.

Zhiyuan Wang, Zeliang Zhang, Siyuan Liang, and Xiaosen Wang. Diversifying the High-level Features for better Adversarial Transferability. In Proceedings of the British Machine Vision Conference , 2023.

Xingxing Wei, Siyuan Liang, Ning Chen, and Xiaochun Cao. Transferable adversarial attacks for image and video object detection. arXiv preprint arXiv:1811.12641 , 2018.

Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A Discriminative Feature Learning Approach for Deep Face Recognition. In Proceedings of the European Conference on Computer Vision , 2016.

Dongxian Wu, Yisen Wang, Shu-Tao Xia, James Bailey, and Xingjun Ma. Skip Connections Matter: On the Transferability of Adversarial Examples Generated with ResNets. In Proceedings of the International Conference on Learning Representations , 2020.

Weibin Wu, Yuxin Su, Xixian Chen, Shenglin Zhao, Irwin King, Michael R Lyu, and Yu-Wing Tai. Boosting the Transferability of Adversarial Samples via Attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1161-1170, 2020.

Chaowei Xiao, Bo Li, Jun-Yan Zhu, Warren He, Mingyan Liu, and Dawn Song. Generating Adversarial Examples with Adversarial Networks. arXiv preprint arXiv:1801.02610 , 2018.

Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan Yuille. Mitigating Adversarial Effects Through Randomization. In Proceedings of the International Conference on Learning Representations , 2018.

Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan L Yuille. Improving Transferability of Adversarial Examples With Input Diversity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2730-2739, 2019.

Yifeng Xiong, Jiadong Lin, Min Zhang, John E. Hopcroft, and Kun He. Stochastic Variance Reduced Ensemble Adversarial Attack for Boosting the Adversarial Transferability. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022 , pages 14963-14972, 2022.

Jianping Zhang, Jen-tse Huang, Wenxuan Wang, Yichen Li, Weibin Wu, Xiaosen Wang, Yuxin Su, and Michael R Lyu. Improving the Transferability of Adversarial Samples by Path-Augmented Method. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8173-8182, 2023.

Zhengyu Zhao, Zhuoran Liu, and Martha Larson. On Success and Simplicity: A Second Look at Transferable Targeted Attacks. Advances in Neural Information Processing Systems , 34:6115-6128, 2021.

A Appendix

In the appendix, we provide extra results about our proposed BPA. We first provide the raw data for Fig. 2. Considering that our main evaluations are conducted on ResNet-50 since the baseline SGM primarily focuses on the residual connection, we also conduct the evaluations on VGG-19 to further validate the effectiveness and generality of our proposed BPA. Then we provide the evaluations using more input transformation based attacks, and different perturbation budgets and also evaluate various methods on CIFAR-10 datasets. To further gain insights into the superiority of BPA, we also offer further discussions and explorations about the motivation and design.

A.1 Raw Data for Fig. 2

Due to the limited space, we only report the average attack success rates in Fig. 2. To validate its generality to various deep models, we also report the raw data in Tab. A1-A3. We can see that the attacker exhibits the same trends on various deep models, which is consistent with the results in Fig. 2.

A.2 Additional Evaluation on Untargeted Attacks

To validate the generality of BPA to various architectures, we further validate the effectiveness of our proposed BPA on VGG-19. Specifically, we first conduct untargeted attacks on VGG-19 following the setting in Sec. 4.2. Here we take LinBP and Ghost as our baselines.

Evaluations on the single baseline . As shown in Table A4, model-related methods consistently achieve better attack performance than the attacks on the original models, showing the effectiveness of these methods. Compared with LinBP and Ghost, our proposed BPA exhibits superior performance across all five attacks. On average, BPA outperforms the runner-up method with a remarkable margin of 14 . 21% , 16 . 44% , 14 . 15% , 11 . 80% , 13 . 64% for PGD, MI-FGSM, VMI-FGSM, ILA, and SSA, respectively. These results are consistent with the findings reported in Sec. 4.2 for ResNet-50. The superior performance of BPA not only validates its effectiveness but also highlights its generality to different architectures.

Evaluations by combining BPA with the baselines . Similar in Sec. 4.2, we also integrate BPA into LinBP and Ghost to further boost the performance. The results in Table A5 indicate that BPA can significantly improve the attack performance of PGD and MI-FGSM. For instance, considering MIFGSMattack, integrating BPA results in a clear performance improvement of 11 . 42% and 16 . 46% for LinBP and Ghost, respectively. These findings are consistent with the results obtained on ResNet-50, as discussed in Sec. 4.2. These results further highlight the effectiveness and superiority of BPA in boosting the adversarial transferability of existing attacks, which are not limited to the surrogate models.

Evaluations on defense methods . Finally, we evaluate these model-related approaches on defense methods and report the results in Table A6. Notably, our BPA method consistently enhances the performance of PGD and MI-FGSM attacks, yielding superior results against the defense methods compared to other model-related methods. On average, BPA outperforms the runner-up method with a

margin of 5 . 32% and 7 . 78% for PGD and MI-FGSM, respectively. These findings further underscore the high effectiveness of BPA in improving the performance of various attacks and highlight its versatility in enhancing adversarial attacks across different architectural models.

In conclusion, the results obtained for untargeted attacks on VGG-19 align with the findings presented for ResNet-50 in Sec. 4.2. The significant and consistent improvement in performance across various architectures validates our motivation that addressing the gradient truncation issue caused by non-linear layers can enhance adversarial transferability. These findings also strongly support the high effectiveness and utility of our BPA to boost adversarial transferability.

A.3 Additional Evaluation on Various Input Transformation based Attacks

In Table 1, we compare our BPA with various model-related attacks when combined with other attacks, including gradient-based (PGD), momentum-based (MI-FGSM, VMI-FGSM), objectiverelated (ILA) and input transformation based (SSA) attack. Due to the page limit, we only evaluate BPA with one up-to-date transformation based attack (SSA). Here we further evaluate its generality to other input transformation based attacks, namely DIM [57], TIM [7], SIN [26] and DA [18] in Table A7. As we can see, our BPA can significantly boost these input transformation based attacks. Compared with existing model-related attacks, BPA consistently exhibits better transferability for TIM, DIM, and DA. For SIN, BPA exhibits comparable attack performance with SGM on standardly trained models but much better transferability on adversarially trained models. To integrate the advantage of SGM and our BPA, we combine BPA with SGM for SIN, which outperforms the baselines with a clear margin. These results further validate its superiority in boosting adversarial transferability.

A.4 Additional Evaluation on Another Widely Adopted Perturbation Budget

Both 8 / 255 and 16 / 255 are widely adopted perturbation budgets in adversarial learning. In Sec. 4.2, we mainly adopt ϵ = 8 / 255 for evaluation. To further validate the effectiveness of our BPA, we also

evaluate BPA using PGD with ϵ = 16 / 255 . As shown in Table A8, BPA consistently outperforms the baselines with a clear margin, showing its remarkable effectiveness in boosting adversarial transferability with various perturbation budgets.

A.5 Additional Evaluation on CIFAR-10 Dataset

Existing transfer-based attacks mainly validate their effectiveness on ImageNet dataset. For a fair comparison, we also evaluate our BPA on ImageNet dataset. Here we also conduct experiments on CIFAR-10 using PGD with ϵ = 8 255 on VGG-19. As shown in Table A9, BPA exhibits better transferability than the baselines, showing its high effectiveness and generality to various datasets and models.

A.6 Additional Evaluation on Targeted Attacks

Targeted attacks are more challenging than untargeted attacks. To further validate the effectiveness and generality of BPA, we also perform the targeted attack on VGG-19, following the experimental settings in Sec. 4.3. The results are summarized in Table A10. It is interesting that LinBP decays the targeted attack performance on VGG-19. Since there is no skip connection in VGG-19, LinBP only modifies the derivative of ReLU, which might introduce an imprecise gradient. This highlights the significance that BPA excludes extreme values from consideration when calculating the gradient for better transferability. It is evident that our BPA achieves the best attack performance among various methods. Overall, BPA outperforms LinBP and Ghost by 8 . 18% and 7 . 90% for PGD, and 2 . 43% and 2 . 46% for MI-FGSM. These results further validate the effectiveness of BPA in targeted attacks, demonstrating its superiority over the baselines. The improved performance of BPA showcases its potential and generality in enhancing targeted attacks on various models.

A.7 Additional Evaluation on Modifications of ReLU Layers with BPA and LinBP

In this work, we find that the truncation of non-linear layers (e.g., ReLU, max-pooling) decays the relevance between the gradient w.r.t . the input and loss. Making the model more linear can enhance such relevance, thus improving the transferability. However, making the model more linear is not optimal. Taking ReLU for example, we compare the transferability of PGD on LinBP and BPA by

solely changing the derivatives of ReLU. Here LinBP makes the model more linear than our BPA. As shown in Table A11, BPA exhibits better transferability than LinBP, which supports our argument.

Table A11: Untargeted attack success rates (%) of PGD on nine models when generating adversarial examples on ResNet-50 with modifications on ReLU layers.

A.8 Comparison between Random Replacement and BPA on Max-pooling Layers

For max-pooling, the initial replacement of zeros in the gradient with ones results in an increase in relevance and subsequent improvement in attack performance. However, as the number of replaced zeros increases, the vallina BP faces challenges in accurately discerning which elements are critical that are related to the magnitude of input values. This indiscriminate replacement introduces a substantial error, leading to a decay in attack performance. Hence, we also compare randomly replacing the zeros with ones using the probability of 0.3 (the best reult in Fig. 2c) and BPA for max-pooling on ResNet-50. As shown in Table A12, BPA exhibits better transferability, which further validates our motivation and indicates that the effectiveness of our BPA is not solely attributed to achieving linearity.

Table A12: Untargeted attack success rates (%) of PGD on nine models when generating adversarial examples on ResNet-50 with modifications on max-pooling layers.

A.9 Relevance between Gradient w.r.t . Input and Loss Function

The relevance between gradient w.r.t . input and loss function indicates the sensitivity of the loss function to changes in the input when taking a small step in the direction of gradient. This relevance can be formally defined as follows:

Definition 1 (Relevance between gradient w.r.t . input and loss function) Given an input x , loss function J ( x ) and a step size ϵ , the relevance between gradient w.r.t . input and objective function can be defined as J ( x + ϵ ·∇ x J ( x )) -J ( x ) ϵ .

ReLU helps address the vanishing gradient issue during the training of deep models by eliminating some gradients. However, BPA focuses on boosting transferability of adversarial examples generated on such ReLU-activated networks. To effectively generate adversarial examples, it is crucial that the gradient w.r.t . the input provides a reliable direction to maximize the loss. Unfortunately, the truncation of ReLU makes the calculated gradient unable to provide such exactly precise direction, e.g ., making the gradient weakened to some extent. Similarly, max-pooling also truncates the gradient during the backpropagation, causing the gradient unable to indicate an exactly precise direction.

As summarized above, the truncation of ReLU and max-pooling drops gradient (introduces zeros) in the backpropagation process, which decays the relevance. We also calculate the Relevance using vallina backpropagation and our backpropagation on ResNet-50 using 1 , 000 images. BPA achieved the relevance of 240 . 51 , while the gradient calculated by the vallina backpropgation achieves lower relevance( 149 . 23 ), which supports our hypothesis that non-linear layers can decay the relevance .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents the Backward Propagation Attack (BPA) as a solution to the issue of gradient truncation in transfer-based attacks within adversarial machine learning. Through empirical validation, the authors highlight the negative impact of gradient truncation on adversarial transferability and propose modifications to the backward propagation process. BPA effectively enhances the relevance of the gradient between the loss function and the input, resulting in significant improvements in various untargeted and targeted transfer-based attacks compared to baseline approaches. This work not only provides valuable insights for enhancing adversarial transferability but also suggests promising directions for bolstering model robustness.\n\nSTRENGTHS:\nThe paper demonstrates originality by identifying and addressing the issue of gradient truncation in transfer-based attacks, which has not been extensively explored before. The proposed Backward Propagation Attack (BPA) introduces modifications to the backward propagation process, mitigating the detrimental effect of gradient truncation. The empirical evaluation on the widely used ImageNet dataset showcases the high quality of the research, with rigorous experimentation and comparisons against baselines. The clarity of the paper is commendable, effectively communicating the problem, the proposed solution, and the experimental methodology. The findings are significant as they contribute to advancing transfer-based attacks, provide new insights into adversarial transferability, and have potential implications for improving model robustness.\n\nWEAKNESSES:\nWhile the paper presents valuable contributions, there are a few areas that could be improved. Firstly, the theoretical underpinnings of the proposed modifications to the gradient truncation during the backward propagation process could be further elaborated to enhance the understanding of the approach. Additionally, the limitations of the proposed techniques could be further discussed to provide a comprehensive view of its effectiveness, such as the potential performance decrease caused by replacing the ReLU function. Furthermore, the paper focuses mainly on the ImageNet dataset, and it would be beneficial to explore the generalizability of the approach on other datasets or domains. Overall, addressing these weaknesses would strengthen the paper and provide further insights into the proposed approach.\n\nQUESTIONS:\n1.\tThe paper proposes modifications to the backward propagation process to mitigate gradient truncation. Could the authors provide more theoretical justification for the choice of these specific modifications?\n2.\tThe experimental evaluation is conducted on the ImageNet dataset. Have you considered testing the proposed BPA method on other datasets to assess its generalizability? It would be valuable to understand how the approach performs across different domains and datasets.\n3.\tIt would be beneficial to discuss the limitations of the proposed BPA method. Are there any specific scenarios or conditions where the effectiveness of BPA may be compromised? Besides, could the authors explain the main improvements of BPA compared with LinBP?\n4.\tThe paper primarily focuses on the enhancement of transfer-based attacks. Could you discuss potential applications or implications of the proposed BPA method in other areas of adversarial training, such as these gradient-based white-box attacks?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper  have developed a new method, Backward Propagation Attack (BPA), to enhance the transferability of adversarial examples. BPA uses a non-monotonic function as the derivative of ReLU and incorporates softmax with temperature to smooth the derivative of max-pooling, reducing information loss during gradient backward propagation. The method has demonstrated significant improvements in adversarial transferability on the ImageNet dataset.\n\nSTRENGTHS:\n**Originality**: To the best of my knowledge, the idea that modifying the derivative of max-pooling, Relu activation for improving adversarial transferability is new. \n\n **Clarity**: This paper is well-structured and easy to follow. \n\n**Significance**: To understand and improve the transferability of adversarial examples is important in the domain of adversarial examples\n\nWEAKNESSES:\n- The chosen baselines are exclusively drawn from model-related attacks. It might be beneficial to include a few from input-transformation-based attacks. Furthermore, it would provide a more comprehensive evaluation to examine if your technique could potentially work in conjunction with input-transformation-based attacks such as TIM[1]/SIN[2]/DIM[3] for further enhancing the transferability.\n\n- There's a relevant piece of work that hasn't been included.[4]\n\n[1]  Dong, Yinpeng, et al. \"Evading defenses to transferable adversarial examples by translation-invariant attacks.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n\n[2] Lin, Jiadong, et al. \"Nesterov accelerated gradient and scale invariance for adversarial attacks.\" arXiv preprint arXiv:1908.06281 (2019).\n\n[3] Xie, Cihang, et al. \"Improving transferability of adversarial examples with input diversity.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.\n\n[4] Huang, Tianjin, et al. \"Direction-aggregated attack for transferable adversarial examples.\" ACM Journal on Emerging Technologies in Computing Systems (JETC) 18.3 (2022): 1-22.\n\nQUESTIONS:\n- Could your technique potentially work in conjunction with input-transformation-based attacks such as TIM/SIN/DIM  for further improving the transferability?\n\nLIMITATIONS:\nthe authors adequately addressed the limitations", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors find that non-linear layers make the gradient imprecise, leading to limited adversarial transferability. Based on this finding, they propose Backward Propagation Attack (BPA) to increase such relevance. In particular, BPA adopts a non-monotonic function as the derivative of ReLU and incorporates softmax function to smooth the derivative of max-pooling. Extensive experiments have shown the high effectiveness of the proposed BPA.\n\nSTRENGTHS:\n1.The paper is well-written and easy to follow.\n2.The motivation is clear and reasonable. Intuitively, the non-linear layers can indeed truncate the gradient during the backward propagation.\n3.The proposed method is novel and interesting. Simply adjusting the gradient calculation of non-linear layers can significantly boost the adversarial transferability, which also validates the motivation.\n4.Different from some works which is limited to some specific architectures, the proposed method is general to existing CNNs.\n5.The authors have conducted experiments on both untargeted and targeted attacks, which can validate its high effectiveness and show good generality when combined with existing model-related methods.\n\nWEAKNESSES:\n1.For ReLU activation, the authors adopt the derivative of SiLU function. As shown in Figure 3, the derivative of SiLU function is mainly different from that of ReLU function in the range of [-5, 5]. I am curious how many elements the derivative matrix are in such range so that BPA can boost the transferability.\n2.Recently, transforming the input image for gradient calculation has shown great effectiveness in boosting adversarial transferability, such as DIM, Admix, SSA and so on. It is expected to see the results on such attacks to further validate the effectiveness of BPA on various transfer-based attacks.\n\nQUESTIONS:\n1.What is the temperature coefficient for max-pooling?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper introduces a new way to improve adversarial transferability by removing gradient truncation in the surrogate model. Specifically, the method replaces gradients of relu and maxpool with softened version to avoid gradient truncation. Experiments are conducted over many models and show gains upon existing methods.

STRENGTHS:
- The method seems well motivated, novel and straightforward.
- The empirical gain is substantial and general across different settings
- The paper is well written overall and easy to follow.

WEAKNESSES:
It is not clear to me whether the improvement is caused by the extra design to align with the function better, or just simply by making the model more linear as suggested in LinBP. For the relu case, it seems that replacing gradient with 1 is the best as shown in prelimiary study. It is not clear whether this is still the case in the general experiments due to lack of ablation. For maxpool, Fig 2 also shows that the attack is best at some middle point. Does this relate to linearity too? And why does the success rate first increases then decreases?

QUESTIONS:
See above

LIMITATIONS:
Not addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Backward Propagation Attack (BPA) to improve the transferability of adversarial examples generated based on the iterative gradient ascent procedure. The paper first empirically demonstrates that the non-linear layers such as ReLU and max-pooling truncate the loss gradient, and such a truncation procedure negatively affects the transferability of adversarial perturbations. Motivated by this observation, the paper proposes to modify the gradient computation for the ReLU and max-pooling layers. Particularly, consider a trained source model from which the adversarial perturbations are generated. BPA proposes to replace the derivative of the ReLU functions in the source model with the derivative of the SiLU function. Similarly, the derivative of the max-pooling function in the source model is replaced with a softmax function with a tunable temperature coefficient. Finally, results on randomly sampled 5000 Imagenet images demonstrate the improved transferability of adversarial perturbations generated based on PGD and MI-FGSM.\n\nSTRENGTHS:\nOriginality: Modifying the backward propagation procedure of the non-linear layers when computing the loss gradient during the process of generating adversarial examples is an interesting approach to improve the transferability of the perturbations. \n\nQuality: The paper is well-written. The proposed BPA method is motivated using empirical observations. The evaluations include the transferability of perturbations under both targeted and untargeted settings.\n\nClarity: The motivation behind the proposed method as well as the structure of the paper is clear.\n\nSignificance: The adversarial robustness of deep learning models against transfer-based perturbations poses a practical security concern. This paper proposes a method that can be used in combination with existing gradient-based attack algorithms to improve the transferability of perturbations.\n\nWEAKNESSES:\nThe clarity of the paper can be improved. There are some terms that are key to understanding the message which the author wants to convey, but the lack of rigorous definitions makes it difficult to understand. (See Questions for a list of clarifications )\n\nThe evaluations in the paper are performed based on perturbations bounded by the L_inf norm with an $\\epsilon$ of $\\frac{8}{255}$. In Ln 242, it states that the choice of $\\epsilon$ aligns with existing work (which one?), whereas most existing work considers $\\frac{16}{255}$ (LBP, MI, VI/VMI, etc). Especially since one of the main contributions of the proposed method is the improvement of transferability over other sota methods (ie. \"remarkable margin\" ln284, ln296, ln 358), it is important to make sure the evaluations and the comparison with other methods are performed fairly.\n\nQUESTIONS:\nQuestions/clarifications:\nIn the abstract, why does the truncation make the gradient imprecise? The ReLU and the max-pooling layers are part of the network definition, so the derivative is exactly what it is supposed to be and I do not see the lack of precision. What does it mean by the relevance between gradient wrt input and the loss function? Since the term \"relevance\" appears later in the paper as well (eg. ln45, 58, 148, 315), having a more rigorous definition is warranted.\n\nFigure 2 is a crucial element in motivating the proposed method. Is the success rate averaged over nine victim models? What is the variance among the results?\n\nLn 124 of Sec. 3.1: the paper first establishes that $z_i$ is the input to the function $f_i$ which returns $z_{i+1}$. When predicting the label, why is $z_l$ used as an input to the function $f_{l+1}$, shouldn't it be $z_{l+1}$?\n\nLn 138: Shouldnt the j-th element of $\\frac{\\partial z_{k+1}}{\\partial z_k}$ be 1 if $f(z_{k})_j >0$? \n\nLn 139 \"As a result, the gradient is effectively limited or weakened to some extent\". ReLU was designed to address the vanishing gradient problem in training deep models. By having a step function as its derivative, it allows some gradient information to pass freely while simultaneously not updating other weights. Therefore, this is exactly how the backward propagation was designed to be with ReLU-activated networks.\n\nLn 414: Similarly, this is exactly how bp is supposed to behave with max-pooling.\n\nLn 149: What does it mean by \"decaying the relevance\"? \n\nLn 185-186: Is this a prior result? or a claim made in this paper supported by evidence?\n\nSuggestions: Could we replace the y-axis (\"Attack success rate\") with the adversarial loss value on the source model? That is to plot $\\ell(x+\\Delta x)$, where $\\Delta x$ is generated using fgsm/i-fgsm/mi-fsgm on the source model modified using the various method. If we observe a higher loss value after the modification, it indeed shows that the derivative of those nonlinear layers prevents the perturbations from reaching maxima. On the other hand, if we observe a lower loss value after the modification, this means the truncation effect of the derivative of the non-linear functions leads to perturbations overfit to the source model, despite having higher losses.\n\nOther minor suggestions:\nFigure 1 should not come after Figure 2.\n\nLIMITATIONS:\nThere is no discussion of the limitation. For instance, the proposed paper is only suitable for attack algorithms that are based on gradient computations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}