SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xiaosen Wang $^{1*}$ , Kangheng Tong $^{2*}$ , Kun He $^{2\dagger}$
$^{1}$ Huawei Singular Security Lab
$^{2}$ School of Computer Science and Technology, Huazhong University of Science and Technology {xiaosen, tongkangheng, brooklet60}@hust.edu.cn
Transfer-based attacks generate adversarial examples on the surrogate model, which can mislead other black-box models without access, making it promising to attack real-world applications. Recently, several works have been proposed to boost adversarial transferability, in which the surrogate model is usually overlooked. In this work, we identify that non-linear layers (e.g. ReLU, max-pooling, etc.) truncate the gradient during backward propagation, making the gradient w.r.t. input image imprecise to the loss function. We hypothesize and empirically validate that such truncation undermines the transferability of adversarial examples. Based on these findings, we propose a novel method called Backward Propagation Attack (BPA) to increase the relevance between the gradient w.r.t. input image and loss function so as to generate adversarial examples with higher transferability. Specifically, BPA adopts a non-monotonic function as the derivative of ReLU and incorporates softmax with temperature to smooth the derivative of max-pooling, thereby mitigating the information loss during the backward propagation of gradients. Empirical results on the ImageNet dataset demonstrate that not only does our method substantially boost the adversarial transferability, but it is also general to existing transfer-based attacks. Code is available at https://github.com/Trustworthy-AI-Group/RPA.
Deep Neural Networks (DNNs) have been widely applied in various domains, such as image recognition [37, 13, 16], object detection [34], face verification [52, 41], etc. However, their susceptibility to adversarial examples [39, 10], which are carefully crafted by adding imperceptible perturbations to natural examples, has raised significant concerns regarding their security. In recent years, the generation of adversarial examples, aka adversarial attacks, has garnered increasing attention [31, 21, 6, 55, 44, 24] in the research community. Notably, there has been a significant advancement in the efficiency and applicability of adversarial attacks [19, 1, 47, 6, 57, 59, 51], making them increasingly viable in real-world scenarios.
By exploiting the transferability of adversarial examples across different models [29], transfer-based attacks generate adversarial examples on the surrogate model to fool the target models [6, 57, 12, 43, 58]. Unlike other types of attacks [2, 19, 1, 47], transfer-based attacks do not require direct access to the victim models, making them particularly applicable for attacking online interfaces. Consequently, transfer-based attacks have emerged as a prominent branch of adversarial attacks. However, it is worth noting that the early white-box attacks [10, 31, 21] often exhibit poor transferability despite demonstrating superior performance within the white-box setting.
*The first two authors contribute equally.
To this end, different techniques have been proposed to enhance adversarial transferability, such as momentum-based attacks [6, 27, 43, 46, 9, 59], input transformations [57, 7, 45, 30, 8, 48, 42], advanced objective functions [17, 54, 49], and model-related attacks [23, 53, 12, 50]. Among these techniques, model-related attacks are particularly valuable due to their ability to exploit the characteristics of surrogate models. Model-related attacks offer a unique perspective on adversarial attacks by leveraging the knowledge gained from surrogate models, which can also shed new light on the design of more robust models. In despite of their potential significance, model-related attacks have been somewhat overlooked compared to other types of transfer-based attacks.
Since transfer-based attacks mainly design various gradient ascend methods to generate adversarial examples on the surrogate model, in this work, we first revisit the backward propagation procedure. We find that non-linear layers (e.g., activation function, max-pooling, etc.) often truncate the gradient of loss w.r.t. the feature map, which diminishes the relevance of the gradient between the loss and input image. We assume and empirically validate that such gradient truncation undermines the adversarial transferability. Based on this finding, we propose Backward Propagation Attack (BPA), which modifies the calculation for the derivative of ReLU activation function and max-pooling layers during the backward propagation process. With these modifications, BPA mitigates the negative impact of gradient truncation and improves the transferability of adversarial attacks.
Our main contribution can be summarized as follows:
- To our knowledge, this is the first work that proposes and empirically validates the detrimental effect of gradient truncation on adversarial transferability. This finding sheds new light on improving adversarial transferability and might provide new directions to boost the model robustness.
- We propose a model-related attack called BPA, that adopts a non-monotonic function as the derivative of the ReLU activation function and incorporates softmax with temperature to calculate the derivative of max-pooling. With these modifications, BPA mitigates the negative impact of gradient truncation and enhances the relevance of gradient between the loss function and the input.
- Extensive experiments on ImageNet dataset demonstrate that BPA could significantly boost various untargeted and targeted transfer-based attacks and outperform the baselines with a substantial margin, emphasizing the effectiveness and superiority of our proposed approach.
In this section, we provide a brief overview of the existing adversarial attacks and defenses.
Existing adversarial attacks can be categorized into two groups based on the access to target model, namely white-box attacks and black-box attacks. In the white-box setting [10, 33, 31, 2], attackers have complete access to the structure and parameters of the target model. In the black-box setting, the attacker has limited or no information about the target model, making it applicable in the physical world. Black-box attacks can be further grouped into three classes, i.e., score-based attacks [4, 19], query-based attacks [3, 22, 47], and transfer-based attacks [6, 57, 43]. Among the three types of black-box attacks, transfer-based attacks generate adversarial examples on the surrogate model without accessing the target model, drawing increasing interest recently.
Since MI-FGSM [6] integrates momentum into I-FGSM [21] to stabilize the update direction and achieve improved transferability, various momentum-based attacks have been proposed to generate transferable adversarial examples. For instance, NI-FGSM [27] leverages Nesterov Accelerated Gradient for better transferability. VMI-FGSM [43] refines the current gradient using the gradient variance from the previous iteration, resulting in more stable updates. EMI-FGSM [46] enhances the momentum by averaging the gradient of several data points sampled in the previous gradient direction.
On the other hand, input transformations that modify the input image prior to gradient calculation have proven highly effective in enhancing adversarial transferability, such as DIM [56], TIM [7], SIM [27], Admix [45], SSA [30] and so on. Among these attacks, Admix introduces a small segment of an image from different categories, while SSA applies frequency domain transformations to the input image, both of which have demonstrated superior performance in generating transferable adversarial examples.
Several studies have explored the utilization of more sophisticated objective functions to enhance transferability in adversarial attacks. ILA [17] employs fine-tuning techniques to increase the similarity of feature differences between the original or current adversarial example and a benign sample. ATA [54] maximizes the disparity of attention maps between a benign sample and an adversarial example. FIA [49] minimizes a weighted feature map in an intermediate layer to disrupt significant object-aware features.
A few works have emphasized the significance of the surrogate model in generating highly transferable adversarial examples. Ghost network [23] attacks a set of ghost networks generated by densely applying dropout at the intermediate features. On the other hand, another line of work focuses on the gradient during backward propagation. SGM [54] adjusts the decay factor to incorporate more gradients from the skip connections of ResNet to generate more transferable adversarial examples. LinBP [12] performs backward propagation in a more linear fashion by setting the gradient of ReLU as a constant of 1 and scaling the gradient of residual blocks. In this work, we find that the gradient truncation introduced by non-linear layers undermines the transferability and modify the backward propagation so as to generate more transferable adversarial examples.
2.2 Adversarial Defenses
The existence of adversarial examples poses a significant security threat to deep neural networks (DNNs). To mitigate this impact, researchers have proposed various methods, among which adversarial training has emerged as a widely used and effective approach [10, 21, 31]. By augmenting the training data with adversarial examples, this method enhances the robustness of trained models against adversarial attacks. For instance, Tramér et al. [40] introduce ensemble adversarial training, a technique that generates adversarial examples using multiple models simultaneously, which shows superior performance against transfer-based attacks.
Although adversarial training is effective, it comes with high training costs, particularly for large-scale datasets and complex networks. Consequently, researchers have proposed innovative defense methods as alternatives. Guo et al. [11] utilize various input transformations such as JPEG compression and total variance minimization to eliminate adversarial perturbations from input images. Xie et al. [56] mitigate adversarial effects through random resizing and padding of input images. Liao et al. [25] propose training a high-level representation denoiser (HGD) specifically designed to purify input images. Nasser [32] introduce a neural representation purifier (NRP) by a self-supervised adversarial training mechanism to purify the input sample. Various certified defenses aim to provide a verified guarantee in a specific radius, such as randomized smoothing (RS) [5].
In this section, we analyze the backward propagation procedure and identify that the gradient truncation introduced by non-linear layers undermines the adversarial transferability. Based on this finding, we propose Backward Propagation Attack (BPA) to mitigate such negative effect and gain more transferable adversarial examples.
3.1 Backward Propagation for Adversarial Transferability
Given an input image  $x$  with ground-truth label  $y$ , a classifier  $f$  with  $l$  successive layers ( $e.g., z_{i+1} = \phi_i(f_i(z_i)), z_0 = x$ ) predicts the label  $f(x) = f_l(z_l) = y$  with high probability. Here  $\phi(\cdot)$  is a non-linear activation function ( $e.g., \text{ReLU}$ ) or identity function if there is no activation function after  $i$ -th layer  $f_i$ . The attacker aims to find an adversarial example  $x^{adv}$  adhering the constraint of  $\| x^{adv} - x \|_p \leq \epsilon$ , but resulting in  $f(x^{adv}) \neq f(x) = y$  for untargeted attack and  $f(x^{adv}) = y_t$  for targeted attack. Here  $\epsilon$  is the maximum perturbation magnitude,  $y_t$  is the target label, and  $\| \cdot \|_p$  denotes the  $p$ -norm distance. For brevity, the following description will focus on non-targeted attacks with  $p = \infty$ . Let  $J(x, y; \theta)$  denote the loss function of classifier  $f$  ( $e.g., \text{the cross-entropy loss}$ ). Existing white-box attacks often solve the following constrained maximization problem using the gradient  $\nabla_x J(x, y; \theta)$ :
$$
x ^ {a d v} = \underset {\| x ^ {\prime} - x \| _ {p} \leq \epsilon} {\operatorname {a r g m a x}} J \left(x ^ {\prime}, y; \theta\right). \tag {1}
$$
Based on the chain rule, we can calculate the gradient as follows:
$$
\nabla_ {x} J (x, y; \theta) = \frac {\partial J (x , y ; \theta)}{\partial f _ {l} (z _ {l})} \left(\prod_ {i = k + 1} ^ {l} \frac {\partial \phi_ {i} \left(f _ {i} \left(z _ {i}\right)\right)}{\partial z _ {i}}\right) \frac {\partial z _ {k + 1}}{\partial z _ {k}} \frac {\partial z _ {k}}{\partial x}, \tag {2}
$$
where  $0 < k < l$  is the index of an arbitrary layer. Without loss of generality, we explore the backward propagation when passing the  $k$ -th layer as follows:
- A fully connected or convolutional layer followed by a non-linear activation function. Taking ReLU activation (i.e.,  $\phi_{k}$ ) for example, the  $j$ -th element in the gradient w.r.t. the  $k$ -th feature,  $[\frac{\partial z_{k+1}}{\partial z_{k}}]_{j}$ , will be one if  $z_{k,j} > 0$  and otherwise,  $[\frac{\partial z_{k+1}}{\partial z_{k}}]_{j}$  will be zero. These zero gradients in  $\frac{\partial z_{k+1}}{\partial z_{k}}$  can lead to the truncation of gradient of the loss function  $\frac{\partial J(x,y;\theta)}{\partial z_{k}}$  w.r.t. the input image. As a result, the gradient is effectively limited or weakened to some extent.
- Max-pooling layer. As shown in Fig. 1, max-pooling calculates the maximum value (orange block) within a specific patch. Hence, the derivative  $\frac{\partial z_{k+1}}{\partial z_k}$  will be a binary matrix, containing only ones at locations corresponding to the orange blocks. In this case, approximately  $3/4$  of the elements in the given sample will be zeros. This means that max-pooling tends to discard a significant portion of the gradient information contained in  $\frac{\partial z_k}{\partial x}$ , resulting in a truncated gradient.
The truncation of gradient caused by non-linear layers (e.g., activation function, max-pooling) can limit or dampen the flow of gradients during backward propagation, which decays the relevance among the gradient between the loss and input. Considering that many existing attacks rely on maximizing the loss by leveraging the gradient information, we make the following assumption:
Assumption 1 The truncation of gradient  $\nabla_{x}J(x,y;\theta)$  introduced by non-linear layers in the backward propagation process decays the adversarial transferability.
To validate Assumption 1, we conduct several experiments using FGSM, I-FGSM and MI-FGSM. The detailed experimental settings are summarized in Sec. 4.1.
Figure 1: A max-pooling layer with  $2 \times 2$  kernel size and stride  $s = 2$  on a  $4 \times 4$  feature map in the forward propagation.
- Randomly mask the gradient. To investigate the impact of gradient truncation on adversarial transferability, we introduce a random masking operation to increase the probability of gradient truncation between stage 3 and stage 2 of ResNet-50. Fig. 2a illustrates the attack performance with various mask probabilities. As the mask probability increases, more zeros appear in the derivative, indicating a higher degree of gradient truncation. Consequently, the larger truncation probability renders the gradient less relevant to the loss function, decreasing the attack performance of the three evaluated methods. These findings validate our hypothesis that the truncation of gradient negatively impacts adversarial transferability and highlight the importance of preserving gradient information to maintain the effectiveness of adversarial attacks across various models.
- Recover the gradient of ReLU or max-pooling layers. In contrast, it is expected that mitigating the truncation of gradient can improve the adversarial transferability. To explore this, we randomly replaced the zeros in the derivative of ReLU or max-pooling operations with ones, using various replacement probabilities. a) In Fig. 2b, as the probability of replacement increases, fewer gradients are truncated across ReLU, resulting in improved adversarial transferability on all the three attacks. Notably, these attacks achieve their best performance when the derivative consists entirely of ones, which aligns with LinBP [12]. b) As illustrated in Fig. 2c, when the ratio of ones in the derivative of max-pooling increases (i.e., the replacement probability increases), the attack performance initially improves, reaching a peak around 0.3. Subsequently, the attack performance gradually decreases but remains superior to vanilla backward propagation. This highlights that we need a more suitable approximation for the derivative calculation of max-pooling, which is detailed in Sec. 3.2. These results suggest that decreasing the probability of gradient truncation in max-pooling is beneficial for enhancing adversarial transferability.
Overall, these findings validate Assumption 1 that the truncation of gradients negatively impacts adversarial transferability. By preserving gradient information and carefully adjusting the replacement probabilities, it is possible to improve the effectiveness of adversarial attacks across different models.
(a) Randomly mask the gradient
Figure 2: Average untargeted attack success rates  $(\%)$  of FGSM, I-FGSM and MI-FGSM when we randomly mask the gradient, recover the gradient of ReLU or max-pooling layers, respectively. The adversarial examples are generated on ResNet-50 and tested on all the nine victim models illustrated in Sec. 4.1. Raw data is provided in Appendix A.1.
(b) Recover the gradient of ReLU
(c) Recover the gradient of max-pooling
3.2 Mitigating the Negative Impact of Gradient Truncation
In Sec. 3.1, we demonstrate that reducing the probability of gradient truncation in non-linear layers can enhance adversarial transferability. However, setting all elements in the corresponding derivative to one is not optimal for generating transferable adversarial examples. Here we investigate how to modify the backward propagation process of non-linear layers to further enhance the transferability.
Within the standard backward propagation procedure, the elements comprising the derivative depend on the magnitudes of the associated feature map. This observation provides an impetus for considering the intrinsic characteristics of the underlying features when diminishing the probability of gradient truncation. To this end, we modify the
gradient calculation for the ReLU activation function and max-pooling in the backward propagation procedure as follows:
Figure 3: Various candidate derivatives of ReLU function.
- Gradient calculation for ReLU. To ensure precise gradient calculation, it is important to exclude extreme values from consideration when calculating the gradient, while still maintaining the relationship between the elements in the derivative and the magnitude of the feature map. Among the family of ReLU activation functions, SiLU [14] provides a smooth and continuous gradient across the entire input range and is less susceptible to gradient saturation issues. Hence, we propose using the derivative of SiLU to calculate the gradient of ReLU during the backward propagation process, i.e.,  $\frac{\partial z_{i+1}}{\partial z_i} = \sigma(z_i) \cdot (1 + z_i \cdot (1 - \sigma(z_i)))$ , where  $\sigma(\cdot)$  is the Sigmoid function. This formulation allows our gradient calculation to reflect the input magnitude mainly within the input range around  $[-5, 5]$ , while closely resembling the behavior of ReLU when the input is outside this range. As shown in Fig. 3, our proposed gradient calculation method demonstrates improved alignment with the input's magnitude compared to both the original derivative of ReLU and the derivative used in LinBP. By leveraging the smoothness and non-monotonicity of SiLU, we can obtain more accurate and reliable gradient information for ReLU.
- Gradient calculation for max-pooling. Similar to the gradient calculation for ReLU, it is essential to exclude extreme values and ensure that the gradient remains connected to the magnitude of the feature map. Furthermore, in the case of max-pooling, the summation of gradients within each window should remain at one to minimize modifications to the gradient. To address these considerations, we propose using the softmax function to calculate the gradient within each window  $w$  of the max-pooling operation:
$$
\left[ \frac {\partial z _ {k + 1}}{\partial z _ {k}} \right] _ {i, j, w} = \frac {e ^ {t \cdot z _ {k , i , j}}}{\sum_ {v \in w} e ^ {t \cdot v}}, \tag {3}
$$
where  $t$  is the temperature coefficient to adjust the smoothness of the gradient. If the feature  $z_{k,i,j}$  is related to multiple windows (i.e., the stride is smaller than the size of max-pooling), we sum its gradient calculated by Eq. 3 in each window as the final gradient.
Attacker	Method	Inc-v3	IncRes-v2	DenseNet	MobileNet	PNASNet	SENet	Inc-v3ens3	Inc-v3ens4	IncRes-v2ens
PGD	N/A	16.34	13.38	36.86	36.12	13.46	17.14	10.24	9.46	5.52
SGM	23.68	19.82	51.66	55.44	22.12	30.34	13.78	12.38	7.90
LinBP	27.22	23.04	59.34	59.74	22.68	33.72	16.24	13.58	7.88
Ghost	17.74	13.68	42.36	41.06	13.92	19.10	11.60	10.34	6.04
BPA	35.36	30.12	70.70	68.90	32.52	42.02	22.72	19.28	12.40
MI-FGSM	N/A	26.20	21.50	51.50	49.68	22.92	30.12	16.22	14.58	9.00
SGM	33.78	28.84	63.06	65.84	31.90	41.54	19.56	17.48	10.98
LinBP	35.92	29.82	68.66	69.72	30.24	41.68	19.98	16.58	9.94
Ghost	29.76	23.68	57.28	56.10	25.00	34.76	17.10	14.76	9.50
BPA	47.58	41.22	80.54	79.40	44.70	54.28	32.06	25.98	17.46
VMI-FGSM	N/A	42.68	36.86	68.82	66.68	40.78	46.34	27.36	24.20	17.18
SGM	50.04	44.28	77.56	79.34	48.58	56.86	32.22	27.72	19.66
LinBP	47.70	40.40	77.44	78.76	41.48	52.10	28.58	24.06	16.60
Ghost	47.82	41.42	75.98	73.40	44.84	52.78	30.84	27.18	19.08
BPA	55.00	48.72	85.44	83.64	52.02	60.88	38.76	33.70	23.78
ILA	N/A	29.10	26.08	58.02	59.10	27.60	39.16	15.12	12.30	7.86
SGM	35.64	32.34	65.20	71.22	34.20	46.72	17.10	13.86	9.08
LinBP	37.36	34.24	71.98	72.84	35.12	48.80	19.38	14.10	9.28
Ghost	30.06	26.50	60.52	61.74	28.68	40.46	14.84	12.54	7.90
BPA	47.62	43.50	81.74	80.88	47.88	60.64	27.94	20.64	14.76
SSA	N/A	35.78	29.58	60.46	64.70	25.66	34.18	20.64	17.30	11.44
SGM	45.22	38.98	70.22	78.44	35.30	46.06	26.28	21.64	14.50
LinBP	48.48	41.90	75.02	78.30	36.66	49.58	28.76	23.64	15.46
Ghost	36.44	28.62	61.12	66.80	24.90	33.98	20.58	16.84	10.82
BPA	51.36	44.70	76.24	79.66	39.38	50.00	32.10	26.44	18.20
Table 1: Untargeted attack success rates (\%) of various adversarial attacks on nine models when generating the adversarial examples on ResNet-50 w/wo various model-related methods.
In practice, we adopt the above two strategies to calculate the gradient of ReLU and max-pooling during the backward propagation process. This approach allows us to circumvent the issue of gradient truncation introduced by these non-linear layers. We refer to this modified backward propagation technique as Backward Propagation Attack (BPA), which can be applied to existing CNNs to adapt to various transfer-based attack methods.
In this section, we conduct extensive experiments on standard ImageNet dataset [35] to validate the effectiveness of the proposed BPA. We first specify our experimental setup, then we conduct a series of experiments to compare BPA with existing state-of-the-art attacks under different settings. Additionally, we provide ablation studies to further investigate the performance and behavior of BPA.
Dataset. Following LinBP [12], we randomly sample 5,000 images pertaining to the 1,000 categories from ILSVRC 2012 validation set [35], which could be classified correctly by all the victim models.
Models. We select ResNet-50 [13] and VGG-19 [37] as our surrogate model for generating adversarial examples. As for the victim models, we consider six standardly trained networks, i.e., Inception-v3 (Inc-v3) [13], Inception-Resnet-v2 (IncRes-v2) [38], DenseNet [16], MobileNet-v2 [36], PNAS-Net [28], and SENet [15]. Additionally, we adopt three ensemble adversially trained models, namely ens3-adv-Inception-v3 (Inc-v3\_ens3), ens4-Inception-v3 (Inc-v3\_ens4), and ens-adv-Inception-ResNet-v2 (IncRes-v2\_ens) [40]. To address the issue of different input shapes required by these models, we adhere to the official pre-processing pipeline, including resizing and cropping techniques.