SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhijin Ge $^{1*}$ , Hongying Liu $^{2*}$ , Xiaosen Wang $^{3*}$ , Fanhua Shang $^{4\dagger}$ , Yuanyuan Liu $^{1\dagger}$
$^{1}$ School of Artificial Intelligence, Xidian University
$^{2}$ Medical College, Tianjin University, China
$^{3}$ Huawei Singular Security Lab
<sup>4</sup>College of Intelligence and Computing, Tianjin University
zhijinge@stu.xidian.edu.cn, hyliu2009@tju.edu.cn
xiaosen@hust.edu.cn, fhshang@tju.edu.cn, yyliu@xidian.edu.cn
Transfer-based attack adopts the adversarial examples generated on the surrogate model to attack various models, making it applicable in the physical world and attracting increasing interest. Recently, various adversarial attacks have emerged to boost adversarial transferability from different perspectives. In this work, inspired by the observation that flat local minima are correlated with good generalization, we assume and empirically validate that adversarial examples at a flat local region tend to have good transferability by introducing a penalized gradient norm to the original loss function. Since directly optimizing the gradient regularization norm is computationally expensive and intractable for generating adversarial examples, we propose an approximation optimization method to simplify the gradient update of the objective function. Specifically, we randomly sample an example and adopt a first-order procedure to approximate the Hessian/vector product, which makes computing more efficient by interpolating two neighboring gradients. Meanwhile, in order to obtain a more stable gradient direction, we randomly sample multiple examples and average the gradients of these examples to reduce the variance due to random sampling during the iterative process. Extensive experimental results on the ImageNet-compatible dataset show that the proposed method can generate adversarial examples at flat local regions, and significantly improve the adversarial transferability on either normally trained models or adversarially trained models than the state-of-the-art attacks. Our codes are available at: https://github.com/Trustworthy-AI-Group/PGN.
A great number of works have shown that Deep Neural Networks (DNNs) are vulnerable to adversarial examples [10, 16, 38, 26, 50], which are generated by applying human-imperceptible perturbations on clean input to result in misclassification. Furthermore, adversarial examples have an intriguing property of transferability [10, 11, 31, 61, 59, 57], i.e., the adversarial example generated from the surrogate model can also misclass other models. The existence of transferability makes adversarial attacks practical to real-world applications because hackers do not need to know any information about the target model, which introduces a series of serious security problems in security-sensitive applications such as self-driving [65, 13] and face-recognition [44, 30].
† Corresponding authors
Although several attack methods [5, 16, 26, 28] have exhibited great attack effectiveness in the white-box setting, they have low transferability when attacking black-box models, especially for some advanced defense models [36, 47]. Previous works [10, 31, 61] attribute that the reason for adversarial examples shows weak transferability due to dropping into poor local maxima or overfitting the surrogate model, which is not likely to transfer across models. To address this issue, many methods have been proposed from different perspectives. Gradient optimization attacks [10, 31, 49, 64] attempt to boost black-box performance by advanced gradient calculation. Input transformation attacks [61, 11, 31, 51, 35, 15, 55] aim to generate adversarial examples with higher transferability by applying various transformations to the inputs. Especially, the above methods are mainly proposed from the perspective of optimization and generalization, which regard the process of generating adversarial examples on the white-box model as a standard neural network training process, and treat adversarial transferability as equivalent to model generalization [31]. Although these methods can improve the transferability of adversarial examples, there are still some gaps between white-box attacks and transfer-based black-box attacks.
Inspired by the observation that flat minima often result in better model generalization [24, 66, 41, 14], we assume and empirically validate that adversarial examples at a flat local region tend to have good transferability. Intuitively, we can achieve the flat local maxima for adversarial examples using a gradient regularization norm but it is computationally extensive to solve such a problem. To address this issue, we theoretically analyze the optimization process and propose a novel attack called Penalizing Gradient Norm (PGN). In particular, PGN approximates the Hessian/vector product by interpolating the first-order gradients of two samples. This approximation allows us to efficiently generate adversarial examples at flat local regions. To eliminate the error introduced by approximation, PGN incorporates the average gradient of several randomly sampled data points to update the adversarial perturbation. Our main contributions can be summarized as follows:
- To the best of our knowledge, it is the first work that empirically validates that adversarial examples located in flat regions have good transferability.
- We propose a novel attack called Penalizing Gradient Norm (PGN), which can effectively generate adversarial examples at flat local regions with better transferability.
- Empirical evaluations show that PGN can significantly improve the attack transferability on both normally trained models and adversarially trained models, which can also be seamlessly combined with various previous attack methods for higher transferability.
In this section, we provide a brief overview of the adversarial attack methods and introduce several studies on the flat local minima for model generalization.
2.1 Adversarial Attacks
In general, adversarial attacks can be divided into two categories, i.e., white-box attacks and black-box attacks. In the white-box setting, the target model is completely exposed to the attacker. For instance, Goodfellow et al. [16] proposed the Fast Gradient Sign Method (FGSM) to generate adversarial examples with one step of gradient update. Kurakin et al. [26] further extends FGSM to an iterative version with a smaller step size  $\alpha$ , denoted as I-FGSM. Madry et al. [36] extends I-FGSM with a random start to generate diverse adversarial examples. Existing white-box attacks have achieved superior performance with the knowledge of the target model. On the other hand, black-box attacks are more practical since they only access limited or no information about the target model. There are two types of black-box adversarial attacks [21]: query-based and transfer-based attacks. Query-based attacks [3, 6, 53] often take hundreds or even thousands of quires to generate adversarial examples, making them inefficient in the physical world. In contrast, transfer-based attacks [61, 49, 52, 4, 54, 56, 48] generate adversarial examples on the surrogate model, which can also attack other models without accessing the target model, leading to great practical applicability and attracting increasing attention.
Unfortunately, adversarial examples crafted by white-box attacks generally exhibit limited transferability. To boost adversarial transferability, various methods are proposed from the perspective of optimization and generalization. MI-FGSM [10] integrates momentum into I-FGSM to stabilize the update direction and escape from poor local maxima at each iteration. NI-FGSM [31] adopts
Nesterov's accelerated gradient [40] to further enhance the transferability. Wang et al. [49] tuned the gradient using the gradient variance of previous iteration to find a more stable gradient direction. Wang et al. [52] enhanced the momentum by accumulating the gradient of several data points in the direction of the previous gradient for better transferability.
Data augmentation, which has shown high effectiveness in improving model generalization [37, 63, 2, 8], has been widely studied to boost adversarial transferability. Xie et al. [61] adopted diverse input patterns by randomly resizing and padding to generate transferable adversarial examples. Dong et al. [11] utilized several translated images to optimize the adversarial perturbations, and further calculated the gradients by convolving the gradient at untranslated images with a kernel matrix for high efficiency. SIM [31] optimizes the adversarial perturbations over several scaled copies of the input images. Admix [51] mixes up a set of images randomly sampled from other categories while maintaining the original label of the input. Spectrum simulation attack (SSA) [35] transforms the input image in the frequency domain to craft more transferable adversarial examples.
Besides, some methods improve adversarial transferability from different perspectives. For instance, Liu et al. [32] proposed an ensemble attack, which simultaneously attacks multiple surrogate models. Wu et al. [59] employed an adversarial transformation network that can capture the most harmful deformations to adversarial noises. Qin et al. [42] injected the reverse adversarial perturbation at each step of the optimization procedure for better transferability.
2.2 Flat Minima
After Hochreiter et al. [20] pointed out that well-generalized models may have flat minima, the connection between the flatness of minima and model generalization has been studied from both empirical and theoretical perspectives [24, 41, 14, 66]. Li et al. [27] observed that skip connections promote flat minima, which helps explain why skip connections are necessary for training extremely deep networks. Similarly, Santurkar et al. [43] found that BatchNorm makes the optimization landscape significantly smooth in the training process. Sharpness-Aware Minimization (SAM) [14] improves model generalization by simultaneously minimizing the loss value and sharpness, which seeks the parameters in the neighborhoods with uniformly low loss values. Jiang et al. [23] studied 40 complexity measures and showed that a sharpness-based measure has the highest correlation with generalization. Zhao et al. [66] demonstrated that adding a gradient norm of the loss function can help the optimizer find flat local minima.
Given an input image  $x$  with its corresponding ground-true label  $y$ , the deep model  $f$  with parameter  $\theta$  is expected to output the prediction  $f(x; \theta) = y$  with high probability. Let  $\mathcal{B}_{\epsilon}(x) = \{x' : \| x' - x \|_p \leq \epsilon\}$  be an  $\epsilon$ -ball of an input image  $x$ , where  $\epsilon > 0$  is a pre-defined perturbation magnitude, and  $\| \cdot \|_p$  denotes the  $L_p$ -norm (e.g., the  $L_1$ -norm). The attacker aims to find an example  $x^{adv} \in \mathcal{B}_{\epsilon}(x)$  that misleads the classifier  $f(x^{adv}; \theta) \neq y$ . Let  $J(x, y; \theta)$  be the loss function (e.g., cross-entropy loss) of the classifier  $f$ . Existing white-box attacks such as [16, 26, 10] usually generate adversarial examples by solving the following maximization problem:
$$
\max  _ {x ^ {a d v} \in \mathcal {B} _ {\epsilon} (x)} J \left(x ^ {a d v}, y; \theta\right). \tag {1}
$$
These attack methods mainly generate adversarial examples through gradient iterations. For instance, I-FGSM [26] iteratively updates the adversarial perturbation as follows:
$$
x _ {t + 1} ^ {a d v} = \Pi_ {\mathcal {B} _ {\epsilon} (x)} \left[ x _ {t} ^ {a d v} + \alpha \cdot \operatorname {s i g n} \left(\nabla_ {x _ {t} ^ {a d v}} J \left(x _ {t} ^ {a d v}, y; \theta\right)\right) \right], \quad x _ {0} ^ {a d v} = x, \tag {2}
$$
where  $\Pi_{\mathcal{B}_{\epsilon}(x)}(\cdot)$  projects an input into  $\mathcal{B}_{\epsilon}(x)$ ,  $\alpha = \epsilon / T$ , and  $T$  is the total number of iterations. For black-box attacks, the gradient is not accessible so that we cannot directly solve Problem (1) like I-FGSM. To address such a issue, transfer-based attacks generate adversarial examples on an accessible surrogate model, which can be transferred to fool the target models.
3.2 Flat Local Maxima Tend to Improve Adversarial Transferability
In general, the adversarial transferability is equivalent to model generalization if we analogize the optimization of perturbation with the model training process [31, 49]. Existing works [24, 66, 41, 14] have demonstrated that flat local minima tend to generalize better than their sharp counterparts. This inspires us to assume that adversarial examples at a flat local region w.r.t. the loss function tend to have better transferability across various models. A rigorous description is given as follows:
Assumption 1. Given any small radius  $\zeta >0$  for the local region and two adversarial examples  $x_{1}^{adv}$  and  $x_{2}^{adv}$  for the same input image  $x$ , if  $\max_{x^{\prime}\in \mathcal{B}_{\zeta}(x_1^{adv})}\| \nabla_{x^{\prime}}J(x^{\prime},y;\theta)\| _2 < \max_{x^{\prime}\in \mathcal{B}_{\zeta}(x_2^{adv})}\| \nabla_{x^{\prime}}J(x^{\prime},y;\theta)\| _2$ , with high probability,  $x_{1}^{adv}$  tends to be more transferable than  $x_{2}^{adv}$  across various models.
SAM [14] has indicated that a flat local minimum is an entire neighborhood having both low loss and low curvature. Zhao et al. [66] also demonstrated that if the loss function has a smaller gradient value, this would indicate that the loss function landscape is flatter. Here we adopt the maximum gradient in the neighborhood to evaluate the flatness of the local region. Following Assumption 1, we introduce a regularizer, which minimizes the maximum gradient in the  $\zeta$ -ball of the input, into Eq. (1) as follows:
$$
\max  _ {x ^ {a d v} \in \mathcal {B} _ {\epsilon} (x)} \left[ J \left(x ^ {a d v}, y; \theta\right) - \lambda \cdot \max  _ {x ^ {\prime} \in \mathcal {B} _ {\zeta} \left(x ^ {a d v}\right)} \| \nabla_ {x ^ {\prime}} J \left(x ^ {\prime}, y; \theta\right) \| _ {2} \right], \tag {3}
$$
where  $\lambda \geq 0$  is the penalty coefficient. During the optimization process, we penalize the maximum gradient to perceive the sharper regions that can not be identified by using the averaged gradients. By optimizing Problem (3), we can find that the adversarial example with a small gradient of data points in its neighborhood should be at a flat local region. However, it is impractical to calculate the maximum gradient in the  $\zeta$ -ball. Hence, we approximately optimize Eq. (3) by randomly sampling an example  $x' \in \mathcal{B}_{\zeta}(x^{adv})$  at each iteration using I-FGSM [26] and MI-FGSM [10], respectively.
As shown in Fig. 1, the regularizer of gradients can significantly boost the adversarial transferability of either I-FGSM or MI-FGSM. In general, the average attack success rate is improved by a clear margin of  $5.3\%$  and  $7.2\%$  for I-FGSM and MI-FGSM, respectively. Such remarkable performance improvement strongly validates Assumption 1. RAP [42] also advocates that the adversarial example should be located at a flat local region. However, to the best of our knowledge, it is the first work that empirically validates that flat local maxima can result in good adversarial transferability. Based on this finding, we aim to improve the adversarial transferability by locating the adversarial examples in a flat local region, and a detailed approach for finding flat local maxima will be provided in Sec. 3.3.
Figure 1: The average attack success rates  $(\%)$  of I-FGSM and MI-FGSM w/wo the gradient regularization on seven black-box models. The adversarial examples are generated on Inc-v3.
3.3 Finding Flat Local Maxima
Although we have verified that adversarial examples at a flat local region tend to have better transferability, the optimization process involved in Eq. (3) is computationally expensive and intractable for generating transferable adversarial examples. The detailed ablation study is summarized in Sec. 4.6. To address this challenge, we propose an approximate optimization approach that efficiently optimizes Eq. (3) to generate more transferable adversarial examples.
Since  $x'$  in Eq. (3) is close to  $x^{adv}$  with a small  $\zeta$ , we assume that the effects of maximizing  $J(x^{adv}, y; \theta)$  and maximizing  $J(x', y; \theta)$  are expected to be equivalent. Then we can approximately simplify Eq. (3) as follows:
$$
\max  _ {x ^ {a d v} \in \mathcal {B} _ {\epsilon} (x)} \mathcal {L} \left(x ^ {a d v}, y; \theta\right) \approx J \left(x ^ {\prime}, y; \theta\right) - \lambda \cdot \| \nabla_ {x ^ {\prime}} J \left(x ^ {\prime}, y; \theta\right) \| _ {2}, \quad \text {s . t .} \quad x ^ {\prime} \in \mathcal {B} _ {\zeta} \left(x ^ {a d v}\right), \tag {4}
$$
where  $\mathcal{L}$  is a loss function. Here a penalized gradient norm is introduced into the original loss function  $J$  for achieving flat local maxima.
Algorithm 1 Penalizing Gradient Norm (PGN) attack method
Input: A clean image  $x$  with ground-truth label  $y$ , and the loss function  $J$  with parameters  $\theta$ .  
Parameters: The magnitude of perturbation  $\epsilon$ ; the maximum number of iterations,  $T$ ; the decay factor  $\mu$ ; the balanced coefficient  $\delta$ ; the upper bound (i.e.,  $\zeta$ ) of random sampling in  $\zeta$ -ball; the number of randomly sampled examples,  $N$ .  
1:  $g_0 = 0$ ,  $x_0^{adv} = x$ ,  $\alpha = \epsilon / T$ ;  
2: for  $t = 0, 1, \dots, T - 1$  do  
3: Set  $\bar{g} = 0$ ;  
4: for  $i = 0, 1, \dots, N - 1$  do  
5: Randomly sample an example  $x' \in \mathcal{B}_{\zeta}(x_t^{adv})$ ;  
6: Calculate the gradient at the sample  $x'$ ,  $g' = \nabla_{x'} J(x', y; \theta)$ ;  
7: Compute the predicted point by  $x^* = x' - \alpha \cdot \frac{g'}{\|\bar{g}'\|_1}$ ;  
8: Calculate the gradient of the predicted point,  $g^* = \nabla_{x^*} J(x^*, y; \theta)$ ;  
9: Accumulate the updated gradient by  $\bar{g} = \bar{g} + \frac{1}{N} \cdot [(1 - \delta) \cdot g' + \delta \cdot g^*]$ ;  
10: end for  
11:  $g_{t+1} = \mu \cdot g_t + \frac{\bar{g}}{\|\bar{g}\|_1}$ ;  
12: Update  $x_{t+1}^{adv}$  via  $x_{t+1}^{adv} = \Pi_{\mathcal{B}_\epsilon(x)}[x_t^{adv} + \alpha \cdot \mathrm{sign}(g_{t+1})]$ ;  
13: end for  
14: return  $x^{adv} = x_T^{adv}$ .  
Output: An adversarial example  $x^{adv}$ .
In practice, it is computationally expensive to directly optimize Eq. (4), since we need to calculate its Hessian matrix. Note that existing adversarial attacks typically rely on the sign of the gradient, rather than requiring an exact gradient value. Thus, we approximate the second-order Hessian matrix by using the finite difference method to accelerate the attack process.
Theorem 1 (Finite Difference Method [1]). Given a finite difference step size  $\alpha$  and one normalized gradient direction vector  $v = -\frac{\nabla_x J(x,y;\theta)}{\|\nabla_x J(x,y;\theta)\|_2}$ , the Hessian/vector product can be approximated by the first-order gradient as follows:
$$
\nabla_ {x} ^ {2} J (x, y; \theta) v \approx \frac {\left. \nabla_ {x} J (x , y ; \theta) \right| _ {x = x + \alpha \cdot v} - \nabla_ {x} J (x , y ; \theta)}{\alpha}. \tag {5}
$$
With the finite difference method, we can solve Eq. (4) approximately by using the Hessian/vector product for high efficiency. In particular, we can calculate the gradient at each iteration as follows.
Corollary 1. The gradient of the objective function (4) at the  $t$ -th iteration can be approximated as:
$$
\nabla_ {x _ {t} ^ {a d v}} \mathcal {L} \left(x _ {t} ^ {a d v}, y; \theta\right) \approx (1 - \delta) \cdot \nabla_ {x _ {t} ^ {\prime}} J \left(x _ {t} ^ {\prime}, y; \theta\right) + \delta \cdot \nabla_ {x _ {t} ^ {\prime}} J \left(x _ {t} ^ {\prime}, y; \theta\right) | _ {x _ {t} ^ {\prime} = x _ {t} ^ {\prime} + \alpha \cdot v}, \tag {6}
$$
where  $x_{t}^{\prime}$  is a point randomly sampled in  $\mathcal{B}_{\zeta}(x_{t}^{adv})$ ,  $\alpha$  is the iteration step size, and  $\delta = \frac{\lambda}{\alpha}$  is a balanced coefficient.
The detailed proof of Corollary 1 is provided in the Appendix. From Corollary 1, we can approximate the gradient of Eq. (4) by interpolating two neighboring gradients. This approximation technique significantly enhances the efficiency of the attack process.
In Eq. (4), we approximate the loss at the input image  $x^{adv}$  and the maximum gradient in  $\mathcal{B}_{\zeta}(x^{adv})$  by randomly sampling an example  $x' \in \mathcal{B}_{\zeta}(x^{adv})$ . However, this approach introduces variance due to the random sampling process. To address this issue, we randomly sample multiple examples and average the gradients of these examples to obtain a more stable gradient. Such averaging helps reduce the error introduced by the approximation and randomness, resulting in a more accurate gradient estimation. This, in turn, allows us to effectively explore flat local maxima and achieve better transferability.
In short, we introduce a novel attack method, called Penalizing Gradient Norm (PGN). The PGN attack aims to guide adversarial examples towards flatter regions by constraining the norm or magnitude of the gradient. The details of the PGN attack are outlined in Algorithm 1. Since PGN is derived by optimizing Eq. (3) to achieve a flat local maximum, it can be seamlessly integrated with existing gradient-based attack methods and input transformation-based attack methods, leveraging their strengths to further improve adversarial transferability.
(a) MI-FGSM
(b) NI-FGSM
(c) VMI-FGSM
Figure 2: Visualization of loss surfaces along two random directions for two randomly sampled adversarial examples on the surrogate model (i.e., Inc-v3). The center of each 2D graph corresponds to the adversarial example generated by different attack methods (see more examples in the Appendix).
(d) EMI-FGSM
(e) RAP
(f) PGN
In this section, we conduct extensive experiments on the ImageNet-compatible dataset. We first provide the experimental setup. Then we compare the results of the proposed methods with existing methods on both normally trained models and adversarially trained models. Finally, we conduct ablation studies to study the effectiveness of key parameters in our PGN. The experimental results were performed multiple times and averaged to ensure the experimental results were reliable.
Dataset. We conduct our experiments on the ImageNet-compatible dataset, which is widely used in previous works [4, 35, 42]. It contains 1,000 images with the size of  $299 \times 299 \times 3$ , ground-truth labels, and target labels for targeted attacks.
Models. To validate the effectiveness of our methods, we test attack performance in five popular pre-trained models, including Inception-v3 (Inc-v3) [45], Inception-v4 (Inc-v4), InceptionResnet-v2 (IncRes-v2) [46], ResNet-101 (Res-101), and ResNet-152 (Res-152) [18]. We also consider adversarially trained models including, Inc-v3\_ens3, Inc-v4\_ens4, and IncRes-v2\_ens [47].
Baselines. We take five popular gradient-based iterative adversarial attacks as our baselines, including, MI-FGSM [10], NI-FGSM [31], VMI-FGSM [49], EMI-FGSM [52], RAP [42]. We also integrate the proposed method with various input transformations to validate the generality of our PGN, such as DIM [11], TIM [61], SIM [31], Admix [51], and SSA [35].
Hyper-parameters. We set the maximum perturbation of the parameter  $\epsilon = 16.0 / 255$ , the number of iterations  $T = 10$ , and the step size  $\alpha = \epsilon / T$ . For MI-FGSM and NI-FGSM, we set the decay factor  $\mu = 1.0$ . For VMI-FGSM, we set the number of sampled examples  $N = 20$  and the upper bound of neighborhood size  $\beta = 1.5 \times \epsilon$ . For EMI-FGSM, we set the number of examples  $N = 11$ , the sampling interval bound  $\eta = 7$ , and adopt the linear sampling. For the attack method, RAP, we set the step size  $\alpha = 2.0 / 255$ , the number of iterations  $K = 400$ , the inner iteration number  $T = 10$ , the late-start  $K_{LS} = 100$ , the size of neighborhoods  $\epsilon_n = 16.0 / 255$ . For our proposed PGN, we set the number of examples  $N = 20$ , the balanced coefficient  $\delta = 0.5$ , and the upper bound of  $\zeta = 3.0 \times \epsilon$ .
4.2 Visualization of Loss Surfaces for Adversarial Example