SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To validate that our proposed PGN method can help the adversarial examples find a flat maxima region, we compare the loss surface maps of the adversarial examples generated by different attack methods on the surrogate model (i.e., Inc-v3). Each 2D graph corresponds to an adversarial example, with the adversarial example shown at the center. We randomly select two images from the dataset and compare the loss surfaces in Fig. 2, each row represents the visualization of one image. From the comparison, we observe that our PGN method can help adversarial examples achieve flatter maxima regions compared to the baselines. The adversarial examples generated by our method are located
Table 1: The untargeted attack success rates ( $\% \pm$ std, over 10 random runs) of various gradient-based attacks in the single model setting. The adversarial examples are crafted on Inc-v3, Inc-v4, IncRes-v2, and Res-101 by MI-FGSM (MI), NI-FGSM (NI), VMI-FGSM (VMI), EMI-FGSM (EMI), RAP, and our PGN attack methods, respectively. Here * indicates the white-box model.
Model	Attack	Inc-v3	Inc-v4	IncRes-v2	Res-101	Inc-v3ens3	Inc-v3ens4	IncRes-v2ens
Inc-v3	MI	100.0±0.0*	51.0±0.47	45.8±0.60	49.0±0.24	22.6±0.52	22.0±0.35	10.9±0.24
NI	100.0±0.0*	61.4±0.42	59.6±0.54	57.2±0.18	22.5±0.37	22.7±0.35	11.5±0.26
VMI	100.0±0.0*	74.8±0.58	69.9±0.92	65.5±0.67	41.6±0.54	41.6±0.54	25.0±0.34
EMI	100.0±0.0*	80.7±0.58	77.1±0.37	72.4±0.83	33.0±0.61	31.9±0.49	17.0±0.48
RAP	99.9±0.10*	84.5±0.69	79.3±0.47	76.5±0.65	56.9±0.84	51.3±0.62	31.9±0.35
PGN	100.0±0.0*	90.6±0.67	89.5±0.75	81.2±0.68	64.6±0.75	65.6±0.94	45.3±0.77
Inc-v4	MI	57.2±0.36	100.0±0.0*	46.1±0.14	51.5±0.33	19.1±0.46	18.4±0.23	10.2±0.36
NI	62.8±0.43	100.0±0.0*	52.7±0.34	56.7±0.19	19.2±0.25	18.3±0.37	11.7±0.29
VMI	77.6±0.65	99.8±0.10*	69.8±0.41	66.7±0.33	41.1±0.87	41.2±0.54	27.0±0.24
EMI	84.2±0.62	99.7±0.10*	75.0±0.70	74.4±0.64	31.5±0.44	28.0±0.65	16.2±0.36
RAP	85.3±0.74	99.5±0.21*	79.5±0.62	77.2±0.42	45.2±0.69	46.8±0.48	29.3±0.51
PGN	91.2±0.58	99.6±0.15*	87.6±0.74	83.5±0.53	67.0±0.68	64.2±0.63	49.1±0.82
IncRes-v2	MI	58.2±0.21	52.4±0.41	99.3±0.21*	50.7±0.26	22.0±0.37	22.0±0.31	13.8±0.43
NI	60.3±0.35	57.1±0.17	99.5±0.17*	55.3±0.35	18.3±0.18	19.3±0.29	12.1±0.16
VMI	78.2±0.64	77.0±0.57	99.1±0.36*	66.0±0.48	47.6±0.69	43.3±0.36	37.7±0.37
EMI	85.2±0.78	83.3±0.29	99.7±0.18*	74.0±0.56	38.4±0.48	33.8±0.53	24.1±0.48
RAP	87.1±0.75	84.2±0.45	99.4±0.28*	79.4±0.64	50.3±0.47	49.8±0.89	40.2±0.54
PGN	92.0±0.69	92.3±0.63	99.8±0.10*	83.5±0.41	74.6±0.75	71.5±0.64	66.62±0.58
Res-101	MI	51.5±0.26	42.2±0.35	36.3±0.24	100.0±0.0*	18.7±0.32	16.6±0.14	9.0±0.22
NI	55.6±0.35	46.9±0.41	40.8±0.28	100.0±0.0*	17.5±0.57	17.6±0.42	9.2±0.24
VMI	75.0±0.40	69.2±0.59	63.0±0.84	100.0±0.0*	35.9±0.41	35.7±0.87	24.1±0.57
EMI	74.3±0.65	71.7±0.47	62.6±0.29	100.0±0.0*	25.7±0.74	24.6±0.98	13.3±0.68
RAP	80.4±0.75	75.5±0.56	68.0±0.84	99.9±0.10*	40.3±0.47	39.9±0.73	30.4±1.03
PGN	86.2±0.84	83.3±0.66	77.8±0.69	100.0±0.0*	63.1±1.32	62.9±0.74	50.8±0.88
in larger and smoother flat local maxima. This confirms that our method can generate adversarial examples located in the flat maximum. With the adversarial examples located in more flat local maxima, PGN exhibits much better transferability than existing attacks as shown in Sec. 4.3- 4.5, which further supports our motivation.
4.3 Attack a Single Model
We conduct a series of gradient-based attacks under a single-model setting and report the attack success rates, which indicate the misclassification rates of the target models when using adversarial examples as inputs. The adversarial examples are generated on four different models: Inc-v3, Inc-v4, IncRes-v2, and Res-101, respectively. The results are summarized in Table 1.
From the results, we observe that our PGN method not only maintains a high attack success rate on white-box models but also significantly improves the attack success rate on black-box models. For example, when generating adversarial examples on Inc-v3, VMI-FGSM, EMI-FGSM, RAP achieve the attack success rates of  $74.8\%$ ,  $80.7\%$  and  $84.5\%$ , respectively on Inc-v4. In comparison, our PGN method achieves an impressive attack success rate of  $90.6\%$ , surpassing RAP (the best baseline) by a margin of  $6.1\%$ . Moreover, when targeting adversarily trained models, our PGN attack method consistently outperforms other gradient-based attacks, improving the attack success rate by at least  $11.8\%$  compared to state-of-the-art methods on average. This convincingly validates the high effectiveness of our proposed method against both normally trained and adversarily trained models. Such outstanding results highlight the effectiveness of locating adversarial examples in flat regions for improved transferability, which is consistent with our motivation.
4.4 Attack an Ensemble of Models
In addition to attacking a single model, we also evaluate the performance of our PGN method in an ensemble-model setting to further validate its effectiveness. In this subsection, we adopt the ensemble attack method in [10], which creates an ensemble by averaging the logit outputs of different models. Specifically, the adversaries are generated by integrating three normally trained models, including
Table 2: The untargeted attack success rates  $(\%)$  of various gradient-based attacks on eight models in the multi-model setting. The adversarial examples are generated on the ensemble models, i.e. Inc-v3, Inc-v4, and IncRes-v2. Here * indicates the white-box model.
Attack	Inc-v3	Inc-v4	IncRes-v2	Res-101	Res-152	Inc-v3ens3	Inc-v3ens4	IncRes-v2ens	Avg.
MI	99.8*	99.5*	97.8*	66.8	68.4	36.7	36.9	22.6	66.06
NI	100.0*	99.9*	99.6*	74.6	74.0	37.7	37.1	22.2	68.14
VMI	99.9*	99.5*	98.0*	82.2	81.6	66.8	64.1	50.8	80.36
EMI	100.0*	99.9*	99.7*	92.3	93.0	62.8	59.5	40.0	80.90
RAP	100.0*	99.4*	98.2*	93.5	93.4	70.1	69.8	58.9	85.41
PGN	100.0*	99.9*	99.6*	94.2	94.6	88.2	86.6	79.2	92.78
Table 3: The untargeted attack success rates  $(\%)$  of our PGN method, when it is integrated with DIM, TIM, SIM, Admix, and SSA, respectively. The adversarial examples are generated on Inc-v3. Here \* indicates the white-box model.
Attack	Inc-v3	Inc-v4	IncRes-v2	Res-101	Inc-v3ens3	Inc-v3ens4	IncRes-v2ens	Avg.
DIM	99.7*	72.2	67.3	62.8	32.8	30.7	16.4	54.56
PGN-DIM	100.0*	93.6	91.9	87.3	78.3	77.5	59.8	84.06
TIM	99.9*	51.6	47.2	47.8	29.6	30.7	20.5	46.76
PGN-TIM	100.0*	87.6	84.1	75.0	78.1	77.6	65.5	81.13
SIM	100.0*	70.5	68.2	63.8	37.5	37.8	22.0	57.11
PGN-SIM	100.0*	92.5	91.2	84.0	76.1	75.7	59.0	82.64
Admix	100.0*	78.6	77.3	69.5	41.6	40.3	24.1	61.63
PGN-Admix	100.0*	93.1	92.2	85.5	76.9	77.2	60.2	83.57
SSA	99.7*	88.3	86.8	77.7	56.7	55.3	35.2	71.39
PGN-SSA	99.8*	89.9	89.7	82.9	69.2	67.8	47.1	78.03
Inc-v3, Inc-v4, and IncRes-v2. All the ensemble models are assigned equal weights and we test the performance of transferability on both normally trained models and adversarially trained models.
The results, presented in Table 2, demonstrate that our PGN method consistently achieves the highest attack success rates in the black-box setting. Compared to previous gradient-based attack methods, PGN achieves an average success rate of  $92.78\%$ , outperforming VMI-FGSM, EMI-FGSM, and RAP by  $12.42\%$ ,  $11.88\%$ , and  $7.37\%$ , respectively. Notably, our method exhibits even greater improvements against adversarially trained models, surpassing the best attack method RAP by over  $18.4\%$  on average. These results validate that incorporating penalized gradient norms into the loss function effectively enhances the transferability of adversarial attacks, which also confirms the superiority of our proposed method in adversarial attacks.
4.5 Combined with Input Transformation Attacks
Existing input transformation-based attacks have shown great compatibility with each other. Similarly, due to the simple and efficient gradient update process, our proposed PGN method can also be combined with these input transformation-based methods to improve the transferability of adversarial examples. To further demonstrate the efficacy of the proposed PGN method, we integrate our method into these input transformations i.e., DIM, TIM, SIM, Admix, and SSA. We generate adversarial examples on the Inc-v3 model and test the transferability of adversarial examples on six black-box models.
The experimental results are shown in Table 3. When combined with our gradient update strategy, it can significantly improve the adversarial transferability of these input transformation-based attack methods in the black-box setting. At the same time, our method also has great improvement after combining these methods. For example, DIM only achieves an average success rate of  $54.56\%$  on the seven models, while when combined with our PGN method it can achieve an average rate of  $84.06\%$ , which is  $29.5\%$  higher than before. Especially, after combining these input transformation-based methods, our PGN tends to achieve much better results on the ensemble adversarially trained models, compared with the results in Table 1. It is a significant improvement and shows that our method
Table 4: Comparison of the approximation effect between directly optimizing the Hessian matrix  $(H_{m})$  and using the Finite Difference Method (FDM) to approximate. "Time" represents the total running time on 1,000 images, and "Memory" represents the computing memory size.
Attack	Hm	FDM	Inc-v3	Inc-v4	IncRes-v2	Res-101	Res-152	Time (s)	Memory (MiB)
I-FGSM	X	X	100.0*	27.8	19.1	38.1	35.2	52.31	1631
✓	X	100.0*	39.2	30.2	47.0	45.5	469.54	7887
✓	✓	100.0*	37.9	28.6	45.7	44.6	96.42	1631
has good scalability and can be combined with existing methods to further improve the success rate of transfer-based black-box attacks. In addition, our PGN method can also be combined with various gradient-based attack methods to enhance the transferability of previous works. The more experimental results are shown in the Appendix.
4.6 Ablation Study on Finite Difference Method
In this subsection, we will analyze and experimentally verify the effectiveness of the finite difference method in accelerating the approximation of the second-order Hessian matrix. We first theoretically analyze the acceleration effect of the finite difference method and substantiate our theoretical analysis with comparative experiments.
Theoretical analysis. For the baseline attack method, I-FGSM [26], the gradient is computed only once per iteration. Thus, its computational complexity is  $O(n)$ , where  $n$  represents the image size. However, when we introduce the penalty gradient term, the need arises to compute the second-order Hessian matrix, leading to a theoretical computational complexity of  $O(n^2)$ . To address this, we propose the finite difference method as an approximation to the Hessian matrix, which requires the computation of the gradient twice in each iteration, effectively yielding a computational complexity of  $O(2n)$ . This theoretically promises significant improvements in computational efficiency.
Experimental comparison. To validate the effectiveness of the finite difference method in accelerating computation, we compare the runtime and computational memory before and after using the finite-difference method. These experiments were conducted using codes executed on an RTX 2080 Ti with a CUDA environment. We employed I-FGSM and evaluated the total running time on 1,000 images (excluding data loading time) and the attack success rate on black-box models. The outcomes are presented in Table 4. Directly optimizing Eq. (4) results in better attack performance with high computational resources. With the finite difference method (FDM), we can better approximate the performance of direct optimization of the second-order Hessian matrix, which significantly reduces the running time and the computational memory. Furthermore, owing to the relatively modest image size and the comparatively small number of parameters compared to the model, the accelerated computing capabilities of CUDA enable the actual running time to be less than the theoretical estimates.
4.7 Ablation Study on Hyper-parameters
In this subsection, we conduct a series of ablation experiments to study the impact of different parameters, including the balanced coefficient  $\delta$  and the upper bound of  $\zeta$ -ball, and the study of the number of sampled examples  $N$  will be illustrated in the Appendix. To simplify our analysis, we only consider the transferability of adversarial examples crafted on the Inc-v3 model.
The balanced coefficient  $\delta$ . In Sec. 3.3, we introduce a balanced coefficient  $\delta$  to represent the penalty coefficient  $\lambda$  (see the Appendix for details). Compared to studying the parameter of  $\lambda$ , studying  $\delta$  will be much easier because it divides the scope of the parameter learning. As shown in Fig. 3a, we study the influence of the  $\delta$  in the black-box setting where  $\zeta$  is fixed to  $3.0 \times \epsilon$ . As we increase  $\delta$ , the transferability increases and achieves the peak for these black-box models when  $\delta = 0.5$ . This indicates that when averaging these two gradients the performance is best. Therefore, we set  $\delta = 0.5$  in our experiments.
The upper bound of  $\zeta$ -ball. In Fig. 3b, we study the influence of the upper bound neighborhood size for random sampling, determined by parameter  $\zeta$ , on the success rates in the black-box setting. In our experiments, we use uniform sampling, which reduces the bias due to uneven sample distribution. As
(a) The hyper-parameter  $\delta$
(b) The hyper-parameter  $\zeta$
Figure 3: The untargeted attack success rates  $(\%)$  on six black-box models with various hyperparameters  $\delta$  or  $\zeta$ . The adversarial examples are generated by PGN on Inc-v3.
we increase  $\zeta$ , the transferability increases and achieves the peak for normally trained models when  $\zeta = 3.0 \times \epsilon$ , but it is still increasing against the adversarially trained models. When  $\zeta > 4.5 \times \epsilon$ , the performance of adversarial transferability will decrease on seven black-box models. To achieve the trade-off for the transferability on normally trained models and adversarially trained models, we set  $\zeta = 3.0 \times \epsilon$  in our experiments.
Inspired by the observation that flat local minima often result in better generalization, we hypothesize and empirically validate that adversarial examples at a flat local region tend to have better adversarial transferability. Intuitively, we can optimize the perturbation with a gradient regularize in the neighborhood of the input sample to generate an adversarial example in a flat local region but it is non-trivial to solve such an objective function. To address such an issue, we propose a novel attack method called Penalizing Gradient Norm (PGN). Specifically, PGN approximates the Hessian/vector product by interpolating the first-order gradients of two samples. To better explore its neighborhood, PGN adopts the average gradient of several randomly sampled data points to update the adversarial perturbation. Extensive experiments on the ImageNet-compatible dataset demonstrate that PGN can generate adversarial examples at more flat local regions and achieve much better transferability than existing transfer-based attacks. Our PGN can be seamlessly integrated with other gradient-based and input transformation-based attacks to further improve adversarial transferability, demonstrating its versatility and ability to improve adversarial transferability across different attack scenarios.
Although we have experimentally verified that flat local minima can improve the transferability of adversarial attacks, there is still a lack of theoretical analysis regarding the relationship between flatness and transferability. An existing perspective is that transferability may be related to the generalization ability of flatness. We will also keep studying the theoretical connection between transferability and flat local minima in our future work. We hope our work sheds light on the potential of flat local maxima in generating transferable adversarial examples and provides valuable insights for further exploration in the field of adversarial attacks.
We want to thank the anonymous reviewers for their valuable suggestions and comments. This work was supported by the National Natural Science Foundation of China (Nos. 62276182, 62072334, and 61976164), and the National Science Basic Research Plan in Shaanxi Province of China (No. 2022GY-061).
A Proof of Corollary 1
In this work, we introduce a penalized gradient norm to the original loss function, which helps the adversarial examples to achieve a flat maximum. Then we randomly sample an example  $x'$  in the neighborhood of the adversarial example  $x^{adv}$  and simplify the objective function as follows:
$$
\max  _ {x ^ {a d v} \in \mathcal {B} _ {\epsilon} (x)} \mathcal {L} \left(x ^ {a d v}, y; \theta\right) \approx \left[ J \left(x ^ {\prime}, y; \theta\right) - \lambda \cdot \| \nabla_ {x ^ {\prime}} J \left(x ^ {\prime}, y; \theta\right) \| _ {2} \right], \quad \text {s . t .} \quad x ^ {\prime} \in \mathcal {B} _ {\zeta} \left(x ^ {a d v}\right). \tag {7}
$$
Gradient-based attacks require calculating the gradient of the objective function during practical optimization, thus the gradient of the current loss function (7) can be expressed as follows:
$$
\begin{array}{l} \nabla_ {x ^ {a d v}} \mathcal {L} \left(x ^ {a d v}, y; \theta\right) \approx \nabla_ {x ^ {\prime}} J \left(x ^ {\prime}, y; \theta\right) - \lambda \cdot \nabla_ {x ^ {\prime}} \left(\| \nabla_ {x ^ {\prime}} J \left(x ^ {\prime}, y; \theta\right) \| _ {2}\right) \\ \approx \nabla_ {x ^ {\prime}} J \left(x ^ {\prime}, y; \theta\right) - \lambda \cdot \nabla_ {x ^ {\prime}} ^ {2} J \left(x ^ {\prime}, y; \theta\right) \cdot \frac {\nabla_ {x ^ {\prime}} J \left(x ^ {\prime} , y ; \theta\right)}{\left\| \nabla_ {x ^ {\prime}} J \left(x ^ {\prime} , y ; \theta\right) \right\| _ {2}}. \tag {8} \\ \end{array}
$$
In practice, it is computationally expensive to directly optimize Eq. (8), since we need to calculate the Hessian matrix. In this work, we approximate the second-order Hessian matrix using the finite difference method to accelerate the attack process. Specifically, local Taylor expansion would be employed to approximate the operation results between the Hessian matrix and the gradient vector.
Proof. According to the Taylor expansion, we have
$$
\nabla_ {x} J (x + \Delta x, y; \theta) = \nabla_ {x} J (x, y; \theta) + \nabla_ {x} ^ {2} J (x, y; \theta) \Delta x + O (\| \Delta x \| ^ {2}), \tag {9}
$$
where  $\Delta x = \alpha \cdot v$ ,  $\alpha$  is a small step size, and  $v$  is a normalized gradient direction vector. Here, we denote  $v = -\frac{\nabla_x J(x,y;\theta)}{\|\nabla_x J(x,y;\theta)\|_2}$ .
Therefore, the Hessian/vector product can be approximated by the first-order gradient as follows:
$$
\nabla_ {x} ^ {2} J (x, y; \theta) v \approx \frac {\nabla_ {x} J (x + \alpha \cdot v , y ; \theta) - \nabla_ {x} J (x , y ; \theta)}{\alpha}. \tag {10}
$$
A.2 Proof of Corollary 1
Proof. From Eqs. (8) and (10), the gradient of the loss function  $\mathcal{L}(\cdot)$  can be expressed as:
$$
\begin{array}{l} \nabla_ {x ^ {a d v}} \mathcal {L} (x ^ {a d v}, y; \theta) \approx \nabla_ {x ^ {\prime}} J (x ^ {\prime}, y; \theta) - \lambda \cdot \nabla_ {x ^ {\prime}} ^ {2} J (x ^ {\prime}, y; \theta) \cdot \frac {\nabla_ {x ^ {\prime}} J (x ^ {\prime} , y ; \theta)}{\| \nabla_ {x ^ {\prime}} J (x ^ {\prime} , y ; \theta) \| _ {2}} \\ \approx \nabla_ {x ^ {\prime}} J \left(x ^ {\prime}, y; \theta\right) + \lambda \cdot \frac {\nabla_ {x ^ {\prime}} J \left(x ^ {\prime} + \alpha \cdot v , y ; \theta\right) - \nabla_ {x ^ {\prime}} J \left(x ^ {\prime} , y ; \theta\right)}{\alpha} \tag {11} \\ = \left(1 - \frac {\lambda}{\alpha}\right) \cdot \nabla_ {x ^ {\prime}} J \left(x ^ {\prime}, y; \theta\right) + \frac {\lambda}{\alpha} \cdot \nabla_ {x ^ {\prime}} J \left(x ^ {\prime} + \alpha \cdot v, y; \theta\right). \\ \end{array}
$$
We introduce a balanced coefficient  $\delta$  and denote it as  $\delta = \frac{\lambda}{\alpha}$ . Hence, the gradient of the objective function (7) at the  $t$ -th iteration can be approximated as:
$$
\nabla_ {x _ {t} ^ {a d v}} \mathcal {L} \left(x _ {t} ^ {a d v}, y; \theta\right) \approx (1 - \delta) \cdot \nabla_ {x _ {t} ^ {\prime}} J \left(x _ {t} ^ {\prime}, y; \theta\right) + \delta \cdot \nabla_ {x _ {t} ^ {\prime}} J \left(x _ {t} ^ {\prime} + \alpha \cdot v, y; \theta\right). \tag {12}
$$
B Visualization of Loss Surfaces
Implementation details. Given that the adversarial example  $x^{adv}$  typically has a large number of dimensions, visualizing the loss function against all dimensions becomes infeasible. To this end, we randomly select two directions, denoted as  $r_1$  and  $r_2$ , from a Gaussian distribution with the same dimension as  $x^{adv}$ . Next, we calculate the loss change by varying the magnitudes of
$k_{1}$  and  $k_{2}$ , representing the scaling factors applied to  $r_1$  and  $r_2$ , respectively, which enables us to visualize the loss function using a two-dimensional plot. This approach provides a slice of the loss function, allowing us to analyze its behavior and understand the impact of perturbations along different directions.
Visualization of loss surfaces for more adversarial examples. We visualize five randomly selected images in the ImageNet-compatible dataset. The adversarial examples are generated by various gradient-based attack methods on Inc-v3. These selected images include the three images that can be successfully transferred using our PGN method but cannot be transferred using other baseline methods. As shown in Fig. 4, we can observe that our method generates visually similar adversaries as other attacks. Hence, our method demonstrates the capability to guide adversarial examples towards larger and smoother flat regions. This observation substantiates the effectiveness of our PGN method in generating adversarial examples that reside within flat regions, thereby shedding light on the potential role of flat local maxima in generating transferable adversarial examples.
C Combined with Other Categories of Attacks
C.1 Gradient-based Attacks
Our PGN attack method can also be combined with various gradient-based attacks. The core of our method involves updating gradients by interpolating the first-order gradients from two samples to approximately minimize the gradient norm. In contrast, conventional gradient-based methods typically utilize a single example for gradient updates. To evaluate the efficacy of our strategy, we incorporate this interpolation approach into previous gradient-based methods, such as I-FGSM (BIM), MI-FGSM, NI-FGSM, VMI-FGSM, EMI-FGSM, and RAP. To simplify the experimental setup, we omitted random sampling and directly substituted the gradient update process of these methods with our proposed strategy.
The experimental results are presented in Table 5. Notably, when our gradient update strategy is integrated, there is a remarkable improvement in the adversarial transferability of the gradient-based attack methods in the black-box setting. For example, RAP alone achieves an average success rate of  $68.80\%$  across the seven models. However, when combined with our PGN method, the average success rate rises to  $73.91\%$ , exhibiting a significant improvement of  $5.11\%$ . This outcome underscores the robust scalability of our approach, as it seamlessly integrates with existing methodologies to further amplify the success rate of transfer-based attacks.