SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Baselines. We adopt three model-related methods as our baselines, i.e., SGM [53], LinBP [12] and Ghost [23], and evaluate their performance to boost adversarial transferability of iterative attacks (PGD [31]), momentum-based attacks (MI-FGSM [6], VMI-FGSM [43]), advanced objective functions (ILA [17]) and input transformation-based attacks (SSA [30]).
Hyper-parameters. We adopt the maximum magnitude of perturbation  $\epsilon = 8 / 255$  to align with existing works. We run the attacks in  $T = 10$  iterations with step size  $\alpha = 1.6 / 255$  for untargeted attacks and  $T = 300$  iterations with step size  $\alpha = 1 / 255$  for targeted attacks. We set the momentum decay factor  $\mu = 1.0$  and sample 20 examples for VMI-FGSM. The number of spectrum transformations and tuning factor is set to  $N = 20$  and  $\rho = 0.5$ , respectively. The decay factor for SGM is
Attacker	Method	Inc-v3	IncRes-v2	DenseNet	MobileNet	PNASNet	SENet	Inc-v3ens3	Inc-v3ens4	IncRes-v2ens
PGD	SGM	23.68	19.82	51.66	55.44	22.12	30.34	13.78	12.38	7.90
SGM+BPA	43.44	38.14	77.66	81.50	41.42	53.56	27.20	22.58	14.70
LinBP	27.22	23.04	59.34	59.74	22.68	33.72	16.24	13.58	7.88
LinBP+BPA	39.08	34.80	77.80	76.86	40.50	50.26	25.66	22.46	15.10
Ghost	17.74	13.68	42.36	41.06	13.92	19.10	11.60	10.34	6.04
Ghost+BPA	34.62	29.28	69.48	69.20	29.98	41.60	22.68	18.88	11.48
MI-FGSM	SGM	33.78	28.84	63.06	65.84	31.90	41.54	19.56	17.48	10.98
SGM+BPA	56.04	49.10	85.32	88.08	52.96	63.30	36.10	29.78	20.98
LinBP	35.92	29.82	68.66	69.72	30.24	41.68	19.98	16.58	9.94
LinBP+BPA	48.74	43.96	83.30	83.52	50.00	59.22	32.60	28.42	20.32
Ghost	29.76	23.68	57.28	56.10	25.00	34.76	17.10	14.76	9.50
Ghost+BPA	50.42	42.84	83.02	81.24	44.70	56.50	32.46	26.82	18.34
$\gamma = 0.5$  and the random range of Ghost network is  $\lambda = 0.22$ . We follow the setting of LinBP to modify the backward propagation of ReLU in the last eight residual blocks of ResNet-50. We set the temperature coefficient  $t = 10$  for ResNet-50 and  $t = 1$  for VGG-19.
4.2 Evaluation on Untargeted Attacks
To validate the effectiveness of our proposed method, we compare BPA with several other model-related methods (i.e., SGM, LinBP, Ghost) on ResNet-50 to boost various adversarial attacks, namely PGD, MI-FGSM, VMI-FGSM, ILA and SSA. Here we adopt ResNet-50 as the surrogate model since SGM is specific to ResNets. However, it is worth noting that BPA is general to various surrogate models with non-linear layers and we also report the results on VGG-19 in Appendix A.2. To further validate the effectiveness of BPA, we also consider more input transformation based attacks, different perturbation budgets and conduct evaluations on CIFAR-10 dataset [20] in Appendix A.3-A.5. We measure the attack success rates by evaluating the misclassification rates of the nine different target models on the generated adversarial examples.
**Evaluations on the single baseline.** We can observe from Table 1 that the model-related strategies can consistently boost performance of the five typical attacks on nine models. Among the baseline methods, LinBP generally achieves the best performance, except for VMI-FGSM where SGM surpasses LinBP. By addressing the issue of gradient truncation, BPA consistently improves the performance of all the five attack methods and achieves the best overall performance. On average, BPA outperforms the runner-up attack by a significant margin of  $7.84\%$ ,  $11.19\%$ ,  $5.08\%$ ,  $9.17\%$ ,  $2.25\%$ , respectively. These results highlight the effectiveness and generality of BPA in generating transferable adversarial examples compared with existing model-related strategies. The performance improvement achieved by BPA on SGM and LinBP, which also modify the backward propagation, validates our hypothesis that reducing the gradient truncation introduced by non-linear layers is beneficial for enhancing the adversarial transferability. This emphasizes the importance of carefully considering the backward propagation procedure when generating transferable adversarial examples.
Evaluations by combining BPA with the baselines. The primary objective of BPA is to mitigate the negative impact of gradient truncation on adversarial transferability, which is not considered by the baselines. Hence, it is expected that BPA can also boost the performance of these baselines. For validation, we integrate BPA with the baseline methods to enhance the performance of PGD and MI-FGSM attacks. The results of these combinations are presented in Table 2. We can observe that BPA can effectively
Table 2: Untargeted attack success rates (\%) of various baselines combined with our method using PGD and MI-FGSM. The adversarial examples are generated on ResNet-50.
Attacker	Method	HGD	R&P	NIPS-r3	JPEG	RS	NRP
PGD	N/A	9.34	5.00	6.00	11.04	8.50	11.96
SGM	16.80	7.50	9.44	13.96	10.50	12.76
LinBP	16.80	7.68	10.08	15.76	10.50	13.14
Ghost	9.60	5.06	6.42	11.92	9.50	12.06
BPA	23.96	12.02	15.60	22.52	14.00	14.08
MI-FGSM	N/A	16.64	8.04	9.92	16.68	13.00	13.32
SGM	24.80	11.02	13.16	20.26	14.00	14.38
LinBP	21.98	10.32	13.26	20.56	12.50	13.22
Ghost	17.98	8.88	10.64	18.52	13.50	13.84
BPA	34.30	17.84	22.04	30.86	17.50	15.96
Table 3: Untargeted attack success rates  $(\%)$  of several attacks on six defenses when generating the adversarial examples on ResNet-50 w/wo various model-related methods.
boost the adversarial transferability of various baselines. On average, BPA can boost the best baseline (i.e., LinBP) with a remarkable margin of  $13.23\%$  and  $20.94\%$  for PGD and MI-FGSM, highlighting
Attacker	Method	Inc-v3	IncRes-v2	DenseNet	MobileNet	PNASNet	SENet	Inc-v3ens3	Inc-v3ens4	IncRes-v2ens
PGD	N/A	0.54	0.80	4.48	2.04	1.62	2.26	0.18	0.08	0.02
SGM	2.56	3.12	15.08	8.68	5.78	9.84	0.62	0.18	0.04
LinBP	5.30	4.84	16.08	8.48	7.26	7.94	1.50	0.54	0.28
Ghost	1.34	2.14	10.24	4.74	3.90	6.64	0.36	0.16	0.10
BPA	8.76	9.74	23.76	13.42	14.66	13.76	2.52	1.02	0.72
MI-FGSM	N/A	0.16	0.26	2.06	0.90	0.42	1.22	0.00	0.02	0.02
SGM	0.74	0.76	5.84	3.24	1.66	3.70	0.00	0.02	0.00
LinBP	3.30	3.00	13.44	6.26	5.50	7.18	0.30	0.10	0.02
Ghost	0.66	0.76	5.48	2.14	1.58	3.38	0.08	0.02	0.00
BPA	5.68	7.30	23.34	12.16	12.50	14.56	0.60	0.12	0.06
Table 4: Targeted attack success rates (%) of various attackers on nine models when generating adversarial examples on ResNet-50 w/wo model-related methods using PGD and MI-FGSM.
the high effectiveness and superiority of BPA. Such high performance also validates its excellent generality to various architectures and supports our hypothesis about gradient truncation.
Evaluations on defense methods. To further evaluate the effectiveness of BPA, we also assess its performance on six defense methods using PGD and MI-FGSM, namely HGD [25], R&P [56], NIPS-r3 $^3$ , JPEG [11], RS [5] and NRP [32]. The results are presented in Table 3. We can observe that our BPA method successfully enhances both the PGD and MI-FGSM attacks, leading to higher attack performance against the defense methods. The results suggest that BPA can effectively enhance adversarial attacks against a range of defense techniques, reinforcing its potential as a powerful tool for generating transferable adversarial examples.
In summary, BPA exhibits superior transferability compared to various baseline methods when evaluated using a range of transfer-based attacks. It also exhibits good generality to further boost existing model-related approaches and achieves remarkable performance on several defense models, highlighting its effectiveness and versatility in generating highly transferable adversarial examples.
4.3 Evaluation on Targeted Attacks
To further evaluate the effectiveness of BPA, we also investigate its performance in boosting targeted attacks. Zhao et al. [60] identified that logit loss can yield better results than most resource-intensive attacks regarding targeted attacks. Here we adopt PGD and MI-FGSM to optimize the logit loss on ResNet-50 w/wo various model-related methods. The results are summarized in Table 4. Without the model-related methods, both PGD and MI-FGSM exhibit poor attack performance. However, when these methods are applied, the attack performance improves significantly. Notably, our BPA method achieves the best attack performance among all the baselines. This highlights the high effectiveness and excellent versatility of our proposed method in boosting targeted attacks and exhibits its potential to improve adversarial attacks in a wide range of scenarios. We also provide the results on VGG-19 in Appendix A.6.
To gain further insights into the effectiveness of BPA, we perform parameter studies on two crucial aspects: the position of the first ReLU layer to be modified and the temperature coefficient  $t$  for max-pooling. Additionally, we conduct ablation studies to investigate the impact of diminishing the gradient truncation of ReLU and max-pooling separately. We also provide more discussions about BPA in Appendix A.7-A.9.
On the position of the first ReLU layer to be modified. ReLU activation functions are densely applied in existing neural networks. For instance, there are totally 17 ReLU activation functions in ResNet-50. Intuitively, the truncation in the latter layers has a greater impact on gradient relevance compared to the earlier layers. As BPA aims to recover the truncated gradients by injecting imprecise gradients into the backward propagation, it is essential to focus on the more critical layers. To identify these important layers and evaluate their impact on transferability, we conduct the BPA attack using MI-FGSM by modifying the ReLU layers starting from the  $i$ -th layer, where  $1 \leq i \leq 17$ . As shown in Fig. 4a, modifying the last ReLU layer alone significantly improves the transferability of the attack, showing its high effectiveness. As we modify more ReLU layers, the transferability further
Figure 4: Hyper-parameter studies on the position of the first ReLu layer to be modified and the temperature coefficient  $t$  for the max-pooling layer.
Attacker	ReLU	Max-pooling	Inc-v3	IncRes-v2	DenseNet	MobileNet	PNASNet	SENet	Inc-v3ens3	Inc-v3ens4	IncRes-v2ens
PGD	X	X	16.34	13.38	36.86	36.12	13.46	17.40	10.24	9.46	5.52
✓	X	29.38	24.00	62.80	61.82	24.98	34.96	17.52	14.38	8.90
X	✓	20.26	16.16	44.66	42.82	17.12	21.52	13.20	11.88	7.74
✓	✓	35.36	30.12	70.70	68.90	32.52	42.02	22.72	19.28	12.40
MI-FGSM	X	X	26.20	21.50	51.50	49.68	22.92	30.12	16.22	14.58	9.00
✓	X	41.50	34.42	74.96	74.42	35.96	47.58	23.34	18.22	10.94
X	✓	34.16	29.02	61.38	59.42	32.24	37.32	21.74	19.96	14.70
✓	✓	47.58	41.22	80.54	79.40	44.70	54.28	32.06	25.98	17.46
Table 5: Untargeted attack success rates (\%) of PGD and MI-FGSM when generating adversarial examples on ResNet-50 w/wo modifying the backward propagation of ReLU or max-pooling.
improves and remains consistently high for most models. However, for a few models (e.g., PNASNet), modifying more ReLU layers leads to a slight decay on performance. To maintain a high level of performance across all nine models, we modify the ReLU layers starting from 3-0 ReLU layer, which is also adopted in LinBP [12].
On the temperature coefficient  $t$  for max-pooling. The temperature coefficient  $t$  plays a crucial role in determining the distribution of relative gradient magnitudes within each window. For example, when  $t = 0$ , the gradient distribution becomes a normalized uniform distribution. To find an appropriate temperature coefficient, we conduct the BPA attack using MI-FGSM with various temperatures. As shown in Fig. 4b, when  $t = 0$ , the attack exhibits the poorest performance but still outperforms the vanilla MI-FGSM. As we increase the value of  $t$ , the attack's performance consistently improves and reaches a high level of performance after  $t = 10$ . By selecting a suitable temperature coefficient, we ensure that the gradient distribution within each window is well-balanced and contributes effectively to the adversarial perturbation. Thus, we adopt  $t = 10$  in our experiments.
Ablation studies on ReLU and max-pooling. As stated in Sec. 3.1, we hypothesize that the gradient truncation caused by non-linear layers, such as ReLU and max-pooling in ResNet-50, has a detrimental effect on adversarial transferability. To further validate this hypothesis, we conduct ablation studies by comparing the performance of PGD and MI-FGSM attacks using the vallina backward propagation, the backward propagation modified by either ReLU or max-pooling, and both modifications combined. As shown in Table 5, adopting the modified backward propagation with either ReLU or max-pooling results in a significant improvement in adversarial transferability for both PGD and MI-FGSM attacks. Considering the presence of only one max-pooling layer in ResNet-50, the average performance improvement of  $4.07\%$  and  $7.58\%$  for PGD and MI-FGSM highlights the high effectiveness of BPA and underscores the efficacy of BPA in addressing the issue of gradient truncation. Furthermore, when both ReLU and max-pooling layers are modified in backward propagation, PGD and MI-FGSM exhibit the best performance. This finding supports the rational design of BPA and highlights the importance of mitigating gradient truncation in both ReLU and max-pooling layers to achieve optimal adversarial transferability.
In this work, we analyzed the backward propagation procedure and identified that non-linear layers (e.g., ReLU and max-pooling) introduce gradient truncation, which undermined the adversarial transferability. Based on this finding, we proposed a novel attack called Backward Propagation Attack (BPA) to mitigate the gradient truncation for more transferable adversarial examples. In particular, BPA addressed gradient truncation by introducing a non-monotonic function as the derivative of the ReLU activation function and incorporating softmax with temperature to calculate the derivative of max-pooling. These modifications helped to preserve the gradient information and prevented significant truncation during the backward propagation process. Empirical evaluations on ImageNet dataset demonstrated that BPA can significantly enhance existing untargeted and targeted attacks and outperformed the baselines by a remarkable margin. Our findings identified the vulnerability of model architectures and raised a new challenge in designing secure deep neural network architectures.
Our proposed BPA modifies backpropagation process for gradient calculation, making it only suitable for gradient-based attacks. Besides, BPA modifies the derivatives of non-linear layers, such as ReLU and max-pooling. Consequently, it may not be directly applicable to models lacking these specific components, such as transformers. In the future, we will investigate how to generalize our BPA to such transformers by refining the derivatives of some components, e.g., softmax. This endeavor to enhance the generality and versatility of BPA will be an essential aspect of ongoing research, paving the way for the broader applicability of the proposed method and facilitating its adoption in various deep learning models beyond those with ReLU and max-pooling layers.
This work is supported by National Natural Science Foundation (U22B2017, 62076105) and International Cooperation Foundation of Hubei Province, China (2021EHB011).
[1] Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models. In Proceedings of the International Conference on Learning Representations, 2018.
[2] Nicholas Carlini and David Wagner. Towards Evaluating the Robustness of Neural Networks. In 2017 IEEE symposium on security and privacy (sp), pages 39-57, 2017.
[3] Jianbo Chen, Michael I Jordan, and Martin J Wainwright. Hopskipjumpattack: A Query-efficient Decision-based Attack. In 2020 IEEE symposium on security and privacy (sp), pages 1277-1294, 2020.
[4] Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models. In Proceedings of the 10th ACM workshop on artificial intelligence and security, pages 15-26, 2017.
[5] Jeremy M. Cohen, Elan Rosenfeld, and J Zico Kolter. Certified Adversarial Robustness via Randomized Smoothing. International Conference on Machine Learning, pages 1310-1320, 2019.
[6] Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting Adversarial Attacks with Momentum. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9185-9193, 2018.
[7] Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Evading Defenses to Transferable Adversarial Examples by Translation-invariant Attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4312-4321, 2019.
[8] Zhijin Ge, Fanhua Shang, Hongying Liu, Yuanyuan Liu, Liang Wan, Wei Feng, and Xiaosen Wang. Improving the Transferability of Adversarial Examples with Arbitrary Style Transfer. In Proceedings of the ACM International Conference on Multimedia, 2023.
[9] Zhijin Ge, Fanhua Shang, Hongying Liu, Yuanyuan Liu, and Xiaosen Wang. Boosting Adversarial Transferability by Achieving Flat Local Maxima. In Proceedings of the Advances in Neural Information Processing Systems, 2023.
[10] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and Harnessing Adversarial Examples. In Proceedings of the International Conference on Learning Representations, 2015.
[57] Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan L Yuille. Improving Transferability of Adversarial Examples With Input Diversity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2730-2739, 2019.
[58] Yifeng Xiong, Jiadong Lin, Min Zhang, John E. Hopcroft, and Kun He. Stochastic Variance Reduced Ensemble Adversarial Attack for Boosting the Adversarial Transferability. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, pages 14963-14972, 2022.
[59] Jianping Zhang, Jen-tse Huang, Wenxuan Wang, Yichen Li, Weibin Wu, Xiaosen Wang, Yuxin Su, and Michael R Lyu. Improving the Transferability of Adversarial Samples by Path-Augmented Method. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8173-8182, 2023.
[60] Zhengyu Zhao, Zhuoran Liu, and Martha Larson. On Success and Simplicity: A Second Look at Transferable Targeted Attacks. Advances in Neural Information Processing Systems, 34:6115-6128, 2021.
In the appendix, we provide extra results about our proposed BPA. We first provide the raw data for Fig. 2. Considering that our main evaluations are conducted on ResNet-50 since the baseline SGM primarily focuses on the residual connection, we also conduct the evaluations on VGG-19 to further validate the effectiveness and generality of our proposed BPA. Then we provide the evaluations using more input transformation based attacks, and different perturbation budgets and also evaluate various methods on CIFAR-10 datasets. To further gain insights into the superiority of BPA, we also offer further discussions and explorations about the motivation and design.
A.1 Raw Data for Fig. 2
Due to the limited space, we only report the average attack success rates in Fig. 2. To validate its generality to various deep models, we also report the raw data in Tab. A1-A3. We can see that the attacker exhibits the same trends on various deep models, which is consistent with the results in Fig. 2.
Prob.	Inv-v3	InvRes-v2	DenseNet	MobileNet	PNASNet	SENet	Inc-v3ens3	Inc-v3ens4	IncRes-v2ens
0.00	26.38	21.38	51.26	49.62	22.64	30.22	15.94	14.44	8.68
0.10	26.14	21.62	51.30	49.78	22.12	29.32	15.92	14.24	8.88
0.20	25.72	21.56	50.72	49.02	22.18	28.90	15.82	13.82	8.38
0.30	25.62	21.24	50.94	48.74	21.84	29.12	15.52	13.82	8.54
0.40	24.96	20.78	50.02	48.32	21.48	28.64	15.04	13.92	8.46
0.50	24.94	20.16	49.52	47.66	20.64	27.92	14.42	13.22	8.28
0.60	24.94	19.70	48.28	47.20	20.34	27.92	14.28	12.84	8.14
0.70	23.18	18.78	46.96	46.08	18.92	26.42	13.96	12.72	7.74
0.80	22.84	17.18	44.52	44.38	17.52	24.92	13.28	11.72	7.00
0.90	20.22	15.78	39.82	41.16	14.80	20.84	11.22	10.54	6.34
1.00	0.02	1.10	0.00	0.02	0.02	0.02	3.76	4.30	1.28
Table A1: Original data for Fig. 2a
Prob.	Inv-v3	InvRes-v2	DenseNet	MobileNet	PNASNet	SENet	Inc-v3ens3	Inc-v3ens4	IncRes-v2ens
0.00	26.38	21.38	51.26	49.62	22.64	30.22	15.90	14.42	8.68
0.10	28.12	23.22	54.40	51.62	24.82	32.38	16.98	15.34	9.26
0.20	29.34	24.80	57.28	55.02	26.60	34.14	17.70	16.12	10.04
0.30	30.32	25.70	59.30	56.52	27.48	36.00	17.92	16.26	10.42
0.40	31.34	26.36	60.50	57.78	28.66	36.68	18.74	16.24	10.18
0.50	31.86	26.76	60.74	58.64	29.46	37.68	19.12	16.74	10.42
0.60	31.68	27.02	61.00	58.66	29.80	37.58	18.94	16.66	10.76
0.70	31.90	26.36	60.62	59.40	29.52	37.90	18.88	16.72	10.40
0.80	31.42	26.82	61.16	59.16	29.42	37.90	18.80	16.40	10.60
0.90	31.80	26.56	60.42	58.90	29.06	37.50	18.98	16.78	10.52
1.00	32.00	26.52	60.44	59.44	29.28	37.46	18.78	16.52	10.30
Table A2: Raw data for Fig. 2b
Prob.	Inv-v3	InvRes-v2	DenseNet	MobileNet	PNASNet	SENet	Inc-v3ens3	Inc-v3ens4	IncRes-v2ens
0.00	26.08	21.40	51.30	49.64	22.50	30.26	15.74	14.44	8.68
0.10	28.64	23.78	56.14	53.10	26.28	32.78	18.20	16.64	11.30
0.20	29.66	24.14	56.98	54.72	27.18	33.54	19.62	18.04	12.86
0.30	28.92	24.06	57.16	54.36	27.74	31.58	20.50	19.28	13.40
0.40	28.22	23.46	56.12	53.16	26.72	30.24	20.62	19.44	14.04
0.50	27.50	21.68	54.40	52.06	25.54	28.48	20.34	19.02	14.14
0.60	26.70	20.72	53.02	50.10	24.24	27.42	19.92	19.30	14.04
0.70	25.82	20.14	51.38	48.36	22.94	26.12	20.24	19.70	14.34
0.80	25.00	19.04	50.02	47.36	21.92	24.12	20.22	19.68	14.56
0.90	24.70	18.34	48.98	46.36	21.36	23.34	20.38	19.68	14.18
1.00	24.04	18.10	48.34	45.08	21.02	22.40	20.38	20.10	14.18
Table A3: Raw data for Fig. 2c
Attacker	Method	Inc-v3	IncRes-v2	DenseNet	MobileNet	PNASNet	SENet	Inc-v3ens3	Inc-v3ens4	IncRes-v2ens
PGD	N/A	12.52	9.70	25.82	32.20	13.18	13.82	7.64	7.60	4.14
LinBP	13.52	10.28	27.60	34.36	14.16	15.12	8.32	7.88	4.20
Ghost	13.18	9.72	25.78	32.50	12.80	13.68	8.12	7.90	4.48
BPA	26.24	27.06	47.98	58.22	34.08	31.42	15.52	14.06	8.78
MI-FGSM	N/A	19.74	15.32	37.02	43.42	21.16	23.02	11.46	10.08	5.96
LinBP	20.28	15.24	36.84	44.44	20.66	23.28	10.92	9.52	5.48
Ghost	19.88	15.34	36.44	43.20	21.84	24.06	11.54	10.30	6.00
BPA	36.88	29.98	61.10	68.58	45.98	43.06	21.44	17.68	11.94
VMI-FGSM	N/A	37.20	29.58	58.20	62.20	40.88	38.86	21.14	17.62	11.10
LinBP	36.18	28.86	55.40	62.46	38.38	39.14	19.20	17.18	10.92
Ghost	36.94	29.75	58.32	62.16	41.32	38.96	21.18	17.58	11.20
BPA	51.60	43.00	74.08	78.74	59.54	54.74	32.88	30.04	20.18
ILA	N/A	16.08	13.8	31.28	42.62	19.72	25.16	8.76	7.70	4.62
LinBP	17.08	14.54	32.74	44.40	20.16	27.08	8.44	7.92	4.54
Ghost	16.56	14.08	31.80	41.90	20.12	25.98	8.84	7.84	4.76
BPA	29.70	25.06	50.84	61.52	38.84	41.20	15.30	12.36	8.30
SSA	N/A	33.52	26.38	50.86	60.26	30.94	30.78	17.06	14.52	8.78
LinBP	35.70	28.08	53.76	63.52	32.32	34.18	18.64	16.10	9.36
Ghost	33.52	25.92	51.31	60.50	30.96	30.02	17.16	14.74	8.74
BPA	50.16	40.68	70.90	78.86	51.64	47.86	29.52	26.50	18.30
Table A4: Untargeted attack success rates (\%) of various adversarial attacks on nine models when generating the adversarial examples on VGG-19 w/wo various model-related methods.
Attacker	Method	Inc-v3	IncRes-v2	DenseNet	MobileNet	PNASNet	SENet	Inc-v3ens3	Inc-v3ens4	IncRes-v2ens
PGD	LinBP	13.52	10.28	27.60	34.36	14.16	15.12	8.32	7.88	4.20
LinBP+BPA	21.10	16.48	40.92	50.54	25.28	24.86	12.34	11.86	7.16
Ghost	13.18	9.72	25.78	32.50	12.80	13.68	8.12	7.90	4.48
Ghost+BPA	26.34	20.22	49.14	58.02	34.96	31.22	15.60	13.60	8.56
MI-FGSM	LinBP	20.28	15.24	36.84	44.44	20.66	23.28	10.92	9.52	5.48
LinBP+BPA	32.08	24.58	53.24	63.16	36.52	36.82	17.18	15.60	10.22
Ghost	19.88	15.34	36.44	43.20	21.84	24.06	11.54	10.30	6.00
Ghost+BPA	37.12	30.50	60.60	69.00	45.80	43.10	21.28	17.38	11.92
Table A5: Untargeted attack success rates (%) of various baselines combined with our method using PGD and MI-FGSM. The adversarial examples are generated on VGG-19.
A.2 Additional Evaluation on Untargeted Attacks
To validate the generality of BPA to various architectures, we further validate the effectiveness of our proposed BPA on VGG-19. Specifically, we first conduct untargeted attacks on VGG-19 following the setting in Sec. 4.2. Here we take LinBP and Ghost as our baselines.