SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Table presents the before and after-accuracy of each method along with the  $l_{2}$  distance of the adversarial patch and the non-normalised residual (NNR) between the adversarial and original image after conducting non-targeted attacks. We provide the mean and variance of each metric over 10 runs.
AT-WideResNet-50-2	AT-ResNet-50
Attack Method	Accuracy	l2	NNR	Accuracy	l2	NNR
-	68.46%	-	-	64.02%	-
CamoPatch	12.98% (0.01)†	0.14 (0.05)†	0.12 (0.07)†	6.00% (0.03)†	0.15 (0.03)†	0.13 (0.03)†
Patch-RS*	14.42% (0.01)‡	0.43 (0.07)‡	0.30 (0.05)‡	12.00% (0.02)‡	0.41 (0.12)‡	0.33 (0.05)‡
Patch-RS	14.42% (0.01)‡	0.74 (0.08)‡	0.42 (0.07)‡	12.00% (0.02)‡	0.74 (0.09)‡	0.43 (0.07)‡
TPA	51.66% (1.3)‡	0.82 (1.21)‡	0.82 (0.07)‡	34.82% (1.41)‡	0.92 (0.05)‡	0.87(0.09)‡
OPA	36.88% (0.1)‡	0.76 (0.20)‡	0.74 (0.05)‡	24.83% (1.12)‡	0.77 (0.14)‡	0.75 (0.04)‡
LOAP	38.85% (0.4)‡	0.56 (0.02)‡	0.46 (0.03)‡	48.89% (0.1)‡	0.72 (0.18)‡	0.64 (0.03)‡
Adv-watermark	52.00% (0.3)‡	0.37(0.05)‡	0.23(0.07)‡	44.00% (0.3)‡	0.42 (0.02)‡	0.29 (0.07)‡
ViT-B/16	BagNet9 with PatchGuard
Attack Method	Accuracy	l2	NNR	Accuracy	l2	NNR1
-	77.91%	-	-	55.1%	-
CamoPatch	8.00% (0.05)†	0.09 (0.02)	0.12 (0.02)	3.20% (0.01)†	0.07(0.03)‡	0.11 (0.01)†
Patch-RS*	19.00% (0.10)‡	0.68 (0.05)†	0.39 (0.07)‡	5.80% (0.02)‡	0.42 (0.05)‡	0.30 (0.05)†
Patch-RS	19.00% (0.10)‡	0.71 (0.12)†	0.41 (0.09)‡	5.80% (0.02)‡	0.62 (0.18)‡	0.57 (0.11)†
TPA	38.12% (0.91)‡	0.59 (0.08)‡	0.54 (0.09)‡	32.87% (1.45)‡	0.62 (0.11)‡	0.61(0.09)‡
OPA	33.09% (0.17)‡	0.68 (0.23)‡	0.68 (0.07)‡	57.89% (2.01)‡	0.61 (0.16)‡	0.67 (0.04)‡
LOAP	43.91% (0.80)‡	0.63 (0.05)‡	0.50 (0.13)‡	72.82% (0.14)‡	0.89 (0.23)‡	0.78 (0.11)‡
Adv-watermark	36.01% (0.12)‡	0.17(0.04)‡	0.28(0.03)‡	42.00% (0.45)‡	0.14(0.01)‡	0.29(0.05)‡
VGG-16	ResNet-50
Attack Method	Accuracy	l2	NNR	Accuracy	l2	NNR
-	73.36%	-	-	76.12%	-
CamoPatch	9.70% (0.03)	0.09 (0.02)†	0.11 (0.02)†	10.00% (0.02)†	0.08 (0.01)†	0.10 (0.01)†
Patch-RS*	6.82% (0.04)	0.42 (0.02)‡	0.30 (0.05)‡	15.92% (0.02)‡	0.45 (0.04)‡	0.31 (0.04)‡
Patch-RS	6.82% (0.04)	0.63 (0.01)‡	0.61 (0.07)‡	15.92% (0.02)‡	0.67 (0.08)‡	0.69 (0.07)‡
TPA	47.11% (1.30)‡	0.61 (0.13)‡	0.55 (0.05)‡	38.98% (1.41)‡	0.61 (0.07)‡	0.58(0.07)‡
OPA	32.19% (0.10)‡	0.71 (0.20)‡	0.64 (0.06)‡	27.91% (1.12)‡	0.71 (0.14)‡	0.66 (0.04)‡
LOAP	37.99% (0.40)‡	0.68 (0.02)‡	0.63 (0.05)‡	47.99% (0.10)‡	0.78 (0.12)‡	0.67 (0.05)‡
Adv-watermark	32.00% (0.10)‡	0.13(0.08)‡	0.25(0.05)‡	35.00% (0.40)‡	0.16(0.01)‡	0.31(0.07)‡
$\dagger$  denotes the performance of the method significantly outperforms the compared methods according to the Wilcoxon signed-rank test [60] at the 5% significance level;  $\ddagger$  denotes the corresponding method is significantly outperformed by the best performing method (shaded).
$t = 300$ ,  $lit = 4$ , and  $N = 100$ . We provide an empirical justification for these specific settings in Section 4.3.
Performance Metrics: We evaluate the performance of all considered algorithms by allowing each method to exhaust the allocated query budget while attacking each classifier. To evaluate the effectiveness of an attack we report the accuracy of the classifier on the generated adversarial images. For the successful adversarial images, we report two additional metrics: (1) the  $l_{2}$  distance between the adversarial patch and the corresponding area of the original image, and (2) the non-normalised residual (NNR) between the adversarial and original image, which measures the absolute difference between the pixel values of the constructed patch and the area of the original image it covers.
Given the stochastic nature of our proposed method and the comparison methods, we follow the setup of [15] and report the mean and variance of each metric over 10 independent runs with different random seeds. We additionally utilize the Wilcoxon signed-rank test [60] at a  $5\%$  significance level to statistically verify whether the improvements by our method over the compared algorithms across the 10 runs are significant.
4.2 Comparison
For the adaptation of LOAP [47], we replace its gradient computation method with the estimation method of [26]. The detailed description of this estimation method can be found in Algorithm 2 in the supplementary document. Additionally, we compare our method with an adapted version of Patch-RS, where we minimize the  $l_{2}$  distance of the constructed patch in a similar manner to our proposed method. For each compared algorithm, we utilize the authors' original implementation and recommended settings. In our black-box adaptation of LOAP [47], we set the number of iterations  $n = 50$  and variance  $\eta = 0.001$  for the gradient estimation method of Ilyas et al..
Figure 3: Adversarial images generated by methods conducting non-targeted attacks on the conventionally trained VGG-16 (top) and adversarial trained WideResNet-50-2 [49] (bottom) classifiers. Whereas adversarial patches generated by state-of-the-art methods are visibly clear, the patches generated by the proposed method are well camouflaged within the image.
Results: Table 1 present the statistical results of non-targeted attacks conducted on the trained ImageNet classifiers. In the tables, "CamoPatch" denotes our proposed method, and "Patch-RS*" refers to the adapted Patch-RS algorithm.
These results demonstrate that the Patch-RS attack, along with our own method, achieves higher attack success rates compared to the other state-of-the-art methods. This result aligns with previous work [15], which demonstrated the superior performance of the Patch-RS algorithm. Despite Patch-RS outperforming our method when attacking the VGG-16 classifier, according to the Wilcoxon signed-rank test, there is no significant difference between the performance of both methods. Alternatively, when attacking the remaining five classifiers, the proposed method is able to significantly outperform Patch-RS and other compared methods according to the Wilcoxon signed-rank test.
Comparing the  $l_{2}$  distance and NNR of adversarial patches generated by the attack methods, the proposed method is able to construct adversarial patches that are far less invasive to the input image. This is supported by the proposed method significantly outperforming all other methods in terms of both  $l_{2}$  distance and NNR, according to the Wilcoxon signed-rank test. This result highlights that the effectiveness of our adversarial patches is not compromised by their perceptibility.
Despite the adapted Patch-RS* algorithm being able to generate patches with lower  $l_{2}$  distances from the original image compared to its original implementation, its use of Square-Attack [2] for patch pattern optimization results in the patch values taking the corners of the color cube [0, 1]. Therefore, its ability for  $l_{2}$  minimization is significantly hampered. Alternatively, the proposed method is able to construct patches with any color, which allows for effective approximations of the original image area. Figure 3 provides a visual comparison of adversarial images generated by each method when attacking the VGG-16 and AT-ResNet-50 classifiers.
We report the results of the targeted attacks in Section 6.1 of the supplementary material.
Robustness Evaluation: Despite the assumption that adversarial trained classifiers have improved robustness compared to their conventionally trained counterparts, our experimental results reveal a different picture. The proposed method achieves higher success rates when attacking the adversarial trained classifier AT-ResNet-50 of Salman et al. compared to the conventionally trained VGG-16 and ResNet-50 classifiers.
However, the results also demonstrate that the adversarial patches generated by the proposed method, when attacking the AT-ResNet-50 classifier, exhibit larger  $l_{2}$  distances from the original image, resulting in larger non-normalised residuals between the entire adversarial image and the original image. This suggests that while the adversarial trained classifier is more susceptible to adversarial patches, these patches require larger distortions to cause the misclassification. On the other hand, conventionally trained classifiers are more susceptible to smaller changes in the original image. This behavior can be attributed to the general procedure of adversarial training, which introduce noise onto images during the training to enhance robustness. Consequently, patches with larger impact decrease the likelihood of the image being representative of the training data, as has been observed in other works [50]. Furthermore, wee see the AT-WideResNet-50-2 classifier exhibits greater robustness to
Table 2: Table presents the before and after-accuracy of each CamoPatch configuration along with the  $l_{2}$  distance of the adversarial patch and the non-normalised residual (NNR) between the adversarial and original image after conducting non-targeted attacks. We provide the mean and variance of each metric over 10 runs.
VGG-16
li	t	N	σ	Accuracy	l2	NNR	Runtime(s)
-	-	-	-	76.12%	-	-	-
1	300	100	0.1	12.88%(1.0)	0.09(0.04)	0.14(0.01)	440.03(10.32)
4	100	100	0.1	10.79%(1.5)	0.09(0.07)	0.13(0.02)	440.01(10.05)
4	300	100	0.1	9.64%(1.0)	0.09(0.05)	0.11(0.02)	440.03(10.32)
4	300	100	0.3	11.66%(2.0)	0.09(0.06)	0.11(0.05)	439.13(10.56)
$^\dagger$  denotes the performance of the method significantly outperforms the compared methods according to the Wilcoxon signed-rank test [60] at the  $5\%$  significance level;  $^\ddagger$  denotes the corresponding method is significantly outperformed by the best performing method (shaded).
our method. Since both AT-WideResNet-50-2 and AT-ResNet-50 models are trained using the same process, these results suggest that the WideResNet architecture is inherently more robust.
The proposed method consists of four tunable parameters:  $li$ ,  $N$ ,  $\sigma$ , and  $t$ . To determine their optimal values, we conduct a grid search over the parameter space. Specifically, we explore  $li \in \{1,4\}$ ,  $N \in \{100,300\}$ ,  $t \in \{100,300\}$ , and  $\sigma \in \{0.1,0.3\}$ . The choice of  $li$  follows the recommendation of Croce et al., while the values of  $\sigma$ ,  $t$ , and  $N$  are commonly used in the evolutionary[52] and computational art [57] communities, respectively.
To evaluate the performance of each parameter configuration, we conduct non-targeted attacks on the VGG-16 ImageNet classifier using 1000 correctly classified images from the validation set. We measure the accuracy of the model on the generated adversarial images,  $l_{2}$  distance and NNR for each configuration over 10 independent runs with different random seeds. Additionally, we compare the computational time required for each configuration to complete an attack on a single image.
Configurations: Table 2 presents the four top performing configurations in terms of the attack accuracy. The results demonstrate that the performance of the proposed method heavily depends on the number of circles  $N$  used to construct the patch pattern. Increasing the number of circles allows for better detailed approximations but also introduces additional complexity. From the results in Table 2, we observe that the best performing configurations all use  $N = 100$ , suggesting that the patch optimizer,  $(1 + 1)$ -ES, struggles with larger numbers of circles. Moreover, we observe longer runtimes for  $N = 300$  due to the increased number of properties that need adjustment. Beyond the number of circles  $N$ , the proposed method achieves improved performance with a larger budget for patch pattern optimization  $(li = 4)$  and a larger exploration parameter for location optimization  $(t = 300)$ . Based on these findings, we set the optimal parameter configuration of the proposed method to  $li = 4$ ,  $t = 300$ ,  $N = 100$ , and  $\sigma = 0.1$ .
Simulated Annealing: To justify the use of simulated annealing for location optimization within the proposed method, we compare its performance with and without simulated annealing. Removing simulated annealing results in a pure random search method similar to existing works. We keep the other parameters of the proposed method constant with those outlined in Section 4.1. The results in Table 3 demonstrate the improved performance exhibited by the proposed method when the simulated annealing policy is employed for location optimization, particularly when attacking adversarial trained classifiers. Despite generating patches with a higher  $l_{2}$  distance, the proposed method with simulated annealing achieves higher success rates. This suggests that more challenging images require larger distortions to cause misclassification, increasing the average  $l_{2}$  distance of the generated successful adversarial patches.
5 Contributions, Limitations and Future Work
Contributions: In this work, we propose CamoPatch, a novel attack method for generating adversarial patches that can blend into the targeted image. We achieve this by constructing the patch
Table 3: Table presents the before and after-accuracy of the CamoPatch method with (CamoPatch) and without (CamoPatch*) the simulated annealing policy for location optimization, along with the  $l_{2}$  distance of the adversarial patch and the non-normalised residual (NNR) between the adversarial and original image after conducting non-targeted attacks. We provide the mean and variance of each metric over 10 runs.
CamoPatch	CamoPatch*
Classifier	ASR	l2	NNR	ASR	l2	NNR
VGG-16	90.30% (0.03)	0.09 (0.02)	0.11 (0.02)	90.01 (0.1)	0.09 (0.01)	0.11 (0.02)
ResNet-50	90.00% (0.02)	0.08 (0.01)	0.08 (0.01)	90.00% (0.01)	0.09 (0.01)	0.1 (0.02)
AT-WideResNet-50-2	87.02% (0.01)†	0.14 (0.05)‡	0.12 (0.07)	83.02% (0.02)‡	0.11 (0.05)†	0.09 (0.02)
AT-ResNet-50	94.00% (0.03)†	0.15 (0.03)	0.13 (0.03)	90.00% (0.01)	0.14 (0.03)	0.13 (0.05)
$^\dagger$  denotes the performance of the method significantly outperforms the compared methods according to the Wilcoxon signed-rank test [60] at the  $5\%$  significance level;  $^\ddagger$  denotes the corresponding method is significantly outperformed by the best performing method (shaded).
pattern using a combination of semi-transparent, RGB-valued circles, which are optimized to cause misclassification and approximate the covered area of the original image. By incorporating a simulated annealing policy for location optimization, our method generates adversarial patches with improved or comparable success rates, while minimizing the visual impact on the target image.
Ethical Considerations: Adversarial patches have gained attention due to their potential real-world applications, where attackers can print and physically place them to deceive real-world implemented DNNs [7, 18]. However, existing methods often generate patches that are visually clear and easily detectable to a human observer. Our work introduces the concept of camouflaged adversarial patches, which are difficult for both humans and computer vision systems to perceive. This raises further concerns about the robustness of DNN classifiers in safety-critical applications. Adversarial training has proven to be an effective method of improving the robustness of DNN classifiers to adversarial patches [47]. Incorporating images with camouflaged adversarial patches into the training process of DNNs may be a promising avenue to enhance their robustness and mitigate the vulnerabilities demonstrated in this work.
Limitations and Future Work: It is important to acknowledge the limitations of the proposed method and identify potential areas for future research. One limitation is that our method assumes the attacker has access to the output probabilities of the targeted DNN, which may not always be the case in real-world scenarios. Future work could explore adapting the proposed method to scenarios where only the predicted label of the input is available, by utilizing estimated loss functions such as the one proposed by Ilyas et al.. Furthermore, techniques from weakly supervised learning can be incorporated into the proposed method to address the label-only setting. Specifically, by using estimation techniques [26] to score constructed adversarial images, the use of weakly supervised image classification models [25] as surrogates could improve the efficiency of the proposed method in addition to providing a direction of the search for the label-only setting. Alternatively, utilizing the stochastic nature of the proposed method to generate a set of non-evaluated candidate solutions, the use of the weakly supervised learning techniques such as semi-supervised learning could be applied [30] to train a surrogate model on both evaluated and non-evaluated adversarial images. Thereby using the surrogate to select predicted-optimal solutions for evaluation.
Another limitation is the DNN query budget assumption in our experiments. In practice, the available query budget might be significantly lower. To address this limitation, future research could extend the proposed method to incorporate surrogates or approximation models that guide the attack process, using fewer DNN queries, similar to that of Williams et al. within the computational art field.
Lastly, the parameter tuning process in our work follows a conventional grid-search approach, which limits the exploration of parameter combinations. Bayesian optimization methods could be employed to automate the configuration of attack algorithms, leveraging Gaussian Process surrogate models to handle the stochastic nature of the proposed method and guide the parameter search [34, 10, 9, 37]. This would provide a more efficient approach for determining the optimal parameter configuration.
This work was supported in part by the UKRI Future Leaders Fellowship under Grant MR/S017062/1 and MR/X011135/1; in part by NSFC under Grant 62376056 and 62076056; in part by the Royal Society under Grant IES/R2/212077; in part by the EPSRC under Grant 2404317; in part by the Kan Tong Po Fellowship (KTP/R1/231017); and in part by the Amazon Research Award and Alan Turing Fellowship.
2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, 2019.
[59] Jonathan Uesato, Brendan O'Donoghue, Pushmeet Kohli, and Aïron van den Oord. Adversarial risk and the dangers of evaluating against weak attacks. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, July 10-15, 2018, Proceedings of Machine Learning Research, 2018.
[60] Frank Wilcoxon. Individual comparisons by ranking methods. In *Breakthroughs in statistics*. Springer, 1992.
[61] Phoenix Neale Williams and Ke Li. Art-attack: Black-box adversarial attack via evolutionary art. CoRR, abs/2203.04405, 2022.
[62] Phoenix Neale Williams and Ke Li. Black-box sparse adversarial attack via multi-objective optimisation CVPR proceedings. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2023.
[63] Phoenix Neale Williams, Ke Li, and Geyong Min. Black-box adversarial attack via overlapped shapes. In GECCO '22: Genetic and Evolutionary Computation Conference, 2022.
[64] Phoenix Neale Williams, Ke Li, and Geyong Min. A surrogate assisted evolutionary strategy for image approximation by density-ratio estimation. In IEEE Congress on Evolutionary Computation, CEC, 2023.
[65] Chong Xiang, Arjun Nitin Bhagoji, Vikash Sehwag, and Prateek Mittal. Patchguard: A provably robust defense against adversarial patches via small receptive fields and masking. In 30th USENIX Security Symposium, USENIX Security 2021, August 11-13, 2021, 2021.
[66] Chenglin Yang, Adam Kortylewski, Cihang Xie, Yinzhi Cao, and Alan L. Yuille. Patchattack: A black-box texture-based attack with reinforcement learning. In Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVI, 2020.
[67] Shasha Zhou, Ke Li, and Geyong Min. Attention-based genetic algorithm for adversarial attack in natural language processing. In Parallel Problem Solving from Nature - PPSN.
[68] Shasha Zhou, Ke Li, and Geyong Min. Adversarial example generation via genetic algorithm: a preliminary result. In GECCO '22: Genetic and Evolutionary Computation Conference, 2022.
[69] Alon Zolfi, Moshe Kravchik, Yuval Elovici, and Asaf Shabtai. The translucent patch: A physical and universal attack on object detectors. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, 2021.