CamoPatch: An Evolutionary Strategy for Generating Camoflauged Adversarial Patches
Authors: Phoenix Neale Williams
Ke Li
Venue: NeurIPS 2023 poster (2023)
Keywords: Evolutionary Strategy
Adversarial Attack
Adversarial Patches
Computational Art
Computer Vision
TL;DR: Our approach constructs adversarial patches that approximate the targeted area of the image,  achieving comparable performance to state-of-the-art methods while minimizing distortion.

Abstract: Deep neural networks (DNNs) have demonstrated vulnerabilities to adversarial examples, which raises concerns about their reliability in safety-critical applications. While the majority of existing methods generate adversarial examples by making small modifications to the entire image, recent research has proposed a practical alternative known as adversarial patches. Adversarial patches have shown to be highly effective in causing DNNs to misclassify by distorting a localized area (patch) of the image. However, existing methods often produce clearly visible distortions since they do not consider the visibility of the patch. To address this, we propose a novel method for constructing adversarial patches that approximates the appearance of the area it covers. We achieve this by using a set of semi-transparent, RGB-valued circles, drawing inspiration from the computational art community. We utilize an evolutionary strategy to optimize the properties of each shape, and employ a simulated annealing approach to optimize the patch's location. Our approach achieves better or comparable performance to state-of-the-art methods on ImageNet DNN classifiers while achieving a lower $l_2$ distance from the original image. By minimizing the visibility of the patch, this work further highlights the vulnerabilities of DNNs to adversarial patches.

CamoPatch: An Evolutionary Strategy for Generating Camouflaged Adversarial Patches
Phoenix Neale Williams
Ke Li Department of Computer Science University of Exeter Exeter, EX4 4RN k. seli@exeter.oom ac uk
Department of Computer Science
University of Exeter Exeter, EX4 4RN
pu384@exeter.. uk
Abstract
Deep neural networks (DNNs) have demonstrated vulnerabilities adversarial examples, which raises concerns about their reliability in safety-critical applications. While the majority of existing methods generate adversarial examples by making small modifications to the entire image, recent research has proposed a practical alternative known adversarial patches. Adversarial patches have shown be highly effective in causing DNNs to misclassify by distorting localized area (patch) of the image. However, existing methods often produce clearly visible distortions since they do not consider the visibility of the patch. To address this, we propose novel method for constructing adversarial patches that approximates the appearance of the area it covers. We achieve this by using set of semi-transparent, RGB-valued circles, drawing inspiration from the computational art community. We utilize evolutionary strategy to optimize the properties of each shape, and employ a simulated annealing approach to optimize the patch's location. Our approach achieves better comparable performance to state-of-the-art methods on ImageNet DNN classifiers while achieving a lower l2 distance from the original image. By minimizing the visibility of the patch, this work further highlights the vulnerabilities of DNNs to adversarial patches.
Introduction
Deep neural networks (DNNs) have revolutionized the field computer vision, demonstrating significant progress in several tasks [8 51 53) Nevertheless, they are not without vulnerabilities. Recent studies highlight a critical weakness: susceptibility to adversarial examples, where subtle, intentionally designed perturbations to input images result in the DNNs misclassification [22 54 43)
The existence of these adversarial examples in the physical world poses a significant threat to
security-critical applications such autonomous vehicles and medical imaging [3] [56 33 5 As
result, developing methods to generate adversarial images has emerged as a critical research area for assessing the robustness of DNNs (
While initial studies emphasized the creation of adversarial examples with no-mo (p can be 1, 2, or oo) constrained perturbations. current research has shifted towards generating sparse perturbations, which alter only a small portion the original image 12 have proven to be effective traditional la -onstraind perturbations. Adversarial patches, localized perturbations affecting a small portion of the image, have garnered significant interest as a type of sparse perturbation [47 66 33 Various methods for generating adversarial patches have been proposed for both white-box (where complete information about the model is known) and black- box (where only the input-output pairs are accessible) scenarios
19, 15 61]. These sparse perturbations
29
66 20
27
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Proposed method
Patch-RS
12 0.014
0.01
1/ 0.78
1/ 0.69
Figure This illustration shows adversarial images generated by two algorithms, the proposed method and Patch-RS [15], both attacking a conventionally trained (left) and adversarialyy trained (right) ImageNet classifiers. While both images are adversarial, the adversarial patch generated by the state-of-the-ar Patch-RS algorithm visibly distorts the image. whereas the proposed method's s's adversarial patch remains more similar to the original image, This similarity is demonstrated by calculating the le distance between the adversarial patches and the area of the original image they are placed upon.
However, a significant challenge remains: the unbounded magnitudes of adversarial patches often lead to noticeable distortions in the original image, as depicted in Figure
contrasting approach to adversarial attacks embodied by minimum-norm attacks, class of strategies that generate adversarial examples through minimizing particular norm of the adversarial perturbation [40 13 58). Due to their ability to measure adversarial accuracy under a range of perturbation budgets, these attacks serve as valuable tools for assessing DNN robustness [45]. However, these methods come with a notable drawback: they rely heavily on numerous DNN queries to substantially decrease the perturbation size. This dependence becomes particularly problematic in black-box scenarios, where the number of queries is often restricted, thus making it more difficult to reduce the perturbation size effectively (26].
Although adversarial patches have proven effective in causing DNN misclassification, existing methods often overlook the necessity of minimizing the visual impact of the patch on the original image. This oversight leads to patches that are easily detectable. To address this shorteoming, we introduce a novel method for generating adversarial patches. Our approach utilizes semi-transparent circles with RGB values, which blend into the original image. We employ simulated annealing for optimal location selection and implement an evolutionary strategy to fine-tune each circle's 's properties. The goal of our method is to induce the desired misclassification while minimizing the l2 distance between the patch and the original image, thereby camouflaging the patch effectively.
The rest of this paper is organized as follows. Section 2 overviews some related works, underscoring their contributions and limitations. In Section 3 we outline our proposed attack scenario and provide an in-depth explanation of our method's implementation. Empirical results are presented and analyzed in Section 4 Section concludes this paper and sheds some light on future directions.
2 Related Works
Adversarial attacks on DNNs have been one of most active fields in the machine learning commu- nity. A common strategy in these attacks is to create imperceptible perturbations, often achieved by constraining the perturbations using the no-norm. These perturbations are typically generated differently depending on the access level to the targeted DNN's information. In white-box scenario, where the attacker has full access to the DNN's details, approaches often leverage the gradient information of the DNN's loss function. This gradient is used within an optimization algorithm to generate adversarial perturbations [54, 22 39. On the other hand, in a black-box scenario, where the attacker's 's access is limited to the DNN's s output probabilities, various attack methods estimate the loss function's 's gradient and use it within a gradient-based optimization technique [26 4 59
Some researchers have also proposed heuristic methods for the black-bor scenario that do not rely on gradient information
2 27 28

---

Our work fits into this landscape by focusing on the black-box scenario. However, unlike many existing methods, we aim to create adversarial patches that are not only effective but also visually blend into the original image.
2.1 Adversarial Patches
Adversarial patches, designed to cause image misclassification, represent unique approach to adversarial attacks. These patches are typically small, visible, and often square-shaped, strategically applied to the targeted image [13] 47 [66. The pioneering work by BBronn al. introduced the concept of universal adversarial patches, which cause misclassification when applied to a variety of images. Using the gradient information of the DNN, they utilized stochastic gradient descent to optimize the patch pattern, which was subsequently superimposed onto a set of target images predetermined locations. Following this, Karmon et al. proposed LAVAN that also generates universal patches. However, they used random search to optimize the patch location. In black-box scenarios, Brown et al. produced universal adversarial patches by executing white-box attacks on four DNN classifiers and transferring the results to an unseen classifier. Croce el al. proposed the Patch-RS method, which generates adversarial patches by minimizing the aggregated loss of set of images, using random search for patch location optimization and the Square-Attack method of Andriishhhhnko ef al. for patch pattern optimization.
In contrast to universal adversarial patch approaches, other researchers have focused on generating image-specific patches. |Fazzi and Frossard Frossardccreatedd patches for individual images by optimizing the position and shape of rectangular patches with predefined monochrome pattern. Built upon the LAVAN concept. |ao er al. proposed alternative techniques for patch location optimization using random search variants. Yang et al. and Croce et al. further advanced the image-specific scenario. with the former using reinforcement learning for patch generation and the latter applying the Patch-RS method to minimize the loss of a single image.
Despite recent advancements in creating both universal and image-specific adversarial patches, the glaring distortions from significant modifications to input images raise practical concerns. This issue also impacts the accurate assessment of DNN robustness against adversarial patches.
2.2 Minimum-Norm Attacks
Minimum-nomm attacks diverge from the traditional adversarial attacks by focusing on finding the smallest perturbation that can lead to misclassification under specific norm. These attacks offer
more comprehensive assessment of DNN robustness [45). Although white-box attacks have made
significant progress in enhancing the query efficiency of minimum norm attacks [45] 41 48). black- box attacks still demand a substantial query budget to achieve effecttiveness. The ZOO algorithm [ aggregated function of the perturbations 22-nrrm and weighted loss
constructs the problem function. Estimating its gradient using -iffe-diifeeennes, the authors make use of coordinate descent to minimize the formulated problem. Tu al. addressed the query inefficiency of ZOO by reducing the size of the perturbation using a trained Auto-Encoder DNN. Ilyas el al, reeooee the need for estimating coordinate-specin gradients by making use of natural evolutionary strategies, reducing the l.ee -nom of the perturbation by iteratively maximizing the adversarial criterion within a given norm constraint, then reducing the norm. Despite the efficiency improvement of gradient estimation, existing black-box methods still require large query budgets. The SimBA method of (23) incrementally adds scaled orthogonal vectors the current perturbation, increasing its no-norm, but unable reduce the n--orm of the perturbation once the desired misclassification has been achieved.
Therefore, while recent works have achieved large performance gains within the white-boo box scenario, black-box box methods suffer from query inefficiency which restricts their applicability to real-world scenarios, particularly when the query budget is limited.
2.3 Evolutionary Strategies for Adversarial Attacks
Many existing studies employ evolutionary strategies (ES) for non-patch-based adversarial attacks. In the black-box scenario, ES has gained popularity due to its independence from gradient information. Notable examples include the works of
who utilize evolutionary algorithms to create Lc constrained adversarial perturbations. For conducting sparse adversarial images Williams and Li
[ 46 36
3

---

Algorithm 1: Evolutionary Strategy for Generating Disguised Adversarial Patches (CamoPatch)
Input: Margin loss L. input x E X' S [0, 1]xxxx3, , query budget K, sparsity 6, initial
temperature f, number of circles N, location schedule li, evolutionary step-size 0
1 8 ve // Patch Side Length
2 < PuthialPatch(N,, s)
3 i ~ u(10, ... w - -s})
4 j u(10, , 8})
5 xx +x
6 Xisit, à Apply patch
L + -(x")
norm + -|xisit..;:st* - 8|12
for k 4-111 1; < K; + + do
10
if modd(k k, li 1) 0 then
11
j, L, norm
LocalionUpdaaee // see Algorithm
else
13
6, L, norm c PatchU pdatee) //see Algorithm 3
return
make use of a multi-objective evolutionary algorithm minimize both the number of modified pixels and magnitude of the pixels modification. ESs have also been used to construct adversarial examples within other domains such natural language processing 68 67]
The use evolutionary algorithms has also been explored for constructing adversarial patches. Chen al. Jaddressed the more-limited decision-only setting (where only the predicted label is accessible to the attacker) by placing localised regions of an image from the target class onto the attacked image. Under the constraint of the patch causing misclassification, the authors optimised the coordinates of the patch by using an adapted differential evolution algorithm to minimise the patch's ' to norm.
3 Proposed Method
In essence, our method strives to generate adversarial patches that seamlessly blend into the targeted image by modeling the superimposed area with semi-transparent, RGB-valued circles. We adopt an approach akin to existing works that generate adversarial patches by iteratively optimizing both the patch and its position on the image. The balance between these steps is managed by a location schedule parameter, (i. In this section, we start by defining the problem formulation, followed by detailed description of our proposed method. The overarching structure of our method is summarized in Algorithm[]
3.1 Problem Formulation
Consider a trained DNN image classifier f : X S [0, 1|xx0x33 - R which takes a benign RGB image x E X' of height h and width and outputs label y argmax fx(x), with P representing
pe(1,.., .P)
the total number of class labels. nar-tereeted attack seeks perturbation 8 satisfying: arginax [p(x + 8) Uga pE(1.--PP)
where y is the original class label for x and y, = argmax fx() a label corresponding to class Φ
other than the true class y. For targeted attacks y, is assigned a target label yt. where yt * y. In the adversarial patch scenario, the number of modified pixels is limited to maintain the semantic content of the image. Hence, the problem cast as:
minimize 8
C(f:x + 8, 5,
(2)
subject to
|81||- S e, x5x+85 1

---

where minimizing the loss function L yields the desired adversarial image.
Most existing algorithms solve (2) by fixing the number of disturbed pixels constant value e, allowing unbounded modifications [15 29 7l. Unlike these, our proposed method aims to create adversarial patches that closely resemble the area of the original image they overlay. We approach 58
this as a constrained optimization problem, akin to the minimum-non setting 45
Thus, our
objective is to generate ô that solves the subsequent optimization problem:
minimize 8
||x
8 |011o
subject to
f( x + 6, Mq <0,
0 x 1.
where the patch is square with a side length of Vε. We ensure that the value of the loss function C(-) is negative when x + ô results in misclassification. This is achieved by defining the loss in the constraint as the margin loss:
c(f:x + 8, 8, fa fug.
(4)
for non-targeted attacks and the cross-entropy loss:
-S log( Σ e
(5)
p=1
for targeted attacks.
3.2 Patch Initialization
We construct an adversarial patch by overlaying N circles black image (see Figure 2). inspired by evolutionary strategies prevalent in computational art 32
21 57. They aim to approximate images using semi-transparent, RGB-valued shapes. Circular shapes, due to their fewer adjustable properties, are popular choice. Furthermore, the use of semi-transparent circular shapes have also been used by Li al. and Zolfi el al. to construct adversarial examples. Whereas this work constructs adversarial patches, Li el al. and Zolfi ef al. simulate stickers placed over camera, modifying entire image.
The adversarial patch 8 represented as the concatenation of N shapes:
8 82
(6)
where @ denotes the concatenation operator. Each shape 8", where f1,..., NN. is represented by a vector comprised of seven elements including the center's ' coordinates (cf, cg), the radius r", the RGB values (R", G". B"), and the shape's 's transparency T These elements, normalized to continuous values between 0 and 1, are initially randomly sampled from a uniform distribution 8" (0, 1). The initial location of patch is also randomly and uniformly sampled from the available pixel locations.
3.3 Patch Optimization
In this paper, we employ a single solution evolutionary strategy, known as (1 + 1)-ES, to modify the properties of each shape 8". This approach has proven to be efficient for approximating images [2] To adjust the properties, we sample values from normal distribution a (0(0 I), where is a tunable parameter controlling the trade-off between exploration (i.e., searching new areas in the solution space) and exploitation (i.e., refining the current solution). A larger or promotes exploration, while a smaller one favors exploitation. We then the updated perturbation 8 to construct an adversarial image x** The solution that satisfies the constraint as per (3) is retained. If both solutions meet this constraint, we opt for the with smaller la distance from the original image. The patch update method is detailed in Algorithm 3 in the supplementary document.
3.4 Location Optimization
5

---

Addressing the discrete nature of pixel locations, many existing methods have employed random search to optimize the position of the patch However, ran-
Tree frog + Grasshopper
within the image 15 29
dom search methods often falter when encoun- tering local optima. To mitigate this, Skiscim and Golden introduced simulated annealing, method that probabilistically accepts worse so- lutions based on the search temperature and the performance difference between current and new solutions. This approach promotes explo- ration of the search space in the early stages of optimization and gradually becomes more selec- tive, favoring solutions with better quality in the later stages.
Figure 2: This illustration shows an adversarial image (left) with the adversarial patch outlined, and the magnified patch (right) for better visibility, This patch is generated by the proposed method using .N- = 100 overlapping circular shapes.
In our work, we leverage the fast simulated an- nealing approach proposed by (Szu and Hartley] to optimize the location of a patch. During each iteration k, we uniformly sample a single location (denoted as (i", from the location space. Then, apply the patch to the new location on the input image x, and construct updated adversarial image x The new solution x** is then evaluated using the loss function L. If both x" and x** satisfy the loss L constraint as per 3). we retain the solution with the lowest (2- 22-morm from the original image. Otherwise, simulated annealing is employed to probabilistically decide the acceptance of the new solution. Specifically, the acceptance probability is defined as xxp(-ddtccurr), where d is the loss difference between the current and new solution, and heur = t/k follows an exponentially decreasing schedule. This formulation ensures better solutions are always selected, while solutions with relatively poor quality are more likely to be accepted in the early search stages for enhanced exploration. The parameter / is predefined, with larger values promoting exploration during a longer portion of the attack process. The detailed location update method can be found in Algorithm 4 in the supplementary document.
4 Empirical Study
In this section, we empirically evaluate our proposed method's effectiveness by attacking classifiers trained on the ImageNet dataset [16]. The experimental setup is outlined in Section [..1] followed by comparative analysis with state-of-the-a arr adversarial patch methods, including Patch-RS [T]. TPA (66), OPA 20). ady-Watermark (27 and a black-box adaptation of LOAP [4) in Section 4.2
Last but not the least, Section 4.3 offers ablation study that scrutinizes the significance of various components and parameters within our proposed method.
4.1 Experimental Setup
Dataset and Classifiers Settings: For our experiments, follow similar setup to preceding works, conducting non-targeted and targeted attacks on DNN classifiers trained on the ImageNet dataset. We specifically target three adversarially trained and defended classifiers, namely AT-ResNet-
50 49 AT-WiddRReNNet-50- 44 and PatchGuard [65], along with three conventionally trained
classifiers, VGG-16 (51), Re [24] and ViT-B/16 [17]. A subset of 000 images, correctly classified by each classifier from the ImageNet validation set, is chosen and resized to dimensions of 224 x 224 x 3. For targeted attacks we randomly select y/: for each image ensuring it is different from the images true label y. The adversarially trained and defended classifiers are implemented using the RobustBench library (14) and authors original implementations, respectively, while the three conventional classifiers are derived from their pre- rreetrained versions available in the Py PyTorch library 44 All experiments were carried out on a system with an NVIDIA GeForee RTX 2080Ti GPU.
Parameter Settings: To select the value of e, we follow the approach of Croce el al. setting
<= 1600. This corresponds to patch size of 40 x 40, which constitutes roughly 3.2% of the total pixel count. We assign a budget of 10, 000 queries for each attack. As discussed in Section 3 our proposed method entails four free parameters: o,, lit, t, and N. For these parameters, we set
0.1.

---

Table 1: Table presents the before and after-accuracy of each method along with the l2 distance of the adversarial patch and the non-normalised residual (NNR) between the adversarial and original image after conducting non-targeted attacks. We provide the mean and variance of each metric over 10 runs.
AT WideltesNer 50
AT ResNet-50
Attack Method
Accuracy
12
NNR
Accuracy
12
68.46%
64.02%
CanioFatch
12.98% (0.01)
0.14 (0.05)
0.12 (0.07)
6.00% (0.03)
0.15 (0.03)
0.13 (0.03)
Patch-RS*
14.42% (0.01)*
0.43 [0.07]*
0.30 (0.06)4
12.00% (0.02)+
0.41 (0.12)*
0.33 (0.0G)*
Patch-RS
14.42% (0.01)*
0.74 (0.08)*
0.42 (0.07)*
12.00% (0.02)3
0.74 (0.00)*
0.43 (0.07)
TPA
61.60% (1.3)
0.82 (1:21))
0.82 (0.07)
34.82% (1.41))
(0.05)
0.87(0.09)
OPA
96.88% (0.1)*
0.76 (0.20)
0.74 (0.05)4
24.83% (1.12)
0.77 (0.14)
0.76 (0.04)*
LOAP
38, 85% (0.4)
0.56 (0.02)
0.46 (0.03))
48. 48.80%
(0.1)*
0.72 18.1
0.64 (0.03))
Ady watermarkk
52. 0.%0% (0.3)
0.37(0.05)*
0.23(0.07)
44.00% (0.3)
0.42 (0.02)
0.29 (0.07))
B/16
BagNet9 with PatchGGard
Attack
Accuracy
1
NNR
Accuracy
1g
NNRI
Method
17.91%
65.1%
CarioPatch
8.00% (0.05)
0.09 (0.02)
0.12 (0.02)
3.20%
(0.01)
0.07(0 03)
0.11 (0.01)
Patch-RS*
19.00% (0.10)+
0.68 (0.05)
0.39 (0.07)
5.80%
(0.02)*
(0.06)+
0.30 (0.05)
Patch-RS
10.00% (031
0.71 (0.12)*
0.41 (0.09)
5.80% (0.02)*
0.62 (0.18)*
0.57 (0.11)1
TPA
38% (0.91)*
(0.08)
(0.09)
32.87% (1.45)
(0.
0.61;0.000
OPA
33.09% (0.17)*
0.68 (0.23)
0.68 (0.07)+
57.89% (2.01)*
0.61 (0. (0.16))
0.67 (0.04)²
LOAP
43.91% (0.8
0.63 (0.05)
0.50 (0. 13)1
72.87% [0.14]
0.89 (0.23) 23)1
0.78 (0.11))
Ady watermark
36.01% (0.12)*
0.17(0.04)
0.28(0.03)
42.00% (0.45)
0.14(0.01)
0,29(0.05)
VGG-16
RosNet-50
Attack Method
Accuracy
12
NNR
Accuracy
12
73.36%
76.12%
Camooatch
9.70% (0.03)
0.09 (0.02)]
0.11 (0.02)1
10.00% (0.02)1
0.08 (0.01)
0.10 (0.01)
Patch-RS*
6.82%
0.42 (0.02)
0.30 (0.06)*
15.92%
(0.02)*
0.45 (0.04)
0.31 (0.04))
Patch-R8
6.82% (0.04)
0.63 (0.01)*
0.61 (0.07)*
15.92% (0.02)1
0.67 (0.08)*
0.69 (0.07)1
TPA
(1.30)
0.61 (0.13)
0.55 (0.05)
36.95% (1.41)
(0.07)
0,58/0.07)
OPA
32, 12.19% (0.10)*
0.71 (0.20)*
0.64 (0.06)*
27.91% (1.12)*
0.71 (0.14)*
0.00 (0.04)*
LOAP
37.99% (0.403*
0.68 [0.02)*
0.63 (0.05)*
47.90% [0.10]"
0.78 (0. 12]
0.67 (0.05)*
Adv watermark
32.00% (0.103"
0.13(0.05)
0.25(0.05))
35.00% (0.40)
0.16(0.01)
0.31(0.07)
t denotes the performance of the method significantly outperforms the compared methods
according to the Wilcoxon signed-rank test [60] at the 5% significance level: denotes the
corresponding method is significantly outperformed by the best performing method (shaded).
t = 300, lit 4, and N 100. We provide empirical justification for these specific settings
in Section 4.3
Performance Metrics: We evaluate the performance of all considered algorithms by allowing each method to exhaust the allocated query budget while attacking each classifier. To evaluate the effectiveness of an attack we report the accuracy of the classifier on the generated adversarial images. For the successful adversarial images, we report two additional metrics: (1) the I2 distance between the adversarial patch and the corresponding area of the original image, and (2) the non-normalised residual (NNR) between the adversarial and original image, which measures the absolute difference between the pixel values of the constructed patch and the area of the original image covers.
Given the stochastic nature of our proposed method and the comparison methods, we follow the setup of 153 and report the mean and variance of each metric over 10 independent runs with different random seeds. We additionally utilize the Wilcoxon signed-rank test [60] at 5% significance level to statistically verify whether the improvements by our method over the compared algorithms across the 10 runs are significant.
4.2 Comparison
For the adaptation of LOAP 47 we replace its gradient computation method with the estimation method of [26]. The detailed description of this estimation method can be found in Algorithm 2 in the supplementary document. Additionally, we compare our method with an adapted version of Patch-RS, where we minimize the l2 distance of the constructed patch in a similar manner to our proposed method. For each compared algorithm, we utilize the authors' original implementation and recommended settings. In our black-box adaptation of LOAP 47), we set the number of iterations
50 and variance 17 0.001 for the gradient estimation method of Ilyas et al.
7

---

Original
CamoPatch
Patch-RS
Patch-RS*
TPA
OPA
LOAP
Figure 3: Adversarial images generated by methods conducting con-targeted attacks on the conven- tionally trained VGG-16 (top) and adversarial trained WideResNet-50. (4 (bottom) classifiers. Whereas adversarial patches generated by state-of-the-art methods are visibly clear, the patches generated by the proposed method are well camouflaged within the image.
Table ! present the statistical results of non-targeted attacks conducted on the trained ImageNet classifiers. In the tables, "CamoPatch" denotes our proposed method, and "Patch-RS** refers to the adapted Patch-RS algorithm.
Results:
These results demonstrate that the Patch-RS attack, along with our own method, achieves higher attack success rates compared to the other state-of of-theeart methods. This result aligns with previous work 15]. which demonstrated the superior performance of the Patch-RS algorithm. Despite Patch-RS outperforming our method when attacking the VGG-16 classifier, according the Wilcoxon signed- rank test, there is no significant difference between the performance of both methods. Alternatively, when attacking the remaining five classifiers, the proposed method is able to significantly outperform Patch-RS and other compared methods according to the Wilcoxon signed-rank test.
Comparing the 22 distance and NNR of adversarial patches generated by the attack methods, the proposed method able to construct adversarial patches that are far less invasive to the input image. This is supported by the proposed method significantly outperforming all other methods in terms of both l2 distance and NNR, according to the Wilcoxon signed-rank test. This result highlights that the effectiveness of adversarial patches is not compromised by their perceptibility.
Despite the adapted Patch-RS* algorithm being able to generate patches with lower la distances from the original image compared to its original implementation, its use of Square-Attack (2) for patch pattern optimization results in the patch values taking the corners of the color cube [o, 1]. Therefore, its ability for (2 minimization is significantly hampered. Alternatively, the proposed method is able to construct patches with any color, which allows for effective approximations of the original image area. Figure provides a visual comparison of adversarial images generated by each method when attacking the VGG-16 16 and AT-ResNet-50 classifiers.
We report the results of the targeted attacks in Section 6.1 of the supplementary material.
Robustness Evaluation: Despite the assumption that adversarial trained classifiers have improved robustness compared to their conventionally trained counterparts, our experimental results reveal a different picture. The proposed method achieves higher success rates when attacking the adversarial trained classifier AT-ResNet-50 of Salman el al. compared to the conventionally trained VGG-16 and ResNet-50 classifiers.
However, the results also demonstrate that the adversarial patches generated by the proposed method, when attacking the AT-ResNet-50 classifier, exhibit larger l2 distances from the original image, resulting in larger non-normalised residuals between the entire adversarial image and the original image. This suggests that while the adversarial trained classifier is more susceptible to adversarial patches, these patches require larger distortions to cause the misclassification. On the other hand, conventionally trained classifiers are more susceptible to smaller changes in the original image. This behavior can be attributed to the general procedure of adversarial training, which introduce noise onto images during the training to enhance robustness. Consequently, patches with larger impact decrease the likelihood of the image being representative of the training data, as has been observed other works [50]. Furthermore, wee see the AT ST-WiidRReNNe-5504 classifier exhibits greater robustness to
8

---

Table 2: Table presents the before and after-accuracy of each CamoPatch configuration along with the l2 distance the adversarial patch and the non-normalised residual (NNR) between the adversarial and original image after conducting non-targeted attacks. We provide the mean and variance of each metric over 10 runs.
VGG-16
N
Accuracy
12
NNR
Runtime(s)
76. 12%
300
100
0.1
12.88%((.0)
0.09(0.04)
0.14(0.01)
440.03 10.32)
100
100
0.1
10, (1..(
0.09(0.07)
0.13(0.02)
440.01
300
100
0.1
(1.0%(1.0)
0.09
(0.0()
440.03( 10.32)
300
100
0.3
11.66%((20)
0.09(0.06)
0.11((.05)
439. 13 10 56
1 denotes the performance of method significantly outperforms the compared methods according to the Wilcoxon signed-rank test [60] at the 5% significance level; denotes the corresponding method is significantly outperformed by the best performing method (shaded).
method. Since both AT-WideResNet-50-22 and AT-ResNet-50 models are trained using the same process, these results suggest that the WideResNNr architecture is inherently more robust.
4.3 Ablation Study
The proposed method consists of four tunable parameters: li, N, and f. To determine their optimal values, we conduct grid search over the parameter space. Specifically, we explore E {1, N 1100,300), t E {100, 300), and E {0.1, 0.3). The choice of li follows the recommendation of Croce et al.) while the values of a. t, and are commonly used in the evolutionary [52 and computational art [57] communities, respectively.
To evaluate the performance of each parameter configuration, we conduct non-targeted attacks on the VGG-16 ImageNet classifier using 1000 correctly classified images from the validation set. We measure the accuracy of the model on the generated adversarial images, l2 distance and NNR for each configuration over 10 independent runs with different random seeds. Additionally, we compare the computational time required for each configuration to complete an attack on a single image.
Configurations: Table 2 presents the four top performing configurations in terms of the attack accuracy. The results demonstrate that the performance of the proposed method heavily depends on the number of circles N used to construct the patch pattern. Increasing the number of circles allows for better detailed approximations but also introduces additional complexity. From the results in Table we observe that the best performing configurations all use N = 100, suggesting that the patch optimizer, (1 + 1) 1)-ES, struggles with larger numbers of circles. Moreover, we observe longer runtimes for N = 300 due to the increased number of properties that need adjustment. Beyond the number of circles N, the proposed method achieves improved performance with a larger budget for patch pattern optimization (li 4) and a larger exploration parameter for location optimization = 300). Based on these findings, we set the optimal parameter configuration of the proposed method to li 4, 300, N 100, and 0.1.
Simulated Annealing: To justify the use of simulated annealing for location optimization within the proposed method, we compare its performance with and without simulated annealing. Removing simulated annealing results in a pure random search method similar to existing works. We keep the other parameters of the proposed method constant with those outlined in Section 4.1 The results in Table Jemonnsaeetst the improved performance exhibited by the proposed method when the simulated annealing policy is employed for location optimization, particularly when attacking adversarial trained classifiers. Despite generating patches with a higher l2 distance, the proposed method with simulated annealing achieves higher success rates. This suggests that more challenging images require larger distortions to cause misclassification, increasing the average l2 distance of the generated successful adversarial patches.
5 Contributions, Limitations and Future Work
Contributions: In this work, we propose CamoPateh, a novel attack method for generating ad- versarial patches that can blend into the targeted image. We achieve this by constructing the patch

---

Table 3:
Table presents the before and after-accuracy of the CamoPatch method with (CamoPatch) and without (CamoPatchh) the simulated annealing policy for location optimization, along with the l2 distance of the adversarial patch and the non-normalised residual (NNR) between the adversarial and original image after conducting non-targeted attacks. We provide the mean and variance of each metric over 10 runs.
Canol'ath
Classitier
ASR
NNR
ASR
NNK
VGGG-16
10.02:
(0.1)
(0.01)
0.11 (0:02)
RowNet-50
90..00%
02)
(0:01)
0.08 (0.01]
90.00%
T0.01)
0.09 (0:01)
(0.02)
Nett
87.02% 01)
0.14 (0.05))
0.12 (0.07: 83.02% (0.02)*
0.11 (0.05)
0.09 (0.02)
A1-ResNet-5t
94.00% (a. 03)
0.15 (0.03)
0.13 (0.03)
90.00% (0.01)
0.14 (0.03)
0.13 (0.06)
i denotes the performance of the method significantly outperforms the compared methods according to the Wilcoxon signed-rank test [60] the 5% significance level; denotes the
corresponding method is significantly outperformed by the best performing method (shaded).
pattern using a combination of semi-transparent, RGB-valued circles, which are optimized to cause misclassification and approximate the covered area of the original image. By incorporating sim- ulated annealing policy for location optimization, our method generates adversarial patches with improved or comparable success rates, while minimizing the visual impact on the target image.
Ethical Considerations: Adversarial patches have gained attention due to their potential real-world applications, where attackers can print and physically place them deceive real-world implemented DNNs [8). However, existing methods often generate patches that are visually clear and easily detectable a human observer. Our work introduces the concept of camouflaged adversarial patches, which are difficult for both humans and computer vision systems to perceive. This raises further concerns about the robustness of DNN classifiers in safety-critical applications. Adversarial training has proven to be an effective method of improving the robustness of DNN classifiers to adversarial patches 47 Incorporating images with camouflaged adversarial patches into the training process of DNNs may be a promising avenue to enhance their robustness and mitigate the vulnerabilities demonstrated in this work.
It is important to acknowledge the limitations of the proposed method and identify potential areas for future research. One limitation is that our method assumes the attacker has access to the output probabilities of the targeted DNN, which may not always be the case in real-world scenarios. Future work could explore adapting the proposed method to scenarios where only the predicted label of the input is available, by utilizing estimated loss functions such as the one proposed by Ilyas et al.) Furthermore, techniques from weakly supervised learning can be incorporated into the proposed method to address the label-only setting. Specifically, by using estimation techniques (26] to score constructed adversarial images, the use of weakly supervised image classification models (25] surrogates could improve the efficiency of the proposed method in addition to providing a direction of the search for the label-only setting. Alternatively, utilizing the stochastic nature of the proposed method to generate set of non-evaluated candidate solutions, the use of the weakly supervised learning techniques such as semi-supervised learning could be applied 300 to train surrogate model on both evaluated and non-evaluated adversarial images. Thereby using the surrogate to select predicted-optimal solutions for evaluation.
Limitations and Future Work:
Another limitation is the DNN query budget assumption in experiments. In practice, the available query budget might be significantly lower. To address this limitation, future research could extend the proposed method to incorporate surrogates or approximation models that guide the attack process, using fewer DNN queries, similar to that of Williams et al. within the computational art field.
Lastly, the parameter tuning process in our work follows a conventional grid-search approach, which limits the exploration of parameter combinations. Bayesian optimization methods could be employed to automate the configuration of attack algorithms, leveraging Gaussian Process surrogate models to handle the stochastic nature of the proposed method and guide the parameter search 10 37 This would provide a more efficient approach for determining the optimal parameter configuration.
10

---

Acknowledgement
This work was supported in part by the UKRI Future Leaders Fellowship under Grant MR/S0170621 and MR/X011 135/1; in part by NSFC under Grant 62376056 and 62076056; in part by the Royal Society under Grant IES/R2/212077; part by the EPSRC under Grant 2404317; part by the Kan Tong Po Fellowship (KTP/R1/231017); and in part by the Amazon Research Award and Alan Turing Fellowship.
References
[1] Moustafa Alzantot, Yash Sharma, Supriyo Chakraborty, and Mani B. Srivastava. Genattack: Practical black-box attacks with gradient-free optimization. CoRR, 2018.
[2] Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: A query-efficienn black-box adversarial attack via random search. In Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII. Springer, 2020.
[3] Tao Bai, Jinqi Luo, Jun Zhao, Bihan Wen, and Qian Wang. Recent advances in adversarial training for adversarial robustness. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event Montreal, Canada, 19-27 August 2021, 2021.
[4] Arjun Nitin Bhagoji, Warren He, Bo Li, and Dawn Song. Practical black-box attacks on deep neural networks using efficient query mechanisms. In Computer Vision ECCV 2018 15th European Conference. Munich, Germany, September 8-14, 2018, Proceedings, Part XII, Lecture Notes in Computer Science, 2018
[5] Gerda Bortsova, Cristina González-Gonzan Suzanne C. Wetstein, Florian Dubost, Ioannis Katramados, Laurens Hogeweg, Bart Liefers, Bram van Ginneken, Josien P. W. Pluim, Mitko Veta, Clara I. Sánchez, and Marleen de Bruijne. Adversarial attack vulnerability of medical image analysis systems: Unexplored factors. Medical Image Anal., 2021.
[6] Wieland Brendel, Jonas Rauber, Matthias Kümmeree, Ivan Ustyuzhaninov, and Matthias Bethge. Accurate reliable and fast robustness evaluation. In Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, 2019.
[7] Tom B. Brown, Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer, Adversarial patch. CoRR, hts//712.09665 2017.
[8] Nicholas Carlini and David A. Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy, SP 2017, San Jose, CA, USA, May 22-26, 2017. IEEE Computer Society, 2017.
[9] Renzhi Chen and Ke Li. Transfer bayesian optimization for expensive black-box optimization in dynamic environment. In 2021 IEEE International Conference on Systems, Man, and Cybernetics, SMC, 2021.
[10] Renzhi Chen and Ke Li. Data-driven evolutionary multi-objective optimization based on Opti-
multiple-gradien descent for disconnected pareto fronts. In Evolutionary mization- 12th International Conference, EMO 2023, Leiden, The Netherlands, March 20-24, 2023, Proceedings, 2023.
[11] Zhaoyu Chen, Bo Li, Shuang Wu, Shouhong Ding, and Wenqiang Zhang. Query-efficient decision-based black-box patch attack. IEEE Trans. Inf. Forensics Secur. 2023.
[12] Francesco Croce and Matthias Hein. Sparse and imperceivable adversarial attacks. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019. IEEE, 2019.
[13] Francesco Croce and Matthias Hein. Minimally distorted adversarial examples with a fast adaptive boundary attack. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, 2020.
11

---

[14] Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: a standardized adversarial robustness benchmark. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021.
[15] Francesco Croce, Maksym Andriushchenko, Naman D. Singh, Nicolas Flammarion, and Matthias Hein. Sparse-rs: A versatile framework for query-efficient sparse black-box adver- sarial attacks. In Thirty-Sixth Sixth AAAI Conference on Artificial Intelligence, AAAI 2022. AAAI Press, 2022.
[16] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition.
[17] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conferencee on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
[18] Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physical- phyri attacks on deep learning visual classification. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, 2018.
[19] Yanbo Fan, Baoyuan Wu, Tuanhui Li, Yong Zhang, Mingyang Li, Zhifeng Li, and Yujiu Yang. Sparse adversarial attack via perturbation factorization. In Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXII, 2020.
[20] Alhussein Fawzi and Pascal Frossard. Measuring the effect of nuisance variables on classifiers. Proceedings of the British Machine Vision Conference 2016, BMVC 2016, York, UK, September 19-22, 2016, 2016.
[21] Julia Garbaruk, Doina Logofatu, Costin Badica, and Florin Leon. Digital image evolution of artwork without human evaluation using the example of the evolving mona lisa problem. Vietnam Journal of Computer Science, 2022.
[22] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adver- sarial examples. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[23] Chuan Guo, Jacob R. Gardner, Yurong You, Andrew Gordon Wilson, and Kilian Q. Weinberger. Simple black-box adversarial attacks. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, Proceedings of Machine Learning Research. PMLR, 2019.
[24] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision ECCV 2016 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, Lecture Notes in Computer Science. Springer, 2016.
[25] Mengying Hu, Hu Han, Shiguang Shan, and Xilin Chen. Weakly supervised image classification through noise regularization. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2019.
[26] Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 Proceedings of Machine Learning Research. PMLR, 2018
[27] Xiaojun Jia, Xingxing Wei, Xiaochun Cao, and Xiaoguang Han. Ady-watermark; A novel watermark perturbation for adversarial examples. ACM, 2020.
12

---

[28] Kaixun Jiang, Zhaoyu Chen, Tony Huang. Jiafeng Wang, Dingkang Yang, Bo Li, Yan Wang, and Wenqiang Zhang. Efficient decision- bassd black-box patch attacks on video recognition. CoRR, 2023.
[29] Danny Karmon, Daniel Zoran, and Yoav Goldberg. Lavan: Localized and visible adversarial noise. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüssann Stockholm, Sweden, July 10-15, 2018, 2018.
[30] Jungtaek Kim. Density ratio estimation-baset bayesian optimization with semi-supervised learning. CoRR, 2023.
[31] Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial examples in the physical world. CoRR, 2016.
[32] Nicholas Lambert, William H. Latham, and Frederic Fol Leymarie. The emergence and growth of evolutionary art: 1980-1999 1993. In International Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2013, Anaheim, CA, USA, July 21-25, 2013, Art Gallery, 2013.
[33] Mark Lee and J. Zico Kolter. On physical adversarial patches for object detection. CoRR, 2019.
[34] Ke Li and Renzhi Chen. Batched data-driven evolutionary multiohjective optimization based on manifold interpolation. IEEE Trans. Evol. Comput., 2023.
[35] Juncheng Li, Frank R. Schmidt, and J. Zico Kolter. Adversarial camera stickers: A physical camera-based attack on deep learning systems. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA. PMLR, 2019.
[36] Chao Li, Handing Wang. Jun Zhang, Wen Yao, and Tingsong Jiang. An approximated gradient sign method using differential evolution for black box adversarial attack. IEEE Transactions on Evolutionary Computation, pages 2022.
[37] Ke Li, Renzhi Chen, and Xin Yao. A data-driven evolutionary transfer optimization for expensive problems in dynamic environments. IEFE Trans. Evol. Comput., 2023. accepted for publication.
[38] Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. In 2nd International Conference on Learning Representations, ICLR 2014, Bansf, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
[39] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. CoRR, 2017.
[40] Apostolos Modas, Seyed-Mohsen Mohsen Moosavi-Dezfooli and Pascal Frossard. Sparsefool: A few pixels make a big difference. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. Computer Vision Foundation / IEEE, 2019.
[41] Seyed-Mohsen and accurate method to fool deep neural networks. In 2016 IEEE Conference Computer Vision and Puttern Recognition, CVPR 2016, Las Vegus, NV, USA, June 27-30, 2016, 2016.
Alhussein Fawzi, and Pascal Frossard. Deepfool: A simple
[42] Misha Paauw and Daan van den Berg. Paintings, polygons and plant propagation. In Compula- tional Intelligence Music, Sound, Ari and Design 8th International Conference, EvoMUSART 2019, Held as Part of EvoStar 2019, Leipzig, Germany, April 24-26, 2019, Proceedings, Lecture Notes in Computer Science. Springer, 2019.
[43] Nicolas Papernot, Patrick D. McDaniel, and Ian J. Goodfellow. Transferability in machine learning: from phenomena to blcck-bxx attacks using adversarial samples. CoRR, 2016.
[44] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. Advances Neural Information Processing Systems 32. 2019.
13

---

[45] Maura Pintor, Fabio Roli, Wieland Brendel, and Battista Biggio. Fast minimum- norm adversarial attacks through adaptive norm constraints. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, 2021.
[46] Hao Qiu, Leonardo Lucio Custode, and Giovanni lacca. Black-box adversarial attacks us- ing evolution strategies. In GECCO '21: Genetic and Evolutionary Computation Conference, Companion Volume, Lille, France. July 10-14, 2021, 2021.
[47] Sukrut Rao, David Stutz, and Bernt Schiele. Adversarial training against location-opimizee adversarial patches. In Computer -CCV 2020 Workshops Glasgoww UK, August 23-28, 2020, Proceedings, Part V, 2020.
[48] Jérôme Rony. Luiz G. Hafemann, Luiz S. Oliveira, Ismail Ben Ayed, Robert Sabourin, and Eric Granger. Decoupling direction and norm for efficient gradient-based L2 adversarial attacks and defenses. In JEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, 2019.
[49] Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do ad- versarially robust imagenet models transfer better? In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020,
December 62, 2020, virtual, 2020.
[50] Vikash Sehwag, Arjun Nitin Bhagoji, Liwei Song, Chawin Sitawarin, Daniel Cullina, Mung Chiang, and Prateek Mittal. Analyzing the robustness of open-world machine learning. In Proceedings of the 12th ACM Workshop on Artificial Intelligence and Security, AlSec CCS 2019, London, UK, November 15, 2019, 2019.
[51] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for targe-scale image recognition. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[52] Christopher C. Skiscim and Bruce L. Golden. Optimization by simulated annealing: A prelimi- nary computational study for the TSP. In Proceedings of the 15th conference on Winter simulation, WSC 1983, Arlington, VA, USA, December 12-14, 1983, 1983.
[53] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller. Striv- ing for simplicity: The all convolutional net. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Workshop Track Proceedings, 2015.
[54] Christian Szegedy, Wojciech Zaremba. Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
[55] Harold Szu and Ralph L.. Hartley. Nonconvex optimization by fast simulated annealing. Proc. IEEE, 1987.
[56] Simen Thys, Wiebe Van Ranst, and Toon Goedemé. Fooling automated surveillance cameras: Adversarial patches to attack person detection. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2019. Computer Vision Foundation IEEE, 2019.
[57] Yingtao Tian and David Ha. Modern evolution strategies for creativity: Fitting concrete images and abstract concepts. In Artificial Intelligence in Music, Sound, Art and Design 11th International Conference, EvoMUSART 2022, Held as Part of EvoStar 2022, Madrid, Spain, April 20-22, 2022, Proceedings, Lecture Notes in Computer Science. Springer, 2022.
[58] Chun-Chen Tu, Pai-Shun Ting, Pin- Yu Chen, Sijia Liu, Huan Zhang, Jinfeng Yi, Cho-Jui Hsieh, and Shin- Cheng. Autozoom: Autoencoder-basee zeroth order optimization method for at- tacking black-box neural networks. In The Thirty-Thin AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-Fire First Innovative Applications of Artificial Intelligence Conference, IAAI
14

---

2019. The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 -February 2019, 2019.
[59] Jonathan Uesato, Brendan O'Donoghue, Pushmeet Kohli, and Aäron van den Oord. Adversarial risk and the dangers of evaluating against weak attacks. In Proceedings of the 35th International Conference on Machine Learning, JCML 2018, Stockholmsmässan Stockholm, Sweden, July 10-15, 2018, Proceedings of Machine Learning Research, 2018.
[60] Frank Wilcoxon. Individual comparisons by ranking methods. In Breakthroughs in statistics. Springer, 1992.
[61] Phoenix Neale Williams and Ke Li. Art-attack: Black-box adversarial attack via evolutionary art. CoRR, fbs//200.004005 2022.
[62] Phoenix Neale Williams and Ke Li. Black-box sparse adversarial attack via multi-objective optimisation CVPR proceedings. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2023.
[63] Phoenix Neale Williams, Ke Li, and Geyong Min. Black- box adversarial attack via overlapped shapes. In GECCO '22: Genetic and Evolutionary Computation Conference, 2022.
[64] Phoenix Neale Williams, Ke Li, and Geyong Min. A surrogate assisted evolutionary strategy for image approximation by density-ratio estimation. In IEEE Congress on Evolutionary Compulation, CEC, 2023.
[65] Chong Xiang, Arjun Nitin Bhagoji, Vikash Sehwag, and Prateek Mittal, Patchguard: provably robust defense against adversarial patches via small receptive fields and masking. In 30th USENIX Security Symposium, USENIX Security 2021, August 11-13, 2021, 2021.
[66] Chenglin Yang, Adam Kortylewski, Cihang Xie, Yinzhi Cao, and Alan L. Yuille. Patchattack:
A black-box texture-based attack with reinforcement learning. In Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVI, 2020.
[67] Shasha Zhou, Ke Li, and Geyong Min, Attention-based genetic algorithm for adversarial attack in natural language processing. In Parallel Problem Solving from Nature PPSN.
[68] Shasha Zhou, Ke Li, and Geyong Min. Adversarial example generation via genetic algorithm: preliminary result. In GECCO '22: Genetic and Evolutionary Computation Conference, 2022.
[69] Alon Zolfi, Moshe Kravchik, Yuval Elovici, and Asaf Shabtai. The translucent patch: A physical and universal attack on object detectors. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, I, virfual, June 19-25, 2021, 2021.
15


Reviews:
[{"review_id": "b1bI2yYCgj", "reviewer": "Reviewer_tLpS", "summary": "This paper presents a novel method for generating adversarial patches that approximates the appearance of the area it covers, achieving better or comparable performance to state-of-the-art methods while minimizing the visibility of the patch. The method uses semi-transparent, RGB-valued circles that blend into the original image with an evolutionary strategy to fine-tune each circle's properties and employs simulated annealing for optimal location selection. This paper also provides an empirical evaluation of the proposed method's effectiveness by attacking classifiers trained on the ImageNet dataset and comparing it with state-of-the-art adversarial patch methods. The contributions of this paper include a new approach to generating adversarial patches that are more difficult to detect, a comparative analysis of state-of-the-art methods, and insights into the limitations and future directions of adversarial attacks on DNNs.", "questions": "1. As the core idea of this paper, ‘camouflaged’ can be presented in more forms such as visual comparison and evaluation on camouflage level.\n\n2. The simulated annealing method, as the important approach to optimize the patch’s location, should be improved or replaced by a novel approach so it can be more consistent with the idea of ‘evolutionary strategy’.\n", "limitations": "It’s appreciated that the authors are up front about the limitations of their work in the paper and introduces the possible solutions in the future work. The authors may draw lessons from weakly supervised learning methods to explore adapting the proposed method to scenarios where only the predicted label of the input is available.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well-written and organized, with clear explanations of the proposed method and the experimental setup. The authors provide detailed descriptions of the algorithms used and the evaluation metrics employed, making it easy for readers to understand the research.\n\n2. The proposed CamoPatch method is a novel approach to generating adversarial patches while minimizing the visibility of the patch. The use of semi-transparent, RGB-valued circles and the evolutionary strategy is a unique contribution to the field of adversarial attacks on DNNs.\n\n3. This paper provides a thorough empirical evaluation of the proposed method's effectiveness. The authors also provide insights into the limitations and future directions of adversarial attacks on DNNs, which adds to the quality of the research. The supplementary material is adequate as well.\n", "weaknesses": "It’s appreciated that the authors provide clear and detailed descriptions of the empirical study, though the work could be more credible if more experiments and evaluation are concerned.\n\n1. This paper only use attack success rate (ASR) to evaluate the attack performance of the method, which is kind of limited evaluation. More metrics should be involved such as non-normalized residual (nq), original accuracy and after-attack accuracy. It would be appreciated if the experiments in terms of targeted attack are concerned.\n\n2. When it comes to evaluating the visibility of the adversarial patches, the SSIM metric seems to be unnecessary because the difference between the proposed method and other methods in this metric is not obvious. The numerical gap is not greater than 0.02.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W#1:** As per requested by the reviewer, we report the original accuracy, after-attack accuracy, non-normalized residual (nq) and $l_2$ norm metrics of conducted targeted and non-targeted attacks in **Table 2 and Table 1 of the global 'author rebuttal PDF'**.\n\n**W#2:** The reason for the small numerical gap is due to the small $40 \\times 40$ patch size, which is roughly 3.2% of the total number of pixels within the image (lines 207-208). Therefore, a numerical gap of 0.02 is significant, considering the small number of changed pixels. \n\n**Q#1:** To evaluate the level of camouflage, we employ SSIM and $l_2$ norm as the performance metrics in our experiments. Moreover, we provide visual comparisons of adversarial images constructed by the proposed method compared to Patch-RS (Figure 1 of our manuscript) and other state-of-the-art algorithms (Figure 3 of our manuscript). Furthermore, to facilitate a visual comparison, we provide an example of an adversarial image with the patch magnified (Figure 2 of our manuscript).\n\n**Q#2:** The selection of simulated annealing (SA) as our location optimizer is based on its proven ability to effectively explore discrete spaces and successfully avoid local optima (lines 169-170). In addition to the performance of the proposed method compared to state-of-the-art methods (Table 1 of our manuscript), we further demonstrate the superior performance achieved when using SA compared to pure random search (Table 3 of our manuscript). In future, alternative methods can be explored to improve the location optimization of a patch location, i.e. the use of surrogates (lines 327-331).\n\n**Limitations:** To enhance and advance the proposed method, we recognize the potential benefits of incorporating techniques from weakly supervised learning as suggested by the reviewer to adapt to the label-only scenario effectively. We believe there are two particularly fruitful directions of incorporating techniques from weakly supervised learning to adapt to the label-only scenario. Firstly, by using estimation techniques (lines 321-326) to label constructed adversarial images, we believe the use of weakly supervised image classification models (Hu et al., Weakly Supervised Image Classification through Noise Regularization, CVPR'19) as surrogates would improve the efficiency of the proposed method in additional to providing direction of the search for the label-only setting. Secondly, utilizing the stochastic nature of the proposed method to generate a set of non-evaluated candidate solutions, the use of weakly supervised techniques such as semi-supervised learning could be applied (Kim et al., Density Ratio Estimation-based Bayesian Optimization with Semi-Supervised Learning, Corr'23) to train a surrogate model on both evaluated and non-evaluated adversarial images. Thereby using the surrogate to select predicted-optimal solutions for evaluation."}, {"review_id": "abN8dV4ORz", "reviewer": "Reviewer_Z9ie", "summary": "This paper proposes a evolutionary strategy based framework for generating imperceptible adversarial patch. Compared with existing methods, the proposed method can generate invisible adversarial patch by iteratively optimizing both the patch and its position on the image. ", "questions": "1. I am curious the computation cost of the proposed method.\n2. I am curious about the performance of the proposed method with different evolutionary strategy.\n3. How is the performance of the proposed method on transformer based vision models?", "limitations": "1. Test the computation time or memory cost of the proposed method.\n2. Test the proposed method with different evolutionary strategy.\n3. Test the proposed method on vision transformers.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The proposed method is novel and performs well on ASR and invisible. \n2. The method analysis is clear and algorithm pipeline is helpful.\n3. The experiments are solid and comprehensive.", "weaknesses": "1. Lack of the efficiency comparison with other methods.\n2. Lack of the different evolutionary strategy comparison.\n3. Lack of the performance on transformer based vision models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W#1, Q#1, L#1:** As shown in Table 2 of our manuscript, we compare the runtime of different hyper-parameter configurations. This partially covers the evaluation of the efficiency of our proposed method. On the other hand, for the sake of fair comparisons, we fixed the computational budget to be 10,000 queries (line 208; lines 212 to 213) for all peer algorithms in our experiments. In particular, we believe the querying of the attacked deep neural network is the most computationally demanding area of an adversarial attack.\n\n**W#2, Q#2, L#2:** In this work we make use of a classic (1+1)-evolutionary strategy (ES) for patch optimization (lines 153 to 154; reference [30]). One of the most important concerns of using other ES, such as CMA-EA and PGPE with ClipUp, for patch optimization is that they usually require a large number of function evaluations to achieve optimal image approximations. This is supported in previous works in the computation creativity community (modern ES for creativity: fitting concrete $481$ images and abstract concepts; reference [44]). Unfortunately, our context is restricted to a fixed query budget. As flagged in the conclusion section, we envisage the use of surrogate model assisted approaches such as Bayesian optimization can be explored for patch optimization to further enhance the efficiency of our proposed method (lines 332 to 336).\n\n**W#3, Q#3, L#3:** At the request of the reviewer, we show the results of conducted non-targeted attacks on a Transformer based DNN classifier (Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, ICLR'21 ) in **Table 1 of the global 'author rebuttal PDF'**."}, {"review_id": "rPypS7EuGA", "reviewer": "Reviewer_eiFr", "summary": "Adversarial examples for a DNN that are not visible to humans are generated with Evolutionary Strategies. The examples are semi-transparent RGB-valued circles. The shape and position are optimized.", "questions": "Questions:\n- Is \\sigma used in Algorithm 1?\n- What post-hoc adjustment did you use?\n- Why only 10 runs?", "limitations": "none", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "- ES and SA are gradient free methods\n- Patch detectability is taken into account\n", "weaknesses": "- How is visibility evaluated? Is L2 always working for the distance\n- No defense to new attack provided\n- Limited presentation of Evolutionary Computation methods for adversarial examples in the related works\n- Limited discussion of computational cost", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W#1:** The norm of a perturbation, including $l_2$, has proven to be a reliable metric of evaluate the visibility of a perturbation, with previous works either constraining (lines 58-60; reference [2]) or minimizing (section 2.2; references [45, 33, 29, 35, 8]) its value to ensure or promote the imperceptibility of the perturbation. We additionally make use of the SSIM metric to measure the structural impact of the patch on the overall image.\n\n**W#2:** The purpose of this work is to further highlight the vulnerability of DNN image classifiers to adversarial patches by proposing a novel attack algorithm that generates adversarial patches with lower visibility compared to current state-of-the-art methods, i.e. camouflaged. Although this work does not propose a defense mechanism to our attack, we discuss possible directions to address the vulnerabilities of DNNs demonstrated by this work in lines 311 to 320 of our manuscript. In particular, incorporating camouflaged adversarial patches in the training dataset, a.k.a. adversarial training, is a promising way to defend against our proposed method.\n\n**W#3:** To the best of our knowledge, the use of evolutionary strategies (ES) have not been explored for the construction of adversarial patches, therefore our discussion of ES is limited. However, within the black-box scenario, the use of meta-heuristics such as ES has becomes a popular approach due to their independence from gradient information. Allowing all pixels of an image to be modified, particular works include that of Alzentot et al., (GenAttack, GECCO'19) who make use of an evolutionary algorithm to evolve a population of adversarial images. Additionally, the work of Qiu et al., (Black-box adversarial attacks using evolution strategies, GECCO'21) apply the popular CMA-ES algorithm to generate adversarial images. Similarly, the work of Li et al. (https://doi.org/10.1109/TEVC.2022.3151373) make use of the differential evolution optimizer to generate adversarial images. Alternatively, the use of ES have also been applied to the sparse adversarial attack scenario. Specifically, the work of Su et al. also make use of differential evolution to generate adversarial images by modifying a single pixel of an image. We will include this discussion within our revisions.\n\n**W#4:** As shown in Table 2 of our manuscript, we compare the runtime of different hyper-parameter configurations. This partially covers the evaluation of the computational cost of our proposed method. On the other hand, for the sake of fair comparisons, we fixed the computational budget to be 10,000 queries (line 208; lines 212 to 213) for all peer algorithms in our experiments. In particular, we believe the querying of the attacked deep neural network is the most computationally demanding area of an adversarial attack.\n\n**Q#1:** $\\sigma$ is a parameter used within the patch optimization stage of the proposed method. It controls the trade-off between exploration and exploitation (lines 154 to 161). We describe this process in Algorithm 3 in the appendix (lines 166 to 167) which also describes the use of $\\sigma$. In the revised version, we will include the arguments of both Location and Patch optimization processes within Algorithm 1 to improve the clarity of $\\sigma$ and other parameters. \n\n**Q#2:** In our experiments, we have conducted a parameter tuning upon four parameters $li,N,\\sigma$ and $t$ involved in our proposed method (lines 272 to 276). In particular, we used the VGG-16 ImageNet model as the victim (please refer to the ablation study, Section 4.3 of our manuscript).\n\n**Q#3:** We follow the settings used in other works (references [2,12]), as cited in lines 218 to 222."}, {"review_id": "cGxUsjYmzg", "reviewer": "Reviewer_1i2p", "summary": "This paper explores black-box score-based patch attacks on image classification. The previous patch attacks are relatively visible to the naked eye, so the authors hope to have a camouflaged adversarial patch. Therefore, the authors propose a novel method for constructing adversarial patches that approximates the appearance of the area it covers. This camouflaged patch is composed of multiple translucent RGB circles. With the help of evolutionary algorithm optimization, the proposed attack can optimize the shape and location of patches. The proposed method achieves better or comparable performance to state-of-the-art methods and is stealthy.", "questions": "See Weaknesses.", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Camouflaged adversarial patches are a more significant threat to security than normal adversarial patches.\n\n2. The proposed method is visually better imperceptible.", "weaknesses": "1. Experimentation is not sufficient. The authors only performed robustness evaluation on two CNNs, and did not test on more CNNs or updated ViT, which limits the generalization of the method;\nThe authors do not have HPA [1], MPA [1], and Adv-watermark [2], the performance of these methods is unknown in imperceptibility, and they are all score-based patch attacks;\nThe author did not test on patch defense, including LGS [3], DW [4], and PatchGuard [5].\n\n2. The idea of a translucent RGB patch is not very novel in patch attacks and has been discussed in [6] and [7]. The differences between this work and its patch form are not discussed.\n\n3. In the absence of experiments on targeted attacks, we believe that the color prior cannot achieve targeted attacks.\n\n4. There is no curve of attack performance changing with the number of queries, which is of great significance to the analysis of algorithm convergence.\n\n[1] PatchAttack: A Black-Box Texture-Based Attack with Reinforcement Learning, ECCV 2020\n\n[2] Adv-watermark: A Novel Watermark Perturbation for Adversarial Examples, ACM MM 2020\n\n[3] Local gradients smoothing: Defense against localized adversarial attacks, WACV 2019\n\n[4] On visible adversarial perturbations & digital watermarking, CVPRW 2018\n\n[5] PatchGuard: A Provably Robust Defense against Adversarial Patches via Small Receptive Fields and Masking, S&P 2021\n\n[6] Adversarial camera stickers: A physical camera-based attack on deep learning systems, ICML 2019\n\n[7] The Translucent Patch: A Physical and Universal Attack on Object Detectors, CVPR 2021", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W#1:** We address the reviewer's concerns from the following two aspects.\n- In fact, We evaluate the proposed method by conducting adversarial attacks on four, including two adversarial and two non-adversarial, classifiers. \n- In this work we compare the proposed method with the TPA algorithm of PatchAttack (reference [1] provided by the reviewer). We additionally make comparisons with the HPA algorithm used within PatchAttack but refer to it as OPA (reference [16] of our manuscript). We do not make comparisons with the MPA algorithm of PatchAttack due to its similarities with the HPA algorithm as well as their similar performance. Finally, our decision was also made by considering the use of MPA from reference [1] provided by the reviewer as motivation for the development of TPA texture-based attack (subsection 3.3 of [48]).\n- At the request of the reviewer, we show the results of conducted non-targeted attacks on a Transformer based DNN classifier (Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, ICLR'21 ) in **Table 1 of the global 'author rebuttal PDF'**. We additionally amend our experiments to include the Adv-watermark algorithm whilst also attacking a PatchGuard (reviewer reference [5]) defended model, due to it being a more recent defence mechanism.\n\n**W#2:** Despite the use of translucent RGB shapes not being novel, we argue that our method for generating adversarial patches is. Specifically, the work 'adversarial camera stickers' (reference [6] provided by the reviewer) focuses on making physical changes by applying semi-transparent stickers over the camera, resulting in modifications to the entire image. Similarly, The 'translucent patch' (reference [7] provided by the reviewer) applies a similar technique to object detection networks by simulating translucent stickers placed upon the lens of a camera. Differently, our work only modify a small localized region of the image (patch) using translucent RGB circles to construct its pattern. Furthermore, the changes made by [6] and [7] to the target image are clearly visible whereas our primary goal is to minimize the visibility of changes made to the target image. We will revise our original manuscript to include the differences between the works [6, 7] and ours.\n\n**W#3:** As requested by the reviewer, we amend our experiments to include targeted attacks on both adversarial and non-adversarial trained classifiers. The results of these experiments can be found in **Table 2 of the global 'author rebuttal PDF'**. \n\n**W#4:** As requested by the reviewer, we include the performance curve of the proposed and compared algorithms in **Figure 1 of the global 'author rebuttal PDF'**."}, {"review_id": "r8Eiovkf0Z", "reviewer": "Reviewer_sS1s", "summary": "Existing methods often produce clearly visible distortions since they do not consider the visibility of the patch. To address this, the authors propose a novel method for constructing adversarial patches that approximates the appearance of the area it covers. They achieve this by using a set of semi-transparent, RGB-valued circles, drawing inspiration from the computational art community. They utilize an evolutionary strategy to optimize the properties of each shape, and employ a simulated annealing approach to optimize the patch’s location.", "questions": "1. The use of the l_2 distance evaluation index may not be suitable for non-patch attacks, as it primarily focuses on measuring the pixel-level differences between the original and perturbed images. As for the experimental comparison algorithm, it would be valuable to investigate if it includes attacks that introduce overall image disturbance, such as global modifications or image-level perturbations. This would provide a more comprehensive evaluation of the algorithm's robustness against different types of attacks.\n2. The number of images chosen for the ablation experiment is relatively small, consisting of only a hundred images. This limited sample size may impact the reliability and generalizability of the experiment's results, especially if the observed effects are not clearly evident. A larger and more diverse set of images would provide a more robust evaluation of the proposed method.\n3. The observation that random methods yield comparable results to the proposed method in this paper raises questions about the effectiveness and uniqueness of the proposed approach. Further analysis is needed to understand the reasons behind this phenomenon.\nPlease do more experiments to compare the random method with the proposed method and explain why they are similar in attacking.\n", "limitations": "The method presented in this paper can be considered as a heuristic black box attack algorithm that incorporates concealment during the attack process. However, it should be noted that this approach is not entirely novel and has been explored to some extent in previous research.  The authors can refer to the following papers:\n\n [1] Zhong Y, Liu X, Zhai D, et al. Shadows can be dangerous: Stealthy and effective physical-world adversarial attack by natural phenomenon[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 15345-15354. \n\n[2] Jia X, Wei X, Cao X, et al. Adv-watermark: A novel watermark perturbation for adversarial examples[C]//Proceedings of the 28th ACM International Conference on Multimedia. 2020: 1579-1587.\n\n[3] Wei X, Guo Y, Yu J. Adversarial sticker: A stealthy attack method in the physical world[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 45(3): 2711-2725.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The research content of the  paper is meaningful and important\n2. The paper is well written\n3. Experiments are relatively abundant", "weaknesses": "Although the method proposed in this paper offers some practical applications, its level of innovation is relatively limited, resulting in incremental improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W#1:** We argue the innovation of our work lies in its consideration of visibility to adversarial patches as well as its methodology for generating adversarial patches. Specifically, previous works do not consider the visibility of a patch resulting in clear distortions to the image as shown in Figure 1 of our manuscript. We argue that this not only questions the practicality of the attack but also the accuracy of their use for DNN robustness evaluation (lines 92 to 94). To construct the pattern of the patch, many works use previously proposed non-patch attack methods (references [12, 34, 7]) which involve the modification of pixel values either through gradient of heuristic methods. Alternatively, other methods make use of pre-defined textures and color (references [48, 16]). Different to existing methods, we construct the patch using a set of semi-transparent RGB circles, which to our knowledge has not been previously proposed. Our novel method of patch construction not only results in more effective attacks (Table 1 in manuscript, **Tables 1 and 2 of the global 'author rebuttal PDF'**) but also generates adversarial patches with low visibility (Figures 1 and 3). Furthermore, whereas random search has shown to be the go-to approach for patch location optimization (lines 169 to 170), we demonstrate our use of simulated annealing results is a more effective method for location optimization (Table 2 of our manuscript). Finally, as our method minimizes the $l_2$ distance to reduce visibility, our patch construction method can be applied to the non-patch minimum-norm attack scenario.\n \n**Q#1:** We address the reviewer's concern from the following two aspects.\n- The $l_2$ distance has been extensively used to promote or ensure low visibility of adversarial perturbation within non-patch adversarial attacks (lines 27 to 28; references [41, 18, 27, 4, 2]), either by its use for constraining the size of the perturbation (lines 58 to 60) or through its minimization (lines 100 to 102; references [33, 29, 25, 8, 19]).\n- In this work focuses on the generation of adversarial patches. Non-patch attacks consider different assumptions and constraints applied to the perturbations (lines 27 to 29; lines 58 to 61), therefore for the sake of fairness we only compare our method to patch adversarial attacks. This is also reflected in previous works (references [2, 12, 4, 48]) that compare methods with similar assumptions and constraints.\n\n**Q#2:** As requested by the reviewer, we have amended our ablation study to 1,000 correctly classified images from the ImageNet validation set. The results of the top $4$ performing configurations are reported in **Table 3 of the global 'author rebuttal PDF'**. From our experiments, we find the observation is similar to our previous setting.\n\n**Q#3:** We argue the uniqueness of our work lies in its consideration of visibility to adversarial patches as well as its methodology for generating adversarial patches. Specifically, previous works do not consider the visibility of a patch resulting in clear distortions to the image (Figure 1). We argue that this not only questions the practicality of the attack but also the accuracy of their use for DNN robustness evaluation (lines 92 to 94). To construct the pattern of the patch, many works use previously proposed non-patch attack methods (references [12, 34, 7]) which involve the modification of pixel values either through gradient of heuristic methods. Alternatively, other methods make use of pre-defined textures and color (references [48, 16]). Different to existing methods, we construct the patch using a set of semi-transparent RGB circles, which to our knowledge has not been previously proposed. This novel method of patch construction not only results in more effective attacks (Table 1 in our manuscript, as well as **Tables 1 and 2 of the global 'author rebuttal PDF'**) but also generates adversarial patches with low visibility (Figures 1 and 3 of our manuscript). Furthermore, whereas random search has shown to be the go-to approach for patch location optimization (lines 169 to 170), we demonstrate our use of simulated annealing results is a more effective method for location optimization (Table 2 of our manuscript).\n\n**Limitations.** We thank the reviewer for providing these three reference. However, we believe our proposed method is unique from the following three aspects.\n- We focus on generating adversarial patches, whereas the works raised by the reviewer aim to generate physical changes which change the scene of the target image (i.e. changes in shadows [1] or stickers placed on a sign [2]).\n- [1] to [3] do not consider the visibility of their changes, instead their changes reflect plausible realistic changes to the image. In contrast, we aim to reduce the visibility of the adversarial patch that is placed upon the target image.\n- [1] to [3] are constrained by the object chosen to be placed upon the target image (i.e. a shadow, a sticker or watermark). Differently, our proposed method constructs adversarial patches by optimizing the characteristics of a set of semi-transparent circles, allowing the patch to approximate any image it is placed upon."}]

Author rebuttal:
Attached is the manuscript containing the tables and figures of reviewer requested experiments and plots.

Meta-review: The paper proposes a localized adversarial attack approach that creates camouflaged perturbed patches using an evolutionary approach. Initial concerns have been well addressed in the author's response, which leads to all 5 reviewers voting for acceptance. AC concurs and believes the paper will be a valuable result for the community. Authors are strongly suggested to reflect the promised changes in the final version and add clarification where requested.
Decision: Accept (poster)