SlowGuess/NeurIPS_2023_OCR huggingface dataset:
CamoPatch: An Evolutionary Strategy for Generating Camouflaged Adversarial Patches
Phoenix Neale Williams  
Department of Computer Science, University of Exeter  
Exeter, EX4 4RN  
pw384@exeter.ac.uk
Ke Li  
Department of Computer Science  
University of Exeter  
Exeter, EX4 4RN  
k.li@exeter.ac.uk
Deep neural networks (DNNs) have demonstrated vulnerabilities to adversarial examples, which raises concerns about their reliability in safety-critical applications. While the majority of existing methods generate adversarial examples by making small modifications to the entire image, recent research has proposed a practical alternative known as adversarial patches. Adversarial patches have shown to be highly effective in causing DNNs to misclassify by distorting a localized area (patch) of the image. However, existing methods often produce clearly visible distortions since they do not consider the visibility of the patch. To address this, we propose a novel method for constructing adversarial patches that approximates the appearance of the area it covers. We achieve this by using a set of semi-transparent, RGB-valued circles, drawing inspiration from the computational art community. We utilize an evolutionary strategy to optimize the properties of each shape, and employ a simulated annealing approach to optimize the patch's location. Our approach achieves better or comparable performance to state-of-the-art methods on ImageNet DNN classifiers while achieving a lower  $l_{2}$  distance from the original image. By minimizing the visibility of the patch, this work further highlights the vulnerabilities of DNNs to adversarial patches.
Deep neural networks (DNNs) have revolutionized the field of computer vision, demonstrating significant progress in several tasks [38, 51, 53]. Nevertheless, they are not without vulnerabilities. Recent studies highlight a critical weakness: susceptibility to adversarial examples, where subtle, intentionally designed perturbations to input images result in the DNNs misclassification [22, 54, 43]. The existence of these adversarial examples in the physical world poses a significant threat to security-critical applications such as autonomous vehicles and medical imaging [31, 56, 33, 5]. As a result, developing methods to generate adversarial images has emerged as a critical research area for assessing the robustness of DNNs [3].
While initial studies emphasized the creation of adversarial examples with  $l_{p}$ -norm ( $p$  can be 1, 2, or  $\infty$ ) constrained perturbations, current research has shifted towards generating sparse perturbations, which alter only a small portion of the original image [12, 19, 15, 61]. These sparse perturbations have proven to be as effective as traditional  $l_{2}$  or  $l_{\infty}$ -constrained perturbations. Adversarial patches, localized perturbations affecting a small portion of the image, have garnered significant interest as a type of sparse perturbation [47, 66, 33]. Various methods for generating adversarial patches have been proposed for both white-box (where complete information about the model is known) and black-box (where only the input-output pairs are accessible) scenarios [29, 7, 47, 15, 66, 20, 28, 11, 27].
$l_{2} = 0.014$
Proposed method
$l_{2} = 0.01$
Patch-RS
$l_{2} = 0.78$
Figure 1: This illustration shows adversarial images generated by two algorithms, the proposed method and Patch-RS [15], both attacking a conventionally trained (left) and adversarially trained (right) ImageNet classifiers. While both images are adversarial, the adversarial patch generated by the state-of-the-art Patch-RS algorithm visibly distorts the image, whereas the proposed method's adversarial patch remains more similar to the original image. This similarity is demonstrated by calculating the  $l_{2}$  distance between the adversarial patches and the area of the original image they are placed upon.
$l_{2} = 0.69$
However, a significant challenge remains: the unbounded magnitudes of adversarial patches often lead to noticeable distortions in the original image, as depicted in Figure 1.
A contrasting approach to adversarial attacks is embodied by minimum-norm attacks, a class of strategies that generate adversarial examples through minimizing a particular norm of the adversarial perturbation [40, 6, 13, 58]. Due to their ability to measure adversarial accuracy under a range of perturbation budgets, these attacks serve as valuable tools for assessing DNN robustness [45]. However, these methods come with a notable drawback: they rely heavily on numerous DNN queries to substantially decrease the perturbation size. This dependence becomes particularly problematic in black-box scenarios, where the number of queries is often restricted, thus making it more difficult to reduce the perturbation size effectively [26].
Although adversarial patches have proven effective in causing DNN misclassification, existing methods often overlook the necessity of minimizing the visual impact of the patch on the original image. This oversight leads to patches that are easily detectable. To address this shortcoming, we introduce a novel method for generating adversarial patches. Our approach utilizes semi-transparent circles with RGB values, which blend into the original image. We employ simulated annealing for optimal location selection and implement an evolutionary strategy to fine-tune each circle's properties. The goal of our method is to induce the desired misclassification while minimizing the  $l_{2}$  distance between the patch and the original image, thereby camouflaging the patch effectively.
The rest of this paper is organized as follows. Section 2 overviews some related works, underscoring their contributions and limitations. In Section 3, we outline our proposed attack scenario and provide an in-depth explanation of our method's implementation. Empirical results are presented and analyzed in Section 4. Section 5 concludes this paper and sheds some light on future directions.
Adversarial attacks on DNNs have been one of most active fields in the machine learning community. A common strategy in these attacks is to create imperceptible perturbations, often achieved by constraining the perturbations using the  $l_{p}$ -norm. These perturbations are typically generated differently depending on the access level to the targeted DNN's information. In a white-box scenario, where the attacker has full access to the DNN's details, approaches often leverage the gradient information of the DNN's loss function. This gradient is used within an optimization algorithm to generate adversarial perturbations [54, 22, 39]. On the other hand, in a black-box scenario, where the attacker's access is limited to the DNN's output probabilities, various attack methods estimate the loss function's gradient and use it within a gradient-based optimization technique [26, 4, 59]. Some researchers have also proposed heuristic methods for the black-box scenario that do not rely on gradient information [1, 2, 27, 28, 11].
Our work fits into this landscape by focusing on the black-box scenario. However, unlike many existing methods, we aim to create adversarial patches that are not only effective but also visually blend into the original image.
2.1 Adversarial Patches
Adversarial patches, designed to cause image misclassification, represent a unique approach to adversarial attacks. These patches are typically small, visible, and often square-shaped, strategically applied to the targeted image [15, 47, 66]. The pioneering work by Brown et al. introduced the concept of universal adversarial patches, which cause misclassification when applied to a variety of images. Using the gradient information of the DNN, they utilized stochastic gradient descent to optimize the patch pattern, which was subsequently superimposed onto a set of target images at predetermined locations. Following this, Karmon et al. proposed LaVAN that also generates universal patches. However, they used random search to optimize the patch location. In black-box scenarios, Brown et al. produced universal adversarial patches by executing white-box attacks on four DNN classifiers and transferring the results to an unseen classifier. Croce et al. proposed the Patch-RS method, which generates adversarial patches by minimizing the aggregated loss of a set of images, using random search for patch location optimization and the Square-Attack method of Andriushchenko et al. for patch pattern optimization.
In contrast to universal adversarial patch approaches, other researchers have focused on generating image-specific patches. Fawzi and Frossard created patches for individual images by optimizing the position and shape of rectangular patches with a predefined monochrome pattern. Built upon the LaVAN concept, Rao et al. proposed alternative techniques for patch location optimization using random search variants. Yang et al. and Croce et al. further advanced the image-specific scenario, with the former using reinforcement learning for patch generation and the latter applying the Patch-RS method to minimize the loss of a single image.
Despite recent advancements in creating both universal and image-specific adversarial patches, the glaring distortions from significant modifications to input images raise practical concerns. This issue also impacts the accurate assessment of DNN robustness against adversarial patches.
2.2 Minimum-Norm Attacks
Minimum-norm attacks diverge from the traditional adversarial attacks by focusing on finding the smallest perturbation that can lead to misclassification under a specific norm. These attacks offer a more comprehensive assessment of DNN robustness [45]. Although white-box attacks have made significant progress in enhancing the query efficiency of minimum-norm attacks [45, 41, 48], black-box attacks still demand a substantial query budget to achieve effectiveness. The ZOO algorithm [8] constructs the problem as an aggregated function of the perturbations  $l_{2}$ -norm and weighted loss function. Estimating its gradient using finite-differences, the authors make use of coordinate descent to minimize the formulated problem. Tu et al. addressed the query inefficiency of ZOO by reducing the size of the perturbation using a trained Auto-Encoder DNN. Ilyas et al. remove the need for estimating coordinate-specific gradients by making use of natural evolutionary strategies, reducing the  $l_{\infty}$ -norm of the perturbation by iteratively maximizing the adversarial criterion within a given norm constraint, then reducing the norm. Despite the efficiency improvement of gradient estimation, existing black-box methods still require large query budgets. The SimBA method of [23] incrementally adds scaled orthogonal vectors to the current perturbation, increasing its  $l_{2}$ -norm, but is unable to reduce the  $l_{2}$ -norm of the perturbation once the desired misclassification has been achieved.
Therefore, while recent works have achieved large performance gains within the white-box scenario, black-box methods suffer from query inefficiency which restricts their applicability to real-world scenarios, particularly when the query budget is limited.
2.3 Evolutionary Strategies for Adversarial Attacks
Many existing studies employ evolutionary strategies (ES) for non-patch-based adversarial attacks. In the black-box scenario, ES has gained popularity due to its independence from gradient information. Notable examples include the works of [1, 46, 36, 63], who utilize evolutionary algorithms to create  $l_{\infty}$  constrained adversarial perturbations. For conducting sparse adversarial images Williams and Li
Algorithm 1: Evolutionary Strategy for Generating Disguised Adversarial Patches (CamoPatch)
Input: Margin loss  $\mathcal{L}$  input  $\mathbf{x} \in \mathcal{X} \subseteq [0,1]^{h \times w \times 3}$ , query budget  $K$ , sparsity  $\epsilon$ , initial temperature  $t$ , number of circles  $N$ , location schedule  $li$ , evolutionary step-size  $\sigma$   
1  $s \gets \sqrt{\epsilon}$  // Patch Side Length  
2  $\delta \gets$  InitialPatch(N,s)  
3  $i \sim \mathcal{U}(\{0,\dots ,w - s\})$   
4  $j \sim \mathcal{U}(\{0,\dots ,h - s\})$   
5  $\mathbf{x}^* \gets \mathbf{x}$   
6  $\mathbf{x}_{i:i+s,j:j+s}^* \gets \delta$  // Apply patch  
7  $L \gets \mathcal{L}(\mathbf{x}^*)$   
8 norm  $\gets ||\mathbf{x}_{i:i+s,j:j+s} - \boldsymbol{\delta}||_2$   
9 for  $k \gets 1; k < K; k \gets k + 1$  do  
10 if mod(k, li + 1) = 0 then  
11  $\begin{bmatrix} i, j, L, & \text{norm} \leftarrow \text{LocationUpdate}() / / \text{see Algorithm 4} \end{bmatrix}$   
12 else  
13  $\delta, L, \text{norm} \leftarrow \text{PatchUpdate}() / / \text{see Algorithm 3}$   
14 return  $\mathbf{x}^*$
make use of a multi-objective evolutionary algorithm to minimize both the number of modified pixels and magnitude of the pixels modification. ESs have also been used to construct adversarial examples within other domains such as natural language processing [68, 67].
The use of evolutionary algorithms has also been explored for constructing adversarial patches. Chen et al. addressed the more-limited decision-only setting (where only the predicted label is accessible to the attacker) by placing localised regions of an image from the target class onto the attacked image. Under the constraint of the patch causing misclassification, the authors optimised the coordinates of the patch by using an adapted differential evolution algorithm to minimise the patch's  $l_{0}$  norm.
In essence, our method strives to generate adversarial patches that seamlessly blend into the targeted image by modeling the superimposed area with semi-transparent, RGB-valued circles. We adopt an approach akin to existing works that generate adversarial patches by iteratively optimizing both the patch and its position on the image. The balance between these steps is managed by a location schedule parameter,  $li$ . In this section, we start by defining the problem formulation, followed by a detailed description of our proposed method. The overarching structure of our method is summarized in Algorithm 1.
Consider a trained DNN image classifier  $f: \mathcal{X} \subseteq [0,1]^{h \times w \times 3} \to \mathbb{R}^P$  which takes a benign RGB image  $\mathbf{x} \in \mathcal{X}$  of height  $h$  and width  $w$  and outputs a label  $y = \operatorname*{argmax}_{p \in \{1, \dots, P\}} f_p(\mathbf{x})$ , with  $P$  representing
the total number of class labels. A non-targeted attack seeks a perturbation  $\delta$  satisfying:
$$
\operatorname {a r g m a x} f _ {p} (\mathbf {x} + \boldsymbol {\delta}) = y _ {q}, \tag {1}
$$
$$
p \in \{1, \dots , P \}
$$
where  $y$  is the original class label for  $\mathbf{x}$  and  $y_{q} = \underset {q\neq y}{\mathrm{argmax}}f_{p}(\mathbf{x})$  is a label corresponding to a class other than the true class  $y$ . For targeted attacks  $y_{q}$  is assigned a target label  $y_{t}$ , where  $y_{t}\neq y$ . In the adversarial patch scenario, the number of modified pixels is limited to maintain the semantic content of the image. Hence, the problem is cast as:
$$
\underset {\delta} {\text {m i n i m i z e}} \quad \mathcal {L} (f; \mathbf {x} + \delta , y _ {q}) \tag {2}
$$
$$
\text {s u b j e c t} \quad | | \boldsymbol {\delta} | | _ {0} \leq \epsilon , 0 \leq \mathbf {x} + \boldsymbol {\delta} \leq 1,
$$
where minimizing the loss function  $\mathcal{L}$  yields the desired adversarial image.
Most existing algorithms solve (2) by fixing the number of disturbed pixels to a constant value  $\epsilon$ , allowing unbounded modifications [15, 29, 7]. Unlike these, our proposed method aims to create adversarial patches that closely resemble the area of the original image they overlay. We approach this as a constrained optimization problem, akin to the minimum-norm setting [45, 48, 58]. Thus, our objective is to generate a  $\delta$  that solves the subsequent optimization problem:
$$
\begin{array}{l} \underset {\delta} {\text {m i n i m i z e}} \quad | | \mathbf {x} - \boldsymbol {\delta} | | _ {2} \\ \begin{array}{l l} \text {s u b j e c t t o} & | | \boldsymbol {\delta} | | _ {0} \leq \epsilon , \\ & \mathcal {L} (f; \mathbf {x} + \boldsymbol {\delta}, y _ {q}) <   0, \end{array} \tag {3} \\ 0 \leq \mathbf {x} + \boldsymbol {\delta} \leq \bar {1}, \\ \end{array}
$$
where the patch is a square with a side length of  $\sqrt{\epsilon}$ . We ensure that the value of the loss function  $\mathcal{L}(\cdot)$  is negative when  $\mathbf{x} + \boldsymbol{\delta}$  results in misclassification. This is achieved by defining the loss in the constraint as the margin loss:
$$
\mathcal {L} (f; \mathbf {x} + \boldsymbol {\delta}, y _ {q}) = f _ {y} - f _ {y _ {q}}, \tag {4}
$$
for non-targeted attacks and the cross-entropy loss:
$$
\mathcal {L} (f;., \mathbf {x} + \vec {\delta}, y _ {t}) = - f _ {y _ {t}} + \log \left(\sum_ {p = 1} ^ {P} e ^ {f _ {p}}\right) \tag {5}
$$
for targeted attacks.
3.2 Patch Initialization
We construct an adversarial patch by overlaying  $N$  circles on a black image (see Figure 2), inspired by evolutionary strategies prevalent in computational art [32, 21, 57]. They aim to approximate images using semi-transparent, RGB-valued shapes. Circular shapes, due to their fewer adjustable properties, are a popular choice. Furthermore, the use of semi-transparent circular shapes have also been used by Li et al. and Zolfi et al. to construct adversarial examples. Whereas this work constructs adversarial patches, Li et al. and Zolfi et al. simulate stickers placed over a camera, modifying the entire image.
The adversarial patch  $\delta$  is represented as the concatenation of  $N$  shapes:
$$
\boldsymbol {\delta} = \boldsymbol {\delta} ^ {1} \oplus \boldsymbol {\delta} ^ {2} \dots \oplus \boldsymbol {\delta} ^ {N}, \tag {6}
$$
where  $\oplus$  denotes the concatenation operator. Each shape  $\delta^a$ , where  $a \in \{1, \dots, N\}$ , is represented by a vector comprised of seven elements including the center's coordinates  $(c_1^a, c_2^a)$ , the radius  $r^a$ , the RGB values  $(R^a, G^a, B^a)$ , and the shape's transparency  $T^a$ . These elements, normalized to continuous values between 0 and 1, are initially randomly sampled from a uniform distribution  $\delta^a \sim \mathcal{U}(0,1)$ . The initial location of a patch is also randomly and uniformly sampled from the available pixel locations.
3.3 Patch Optimization
In this paper, we employ a single solution evolutionary strategy, known as  $(1 + 1)$ -ES, to modify the properties of each shape  $\delta^a$ . This approach has proven to be efficient for approximating images [42]. To adjust the properties, we sample values from a normal distribution  $\sigma \cdot \mathcal{N}(0,I)$ , where  $\sigma$  is a tunable parameter controlling the trade-off between exploration (i.e., searching new areas in the solution space) and exploitation (i.e., refining the current solution). A larger  $\sigma$  promotes exploration, while a smaller one favors exploitation. We then use the updated perturbation  $\delta^{*}$  to construct an adversarial image  $\mathbf{x}^{**}$ . The solution that satisfies the constraint as per (3) is retained. If both solutions meet this constraint, we opt for the one with a smaller  $l_{2}$  distance from the original image. The patch update method is detailed in Algorithm 3 in the supplementary document.
3.4 Location Optimization
Addressing the discrete nature of pixel locations, many existing methods have employed random search to optimize the position of the patch within the image [15, 29, 47]. However, random search methods often falter when encountering local optima. To mitigate this, Skiscim and Golden introduced simulated annealing, a method that probabilistically accepts worse solutions based on the search temperature and the performance difference between current and new solutions. This approach promotes exploration of the search space in the early stages of optimization and gradually becomes more selective, favoring solutions with better quality in the later stages.
In our work, we leverage the fast simulated an
Figure 2: This illustration shows an adversarial image (left) with the adversarial patch outlined, and the magnified patch (right) for better visibility. This patch is generated by the proposed method using  $N = 100$  overlapping circular shapes.
nealing approach proposed by Szu and Hartley to optimize the location of a patch. During each iteration  $k$ , we uniformly sample a single location (denoted as  $(i^{*}, j^{*})$ ) from the location space. Then, we apply the patch to the new location on the input image  $\mathbf{x}$ , and construct an updated adversarial image  $\mathbf{x}^{**}$ . The new solution  $\mathbf{x}^{**}$  is then evaluated using the loss function  $\mathcal{L}$ . If both  $\mathbf{x}^{*}$  and  $\mathbf{x}^{**}$  satisfy the loss  $\mathcal{L}$  constraint as per (3), we retain the solution with the lowest  $l_{2}$ -norm from the original image. Otherwise, simulated annealing is employed to probabilistically decide the acceptance of the new solution. Specifically, the acceptance probability is defined as  $\exp(-d / t_{\text{curr}})$ , where  $d$  is the loss difference between the current and new solution, and  $t_{\text{curr}} = t / k$  follows an exponentially decreasing schedule. This formulation ensures better solutions are always selected, while solutions with relatively poor quality are more likely to be accepted in the early search stages for enhanced exploration. The parameter  $t$  is predefined, with larger values promoting exploration during a longer portion of the attack process. The detailed location update method can be found in Algorithm 4 in the supplementary document.
4 Empirical Study
In this section, we empirically evaluate our proposed method's effectiveness by attacking classifiers trained on the ImageNet dataset [16]. The experimental setup is outlined in Section 4.1, followed by a comparative analysis with state-of-the-art adversarial patch methods, including Patch-RS [15], TPA [66], OPA [20], Adv-Watermark [27] and a black-box adaptation of LOAP [47] in Section 4.2. Last but not the least, Section 4.3 offers an ablation study that scrutinizes the significance of various components and parameters within our proposed method.
Dataset and Classifiers Settings: For our experiments, we follow a similar setup to preceding works, conducting non-targeted and targeted attacks on DNN classifiers trained on the ImageNet dataset. We specifically target three adversarially trained and defended classifiers, namely AT-ResNet-50 [49], AT-WideResNet-50-2 [49] and PatchGuard [65], along with three conventionally trained classifiers, VGG-16 [51], ResNet-50 [24] and ViT-B/16 [17]. A subset of 1,000 images, correctly classified by each classifier from the ImageNet validation set, is chosen and resized to dimensions of  $224 \times 224 \times 3$ . For targeted attacks we randomly select  $y_{t}$  for each image ensuring it is different from the images true label  $y$ . The adversarially trained and defended classifiers are implemented using the RobustBench library [14] and authors original implementations, respectively, while the three conventional classifiers are derived from their pre-trained versions available in the PyTorch library [44]. All experiments were carried out on a system with an NVIDIA GeForce RTX 2080Ti GPU.
Parameter Settings: To select the value of  $\epsilon$ , we follow the approach of Croce et al., setting  $\epsilon = 1600$ . This corresponds to a patch size of  $40 \times 40$ , which constitutes roughly  $3.2\%$  of the total pixel count. We assign a budget of 10,000 queries for each attack. As discussed in Section 3, our proposed method entails four free parameters:  $\sigma$ , lit,  $t$ , and  $N$ . For these parameters, we set  $\sigma = 0.1$ ,
Tree frog  $\rightarrow$  Grasshopper