IMPACT: Irregular Multi-Patch Adversarial Composition Based on Two‑Phase Optimization
Authors: Zenghui Yang
Xingquan Zuo
Hai Huang
Gang Chen
Xinchao Zhao
Tianle Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Adversarial Patch Attack
Evolutionary Algorithms
Irregular Shapes
Multi-Patch Attack
Black-Box Attack
Primary area: optimization

Abstract: Deep neural networks have become foundational in various applications but remain vulnerable to adversarial patch attacks. Crafting effective adversarial patches is inherently challenging due to the combinatorial complexity involved in jointly optimizing critical factors such as patch shape, location, number, and content. Existing approaches often simplify this optimization by addressing each factor independently, which limits their effectiveness. To tackle this significant challenge, we introduce a novel and flexible adversarial attack framework termed IMPACT (Irregular Multi-Patch Adversarial Composition based on Two-phase optimization). IMPACT uniquely enables comprehensive optimization of all essential patch factors using gradient-free methods. Specifically, we propose a novel dimensionality reduction encoding scheme that substantially lowers computational complexity while preserving expressive power. Leveraging this encoding, we further develop a two-phase optimization framework: phase 1 employs differential evolution for joint optimization of patch mask and content, while phase 2 refines patch content using an evolutionary strategy for enhanced precision. Additionally, we introduce a new aggregation algorithm explicitly designed to produce contiguous, irregular patches by merging localized regions, ensuring physical applicability. Extensive experiments demonstrate that our method significantly outperforms several state-of-the-art approaches, highlighting the critical benefit of jointly optimizing all patch factors in adversarial patch attacks.

IMPACT: Irregular Multi-Patch Adversarial Composition Based on Two-Phase Optimization
Zenghui Yang",, Xingquan Zuo
Hai Huang Gang Chen', Xinchao Zhao', Tianle Zhang
Shool of Cyberspace Security, Beijing University of Posts and Telecommunications
School of Computer Science, Beijing University of Posts and Telecommunications
School of Engineering and Computer Science, Victoria University of Wellington
Schooo of Science, Beijing University of Posts and Telecommunications Key Laboratory of Trustworthy Distributed Computing and Service Lyangzh, zuoxq, zuoxq,uuuang h.hahgg ahhuang,zhaoxe zhaoxc, wwwehanggeabiitt @bupt. edu. en aaron. chengecs vuw. ac.nz
Abstract
Deep neural networks have become foundational in various applications but remain vulnerable to adversarial patch attacks. Crafting effective adversarial patches is inherently challenging due to the combinatorial complexity involved in jointly optimizing critical factors such as patch shape, location, number, and content. Existing approaches often simplify this optimization by addressing each factor independently, which limits their effectiveness. To tackle this significant challenge, we introduce novel and flexible adversarial attack framework termed IMPACT (Irregular Multi-Patch Adversarial Composition based on Two-phase optimization). IMPACT uniquely enables comprehensive optimization all essential patch fac- tors using gradient-free methods. Specifically, we propose novel dimensionality reduction encoding scheme that substantially lowers computational complexity while preserving expressive power. Leveraging this encoding, further develop a two-phase optimization framework: phase employs differential evolution for joint optimization of patch mask and content, while phase refines patch content using an evolutionary strategy for enhanced precision. Additionally, we introduce new aggregation algorithm explicitly designed produce contiguous, irregular patches by merging localized regions, ensuring physical applicability. Exten- sive experiments demonstrate that our method significantly outperforms several state-of-the-arr approaches, highlighting the critical benefit of jointly optimizing all patch factors in adversarial patch attacks. Our source code is available https:/lyangzh22 w.w.hh216...tte http://yyagga22lllgggbbb/////PP github.io//MPPAC..
Introduction
Deep Neural Networks (DNNs) have become one of the core technologies modern artificial intelli- gence. With their exceptional learning ability, DNNs have demonstrated outstanding performance in fields such as image classification (21) object detection (3), and natural language processing (34), driving the rapid advancement of numerous real-world applications. However, recent studies demon- strate a concerning vulnerability: DNNs are susceptible to adversarial attacks [35], where minor, specifically designed perturbations input images significantly degrade their performance [4] While early adversarial attack methods primarily relied on global perturbations [35] constrained by & fo norms and sparse perturbations 7
g constrained by Eo norm, recent approaches have
corresponding authors: zuoxq @ bupt.edu.cn
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

shifted focus towards localized perturbations, known as adversarial patches [12 42 16] more visually conspieuous, they have greater practical applicability and effectiveness
While being 25
Adversarial patches, characterized by distinct shapes, locations, number, and content, significantly complicate the attack optimization process due to their high-dimensional and combinational nature. Previous studies typically simplify this complex problem by independently optimizing patch con- tent [20] or location [28, 41), often assuming fixed shapes and limiting patches to single or basic geometric forms 45 27 This simplification, however, considerably restricts the adversarial potential and practicality of these attacks, leaving open a critical research gap: the joint optimization of multiple essential patch factors, including shapes, locations, number, and content.
Motivated by this critical challenge, we propose IMPACT (Irregular Multi-Patcl Adversarial Com- position Based on Two-Phase Optimization), a flexible and novel adversarial attack framework specifically designed to support comprehensive optimization of multiple adversarial patch factors. The IMPACT framework is inherently versatile, allowing the integration of any gradient-free opti- mization algorithms tailored to specific adversarial objectives or constraints. To demonstrate the effectiveness and practicality of IMPACT, we present one concrete implementation leveraging evolu- tionary algorithms (EAs), renowned for their efficacy in black-box optimization tasks lacking gradient information. Our EA-based implementation of IMPACT introduces a two-phase optimization scheme:
This phase jointly optimizes patch masks (defining shape and location) and patch content. To overcome the curse of dimensionality, we introduce a novel dimensionality reduction encoding scheme, reducing computational complexity while maintaining solution expres- siveness. Additionally, we develop a random aggregation algorithm that plays a crucial role in
Joint Optimization Phase:
generating practical adversarial patches. Unlike methods relying on fixed shapes 43 36 27
algorithm produces diverse, irregular patch geometries by merging local regions. This ensures each patch is locally well formed, making them suitable for physical application. The irregularity further enhances their adversarial potential. This phase is implemented with differential evolution (DE),
This phase precisely refines patch content at the pixel level, transitioning from global exploration in phase to targeted local exploitation, thus further optimizing attack effectivenesss. In our implementation, this phase uses (1+1)-ES to balance computational cost and attack efficacy.
Refinement Phase:
Phase One
Phase Two
White-Box
Figure I:
Adversarial patches by IMPACT. Phase One: Dp-omiiiized block content. Phase Two: (1+1)-ES refined pixel content. White-Box: IMPACT mask with gradient-optimized content. Fxam- ples show diverse shapes, locations, and content.
Extensive experiments conducted on widely-used benchmark models (ResNet50 [17]. VGG16 [32], ViT-B (T) confirm that IMPACT outperforms multiple state-of-the-art patch attack methods in black-box scenarios. Our results underline the importance of jointly considering all key patch factors, demonstrating that IMPACT can significantly improve adversarial effectiveness. Figure F presents examples of adversarial patches generated by IMPACT. These patches exhibit locally coherent structures, making them physically printable and suitable for real-worke deployment. Their irregular, optimized designs further enhance adversarial effectiveness by increasing the perturbation diversity and expressiveness. Our main contributions are summarized as follows:
We introduce IMPACT, a flexible framework for adversarial patch attacks that allows seamless integration of diverse madieent-free optimization algorithms. The presented EA-

---

based implementation showcases the framework's 's capability for joint optimization of all critical adversarial patch factors.
In line with the IMPACT framework, we propose novel dimensionality reduction encoding scheme, simplifying the high-dimensional solution space while preserving the solution's s diversity and quality. We further develop new aggregation algorithm that merges scattered patch elements into locally coherent and irregular shapes, enabling physically printable adversarial patches.
Through comprehensive evaluations against sate-of-theear methods, we demonstrate the superior effectiveness and efficiency of our IMPACT implementation. This systematic analysis provides valuable insights into how jointly optimizing multiple patch factors can substantially enhance adversarial attack success rates.
2
Related Work
This section provides an overview of two related research directions: Adversarial patch attacks, which craft localized, visible perturbations, and EA-based methods for black-box adversarial attacks.
2.1 Adversarial Patch Attacks
Depending on the level of knowledge attacker has about the target model, adversarial patches can be categorized into white-box attacks and black-box attacks.
For the white-box attack, most studies focus primarily on optimizing patch content through gradient- based techniques. Brown et al. [2] proposed a universal adversarial patch for real-world targeted attacks. Karmon et al. (20) introduced LaVAN for generating visible localized noise patches. Their work focused on adversarial attacks in the digital domain, where modifications are made directly to the pixel values of digital images in a dataset. This approach enabled successful attacks using significantly smaller visible perturbations. Unlike earlier studies on universal patches designed to work across various images, Rao et al. (28] showed that image-specific patches, optimized for individual images, offer more powerful alternative by leveraging the unique characteristics of each image. They emphasized the importance of patch location, proposing an optimization algorithm to determine the most effective location for the patch. Chen al. (S] recognized that the patch shape is an equally critical factor. Building on this insight, they proposed the deformable patch attack. However, their approach was limited to generating a single patch. the context of multi multi-patch attacks, Fu et al. [14] introduced PATCH-FOOL, which leverages attention-awar patch selection mechanism to generate multiple patches simultaneously. Sharma et al. 31 further explored multi- patch attacks, demonstrating the advantages of using multiple patches over single-patch attacks. In addition, Huang et al. [19] proposed a mooli-mmini-patee adversarial attack for remote sensing image classification. However, these multi-patch schemes are restricted to rectangular shaped patches.
For the black-box attack, random search is the dominating approach for optimizing patch content. Fawzi et al. [13] were among the first to explore patch-based black-box attacks. Their method can generate rectangular, monochromatic patches with optimized shape and placement. Unfortunately. the attack's 'ss effectiveness was limited due to the simplicity of the patch content. To address this issue, Yang et [45) proposed TPA, where reinforcement learning was employed to optimize both the position and texture parameters of each patch. Croce et al. [8] introduced robust adversarial attack framework Patch-RS based on random scarch. However, the patches were limited to fixed square shapes. To investigate the impact of different shapes, Ran et al. (27] proposed cross-shaped adversarial patch, consisting of two intersecting line segments extending toward the corners of the input image. Using random search to optimize position and content, the method achieved high attack success rate. However, the global perturbation structure caused the line segments to become excessively thin, rendering it challenging apply in the physical world.
In summary, existing adversarial patch attack methods predominantly focused on optimizing one or two factors, such as patch shape, location, or content, while neglecting comprehensive, joint optimization of all critical factors. This narrow focus restricts their capacity to fully leverage the optimization space, highlighting the potential for more comprehensive and effective approaches.
3

---

2.2 Adversarial Attacks Based on EAs
In the field of adversarial attacks, EAs have demonstrated its unique strength of effectively optimizing arbitrary target models without using gradient information [TO 43 36 22 Several studies 26 [2
have investigated the application of EA to generate adversarial examples under Ez and loo constraints. have focused on sparse adversarial attacks under to constraints.
Other works 33 38 37
Very recently, EAs have been increasingly used generate adversarial patches; however, existing encoding schemes in these approaches do not support full optimization across all patch factors. For example, Williams et al. 45] introduced CamoPatch as an EA approach generate camouflaged adversarial patches. Unlike traditional adversarial patches, this approach focuses reducing the patch's 's visibility. Hu deployable in the physical world. However, due to the limitations of their encoding scheme, their method only supports rectangular patch shapes and monochromatic patch content. Tang et al. [36] introduced dimensionality reduction strategy focused on patch content, leveraging duplicating and tiling to upscale decision variables from low-dimensional space to a higher-dimensionall space. While this approach enhances the optimization process for patch content, their encoding scheme has notable limitations. Specifically, it only addresses patch content optimization while ignoring the patch mask, which remains in fixed square shapes.
[8] proposed AdvIB, leveraging DE create adversarial patches
Overall, most existing methods remain focused on designing encoding schemes to optimize patch content, while a comprehensive approach that integrates patch shape, location, number, and content into the encoding process is still lacking. This gap motivated us develop a novel encoding scheme that supports joint optimization across all these critical patch factors.
3 Proposed Method
We present IMPACT, framework that simultancously optimizes both the patch mask and its content. Specifically, the IMPACT framework comprises two optimization phases. In phase 1, IMPACT operates block-level to simultaneously optimize the patch mask and the patch content. This phase aims for broad exploration of the solution space to identify promising patch configurations. Following phase 1, phase 2 focuses meticulous, pixel-level optimization of the patch content. In this section, we first formulate the problem of adversarial patch attacks. Subsequently, we present the detailed EA-based implemennation of IMPACT. Algorithm Secoritmmouutlneeee the overall procedure of using EA-based IMPACT for black-box adversarial patch attacks. Descriptions the key functions within Algorithm are provided in Appendix A
Algorithm Irregular Multi-Patch Adversarial Attack Based on Two-Phase Optimization
Input:
Model f. original example true label num-
14:
break
ber of thin--patches n, number of patches k pop- ulation size N, DE iterations Ta, (1+1)-ES itera-
15:
end if
16:
end for
tions Te
Output: Adversarial example & // Phase 1: DE for Joint Optimization
// Phase 2: (1+1)-ES for Content Refinement
17: for L == 1,..., .Te do
1: Initialize population Po +- PopInit(n,, k N)
18:
successful = True then break
Compute initial fitness Fo <- (Fone f.
19:
3: for 1,.... Tu do
20:
end if
4: 1/2 + Mutation( P:-1)
21:
Add Gaussian noise dunise: 8' = 5 Snoise
5:
U, + Crossover (P--,, Vi)
22:
Generate &' using (8', M)
6:
U, Aggregation(Ue, k))
23:
fitness(#') fitness(E) then
7:
F, + fU,,
24:
Update 8' f end if
8:
Pr, F. + Selection( Pt-1, F Ur, F.))
25:
9:
p' < SelectBest (?,, FF)
26: if f() * then
10:
Construct patches (8, MM) BuildPatch(p")
27:
successful True
Generate è using (8, M) according Eq.
28: end
12:
if f() # y then
29: end for
13:
successful = True
30: return Adversarial example à

---

3.1 Problem Formulation
Given an original example x Rexhxw, the objective of adversarial patch attacks is create an adversarial example λ Rex that can mislead the model into making incorrect predictions [35]. Here, c, h, and w correspond the number of channels, height, and width of the example, respectively. An adversarial patch consists of two components: a mask M E {0, 1 1jexhxu, which determines the shape and location of the patch, and a perturbation E RexAxw, which defines the patch content. By combining x, M, and 6, the adversarial example & can be defined as follows:
i @ (1 M) + 3 M,
(1)
where @ is the element-wise Hadamard product. As a result, the perturbation is applied in regions with Mi I, while regions with M,j = 0 retain the original image content.
Performing adversarial patch attacks requires solving the following optimization problem 27
arg min [(f(2 o - M) + & M). M.I. <, 5.M
where f denotes the image classification model, C is its loss function, and || llo c imposes an ly-norm constraint to limit the patch area. For the untargeted attack, ÿ can be any label other than the original label y. For the targeted attack, 9 is
the target label y/c-
The problem defined Equation 2 is a joint optimization problem where M and jointly define the optimization solution space. Many existing adversarial patch attack methods simplify the problem by decoupling these two components (20 42) Methods that pursue a truly joint optimization of both mask and perturbation remain conspicuously absent.
3.2 Phase 1: Joint Optimization
Phase of IMPACT tackles the challenging problem of jointly optimizing the patch mask and content, which defines a complex and high-ding high-dimensional search space [36]. Direct pixel-level optimization is computationally prohibitive due to the sheer number of variables involved. Additionally, the binary nature of the mask often leads to fragmented, incoherent shapes that not physically realizable 42) To overcome these challenges, we introduce two key innovations: a dimensionality reduction encoding scheme to compress the solution space, and new aggregation algorithm to ensure locally coherent and contiguous patch shapes. Within this framework, we employ DE as the core optimization engine, leveraging its proven effectiveness in black-box adversarial settings for efficiently exploring complex solution spaces 36 6].
3.2.1 Dimensionality Reduction Encoding
We develop a novel dimensionality reduction method that significantly reduces the encoding length while addressing limitations in existing approaches. Unlike traditional methods 36), our encoding method supports the first time joint optimization of all critical patch factors. Such individual representation not only enhances optimization efficiency but also enables a more comprehensive and effective exploration of the adversarial patch design space.
In our framework, the first phase of optimization utilizes DE, which is oppulation-based algorithm. This means it maintains a population, which is a set of candidate solutions. We pt to repre- sent the i-th individual in the population, where i [1, and N denotes the population size. Each individual p; in the population represents candidate solution that encodes all the neces- sary information to generate the patches. Con- cretely, each individual p; is encoded into two parts: (m,, r,). The first component m; represents the mask M, and the second part r; represents the perturbation 8. The binary array m = [b,, b2, ...; br] used to encode the mask M. where b; {0, 1} and / is the encoding length of
D
0
0
0
8
0
0
0
tiling
[0.1000..00....3]]
Figure 2: Example of mask encoding. A 9- element binary array 1,, 0, can represent a 12 x 12 mask.
5

---

m, depending on the size h x w of the original mask. When we set = (h/4) (w/4), each element in m corresponds to a 4 block in the mask M. We can use method called "4 x 4 tillng" to reconstruct m back to the original size of the mask. An example of this encoding method is shown in Figure 2 This method significantly compresses the optimization search space and reduces its complexity. For the 224 x 224 images in the ImageNet dataset, the mask M can be reduced to a size of 56 x 56. For the encoding of the perturbation é, we use a three-channel matrix r E R where is the number of one-valued elements in and R. is constrained to [0, 255]. This design exploits the fact that the perturbation applied only at positions where Mjj = 1 according to Equation (I. Hence optimization is restricted to those specific pixel locations. Notably, each element in m represents a 4 x 4 mini mini-patch, and each three-channel pixel in r encodes the color information of the corresponding x 4 patch.
3.2.2 Random Aggregation
After defining our encoding scheme, the resulting encoded mask m E {0, 1}/ will contain n active elements. However, these elements may be spatially dispersed. To address this issue, we propose new aggregation algorithm to transform m into a new mask in where these n active elements are consolidated into k locally connected, and irregular patches. The algorithm proceeds follows:
The ID input mask encoding m {0, 13 is first reshaped into 2D binary matrix M' {0, 13 vivi for spatial processing. The set of coordinates of the n active elements in M' is extracted:
X' (p) (Iz, My) M [p,]
1,j n..
These coordinates X' are then partitioned into k clusters using the K-Means algorithm:
C = {C1, C2. C+) KMeans (X, k).
(4)
The aggregation process aims to form a single connected shape using the elements assigned to C. For each cluster, we randomly select point within the cluster as the aggregation center. For each point in a cluster, we select a target point uniformly at random from the aggregation center's 8-neighborhood and move the point toward that target point, repeating this process until the point becomes adjacent to the existing connected component. During each move, we randomly choose whether to prioritize a horizontal vertical move. The randomness in selecting aggregation center, the target point, and the order of move attempts contributes to the diversity of the resulting aggregated shapes. A detailed description of the random aggregation algorithm (Algorithm ( along with visual explanation can be
found in Appendix
B
Powered by the newly developed encoding scheme and the random aggregation algorithm, we further adopt DE to optimize the patch mask and content in phase The detailed design of our DE algorithm is provided in Appendix C
3.3 Phase 2: Content Refinement
Phase 2 aims to further improve attack effectiveness through pixel-level refinement of the patch content, using the mask and initial content established in Phase 1. This refinement is particularly important when phase 1 does not yield a successful attack. The solution candidate in this phase is represented by the full-resolution perturbation 8, which has dimensions 3 x h. x w, matching the input image resolution. We employ the (1+1) -ES for fine-grained content refinement in this phase, because it is notably simple to implement, and computationally efficient due to evaluating only one candidate solution per iteration. Note that the IMPACT framework is flexible, and any black-bar box optimizers could be employed for both phases.
The initial 6 are inherited from phase In phase 2, M remains fixed while 8 is refined through iterative application of Gaussian noises. At each iteration, noise &noise sampled from A ((0, o2), where controls the perturbation magnitude, is added to the current 8. The updated perturbation is then applied to the input image using the fixed mask M to generate a new adversarial example. If the new perturbation improves fitness, replaces the current best solution. This process continues until a successful attack is achieved or the maximum number of iterations is reached. By transitioning from block-wel wise to pixel-level optimization, phase overcomes the limitations of using coarse 4 x perturbations, enabling precise adjustments that significantly enhance the attack's 's effectiveness.

---

4
Experiments
In this section, we conduct experimental evaluation to assess the effectiveness of our proposed IMPACT method. Section 4.1 provides a detailed overview of the experimental setup. Section(4.2 presents a comparative analysis of IMPACT against state-of-the ah-tthe-rt patch-based attack methods. Section 4.3 offers ablation studies to analyze the contribution of key components. Finally, Section 4.4 explores the significance and interplay of various hyperparameters in our method.
4.1 Experimental Setup
Dataset and Models:
Following previous works 8 43 27,, we use ImageNet [ for evaluation due to its diverse object categories and real-world scenarios, enabling a comprehensive assessment of our method's effectiveness. Additional experiments on more datasets are provided Appendix E.5 Following the same setup as Patch-RS 8 we randomly select subset of 500 images from the validation set of ImageNet for our experiments. For the victim models, we employ three widely adopted architectures: ResNet50 (17], VGG16 [32), and ViT-B [I]. These models encompass diverse architectural designs, enabling comprehensive evaluation of IMPACT's effectiveness across varying network architectures. All input images are resized to standard size of 224 x 224, consistent with the requirements of the experimented models. The models are officially pre-trained on the full ImageNNet training set, ensuring a robust and reliable baseline for evaluation. All experiments were conducted on system equipped with an NVIDIA GeForce RTX 4090 GPU. The detail parameter settings are provided in Appendix E.1
Evaluation Metrics: We use the attack success rate (ASR) as the evaluation metric, considering only input images the deep model classifies correctly in the absence of any attack. For untargeted attacks, ASR measures the proportion of such images where the adversarial patch successfully causes the model to misclassify them. For targeted attacks, ASR evaluates the percentage of input images where the adversarial patch forces the model to classify the input into specifie, pre-defined target class. For query efficiency, we utilize the average query count (AQ) to measure the average number of queries the attack algorithm requires to successfully craft adversarial patches. Definitions of these performance metrics are presented in Appendix E.2 In addition, due to the stochastic nature of IMPACT, we use the same group of random seeds for evaluations to ensure reproducibility. The influence of multiple seeds is further discussed in Appendix E.3
4.2 Performance Comparison
To evaluate our proposed IMPACT method, we conduct comparisons with state-of-the-art adversarial patch attack methods, including Patch-RS [8. TPA [45) Patch-Fool [14]. Our primary evaluation focuses on IMPACT's performance in challenging black-box scenarios. Additionally, Appendix [E.4] presents a comparison between the white-box variant of IMPACT and Patch-Fool. Furthermore, to assess its robustness, IMPACT's effectiveness against various common defense mechanisms is evaluated in Appendix E.6 Appendix [E. provides the results of physical-woll world experiments, further demonstrating the practicality of IMPACT. Appendix [E8] reports a detailed runtime analysis. Below, present detailed analysis of our experimental comparison results.
For the box comparison, Table I presents the statistical results of adversarial attacks conducted on various ImageNet classification models. Here, The Query represents the query budgets, and 1 1%, 2% are the percentages of perturbation areas. The ASR is expressed as percentage, and for simplicity, we have omitted the unit in the Table We select Patch-RS and TPA as baselines, as they focus on optimizing patch content and location, although their shapes remain fixed as rectangles.
The experimental results consistently demonstrate the superiority of our IMPACT method, which achieves higher ASR and generally lower AQ compared to the baselines across different models and settings. As shown in Table IMPACT exhibits strong performance in untargeted scenarios. For instance, when attacking ResNet50 with query budget of 10,000 and 2% perturbation area, our method achieves a maximum ASR of 96.4%, while Patch-RS achieves 93.6%. Additionally, our AQ is 1044, which is better than Patch-RS's 1408. For targeted attacks, our method demonstrates even greater advantages. On the ResNet50 model, we achieve maximum targeted ASR of 57.8%, whereas Patch-RS RS only reaches 20.0%. Moreover, our AQ 7148, which is significantly smaller than Patch-RS's 8885. We attribute IMPACT's enhanced effectiveness primarily to its joint optimization
7

---

Table 1: Performance comparison for black-box adversarial patch attacks.
Untargeted Attack
Targeted Attack
Model
Query
Method
2%
1%
2%
ASR
AQ
ASR
AQ
ASR
AQ
ASR
AQ
IMPACT
87.2
1236
94.2
676
24.6
4379
38.4
4126
5000
Patch-RS
82.4
1360
89.8
7.6
4790
12.2
4711
TPA
38.0
3519
51.0
2807
4.3
4888
7.8
4781
ResNet50
IMPACT
90.0
1518
96.4
1044
38.4
8239
57.8
10000
Patch-RS
88.2
1990
93.6
1408
10.8
9359
20.0
8885
TPA
51.0
5705
57.0
5091
7.6
9730
15.4
9479
IMPACT
92.8
862
94.2
841
16.2
4635
34,6
4327
5000
Patch-RS
88.6
1114
92.8
832
10.8
4704
15.6
4640
TPA
42.2
3416
53.6
2686
5.6
4880
8.2
4805
VGG16
IMPACT
94.6
1310
95.2
1069
27.6
8893
35.4
8295
10000
Patch-RS
92.4
1562
94.4
1191
16.6
9073
30.8
8425
TPA
56.2
5204
59.8
4777
9.4
9598
16.8
9321
IMPACT
85.6
1315
92.8
18.4
4552
30.2
4311
5000
Patch-RS
80.2
1457
87.4
1052
6.2
4825
10.6
4786
TPA
35.6
3607
48.2
2956
3.8
4942
6.2
4855
VIT-B
IMPACT
88.4
1652
95.0
1157
30,6
8553
48.2
7552
10000
Patch-RS
86.2
2175
91.8
1524
9.2
9456
17.8
9057
TPA
48.4
5854
55.2
5237
6.8
9828
13.6
9553
of multiple patch factors, including irregular shapes, locations, number, and content. This holistic approach allows IMPACT to explore more expressive solution space, enabling the discovery of more potent adversarial patches. For deeper understanding of how IMPACT achieves this by influencing the model's internal mechanisms, we provide an effectiveness analysis in Appendix D
4.3 Ablation Study
key components, we perform ablations on DE, (1+1)-ES, and dimensionality reduction encoding. Note that we do not ablate the random aggregation algorithm because it is essential for transforming sparse modifications into patch-shaped perturbations, and removing would change the attack type. However, to validate its design, investigate the impact of the stochastic elements within this algorithm itself in Appendix E.9 All ablation experiments were performed the ImageNet against the ResNet50 model under total query budget of 5000, with a 2% patch area distributed across patches. Similar trends were observed on other model architectures.
To assess IMPACT's
Table 2: Ablation study components of IMPACT.
Table Encoding granularity ablation. For each tile size t x t. the number of mini-patches was adjusted to maintain the total patch area fixed at 2% of the image.
Method
Phase
Phase 2
ASR
AQ
IMPACT
DE
(1+1)-ES
94.2
676
Effectiveness of DE in Phase 1:
Tile Size
Mini-Patch
ASR
AQ
(1+1)-ES-only
None
(1+1)-ES
60.9
2269.82
RS (1+1)-ES
RS
(1+1)-ES
78.4
1698.7
X
1024
87.1
751.5
+ (1+1)-ES
GA
(1+1)-ES
81.8
1523.8
2 x
256
89.2
691
4 4
64
94.2
676
Effectiveness (1+1J-ES in Phase
8 x 8
16
86.3
822.5
DE-Only
DE
None
90.2
748
16 x
80.5
1099.5
DE+R. RS
DE
RS
92.3
842.1
32 x 32
57.2
1824
Table 2 presents the results comparing our full IMPACT framework against several variants designed assess the roles of its constituent optimizers and the two-phase structure. To validate the choice of DE, we compared IMPACT against variants where DE was replaced by Random Search (RS) and Genetic Algorithm (GA). These results underscore
superior exploratory capabilities of DE in navigating the complex, high-dimensional search space of joint mask and content optimization. To demonstrate the necessity the second refinement phase, we compare IMPACT with a DE-Onlly variant and DE+RS. In DE-Only, the entire 5000 query budget is allocated to the DE algorithm for

---

joint mask and content optimization, omitting the (1+1)-ES refinement. As shown, IMPACT achieves an ASR of 94.2% with 676 AQ, whereas DE-Only only reaches 90.2% ASR with 748 AQ. This improvement highlights that the fine-grained adjustments performed by (1+1)-ES in Phase 2 are crucial for converting nucc-successful patches into effective adversarial examples.
Furthermore, IMPACT employs 4 x 4 tiling strategy as the default for its dimensionality reduction encoding scheme. To validate this choice and understand the impact of different encoding granulari- ties, we conducted an encoding granularity ablation by varying the tile size. The results are presented in T Using smaller tiles resulted in a dimensoalmen search space. While offering finer granulanity this increased complexity led to slightly lower ASR. Moreover, excessively small tile sizes incur prohibitively long optimization times. Using larger tiles can significantly reduce the dimensionality. However, this led to a marked decrease in ASR. This indicates that the coarse granu- larity severely limited the ability to form effective adversarial patches. These findings confirm that the 4x4 tiling offers an effective balance between dimensionality and spatial resolution to construct diverse and potent irregular adversarial patches.
4.4 Parameter Sensitivity Analysis
Our method involves five important parameters that can be adjusted: k, N. Ta. and Te. Here is the number of 4 x mini-patches in the mask, controlling the perturbation area. k denotes the number of patches. N represents the population size. Ta and Ta refer to the iterations of the DE and (1+1)-ES algorithm, respectively. Different parameter settings can lead to varying attack effectiveness.
4.8. 99
A38
25 75- 200
Ferumaiie Amm Terahudei Aie
Pacch Narter
50, 7-m 150
Fakch Nacler
New 120, =5
.... Quey Selgtt-139
Uaey Bedpe 1993
NW 150, -50
1906
Budgea
Query
Quary
Figure 3: ASR query budgets. Effect of perturbation areas (Left). Effect of patch number (Center). Effect N and Ta (Right).
To better illustrate the impact of different parameters, we plot the variation curves of success rates under different query budgets in Figure B] The results demonstrate that larger perturbation area and a greater number of patches contribute to higher ASR. Moreover, while different combinations of N and Te show similar performance under high query budgets, smaller populations with more iterations excel under lower budgets. This suggests that prioritizing iteration count over population size can improve efficiency, especially with limited query resources. In addition these primary analyses, we conducted further parameter experiments, including an investigation into the impact of DE's mutation factor and crossover probability. These results demonstrate the relative robustness of IMPACT to these parameters. Detailed experimental data for those studies can be found in Appendix E.10
Our IMPACT supports generating multiple patches simultaneously. To demonstrate the im- provement in attack effectiveness achieved by us- ing multiple patches, we conduct an experiment to assess the impact of varying the number of patches. As depicted in Figure 4 across all tested total per- turbation areas, increasing the number of patches generally leads to a noticeable improvement in ASR. This trend highlights a key advantage of multi-patch strategies: by distributing adversarial perturbations across multiple smaller, strategically placed regions, IMPACT is able to degrade model accuracy more effectively than using a single con- centrated patch. This ability to alter diverse local regions within the image significantly enhances the overall attack effectiveness.
42
42
2%
Figure 4: Impact of the number of patches (k) ASR for different total perturbation areas (1%, 2%, 3%). Experiments were conducted on ResNet50 with N 150, T. 50, T. 2500.

---

5 Conclusion
In this paper, we introduced novel irregular multi-patch adversarial attack framework that supports simultaneously optimizing all critical patch factors, including shape, location, number, and content. Under our IMPACT framework, DE was used first jointly optimize the patch mask and content, supported by a unique dimensionality reduction encoding scheme. Secondly, (I+I)-ES further refined the patch content for improved precision. Additionally, we proposed random aggregation algorithm to generate diverse, irregular patch designs for practical use. Extensive experiments demonstrated that our method outperformed state-of-the-an approaches, significantly enhancing attack success rates. This work provides valuable insights into adversarial patch design and optimization, paving the way for practical methodologies in this domain.
Limitations: This work focuses on score-based black-box attacks, where access to the model's 's output probabilities enables effective guidance of the evolutionary optimization. In contrast, based black-box attacks, which provide only the model's top-1 predicted label, pose fundamentally different and more restrictive challenge. The absence of probability scores makes significantly harder to infer meaningful search directions, especially when jointly optimizing multiple patch factors. Addressing this setting would require substantially different algorithmic designs and therefore beyond the scope of this paper. Nevertheless, extending IMPACT to support decision- remains an important direction for future research. Moreover. as IMPACT is primarily an empirical approach, more rigorous theoretical analysis of its underlying mechanisms constitutes an important direction for future research.
attacks
This work introduces IMPACT, a framework for crafting irregular multi- nulti-patch adversarial attacks to uncover vulnerabilities in deep vision models. While these attacks are essential for developing stronger defenses and improving model robustness, they also carry the risk of malicious misuse. We advocate for the responsible and ethical application of such technologies, emphasizing their use as tools for the advancement of trustworthy AI, rather than for purposes that could cause harm or compromise system integrity.
Ethics Statement:
Acknowledgments
This work was supported in part by the National Natural Science Foundation of China (N0.62272117 and No.62476030).
References
[1] Moustafa Alzantot, Yash Sharma, Supriyo Chakraborty, Huan Zhang, Cho-Jui Hsieh, and Mani B Srivastava. Genattack: Practical black-box attacks with gradient-free optimization. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 11 111199 Association for Computing Machinery, 2019.
[2] Tom B Brown, Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer. Adversarial patch. arXiv preprint arXiv:/7/..09666, 2017.
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pages
Springer, 2020.
[4] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks, In 2017 IEEE Symposium on Security and Privacy (SP), pages 39-57. IEEE, 2017.
[5] Zhaoyu Chen, Bo Li, Shuang Wu, Jianghe Xu, Shouhong Ding, and Wenqiang Zhang. Shape matters: deformable patch attack. In European Conference on Computer Vision, pages 529-548. Springer, 2022.
[6] Zhaoyu Chen, Bo Li, Shuang Wu, Shouhong Ding, and Wenqiang Zhang. Query-Ffffiiien Decision- Baseed Black-Box Patch Attack. IEEE Transactions on Information Forensics and Security, 18:5522-5536, 2023. ISSN 1556-6013, 1556-6021.
[7] Francesco Croce and Mauhias Hein. Sparse and imperceivable adversarial attacks. In IEEE International Conference on Computer Vision, pages 4724 4732. IEEE, 2019.
[8] Francesco Croce, Maksym Andriushchenko, Naman D Singh, Nicolas Flammarion, and Matthias Hein. Sparse-rs: a versatile framework for query-efficient sparse black- box adversarial attacks. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pages 6437-6445. AAAI Press, 2022.
10

---

[9] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In JEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 248-255. IEEE, 2009.
[10] Yinpeng Dong. Hang Su, Banyuan Wu, Zhifeng Li, Wei Liu, Tong Zhang, and Jun Zhu. Efficient decision- based black-box adversarial attacks on face recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages
IEEE, 2019.
[11] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. 9th International Conference on Learning Representations. OpenReview.net, 2021.
[12] Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physical physical-world attacks deep learning visual classification. In IEEE Conference Computer Vision and Pattern Recognition, pages 1625-1634 IEEE, 2018.
[13] Alhussein Fawzi and Pascal Frossard. Measuring the effect of nuisance variables on classifiers. In Proceedings of the British Machine Vision Conference, pages 137-1. BMVA Press, 2016.
[14] Yonggan Fu, Shunyao Zhang. Shang Wu, Cheng Wan, and Yingyan Lin. Patch-fool: Are vision transformers always robust against adversarial perturbations? In The Tenth International Conference on Learning Representations. OpenReview.net, 2022.
[15] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint 4rX25:1412.6572, 2014.
[16] Chaoxiang He, Xiaojing Ma, Bin B. Zhu, Yimiao Zeng, Hanqing Hu, Xiaofan Bai, Hai Jin, and Dongmei Zhang. Dorpatch: Distributed and occlusion-rohust adversarial patch to evade certifiable defenses. 31t Annual Network and Distributed System Security Symposium. The Internet Society, 2024.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference Computer Vision and Pattern Recognition, pages 770-778. IEEE, 2016.
[18] Chengyin Hu, Weiwen Shi, Tingsong Jiang, Wen Yao, Ling Tian, Xiaoqian Chen, Jingzhi Zhou, and Wen Li. Adversarial infrared blocks: A multi-view black-box attack to thermal infrared detectors in physical world. Neural Networks, 175:1063 10, 2024.
[19] Jun-Jic Huang, Ziyue Wang, Tianrui Liu, Wenhan Luo, Zihan Chen, Wentao Zhao, and Meng Wang. Dempaa: Deployable multi-mini-ppe mini-patch adversarial attack for remote sensing image classification. IEEE Transactions on Geoscience and Remote Sensing, 62:1-13, 2024.
[20] Danny Karmon, Daniel Zoran, and Yoav Goldberg. Lavan: Localized and visible adversarial noise. In International Conference on Machine Learning, pages 2507-2515. PMI.R, 2018.
[21] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, volume 25, pages 1106-1114 Curran Associates, Inc., 2012.
[22] Chao Li, Handing Wang, Jun Zhang, Wen Yao, and Tingsong Jiang. An approximated gradient sign method using differential evolution for black-box adversarial attack. IEEE Trans. Evol. Comput., 26(5))976-995 2022.
[23] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations. OpenReview.net, 2018
[24] Apostolos Modas, Seyed- Moosavi- Dezaavli, and Pascal Frossard. Sparsefool: a few pixels make a big difference. In IEEE Conference Computer Vision and Pattern Recognition, pages 9087-9096. IERR, 2019.
[25] Kien Nguyen, Tharindu Fernando, Clinton Fookes, and Sridha Sridharan. Physical adversarial attacks for surveillance: A survey, IEEE Trans. Neural Networks Learn Syst., 35())): 01777706677777 2024.
[26] Hao Qiu, Leonardo Lucio Custode, and Giovanni lacca. Black-bor adversarial attacks using evolution strategies. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pages 1827-1833, Association for Computing Machinery, 2021.

---

[27] Yu Ran. Weijia Wang. Mingjie Li, Lin-Cheng Li. Yuan-Gen Wang, and Jin Li. Cross-shaped adversarial patch attack. IEEE Trunsactions Circuity and Systems for Video Technology, 34(4):2289-2333, 2023.
[28] Sukrut Rao, David Stutz, and Bernt Schiele. Adversarial training against location-optimized adversarial patches. In European Conference on Computer Vision, pages 429- 448. Springer. 2020.
[29] Hadi Salman, Andrew Ilyas, Logan Engstrom. Ashish Kapoor, and Aleksander Madry. Do adversarially robust imagenet models transfer better? In Advances in Neural Information Processing Systems, 2020.
[30] Ramprasaath R. Sclvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In IEEE International Conference on Computer Vision, pages 618 6-6.. IEEE Computer Society, 2017.
[31] Abhijith Sharma, Yijun Bian, Vatsal Nanda, Phil Munz, and Apurva Narayan. Vulnerability of cnns against multi-patch attacks. Proceedings of the 2023 ACM Workshop on Secure and Trustworthy Cyber-Physical Systems, pages 23-32. ACM, 2023.
[32] Karen Simonyan. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.14 1536, 2014.
[33] Jiawei Su, Danilo Vasconcellos Vargas, and Kouichi Sakurai. One pixel attack for fooling deep neural networks. IEEE Transactions on Evolutionary Computation, 23(5):828-844 -411, 2019.
[34] Ilya Sutskever, Oriol Vinyals, and Quoc Le. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pages 3104-31 12. Curran Associates, Inc., 2014.
[35] C Szegedy. Intriguing properties of neural networks. arXiv preprint 132.v:13 2013.
[36] Guijian Tang, Wen Yao, Chao Li, Tingsong Jiang, and Shaowu Yang. Black-box adversarial patch attacks using differential evolution against aerial imagery object detectors. Eng. Appl. Artif. Intell., 137:109141, 2024.
[37] Tian, Jingwen Pan, Shangshang Yang, Xingyi Zhang, Shuping He, and Yaochu Jin. Imperceptible and sparse adversarial attacks via dutal-population-baeeee constrained evolutionary algorithm. IEEE Trans. Artif. Intell., 4(2):268-21,, 2023.
[38] Viet Quae Vo, Ehsan Abbasnejad, and Damith Ranasinghe, Query efficient decision based sparse attacks against black-box deep learning models. In The Tenth International Conference on Learning Representa- tions. OpenReview.net, 2022.
[39] Yichen Wang. Yuxuan Chou, Ziqi Zhou, Hangtao Zhang, Wei Wan, Shengshan Hu, and Minghui Li. Breaking barriers in world adversarial examples: Improving robustness and transferability via robust feature. Proceedings of the AAAI Conference on Artificial Intelligence. volume 39, pages 8069-8077. AAAI Press, 2025.
[40] Hui Wei, Hao Tang, Xuemei Jia, Zhixiang Wang. Hanxun Yu, Zhubo Li, Shin' 'ichi Satoh, Lue Van Gool, and Zheng Wang. Physical adversarial attack meets computer vision: A decade survey. IEEE Trans. Pattern Anal. Mach. Intell., 46 96(12)97777 (7911)))77771777711 2024.
[41] Xingxing Wei, Ying Guo, Jie Yu, and Bo Zhang, Simultaneously optimizing perturbations and positions for black-box adversarial patch attacks. JEEE transactions on pattern analysis and machine intelligence, 45(7):9041-905,, 2022.
[42] Xingxing Wei, Jie Yu, and Yao Huang. Physically adversarial infrared patches with learnable shapes and locations. In IEEE Conference on Computer Vision and Pattern Recognition, pages 12334-12342 IEEE, 2023.
[43] Phoenix Williams and Ke Li. Camopatch: An evolutionary strategy for generating camoflauged adversarial patches. Advances in Neural Information Processing Systems, volume 36, pages 67269 67283,, Curran Associates, Inc., 2023.
[44] Chong Xiang, Arjun Nitin Bhagoji, Vikash Schwag, and Prateek Mittal. Patchguard: A provably robust defense against adversarial patches via small receptive fields and masking. In 30th USENIX Security Symposium, pages 2237-2254, USENIX Association, 2021.
[45] Chenglin Yang, Adam Kortylewski, Cihang Xie, Yinzhi Cao, and Alan L. Yuille. Patchattack: black-box texture-based attack with reinforcement lcarning. In European Conference on Computer Vision, volume 12371, pages 681-698. Springer, 2020.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly articulate the contributions of this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of this work the Conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
13

---

This work focuses on empirical algorithm design and evaluation and does not present formal theoretical results.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: This paper provides detailed parameters, configurations, and source code for our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
14

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
This paper provides open access to the data and code anonymously.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipss.cc/pill wttpcccppppppppeeeeeeeeeeeeeeeee httaa//gggiiee//eeuuabiisisssss /Cc////oddiiisssseeeeeee ssiiiiiisiyyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
All parameters and settings are included in our paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We initialized the random seed and conducted multiple experiments to average results experiments involving randomness.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
15

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide sufficient information the computational resources the paper.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Our work adheres to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification; Our work advances understanding of model vulnerabilities by generating irreg- ular adversarial patches, which can be used to improve defense mechanisms via robustness evaluation.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
16

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We mention the assets used and adhere to their respective licenses and terms of use.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
17

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: This paper releases the source code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing experiments research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
18

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
19

---

Appendix
A Function Explanation
To provide additional clarity, summarizes the functions used in Algorithm Each function plays a specific role the EA-based optimization framework of IMPACT.
This function initializes a population Po of N individuals to ensure diverse starting points for the search process. Each individual structured according to our encoding scheme, consisting of two components:
PopInit:
binary encoded mask and a continuous encoded content r.
Mutation: This core DE function creates mutant vector v; for each individual p: by differential combination of other population members. This process introduces new variations into the search.
Crossover: This function mixes the components of the parent individual p; and its corresponding mutant vector v; to create a new trial individual uy This enhances population diversity.
Aggregation: This is component of our framework. It takes the sparse binary mask from trial individual and transforms its scattered active elements into physically contiguous, irregular patches.
Fitness: This function evaluates the quality of trial individual. reconstructs the full patch, applies the image, and calculates the model's cross-entropy loss.
Selection: This function compares the fitness of the trial individual u; with the parent p.. The one with the better fitness score survives into the next generation's 's population.
SelectBest: This is simple function that iterates through the final population and returns the single individual with the best fitness score found during the optimization.
B Random Aggregation Details
To provide more detailed exposition of our random aggregation algorithm, we present it in Al- gorithm 2 This algorithm a crucial component of our IMPACT framework, responsible for transforming the spatially dispersed active mini-patches into coherent, irregular patch structures.
Algorithm 2 Random Aggregation Algorithm
Input: Mask encoding m E {0, 13', patch number k Output: Aggregated mask encoding
1: Reshape m into a 2D binary matrix M E 10,11v
2: Extract coordinates of one-valued elements: x'= {(a, b) Mía
3: Perform clustering:
K K Maass(X, k),, where C = (C1, C2...., C
4: for each C E C do
5: Select Seenee E C2 uniformly random.
6: Create aggregated region Al ASSenterr.
7: for each 8 E C. {Scenter} do
8: Sample surget E (eeenern) where Nb(scemte) is the neighborhood of Scenter-
9:
Snew + (s, Surget)) ).
10:
if Snew .Nb (Nb(A(sscnnter) then
11:
Update A( A(Scenter) + A( (SSeme))) {SSoww }-
12:
Update M: [s] <- O, M [sow]] f- 1.
end if
14:
end for
end for
16: Flatten M to m E {0.13'
17: return m
Furthermore, to visually illustrate the algorithm's 's operation, Figure 5] depicts an example of the aggregation process. In this case, the initially scattered active elements are first partitioned into three distinct clusters by K-Means. Subsequently, our random aggregation algorithm processes each cluster, resulting the formation of three distinct patches, each exhibiting an irregular and locally connected shape.
20

---

Patch 2
Cluster 2
Patch
Patch 3
1
Cluster
Cluster
I
Figure 5: Example of random aggregation.
Figure 6 further demonstrates the practical outcome of this aggregation process. As can be seen, the algorithm effectively consolidates initially scattered units into locally connected, irregular shapes, thereby forming patches that are well-formed and readily applicable for adversarial attacks.
16 mini--pach
mini-patch 8-8 tiling size
64 mini-patch
16x16 tiling size
4x4 tiling size
Refore
Figure 6: Visualization of the Random Aggregation algorithm's ' effect. The top row shows the spatial distribution of active mini-patches before aggregation, for different total mini-patch counts and corresponding tiling sizes. The bottom row illustrates how these scattered mini- mani-patches are consolidated into coherent, irregular patch structures by our aggregation algorithm. Each column represents different configuration. Left: 16 mini-patche and 16x 16 tiling size. Center: 32 mini- patches and 8x8 tiling size. Right: 64 mini-patches and 4x4 tiling size.
C DE Components
C.1 Initialization
The initial population Pd for the DE algorithm, consisting of individuals, is generated randomly to ensure diverse starting points for the search process. Each individual p E Pi is structured according to our dimensionality reduction encoding scheme, consisting of two components: a binary encoded mask and continuous encoded content
The mask m a binary vector length satisfy the constraint on the total number active mini mini-patched patches, exactly elements of m randomly selected and set to I, while the remaining |
21

---

elements are set to O. This ensures that every individual in the initial population adheres to the specified patch area from the beginning the optimization.
The content component r corresponds to the initial color information for the n active mini-patches in matrix of shape x The values within this matrix
its associated mask It is represented initialized by sampling uniformly at random from the valid pixel intensity range [0, 255].
This random initialization process generates diverse population Po {pI.... . ,N where each individual satisfies the patch area requirements and possesses varied initial content, providing a robust foundation for the subsequent evolutionary operations of mutation, crossover, and selection.
C.2 Mutation
Building upon our dimensionality reduction encoding, we further tailor the core DE operators. In particular, mutation must be adapted to handle the distinct binary nature of the encoded mask and the continuous nature of the encoded content. We present these customized mutation procedures below.
For each individual P;, we first randomly select three distinct individuals Pa = (m,, ra), Ps = (ms, ro), and Pe (m,, re) from the population. These individuals serve the basis for generating the mutated individual 02 = (m, Fi), as detailed below. The mutation operation is applied separately the two components of the individual.
For the mask component, we adopt a mutation strategy in the binary space. Using the selected mask components ma, my, and m. of the chosen individuals, the mutated mask in is computed follows:
m; (m, + (ms me)) mod! 2.
(5)
Here, the modulo operation ensures that the resulting tn, remains valid binary vector.
For the perturbation component in the continuous space, using the selected perturbation components Fas rb, and r., the mutated component is calculated
ra F (ID (eb-rre) re).
(6)
where F a mutation factor. The crossover and selection mechanisms, which follow standard DE practices (c.g., binomial crossover and greedy selection), along with the detailed fitness function calculation, are described below.
C.3 Crossover
Each mutated individual e proceeds to the crossover operation, which is performed as follows:
U4 = Vi.sr if rand, (0, 1) < C.R. or = Jrand
Piiht otherwise
where v.j is the j-th element of the mutant individual ey. Pt, is the j-th element of the parent individual The crossover operation combines i and p; to create trial individual u, CR e [0, 1] the crossover probability, which determines the likelihood of inheriting elements from the mutant individual. To ensure that at least one element is inherited from the mutant individual, Frand is randomly chosen index that guarantees Vi, is selected for the corresponding j = Frand-
C.4 Repair
Following the mutation and crossover operations, the total number of elements with value 1 may deviate from the pre-defined constraint. We should repair the count of 1s to precisely match the target patch area. Let n denote the desired number of elements with value 1, and n. denote the current number of 1s in m. If n. > n, randomly select n. elements with value and set them to o. If nhe < n,, we randomly select n ne elements with value and set them to 1. This procedure ensures that the total perturbation area of the patch remains constant.
C.5 Fitness Function
The fitness function evaluates the quality of each individual during the selection process, retaining those with higher fitness and discarding others. It is designed based on the cross-entropy loss LoE.
22

---

Specifically, pixel values from are extracted and mapped to positions where m[i] = 1. forming perturbation ô with dimensions 3 x $ +- Using 4 4-tiling, and are resized to match the original image dimensions, i.e., MM and respectively. The perturbed example is then constructed using Equation (I) and fed into the target model. Subsequently, the cross-entropy loss regarding the model's 's output is used to assess the perturbation's ' impact. For untargeted attacks, the fitness function is Lo (+)) y), where y is the true label. For targeted attacks, is LC (8), yt), where y, is the target label.
C.6 Selection
The selection operator in our DE algorithm determines which individuals, between the current population and the newly generated trial vectors, will survive form the population for the next generation. This process employs a one-to-one greedy selection strategy based on fitness values.
Let P (pi.-... .......PP' be the current population at generation i, where each individual p; (m, r;) consists of an encoded mask and content. Let U = {u1..... uN } be the population of trial vectors generated through mutation and crossover from P, where each u, (m', r.)). Let F(p) denote the fitness function, where a higher fitness value indicates better solution.
For each pair of corresponding individuals, their fitness values are compared. The individual with the superior fitness is selected to be part of the population for the next generation.
(0) u
iF F(p!")
((+1) p;
(8)
otherwise
This elitist selection mechanism ensures that the fitness of the population is non-decreasing from generation to the next, preserving good solutions found so far and driving the search towards more promising regions of the solution space.
D Effectiveness Analysis
To gain deeper insights into how IMPACT influence the model's decision-makinn process, we employ class activation mapping (CAM) [30] to visualize the regions the model focuses when making predictions. Figure presents CAM visualizations for several examples from the ImageNet dataset, showing the model attention both original images and their corresponding adversarial versions generated by IMPACT.
After applying the IMPACT-generater patches, significant shift in the model's attention is observed in the corresponding CAMs. The model's focus is now often distracted or drawn towards the locations of our strategically placed irregular patches leading to misclassification. This suggests that the patches introduce features that the model deems highly indicative the incorrect class or sufficiently disruptive to confuse the features of the true class. The IMPACT method effectively manipulates the model's learned saliency. The irregular, multi-patch design appears adept at either creating new, highly salient focal points or disrupting the existing saliency map of the true object. The distributed nature of the multi-patches allows for influencing model attention at several locations simultaneously, potentially being more effective than a single, larger patch in confusing the model's global understanding of the scene.
The CAM results provide qualitative evidence for the effectiveness of our IMPACT. The patches do not need to be large or overtly cover the main object to be effective; instead, their carefully optimized content and placement, facilitated by our DE-based joint optimization and irregular shape generation, are sufficient to significantly alter the model's feature interpretation and subsequent attention, leading to successful attacks.
23

---

Original Image
Original Image CAM
Adversarial Image
Adversarial Image CAM
Figure 7: The figure is organized into four columns per example: (1) the original image, (2) the CAM for the original image correctly classified by the ResNet50 model, (3) the adversarial image generated by IMPACT, and (4) the CAM for the adversarial image, misclassified by the model.
E Experimental Details
E.1 Parameter Settings
We evaluate IMPACT under different perturbation areas, with parameters 32, 64 controlling perturbation areas of 1% and 2%, respectively. The remaining parameters of IMPACT are set as follows: & = 3, N 50, Ta = 150, and T. = 2500. For the comparison methods, since these methods also support the ImageNet dataset, we set the parameters to their recommended values as suggested the respective papers.
Black-box Comparison:
The parameters of IMPACT are N = 50, Ta = 150, and T' = 2500. For perturbation area of 0.5%, we set n 16 and l: For perturbation area of 1%, we set 32 and & = 2. For perturbation area of 1.5% we set = 48 and & = 3. To ensure fairness in comparison, we align the patch number for Patch-Fool with that of IMPACT different perturbation areas. Additionally, when optimizing patch content, both IMPACT and Patch-Fool use 250 iterations of PGD.
White-box Comparison:
E.2 Evaluation Metrics
Here, we present the formulas for ASR and AQ. For the input images Netemn which the model correctly classifies without any attack, the untargeted ASR is defined as:

---

N miscllosified
ASR untargeted
x 100%
Nelcan
where Niisclassified is the subset of Nelean that is misclassified after applying adversarial patches. |+| is the size the set. The targeted ASR is defined as:
Nurgeted
ASR aargeeed
X 100%
[Neeean
where Nargeted is the subset of Netemn where adversarial patches successfully causes the model to classify the images into a specific, pre-defined target class.
AQ measures the average number of queries required to successfully craft adversarial patches. For the set of input images Netean is defined as:
EieNden qi
AQ
|elean
where qi is the number of queries required to successfully attack the i-th image. For unsuccessful attacks, qi is set to the query limit Qmas-
E.3 Sensitivity of Random Seeds
Given the stochastic elements inherent in our IMPACT framework, we investigated the method's sensitivity random seed. We performed 10 independent runs for untargeted attacks with both I 1% and 2% total perturbation areas against the ResNet50 model, each run utilizing a different random seed. The total query budget was fixed at 5000. Figures illustrates the distribution of the ASR and AQ across seeds.
Distribution Across Saads
AQ Distribution Across Seeds
1200
34-
ASR
58
800
600
82
Figure 8: Violin plots of ASR and AQ across 10 random seeds for two patch budgets (1 % of patch area). The shaded violins depict the full distribution density, the black boxes show the interquartile range with median lines, and the orange ticks mark the mean. Allowing larger patch area (2 %) yields higher and more consistent ASR while reducing the number of queries needed.
The violin plots for both ASR and AQ demonstrate that our IMPACT method exhibits good robustness initialization stochasticity. While some variation expected in heuristic search algorithms, the results are largely consistent, especially at a 2% perturbation area where high success rates are reliably achieved with stable query efficiency. This indicates that IMPACT's performance is not overly sensitive to the specific random seed chosen, making it a reliable method for generating adversarial patches.
E.4 Comparison of White-Box Attack Methods
We designed white-box experiment to evaluate the contribution of our patch mask generation strategy. To ensure fair comparison, create white-box version of IMPACT method, referred to
25

---

as IMPACT-W. Specifically, after obtaining the multi- patch mask M from DE, we use Projected Gradient Descent (PGD) [23 to optimize the patch content. Patch-Fool [4] is state-of-the-an
methods, and it has shown superior performance compared to other white box methods, such as LOAP 28 DPA [5] Patch-Fool offers multiple strategies for selecting patch locations. We focus on two: saliency-based selection (Patch-Fool-S) and random selection "Patch-Fool-R, Table presents the comparison with Patch-Fool, based on consistent perturbation areas and patch counts: 0.5% (1 patch), 1% (2 patches), and 1.5% (3 patches). The parameter settings of IMPACT-W and Patch-Fool are Appendix E.. Results show our method outperforms Patch-Fool, highlighting the advantage of optimizing patch shapes and demonstrating the superiority of our patch mask generation approach.
Table 4: Performance comparison of white-box methods.
ASR
Model
Method
0.5%
1 1
1.5%
IMPACT-W
80.50
95.40
98.50
ResNet50
Patch-Fool-S
76.00
93.40
97.40
Patch-Fool-R
68.40
90.40
96.80
E.5 Evaluation on More Datasets
To further validate the generalization capability of IMPACT, we extended our experiments beyond ImageNet to include CIFAR-10 and CIFAR- 100 using ResNet50 as the backbone model. Table 5
summarizes the results under both untargeted and targeted settings. Given the lower resolution of CIFAR images (32 x 32), we adjusted the encoding to employ a tiling configuration. This modification ensures maximum granularity while maintaining manageable search space, highlighting the flexibility of our encoding design.
Table 5: Performance on CIFAR-10 10 and CIFAR-100 datasets using ResNet50, with 5000 query
budget and 5% perturbation area.
Dataset
Attack Type
Method
AQ
ASR (%) 1
Patch-RS
156.78
97.8
Untargeted
IMPACT
113.98
99.2
CIFAR-10
Patch-RS
2155.47
85.3
Targeted
IMPACT
1634.36
92.7
Patch-RS 98.52
98.1
Untargeted
IMPACT
72.81
99.3
CIFAR-100
Patch-RS
2310.80
82.5
Targeted
IMPACT
1556.17
90.1
As shown in Table IMPACT achieves near-perfect ASRs exceeding 99% in untargeted attacks across both CIFAR-T TO and (IFAR-100, outperforming the strong baseline Patch-RS. Moreover, even in the more challenging targeted setting, IMPACT maintains ASRs above 90%, significantly reducing the query cost compared to Patch-RS. These results demonstrate that IMPACT's effectiveness is not limited to a particular data distribution such as ImageNet, but rather generalizes well to datasets with distinct statistical and visual characteristics.
E.6 Evaluation on Defense Models
To further assess the robustness of our IMPACT method, we evaluated its performance against several
defense mechanisms. Similar to reference 43 we evaluate the effectiveness of our IMPACT method against adversarially trained models 29 and the PatchGuard defense [44].
Table summarizes the ASR and AQ our IMPACT method when attacking these defended models, compared to its performance against a standard, non-defended ResNet50 model. The results demonstrate that while defenses can impact performance, our IMPACT method exhibits considerable resilience, particularly against the tested adversarially trained models. Against the specialized
26

---

PatchGuard defense, IMPACT's success rate, while reduced, remains significant, indicating its potential to overcome defenses specifically designed for patch attacks. The increased query cost in this scenario highlights the added difficulty imposed by such a defense.
Overall, these findings underscore the strength of IMPACT as black-box patch attack. Its ability to jointl optimize diverse patch characteristics allows it to remain effective even when faced with common and specialized defense strategies, motivating further research into more comprehensive defense mechanisms.
Table 6: ASR and AQ of IMPACT against standard and defended models (Black-box, untargeted, 5000 queries, 2% area, patches).
Target Model
ASR (%)
AQ
ResNet50 (No Defense)
94.2
676
AT-ResNet50--2 (c 3.0)
92.2
520.6
AT-ResNet50--intt (c = 4/255)
93.8
508.1
bagnet17 with PatchGuard
83.4
1920.6
E.7 Physical- World Evaluation
To assess the real-world applicability of IMPACT, we conducted physical-world experiments follow- ing Wang et al. [39]. Specifically, we randomly selected 100 images with ad adversarial patches and printed each on a 10 cm 10 white paper. Using iPhone 15, we photographed each printed image from different distances (10 cm, 15 cm, and 20 cm) and viewing angles (0", 15%, and 30"). The captured photos were then resized 224 224 pixels for input to the ResNet50 model. All experiments were conducted under the targeted attack setting, where the goal was to force the model to predict a specific incorrect class.
Table Physical -ooldd performance of IMPACT. The targeted ASR (%) is reported across varying distances and viewing angles for ResNet50 model. "Digital ASR" denotes the corresponding performance in the digital domain for the same 100 images.
Model
Digital ASR
Angle
ASR 10cm
ASR@ 15em
ASR@ 20cm
0"
41.0
37.0
32.0
ResNet50
53.0
15e
35.0
31.0
26.0
30"
28.0
25.0
21.0
As shown in Table 7 IMPACT remains effective in real-world settings. Even under challenging conditions (20 cm distance and 30 viewing angle). it achieves 21.0% targeted ASR. The observed degradation from the digital to the physical domain is expected, as variations in distance and viewing angle inevitably diminish patch detail captured images. Despite this, IMPACT retains meaningful attack performance, demonstrating its practical feasibility.
E.8
Consumption Time Analysis
To assess the computational overhead of IMPACT, we follow Williams et al. 43 measuring the average time required to successfully complete attack single image. The experiments were conducted on a single NVIDIA RT' 4090 GPU under the untargeted attack setting against the ResNe150 model, with query budget of 5000 and perturbation area of 2%. The results summarized in Table g
Table 8: Comparison of computational time under the untargeted attack setting ResNet50.
Method
ASR (%)
AQ 4
Runtime (s) 4
IMPACT
94.2
676
29.16
Patch-RS
89.8
982
1.78
As shown in Table 8 primarily because it employs population-based optimization algorithm that evaluates multiple
IMPACT requires more time per attack (29.16s) than Patch-RS (1.78
27

---

candidate solutions per generation. In contrast, Patch-RS relies on simpler random sampling strategy with lower per-iteration computational cost.
Despite the higher runtime, IMPACT significantly outperforms Patch-RS in the metrics that matter most in black-box attack scenarios. Specifically, it achieves higher attack success rate (94.2%) % vs. 89.8% %) and requires substantially fewer model queries (676 vs. 982 on average). In real-world applications, each query can be expensive and increases the risk of detection, making query efficiency far more critical than raw computational time. Thus, while IMPACT incurs a higher runtime than Patch-RS, Rs, this is justified by its substantially better attack quality and query efficiency, and the overall runtime remains practical for real-world deployment.
E.9 Ablation Study on Random Aggregation
Our Random Aggregation algorithm incorporates stochasticity at several key junctures to foster diversity in the generated irregular patch shapes. Specifically, these include: (1) Random aggregation center. The center for aggregation within each K-Means cluster is chosen randomly from the cluster's s members. (2) Random target in neighborhood. When an element is being moved, the target towards which it moves is randomly selected from the neighborhood of the aforementioned aggregation center. (3) Random movement direction. If multiple directions (e.g., horizontal and vertical) would reduce an element's distance to its target, one is chosen randomly. To understand contributions of these stochastic elements, we conducted an ablation study in Table 9
Table 9: Ablation study on stochastic components within random aggregation.
Aggregation Variant
ASR (%)
AQ
Full random
94.2
676
No random aggregation center
89.5
782
No random target in neighborhood
92.1
727
No random movement direction
92.4
684
For no random aggregation center, instead of randomly selecting an aggregation center from within the cluster members, we deterministically the aggregation center for each cluster. All other stochastic elements remain active. For no random target in neighborhood, elements are moved directly towards the aggregation center itself, rather than to a randomly chosen point within its neighborhood. For no random movement direction, the algorithm always attempts a horizontal move first, followed by vertical move.
the centroid calculated by the K-Means algorithm as
This ablation study highlights the importance of the stochastic elements within our Random Aggre- gation algorithm. The random selection of the aggregation center appears to be the most impactful, significantly contributing to both attack success and efficiency. Randomness in selecting the target within the center's neighborhood also provides noticeable benefit. While randomizing the movement direction has a smaller effect on the primary metrics, retaining all three sources of randomness likely contributes to the overall diversity and robustness of the patch generation process. These findings justify our design choices to incorporate these levels of stochasticity to enhance the exploration of diverse and effective irregular patch shapes.
E.10 Parameter Experiment Details
In our method, the number of queries is influenced by three parameters: N, Ta. and Te. The total number of queries is calculated as Q = N x Ta + Te. To assess the impact of these parameters on optimization performance, of 10,000 while keeping n, k, and T. constant. This approach resulted various combinations of N and Tr. Table 10 illustrates the effect of these different combinations. It can be observed that the highest ASR 95.20% is achieved when = 50 and T₃ = 150. Under a fixed query budget, prioritizing higher number of iterations over a larger population size leads to better optimization and more effective attacks.
design experiments focusing N and Tg. We set a total query budget
We further investigate the sensitivity of the DE phase to its two core hyperparameters: the mutation factor F and the crossover probability CR. All experiments below use only Phase optimization
with N = 50, Ta 100, and fixed total query budget of Q 5 000. We sweep the two core DE
28

---

Table 10: The impact of different parameter combinations.
Te
.V
Ta
ASR(%)
25
300
94.60
150
95.20
64
2500
100
75
94.00
150
50
93.80
hyperparameters: mutation factor F and crossover rate CR, while holding the total patch area at 2% and the number of patchesk 3 constant. Specifically, when evaluating CRR fix F = 2, and when evaluating F, we fix C-R = 0.8. The results are reported Table 11 and Table T2 respectively.
Table 11: Effect of crossover rate CR.
Table 12: Effect of mutation factor F.
CR
ASR (%)
AQ
F
ASR (%)
AQ
0.2
90.4
835
0.5
85.4
1058
0,4
88.2
840
1.0
87.8
835
0.6
89,2
785
2.0
90.8
748
0.8
90.8
748
3.0
89.6
718
1.0
89.4
727
4.0
89.2
743
Our analysis of these hyperparameter sweeps reveals that IMPACT's performance is relatively robust to variations in the mutation factor F and crossover rate CR within typical ranges. However, to optimize the balance between success rate and query efficiency, we identified CR 0.8 and F = 2.0 as offering marginally superior trade-off. Consequently, these values are adopted as defaults in other reported experiments.
29


Reviews:
[{"review_id": "2pqrTzUbFr", "reviewer": "Reviewer_d9ew", "summary": "This paper claims to propose a novel adversarial patch generation method that enables joint optimization of the patch’s position, number, shape, and content, without relying on gradient information. The core techniques include dimensionality-reducing encoding and a random aggregation algorithm. The proposed method shows significant improvements over baseline approaches on ImageNet.", "questions": "1. In line 181, what does pi refer to? Its definition is not explained.\n2. In the ablation study (Table 3), the authors claim that 4×4 is the optimal tile size. However, as mentioned by the authors in line 316, smaller tile sizes should offer a higher-dimensional search space and finer granularity. So why does the attack success rate decrease with smaller tiles?\n3. The paper repeatedly emphasizes the physical feasibility of the proposed method. Can experimental results be provided to demonstrate the advantages of the method in physical-world attacks?", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The introduction of the proposed method is very detailed and easy to understand.\n2. The dimensionality-reducing encoding and random aggregation algorithms are simple yet effective.\n3. The experimental results are convincing.\n\nWeaknesses:\n1. The authors claim that their main contribution is proposing a method capable of jointly optimizing the position, shape, content, and number of adversarial patches. However, the method does not unify all these elements into a common feature space; instead, it simply uses two separate feature vectors to represent the various factors. This approach does not appear to differ significantly from previous methods.\n2. The experiments are conducted only on ImageNet, lacking diversity in datasets.\n3. The authors did not provide the full executable code—only code for key components—making it difficult to verify reproducibility.", "was_revised": true, "final_justification": "In the initial review, I raised concerns about the novelty of the proposed method, the completeness of the experiments, and the details of the ablation studies. During the rebuttal phase, the authors addressed some of these concerns through explanations and additional experiments. Therefore, I raised my score to 3.", "rebuttal": "We thank the reviewer for their insightful and constructive comments. We have carefully addressed each concern below and will incorporate improvements into the revised paper accordingly.\n\n### **W1. Clarify Contributions:**\n\nWe appreciate the opportunity to clarify our joint optimization framework. IMPACT's key innovation lies in its ability to simultaneously optimize **all critical patch factors**, including shape, location, number, and content, using a novel unified encoding.\n\nWhile we employ a two-part vector encoding $(\\mathbf{m},\\mathbf{r})$, **these two components are not optimized in isolation**. Instead, they form **a unified representation** and are **jointly optimized in a common solution space**. The mask vector $\\mathbf{m}$ implicitly defines patch shape, location, and number, while the content vector $\\mathbf{r}$ specifies the corresponding pixel values. Vectors $\\mathbf{m}$ and $\\mathbf{r}$ compose an individual (a solution), and jointly optimized by DE.\n\nOur work addresses the problem of how to optimally craft adversarial patches by considering all critical patch factors simultaneously. In contrast, prior work has typically simplified this problem by focusing on only a subset of these factors. Technically speaking, IMPACT's holistic treatment marks a significant departure from prior methods, which typically fix specific patch parameters (e.g., shape or location) or optimize them sequentially in isolation. Such decomposed approaches fail to model the complex interdependencies among patch factors, often leading to **suboptimal adversarial effectiveness** [1-3]. In contrast, IMPACT jointly evolves both the mask vector and the content vector in a unified search space. This enables discovery of synergistic configurations that adaptively balance spatial arrangement and patch content, resulting in more effective and flexible attacks across diverse scenarios.\n\n### **W2. Dataset:**\n\nThank you for suggesting broader dataset evaluation. We have now conducted additional experiments on **CIFAR-10** and **CIFAR-100** using ResNet-50. The results (see Table 1 below) confirm that IMPACT **generalizes well** beyond ImageNet, maintaining high attack success rates (ASR) and strong query efficiency. Following your helpful advice, we will include these results and analysis in the final version of the paper.\n\n|Dataset|Attack Type|Method|AQ|ASR (%)|\n|-|-|-|-|-|\n|**CIFAR-10**|Untargeted|Patch-RS|156.78|97.8|\n|||**IMPACT**|**113.98**|**99.2**|\n||Targeted|Patch-RS|2155.47|85.3|\n|||**IMPACT**|**1634.36**|**92.7**|\n|**CIFAR-100**|Untargeted|Patch-RS|98.52|98.1|\n|||**IMPACT**|**72.81**|**99.3**|\n||Targeted|Patch-RS|2310.80|82.5|\n|||**IMPACT**|**1556.17**|**90.1**|\n\n*Table 1: IMPACT on CIFAR-10 and CIFAR-100. Performance on ResNet-50, 5000 Query Budget, and 5% Perturbation Area.*\n\nNote that due to the **lower resolution of CIFAR images (32 × 32)**, we adjusted the encoding to use a 1 × 1 tiling. This provides maximum granularity while keeping the search space manageable, demonstrating the **adaptability of our framework**. As the results in the table demonstrate, our method achieves near-perfect ASRs of **over 99%** in the untargeted setting on both CIFAR-10 and CIFAR-100, **surpassing** the already strong performance of Patch-RS. Notably, IMPACT consistently achieves a high attack success rate **above 90%**, even in the more challenging targeted attack setting across both CIFAR-10 and CIFAR-100. These results show that IMPACT's effectiveness is not confined to a specific data distribution like ImageNet. It generalizes well to datasets with varying characteristics.\n\n### **W3. Code:**\n\n**We confirm that the complete, well-documented code will be released upon paper acceptance.** Our current supplementary code covers the **core components**.\n\n### **Q1. Definition of $p_i$:**\n\nWe thank the reviewer for pointing out this ambiguity. In our framework, the first phase of optimization utilizes a Differential Evolution (DE) algorithm. DE is a population-based algorithm. This means it maintains a **population**, which is a set of **candidate solutions**, and iteratively refines this population to find better solutions. We use $p_i$ to represent the $i$-th **individual** in the population, where $i \\in [1, N]$ and $N$ denotes the population size. Each individual $p_i$ in the population represents a candidate solution that encodes all the necessary information to generate the patches. As we state in the paper, this information is divided into two components: the encoded mask $\\mathbf{m}_i$ and the encoded content $\\mathbf{r}_i$.\n\nTo ensure this is clear in the final version of the paper, we will revise the paragraph starting at line 181. Furthermore, we will conduct a thorough review of the entire paper to identify and correct similar notational ambiguities.\n\nWe appreciate the reviewer's careful attention to detail, which has helped us strengthen the clarity of our paper.\n\n### **Q2. Analyze of the 4 × 4 Tiling Encoding:**\n\nWe sincerely thank the reviewer for this insightful question, which highlights a critical **trade-off** in our design. We elaborate below on the rationale for adopting the 4 × 4 tiling scheme, highlighting trade-offs across **granularity**, **dimensionality**, and **optimization efficiency**:\n\n- **Granularity vs. Dimensionality**: The drop in ASR with smaller tile sizes is due to the **increased optimization difficulty under limited queries**. A 1 × 1 tiling provides the finest possible granularity, allowing precise per-pixel patch shaping. However, this results in a prohibitively large binary vector (224 × 224 = **50,176 dimensions**), which severely hampers the search efficiency of population-based optimization methods under black-box query constraints. Conversely, coarse tiling (e.g., 8 × 8 or larger) significantly **reduces dimensionality but lacks flexibility** in representing complex patch contents or irregular patch shapes.\n\n- **Empirical Trade-Off Analysis**: We performed extensive ablation studies with multiple tile granularities and observed that **4 × 4 tiling offers a favorable balance**: it maintains enough expressiveness to represent irregular patches while compressing the search space to a tractable size. This enables effective convergence within **limited query budgets**.\n\n- **Compatibility with Random Aggregation**: The 4 × 4 tiling grid also integrates well with our random aggregation algorithm. **Smaller tiles would create a more fragmented starting point, complicating the clustering process, while larger tiles would limit the shape's expressiveness.** The 4 × 4 tiling enables contiguous, locally coherent regions to **form effectively**, enhancing the real-world printability of generated patches.\n\nWe will incorporate this expanded justification into Section 4.3 and include supporting ablation results. We thank the reviewer again for prompting this important clarification.\n\n### **Q3. Performance in the Physical World:**\n\nWe sincerely thank the reviewer for this important question. Demonstrating the physical-world applicability of our method is indeed a critical part of our evaluation. \n\nTo validate the physical-world effectiveness of our method, we conducted real-world experiments following Wang et al. [4]. Specifically, we randomly selected 100 images with adversarial patches and printed each on a 10cm × 10cm white paper. Then, using an iPhone 15, we photographed each printed image at various distances and angles. All photos were then resized to 224 × 224 pixels for ResNet50 model input. Table 2 reports the resulting ASR across the different shooting distances and viewing angles. Note that all these results are for **targeted attacks**. This means the goal was not simply to cause a misclassification, but to force the model to predict a specific, pre-determined incorrect class.\n\n|Model|Digital ASR|Angle|ASR at 10cm|ASR at 15cm|ASR at 20cm|\n|-|-|-|-|-|-|\n|ResNet50|53.0|0°|41.0|37.0|32.0|\n|||15°|35.0|31.0|26.0|\n|||30°|28.0|25.0|21.0|\n\n*Table 2: Physical-world performance of IMPACT under different conditions. The targeted attack success rates ASR (%) is reported across varying distances and viewing angles for ResNet50 model. The \"Digital ASR\" indicates the attack success rate of the same 100 selected images in the digital domain.*\n\nThese results confirm that our irregular patches **remain effective when physically deployed**. Even under the challenging condition (20cm distance and a 30° viewing angle), IMPACT still achieves a 21.0\\% ASR. It is worth noting that even a strong baseline like Patch-RS only achieves a 20.0% ASR in the digital-world targeted attack setting (Table 1 in our main paper). In the physical world, varying distances and angles inevitably reduce the patch's detail in the captured image. Consequently, a degradation in performance when moving from the digital domain to the physical world is an expected phenomenon. **Despite these challenges, our method demonstrates resilience.** This validates the practical applicability and physical feasibility of IMPACT.\n\nIn the final version of the paper, we will include the analysis of these real-world experiments. All physical-world images will be released upon acceptance of the paper.\n\n**We trust that our detailed responses have resolved your concerns. We would be sincerely grateful if you could kindly reconsider your assessment in light of these clarifications. Please feel free to let us know if there are any further questions. We would be happy to provide additional responses.**\n\n**Reference:**\n\n[1] Croce et al. Sparse-RS: A Versatile Framework for Query-Efficient Sparse Black-Box Adversarial Attacks. AAAI 2022\n\n[2] Chen et al. Query-Efficient Decision-Based Black-Box Patch Attack. TIFS 2023\n\n[3] Rao et al. Adversarial Training Against Location-Optimized Adversarial Patches. ECCV 2020\n\n[4] Wang et al. Breaking Barriers in Physical-World Adversarial Examples: Improving Robustness and Transferability via Robust Feature. AAAI 2025"}, {"review_id": "bNyYIBsJyv", "reviewer": "Reviewer_sVYk", "summary": "This paper introduces IMPACT, a novel framework for generating adversarial patches through the joint optimization of multiple critical factors (shape, location, number, and content). Addressing limitations of prior works that optimize these factors independently, IMPACT features three key innovations: (1) A dimensionality reduction encoding scheme that compresses the high-dimensional search space while preserving expressiveness; (2) A random aggregation algorithm that merges scattered modifications into physically realizable, contiguous irregular patches; and (3) A two-phase optimization approach combining differential evolution for global exploration with (1+1)-ES for local refinement. Extensive experiments on ImageNet demonstrate IMPACT's superiority over state-of-the-art methods", "questions": "1. Could you demonstrate the effectiveness of IMPACT-generated patches in physical-world scenarios (e.g., printed patches under varying lighting/viewing angles)?\n2. Analyze how the 4×4 tiling encoding performs on higher-resolution images (e.g., 512×512). Does the method maintain query efficiency?\n3. How does IMPACT compare if baselines (e.g., Patch-RS) are allowed similar query budgets for sequential optimization of shape/location/content?", "limitations": "Yes.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. Comprehensive experiments on multiple architectures (ResNet50, VGG16, ViT-B) with clear benchmarks against SOTA methods。\n2. Detailed ablation studies validating design choices (e.g., Table 2 shows 94.2% vs 90.2% ASR for full IMPACT vs DE-only).\n3. The two-phase optimization framework is well-motivated and clearly explained (Algorithm 1, Section 3.2–3.3).\n \nWeaknesses\n1. While IMPACT outperforms baselines, the absolute gains in ASR are modest (e.g., +2.8% for ResNet50)\n2. Comparison to white-box methods (Table 4) is less compelling since IMPACT-W uses gradient-based refinement.", "was_revised": true, "final_justification": "The authors partially solved my concerns. After reading the rebuttals and the reviews from other reviewers, I would like to keep the score.", "rebuttal": "Thank you for your valuable comments.\n\n### **W1. Modest Performance:**\n\nWe thank the reviewer for this comment. While the ASR gain may appear modest in isolation in some untargeted settings, it is accompanied by a **substantial reduction in query cost** (e.g., a significant decrease of 472 queries). In practice, higher query efficiency is often as important as peak ASR in black-box setting [1]. \n\nMoreover, **in targeted attacks**, where success is harder to achieve, IMPACT shows **significant gains** (e.g., 24.6% vs. 7.6% ASR on ResNet50 with 5k queries, 1% area).\n\nAs far as we are aware, IMPACT is the first method to unify all patch factors under a unified optimization framework. In our humble opinion, even modest ASR improvements reflect a nontrivial **architectural advance**.\n\n\n### **W2. White-box Comparison:**\n\nWe appreciate the reviewer's observation. In the white-box comparison, both our IMPACT-W variant and the baseline Patch-Fool utilize gradient-based optimization to refine the patch content. We designed this experiment to evaluate the contribution of our **patch mask generation strategy**.\n\nIMPACT-W and Patch-Fool differ only in mask generation: IMPACT-W uses DE to optimize irregular masks, while Patch-Fool uses saliency-based or random placement with fixed shapes.\n\nIn particular, both approaches use PGD to optimize content. Hence, we believe the superior performance of IMPACT-W directly demonstrates the benefit of jointly optimizing irregular mask structures in IMPACT-W.\n\nWe will clarify the motivation and conclusion of this experiment in the final paper to better reflect **the advantage of IMPACT's mask optimization strategy**.\n\n\n\n### **Q1. Performance in the Physical World:**\n\nWe sincerely thank the reviewer for this important question. Demonstrating the physical-world applicability of our method is indeed a critical part of our evaluation. \n\nTo validate the physical-world effectiveness of our method, we conducted real-world experiments following Wang et al. [2]. Specifically, we randomly selected 100 images with adversarial patches and printed each on a 10cm × 10cm white paper. Then, using an iPhone 15, we photographed each printed image at various distances and angles. All photos were then resized to 224 × 224 pixels for ResNet50 model input. Table 2 reports the resulting ASR across the different shooting distances and viewing angles. Note that all these results are for **targeted attacks**. This means the goal was not simply to cause a misclassification, but to force the model to predict a specific, pre-determined incorrect class.\n\n\n| Model    | Digital ASR   | Angle| ASR at 10cm  | ASR at 15cm  | ASR at 20cm  |\n|----------|-----------|------|-------|-------|-------|\n| ResNet50 | 53.0      | 0°   | 41.0    | 37.0    | 32.0    |\n|          |           | 15°  | 35.0    | 31.0    | 26.0    |\n|          |           | 30°  | 28.0    | 25.0    | 21.0    |\n\n*Table 2: Physical-world performance of IMPACT under different conditions. The targeted attack success rates ASR (%) is reported across varying distances and viewing angles for ResNet50 model. The \"Digital ASR\" indicates the attack success rate of the same 100 selected images in the digital domain.*\n\n\n\nThese results confirm that our irregular patches **remain effective when physically deployed**. Even under the challenging condition (20cm distance and a 30° viewing angle), IMPACT still achieves a 21.0\\% ASR. It is worth noting that even a strong baseline like Patch-RS only achieves a 20.0\\% ASR in the digital-world targeted attack setting (Table 1 in our main paper). In the physical world, varying distances and angles inevitably reduce the patch's detail in the captured image. Consequently, a degradation in performance when moving from the digital domain to the physical world is an expected phenomenon. **Despite these challenges, our method demonstrates resilience.** This validates the practical applicability and physical feasibility of IMPACT.\n\nIn the final version of the paper, we will include the analysis of these real-world experiments. All physical-world images will be released upon acceptance of the paper.\n\n### **Q2. Scalability for Higher-Resolution Images:**\n\nWe thank the reviewer for this excellent question regarding the scalability of our approach to higher-resolution images. This is a crucial aspect for practical applications, and we appreciate the opportunity to elaborate on it.\n\nOur 4 × 4 tiling strategy was **empirically optimized** for standard 224 × 224 ImageNet images, where it offers a practical balance between granularity and search space dimensionality.\n\nHowever, when applied to larger inputs such as 512 × 512 images, the resulting binary mask vector grows to 16,384 dimensions, over five times larger than for 224 × 224 images. This substantial increase imposes a much **heavier burden** on the DE-based optimizer, especially under fixed query budgets.\n\nAs evidenced in our ablation study (Table 3), **larger search spaces may degrade query efficiency**, even when spatial expressiveness improves, especially if the 4 × 4 scheme is applied naively to images with higher resolutions.\n\nThat said, we believe this limitation stems from using a fixed tiling granularity, not from our proposed framework itself. IMPACT is inherently **flexible** and supports modular adjustments to the encoding scheme. For example, a scalable variant could adapt the tile size to the input resolution (e.g., using 8 × 8 or 16 × 16 tiling for 512 × 512 images), preserving both search tractability and attack flexibility. We plan to explore such **adaptive strategies** in future work and will incorporate relevant discussions in the revised paper.\n\n### **Q3. Compare with Sequential Optimization:**\n\nThis is a insightful question that gets to the heart of the difference between our joint optimization approach and other exiting methods. We thank the reviewer for proposing this hypothetical comparison, as it allows us to highlight the fundamental advantages of IMPACT.\n\nFirst, we would like to clarify that the core methodology of Patch-RS is to fix the patch shape as a square and then optimize its location and content. Within this design, **'shape' is not treated as an optimizable variable**. This fundamental limitation is exactly the problem that IMPACT is designed to solve. Our framework is the first to treat shape, location, and content as **interdependent factors for joint optimization**. \n\nSecondly, even if a sequential optimization strategy were implemented, it would remain fundamentally limited compared to our joint optimization framework. This is due to the strongly **entangled nature** of all patch factors, including shape, location, number, and content. Optimizing these components in isolation, or in a fixed sequence, restricts the search to a **narrower subspace** and often results in **suboptimal configurations**.\n\nFor example, selecting a location and content under the assumption of a fixed square shape may yield a solution well-suited for that specific constraint. However, when the shape is later altered, the previously selected location and content may no longer be effective, leading to suboptimal attacks or even total failure. In contrast, IMPACT jointly optimizes all patch factors within a unified encoding space, allowing it to adaptively discover **synergistic configurations** that would likely be inaccessible to sequential or factor-decoupled methods. This tight coupling is critical to achieving high adversarial success with limited query budgets.\n\n\n**We trust that this response has clarified our work and addressed your valuable points. We would be deeply grateful if you would reconsider our work in light of these clarifications. Please do not hesitate to let us know if any questions remain.**\n\n**Reference:**\n\n[1] Chen et al. Query-Efficient Decision-Based Black-Box Patch Attack. TIFS 2023\n\n[2] Wang et al. Breaking Barriers in Physical-World Adversarial Examples: Improving Robustness and Transferability via Robust Feature. AAAI 2025"}, {"review_id": "I0FS09a1rb", "reviewer": "Reviewer_eyLv", "summary": "This work mainly focuses on how to generate adversarial patches to improve the success rate of adversarial attacks. The authors proposed a flexible adversarial attack framework based two phases optimization and introduced a new aggregation algorithm explicitly designed to produce contiguous, irregular patches. The experiments have shown the effectiveness of the method.", "questions": "1. Could the authors provide information about the differences in the calculation of consumption time between IMPACT and the previous method? \n2. I am confused about some of the function names in Algorithm 1, such as PopInit, Fitness, Mutation and so on. Could the author provide some more precise formulas in the main paper so that the reader can make sense of them.\n3. The results of IMPACT on some defense models in Table 5 are very good, how about the results on diffusion-based defense methods such as Diffpure[1]?\n4. Have the authors printed patches attacking some models in the physical world, as mentioned in line 73?\n\n[1] Nie et al. Diffusion Models for Adversarial Purification. ICML 2022.", "limitations": "No discussion of the time consumption of the proposed method compared to previous methods, and evaluation with more advanced defense models.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Paper strengths:\n1. Unlike the independent optimization of patch and position, this paper presents a novel joint optimization strategy based on evolutionary algorithms.\n2. They develop a new dimensionality reduction method that significantly reduces the encoding length while addressing limitations in existing approaches. Figure 2 gives a clear presentation of this approach.\n3. The experimental results demonstrate the effectiveness of the current method.\n\nPaper Weakness:\n1. The authors did not evaluate the time consumed for joint optimization using evolutionary algorithms compared to some previous work.\n2. There are some function names in Algorithm 1 that the authors do not explain in the main paper.", "was_revised": true, "final_justification": "The author has addressed my concerns in the rebuttal, and I am willing to raise my rating to “accept.”", "rebuttal": "Thank you for your valuable comments.\n\n### **Q1. Consumption Time:**\nWe thank the reviewer for questioning the **computational time**. To assess the computational overhead, we follow Williams et al. [1], measuring the average time required to successfully complete an attack on a single image. The experiments were conducted on a single NVIDIA RTX 4090 GPU under the untargeted attack setting against ResNet50 (5000 query budget and 2\\% perturbation area). The results are presented in Table 1.\n\n| Method   | ASR (%)         | AQ              | Runtime (s)     |\n| :------- | :-------------- | :-------------- | :-------------- |\n| IMPACT   | **94.2**        | **676**         | 29.16       |\n| Patch-RS | 89.8            | 982             | **1.78**      |\n\n*Table 1: Comparison of Performance and Computational Time\n(Untargeted Attack on ResNet50, 5000 Query Budget, 2% Perturbation Area).*\n\nAs shown in Table 1, IMPACT takes more time per attack (29.16s) than Patch-RS (1.78s), primarily because it employs a **population-based** optimization algorithm (Differential Evolution) that **evaluates multiple candidate solutions per generation**. In contrast, Patch-RS relies on a simpler random sampling strategy with lower per-iteration cost.\n\nDespite the longer runtime, IMPACT significantly outperforms Patch-RS in the metrics that matter most in black-box attack scenarios. Specifically, it achieves a significantly **higher attack success rate** (ASR: 94.2% vs. 89.8%) and requires substantially **fewer model queries** (676 vs. 982 on average), which is critical in real-world applications where each query is expensive and increases the risk of detection. In fact, according to our knowledge, **query count**, rather than computational time, is widely regarded as the primary bottleneck in black-box attacks. Many state-of-the-art methods [2-5] focus on minimizing query cost since queries dominate both efficiency and stealth.\n\nFinally, we note that IMPACT remains computationally practical: its average runtime (29.16s) is **more than an order of magnitude faster** than the 440s reported for CamoPatch [1], a state-of-the-art patch-based method.\n\nThus, while IMPACT incurs a higher runtime than Patch-RS, this is **justified by substantially better attack quality and efficiency**. In view of the newly reported results, we believe the runtime of IMPACT remains practical, particularly faster than other complex methods like CamoPatch.\n\n### **Q2. Function Explanation in Algorithm 1:**\n\nWe thank the reviewer. In the revised paper, we will add a concise summary of each function (PopInit, Mutation, Crossover, Aggregation, Fitness, Selection, SelectBest) immediately following Algorithm 1, with references to relevant appendices.\n\nTo address the reviewer's concern, we provide a concise summary of these functions here.\n\n- **PopInit** (Initialization, detail in the Appendix B.1): This function initializes a population $P_0$ of $N$ individuals to ensure diverse starting points for the search process. Each individual is structured according to our dimensionality reduction encoding scheme, consisting of two components: a binary encoded mask $\\mathbf{m}$ and a continuous encoded content $\\mathbf{r}$.\n\n- **Mutation** (Details in the Appendix B.2): This core DE function creates a mutant vector $v_i$ for each individual $p_i$ by differential combination of other population members. This process introduces new variations into the search.\n\n- **Crossover** (Details in the Appendix B.3): This function mixes the components of the parent individual $p_i$ and its corresponding mutant vector $v_i$ to create a new trial individual $u_i$. This enhances population diversity.\n\n- **Aggregation** (Details in the Section 3.2.2 Random Aggregation): This is a component of our framework. It takes the sparse binary mask $m$ from a trial individual and transforms its scattered active elements into $k$ physically contiguous, irregular patches.\n\n- **Fitness** (Details in the Appendix B.5): This function evaluates the quality of a trial individual. It reconstructs the full patch, applies it to the image, and calculates the model's cross-entropy loss. \n\n- **Selection** (Details in the Appendix B.6): This function compares the fitness of the trial individual $u_i$ with the parent $p_i$. The one with the better fitness score survives into the next generation's population.\n\n- **SelectBest**: This is a simple function that iterates through the final population and returns the single individual with the best fitness score found during the optimization.\n\n### **Q3. Performance on Diffusion-Based Defense Methods:**\n\nWe thank the reviewer for raising this question regarding diffusion-based defense methods like DiffPure. \n\nWe agree that testing against diffusion-based defenses is a **crucial next step**. In our initial experimental design, following Williams et al. [1], we focused on more established defense benchmarks, such as adversarially trained models and PatchGuard.\n\nWhile we have not yet conducted these specific experiments, we can offer an **analysis** of how IMPACT might interact with a defense like DiffPure. Diffusion-based defenses work by purifying the input image. They use a diffusion model to effectively remove high-frequency adversarial perturbations. This process fundamentally alters the gradient landscape, making it very difficult for gradient-based attacks to succeed.\n\nIMPACT generates **irregular**, **multi-patch** attacks which are **structurally very different** from the subtle, pixel-level noise that diffusion models are primarily trained to remove. It is plausible that our patches might partially survive the purification process, or that they could be optimized to create features that mislead the model even after purification. \n\nThe reviewer's question highlights an exciting avenue for future research. We are highly motivated to rigorously evaluate IMPACT against a suite of diffusion-based defenses in our future work. It is also worthwhile to emphasize that **the design and development of IMPACT itself represents a substantial technical contribution**. IMPACT introduces a novel encoding scheme and a joint optimization framework capable of evolving shape, location, and content of patches under strict black-box constraints. This unified approach addresses key limitations of prior methods and advances the state-of-the-art in patch-based attacks, independent of downstream defense evaluations.\n\n### **Q4. Performance in the Physical World:**\n\nWe sincerely thank the reviewer for this important question. Demonstrating the physical-world applicability of our method is indeed a critical part of our evaluation. \n\nTo validate the physical-world effectiveness of our method, we conducted real-world experiments following Wang et al. [6]. Specifically, we randomly selected 100 images with adversarial patches and printed each on a 10cm × 10cm white paper. Then, using an iPhone 15, we photographed each printed image at various distances and angles. All photos were then resized to 224 × 224 pixels for ResNet50 model input. Table 2 reports the resulting ASR across the different shooting distances and viewing angles. Note that all these results are for **targeted attacks**. This means the goal was not simply to cause a misclassification, but to force the model to predict a specific, pre-determined incorrect class.\n\n\n| Model    | Digital ASR   | Angle| ASR at 10cm  | ASR at 15cm  | ASR at 20cm  |\n|----------|-----------|------|-------|-------|-------|\n| ResNet50 | 53.0      | 0°   | 41.0    | 37.0    | 32.0    |\n|          |           | 15°  | 35.0    | 31.0    | 26.0    |\n|          |           | 30°  | 28.0    | 25.0    | 21.0    |\n\n*Table 2: Physical-world performance of IMPACT under different conditions. The targeted attack success rates ASR (%) is reported across varying distances and viewing angles for ResNet50 model. The \"Digital ASR\" indicates the attack success rate of the same 100 selected images in the digital domain.*\n\nThese results confirm that our irregular patches **remain effective when physically deployed**. Even under the challenging condition (20cm distance and a 30° viewing angle), IMPACT still achieves a 21.0\\% ASR. It is worth noting that even a strong baseline like Patch-RS only achieves a 20.0\\% ASR in the digital-world targeted attack setting (Table 1 in our main paper). In the physical world, varying distances and angles inevitably reduce the patch's detail in the captured image. Consequently, a degradation in performance when moving from the digital domain to the physical world is an expected phenomenon. **Despite these challenges, our method demonstrates resilience.** This validates the practical applicability and physical feasibility of IMPACT.\n\nIn the final version of the paper, we will include the analysis of these real-world experiments. All physical-world images will be released upon acceptance of the paper.\n\n**We hope that our clarifications have resolved the issues you raised. We believe the manuscript is significantly stronger thanks to your feedback, and we would sincerely appreciate it if you would take our responses into account in your final assessment. We remain available for any further questions you may have.**\n\n**Reference:**\n\n[1] Williams et al. CamoPatch: An Evolutionary Strategy for Generating Camouflaged Adversarial Patches. NeurIPS 2023\n\n[2] Chen et al. Query-Efficient Decision-Based Black-Box Patch Attack. TIFS 2023\n\n[3] Croce et al. Sparse-RS: A Versatile Framework for Query-Efficient Sparse Black-Box Adversarial Attacks. AAAI 2022\n\n[4] Fu et al. PATCH-FOOL: Are Vision Transformers Always Robust Against Adversarial Perturbations? ICLR 2022 \n\n[5] Ran et al. Cross-Shaped Adversarial Patch Attack. IEEE Transactions on Circuits and Systems for Video Technology 2024\n\n[6] Wang et al. Breaking Barriers in Physical-World Adversarial Examples: Improving Robustness and Transferability via Robust Feature. AAAI 2025"}, {"review_id": "l5CNhOEUOv", "reviewer": "Reviewer_Tn9c", "summary": "This paper introduces IMPACT, a novel adversarial patch attack framework that jointly optimizes all critical patch factors—shape, location, number, and content—through a two-phase, gradient-free optimization process. The authors propose a dimensionality-reduced encoding scheme and a random aggregation algorithm to produce irregular but physically feasible patches. The method demonstrates strong performance across multiple black-box attack settings and outperforms several state-of-the-art baselines in both untargeted and targeted attacks.", "questions": "Please refer to the Weaknesses section and address each point.", "limitations": "Even the proposed method belongs to the black-box AA category, the method assumes access to model confidence scores. This limits its applicability to real-world systems that only expose top-1 labels. Decision-based adaptation is mentioned as future work but not explored.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper tackles the underexplored problem of joint optimization over all patch attributes, including irregular geometry and multiple patch composition—areas often overlooked or simplified in prior works.\n2. The dimensionality-reduction strategy via 4×4 tiling and the stochastic patch aggregation allow for tractable search while producing diverse, physically printable adversarial examples.\n3. The paper evaluates the contribution of each phase, different optimizers, and tile granularity, supporting the design choices with data.\n\nWeaknesses:\n1. The method assumes access to model confidence scores. This limits its applicability to real-world systems that only expose top-1 labels. Decision-based adaptation is mentioned as future work but not explored.\n2. While effective empirically, the method lacks any theoretical insight into convergence behavior, success conditions, or the impact of irregular geometry on attack transferability.", "was_revised": true, "final_justification": "The paper provides solid experimental contributions, including physical validation, but novelty and theoretical depth are limited. Concerns on assumptions and runtime remain, so I keep Borderline Accept.", "rebuttal": "Thank you for your valuable comments.\n\n### **W1. Decision-Based Adaptation:**\n\nWe sincerely thank you for this important question. We agree that the decision-based setting is equally important and poses stricter constraints. However, the score-based setting **remains highly relevant in practice** [1-3], as many real-world systems (e.g., commercial APIs) expose confidence scores. Our focus on this setting reflects a deliberate effort to tackle the under-explored challenge of jointly optimizing multiple entangled patch factors under non-differentiable, black-box conditions. We believe this problem is both technically rich and practically impactful.\n\nMeanwhile, extending to decision-based attacks requires fundamentally different query strategies and objective formulations, as only the top-1 predicted label is available and class probabilities are inaccessible. This setting severely limits gradient-free optimization, especially when optimizing multiple entangled patch factors, as there is no feedback on directionality or partial improvement. Adapting IMPACT to this restrictive scenario would necessitate new strategies, such as label-guided boundary estimation or discrete search refinements, which go beyond the current scope but represent a **promising direction for future work**.\n\nIn the revised paper, we will expand the discussion of future work to explicitly include decision-based attacks. We will highlight the importance of this setting and outline how IMPACT could be extended to cope with the associated challenges. By positioning our work as a **foundation for such extensions**, we aim to emphasize the broader applicability and long-term potential of the IMPACT framework across diverse black-box threat models.\n\n### **W2. Theoretical Insight:**\n\nWe acknowledge that theoretical analysis would strengthen the paper. However, analyzing population-based algorithms like DE on high-dimensional, non-convex black-box objectives **remains an open challenge**. To this end, we focus on conducting a **comprehensive and in-depth empirical evaluation**, including detailed CAM visualizations and ablation studies, to support our design choices and hypotheses. \n\nThis empirical approach follows **a well-established practice in adversarial machine learning**, where many impactful papers published in top-tier conferences and journals [1-4] have similarly relied on **strong empirical validation in lieu of theoretical guarantees**.\n\nOur empirical results consistently demonstrate the effectiveness of irregular multi-patch attacks that are optimized jointly across shape, location, number, and content. This unified optimization framework enables more effective perturbations that significantly degrade model performance. The optimized patches can also disrupt learned saliency patterns by directly interfering with the model’s internal mechanisms for identifying and attending to salient input regions (i.e., saliency or attention attribution). Hence, our empirical study comprehensively and strongly supports the major claim of this paper that irregular, multi-patch configurations are particularly effective in **diffusing attention and triggering distributed misclassification**.\n\n**We hope our responses have fully addressed your concerns. We would deeply appreciate it if you would consider our clarifications in your final evaluation. Please feel free to let us know if there are any further questions. We would be happy to provide additional responses.**\n\n\n**Reference:**\n\n[1] Croce et al. Sparse-RS: A Versatile Framework for Query-Efficient Sparse Black-Box Adversarial Attacks. AAAI 2022\n\n[2] Williams et al. CamoPatch: An Evolutionary Strategy for Generating Camouflaged Adversarial Patches. NeurIPS 2023\n\n[3] Yang et al. PatchAttack: A Black-Box Texture-Based Attack with Reinforcement Learning. ECCV 2020\n\n[4] Chen et al. Query-Efficient Decision-Based Black-Box Patch Attack. TIFS 2023"}]

Author rebuttal:

Meta-review: IMPACT delivers an advance​ in adversarial patch attacks by unifying previously disjointed optimization tasks into a cohesive, efficient framework. Its encoding scheme and two-phase optimization resolve core limitations of prior work, achieving ​superior attack success​ with ​greater practicality. The authors' rigorous validation—spanning digital, physical, and cross-dataset settings—and responsive engagement with reviewers solidify this contribution. Given the technical novelty, empirical strength, and resolved reviewer consensus, I endorse ​acceptance.
Decision: Accept (poster)