Title: IMPACT: Irregular Multi-Patch Adversarial Composition Based on Two‑Phase Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Adversarial Patch Attack

Evolutionary Algorithms

Irregular Shapes

Multi-Patch Attack

Black-Box Attack
Abstract: Deep neural networks have become foundational in various applications but remain vulnerable to adversarial patch attacks. Crafting effective adversarial patches is inherently challenging due to the combinatorial complexity involved in jointly optimizing critical factors such as patch shape, location, number, and content. Existing approaches often simplify this optimization by addressing each factor independently, which limits their effectiveness. To tackle this significant challenge, we introduce a novel and flexible adversarial attack framework termed IMPACT (Irregular Multi-Patch Adversarial Composition based on Two-phase optimization). IMPACT uniquely enables comprehensive optimization of all essential patch factors using gradient-free methods. Specifically, we propose a novel dimensionality reduction encoding scheme that substantially lowers computational complexity while preserving expressive power. Leveraging this encoding, we further develop a two-phase optimization framework: phase 1 employs differential evolution for joint optimization of patch mask and content, while phase 2 refines patch content using an evolutionary strategy for enhanced precision. Additionally, we introduce a new aggregation algorithm explicitly designed to produce contiguous, irregular patches by merging localized regions, ensuring physical applicability. Extensive experiments demonstrate that our method significantly outperforms several state-of-the-art approaches, highlighting the critical benefit of jointly optimizing all patch factors in adversarial patch attacks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces IMPACT, a novel adversarial patch attack framework that jointly optimizes all critical patch factors—shape, location, number, and content—through a two-phase, gradient-free optimization process. The authors propose a dimensionality-reduced encoding scheme and a random aggregation algorithm to produce irregular but physically feasible patches. The method demonstrates strong performance across multiple black-box attack settings and outperforms several state-of-the-art baselines in both untargeted and targeted attacks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper tackles the underexplored problem of joint optimization over all patch attributes, including irregular geometry and multiple patch composition—areas often overlooked or simplified in prior works.\n2. The dimensionality-reduction strategy via 4×4 tiling and the stochastic patch aggregation allow for tractable search while producing diverse, physically printable adversarial examples.\n3. The paper evaluates the contribution of each phase, different optimizers, and tile granularity, supporting the design choices with data.\n\nWeaknesses:\n1. The method assumes access to model confidence scores. This limits its applicability to real-world systems that only expose top-1 labels. Decision-based adaptation is mentioned as future work but not explored.\n2. While effective empirically, the method lacks any theoretical insight into convergence behavior, success conditions, or the impact of irregular geometry on attack transferability.\n\nQUESTIONS:\nPlease refer to the Weaknesses section and address each point.\n\nLIMITATIONS:\nEven the proposed method belongs to the black-box AA category, the method assumes access to model confidence scores. This limits its applicability to real-world systems that only expose top-1 labels. Decision-based adaptation is mentioned as future work but not explored.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work mainly focuses on how to generate adversarial patches to improve the success rate of adversarial attacks. The authors proposed a flexible adversarial attack framework based two phases optimization and introduced a new aggregation algorithm explicitly designed to produce contiguous, irregular patches. The experiments have shown the effectiveness of the method.\n\nSTRENGTHS AND WEAKNESSES:\nPaper strengths:\n1. Unlike the independent optimization of patch and position, this paper presents a novel joint optimization strategy based on evolutionary algorithms.\n2. They develop a new dimensionality reduction method that significantly reduces the encoding length while addressing limitations in existing approaches. Figure 2 gives a clear presentation of this approach.\n3. The experimental results demonstrate the effectiveness of the current method.\n\nPaper Weakness:\n1. The authors did not evaluate the time consumed for joint optimization using evolutionary algorithms compared to some previous work.\n2. There are some function names in Algorithm 1 that the authors do not explain in the main paper.\n\nQUESTIONS:\n1. Could the authors provide information about the differences in the calculation of consumption time between IMPACT and the previous method? \n2. I am confused about some of the function names in Algorithm 1, such as PopInit, Fitness, Mutation and so on. Could the author provide some more precise formulas in the main paper so that the reader can make sense of them.\n3. The results of IMPACT on some defense models in Table 5 are very good, how about the results on diffusion-based defense methods such as Diffpure[1]?\n4. Have the authors printed patches attacking some models in the physical world, as mentioned in line 73?\n\n[1] Nie et al. Diffusion Models for Adversarial Purification. ICML 2022.\n\nLIMITATIONS:\nNo discussion of the time consumption of the proposed method compared to previous methods, and evaluation with more advanced defense models.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces IMPACT, a novel framework for generating adversarial patches through the joint optimization of multiple critical factors (shape, location, number, and content). Addressing limitations of prior works that optimize these factors independently, IMPACT features three key innovations: (1) A dimensionality reduction encoding scheme that compresses the high-dimensional search space while preserving expressiveness; (2) A random aggregation algorithm that merges scattered modifications into physically realizable, contiguous irregular patches; and (3) A two-phase optimization approach combining differential evolution for global exploration with (1+1)-ES for local refinement. Extensive experiments on ImageNet demonstrate IMPACT's superiority over state-of-the-art methods\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. Comprehensive experiments on multiple architectures (ResNet50, VGG16, ViT-B) with clear benchmarks against SOTA methods。\n2. Detailed ablation studies validating design choices (e.g., Table 2 shows 94.2% vs 90.2% ASR for full IMPACT vs DE-only).\n3. The two-phase optimization framework is well-motivated and clearly explained (Algorithm 1, Section 3.2–3.3).\n \nWeaknesses\n1. While IMPACT outperforms baselines, the absolute gains in ASR are modest (e.g., +2.8% for ResNet50)\n2. Comparison to white-box methods (Table 4) is less compelling since IMPACT-W uses gradient-based refinement.\n\nQUESTIONS:\n1. Could you demonstrate the effectiveness of IMPACT-generated patches in physical-world scenarios (e.g., printed patches under varying lighting/viewing angles)?\n2. Analyze how the 4×4 tiling encoding performs on higher-resolution images (e.g., 512×512). Does the method maintain query efficiency?\n3. How does IMPACT compare if baselines (e.g., Patch-RS) are allowed similar query budgets for sequential optimization of shape/location/content?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper claims to propose a novel adversarial patch generation method that enables joint optimization of the patch’s position, number, shape, and content, without relying on gradient information. The core techniques include dimensionality-reducing encoding and a random aggregation algorithm. The proposed method shows significant improvements over baseline approaches on ImageNet.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The introduction of the proposed method is very detailed and easy to understand.\n2. The dimensionality-reducing encoding and random aggregation algorithms are simple yet effective.\n3. The experimental results are convincing.\n\nWeaknesses:\n1. The authors claim that their main contribution is proposing a method capable of jointly optimizing the position, shape, content, and number of adversarial patches. However, the method does not unify all these elements into a common feature space; instead, it simply uses two separate feature vectors to represent the various factors. This approach does not appear to differ significantly from previous methods.\n2. The experiments are conducted only on ImageNet, lacking diversity in datasets.\n3. The authors did not provide the full executable code—only code for key components—making it difficult to verify reproducibility.\n\nQUESTIONS:\n1. In line 181, what does pi refer to? Its definition is not explained.\n2. In the ablation study (Table 3), the authors claim that 4×4 is the optimal tile size. However, as mentioned by the authors in line 316, smaller tile sizes should offer a higher-dimensional search space and finer granularity. So why does the attack success rate decrease with smaller tiles?\n3. The paper repeatedly emphasizes the physical feasibility of the proposed method. Can experimental results be provided to demonstrate the advantages of the method in physical-world attacks?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}