Title: Exploring Semantic-constrained Adversarial  Example with Instruction Uncertainty Reduction
Venue: NeurIPS 2025
Accepted: yes
Keywords: Adversarial Example

Generative Models
Abstract: Recently, semantically constrained adversarial examples (SemanticAE), which are directly generated from natural language instructions, have become a promising avenue for future research due to their flexible attacking forms, but have not been thoroughly explored yet.
To generate SemanticAEs, current methods fall short of satisfactory attacking ability as the key underlying factors of semantic uncertainty in human instructions, such as $\textit{referring diversity}$, $\textit{descriptive incompleteness}$, and $\textit{boundary ambiguity}$, have not been fully investigated.
To tackle the issues, this paper develops a multi-dimensional $\textbf{ins}$truction $\textbf{u}$ncertainty $\textbf{r}$eduction ($\textbf{InSUR}$) framework to generate more satisfactory SemanticAE, $\textit{i.e.}$, transferable, adaptive, and effective.
Specifically, in the dimension of the sampling method, we propose the residual-driven attacking direction stabilization to alleviate the unstable adversarial optimization caused by the diversity of language references.
By coarsely predicting the language-guided sampling process, the optimization process will be stabilized by the designed ResAdv-DDIM sampler‌, therefore releasing the transferable and robust adversarial capability of multi-step diffusion models.
In task modeling, we propose the context-encoded attacking scenario constraint to supplement the missing knowledge from incomplete human instructions.
Guidance masking and renderer integration are proposed to regulate the constraints of 2D/3D SemanticAE, activating stronger scenario-adapted attacks.
Moreover, in the dimension of generator evaluation, we propose the semantic-abstracted attacking evaluation enhancement by clarifying the evaluation boundary based on the label taxonomy, facilitating the development of more effective SemanticAE generators.
Extensive experiments demonstrate the superiority of the transfer attack performance of InSUR.
Besides, it is worth highlighting that we realize the reference-free generation of semantically constrained 3D adversarial examples by utilizing language-guided 3D generation models for the first time.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper \"Exploring Semantic-constrained Adversarial Example with Instruction Uncertainty Reduction\" introduces a novel framework, InSUR, for generating semantic-constrained adversarial examples (SemanticAEs) directly from natural language instructions. It addresses the challenges of semantic uncertainty in instructions, categorized as referring diversity, descriptive incompleteness, and boundary ambiguity. The InSUR framework proposes three key contributions: (1) a residual-driven attacking direction stabilization using the ResAdv-DDIM sampler to enhance adversarial optimization stability in diffusion models, (2) context-encoded attacking scenario constraints to improve scenario-adapted 2D and 3D SemanticAE generation by integrating external knowledge, and (3) a semantic-abstracted attacking evaluation enhancement based on label taxonomy to provide robust evaluation metrics. The paper demonstrates improved transfer attack performance for 2D SemanticAEs and achieves the first reference-free generation of 3D SemanticAEs using language-guided 3D generation models. Extensive experiments validate the effectiveness of the proposed methods across ImageNet-based tasks and novel abstracted label evasion tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1、The paper presents a well-designed multi-dimensional framework (InSUR) that systematically addresses three forms of semantic uncertainty (referring diversity, descriptive incompleteness, and boundary ambiguity). The proposed techniques, such as the ResAdv-DDIM sampler, context-encoded scenario constraints, and semantic-abstracted evaluation, are theoretically grounded and build on established methods.\n\n2、The experimental evaluation is robust, testing the framework on ImageNet 1000-class label evasion and a novel abstracted label evasion task. The use of multiple surrogate models and transfer attack methods ensures a thorough assessment. The reported results, such as a 1.19× average attack success rate (ASR), demonstrate the framework’s effectiveness.\n\n3、Figures 2–6 and 9–11 effectively illustrate the framework, optimization challenges, and generated SemanticAEs. The visualizations of 2D and 3D samples (Figures 10 and 11) are compelling, showcasing the naturalness and adversarial effectiveness of the generated examples.\n\nWeaknesses:\n\n1、While the paper mentions potential real-world applications, it lacks empirical validation in physical-world scenarios (e.g., testing 3D SemanticAEs in simulated environments). This limits the immediate practical significance of the results.\n\n2、The 3D generation relies on the Trellis framework [41], which may reduce the perceived originality of the 3D pipeline, as it leverages an existing tool rather than proposing a fully new 3D generation method.\n\n3、Some components, such as the use of guidance masking for 2D generation, are less novel and resemble techniques in prior diffusion-based adversarial works (e.g., AdvDiff [16]). While the integration is creative, these elements may be seen as incremental improvements.\n\nQUESTIONS:\n1、Please clarify the residual-driven attacking direction stabilization mechanism. You should provide a detailed explanation of the residual term’s role in stabilizing adversarial direction. The ResAdv-DDIM sampler is a core contribution, proposed to stabilize adversarial optimization under instruction uncertainty by introducing a residual-driven attacking direction. The formulation (Eq. 2–4) adapts the DDIM process with a residual term and momentum optimization, but the technical details are dense, and the rationale for design choices (e.g., choice of momentum parameters) is not fully explained.\n\n2、This paper should include a quantitative analysis comparing ResAdv-DDIM to standard DDIM in terms of convergence stability (e.g., loss curves, gradient variance) or attack success rate (ASR) under varying instruction diversity.\n\n3、The paper emphasizes addressing instruction uncertainty (referring diversity, descriptive incompleteness, boundary ambiguity) but does not provide details on the natural language instructions used in experiments. The lack of information about instruction diversity (e.g., length, complexity, semantic variation) or their construction (e.g., manual, dataset-derived, or programmatic) makes it difficult to assess the robustness of the InSUR framework across varied inputs.\n\n4、Please validate or discuss real-world applicability of 3D semanticAEs.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper mainly try to addresses the problem of generating semantic-constrained adversarial examples (SemanticAEs) directly from natural language instructions, a challenging task that moves beyond traditional perturbation-based attacks. The authors identify a core challenge: \"instruction uncertainty,\" which they thoughtfully categorize into three key issues. To tackle these issues, the paper proposes a multi-dimensional framework named Instruction Uncertainty Reduction (InSUR). The authors conduct extensive experiments on 2D and 3D tasks, demonstrating that InSUR significantly outperforms existing state-of-the-art methods in generating transferable and effective SemanticAEs.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n1. The proposed InSUR framework is technically sound and novel. The ResAdv-DDIM sampler is a clever solution to the problem of unstable gradients in multi-step generative processes.\n\n2. Integrating a language-guided 3D generator with an adversarial objective to produce reference-free 3D objects that can fool 2D classifiers is a major step forward for the field and demonstrates the flexibility of the proposed framework.\n\n3. The authors compare against a strong set of recent baselines (AdvDiff, SD-NAE, VENOM) across multiple surrogate models (ResNet, ViT, ConvNeXt), demonstrating robust transferability.\n\n# Weaknesses\n\n1. Regarding the ResAdv-DDIM sampler (Section 3.2): Could you provide more intuition on the choice of $k$ (the number of coarse prediction steps)? Is there a principle for selecting the optimal $k$, or is it purely an empirical hyperparameter? \n\n2. Regarding 3D generation (Table 2): The results are extremely impressive, with ASR jumping to $92.2\\%$. The non-adversarial baseline already has a classification accuracy of $21.5\\%$ on the target model. Does this suggest that classifiers are inherently less robust to rendered 3D objects, making them an easier target for adversarial attacks compared to real 2D images? \n\n3. Regarding the relationship between referring diversity and descriptive incompleteness. The paper neatly separates these concepts. However, could there be an interplay? For example, if an instruction is incomplete (\"image of a car\"), does this increase the diversity of valid references, making the optimization problem harder? How does your framework handle this potential interaction?\n\nQUESTIONS:\nSee weakness\n\nLIMITATIONS:\nThe paper does not discuss the limitations of the proposed methods within the main text.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of generating semantically constrained adversarial examples (SemanticAEs) from natural language instructions, which suffer from instruction uncertainty due to referring diversity, descriptive incompleteness, and semantic boundary ambiguity. The authors introduce the InSUR framework (Instruction Uncertainty Reduction), which combines a novel ResAdv-DDIM sampler for stabilizing adversarial optimization, context-encoded constraints for adapting to incomplete instructions in both 2D and 3D settings, and a semantic-abstracted evaluation using label taxonomies. On the ImageNet abstracted label evasion task, the proposed method achieves a 55.4% attack success rate (ASR) with the ViT-B surrogate model, outperforming prior approaches in transferability and semantic fidelity (Table 1).\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n - The reviewer likes the approach of InSUR that introduces a multi-dimensional approach combining residual-based diffusion guidance, context-aware 2D/3D generation, and semantic evaluation via label taxonomies, offering a robust pipeline for reference-free, instruction-guided 3D adversarial generation.\n - The proposed method significantly outperforms prior methods (SD-NAE, VENOM, AdvDiff). For example, on the abstracted label evasion task with ViT-B surrogate, the proposed method achieves 55.4% ASR versus 33.6% for SD-NAE and 40.3% for VENOM, while maintaining high perceptual quality (e.g., LPIPS of 0.039).\n - The paper is clear and concise in writing.\n\n----\n**Weaknesses:**\n - While the paper evaluates transferability across six standard architectures (e.g., ResNet50, ViT-B/16, ConvNeXt), it does not test robustness against, foundation models like CLIP (despite using CLIP-IQA as a metric), as   semanticAEs are meant to probe misalignment in general perception models. The reviewer feels that restricting evaluation to classification networks undercuts the paper's broader alignment claims.\n - In the 3D pipeline, the choice to freeze the structured latent (z₀^slat) is heuristic. There’s no ablation to show whether optimizing over it would help or hurt performance.\n - Given that instruction-driven generation is central, it’s surprising that prompt attacks on vision-language systems like [1] ([13] in the paper) are not used as baselines. \n - Typos\n    - Line 39: \"SemanticAEhas\" → should be \"SemanticAE has\" (missing space).\n    - Line 151: \".. improve the approximation...\" → should be \"by improving\".\n    - Line 203: \"Senario knowledge...\" → should be \"Scenario knowledge\".\n    - Line 298: \"Tabel 2\" → should be \"Table 2\".\n----\n[1] An LLM can Fool Itself: A Prompt-Based Adversarial Attack, ICLR 2024\n\nQUESTIONS:\n- In Eq. 4, how sensitive is the ResAdv-DDIM sampler to the choice of k?\n - In Eq. 6, could you clarify how the thresholds ξ₁ = 0.1 and ξ₂ = 0.01 were chosen? Were these hyperparameters tuned for attack performance, or are they fixed across all tasks/models?\n - In Line 192–193, the author’s mention “M is the guidance masking that regularizes the ....\". It is unclear how M is defined. \n - In Table 2, given that generation times vary widely (e.g., up to 59s), is most of this cost from the Gaussian splatting renderer, diffusion steps, or EoT sampling? Can you attribute bottlenecks quantitatively?\n\nLIMITATIONS:\nyes (although provided in the Conclusion section)", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the topic of semantic-constrained adversarial examples. This paper develops a multi-dimensional instruction uncertainty reduction (InSUR) framework to generate more satisfactory SemanticAE. Their framework proposes residual-driven attacking direction stabilization, context-encoded attacking scenario constraint, guidance masking and renderer integration techniques to solve the uncertainty problems. Extensive experiments demonstrate the superiority of the transfer attack performance of InSUR.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n+ Semantic AE is interesting\n+ The approach is concrete\n\nWeakness\n\n-\tThe naturalness of adversarial examples cannot be guaranteed\n-\tThe real use case is unclear\n-\tDifferent abstracting levels should be discussed.\n\nQUESTIONS:\nHow to control the abstracting level of the antonyms of the Text. Cat is an antonym of dog, and car is also an antonym of dog, but they are different-level antonyms. How do you consider this kind of different-level antonyms?\n\nHow to guarantee the naturalness of the proposed adversarial attacks. We know that diffusion models can generate realistic images, but some of the details are unreal. Your approach utilizes the diffusion models, so how to guarantee the naturalness of the generated images?\n\nI am not sure about the real use cases of semantic AE. Is the semantic AE possible in the real world or does the semantic AE have real world application?\n\nI understand that the adversarial examples can be used for single object evaluation, but can you extend the approach for more complex scenarios in general VQA tasks, e.g., the image contains multiple objects? Can your adversarial examples be able to generate such adversarial images.\n\n(Minor) Typos Line39: SemanticAEhas => SemanticAE has\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-12"}