Exploring Semantic-constrained Adversarial  Example with Instruction Uncertainty Reduction
Authors: Jin Hu
Jiakai Wang
linna Jing
Haolin Li
Haodong Liu
Haotong Qin
Aishan Liu
Ke Xu
Xianglong Liu
Venue: NeurIPS 2025 poster ()
Keywords: Adversarial Example
Generative Models
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: A multi-dimensional instruction uncertainty reduction (InSUR) framework is introduced to generate highly transferable and robust 2D & 3D adversarial examples directly from natural language instructions.

Abstract: Recently, semantically constrained adversarial examples (SemanticAE), which are directly generated from natural language instructions, have become a promising avenue for future research due to their flexible attacking forms, but have not been thoroughly explored yet.
To generate SemanticAEs, current methods fall short of satisfactory attacking ability as the key underlying factors of semantic uncertainty in human instructions, such as $\textit{referring diversity}$, $\textit{descriptive incompleteness}$, and $\textit{boundary ambiguity}$, have not been fully investigated.
To tackle the issues, this paper develops a multi-dimensional $\textbf{ins}$truction $\textbf{u}$ncertainty $\textbf{r}$eduction ($\textbf{InSUR}$) framework to generate more satisfactory SemanticAE, $\textit{i.e.}$, transferable, adaptive, and effective.
Specifically, in the dimension of the sampling method, we propose the residual-driven attacking direction stabilization to alleviate the unstable adversarial optimization caused by the diversity of language references.
By coarsely predicting the language-guided sampling process, the optimization process will be stabilized by the designed ResAdv-DDIM sampler‌, therefore releasing the transferable and robust adversarial capability of multi-step diffusion models.
In task modeling, we propose the context-encoded attacking scenario constraint to supplement the missing knowledge from incomplete human instructions.
Guidance masking and renderer integration are proposed to regulate the constraints of 2D/3D SemanticAE, activating stronger scenario-adapted attacks.
Moreover, in the dimension of generator evaluation, we propose the semantic-abstracted attacking evaluation enhancement by clarifying the evaluation boundary based on the label taxonomy, facilitating the development of more effective SemanticAE generators.
Extensive experiments demonstrate the superiority of the transfer attack performance of InSUR.
Besides, it is worth highlighting that we realize the reference-free generation of semantically constrained 3D adversarial examples by utilizing language-guided 3D generation models for the first time.

Exploring Semantic-constrained Adversarial Example with Instruction Uncertainty Reduction
Jin Hu! 1.2
Jiakai Wang 2
Linna Jing'
Haolin Li Haodong Liu
Haotong Qin Aishan Liu²
Ke Xu Xianglong Liu
State Key Laboratory of Complex & Critical Software Environment (CCSE), Beihang University 27hongguancun Laboratory SSchool of Computer Science and Engineering. Beihang University *Department of Information Technology and Electrical Engineering, ETH Zurich huj in@iuanuaa... edwaa.edducoom cn wangjkOmail. zgclab. edu. cn
Abstract
Recently, semantically constrained adversarial examples (SemanticAE)), which are directly generated from natural language instructions, have become a promising avenue for future research due their flexible attacking forms, but have not been thoroughly explored yet. To generate SemanticAEs,, current methods fall short of satisfactory attacking ability the key underlying factors of semantic uncertainty in human instructions, such as referring diversity, descriptive incompleteness, and boundary ambiguity, have been fully investigated. To tackle the issues, this paper develops muuti-diiensionall instruction uncertainty reduction (InsUR) framework to generate more satisfactory SemanticAAE, iee. transferable, adaptive, and effective. Specifically, in the dimension of the sampling method, we propose the residual-driven attacking direction stabilization to alleviate the unstable adver- sarial optimization caused by the diversity of language references. By coarsely predicting the aagguage-guidee sampling process, optimization process will be stabilized by the designed ResAdy-DDIM sampler, therefore releasing the transferable and robust adversarial capability multi-step diffusion models. task modeling, we propose the context-encee -eccdeed attacking scenario constraint supplement the missing knowledge from incomplete human instructions. Guid- ance masking and renderer integration proposed regulate the constraints 2D/3D SemanticAE, activating stronger senario-adapted attacks. Moreover, in the dimension of generator evaluation, propose the semantic-abstracted at- tacking evaluation enhancement by clarifying the evaluation boundary based on the label taxonomy, facilitating the development of more effective SemanticAF: generators. Extensive experiments demonstrate the superiority of the transfer at- tack performance of InSUR. Besides, it worth highlighting that realize the reference-fir generation of semantically constrained 3D ad adversarial examples by utilizing language-guidee 3D generation models for the first time.
Introduction
Adversarial example (AF), showing that small perturbations can impact the performance of deep learning models, is broadly focused due to its potential to promote model robustness and secure applications in practice. A series of studies has uncovered several forms of AEs, including physical- world AEs î 4], transfer AEs
55 3 and naturalistic AEs 7,, well the applications in
evaluating real-world recognition g autonomous driving [10) [1 12 or LLM systems [13 [4].
While most adversarial example research focuses on finding AES around existing data, generating AEs from natural language instructions without referenced data has not yet been thoroughly explored, i.e., to find Semantic- Constrained Adversarial Examples (SemanticAE). Specifically, given a certain
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

natural language description, we aim to generate the data that corresponds to its real semantic meaning but is hardly to be correctly recognized by deep learning models trained in related tasks. Recent works have employed techniques related to naturalistic AEs to accomplish a similar objective 13
16 17
but the de facto potential of SemanticAF, has still not been fully released performing transferable, adaptive, and effective attacks, limiting the applicability. In light of the recent advancements in language-driven multimodal intelligence and the increasing demand for alignment [88]
19 20), is
necessary to take step further in SemanticAE generation and facilitate more versatile AE generation.
To push the boundary of the current technology, we focus on the key underlying factor limiting the adversarial capability of Seman- ticAEs: the inherent uncertainty within human instructions that defines semantic constraints. We categorize three major forms of uncertainty in instructions: Referring diversity introduces barrier in SemanticAR. optimization via the multi-step generative models, since it leads the inconsistent aunguage-guidanee that the adversarial optimization should collaborate with. @ Descrip- tive incompleteness, which conceptualizes the gap between the precise model of the attack scenario and the instructions given by potential users, restricts the application scenarios. @ Boundary ambiguity of the semantic constraint is hard to characterize in task definitions, affecting the evaluation of SemanticAE generators.
"Image
"Imagee of Airship"
INSUR Generator
Surrogate Model
Targat
Transfer Robust
Figure 1: SemanticAEs are gen- erated directly by instructions.
We propose a multidimensional instruction uncertainty reduction (InSUR) framework to tackle the issues and generate more transferable, adaptive, and effective SemanticAF. Specifically, for referring diversity, we propose reridnal-driven attacking direction stabilization via the novel ResAdv- DDIM sampler that stabilizes optimization through coarsely predicting the language-guidet sampling process, releasing the capability of multistep diffusion models on adversarial transferability and robustness. For descriptive incompleteness, we propose the context-encoded attacking scenario constraint for both 2D and 3D generation problems by scenario knowledge integration, tackling the scenario adaptation problem by addressing the descriptions' incompleteness problem, achieving the first 3D SemanticAE generation. For boundary ambiguity, we propose the attacking evaluation enhancement based on label taxonomy. Our contribution can be summarized as:
We conceptualize the SemanticAE generation problem and propose multi-dimensional instruc- tion uncertainty reduction framework, InSUR, to address the challenges.
In the dimension of the sampling method, we propose the residual-driver attacking direction stabilization to achieve better adversarial optimization. In task modeling, we propose the context- encoded attacking scenario constraint to realize scenario-adapted attacks. In generator evaluation, we propose the semantic-abstracter attacking evaluation enhancement to facilitate the develop- ment of Semantic generators.
Extensive experiments demonstrate the superiority in the transfer attack performance of gen- erated 2D SemanticAEs, and for the first time, we realize the reference-free generation of 3D SemanticAE by utilizing language-guided 3D generation models.
2 Backgrounds
Adversarial Example Generation Adversarial attack generating algorithms can be categorized as iterative optimization the data space, e.g. FGSM [21], PGD (2), AutoAttack 23 iterative
optimization in the latent space of generative models, e.g. NAP 224, AdvFlow (25) and DiJPPG (7).
training a neural network for generation 26 27 , and applying zero-order optimizations 28% 9.
Adversarial examples may not be robust in the physical world. For physical attacks, expectation-over-
transformation (EoT) [29] and 3D simulation [30 31] are proposed to bridge the digital-physical gap. Research has also focused on attacking unknown models, i.e, transfer attacks 5 32) An extended technical background and related works are provided in Appendix A
Semantic-constrained Adversarial Example (33 first proposes the Unrestricted Adversarial Example(UABB) of which the restriction is defined by the human's cognition instead of lp no-m on existing data, and proposes generative learning method for its generation. line of studies further develops optimization techniques in latent spaces (4 24 35, and terms it as Natural Adversarial Example (NAE), while another line of study focuses on constructing the perceptual constraints for UAEs. A difference between them is that NAE studies also focus on generating

---

Reciduaf-Srive Direction Suubilization
Jestratiion
Scuurtic Scmumik-absracted Aftaching Evalortion
X
Mis
Xx
X
Create [Ttperf like Adverzorial Model
Noie Term
Reckdr-Diil udc Optraictioon
Contittnt
Longrage Gesled Prsorrter Sampling
Input
Optimization
Diffucion Guikted Consbrained Exploratin
Enhaneed Benchmark
Trranter Alladia
Rehoe
Encodin
Realitkk Rattorns
Atticka
Scmurtta
Rastern
Model
Mtokin
Spoir Achorcrrial Ennepple
Scenario Knowledge
29 Spakel Comstobe
In Anedining Alerlir Aupgeatio
Embedded
Mooil
Contest-enonled Attcking Scomario Constralut
Adversutaal Oitject
Eurmple
Proposol
Bllarr
Diffucion Sarpping
Metastolgg Application
Constraised Explonttios Wosliess
Effective Altacks under Semantic Coustraint
Advenuaal Quadicto
Figure 2: Overview of multi-dimensional instruction uncertainty reduction (InSUR) framework.
adversarial examples without referencing static image. We formulate the adversarial example generating task constrained by natural language's semantics as the semantically-
constrained adversarial example generation problem. Recent works ((7)5) 35 36 [6 17]) focus
integrating the pre-trained diffusion model and iterative optimization to constrain the naturalness and improve transferability. Furthermore, generating 3D adversarial examples that are more aligned with the physical world and satisfy the semantic constraints is still an open problem.
3 Methodology
3.1 Problem Formulation and Analysis
Adversarial Example (SemanticAE) Generation Problem We define SemanticAF. generation problem as generating an adversarial example Tady that fools the target model and satisfies the semantics constraint defined by the user's instruction Text. Formally, formulate the SemanticAE generation problem as follows:
find Tady S(Text) s.t. M((adv) E A
(1)
where S(Text) is the set of data with semantic meaning corresponding to Text, .M represents the target model, and AText defines the types of target model's output that conceptually different from Text, representing a successful attack. In the strict black-box setting, which is the focus of this paper, both S and M unknown the generation algorithm.
The goals of SemanticAE, generation are to build red-team model that automatically finds the alignment problem between the intelligent model M and the implicit semantics S(Text) reflecting the social consensus or the physical world, This goal leads to the following constraints: firstly, to achieve automatic alignment with limited supervision, the instruction Text is not required to characterize semantic constraints precisely. Secondly, from the perspective of data value, the generated SemanticAE Tadv should be able to perform transfer attacks.
Challenges in SemanticAE Generation As shown in the middle card of Figure 2 generative or diffusion models can constrain the pattern of generated AEs and facilitate transfer attacks with better in-manifold constraint [37]. We take a step further in SemanticAE, focusing the inherent challenge related instruction uncertainty:
Reference diversity challenges adversarial optimization. The language guidance that is learned from the mapping between Text and S ((Text) is non-linear since ((Txx)) diverse. This makes collaborating with adversarial optimization and the diffusion model for better transfer attacks and robust attacks non-trivial problem. 2 Descriptive incompleteness requires secnario-knoleedee integration for acenario-adapted generation. The challenges are iden- tifying the missing contexts in pretrained models and establishing practical knowledge embedding methodologies, thereby further eliminating the reference diversity from the external perspective. O Boundary ambiguity makes defining S and A for evaluating the generator also challenging, which lies in the fact that inappropriate evaluation leads to inaccurate results.
Multi-dimennional Instruction Uncertainty Reduction (InSUR) Framework As shown in Figure 2 for the reference diversity problem, propose the residual-driven attacking direction
3

---

stabilization with the designed ResAdv-DDIM sampler. For the contextual incompleteness, we propose the context-encoded attacking scenario constraint methods for gration in representative 2D and 3D SemanticAF, generation tasks. Moreover, since the unclear semantic boundary makes evaluating the generator difficult, semantic-abstracted attacking evaluation enhancement is proposed to facilitate further developments of SemanticAE, generation.
inte-
3.2 Residual-driven Attacking Direction Stabilization with ResAdy-DDIM
Semantic-constrained Optimization Problem Referring to the generatiee-model-osss adversarial examples, we solve SemanticAE generation by maximizing the loss LATK under the constraint of the posterior sampling process defined by the natural language guidance Text. However, if the posterior sampling process more complex, egg,, multi-step diffusion de-noising, tackling this maximization problem is challenging. Recent work utilizes deterministic sampling process, e.g., DDIM [38], as a constraint defined by Text [5]. For simplicity, we denote the sampling step as (1,) ~ qo (z [ Lo. Text), and such optimization can be formulated as: max LATK (M S0,AT ..... (IT)))
22
T/AT times
where denotes function composition. However, this may trigger the robust problem of f. since is hard to determine whether is an adversarial example of f or M. This results in the instruction misalignment of SD-NAE shown in section 4.4 Also, this optimization computationally expensive. Another solution is to tackle the challenges by approximating the gradient 7
or directly altering the
sampling process 16. which can be fermuuulltedd as:
Tt-A7 = J0,AT arg max LATK (MMTk(M(2())) VI (AT,4)) x;
where t₂ is a selected intermediate step, and the max,, optimization could be single iteration. An advantage is that, since the maximization algorithm does not retrieve the information of f. or has been protected from adversarial attacks, alleviates the robustness problem of f. However, the optimization used V 22 LATK to approximate LATKAT and the inconsistency introduces noise to the optimization. As shown in Figure 3 the optimization direction may vary, or be non-linear, with respect to different Zt. This misalignment makes the adversarial pattern optimized in the initial denoising stage ineffective in the latter stage, thereby limiting the multi-step regularization opportunity of diffusion models for better transfer attacks.
Overall, this technical challenge originates from the conflict between (1) the accurate estimation of V., LAT, causing the robust problems of the language guidance defined by f and the computational problems, and (2) the approximated estimation of V., CATK, causing the non-optimality of attack optimization. We solve the problem by improving the approximation with the novel ResAdv-DDIM posterior sampler, enabling the discovery of more robust adversarial patterns with better regularization.
Diffeent Adveriaiial
Original Adversarial Guidance: Optimiaaioon
Diverse Synthetic Data Modeled by Generative Models
Space of Effective Patterns
Diections
Linguage Condarce Advecataa Gu dance
Sampling Process
Residusi-GGil Sampling:
Direction Stabi zation Preducting Samping Hlsuuts
Figure 3: Inconsistent adv. direction problem.
Figure 4: Residual-driven attacking direction stabilization. ResAdv-DDIM is designed for the efficient and thorough exploration of new ad adversaiial patterns constrained by multi-step sampling processes.
Residual-Guided Adversarial DDIM Sampler Inspired by Learning to Optimize [39], we handle the challenge by predicting a coarse sketch of the future-step denoising result wo for estimating the attack optimization direction with LATK. Our key insight is that since multi-modal models acquire general capabilities through training in the task of predicting diverse human responses and complementing missing information across diverse data, should fully leverage the model's intrinsic multi- -uati-gra predictive capabilities to achieve stable generation under semantic uncertainty.
Specifically, we leverage DDIM's multi-step posterior sampling capabilities to achieve a coarse prediction of zo from current Ft, i.e., (0()),, which allows for more accurate estimate of V., LXTK compared to directly using V CATK. We formulate the generation process

---

k
go(x)) = fo,AT, o fo, .... S0, (It), where Σ
AT, =
times, kETTAT
i=1
(4)
PT = fo.AT (arg max
Vt t.).
The notation is the same Eq 2 go (x2))) is the coarse estimation of To. and & is small number of iterations that could be selected from {1, 2. 4). Since the sampling process takes a residual shortcut Tp. name it as Residual-Guider Adversarial DDIM Sampler (ResAdy-DDIM),
To further establish the concrete adversarial attack algorithm, we further propose the following method. (1) Constraining the Semanties. To ensure the generated sample satisfies Zadv E ((Txx)), we constrain the discrepancy the sample trajectory with the nn-norm between DDIM-generated samples and the adversarially optimized samples after determining Zx,;
[[Denoisennm((>., AT) Demoisead (æe, AT) <.
(5)
Such constrained optimization problem could be performed by simultancously sampling with DenoiSCpDIM and Denoiscadv- and clipping z in each step. (2) Adaptive Attack Optimization. To we introduce an early-stop mechanism that terminates optimization:
solve the maximization in Eq. [
(1) after the first iteration, and the estimated probability of unsuccessful attack < Cr,
(2) at the first iteration, when this probability satisfies stricter threshold & E1-
The probabilities are estimated from M (x+)). These conditions reduce the expected number and lower bound of optimization steps, while maintaining attack performance alongside the denoising step, which degrades the adversarial capability. Lower threshold could result in better attack performance and slower optimization, and we set 0.1. & 0.01 as a feasible setting across all experiments. In addition, we integrate momentum optimization, introduced in [7 5 to improve the attack transferability. Detailed implementation and analysis are shown in Appendix B.
3.3 Context-eneoded Attacking Scenario Constraint for 2D and 3D Generation
In the application scenarios, the instruction Text might be ambiguous incomplete, which requires integrating learned guidance with external knowledge. For effective task adaptation, we provide knowledge embedding strategies the key data structures that collaborate with the ResAdv-DDIM sampler, achieving better 2D SemanticAE generation and realizing 3D SemanticAE generation.
Spatial Constraint Complement for 2D SemanticAE Generation We focus on the problem of the incompleteness of the spatial constraint given by the instruction Text representing the object label. Specifically, an effective SemanticAE generator shall leverage the optimization space the image backgrounds and generate patterns that amplify the attack's 's effectiveness. However, the diffusion model's conditional background generation is overly uniform because the attack functionalities not considered during the original training. To solve the problem, encode the context of the attack application through the fine-graie tieeggrineed control of the denoising guidance. We leverage the application of guidance-masking from edit control of single guidance (40) the re-distribution of multiple guidance, and embed the masking into the key guidance function So applied in both
posterior sampling and adversarial optimization process in ResAdv-DDIM (as shown in Figure 2 Together with the deterministic DDIM, fa is formulated as:
fo (e, =ag-AT//a (z+ vI VI- (F)) + 11-5--Ar E0( 60 c0(x,, 1) = M) 60. (nconditional(::: + M 60, No,CCodiitioal (Ft, Text),
(6)
where a defines the noise ratio in the diffusion model, co is the noise estimating network, and M is the guidance masking that regularizes the spatial distribution of semantic guidance Text. Detailed definition of M is in Appendix B. and the integration is illustrated in Figure 2
Differentiable Rendering Pipeline Integration for 3D SemanticAE Generation 3D Data is (3D)
valuable for world modeling [1] 42). We focus on the problem of generating 3D SemanticAE. x target models M operated under the 2D inputs. fill the gap between the 3D scenarios 2D target models, additional physical rendering knowledge shall be efficiently encoded. Leveraging the proposed ResAdy-DDIM sampler and the advancements in Gaussian-sii caussian-splatting [42), we fill the gap between the 3D generation and 2D target models. As shown in Figure
optimize latents with the
5

---

"Dog"
(0
30 Gaussin Model
Attack
Image
Lads 1
Language Model
Insrurtion Erbscdding
ResAdh- DDIM
Gaussian Model Decader
Gansian Benderer
Target Model M
Gemeration
Gradient Estimation
3D Space
2D Space
Figure S: 3D optimization pipeline. Scenario knowledge encoded with the Gaussian renderer.
gradient pocck-ropaggted through the 3D data structure and integrate the scenario knowledge during the differentiable rendering process. The optimization pipeline could be implemented concisely with the Trellis [4] framework, which is recently published diffusion-basel 3D access generation framework. For simplicity, we reformulate the Trellis sampling and rendering process as:
pos = Coords( stat slae(22) (2olat))) Modelgs = Das(=u, pos). x ReRnereergs (Modclas, Camera), Des! {(2), pos")) {[( c stt
where Zn and zglat are latents sampled by the diffusion model, and represented by sparse and dense tensors, respectively. Datat the coarse structure decoder, Coords transforms the voxel to point positions pos, DGs the refined structure decoder that decodes each vertex into multiple Gaussian points and Renderergs renders the Gaussian model to 2D images æ with the camera parameter. For SemanticAE generation, the refined feature generation process { AT- .., =03 is replaced with ResAdv-DDIM sampling in Eq. with the rendering model embedded in go:
go (t,, pos, Camera) = Renderergs (Das(fe,AT) o .... fo, (2,, pos), pos), Camera)
(arg AT fe. (arg max ECamera Peam [CTT (MTK((M(p) (Mg (go Camera, pos)))!) (8)
We the EoT method with gradient accumulation to optimize 2, for unknown camera positioning. Le., samples Camera from Pcam in each iteration. The scenario knowledge is encoded in Pcam and the rendering background. Due to the stabilized guidance ResAdv-DDIM, the gradient of the previous steps could be utilized current-step gradient estimation, and therefore, fewer EoT steps are required. Since texture and localized positioning perturbation are enough for adversarial attacks, slat is not included in the parameter space and serves semantic anchor. With the collaborative constraint of 3D diffusion and renderer model, comantically-constraiine and multi-view adapted SemanticAEs are efficiently generated. Detailed implementation is shown in Appendix B.2
40
3.4 Semantic-abstracter Attacking Evaluation Enhancement with Label Taxonomy
The evaluation of SemanticAE generator requires the benchmark to judge whether æ E S(Txxt and detine AText, which determines the adversarial attack and semantic alignment performance of the generator, and is still blank in practice. To address the issue, we provide a task construction method for automatic evaluation based on the application goal of the SemanticAE generation task. Note that our method evaluates the SemanticAE generator instead of the adversarial example.
Hyponymic Graph Construction WordNet Taxonomy
Abstraction Level Selection
Attack Goal: Evade Coarse Label
Ovee-coorse
Eluaraation
SemanticAE
Aniral
A
B
Fish
Goldfish
Fivas Wane
Stels
Perception N
Xadv
Semantic
ImageNeetLbbiss
Real Semantics of
Constraints
Figure 6: Construction of abstract label evasion evaluation task.
The task constructed based on semantic abstracting with the label taxonomy. Firstly, the attack targets of existing non-target evaluation methods based on ImageNet labels are often too simple, while the constraint space SemanticAE is relatively loose, making it easy for the attack generation model G to achieve successful attacks easily. For example, it is unreasonable to use the ImageNet label "tiger-shark" as the misclassification category AText for the instruction Text "great- white-shark", since achieving successful attacks in this task may not show the capability of successful attacks in real scenarios. To clarify the boundary, we re-construct the evaluation label with a better abstraction level by leveraging WordNer [44) taxonomy. As shown in Figure 6 firstly construct the hyponymic

---

graph based on the hyponymic relation defined by WordNer, then select the proper abstraction level, and finally define the attack goal as the evasion attack on the abstracted label. Specifically, under the definition of SemanticAF generation, the evaluation task is formulated as:
Text == "Realistic image of |bstractedLabbll, specifically, [label]", Text = {labeladv AbstractedLabel & Ancestors(labelaa.))
AbstractedLabel E E E L' 31 E s.t. c E Ancestors(I) Count-Chitdren (c) 0},
where LL is the transitive closure of ImageNer labels on the hyponymic graph, the construction of A Labe is equivalent
(1) Remove overly coarse-grained labels through annotation to obtain the label subset T/. (2) For each linear path, select the node with the lowest height as a candidate label, constraining the upper bound of the abstracting level. (3) Eliminate descendant labels of the candidate labels to constrain the lower bound of the abstracting level.
Secondly, from the perspective of semantic constraint evaluation, using another deep-learning model for evaluation, egg., CLIP, will limit the benchmark to the robust region such models. Drawing upon previous discussions and attempts in evaluation enhancement [17. further conceptualize the sub-task of non- adversarial exemplar generation. As shown in Figure 6 the adversarial generator G is required to simultancously generate nearby sample Texemplar E Xexemplar proof that Tadv complies with semantic constraints. We further propose the evaluation method based on attack success rate and pair-wise semantic metric complement to the single-image assessments:
Attack Success (I adv) A Classification Correct oxemmlar)
(
Rtelative
E [0, (10)
K- Accuracy (ecuuray(Xeexmphaar Nexemplar)
SemanticDiffs = =x (Texemplar; S.
where K' is the amount of samples, S is visual similarity metric, such as LPIPS or MS- SSIM Measurinn local similarity is easier since high-level feature extraction, which could be attacked, is less required. By assuming the generator G is not motivated towards finding positive adversarial example, achieving a high score on both metrics can sufficiently show both adversarial capability and instruction compliance of G. Notably, AS AReaaivv evaluation metric imposes a more rigorous as- sessment for the masked language guidance in Section as it eliminates the confounding variations in benign example generation methods through regularizing with Classification Correct (Zeeepl(Ze
4 Experiments
4.1 Experiment Settings
Tasks and Baselines We evaluate different generation methods by generating 6 samples for each label in the ImageNet 1000-class label evasion task and the proposed abstracted label evasion task. The baseline method is constructed by combining diverse O Surrogate models, @ Transfer attack methods (MI-FGSM (5). DeCoWA (45),, and @ Diffusion-based naturalistic AE generation methods (AdvDiff [6), SD-NAE [15), VENOM (17). Note that DeCoWa develops on top of MI-FGSM. For fairness, we incorporate the same pretrained diffusion model as SD-NAE and VENOM. For 3D SemanticAE we evaluate the classification models' performance on the generated video showing the rotating object. Detailed implementation and settings are shown in Appendix C
Evaluation Metrics Referenced image quality assessment metrics, including LPIPS [46] and MSSSIM [47] are been employed to measure the similarity between Texemplar and Zadv under the proposed aon-adversarial exemplar evaluation. Also, we supplement the non-reference image quality assessment CLIP-IQA (48) for the generated images. We do not FID and IS as a primary metric since their adapted vision backbones are simple and might be adversarially attacked, and keeping the feature distribution consistent is not the primary goal of the SemanticAE generation. For attack evalu- ation, we computed the classification accuracy and AS ARRelttvee defined in Eq. 10 across diverse targets set T = [ResNe150 [4), Vi7-B/16 [50), ConvNeXI-T [51), R5sNer Inception V3 (52). SyiinTransformer-a [53] }. The first three models are used individually as surrogates in experiments. The main paper presents the average Rrelavve and accuracy (ACC) on the target model, while the detailed transfer attack performance on different target models is shown in Appendix D.I In addition, we fuse the imputtrransformatinn based module DeCoWa with ResNet50 as one of the surrogate models to evaluate the collaboration capability of the generation algorithms. 2D / 3D generation times are benchmarked on single 4090 or A800 GPU, respectively. by generating 100 samples with abstracted labels, and are presented in the results with standard deviation.
7

---

YIMMN GENAS
AVR. Toleet E 5.
12MP (Loves Botee)
EPPS CLosen Serte)
LPMP (Laner Bocke
LPMS (Loers Beneo
Figure 7: ImageNet label results.
Figure 8: Abstracted label results.
Table I: Results on more surrogate models, Appendix D shows our Pareto optimality in each setting.
Attacker Settings
ImageNer Label SemanticAE Generation Tack LPIPS
Course Label Evasion
Task (Section 3.4
Surmggats
Method
Azc
ASR
Clipp
MSSSIM
Acc.
ASR
Clipo!
MSSSIM:
TPIPS.
Time(s)]
MI-FGSM
33.4%
41.5%
0,548
0.850
0.201
61.3%
22.8%
0.551
0.885
0.198
1.43
0.634
0.621
19.6.00
ResNet50
SD-NAF
37.1%
47.4%
0.841
0.435
0.457
58.698
31.8%
0.771
0.5
D:308
24 24.4
VENOM
14.5%
34.4%
0.795
0.972
0.023
$1.055
14.9%
0.779
0.951
D.043
3.09
35.2%
47.9%
0.808
0.958
2.26
MI-FGSM
32.7%
42.4%
0.524
0.855
0.205
58. 2%
25.7%
0.521
0.860
0.207
1.462001
AdvIbif
65 6%
30.3%
0.638
0.430
0.390
94. 156
2.2%
016228
0.972
0.026
20. 20.52001
VIT-B
51.7%
0.844
56.15
0.787
24.520:00
VENOM
30.5%
40.6%
0.796
0,977
0.021
46.31g
40.3%
0.780
0.958
0.040
3.07 3.07
Qurs
10.9%
69.7
0.815
0.956
0.038
28.7%
55.4%
0.814
0.955
0.039
7.23
MI-FGSM
0.543
1.46 1.46_oo
AdvDiff
$2.9%
44.4%
0.636
0.312
0.471
93.356
3.2%
0.627
0.985
0.017
19.9
ConvNeXr
SD-NAF
22.4%
67 3%
0.848
0.432
0.458
53.69
36, 1%
0.7882
0.003
0.308
24. 24.5
VENOM
0.796
0.785
3.1
Ours
9.1%
75.8%
0.817
0.958
0.036
28.6%
57.4%
0.812
0.957
0.036
6.96 98
MI-KGSM
23.6%
58 8%
0,535
0.869
0.201
27.9%
63.7%
01-474
0.870
0.247
2.01
ResNer50 DeCoWW
SD-NAE
26.6%
67.9%
61.7%
0.597
0.845
0.421
0.470
64.4%
26.3%
0.782
0.568
0.351
24.3
20.8
VENOM
36.1%
31.4%
0.805
0.968
0.027
56.2%
28.1%%
0.796
0.944
0.048
4,93
Ours
10.1%
75.8%
0.810
0.947
0.044
30.35
57.4%
0.808
0.943
0.048
10.72800
Attack performance averaged across targets. Detailed results with the specified target models shown in Appendix
4.2 Overall Performance Evaluation
2D SemanticAE Generation The overall results on 2D Semantic generation is shown in
Figure 7 and 8 with the strength of semantic constraint of our method set in {1.5, 2, 2.5, 3} and = (2, 3,4] via Eq. respectively. Multiple e are applied since it is difficult to control and align the distortion strength for baselines. The min/max ASR across target models and the standard deviation of LPIPS across generated images are plotted as bars. Table [ shows the results of our method set = 2.5. More results in the appendix. Overall, in any of the 4 surrogate and 2 task settings, InSUR is able to achieve at least 1.19x average ASR and 1.08x minimal ASR across all target models in T. and maintains with lower LPIPS (unsuccessful baseline generation with avg. ASR < 5% are not considered), showing the consistent superiority. The Pareto improvement shown by the figure is more significant. Moreover, 1 c-based semantic constraint in Eq. Jachieess more consistent LPIPS across images generated in identical settings. @ For Clipo. our method performs better in the challenging abstracted-label evasion task, while SD-NAB higher in original tasks.
3D SemanticAF. Generation We export the video visualization of the object under MPEG4 encoding, and evaluate the attack performance by reading it. The surrogate and target models are both ResNet50. The results are in Table 2 There is no 3D Semanti- cAE previously available. It shows that our results show satisfactory attack performance, validating the cross-task scalability of InSUR. Note that since 3D-difuusion research is still under development, the clean accuracy on the generated 3D samples is not high, while making InSUR a growable research.
Table 2: 3D generation results.
Generator
Acc.
ASR
MSSSIM
LPIPS
Non-Adversarial
Ours w/o ResAdv
17.9%
45.1%
0.658
0.261
Ours 2.8%
92.2%
0.665
0.258
4.3 Key Ablation Studies
Residual Approximation We evaluate the effects of residual approximation steps à in the adversar- ial guidance g in Eq[4 under the 2D abstracted label task and the 3D task. In implementation, l is controlled by the upper-bound parameter K Eq. 13 in Appendix [B. The performance improve- ment is significant and consistent compared to the result without future-step sampling prediction (K = 0), which represents the original DDIM sampling with adversarial guidance. The naturalness metrics, including Clipo, MSSSIM, and LPIPS, are also slightly improved. Morcover, attributed to
8

---

the adaptive iteration mechanism, more accurate estimation leads to lower optimization steps, and therefore, the increase time consumption is sub-linear. Detailed results on more settings are shown in the Appendix D.3 Parameter and time consumption analysis are shown with the implementation details in Appendix
The effect of different k on the process of adversarial optimization is shown in Figure 9 The setting of Text is a dog sitting on the floor, and the evasion label is dog. As increases, the estimated ASR after adversarially sampling 2 increases earlier. Since the white-box ASR in both settings is nearly 100%, the improvements are from: (I) by eliminating the referring diversify, the initial sampling steps receive more accurate guidance. (2) More effective adversarial optimization on earlier diffusion denoising steps provides better on on-manifold regularization, leading to better visual quality and better adversarial transferability.
Table 3: Ablation of residual approximation
Surro.
Acc.
ASK
Clipo
MSSSIM
LPIPS
Time
43.1%
0.794
0,941
0.055
2.5
VIT-B
33,5%
54.7%
0.812
0.942
0.049
6.87=1.48
31.3%
56.6%
0.816
0,942
0.049
7.44
29.2%
57.5%
0.807
0.943
0.048
7.75=2.07
27.7%
60.1% %
0.813
0.944
0.047
7.87
ResNer50 +DeCoWa
39.8%
47.1%
0.764
0,946
0.053
5.65 20:38
36.3%
50.4%
0.812
0.954
0.040
9.64-22.4
32.8%
52.6%
0.818
0.955
0.039
10.7
30.4%
53.9%
0.817
0.955
0.039
11.2
29.5%
54.2%
0.814
0.955
0.0399
11.5 11.00
Figure 9: Estimated ASR of zo after
17.9%
45.1%
0.658
0.261
2.4
adv. sampling ot, under different set- tings of & in Eq. 4 Earlier increase of estimated ASR leads to better natural-
3.4%
90.2%
0.658
0.262
30.4:35.02
3D
3.4%
91.2%
0.659
0.261
32.7
Gen.
2.9%
91.2%
0.671
0.255
25.1250.76
4
2.8%
92.2%
0.665
0.258
40.1 2:50.:50.26
ness and higher attack transferability.
Spatial Masking of Language Guidance We evaluate the effect of guidance masking by setting Meage/ MMid to different values in Eq. 6 and setting to 1.0 represent removing the masking. The results are shown in Table 4 and Figure 1O indicating that (1) decreasing Medge leads to an increase in unconditional guidance, which enriches the background diversity as shown in the figure, and may lead to the improvements of Clipo. (2) When the budget e is small (E 2) and the optimization space is relatively narrow, diversifying the background is beneficial. Note that e is large, the improvement is marginal since there is no need for expanding the optimization space. Overall, the guidance masking design improves diffusion models' adaptability to strongly constrained Semantic generation.
Table 4: Ablation of guidance masking.
Instraction: Realistic image of boat, specifically, canoe. Evasion label: beot.
Metgel Maia
Acc:
ASR
Clipo
MSSSIM
LPIPS
0.0
0.813
0.958
0.035
2
0.1
34.5%
50.2%
0.809
0.957
0.035
2
1.0
36.9%
48.3%
0.788
0.958
0.034
0.1
0.4
0..
1.01
4
0.0
16.6%
75.9%
0.804
0.901
0.087
Figure 10: Vis. of different guidance masking.
1.0
17.3%
75.9%
0.775
0.904
0,084
The value under images denotes Modge/ /MMia-
4.4
Visualization of Generated SemanticAEs
We selected the 2D/ / 3D samples with Texemplar correctly classified and visualized in Fig. 11 and Fig. 12 2D samples are generated with the original ImageNet-label and the surrogate model is DeCoWa+Res
3D samples are generated and visualized as the main experiment. The results are coherent with the main table, showing that MI-FGSM is not natural, SD-NAE disturbs more global semantics, while ours achieves both global semantic preservation and naturalness. Through observation, our method generates local in-manifold patterns to achieve the strong attacks, e,g, adding fog in the castle image altering the lightning in the jellyfish image. For 3D results, although the MSSSIM and LPIPS metrics are not excellent in the main experiment, the generated 3D objects are natural and follow the semantic constraint. More visualizations shown in Appendix E
4.5 Discussions
Extending to Attacking Large Vision Language Models The InSUR framework is agnostic to specific target models and tasks, and can be directly applied to adversarial attack evaluation

---

Oriin
Curs
50
Rablit
Olyynt Seiie
Figure 11: Comparison of 2D SemanticAEs.
Figure 12: 3D SemanticAEs
in Visual Question Answering (VQA) scenarios. To illustrate the capability, we conduct VQA experiments using OpenAl CLIP (54) model RN50 (with DeCoWA augmentation) as the surrogate, using
[55] the diffusion generator, and generate adversarial examples for the VQA problem. The question and options are "What is in the picture:" and (Police car, Ambulance, Taxi, School Bus/ We evaluate weak-to-strong transferability across OpenAl CLIP variants (RN50, RN101, , RN50x4, RN50x16 16, ViT-B/32, ViT-B/16) and larger vision language models (VLM), LLaVA [56] and Owen-7B (57]. The results in Table [5] showing that technical contribution is also effective in the VQA transfer attack task.
Table 5: ASRyelative (%) of attacks. The surrogate model is ResNet50 CLIP model.
Residual Approxination
Target Model VIT-B/32
RN50
RN101
RN50x4
RN50x16
VTT-B116
LLaVa1.5-7B
Qwen2.5VL-7B
X(K 0)
77.50
27.50
30.00
11.11
12.50 40.00
10.53 47.37
11.11 22.22
16.22
/ (K
3)
97.50
$7.50
62.50
27.78
35.14
Broader Applications Our approach also has the po- tential to extend to real-world adversarial attacks under the 3DGS representations (II, providing technical tools for security-related research. We tested our generated 3D SemanticAE in with lightning, material, and camera set- tings with Blender environment, and the ResNet50 ACC 59.1 %, 20.0%, for the two settings in Figure [3] re- spectively. Although the attack performance is worse than the original rendering (ACC=7.7%). the relative improve- ment is significant. Our framework also be integrated with existing 3D scene generation pipelines [58), enabling more effective incorporation of diffusion-based adversarial optimization into world models. This supports safety- -afety-critical scenario generation for applications such as autonomous driving [59, (60) and embodied agents [6]]. On another front, recent work [62] demonstrates that diffusion-genett generated hard samples can achieve higher sample efficiency, especially with adversarial guidance, suggesting that InSUR framework also holds promise for supporting the generation of adversarial training data.
(a) w/o ResAdv.
(b) with ResAdv.
Figure 13: Blender re-rendered 3D Se- manticAEs with altered lightning.
Conclusion
This paper proposes multi-diimensoonll uncertainty reduction frameworks for SemanticAE generation, pushes the boundary of the 2D generation, and opens the door to 3D generation. The proposed technology consistently improves the attack performance and has the potential to scale to other tasks. Moreover, believe it could provide valuable insights for the test-time scaling of the red-teaming framework. For limitations, there is scope for improvement in the generation quality, the evaluation on larger models, and the application the real world, suggesting future research avenues of the SemanticAB generation algorithms based on concrete generative models and the scenario adaptation methods oriented to real-world scenarios.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the National Natural Science Foundation China (NSFC) under Grant 62476018, and was supported by Zhongguaneun Laboratory.
References
[1] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint arXiv: 1607.02330 2016.
[2] Aishan Liu, Xianglong Liu, Jiaxin Fan, Yuqing Ma, Anlan Zhang, Huiyuan Xie, and Dacheng Tao. Perceptual sensitive gan for generating adversarial patches. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 1028-1035, 2019.
[3] Jiakai Wang, Aishan Liu, Zixin Yin, Shunchang Liu, Shiyu Tang, and Xianglong Liu. Dual attention suppression attack: Generate adversarial camouflage in physical world. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8574,, 2021.
[4] Hui Wei, Hao Tang, Xuemei Jia, Zhixiang Wang, Hanxun Yu, Zhubo Li, Shin ichi Satoh, Luc Van Gool, and Zheng Wang- Physical adversarial attack meets computer vision: A decade survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):9797-911 2024.
[5] Yinpeng Dong, Fangzhou Liao, Tianyu Pang. Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9185-9193, 2018
[6] Yu-Chih-Tuaa Hu, Bo-Han Kung, Daniel Stanley Tan, Jun-Cheng Chen, Kai-Lung Hua, and Wen-Huang Cheng- Naturalistic physical adversarial patch for object detectors. Proceedings of the IEEE/CVF international conference computer vision, pages 7848-7857, 2021.
[7] Haotian Xue, Alexandre Araujo, Bin Hu, and Yongxin Chen. Diffussin- based adversarial sample generation for improved stealthiness and controllability. In Proceedings of the Advances in Neural Information Processing Systems, volume 36, pages 2894-2921, 2023.
[8] Shunchang Liu, Jiakai Wang, Aishan Liu, Yingwei Li, Yijie Gao, Xianglong Liu, and Dacheng Tao. Harnessing perceptual adversarial patches for crowd counting. In Proceedings of the 2022 ACM SIGSAC conference on computer and communications security, pages 2055 2069,, 2022.
[9] Aishan Liu, Jun Guo, Jiakai Wang. Siyuan Liang, Renshuai Tao. Wenbo Zhou, Cong Liu, Xianglong Liu, and Dacheng Tao. X-Adv: Physical adversarial object attacks against x-ray prohibited item detection. In 32nd USENIX Security Symposium (USENIX Security 23), pages 3781 3781-3798, 2023.
[10] Zelun Kong, Junfeng Guo, Ang Li, and Cong Liu. Physgan: Generating physical-world-reslleee adversarial examples for autonomous driving. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 14254-14263, 2020.
[11] Wenhao Ding, Chejian Xu, Mansur Arief, Haohong Lin, Bo Li, and Ding Zhao. A survey on safety-critical driving scenario generation a methodological perspective. IEEE Transactions on Intelligent Transportation Systems, 24(7):6971-6988 2023.
[12] Chen Ma, Ningfei Wang. Qi Alfred Chen, and Chao Shen. Slowtrack: Increasing the latency of camera-based perception in autonomous driving using adversarial examples. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4062-4070, 2024.
[13] Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint 10454 2023.
11

---

[14] Jia-Yu Yao, Kun-Peng Ning, Zhen-Hui Liu, Mu-Nan Ning, Yu- Yang Liu, and Li Yuan. Lim lies: Hallucinations are not bugs, but features as adversarial examples. arXiv preprint arXiv:2110.01469, 2023.
[15] Yueqian Lin, Jingyang Zhang, Yiran Chen, and Hai Li. SD-NAE:
Generating natural adversarial examples with stable diffusion. In The Second Tiny Papers Track at ICLR 2024, 2024.
[16] Xuclong Dai, Kaisheng Liang, and Bin Xiao. Advdiff: Generating unrestricted adversarial examples using diffusion models. In European Conference on Computer Vision, pages 93-109. Springer, 2024.
[17] Hui Kuurila-Zhang, Haoyu Chen, and Guoying Zhao. Venom: Text-driven unrestricted adver- sarial example generation with diffusion models. arXiv preprint urXiv:2501.07922 2025.
[18] Songtao Li and Hao Tang. Multimodal alignment and fusion: A survey. arXiv preprint arXiv:2411.170k 2024.
[19] Yixu Wang, Jiaxin Song, Yifeng Gao, Xin Wang, Yang Yao, Yan Teng, Xingjun Ma, Yingchun Wang, and Yu-Gang Jiang. Safevid: Toward safety aligned video large multimodal models. arXiv preprint 2rXiv:2005.11926 2025.
[20] Shiji Zhao, Ranjie Duan, Fengxiang Wang. Chi Chen, Caixin Kang, Shouwei Ruan, Jialing Tao, YueFeng Chen, Hui Xue, and Xingxing Wei. Jailbreaking multimodal large language models via shuffle inconsistency. ICCV, 2025.
[21] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adver- sarial examples. In International Conference on Learning Representations, 2015.
[22] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu, Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
[23] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pages 2206-2216. PMLR, 2020.
[24] Yu-Chih- Tuan Hu, Bo-Han Kung, Daniel Stanley Tan, Jun-Cheng Chen, Kai-Lung Hua, and Wen-Huang Cheng. Naturalistic physical adversarial patch for object detectors. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7848-7857, October
2021.
[25] Hadi Mohaghegh Dolatabadi, Sarah Erfani, and Christopher Leckie. Advflow: Inconspicuous black-box adversarial attacks using normalizing flows. In Neural Information Processing Systems, volume 33, pages 15871-15884, 2020.
[26) Chaowei Xiao, Bo Li, Jun-Yan Yan Zhu, Warren He, Mingyan Liu, and Dawn Song. Generating adversarial examples with adversarial networks. In Proceedings of the International Joint Conference on Artificial Intelligence, pages 3905-3911, 2018
[27] Jin Hu, Xianglong Liu, Jiakai Wang, Junkai Zhang, Xianqi Yang, Haotong Qin, Yuqing Ma, and Ke Xu. DynamicPAE: Generating scene-awand physical adversarial examples in real-time. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
[28) Pin- Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM workshop on artificial intelligence and security, pages 15-26, 2017.
[29] Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adver- sarial examples. In International conference on machine learning, pages 284-293. PMLR, 2018.
[30] Yanjie Li, Bin Xie, Songtao Guo, Yuanyuan Yang, and Bin Xiao. A survey of robustness and safety of 2d and 3d deep learning models against adversarial attacks. ACM Comput. Surv., 56(6), January 2024.
12

---

[31] Tianrui Lou, Xiaojun Jia, Siyuan Liang, Jiawei Liang, Ming Zhang, Yanjun Xiao, and Xiaochun Cao. 3d gaussian splatting driven multi-view robust physical adversarial camouflage generation. International Conference on Computer Vision, 2025.
[32] Jiakai Wang, Xianglong Liu, Zixin Yin, Yuxuan Wang, Jun Guo, Haotong Qin, Qingtao Wu, and Aishan Liu. Generate transferable adversarial physical camouflages via triplet attention suppression. Int. J. Comput. Vis., 132(1 1:5004-51100, 2024.
[33] Yang Song, Rui Shu, Nate Kushman, and Stefano Ermon. Constructing unrestricted adversarial examples with generative models. In Proceedings of the Advances in Neural Information Processing Systems, volume 31, 2018.
[34] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15262-15271, 2021.
[35] Xinquan Chen, Xitong Gao, Juanjuan Zhao, Kejiang Ye, and Cheng-Zhong Xu. Advdiffuser: Natural adversarial example synthesis with diffusion models. In Pruceedings of the IEEF/CVF International Conference on Computer Vision, pages 4562-4572 2023.
[36] Jianqi Chen, Hao Chen, Keyan Chen, Yilan Zhang, Zhengxia Zou, and Zhenwei Shi. Diffusion models for imperceptible and transferable adversarial attack. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(2):961-977, 2025.
[37] Yanbo Chen and Weiwei Liu. A theory of transfer-based black-box attacks: Explanation and implications. In Proceedings of the Advances in Neural Information Processing Systems, volume 36, pages 13887-13907, 2023.
[38] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
[39] Tianlong Chen, Xiaohan Chen, Wuyang Chen, Howard Heaton, Jialin Liu, Zhangyang Wang, and Wotao Yin. Learning to optimize: A primer and benchmark. Journal of Machine Learning Research, 23(18)):1559 2022.
[40] Guillaume Couairon, Jakob Verbeek, Holger Sehwenk, and Matthicu Cord. Diffedit: Diffusion- based semantic image editing with mask guidance. In The Eleventh International Conference on Learning Representations, 2023.
[41] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications the ACM, 6561 2021.
[42] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4):139- 2023.
[43] Jianfeng Xiang, Zelong Lv, Sicheng Xu, Yu Deng, Ruicheng Wang. Bowen Zhang, Dong Chen, Xin Tong, and Jiaolong Yang. Structured 3d latents for scalable and versatile 3d generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 21469-21480, 2025.
[44] George Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39-4) 41, 1995.
[45] Qinliang Lin, Cheng Luo, Zenghao Niu, Xilin He, Weicheng Xie, Yuanbo Hou, Linlin Shen, and Siyang Song. Boosting adversarial transferability across model genus by deformation- constrained warping. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 38, pages 3459-3467, 2024.
[46] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 586-595, 2018
13

---

[47] Zhou Wang, Eero P Simoncelli, and Alan C Bovik. Multiscale structural similarity for im- age quality assessment In The Phrity-Seventl Asilomar Conference on Signals, Systems & Computers, 2003, volume 2, pages 1398-1402. leee, 2003.
[48] Jianyi Wang, Kelvin CK Chan, and Chen Change Loy. Exploring clip for assessing the look and feel of images. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 2555-2563, 2023.
[49] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[50] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[51] Zhuang Liu, Hanzi Mao, Chao-Yuaa Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 11976- 11976-19986, 2022.
[52] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zhigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818 118-2826,222 2016.
[53] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[54] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8788888 8763. PmLR, 2021.
[55] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, June 2022.
[56] Haotian Liu, Chunyuan Li. Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Proceedings of the Advances in neural information processing systems, volume 36, pages 11892-34916 2023.
[57] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint 2rXiv:2502.1392. 2025.
[58] Beichen Wen, Haozhe Xie, Zhaoxi Chen, Fangzhou Hong, and Ziwei Liu, 3d scene generation: survey, arXiv preprint wxXiv2500.00547. 2025.
[59] Yuping Wang, Shuo Xing, Cui Can, Renjie Li, Hongyuan Hua, Kexin Tian, Zhaobin Mo, Xiangbo Gao, Keshu Wu, Sulong Zhou, et al. Generative ai for autonomous driving: Frontiers and opportunities. arXiv preprint 2xXi:2505.0885. 2025.
[60] Chejian Xu, Aleksandr Petiushko, Ding Zhao, and Bo Li. Diffscene: Diffusion-based safety- critical scenario generation for autonomous vehicles. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 8797-8805, 2025.
[61] Xinjie Wang, Liu Liu, Yu Cao, Ruiqi Wu, Wenkang Qin, Dehui Wang, Wei Sui, and Zhizhong Su. Embodiedgen: Towards generative 3d world engine for embodied intelligence. arXiv preprint arxiv:2506. 10600, 2025.
[62] Reyhane Hemmm,,, Mohammad Pezeshki, Elvis Dohmatob, Florian Bordes, Pietro Astolfi, Melissa Hall, Jakob Verbeek, Michal Drozdzal, and Adriana Romero-Soriant Improving the scaling laws of synthetic data with deliberate practice, International Conference on Machine Learning. PMLR, 2025.
14

---

[63] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
[64] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In International Conference on Machine Learning, pages 11221-32252, PMLR, 2023.
[65] Zhengyang Geng, Mingyang Deng, Xingjian Bai, J Zico Kolter, and Kaiming He. Mean flows for one-step generative modeling. In Proceedings of the Advances in neural information processing systems, 2025.
[66] Viraj Prabhu, Sriram Yenamandra, Prithvijit Chattopadhyay, and Judy Hoffman. Lance: Stress- testing visual models by generating aogguage-guided counterfactual images. In Proceedings of the Advances in Neural Information Processing Systems, volume 36, pages 25165-25184, 2023.
[67] Chenshuang Zhang, Fei Pan, Junmo Kim, In So Kweon, and Chengzhi Mao. Imagenet-d: Benchmarking neural network robustness diffusion synthetic object. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21752-21762, 2024.
[68] Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, Utkarsh Tyagi, S. Sakshi, Sanjoy Chowdhury, and Dinesh Manocha. ASPIRE: language-guided data augmentation for improving robustness against spurious correlations. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thai- land and virtual meeting, August 11-16, 2024, pages 386-406. Association for Computational Linguisties, 2024.
[69] Xiaopei Zhu, Peiyang Xu, Guanning Zeng. Yinpeng Dong, and Xiaolin Hu. Natural language induced adversarial images. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 10872-10881, 2024.
[70] Wenkai Yang, Shiqi Shen, Guangyao Shen, Wei Yao, Yong Liu, Zhi Gong, Yankai Lin, and JonR Wen. (aaerfffiia)-alinmmenn Strong models may deceive weak models in weak-to- strong generalization. In The Thirteenth International Conference on Learning Representations, 2025.
[71] Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attacks. In International Conference on Learning Representations, 2017.
[72] Martin Gubri, Maxime Cordy, Mike Papadakis, Yves Le Traon, and Koushik Sen. Lgv: Boosting adversarial example transferability from large geometrie vicinity. In European Conference on Computer Vision, pages 603-618. Springer, 2022.
[73] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness- aware min- imization for efficiently improving generalization. In International Conference on Learning Representations, 2021.
[74] Kunyu Wang, Xuanran He, Wenxuan Wang. and Xiaosen Wang. Boosting adversarial transfer- ability by block shuffle and rotation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 22336,2444, 2024.
[75] Xin Wang, Jie Ren, Shuyun Lin, Xiangming Zhu, Yisen Wang, and Quanshi Zhang. A unified approach to interpreting and boosting adversarial transfcrabiliity In The Ninth International Conference Learning Representations, 2021.
(76) Guandao Yang, Xun Huang, Zekun Hao, Ming- Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4541-4554 2019.
[77] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. Proceedings of the European conference on computer vision, pages 52-67, 2018
15

---

[78] Chaowei Xiao, Dawei Yang, Bo Li, Jia Deng, and Mingyan Liu. Meshadv: Adversarial meshes for visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pages 6898-6907, 2019.
[79] Yao Huang, Yinpeng Dong, Shouwei Ruan, Xiao Yang, Hang Su, and Xingxing Wei. Towards transferable targeted 3d adversarial attack in the physical world. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 24512 -45222, 2024.
[80] Yanjie Li, Kaisheng Liang, and Bin Xiao. UV-attack: Prysical-worle adversarial attacks on person detection via dynamic-neRf-based UV mapping. In The Thirteenth International Conference Leurning Representations, 2025.
[81] Sergey Kastryulin, Jamil Zakirov, Denis Prokopenko, and Dmitry Dylov. Pytorch image quality: Metrics for image quality assessment. arXiv preprint arXiv:2208. 14818, 2022.
[82] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free evaluation metric for image captioning. In Proceedings of the Empirical Methods in Natural Language Processing, pages 7514 7514-7528, 2021.
16

---

Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The primary claims of the paper are addressed with the provided methodology and the experiment evaluates the effectiveness.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
We discuss the limitations in the conclusion section of the generation perfor- mance, the evaluation on larger models, and the application the real world
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
17

---

Answer: [NA]
Justification:
No theoretical results are included in the main paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We make it clear how to reproduce that algorithm new algorithm. The detailed implementation is provided Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
18

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We will release the code after publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ praips.cc/ubblic/guuueee CoodeSumeeieeeeee ssi iessonPolll mollicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:[Yor Yes]
Justification: We specify the Experimental setting in the Experimental section and will include the details Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
We have selected statistical significance verification methods based on the properties of the metrics, including the standard deviation, extremum difference across different task settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
19

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the execution time Table I and on a single 4090 or A800 GPU. Detailed experiment settings are shown in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttt:://neut http:///wwrrrpps.ppppeeeeeeeeeee http://s.....pple/cccccccccdddd.
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with
theNeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Societal impacts are expounded upon in the discussion section in the appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
20

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have reflected Licenses for existing assets in our references.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
21

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
22

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy buttps://neure wwwpariipp..ccoooeeeeeeeeeec for what should or should be described.
23

---

Contents
Introduction
2
Backgrounds
3
Methodology
3.1
Problem Formulation and Analysis
3.2
Residual-driven Attacking Direction Stabilization with ResAdv-DDIM
3.3
Context-encoded Attacking Scenario Constraint for 2D and 3D Generation
3.4
Semmntic-abstracteed Attacking Evaluation Enhancement with Label Taxonomy
4
Experiments
7
4.1
Experiment Settings
4.2
Overall Performance Evaluation
4.3
Key Ablation Studies
4.4
Visualization of Generated SemanticAEs
4.5
Discussions
Conclusion
10
Technical Background and Related Works
25
A.I
Diffusion models
25
A.2
Generating Hard Samples from Language
25
A.3 Transfer Attack Methodology
26
A.4
3D Generation and Gaussian Splatting
26
Detailed Implementation of InSUR Framework
26
B.1
2D Image Generation
26
B.2 3D Object Generation
29
B.3 Construction and Analysis of Semantic- Abstracted Evaluation Task
remantic-Abstractee
31
Detailed Experiment Settings
33
C.1 2D Experiment Settings
33
3D Experiment Settings
34
D Detailed Results and Discussions
35
D.I Transfer Attack Analysis
35
D.2 Attack Robustness Analysis
36
D.3 On the Role of Residual-driven Attacking Direction Stabilization in 2D and 3D
SemanticAE Applications]
36
D.4 On the Potential Adversarial Transferability to Semantic Evaluator
38
Results Visualization
39
E. 2D Visualized Results
39
3D Visualized Results
40
Boarder Impacts
41
24

---

A Technical Background and Related Works
A.1 Diffusion models
This work mainly applies the diffusion model as the language-guided data generator. As brief review, diffusion models establish the theoretical and technical route of estimating the score function x p(gp(X) of the data distribution X by training UNet models on the dataset {x), and sampling the data from the score function with numerical methods. One of the key insights of diffusion models is alleviating the training difficulty by disturbing the data x gradually with noise, and models it as a forward process from xo X to JT (0, with the Markov process Xx = -I _,X,-1 + /B,N(O, /) scheduled by B. The training is performed by learning a denoising process, i.e., learn to predict the distribution (0 (arz) with the neural network eg. During inference, the data is sampled from 2T ~ NNO,, 7) to I. step by step with q. DDIM model reinterprets the forward process as 2 To). which abandons the Markov property, and constructs the sampling method by finding qe (0(z-11 1 σo). It also provides the theoretical grounding for the step jumping the sampling process. The DDIM [8] sampling procedure could be formulated as (deterministic version):
Diffusiio Model
TT-AT Vā, V /ā (z+ v1 (xeox, + V -----AT--e
(II)
where ā = -I ==1 (1 (11--1),, 3,), and AT is the step interval. We adapted this notation in the main paper. The language- -ugiddddg generation problem is modeled by adding the conditional term Context that corresponds to the data x and sampling the data by Eg t, Context), where Context is the encoding given by the language model. To balance the generation diversity and the instruction following, conditional and unconditional guidance are integrated, Lee., 6g = -wro (x,, f, Unconditional) + (1
(x) it Context), in the classifier-free guidance 63
Discussions In the application in content edit 40 masking has been applied to constrain the
editing area, i.e. It-1 = Mask eeiit + (1 Mask) 2T -riiginnaall- In our 2D generation task, we
take further step to model the interaction between conditional and unconditional guidance, and achieve the new function of re-distributing the spatial strength of the semantic constraint. In recent years, training techniques that enable single-step generation have been proposed [64] 665) Although our method is designed with multi-step diffusion models, our method still has practical value since (I) there is a performance gap between multi-step and single-step diffusion models, and (2) as the adversarial optimization is performed with local perturbations, sampling in small step sizes might provide a better regularization for transfer attacks.
A.2 Generating Hard Samples from Language
Language-guided adversarial example generation is still under development. We categorize two major paradigms. Perturbing the input text without altering the language-guider data generation model. A line of study has focused on utilizing the language-guided image edition model to evaluate the robustness of visual models [66] 67 688. Compared to adversarial attacks, they focus more on enriching the dataset than finding hard examples for victim models, while the adversarial capability is relatively limited. Related to the discussion in Section 3.2 of the main paper, overly perturbing the text guidance may lead to unsatisfactory semantic alignment. To solve this problem, [69] integrates the additional CLIP supervision on the generated image. @ Altering the language-guin -uigged data generation model by introducing the adversary capability. Although it has difficulties in global optimization, this paradigm has an advantage in the fine-grained control of the generated samples.
We focus on the latter paradigm, since (1) Perturbing the text input only will limit the capability of the generation method within the semantic knowledge learned by both discriminative and generative models. e.g.& a CLIP-based semantic supervision may not attack the CLIP model itself. (2) Optimizing the input text for the entire data generation process with the adversarial feedback time-consuming. These features are crucial from the perspective of AuperAAignment [70), i.e, creating efficient methodologies that utilize small models with limited capability to build the oversight framework for larger models. Also, we believe that the two methods could be implemented into an integrated red-team system, i.e, the language-space reject sampling as the high-level generation method and the latent-space optimization as the mid or low-level generation module. Therefore, our proposed evaluation task focuses on the second paradigm, serving as an evaluation of the key component of the red-team framework.
25

---

A.3 Transfer Attack Methodology
Transferable attack denotes finding adversarial examples that can attack other unknown or even stronger black-box models, which is practical for revealing real-world Al safety problems. The general method is to find a surrogate model(s) in the related task and optimize the adversarial example for it with the regularization. From the perspective the optimization pipeline, regularization on three modules has been investigated to improve transferabilityy O The gradient decent. Momentum is the general method boost the transferability. enhance transferabilityy by combining gradient features from multiple distinct models 77 72 Constructing a proper objective using the Sharpness-Awe Aware Minimization (SAM) 73 or the attention mechanism B to model the ross- vulnerability. is also beneficial for transferability. 0 The input transformation. Transfer improvement could be achieved by inserting a random transformation between the current-step adversarial example and the surrogate model's input, the [74 45 This could be regarded as improving the diversity of the surrogate model. From the theoretical perspective, decreasing the cooperation within the adversarial pattern 775] and in-manifold constraint [37] is beneficial for transferability. In the adversarial example generation pipeline, the generative-moder based method, which this work focuses could be regarded replacement of gradient descent with a better in-manifold constraint. We construct baselines and evaluation tasks based on this background.
The surrogate model. Model ensemble can effectively
A.4 3D Generation and Gaussian Splatting
3D geometric data employs diverse representations that are crucial in 3D generation. These repre- sentations be categorized into three types: O explicit representations, such as point clouds [76 and meshes (77). @implicit representations, such as Neural Radiance Fields (NeRFs) [1]. @hybrid representations, such as 3D Gaussian [42]. Currently, 3D Gaussian splatting widely used in 3D generation due to the geometric editability, high-frequency detail capture capability and real-time ren- dering efficiency. Each 3D Gaussian point can be represented by the following parameters: position r, spherical harmonics coefficients c, opacity 3D Gaussian points can be projected onto the image plane via the viewing transformation W and the Jacobi affine approximation matrix of the projection transformation matrix in geometry. In terms of appearance, we can calculate the color of every pixel the 2D image by blending N ordered points overlapping the pixel using spherical harmonics coefficients e and opacities a.
rotation matrix and a scaling matrix s. Then,
With the development of 3D representations, 3D) adversarial examples for different 3D structures have also been advanced 778 geometric or texture data of existing objects, for the first time, we realize the reference-free generation of semantically constrained 3D adversarial examples by utilizing language-guidee 3D generation models. We implement language-guidee DDIM sampler referencing the optimization pipeline of Trellis in latent space and decode into 3D Gaussian formats. Then, we achieve 3D adversarial attacks on 2D target models using 3D Gaussian rendering.
79 80]. Unlike other adversarial methods, which are based on the
3D adversarial example generation through the ResAdv-
B Detailed Implementation of InSUR Framework
Adversarial attacks often require reengineering existing tools achieve new functions, and their implementation not simple, especially when achieving new characteristics. Therefore, we provide the design principles and the key techniques in the main paper, and supplement the detailed implemen- tation in this section as a complement. the following subsections, we describe the implementation of the SemanticAE generator based on the different scenarios, and then provide more details about the cemantic-abstractie Attacking Evaluation Enhancement.
B.1 2D Image Generation
ImageNet Object Generation with Guidance Masking For the classical image generation prob- lem, we start with the original DDIM sampling process:
It-AT Va,-AT/ aa (xt vI (zeo(ze,t)) + vI ar-AT--
(12)
26

---

where â = IT:=1 - 3.) ) is the sampling parameter, and AT' is the step interval. As described in section 3.3, masked guidance is adapted as the conditional diffusion guidance, which formulates the denoise function 2T-AT = Jo,Ar(1 (1:) as:
(1,TTex)) JO,A (1ta) - AT (ar - v1 0 (34,, Text)) / or aT-AT co 4, Text),
ca( t, Text): = M) 68, Uncouditional t) + M Eg, Conditiionl t, Text).
(13)
The guidance mask is defined as matrix with different values in the border elements:
Mmid Modge
h
Sii<
15h 16
2 16
S
15
My
16
(14)
otherwise.
In the main paper, use the simplified notation of f without parameters Text and explicitly written. Here we use the detailed notations. For the experiments, we adapted Muid = 3.0 and Medge 0.3 in the main experiment and selected Medge {0.0, 0.3, 3.0} in the ablation study.
Residual Approximation Recall that ResAdv-DDIM defines a residual estimation function g for the adversarial feedback of the target model. We construct the step size of g as evenly distributed:
k
(1. Text) 99
(xt)
(ATi,,
(AAT2+ ATI Text) .... S8,AT2
Text)) J
(It), where Σ
ATi =t
JAATI
& times
(15)
t [T/K] 1, AT(() = [ mod [T/K], i=1
i<isk
k
where K is the maximal iteration number, corresponding to Itermax in the ablation study in the main paper. For brevity, define T as the number of sampling steps of the original DDIM generator and use the sampling interval the original DDIM generator as the unit of AT. We let f, (default set as 0.75) be the start step of the adversarial optimization, and the sampling process with adversarial optimization is formulated as:
(exTex J0,T
(1,Toxt) (x:)))
arg maxx, LATK (M o go
S bas t> t
T4- AT
TexT (zz),
(16)
Collaborating with Guidance Masking To generate attack related image backgrounds without disturbing the foreground semantics, as the goal of Context-encoded Attacking Scenario Constrain, we let the exemplar generation x/ - a/-AT communicate with the adversarial generation it - It AT- Specifically, at the beginning of the adv. optimization in step lx, we set the benign sample generated from x/_ + Fy, as the initialization of the constraint anchor. At the end of the adversarial optimization,
1 the optimized background written back o [M = MedgelSelect + 2 (ro + x%) [M = Medge] Melee] Select after the generation.
A daptive Optimization Iteration To improve the efficiency in multi-step diffusion- adversar- ial optimization, implement the adaptive iteration mechanism for ResAdy-DDin by early-stopping the optimization problem in Eq 16 which is formulated as:
M, 1= L-LL V 4,
=
m, otherwise.
PerformOptimize
(isnA A arg max P(M(I.) = E1) V A
(17)
If confidences'En early stop. Maximal optimize iterations
(i (i=1) A arg max P(M(a,) = E2), Ig
step. where the current iteration number, and n stands for the maximal adversarial optimization iteration in the single diffusion step. We set 0.1, &2 = 0.01, and set 3 and M = 10 for diversified
confidence Ep. optimize
27

---

Table Analysis of the different setting of
0.15
0.15
0.15
0.1
0.1
0.1
0.05
0,05
0.05
0.02
0.01
0.005
0.02
0.01
0.005
0.02
0.01
0.005
ResNet50 ACCJ
0.0224
0.0192
0.0214
0.0246
0.0246
0.0246
0.0224
0.0256
0.0246
VIT-B/16 ACC|
0.2949
0.2885
0.2885
0.2853
0.2724
0.2714
0.2404
0.2382
0.2511
strategy in different sampling steps, i.e., the initial and final steps are with higher maximal iteration number (n = M.
In T we present parameter analysis of & and &2 using ResNet50 the surrogate model on the Abstract Label Evasion Task. In white-box attacks, all tested threshold values achieve strong performance (accuracy after attack < 3%). In black-box attacks, smaller & values yield higher transferability, because lower Ç1 leads to more aggressive adversarial optimization in early denoising opeps-ootimizationn that tends be more transferable (this is consistent with the design principle of ResAdv-DDIM, which aims to enhance early-step optimization). However, this also increases computational cost. A practical deployment could balance effectiveness and efficiency.
Attack Loss Construction For the classification task, given the incorrect label set AText we implement the loss as:
LATK Log
(18)
Where Logits the model's prediction, and Lar is the estimated highest confidence label in the incorrect label set ATex-- We maximize LATK to perform the attack. For the current denoising step It, the logits are estimated based on the residual approximation function and the surrogate model, i.., Logits = M(g (t, (e.Text) ). To further eliminate guidance fluctuation, we adapt the fixed Lua after initialization in the initial steps of attack optimization (target label update delay). Note that our adversarial attack method is not designed for the classification task only, and the loss construction could be substituted regarding the specific adversarial example generation task.
The overall SemanticAE generation algorithm is constructed by further integrating global perturbation constraints, momentum gradient optimization, target label update delay (after < 44 0.4T) through the bi-level optimization. The pseudo-code is shown in Algorithm] [ The default configuration for other parameters is coherent with related baselines, i.e., 0.5, 0.7, T 100.
Algorithm 1: ResAdv-DDIM
Require: Input: Text, M, AText f. optimization parameters T, K, t,, th, 3, Init .T ~ (N, I), Ur < 0, 57 +- Fr.
for t = T.... 1 do
< t, then
for i 1....... do
Perform Ontrnnie (Eq. 17 then break
/* Adaptive Iteration */
if / = 1, V < 4 then
bTw arg max L
M(g (txText) (x. logica [.]
/* Target Update */
Up < B02- (1 B-V CATK.
/* Momentum Updates */
E1 < T1 + s U2-
/* Adversarial Optimization */
= 1, then
L x t- I4-1
Determine Exemplar Step
+/
Tex,))xt) (zx)
Et- + f0.1
/* DDIM Sampling Step for Tady */
Tex, (x²)
/ + f01
/* DDIM Sampling Step for Texemplar */
It-1 f It- + min(e, ||o6_1 21-1112) | 26 /* Semantic Constraint */
return Fg, 2/6-
28

---

By counting, the step number of the forward or backward process of diffusion Luffusiin-UNNet UNet is less than (2mt, + 8M)K tt, T. The maximal memory cost of the backward process |T/K] X the parameters in the feature maps the Liffusion-UNet, combined with other modules, including the surrogate model, the imput transformation, and the VAE-decoder in the optimization pipeline. In practice, the time consumption is significantly lower than the upper bound due to the adaptive iteration mechanism, and the lower bound is characterized by:
Proposition B.1 (Lower Bound of the Diffusion Step). For ResAdy-DDDiii with the total sampling step T, the parameter of approximate iterations in g as K, the timestep of start adversarial optimization as l. the lower bound of the diffusion step is:
K 1.
44 + T,
+
(19)
T
if K and la are set as K|T and I
K 16..
Proof. From K T and F lt. T = K d where d E d optimal implementation, the forward process in the early-stop judgment and the optimization step are reused. Consider the case of always early-stopping, the total number of approximate iterations in is Et [T/K]]
where E Under the
we have:
dm
kd + d
(d k) dm((m + 1)
Σ
Σ
(20)
T/K]
t=1
k=
t=1
By combining the total denoising step of Zadv and Texemplar generation, the total step is:
dm((m +1 +
Ls
K ls
K
+
+1) + +T
+ +3)
+
(21)
2
2
T
T 2
This completes the proof.
B.2 3D Object Generation
Base 3D Generation Method (Trellis) Representing 3D data as matrices is inefficient and impractical due to computational problems. Our method is developed based on the Trellis model, which bridges the gap between 3D structure and the diffusion process with the structured latent (SLAT). The overall generation process could be formulated as:
Diffusion Sampling(estats =t zalat Text), =7 zglat NO0 pos Coords(Dstatt (z (=aaat)), Coords Rbxbxwxd + 2xxx2
alat
20 Diffusion Sampling(e, æT, pos, Text), 2T (0,,1)
(22)
Modelas Dzs pos), Des {(2², "es,poo")) - {l(xtt oi o a De
x = Renderergs (MoccesssMMo (Modeles, Camera).
where 2n and zolat are latents sampled by the diffusion model, and are represented by sparse and dense tensors, respectively. Dztat is the coarse structure decoder, Coords transforms the voxel to point positions pos, DGs is the refined structure decoder that decodes each vertex into multiple Gaussian points, and Renderergs renders the Gaussian model 2D images x with the camera parameter. Since the refined position is also encoded in zo. implement the proposed ResAdv-DDIM with the noise estimation network that models the refined structure.
Renderercs is the Gaussian renderer proposed in Gaussian splatting. Due to its advantage in optimization, we select this representation as the intermediate 3D data structure for gradient estimation. Specifically, the 3D Gaussian with center point can be expressed as the Gaussian function:
G(p)
T 1p
(23)
To get 2D projection image from 3D Gaussian point in world coordinate with viewing transformation W, the 2D covariance matrix Σ as:
= JWEW J.
(24)
29

---

where J/ is the Jacobi affine approximation matrix of the transformation matrix. To render the entire Gaussian model, the color of every pixel in the 2D image is computed by blending N ordered points overlapping the pixel using the following equation:
C= Σ Ga JE
(g) ==1
(25)
where o, is the color of each point evaluated by spherical harmonics (SH) color coefficients and (a:) determined by 2D Gaussian with covariance and optimizable per-point opacity.
Residual Approximation with EoT We adapt the appectation-vver-taaaffommmt to bridge the 2D and 3D adversarial generation. By integrating the renderer, the residual approximation and the adversarial optimization represented
96
(t. Text)
Ee pos, Camera) =
Renderergs (DGs(Je,A7) AAT,,
((t,Text) Jo.AT (zt, pos), pos), Camera)
(26)
Text) 21 AT = JO
(ATK( (t,Text) (ze, Camera, pos)))
arg max ECameran Zt
The camera settings are sampled based on the original configuration in Trellis framework that defines Poam i.e::
A yaw
u
A pitch U1
4'
4
[sin( (fyy
A yaw cos cos(Ogitch Ppich
A pisch)
eye = 2
cosl 0yaw
cos ppich + A Apich)
yaw
(27)
sin pitch Apich
R LookA((From-eye, To= UpAxis=Z),
R 0
Extrinsics
Camera Extrinsics, Intrinsicsoo==0rl)
where the eye position eye is sampled on the sphere with r = 2, and the camera is toward the coordinate origin. Based on the practice of expectation-over transformation(Bo)) irrerinnoooe-traaffmmaaatiooooo optimization performed by averaging the gradient from different sampled cameras:
(29], the inner-loop
Cam[1,2, E] f- Sample(Pc.nm
grad=
(ATK(M g
(t. (t.T)x (2t, pos, Camera))
(28)
U, B8, + (1 3) grad
2e - 2 + Uga
where is the step of EoT with the default value For the optimization- configurations. we maintained most of the parameters in 2D generation:
10. K 4. MM = 30/ E, & = & 0.01, s 0.5 and t, 0.75T. The target label delay update mechanism is not applied. Diffusion-related configurations are coherent with the original pipeline, i.e., T 50.
Overall time Consumption We conducted timing analysis single GPU using 100 labels from the ImageNet-Label dataset. The results are shown in Tabel 7 Gradient computation and residual approximation account for the majority of the computational cost. Meanwhile, EoT sampling is implemented via multi-view rendering using the Gaussian splatting renderer, while the rendering parameter computation does not occupy CUDA execution time. The variation generation time arises from the adaptive number of optimization steps and differences in the size of sparse tensors in the Trellis framework. Specifically, in single iteration, fluetattoo computation speed primarily caused by inconsistent data sizes in the Sparse Tensor. For the entire optimization, the fluctuation in
30

---

Table GPU Time Consumption for Adversarial Optimization
Process
Single Iteration (ms)
Entire Optimization (ms)
Denoise Sampling
123.321 + 36.878
2994.933 + 269.84, (69.841
Residual Approximation
147.256 + 76.243
7040.238 + 8915.112
Gaussian Decoding
21.547 = 11.868
1029.394 1185.636
Gaussian Rendering
28.556 1.76:77..000 7.706
1364.221 = 1732.996
Surrogate Model
6.777 + 0.745
323.782 423.503
Backward Process
208.471 + 78.039
6261.577 + 12119.871
entity, physical entity, object, ungulate, whole, animal, organism, vertebrate, vascular vascular_plam __lat, instrumentality. mammal, placental, carnivore, vehicle, herb, self-propelled_ proheelled_eehhcleee phibian, canine, comnettccaanimne veyance, commodity, monkey, abstraction, consumer_gooe goods, structure, invertebrate, artifact, ruminant, invertebrate, matter, wheeled_vchicle, arthropod, causal_agent, reptile, equip- ment, implement, even-toed ween-toed_ungulatee, garment, game, diapsid, primate, formaatin, at- relation, restraint, ape, natural_object, psychological feature, geological tribute, starches, obstruction, gqquatic_vertbbee
vehicle, am- peccctronic_qqiipeeet device, container, covering, con-
old_world, workd_monkey, process, barrier, new new__oweee world monkey, substance, communication, establishment, feline, tool, clothing, food, solid, piece_ of cloth, brass, screen, shelter, grouse, machine, vessel, craft, arachnid, fabric, durables, thing. place_ of business, reproductive sttuctuuteuteeerrt plant, event, material, fastener, woody_ _llntt,lnnttttt measure, game_ equipment, shape, rodent, cellitary-vehiche area, mechanical device, substance, nu- pechanncal_deevce,,
mechanism, seafood, cognition, part, organ, group,
triment, amphibian, salamander, support, produce, natural natural_elevation elevation, mollusk, crustacean, act, public_tre traniis_ttnsoort,,, hand stol_ medium, box,
aquatic_man mammal, signal, indefinite_quani state, kitchen srcchen_applincee, edible ffuit, toiletry, shellfish, ware, utensil, fur, foodstuff, cloak, big_cat, footwear, ball, instrument, person, worker, insect, computer, lepidopterous_inee insect, vine
sports_ pports_equipmet stick,
Figure 14: Overly polysemous tags filtered out
computation speed is also attributed the adaptive optimization steps. Specifically, the average count of optimization steps for the inner optimization is: 47.810 (666)).. Attributed to the proposed early stopping mechanism, easier attack cases require fewer iterations and thus less time. The large variance in generation time reflects significant differences in attack difficulty across tasks, which highlights the effectiveness of our adaptive step design.
B.3 Construction and Analysis of Semantic-Asstracted Evaluation Task
Abstracted Label Set Construction As described in the main paper, we construct the coarse label based on three steps: (1) hyponymic graph based on the hyponymic relation defined by WordNet, (2) select the proper abstraction level, (3) define the attack goal as the evasion attack on the abstracted label. Specifically, the hyponymic graph defined
directed graph G EV, E V V. where each vertex V represents a word, each edge e U1 + denotes that the word 02 is hypernym of V1. We denote the ImageNet label set as the vertex set C V. and construct the subgraph G' = (V', F²)
V' = TCo(L) = - VEEV Bu L, L, (,
(29)
G -V' E')) where E' ='= u.E
Where TC denotes the transitive closure and G' is the induced subgraph. Next, select the abstraction level. We first annotated and filtered out the following over-coarse labels from the vertex set V", resulting in the label set L'. We also filter polysemous tags listed in Figure 14
Then, for each linear path, we select the node with the lowest height (or has more than one direct hyponym) candidate label, constraining the upper bound of the abstracting level, and climinate
31

---

descendant labels of the candidate labels to constrain the lower bound of the abstracting level. These labels are represented as the Absbr. We construct the SemanticAE generation task by which is formulated as:
evading the A
find zadv E S(Text) s.t. .M(zadv) AText,
Text = 'Realistic image of AbstrractedLabel, specifically, [labe]]",
(30)
ATex {labeladv AbstractedLabel (ncestors(labelaa)))
AbstractedLabel E { 3I s.t. c Ancestors()) A Countchhllren (c) > 0),
For simplicity, we use the term e E A Ancestors()) to denote there exists a path from to c, and the term CountChiildeen
denote the in-degree of 0. We select the abstracted label with more than 3 hyponym Image labels as the label set in the experiment. The abstracted labels and the corresponding hyponym imagenet labels are shown Table
Discussions on the AeRrelative metric. We design AS Rrelativee to match the goal of facilitating the red-teaming framework, as described in Section 3.1 of the main paper The following proposition describes the relation between the AS Reelattive evaluation metrics of SemanticAE generator and the application scenario the multi-round reject-sampling based data generation pipeline.
Proposition B.2 (AS Rrelavvvee characterize the upper-bound probability of the successful attack). For any adversarial sampling algorithm K that generutes the SemanticAE with ASRralative = p on the evaluated black-box model Mr and the surrogate model MMs, there exists attack algorithm that achieves the successful attack with at least probability p (1 E) on Mr, the average generation log(e)
times less than L/ps, and the maximal generation times following assumption holds true:
for any 0 e 1. if the
log p.)
1. Non-Positive Attack:
For the sample generated from the instruction Text and the generation algorithm does not perform adversarial optimization towards misleading M towards the labels corresponding to Text, the correct classification leads to semantic alignment, i.e., M(x) E TText - (x + 8) E S (Text)), where LTexa is the correct label corresponding to Text and LText AText, and â is small perturbation with 5 > (Texemplar, Tadv)-
2. The sampling algorithm can generate sample satisfying M(x) E -Texxt at least proba- bility of ps by only uccessing the instruction Text.
3. Adjacency Assumption:
(FMrv) Ltext I MM((eeemplaa)) E L > Ms Mg(Fexemplar) E LText), where Ms is the surrogate model,
P( ((My(Fadv) E L/Text
Mr is the target model.
4. Consistency Assumption: ASRrelative = P(Mr(z) Atea [M((x) E LTex) for the instruction Text given in the attack scenario.
Proof. We construct the Las stgas-style sampling algorithm with the surrogate model Ms. The sampling algorithm is re-run if MM (Texemppla)) LText until it reaches the upper-bound iteration
log(e)
log(1
For the case of Ms (Zexemplar) E Lexxs based on assumption (1), Fady STeext). Based on assumptions (3) and (4), we have
(FMrr E AToxt) = P(Mr(Fadv) E AText MM (ssSexemplar) E ITTaa)
= 1 P(Mr (Radv) E Lrext Ms MM((eexempla (Texemplar) E Lrext)
P(Mr (Radv) E Lrext My (Fexempll))) Trea)) (by Assumption (3))
P(MrrIadds E AText MM (#exeuplaa)) E Trea)
AS = (by Assumption (4))
(31)
Therefore, with probability p, Tady is successful attack. Since the final success attack rate is
E Treas) Ps.
Jog(I Pc) 1) >>pp (1 (1 p log)-po =p. (1-e)
log(e)
Panal p- (1 (1 P.))
(32)
32

---

The expected execution time of the sampling is:
logfe?
Tog Pal
00
E
(1 (1-Pa) pa)a
k -p-
(1 pa) )a-
Ps Ps (1- (1 -p,))² ps
This completes the proof.
We acknowledge that the evaluation might still not be adequate it requires the assumption of Non-Pot Non-Positiiv Attack. However, the defect the original non-reference evaluation is already shown in experiments (detailed in Appendix D.4.
C Detailed Experiment Settings
C.1
2D Experiment Settings
Our baseline method is constructed based on the categorization of the transfer attack method in Appendix A.3 and based the discussions in Appendix A.2 Since proposed module belongs the intersection of the diffusion-based adversarial attack and the optimization methodology. we select the classical MI-FGSM [5]. which is the base method of recent transfer attacks, and three diffusion- based adversarial optimization methods that are recently proposed and are suitable for SemanticAE including AdvDiff (16). VENOM [7 and SD-NAE [15]. SD-NAE applics the gradient back propagatonn optimization over the full diffusion steps, with the optimization formula as follows. It belongs to the first optimization paradigm discussed in Section
Baseline Descriptions
max TextEibedding
LATK (M( (
Embeddiing 0,AT
fo.At
Textddneddinng Embeddinn .... fo
(TT (IT))) )),, (33)
T/AT times
AdvDiff and VENOM alter the latent embedding 24 during the diffusion denoising process, and belong to the second optimization paradigm discussed in Section Compared to AdvDiff, VENOM applied the additional momentum mechanism and the carly-stopping mechanism to the optimization process. Also, tries resample zT based on the adversarial direction of xo when the optimization fails on the surrogate model. Our method, besides the technical improvement discussed in the main paper, adds a more one-grained scheduling of optimization steps based on the new optimization paradigm. And we do not add the resample mechanism since it is not redundant if our method is used as a sampling module an attack/data generation system that could perform reject sampling based on the application scenarios.
We integrate an inpuuttranssormaaion-oooo evaluate the collaboration capability of our method and other methods. We set the num_ warping=2 and our attacks, and num warping=10 for MI-FGSM. The latter setting is consistent with the overall attack pipeline evaluated in the DeCoWA paper.
method (DeCoWA 45 as the surrogate model to
for diffusion-baas
For the diffusion baselines, AdyDiff adapts the classifier guidance and takes the image class the input of the diffusion model, while VENOM and SD-NAE adapt the classifier-free guidance. SD-NAE alters the selected text embedding by solving the maximization problem described in the main paper, and therefore max LATK M(fe.AT o fa.At o ... o fo.se (xT) ))), Advdiff adapts the approximated and VENOM introduces conditional opti- mization and momentum mechanisms on it to stabilize the optimization. The diffusion model applied in VENOM and SD-NAE is bguisard/stable-difffff delee-ddfff---nnacc - and latent- diffiusion/cin256-v is applied for AdvDiff. We implemented our code based on the baselines VENOM and SD-NAE, and adapted the same diffusion model for consistent evaluation. In principle, our method can scale to larger models and is evaluated on the Trellis 3D generation model.
optimization y- T fe, (arg (arg max,; (ATK(M(2())
Surrogate and Target Models described in the main paper, we adapt the target model set as T (ResNe550 49), VIT-B/16 [50]. ComvNeXi-T [5T), R5sNet/52, Inception (52). Swin- -rin-Trannformer-- B [53] and the surrogate model set as |ResNet50, VIT-B/16, ConvNext- T, ResNee50+DeCoWA),
33

---

Loss Function We employ the same loss function as the original implementation for the baseline methods. For our model, we set the loss function in both 2D and 3D SemanticAE generation task as:
1
Cauk(logits, A Iear) LogSottMaallogit))
Σ
LogSoftMax(logtts)]3 [t],
[A
iEAre
(34)
Where Laar is the currently selected label (the label with the highest confidence in the set Arezt). and log softmax denotes log Σ gx
Image Quality Assessment Metrics To evaluate semantic constraints, the pairwise semantic metric proposed to measure the similarity between Texemplar and Zadv. which defines as follows in the main body of our work:
SemanticDiffs = (Texemplars Tadv)s:
(35)
S is visual similarity metric; employed LPIPS and MS-SSIM for evaluation. Parameters of the evaluation metrics are adapted as common practice. For MS-SSIM, we adapt kernelsize 11 and σ 1.5. For LPIPS, we adapt AlexNet as the local feature extractor. For Clipq. we the implementation of piq [81] and adapt openai/cli-vvit-buse patii-hhs--- as the image embedding extractor.
C.2 3D Experiment Settings
This section details the comprehensive framework established for the evaluation of 3D video genera- tion models, encompassing dataset preparation, video synthesis methodologies for both benign and adversarial examples, and the metries employed for performance assessment.
Dataset Preparation
The ImageNet dataset, while extensive, contains labels with fine-graines semantic distinctions that can challenging for text-to-3D video generation models to differentiate effectively. A coarse- graniiig procedure was applied to the Original Imagenet labels to address this.
Specifically, a predefined mapping, detailed in Table 9 was utilized to merge semantically similar labels. This process involved replacing the Original ImageNet labels with their corresponding Abstracted Labels. The resultant collection of these processed Abstracted Labels served as the prompt dataset for the subsequent video generation tasks. Let L be the set of Original ImageNet labels and Teoarse be the set of Abstracted Labels. The mapping function M - Tooarse transforms each Original ImageNet label to its Abstracted Label. The set of prompts used for generation is P = {1]6 E T.
3D Video Generation Two categories of video samples were generated: clean samples and ad- versarial examples. Clean video samples were synthesized using the TRELLIS (33 model. For each prompt p E P, corresponding clean video Vatean was generated. Adversarial examples were generated based on the TRELLIS[43) model (version TRELLIS-rext-ee ImageNet, the surrogate model for guiding the adversarial attack. The
employing a ResNet-50
model, pre-trained generation of these adversarial examples was performed using our proposed methodology. During each generation instance, an abstracted text label p E was used as the input prompt. The target label for the attack was set to any Original ImageNet label laaget E IL such that M (ltarget) p. A constant perturbation strength, denoted as c, was maintained at 10.0 across all adversarial generation processes. All videos are captured by the original rendering pipeline, i.e., camera surrounding the object.
Evaluation Metrics The evaluation of the generated videos involved frame-by brree-bb--rraeee analysis using a pre-trained ResNet50 classifier.
For a given video V, consisting of N frames (f1, f2, fN), each frame fa was individually classified by the ResNet-50 model. This yields a sequence of Original Imagenet labels, e ResNe(50(ff). Each e; was then mapped to its Abstracted Label (c.). The overall model prediction for the video V, denoted as Pv. was determined by the mode of these frame-level Abstracted Label predictions:
py mode (f1,
where mode(-) returns the most frequently occurring element in the set.
34

---

A video V was deemed correctly classified if its model prediction Pv matched its ground truth Abstracted Label Gv. It was observed that, under certain parameter configurations (particularly for varying K) minority of video generation attempts might fail. To ensure rigorous and controlled comparison, a data curation step was implemented. The intersection of successfully generated videos across all conditions clean samples (Veiean) and all sets of adversarial examples (Vil)) v-I)) v2(2) v-(3) Vodee- was taken. Only videos present in this intersection were considered for the final evaluation. This ensures that performance metrics are calculated over an identical set of video instances, thus isolating the impact of the varied adversarial generation parameters.
add+
Following the procedures outlined above, the Accuracy (ACC) and Attack Success Rate (ASR) were calculated. The specific mathematical formulations ASR provided in the main body of this work.
D Detailed Results and Discussions
D.1 Transfer Attack Analysis
Experimental settings We select VENOM, MI-FGSM, SD-NAE, and AdvDiff as baseline meth- ods, employing four surrogate models: ResNet50, DeCoWa, ConvNext-T, and ViT-B/16. Six target models are evaluated: ResNer50, ResNet152, ConvNext-T, ViT-B/16, Swin-B, and Inception For Abstracted Label tasks, our method adopts four different perturbations (2, 2.5, 4]. while Origi- nal Imagenet label tasks use e = {1.5, 2, 2.5, 3). For each surrogate-targe model pair, adversarial examples are crafted using both our method and baselines. Evaluation metries include Attack Success Rate (ASR), Accuracy (ACC) and LPIPS (lower values indicate better perceptual quality).
Data presentation Due to the inconsistency between different surrogate-target model pairs, we chose to present the experimental results using subplots. The x-axis represents the selected surrogate models, and the y-axis represents the attacked target models, with a total of 24 subplots. The experimental results shown in Figure 20 21 22 23 Figure 20 [2) display the relationship between LPIPS and ASR/ACC for examples generated by different methods in the abstracted label task. show the relationship between LPIPS and ASR/ACC for examples generated by different methods in the original Imagenet label task. In the figures, data points of different colors represent different methods. The data points of our method under varying perturbation strengths connected by lines, and its trend is fitted with a black dashed line. As a supplement, the numerical results with the standard deviation of the final metric over random seeds are shown in Table 11 and demonstrating the stability of the results.
Figure 21 2
Table II
Analysis Based on the observations from Figure 20 the following conclusions can be derived:
In our method, as the perturbation parameter gradually increases, ASR also rises, but the LPIPS increases as well. The relationship between ASR and the Natural logarithm of LPIPS essentially forms straight line with positive slope.
The adversarial examples produced by SD-NAE (yellow markers) exhibit higher LPIPS in almost all cases, indicating that this method introduces more noticeable perturbations. However, compared to other baseline methods, SD-NAE does not always achieve higher attack success rate.
The adversarial examples produced by MI-FGSM (purple markers)have lower LPIPS than SD-NAE, but still significantly higher than our proposed attack method.
The LPIPS of adversarial examples generated by VENOM (green markers) is comparable to our method. However, at similar LPIPS levels, our method achieves a higher ASR in most cases.
AdvDiff (blue markers) produces adversarial examples with the lowest LPIPS, indicating better stealthiness. However, its attack success rate is significantly lower than all other methods.
To summarize, in all 48 settings of this study, except for two cases (the first row, first column and first row, fourth column Figure 20) where our method was slightly worse than the VENOM method, the remaining ASR-LPIPS curves of our method were all located above and to the
Conclusion
35

---

left of the comparison methods, In Figure 22 the two subfigures of Figure the first row and first column, and the first row and fourth column), are on the ACC-LPIPS curves of our method. This means our method achieved higher attack success rates at the same LPIPS levels. These results clearly demonstrate that our method outperforms the comparison methods in most cases (46/48). In addition, these exceptions both occurred in white-box attack scenarios, which were not our primary focus. VENOM achieved higher white-box accuracy by repeatedly resampling through rejection, while we treated this module separate from the Adversarial Sampling algorithm.
23 all the methods, except for the VENOM method in
Therefore, our InSUR framework shows significant superiority in adversarial transferability.
D.2 Attack Robustness Analysis
Experimental settings This experiment discusses the performance of our method compared to baseline methods when facing adversarial defenses. The defense methods used are JPEG and DiffPure. JPEG applies lossy compression to images (with a quality factor of 75 in this experiment), while DiffPure removes adversarial perturbations by feeding samples into a diffusion model for regeneration.
Data presentation Figure 24 25 show the results of method and comparison methods both defended and undefended models. Solid dots represent undefended results, while hollow dots represent defended results, with arrows indicating the impact of applying defenses.
Analysis
MI-FGSM: JPEG a rule-based defense method, whereas DiffPure is a defense based on the in-manifold assumption. Due to the lack of in-manifold regularization, MI-FGSM- generated adversarial examples are less robust against DiftPure, showing a noticeable performance drop. This be seen in Figure 24
Figure 25
AdvDiff: As diffusion-based adversarial example generation method,it exhibit strong robust- ness against DiftPure (also diffusion harpiionn-baddd.iii..g. Figure 25 shows that their attack success rates (ASR) even increase after DiffPure defense, though they still underperform
method.
SD-NAE: SD-NAE is also a diffusion-based adversarial example generation method, so it
exhibits strong robustness against DiffPure. Besides, SD-NAE applies perturbations through text embeddings, which results in better performance in transfer attacks. However, this does not necessarily indicate an advantage, as the visualization results suggest that it may lead to uncontrollable semantic deviations. The high transfer attack success rate could be attributed to inherent changes in global semantics. Additionally, its optimization for white-box attacks is inadequate. For undefended white-box models, the success rate is lower than that of our method and VENOM.
Conclusion Our method shows a decrease in ASR in most cases when facing JPEG and DifiPure defenses, but this effect is limited, and in the face of JPEG defense, our method is the only one able to keep the attack success rate ASR consistently above 80% when the target model is the same as the surrogate model (see Figure 24). When the DiffPure defense leads to decrease in the attack success rate of our method and increase in the attack success rate the SD-NAE method, we are still able to ensure that at least one data point of our method outperforms the SD-NAE (e.g., the subplot in the first column of the fifth row of Figure 25
In summary, our method enhances the optimization exploration capability for adversarial examples while maintaining In Manifold Regularization (outeeroooos the comparison methods all cases (48/48)). This ensures that our approach remains effectively aggressive against defense methods.
D.3
On the Role of Residual-driven Attacking Direction Stabilization in 2D and 3D SemanticAE Applications
Boosting Multi Multi-diffusion-step Regularized Adversarial Optimization Recall that, to solve the problem of collaborating the adversarial optimization with the diffusion model for better transfer attacks and robust attacks, we introduce the residual approximation in
36

---

We further analyze it in the more refined evaluation tasks. The experiment is conducted with the surrogate model of ViT-B/16 with 6 different target models on the abstracted label evasion task. The parameters of the maximal approximate iteration K and the l./TT = 0.25, 0.5, 0.75 have been evaluated. The results are shown Figure [5] Since altering the parameter changes the behavior of the semantic alignment, the semantic difference measurement is included, and the results are plotted in figure. K indicates the setting without the residual approximation.
The figure shows that:
When f,/T 0.25, both LPIPS and ASR are higher, and the introduction of the residual approximation lets the balance point between LPIPS and ASR offset, more biased towards LPIPS optimization. This is due to (1) since the white box results, shown in the upper left figure, indicate that the successful ASR has already been achieved, and there is need to improve the performance of adversarial optimization. Therefore, improvement is on LPIPS. (2) Since the residual approximation is not designed for regularizing transfer attacks, the transfer attack performance declines in ResNet and Inception models.
When t {0.5, 0.75). the result is different. Specifically. (1) the white-box results indicate that the adversarial optimization is non-optimal. (2) The residual approximation im- proves the performance of the white-box attacks and significantly improves the performance of transfer attacks. Improvements exist in both LPIPS and ASR.
Simply increasing the step (the blue arrows) of undergoing diffusion steps of adversarial optimization may not improve adversarial transferability. However, may decrease adver- sarial optimization performance under the same adaptive optimization mechanism, as the diffusion process may purify the adversarial optimization.
Increasing t, together with the residual approximation solves the collaboration problem between adversarial optimization and diffusion purification, leading to highly transferable SemanticAE.
-Drpec
Target Resters0
Dojet Iocer
1.9
1
0.12
S-M4
355
1.5
389
037
24:07
4+177
4+37
Terast: Com//NecT
Terget: Satts
Terge::
0.00
0.139
200 6010
0.53
1254
6449
(PI5 Tower Sheteen
LSPS Hivver seteet
LRPS Sower betee
Figure 15: Parameter analysis with surrogate model ViT-B/16. Red arrows denote the performance before and after adding the residual approximation for g(a)). In each setting of 4 2 0.5, applying residual approximation achieves significant improvements (The upper left corner indicates a strictly superior direction). This result is coherent with the design goal of ResAdv-DDIM.
Collaboration with EoT Different from 2D optimization, the global gradient of 3D models cannot be obtained within a single iteration. Therefore, the EoT optimization is applied, accumulating the gradients across different perspectives. This further challenges the adversarial optimization capability. However, as shown in Table R] with the residual approximation (K > 0), our method can collaborate well with EoT' with different numbers of EoT steps, resulting in different gradient
37

---

optimization step sizes. The visualized comparison is shown in Figure 16 and the supplementary videos in httis///emanticA.giiuun higher than without residual approximation (K = 0), showing that the necessity of the residual approximation under diffusion+EoT generation pipeline.
The attack perfonmance on different views is significantly
Table 8: Comparison between residual approximation and expectation over transformation (EoT). The total iteration (EoT step gradient descent step) is consistent. The gradient optimization stepsize is larger if the EoT step higher.
EoT step Residual approximation step (K)
3
1
1
1
4
4
4
3
2
0
ASR
0.922
0.913
0.922
0.912
0.912
0.902
0.451
Clean sample
Adr sample w/o ResAdr-DDIM
Adv sample with RDDMM
Accuracy. 237/300 Frames
Accuracy: 276/300 Frames
Accuracy: 23/300 Frames
True: forklift
Trne: focklift
True: forklift
Predict forklift
Predict forklift
Predict: motor matttooyeolleeeeeeeeeoe yelicle
Figure 16: 3D Visual Results Ablation.
D.4 On the Potential Adversarial Transferability to Semantic Evaluator
We analyze the character between and find the clue of the reference-free semantic evaluation metric being transferred in SemanticAE evaluation. Our experiment is based on hypothesis testing. Specifi- cally, we evaluate the results from our method under setting of Appendix D.I] which is rounds of generation for each of the original ImageNet label evasion task and the abstracted label evasion task, Additionally, we evaluate the clip-score
[2
metric with the settings:
Prompt ImageNetLabel
Task ImageNet
(36)
AbstractedLabel, ImageNetLabel
Task = Abstracted
The backbone of the clip-score is V/T-B/32. Then we apply the linear regression the clip-score, as the clip Semantic metric, and the LPIPS score on the factor of whether the surrogate selects VIT-B/16. The results shown in Figure [7 Figure [8] and Figure 19 rejected with high confidence (p < 0.02):
The following hypothesis is
Under the same generation settings, the CLIP semantic evaluation results are independent of the surrogate model selection.
Due to the high p-value associated with LPIPS, its correlation with surrogate model selection is relatively low. Although there are differences in model configurations and training tasks, both clip- score and the surrogate model adopt the VIT architecture. Therefore, we have reason to believe that the transfer attack has affected the evaluation of semantic similarity metrics.
We believe that the potential adversarial transferability to the deep-learning semantic evaluation model is difficult to tackle within the models, especially for the potential adversarial example that
38

---

could perform weak-to-strong strong attacks. As discussed in Appendix B.3 our exemplar-based evaluation task provides an alternative way show the semantic alignment, avoids the requirement of non- referencing semantic evaluation, and directly shows the adversarial capability of the generated data.
4014
4114
004
603
401
600
600
601
C33
600
Figure 17: Results on the Original ImageNet Label Evasion SemanticAEs
Figure 18: Results on the Abstracted Label Eva- sion SemanticAEs
0.75
Lincar Regression (Rtw0.25 pw0.011)
Regression (Rt00000, p=0.53)
0.745
0.08
0.725
0.06
0.730
0.04
0.725
0.715
2.00
ISurragate
VIT3
(Surogete
V.T]
Figure 19: Combined distribution of the semantic metric the ImageNet and abstracted label evasion SemanticAEs. With high confidence (p < 0.02), the CL/P semantic metric is affected by the attack transferability.
E
Results Visualization
E.1
2D Visualized Results
To intuitively visualize the samples generated under the semantic constraints of original ImageNet label and abstracted label, we select SemanticAEs aadv and corresponding nearby samples Texemplar of seven original ImageNet labels (monarch, castle, goldfinch, aircraft carrier, vase, jellyfis)) and six abstracted labels (aircraft, bag, beetle, bird, boat, fish) for visualization, as shown in Figure 26 and
27 The surrogate model is DeCoWa ResNet. For our attack method. We set different parameters {1.5, 2, 2.5, 3} and e = {2, 2.5. 3, 4}. respectively. for 2D ImageNet- label evasion attacks and 2D abstracted-late evasion attacks, which determines the strength of the semantic constraint. We also present the generation results of the baselines. For single image I. we report confidence and mark the classification label y with different colors employing ResNet50 and ViT-B/16 as target models. In 2D ImageNet-labed evasion attacks, Green is for the same classification label and ImageNet label corresponding to the semantic constraint; otherwise, red. In 2D Abstracted- evasion attacks, Green indicates that the classification label y belongs to the abstracted label corresponding to the semantic constraint. Therefore, a successful attack defined as follows: for a pair of Tadv and Texemplar Texemplar is classified correctly (green), while Zadv is classified incorrectly (red).
Analyzing 2D Visualized Results conclusions can be derived:
Based on the observations of Figure 26 and 27 the following
39

---

For our attack method, as the parameter gradually increases, signifying a progressive weakening of the semantic constraint strength, the number of successful attacks correspondingly increases. However, this also results in unnaturalness of SemanticAFs. For instance, the Tadv with the original ImageNet label "castle" has a castle with a blurred top when e = 3, and the Tadv with the abstracted label "ship" exhibits a blurry mast when c = 4.
SemanticAEs generated by SD-NAE disturb more global semantics, which leads to semantic drift and dissimilarity between the Zexemplar and Zadv. It is consistent with the results shown in Appendix [ SD-NAE has a lower MS-SSIM score and a higher LPIPS score, with the same surrogate model, compared with other attack methods.
As for MI-FGSM, noticeable noise can be observed in the background area of the SemanticAEs, which are not natural. In the main paper, MI MI-FGSM corresponds lower CLIP-QAI score related to noisiness.
AdvDiff generates adversarial examples with artifacts the edges in ImageNet-label evasion attacks, such as the Tadv of label "castle" and "aircraft carrier", which is a manifestation of low image quality. In abstracted-label evasion attacks, SemanticAEs hardly attack target models successfully.
The adversarial examples of VENOM are natural. However, they struggle to effectively attack the ViT-B/16 model.
In conclusion, method generates local in in-manifold patterns to achieve strong attacks.
E.2 3D Visualized Results
To qualitatively substantiate the efficacy of our proposed methodology, conducted a visual comparison of the generated video samples. Initially, ten unique labels were randomly selected from the complete set of Abstracted Labels. Subsequently, for each of these selected labels, the corresponding clean video sample and the adversarial video sample generated with K were chosen. From each of these videos, five frames were extracted at equidistant intervals. The visual results of these extracted frames are presented Figure 28 A comparative analysis of each pair of clean and adversarial examples reveals that the adversarial counterparts maintain a high degree of visual similarity to the benign videos. Despite this perceptual resemblance, the adversarial examples demonstrate high probability of inducing misclassification by the ResNet-50 model, thereby underscoring the effectiveness of our approach in generating robust yet inconspicuous adversarial attacks.
Image Visualization
Furthermore, we performed a comparative analysis of adversarial examples generated under varying K parameter settings to demonstrate the rationale behind our parameter selection. Specifically, selected the same video instance from the clean samples, the adversarial examples generated with K K==0 0, and those generated with K 4. A single frame was extracted from each of these three video versions for visualization, as depicted in Figure 16. It is observable from the figure that when K = O, the classification outcome for the adversarial example paradoxically improves compared to the clean sample, signifying failure of the adversarial attack. Conversely, for K = 4. the adversarial example exhibits visual characteristics closely resembling those of the K sample. However, its efficacy in deceiving the classifier is significantly enhanced. This comparative visualization corroborates the superiority of our chosen parameter configuration (K- 4) in achieving strong attack effect while preserving visual quality.
Video Visualization We performed a comparative analysis of adversarial examples generated under varying K parameter settings to demonstrate the rationale behind our parameter selection. Specifically, we selected three video instances, identified by their primary content as the forklift video, the Ilama video, and the volcano video. For each of these videos, considered the clean sample, the adversarial example generated with K (without residual approximation). and that generated with K = 4. The videos are shown in https://semanticAE..C Aithhub..
It is observable from our quantitative analysis that the outcomes for K varied across the different videos. For the forklift video, the classification accuracy of the adversarial example generated with K T=0 paradoxically improved compared to its clean sample, signifying a failure the adversarial attack under this specific setting. In contrast, for both the Ilama and volcano videos, the adversarial examples generated with K 0 achieved lower classification accuracies than their respective clean samples, indicating some attack effect. However, their accuracies were still notably higher than those
40

---

of the adversarial examples generated with K = 4. This demonstrates that for the laaaa and volcano videos, while K O initiated an attack, its deceiving capability was considerably weaker than that of K =4
Conversely, for K 4, the adversarial examples for all three videos (forklift, Ilama, and volcano) exhibit visual characteristics closely resembling those of the K = samples. However, their efficacy deceiving the classifier is significantly enhanced across all instances. This comparative visualization and analysis corroborate the superiority of our chosen parameter configuration (K = 4) in achieving a strong attack effect while preserving visual quality.
F
Boarder Impacts
While our goal is catalyze the development of the red-teaming framework and trustworthy AI, we acknowledge that the proposed technology might be misused, including: (I) extending the proposed transfer attack improvement methods to jailbreak multi-modal LLMs. (2) extending the proposed 3D attack methods to generate physical adversarial examples to attack biometric authentication systems. However, our framework is not directly designed for these scenarios and requires further integration.
To protect from potential attacks in applications, suggest developing the following closed-loop framework complement to traditional defense methods tested in Appendix D:
Collect data generated by the proposed InSUR framework.
Annotate the data with human feedback or rule-based models.
Improve the alignment of the
models through fine-tuning the dataset.
As a tool for data generation, we believe our framework is more beneficial for the model holder. For responsibility, will release the code of InSUR framework after the paper is published for reference.
41

---

Table 9: Abstracted Label Mapping from ImageNet Numerical IDs
Abstracted Label
Original ImageNet Label IDs
dog
151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164,
165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178,
179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192.
193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206,
207, 208, 209, 210, 211, 213, 214, 215, 216, 217, 218, 220,
221, 222, 223, 224, 225, 226, 228, 229, 230, 231,232 233, 234,
235, 236, 237, 238, 239, 240,
243, 244, 245, 246, 247, 248,
249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268
bird
7, 7,8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 23, 24, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93,94, 95, 96, 97, 98, 99, 100,
127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140,
141, 142, 143, 144, 145, 146
musical instrument
401, 402, 420, 432, 486, 494, 513, 541, 546, 558, 566, 577, 579, 593,
594, 641, 642, 683, 684, 687, 699, 776, 822, 875, 881, 889
furnishing
423, 431, 453, 493, 495, 516, 520, 526, 532, 548, 553, 559, 564, 648,
703, 736, 741, 765, 794, 831, 846, 854, 857, 861, 894
motor_vehiele
407, 408, 436, 468, 511, 555, 569, 573, 575, 609, 627, 656, 661, 665,
717, 734, 751, 803, 817, 864, 867
snake
52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68
fish
0, 2, 5, 6, 389, 390, 391, 392, 393, 394, 395, 396, 397
building
410, 425, 449, 497, 498, 580, 598, 624, 663, 668, 698, 727, 762, 832
saurian
38, 39, 40, 41, 42, 43, 44, 46, 47, 48
ball
429, 430, 522, 574, 722, 747, 768, 805, 852, 890
headdress
433, 439, 452, 515, 518, 560, 667, 793, 808
beetle
300, 301, 302, 303, 304, 305, 306, 307
timepiece
409, 530, 531, 604, 704, 826, 835, 892
shop
415, 424, 454, 467, 509, 788, 860, 865
dish
925, 926, 933, 934, 962, 963, 964, 965
cat
281, 282, 283, 284, 285, 286, 287
weapon
413, 456, 471, 657, 744, 763, 764
bottle
440, 720, 737, 898, 899, 901, 907
fungus
991, 992, 993, 994, 995, 996, 997
spider
72, 73, 74, 75. 76, 77
ship
403, 510, 628, 724, 833, 913
boat
472, 554, 576, 625, 814, 914
turtle
33, 34, 35, 36, 37
bag
414, 636, 728, 748, 797
housing
500, 660, 663, 698, 915
crab
118, 119, 120, 121
wolf
269, 270, 271, 272
fox
277, 278, 279, 280
bear
294, 295, 296, 297
aircraft
404, 405, 417, 895
armor
465, 490, 524, 787
wheel
479, 694, 723, 739
fence
489, 716, 825, 912
personal_computer
527, 590, 620, 681
roof
538, 853, 858, 884
overgarment
568, 617, 735, 869
skirt
601, 655, 689, 775
bread
930, 931, 932, 962
squash
939, 940, 941, 942
42

---

1010
11200
1.20
1030
1914.
Fel
Unitowr 8011
UMMoow
F+ll
720
122
Figure 20: ASR of Abstracted Label
43

---

1248
7000
1910
1100
PITGOW BROC
UMMoow
1912
7230
1200
202
1248
Figure 21: ASR of Original Imagenet label
44

---

2.20
Unitowe Best
1.00
Figure 22: ACC of Abstracted Label
45

---

2250
150
2000
Figure 23: ACC of Original Imagenet label
46

---

8-
Figure 24: ASR JPEG Defense
47

---

Figure 25: ASR DiffPure Defense
48

---

doon
Our m-1.5
Nd.
Not aeaaac
0.00
220
4.79
coan
Our e42.
Net cara
0.00
dean
Ouea amd
ABV
+33
ooan
SONAZ
6.79
claan
M-TGOM
0,00
Net
5.0
0.10
0.00
doan
IP
Ad
ooan
VENOO
0.79
140
cack
Midi
Mart Age
IA4: PARA
Figure 26: Visualization of 2D ImageNet-label Evasion Attacks. ResNet50+DeCoWA.
Surrogate model is
49

---

Our
coam
Ours 0x25
Arc www.
ABV
Arruhtts
dlan
Ourar
K-7
ebaan
1.Brppelvvee 114. CR HEEVISS
AdN
deaon
NB
2.10
don
Figure 27: Visualization of 2D Abstracted-label Evasion Attacks. Surrogate model is ResNet50+DeCoWA.
50

---

Insrnction: supp depenser. Orginal Objeest RoiNet Accuracy: 214,300 Framee
Instrecion: strwberry Otiginal Object ResNet Accurscy: 2477300 Trames
Olbjoct ReaNet Classlicatim patlune: 298, saup dispatee::
Francs
Adv. Clyycct Classfication 109, jellyfish: 55, strawbery:
triumphal
Original Objecl
Acemney: 170:00
Object Ascuracy: 163/300
Ofjject Classiiaation
hidiosse
Objjst Chossfiatiio
laillopper
Instraction: padestal. Original ResNet Accunacy: 300/300 Frames
Instmetioo: hell papper Orriginal Objject Reshet Aecuracy: 200/300 U'mmes
Adv. Object ResNet Clusificttion hindhouse: 238, pedestal: 35,
Adv. Objoc, RecNel Choceiication
emaile
Band
26, bell pepper: S
Instrnetio: stariih Original Object ReaNer Accurcy: 3003300 Frnmos
Instrection: batclst Original Object ResNer
Aeemmey: 229/300 Frnmes
Adv. Object ResNet
Classification daisy: 181, bird: 97, starfish:
Adv. Object ResNer
Classification nail: 243, hammer: 22, hatchet: 0
Instretioon condle. Original Object ResNet Accenncy: 214/300 Fraenes
Instruction: piteher Original Object ResNet Accureyy: 285/500 Frmmes
Adv: Object ResNet Classification plunger: 297, candle: Frames
Adv: Object ResNet Classification hottle: 255, conch:
31, pitcher:
Figure 28: 3D Visual Results, Surrogate Model is ResNet50.
51

---

ACC
0.513.m001
0.568-Huu
0.4025100
0.440 moni
0.598
0.3582003
0,390a003
0:5122000
8:320:000
0.563 20ta
8.451.000 200z
8.27
0.29 mnn
f1418
0.684 100z
0 0,22 mns
0:47-mmm mum
0,7.3 c0.02
0.687 200m
0.622
0.579
0.725 0.725
0,708 6.70200m
0.065
0.649 20m0
0.962
0.908 1008tmm
1000
0.955 (.955.mm
0.513
0.568
0.402
01420
0:299
0:476
0.687
0.960
Swis-B
A3R
0.319
0.319-c.00
-0.00
0,199
0.199-c.ouz
-0.00
0.408
0,10
0,303
0.303
0.26650002
0472
0,462
0,479
0.287=50000
0.52
0,492
0,590
0L3700 00
0.5975000
0,580 -c.0
0,483 L000
0.178.c.. c000
0.248-c.mm
0.248
0,408
ccm
0,097
0.150=a.m
0.241 salas
0,291 10laa
0.291
0,194 Mosoo
0.201 sole
0.241 snla
0.262-.000
0.01 50500000
0,000
0.01 0.01=5000 sn:00
0,022
0.022.coo
ACC
0.201 0000
0.460 0.00
0. 0.355
0.355-5000
0.000
0.010
0.354 0.0
0-115 0-155-000
0. 0.323 0.523_0000
0.004
252 2010
1.271-000
1.271
0.004
11.34 sn
0.302-0000
2.2300000
0.000
0,274
0.625 ..00
0.396 c.m
0.355
0,872
5.00
-557-mmm
0.009
0.676
0.653 =000
0.004
D. 0..36
D 0.948
0.951 200m
1194--000 c000
0,907 0.907_000
ComyNest-
0
r0.00
0.306
0.429
0,508 14.00
0.381 0.381
2m00s
0.492 0.4922000
11500
1:500
0.42 1.421 200ta
I.57220
0,5420000 no
10
0,998 19,80
10.22
0.346 0.366. moat
0.52 10.205
0.993 10.000
0334
0.900
0.240
0.256mm00
0.552
0,22 0,22
0.133 0.13
2002)0000
0000
A3R
0.415
0.306
11
0,992
11707
1,641
0.521
0 11.22
ACC
0.439 nu42 m00
05m3-uum
0.06
3.38%
0.407
0-176.umm
0.003
0.358an00
0.300
0.4830001
81022
0.316gno)
0.3893001
0.000.nos
0,245 0.245 (05)
01344 (1.443,mm)
0.647 mo0
i002
0.577
-m777mmm
num
0.515
0.515
0,683 (no
0.4775mm4
0.005amm:
0.542
0.667
0.667.0m.
0.655um
0:377amm
01.66%
0.949
0.955
0.925.001
01.93% 0,933_mm2 2002
ViT-8/16
0
A3R
0.365
cocm
0,240
rcom
0,9675..00 s5.00
0,967
01,22
0.43 so.00
0.296 an
0.296,
0,994
on 01,644
0.500 socoo
0.341an000 a0
0.996 2: on
0.522
016913
gontanoo on
an 0o
1,000
0,629
01552
0.552.5.0
0.182 iocc
0.221 corm
0,327
0,108 c0.00
socm
0.991 0.991 scom
0.282 ann
0.282
0.227
0,227 35.00
0.236 o0
0555
0.323
0.020
0.010
0.061 an on
0.030
Table 10: Abstracted label transfer attack results.
ACC
0.076
0.00
0.077 c..
0,378
569
0,055
0.044 0.04-...m coom
0 0.33220000
(1 1.38att.o
0137 SRAM
200220000
(1 12222.000
0.279
0.299g5.0
(1120
01000-50100 000
0.365 2500
0,200
(1 (1994 5mm
0L533, 16:0
0.515 c.m
0 0,36]
01119
5.00
0.006-0.om
co.m
0.54
0.453 0.43-..0
0.573 0.573.5..0
0 0.350 200m
0.567 207700m
01825 01825_000
01 889
(1917
20
(193321000 un
(1926
ReNNa50
01
ASR
0.871
1083
0,966
0.399
suXs
0-467
0.90
mazz
0.92 p003
0:467 sooee
0:467
0:50
0103
0:407 80
D-49
0.576
DOKG s0
209
201
0.572
0.6.9
0.736,m000
0.736
(022
0.291 inxaa
0.521 0.521
0:41
0.41l
0.340
0.576anas5
0:364 sore
0177
0.000 anae
10 soar
0,029 0.129 aorre
Incepiioe
ACC
0.356
o.92
0.425
nuns
0.394
0.373
0.327
0.382
0.365
0.333
0.3715000
0.342.001
0.31
11 770000
5005
0.292 m0m
0,245 (24)
0,226, no0
0,559 mmm
0,351
0,621
0.566
0.538-0m
0.485
0-485
0.603
0.382
13522000
0.573500m
0.72%
(m))
0974
0.93
1002
0.926,
.092
0002
(1
ASA
0.466
0.357
nus
0.379 m0m
0:33-30m
0.337
0.30
0-422 200r
0--22
01-92
0,500 2000
0.488
0.332
0,779 moon
00000-005
0.354
0.665
0.,94 mos
0,282
mum
0,538
mum
0, m00
0,26700m4
0, 0,300
0.300aum:
20m
0.298a0ms
0.298
0,321-m00
0.359 0.35%
0.07750m
0.077
0,022
8.022a0m
0,020.001
0,024 (no:
cz
0.391
0.276 soomn
0,341 0.341
0:200.0000
00
0:314.0000 /0
0,28555.00
sn on
10.02
cad
15:00
0.572
04188s5.o
so0g
0.529-0000
ReNNel
ACC
0.284
0.340
0.391
0.340
0.110
0,
0,285
0,099
0.23
0.212
0,231
0.574
0.594
0.11
0,507
0,499
0.118
0,577
0.529
0.3
0955
0.47
ASA
0,576
0.169 m-169
mus
0.399
04483
0070
0558
0.475300s
0.51
0,73
0,58
0.117a0006
0.5
1054% 13
0,000
0.59410044
0.678
0,715
0.470
0.2
0319
0.376puss
0 0.352mass
033340022
022440002
0.32940r5
0037
D01
8018
0022
MISSSIM
0.965
0.9677
0,943
0.958
0.955
0,920
0.944.0 a0 on
on
0.9460.000
0,901 1000la0w wa
0,929
0.922 10
0,926
0,000 aas
0.860 1.860 inos
0,000
0,311
0,311
0,95 0,951 sno
0.958.000
0.96
0,568
0,59
0.603
0,934auus
0,992
0197210-0
0,985 0.98510.0
ocr
Metic
LPTS
mm
0.025 mm
0030-um:
0.300
0,72
0,048
mam:
0.59
0,000
0.022000
0.066
w0
200
0.770.0mm
0.084
0.301 2o0s
0.207-0ms
0,207
imm
0,202
0,043 20m3
0.000a004
6.57
0.33
0.30aamms
0.308_0ms
6.881
6,081
0.011.cm
0.011
som:
6.026-00m
8.17 0.07.0m1
2001
10.000
0.815 10.00
0.815 0.815
0,114 20001
50.0
0,808 0.808
0811 01
0,79955tci ci0
0.00555001
0.434 20.0
0,55 10:01
0.521 Hoci
0.530 0.530.2.0
0,7% 10.0
0,770 6.779
0.780
0.7 s7.00
0.780 0.710-2.01 20
0,621 sovor
0.6.2 an oo
0.814
0.12
0,808
0,808
0,785
0,620
0.211
NO CA
Mes Nete DeC.WA
Rex Neve WWA
Res Novi Now- DAC
Res Nort Nort-DDCooO DACA
Summgglt
50
VIT-8/16
ComvNast-T
Res 5o0
VI1-8816
ComvNeut-T
Kes Ne50
V1T-8816
ComyNost-T
VIT 1V16
TomnNost-T
Re:Ne:50
ConvNest-T
50
VITERVI6
ConvNat-T
ResNe:50
VITEBVI6
ConvNos--1
Ner3o
VIT-1V16
008%
ours....
mifgam

---

ACC
1.122000
0.228.000
0.27%
0.005
m005
0.112mmm mam
0.11 200m
0.243
0,007ano0
1:3920000
n.2252001
0,00
(119
(.10020000
0.20)
1.20).0mm
0.000.mm
0.456mmm
0.359amm
0:403
0.36%
0.453
0.4532000
0.448-000
0.085
0.245200
0.2942000
0.392
0.354mmm
0.545
0.733
0.73
0.895 0.895mmm
0.636.uuxs
Vit B/16
ASR
0.459 0.459
2003
0.311
0.311anmm
inns
0,978
0.019 mons
0.506
200ss
0,384
0.902
0.000
11002m00r
01664200
n44322001
0.906
U,745
8.7402000
0.49) 0.49) anos
1.000
1.00
01.24 0.244 am0m
1.199 1007
0.367 2.267
0,25 23nn
0.20 20
0.201
11442001
0.990
0,507 2002000
0.454
0.4541000
0.501 0.50l inm:
0.430
0.225 (000
0.035 m0m
0.327 mwnz
0.327
0.
Clevv Coov-Nest Next T
ACC
0.204
0.204__no
0.226
0.1 15
0,013 0.00
0.150
6.150_c.om s.mm
0,194
20000
0:1201-tton
0.10X_tion
0,102
0,001
E0 (0
0.066-000
0.127_5.0
0.0R6
-0000
(.397.n.0m
n.00
0.276
0,439
0.010
0.550mmmm
0.195
0.279
0.27920000
0.377-000
0.340 c.
0,503
0,726
6,226_.0000
0.581 -com
0.658 cue
0.1
ASR
0:981-000
mm
0.482_nm
0,300 mam
0.621 2.0
0,622 6007eton
0.701 0.701=noo =nou
0.535 0.585-0o
-00000000
0.608-nm
0.245 m.
(1090 -0:00
0.273 m.02
0.21-aam
0,982 2000000000
6.186 180m
0.201-moo coom
0.,08_000 2000
d.700 d.700
0.502 20002.000
0.458 0.258-000
0.508 an
m00
nuzz
0:981
0.484
0.59%
0.543
0--96-
0.90
0.697
0.999
0.008
0-09K
0.21
0.201
0.405
0.222
0.29%
ACC
a1 00
0.019
0.001
20.000
0.182
0.007 0.007-00.01
01000
0.1 1155ta0.00
0.100
0:003
00
0. 1.130g0.00
0.000 0o
0.000-5.00
0,110
(1.35
10.00
0.019
0.019.m.00
01420
0.30
0,040
20.00
0.541-1 10000
d:21
d2211
0.200
0.530
0.350-m.00
0.302
0.302mm.m a..
01527
0l.79 1.479-hmm)
0.831 soue
0.060
ReeNerS
Table 11: Original ImageNer label transfer attack results.
ASR
(0571a 00
0.940
0,9977
0.9977..00
1::00
0,470
0.999
0.557a3:00
0.712 s0.0z
0.991
0199120:00
20:00
1.00
0.62790000
0,772 on
(1599190000 :00
1.000.0.00
0.682
0,800
0.999 cocm
0.967 sm:mi
0.249
0,30
0.917 c0
0.99120:00 go:0n
0.310
01689
0,601
0.601
0.1889200
0.566...0
0,400
0,487
0.091-ccom rocm
10300.ccaa
ACC
0.3000.000 10:00
(1528 01522860000
0.212
15.00
0 0215 55.00
0.305
0 1770s:c00
0244 20:00
(127250000
01143
oi
1333355000 on
(200g5000
[21
(L
20:00
0.674 1c.0
(.303 11303sc.om sccmm
01493
(1903
0.400
(1484 20:00
0.1
0411=5000
15:000
(1305
(1.222- 122
(1423 s500000
0.385 (.383.c.om
.c.00
(USB3
0.765 25:00
0.883 -0.02
0 sc.22
Swin B
ASR
035420008
0.374
0.275
0.478
0.030 funce
0.179
0.336
0.577nm)
11000
0.562200c
0.399
0.644.00)
0.535nmm
0.7010as
0.23 Lonce
joucs
0.35%anaci
119220ss
0.27
11477001
0.2674008
0.5492001
0.400g0001
0.467 44671003
0.433
0. 01180000)
0.295 Haa2
0 0,701
0.5
0
ACC
0.1081000
0. 117740000
0 1600nn0
0..04 100R
0199 0199
0.142 21:420001
0. 42,401
11122
0. 10
0 23 10.0
0.334
0.138 1oate
0191 0190 aooii
0 sood
0.376
1337620001
2005
0.348.0000
0,2290
0.3,5 (0/5)
0.608 10.001
0.651 muzz
IncepttonVS
0.108
0.190
0 88
0-41
0.303
0.200
0.521
ASR
0.500.00
0.567
(.567.mm 2000
0,465 (002
0,470
0.680
0.517am:
0.548an00
0.762200
(10072000
2003
0.599
0.767 2012
2000
0.699,
(.649.000
0.651
(1.39 2000
mo2
0.600.um
0.25
1,25:amm ao0e
0. 0.XX
(.247
24777mmm
1.288-00
0 1.228%-0m
0.699
(.502.mm)
0,530 am
0.448.000
0.247.no.
0.050
m.050 mm:
0 0.304 0.304.uuz:
0.
0.
ACC
0.133 (1:33.m) m000
1.157.m
0.157
188
0.08 0.084.0m Jume
0,123 200s
0 1888
010000 sum
01050
(11982000
0. 1.6
(084
0.079 20m:
0.12
0.3.3
1.3.3.mm in0
mo:
0:45gmm
0.338 1338.mm
0. 1.300 18tzomm
2.3722000
6 1.314
0.221 0.221
6.280a000
(.358.mm)
1.327-mm
0.327
0.341 0.34
0.672 0.672.0m (000
mum:
0.673
suic:
RexNer1
0.
0.
ASR
Lovil
0.654
Loni
0,559
0,473
0.004
0,782 7722amm mmmn
0.650
0.553
0,7500
8.724gmeo
0.619
01.900 P
0:9122001
0.7:5
8.775.mmm
0.687
0.3022
inen
0.400 mon
0.697 11977amm summ
0 0,221 1222 anted
0.350 snau
0.257 annm
0,274amm
0.673
0.000.000
0,500
0.550amm
0.460laa
0,289
mnmi
0.046
mun
0,295 (1.95..uu
MSSSIM
0.923
0.923_0000
1000
0.979
0.977
c.om
6.00
..60.cmm
0.960
0,970_5.00
0.967 01967.1.00
0.230 cOM
01947
11977.000
0.961
-1.00
0.956
(1347
0,952
0,945
0.577
(100
c.m
0.889
sco
856-
0.978
0.900
0.977
01432
(1.21 o00
0433-0017
0.41-ccm
0.44
0.312-601
0.761
-:00
c.m
0,400 c.m
Metiics
1PPPS
-000-0000
0.002
-000
0.015
0.019
000
0.033
0.023
0.028
01056 -000
01044 -000
0.111 0.111-000
=000
0.038
(LAK3 (000
0455-0000
0.099 -non
018
0.00
21 mi0
0.209 0000
0.208
0 0.205-.
(1,20
non
0.027 =000
04033 2000
0.021 0.021-0m
(L-700 no
0.457- -000
0.459 aan
0.471=00:
0.471
0 1.333 -001
0.0K5 am
0 10300
3300-uzz
Clipp
1
100000
1
19.8
a0.05
0.819
0.81
0,85500000
018 0.815
0.822...0
01543
0.548 oo.0
0.535
0.559
0,803 200
0.796.2200
11348gn.000
01845
Rc 0.84130011
0.84...0
0.592
0.638 10.004
0.821
0.820
0.:96
Sumognte
ResNettDDeC
RecNe50
V4-8/166
1
ResNeHDDCOMN
Vi-3/11
T
ReaNGHDDCOON
Ver 18/6
Se
Vir B/16
ComvNEne
S0
V1-8/16
ResNorDDCCOMi
50)
Vir Vir-
DesNaaDDeOOMi DeOloi
Via 8/16
Com Commiert Next
Resheett DeCooWa
ResNee50
V2-8/116
methed
ours
MLROSM
VENOM
SD-NAE
53


Reviews:
[{"review_id": "Ag9gaiHIqI", "reviewer": "Reviewer_t5zq", "summary": "This paper focuses on the topic of semantic-constrained adversarial examples. This paper develops a multi-dimensional instruction uncertainty reduction (InSUR) framework to generate more satisfactory SemanticAE. Their framework proposes residual-driven attacking direction stabilization, context-encoded attacking scenario constraint, guidance masking and renderer integration techniques to solve the uncertainty problems. Extensive experiments demonstrate the superiority of the transfer attack performance of InSUR.", "questions": "How to control the abstracting level of the antonyms of the Text. Cat is an antonym of dog, and car is also an antonym of dog, but they are different-level antonyms. How do you consider this kind of different-level antonyms?\n\nHow to guarantee the naturalness of the proposed adversarial attacks. We know that diffusion models can generate realistic images, but some of the details are unreal. Your approach utilizes the diffusion models, so how to guarantee the naturalness of the generated images?\n\nI am not sure about the real use cases of semantic AE. Is the semantic AE possible in the real world or does the semantic AE have real world application?\n\nI understand that the adversarial examples can be used for single object evaluation, but can you extend the approach for more complex scenarios in general VQA tasks, e.g., the image contains multiple objects? Can your adversarial examples be able to generate such adversarial images.\n\n(Minor) Typos Line39: SemanticAEhas => SemanticAE has", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n+ Semantic AE is interesting\n+ The approach is concrete\n\nWeakness\n\n-\tThe naturalness of adversarial examples cannot be guaranteed\n-\tThe real use case is unclear\n-\tDifferent abstracting levels should be discussed.", "was_revised": true, "final_justification": "After the rebuttal, I increase my score to borderline accept.", "rebuttal": "We sincerely appreciate your constructive feedback. Below are our responses to the questions raised.\n\n**Q1:** How to control the abstracting level of the antonyms of the Text. Cat is an antonym of dog, and car is also an antonym of dog, but they are different-level antonyms. \n\n**R1:**  We acknowledge that the term *antonyms* in the original manuscript (Section 3.1) was imprecisely used and could lead to misunderstanding. Specifically, $A_{\\text{Text}}$ refers to output types that are conceptually *different* from the original text, not necessarily strict linguistic antonyms. In fact, in the untargeted attack scenario, these should be understood as *non-synonyms* (e.g., *cat* is not an antonym of *dog*, but it is a non-synonym). As in prior transfer-based attack works, our experiments focus on the untargeted setting to evaluate effectiveness.\n\nOur proposed benchmark already addressed the **evaluation bias** caused by overly fine-grained semantic abstraction. We agree with the reviewer that different abstraction levels inherently lead to different attack difficulties, as supported by prior work [1]. To account for this, we define the abstraction level of $A_{\\text{Text}}$ via the scope of synonym coverage: broader synonym sets imply more distant \"antonyms.\" In our experiments, we evaluate across both *Abstracted Labels* and *Refined Labels*—representing different abstraction levels—to comprehensively assess the performance of SemanticAE methods under varying semantic granularity.\n\nThe reviewer’s suggestion, “Different abstraction levels should be discussed”, can be further addressed through finer-grained evaluation, which we support. We also agree that VQA provides a more general and flexible framework for multi-level evaluation, potentially extending to alignment studies. We present VQA results under different abstraction levels in our response to Q3.\n\n---\n**Q2:** We know that diffusion models can generate realistic images, but some of the details are unreal. Your approach utilizes the diffusion models, so how to guarantee the naturalness?\n\n**R2:** While generating natural adversarial examples has been studied by many prior works, our core contribution lies not in naturalness, but in exploring and constructing adversarial examples in a broader semantic space, with a focus on key properties such as *transferability* and *robustness*. Hence, our title emphasizes *Semantically Constrained Adversarial Examples*, not *Natural* ones.\n\nIn our technology contribution, diffusion models are primarily used for *on-manifold regularization* to enhance transferability. Nevertheless, we do incorporate naturalness constraints by adapting existing techniques to ResAdv-DDIM. Specifically, we constrain the difference between non-adversarial and adversarial denoising steps during diffusion sampling (as shown in Eq. 7):\n$$\n\\|\\|\\textrm{Denoise}\\_\\textrm{DDIM}(x\\_{t\\_s - \\Delta T})-\\textrm{Denoise}\\_\\textrm{Adv}(x\\_{t\\_s - \\Delta T})\\|\\|\\_2 < \\epsilon\n$$\nBoth visualizations and quantitative results confirm that our generated samples exhibit sufficient naturalness.\n\nMoreover, we argue that the loose realistic constraint does not challenge the significance of the study, specifically:  \n1.  **From an attack system perspective**, if our method is used as a module in a real system, generated samples can be filtered using semantic plausibility checks. If both generator $A$ and discriminator $B$ (not under attack) deem an image $X$ semantically valid, we have strong justification to treat $X$ as natural. We formalize this in Appendix B.3 via a proposition linking our evaluation metric to practical utility. In real attack scenarios—whether physical attacks or jailbreaks—the targets are often automated systems; our evaluation aligns well with modeling such defenses. We further analyze attack performance under defenses in Appendix D.  \n2. **For adversarial training**, recent work [2] (ICML 2025, oral) shows that diffusion-generated samples—even with minor imperfections—can significantly improve training efficiency and model performance, enhancing scaling laws. Thus, minor realism flaws do not undermine their utility. We detailed this work in our response to Q3.\n\nThe *realism* of generated images is more central to diffusion model alignment research. Our framework may also inform *Test-Time Scaling of Diffusion Model Alignment*. While recent works[3] use reward models to refine generation, we explore the inverse: what optimization structures induce robustness failures in such reward models?\n\n---\n**Q3:** Is the semantic AE possible in the real world or does the semantic AE have real world application? I understand that the adversarial examples can be used for single object evaluation, but can you extend the approach for more complex scenarios in general VQA tasks, e.g., the image contains multiple objects? Can your adversarial examples be able to generate such adversarial images.\n\n**R3:**  Attributed to the technical improvement on the key problems, our framework has broad applications:  \n1. **Adversarial training**: Our framework can adapt to a concurrent ICML 2025 [2] work proposing diffusion-based data generation for training, which uses the target model $f_\\phi$’s prediction entropy $H$ to guide diffusion sampling to generate challenging samples: \n$$\\textrm{Guidance Signal} \\propto \\omega \\nabla_{x_t} H(f_\\phi(\\hat{x}_{0,t}))$$ Our **ResAdv-DDIM** can enhance this pipeline by stabilizing multi-step adversarial directions through a refined estimation of $\\hat{x}\\_{0,t}$. This improves sampling accuracy and speed, while offering new insights into transferability. Since high-transferability adversarial examples expose non-robust features shared across models, they may offer advantages in adversarial training. We will include this discussion in the revised manuscript.\n\n2. **Jailbreak and physical attacks**: We have already demonstrated effectiveness in cross-domain settings (2D → 3D). In complex systems, *inaccurate adversarial feedback* is a key challenge. The success of our method in 3D scenarios—where multi-view feedback is often incomplete—shows it can handle such uncertainty. We elaborate further in our response to Reviewer vVyk.\n\n3. **Extension to complex VQA tasks**: Both the generator and the surrogate model are replaceable in our framework. Our 3D generation experiment validates the cross-task extension capability. For images with multiple objects, one can simply include multiple object descriptions in the diffusion model’s prompt. For applications in autonomous driving, our framework could guide the adversarial data generation techniques developed based on the scene generation diffusion models in the simulation pipeline.\n\nWe conducted additional VQA experiments using OpenAI CLIP models {RN50, ViT-B/32, RN50x16} (with DeCoWA) as surrogates, generating adversarial examples under three different *abstraction levels*. For each answer option, InSUR generates samples satisfying the corresponding semantic constraint $\\mathcal{S}_\\text{Text}$ ($\\text{Text} \\in \\{\\text{Options}\\}$), using stabilityai/stable-diffusion-2-1-base. We evaluate transferability across OpenAI CLIP variants (RN50, RN101, RN50x4, RN50x16, ViT-B/32, ViT-B/16) and large VLMs (LLaVA, Qwen 7B). Results show:  \n- InSUR achieves strong cross-model transferability and enables weak surrogates to attack stronger models via diffusion assistance, though transfer weakens with greater semantic distance.  \n- Due to the widespread adoption of ViT architectures in modern VLMs, ViT-based surrogates show stronger transfer to LLaVA and Qwen.\n\nWe believe these findings are valuable for both alignment and *super-alignment* research. We will add the discussion to the camera-ready version.\n\n--- \n **Experimental Results:** \n\n\n**Question:** \"What brand is this car?\", **Options:** [\"Audi\", \"Mercedes\", \"Ferrari\", \"Toyota\"]\n\n**Accuracy(%) After Attack ($\\downarrow$)**\n\n|Surrogate|RN50|RN101|RN50x4|RN50x16|ViT-B/32|ViT-B/16|Llava1.5-7B|Qwen2.5VL-7B|\n| - | - | - | - | - | - | - | - | - |\n| RN50 | 2.5| 30.0| 37.5| 47.5| 55.0| 50.0| 67.5| 70.0  |\n| ViT-B/32  | 32.5 | 37.5| 45.0| 65.0| 0.0 | 40.0| 57.5   | 32.5  |\n| RN50x16 | 47.5 | 52.5| 37.5| 5.0| 70.0| 65.0| 67.5   | 57.5  |\n\n**Question:** \"What is in the picture:\", **Options:** [\"Police car\", \"Ambulance\", \"Taxi\", \"School Bus\"]\n\n**Accuracy(%) After Attack ($\\downarrow$)**\n\n|Surrogate|RN50|RN101|RN50x4|RN50x16|ViT-B/32|ViT-B/16|Llava1.5-7B|Qwen2.5VL-7B|\n| - | - | - | - | - | - | - | - | - |\n| RN50|0.0| 32.5  | 45.0 | 50.0  | 40.0| 52.5| 70.0 | 60.0 |\n| ViT-B/32| 10.0 | 7.5  | 17.5 | 17.5 | 0.0 | 10.0| 25.0| 12.5|\n| RN50x16| 30.0 | 40.0 | 32.5| 0.0| 55.0| 52.5| 65.0| 57.5|\n\n**Question:** \"What is in the picture:\", Options: [\"Air Liner\", \"Car\", \"Helicopter\", \"Cruise Ship\"]\n\n**Accuracy(%) After Attack ($\\downarrow$)**\n\n|Surrogate|RN50|RN101|RN50x4|RN50x16|ViT-B/32|ViT-B/16|Llava1.5-7B|Qwen2.5VL-7B|\n| - | - | - | - | - | - | - | - | - |\n| RN50 | 5.0  | 70.0  | 72.5 | 92.5 | 92.5| 85.0| 90.0| 82.5|\n| ViT-B/32  | 47.5 | 65.0  | 67.5| 77.5 | 0.0 | 62.5| 70.0 | 67.5 |\n| RN50x16 | 60.0 | 67.5  | 75.0| 10.0| 85.0| 92.5| 92.5 | 87.5 |\n\n**Overall Accuracy After Attack(%)($\\downarrow$)**\n\n|Surrogate|RN50|RN101|RN50x4|RN50x16|ViT-B/32|ViT-B/16|Llava1.5-7B|Qwen2.5VL-7B|\n| - | - | - |-|-| - |-|-| - |\n| RN50 | 2.5  | 44.2| 51.7| 63.3| 62.5 | 62.5 | 75.8| 70.8 |\n| ViT-B/32| 30.0 | 36.7| 43.3| 53.3| 0.0| 37.5 | 50.8| 37.5 |\n| RN50x16| 45.8 | 53.3| 48.3| 5.0 | 70.0 | 70.0 | 75.0| 67.5 |\n\n[1] Wang, Jiakai, et al. \"Generate transferable adversarial physical camouflages via triplet attention suppression.\" *International Journal of Computer Vision* 132.11 (2024): 5084-5100.\n\n[2] Askari-Hemmat, Reyhane, et al. \"Improving the Scaling Laws of Synthetic Data with Deliberate Practice.\" arXiv preprint arXiv:2502.15588 (2025).\n\n[3] Jain, Vineet, et al. \"Diffusion Tree Sampling: Scalable inference-time alignment of diffusion models.\" *arXiv preprint arXiv:2506.20701* (2025)."}, {"review_id": "hFJ0iL8Doc", "reviewer": "Reviewer_rjVJ", "summary": "This paper addresses the challenge of generating semantically constrained adversarial examples (SemanticAEs) from natural language instructions, which suffer from instruction uncertainty due to referring diversity, descriptive incompleteness, and semantic boundary ambiguity. The authors introduce the InSUR framework (Instruction Uncertainty Reduction), which combines a novel ResAdv-DDIM sampler for stabilizing adversarial optimization, context-encoded constraints for adapting to incomplete instructions in both 2D and 3D settings, and a semantic-abstracted evaluation using label taxonomies. On the ImageNet abstracted label evasion task, the proposed method achieves a 55.4% attack success rate (ASR) with the ViT-B surrogate model, outperforming prior approaches in transferability and semantic fidelity (Table 1).", "questions": "- In Eq. 4, how sensitive is the ResAdv-DDIM sampler to the choice of k?\n - In Eq. 6, could you clarify how the thresholds ξ₁ = 0.1 and ξ₂ = 0.01 were chosen? Were these hyperparameters tuned for attack performance, or are they fixed across all tasks/models?\n - In Line 192–193, the author’s mention “M is the guidance masking that regularizes the ....\". It is unclear how M is defined. \n - In Table 2, given that generation times vary widely (e.g., up to 59s), is most of this cost from the Gaussian splatting renderer, diffusion steps, or EoT sampling? Can you attribute bottlenecks quantitatively?", "limitations": "yes (although provided in the Conclusion section)", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n - The reviewer likes the approach of InSUR that introduces a multi-dimensional approach combining residual-based diffusion guidance, context-aware 2D/3D generation, and semantic evaluation via label taxonomies, offering a robust pipeline for reference-free, instruction-guided 3D adversarial generation.\n - The proposed method significantly outperforms prior methods (SD-NAE, VENOM, AdvDiff). For example, on the abstracted label evasion task with ViT-B surrogate, the proposed method achieves 55.4% ASR versus 33.6% for SD-NAE and 40.3% for VENOM, while maintaining high perceptual quality (e.g., LPIPS of 0.039).\n - The paper is clear and concise in writing.\n\n----\n**Weaknesses:**\n - While the paper evaluates transferability across six standard architectures (e.g., ResNet50, ViT-B/16, ConvNeXt), it does not test robustness against, foundation models like CLIP (despite using CLIP-IQA as a metric), as   semanticAEs are meant to probe misalignment in general perception models. The reviewer feels that restricting evaluation to classification networks undercuts the paper's broader alignment claims.\n - In the 3D pipeline, the choice to freeze the structured latent (z₀^slat) is heuristic. There’s no ablation to show whether optimizing over it would help or hurt performance.\n - Given that instruction-driven generation is central, it’s surprising that prompt attacks on vision-language systems like [1] ([13] in the paper) are not used as baselines. \n - Typos\n    - Line 39: \"SemanticAEhas\" → should be \"SemanticAE has\" (missing space).\n    - Line 151: \".. improve the approximation...\" → should be \"by improving\".\n    - Line 203: \"Senario knowledge...\" → should be \"Scenario knowledge\".\n    - Line 298: \"Tabel 2\" → should be \"Table 2\".\n----\n[1] An LLM can Fool Itself: A Prompt-Based Adversarial Attack, ICLR 2024", "was_revised": true, "final_justification": "The reviewer read through the comments of other reviewers and the rebuttal and found the rebuttal to be satisfactory. Hence, the reviewer is updating the rating to \"Accept\".", "rebuttal": "We sincerely appreciate the reviewer's positive feedback on our approach and their meticulous proofreading to improve the readability. Below are our responses to the question raised.\n\n**Response to Weakness 1: limited evaluation on foundation models.**\n\nWe thank the reviewer for the comment. While our primary evaluation focuses on standard architectures, we acknowledge the importance of testing on foundation models, given the paper’s emphasis on semantic alignment. To address this, we have conducted additional experiments on VQA tasks, as detailed in our response to Reviewer t5zq, with results presented at the end of that reply.\nFurthermore, in Appendix D.4 (provided in the supplementary material), we show that our generated adversarial examples effectively perturb the CLIP-VQA evaluation metric itself, indicating that semantic assessment based on CLIP is also vulnerable to such attacks. This highlights a key challenge: evaluating adversarial examples becomes difficult when the evaluator model is not robust.\nTo mitigate this dependency, we propose the $ASR_\\mathrm{relative}$ metric, which reduces reliance on absolute semantic evaluation by using relative performance comparison. We believe these findings are valuable for research on alignment and even *super-alignment*, as they reveal vulnerabilities not only in perception models but also in the metrics used to assess them.\n\n---\n**Response to Weakness 2: freeze the structured latent (z₀^slat) is heuristic**\n\nOur choice of freezing the structured latent is not purely heuristic.  The reasons are:\n1. $z_0^{slat}$ encodes the 3D positioning data, while the accurate positioning is also rectified by $z_0$ in the *Trellis* framework. We could achieve the positioning disturbance also by optimizing $z_0$.\n2. Estimating the adversarial feedback from the surrogate model to  $z_0^{slat}$ is not computationally efficient, since it shall undergo the entire denoising diffusion process.\n3. The computation from $z_0^{slat}$ to $\\mathbf{pos}$ contains a quantization operation, which is non-differentiable. \n\nFurthermore, $z_0^{slat}$ is an internal data structure of Trellis, and the optimization method customized specifically for this structure will be constrained by the *Trellis* framework. Besides, even with z₀^slat frozen, we still achieve favorable optimization results. We believe such optimization shall be developed together with harder attack tasks.\n\n---\n**Response to Weakness 3: limited evaluation on prompt-based attacks.**\n\n\nThanks for the comment. As stated in Section 3.1, our work focuses on a fixed instruction: *how to generate an image that satisfies the semantic constraints of that instruction while being adversarial*. We acknowledge that perturbing the instruction itself (i.e., prompt attacks) can also lead to adversarial effects. However, most existing prompt attacks are not designed for tasks similar to SemanticAE.\n\nOur baseline, **SD-NAE**, performs attacks by perturbing the prompt embedding, making it approximately equivalent to prompt attack methods, but more aligned with our task objective. We will include a detailed description of SD-NAE in the Appendix, and ensure it is appended to the main paper in the camera-ready version.\n\nMoreover, we discuss this distinction in Appendix A.2. Specifically, generating hard samples from language can be approached via two paradigms:  \n(1) *Perturbing the input text without altering the language-guided generation model*. \n(2) *Altering the generation process by introducing adversarial capabilities into the model*.  \nOur work focuses on the latter. In practice, these two approaches can be combined into a unified pipeline. Our evaluation could be regarded as assessing the effectiveness of our method as a *module* within the red-team framework.\n\nWe further tested a recent method that uses prompt optimization to generate adversarial images [1]. In our local single-GPU setup, it took **several hours** to find a single adversarial prompt, which is thousands of times slower than our method and the baselines we compare against. This performance gap significantly limits its applicability in scalable red-teaming systems, and it shall be deployed in different parts of the system compared to our method.\n\n\n[1] Zhu, Xiaopei, et al. \"Natural language induced adversarial images.\" *Proceedings of the 32nd ACM International Conference on Multimedia*. 2024.\n\n---\n\n**Q1**: In Eq. 4, how sensitive is the ResAdv-DDIM sampler to the choice of $k$?\n\n**R1**: We conducted ablation studies comparing different $k$ value settings (Iter$_\\mathrm{max}$ in Table 3 of the main paper). We will clarify it to make it clearer. As $k$ increases, attack effectiveness gradually improves, but so does computational cost. In practice, $k$ can be adjusted based on the specific application. Furthermore, following the suggestion from Reviewer vVyk, we provide additional analysis on the stability of adversarial optimization under semantic guidance instructions with varying ambiguity.\n\n---\n\n**Q2**:  In Eq. 6, could you clarify how the thresholds $\\xi_1 = 0.1$ and $\\xi_2 = 0.01$ were chosen? Were these hyperparameters tuned for attack performance, or are they fixed across all tasks/models?\n\n**R2**: They are fixed across all tasks and models. In our framework, $\\mathrm{Confidence} < \\xi_1$ indicates that the \"residual approximation\" predicts the attack performance will be sufficiently strong, so the inner optimization at the current denoising step is terminated and the process proceeds to the next denoising step. When $\\mathrm{Confidence} < \\xi_2$, it means the predicted attack performance already exceeds the task requirement—further optimization is unnecessary, and even if subsequent denoising steps improve naturalness at the cost of reduced attack strength, this is acceptable. These conditions reduce the expected number and lower bound of optimization steps. In Appendix B.1, we detail analyze the benefits of this design on generation efficiency (Proposition B.1 and related discussions).\n\nWe chose these threshold values heuristically and used them consistently across all tasks without manual tuning for attack performance or computational speed.\n\nBelow, we present results using ResNet50 as the surrogate model on the Abstract Label Evasion Task. In white-box attacks, all tested threshold values achieve strong performance (accuracy after attack < 3%). In black-box attacks, smaller $\\xi_1$ values yield higher transferability, because lower $\\xi_1$ leads to more aggressive adversarial optimization in early denoising steps—optimization that tends to be more transferable (this is consistent with the design principle of ResAdv-DDIM, which aims to enhance early-step optimization). However, this also increases computational cost. A practical deployment could balance effectiveness and efficiency.\n\n|$ξ_1$|$ξ_2$|Resnet50-ACC$\\downarrow$|ViT-B/16-ACC$\\downarrow$|\n|-|-|-|-|\n|0.15|0.02|0.0224|0.2949|\n|0.15|0.01|**0.0192**|0.2885|\n|0.15|0.005|0.0214|0.2885|\n|0.1|0.02|0.0246|0.2853|\n|0.1|0.01|0.0246|0.2724|\n|0.1|0.005|0.0246|0.2714|\n|0.05|0.02|0.0224|0.2404|\n|0.05|0.01|0.0256|**0.2382**|\n|0.05|0.005|0.0246|0.2511|\n\n**Q3:** In Line 192–193, the authors mention “M is the guidance masking that regularizes the ....\". It is unclear how M is defined.\n\n**R3:** The definition of $M$ is explained in Appendix B.1 (in the supplementary material). We will include the appendix after the main text in the camera-ready version and add a reference to it after this sentence. Specifically,\n$$\n\\begin{equation}\n    M_{ij} := \\begin{cases}\nM_\\mathrm{mid} & \\frac{h}{16} \\le i < \\frac{15h}{16}, \\frac{w}{16} \\le j < \\frac{15w}{16}, \\\\\\\\\nM_\\mathrm{edge} & \\text{otherwise}.\n    \\end{cases}\n\\end{equation}\n$$\nIn the main experiments, we set $M_\\mathrm{mid} = 3.0$ and $M_\\mathrm{edge} = 0.3$. We further test $M_\\mathrm{edge} = \\{0.0, 0.3, 3.0\\}$ in the ablation study.\n\n---\n\n**Q4:** In Table 2, given that generation times vary widely (e.g., up to 59s), is most of this cost from the Gaussian splatting renderer, diffusion steps, or EoT sampling? Can you attribute bottlenecks quantitatively?\n\n**R4:**  The variation in generation time arises from the adaptive number of optimization steps and differences in the size of sparse tensors in the Trellis framework. Attributed to the proposed early stopping mechanism, easier attack cases require fewer iterations and thus less time. The large variance in generation time reflects significant differences in attack difficulty across tasks, which highlights the effectiveness of our adaptive step design.\n\nWe further conducted a timing analysis on a single GPU using 100 labels from the Imagenet-Label dataset. The results are shown below. Gradient computation and residual approximation account for the majority of the computational cost. Meanwhile, *EoT sampling* is implemented via multi-view rendering using the Gaussian splatting renderer, while the rendering parameter computation does not occupy CUDA execution time.\n\nFor the single adversarial optimization iteration:\n - Denoise Sampling: 123.321 (std=36.878)ms\n -  Residual Approximation: 147.256 (std=76.243)ms\n -  Gaussian Decoding: 21.547 (std=11.868)ms\n  - Gaussian Rendering: 28.556 (std=7.706)ms\n -  Surrogate Model: 6.777 (std=0.745)ms\n  - Backward Process: 208.471 (std=78.039)ms\n\nThe fluctuation in computation speed is primarily caused by inconsistent data sizes in the Sparse Tensor.\n\nFor the entire adversarial optimization:\n  - Denoise Sampling: 2994.933 (std=269.841)ms\n -  Residual Approximation: 7040.238 (std=8915.112)ms\n -  Gaussian Decoding: 1029.394 (std=1185.636)ms\n -  Gaussian Rendering: 1364.221 (std=1732.996)ms\n  - Target Model: 323.782 (std=423.503)ms\n -  Backward Process: 6261.577 (std=12119.871)ms\n\nThe fluctuation in computation speed is also attributed to the adaptive optimization steps. Specifically, the average count of optimization steps for the inner optimization is: 47.810 (std=63.566)."}, {"review_id": "q32JWjvIQ4", "reviewer": "Reviewer_FdBW", "summary": "This paper mainly try to addresses the problem of generating semantic-constrained adversarial examples (SemanticAEs) directly from natural language instructions, a challenging task that moves beyond traditional perturbation-based attacks. The authors identify a core challenge: \"instruction uncertainty,\" which they thoughtfully categorize into three key issues. To tackle these issues, the paper proposes a multi-dimensional framework named Instruction Uncertainty Reduction (InSUR). The authors conduct extensive experiments on 2D and 3D tasks, demonstrating that InSUR significantly outperforms existing state-of-the-art methods in generating transferable and effective SemanticAEs.", "questions": "See weakness", "limitations": "The paper does not discuss the limitations of the proposed methods within the main text.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\n1. The proposed InSUR framework is technically sound and novel. The ResAdv-DDIM sampler is a clever solution to the problem of unstable gradients in multi-step generative processes.\n\n2. Integrating a language-guided 3D generator with an adversarial objective to produce reference-free 3D objects that can fool 2D classifiers is a major step forward for the field and demonstrates the flexibility of the proposed framework.\n\n3. The authors compare against a strong set of recent baselines (AdvDiff, SD-NAE, VENOM) across multiple surrogate models (ResNet, ViT, ConvNeXt), demonstrating robust transferability.\n\n# Weaknesses\n\n1. Regarding the ResAdv-DDIM sampler (Section 3.2): Could you provide more intuition on the choice of $k$ (the number of coarse prediction steps)? Is there a principle for selecting the optimal $k$, or is it purely an empirical hyperparameter? \n\n2. Regarding 3D generation (Table 2): The results are extremely impressive, with ASR jumping to $92.2\\%$. The non-adversarial baseline already has a classification accuracy of $21.5\\%$ on the target model. Does this suggest that classifiers are inherently less robust to rendered 3D objects, making them an easier target for adversarial attacks compared to real 2D images? \n\n3. Regarding the relationship between referring diversity and descriptive incompleteness. The paper neatly separates these concepts. However, could there be an interplay? For example, if an instruction is incomplete (\"image of a car\"), does this increase the diversity of valid references, making the optimization problem harder? How does your framework handle this potential interaction?", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your recognition of our work's technical rigor and innovation. Below are our responses to the question raised.\n\n**Q1:** Regarding the ResAdv-DDIM sampler (Section 3.2): Could you provide more intuition on the choice of $k$ (the number of coarse prediction steps)? Is there a principle for selecting the optimal $k$, or is it purely an empirical hyperparameter?**\n\n**R1:** In principle, within a certain range, a higher $k$ leads to stronger adversarial optimization in the early denoising steps of diffusion, resulting in higher attack transferability—but at the cost of increased computational time. We provide a detailed ablation study analyzing this trade-off in *Table 3* of the main paper (partial results shown below). In the original table, *Iter$_\\mathrm{max}$* corresponds to the maximum number of coarse prediction steps. The specific scheduling strategy for $k$, i.e., how many prediction steps to use at each denoising step based on remaining timesteps, is described in *Appendix B.1 (in the supplementary material)*, with $K$ being the maximum allowed prediction step.\n\n| $K$  | Acc. ($\\downarrow$) | ASR ($\\uparrow$) | ClipQ ($\\uparrow$) | MSSSIM ($\\uparrow$) | LPIPS ($\\downarrow$) | Time ($\\downarrow$) |\n|------|---------------------|------------------|--------------------|---------------------|-----------------------|---------------------|\n| 0    | 43.10%              | 43.30%           | 0.794              | 0.941               | 0.055                 | 4.53±0.19           |\n| 1    | 33.50%              | 54.70%           | 0.812              | 0.942               | 0.049                 | 6.87±1.48           |\n| 2    | 31.30%              | 56.60%           | 0.816              | 0.942               | 0.049                 | 7.44±1.92           |\n| 3    | 29.20%              | 57.50%           | 0.807              | 0.943               | 0.048                 | 7.75±2.37           |\n| 4    | 27.70%              | 60.10%           | 0.813              | 0.944               | 0.047                 | 7.87±2.31           |\n\nFurther mechanistic analysis on the impact of $k$ on optimization stability is provided in our response to Reviewer vVyk. We believe that the choice of $K$ should be made based on the specific application scenario, balancing effectiveness and efficiency. In practice, $K$ can be selected adaptively by integrating with AutoML frameworks, using example instructions $\\mathrm{Text}$ and stability metrics to determine an appropriate $K$ for the deployment environment.\n\n---\n\n**Q2:** Regarding 3D generation (Table 2): The results are extremely impressive, with ASR jumping to 92.2. The non-adversarial baseline already has a classification accuracy of 21.5 on the target model. Does this suggest that classifiers are inherently less robust to rendered 3D objects, making them an easier target for adversarial attacks compared to real 2D images?\n\n**R2:** We believe this is due to the nature of the task. Nevertheless, our results sufficiently validate the effectiveness of the proposed method. Specifically, \n1. Although the average clean accuracy across the dataset is low, for certain labels, the generated 3D objects achieve high clean accuracy under multi-view sampling. On these labels, our method still successfully generates effective 3D adversarial examples (visualized in Figure 11 of the main paper and the supplementary videos).  \n2. Our $ASR_\\mathrm{relative}$ metric is designed precisely to address this issue: it excludes samples where the clean version is already misclassified, eliminating the influence of inherently hard-to-recognize objects. $ASR_\\mathrm{relative}$ on clean samples could be regarded as 0%.  \n3.  Ablation studies show that ResAdv-DDIM significantly improves attack performance, indicating this is a non-trivial problem.  \n\n\n---\n\n**Q3:** Regarding the relationship between referring diversity and descriptive incompleteness. The paper neatly separates these concepts. However, could there be an interplay? For example, if an instruction is incomplete (\"image of a car\"), does this increase the diversity of valid references, making the optimization problem harder? How does your framework handle this potential interaction?\n\n**R3:** We agree with the reviewer that these concepts are interrelated. As the reviewer points out, we view *referring diversity* as having a *direct* impact on optimization, while *descriptive incompleteness* has an *indirect* effect. From a design perspective, we believe it is reasonable to first develop optimization methods targeting referring diversity, and then define contextural knowledge embedding methods that account for issues arising from descriptive incompleteness. Specifically, referring diversity characterizes the *optimization challenge* faced by transfer attack algorithms, while descriptive incompleteness relates to how the *optimization objective* is defined. InSUR provides a collaborative design solution for these two issues. We will add further clarification on this methodological design principle at the beginning of the methodology section to better address this point."}, {"review_id": "0boO4BQerN", "reviewer": "Reviewer_vVyk", "summary": "The paper \"Exploring Semantic-constrained Adversarial Example with Instruction Uncertainty Reduction\" introduces a novel framework, InSUR, for generating semantic-constrained adversarial examples (SemanticAEs) directly from natural language instructions. It addresses the challenges of semantic uncertainty in instructions, categorized as referring diversity, descriptive incompleteness, and boundary ambiguity. The InSUR framework proposes three key contributions: (1) a residual-driven attacking direction stabilization using the ResAdv-DDIM sampler to enhance adversarial optimization stability in diffusion models, (2) context-encoded attacking scenario constraints to improve scenario-adapted 2D and 3D SemanticAE generation by integrating external knowledge, and (3) a semantic-abstracted attacking evaluation enhancement based on label taxonomy to provide robust evaluation metrics. The paper demonstrates improved transfer attack performance for 2D SemanticAEs and achieves the first reference-free generation of 3D SemanticAEs using language-guided 3D generation models. Extensive experiments validate the effectiveness of the proposed methods across ImageNet-based tasks and novel abstracted label evasion tasks.", "questions": "1、Please clarify the residual-driven attacking direction stabilization mechanism. You should provide a detailed explanation of the residual term’s role in stabilizing adversarial direction. The ResAdv-DDIM sampler is a core contribution, proposed to stabilize adversarial optimization under instruction uncertainty by introducing a residual-driven attacking direction. The formulation (Eq. 2–4) adapts the DDIM process with a residual term and momentum optimization, but the technical details are dense, and the rationale for design choices (e.g., choice of momentum parameters) is not fully explained.\n\n2、This paper should include a quantitative analysis comparing ResAdv-DDIM to standard DDIM in terms of convergence stability (e.g., loss curves, gradient variance) or attack success rate (ASR) under varying instruction diversity.\n\n3、The paper emphasizes addressing instruction uncertainty (referring diversity, descriptive incompleteness, boundary ambiguity) but does not provide details on the natural language instructions used in experiments. The lack of information about instruction diversity (e.g., length, complexity, semantic variation) or their construction (e.g., manual, dataset-derived, or programmatic) makes it difficult to assess the robustness of the InSUR framework across varied inputs.\n\n4、Please validate or discuss real-world applicability of 3D semanticAEs.", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1、The paper presents a well-designed multi-dimensional framework (InSUR) that systematically addresses three forms of semantic uncertainty (referring diversity, descriptive incompleteness, and boundary ambiguity). The proposed techniques, such as the ResAdv-DDIM sampler, context-encoded scenario constraints, and semantic-abstracted evaluation, are theoretically grounded and build on established methods.\n\n2、The experimental evaluation is robust, testing the framework on ImageNet 1000-class label evasion and a novel abstracted label evasion task. The use of multiple surrogate models and transfer attack methods ensures a thorough assessment. The reported results, such as a 1.19× average attack success rate (ASR), demonstrate the framework’s effectiveness.\n\n3、Figures 2–6 and 9–11 effectively illustrate the framework, optimization challenges, and generated SemanticAEs. The visualizations of 2D and 3D samples (Figures 10 and 11) are compelling, showcasing the naturalness and adversarial effectiveness of the generated examples.\n\nWeaknesses:\n\n1、While the paper mentions potential real-world applications, it lacks empirical validation in physical-world scenarios (e.g., testing 3D SemanticAEs in simulated environments). This limits the immediate practical significance of the results.\n\n2、The 3D generation relies on the Trellis framework [41], which may reduce the perceived originality of the 3D pipeline, as it leverages an existing tool rather than proposing a fully new 3D generation method.\n\n3、Some components, such as the use of guidance masking for 2D generation, are less novel and resemble techniques in prior diffusion-based adversarial works (e.g., AdvDiff [16]). While the integration is creative, these elements may be seen as incremental improvements.", "was_revised": true, "final_justification": "Thank you for your detailed response. I have reviewed your revisions. My original concerns have been addressed. Therefore, I am satisfied with the revisions and will maintain my original recommendation.", "rebuttal": "We are truly grateful for your valuable suggestions. Below are our responses to the questions raised.\n \n**Response to Weekness 1 & Q4: Real-world applicability of 3D SemanticAEs**\n\nIn our supplementary material, we include a simulation video showing adversarial attacks under continuous camera rotation around the 3D object, demonstrating strong multi-view robustness. Following the reviewer's suggestion, we conducted a Blender-based physical simulation using sunlight illumination. The tested model is the forklift model in the supplementary video. Under the viewpoints, material and lighting settings different from the training stage, the ResNet50 classification accuracy on clean samples was 59.41%, while on adversarial samples it dropped to 20.0%. Crucially, the two models are visually indistinguishable, indicating that the generated 3D adversarial objects maintain natural appearance while achieving strong attack performance.\nDue to NeurIPS policy restrictions, we are unable to include visualizations, but we will release the simulation results in the camera-ready version.\n\nAdditionally, a concurrent work at ICCV 2025 [1] (preprinted after our initial submission) demonstrates that adversarial examples in 3D Gaussian representations can effectively enable physical-world attacks. Our semantic-level control advances this direction. We discuss the relationship between that work and ours below. LLM-related application is discussed in our response to Reviewer t5zq.\n\n---\n\n**Response to Weakness 2: Use of Trellis may reduce perceived originality**\n\nWe clarify that our framework is **not limited to Trellis** and can be adapted to other diffusion-based 3D generation models—similar to how Hugging Face Diffusers supports multiple architectures. Trellis serves as one implementation backend, and our design choices prioritize compatibility with existing open-source tools. From the perspective of AI safety governance, we argue that effectively leveraging widely adopted, open models and frameworks is not only realistic but essential. It addresses the economic and scalability challenges in adversarial robustness research, enabling broader access and faster iteration.\n\nMoreover, to the best of our knowledge, no prior work had applied 3D Gaussian Splatting to adversarial example generation. We note that a concurrent ICCV 2025 paper [1] also adopts 3D Gaussian rendering and reports improved physical-world attack performance. While we omit extensive discussion of 3D reconstruction mechanics due to space constraints in the main paper, we believe that since 3D Gaussian Splatting projects 2D gradient feedback onto spherical harmonic bases, it effectively reduces optimization uncertainty and enables efficient inverse optimization of adversarial patterns.\nBy integrating this with ResAdv-DDIM, we establish a *multidimensional uncertainty reduction framework*, stabilizing both the semantic (via residual approximation) and geometric (via 3D Gaussian representation) optimization. We believe this paradigm provides valuable guidance for scaling red-teaming frameworks to complex tasks.\n\n---\n\n**Response to Weakness 3: Perceived incremental nature of some components**\n\nThe core innovation of this paper lies in constructing an integrated robust attack framework through Instruction Uncertainty Reduction. Regarding component-level contributions:\n\n1. By combining residual approximation and adaptive optimization, we establish a new technical solution for transferability enhancement, while unlocking the potential of multi-step diffusion models in improving adversarial transferability and robustness.\n\n2. Motivated by scenario adaptability, we design an effective knowledge injection mechanism for ResAdv-DDIM from both internal sampling model and external guidance perspectives, supporting cross-task applicability.\n\n3. To address ambiguity in evaluation, we propose an exemplar-based metric that alleviates the metric robustness problem posed by Clip-VQA-style assessment in the SemanticAE research. Detailed discussions are provided in Appendix B.3 and Appendix D.4.\n\nRegarding the use of masked guidance, we have discussed its role in Appendix A.1. Masked guidance has been primarily used in diffusion-based image editing to constrain the edited regions, i.e., constraining $x$, with limited exploration on the guidance model $\\epsilon$. As stated in Appendix A.1, *in our 2D generation task, we take a further step by modeling the interaction between conditional and unconditional guidance, achieving the new function of re-distributing the spatial strength of the semantic constraint.* This approach has not yet been explored in the adversarial attack domain.\n\n\n\n---\n\n**Response to Question 1**\n\nThanks for the suggestion. We will provide a more detailed explanation in the camera-ready version. Specifically, at diffusion step $t$, the adversarial update $\\Delta x_t$ is computed as:\n$$\nx_t \\to \\hat{x}_0 \\to \\mathcal{M}(\\hat{x}_0) \\to \\Delta \\hat{x}_0 \\to \\Delta x_t,\n$$\nwhere $\\hat{x}_0 \\sim \\mathrm{ResApprox.}[x_t]$. In principle, the higher the number of internal sampling steps in the $\\mathrm{ResApprox.}$ process, the more accurate the estimation of $\\hat{x}_0$, leading to lower variance $\\mathrm{Var}[\\Delta x_t]$. This results in more stable optimization, as reflected by fewer abrupt drops in attack strength between consecutive denoising steps. We verify this phenomenon in the following supplementary experiments.\n\nFurthermore, our contribution goes beyond improving $\\hat{x}_0$ estimation: by enabling stronger adversarial optimization in early denoising steps via ResAdv-DDIM, we enhance the transferability of the final adversarial sample. Since early steps determine the global structure of the generated image, focusing on attack optimization in early steps leads to more structural and semantically stable features, which are more likely to transfer across models.\n\nWe do not modify the momentum mechanism used in prior adversarial optimization methods. To isolate the effect of our design, we keep the momentum parameter identical to the baseline (VENOM). A brief review of momentum-based transfer attacks and their mechanisms is provided in Appendix A.3, and detailed implementation is described in Appendix B.1. The choice of parameter $\\xi$ is discussed in our response to Reviewer rjVJ’s Question 2.\n\n---\n\n**Response to Question 2**\n\nThanks for the suggestion. We analyze convergence stability by measuring the mean and fluctuation of the estimated confidence of successful attack across different denoising stages of the diffusion model. Due to NeurIPS policy restrictions, we are unable to include visual plots. The quantitative results are presented below. When $k=0$, the sampler degrades to DDIM.\n\n---\n\n**Response to Question 3**\n\nOur language instructions are formally defined in the evaluation methodology section (Section 3.4, Eq. 10). The specific label configurations are listed in the Appendix. Regarding instruction diversity, we conduct experiments under different levels of *referring diversity* to evaluate robustness. The results are shown below.\n\n---\n**Supplementary Experiment**\n\nWe evaluate the impact of different *referencing diversity* levels and varying *residual approximation* steps $k$. The surrogate model is ResNet50, and the diffusion model is `stabilityai/stable-diffusion-2-1-base`. We define four prompt sets with decreasing referencing diversity:\n\n- P0: *a dog on the floor*  \n- P1: *a dog sitting on the floor*  \n- P2: *a brown Labrador sitting on the wooden floor, facing and gazing us directly*\n\nThe experiment is conducted 10 trials for each setting with random seeds from 0 to 9.\nTo measure optimization stability, we define **Fluctuations** as the cumulative drop in attack success rate (ASR) during the denoising process $t = [T, T-1, ..., 0]$:  \n$$\n\\text{Fluctuations} = \\sum \\|\\mathrm{clip}_{\\min=0}(ASR[t] - ASR[t-1])\\|^2\n$$  \nAdditionally, we compute the **Estimated ASR**—the predicted success rate after removing adversarial optimization in later steps—as a measure of the effectiveness of the attack at step $t$.\n\nThe experimental results are summarized in the table below.\n\n\n|Prompt| k| Fluctuations $\\downarrow, t\\in [750, 500) $ | Estimated ASR$\\uparrow, t\\in [750, 500)$ | Fluctuations $\\downarrow, t\\in [500, 250)$|Estimated ASR$\\uparrow, t\\in [500, 250)$ | Fluctuations $\\downarrow,t\\in [250, 0]$ | Estimated ASR$\\uparrow,t\\in [250, 0]$ |\n|-|-|-|-|-|-|-|-|\n|P0|0|0.1843|$\\underline{0.611}$|0.1473|0.674|0.1451|0.817|\n|P0|1|0.2036|0.538|0.0843|0.560|0.0805|0.795|\n|P0|2|**0.1375**|0.560|**0.0322**|$\\underline{0.727}$|$\\underline{0.0080}$|**0.934**|\n|P0|3|$\\underline{0.1421}$|**0.636**|$\\underline{0.0578}$|**0.767**|**0.0042**|$\\underline{0.877}$|\n|||||||||\n|P1|0|0.2110|0.561|0.0722|0.617|0.0215|0.833|\n|P1|1|0.2074|0.491|0.1225|0.524|0.0559|0.781|\n|P1|2|$\\underline{0.1214}$|**0.637**|**0.0088**|$\\underline{0.806}$|$\\underline{0.0202}$|$\\underline{0.894}$|\n|P1|3|**0.1017**|$\\underline{0.635}$|$\\underline{0.0170}$|**0.865**|**0.0005**|**0.940**|\n|||||||||\n|P2|0|0.1025|0.369|0.0570|0.369|0.0256|0.602|\n|P2|1|0.1687|0.355|0.0626|0.365|$\\underline{0.0128}$|0.672|\n|P2|2|$\\underline{0.0660}$|$\\underline{0.446}$|$\\underline{0.0206}$|$\\underline{0.567}$|0.0241|$\\underline{0.853}$|\n|P2|3|**0.0279**|**0.627**|**0.0122**|**0.814**|**0.0070**|**0.930**|\n\nOverall, as $k$ increases, fluctuations decrease and Estimated ASR improves across denoising stages. Prompts with higher specificity induce smaller fluctuations, while more ambiguous prompts lead to larger fluctuations, especially when $k=0$. Due to stronger semantic constraints, more precise prompts may result in lower average ASR. We will incorporate these analysis, along with visualization results, into the main text in the camera-ready version.\n\n\n[1] Lou T, Jia X, Liang S, et al. 3D Gaussian Splatting Driven Multi-View Robust Physical Adversarial Camouflage Generation[J]. arXiv preprint arXiv:2507.01367, 2025."}]

Author rebuttal:

Meta-review: The paper proposes the InSUR framework for generating semantically constrained adversarial examples and addresses instruction uncertainty through residual-driven direction stabilization, context-encoded constraints and semantic-abstracted evaluation.
The reviewers generally agree that the paper is technically solid, novel in framing instruction uncertainty, and supported by extensive experiments in both 2D and 3D settings.
Furthermore, the rebuttal addressed reviewer concerns with additional experiments (including VQA and CLIP-based evaluations), parameter studies, and clarifications on the role of ResAdv-DDIM, Trellis integration and guidance masking. 
After rebuttal and discussion, the consensus leans positive.
The work makes a meaningful contribution to the field of semantic adversarial examples.
Decision: Accept (poster)